The production of peer reviewed scholarship is the single most important activity for professional advancement in academe, including tenure, promotion, and salary increases. The development of software for Humanities Computing has been identified as a crucial need in our field [1], and yet, because of a lack of peer review mechanism for software, computing humanists lack an important incentive to engage fully in programming activities. Why divert valuable time and effort to software development when the payoffs are generally much greater for the production of articles and books? We believe that this conundrum is a dominant factor in the current dearth of specialised text analysis tools.

In addition to encouraging software development by computing humanists – those best suited to understand the needs of the community – a peer review process would establish best practices and guidelines that would likely be useful to most developers. A significant number of developers in the Humanities are programming autodidactics who have had little or no formal training in software development and who would especially benefit from guidelines established specifically for the circumstances of Humanities Computing. Guidelines would also be an extremely valuable resource for instructors who wish to teach programming to Liberal Arts students (computer science courses tend to focus on business, scientific and engineering problems using lower-level, strongly-typed languages). Though guidelines for software development could be formulated independently, a peer review mechanism would provide a strong impetus for doing so.

Peer review would also promote a sustained discussion on the software needs of the Humanities Computing community, and future directions for development. Assessing the value of a particular piece of software assumes a reasonable notion of what already exists and what would be most useful to have. A review process would be an opportunity for both reviewers and developers to examine critically the current state of affairs and reflect on where efforts would best be concentrated.

The theoretical and practical challenges to formulating a peer review process for software development are numerous. In order to consider whether or not software should even be considered for peer review - as Humanities Computing scholarship - we need to recognise that software packages can be complex objects consisting of multiple dynamic parts, including:

• code (new, modified, or integrated)
• interface (design, frameworks for various platforms or languages)
• documentation (as comments in code, APIs or instructions for developers, and help for end-users)
• other documents (research statements, technical articles, representative results)

Assessment of each of these components, together or separately, can be further complicated by the circumstances of the development team: project leaders who may not have done any coding, research assistants who may not have contributed to more recent versions, components that may have been developed by external contractors, etc. Software resources tends to be organic entities that resist the fixity to which review of published materials is accustomed, and perhaps dependent.

Meshed with the challenges of defining the nature of the object(s) to be reviewed is the question of who would be qualified and appropriate to do the reviewing of which parts, and based on what general principles and criteria. The relatively small number of developers in Humanities Computing poses a heightened risk of biased reviews, at least for the code component. However, professional programmers (in the public or private sectors) may not be an appropriate alternative, since their concerns and priorities would likely be quite different.

There are useful materials that can provide some guidance as we work through these issues, such as guidelines produced by the MLA [2], and "The Stoa: A Consortium for Electronic Publication in the Humanities" http://www.stoa.org/.

This cluster of articles will examine the issues surrounding software development in Humanities Computing, and explore possible mechanisms for peer review. Among the questions that the collection considers and attempts to address are the following:

1. Can software tools be considered original contributions to the field comparable to other contributions?
2. Practically speaking, can software be reviewed? Can peers be found who can review software and would they do it? What would they be expected to do when reviewing?
3. What exactly would be reviewed? Functionality, appropriate ease of use for humanists, new algorithms, multimedia...
4. Peer review is usually part of a larger process of publication. How would peer review intergrate into a publication cycle? What outcomes would there be for a community that supported this?

Despite the challenges involved, the Humanities Computing community is in urgent need of software that is worthy of the sophisticated text encoding schemes that exist for editing and publication. In order to spark a concerted and sustained effort of development, we need to establish mechanisms to recognise institutionally the time and effort that are required, and the valuable contribution to Humanities Computing scholarship that well developed software represents.

Notes:

1) For enlightening details on the history and function of peer review, see "Peer Review and Imprint" in The Credibility of Electronic Publishing: A Report to the Humanities and Social Sciences Federation of Canada, Ray Siemens (Project Co-ordinator), http://web.mala.bc.ca/hssfc/Final/PeerReview.htm.

2) See especially the 2000 MLA report entitled "Guidelines for Evaluating Work with Digital Media in the Modern Languages," http://www.mla.org/reports/ccet/ccet_guidelines.htm.

• (comment from StefanSinclair - 15 Jun 2006 00:54:28): I'm not sure if it makes sense to keep this as an introduction or to spawn a more autonomous article. In any case, I do need to bring it up-to-date, mentioning things like http://tada.mcmaster.ca/