Tooling Around Before: Reverse Engineering a Peer Review Process of Techniques

Previously “Tool or/and Die” Geoffrey Rockwell, McMaster University Delivered as part of a panel on “Peer Review of Humanities Computing Software” at the ACH/ALLC in June, 2003 at Athens, Georgia.

One way to approach the issue of peer review of tools is to ask what difference it would make if there were an accepted peer review process in place? Imagine that we had a functioning process of peer review in place 10 years from now, what would be different? What would we gain? I will propose three welcome outcomes that could define what peer review of tools could be.

Encouraging Development

First, and the most obvious outcome of peer review of tools would be that academic staff would be rewarded for the development of tools. Given the importance of peer review to the system of assessment and rewards in the academy, were there a viable and acceptable process there would probably be more development in the academy. This would be especially true of tools that are too specialized to have any commercial potential.

As it is, most development in the academy is ad hoc. There are few incentives to do more than what you need to get results and publication. Most tools are developed for a particular project – custom programming for a particular problem. Such tools are not generalized – made useful and available to the rest of us, because there is no incentive to do the extra work once the research is enabled, research for which one does get promoted.

Were there an appropriate review process we would be more likely to see useful tools in the field. In particular we would be more likely to see tools by and for research for which other forms of development support are not available.

Erase the Difference

The second welcome outcome would be to erase one of the differences between electronic research products and print products. There is no a-priori reason the products of original research in one form, print, should be treated differently from other performances . There are all sorts of reasons that print publications are priviledged, they are easier to archive, we are used to assessing their value, they come in standard chunks, but none of these are good reasons to deny that digital contributions are valuable research products. Were tool review accepted in the field it would make it easier to argue the case of the variety of electronic artifacts that we are justifiably proud of from electronic editions to Web sites. Tools are the extreme case, it is easier to argue that an electronic edition is comparable in research contribution to a print publication. With the extreme case having an accepted review process then the hybrid works ….

Further, a tool review process would assist in the assessment of hybrid digital contributions. What often is not appreciated in the assessment of complex Web research sites is that they are not just content arranged in a new form, but the design of the software delivery framework is an integral part of the intellectual development of the research work. Decisions taken early on as to what technologies to use, how to deploy them, and how the interface should work are not mere design afterthoughts to be passed off to programming drones – these decisions and the development of the software framework constrain or make possible interpretation. As we know they are rhetorical decisions – how do we want to render information to the screen for who and why? Should tool development get its due respect it would be easier to negotiate review of hybrid products that are our bread and butter – products that combine tools and content in innovative ways.

Developing Technique

Both of the preceding are side effects of peer reviewing tools. They do not, in my mind constitute by themselves a sufficiently compelling case. An outcome that would justify the work of review needs to be not a side effect of review but an outcome of review. In other words the review should itself be a desirable practice not just drudgery we need to do to get something else. Take the review of essays – the activity of reviewing is a formal version of the way we engage essays anyway – we read them, take notes, and assess them in light of what we know and what we are interested in. Is there an analogue to reading-review that applies to tools and is valuable in of itself?

One way to review tools is to try them out and compare what they do with what their developers say they do. In other words a form of disciplined play where you don’t just fiddle with them, but you fiddle with them to figure out how they work and what exactly they do. This is after all what we do when we get new software – we fool around with it to figure out how it works. We test it in order to understand what it might be useful for. In other words we try to understand it as a tool that is ready-at-hand, to use Heidegger’s phrase, ready to be used for some task that will come. It doesn’t really become a tool until we have played with it to understand its potential application to problems. The playing is not the using as tool, it is the preparation for use, the doodling that practices using without an end.

The playing we all do instead of work is not disciplined. The discipline comes in when you replay in a fashion so as to share the setting-aside as ready-at-hand. Playing for others, or before others, is a generous act that contributes to the discipline, and therefore is disciplined in the sense that it prepares others. Playing for others is playing in a way that informs others and allows them to see the tool emerge in its potential. It is showing them how the tool could be used so they could imagine using it for their ends. It is play as public performance – tooling around for others.

The key to disciplined tooling around is the revealing of technique. The play reveals the tool in light of anticipated and unanticipated hermeneutical trajectories.

1. Disciplined play pretends you have a hermeneutical perspective and tries a tool in that light
2. Disciplined play therefore interrogates the hermeneutical presuppositions of a tool
3. Disciplined play is about the match between theory of interpretation and method implemented in tools

Reverse Engineering the Outcome

If encouraging development, erasing boundaries, and developing technique would be welcome outcomes of a process we can now reverse engineer some of the features of a peer review process. The following are some of the features to a review process that would be needed to get the desired outcomes.

1. Documenting Process. The peer review process would have to develop the credibility of current print publication processes. It would have to be something that those who assess academic performance could understand and use for purposes of hiring, tenure, and promotion. The onus will initially be on those who claim to do academically sound reviews of tools to explain how the review process works and how it is comparable to review for publication. We will need to document these processes in a fashion that they will achieve the desired goal. This is also true of peer review processes for online journals. For a process to be credible and therefore have the intended effect on administrative processes will we need to explain reviews are done and explain that they are done to our colleagues expectations.
2. Association Defined Processes. To legitimize peer review software venues it would be useful if the Association for Computing in the Humanities defined an acceptable process in general terms that can be used by particular venues. The development of the process in the abstract is what Associations like the ACH should do.
3. Starting by Imitating Print Processes. For new media tools and publications to achieve the credibility of print publications, review processes will have to be comparable to print process, at least initially. While eventually they may work differently, they will have to evolve from something that looks indisputably like a print process to something that the community negotiates as a acceptable.
4. Interesting Tasks. Review processes that can further research should be made up of tasks we are likely to do anyway, though in a more disciplined form. Any task in the process that is not interesting to do will add to the cost of the process.
5. Reviewing the Application of Theory. The review of tools should not just look at code, but be capable of reviewing the intersection of theory, methodology technique and tool implementation. One thing that should be reviewed is the implementation of the tool in light of the creator’s discussion of its hermeneutical purpose. This would make the review process uniquely a humanities process – the review would be of the implementation in code of methodology of interpretation.
6. Valuing Reflection on Theory and Tools. The theoretical discussion of tools should itself become a research topic in humanities computing. We should be willing, through an open discussion of the place of computing in the humanities, to reopen basic questions about knowledge – its form, its acquisition through methods, and its transmission. This is, after all, what has happened around text encoding – it has reopened the question of what a text is. The review of tools can remind humanists that publication is not an end in itself and that there are other forms of valuable research expression including software.

Conclusion

To conclude, the point of developing peer review of humanities computing software is not to develop one process that guarantees one type of outcome. Just as different journals have different criteria for acceptance, what we want is for there to be different peer review outlets that use different criteria. The process might be similar, but the criteria should be different from one venue to another.

Comments

Steve Ramsay says . . .

I'm not sure if anyone has thought about the order of the essays, but this one strikes me as belonging in either first or last place. It give a solid, concrete overview of what we're talking about. But see the second of my two points:

1. "Tools are the extreme case, it is easier to argue that an electronic edition is comparable in research contribution to a print publication. With the extreme case having an accepted review process then the hybrid works …."

I agree (I think) with the general point being made, but I definitely want to hear more. I'm also not sure that resolution of the ordinary case would resolve the extreme case.

2. "It doesn’t really become a tool until we have played with it to understand its potential application to problems. The playing is not the using as tool, it is the preparation for use, the doodling that practices using without an end."

This, and what follows, constitutes a bold philosophical statement about which I want to hear more. In fact, I wonder if it would be possible (sorry, Geoff) to make the essay be about this process -- this "doodling" that is somehow apart from normal software testing, but which may form a useful analogue to the review of essays.

• (comment from GeoffreyRockwell - 23 Dec 2005 15:13:35): Geoffrey Rockwell says ...

Steve - Good idea # 2. I will see what I can do. I think you have written on this too and will try to connect to your paper at the ACH 05.