London Tool Meeting Notes
These notes are my impressions of the issues that emerged in the meeting organized at the
AHRC ICT Methods Network at King's College London by Lorna Hughes. These are not an official report. The major issues identified around the creation of research tools for the arts and humanities were:
What do we do? What do we know about what we do?
In order to develop tools that extend or automate research practices we need to know what it is we do. By and large we don't actually know how we do research, or, atleast we could know more. Humanists reflect on methods in an abstract way as a theoretical issue, but there are few studies of what we actually do.
The importance of the social dimension of tools and design research.
The design of tools should be grounded in design research that is attentive to the social dimension of tools. Tools are used in contexts, by groups of people negotiating their use and interpretation. Often tool development is done by contract programmers (often undergraduates) with little knowledge or committment to the research practices of the subject and little interaction with their prospective users. The lucky few academics who get funding often have little knowledge of software engineering, but a personal vision of what is needed. They contract with programmers with little knowledge of the field to produce tools in ways that don't reflect best practices in research project management. This is partly because we have little experience in the humanities with project management, usability research, or software engineering. In principle every tool development project should be a research project in itself, but we need paradigms for this. One paradigm is how our colleagues in design fields approach research where the develolpment of a new design is itself a self-conscious design project that integrates creative and research practices. A good place to start might be the collection edited by Brenda Laurel,
Design Research.
Can we separate data from analysis?
Most computing tools presuppose a clear separation between what is data and what is process implemented by a tool. We need to better theorize this difference and account for situations where the preparation of knowledge rich data is the tool. John Bradley spoke to this text enrichment paradigm and showed PLINY which is build on Eclipse to provide a rich editing, tagging and analysis environment for those who work with electronic text objects. Geraint Wiggins made the point that it should not be about tools but about the underlying data, the knowledge, and the inferences we can make from the knowledge. A tool is just part of a larger knowledge representation issue. My sense is that archaeology is where this is discussed openly in the discipline. Textual disciplines are still comfortable with the library and its virtual versions.
Sustainability and Dissemination
We need models for how to maintain tools. Often tools are created with a grant, but once the grant is spent there is no ongoing funding to upgrade the tool, fix the inevitable bugs, maintain it and provide support. Some of the models floated were:
- Work with commercial developers.
- Create a consortium that is funded by members.
- Sell the software.
- License technology developed in labs to entities capable of commercializing it.
- Sell services based on technology, not the technology.
- Continue getting grants to sustain the project. Advocate for grant programs for sustaining useful research tools.
Connected to this is an issue around "marketing". Researchers are bad at promoting innovation. We tend to create a neat tool, publish on it, but not promote it to those who could use it.
Will the research communty accept new forms of research outcomes?
New tools don't always lead to new insights that can be published in traditional forms. Visualization tools like GIS produce visual outcomes. Acceptance and use of such tools will depend in part on acceptance of the new forms of research output. For this we need to rethink how research results look, get assessed, and get disseminated.
Ric Allsop talked about the relationship between research, print and performance. He has been experimenting with
Liquid Reader to explore alternative forms of publication or performance and performance of publication.
Interoperability and Collaboration
Sheila Anderson from the AHDS pointed out that we are burning knowledge in that projects get funded to create digital knowledge, but that knowledge can't be reused. Digital projects don't share their data, they create it in accessible forms, and we don't have models for interoperability. Likewise tools don't interoperate something John Bradley and I argued for back in 1998 in
Eye-ConTact: Towards a New Design for Text-Analysis Tools.
Is it knowledge if it can't be reused? Are we witnessing the equivalent to book burning for data?
There is a second form of interoperation or collaboration that is important, and that is the collaboration with other disciplines like computer science and information science. The issue of genuine research collaboration between the creative/interpretative disciplines and the sciences has a long history documented in places like the second chapter of
Beyond Productivity.
What would a tool program look like?
Grant programs need not be responsive, they can be creative in the sense that they can encourage movement in the disciplines towards new practices. Here is an outline of what I imagine a tools grant program should look like:
- It should have three funding stages with reports expected at the end of each.
- Prototype Phase (2 years, up to $100K) The first phase would involve prototyping and design research. Grantees would be expected to develop the prototype through a recognized and documented design research process that identifies the audience of the tool and probes the needs of the audience. This doesn't mean developing only things that people think they need, but it does mean the grantee needs to propose a design research methodology suited to the project. Lots of small prototype grants would be given out.
- Development Phase (3 years, up to $350K) At the end of the prototype phase (2 years) projects could apply for funding for a Development Phase. The design research outcomes of the first phase would be used to assess which projects were viable. This phase would be more competetive. Proposals would need to include a software engineering management discussion, usability research plan, testing plan associated with a reference project (see below), and a preliminary dissemination strategy.
- Outreach Phase (5 years, up to $250K) At the end of development projects could apply for funding to promote and maintain the tool. Where the development had met its goals this funding should be automatic. A full dissemination plan should be part of the proposal for Outreach funding.
- Reference Project. Every tool project should be allied with a reference research project in the arts and humanities where the tool will be used for real research.
- The program should be assessed in terms of the outreach into the research community, not just in terms of the tools. In some ways the development of new methods and practices is more important than the tools. A project that developed a great algorithm and made it available for reimplementation in comprehensible pseudocode would be more useful than a black-box tool that no one understood.
- We should expect reflection on tools, methods, knowledge representation, and practices to be part of these projects.
--
GeoffreyRockwell - 27 Jun 2006
