1. Research Using Web Sources. Academics have serious anxieties about letting students use Web sources in traditional research instead of print resources. Colleagues will ban the Wikipedia as a source in the fond hope that that will force students into the library (which of course is rapidly divesting itself of print resources in order to get more electronic resources.) A reasonable and common approach is to help student understand how to assess Web sources. The second link if you Google "web research" is A Student's Guide to Research with the WWW, a guide that "will help you explore the resources of the World Wide Web for your research, and introduce you to some strategies for evaluating Web sites." Numerous books are also available on this, like Find It Online, though they are often aimed at a broader audience than just students. (See also Online Research for High School and College Students.) The seventh Google result is an example of another type of response to the issue of quality research on the web, it is A Reader's Guide to Canadian Military History put together by the Library and Archives of Canada to guide readers to appropriate resources on a topic.

See Also

• Anatomy of Web Intelligence by Stéfan Sinclair
• Strategic Web intelligence interface Technologies is a draft proposal for commercialization research

• Crawlers and Spiders are tools that harvest information off the Web, often crawling for pages to index for search engines. They are programs that run unattended and are also called bots, ants, or spiders. openkapow is a tool for developers to create robots (bots) crawl the web and gather results.

• Text Mining tools like T2K - Text to Knowledge and Philomin let you discover patterns in large collections of documents.

See also Anatomy of Web Intelligence a white paper that outlines the types of tools in detail.

• News Services like the CNW Group or Cormex Research provide customized news feeds (and research) as a service.

• It is accessible to crawlers, bots, search engines and aggregators so it can be harvested into study sets

1. Change Research. Research to help organizations deal with the disruptive changes brought about by the Internet and the Web. For example, if you Google.ca "web research" the top link is to a New Zealand company, WEB Research, that helps "organisations create a platform for innovation during periods of rapid change and uncertainty."(What WEB Does.)

1. Research Using Web Sources. Academics have serious anxieties about letting students use Web sources in traditional research instead of print resources. Colleagues will ban the Wikipedia as a source in the fond hope that that will force students into the library (which of course is rapidly divesting itself of print resources in order to get more electronic resources.) A reasonable and common approach is to help student understand how to assess Web sources. The second link if you Google "web research" is A Student's Guide to Research with the WWW, a guide that "will help you explore the resources of the World Wide Web for your research, and introduce you to some strategies for evaluating Web sites." Numerous books are also available on this, like Find It Online, though they are often aimed at a broader audience than just students. The seventh Google result is an example of another type of response to the issue of quality research on the web, it is A Reader's Guide to Canadian Military History put together by the Library and Archives of Canada to guide readers to appropriate resources on a topic.

1. Web Usability Research. Research into how the Web works at the micro level - how people behave when using the web. The third link on Google is to an Alertbox by Jakob Nielsen on Web Research: Believe the Data.

1. Research about the Web. Research at the macro level about Internet statistics, how the Web is structured, social networking on the Web, and about Semantic Web research. The fourth and sixth results in Google.ca, for example, link to Semantic Web Research sites, but there are other types of research about the web, like all the research listed on the Complete Guide to Internet Statistics and Research site. This is obviously related to sense 3, but is on a different scale and comes from a different tradition. Web usability research is usually coming out of Human Computer Interface research, while research about the web has elements of market research, network research, and statistics. Web usability often uses eyetracking to look at how individuals use a web page, research about the web looks at traffic logs and often graphs them into cyberatlases.

1. Web Mining. Research that uses the Web as a source of evidence. This is different from 2 - using Web resources, but in traditional craft research ways. Web mining harvests (mines) the Web on a large scale for information and then applies text analysis and statistical techniques to the resulting repositories of unstructured data. Web mining treats the Web as a whole as evidence rather than as an electronic library. It is a form of research that has obvious applications to strategic business intelligence and marketing, but is also relevant to humanists and social scientists. The ninth result from Google, the Chilean Centre for Web Research, for example, is doing basic research on "Data mining" and "Data extraction from the web." This is the sense that is developed in this white paper.

• Data Entry Services like Hi-Tech Export which advertises on Google.ca that they do "Online data collection, website mining services in India." They offer Web searching, market research, analysis, and mining.

Why the Web?

• It is already digital so it is tractable to text analysis and mining techniques
• It is accessible to crawlers, bots, search engines and aggregators
• Much of the evidence is primary in the sense that it is the uninterpreted opinions and behaviour of individuals as opposed to secondary research evidence. In fact you can find both online.
• It can be considered as published information so you can get personal information without waivers. (That doesn't mean there aren't ethical problems with Web Research.)
• There are a wealth of useful tools for research using the Web like search engines and archives
• Above all, it provides an deep and wide amount of evidence of human communication on just about any subject

1. Research Using Web Sources Academics have serious anxieties about letting students use Web sources in traditional research instead of print resources. Colleagues will ban the Wikipedia as a source in the fond hope that that will force students into the library (which of course is rapidly divesting itself of print resources in order to get more electronic resources.) A reasonable and common approach is to help student understand how to assess Web sources. The second link if you Google "web research" is A Student's Guide to Research with the WWW. " This guide will help you explore the resources of the World Wide Web for your research, and introduce you to some strategies for evaluating Web sites." Numerous books are also available on this, see Find It Online.

1. Web Usability Research Research into how the Web works at the micro level - how people behave when using the web. The third link on Google is to an Alertbox by [[http://www.useit.com/jakob/][Jakob Nielsen] on Web Research: Believe the Data.

Examples of Web Mining for Research

• Studying Popular Culture is possible using the Web. You can track the controversy around the Dixie Chicks and their comments about George W. Bush through the news stories and blogs about them.

• Studying the Epidemiology of Ideas is possible using the Web. You can look at how "postmodernism" appears in academic discourse - how it is taught and how it discussed outside the academy.

• Studying Everyday Language Use is possible with the Web. You can gather examples of usage of words or patterns in blogs or discussion lists.

• Studying a Brand like "Nike" is possible by gathering examples of how the brand is discussed by a target population in their blogs or web sites.

• Studying a Community like Hamilton teens is possible if you can identify teen web sites and blogs from Hamilton. What are they interested in? How are their concerns different from Toronto teens?

Philosophical Analysis and the Web

Philosophical analysis is the analysis of concepts. Concepts are words in their sites. Sites include sentences, uttered or transcribed, always in a larger site of neighborhood, institution, authority, language. If one took seriously the project of philosophical analysis, one would require a history of the words in their sites in order to comprehend what the concept was. But isn't "analysis" a breaking down, a decomposition into smaller parts, atoms? Not entirely; for example, "analysis" in mathematics denotes the differential and integral calculus, among other things. Atomism is one kind of analysis ... (p. 68)

Web mining and text analysis lets us look at "words in their sites", something philosophers have only done anecdotally until now. It isn't the end of philosophical analysis, but it is a way to get at the way concepts are used.

Tools and Services

What tools are there to help with Web mining research? Some of the types of tools and services include:

• News Services like the CNW Group provide customized news feeds as a service.

• Data Entry Services like Hi-Tech Export which offers Web research and mining including sarching, market research, analysis, and mining.

• Crawlers and Spiders are tools that harvest information off the Web, often crawling for pages to index for search engines. They are programs that run unattended and are also called bots, ants, or spiders. http://openkapow.comopenkapow]] is a tool for developers to create robots (bots) crawl the web and gather results.

• Metasearch Engines are tools that gather results from other search engines and indexes providing one-stop searching. Dogpile is an example. HighBeam Research is another that is optimized for research.

• Aggregators are tools that gather information from multiple web sites into a collection for analysis. The TAPoRware [[http://taporware.mcmaster.ca/~taporware/otherTools/googlizer.shtml][Googlizer] and Aggregator are examples.

• Web Clipping tools like NetSnippets let you clip Web pages or passages from them to a personal collection that can then be organized and shared.

• Text Analysis tools like those available from the Portal let you analyze electronic texts.

• Text Mining tools like T2K - Text to Knowledge let you discover patterns in large collections of documents.

Conclusions

The Summit on Digital Tools for the Humanities called for the development of four types of tools including tools for the "Exploration of Resources" which would allow for the aggregation of studysets, the sharing of studysets, the exploration of large studysets, and the visualization and presentation of such sets. (Final Report) At the moment people who do Web research, whether informally or formally, do so using what is at hand from Google to personal text analysis tools. The potential for Web mining for research depends on:

• the discussion of web research methodologies and the web as evidence,
• the development of better tools, especially gathering and aggregating tools, and
• these tools being interoperable so researchers can combine them in innovative ways.

Research in the digital age will not simply be a matter of using online facsimiles of print resources, or creating ever larger textbases of the same stuff we studied before. Digital research has to consider the Web as evidence of human behaviour, art and commerce. Our research practices have to evolve to not just use the web but to think through it.

Bibliography

Complete Guide to Internet Statistics and Research This site is a good place to start looking for statistics about the Web.

Hacking, Ian. Historical Ontology. Harvard University Press: Cambridge, MA, 2002. HUP Site

Jones, Steve. Doing Internet Research: Critical Issues and Methods for Examining the Net. SAGE: Thousand Oaks, CA, 1999. On Amazon

Newhagen, John E. and Rafaeli Sheizaf. "Why Communication Researchers Should Study the Internet: A Dialogue". (Originally in Journal of Computed-Mediated Communication [Online], 1996, vol. 1, no. 4.) Now available Online. Also at Blackwell Synergy.

Schlein, Alan M. Find It Online. 4th Ed. Fact on Demand Press: Tempe, AZ, 2004. On Amazon

Summit on Digital Tools for the Humanities: Report on Summit Accomplishments

Zillman, Marcus P. White Papers by Marcus P. Zillman includes links to white papers and annotated link compilations on subjects like Business Intelligence Online Resources.

Credits

Thanks to Terry Flynn, Stéfan Sinclair, Laurence Mussio and others of the SWiiT group for help with this.

Web Mining for Research

TOC: No TOC in "Main.WhatTM"

The Web has the following features that make it useful evidence for research into contemporary culture, politics, marketing, and ideas:

• Much of the evidence is primary in the sense that it is the uninterpreted opinions and behaviour of individuals as opposed to secondary research evidence. In fact you can find both online.
• It can be considered as published information so you can get personal information without waivers. (That doesn't mean there aren't ethical problems with Web Research.)
• Above all, it provides an deep and wide amount of evidence of human communication on just about any subject

As Newhagen and Rafaeli put it in a dialogue published in 1996 online:

Not only does it occur on a computer, communication on the Net leaves tracks to an extent unmatched by that in any other context-the content is easily observable, recorded, and copied. Participant demography and behaviors of consumption, choice, attention, reaction, learning, and so forth, are widely captured and logged. (Newhagen and Rafaeli)

Senses of "Web Research"

1. Change Research Research to help organizations deal with the disruptive changes brought about by the Internet and the Web. For example, if you Google "web research" the top link is to a New Zealand company, WEB Research, that helps "organisations create a platform for innovation during periods of rapid change and uncertainty."What WEB Does. (Note)

1. Research Using Web Sources Academics have serious anxieties about letting students use Web sources in traditional research instead of print resources. Colleagues will ban the Wikipedia as a source in the fond hope that that will force students into the library (which of course is rapidly divesting itself of print resources in order to get more electronic resources.) A reasonable and common approach is to help student understand how to assess Web sources. The second link if you Google "web research" is A Stident's Guide to Research with the WWW. " This guide will help you explore the resources of the World Wide Web for your research, and introduce you to some strategies for evaluating Web sites."

1. Web Usability Research Research into how the Web works at the micro level - how people behave when using the web. The third link on Google is to an Alertbox by [[http://www.useit.com/jakob/][Jakob Nielsen] on Web Resarch: Believe the Data.

1. Research about the Web Research at the macro level about Internet statistics, how the Web is structured, social networking on the Web, and about Semantic Web research. This is obviously related to sense 3, but is on a different scale and comes from a different tradition. Web usability research is usually coming out of Human Computer Interface research, while research about the web has elements of market research, network research, and statistics. Web usability often uses eyetracking to look at how individuals use a web page, research about the web looks at traffic logs and often graphs them into cyberatlases.

1. Web Mining Research that uses the Web as a source of evidence. This is different from 2 - using Web resources, but in traditional craft research ways. Web mining harvests (mines) the Web on a large scale for information and then applies text analysis and statistical techniques to the resulting repositories of unstructured data. Web mining treats the Web as a whole as evidence rather than as an electronic library. It is a form of research The ninth result from Google, the Chilean Centre for Web Research, for example, is doing basic research on "Data mining" and "Data extraction from the web." This is the sense that is developed in this white paper.

Defining Web Mining for Research

Web mining is a subset of Internet Research that is sometimes called text mining or web knowledge-discovery in data mining circles. In the hyped language of [[http://en.wikipedia.org/wiki/Knowledge_management][knowledge management], knowledge-discovery is the extraction of useful information from heterogeneous data-sets like the Web. Because the Web is mostly unstructured text data with embedded multimedia objects, web mining, typically involves forms of text analysis and mining. We define web mining as having the following phases:

1. Gathering Web mining involves gathering a subset of the whole web as a study repository. In principle it is possible to use the whole Web and its archives, but that is not practical so most web mining involves carving out a subset that is amenable to analysis and mining technologies. In cases where the studyset is a set of changing documents like RSS feeds it can be called a stream. Gathering is usually done with scrapers or semi-automated techniques.

1. Structuring Repositories or streams are often enhanced with metadata, edited to remove irrelevant data, and structured so that they can be manipulated in the analysis phase. Typically repositories are so large they are also pre-indexed for rapid search, retrieval, and mining.

1. Analysis Finally, repositories are studied using text analysis techniques and mining techniques.

Web mining for research has evolved out of social science and communications research practices that looked at identity and social communication using computer-mediated communications like e-mail and chat transcripts.

Bibliography

White Papers by Marcus P. Zillman includes links to white papers and annotated link compilations on subjects like Business Intelligence Online Resources.

Complete Guide to Internet Statistics and Research This site is a good place to start looking for statistics about the Web.

Mann, Chris and Stewart, Fiona. Internet Communication and Qualitatieve Research: A Handbook for Researching Online. SAGE: London, 2000. On Amazon

Schlein, Alan M. Find It Online. 4th Ed. Fact on Demand Press: Tempe, AZ, 2004. On Amazon

Jones, Steve. Doing Internet Research: Critical Issues and Methods for Examining the Net. SAGE: Thousand Oaks, CA, 1999. On Amazon

Neuendorf, Kimberly A. The Content Analysis Gudiebook. SAGE: Thousand Oaks, CA, 2002. On Amazon

Newhagen, John E. and Rafaeli Sheizaf. "Why Communication Researchers Should Study the Internet: A Dialogue". (Originally in Journal of Computed-Mediated Communication [Online], 1996, vol. 1, no. 4.) Now available Online. Also at Blackwell Synergy.

Journal of Computer-Mediated Communication has articles both about Internet communications research methods and articles that are based on computer-mediated communication methods.

Moretti, Franco. Graphs, Maps, Trees: Abstract Models for Literary History. Verso, 2005. On Amazon

Notes

Web Mining for Research

The Web provides an unprecedented opportunity for research. The Web has the following features that make it useful evidence for research into contemporary culture:

• It is digital so it is tractable to text analysis techniques
• It is accessible to scrapers, bots, search engines and aggregators
• Much of the evidence is primary in the sense that it is the uninterpreted opinions as opposed to secondary research evidence. In fact you can find both online.
• Much of it

But what is Web Research?

Web Research typically refers to research about the Web or for the Web. Here I will focus on using the Web as evidence to study contemporary politics, culture, and ideas.

Some of the senses of Web Research are:

• Change Research Research to help organizations deal with the disruptive changes brought about by the Internet and the Web. For example, if you Google "web research" the top link is to a New Zealand company, WEB Research, that helps "organisations create a platform for innovation during periods of rapid change and uncertainty."What WEB Does.Note 1

• Research Using Web Sources Academics have serious anxieties about letting students use Web sources in traditional research instead of print resources. Colleagues will ban the Wikipedia as a source in the fond hope that that will force students into the library (which of course is rapidly divesting itself of print resources in order to get more electronic resources.) A reasonable and common approach is to help student understand how to assess Web sources. The second link if you Google "web research" is A Stident's Guide to Research with the WWW. " This guide will help you explore the resources of the World Wide Web for your research, and introduce you to some strategies for evaluating Web sites."

Search performed Feb. 22nd, 2007. I should note that using Google this way is an example of Web Research where you use Google's Page Ranking as an indication of popularity.

-- GeoffreyRockwell - 22 Feb 2007