To clarify:
Similar to a spider, but with more of a focus on simply compiling rather than indexing the web content encountered during the process.
The following seem worthy of investigation:
Similar to a spider, but with more of a focus on simply compiling rather than indexing the web content encountered during the process.
The following seem worthy of investigation:
- Heritrix
Heretrix is being used by LAC to crawl the GOvernment of Canada site, as per their legislated mandate.
A user manual is available. - WebSPHINX A mature java class library.
- Nutch
- LARM
- Aperture
- larbin
- WebVac This a PERL-based script used by Stanford's WebBase? Project.
- WIRE - Web Information Retrieval Environment A Chilean project in C++. There is an additional library available for user with Ruby.
- Noviway Web Crawler C#-based
- YaCy A java-based P2P web crawler.
- Ruya A python-based crawler
- Univeral Information Crawler Another Python-based web crawler.
- Squzer Distributed Crawler A final Python Scripted web crawler.