|
The following is the present list of supported Robots and Crawlers.
Robot /Crawler
|
Robot Purpose
|
Googlebot
|
Indexing
|
Inktomi Slurp
|
Indexing, Statistics
|
MSNBot
|
MSN Search Crawler
|
AskJeeves
|
Indexing, Maintenance
|
Scooter (Alta Vista)
|
Indexing
|
ABCdatos BotLink
|
Maintenance
|
Acme.Spider
|
Indexing Maintenance Statistics
|
Ahoy! The Homepage Finder
|
Maintenance
|
Alkaline
|
Indexing
|
ananzi
|
Indexing
|
Anthill
|
Indexing
|
Arachnophilia
|
|
Arale
|
Maintenance
|
Araneo
|
Indexing, Statistics
|
AraybOt
|
Indexing Maintenance
|
ArchitextSpider
|
Indexing, Statistics
|
Aretha
|
|
ARIADNE
|
Statistics, Development of Focused Crawling Strategies
|
arks
|
Indexing
|
ASpider (Associative Spider)
|
Indexing
|
ATN Worldwide
|
Indexing
|
Atomz.com Search Robot
|
Indexing
|
AURESYS
|
Indexing,Statistics
|
BackRub
|
Indexing, Statistics
|
BBot
|
Indexing, Maintenance
|
Big Brother
|
Maintenance
|
Bjaaland
|
Indexing
|
BlackWidow
|
Indexing, Statistics
|
Bloodhound
|
Web Site Download
|
Borg-Bot
|
Indexing Statistics
|
BoxSeaBot
|
Indexing
|
bright.net caching robot
|
caching
|
BSpider
|
Indexing
|
CACTVS Chemistry Spider
|
Indexing
|
Calif
|
Indexing
|
Cassandra
|
Indexing
|
Checkbot
|
Maintenance
|
ChristCrawler.com
|
Indexing
|
churl
|
Maintenance
|
cIeNcIaFiCcIoN.nEt
|
Indexing
|
CMC/0.01
|
Maintenance
|
Collective
|
|
Combine System
|
Indexing
|
ComputingSite Robi/1.0
|
Indexing, Maintenance
|
ConfuzzledBot
|
Indexing of luxembourgish sites
|
CoolBot
|
Indexing
|
Cusco
|
Indexing
|
CyberSpyder Link Test
|
link validation, some html validation
|
CydralSpider
|
gather Web content for image search engine service
|
Desert Realm Spider
|
Indexing
|
DeWeb(c) Katalog/Index
|
Indexing, Mirroring, Statistics
|
Die Blinde Kuh
|
Indexing
|
DienstSpider
|
Indexing
|
Digger
|
Indexing
|
Digimarc MarcSpider
|
Maintenance
|
Digital Integrity Robot
|
WWW Indexing
|
Direct Hit Grabber
|
Indexing and Statistics
|
DNAbot
|
Indexing
|
DownLoad Express
|
graphic download
|
DragonBot
|
Indexing
|
DWCP (Dridus' Web Cataloging Project)
|
Indexing, Statistics
|
EbiNess
|
Statistics
|
e-collector
|
email collector
|
EIT Link Verifier Robot
|
EIT-Link-Verifier-Robot/0.2
|
ELFINBOT
|
Indexing for the Lets Find It Now search Engine
|
Emacs-w3 Search Engine
|
Indexing
|
esculapio
|
link validation
|
Esther
|
Indexing
|
Evliya Celebi
|
Indexing turkish content
|
FastCrawler
|
Indexing
|
Felix IDE
|
Indexing, Statistics
|
FetchRover
|
Maintenance, Statistics
|
fido
|
Indexing
|
Fish search
|
Indexing
|
Fluid Dynamics Search Engine robot
|
Indexing
|
Fouineur
|
Indexing, Statistics
|
Freecrawl
|
Indexing
|
FunnelWeb
|
Indexing, statisitics
|
gammaSpider, FocusedCrawler
|
Indexing, Maintenance
|
gazz
|
Statistics
|
GCreep
|
Indexing
|
GetBot
|
Maintenance
|
GetterroboPlus Puu
|
Indexing
|
GetURL
|
Maintenance, Mirroring
|
Golem
|
Maintenance
|
Grapnel/0.01 Experiment
|
Indexing
|
Griffon
|
Indexing
|
Gromit
|
Indexing
|
Gulper Bot
|
Indexing
|
Hamahakki
|
Indexing
|
HamBot
|
Indexing
|
Harvest
|
Indexing
|
havIndex
|
Indexing
|
HI (HTML Index) Search
|
Indexing
|
HKU WWW Octopus
|
Indexing
|
Hometown Spider Pro
|
Indexing
|
ht://Dig
|
Indexing
|
html_analyzer
|
maintainance
|
HTMLgobble
|
mirror
|
Hyper-Decontextualizer
|
Indexing
|
I, Robot
|
Indexing
|
iajaBot
|
Indexing
|
IBM_Planetwide
|
Indexing, Maintenance, Mirroring
|
image.kapsi.net
|
Indexing
|
Imagelock
|
Maintenance
|
IncyWincy
|
|
Informant
|
Indexing
|
InfoSeek Robot 1.0
|
Indexing
|
Infoseek Sidewinder
|
Indexing
|
InfoSpiders
|
search
|
Ingrid
|
Indexing
|
Inspector Web
|
Maintenance: link validation, html validation, image size validation, etc
|
IntelliAgent
|
Indexing
|
Internet Cruiser Robot
|
Indexing
|
Internet Shinchakubin
|
Locate New Links and Changed Pages
|
Iron33
|
Indexing, Statistics
|
Israeli-search
|
Indexing
|
JavaBee
|
Stealing Java Code
|
JBot Java Web Robot
|
Java web crawler to download web sites
|
JCrawler
|
Indexing
|
JoBo Java Web Robot
|
downloading, Mirroring, Indexing
|
Jobot
|
Indexing
|
JoeBot
|
|
JumpStation
|
Indexing
|
Katipo
|
Maintenance
|
KDD-Explorer
|
Indexing
|
Kilroy
|
Indexing,Statistics
|
KIT-Fireball
|
Indexing
|
KO_Yappo_Robot
|
Indexing
|
LabelGrabber
|
Grabs PICS labels from web pages, submits them to a label bueau
|
larbin
|
General
|
legs
|
Indexing
|
Link Validator
|
Maintenance
|
LinkScan
|
Link checker, SiteMapper, and HTML Validator
|
LinkWalker
|
Maintenance, Statistics
|
Lockon
|
Indexing
|
logo.gif Crawler
|
Indexing
|
Lycos
|
Indexing
|
Mac WWWWorm
|
Indexing
|
Magpie
|
Indexing, Statistics
|
marvin/infoseek
|
Indexing
|
Mattie
|
Procurement Spider
|
MediaFox
|
Indexing and Maintenance
|
MerzScope
|
WebMapping
|
MindCrawler
|
Indexing
|
mnoGoSearch
|
Indexing
|
moget
|
Indexing, Statistics
|
MOMspider
|
Maintenance, Statistics
|
Monster
|
Maintenance, Mirroring
|
Motor
|
Indexing
|
Muncher
|
Indexing
|
Muninn
|
Indexing
|
Muscat Ferret
|
Indexing
|
Mwd.Search
|
Indexing
|
NDSpider
|
Indexing
|
NEC-MeshExplorer
|
Indexing
|
Nederland.zoek
|
Indexing
|
NetCarta WebMap Engine
|
Indexing, Maintenance, Mirroring, Statistics
|
NetMechanic
|
Link and HTML validation
|
NetScoop
|
Indexing
|
newscan-online
|
Indexing
|
NHSE Web Forager
|
Indexing
|
Nomad
|
Indexing
|
Northern Light Gulliver
|
Indexing
|
nzexplorer
|
Indexing, Statistics
|
ObjectsSearch
|
Indexing
|
Occam
|
Indexing
|
OntoSpider
|
Statistics
|
Openfind Data Gatherer
|
Indexing
|
Orb Search
|
Indexing
|
Pack Rat
|
Maintenance and Mirroring
|
PageBoy
|
Indexing
|
ParaSite
|
Indexing
|
Patric
|
Statistics
|
Pegasus
|
Indexing
|
PerlCrawler
|
Indexing
|
Phantom
|
Indexing
|
PhpDig
|
Indexing
|
PiltdownMan
|
Statistics
|
Pimptrain.com's robot
|
Indexing
|
Pioneer
|
Indexing, Statistics
|
Poppi
|
Indexing
|
Popular Iconoclast
|
Statistics
|
Portal Juice Spider
|
Indexing
|
PortalB Spider
|
Indexing
|
psbot
|
Indexing
|
Raven Search
|
Indexing
|
RBSE Spider
|
Indexing, Statistics
|
Resume Robot
|
Indexing
|
RixBot
|
Indexing
|
Road Runner: The ImageScape Robot
|
Indexing
|
RoadHouse Crawling System
|
Indexing
|
Robbie the Robot
|
Indexing
|
RoboCrawl Spider
|
Indexing
|
RoboFox
|
Site Download
|
Robot Francoroute
|
Indexing, Mirroring, Statistics
|
Robozilla
|
Maintenance
|
Roverbot
|
Indexing
|
RuLeS
|
Indexing
|
SafetyNet Robot
|
Indexing
|
Search.Aus-AU.COM
|
Indexing
|
SearchProcess
|
Statistics
|
Senrigan
|
Indexing
|
SG-Scout
|
Indexing
|
ShagSeeker
|
Indexing
|
Shai'Hulud
|
Mirroring
|
Sift
|
Indexing
|
Simmany Robot Ver1.0
|
Indexing, Maintenance, Statistics
|
Site Searcher
|
Indexing
|
Site Valet
|
Maintenance
|
SiteTech-Rover
|
Indexing
|
Skymob.com
|
Indexing
|
SLCrawler
|
Sitemap Builder
|
Sleek
|
Indexing
|
Smart Spider
|
Indexing
|
Snooper
|
Indexing
|
Solbot
|
Indexing
|
Speedy Spider
|
Indexing
|
spider_monkey
|
Indexing
|
SpiderBot
|
Indexing, Mirroring
|
Spiderline Crawler
|
Indexing
|
SpiderMan
|
Indexing
|
SpiderView(tm)
|
Maintenance
|
Spry Wizard Robot
|
Indexing
|
Suke
|
Indexing
|
Suntek search engine
|
Indexing
|
Sven
|
Indexing
|
Sygol
|
Indexing
|
TACH Black Widow
|
Maintenance: link validation
|
Tarantula
|
Indexing
|
Tarspider
|
Mirroring
|
Tcl W3 Robot
|
Maintenance, Statistics
|
TechBOT
|
Statistics, Maintenance
|
Templeton
|
Mirroring, Mapping, Automating Web Applications
|
The Jubii Indexing Robot
|
Indexing, maintainance
|
The NorthStar Robot
|
Indexing
|
The NWI Robot
|
Indexing, Statistics
|
The Peregrinator
|
Indexing
|
The Python Robot
|
|
The TkWWW Robot
|
Indexing
|
The Web Moose
|
Statistics, Maintenance
|
TITAN
|
Indexing
|
TitIn
|
Indexing, Statistics
|
TLSpider
|
Indexing
|
UCSD Crawl
|
Indexing, Statistics
|
UdmSearch
|
Indexing, validation
|
UptimeBot
|
Indexing, Statistics
|
URL Check
|
Maintenance
|
URL Spider Pro
|
Indexing
|
Valkyrie
|
Indexing
|
Verticrawl
|
Indexing, Seartching, Classyfing
|
Victoria
|
Maintenance
|
vision-search
|
Indexing
|
void-bot
|
Indexing, Maintenance
|
Voyager
|
Indexing, Maintenance
|
VWbot
|
Indexing
|
w@pSpider by wap4.com
|
Indexing, Maintenance (dedicated to wap/wml pages)
|
W3M2
|
Indexing, Maintenance, Statistics
|
w3mir
|
Mirroring
|
Walhello appie
|
Indexing
|
WallPaper / Crawlpaper
|
Indexing
|
Web Core / Roots
|
Indexing, Maintenance
|
Web Wombat
|
Indexing, Statistics.
|
WebBandit Web Spider
|
Resource Gathering / Server Benchmarking
|
WebCatcher
|
Indexing
|
WebCopy
|
Mirroring
|
webfetcher
|
Mirroring
|
Webfoot Robot
|
|
Webinator
|
Information Retrieval
|
weblayers
|
maintainance
|
WebLinker
|
Maintenance
|
Weblog Monitor
|
Statistics
|
WebMirror
|
Mirroring
|
WebQuest
|
Indexing
|
WebReaper
|
Indexing/ Offline Browsing
|
webs
|
Statistics
|
Websnarf
|
|
WebSpider
|
Maintenance, Link Diagnostics
|
WebStolperer
|
Indexing
|
WebVac
|
Mirroring
|
webwalk
|
Indexing, Maintenance, Mirroring, Statistics
|
WebWalker
|
Maintenance
|
WebWatch
|
maintainance, Statistics
|
Wget
|
Mirroring, Maintenance
|
whatUseek Winona
|
Site-level Search and Meta-Search Engines.
|
WhoWhere Robot
|
Indexing
|
Wild Ferret Web Hopper #1, #2, #3
|
Indexing, Maintenance Statistics
|
World Wide Web Wanderer
|
Statistics
|
World Wide Web Worm
|
Indexing
|
WWWC Ver 0.2.5
|
Maintenance
|
XGET
|
Mirroring
|
XYLEME Robot
|
Indexing
|
Advanced Robots.txt Generator Website.
|