This is the current list of supported robots and crawlers.
Robot /Crawler |
Robot Purpose |
Googlebot |
Indexing |
Inktomi Slurp |
Indexing, Statistics |
MSNBot |
MSN Search Crawler |
AskJeeves |
Indexing, Maintenance |
Scooter (Alta Vista) |
Indexing |
ABCdatos BotLink |
Maintenance |
Acme.Spider |
Indexing Maintenance Statistics |
Ahoy! The Homepage Finder |
Maintenance |
Alkaline |
Indexing |
ananzi |
Indexing |
Anthill |
Indexing |
Arachnophilia |
|
Arale |
Maintenance |
Araneo |
Indexing, Statistics |
AraybOt |
Indexing Maintenance |
ArchitextSpider |
Indexing, Statistics |
Aretha |
|
ARIADNE |
Statistics, Development of Focused Crawling Strategies |
arks |
Indexing |
ASpider (Associative Spider) |
Indexing |
ATN Worldwide |
Indexing |
Atomz.com Search Robot |
Indexing |
AURESYS |
Indexing,Statistics |
BackRub |
Indexing, Statistics |
BBot |
Indexing, Maintenance |
Big Brother |
Maintenance |
Bjaaland |
Indexing |
BlackWidow |
Indexing, Statistics |
Bloodhound |
Web Site Download |
Borg-Bot |
Indexing Statistics |
BoxSeaBot |
Indexing |
bright.net caching robot |
caching |
BSpider |
Indexing |
CACTVS Chemistry Spider |
Indexing |
Calif |
Indexing |
Cassandra |
Indexing |
Checkbot |
Maintenance |
ChristCrawler |
Indexing |
churl |
Maintenance |
cIeNcIaFiCcIoN.nEt |
Indexing |
CMC/0.01 |
Maintenance |
Collective |
|
Combine System |
Indexing |
ComputingSite Robi/1.0 |
Indexing, Maintenance |
ConfuzzledBot |
Indexing of luxembourgish sites |
CoolBot |
Indexing |
Cusco |
Indexing |
CyberSpyder Link Test |
link validation, some html validation |
CydralSpider |
gather Web content for image search engine service |
Desert Realm Spider |
Indexing |
DeWeb(c) Katalog/Index |
Indexing, Mirroring, Statistics |
Die Blinde Kuh |
Indexing |
DienstSpider |
Indexing |
Digger |
Indexing |
Digimarc MarcSpider |
Maintenance |
Digital Integrity Robot |
WWW Indexing |
Direct Hit Grabber |
Indexing and Statistics |
DNAbot |
Indexing |
DownLoad Express |
graphic download |
DragonBot |
Indexing |
DWCP (Dridus' Web Cataloging Project) |
Indexing, Statistics |
EbiNess |
Statistics |
e-collector |
email collector |
EIT Link Verifier Robot |
EIT-Link-Verifier-Robot/0.2 |
ELFINBOT |
Indexing for the Lets Find It Now search Engine |
Emacs-w3 Search Engine |
Indexing |
esculapio |
link validation |
Esther |
Indexing |
Evliya Celebi |
Indexing turkish content |
FastCrawler |
Indexing |
Felix IDE |
Indexing, Statistics |
FetchRover |
Maintenance, Statistics |
fido |
Indexing |
Fish search |
Indexing |
Fluid Dynamics Search Engine robot |
Indexing |
Fouineur |
Indexing, Statistics |
Freecrawl |
Indexing |
FunnelWeb |
Indexing, statisitics |
gammaSpider, FocusedCrawler |
Indexing, Maintenance |
gazz |
Statistics |
GCreep |
Indexing |
GetBot |
Maintenance |
GetterroboPlus Puu |
Indexing |
GetURL |
Maintenance, Mirroring |
Golem |
Maintenance |
Grapnel/0.01 Experiment |
Indexing |
Griffon |
Indexing |
Gromit |
Indexing |
Gulper Bot |
Indexing |
Hämähäkki |
Indexing |
HamBot |
Indexing |
Harvest |
Indexing |
havIndex |
Indexing |
HI (HTML Index) Search |
Indexing |
HKU WWW Octopus |
Indexing |
Hometown Spider Pro |
Indexing |
ht://Dig |
Indexing |
html_analyzer |
maintainance |
HTMLgobble |
mirror |
Hyper-Decontextualizer |
Indexing |
I, Robot |
Indexing |
iajaBot |
Indexing |
IBM_Planetwide |
Indexing, Maintenance, Mirroring |
image.kapsi.net |
Indexing |
Imagelock |
Maintenance |
IncyWincy |
|
Informant |
Indexing |
InfoSeek Robot |
Indexing |
Infoseek Sidewinder |
Indexing |
InfoSpiders |
search |
Ingrid |
Indexing |
Inspector Web |
Maintenance: link validation, html validation, image size validation, etc |
IntelliAgent |
Indexing |
Internet Cruiser Robot |
Indexing |
Internet Shinchakubin |
Locate New Links and Changed Pages |
Iron33 |
Indexing, Statistics |
Israeli-search |
Indexing |
JavaBee |
Stealing Java Code |
JBot Java Web Robot |
Java web crawler to download web sites |
JCrawler |
Indexing |
JoBo Java Web Robot |
downloading, Mirroring, Indexing |
Jobot |
Indexing |
JoeBot |
|
JumpStation |
Indexing |
Katipo |
Maintenance |
KDD-Explorer |
Indexing |
Kilroy |
Indexing,Statistics |
KIT-Fireball |
Indexing |
KO_Yappo_Robot |
Indexing |
LabelGrabber |
Grabs PICS labels from web pages, submits them to a label bueau |
larbin |
General |
legs |
Indexing |
Link Validator |
Maintenance |
LinkScan |
Link checker, SiteMapper, and HTML Validator |
LinkWalker |
Maintenance, Statistics |
Lockon |
Indexing |
logo.gif Crawler |
Indexing |
Lycos |
Indexing |
Mac WWWWorm |
Indexing |
Magpie |
Indexing, Statistics |
marvin/infoseek |
Indexing |
Mattie |
Procurement Spider |
MediaFox |
Indexing and Maintenance |
MerzScope |
WebMapping |
MindCrawler |
Indexing |
mnoGoSearch |
Indexing |
moget |
Indexing, Statistics |
MOMspider |
Maintenance, Statistics |
Monster |
Maintenance, Mirroring |
Motor |
Indexing |
Muncher |
Indexing |
Muninn |
Indexing |
Muscat Ferret |
Indexing |
Mwd.Search |
Indexing |
NDSpider |
Indexing |
NEC-MeshExplorer |
Indexing |
Nederland.zoek |
Indexing |
NetCarta WebMap Engine |
Indexing, Maintenance, Mirroring, Statistics |
NetMechanic |
Link and HTML validation |
NetScoop |
Indexing |
newscan-online |
Indexing |
NHSE Web Forager |
Indexing |
Nomad |
Indexing |
Northern Light Gulliver |
Indexing |
nzexplorer |
Indexing, Statistics |
ObjectsSearch |
Indexing |
Occam |
Indexing |
OntoSpider |
Statistics |
Openfind Data Gatherer |
Indexing |
Orb Search |
Indexing |
Pack Rat |
Maintenance and Mirroring |
PageBoy |
Indexing |
ParaSite |
Indexing |
Patric |
Statistics |
Pegasus |
Indexing |
PerlCrawler |
Indexing |
Phantom |
Indexing |
PhpDig |
Indexing |
PiltdownMan |
Statistics |
Pimptrain.com's robot |
Indexing |
Pioneer |
Indexing, Statistics |
Poppi |
Indexing |
Popular Iconoclast |
Statistics |
Portal Juice Spider |
Indexing |
PortalB Spider |
Indexing |
psbot |
Indexing |
Raven Search |
Indexing |
RBSE Spider |
Indexing, Statistics |
Resume Robot |
Indexing |
RixBot |
Indexing |
Road Runner: The ImageScape Robot |
Indexing |
RoadHouse Crawling System |
Indexing |
Robbie the Robot |
Indexing |
RoboCrawl Spider |
Indexing |
RoboFox |
Site Download |
Robot Francoroute |
Indexing, Mirroring, Statistics |
Robozilla |
Maintenance |
Roverbot |
Indexing |
RuLeS |
Indexing |
SafetyNet Robot |
Indexing |
Search.Aus-AU.COM |
Indexing |
SearchProcess |
Statistics |
Senrigan |
Indexing |
SG-Scout |
Indexing |
ShagSeeker |
Indexing |
Shai'Hulud |
Mirroring |
Sift |
Indexing |
Simmany Robot |
Indexing, Maintenance, Statistics |
Site Searcher |
Indexing |
Site Valet |
Maintenance |
SiteTech-Rover |
Indexing |
Skymob.com |
Indexing |
SLCrawler |
Sitemap Builder |
Sleek |
Indexing |
Smart Spider |
Indexing |
Snooper |
Indexing |
Solbot |
Indexing |
Speedy Spider |
Indexing |
spider_monkey |
Indexing |
SpiderBot |
Indexing, Mirroring |
Spiderline Crawler |
Indexing |
SpiderMan |
Indexing |
SpiderView(tm) |
Maintenance |
Spry Wizard Robot |
Indexing |
Suke |
Indexing |
Suntek search engine |
Indexing |
Sven |
Indexing |
Sygol |
Indexing |
TACH Black Widow |
Maintenance: link validation |
Tarantula |
Indexing |
Tarspider |
Mirroring |
Tcl W3 Robot |
Maintenance, Statistics |
TechBOT |
Statistics, Maintenance |
Templeton |
Mirroring, Mapping, Automating Web Applications |
The Jubii Indexing Robot |
Indexing, maintainance |
The NorthStar Robot |
Indexing |
The NWI Robot |
Indexing, Statistics |
The Peregrinator |
Indexing |
The Python Robot |
|
The TkWWW Robot |
Indexing |
The Web Moose |
Statistics, Maintenance |
TITAN |
Indexing |
TitIn |
Indexing, Statistics |
TLSpider |
Indexing |
UCSD Crawl |
Indexing, Statistics |
UdmSearch |
Indexing, validation |
UptimeBot |
Indexing, Statistics |
URL Check |
Maintenance |
URL Spider Pro |
Indexing |
Valkyrie |
Indexing |
Verticrawl |
Indexing, Seartching, Classyfing |
Victoria |
Maintenance |
vision-search |
Indexing |
void-bot |
Indexing, Maintenance |
Voyager |
Indexing, Maintenance |
VWbot |
Indexing |
w@pSpider by wap4.com |
Indexing, Maintenance (dedicated to wap/wml pages) |
W3M2 |
Indexing, Maintenance, Statistics |
w3mir |
Mirroring |
Walhello appie |
Indexing |
WallPaper / Crawlpaper |
Indexing |
Web Core / Roots |
Indexing, Maintenance |
Web Wombat |
Indexing, Statistics. |
WebBandit Web Spider |
Resource Gathering / Server Benchmarking |
WebCatcher |
Indexing |
WebCopy |
Mirroring |
webfetcher |
Mirroring |
Webfoot Robot |
|
Webinator |
Information Retrieval |
weblayers |
maintainance |
WebLinker |
Maintenance |
Weblog Monitor |
Statistics |
WebMirror |
Mirroring |
WebQuest |
Indexing |
WebReaper |
Indexing/ Offline Browsing |
webs |
Statistics |
Websnarf |
|
WebSpider |
Maintenance, Link Diagnostics |
WebStolperer |
Indexing |
WebVac |
Mirroring |
webwalk |
Indexing, Maintenance, Mirroring, Statistics |
WebWalker |
Maintenance |
WebWatch |
maintainance, Statistics |
Wget |
Mirroring, Maintenance |
whatUseek Winona |
Site-level Search and Meta-Search Engines. |
WhoWhere Robot |
Indexing |
Wild Ferret Web Hopper |
Indexing, Maintenance Statistics |
World Wide Web Wanderer |
Statistics |
World Wide Web Worm |
Indexing |
WWWC |
Maintenance |
XGET |
Mirroring |
XYLEME Robot |
Indexing |
Other Various robots
Googlebot
Inktomi Slurp
MSNBot
AskJeeves
Scooter (Alta Vista)
µTorrent
123spider
1noon
1X
21seek.com
2Bone
8484 Boston
Abacho
ABCdatos
ABCdatos BotLink
Aberja
abot
About robot
About.com robot
Accela RSS
Accoona
Accoona Search
Ace Explorer
Ackerm
Acme.Spider
Acoi
Acoon.de
Acorn
Activeworlds
Ad Muncher
Adaxas
Advanced Site Crawler
Aesop
Agada
Ahoy! The Homepage Finder
AideRss
Aladin
Aleksika
Alexa I
Alexa
Alkaline
Allesklar
Alligator
Altavista
Amansoft WebBug
Amazon.com
Amazon.com robot
Amfibi
Amidalla
ananzi
Annomille
Anonymizer/1.1
Anonymous
AnswerBus
AnswerChase
Answerchase PROve
ANT Fresco
Anthill
Antibot
Anzwers
AOL Search / Pacific Internet Exchange
A-Online
Apexoo
Aplix
Aport
Araby
Arachnophilia
Arale
Araneo
AraybOt
ArchitextSpider
Aretha
Arexera
ARIADNE
Arianna
Arikus inContext
arks
Asaha
ASAHINA
Ask 24x Info
Ask About Oil
askEd
AskJeeves
ASpider (Associative Spider)
ASPSeek
Associative Sort
Atlocal
ATN Worldwide
Atomz.com Search Robot
atSpider
Attentio
Augurnet
AURESYS
Auto Email
autohttp
AV Fetch
AVSearch-1.0
AWeb
Axada
Axmo
Ay-Up
Azureus
BaBoom
BackRub
BackStreet
Baidu
Balihoo
BanBots
Barca
Barrahome
Batsch
BB4
BBot
bCentral
BD-Brandprotect
Bdncentral
beammachine
Beauty
BebopBot
Big Brother I
Big Brother
BigClique
Biglotron
Bigsearch.ca
Bilbo
Bilgi
Bisnisseek (was www.bisnisseek.com) robot
Bitacle
Biyubi
Bjaaland
Black Widow
BlackWidow
Blaiz
Blitzsuche
BlockNote
Blogbot
BlogBridge
blogdex
Bloglines
Bloglines-Images
BlogsNow
blogWatcher
BlogzIce
Bloodhound
Bluefish
BMCLIENT
BMLAUNCHER
BoardReader
Bobby
Boitho
Bookdog
Borg-Bot
BoxSeaBot
Brandimensions
BravoBrian
Breaking Blogs
bright.net caching robot
Bruinbot
BSDSeek
BSpider
Btbot
BuildCMS
Bulkfeeds
BullsEye/Intelliseek
BunnySlippers
Burf
BurstFindCrawler
Buscaplus
Business.com
Business-Socket
BW-C-2.0
CACTVS Chemistry Spider
Caesius WebQL
Calif
CamelHttpStream
Cassandra
Catall
Cazoodle
Ccubee
CE-Preload
CFC crawler
cfetch
CFNetwork
cg-eye
Checkbot
Checklinks
CheckWeb
Chikayama-Taura Lab Shim
Chilkat
Chinatelecom
Christcrawler
CHttpClient
churl
cIeNcIaFiCcIoN.nEt
CipinetBot
CJ Spider
CJB Net
Claria I
Claria
Claymont I
Claymont
Clush
CMC/0.01
COAST Webmaster
Cobion
CoBITSProbe
Cocoa del.icio.us
CodeBitch
ColdFusion
collage
Collective
Combine
cometrics
Cometsystems
ComputingSite Robi/1.0
ComRite
ConfuzzledBot
ContactBot
ContentSmartz
contype
Converas
CoolBot
Coral
Cosmix
Cosmix project
Coteo
Cougarsearch
CrawlConvera
Crawlly
Creatice Commons
Croccrawler
csci_b659
Cuam
Cuasar
Curl
CurryGuide
Cusco
Custo
Cyberdog
CyberSpyder
DataCha0s
Datafire
DataparkSearch
DataSpear
Daum
DAUMOA
Daumsoft Talkro
Daypop
dbDig
Dbrowse
DC-Sakura
dCSbot
DDD
dds explorer
Dead-Links.com
deepak-USC/ISI
Deepindex robot
del.icio.us pictures
DeleGate
Denmex
Der Bot
Desert Realm Spider
DevComponents
DeWeb(c) Katalog/Index
DiaGem
Die Blinde Kuh
Die Kraehe
DienstSpider
Digger
Diggit!
Digimarc MarcSpider
Digital Integrity Robot
Dir.com / Iliad
Direct Hit Grabber
DISCo Pump
disco/Nutch
Ditto
DLC
dlman
DNAbot
DocZilla
dodgebot
Domanova
Donut P
Doubanbot
DoWeb UK
Download Demon
Download Direct
Download Druid
Download Express
Download Master
Download Ninja
Download Wonder
download.exe
Download-Tipp I
Download-Tipp
Dr.WEB
DragonBot
Dragonfly
DreamCast DreamPassport
Drecom
Drupal
DSurf15a 01
DSurf15a 71
DSurf15a 81
DSurf15a VA
dt Search Spider
DTAAgent
Dual Proxy
Dumbot
DWCP (Dridus' Web Cataloging Project)
Earthcom
EbiNess
EBrowse
eCatch
Echo.fr
e-collector
Egoto
EIT Link Verifier Robot
eJupiter
ELFINBOT
Elftales
Elinks
Eliyon
Emacs-w3 Search Engine
EmailSiphon
EmailWolf
EmeraldShield
Empas
endo
Enfish
Enterprise Search
EnterpriseSearch
Entireweb Speedy
Envolk
EroCrawler
esculapio
e-Sense
ESI Smart-Spider
e-Society
Esther
e-Style
ESurf15a 15
Eurip
Euroseek
Evaal
Eventax
Evliya Celebi
eXact Search
ExactSEEK
ExactSEEK (Jayde Online)
Exalead
Exalead image
Excalibur (Convera)
Excite
Execrawl
Exisoftware image grabber
Expert HTML
Expired Domain Sleuth
Express Web Pictures
Extractor Pro
Factbites
FaEdit
FairAd user
FARK
Fast/Alltheweb
Fastbot
FastCrawler
FavIconizer
favo.eu
FavOrg
Favorites Sweeper
FaXo Search
FDM
Feed Seeker
Feed::Find
Feed24
Feedable
FeedChecker
FeedForAll
FeedMap / BlogMap
FeedShow
FeedshowOnline
Feedster
FeedZcollector
Felix IDE
fetch libfetch
FetchLinks
FetchRover
fido
Filangy
File Boost
FileHeap
FileHound
Finacialbot
Find An ISP
FindPal
Findsame
Finesearch robot
Finjan
Fireball
Fireball
FirstGov.gov
Firstsfind
Fish search
Flaptor
FlatArts
Flatland
FlickBot
Fluid Dynamics Search Engine robot
FLY
fly*
FocusedSampler
Fooky
Fouineur
Franklin Locator
Freecrawl
FreeFind
Frequent Finders
Fresh Download
FreshNotes
FSurf15a 01
Full Web Bot 0416B
Full Web Bot 0516B
Full Web Bot 2816B
FunnelWeb
FuseBulb
FyberSearch
Gagglebot
Gaislab
Galaxy
Gallent
Gamekit
GameSpyHTTP
gammaSpider, FocusedCrawler
GammaWare GammaSpider
gazz
GCreep
Geckobot
generic_crawler
GenieKnows.com
GeoBot
Geona
GeoURL ICBM
GeoURLBot
Getbot
GetBot
GetRight
GetRight download manager
GetterroboPlus Puu
GetURL
GetWeb
GigaBaz
GigaBaz Brainbot
Gigablast
Gigablast SiteSearch
giveRAMP
GlobalSpec
GNODSPIDER
Go!Zilla
GoForIt
GoGuides
GoldenFeed
Goldfire
Golem
gonzo1
Goo
Goo Japan
Goo Japan / Inktomi
Goofer
Google AdSense
Google Feedfetcher
Google image crawler
Google Mobile
Google Search Appliance
Google WAP
Googlebot
GoonGee
GotIt
Grapeshot
Grapnel/0.01 Experiment
Greasemonkey
GreatNews
GreenBrowser
gridwell
Griffon
Grigor
Gromit
Gromit
Grub
Gseek.com
GSiteCrawler
Guestbook spamming
Gulper Bot
Gungho
guruji
Hämähäkki
HamBot
Happy Fun
Harvest
Harvest-NG
Haste
Hatena
havIndex
Helix
Heritrix I
Heritrix
hgrepurl
HI (HTML Index) Search
HiDownload
Hippias
Hitwise
HKU WWW Octopus
HLoader
HomePageSearch
Homerweb
Hometown Spider Pro
Honda-Search
HooWWer
Hoppa
HotJava
Hotzuno - Japanese BBS reader client
HPL/Nutch
ht://Dig
htdig
Html Link
HTML Parser
html_analyzer
HTML2JPG
HTMLgobble
HTTP Retriever
http://www.ianett.com robot
HTTP::Lite
HTTPResume
HttpUnit
Hybrid
Hyper Estraier
Hyper-Decontextualizer
I, Robot
i1search
iajaBot
Iask
iaskspider
IBM Lotus Discovery
IBM_Planetwide
IBM's Almaden
iCab
ICC-Crawler
IceRocket
IceRocket Web search robot
Iconsurf
ICRA
ICS Robot
iearthworm
IEFav172Free
iGetter
Il Trovatore
Ilial
Ilse
Ilse Netherlands
image.kapsi.net
Imagelock
ImageVisu
Incutio
IncyWincy I
IncyWincy II
IncyWincy III
Index the Web
Industry Program
iNet Grabber
Inet Library
InetURL
Infobee.ne.jp
infoConveraCrawler
InfoFly
InfoLink
INFOMINE
INFOMINE/iVia
InfoNaviRobot
Informant
Infoseek
InfoSeek Robot 1.0
Infoseek robot
Infoseek Sidewinder
InfoSpiders
Ingrid
Inktomi Slurp
Innerprise URL Spider Pro
InnerpriseBot
Innova/IBM Rational
Inria
Insignias Jeode
Insitor
Insitornaut
Inspector Web
IntelliAgent
Intelliseek's BlogPulse
Interarchy
Internet Cruiser Robot
Internet Link Agent
Internet Shinchakubin
Internetseer
intraVNews
IP*Works!
IP2Location
IP2Map
Ipselon
Iria
IRL-crawler
Iron33
Irssi
Irvine
iSiloX
Israeli-search
Istarthere
isurf
iVia
IWAgent
Jakarta Commons
JamBot
Janas I
Janas
JavaBee
Jayde
JBot Java Web Robot
jBrowser
JCheckLinks
JCrawler
JetCar / Flashget
JetEye
Jigsaw
JoBo Java Web Robot
Jobot
JoeBot
Jordo Media RSS
Journster
J-PHONE
JRTwine
JumpStation
Jyxo
KakleBot
Kapere
Katipo
Kazehakase
KDD-Explorer
Kenjin
Keywen
KFSW-Bot
Kilroy
Kinja
Kinja Imagebot
KIT-Fireball
Klondike
K-meleon
KnowItAll
Knowledge.com
KnowledgeStorm
KO_Yappo_Robot
Kolibri.de
Kontiki
Krugle
Kuloko
kulturarw3
KummHttp/1.1
LabelGrabber
Lachesis
lanshanbot
Lapozz
larbin
Laurions Ipium
LawInfo
lcabotAccept
LECodeChecker
Leechget
legs
Lets crawl!
Lexibot I
Lexibot
Lexis-Nexis
libcurl
LibertyW
LibMaster
libWeb/clsHTTP
Liferea
Lightning Download
Lincoln State Web Browser
Link Valet
Link Validator
link_check3
Linkalarm
Linkbot Pro
Linkcheck
LinkcheckerBeepware
Linklint
Linknz
Linkomatic
Link-Pimp
Links
Links4us
LinkScan
Linksmanager bookmark checking
LinkSonar
LinkSweeper
LinkWalker
ListBidBot
LiveJournal picture
llaut
LMQueueBot/0.2
lmspider
Local.com
LocalBot
Lockon
Lockstep
logo.gif Crawler
Look.com
Lookseek
Looksmart
Looksmart page analysis
Loop Improvements
Luchs
Lunascape
Lycos
Mac Finder
Mac WWWWorm
MagicWML/1.0
Mag-Net
Magpie
Magpie RSS
Mail.ru
Majestic-12 Dsearch
mammoth
MapoftheInternet
Marketscore
Marvin
marvin/infoseek
Maskbit Surfmaster
Masterseek
Mattie
Maxbot
Maxomo
McBot/5.001
Media Find
MediaFox
Medkatalog
MegaSheep
Mercator
Mercator-Scrub
MerzScope
Metacarta
Metacarta / Cogent
MetaEuro
MetaGer
MetaGer search robot (Germany)
MetaGer-LinkChecker
Metamark URL Shorten Service
MetaQuerier
Metaspinner
Metatagsdir
Microsoft MSRBot
Microsoft.com user agent
MicrosoftPrototypeCrawler
Microton Intelix
MindCrawler
Mindjet MindManager
mini- Rank
minibot
MiracleAlphaTest
Mirago I
Mirago II
Mirago (UK)
Mirago France
Missauga Locate
Missouri College Browse
Mister PiX
Misterbot
Miva / ex Findwhat.com
MixCat robot
MixCat
Mizzu Labs 2.2
mnoGoSearch I
mnoGoSearch
Mo College 1.9
moget
moiNag
MojeekBot
MOMspider
Monogol
Monster
MoonBrowser
Moreover / FeedDirect
Morfeo / Centrum
Motor
Motoricerca
Mozdex
Mozi!
MPRM
MSN media
MSN Search
MSNBot
Multiblocker
MultiCrawler
MultiText
Muncher
Muninn
Muscat Ferret
MVAClient
Mwd.Search
My UrlTrends
myDaemon
Mylinea
MySpiders
Nameprotect I
Nameprotect II
Nameprotect copyright
naoFavicon4IE
NASA Search
Nationaldirectory
Naver I
Naver
Naver Japan / Korea
Naver Japan / Korea robot I
Naver Japan / Korea robot
Navisso
NDSpider
Nebulla.info
NEC Researchindex
NEC-Hayek
NEC-MeshExplorer
Nederland.zoek
neofonie
Neomo
NetAnts
NetCarta WebMap Engine
NetComplete
Netinfo.bg
Netlookout
Netluchs
NetMechanic
NetNoseCrawler
Netprospector
Netpumper
NetScoop
NetShow Media Player
NetWhat
Netzip/Smartdownload
Netzippy robot
NeuralBot
newscan-online
newsearchengine
NewsGator Net
NewsGator
NextLinks
Nextopia
NG-Search
NHSE Web Forager
nicebot
NicoZone
Nikita the Spider
NITLE Blog Spider
Nitro
Noago
Nocilla
Nokodo
Nomad
Northern Light Gulliver
Northernligh
noXtrum
Noyona
Nsauditor
NTL user agent
NTT Directory
NTT DoCoMo
Nucleus CMS SiteList
nuSearch s
Nutch
NutchCrawler
nzexplorer
obidos-bot
ObjectsSearch
Occam
Octopus
Octora RSS feed
ODIN Directory
Offlinebrowser
Omea RSS
OmniExplorer
Online24
Only.com
onsearch
OntoSpider
open i
Openfind
Openfind Data Gatherer
OpenPortal4U
Opentagger
OpenText
OpenWebSpider
Opidoo
Oracle interMedia Text
Oracle Search
Orangeslicer
Orb Search
Orbiter - DailyOrbit
Orca
OutfoxBot
OutfoxMelonBot
Overture/Fast/Alltheweb
OWR_Crawler
Ozelot
Pack Rat
Pagebites
PageBoy
Pagebull
PagmIEDownload
Parallel URL
ParaSite
Patric
Patsearch
Pavuk
Pbrowse
PC Magazin SiteSnagger
Pear
Peerbot
Pegasus
PeopleChat
PerlCrawler
Peval
Phantom
PhpDig
Pico
Picsearch robot
Piffany
Pilot Hitlist I
Pilot Hitlist
PiltdownMan
Pimptrain.com's robot
PingALink
Pingdom
Pinpoint
Pioneer
pipeline
Pita
Pizilla
Plagger
PlagiarBot
Plantynet
plinki
Pluck RSS
Pluggd Podcast Crawler
Pockey
POE HTTP user-agent
Poirot
Polybot
Popdex
PopJapanSearch
Poppi
Popular Iconoclast
Port Huron
Portal Juice Spider
PortalB Spider
Portaljuice
portalmmm Imode
Post.sk / Eurotel.sk robot
Potbot
PrivacyFinder
Privoxy
Privoxy
Production Bot
ProgramacionWeb.net
ProWebguide
psbot
PSurf15a 11
PSurf15a 51
psycheclone
PubCrawl
PureSight
Puxa Rapido
PycURL
Qango
Qualigo
Quarterdecks WebCompass
Quepasa!com
QueryN Metasearch
Quicktime
Qweery
Qweerybot
rabaz
Rainbot
Rambler
ramBot
Rank Exec
RankMeter
Raven Search
RBSE Spider
Read A Blog
Real Media
RealDownload
Reaper
REBOL
Rebusnet
RedCarpet
rediff.com
RedKernel Softwares robot
REL Link Checker
Relsoft
RepoMonkey
Resume Robot
Rewebber
rico
RixBot
RixBot Rebol
Road Runner: The ImageScape Robot
RoadHouse Crawling System
ROADS
Robbie the Robot
RoboCrawl
RoboCrawl Spider
RoboFox
Robot Francoroute
Robozilla
Robozilla
Roverbot
RPT-HTTPClient
RSS Bandit
RSS Micro
RSSOwl
RSurf15a
rubhub
RufusBot
RuLeS
Rumours-Agent
RX Bar
SafariBookmarkChecker
Safe-t.net
SafetyNet Robot
Sagool
Sandcrawler
ScanWeb
SCAVR
ScholarUniverse
Schwarzmann
Scooter (Alta Vista)
Scope
ScoutAbout
ScoutMaster
Scrub the web
Search & Links
Search.Aus-AU.COM
Search.ch
SearchByUSA
Searchday
Searchengine
Searchexpress
Searchguild
Search-Info ODP/DMOZ spider
Searchit
Searchmee!
SearchProcess
SearchSight
Searchspider
SearchSpider
Secure Computing SmartFilter
Seed Search robot
Seekbot
semaforo
Semager
Semantic Discovery
Senrigan
Sensis Australia
SEO-Tools Stalker
Seznam I
Seznam
SG-Scout
ShagSeeker
Shai'Hulud
Shareaza P2P
SharewarePlaza I
SharewarePlaza
ShopWiki
Shoula
Sicher-durchs-Netz
Sidewinder G2
Siets
Sift
Siigle
Simmany Robot Ver1.0
SimpleFavPanel
Simpy
Simpy Bookmarklet
Singingfish
Sirketçe
Site Searcher
Site Spider
Site Valet
SiteBar
SiteRecon
SiteSell
SiteSucker
SiteTagger
SiteTech-Rover
SiteTruth
SitiDi.net
Sixxs
Skaffe
Skymob.com
SLCrawler
Sleek
Sleipnir
Slider Search I
Slider Search
Slim Browser
Slysearch
Smart Spider
SmiffyDCMetaSpider
Snapbot
Snarfer RSS
Snooper
SnoopRob
Snoopy
Snoopy PHP-client
Snyke
social search application
SocSciBot ()
Soft Hypermarket
Softizer
Sohu
Solbot
Some spam bot
SoMusical!
Sootle
Space Bison
Speed Download
Speedfind
Speedy Spider
Sphider
Spida
Spider
spider_monkey
SpiderBot
SpiderKU
Spiderline Crawler
SpiderMan
SpiderMonkey
SpiderView(tm)
Spock
Sportsuchmaschine
Sproose
Spry Wizard Robot
SQ Webscanner
Squidclam
Squid-Prefetch
Sqworm
SSurf15a
Stamina
Star Downloader
stat statcrawler
Steeler
Strategic Board
StreamBox VCR
Suchbaer.de (Germany) search robot
Suchbot
Suchknecht Austria
Suchmaschine21
suchpad
Suke
Sunrise
Suntek search engine
SuperBot
Supersnooper
SurfControl
Surfsafely
SurveyBot
Sven
SWB
Swooglebot
SWSBot - SmartWareSoft
Sygol
Sygol Search
Sylera
Sync2It
SyncBot
SyncIT
Syndirella
Synomia
Syntryx Solution Suite
Szukacz
Szukaj
TACH Black Widow
TafWeb
tags2dir
TAGword DMOZ
Tagyu - del.icio.us
TAMU_CS
Tarantula
Target Your News
Targetblaster
Tarspider
TCDBOT
Tcl W3 Robot
TechBOT
Tecomac
Tecomi
Teleport (website) downloading tool
Templeton
Teoma
Teradex
Teragram
Terraminds
TerraNautic
Terrawiz
The Jubii Indexing Robot
The NorthStar Robot
The NWI Robot
The Peregrinator
The Python Robot
The TkWWW Robot
The Web Moose
TheSuBot
ThumbShots
Time Warner Telecom
TimelyWeb
TITAN
TitIn
Tivra
Tjgroup
TJvHttpGrabber
Tkensaku
TLSpider
toCrawl/UrlDispatcher
T-Online
Topodia
Topos
Traazi!
Trailfire
Trampelpfad Webkatalog
TrendTech
Tricos
TSurf15a
Tulip Chain
Turnitin / SlySearch
TurnPike Emporium
TutorGig
Twiceler
Twisted Python
Twotrees I
Twotrees
Tygo
TygoProwler
Ubbi Superdownloads
UCMore
UCSD Crawl
UdmSearch / MySearch / mnoGoSeach
UdmSearch*
UKWizz
UKWizz II
Ultimate Search / Smartdesk
UnCHAOS
Under the Rainbow
unknown
UofTDB
Updated!
UptimeBot*
URI::Fetch
URL Check
URL Search+
URL Spider Pro
URLBase
URLBlaze
URLGetFile
USyd-NLP-Spider
Vakes
Valkyrie
VayalaCreep
VDOG
Verity K2 Spider
Verity vspider
Versus Project
Verticrawl
VeryGoodSearch.com
Verzamelgids NL
Vespa Crawler
Vestris
Vias
Victoria
Virgilio
Visbot
Visicom
Vision research
vision-search
Visvo
Vivisimo
VLC
VM - Vertical Search Engine
void-bot
Voila.fr
Vortex
Voyager
Vulcan
VWbot
VWBot - MetaQuerier
w@pSpider by wap4.com
W3 Site Search
W3C Line Mode
W3C Link Checker
w3m
W3M2
w3mir
Wadain
Walhello
Walhello appie
WallPaper / Crawlpaper
Wanadoo Rechereche
Wanna-Be
WapOnWindows
Watchfire WebXM
WAVcheck
Wavefire
Waypath
Waypath blog
WDG validator
Web Bandit
Web Core / Roots
Web Thief Site Grabber
Web Trends
Web Vulnerability
Web Watcher Monitor
Web Wombat
Web2Express / Web2x
WebAlta
Webaroo
WebarooBot / RufusBot
WebBandit Web Spider
Web-bekannt
Web-Bot
WebCatcher
WebClipping.com
WebCollage Syndicator
WebCon
WebCopier
WebCopy
WebCorp
Webcrawl
WebDownloader
Webdup
webfetcher
WebFilter
Webfind
Webfoot Robot
Webglimpse
webGobbler
webhack
WebImages
Webinator
Webkhoj
weblayers
WebLight
WebLinker
WebLink's
Weblog Attitude Diffusion
Weblog Monitor
webmeasurement-bot
WebMiner
WebMirror
WeBoX
WebPix
WebQuest
WebRACE
WebRankSpider
Webreaper
Web-Robot
webs
WebSavvy
Websearch Australia
WebSearchBench
WebSearchBench
Website Explorer
Website eXtractor
Website Worth
Websnarf
WebSpider
Webspinne
Websquash
WebStat
Webster
Websters Webmaster Archive
WebStolperer
WebStripper
WebTrafficExpress
WebVac
WebVal
Webverzeichnis
webwalk
WebWalker
Webwasher
WebWasher ads
WebWatch
WebWobot UK
WebZip
Wells Search II
WEP Search
WFARC
Wget
Whatsup
WhatUSeek / Chubba
whatUseek Winona
WhizBang! Labs
Whois Source Crawler
Wholeweb
WhoWhere Robot
Wild Ferret Web Hopper #1, #2, #3
Wildsoft Surfer
WinampMPEG
WincerSong
Windows Live Products
WingFlyer WebFetch
WinInet
Wink
WinPodder
WinWap
WIRE
Wirtualna Polska / Netsprint
WiseGuys
Wisenut I
Wisenut
Wisewire
WISH
Wordchamp
WORIO
World Wide Weasel
World Wide Web Wanderer
World Wide Web Worm
WorldLight
WorQmada
Wotbox
WUME
Wusage
wwIPStuff
WWLib
WWSBOT
WWW.fi Media
www4mail
WWWC Updating check of Web pages. (Japanese only)
WWWC Ver 0.2.5
WWW-Mechanize
WWWoffle
wwwster
Xaldon WebSpider
Xdefine
Xenu
XerKa
XGET
xine
XIRQ
XMLSlurp
XP5 robot
xpc-mii.net
Xtreem Search Engine Studio
Xtreeme SiteXpert
Xyleme
Y!TunnelPro
Yacy
Yahoo / Overture
Yahoo blogs
Yahoo favorites
Yahoo Mindset
Yahoo multimedia
Yahoo picture
Yahoo Pipes
Yahoo Product
Yahoo Publisher
Yahoo robot (64.157.137.xxx)
YahooJapan
Yammba
Yanasoft WebAuto
Yandex
Yarienavoir
Yellopet
Yelo.no
yggdrasil
Ykoon RssReader
Yodao
Yokogao
Yoogli
Yoono
YottaCars
YottaShopping
Yuntis / GulperBot
Z-Add
Zao
Ze.bz
Zearchit
ZedZo
Zend
Zerx
Zeus
Zeus I
Zeus*
Zeusbot / Ulsysseek
Ziggy
Zipp I
Zipp
Zippyfinder
Zoo Tycoon
Zoom
zspider
Pasiphae
Alexibot
Aqua_Products
BackDoorBot
BackDoorBot/1.0
Black.Hole
BlackWidow
BlowFish
BlowFish/1.0
Bookmark search tool
Bot mailto:craftbot@yahoo.com
BotALot
BotRightHere
BuiltBotTough
Bullseye
Bullseye/1.0
BunnySlippers
Cegbfeieh
CheeseBot
CherryPicker
CherryPickerElite/1.0
CherryPickerSE/1.0
ChinaClaw
Copernic
CopyRightCheck
Crescent
Crescent Internet ToolPak HTTP OLE Control v.1.0
Custo
DISCo
DISCo Pump 3.0
DISCo Pump 3.2
DISCoFinder
DittoSpyder
Download Demon
Download Demon/3.2.0.8
Download Demon/3.5.0.11
EirGrabber
EmailCollector
EmailSiphon
EmailWolf
EroCrawler
Express WebPictures
Express WebPictures (www.express-soft.com)
ExtractorPro
EyeNetIE
FairAd Client
Flaming AttackBot
FlashGet
FlashGet WebWasher 3.2
Foobot
FrontPage
FrontPage [NC,OR]
Gaisbot
GetRight
GetRight/2.11
GetRight/3.1
GetRight/3.2
GetRight/3.3
GetRight/3.3.3
GetRight/3.3.4
GetRight/4.0.0
GetRight/4.1.0
GetRight/4.1.1
GetRight/4.1.2
GetRight/4.2
GetRight/4.2b (Portuguxeas)
GetRight/4.2c
GetRight/4.3
GetRight/4.5
GetRight/4.5a
GetRight/4.5b
GetRight/4.5b1
GetRight/4.5b2
GetRight/4.5b3
GetRight/4.5b6
GetRight/4.5b7
GetRight/4.5c
GetRight/4.5d
GetRight/4.5e
GetRight/5.0beta1
GetRight/5.0beta2
GetWeb!
Go!Zilla
Go!Zilla (www.gozilla.com)
Go!Zilla 3.3 (www.gozilla.com)
Go!Zilla 3.5 (www.gozilla.com)
Go-Ahead-Got-It
Googlebot-Image
GrabNet
Grafula
HMView
HTTrack
HTTrack 3.0
HTTrack [NC,OR]
Harvest
Harvest/1.5
Image Stripper
Image Sucker
Indy Library
Indy Library [NC,OR]
InfoNaviRobot
InterGET
Internet Ninja
Internet Ninja 4.0
Internet Ninja 5.0
Internet Ninja 6.0
Iron33/1.0.2
JOC Web Spider
JennyBot
JetCar
Kenjin Spider
Kenjin.Spider
Keyword Density/0.9
Keyword.Density
LNSpiderguy
LeechFTP
LexiBot
LinkScan/8.1a Unix
LinkScan/8.1a.Unix
LinkWalker
LinkextractorPro
MIDown tool
MIIxpc
MIIxpc/4.2
MSIECrawler
Mass Downloader
Mass Downloader/2.2
Mata Hari
Mata.Hari
Microsoft URL Control
Microsoft URL Control - 5.01.4511
Microsoft URL Control - 6.00.8169
Microsoft.URL
Mister PiX
Mister PiX version.dll
Mister Pix II 2.01
Mister Pix II 2.02a
Mister.PiX
NICErsPRO
NPBot
NPbot
Navroad
NearSite
Net Vampire
Net Vampire/3.0
NetAnts
NetAnts/1.10
NetAnts/1.23
NetAnts/1.24
NetAnts/1.25
NetMechanic
NetSpider
NetZIP
NetZip Downloader 1.0 Win32(Nov 12 1998)
NetZip-Downloader/1.0.62 (Win32; Dec 7 1998)
NetZippy+(http://www.innerprise.net/usp-spider.asp)
Octopus
Offline Explorer
Offline Explorer/1.2
Offline Explorer/1.4
Offline Explorer/1.6
Offline Explorer/1.7
Offline Explorer/1.9
Offline Explorer/2.0
Offline Explorer/2.1
Offline Explorer/2.3
Offline Explorer/2.4
Offline Explorer/2.5
Offline Navigator
Offline.Explorer
Openbot
Openfind
Openfind data gatherer
Oracle Ultra Search
PageGrabber
Papa Foto
PerMan
ProPowerBot/2.14
ProWebWalker
Python-urllib
QueryN Metasearch
QueryN.Metasearch
RMA
Radiation Retriever 1.1
ReGet
RealDownload
RealDownload/4.0.0.40
RealDownload/4.0.0.41
RealDownload/4.0.0.42
RepoMonkey
RepoMonkey Bait & Tackle/v1.01
SiteSnagger
SlySearch
SmartDownload
SmartDownload/1.2.76 (Win32; Apr 1 1999)
SmartDownload/1.2.77 (Win32; Aug 17 1999)
SmartDownload/1.2.77 (Win32; Feb 1 2000)
SmartDownload/1.2.77 (Win32; Jun 19 2001)
SpankBot
Sqworm/2.9.85-BETA (beta_release; 20011115-775; i686-pc-linux
SuperBot
SuperBot/3.0 (Win32)
SuperBot/3.1 (Win32)
SuperHTTP
SuperHTTP/1.0
Surfbot
Szukacz/1.4
Teleport
Teleport Pro
Teleport Pro/1.29
Teleport Pro/1.29.1590
Teleport Pro/1.29.1634
Teleport Pro/1.29.1718
Teleport Pro/1.29.1820
Teleport Pro/1.29.1847
TeleportPro
Telesoft
The Intraformant
The.Intraformant
TheNomad
TightTwatBot
Titan
True_Robot
True_Robot/1.0
TurnitinBot
TurnitinBot/1.5
URL Control
URL_Spider_Pro
URLy Warning
URLy.Warning
VCI
VCI WebViewer VCI WebViewer Win32
VoidEYE
WWW-Collector-E
WWWOFFLE
Web Image Collector
Web Sucker
Web.Image.Collector
WebAuto
WebAuto/3.40 (Win98; I)
WebBandit
WebBandit/3.50
WebCapture 2.0
WebCopier
WebCopier v.2.2
WebCopier v2.5
WebCopier v2.6
WebCopier v2.7a
WebCopier v2.8
WebCopier v3.0
WebCopier v3.0.1
WebCopier v3.2
WebCopier v3.2a
WebEMailExtrac.*
WebEnhancer
WebFetch
WebGo IS
WebLeacher
WebReaper
WebReaper [info@webreaper.net]
WebReaper [webreaper@otway.com]
WebReaper v9.1 - www.otway.com/webreaper
WebReaper v9.7 - www.webreaper.net
WebReaper v9.8 - www.webreaper.net
WebReaper vWebReaper v7.3 - www,otway.com/webreaper
WebSauger
WebSauger 1.20b
WebSauger 1.20j
WebSauger 1.20k
WebStripper
WebStripper/2.03
WebStripper/2.10
WebStripper/2.12
WebStripper/2.13
WebStripper/2.15
WebStripper/2.16
WebStripper/2.19
WebWhacker
WebZIP
WebZIP/2.75 (http://www.spidersoft.com)
WebZIP/3.65 (http://www.spidersoft.com)
WebZIP/3.80 (http://www.spidersoft.com)
WebZIP/4.0 (http://www.spidersoft.com)
WebZIP/4.1 (http://www.spidersoft.com)
WebZIP/4.21
WebZIP/4.21 (http://www.spidersoft.com)
WebZIP/5.0
WebZIP/5.0 (http://www.spidersoft.com)
WebZIP/5.0 PR1 (http://www.spidersoft.com)
WebZip
WebZip/4.0
WebmasterWorldForumBot
Website Quester
Website Quester - www.asona.org
Website Quester - www.esalesbiz.com/extra/
Website eXtractor
Website eXtractor (http://www.asona.org)
Website.Quester
Webster Pro
Webster.Pro
Wget
Wget/1.5.2
Wget/1.5.3
Wget/1.6
Wget/1.7
Wget/1.8
Wget/1.8.1
Wget/1.8.1+cvs
Wget/1.8.2
Wget/1.9-beta
Widow
Xaldon WebSpider
Xaldon WebSpider 2.5.b3
Xenu's
Xenu's Link Sleuth 1.1c
Zeus
Zeus 11389 Webster Pro V2.9 Win32
Zeus 11652 Webster Pro V2.9 Win32
Zeus 18018 Webster Pro V2.9 Win32
Zeus 26378 Webster Pro V2.9 Win32
Zeus 30747 Webster Pro V2.9 Win32
Zeus 32297 Webster Pro V2.9 Win32
Zeus 39206 Webster Pro V2.9 Win32
Zeus 41641 Webster Pro V2.9 Win32
Zeus 44238 Webster Pro V2.9 Win32
Zeus 51070 Webster Pro V2.9 Win32
Zeus 51674 Webster Pro V2.9 Win32
Zeus 51837 Webster Pro V2.9 Win32
Zeus 63567 Webster Pro V2.9 Win32
Zeus 6694 Webster Pro V2.9 Win32
Zeus 71129 Webster Pro V2.9 Win32
Zeus 82016 Webster Pro V2.9 Win32
Zeus 82900 Webster Pro V2.9 Win32
Zeus 84842 Webster Pro V2.9 Win32
Zeus 90872 Webster Pro V2.9 Win32
Zeus 94934 Webster Pro V2.9 Win32
Zeus 95245 Webster Pro V2.9 Win32
Zeus 95351 Webster Pro V2.9 Win32
Zeus 97371 Webster Pro V2.9 Win32
Zeus Link Scout
asterias
b2w/0.1
cosmos
eCatch
eCatch/3.0
hloader
httplib
humanlinks
ia_archiver
larbin
larbin (samualt9@bigfoot.com)
larbin samualt9@bigfoot.com
larbin_2.6.2 (kabura@sushi.com)
larbin_2.6.2 (larbin2.6.2@unspecified.mail)
larbin_2.6.2 (listonATccDOTgatechDOTedu)
larbin_2.6.2 (vitalbox1@hotmail.com)
larbin_2.6.2 kabura@sushi.com
larbin_2.6.2 larbin2.6.2@unspecified.mail
larbin_2.6.2 larbin@correa.org
larbin_2.6.2 listonATccDOTgatechDOTedu
larbin_2.6.2 vitalbox1@hotmail.com
libWeb/clsHTTP
lwp-trivial
lwp-trivial/1.34
moget
moget/2.1
pavuk
pcBrowser
psbot
searchpreview
spanner
suzuran
tAkeOut
toCrawl/UrlDispatcher
turingos
webfetch/2.1.0
Is your robot not in this list? Do you know of a robot not included here? Simply email us at bots@basisoft.com so we can include it immediately.
del.icio.us
Digg
Furl
Reddit