The Robots Exclusion Protocol

The Robots Exculsion Protocol is a mechanism for telling robots and crawlers which files and directories they can access as they flow through your website.

When a well behaved robot starts crawling a website it checks for the robots.txt file, analyses its contents and cralws only through the directories and files which are allowed by the contents of the robots.txt file.

The syntax of the entries in the robots.txt file is of the type:

User-agent: <Name of user agent>
Dissallow: <Name of directory>

Wild cards are not allowed and each of the directories or files that you need to exclude for a specific robot should be denoted explicitly.

Advanced Robots.txt Generator allows you to select robots and directories in a WYSIWYG manner and has the most updated list of robots available. It will take seconds to create and upload your robots.txt file and maintaining your robots.txt files can be done at a glance.

Check the full features and download Advanced Robots.txt Generator!

Advanced Robots.txt Generator Standard offers complete control of more than 400 robots and crawlers within minutes.

Advanced Robots.txt Generator Professional is an advanced tool supporting close to 2000 robots and crawlers and allows for use in dynamic (PHP, ASP etc.) websites.

The Professional edition is available for $35, offers weekly updates of the most current robots and crawlers and naturally comes with free documentation and customer support. We also honor a 30 day no-questions asked refund policy.

See complete features list

Creating robots.txt files with Advanced Robots.txt Generator couldn't be simpler. It is just a matter of three steps:

1. Select robots/crawlers from the list of 300 robots.
2. Select the files / directories of your website.
3. Compile robots.txt - and upload automatically to your webserver.

Download Advanced Robots.txt Generator and try it now!