Robots, crawlers, spiders.

Robots crawlers and spiders are computer software which go through the web retrieving all possible content automatically,' travelling' through the hyper links from one page to the next.

There are various kinds of robots which crawl web and are usually used for:

• Indexing
• Mirroring
• Monitoring
• Validation.

Robots.txt files are a way to stop robots crawling through certain parts of the website.

Why use a Robots.txt file?

The use of a robots.txt file on your website allows you to control moderately well the behavior of spiders and crawlers on your pages. As a consequence with the use of the robots.txt you can

• Manage your server resources.
• Manage your bandwidth
• Control page access for SEO purposes
• Limit page access for Copyright reasons.

Advanced Robots.txt Generator allows you to create your Robots.txt files within seconds, maintain them for changes in your needs and site parameters and compare with your previously saved projects.

Download Advanced Robots.txt Generator and experience the full set of features.

Advanced Robots.txt Generator Standard offers complete control of more than 400 robots and crawlers within minutes.

Advanced Robots.txt Generator Professional is an advanced tool supporting close to 2000 robots and crawlers and allows for use in dynamic (PHP, ASP etc.) websites.

The Professional edition is available for $35, offers weekly updates of the most current robots and crawlers and naturally comes with free documentation and customer support. We also honor a 30 day no-questions asked refund policy.

See complete features list

Creating robots.txt files with Advanced Robots.txt Generator couldn't be simpler. It is just a matter of three steps:

1. Select robots/crawlers from the list of 300 robots.
2. Select the files / directories of your website.
3. Compile robots.txt - and upload automatically to your webserver.

Download Advanced Robots.txt Generator and try it now!