|
The Robots META tag |
Top Previous Next |
|
Some web servers will not allow you to upload a robots.txt file so you will need to use an alternative mechanism to disallow crawlers from crawling specific pages. This can be done through the use of the Robots META tag which you can include in the <HEAD> section of your HTML documents that should not be spidered:
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
However, not all crawlers support the Robots META tag.
Advanced Robots.txt Generator Website. |