Robots.txt and Sitemaps

Last week at the NYC SES Conference it was announced that all 4 major search engines (Google, Yahoo, MSN and Ask) now not only support the sitemap protocol, but also support the new auto-discovery feature for the robots.txt file!

In order to use the auto-discovery feature, you simply add the following line to your robots.txt file:

Sitemap: http://sitemap_location
Example Robots.txt file:

User-agent: *

I have updated my free Robots.txt Generator so that it will generate this properly for you.


No comments: