Robots.txt and Sitemaps

Last week at the NYC SES Conference it was announced that all 4 major search engines (Google, Yahoo, MSN and Ask) now not only support the sitemaps.org sitemap protocol, but also support the new auto-discovery feature for the robots.txt file!

In order to use the auto-discovery feature, you simply add the following line to your robots.txt file:

Sitemap: http://sitemap_location

Example Robots.txt file:

User-agent: *
Disallow:
Sitemap: http://www.mcanerin.com/sitemap.gz

I have updated my free Robots.txt Generator so that it will generate this properly for you.

Ian