Cuil Added to Robots.txt Generator

As many of you know, I run one of the most popular robots.txt generators on the web. Some time ago, I sent an email to up and coming search engine "Cuil" for information so I could add their crawler to the tool.

Once they came out of stealth mode, I was sent an email with the information (which I've been procrastinating on) and just today discovered that Victor mentioned my tool in his blog.

Well, I could not procrastinate any more after that! So now, the robots.txt generator now fully supports Cuil and it's special features (ie crawl-delay).

If you haven't tried Cuil, you should. It's growing very rapidly and currently has the largest index of sites of all the search engines. It's pretty good at finding hard to locate information.

Ian