Cuil Added to Robots.txt Generator
As many of you know, I run one of the most popular robots.txt generators on the web. Some time ago, I sent an email to up and coming search engine "Cuil" for information so I could add their crawler to the tool.
Once they came out of stealth mode, I was sent an email with the information (which I've been procrastinating on) and just today discovered that Victor mentioned my tool in his blog.
Well, I could not procrastinate any more after that! So now, the robots.txt generator now fully supports Cuil and it's special features (ie crawl-delay).
If you haven't tried Cuil, you should. It's growing very rapidly and currently has the largest index of sites of all the search engines. It's pretty good at finding hard to locate information.
Ian
Posts
1 comment:
I know this is a little late in the day, but Cuil just ignores robots.txt which has earned it my ire after it laboriously crawled huge numbers of threads in my site's forum and gobbled up > 1GB in doing so. I've blocked it through htaccess.
Post a Comment