Bookmark of " Bots, Spiders, and Crawlers - The Results – Manu"Fundor 333A good test about trying to understand how much the crawlers follows the Robots.txt
