Bots, Spiders, and Crawlers - The Results – Manu

A good test about trying to understand how much the crawlers follows the Robots.txt

Fundor 333