Bookmark " Bots, Spiders, and Crawlers - The Results – Manu"
A good test about trying to understand how much the crawlers follows the Robots.txt
The floor underneath you suddenly gives way! You find yourself rapidly sliding down into an Internet hole. Memes and low-resolution cat pictures go flying by. Flailing, you reach out for something that seems to resemble good advice and catch hold of it, halting your fall and climbing to your feet. You emerge in micro-blogging territory.