Reply To: What is robots.txt?

[__]

Robots.txt is essentially a configuration file where you tell web crawlers what to do.

…and then, the robots decide whether or not to listen to you. Legitimate bots (google, etc.) will honor your wishes, but keep in mind that nothing forces them to. Malicious bots may not even bother reading the file.