The forums ran from 2008-2020 and are now closed and viewable here as an archive.

Home Forums Other What is robots.txt? Reply To: What is robots.txt?


Robots.txt is essentially a configuration file where you tell web crawlers what to do.

…and then, the robots decide whether or not to listen to you. Legitimate bots (google, etc.) will honor your wishes, but keep in mind that nothing forces them to. Malicious bots may not even bother reading the file.