Home › Forums › Other › What is robots.txt? › Reply To: What is robots.txt?
March 20, 2014 at 12:28 pm #166405
Robots.txt is essentially a configuration file where you tell web crawlers what to do.
…and then, the robots decide whether or not to listen to you. Legitimate bots (google, etc.) will honor your wishes, but keep in mind that nothing forces them to. Malicious bots may not even bother reading the file.