The forums ran from 2008-2020 and are now closed and viewable here as an archive.

Home Forums Other What is robots.txt? Reply To: What is robots.txt?


…and then, the robots decide whether or not to listen to you.

Yeah… this is true. I mean, for Ess Eee Oh purposes, it generally works. Google is good about it. That being said, there’s no international bot governing body that makes them properly access your site. They’re definitely by no means a security measure.