The forums ran from 2008-2020 and are now closed and viewable here as an archive.

Home Forums Other What is robots.txt? Reply To: What is robots.txt?


Robots.txt is essentially a configuration file where you tell web crawlers what to do. You can tell them not to look at certain pages for SEO reasons… or you can draw their attention to deeply nested pages, though they’ll probably find them anyways.

Robots.txt is ESPECIALLY important if you keep a blog as there is likely to be MULTIPLE copies of blog posts.