Home › Forums › Other › What is robots.txt? › Reply To: What is robots.txt?
March 20, 2014 at 3:48 am #166318
Robots.txt is essentially a configuration file where you tell web crawlers what to do. You can tell them not to look at certain pages for SEO reasons… or you can draw their attention to deeply nested pages, though they’ll probably find them anyways.
Robots.txt is ESPECIALLY important if you keep a blog as there is likely to be MULTIPLE copies of blog posts.