What is robots.txt?

[ranjeet]

Hi friends, can anyone help me with robots.txt file, what role it plays in a website?

[nixnerd]

Robots.txt is essentially a configuration file where you tell web crawlers what to do. You can tell them not to look at certain pages for SEO reasons… or you can draw their attention to deeply nested pages, though they’ll probably find them anyways.

Robots.txt is ESPECIALLY important if you keep a blog as there is likely to be MULTIPLE copies of blog posts.

[__]

Robots.txt is essentially a configuration file where you tell web crawlers what to do.

…and then, the robots decide whether or not to listen to you. Legitimate bots (google, etc.) will honor your wishes, but keep in mind that nothing forces them to. Malicious bots may not even bother reading the file.

[nixnerd]

…and then, the robots decide whether or not to listen to you.

Yeah… this is true. I mean, for Ess Eee Oh purposes, it generally works. Google is good about it. That being said, there’s no international bot governing body that makes them properly access your site. They’re definitely by no means a security measure.

[kevsyers]

They are a great help for SEO purposes. But like others stated, they might not always be followed.

[Author]

Posts