Meta Tags for Instructing Search Bots

Avatar of Chris Coyier
Chris Coyier on

Straight from Google:

Meta tags are a great way for webmasters to provide search engines with information about their sites. Meta tags can be used to provide information to all sorts of clients, and each system processes only the meta tags they understand and ignores the rest.

If you want to totally block all search engines:

<meta name="robots" content="noindex, nofollow">
<!-- or -->
<meta name="robots" content="none">
<!-- or if you only care about Google... -->
<meta name="googlebot" content="noindex, nofollow">

Why?

Perhaps it’s a somewhat private URL on your site, but not protected in a such a way that a search bot couldn’t see it. Like if a page requires a log in to see it, that’s blocked from a search bot anyway. But if it’s just an obfuscated URL, who knows how search bots get around, so you might wanna prevent the indexing.

Search results is another common one. Weird to have a search result be search results, perhaps. WordPress themes tend to include this:

<?php if (is_search()) { ?>
  <meta name="robots" content="noindex, nofollow">
<?php } ?>

I might alter that a little and tell bots, hey, you can follow links here to help continue indexing the site, but don’t index this page specifically.

<meta name="robots" content="noindex, follow">

Others

Search engines tend to use the description tag for text content next to the <title>:

<meta name="description" content="A description of the page" />

See the Google page for other Google-specific ones like the ability to stop Google-specific UI things like their “site link search box”.