Meta Tags for Instructing Search Bots

Straight from Google:

Meta tags are a great way for webmasters to provide search engines with information about their sites. Meta tags can be used to provide information to all sorts of clients, and each system processes only the meta tags they understand and ignores the rest.

If you want to totally block all search engines:

<meta name="robots" content="noindex, nofollow">
<!-- or -->
<meta name="robots" content="none">
<!-- or if you only care about Google... -->
<meta name="googlebot" content="noindex, nofollow">

Why?

Perhaps it's a somewhat private URL on your site, but not protected in a such a way that a search bot couldn't see it. Like if a page requires a log in to see it, that's blocked from a search bot anyway. But if it's just an obfuscated URL, who knows how search bots get around, so you might wanna prevent the indexing.

Search results is another common one. Weird to have a search result be search results, perhaps. WordPress themes tend to include this:

<?php if (is_search()) { ?>
  <meta name="robots" content="noindex, nofollow">
<?php } ?>

I might alter that a little and tell bots, hey, you can follow links here to help continue indexing the site, but don't index this page specifically.

<meta name="robots" content="noindex, follow">

Others

Search engines tend to use the description tag for text content next to the <title>:

<meta name="description" content="A description of the page" />

See the Google page for other Google-specific ones like the ability to stop Google-specific UI things like their "site link search box".

Comments

  1. User Avatar
    zaimazhar
    Permalink to comment#

    Might be useful, will give it a shot! Thank you.

  2. User Avatar
    Shreeya Web Solutions
    Permalink to comment#

    Meta tags are very important for any website. With the help of these tags, a website is optimized for search engines.

Leave a Comment

Posting Code!

You may write comments in Markdown. This makes code easy to post, as you can write inline code like `<div>this</div>` or multiline blocks of code in triple backtick fences (```) with double new lines before and after.

Code of Conduct

Absolutely anyone is welcome to submit a comment here. But not all comments will be posted. Think of it like writing a letter to the editor. All submitted comments will be read, but not all published. Published comments will be on-topic, helpful, and further the discussion or debate.

Want to tell us something privately?

Feel free to use our contact form. That's a great place to let us know about typos or anything off-topic.

icon-anchoricon-closeicon-emailicon-linkicon-logo-staricon-menuicon-nav-guideicon-searchicon-staricon-tag