Meta Tag to Prevent Search Engine Bots

To prevent all search bots from indexing a page:

<meta name="robots" content="noindex">

To prevent just Google:

<meta name="googlebot" content="noindex">

If you have control over it, you probably want to add nofollow to links that point to that page as well:

<a href="privatepage.html" rel="nofollow">Link to private page</a>

Theoretically that's not needed for Google, which claims to drop all pages with noindex from their directory. But there are more search engines out there and it doesn't hurt.

Reference URL

Comments

  1. User Avatar
    Felix Kiunke
    Permalink to comment#

    Also, you might want to add a robots.txt to prevent the bots from indexing your directories, it might look like this:

    User-agent: *
    Disallow: /

    which would prevent all bots from indexing; of course you can also only block certain bots by their user agent; google it :-)

  2. User Avatar
    AkmaOmar
    Permalink to comment#

    Hi.. if use nofollow, that link will be index or not?

    • User Avatar
      radu
      Permalink to comment#

      I’m not sure, but:
      User-agent: *
      Disallow: /
      will do the job!

  3. User Avatar
    Patricio Mendes
    Permalink to comment#

    Very useful especially for admin pages.

  4. User Avatar
    AimIT
    Permalink to comment#

    Hello,

    User-agent: *
    Disallow: /
    If you used this robot txt , than any search engine don’t index full your web site.

    and

    User-agent: *
    Disallow:
    Disallow: /your-folder/
    Disallow: /page.html

    This robot txt help search engine to index full your site but don’t index your-folder and page.html .

  5. User Avatar
    Malik
    Permalink to comment#

    This Might Help ..

    User-agent: Mediapartners-Google
    Disallow:

    User-agent: *
    Disallow: /search
    Allow: /

    Sitemap: http://www.malikshahbaz.com/feeds/posts/default?orderby=UPDATED

Posting Code

You may write comments in Markdown. This makes code easy to post, as you can write inline code like `<div>this</div>` or multiline blocks of code in triple backtick fences (```) with double new lines before and after.

Code of Conduct

Absolutely anyone is welcome to submit a comment here. But not all comments will be posted. Think of it like writing a letter to the editor. All submitted comments will be read, but not all published. Published comments will be on-topic, helpful, and further the discussion or debate.

Want to tell us something privately?

Feel free to use our contact form. That's a great place to let us know about typos or anything off-topic.

Submit a Comment

icon-closeicon-emailicon-linkicon-menuicon-searchicon-tag