Treehouse: Grow your CSS skills. Land your dream job.

Last updated on:

Meta Tag to Prevent Search Engine Bots

To prevent all search bots from indexing a page:

<meta name="robots" content="noindex">

To prevent just Google:

<meta name="googlebot" content="noindex">

If you have control over it, you probably want to add nofollow to links that point to that page as well:

<a href="privatepage.html" rel="nofollow">Link to private page</a>

Theoretically that's not needed for Google, which claims to drop all pages with noindex from their directory. But there are more search engines out there and it doesn't hurt.

Reference URL

Comments

  1. Felix Kiunke
    Permalink to comment#

    Also, you might want to add a robots.txt to prevent the bots from indexing your directories, it might look like this:

    User-agent: *
    Disallow: /

    which would prevent all bots from indexing; of course you can also only block certain bots by their user agent; google it :-)

  2. AkmaOmar
    Permalink to comment#

    Hi.. if use nofollow, that link will be index or not?

    • radu
      Permalink to comment#

      I’m not sure, but:
      User-agent: *
      Disallow: /
      will do the job!

  3. Patricio Mendes
    Permalink to comment#

    Very useful especially for admin pages.

  4. AimIT
    Permalink to comment#

    Hello,

    User-agent: *
    Disallow: /
    If you used this robot txt , than any search engine don’t index full your web site.

    and

    User-agent: *
    Disallow:
    Disallow: /your-folder/
    Disallow: /page.html

    This robot txt help search engine to index full your site but don’t index your-folder and page.html .

  5. Malik
    Permalink to comment#

    This Might Help ..

    User-agent: Mediapartners-Google
    Disallow:

    User-agent: *
    Disallow: /search
    Allow: /

    Sitemap: http://www.malikshahbaz.com/feeds/posts/default?orderby=UPDATED

Leave a Comment

Posting Code

We highly encourage you to post problematic HTML/CSS/JavaScript over on CodePen and include the link in your post. It's much easier to see, understand, and help with when you do that.

Markdown is supported, so you can write inline code like `<div>this</div>` or multiline blocks of code in in triple backtick fences like this:

```
<script>
  function example() {
    element.innerHTML = "<div>code</div>";
  }
</script>
```