Forums

The forums ran from 2008-2020 and are now closed and viewable here as an archive.

Home › Forums › Other › Robots.txt or <meta name="robots"> ?

  • This topic is empty.
Viewing 4 posts - 1 through 4 (of 4 total)
  • Author
    Posts
  • June 4, 2009 at 2:27 pm #25046
    jerred
    Member

    I want to keep dev.domain.com from being indexed as I’m working on a redesign and would like to keep the content from being indexed.

    I currently have the dev.domain.com subdomain in the folder /htdocs/dev.

    What is the best way to keep the entire dev subdomain (or the /htdocs/dev folder) from being indexed?

    June 8, 2009 at 7:23 pm #58802
    davesgonebananas
    Member

    Put the following in /htdocs/dev/robots.txt

    Code:
    User-agent: *
    Disallow: /
    June 9, 2009 at 11:07 am #58837
    jerred
    Member

    Thanks Dave.

    Done and Done!

    June 17, 2009 at 12:18 pm #59265
    Vinnix
    Member

    The Robots.txt or meta Robots caches the site for SEO right?
    It seems like it does that once in a while I am assuming based on traffic flow, my question is that can you cache it yourself without the search engine crawling for it?

  • Author
    Posts
Viewing 4 posts - 1 through 4 (of 4 total)
  • The forum ‘Other’ is closed to new topics and replies.