CSS
-Tricks
treehouse :
what would you like to learn today?
Web Design
Web Development
iOS Development
Show search box
Search
Search in:
All
Articles
Forums
Snippets
Videos
✕
Home
Forums
Snippets
Gallery
Videos
Almanac
Demos
Lodge
Navigation 'n' Search
Forums
Illustration by Nick Sirotich
Forums
»
Other Discussions
Robots.txt or ?
jerred
Permalink to comment
#
June 2009
I want to keep dev.domain.com from being indexed as I'm working on a redesign and would like to keep the content from being indexed.
I currently have the dev.domain.com subdomain in the folder /htdocs/dev.
What is the best way to keep the entire dev subdomain (or the /htdocs/dev folder) from being indexed?
davesgonebananas
Permalink to comment
#
June 2009
Put the following in /htdocs/dev/robots.txt
User-agent: *
Disallow: /
jerred
Permalink to comment
#
June 2009
Thanks Dave.
Done and Done!
Vinnix
Permalink to comment
#
June 2009
The Robots.txt or meta Robots caches the site for SEO right?
It seems like it does that once in a while I am assuming based on traffic flow, my question is that can you cache it yourself without the search engine crawling for it?
Add a Comment
I currently have the dev.domain.com subdomain in the folder /htdocs/dev.
What is the best way to keep the entire dev subdomain (or the /htdocs/dev folder) from being indexed?
Done and Done!
It seems like it does that once in a while I am assuming based on traffic flow, my question is that can you cache it yourself without the search engine crawling for it?