My site is up and running but I'd like google to not crawl that site since I don't want any canonical problems or anything. Is there any Media Temple user here who has found a solution to this problem?
Thanks Mark! I guess they do some kind of minor blocking which is why Google only indexes it if you make it public.
I did actually ask them and they gave a similar robots.txt answer. I'm not really sure how to specifically block http://url.s83596.gridserver.com/ with the robots.txt - as far as I know you can block files and folders, not urls.
But it seems it won't be a problem since no one else has really complained about this before.
The basic rule of search engines is that the only way a page can be indexed is if you either submitted it to them to index, or they could crawl to the site from another URL.
If there are no links to the site on the internet then the site should not be indexed.
These are no-follow links on CSS Tricks so they should not be indexed (yes spammers, you are wasting your time :-) )
My site is up and running but I'd like google to not crawl that site since I don't want any canonical problems or anything. Is there any Media Temple user here who has found a solution to this problem?
https://twitter.com/#!/mediatemple/status/146161602700378112
I did actually ask them and they gave a similar robots.txt answer. I'm not really sure how to specifically block http://url.s83596.gridserver.com/ with the robots.txt - as far as I know you can block files and folders, not urls.
But it seems it won't be a problem since no one else has really complained about this before.
Thanks again
If there are no links to the site on the internet then the site should not be indexed.
These are no-follow links on CSS Tricks so they should not be indexed (yes spammers, you are wasting your time :-) )
http://www.raymondfong.net/search-engines/getting-delisted-from-google-counter-seo/