It’s extremely useful to have a testing environment for your website, where you can try out new features, tweaks or strategies without effecting your live site. In most cases, however, you will want your sandbox environment kept out of the search engines, so here are a few tips on how to keep it under wraps and out of Google’s index.
Is Your Test Server Currently Being Indexed?
It’s as simple as running a simple Google search query to see if any pages from your development site have been indexed. Use site:yourdomain.co.uk and see how many results are being served, or you can use third-party software to achieve the same end result, it’s entirely down to your preference.
If your site is being indexed, there are a number of ways to correctly prevent this from happening in the future. You can also submit a URL removal request in the Google Search Console, which should remove the page from the index for around 90 days, so you have time to put corrective measures in place.
Any sensitive areas make great use of HTTP Authentication. Anything you want to be hidden from the public eye and out of the search engines can be secured using server-side authentication.
Another great way of securing your content is by only allowing known IP address to see it. This could be your internal network, clients etc.
Robots.txt and Noindex
Unfortunately Noindex in your Robots.txt file isn’t officially supported, and may not work in the long run and for all search engines. It also gives a pointer toward where people shouldn’t be looking!
You can also add a Noindex tag in the robots meta tag or n X-Robots-Tag in the HTTP header. However, this means that Google still has to crawl the pages to even see the tag.