07480 257117 Talk to us
EngineeringField notes

Our staging sites say noindex. Google may never read it.

Every preview and staging site we deploy carries a noindex header and a robots.txt that blocks all crawling. Google's own docs say the second can stop it ever seeing the first. We checked what we actually serve.

2 set large on a pale grey background beside a terminal listing what a staging site sends: a noindex header, a robots.txt of Disallow slash, and the result that a crawler which obeys robots.txt may not fetch the page, above the words: Instructions on every staging site, and the second can hide the first
2instructions sent by every staging site we deploy
15sites with a deploy config in our websites folder
7of them marked not live, so every environment sends both
8live sites that still send both on preview and staging

On 15 November we asked this blog’s staging address for a page and read what came back. Two separate instructions told search engines to stay away. One was a header on the response: X-Robots-Tag: noindex, nofollow. The other was a file: robots.txt, containing Disallow: /.

2instructions on every staging site we deploy, and Google's docs say the second can hide the first

That looks careful. It may be a mistake, and it is one our own deploy tool makes on purpose.

What Google says the pair does

Google’s page on blocking indexing says a noindex rule only works if Google can read the page. If robots.txt blocks the page, the crawler never sees the rule, and the page can still appear in results, for example when other pages link to it.

Its robots.txt guide goes further. A robots.txt file, it says, is not a way to keep a page out of Google. A blocked address can still show up in results, without a description, because Google can list a URL it has never visited. The guide’s own advice for hiding a page is to password-protect it or use noindex.

So the two instructions are not two locks on one door. A noindex rule has to be read to work, and the robots.txt file stops it being read. A crawler that obeys robots.txt reads Disallow: /, turns round, and never meets the header.

We tested that mechanism with Python’s standard robots.txt parser, standing in for a well-behaved crawler. Given our staging file, it says a crawler named Googlebot may not fetch any page. Given a file that says Allow: /, it says it may. That is a stand-in, not Google, but it is the rule Google’s docs describe.

What our tool sends

Our own rules say a site that is not live is served with the noindex header and Disallow: /, and the deploy script adds both to a copy of the site just before it goes up. It applies to every environment of a site marked not live, production included, and to preview and staging on a site that is live.

Sites with a deploy config, by what they send

Counted from our websites folder on 15 November 2026. Test fixtures and a mock-up copy left out.

Not live: both, in everyenvironment7Live: both, on preview andstaging only8

Source: zxn.uk's own deploy configs, 15 November 2026

On the blog itself the staging address returned the header, and its robots.txt was the two-line block. The live blog at zxn.uk returns neither: its robots.txt allows everything and points to the sitemap. That half is right.

Where it has not bitten

We have not seen a staging address of ours listed. We also have not looked. We did not search Google for them, and we did not run URL Inspection. A blocked address can only be listed if a link to it exists somewhere, and we have not audited where each of ours has been pasted.

So this is a flaw we found by reading, not a fire. The risk is small, but the effect would be the opposite of what the setup says it does: an unfinished client site appearing as a bare link in search is exactly what noindex on staging is meant to prevent.

The working version is already in this blog. The search page, the 404 and thin tag pages carry noindex in the HTML while the live robots.txt lets crawlers in. Google can read the rule, so the rule works. It is the same raw-page view we used when we loaded every post with scripts off: what Google gets from the server is what counts.

What we will change

Google offers two ways to keep a page out: noindex that the crawler can read, or a password. We will use one of each, never the pair.

The price is honest. With no Disallow, a staging page is fetchable by any crawler that finds it, so a password is the answer for anything we would mind someone reading. We have filed that change as a task. Our deploy tool still sends both today.

We described the pair as a feature in our launch checklist, and added a dated correction there. The rest of that list still stands.

Update, 15 November 2026

Later the same day we looked. We ran a Google site search for each of our 15 project addresses, the part of each address that covers its preview and staging copies. Seven belong to sites that are not live and eight to sites that are. Google returned no pages for any of them. One of the 15 was a guess, because that project is missing from our Cloudflare list. As a control, the same kind of search for the live blog returned its pages, so the method does find listed pages.

That is a clean result and a limited one. It is one search per address, on one day, in one search engine. It says nothing about Bing or about listings that appear later. We still have not run URL Inspection, which would need each address verified in Search Console first. So far the flaw has cost us nothing we can see. The change still stands, because the setup would fail the same way the day a link to one of those addresses gets out.

Where the numbers come from

  1. Google Search Central, Block Search indexing with noindex, read 15 November 2026.
  2. Google Search Central, Introduction to robots.txt, read 15 November 2026.
  3. zxn.uk's own deploy script and websites rules, read 15 November 2026: what is added to a site that is not live, and the count of site configs and their live flag.
  4. zxn.uk's own check of this blog's staging address, 15 November 2026: the response header and the robots.txt it serves.
  5. A test with Python's standard robots.txt parser, 15 November 2026, standing in for a crawler that obeys robots.txt. It is not Googlebot.
  6. zxn.uk's own Google searches, later on 15 November 2026: one site search for each of 15 project addresses, and one for the live blog as a control.

Mark Lovelady, Director. Writes a lot of our code and learns from every project.