On 15 November we asked this blog’s staging address for a page and read what came back. Two separate instructions told search engines to stay away. One was a header on the response: X-Robots-Tag: noindex, nofollow. The other was a file: robots.txt, containing Disallow: /.
That looks careful. It may be a mistake, and it is one our own deploy tool makes on purpose.
What Google says the pair does
Google’s page on blocking indexing says a noindex rule only works if Google can read the page. If robots.txt blocks the page, the crawler never sees the rule, and the page can still appear in results, for example when other pages link to it.
Its robots.txt guide goes further. A robots.txt file, it says, is not a way to keep a page out of Google. A blocked address can still show up in results, without a description, because Google can list a URL it has never visited. The guide’s own advice for hiding a page is to password-protect it or use noindex.
So the two instructions are not two locks on one door. A noindex rule has to be read to work, and the robots.txt file stops it being read. A crawler that obeys robots.txt reads Disallow: /, turns round, and never meets the header.
We tested that mechanism with Python’s standard robots.txt parser, standing in for a well-behaved crawler. Given our staging file, it says a crawler named Googlebot may not fetch any page. Given a file that says Allow: /, it says it may. That is a stand-in, not Google, but it is the rule Google’s docs describe.
What our tool sends
Our own rules say a site that is not live is served with the noindex header and Disallow: /, and the deploy script adds both to a copy of the site just before it goes up. It applies to every environment of a site marked not live, production included, and to preview and staging on a site that is live.
Sites with a deploy config, by what they send
Counted from our websites folder on 15 November 2026. Test fixtures and a mock-up copy left out.
Source: zxn.uk's own deploy configs, 15 November 2026
On the blog itself the staging address returned the header, and its robots.txt was the two-line block. The live blog at zxn.uk returns neither: its robots.txt allows everything and points to the sitemap. That half is right.
Where it has not bitten
We have not seen a staging address of ours listed. We also have not looked. We did not search Google for them, and we did not run URL Inspection. A blocked address can only be listed if a link to it exists somewhere, and we have not audited where each of ours has been pasted.
So this is a flaw we found by reading, not a fire. The risk is small, but the effect would be the opposite of what the setup says it does: an unfinished client site appearing as a bare link in search is exactly what noindex on staging is meant to prevent.
The working version is already in this blog. The search page, the 404 and thin tag pages carry noindex in the HTML while the live robots.txt lets crawlers in. Google can read the rule, so the rule works. It is the same raw-page view we used when we loaded every post with scripts off: what Google gets from the server is what counts.
What we will change
Google offers two ways to keep a page out: noindex that the crawler can read, or a password. We will use one of each, never the pair.
- Shareable previews: keep the header, drop the
Disallow. The crawler can fetch the page, reads the header and leaves it out. - Anything private: put it behind a password. Then there is nothing to read, and blocking crawling costs nothing.
The price is honest. With no Disallow, a staging page is fetchable by any crawler that finds it, so a password is the answer for anything we would mind someone reading. We have filed that change as a task. Our deploy tool still sends both today.
We described the pair as a feature in our launch checklist, and added a dated correction there. The rest of that list still stands.
Update, 15 November 2026
Later the same day we looked. We ran a Google site search for each of our 15 project addresses, the part of each address that covers its preview and staging copies. Seven belong to sites that are not live and eight to sites that are. Google returned no pages for any of them. One of the 15 was a guess, because that project is missing from our Cloudflare list. As a control, the same kind of search for the live blog returned its pages, so the method does find listed pages.
That is a clean result and a limited one. It is one search per address, on one day, in one search engine. It says nothing about Bing or about listings that appear later. We still have not run URL Inspection, which would need each address verified in Search Console first. So far the flaw has cost us nothing we can see. The change still stands, because the setup would fail the same way the day a link to one of those addresses gets out.



