How do you test for a soft 404 in one command?

Request a URL you invented on the spot and read only the status code: `curl -o /dev/null -w "%{http_code}" https://example.com/definitely-not-real-xyz123`. A correct site answers 404. If it answers 200 you have a soft 404, and every presence check you ran against that host before this moment is void — including the ones that passed.

Is allowing AI crawlers in robots.txt enough to get cited?

No. Robots.txt controls fetching, and fetching is a precondition, not a result. An engine can crawl a page and still never quote it. Crawl permission is the floor: the page still has to answer a real question in a self-contained passage, carry sources a model can check, and exist at a URL that resolves the same way twice.

Is llms.txt an adopted standard?

No. It is a proposal published by Jeremy Howard in September 2024, and llmstxt.org says so on its own page. No search or answer engine has committed to reading it. We publish one because it costs nothing to generate from the same registry as the sitemap — and we never describe having it as standards compliance, because there is no standard to comply with.

We Checked Whether Our Own Site Was AI-Ready. It Wasn't.

Before you can ask whether AI engines cite your site, you have to ask whether they can read it. Ours could not. Every URL we invented returned 200 and served the homepage, which meant robots.txt, sitemap.xml and llms.txt only appeared to exist. Five fixes shipped the same day.

Why did our presence check almost pass?

We set out to measure something specific: whether answer engines quote us. The first step was supposed to be trivial — confirm the three files an engine looks for are actually served. We requested robots.txt. It returned 200. We requested sitemap.xml. 200. Then llms.txt. 200 again. Three out of three. We were one keystroke away from writing "crawlability: fine" in the baseline and moving on to the interesting part.

None of the three files existed. Not one. The site was answering 200 to everything, and we had just run three checks that could not fail.

How do you prove a file actually exists?

By asking for one that does not. Before recording the result we requested a path we had invented on the spot — a random string no deployment could plausibly contain. It returned 200 and served the homepage. That single request invalidated the three that came before it, because a server that answers 200 to an address that cannot exist is not telling you about files. It is telling you nothing at all.

200 is not proof of existence. Every presence check has to include one address that is supposed to fail.

This is now a fixed rule in our measurement spec, and it generalises well past web servers. A check that cannot produce a negative result is not a check. If you have never seen your test fail, you do not know that it runs — which is the same reason we now require every new build gate to ship with cases that must break it.

Why is a soft 404 worse than a missing file?

Google's own crawling documentation names the family: when a URL returns a 2xx status while its content signals an error, Search Console reports a soft 404. Ours was the broader version of the same fault — the homepage behind every address, rather than an error page behind a 200 — and Google's crawl-budget guidance asks for the same remedy: eliminate soft 404s, and return a 404 or 410 for pages that are gone. The reason is mechanical. When every wrong address serves the homepage, the site has an unbounded number of valid URLs, all carrying identical content. A crawler cannot build a map of what exists, because nothing is ever reported as absent.

There is a second cost, and it is the one that made this urgent for us rather than merely untidy. Google's spam policies define scaled content abuse as many pages generated "for the primary purpose of manipulating search rankings and not helping users". That definition turns on intent, and ours was a missing configuration file, not a scheme — so we are not claiming we tripped the policy. The narrower point is the one that made this urgent: what a crawler actually sees, an unbounded set of URLs all serving the same page, is the shape the policy is written about, and we would rather not be in a position where the difference is an argument about our intentions.

The three files that appeared to work are all specified documents, not conventions we guessed at. Robots exclusion is standardised in RFC 9309, sitemaps in the sitemaps.org protocol, and llms.txt is a proposal published in September 2024 — which llmstxt.org describes as a proposal on its own page, and which we therefore implement without ever calling it standards compliance. Three real specifications, three files we did not have, three checks that told us we did.

What was already correct on the site?

Not everything was broken, and saying so plainly matters more than the finding. The two articles already live had been built correctly: each had a unique canonical URL, working reciprocal hreflang links between its Hebrew and English versions, and complete article structured data. The bilingual structure held. What failed was the layer underneath it — the part nobody looks at because it is generated, and generated things are assumed to be there.

That is the shape of the whole failure, and it is not really a web-server story. Every individual signal was green. The audit only turned up anything because one request was designed to come back red.

Why did crawlability come before measurement?

We have no clients. That is a real constraint and we state it on this site rather than dressing it up: our own visibility inside AI answers is the only evidence we will be able to show anyone, which makes measuring it the product rather than a metric attached to one.

But there is no sense asking whether an engine quotes you while the engine cannot map your site. So the order was forced: fix, then measure. Five fixes shipped the same day — a real 404 page, a robots.txt that explicitly welcomes answer-engine crawlers, a sitemap.xml generated from the article registry itself so it cannot drift from what actually ships, an llms.txt from that same registry, and organization structured data on the homepage. Each one was verified against the same control address that exposed the fault. This time it returned 404.

Worth being precise about what those fixes buy, because it is less than it sounds. Google's own guidance on optimising for its AI features says there is no special markup and no separate file that makes content eligible, and warns against writing a second version of your content aimed at machines. Crawlability is the floor, not the lever.

attributed — someone else's measurement, not ours

The lever we had skipped is one the research puts near the top. Aggarwal et al., Generative Engine Optimization (KDD 2024), name Cite Sources — alongside Quotation Addition and Statistics Addition — as their three top-performing methods, together reaching a relative improvement of 30–40% on their Position-Adjusted Word Count visibility metric. Citations were not the single strongest method: the paper reports Cite Sources as "relatively less effective when used alone (8% lower than Quotation Addition)", and says effectiveness varies across domains. That is their result on their corpus, not ours on ours. We report it because it is why this article was rewritten, and we do not promise it transfers.

What did we still get wrong?

Two things, and both were worse than the original bug.

The first, the same day the fixes shipped and a day before this article went live: a different site we had built — the one that displays our own build stages — had the identical fault. We had fixed the product and not the tool we built to report on the product. Someone else caught it. The correction is logged under our own name in the same rewind log as everything else.

The second came after publication, and is the reason this page has a revision date on it. The first version of this article contained no outbound links at all. It described a study about citing sources, and cited nothing. Our own spec required links to primary sources, and the article arguing for the rule was the one page that broke it. We could have added the links quietly. Correcting visibly, with the date attached, is the rule we published — so this is us following it.