Twelve URLs for one article

Every post on this blog lives at twelve addresses. Eleven carry a language prefix — /en/blog/…, /de/blog/… and nine more — and the twelfth has no prefix at all. That one reads the browser's Accept-Language header and answers in whichever of the eleven it recognises.

Over the 27th and 28th of August, crawlers fetched 1,609 blog pages from this site. Readers fetched 628.

What the pages already tell each other

All twelve say the same three things in their head, and I checked all three today rather than trusting the code.

The canonical link on each page points at itself. Eleven hreflang lines name the sibling language versions, and a twelfth names the prefix-less address as x-default, because that address belongs to no single language. The prefix-less address answers with Vary: Accept-Language, Accept-Encoding, so anything caching in front of it knows the body depends on a request header.

The negotiation works. Asking for the same URL with Accept-Language: de returns a page marked lang="de", with ja a Japanese one, with a code nobody uses, English. Twelve requests alternating between four languages returned four correct bodies three times over, so the page cache is keyed on the language and not only on the path.

This is what the specification asks for: a complete, reciprocal group where every member names itself and all its siblings. It is the arrangement that lets a search engine treat twelve addresses as one article in twelve languages instead of twelve articles competing with each other.

It does not make anyone fetch less

It also does not reduce fetching, and the access log says so without much room for interpretation.

ClaudeBot made 396 blog requests across 36 posts — exactly 11.0 language versions each, every language, every time. bingbot averaged 8.7 across 37 posts, PetalBot 6.0, Applebot 4.3. SemrushBot managed 1.8. Googlebot made 72 requests across 42 posts, an average of 1.5 language versions per post, and it is the only one of the six that behaves as though the annotation reached it.

Across all crawlers, 73 of the 76 posts were touched at least once, and 68 of those in more than one language. The average was 7.84 language versions per post out of eleven.

None of this is a violation. hreflang is a hint about how pages relate, addressed to the part of a search engine that decides which result to show. It is not an instruction to the part that decides what to download. A crawler that has never seen /uk/blog/… has no way of knowing it is a translation until it fetches it and reads the head — the annotation can only take effect after the request it might have saved.

What it costs

Less than the paragraph above suggests. Those 1,689 crawler requests to blog pages are 0.5 per cent of the 373,019 requests this virtual host answered in two days, and 19.9 MB of the 4,363 MB it sent. A blog page comes back in about a tenth of a second, so the whole eleven-fold crawl costs somewhere around eighty seconds of processor time a day.

The sitemap costs a little more than I expected. It lists 3,136 addresses with 13,248 alternate-language annotations attached to them: 1.85 MB of XML, 35 KB once compressed. It is rebuilt at most every six hours, which is why a post published at ten past one in the afternoon appears in it complete some hours later and not immediately.

Nothing in those numbers is worth changing. The only lever that would reduce the fetching is not having the addresses, and the addresses are the point.

Readers and crawlers want different things

The same two days, split by language, show two quite different habits.

Crawlers spread almost evenly: between 5.5 and 10.5 per cent of their blog requests went to each of the eleven prefixed versions, plus 16 per cent to the prefix-less one. That is close to what you would expect from something walking a list.

Readers do not spread evenly. 38 per cent of their requests went to the prefix-less address and another 18 per cent to /en/, so more than half arrived either in English or by negotiation. The rest divides across ten languages, the largest German at 7 per cent, the smallest French at 2 per cent.

That remaining 44 per cent is the entire reason the language paths exist. It is also small enough that it would be easy to talk yourself out of them — which is the same mistake as reading the crawler column and concluding the pages are popular.

What generalizes

A correct hint is not an instruction. Everything in the head of these pages is right, reciprocal and verifiable, and five of six crawlers fetch as if none of it were there. The specification describes what the annotation means, not what any particular program will do when it sees it.

So the useful question was never whether the markup is correct. It was how many times each fetcher actually comes, which is a question only the log can answer — and having answered it, the honest conclusion is that the eleven-fold crawling costs half a per cent and should be left alone.