The watchdog that refuses to guess

A counter can break quietly. The site gets redesigned, the embed does not survive the move, and nothing announces it — the statistics simply stop, and the owner finds out months later, if at all.

An obvious feature suggests itself: notice the silence and send a note. The obvious way to build it is to infer breakage from the numbers, which are already there. That version was built, measured, and thrown away.

The measurement that killed it

The rule was: counted regularly, then nothing for fourteen days. To test it honestly you cannot look forward, so it was run backwards — against three cut-off dates 30, 45 and 60 days in the past — and then checked which of the counters it would have reported were in fact still running afterwards.

threshold	reported	wrong	correct
active on 10 of 42 days	79	56	29 %
active on 20 of 42 days	39	29	26 %
active on 30 of 42 days	10	7	30 %
active on 35 of 42 days	5	3	40 %

No threshold got past 40 %. Tightening it did not help; it only reduced the number of guesses while leaving most of them wrong.

Why, and it is not a bug

The reason is the audience. Of 2,282 counters with traffic, 60 % record something on at most six days out of forty-two. These are small sites. A gap is their normal condition. "Fourteen days of nothing" almost always means a holiday, a quiet month, a page nobody happened to open — not a broken embed.

A watchdog that is wrong two times in three does not merely fail to help. It teaches people to dismiss it, which costs them the one time it is right.

So it looks instead of guessing

The silence is still used, but only as a pre-filter. It is free, it runs over data already collected, and it is allowed to be imprecise because it decides nothing. The decision is made by fetching the page and checking whether the counter is still in it. A 30 % guess becomes an observation.

Three things follow from that, and each of them narrows the feature:

• No page address, no letter. Only 41 % of active counters have one; for the rest the page is recorded as unknown, because the embed cannot report it. Warning four people correctly beats warning thirteen with noise.
• Unclear is not broken. If the page cannot be judged — a caching layer, a JavaScript shell, a server error — the result is "unclear". We did not see it, which is a different statement from it not being there.
• A suspicion must survive two nights. A site that fails to answer for one evening is not a reason to send mail.

And the mail itself goes only to people who confirmed they want it. There are 4,145 addresses in the old counter table, left over from the previous service. They are allowed to answer a request; they never asked to be written to, and the difference between those two things is the whole of it.