Testing with curl tests the wrong thing

A counter request that arrives without a browser identity writes into two of the twelve places a counted visit normally touches. It answers 200. It returns a valid counter image. Nothing anywhere says that ten of the twelve were skipped.

What the filter is for, and what it costs you

Counters here separate robots from people, and the separation is done on the identity the client gives for itself. The list of markers is short and deliberately unambiguous, and curl/ is on it. So is an empty identity, on the reasoning that a request with no user agent at all is not a browser.

Both of those are correct. The service is not wrong to treat a bare curl as a robot; that is what it is. The problem is that the person driving the curl is usually not testing whether robot detection works. They are testing whether counting works, and they have quietly asked for the robot path.

What they get: the day's hit total goes up by one, the bot column goes up by one, and that is all. No unique-visitor row, so nothing about visitors. No entry in the online list. No page, no country, no browser, no operating system, no device, no hour. The parts of the system that anybody would actually want to check are the parts that did not run.

The failure has no symptom

This is the part worth dwelling on. A test that measures nothing usually announces itself: an error, an empty result, a zero where a number should be. Here the response is 200. The body is a real counter image, 3,361 bytes of it. The number in the picture even went up, because the hit total is one of the two things that did happen.

So the probe looks like a pass. Everything downstream of it — "the counting path works", "the new column is being written", "the change did not break anything" — is a conclusion drawn from a run that skipped most of the code it was supposed to exercise.

It happened while writing this

The first version of the measurement behind this post requested /c/<number>. That is not the address of a counter image; the real one carries the design and an extension, /c/<number>-<design>.png. The request returned 404.

The script printed, three times: writes into 0 of 12 places. Which is true, and which is also exactly what a working bot filter would look like if you squinted. The 404 was in the output, in the same table, one row above the zeros. It went unread for a minute because the zeros were the interesting part and they matched what was expected.

That is the same mistake as the one this post is about, one level up: a measurement that produced a plausible answer for a reason nobody checked. It was caught only because the script counted rows in twelve tables instead of trusting the request — a request that fails and a request that is filtered look identical from the outside.

What to do instead

Send a browser identity. One flag, -A with a real user agent string, and the same request writes into ten of the twelve places instead of two.

And count something on the other side. The response code tells you the request arrived; it does not tell you what the request did. For anything where the point is the side effect — a counter, a queue, a log line, a row somewhere — the probe has to look at the side effect. Before-and-after counts across every table involved took twenty lines here and turned an ambiguous result into an unambiguous one twice: once for the filter, once for the 404.

Two of the twelve places stayed empty even with a browser identity: the bot column, correctly, and the referrer list, which did not record the referrer that was sent. That second one is not explained here, because it has not been chased down yet. It is reported because the alternative is to publish a table with eleven rows and call it twelve.