Counters have been counting robots
Counters here have been counting robots. All of them, without exception. Until this month nobody here could say how many, because nothing was measuring it — and a crawler that requests a counter image looks exactly like a person who requests a counter image.
Now it is measured. The numbers are worse than expected.
What the first days show
Counting started on 9 August. Across all counters on this site, the share of hits coming from crawlers rather than people:
- 9 August — 1,193 of 16,070 hits, about 7%
- 10 August — 1,624 of 20,181
- 11 August — 1,954 of 20,918
- 12 August — 2,707 of 22,740, about 12%
Somewhere between seven and twelve percent, then. Four days is not a trend and will not be presented as one — the variation may be the crawlers, or it may be detection catching more as the list of signatures grows.
The number that is actually striking
The average hides the interesting part. Looking at individual counters day by day, there were 2,412 counter-days measured, and on 631 of them the majority of counted hits were bots.
On roughly a quarter of counter-days, in other words, most of the "visitors" were not visitors. That is what a quiet site looks like from the inside: a handful of real people, and a steady patrol of crawlers that never stops, never sleeps, and never reads anything.
On a small site, the count that creeps up when nobody could possibly be reading now has an explanation, and it sits on the statistics page as Bots (30 days).
How they are recognised
By the name the client gives for itself. Baiduspider, Googlebot, Amazonbot, AhrefsBot, the various uptime monitors, and the plain tools that do not pretend otherwise: curl, wget, python-requests, Scrapy. A request that gives no name at all is treated as a bot too, because browsers always give one.
The list is deliberately short. It only matches signatures that are unambiguous, which means well-behaved crawlers get caught and a crawler that lies about being Chrome does not. That is the right way round: a bot counted as a person is a small error, and a person discarded as a bot is data that can never be got back.
The decision that has not been made
Here is the honest part. Bots are still counted in the totals. They are shown separately, but the big number on the counter still includes them.
Excluding them is a single setting, and it is deliberately switched off. Turning it on would drop the visible count of every counter on this site by around a tenth, overnight, without anybody having been asked. For counters that have been running since 2007 and have become a small point of pride on somebody's page, quietly deducting a chunk of their total is not a neutral technical correction.
So the number is being shown first. If it should be deducted rather than merely reported, that is worth hearing about — the support form is on this site, and there is a real person on the other end of it.