Why the model runs on another machine

No language model runs on the machine that serves this site. The summaries are computed on a second device that fetches jobs, does the work, and brings the sentences back.

The reason is arithmetic. A 3B model occupies over a gigabyte of memory and needs seconds per answer. The heaviest database query in this entire application takes 581 milliseconds. Putting the two on the same board means the counter images of 379,000 counters wait behind a text generator, and they are the actual product.

What that costs, and what it buys

The cost is a second machine and a queue. Jobs are handed out at most twenty at a time; the queue has a ceiling, and when it is full nothing new is created — the sentences already written stay visible. A summary that is a week old is fine. A summary that is missing is fine. A statistics page that is slow because something else on the board is thinking is not.

What it buys is that the counting path never shares a machine with anything unpredictable. Drawing a counter image is a small, bounded amount of work. Generating text is not: the measured range for one sentence here is 5.4 to 54.3 seconds, a factor of ten between the fast case and the slow one, decided by nothing the Pi can see in advance.

What leaves the house

This is the part that had to be decided before any of it was built, because it is the part that cannot be fixed afterwards.

The second machine receives only numbers that are already public on the counter's own statistics page, and only from counters whose statistics page is public at all — not private, not password-protected. The worker learns nothing that a visitor to that page could not read by opening it.

That is not a happy side effect of how the code turned out. The candidate query filters on it, and without that filter the whole feature would be the transfer of other people's data to a second device. There is no version of "we'll be careful with it" that is as good as not sending it.

The general shape

Two rules came out of this that are worth more than the feature itself.

Keep unbounded work off the bounded path. Anything whose runtime you cannot predict within an order of magnitude does not belong on the machine that has to answer in milliseconds. Not "usually fine" — the slow case is the one that matters, and it always arrives.

Decide what may leave before deciding what to build. If the answer had been "the worker needs the private counters too", the correct response was not to secure the transfer. It was that the feature does not cover private counters, which is what it does.