Contact support

We reply by e-mail, usually within two days.

Google reCAPTCHA checks this submission against abuse; data is sent to Google. The script loads only when this form is opened.

← All posts

The tidy-up that would have made 180 addresses vanish

· Lukas Wojcik

Two days in August carry thirteen and seven posts. Every other day of the month carries three. Evening it out looks like a five-minute job, and it would have taken 180 working addresses offline.

the twenty posts of 27 and 28 August, when the dates were to be tidied already fetched by crawlers never fetched fifteen of twenty had 21 to 38 crawler fetches each moving them forward would have turned 180 addresses into 404 for days the five on the right cost nothing to move — four of them say «today»

What a future date does

Scheduling here is one condition in the query: a post is only fetched if its publication time has passed. That is what makes a queue possible, and it also means a post moved into the future stops existing. Its own address answers 404, it leaves the list, it leaves the sitemap. All three agree with each other, which is the right design and exactly the problem.

Each post lives at twelve addresses: eleven language paths and one that negotiates. Fifteen posts moved forward is 180 addresses answering 404 until their new date arrives.

Which of them would have noticed

The log answers that precisely. Of the twenty posts, fifteen had already been fetched by crawlers — between 21 and 38 fetches each, with Googlebot on up to five of them and bingbot on up to twelve. Those are pages that are in an index somewhere. Turning them into errors for several days is not tidying, it is a withdrawal and a re-submission.

The remaining five had never been fetched by anything. Moving those costs nothing at all, and that is the useful half of the measurement: it is not that scheduling is dangerous, it is that it is only free before anyone has looked.

The other half of the answer

Four of those five could not be moved either, for a reason that has nothing to do with crawlers. They say today, and yesterday, and the day before — thirteen such phrases between them. A post dated three days later that says «yesterday» is simply wrong, in eleven languages at once.

The alternative to moving forward is moving backwards, and that is worse: these posts were written on the days they carry. Backdating them to make a calendar look even would be a small lie told to a reader in order to please a chart.

What we did instead

Nothing to the past, and a tool for the future. New posts now take the next free slot in a grid of three a day, and the tool refuses to move a post that is already published unless it is told twice — and prints the number of addresses that would go dark before it does.

The uneven days stay uneven. They are what happened.

What generalizes

The cost of a change is not in the change, it is in what has already happened to the thing being changed. A publication date is a number in a column right up until somebody has fetched the page it belongs to, and after that it is a promise.

Both halves of that were in the same log, and neither was in the code.

Advertisement