Contact support

We reply by e-mail, usually within two days.

Google reCAPTCHA checks this submission against abuse; data is sent to Google. The script loads only when this form is opened.

← All posts

An 870 MB file in a tmpfs, and the reboot that followed

· Lukas Wojcik

On 28 August at 19:35 I unpacked a day of access log into /tmp so that two days could be read in one pass. At 19:44 the machine rebooted.

what the machine has, and what one file took 3,794 MB of memory, of which /tmp can hold 1,900 870 MB written into it at 19:35 19:35 — a day of log unpacked into /tmp to read two days at once 19:44 — SSH stops answering, the site takes 11 s instead of 0.1 s 19:47 — the machine is back, uptime 0 minutes MariaDB comes up with crash recovery: it was not a clean shutdown the journal does not survive a reboot, so this cannot be proven — only dated

The property that was not checked

On this machine /tmp is a tmpfs. It is memory. df shows 1.9 GB free and says nothing at all about where that space comes from, and the machine has 3,794 MB in total.

So an 870 MB file went into a quarter of the memory of a small server that was also serving a site and running somebody else's test job. Nine minutes later SSH stopped accepting connections, the site answered in 11 seconds instead of 0.1, and shortly after that it was back with an uptime of zero minutes.

What can and cannot be said

MariaDB came up with crash recovery, which means the shutdown was not clean. Beyond that, the honest answer is that this cannot be proven: the journal on this machine is volatile, so it does not survive a reboot, and there is no kern.log carrying an out-of-memory line. The evidence is a timestamp nine minutes before the event and a file in the wrong place.

That is worth writing down as it is, rather than as a conclusion. «Filled the memory and the kernel gave up» is a plausible story. It is not a measured one, and the difference matters more here than the certainty would.

What the damage was

None that could be found. CHECK TABLE on the five most important tables came back OK, the row counts were unchanged, counting was still recording that same day, and all sixteen scheduled jobs were intact. The site answered in 0.14 seconds again.

One thing was lost: a long-running job belonging to a different session on the same machine. It had nothing to do with this work and it went down with everything else, which is the part of an outage that no local check reports.

The rule that came out of it

Log files here run 700 to 900 MB a day. Unpacking one into /tmp means putting a quarter of the machine's memory somewhere that looks like disk and behaves like RAM.

Large intermediate files go to /var/tmp or a home directory, both of which are on the disk. Better still, they do not get unpacked at all: zcat -f file.gz | awk ... reads in a stream and needs no destination. Every measurement in the posts that followed was taken that way.

What generalizes

A filesystem reports how much room it has. It does not report what the room is made of, and the two questions have different answers on exactly the systems where it matters most.

The check is one line and it was skipped because the file was «temporary». Temporary was true. It was the wrong property to be thinking about.

Advertisement