Some of what gets blamed on “flaky” software is the hardware: a memory bit that flips, a process killed and quietly restarted, a disk that logs errors for weeks. A test crash on our own build server taught us how easy that is to miss, and changed what Infinite Haiku (IH) watches on every node.

Three crashes, three places

Our build server is a 32-core machine in the lab cluster that compiles and tests every change. On 2026-10-12 one run of our datastore’s test suite (Poet, about a thousand tests) died with a segfault and no Rust message. Rust code doesn’t do that on its own, so we went looking for the bug. We pulled the crash records from the machine and turned the addresses into function names. Each crash was somewhere else:

• inside the memory allocator, freeing memory as a test finished;
• walking an ordered map;
• serialising a metric definition.

Ten more full runs of the same binary under a debugger: no crash. The test that died passed on its own. Code bugs repeat; these didn’t.

Ask the machine, not the code

The kernel log of the build server held 22 crashes since 29 September, and most were not ours: the Rust compiler, the LLVM optimiser, a video encoder. Programs that share no code were dying at unrelated addresses.

Then a direct test. We wrote patterns to memory and read them back, on two groups of cores at the same time, for 15 minutes:

The machine’s memory has no error correction, so nothing had ever reported a flipped bit. The data was simply wrong, now and then: a compiler cache here, a test binary there. The fix belongs to the machine, and the hardware stays in the lab on purpose: a faulty part is a free test case for an observability product.

What IH watches now

The lesson for a monitoring tool is uncomfortable: the evidence sat in the kernel log, and nothing we ran looked there. So IH now does. A new Muse (IH’s name for a telemetry source) runs on every node, reads the kernel’s log read-only, without root or extra privileges, and sends each fault to the Poet as an event on that node, at the moment the kernel logged it:

• segmentation faults and protection faults, with the program, the library and the CPU core;
• machine-check and memory-controller errors, where the hardware reports them;
• disk and filesystem errors;
• out-of-memory kills.

It also counts faults per node and per core, so a dashboard can show “which cores keep crashing” beside the services running on them. Our build server’s pattern would have been visible on the first day: most crashes landed on three of its sixteen cores.

Its first minutes on our own cluster showed something nobody had noticed: our datastore had been killed for running out of memory 20 times on one of the small lab nodes since it last booted. It restarted quietly each time. Now each kill is an event on the timeline, next to the memory graph that explains it.

What this means for you

Silent memory errors, killed processes that restart quickly, a disk that logs errors for weeks: with IH they show up where you already look, on the same timeline as your services, per node and per core. The Muse reads a log your nodes already write; it needs no root access and runs within a tenth of a core and 128 MB, small enough for a Raspberry Pi.

Want IH to watch your own nodes? Ask us for a pilot.