A monitoring graph makes claims: this is how busy the machine was, this is what nothing looked like, these parts add up to that whole. If those claims are wrong, you make decisions on a picture of a system you do not have. We got several of them wrong in Infinite Haiku and fixed them. This post covers four: why a process at 400% CPU can be correct, why missing data must never be drawn as zero, why a stacked chart needs exactly one additive parent, and how one metric turned into a menu entry per container.

Infinite Haiku’s timeline renderer, AGS, draws hierarchical systems as stacked areas over time: a host, its application groups, applications and processes; a cluster, its nodes, pods and containers. Stacking is powerful because it lets you see a whole and its parts at once. It is also easy to make it lie.

400% is not a bug

A user looked at a laptop in AGS and saw a process at 400% CPU. The laptop was not over capacity. The number was in core percent: 100% means one logical CPU fully busy, so 400% means four. That is the convention of the sysinfo crate our macOS Muse uses.

Core percent is a perfectly good unit, but it is a bad default for a stacked host view. The reader’s question is usually “how much of this machine is being used?”, and the answer to that has a different denominator. So the macOS Muse now reports two process CPU metrics:

• process.cpu.capacity_percent, the default: core percent divided by the number of logical CPUs, so 100% is the whole machine;
• process.cpu.usage_percent, shown as “Process CPU (core %)”, unchanged. Its history is not reinterpreted.

Two caveats are spelled out rather than hidden. Logical CPUs are weighted equally, although performance and efficiency cores do not do the same work. And the sum of collected processes need not equal host busy CPU, because some processes are not visible to the collector. A stacked view can show the gap as an explicit remainder; it must not rescale the children until they fit.

Missing is not zero

If the capacity cannot be computed because the logical CPU count is unknown, the Muse emits no measurement, not a zero. This sounds small and is the rule we break most easily.

A gap in telemetry can mean many things: the collector was slow, the source was unavailable, the query has not loaded yet, the data aged out of the retained tier, or the process really was idle. Only the last one is a zero. Drawing the others as zero invents a measurement and, worse, one that looks reassuring. A crash-looping container that reports no CPU is not using 0% CPU.

So we keep the distinction all the way through:

• Coarse queries return the mean of observed samples, with the observed minimum and maximum, the observed count, and the expected count. Missing samples are left out of the mean, not averaged in as zeros.
• With partial coverage, a stacked area shows the known contribution per expected sample, and the summary reports coverage separately.
• AGS distinguishes “Loading” from a confirmed “No observation” band, and both from a measured zero, which is valid coverage.
• A short-gap connector is drawn dashed and unfilled, and never extends what the query claims to cover.
• We do not hold the last value indefinitely.

Our tests must distinguish missing data from measured zero, and the dashboards follow the same rule: measured zero, no data and “not reported” are three different states.

Stacking needs one additive parent

A stack says “these parts sum to that whole”. That is only true for one disjoint partition at a time. A container belongs to a pod, which belongs to a node, and also to a Deployment and a namespace. Those are different partitions of the same CPU. Stack a container under two parents and you count it twice.

AGS therefore stacks along exactly one additive hierarchy at a time and shows the others as alternative views or relationship overlays. The same rule applies to units:

• percentages can be added only when every child shares a denominator; core percent from a four-core host and a sixteen-core host do not add up;
• process RSS includes shared memory, so summing it is not host memory in use;
• latency quantiles cannot be summed, and averages cannot be averaged without their counts;
• a trace tells you which tasks ran in a process, not how its CPU divides between them, so we never attribute process CPU to spans by duration and call it measured.

When readings disagree, we show the disagreement, for example as an “unattributed” or “shared” remainder. Silent rescaling makes inconsistent data look correct, which is the one thing a monitoring tool must not do.

One metric, one menu entry

The Kubernetes version of these bugs was a menu. Kabuki’s metric menu showed one CPU entry per container instead of a single “CPU” with containers, pods, nodes and the cluster underneath.

The cause: the Kubernetes Muse described the container inside the metric’s description, and the menu treated every distinct description as a distinct metric. Our checks confirmed that samples existed, which they did. Nothing checked that one metric appeared once with many entities below it.

The fix had three parts: the Muse emits stable metric descriptions (the live cluster now reports shared k8s.cpu.usage and k8s.memory.usage metrics), metrics are identified by their meaning rather than their full description, and an offline regression test catches the split before anything is deployed. Every Muse now needs a fast contract test that checks metric identity, entity identity, hierarchy and menu grouping. “There is data” is not the same as “the data has the right shape”.

The common thread

Each of these was a graph that looked plausible and meant something else. Honest graphs come from a few boring rules applied everywhere: name the denominator, keep missing separate from zero, stack along one partition, and test the shape of the data as well as its existence. For you, that means a graph you can act on without first asking what it is hiding.

Follow the build on this blog, or ask us for a pilot if you want graphs like these on your own systems.