Most observability products ship every measurement you produce to their cloud, store it there and bill you for it, and your network bill charges you again for the egress. In the summer of 2024 Infinite Haiku’s renderer got a home, Kabuki, and a source of real data, Poet, designed the other way round: it runs in your infrastructure, next to the systems it watches, and hands the browser exactly what it needs to draw. Origins, part 2 of 3.

A home for the renderer

AGS could draw, but a canvas is not a product. On 2 June 2024 work started on Kabuki, the web application around it, written in Rust with Leptos. The first features were deliberately ordinary: a landing page, sign-up, login and protected pages.

The interesting part was the boundary. The GPU’s event loop and the web app’s reactive state had to live together in one browser tab, so AGS got an interface that separates drawing from fetching data. A proof of concept drew two graphs with linked time ranges: scroll one, and the other follows. By the end of June, Kabuki could host AGS. What it lacked was real data.

Why the datastore lives with you

Poet runs next to the system it watches, receives measurements there and prepares exactly what the visualisation needs. You provide the compute. We take on the job of making that compute small.

From generated data to a real model

Poet started in July with a small in-memory store and a plain web interface, plus a generated data source to measure how fast it could receive and answer. The model grew in layers:

• element kinds describe the levels of a hierarchy, such as cluster, node, pod and container;
• elements have lifetimes, parents and metadata;
• metrics hold what a source, which we call a Muse, observed;
• relative metrics place each child continuously inside its parent, so the parts always add up to the whole on screen;
• haikus package the result for AGS.

Keeping the parts consistent with the whole is harder than it sounds. A child’s value moves every sibling after it and can reveal that the parent total does not match. Recomputing the whole tree for every update would not scale, so Poet does the work incrementally.

More than one Poet

A datastore that lives in your infrastructure has to survive it. Over the summer, Poets learned to find known peers, elect a leader, exchange heartbeats, share out ranges of identifiers and find elements held by another Poet. Tests ran two and three real Poet processes, killed leaders and checked that the survivors carried on.

Why it matters to you

By the end of September 2024, Poet was a tested model with a working coordination protocol, and a clear direction: the heavy lifting happens next to your data, and only the result travels. Today Poet is a native store for metrics, traces and logs that replicates between Poets and reports what replication costs it. The principle has not changed.

Next in the series: Your data takes the short way. Previous: Teaching metrics to take shape.