Metrics, logs and traces answer different questions

By Readout editorial teamInfrastructure6 min read

Three shapes of telemetry dominate, they are frequently sold as three products, and they are better chosen by starting from the question you cannot currently answer.

What each shape is good at

Metrics are numbers aggregated over time intervals. They are small, regular, cheap to store and fast to query across long spans, which makes them the right basis for alerting and for trends. What they cannot do is explain a single case: a metric knows that the ninety-ninth percentile rose and has no idea which request that was.

Logs are discrete events with detail attached. They answer what exactly happened, in order, with the specifics. The cost is that making arbitrary text searchable is expensive, and the index rather than the storage is what appears on the bill.

Traces record one operation as it moves across services, with the causal relationships preserved. They answer where the time went and which component in a chain was responsible, which neither of the others can do once a request touches more than one service.

Start from the question

Is something wrong, and how much? Metrics. What exactly happened to this order at eleven o'clock? Logs. This request took four seconds, which of the eleven services it touched spent them? Traces.

Organisations frequently buy all three and answer one question three times, which is expensive and slow. The useful audit is to list the questions actually asked during the last five incidents and see which shape was needed for each.

The boundaries are softer than the diagram

Logs with structured fields can be aggregated into metrics, and doing so is often the fastest way to get a measurement out of an application nobody wants to modify. It works, and it pays log prices for metric answers, which is fine at low volume and ruinous at high.

Traces are made of events and can be sampled into metrics. Metrics can carry a pointer to an example trace, which is the practical trick that connects a spike on a graph to a single slow request without keeping everything.

Sampling is where traces become affordable

Recording every operation across every service is usually unaffordable, so most of them are discarded. The decision of which to keep is made either at the start, before anything is known about the request, or at the end, once its outcome is visible.

Deciding at the start is cheap and keeps a random selection, which means the interesting slow ones are kept only in proportion to how often they occur. Deciding at the end can keep every error and every slow request, which is what you actually want, at the cost of holding everything in memory until the operation finishes.

This choice determines whether traces are useful during an incident, and it is usually made by a default nobody examined.

Cardinality is where metrics stop being cheap

The claim that metrics are the cheap shape holds only while the number of distinct series stays bounded. Every label attached to a measurement multiplies the series count by the number of values that label can take, and one label carrying a customer identifier or a request path turns a handful of series into hundreds of thousands.

At that point metrics acquire the cost profile of logs while keeping the poor explanatory power of metrics, which is the worst available combination. The next article in this section is about that number specifically, because it is the single most common reason a monitoring bill becomes indefensible.

Instrumentation and backend are separable, and should be

The largest lock-in lever in this market is that instrumentation has historically been vendor-specific: code annotated with one company's library only speaks to that company's product, so changing supplier means changing the application.

A vendor-neutral standard for producing telemetry now exists, governed in the open by a foundation rather than by a supplier, and it decouples the two. Instrument once, choose a backend, change the backend later without touching the code. Whether a product accepts that format is a procurement question worth more than most feature comparisons, because it is the difference between a decision that can be revisited and one that cannot.

The three-shape framing is a taxonomy, not a design

The habit of calling these pillars comes largely from vendor marketing, and it quietly suggests that a complete setup requires all three at full coverage. It does not. It suggests three budgets, three products and three integrations, and treating a taxonomy of data shapes as an architecture is how a monitoring bill arrives at a number nobody can defend.

Other shapes exist and matter: profiles from running processes, discrete business events, and in some systems the contents of a queue. The question remains what it was: which question cannot be answered today.

What we cannot verify

Cost comparisons between the three shapes depend entirely on volume, cardinality, retention and the pricing model of a specific vendor, and the vendors publish those models themselves. We quote none. The relative strengths described here are structural properties of the data shapes rather than measurements of any product, and any specific claim that one tool ingests or queries faster than another comes from whoever sells it.