Four hundred milliseconds. Sixty-two per cent. Three hours of active time. None of those means anything, and no amount of staring at a dashboard will make them mean anything, until somebody can say what the same measurement looked like before.
A baseline is the distribution of normal for a specific thing, over a period long enough to contain its cycles. It is the reference that turns a reading into information, and it is the piece most often skipped, because collecting it takes time and a dashboard can be installed in an afternoon.
Long enough to contain the cycles
Almost everything worth measuring is periodic. Traffic has a daily shape and a weekly shape. Support volume spikes on Mondays. Finance systems have a month end. Retail has a season that dwarfs everything else. A workforce has a Friday.
A baseline taken across one week cannot describe a month-end, and anything it flags at month-end will be flagged wrongly. The baseline period has to be at least one full turn of the longest cycle that matters, which for most businesses is a year and for most practical purposes gets truncated to a quarter with the annual pattern noted separately.
The average is the wrong summary
Response times, task durations and most other measures of work are not symmetric. They have a floor, no ceiling, and a long tail, so the mean sits somewhere with very few actual observations near it and moves around when a handful of slow cases appear.
Percentiles describe the shape instead. The median says what a typical case looked like; the ninety-fifth and ninety-ninth say what the unlucky cases looked like, and the unlucky cases are usually the ones people complain about.
Two mistakes follow from this and both are common. The first is quoting a mean where a percentile was needed, which hides exactly the population that is suffering. The second is arithmetically worse: percentiles cannot be averaged. The mean of the ninety-fifth percentiles reported by ten servers is not the ninety-fifth percentile of the ten servers together, and treating it as one produces a number that is not any statistic of anything.
Tails matter more than they look
Work published by engineers at Google on what they called the tail at scale set out the reason plainly. When one user request fans out to many internal services and waits for all of them, the chance that at least one of those calls lands in its own slow tail rises quickly with the number of calls. A per-service ninety-ninth percentile that looks like a rare inconvenience becomes a common experience at the level the user actually occupies.
The source there is a paper by practitioners describing their own production systems, which is stronger than vendor material and weaker than an independent study, and it is worth reading rather than quoting.
Normal per what
A baseline belongs to a specific thing under specific conditions. Aggregate it across things that differ and it stops describing any of them.
This bites hardest in the workforce market. Activity distributions differ enormously between roles, and comparing an engineer's captured activity against a support agent's compares two unrelated jobs. Within a single role, variation between individuals is large and mostly not about effort. A baseline that averages a department produces a number against which most of the department is abnormal.
Baselines inherit whatever was happening
A baseline records the state of the world during the period it was taken, including the parts nobody wanted to keep. Take one during a crunch and overwork becomes normal. Take one while an unnoticed regression is already present and the regression becomes the reference. Take one during a quiet fortnight and the ordinary week that follows looks like an incident.
Two habits address it. Record what was happening during the baseline period alongside the numbers, so the reference can be judged later. And re-take it deliberately after any significant change, rather than letting it drift, because a baseline that updates itself continuously will quietly absorb a slow degradation and stop reporting it.
Measure before you change anything
This is the practical instruction that follows from all of the above, and it applies identically to a new alerting policy and to a monitoring rollout across a team. Whatever you intend to change, capture the distribution of the thing you care about before the change lands. Without it, the effect cannot be attributed, and every discussion afterwards is about impressions.
What we cannot verify
Automatic baselining and anomaly detection features are described by the companies that sell them, generally without stating the model, the window, or how seasonality is handled, and we reproduce none of their claims. The statistical points here are general rather than measurements of any system. Whether a particular baseline is long enough is a question about that system's cycles, answerable only by looking at its own history.
The short version
- A reading is uninterpretable until somebody can say what it looked like before.
- The baseline must span at least one full turn of the longest cycle that matters.
- Durations have long tails, so the mean describes almost nobody.
- Percentiles cannot be averaged across machines, and doing it produces a non-statistic.
- Aggregating a baseline across different roles or services describes none of them.
- Capture the distribution before the change, or the effect cannot be attributed.