Support hears about it first. The graphs are normal. By the time anything monitored moves, the complaints have been arriving for forty minutes and somebody senior is asking why nobody knew.
This is not a failure of the tooling to work. It is the tooling working correctly on the wrong questions, and the gap between the first complaint and the first alert is the thing to attack.
Why the graphs are late
Four reasons account for nearly all of it, and they are diagnosable without buying anything.
The measurement is server-side and the problem is not. Everything after the response leaves is invisible: the client, the network, a third-party script, a device class. The article on synthetic and real user data covers the split.
Averaging hides the affected group. If one region, one browser or one customer is failing, a global rate barely moves. The article on baselines argues for percentiles; this is the case where segmentation matters more than percentiles do.
The alert is on a cause, not a symptom. Resource saturation and error counts move when a failure is infrastructural. A service returning quick, cheerful, wrong answers moves none of them.
The threshold is calibrated for total failure. Twelve per cent of users failing is an emergency to those users and a rounding error against a threshold written for an outage.
The cheapest instrument is the complaints themselves
Support tickets and their timestamps are already collected, and nearly nobody feeds them back into monitoring. A simple count of incoming contacts per hour, compared against its own baseline, is a genuine detector: it is slow, it is noisy, and it fires on the failures that every technical measure missed, because it observes the only population that matters.
The objective is the fix, not another dashboard
Every one of the four causes above has the same underlying shape: the thing being measured is not what the user gets. The durable correction is to state an objective in terms of the user, and to alert on that rather than on its components.
Requests that returned the right answer in an acceptable time, as a proportion of requests attempted, for a defined population. That single statement is measurable, it moves when a region fails, it moves when answers are wrong, and it does not move when a processor is briefly busy.
Getting it agreed is harder than building it, because it requires somebody to say what acceptable means, and the argument about that number is the useful part of the exercise rather than an obstacle to it.
Real user measurement in the client
- Best forAnyone whose users are on the far side of a browser or an app
- PricingCommonly bundled into a larger platform; standalone products exist
- StandoutSees the part of the experience that happens after your server is finished
- Watch out forBlind to users who never loaded enough to report, which are the worst-affected
Segmented objectives rather than global rates
- Best forServices with distinguishable populations: regions, plans, customers
- PricingA configuration change in most metrics products, not a purchase
- StandoutCatches a failure confined to a group, which a global rate cannot show
- Watch out forMultiplies the number of alerting rules, and the article on deleting alerts applies
Correctness checks, not availability checks
- Best forAnything that can fail while still returning a response
- PricingTrivial to build; a few lines in whatever runs your synthetic checks
- StandoutCatches the cheerful error page, which status codes never will
- Watch out forSomebody has to maintain the expected answer as the product changes
Support volume as a signal
- Best forEvery organisation, and it is already being collected
- PricingFree; the work is connecting two systems that do not usually meet
- StandoutObserves the only population that counts, and fires when everything technical missed it
- Watch out forSlow, noisy, and dependent on people bothering to complain, which most do not
Close the loop deliberately
After each of these incidents, the useful question is not what broke. It is which measurement would have moved, and whether it exists. That question turns a recurring embarrassment into a short list, and the list is usually three items long rather than a programme.
The article on incident review argues for few actions rather than many. This is the one worth privileging, because it is the only kind of finding that shortens the next gap.
What we cannot verify
Detection latency depends on traffic, on the failure and on the population affected, and no general figure would transfer. Product claims about catching problems earlier come from their vendors. Your own gap is measurable from your own records: take the last five incidents, compare the first complaint against the first alert, and the distribution of that difference is the number this article is about.
The short version
- The tooling is working correctly on the wrong questions.
- Server-side measurement is blind to everything after the response leaves.
- A global rate barely moves when one region or one customer is failing.
- Thresholds calibrated for total failure ignore the twelve per cent who are down.
- Support volume is already collected and is a genuine detector nobody connects.
- After each incident, ask which measurement would have moved, and whether it exists.