An alert differs from a dashboard, a log line and a ticket in exactly one respect: it interrupts a person. Everything else about it, the threshold, the query, the routing, follows from whether that interruption is justified.
So the definition worth holding is behavioural rather than technical. An alert is a request for a human being to stop what they are doing. Written that way, most of the alerts in most systems do not qualify, and the ones that survive are considerably more useful.
Three questions, and any no disqualifies
Is a human needed? If the system can recover by itself, or already has, the event belongs in a record rather than in somebody's evening. A disk that is cleaned automatically at ninety per cent does not need to announce eighty.
Is a human needed now? Plenty of real problems can wait until Tuesday. Certificate expiring in three weeks is a ticket. Certificate expired is a page. The same condition at two distances produces two different destinations, and conflating them is what turns a rotation into attrition.
Is there something a human can do? An alert that reports an upstream provider's outage, with no failover available and no action possible, informs somebody that they are powerless at three in the morning. That is worth knowing during business hours and worth nothing at all in the night.
Alert on what the user experiences
The practice literature published by engineers who run large systems is consistent on this, and the reasoning follows the argument in the first section of this site. Page on symptoms, which are close proxies for user harm. Diagnose with causes, which sit further away.
Processor saturation is a cause. It may accompany a problem, and it routinely occurs without one, so paging on it means waking somebody for a condition that is frequently fine. Requests failing is a symptom. It means somebody is not getting what they came for, regardless of which of the forty possible causes produced it.
The practical consequence is a small number of symptom alerts and a large number of cause dashboards, which is the opposite of the arrangement most systems accumulate.
What an alert has to be able to say
Four things, and if it cannot state them it is not ready to wake anyone. What is broken from the point of view of somebody using the system. How many of them are affected. What the recipient is expected to do. And what happens if it is ignored until morning.
That last question is the useful filter, because answering it honestly converts a great many pages into tickets. If the answer is "nothing much", the alert has been asking for an interruption it did not need.
Rate of consumption, rather than instantaneous state
A threshold on a current value fires on a momentary excursion and misses a slow degradation that is doing far more damage over a day. The alternative used in objective-based practice is to alert on how fast an allowance for failure is being consumed relative to the period it has to last.
A brief spike consumes a negligible share and does not fire. A modest elevation sustained for six hours consumes a large share and does. This is closer to the question anyone actually cares about, which is whether the service will fail its commitment, and it produces markedly fewer interruptions for the same coverage.
Every alert needs an owner and an expiry
Alerts are written during incidents, by people who are tired, to prevent a recurrence of something specific. They then persist, outliving the system they described, the person who wrote them and the reason.
Two record-keeping habits fix most of it. Each alert names an owner, so that a question about it has a destination. And each has a review date, because nothing about an alert announces that it has stopped being relevant, and the article on deleting alerts is about the maintenance that follows.
Count the cost, because it is not zero
A page at three in the morning costs the rest of that night, a degraded day after it, and a share of the recipient's willingness to carry a pager at all. Ten false ones cost the credibility of every alert in the system, which is the mechanism the article on alert fatigue covers.
Against that, the cost of a missed problem. The article on thresholds argues that choosing where to set one is choosing a ratio between those two costs. That choice is made whether or not anyone states it, and stating it is the whole of alert design.
What we cannot verify
The symptom-first practice described here comes from published accounts by engineers at large operators, describing their own systems. That is a strong source and not a controlled study, and their scale differs from most readers'. Claims that particular products reduce alert volume come from those products' vendors. We have measured nothing, and the honest test is to count your own pages for a month and mark how many resulted in an action.
The short version
- An alert is a request for a human to stop what they are doing.
- Is a human needed, needed now, and able to act; any no disqualifies it.
- Page on symptoms, which are close to the user; diagnose with causes, which are not.
- If nothing much happens when it is ignored until morning, it was a ticket.
- Alerting on how fast the failure allowance is consumed beats instantaneous thresholds.
- Every alert needs a named owner and a review date, because none expire by themselves.