An alert arrives at the moment its recipient has the least context available: asleep, away from a desk, reading a phone screen in the dark, with no idea what has been happening for the last four hours.
Everything about what an alert should contain follows from that. The question is not what would be nice to include. It is what a person in that state needs in order to decide what to do in the next sixty seconds.
What the notification itself has to carry
What is broken, in the words a user would use. Not a metric name and a hostname, but the fact that people cannot check out, or that the search returns nothing.
How large it is, and which direction it is moving. Twelve per cent of users is a different night from every user, and rising is different from recovering.
Since when. The start time locates it against everything else that happened, which is how the next item becomes useful.
What changed recently. Most incidents follow a change, and putting the last deployment or configuration change into the alert saves the first ten minutes of every investigation. This single addition is the highest return available in alert design.
What is expected of the recipient, and what happens if they ignore it until morning. The article on justifying an alert argues that the second question decides urgency, and it belongs in the alert rather than in the recipient's head.
Link to the answer, not to the front page
A link to a dashboard's home page hands somebody a navigation problem. A link that opens the right dashboard, filtered to the right service and region, scoped to the window around the start time, hands them the picture.
The difference is a few minutes per incident and a great deal of error, because a tired person navigating a dashboard will look at the wrong time range at least once.
A runbook that says how to confirm, not how to fix
Runbooks decay because they are written as repair procedures, and repair procedures go stale with the system. Worse, any step mechanical enough to be written down precisely is a step that should have been automated, so a good runbook is constantly being eaten by scripts.
What does not go stale is the other material. How to confirm the condition is real rather than a monitoring artefact. How to establish who is affected. What mitigations exist and what each costs. When to escalate, and to which of the two ladders. What must not be done, which is often the most valuable line in the document.
Say how to silence it, and for how long
Somebody will want to stop the alert while working on the underlying problem, and if the only mechanism available is broad, they will use the broad one. Silences applied at three in the morning with no expiry are the most common way a working alert is permanently switched off, and nothing later reminds anyone.
Every silence should require a duration and a reason, and expiring silences should be visible somewhere. This is the maintenance the article on deleting alerts is about, and it starts here.
Fit the first screen
The first contact is usually a notification on a locked phone. If the essential facts are below the fold of that, they are not in the alert. Put the user-visible statement, the scale and the link in the first lines, and everything else behind them. A page that requires opening a laptop to understand has spent five minutes before anyone has done anything.
Make it findable afterwards
Each alert should carry a stable identifier for the rule that produced it. Six months later somebody will want to know how often this fired, whether it ever led to an action, and whether it should still exist. That analysis is trivial if the rule can be counted and impossible if every instance describes itself differently.
This is the same requirement as the measurement described in the article on fatigue, arriving from the other end: the numbers that make the alerting system reviewable are collected by the alerts themselves.
What we cannot verify
The claim that most incidents follow a change is widely reported by practitioners and rarely quantified in any way that transfers between organisations; we state it as common experience rather than as a measured fact, and it is checkable against your own incident record. Product capabilities for linking, silencing and templating differ and are described by their vendors. Nothing here has been tested by us.
The short version
- The alert arrives when its recipient has the least context available.
- State what is broken in the words a user would use, not a metric name.
- Recent changes in the alert save the first ten minutes of every investigation.
- Link to the filtered, time-scoped view rather than to a dashboard home page.
- A runbook should say how to confirm and what not to do, not how to repair.
- Silences need a duration and a reason, or a working alert dies quietly.