Alerting

Deleting alerts is the maintenance nobody schedules

Removing a rule is attributable and keeping one costs everybody and nobody, so the rational individual keeps everything.

Teams applying this principle can also compare practical guidance on workforce analytics, keeping time and activity records separate from the judgement they are meant to inform.

Alert rules accumulate monotonically. Every incident produces at least one new rule, written by somebody determined that this will not happen again. Nothing in the ordinary run of work produces a rule's removal, so the count only ever climbs.

Three years of that leaves a system where nobody can say what most of the rules are for, and the article on fatigue describes what it costs.

Why nobody deletes anything

The reason is not carelessness, and framing it as diligence is why the problem survives every attempt to solve it with discipline.

Delete a rule, and if the thing it watched fails afterwards, the deletion is in the history with a name attached. Keep a rule, and the hundred wasted interruptions it causes over the next two years are spread across everyone, attributable to nobody, and never counted.

The individually rational choice is to keep everything, and every individual makes it. The countermeasure has to change that asymmetry rather than exhort people to be braver about it.

Make removal collective, scheduled and reversible

Three changes do most of the work.

Put the decision in a scheduled review with several people present, so that removal is a group act rather than an individual's exposure.

Bring the numbers from the article on fatigue. A rule that fired forty times and led to no action twice is not a judgement call, and the argument stops being about intuition.

Keep alert definitions in version control, so that deleting one is a revert away from being restored. Reversibility is what makes the decision cheap, and a cheap decision is one that actually gets taken.

Demote before deleting

The politically easiest and technically safest move is not deletion. Route the rule to a ticket queue or a dashboard for a quarter and see whether anybody misses it.

If it mattered, somebody notices within a month and it comes back with evidence. If it did not, it can be removed at the next review with no argument at all. Most rules never come back.

alert rules in the systemtwo clear-outs in four yearsevery incident adds one. nothing removes any.delete one, anda failure is traceable to youkeep one, anda hundred wasted nightsbelong to nobodythe individually rational choice is to keep everything
Figure 1Rules climb for four years with two clear-outs, and the asymmetry on the right explains why: one decision is traceable and the other is spread across everybody.

What to look for at the review

Rules that have not fired in a year. Either the condition never happens, in which case the rule earns nothing, or it fires and nobody noticed, which is worse.

Rules that fire constantly and lead to nothing, which are the ones doing active harm.

Causes duplicated by a symptom. If a user-facing alert already covers the impact, the six cause alerts underneath it are diagnostic material and belong on a dashboard.

Rules watching systems that no longer exist, which are more common than anyone expects, because decommissioning a service is a project and tidying its monitoring is nobody's task.

And rules whose owner cannot be named. An unowned rule has no one to ask about it, which means it will never be changed, only silenced.

The migration that mutes everything

There is one reliable way to lose an entire alerting configuration at once, and it is not a decision anybody makes. During a large migration, someone mutes broadly so that the expected noise does not drown the rotation. The migration takes three weeks longer than planned, the person who applied the mute moves on to the next thing, and nothing ever prompts anyone to reverse it.

Months later the system is quiet, everybody assumes it is covered, and the first evidence to the contrary arrives from a customer. The only practical defence is that broad mutes carry an expiry by construction, short enough to be inconvenient, so that renewing one is a conscious act somebody has to perform in daylight.

Silences are deletions performed badly

A silence with no expiry removes a rule from service without removing it from the system. It leaves the rule in the count, in the documentation and in the false impression that the condition is covered, while producing exactly nothing.

This is the worst of the available outcomes: the interruption is gone and so is the protection, and no review will ever surface it because the rule still exists and looks healthy. Expiring silences by default, and listing active ones somewhere visible, converts an invisible failure into a small recurring annoyance, which is the correct trade.

The same accumulation, on the other side

Workforce tools accumulate rules identically: flags for particular applications, alerts on activity patterns, categories assigned during onboarding to roles that have since changed. The asymmetry is the same one, with an additional weight, because the rule that nobody dares remove is producing records about people.

The review is the same review, and it is performed even less often.

What we cannot verify

The behavioural asymmetry described here is an argument from incentives supported by widespread practitioner experience rather than by a study we can point to. Figures on how many rules a typical system carries vary enormously and we quote none. Your own numbers, by rule name, are the only ones that matter and can be produced this afternoon from records you already hold.

The short version

  1. Rules accumulate monotonically; nothing in normal work removes any.
  2. Deleting one is attributable; keeping one costs everyone and nobody.
  3. Make removal collective, scheduled and revertible, and the decision gets taken.
  4. Demote to a ticket for a quarter and see whether anyone misses it.
  5. A rule whose owner cannot be named will never be changed, only silenced.
  6. A silence with no expiry removes the protection and leaves the appearance of it.

Further context

For a primary, standards or institutional reference, see the NIST Cybersecurity Framework.

Start from the class of problem, not the list of tools

Every selection here names the situation first and the criteria second. Product names come last, and each one carries the line describing what it costs you.