A threshold is a classifier. Every reading it sees is sorted into "worth telling someone" and "not worth telling someone", and like every classifier it is wrong in two different directions with two different costs.
It raises alarms when nothing is wrong, and it stays silent when something is. Moving the threshold trades one for the other. It does not reduce both, and no amount of tuning changes that, because the trade is a property of where the two distributions overlap rather than of the number you picked.
Only a better signal reduces both kinds of error. That is the sentence worth carrying out of this article, because the usual response to a noisy alert is to move the threshold, and the usual result is a quieter system that misses more.
Rare events defeat accurate detectors
Here is the arithmetic that surprises people, and it is ordinary conditional probability rather than anything exotic.
Suppose real problems are rare: one interval in a thousand. Suppose a detector is right ninety-nine per cent of the time in both directions. Over a thousand intervals it catches the one real problem, and it also raises an alarm on about one per cent of the nine hundred and ninety-nine quiet ones, which is roughly ten false alarms.
So a detector that is right ninety-nine times out of a hundred produces alerts that are wrong about ten times out of eleven. Nothing is broken. The detector performs exactly as specified. The base rate is doing the damage, and this is why accuracy is close to useless as a description of whether an alert deserves to wake anyone.
The consequence is directional and worth stating: the rarer the event, the better the detector has to be before it is worth acting on, and beyond a certain rarity no threshold on a single signal is good enough.
The two errors do not cost the same
Choosing a threshold means choosing a ratio between two costs, whether or not anyone says so out loud.
A missed outage costs revenue, reputation and possibly a contractual penalty. A false alarm at four in the morning costs an hour of sleep, and the twentieth one costs the credibility of the whole alerting system, which is a much larger and much slower bill.
In the workforce market the asymmetry is not merely financial. A false positive on an idle-time threshold is a person wrongly recorded as not working, and the record persists in a system a manager will read later. The cost of that error is borne by someone who did not set the threshold and usually does not know it exists.
The useful question is therefore never "what number is correct". It is "how many false alarms is one catch worth here", and the answer differs between a payment system and a batch report, and between a security control and a productivity dashboard.
Crossing is not the same as being over
A measurement that hovers near a threshold crosses it repeatedly, and a naive rule fires every time. The result is a burst of alerts that describe one situation and read as several.
Two mechanisms fix it, and both are ordinary. Require the condition to persist for a stated duration before it counts, which converts a spike into a state. And separate the level at which an alert clears from the level at which it fires, so that a reading has to travel a distance before the system changes its mind. Without that separation, anything sitting exactly on the line generates alternating alerts and recoveries indefinitely.
Static numbers age
A threshold set once encodes assumptions about traffic, hardware, team size and workload at the moment it was written. All four move. A disk warning at eighty per cent was reasonable when the disk was small and the growth was slow; on a larger disk with faster growth the same percentage may leave less warning than it takes to order more capacity.
The failure is slow and produces no error message. Thresholds do not announce that they have stopped being sensible, so the review has to be scheduled rather than triggered, which is the subject of the article on deleting alerts.
What a noisy threshold does downstream
False alarms are not free even when they are dismissed quickly, because dismissal is a learned behaviour that generalises. A system that is wrong ten times out of eleven trains its recipients to close alerts without reading them, and it trains them on the real one too. That mechanism has its own article in the alerting section.
What we cannot verify
The probabilities used above are illustrative, chosen to show the mechanism rather than taken from any measured system, and they are labelled as such. Real base rates differ by system and are rarely measured at all, which is itself part of the problem. Claims that a particular product reduces false positives come from the companies selling it, without a stated base rate or cost ratio, and we reproduce none of them.
The short version
- A threshold is a classifier and is wrong in two directions at once.
- Moving it trades misses for false alarms; only a better signal reduces both.
- When events are rare, an accurate detector still produces mostly false alarms.
- Choosing a threshold means choosing a ratio between two unequal costs.
- In workforce tools the false positive is borne by someone who never saw the setting.
- Require duration, and separate the clear level from the fire level.