A threshold is a sentence about a system, written at a particular moment, in a particular configuration, usually by somebody who had just been woken up. The system then changes and the sentence does not.
What moves underneath it
Traffic grows. Hardware is replaced with hardware that behaves differently. The mix of work changes because a new feature is popular. Deployments become more frequent. A dependency gets slower. The team gets smaller. Every one of those alters the relationship between the number and the situation it was supposed to describe, and none of them produce any notification that a threshold has stopped making sense.
Nothing announces that a threshold has expired. It does not fail; it starts being wrong in one of two directions, and both are quiet.
Percentages hide the thing you care about
Eighty per cent of a hundred-gigabyte disk leaves twenty gigabytes. Eighty per cent of a four-terabyte disk leaves eight hundred. The same threshold on the same graph describes two entirely different amounts of warning.
What anyone actually wants to know is how long there is before the thing runs out, and that depends on the rate of growth rather than on the current level. A disk filling at one gigabyte an hour with two hundred free is a comfortable Tuesday. The same disk filling at a hundred an hour is an emergency, and a percentage threshold cannot tell them apart.
The general form of the rule: where the concern is exhaustion, alert on time remaining, not on how much is used. It survives hardware changes, growth and reprovisioning, which no percentage does.
The ratchet only turns one way
This is the drift that matters most and it is entirely social.
A threshold fires at an inconvenient hour. Somebody investigates, finds nothing wrong, and raises it so it will not fire again. That is the correct decision in the moment and it is made under fatigue, at night, without a review.
Nobody ever performs the opposite operation. There is no equivalent moment at which a person is prompted to lower a threshold, because a threshold that is too loose produces silence, and silence prompts nothing. Over a few years the numbers migrate steadily upward, and the system that once alerted usefully now alerts only when the situation has already become obvious to everybody.
Dynamic thresholds move the problem rather than solving it
Deriving the threshold automatically from recent history removes the manual ratchet and introduces the failure described in the article on baselines: a reference that updates itself will absorb a slow degradation and call it normal.
A service that gets four per cent worse every week is never anomalous against last week. After a year it is unrecognisable and has never triggered anything. Automatic baselining is good at catching sudden changes and structurally blind to gradual ones, which is the opposite of what most people assume they are buying.
Seasonality causes the other half of the trouble. A model trained on recent weeks treats the first day of a seasonal peak as an anomaly and the last quiet day before it as normal, producing a burst of alerts at precisely the busiest moment of the year.
Write down why the number is the number
The single most useful practice here costs a line of text. Record, next to the threshold, what it was derived from: the observed rate of growth, the incident that prompted it, the capacity it assumes, the date.
Without that, the next person has a number and no way to judge it, so they will do the only safe thing available and leave it alone. With it, the threshold can be re-derived when the assumption changes, which is the whole of the maintenance.
The same ratchet, on the other side
Workforce tools have thresholds too: minutes of inactivity before a person is recorded as idle, the definition of a productive application, the level at which an alert is raised about somebody.
Each of them encodes an assumption about what the work looks like, and they are set once, usually by whoever configured the tool during onboarding, for a set of roles that has since changed. They ratchet in the same direction and for the same reason: a setting that produces complaints gets adjusted, and a setting that produces quiet does not.
What we cannot verify
The rates and capacities used above are illustrative. Anomaly detection behaviour differs between products, is described by their vendors, and the models are generally not disclosed in enough detail to predict how they handle seasonality; we reproduce no claims about them. Whether your own thresholds have drifted is answerable from your own change history, and the quickest test is to ask when each one was last edited and why.
The short version
- A threshold is a sentence about a system, written once, while the system keeps moving.
- Nothing announces that a threshold has expired; it just starts being quietly wrong.
- Where the concern is exhaustion, alert on time remaining rather than percentage used.
- Thresholds are raised at three in the morning and never lowered, so they migrate upward.
- Self-updating baselines absorb slow degradation and call it normal.
- Record what the number was derived from, or the next person can only leave it alone.