Every system at scale throws data away. The only question is whether the discarding was designed or happened by default, and the difference shows up on the day somebody needs a record that no longer exists.
Two purposes that want opposite rules
Sampling for measurement keeps enough records to estimate a population. If ninety-nine per cent of successful requests are discarded and the remainder is scaled correctly, the estimated error rate, the estimated throughput and the estimated median all survive, because those are properties of a distribution and a random subset preserves them.
Sampling for investigation keeps particular records because somebody will want to look at them individually. Here a random subset is close to useless: the case that will be asked about is the unusual one, and a one per cent lottery keeps it one per cent of the time.
Most arguments about sampling are two people optimising for different purposes without saying which. The system needs both, and it needs them configured separately.
Deciding at the start, or at the end
A decision taken when an operation begins is cheap: hash an identifier, keep or drop, propagate that decision to everything downstream so the record stays whole. Nothing has to be held in memory. What it cannot do is prefer interesting operations, because at the start nothing is known about how the operation will turn out.
A decision taken when the operation finishes can keep every error and every slow case and discard the ordinary successes. That is exactly the selection anyone wants during an incident. The cost is that the system must buffer every operation until it completes, which is a real architectural burden and the reason this option carries a price.
The correctness trap in weighted sampling
Once records are kept at different rates, the surviving set no longer resembles the population. Errors are over-represented by construction. Computing an error rate from that set produces a number that is wildly wrong and looks entirely plausible.
The fix is to carry the sampling rate with each retained record and weight by it when aggregating. Systems that do this are correct; systems that keep errors preferentially and then compute rates from what remains are producing fiction. It is worth asking a vendor which of the two their product does, because the answer is rarely on the feature list.
Keep a whole thing or none of it
A trace sampled independently at each service produces fragments: some spans kept, others discarded, and a picture with holes in exactly the places that matter. The decision has to be made once and honoured by every participant, which is why sampling decisions travel with the request rather than being taken locally.
The same applies to logs written by several components about one event. Sampling them independently produces a record where the cause was discarded and the consequence was kept.
What must never be sampled
Records kept for audit, for billing, or to satisfy an obligation are not telemetry and are not subject to a sampling budget. A system that applies one policy across everything will eventually discard something that had to be retained, and the discovery is made by an auditor rather than by an engineer.
Separating those streams at the point of collection is the only arrangement that survives a later cost reduction, because the cost reduction will be applied by somebody who was not in this conversation.
Screenshots are sampling, and nobody calls them that
On the workforce side of this site the same mathematics applies without the vocabulary. A tool capturing a screenshot every ten minutes is sampling the working day at roughly one part in sixty, and every conclusion drawn from it is a statistical claim about a population from a very small sample, made without anyone saying so.
Two consequences follow directly from the sections above. The sample is uniform, so it is systematically bad at catching the rare event, which is usually what somebody is looking for. And the interval is known to the person being sampled, which makes it exactly the kind of measure the article on targets describes.
What we cannot verify
Sampling rates, buffering limits and the correctness of weighting are implementation details of specific products, described by their vendors, and we have tested none of them. The proportions used above are illustrative. Whether a given setup preserves aggregate accuracy is answerable by comparing its reported rates against an unsampled source for a period, which is a test worth running once and almost nobody runs.
The short version
- Sampling for measurement and sampling for investigation want opposite rules.
- A one per cent lottery keeps the case you will be asked about one per cent of the time.
- Deciding at the end can keep every error; it costs buffering until the operation finishes.
- Keep the sampling rate with the record, or aggregates computed from it are fiction.
- Sample a whole trace or none of it, or the cause is discarded and the consequence kept.
- A screenshot every ten minutes is a one-in-sixty sample nobody describes as one.