Retention: what you keep, for how long, and what that costs

By Readout editorial teamMeasurement6 min read

Retention looks like a storage question and is really a question about which investigations remain possible. What you keep decides what can be asked later, and the decision is usually made by whoever configured the default.

Resolution decays, and takes the short events with it

Nobody keeps everything at full resolution forever. The standard arrangement collapses old data into coarser buckets: fine detail for a few days, minute averages for a month, wider averages for a year.

That trade is sensible and it has a specific cost. A saturation lasting thirty seconds is plainly visible in ten-second samples and disappears entirely into a five-minute average. Once the rollup has happened, no query recovers it. The question "was there a spike at that moment" stops being answerable, not because the system failed to record it, but because the recording was later summarised.

So the retention policy is a list of the questions the organisation has decided it will not be able to ask about anything older than a given age. Written that way it usually gets more attention.

The bill is not made of bytes

Raw storage is cheap and is rarely the driver. What costs money is making the data searchable: indexes, cardinality, the compute that runs queries, and in most commercial pricing models the volume ingested rather than the volume kept.

This is why logs dominate a monitoring bill while metrics rarely do. Metrics are small, regular and compress well. Logs are unbounded text that somebody wants to search by arbitrary fields, and every field made searchable multiplies the index.

Tiering exists for exactly this: keep recent data hot and searchable, push older data to cheap storage where retrieval is slow and search is limited. It works, and it means the answer to "can we look at last March" becomes "yes, tomorrow", which is a different answer from "yes".

Ask what the old data is for

Three purposes cover most of it, and each wants a different shape.

Capacity and trend work needs a long span at low resolution. Incident investigation needs high resolution and almost never reaches back beyond a few weeks. Audit and dispute need the original records unaggregated, for whatever period the obligation names, and cannot use a rollup at all.

A single retention setting serves none of them well. Separate policies per purpose cost a little configuration and save both money and the argument that happens when somebody needs March.

Retention decides what a baseline can be

The article on baselines argues that a reference period must span at least one full turn of the longest cycle that matters, which for most organisations is a year. A system keeping thirty days cannot produce that reference, and no amount of statistical work recovers it.

This is worth checking before the first monitoring bill arrives, because the cheap plan and the useful baseline are frequently incompatible, and discovering it eleven months later costs another year.

Kept data is a liability as well as an asset

Everything retained can be breached, subpoenaed, exported by a departing employee, or read by somebody who joins the company next year and has the same access as everybody else on the team.

That is uncomfortable for infrastructure telemetry and serious for workforce monitoring, where the retained material is personal data about identifiable people. Data protection regimes of the European kind impose storage limitation: personal data is kept no longer than necessary for the purpose it was collected for, and "it might be useful" is not a purpose. Requirements differ by jurisdiction and nothing here is legal advice, but the direction is consistent, and the article on retention and access in the workforce section goes through it.

The two pressures point opposite ways. Defensible records argue for keeping; storage limitation and breach exposure argue for deleting. They are reconciled by separating the categories rather than by picking a single number: the record needed to settle an invoice is not the screenshot archive, and they do not need the same lifetime.

Deletion has to be real

A retention policy that only governs the primary store is a policy about one copy. Backups, replicas, exports somebody scheduled, a warehouse that was fed for analytics, and a spreadsheet a manager downloaded all outlive it.

The check worth running is not whether the policy exists but whether anyone has traced where the data goes. Every destination inherits the obligation, and the ones nobody remembers are the ones that appear in a breach notification.

What we cannot verify

Pricing models differ between vendors, change frequently, and are described by the vendors themselves; we reproduce no rates here and every figure elsewhere on this site carries the date it was read. The rollup intervals described are conventional rather than standard. Legal obligations vary by jurisdiction and by sector, and the applicable rule is the one that applies to your organisation, established with advice.