Cloud recording storage cleanup

Before and after

The problem

• Retention is 373 days: 90 days on local Ceph storage, then 283 days in the AWS bucket.
• At 373 days the platform hides the recording from the customer, but it never deletes the S3 object. The vendor's auto-expiry had been designed years earlier and never enabled for Beanfield.
• The platform's own catalogue was not purged either: a live read found 55,600 recording rows older than 373 days, with history back to January 2021.
• So the bucket grew without limit, at standard storage rates, with nobody choosing to keep 2021 recordings in 2026.

How I found it

1. 9 Jun 2026: I set the AWS cost line against what the platform should be storing. Storage alone could not explain the bill unless the bucket was never pruned. Then I checked the bucket: about 305 TB against roughly 57 TB at steady state.
2. 10 Jun: I pulled the 2025 finance actuals (AWS averaged about USD 10,500 a month, S3 storage USD 6,400 to 6,900 of it) and live catalogue reads, and collected the edge-server playback logs from all three nodes.
3. 10 Jun: I sent an evidence pack and an executive report to the senior director. They had the root cause, the ticket history back to 2020 and the savings options, each stated separately.
4. The ticket history showed the gap had been named by the vendor in 2020 and left open since.

The proof: nobody watches recordings past retention

Deleting customer recordings is irreversible, so "it is old" was not enough. I needed proof that customers were not, and could not be, playing them. I built it from two independent sources.

1. Edge playback logs (June)

• About 59,500 requests for AWS-archived recordings over about 14 days, from all three edge nodes.
• No request for a recording older than 368 days at the time it was requested. The newest AWS plays sat exactly at the local-storage boundary.
• Conclusion: the bucket only ever serves the 186 to 373 day slice. Everything older is never requested.

2. SmartCARE customer viewing history (October)

I built a report over every customer recording-library playback session SmartCARE holds: 98,633 sessions from 1,081 accounts, 4 Apr to 7 Oct 2026. For each session, the age of the recording is the play start minus the programme's air time. Test and vendor playback (153 sessions) was excluded.

Zero sessions and zero accounts past retention. The report ran on 7 Oct, and the next delete ran on 8 Oct.

The money

I kept each saving option separate rather than adding them up. Deleting past-retention objects was a pure cleanup with zero customer impact. Shortening retention was a policy decision, and I left it to management with the impact stated: about 1% of plays.

The vendor purge, June 2026

1. 15 Jun: after my pushes on the vendor ticket, the vendor started removing old recordings from AWS.
2. 15 to 17 Jun: 836,199 objects and 272.5 TB removed. By 19 Jun the bucket was down from about 336 TB to 63.3 TB.
3. 17 Jun: the vendor then started copying recordings older than 3 months back into the bucket. I pushed back that this was the wrong direction.
4. 7 Jul: the vendor confirmed auto-removal had never been enabled and proposed an S3 lifecycle rule.
5. 14 Jul: I wrote the lifecycle and retention up as a knowledge article. The senior director approved me running the deletes directly from then on.

My delete runs

The 8 Oct run, batch by batch: a 50-folder canary, three batches of 750 and a final 160. Every batch was verified byte-exact: the bucket's object count and size dropped by exactly what the list said, with zero survivors.

Safety design

• The cut-off has a margin. Policy is 373 days and the working cut-off is 375, counted from the air date, never the upload date. Nothing younger than 373 days is ever touched.
• Nine gates before any delete, including the cut-off floor, a clock check against the S3 server's own date, a listing that must equal the measured size exactly, and a strict folder-name pattern.
• Batches grow only after proof: canary 50, then 250, then 700 to 750. A token binds each approval to the exact list, so a changed list cannot ride on an old yes.
• A planner that never deletes. It builds the candidate list, totals and batch plan, then verifies each finished batch. The delete itself is a separate, approved step.
• A guard on the workstation blocks any bucket delete that lacks a dated approval and a change-register entry.
• No automated deletes. An automated delete job was built, run disarmed, and removed. Monitoring only reports.
• Learned the hard way, 22 Jul: the live bucket held 1,077 folders against a list of 1,664 entries. The per-batch verify caught it on the first attempt, which is exactly why it exists.

What came next

• 21 Oct: a "past retention over time" chart and a growth panel on the monitoring Ceph page. The bucket grows about 249 GB a day, so expired data builds up again unless it is deleted.
• 2 Nov: a briefing to the senior director on retention for new packages. About 5 TB was past retention again, heading for about 18 TB by year end if left alone.
• 2 Nov: management set 30-day retention for new recordings, with my viewing data as evidence.
• 4 Nov: a follow-up watched-band report on recordings aged 30 to 373 days: 18,718 recordings, 7.6% watched at all.

The four roles in this project

• Engineer: traced cost to cause across the finance sheet, the bucket, the catalogue and the edge logs.
• Designer: built a viewing-age report management could read in one table.
• System engineer: mapped the full recording lifecycle (local, offload, hide, never delete) and where each layer failed.
• Innovator: reused customer telemetry as deletion evidence, and built a delete method where every step proves itself.

Lessons

• Hiding is not deleting: check every layer a retention rule should reach.
• Irreversible actions need two independent proofs, not one.
• The size walk (3 to 4 minutes) is a fixed cost per batch and dominates the run time.
• Hard line: no bucket write or delete without a verified list and a clean dry run.

Stack