Platform tooling and reliability

Provisioning repair tool

Problem. A customer buys TV in billing, but the platform never gets a working account because billing's own call failed silently: an "account not found" error, or an HTTP 200 that changed nothing. Repairs were manual.

1. Read billing's own calls in the web-server log and counted them: 1,031 subscribe calls, 163 account updates, 119 terminal adds, with parameters and return codes documented.
2. Built a 339-line repair tool with sub-commands to trail, inspect, plan, create from the log, subscribe, activate and rebind.
3. Every write is a dry run until confirmed, refused without an approval stamp, and prints a before and after diff of the account's services, so a silent 200 is caught.
4. Calls go through the same front door billing uses, so repairs land in the same log; customer sign-in details found in logs are never printed.
5. Wired to a 5-minute alert and the BOSS Provisioning page in the monitoring GUI; every write and its reversal is logged.

Read-only gateways to live systems

Four small servers that let tools and scripts query production safely, each read-only by construction:

Security tests cover redirects, filter operators, parameter case, error shapes and preview overrides; diagnostics never include request parameters. About 6,273 lines including tests, plus command-line tools for tickets, cases and routing.

Storage cluster (Ceph)

• Fixed event-log clocks on 6 of 7 storage nodes and raised the sync from daily to every 30 minutes.
• Repaired inconsistent placement groups with before and after state recorded.
• 14 Oct 2026 outage: a newly joined disk took IPTV down. I took it out of service: inactive placement groups went from 74 to 3 in 30 seconds, all 6,568 active within minutes, edge storage answering in 2 to 5 ms. Proved the cause from firewall drop counters on seven hosts (549 to 1,097 drops each): the join had skipped the runbook.
• Wrote the production-join runbook (weight 0, no-in, peer firewall, night window) and built the Ceph guard into the monitoring platform.

Platform baseline

Golden traces of the platform's six flows, a restore reality map, an outage scenarios and recovery playbook, a known-defects register, and a live service check every 30 minutes across eight service lines, in shadow mode first.

Set-top-box log platform (ELK)

Rebuilt and extended the box log cluster: retention to 60 days, dropped log lines from 18.08% (30 Jul) to about 4% (23 Sep), node disks grown, an allow-list for noisy sources. These logs feed the fleet-stall detector and every freeze diagnosis.

Channels and captions

• Rebuilt the channel reference from live recorder config: 194 channels, 184 active, 10 paused, 23 deviations documented; deleted the stale matrix instead of patching it.
• Probed all 193 channel sources and fixed 23 mislabelled caption tracks, restoring French captions on Quebec channels.
• Found why the guide image volume kept filling: images are re-fetchable from 67,190 CDN URLs and storage hard links double-count space; proposed the reclaim process.

More builds