Category
3Incidents
Reliability
Two Outages Where We Told Envoy Something We Did Not Mean
- Sev2
- Reliability
- Kubernetes
- 21 Oct 2026
A service mesh caused two outages, both self-inflicted and both configuration rather than software: a VirtualService applied before its DestinationRule, and a PeerAuthentication scoped to the wrong namespace.
Three Dashboards Reported Zero Revenue, and Every Job Exited 0
- Sev2
- Overnight
- Reliability
- Data Engineering
- 07 Oct 2026
A vendor changed an SFTP filename. One cron job silently produced a zero-row file, six downstream jobs consumed it happily, and by morning three dashboards reported zero revenue. Nothing alerted, because nothing had technically failed.
A Regional Control-Plane Failure Took a Payments API Offline
- Sev1
- A few hours
- Reliability
- Architecture
- 08 Sept 2026
A money-movement API ran in one AWS region for years. A regional control-plane incident took it offline for hours with nowhere to fail over to. The fix was idempotency first, then a second region.