Mohammad Abu Mattar | Incidents

Two Outages Where We Told Envoy Something We Did Not Mean

Mohammad Abu Mattar — Mon, 26 Oct 2026 19:05:56 GMT

Summary

Did the sidecars cause outages? Two, both self-inflicted, and both configuration rather than software. One was a DestinationRule ordering mistake. The other was a PeerAuthentication scoped to the wrong namespace, which enforced strict mTLS against a service whose caller was not yet meshed. Neither was Envoy failing; both were us telling it something we did not mean.

Impact

Every request routed to an undefined subset failed, on the first outage. The error names the destination service, so the alert fired on payments, the on-call engineer paged the payments team, and the payments team correctly reported that they were fine.
A service whose caller was not yet meshed lost its callers, on the second, when strict mTLS arrived in a namespace it did not belong in.
Before the mesh existed at all, the diagnosis cost was its own incident class: finding which hop was slow took hours, sometimes a day, because each team had instrumented their own service differently, or not at all.

Timeline

No clock for either outage; both were short, caught by their own blast radius, and reverted. What the record holds is the mechanism, and both mechanisms are deterministic enough to deserve the space:

Outage one: a VirtualService ("send 5% to subset v2") applied before the DestinationRule that defines what v2 means. Every request routed to the undefined subset fails immediately.
Outage two: a PeerAuthentication scoped to the wrong namespace, enforcing strict mTLS against a service whose caller was not yet meshed.

Root cause

Configuration expressed real intent poorly. The VirtualService and the DestinationRule are two halves of one routing decision that Kubernetes happily applies in either order, and only one order is correct. The namespace scope on the PeerAuthentication was written once and read everywhere it mattered.

Contributing factors

mTLS arrived by accumulation in reverse. The project started because an auditor asked how we encrypted traffic between services and the honest answer was that we mostly did not: everything inside the VPC talked over plain HTTP, roughly 15% encrypted before the rollout.

The migration had no owner for three months. The mesh went un-upgraded through two releases while it sat on the platform team's list somewhere below everything with a deadline. A mesh with no owner becomes the thing every team blames when something is slow and nobody has the standing to change.

Retries at two layers. We already had retries and timeouts in application code. Adding them at the mesh layer meant that during one incident a request was being retried three times by the sidecar and twice by the caller, turning a slow dependency into a hammered one.

What changed

The full rollout is the case study: eleven weeks of PERMISSIVE mode, flipped to STRICT only after a daily plain-text check had returned nothing for five consecutive days. The incident-specific changes:

Apply the DestinationRule first, always. The ordering rule is now written into the rollout procedure, not carried in one person's head.
Namespace scoping reviewed per policy, not inherited.
Mesh-level retries removed; the code-level ones kept. One layer owns retry behaviour.
Ownership named before the next migration, after three months proved what ownerless infrastructure decays into.

Did it hold

Flipping PERMISSIVE to STRICT across everything was a one-line change and a non-event, because the telemetry had been checked first. Encrypted service-to-service traffic went from roughly 15% to 100%. Time to identify a slow hop went from hours, sometimes a day, to minutes, measured against four incidents after the rollout. Seven canary-caught regressions in six weeks never became incidents at all.

What I would do differently

Treat config ordering as part of the interface. "Two objects that only work in one order" is a design smell in any system, and in a mesh it is an outage with a two-week fuse.

The full rollout, the mTLS migration path, and the measured numbers are written up in Service Mesh Rollout: When It Is Worth It.

Three Weeks of "Sometimes Doesn't Load", While p50 Stayed Green

Mohammad Abu Mattar — Sun, 18 Oct 2026 16:25:54 GMT

Summary

The complaint that started this was not about speed. It was a support ticket saying the dashboard "sometimes doesn't load", from a customer whose account was roughly forty times larger than our median account. Our latency dashboard showed a p50 of 90ms and a green tick, so for about three weeks nobody believed them.

They were right, and our monitoring was lying by omission.

Impact

A real customer, on one of our largest accounts, intermittently unable to load their dashboard for about three weeks.
The p99 sat at 4.2 seconds against a p50 of 90ms. That 46x spread is not a uniformly slow system; it is a small class of requests doing something completely different from the rest.
On the account that complained, the largest-account p99 reached 9.8 seconds.
Large-account requests thrashed Aurora's buffer pool, so they slowed everyone else down too.

Timeline

No dated clock survives. The shape of the three weeks does:

A support ticket arrives: the dashboard "sometimes doesn't load".
The p50 dashboard shows green, and for about three weeks the ticket loses to the graph every time it is looked at.
Someone finally slices by percentile and account. The p99 is 4.2s, and on the complaining account the endpoint runs its join 214 times and takes just over four seconds.

From our own postmortem, three weeks in: "We spent the first sprint optimising the wrong thing, because the p50 said the system was fine and the p50 is what the dashboard showed by default."

Root cause

The offender was GET /api/v1/projects, which built each row from a nine-table join covering the project, its owner, its current status, its two most recent activity entries, and a permissions check. On an account with three projects it ran the join three times and returned in 60ms. On the account that complained it ran 214 times.

The join included a correlated subquery against the activity table, which had grown past 400 million rows, so its cost scaled with account age rather than account size.

Contributing factors

The dashboard defaulted to p50. The median of a bimodal distribution describes neither mode. Three weeks of disbelief came from one default.

Cache keys were built wrong before they were built right. The first caching implementation keyed on the incoming request, including a header the load balancer sets per request. Hit ratio zero, Redis full of single-use entries, and a 4% "improvement" that was noise. It took two weeks and a colleague asking why key cardinality exceeded the user count to catch it.

KEYS in production. The first invalidation implementation used KEYS project:*, which blocks Redis while it scans every key. It worked in staging with 400 keys and caused a 900ms stall in production with 2 million.

What changed

The full rearchitecture is the case study: CDN and Redis layers in front of the join, with measured before-and-after numbers. The incident-specific changes:

Percentile and account-size slicing in the dashboards, so a large-account tail cannot hide inside a green median again.
Key cardinality tracked against expected cardinality as a dashboard panel; it would have caught the cache-key bug on day one.
A hard rule from the incident review: no change could increase the blast radius of an Aurora failover.
No pattern-based deletion anywhere, SCAN-based invalidation from a centralised key map instead, because every early staleness bug was a write path that forgot one of the three keys it should have touched.

Did it hold

Measured four weeks before and after at comparable traffic: p99 4.2s to 1.6s, largest-account p99 9.8s to 2.1s, Aurora read IOPS down 71%. The caveat the review made us write down: caching moved the failure mode, it did not remove it. Two of three post-rollout incidents were correctness bugs, and neither triggered a latency alert, because both were fast.

What I would do differently

Put the tail on the dashboard before a customer has to put it in a ticket. The p50 default was not a monitoring decision; it was a default.

The full rearchitecture, including the stale-on-error design that turned an Aurora failover into slightly old data, is written up in A p99 Latency Caching Rearcitecture.

Three Dashboards Reported Zero Revenue, and Every Job Exited 0

Mohammad Abu Mattar — Wed, 07 Oct 2026 19:41:35 GMT

Summary

The failure mode that finally forced the pipeline rewrite was mundane. A vendor changed an SFTP filename. One job silently produced a zero-row file, six downstream jobs consumed that file happily, and by morning three internal dashboards were confidently reporting zero revenue.

Nothing alerted, because nothing had technically failed. Every job exited 0.

Impact

Three internal dashboards reporting zero revenue, believed by everyone who opened them, because the number looked like a number.
Nine overnight pages in the eight weeks before this incident class was retired, and four silent failures found only after the fact.
Median time to identify a failing job was about 35 minutes, and that was when someone already knew a job had failed. The first we usually heard of an overnight break was a customer asking why their report was empty.

Timeline

There is no incident clock. The record that survives is the shape of the night: the zero-row file was produced overnight, consumed by six downstream jobs, and discovered in the morning by people reading dashboards, not by any alert.

What is established about the surrounding regime, because it was every night, not one night:

jobs chained by sleep 900, scheduled by hope: job A at 02:00, job B at 02:15, because A "usually takes ten minutes", until A took eighteen and B read a half-written file, which had happened at least twice before
a failed job at 02:14 was simply not done; several were unsafe to run by hand twice, which nobody had documented, so re-running was a small act of faith
output went to /var/log/cron on whichever of 6 hosts had drawn the job, rotated weekly, with no record of normal runtimes
of the 41 jobs, we could name a current owner for 12

Root cause

Two causes, one on each side of the interface.

The vendor changed a filename without a contract that made that impossible to do quietly. And our jobs treated an empty input as success: a zero-row file was processed completely, exited 0, and passed the emptiness downstream, where six more jobs did the same.

Contributing factors

No retries existed. A failed job was missing until run by hand; the retry story after the migration took a week of chasing duplicate side effects to get right, because retry and idempotency have to be designed together.

Detection depended on customers. With no alerting, no runtime history, and no owners, the discovery path for a broken overnight job was a customer asking about an empty report.

Forty minutes to find a host. From the handover: "The thing I remember is spending forty minutes just working out which host a job ran on. Not fixing it. Finding it."

What changed

The full rewrite is the case study: 41 cron jobs across 6 hosts became 3 scheduled jobs over an SQS-driven event pipeline. The pieces this incident directly demanded:

Empty is a failure. A zero-row input trips an alarm instead of exiting 0.
Retries with idempotency keys, so a retry is the same work, not new work. The first key design was wrong (it included a timestamp) and produced duplicate PDFs and a very confusing morning; the fixed version later caught a resurrected cron entry on its own.
The alarm moved to DLQ depth, which cut overnight pages more than any reliability improvement did: 9 in 8 weeks before, 2 in 12 weeks after.

Did it hold

The two remaining pages were both genuine: a vendor served malformed data for two days, and the DLQ caught every affected message. A commented-out cron entry, uncommented by accident six weeks later, re-ran an already migrated job, and the idempotency layer absorbed the duplicate it was built for. Silent failures after the fact: four before, zero after.

What I would do differently

Delete rather than disable. The old cron entries left commented out "just in case" were a live footgun for six weeks, and one of them fired.

The full migration, including the visibility-timeout lesson and the measured before-and-after, is written up in From 41 Cron Jobs to an Event-Driven Pipeline.

The Admission Gate Denied an Image Built from a Fork

Mohammad Abu Mattar — Sun, 20 Sep 2026 00:00:00 GMT

Summary

The enforcement phase of a supply-chain hardening program denied 23 pods in its first six weeks. Most were rollout tail: twenty stale base images, two images built by hand on a developer laptop. The last one was a genuine incident.

An image in a production-bound pod had been built from a fork by a misconfigured pipeline in a sandbox account. Nothing about it was malicious. The admission check denied it because it could not prove the image was built by the pipeline, from the commit it claimed, which is exactly the question the gate exists to ask.

Under the previous setup, that image runs, and nobody finds out.

Impact

No user-facing impact, which is the point of catching it at admission rather than after. The real exposure was what the old system could not see:

builds ran on long-lived shared EC2 runners, where a job could read files another job left behind
registry credentials were static access keys stored as CI variables, rotated when someone remembered, and a compromised build in a low-value repo held the same credentials as everything else
deployments referenced mutable tags, so anyone with push access could change what prod meant on any cluster, with no review

Timeline

There is no clock, and there would not have been one under the old setup either. The detection method was the gate itself: the pod was denied at admission, the denial was logged, and the 20/2/1 breakdown of all 23 denials comes from those admission logs.

Root cause

The image came from a misconfigured pipeline in a sandbox account, building from a fork of the repository rather than the repository. Nothing in the old deployment path would have surfaced that: the manifest named a tag, the registry had the tag, and the cluster pulled it.

Contributing factors

Two builds of the same tag were not the same image. While investigating something unrelated, we found a 40MB difference between two builds of the same tag, a week apart. Dependencies resolved at build time from public registries by floating version range. An image with no stable identity cannot have a provenance story worth checking.

Nobody could answer the auditor's question. The program started from a fair question: can you prove the container running in production is the one your pipeline built from the commit you think it was? We could tell a plausible story. We could not produce evidence.

The blast radius of one credential was everything. Static, shared, rarely rotated: the fork-built image was benign, but the same trust model would have carried a hostile one just as quietly.

What changed

The full program is the case study: hermetic builds, signed provenance, digest pinning, and the admission gate. The pieces this incident proved out:

Admission enforcement with telemetry first. Two weeks of logging verdicts and denying nothing found 11 problems that would each have been a production incident on enforcement day.
Reproducible builds. Rebuilding the same commit later produced the same digest for 52 of 60 services; the eight exceptions embed build timestamps and are on the backlog.
No static registry credentials. Eleven before, zero after, measured by secret scan.
A documented two-approver bypass. Expected to be abused, used twice in six weeks, both times legitimately during an incident.

Did it hold

Six weeks of post-enforcement telemetry kept the measurement honest: images deployable without provenance went from all to zero, services deploying by mutable tag from 60 of 60 to 0 of 60, and "which commit is running" went from a twenty-minute archaeology exercise to one command. The break-glass procedure was rehearsed twice in game days and used zero times in anger.

What I would do differently

Trust an image you cannot reproduce. The 40MB divergence was found by accident while investigating something else; reproducibility should have been a standing check, not a discovery.

The full program, including the nine-week rollout and the measured results, is written up in Hardening a CI/CD Supply Chain to SLSA Level 3.

A Regional Control-Plane Failure Took a Payments API Offline

Mohammad Abu Mattar — Tue, 08 Sep 2026 00:00:00 GMT

Summary

A payments API that moves real money had run in a single AWS region for years. A regional control-plane incident took the whole service offline for a few hours. There was no second region to fail over to, so there was nothing to do but wait for the region to come back.

Recovery time was whatever the provider's recovery time turned out to be. That is the finding.

Impact

Total loss of service for the duration. A payments API fails worse than most things at the same downtime:

settlements stuck mid-flight rather than merely delayed
partners escalating against a system that could not answer them
an availability conversation to have afterwards, because the product is regulated

Timeline

There isn't one, and I am not going to invent it. The engineering record of the work that followed puts the outage at a few hours but never captured when we detected it or when it cleared.

What is established: the region's control plane degraded, the service had no second region to move to, and it stayed down until the region recovered. No mitigation existed on our side. A timeline of our actions would be a list of things we could not do.

Root cause

Two separate things, and only one of them was ever ours to fix.

The trigger was a control-plane incident in the provider's region. Rare, outside our control, and not something an architecture review would have prevented.

The cause was that we ran in one region with no failover target. When the only region is unavailable there is no operational response available. No runbook helps, because every path in it ends at the same unreachable place.

Contributing factors

Years of reliability had been reading as evidence of resilience. The service had been dependable in one region for a long time, and that record was doing the work of an argument that one region was enough. It was never that. It only meant the question had not come up yet.

Write paths were not idempotent. This did not cause the outage, but it shaped what a fix could look like. When a region gets shaky, clients, proxies and queues all start retrying at once. Without idempotency, that retry storm turns one payment into several, so any failover built on those write paths would have swapped an outage for double charges.

No recovery objective was written down. With no agreed RTO or RPO, there was no number a single-region design had to answer to.

What changed

The mandate was to survive the loss of an entire region without losing a committed payment or charging anyone twice. In order:

Idempotency keys in a globally replicated store, so a retry returns the original result instead of performing the operation again. This came first on purpose. The real problem was never "run in two regions", it was "make every money-touching operation safe to repeat, then run in two regions."
Active-active across two regions rather than a warm standby, so normal traffic exercises the failover path instead of an emergency being the first time anyone tries it.
Aurora Global Database with sub-second replication and a promotable writer in each region. It fit the EKS and Aurora PostgreSQL the team already ran, and data-residency rules ruled out a single global write master.
A written RTO and RPO, proven monthly, by deliberately failing a region out under production-like conditions.

On-call needed a failover they could trust at 3am without manual database promotion steps, because manual steps under pressure are how a recoverable incident becomes a data-loss incident. That constraint is why the failover is automatic rather than a runbook.

Did it hold

In the quarter after cutover the primary region degraded twice. Both times traffic moved to the healthy region inside the target window, with no customer-visible errors and no duplicate settlements.

Finance and risk watched one number, the double-charge count, and it stayed at zero. Failures kept happening. Every write path had become safe to repeat.

What I would do differently

Write the recovery objective down before you need it. The missing RTO was not a documentation gap. It was the reason nobody ever had to defend running in one region, and a number on paper forces that conversation while there is still time to have it.

Read a long clean record as the weak evidence it is. Years of uptime in one region tells you nothing about losing that region. It tells you the question has not been asked.

The full build, including the architecture, the idempotent request flow, the failover timeline and what broke during the game-days, is written up in Multi-Region Active-Active for a Payments API.