Summary
Did the sidecars cause outages? Two, both self-inflicted, and both
configuration rather than software. One was a DestinationRule ordering
mistake. The other was a PeerAuthentication scoped to the wrong namespace,
which enforced strict mTLS against a service whose caller was not yet
meshed. Neither was Envoy failing; both were us telling it something we did
not mean.
Impact
- Every request routed to an undefined subset failed, on the first outage.
The error names the destination service, so the alert fired on
payments, the on-call engineer paged the payments team, and the payments team correctly reported that they were fine. - A service whose caller was not yet meshed lost its callers, on the second, when strict mTLS arrived in a namespace it did not belong in.
- Before the mesh existed at all, the diagnosis cost was its own incident class: finding which hop was slow took hours, sometimes a day, because each team had instrumented their own service differently, or not at all.
Timeline
No clock for either outage; both were short, caught by their own blast radius, and reverted. What the record holds is the mechanism, and both mechanisms are deterministic enough to deserve the space:
- Outage one: a
VirtualService(“send 5% to subset v2”) applied before theDestinationRulethat defines whatv2means. Every request routed to the undefined subset fails immediately. - Outage two: a
PeerAuthenticationscoped to the wrong namespace, enforcing strict mTLS against a service whose caller was not yet meshed.
Root cause
Configuration expressed real intent poorly. The VirtualService and the
DestinationRule are two halves of one routing decision that Kubernetes
happily applies in either order, and only one order is correct. The
namespace scope on the PeerAuthentication was written once and read
everywhere it mattered.
Contributing factors
mTLS arrived by accumulation in reverse. The project started because an auditor asked how we encrypted traffic between services and the honest answer was that we mostly did not: everything inside the VPC talked over plain HTTP, roughly 15% encrypted before the rollout.
The migration had no owner for three months. The mesh went un-upgraded through two releases while it sat on the platform team’s list somewhere below everything with a deadline. A mesh with no owner becomes the thing every team blames when something is slow and nobody has the standing to change.
Retries at two layers. We already had retries and timeouts in application code. Adding them at the mesh layer meant that during one incident a request was being retried three times by the sidecar and twice by the caller, turning a slow dependency into a hammered one.
What changed
The full rollout is the case study: eleven weeks of PERMISSIVE mode,
flipped to STRICT only after a daily plain-text check had returned
nothing for five consecutive days. The incident-specific changes:
- Apply the
DestinationRulefirst, always. The ordering rule is now written into the rollout procedure, not carried in one person’s head. - Namespace scoping reviewed per policy, not inherited.
- Mesh-level retries removed; the code-level ones kept. One layer owns retry behaviour.
- Ownership named before the next migration, after three months proved what ownerless infrastructure decays into.
Did it hold
Flipping PERMISSIVE to STRICT across everything was a one-line change
and a non-event, because the telemetry had been checked first. Encrypted
service-to-service traffic went from roughly 15% to 100%. Time to identify
a slow hop went from hours, sometimes a day, to minutes, measured against
four incidents after the rollout. Seven canary-caught regressions in six
weeks never became incidents at all.
What I would do differently
Treat config ordering as part of the interface. “Two objects that only work in one order” is a design smell in any system, and in a mesh it is an outage with a two-week fuse.
The full rollout, the mTLS migration path, and the measured numbers are written up in Service Mesh Rollout: When It Is Worth It.
