---
title: "reliability"
lang: "en"
author: "Mohammad Abu Mattar"
canonical: https://mkabumattar.com/incidents/categories/reliability
---

[Home](/)›[All Incidents](/incidents)›[All Categories](/incidents/categories)›[Reliability](/incidents/categories/reliability)

Category

3Incidents

# Reliability

CategoryAllReliability (3)Architecture (1)CI/CD (1)Data Engineering (1)Kubernetes (1)Observability (1)Performance (1)Security (1)TagAllPostmortem (5)Admission Control (1)AWS (1)Caching (1)Configuration (1)Cron (1)Data Pipeline (1)Event Driven (1)Failover (1)Idempotency (1)Istio (1)Kubernetes (1)Latency (1)Monitoring (1)MTLS (1)Multi Region (1)Payments (1)PostgreSQL (1)Provenance (1)Redis (1)Service Mesh (1)Silent Failure (1)SLSA (1)SQS (1)Supply Chain (1)SortNewest firstOldest firstA to Z

## [Two Outages Where We Told Envoy Something We Did Not Mean](/incidents/post/mesh-config-ordering-outages)

-   Sev2
-   [Reliability](/incidents/categories/reliability)
-   [Kubernetes](/incidents/categories/kubernetes)
-   21 Oct 2026

A service mesh caused two outages, both self-inflicted and both configuration rather than software: a VirtualService applied before its DestinationRule, and a PeerAuthentication scoped to the wrong namespace.

[#Postmortem](/incidents/tags/postmortem)[#Service Mesh](/incidents/tags/service-mesh)[#Istio](/incidents/tags/istio)+2 tags

[Read Postmortem](/incidents/post/mesh-config-ordering-outages)

## [Three Dashboards Reported Zero Revenue, and Every Job Exited 0](/incidents/post/cron-silent-zero-revenue-failure)

-   Sev2
-   Overnight
-   [Reliability](/incidents/categories/reliability)
-   [Data Engineering](/incidents/categories/data-engineering)
-   07 Oct 2026

A vendor changed an SFTP filename. One cron job silently produced a zero-row file, six downstream jobs consumed it happily, and by morning three dashboards reported zero revenue. Nothing alerted, because nothing had technically failed.

[#Postmortem](/incidents/tags/postmortem)[#Cron](/incidents/tags/cron)[#SQS](/incidents/tags/sqs)+3 tags

[Read Postmortem](/incidents/post/cron-silent-zero-revenue-failure)

## [A Regional Control-Plane Failure Took a Payments API Offline](/incidents/post/single-region-payments-api-regional-outage)

-   Sev1
-   A few hours
-   [Reliability](/incidents/categories/reliability)
-   [Architecture](/incidents/categories/architecture)
-   08 Sept 2026

A money-movement API ran in one AWS region for years. A regional control-plane incident took it offline for hours with nowhere to fail over to. The fix was idempotency first, then a second region.

[#Postmortem](/incidents/tags/postmortem)[#AWS](/incidents/tags/aws)[#Multi Region](/incidents/tags/multi-region)+3 tags

[Read Postmortem](/incidents/post/single-region-payments-api-regional-outage)