Postmortem the Near Miss

A quarter with zero customer outages feels like a win. In practice, a green dashboard often measures how many times luck stepped in before the last barrier fell.

We call postmortems when a database dies, payments halt, or customer data leaks. The smoke is out in the open. Leaders want answers. The team pays its retrospective tax.

Now take an unindexed database migration. It locks the primary table for forty seconds at three on Sunday morning. The on-call engineer sees no alerts. They close the laptop. They post a relief GIF in Slack and move to the next sprint ticket.

That asymmetry degrades your engineering culture. Unexamined near misses do not leave an organization safe: they train teams to become more reckless.

What the research shows

Decision science explains why close calls distort human judgment.

In lab experiments, Robin Dillon and Catherine Tinsley (2008, 2012) tested how people handle close calls. When subjects saw near misses without review, they took bigger risks than subjects with no prior data.

Cognitive optimism drives the shift. When a risky action causes no visible harm, people treat it as a success. Decision-makers assume the system is safer than it is. Their fear of danger drops.

That habit pairs with outcome bias, documented by Jonathan Baron and John Hershey (1988). Teams judge a choice by its end result, not by the quality of the thinking behind it. A reckless production hotpatch that dodges a crash gets praised for speed. A careful rollout that hits an unexpected network drop gets blamed.

Field data from high-hazard industries shows the same pattern at scale. The gold standard is commercial flight. In 1997, the Federal Aviation Administration launched the Commercial Aviation Safety Team.

Regulators built the NASA Aviation Safety Reporting System to give pilots immunity for honest reports. Flight crews began studying close calls and path errors instead of waiting for crashes. CAST helped cut the fatal accident risk for US commercial flights by 83 percent between 1998 and 2008. When major crashes are rare, close calls are the only dense signal you have.

In healthcare, Amy Edmondson’s field studies (1996, 1999) found a famous paradox: the strongest hospital nursing units reported more medication errors than struggling units. Better teams were not making more errors. They had the psychological safety to speak up about near misses before patients got hurt.

When teams ignore those weak signals, they drift. Sociologist Diane Vaughan (1996) studied the Challenger disaster and named this normalization of deviance. Minor gas leaks on early shuttle flights became accepted risk because the rockets kept landing safely. Each flight expanded the boundary of acceptable damage until the system broke.

In software, Courtney Nash’s VOID Report (2022) analyzed thousands of public incidents and showed that incident duration does not track with incident severity. Public incident logs still miss almost all near misses, capturing only the outages that hit customers.

Then there is the operational baseline from distributed systems. In How Complex Systems Fail (2000), Dr. Richard Cook showed that complex systems run as broken systems with hidden flaws. Big outages need multiple small failures to line up at once. Single flaws rarely take down modern infrastructure; disaster needs a chain of hidden defects.

A near miss proves that every defense layer broke except the last one. In my experience running production infrastructure, treating that escape as proof of strength is the fastest way to cause an outage.

The reporting tax

If near-miss reviews are so valuable, why do most engineering teams skip them?

The main barrier is the reporting tax. In most companies, writing a postmortem brings an admin penalty: an eight-page doc template, three review meetings, and thirty Jira tickets for the author.

A postmortem process that punishes the reporter with red tape guarantees that engineers will quietly fix close calls and tell no one.

When filing a report brings red tape, rational engineers hide the signal. They push a fast patch, write a vague commit message, and go back to sprint work. The dashboard stays green, leadership assumes the systems are fine, and the hidden flaw waits for peak traffic to break the rest of the chain.

The 15-minute async brief

Great engineering teams do not put near misses through four-hour meetings. They drop the friction to zero.

First, set clear triggers so engineers do not debate whether an event counts:

• A canary deploy stops on a panic or bad data path that passed continuous integration.

• An admin script runs with wide write permissions but hits safe rows by pure luck.

• A service hits 95 percent thread or connection limits during off-peak hours.

• A deployment requires an engineer to turn off a safety check or force a release lock.

Second, use a fifteen-minute async template with four direct questions:

1. The hidden flaw: What dormant weakness or unexpected link lived in the code before today?

2. The trigger: What active change or traffic spike woke that flaw up?

3. The barrier that held: What circuit breaker, rate cap, or quick human fix stopped the blast radius?

4. The counterfactual: What single change in timing or load would have turned this into an outage?

Third, cap action items at two structural fixes, and publicly thank the engineer who called out the risk. When finding a near miss is treated as good stewardship rather than a confession, weak signals surface on their own.

An outage shows where your defenses broke. A near miss shows how your defenses almost broke, studied with a resting heart rate and no angry users on the phone.

The teams that survive complex systems do not wait for smoke to find the fire.