← Articles

Writing

MTTR > CAB

Why Change Advisory Boards fail to prevent outages, and why optimizing for recovery speed beats pre-release gatekeeping.

Engineering

Most tech teams that pass fifty engineers run into the change board.

The committee arrives dressed as prudence after an outage hits production. Leaders demand answers. Management sets up a board to review code before it ships.

Every release needs a ticket, a risk score, and a meeting. Engineers fill out forms describing changes. Managers nod at slides they cannot parse. The release gets approved, pushed to users, and breaks anyway.

The Change Advisory Board is risk theater. It does not stop outages. It creates them.

What the data says

The debate over change boards was settled by data years ago.

In the 2016 State of DevOps Report, researchers tracked change approval across thousands of tech teams. They tested whether external boards cut production failures.

The data was clear. External approval bodies do not cut change failure rates. Teams that used them had the same failure rates as teams with no formal process. But they suffered longer lead times and lower speed.

The 2018 report confirmed the pattern. External approval gates brought slower delivery and zero gain in stability.

Peer review and automated tests drove high stability and fast speed.

The false economy of failure prevention

Change boards operate on a broken idea: that you can test quality into code right before release.

The true damage of a change board is the batch size growth it forces. When shipping code takes a board meeting, engineers stop making small updates. They bundle two weeks of code into one big release to avoid facing the board twice.

Large batches are risky. When a big release fails, finding the root bug takes hours of triage across teams.

The cost of downtime keeps mounting. In DORA’s 2020 study, citing IDC data, hourly downtime costs for Fortune 1000 firms range from $1.25 billion to $2.5 billion. Rework eats 19 to 20 percent of total engineering time across all teams.

When an outage hits, what matters is how fast you restore service.

Fast automated recovery protects uptime. Manual approval committees merely delay failure while maximizing blast radius.

The compliance myth

The main defense of a change board is audit compliance.

Leaders claim that SOC 2 or auditors demand an approval meeting before every release. This is a mix-up of audit rules.

Auditors care about split roles, clear change logs, and proof of testing. They do not care about a weekly meeting.

A manual change board gives weak audit proof. A sign-off proves only that someone clicked approve. It offers zero technical proof of what code ran.

A modern delivery pipeline provides clear audit trails automatically:

First, branch protection rules ensure that no dev pushes code to main without peer review.

Second, CI runners log every test, scan, and build hash automatically.

Third, pipeline credentials ensure that only tested code reaches servers.

When auditors ask for change logs, a clean git history with signed CI proofs beats a sign-off sheet every time.

The recovery stack

Replacing a change board means replacing human guesswork with automated checks.

A modern pipeline meets audit needs in code and cuts recovery times to seconds:

  • Automated canary analysis. Deploy code to one percent of traffic, track error rates, and roll back if metrics degrade.
  • Feature flags. Split code deploys from feature releases, and toggle broken code off instantly without redeploying.
  • Immutable artifacts. Build a container once, test it in CI, and promote that exact binary across servers.
  • Peer review in git. Enforce branch protection rules that require clean CI checks and peer reviews before code merges.

The sixty-second recovery contract

To make fast recovery real, teams need a concrete operating contract.

First, keep all data changes backward-compatible. Never alter a database schema in a way that breaks running code. Use expand-and-contract patterns so that rolling back application code never corrupts user data.

Second, automate the rollback trigger. Do not wait for a human to log into a dashboard during an outage. If error rates climb after a deploy, the deployment pipeline should roll back automatically within sixty seconds.

Third, decouple deploy from release. Ship code dark behind feature flags. Turn features on for internal staff first, then five percent of users, then all users. If a bug appears, flip the flag off in five seconds.

Fourth, keep deployments small. When changes contain fewer than fifty lines of code, finding and fixing bugs takes minutes.

The operational mandate

If you want reliable systems, stop inspecting releases in committees.

Put your engineering hours into fast rollback tools, canary automation, and reliable test suites. Make deployments small, routine, and boring.

When a bad change reaches production, fix the rollback path and shrink the batch size rather than adding another approval gate.

Fast recovery beats board permission every time.