The agentic maturity ladder

Two candidates. Both say yes when you ask whether they use AI to write code. Everyone I interview says yes now; in my hiring, that question stopped discriminating about a year ago.

One types a prompt into a chat box and pastes errors back until things look green. The other writes a plan, hands it to an agent, has a second agent drive the app to break it, and reads the diff against the plan. Same interview answer. Wildly different engineers.

The question that separates them is “show me your loop.” Grading the answers takes a clear scale. Here is mine: six rungs, defined by what the loop looks like and who does the verification. That second part matters more than the first.

The six rungs

Level 0: no agentic use. Still common, and not always the engineer’s fault. I hire across sectors; banks, defense shops, and health companies are laggards at adoption. Before grading a zero, ask whether it was policy or apathy. Someone who built agent workflows on side projects because work forbade them grades differently from someone who never got curious. Charity Majors calls engineering an apprenticeship industry; a level 0 with hunger may be an apprentice rather than a reject.

Level 1: turn-by-turn. Conversational coding. Prompt, skim, prompt again. Verification is vibes: it compiles, the happy path works, ship it. Vibe coding, in Simon Willison’s precise definition, is building software with an LLM without reviewing the code it writes. His verdict is celebratory: fine for personal tools, worth a check from someone experienced once other people depend on it.

In an interview, level 1 sounds like describing a chat. No plan exists before the code does.

Level 2: plan, then develop. The engineer specifies what should happen before generating code, points an agent at the plan, and reviews the output against the spec rather than their mood. Verification moved from “seems fine” to “matches what I specified.” That is a bigger jump than it sounds, and it is where most strong engineers sit today.

Level 3: plan, develop, agentic QA. The engineer stops manually verifying everything and designs the checks instead. Agents drive the app through computer use, click the flows, and test the edge cases. The human decides what needs proving and how it could fail. Willison’s account of serious agentic work puts the checking back on the engineer: agents can claim something works without testing it, so tests and manual QA stay yours, along with an instinct for what can be outsourced. Level 3 engineers talk about their checks the way level 1 engineers talk about their prompts.

Level 4: software factories. Parallel agents on decomposed, long-horizon workstreams. Verification lives in the structure: test suites and CI gates agents must pass before a human looks. The engineer’s output is the system that produces software. Almost nobody I have interviewed operates here yet, and that is fine.

The signal is whether a candidate tracks the frontier. Gergely Orosz’s reporting lands nearby: senior engineers use these tools more efficiently, and AI seems to amplify coding knowledge for seniors. Grade the artifact: the weekend rig, and what broke.

Level 5: autonomous systems. Closed-loop system stewardship. The loop moves from discrete tasks to continuous background operation. Agents read telemetry, triage errors, write tests, patch bugs, and deploy canary builds under risk limits. Verification lives in the environment: formal boundary rules, automatic rollbacks, and regression suites that prove safety before release. The engineer’s job shifts to setting policy, system limits, and risk budgets.

Almost nobody runs this in production today. The diagnostic signal is failure modes. Ask how their setup prevents invariant drift and reward hacking. A real practitioner knows an unchecked agent will happily pass a test suite by deleting the failing assert.

Running the probe

Twenty minutes, anchored to an artifact.

“Walk me through the last task you handed an agent. What did you write down first?” A strong answer reaches for an artifact: a plan doc, a list of constraints. A performed answer reaches for the prompt. Someone whose first written artifact was the prompt is at level 1, whatever vocabulary they use.

“What did it get wrong, and how did you find out?” A strong answer names a specific miss and the check that caught it: the eval gate that fired, the diff line that broke the plan. A performed answer says “nothing major,” or describes rereading the output until it passed a skim. A spotless record usually means nobody checked.

“What did your setup look like six months ago?” A strong answer has a plot: a step that was manual then and is automated now, a check that catches today what slipped through then. A performed answer has the same loop with a newer model.

Real practitioners answer with specifics and complaints: the plan that drifted, the test the agent gamed, the afternoon lost to a hallucinated package. People performing adoption answer with marketing.

Grade two observable things: the structure of the loop, and who does the verification. The second part matters more.

The Goodhart problem

Publish a ladder and you build a Goodhart target on purpose. Candidates rehearse it; interview prep drills the vocabulary. Two countermeasures keep the probe honest.

The first is texture. Rehearsed claims are thin, and failure stories are expensive to fake. Anyone can memorize rung names; almost nobody can invent the afternoon their agent gamed a test. Grade artifacts and scar tissue rather than vocabulary. A candidate who says “software factory” scores nothing; a candidate who describes the eval gate that caught their agent cheating scores even without the label.

The second is rotation. The ladder has an expiry date; a scale that stops discriminating gets rewritten or retired. The labels go stale fast; a live loop is harder to rehearse.

Two objections, because I had them too

This looks like CMMI. Maturity models have a deserved reputation for rewarding ceremony. The defense is the one above: a grade that rewards vocabulary over scar tissue has become the ceremony.

The rungs will move. Also true. The durable grade is the derivative: where were they six months ago, and where are they now? A candidate moving fast from level 1 beats a candidate parked comfortably at level 2. Trajectory stories can be rehearsed too; grade their artifacts and rotate when they go stale. You are hiring the rate of ascent; the current rung is context.

Publish your ladder internally so interviewers grade against the same scale. Then write the date on it, in pencil.