Writing
Evals > Vibes
Agent skills are code now. Almost nobody tests them, and the few honest measurements should worry you.
There’s a release process that has become common for agent skills. Write the skill, run it twice, feel good about the output, push. Maybe a colleague glances at it. The same engineers who would never merge untested code ship untested prompt scaffolds by the thousands and call it a productivity system.
Philipp Schmid, a staff engineer at Google DeepMind, gave a talk at this year’s AI Engineer conference titled like an indictment: don’t ship skills without evals. The index he cites is Skills Bench. It has catalogued on the order of 50,000 published entries. Almost none carried tests. Most were AI-written.
Skills Bench 1.1, as he reported it, ran about 100 coding and productivity tasks across open and closed models in different harnesses. Average performance rose by roughly 15 percent. AI-generated files, the ones the agent wrote about itself while you watched, can make performance worse. Those figures are a talk relay of a public leaderboard. I did not re-run the table. The median published skill is still untested, machine-written, and of unknown sign.
This isn’t laziness. The vibe check and the eval point at the same object. Only one of them can tell you no.
Why the vibe check always says yes
Load a structured thinking framework into a coding agent, a pre-mortem, say, or a first-principles decomposition, and give it a task. The output reads smarter. It’s organized, it names its assumptions, it walks through failure modes in order. The temptation to declare victory is enormous.
That impression is a claim about structure. Structured scaffolds reliably change the form of an agent’s reasoning: more legible, more auditable, easier to review. Those are real properties and worth wanting. Whether the agent gets more answers right is a separate claim, and the vibe check can’t evaluate it.
Start with non-determinism. Run the same task twice and you get different answers, sometimes a different impression of the skill itself. A framework that “worked” in your session is one sample from a noisy distribution. Small real effects probably exist in there, but you can’t separate them from variance without replication. One manual run is the opposite of replication.
The other confound is task difficulty. When a run fails, you can’t tell whether the skill was bad or the task was simply too hard for the model. Without an ablation, the same task with and without the skill loaded, every judgment about the skill is mixed up with the work you happened to test it on.
Two layers, and only one can say it helped
The harness that fixes this is smaller than it sounds. It has two layers. Conflating them is how teams fool themselves a second time.
The first layer is contract tests: cheap, deterministic checks on behavior. Did the skill trigger when it should and stay silent when it shouldn’t? Did the output use the current API and avoid the deprecated pattern? Did it follow the required structure? Most of these are regex asserts over the agent’s output and traces. They cost nearly nothing to run, and ten to twenty cases per skill is enough to start.
Write the negative cases too, five prompts where the skill should fire and five where it shouldn’t, because an over-triggering skill quietly taxes every session it doesn’t belong in. Contract tests catch regressions and mis-triggering. They cannot tell you whether the skill made the agent any better at the job.
That second claim, the one every skill README implies, needs outcome evals: graders that score whether the task was actually done well. For tasks with checkable answers, that’s a deterministic grader. Does the code run? Does the answer match? Is the plan within the stated constraints?
For fuzzier work, it’s blinded judging against a rubric, where the judge can’t tell which output had the skill loaded. Outcome evals are where the stricter controls earn their keep. Length control, so a skill can’t win just by making outputs longer, which graders of all kinds, human and machine, reliably mistake for quality. Ablation as the default. Replication gates, because a lift that appears once is an anecdote with a score attached.
The uncomfortable part is what this rigor tends to find. Projects that apply it to their own skill libraries usually discover much less than they hoped. Structured-reasoning scaffolds that feel like a breakthrough in use often show no replicated accuracy gain at all when measured properly. The few maintainers who publish those results openly, inconvenient findings at the top of the README, are doing something the prompt-pack economy almost never rewards: telling you how strong the evidence is.
The eval outlives the skill
Schmid draws a distinction worth building around. Capability skills teach a model something it can’t yet do consistently: a new API, an unfamiliar workflow. Preference skills encode how your team wants things done. The first kind is temporary by construction. Models turn over every few months, and each generation absorbs capabilities that needed scaffolding the generation before. Yesterday’s essential skill becomes dead weight in your context window.
The only way to know a skill has expired is to keep measuring after you suspect you no longer need it. Run the eval with the skill and without. When the gap closes, retire the skill and keep the eval. The eval is now your regression alarm for the day a model update loses the capability again.
The eval is the durable artifact. The skill was scaffolding all along.
Skills are code now. They ship to production, they change agent behavior, and they rot. They deserve code’s discipline: a change to a skill runs its tests before it merges, and a skill without tests gets treated the way you’d treat a pull request without them.
If you ship skills, the homework is one afternoon. Pick your most-used skill and write ten prompts, five where it should trigger and five where it shouldn’t. Run them with the skill loaded and without. You’ll learn something real before the tenth prompt, and there’s a decent chance it contradicts the vibe.