The One Part No AI Lab Will Build for You

Building an agent fleet in 2026 means building scaffolding around a capability gap that is actively closing. The standard recommendation — a review agent, a coding agent, a test agent, and an orchestrator to coordinate them — mirrors how human teams divide work, which is why it sounds sensible. I think it is the wrong layer to invest in.

The labs are training that division of labour into the models. Planning, sub-agents, self-review, tool use: every generation absorbs another layer of what was orchestration code twelve months earlier. You will maintain that scaffolding long after the gap is gone.

This is the contested part of my own argument, and I would rather say so than have it found: there is a serious counter-position that harness-level engineering keeps producing gains the model does not absorb. What follows holds either way, because none of it depends on the fleet being obsolete next quarter — only on where the work goes when it is.

One clarification before the objection arrives. What models absorb is planning: decomposition, delegation, self-review, deciding what to do next. What they do not absorb is durable execution — retries, scheduling, state that survives a crash. That is a reliability problem, not a reasoning problem, and it will still be there when the planning code is gone. Buy that layer, don't build it.

Two things move now. You only log one.

The harness — the loop that manages context, executes tools, holds permissions, sandboxes the environment — has become as consequential as the model, and it moves independently of it. Teams now run Claude Code as their harness with a cheaper model wired in behind it, purely on cost. That would have been incoherent two years ago, when the tool and the model were one product.

It is not a small effect either. One published benchmark found roughly twenty-four points of score movement on an identical model, from harness changes alone — model-generation size, from the layer everyone treats as plumbing.

So you have two independent variables. The model gets better and absorbs your instructions. The harness changes how context is loaded and how many turns a task takes. Everything you built assuming a fixed combination quietly stops fitting.

The consequence is unglamorous and almost nobody does it: log the harness version, the skill version and the inference settings on every run, next to the model version. Without that, a regression has two plausible causes and no way to tell them apart. No benchmark, cost figure or quality complaint measures a model — they measure model times harness times settings, and if you record one of the three you are not measuring anything you can act on.

The price you can see is not the price you pay

Which is where the swap-on-price instinct goes wrong.

Token price is close to useless as a planning quantity. The published rate covers input and output; the cost mass sits elsewhere — reasoning tokens spent before anything you read appears, retries after a tool returns an error, tool round trips that each drag their result back into context, and re-reads of a context that grew with every one of those steps. None of that was in the number you compared. All of it is on the invoice.

That is why a cheaper model can be more expensive, and routinely is. A lower rate with a lower success rate means more attempts per finished task, each billed in full. The run that goes seven turns deep, gets it wrong and is thrown away costs more than the one that got it right on the second turn at twice the rate. Throughput hides this: tokens flow, tasks move, and the bottleneck has quietly relocated into verification and rework.

So pick a model on cost per successfully completed task — not peak score, not price per million tokens. Run the platform on the same number, plus the ratio of failed to successful runs; that second one tells you whether you have a model problem, a skill problem or a missing test. In most cases I have looked at, a large share of consumption goes to runs that produced nothing anybody kept. That is not a rounding error. That is the budget.

What belongs in a skill

An agent is adding an endpoint to a payments service. Three lines from the skill it loads:

Keep functions small and use descriptive names. The shared auth middleware is in platform/auth-mw/v2. Payments never write to the database directly. They go through the ledger service — decided after the double-charge incident.

The first changes nothing. It is true at every company on earth, it overwrites training with opinion, and it costs you context on every single run. The test is simple: if an instruction would be equally true at any other company, delete it and let the model be good at its job.

The second is true until someone ships v3. Then it is a lie that keeps working — the agent imports the old version, it still compiles, and you find out in review. Facts that live in a system belong in a tool that reads that system, not in instruction text. A skill that copies data has created a second source of truth, and it is the one nobody updates.

The third is the one that pays for the skill. It is in nobody's code — the constraint exists because of an incident, not because of an interface — and it stops the agent from writing the exact thing your review would have rejected. Same category: which team owns which service, what a deployment needs before it goes near production, that a service is not finished until it appears in the catalogue.

Underneath those three lines is one division of labour, narrower than most skills assume. The model brings the craft: language, patterns, idioms, the thousand small decisions of writing code, and it is better at that than any instruction you can write. The skill brings the decisions: the architecture, the approved components, the boundaries, and the route to where the real context lives. Every line that crosses from the second category into the first makes the output worse, and it usually looks like diligence while doing it.

The route is the part people skip. Without one, the model guesses paths, opens files that do not exist, and reimplements what already sits in the repository next door — every dead end billed in tokens and paid for again in context noise. The fix is not pasting everything in, which explodes in size and goes stale within a sprint. It is a signpost: which repository holds the shared libraries, which directory has the examples worth copying, which system to ask for the rest. Same argument I made about MCP servers, where filtering at the source lets the model see the relevant five percent instead of the whole log. An MCP narrows the data, a skill narrows the search space, and together they remove the two most expensive habits an agent has.

You gave the agent write access to its own exam

A model with no way to check its work does not stop. It reads its own output, decides it could be better, revises, re-reads, revises again. You pay for every round, and the last five rarely change anything.

One test the model can run itself ends that: green or red, instead of ten rounds of self-reassurance. Two things have to be true for it to work, and usually neither is.

The test has to actually run. If checking a change needs three services, a seeded database and a VPN, the model will not check. It will guess, and it will sound certain.

And the test cannot belong to the agent. Your test suite sits in the repository you just gave it write access to. Nothing dramatic happens — an assertion gets loosened, a case gets skipped with a reasonable-sounding comment, a fixture gets nudged until it passes. Researchers have watched agents raise their own pass rates while real accuracy fell. So keep the criteria somewhere the agent cannot reach. It can suggest a test. Merging one is a human decision.

Even then: a test only checks what you thought to write down. Green is not correct. Green is not-red.

Which makes platform engineering and AI enablement the same job. What makes agents reliable is not a better prompt — it is a test suite that runs with one command, seeded fixtures, and an environment that comes up in a container instead of a ticket. Teams who did that work for their humans already have it. Teams who never did are paying for it twice.

A skill says how, not whether

A skill tells the agent how things are built here. It says nothing about what the agent is allowed to do: what it may push, what it may merge, what it may touch anywhere near production.

That question sharpens the moment the agent stops being triggered by a person. A scheduled or event-driven run has nobody at the screen, which is when a wrong action goes unnoticed longest. Permissions are their own layer. They encode your risk appetite rather than general competence, and no lab is going to absorb them.

A skill is a cost centre, and a dependency

I have seen a client produce an uncomfortable token bill with nothing more exotic than badly written skills. No runaway agent — just skills that loaded more context than they needed, pulled every available MCP into scope, and left the model to decide when a task was done. Each is individually defensible; together they multiply, and multiplication is what shows up on the invoice.

So a skill is a cost centre with a rate, charged every time anyone uses it, by everyone who uses it, for as long as nobody looks at it. It is also a dependency, which is what the weekend framing misses: it runs with the agent's privileges and carries instruction authority — everything a compromised package gets you, plus the ability to tell the agent what to do next. Public scans of shared agent skills have already turned up malicious ones, mostly working through injected instructions in the skill text rather than through code. If you would not install an unsigned npm package into your build, do not install an unsigned skill into your agent.

And because both variables keep moving, none of this settles. The model changes and half your instructions become redundant but still get sent. The harness changes how context is assembled, and a skill that was lean last quarter now drags a directory tree into every turn. Somebody has to sit with the usage data, find the skills responsible for the largest share of tokens, and rewrite them against the current combination. That is a standing role with more leverage than almost anything else on a platform team: one person iterating on the ten most-used skills moves the bill for everyone, in a way no developer optimising their own prompts ever will. FinOps, applied to tokens rather than instances — cost that nobody owns is cost that only grows.

The half I still cannot measure

The machine half has a denominator now: cost per successfully completed task, and the failed-to-successful ratio, against a pinned combination of model, harness and settings. Not a satisfying dashboard, but a real number that moves when you do something.

The human half I cannot instrument. What matters is the token bill plus the developer's salary, measured against what actually shipped — a skill costing thirty percent more per run but saving two hours of work is cheap. Every proxy collapses under weight: lines of code rewards the wrong behaviour, story points get renegotiated the moment they are used for evaluation, delivery metrics measure flow rather than whether the output was worth the input. And the real variance stays invisible — same skill, same model, two developers, one ships in an hour and the other spends a day arguing with it. I don't think that half gets solved by a better metric. It gets managed by keeping the loop short enough to notice. If you have found something that works at scale, I would genuinely like to hear it.

Two questions at the next release

Which instruction in here is now redundant, because the model has learned it? Delete those lines — against the eval, not against your judgement, because that is a change to the instruction layer like any other. A well-built skill shrinks with every generation, because it only ever contained what the model could not know.

And: which check in here is now gameable, because the model got better? A more capable optimiser finds the slack in a test that held fine last quarter. The skill shrinks. The verification around it does not — it has to be re-hardened at the same cadence as the thing it measures.

An agent fleet does the opposite of both. Every release makes more of your orchestration redundant, but the code stays, and someone has to maintain it, explain it, and eventually justify it.

The labs will keep absorbing capability, and the harness will keep moving underneath you. Let both. Spend the effort on the part neither of them will ever have: your architecture, your components, a route to your own code, a boundary around what the agent may do, and a test that proves the answer was right — one you still own.