Now we’re building for a different kind of user: AI agents. Increasingly, agents are taking action on behalf of users and developers. They’re reading documentation, inspecting code, calling APIs and CLIs, and using MCP tools to interact directly with products. And just as good UX and DX require designing around the needs of their users, products need to account for how agents find information, choose tools, make decisions, and take action.

The industry has started calling this agent experience (AX). As more work moves through agents, their ability to understand and operate a product becomes part of the product experience itself. A platform can have excellent UX and DX and still be difficult for an agent to use reliably. At CircleCI, we’ve made solving that problem a core product goal: making CircleCI work just as well for agents as it does for the developers they support.

In this issue we’re sharing what we’ve learned redesigning CircleCI for agents. We’ll start with the product problem that pushed us into AX, then walk through five findings from our testing that changed how we think about context, tools, evaluation, and the way agents interact with a product.

Subscribe now

We started with broken builds

The need to design for agents is already showing up in how customers use CircleCI. In our 2026 software delivery data, average daily workflow volume increased 59% year over year, the largest increase we’ve measured. More agent-generated code means more builds, tests, and opportunities for failures to interrupt the development loop.

Broken builds were the highest-value place to start. If agents can produce code quickly, they also need an efficient way to recover when validation fails. Our first approach followed what Netlify CEO Mathias Biilmann describes as the closed model of agent experience. The original version of Chunk was a hosted agent that could diagnose a failing build, propose a fix, and open a pull request.

In production, Chunk turned CI green on around half of its attempts, but developers chose to merge less than 25% of its fixes. The takeaway was clear: engineers wanted more control over the fixing process. So we moved toward the open model, giving their existing agents direct access to CircleCI data and tools, so the developer could stay informed and in control of the recovery process.

Early benchmark testing showed how sensitive agent performance was to the environment around the model. In a 100-sample benchmark of CI repair tasks, better log preprocessing increased the share of proposed fixes that closely matched the known solution from about 25% to 53%. The size of the improvement made one thing clear: context, tools, evaluation, and workflow design could materially change how well the agent performed. We started testing those variables more deliberately, and five findings stood out.

1. Agents need finely tuned context

Our first instinct was to give the agent as much information as possible. The results moved in the opposite direction.

A raw build log can contain thousands of lines covering environment setup, dependency installation, test output, retries, warnings, and other execution details. Most of that information has little bearing on the failure the agent is trying to diagnose.

When we removed about 99% of the log content, the rate of fixes that took the wrong approach fell from 48% to 18%.

The experiment also showed that context placement mattered. Adding git context as a file did not improve performance, but putting the same information directly into the message produced our strongest run at that point. The share of fixes that followed project conventions increased from 59% to 90%.

For teams building agent workflows, context is worth treating as an input you design and evaluate. Test what information you provide, how much you provide, and where the agent encounters it.

2. Keep the tool set focused

Better context improved performance, so we tested whether giving the agent more capabilities would help further. We added skills, forced reasoning steps, and a much larger set of MCP tools.

More capability did not consistently produce better results. Added skills used about 24% more tokens without improving quality, and forced workflow steps reduced fix rates in our tests. In our largest MCP test, the server exposed 138 tools for the agent to choose from. The agent consistently relied on just 10 of them, less than 10% of the available tool set.

A developer who knows a product can navigate a broad API surface and choose the right endpoint. An agent has to select among MCP tools based on their names, descriptions, and the context available to it. Exposing every underlying API capability as a separate tool can make that choice harder without helping the agent complete the task.

For teams designing MCP servers and other agent interfaces, start with the actions required to complete the job. Add tools when testing shows that the additional capability improves performance.

3. Evaluate the result you want to ship

One of our early evaluation criteria was simple: did the agent make the build pass?

A passing build turned out to be an incomplete measure. An agent can turn CI green by increasing a timeout, loosening an assertion, excluding a failing package, or changing the failing test instead of fixing the code that caused the failure.

We started measuring merge quality separately from fix rate: not only whether CI passed, but whether the proposed fix was good enough to merge.

The benchmark exposed a meaningful gap: about 90% of fixes made CI pass, but only about 77% met our merge-quality bar. A green build alone was not always a sufficient measure of success.

The same evaluation problem applies to other agent systems. Intermediate metrics are useful, but the final evaluation needs to represent the outcome the user would accept.

4. Test the same task more than once

Changing the evaluation criteria also exposed variation between repeated runs.

The agent usually diagnosed the problem correctly, but the same failure did not always produce the same quality of fix. Incomplete changes and fixes that expanded beyond the required scope accounted for much of the variation.

Across repeated benchmark runs, 27% of cases flipped between pass and fail.

Clearer task boundaries helped reduce some of the variation. When the requested scope was narrow and explicit, the agent had less room to make unrelated changes or expand the solution beyond the failure.

For teams evaluating agent workflows, repeat the same tasks and look at the distribution of outcomes. A strong average can hide variation that matters when the system is used repeatedly.

5. Give agents direct access to product data

Cleaner inputs helped once the agent had the information it needed. We also found that agents spent a significant part of each run obtaining and moving information before they could work on the fix.

CircleCI already has much of what an agent needs to begin: which workflow failed, which job produced the error, which commit was running, what tests failed, and what happened during execution.

In our testing, only 20–30% of a run was spent reading the error and editing code. The rest included surrounding work such as exploring the repository, working with git, and moving information between systems.

Longer feedback loops also added measurable cost. In one iterative experiment, runtime increased from about 10 to 20 minutes and token usage from 2.4M to 3.6M as the agent kept working toward a passing result.

For teams adding agent access to an existing product, look closely at the information your product already has when an agent begins a task. Direct access can reduce the searching and reconstruction the agent would otherwise need to perform itself.

How we’re improving agent experience at CircleCI

Across the experiments, we eventually reached a point where a capable model could turn CI green on roughly nine in ten benchmark cases: 90% in one run and 84% in the repeat.

Reaching that level required substantial experimentation with context, tools, evaluation, and feedback loops. About 77% of benchmark fixes met our merge-quality bar, but production usage showed that developers wanted to stay closer to the fixing process, and the share of fixes they actually merged remained much lower than we wanted.

Those findings shaped the AX work we’re shipping now. We’re taking what made the fixer effective and building it into interfaces that give developers’ chosen agents first-class access to CircleCI data and actions.

A lot of the infrastructure behind the new experience is new or substantially rewritten, and we’re rolling out improvements on a daily cadence.

Recent changes include:

• A ground-up rewrite of the CircleCI CLI for developers and agents, with predictable JSON, stable exit behavior, agent-readable errors, and a built-in MCP server.

• circleci run get --failure-report, which gives agents condensed failure diagnostics instead of requiring them to work through complete build logs.

• Copy Fix Prompt, which packages the relevant CircleCI failure context into a prompt developers can paste into the coding agent they already use.

• Expanded MCP tools for inspecting runs, jobs, build output, and test results and taking CI actions from an agent’s development environment.

• OAuth 2.0 with Dynamic Client Registration and PKCE, giving local tools and agent integrations a standard way to request CircleCI API access without a manually provisioned client secret.

The new Copy Fix Prompt option is one of the ways we’re making CircleCI more friendly to agentic workflows.

The details are still moving as we put these interfaces in front of more real agent workflows. Our goal is to improve both sides of the experience: help agents diagnose and fix CI problems well, while giving developers the visibility and control they need to work with those fixes.

You can try the current fixing flow from any failing CircleCI workflow:

• Install the new CircleCI CLI: brew install circleci

• Run circleci run get --failure-report on a failing run to get condensed diagnostics for your local agent

• Or use Copy Fix Prompt from a failing workflow for a manual handoff

• Need to sign up? Run circleci onboard from the CLI to get started, or create a free CircleCI account from our signup page.

The biggest lesson from our testing is that agent experience is shaped by far more than the model. Context, tool design, evaluation, repeatability, and direct access to product data all changed how reliably agents could work with CircleCI. For teams building for agents, AX means designing those surrounding systems as deliberately as you design the product experience for people.

Median teams sit around 3.9. The top 5% run about 2.6. An elite cohort of twenty orgs is already near 1.3.

Most of the industry is still grinding through several rounds of feature-branch rework before earning the right to merge to main. The leaders are closer to one.

When coding agents are the ones waiting on those rework cycles, MER is not a vanity metric. It is the clock and the bill. Extra loops mean more tokens burned, more CI credits spent, and more agent context rotting on a red light from CI.

The Q2 Pulse Report is blunt: lower MER is a DX win, an AX win, and a cost win.

I ran a small AFK lab experiment that rhymes with that report. Not an org-level MER study. Same shape of problem, inside a single agent setup.

Subscribe now

Hypothesis

If a green PR is already likely, pushing only once at the end should burn far fewer outer-loop CI credits than pushing after every task. And if deterministic checks live on the inner loop once, you stop paying time and money three times for the same linter.

Pushing once at the end is only safe once the agent can reliably go green.

Setup

In our core experiment to determine the cost of a green PR, we found that if you give the agent a sneak peek at the CI setup, let it take notes before coding begins, and let it do a practice run on inner-loop checks before pushing to outer-loop CI, you can achieve 100% green PRs with zero rework.

The sneak peek is actually quite simple. Before task one, before a line of game code gets written, the agent receives a short inventory of the CI setup: which checks fire on the inner-loop practice run, which jobs fire on the outer-loop CI pipeline, and pointers to the scripts and configs that define them. This information is descriptive, but not an answer key. The agent writes its own notes from that inventory preflight.md) and can @-read those configs whenever it wants more. In our codebase, the inventory is a “CI validation manifest,” which tells the agent what will be graded and where the grading logic lives, before it starts guessing from vibes.

Fix loops drop to zero. Green PRs go to 100%.

This piece starts there. Same AFK Snake game build. Same seven tasks. Same two cadences, now with the sneak peek on.

• Per-task-push: a pipeline after every task. Seven tasks, seven trips.

• Single-push: finish the stack, clear practice run, push once at the end.

How often you push is the lever under test. First you have to be equipped to push green.

A linter is a linter. Compute is compute, whether it runs on a laptop, in a Chunk sidecar microbuild, or in a cloud job financed with CI credits.

There is no real reason to run the same deterministic check on localhost, again on the sidecar, and again on the outer-loop CI pipeline, as if three identical stamps make the code more correct.

That stack is a relic of human by-hand engineering. A person in flow forgets to run the test suite, pushes red, and we paper over forgetfulness by re-running the universe on every layer. Coding agents do not forget the same way. You can (and should) instruct them to clear inner-loop checks before they spend a single outer-loop CI credit.

In this calibration we moved inner-loop checks fully onto the sidecar and left thick outer-loop CI as the honest final test. Slimming true duplicates on the outer-loop surface is later work. The principle does not wait though: do not triple-pay for the same answer.

Results

Headline: once sneak peek made 100% green likely, a single end-of-run push cut estimated CI credits from ~81 to ~14.

• Per-task-push: 7 pipelines. ~81.4 estimated CI credits.

• Single-push: 1 pipeline. ~14.0 estimated credits.

About 5 to 6x fewer CI credits on the single-push run.

Wall-clock stayed in the 50 to 60 minute band either way (55.5 vs 56.7). LLM dollars stayed in the $15 to $17 band (17.15 vs 15.73). Tokens still track agent work. The CI credit collapse comes from the push cadence, not a cheaper model.

That bargain only holds when a sneak peek plus inner-loop checks make a 100% green PR likely. If you pile up unproven work and trigger the outer-loop pipeline once cold, you have not saved anything. You have postponed finding out you are red. When that one pipeline fails, you still pay the fix loop, and a retry can pick up new last-mile surprises.

Earn one take. Then trigger outer-loop CI once.

TL;DR

Elite engineering teams do not win by loving rework. They win by needing less of it.

Think of agent-era validation as a continuum. Inner loop to outer loop, hybrid on purpose. Each check earns its place.

• Deterministic checks live inward: cheap, early, once.

• World-shaped risk lives outward: policy, last-mile, flaky integrations, the CVE that did not exist at 9am. You name it.

Place each check where its information is unique. Everything else is just another duplicate cost for the same answer.

Outer-loop CI should still exist. It is where the world gets a vote. It should not be a museum of jobs you already passed in identical form on the previous two layers of validation.

At AFK scale, the inheritance path looks like this:

1. Make preventable reds disappear (push a /green PR in one take).

2. Keep remaining retries cheap (use the --failure-report flag).

3. Put deterministic checks on the inner loop once. Push to outer-loop CI as often as your green rate deserves.

Once 100% green is something you can count on, pushing only once at the end of the run is an obvious CI credit win.

Bake that into your agent setup. Low MER should not be a hero dashboard you inspect after the damage. It should be the default the agent loop was engineered to produce.

Rework should be reserved for unknown failures from the real world. Preventable failures should be totally eliminated.

Dennis Pilarinos has been thinking about this problem for years. He’s the founder and CEO of Unblocked, which he describes as the context layer for modern engineering teams: a system that surfaces the tribal knowledge inside an organization and makes it available to both the people doing the work and the agents working alongside them. Before Unblocked, Dennis spent years at Microsoft on the early Azure team, then moved through mobile CI and developer tooling at a time when, as he puts it, investors were telling founders “there’s no money in DevTools.”

There’s money in DevTools now. But the more interesting question is whether teams are building the right things into them. Rob and Dennis went deep on that question: why agents fail in ways humans don’t, what happens to the pull request when agents are writing the code, and whether we’re finally close to being able to draw a line from “we ran this agent” to “we moved this metric.”

1. Every agent you spin up starts on day one. Every time.

The framing Dennis uses here is hard to shake: imagine joining a company and being asked to fix a bug. You’re reasonably competent. But you’re up against someone who has three to seven years of context on that codebase. Who gets it done better? Almost certainly not you, not yet.

That’s the situation for every agent, every session. It doesn’t accumulate experience across runs. It doesn’t remember the incident channel discussion from three months ago, or why that particular API pattern was chosen over the obvious alternative. Each session starts cold.

“When you spin up an agent, it is basically like me joining CircleCI on day one. It has no meaningful historical context for which to draw upon. And so it does what I would do. I would probably dig through a whole bunch of documentation. I would go down paths that don’t actually lead to productive outcomes.”

Dennis Pilarinos, Founder and CEO, Unblocked

The practical takeaway is this: if your agents are producing messy, roundabout solutions, the problem is likely the missing context. What the agent needs is often sitting in Slack, in a closed issue, in a Confluence doc nobody’s opened in 18 months.

2. The code alone is not enough context.

Dennis is direct about something most teams discover the hard way: indexing the codebase is a starting point, not a solution. The reasoning behind the code lives elsewhere.

“I have never ever met a company that’s like, our docs are super well organized and completely up to date. That just doesn’t exist.”

Dennis Pilarinos, Founder and CEO, Unblocked

When Unblocked pulls in Slack threads, bug trackers, incident channels, and documentation alongside source code, the quality of outcomes goes up significantly. This isn’t a surprise in retrospect. It’s exactly how a person learns a codebase. They read pull request comments. They get feedback on their first few contributions. They sit in on postmortems. The “scar tissue,” as Dennis calls it, is where the real knowledge lives.

Teams building context systems that rely only on source code are building the equivalent of onboarding a new engineer with read-only access to the repo and no other communication. It doesn’t work for people either.

3. AI code review is only as good as the context behind it.

Unblocked started using its own context layer to build an internal code review tool after surveying the existing market and finding it underwhelming. The tools they evaluated felt like early-stage engineers leaving noise in PRs, sometimes literally leaving haikus. Not useful.

When they built review on top of a context layer that knew the history of decisions, past incidents, and team-specific conventions, the results were different. The reviewer understood why certain patterns existed.

“It’s not just limited to the source code. It knows conversations you’ve had about incidents, or some of the best practices that you might have written down somewhere else that other tools might not necessarily know.”

Dennis Pilarinos, Founder and CEO, Unblocked

If you’re evaluating AI code review tools and finding them shallow, this is worth paying attention to. The limiting factor is usually not the model’s ability to read code. It’s whether the model understands the context in which that code was written. Most tools don’t have that. The ones that will matter will.

4. The pull request is two tools in a trench coat.

Rob and Dennis had one of the more useful framings of this conversation on the subject of pull requests. The PR solves two distinct problems: it’s an information-sharing mechanism (here’s what changed and why) and a change management mechanism (here’s how we control what goes to production). Those are different problems, and they’ve been bundled together for so long that most teams have stopped asking whether they need to be.

As agents generate more code, the PR review bottleneck gets real fast. Teams that Dennis talks to are saying their engineers are backed up on review, not on writing. The question becomes: can you decompose what the PR was doing and solve each part better?

“Go back to first principles. Pull that apart and say, what would be the way we would solve for this piece of it, for this specific problem in this new context? It doesn’t have to be one solution. It just happens to be now, because it was there.” Rob Zuber, CTO, CircleCI

The implication for engineering leaders: look at every gate in your delivery pipeline and ask what problem it was originally solving. The answer might still be worth solving. The mechanism might not be the right one anymore.

5. AI adoption inside one company spans an enormous range.

This was one of the more grounding moments in the conversation. Dennis described organizations where some engineers are just discovering tab-completion and others are running thousands of autonomous agents handling multi-hour implementation tasks. Same company. Same week.

He puts roughly three to five percent of engineering teams in the “AI pilled” category: people who have experienced enough of the technology to feel acutely where the gaps are. They’re the ones who find the lack of organizational context genuinely painful, because they’ve hit the ceiling. The rest of the org might not have reached that ceiling yet.

This matters for how you roll out tooling and expectations. Someone whose primary interaction with AI is autocomplete doesn’t have the same mental model of what’s possible, or of what’s broken. The journey isn’t linear, and it’s not uniform.

6. The ROI problem is closer to being solvable than it’s ever been.

This was the most forward-looking thread in the conversation, and both Rob and Dennis were careful about timelines. But the observation is worth sitting with: agents are instrumented in ways that people never were. You can actually see what it cost to get to a solution. The question is whether you can connect that to business outcomes.

“We can say: what did it take in order for you to come to this conclusion or to this solution? And can you do it in a token-efficient way? But what will happen next is: how do we reconcile that execution relative to business objectives? Not just how much did it cost, but how did it actually move the needle for the business?”

Dennis Pilarinos, Founder and CEO, Unblocked

Finance teams are noticing the spike in costs. The engineering side hasn’t yet built the case for what that cost produced. Closing that gap, drawing the through-line from token spend to business outcome, is the work in front of engineering leaders right now. Dennis thinks we’re closer than we’ve ever been. Rob’s take: we’ve taken one step on a very long journey, but it’s a real step.

Confident Commit is CircleCI’s publication for engineering leaders and practitioners. Subscribe to get episodes and companion posts delivered directly.

Our current CircleCI APIs use version numbers in the URLs. V1, V1.1 and V2. They had accumulated four incompatible URL patterns, three ID formats and a different response shape per endpoint, effectively ten years of drift. The pitch for a new API V3 was explicitly about consistency for agents: one URL grammar, UUIDs everywhere, one response envelope, one error shape. An LLM should be able to predict /api/v3/{entity}/:id from the entity name alone instead of carrying the whole spec in its context.

We followed all the current advice on how to make the API discoverable by agents. We published the OpenAPI spec at a stable URL. We added /.well-known/api-catalog per RFC 9727, so a machine could discover the machine-readable descriptions. We generated an llms.txt and an llms-full.txt. We rendered the whole reference as markdown as well as HTML. We put the conventions - error shapes, idempotency, which operations are irreversible - into the spec description so they travel with every entity document, not just the index.

This post answers one question: did any of that work?

Hypothesis

My assumption at the outset was that agents would want to work with a consistent API, and that they would be able to easily find it.

I expected consistency to be the hard part and discovery to be the easy win.

Specifically:

1. Agents would find V3 on their own through the discovery surface - the catalog, the OpenAPI spec, llms.txt.

2. Once they found it, V3’s consistency would keep them there, because guessing a V3 URL is easier than looking one up in V2.

3. V2 usage by agents would decline as agents discovered the better option.

The first and third were wrong in ways more interesting than being right.

Setup

I first proposed building a new consistent API 3 years ago. At the time, it would have been a massive time expenditure likely not worth the investment. But by the time I pitched it again earlier this year, the execution had changed; we could use a swarm of agents with good skills. Agentic workflow enabled us to decide that it was a low enough cost to take on an experiment previously considered to be very high cost, and impacting every team on the org. I got the green light.

I tend to be more pragmatic than scientific. I understand the scientific method, but can also plan ahead to mitigate the risks of just diving in and trying things. In the case of the API, although it’s fairly high risk in some ways, I knew that if we didn’t publicize it fully, we could always stop and roll back.

Every V3 route ingresses through one gateway service. That was a deliberate call made for enforcement reasons, and it is also what makes this measurable: one dataset holds every public API request with its route, method and user agent.

After agreeing on the consistent flexible API shape, we built skills internally for teams to use to guide their agents, and within a few months we had V3 APIs in place that could replace 90% of existing API traffic.

Measuring agent traffic hitting our API is an inexact science, relying on parsing the User-Agent in the inbound requests.

Real strings from production for requests coming in via our CLI:

circleci-cli (darwin/arm64; 1.0.48773; claude-code_2-1-246_agent)
circleci-cli (darwin/arm64; 1.0.48692; codex)
circleci-cli (darwin/arm64; 1.0.48773; opencode)
circleci mcp (755dd082f9ef571f74ac433160ab068e10c9ef25)

That is self-reported identity, not inference from traffic shape. If you want to measure agent usage of your own API and you control a client, stamping the harness name into the user agent is the highest-value thing you can do. It costs one line and turns an unanswerable question into a query.

Two caveats to keep in mind if you try something similar: An agent that hand-rolls its own HTTP calls is invisible to this method. I checked the bare-HTTP-client population and found roughly 41,000 spans a day where an agent could hide indistinguishably from a shell script (the span counts come from a sampled tracing dataset: the agent-filtered queries returned unsampled, the whole-traffic ones at 1.7× mean sample rate.)

Results

Interestingly, we realized the agents don’t care about the API directly. Instead, their focus is the MCP and CLI. Having a consistent API made it easier to build a good CLI. It’s only in this roundabout way that the agents are interested in using the API.

Agents overwhelmingly use V3. But not by migrating.

Hypothesis 2 confirmed, emphatically. Now the same measurement seven weeks earlier:

In those seven weeks, agent V3 share went from 57% to 97%. But look at the V2 column: 17,641 to 22,375. V2 agent traffic did not decline. It grew slightly. V3 did not win by converting anyone. It won by absorbing 760,000 calls of agent traffic that did not exist seven weeks earlier.

Hypothesis 3 was wrong: the migration story I expected to tell was actually a growth story.

For contrast, across all clients including browsers and scripts over that same recent week, V2 still outweighs V3 - 24,067,618 calls against 16,978,477. Humans and legacy integrations have not moved. Agents are a separate population behaving differently.

No agents found the V3 API on its own

Here is the entire discovery surface over thirty days. Every file we published for machines to find:

Discovery endpoints · All clients · 30 days to 28 Aug 2026

Agents made 807,209 API calls in a week. The entire discovery surface got 6,776 hits in a month.

Then I broke those 6,776 hits down by user agent, expecting a long tail of coding agents. There are three populations in there, and none of them is the one I was looking for.

Population one: nobody in particular

Crawlers, scanners, and humans

The largest single consumer of /.well-known/api-catalog is a bot that exists to grade whether your API is agent-ready. We are being audited for agent-readiness by a crawler while the agents themselves never look.

Population two: agents, but only when a human points at them

These are visible and separable, because a directed fetch carries a different user agent than an agent doing its own work:

Claude-User (claude-code/2.1.220; +https://support.anthropic.com/)

That is Claude Code’s web fetch, which fires when a person says “go read this URL”. Across the whole discovery surface in thirty days it accounts for 23 requests - 6 on the static OpenAPI HTML, 5 on /docs/api/v3, 4 on /fullopenapi.json, 4 on /.well-known/api-catalog, and single hits elsewhere. Two more came from Claude-User/1.0, the claude.ai equivalent.

So the honest version of the finding is not “no agent has ever read it”:

Zero agents read the discovery surface unless a human explicitly directed them to it. Twenty-three times in thirty days, someone did.

Every one of those 23 was a person (probably one of us) saying “fetch the catalog”. Not one was an agent concluding on its own that a catalog might exist and going to look.

Population three: the training and search pipelines

This is the part that gives me some hope. Around 203 of the 6,776 hits are corpus builders and search crawlers:

They are all reading /openapi.json. The spec is being ingested, but just not into a working agent’s context. It is going into the next generation of training data and the search indexes agents fall back to. The discovery surface is on a much slower clock than I assumed.

Which means the optimistic reading of this whole post is available, but it is not the one I would have guessed. It is tempting to hope that the next round of models will start consulting catalogs and llms.txt properly, the way the standards intend. I do not think that is what will happen. What will happen is that GPTBot and ClaudeBot read our openapi.json, V3 ends up in the weights, and the next generation answers V3 from memory… exactly the way this generation answers V2 from memory.

The fix for stale training data is more training data. The catalog’s delivery vehicle is the corpus, and the crawler is the courier.

So the discovery surface does pay off. Just not as discovery. And the loop it runs on is a model generation long, which is not something you can shorten by publishing harder. The only lever you control on that timescale is still the client.

Why agents don’t discover: two layers of stale

Watching agents work with our API, the failure has two stages and neither involves discovery.

STAGE 1: Training data answers first

Ask an agent about the CircleCI API and it answers immediately and confidently with V2 shapes - /api/v2/project/{project-slug}/pipeline, slugs in paths, the old status enum. It is not looking anything up. V2 has been in the corpus for years; V3 has existed for months.

STAGE 2: The search index agrees with the training data

Agents will often announce they are searching the web when they are reaching into training data. When they do genuinely search, the indexes rank a decade of V2 documentation, V2 Stack Overflow answers and V2 blog posts far above anything we published this year. Two independent mechanisms, same wrong answer.

And here is the twist that makes discovery standards nearly useless in this shape: by the time an agent finally reaches a spec or a catalog, it already believes it is looking for V2. It arrives with a target. We updated the spec descriptions to say prominently that a newer V3 API exists, and an agent that has decided it needs the V2 pipeline endpoint reads past that to go looking for the V2 pipeline endpoint.

An in-context hint only helps a reader who has not yet decided what they are looking for. An agent has always already decided.

What actually moved the traffic

We updated the CLI to use V3, and pointed the built-in MCP server at the same code.

My colleague Pete has written up the rebuild this came out of (Rebuilding the CircleCI CLI from scratch) and it is worth reading alongside this one, because his half of the story is the half that actually moved the numbers below.

Roughly 57% of all CLI traffic to the public API now carries an agent harness stamp. Every one of the 807,209 agent calls arrived through the CLI or the MCP server. None arrived from an agent constructing its own HTTP request.

Agents do not discover APIs. They use the tools already in front of them. The CLI is already installed, already authenticated, already on the agent’s PATH, and --help is right there. The MCP server appears in the tool list. Neither requires discovery, a search, or a spec.

The lever was never the discovery standard. The lever was the client.

TL;DR

No agent reads your discovery surface unless a human points at it. The catalog, llms.txt and the OpenAPI spec drew 6,776 hits in a month. Agent reads: 23, every one a directed fetch. Self-directed: zero.

Publish it anyway, but price it as a training-data bet. 203 of those hits were GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot and MistralAI-User. The spec is being ingested, just not into a working agent’s context.

Expect the fix to arrive through the weights, not the standard. The next model generation won’t consult your catalog. It’ll just already know your API. That’s a model-generation-long loop you can’t shorten by publishing harder.

In-context hints only help a reader who hasn’t decided yet. We put “there is a newer V3” in the spec description. An agent that has already concluded it needs a V2 endpoint reads straight past it.

Ship the client and the MCP server. 100% of measured agent traffic arrived through a wrapper that was already installed and already authenticated. None of it was hand-rolled HTTP. If you want agents on your new API, update the tools they already have.

$ circleci run trigger --help | head -40

I hadn’t asked for head -40. Claude added it, and it does that constantly; help output is untrusted input of unknown size, and pulling an unbounded page into a context window is a bad trade. Forty lines is the interface. Below line 40 was the part that tells you how to use the command.

Hypothesis

As I shared in a previous blog post, when I started rewriting the CLI, I already understood we were writing for agents; that was the premise of the rewrite. Every data-returning command has --json with its fields enumerated, and the command tree generates an MCP server, so each command’s description is a tool description.

Two things we didn’t know:

1. We didn’t know there was a length limit, so we never measured against one (”written for agents” was a set of content decisions, and none of them had a size).

2. We didn’t know that we were saying everything three times: the accepted values for --event-preset, the default for --provider, all of it in the examples, in the flag table, and again in the prose. Each section had been written to stand on its own, and nobody diffs a paragraph against a table.

So the fix wasn’t “write less”. If the length was repetition, most of it would come out mechanically; and if prose went last, truncation would cut the cheapest content rather than the most expensive.

Setup

I measured before editing anything, because “is this help text too long?” is otherwise a taste argument nobody wins. Walk the command tree, render every page the way a real invocation would, and count. Not total lines, but whether each section finishes inside the first 40. A flag table that starts on line 38 isn’t visible; it’s a teaser.

Across 169 command pages, the mean was 49 lines, the worst 91, and only 15% got their examples inside the window (the section an agent actually copies from).

Three passes. Boilerplate out: the banner, the per-page table re-printing the same four global flags, the repeated “Learn More” links. Seventeen lines a page, no prose rewritten. Reorder: Short → Usage → Arguments → Flags → Examples → Details, so truncation eats prose first. Trim the duplication, never dropping a flag row or an example to make space. project trigger create went from 91 lines to 42.

Then keep it fixed. Order and boilerplate are properties of the help template now; length is a test that asserts every page fits 40 lines, with individual caps for the twenty-five commands that genuinely can’t, and those caps only ratchet down. And since these commands are largely written by an agent working from guidelines in the repo, the rule went there too.

The PR: https://github.com/CircleCI-Public/circleci-cli/pull/1667. 449 files, about 2,600 net lines of help text deleted.

Results

The three passes cut both the mean and the worst case, but what really matters is whether the sections an agent actually uses (flag table and the examples) finish rendering before truncation. A page that runs to 70 lines but front-loads its flags is a better interface than a 45-line page that buries them. The reorder pass was designed around this: put the cheap content last, so if something gets cut, it isn’t the part Claude copies from.

No task-success delta, because I don’t have one. What changed is the interface: every command’s complete flag list now sits inside the window an agent reads, and three in four get their examples there too, against one in seven before. The old failure mode was silent, which is why it lasted. Claude wasn’t erroring; it was succeeding at a degraded version of the task on whatever fragment of the flag list it had seen. Worse than a stack trace, because nothing tells you to go and look.

TL;DR

“Designed for agents” is a budget, not a style. We had the content decisions right and still shipped help that couldn’t be read, because we never asked how much of it arrives.

Pick one canonical place for each fact. The duplication wasn’t sloppiness; it was three self-sufficient sections maintained separately. The flag table is canonical now, and prose restating it is a defect.

Make it a test. A style guide asking for 40 lines would have lasted a month; the next person adding a command, human or otherwise, has no reason to know the number exists.

The pages are better for humans now, which I didn’t expect; not because agents and people want the same thing, but because neither mistake was agent-specific. Nobody benefits from reading a flag’s values three times, and nobody was reading line 70 either.

At roughly the same time, I’d been spending time considering how I might be able to use Chunk sidecars for related tasks outside of testing. I wondered: Do sidecars have the potential for a wider application?

I’d say that’s representative of my style as a whole: When conducting experiments, I don’t just think about the results, I try to think about new ways of conducting the experiment that can be applied to other experiments in the future to create additional efficiencies.

In the case of this specific experiment, I was fairly certain that the problem would be easily divided and conquered by using multiple computational resources. Without that approach, the experiment would be costly and cumbersome. Chunk sidecars provided the best route forward, and resulted in a method that I (and any other CircleCI customer) can replicate within future experiments too.

Hypothesis

Config validation is embarrassingly parallel. Each file is independent. There’s no shared state, no ordering requirement, no reason one file’s result should wait on another’s. If we could distribute validation across a fleet of Chunk sidecars, the wall-clock time should drop to roughly: (time to validate one file) / (number of sidecars running in parallel).

The biggest question was how to keep this cost effective from a time expenditure perspective, given that sidecars take about 30 seconds each to spin up. If each sidecar only validated a handful of files before its setup overhead dominated, the fleet wouldn’t be faster than sequential. It would just be more expensive.

Setup

Chunk sidecars are lightweight remote microVMs that run microbuilds alongside a developer’s work session. Each one is a full Linux environment: it can run the compiler, execute arbitrary code, and report results back.

It was important to me to conduct this work in a way the customer could reproduce. Don’t just trust me – try the same validation on your end as well to prove that the pipeline won’t be broken when the change is introduced.

Makoto Mizukami, Senior Field Engineer JAPAC, CircleCI

Knowing that the customer wouldn’t have access to our database (making the experiment potentially costly for them to run themselves) I made sure to use resources that would be available to them as well – the CircleCI API and CLI.

For this experiment, I partitioned the customer’s 10,000+ configs into batches and assigned each batch to a sidecar. Each sidecar:

1. Spun up

2. Received its batch of projects

3. Walked through each project to fetch its config through CircleCI API and run it through circleci config validate--next

4. Reported pass/fail results, and failure reasons if any

The fleet size was 100 sidecars in parallel at maximum. Task orchestration, command dispatches, and result aggregations were all done by a single Claude Code agent session, allowing me to save tokens (unlike Claude subagents, which hungrily consume tokens). The work was instrumented with Honeycomb throughout: 32,359 spans across the full run.

Results

The fleet economics held up. Setup overhead for the 100 sidecars was 2 minutes in total — small enough that the parallelism paid off at this batch size.

One finding worth noting: We also observed some specific patterns in validation failures. For example, certain series of strings tended to raise a validation error. By analyzing the result of the large-scale process further, we were also able to identify some “template” projects, from which most of the 120 configs derived.

The reason that the setup was reasonably easy was that Chunk sidecars provide snapshots. I could install a basic toolset in the sidecar, take a snapshot and apply the snapshot to the sidecars. Without that, this experiment would be impossible to conduct efficiently.

TL;DR

Embarrassingly parallel tasks don’t need clever algorithms. They need an environment that can run many agents at once. The compiler logic didn’t change. The validation logic didn’t change. What changed was the execution model: instead of one process working through a queue, a fleet of processes each worked through a slice.

Running them sequentially would have taken hours. A Chunk sidecar fleet did it in 4 minutes.

The thing to check before you do this: Make sure each agent knows exactly what its job is. Per-agent setup costs compound. If your task is trivial and your setup is heavy, the fleet won’t help. In this case, validation was fast enough and setup was cheap enough that the math worked. Instrument it first and check.

The customer got their answer before the breaking change shipped. It was an answer we could never have gotten in a timely manner without the massive parallelism Chunk sidecars offered. It proved my hypothesis that Chunk sidecars have multiple uses, and aren’t just limited to testing. As it turns out, sidecars are well implemented so that agents can easily leverage and consume the data, making them a perfect vehicle for experiments like this one.

We complain a lot about AI agent output. But we need to be thinking more about how we’re setting our agents up for success, and whether we’re providing them with sufficient power to do their job. Sidecars are one of the best ways to prepare agents to do great work.

Then suppose that same student could peek at the test in advance and take notes.

One catch. The sneak peek does not include any of the last-minute trick questions that might show up on the real exam without warning. Those get written by “the world” the moment the exam starts.

Nobody on the teaching staff knows the trick questions in advance, and new ones can show up on a retry.

Pencils down.

That is coding agents and CI. Merge-ready green is the only passing grade. Outer-loop CI is the real final exam. A Chunk sidecar is the timed practice run. Every red→green fix loop (diagnose, patch, push again) charges a little tuition: LLM dollars, CI credits, wall-clock.

I wanted to know what a green PR actually costs on a fully-automated AI coding agent run. Time. Tokens. Credits. The whole bill.

We moved the needle a bit on time and tokens. Fine. Not the story.

The story is rework going to zero on the stuff we can predict. Deliver the PR in one take. Ace it at 100%. The only red you should tolerate is the unknowable.

Hypothesis

If you show the agent what CI will check before it writes product code, give it a page of notes, and make it pass a real practice run on a sidecar, preventable outer-loop failures should stop showing up. Fix loops should not be necessary. Green commits should hit 100%.

Not “prompt harder.” Setup for success. Ace it at 100%. One take.

Setup

In this lab, an A- is still red. Local lint that smiled while outer-loop CI frowned still fails. A green sidecar run that dies on the real pipeline still fails.

Two cadences. Same bar: every commit you push to outer-loop CI is green.

• Per-task-push: after each task clears practice run, push. Seven tasks, seven trips. Ace every one.

• Single-push: finish the stack, clear practice run, push once at the end. Ace that one.

How often you push is a later lever. First you have to be equipped to push green.

Without the sneak peek

Three calibrations on the same AFK classic Snake game build. Seven tasks. Hands off the keyboard.

No inventory of checks up front. No notes. Inner-loop checks existed, but they were split between the localhost and a sidecar.

They finished, but rework kept piling up. When we feed agents endless retries and mountains of feedback, we’re just repeating the same habits we once drilled into human developers, now adopted by their AI counterparts.

*NOTE: We can make retries cheaper with a structured CI pipeline --failure-report, but that is its own write-up. Spoiler: even if you cheapen input tokens on each retry, you still have to pay time and money to retry.

Then we showed it the test

Before task one, before a line of game code, the agent got a short inventory: which checks fire on the practice run, which jobs fire on the thick outer pipeline, and pointers to the scripts and configs that define them. Descriptive. Not an answer key.

It took notes (preflight.md). It could @-read those configs when it needed more.

In our codebase that inventory is a “CI validation manifest”, which basically tells the agent what will be graded, and where the grading logic lives, before it starts guessing from vibes.

Inner-loop checks moved fully onto the sidecar, and we stopped running the same exact lint and test loops on the localhost. Outer-loop CI stayed the honest final exam, including whatever the world still wanted to throw at it.

Results

Headline: 0 CI fix loops on both sneak-peek arms.

• Per-task-push: 7/7 pushed commits green.

• Single-push: the one end-of-run push went green on first contact.

Zero outer-loop CI failures on either arm.

Diagnosis tokens on the rework line: $0. That budget can fund first-pass product code generation instead of burning LLM dollars and CI credits on red -> green archaeology.

Wall-clock did not collapse. Still about 50 to 60 minutes. Coding time dominated. LLM dollars landed about $15 to $17, in line with earlier attempts. The key difference: rework nearly disappeared.

Quiet rework is the same muscle Merge Efficiency Ratio names in CircleCI’s State of Software Delivery Q2 Pulse report: how many validation cycles before a change is actually done. This lab is that problem inside one agent setup. Once green is likely, push cadence and where you place the checks are how you bake low MER into the loop.

TL;DR

A coding agent can only prevent what we can predict. Say that out loud.

Linters. Unit tests. Typechecks. Formatters. Lockfile rules. Contract tests you already wrote. The deterministic stuff with a known answer. If it is on the syllabus, the agent can study it, practice it, and clear it before you spend an outer-loop credit.

It cannot prevent what the world invents after the peek is printed.

Late-breaking CVEs mid-run. A registry serving a different tarball than the one you resolved an hour ago. A base image that rotated overnight. An org policy that fails a dependency you did not touch. A flaky third-party API. A secrets scanner lighting up on a fixture. A new advisory. A mirror outage. A quota. Runner drift. “Works on my machine” that is really the cloud’s clock, certs, or DNS.

Platform engineers, harness authors, and coding agents cannot predict this. It arrives when the CI pipeline is already running.

Design for what the agent can own. Leave outer-loop CI in place for the rest. Ideally one take. At most two, once the world gets a vote.

Build the agent setup so it can ship the application end-to-end with zero CI failures from known deterministic sources.

Inventory of checks. Notes. A practice surface that is real (sidecar, not theater). Then push.

Do not “prompt harder” and hope the fix loop works overtime.

One take. Let’s go.

We wanted to understand: How effective was Chunk? Was there something we could do to improve the fixes given to the users? And perhaps most importantly, was there a way to improve the accessibility of the failure data to agents?

We tested the effectiveness in a controlled eval benchmark to measure the actual impact. The same dataset, the same model (claude-sonnet-5), against three different setups: no CLI, CLI with free use, and CLI with --failure-report. Here is what we found.

The context

Our first experiments were designed to mimic how Chunk works: giving it a file, then a generic prompt written by the team (for example “you are a coding agent, you need to solve this error” etc). Essentially at the beginning, we were trying to change the prompt to see if the fixes improved, and if giving certain instructions, steps, or validations would move the needle. The reality? The prompt didn’t seem to matter much.

Our team was constantly reading blog posts from other AI or CI/CD companies about what they were doing to solve this problem, and eventually we came across a paper by Bytedance. The paper shared that when cleaning log output, most of it is useless so the best thing to do is focus on the part of the log that contains the error.

We began to wonder: How do we remove the unnecessary characters and do a diff between the good and bad logs? That was the first successful experiment that gave us good results. It followed the way Chunk works. Because of that, we released the API of step output condensed. You fetch the output of a failed step and process to clean it, so you give better context to the agent.

From there, our focus eventually moved away from Chunk and toward the inner loop. We redesigned the experiment to mimic the usual developer flow.

“I thought: ‘If I was a dev working on a project and my pipeline failed, how would I fix it with an agent? Probably by copying the pipeline URL and giving it to an agent to fix. I saw that it had to follow a lot of steps to get to the pipeline output. The agent has context to the pull request, but it doesn’t necessarily have sufficient context.’”

Joaquin Sandoval Miramontes, Senior Software Engineer, CircleCI

Hypothesis

The hypothesis we arrived at was this: CI failure data is structured for humans reading a dashboard: status icons, log streams, nested UI. An agent working from that raw output burns context on navigation before it can start fixing. Therefore, restructuring the same data for agent consumption should reduce diagnostic turns and get the agent to fix faster.

Setup

I always try to do experiments in a structured fashion. I want to make sure that I have a way to repeat my own results over and over, and I’m particular about designing experiments with a hypothesis, expected results, and conclusions – I try to follow the scientific process wherever possible.

So here’s what we tried.

New flag: circleci run get [run_id] –failure-report

Output format: condensed, organized [workflow → job → step], showing only what failed. Built for piping directly into an agent’s context window.

circleci run get [run_id] --failure-report | claude "fix the failing workflow"

We compared three conditions on the same real failed pipelines from production:

• An agent given authenticated access to the CircleCI API

• An agent given free use of the CircleCI CLI

• An agent given access to the CircleCI CLI instructed to use --failure-report command

Results

Efficiency

The flag delivers real efficiency gains:

Agents using --failure-report spent fewer turns overall and finished faster. The CLI-only variant used the most turns and tokens due to agents navigating the CLI tool chain to resolve which job failed.

Fix Quality

Fix quality is flat across all variants. Fix rate difference is within the regular variation across agentic runs.

TL;DR

It used to be that an agent, when handed a failed CI run, spent many turns just figuring out what broke. The problem: the failure data was there, it just wasn’t accessible by an agent. The new command circleci run get [run_id] –failure-report changes that.

Previously, the agent was taking too many steps to get to the error itself, so we summarized it into one line. More context for the agent means better fixes, and spending less.

“If you have repeated tasks for your agent, give them tools to get it done faster. Don’t make your agent do 3-4 things before it gets started.”
Joaquin Sandoval Miramontes, Senior Software Engineer, CircleCI

The same failure data, restructured for agent consumption, changes how efficiently the agent works. The agent still needs to gather context from the source files, understand the failure and write the fix. What it no longer needs to do is spend 4-5 turns navigating a log hierarchy to find the failing step.

The practical conclusion: agents using --failure-report reach the same fixes in about ⅓ less time and ¼ fewer tokens. At scale, given thousands of CI fixes per day, that efficiency difference is real.

Has your team run any similar experiments? What have you learned about restructuring failure data for agent consumption? We’d love to hear from you in the comments below.

Data: 84 real CI failures from CircleCI. Model: claude-sonnet-5. Quality Judge Model: Claude-opus-4.6.

That loop runs on my machine most days. I wanted a cheaper version of it, and I had a specific idea about where the savings were hiding.

So I broke six CircleCI pipelines on purpose. Then I saved one specific failed run of each, so every model would stare at the exact same wreck. Three AI coding setups, one instruction: fix this.

Grok 4.6 fixed all six. Twenty-two cents a fix.

Claude Sonnet 5, turned all the way up, fixed three. Every fix it did land cost about $1.11.

I walked in looking for the opposite result. The chatter said Grok would be the one to stumble on the terminal work.

What I was actually hunting

A routing rule, which is a boring thing with a boring name: a policy for which model gets which wreck. Teams already do this with humans. The YAML typo goes to whoever’s on rotation. The bash failure that’s been red since Tuesday goes to the one person who can actually read a wait loop. I wanted that same escalation in software, with a price tag on it. Hard shell problems to the expensive model, broken config and missing test results to the cheap one. Same quality of fix, smaller bill at the end of the month.

The stake is your inner loop. Every time an agent picks up a red pipeline, reads the logs, edits the config, triggers a run, and reads that too, somebody is getting billed. Running all of it on the expensive model, all week, is a line item.

That split never showed up.

What did show up is a way to measure this stuff without fooling myself. I connected CircleCI’s command line tools to both Claude Code and Cursor. I saved a red run, so nobody could grade a different pipeline than the one I picked. Then I refused to grade the model’s essay for a proposed fix. By “essay” I mean the model’s written diagnosis of what broke and what it would change, which is a completely different thing from whether CircleCI actually went green.

Agents are already good at the essay. The actual pipeline rerun is the score.

The essay is the menu. Green CI is the meal. I came to eat.

Hypothesis

I expected Grok 4.6 to struggle on the bash-flavored failures and hold its own everywhere else.

That was the prediction. I went in with a bias: if any model was going to struggle on shell, it would be Grok 4.6 once it had to drive a terminal. Type a command, read what came back, adjust, type the next one. Shell breakage is exactly that, over and over, so that’s where I thought a gap would show.

If the reputation held, the plan was already written. Sonnet 5 at effort: low takes the bash cases, Grok takes the rest, and I pay less without shipping worse fixes.

Second question, cheaper to ask and just as useful: does paying for Sonnet 5 at effort: high buy enough extra passes to justify the receipt?

Setup: six puzzles, three setups, one judge

The puzzles came from the CircleCI Training Koans, a public set of short exercises that break a pipeline on purpose so you can practice un-breaking it. I used six on a throwaway project, two per flavor of broken.

Three flavors went in. Broken config, where the pipeline file itself is wrong. Tests that don’t report, where the tests genuinely run and pass but CircleCI never receives the results, so the job looks fine and tells you nothing. And shell problems: bash logic, wait loops, cache commands, container images, services.

Then I froze them: one saved failed run per puzzle, so “whatever failed most recently” couldn’t sneak in and change the question halfway through.

Three setups, same prompt every time.

Those dollar figures are sticker rates for tokens, not the cost of a run. What each trial actually spent shows up in Results.

Changed: the setup. Held constant: the saved failed run, the prompt, the isolation, the spend ceiling.

Every trial started in a fresh clone. No resumed sessions. Sonnet never got to peek at what Grok had already worked out, and the two Sonnet effort settings didn’t share a session either. Each agent had to check its config change, then trigger a brand new CircleCI run on its own throwaway branch. Never main. Never the saved branch.

Now the grade, which is the part that maps to your actual day. Nobody merges the model’s write-up. You merge green CI. So a trial passes only if the pipeline that agent triggered finished green. On the tests-that-don’t-report puzzles, a green job with an empty test results panel (CircleCI’s Test Summary) is still a fail, because tests that pass without reporting anything back aren’t a fix. Afterward I re-checked every run against CircleCI’s own record, so the numbers below come from CircleCI and not from an agent’s summary of its own brilliance.

I capped spending too. I watched the expensive setup burn money during calibration, took its 95th percentile, and set the ceiling at $1.25 so a trial could go long without going stupid. Claude Code can hard-stop at that number. Cursor writes it down and keeps going.

Claude dollars are whatever Claude Code billed for the whole session, including the Haiku 4.5 helper that sometimes rides along. Grok dollars are tokens in and out times the published rate, not an invoice from Cursor.

One honest wrinkle: model and host are welded together here. Sonnet only runs on Claude Code, Grok 4.6 only on Cursor CLI. I can’t pull those apart, and I’d rather say it out loud than bury it in a footnote.

Eighteen isolated trials. Then a repeat round on the same six puzzles, then one genuinely messy red Playwright job. Three separate scoreboards, and averaging them would only make the numbers look tidier than the evidence is.

Results

The pilot, scored the day it ran.

All three greened both shell puzzles. The terminal gap I came hunting for never showed.

One puzzle was a config file so mangled that CircleCI rejected it before any job started. That’s why the original run was red: nothing ran at all. Fixing it meant repairing the file, triggering a new pipeline, and getting that new pipeline green. Only Grok’s new run went green. Sonnet at effort: low spent the entire $1.25 cap on that one puzzle and got cut off. Sonnet at effort: high failed it too.

Separately, Sonnet at high effort missed both puzzles where the tests ran but CircleCI never received the results. Zero for two.

I ran the same six puzzles a second time. Fresh start, no memory of the first try. Grok fixed all six again. Add both tries together and Grok is 12 for 12, Sonnet at low effort is 9 for 12, Sonnet at high effort is 7 for 12. Same homework twice. I wanted to know if Grok just got lucky on Monday. It didn’t.

Then the follow-up. I cloned a Playwright job that had been going red over and over, again a throwaway, not production. The bug was a locator: getByText matched both a heading and a button, so the test found two things where it wanted one and refused to guess.

All three models got that one pipeline green. One try each. Nobody deleted tests. Nobody touched .circleci/config.yml. Every one of them edited the Playwright test and switched to getByRole(”heading”, ...). Inside each green job, CircleCI’s test results panel listed six Playwright tests, and all six passed.

Grok was the cheapest and the slowest. High effort and low effort wrote the same fix. One case, three trials, which makes it an anecdote, and I’m labeling it one.

TL;DR

I didn’t get a routing policy. Two puzzles per flavor of broken is nowhere near enough to write a company rule about where your model budget goes. Grok held the shell cases, too. And paying for effort: high didn’t buy extra passes. On these puzzles it bought fewer.

What I did get is a test that refuses to lie to me. Most evals grade what the model wrote. This one waits for CircleCI to finish, then takes the verdict from the run itself. Execution-graded, meaning CircleCI’s pass or fail is the grade, not a rubric and not the model’s own confidence.

If you’re wiring agents into CI, that distinction is the entire ballgame.

Cheap can win when execution is the judge. Twenty-two cents a successful fix against $1.11 isn’t a brand story. It’s a 5x token bill, on this lab set, every time an agent tries to get a red pipeline green. I’m not going to invent your volume. A team of ten, five of those loops a week, is about $45 extra to pay for high effort instead of Grok. That’s the unit. Not a finance-team emergency. Still a 5x.

I evaluate models for a living. Then I have to evaluate the eval.

Scoring a write-up is asking somebody how they feel. Scoring a CI pipeline rerun is taking their actual blood pressure.

RalphCI already showed that green on your laptop isn’t green in CI. This one asks the ruder follow-up: hand an agent the logs, the CircleCI tools, and one saved red run, and can it actually fix the pipeline? On this lab set, yes. Often. The expensive knob wasn’t the reason.

I’m not saying always use Grok instead of Sonnet. Hosts behave differently, Cursor can’t enforce a dollar cap, and these were clean training puzzles plus one Playwright locator.

I’m saying this: if your agent loop can’t trigger a rerun and read the test results panel, you’re grading the menu instead of the meal.

Yesterday, we published our Q2 Pulse report, a follow-up to our annual State of Software Delivery that digs into a question engineering leaders are running into everywhere: why do some teams turn AI-generated code into shipped software while others watch feature branches pile up?

AI has made writing code cheaper and faster than ever. Every team we look at is generating more of it than they were a year ago. But the data keeps showing the same split: code volume is up almost everywhere. Shipped changes are not.

The bottleneck is validation. A change goes out, a pipeline takes several minutes to run, something fails, and the cycle restarts before anything merges into main. Each round trip drains both engineering time and delivery budget. In our previous issue, we modeled how a 50-developer team can spend close to $900,000 a year on avoidable validation cycles, driven by repeat CI runs and the tokens agents spend reloading context after slow feedback breaks the flow.

The most productive teams are pulling away because they ship in fewer cycles. That lifts individual and team velocity while driving down the cost of each change. In our Q2 Pulse report, we went behind the scenes with the most productive organizations on CircleCI to isolate the habits that help them break the delivery bottleneck without breaking the bank. Below, we’ll share three of them.

The teams setting the pace

The cohort we studied includes 20 anonymized organizations with the highest main-branch throughput on our platform. They span sizes, regions, and industries, with a concentration in security and developer tooling and additional representation from fintech, e-commerce, logistics, and energy.

As a group, they run an average of 2,165 production-branch workflows every day, a 72% jump from a year ago. That builds on the AI-driven volume surge we flagged in our Q1 report, when average daily workflow runs across CircleCI were up 59% year over year. The new finding is sharper: among the teams pushing hardest, increased activity is translating into dramatically higher production-branch throughput.

The gap between these teams and everyone else is widening. Across the platform, the most productive teams now ship roughly 9x the validated code of a typical team, up from 8x just a quarter ago. Their operational advantage comes down to three distinct habits.

Habit 1: Every engineer ships more finished work

The most productive teams get more production-ready work through the system per contributor. We measure this as main-branch workflows per contributor per day: a practical signal for how often validated work is reaching the production branch.

On a typical team, that comes out to about 1 main-branch workflow per contributor per day. Among high performers, it’s closer to 3. Among the 20 most productive teams, it’s around 12. That puts them at more than 10x the output of a typical team. It also shows how quickly the bar is rising: this same set of elite teams has nearly doubled per-contributor throughput in just one year.

High per-contributor throughput is the natural result of a delivery system built around frequent integration. Branches stay short-lived, validation runs continuously, and changes do not sit for days waiting on review or late-stage cleanup.

Frequent integration matters even more with AI in the loop. As more code enters the system, the teams that pull ahead are the ones that can move changes quickly from generation to validation to merge.

Habit 2: Validation runs beyond the manual push

The mechanics behind high volume become clear when looking at how top teams trigger validation. Engineers are not manually pushing code every single time they need to find out whether something works.

Most teams only run CI when someone pushes code, and the median team triggers nearly 100% of its pipelines that way. The most productive teams do not:

• Direct pushes: Account for only about 68% of their triggers.

• Automated triggers: The remaining 32% comes from scheduled runs, API calls, dependency-update checks, and post-deploy verification.

Their CI keeps validating the codebase and deployment lifecycle even after devs have closed their laptops.

A broader trigger mix gives teams more places to catch problems, from dependency health to release verification. Done well, it can also keep push-driven feedback focused on the change a developer is trying to merge, instead of mixing that signal with unrelated failures that could have been caught elsewhere.

Habit 3: Fewer validation cycles per shipped change

Throughput and automation drive volume. Efficiency keeps that volume affordable.

We track efficiency with a metric called Merge Efficiency Ratio, or MER: the number of feature-branch workflow runs for every workflow that runs on main. A high MER means work takes more rounds of pre-merge validation before it’s ready.

Every extra feature-branch cycle means more compute spent validating the same change. In an agentic workflow, it can also mean more tokens spent reloading context and generating fixes each time a slow run reports back. A team can ship a lot and still bleed money if every change takes five or six cycles to ship. The most productive teams keep cycles low, so their higher volume moves into production without a runaway bill behind it.

Less waiting, more shipping

Higher output per engineer, validation beyond the push, and fewer cycles per change all point to the same operating principle: the best teams reduce the distance between writing code, validating it, and merging it.

That is how more code turns into more shipped software. Feedback stays fast, branches stay short, and changes reach production instead of piling up. Just as important, the cost per change goes down instead of up.

The full Q2 Pulse report lays out the cohort behind the numbers, the implementation patterns behind each habit, and what it costs to operate at this level.

Read the Q2 Pulse report →

The report focuses on the operating model behind high-throughput teams. Chunk is one practical way to start applying the same principle: move routine validation out of the slow CI loop and into the moment code is being written.

Chunk sidecars run validation in the inner loop, right alongside your agent, so lint failures, broken tests, and syntax errors get caught in seconds while the context is still warm, instead of after a five-minute round trip through CI. In our own testing, microbuilds delivered feedback with up to 30x less compute usage than a full pipeline run and about 5x fewer tokens.

Chunk sidecars are available now, free on every CircleCI plan. One command gets you started: brew install CircleCI-Public/circleci/chunk

Read the five-minute setup guide →

Earlier this year, the conversation in software was all about maximizing agent output. Tokenmaxxing. Jensen Huang telling engineers to spend $250K a year on tokens. Every benchmark was about throughput: how much code could your AI generate per hour?

That mood has shifted fast.

Uber reportedly used its entire 2026 AI coding tools budget in four months. Microsoft started pulling back internal Claude Code licenses. Amazon just scrapped an internal AI leaderboard after employees started running up token costs to climb the rankings instead of shipping useful work. Across the industry, orgs are staring at massive AI bills and wondering where the ROI went.

In our recent research, we found a big part of the answer: teams are generating changes faster than ever but struggling to validate and ship them at the same pace. The delta between code written and code in production has never been wider, and it carries a hefty price tag.

In this issue, we’re digging into why teams are paying more than they should to move code into production. We’ll preview some new findings from an upcoming update to our State of Software Delivery report, show how moving validation into the inner loop of local, agent-driven development can dramatically reduce delivery time and cost, and give you practical steps to get started.

Merge efficiency (and why it matters)

One of the most interesting findings from the recent State of Software Delivery was that median throughput increased 15% on feature branches but declined 7% on main branches. AI makes writing code trivial, but it adds complexity downstream when those changes hit the shared codebase.

We wanted to dig into what that burden actually looks like for a typical team. How many cycles does it take to get a piece of code into production?

To find out, we looked at the ratio of feature branch runs to main branch runs across CircleCI in March 2026. This gives us an imperfect but directionally useful proxy for the total CI overhead required to ship a single change.

Here’s what we found:

• For the median org, every main branch workflow is accompanied by 3.9 workflow runs on a feature branch. At the mean, that number rises to 8.6. At the p95 level, it rises to 23.7.

• Using main branch workflows as a rough proxy for a merged change, that means every production deploy requires 5+ total CI runs (~4 on the feature branch and 1 on main) to complete.

• Main branch workflows also fail 20-30% of the time, adding at least one more cycle when they do.

In a human-paced development context, these numbers wouldn’t be terrible. A developer pushes a change, waits a few minutes for feedback, and iterates through a couple of build-test-fix cycles before merging to main and wrapping up the workday. That’s how you get to the canonical advice that “everyone pushes to main daily.”

In an agentic world, merge efficiency matters a lot more. When full CI runs take 5-10 minutes to deliver feedback, the agent loses context. Fixing the failure means starting a new cycle: reloading context, re-examining the change, potentially redoing work that was already completed.

That not only derails productivity but also creates massive cost inefficiencies. At 5+ CI runs for every shipped change, the math gets dire quickly.

Where costs pile up

Let’s make this concrete.

Imagine an AI agent is tasked with adding a new feature to a medium-sized codebase. The full repository context, including system instructions, tool definitions, and source files, comes to about 200,000 tokens.

Modern frontier models use prompt caching to make reading massive contexts affordable. The first time the agent reads your codebase, it’s a “cold write” at a premium price. On Opus 4.7, the cost is $6.25 per million tokens. On subsequent steps, the model reads from a warm cache at a 90% discount: just $0.50 per million tokens.

But (as usual) there’s a catch: prompt caches expire after about 5 minutes of inactivity.

Here’s how that plays out in a typical delivery workflow:

1. The agent loads the codebase (200,000 tokens) and generates the code change. API cost: ~$1.28 (plus output tokens).

2. The agent pushes the change, triggering a CI pipeline. The CI platform queues the job, spins up a fresh VM, pulls dependencies, compiles the project, and runs the full test suite. The pipeline consumes about 5 minutes of billable compute across its parallel jobs. But the total round trip from push to feedback, including queue time and provisioning, stretches well past the 5-minute mark.

3. The prompt cache goes cold. By the time CI reports back, the cache window has closed. The cache is wiped.

4. The build fails on a minor syntax error. The agent needs to fix it, but the cache is gone. It pays the full cold-write price to reload the codebase, and the context has grown: the previous attempt’s output and the CI build logs have pushed it from 200,000 to roughly 230,000 tokens. API cost: ~$1.44. Then the agent regenerates the fix from scratch, burning another round of output tokens at $25 per million.

5. The agent pushes the fix. Your CI provider spins up another fresh environment and runs the full suite again. Another 5 minutes of billable compute.

The merge efficiency data shows it takes an average of 5+ total CI runs to ship a single change. And with each cycle, the context grows: the agent retains the full session history, so each retry passes the original codebase plus every previous attempt along with its CI build logs. By run 5, input context can double.

In this scenario, input tokens that should cost under $2 with a warm cache balloon to over $13 across 5 cold reloads, purely because the feedback loop is slower than the agent’s cache. Add output token rework at $25 per million and $1-3 in CI compute, and the total cost per change runs to around $25.

Now imagine this at scale. A 50-developer team at agentic pace ships about 3,000 changes a month (roughly 3 per developer per working day). At ~5 CI runs per change, that’s 15,000 pipeline runs and nearly $1 million per year in token and compute costs. At 500K tokens (common for larger production codebases), it crosses $1.5 million. And with the continued increases in change volume that our data shows, those numbers climb every quarter.

Most of that spend is going to cycles that shouldn’t exist.

Validation in the inner loop

Scenarios like this play out because all of the validation is happening in the outer loop: CI pipelines, shared infrastructure, billable compute. CI is the right place to catch integration failures, security issues, and anything that requires a full environment. But when a syntax error or a failing unit test has to round-trip through a full pipeline run before your agent gets the signal, you can quickly accumulate a ton of unnecessary costs.

What if those cheap failures could be caught earlier, before they ever reach CI? The way to do that is to move validation into the inner loop, where agents iterate locally before the push.

In a healthy system, the inner loop handles the basic checks so the outer loop only runs on code that’s ready for deeper validation. Without a quality gate before the push, your outer loop becomes a bottleneck: flooded with changes, catching failures that range from trivial to critical, burning compute and tokens on every cycle.

The path forward is to raise the quality bar in the inner loop. Validate before the push. Catch the lint failures, the broken tests, the syntax errors while the agent still has context and the fix costs almost nothing.

Introducing Chunk sidecars

Moving validation into the inner loop means having an environment that can run real checks fast enough for an agent to act on the results. To get there, we recently released Chunk sidecars: sandbox environments spun up by the Chunk CLI that mirror your CI stack.

Rather than waiting for a full pipeline run to surface a failing unit test or a syntax error, the sidecar runs the validation checks you’ve configured: linting, unit tests, build validation, or whatever your stack requires. Failure feedback comes back to the agent while it still has context. The agent fixes the issue and the hook fires again, repeating until the checks pass and the change is ready to push.

This keeps the feedback loop tight enough that agents never lose context between a failure and its fix. To measure what that’s actually worth, we took real failures from the chunk-cli CI pipeline and ran them two ways: through CI as normal, and through Chunk sidecars using microbuilds. We measured compute time and token consumption across four pipelines with warm snapshots.

Here’s how those numbers affect the costs we saw in the scenario introduced earlier:

• Input tokens: $13 → under $2. The 27-second feedback time keeps the agent inside the 5-minute cache window. Five cold reloads become one cold load plus four warm cache reads.

• Output tokens: 5 full regenerations → targeted fixes. The agent gets a focused failure signal instead of a full CI log dump, so it applies a small fix instead of rebuilding context and regenerating the entire change.

• CI compute: 5+ runs → 1–2. Fifteen to twenty minutes of billable compute per change drops to a single pipeline run on code that has already passed its checks.

The change that was costing ~$25 in tokens and compute drops to around $6. For the 50-person team from our model, that takes annual token and compute costs from approximately $900k to around $200k. Total savings for a 50-person team: $700k or more.

Stop wasting your delivery dollars

Concerns over runaway AI spend are volume problems at their core. When you scale agentic workflows without changing the feedback architecture underneath them, costs accumulate with every additional change.

Teams who have closed this loop are already separating from the pack, and that difference is widening every quarter.

If you want more return on every token and every pipeline run, look at where your validation is actually catching failures today. If most of them are happening in CI, after a push, there’s a better way.

Chunk sidecars are available now on all CircleCI plans, including free. Install the Chunk CLI with brew install CircleCI-Public/circleci/chunk, run chunk init in your project, and start a sidecar session from your agent. The setup takes minutes, and the savings start on the first build.

Subscribe now

You can tell a lot about a developer from the tools they use. Editor, OS, language, framework. Everyone has a side, and everyone is a little suspicious of the other side’s choices. Version control is no exception. Ask a GitHub team and a GitLab team how they ship software, and you will get two pretty different descriptions of what normal looks like.

But do these preferences (and their downstream impacts) actually show up in delivery metrics? Each VCS attracts a certain kind of team, and each platform builds features to reinforce those tendencies. If teams really do self-select based on their priorities, then those priorities should show up in the delivery data.

In this issue of Confident Commit, we break down data from our most recent State of Software Delivery by code hosting platform to find out.

The evolving source of truth

Git has been the foundation of collaborative software development since Linus Torvalds created it in 2005 to manage the Linux kernel. It quickly became the industry standard, displacing older centralized systems like Subversion and CVS.

Hosting platforms arrived shortly after to enable team collaboration at scale. GitHub launched in 2008, Bitbucket the same year (acquired by Atlassian in 2010), and GitLab in 2011. Between them, these three cover the vast majority of professional engineering teams today.

Despite sharing the same underlying technology, these platforms evolved to serve different kinds of teams and workflows, making unique design choices along the way:

• GitHub is built around pull-request-driven social coding, making it the natural home for high-velocity, branch-heavy workflows.

• Bitbucket is tightly coupled to Jira and the Atlassian suite, attracting teams that prioritize rigorous change management and cross-functional visibility.

• GitLab serves as an integrated, governance-first platform appealing to teams that require strict, centralized control over security and compliance.

Given those philosophies, you would expect GitHub teams to iterate fastest, while Bitbucket teams move with more deliberate enterprise ceremony. GitLab teams, often working in complex and regulated environments, should land somewhere in between.

CircleCI integrates with all three platforms, so we can test those expectations directly to see if they hold up. Let’s take a look at how the hosting platforms stack up across the four key software delivery metrics.

Duration: how long is the feedback loop?

Duration measures the time from when a pipeline starts to when it finishes and determines how long a developer waits to find out whether a change passed or failed. Teams optimizing for rapid iteration want this number as low as possible. Teams running deeper validation, with more extensive tests and approval gates, accept longer durations in exchange for higher confidence in what they ship.

Workflow duration is generally consistent across all three platforms. At the median, the three cohorts are within a few seconds of each other: GitHub at 2m 58s, GitLab at 3m 4s, and Bitbucket at 2m 58s.

GitLab teams running slightly longer workflows is consistent with the hypothesis that these teams run governance-heavy pipelines, but the overall story here is similarity rather than divergence. Even at p90, pipeline durations are closely clustered at 493, 489, and 484 minutes respectively.

Throughput: how much is moving through the pipeline?

Throughput is the number of workflows a team runs per day. It can reveal how actively a team is pushing changes, and is often used as a directional indicator of team productivity.

Looking across all branches, Bitbucket teams run 1.71 workflows per day, while GitHub teams run 2.25 and GitLab teams 2.55.

But the branch-level breakdown is where the numbers get interesting. Throughput on feature branches tells a story about iteration speed: how often developers are committing, running CI, and testing changes before they are ready to merge. On the main branch, throughput is a proxy for how often code moves toward production. Given what we know about each platform’s user base, you would expect GitHub teams, as the most iteration-focused cohort, to lead on throughput.

As expected, GitHub is the runaway leader on feature branches at 3.39 workflows per day versus 2.52 for GitLab and 1.71 for BitBucket. This makes sense considering GitHub’s user base skews toward smaller teams, open-source projects, and startups that prioritize iteration speed over formal approval processes.

But on the main branch, GitLab teams run the most workflows at 2 per day compared to 1.68 for GitHub and 1 for BitBucket. While this runs contrary to what we might first expect, the finding is consistent with previous research we’ve done showing compliance-heavy teams shipping more frequently but hitting friction in other areas of the SDLC (more on that below).

Success rate: how healthy is the path to production?

Success rate is the percentage of workflow runs that complete successfully. The default branch is where this metric matters most. Teams with efficient delivery processes catch errors locally or on feature branches before merging, so a high success rate on main is a signal that the upstream process is working.

Average success rate on main-branch workflows is highest for Bitbucket teams at 73.7% and falls to 70.6% for GitHub and 61.3% for GitLab.

Bitbucket teams running the fewest pipelines but passing more of them is consistent with a stability-leaning culture where more review happens before a merge. GitHub sits in the middle, reflecting a culture comfortable operating at tempo while keeping things green. GitLab’s lower success rate likely reflects the complexity of the environments these teams operate in, something the recovery data makes even clearer.

Time to recovery: how long until you’re back to green?

Time to recovery measures resilience: how long it takes to fix a broken build and get back to a successful state.

Across all branches, GitHub teams recover in 72 minutes, Bitbucket in 80, and GitLab in 124.

On the main branch specifically, those differences are magnified: GitHub teams recover in under an hour, BitBucket teams take about 80 minutes, and GitLab teams require nearly 8 hours to get back to green.

This may seem like a large delta. However, these results are consistent with what we’ve seen previously regarding high complexity, compliance-heavy workflows. When things go sideways on main in a complex environment, the need for more extensive root cause analysis, documentation, and verification before merging a fix can add significant time to the recovery process.

Of all four metrics, time to recovery is where the self-selection pattern shows up most dramatically. The teams who chose the governance-first platform are the ones paying the governance tax on recovery.

What did we learn?

While impactful, tool choices alone are never the determining factor for how a team performs. Most of that comes down to experience, engineering vision, processes, and how you use the tools at your disposal. But it’s also true that teams gravitate toward the tools that map to their way of doing things, and over time, those choices become part of a team’s identity.

Each cohort’s delivery performance aligns closely with what their platform was designed to support. Teams self-select into the tool that matches their priorities, and those same priorities show up in the data.

• GitHub teams prioritize velocity: more feature branch workflows, faster iteration, quicker recovery. This tracks with GitHub’s pull-request-driven culture where branching is cheap, experimentation is encouraged, and speed matters more than gating every change.

• Bitbucket teams prioritize stability: fewer workflows, highest success rate on main. The Atlassian ecosystem emphasizes structured workflows and cross-functional coordination. Teams catch more issues before merge, but that deliberation costs iteration tempo.

• GitLab teams prioritize governance: most main-branch workflows, longest recovery times. This reflects GitLab’s governance-first design. Teams in regulated environments often ship frequently despite their additional constraints, but when things break, the overhead of meeting compliance requirements extends recovery.

Each platform attracts teams with different priorities, and those priorities shape delivery outcomes. So while your VCS doesn’t determine your delivery destiny, it does reflect the outcomes you’re optimizing for.

Go deeper

VCS choice is one way to see how different choices and team characteristics shape delivery performance. The 2026 State of Software Delivery takes a deeper look at what drives these differences, cutting the same metrics by team size, industry, and geography to show how delivery patterns shift depending on who you are, where you are, and what you are building.

Download the report to see the full analysis, or explore the data and benchmark your team against similar organizations in the interactive Software Delivery Data Explorer. The more you understand about how teams like yours ship, the easier it is to know what to optimize next.

Subscribe now

The 2026 State of Software Delivery found the largest year-over-year increase in throughput ever recorded. Workflow volume is up 59%, but on the flip side, main branch activity is falling. Teams are generating a lot more code, but most are struggling to turn that activity into shipped software.

How the AI delivery bottleneck affects your team has as much to do with your specific business requirements as it does your ability to integrate AI into your workflows. Breaking down the delivery data by industry, then, can give us useful insights into different ways the problem shows up and what teams like yours are doing to solve it.

In this issue of The Confident Commit, we’re looking at three industries that stand out in this year’s data, each for unique reasons: civil engineering, utilities, and retail. We’ll break down what each one is doing well, where they’re still falling short, and what their results can tell you about closing the gap between code generation and code delivery on your own team.

Civil engineering

Civil engineering teams validate more changes per day than any other industry. The typical project triggers 9.9 workflows every day, nearly 6x the median result across the entire dataset. On feature branches, where developers build and test new work before it’s ready for production, activity is 6.8 workflows per day.

That high volume likely reflects how civil engineering projects are structured: multiple parallel workstreams across structural, mechanical, electrical, and site systems, each being developed simultaneously. But the picture changes when you look at the main branch, where code actually ships to production. Throughput there drops to 1.8, right in line with what most teams run.

That’s a clear illustration of the AI bottleneck in action. These teams are generating massive amounts of code, but they are paying a heavy integration tax. Their complex, interconnected systems have to be validated against each other before anything can ship. That takes time, and it doesn’t scale linearly with volume.

Given the quality and compliance constraints in this vertical, a bottleneck at the integration stage makes sense. But it also means that despite having the highest overall throughput of any industry, civil engineering’s internal velocity rarely translates into faster project delivery

Utilities

So what does it look like when a team actually breaks through the production bottleneck?

Utilities teams run 7.6 workflows per day on feature branches and 7.8 on main. Both numbers are about 4x the current global median and represent a more than 2x year-over-year increase for this vertical. But what’s especially notable is that activity on main is keeping pace with feature branches. Where civil engineering teams are increasing code volume but struggling to integrate it, utilities teams are writing and shipping more code.

One possible explanation is business pressure. Utilities are critical infrastructure. Grid management, outage response, and regulatory reporting can’t wait. That urgency may push teams to ship smaller, more isolated changes that integrate cleanly rather than letting work pile up on feature branches. Civil engineering projects operate on longer timelines with less immediate consequence for any individual change, which may explain why parallel workstreams pile up and create integration complexity at main.

Whatever the reason, the data shows that utilities teams are shipping code as fast as they write it. But that’s not the complete story.

Utilities teams take approximately 400 minutes to recover from a failed build on main, compared to a global median of just 59 minutes. When something fails in a high-stakes pipeline, formal investigation steps take priority over speed regardless of how fast your CI system can rerun a build. But for a critical service that people depend on, almost 7 hours of blocked deployments has real consequences.

Retail

Like utilities, retail teams depend on 24/7 availability for their customers. While downtime for a retailer may not carry the same consequences as a utility outage, the impact on revenue and customer sentiment can be massive. That pressure has pushed retailers to optimize for uptime and fast recovery.

The typical retail team recovers from a failed build on main in 20 minutes. Most non-retail teams take 60 minutes or more.

Not only does retail recover the fastest, but it’s also among the top performers on success rates. Retail workflows running on main succeed 85.2% of the time, placing them fifth among all industries. With these two metrics combined, retail is the strongest all-around performer on the metrics that matter most for reliability.

Banking tells a similar story. The typical banking team recovers from a failed build in 41 minutes, second only to retail, and maintains an 80.1% success rate on main. What retail and banking have in common is that the business cost of downtime makes recovery speed an organizational priority, not just an engineering goal. For these teams, MTTR is a financial metric.

What this means for your team

Each of these industries has optimized for the thing their business pressure demands and accepted a tradeoff elsewhere. Civil engineering iterates at massive volume but ships to production at an average pace. Utilities ships to production at a very high rate, but recovery takes nearly seven hours when something breaks. Retail recovers faster than any other industry because the revenue impact of downtime made it an organizational priority.

The question for your team is which of these dimensions is actually constraining you right now. If your feature branches are active but production is flat, you have a throughput-to-production problem, the same one civil engineering is paying for at scale. If your success rate is below 80%, you’re burning engineering time and tokens on failures that stack up with every change you generate. If recovery takes more than an hour, every failure blocks not just the build that broke but everything behind it in the queue.

This is what we’re building autonomous validation to solve: faster feedback, deeper context across your build history, and the ability to automatically resolve common failures before they turn into 400-minute recovery stalls or integration bottlenecks that keep code from reaching customers.

See where your team stands

The full 2026 State of Software Delivery breaks down performance by industry, region, and team size.

Want to benchmark your own metrics against these industries? The Software Delivery Data Explorer lets you compare your throughput, success rate, duration, and recovery time against teams in your specific segment.

Ready to start closing the gap between generation and delivery? CircleCI is free to get started.

Subscribe now

In our last issue, we shared a preview of data from our upcoming 2026 State of Software Delivery showing that the promised AI productivity boom isn’t all hype. Throughput across the CircleCI platform increased 59% year-over-year, by far the largest productivity jump we’ve ever recorded and a clear indication that AI-assisted coding is driving massive increases in change volume.

But the gains weren’t evenly distributed. The top 5% of teams nearly doubled their output, while the median team improved by just 4%.

Since then, we’ve released the full report. Alongside it, we launched the CircleCI Software Delivery Data Explorer, an interactive tool built on tens of millions of real CircleCI workflows that lets you benchmark your team’s throughput, duration, success rate, and MTTR against peers in your industry, region, or team size.

In this issue, we’re digging deeper into the performance divide we called out last time. What’s actually driving this separation, and what can you do about it?

The branch data tells the real story

To understand what’s causing the AI performance gap, it helps to know a little about how modern software gets built.

When a developer starts on something new, they typically don’t write code directly into the main codebase. They create a separate copy called a branch, a safe workspace where they can build and test without affecting anything in production or blocking their team from shipping other changes. When the work is done and the code passes automated checks, it gets reviewed and merged back into that main codebase, where it can be deployed to users.

Those two events mean very different things. Activity on feature branches tells you developers are working. Activity on the main branch tells you software is shipping. When you split the throughput data along those lines, the picture changes considerably.

For the top 5% of teams, throughput increased 26% on the main branch and 85% on feature branches between 2024 and 2025. Why is the jump so much bigger on feature branches? Because AI is particularly well-suited to the work that happens there: working out complex problems quickly, spinning up prototypes, and iterating quickly until the approach is right.

But the most important story is the increase in activity on main. Top performers are generating and shipping significantly more than they were just one year ago. That’s what it looks like when AI acceleration runs all the way through the delivery pipeline.

For everyone else, the picture is different. Teams in the top 10% saw feature branch throughput rise nearly 50%, but main branch activity was essentially flat at 1%.

And at the median, feature branch activity was up 15%, while main branch throughput actually declined by 7%.

In other words, the typical software team got marginally better at generating changes last year, but they got worse at getting those changes into customers’ hands.

For any business leader, a regression in the organization’s ability to deliver value is going to set off alarm bells. So what’s behind it? The short answer is that AI has made it significantly easier to write code, but the process of integrating that code into a production system involves a different set of challenges. And those challenges get harder, not easier, as the volume of incoming changes goes up.

Why AI-generated code is harder to land

It’s easy to say “integration is the bottleneck” and leave it there. But there are several distinct places between a developer finishing a piece of code and that code reaching production where things can slow down or break, and AI is putting pressure on all of them.

The first place AI generated changes run into trouble is code review. Most teams still require proposed changes to pass a manual review before they can be merged, and AI is making that queue harder to manage on two dimensions: volume and complexity.

We’ve already shown that AI is driving big increases in code activity for teams across the board. Those changes are not only happening more frequently, they’re also larger, denser, and touch more of the codebase than a human would in the same pass. Research by Faros AI and others has shown that AI adoption increases the size of pull requests by more than 150% on average.

What this means is that senior engineers tasked with reviewing AI-generated pull requests are facing an ever-growing review queue that is more difficult than ever to process. And they’re reporting all kinds of negative effects from it, from feelings of overwork and burnout to the tendency to take shortcuts and skim potentially risky code.

While the former is a serious risk to employee wellbeing and business continuity, only the latter is measurable in delivery data. And the impact is hard to ignore. Main branch success rates fell to 70.8% this year, the lowest in over five years and well below the 90% benchmark that indicates a healthy pipeline.

Roughly 3 in 10 attempts to merge changes into production are now failing, and when the main branch breaks, nothing else can ship until it’s fixed.

That brings us to a third place AI code tends to get stuck: fixing it when it breaks. Recovery times for failed builds have been climbing since AI-assisted development went mainstream in 2022. This year, the typical team takes 72 minutes to get back to green, a 13% increase from the previous year.

Recovery times have been getting longer as teams incorporate more AI-generated code into their workflows. The same challenges that slow down manual review – PR size and complexity – also make failures harder to diagnose. Larger changes create more places for bugs to hide. And when something breaks in code a model generated, the developer debugging it often has little intuition for why it was written the way it was or what it might have affected downstream.

CI/CD has to keep up

In a recent newsletter, we made the case for autonomous validation as the next evolution of CI/CD. The data in this report is the clearest argument yet for why that evolution is necessary.

The AI choke points we described are all symptoms of the same underlying weakness: delivery infrastructure built for a world where humans produced code at a human pace. That’s why we’re evolving CircleCI to deliver

• Faster feedback so issues surface while context is still fresh, not after developers have moved on to the next AI-generated change

• Deeper context across your build history, dependencies, configs, and policies to surface risk in AI-generated changes that a diff alone can’t reveal

• Agentic capabilities that can resolve common, repeatable failures autonomously, without pulling an engineer off their work every time a flaky test or dependency conflict blocks the pipeline

When delivery pipelines can’t keep up, AI code is more likely to stall at review, break on merge, and burn engineering time on recovery. By rethinking CI/CD to operate at the same speed and scale as AI development, we’re taking concrete steps to remove the sources of friction that keep most teams from turning AI-generated code into shipped software.

Where do you stand?

If your team is generating more code than ever but struggling to get it across the finish line, you’re not alone. That’s the reality for the majority of teams in this year’s data.

You can learn more in the 2026 State of Software Delivery. It breaks down exactly where things are slowing down and what high performers are doing differently.

Download the report →

Want to see where your own numbers land? The Data Explorer lets you benchmark your delivery metrics against teams in your industry, region, and size band.

Benchmark your team in the Data Explorer →

And when you’re ready to turn all that AI-generated code into shipped software, CircleCI is free to get started.

Start building for free →

Subscribe now

If AI can write the code, why train someone when the model can just do it?

Rob Zuber has heard this take. He thinks it’s wrong, but not in the hand-wavy “humans will always matter” sense. He thinks it’s wrong because it misidentifies what the job actually is, and therefore misidentifies what we’d be losing.

In this episode of Confident Commit, Rob sat down with two colleagues from CircleCI’s software factory team: Michael Webster, a software engineer with more than a decade in the industry, and Hanabel Mengistu, who joined CircleCI less than a year ago straight out of college. The result is one of the more honest conversations about the current state of software development: what experienced engineers are struggling to let go of, what new engineers are struggling to find, and where those two things collide.

1. The tools changed faster than the job descriptions did

2025 brought a lot of change, fast.

Webster framed the inflection point precisely: “Around the end of last year, we kinda had that Sonnet 4 / Opus 4 moment where it’s like, hey, something’s happened with these models. They’re able to do a lot more.”

The instinct, for a team that wanted to push hard on AI-assisted development, was to try running multiple agents in parallel and see what broke. The answer came fast.

“I think within the first forty-five minutes to an hour of starting, a lot of our instincts immediately started breaking down. The merge conflicts, the trying to stack them, the manual review overhead. Sub two hours, we were like, yeah, we’re gonna have to figure some things out here. Code generation time outpacing CI build times, all of that was pretty eye-opening.”

Michael Webster, Principal Engineer, CircleCI

The problem wasn’t the AI. The problem was that the existing process, designed around the constraint of human coding speed, was suddenly the wrong process.

Rob’s framing: “We fundamentally changed our constraints. But as humans, we’re not good at massive leaps in approach. So we’re clinging to the processes that we have and trying to tweak them with this fundamentally new toolkit instead of asking: what was it we were trying to do, exactly?”

2. What experienced engineers are actually clinging to (and why it’s hard)

Webster put something into words that most senior engineers are circling around but haven’t quite said directly:

“What if the stuff that I like doing is the bottleneck? What if the stuff that has saved me production incidents and helped me build a career, what if that is now the bottleneck? That’s a really challenging thing to work through.”

Michael Webster, Principal Engineer, CircleCI

If your expertise is in a thing that’s now being commoditized, the rational move (reprioritize, go further up the stack) is emotionally costly in a way that’s hard to explain to someone who hasn’t been through it.

Webster’s working theory: go back further than you think you need to. Sharpen the intuition, not the implementation. He cited Fred Brooks’ “No Silver Bullet” and the distinction between essential and incidental complexity. AI is handling a lot of the incidental complexity that accumulated over decades. The essential complexity of building systems that solve real problems for real people has not gone anywhere.

“You’re just moving further up the stack,” Webster said, “and that’s a skill a lot of folks don’t have, because for a long time it was really hard to do those other things and now those things are easier.”

3. College prepared Hanabel for a job that barely exists anymore

Hanabel graduated about a year before joining CircleCI. She came up through a computer science program that treated AI as a form of academic dishonesty: sign at the bottom of every test saying you didn’t use it, even as everyone quietly used it.

The irony Rob raised: using AI would have been cheating. Using AI would have also prepared her for her actual job.

Like most fresh graduates, Hanabel found herself equipped with technical skills, but needing experience in everything around them.

“I feel like that’s me during all the meetings. I knew nothing about the process or the delivery side of things. I didn’t know what that would look like outside of college.”

Hanabel Mengistu, Software Engineer, CircleCI

Rob’s take is worth sitting with: “A lot of the day-to-day job is not actually about the structure of software. It’s: How do we decide what to work on? How do we communicate about what we’re working on? How do we decide the risk involved? How do we deal with breakage? That’s probably more than 50% of the job. And then we took the part we did focus on in school and said, don’t worry about that anymore.”

Subscribe now

4. Juniors aren’t being mentored, and it’s showing up as loneliness

One of the quieter observations in this episode: junior engineers are feeling isolated, and it’s not entirely their fault or anyone else’s fault specifically. The mentorship pipeline has a structural problem.

Hanabel described what she’s hearing from peers:

“A lot of my friends that are in the industry feel a little bit alone. People aren’t mentoring other juniors anymore because it’s something AI can do. Not really, but for small questions, you can just ask Claude. And asking AI is not always the most valuable way to learn.”

Hanabel Mengistu, Software Engineer, CircleCI

The deeper issue Rob named is a two-sided dynamic: senior engineers assume juniors can just ask the model, so they stop offering; junior engineers sense they shouldn’t bother asking, so they stop reaching out. But the value a good mentor provides isn’t just the answer to the question. It’s the context around the question, the adjacent things worth knowing, and the pattern recognition that says “you’re asking about X because you’re missing Y.”

5. The great leveling might actually help juniors more than it hurts them

The most optimistic thread in this conversation came from an unexpected place. Hanabel pushed back, gently, on the framing that the current moment is particularly hard for new engineers. Her read: it might actually be easier in one specific way.

“I feel like it’s comforting. It’s nice to be able to figure stuff out with other people and not feel like I’m the worst. I’ve been able to contribute so quickly. I wouldn’t be able to do all the things I’m contributing to if it weren’t for that.”

Hanabel Mengistu, Software Engineer, CircleCI

Claude Code went GA in June 2025. ChatGPT went GA in November 2022. Nobody has ten years of experience in any of this.

The experience gap that normally takes years to cross hasn’t had time to form in the places that matter most for this transition. For junior developers, this means that the playing field has flattened.

A curious generalist with no bad habits to break might actually have an edge over a deeply experienced engineer who has spent a decade optimizing for a set of constraints that no longer apply.

Subscribe to Confident Commit for more conversations about how software delivery is changing, from the people figuring it out in real time: confidentcommit.com

Rob sat down with Luca to talk about what high-performing teams are actually getting right with AI in 2026, why code review was always more questionable than we admitted, and how Luca built a three-part framework: guides, gates, and guards, to keep AI agents honest in production code.

1. 2026 is the year of reckoning, not adoption

Last year, AI tooling spread fast. Most teams added something. Some added everything. This year, the question has shifted from “are we using AI?” to “what are we actually getting for it?”

Luca has been watching this across a wide range of teams, and the diversity is striking. There’s no consensus, no single playbook, and the gap between teams who are extracting real value and teams who are just spending tokens keeps widening.

“2025 was more like the year of adoption. Eventually we got to: ‘AI is everywhere, 95 plus percent of engineers using AI’. And this year is more like okay, show me what you’re getting out of it, as the token spending keeps growing for everybody.”

Luca Rossi, Founder, Refactoring.fm

The teams pulling ahead are the ones who’ve gotten specific about where AI actually helps and where the friction still lives.

2. You have to know your SDLC before you can improve it

This sounds obvious. It isn’t. Luca’s observation is that the single biggest differentiator between the teams getting results and those who aren’t is whether engineering leaders actually have visibility into what’s happening in their delivery process. It’s about real awareness of where things slow down, where handoffs break, and where decisions stall.

“Just being aware of what’s going on in your developer process. Which means talking with your engineers, knowing where the problems are, where the bottlenecks are, having a good feedback loop where you can actually work on things that you know can be improved.”

Luca Rossi, Founder, Refactoring.fm

Rob noted that this insight has been true for 30 years. But the AI era has made it more consequential because improvements compound faster when you know where to aim them.

3. Guides, gates, and guards: a framework for trusting AI agents

Luca has been building Tolaria largely without writing code himself, directing AI agents through the full development workflow. That experiment forced him to get precise about trust. The result is a three-part framework he calls guides, gates, and guards.

Guides are the instructions agents follow: rules in the agents.md file about how code should be written, covering things like TDD, the Boy Scout rule, analytics instrumentation, and localization.

Gates are deterministic checks that run before anything gets committed. Because agents ignore the guides roughly 10% of the time, you need enforcement that doesn’t rely on the agent’s judgment. Luca uses local hooks to check code health, quality, and security and blocks commits that don’t clear a threshold.

“AI can happily ignore these rules every now and then, like ten percent of the time they just don’t care and do things how they like to do. And so you need deterministic gates to avoid [the fact that] bad things happen in production and get committed and pushed.”

Luca Rossi, Founder, Refactoring.fm

Guards are nightly or weekly audits that catch judgment calls and bigger-picture issues. Did the agent forget to write an architecture decision record? Have performance metrics degraded since the last release? Are there refactoring opportunities the agent couldn’t see at the task level?

Subscribe now

4. Velocity without visibility is just faster drift

One of the clearest tensions in the conversation: AI compresses the coding loop, but that doesn’t automatically speed up software delivery. It can just mean you’re drowning somewhere else.

Luca pointed to code reviews and product planning as the two places teams tend to hit the ceiling. Code gets written faster, but review queues get longer. Requirements get built faster, but upstream work to define what to build doesn’t scale with the throughput.

“We’re just getting faster at coding, but people are drowning either in code reviews or are too slow at creating new requirements or measuring what’s been done. So we have to take a holistic view about the whole thing.”

Luca Rossi, Founder, Refactoring.fm

The teams breaking through aren’t asking “how do I make each stage better?” They’re asking “what am I actually trying to get out of this process?” and redesigning from that question with the tools they now have.

5. Code review was already questionable. Now it’s untenable.

Luca has been skeptical of traditional code review for longer than AI has been part of the conversation. The asynchronous process was a time sink before agents started generating code at scale. Teams would fight to shave 10 minutes off CI times while PRs sat idle for 18 hours. And the assumption that humans reliably catch important issues by scanning unfamiliar code quickly has always been optimistic.

Rob agreed: We were never that excited about reviewing our colleagues’ code. We tolerated it because it was the best available option.

“Even if you don’t account for the quantity, it’s just a miserable experience. You don’t want to corner people into spending a lot of their time at work doing miserable things like reviewing AI code.”

Luca Rossi, Founder, Refactoring.fm

Now that AI agents can generate code far faster than any team can review it, the math is clearly broken. But the answer isn’t faster humans. It’s rethinking what “review” is actually trying to accomplish and finding better mechanisms for that. Deterministic gates, paired programming for high-stakes changes, and automated quality enforcement upstream before anything reaches a human.

6. Building for yourself is a cheat code with limits

Luca’s most consistent lens across everything he talked about, including the newsletter, the app, and the AI workflow, is that building something you personally need gives you a feedback loop most teams don’t have. You know the problem because you’re living it. You know when the solution works because you feel it.

He’s explicit that his experience building Tolaria is privileged. He’s working alone, with no legacy code, and can afford to experiment in ways that most real-world teams can’t. But the underlying principle transfers: the teams getting the most out of AI tools right now are the ones with enough self-knowledge to know what they’re actually trying to solve.

Confident Commit is CircleCI’s podcast for engineering leaders who want to understand how software delivery is actually changing. Subscribe at confidentcommit.com to get every episode plus companion posts like this one.

Laurie Voss has watched this pattern play out at scale. As co-founder of npm (the package registry that now serves hundreds of millions of JavaScript developers), a DevRel leader at Netlify, and now Head of Developer Relations at Arize, she has spent her career at the intersection of data and developer behavior. She knows how software communities learn, and she knows how they avoid learning.

The current moment in AI development, she argues, looks a lot like the early days of software testing: everyone knows tests exist, and most people are not writing them.

Rob Zuber, CTO at CircleCI, sat down with Laurie to talk about what “real evals” actually are, why the eval loop is going to change how software gets built, and whether we are anywhere close to agents building production software without humans in the loop.

1. Evals are just tests. Stop calling them anything else.

The first problem Laurie has with the word “evals” is the word itself. It is borrowed from the world of ML, and it lands with a thud in the ears of engineers who came up through web development, backend systems, or platform work.

“As soon as you say traces are logs and evals are tests, AI engineers get the story much better. You’re supposed to be writing tests.”

Laurie Voss, Head of DevRel, Arize

The terminology barrier is not trivial. If engineers hear “evals” as an ML research concept, it stays over there, in someone else’s domain. If they hear “tests,” the mental model clicks and the excuses disappear. The biggest competition for an evals company, Laurie points out, is not a rival product. It is engineers who just do not do any evals at all.

The implication is worth sitting with: a large fraction of AI applications going to production right now have no systematic validation. If you are running evals, you already have an advantage.

2. “Vibe-based testing” is why people hate AI.

There is a phrase that stuck: vibe-based testing. It’s what happens when a developer makes a change, runs a favorite query or two, feels good, and ships. It works fine for the happy path. It does nothing for the other 80% of inputs users will eventually throw at the application.

“A whole lot of software is getting all the way to production right now using nothing more than vibes. And it is showing up as really poor reliability. And it is making people hate AI because they interact with AI that’s busted all the time.”

Laurie Voss, Head of DevRel, Arize

This is not a new failure mode. Anyone who has watched engineers click-test a UI before a release will recognize the shape of it. The specific damage here is that broken AI applications have the potential to erode trust in AI as a category. The cost is the whole wave of adoption getting slower because people have been burned.

If you are building AI applications, the quality of your evals is now a competitive differentiator.

3. LLM-as-judge is the only way to scale, but it needs its own tuning.

Non-determinism is the real wrench. Traditional tests are essentially string matching: you put in an input, you expect an output, you check if it matches. With AI applications, the same input can produce a million different outputs, and a meaningful fraction of them will be correct. You cannot just check for a specific string.

For some cases, smarter string matching helps: check for “one hour ago” and “60 minutes ago” and other valid phrasings. But for anything requiring judgment, you need a judge. In production, at scale, that judge has to be another LLM.

The catch is that the judge LLM introduces its own layer of non-determinism. It has its own prompt, and that prompt needs to be tuned against real output to confirm it is actually measuring what you want it to measure, rather than producing good/bad labels at random.

If you are running evals in CI, where they need to execute reliably thousands of times a day, this is the architecture you are building toward: a test suite powered by an LLM that has itself been validated as a good judge.

Subscribe now

4. Regression evals vs. capability evals: two different jobs.

One of the most useful frames in the conversation was the distinction between regression evals and capability evals. They are not interchangeable, and conflating them leads to poorly designed test suites.

Regression evals are what most people think of first. They are close to traditional tests: can my application still do the things I already know it can do? They are the safety net you run continuously.

Capability evals are different in a fundamental way.

“Capability evals are a test I expect you to fail. This is a test where I expect you to score 20 percent and then do better the next time and climb a hill of capability.”

Laurie Voss, Head of DevRel, Arize

The combination opens up serious possibilities. Because evals powered by LLMs include an explanation field alongside a score, you can feed the failure explanations from 100 runs back into a coding agent and ask it to improve the software based on what went wrong. No human required. The eval loop drives the development loop. This has not been fully productionized yet, but the pieces exist.

5. Context engineering is the whole game.

The conversation shifted toward agentic coding and the question of how close we are to agents that can build production software end to end. Laurie’s answer was grounded in a single concept: context engineering.

Every capability question in AI development comes back to context. Can this LLM be a doctor? That requires a human lifetime of context, and it does not fit in a window. Can this LLM build a website? Websites are a narrow enough domain that Laurie thinks we are close to an agent harness that can handle it reliably. Can this LLM build production software in general? Software development as a domain is too large for any current window or graph structure to cover well enough to one-shot it.

Context graphs, one of the tools Arize is working on, are a form of context compression: a way of defining which topics connect to which other topics so an LLM can navigate a large domain without losing things out the bottom of its window.

The practical takeaway for engineering teams building on AI today: the investment in skills files, harnesses, and structured context is what determines whether your agent makes good decisions or system-level mistakes.

6. Software developers always solve their own problems first.

There is a pattern in software development that Laurie named plainly: the community perfects tooling for its own domain before solving anyone else’s. The reason there are so many frameworks is that developers made their own environments excellent before turning to enterprise software, which is still, by most measures, remarkably bad.

The same dynamic is happening now. AI coding tools are improving fast because developers are building them and using them on their own work. The patterns being developed in AI coding evals, harness design, and context engineering will eventually transfer to other domains. But that transfer comes later.

For teams building AI products now, the implication is to pay attention to what the software development domain is learning. The patterns being forged here will be the foundations for whatever comes next.

If this episode got you nodding your head, subscribe to Confident Commit for more conversations like this one.

Five months later, I am still coding the same game of Snake. Now 100% AFK, with much more confidence than when I started. Zero human supervision.

The first half of 2026 has been fun, but not in the arcade sense. It has been fun in the “I haven’t touched my keyboard in twenty minutes, and every check on the branch is green” sense. We kept building the same Snake game. Same 20x20 grid. Same arrow keys. Same TDD spec. Same retro aesthetic.

The game has barely changed, but the delivery loop has changed everything.

If you have been watching the AI agent hype cycle from a safe distance, here is the compressed version of what we learned by staying stubbornly on one benchmark: you can go AFK with rising confidence, greener pull requests, and faster feedback than we have ever measured. Not by trusting the agent more. By giving it better and quicker back-pressure at each layer of validation.

The benchmark nobody asked for (but everybody needed)

Snake is a toy. That is the point.

A toy spec is small enough to run ten times in a week. Small enough to isolate one variable at a time. Small enough that when something breaks, you can actually read the diff instead of drowning in a monorepo.

We gave agents the same task repeatedly: build a playable Snake game from scratch using test-driven development (TDD). Seven tasks. Canvas rendering. Collision detection. Score tracking. Push commits. Open a PR. Walk away.

Every run finished locally. Every agent said the tests passed. Every game worked on the laptop and was pretty fun to play.

January: everything is a Ralph loop

In January, Geoffrey Huntley published everything is a ralph loop. The mindset shift is not “use AI to type faster.” It is “program the loop.”

Allocate a goal. Run the loop. Watch the failures. Fix the failure domain so it never happens again. Repeat until done or until you hit CTRL+C and take the wheel back.

Basic Ralph loops can build software and generate PRs AFK. That part worked decently well (with some light human supervision).

The PR at the end might be green. It might be red. The agent does not inherently know which until something external tells it.

We were building Snake games in loops before we had a name for what we were doing.

The lesson from January: autonomy without validation is just faster guessing.

February: does the pipeline agree with local tests?

By February we had data, not vibes, on whether the pipeline agreed.

Our first study ran ten controlled Snake builds. Same spec. Same model scaffolding. One variable: five runs wired to CircleCI through RalphCI, five runs without.

All ten agents completed every task. All ten passed local tests. All ten declared victory.

Eighty percent of the no-CI runs shipped code the pipeline rejected. The failure was mundane: single quotes where ESLint wanted double quotes. Local tests do not check lint rules. CI does. The gap between “works on my machine” and “works in your pipeline” is where agents lie with confidence.

RalphCI fixed that by injecting real pipeline status into the agent loop after push. When CI failed, a CI Doctor agent read the logs, applied fixes, and pushed again. Twelve failures across five runs. Twelve autonomous fixes. Zero human triage.

The breakthrough in February: agentic loops with CI back-pressure can land green PRs AFK. The final PR passes CI. Commits inside that PR might still fail along the way. Red commits on the branch. Green at the end. Progress, not perfection.

Early May: every commit has to be green

Three months later we hit the next failure domain.

Local pnpm test:run kept passing on my Mac. Chunk sidecars kept failing on Linux. Missing LEGAL_DISCLAIMER.md. Missing SNAKE.md experiment marker. A TypeScript compile error the Snake workspace’s local tests never exercised but a full pnpm build caught.

Same pattern, five runs in a row. Local green. Sidecar red. CI Doctor lands a fix(ci-sidecar): commit. Sidecar green. Push allowed.

Our early May study wired Chunk sidecars into RalphCI’s Review Gate. A sidecar is a lightweight remote microVM: your workspace syncs there, Chunk runs a microbuild that mirrors your pipeline intent, and you get CI-shaped feedback before the branch leaves your laptop.

Five Snake builds. Five 100% green PRs. Every commit green. Not just the last one.

I had not touched the keyboard in twenty minutes. The final PR (and every commit on it) was green anyway.

This screenshot is what HIGH CONFIDENCE looks like.

The May breakthrough: three validation layers (local, sidecar, pipeline) produce PRs where humans review green branches only. Sidecars catch CI-shaped failures while the agent still has context. CircleCI remains the authority at merge time. Sidecars extend CI into the inner loop. They do not replace it.

Late May: deliver the same green outcome, faster

100% Green PRs answered “can we merge with confidence?” A follow-up experiment asked “how fast can we merge with confidence?”

Our late May study ran the same ten Go tasks two ways on chunk-cli: sidecar validation per iteration versus commit-push-poll CircleCI per iteration. Same agent. Same model. Same gate jobs lint and test). Different machinery.

Sixty-nine seconds. Every iteration. Five replicates in a row. That is not random noise in our harness. That is the queue tax on push-per-task CI.

Twenty-two seconds on a warmed sidecar snapshot. Same question: are lint and test happy right now? Same answer. Different delivery speed.

Token spend was flat because in this particular experiment tokens tracked fixes, not wait. The CI arm burned more clock time, not more intelligence, so sidecars did not shrink the LLM bill here. They shrunk idle time. On a ten-task run that is roughly eight minutes of gate-waiting saved before the final pipeline epilogue.

The May speed breakthrough: you still want an outer loop. Sidecar runs finished with a full ci workflow epilogue after one push. Fast inner loop on each task, authoritative pipeline confirmation at the end. Not sidecar-only cowboy coding. Not push-and-pray either.

Four layers, one arc

Strip away the Snake skin and the arc looks like this:

We are not gaining more AFK confidence because we trust agents more (although the models have improved since January). We are gaining more AFK confidence because each failure domain we hit got engineered out of the loop. And we’ve even unleashed some speed gains along the way.

Watch the loop. That is Huntley’s line and it is the whole game.

• January taught us to program the AFK agent loop.

• February taught us the agent’s “I passed” is local theater without CI.

• Early May taught us that 100% green PRs are possible, given sidecar validation before commit and push.

• Late May taught us that 100% green PRs are still the prize, and sidecars get us there faster than push-per-task CI.

Snake games did not teach us that AI writes good code. They taught us where validation has to live when the coder is an agent loop and the reviewer is still a human with a merge button.

What I am actually doing with this

I am obviously not shipping Snake to production. (Unless you’ve been craving retro gaming experiences. DM me.)

I am shipping the stack: RalphCI orchestration, Review Gate with local checks plus Chunk sidecar validation, CI Doctor for autonomous repair, CircleCI as the outer loop authority. Same pattern we pressure-tested on a toy game because toys fail fast and teach faster.

Four layers, one arc. Program the loop. Wire in CI back-pressure. Validate on sidecars before push. Maintain CI pipeline authority at the end.

Each layer kills a failure domain the last one missed. Confidence climbs. PRs go greener. Feedback gets faster. The human at the merge button only reviews green.

That is what five months of the same Snake game bought us.

Green CI is still priceless. Everything else is noise.

Where to read the lab reports

• everything is a ralph loop (Geoffrey Huntley, Jan 2026)

• We Let an AI Agent Say “I Passed.” Was It Actually Good? (Feb 2026)

• AFK Builds with 100% Green PRs (May 2026)

• The Sidecar Race: 22 Seconds vs 69 Seconds (May 2026)

More Snake runs coming. Heavier repos. Flakier tests. Parallel agents. Improved CLI. Better agent-facing APIs. You name it.

The benchmark stays small on purpose, and we intend to scale the lessons as we go.

Hywel Carver has been thinking about this longer than most. He’s the founder and CEO of Skiller Whale, a company built on the premise that engineers learn best by doing, not by watching. When he first talked publicly about what AI-generated code would mean for developer skill development, it was 2023, at a conference. The consensus at the time was still pretty comfortable. Things have gotten less comfortable since.

Rob sat down with Hywel to work through what it actually means to grow engineers in a world where most code is generated, not written.

1. Knowledge transfer is not the same thing as learning

The way most organizations approach skill development is to throw information at people and call it training. Watch this video. Read this documentation. Here’s a course with 47 modules.

“Very often the way that we expect people to learn is by just giving them a load of information. The actual practice of trying to do that thing, we leave you on your own to do in isolation. And that’s where the rubber hits the road. That’s when you find out whether you actually can do it or not.”

Hywel Carver, Founder and CEO, Skiller Whale

The problem is that information without practice doesn’t produce skill. Hywel’s framing: knowing and understanding are the first two rungs of a six-level learning hierarchy. Actually being able to apply something, evaluate someone else’s work, and eventually create something new comes later, and it only gets there through doing. If your development program stops at the second rung and calls it done, you haven’t built anything.

2. Bloom’s taxonomy explains exactly why AI code review is harder than it looks

Hywel referenced Bloom’s taxonomy of cognitive learning as a framework for understanding the current problem. The six levels build on each other: know, understand, apply, analyze, evaluate, create.

The way engineers have historically gotten good at reviewing code is by writing a lot of it themselves and having it reviewed in return. That cycle is how you build the mental model that lets you look at a loop and immediately think about off-by-one errors and variable state, rather than reading each line individually.

“We are used to a world where applying the skill of writing software is how we get good enough to evaluate it, to look at what someone else has done and say, yes, I am happy to put my name to that and for it to go into production.”

Hywel Carver, Founder and CEO, Skiller Whale

If 90% of code is now AI-generated, the training loop that used to happen organically through writing code no longer fires at the same rate. You can still develop those mental models. You just have to do it deliberately, through structured practice designed to build them, rather than picking them up as a byproduct of shipping.

3. AI and humans are bad at different things, and that’s precisely why both matter

Another interesting argument from the conversation: the error categories that AI is susceptible to are largely not the ones humans fall into, and the reverse is also true. AI is excellent at catching syntax errors, off-by-one issues, and assignment-vs-comparison bugs. Humans are not particularly good at those.

Humans are good at something different: reasoning about real-world consequences that weren’t in the scope of the ticket. The change that would give a specific category of users infinite free access. The knock-on effect that shows up in a workflow nobody thought to include in the spec.

“We are good at detecting things like: this is going to have a knock-on effect to that category of user that wasn’t in the scope of the change request. Humans are good at spotting that kind of thing.”

Hywel Carver, Founder and CEO, Skiller Whale

Hywel presented an example of an AI agent that found production credentials in a repo and wiped the database in nine seconds. It’s an argument for keeping human and AI review in parallel, because they’re checking for fundamentally different things.

Subscribe now

4. Using AI well is a skill, and it’s learnable

There’s an assumption baked into a lot of AI tooling rollouts: give people access to the tool, maybe run a vendor demo, and let adoption happen. Skiller Whale ran a study that suggests this assumption is wrong.

They worked with part of an engineering organization while the rest of the org had access to the same tools but no structured training. Both groups had the vendor talk. Both had access to the AI coding assistant.

“The group we worked with had 90% weekly active users where the rest of the org had 50% weekly active users. When we looked at their main measure of productivity, which was PR throughput, the group we worked with saw a roughly doubling in their PR throughput. The rest of the org actually went down very slightly.”

Hywel Carver, Founder and CEO, Skiller Whale

A doubling versus a slight decline, same tools, same company. The difference was whether engineers had built mental models for how the tools actually work. People who understand what an LLM is doing underneath prompt differently, catch failures differently, and know what to try when the output is wrong.

5. Mental models don’t expire, even when implementations do

The pace of change in AI tooling feels genuinely overwhelming if you try to keep up at the surface level: new models, new frameworks, new prompting techniques, new papers every week. Hywel’s argument is that this is the wrong level to track.

“If you have a good sense of how an LLM works, and then someone says, one of the things an LLM can do is produce output that says do a function call, you see how tools work and you’re like, okay, that makes sense to me. You have a way better understanding of it than someone who’s just been a user of those systems.”

Hywel Carver, Founder and CEO, Skiller Whale

His analogy: if you had a solid mental model of HTML, CSS, JavaScript, and how a browser renders pages in the mid-1990s, that model has not meaningfully changed today. The implementations are faster, the APIs are cleaner, but the underlying structure is the same. The mental model carries you through decades of surface-level change. The same logic applies to LLMs. The tips-and-tricks content will be obsolete in six months. But the architecture of how they’re trained and how they produce output hasn’t changed that much.

6. The job title is changing faster than anyone expected

One thing Hywel raised that didn’t get a full chapter in the conversation but probably deserves one: after teams go through structured AI training, the next thing they come back asking for is communication skills. Engineers are having more contact with stakeholders. Product functions at some companies have gotten smaller. Individual engineers are doing more.

That sounds like good news for engineering influence. It also means engineers who were hired to write code are now expected to run conversations they weren’t trained for, translate technical decisions for non-technical audiences, and navigate organizational dynamics that used to be someone else’s job.

The skill set that made someone an excellent engineer in 2020 is still valuable, but it needs some expansion in order to meet the role’s new level of complexity.

Subscribe to Confident Commit for new episodes and the data behind how software actually ships: confidentcommit.com.

Twenty-two seconds is what the other arm averaged for the same question on a Chunk sidecar.

Same repo. Same ten Go tasks. Same Claude Agent SDK edits. Same gate jobs. Different loop.

Three weeks ago I published AFK Builds with 100% Green PRs. Five Snake games. RalphCI orchestration. Sidecars in the Review Gate so agents stopped feeding red commits to GitHub. Green PRs were the win.

This follow-up is narrower and, I think, more revealing for anyone running agents today: not whether sidecars help you merge green, but how fast the agent gets CI-shaped feedback while the file is still open.

I kicked the tires on sidecars again.

They are fast.

Not a much cheaper agent. A less idle one. Token spend was flat in this particular experiment (more on that later).

The win was clock time.

Hypothesis

If each agent task is validated with chunk sidecar sync plus chunk validate --remote lint and test-changed) instead of commit → push → poll CircleCI for the same lint and test jobs, then:

1. Median time to signal (TTS) per iteration will be materially lower on the sidecar arm.

2. LLM cost will be similar (same agent, same ten prompts, same model fixing the same mistakes).

3. CircleCI still matters for pipeline-level confirmation (sidecar runs end with a full ci workflow epilogue after one final push).

I expected push-per-task CI to pay a queue tax every iteration. However, I did not expect the median to land on 69 seconds five replicates in a row.

Setup

Repo:

• CircleCI-Public/chunk-cli

• Experiment PR #370

Tasks: Ten cumulative edits on a small Go fixture internal/racefixture/ on experimental run branches).

Replicates: Five labels 001–005) per arm. 10 recorded runs total.

Metric: TTS = wall-clock seconds from iteration start until both gates report pass/fail (logged in results.csv`).

CircleCI CTO Rob Zuber frames this as rebalancing inner and outer loop validation. My May post showed the outcome layer (green PRs). This post measures the wait layer.

Results

Headline: median time to signal

Aggregate: median of per-run medians across five replicates.

Five replicates at a glance

Sidecar clustered 20–22s. CI sat on 69s every time. That stability is either a real push-and-queue signature in our harness or a coincidence at lab scale. I am reporting it, not universalizing it.

p95 TTS: sidecar ~23–25s; CI ~72–99s (tails worse on push-per-task).

LLM cost comparison explained

The speed table is not the only controlled comparison in this harness. Token spend is measured the same way: same model claude-sonnet-4-20250514), same ten tasks, same agent, same fix-and-retry pattern. We changed the validation path, not the work.

Both arms hit the same lint and test failures and applied the same corrections. Sidecar surfaced them via chunk validate --remote on the warmed snapshot (~22s median). The CI arm surfaced them after push to GitHub and a ~3× longer wait for the same gate jobs (~69s median). Same signal, different delivery speed.

That is why token spend landed flat. Tokens track fixes, not wait. The CI arm cost more clock time, not more tokens. Sidecars did not shrink the LLM bill here because there was nothing to shrink: identical failure set, identical agent work on both arms. On a messier repo, especially if remote CI surfaces failures the sidecar doesn’t mirror, token spend might diverge. This harness didn’t test that.

So treat ~$0.90 per run as what this harness actually cost for this specific experiment, not a budget for production agent loops. Whether faster feedback trims token burn on messier work and CI failure modes (i.e. given fewer stale-context retries, less thrash, and less environment drift) is a follow-on question for a future study (coming soon). This race did not answer it.

This experiment answers a narrow question cleanly: when the agent does the same fixes either way, faster feedback does not change token spend. It changes how long the agent sits idle.

Same gate jobs. Different machinery.

We timed the same question on both arms: did lint and test pass? We did not run identical validation. Sidecar answers via chunk validate --remote on a warmed-up Linux snapshot. CI answers via real CircleCI jobs after you push.

Early iterations on both arms still saw lint failures while the agent fixed mistakes (about two failing lint iters per run in the rollup). The race is not “sidecar never fails.” It is how long you wait to learn.

Illustrative lab math only: ten tasks × ~47s saved ≈ ~8 minutes less gate-waiting per replicate before the sidecar epilogue. Your mileage will vary.

Takeaway

Green PRs are the outcome. Time to signal is the throttle.

The May experiment showed sidecars help agents stop polluting branches with commits CI would reject. This one shows why that matters in wall clock terms: the coding agent is not stuck in queue while context goes cold.

What held across ten runs in chunk-cli:

1. Sidecars save feedback latency on mirrored gate jobs (~3.1× median TTS here).

2. LLM spend was flat (~$4.64 sidecar vs ~$4.73 CI) in the same apples-to-apples comparison: same failures, same fixes, CI just took ~3× longer to surface them. Tokens track fixes, not wait.

3. Push-per-task CI is a predictable wait in this harness (69s median, every replicate).

4. You still want an outer loop. Sidecar runs finished with a full ci workflow epilogue. Fast inner loop on each task, plus an authoritative pipeline confirmation at the end. Not sidecar-only cowboy coding.

Please re-read #4 above: I am not at all suggesting you delete your CI pipelines. Quite the opposite: I am saying that if your agent loop is push → wait → fix → push again, you are paying for that wait every iteration.

Sidecars moved the same exact question to a local, warmed-up Linux snapshot...

QUESTION: Are lint and test happy on this repo right now?

...and the same answers came with a much tighter feedback loop. 3.1× faster.

What’s Next

• Combine this harness with RalphCI orchestration (Snake outcome study + TTS race in one stack).

• Heavier repos, flaky tests, parallel agents. Find where Chunk sidecars and snapshot sync stops winning.

• Agent economics at scale: cost per green iteration (LLM + validation path), not just TTS. A proper cohort study on heavier repos, not extrapolated from this ten-run table alone.

• Demo recordings on various Chunk sidecar and snapshot sync setups. Stay tuned!

Artifacts:

• FINDINGS.md

• Experiment PR #370