CI can tell you an agent-built app compiles, lints, and passes its tests. It can’t tell you whether a person can actually use the thing. That check usually falls to a human clicking around and trying things out before release: exploratory testing, manual QA, a “human in the loop”, whatever your team calls it. Your other option is hand-coding a bot to do the clicking, and then maintaining that bot forever.

I built a small research test rig called Understudy to find out how much of that human click-through an AI player can cover, and whether it beats the hand-coded bot. Snake keeps the question easy to score: can somebody get through this, and how far? This is Understudy’s first real race.

Why “Understudy”? In theater, understudies learn the part so they can go on when the lead can’t. Same idea here: an agent that stands in for the human click-through, not a replacement for the show.

Here’s the setup in plain English. Twenty-one Snake games, all built by coding agents in earlier Loop Lab experiments. Two players:

• Greedy bot I vibecoded by hand in JavaScript: a short, fixed rulebook that chases the nearest food unless that would trap it.

• Jev, a decision model from Typesafe that picks one answer from a fixed list instead of writing paragraphs of advice like a typical LLM would.

Both see the same board. Both get the same list of moves that won’t kill the snake on each tick of the game. Each game runs until the snake dies, and the score is how much food it ate.

Jev averaged 95.0 points per game. The greedy bot averaged 72.6. A second Jev setup with extra strategy tips in its prompt averaged 94.5, a dead heat with the first. On average, the tips bought nothing. But, that’s a second story. The main story is that Jev outscored the greedy bot on 19 of 21 builds.

Think of it as bowling with bumpers. Both bowlers had the gutters blocked, so nobody threw a gutter ball. The question was only who knocked down more pins.

Subscribe now

Hypothesis

If an AI decision model is held to the same safe-move list as a hand-coded bot baseline and scored until death on a fixed set of games, it should beat the baseline on mean score.

Why that bar? If the model can’t outscore a rule book that fits on one screen, in a game with five buttons, it has no business standing in for a human tester on anything harder.

And the CI idea I actually care about (i.e., freeze a set of scenarios you replay every night or before a release, keep a cheap deterministic player as the floor, run an agent “manual tester” against the same scenarios, compare) only makes sense if the agent beats the free player on something you can measure: higher scores and longer runs on the same games. If it can’t, just run the free bot and save the tokens.

Setup

The games. The corpus is 21 Snake builds pulled from half a year of Loop Lab Snake testing: six from cost-of-a-green-PR calibration runs, ten from the CI vs no-CI study, and five from the Chunk sidecars / 100% green PR runs.

Same spec, different agent runs, so 21 slightly different versions of one app. Food uses each game’s own random spawner (Math.random, like a real browser session), so no two runs see the same food sequence. Runs end at death, with two safety valves: a 10,000-tick ceiling and a stuck detector that kills a run after 2,500 ticks with no score change. A tick is one step of the game clock, one move.

The shared move surface. Before either player picks a direction, a shared scorer looks at the board and grades every option. That scorer lives in one file (evaluate.ts) that both the greedy bot and Jev read from, so the rules can’t drift between them. For each direction it reports:

• Fatal or not: reversing into your own neck, hitting a wall, or hitting your body

• Distance: how far the next square is from the food

• Space: a flood-fill count, meaning how many open squares the head could still reach from there

• Need: snake length plus two (plus one if the move eats), the room a snake needs to not box itself in

• Escape routes: how many safe moves exist after this one

Those are the bumpers.

SAFE means a direction that is not fatal on this tick: it won’t reverse into the neck, walk off the grid, or land on the body.

The players:

Here’s the heart of what Jev sees each tick, in plain English, expressed as human-readable intent. Only SAFE directions are listed. Then:

Among the listed SAFE choices:
  1. Prefer moves that still have an escape after this one.
  2. Prefer roomy squares (enough open space for the snake's length).
     If every option is tight, don't take food when a roomier non-food
     move is on the list.
  3. Among what's left: closer to food, then more open space,
     then keep the current facing on ties.
Pick exactly one listed direction.

That’s it. The model makes the final call.

The Jev strategy profile specifically adds four short tactics on top: keep an escape into open space, don’t hug walls early, don’t cut off your own tail, break distance ties toward open space or the center. On average, those tips bought nothing on the scoreboard. Different tips might. That’s a future experiment.

The alternative is the Greedy player: lengthy JavaScript that encodes the same procedure as hard-coded branches, and has to be extended and maintained every time you want a smarter heuristic.

Jev gets the procedure as readable policy text, plus the scored candidates from evaluate.ts. Same bumpers. Less glue code to own.

Jev ran as jev-latest through Typesafe’s System One API, one call per tick, answering a single multiple-choice question: which direction? Only safe moves were offered whenever any existed. If Jev ever answered with a move outside that list, Understudy swapped in greedy’s pick for that tick.

Variables. Changed: the player. Held constant: the 21 builds, the move surface, the safe-move filter, one decision per tick, and the run caps.

Cost. Greedy runs free on my laptop’s CPU. Both Jev passes together cost about $12 on my personal Typesafe account, covering more than 109,000 decisions.

Results

All 21 games finished at a normal game over for both Jev profiles. Greedy is $0 (i.e. FREE) CPU on my laptop, including all 105 runs. Each Jev profile was one pass over the 21 games at about $6; both together about $12 on my personal Typesafe account.

Quick unit math on that bill, using the run totals (one API call per tick, so ticks ≈ decisions):

So: about 29¢ per game, and about 11¢ per thousand decisions. Cheap for a nightly corpus job. Still a real line item if you fire it on every commit.

Head-to-head, Jev strategy against Jev parity on the same 21 games:

NOTE: One run went sideways. Strategy’s first try on one game variant ate only one food pellet, then circled the next random food pellet for 2,516 ticks without scoring until the stuck detector ended it at a score of 1. No API errors. Every move stayed on the safe list. Since that low score was merely bad luck due to bad luck with food placement, I archived that replay and reran the game: 106. The table uses the rerun. If I had kept the score of 1 instead and strategy’s mean dropped to 89.5, it would still be well above greedy and still no better than parity.

By the way, the above anomaly is not necessarily a problem with the Jev strategy prompt, but rather a limitation of attempting to automate a human in the loop. A human being would never play a game like that.

What a CI pipeline config would read

Every run writes a trace (JSON, every tick) and a self-contained replay page that opens in any browser. That’s the raw material for a CI gate. I have not wired this into a pipeline. The sketch below shows the shape, and the gate script in it doesn’t exist yet:

# Sketch only. score-gate.mjs is hypothetical.
jobs:
  understudy-play:
    docker:
      - image: cimg/node:22.13
    steps:
      - checkout
      - run: pnpm install
      - run: pnpm sim:batch -- --player greedy --results-name control
      - run: pnpm sim:batch -- --player jev --results-name agent
      - run: node scripts/score-gate.mjs --floor control --candidate agent
      - store_artifacts:
          path: results

The control run sets the floor. The agent run has to clear it. The replays go up as build artifacts so a human can watch the worst game in a couple of minutes instead of replaying all 21 by hand.

TL;DR

The lab claim held. With identical bumpers, Jev outscored a strong hand-coded bot by about 22 points per game and lived roughly 900 ticks longer. That wasn’t one hot streak either: parity beat greedy’s per-game average on 19 of 21 builds.

So here’s my recommendation. If you’re choosing between hand-coding a bot and paying a human to click through every build, a decision model is worth a serious look as the stand-in. The policy is a paragraph of readable text, not a pile of branches you maintain forever, and it played better.

Coaching it to “think strategically” hit diminishing returns. That’s the secondary finding, not the headline. My quick four-tip prompt left the average flat: parity and strategy split 9 to 9, with strategy about 0.6 points behind. The best and worst games did nudge up (70 / 128 against parity’s 66 / 119, counting the rerun), but on one pass per profile I won’t call that a win.

My read: most of Jev’s edge comes from weighing the features it’s handed, so put the knowledge in the state you pass it (escape routes, open space) before you write a paragraph of advice. The right coaching might still help. These four tips weren’t it, and I need more runs before I say anything definitive.

The practice claim is an implication, not a measurement. Nothing here ran inside a CircleCI pipeline. But the shape works: frozen scenarios, a free deterministic control, an agent player, scores plus replays (as HTML artifacts). I’d use it when:

• The question is “can somebody get through this?” and you can score it (a game, a checkout flow, a signup)

• You have a fixed scenario set the agent replays every night or before a release

• You want a human to watch a two-minute replay of the worst run, not click through everything

Btw, watching a snake circle one food pellet for 2,516 ticks until the stuck detector fires is the best part of the job. Same for the unwinnable food placements: maybe a game should never spawn food you can’t reach. Every replay artifact teaches something. None of them were written by hand. 100% of them were written by my Understudy agent.

I wouldn’t use Understudy on every push of every PR. One API call per tick, in sequence, takes hours per corpus pass, and tokens are the tax the free control never pays. $12 for 42 full games is cheap for a nightly job. Per commit might get a bit pricey, depending on the app.

I also wouldn’t ask Understudy about taste. A high Snake score means the game is playable to the end. It says nothing about whether the game is ugly, confusing, or fun, although we are still working on that.

Limits, plainly: one Jev run per game against five for greedy, one app shape, and no quality-judge pass on these traces yet. I stopped after one full corpus pass per Jev profile because the gap over greedy was already clear and the token bill is real. More replicates would tighten the error bars. They would not change the headline.

A better next step than burning five times the API spend is pointing the same pattern at an app with screens to click through instead of a snake to steer. That’s the next experiment.

Bumpers keep the ball out of the gutter. A human in the loop still has to watch the replay. Understudy’s job is making sure there’s a replay worth watching before that human ever gets involved.

In the past, that kind of review meant walking through the process set out in the existing docs myself, and taking notes and screenshots for each step in order to find gaps and outdated content. This job has expanded over recent months. The paths into a product have multiplied: People, or agents, might be doing everything from the terminal. You might use a skill, a plugin with your agent, or ask your agent to use the CLI, APIs, or MCP. There are now many ways to get set up, and I had to understand each to make sure we have them documented correctly.

So instead of just screenshotting the app, I planned to draw up journeys in a Figma file for each of the different ways you can get a project set up, with screenshots or a note of the command and outcome at every step.

Partway through, something odd happened. I had previously installed the CircleCI plugin while working on docs about how to use it. Since this is how Claude is designed to work (pick up cues from our interactions and decide how to best approach the task), my Claude session found it, decided it was relevant, and used it without my asking. It then sent me to a URL that doesn’t exist. Meanwhile, the CLI running in the same session had shown me a list of my orgs and asked me to pick one.

Three things didn’t add up:

• I’m used to Claude knowing which org I’m in, so being asked for one felt off.

• The plugin sent me to a browser, outside the loop of what Claude and I were working on.

• The CLI could already do what the plugin was asking me to do elsewhere. It lists your orgs, and it can create one, so there was no reason to open a browser and click through a bunch of buttons.

Put together, these pointed to one conclusion: Claude was following stale instructions. The skill and the tool had drifted apart, and Claude followed the skill.

What I can’t tell you is how the skill got that way. It may have been written by an agent that made up the URL, or a review may have been missed, or the URL may once have existed and since been removed. Whichever it was, the result for me was the same.

Claude gave no sign that anything was wrong. Agents don’t signal uncertainty when they follow stale skills. It didn’t say “I found a skill, it might be out of date, here’s what it says.” It just followed the skill. The skill had become invisible infrastructure: something I didn’t know was running until it sent me somewhere wrong. With default settings, even when skills are packaged as plugins with versioned releases, an agent or human user won’t automatically know their skills are outdated, unless the individual user has previously opted into that configuration.

Today, I’ll be sharing how I worked through this challenge.

Subscribe now

Hypothesis

I went in with a simple hypothesis: a Claude session with a relevant skill installed would perform at least as well as the underlying tool. Skills are supposed to give agents richer context than a --help flag alone, so I expected the plugin path to match the CLI at minimum, and probably improve on it.

That hypothesis was wrong. The skill-assisted session did worse than the CLI, and it did so without telling me. Several assumptions sat underneath my hypothesis, and each one fell apart:

• I assumed a skill would stay in step with its tool. I didn’t realize a skill could drift from the underlying tooling in such an extreme way, or that it could happen so quickly.

• I assumed I’d know when a skill was running. I’d forgotten I had the plugin installed at all, since I’d added it for a different line of work. In a natural-language chat, you don’t know what the agent is doing unless you check.

• I assumed the agent would choose sensibly between sources. The skill took precedence over the CLI, and nothing told me it had.

• I assumed there were only a few ways to get set up. There were far more paths than I’d expected, and they behaved very differently.

Being wrong meant changing how I tested. I’d assumed the plugin’s instructions would stay in step with the CLI, so I’d treated the two as interchangeable. They aren’t. Each way of getting set up can drift on its own, so each one needs its own test.

Setup

It became clear that I would need to check all of the different ways to set up my project. I began with just the CLI in my terminal. I asked Claude to run through the steps using only the CLI, the underlying tooling, being explicit about which parts of the product to use. I asked Claude to enable the CLI MCP and get set up. All while noting the steps in my Figma file.

The conditions for the test were simple: I already had a CircleCI account, I was admin of an org, and I had a generic GitHub project with no CircleCI config. The task was just to set up that project, with my orgs already in place. It shouldn’t have been difficult.

I tested three (of many) paths for the same task:

1. CLI directly in the terminal

2. CLI in a Claude Code session, no plugin installed

3. Claude Code session with the public CircleCI plugin installed (https://github.com/CircleCI-Public/skills)

In the third session, the plugin’s onboarding skill loaded without my explicitly invoking it. Claude found it, treated it as authoritative, and followed it.

I’d expected the three paths to feel broadly similar. It surprised me how different they were. To get through the process with the skill, I had to report the issue to the engineering team, ask for the problem to be fixed, and then go back and check that it was working.

It didn’t stop me from doing the work. The how-to guide I was updating doesn’t explicitly say “install the plugin, tell your agent to do this,” so I could carry on. But I had to pause, report to the team that something was off, and return to it later.

This would have come up eventually, and it might well have been a customer who hit it first. Writing or updating guides is a common way to uncover bugs, and catching them before customers do is one of the ways the docs team can bring the voice of the customer back to engineering teams directly.

Results

The results clearly showed that the skill and the CLI had diverged. The CLI had been updated to handle org selection in the terminal. The skill still pointed to a browser flow with a URL that didn’t exist. Claude had no way to know which was fresher. It used what it found.

As mentioned earlier, Claude also asked me for an org slug (something the CLI can already see when I’m authenticated) and wanted me to create a new org in the app, at a URL I didn’t recognize. I’m quite familiar with our URL patterns, and from working on the CLI docs, I know the CLI and API can create an org. There should have been no clunky opening of a browser and clicking of buttons.

Note: The agentic experience team has since fixed the skill. Instead of sending users to a broken URL in a browser, the agent now prompts them to pick an org in the session, so no URL is needed.

What this says about our docs

Many of our user guides were written from the viewpoint of clicking buttons in the web app. Today, we need to document more use cases for using our tools from a terminal. The fact is that every guide needs reviewing from the point of view of an agent doing the work, with a human in the loop or without one.

That’s a different level of review. You can get agents to write your docs for you, but you still need to check that they’re correct for the various ways your audiences work. Today, there are a lot more steps to cover, and getting through the work quickly and accurately is the challenge.

What happened to my Figma file

To record each journey, I’d been building a Figma file with screenshots of every step, or a small box with the command and outcome at each step from A to B. By the time I finished and found this problem, the “Using the Skill” section of the Figma was out of date the same day I created it. The artifact is still useful as a reference but by no means a static source of truth to build on. It’s a snapshot in time that will need to be constantly reviewed, just like the doc itself. A scratchpad.

It was a useful exercise, but the artifact that ultimately matters is the documentation, and even that may be a losing battle because the skills have already changed. I probably wouldn’t do it that way again.

Ideally, we’d move to automated docs generation from the code, with an added human review to confirm the reality of the experience. The challenge will be extracting what parts of this process we can optimize and automate for speed of delivery to keep things consistently up to date. However this rolls out, the most reliable way to ensure that docs are correct will always be returning to the product as it is, on the day of the update.

Keeping docs from going stale

Rather than constantly rebuilding Figma journeys, I’d now lean on a few other things:

• Community feedback, which now includes agents. An agent could report, via an integration with our chatbot, something like “when I was answering this customer question, I couldn’t give them an accurate answer because...”

• An agentic review on a rolling basis, so docs don’t go stale between manual passes.

• Watching for changes in the relevant codebase and automating change requests in the corresponding docs. For example, the code change could flag an integration to create a Linear ticket, the agent would read the ticket scope and draft the change, and finally a human would review the change against real product use.

TL;DR

Every public skill, MCP description, or structured doc you’ve published is now part of your agent surface, and agents will act on it as written. Skills are instructions for agents, so they need an owner who updates them whenever the tool changes, just like product docs.

The practical check: when did you last run your own onboarding flow with a plugin installed? The people maintaining the CLI aren’t always the people maintaining the skill, so the two can disagree without anyone noticing.

The app’s UI text, the skill, and the docs all carry the same instructions now, and every copy has to match; both each other and how the product actually works.

The pipeline is still running.

When new repos appear, they get fixed too.

That pipeline is the repo fixer. Here is how it works and why I think the pattern is worth stealing.

Hypothesis

Every codebase accumulates patterns that made sense at the time and don’t anymore later on. Hardcoded image tags. Old resource classes. Config patterns that predate a better standard. Everyone knows these should be cleaned up. Nobody does it because the cost is too high: you have to touch every repo, write the same PR description forty times, and then wait for forty teams to review something trivial.

My hypothesis was that LLMs would be good enough at narrow, targeted transformations to make this class of work economically viable again. Not “write me an application” prompts. Specific, bounded prompts: “find config files that reference this executor, update them to use this one instead, explain what you changed and why.” If that worked, and if I made it easy enough for other engineers to define their own fixers, people would use it without being asked.

Both turned out to be true.

Subscribe now

Setup

My process always reflects a focus on “the customer”; in this case the customers were my fellow teammates, divided into 2 distinct groups:

Customer 1: The SME who knows the fix needs to happen but doesn’t want to make the PRs. They write the prompt once, defining what the broken pattern looks like and how to fix it. That’s the end of their work. They never touch a repo directly.

Customer 2: The engineer responsible for actually merging the fix in their repo. They get a well-described draft PR from a bot user, can review it quickly, and merge on their own schedule without being chased.

I consciously built for both from the start.

Repo fixer is a CircleCI pipeline that runs on a schedule across every repo in the org. The engineer who knows a fix needs to happen writes a fixer once. After that, the pipeline handles everything.

Each fixer has two phases.

The first phase is detection. Before spending money on an LLM, the pipeline checks whether this repo even needs the fix. This is done with cheap tools: bash, grep, or a code search query. If the pattern isn’t present, the fixer exits immediately. Most repos exit here. You only pay for the LLM on the repos that actually need work.

The second phase is the fix itself. If detection passes, the pipeline runs the fixer, which is defined as a bash script, an LLM prompt, or a combination of both. The prompt is written by the person who knows the codebase pattern best, so it’s specific. When the gen2 migration ran, the prompt knew exactly which executor names to look for, what to replace them with, and what edge cases to watch for.

It’s worth noting that the gen2 fixer used an LLM prompt, but it could have been written with ast-grep or bash tools with more work. LLMs handle edge cases you didn’t even think of very well and I knew that it would only run once because of the gating detections, so I decided that LLM prompt was the way to go here. I could have spent hours finding and tuning the script to work on all the edge cases but… I didn’t have to.

When the fixer decides to make a change, it commits the diff and opens a PR as a bot user. Using a bot user means any engineer on the team can approve and merge the PR. If the author opened it, they couldn’t merge their own work, which would make the whole point moot.

The PR includes everything a reviewer needs: what changed, why, which fixer generated it, and a link to context. PRs open as drafts by default. I review them in bulk before they go out, then pass them off to each team. That review gate is not optional. When you’re opening hundreds of PRs at once, you need to QA the output before creating noise for everyone downstream.

There are also two Claude skills for managing the system day-to-day: one to trigger a fixer immediately instead of waiting for the next scheduled run, and one to view and triage the PRs that are waiting to be merged.

Personally, I’m not a “wait to act until it’s perfect” person. I’m a “get something working and then iterate” person. So the tooling itself went through several iterations.

The first version was a pile of bash scripts. Moving it to a Go binary made a real difference: writing a new fixer became a much better experience for the subject matter expert, which is the whole point. If defining a fixer is painful, people won’t write them. The PR descriptions also evolved. The first version shipped with an empty description. It now includes a disclosure that the change was generated by a tool, signed commits, and enough context that a reviewer can approve it in under a minute.

The inspiration for the pattern was Sourcegraph batch changes. The key difference is that repo fixer keeps running on a schedule forever. Batch changes are a one-time operation. Patterns keep appearing in new repos, so a one-time run puts you back where you started six months later.

Results

The cleaner result is behavioral. I expected to have to suggest the repo fixer to people. For the first few fixers, I did. Now engineers come to me asking to add one. The SME who knows a pattern needs fixing writes a prompt, and their work is done. The PRs appear on a schedule without them doing anything else.

One thing that caught me off guard: not all of our repos had the same default branch. The pipeline had to handle that variation, which wasn’t something I had thought through on the first pass. It’s worth building for early if you’re doing this across a large org.

TL;DR

The LLM is not the interesting part of this. The interesting part is the structure around it.

A detection pass before the LLM call is not optional at scale. If you skip it and run the prompt on every repo unconditionally, you burn money and slow everything down. The cheap filter is what makes the system viable to run on a schedule forever.

Draft PRs with a review gate before merge are also not optional. LLM-generated PRs at volume create a lot of noise. You need a way to verify the output before it lands in forty engineers’ review queues.

The bot user pattern solves another big problem. If a human opens a PR, that human is now in the critical path for every merge. A bot user removes that dependency entirely. Anyone can approve. Teams merge on their own schedule.

And the scheduling matters more than I expected. The value of repo fixer is the tenth run, six months later, when a new repo appears with the old pattern and gets a PR the next morning without anyone having to remember to file a ticket.

I had assumed this would be a slam dunk. But the numbers we saw on our analytics dashboard didn’t add up: More customers clicked the button than ended up running the --failure-report command. This was odd, given that the prompt explicitly told the agent to use it.

At the same time, in my daily work, I tried using the CLI for my own work to fix pipelines and review runs. And I started noticing that even if I told the agent to use the CLI, it would use the CircleCI MCP (which I also have installed).

It left me with a lot of questions. Was this a coincidence? Where were all these clicks going? Were customers using different tools? Was the prompt not working? Was it a CLI distribution problem? Or maybe there were other factors influencing this behavior: was it the available context? Hidden preferences in the agent’s system prompt? Was the CLI too confusing to use?

To answer these questions, I built some evals and began investigating.

Subscribe now

Hypothesis

My initial hypothesis was that the agent had certain preferences when it comes to which tools to use for each task (in this case, to fetch CI context). These preferences could be related to the model’s training (system prompt), the instructions given (prompt engineering) or to the available context for the agent. Which of these factors had an actual impact on the agent’s behavior, and why?

If we found what caused the agent to pick one tool over another, we could better design and distribute the tools we’ve been building for agents and engineers.

Setup

From my previous work, I had already set up a controlled environment with CircleCI tools that let me run evals on agent behavior and grade the results directly. I decided to reuse that project for this new behavioral experiment.

The task was simple: give the agent a failed pipeline URL and ask it to fetch and diagnose the error. Not fix it… just diagnose. That let me isolate the tool-choice behavior without mixing it with the ability to actually fix the problem (a whole different goal).

I set up multiple environments, varying two things:

Tool availability: CLI only, MCP only, REST API only, and combinations (all three, CLI + MCP, CLI + API, MCP + API).

Instructions: either free choice (use whatever tool you want) or an explicit instruction to use the CLI.

We had 10 different conditions. We ran each condition across 27 different samples using Claude-Sonnet-5 and recorded which tool the agent picked.

After viewing the initial results, I had some visibility on the preferences, but I was still missing the reasoning behind them. So I ran a second pass and added one line to the prompt: “Justify why you chose that tool.” I wanted the agent’s reasoning in plain language, not just the outcome.

Results

All the arms had a CircleCI Token for them to make authenticated requests to the API. This was mentioned to the agent in some experiments, and not in others, to analyze whether the agent tried to hit the API or not.

As you can see, there is a clear preference for MCP even when instructed to only use the CLI. When the MCP is available, it almost always goes that way. When not, agents showed a strong preference for the API over the CLI. Even when using the CLI, the API was still called a couple of times.

These results were already surprising. The hypothesis was somewhat correct: agents do prefer MCP over any other tool. But we still did not know why they were picking those tools; we only knew that they did. So we added an extra step and re-ran the free-choice experiments, asking the agents to explain why they chose the tool they chose.

The agents provided the following explanation of why they were choosing the API over the CLI:

“I’ll use the CircleCI REST API via curl since it gives structured JSON I can parse directly without needing interactive CLI auth setup.”

And for choosing the MCP over the CLI when both were available:

“MCP tools provide structured access to pipeline data without needing to construct API calls manually.”

Both explanations are wrong about the facts. The CLI in our setup was already authenticated via CIRCLE_TOKEN. It has a --json flag that produces the same structured output. The agent didn’t know either of those things. The information existed, but since it wasn’t in the prompt, it was missed.

So we updated the CLI prompt to state three things explicitly: the CLI is pre-authenticated via CIRCLE_TOKEN, it supports --json for structured output, and here are the exact command examples.

We ran the experiment again. The agent started using the CLI. We went from 0% of sessions using the CLI to 100% of sessions using it, while others also kept using the API for additional context. Nevertheless, when the MCP was available, it was still the agent’s preferred tool.

TL;DR

The agent isn’t reading your mind or your README. It’s making tool-selection decisions based on what’s in its context at the moment it has to choose. If that context is incomplete (if it doesn’t know the CLI is authenticated, or that --json exists), it will make a reasonable-sounding choice that happens to be wrong.

Two things follow from this.

MCP schema presence outweighs instructions. When an MCP server is configured, its tool definitions are already parsed into the model’s context. That’s a structural advantage over a CLI that has to be described in prose. Telling the agent to use the CLI is fighting uphill against a schema the model has already internalized. If you want the CLI to win in a mixed environment, you have to make the CLI’s capabilities as legible as the MCP’s, which means prompt engineering.

The agent’s wrong choice was entirely predictable from its context. It wasn’t hallucinating. It was reasoning correctly from incomplete information. “Doesn’t need interactive auth setup” is a reasonable thing to prefer, but it just wasn’t true in our environment, and our prompt didn’t say so. Once the prompt said so, the behavior changed.

The broader point: any time an agent is choosing between tools, the choice is a function of what it knows about each option at selection time. If you’re not getting the behavior you want, the first question is what the agent thinks it knows… and whether that’s accurate.

Recently I’ve been curious about decision models: a newer kind of AI built to pick one option from a list, quickly and cheaply. Could one handle choices like these? Real pipelines are a hard place to find out, because you can’t replay the same Tuesday twice. So I decided to create a simulation.

A building shares the same pipeline problem with its elevators. The cars are your runners, the button presses are your commits, and a dispatcher has to pick a car for every call, instantly and forever. An elevator bank has a clean score (how long people wait) and can be replayed exactly. Add a rule for each new edge case and you end up with the same pile of special cases.

Like your pipeline, when it chooses well, no one notices. But when it chooses badly, people stand in lobbies, checking their watches, never knowing why.

What we can build is limited by the pieces we can ship. You can solve a surprising number of problems with plain rules, but every new variable multiplies the cases. A heavy group, a broken car and a meeting ending on the fifth floor can all arrive in the same minute. Write a rule for each combination, and soon you are tending a jungle, and somewhere in it there is always a case nobody thought of.

So the question behind this experiment is simple to ask and hard to answer: can we take a small part of what the language-model revolution has taught us, and use it to put smart, affordable parts into ordinary things, like the lift in an office? That is also why the benchmark is an animated building rather than a silent script: I want you to watch how quickly a few simple things turn into a moment nobody planned for.

Subscribe now

Hypothesis

We tested two decision models: TypeSafe’s Jev, a hosted model getting a lot of attention right now, and Laya, a small open-weight model you can run and train yourself. Our central hypothesis was that a small, affordable AI decision model could make a repeated, real-world choice (here, which elevator answers a call) as well as a well-tuned classic rule, but only under two conditions:

• First, the model has to be given the right information in the right form.

• Second, a model built to be trained has to actually be trained, on the system’s own decisions.

We did not expect these models to beat the rule. The question was narrower and more practical: what does it take to match the baseline you already have? That is the question a CI/CD team faces before handing decisions like runner selection or test selection to a model. Before it can improve on your rules, it has to equal them. We broke the idea into three testable hypotheses:

1. For Jev, how we describe the cars matters as much as the model.

2. Out of the box, the open model is not ready: its makers call it “a fast base to specialise, not a zero-shot decision engine”.

3. Fine-tuned on the building’s own decisions, the open model can reach the level of the best dispatchers.

We explored these first on ten seeds, then re-tested the headline claims on fresh seeds (see “Then we checked”). Each one is tested in the experiments below.

The contenders

Decision models are a different kind of AI. A chat model writes; a decision model chooses. You send it the facts and a typed question with a set of options, and it returns one option, with a probability for every option, in a single step. There is no text to parse and nothing to make up. For elevators, the question is one per call: “Which car should answer this hall call?”, asked only among the cars that can legally take it.

• Jev is TypeSafe’s hosted decision model (TypeSafe calls it a System One model), reached through an API. It can’t be fine-tuned: “the same weights serve every account”.

• Laya is Convai Innovations’ open-weight model (Apache 2.0, 421 million parameters). You run it on your own machine, and you can train it.

Both speak the same API, so we could send them byte-for-byte identical requests, with only the model name changed. That is rare in AI comparisons.

(to watch the video, ensure you’re reading this article in your web browser or Substack app.)

The setup

We built Elevator Decision Arena, an open-source benchmark that lives inside a simulation. It races two dispatchers side by side, in two copies of the same building, with the same passengers and the same surprises. The building and its traffic are simulated. The decisions are real: real algorithms, and real calls to both models. The results describe our simulated building, not any real one.

Jev (left) and base Laya (right) dispatching the same morning rush, 48 seconds in. Same building, same passengers, same moment; in both buildings, car 3 is out of service.

• Building: 24 floors and 4 cars. Two kinds of traffic: Normal Traffic, a steady day, and Morning Wave, a lobby rush with a heavy group, a broken car and a sudden crowd.

• Seeds: a seed is a whole morning, replayed exactly. Every dispatcher faces the same people, and we compare them seed by seed. (This is the same idea as replaying the same commit history through two different pipeline configurations, so the only thing that differs is the decision-maker.)

• Fixed decision time: every decision takes 0.25 s, whoever makes it, so only the quality of the choice counts.

• Yardsticks: Nearest-Car ETA, a classic rule that sends the car with the lowest estimated arrival time (versions v1.2.0 and a load-aware v1.3.0), and round robin, which hands calls to the cars in turn, wherever they are.

• Pinned versions: jev-1.13.0, checked on every answer; Laya at a reviewed commit, on a laptop GPU (Apple M4 Pro).

Experiment 1: How you ask matters

We asked Jev the same question three ways. For each car, it read either raw numbers, friendlier words, or an arrival estimate worked out by our code:

• Raw: floor 3.2, moving up, doors closed, 150 of 630 kg, 2 stops, 1 call

• Words: 4 floors below the call, moving towards it, about a quarter full, 2 stops planned

• Estimate: arrives in about 9 s, about a quarter full (480 kg of room), 2 stops planned

(These examples come from a hand-built test request, whose cars are larger than the simulator’s.)

On raw numbers, Jev did about as well as taking turns. TypeSafe’s own docs explain why: Jev “is not a calculator”, and the advice is to “keep the arithmetic in code”. With the estimate, Jev came within a second of the classic rule, too close for ten seeds to call.

One caveat: the estimate is the classic rule’s own formula, and our question asks for the car that will arrive soonest. Jev is still free to prefer a slower car with room, but we handed it the key number. On 1,601 separate validation decisions, it picked the rule’s car 90.3% of the time.

Experiment 2: The open model, out of the box

We sent base Laya the same raw-number requests, untrained.

Used zero-shot, in our question format, base Laya did worse than round robin, a dispatcher that doesn’t even look at where the cars are. With the arrival estimate it improved (40.3 s and 50.7 s) but still trailed taking turns. On the 1,601 validation decisions it agreed with the classic rule 24.9% of the time, a little below a random pick.

None of this contradicts Laya’s documentation, which says base checkpoints are “near chance on typed-decisions zero-shot”. A fair reader could also point to our side: we wrote the question format, and in these runs the cars were always listed in the same order. Reading raw numbers, base Laya picked the first car on the list far more often than the rule would.

Where Laya already wins: it answers in about 40 ms on a laptop (Jev: about 200 ms over the network), costs nothing per decision, gives the same answer to the same request, works offline, and can be taught.

Experiment 3: Teaching the open model

We then taught Laya with the one thing Jev can never be trained on: the building’s own decisions, generated by the simulator. The seeds we raced on were never used in training.

• Copy the rule (imitation): 4,382 decisions labeled with the classic rule’s choice; about 24 minutes on a laptop GPU. Agreement with the rule rose from 24.9% to 92.8%. In the races it waited 9.4 s and 7.3 s less than base Laya, on every seed, and came level with the rule and with Jev. A student who copies a teacher can, at best, reach the teacher, and that is what it did.

• Learn from outcomes (hindsight): At each decision the simulator tried every car, played 90 seconds forward, and labeled the car that led to the least waiting. A dispatcher acting on those labels was about 6 s better than everything, but it is an oracle: its forward plays see the real future, including who walks in next. Laya trained on those labels came out level again, and captured none of that gap. The labels knew the future; the model can only see the present.

• A model for another building: In a second building (12 floors, 3 cars, evening traffic heading down), the model we had fine-tuned on the first building’s decisions, with no training on this one, performed at the same level as the classic rule. A model trained only on the second building’s 2,109 decisions did far worse: it sent car 1 every time. Its 43% accuracy was misleading, because car 1 happened to be the correct answer in 43% of the training labels.

Then we checked, properly

All of the above was exploration: we chose the best way to ask while watching the same ten mornings. When you look at the answer while you tune the question, you can fool yourself without meaning to.

So we wrote the tests down before running them, committed the plan to the repository, and only then made a single call. We did two runs, each on thirty fresh mornings nobody had ever run, with a correction for testing many claims at once, and a margin set in advance for what counts as “equal”: 1.5 seconds, smaller than the gap between the rule and taking turns. The plans, the results and the full analysis are public, in docs/confirmation.

All thirteen pre-registered claims held, including the fine-tune’s level in the second building. The effects shrank a little on fresh mornings, as honest effects usually do. And shuffling the cars made no difference to the fine-tune either: it had learned the facts, not the positions.

Two things we report as we found them:

• One earlier claim did not survive: On the first ten seeds, the fine-tune looked 1.6 s better than the load-aware rule in the second building. On fresh seeds it was simply level. We dropped the claim.

• Round robin surprised us: In the morning rush, taking turns was level with the classic rule and the best models, on both sets of fresh mornings. Simple things can be stronger than they look.

Speed and cost

• Speed: With Jev’s real 200 ms round trip counted in the simulation, it stayed within a second of the rule. For elevators, a blink doesn’t seem to matter.

• Cost: Jev costs about $0.00002 per decision at TypeSafe’s list price of $0.042 per million input tokens (as of October 2026; pricing). The whole study, about 25,000 Jev decisions, cost less than half a dollar. Laya and its fine-tunes cost nothing per call.

For pipelines, this suggests that latency and per-decision cost may not be the main obstacle for decisions that happen between jobs, such as picking a runner or choosing a test subset, where a fraction of a second is small next to a build that takes minutes. We did not test this on real workloads.

Results

Across three experiments on ten seeds, and a pre-registered re-test on fresh mornings in which all thirteen claims held, the answer came down to a few things: how the question is asked, whether the model has been trained, and what it is measured against. Five lessons follow. They hold inside our simulated building, and any link to your pipeline is an analogy, not something we measured.

1. What you show a model matters as much as the model. The biggest step came from giving Jev a number our code had already worked out.

2. Out of the box, the generalist wins. Jev was sensible from day one; base Laya was not, as its makers say.

3. A small open model can be taught to the level of a good rule in 24 minutes on a laptop, and then will run with no per-decision fees.

4. Beating the rule needs more than better labels. The model has to see something the rule ignores, such as recent calls on each floor. That is our next experiment.

5. Test on mornings you haven’t seen. Writing the test down first is cheap, and it is the difference between a hunch and a result.

Limits

• A simulated building with made-up traffic, two building sizes and three traffic patterns. The comparisons hold inside this world, not beyond it.

• “Equal” means within the pre-set 1.5 s margin, not identical. Results marked exploratory (the hindsight fine-tune, the collapse, real-latency runs, Jev vs Laya head to head) were not re-tested.

• One base Laya checkpoint, one training recipe, one device. Other settings may do better or worse.

• Identical requests are not identical chances: a model built to be fine-tuned, asked zero-shot in our format, was tested on our terms, not its own.

• We did not test any CI/CD workload. The pipeline connections in this article are analogies drawn from the elevator results, not measured findings. Elevator dispatch also has one clean goal (average wait), whereas pipeline decisions often balance speed, cost and risk at once, which is harder to benchmark.

TL;DR

In our small building, a simple rule held its own, and even plain turn-taking held its ground in the morning rush. Nothing here shows that decision models are needed yet. But these models are improving quickly, and I believe that for many everyday decisions a decision model will become the cheapest way to handle the cases our rules never imagined: the crowded minute, the broken car, the pattern nobody wrote down. That is my personal bet, not a finding. Only time can prove it, or prove me wrong.

If you build and run pipelines, here is what we’d take from the experiment:

• Present the data well: Let your code compute the key number (an estimated wait, a queue depth, a historical duration) and let the model weigh it. Handing it raw metrics performed about as well as no intelligence at all.

• Benchmark against the dumbest baseline: Round robin tied the best approaches in the rush. If a trivial rule matches your model, you don’t need the model.

• Hosted generalist or small open model is a real trade-off: The hosted model was sensible from day one but can’t be customized. The open model was poor until trained, then matched a good rule, runs offline, and costs nothing per decision.

• Training on your own history has a ceiling: A model that learns to copy your rules will, at best, equal your rules. To do better it needs to see signals the rules ignore.

• Distrust a metric that looks like knowledge: A model that always picks the same answer can post an impressive accuracy number. Compare every score with what a trivial guess would get.

• Evaluate on data you haven’t tuned against: Write the test down first, run it on fresh data, and be ready to drop a claim that doesn’t hold up. One of ours didn’t.

• Latency and cost are probably not the blocker: A 200 ms network call didn’t hurt, and 25,000 decisions cost under fifty cents. Quality of the decision, and how you ask, are what matter.

After ~45 experiments we reached a plateau: we were getting a consistent 85% +/- 3% pipeline fix rate in different runs, no matter how the agent fetched the failed pipeline context. As a result, we wanted to further examine the details of each run, asking ourselves the following questions:

• How many samples does the agent always fix?

• How many samples does the agent never fix?

• How many samples would switch fix-result run over run?

The results were a surprise: 60% were always fixable, 15% were never fixed, and 25% would be fixed in some runs but not in others. We named the last share flip rate: the percentage of tasks whose outcome changes across repeated runs on the same input. A high flip rate means any single run is unreliable: differences between experiments may be noise rather than a real effect.

The next question became: Why? What changed across runs that caused this variation of outcomes?

We looked at the different fixes the agent did to these swapping samples and the answer was more simple than we predicted: This variation was a result of agentic non-determinism; the agent would sometimes come up with the correct solution, but sometimes it would miss certain aspects of the fix, or try something completely different. Note that this is distinct from hallucination since agents were not making up a fix; they were looking at different paths that seemed valid and landed in different places.

With this in mind, we moved on to creating a solution. How can we make more consistent fixes? Which tools can help with this? As we started researching what other people were doing, a trend emerged: Helping an agent better manage its memory/context reduces the unexpected behavior and creates better quality outputs.

For more information on the research we were consulting to inform our work, check out code rabbit memory and Mendral.

Subscribe now

Hypothesis

The agent fails to be consistent across runs, so giving the agent an external memory should help reduce this inconsistent work.

Before stepping into developing a full context graph or anything too complex, I decided to try the least complicated implementation: a simple note-taking tool. I anticipated that this would give the agent a way to annotate its most important findings and read them at will, thus producing better and more consistent output.

Setup

In designing the project’s structure, we wanted a way to prove that what we added actually had an impact. As I’ve mentioned in a past article, I’m a fan of the scientific method and a big believer in repeatable structure. So, we designed this as a formal experiment with a baseline and a target to measure against. Also, due to the non-deterministic behavior of agents, we made sure to run the same experiment multiple times to validate that our results maintained across runs.

Two tools were added to the agent:

• A tool to save a note with a given key, similar to a dictionary.

• A function to return all the stored notes.

We tried three different setups:

1. Basic: Give the tools an instruction to “save each finding, read before finalizing”.

2. Formal read protocol: Tools and instruction to read before diagnosis and before validation (mandatory).

3. Auto-surface: When saving a note, it would return the full accumulated notes on every call, reaching context automatically.

Across the 25% fix rate samples, we tested this against Claude Code with Sonnet 5, running each setup three times to measure actual flip rate. Each run started fresh, so no context is shared across runs. To measure success, we ran the real pipeline to capture outcome, and we also used an Agent-As-Judge (Claude Opus 4.6) to evaluate the quality of the fix, focusing on if it was “mergeable or not” (we call this “merge quality”).

Results

The three mechanisms reduce flip-rate on a certain level, producing more consistently passing pipelines across attempts, but fix rate and merge quality had a different behavior across setups.

Basic setup dropped on flip rate, but hurt on quality. Looking into the behavior of the agent, it was writing a lot, but it almost never read what it wrote. This affected the solution’s quality, dropping from 72% from the control experiment, to 60%.

The second setup was the other way around, it achieved the biggest reduction in flip-rate (from 50% to 25%) and quality remained the same. Adoption from the agent was visible too, with all the sessions reading the notes as instructed.

Finally, the third setup also decreased the flip rate to 30% and improved quality by a small margin, proving again that giving the agent a place to better manage its memory/context helps to reduce the uncertainty of its behavior.

Since the goal was also to have the smallest impact possible in token usage, we tested the second setup with the full dataset, three times, to measure actual impact on reducing the flip-rate without having regressions. The results can be found in the table below.

The benefits are clear: reducing the flip rate by ⅓ without dropping fix rate nor quality, costing less than 10% more tokens per sample, and taking less than 5% more time.

Sidebar: In a separate experiment, we found that testing with Claude Opus 5 showed a fix rate improvement of +5% and quality improvement of +7%, at 36% higher cost, which is worth knowing before switching models.

TL;DR

It is clear that nowadays we cannot rely only on the agent’s context to get the work done.

As problems grow, external memory tools do improve on achieving more consistent results. These experiments back up that trend: reducing the variability of results with only a 7% increase on token consumption is a really good deal, rather than using a bigger (more expensive) model with a bigger context window.

Our findings here mean we can now consider other memory tools in our day-to-day agent usage.

Following my last role, I developed a personal habit: whenever a new runtime shows up (a microVM, AWS Lambda’s PHP runtime, etc.), I like to see if WordPress runs on it. Chunk sidecars (a remote microVM that’s usable with little burden on your local PC, right inside a dev session) felt like they could provide an ideal fit for my WordPress test.

This is how I used Claude Code to spin up a full WordPress environment inside a Chunk sidecar, what got in the way, and why the whole thing turned out to be more useful than I expected.

Hypothesis

Chunk sidecars are designed for inner-loop CI verification: sync your working tree to a remote microVM, run your tests or linting there, and validate before pushing. That’s the intended use. I wanted to see if the environment could hold something heavier.

My hypothesis was that Claude Code could set up a full WordPress installation inside a single sidecar with minimal input from me. WordPress normally wants MySQL, but running a second service inside a microVM meant more moving parts and more weight. SQLite (the same backend WordPress Playground uses) keeps the stack to one process. My assumption going in was that the lighter the environment, the better the odds the sidecar would hold.

Subscribe now

Setup

Chunk sidecars are designed to hand the contents of the microVM to an agent like Claude Code. So I delegated. I started by telling Claude Code the conditions: the snapshot ID of the microVM I wanted, and the requirement to keep the database lightweight. Claude Code can manage Chunk sidecar startup operations through the CLI, so I handed it the implementation from there.

Next came the database choice. A standard WordPress LAMP stack needs Apache, PHP, and a running MySQL or MariaDB instance. That’s a lot of moving parts for a verification VM. What is less widely known is that WordPress can swap its database backend to SQLite via the official SQLite Database Integration plugin; the same mechanism WordPress Playground uses to run WordPress entirely in a browser. I specified SQLite to keep the VM fast to start and easy to maintain.

Claude Code handled the setup: creating the db.php file, writing wp-config.php, and wiring the SQLite connection via the chunk command.

After 6 minutes, 42 seconds, WordPress was running inside the microVM.

Bonus data: A later experiment found that restarting from a snapshot (rather than building from scratch) took ~14–28 sec to reach a verified working state; roughly 27–29x faster than the from-scratch build.

Getting the sidecar up takes one command. There’s no image argument on this first run; nothing had been snapshotted yet, so a bare create provisions a fresh environment. From the second launch onward, you boot from the snapshot instead, with chunk sidecar create --image <snapshot-id>.

Results

WordPress came up cleanly. The next step was WP-CLI. Since the sidecar runs in an isolated VM with no inbound internet access, I wanted it for demonstration purposes rather than production use.

That is where Claude Code’s auto-mode stopped me:

Permission denied by the Claude Code auto mode classifier.
Reason: [Code from External] The agent downloads and executes a wp-cli.phar binary
from raw.githubusercontent.com (an external, agent-chosen source) ...

Auto-mode will not download and execute external binaries on its own. I approved it manually, and the install proceeded. After that, things moved quickly:

wp plugin install hello-dolly --activate
wp post create --post_title="Test post" --post_status=publish
wp post list

Posts created, listed, plugin activated. Full WordPress environment, running inside a remote microVM, operated entirely through CLI.

Worth noting: I took a snapshot after the initial setup, then created a new sidecar from it and it came up already fully working (Apache running, WordPress serving over HTTP 200) without re-running any of the install steps. I also re-tested this separately with 5 restarts from a snapshot, all successful, previously-created posts still present each time.

TL;DR

I went in thinking this was a curiosity exercise. I came out realizing Chunk sidecars might be a serious option for WordPress plugin development.

Here’s why: once WordPress is set up inside the VM, you can snapshot that state. The next time you need the environment, it starts from the snapshot instead of from scratch. Combine that with Chunk sidecar sync to push your local plugin code into the running VM, and you have a tight loop: edit locally, sync to the sidecar, run PHPUnit there, get results. No Docker daemon pulling CPU on your laptop. No Vagrant box eating RAM.

The part that makes it interesting for plugin work specifically is parallel environments. Because sidecars are remote and lightweight, you can run multiple instances simultaneously against different PHP versions. That is harder to manage with a local Docker setup.

By way of proof: I built three snapshots (PHP 8.1 / 8.3 / 8.4, each with WordPress + SQLite Database Integration installed) and launched all three sidecars simultaneously. There were no conflicts: no port collisions, no resource contention… all three came up clean. Verified php -v on each matched the intended version, and each WordPress instance responded with HTTP 200.

One caveat: .chunk/config.json only holds a single validation.sidecarImage value. There’s no built-in way to map multiple snapshots to different PHP versions. So running this in practice means either passing --image <snapshot-id> explicitly each time, looking snapshots up by name via Chunk sidecar snapshot list, or wiring the IDs into a CI matrix config.

The auto-mode block on WP-CLI was worth noting too. It was auto-mode doing exactly what it is supposed to do: pausing before it runs an external binary it sourced itself, rather than executing it without asking. One manual approval, and it moved on. That’s the right behavior for an agent running in a verification environment.

When a pipeline came back red, the local fix loop got triggered: read the failure, patch it, push again.

That loop is where an AFK run burns dollars and wall-clock nobody budgeted for. And a fat slice of that bill is nothing more than the CI pipeline failure context we shovel into the model when the agent asks for it.

So I ran the build again and changed exactly one thing: what the fix agent reads when CI goes red. Push cadence held still. Same model, same task list, same expectation (a merge-ready green PR at the end of the run).

One arm was given the entire raw CI pipeline failure output. The other got a compact, structured report from the following CircleCI CLI command: circleci run get --failure-report.

Full pipeline failure output: ~800,898 characters.

--failure-report: ~19,936.

About 40x less stuff stuffed into the prompt.

Across both arms, fix loops stayed at 3, and pipelines stayed at 8.

That is the finding. Compact, structured failure context makes a retry cheaper. It does not, by itself, make retries disappear.

Nothing here prevents the red. It makes the red cheaper to read.

Subscribe now

Hypothesis

If the CI fix loop reads a compact, structured failure report instead of the whole log dump, CI pipeline failure diagnosis input tokens should drop. Retry count should not.

Same red pipelines fed to the fix agent. Thinner diet.

Setup

Same Snake build both arms. Seven tasks. AFK. Push after every task that cleared inner-loop checks.

The variable was failure shape, not how often we triggered outer-loop CI.

• per-task-push (full log): when a pipeline went red, the fix agent fetched the entire CI pipeline failure output.

• per-task-push --failure-report): same cadence. Failures shaped with the CircleCI CLI command circleci run get --failure-report. Compact. Structured. Sized for an agent.

Everything else that could wander, I tried to hold still: model, task list, merge-ready green PR at the end of the run.

Results

Both arms finished green. Both still took the outer pipeline eight times. Both still ran the fix loop three times.

So why care about fewer failure-log characters if cost, wall-clock, pipelines, and fix loops look basically the same?

Because the totals are misleading.

Most of that ~$15 bill is the agent writing the game, not reading CI failures. That build cost jumps around run to run (call it $10 to $17). So even when we cut failure-log input from ~800k characters to ~20k, the total check can only drop about $1 to $2. Wall-clock barely moves either. Reading a thinner failure report might save milliseconds or seconds, not minutes.

The savings are still real on every fix. Every time the agent has to diagnose a red pipeline, you stuff ~40x less into the prompt. That is input tokens you do not buy. On this pair of runs the whole-run LLM bill moved from $16.75 to $14.27. Do not hang a strategy on that exact delta. Codegen variance is loud. The quiet, repeatable tell is the input diet: ~800k → ~20k characters of failure context, with zero change in how often we came back for another CI pipeline run.

Extremely detailed feedback on every attempt. Same number of attempts. Cheaper to read each time.

TL;DR

--failure-report is graded comments, not the whole exam booklet: failed steps, condensed output, enough to fix without hauling the universe into the context window.

So treat it as hygiene. Use the flag. Always. Pull it from the CircleCI CLI, pipe it into the fix loop, point the agent at it, and stop letting that loop read raw logs. It belongs in any decent agent harness. For the longer cut on tokens and fix time versus raw logs, see the --failure-report write-up.

Cheaper retries are good. Zero retries are better.

If the goal is a green PR in one take, this flag will not get you there. Showing the agent what CI will check before it writes product code will. That is a different experiment, and it is the one that drove fix loops from 3 to 0.

Clicking the same link again sometimes worked. Same link, same machine, different result. Raise your hand if you love debugging exasperating intermittent issues! No one? Ok, let’s move on.

The prompt passes through a command line. Inspecting the running process showed a command line exactly 1024 bytes long, not 1025 or 1023. And software engineers know that a round number like 1024 points to a full buffer somewhere.

Breaking down the command:

113 bytes  program path + flags
911 chars  encoded prompt, all remaining data
---------
1024 bytes  hard stop

Those 911 encoded characters unpack to 683 real ones, matching the truncated prompt character for character.

Subscribe now

Hypothesis

A browser click does not start Claude. The browser hands the URL to macOS, macOS hands it to the registered handler, and the handler opens a Terminal window and types a command into the shell, encoded prompt and all, as one long line.

So the prompt has to survive the connection between Terminal and the shell in that window. The same channel your keyboard uses.

That channel was the suspect. Stated plainly, the hypothesis had three parts:

• The channel holds about 1024 bytes. Anything beyond that gets dropped, with no error.

• The drop only happens when the shell is not reading at that moment. A shell that reads continuously drains the channel and lets any length through, which would explain why the same link succeeds and fails.

• Therefore the prompt survives only while 113 + encoded length <= 1022, or 681 characters.

The 681 figure came out of (1022 - 113) / 4 * 3 before any test ran.

Setup

I could have planned many experiments, trying to foresee limitations and potential routes and optimize based on what I thought might be quicker and more robust. But I would always rather go directly into experimentation mode, invest that planning time in finding leads on the ground, and iterate from there.

So I started by manually replicating this intermittent issue. Once I found where Claude got an actual truncated prompt, I asked it (given the relationship we’ve built over the years, it feels weird calling Claude “it”, but for the sake of professionalism I’ll stick to that) “check the parameters/arguments used to open this Claude instance, and figure out why the prompt was only prefiled with this instead of the whole prompt”. From there, I dove into the rabbit hole, scripting the browser’s behavior:

• Fire the same URL macOS would receive from a click

• Snapshot running processes before and after

• Capture the full command line of the launched process

• Decode the prompt and compare received text with sent text

• Kill only the processes the script started

That script measures two hops: what the handler built, and what the launched program received. If the two differ, the loss happened in the terminal hop.

A second script went after the boundary directly. It built the real command line for prompts of every length, pushed each one through a real channel held in the failing state, and bisected the largest prompt that survived.

About 400 measured runs in total, all on the same machine.

Results

The two hops differed. The handler always built the full 1388-character payload; the program sometimes received less. The loss happens in the terminal hop, as predicted.

1. Is the channel the bottleneck?

I fed some text while the receiving program was not reading, then counted what got through.

The cap holds at 1022 bytes no matter how much gets sent. Retrying the send does not help. Once the buffer is full, it stays full.

2. What makes some runs work?

One condition delivers everything: the receiver has to be actively reading at the same moment. Shifting the start of reading by a few hundred milliseconds flipped otherwise identical runs. Ten arrived full, fourteen truncated.

3. Where exactly is the line?

681 characters, exactly where the formula put it.

What this means

All three parts of the hypothesis held. The channel between Terminal and a shell holds about 1024 bytes. If the shell is reading when the command lands, it drains input continuously and any length gets through. If the shell is busy for a moment, redrawing or running a prompt hook, the buffer fills, the channel drops the overflow, and the shell runs a cut-off command.

No error appears anywhere. The prompt is just shorter than the one you sent.

Timing decides, which is why the bug is rare. Twelve emulated clicks in a row delivered the full prompt. Two clicks four minutes apart in the original session did not.

Two false alarms

Both wrong answers came from broken measurement.

The emulator matched the wrong process ID and decoded with standard base64 when the encoding is base64url, so it reported “truncated” on runs where the full prompt had arrived. Then a timing bug in the second script let the receiver wake 193 ms in, inside the sender’s 200 ms deadline. Everything passed, and the reported limit came out at 1040 characters, which meant nothing.

Problematic, because a test that passes for the wrong reason is worse than one that fails.

The fix

Prompts below 681 characters never hit this. Above 681, success depends on timing, and most clicks still work, which is precisely what makes it hard to diagnose.

For our experimental case (opening prefilled Claude Code prompts from the web) the honest answer is: don’t go this route. 🙂

For CLI scripts, keep long prompts off the command line:

claude "$(cat prompt.txt)"
# or
cat prompt.txt | claude -p

Both skip the terminal channel entirely, so the limit never applies.

TL;DR

Clicking a claude-cli:// link makes macOS open a Terminal window and type a long command into it, prompt encoded inside. That typing channel holds about 1024 bytes. If the shell is not reading at that exact moment, everything past the limit is dropped with no error and Claude opens with a half-finished prompt.

The cutoff is 681 characters. Below that, prompts always arrive. Above it, most clicks still work, which is why the bug looks random.

Don’t prefill long prompts from the browser. In CLI scripts, use claude “$(cat prompt.txt)” or cat prompt.txt | claude -p. Neither goes through the terminal.

Median teams sit around 3.9. The top 5% run about 2.6. An elite cohort of twenty orgs is already near 1.3.

Most of the industry is still grinding through several rounds of feature-branch rework before earning the right to merge to main. The leaders are closer to one.

When coding agents are the ones waiting on those rework cycles, MER is not a vanity metric. It is the clock and the bill. Extra loops mean more tokens burned, more CI credits spent, and more agent context rotting on a red light from CI.

The Q2 Pulse Report is blunt: lower MER is a DX win, an AX win, and a cost win.

I ran a small AFK lab experiment that rhymes with that report. Not an org-level MER study. Same shape of problem, inside a single agent setup.

Subscribe now

Hypothesis

If a green PR is already likely, pushing only once at the end should burn far fewer outer-loop CI credits than pushing after every task. And if deterministic checks live on the inner loop once, you stop paying time and money three times for the same linter.

Pushing once at the end is only safe once the agent can reliably go green.

Setup

In our core experiment to determine the cost of a green PR, we found that if you give the agent a sneak peek at the CI setup, let it take notes before coding begins, and let it do a practice run on inner-loop checks before pushing to outer-loop CI, you can achieve 100% green PRs with zero rework.

The sneak peek is actually quite simple. Before task one, before a line of game code gets written, the agent receives a short inventory of the CI setup: which checks fire on the inner-loop practice run, which jobs fire on the outer-loop CI pipeline, and pointers to the scripts and configs that define them. This information is descriptive, but not an answer key. The agent writes its own notes from that inventory preflight.md) and can @-read those configs whenever it wants more. In our codebase, the inventory is a “CI validation manifest,” which tells the agent what will be graded and where the grading logic lives, before it starts guessing from vibes.

Fix loops drop to zero. Green PRs go to 100%.

This piece starts there. Same AFK Snake game build. Same seven tasks. Same two cadences, now with the sneak peek on.

• Per-task-push: a pipeline after every task. Seven tasks, seven trips.

• Single-push: finish the stack, clear practice run, push once at the end.

How often you push is the lever under test. First you have to be equipped to push green.

A linter is a linter. Compute is compute, whether it runs on a laptop, in a Chunk sidecar microbuild, or in a cloud job financed with CI credits.

There is no real reason to run the same deterministic check on localhost, again on the sidecar, and again on the outer-loop CI pipeline, as if three identical stamps make the code more correct.

That stack is a relic of human by-hand engineering. A person in flow forgets to run the test suite, pushes red, and we paper over forgetfulness by re-running the universe on every layer. Coding agents do not forget the same way. You can (and should) instruct them to clear inner-loop checks before they spend a single outer-loop CI credit.

In this calibration we moved inner-loop checks fully onto the sidecar and left thick outer-loop CI as the honest final test. Slimming true duplicates on the outer-loop surface is later work. The principle does not wait though: do not triple-pay for the same answer.

Results

Headline: once sneak peek made 100% green likely, a single end-of-run push cut estimated CI credits from ~81 to ~14.

• Per-task-push: 7 pipelines. ~81.4 estimated CI credits.

• Single-push: 1 pipeline. ~14.0 estimated credits.

About 5 to 6x fewer CI credits on the single-push run.

Wall-clock stayed in the 50 to 60 minute band either way (55.5 vs 56.7). LLM dollars stayed in the $15 to $17 band (17.15 vs 15.73). Tokens still track agent work. The CI credit collapse comes from the push cadence, not a cheaper model.

That bargain only holds when a sneak peek plus inner-loop checks make a 100% green PR likely. If you pile up unproven work and trigger the outer-loop pipeline once cold, you have not saved anything. You have postponed finding out you are red. When that one pipeline fails, you still pay the fix loop, and a retry can pick up new last-mile surprises.

Earn one take. Then trigger outer-loop CI once.

TL;DR

Elite engineering teams do not win by loving rework. They win by needing less of it.

Think of agent-era validation as a continuum. Inner loop to outer loop, hybrid on purpose. Each check earns its place.

• Deterministic checks live inward: cheap, early, once.

• World-shaped risk lives outward: policy, last-mile, flaky integrations, the CVE that did not exist at 9am. You name it.

Place each check where its information is unique. Everything else is just another duplicate cost for the same answer.

Outer-loop CI should still exist. It is where the world gets a vote. It should not be a museum of jobs you already passed in identical form on the previous two layers of validation.

At AFK scale, the inheritance path looks like this:

1. Make preventable reds disappear (push a /green PR in one take).

2. Keep remaining retries cheap (use the --failure-report flag).

3. Put deterministic checks on the inner loop once. Push to outer-loop CI as often as your green rate deserves.

Once 100% green is something you can count on, pushing only once at the end of the run is an obvious CI credit win.

Bake that into your agent setup. Low MER should not be a hero dashboard you inspect after the damage. It should be the default the agent loop was engineered to produce.

Rework should be reserved for unknown failures from the real world. Preventable failures should be totally eliminated.

Our current CircleCI APIs use version numbers in the URLs. V1, V1.1 and V2. They had accumulated four incompatible URL patterns, three ID formats and a different response shape per endpoint, effectively ten years of drift. The pitch for a new API V3 was explicitly about consistency for agents: one URL grammar, UUIDs everywhere, one response envelope, one error shape. An LLM should be able to predict /api/v3/{entity}/:id from the entity name alone instead of carrying the whole spec in its context.

We followed all the current advice on how to make the API discoverable by agents. We published the OpenAPI spec at a stable URL. We added /.well-known/api-catalog per RFC 9727, so a machine could discover the machine-readable descriptions. We generated an llms.txt and an llms-full.txt. We rendered the whole reference as markdown as well as HTML. We put the conventions - error shapes, idempotency, which operations are irreversible - into the spec description so they travel with every entity document, not just the index.

This post answers one question: did any of that work?

Hypothesis

My assumption at the outset was that agents would want to work with a consistent API, and that they would be able to easily find it.

I expected consistency to be the hard part and discovery to be the easy win.

Specifically:

1. Agents would find V3 on their own through the discovery surface - the catalog, the OpenAPI spec, llms.txt.

2. Once they found it, V3’s consistency would keep them there, because guessing a V3 URL is easier than looking one up in V2.

3. V2 usage by agents would decline as agents discovered the better option.

The first and third were wrong in ways more interesting than being right.

Setup

I first proposed building a new consistent API 3 years ago. At the time, it would have been a massive time expenditure likely not worth the investment. But by the time I pitched it again earlier this year, the execution had changed; we could use a swarm of agents with good skills. Agentic workflow enabled us to decide that it was a low enough cost to take on an experiment previously considered to be very high cost, and impacting every team on the org. I got the green light.

I tend to be more pragmatic than scientific. I understand the scientific method, but can also plan ahead to mitigate the risks of just diving in and trying things. In the case of the API, although it’s fairly high risk in some ways, I knew that if we didn’t publicize it fully, we could always stop and roll back.

Every V3 route ingresses through one gateway service. That was a deliberate call made for enforcement reasons, and it is also what makes this measurable: one dataset holds every public API request with its route, method and user agent.

After agreeing on the consistent flexible API shape, we built skills internally for teams to use to guide their agents, and within a few months we had V3 APIs in place that could replace 90% of existing API traffic.

Measuring agent traffic hitting our API is an inexact science, relying on parsing the User-Agent in the inbound requests.

Real strings from production for requests coming in via our CLI:

circleci-cli (darwin/arm64; 1.0.48773; claude-code_2-1-246_agent)
circleci-cli (darwin/arm64; 1.0.48692; codex)
circleci-cli (darwin/arm64; 1.0.48773; opencode)
circleci mcp (755dd082f9ef571f74ac433160ab068e10c9ef25)

That is self-reported identity, not inference from traffic shape. If you want to measure agent usage of your own API and you control a client, stamping the harness name into the user agent is the highest-value thing you can do. It costs one line and turns an unanswerable question into a query.

Two caveats to keep in mind if you try something similar: An agent that hand-rolls its own HTTP calls is invisible to this method. I checked the bare-HTTP-client population and found roughly 41,000 spans a day where an agent could hide indistinguishably from a shell script (the span counts come from a sampled tracing dataset: the agent-filtered queries returned unsampled, the whole-traffic ones at 1.7× mean sample rate.)

Results

Interestingly, we realized the agents don’t care about the API directly. Instead, their focus is the MCP and CLI. Having a consistent API made it easier to build a good CLI. It’s only in this roundabout way that the agents are interested in using the API.

Agents overwhelmingly use V3. But not by migrating.

Hypothesis 2 confirmed, emphatically. Now the same measurement seven weeks earlier:

In those seven weeks, agent V3 share went from 57% to 97%. But look at the V2 column: 17,641 to 22,375. V2 agent traffic did not decline. It grew slightly. V3 did not win by converting anyone. It won by absorbing 760,000 calls of agent traffic that did not exist seven weeks earlier.

Hypothesis 3 was wrong: the migration story I expected to tell was actually a growth story.

For contrast, across all clients including browsers and scripts over that same recent week, V2 still outweighs V3 - 24,067,618 calls against 16,978,477. Humans and legacy integrations have not moved. Agents are a separate population behaving differently.

No agents found the V3 API on its own

Here is the entire discovery surface over thirty days. Every file we published for machines to find:

Discovery endpoints · All clients · 30 days to 28 Aug 2026

Agents made 807,209 API calls in a week. The entire discovery surface got 6,776 hits in a month.

Then I broke those 6,776 hits down by user agent, expecting a long tail of coding agents. There are three populations in there, and none of them is the one I was looking for.

Population one: nobody in particular

Crawlers, scanners, and humans

The largest single consumer of /.well-known/api-catalog is a bot that exists to grade whether your API is agent-ready. We are being audited for agent-readiness by a crawler while the agents themselves never look.

Population two: agents, but only when a human points at them

These are visible and separable, because a directed fetch carries a different user agent than an agent doing its own work:

Claude-User (claude-code/2.1.220; +https://support.anthropic.com/)

That is Claude Code’s web fetch, which fires when a person says “go read this URL”. Across the whole discovery surface in thirty days it accounts for 23 requests - 6 on the static OpenAPI HTML, 5 on /docs/api/v3, 4 on /fullopenapi.json, 4 on /.well-known/api-catalog, and single hits elsewhere. Two more came from Claude-User/1.0, the claude.ai equivalent.

So the honest version of the finding is not “no agent has ever read it”:

Zero agents read the discovery surface unless a human explicitly directed them to it. Twenty-three times in thirty days, someone did.

Every one of those 23 was a person (probably one of us) saying “fetch the catalog”. Not one was an agent concluding on its own that a catalog might exist and going to look.

Population three: the training and search pipelines

This is the part that gives me some hope. Around 203 of the 6,776 hits are corpus builders and search crawlers:

They are all reading /openapi.json. The spec is being ingested, but just not into a working agent’s context. It is going into the next generation of training data and the search indexes agents fall back to. The discovery surface is on a much slower clock than I assumed.

Which means the optimistic reading of this whole post is available, but it is not the one I would have guessed. It is tempting to hope that the next round of models will start consulting catalogs and llms.txt properly, the way the standards intend. I do not think that is what will happen. What will happen is that GPTBot and ClaudeBot read our openapi.json, V3 ends up in the weights, and the next generation answers V3 from memory… exactly the way this generation answers V2 from memory.

The fix for stale training data is more training data. The catalog’s delivery vehicle is the corpus, and the crawler is the courier.

So the discovery surface does pay off. Just not as discovery. And the loop it runs on is a model generation long, which is not something you can shorten by publishing harder. The only lever you control on that timescale is still the client.

Why agents don’t discover: two layers of stale

Watching agents work with our API, the failure has two stages and neither involves discovery.

STAGE 1: Training data answers first

Ask an agent about the CircleCI API and it answers immediately and confidently with V2 shapes - /api/v2/project/{project-slug}/pipeline, slugs in paths, the old status enum. It is not looking anything up. V2 has been in the corpus for years; V3 has existed for months.

STAGE 2: The search index agrees with the training data

Agents will often announce they are searching the web when they are reaching into training data. When they do genuinely search, the indexes rank a decade of V2 documentation, V2 Stack Overflow answers and V2 blog posts far above anything we published this year. Two independent mechanisms, same wrong answer.

And here is the twist that makes discovery standards nearly useless in this shape: by the time an agent finally reaches a spec or a catalog, it already believes it is looking for V2. It arrives with a target. We updated the spec descriptions to say prominently that a newer V3 API exists, and an agent that has decided it needs the V2 pipeline endpoint reads past that to go looking for the V2 pipeline endpoint.

An in-context hint only helps a reader who has not yet decided what they are looking for. An agent has always already decided.

What actually moved the traffic

We updated the CLI to use V3, and pointed the built-in MCP server at the same code.

My colleague Pete has written up the rebuild this came out of (Rebuilding the CircleCI CLI from scratch) and it is worth reading alongside this one, because his half of the story is the half that actually moved the numbers below.

Roughly 57% of all CLI traffic to the public API now carries an agent harness stamp. Every one of the 807,209 agent calls arrived through the CLI or the MCP server. None arrived from an agent constructing its own HTTP request.

Agents do not discover APIs. They use the tools already in front of them. The CLI is already installed, already authenticated, already on the agent’s PATH, and --help is right there. The MCP server appears in the tool list. Neither requires discovery, a search, or a spec.

The lever was never the discovery standard. The lever was the client.

TL;DR

No agent reads your discovery surface unless a human points at it. The catalog, llms.txt and the OpenAPI spec drew 6,776 hits in a month. Agent reads: 23, every one a directed fetch. Self-directed: zero.

Publish it anyway, but price it as a training-data bet. 203 of those hits were GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot and MistralAI-User. The spec is being ingested, just not into a working agent’s context.

Expect the fix to arrive through the weights, not the standard. The next model generation won’t consult your catalog. It’ll just already know your API. That’s a model-generation-long loop you can’t shorten by publishing harder.

In-context hints only help a reader who hasn’t decided yet. We put “there is a newer V3” in the spec description. An agent that has already concluded it needs a V2 endpoint reads straight past it.

Ship the client and the MCP server. 100% of measured agent traffic arrived through a wrapper that was already installed and already authenticated. None of it was hand-rolled HTTP. If you want agents on your new API, update the tools they already have.

$ circleci run trigger --help | head -40

I hadn’t asked for head -40. Claude added it, and it does that constantly; help output is untrusted input of unknown size, and pulling an unbounded page into a context window is a bad trade. Forty lines is the interface. Below line 40 was the part that tells you how to use the command.

Hypothesis

As I shared in a previous blog post, when I started rewriting the CLI, I already understood we were writing for agents; that was the premise of the rewrite. Every data-returning command has --json with its fields enumerated, and the command tree generates an MCP server, so each command’s description is a tool description.

Two things we didn’t know:

1. We didn’t know there was a length limit, so we never measured against one (”written for agents” was a set of content decisions, and none of them had a size).

2. We didn’t know that we were saying everything three times: the accepted values for --event-preset, the default for --provider, all of it in the examples, in the flag table, and again in the prose. Each section had been written to stand on its own, and nobody diffs a paragraph against a table.

So the fix wasn’t “write less”. If the length was repetition, most of it would come out mechanically; and if prose went last, truncation would cut the cheapest content rather than the most expensive.

Setup

I measured before editing anything, because “is this help text too long?” is otherwise a taste argument nobody wins. Walk the command tree, render every page the way a real invocation would, and count. Not total lines, but whether each section finishes inside the first 40. A flag table that starts on line 38 isn’t visible; it’s a teaser.

Across 169 command pages, the mean was 49 lines, the worst 91, and only 15% got their examples inside the window (the section an agent actually copies from).

Three passes. Boilerplate out: the banner, the per-page table re-printing the same four global flags, the repeated “Learn More” links. Seventeen lines a page, no prose rewritten. Reorder: Short → Usage → Arguments → Flags → Examples → Details, so truncation eats prose first. Trim the duplication, never dropping a flag row or an example to make space. project trigger create went from 91 lines to 42.

Then keep it fixed. Order and boilerplate are properties of the help template now; length is a test that asserts every page fits 40 lines, with individual caps for the twenty-five commands that genuinely can’t, and those caps only ratchet down. And since these commands are largely written by an agent working from guidelines in the repo, the rule went there too.

The PR: https://github.com/CircleCI-Public/circleci-cli/pull/1667. 449 files, about 2,600 net lines of help text deleted.

Results

The three passes cut both the mean and the worst case, but what really matters is whether the sections an agent actually uses (flag table and the examples) finish rendering before truncation. A page that runs to 70 lines but front-loads its flags is a better interface than a 45-line page that buries them. The reorder pass was designed around this: put the cheap content last, so if something gets cut, it isn’t the part Claude copies from.

No task-success delta, because I don’t have one. What changed is the interface: every command’s complete flag list now sits inside the window an agent reads, and three in four get their examples there too, against one in seven before. The old failure mode was silent, which is why it lasted. Claude wasn’t erroring; it was succeeding at a degraded version of the task on whatever fragment of the flag list it had seen. Worse than a stack trace, because nothing tells you to go and look.

TL;DR

“Designed for agents” is a budget, not a style. We had the content decisions right and still shipped help that couldn’t be read, because we never asked how much of it arrives.

Pick one canonical place for each fact. The duplication wasn’t sloppiness; it was three self-sufficient sections maintained separately. The flag table is canonical now, and prose restating it is a defect.

Make it a test. A style guide asking for 40 lines would have lasted a month; the next person adding a command, human or otherwise, has no reason to know the number exists.

The pages are better for humans now, which I didn’t expect; not because agents and people want the same thing, but because neither mistake was agent-specific. Nobody benefits from reading a flag’s values three times, and nobody was reading line 70 either.

At roughly the same time, I’d been spending time considering how I might be able to use Chunk sidecars for related tasks outside of testing. I wondered: Do sidecars have the potential for a wider application?

I’d say that’s representative of my style as a whole: When conducting experiments, I don’t just think about the results, I try to think about new ways of conducting the experiment that can be applied to other experiments in the future to create additional efficiencies.

In the case of this specific experiment, I was fairly certain that the problem would be easily divided and conquered by using multiple computational resources. Without that approach, the experiment would be costly and cumbersome. Chunk sidecars provided the best route forward, and resulted in a method that I (and any other CircleCI customer) can replicate within future experiments too.

Hypothesis

Config validation is embarrassingly parallel. Each file is independent. There’s no shared state, no ordering requirement, no reason one file’s result should wait on another’s. If we could distribute validation across a fleet of Chunk sidecars, the wall-clock time should drop to roughly: (time to validate one file) / (number of sidecars running in parallel).

The biggest question was how to keep this cost effective from a time expenditure perspective, given that sidecars take about 30 seconds each to spin up. If each sidecar only validated a handful of files before its setup overhead dominated, the fleet wouldn’t be faster than sequential. It would just be more expensive.

Setup

Chunk sidecars are lightweight remote microVMs that run microbuilds alongside a developer’s work session. Each one is a full Linux environment: it can run the compiler, execute arbitrary code, and report results back.

It was important to me to conduct this work in a way the customer could reproduce. Don’t just trust me – try the same validation on your end as well to prove that the pipeline won’t be broken when the change is introduced.

Makoto Mizukami, Senior Field Engineer JAPAC, CircleCI

Knowing that the customer wouldn’t have access to our database (making the experiment potentially costly for them to run themselves) I made sure to use resources that would be available to them as well – the CircleCI API and CLI.

For this experiment, I partitioned the customer’s 10,000+ configs into batches and assigned each batch to a sidecar. Each sidecar:

1. Spun up

2. Received its batch of projects

3. Walked through each project to fetch its config through CircleCI API and run it through circleci config validate--next

4. Reported pass/fail results, and failure reasons if any

The fleet size was 100 sidecars in parallel at maximum. Task orchestration, command dispatches, and result aggregations were all done by a single Claude Code agent session, allowing me to save tokens (unlike Claude subagents, which hungrily consume tokens). The work was instrumented with Honeycomb throughout: 32,359 spans across the full run.

Results

The fleet economics held up. Setup overhead for the 100 sidecars was 2 minutes in total — small enough that the parallelism paid off at this batch size.

One finding worth noting: We also observed some specific patterns in validation failures. For example, certain series of strings tended to raise a validation error. By analyzing the result of the large-scale process further, we were also able to identify some “template” projects, from which most of the 120 configs derived.

The reason that the setup was reasonably easy was that Chunk sidecars provide snapshots. I could install a basic toolset in the sidecar, take a snapshot and apply the snapshot to the sidecars. Without that, this experiment would be impossible to conduct efficiently.

TL;DR

Embarrassingly parallel tasks don’t need clever algorithms. They need an environment that can run many agents at once. The compiler logic didn’t change. The validation logic didn’t change. What changed was the execution model: instead of one process working through a queue, a fleet of processes each worked through a slice.

Running them sequentially would have taken hours. A Chunk sidecar fleet did it in 4 minutes.

The thing to check before you do this: Make sure each agent knows exactly what its job is. Per-agent setup costs compound. If your task is trivial and your setup is heavy, the fleet won’t help. In this case, validation was fast enough and setup was cheap enough that the math worked. Instrument it first and check.

The customer got their answer before the breaking change shipped. It was an answer we could never have gotten in a timely manner without the massive parallelism Chunk sidecars offered. It proved my hypothesis that Chunk sidecars have multiple uses, and aren’t just limited to testing. As it turns out, sidecars are well implemented so that agents can easily leverage and consume the data, making them a perfect vehicle for experiments like this one.

We complain a lot about AI agent output. But we need to be thinking more about how we’re setting our agents up for success, and whether we’re providing them with sufficient power to do their job. Sidecars are one of the best ways to prepare agents to do great work.

Then suppose that same student could peek at the test in advance and take notes.

One catch. The sneak peek does not include any of the last-minute trick questions that might show up on the real exam without warning. Those get written by “the world” the moment the exam starts.

Nobody on the teaching staff knows the trick questions in advance, and new ones can show up on a retry.

Pencils down.

That is coding agents and CI. Merge-ready green is the only passing grade. Outer-loop CI is the real final exam. A Chunk sidecar is the timed practice run. Every red→green fix loop (diagnose, patch, push again) charges a little tuition: LLM dollars, CI credits, wall-clock.

I wanted to know what a green PR actually costs on a fully-automated AI coding agent run. Time. Tokens. Credits. The whole bill.

We moved the needle a bit on time and tokens. Fine. Not the story.

The story is rework going to zero on the stuff we can predict. Deliver the PR in one take. Ace it at 100%. The only red you should tolerate is the unknowable.

Hypothesis

If you show the agent what CI will check before it writes product code, give it a page of notes, and make it pass a real practice run on a sidecar, preventable outer-loop failures should stop showing up. Fix loops should not be necessary. Green commits should hit 100%.

Not “prompt harder.” Setup for success. Ace it at 100%. One take.

Setup

In this lab, an A- is still red. Local lint that smiled while outer-loop CI frowned still fails. A green sidecar run that dies on the real pipeline still fails.

Two cadences. Same bar: every commit you push to outer-loop CI is green.

• Per-task-push: after each task clears practice run, push. Seven tasks, seven trips. Ace every one.

• Single-push: finish the stack, clear practice run, push once at the end. Ace that one.

How often you push is a later lever. First you have to be equipped to push green.

Without the sneak peek

Three calibrations on the same AFK classic Snake game build. Seven tasks. Hands off the keyboard.

No inventory of checks up front. No notes. Inner-loop checks existed, but they were split between the localhost and a sidecar.

They finished, but rework kept piling up. When we feed agents endless retries and mountains of feedback, we’re just repeating the same habits we once drilled into human developers, now adopted by their AI counterparts.

*NOTE: We can make retries cheaper with a structured CI pipeline --failure-report, but that is its own write-up. Spoiler: even if you cheapen input tokens on each retry, you still have to pay time and money to retry.

Then we showed it the test

Before task one, before a line of game code, the agent got a short inventory: which checks fire on the practice run, which jobs fire on the thick outer pipeline, and pointers to the scripts and configs that define them. Descriptive. Not an answer key.

It took notes (preflight.md). It could @-read those configs when it needed more.

In our codebase that inventory is a “CI validation manifest”, which basically tells the agent what will be graded, and where the grading logic lives, before it starts guessing from vibes.

Inner-loop checks moved fully onto the sidecar, and we stopped running the same exact lint and test loops on the localhost. Outer-loop CI stayed the honest final exam, including whatever the world still wanted to throw at it.

Results

Headline: 0 CI fix loops on both sneak-peek arms.

• Per-task-push: 7/7 pushed commits green.

• Single-push: the one end-of-run push went green on first contact.

Zero outer-loop CI failures on either arm.

Diagnosis tokens on the rework line: $0. That budget can fund first-pass product code generation instead of burning LLM dollars and CI credits on red -> green archaeology.

Wall-clock did not collapse. Still about 50 to 60 minutes. Coding time dominated. LLM dollars landed about $15 to $17, in line with earlier attempts. The key difference: rework nearly disappeared.

Quiet rework is the same muscle Merge Efficiency Ratio names in CircleCI’s State of Software Delivery Q2 Pulse report: how many validation cycles before a change is actually done. This lab is that problem inside one agent setup. Once green is likely, push cadence and where you place the checks are how you bake low MER into the loop.

TL;DR

A coding agent can only prevent what we can predict. Say that out loud.

Linters. Unit tests. Typechecks. Formatters. Lockfile rules. Contract tests you already wrote. The deterministic stuff with a known answer. If it is on the syllabus, the agent can study it, practice it, and clear it before you spend an outer-loop credit.

It cannot prevent what the world invents after the peek is printed.

Late-breaking CVEs mid-run. A registry serving a different tarball than the one you resolved an hour ago. A base image that rotated overnight. An org policy that fails a dependency you did not touch. A flaky third-party API. A secrets scanner lighting up on a fixture. A new advisory. A mirror outage. A quota. Runner drift. “Works on my machine” that is really the cloud’s clock, certs, or DNS.

Platform engineers, harness authors, and coding agents cannot predict this. It arrives when the CI pipeline is already running.

Design for what the agent can own. Leave outer-loop CI in place for the rest. Ideally one take. At most two, once the world gets a vote.

Build the agent setup so it can ship the application end-to-end with zero CI failures from known deterministic sources.

Inventory of checks. Notes. A practice surface that is real (sidecar, not theater). Then push.

Do not “prompt harder” and hope the fix loop works overtime.

One take. Let’s go.

We wanted to understand: How effective was Chunk? Was there something we could do to improve the fixes given to the users? And perhaps most importantly, was there a way to improve the accessibility of the failure data to agents?

We tested the effectiveness in a controlled eval benchmark to measure the actual impact. The same dataset, the same model (claude-sonnet-5), against three different setups: no CLI, CLI with free use, and CLI with --failure-report. Here is what we found.

The context

Our first experiments were designed to mimic how Chunk works: giving it a file, then a generic prompt written by the team (for example “you are a coding agent, you need to solve this error” etc). Essentially at the beginning, we were trying to change the prompt to see if the fixes improved, and if giving certain instructions, steps, or validations would move the needle. The reality? The prompt didn’t seem to matter much.

Our team was constantly reading blog posts from other AI or CI/CD companies about what they were doing to solve this problem, and eventually we came across a paper by Bytedance. The paper shared that when cleaning log output, most of it is useless so the best thing to do is focus on the part of the log that contains the error.

We began to wonder: How do we remove the unnecessary characters and do a diff between the good and bad logs? That was the first successful experiment that gave us good results. It followed the way Chunk works. Because of that, we released the API of step output condensed. You fetch the output of a failed step and process to clean it, so you give better context to the agent.

From there, our focus eventually moved away from Chunk and toward the inner loop. We redesigned the experiment to mimic the usual developer flow.

“I thought: ‘If I was a dev working on a project and my pipeline failed, how would I fix it with an agent? Probably by copying the pipeline URL and giving it to an agent to fix. I saw that it had to follow a lot of steps to get to the pipeline output. The agent has context to the pull request, but it doesn’t necessarily have sufficient context.’”

Joaquin Sandoval Miramontes, Senior Software Engineer, CircleCI

Hypothesis

The hypothesis we arrived at was this: CI failure data is structured for humans reading a dashboard: status icons, log streams, nested UI. An agent working from that raw output burns context on navigation before it can start fixing. Therefore, restructuring the same data for agent consumption should reduce diagnostic turns and get the agent to fix faster.

Setup

I always try to do experiments in a structured fashion. I want to make sure that I have a way to repeat my own results over and over, and I’m particular about designing experiments with a hypothesis, expected results, and conclusions – I try to follow the scientific process wherever possible.

So here’s what we tried.

New flag: circleci run get [run_id] –failure-report

Output format: condensed, organized [workflow → job → step], showing only what failed. Built for piping directly into an agent’s context window.

circleci run get [run_id] --failure-report | claude "fix the failing workflow"

We compared three conditions on the same real failed pipelines from production:

• An agent given authenticated access to the CircleCI API

• An agent given free use of the CircleCI CLI

• An agent given access to the CircleCI CLI instructed to use --failure-report command

Results

Efficiency

The flag delivers real efficiency gains:

Agents using --failure-report spent fewer turns overall and finished faster. The CLI-only variant used the most turns and tokens due to agents navigating the CLI tool chain to resolve which job failed.

Fix Quality

Fix quality is flat across all variants. Fix rate difference is within the regular variation across agentic runs.

TL;DR

It used to be that an agent, when handed a failed CI run, spent many turns just figuring out what broke. The problem: the failure data was there, it just wasn’t accessible by an agent. The new command circleci run get [run_id] –failure-report changes that.

Previously, the agent was taking too many steps to get to the error itself, so we summarized it into one line. More context for the agent means better fixes, and spending less.

“If you have repeated tasks for your agent, give them tools to get it done faster. Don’t make your agent do 3-4 things before it gets started.”
Joaquin Sandoval Miramontes, Senior Software Engineer, CircleCI

The same failure data, restructured for agent consumption, changes how efficiently the agent works. The agent still needs to gather context from the source files, understand the failure and write the fix. What it no longer needs to do is spend 4-5 turns navigating a log hierarchy to find the failing step.

The practical conclusion: agents using --failure-report reach the same fixes in about ⅓ less time and ¼ fewer tokens. At scale, given thousands of CI fixes per day, that efficiency difference is real.

Has your team run any similar experiments? What have you learned about restructuring failure data for agent consumption? We’d love to hear from you in the comments below.

Data: 84 real CI failures from CircleCI. Model: claude-sonnet-5. Quality Judge Model: Claude-opus-4.6.

That loop runs on my machine most days. I wanted a cheaper version of it, and I had a specific idea about where the savings were hiding.

So I broke six CircleCI pipelines on purpose. Then I saved one specific failed run of each, so every model would stare at the exact same wreck. Three AI coding setups, one instruction: fix this.

Grok 4.6 fixed all six. Twenty-two cents a fix.

Claude Sonnet 5, turned all the way up, fixed three. Every fix it did land cost about $1.11.

I walked in looking for the opposite result. The chatter said Grok would be the one to stumble on the terminal work.

What I was actually hunting

A routing rule, which is a boring thing with a boring name: a policy for which model gets which wreck. Teams already do this with humans. The YAML typo goes to whoever’s on rotation. The bash failure that’s been red since Tuesday goes to the one person who can actually read a wait loop. I wanted that same escalation in software, with a price tag on it. Hard shell problems to the expensive model, broken config and missing test results to the cheap one. Same quality of fix, smaller bill at the end of the month.

The stake is your inner loop. Every time an agent picks up a red pipeline, reads the logs, edits the config, triggers a run, and reads that too, somebody is getting billed. Running all of it on the expensive model, all week, is a line item.

That split never showed up.

What did show up is a way to measure this stuff without fooling myself. I connected CircleCI’s command line tools to both Claude Code and Cursor. I saved a red run, so nobody could grade a different pipeline than the one I picked. Then I refused to grade the model’s essay for a proposed fix. By “essay” I mean the model’s written diagnosis of what broke and what it would change, which is a completely different thing from whether CircleCI actually went green.

Agents are already good at the essay. The actual pipeline rerun is the score.

The essay is the menu. Green CI is the meal. I came to eat.

Hypothesis

I expected Grok 4.6 to struggle on the bash-flavored failures and hold its own everywhere else.

That was the prediction. I went in with a bias: if any model was going to struggle on shell, it would be Grok 4.6 once it had to drive a terminal. Type a command, read what came back, adjust, type the next one. Shell breakage is exactly that, over and over, so that’s where I thought a gap would show.

If the reputation held, the plan was already written. Sonnet 5 at effort: low takes the bash cases, Grok takes the rest, and I pay less without shipping worse fixes.

Second question, cheaper to ask and just as useful: does paying for Sonnet 5 at effort: high buy enough extra passes to justify the receipt?

Setup: six puzzles, three setups, one judge

The puzzles came from the CircleCI Training Koans, a public set of short exercises that break a pipeline on purpose so you can practice un-breaking it. I used six on a throwaway project, two per flavor of broken.

Three flavors went in. Broken config, where the pipeline file itself is wrong. Tests that don’t report, where the tests genuinely run and pass but CircleCI never receives the results, so the job looks fine and tells you nothing. And shell problems: bash logic, wait loops, cache commands, container images, services.

Then I froze them: one saved failed run per puzzle, so “whatever failed most recently” couldn’t sneak in and change the question halfway through.

Three setups, same prompt every time.

Those dollar figures are sticker rates for tokens, not the cost of a run. What each trial actually spent shows up in Results.

Changed: the setup. Held constant: the saved failed run, the prompt, the isolation, the spend ceiling.

Every trial started in a fresh clone. No resumed sessions. Sonnet never got to peek at what Grok had already worked out, and the two Sonnet effort settings didn’t share a session either. Each agent had to check its config change, then trigger a brand new CircleCI run on its own throwaway branch. Never main. Never the saved branch.

Now the grade, which is the part that maps to your actual day. Nobody merges the model’s write-up. You merge green CI. So a trial passes only if the pipeline that agent triggered finished green. On the tests-that-don’t-report puzzles, a green job with an empty test results panel (CircleCI’s Test Summary) is still a fail, because tests that pass without reporting anything back aren’t a fix. Afterward I re-checked every run against CircleCI’s own record, so the numbers below come from CircleCI and not from an agent’s summary of its own brilliance.

I capped spending too. I watched the expensive setup burn money during calibration, took its 95th percentile, and set the ceiling at $1.25 so a trial could go long without going stupid. Claude Code can hard-stop at that number. Cursor writes it down and keeps going.

Claude dollars are whatever Claude Code billed for the whole session, including the Haiku 4.5 helper that sometimes rides along. Grok dollars are tokens in and out times the published rate, not an invoice from Cursor.

One honest wrinkle: model and host are welded together here. Sonnet only runs on Claude Code, Grok 4.6 only on Cursor CLI. I can’t pull those apart, and I’d rather say it out loud than bury it in a footnote.

Eighteen isolated trials. Then a repeat round on the same six puzzles, then one genuinely messy red Playwright job. Three separate scoreboards, and averaging them would only make the numbers look tidier than the evidence is.

Results

The pilot, scored the day it ran.

All three greened both shell puzzles. The terminal gap I came hunting for never showed.

One puzzle was a config file so mangled that CircleCI rejected it before any job started. That’s why the original run was red: nothing ran at all. Fixing it meant repairing the file, triggering a new pipeline, and getting that new pipeline green. Only Grok’s new run went green. Sonnet at effort: low spent the entire $1.25 cap on that one puzzle and got cut off. Sonnet at effort: high failed it too.

Separately, Sonnet at high effort missed both puzzles where the tests ran but CircleCI never received the results. Zero for two.

I ran the same six puzzles a second time. Fresh start, no memory of the first try. Grok fixed all six again. Add both tries together and Grok is 12 for 12, Sonnet at low effort is 9 for 12, Sonnet at high effort is 7 for 12. Same homework twice. I wanted to know if Grok just got lucky on Monday. It didn’t.

Then the follow-up. I cloned a Playwright job that had been going red over and over, again a throwaway, not production. The bug was a locator: getByText matched both a heading and a button, so the test found two things where it wanted one and refused to guess.

All three models got that one pipeline green. One try each. Nobody deleted tests. Nobody touched .circleci/config.yml. Every one of them edited the Playwright test and switched to getByRole(”heading”, ...). Inside each green job, CircleCI’s test results panel listed six Playwright tests, and all six passed.

Grok was the cheapest and the slowest. High effort and low effort wrote the same fix. One case, three trials, which makes it an anecdote, and I’m labeling it one.

TL;DR

I didn’t get a routing policy. Two puzzles per flavor of broken is nowhere near enough to write a company rule about where your model budget goes. Grok held the shell cases, too. And paying for effort: high didn’t buy extra passes. On these puzzles it bought fewer.

What I did get is a test that refuses to lie to me. Most evals grade what the model wrote. This one waits for CircleCI to finish, then takes the verdict from the run itself. Execution-graded, meaning CircleCI’s pass or fail is the grade, not a rubric and not the model’s own confidence.

If you’re wiring agents into CI, that distinction is the entire ballgame.

Cheap can win when execution is the judge. Twenty-two cents a successful fix against $1.11 isn’t a brand story. It’s a 5x token bill, on this lab set, every time an agent tries to get a red pipeline green. I’m not going to invent your volume. A team of ten, five of those loops a week, is about $45 extra to pay for high effort instead of Grok. That’s the unit. Not a finance-team emergency. Still a 5x.

I evaluate models for a living. Then I have to evaluate the eval.

Scoring a write-up is asking somebody how they feel. Scoring a CI pipeline rerun is taking their actual blood pressure.

RalphCI already showed that green on your laptop isn’t green in CI. This one asks the ruder follow-up: hand an agent the logs, the CircleCI tools, and one saved red run, and can it actually fix the pipeline? On this lab set, yes. Often. The expensive knob wasn’t the reason.

I’m not saying always use Grok instead of Sonnet. Hosts behave differently, Cursor can’t enforce a dollar cap, and these were clean training puzzles plus one Playwright locator.

I’m saying this: if your agent loop can’t trigger a rerun and read the test results panel, you’re grading the menu instead of the meal.

Five months later, I am still coding the same game of Snake. Now 100% AFK, with much more confidence than when I started. Zero human supervision.

The first half of 2026 has been fun, but not in the arcade sense. It has been fun in the “I haven’t touched my keyboard in twenty minutes, and every check on the branch is green” sense. We kept building the same Snake game. Same 20x20 grid. Same arrow keys. Same TDD spec. Same retro aesthetic.

The game has barely changed, but the delivery loop has changed everything.

If you have been watching the AI agent hype cycle from a safe distance, here is the compressed version of what we learned by staying stubbornly on one benchmark: you can go AFK with rising confidence, greener pull requests, and faster feedback than we have ever measured. Not by trusting the agent more. By giving it better and quicker back-pressure at each layer of validation.

The benchmark nobody asked for (but everybody needed)

Snake is a toy. That is the point.

A toy spec is small enough to run ten times in a week. Small enough to isolate one variable at a time. Small enough that when something breaks, you can actually read the diff instead of drowning in a monorepo.

We gave agents the same task repeatedly: build a playable Snake game from scratch using test-driven development (TDD). Seven tasks. Canvas rendering. Collision detection. Score tracking. Push commits. Open a PR. Walk away.

Every run finished locally. Every agent said the tests passed. Every game worked on the laptop and was pretty fun to play.

January: everything is a Ralph loop

In January, Geoffrey Huntley published everything is a ralph loop. The mindset shift is not “use AI to type faster.” It is “program the loop.”

Allocate a goal. Run the loop. Watch the failures. Fix the failure domain so it never happens again. Repeat until done or until you hit CTRL+C and take the wheel back.

Basic Ralph loops can build software and generate PRs AFK. That part worked decently well (with some light human supervision).

The PR at the end might be green. It might be red. The agent does not inherently know which until something external tells it.

We were building Snake games in loops before we had a name for what we were doing.

The lesson from January: autonomy without validation is just faster guessing.

February: does the pipeline agree with local tests?

By February we had data, not vibes, on whether the pipeline agreed.

Our first study ran ten controlled Snake builds. Same spec. Same model scaffolding. One variable: five runs wired to CircleCI through RalphCI, five runs without.

All ten agents completed every task. All ten passed local tests. All ten declared victory.

Eighty percent of the no-CI runs shipped code the pipeline rejected. The failure was mundane: single quotes where ESLint wanted double quotes. Local tests do not check lint rules. CI does. The gap between “works on my machine” and “works in your pipeline” is where agents lie with confidence.

RalphCI fixed that by injecting real pipeline status into the agent loop after push. When CI failed, a CI Doctor agent read the logs, applied fixes, and pushed again. Twelve failures across five runs. Twelve autonomous fixes. Zero human triage.

The breakthrough in February: agentic loops with CI back-pressure can land green PRs AFK. The final PR passes CI. Commits inside that PR might still fail along the way. Red commits on the branch. Green at the end. Progress, not perfection.

Early May: every commit has to be green

Three months later we hit the next failure domain.

Local pnpm test:run kept passing on my Mac. Chunk sidecars kept failing on Linux. Missing LEGAL_DISCLAIMER.md. Missing SNAKE.md experiment marker. A TypeScript compile error the Snake workspace’s local tests never exercised but a full pnpm build caught.

Same pattern, five runs in a row. Local green. Sidecar red. CI Doctor lands a fix(ci-sidecar): commit. Sidecar green. Push allowed.

Our early May study wired Chunk sidecars into RalphCI’s Review Gate. A sidecar is a lightweight remote microVM: your workspace syncs there, Chunk runs a microbuild that mirrors your pipeline intent, and you get CI-shaped feedback before the branch leaves your laptop.

Five Snake builds. Five 100% green PRs. Every commit green. Not just the last one.

I had not touched the keyboard in twenty minutes. The final PR (and every commit on it) was green anyway.

This screenshot is what HIGH CONFIDENCE looks like.

The May breakthrough: three validation layers (local, sidecar, pipeline) produce PRs where humans review green branches only. Sidecars catch CI-shaped failures while the agent still has context. CircleCI remains the authority at merge time. Sidecars extend CI into the inner loop. They do not replace it.

Late May: deliver the same green outcome, faster

100% Green PRs answered “can we merge with confidence?” A follow-up experiment asked “how fast can we merge with confidence?”

Our late May study ran the same ten Go tasks two ways on chunk-cli: sidecar validation per iteration versus commit-push-poll CircleCI per iteration. Same agent. Same model. Same gate jobs lint and test). Different machinery.

Sixty-nine seconds. Every iteration. Five replicates in a row. That is not random noise in our harness. That is the queue tax on push-per-task CI.

Twenty-two seconds on a warmed sidecar snapshot. Same question: are lint and test happy right now? Same answer. Different delivery speed.

Token spend was flat because in this particular experiment tokens tracked fixes, not wait. The CI arm burned more clock time, not more intelligence, so sidecars did not shrink the LLM bill here. They shrunk idle time. On a ten-task run that is roughly eight minutes of gate-waiting saved before the final pipeline epilogue.

The May speed breakthrough: you still want an outer loop. Sidecar runs finished with a full ci workflow epilogue after one push. Fast inner loop on each task, authoritative pipeline confirmation at the end. Not sidecar-only cowboy coding. Not push-and-pray either.

Four layers, one arc

Strip away the Snake skin and the arc looks like this:

We are not gaining more AFK confidence because we trust agents more (although the models have improved since January). We are gaining more AFK confidence because each failure domain we hit got engineered out of the loop. And we’ve even unleashed some speed gains along the way.

Watch the loop. That is Huntley’s line and it is the whole game.

• January taught us to program the AFK agent loop.

• February taught us the agent’s “I passed” is local theater without CI.

• Early May taught us that 100% green PRs are possible, given sidecar validation before commit and push.

• Late May taught us that 100% green PRs are still the prize, and sidecars get us there faster than push-per-task CI.

Snake games did not teach us that AI writes good code. They taught us where validation has to live when the coder is an agent loop and the reviewer is still a human with a merge button.

What I am actually doing with this

I am obviously not shipping Snake to production. (Unless you’ve been craving retro gaming experiences. DM me.)

I am shipping the stack: RalphCI orchestration, Review Gate with local checks plus Chunk sidecar validation, CI Doctor for autonomous repair, CircleCI as the outer loop authority. Same pattern we pressure-tested on a toy game because toys fail fast and teach faster.

Four layers, one arc. Program the loop. Wire in CI back-pressure. Validate on sidecars before push. Maintain CI pipeline authority at the end.

Each layer kills a failure domain the last one missed. Confidence climbs. PRs go greener. Feedback gets faster. The human at the merge button only reviews green.

That is what five months of the same Snake game bought us.

Green CI is still priceless. Everything else is noise.

Where to read the lab reports

• everything is a ralph loop (Geoffrey Huntley, Jan 2026)

• We Let an AI Agent Say “I Passed.” Was It Actually Good? (Feb 2026)

• AFK Builds with 100% Green PRs (May 2026)

• The Sidecar Race: 22 Seconds vs 69 Seconds (May 2026)

More Snake runs coming. Heavier repos. Flakier tests. Parallel agents. Improved CLI. Better agent-facing APIs. You name it.

The benchmark stays small on purpose, and we intend to scale the lessons as we go.

Twenty-two seconds is what the other arm averaged for the same question on a Chunk sidecar.

Same repo. Same ten Go tasks. Same Claude Agent SDK edits. Same gate jobs. Different loop.

Three weeks ago I published AFK Builds with 100% Green PRs. Five Snake games. RalphCI orchestration. Sidecars in the Review Gate so agents stopped feeding red commits to GitHub. Green PRs were the win.

This follow-up is narrower and, I think, more revealing for anyone running agents today: not whether sidecars help you merge green, but how fast the agent gets CI-shaped feedback while the file is still open.

I kicked the tires on sidecars again.

They are fast.

Not a much cheaper agent. A less idle one. Token spend was flat in this particular experiment (more on that later).

The win was clock time.

Hypothesis

If each agent task is validated with chunk sidecar sync plus chunk validate --remote lint and test-changed) instead of commit → push → poll CircleCI for the same lint and test jobs, then:

1. Median time to signal (TTS) per iteration will be materially lower on the sidecar arm.

2. LLM cost will be similar (same agent, same ten prompts, same model fixing the same mistakes).

3. CircleCI still matters for pipeline-level confirmation (sidecar runs end with a full ci workflow epilogue after one final push).

I expected push-per-task CI to pay a queue tax every iteration. However, I did not expect the median to land on 69 seconds five replicates in a row.

Setup

Repo:

• CircleCI-Public/chunk-cli

• Experiment PR #370

Tasks: Ten cumulative edits on a small Go fixture internal/racefixture/ on experimental run branches).

Replicates: Five labels 001–005) per arm. 10 recorded runs total.

Metric: TTS = wall-clock seconds from iteration start until both gates report pass/fail (logged in results.csv`).

CircleCI CTO Rob Zuber frames this as rebalancing inner and outer loop validation. My May post showed the outcome layer (green PRs). This post measures the wait layer.

Results

Headline: median time to signal

Aggregate: median of per-run medians across five replicates.

Five replicates at a glance

Sidecar clustered 20–22s. CI sat on 69s every time. That stability is either a real push-and-queue signature in our harness or a coincidence at lab scale. I am reporting it, not universalizing it.

p95 TTS: sidecar ~23–25s; CI ~72–99s (tails worse on push-per-task).

LLM cost comparison explained

The speed table is not the only controlled comparison in this harness. Token spend is measured the same way: same model claude-sonnet-4-20250514), same ten tasks, same agent, same fix-and-retry pattern. We changed the validation path, not the work.

Both arms hit the same lint and test failures and applied the same corrections. Sidecar surfaced them via chunk validate --remote on the warmed snapshot (~22s median). The CI arm surfaced them after push to GitHub and a ~3× longer wait for the same gate jobs (~69s median). Same signal, different delivery speed.

That is why token spend landed flat. Tokens track fixes, not wait. The CI arm cost more clock time, not more tokens. Sidecars did not shrink the LLM bill here because there was nothing to shrink: identical failure set, identical agent work on both arms. On a messier repo, especially if remote CI surfaces failures the sidecar doesn’t mirror, token spend might diverge. This harness didn’t test that.

So treat ~$0.90 per run as what this harness actually cost for this specific experiment, not a budget for production agent loops. Whether faster feedback trims token burn on messier work and CI failure modes (i.e. given fewer stale-context retries, less thrash, and less environment drift) is a follow-on question for a future study (coming soon). This race did not answer it.

This experiment answers a narrow question cleanly: when the agent does the same fixes either way, faster feedback does not change token spend. It changes how long the agent sits idle.

Same gate jobs. Different machinery.

We timed the same question on both arms: did lint and test pass? We did not run identical validation. Sidecar answers via chunk validate --remote on a warmed-up Linux snapshot. CI answers via real CircleCI jobs after you push.

Early iterations on both arms still saw lint failures while the agent fixed mistakes (about two failing lint iters per run in the rollup). The race is not “sidecar never fails.” It is how long you wait to learn.

Illustrative lab math only: ten tasks × ~47s saved ≈ ~8 minutes less gate-waiting per replicate before the sidecar epilogue. Your mileage will vary.

Takeaway

Green PRs are the outcome. Time to signal is the throttle.

The May experiment showed sidecars help agents stop polluting branches with commits CI would reject. This one shows why that matters in wall clock terms: the coding agent is not stuck in queue while context goes cold.

What held across ten runs in chunk-cli:

1. Sidecars save feedback latency on mirrored gate jobs (~3.1× median TTS here).

2. LLM spend was flat (~$4.64 sidecar vs ~$4.73 CI) in the same apples-to-apples comparison: same failures, same fixes, CI just took ~3× longer to surface them. Tokens track fixes, not wait.

3. Push-per-task CI is a predictable wait in this harness (69s median, every replicate).

4. You still want an outer loop. Sidecar runs finished with a full ci workflow epilogue. Fast inner loop on each task, plus an authoritative pipeline confirmation at the end. Not sidecar-only cowboy coding.

Please re-read #4 above: I am not at all suggesting you delete your CI pipelines. Quite the opposite: I am saying that if your agent loop is push → wait → fix → push again, you are paying for that wait every iteration.

Sidecars moved the same exact question to a local, warmed-up Linux snapshot...

QUESTION: Are lint and test happy on this repo right now?

...and the same answers came with a much tighter feedback loop. 3.1× faster.

What’s Next

• Combine this harness with RalphCI orchestration (Snake outcome study + TTS race in one stack).

• Heavier repos, flaky tests, parallel agents. Find where Chunk sidecars and snapshot sync stops winning.

• Agent economics at scale: cost per green iteration (LLM + validation path), not just TTS. A proper cohort study on heavier repos, not extrapolated from this ten-run table alone.

• Demo recordings on various Chunk sidecar and snapshot sync setups. Stay tuned!

Artifacts:

• FINDINGS.md

• Experiment PR #370

If sidecars are new to you, a Chunk sidecar is a lightweight remote microVM that sits in your inner loop. Your workspace syncs there, Chunk runs a microbuild (lint, tests, build, and the repo policy hooks you configured), and you get CI-shaped feedback on Linux before your branch leaves the laptop.

CircleCI CTO Rob Zuber describes the goal as rebalancing inner and outer loop validation so agents are not learning about failures only after push.

Our three early sidecar failures were unrelated. First, a missing LEGAL_DISCLAIMER.md at the repo root (legal policy gate). Second, a missing SNAKE.md marker file in the experiment workspace (documentation policy gate). Third, a TypeScript compile error the Snake workspace’s local tests never exercised but the sidecar’s full pnpm build caught.

All three times, the lint-and-test run on my Mac localhost said green, but the red Chunk sidecar said “fix this stuff.” CI Doctor landed one fix(ci-sidecar): commit per issue. Only then did my local Review Gate allow a push to GitHub.

That is a small lab story. We think it is a bigger CI story.

We humbly believe we are building toward the future of continuous integration in the agent era: validation woven through the whole loop, ending on a 100% green pull request you can merge with confidence.

Local lint and tests first. A Chunk sidecar microbuild that partially mirrors your pipeline while the agent still has context. Then CircleCI on the branch, doing what CI has always done, checking integration before a human approves merge.

AFK application builds that land as 100% green PRs are what we can demo today. The thesis underneath: CI validation that keeps up with the pace of your local coding agents.

We ran five Snake game builds to pressure-test that future. What follows is the lab report.

Hypothesis

If RalphCI runs Chunk sidecar validation chunk validate --remote) after local lint:fix and test:run, then:

1. A coding agent can run mostly unattended through build and fix cycles (while a CI Doctor agent handles sidecar and pipeline failures locally).

2. Commits that pass the local Review Gate and push to GitHub show up on the branch as a 100% green PR: with every mirrored CI pipeline passing on the branch.

3. CircleCI remains the authority at merge time. Sidecars extend CI into the inner loop; they do not replace it.

Our February 10 study already showed CI-aware RalphCI can get to green pipelines on this Snake benchmark when the agent answers to CircleCI after push. This experiment adds pre-push CI parity via sidecars so the PR itself is green top to bottom, and not just at the end of the PR.

Setup

Benchmark: Five independent runs in ralph-ci/experiments/w_chunk-sidecars/iteration-{1..5}/. Same Snake spec as prior Loop Lab RalphCI work: 7 tasks, Vitest TDD, canvas game.

The stack (from top to bottom):

Chunk mirrors CI: .chunk/config.json runs policy gates --gate-legal, --gate-snake) plus remote pnpm lint, pnpm test:run, pnpm build, matching CircleCI jobs on the branch.

Sample size: 5 runs. Lab notebook scale.

Results

Five runs at a glance

Totals: 57 RalphCI iterations, 16 sidecar repair loops, 5/5 runs ended with all tasks complete and green CircleCI on the branch.

Source: iteration-*/metrics.json and activity.md.

Local green is not PR green

Every run, early iterations: Review Gate fails on Chunk sidecar, not local lint-and-test run. On all runs, the first three sidecar fixes were: restore LEGAL_DISCLAIMER.md (legal gate), add the SNAKE.md experiment marker file (documentation gate), then fix a tsc error in run-ci.ts surfaced by the monorepo build step.

Same pattern across runs: failures local tests never see. CI Doctor agent fixes them without a human in the loop. Only then does the local Review Gate allow a push to GitHub.

After the sidecar gate cleared, pushes stayed green on CircleCI. All CircleCI pipelines passed upon push to remote. No surprise reds on mirrored work.

Compared to our February baseline

On the same Snake benchmark without sidecars in the Review Gate, agents often finished locally while the branch still failed CircleCI.

With sidecars before push plus CI on the branch, five for five ended as 100% green PRs in our runs: build, lint, policy gates, and test jobs all passing on the merge target after push to GitHub.

The continuous integration breakthrough is three layers: local, sidecar, pipeline. AFK works when failure shows up in layer two, so layer three confirms instead of contradicts.

Takeaway

Rob Zuber framed inner-loop validation for agents. RalphCI plus Chunk sidecars are one way to wire that into an orchestration loop so AFK does not mean “ship and pray.”

What held across five Snake game builds:

1. Sidecars catch CI-shaped failures while the agent still has context.

2. A 100% green PR needs three layers: local, sidecar, pipeline. Skip one, get surprised downstream.

3. Humans review green branches ONLY. They rarely see red, not even on a single commit.

Green CI is still priceless. Chunk sidecars are how you earn a 100% green PR before you even ask a human to look, and a green PR instills more confidence in the agent’s work than a red one.

The artifact: a 100% green PR ready for review

This is the outcome we care about. A pull request an agent built while I was AFK, with every last CircleCI check green on the branch after push to GitHub.

Twelve commits on the branch. Task work, three fix(ci-sidecar): repairs, finalize, metrics. At the bottom of the page: All checks have passed. Five green jobs:

• ci/circleci: build

• ci/circleci: lint

• ci/circleci: require-legal-disclaimer-md

• ci/circleci: require-snake-md

• ci/circleci: test

No conflicts with base. Merge pull request is the honest end state.

That screenshot is what we mean by AFK builds and 100% green PRs: every CircleCI check green before a human opens the branch, because validation ran local, then on a Chunk sidecar, then on the pipeline, and the agent fixed failures in between.

Five Snake game builds proved the workflow. In future runs, we may expand the repo complexity and scope, and the review gates along with it, to see how far we can push this thesis.

But we will keep the overall stack, because we humbly think that this is where continuous integration is going.

Onboarding for those teams is never “read the README and go.” It is access to the right systems, a path through tooling that matches how work really flows, habits that stay safe under pressure, and knowing who to ping when something breaks.

We stopped treating that ramp like a scavenger hunt through stale wikis and shipped it as something you can run: a team-scoped Claude Code plugin made of skills, small playbooks the agent follows, with MCP checks so “connected” means verified, not hoped for.

Hypothesis

A Team Onboarding Buddy should beat the default experience: read twelve pages, miss the thirteenth, ping three people, discover your MCP tokens are wrong, and lose a week proving you are not incompetent, you are just new.

If onboarding is small, invocable skills with explicit triggers, and those skills include MCP health checks plus a real access checklist, a new hire should get to first useful work with less thrash. The skills do not replace humans. They replace the worst part of onboarding: reconstructing tribal knowledge from stale breadcrumbs.

Setup

What we built. A team-scoped plugin: four skills in one bundle, installable the same way you would ship any other Claude Code plugin to a group that should share one playbook.

Our first internal cut targeted one engineering sub-team’s playbook before we framed this as a pattern anyone could clone.

Repo shape (quick and dirty). In a Claude Code plugin marketplace repo you usually get one folder per plugin: a small manifest under .claude-plugin/ and a skills/ directory full of SKILL.md playbooks. The folder name should match the plugin id in plugin.json (that becomes the /namespace:skill prefix in Claude Code). Rename team-name to whatever id you pick in plugin.json.

Examples of folder / namespace names (illustrative only): engineering, marketing, product, design, legal, finance, people-ops.

Why this shape. The skills are modular on purpose. Nobody wants a forty-page monolith. A new hire can pick sections like a menu instead of pretending they will read the whole handbook on day one.

The MCP angle. The access skill does something wikis are bad at. It tells the agent to verify integrations with real reads through connected tools, not to assume OAuth worked because nothing exploded yet.

The human angle. Long-form runbooks still live wherever your company keeps them. Escalation still names real humans. The goal is not autonomous HR. The goal is fewer round trips for questions the team has already answered a hundred times.

Results

What we shipped.

• A versioned, team-scoped bundle so one group can adopt the buddy without forking the whole company agent config.

• Hundreds of lines of structured guidance across the four skills: MCP verification steps, access and tooling checks, and a clear path from “green integrations” toward first safe change.

What we learned in practice (qualitative).

• The entry skill actually gets used. People do not have to remember the “right” order. The plugin encodes it.

• MCP verification turns a vague “I think I am set up” into a pass or fail list. When something is wrong, the output is actionable instead of confidently wrong.

• The team skill fills a gap most organizations leave empty: who shows up from outside the group, for what reasons, and where the sharp edges are. That is context people often learn only after they step on a rake.

How to sanity-check your own build (five minutes).

• Invoke the entry skill and confirm it routes into access, local-dev, and team flows in a sensible order.

• Run the MCP verification path and confirm failures degrade gracefully: they say what is broken and what to check next, instead of inventing green checks.

On metrics. We have strong internal signal on usability and fewer “false start” access threads. A controlled before/after (time to first local run, time to green MCP matrix, week-one ticket volume, short confidence survey) is the right next step if we want a chart in a sequel post. This write-up is the pattern and the ship story, not that chart yet.

Takeaway

Onboarding is not an information problem. It is a routing problem. You do not need more wiki pages. You need a default path that is hard to screw up, plus checks that fail loud when your tooling is not actually connected.

Packaging that path as Claude Code skills is a bet on where engineers already live (NOTE: It could be Cowork, in the case of a non-engineering team.).

Whether Claude Code or Cowork, the buddy belongs in the editor session, not on page seventeen of a PDF nobody opened.

The bundle bakes in habits that matter whenever financial correctness, access control, or customer-visible production behavior is on the line: look before you change anything (read-only checks first), exercise mutations in a safe mode (dry-run or sandbox before you touch live state), and do not solo your first production-adjacent change (pair with someone who has muscle memory).

That is not moralizing. It is how you avoid the expensive failure mode where a small onboarding shortcut becomes a customer-visible incident or a long support recovery.

Credits

Team Onboarding Buddy was a collaboration. Thanks to the team of engineers who built, reviewed, and tried it in real onboarding flows:

• Andrea Morato Dixon

• Fernando Lima

• Ryan Hamilton (yours truly)

If this pattern helps your org, please let us know!

What’s Next

1. Run a small cohort study with agreed metrics so we can publish numbers, not just vibes.

2. Reuse the skeleton across the org, for other teams: same four-skill shape onboarding, access, local-dev, team), different playbooks for both engineering and non-engineering teams.

3. Treat rosters and paths as perishable data. Point people at live chat and authoritative docs, refresh the skills when reality drifts.

Bottom line for anyone pitching this upstairs

This is not “AI replaces onboarding.” It is AI holds the checklist and verifies the wiring so humans spend time on judgment, not detective work.