Median teams sit around 3.9. The top 5% run about 2.6. An elite cohort of twenty orgs is already near 1.3.

Most of the industry is still grinding through several rounds of feature-branch rework before earning the right to merge to main. The leaders are closer to one.

When coding agents are the ones waiting on those rework cycles, MER is not a vanity metric. It is the clock and the bill. Extra loops mean more tokens burned, more CI credits spent, and more agent context rotting on a red light from CI.

The Q2 Pulse Report is blunt: lower MER is a DX win, an AX win, and a cost win.

I ran a small AFK lab experiment that rhymes with that report. Not an org-level MER study. Same shape of problem, inside a single agent setup.

Subscribe now

Hypothesis

If a green PR is already likely, pushing only once at the end should burn far fewer outer-loop CI credits than pushing after every task. And if deterministic checks live on the inner loop once, you stop paying time and money three times for the same linter.

Pushing once at the end is only safe once the agent can reliably go green.

Setup

In our core experiment to determine the cost of a green PR, we found that if you give the agent a sneak peek at the CI setup, let it take notes before coding begins, and let it do a practice run on inner-loop checks before pushing to outer-loop CI, you can achieve 100% green PRs with zero rework.

The sneak peek is actually quite simple. Before task one, before a line of game code gets written, the agent receives a short inventory of the CI setup: which checks fire on the inner-loop practice run, which jobs fire on the outer-loop CI pipeline, and pointers to the scripts and configs that define them. This information is descriptive, but not an answer key. The agent writes its own notes from that inventory preflight.md) and can @-read those configs whenever it wants more. In our codebase, the inventory is a “CI validation manifest,” which tells the agent what will be graded and where the grading logic lives, before it starts guessing from vibes.

Fix loops drop to zero. Green PRs go to 100%.

This piece starts there. Same AFK Snake game build. Same seven tasks. Same two cadences, now with the sneak peek on.

• Per-task-push: a pipeline after every task. Seven tasks, seven trips.

• Single-push: finish the stack, clear practice run, push once at the end.

How often you push is the lever under test. First you have to be equipped to push green.

A linter is a linter. Compute is compute, whether it runs on a laptop, in a Chunk sidecar microbuild, or in a cloud job financed with CI credits.

There is no real reason to run the same deterministic check on localhost, again on the sidecar, and again on the outer-loop CI pipeline, as if three identical stamps make the code more correct.

That stack is a relic of human by-hand engineering. A person in flow forgets to run the test suite, pushes red, and we paper over forgetfulness by re-running the universe on every layer. Coding agents do not forget the same way. You can (and should) instruct them to clear inner-loop checks before they spend a single outer-loop CI credit.

In this calibration we moved inner-loop checks fully onto the sidecar and left thick outer-loop CI as the honest final test. Slimming true duplicates on the outer-loop surface is later work. The principle does not wait though: do not triple-pay for the same answer.

Results

Headline: once sneak peek made 100% green likely, a single end-of-run push cut estimated CI credits from ~81 to ~14.

• Per-task-push: 7 pipelines. ~81.4 estimated CI credits.

• Single-push: 1 pipeline. ~14.0 estimated credits.

About 5 to 6x fewer CI credits on the single-push run.

Wall-clock stayed in the 50 to 60 minute band either way (55.5 vs 56.7). LLM dollars stayed in the $15 to $17 band (17.15 vs 15.73). Tokens still track agent work. The CI credit collapse comes from the push cadence, not a cheaper model.

That bargain only holds when a sneak peek plus inner-loop checks make a 100% green PR likely. If you pile up unproven work and trigger the outer-loop pipeline once cold, you have not saved anything. You have postponed finding out you are red. When that one pipeline fails, you still pay the fix loop, and a retry can pick up new last-mile surprises.

Earn one take. Then trigger outer-loop CI once.

TL;DR

Elite engineering teams do not win by loving rework. They win by needing less of it.

Think of agent-era validation as a continuum. Inner loop to outer loop, hybrid on purpose. Each check earns its place.

• Deterministic checks live inward: cheap, early, once.

• World-shaped risk lives outward: policy, last-mile, flaky integrations, the CVE that did not exist at 9am. You name it.

Place each check where its information is unique. Everything else is just another duplicate cost for the same answer.

Outer-loop CI should still exist. It is where the world gets a vote. It should not be a museum of jobs you already passed in identical form on the previous two layers of validation.

At AFK scale, the inheritance path looks like this:

1. Make preventable reds disappear (push a /green PR in one take).

2. Keep remaining retries cheap (use the --failure-report flag).

3. Put deterministic checks on the inner loop once. Push to outer-loop CI as often as your green rate deserves.

Once 100% green is something you can count on, pushing only once at the end of the run is an obvious CI credit win.

Bake that into your agent setup. Low MER should not be a hero dashboard you inspect after the damage. It should be the default the agent loop was engineered to produce.

Rework should be reserved for unknown failures from the real world. Preventable failures should be totally eliminated.

Our current CircleCI APIs use version numbers in the URLs. V1, V1.1 and V2. They had accumulated four incompatible URL patterns, three ID formats and a different response shape per endpoint, effectively ten years of drift. The pitch for a new API V3 was explicitly about consistency for agents: one URL grammar, UUIDs everywhere, one response envelope, one error shape. An LLM should be able to predict /api/v3/{entity}/:id from the entity name alone instead of carrying the whole spec in its context.

We followed all the current advice on how to make the API discoverable by agents. We published the OpenAPI spec at a stable URL. We added /.well-known/api-catalog per RFC 9727, so a machine could discover the machine-readable descriptions. We generated an llms.txt and an llms-full.txt. We rendered the whole reference as markdown as well as HTML. We put the conventions - error shapes, idempotency, which operations are irreversible - into the spec description so they travel with every entity document, not just the index.

This post answers one question: did any of that work?

Hypothesis

My assumption at the outset was that agents would want to work with a consistent API, and that they would be able to easily find it.

I expected consistency to be the hard part and discovery to be the easy win.

Specifically:

1. Agents would find V3 on their own through the discovery surface - the catalog, the OpenAPI spec, llms.txt.

2. Once they found it, V3’s consistency would keep them there, because guessing a V3 URL is easier than looking one up in V2.

3. V2 usage by agents would decline as agents discovered the better option.

The first and third were wrong in ways more interesting than being right.

Setup

I first proposed building a new consistent API 3 years ago. At the time, it would have been a massive time expenditure likely not worth the investment. But by the time I pitched it again earlier this year, the execution had changed; we could use a swarm of agents with good skills. Agentic workflow enabled us to decide that it was a low enough cost to take on an experiment previously considered to be very high cost, and impacting every team on the org. I got the green light.

I tend to be more pragmatic than scientific. I understand the scientific method, but can also plan ahead to mitigate the risks of just diving in and trying things. In the case of the API, although it’s fairly high risk in some ways, I knew that if we didn’t publicize it fully, we could always stop and roll back.

Every V3 route ingresses through one gateway service. That was a deliberate call made for enforcement reasons, and it is also what makes this measurable: one dataset holds every public API request with its route, method and user agent.

After agreeing on the consistent flexible API shape, we built skills internally for teams to use to guide their agents, and within a few months we had V3 APIs in place that could replace 90% of existing API traffic.

Measuring agent traffic hitting our API is an inexact science, relying on parsing the User-Agent in the inbound requests.

Real strings from production for requests coming in via our CLI:

circleci-cli (darwin/arm64; 1.0.48773; claude-code_2-1-246_agent)
circleci-cli (darwin/arm64; 1.0.48692; codex)
circleci-cli (darwin/arm64; 1.0.48773; opencode)
circleci mcp (755dd082f9ef571f74ac433160ab068e10c9ef25)

That is self-reported identity, not inference from traffic shape. If you want to measure agent usage of your own API and you control a client, stamping the harness name into the user agent is the highest-value thing you can do. It costs one line and turns an unanswerable question into a query.

Two caveats to keep in mind if you try something similar: An agent that hand-rolls its own HTTP calls is invisible to this method. I checked the bare-HTTP-client population and found roughly 41,000 spans a day where an agent could hide indistinguishably from a shell script (the span counts come from a sampled tracing dataset: the agent-filtered queries returned unsampled, the whole-traffic ones at 1.7× mean sample rate.)

Results

Interestingly, we realized the agents don’t care about the API directly. Instead, their focus is the MCP and CLI. Having a consistent API made it easier to build a good CLI. It’s only in this roundabout way that the agents are interested in using the API.

Agents overwhelmingly use V3. But not by migrating.

Hypothesis 2 confirmed, emphatically. Now the same measurement seven weeks earlier:

In those seven weeks, agent V3 share went from 57% to 97%. But look at the V2 column: 17,641 to 22,375. V2 agent traffic did not decline. It grew slightly. V3 did not win by converting anyone. It won by absorbing 760,000 calls of agent traffic that did not exist seven weeks earlier.

Hypothesis 3 was wrong: the migration story I expected to tell was actually a growth story.

For contrast, across all clients including browsers and scripts over that same recent week, V2 still outweighs V3 - 24,067,618 calls against 16,978,477. Humans and legacy integrations have not moved. Agents are a separate population behaving differently.

No agents found the V3 API on its own

Here is the entire discovery surface over thirty days. Every file we published for machines to find:

Discovery endpoints · All clients · 30 days to 28 Aug 2026

Agents made 807,209 API calls in a week. The entire discovery surface got 6,776 hits in a month.

Then I broke those 6,776 hits down by user agent, expecting a long tail of coding agents. There are three populations in there, and none of them is the one I was looking for.

Population one: nobody in particular

Crawlers, scanners, and humans

The largest single consumer of /.well-known/api-catalog is a bot that exists to grade whether your API is agent-ready. We are being audited for agent-readiness by a crawler while the agents themselves never look.

Population two: agents, but only when a human points at them

These are visible and separable, because a directed fetch carries a different user agent than an agent doing its own work:

Claude-User (claude-code/2.1.220; +https://support.anthropic.com/)

That is Claude Code’s web fetch, which fires when a person says “go read this URL”. Across the whole discovery surface in thirty days it accounts for 23 requests - 6 on the static OpenAPI HTML, 5 on /docs/api/v3, 4 on /fullopenapi.json, 4 on /.well-known/api-catalog, and single hits elsewhere. Two more came from Claude-User/1.0, the claude.ai equivalent.

So the honest version of the finding is not “no agent has ever read it”:

Zero agents read the discovery surface unless a human explicitly directed them to it. Twenty-three times in thirty days, someone did.

Every one of those 23 was a person (probably one of us) saying “fetch the catalog”. Not one was an agent concluding on its own that a catalog might exist and going to look.

Population three: the training and search pipelines

This is the part that gives me some hope. Around 203 of the 6,776 hits are corpus builders and search crawlers:

They are all reading /openapi.json. The spec is being ingested, but just not into a working agent’s context. It is going into the next generation of training data and the search indexes agents fall back to. The discovery surface is on a much slower clock than I assumed.

Which means the optimistic reading of this whole post is available, but it is not the one I would have guessed. It is tempting to hope that the next round of models will start consulting catalogs and llms.txt properly, the way the standards intend. I do not think that is what will happen. What will happen is that GPTBot and ClaudeBot read our openapi.json, V3 ends up in the weights, and the next generation answers V3 from memory… exactly the way this generation answers V2 from memory.

The fix for stale training data is more training data. The catalog’s delivery vehicle is the corpus, and the crawler is the courier.

So the discovery surface does pay off. Just not as discovery. And the loop it runs on is a model generation long, which is not something you can shorten by publishing harder. The only lever you control on that timescale is still the client.

Why agents don’t discover: two layers of stale

Watching agents work with our API, the failure has two stages and neither involves discovery.

STAGE 1: Training data answers first

Ask an agent about the CircleCI API and it answers immediately and confidently with V2 shapes - /api/v2/project/{project-slug}/pipeline, slugs in paths, the old status enum. It is not looking anything up. V2 has been in the corpus for years; V3 has existed for months.

STAGE 2: The search index agrees with the training data

Agents will often announce they are searching the web when they are reaching into training data. When they do genuinely search, the indexes rank a decade of V2 documentation, V2 Stack Overflow answers and V2 blog posts far above anything we published this year. Two independent mechanisms, same wrong answer.

And here is the twist that makes discovery standards nearly useless in this shape: by the time an agent finally reaches a spec or a catalog, it already believes it is looking for V2. It arrives with a target. We updated the spec descriptions to say prominently that a newer V3 API exists, and an agent that has decided it needs the V2 pipeline endpoint reads past that to go looking for the V2 pipeline endpoint.

An in-context hint only helps a reader who has not yet decided what they are looking for. An agent has always already decided.

What actually moved the traffic

We updated the CLI to use V3, and pointed the built-in MCP server at the same code.

My colleague Pete has written up the rebuild this came out of (Rebuilding the CircleCI CLI from scratch) and it is worth reading alongside this one, because his half of the story is the half that actually moved the numbers below.

Roughly 57% of all CLI traffic to the public API now carries an agent harness stamp. Every one of the 807,209 agent calls arrived through the CLI or the MCP server. None arrived from an agent constructing its own HTTP request.

Agents do not discover APIs. They use the tools already in front of them. The CLI is already installed, already authenticated, already on the agent’s PATH, and --help is right there. The MCP server appears in the tool list. Neither requires discovery, a search, or a spec.

The lever was never the discovery standard. The lever was the client.

TL;DR

No agent reads your discovery surface unless a human points at it. The catalog, llms.txt and the OpenAPI spec drew 6,776 hits in a month. Agent reads: 23, every one a directed fetch. Self-directed: zero.

Publish it anyway, but price it as a training-data bet. 203 of those hits were GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot and MistralAI-User. The spec is being ingested, just not into a working agent’s context.

Expect the fix to arrive through the weights, not the standard. The next model generation won’t consult your catalog. It’ll just already know your API. That’s a model-generation-long loop you can’t shorten by publishing harder.

In-context hints only help a reader who hasn’t decided yet. We put “there is a newer V3” in the spec description. An agent that has already concluded it needs a V2 endpoint reads straight past it.

Ship the client and the MCP server. 100% of measured agent traffic arrived through a wrapper that was already installed and already authenticated. None of it was hand-rolled HTTP. If you want agents on your new API, update the tools they already have.

$ circleci run trigger --help | head -40

I hadn’t asked for head -40. Claude added it, and it does that constantly; help output is untrusted input of unknown size, and pulling an unbounded page into a context window is a bad trade. Forty lines is the interface. Below line 40 was the part that tells you how to use the command.

Hypothesis

As I shared in a previous blog post, when I started rewriting the CLI, I already understood we were writing for agents; that was the premise of the rewrite. Every data-returning command has --json with its fields enumerated, and the command tree generates an MCP server, so each command’s description is a tool description.

Two things we didn’t know:

1. We didn’t know there was a length limit, so we never measured against one (”written for agents” was a set of content decisions, and none of them had a size).

2. We didn’t know that we were saying everything three times: the accepted values for --event-preset, the default for --provider, all of it in the examples, in the flag table, and again in the prose. Each section had been written to stand on its own, and nobody diffs a paragraph against a table.

So the fix wasn’t “write less”. If the length was repetition, most of it would come out mechanically; and if prose went last, truncation would cut the cheapest content rather than the most expensive.

Setup

I measured before editing anything, because “is this help text too long?” is otherwise a taste argument nobody wins. Walk the command tree, render every page the way a real invocation would, and count. Not total lines, but whether each section finishes inside the first 40. A flag table that starts on line 38 isn’t visible; it’s a teaser.

Across 169 command pages, the mean was 49 lines, the worst 91, and only 15% got their examples inside the window (the section an agent actually copies from).

Three passes. Boilerplate out: the banner, the per-page table re-printing the same four global flags, the repeated “Learn More” links. Seventeen lines a page, no prose rewritten. Reorder: Short → Usage → Arguments → Flags → Examples → Details, so truncation eats prose first. Trim the duplication, never dropping a flag row or an example to make space. project trigger create went from 91 lines to 42.

Then keep it fixed. Order and boilerplate are properties of the help template now; length is a test that asserts every page fits 40 lines, with individual caps for the twenty-five commands that genuinely can’t, and those caps only ratchet down. And since these commands are largely written by an agent working from guidelines in the repo, the rule went there too.

The PR: https://github.com/CircleCI-Public/circleci-cli/pull/1667. 449 files, about 2,600 net lines of help text deleted.

Results

The three passes cut both the mean and the worst case, but what really matters is whether the sections an agent actually uses (flag table and the examples) finish rendering before truncation. A page that runs to 70 lines but front-loads its flags is a better interface than a 45-line page that buries them. The reorder pass was designed around this: put the cheap content last, so if something gets cut, it isn’t the part Claude copies from.

No task-success delta, because I don’t have one. What changed is the interface: every command’s complete flag list now sits inside the window an agent reads, and three in four get their examples there too, against one in seven before. The old failure mode was silent, which is why it lasted. Claude wasn’t erroring; it was succeeding at a degraded version of the task on whatever fragment of the flag list it had seen. Worse than a stack trace, because nothing tells you to go and look.

TL;DR

“Designed for agents” is a budget, not a style. We had the content decisions right and still shipped help that couldn’t be read, because we never asked how much of it arrives.

Pick one canonical place for each fact. The duplication wasn’t sloppiness; it was three self-sufficient sections maintained separately. The flag table is canonical now, and prose restating it is a defect.

Make it a test. A style guide asking for 40 lines would have lasted a month; the next person adding a command, human or otherwise, has no reason to know the number exists.

The pages are better for humans now, which I didn’t expect; not because agents and people want the same thing, but because neither mistake was agent-specific. Nobody benefits from reading a flag’s values three times, and nobody was reading line 70 either.

At roughly the same time, I’d been spending time considering how I might be able to use Chunk sidecars for related tasks outside of testing. I wondered: Do sidecars have the potential for a wider application?

I’d say that’s representative of my style as a whole: When conducting experiments, I don’t just think about the results, I try to think about new ways of conducting the experiment that can be applied to other experiments in the future to create additional efficiencies.

In the case of this specific experiment, I was fairly certain that the problem would be easily divided and conquered by using multiple computational resources. Without that approach, the experiment would be costly and cumbersome. Chunk sidecars provided the best route forward, and resulted in a method that I (and any other CircleCI customer) can replicate within future experiments too.

Hypothesis

Config validation is embarrassingly parallel. Each file is independent. There’s no shared state, no ordering requirement, no reason one file’s result should wait on another’s. If we could distribute validation across a fleet of Chunk sidecars, the wall-clock time should drop to roughly: (time to validate one file) / (number of sidecars running in parallel).

The biggest question was how to keep this cost effective from a time expenditure perspective, given that sidecars take about 30 seconds each to spin up. If each sidecar only validated a handful of files before its setup overhead dominated, the fleet wouldn’t be faster than sequential. It would just be more expensive.

Setup

Chunk sidecars are lightweight remote microVMs that run microbuilds alongside a developer’s work session. Each one is a full Linux environment: it can run the compiler, execute arbitrary code, and report results back.

It was important to me to conduct this work in a way the customer could reproduce. Don’t just trust me – try the same validation on your end as well to prove that the pipeline won’t be broken when the change is introduced.

Makoto Mizukami, Senior Field Engineer JAPAC, CircleCI

Knowing that the customer wouldn’t have access to our database (making the experiment potentially costly for them to run themselves) I made sure to use resources that would be available to them as well – the CircleCI API and CLI.

For this experiment, I partitioned the customer’s 10,000+ configs into batches and assigned each batch to a sidecar. Each sidecar:

1. Spun up

2. Received its batch of projects

3. Walked through each project to fetch its config through CircleCI API and run it through circleci config validate--next

4. Reported pass/fail results, and failure reasons if any

The fleet size was 100 sidecars in parallel at maximum. Task orchestration, command dispatches, and result aggregations were all done by a single Claude Code agent session, allowing me to save tokens (unlike Claude subagents, which hungrily consume tokens). The work was instrumented with Honeycomb throughout: 32,359 spans across the full run.

Results

The fleet economics held up. Setup overhead for the 100 sidecars was 2 minutes in total — small enough that the parallelism paid off at this batch size.

One finding worth noting: We also observed some specific patterns in validation failures. For example, certain series of strings tended to raise a validation error. By analyzing the result of the large-scale process further, we were also able to identify some “template” projects, from which most of the 120 configs derived.

The reason that the setup was reasonably easy was that Chunk sidecars provide snapshots. I could install a basic toolset in the sidecar, take a snapshot and apply the snapshot to the sidecars. Without that, this experiment would be impossible to conduct efficiently.

TL;DR

Embarrassingly parallel tasks don’t need clever algorithms. They need an environment that can run many agents at once. The compiler logic didn’t change. The validation logic didn’t change. What changed was the execution model: instead of one process working through a queue, a fleet of processes each worked through a slice.

Running them sequentially would have taken hours. A Chunk sidecar fleet did it in 4 minutes.

The thing to check before you do this: Make sure each agent knows exactly what its job is. Per-agent setup costs compound. If your task is trivial and your setup is heavy, the fleet won’t help. In this case, validation was fast enough and setup was cheap enough that the math worked. Instrument it first and check.

The customer got their answer before the breaking change shipped. It was an answer we could never have gotten in a timely manner without the massive parallelism Chunk sidecars offered. It proved my hypothesis that Chunk sidecars have multiple uses, and aren’t just limited to testing. As it turns out, sidecars are well implemented so that agents can easily leverage and consume the data, making them a perfect vehicle for experiments like this one.

We complain a lot about AI agent output. But we need to be thinking more about how we’re setting our agents up for success, and whether we’re providing them with sufficient power to do their job. Sidecars are one of the best ways to prepare agents to do great work.

Then suppose that same student could peek at the test in advance and take notes.

One catch. The sneak peek does not include any of the last-minute trick questions that might show up on the real exam without warning. Those get written by “the world” the moment the exam starts.

Nobody on the teaching staff knows the trick questions in advance, and new ones can show up on a retry.

Pencils down.

That is coding agents and CI. Merge-ready green is the only passing grade. Outer-loop CI is the real final exam. A Chunk sidecar is the timed practice run. Every red→green fix loop (diagnose, patch, push again) charges a little tuition: LLM dollars, CI credits, wall-clock.

I wanted to know what a green PR actually costs on a fully-automated AI coding agent run. Time. Tokens. Credits. The whole bill.

We moved the needle a bit on time and tokens. Fine. Not the story.

The story is rework going to zero on the stuff we can predict. Deliver the PR in one take. Ace it at 100%. The only red you should tolerate is the unknowable.

Hypothesis

If you show the agent what CI will check before it writes product code, give it a page of notes, and make it pass a real practice run on a sidecar, preventable outer-loop failures should stop showing up. Fix loops should not be necessary. Green commits should hit 100%.

Not “prompt harder.” Setup for success. Ace it at 100%. One take.

Setup

In this lab, an A- is still red. Local lint that smiled while outer-loop CI frowned still fails. A green sidecar run that dies on the real pipeline still fails.

Two cadences. Same bar: every commit you push to outer-loop CI is green.

• Per-task-push: after each task clears practice run, push. Seven tasks, seven trips. Ace every one.

• Single-push: finish the stack, clear practice run, push once at the end. Ace that one.

How often you push is a later lever. First you have to be equipped to push green.

Without the sneak peek

Three calibrations on the same AFK classic Snake game build. Seven tasks. Hands off the keyboard.

No inventory of checks up front. No notes. Inner-loop checks existed, but they were split between the localhost and a sidecar.

They finished, but rework kept piling up. When we feed agents endless retries and mountains of feedback, we’re just repeating the same habits we once drilled into human developers, now adopted by their AI counterparts.

*NOTE: We can make retries cheaper with a structured CI pipeline --failure-report, but that is its own write-up. Spoiler: even if you cheapen input tokens on each retry, you still have to pay time and money to retry.

Then we showed it the test

Before task one, before a line of game code, the agent got a short inventory: which checks fire on the practice run, which jobs fire on the thick outer pipeline, and pointers to the scripts and configs that define them. Descriptive. Not an answer key.

It took notes (preflight.md). It could @-read those configs when it needed more.

In our codebase that inventory is a “CI validation manifest”, which basically tells the agent what will be graded, and where the grading logic lives, before it starts guessing from vibes.

Inner-loop checks moved fully onto the sidecar, and we stopped running the same exact lint and test loops on the localhost. Outer-loop CI stayed the honest final exam, including whatever the world still wanted to throw at it.

Results

Headline: 0 CI fix loops on both sneak-peek arms.

• Per-task-push: 7/7 pushed commits green.

• Single-push: the one end-of-run push went green on first contact.

Zero outer-loop CI failures on either arm.

Diagnosis tokens on the rework line: $0. That budget can fund first-pass product code generation instead of burning LLM dollars and CI credits on red -> green archaeology.

Wall-clock did not collapse. Still about 50 to 60 minutes. Coding time dominated. LLM dollars landed about $15 to $17, in line with earlier attempts. The key difference: rework nearly disappeared.

Quiet rework is the same muscle Merge Efficiency Ratio names in CircleCI’s State of Software Delivery Q2 Pulse report: how many validation cycles before a change is actually done. This lab is that problem inside one agent setup. Once green is likely, push cadence and where you place the checks are how you bake low MER into the loop.

TL;DR

A coding agent can only prevent what we can predict. Say that out loud.

Linters. Unit tests. Typechecks. Formatters. Lockfile rules. Contract tests you already wrote. The deterministic stuff with a known answer. If it is on the syllabus, the agent can study it, practice it, and clear it before you spend an outer-loop credit.

It cannot prevent what the world invents after the peek is printed.

Late-breaking CVEs mid-run. A registry serving a different tarball than the one you resolved an hour ago. A base image that rotated overnight. An org policy that fails a dependency you did not touch. A flaky third-party API. A secrets scanner lighting up on a fixture. A new advisory. A mirror outage. A quota. Runner drift. “Works on my machine” that is really the cloud’s clock, certs, or DNS.

Platform engineers, harness authors, and coding agents cannot predict this. It arrives when the CI pipeline is already running.

Design for what the agent can own. Leave outer-loop CI in place for the rest. Ideally one take. At most two, once the world gets a vote.

Build the agent setup so it can ship the application end-to-end with zero CI failures from known deterministic sources.

Inventory of checks. Notes. A practice surface that is real (sidecar, not theater). Then push.

Do not “prompt harder” and hope the fix loop works overtime.

One take. Let’s go.

We wanted to understand: How effective was Chunk? Was there something we could do to improve the fixes given to the users? And perhaps most importantly, was there a way to improve the accessibility of the failure data to agents?

We tested the effectiveness in a controlled eval benchmark to measure the actual impact. The same dataset, the same model (claude-sonnet-5), against three different setups: no CLI, CLI with free use, and CLI with --failure-report. Here is what we found.

The context

Our first experiments were designed to mimic how Chunk works: giving it a file, then a generic prompt written by the team (for example “you are a coding agent, you need to solve this error” etc). Essentially at the beginning, we were trying to change the prompt to see if the fixes improved, and if giving certain instructions, steps, or validations would move the needle. The reality? The prompt didn’t seem to matter much.

Our team was constantly reading blog posts from other AI or CI/CD companies about what they were doing to solve this problem, and eventually we came across a paper by Bytedance. The paper shared that when cleaning log output, most of it is useless so the best thing to do is focus on the part of the log that contains the error.

We began to wonder: How do we remove the unnecessary characters and do a diff between the good and bad logs? That was the first successful experiment that gave us good results. It followed the way Chunk works. Because of that, we released the API of step output condensed. You fetch the output of a failed step and process to clean it, so you give better context to the agent.

From there, our focus eventually moved away from Chunk and toward the inner loop. We redesigned the experiment to mimic the usual developer flow.

“I thought: ‘If I was a dev working on a project and my pipeline failed, how would I fix it with an agent? Probably by copying the pipeline URL and giving it to an agent to fix. I saw that it had to follow a lot of steps to get to the pipeline output. The agent has context to the pull request, but it doesn’t necessarily have sufficient context.’”

Joaquin Sandoval Miramontes, Senior Software Engineer, CircleCI

Hypothesis

The hypothesis we arrived at was this: CI failure data is structured for humans reading a dashboard: status icons, log streams, nested UI. An agent working from that raw output burns context on navigation before it can start fixing. Therefore, restructuring the same data for agent consumption should reduce diagnostic turns and get the agent to fix faster.

Setup

I always try to do experiments in a structured fashion. I want to make sure that I have a way to repeat my own results over and over, and I’m particular about designing experiments with a hypothesis, expected results, and conclusions – I try to follow the scientific process wherever possible.

So here’s what we tried.

New flag: circleci run get [run_id] –failure-report

Output format: condensed, organized [workflow → job → step], showing only what failed. Built for piping directly into an agent’s context window.

circleci run get [run_id] --failure-report | claude "fix the failing workflow"

We compared three conditions on the same real failed pipelines from production:

• An agent given authenticated access to the CircleCI API

• An agent given free use of the CircleCI CLI

• An agent given access to the CircleCI CLI instructed to use --failure-report command

Results

Efficiency

The flag delivers real efficiency gains:

Agents using --failure-report spent fewer turns overall and finished faster. The CLI-only variant used the most turns and tokens due to agents navigating the CLI tool chain to resolve which job failed.

Fix Quality

Fix quality is flat across all variants. Fix rate difference is within the regular variation across agentic runs.

TL;DR

It used to be that an agent, when handed a failed CI run, spent many turns just figuring out what broke. The problem: the failure data was there, it just wasn’t accessible by an agent. The new command circleci run get [run_id] –failure-report changes that.

Previously, the agent was taking too many steps to get to the error itself, so we summarized it into one line. More context for the agent means better fixes, and spending less.

“If you have repeated tasks for your agent, give them tools to get it done faster. Don’t make your agent do 3-4 things before it gets started.”
Joaquin Sandoval Miramontes, Senior Software Engineer, CircleCI

The same failure data, restructured for agent consumption, changes how efficiently the agent works. The agent still needs to gather context from the source files, understand the failure and write the fix. What it no longer needs to do is spend 4-5 turns navigating a log hierarchy to find the failing step.

The practical conclusion: agents using --failure-report reach the same fixes in about ⅓ less time and ¼ fewer tokens. At scale, given thousands of CI fixes per day, that efficiency difference is real.

Has your team run any similar experiments? What have you learned about restructuring failure data for agent consumption? We’d love to hear from you in the comments below.

Data: 84 real CI failures from CircleCI. Model: claude-sonnet-5. Quality Judge Model: Claude-opus-4.6.

That loop runs on my machine most days. I wanted a cheaper version of it, and I had a specific idea about where the savings were hiding.

So I broke six CircleCI pipelines on purpose. Then I saved one specific failed run of each, so every model would stare at the exact same wreck. Three AI coding setups, one instruction: fix this.

Grok 4.6 fixed all six. Twenty-two cents a fix.

Claude Sonnet 5, turned all the way up, fixed three. Every fix it did land cost about $1.11.

I walked in looking for the opposite result. The chatter said Grok would be the one to stumble on the terminal work.

What I was actually hunting

A routing rule, which is a boring thing with a boring name: a policy for which model gets which wreck. Teams already do this with humans. The YAML typo goes to whoever’s on rotation. The bash failure that’s been red since Tuesday goes to the one person who can actually read a wait loop. I wanted that same escalation in software, with a price tag on it. Hard shell problems to the expensive model, broken config and missing test results to the cheap one. Same quality of fix, smaller bill at the end of the month.

The stake is your inner loop. Every time an agent picks up a red pipeline, reads the logs, edits the config, triggers a run, and reads that too, somebody is getting billed. Running all of it on the expensive model, all week, is a line item.

That split never showed up.

What did show up is a way to measure this stuff without fooling myself. I connected CircleCI’s command line tools to both Claude Code and Cursor. I saved a red run, so nobody could grade a different pipeline than the one I picked. Then I refused to grade the model’s essay for a proposed fix. By “essay” I mean the model’s written diagnosis of what broke and what it would change, which is a completely different thing from whether CircleCI actually went green.

Agents are already good at the essay. The actual pipeline rerun is the score.

The essay is the menu. Green CI is the meal. I came to eat.

Hypothesis

I expected Grok 4.6 to struggle on the bash-flavored failures and hold its own everywhere else.

That was the prediction. I went in with a bias: if any model was going to struggle on shell, it would be Grok 4.6 once it had to drive a terminal. Type a command, read what came back, adjust, type the next one. Shell breakage is exactly that, over and over, so that’s where I thought a gap would show.

If the reputation held, the plan was already written. Sonnet 5 at effort: low takes the bash cases, Grok takes the rest, and I pay less without shipping worse fixes.

Second question, cheaper to ask and just as useful: does paying for Sonnet 5 at effort: high buy enough extra passes to justify the receipt?

Setup: six puzzles, three setups, one judge

The puzzles came from the CircleCI Training Koans, a public set of short exercises that break a pipeline on purpose so you can practice un-breaking it. I used six on a throwaway project, two per flavor of broken.

Three flavors went in. Broken config, where the pipeline file itself is wrong. Tests that don’t report, where the tests genuinely run and pass but CircleCI never receives the results, so the job looks fine and tells you nothing. And shell problems: bash logic, wait loops, cache commands, container images, services.

Then I froze them: one saved failed run per puzzle, so “whatever failed most recently” couldn’t sneak in and change the question halfway through.

Three setups, same prompt every time.

Those dollar figures are sticker rates for tokens, not the cost of a run. What each trial actually spent shows up in Results.

Changed: the setup. Held constant: the saved failed run, the prompt, the isolation, the spend ceiling.

Every trial started in a fresh clone. No resumed sessions. Sonnet never got to peek at what Grok had already worked out, and the two Sonnet effort settings didn’t share a session either. Each agent had to check its config change, then trigger a brand new CircleCI run on its own throwaway branch. Never main. Never the saved branch.

Now the grade, which is the part that maps to your actual day. Nobody merges the model’s write-up. You merge green CI. So a trial passes only if the pipeline that agent triggered finished green. On the tests-that-don’t-report puzzles, a green job with an empty test results panel (CircleCI’s Test Summary) is still a fail, because tests that pass without reporting anything back aren’t a fix. Afterward I re-checked every run against CircleCI’s own record, so the numbers below come from CircleCI and not from an agent’s summary of its own brilliance.

I capped spending too. I watched the expensive setup burn money during calibration, took its 95th percentile, and set the ceiling at $1.25 so a trial could go long without going stupid. Claude Code can hard-stop at that number. Cursor writes it down and keeps going.

Claude dollars are whatever Claude Code billed for the whole session, including the Haiku 4.5 helper that sometimes rides along. Grok dollars are tokens in and out times the published rate, not an invoice from Cursor.

One honest wrinkle: model and host are welded together here. Sonnet only runs on Claude Code, Grok 4.6 only on Cursor CLI. I can’t pull those apart, and I’d rather say it out loud than bury it in a footnote.

Eighteen isolated trials. Then a repeat round on the same six puzzles, then one genuinely messy red Playwright job. Three separate scoreboards, and averaging them would only make the numbers look tidier than the evidence is.

Results

The pilot, scored the day it ran.

All three greened both shell puzzles. The terminal gap I came hunting for never showed.

One puzzle was a config file so mangled that CircleCI rejected it before any job started. That’s why the original run was red: nothing ran at all. Fixing it meant repairing the file, triggering a new pipeline, and getting that new pipeline green. Only Grok’s new run went green. Sonnet at effort: low spent the entire $1.25 cap on that one puzzle and got cut off. Sonnet at effort: high failed it too.

Separately, Sonnet at high effort missed both puzzles where the tests ran but CircleCI never received the results. Zero for two.

I ran the same six puzzles a second time. Fresh start, no memory of the first try. Grok fixed all six again. Add both tries together and Grok is 12 for 12, Sonnet at low effort is 9 for 12, Sonnet at high effort is 7 for 12. Same homework twice. I wanted to know if Grok just got lucky on Monday. It didn’t.

Then the follow-up. I cloned a Playwright job that had been going red over and over, again a throwaway, not production. The bug was a locator: getByText matched both a heading and a button, so the test found two things where it wanted one and refused to guess.

All three models got that one pipeline green. One try each. Nobody deleted tests. Nobody touched .circleci/config.yml. Every one of them edited the Playwright test and switched to getByRole(”heading”, ...). Inside each green job, CircleCI’s test results panel listed six Playwright tests, and all six passed.

Grok was the cheapest and the slowest. High effort and low effort wrote the same fix. One case, three trials, which makes it an anecdote, and I’m labeling it one.

TL;DR

I didn’t get a routing policy. Two puzzles per flavor of broken is nowhere near enough to write a company rule about where your model budget goes. Grok held the shell cases, too. And paying for effort: high didn’t buy extra passes. On these puzzles it bought fewer.

What I did get is a test that refuses to lie to me. Most evals grade what the model wrote. This one waits for CircleCI to finish, then takes the verdict from the run itself. Execution-graded, meaning CircleCI’s pass or fail is the grade, not a rubric and not the model’s own confidence.

If you’re wiring agents into CI, that distinction is the entire ballgame.

Cheap can win when execution is the judge. Twenty-two cents a successful fix against $1.11 isn’t a brand story. It’s a 5x token bill, on this lab set, every time an agent tries to get a red pipeline green. I’m not going to invent your volume. A team of ten, five of those loops a week, is about $45 extra to pay for high effort instead of Grok. That’s the unit. Not a finance-team emergency. Still a 5x.

I evaluate models for a living. Then I have to evaluate the eval.

Scoring a write-up is asking somebody how they feel. Scoring a CI pipeline rerun is taking their actual blood pressure.

RalphCI already showed that green on your laptop isn’t green in CI. This one asks the ruder follow-up: hand an agent the logs, the CircleCI tools, and one saved red run, and can it actually fix the pipeline? On this lab set, yes. Often. The expensive knob wasn’t the reason.

I’m not saying always use Grok instead of Sonnet. Hosts behave differently, Cursor can’t enforce a dollar cap, and these were clean training puzzles plus one Playwright locator.

I’m saying this: if your agent loop can’t trigger a rerun and read the test results panel, you’re grading the menu instead of the meal.

Five months later, I am still coding the same game of Snake. Now 100% AFK, with much more confidence than when I started. Zero human supervision.

The first half of 2026 has been fun, but not in the arcade sense. It has been fun in the “I haven’t touched my keyboard in twenty minutes, and every check on the branch is green” sense. We kept building the same Snake game. Same 20x20 grid. Same arrow keys. Same TDD spec. Same retro aesthetic.

The game has barely changed, but the delivery loop has changed everything.

If you have been watching the AI agent hype cycle from a safe distance, here is the compressed version of what we learned by staying stubbornly on one benchmark: you can go AFK with rising confidence, greener pull requests, and faster feedback than we have ever measured. Not by trusting the agent more. By giving it better and quicker back-pressure at each layer of validation.

The benchmark nobody asked for (but everybody needed)

Snake is a toy. That is the point.

A toy spec is small enough to run ten times in a week. Small enough to isolate one variable at a time. Small enough that when something breaks, you can actually read the diff instead of drowning in a monorepo.

We gave agents the same task repeatedly: build a playable Snake game from scratch using test-driven development (TDD). Seven tasks. Canvas rendering. Collision detection. Score tracking. Push commits. Open a PR. Walk away.

Every run finished locally. Every agent said the tests passed. Every game worked on the laptop and was pretty fun to play.

January: everything is a Ralph loop

In January, Geoffrey Huntley published everything is a ralph loop. The mindset shift is not “use AI to type faster.” It is “program the loop.”

Allocate a goal. Run the loop. Watch the failures. Fix the failure domain so it never happens again. Repeat until done or until you hit CTRL+C and take the wheel back.

Basic Ralph loops can build software and generate PRs AFK. That part worked decently well (with some light human supervision).

The PR at the end might be green. It might be red. The agent does not inherently know which until something external tells it.

We were building Snake games in loops before we had a name for what we were doing.

The lesson from January: autonomy without validation is just faster guessing.

February: does the pipeline agree with local tests?

By February we had data, not vibes, on whether the pipeline agreed.

Our first study ran ten controlled Snake builds. Same spec. Same model scaffolding. One variable: five runs wired to CircleCI through RalphCI, five runs without.

All ten agents completed every task. All ten passed local tests. All ten declared victory.

Eighty percent of the no-CI runs shipped code the pipeline rejected. The failure was mundane: single quotes where ESLint wanted double quotes. Local tests do not check lint rules. CI does. The gap between “works on my machine” and “works in your pipeline” is where agents lie with confidence.

RalphCI fixed that by injecting real pipeline status into the agent loop after push. When CI failed, a CI Doctor agent read the logs, applied fixes, and pushed again. Twelve failures across five runs. Twelve autonomous fixes. Zero human triage.

The breakthrough in February: agentic loops with CI back-pressure can land green PRs AFK. The final PR passes CI. Commits inside that PR might still fail along the way. Red commits on the branch. Green at the end. Progress, not perfection.

Early May: every commit has to be green

Three months later we hit the next failure domain.

Local pnpm test:run kept passing on my Mac. Chunk sidecars kept failing on Linux. Missing LEGAL_DISCLAIMER.md. Missing SNAKE.md experiment marker. A TypeScript compile error the Snake workspace’s local tests never exercised but a full pnpm build caught.

Same pattern, five runs in a row. Local green. Sidecar red. CI Doctor lands a fix(ci-sidecar): commit. Sidecar green. Push allowed.

Our early May study wired Chunk sidecars into RalphCI’s Review Gate. A sidecar is a lightweight remote microVM: your workspace syncs there, Chunk runs a microbuild that mirrors your pipeline intent, and you get CI-shaped feedback before the branch leaves your laptop.

Five Snake builds. Five 100% green PRs. Every commit green. Not just the last one.

I had not touched the keyboard in twenty minutes. The final PR (and every commit on it) was green anyway.

This screenshot is what HIGH CONFIDENCE looks like.

The May breakthrough: three validation layers (local, sidecar, pipeline) produce PRs where humans review green branches only. Sidecars catch CI-shaped failures while the agent still has context. CircleCI remains the authority at merge time. Sidecars extend CI into the inner loop. They do not replace it.

Late May: deliver the same green outcome, faster

100% Green PRs answered “can we merge with confidence?” A follow-up experiment asked “how fast can we merge with confidence?”

Our late May study ran the same ten Go tasks two ways on chunk-cli: sidecar validation per iteration versus commit-push-poll CircleCI per iteration. Same agent. Same model. Same gate jobs lint and test). Different machinery.

Sixty-nine seconds. Every iteration. Five replicates in a row. That is not random noise in our harness. That is the queue tax on push-per-task CI.

Twenty-two seconds on a warmed sidecar snapshot. Same question: are lint and test happy right now? Same answer. Different delivery speed.

Token spend was flat because in this particular experiment tokens tracked fixes, not wait. The CI arm burned more clock time, not more intelligence, so sidecars did not shrink the LLM bill here. They shrunk idle time. On a ten-task run that is roughly eight minutes of gate-waiting saved before the final pipeline epilogue.

The May speed breakthrough: you still want an outer loop. Sidecar runs finished with a full ci workflow epilogue after one push. Fast inner loop on each task, authoritative pipeline confirmation at the end. Not sidecar-only cowboy coding. Not push-and-pray either.

Four layers, one arc

Strip away the Snake skin and the arc looks like this:

We are not gaining more AFK confidence because we trust agents more (although the models have improved since January). We are gaining more AFK confidence because each failure domain we hit got engineered out of the loop. And we’ve even unleashed some speed gains along the way.

Watch the loop. That is Huntley’s line and it is the whole game.

• January taught us to program the AFK agent loop.

• February taught us the agent’s “I passed” is local theater without CI.

• Early May taught us that 100% green PRs are possible, given sidecar validation before commit and push.

• Late May taught us that 100% green PRs are still the prize, and sidecars get us there faster than push-per-task CI.

Snake games did not teach us that AI writes good code. They taught us where validation has to live when the coder is an agent loop and the reviewer is still a human with a merge button.

What I am actually doing with this

I am obviously not shipping Snake to production. (Unless you’ve been craving retro gaming experiences. DM me.)

I am shipping the stack: RalphCI orchestration, Review Gate with local checks plus Chunk sidecar validation, CI Doctor for autonomous repair, CircleCI as the outer loop authority. Same pattern we pressure-tested on a toy game because toys fail fast and teach faster.

Four layers, one arc. Program the loop. Wire in CI back-pressure. Validate on sidecars before push. Maintain CI pipeline authority at the end.

Each layer kills a failure domain the last one missed. Confidence climbs. PRs go greener. Feedback gets faster. The human at the merge button only reviews green.

That is what five months of the same Snake game bought us.

Green CI is still priceless. Everything else is noise.

Where to read the lab reports

• everything is a ralph loop (Geoffrey Huntley, Jan 2026)

• We Let an AI Agent Say “I Passed.” Was It Actually Good? (Feb 2026)

• AFK Builds with 100% Green PRs (May 2026)

• The Sidecar Race: 22 Seconds vs 69 Seconds (May 2026)

More Snake runs coming. Heavier repos. Flakier tests. Parallel agents. Improved CLI. Better agent-facing APIs. You name it.

The benchmark stays small on purpose, and we intend to scale the lessons as we go.

Twenty-two seconds is what the other arm averaged for the same question on a Chunk sidecar.

Same repo. Same ten Go tasks. Same Claude Agent SDK edits. Same gate jobs. Different loop.

Three weeks ago I published AFK Builds with 100% Green PRs. Five Snake games. RalphCI orchestration. Sidecars in the Review Gate so agents stopped feeding red commits to GitHub. Green PRs were the win.

This follow-up is narrower and, I think, more revealing for anyone running agents today: not whether sidecars help you merge green, but how fast the agent gets CI-shaped feedback while the file is still open.

I kicked the tires on sidecars again.

They are fast.

Not a much cheaper agent. A less idle one. Token spend was flat in this particular experiment (more on that later).

The win was clock time.

Hypothesis

If each agent task is validated with chunk sidecar sync plus chunk validate --remote lint and test-changed) instead of commit → push → poll CircleCI for the same lint and test jobs, then:

1. Median time to signal (TTS) per iteration will be materially lower on the sidecar arm.

2. LLM cost will be similar (same agent, same ten prompts, same model fixing the same mistakes).

3. CircleCI still matters for pipeline-level confirmation (sidecar runs end with a full ci workflow epilogue after one final push).

I expected push-per-task CI to pay a queue tax every iteration. However, I did not expect the median to land on 69 seconds five replicates in a row.

Setup

Repo:

• CircleCI-Public/chunk-cli

• Experiment PR #370

Tasks: Ten cumulative edits on a small Go fixture internal/racefixture/ on experimental run branches).

Replicates: Five labels 001–005) per arm. 10 recorded runs total.

Metric: TTS = wall-clock seconds from iteration start until both gates report pass/fail (logged in results.csv`).

CircleCI CTO Rob Zuber frames this as rebalancing inner and outer loop validation. My May post showed the outcome layer (green PRs). This post measures the wait layer.

Results

Headline: median time to signal

Aggregate: median of per-run medians across five replicates.

Five replicates at a glance

Sidecar clustered 20–22s. CI sat on 69s every time. That stability is either a real push-and-queue signature in our harness or a coincidence at lab scale. I am reporting it, not universalizing it.

p95 TTS: sidecar ~23–25s; CI ~72–99s (tails worse on push-per-task).

LLM cost comparison explained

The speed table is not the only controlled comparison in this harness. Token spend is measured the same way: same model claude-sonnet-4-20250514), same ten tasks, same agent, same fix-and-retry pattern. We changed the validation path, not the work.

Both arms hit the same lint and test failures and applied the same corrections. Sidecar surfaced them via chunk validate --remote on the warmed snapshot (~22s median). The CI arm surfaced them after push to GitHub and a ~3× longer wait for the same gate jobs (~69s median). Same signal, different delivery speed.

That is why token spend landed flat. Tokens track fixes, not wait. The CI arm cost more clock time, not more tokens. Sidecars did not shrink the LLM bill here because there was nothing to shrink: identical failure set, identical agent work on both arms. On a messier repo, especially if remote CI surfaces failures the sidecar doesn’t mirror, token spend might diverge. This harness didn’t test that.

So treat ~$0.90 per run as what this harness actually cost for this specific experiment, not a budget for production agent loops. Whether faster feedback trims token burn on messier work and CI failure modes (i.e. given fewer stale-context retries, less thrash, and less environment drift) is a follow-on question for a future study (coming soon). This race did not answer it.

This experiment answers a narrow question cleanly: when the agent does the same fixes either way, faster feedback does not change token spend. It changes how long the agent sits idle.

Same gate jobs. Different machinery.

We timed the same question on both arms: did lint and test pass? We did not run identical validation. Sidecar answers via chunk validate --remote on a warmed-up Linux snapshot. CI answers via real CircleCI jobs after you push.

Early iterations on both arms still saw lint failures while the agent fixed mistakes (about two failing lint iters per run in the rollup). The race is not “sidecar never fails.” It is how long you wait to learn.

Illustrative lab math only: ten tasks × ~47s saved ≈ ~8 minutes less gate-waiting per replicate before the sidecar epilogue. Your mileage will vary.

Takeaway

Green PRs are the outcome. Time to signal is the throttle.

The May experiment showed sidecars help agents stop polluting branches with commits CI would reject. This one shows why that matters in wall clock terms: the coding agent is not stuck in queue while context goes cold.

What held across ten runs in chunk-cli:

1. Sidecars save feedback latency on mirrored gate jobs (~3.1× median TTS here).

2. LLM spend was flat (~$4.64 sidecar vs ~$4.73 CI) in the same apples-to-apples comparison: same failures, same fixes, CI just took ~3× longer to surface them. Tokens track fixes, not wait.

3. Push-per-task CI is a predictable wait in this harness (69s median, every replicate).

4. You still want an outer loop. Sidecar runs finished with a full ci workflow epilogue. Fast inner loop on each task, plus an authoritative pipeline confirmation at the end. Not sidecar-only cowboy coding.

Please re-read #4 above: I am not at all suggesting you delete your CI pipelines. Quite the opposite: I am saying that if your agent loop is push → wait → fix → push again, you are paying for that wait every iteration.

Sidecars moved the same exact question to a local, warmed-up Linux snapshot...

QUESTION: Are lint and test happy on this repo right now?

...and the same answers came with a much tighter feedback loop. 3.1× faster.

What’s Next

• Combine this harness with RalphCI orchestration (Snake outcome study + TTS race in one stack).

• Heavier repos, flaky tests, parallel agents. Find where Chunk sidecars and snapshot sync stops winning.

• Agent economics at scale: cost per green iteration (LLM + validation path), not just TTS. A proper cohort study on heavier repos, not extrapolated from this ten-run table alone.

• Demo recordings on various Chunk sidecar and snapshot sync setups. Stay tuned!

Artifacts:

• FINDINGS.md

• Experiment PR #370

If sidecars are new to you, a Chunk sidecar is a lightweight remote microVM that sits in your inner loop. Your workspace syncs there, Chunk runs a microbuild (lint, tests, build, and the repo policy hooks you configured), and you get CI-shaped feedback on Linux before your branch leaves the laptop.

CircleCI CTO Rob Zuber describes the goal as rebalancing inner and outer loop validation so agents are not learning about failures only after push.

Our three early sidecar failures were unrelated. First, a missing LEGAL_DISCLAIMER.md at the repo root (legal policy gate). Second, a missing SNAKE.md marker file in the experiment workspace (documentation policy gate). Third, a TypeScript compile error the Snake workspace’s local tests never exercised but the sidecar’s full pnpm build caught.

All three times, the lint-and-test run on my Mac localhost said green, but the red Chunk sidecar said “fix this stuff.” CI Doctor landed one fix(ci-sidecar): commit per issue. Only then did my local Review Gate allow a push to GitHub.

That is a small lab story. We think it is a bigger CI story.

We humbly believe we are building toward the future of continuous integration in the agent era: validation woven through the whole loop, ending on a 100% green pull request you can merge with confidence.

Local lint and tests first. A Chunk sidecar microbuild that partially mirrors your pipeline while the agent still has context. Then CircleCI on the branch, doing what CI has always done, checking integration before a human approves merge.

AFK application builds that land as 100% green PRs are what we can demo today. The thesis underneath: CI validation that keeps up with the pace of your local coding agents.

We ran five Snake game builds to pressure-test that future. What follows is the lab report.

Hypothesis

If RalphCI runs Chunk sidecar validation chunk validate --remote) after local lint:fix and test:run, then:

1. A coding agent can run mostly unattended through build and fix cycles (while a CI Doctor agent handles sidecar and pipeline failures locally).

2. Commits that pass the local Review Gate and push to GitHub show up on the branch as a 100% green PR: with every mirrored CI pipeline passing on the branch.

3. CircleCI remains the authority at merge time. Sidecars extend CI into the inner loop; they do not replace it.

Our February 10 study already showed CI-aware RalphCI can get to green pipelines on this Snake benchmark when the agent answers to CircleCI after push. This experiment adds pre-push CI parity via sidecars so the PR itself is green top to bottom, and not just at the end of the PR.

Setup

Benchmark: Five independent runs in ralph-ci/experiments/w_chunk-sidecars/iteration-{1..5}/. Same Snake spec as prior Loop Lab RalphCI work: 7 tasks, Vitest TDD, canvas game.

The stack (from top to bottom):

Chunk mirrors CI: .chunk/config.json runs policy gates --gate-legal, --gate-snake) plus remote pnpm lint, pnpm test:run, pnpm build, matching CircleCI jobs on the branch.

Sample size: 5 runs. Lab notebook scale.

Results

Five runs at a glance

Totals: 57 RalphCI iterations, 16 sidecar repair loops, 5/5 runs ended with all tasks complete and green CircleCI on the branch.

Source: iteration-*/metrics.json and activity.md.

Local green is not PR green

Every run, early iterations: Review Gate fails on Chunk sidecar, not local lint-and-test run. On all runs, the first three sidecar fixes were: restore LEGAL_DISCLAIMER.md (legal gate), add the SNAKE.md experiment marker file (documentation gate), then fix a tsc error in run-ci.ts surfaced by the monorepo build step.

Same pattern across runs: failures local tests never see. CI Doctor agent fixes them without a human in the loop. Only then does the local Review Gate allow a push to GitHub.

After the sidecar gate cleared, pushes stayed green on CircleCI. All CircleCI pipelines passed upon push to remote. No surprise reds on mirrored work.

Compared to our February baseline

On the same Snake benchmark without sidecars in the Review Gate, agents often finished locally while the branch still failed CircleCI.

With sidecars before push plus CI on the branch, five for five ended as 100% green PRs in our runs: build, lint, policy gates, and test jobs all passing on the merge target after push to GitHub.

The continuous integration breakthrough is three layers: local, sidecar, pipeline. AFK works when failure shows up in layer two, so layer three confirms instead of contradicts.

Takeaway

Rob Zuber framed inner-loop validation for agents. RalphCI plus Chunk sidecars are one way to wire that into an orchestration loop so AFK does not mean “ship and pray.”

What held across five Snake game builds:

1. Sidecars catch CI-shaped failures while the agent still has context.

2. A 100% green PR needs three layers: local, sidecar, pipeline. Skip one, get surprised downstream.

3. Humans review green branches ONLY. They rarely see red, not even on a single commit.

Green CI is still priceless. Chunk sidecars are how you earn a 100% green PR before you even ask a human to look, and a green PR instills more confidence in the agent’s work than a red one.

The artifact: a 100% green PR ready for review

This is the outcome we care about. A pull request an agent built while I was AFK, with every last CircleCI check green on the branch after push to GitHub.

Twelve commits on the branch. Task work, three fix(ci-sidecar): repairs, finalize, metrics. At the bottom of the page: All checks have passed. Five green jobs:

• ci/circleci: build

• ci/circleci: lint

• ci/circleci: require-legal-disclaimer-md

• ci/circleci: require-snake-md

• ci/circleci: test

No conflicts with base. Merge pull request is the honest end state.

That screenshot is what we mean by AFK builds and 100% green PRs: every CircleCI check green before a human opens the branch, because validation ran local, then on a Chunk sidecar, then on the pipeline, and the agent fixed failures in between.

Five Snake game builds proved the workflow. In future runs, we may expand the repo complexity and scope, and the review gates along with it, to see how far we can push this thesis.

But we will keep the overall stack, because we humbly think that this is where continuous integration is going.

Onboarding for those teams is never “read the README and go.” It is access to the right systems, a path through tooling that matches how work really flows, habits that stay safe under pressure, and knowing who to ping when something breaks.

We stopped treating that ramp like a scavenger hunt through stale wikis and shipped it as something you can run: a team-scoped Claude Code plugin made of skills, small playbooks the agent follows, with MCP checks so “connected” means verified, not hoped for.

Hypothesis

A Team Onboarding Buddy should beat the default experience: read twelve pages, miss the thirteenth, ping three people, discover your MCP tokens are wrong, and lose a week proving you are not incompetent, you are just new.

If onboarding is small, invocable skills with explicit triggers, and those skills include MCP health checks plus a real access checklist, a new hire should get to first useful work with less thrash. The skills do not replace humans. They replace the worst part of onboarding: reconstructing tribal knowledge from stale breadcrumbs.

Setup

What we built. A team-scoped plugin: four skills in one bundle, installable the same way you would ship any other Claude Code plugin to a group that should share one playbook.

Our first internal cut targeted one engineering sub-team’s playbook before we framed this as a pattern anyone could clone.

Repo shape (quick and dirty). In a Claude Code plugin marketplace repo you usually get one folder per plugin: a small manifest under .claude-plugin/ and a skills/ directory full of SKILL.md playbooks. The folder name should match the plugin id in plugin.json (that becomes the /namespace:skill prefix in Claude Code). Rename team-name to whatever id you pick in plugin.json.

Examples of folder / namespace names (illustrative only): engineering, marketing, product, design, legal, finance, people-ops.

Why this shape. The skills are modular on purpose. Nobody wants a forty-page monolith. A new hire can pick sections like a menu instead of pretending they will read the whole handbook on day one.

The MCP angle. The access skill does something wikis are bad at. It tells the agent to verify integrations with real reads through connected tools, not to assume OAuth worked because nothing exploded yet.

The human angle. Long-form runbooks still live wherever your company keeps them. Escalation still names real humans. The goal is not autonomous HR. The goal is fewer round trips for questions the team has already answered a hundred times.

Results

What we shipped.

• A versioned, team-scoped bundle so one group can adopt the buddy without forking the whole company agent config.

• Hundreds of lines of structured guidance across the four skills: MCP verification steps, access and tooling checks, and a clear path from “green integrations” toward first safe change.

What we learned in practice (qualitative).

• The entry skill actually gets used. People do not have to remember the “right” order. The plugin encodes it.

• MCP verification turns a vague “I think I am set up” into a pass or fail list. When something is wrong, the output is actionable instead of confidently wrong.

• The team skill fills a gap most organizations leave empty: who shows up from outside the group, for what reasons, and where the sharp edges are. That is context people often learn only after they step on a rake.

How to sanity-check your own build (five minutes).

• Invoke the entry skill and confirm it routes into access, local-dev, and team flows in a sensible order.

• Run the MCP verification path and confirm failures degrade gracefully: they say what is broken and what to check next, instead of inventing green checks.

On metrics. We have strong internal signal on usability and fewer “false start” access threads. A controlled before/after (time to first local run, time to green MCP matrix, week-one ticket volume, short confidence survey) is the right next step if we want a chart in a sequel post. This write-up is the pattern and the ship story, not that chart yet.

Takeaway

Onboarding is not an information problem. It is a routing problem. You do not need more wiki pages. You need a default path that is hard to screw up, plus checks that fail loud when your tooling is not actually connected.

Packaging that path as Claude Code skills is a bet on where engineers already live (NOTE: It could be Cowork, in the case of a non-engineering team.).

Whether Claude Code or Cowork, the buddy belongs in the editor session, not on page seventeen of a PDF nobody opened.

The bundle bakes in habits that matter whenever financial correctness, access control, or customer-visible production behavior is on the line: look before you change anything (read-only checks first), exercise mutations in a safe mode (dry-run or sandbox before you touch live state), and do not solo your first production-adjacent change (pair with someone who has muscle memory).

That is not moralizing. It is how you avoid the expensive failure mode where a small onboarding shortcut becomes a customer-visible incident or a long support recovery.

Credits

Team Onboarding Buddy was a collaboration. Thanks to the team of engineers who built, reviewed, and tried it in real onboarding flows:

• Andrea Morato Dixon

• Fernando Lima

• Ryan Hamilton (yours truly)

If this pattern helps your org, please let us know!

What’s Next

1. Run a small cohort study with agreed metrics so we can publish numbers, not just vibes.

2. Reuse the skeleton across the org, for other teams: same four-skill shape onboarding, access, local-dev, team), different playbooks for both engineering and non-engineering teams.

3. Treat rosters and paths as perishable data. Point people at live chat and authoritative docs, refresh the skills when reality drifts.

Bottom line for anyone pitching this upstairs

This is not “AI replaces onboarding.” It is AI holds the checklist and verifies the wiring so humans spend time on judgment, not detective work.

If you are landing here cold: when we say “the agent’s work is done”, we mean “the CI pipeline is green”, not only “the local tests pass on your laptop”.

Here’s how it all fits together, step by step. The February study already showed the problem: local tests might pass, but without CI in the loop, pipelines often fail; when CI is part of the loop, results reliably match what ships. This experiment is about putting that hardened, CI-driven loop into the wild under an MIT license: tightening rough edges and preserving the multi-agent structure that keeps each role clear.

Code is on GitHub. Video walkthrough: Loop Lab demo on YouTube.

Hypothesis

I expected we could keep the behavior we measured in the February study, address issues that only show up under long runs, and ship RalphCI as a small CLI that other people can run without a custom one-off setup.

Stated as an if-then: if CI status check stays at the start of each task iteration, CI failures stay ahead of new feature work, and a deterministic review gate still runs before push, then tightening up the loop by open sourcing the project does not weaken the rule that completion waits on the real CI pipeline green state. In fact, it should make it stronger.

This is an engineering follow-up, not another n=10 Snake batch. The February study supplies the quantitative baseline.

This piece is what we changed to make that baseline shippable.

Setup

How RalphCI fits together (plain language)

Think of the ralphci CLI as a team of coding and review agents that sits between your repo, version control, and your CI pipeline. The model still edits files and runs tests on your machine. The CLI decides what kind of help the coding agent gets next and when commits actually get pushed to version control, so your CI bill and your definition of “green” stay connected to reality.

Iterations. One task iteration is one pass:

1. Check pipeline status

2. Run the agent on the current task

3. Run local checks

4. Commit if appropriate

5. Repeat until your task list is complete and, when CI is enabled, the remote pipeline is in a passing state.

CI status early each iteration. At the start of an iteration, the tool asks CircleCI how the latest work on your branch is doing (pass, fail, or still running). Results can be cached so the CLI is not calling the API when nothing new was pushed. That is how the agent gets a factual snapshot of the server-side build instead of guessing.

Build Agent versus CI Doctor. These are not two separate products. They are two modes the orchestrator selects:

• Build Agent handles normal forward progress when you are not blocked on a red pipeline (or CI is not in play). The prompt stays lighter and focused on the task list.

• CI Doctor turns on when the pipeline failed. The model receives full failure context (including logs) and is steered to fix what broke in CI before piling on new features. Same model; different job description. That is the “multi-agent” shape: split responsibilities instead of one blob of instructions.

Review Gate. Before a git push, the CLI runs your repo’s lint and test commands as ordinary shell steps, with a timeout so nothing hangs forever. No LLM in that step. If lint or tests fail, the push does not go out. The point is to catch formatting and unit-test issues locally before they become another failing pipeline run.

Smart Push. This label means the tool does not push on every save or every commit by default. It pushes when local checks (including the Review Gate) succeed, so you are less likely to burn CircleCI credits and time on work that already failed lint or tests on your machine. Push triggers CI; Smart Push aims to commit and push to version control only when the branch already survived those local checks.

When is the run “done”? With CI integration enabled, finishing your tasks is not enough if the pipeline is still red. The loop keeps going until the real CircleCI pipeline is green (per your config), not only until local tests pass. That mismatch between local and remote is exactly what the February study measured in numbers.

Everything above is the default CI-on story. You can tune behavior in ralphci.json and CLI flags without changing that core: for example branch strategy (feature branch vs pushing straight to main), draft vs ready PRs, whether the loop auto-pushes or waits for you, approval gates before you call a run complete, and ralphci run --no-ci when you want local-only iteration with no CircleCI token. See the repo README and AGENTS.md for the full matrix. Those options are packaging; the loop in this section is the part that stays constant when CI is enabled.

Where the baseline came from

The February study holds the detail: same Snake spec, same tasks, same model family, five runs with CI off and five with CI on. One headline from it: 100% of CI-enabled runs ended green on CI; 20% of local-only runs did, with local tests passing in both groups. If you only read one piece from this line of work, that article is the quantitative backbone.

What we changed after that baseline

Work between the February study snapshot and the public repo focused on reliability, clarity, and running on someone else’s machine, not on a new product story.

• The core loop (Build Agent, CI Doctor, Review Gate, Smart Push, and requiring CI to pass before you call the run complete) stayed the same. The main engineering effort focused on improving how each part handles tricky situations and exceptions. For example, we fixed cases where the CI Doctor would keep running even if a fix had already been applied locally. Pipeline status checks are now limited to just the current branch, preventing confusion with unrelated CI runs. When the pipeline is still running, the tool now polls periodically instead of waiting indefinitely. We also resolved issues with duplicate commits and inaccurate activity metrics, making the logs more reliable and easier to interpret.

• We removed non-essential paths. Experimental review-agent code came out. The CLI stays centered on this CI-aware, multi-agent-shaped loop.

• We made the tool easy to run from source. The ralphci binary runs TypeScript through tsx, without a separate build step before you iterate.

• We licensed it for reuse. MIT. github.com/CircleCI-Research/ralph-ci

• We recorded a demo. The YouTube recording walks through setup, a run, and CI feedback in the activity log.

Variables

Results

Quantitative bar (from the February study)

You do not need the February study to use the tool, but it is where these before/after numbers come from. From the same article:

This open-source release assumes that gap still matters. We did not run another ten Snake iterations for this document.

What follows is shipping and engineering evidence, not a second controlled trial.

Hardening phase (this release)

• Repository: github.com/CircleCI-Research/ralph-ci under MIT.

• CI Doctor and Review Gate: Fixes for cases such as doctor spinning and branch-scoped checks, covered by tests in the repo.

• Contributor flow: tsx entry, documented pnpm lint:fix && pnpm format:fix && pnpm test:run gate, conventional commit helpers for orchestration.

• Demo: Loop Lab recording on YouTube.

ESLint and environment drift

The February study often surfaced ESLint quote style drift. That is representative of what CI is for when agent defaults do not match project config. Hardening does not remove linting; it reduces thrash while the agent aligns with the rules.

Takeaway

The February study asked whether CI feedback changes outcomes when other factors are held constant. This phase asked whether we could publish the mechanism behind those outcomes while keeping the same accountability: green CI pipeline state matters for completion, not only local tests, and Smart Push (see above) still limits unnecessary CircleCI runs.

The design worth naming explicitly is specialization over one generic agent. CI debugging and feature work pull different context and different prompts. The Review Gate keeps a slice of quality checks out of the model entirely and makes them more deterministic. That separation is what we hardened for open source, not a bigger single prompt.

In the February study, we reference breadth-first agent work. That is the idea of trying several variants (more than one agent run or branch on the same spec) instead of one endless depth-first pass. It only makes sense if each variant can ship, meaning CI passes. The study’s main table is what happens when that is not true: same agent, same tasks, locals green, pipeline often still red. Open-sourcing RalphCI is about keeping that bar honest when you explore in parallel.

What’s Next

• Larger codebases than the Snake exercise. Real services, flaky tests, multi-job pipelines.

• More models and runners. Same contract, different agents; compare cost and failure handling.

• Community usage. Feedback from people wiring their own CircleCI projects.

• Tighter feedback loops. A game that can build itself in 20 to 30 minutes, 100% AFK, is great. Ten minutes would be better. We are tinkering with faster, smaller validation cycles using sandboxes the local agent can drive directly, so not every check has to wait on a full commit, push, and remote pipeline round trip. The same thread shows up in CircleCI’s autonomous validation framing: delivery that keeps pace with AI-era churn, including selective testing (run tests tied to what changed instead of the full suite on every pass) so you get quicker signal without abandoning coverage discipline.

• Failure-pattern intelligence. Today the loop is built to clear the red build in front of you. The next layer is memory across incidents: jobs that keep flaking, lint rules that trip the same way, fixes that do not stick. That is different from reacting to the latest log line. Chunk CLI sits near that problem: it mines GitHub PR review comments into markdown prompts for agents, wires validation into coding-agent hooks, and can run checks in cloud sandboxes before push. It is not wired into RalphCI, but it is a plausible place to experiment when we want CI assistance to learn from repetition, not only from the current failure.

• Live voice narration. We are also experimenting with voice-led tooling in the spirit of Claude LiveCaster: seven personalities across the SDLC, YAML-driven personas and /loop-style narration on top of real local agent and CI pipeline activity.

• New controlled runs when the question warrants a fresh n=10 design. This write-up connects the numbers from the February study to the current codebase; it does not replace new statistics.

For the original comparison tables, see the February study. For a walkthrough, use the YouTube demo. To build or fork, start from RalphCI on GitHub.

That worked. So I did what any reasonable person would do: I gave it six more personalities and told each one to narrate a different phase of the software development lifecycle.

A rodeo announcer for code reviews. A mission control operator for production deploys. A war room coordinator for incident response. A pair programmer narrating a solo app build. And a two-person broadcast booth for CI pipeline races, complete with a technical anchor and an excitable field correspondent speaking through different TTS voices.

Watch the demo on YouTube.

Hypothesis

I expected a YAML-driven configuration layer to make the stateless-cron architecture generalize. Swap the YAML, swap the entire narration experience, without changing the core announcer slash command.

The variable I was really testing: does the LLM actually follow persona instructions well enough to produce distinct commentary across wildly different scenarios? Or does everything flatten into generic “sports announcer narrates a process” regardless of what the persona field says?

Setup

The Stack

• Claude LiveCaster: the open-source slash command toolkit from the first experiment

• Cursor: wrote the simulation engine and all 7 YAML scenario configs

• Kokoro TTS: local text-to-speech, 8 voices across 7 scenarios

• Python 3.9+: data-driven simulation script scripts/simulate.py)

What Changed

The core announcer command /livecaster-announce) reads livecaster.yaml from the project root each iteration. The first experiment hardcoded everything. This time I made the config do the heavy lifting.

Each scenario YAML file defines:

• Contestants (who or what is competing, or solo subject)

• Tasks (the work items being processed, with scenario-appropriate names)

• Persona (multi-sentence prompt describing the announcer’s character and style)

• Voice (which Kokoro TTS voice to use, and at what speed)

• Error messages (scenario-specific failures like “MERGE CONFLICT: 4 files conflict with main” or “ROLLBACK CONSIDERED: anomaly detected in checkout flow”)

The simulation script simulate.py) got a --scenario flag. Pass it a YAML path and it generates log output using that scenario’s contestants, tasks, timing, and error profiles. Backward compatible. No flag means original defaults.

The Seven Scenarios

Races (multi-contestant):

Solo narrations (single process):

Dual-Voice Mode

Pipeline Wars introduced a second TTS voice. Instead of a single voice key in the YAML, it uses a voices list:

The announcer detects this list, generates labeled commentary [ANCHOR] / [CORRESPONDENT]), and speaks each section with the assigned voice. The anchor delivers measured factual updates. The correspondent reacts emotionally to failures and celebrates green builds.

Variables

• Changed: The simulation scenario YAML (persona, contestants, tasks, voice, errors)

• Controlled: Core slash commands, TTS pipeline, /loop scheduling, filesystem-backed state pattern

Results

Persona Fidelity Was the Surprise

I honestly expected the persona instructions to be a suggestion the LLM would half-follow. The opposite happened.

The rodeo announcer for Code Review Roundup actually used rodeo metaphors. “feat/auth-overhaul is trying to hang on through a nasty merge conflict” is exactly the kind of thing a rodeo commentator would say about a bronc ride. The war room coordinator for Incident Response stayed procedural and urgent without tipping into panic. “Root cause confirmed. Connection leak in the ORM layer. Hotfix is in staging now. Lets see if this holds.” That reads like a real incident channel update.

The mission control persona for Deploy Day was my favorite. Treated canary percentage increases like orbital stage separations. “Canary at 25 percent. All systems nominal. Proceeding to 50.” You could close your eyes and picture Houston.

Dual-Voice Actually Worked

I expected the labeled output to be brittle. It wasn’t. The announcer consistently generated [ANCHOR] and [CORRESPONDENT] labels, and the text extraction held across every run I tested. The anchor stayed measured. The correspondent stayed excitable. The contrast made Pipeline Wars the most entertaining scenario of the seven.

Having two distinct TTS voices alternate added something I didn’t anticipate: rhythm. The switch between bf_emma’s BBC-anchor delivery and af_bella’s animated correspondent created a cadence that single-voice scenarios don’t have.

Solo Narrations Needed Different Thinking

Races have a built-in narrative: who’s ahead, who’s falling behind, lead changes. Solo narrations have none of that. There’s one contestant progressing through tasks sequentially.

The personas handled this by shifting from competition to progression. The pair programmer in App Build Journey tracked milestones: “Auth is in place, the dashboard is looking great. Now we are wiring up Stripe.” The mission control operator in Deploy Day built tension through the canary ramp. The war room coordinator in Incident Response tracked investigation phases: triage, diagnosis, mitigation, recovery.

The leaderboard command in the YAML adapts too. Races count completed tasks per contestant. Solo narrations show the last 8 completed steps instead. Same command template, different shell pipeline.

Takeaway

The persona YAML field is the real primitive. Not the voice. Not the simulation engine. Not the cron scheduling.

In the first experiment, I found that /loop with filesystem-backed state is a reliable architecture for Claude Code automations. This experiment found something built on top of that: a short natural-language persona prompt, embedded in YAML and read fresh each /loop iteration, is enough to produce genuinely distinct narration styles from the same code path.

That’s the pattern worth paying attention to. You can ship a single announcer command and make it sound like seven completely different tools by swapping a config file. The architecture doesn’t care whether it’s narrating a CI pipeline race, a production deploy, or a P1 incident. It reads the persona, it reads the log, it speaks.

This maps to something bigger than voice announcing. Any recurring Claude Code automation that needs contextual awareness (code review bots, monitoring narrators, build commentators) can use the same trick: define the persona in config, let the slash command read it each iteration, and the LLM adapts.

What’s Next

The seven scenarios cover a lot of SDLC ground, but they’re all running on synthetic log data generated by the simulation script. Every task completes. Every error is scripted. The narration is convincing, but it’s narrating a simulation.

Things I want to test:

• Real CI pipeline narration: Point LiveCaster at an actual CircleCI pipeline log instead of synthetic data. Does the commentary hold up when the data is messy and unpredictable?

• Custom scenario authoring: How fast can someone who’s never seen LiveCaster create their own scenario YAML? The format is simple, but I haven’t tested it with anyone else.

• Longer dual-voice runs: Pipeline Wars worked in short simulations. What happens with a 30-minute dual-voice broadcast? Does the LLM maintain persona separation, or do the voices start to bleed together?

• Three-voice mode: The architecture supports N voices in the voices list. An anchor, a color commentator, and a sideline reporter? This might be absurd. I want to try it anyway.

The open-source repo is here. Clone it, run /livecaster-simulate with-announcer, and pick a scenario. The whole thing runs locally, no API keys beyond Claude Code itself.

Ship it. 🎙️

This experiment was conducted as part of the Loop Lab program at CircleCI. It is a follow-up to /loop Gave Claude a Heartbeat. I Gave It a Microphone. The code lives at CircleCI-Research/claude-livecaster. Full demo: YouTube. PR: #2.

Sixteen commits later, it works. Six AI models racing head-to-head on eval tasks, and Claude is calling the action in real time like a motorsport broadcaster with a Southern drawl and opinions about latency.

Watch the full demo on YouTube.

Hypothesis

I expected Claude Code’s /loop command to be stable enough to serve as a cron-style backbone for a recurring, stateful task. Specifically: read live log data every N minutes, generate contextual commentary based on what changed since last check, synthesize it to speech, and auto-cancel the cron when the race finishes.

My reasoning: slash commands already work for one-shot jobs. The /loop extension schedules them on an interval. If the execution stays in the main session (where permissions are already granted), it should hold together. But I’d never run anything on /loop for 30+ minutes straight, so I genuinely didn’t know.

Setup

The Stack

• Claude Code with /loop slash command support

• MindTrial: an open-source AI model eval framework I forked from a colleague, Petr Malik, on the CircleCI AI Team. Runs tasks against multiple LLM providers (OpenAI, Google, Anthropic, DeepSeek, Mistral, xAI) simultaneously and scores the results.

• Kokoro TTS: open-source text-to-speech running locally. No API calls, no latency. Sounds surprisingly good.

• Custom slash commands I built for the experiment:

  • /run-model-comparison builds MindTrial, launches the eval, initializes a commentary transcript

  • /simulate-model-comparison generates fake log output in the same format, so I could test end-to-end without burning API tokens

  • /announce-model-comparison is the single announcer iteration, designed to be called by /loop

  • /stop-model-comparison kills the eval, finalizes the transcript, shows the leaderboard

How It Works

You kick off an eval race. Six model configs start processing tasks in parallel (I patched MindTrial’s Go code so runTasks launches each config in its own goroutine instead of running sequentially within a provider). Then you start the announcer:

Every 5 minutes, Claude wakes up, tails the eval log, parses the current leaderboard, figures out what changed since last time (lead changes, new completions, errors, scoring gaps), writes Ken Squier-style race commentary, speaks it out loud via Kokoro, and appends to a running transcript.

When the race finishes, the announcer detects it from the log, delivers a final wrap-up with standings and notable moments, and auto-cancels its own cron job via CronList + CronDelete. Zero human intervention after you start it.

The Variable

I started by trying to run the announcer as a background Task agent. That’s the variable I was really testing at first. But Task agents lose Bash permissions mid-run. I discovered this the hard way when the announcer went silent 12 minutes in. So I pivoted to /loop, which runs in the main Claude Code session where permissions persist.

That pivot turned out to be the more interesting architecture anyway.

Results

The Background Task Discovery

The first architecture I tried was launching the announcer as a background Task agent. Makes sense in theory: background agent watches the race, main session stays free. In practice, Task agents lose Bash permissions after about 10-15 minutes. The agent can still think, but it can’t execute shell commands, which means it can’t tail the log, can’t run TTS, can’t do anything useful.

This isn’t documented anywhere I could find. I discovered it when the commentary just... stopped. Checked the Task agent output and found it apologizing for being unable to execute commands.

The /loop Fix

/loop runs in the main session. Permissions persist because they were already granted when you started the session. Each iteration gets a fresh context call, but the slash command reads from the same log file and transcript, so state is maintained through the filesystem rather than through agent memory.

This is a better pattern. The announcer doesn’t need to hold state in memory between calls. It reads the log, diffs against the transcript, generates commentary, writes it back. Stateless iterations with filesystem-backed state. Feels like a cron job because it is one.

TTS Quirks

One thing I didn’t anticipate: model version numbers. Kokoro TTS reads “4.6” as “four six” instead of “four point six.” I had to add a decimal-to-word conversion step in the announcer pipeline. Small thing, but it matters when you’re narrating “Claude 3 point 5 Sonnet pulls ahead of GPT 4 point 1” every few minutes.

Also had to tune the dynamic tail sizing. The announcer reads N lines from the end of the log each iteration. If your /loop interval is 1 minute, you need fewer lines than if it’s 10 minutes. I settled on a LOG_LINES_PER_MINUTE=50 constant, clamped between 60 and 500 lines, scaled by the interval. Gets the right amount of context without drowning in log noise.

The Session

16 commits. 1,056 lines added. Started at 1:17 AM, merged at 3:35 PM. The commit history tells the story: initial build, rename and refactor, simulation mode for testing, UX improvements, the Task-agent-to-loop pivot, opt-in announcer with interactive prompts, interval selection, and docs.

Built the whole thing with Claude Code writing the code while I directed architecture decisions. Ate its own dog food: Claude Code building a Claude Code automation.

Takeaway

/loop is the more interesting primitive here, not the voice announcer itself. The announcer is fun (genuinely fun, the Ken Squier voice calling lead changes between Gemini and Claude is something I didn’t know I needed). But the pattern underneath is the finding.

Stateless cron iterations with filesystem-backed state is a reliable architecture for Claude Code automations. You don’t need the agent to remember anything between calls. Write state to disk. Read it back next iteration. Let /loop handle the scheduling. Let the slash command handle the logic. The agent stays in the main session where permissions work, and each iteration is a clean execution.

This pattern applies to way more than race announcing. Monitoring dashboards. Periodic code review sweeps. Build status narration. Anything you’d write a cron job for, you can now write as a slash command and schedule with /loop.

The constraint that makes it work is the same constraint that makes cron jobs work: each invocation is independent. Don’t rely on agent memory. Rely on the filesystem.

What’s Next

I extracted the announcer into a standalone open-source project: claude-livecaster. It’s the slash commands, the TTS pipeline, and the simulation harness, decoupled from MindTrial so anyone can wire it into their own long-running processes.

Things I want to test next:

• /loop at scale: What happens when you run it for 4+ hours? Or days? Does it stay stable? Are there memory or context limits I haven’t hit yet?

• Multiple /loop jobs simultaneously: Can you run two different slash commands on different intervals in the same session?

• CI pipeline narration: Same architecture, but the announcer watches a CircleCI pipeline instead of an eval race. Every few minutes, Claude narrates what’s happening in your build. I have a feeling this is a better status dashboard than any dashboard.

• Other TTS voices: Kokoro is good but limited. Want to try ElevenLabs for higher fidelity and see if the latency trade-off (API call vs. local) matters at 5-minute intervals.

If Background Task agents ever get persistent Bash permissions, I’d revisit that architecture too. Running the announcer in the main session works, but it does block you from using that session for other things while the race is running.

Sixteen commits before lunch. Ship it. 🤖🎙️ 🚀

This experiment was conducted as part of the Loop Lab program at CircleCI. The original implementation lives in MindTrial PR #2. The extracted open-source project is claude-livecaster. Full demo: YouTube.

There’s a lot of noise right now about AI agents writing code. Faster commits. More output. Ship it yesterday.

But here’s a question we haven’t seen anyone answer with data: When the agent says “all tests pass,” is the code actually good?

We ran the experiment. Here’s what we found.

The Setup

We gave an AI agent the same task 10 times: build a Snake game from scratch using TDD. Same spec. Same model. Same prompt scaffolding. 20x20 grid, arrow key controls, collision detection, score tracking, retro aesthetic. The whole thing.

The only variable: 5 runs had CI pipeline integration. 5 didn’t.

Both groups followed test-driven development. Both wrote comprehensive test suites. Both committed and pushed code. The CI-enabled group got one additional thing: real-time feedback from a CircleCI pipeline after every push. In our experiment, we created a RalphCI loop (i.e. a CI-enabled Ralph loop) that pre-fetches pipeline status and injects it into the agent’s prompt context, so the agent can see and act on CI results as part of its development cycle.

That’s it. One variable.

What Happened

All 10 runs finished. All 10 completed every task. All 10 produced a playable Snake game with full test coverage. All 10 passed local tests.

Every single agent said “passed.”

Then we checked the CI pipeline.

80% of the runs without CI integration shipped code that fails the pipeline. The agent ran the tests on its machine, saw green, and declared victory. The pipeline disagreed.

What Went Wrong (and Why It’s Instructive)

The failure was an ESLint configuration mismatch. The agent defaulted to single quotes. The project’s linter required double quotes. Local tests don’t check lint rules. The CI pipeline does.

This isn’t a dramatic failure. It’s a mundane one. And that’s the point.

AI agents don’t inherently know your project’s lint config, your environment variables, your CI-specific test runners, or your integration-level constraints. Local tests verify logic. CI verifies deployability. These are different things, and the gap between them is where “works on my machine” lives.

The one non-CI run that passed? The agent happened to discover the project’s lint:fix command on its own. Lucky, not reliable. Not a strategy.

What the CI-Enabled Agent Actually Did

When CI failed, the agent didn’t just see a red X. The RalphCI loop pre-fetches the pipeline status and injects it into the agent’s prompt context at the start of its next iteration. The agent sees something like:

CI Status

• Pipeline #551 FAILED with linting errors

• Issue: game.test.ts using single quotes instead of double quotes

Here’s what happened next, across all 5 CI-enabled runs:

• 12 CI failures appeared in total

• The agent fixed all 12 autonomously

• Zero human intervention required

In every case, the agent dropped its planned task, diagnosed the root cause from the failure log, applied the fix, re-verified locally, pushed, and resumed its work once the pipeline came back green.

It refused to mark the PR as “Ready for review” until CI was green.

Or as Dan Lorenc put it: “Code is cheap. Green CI is priceless.”

The Behavioral Shift

The most interesting thing in this experiment isn’t the pass rate. It’s how the agent’s behavior changes when it has access to pipeline feedback.

Without CI, the agent’s definition of “good” is: local tests pass. It has no reason to think otherwise. It runs the suite, sees green, declares victory, and moves on. It’s not being careless. It literally has no other signal available.

With CI, the agent operates differently. It treats the pipeline as the source of truth. I watched it:

• Query pipeline status before starting new work

• Drop planned tasks to prioritize CI fixes

• Parse failure logs to diagnose root causes

• Re-verify both locally and through the pipeline before continuing

• Wait for pipeline completion before declaring a PR ready

This is closer to how a senior engineer works than a junior one. Not because the model is smarter. It’s the same model in both groups. The feedback loop gives it the information it needs to operate at a higher level.

An Unexpected Finding: CI Makes Agents Write Better Tests

CI-enabled runs consistently produced more tests: 28-37 per run versus 21-28 for non-CI runs.

We didn’t instruct the agent to write more tests in CI mode. The prompt was identical. But when the agent operates within a feedback loop that includes external verification, it appears to write more thorough test suites on its own.

We don’t want to overclaim causation from 10 runs, but the pattern was consistent across all 5 CI-enabled experiments. If I had to guess: agents test more thoroughly when they know their work will be checked by something other than themselves. Which sounds a lot like what humans do.

The Velocity Problem Nobody’s Talking About

Everyone’s focused on making AI agents faster at writing code. But code generation was never the bottleneck, and it definitely isn’t now. The bottleneck is knowing whether the code you generated is actually safe to ship. As agents accelerate how fast code gets written, the gap between “code produced” and “code that’s actually deployable” doesn’t shrink. It widens. More code, generated faster, with the same (or less) visibility into whether it integrates, passes lint, respects environment config, and behaves correctly outside the agent’s local context.

That gap is where our 80% failure rate lives.

CI is what closes it. Not by slowing agents down, but by giving them the signal they need to self-correct. The CI-enabled runs in our experiment took more iterations and more time. They also produced 100% deployable code. The non-CI runs were “faster” and 80% broken.

Speed without confidence isn’t velocity. It’s drift.

The teams that will move fastest in the agent era won’t be the ones generating the most code. They’ll be the ones with the tightest feedback loops between code generation and verification, the ones who can point an agent at a problem and trust the pipeline to keep it honest.

What This Means If You’re Building with AI Agents

This experiment was small and controlled. A Snake game, not a production codebase. But the finding generalizes to a principle:

The feedback loop defines the output quality.

Same agent. Same task. Same model. Same prompt. The only difference was whether the agent could see what the pipeline saw. That single variable moved the CI pass rate from 20% to 100%.

If you’re running AI agents against real codebases without CI in the loop, you’re likely in the same position as our non-CI runs: the agent is confident, the code passes local tests, and you have no signal on whether it actually integrates.

Three things we’d suggest based on what we found:

Close the feedback loop. If your agent can’t see CI results, it can’t fix CI failures. This sounds obvious, but most agent setups today push code and move on. The agent needs to see the pipeline status and have the opportunity to act on it.

Treat systematic failures as tuning opportunities. The ESLint quote mismatch appeared in every single run. That’s not randomness, it’s a trainable pattern. Injecting your project’s lint rules into the agent’s prompt context, or adding a pre-commit hook, could eliminate the most common failure before it ever hits CI.

Invest in your pipeline, not just your prompts. The agent’s ceiling is defined by the infrastructure around it. A better prompt might make the agent write slightly better code. A CI feedback loop changes what “done” means.

What We’re Doing Next

This was our first controlled study with proper sample size and identical configurations across groups. We’re planning to run the same experiment design against production-scale codebases with more complex CI pipelines, compare across different models, and explore whether smarter prompt context (feeding lint rules and failure patterns directly to the agent) can reduce CI failures before they happen.

We’re also interested in what this enables when you run agents in parallel: if each CI-verified run produces a genuinely deployable artifact, you can spin up 10 variants of a feature, all tested, all passing CI, and pick the best one. That changes how you think about prototyping.

All 10 runs finished before lunch. Ship it. With receipts. 🚀

This experiment was conducted by the CircleCI AI Testing Lab using RalphCI, a soon-to-be open-source tool for integrating AI coding agents with CI pipelines. The full dataset (10 runs, per-iteration metrics, activity logs) will be published alongside the repo. Stay tuned.

We’re sharing this because we think developers deserve data, not just opinions, on how to work effectively with AI agents. If you’ve run similar experiments, we’d love to compare notes.