Imagine a student who must score 100% on a final exam to pass a course. Not an A-. That would fail them. They can retake the exam as many times as they want, and every attempt comes back with painfully detailed feedback.
Then suppose that same student could peek at the test in advance and take notes.
One catch. The sneak peek does not include any of the last-minute trick questions that might show up on the real exam without warning. Those get written by “the world” the moment the exam starts.
Nobody on the teaching staff knows the trick questions in advance, and new ones can show up on a retry.
Pencils down.
That is coding agents and CI. Merge-ready green is the only passing grade. Outer-loop CI is the real final exam. A Chunk sidecar is the timed practice run. Every red→green fix loop (diagnose, patch, push again) charges a little tuition: LLM dollars, CI credits, wall-clock.
I wanted to know what a green PR actually costs on a fully-automated AI coding agent run. Time. Tokens. Credits. The whole bill.
We moved the needle a bit on time and tokens. Fine. Not the story.
The story is rework going to zero on the stuff we can predict. Deliver the PR in one take. Ace it at 100%. The only red you should tolerate is the unknowable.
Hypothesis
If you show the agent what CI will check before it writes product code, give it a page of notes, and make it pass a real practice run on a sidecar, preventable outer-loop failures should stop showing up. Fix loops should not be necessary. Green commits should hit 100%.
Not “prompt harder.” Setup for success. Ace it at 100%. One take.
Setup
In this lab, an A- is still red. Local lint that smiled while outer-loop CI frowned still fails. A green sidecar run that dies on the real pipeline still fails.
Two cadences. Same bar: every commit you push to outer-loop CI is green.
Per-task-push: after each task clears practice run, push. Seven tasks, seven trips. Ace every one.
Single-push: finish the stack, clear practice run, push once at the end. Ace that one.
How often you push is a later lever. First you have to be equipped to push green.
Without the sneak peek
Three calibrations on the same AFK classic Snake game build. Seven tasks. Hands off the keyboard.
No inventory of checks up front. No notes. Inner-loop checks existed, but they were split between the localhost and a sidecar.
They finished, but rework kept piling up. When we feed agents endless retries and mountains of feedback, we’re just repeating the same habits we once drilled into human developers, now adopted by their AI counterparts.
*NOTE: We can make retries cheaper with a structured CI pipeline --failure-report, but that is its own write-up. Spoiler: even if you cheapen input tokens on each retry, you still have to pay time and money to retry.
Then we showed it the test
Before task one, before a line of game code, the agent got a short inventory: which checks fire on the practice run, which jobs fire on the thick outer pipeline, and pointers to the scripts and configs that define them. Descriptive. Not an answer key.
It took notes (preflight.md). It could @-read those configs when it needed more.
In our codebase that inventory is a “CI validation manifest”, which basically tells the agent what will be graded, and where the grading logic lives, before it starts guessing from vibes.
Inner-loop checks moved fully onto the sidecar, and we stopped running the same exact lint and test loops on the localhost. Outer-loop CI stayed the honest final exam, including whatever the world still wanted to throw at it.
Results
Headline: 0 CI fix loops on both sneak-peek arms.
Per-task-push: 7/7 pushed commits green.
Single-push: the one end-of-run push went green on first contact.
Zero outer-loop CI failures on either arm.
Diagnosis tokens on the rework line: $0. That budget can fund first-pass product code generation instead of burning LLM dollars and CI credits on red -> green archaeology.
Wall-clock did not collapse. Still about 50 to 60 minutes. Coding time dominated. LLM dollars landed about $15 to $17, in line with earlier attempts. The key difference: rework nearly disappeared.
Quiet rework is the same muscle Merge Efficiency Ratio names in CircleCI’s State of Software Delivery Q2 Pulse report: how many validation cycles before a change is actually done. This lab is that problem inside one agent setup. Once green is likely, push cadence and where you place the checks are how you bake low MER into the loop.
TL;DR
A coding agent can only prevent what we can predict. Say that out loud.
Linters. Unit tests. Typechecks. Formatters. Lockfile rules. Contract tests you already wrote. The deterministic stuff with a known answer. If it is on the syllabus, the agent can study it, practice it, and clear it before you spend an outer-loop credit.
It cannot prevent what the world invents after the peek is printed.
Late-breaking CVEs mid-run. A registry serving a different tarball than the one you resolved an hour ago. A base image that rotated overnight. An org policy that fails a dependency you did not touch. A flaky third-party API. A secrets scanner lighting up on a fixture. A new advisory. A mirror outage. A quota. Runner drift. “Works on my machine” that is really the cloud’s clock, certs, or DNS.
Platform engineers, harness authors, and coding agents cannot predict this. It arrives when the CI pipeline is already running.
Design for what the agent can own. Leave outer-loop CI in place for the rest. Ideally one take. At most two, once the world gets a vote.
Build the agent setup so it can ship the application end-to-end with zero CI failures from known deterministic sources.
Inventory of checks. Notes. A practice surface that is real (sidecar, not theater). Then push.
Do not “prompt harder” and hope the fix loop works overtime.
One take. Let’s go.
Retrying a flaky build until it passes turns CI into a probabilistic lottery that actively masks race conditions and concurrency deadlocks. The real expense is not compute minutes, but engineering hours lost chasing phantom bugs in production that CI already caught and hid.