Your pipeline goes red. You hand it to an agent. It reads the logs, writes you a confident paragraph about what broke, edits the config, triggers a rerun. You pay for every token of that loop whether CI comes back green or not.
That loop runs on my machine most days. I wanted a cheaper version of it, and I had a specific idea about where the savings were hiding.
So I broke six CircleCI pipelines on purpose. Then I saved one specific failed run of each, so every model would stare at the exact same wreck. Three AI coding setups, one instruction: fix this.
Grok 4.6 fixed all six. Twenty-two cents a fix.
Claude Sonnet 5, turned all the way up, fixed three. Every fix it did land cost about $1.11.
I walked in looking for the opposite result. The chatter said Grok would be the one to stumble on the terminal work.
What I was actually hunting
A routing rule, which is a boring thing with a boring name: a policy for which model gets which wreck. Teams already do this with humans. The YAML typo goes to whoever’s on rotation. The bash failure that’s been red since Tuesday goes to the one person who can actually read a wait loop. I wanted that same escalation in software, with a price tag on it. Hard shell problems to the expensive model, broken config and missing test results to the cheap one. Same quality of fix, smaller bill at the end of the month.
The stake is your inner loop. Every time an agent picks up a red pipeline, reads the logs, edits the config, triggers a run, and reads that too, somebody is getting billed. Running all of it on the expensive model, all week, is a line item.
That split never showed up.
What did show up is a way to measure this stuff without fooling myself. I connected CircleCI’s command line tools to both Claude Code and Cursor. I saved a red run, so nobody could grade a different pipeline than the one I picked. Then I refused to grade the model’s essay for a proposed fix. By “essay” I mean the model’s written diagnosis of what broke and what it would change, which is a completely different thing from whether CircleCI actually went green.
Agents are already good at the essay. The actual pipeline rerun is the score.
The essay is the menu. Green CI is the meal. I came to eat.
Hypothesis
I expected Grok 4.6 to struggle on the bash-flavored failures and hold its own everywhere else.
That was the prediction. I went in with a bias: if any model was going to struggle on shell, it would be Grok 4.6 once it had to drive a terminal. Type a command, read what came back, adjust, type the next one. Shell breakage is exactly that, over and over, so that’s where I thought a gap would show.
If the reputation held, the plan was already written. Sonnet 5 at effort: low takes the bash cases, Grok takes the rest, and I pay less without shipping worse fixes.
Second question, cheaper to ask and just as useful: does paying for Sonnet 5 at effort: high buy enough extra passes to justify the receipt?
Setup: six puzzles, three setups, one judge
The puzzles came from the CircleCI Training Koans, a public set of short exercises that break a pipeline on purpose so you can practice un-breaking it. I used six on a throwaway project, two per flavor of broken.
Three flavors went in. Broken config, where the pipeline file itself is wrong. Tests that don’t report, where the tests genuinely run and pass but CircleCI never receives the results, so the job looks fine and tells you nothing. And shell problems: bash logic, wait loops, cache commands, container images, services.
Then I froze them: one saved failed run per puzzle, so “whatever failed most recently” couldn’t sneak in and change the question halfway through.
Three setups, same prompt every time.
Those dollar figures are sticker rates for tokens, not the cost of a run. What each trial actually spent shows up in Results.
Changed: the setup. Held constant: the saved failed run, the prompt, the isolation, the spend ceiling.
Every trial started in a fresh clone. No resumed sessions. Sonnet never got to peek at what Grok had already worked out, and the two Sonnet effort settings didn’t share a session either. Each agent had to check its config change, then trigger a brand new CircleCI run on its own throwaway branch. Never main. Never the saved branch.
Now the grade, which is the part that maps to your actual day. Nobody merges the model’s write-up. You merge green CI. So a trial passes only if the pipeline that agent triggered finished green. On the tests-that-don’t-report puzzles, a green job with an empty test results panel (CircleCI’s Test Summary) is still a fail, because tests that pass without reporting anything back aren’t a fix. Afterward I re-checked every run against CircleCI’s own record, so the numbers below come from CircleCI and not from an agent’s summary of its own brilliance.
I capped spending too. I watched the expensive setup burn money during calibration, took its 95th percentile, and set the ceiling at $1.25 so a trial could go long without going stupid. Claude Code can hard-stop at that number. Cursor writes it down and keeps going.
Claude dollars are whatever Claude Code billed for the whole session, including the Haiku 4.5 helper that sometimes rides along. Grok dollars are tokens in and out times the published rate, not an invoice from Cursor.
One honest wrinkle: model and host are welded together here. Sonnet only runs on Claude Code, Grok 4.6 only on Cursor CLI. I can’t pull those apart, and I’d rather say it out loud than bury it in a footnote.
Eighteen isolated trials. Then a repeat round on the same six puzzles, then one genuinely messy red Playwright job. Three separate scoreboards, and averaging them would only make the numbers look tidier than the evidence is.
Results
The pilot, scored the day it ran.
All three greened both shell puzzles. The terminal gap I came hunting for never showed.
One puzzle was a config file so mangled that CircleCI rejected it before any job started. That’s why the original run was red: nothing ran at all. Fixing it meant repairing the file, triggering a new pipeline, and getting that new pipeline green. Only Grok’s new run went green. Sonnet at effort: low spent the entire $1.25 cap on that one puzzle and got cut off. Sonnet at effort: high failed it too.
Separately, Sonnet at high effort missed both puzzles where the tests ran but CircleCI never received the results. Zero for two.
I ran the same six puzzles a second time. Fresh start, no memory of the first try. Grok fixed all six again. Add both tries together and Grok is 12 for 12, Sonnet at low effort is 9 for 12, Sonnet at high effort is 7 for 12. Same homework twice. I wanted to know if Grok just got lucky on Monday. It didn’t.
Then the follow-up. I cloned a Playwright job that had been going red over and over, again a throwaway, not production. The bug was a locator: getByText matched both a heading and a button, so the test found two things where it wanted one and refused to guess.
All three models got that one pipeline green. One try each. Nobody deleted tests. Nobody touched .circleci/config.yml. Every one of them edited the Playwright test and switched to getByRole(”heading”, ...). Inside each green job, CircleCI’s test results panel listed six Playwright tests, and all six passed.
Grok was the cheapest and the slowest. High effort and low effort wrote the same fix. One case, three trials, which makes it an anecdote, and I’m labeling it one.
TL;DR
I didn’t get a routing policy. Two puzzles per flavor of broken is nowhere near enough to write a company rule about where your model budget goes. Grok held the shell cases, too. And paying for effort: high didn’t buy extra passes. On these puzzles it bought fewer.
What I did get is a test that refuses to lie to me. Most evals grade what the model wrote. This one waits for CircleCI to finish, then takes the verdict from the run itself. Execution-graded, meaning CircleCI’s pass or fail is the grade, not a rubric and not the model’s own confidence.
If you’re wiring agents into CI, that distinction is the entire ballgame.
Cheap can win when execution is the judge. Twenty-two cents a successful fix against $1.11 isn’t a brand story. It’s a 5x token bill, on this lab set, every time an agent tries to get a red pipeline green. I’m not going to invent your volume. A team of ten, five of those loops a week, is about $45 extra to pay for high effort instead of Grok. That’s the unit. Not a finance-team emergency. Still a 5x.
I evaluate models for a living. Then I have to evaluate the eval.
Scoring a write-up is asking somebody how they feel. Scoring a CI pipeline rerun is taking their actual blood pressure.
RalphCI already showed that green on your laptop isn’t green in CI. This one asks the ruder follow-up: hand an agent the logs, the CircleCI tools, and one saved red run, and can it actually fix the pipeline? On this lab set, yes. Often. The expensive knob wasn’t the reason.
I’m not saying always use Grok instead of Sonnet. Hosts behave differently, Cursor can’t enforce a dollar cap, and these were clean training puzzles plus one Playwright locator.
I’m saying this: if your agent loop can’t trigger a rerun and read the test results panel, you’re grading the menu instead of the meal.