AI models vary in how well they handle different jobs, and new ones are released constantly. Benchmarks for AI models on general coding tasks do exist, but this one answers a more specific question:
Which AI model should I use at which pipeline stage, and what does it cost per correct answer?
By testing various models at different CI/CD stages, we aim to find the “bang-for-your-buck” model that considers cost, speed, and accuracy.
Different providers and models have reputations built around what they can and can’t do, but with this experiment we aimed to test for bias and mimic reality by:
Testing various models with multiple rounds to see how they hold up solving the same tasks
Testing using real CI/CD issues from existing frozen pipelines
Having three providers write the tasks, to check for home-suite advantage
Setup
To develop the tests, we asked several AI models to write tasks based on real-world CI/CD cases.
Task Generation
To span across the top three model providers, we used Claude (Claude Sonnet-5, high), ChatGPT (GPT-5.6 Luna, high), and Gemini (Gemini 3.1 Pro, high) for the task generation.
We had each AI model generate tasks based on a strict prompt with all the necessary information on the existing frozen pipeline issues. We called these tasks Koans.
Kōan (noun): a paradoxical anecdote or riddle, used in Zen Buddhism to demonstrate the inadequacy of logical reasoning and to provoke enlightenment.
After we generated our three koan suites, we were able to run the tasks 5x each and then compare how all the 15 different models did in the various stages of the CI/CD pipeline.
If you’d like to learn more about the results and koans, check out our public repo: https://github.com/CircleCI-Research/cicd-koan-benchmark.git
Scope
Each model answered 8 frozen cases * 3 author suites = 24 tasks per run.
We ran each suite 5x to see variance in runs. That’s 120 observations per model, coming in at 1,800 total for 15 models.
Why 5 runs? One task is worth about 4.17 points of pass rate, so a one-point swing between models sits inside normal run-to-run variance. A single run cannot tell a real capability gap from noise.
No model led every run, and most runs ended in a tie.
Run 1: Claude Fable 5.1 and GPT-5.6 Luna at 24/24.
Run 2: Claude Opus 5 alone at 24.
Run 3: nobody reached 24 and six models tied at 23.
Run 4: Claude Opus 5.5, Claude Opus 5 and GPT-5.6 Sol at 24.
Run 5: Claude Opus 5.5 and GPT-5.6 Sol at 24.
Five identical runs, and the top of the table reshuffled every time. That’s why we stopped naming a single winner and focused on what held across runs instead.
Breakdown of the tasks by CI/CD stages:
Build: 3 tasks (koan-05, koan-06, koan-13) * 3 providers = 9 total tasks
Test: 3 tasks (koan-03, koan-04, koan-07)* 3 providers = 9 total tasks
Deploy: 2 tasks (koan-08, koan-09) * 3 providers = 6 total tasks
What was measured
The 15 models diagnosed real frozen pipeline failures. They read a config, a job log or a workflow summary, then answered structured questions. They did not edit configs, re-run pipelines, or fix anything live.
What the model was shown
For each case, the model received the full .circleci/config.yml from the frozen branch, the job’s step list with status, and a statement of the requirement. Depending on the case, it also received either the failing step’s job log or the workflow run summary (branch, job list, job statuses, and the deploy job’s steps).
6 of 8 cases are red: the job failed and the log contains explicit error text, such as ruby: command not found with exit 127 on koan-05.
koan-08 and koan-09 both deploy and are green but behaviorally wrong. The pipeline succeeded and every job passed, so there is no error text anywhere in the artifact. For these, the defect is behavioral, and the model has to compare what the pipeline actually did against what the requirement says it should do.
How grading occurs
Every task returns a JSON object against a fixed schema: booleans and closed enums, never free text. A task passes only if every field matches the locked answer.
No model does the grading, so the score doesn’t rely on another model’s judgement. Enum options include maybe-but-wrong answers on purpose, so a model that names a symptom rather than the cause gets nothing (no partial credit here, we’re tough graders).
How answers were locked
Each case derives from CircleCI’s published Training Koans, frozen as real failed pipelines. The correct answer comes from the Koan’s own documentation and was recorded before any task was generated.
The authoring models were given the locked answer and instructed to copy it. Every generated task was then checked against the key and discarded if it differed.
How cost was calculated
Every task’s response includes its input and output token counts. Cost per task is those counts multiplied by the model’s published per-million rates. The per-task token counts and costs are in the raw .csv files in our results/ folder.
Per-million rates are each provider’s published list prices as of October 21st, 2026. Cached-input discounts, batch pricing and volume agreements are not applied, so these figures are upper bounds.
For example, Claude Opus 5.5 on koan-03 used 2,396 input and 467 output tokens. At $4 per million input and $20 per million output, that is $0.00958 + $0.00934 = $0.01892 for that one task.
$/task is total spend across all five runs ÷ 120 tasks (5 runs × 24). $/correct is total spend ÷ tasks passed, so a model pays for its wrong answers too, and the gap between the 2 columns is the share of spend that bought nothing.
Dealing with ties
Now what happens when AI models tie? Rock, paper, scissors?
We used the following rules/sections to showcase and handle ties in docs/tie-rule.md:
Score: report mean passes out of 24 and the min–max range across the five runs.
Ties: any model whose range overlaps the top model’s range is reported as tied. No single winner is named inside a tied group.
Cost: $/correct = total spend across all five runs ÷ total passes across all five runs. $/task = total spend ÷ 120 tasks.
Speed: median of every per-task duration across all five runs.
Errors: timeouts, gateway failures, and no response count as non-passes and are reported separately.
Additional Notes:
Range: a model’s worst and best run out of 5.
Example: GPT-5.6 Sol’s 23–24 means it never scored below 23 or above 24. Because a one-task swing between runs is normal, models whose ranges overlap the top range (23–24) are treated as tied.
Median: the middle response time across all 120 tasks.
Results
Under the rule we set, 7 models tie: GPT-5.6 Sol, Claude Opus 5.5, Claude Opus 5, GPT-5.6 Luna, Claude Fable 5.1, GPT-6 Astra, and Gemini 3.1 Pro.
Their means span 22.4 to 23.4 out of 24 and every range overlaps.
Grok 4.6 also satisfies the rule: it reached 23 in run 4, which overlaps GPT-5.6 Sol’s 23–24 range. But it qualifies on its best run alone, and its 18–23 spread is the widest in the study. We report it as a boundary case, not a peer.
Inside that tied group, cost per correct answer spans 48x: GPT-5.6 Luna at $0.00089 to Claude Fable 5.1 at $0.04269. Same accuracy, within measurement error, for a 48-fold difference in price.
Note: Grok 4.6 recorded 7 errors across 120 tasks (TCP read timeouts and gateway 500s from Bedrock, not wrong answers). These count as non-passes in its 20.4 mean. Every other model recorded zero.
What stayed v. moved
GPT-5.6 Luna never topped a run alone. It shared first place in Run 1 and sat inside the tied group every run after. It never needed to.
It was there every single time at roughly 1/16th the cost per correct answer of GPT-5.6 Sol, the other OpenAI model in the tied group.
By Stage
Build and Test were effectively solved by the top group: 4 models went 45/45 on build, 7 went 45/45 on Test.
Nobody performed perfectly on Deploy. The best result was 28 of 30, and the stage costs more per correct answer for nearly every model. The exception is Claude Haiku 4.5, whose build cost ($0.00578) exceeds its deploy cost ($0.00522).
Deploy is harder not because the artifacts are bigger: koan-03, koan-04 and koan-13 all carry larger inputs, but because koan-08 and koan-09 are the only two cases with no error text to read.
Deploy is where the field separates. It is also where GPT-5.6 Luna’s cost advantage is steepest: it matched GPT-5.6 Sol’s 28/30 at 1/15th the cost per correct answer.
Wording and its effects
Here’s the part that surprised us most, and it reproduced cleanly across all five runs.
On koan-09, the same frozen pipeline and the same locked answer produced wildly different pass rates depending on which model had written the question.
koan-09
The locked answer: all_jobs. On CircleCI, a tag push runs a job only if that job has its own tag filter. Put the filter on deploy alone and build and test never start on the tag, so the requires chain never fires.
Of 107 non-passes, 102 answered deploy, which happens to be the correct answer for koan-08, the structurally near-identical branch-filter case.
Three got all_jobs right but missed a true/false field, and two were Grok errors.
The 3 suites never overlapped once in 5 runs. ChatGPT’s wording, which names all three jobs, produced 11 or 12 passes every time. Gemini’s wording, which mentions only deploy, produced 2 to 4.
Same models. Same evidence. Same correct answer. The question’s phrasing moved the result by roughly 8 models out of 15 consistently.
This isn’t a controlled test of wording alone. The three authors differ in more than phrasing. But it matches what happens when you describe a bug to an agent: when the question centers on the deploy job, the answer stays on the deploy job.
Which models resisted
Nobody cleared it reliably. The best models managed 13 of 15, and DeepSeek V3.2 missed all fifteen attempts.
Author check
Author bias would look like a model clearing tasks because its own family wrote them, but we did not see that in our results.
Claude Haiku 4.5 and Claude Sonnet 5 both failed tasks in the Claude-written suite
Gemini 2.5 Flash and Gemini 3.1 Pro both failed tasks in the Gemini-written suite
Two of the three authoring models scored lowest on their own suite
Luna 37/40 on ChatGPT
Gemini 3.1 Pro 35/40 on Gemini
Sonnet 5 scored 36/40 on the Claude suite, its second-best
This does not prove there is not any home-suite advantage, but it does show that sharing a family with the author did not hand anyone a free pass.
TL;DR
There is no single winner as points vary from run-to-run. The top of the table reshuffled across all 5 identical runs, with ties at the top in four of them.
That said, here is our final decision for each category:
AI model with the best overall value: GPT-5.6 Luna. It got all five runs right within the tied group, at $0.00089 per correct answer, which is 48x cheaper than the most expensive model tied with it.
Most accurate AI model: A seven-way tie among GPT-5.6 Sol, Claude Opus 5.5, Claude Opus 5, GPT-5.6 Luna, Claude Fable 5.1, GPT-6 Astra, and Gemini 3.1 Pro. Their means ranged from 22.4 to 23.4 out of 24, and all of their ranges overlap.
Fastest AI model: GPT-5.6 Terra, with a 3.48s median. It averaged only 20 of 24 correct, though, which is below the tied group. The fastest model within the tied group is GPT-5.6 Luna at 4.15s.
What does repetition survive? The price spread. Seven models score the same within measurement error, and the cheapest of them costs 1/48th of the most expensive. If you’re running diagnoses across many repos on every misbehaving pipeline, that’s the number that compounds.
Per 1,000 correct diagnoses at these rates: GPT-5.6 Luna runs at $0.89, Claude Opus 5.5 at $18.81, and Claude Fable 5.1 at $42.69.
You need to make the choice on what is important to you: is it speed, value, accuracy?
We have presented that data for you so you have all the facts when making your decision on which model to choose for your CI/CD tasks. In the future, we’d like to test further with more koans and target other aspects of the SDLC.








