When I joined CircleCI in December of 2025, Chunk was already in GA. At the time, we were testing its ability to offer PRs to fix broken builds. But we had 2 big problems: people weren’t using it, and the people who were, weren’t really merging the fixes that Chunk provided for their broken pipelines.
We wanted to understand: How effective was Chunk? Was there something we could do to improve the fixes given to the users? And perhaps most importantly, was there a way to improve the accessibility of the failure data to agents?
We tested the effectiveness in a controlled eval benchmark to measure the actual impact. The same dataset, the same model (claude-sonnet-5), against three different setups: no CLI, CLI with free use, and CLI with --failure-report. Here is what we found.
The context
Our first experiments were designed to mimic how Chunk works: giving it a file, then a generic prompt written by the team (for example “you are a coding agent, you need to solve this error” etc). Essentially at the beginning, we were trying to change the prompt to see if the fixes improved, and if giving certain instructions, steps, or validations would move the needle. The reality? The prompt didn’t seem to matter much.
Our team was constantly reading blog posts from other AI or CI/CD companies about what they were doing to solve this problem, and eventually we came across a paper by Bytedance. The paper shared that when cleaning log output, most of it is useless so the best thing to do is focus on the part of the log that contains the error.
We began to wonder: How do we remove the unnecessary characters and do a diff between the good and bad logs? That was the first successful experiment that gave us good results. It followed the way Chunk works. Because of that, we released the API of step output condensed. You fetch the output of a failed step and process to clean it, so you give better context to the agent.
From there, our focus eventually moved away from Chunk and toward the inner loop. We redesigned the experiment to mimic the usual developer flow.
“I thought: ‘If I was a dev working on a project and my pipeline failed, how would I fix it with an agent? Probably by copying the pipeline URL and giving it to an agent to fix. I saw that it had to follow a lot of steps to get to the pipeline output. The agent has context to the pull request, but it doesn’t necessarily have sufficient context.’”
Joaquin Sandoval Miramontes, Senior Software Engineer, CircleCI
Hypothesis
The hypothesis we arrived at was this: CI failure data is structured for humans reading a dashboard: status icons, log streams, nested UI. An agent working from that raw output burns context on navigation before it can start fixing. Therefore, restructuring the same data for agent consumption should reduce diagnostic turns and get the agent to fix faster.
Setup
I always try to do experiments in a structured fashion. I want to make sure that I have a way to repeat my own results over and over, and I’m particular about designing experiments with a hypothesis, expected results, and conclusions – I try to follow the scientific process wherever possible.
So here’s what we tried.
New flag: circleci run get [run_id] –failure-report
Output format: condensed, organized [workflow → job → step], showing only what failed. Built for piping directly into an agent’s context window.
circleci run get [run_id] --failure-report | claude "fix the failing workflow"We compared three conditions on the same real failed pipelines from production:
An agent given authenticated access to the CircleCI API
An agent given free use of the CircleCI CLI
An agent given access to the CircleCI CLI instructed to use --failure-report command
Results
Efficiency
The flag delivers real efficiency gains:
Agents using --failure-report spent fewer turns overall and finished faster. The CLI-only variant used the most turns and tokens due to agents navigating the CLI tool chain to resolve which job failed.
Fix Quality
Fix quality is flat across all variants. Fix rate difference is within the regular variation across agentic runs.
TL;DR
It used to be that an agent, when handed a failed CI run, spent many turns just figuring out what broke. The problem: the failure data was there, it just wasn’t accessible by an agent. The new command circleci run get [run_id] –failure-report changes that.
Previously, the agent was taking too many steps to get to the error itself, so we summarized it into one line. More context for the agent means better fixes, and spending less.
“If you have repeated tasks for your agent, give them tools to get it done faster. Don’t make your agent do 3-4 things before it gets started.”
Joaquin Sandoval Miramontes, Senior Software Engineer, CircleCI
The same failure data, restructured for agent consumption, changes how efficiently the agent works. The agent still needs to gather context from the source files, understand the failure and write the fix. What it no longer needs to do is spend 4-5 turns navigating a log hierarchy to find the failing step.
The practical conclusion: agents using --failure-report reach the same fixes in about ⅓ less time and ¼ fewer tokens. At scale, given thousands of CI fixes per day, that efficiency difference is real.
Has your team run any similar experiments? What have you learned about restructuring failure data for agent consumption? We’d love to hear from you in the comments below.
Data: 84 real CI failures from CircleCI. Model: claude-sonnet-5. Quality Judge Model: Claude-opus-4.6.
