As part of a larger initiative to build next-generation config tooling, a breaking change was shipped to the config compiler. The purpose? Stricter and more predictable validation gives us the reliable foundation needed to support more flexible config in the future. The problem? A customer had more than 10,000 config.yml files across their organization, and needed to know how many would break before the change went live, not after.
At roughly the same time, I’d been spending time considering how I might be able to use Chunk sidecars for related tasks outside of testing. I wondered: Do sidecars have the potential for a wider application?
I’d say that’s representative of my style as a whole: When conducting experiments, I don’t just think about the results, I try to think about new ways of conducting the experiment that can be applied to other experiments in the future to create additional efficiencies.
In the case of this specific experiment, I was fairly certain that the problem would be easily divided and conquered by using multiple computational resources. Without that approach, the experiment would be costly and cumbersome. Chunk sidecars provided the best route forward, and resulted in a method that I (and any other CircleCI customer) can replicate within future experiments too.
Hypothesis
Config validation is embarrassingly parallel. Each file is independent. There’s no shared state, no ordering requirement, no reason one file’s result should wait on another’s. If we could distribute validation across a fleet of Chunk sidecars, the wall-clock time should drop to roughly: (time to validate one file) / (number of sidecars running in parallel).
The biggest question was how to keep this cost effective from a time expenditure perspective, given that sidecars take about 30 seconds each to spin up. If each sidecar only validated a handful of files before its setup overhead dominated, the fleet wouldn’t be faster than sequential. It would just be more expensive.
Setup
Chunk sidecars are lightweight remote microVMs that run microbuilds alongside a developer’s work session. Each one is a full Linux environment: it can run the compiler, execute arbitrary code, and report results back.
It was important to me to conduct this work in a way the customer could reproduce. Don’t just trust me – try the same validation on your end as well to prove that the pipeline won’t be broken when the change is introduced.
Makoto Mizukami, Senior Field Engineer JAPAC, CircleCI
Knowing that the customer wouldn’t have access to our database (making the experiment potentially costly for them to run themselves) I made sure to use resources that would be available to them as well – the CircleCI API and CLI.
For this experiment, I partitioned the customer’s 10,000+ configs into batches and assigned each batch to a sidecar. Each sidecar:
Spun up
Received its batch of projects
Walked through each project to fetch its config through CircleCI API and run it through
circleci config validate--nextReported pass/fail results, and failure reasons if any
The fleet size was 100 sidecars in parallel at maximum. Task orchestration, command dispatches, and result aggregations were all done by a single Claude Code agent session, allowing me to save tokens (unlike Claude subagents, which hungrily consume tokens). The work was instrumented with Honeycomb throughout: 32,359 spans across the full run.
Results
The fleet economics held up. Setup overhead for the 100 sidecars was 2 minutes in total — small enough that the parallelism paid off at this batch size.
One finding worth noting: We also observed some specific patterns in validation failures. For example, certain series of strings tended to raise a validation error. By analyzing the result of the large-scale process further, we were also able to identify some “template” projects, from which most of the 120 configs derived.
The reason that the setup was reasonably easy was that Chunk sidecars provide snapshots. I could install a basic toolset in the sidecar, take a snapshot and apply the snapshot to the sidecars. Without that, this experiment would be impossible to conduct efficiently.
TL;DR
Embarrassingly parallel tasks don’t need clever algorithms. They need an environment that can run many agents at once. The compiler logic didn’t change. The validation logic didn’t change. What changed was the execution model: instead of one process working through a queue, a fleet of processes each worked through a slice.
Running them sequentially would have taken hours. A Chunk sidecar fleet did it in 4 minutes.
The thing to check before you do this: Make sure each agent knows exactly what its job is. Per-agent setup costs compound. If your task is trivial and your setup is heavy, the fleet won’t help. In this case, validation was fast enough and setup was cheap enough that the math worked. Instrument it first and check.
The customer got their answer before the breaking change shipped. It was an answer we could never have gotten in a timely manner without the massive parallelism Chunk sidecars offered. It proved my hypothesis that Chunk sidecars have multiple uses, and aren’t just limited to testing. As it turns out, sidecars are well implemented so that agents can easily leverage and consume the data, making them a perfect vehicle for experiments like this one.
We complain a lot about AI agent output. But we need to be thinking more about how we’re setting our agents up for success, and whether we’re providing them with sufficient power to do their job. Sidecars are one of the best ways to prepare agents to do great work.
