Last month, Claude converted our web app’s CI from dynamic config (a setup job running a few thousand lines of TypeScript to generate the real pipeline) into plain YAML. It took one pass, deleted over 5,000 lines, and was green on its first run.
A month earlier I wouldn’t have let it try. The difference was the feedback Claude got while editing: after every change, it saw what the compiler would make of its config. This post covers how we got the compiler’s answers in front of Claude, and how we made sure they were correct.
Hypothesis
DSLs, or domain-specific languages, are easy for an agent to write and easy to get subtly wrong. Everything parses; the mistakes are in what the keys mean. If your DSL is GitHub Actions, the model has seen so much of it that it mostly gets the meaning right from memory. Most DSLs don’t get that. CircleCI config is well known but nowhere near as common in training data, and your company’s internal deploy format is probably not in there at all. An agent writing those produces plausible YAML with an invalid resource class or a misspelled parameter, and finds out when the pipeline fails, minutes later.
So the knowledge has to come to the agent at edit time. We already have circleci config validate, which sends the file to the real compiler. It’s authoritative, but it’s an answer after the fact: the agent has to remember to run it, it needs the network, and it returns a list of messages with no positions. A language server is better suited. Claude Code’s plugins can run one, and the diagnostics it publishes after each edit go straight into the model’s context, with hover docs and go-to-definition on top.
The CircleCI YAML language server existed, but it had fallen behind. Functions, code signing and the expression language had all shipped since anyone last gave it real attention. Worse for an agent, it wasn’t always right, and a wrong diagnostic is worse than none: an agent will dutifully “fix” an error that isn’t there.
I assumed the work was mostly missing features, and that Claude could do most of it, provided it never had to guess what correct meant. A spec tells you what a config should mean. A reference implementation tells you what it does mean, including the parts nobody wrote down. CircleCI has two compilers, the original Clojure one and a Java rewrite. If Claude could read both and check its work against them, any disagreement would show up as a diff.
Setup
The work ran through an internal, experimental harness that has Claude plan, build and review changes in small units. Over about a month it produced most of the 333 PRs that merged. My job was mostly choosing what to measure and reading the results.
There were three oracles, each catching something the others missed:
The compiler source. The rule I gave Claude: where the compiler accepts something, we don’t report an error, and where we cover the same rule, we reuse its wording.
The compiler’s own test fixtures. Nearly 700 input configs, about 300 with expected errors, run through our diagnostics. Every mismatch was either our bug or a deliberate decision. This is where Claude learned that a matrix may expand to 128 jobs, not the 10,000 we had hard-coded (#554), and that a bare
yesoroffis a boolean, because the compiler reads YAML 1.1 (#727).A survey of real configs. A small tool that fetches the
.circleci/directory of every repo in CircleCI’s two GitHub orgs that isn’t archived or a fork and has been pushed to in the last two years, runs diagnostics over all of it, and diffs one run against the next (#565). These configs all run, so every error it reports is a claim that something working is broken. It started as throwaway scripts, and I only checked it in once it reproduced their numbers exactly.
I measured before fixing anything, as I did last time, so I could tell whether each fix reduced the noise.
Robustness got the same treatment. An agent edits a file the way a fuzzer does, one half-finished token at a time. In a language server, one panic (Go’s term for an unhandled crash) takes down every open document, not just the request that caused it (#490). So the tests type each config one token at a time, cut every line off mid-edit, and fuzz every position-based request against malformed input (#500). That found five crashes.
My favorite was an anchor that merges itself (build: &a then <<: *a), which recursed until the stack overflowed. In Go that’s fatal rather than a panic, so no amount of recover would have saved the process. Another was YAML error lines read as zero-based when they’re one-based, which had quietly put every syntax error one line too low.
Results
The first survey run was humbling: 770 errors across 599 configs that run green every day. One fix removed 484 of them. Orbs referenced by URL were looked up in the public registry, reported as missing, and then every job, command and executor used from them failed too. One bug was two-thirds of the noise, and no amount of feature work would have touched it.
The rest was the long tail you’d expect: code_signing rejected on macOS, an integer no_output_timeout, quoted job names, << >> parameters in typed fields. The single biggest leftover was “version is required” on every YAML file in .circleci/ that isn’t pipeline config, such as Smarter Testing’s test-suites.yml: 24 errors, one of them on this repo’s own file (#515).
The survey also counted things I chose to leave alone: 448 hints to add store_test_results, and 201 warnings about a workflow sharing a job’s name. Those are advice, so I left them, and the survey showed how often they would fire. Our real configs are private, so each false error survives as one of 31 sanitized acceptance tests, and all but one of them fail on the old server.
None of the performance work was guesswork; each fix came from a profile. Diagnostics had been rebuilding schemas, queries, and regexps on every request, while expensive tree-sitter look-ups slowed larger configs. Even after adding thousands of malformed-input requests, the full test suite grew by just four seconds, from 25 to 29. Faster diagnostics also let the fuzzer test more inputs per minute, so more bugs per minute, and a test loop short enough that Claude ran it on every change.
With false errors under control, the features went in, and the most useful ones catch exactly the mistakes agents make. CircleCI renders every value in a config as a template, so an ordinary heredoc in a run step, cat <<EOF, is an unclosed << tag and fails the compile; it has to be written cat \<<EOF (#643). A misspelled << pipeline.git.brnach >> doesn’t fail at all: the compiler quietly substitutes an empty string, so it’s now a warning, as is a deprecated pipeline value, with its replacement named (#652). And a branch filter like ^(?!main).*$ looks fine and isn’t, because the compiler’s regexes have no lookarounds (#626). Each of those is the kind of thing a model writes confidently from general knowledge, and each is invisible until a pipeline runs.
The surprise was that the oracle needed an oracle. Claude first named matrix jobs the way the Java rewrite’s code reads, with parameters in declared order. That was wrong: the original Clojure compiler sorts them by key (#810), and the Java code carries a comment admitting it copies legacy quirks “for parity during cutover”. A reference implementation tells you what happens. Sometimes you need the older one to tell you why.
We also decided where to be stricter than the compiler, and wrote it down. The compiler silently drops an empty requires:, which appears in 22 of its own fixtures; we flag it. Interpolating a list into a string renders it in Clojure syntax, “which is never what anyone meant”, so we flag that too. Later, Dan Mullineux added a corpus harness that parses 2,251 configs, from our testdata, the compiler’s fixtures and surveys of 15 orgs. It found zero panics, and still caught context ordering that changed between parses, failing 8 runs in 50 (#804).
Then the agent. The CircleCI plugin for Claude Code now runs the server, through a small launcher that downloads the release binary, verifies its checksum and caches it. Two details mattered more than I expected. Claude Code routes language servers by file extension only, so the plugin has to claim every .yml in the repo. The server now ignores anything outside .circleci/; before that, a GitHub Actions workflow got four false errors, which is exactly the kind of noise that sends an agent off fixing things that aren’t broken. And Claude Code doesn’t wait for diagnostics: it attaches whatever has arrived by its next step. With the server’s one-second debounce, Claude was always reading the previous edit’s errors. Setting the debounce to zero for agents fixed it. Now when Claude writes this:
jobs:
test:
docker:
- image: circleci/node:14
resource_class: huge
workflows:
main:
when: pipeline.git.brnach == "main"
jobs: [tset]it hears about the invalid resource class, the deprecated image, the misspelled pipeline value and the job that doesn’t exist on the same step, before anything is pushed. That loop, repeated a few hundred times, is how the web app conversion got done in one pass.
There’s still a gap. Claude Code has no way to hand the server a CircleCI token yet, so private orbs and contexts go unchecked there, unlike in the editors. That’s next.
TL;DR
If your DSL isn’t the popular one, bring the knowledge to the edit. The model won’t remember your config format. Put your compiler’s answers where the agent reads them, after every edit, not in a command it has to remember to run.
Give the agent an oracle, not a spec. Reference implementations and a real-world corpus turned correctness into a diff. Claude was fast because it rarely had to guess.
Measure false positives on real input first. A noisy checker trains an agent to make wrong fixes. One bug was two-thirds of ours.
Oracles disagree. Write down your tie-breaks. Decide where you match the reference and where you’re stricter, and keep it in the repo where the next agent reads it.
The server is better for people in VS Code and JetBrains too. Humans didn’t want false errors either; they’d just learned to ignore them. I’ll write about the editor plugins in a follow-up.




