Every company with an API has been busy making it AI friendly, with the goal of improving how agents interact with their products. That is part of the advice that big model providers suggest.
Our current CircleCI APIs use version numbers in the URLs. V1, V1.1 and V2. They had accumulated four incompatible URL patterns, three ID formats and a different response shape per endpoint, effectively ten years of drift. The pitch for a new API V3 was explicitly about consistency for agents: one URL grammar, UUIDs everywhere, one response envelope, one error shape. An LLM should be able to predict /api/v3/{entity}/:id from the entity name alone instead of carrying the whole spec in its context.
We followed all the current advice on how to make the API discoverable by agents. We published the OpenAPI spec at a stable URL. We added /.well-known/api-catalog per RFC 9727, so a machine could discover the machine-readable descriptions. We generated an llms.txt and an llms-full.txt. We rendered the whole reference as markdown as well as HTML. We put the conventions - error shapes, idempotency, which operations are irreversible - into the spec description so they travel with every entity document, not just the index.
This post answers one question: did any of that work?
Hypothesis
My assumption at the outset was that agents would want to work with a consistent API, and that they would be able to easily find it.
I expected consistency to be the hard part and discovery to be the easy win.
Specifically:
Agents would find V3 on their own through the discovery surface - the catalog, the OpenAPI spec,
llms.txt.Once they found it, V3’s consistency would keep them there, because guessing a V3 URL is easier than looking one up in V2.
V2 usage by agents would decline as agents discovered the better option.
The first and third were wrong in ways more interesting than being right.
Setup
I first proposed building a new consistent API 3 years ago. At the time, it would have been a massive time expenditure likely not worth the investment. But by the time I pitched it again earlier this year, the execution had changed; we could use a swarm of agents with good skills. Agentic workflow enabled us to decide that it was a low enough cost to take on an experiment previously considered to be very high cost, and impacting every team on the org. I got the green light.
I tend to be more pragmatic than scientific. I understand the scientific method, but can also plan ahead to mitigate the risks of just diving in and trying things. In the case of the API, although it’s fairly high risk in some ways, I knew that if we didn’t publicize it fully, we could always stop and roll back.
Every V3 route ingresses through one gateway service. That was a deliberate call made for enforcement reasons, and it is also what makes this measurable: one dataset holds every public API request with its route, method and user agent.
After agreeing on the consistent flexible API shape, we built skills internally for teams to use to guide their agents, and within a few months we had V3 APIs in place that could replace 90% of existing API traffic.
Measuring agent traffic hitting our API is an inexact science, relying on parsing the User-Agent in the inbound requests.
Real strings from production for requests coming in via our CLI:
circleci-cli (darwin/arm64; 1.0.48773; claude-code_2-1-246_agent)
circleci-cli (darwin/arm64; 1.0.48692; codex)
circleci-cli (darwin/arm64; 1.0.48773; opencode)
circleci mcp (755dd082f9ef571f74ac433160ab068e10c9ef25)That is self-reported identity, not inference from traffic shape. If you want to measure agent usage of your own API and you control a client, stamping the harness name into the user agent is the highest-value thing you can do. It costs one line and turns an unanswerable question into a query.
Two caveats to keep in mind if you try something similar: An agent that hand-rolls its own HTTP calls is invisible to this method. I checked the bare-HTTP-client population and found roughly 41,000 spans a day where an agent could hide indistinguishably from a shell script (the span counts come from a sampled tracing dataset: the agent-filtered queries returned unsampled, the whole-traffic ones at 1.7× mean sample rate.)
Results
Interestingly, we realized the agents don’t care about the API directly. Instead, their focus is the MCP and CLI. Having a consistent API made it easier to build a good CLI. It’s only in this roundabout way that the agents are interested in using the API.
Agents overwhelmingly use V3. But not by migrating.
Hypothesis 2 confirmed, emphatically. Now the same measurement seven weeks earlier:
In those seven weeks, agent V3 share went from 57% to 97%. But look at the V2 column: 17,641 to 22,375. V2 agent traffic did not decline. It grew slightly. V3 did not win by converting anyone. It won by absorbing 760,000 calls of agent traffic that did not exist seven weeks earlier.
Hypothesis 3 was wrong: the migration story I expected to tell was actually a growth story.
For contrast, across all clients including browsers and scripts over that same recent week, V2 still outweighs V3 - 24,067,618 calls against 16,978,477. Humans and legacy integrations have not moved. Agents are a separate population behaving differently.
No agents found the V3 API on its own
Here is the entire discovery surface over thirty days. Every file we published for machines to find:
Discovery endpoints · All clients · 30 days to 28 Aug 2026
Agents made 807,209 API calls in a week. The entire discovery surface got 6,776 hits in a month.
Then I broke those 6,776 hits down by user agent, expecting a long tail of coding agents. There are three populations in there, and none of them is the one I was looking for.
Population one: nobody in particular
Crawlers, scanners, and humans
The largest single consumer of /.well-known/api-catalog is a bot that exists to grade whether your API is agent-ready. We are being audited for agent-readiness by a crawler while the agents themselves never look.
Population two: agents, but only when a human points at them
These are visible and separable, because a directed fetch carries a different user agent than an agent doing its own work:
Claude-User (claude-code/2.1.220; +https://support.anthropic.com/)That is Claude Code’s web fetch, which fires when a person says “go read this URL”. Across the whole discovery surface in thirty days it accounts for 23 requests - 6 on the static OpenAPI HTML, 5 on /docs/api/v3, 4 on /fullopenapi.json, 4 on /.well-known/api-catalog, and single hits elsewhere. Two more came from Claude-User/1.0, the claude.ai equivalent.
So the honest version of the finding is not “no agent has ever read it”:
Zero agents read the discovery surface unless a human explicitly directed them to it. Twenty-three times in thirty days, someone did.
Every one of those 23 was a person (probably one of us) saying “fetch the catalog”. Not one was an agent concluding on its own that a catalog might exist and going to look.
Population three: the training and search pipelines
This is the part that gives me some hope. Around 203 of the 6,776 hits are corpus builders and search crawlers:
They are all reading /openapi.json. The spec is being ingested, but just not into a working agent’s context. It is going into the next generation of training data and the search indexes agents fall back to. The discovery surface is on a much slower clock than I assumed.
Which means the optimistic reading of this whole post is available, but it is not the one I would have guessed. It is tempting to hope that the next round of models will start consulting catalogs and llms.txt properly, the way the standards intend. I do not think that is what will happen. What will happen is that GPTBot and ClaudeBot read our openapi.json, V3 ends up in the weights, and the next generation answers V3 from memory… exactly the way this generation answers V2 from memory.
The fix for stale training data is more training data. The catalog’s delivery vehicle is the corpus, and the crawler is the courier.
So the discovery surface does pay off. Just not as discovery. And the loop it runs on is a model generation long, which is not something you can shorten by publishing harder. The only lever you control on that timescale is still the client.
Why agents don’t discover: two layers of stale
Watching agents work with our API, the failure has two stages and neither involves discovery.
STAGE 1: Training data answers first
Ask an agent about the CircleCI API and it answers immediately and confidently with V2 shapes - /api/v2/project/{project-slug}/pipeline, slugs in paths, the old status enum. It is not looking anything up. V2 has been in the corpus for years; V3 has existed for months.
STAGE 2: The search index agrees with the training data
Agents will often announce they are searching the web when they are reaching into training data. When they do genuinely search, the indexes rank a decade of V2 documentation, V2 Stack Overflow answers and V2 blog posts far above anything we published this year. Two independent mechanisms, same wrong answer.
And here is the twist that makes discovery standards nearly useless in this shape: by the time an agent finally reaches a spec or a catalog, it already believes it is looking for V2. It arrives with a target. We updated the spec descriptions to say prominently that a newer V3 API exists, and an agent that has decided it needs the V2 pipeline endpoint reads past that to go looking for the V2 pipeline endpoint.
An in-context hint only helps a reader who has not yet decided what they are looking for. An agent has always already decided.
What actually moved the traffic
We updated the CLI to use V3, and pointed the built-in MCP server at the same code.
My colleague Pete has written up the rebuild this came out of (Rebuilding the CircleCI CLI from scratch) and it is worth reading alongside this one, because his half of the story is the half that actually moved the numbers below.
Roughly 57% of all CLI traffic to the public API now carries an agent harness stamp. Every one of the 807,209 agent calls arrived through the CLI or the MCP server. None arrived from an agent constructing its own HTTP request.
Agents do not discover APIs. They use the tools already in front of them. The CLI is already installed, already authenticated, already on the agent’s PATH, and --help is right there. The MCP server appears in the tool list. Neither requires discovery, a search, or a spec.
The lever was never the discovery standard. The lever was the client.
TL;DR
No agent reads your discovery surface unless a human points at it. The catalog, llms.txt and the OpenAPI spec drew 6,776 hits in a month. Agent reads: 23, every one a directed fetch. Self-directed: zero.
Publish it anyway, but price it as a training-data bet. 203 of those hits were GPTBot, ClaudeBot, OAI-SearchBot, PerplexityBot and MistralAI-User. The spec is being ingested, just not into a working agent’s context.
Expect the fix to arrive through the weights, not the standard. The next model generation won’t consult your catalog. It’ll just already know your API. That’s a model-generation-long loop you can’t shorten by publishing harder.
In-context hints only help a reader who hasn’t decided yet. We put “there is a newer V3” in the spec description. An agent that has already concluded it needs a V2 endpoint reads straight past it.
Ship the client and the MCP server. 100% of measured agent traffic arrived through a wrapper that was already installed and already authenticated. None of it was hand-rolled HTTP. If you want agents on your new API, update the tools they already have.
