We wanted to help customers use the CLI to debug failing pipelines faster. So we added a button in the UI that copies a ready-made AI prompt. The idea: click the button, drop the prompt into your AI agent, and get a diagnosis without hunting through logs. The prompt was built around a new command, circleci run get <id> --failure-report, which pulls all the context you need about a failing job in a single call instead of several.
I had assumed this would be a slam dunk. But the numbers we saw on our analytics dashboard didn’t add up: More customers clicked the button than ended up running the --failure-report command. This was odd, given that the prompt explicitly told the agent to use it.
At the same time, in my daily work, I tried using the CLI for my own work to fix pipelines and review runs. And I started noticing that even if I told the agent to use the CLI, it would use the CircleCI MCP (which I also have installed).
It left me with a lot of questions. Was this a coincidence? Where were all these clicks going? Were customers using different tools? Was the prompt not working? Was it a CLI distribution problem? Or maybe there were other factors influencing this behavior: was it the available context? Hidden preferences in the agent’s system prompt? Was the CLI too confusing to use?
To answer these questions, I built some evals and began investigating.
Hypothesis
My initial hypothesis was that the agent had certain preferences when it comes to which tools to use for each task (in this case, to fetch CI context). These preferences could be related to the model’s training (system prompt), the instructions given (prompt engineering) or to the available context for the agent. Which of these factors had an actual impact on the agent’s behavior, and why?
If we found what caused the agent to pick one tool over another, we could better design and distribute the tools we’ve been building for agents and engineers.
Setup
From my previous work, I had already set up a controlled environment with CircleCI tools that let me run evals on agent behavior and grade the results directly. I decided to reuse that project for this new behavioral experiment.
The task was simple: give the agent a failed pipeline URL and ask it to fetch and diagnose the error. Not fix it… just diagnose. That let me isolate the tool-choice behavior without mixing it with the ability to actually fix the problem (a whole different goal).
I set up multiple environments, varying two things:
Tool availability: CLI only, MCP only, REST API only, and combinations (all three, CLI + MCP, CLI + API, MCP + API).
Instructions: either free choice (use whatever tool you want) or an explicit instruction to use the CLI.
We had 10 different conditions. We ran each condition across 27 different samples using Claude-Sonnet-5 and recorded which tool the agent picked.
After viewing the initial results, I had some visibility on the preferences, but I was still missing the reasoning behind them. So I ran a second pass and added one line to the prompt: “Justify why you chose that tool.” I wanted the agent’s reasoning in plain language, not just the outcome.
Results
All the arms had a CircleCI Token for them to make authenticated requests to the API. This was mentioned to the agent in some experiments, and not in others, to analyze whether the agent tried to hit the API or not.
As you can see, there is a clear preference for MCP even when instructed to only use the CLI. When the MCP is available, it almost always goes that way. When not, agents showed a strong preference for the API over the CLI. Even when using the CLI, the API was still called a couple of times.
These results were already surprising. The hypothesis was somewhat correct: agents do prefer MCP over any other tool. But we still did not know why they were picking those tools; we only knew that they did. So we added an extra step and re-ran the free-choice experiments, asking the agents to explain why they chose the tool they chose.
The agents provided the following explanation of why they were choosing the API over the CLI:
“I’ll use the CircleCI REST API via curl since it gives structured JSON I can parse directly without needing interactive CLI auth setup.”
And for choosing the MCP over the CLI when both were available:
“MCP tools provide structured access to pipeline data without needing to construct API calls manually.”
Both explanations are wrong about the facts. The CLI in our setup was already authenticated via CIRCLE_TOKEN. It has a --json flag that produces the same structured output. The agent didn’t know either of those things. The information existed, but since it wasn’t in the prompt, it was missed.
So we updated the CLI prompt to state three things explicitly: the CLI is pre-authenticated via CIRCLE_TOKEN, it supports --json for structured output, and here are the exact command examples.
We ran the experiment again. The agent started using the CLI. We went from 0% of sessions using the CLI to 100% of sessions using it, while others also kept using the API for additional context. Nevertheless, when the MCP was available, it was still the agent’s preferred tool.
TL;DR
The agent isn’t reading your mind or your README. It’s making tool-selection decisions based on what’s in its context at the moment it has to choose. If that context is incomplete (if it doesn’t know the CLI is authenticated, or that --json exists), it will make a reasonable-sounding choice that happens to be wrong.
Two things follow from this.
MCP schema presence outweighs instructions. When an MCP server is configured, its tool definitions are already parsed into the model’s context. That’s a structural advantage over a CLI that has to be described in prose. Telling the agent to use the CLI is fighting uphill against a schema the model has already internalized. If you want the CLI to win in a mixed environment, you have to make the CLI’s capabilities as legible as the MCP’s, which means prompt engineering.
The agent’s wrong choice was entirely predictable from its context. It wasn’t hallucinating. It was reasoning correctly from incomplete information. “Doesn’t need interactive auth setup” is a reasonable thing to prefer, but it just wasn’t true in our environment, and our prompt didn’t say so. Once the prompt said so, the behavior changed.
The broader point: any time an agent is choosing between tools, the choice is a function of what it knows about each option at selection time. If you’re not getting the behavior you want, the first question is what the agent thinks it knows… and whether that’s accurate.
