Dennis Pilarinos has been thinking about this problem for years. He’s the founder and CEO of Unblocked, which he describes as the context layer for modern engineering teams: a system that surfaces the tribal knowledge inside an organization and makes it available to both the people doing the work and the agents working alongside them. Before Unblocked, Dennis spent years at Microsoft on the early Azure team, then moved through mobile CI and developer tooling at a time when, as he puts it, investors were telling founders “there’s no money in DevTools.”

There’s money in DevTools now. But the more interesting question is whether teams are building the right things into them. Rob and Dennis went deep on that question: why agents fail in ways humans don’t, what happens to the pull request when agents are writing the code, and whether we’re finally close to being able to draw a line from “we ran this agent” to “we moved this metric.”

1. Every agent you spin up starts on day one. Every time.

The framing Dennis uses here is hard to shake: imagine joining a company and being asked to fix a bug. You’re reasonably competent. But you’re up against someone who has three to seven years of context on that codebase. Who gets it done better? Almost certainly not you, not yet.

That’s the situation for every agent, every session. It doesn’t accumulate experience across runs. It doesn’t remember the incident channel discussion from three months ago, or why that particular API pattern was chosen over the obvious alternative. Each session starts cold.

“When you spin up an agent, it is basically like me joining CircleCI on day one. It has no meaningful historical context for which to draw upon. And so it does what I would do. I would probably dig through a whole bunch of documentation. I would go down paths that don’t actually lead to productive outcomes.”

Dennis Pilarinos, Founder and CEO, Unblocked

The practical takeaway is this: if your agents are producing messy, roundabout solutions, the problem is likely the missing context. What the agent needs is often sitting in Slack, in a closed issue, in a Confluence doc nobody’s opened in 18 months.

2. The code alone is not enough context.

Dennis is direct about something most teams discover the hard way: indexing the codebase is a starting point, not a solution. The reasoning behind the code lives elsewhere.

“I have never ever met a company that’s like, our docs are super well organized and completely up to date. That just doesn’t exist.”

Dennis Pilarinos, Founder and CEO, Unblocked

When Unblocked pulls in Slack threads, bug trackers, incident channels, and documentation alongside source code, the quality of outcomes goes up significantly. This isn’t a surprise in retrospect. It’s exactly how a person learns a codebase. They read pull request comments. They get feedback on their first few contributions. They sit in on postmortems. The “scar tissue,” as Dennis calls it, is where the real knowledge lives.

Teams building context systems that rely only on source code are building the equivalent of onboarding a new engineer with read-only access to the repo and no other communication. It doesn’t work for people either.

3. AI code review is only as good as the context behind it.

Unblocked started using its own context layer to build an internal code review tool after surveying the existing market and finding it underwhelming. The tools they evaluated felt like early-stage engineers leaving noise in PRs, sometimes literally leaving haikus. Not useful.

When they built review on top of a context layer that knew the history of decisions, past incidents, and team-specific conventions, the results were different. The reviewer understood why certain patterns existed.

“It’s not just limited to the source code. It knows conversations you’ve had about incidents, or some of the best practices that you might have written down somewhere else that other tools might not necessarily know.”

Dennis Pilarinos, Founder and CEO, Unblocked

If you’re evaluating AI code review tools and finding them shallow, this is worth paying attention to. The limiting factor is usually not the model’s ability to read code. It’s whether the model understands the context in which that code was written. Most tools don’t have that. The ones that will matter will.

4. The pull request is two tools in a trench coat.

Rob and Dennis had one of the more useful framings of this conversation on the subject of pull requests. The PR solves two distinct problems: it’s an information-sharing mechanism (here’s what changed and why) and a change management mechanism (here’s how we control what goes to production). Those are different problems, and they’ve been bundled together for so long that most teams have stopped asking whether they need to be.

As agents generate more code, the PR review bottleneck gets real fast. Teams that Dennis talks to are saying their engineers are backed up on review, not on writing. The question becomes: can you decompose what the PR was doing and solve each part better?

“Go back to first principles. Pull that apart and say, what would be the way we would solve for this piece of it, for this specific problem in this new context? It doesn’t have to be one solution. It just happens to be now, because it was there.” Rob Zuber, CTO, CircleCI

The implication for engineering leaders: look at every gate in your delivery pipeline and ask what problem it was originally solving. The answer might still be worth solving. The mechanism might not be the right one anymore.

5. AI adoption inside one company spans an enormous range.

This was one of the more grounding moments in the conversation. Dennis described organizations where some engineers are just discovering tab-completion and others are running thousands of autonomous agents handling multi-hour implementation tasks. Same company. Same week.

He puts roughly three to five percent of engineering teams in the “AI pilled” category: people who have experienced enough of the technology to feel acutely where the gaps are. They’re the ones who find the lack of organizational context genuinely painful, because they’ve hit the ceiling. The rest of the org might not have reached that ceiling yet.

This matters for how you roll out tooling and expectations. Someone whose primary interaction with AI is autocomplete doesn’t have the same mental model of what’s possible, or of what’s broken. The journey isn’t linear, and it’s not uniform.

6. The ROI problem is closer to being solvable than it’s ever been.

This was the most forward-looking thread in the conversation, and both Rob and Dennis were careful about timelines. But the observation is worth sitting with: agents are instrumented in ways that people never were. You can actually see what it cost to get to a solution. The question is whether you can connect that to business outcomes.

“We can say: what did it take in order for you to come to this conclusion or to this solution? And can you do it in a token-efficient way? But what will happen next is: how do we reconcile that execution relative to business objectives? Not just how much did it cost, but how did it actually move the needle for the business?”

Dennis Pilarinos, Founder and CEO, Unblocked

Finance teams are noticing the spike in costs. The engineering side hasn’t yet built the case for what that cost produced. Closing that gap, drawing the through-line from token spend to business outcome, is the work in front of engineering leaders right now. Dennis thinks we’re closer than we’ve ever been. Rob’s take: we’ve taken one step on a very long journey, but it’s a real step.

Confident Commit is CircleCI’s publication for engineering leaders and practitioners. Subscribe to get episodes and companion posts delivered directly.

If AI can write the code, why train someone when the model can just do it?

Rob Zuber has heard this take. He thinks it’s wrong, but not in the hand-wavy “humans will always matter” sense. He thinks it’s wrong because it misidentifies what the job actually is, and therefore misidentifies what we’d be losing.

In this episode of Confident Commit, Rob sat down with two colleagues from CircleCI’s software factory team: Michael Webster, a software engineer with more than a decade in the industry, and Hanabel Mengistu, who joined CircleCI less than a year ago straight out of college. The result is one of the more honest conversations about the current state of software development: what experienced engineers are struggling to let go of, what new engineers are struggling to find, and where those two things collide.

1. The tools changed faster than the job descriptions did

2025 brought a lot of change, fast.

Webster framed the inflection point precisely: “Around the end of last year, we kinda had that Sonnet 4 / Opus 4 moment where it’s like, hey, something’s happened with these models. They’re able to do a lot more.”

The instinct, for a team that wanted to push hard on AI-assisted development, was to try running multiple agents in parallel and see what broke. The answer came fast.

“I think within the first forty-five minutes to an hour of starting, a lot of our instincts immediately started breaking down. The merge conflicts, the trying to stack them, the manual review overhead. Sub two hours, we were like, yeah, we’re gonna have to figure some things out here. Code generation time outpacing CI build times, all of that was pretty eye-opening.”

Michael Webster, Principal Engineer, CircleCI

The problem wasn’t the AI. The problem was that the existing process, designed around the constraint of human coding speed, was suddenly the wrong process.

Rob’s framing: “We fundamentally changed our constraints. But as humans, we’re not good at massive leaps in approach. So we’re clinging to the processes that we have and trying to tweak them with this fundamentally new toolkit instead of asking: what was it we were trying to do, exactly?”

2. What experienced engineers are actually clinging to (and why it’s hard)

Webster put something into words that most senior engineers are circling around but haven’t quite said directly:

“What if the stuff that I like doing is the bottleneck? What if the stuff that has saved me production incidents and helped me build a career, what if that is now the bottleneck? That’s a really challenging thing to work through.”

Michael Webster, Principal Engineer, CircleCI

If your expertise is in a thing that’s now being commoditized, the rational move (reprioritize, go further up the stack) is emotionally costly in a way that’s hard to explain to someone who hasn’t been through it.

Webster’s working theory: go back further than you think you need to. Sharpen the intuition, not the implementation. He cited Fred Brooks’ “No Silver Bullet” and the distinction between essential and incidental complexity. AI is handling a lot of the incidental complexity that accumulated over decades. The essential complexity of building systems that solve real problems for real people has not gone anywhere.

“You’re just moving further up the stack,” Webster said, “and that’s a skill a lot of folks don’t have, because for a long time it was really hard to do those other things and now those things are easier.”

3. College prepared Hanabel for a job that barely exists anymore

Hanabel graduated about a year before joining CircleCI. She came up through a computer science program that treated AI as a form of academic dishonesty: sign at the bottom of every test saying you didn’t use it, even as everyone quietly used it.

The irony Rob raised: using AI would have been cheating. Using AI would have also prepared her for her actual job.

Like most fresh graduates, Hanabel found herself equipped with technical skills, but needing experience in everything around them.

“I feel like that’s me during all the meetings. I knew nothing about the process or the delivery side of things. I didn’t know what that would look like outside of college.”

Hanabel Mengistu, Software Engineer, CircleCI

Rob’s take is worth sitting with: “A lot of the day-to-day job is not actually about the structure of software. It’s: How do we decide what to work on? How do we communicate about what we’re working on? How do we decide the risk involved? How do we deal with breakage? That’s probably more than 50% of the job. And then we took the part we did focus on in school and said, don’t worry about that anymore.”

Subscribe now

4. Juniors aren’t being mentored, and it’s showing up as loneliness

One of the quieter observations in this episode: junior engineers are feeling isolated, and it’s not entirely their fault or anyone else’s fault specifically. The mentorship pipeline has a structural problem.

Hanabel described what she’s hearing from peers:

“A lot of my friends that are in the industry feel a little bit alone. People aren’t mentoring other juniors anymore because it’s something AI can do. Not really, but for small questions, you can just ask Claude. And asking AI is not always the most valuable way to learn.”

Hanabel Mengistu, Software Engineer, CircleCI

The deeper issue Rob named is a two-sided dynamic: senior engineers assume juniors can just ask the model, so they stop offering; junior engineers sense they shouldn’t bother asking, so they stop reaching out. But the value a good mentor provides isn’t just the answer to the question. It’s the context around the question, the adjacent things worth knowing, and the pattern recognition that says “you’re asking about X because you’re missing Y.”

5. The great leveling might actually help juniors more than it hurts them

The most optimistic thread in this conversation came from an unexpected place. Hanabel pushed back, gently, on the framing that the current moment is particularly hard for new engineers. Her read: it might actually be easier in one specific way.

“I feel like it’s comforting. It’s nice to be able to figure stuff out with other people and not feel like I’m the worst. I’ve been able to contribute so quickly. I wouldn’t be able to do all the things I’m contributing to if it weren’t for that.”

Hanabel Mengistu, Software Engineer, CircleCI

Claude Code went GA in June 2025. ChatGPT went GA in November 2022. Nobody has ten years of experience in any of this.

The experience gap that normally takes years to cross hasn’t had time to form in the places that matter most for this transition. For junior developers, this means that the playing field has flattened.

A curious generalist with no bad habits to break might actually have an edge over a deeply experienced engineer who has spent a decade optimizing for a set of constraints that no longer apply.

Subscribe to Confident Commit for more conversations about how software delivery is changing, from the people figuring it out in real time: confidentcommit.com

Rob sat down with Luca to talk about what high-performing teams are actually getting right with AI in 2026, why code review was always more questionable than we admitted, and how Luca built a three-part framework: guides, gates, and guards, to keep AI agents honest in production code.

1. 2026 is the year of reckoning, not adoption

Last year, AI tooling spread fast. Most teams added something. Some added everything. This year, the question has shifted from “are we using AI?” to “what are we actually getting for it?”

Luca has been watching this across a wide range of teams, and the diversity is striking. There’s no consensus, no single playbook, and the gap between teams who are extracting real value and teams who are just spending tokens keeps widening.

“2025 was more like the year of adoption. Eventually we got to: ‘AI is everywhere, 95 plus percent of engineers using AI’. And this year is more like okay, show me what you’re getting out of it, as the token spending keeps growing for everybody.”

Luca Rossi, Founder, Refactoring.fm

The teams pulling ahead are the ones who’ve gotten specific about where AI actually helps and where the friction still lives.

2. You have to know your SDLC before you can improve it

This sounds obvious. It isn’t. Luca’s observation is that the single biggest differentiator between the teams getting results and those who aren’t is whether engineering leaders actually have visibility into what’s happening in their delivery process. It’s about real awareness of where things slow down, where handoffs break, and where decisions stall.

“Just being aware of what’s going on in your developer process. Which means talking with your engineers, knowing where the problems are, where the bottlenecks are, having a good feedback loop where you can actually work on things that you know can be improved.”

Luca Rossi, Founder, Refactoring.fm

Rob noted that this insight has been true for 30 years. But the AI era has made it more consequential because improvements compound faster when you know where to aim them.

3. Guides, gates, and guards: a framework for trusting AI agents

Luca has been building Tolaria largely without writing code himself, directing AI agents through the full development workflow. That experiment forced him to get precise about trust. The result is a three-part framework he calls guides, gates, and guards.

Guides are the instructions agents follow: rules in the agents.md file about how code should be written, covering things like TDD, the Boy Scout rule, analytics instrumentation, and localization.

Gates are deterministic checks that run before anything gets committed. Because agents ignore the guides roughly 10% of the time, you need enforcement that doesn’t rely on the agent’s judgment. Luca uses local hooks to check code health, quality, and security and blocks commits that don’t clear a threshold.

“AI can happily ignore these rules every now and then, like ten percent of the time they just don’t care and do things how they like to do. And so you need deterministic gates to avoid [the fact that] bad things happen in production and get committed and pushed.”

Luca Rossi, Founder, Refactoring.fm

Guards are nightly or weekly audits that catch judgment calls and bigger-picture issues. Did the agent forget to write an architecture decision record? Have performance metrics degraded since the last release? Are there refactoring opportunities the agent couldn’t see at the task level?

Subscribe now

4. Velocity without visibility is just faster drift

One of the clearest tensions in the conversation: AI compresses the coding loop, but that doesn’t automatically speed up software delivery. It can just mean you’re drowning somewhere else.

Luca pointed to code reviews and product planning as the two places teams tend to hit the ceiling. Code gets written faster, but review queues get longer. Requirements get built faster, but upstream work to define what to build doesn’t scale with the throughput.

“We’re just getting faster at coding, but people are drowning either in code reviews or are too slow at creating new requirements or measuring what’s been done. So we have to take a holistic view about the whole thing.”

Luca Rossi, Founder, Refactoring.fm

The teams breaking through aren’t asking “how do I make each stage better?” They’re asking “what am I actually trying to get out of this process?” and redesigning from that question with the tools they now have.

5. Code review was already questionable. Now it’s untenable.

Luca has been skeptical of traditional code review for longer than AI has been part of the conversation. The asynchronous process was a time sink before agents started generating code at scale. Teams would fight to shave 10 minutes off CI times while PRs sat idle for 18 hours. And the assumption that humans reliably catch important issues by scanning unfamiliar code quickly has always been optimistic.

Rob agreed: We were never that excited about reviewing our colleagues’ code. We tolerated it because it was the best available option.

“Even if you don’t account for the quantity, it’s just a miserable experience. You don’t want to corner people into spending a lot of their time at work doing miserable things like reviewing AI code.”

Luca Rossi, Founder, Refactoring.fm

Now that AI agents can generate code far faster than any team can review it, the math is clearly broken. But the answer isn’t faster humans. It’s rethinking what “review” is actually trying to accomplish and finding better mechanisms for that. Deterministic gates, paired programming for high-stakes changes, and automated quality enforcement upstream before anything reaches a human.

6. Building for yourself is a cheat code with limits

Luca’s most consistent lens across everything he talked about, including the newsletter, the app, and the AI workflow, is that building something you personally need gives you a feedback loop most teams don’t have. You know the problem because you’re living it. You know when the solution works because you feel it.

He’s explicit that his experience building Tolaria is privileged. He’s working alone, with no legacy code, and can afford to experiment in ways that most real-world teams can’t. But the underlying principle transfers: the teams getting the most out of AI tools right now are the ones with enough self-knowledge to know what they’re actually trying to solve.

Confident Commit is CircleCI’s podcast for engineering leaders who want to understand how software delivery is actually changing. Subscribe at confidentcommit.com to get every episode plus companion posts like this one.

Laurie Voss has watched this pattern play out at scale. As co-founder of npm (the package registry that now serves hundreds of millions of JavaScript developers), a DevRel leader at Netlify, and now Head of Developer Relations at Arize, she has spent her career at the intersection of data and developer behavior. She knows how software communities learn, and she knows how they avoid learning.

The current moment in AI development, she argues, looks a lot like the early days of software testing: everyone knows tests exist, and most people are not writing them.

Rob Zuber, CTO at CircleCI, sat down with Laurie to talk about what “real evals” actually are, why the eval loop is going to change how software gets built, and whether we are anywhere close to agents building production software without humans in the loop.

1. Evals are just tests. Stop calling them anything else.

The first problem Laurie has with the word “evals” is the word itself. It is borrowed from the world of ML, and it lands with a thud in the ears of engineers who came up through web development, backend systems, or platform work.

“As soon as you say traces are logs and evals are tests, AI engineers get the story much better. You’re supposed to be writing tests.”

Laurie Voss, Head of DevRel, Arize

The terminology barrier is not trivial. If engineers hear “evals” as an ML research concept, it stays over there, in someone else’s domain. If they hear “tests,” the mental model clicks and the excuses disappear. The biggest competition for an evals company, Laurie points out, is not a rival product. It is engineers who just do not do any evals at all.

The implication is worth sitting with: a large fraction of AI applications going to production right now have no systematic validation. If you are running evals, you already have an advantage.

2. “Vibe-based testing” is why people hate AI.

There is a phrase that stuck: vibe-based testing. It’s what happens when a developer makes a change, runs a favorite query or two, feels good, and ships. It works fine for the happy path. It does nothing for the other 80% of inputs users will eventually throw at the application.

“A whole lot of software is getting all the way to production right now using nothing more than vibes. And it is showing up as really poor reliability. And it is making people hate AI because they interact with AI that’s busted all the time.”

Laurie Voss, Head of DevRel, Arize

This is not a new failure mode. Anyone who has watched engineers click-test a UI before a release will recognize the shape of it. The specific damage here is that broken AI applications have the potential to erode trust in AI as a category. The cost is the whole wave of adoption getting slower because people have been burned.

If you are building AI applications, the quality of your evals is now a competitive differentiator.

3. LLM-as-judge is the only way to scale, but it needs its own tuning.

Non-determinism is the real wrench. Traditional tests are essentially string matching: you put in an input, you expect an output, you check if it matches. With AI applications, the same input can produce a million different outputs, and a meaningful fraction of them will be correct. You cannot just check for a specific string.

For some cases, smarter string matching helps: check for “one hour ago” and “60 minutes ago” and other valid phrasings. But for anything requiring judgment, you need a judge. In production, at scale, that judge has to be another LLM.

The catch is that the judge LLM introduces its own layer of non-determinism. It has its own prompt, and that prompt needs to be tuned against real output to confirm it is actually measuring what you want it to measure, rather than producing good/bad labels at random.

If you are running evals in CI, where they need to execute reliably thousands of times a day, this is the architecture you are building toward: a test suite powered by an LLM that has itself been validated as a good judge.

Subscribe now

4. Regression evals vs. capability evals: two different jobs.

One of the most useful frames in the conversation was the distinction between regression evals and capability evals. They are not interchangeable, and conflating them leads to poorly designed test suites.

Regression evals are what most people think of first. They are close to traditional tests: can my application still do the things I already know it can do? They are the safety net you run continuously.

Capability evals are different in a fundamental way.

“Capability evals are a test I expect you to fail. This is a test where I expect you to score 20 percent and then do better the next time and climb a hill of capability.”

Laurie Voss, Head of DevRel, Arize

The combination opens up serious possibilities. Because evals powered by LLMs include an explanation field alongside a score, you can feed the failure explanations from 100 runs back into a coding agent and ask it to improve the software based on what went wrong. No human required. The eval loop drives the development loop. This has not been fully productionized yet, but the pieces exist.

5. Context engineering is the whole game.

The conversation shifted toward agentic coding and the question of how close we are to agents that can build production software end to end. Laurie’s answer was grounded in a single concept: context engineering.

Every capability question in AI development comes back to context. Can this LLM be a doctor? That requires a human lifetime of context, and it does not fit in a window. Can this LLM build a website? Websites are a narrow enough domain that Laurie thinks we are close to an agent harness that can handle it reliably. Can this LLM build production software in general? Software development as a domain is too large for any current window or graph structure to cover well enough to one-shot it.

Context graphs, one of the tools Arize is working on, are a form of context compression: a way of defining which topics connect to which other topics so an LLM can navigate a large domain without losing things out the bottom of its window.

The practical takeaway for engineering teams building on AI today: the investment in skills files, harnesses, and structured context is what determines whether your agent makes good decisions or system-level mistakes.

6. Software developers always solve their own problems first.

There is a pattern in software development that Laurie named plainly: the community perfects tooling for its own domain before solving anyone else’s. The reason there are so many frameworks is that developers made their own environments excellent before turning to enterprise software, which is still, by most measures, remarkably bad.

The same dynamic is happening now. AI coding tools are improving fast because developers are building them and using them on their own work. The patterns being developed in AI coding evals, harness design, and context engineering will eventually transfer to other domains. But that transfer comes later.

For teams building AI products now, the implication is to pay attention to what the software development domain is learning. The patterns being forged here will be the foundations for whatever comes next.

If this episode got you nodding your head, subscribe to Confident Commit for more conversations like this one.

Hywel Carver has been thinking about this longer than most. He’s the founder and CEO of Skiller Whale, a company built on the premise that engineers learn best by doing, not by watching. When he first talked publicly about what AI-generated code would mean for developer skill development, it was 2023, at a conference. The consensus at the time was still pretty comfortable. Things have gotten less comfortable since.

Rob sat down with Hywel to work through what it actually means to grow engineers in a world where most code is generated, not written.

1. Knowledge transfer is not the same thing as learning

The way most organizations approach skill development is to throw information at people and call it training. Watch this video. Read this documentation. Here’s a course with 47 modules.

“Very often the way that we expect people to learn is by just giving them a load of information. The actual practice of trying to do that thing, we leave you on your own to do in isolation. And that’s where the rubber hits the road. That’s when you find out whether you actually can do it or not.”

Hywel Carver, Founder and CEO, Skiller Whale

The problem is that information without practice doesn’t produce skill. Hywel’s framing: knowing and understanding are the first two rungs of a six-level learning hierarchy. Actually being able to apply something, evaluate someone else’s work, and eventually create something new comes later, and it only gets there through doing. If your development program stops at the second rung and calls it done, you haven’t built anything.

2. Bloom’s taxonomy explains exactly why AI code review is harder than it looks

Hywel referenced Bloom’s taxonomy of cognitive learning as a framework for understanding the current problem. The six levels build on each other: know, understand, apply, analyze, evaluate, create.

The way engineers have historically gotten good at reviewing code is by writing a lot of it themselves and having it reviewed in return. That cycle is how you build the mental model that lets you look at a loop and immediately think about off-by-one errors and variable state, rather than reading each line individually.

“We are used to a world where applying the skill of writing software is how we get good enough to evaluate it, to look at what someone else has done and say, yes, I am happy to put my name to that and for it to go into production.”

Hywel Carver, Founder and CEO, Skiller Whale

If 90% of code is now AI-generated, the training loop that used to happen organically through writing code no longer fires at the same rate. You can still develop those mental models. You just have to do it deliberately, through structured practice designed to build them, rather than picking them up as a byproduct of shipping.

3. AI and humans are bad at different things, and that’s precisely why both matter

Another interesting argument from the conversation: the error categories that AI is susceptible to are largely not the ones humans fall into, and the reverse is also true. AI is excellent at catching syntax errors, off-by-one issues, and assignment-vs-comparison bugs. Humans are not particularly good at those.

Humans are good at something different: reasoning about real-world consequences that weren’t in the scope of the ticket. The change that would give a specific category of users infinite free access. The knock-on effect that shows up in a workflow nobody thought to include in the spec.

“We are good at detecting things like: this is going to have a knock-on effect to that category of user that wasn’t in the scope of the change request. Humans are good at spotting that kind of thing.”

Hywel Carver, Founder and CEO, Skiller Whale

Hywel presented an example of an AI agent that found production credentials in a repo and wiped the database in nine seconds. It’s an argument for keeping human and AI review in parallel, because they’re checking for fundamentally different things.

Subscribe now

4. Using AI well is a skill, and it’s learnable

There’s an assumption baked into a lot of AI tooling rollouts: give people access to the tool, maybe run a vendor demo, and let adoption happen. Skiller Whale ran a study that suggests this assumption is wrong.

They worked with part of an engineering organization while the rest of the org had access to the same tools but no structured training. Both groups had the vendor talk. Both had access to the AI coding assistant.

“The group we worked with had 90% weekly active users where the rest of the org had 50% weekly active users. When we looked at their main measure of productivity, which was PR throughput, the group we worked with saw a roughly doubling in their PR throughput. The rest of the org actually went down very slightly.”

Hywel Carver, Founder and CEO, Skiller Whale

A doubling versus a slight decline, same tools, same company. The difference was whether engineers had built mental models for how the tools actually work. People who understand what an LLM is doing underneath prompt differently, catch failures differently, and know what to try when the output is wrong.

5. Mental models don’t expire, even when implementations do

The pace of change in AI tooling feels genuinely overwhelming if you try to keep up at the surface level: new models, new frameworks, new prompting techniques, new papers every week. Hywel’s argument is that this is the wrong level to track.

“If you have a good sense of how an LLM works, and then someone says, one of the things an LLM can do is produce output that says do a function call, you see how tools work and you’re like, okay, that makes sense to me. You have a way better understanding of it than someone who’s just been a user of those systems.”

Hywel Carver, Founder and CEO, Skiller Whale

His analogy: if you had a solid mental model of HTML, CSS, JavaScript, and how a browser renders pages in the mid-1990s, that model has not meaningfully changed today. The implementations are faster, the APIs are cleaner, but the underlying structure is the same. The mental model carries you through decades of surface-level change. The same logic applies to LLMs. The tips-and-tricks content will be obsolete in six months. But the architecture of how they’re trained and how they produce output hasn’t changed that much.

6. The job title is changing faster than anyone expected

One thing Hywel raised that didn’t get a full chapter in the conversation but probably deserves one: after teams go through structured AI training, the next thing they come back asking for is communication skills. Engineers are having more contact with stakeholders. Product functions at some companies have gotten smaller. Individual engineers are doing more.

That sounds like good news for engineering influence. It also means engineers who were hired to write code are now expected to run conversations they weren’t trained for, translate technical decisions for non-technical audiences, and navigate organizational dynamics that used to be someone else’s job.

The skill set that made someone an excellent engineer in 2020 is still valuable, but it needs some expansion in order to meet the role’s new level of complexity.

Subscribe to Confident Commit for new episodes and the data behind how software actually ships: confidentcommit.com.

AI changed the math.

On this episode of the Confident Commit podcast, Rob sat down with Chris Kelly, Head of Product at Augment Code, to dig into what’s actually shifting in how software gets built and delivered. Chris is unusually positioned to see both sides of this: he’s a product leader who recently built the first version of Augment’s CLI himself, over a weekend, to prove a point. Did he prove it? Read on for the takeaways.

1. Prototypes beat specs now. This changes everything upstream.

Chris built the Augment CLI because he was tired of spending time in meetings convincing people of an idea in the abstract, so he just built it.

“In order to make it happen, I just prototyped the first version over a weekend... Everyone’s like, this is great. Which is like, especially as a product person, I don’t have to spend countless hours in meetings trying to convince you of an idea in the abstract. I can actually just show it to you.”

Chris Kelly, Head of Product, Augment Code

This isn’t new advice. Danish programmer David Heinemeier Hansson said “code wins arguments” years ago. What’s new is that it’s now accessible to more people, faster, at lower cost. The threshold for “just build it” has dropped so far that it’s starting to break how we think about planning cycles, design reviews, and requirements docs.

If you can build the thing faster than you can finish the meeting about the thing, the meeting needs to change.

2. The real savings isn’t faster code. It’s less coordination.

A lot of the ROI conversation around AI tools focuses on lines of code per hour. That’s the wrong unit. The underappreciated savings, as Chris framed it, is coordination cost: the energy that goes into aligning people, running standups, writing meeting notes, keeping everyone in sync.

“It’s not just can I write code faster? But can I decide what to build and build it faster because I have less human beings that I have to collaborate with. So then you just get smaller pods of people doing a lot more work. That I think is the unappreciated cost that we’re actually saving.”

Chris Kelly, Head of Product, Augment Code

Smaller teams with less coordination overhead have always outperformed larger teams with more. The ROI question gets reframed entirely. It’s not “are my engineers using tokens efficiently?” It’s “what would it have cost me to coordinate the team required to do this work before?”

3. Any IC engineer now has signing authority to increase engineering capacity.

In the old model, scaling up engineering capacity meant a long conversation about hiring budgets, ramp time, and headcount. It was slow by design.

“Any IC engineer right now has the signing authority because all they do is run more prompts. They have the signing authority to increase the capacity of the engineering organization. It really puts us in a really bizarre state.”

Rob Zuber, CTO, CircleCI

That’s not an exaggeration. The gates that used to govern capacity growth (budget approvals, hiring pipelines, onboarding time) don’t apply to agents. An engineer who decides to spin up additional agents for a project has effectively made a capacity decision unilaterally.

That’s either terrifying or exciting depending on how you’re set up. The engineering leaders figuring it out now are the ones who will be ahead when everyone else catches up.

Subscribe now

4. Think of agents like dynamic cloud compute, not additional headcount.

Chris offered what might be the clearest mental model for the current moment: the shift from provisioning racks of servers to dynamic cloud compute.

Before the cloud, you provisioned for peak. You either had too much capacity or not enough, and both were expensive. The cloud let you pay for what you actually used and scale in real time. Agents do the same thing for engineering output.

“Do agents allow us to keep our staffing levels and our teams the right size and then be able to respond when things change? Do we get closer to that in a healthier way?”

Chris Kelly, Head of Product, Augment Code

This maps onto the over-hiring cycles we’ve seen at large companies too. If you can scale capacity dynamically with agents, you don’t need to staff up for a product that might not take off. You prototype it cheaply, validate the bet, and hire when you know it’s working, not before.

The question isn’t “how many engineers do I need for this project?” It’s “what’s the right human baseline, and what can I run dynamically on top of it?”

5. Senior engineers are high-leverage (but only if they get invested).

Rob and Chris both circled back to a tension that comes up in almost every conversation about AI adoption: senior engineers often get the best results from AI tools, but they’re not always the first to reach for them. They already know how to do the work. Why hand it off?

The problem is what they leave on the table when they don’t.

“It was always more valuable that a senior engineer was mentoring and helping five junior engineers than having them handwrite a bunch of code on their own. And so now, how do we do this? Make everything shareable in that way. If I build an agent that does a specific task and it’s really good at it, it turns into an institutional agent that we run on all of our PRs now.”

Chris Kelly, Head of Product, Augment Code

When a senior engineer builds a skill, a prompt, or an agent workflow that solves a recurring problem, they’re encoding institutional knowledge in a form the whole team can run. That’s leverage in a way that handwriting code never was.

The engineering cultures that thrive will be the ones that treat those artifacts the same way they’ve always treated scripts that became services: write it once, get it into the shared system, and stop making the same decision twice.

6. Agents need to get better on their own. Most aren’t designed to yet.

The last thread Rob and Chris pulled on was the one furthest out on the horizon: self-improving agents. Right now, most agent workflows are designed as static patterns: trigger an action, get an output, repeat. That’s useful, but it’s leaving something significant on the table.

“I want my agents to always be improving their own prompts, their own systems and having their own memory. And that’s really where we’re focused a lot. And it’s been really exciting to see what’s possible when that comes true.”

Chris Kelly, Head of Product, Augment Code

The analogy Rob reached for is incident response. The most valuable engineer in a production incident isn’t necessarily the best coder. It’s the one who’s been around long enough to say “I’ve seen this before, it’s probably that.” That knowledge lives in their head, and it walks out the door when they leave.

The teams that are ahead are already asking: how do we design agents that write their own memories? How does a fix in one agent make every future agent smarter? That’s where teams should be heading right now.

Want more on how software delivery is actually changing? Subscribe to Confident Commit and join Rob for conversations with the people building through it.

The conversation covers how Loïc drove AI tool adoption across his engineering org without mandates (and which senior engineer's change of heart became a cultural turning point), why great UX is still the real moat in an age where anyone can ship an average product fast, and how email, despite everything, remains the connective tissue of professional life. Plus: what it's like to rethink your entire SDLC when the economics of building software change overnight.

The conversation explores the paradox of our current moment—where developers can generate more code than ever, yet senior engineers wake up to 20 AI-generated PRs that still require careful human review. Greg discusses why companies like Google’s techniques for handling high volumes of code changes are suddenly relevant again, the emerging role of AI code review as a "super linter," and why he believes the best engineering cultures prioritize being "urgent, expert, and kind." Whether you're an engineering leader navigating the AI transformation or a developer trying to understand how code review needs to evolve, this episode delivers practical insights on maintaining quality and velocity as AI reshapes software delivery.

Have someone you'd like to hear on the show? Reach out and let us know on X at @CircleCI!

Have someone you'd like to hear on the show? Reach out and let us know on X at @CircleCI!

The conversation explores the unique challenges of product engineering in media, from building cooking products for users across the entire spectrum of culinary expertise to navigating digital transformations that honor legacy while embracing an uncertain future. Meg discusses the importance of mission-driven leadership through technological transitions, why she prioritizes helping her engineers stay valuable in the marketplace, and her unconventional path from elementary education to retail management to engineering leadership. Whether you're an engineering leader managing AI adoption or someone thinking about the intersection of technology and human-centered products, this episode delivers insights on building resilient teams that can adapt while maintaining their core purpose.

Have someone you'd like to hear on the show? Reach out and let us know on X at @CircleCI!

The conversation explores the parallels between building tech products and sports organizations, from using merchandise as a "landing page" to prove market demand, to the critical importance of obsessing over customer-facing touchpoints while maintaining operational flexibility behind the scenes. Edreece discusses the challenges of scaling while preserving authentic brand identity, why purpose should lead to profit (not the other way around), and how hiring good humans with strong moral compasses becomes the foundation for sustainable growth. Whether you're a startup founder looking to build authentic community engagement or a leader working to maintain organizational purpose at scale, this episode delivers actionable insights on creating brands that people proudly represent in their daily lives.

Have someone you'd like to hear on the show? Reach out and let us know on X at @CircleCI!

The conversation explores how to shift engineering teams from a "ticket factory" mindset to outcome-focused delivery, the tension between servant leadership and strategic leadership, and why the best engineering cultures encourage productive disagreement. Whether you're an individual contributor looking to take ownership of your trajectory or a leader working to create environments where people thrive, this episode delivers actionable wisdom for engineering professionals at every level.

Have someone you’d like to hear on the show? Reach out and let us know on X at @CircleCI!

Pat reveals how today's engineering leaders are navigating unprecedented industry turbulence: from AI disruption and talent shifts to economic uncertainty and constant organizational change. Drawing from his 20+ years coaching CTOs, VPs of Engineering, and emerging tech leads, he shares practical strategies for making decisions without perfect information and building resilient, adaptable teams.

Key insights include:

• Why embracing uncertainty is a defining leadership skill, not a weakness

• How to apply agile principles beyond code—from team structures to strategic decisions

• The critical importance of creating "forcing functions" that drive continuous learning

• Why exposure to high-performing teams transforms your entire worldview

• Practical approaches for leading through AI adoption and industry shifts

From small startups to global enterprises, Pat's guidance helps leaders focus on what they can control while building systems that thrive in constant change. Whether you're a first-time manager or seasoned executive, discover how to turn uncertainty into competitive advantage.

Have someone in mind you'd like to hear on the show? Let us know on X at @CircleCI!

Corey challenges the myth that creativity is just for artists, revealing how engineering workflows are fundamentally creative processes. Her "Ideas, Not Art" framework shows how visual communication, coaching techniques, and strategic problem-solving help engineers and leaders thrive. From sketch notes to systems thinking, discover practical approaches for building stronger teams and accelerating personal growth in engineering.

Have someone in mind you’d like to hear on the show? Let us know on X at @CircleCI!

Hernandez shares insights from scaling MongoDB's proprietary CI system—processing 10 engineer years of compute daily—and reveals how organizations can evaluate when custom infrastructure drives competitive advantage versus when strategic partnerships accelerate growth. Her perspective on navigating the evolving landscape of CI/CD tooling offers actionable guidance for engineering leaders balancing innovation with operational efficiency.

Have someone in mind you'd like to hear on the show? Reach out to us on X at @CircleCI!

Brian challenges conventional CI/CD wisdom, arguing that asynchronous pull request processes create hidden context transfer costs that cripple time-to-market. His "Move Faster Manifesto" reveals how continuous integration was originally designed for speed, the real cost of feature branching versus trunk-based development, and why reducing integration problems translates directly to competitive advantage.

Have someone you'd like to hear on the show? Let us know on X at @CircleCI!