AI has taken software program growth by storm. Between the 2 of us, we construct merchandise for software program engineers and client merchandise for thousands and thousands of on a regular basis customers, so we’ve got pores and skin within the sport. We would like the AI productiveness story to be true. Extra output, tighter timelines, happier and extra productive engineers. Who wouldn’t?

However after we have a look at the precise analysis after which have a look at what’s occurring in the actual world, we are able to’t make them agree. Or relatively we are able to, however provided that we’re prepared to confess that “productive” doesn’t imply what many of the latest discourse thinks it means.

Probably the most uncomfortable discovering first

In early 2025, a analysis group, METR, ran a managed experiment with open supply builders. They discovered that (in distinction of what the trade was anticipating) engineers utilizing AI instruments took 19% longer than these working with out them, with a confidence interval of +2% to +39%. The slowdown was statistically sturdy. This was a distinct time within the trade. Claude hadn’t launched its Opus fashions, the trade was determining what AI can and may’t do, however what makes this exceptional isn’t the slowdown, it’s that engineers believed they have been roughly 20% sooner whereas the information indicated in any other case, uncovering a big hole between notion and actuality.

Think about this discovering for a second earlier than we pile the remainder of the proof on prime of it as a result of it adjustments the way you learn all the things else.

METR tried a follow-up research beginning in August 2025, and what occurred to that research is arguably extra revealing than the unique outcome. In February 2026 they revealed a put up explaining why they deserted the experimental design. The issue was that too many builders refused to take part until they may use AI for all their duties. Between 30% and 50% of remaining members reported selectively avoiding submitting duties they didn’t wish to do with out AI. The pattern grew to become systematically biased towards the builders and duties least prone to present the worth of AI.

Knowledge from the late 2025 research exhibits an enchancment in traits. For the subset of unique builders who returned, the estimated impact shifted to an 18% enchancment in pace (confidence interval: -38% to +9%). Amongst newly recruited builders, there was a 4% enchancment in pace (-15% to +9%). However METR flagged these numbers as probably a decrease sure as a result of many individuals self-selected out. Their conclusion: AI instruments have gotten extra helpful since early 2025, however the choice results are actually so extreme that managed measurement is sort of unimaginable. The builders most obsessed with AI will not work with out it to function a management group. That’s not a failure of METR’s methodology. It’s a sign about the place we’re and the place we’re headed.

Three extra information factors

A number of extra research landed over the course of late 2025 and early 2026.

Anthropic surveyed 132 of its personal engineers in late 2025, carried out 53 interviews, and analyzed 200,000 Claude Code transcripts. Workers reported reaching a 50% productiveness increase. Because the engineering group and utilization of Claude grew, they claimed that pull requests per engineer per day have been up 67%. Anthropic engineers use Claude in 60% of each day work, and Claude performs extra duties autonomously.

CircleCI analyzed 28 million CI workflows throughout hundreds of groups. Workflow throughput was up 59%, however foremost department throughput for the median workforce declined 7%. Construct success charges fell to 70.8%, which is a five-year low. Extra code exists than ever, however much less of it reaches manufacturing, and the CI is turning into a chokepoint.

Harvard Enterprise College researchers studied 78 employees utilizing synthetic intelligence to carry out duties exterior their experience. AI helped everybody brainstorm equally properly, however on execution, employees whose abilities have been removed from the area underperformed area specialists by 13%. The hole that AI appeared to shut in planning reemerged in supply.

METR’s Might 2026 survey of 349 technical employees—which was carried out after the experimental design broke down—discovered self-reported productiveness worth features of 1.4x to 2x from synthetic intelligence instruments. However METR’s personal analysis employees, the individuals most calibrated on the notion bias they documented in 2025, reported the bottom features of any subgroup in that survey.

What this appears to be like like in follow

Right here’s a state of affairs that can really feel acquainted to some readers: Engineer exercise metrics look nice on the floor. Pull requests are rising, code commits are up, velocity factors are being closed at a tempo the workforce hasn’t hit in years. The management workforce is completely happy, engineers really feel extra productive. Then somebody—probably a PM—asks why the roadmap objects marked “in progress” six weeks in the past are nonetheless in progress.

Everybody involves the identical realization : The characteristic timelines haven’t actually modified. What’s occurred is that AI has dramatically diminished the price of beginning work, however production-ready polish stays a problem. First draft features, boilerplate, scaffolding, and check writing explanations for unfamiliar code have all gotten considerably cheaper. However the bottlenecks on transport have been by no means these duties. They have been product choices, design critiques, QA, compliance, infrastructure, launch processes. If you pace up coding, you find yourself jamming extra work-in-progress objects in opposition to the identical downstream chokepoints. The CircleCI information on 28 million workflows is, partially, an image of what that appears like at scale: large exercise in characteristic branches with flat or declining throughput on foremost.

This isn’t only a sample in combination information. As Fiona Fung, a director of engineering for Claude Code at Anthropic, defined at a June 2026 speak, writing code, writing exams, and refactoring hardly ever slows her workforce down anymore, however the bottlenecks didn’t disappear. Verification, code evaluation, and safety took their place. She flagged CI particularly. As groups generate extra code, construct programs and CI pipelines can battle to maintain up. That’s a workforce operating probably the most AI-accelerated engineering orgs on this planet hitting the identical constraint wall the CircleCI information describes. The ceiling isn’t code authoring pace anymore; it truly by no means was.

Anthropic’s discovering that 27% of AI-assisted work wouldn’t have occurred in any other case cuts each methods. A few of that work is genuinely beneficial, like prototype explorations that inform actual choices, documentation that really will get written. A few of it’s work no person prioritized as a result of it merely wasn’t vital sufficient. Now it’s burning evaluation cycles and CI assets as a result of constructing it grew to become almost free, whereas reviewing, testing, and sustaining it didn’t.

The competence-confidence hole

The HBS research identifies a selected mechanism: AI closes the confidence hole between novices and specialists. It provides everybody equal entry to plans, explanations, and first drafts. But it surely doesn’t shut the competence hole. When a backend engineer builds a frontend characteristic with AI help, they produce one thing that appears proper. The issues are beneath, within the choices they didn’t know to query and the sting instances they didn’t know to check.

The early METR outcome suggests this extends even to skilled practitioners working in their very own domains. The AI doesn’t make them incompetent; it truly makes them really feel extra succesful than their output justifies. And as METR’s follow-up collapse demonstrated, as soon as builders combine AI deeply sufficient, they lose the power to work with out it as a reference level in what researchers have known as automation bias.

That is the half that ought to concern engineering leaders. You may’t repair what you possibly can’t see. If each engineer in your workforce sincerely believes they’re 50% extra productive and your ship dates haven’t moved, there’s an issue that no person thinks exists.

What makes synthetic intelligence native growth sustainable

Make code evaluation extra rigorous, not sooner. AI-generated code passes floor checks simply—clear formatting, constant conventions, no linter complaints, and many others.—which is strictly why it’s harmful. The issues are the sort a reviewer gained’t catch from skimming a diff.

I’ve been calling this “affordable doubt evaluation.” The follow is to start out from skepticism relatively than belief, asking, “What might be flawed right here that I wouldn’t catch from the diff?” Particularly, what assumptions did the mannequin make that aren’t seen within the output? What edge instances does this silently fail on? The place does this couple to one thing the creator may not have been enthusiastic about?

That is slower. That’s the purpose. It’s additionally not infinitely scalable, which is why it must be paired with automation on the issues that don’t require judgment and human consideration targeting the place it does.

The Claude Code workforce’s method is an effective instance: Let AI deal with type, linting, bug-catching, and check era as a primary move, however route security-sensitive code, belief boundaries, and something touching authorized threat on to area specialists. The division isn’t “AI critiques smaller, low-risk adjustments and people evaluation greater, higher-risk adjustments.” It’s “AI handles floor correctness, people personal consequential judgment.” That’s a significant distinction. Quite a lot of groups are doing the primary whereas pondering they’re doing the second.

Adapt your CI to the brand new failure modes. CircleCI’s construct success price hitting a five-year low whereas throughput exploded suggests most groups haven’t up to date their pipelines to catch how AI-generated code breaks. AI-generated code fails otherwise than human-generated code. It’s extra prone to be domestically right however architecturally inconsistent, move unit exams and fail integration exams, and respect operate signatures whereas violating the assumptions that these features have been constructed round. Integration exams, contract exams, and structure health features that implement your system’s constraints within the pipeline will catch extra of this than a linter or a sort checker. If AI-generated code violates your patterns, the construct ought to catch it earlier than a reviewer opens the diff. This addresses what is going to turn into your evaluation drawback and your infrastructure drawback.

Ship behind characteristic flags and monitor aggressively. Settle for that you’ll not catch all the things earlier than deployment. As an alternative of betting totally on premerge high quality—which the proof suggests is more durable to evaluate than it feels—deploy to 1% of customers, watch the dashboards, and roll again quick when one thing’s flawed. This method additionally forces funding in observability, which pays for itself independently of the AI query.

Require human-written exams for AI-assisted code (till AI can confidently generate deterministic exams). Human-written exams, particularly for edge instances and boundary circumstances. The self-discipline of writing the check forces the developer to assume by means of the habits relatively than settle for the output at face worth. If an engineer can’t write the check, they most likely don’t perceive the code properly sufficient to ship it. That’s a helpful sign, not a failure state.

Shield deliberate knowledge-sharing time. The Anthropic research discovered that mentorship was quietly eroding as Claude changed the conversations engineers used to have with one another. That is the long-horizon threat within the information. Structure choice information, rotating system walkthroughs, and pairing classes the place a senior and junior work by means of an issue collectively really feel inefficient subsequent to asking an AI, however they’re how groups construct the shared understanding that forestalls the identical errors from being rebuilt in better-formatted code each six months.

The measurement drawback

So does this imply we cease utilizing AI? No. Use AI and use it aggressively the place it clearly helps tedious duties, prototyping, and exploratory work, something you possibly can confirm shortly. The features on well-scoped, independently verifiable work are actual.

However should you’re attempting to measure whether or not AI is definitely serving to your workforce ship, PR rely and self-reported velocity are the flawed devices. The 4 research we evaluated taken collectively point out that these aren’t simply measurement issues, they’re a warning signal that the suggestions loops we’d usually depend on to detect whether or not one thing is working have modified considerably.

The more durable query—the one that every one the analysis research increase with out fairly answering—is what the measurement would truly inform you. Cycle time from characteristic conception to supply, or the speed at which merged code reaches manufacturing with out rollback, is perhaps higher metrics. Or the hole between deliberate and precise scope on the finish of a dash. Or perhaps a bit extra abstracted: firm income progress correlated with the AI funding (tooling, infrastructure, and OpEx).

None of those are simple to instrument. The query you ought to be asking of your groups isn’t “How productive can we really feel?” It’s “What would we have to measure to know?”

Word: The analysis work pertaining to this text was achieved in a private capability. Views are our personal and don’t replicate the views of our employers in any approach.