Developers Felt 20% Faster With AI. Timed, They Were 19% Slower.
People trusted the feeling. The stopwatch told a different story.
Every owner has heard the same sentence from a team that has just been given an AI tool: this is much faster. METR decided to time it. The result is uncomfortable, useful, and exactly the kind of evidence a small firm should want before it prices a job, promises a deadline, or cuts a role.
METR is a non-profit research group that studies what AI systems can actually do. On 10 July 2025 it published a randomised controlled trial of experienced open-source developers. Sixteen developers, each working in large repositories they knew well, supplied 246 real issues — bug fixes, features and refactors that were part of their normal work. Each issue was randomly assigned to allow AI or to forbid it. When AI was allowed, developers mainly used Cursor Pro with Claude 3.5 and 3.7 Sonnet, the frontier tools of early 2025. Tasks averaged about two hours. Developers recorded their screens and reported the time taken. The study is on arXiv as 2507.09089 and summarised on METR's blog.
The stopwatch disagreed with the feeling
When developers were allowed to use AI, they took 19% longer to finish an issue than when they worked without it. Before the tasks, they had forecast that AI would make them 24% faster. After living through the slowdown, they still believed AI had made them about 20% faster. That gap — felt speed against timed speed — is the whole lesson. People trusted the feeling. The stopwatch told a different story.
METR checked the obvious explanations. Developers used frontier models, followed their assigned condition, did not quietly drop the hard AI-free tasks, and submitted pull requests of similar quality in both conditions. The slowdown survived different ways of measuring and slicing the data. Five contributing factors looked likely in their analysis of twenty possibilities, including time spent prompting, reviewing and correcting plausible-looking output. In plain terms, attention went into supervising suggestions instead of doing the work. When a tool absorbs routine typing but taxes judgment on every line, the budget that matters is attention, and it was being spent in the wrong place.
What this does not prove
METR is unusually clear about the limits, and a fair summary has to carry them. The study does not show that AI fails to speed up most developers, or people in other kinds of work, or the same developers with better methods. Sixteen experienced specialists in high-quality open-source codebases, with strict style, testing and documentation habits, are not your bookkeeper, your paralegal or a junior hire learning a new codebase. Tasks averaged twenty minutes to four hours. Settings where you can sample many attempts, accept a rough prototype, or learn over hundreds of hours may behave differently. The honest claim is narrow: in this setting, with early-2025 tools, felt speed was a poor guide to measured speed.
The February 2026 caveat — and why it strengthens the piece you are reading
In February 2026 METR published a follow-up and changed its own reading. Its later experiment, using late-2025 tools, produced raw estimates pointing the other way — about an 18% speedup for returning developers and 4% for newly recruited developers — but with confidence intervals crossing zero and, in METR's words, unreliable signal. Developers increasingly refused to work without AI or withheld the tasks they most expected AI to help with, which biases the sample. Timing also became hard to interpret when people ran several agents at once. METR concluded it is likely developers are more sped up in early 2026 than in early 2025, while stressing that its new data are only very weak evidence for how large that increase is. The original page now says plainly: these results are out of date. Any article that quotes 19% without that paragraph is selling you a headline, not evidence.
So why publish the older result at all? Because the durable finding is not a percentage. It is that self-reported speed can be wrong by a wide margin, in a positive direction, even among experts who just lived the task. Tools improve. That measurement trap does not retire with a model release. It is the same trap behind the wider productivity debate covered in the MIT article: pilots feel busy and modern while the finished, accepted work barely moves.
Price the finished task, not the licence
For a small firm the practical unit is cost per finished task — a completed, checked, billable piece of work — not cost per seat and not minutes that felt saved. Our guide to counting cost per finished job makes the same point, and the AI ROI page shows how to frame it before you buy. Pick one real workflow. Time ten examples without AI and ten with it, including review and rework. Count only work you would send to a client. If the tool wins, expand it. If it merely feels faster, you have learned something cheap.
That is symbiosis in operator language: let the machine absorb routine formulation so scarce attention stays on judgment — and prove, with a timer, that attention really was freed. The failure mode METR caught is the opposite bargain, where people keep the judgment and also inherit a new reviewing job.
Before you roll a tool out across a team, run that check on your own work. It takes an afternoon and it beats any vendor benchmark.
One more discipline from the trial design is worth stealing. METR assigned the tool by task, not by team mood on the day. Your version can be simpler — alternate weeks, or alternate the same recurring job between two people — but keep the comparison inside the same kind of work. Timing AI on easy tickets against no-AI on hard ones manufactures whatever result you wanted. Same task, same standard of done, tool on or off: that is the whole method, and an afternoon of it tells you more than a quarter of testimonials.
Give Ada one task your team swears AI speeds up. She will help you time it properly — finished work, review included — and build it into your free report. Start at /audit/ and talk to the agent. No paid audit, no invented saving: just a measured answer on one workflow before you spend on the rest.
Sources: METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, 10 July 2025; arXiv:2507.09089; METR, We are Changing our Developer Productivity Experiment Design, 24 February 2026. Figures are METR's; the February 2026 caveat is METR's own updated reading.
