Research Corner: What the Studies Show

Why Practitioners Should Read the Research

Your information diet about AI coding tools is, by default, systematically biased. Vendor demos are selected from many takes. Conference talks feature the project that worked, not the three that were quietly abandoned. Social media rewards the "I built an app in 45 minutes" post and buries the "I spent two days untangling what the agent did to my auth layer" post. None of these sources are lying, exactly — they are just drawn from the extreme right tail of a distribution you never get to see.

Controlled studies are the corrective. They measure the whole distribution: median outcomes, not highlight reels; time actually elapsed, not time as remembered; defects that shipped, not demos that dazzled. Reading them does three things for you as a working engineer:

• It calibrates your expectations. When you know what effect sizes look like under measurement, you stop being whipsawed between "this changes everything" and "this is all hype" — both of which are wrong in instructive ways.
• It tells you where the payoff lives. The studies do not say "AI helps" or "AI hurts." They say AI helps here and hurts there, and the boundary between here and there is precisely the thing you need to know to deploy your own effort well.
• It protects you from your own perception. The single most repeated finding in this literature is that developers' felt experience of speedup is a poor guide to their measured speed. You cannot introspect your way to the truth here. You need external measurement, and the studies are where that measurement lives.

Here is the punchline of this entire module, stated up front so you can check everything that follows against it: the research consistently finds that AI-assisted development pays off only under discipline — clear specifications, small verifiable steps, real review, honest measurement. Undisciplined adoption shows up in the data as churn, instability, and confidently-felt slowdowns. That is not a coincidence. It is the reason the rest of this board exists.

The Productivity Evidence: A Speedup and a Slowdown, Both Real

The productivity literature contains two headline results that appear, at first glance, to flatly contradict each other. Holding both in your head at once — and understanding why they do not actually conflict — is the core calibration skill this module wants to leave you with.

Result one: the controlled-task speedups

The early controlled experiments on AI code assistants found large, real speedups. The best known is the GitHub Copilot controlled experiment (Peng and colleagues, published in 2023), in which developers were recruited and randomly assigned to complete the same task — implementing an HTTP server in JavaScript — with or without Copilot. The assisted group finished dramatically faster: the reported effect was on the order of completing the task in roughly half the time of the control group. Other vendor and academic studies in the same period reported speedups in a similar direction, if not always the same magnitude, on comparable well-specified tasks.

Look closely at the design, because the design is what the result generalizes to. The task was greenfield (no existing codebase to understand), well-specified(a standard problem with a clear definition of done), self-contained (no cross-repo conventions, no tribal knowledge, no legacy constraints), and short (completable in a sitting). Under those conditions, an AI assistant is operating at maximum advantage: the entire problem fits in its context, the specification is crisp, and the developer's own familiarity advantage is near zero because there is nothing to be familiar with.

Result two: the METR slowdown

In 2025, the research organization METR published a randomized controlled trial that measured something much closer to real professional work — and got the opposite sign. Experienced open-source developers worked on real issues in their own mature repositories, codebases they had often maintained for years. Tasks were randomly assigned to allow or disallow AI tooling, and completion times were measured.

The result: with AI assistance allowed, developers took roughly 19% longer to complete their tasks. And the finding that made the study famous is not the slowdown itself but the perception gap wrapped around it: participants expected AI to speed them up substantially before the study, and — this is the remarkable part — after finishing, they still believed it had sped them up, even as the clock said otherwise. The felt experience of AI assistance and its measured effect pointed in opposite directions.

Reconciling the two: what flips the sign

Neither study is wrong. They measured different situations, and the differences between those situations are the variables that flip the sign of the effect:

Read as a pair, the studies sketch a coherent picture: the payoff of AI assistance is largest where your own context is thinnest, and smallest — possibly negative — where you are already the expert. On greenfield work, boilerplate, unfamiliar languages, or well-specified standalone tasks, the model's general knowledge is a genuine addition to yours. On a mature codebase you maintain, your head already holds most of what matters; the model's contribution shrinks while its costs — prompting time, reviewing plausible-but-wrong suggestions, correcting violations of conventions it cannot see — stay fixed. When the costs exceed the shrunken contribution, the sign flips.

Notice this is not an argument against using agents on mature codebases — it is an argument for exactly what the earlier modules taught. The METR participants were largely using early-2025 tools with their own ad-hoc workflows on tasks where their expertise was maximal. The costs that ate the speedup — vague prompts, oversized diffs, review of unfamiliar generated code — are precisely the costs that specification discipline, chunking, and durable context files are designed to shrink. The study measures what happens without that discipline; it does not measure its ceiling.

The Quality Trendlines: Faster and Sloppier, at Scale

Speed is only half the ledger. The other half is what the code looks like afterward — and here the large-scale observational evidence tells a consistent, uncomfortable story about what AI-era code looks like in the absence of discipline.

GitClear: churn up, refactoring down

GitClear, an analytics company that studies large volumes of commit data, has published a series of repository analyses covering the years when AI assistance went mainstream. The reported trendlines, across hundreds of millions of changed lines: code churn rising (code committed and then rewritten or reverted shortly after — a proxy for "shipped before it was right"), copy-paste and duplicated code rising (blocks of repeated code growing as a share of changes), and refactoring falling (specifically "moved" code, the signature of consolidating existing logic, declining as a share of activity). The exact percentages vary by report year and should be held loosely — this is observational data, not a controlled experiment — but the direction is consistent across their reports: more code, written faster, duplicated more, consolidated less.

The mechanism is not mysterious. An assistant will happily generate a fresh implementation of something that already exists two directories away, because it does not know the codebase has one and the developer did not check. Generating is cheaper than searching; accepting is cheaper than refactoring. Every default points toward duplication, and the aggregate data shows the defaults winning.

DORA: shipping more, breaking more

The 2024 DORA report (the long-running State of DevOps research program, which surveys tens of thousands of practitioners and correlates practices with delivery outcomes) examined AI adoption directly. The finding was mixed in a specific and telling way: AI adoption was associated with gains on several individual and flow measures — but with a decrease in software delivery stability, and the report found delivery throughput did not improve either. Teams adopting AI were, in aggregate, not shipping meaningfully faster end-to-end, and what they shipped broke more.

A plausible reading, and the one the DORA authors themselves gesture at: AI accelerates the writing of code, which produces larger changes moving through delivery pipelines built for smaller ones. Larger changesets are the best-established predictor of delivery instability in the entire DORA research tradition — and AI makes large changesets effortless to produce. Speed at the keyboard became risk at the release.

That is the right way to metabolize the quality evidence. It is not a verdict that AI-assisted code must be worse. It is a measurement of the default trajectory — the one you get for free, without trying. The earlier modules on reviewing AI code and working in small verified chunks are, in effect, the countermeasures to exactly these trendlines. The studies tell you what happens to teams that skip them.

Reading Capability Benchmarks Sanely

Every model release arrives wrapped in benchmark numbers, and the one you will see most often for coding is SWE-bench. Knowing what it measures — and what a score on it does and does not predict about your work — is basic literacy for anyone working with these tools.

What SWE-bench actually is

SWE-bench, introduced by academic researchers in late 2023, is built from real GitHub issues in popular open-source Python repositories. The model is given the issue and the repository and must produce a patch; the patch is scored by running the repository's own tests. This was a genuine step forward over earlier coding benchmarks, which were mostly self-contained puzzle-style problems: SWE-bench requires navigating a real codebase to fix a real reported problem. A widely used subset, SWE-bench Verified, was later human-reviewed to filter out ill-specified or broken tasks, and became the field's reference number for agentic coding capability.

The trajectory on it has been steep. When the benchmark launched, the best systems resolved only a tiny fraction of tasks — low single-digit percentages. Within roughly two years, frontier models with agentic scaffolding were reported resolving well over half of the Verified set, with scores continuing to climb. That climb is real information: the models are genuinely, rapidly getting better at self-contained repository-level bug fixing.

What the score does — and does not — tell you

Two caveats deserve emphasis. First, the benchmark's tasks skew toward a particular shape of work — Python, popular open-source repos, issues resolvable by a patch that existing tests can verify. The distance between that distribution and your daily work is the distance the score has to travel before it predicts anything for you. Second, and more fundamental: a benchmark reports whether a task was solved, but production cares about whether a task is solved every time. A step that succeeds 90% of the time looks impressive in isolation and is a disaster inside a twenty-step chain — this is the compounding-error doctrine from the Agentic Theory board, and it is why a rising benchmark score does not by itself retire the review disciplines this board teaches. Capability climbing is real. Capability climbing is not the same as reliability arriving.

What the Labs Themselves Tell You: The Practitioner Literature

Alongside the academic studies sits a practitioner literature written by the AI labs about how to actually use these systems — and it is worth reading precisely because of what it does not say. The labs, who have every commercial incentive to tell you their models can be trusted with everything, consistently publish guidance that is strikingly conservative.

Anthropic's published guidance on building effective agents argues for simple, composable patterns over heavyweight frameworks: start with a plain augmented model call, add structure only when the task demonstrably needs it, and prefer predictable workflows (fixed sequences of model calls with checks between them) over open-ended autonomous agents wherever the problem allows. Its Claude Code best-practices material lands on the same doctrine at the workflow level: have the model plan before it executes and review the plan; maintain context files that carry project conventions across sessions; work in small, verifiable steps with tests or other checks closing each loop; and keep a human checkpoint in agentic loops at the points where errors would be expensive to unwind.

Now put that next to everything else in this module. The controlled studies say AI pays off where specifications are crisp and context is supplied. The METR trial says undisciplined use on expert territory can cost you time while feeling like a gain. The GitClear and DORA trendlines say scale-adoption without review and chunking discipline produces churn and instability. And the labs' own guidance says: plan first, keep steps small, supply durable context, review everything, checkpoint the loop.

Three independent lines of evidence — academic measurement, industry telemetry, and lab practice guides — keep converging on the same doctrine. That convergence is itself evidence. When the people selling the tools, the people measuring the tools, and the people mining the commit data all arrive at the same short list of disciplines, the reasonable conclusion is that the disciplines are load-bearing. That short list is this board's curriculum. It was not invented here; it was converged upon everywhere.

Staying Calibrated in a Field That Moves Monthly

Everything above has a shelf life. Models improve, tooling matures, and a finding about early-2025 systems may not describe late-2026 systems. That does not mean throwing up your hands and believing the loudest recent claim — it means having a method for updating. Four habits keep you calibrated without drowning:

• Read the study, not the headline. Nearly every result in this module has circulated in a distorted headline form — "AI makes developers 19% slower" with no mention of mature-repo experts, "55% faster" with no mention of greenfield tasks. The methods section is where a result's generalization limits live, and it is usually two paragraphs long. Reading it is the difference between owning a finding and repeating a rumor.
• Hold surprising results loosely until replicated.A single study — including the ones in this module — is one sample from a noisy process. The right response to a surprising result is neither dismissal nor conversion; it is a provisional update and an eye out for replication. Fields move on accumulations, not announcements.
• Run your own experiments. Your personal effect size beats any study's average, because you are not the average: your codebase, your stack, and your workflow are a specific point in the space the studies average over. The experiment is cheap: pick comparable recurring tasks, time yourself with and without agent assistance, and count the rework. A month of honest timestamps on your own work tells you more about your sign and magnitude than any paper can — and given the perception gap, the timer is not optional. Feel is exactly the instrument the METR study showed to be broken.
• Share what you find internally. A finding that stays in your head helps one engineer; posted to a public channel or captured in session notes, it compounds. "Agent work on the billing service keeps failing on our custom ORM — supplying the ORM conventions doc up front fixed it" is a research finding, locally scoped and immediately actionable, and it is worth more to your team than most published papers. This is the same session-notes and work-in-public culture from earlier boards, applied to evidence.

The Honest Summary

The controlled greenfield studies and the METR trial are both right: AI assistance produces large speedups where your context is thin and the task is crisp, and can produce measured slowdowns — wrapped in a felt speedup — where you are already the expert on a mature codebase. The quality trendlines from GitClear and DORA show what scale adoption looks like without review and chunking discipline: churn, duplication, and delivery instability. Benchmarks like SWE-bench show real, steep capability gains while measuring neither your codebase nor reliability. And the labs' own guidance converges on the same doctrine as the independent evidence: plan first, small verifiable steps, durable context, review everything, humans at the checkpoints. The research does not say AI coding works or does not work. It says AI coding pays off under discipline — and then it hands you the discipline. That is what this board has been for.

Knowledge Check

Five scenarios. Each one tests whether you can reason from the evidence — including the evidence that cuts against the hype — rather than from the headlines.