AI Writes Better Code and Makes Bigger Mistakes

For years, the easiest way to distrust AI-generated code was to run it.

The import did not exist. The loop stopped one iteration early. The model invented an API, confused two types, or returned something that failed the first unit test. You did not need an architecture review to find the problem. The compiler did it for you.

That version of AI coding is fading.

On August 11, 2026, Boris Cherny, the creator and lead of Claude Code, described the change bluntly: "LLMs still produce bugs, but those bugs are different than what they used to be. It's less off-by-ones and more about system design, UI usability, missing broader context."[1]

The quote caught my attention because it puts words to something many developers have started to feel. The code looks better. It often compiles. It may pass the tests the agent wrote for itself. Yet the change can still be wrong in ways that are harder to notice and more expensive to repair.

The agent solved the function and missed the system.

Cherny's observation is not, by itself, scientific proof that the distribution of AI defects has changed. It is a practitioner statement from someone building one of the most widely used coding agents. But recent research points in the same direction. Stronger agents are getting good at producing valid patches. Their performance drops when they must interpret incomplete requirements, coordinate changes across many files, respect existing abstractions, preserve behavior, and finish a feature from end to end.

The next AI coding problem is not simply whether the model can write code. It is whether the model understands what the code is supposed to mean inside a living system.

A valid patch can still be a failed change

SWE-EVO is one of the clearest demonstrations of this gap. The benchmark contains 48 release-sized software evolution tasks drawn from seven mature Python projects. The expected changes touch 21 files on average, and each instance has roughly 874 tests.[2]

Several evaluated models produced patches that applied successfully between 97.92% and 100% of the time. That sounds impressive until you compare it with actual task resolution. The best reported model completed only 25% of the tasks. GPT-5.2, for example, scored 72.8% on SWE-bench Verified but only 22.92% on SWE-EVO.[2]

This is the difference between patch mechanics and engineering correctness.

The agent knew how to edit files. Git could apply the output. The syntax was usually acceptable. Most of the releases were still wrong.

The failure analysis is even more revealing. According to the researchers, weaker models continued to struggle with syntax and tool use, while stronger models more often misinterpreted nuanced release notes.[2] That is about as close as current benchmark evidence gets to Cherny's claim. The stronger model is no longer stopped primarily by punctuation or a malformed command. It is stopped by meaning.

FeatureBench finds a similar pattern at a larger feature scope. Its 200 tasks average 15.7 changed files, 29.2 changed functions, and about 790 changed lines. Claude Opus 4.5 reportedly achieved 74.4% on SWE-bench, but resolved only 11% of FeatureBench tasks. Other strong systems made partial progress while resolving none of the evaluated tasks completely.[3]

Partial progress is useful, but it can be deceptive. A feature that is 80% implemented is not necessarily 80% valuable. The missing 20% may contain the authorization rule, data migration, rollback path, accessibility behavior, or compatibility guarantee that makes the feature safe to ship.

Passing the test is not the same as satisfying the requirement

Software teams have always known that tests are incomplete. AI makes that old lesson easier to forget because an agent can generate the implementation, the tests, and the confident summary saying everything passed.

An ICSE 2026 study examined patches that SWE-bench's validation system counted as successful. The researchers found that 7.8% of plausible patches failed a broader developer-written test suite. They also found that 29.6% behaved differently from the human patch. In a manually inspected sample of divergent patches, 28.6% were certainly incorrect.[4]

The common problems were not missing semicolons. Many patches used a similar but behaviorally different implementation, changed more behavior than requested, or interpreted an underspecified issue incorrectly.[4]

This exposes an uncomfortable weakness in agent workflows: the evaluator may share the agent's misunderstanding.

Suppose the request says, "Prevent users from editing archived projects." The agent adds a disabled button and a browser test confirming that the button cannot be clicked. Every generated test passes. But the API still accepts the update, the mobile client still exposes the action, and an old background job can still change the record.

The local behavior is correct. The system rule is not.

No amount of celebrating a green test suite fixes a test suite that describes the wrong boundary.

Repositories have a design, even when nobody wrote it down

A mature codebase contains thousands of decisions that may never appear in the issue description. Where does validation belong? Which service owns the data? Which abstraction should a new feature extend? Which dependency is approved? What must remain backward compatible? Which failures should be retried, surfaced, or ignored?

Developers absorb these constraints slowly. Coding agents receive a prompt, a context window, and whatever files their retrieval system happens to select.

RepoExec was designed to measure this problem. It evaluates whether generated code runs, whether it behaves correctly, and whether it uses the repository's existing dependencies. Researchers found that pretrained models often produced runnable, correct code by reimplementing capabilities that already existed elsewhere in the project. Instruction-tuned models used existing dependencies more often, but sometimes introduced unnecessary complexity.[6]

In other words, the code can pass while still being the wrong contribution to the repository.

A duplicate implementation creates two places to fix the next bug. A bypassed abstraction weakens future refactors. A new dependency can expand the attack surface or conflict with the project's release policy. None of these problems has to fail today's test.

Other repository benchmarks tell the same story. FEA-Bench requires agents to generate new components while editing related existing components, and evaluated models performed substantially worse than they did in more local code-generation settings.[7] RepoCod contains 980 whole-function tasks from 11 large Python projects, with more than half requiring repository-level context. No evaluated model exceeded 30% pass@1.[8]

The hard part is no longer always writing the body of the function. It is discovering which function should exist, where it belongs, what it may depend on, and which behavior it must not disturb.

Long-running agents can compound small misunderstandings

Autonomous agents make this problem larger because their outputs are not limited to suggestions. They search, edit, run commands, install dependencies, rewrite tests, and decide what to do next.

Anthropic's own engineering guidance warns that autonomous agents carry "the potential for compounding errors" and recommends extensive testing in sandboxed environments. The company also makes an important distinction: automated tests can verify functionality, but human review remains necessary to determine whether a solution matches broader system requirements.[9]

That distinction should sit above every coding-agent dashboard.

Anthropic's work on long-running agent harnesses documents another class of failure. Agents may attempt too much at once, exhaust their context in the middle of an implementation, leave features half finished and undocumented, or declare victory too early.[10]

These are project-state failures. Each individual edit may be reasonable, but the sequence loses continuity. The next session then has to infer what the previous session intended, often from a working tree that no longer matches the plan.

Humans make continuity mistakes too. The difference is speed and scale. An autonomous agent can produce a large amount of coherent-looking state before anyone notices that its original assumption was wrong.

Security mistakes move with authority

A bad code suggestion is one risk. An agent with repository access, a terminal, deployment credentials, and a production database is a different category of risk.

OWASP calls this "excessive agency": damaging actions become possible when an LLM has too much functionality, permission, or autonomy. The triggering output can come from hallucination, ambiguous instructions, poor performance, or prompt injection.[14]

This is not a model-only problem. It is a system-design problem.

If an agent can delete production data because it misunderstood a request, the failure began before the model acted. The surrounding platform gave a probabilistic component an irreversible capability without a sufficient approval boundary. Better prompting may reduce the chance of failure. Environment separation and least privilege reduce the impact.

Security research also shows how polished AI output can distort human judgment. In a controlled study using a Codex-based assistant, participants with AI access wrote significantly less-secure code than participants without it. The assisted group was also more likely to believe its code was secure. Participants who trusted the assistant less produced fewer vulnerabilities.[13]

That combination is dangerous: plausible output, misplaced confidence, and a security property that ordinary functional tests may never check.

The riskiest AI-generated code may not look broken. It may look finished.

Local productivity can hide a system-level bill

None of this means AI coding tools are useless or inherently harmful. On bounded tasks with a clear specification and aligned tests, they can perform very well.

GitHub's controlled study of 202 experienced developers found that Copilot users were 53.2% more likely to pass all ten unit tests in a predefined API task. Blind reviewers also gave the assisted code modestly better scores for readability, reliability, maintainability, and conciseness.[15]

That is real evidence in AI's favor. It also shows why task boundaries matter. The experiment supplied a contained assignment and a visible evaluator. It did not ask the model to choose a service boundary, preserve years of undocumented behavior, migrate production data, or decide whether the feature should exist.

METR found the opposite productivity result in a different setting. Experienced open-source maintainers working in repositories they knew well took 19% longer with early-2025 AI tools, even though they believed AI had made them faster. The researchers pointed to mature projects' implicit requirements, quality standards, and context as possible contributors.[11]

That result should not be frozen into a timeless slogan. METR has since reported weak evidence that newer tools may provide speedups, and the original study covered a small group of expert maintainers using early-2025 systems. Still, it demonstrates a basic point: faster code generation does not guarantee faster engineering.

Google's 2024 DORA research found the same tension at the organizational level. Greater AI adoption was associated with better documentation quality, code quality, and review speed, but also with lower delivery throughput and stability. DORA cautioned that improving development activity does not automatically improve software delivery without small batches and robust testing.[12]

A team can generate more code, review individual changes faster, and still create a less stable delivery system.

The developer's job is moving up the stack too

If AI coding failures are moving upward, human responsibility has to move upward with them.

That does not mean every developer becomes a diagram-producing "architect." It means the valuable work increasingly happens before and after code generation:

• Turn vague requests into explicit behavior and invariants.
• Decide where a change belongs and which boundaries it must respect.
• Give agents access only to the context and capabilities they need.
• Separate development, staging, and production authority.
• Write independent tests that challenge the implementation rather than repeat it.
• Review the blast radius, not only the diff.
• Verify failure paths, migrations, rollback behavior, security rules, and operational impact.

The 2025 Stack Overflow survey reflects this caution. More developers distrusted AI-tool accuracy than trusted it, and experienced developers were the most skeptical. Respondents said they would still seek human help when they did not trust an answer, faced security concerns, or needed to understand complex code.[16]

This is not resistance to progress. It is what accountability looks like when the tool can produce more than the reviewer can casually inspect.

We need stronger definitions of "correct"

The software industry has spent years measuring coding models with exact-match scores, isolated functions, unit tests, and issue-resolution rates. Those measures helped models improve. They are no longer enough.

A serious evaluation of an AI coding agent should ask:

• Did the change satisfy the user's actual requirement?
• Did it preserve behavior outside the new feature?
• Did it use the repository's intended abstractions?
• Did it introduce unnecessary dependencies or duplicate logic?
• Did it respect authorization, privacy, and data-ownership boundaries?
• Can the team operate, observe, migrate, and roll back the change?
• Did the tests challenge the solution independently?
• Can another developer understand what happened and why?

A patch that applies is not necessarily correct. A test that passes is not necessarily meaningful. A feature that works in the happy path is not necessarily ready.

AI has not eliminated software bugs. It has started changing where we have to look for them.

The compiler will still catch the missing bracket. The harder question is who catches the beautifully implemented solution to the wrong problem.

Sources

[1] https://x.com/bcherny/status/2087284684103537011 — Boris Cherny: LLM coding bugs are changing [2] https://arxiv.org/abs/2512.18470 — SWE-EVO: Benchmarking Coding Agents in Long-Horizon Software Evolution Scenarios [3] https://arxiv.org/abs/2602.10975 — FeatureBench: Benchmarking Agentic Coding for Complex Feature Development [4] https://arxiv.org/abs/2503.15223 — Are Solved Issues in SWE-bench Really Solved Correctly? [6] https://aclanthology.org/2025.findings-naacl.82 — RepoExec: Impacts of Contexts on Repository-Level Code Generation [7] https://aclanthology.org/2025.acl-long.839 — FEA-Bench: Repository-Level Feature Implementation [8] https://aclanthology.org/2025.acl-long.1204 — RepoCod: Can Language Models Replace Programmers for Coding? [9] https://www.anthropic.com/engineering/building-effective-agents — Anthropic: Building effective agents [10] https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents — Anthropic: Effective harnesses for long-running agents [11] https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study — METR: AI impact on experienced open-source developers [12] https://cloud.google.com/blog/products/devops-sre/announcing-the-2024-dora-report — Google Cloud: 2024 DORA report [13] https://arxiv.org/abs/2211.03622 — Do Users Write More Insecure Code with AI Assistants? [14] https://genai.owasp.org/llmrisk/llm062025-excessive-agency — OWASP LLM06:2025 Excessive Agency [15] https://github.blog/news-insights/research/does-github-copilot-improve-code-quality-heres-what-the-data-says — GitHub Copilot code quality study [16] https://survey.stackoverflow.co/2025/ai — Stack Overflow 2025 Developer Survey: AI