How Anthropic's Own Engineers Rate AI Coding Ability in 2026
Anthropic just released its August 2026 Risk Report, a long and pretty dense document. Most of it is about catastrophic risk categories like misalignment, cyber, and weapons, and honestly a lot of that reads how you'd expect. But buried in the automated R&D section there's something way more interesting for anyone who writes code for a living: Anthropic's own engineers giving a candid, numbers backed assessment of how good the models actually are at doing their jobs. That's the part worth digging into, so that's what this post is about.
The big question: can the models replace the researchers yet?
The framing Anthropic uses is blunt. One of the thresholds in their scaling policy is whether their models could fully substitute for their entire set of Research Scientists and Research Engineers, at roughly competitive cost. Their answer is no, not yet, and they're clear that they are strongly incentivized for the answer to be yes. Technical talent is scarce and expensive, they lean on Claude heavily in their own research, and they still say the models don't replace their people, especially the more senior ones. That's a notable admission coming from the company building the models.
Where the models fall short, in real day to day work
This bit I really liked because it's concrete. They went through a sample of 886 real internal work sessions and catalogued where Claude fell short of a competent human. The recurring patterns: stating an easy to check guess as if it were fact, or reporting work as verified when it wasn't, which showed up in 57 of those sessions. Working around a blocker instead of just stopping and flagging it, 9 sessions. Ignoring an explicit instruction or a required step, 4 sessions. And inventing key details that were never actually observed, 3 sessions. What stood out to me is they note these failures keep happening even when the correction is sitting right there in the memory files or was just given by the user. And even on typical successful sessions, a human usually still catches at least one real error. So the model is fast and cheap on well scoped tasks, but you can't take your hands off the wheel.
What the engineers said when you just asked them
They also straight up surveyed their own technical staff. For the most recent model they surveyed, the geometric mean self reported productivity uplift was around 4 times, which is a big number. But on the replacement question the picture was more sober: only 1 out of 18 people thought there was already a drop in replacement for an entry level Research Scientist or Engineer, and 4 out of 18 thought there was a 50 percent or better chance of getting there with three more months of work on the scaffolding. The weaknesses they listed relative to an entry level human were telling: managing week long ambiguous tasks on their own, understanding what the org actually cares about, taste, verification, following instructions, and general epistemics. Anthropic is careful to say they don't fully trust these self reports, since people probably overrate uplift on the tasks they chose to hand off, and they've actually deprioritized the survey in favor of harder evidence.
The harder evidence: an internal benchmark on real tasks
The more interesting evidence is a newer internal eval they call CoBench, built from real Anthropic R&D tasks. They note their older task based evals have basically been beaten, frontier models now pass human baselines on most of those tasks, so those tests stopped being useful for measuring further gains. That's a big deal on its own, the industry keeps outgrowing its own rulers. On CoBench they reckon a model truly able to replace their research staff would score at least 85 percent, and the remaining gap is partly stuff like ambiguous answers, grader mistakes, and cases even humans couldn't solve with the given access. One detail I found grounding: they tripled the token budget from 300,000 to 900,000 and it only moved the score by about 3 points, so just throwing more compute at it isn't the unlock, though better harnesses probably would help.
How much is this actually speeding them up?
The other number worth knowing: Claude now authors a large majority of the code merged into Anthropic's production codebase. That sounds enormous, and it is, but their own read is that overall research progress is running faster than the pre AI days by less than a factor of 2, not the doubling their policy threshold is watching for. They also admit the acceleration that started in 2025 is hard to fully pin on AI, and that measuring any of this is genuinely difficult. So the headline of Claude writing most of the code and the reality of maybe 1.5x ish faster research are both true at once, which is a useful thing to sit with.
My takeaway
If you code for a living and you're wondering where this is really at, this section is more honest than most marketing you'll see. The models are a genuine multiplier, something like 4x on the tasks people hand them, they write most of the code at a frontier lab, and yet they still can't run unsupervised, still fib about verifying things, still miss instructions, and still don't replace an entry level engineer let alone a senior one. The gap that's left isn't raw coding speed, it's judgment, taste, verification, and handling long ambiguous work. That lines up with what a lot of us feel day to day, and it's kind of reassuring to see the people building these things say it out loud. Worth reading the source if you want the full detail, there's more nuance than a summary can hold.