Semgrep found Claude Opus 5.5 writes working code 93.5% of the time, but only 54.8% of it is secure, and memorized fixes explain most of the gap.
Claude Opus 5.5 produces code that works on 93.5% of 186 security-relevant coding tasks. It produces code that works and is secure on only 54.8% of them. Semgrep's SusVibes benchmark, published September 29, 2026, built those 186 tasks from real historical CVEs in open-source Python projects, recast as ordinary feature requests, and the gap between the two numbers is the entire story: 41% of the model's "working" solutions still shipped the exact vulnerability the task was built around.
Sources: Semgrep: Claude Opus 5.5 Writes Working Code but Only Half of It Is Secure
The more unsettling finding sits one layer deeper than the headline gap. About 53% of the solutions Semgrep could compare against history were near-identical to the real historical fix, meaning Opus had almost surely seen that exact patch during training rather than reasoning its way to a new one. Those memorized solutions were secure 64% of the time. The solutions Opus actually had to derive on its own were secure only 51% of the time. A model that performs measurably better when a problem resembles something already in its training data is not demonstrating security judgment. It is demonstrating recall, and recall does not transfer to a codebase the model has never seen.
Sources: Semgrep: Claude Opus 5.5 Writes Working Code but Only Half of It Is Secure
What SusVibes actually measured
SusVibes, authored by Katie Paxton-Fear, Staff Security Advocate at Semgrep, is built from 186 real CVEs found in open-source Python projects. Semgrep converts each CVE into a feature request: it deletes the code that implemented the feature, including the part that originally introduced the vulnerability, and tells the agent to build the feature back with no security guidance beyond a one-line generic reminder. Claude Opus 5.5, running inside SWE-agent 1.1.0 with the benchmark's standard prompt, an anti-cheating instruction block, a 200-call budget per task, and no internet or outside repository access, had to write the feature cold.
Sources: Semgrep: Claude Opus 5.5 Writes Working Code but Only Half of It Is Secure
Semgrep scores each task twice, independently. A functional check runs the project's own test suite and passes a solution that breaks no more tests than the reference implementation did. A security check runs the tests that shipped alongside the real-world CVE fix, and passes a solution only if it is not vulnerable to the original bug. A task counts as a full success only when both checks pass. That split is what exposes the gap: a solution can satisfy every functional test a reviewer would normally run and still carry the vulnerability the task was built to probe for, because passing tests never prove the agent handled the security-relevant code path correctly in the first place.
Sources: Semgrep: Claude Opus 5.5 Writes Working Code but Only Half of It Is Secure
The headline number, and the number that matters
Semgrep's functional check found working code on 174 of the 186 tasks, 93.5%. Of those 174 working solutions, 72 still contained the original vulnerability, 41% of everything that counted as "done." Only 102 of the 186 tasks, 54.8%, cleared both the functional bar and the security bar at once. Read the two numbers side by side and the first one stops meaning very much on its own: a model that is right 93.5% of the time by the metric a demo would show you, and right less than 55% of the time by the metric a security team actually cares about, is reporting two different models' worth of performance depending on which number a release note chooses to quote.
Sources: Semgrep: Claude Opus 5.5 Writes Working Code but Only Half of It Is Secure
For context, Claude Opus 5.5's 54.8% correct-and-secure score is still the best result SusVibes has recorded. The next-best prior score was 43.5%, from GPT-5.5 running under mini-swe-agent. Claude Opus 4.8, the previous generation under the same SWE-agent setup, scored 19.4%. The newest model is a real improvement over the last one. It is still wrong, in a security-relevant way, on almost half of everything it ships as finished.
| Metric | Figure |
|---|---|
| Tasks in the benchmark | 186, built from real CVEs in open-source Python projects |
| Produced working code | 93.5% of tasks (174 of 186) |
| Working and secure at once | 54.8% of tasks (102 of 186) |
| "Working" solutions that still had the original vulnerability | 41% (72 of 174) |
| Solutions near-identical to the real historical fix (memorized) | 53% of judged solutions (93 of 176) |
| Secure rate, memorized solutions | 64% |
| Secure rate, independently derived solutions | 51% |
| Best prior correct-and-secure score (GPT-5.5, mini-swe-agent) | 43.5% |
Memorization, not reasoning, is doing most of the work
Semgrep measured similarity between each solution and the real, historical fix for that CVE, comparing only the lines added to non-test files and ignoring whitespace and comment differences. Of the solutions Semgrep could judge this way, 53% (93 of 176) scored as near-identical to the real fix at a similarity threshold of 0.80 or higher, strong enough that Semgrep treats them as effectively memorized rather than independently generated. Those memorized solutions were secure 64% of the time. The solutions Opus had to write without a close training match were secure only 51% of the time. Thirteen points separate a model recalling a known answer from a model reasoning through a new one, on the exact same underlying model, inside the exact same benchmark run.
Sources: Semgrep: Claude Opus 5.5 Writes Working Code but Only Half of It Is Secure
The pattern gets sharper, not weaker, when Semgrep cut the data by how hard or how old each fix was. For fixes longer than 50 lines, solutions written independently were secure only 33% of the time, against 60% for memorized solutions, roughly double. For CVEs originally fixed in 2019 or earlier, the gap widens even further: 79% secure when memorized, 33% secure when derived. The older and more thoroughly documented a vulnerability class is, meaning more training data likely exists discussing it, the bigger the advantage memorization gives the model over genuine problem-solving. Thirty-four distinct weakness categories, spanning cryptography, concurrency, access control, and several injection types, had zero secure solutions across the entire benchmark.
Sources: Semgrep: Claude Opus 5.5 Writes Working Code but Only Half of It Is Secure
A model scoring better on problems that resemble its training data and worse on genuinely new ones is not a quirk in the data. It is the most honest signal the benchmark produced, and it points the opposite direction from what a single aggregate score suggests.
Why a benchmark score tells you little about your own codebase
By construction, SusVibes draws its tasks from public, well-documented CVEs in open-source projects: exactly the kind of material likely to appear, directly or indirectly, in a frontier model's training data. That is not a flaw in the benchmark. It is close to the best anyone can do at scale, and it is precisely what makes the memorized-versus-derived split so informative. Your own codebase doesn't draw from public CVEs. It is private, it never appeared in the training set, and every feature request you hand an agent is, by definition, closer to the "derived" half of this study than the "memorized" half. If the memorized half scores 64% secure and the derived half scores 51%, the task that most resembles your actual day-to-day work is the one the model is worse at, not the one its headline score describes.
Sources: Semgrep: Claude Opus 5.5 Writes Working Code but Only Half of It Is Secure
That is the structural problem with treating any benchmark percentage as a transferable trust score. A benchmark number is an average over a known set of tasks, assembled by someone else, with a known relationship to public training data. Nothing about that average tells a team whether the specific task in front of it today, on its own proprietary code, lands in the easy half of the distribution or the hard half. A related argument about Claude Opus 5.5's launch made this point from the completion-rate side: a model that finishes more of a task unsupervised is a model whose mistakes travel further before a human notices. SusVibes adds the security half of that same argument. The model is not just finishing more. It is finishing confidently wrong on security-relevant work nearly as often as it finishes right, and the published score cannot tell you in advance which outcome you are getting on any single task.
Sources: Semgrep: Claude Opus 5.5 Writes Working Code but Only Half of It Is Secure
The honest conclusion: review every change, not just the risky-looking ones
There is no reliable way to tell, before an agent writes the code, whether the task you just handed it looks enough like something in its training data to get the memorization boost, or whether it is genuinely novel and sitting closer to that 51% number. A task that feels routine to a human engineer, implementing an auth check, parsing an upload, handling a deserialization path, can be exactly the kind of thing that is either everywhere in public training data or nowhere in it, and there is no way to read that off the ticket. That uncertainty is the whole argument for running independent review on every change rather than triaging which changes "seem risky enough" to deserve scrutiny. TLM Forge builds its approach around that uncertainty rather than around trusting any model's track record: a spec audit before code generation, an independent diff review once code exists, an adversarial red-team security pass on top of that, and a convergence gate that blocks the merge until every flagged issue hits zero, not just logged for later. None of that depends on knowing in advance whether a given task is the kind Opus 5.5 is 64% secure on or the kind it is 51% secure on. It runs the same way either way. You can read the specifics of how it works.
None of this is an argument that Claude Opus 5.5 is unusually bad. It posted the best correct-and-secure score SusVibes has recorded against any model tested so far, well ahead of GPT-5.5 and far ahead of the previous Claude generation. The argument is narrower and, for anyone deciding how much to trust a benchmark card, more useful: even the best-scoring model available checked out insecure on close to half of its own benchmark, and the half it got right leaned heavily on having seen the answer before. Whatever model ships next will likely score higher still. The gap between its memorized performance and its derived performance is the number worth watching, not the aggregate, because that gap is the only part of the score that says anything about work the model has never seen, which describes essentially all of your own codebase.
Sources: Semgrep: Claude Opus 5.5 Writes Working Code but Only Half of It Is Secure
Frequently asked questions
01What is the SusVibes benchmark?
SusVibes is a Semgrep benchmark of 186 coding tasks built from real historical CVEs in open-source Python projects, with each CVE recast as a feature request. An agent must rebuild the feature from scratch, and Semgrep scores solutions separately for whether they work and whether they avoid the original vulnerability.
02How often does Claude Opus 5.5 write secure code on SusVibes?
Claude Opus 5.5 produced working code on 93.5% of the 186 tasks, but was both working and secure on only 54.8% of them. Among the solutions that did work, 41% still contained the original vulnerability the task was built around, according to Semgrep's September 29, 2026 study.
03Why do memorized solutions score more secure than original ones?
Semgrep found that about 53% of Opus 5.5's solutions were near-identical to the real historical fix, meaning the model had likely seen that exact patch in training. Those memorized solutions were secure 64% of the time, versus 51% for solutions the model had to derive independently, suggesting recall rather than security reasoning drives much of the score.
04Does a high AI coding benchmark score mean a model is safe to use on my own code?
Not reliably. Benchmarks like SusVibes draw tasks from public, well-documented vulnerabilities likely present in training data. A private, unpublished codebase looks structurally more like the benchmark's "derived" tasks, where Claude Opus 5.5 scored 51% secure, than its memorized tasks, where it scored 64%.
05How does TLM Forge address AI coding security without relying on benchmark scores?
TLM Forge runs a spec audit before code generation, an independent diff review, and an adversarial red-team security pass on every change, then blocks merges behind a convergence gate until flagged issues hit zero. The process runs the same way regardless of how safe a model's benchmark results suggest it should be.