AI-Rewritten Linux Utilities: Fewer Crashes, More Hangs

A study fuzzed 10 AI-rewritten Linux utilities against their human originals: fewer memory-safety crashes, but a larger share of hangs.

Researchers at the University of Wisconsin-Madison fuzz-tested AI-rewritten versions of 10 well-known Linux utilities against the current human-written originals. The AI-generated code failed less often overall and had fewer memory-safety crashes, such as buffer overflows, but a much larger share of its failures were hangs and infinite loops.

The study, published as a Trusted CI report and posted to arXiv on September 16, 2026, comes from Ayesha Shafique, Barton P. Miller, and Elisa R. Heymann. It is funded by the National Science Foundation, not by an AI vendor, so the incentive to make the result look flattering in either direction is low. Miller has been fuzzing Unix utilities since 1990, when he and two colleagues published the paper that effectively introduced random fuzz testing as a technique. Thirty-six years later, he pointed the same method at code an agent wrote from a blank file.

Sources: A Study of the Reliability of Agentic AI-Generated Programs (arXiv:2609.18298)

A clean-room rewrite, not a refactor

The researchers picked ten established Linux utilities: cat, dash, dc, grep, less, make, ptx, strings, tac, and tnftp. Program size ranged from around 400 to 18,000 lines of C. For each one, they ran a five-stage agentic workflow built on Claude Code Opus 4.8: derive a feature checklist and a test oracle from the program's own documentation, design an architecture, implement it with test-driven development, run an independent specification audit, then run a security audit using static analyzers, sanitizers, Valgrind, and GDB. A failed audit sent the program back to implementation, and the loop repeated until the audit accepted it.

The agents never saw the original source code and never ran the reference binary. Their only specification was the utility's published documentation, sometimes a short man page, sometimes, as with grep, roughly 17 pages of manual. The researchers required the AI version to implement the documented behavior, not to match the original output byte for byte. Some of the results were smaller than the code they replaced: tac shrank from 2,140 lines to 412. Others were not: less grew from 17,517 lines to 19,582.

Fuzzing, not unit tests, is what surfaced the gap

To measure reliability, the team ran both classic black-box fuzzing and modern coverage-guided fuzzing with AFL++, feeding each program roughly 4,280 test cases and about 12 GB of randomized data across four size ranges, from under 1 KB up to 100 MB. Every utility ran inside a Docker container so a wayward shell command or a genuinely stuck process could not affect the host or get miscounted as a failure. The two fuzzing methods were not redundant: black-box testing found 7 crashes and 4 hangs, AFL++ found 10 crashes and 7 hangs, and the two methods overlapped on only 4 hangs, for a combined total of 17 unique crashes and 7 unique hangs across every program tested.

Sources: A Study of the Reliability of Agentic AI-Generated Programs, Table 2

How many total failures did fuzzing find?

Across all ten programs, fuzzing exposed 24 unique failures in total: 19 in the human-written repository versions and 5 in the AI-generated versions. Six of the ten AI rewrites, cat, dash, less, make, strings, and tac, had zero failures of any kind. Only two of the ten human originals, cat and strings, could say the same.

Crashes made up 79 percent of the human code's failures and hangs made up 21 percent. In the AI code, that ratio nearly inverted: 40 percent crashes, 60 percent hangs. Raw counts favor the AI version outright, since it had five failures against the human version's nineteen, but the composition of those failures is where the two-sided finding lives: not just fewer failures, a different kind of failure.

Where the AI avoided a crash entirely

The clearest examples are in less and make. The human-written less had three distinct memory-safety crashes under fuzzing: a heap buffer over-read in the table that tracks screen-row positions, an invalid free triggered when a pointer that should have pointed to heap memory instead pointed to a string literal, and an out-of-bounds read one byte before the start of a completion buffer. The AI-generated less had none of these, not because it added extra bounds checks, but because it used different data structures altogether: no screen-row table to overrun, and numbered completion candidates instead of a pointer walked backward through a buffer.

Make tells a similar story. The human-written version crashed on unbounded recursive macro expansion: a macro that referenced itself indirectly through a construct like $(call f) expanded forever until the call stack ran out and the process segfaulted. The AI-generated version added an explicit recursion-depth limit and reported a clean error instead. Dash's human-written version crashed twice, once from treating input bytes as signed characters and indexing a lookup table with a negative value, a class of C bug that predates most of today's programmers, and once from a NULL-pointer bug in nested here-document parsing. The AI-generated dash had neither failure.

Where the AI introduced a failure mode of its own

Not every AI failure came from an inherited library. In ptx, a text-indexing utility, the AI-generated version built its output lines by repeatedly rescanning the surrounding text and assembling the result one character at a time. On ordinary input this was just slow. On adversarial fuzz input it turned pathological: a 100 MB input took about 10 minutes versus roughly 15 seconds for the human-written version, and a 2.6 MB input with no newlines, which forced the whole file to be treated as one text region, took over 10 minutes on its own. That is quadratic-time behavior hiding inside code that otherwise passed every test the agent wrote for it.

Sources: A Study of the Reliability of Agentic AI-Generated Programs, Section 6.7

Two other hangs, in dc and grep, were not AI-specific at all. Both implementations shared the same underlying issue: dc's arbitrary-precision arithmetic treats hexadecimal letters as valid digits even in base ten, and grep relies on the GNU regular-expression library, whose pattern compiler can blow up exponentially on strings of consecutive plus-quantifiers. Either version, human or AI, would have hit both bugs, since neither program controls that shared code path.

What fuzzing caught that the agents' own audits missed

The most useful part of the paper may not be the crash counts at all. It is the description of how unreliable the agents' self-reported progress turned out to be. In less, eleven command-line options were parsed and stored but never wired into any actual behavior. That gap survived four audit rounds and a test suite where 1,217 of 1,221 agent-written tests passed. The tests checked that the options were accepted, not that they changed anything.

Make is the sharper example. The agent-generated version passed every test it wrote for itself, yet could initially build little more than a trivial C program. The gap only became visible when the researchers used it to build real software: the Linux kernel in three configurations, plus PostgreSQL, Redis, BusyBox, SQLite, FFmpeg, OpenSSL, curl, binutils, and 7-Zip. That exposed missing and incomplete features the specification-derived tests never touched, and cost roughly another week of implementation and debugging to close.

This is exactly the trap covered in your agent passed the tests by luck: a passing test suite is evidence about the specific checks it runs, not proof that the underlying implementation is sound. The researchers put it almost as bluntly, writing that claims produced by an agent should be treated as hypotheses to verify rather than as the truth. Testing AI-generated code well means going past whatever tests the agent wrote for itself, and mutation testing is one way to check whether those tests would even catch a real defect if one were introduced. Fuzzing caught a different class of gap here: behavior nobody had gotten around to specifying in the first place.

Does this study prove AI-generated code is safer than human code?

No, and the paper does not claim otherwise. Ten command-line C utilities is a narrow sample. A web service, a data pipeline, or anything written in a memory-safe language would change the picture, since the entire crash side of this comparison is specific to languages where a buffer overflow is even possible. The researchers also stress that the outcome depended heavily on how the workflow was run: reliable output required careful prompts, purpose-built skills reused across every program, and a human resolving ambiguities the documentation left open. A team skipping those three things should not expect these numbers to hold.

Practical takeaways for teams shipping AI-written code

• Fuzz test AI-generated code the same way you would legacy C, especially anything that parses untrusted input. The crash counts here came from fuzzing, not from the agents' own test suites, which caught none of them.
• Do not treat an agent's "all tests passed" report as proof of completeness. Make's AI rewrite passed every test it wrote for itself and still could not build the Linux kernel.

The gap between what these agents reported and what fuzzing and real workloads actually found is the same gap TLM Forge is built to close. It enforces test-driven development with full-suite regression evidence captured at every stage, so "tests passed" is a reproducible run rather than an agent's own summary of its work. It also requires every behavior claim to cite the specific file and line that backs it, which is the direct opposite of a checklist item marked complete because an option was parsed but never wired into real behavior. A reviewer can check the claim against the code instead of taking the agent's word for it.