Mostly software Zola 2026-07-12T00:00:00+00:00 https://jhugman.com/tags/fuzzing/atom.xml 2026-07-12T00:00:00+00:00 2026-07-12T00:00:00+00:00 James Hugman https://jhugman.com/posts/automated-bug-finding/ <p>My current side project, the one that's been consuming my hobby-less time for the past 15 years or so<sup class="footnote-reference"><a href="#1">1</a></sup> is a small programming language and associated tools. The language itself is called turo or turo-lang.</p> <p>This post is not about that project or the language, but about an emergent workflow which is quite interesting, and transferable to other projects.</p> <div class="footnote-definition" id="1"><sup class="footnote-definition-label">1</sup> <p>it's on its fourth iteration now, and I put it down for more than a decade, so it's not that old or mature.</p> </div> <h2 id="the-building-blocks">The Building Blocks</h2> <p>I say emergent, so these are the building blocks:</p> <ol> <li>I have a detailed <code>SPEC.md</code>.</li> <li>The first thing I build is: a CLI to exercise the language. There's a <code>--json</code> flag for agents to use, and <code>--debug</code> flag to turn on verbose logging.</li> <li>The second thing I build is: a test runner which reads YAML example files and generates tests. Inputs and outputs, but can probe the output, the current turo-lang scope, the types of each of the turo-lang variables etc.</li> </ol> <p>The example files keep the tests readable and implementation agnostic, without the ceremony of setup or teardown. It becomes a real repository of tests from test-driven development, from regressions, and debugging traces. So far, so <a class="external-link" href="https://en.wikipedia.org/wiki/Model-based_testing">model-based testing</a>.</p> <p>Next is a skill I made. It's a non-invasive debugging flow, which I've coined as doctor-patient-debugging<sup class="footnote-reference"><a href="#2">2</a></sup>:</p> <ol> <li>given a turo-lang program as input, evaluate the program in your head, then use the CLI. Notice how it succeeds or fails</li> <li>make variations of the input across different axes; to survey the shape of the error, testing with the CLI each time</li> <li>form theories based on this survey: is it a particular subsystem, or an interaction between subsystems</li> <li>use those theories to make predictions which can be used to prove or disprove those theories.</li> <li>only after proving a solid theory about what is happening should you open a source file.</li> <li>you can then convert the CLI probes into YAML fixtures, to feed into your RED GREEN REFACTOR process.</li> </ol> <div class="footnote-definition" id="2"><sup class="footnote-definition-label">2</sup> <p>I name it that here, but I'm sure this is an already named process. I have heard this called <a class="external-link" href="https://csc151.cs.grinnell.edu/readings/hypothesis-driven-debugging.html">hypothesis-driven-debugging</a>, but I prefer my name.</p> </div> <p>The skill relies on a rich input language, a debugging interface to a CLI, and a fast and synchronous program under test.</p> <h2 id="the-bug-finding-workflow">The Bug-finding workflow</h2> <p>The final part automates the process with a dynamic workflow, with a series of agents:</p> <ul> <li>fuzzers</li> <li>doctor-patient debuggers</li> <li>triagers</li> <li>fixers</li> </ul> <p>Have a fuzzer subagent:</p> <ul> <li>Read the SPEC.md</li> <li>Use the CLI to test a named area of the language. Once you find a bug, don't hammer on it, jump to a new idea.</li> <li>Once you have found a handful of bugs in an area, stop and report back the input strings and expected behaviours.</li> </ul> <p>You can get the workflow script to dispatch multiple fuzzers at once, one for each area in the spec. I like to set a "difficulty level" of the bugs: find the easily surfaced bugs which a user will find within minutes, rather than going deep on a subsystem which looks solid.</p> <p>Once all the fuzzers come back, you'll have perhaps 10–50 input strings to work through, so the next is to spin up doctor-patient debug agents that develop theories of the bug. Each of these returns a packet of YAML fixtures, an investigation trace, and a likely intervention that will fix that class of errors.</p> <p>Once the doctor-patient agents come back, we can have triage agents review prioritize and dedupe the bug fixes. This is the agent that decide if the bugfix is too invasive, and needs design work, or that it is the SPEC that is incorrect, or we're missing a feature, not just a bug.</p> <p>At this point we should have a prioritized list of tests and bugfixes which we can apply as a part of an ordinary RED GREEN REFACTOR loop.</p> <p>Now, we get one fixer agent to fix one bug at a time, each bug with a minimal reproduction, some boundary tests, an investigation and a likely fix. We could parallelize more by making worktrees, but I think that's perhaps a little heavyweight for this.</p> <p>Once the fixer agent is done, then we can loop back to the fuzzers again, but with the difficulty level turned up.</p> <p>The fixer is the <em>only</em> agent that actually changes the code, so has to be running sequentially; every other agent scales horizontally.</p> <p>After a few goes round this outer loop, you will have found <em>lots</em> of bugs.</p> <h2 id="results">Results</h2> <p>For turo, the summary was:</p> <blockquote> <p>~400+ agent-runs, &gt;45M tokens, ~45+ hours of compute across the line. 1,900 cases across 20 themed buckets — 1,473 green / 427 ignored. Landed via 133 commits touching the corpus, 105 of them <code>fix(...)</code> behaviour fixes.</p> </blockquote> <p>For a project with 60k lines of Rust (25k production), that doesn't sound too shabby.</p> <p>The fan-out of the fuzzing and the root-cause analysis from the hypothesis-driven debugging pair very well.</p> <p>## Next</p> <p>My next experiment with this flow, inspired by the <a class="external-link" href="https://bun.com/blog/bun-in-rust">Bun In Rust blog post</a>: apply all the fixes all at once, then bug fix the bug fix.</p> <p>I'd also like to try with different fuzzing agents: an hour of release time with a thousand users finds a different set of bugs than a single manual tester doing 1000 hours of exploratory testing.</p>