Auto-research is wild

2026-03-17T00:00:00+00:00

This is pretty exciting.</p>

I heard about Kaparthy's Autoresearch</a> only vaguely: it's a way of getting the LLM to run experiments on itself(?), running for long periods of time.</p>

It wasn't until I read David Cortés' autoresearch-pi</code> skill</a> that I understood the significance. The skill lets you give your coding agent an optimization target for some or all of your code, then it:</p>


comes up with ideas,</li>
tries each of them out,</li>
keeping the ideas that make the optimization target better, but</li>
reverts the ones that don't.</li>

it comes up with more ideas in a loop.</li>
</ul>
I tried it on improving the integration test performance on a project I work on</a>. The details here are not important, but I include them here just to give a sense of how many moving parts there are involved.</p>
The tests are testing the correctness of the FFI generator. It's a complicated test set up:</p>

A Rust test file with a macro, which generates the actual test function.</li>
The generated test function takes the fixture crate, compiles it then generates the FFI for it the crate.</li>
The generated FFI is itself part typescript and part C++.</li>
The actual test is Typescript file which is written for the fixture crate.</li>
The test.ts</code> and the Typescript part of the generated FFI are bundled together, with tsc</code> and metro</code></li>
The bundle is sent off to a hand made C++ test runner which runs Typescript in a Hermes execution environment</a>.</li>
</ol>
There is a similarly convoluted set up to run the same tests with the same fixture crates, but in a WASM environment.</p>
The tests are pretty extensive, and on my machine, they take about 30-40 minutes. They take so long, I don't tend to run them all at once very often, preferring to run them one at a time, or just JSI or just WASM.</p>
Over the past couple of years I've tried to improve the performance, but it's always ended by backing away from it: it's too delicate and too important to break, but not important enough to make faster.</p>
After setting up pi.dev</a> and installing the plugin, I answered a few questions. These were essentially:</p>

what do I want to optimize and how do I measure it?</li>
how do we know an experiment didn't break anything?</li>
are there any ideas that I wanted to try first?</li>
</ul>
The plugin is a skill, which I guess could be ported to Claude or Codex, or whatever, and a gadget which updates the status bar.</p>
I chose to optimize the JSI test performance; picking out a couple of representative tests. It should run the entire test suite for JSI to check nothing had broken.</p>
Then went to bed.</p>
I came back in the morning, and it had got the who test suite down to 74 seconds.</p>
Its most significant win was to string together three ideas which together unlocked parallel test running.</p>
Since then, I have run it twice more: once for the WASM test suite, and then once when both of them were sub two minutes and running in parallel. I could get it to optimize for a realistic development scenario: mutating a template which changes a generated FFI, so that only code changes that cause a change in generated code need trigger an expensive re-compile of the C++. For these second and third times, I limited the area of code to change to one crate.</p>
Now the entire test-suite, after two nights optimizing and almost no effort from me, is now 1m42s.</p>
A powerful new tool for our toolsets has been discovered.</p>

Notes to a Future Self - tips

Auto-research is wild

Tip: Being Karen to an agent