RL test generation

Rationale

Automated test generation runs into a circular dependency: evaluating test quality usually requires already having high-quality tests, and coverage. We approached this by optimizing directly for bug detection instead of for resembling human-written tests, since they can be cheaply generated by machine perturbations.

Methods

Given a specification and a correct implementation, GPT-4 acts as a fixed "bug fuzzer," producing perturbations across four categories: logic, boundary, type, and incomplete implementations. A generated test suite scores the fraction of those bugs it catches, with a penalty for failing on correct code, and that score is the reward signal.

We trained Qwen3-8B against it with GRPO on 38K functions pulled from 61 open-source Python repositories, with 116K injected bugs between them, running every suite in a sandboxed container. On a held-out benchmark of 1,000 high-complexity functions, the RL-trained model outperformed the base model.

Held-out benchmark, k = 5

metric	base	rl-trained	Δ
pass@5	56.50%	74.19%	+17.69
line coverage	96.03%	99.22%	+3.19
branch coverage	96.03%	99.22%	+3.19
mutation score	74.99%	100.00%	+25.01

Limitations

A bug in the evaluation pipeline silently filtered out cases where the model generated no tests at all — those are hard failures and should have counted as such, so the absolute scores overestimate reliability.

The benchmark was also small, and it measured generic functional correctness rather than the specific bug categories we trained against, which means it doesn't quite answer the question we started with.

Mechanical AST mutations used in our dataset produced many trivially detectable cases, while LLM-written bugs were more realistic and transferred better. Validating against real bugs from real projects would likely make this project more informative.