Withdrawn: ten mined examples and a 1.5B bug fixer
I published this post on 2026-05-26 and withdrew it on 2026-09-23. It made two claims about a synthetic Python bug-repair benchmark. The first was that a LoRA (a small set of added weights trained while the base model stays frozen), trained on ten mined fixes, lifted Qwen2.5-1.5B-Instruct from zero and repaired bugs in categories absent from its training. The mined fixes were patches the model itself wrote that the benchmark’s hidden tests accepted. The second was that fixes mined with diagnostic test feedback, which reported the failing tests and their errors, carried over to those unseen categories better than fixes mined with pass/fail feedback.
Why I withdrew it
Rechecking the run records turned up three problems that rewording could not fix:
- The test prompts named the bug category. Each test prompt carried the task ID and a "bug category hint", so the model was told what kind of bug to look for. My own run notes flag the resulting scores as inflated by those hints.
- The feedback comparison was not isolated. The two mining conditions used differently worded prompts. The diagnostic prompt mentioned mutation and type conversion, two of the three held-out categories (bug types kept out of training to test transfer). And its training sets were drawn from mined fixes covering five bug categories, against three for the pass/fail set. Any of these, rather than the type of feedback, could explain the held-out gap of 3–4 against 0–1 of 6 programs.
- The starting zero was mostly a format failure. 374 of the 384 baseline outputs contained no patch that the evaluation harness, the code that applies each patch and runs the tests, could parse. So much of the lift from zero came from learning the output format.
One later result is not among these reasons. In a follow-up experiment, not yet published, a 1.5B Qwen2.5 model trained on fixes it had mined from real Python bugs repaired 1 of 40 held-out real programs. That figure came from real-bug training, not from the LoRA in this post, so it shows low absolute performance on real code, not a measured failure of transfer from the synthetic benchmark.
The original post is archived as last published, under a withdrawal banner. If you cited it, please treat its conclusions as withdrawn. Corrections and questions are welcome.