Exploratory follow-up on training with partially solved math problems

[ 2026-04-25 · updated 2026-10-23 ] // Qwen3-30B-A3B // 8 MIN READ

One reinforcement-learning run on the competition problems a panel-format Qwen3 adapter solved only some of the time raised its score on 100 held-out problems from 14% to 29% by its last evaluation. With no comparison curriculum, it does not show that choosing those problems beats training on others.

The question

The training loop scores each sampled answer against the others for the same problem, as in GRPO: a sample's advantage is its reward (1 if correct, minus 0.2 for missing or malformed answer tags) minus its group's mean. When every sample is right, or every one wrong, the correctness part cancels and only the small tag penalty can differ, so mixed problems are where the learning signal comes from.

That suggests a curriculum (a rule for choosing what to train on) made of the problems a model solves some of the time. I ask where each model's mixed problems sit, and whether training the panel adapter on its own mixed problems raises its accuracy on problems it never trained on.

Setup

The two models are described in the main study: a rank-32 LoRA adapter (a small set of trainable weights added to a frozen Qwen3-30B-A3B-Base) trained with RL to reason as a panel of experts, and Qwen's production Qwen3-30B-A3B with thinking on.

This run started from the panel adapter after its 128-step GSM8K stage (RL on grade-school math word problems, run on Tinker, the hosted training service I used), before the MATH stage (RL on MATH competition problems) that produced the adapter released with the main study. The band sweep and the RL log both load that checkpoint.

The pool has 877 problems: HMMT February 2024 and 2025 (60), AIME 2024 and 2025 (60), OlympiadBench's English text-only math competition set (674) and AMC (83). I sampled each problem 8 times from each model at temperature 1.0 with an 8,192-token limit and put it in one of three bands: all_zero (0 of 8 correct), variance_band (1 to 7, which I call mixed) or all_one (8 of 8).

Where each model's mixed problems sit

Having more mixed problems is not the same as exploring more. A problem a model solves with probability p per sample is mixed in a group of G with probability 1 − p^G − (1 − p)^G: with G = 8, 99% at p = 0.5, 57% at p = 0.9 and 34% at p = 0.05. Many mixed problems means many problems near the middle of a model's ability.

Table 01 fits that reading: most of the panel's mixed problems (280 of 382) are ones the thinking model got 8 of 8, and the thinking model's largest mixed group (137) are ones the panel got 0 of 8. The counts show where each model is still learning on this pool. The models also differ in post-training (see the main study), so this compares two models, not two formats.

The counts are approximate. The bands used 8 samples but RL used groups of 16, so a problem at 0 of 8 can still give mixed groups of 16. And the 8,192-token limit was tight for the thinking model: its first-step RL samples averaged about 11,100 tokens (under a 12,288-token limit) against the panel's 586, so the sweep likely cut off many of its samples before an answer.

One RL run on the panel's mixed problems

I set aside 100 problems, a random 25 from each source, and removed them from both models' training sets, leaving 354 mixed problems for the panel and 180 for the thinking model. No training step used the 100, but they are not an untouched test: I scored them every 10 steps during the run, with one sampled answer per problem.

Settings: LoRA rank 32, learning rate 5×10⁻⁶, groups of 16 samples, 8 problems per step, temperature 1.0, a 12,288-token limit and 100 steps. The loop drew two problems from each source at every step, so each source supplied 200 of the 800 draws regardless of its size.

The score dipped at step 10, rose to 21% by step 50, and read 29%, 27% and 29% at steps 70, 80 and 90. I quote step 90 because it was the last evaluation, not because it was the best; it tied step 70.

All four sources went up, by 6 of 25 held-out problems on OlympiadBench, 5 on AMC, 3 on AIME and 1 on HMMT. OlympiadBench made up 88% of the training pool but supplied only a quarter of the draws. With 25 problems per source and one sample per problem, these changes are noisy; the overall change is more reliable.

The thinking arm

I started the matching run on the thinking model's 180 mixed problems three times, and each stopped within its first three steps, two of them when the compute budget ran out; the thinking model's samples were about 19 times as long as the panel's. Without a thinking-model curve, this note cannot say which model improves faster under RL.

What this shows, and what it does not

The run establishes one result: from the GSM8K-stage adapter, RL on its own mixed problems raised its monitored held-out score from 14% to 29% at step 90, the last evaluation. That is still well below the thinking model's 54% to 61% on the same set before any RL. The band counts add a second result: on this pool, the two models' mixed problems mostly differ.

The run does not show that selecting mixed problems beats another curriculum: RL on other problems from the same pool might have produced a similar gain. The missing comparison, the first item under Next experiment, is the same starting adapter trained on mixed-outcome problems and on a random sample or the full pool under one matched budget. Finishing the thinking-model arm would not isolate the selection, because that arm also changes the model, its reasoning format, its post-training and its training problems.

The gain rests on one completed run with one seed, scored with one sampled answer per problem and no confidence interval. Single-sample scores on 100 problems move by several points, so the exact size of the gain is uncertain.

Next experiment

• Test the selection. From the GSM8K-stage adapter, train a mixed-outcome arm, a random-sample arm and a full-pool arm under one budget: 100 steps, 8 problems per step, 16 samples per problem, two problems per source per step, with at least two seeds per arm. Score a fixed final checkpoint of each arm with several samples per problem on problems no arm trained on or was monitored with.
• Then run the thinking arm: recompute both models' bands with 16 samples and a token limit that fits the thinking model's traces, and train 100 steps on its mixed problems (data/olympiad_pool/thinking_train.jsonl) with the same settings.
• Repeat the panel run once from the post-MATH adapter released with the main study.

Methods notes

Training-side metrics. On the training problems, the share of correct samples rose from 23% (steps 0–9) to 58% (steps 90–99). Mean per-token entropy fell from 0.591 to 0.158 over those windows and within-group length spread rose (coefficient of variation 0.36 at step 0, 0.66 at step 90); neither shows whether samples used different solution strategies, so they are not evidence about diversity.

Starting adapter. Reproducing this run means retraining the GSM8K stage to 128 steps (its script defaults to 80) and passing that checkpoint to scripts/rl_panel_olympiad.py as load_checkpoint_path, overriding its post-MATH default (PANEL_MATH_CHECKPOINT).

Reproducibility