Training on sometimes-solvable math problems: one RL run

[ 2026-04-25 · updated 2026-09-23 ] // Qwen3-30B-A3B // 6 MIN READ

A panel-of-experts adapter on Qwen3-30B-A3B-Base was trained with RL on the competition problems it solved on some but not all sampled attempts. In a single run, its held-out score rose from 14% to 29%. The matching run for Qwen's thinking model was not completed.

Setup

The two models are described in the main post: a rank-32 LoRA adapter on Qwen3-30B-A3B-Base that reasons as a panel of experts, and Qwen's production Qwen3-30B-A3B with thinking turned on. The run logs and band summary show that this work used the panel adapter from after the GSM8K stage (Tinker session e1a9b8bf…), not the post-MATH adapter evaluated in the main post, although the repo's write-up calls it the post-MATH adapter.

The pool has 877 problems: HMMT February 2024 and 2025 (60), AIME 2024 and 2025 (60), the English text-only math competition set from OlympiadBench (674), and AMC (83). I sampled each problem 8 times from each model at temperature 1.0, with a limit of 8,192 tokens for both. Each problem then falls into one of three bands: all_zero (0 of 8 correct), variance_band (1 to 7 of 8 correct, which I call mixed) and all_one (8 of 8 correct).

Mixed problems matter because group-relative RL methods such as GRPO score each sample against the other samples on the same problem. The reward in this run is 1 for a correct answer, minus 0.2 when the answer tags are missing or malformed. If every sample in a group gets the same correctness score, the correctness term gives each sample zero advantage relative to the group. The tag penalty can still differ within such a group, so a small format signal remains.

Where each model's mixed problems sit

Having more mixed problems is not the same as exploring more. If a model solves a problem with probability p on each independent sample, a group of G samples is mixed with probability 1 − p^G − (1 − p)^G, highest at p = 0.5. With G = 8, a problem at p = 0.5 is mixed 99% of the time, one at p = 0.9 57% of the time, and one at p = 0.05 34% of the time. A model gets many mixed problems when many problems sit near the middle of its ability.

Table 01 fits that reading. Most of the panel's mixed problems (280 of 382) are ones where the thinking model got 8 of 8. The thinking model's largest group of mixed problems (137) are ones where the panel got 0 of 8. The counts locate each model's frontier on this pool. They do not show that either model searches more widely.

Two details make these counts approximate. The bands were set with 8 samples, but the RL run below used groups of 16, so a problem at 0 of 8 can still give mixed groups of 16 and a mixed problem can give a group with no correct sample. And the 8,192-token limit was tighter for the thinking model: in the RL logs, at a 12,288-token limit, its samples averaged about 11,100 tokens at the first step against the panel's 586, so the band sweep likely cut off many thinking samples before an answer.

One RL run on the panel's mixed problems

I held out 100 problems from the pool, 25 from each source, and removed them from both models' training sets. That left 354 mixed problems for the panel and 180 for the thinking model. Settings: LoRA rank 32, learning rate 5×10⁻⁶, groups of 16 samples, 8 problems per step, temperature 1.0, a 12,288-token limit and 100 steps. I evaluated every 10 steps with one sample per held-out problem.

The score dipped at step 10, rose to 21% by step 50 and reached 29% at step 70. It was 27% at step 80 and 29% at step 90, the last evaluation.

All four sources went up, most on OlympiadBench, which made up 88% of training. Each source has 25 held-out problems, so the AIME gain is 3 problems and the HMMT gain is 1.

Two rough signals of how varied the panel's samples stayed during training:

Length spread says nothing about content.

The thinking arm

I started the matching run on the thinking model's 180 mixed problems three times, with the same settings, and stopped each within its first three steps when I ran out of budget; its samples were about 19 times as long as the panel's. With no thinking-model training curve, this post cannot say which model improves faster under RL.

Interpretation and limits

On this pool, the two models are sometimes right on different problems: the panel mostly where the thinking model got 8 of 8, and the thinking model mostly where the panel got 0 of 8. One RL run on the panel's mixed problems raised its held-out score from 14% to 29%, still below the thinking model's score before RL.

• The 15-point gain comes from one run with one seed, scored with one sample per held-out problem, and has no confidence interval.
• The two models differ in post-training as well as in reasoning format (see the main post), so the band comparison is between these two models, not between formats.

Next experiment

• Run the thinking arm: 100 steps on its 180 mixed problems (data/olympiad_pool/thinking_train.jsonl) with the same settings, and plot both models on the same held-out set.
• First, recompute both models' bands with 16 samples per problem and a token limit that fits the thinking model's traces.
• Repeat the panel run with a second seed, and once from the post-MATH adapter used in the main post.
• Measure the panel's embedding dispersion after RL.

Reproducibility