Training on random integers changes fixed-list choices across tasks
Fifty RL steps on one pick-a-number task moved Qwen3-30B-A3B-Instruct's choices toward uniform on all nine untrained pick-one tasks, measured over each task's fixed answer list. On Llama-3.1-8B-Instruct the shift was narrower and concentrated on integer tasks.
Finding. Asked for a random integer from 1 to 100, Qwen3-30B-A3B-Instruct put more than 95% of its probability on 4, 42 and 47. After 50 steps of reinforcement learning (GRPO) on that one task, its distance from uniform (how far its probabilities over a task's list are from an even spread) fell on all nine untrained tasks, from a mean of 0.74 to 0.38. On Llama-3.1-8B-Instruct the same recipe moved 7 of 9 untrained tasks toward uniform, with the large moves on integer ranges.
What was measured. The main numbers are candidate scores over each task's fixed list of valid answers; sampled checks cover three tasks for the trained model. Neither measures open-ended writing.
Previously titled "A 50-step RL update reduces categorical sampling bias". I renamed it for what was measured, separated candidate scoring from sampling, narrowed the capability claims and marked the Llama report's stronger wording as superseded. The fruit uniform reference is corrected to 0.251 (previously 0.250), and the Qwen MMLU check is described as a subject-stratified subset (200 questions requested); other measurements are unchanged.
Why this question
Instruction-tuned models asked to pick at random from a list tend to return the same few favorites, which matters when a model is asked repeatedly to choose from a fixed set, as in simulating a random draw or producing many variants of a synthetic-data item. A per-task fix such as logit correction (offsetting each candidate's score until one task is exactly uniform) does not carry over: offsets for "integer 1 to 100" say nothing about "color". So I asked whether a small update on one choice task changes how the model spreads probability over other tasks' lists.
The starting observation follows exmergo's experiment, which asked gpt-4.1 10,000 times for a number between 1 and 100. Zhang et al. (2024) fine-tuned models to spread probability over valid choices and reported that their method generalized across tasks. This run tests a different update: a 50-step RL pass on one integer task.
The setup
I trained Qwen3-30B-A3B-Instruct, a mixture-of-experts model, for 50 steps of GRPO (Group Relative Policy Optimization) on the integer 1 to 100 task. GRPO samples a group of answers per prompt and moves the model toward those that score above the group average; here rarer answers within a group of 16 scored higher. The update is a LoRA adapter (a small set of low-rank weight changes to the frozen model), published on Hugging Face; the run cost about $25 in Tinker compute.
The exact candidate-scoring prompt for one task, a flat string with no chat template:
System: You are a uniform random sampler. Output ONLY one card suit from the list below, exactly as written. No punctuation, no extra words. Valid card suits: Spades, Hearts, Diamonds, Clubs User: Pick a random card suit. Assistant:
Each candidate is scored as the text after Assistant:, averaged over paraphrases of the user line. Training used the same flat format and the first user line, with a random request ID added.
Who did what. I chose the question, and Claude coding agents co-authored the code, the run reports and the Llama repeat. Their work was checked by the pipeline's own thresholds for transfer and capability regression, the repo's unit tests, and a September 2026 review, also run with a Claude coding agent, that checked the published claims against the run records. That review corrected the README and added a scoping note to the Llama report (PR #1), and corrected the model card and this post. None of these checks is an outside human review.
Results on ten tasks
| Task | K | Max | Before | After |
|---|---|---|---|---|
| integer 1 to 10 | 10 | 0.900 | 0.890 | 0.115 |
| fruit | 20 | 0.950 | 0.461 | 0.194 |
| first name | 20 | 0.950 | 0.607 | 0.300 |
| color | 20 | 0.950 | 0.889 | 0.311 |
| card suit | 4 | 0.750 | 0.712 | 0.370 |
| integer 1 to 100 (trained) | 100 | 0.990 | 0.953 | 0.413 |
| emoji | 20 | 0.950 | 0.771 | 0.420 |
| animal | 20 | 0.950 | 0.620 | 0.522 |
| integer 1 to 1000 | 1000 | 0.999 | 0.942 | 0.571 |
| word | 50 | 0.980 | 0.770 | 0.581 |
Every task moved toward uniform. Integer 1 to 10 moved most, from near its 0.90 ceiling to 0.115; animal moved least, from 0.620 to 0.522. Across the nine untrained tasks the mean fell from 0.74 to 0.38.
The chart and table use candidate scoring, which renormalizes the listed answers' scores over the list. Probability outside the list, such as an unlisted fruit, a sentence or a refusal, is left out, so these numbers describe the model's preferences among the listed answers, not what it would write. Sampling generates and counts answers, and it used a different input: the same instruction in the model's chat template, where training and candidate scoring used the flat string above. A gap between scored and sampled numbers can come from either difference.
Capability checks. I saw no decline on the small Qwen checks: a subject-stratified MMLU subset (200 questions requested), 50 GSM8K problems answered greedily, and the 25 GSM8K problems sampled at T=1.0 for the appendix. These checks leave instruction following, calibration, safety and tool use untested.
Llama-3.1-8B-Instruct (dense, 8B), same 50-step recipe and candidate scoring: the trained task fell from 0.474 to 0.094. Distance from uniform fell on 7 of 9 untrained tasks, and 5 of 9 cleared my transfer threshold of a 0.05 drop (mean drop 0.105). The transfer concentrated on integers (integer 1 to 10: 0.527 to 0.116; integer 1 to 1000: 0.386 to 0.123). First name and card suit each got 0.008 worse; on Llama both started much closer to uniform (0.372 and 0.134) than on Qwen (0.607 and 0.712). The small checks showed no decline: 57 MMLU questions (one per subject), 54.4% before and after; 30 GSM8K problems, 13.3% before and after; and 60 prompts from four hand-written instructions (labeled IFEval in the code, but not the benchmark), 50.0% before and 56.7% after. Full report, now with a dated correction (24 October 2026) that marks its stronger readings superseded.
Reading the Llama report. The report's stronger reading is superseded. It calls capability "preserved", but three small checks do not establish that; it calls the bias "shallowly encoded and cheaply correctable", but a 50-step LoRA does not show where a behavior is encoded; and it calls the correction "reproducible across architectures", but that rests on two model families, one of which transferred narrowly. Its Qwen mean of 0.79 to 0.43 mixes ten sampled tasks before training with three after; candidate scoring gives 0.76 to 0.38 over all ten.
Sampled answers
A user sees sampled text, not probabilities. I sampled 50 answers per cell from Qwen, with two paraphrases of the user line: the original model on all ten tasks at T=1.0 and 1.5, and the trained model on three tasks at T=1.0 before the run was cut short (23 of 40 planned cells). The comparison below covers those three. Outputs that do not parse to a listed candidate are dropped, and distance from uniform is computed over the valid outputs only.
Temperature above 1.0 gives less likely answers more weight. Raising the original model from T=1.0 to 1.5 moved all three tasks toward uniform, but each stayed further from uniform than the trained model at 1.0:
| Task | Original, T=1.0 | Original, T=1.5 | Trained, T=1.0 | Uniform sampler |
|---|---|---|---|---|
| integer 1 to 100 (K = 100) | 0.98050/50 | 0.94046/50 | 0.60050/50 | 0.6050.56 to 0.65 |
| color (K = 20) | 0.90050/50 | 0.74146/50 | 0.42849/50 | 0.2530.18 to 0.34 |
| fruit (K = 20) | 0.51050/50 | 0.39647/50 | 0.25050/50 | 0.2510.18 to 0.33 |
What 50 samples can show. Even a perfectly uniform sampler does not produce uniform counts. With 50 draws over 100 integers its distance from uniform cannot fall below 0.50 and averages (1 − 1/100)50 = 0.605. Trained integer 1 to 100 (0.600) and fruit (0.250) fall inside the uniform sampler's range: at this sample size the measurement is saturated and cannot tell the trained model from uniform. It does not show the model is uniform. Color (0.428) and every original-model cell are above the range.
What this shows, and what it does not
Fifty GRPO steps on one random-integer task changed how Qwen3-30B-A3B-Instruct spreads probability over the fixed lists of all nine untrained tasks. Sampled at T=1.0 on the three tasks checked, the trained model was also closer to uniform than the original at T=1.0 or 1.5. On Llama-3.1-8B-Instruct the change was narrower, mostly on the other two integer ranges. The finding is about choices over fixed lists: the experiment did not measure open-ended writing or mode collapse in general (post-trained models concentrating on a few outputs), and the appendix's GSM8K proxy does not measure solution strategies.
The most consequential limitation is the gap between scores and samples: the scores leave out off-list probability, and the trained model was sampled on only three tasks, at 50 samples each and in a different prompt format, with two of those cells already inside the uniform sampler's range. Whether its sampled answers move as far as its scores on all ten tasks is open. Uniform is also the right target for integers or card suits, but not clearly for fruit or words.
Next experiment
- The full sampled comparison, which would settle that limitation: all ten tasks, both models, T=1.0 and 1.5, one prompt format for training, scoring and sampling, and sample counts set by each task's K, or each cell compared with a matched finite-sample uniform reference. A flat 200 samples would not do: a uniform sampler over 1,000 integers still averages 0.819 at that count.
- Untested behaviors: instruction following beyond the small Llama check, calibration, safety, tool use, JSON formatting, full capability benchmarks, and free-form generation such as essays (an earlier embedding-based pilot on free text was inconclusive).
- Inference-time baselines: contrastive decoding, base-model-assisted decoding, rejection sampling.
- Other training tasks, model sizes and families.
- Settings where a fixed candidate list matters in practice, such as multiple-choice evaluation with shuffled options or synthetic-data generation.
Reproducibility
| Model | Qwen/Qwen3-30B-A3B-Instruct-2507 plus the LoRA adapter from step 50. Second family: meta-llama/Llama-3.1-8B-Instruct. The Llama step-50 adapter is not published. |
|---|---|
| Experiment | 50 GRPO steps on integer 1 to 100 (run_bee_v0_4_stage1.py, Tinker): 8 groups of 16 samples per step, LoRA rank 16, learning rate 2e-5, at most 8 response tokens; reward −log of the answer's share of its group, with a fixed penalty for unparseable outputs (grpo_env.py). Candidate scoring, flat prompt: eval_lane_a.py. Sampling, chat template: run_bee_v0_4_temp_ablation.py, distance over valid outputs in sample_eval.py. Capability checks: capability_eval.py. Numeric signatures: run_bee_v0_4_2_path_diversity.py. |
| Data | Ten random-pick tasks with fixed candidate lists (distribution_tasks.py): 1 trained, 9 held out. Unit: one task's distribution over its list. Sampled: 23 cells of 50 samples each, invalid outputs dropped. GSM8K test items 0 to 24, 10 samples each. Capability: Qwen, a subject-stratified MMLU subset (200 requested) and 50 GSM8K problems; Llama, 57 MMLU questions, 30 GSM8K problems and 60 instruction prompts. |
| Results | Not published: the Qwen per-task candidate scores, the Qwen capability checks, and the Llama run folders behind the Llama report (my run records). Published: sampled cells (CSV), numeric-signature report with per-problem CSVs and completions, Llama report with its per-task and capability tables (its stronger wording is superseded; see Results and the report's dated correction). |
| Code | scasella/research-uniform-random-from-llms @ c11534f. The README's Path 1 recomputes the appendix GSM8K signature averages from the committed CSVs without an API key. Retraining runs on Tinker (about $25 in compute); the published Qwen adapter loads on the base model with PEFT instead. |
| Status | Published 2026-05-26 · updated 2026-10-23 · Open issues: the Qwen per-task and capability records and the raw Llama run folders are not published; the sampling run stopped at 23 of 40 planned cells; at 50 samples, sampled distance from uniform is saturated for integer 1 to 100 and fruit. |
Appendix: a GSM8K side check, and how formatting misleads a proxy
For 25 GSM8K problems (10 samples each) I counted distinct numeric signatures (every number in the body, sorted, excluding the final answer line) among correct solutions as a proxy for varied working. At T=1.0 the trained model averaged 8.44 per problem against the original's 7.40 (paired difference +1.04, 95% bootstrap CI +0.40 to +1.68), about the same as the original at T=1.5 (8.40; difference +0.04, CI −0.60 to +0.60). Both models averaged 9.80 of 10 correct at T=1.0 (at T=1.5: original 9.68, trained 9.40).
The signature tracks layout as much as method. On problem 0, two original-model solutions with the same arithmetic count twice because their step numbering and restated quantities differ. So the count says little about solution strategies.