ICML 2026 Workshop · RLxF: Reinforcement Learning from World Feedback

Playing with Fire: What Transfers When RL Trains a Language Agent?

Scaffold-local gains, competence-gated transfer, and trainability transfer in Hanabi

Mahesh Ramesh^1* · Kaousheik Jayakumar^2* · Hemanth Ram³ · Pavan Thodima¹ · Ramani Duraiswami² · Dinesh Manocha² · Aniket Rege¹ · Emmanouil-Vasileios Vlatakis-Gkaragkounis¹

¹University of Wisconsin, Madison · ²University of Maryland, College Park · ³The University of Texas at Austin · *equal contribution

Three-panel overview: low base competence gives scaffold-local gains, high base competence transfers across scaffolds, and hard-task RL accelerates downstream convergence. — Whether RL gains transfer across scaffolds depends on the base model's scaffold-independent competence. Low-competence models improve only where they were trained; high-competence models transfer everywhere; hard-task RL speeds up later training even when the skill itself does not transfer.

THE CENTRAL QUESTION

When RL improves an LM agent in one scaffold, what actually transfers to another?

Agent benchmarks wrap models in scaffolds, and the same model can score very differently across them. A higher reward can reflect idiosyncrasies of one training setup rather than transferable skill.

What actually transfers?

We train language agents with RL to play Hanabi, a cooperative hidden-information card game, and evaluate the same game through three different prompt scaffolds. RL post-training separates into three distinct outcomes.

01 · SCAFFOLD-LOCAL GAINS

When the base model lacks usable task competence, RL lifts the trained scaffolds 4 to 15x, yet the held-out Watson scaffold, which demands internal game knowledge, barely moves.

Qwen3-4B, trained on Sherlock + Mycroft

02 · COMPETENCE-GATED TRANSFER

A model that already plays Hanabi without prompt-supplied rules transfers its RL gains. Trained on Mycroft alone, it improves on all three scaffolds, including both held-out ones.

GPT-OSS-120B, trained on Mycroft only

03 · TRAINABILITY TRANSFER

Hard state-tracking RL does not transfer zero-shot across scaffolds, but it reshapes computation: longer, more persistent rollouts, 4x better downstream Pass@8, and roughly half the RL steps to reach the same reward.

Qwen-ST, trained on FIREWORKS

One game, three scaffolds

The underlying Hanabi game never changes. What changes is how much support the prompt supplies: rules, legal moves, engine-computed deductions, and who has to maintain the belief state.

Watson least support

Public game state, visible hands, clue-derived own-card info, legal moves. No rules, no deductions. Strong play here means the model has internalized Hanabi, so Watson serves as the competence test.

Sherlock richest support

Everything Watson has, plus full Hanabi rules and engine-computed deductions over possible card identities from the Hanabi Learning Environment.

Mycroft most demanding

Only the previous turn's state plus all intervening moves. The model must reconstruct per-player belief states itself before acting, a direct test of state tracking and theory-of-mind reasoning.

Results

Transfer is gated by pre-RL competence, not model scale alone. And even when a trained skill does not transfer zero-shot, the checkpoint it produces is a better starting point.

Low game knowledge: gains stay scaffold-local

Qwen3-4B lineage · IQM over seeds 1001 to 1010 · Hanabi score out of 25

Model	Watson · held out				Sherlock · trained				Mycroft · trained
Model	2P	3P	4P	5P	2P	3P	4P	5P	2P	3P	4P	5P
Qwen3-4B	1.67	1.67	1.50	0.67	1.33	2.33	5.00	3.17	0.67	0.50	1.67	0.67
Qwen-ST	1.50	2.67	2.33	2.00	3.67	10.67	7.67	7.00	2.50	3.50	3.17	2.17
Qwen-Hanabi	2.00	2.00	2.50	2.33	11.00	14.67	13.67	13.83	12.33	12.83	13.50	12.67
Qwen-Hanabi+	2.67	2.33	2.50	2.33	5.33	11.83	14.00	12.83	12.17	13.67	14.33	13.33

RL lifts the trained Sherlock and Mycroft scaffolds 4 to 14x, but held-out Watson barely moves: the model does not acquire enough scaffold-independent Hanabi knowledge to transfer. Continued Mycroft RL even degrades Sherlock.

High game knowledge: gains transfer across scaffolds

GPT-OSS-120B lineage · Mycroft-only RL · IQM · Hanabi score out of 25

Model	Watson · held out				Sherlock · held out				Mycroft · trained
Model	2P	3P	4P	5P	2P	3P	4P	5P	2P	3P	4P	5P
GPT-OSS-120B	13.67	13.33	13.33	12.17	14.83	14.50	15.17	13.17	13.50	12.67	12.83	11.17
+ Phase I	13.83	13.33	13.33	12.33	15.33	14.67	14.50	12.83	14.00	12.83	11.83	12.00
+ Phase II	14.83	13.50	14.00	13.00	15.50	16.50	15.67	14.83	14.67	14.33	13.83	13.17
+ Phase III	13.17	13.33	14.00	12.83	16.67	15.33	15.33	13.83	13.50	13.17	13.33	12.33
+ Phase IV	14.00	14.00	14.17	13.67	16.83	16.17	16.17	15.00	15.33	14.67	14.33	13.67

A stronger base model trained on Mycroft alone improves on all three scaffolds. Cross-scaffold transfer is gated by pre-RL competence, not model scale alone. Training on the hard failure subset (Phase IV) elicits deliberation and gives the strongest transfer.

GPT-OSS-120B training curves across Phases I-IV: state-tracking reward, move-rating reward, mean Hanabi score per phase, and training tokens per turn. — GPT-OSS-120B across the four Mycroft training phases: state-tracking reward climbs steadily, mean Hanabi score improves on all three scaffolds, and Phase IV's hard failure data lengthens per-turn computation while trading off the move-rating reward.

Trainability transfer: hard-task RL is a better starting point

COMPUTATION STYLE

FIREWORKS RL lengthens completions from about 1.5K to 8K tokens as reward rises from near 0 to about 0.9. Initialized from Qwen-ST, downstream Mycroft RL reaches the same state-tracking reward in roughly half as many steps as a cold start, which lags from reward interference between the joint objectives.

HELD-OUT LOGIC · PI-LOGIC PASS@8

Base model

0.15

Hard-task RL

0.37

+ Hanabi RL

0.61

FIREWORKS state-tracking RL lifts a held-out, non-Hanabi logic task 4x, and continued Hanabi RL raises it further, even though the exact belief-reconstruction skill does not transfer zero-shot to Mycroft.

4-15x

in-domain gain on trained scaffolds for the low-competence 4B model, with almost no held-out improvement

~1/2

the RL steps to reach the same state-tracking reward when initializing from the FIREWORKS checkpoint instead of a cold start

Pass@1 for Qwen3-4B on FIREWORKS exact belief reconstruction before training, across 2 to 5 player games

BENCHMARK

FIREWORKS: strict state tracking

A 10K-example state-transition dataset across 2 to 5 player Hanabi games. The model sees the previous and current public state and must reconstruct the full hidden-belief state exactly. Reward is 1 only when every card belief is correct, 0 otherwise. The base model's Pass@1 is near zero.

We also release 18K Mycroft-scaffold gameplay samples with verifiable state-tracking rewards, enabling deterministic reward signals during Hanabi RL.

FIREWORKS RL training dynamics: average rollout completion length and FIREWORKS reward versus global training step. — FIREWORKS RL training dynamics: completion length grows from about 1.5K to 8K tokens as the strict state-tracking reward rises from near 0 to about 0.9.

Multi-stage training

Qwen3-4B trains in three stages: FIREWORKS state tracking (Qwen-ST), mixed Sherlock + Mycroft gameplay (Qwen-Hanabi), then continued Mycroft-only training (Qwen-Hanabi+). GPT-OSS-120B trains across four phases of Mycroft data with progressively expanding game coverage.

Checkpoint lineage diagram: the Qwen path from Qwen3-4B through Qwen-ST, Qwen-Hanabi, and Qwen-Hanabi+, and the GPT-OSS path from GPT-OSS-120B through Phases I-IV of Mycroft training. — Checkpoint lineage for both model families: the Qwen path through FIREWORKS state tracking and mixed-scaffold gameplay RL, and the GPT-OSS path through four phases of Mycroft training with expanding game coverage.

Takeaways

RL training scores can be deceiving: gains on one scaffold can be brittle elsewhere.

Injecting new knowledge through SFT or on-policy distillation beats doing RL from scratch.

Hard-task RL is useful when the model has the needed knowledge or receives it in context.

Hard-task RL enables faster downstream convergence, even when the skill itself does not transfer zero-shot.

Abstract

What does reinforcement learning teach an agentic language model: task-transferable competence or scaffold-specific behavior? We study this question in Hanabi, a cooperative hidden-information card game, by training and evaluating models across three Holmesian scaffolds that present the same game with different prompt support. A Qwen3-4B model with weak scaffold-independent Hanabi competence, trained with RL on the supported Sherlock and Mycroft scaffolds, achieves large in-domain gains but fails to transfer to held-out Watson, where rules are not provided in context. In contrast, GPT-OSS-120B, which starts with stronger scaffold-independent competence, improves on both held-out scaffolds after RL on Mycroft alone. To evaluate long-horizon state tracking we construct FIREWORKS, a 10K benchmark requiring exact reconstruction of per-player hidden card beliefs. Together, our results separate RL post-training into three outcomes: scaffold-local gains, competence-gated transfer, and computation-style and trainability transfer, where hard-task RL changes rollout behavior and improves downstream optimization even when the trained zero-shot skill does not transfer across scaffolds.

Citation

@inproceedings{ramesh2026playingwithfire,
  title     = {Playing with Fire: What Transfers When RL Trains a Language Agent?},
  author    = {Ramesh, Mahesh and Jayakumar, Kaousheik and Ram, Hemanth and
               Thodima, Pavan and Duraiswami, Ramani and Manocha, Dinesh and
               Rege, Aniket and Vlatakis-Gkaragkounis, Emmanouil-Vasileios},
  booktitle = {ICML 2026 Workshop on RLxF: Reinforcement Learning from World Feedback},
  year      = {2026}
}