Learning MegaGem, from self-play to price discovery

The 30-gem deck contains six gems of each color, and the terminal Value Display is the 15 gems dealt into players’ hands. From your five-card hand and the two face-up auction gems, 23 gems remain unseen and the opponents’ ten hand cards are drawn from them. Let $N_R$ and $N_Y$ be the terminal display counts for Red and Yellow, and $V$ the corresponding value dictated by the value chart A, which pays 4 points per displayed gem up to 5. Then, the EV is

Winning the Blue–Green auction changes your collection to Red–Red–Blue–Blue–Green, completing two of the missions for 25 points total. Factoring in the blue and green gems from the lot and their values on the display, we know that winning this lot is worth at least $25+2\cdot4+2\cdot4=41$. Because neither Alice nor Bob has more than 12 coins and you are first in tiebreak order, you can win the auction for 12 and earn at least $41-12=29$ in positive EV. Their lack of liquidity lets you take the auction much more cheaply.

Because you are last in tiebreak order, matching Alice’s bid is a losing action: to clear a predicted bid of $b$, you must bid $b+1$. Alice can bid at most 12 and Bob at most 6, so 13 is the minimum bid to guarantee the lot. However, this increase must still be weighed against the gem’s value and the option value of keeping that coin for the remaining auctions; it may make the lot no longer worth bidding on.

Both cards enter the Value Display at game end, so revealing Red or Purple does not change final gem values, but it changes what other players are willing to pay before then. Revealing Purple moves its public count from 2 to 3 and its current quote from 8 to 12. Bob already owns Yellow gems, so another Purple would also complete the 5-point Pendant mission. This reveal raises his visible reservation value for a future Purple from roughly $8+5=13$ to $12+5=17$. That can be useful if the goal is to induce Bob to spend his six coins, but it also makes a strategically important Purple harder to buy or deny, especially when you have only two coins and act last in ties.

Revealing Red has the opposite advantage. Moving Red’s quote from 4 to 8 has a null effect on public perception because there are no more red gems to be auctioned off (all 6 are either in a collection or will be in the value display). It also keeps Purple quoted at 8 while your hidden Purple guarantees that its terminal supply is higher than the table can see. This usually makes Purple easier to snipe later (for example, to block Bob’s pendant). Revealing Red does advertise more of your lead (making you a bigger threat to other players who want to block you), but its direct price impact is confined to a market that has already closed.

The 20 borrowed coins are matched by a 20-point repayment at scoring, so the principal itself creates no terminal value. Winning the loan for 10 changes your available cash to $C_{\mathrm{after}}=2+20-10=12$, but carrying those coins to the end would leave you 10 points worse off. For example, if you win and have 14 coins, you can guarantee winning the last two auctions by first paying 7 (beating Bob in the tiebreak) and paying 7 again (since you just won, you are last in the tiebreak and need to one-up Bob’s liquidity of 6).

Of the five cards still hidden, two are Purple, which puts a Purple gem’s expected value at $0.1(12)+0.6(16)+0.3(20)=16.8$.

With 2 coins, under rational play by your opponents, you would never be able to win the last two auctions. But for the reasons explained above, the loan gives you an out to win these last two, and because the EV suggests a positive trade, it is worth winning the loan auction.

Before the final lot, terminal-value accounting puts you at 77, Alice at 75, and Bob at 63. The Purple gem is worth 16 on Chart A; it would also complete Bob’s 5-point Pendant mission, but Bob has only 1 coin. The score branches are:

Your tie priority beats Bob, so a bid of 1 covers his branch. Alice, however, acts before you so 5 is the smallest bid that covers both stacks. Any bid from 6 through 12 buys the same gem while needlessly giving away score. The asset is fundamentally worth 16, but the correct price is 5 because the opponents’ remaining cash sets the clearing threshold.

Chess, Go, and heads-up poker are standard examples. One player’s gain is the other’s loss, so a value function $V(s)$ can describe the expected outcome from a position for either player, with the sign reversed for the opponent. This is the same object visualized by the evaluation bar beside an online chess board.

Under pure opposition, you must assume the opponent chooses whatever hurts you most. You therefore choose the strategy with the best worst case, the maximin. Your opponent does the converse, minimizing the maximum value you can obtain, the minimax. Von Neumann’s minimax theorem says these values are equal.

If neither player can improve by deviating alone, the joint policy is in Nash equilibrium. In a 2p0s game, its equilibrium strategies are also minimax. This supplies a clean progress measure: exploitability, or how much a worst-case opponent could gain by deviating. A lower exploitability means the strategy is becoming harder for any opponent to exploit.

Ordinary policy-gradient self-play can still cycle (think of rock-paper-scissors) in the latest policy. Practical systems therefore often average policies or train with fictitious self-play, exploiters, and wider populations, and these mechanisms can stop the fresh policy from forgetting old weaknesses.

General-sum games remove the requirement that the players’ payoffs add to a constant in every outcome.

1. In mixed-motive games, interests are neither perfectly opposed nor perfectly aligned. In the prisoner’s dilemma, both players prefer mutual cooperation to mutual defection, but each is individually incentivized to deviate. Settlers of Catan has the same flavor through beneficial trades, temporary coalitions, and coordinated attempts to block (or “plow”) a leader.
2. In fully cooperative games, every player receives the same payoff. Self-play can still learn arbitrary private conventions that fail when a new partner has learned a different convention. Hanabi is the standard example: two policies can each be effective with copies of themselves yet coordinate poorly with one another.

If you’re interested in the cooperative side of multi-agent RL, Arturo wrote a great hands-on account of training LLM agents to play Hanabi, including the communication conventions that emerge under self-play.

General-sum rewards give each player a different value for the same position. Multiple equilibria may exist, so different conventions or tradeoffs are common.

MegaGem adds more players. Score changes are not constrained to sum to zero, and an action can redistribute advantage among players in ways that barely affect the actor. A losing player can decide which leader wins. Two players can pressure the leader without sharing a formal team reward.

A single optimal bid also breaks down because best play is contingent on opponents. Against a player who always bids 0, bidding 1 wins cheaply; against a rational player, the same bid loses. If a kingmaking action barely changes the acting player’s own reward, a policy-gradient update receives almost no direct signal for which rival benefits.

We can measure deviation gains around a specified joint policy. No unique opponent-independent strategy supplies a universal reference point. In a general-sum game, the word “improvement” must name an opponent distribution, equilibrium-selection rule, or population game. The self-play reward describes the population that produced it.

First, deals are stochastic. A strong hand can raise terminal score without reflecting a better decision. Same-deal comparisons remove much of this variance.

Second, MegaGem has imperfect information. A player observes an information state consisting of its private hand and the public history; the hidden state remains unknown. Formally, the observation defines a set of hidden worlds still consistent with what the player has seen. New bids and reveals update the player’s belief over those worlds. A poker four-bet leaves the cards hidden while shifting the belief toward stronger hands.

This makes MegaGem a partially observable Markov decision process (POMDP) because the relevant value depends on the full belief state, not just the current board. The effective state space is therefore much larger and harder to estimate over.

Because bids are simultaneous, action selection also requires a distribution over clearing prices. For a candidate bid $b$, we need $P(\text{win}\mid b)$, the probability that the bid beats both opponents. Every bid trades the surplus retained after paying against the probability of winning.

Also, terminal gem values are fixed by the initial deal but revealed endogenously through play. Every private gem eventually enters the public Value Display, so the final display does not change. The order of reveals changes what each player believes before the remaining auctions. A reveal can therefore move perceived value even when it does not move terminal value.

For the common-value component of a lot, winning also carries information. If both opponents decline to pay your price, you may have overvalued the asset, the auction version of the winner’s curse. Private mission value weakens that inference because the same lot can rationally be worth different amounts to different seats.

We choose the teachers on two criteria:

• skill: The initial screen, run before the final benchmark ranking, compares Gemini 3 Pro, Gemini 3 Flash, and Claude Opus 4.5. A full-field score can conflate strategic skill with exploiting weaker opponents, so we also run a 24-game-per-model probe with three copies of each model. Clone-game scores also reflect how a policy changes total surplus, so we use this result only for screening.
• length and cost: Gemini 3 Flash is about fourteen times cheaper per game than Claude Opus. A blend supplies enough long rationales to diversify the data without making every game Opus-priced.

For production collection, we keep Gemini 3 Flash and replace the screened Claude Opus 4.5 with its newer Claude Opus 4.6 successor. We choose the successor based on model family; the screen itself tests only Claude Opus 4.5.

Keeping only the winner’s traces would waste data and select the luckiest copy of an otherwise identical policy. It would also remove the lower liquidity and slower tempo states that Qwen3-4B-Instruct needs to handle. So instead, we consider the top two models’ traces from each game while removing unparseable or illegal turns (only 10 training turns fail those validity gates). The final split contains:

Checkpoint selection uses a fixed 6-model panel on the 10 validation deals. Qwen faces two copies of each opponent and occupies all three seats, for 180 panel games, plus 10 self-play games.

The self-play mean is only a sanity check, since all three copies co-evolve, but it does show that the model hasn’t degenerated.

The configurations use the same DAPO-style clipped objective; the labels below describe what changed in the training pool or schedule. Snapshots are lagged copies of the learner, and SFT anchor is the frozen step-0 policy. The two snapshots-plus-Flash rows are separate replications of the same basic pool. The frontier-API pool substitutes Gemini 3.1 Pro, Claude Opus 4.6, and Claude Sonnet 4.5; the 400-step run doubles the usual training horizon; seat rotation moves the learner through all three seats; and the final mixed pool combines the SFT anchor, lagged snapshots, and a decaying scripted-heuristic share. The figure abbreviates that final pool as “heterogeneous.”

The heuristic contrasts use 60 matched seed groups with eight samples each, except the 24-game seat-rotation diagnostic. The optimizer can learn a narrow exploit.

For player $i$, playing against $j$ and $k$, define competitive margin as $m_i=s_i-\frac{s_j+s_k}{2}$. The terminal reward is

We calibrate the scale 19.6 so the median absolute margin maps to 0.5 via $\tanh\left(\frac{10.75}{19.6}\right) \approx 0.5$. This compresses extreme wins and losses while preserving their order via monotonicity. We add a shaping channel (weighted by 0.01) that assigns intermediate credit from reconstructed public-score changes between rounds. Then, a terminal correction aligns its cumulative accounting with the final margin. Illegal actions receive $-0.5$.

Let $r_u$ denote the terminal, legality, or shaping reward assigned at turn $u$. For each trainable turn $q$, define the return $G_q=\sum_{u\ge q}r_u$ and subtract an exponential-moving-average baseline $b_q$ indexed by game phase and seat role. Let $\mathcal G$ contain all trainable turns from the eight sibling games. Then, we define the advantage here as

Only turns from the trainable seat receive gradients.

We sample at temperature 1.0 and top-$p=0.95$, then update a rank-32 LoRA with learning rate $10^{-5}$ and KL coefficient 0.01.

The SFT-anchor mixed-pool run completes 100 optimizer steps with 192 games per rollout generation. Taken together, the diagnostics below rule out a dead adapter, broken reward ordering, or format collapse as explanations for the null effect.

A second problem is assigning those wins to individual decisions.

trajectory-level credit

GRPO is easy to engineer in this setup. Its statistical fit is weak for this task. It avoids training a separate critic, yet this setup is far from the typical single-turn GRPO task. A game lasts roughly 15–25 rounds of interleaved bids, reveals, loans, and investments. Most reward arrives at the end, so an early bid and a late reveal can inherit nearly the same verdict even though either may have changed every state that follows. The same-deal comparison removes hand luck. It leaves within-trajectory credit unresolved.

We don’t claim that GRPO cannot work on long-horizon or multi-turn tasks. Dense verifiable rewards, process supervision, much larger groups, or environments in which one decision dominates the outcome can still make trajectory-level comparison useful.

state-value probe

For each fixed board $b$, let $\bar Y_b$ and $s_b^2$ be the sample mean and within-board variance of its rollout returns. Because the observed variance of board means includes finite-sample noise, we estimate the true between-board variance as

with within-board variance $\hat\sigma^2_{\text{trajectory}}=\mathbb E_b[s_b^2]$. In the probe, 99.8% of the estimated variance lies among trajectories from the same board, while only 0.2% lies between corrected board means. The within-board term contains both action effects and continuation noise; it isn’t a clean action signal. A state-value critic must learn the much smaller between-board component from labels dominated by within-board variation.

Under relative-margin self-play, all three copies benefit from the same favorable deal, which suppresses most common board advantage. The fixed-heuristic probe allows repeated outcomes to recover more of the position’s value.

The probe argues against learning $V(I)$ from these particular relative-margin labels but doesn’t rule out state values in general. However, an action-conditioned value $Q(I,a)$ remains viable because it could still distinguish bids within the same state.

rollout-$Q$ distillation with piKL

The first plausible fix is a piKL-style target, inspired by Meta’s work on Diplomacy, a seven-player game outside the 2p0s setting. It builds a search distribution that favors bids with higher action value $Q(I,a)$ while paying a KL penalty for moving too far from the SFT policy. Here, $Q(I,a)$ means the expected outcome after taking bid $a$ from information state $I$ and playing the game to completion.

We estimate $Q$ by simulation. The acting player cannot see the opponents’ hands or the undealt deck order, so we sample several worlds $w$: complete assignments of the hidden information that remain consistent with the player’s hand and the public history. In each world, we force candidate bid $a$, let the SFT continuation policy $\pi$ finish the game in the acting seat while policy $\omega$ controls the opponents, and read off the terminal score margin:

Here, $\xi$ is the continuation of the game generated by $\pi$ and $\omega$. We score every candidate on the same sampled worlds to reduce variance. The estimate $Q$ still depends on $\omega$ because changing the continuation opponent can reorder bids even when $(I,a)$ stays fixed.

piKL combines these action values with sampled action frequencies $\hat\tau$ from the SFT policy, upweighting high-$Q$ bids while remaining KLed to the SFT policy:

The temperature $\lambda_{\mathrm{KL}}$ controls the strength of the anchor. As $\lambda_{\mathrm{KL}}\to\infty$, the $Q$ tilt vanishes and the distribution returns to the SFT frequencies $\hat\tau$; as $\lambda_{\mathrm{KL}}\to0$, it concentrates on the sampled candidate with the largest $Q(I,a)$. The $\lambda_{\mathrm{KL}}=0$ arm in the table implements that limiting argmax directly.

We label decisions with this $Q$, build piKL targets at several $\lambda_{\mathrm{KL}}$ values, and distill them with LoRA SFT. Each candidate then occupies every seat against two frozen SFT copies over the same 120 games, 40 per seat.

Mean-margin arm intervals use the normal approximation; paired intervals use a $t$ interval over matched games.

At 40 bid nodes with up to six candidates each, changing only the continuation opponent from self-play to base Qwen reduces median action-rank correlation with the self-play ordering from 1.0 to 0.43, while top-action agreement falls from 100% to 27.5%. The sign of each candidate’s advantage over the reference bid still agrees 72.5% of the time. The probe is too small, and the alternative opponent too weak, to establish the mechanism by itself. It does show that fine $Q$ rankings can depend strongly on the continuation policy.

A second five-arm sweep replaces the self-play continuation with Gemini 3 Flash. Relative to its $\lambda=\infty$ control, the $\lambda=1$ and argmax contrasts are +6.10 [0.38, 11.82] and +5.81 [0.70, 10.93]. These comparisons are exploratory, uncorrected for multiplicity, and unreplicated.

On a separate 43-node offline proxy, aggregating $Q$ over five fixed continuation types also never beats the single-self-play-$Q$ baseline. This diagnostic uses an offline proxy and provides no live policy comparison.

action-critic pilot

GRPO’s group return cannot identify which bid changes the outcome. PPO could instead update from an action-specific advantage supplied by a learned critic. Before paying for an actor update, we test whether such a critic can rank candidate bids reliably enough to be useful.

critic target

The actor produces a bid using $I$ before the opponents’ bids are known. Once the auction resolves, those bids become public. During training, a critic can use this newly revealed information even though the actor never receives it at inference (centralized training with decentralized execution). We can fix the observed opponent bids $\mathbf b_{-i}^{\mathrm{obs}}$ to determine what happens when only our bid changes.

Most raw $Q$ variation reflects whether the state itself is favorable. To isolate differences among bids, we center each target over the retained candidate set $C(I)$:

This target centers candidates with equal weight. A PPO update would instead require an advantage weighted by the actor policy.

rollout labels

At each treasure-bid state reached by the actor, we sample $K=16$ bids and compress them to 6 diverse candidates. The selection retains the played bid while approximately preserving both the policy’s sampled mass and the numerical range of bids. We score every candidate on the same sampled worlds and random seeds, and hold the opponents’ observed bids fixed. After the forced bid, a deterministic frozen SFT policy completes the game. The labels therefore apply to that frozen continuation policy; an updated PPO actor would induce a different distribution.

world-sampling budget

For 32 games, we evaluate each candidate with $M=64$ hidden-world samples to get 379 bid states and ~1.5k candidate rows. Because smaller estimates reuse prefixes of the same 64 worlds, we can compare 4, 8, 16, or 32 worlds without further rollouts. “Pair-sign agreement” measures whether actions separated by a $Q$ gap of at least 0.05 and 95% paired sign confidence retain their ordering while regret measures how much reference $Q$ is lost by selecting the lower-sample-budget argmax.

Since the later training set retains confidently separated pairs, we adopt 32 worlds at half the rollout cost.

structured-model probe

Across all 379 calibration states, the mean spread from worst to best retained action is 0.2149 in $\tanh$-margin utility, and the estimated best action improves 0.0931 over the SFT-weighted candidate mean. That second number is an in-sample rollout-oracle bound, not realizable policy uplift. The average gap between the top two is only 0.0599, and the SFT policy’s played action is already best 36.1% of the time.

Most candidate-$Q$ variation is irrelevant to choosing a bid: 84.8% lies between states, while only 15.2% separates actions within a state. To be clear, this split measures paired rollout-$Q$ estimates grouped by decision state while the earlier 99.8% figure measures raw whole-trajectory variation across replicas grouped by initial deal. PPO can use only the within-state component because a state-wide offset raises or lowers every candidate together.

Across 4 split seeds, 4-fold cross-validation fits small structured models to multi-action states from the $M=64$ reference labels. In the table, “uplift” is offline $Q$ improvement over the SFT candidate mixture.

Each non-control row is a gradient-boosted tree ensemble fitted to the candidate-$Q$ labels. The centralized model receives the observed opponent bids, while the actor-visible model does not. Two negative controls either shuffle the targets or drop the candidate bid, leaving a model that can predict only common state value. “Filtered pair accuracy” scores only the confidently separated action pairs defined above.

4B critic results

We reuse three confidence gates from calibration: gap $\ge0.05$, 95% sign confidence, and at most 6 pairs per state. The SFT-initialized critic’s Bradley–Terry loss ranks the better bid above the worse one, while a Huber regression term trains the numeric centered value at weight 0.25.

Even with the preregistered gates, every neural run misses the 55% argmax threshold and the 40% linear-bid baseline.

compute decision

The next meaningful rung uses 128 games and roughly 1,485 decision states. It costs an estimated 328 H200-hours, with $\pm40\%$ uncertainty. Reaching 1,024 games and then evaluating on the sequestered 128-game final test approaches 3,000 H200-hours.

PPO could in principle improve beyond a fixed expert, but it must first compress expensive rollouts into small critic differences and then refresh them as the actor’s state distribution moves.

We therefore stop after the critic diagnostics: no production label corpus and no actor update.

One final search variant keeps the terminal rollouts but replaces the opponents’ current bids with deterministic predictions from a fitted market model. The best arm changes paired margin by only +0.58 [−5.6, +6.8], therefore unresolved.

A separate diagnostic asks whether a much better point predictor would solve the problem. A gradient-boosted tree trained on ~3.2k logged Gemini 3 Flash bids reaches an out-of-fold mean absolute error (MAE) of 1.10 coins with essentially zero bias (all variance). However, it still predicts the wrong side of the actual win/loss threshold on 26.1% of auctions. 55% of auctions resolve within one coin, where Flash’s own sampling noise straddles the boundary even when the conditional mean is accurate.

We check ordinary chosen-action calibration on the games used for the myopic selector evaluation. Across ~1.8k seat-0 treasure bids against 2x Gemini 3 Flash opponents, mean predicted win probability is 40.3% and empirical win frequency is 36.2%. The model is close at the extremes but isn’t as calibrated in several middle-probability bins, which should roughly be expected.

For each seat-0 decision, the final display reveals the treasure lot’s realized value $V$. Let $p$ be the highest recorded opponent bid. The ideal clearing-price surplus is $S^*=\max(0,V-p)$. For a counterfactual bid $b$, we resolve the auction against the opponent bids, giving realized surplus $S(b)=(V-b) \cdot \mathbf 1[b\text{ wins}]$. Then just define decision regret as

Zero regret can be unattainable when the budget cannot beat a profitable clearing price, or when $V>p$ but losing the tiebreak makes the cheapest winning bid $p+1$ instead of $p$.

We introduce a control variate to reduce auction-resolution noise: a decision should not receive full credit because an opponent happens to sample below its usual price, or full blame because the opponent samples above it.

For each treasure decision $d$, let $y_d=1$ if the bid wins and $y_d=0$ otherwise, let $p_d$ be the model’s predicted win probability, and let $V_d-b_d$ be the lot’s estimated surplus. Define the decision-level resolution surprise as $x_d=(y_d-p_d)(V_d-b_d)$. This converts auction luck into score units: a surprising outcome matters more when the lot’s surplus is larger. For arm $a$, game $g$, and seat $k$, sum the surprises over that seat’s decisions:

For the adjustment to be unbiased, the model must satisfy the stronger, value-weighted calibration condition

where $I_d$ is the information available at the decision. Ordinary calibration of $p_d$ alone isn’t sufficient because the surplus $V_d-b_d$ varies across decisions. For each matched seed, define the seatwise luck contrast $Z_{g,k}=X_{\mathrm{on},g,k}-X_{\mathrm{off},g,k}$ and the paired margin difference $D_g=\text{margin}_{\mathrm{on},g}-\text{margin}_{\mathrm{off},g}$. We regress

The intercept $\tau$ is the adjusted treatment effect. In a paired pilot, this regression explains 42% of the paired variance, a 1.72× reduction. No direct validation of the stronger value-weighted calibration condition is available, so later gates report the raw paired effect as the primary evidence and the adjusted estimate as model-assisted corroboration.

Adjustment removes about 3 points of adverse resolution luck, yielding −1.88 [−6.57, +2.81]. The adjusted history-repair arm is harmful at −5.16 [−9.90, −0.41]. Neither adjusted estimate overturns the raw result.

ExIt alternates between a policy that visits states and an expert that improves the action taken in those states. If the current policy $\pi$ induces a state distribution $I\sim d_\pi$, the expert supplies

and the resulting dataset $\mathcal D_E$ trains a new policy by imitation:

The new policy can generate another state distribution, which the expert can label again, giving the loop $\pi_k\rightarrow d_{\pi_k}\rightarrow\mathcal D_E\rightarrow\pi_{k+1}$. Unlike ordinary SFT from a fixed teacher corpus, the supervision is concentrated on states the current policy actually reaches and on actions the expert would change.

1. deviations: replace the SFT bid with the analytic expert’s target and expose the quantities that determine it: $\hat V$, opponent price means, $P(\text{win})$ for both the SFT bid and the selected bid, budget, and paced surplus
2. pass-throughs: retain the raw SFT response when the selector agrees or finds less than one coin of EV lift, so the adapter does not relearn every sound decision
3. loan and investment anchors: retain the SFT policy’s original financing responses, so an improvement to treasure bidding does not erase the liquidity discipline that makes it work

Pass-through and financing anchors come from all three seats. Only seat 0 contains selector deviations, so all-seat anchors reduce the risk that the adapter learns a player_id = 0 shortcut.

In one game, round 9, seat 0 has 22 coins and a public collection of three Green and one Yellow gems. The current auction offers two more Green gems, enough to complete a mission, and the ordinary SFT policy samples a bid of 22.

The frozen expert values the lot at 30.59 coins after the calibration correction, including the 10-point mission bonus it would complete, and predicts opponent mean bids of 10 and 11. A bid of 22 wins with certainty, but once each coin’s option value is priced in, its paced utility is negative. A lower bid can preserve most of the win probability at a far better price:

The final 12 percentage points of win probability cost too much, so the expert shades the bid to 13. The training target reads:

This treasure is worth about 31 to me net of bias. 22 would overpay once each coin’s option value is priced in (budget 22); shading to 13 keeps the win probability at ~88% and earns more surplus per coin. Expected surplus favors 13.

{"bid": 13}

That prose comes from a template filled with four logged quantities: $\hat V$, both bids, win probability, and remaining budget. Every deviation target initially has the same synthetic voice.

The log preserves neither the SFT policy’s free-form rationale nor a student response. This row therefore illustrates the supervision target.

Closure measures how much of the expert’s utility improvement the student recovers on held-out deviation states, where the denominator is positive by construction. A value of 0 means no utility gain over the recorded SFT bid; 1 means equal utility to the expert, not necessarily the same bid; values above 1 are possible. The “SFT reference” below is a fresh stochastic response to the same prompts, so it need not reproduce the recorded $b_{\mathrm{SFT}}$ and can score above zero.

Across separate experiments, adjusted point estimates leave a 2.99-point gap; the direct within-run selector increment is +3.95 [−0.83, 8.74]. Applying the two measured distillations’ 73–78% retention range to that gap suggests only ~2.2 points from another round, but this estimate is just an extrapolation.

Here, the model plays seat 0 against GPT-5.5 and Gemini 3.1 Pro. By round 8, it has 14 coins and a collection of Yellow and Purple. A Blue gem would complete the 5-point Blue–Yellow mission, and revealing the Blue in its private hand would move Blue from 4 to 8 points per gem. One opponent has only 6 coins, while the model has tiebreak priority over the other.

The model writes:

Winning this Blue gem would immediately complete the “specific_2_by” mission (Blue + Yellow) for 5 coins and put me one step closer to the “specific_3_bgp” mission (Blue + Green + Purple) for 10 coins.

A bid of 6 ensures I beat Player 1’s maximum possible bid. If Player 2 bids 6, I win on tiebreak. If Player 2 bids higher, they are spending a significant portion of their capital early. Given the mission rewards and the value of the gem, 6 is a strong but efficient bid that preserves my coins for future rounds.

{"bid": 6}

The bids are 6, 6, and 5. The model wins the tie, claims the mission, and reveals Blue. Ignoring any later increase in Blue’s price, the immediately visible surplus is already at least

Recognizing both the value/surplus of Blue while accounting for covering price and coin preservation is what allows it to eventually win the game 93–89–59.

The same game produces a cleaner test of state tracking in rounds 13 and 14. Both opponents have exhausted their cash. In round 13, the model is first in tiebreak order, so it recognizes that a zero bid wins the Blue gem:

Player 1 has 0 coins, Player 2 has 0 coins. Neither can bid above 0!

I’m first in tiebreak order, so if we all bid 0, I win.

This is a free win at 0 coins! Since both opponents have 0 coins, they can only bid 0. I have tiebreak priority, so bidding 0 guarantees I win the Blue gem for free.

{"bid": 0}

All three players bid 0, and the model wins. Winning moves it to the back of the tiebreak order. On the next auction, a two-gem Yellow–Blue lot, it updates rather than repeating the previous action:

Player 2 > Player 1 > Me. So if everyone bids 0, Player 2 wins. I need to bid at least 1 to guarantee winning.

Bidding 1 is clearly optimal—it guarantees the win while costing almost nothing.

{"bid": 1}

The other players again bid 0, so the model buys both gems for 1, and these two purchases contribute to the same 93-point win above.

In another game, the model plays seat 1 against Gemini 3.1 Pro and GPT-5.5. In round 13 it has 14 coins, while GPT-5.5 has 7 and acts earlier in ties. Purple would complete the 5-point Red–Purple mission, so the model chooses the exact guaranteed price:

Player 2 has 7 coins and is ahead of me in the tiebreak order. To guarantee a win against Player 2, I must bid 8.

Spending 8 now for a gem that is worth at least 12–16 coins (plus the 5-coin mission) is a net positive, and I still retain 6 coins for future rounds.

{"bid": 8}

The bids are 0, 8, and 7, so the model wins Purple and the mission. It now has only 6 coins. The next card is a 10-coin loan, and its rationale connects that financing decision to the remaining mission:

Winning this loan would provide immediate liquidity (10 - bid) to compete for the remaining 4 auctionable gems, which is crucial because I need a Yellow gem to complete the “specific_3_rgy” mission (10 coins) and a second Purple for the “same_2_purple” mission (5 coins). However, bidding too high on a loan is counterproductive as it increases my end-game debt without adding net value.

{"bid": 5}

The loan bids are 2, 5, and 4. Winning raises the model’s available cash to $6+(10-5)=11$. The next auction is another Purple. GPT-5.5 still has 7 coins and the better tiebreak position, so the covering price is again $7+1=8$:

Player 2 has 7 coins max. They might bid up to 7. I should bid 8 to guarantee winning over Player 2’s maximum.

Given the expected surplus-maximizing bid is 8, and this is worth ~17+ to me, bidding 8 is clearly optimal.

{"bid": 8}

The bids are again 0, 8, and 7. The model wins its second Purple and the 5-point mission. It finishes first, 72–66–49, with

This sequence is not enough either: an exact covering bid creates a liquidity shortage which a subsequent loan repairs to fund a second exact covering bid that wins the mission.