1 Introduction↩︎

Learning to play imperfect-information card games well is a long-standing testbed for game AI. Games such as poker, DouDizhu, Mahjong, and Gin Rummy hide the opponent’s hand and the order of the deck, so a player must act under uncertainty, plan over a long horizon, and adapt to whoever sits across the table. The dominant recipe for building agents in these games is deep reinforcement learning (RL) with self-play, which has produced superhuman play in several of them [1]–[3]. Behind those headline results, though, a practitioner who wants a small and fast agent runs into two practical problems that the headline results do not solve.

The first is the opponent bottleneck. An RL agent is only as good as the opponents it practices against. Training against a weak fixed opponent teaches a weak ceiling, and pure self-play can chase its own tail, cycling through strategies that beat the last version without getting stronger in any absolute sense. The second problem is that, for most of these games, the practitioner has no cheap absolute yardstick. Strength is usually measured against random play or against the agent’s own past checkpoints, both of which can flatter a mediocre agent. As a result, the design choices that actually build a strong lightweight agent (which RL algorithm, how to shape the reward, how to schedule opponents, how to represent the state, which checkpoint to ship) are tuned against references that are either too weak or constantly moving. The field has many strong single agents but few clean, controlled accounts of which of these choices matter and why. If we had a fixed, strong, and cheap reference opponent, we could measure these choices cleanly and turn them into a reusable recipe.

Existing lines of work do not fill this gap directly. The famous self-play systems [4]–[6] answer what is possible with very large compute, not which lightweight choices matter on a single GPU. Search and equilibrium methods such as counterfactual regret minimization and its descendants [7]–[9] produce extremely strong play, but they are a different family of method (search over the game tree), and they do not tell you how to set the reward or curriculum for a compact reactive policy. Prior Gin Rummy work studies single methods in isolation, for example temporal-difference self-play [10] or hand-tuned heuristic and ensemble agents [11], rather than a controlled head-to-head of training choices graded against one fixed expert.

We take a deliberately simple position. For Gin Rummy we build a strong, fixed, deterministic expert that solves the one subproblem with a clean optimum (meld decomposition, that is, the lowest-deadwood way to arrange a hand) and otherwise plays principled endgame moves. It is not a game-theoretic optimum for the full imperfect-information game, and we do not claim it is; what matters is that it is strong, reproducible, and cheap, so it makes a fair yardstick. We never train against it. We then run more than one hundred controlled experiments that change one factor at a time and grade every result against this fixed expert. This paper makes the following contributions.

• A reproducible gold-standard expert benchmark for Gin Rummy, and the empirical finding that strong play almost never gins. The expert gins in 0.7 to 1.7 percent of games and wins by knocking early with low deadwood. This single fact reframes how reward should be designed for the game.

• A controlled, apples-to-apples study, graded against the fixed expert, of which lightweight training choices actually help. Trust-region updates, a knock-first reward, a rising opponent curriculum, warm-starting, and keeping the best checkpoint each help; learned state embeddings, dense step rewards, imitation learning, and a live large-language-model (LLM) opponent each fail, each for an identified reason.

• The result that reward shaping cannot induce the human-intuitive but losing habit of chasing gin. Paying three times more for a gin than a knock leaves the gin rate under one percent, the same as not rewarding gin at all. We connect the parallel failure of imitation learning to causal confusion.

• Evidence that the ceiling is information-bound, not capacity-bound. Across MLP width and depth, convolutional, permutation-invariant set, and recurrent encoders, win-rate against the expert stays in a narrow band with overlapping confidence intervals, and the ranking is robust across training recipes. A determinized information-set Monte-Carlo search, graded fairly (hidden cards re-dealt), is in fact weaker than our trained agents and loses to them head-to-head; an oracle variant that may see the hidden cards reaches far higher. The gap between the two quantifies the value of the hidden information, which is what sets the ceiling.

• A game-agnostic release of the pipeline, with the method reproduced on a second game, Leduc Hold’em, where a counterfactual-regret optimum is computable: a tabular learner graded against that optimum reaches near parity, showing the expert-yardstick study is not tied to Gin Rummy.

2 Related Work↩︎

2.0.0.1 Deep RL for imperfect-information card games.

Self-play deep RL has mastered several hidden-information card and tile games. Neural Fictitious Self-Play approximates a Nash equilibrium by mixing a best-response network with an average-policy network [1]. Suphx reached expert human level at Mahjong with global reward prediction and oracle guiding [2], and DouZero mastered the three-player game DouDizhu by combining deep networks with parallel Monte-Carlo self-play [3]. RLCard provides a common toolkit and environments for this line of work, including Gin Rummy [12]. Within Gin Rummy specifically, [10] compared temporal-difference self-play with coevolution, and the EAAI undergraduate challenge produced strong hand-tuned and ensemble agents [11]. These works each introduce a single strong agent. We differ in goal: rather than proposing one more agent, we hold a fixed expert constant and use it to measure, in controlled experiments, which training choices make a lightweight agent stronger.

2.0.0.2 Self-play, leagues, and opponent curricula.

AlphaGo Zero and AlphaZero showed that pure self-play with search can reach superhuman play in perfect-information games [4], [13]. AlphaStar added a league with prioritized fictitious self-play (PFSP), sampling harder opponents more often [5], and OpenAI Five scaled self-play to a complex video game [6]. Curriculum learning more broadly orders training from easy to hard [14], and a large literature studies curricula for RL [15]. We adopt a small opponent curriculum and a PFSP variant, but our question is not how far self-play scales; it is which curriculum and checkpoint choices matter at small scale, measured against a fixed expert rather than against the moving target of self-play.

2.0.0.3 Search and equilibrium solving.

Counterfactual regret minimization [7] and its deep variants [9] underpin the superhuman poker agents Libratus and Pluribus [8], [16], and OpenSpiel collects many such algorithms and small benchmark games [17]. Determinized search, or perfect-information Monte-Carlo, samples the hidden state and searches the resulting perfect-information games; information-set Monte-Carlo tree search (ISMCTS) is the canonical form [18], [19]. We use these as baselines rather than as our method: we run a determinized ISMCTS for Gin Rummy, and NFSP and CFR on a second game, and we use the family to frame the ceiling we observe. A key point we make precise later is that a determinized search graded fairly (without seeing the hidden cards) is far weaker than the same search given oracle access.

2.0.0.4 Network architectures for card observations.

A hand is an unordered set of cards, which invites permutation-invariant encoders such as Deep Sets [20] and self-attention [21]; convolutional stacks and recurrent networks [22] are the other standard choices for structured or sequential inputs. We hold our training recipe fixed and sweep these encoders to ask whether representation, rather than reward or curriculum, is the lever that breaks the ceiling.

2.0.0.5 Reward shaping and imitation learning.

Potential-based reward shaping preserves the optimal policy while changing the learning signal [23]; we use this view to interpret why some shaped rewards help and others hurt. DAgger reduces imitation learning to no-regret online learning by querying an expert on states the student visits [24]. We find that imitation collapses here, and we attribute the collapse to causal confusion, the phenomenon in which a cloned policy latches onto spurious correlates of the expert action and breaks under distribution shift [25].

2.0.0.6 Language models as game players.

Cicero combined a language model with planning and RL to reach human-level play in Diplomacy, a game built on negotiation [26]. Motivated by such results, we test a modern instruction-tuned LLM [27] as a live opponent inside the training loop. We find it plays competently but is far too slow to serve the millions of moves that RL needs, which is itself a useful negative result for anyone considering live LLM opponents.

2.0.0.7 Action masking and tooling.

Invalid-action masking is the standard way to keep a policy-gradient agent inside the legal action set [28]. We build on PettingZoo for the multi-agent interface [29], Stable-Baselines3 for PPO and TRPO [30]–[33], and a paged-attention server for LLM inference [34].

3 Problem Setup↩︎

We study two-player Gin Rummy as implemented in RLCard and exposed through PettingZoo as gin_rummy_v4 [12], [29]. Figure 1 shows how the pieces fit together: a masked RL learner, a curriculum of opponents, and the fixed expert that grades agents without ever entering training.

3.0.0.1 The game.

Gin Rummy is a two-player draw-and-discard game played with a standard \(52\)-card deck; each player holds ten cards.

• Melds and deadwood: group cards into runs (three or more consecutive cards of a suit) and sets (three or four of a kind); the ungrouped cards are deadwood, and a card cannot serve in two melds at once.

• Turn: draw the top of the stock or the discard pile, then discard one card face up.

• Ending: a player may knock once its deadwood is at most ten, or declare gin at zero deadwood, which ends the hand.

• Score: the knocker scores the opponent’s deadwood minus its own; gin earns a fixed bonus, and a defender whose deadwood is no larger undercuts and scores instead.

Winning therefore rewards knocking early with low deadwood far more than holding out for the rarer, riskier gin.

3.0.0.2 Observation and actions.

The agent sees a \(4\times52\) binary tensor: its own hand, the top of the discard pile, the other visible discards, and the set of unknown cards (in the opponent’s hand or the stock). It never sees the opponent’s hand. The action space has \(110\) discrete actions (draw, pick up, declare, and one discard or knock per card), and at every step the environment supplies a binary mask of the legal actions.

3.0.0.3 Single-agent reduction.

We turn the two-player game into a single-agent learning problem: one seat is the learner and the other is an opponent drawn from a curriculum (described below). The wrapper steps the opponent to completion between the learner’s decisions and rotates seats across games so that neither position is favored. This reduction is what lets us swap in any opponent, including the fixed expert for evaluation and the LLM for the live-opponent test, without changing the learner.

3.0.0.4 Masked actor-critic.

Our policy is a masked actor-critic shared by PPO and TRPO [31], [32]. We compute action logits, then set the logits of illegal actions to a large finite negative value before the softmax. We use a large finite value rather than negative infinity on purpose: PPO tolerates infinities, but the conjugate-gradient and KL computations in TRPO produce NaNs on them, so a large finite fill gives near-zero probability to illegal moves while keeping the same network usable by both algorithms. At evaluation we always take the highest-probability legal action. We never fall back to a random legal move, because a random fallback would quietly inflate measured strength by hiding states where the policy is undecided.

4 The Gold-Standard Expert↩︎

The center of our method is a fixed reference opponent that is strong, cheap, and fully reproducible. We build it from one exact component and a few principled heuristics, and we are precise about which is which.

4.0.0.1 Exact melding.

Given a hand, the lowest achievable deadwood is a well-defined optimization: enumerate the valid ways to group cards into runs and sets, and keep the arrangement with the least leftover value. We solve this exactly using RLCard’s meld enumeration, memoized over hands so that the whole benchmark runs quickly. This is the one place where the expert is provably optimal, and it is optimal only for this subproblem, not for the full game.

4.0.0.2 Principled endgame.

On top of exact deadwood, the expert draws the up-card only when doing so strictly lowers its best achievable deadwood, discards the card that minimizes resulting deadwood (breaking ties by shedding the highest-value card), and knocks as soon as it legally can, declaring gin only when gin is reachable without delay. These are sensible, deterministic rules. They are not tuned to the opponent and they do not reason about the opponent’s hidden cards, so the expert is not a game-theoretic optimum.

4.0.0.3 Why this is the right yardstick.

A game-theoretic optimum for imperfect-information Gin Rummy is not available at low cost, and we do not need one. What a controlled study needs is a reference that is (i) strong enough to separate good agents from weak ones, (ii) identical in every comparison, and (iii) cheap enough to grade thousands of games. Our expert satisfies all three. As we show next, it beats every agent we trained by a wide margin, so it is a meaningful ceiling, and because it is fixed and deterministic, a win-rate against it is directly comparable across every experiment in the paper. We use the expert only to score agents; it never appears in the training loop, so no result below is contaminated by training against the thing we measure with.

5 Training Methods Studied↩︎

All learning agents share the masked actor-critic above. Around it we vary one ingredient at a time. This section describes the ingredients; the next two describe how we evaluate them and what we found.

5.0.0.1 Algorithm.

We compare PPO, which clips the policy update, with TRPO, which bounds the update by a trust region on the policy’s KL divergence [31], [32]. Both use generalized advantage estimation [33] and the same network, curriculum, and reward, so any difference is the algorithm.

5.0.0.2 Reward shaping.

The native reward is sparse: a score at the end of the game. We study shaped rewards that change the relative payoff of knocking and ginning, and a small dense bonus for reducing one’s own deadwood. We read these through the lens of potential-based shaping [23]: a shaping term that tracks real progress should help, while one that rewards a proxy can pull the agent off the true objective.

5.0.0.3 Opponent curriculum.

We schedule opponents in three stages: first random play, then a growing pool of past checkpoints, then a mix that adds self-play. The pool can be sampled uniformly over recent checkpoints or with a PFSP weighting that practices more against opponents the agent is currently losing to [5]. The stage boundaries are set as fractions of the training budget and synchronized across parallel workers through a small shared state file.

5.0.0.4 Warm-start and keep-the-best.

We optionally warm-start a run from a strong earlier agent rather than from scratch. During training we evaluate against the hardest reference every fixed number of steps and save a copy whenever the agent improves, then ship that best checkpoint instead of the final one. The motivation, which the results confirm, is that training in a shifting self-play environment often drifts past its own peak.

5.0.0.5 State representation.

The native observation is a sparse \(4\times52\) tensor. We test whether a compact dense embedding helps by squeezing it into a small vector two ways: one learned to place game states that lead to similar outcomes near each other, and one where an LLM judges state similarity and we fit an embedding to those judgments. We test several embedding sizes and both frozen and unfrozen variants.

5.0.0.6 Imitation and dense supervision.

We test DAgger, which trains the student to copy the expert’s action on the states the student actually visits [24], and a short-horizon dense reward that scores each move by agreement with an evaluator over the next \(L\) steps. Both are popular shortcuts for injecting expert knowledge.

5.0.0.7 Live LLM opponent.

We test whether a modern instruction-tuned LLM can serve as a strong in-the-loop opponent. Because a single RL run issues tens of thousands of opponent queries, we serve the model through a caching CPU master over a pool of GPU workers, so the queries stay affordable [27], [34].

5.0.0.8 Network architecture.

Holding the best recipe fixed, we vary only the policy network and retrain from scratch (a different-shaped network cannot warm-start from the MLP champion, so this is a clean relative ranking at a fixed budget). We sweep MLP width and depth, a convolutional stack over the card planes, a permutation-invariant Deep Sets encoder [20], a self-attention encoder over card tokens [21], an LSTM [22], activation functions, and weight decay. To test whether any gain is recipe-specific, we also rerun the top encoders under two alternative recipes (PPO in place of TRPO, and PFSP sampling).

5.0.0.9 Search and learning baselines.

We add two non-learned and learned references. The first is a determinized information-set Monte-Carlo search (ISMCTS/PIMC) [18], [19] that, before each rollout, re-deals the cards it cannot see (the opponent’s hand and the stock) uniformly from the unseen pool, so it never uses the true hidden cards; we vary the per-move rollout budget. For contrast we also run an oracle variant that searches from the true state, as an upper bound. The second is Neural Fictitious Self-Play [1], the canonical equilibrium-learning baseline, run through OpenSpiel [17].

5.0.0.10 A second game.

To check that the method is not tied to Gin Rummy, we repeat the expert-yardstick study on Leduc Hold’em, a small poker game whose game-theoretic optimum is computable. We solve a near-optimal expert with counterfactual regret minimization [7], [17] and grade a tabular self-play learner and NFSP against it, the exact analogue of grading our Gin Rummy agents against the fixed expert.

6 Experimental Setup↩︎

6.0.0.1 One metric, applied the same way everywhere.

Every agent is graded by its win-rate against the fixed expert, with seats rotated so that each agent plays both positions equally. We use this single metric because the expert is strong, identical across comparisons, and cheap, so a win-rate against it isolates the effect of whatever ingredient we changed. We report two supporting numbers where useful: win-rate against the prior self-play champion (a moving but familiar reference) and against random play (a floor that saturates). For the headline agents we run \(2000\) games and report a \(95\) percent confidence interval; for the broad sweeps we run \(400\) to \(600\) games per cell, which is enough to rank regimes. For multi-seed comparisons such as the architecture sweep we follow current guidance on RL evaluation and report the interquartile mean (IQM) with stratified bootstrap confidence intervals rather than a single best run [35], [36].

6.0.0.2 Protocol.

Each study changes one factor and holds the rest fixed. Algorithm runs use two random seeds at two million steps. The reward and curriculum sweep changes one setting at a time from a common baseline. Representation runs hold the trainer fixed and swap only the input. Evaluation always uses the highest-probability legal action, never a random fallback, for the reason given above. Win-rates count a positive game score as a win.

7 Results↩︎

7.1 The expert is strong, and it almost never gins↩︎

The expert beats every agent we trained between \(70\) and \(99\) percent of the time: for example \(70.2\) percent against the self-play champion and \(99.5\) percent against random play (Figure 2, left). The surprise is on the right of Figure 2: despite gin scoring the most points, the expert gins in only \(0.7\) to \(1.7\) percent of games. It wins by knocking early with low deadwood. Chasing gin, the move a beginner reaches for, is a trap. The best style is patient, low-risk knocking. Every reward result below should be read with this in mind.

7.2 Algorithm: trust-region updates win↩︎

With the same masking, curriculum, and reward at two million steps, TRPO beats PPO on every opponent and both seeds. Against the expert, TRPO reaches \(22.5\) percent and PPO \(15.0\) percent; against the champion, \(50.5\) percent versus \(31.0\) percent (Figure 3, left). The reason fits the setting: rewards here are rare and the opponent keeps changing, and a trust region takes smaller, safer steps that do not overreact to a lucky or unlucky batch. We note that some popular alternatives do not apply here: methods like GRPO and DPO are built for aligning language models from preference data, not for per-move control in a game environment, so they are out of scope.

7.3 Reward: you cannot pay the agent into ginning↩︎

The gin-to-knock payoff is a real dial for style: when we reward gin heavily the agent gins more often against weak opponents, and when we reward knocking it knocks almost always. Against random play, a knock-first reward yields \(97\) percent knocks at a \(99.4\) percent win-rate, and a gin-first reward yields \(22\) percent gins at a \(98.3\) percent win-rate. But style is not strength. When graded against the expert, the gin rate collapses no matter how we set the reward. Across the controlled sweep, paying three times more for a gin than a knock leaves the gin rate under one percent, statistically the same as not rewarding gin at all (Figure 3, right). Just like the expert, the agent works out on its own that against strong play, chasing gin loses. This is the cleanest result in the paper: a shaped reward can set the agent’s style, but it cannot bribe the agent into a habit that loses.

7.4 Representation: the raw sparse input wins↩︎

Compressing the sparse \(4\times52\) observation into a small dense embedding hurt in every variant we tried. Against the expert, the raw sparse input reaches about \(15\) percent, while learned and LLM-judged embeddings land between \(6\) and \(14\) percent depending on size, and never above the sparse baseline. Larger embeddings and unfreezing the encoder did not close the gap. The reason is that a fixed low-dimensional bottleneck is good at saying whether two states are broadly similar, but it discards the fine detail (which exact cards are where) that the policy needs to act well. Dense embeddings earn their keep in NLP for two reasons that do not hold here: the vocabulary is enormous, so mapping one-hot tokens into a small dense space is a genuine compression, and word order carries meaning, so the embedding can capture sequence structure. Our \(4\times52\) observation is already small and its cards carry no meaningful order, so there is little to compress and no sequence to encode. The raw sparse tensor is already close to the right representation.

7.5 Curriculum, warm-start, and keep-the-best↩︎

Self-play makes difficulty free: a three-million-step self-play agent beats its own parent \(61\) percent of the time. But unguided self-play is fragile; a pool of past selves with no schedule broke down after about ten million steps as the agents chased circular strategies. A rising curriculum (random, then a pool, then self-play) keeps the challenge fair but always a little harder, and avoids that collapse. Two cheap additions matter on top of it. Warm-starting from the champion starts a run strong and lets it specialize. Keeping the best checkpoint matters most: because training drifts past its peak, evaluating every million steps and shipping the best checkpoint recovers two to three points for free (Figure 4). A more elaborate opponent-picking scheme (PFSP) was about even with a simple schedule here, and a longer reward horizon (a higher discount) slightly hurt, adding noise rather than foresight.

Stacking the ingredients that help (TRPO, a knock-first reward with a small deadwood-reduction bonus, a curriculum seeded with the strongest earlier agents, warm-starting, and keep-the-best) and checking over \(2000\) games, the strongest agent reaches \(\mathbf{34.2 \pm 2.1}\) percent against the expert and \(51.4\) percent against the old champion, up from the champion’s roughly \(30\) percent (Table 1). Two sibling agents trained with PFSP reach \(34.0\) percent, statistically the same. Table 2 summarizes every ingredient and its verdict.

8 Architecture, Baselines, and a Second Game↩︎

Having tuned the algorithm, reward, curriculum, and checkpointing to the mid-thirties, we ask the natural follow-up: is the network the missing lever, and how do principled baselines compare? The answer, in both cases, points at the same conclusion: the ceiling is set by the hidden information, not by the model.

8.1 Architecture does not break the ceiling↩︎

We hold the winning recipe fixed and vary only the network, retraining from scratch. Across MLP width and depth, convolutional, permutation-invariant Deep Sets, and recurrent encoders, the win-rate against the expert stays in a narrow band: the best IQMs are the structured encoders (convolutional \(31.1\) percent, Deep Sets \(30.4\) percent) and the plain MLP anchor is \(26.7\) percent, but the \(95\) percent bootstrap intervals overlap throughout (Figure 5a), so the architectures are statistically indistinguishable. Making the MLP wider, deeper, or extra-wide does not help, and weight decay slightly hurts. The finding is robust to the training recipe: rerun under PPO instead of TRPO, or with PFSP sampling, the encoders keep the same ordering and the best cell reaches \(30.2\) percent, in the same band. Recurrence (an LSTM) beats its matched feed-forward control (\(24.3\) versus \(19.2\) percent) but both trail the TRPO anchor, so memory helps relative to its own baseline without breaking the ceiling. A self-attention encoder cannot be trained under our trust-region recipe (differentiating the transformer twice for the natural-gradient step is prohibitive on CPU and unstable on GPU); under PPO against a matched MLP it does not beat a plain MLP (\(17\) versus \(21\) percent). Capacity and inductive bias are not the bottleneck.

8.2 A fair search is weak; only an oracle beats the expert↩︎

We then run a determinized ISMCTS, the classic non-learned reference for hidden-information card games. Graded fairly, re-dealing the unseen cards before every rollout, it is surprisingly weak: win-rate against the expert rises with the per-move budget but only to \(10\), \(17\), \(21\), and \(26\) percent at \(10\), \(30\), \(60\), and \(120\) rollouts (Figure 5b), below our trained agents. Gin Rummy hides a full hand plus the deck, so averaging over sampled deals is high-variance. Head to head against the fair search, every trained agent wins comfortably (\(66\) to \(69\) percent at \(60\) rollouts; Table 3). For contrast, an oracle search that may see the hidden cards reaches \(41\) to \(85\) percent over the same budgets. The \(26\)-versus-\(85\) gap is the crux: the information, not the search, is what wins, which is direct evidence that the ceiling is information-bound. Naive search is not a free way past it.

8.3 Generality: a second game with a computable optimum↩︎

Finally we repeat the study on Leduc Hold’em, where a counterfactual-regret solution is computable (our CFR expert reaches exploitability \(0.026\)). Graded against that near-optimal expert over eight seeds, a tabular self-play learner reaches near parity (mean return \(-0.085\); random is about \(-0.78\)), reproducing the Gin Rummy story on a game where the true optimum is available. NFSP, the neural equilibrium baseline, was far more sample-hungry: even at three million episodes it averaged \(-0.71\), near the random floor, a reminder that the neural baseline needs much more data than the tabular learner on this size of game. Table 3 collects the baseline and second-game numbers.

9 Discussion, Limitations, and Future Work↩︎

9.0.0.1 An information-bound ceiling.

Stacking every choice that helps moves a lightweight agent from roughly \(30\) to about \(34\) percent against the expert (Table 1), and three independent lines of evidence say this ceiling is set by the hidden information, not by the method or the model. First, network architecture does not move it: MLP, convolutional, set-based, and recurrent encoders are statistically indistinguishable, so capacity and inductive bias are not the bottleneck. Second, a determinized search graded fairly is weaker than our agents (\(26\) percent), yet the same search given oracle access to the hidden cards jumps to \(85\) percent; the gap is the value of the information. Third, our reactive policy has no memory and does not estimate the opponent’s hand, which is exactly the information the oracle exploits. The ceiling is therefore a measurement, not a disappointment: it says that with the available information a lightweight reactive agent is already near the practical frontier, and that going clearly past it requires more information (opponent-hand inference) or much heavier search, not a bigger network or more reward tuning.

9.0.0.2 What did not work, and why.

Three popular shortcuts each failed, and the failures are as informative as the successes. Imitation learning (DAgger) drove the training loss to near zero but produced an agent that wins almost no games: a textbook case of causal confusion [25], where the student reproduces the expert’s move in familiar positions without learning the reasoning behind it (tracking what the opponent is collecting), so it falls apart the moment the game leaves the demonstrated distribution. Dense step rewards made the agent short-sighted; a horizon of two showed no real learning and a horizon of five stopped improving after about five hundred thousand steps, at which point the agent farmed instant points instead of trying to win, exactly the failure mode the theory of reward shaping warns about when the bonus is not aligned with the true return [23]. A live LLM opponent played competently with a chain-of-thought prompt (up to \(98.2\) percent legal moves, against \(79.3\) with a terse prompt) and one even beat our self-play agent three games to two, but at \(9\) to \(27\) seconds per move it is far too slow to supply the millions of moves an RL run needs, and a vision-language variant fed the board as an image failed outright (about ten percent legal moves), confirming this is a text task. Each negative is consistent with the same ceiling: the shortcuts that fail try to substitute supervision, shaping, or a bigger opponent for information the agent does not have.

9.0.0.3 Limitations.

We study one game, so the specific numbers are about Gin Rummy. Our expert is a strong heuristic, not a game-theoretic optimum, so “win-rate against the expert” is a fixed and meaningful yardstick rather than a distance from perfect play, and we have been careful to frame it that way. Our headline learner is a reactive feed-forward policy; we did test recurrence, but an LSTM helped only relative to its own control and did not break the ceiling, so a stronger form of memory or explicit opponent modeling is likely needed to track hidden information. And we tested the LLM only as a live opponent; we did not use it to generate an offline dataset, which our serving stack was in fact built to support.

9.0.0.4 Future work.

The evidence points squarely at the hidden information. A naive determinized search did not break the ceiling at the budgets we tried, so the lever is not search alone but reasoning about what the opponent holds: explicit opponent-hand estimation (a thread already present in the Gin Rummy literature), belief-state or information-set search that conditions on inferred cards, and regret-minimization or equilibrium-finding methods [1], [7], [9] that target hidden information directly. Stronger memory than the LSTM we tested is a second axis. Finally, because our LLM serving stack can produce strong play offline even though it is too slow online, distilling a large batch of LLM-versus-LLM games into the policy with offline RL is a promising way to use the LLM without the latency that killed the live-opponent approach.

9.0.0.5 Beyond one game.

Nothing in the method is specific to Gin Rummy. The recipe is a pattern: build a fixed strong reference for the game, grade every training choice against it, schedule a rising curriculum of opponents, and ship the best checkpoint rather than the last. We package the pipeline so that pointing it at another two-player game in the same multi-agent interface requires changing only the environment, and we hope the controlled, expert-graded style of study is useful to others building lightweight agents for interactive games.

10 Conclusion↩︎

We set out to answer two questions about a small, fast Gin Rummy agent: how strong it can get, and which choices make it strong. A fixed expert and more than a hundred controlled runs turned those questions into measurements. Strong play knocks early and almost never gins, and no reward could tempt the agent into chasing it. Trust-region self-play with a knock-first reward, a rising curriculum, warm-starting, and keeping the best checkpoint is the recipe that works, reaching about \(34\) percent against the expert, while learned embeddings, imitation, dense rewards, and a live LLM opponent each fail for a reason we can name. The ceiling that remains is not an engineering failure: a bigger network does not move it and a fair search trails our agents, yet an oracle with the hidden cards pulls away, so the agent is capped by missing information, not missing capacity. The same story holds on Leduc Hold’em, where the optimum is computable.

The lasting lesson is about method. Strength in these games is easy to overstate against random or self-referential opponents; a fixed, strong, reproducible reference turns a pile of plausible training tricks into a study that says which choices truly matter. Build that reference first, and the rest measures itself. We release the pipeline so the next game can be studied the same way.

10.0.0.1 Reproducibility.

The training pipeline, the fixed expert, the LLM serving stack, and the human-play web client are available at https://github.com/Nikelroid/adversarial-coevolution.

References↩︎