1 Introduction↩︎

Complex games have long served as useful benchmarks for evaluating the capabilities of learning agents, as they offer clearly defined rules and measurable objectives. Collectible card games extend the scope as they allow for the rule set to be altered. The game pieces are a growing collection of cards, each with the ability to introduce novel mechanics, modify existing rules, or combine with previous cards to re-contextualize their utility. As a result, learning a single, global policy that fully captures the game is impractical: mastery requires continual adaptation to new content and the ability to encode generic game pieces and how they change the game rules.

In this work, we focus on Magic: The Gathering (MTG), a game with thousands of new cards released every year, and some of the most complex rules in games. We restrict our attention to the Draft format, a variant we describe in Section 2, full of highly non-local decisions: the value of a card depends not only on its intrinsic properties, but also on previously selected cards, the previously seen cards that could wheel back, and signals from passed cards giving noisy information about the draft choices made by the other drafters. We use the outcome of games subsequent to the Draft as an evaluation metric, making our approach outcome-driven.

Limited work exists on applying machine learning methods to MTG in general, mostly limited to predicting human pick choices. While such approaches provide useful signals, they limit themselves to descriptive modeling of player behavior. In contrast, our goal is to directly predict draft outcomes: given a partial or complete draft sequence, we estimate the expected win-rate of the resulting deck in subsequent games.

We summarize our contributions as follows:

1. A procedure to transform raw card features into contextualized embeddings given a draft environment.

2. DraftEncoder, an encoder-based architecture designed to model drafting as a sequence of combinatorial decisions, updating card embeddings dynamically as the draft progresses, in order to reason about both local card quality and global deck composition.

3. An evaluation of our model and linear baselines on large-scale real-world draft data from MTG Arena and 17Lands [1], which we believe is the first benchmark for predicting MTG draft performance from draft data.

2 Magic: the Gathering (MTG)↩︎

MTG is the oldest collectible card game, providing a rich history of cards and ways to play, all of which consist of building a deck following some rules, then playing with that deck against other players.

Drafting process. During a draft, up to 8 players are arranged around a table, and they open a pack each, containing a fixed amount of cards (varying between 13 and 15 depending on the drafted expansion). From their pack they select exactly one card, pass the remaining cards to the player on their left, and repeat this process until all cards have been picked. Then this process is repeated two more times with new packs, alternating the passing direction from right to left. From the perspective of a single player, the draft can be viewed as a sequence of \(K \in \{39, 42, 45\}\) decisions, where at step \(k\) the player chooses one card from an ever changing candidate set of size varying between 1 and 15. A choice is informed by previously selected cards, as well as on implicit signals inferred from which cards remain available in later picks.

Once all cards have been picked, they are used to construct a deck containing at least 40 cards, usually including around 17 extra basic lands.

Draft representation. We model a draft for one player as a sequence \(\mathcal{D} = (({(p_{k,i})}_i, c_k))_{k=1}^{K}\), where \({(p_{k,i})}_i\) are the available cards at pick \(k\) and \(c_k\) is the selected card.

4pt

17Lands draft data. We use draft data collected from MTG Arena via the 17Lands project [1], summarized in Table ¿tbl:tbl:17lands?, which aggregates anonymized draft logs from players across a range of skill levels (The average win rate is typically around 55%, inflated by self-selection bias). Each draft record contains the full sequence of observed packs and picks by an individual player, the player rank², and the number of wins and losses.

Outcome definition. We observe the win-rate of the drafted deck as our measured outcome, stored as the raw number of wins and losses \((W, L)\). For the data we used, drafted decks are played until either 7 wins or 3 losses are reached. As a result, the observed proportion of wins is a censored estimate of the deck’s true win-rate \(p_{\text{win}}\).

Learning objective. Given a partial or complete draft sequence \(\mathcal{D}_{\le k}\) in the expansion \(\mathcal{E}\), we aim to learn a function predicting the true win-rate: \(f_\mathcal{E}(\mathcal{D}_{\le k}) \rightarrow p_{\text{win}} \in [0,1]\).

3 Related Work↩︎

Models for card game drafts. Prior work [2], [3] on modeling drafting decisions in Magic: the Gathering has primarily focused on estimating pick order from human draft data. Bertram et al. [3] studies optimal card representations for draft pick prediction in MTG draft logs. However, this approach focuses on predicting individual picks rather than modeling downstream deck performance or win-rate. Some work exists for deriving a drafting policy combined with a playing policy for Hearthstone [4], but relies on being able to simulate the simpler gameplay.

Natural Language Processing. Converting textual data to a vector of real values is a recurring task in machine learning. For our work, we use a pre-trained model, EmbeddingGemma [5], that uses a transformer encoder to encode an input string of unspecified length into a single embedding of size 768. As drafting is a sequential process where each decision is highly conditional on both previous decision and previous potential decisions (which cards were in the passed packs), a model based on transformer architectures initially introduced by Vaswani et al.[6] seems natural.

4 Our model: DraftEncoder↩︎

To predict a draft outcome, we first use a Transformer Encoder architecture to produce card embeddings contextualized by the current draft expansion and player rank. Then, the draft state embeddings for each pick are initialized with those card embeddings. Finally, another attention mechanism updates the draft state embeddings using the information from previous picks. We give an overview of this process in Figure 1.

4.1 Encoding Cards↩︎

To construct card features, we convert all the information in a card into a string (see Figure 2) and use EmbeddingGemma [5] to convert it to a static embedding vector \(E_c^{(T)}\) of size \(d_t=768\). To evaluate the model comprehension, we also compute statistical features \(E_c^{(N)}\) for each card, following the features of Bertram et al. [3] (i.e. the average card win rate when it is in the opening hand, drawn, tutored, in the deck, in the sideboard, or seen during the game). In our results, we indicate when a model concatenates these features to the textual features.

We reduce the dimension of the initial card features using a linear layer, a LeakyReLU non-linearity, and a dropout layer. \[\begin{align} \bar h_c^{(0)} &= \mathop{\mathrm{DO}}\left(\mathop{\mathrm{LeakyReLU}}\left(W^I \left[E_c^{(T)}||E_c^{(N)}\right] + b^I\right)\right) \end{align}\]

To let player rank condition card evaluation, we learn an embedding vector for each rank, feed it through a two-layer Multi-Layer Perceptron (MLP) to get two vectors \(\gamma,\beta\in\mathbb{R}^d\). We use each dimension of those vectors to alter the matching dimension of the card features individually, following the FiLM layer (Feature-wise Linear Modulation) of Perez et al. [8], to control mean and variance, yielding our initial card embeddings \(h_\cdot^{(0)}\): \[\begin{align} \gamma^*(r), \beta(r) &= \mathop{\mathrm{MLP}}(x_r) \quad \gamma(r) = 1 + 0.1 \tanh(\gamma^*(r)) \\ h_c^{(0)} &= \gamma(r) \odot \bar h_c^{(0)} + \beta(r) \end{align}\]

\(h_c^{(0)}\) should encode absolute information about card \(c\), contextualized by the skill level of the drafting player, and optionally by empirical meta-stats about that card in the currently drafted expansion, but is mostly agnostic relative to other cards in the same expansion. To update this encoding with the context of the current expansion and other cards that could be drafted with it, we implement a standard encoder-only architecture in which each card attends to other cards in the same expansion. This consists of \(N_\mathcal{E}\) residual attention layers, producing contextualized card embeddings \(h_c^{(N_\mathcal{E})}\).

4.2 Encoding draft picks↩︎

The sequence of information gained during a draft and the decisions taken is encoded using two types of tokens:

• Pack cards: For each pack card \(p_{k, i}\), we keep track of our evaluation of that card relative to the previous picks, starting with the feature vector \(h_{k, i}^{(0)}=h_{p_{k, i}}^{(N_\mathcal{E})}\).

• Draft state: We also track the current state of the draft after \(k\) picks, starting by only considering the picked cards in isolation with the feature vector \(s_k^{(0)}=h_{p_{k,c_k}}^{(N_\mathcal{E})}\)

Those encodings are updated with a bidirectional pre-norm Transformer encoder block, first updating the pack cards: \[\begin{align} \bar h_{k,i}^{(n+1)} &= h_{k,i}^{(n)} + \mathop{\mathrm{DO}}(\mathop{\mathrm{Attn}}(\mathop{\mathrm{R}}(\mathop{\mathrm{LN}}(h_{k,i}^{(n)})), \mathop{\mathrm{R}}(s_{1\ldots k}^{(n)}), s_{1\ldots k}^{(n)})) \\ h_{k,i}^{(n+1)} &= \bar h_{k,i}^{(n+1)} + \mathop{\mathrm{DO}}(\mathop{\mathrm{MLP}}(\mathop{\mathrm{LN}}(\bar h_{k,i}^{(n+1)}))) \end{align}\] applying a rotary positional encoding [9] \(\mathop{\mathrm{R}}\) to the query and key, before updating the draft states: \[\begin{align} \bar s_k^{(n+1)} &= s_k^{(n)} + \mathop{\mathrm{DO}}(\mathop{\mathrm{Attn}}(h_{k,c_k}^{(n+1)}, h_{k,\cdot}^{(n+1)}, h_{k,\cdot}^{(n+1)})) \\ s_k^{(n+1)} &= \bar s_k^{(n+1)} + \mathop{\mathrm{DO}}(\mathop{\mathrm{MLP}}(\mathop{\mathrm{LN}}(\bar s_k^{(n+1)}))) \end{align}\] to obtain \(s_k^{(n+1)}\).

After \(N_\mathcal{D}\) such updates, we obtain final draft states \(s_k^{(N_\mathcal{D})}\), that we concatenate to the rank embeddings \(x_r\) and feed through an MLP with 3 layers, \(\mathop{\mathrm{LeakyReLU}}\) activations, and a final sigmoid to output after each pick a win-rate between 0 and 1.

4.3 Training metric↩︎

As our target outcome is biased to have at most 7 wins and 3 losses, we train our model with an appropriate Negative Log-Likelihood (NLL) loss. \[\begin{align} \ell(p_i; W, L) &= \begin{cases} -\log \left[ \binom{6 + L}{6} \, p^{7} (1 - p)^{L} \right] & \text{if } W = 7, \\ -\log \left[ \binom{W + 2}{2} \, p^{W} (1 - p)^{3} \right] & \text{if } L = 3 , \end{cases} \\ L(p, W, L) &= \sum_{i=1}^K \frac{i}{K} \ell(p_i; W, L) \end{align}\] We weight later predictions higher as they were made with more information, and \(p\) is the win-rate predicted by the model.

5 Experimental Setup↩︎

5.1 Experiments↩︎

Predictive accuracy. We first evaluate our model on predicting draft outcomes using the same expansions it was trained on. We only use BLB, FDN, and TDM for this experiment, split temporally using the last 6 weeks of data into chunks of 2 weeks assigned to training, validation, and testing chronologically, for around 120,000 training drafts.

New expansion and cards generalization. We also evaluated the ability of our model to generalize to unseen expansions and cards. To do so, we trained and validated on the last 4 weeks of every expansion listed in Table ¿tbl:tbl:17lands? except FDN, and evaluated on the last 2 weeks of FDN, for around 400,000 training drafts. Then, we fine-tune each model using the first four days of data of FDN.

5.2 Evaluated Models↩︎

We compare DraftEncoder against these baselines:

• Heuristic, that computes and outputs the average "in deck" win rate from 17Lands of the best 23 picked cards.

• LR, a linear regression, using the features of the first \(i\) picked cards and an embedding player rank vector to predict the win rate after \(i\) picks.

We train the linear regression and our model for 100 epochs, with an inner dimension of 64, \(N_\mathcal{E}=10\), \(N_\mathcal{D}=5\), 2 attention heads, a batch size of 256, optimizing with AdamW [10], and a peak learning rate of 1e-4.

6 Results↩︎

We report, as mean \(\pm\) standard deviation over 3 random seeds for the epoch with the best validation loss, the Spearman’s rank correlation coefficient between the predicted win-rates and the observed (biased) win-rates \(\frac{W}{W+L}\)³. We use rank correlation since we care about relative deck strength rather than calibrated probability estimates. Our models were coded in Jax [11] and Equinox [12], and trained on 4 Nvidia RTX A5000 GPUs, using around an hour per seed.

As seen in Figure 3, DraftEncoder is able to use its non-linear architecture to outperform the baselines. Although DraftEncoder has slightly worse predictions for the first pick, it quickly grows to almost twice the rank correlation of the baselines, and is close to reaching the saturation value of around \(0.23\). Adding meta features consistently improves performance, which we assume means our model is able to learn enough about the cards from the training outcomes alone.

However, as shown in Table 1, the model struggles to generalize to new sets, without having access to the meta features, only matching the performance of the linear model without meta features (and falling short of the heuristic, which has implicit access to meta statistics for the full expansion), but does manage to be competitive with the linear model with meta features. Fine-tuning using the first four days of data lowers the importance of the meta features a lot, as DraftEncoder then outperforms an untuned DraftEncoder+Meta, and is only worse than the tuned version by 10%. This shows that our model can learn a lot with only a few days of data, making it useful and relevant early in the lifespan of a newly released expansion.

7 Conclusion↩︎

In this work, we introduced DraftEncoder, the first learned benchmark for predicting Magic: the Gathering draft outcomes. Our model outperforms baselines by nearly 2\(\times\) in rank correlation, and approaches the estimated noise ceiling of \(0.23\) imposed by sampling noise inherent to the collected data.

The main remaining challenge to address is generalization to unseen expansions and cards, though fine-tuning on as little as four days of initial data largely closes the gap to meta-augmented models, suggesting that our model learns transferable draft representations that adapt quickly to new cards. Future work could explore incorporating gameplay simulation or a stronger natural language processing (NLP) of the cards, potentially including a fine-tuned rather than frozen text encoder.

References↩︎