From Multimodal Perception to Strategic Reasoning: A Survey on AI-Generated Game Commentary

Qirui Zheng\(^{1,*}\), Xingbo Wang\(^{1,*}\), Keyuan Cheng\(^{1,*}\), Yunlong Lu\(^{1}\), Muhammad Asif Ali\(^{2}\), Lingfeng Li\(^{1}\), Yongyi Wang\(^{1}\) and Wenxin Li\(^{1,\text{\Letter}}\)
\(^1\)Peking University
\(^2\)King Abdullah University of Science and Technology
zzzqr@stu.pku.edu.cn, jacksimbol@stu.pku.edu.cn, lwx@pku.edu.cn


Abstract

The advent of artificial intelligence has propelled AI-Generated Game Commentary (AI-GGC) into a rapidly expanding research area, offering advantages such as scalable availability and personalized narration. However, existing studies remain fragmented, and a systematic survey that unifies prior efforts is still lacking. To bridge this gap, our survey introduces a unified framework that systematically organizes the AI-GGC landscape. We present a novel taxonomy focused on three core commentator capabilities: Live Observation, Strategic Analysis, and Historical Recall, and further categorize commentary into three corresponding types: Descriptive Commentary, Analytical Commentary, and Background Commentary. Building on this structure, we provide an in-depth review of methods, datasets, and evaluation metrics, analyzing their strengths and limitations. Finally, we highlight key challenges and point out promising directions for future research in AI-GGC.

1 Introduction↩︎

Game commentary is the real-time narration and analysis of gameplay, which enhances the viewing experience by elucidating strategies and highlighting pivotal moments [1]. Game commentary often appear in three major domains: board games (Chess, Go), sports (basketball, soccer), and esports (League of Legends (LoL), Dota 2). Its importance is evidenced by the massive viewership of major tournaments, such as the World Cup Final, which attract billions of concurrent viewers and rely heavily on commentary to sustain engagement and immersion.

The rapid advancement of AI has made AI-Generated Game Commentary (AI-GGC) feasible. AI commentators present several advantages over human commentators. Primarily, they ensure unlimited availability, covering matches that cannot be covered by human commentators due to limited broadcast resources [2]. Furthermore, they offer unparalleled personalization [3]. Through tailored training, AI commentators can adapt to diverse viewer preferences, including the choice of language, voice, favored teams, and the level of analytical detail.

Although considerable research has been conducted, the field still lacks a systematic survey that consolidates these fragmented endeavors. This fragmentation stems primarily from two fundamental challenges inherent to the domain:

In board games such as Chess and Go, while RL-based AI systems have achieved superhuman performance [4], pure Large Language Models (LLMs) often struggle to provide deep, principled analysis of the reasoning behind each move. They can describe the moves made but frequently fail to articulate the underlying strategic reasoning and long-term planning behind them—reflecting their limited capacity for complex logical reasoning and long-horizon strategic inference.

Different game genres essentially "speak different languages". Board games rely on simple, symbolic formats; sports games are captured as fast-paced, visually rich videos; while e-sports generate high-frequency, complex metadata streams. The entirely different techniques needed to process each format thus segregate research into isolated camps, preventing a unified methodology.

To address this gap, Section 2 introduces a systematic survey scheme for analyzing AI commentators, guided by two key questions: “What makes a proficient commentator?” and “What defines effective commentary?” We clarify the three core capabilities required of AI commentators and propose a taxonomy that classifies commentary into three types by content and purpose. Building on this foundation, Section 3 reviews technical methodologies for these core capabilities. Section 4 surveys datasets by game genre and data practices, while Section 5 analyzes evaluation metrics. Figure 2 presents our integrated taxonomy of methods, datasets, and metrics. Finally, Section 6 discusses open challenges and future directions, and Section 7 concludes the paper. To summarize, this survey makes following key contributions:

  1. We propose a unified survey scheme that organizes AI-GGC along game genres, core commentator capabilities, and commentary types, providing a principled structure for analyzing existing methods and their trade-offs.

  2. Building on this scheme, we present the first comprehensive survey of AI-GGC spanning methods, datasets, and evaluation, and introduce a systematic taxonomy that synthesizes fragmented literature into a coherent framework.

  3. We critically examine the current AI-GGC landscape, highlighting challenges in capability coverage and maturity, system-Level coordination, and evaluation.

2 AI-GGC Survey Scheme↩︎

In this section, we propose a systematic survey scheme, illustrated in Figure 1, which organizes and summarizes the field of AI-GGC along three principal dimensions: game genre, foundational AI capabilities, and commentary types.

Figure 1: Overview of the proposed AI-GGC survey scheme, systematically summarizing the field along three key dimensions: (a) Game genres; (b) Foundational capabilities of AI commentators; (c) Commentary types.

2.1 Scope and Game Genres↩︎

Our survey scheme is anchored in three representative and distinct game genres: (1) Board Games. Covers games with perfect information (Chess) or imperfect information (Mahjong). Commentary focuses on reasoning, move quality evaluation, and strategic analysis. (2) Esports. Includes fast-paced digital titles like Dota and LoL. Commentary analyzes real-time decision-making and both individual and team-based strategies. (3) Sports. Physical, dynamic games like soccer and basketball. Commentary emphasizes real-time movement, key events, and tactical coaching decisions.

In Section 4, we categorize and analyze existing AI-GGC datasets according to these game genres.

2.2 Core Capabilities of AI Commentators↩︎

Drawing inspiration from the expertise of human commentators across various game genres, we abstract the core cognitive modules that underpin high-quality AI-GGC systems.

(1) Live Observation (LO) is responsible for the accurate perception and interpretation of real-time events and fine-grained details within the game environment. It serves as the primary data source for generating descriptive commentary, such as identifying the placement of pieces in chess, or describing shots and passes in soccer, kills and assists in LoL.

(2) Strategic Analysis (SA) encompasses the analytical abilities required to comprehend the current game state, interpret strategic intent, evaluate decisions, and anticipate future developments. It is particularly vital in board games, where understanding the intention and implications of each move is essential for insightful commentary.

(3) Historical Recall (HR) involves leveraging extensive game knowledge, including player histories, classic match moments, and community-specific terminology. It enables the commentator to provide meaningful context, such as recognizing signature moves or recalling iconic past plays, thereby enhancing the informativeness and engagement.

A systematic discussion of how existing methods implement these foundational capabilities is provided in Section 3.

2.3 Taxonomy of Commentary Types↩︎

According to the three cognitive modules, we classify commentary types, which provides a new evaluation dimension.

(1) Descriptive Commentary. This is the most fundamental type, directly derived from the LO capability. It objectively addresses the question, “What is happening?” by narrating immediate, observable events in games.

(2) Analytical Commentary. Derived from SA, this commentary transcends basic description by evaluating the quality of decisions or actions (“How good?”), interpreting player intent (“Why?”), and predicting strategic outcomes within a broader tactical context (“What’s next?”).

(3) Background Commentary. Enabled by the HR capability, this commentary type enriches the live narrative by integrating broader contextual information, such as detailed team and player backgrounds, game history, and cultural references. It addresses the question “What is the significance of this?” by referencing past performances, historical match situations, team history, and community knowledge.

= [ rectangle, draw=hidden-draw, rounded corners, text opacity=1, minimum height=1.5em, minimum width=5em, inner sep=2pt, align=center, fill opacity=.5, ] =[my-box, minimum height=1.5em, fill=blue!15, text=black, align=left,font=, inner xsep=2pt, inner ysep=4pt, ]

Figure 2: Taxonomy of Methods, Datasets and Metrics in AI-GGC

3 Methods↩︎

This section reviews technical pathways towards core capabilities of AI commentators introduced in Section 2. We also expand our discussions to architectural designs that wire key function modules to produce end-to-end game commentaries.

3.1 Live Observation↩︎

LO requires AI-GGC systems to process complex, multimodal game data under time constraints. Board games typically rely on direct visual observation of game board or structured textual representation that encode game state into compact strings (e.g. Forsyth-Edwards Notation for chess), while sports and e-sports rely mostly on videos. However, explaining “what is going on” in fast-paced games involves more than multimodal perception. A good commentary should be able to identify focal points in chaotic gameplays and summarize them using professional terminology, balancing detail and conciseness under time and comprehension constraints

Early technical trials attempting to address this challenge primarily adopt Event Spotting (ES), which reduces the complexity of raw game video processing by converting visual inputs into textual labels. Specifically, predefined event labels are assigned to corresponding timestamps in game videos and subsequently used as inputs to the commentary generation model, instead of raw video streams. Early works [5], [6] rely on human-annotated domain-specific datasets to train models for ES which may suffer from generalization issues. Detection-based ES reduces annotation labor by using object-detection models to identify actions and entities in videos. Jung  detect players and the ball using YOLOv8 and perform ES by combining spatial-temporal trajectories, action recognition, and rule-based reasoning to identify key basketball events. Similar pipelines have also been explored using various object detection backbones [7].

While effective in reducing perceptual complexity, this representation inevitably introduces information loss by discarding fine-grained details beyond the detected events.

Direct Encoding approaches map multimodal inputs (e.g. video streams and metadata) into dense representations in order to preserve information completeness. The advancing multimodal large language models has made it possible. Initial efforts focused on encoding structured representations for board games, directly encoding chessboard states into vectors [1], [8], until recent advance in models’ multimodal understanding capabilities enable the extension of this approach to visual inputs from sports. Jhamtani  combined video transformers for visual input and BERT for textual descriptions, fusing these features with a transformer encoder guided by a soft-prompt mechanism for better multimodal perception.

Despite recent progress, several direct encoding approaches still rely on relatively early visual encoding backbones [9], which may limit their capabilities for modeling complex visual information. More fundamentally, recent studies show that even state-of-the-art LLMs struggle with visual perception in low-dynamic board games [10],raising concerns about their scalability to fast-paced and highly dynamic sports or esports.

Methodologically, LO paradigms in AI-GGC have evolved in response to changing modeling constraints and opportunities, as reflected in the temporal distribution of methods in Figure 3 (b). Early work almost exclusively adopted ES to simplify video modeling under limited capacity, inevitably introducing information loss. With advances in DL, direct encoding emerged as a parallel paradigm between 2016 and 2022, aiming to preserve richer visual information. Since 2023, the rise of LLMs has renewed interest in event-based representations, as discrete events can be readily transformed into textual or structured prompts without resolving visual perception, making ES attractive from an engineering perspective. Importantly, these trends do not imply that direct encoding is a dead end. Rather, its practicality remains contingent on advances in multimodal understanding, suggesting that it may regain relevance as model capabilities continue to evolve.

a
b

Figure 3: Methodological patterns in AI-GGC research across capabilities and time. All statistics are reported by paper count.. a — Methods count of AI-GGC systems across core commentator capabilities, grouped by game genres., b — Temporal distribution of LO method counts in AI-GGC.

3.2 Strategic Analysis↩︎

A key challenge for AI-GGC is enabling deep SA, a capability that places substantial demands on the logical reasoning ability. Beyond LO, a good commentator is expected to address “How good”, explain “Why”, and anticipate “What’s next” within the same logic. Depending on whether SA relies solely on the model itself, we categorize existing AI-GGC methods for SA into Prompt-based Chain-of-Thought(CoT) and Externally Augmented Reasoning [11].

Prompt-based CoT refers to eliciting step-by-step SA purely through instructions in the prompt, guiding the model to carry out deeper reasoning without introducing external modules, search procedures, or additional training. In AI-GGC, prompting is not only used to steer how the model analyzes the current situation, but also to inject details of game rules and gameplay common sense that LLMs may be unfamiliar with (e.g., in Guandan, three-with-a-pair is allowed whereas three-with-a-single is not). Prior work [12], [13] has shown that such prompt-based guidance can better elicit SA ability of LLMs and improve the quality of generated commentary.

Despite its simplicity and effectiveness, this method mainly elicits SA already latent in the model, without extending its intrinsic reasoning capacity. As a result, its ability to support deeper strategic inference and forward-looking anticipation remains bounded by the model itself.

Externally Augmented Reasoning refers to enhancing SA by introducing auxiliary components beyond the language model itself, such as advanced algorithms or expert knowledge, to compensate for limited reasoning capacity of models. Such augmentation typically takes the form of external guidance, which can be broadly categorized into guidance derived from Human Expertise and guidance provided by Game AIs.

Incorporating strategic principles from Human Expertise allows LLMs to perform more insightful analysis, analogous to expert commentators explaining key moves and tactical intentions. The main challenge lies in embedding such principles effectively into the model’s reasoning process. Kim  addressed this by manually mapping baseball events to strategic concepts and provided both as model inputs, enabling interpretation within an explicit strategic context. In chess, Kim  extracted key strategic concepts (e.g., king safety) using a game engine and compared their representations before and after each move to identify salient strategic changes for commentary generation. For imperfect-information games such as GuanDan, Ma  proposed a two-level prompting scheme grounded in Theory of Mind (ToM) [14], where one prompt guided the model to analyze its own strategy while the other encouraged inference over opponents’ possible hands and intentions.

Augmenting strategy analysis with powerful Game AIs is another natural idea, mirroring common practice among human commentators, particularly in some board games where game AIs dominant. However, this approach introduces additional challenges. Despite their strength, RL-based game AIs are often black boxes, limiting interpretability and making it difficult to directly guide logical reasoning. Consequently, existing approaches focus on transforming unexplainable outputs into more explainable signals. An early idea is to achieve this through controllable search or rule-based mechanisms. Kameko  employed rule-based scoring functions and limited-depth search in Shogi to generate candidate game trees andF derive commentary from simulated outcomes. Another idea is to translate engine outputs into interpretable linguistic signals. Lee  quantified “move quality” by measuring win rate difference between the played move and the AI-recommended best move, and this difference was mapped to discrete natural-language categories. Zhang  jointly trained a chess engine and a language model in a multi-task framework, allowing the system to simultaneously learn game understanding and commentary generation.

Although externally augmented reasoning substantially improves the logical coherence and correctness of analytical commentary, it also faces inherent limitations. First, both sources of external guidance exhibit inherent weaknesses. Human expertise provides only partial coverage of the strategic space, generalizes poorly to complex or unseen situations, and may become outdated over time. Game AI–based augmentation, while powerful, is constrained by the black-box nature of modern engines; simple mappings from engine outputs to linguistic labels are often insufficient for rich strategic explanations. More critically, in highly complex sports and esports domains, dominant game AIs are still largely unavailable, severely limiting the applicability of AI-driven augmentation. Second, external augmentation introduces capabilities absent from the model itself, but these remain externally provided rather than internalized as intrinsic reasoning ability.

Beyond method-specific limitations, existing AI-GGC approaches share a fundamental weakness in SA: they mainly focus on short-horizon quality assessment (“How good?”), without constructing coherent multi-step strategic narratives. As a result, reasoning about “Why?” is often incomplete, leading to unreliable prediction of “what’s next” based on such ligic. And as also discussed above, current approaches rarely internalize strategic reasoning into the model itself, instead mainly relying on externally provided signals.

3.3 Historical Recall↩︎

Historical recall enriches background commentary by integrating external knowledge. Previous works have approached this primarily as an information retrieval task. Lee  ranked stories from a database based on their similarity to the current game state using AdaRank [15], then filtered candidates with a regression model before inserting into commentary. Andrews  encoded player information with SBERT to retrieve relevant player or team data.

3.4 Content Organization↩︎

While techniques in former sections have enabled rich contents and details based on raw game states and data, real game commentary is usually restricted by limited time and audiences’ capabilities to absorb the game. Thus AI-GGC faces an inherent challenge: transforming unstructured segments into logically structured, concise narratives through strategic selection and prioritization to maintain viewer comprehension without cognitive overload. Mkhallati  proposed a spotting head that predicts when commentary should be generated for each time window, using non-maximum suppression to eliminate redundancy. Mei  introduced a coarse-to-fine aligner that filters out irrelevant records and reweights alignment scores to focus on salient events. Likewise, Ranella  used an event-priority and delayed-confirmation mechanism to filter key events in real time.

3.5 Commentary Generation and Rendering↩︎

Commentary generation, the final stage of AI-GGC, transforms processed information into natural language. These methods fall into three main categories: template-based, DL-based, and LLM-based. Template-based approaches are efficient and reliable [16][18], but often yield repetitive outputs. DL-based methods, typically using LSTM decoders [19][21], provide more diverse and context-aware commentary. LLMs further improve fluency and strategic reasoning, with prompt-based techniques explored in recent work [3], [12], [22], though they can be slower, more costly, and prone to hallucinations.

Multimodal rendering is an emerging area, with most non-textual commentaries generated by converting text to speech via TTS APIs. For example, Xu  generated both text and audio parameters for expressive speech, while Siu  combined TTS and StyleGAN to create a virtual commentator with realistic facial expressions and lip-syncing.

3.6 A Systematic Gap in Capability Coverage↩︎

Figure 3 (a) reveals a pronounced structural gap in current AI-GGC research: most systems focus on LO and fail to cover the full set of core commentator capabilities, with comparatively limited attention to SA, HR, and CO. This imbalance manifests differently across game genres due to their distinct characteristics. For board games, longer turn intervals and high demands for logical reasoning lead to greater research attention on SA, while reducing the relative need for CO. Powerful game AIs provide strong support for this ability, reinforcing this focus. In contrast, the limited attention to HR is difficult to justify. For sports and esports, the dominance of LO is largely driven by the intrinsic difficulty of multimodal perception in fast-paced and visually complex scenes, leaving other capabilities substantially under-explored despite their importance for effective commentary.

4 Datasets and Benchmarks↩︎

This section reviews dataset according to game gernes, along with relevant data processing techniques.

4.1 Board Game Commentary Datasets↩︎

Commentary datasets for board games have been developed for chess, Go, Shogi (Japanese chess), GuanDan (Chinese card game). The turn-based nature of board games ensures natural alignment between game states and commentary, and the slower pace offers sufficient time for deep reasoning.

Most datasets consist of aligned pairs of game states and commentaries. Jhamtani   introduced a large chess dataset with 298K move-by-move annotations, establishing a benchmark for AI-GGC in chess. Additional datasets exist for Shogi [23] and Go [24].

Some datasets are annotated beyond commentary text for better commenting. A Shogi commentary corpus [25] annotates modality expressions and event factuality, enabling fine-grained modeling of uncertainty and future-oriented reasoning in strategic commentary.

4.2 Sports Commentary Datasets↩︎

As mentioned before, in sport games, ES is widely used to simplify percetion of complex multimodal input. ES-based datasets cover soccer [26], basketball [20], baseball [27], and tennis [19]. SoccerNet [5] is a prominent example, providing soccer videos with event labels and over 500K commentaries. It expanded to SoccerNet-v2 [6] with 17 event categories and SoccerNet-v3 [28] with spatial annotations. To compensate for the information loss caused by ES, some datasets also incorporate additional modalities such as player tracking [29] and play data [30], enabling richer perception and reasoning.

Manual alignment of game states and commentary remains common but is labor-intensive. To automate alignment, Rao  developed a pipeline combining LLaMA-3 and contrastive learning, trained on a small set of aligned data.

Alternative alignment strategies have been introduced for specific tasks, including timestamp-based sparse alignment to support long-video commenting [31] and benchmarks where commentaries outnumber events, reflecting real-world settings [32].

Beyond commentary datasets, some auxiliary resources can be used to enrich background commentary by providing anecdotes, such as story libraries for cricket [33] and baseball [34], [35].

4.3 Esports Commentary Datasets↩︎

Esports datasets, like those for sports, often use ES to manage complex video inputs. A unique advantage in esports is the availability of official APIs, which provide precise access to rich multimodal signals such as player statistics, game state metadata, and audio streams [36], placing higher demands on multimodal perception.

Esports commentary datasets cover games such as LoL [37], Dota 2 [38], and RoboCup [39], and often include rich multimodal data including audio features [40], sensor data [41], or multiple viewpoints [42]. Ringer  introduced affect labels capturing both game context and emotional dimensions, enabling affect-aware commentary generation.

5 Evaluation Metrics↩︎

Evaluating AI-GGC systems is inherently challenging for three reasons: (1) AI-GGC is a multi-faceted task involving various core commentator capabilities, with no clear way to aggregate them into a single score. (2) It is an open-ended NLG task with no gold reference, as different commentators may produce diverse yet equally valid commentaries for the same game state. (3) Analytical commentary often requires long chains of strategic reasoning that are difficult to formalize, making correctness hard to assess with existing metrics. Current AI-GGC work relies on evaluation methods that only partially address these challenges and have substantial limitations, which can be broadly grouped into four categories:

Human-Centric Metrics. Human-centric metrics remain the gold standard for assessing commentary quality, typically using questionnaires like Likert-scale. This metric of course can ignore the three challenges due to its directly reflecting human preference, but they are expensive, require domain expertise, and highly subject to individual variability.

Untrained Automatic Metrics. Untrained automatic metrics, especially n-gram overlap measures, remain the most widely used tools for AI-GGC evaluation. As shown in Figure 3 (a), there are nearly half of AI-GGC works using it as the only evaluation metric. Metrics such as BLEU [43], ROUGE [44], and METEOR [45] quantify lexical similarity to reference texts and are popular due to their low cost and reproducibility. However, n-gram overlap metrics are inherently misaligned with the task and correlate weakly with human judgments because AI-GGC lacks definitive ground-truth references [46], making them ineffective for addressing any of the three challenges discussed above.

Figure 4: Distribution of evaluation strategies used in AI-GGC.

ML-based Metrics. ML-based metrics instead use learned models to approximate semantic similarity or other properties of commentary. Zhang  use BERTScore to evaluate esports commentary. However, it is still a reference-based measure of similarity to human texts and thus shares the same fundamental limitation as n-gram metrics in AI-GGC. Worth mentioning, Ma  apply SnowNLP, a Naïve Bayes classifier, to estimate sentiment polarity of commentary. Although it is a very coarse signal and should not be treated as a primary metric, it has limited value as an auxiliary indicator of whether the overall emotional tone is plausible.

LLM-based Metrics move beyond simple similarity measures by providing reference-free, multi-dimensional assessments that better align with human preferences through prompting or post-training. For instance, Kim  introduced GCC-Eval, which integrates expert chess knowledge with G-Eval [47] to evaluate chess commentary along four dimensions: relevance, completeness, clarity, and fluency. However, it applies the same dimension weights to all commentary types and avoids assigning reasoning correctness to the LLM, relying instead on human annotations. Although still an emerging approach, LLM-as-a-judge currently appears to be the most promising direction for evaluating AI-GGC. However, existing work also shows that such judges remain unstable when verifying long chains of reasoning and exhibit systematic biases, such as a tendency to prefer longer [48] and position bias [49], which limits their current reliability for AI-GGC.

Overall, existing AI-GGC evaluation remains only partially aligned with the task: untrained and ML-based metrics struggle with open-endedness and multidimensionality, while LLM-based judges are still unreliable for assessing long-chain reasoning. To address these limitations, we argue that the taxonomy of descriptive, analytical, and background commentary offers a natural basis for type-specific evaluation, as discussed in Section 6.

6 Challenges and Future Directions↩︎

AI-GGC is rapidly emerging as a prominent application area for LLMs, where technical innovation meets growing real-world demand. While recent research and practical deployments highlight the field’s promise, they also expose fundamental challenges and point to key directions for future work.

Incomplete Coverage of Core Commentator Capabilities. A common limitation of existing AI-GGC systems is their incomplete coverage of core AI commentator capabilities, as shown in Figure 3 (a) and discussed in Section 3.5. Future AI-GGC research should therefore place balanced emphasis on all core commentator capabilities, treating them as equally important components in building more informative, interpretable, and complete commentary systems.

Shallow Realization of LO. LO requires AI commentators to accurately capture and summarize ongoing games to guide audience attention and comprehension. Although a considerable amount of work has already been devoted to processing raw game data and extracting fundamental game events, these descriptions are still far from clear, specific, and easily understandable game descriptions. While pattern-recognition-based processing relies on models trained on annotated data with predefined event labels [50], this class of methods often lead to significant information loss and preclude the possibility of subsequent fine-grained scene and strategy analysis. We advocate that the introduction of common sense and reasoning could pave the way towards ideal LO commentaries. On capturing atomic actions of game entities, using generic object detection techniques, the model can perform reasoning by: (1) constructing relational graphs among multiple entities, (2) conducting cross-temporal alignment with observation history, and (3) referring to game-specific knowledge and commonsense, thereby yielding fine-grained, comprehensive cognition of the current game state.

Shallow Realization of SA. Current AI-GGC systems exhibit several critical limitations in SA. First, most systems lack long-horizon match memory, relying primarily on current actions and states for analysis, which makes it difficult to explain “Why?” or reliably anticipate “What’s next?”, both of which depend on broader historical context. Maintaining memory of past actions and intermediate analytical states could help models form more coherent strategic interpretations. Second, the modeling space of external augmentation remains underexplored. Explainable AI methods provides a direct way to understand black-box game AIs, particularly interpretable game AI models [51]. Meanwhile, deeper ToM modeling can help LLMs reason about players’ intentions and beliefs [52]. Finally, SA is rarely internalized within LLMs, with most systems relying on external analytical cues, constraining reasoning depth and consistency. Aligning LLMs with strategic reasoning behaviors of game AIs at the parameter level is a promising direction for more robust analytical commentary.

Lack of Cross-Capability Coordination. A common limitation of existing AI-GGC systems is the lack of coordination across core commentator capabilities. LO, SA, and HR are often treated as independent modules rather than as an integrated whole, with limited communicating or joint decision-making. Consequently, capabilities are optimized isolately, resulting in fragmented understanding and inconsistent commentary logic. While You  adopts an end-to-end formulation, it primarily couples LO with commentary generation, leaving other core capabilities uncoordinated. Cross-capability coordination can be promoted from both informational and functional perspectives. At the informational level, intermediate information produced by different capabilities can be partially shared. At functional level, SA and HR can impose fine-grained information demands on LO, specifying which details should be emphasized. Such demand-driven coordination remains unexplored in existing AI-GGC systems.

Lack of a Standardized Benchmark and Reliable Evaluation. Despite the growing number of AI-GGC datasets spanning multiple modalities, game genres, and languages, the field still lacks a unified and standardized benchmark, making fair comparison across AI-GGC works difficult. Moreover, as discussed in Section 5, this problem is further exacerbated by the lack of reliable and convincing evaluation metrics. A pressing direction is to develop a large-scale, multimodal AI-GGC benchmark with standardized data formats and fair evaluation. For evaluation, fine-tuning preference models from human labels is feasible but costly. In contrast, the taxonomy of commentary types proposed in our work enables more targeted LLM-as-a-Judge designs for different commentary types, alleviating challenges of multi-dimensionality and open-endedness. The relative importance and weights of evaluation dimensions can be examined through expert assessment or audience feedback [53]. For analytical correctness, a promising direction is to decompose long reasoning chains into shorter, verifiable sub-chains and validate them progressively, reducing reliance on LLMs’ long-horizon reasoning evaluation.

7 Conclusion↩︎

In this survey, we provide a systematic and comprehensive overview of AI-GGC. We propose a unified survey scheme that organizes existing work by game genres, core commentator capabilities, and commentary types, and use it to analyze methods, datasets, and evaluation metric. Moreover, we highlight key research gaps in current AI-GGC works and identify several promising directions for future work. We hope this survey offers a clear picture of the field and helps guide future research toward more mature AI commentators.

Contribution Statement↩︎

Qirui Zheng, Xingbo Wang, and Keyuan Cheng contributed equally to this work. Wenxin Li is the corresponding author.

References↩︎

[1]
H. Jhamtani, V. Gangal, E. Hovy, G. Neubig, and T. Berg-Kirkpatrick, “Learning to generate move-by-move commentary for chess games from large-scale social forum data,” in Proceedings of ACL, 2018, pp. 1661–1671.
[2]
W.-C. Siu et al., “On the completion of automatic football game commentary system with deep learning,” in IWAIT, 2023.
[3]
P. Andrews, O. E. Nordberg, N. Borch, F. Guribye, and M. Fjeld, “Designing for automated sports commentary systems,” in IMX, 2024, pp. 75–93.
[4]
D. Silver et al., “Mastering the game of go with deep neural networks and tree search,” nature, vol. 529, no. 7587, pp. 484–489, 2016.
[5]
S. Giancola, M. Amine, T. Dghaily, and B. Ghanem, “SoccerNet: A scalable dataset for action spotting in soccer videos,” in CVPRW, 2018, pp. 1792–179210, doi: 10.1109/cvprw.2018.00223.
[6]
A. Deliège et al., “SoccerNet-v2: A dataset and benchmarks for holistic understanding of broadcast soccer videos,” in CVPRW, 2021, pp. 4503–4514, doi: 10.1109/CVPRW53098.2021.00508.
[7]
N. Ranella and M. Eger, “Towards automated video game commentary using generative AI,” in EXAG@AIIDE, 2023, [Online]. Available: https://api.semanticscholar.org/CorpusID:264559591.
[8]
H. Zang, Z. Yu, and X. Wan, “Automated chess commentator powered by neural chess engine,” in Proceedings of ACL, 2019, pp. 5952–5961, doi: 10.18653/v1/P19-1597.
[9]
D. Ghosh et al., “Automated cricket commentary generation using deep learning,” ICIASC, 2023, [Online]. Available: https://api.semanticscholar.org/CorpusID:268587932.
[10]
X. Wang, B. Zhuang, and Q. Wu, “Are large vision language models good game players?” arXiv preprint, 2025.
[11]
Y. Li et al., “Perception, reason, think, and plan: A survey on large multimodal reasoning models,” ArXiv, 2025, [Online]. Available: https://api.semanticscholar.org/CorpusID:278394529.
[12]
J. Kim, J. Goh, I. Hwang, J. Cho, and J. Ok, “Bridging the gap between expert and language models: Concept-guided chess commentary generation and evaluation,” in Proceedings of NAACL-HLT, 2025.
[13]
F. Ma, X. Liang, J. Long, and M. TAo, “Strategies for enhancing card game analysis: Exploring large language models in imperfect information settings,” IEEE Access, 2025.
[14]
C. Frith and U. Frith, “Theory of mind,” Current biology, vol. 15, no. 17, pp. R644–R645, 2005.
[15]
J. Xu and H. Li, “AdaRank: A boosting algorithm for information retrieval,” ACM, 2007.
[16]
T. Kumano et al., “Generation of automated sports commentary from live sports data,” in BMSB, 2019, pp. 1–4, doi: 10.1109/BMSB47279.2019.8971879.
[17]
C.-Y. Chan, C.-C. Hui, W.-C. Siu, S. Chan, and H. A. Chan, “To start automatic commentary of soccer game with mixed spatial and temporal attention,” in Proceedings of TENCON, 2022, pp. 1–6.
[18]
Y. Zhang, A. Agung, P. Paliyawan, and R. Thawonmas, “An audience participation game with difficulty adjustment and rap-style commentary based on audience inputs,” in GCCE, 2021, pp. 845–846, doi: 10.1109/GCCE53005.2021.9621912.
[19]
F. Yan, K. Mikolajczyk, and J. Kittler, “Generating commentaries for tennis videos,” in ICPR, 2016, doi: 10.1109/ICPR.2016.7900036.
[20]
Y. Yan et al., “Fine-grained video captioning via graph-based multi-granularity interaction learning,” IEEE TPAMI, 2022, doi: 10.1109/TPAMI.2019.2946823.
[21]
H. Mei, M. Bansal, and M. R. Walter, “What to talk about and how? Selective generation using LSTMs with coarse-to-fine alignment,” in Proceedings of NAACL-HLT, 2016, doi: 10.18653/v1/N16-1086.
[22]
P. Andrews et al., “AiCommentator: A multimodal conversational agent for embedded visualization in football viewing,” in Proceedings of IUI, 2024, pp. 14–34.
[23]
H. Kameko, S. Mori, and Y. Tsuruoka, “Learning a game commentary generator with grounded move expressions,” in CIG, 2015, doi: 10.1109/CIG.2015.7317930.
[24]
N. Tomlin, A. He, and D. Klein, “Understanding game-playing agents with natural language annotations,” in Proceedings of ACL, 2022, pp. 797–807.
[25]
S. Matsuyoshi, H. Kameko, Y. Murawaki, and S. Mori, “Annotating modality expressions and event factuality for a japanese chess commentary corpus,” in Proceedings of LREC, 2018.
[26]
J. Qi et al., “GOAL: A challenging knowledge-grounded video captioning benchmark for real-time soccer commentary generation,” in Proceedings of CIKM, 2023, pp. 5391–5395.
[27]
B. J. Kim and Y. S. Choi, “Automatic baseball commentary generation using deep learning,” in Proceedings of SAC, 2020, pp. 1056–1065.
[28]
A. Cioppa, A. Deliège, S. Giancola, B. Ghanem, and M. V. Droogenbroeck, “Scaling up SoccerNet with multi-view spatial localization and re-identification,” Scientific Data, vol. 9, 2022, [Online]. Available: https://api.semanticscholar.org/CorpusID:249894209.
[29]
A. Vijayakumar, A. Toms, and S. Vadivu, “Player tracking-integrated soccer game commentary generation,” IJSAT, vol. 16, no. 2, 2025.
[30]
Y. Taniguchi, Y. Feng, H. Takamura, and M. Okumura, “Generating live soccer-match commentary from play data,” in AAAI, 2019, [Online]. Available: https://api.semanticscholar.org/CorpusID:57854642.
[31]
H. Mkhallati, A. Cioppa, S. Giancola, B. Ghanem, and M. V. Droogenbroeck, “SoccerNet-caption: Dense video captioning for soccer broadcasts commentaries,” CVPRW, 2023, [Online]. Available: https://api.semanticscholar.org/CorpusID:258049025.
[32]
R. Zhang and C. Eickhoff, “SOCCER: An information-sparse discourse state tracking collection in the sports commentary domain,” in Proceedings of NAACL, 2021.
[33]
S. Chitrakala, K. Akshaya, and S. Nisha, “STORY SELECTION AND RECOMMENDATION SYSTEM FOR COLOUR COMMENTARY IN CRICKET,” 2015.
[34]
G. Lee, V. Bulitko, and E. Ludvig, “Sports commentary recommendation system (SCoReS): Machine learning for automated narrative,” in Proceedings of the 8th AAAI conference on AIIDE, 2012, pp. 32–37.
[35]
G. Lee, V. Bulitko, and E. A. Ludvig, “Automated story selection for color commentary in sports,” IEEE T-CIAIG, vol. 6, no. 2, pp. 144–155, 2014, doi: 10.1109/TCIAIG.2013.2275199.
[36]
C. Ringer, J. A. Walker, and M. A. Nicolaou, “Multimodal joint emotion and game context recognition in league of legends livestreams,” in IEEE CoG, 2019, doi: 10.1109/CIG.2019.8848060.
[37]
Z. Wang and N. Yoshinaga, “From eSports data to game commentary: Datasets, models, and evaluation metrics,” in DEIM, 2021, [Online]. Available: https://api.semanticscholar.org/CorpusID:235327865.
[38]
D. Zhang, S. Wu, Y. Guo, and X. Chen, MOBA-E2C: Generating MOBA game commentaries via capturing highlight events from the meta-data,” in Findings of the EMNLP, 2022, pp. 4545–4556, doi: 10.18653/v1/2022.findings-emnlp.333.
[39]
P. Liang, M. I. Jordan, and D. Klein, “Learning semantic correspondences with less supervision,” in Proceedings of ACL-IJCNLP, 2009, pp. 91–99.
[40]
Z. Zhang, F. Cao, Y. Mo, Y. Zhang, J. Poon, and S. C. Han, “Game-MUG: Multimodal oriented game situation understanding and commentary generation dataset,” ArXiv, 2024, [Online]. Available: https://api.semanticscholar.org/CorpusID:269456978.
[41]
D. L. Chen, J. Kim, and R. J. Mooney, “Training a multilingual sportscaster: Using perceptual context to learn language,” JAIR, vol. 37, pp. 397–435, 2010.
[42]
T. Ishigaki, G. Topić, Y. Hamazono, I. Kobayashi, Y. Miyao, and H. Takamura, “Audio commentary system for real-time racing game play,” in Proceedings of INLG: System demonstrations, 2023.
[43]
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, Bleu: A method for automatic evaluation of machine translation,” in ACL, 2002, pp. 311–318, [Online]. Available: https://aclanthology.org/P02-1040/.
[44]
C.-Y. Lin, ROUGE: A package for automatic evaluation of summaries,” in Text summarization branches out, 2004, pp. 74–81, [Online]. Available: https://aclanthology.org/W04-1013/.
[45]
M. Denkowski and A. Lavie, “Meteor universal: Language specific translation evaluation for any target language,” in Proceedings of WMT, 2014, pp. 376–380.
[46]
E. Reiter and A. Belz, “An investigation into the validity of some metrics for automatically evaluating natural language generation systems,” Computational Linguistics, vol. 35, no. 4, pp. 529–558, 2009, doi: 10.1162/coli.2009.35.4.35405.
[47]
Y. Liu, D. Iter, Y. Xu, S. Wang, R. Xu, and C. Zhu, “G-eval: NLG evaluation using gpt-4 with better human alignment,” in Proceedings of EMNLP, 2023, pp. 2511–2522.
[48]
J. Ye et al., “Justice or prejudice? Quantifying biases in llm-as-a-judge,” arXiv preprint, 2024.
[49]
L. Shi, C. Ma, W. Liang, X. Diao, W. Ma, and S. Vosoughi, “Judging the judges: A systematic study of position bias in llm-as-a-judge,” arXiv preprint, 2024.
[50]
S. Jung, H. Kim, H. Park, and A. Choi, “Integrated AI system for real-time sports broadcasting: Player behavior, game event recognition, and generative AI commentary in basketball games,” Applied Sciences, 2025.
[51]
L. Li, Y. Lu, Y. Wang, Q. Zheng, and W. Li, “Mxplainer: Explain and learn insights by imitating mahjong agents,” Algorithms, vol. 18, no. 12, p. 738, 2025.
[52]
F. B. Von Der Osten, M. Kirley, and T. Miller, “The minds of many: Opponent modeling in a stochastic game.” in IJCAI, 2017.
[53]
A. Valdivia and P. Burelli, “Evaluating quality of gaming narratives co-created with AI,” in CoG, 2025.