1 Introduction↩︎

The ability to flexibly combine existing knowledge to solve new problems is a hallmark of human intelligence [1], [2]. This capacity, termed as compositional generalization [3], [4], enables us to generate a potentially infinite range of complex ideas and behaviors from a finite set of known components [5]. Modern artificial intelligence, particularly large-scale models, has demonstrated remarkable capabilities in processing and generating human-like data[6]. Yet, a critical gap persists between their pattern-matching prowess and the robust, systematic reasoning characteristic of human intelligence[7]. These models often fail at tasks requiring logical deduction and multi-step planning, revealing a fundamental weakness in their ability to generalize systematically [8], [9].

To address this, the community has developed benchmarks such as SCAN [10] and COGS [11] to test for compositional generalization, primarily in the domain of natural language processing. However, a gap exists for benchmarks that test compositional reasoning in more structured, symbolic problem-solving domains [12]. Such domains require not only recognizing patterns but also building and executing explicit, multi-step strategies.

In this paper, we introduce ClassicLogic, a novel benchmark designed to provide a transparent and challenging testbed for compositional reasoning. Our contribution is a suite of four classic logic puzzles—Sudoku, KenKen, Kakuro, and Futoshiki—that are procedurally generated and comes with a hierarchical knowledge base (KB) of solving strategies, where complex solving strategies are explicitly defined as compositions of simpler, atomic rules. For instance, advanced strategies are explicitly defined as compositions of more fundamental ones (e.g., identifying a ‘naked_pair’ in Sudoku is a composition of the ‘naked_single’ and ‘constraint_propagation’ strategies). This structure allows us to dissect the reasoning process and evaluate three distinct and crucial forms of compositional generalization: (1) Entity Composition, such as identifying handwritten digits as numerical entities within the puzzle grid; (2) Relational Composition, such as integrating cell-value states with arithmetic or inequality constraints; and (3) Procedural Composition, such as chaining simple solving rules into complex, multi-step strategies.

By providing a framework to test these multi-facets of reasoning, ClassicLogic offers a more nuanced evaluation than simple accuracy metrics. In summary, our main contributions are:

• A novel, open-source benchmark (ClassicLogic) featuring four challenging logic puzzles with procedurally generated instances and guaranteed unique solutions.

• A hierarchical knowledge base (KB) for each game base, where complex strategies are explicitly composed of simpler ones, allowing for fine-grained control over task difficulty and required reasoning depth.

• To our knowledge, the first unified framework to explicitly disentangle and evaluate three distinct forms of compositional generalization: Entity Composition (perception), Relational Composition (rule-following), and Procedural Composition (strategic planning).

It enables researchers to diagnose why a model fails—whether it is unable to learn base rules, combine them, or transfer them. As such, ClassicLogic provides a crucial new tool to guide the development of more robust, interpretable, and systematically intelligent AI systems.

2 Related Work↩︎

The evaluation of compositional generalization has been a central theme in cognitive science and AI. Early work highlighted the systematicity of human thought as a core property of intelligence [5], a concept that modern benchmarks aim to quantify.

2.1 Benchmarks for Compositional Generalization↩︎

Existing benchmarks have largely focused on linguistic and visual domains. In linguistics, datasets like SCAN [10] and COGS [11] evaluate whether models can generalize to novel combinations of familiar words and commands. Similarly, alongside them, others have emerged to test more complex facets. For instance, gSCAN [13] tests generalization in visually-grounded instruction following , while CFQ provides a more realistic testbed based on querying a large knowledge base [14]. In vision, the CLEVR dataset [15] evaluates for compositional reasoning about objects and their spatial and semantic relationships. Subsequent work like CLEVRER added a temporal dimension to probe causal reasoning [16], and GQA introduced a more naturalistic and compositionally complex question-answering dataset [17]. While these benchmarks are foundational for assessing generalization on perceptual and sequential data, they primarily test for an implicit understanding of composition. They are not designed to evaluate an agent’s ability to learn and execute an explicit, multi-step procedural strategy from a symbolic knowledge base. However, ClassicLogic provides a ground-truth and symbolic hierarchy of strategies. This allows for direct evaluation of an agent’s ability to build and execute complex plans in a discrete, symbolic environment.

2.2 AI in Games and Automated Reasoning↩︎

The use of games as a crucible for artificial intelligence is a long and storied tradition. Landmark achievements, such as DeepMind’s AlphaGo for GO [18] and Pluribus for poker [19], have demonstrated superhuman performance in complex strategy games. Such immensely powerful systems often rely on reinforcement learning, trained over massive volumes of self-play and functioning as opaque, “black-box" agents. The strategies they learn are emergent and subsymbolic, not explicitly represented in a human-interpretable format [20]. In contrast to opaque deep RL agents, ClassicLogic provides a transparent evaluation environment where the goal is not to create an optimal player, but to diagnose if an agent can learn and deploy human-cognizable strategies.

Moreover, there is extensive research on automated puzzle solvers, particularly for logic puzzles [21]–[23]. These problems are often formally framed as Constraint Satisfaction Problems (CSPs) [24], [25], for which highly efficient and specialized solvers exist. However, the goal of these systems is to find a valid solution as quickly as possible because they are optimized for the final product. They do not evaluate whether an agent can learn the human-like reasoning process required to get there [26]. Whereas, ClassicLogic is uniquely focused on evaluating the reasoning process itself. It provides a framework to test how an agent learns, not just whether it can produce a correct answer. It is not designed to compete with bespoke CSP solvers, but rather to serve as a benchmark that tests whether a learning agent can build an understanding of the puzzle’s underlying principles by learning and composing strategies from our hierarchical knowledge base.

2.3 Neuro-Symbolic Approaches↩︎

A rising field that directly intersects with our work is Neuro-Symbolic (NeSy) AI. The goal of NeSy is to integrate the pattern-recognition strengths of neural networks with the reasoning capabilities of symbolic systems [27]–[29]. Systems like DeepProblog [30] and Neural Logic Machines [31] attempt to build models that can perform explicit logical reasoning on top of neural perception. ClassicLogic provides an ideal testbed for such systems, as it contains both perceptual components (e.g., recognizing MNIST digits) and a complex, symbolic reasoning structure (the strategy KB). Currently, a major challenge for the NeSy field is the lack of comprehensive benchmarks that require deep, compositional, and symbolic reasoning; we aim to fill this gap [12]. In regards to that, our benchmark serves as a testbed for NeSy systems as it provides a challenging new environment that requires the tight integration of perception and multi-step symbolic reasoning.

3 The ClassicLogic Benchmark Suite↩︎

The ClassicLogic benchmark is a suite of four procedurally generated logic puzzle environments designed to facilitate the diagnostic evaluation of compositional reasoning in artificial agents. In this section, we formalize its architecture, the structure of its game environments, and its core innovation: the hierarchical knowledge base of solution strategies.

3.1 Design Principles↩︎

The development of ClassicLogic is guided by four core principles to ensure a rigorous and fair evaluation platform for end-to-end reasoning systems:

• Perceptual Grounding: All initial puzzle states are presented to the agent not as symbolic matrices, but as visual grids rendered with MNIST digit images. This design choice necessitates an initial perceptual function, \(\Phi: I \rightarrow G\), to map the visual input from the image space \(I\) to a symbolic grid state \(G\), creating a unified testbed for integrated perception and reasoning.

• Strategy-Driven Generation: A puzzle instance \(P\) is not generated randomly, but is constructed to require a specific, minimal set of strategies \(\Sigma_{\text{req}}\) for its solution. This allows for targeted testing of an agent’s ability to deploy specific reasoning patterns.

• Validated Difficulty Scaling: The difficulty of a puzzle \(P\), denoted by a function \(D(P)\), is directly correlated with the compositional depth of the strategies in \(\Sigma_{\text{req}}\). We define compositional depth, \(\delta(s)\), of a strategy \(s\) recursively, providing a formal measure of puzzle complexity.

• Guaranteed Uniqueness: For every puzzle instance \(P\) generated by the benchmark, there exists a unique solution state \(S_{\text{sol}}\). This eliminates ambiguity in evaluation and ensures that success is based on valid logical deduction.

3.2 Game Environments↩︎

The suite comprises four classic logic puzzles, each formalized as a distinct environment, as mentioned in Table 1. Let a grid state be a matrix \(G\) of size \(N \times N\). The state of any puzzle, \(P\), can be represented as a tensor \(T \in \mathbb{R}^{N \times N \times K}\), where \(N\) is the grid dimension and \(K\) represents the features of each cell (e.g., a one-hot vector for candidate digits). The action space \(\mathcal{A}\) is typically discrete, allowing an agent to set a value \(v\) in a cell \((i, j)\), i.e., \(\mathcal{A} \subseteq \{ (i, j, v) \}_{i,j,v=1}^{N}\).

3.2.0.1 Sudoku

The environment is a partially filled \(9 \times 9\) grid \(G\). The objective is to fill the empty cells with digits \(d \in \{1, ..., 9\}\) subject to the constraint that for any cell \(G_{ij}\), its value must be unique in its row, column, and designated \(3 \times 3\) subgrid. Formally, \(\forall i, j, k \in \{1, ..., 9\}\) where \(i \neq k\), \(G_{ij} \neq G_{kj}\); where \(j \neq k\), \(G_{ij} \neq G_{ik}\); and for all cells \((i', j')\) in the same \(3 \times 3\) block as \((i, j)\), \(G_{ij} \neq G_{i'j'}\).

3.2.0.2 KenKen

An \(N \times N\) grid is partitioned into a set of non-overlapping cages \(C = \{c_1, c_2, ..., c_m\}\). Each cage \(c_k \in C\) is defined by a tuple \((\mathcal{Z}_k, V_k, O_k)\), where \(\mathcal{Z}_k\) is the set of cell coordinates in the cage, \(V_k \in \mathbb{N}^+\) is a target value, and \(O_k \in \{+, -, \times, \div\}\) is a binary arithmetic operator. The constraint for each cage is that the application of operator \(O_k\) to the values in the cells of \(\mathcal{Z}_k\) must result in \(V_k\).

3.2.0.3 Kakuro

The grid consists of “clue cells" and”entry cells." A clue cell contains one or two clue values, \((c_{\text{down}}, c_{\text{right}})\), where \(c \in \mathbb{N}^+\). The constraint is that the sum of the digits in the contiguous block of entry cells below the clue must equal \(c_{\text{down}}\), and the sum of the digits in the block to its right must equal \(c_{\text{right}}\). Furthermore, all digits within any given block must be unique.

3.2.0.4 Futoshiki

An \(N \times N\) grid must be filled with digits \(\{1, ..., N\}\) adhering to Latin square rules (uniqueness in rows and columns). Additionally, the environment contains a set of relational constraints \(\mathcal{R}\) of the form \(G_{ij} \prec G_{i'j'}\), where \((i,j)\) and \((i',j')\) are adjacent cell coordinates and \(\prec \in \{<, >\}\).

To provide a concrete visualization of these environments, Figure 1 presents sample instances for Sudoku, KenKen, and Futoshiki. Each example displays both the initial puzzle state, as it would be presented to an agent, and its corresponding unique solution. These samples illustrate the diversity of constraints and logical structures that agents must handle within the ClassicLogic benchmark.

3.3 The Hierarchical Knowledge Base (KB)↩︎

The central innovation of ClassicLogic is its explicit, hierarchical knowledge base. Formally, the KB for each game is a tuple \(\mathcal{K} = (\mathcal{S}, \mathcal{C})\), where \(\mathcal{S}\) is a set of strategies and \(\mathcal{C}\) is a composition relation defined on \(\mathcal{S}\).

The set of strategies \(\mathcal{S}\) is partitioned into a set of atomic base strategies \(\mathcal{S}_{\text{base}}\) and a set of composed strategies \(\mathcal{S}_{\text{comp}}\). A strategy \(s \in \mathcal{S}\) is a function that maps a puzzle state \(P_t\) to a new, more constrained state \(P_{t+1}\) by making a valid logical deduction, i.e., \(s: P_t \mapsto P_{t+1}\).

The composition relation \(\mathcal{C}\) defines how strategies in \(\mathcal{S}_{\text{comp}}\) are constructed. For any composed strategy \(s_c \in \mathcal{S}_{\text{comp}}\), there exists an ordered set of constituent strategies \(\{s_1, s_2, ..., s_k\} \subseteq \mathcal{S}\) such that \(s_c\) represents the sequential application or logical combination of these constituents. We can denote this composition as \(s_c := \bigoplus_{i=1}^{k} s_i\).

For example, consider the state of a Sudoku puzzle \(P\) as a grid of candidate sets, where \(P_{ij} \subseteq \{1, ..., 9\}\) is the set of possible values for cell \((i, j)\). A base strategy, Constraint Propagation (\(s_{\text{cp}}\)), takes a confirmed cell value and removes it from the candidate sets of all peer cells. This simple action is a foundational component for more complex strategies. A moderate strategy like Naked Pair (\(s_{\text{np}}\)) is composed of a pattern-recognition step followed by this action, which we can denote as \(s_{\text{np}} := s_{\text{find\_pair}} \oplus s_{\text{cp}}\). The sub-procedure, \(s_{\text{find\_pair}}\), identifies when two cells in a unit share the exact same two candidates (e.g., \(P_{ij} = P_{ik} = \{a, b\}\)). This finding then enables the application of \(s_{\text{cp}}\) to eliminate candidates \(a\) and \(b\) from all other cells in that unit.

This compositional principle extends to more advanced, multi-unit strategies as visualized in Figure 2. An X-Wing (\(s_{\text{xw}}\)) is a more complex moderate strategy that is similarly composed of a sophisticated pattern-recognition procedure followed by the same base action: \(s_{\text{xw}} := s_{\text{find\_xwing}} \oplus s_{\text{cp}}\). Here, \(s_{\text{find\_xwing}}\) is itself a composition of a more basic procedure, let’s call it Unit Scan (\(s_{\text{scan}}\)), which finds the positions of a candidate in a unit. An X-Wing pattern is identified for a digit \(d\) when there exist two rows \(r_1, r_2\) and two columns \(c_1, c_2\) such that the repeated application of \(s_{\text{scan}}\) confirms that \(\text{Positions}(d, r_1) = \{c_1, c_2\}\) and \(\text{Positions}(d, r_2) = \{c_1, c_2\}\).

The hierarchy deepens further with hard strategies like Swordfish (\(s_{\text{sf}}\)), which is a direct extension of the X-Wing logic. It is composed as \(s_{\text{sf}} := s_{\text{find\_swordfish}} \oplus s_{\text{cp}}\), where \(s_{\text{find\_swordfish}}\) is a scaled-up version of the X-Wing’s pattern recognition. It applies the same foundational \(s_{\text{scan}}\) procedure but seeks a 3x3 pattern across three rows and three columns. This demonstrates a deep compositional structure where advanced strategies are formed not just by combining different simple rules, but by scaling and reapplying the same underlying logical patterns in a more complex configuration.

3.4 Procedural Generation and Difficulty Calibration↩︎

A central requirement for a robust benchmark is the ability to generate a vast and diverse set of puzzles with precisely controlled properties. Standard reverse-solving algorithms, which iteratively remove clues from a solved grid, are often computationally intractable. To address this, we employ a sophisticated two-stage hybrid generation process. This methodology separates the computationally expensive task of logical structure generation from the lightweight task of producing playable instances, providing both rigor and real-time performance.

3.4.1 Stage 1: Strategy-Driven Template Generation↩︎

The goal of this stage is to produce a library, \(\mathbb{TPL}\), of minimal, abstract puzzle structures, or templates. Each template represents the logical essence of a puzzle class with a specific difficulty.

We formally define a puzzle instance as a tuple \(P = (\mathcal{G}, \mathcal{C}_P)\), where \(\mathcal{G}\) is a grid structure and \(\mathcal{C}_P\) is a set of concrete clues (e.g., pre-filled cell values, arithmetic cage definitions). A solution \(S\) is a complete assignment of values to \(\mathcal{G}\) that satisfies all game rules and the clues in \(\mathcal{C}_P\).

A template is an abstracted puzzle, \(Tpl = (\mathcal{L}, \mathcal{C}_{\text{abs}}, \Sigma_{\text{req}})\), where \(\mathcal{L}\) is a logical layout (e.g., a graph representation of the grid), \(\mathcal{C}_{\text{abs}}\) is a set of abstract placeholder constraints, and \(\Sigma_{\text{req}}\) is the pre-analyzed, minimal set of strategies required to solve any valid instantiation of this template.

Template generation (Algorithm 3) begins with a canonical solution grid, \(S_{\text{sol}}\). It then performs an iterative clue ablation process. For each clue \(c \in \mathcal{C}_{S_{\text{sol}}}\), it creates a candidate puzzle \(P' = (\mathcal{G}, \mathcal{C}_{P} \setminus \{c\})\). This candidate is validated against two criteria using two distinct solvers:

1. Uniqueness Check: A complete, optimal solver, \(\texttt{FullSolver}(P')\), which returns the set of all valid solutions. The removal is valid only if \(| \texttt{FullSolver}(P') | = 1\).

2. Difficulty Check: A constrained solver, \(\texttt{LimitedSolver}(P', \mathcal{K}_{\leq \mathcal{D}_{\text{target}}})\), equipped only with strategies from the Knowledge Base whose compositional depth \(\delta(s)\) is less than or equal to the target difficulty \(\mathcal{D}_{\text{target}}\). The removal is valid only if this solver finds the solution, ensuring the puzzle does not require harder strategies.

The final, minimal puzzle \(P_{\text{final}}\) is then abstracted into a canonical template \(Tpl\) by a function \(f_{\text{abs}}: P \rightarrow Tpl\). This intensive process is run to populate the template library \(\mathbb{TPL}\). To validate our formal difficulty metric, we conducted an empirical study correlating \(D(P)\) with the performance of our baseline heuristic solver (see Section 4.1).

3.4.2 Stage 2: Real-time Puzzle Instantiation↩︎

With a rich library \(\mathbb{TPL}\) of pre-generated and validated templates, creating a playable puzzle instance is computationally trivial. This process (Algorithm 4) involves template selection followed by instantiation and transformation.

The instantiation function, \(f_{\text{inst}}: Tpl \rightarrow P_{\text{base}}\), maps the template’s abstract structure \(\mathcal{L}\) and constraints \(\mathcal{C}_{\text{abs}}\) to a concrete puzzle instance \(P_{\text{base}}\). This is achieved via a randomized backtracking search that assigns concrete values while satisfying the template’s constraints.

To generate a wide variety of structurally equivalent but superficially distinct puzzles from a single template, we apply a transformation \(\tau\) from a set of symmetry-preserving transformations \(\mathcal{T}\) (e.g., rotation, reflection, and value permutations). The final puzzle is \(P_{\text{final}} = \tau(P_{\text{base}})\). This two-stage architecture ensures both logical rigor and real-time performance.

3.5 A Framework for Evaluating Compositional Generalization↩︎

The ClassicLogic benchmark provides a structured framework to evaluate three distinct and critical forms of compositional generalization. We formalize these as follows, from basic perception to multi-step strategic planning.

3.5.0.1 1. Entity Composition

is the foundational ability to learn a mapping function, \(\Phi: I \rightarrow E\), from the raw perceptual input space \(I\) (pixel grids) to a set of symbolic entities \(E\). An entity \(e \in E\) is a tuple representing a fundamental object, such as \((\text{digit}, 7)\) or \((\text{operator}, >)\). An agent demonstrates successful entity composition if, for every initial clue image \(i_{r,c}\) at grid location \((r,c)\), its inferred entity \(\hat{e}_{r,c} = \Phi(i_{r,c})\) equals the ground-truth entity \(e_{r,c}\).

• Example 1 (KenKen): An agent is presented with a visual cage containing the clue “7+". Successful composition requires generating the entity set \(\{(\text{digit}, 7), (\text{operator}, +)\}\). An output of \(\{(\text{digit}, 1), (\text{operator}, +)\}\) would represent a critical failure in entity recognition.

• Example 2 (Futoshiki): Given a grid of MNIST digits and \(>\) symbols, a successful agent must parse the entire initial state \(P_0\) into a set of symbolic facts, such as \(\{\text{Value}(r_1, c_1, 5), \text{Constraint}(r_2, c_2, >) ...\}\). Misclassifying a handwritten digit or an operator indicates a failure at this basic compositional level.

3.5.0.2 2. Relational Composition

is the ability to apply a set of universal relational rules, \(\mathcal{R}\), to the set of entities \(E\) to infer new facts or validate state transitions. A rule \(R \in \mathcal{R}\) is a predicate over one or more entities that must hold true for a state to be valid. An agent’s action \(a_t\) at state \(S_t\) is valid only if the resulting state \(S_{t+1}\) satisfies all rules, i.e., \(\forall R \in \mathcal{R}, R(S_{t+1}) = \text{true}\). An illegal move signals a failure in relational composition.

• Example 1 (Futoshiki): Given the symbolic facts \(\text{Constraint}(c_1, c_2, >)\) and \(\text{Value}(c_2, 4)\), the agent must apply the rule: \[\forall c_1, c_2, v_1, v_2: (\text{Value}(c_1, v_1) \land \text{Value}(c_2, v_2) \land \text{Constraint}(c_1, c_2, >)) \rightarrow v_1 > v_2\] By composing the rule with the known facts, a successful agent infers that the candidate set for \(c_1\) is now constrained, \(\text{Candidates}(c_1) \subseteq \{5,6,7,8,9\}\).

• Example 2 (Sudoku): An agent in a state with \(\text{Value}(1,2, 7)\) considers the action \(a_t = \text{place}(1,5,7)\). To validate this, it must apply the row-uniqueness rule: \[\forall r, c_1, c_2, v: (c_1 \neq c_2 \land \text{Value}(r, c_1, v)) \rightarrow \neg \text{Candidate}(v, r, c_2)\] Since this rule is violated by the proposed action, a successful agent would prune this action from its policy.

3.5.0.3 3. Procedural Composition

This is the highest form of compositionality, representing the ability to construct a new procedure (strategy), \(s_{\text{new}}\), by sequencing a set of known, simpler procedures \(\{s_1, ..., s_k\} \subseteq \mathcal{S}_{\text{known}}\). Formally, \(s_{\text{new}} := s_k \circ ... \circ s_1\), where \(\circ\) represents sequential application or functional composition. We train an agent on puzzles \(\mathcal{P}_{\text{base}}\) solvable only by procedures in \(\mathcal{S}_{\text{base}}\). We then test it on puzzles \(\mathcal{P}_{\text{comp}}\) that require at least one procedure \(s_c \notin \mathcal{S}_{\text{base}}\), where \(s_c\) is a known composition of procedures from \(\mathcal{S}_{\text{base}}\). Success on \(\mathcal{P}_{\text{comp}}\) provides direct evidence of procedural composition.

• Example 1 (Sudoku): An agent knows the procedure \(s_{\text{pattern\_rec}}\) (recognize when two cells in a row have candidates \(\{2,8\}\)) and \(s_{\text{elimination}}\) (apply constraint propagation). It must compose these to form a novel strategy, \(s_{\text{naked\_pair}} := s_{\text{elimination}} \circ s_{\text{pattern\_rec}}\), where the output of the pattern recognition (the cells and candidates to eliminate) becomes the input for the elimination procedure.

• Example 2 (Kakuro): An agent knows \(s_{\text{unique\_sum}}\) (e.g., a 2-cell sum of 3 must be \(\{1,2\}\)) and \(s_{\text{cross\_ref}}\) (use a confirmed value to constrain an intersecting block). Faced with intersecting blocks, it must compose a new procedure: first, apply \(s_{\text{unique\_sum}}\) to reduce one block’s candidates to \(\{1,2\}\); then, apply \(s_{\text{cross\_ref}}\) using a known digit from the second block to disambiguate the first. This chain of dependent procedures is a hallmark of procedural composition.

3.6 Implementation and Accessibility↩︎

To maximize reproducibility and ease of adoption, the ClassicLogic benchmark is implemented with a strong focus on accessibility and a streamlined setup process. The entire suite is developed in Python 3 and its dependencies are explicitly managed. A key feature is the unified environment setup. We provide automated scripts designed to create a self-contained virtual environment and install all necessary packages with a single command. The repository includes ‘unified_env_setup.py’ (for POSIX systems) and unified_env_quick_setup.bat (for Windows), which handle the environment creation and dependency installation listed in the ‘requirements.txt’ file. This eliminates common setup hurdles and ensures that researchers can begin experimentation with minimal friction. Furthermore, environment configurations are managed through a central ‘classic_games_env_info.json’ file. This file contains metadata and settings for the benchmark, allowing for easy modification and inspection of the environment’s parameters without altering the core source code. The framework is also designed with modularity and extensibility in mind. Adding a new puzzle to the suite follows a well-defined template, primarily requiring the implementation of the game’s state transition logic and its corresponding strategy knowledge base. This structure encourages community contributions and the long-term expansion of ClassicLogic with new reasoning challenges.

The entire framework, including the source code, setup scripts, and documentation, is publicly available under an MIT License on GitHub¹ to encourage open collaboration and extension for the research community.

4 Experiments↩︎

We conducted a set of experiments to demonstrate how this benchmark can be utilized to evalute an agent’s compositional reasoning abilities using ClassicLogic.

4.1 Preliminary Validation of the Difficulty Metric↩︎

4.1.0.1 Objective

Before using ClassicLogic to evaluate AI agents, we first conducted a preliminary experiment to empirically validate that our theoretical difficulty metric, \(D(P)\), derived from the Knowledge Base, correlates with practical computational hardness. To establish both internal and external validity, we measured performance using two distinct automated solvers.

4.1.0.2 Methodology

For this validation study, we focused on Sudoku as a representative environment, given its well-understood properties and rich strategy space. We generated a validation set of 99 Sudoku puzzles, comprising 33 unique instances for each difficulty category (Easy, Moderate, Hard). We then used two distinct automated solvers to solve each puzzle:

• Glass-Box Solver: Our implemented baseline agent that greedily applies strategies from the Knowledge Base \(\mathcal{K}\). We measured the total number of distinct strategies applied to reach a solution.

• Black-Box Solver: A standard, optimized Constraint Satisfaction Problem (CSP) solver. This solver is unaware of our KB. We measured the wall-clock solve time in milliseconds.

Our hypothesis is that as the difficulty category progresses from Easy to Hard, the metrics from both solvers will increase significantly. We treated the categories as ranks (Easy=1, Moderate=2, Hard=3) and computed the Spearman’s rank correlation coefficient (\(\rho\)).

4.1.0.3 Results

The results, summarized in Table 2 and visualized in Figure 5, confirm our hypothesis. The number of strategies applied by the glass-box solver and the solve time for the black-box CSP solver both increase substantially with each difficulty category, demonstrating a strong, positive, and monotonic relationship.

The strong correlation with an independent, black-box solver (\(\rho = 0.98, p < 0.01\)) provides the evidence that our formal difficulty metric is a valid proxy for intrinsic computational hardness. This validation of our methodology on a representative puzzle provides confidence that the difficulty scaling is a robust feature of the entire benchmark suite.

4.2 Evaluating Compositional Generalization↩︎

We evaluated three representative neuro-symbolic (NeSy) models using the ClassicLogic benchmark to assess their compositional generalization capabilities. The models were chosen to represent different approaches to integrating neural and symbolic reasoning:

• SATNet [32]: An architecture that integrates a differentiable MAX-SAT solver, allowing a neural network to learn the logical constraints of a problem.

• Neural Module Network (NMN) [33]: A model that dynamically assembles a custom neural network from smaller, specialized modules to create a task-specific reasoning pipeline.

• Logical Neural Network (LNN) [34]: A framework where each neuron has a clear logical meaning (e.g., AND, OR), creating a highly interpretable network that adheres to formal logic.

The following experiments are intended to demonstrate the benchmark’s utility in evaluating and differentiating the capabilities of various models across all four game environments. We conducted two distinct sets of experiments to probe different facets of compositional generalization. First, in Section 4.3, we evaluated the ability of models to perform zero-shot transfer of perceptual and relational knowledge from a single training game to three unseen games. Second, in Section 4.4, we test for in-domain procedural generalization, evaluating if models can compose known strategies into novel, more complex ones within each game environment.

4.3 Entity and Relational Composition Performance↩︎

4.3.0.1 Training Protocol

To rigorously test for zero-shot generalization, all models were trained exclusively on the Sudoku environment. For Entity Composition, the models’ perceptual frontends were trained to recognize the MNIST digits used as clues in Sudoku puzzles. For Relational Composition, the models were trained only on Sudoku’s specific rule set: the uniqueness constraint within rows, columns, and 3x3 blocks. Their performance was then evaluated in a zero-shot setting on the unseen visual and logical structures of KenKen, Kakuro, and Futoshiki.

4.3.0.2 Results

The results, summarized in Table 3, reveal the profound challenge of out-of-domain generalization for these models. In Entity Composition, while all models learned to parse Sudoku digits well, their accuracy dropped significantly on the other games. The LNN performed best, maintaining accuracy in the low 80s, but all models struggled, indicating that their visual representations were not abstract enough to generalize to the novel layouts and symbols (e.g., cages, operators) in the other puzzles. The performance on Relational Composition tells a similar story. The models found it extremely difficult to transfer the learned Sudoku rules to the fundamentally different constraints of the other games. The LNN was the most robust, maintaining the lowest illegal move rate, which is consistent with its logic-based architecture. However, even it made frequent errors when faced with the arithmetic and ordinal logic of KenKen and Futoshiki. SATNet and NMN struggled even more, with illegal move rates reaching as high as 45.5% for the NMN in the Kakuro environment. This demonstrates a critical failure to generalize abstract logical principles, a key weakness that this benchmark effectively exposes.

4.4 Procedural Composition Performance↩︎

4.4.0.1 Training Protocol

This experiment tests the most challenging facet of reasoning: composing known rules into novel, multi-step strategies. Unlike the previous zero-shot transfer task, this is a test of in-domain procedural generalization. For each of the four games, all three models were trained on a curriculum of “Easy" and”Moderate" puzzles. This training corpus exposed them to all the base and moderate-level strategies in the Knowledge Base. The models were then tested on a set of “Hard" puzzles, which are guaranteed to require composed strategies in sequences not seen during training.

4.4.0.2 Results

The results, shown in Figure 6, are striking and reveal a critical failure point for all tested architectures. Despite their good performance on adhering to static rules (Relational Composition), all models showed an inability to compose those rules into novel, multi-step strategies. The success rates on the Hard puzzles collapsed to below 25% across the board.

This demonstrates a stark disconnect between knowing the rules of a game and knowing how to play it strategically. The LNN, which was the most robust in the relational task, remains the top performer here, but its low success rate (20-24%) underscores the immense difficulty of the challenge. The NMN, which might have been expected to perform well due to its modularity, struggled the most, suggesting its dynamic assembly process failed to generalize to more complex procedures. These results powerfully illustrate that even models with strong logical priors are not inherently equipped for the flexible, sequential planning required by our benchmark.

5 Discussion↩︎

Our experiments demonstrate that ClassicLogic can effectively diagnose a critical failure mode in modern neuro-symbolic systems. The ability to pinpoint this specific gap between relational and procedural generalization is a direct result of the benchmark’s design, which provides a unified framework for evaluating these distinct layers of compositionality in a single, coherent environment. While models adept at learning static facts and rules, they remain brittle at composing those rules into dynamic, multi-step procedures. The severe performance collapse on “Hard" puzzles is not merely a quantitative failure; it is a qualitative one that exposes a deep gap between pattern recognition and true strategic reasoning.

This finding suggests that many current architectures, even those with strong logical priors like LNNs and SATNet, lack the mechanisms for flexible, sequential planning. Knowing the rules of chess (relational composition) is fundamentally different from knowing how to play it well (procedural composition).

Ultimately, this validates ClassicLogic not just as another benchmark, but as a necessary course correction for the field. It challenges the community to move beyond tasks that can be solved with static rule application and to focus on the more difficult, more human-like skill of composing knowledge into novel plans. We believe that progress on the kind of challenges presented in our benchmark is a prerequisite for building the next generation of robust, trustworthy, and truly intelligent AI systems.

6 Limitations↩︎

While ClassicLogic provides a comprehensive framework, it is important to acknowledge its scope and inherent limitations to properly situate its contributions and guide future work. The four puzzles in the suite were selected to cover a diverse range of logical constraints (e.g., arithmetic, relational, spatial). However, they represent only a subset of the vast world of logic puzzles. The benchmark does not currently include other families such as graph-coloring puzzles, temporal logic puzzles, or those requiring extensive common-sense knowledge. Moreover, we have curated a hierarchically structured KB for each game based on well-documented human strategies. However, we do not claim this KB is exhaustive. Humans may employ novel or idiosyncratic strategies not captured in our formal system.

These limitations do not diminish the benchmark’s utility but rather clarify its intended purpose: to provide a deep, diagnostic tool for a specific, vital, and currently under-evaluated facet of artificial intelligence. We believe these focused evaluations are a prerequisite for building more general and robust AI systems, and we welcome community contributions to expand the benchmark’s scope in the future.

7 Conclusion↩︎

In this work, we introduced ClassicLogic, which to best of our knowledge is the first benchmark to provide a unified framework for evaluating the full stack of compositional reasoning: from Entity Composition (perception) to Relational Composition (rule-following) and finally to Procedural Composition (strategic planning). Our illustrative experiments revealed a profound gap: while current neuro-symbolic models excel at adhering to static rules, they fail catastrophically when required to compose known procedures into novel, multi-step plans. This finding validates ClassicLogic as a crucial diagnostic tool that moves beyond simple rule-following to probe the deeper challenge of strategic reasoning. By providing a clear and challenging measure of procedural generalization, we hope to guide the community toward building the next generation of AI that can not only learn facts but can compose them into robust and trustworthy strategies.

References↩︎

---

1. https://github.com/Place-Beyond-Bytes/classic_games_benchmark.git↩︎