benchmark, dataset generation, mathematical reasoning, information acquisition, clarification, partial observability, large language models

1 Introduction↩︎

Large language models are increasingly evaluated on mathematical reasoning, tool use, and interactive problem solving. Yet many high-profile math benchmarks assess only the final answer under full information [1]–[3], while broad interactive benchmarks mix multiple sources of difficulty, including tool selection, retrieval, web navigation, external APIs, and long-context management [4]–[6]. These settings are valuable, but they make it difficult to isolate a basic bottleneck that appears whenever a model does not initially possess all relevant information: can the model recognize the missing fact, ask for that fact precisely, and use the answer correctly?

MIRA-Math is designed to isolate this capability. Each benchmark instance is generated from a complete mathematical state with a unique answer, but the solver model receives only a private view that is deliberately insufficient, missing exactly one necessary atomic fact. A fixed information-holder model receives only the missing fact and is constrained by a structured-output protocol: if the solver’s request semantically matches the information it holds, it returns that fact; otherwise, it returns a declination stating that it does not have the requested information. The information holder does not solve, explain, negotiate, or volunteer extra hints. The benchmark therefore measures minimal information requesting under a controlled responder channel, not open-ended multi-agent collaboration.

This framing is intentionally data-centric. Following the emphasis on public benchmark infrastructure and responsible dataset design in recent data-centric machine-learning work [7], [8], MIRA-Math includes generator code, exact verifiers, canonical hint specifications, prompt templates, run metadata, and dataset documentation. The benchmark is closest to diagnostic benchmark datasets that expose hidden structure or evaluation gaps through carefully controlled data construction [9]–[12], but its target capability is different: information acquisition under partial mathematical observability.

1.0.0.1 Contributions.

This paper makes three contributions:

• We define a reproducible protocol for minimal information requesting in mathematical reasoning, with typed atomic hints, a fixed constrained LLM responder using structured offer/decline outputs, deterministic dataset validation, and exact answer checking.

• We report baseline evaluations across multiple solver models and prompt regimes, over a benchmark of 2,310 instances showing that first-request success, request hit rate, and final accuracy capture distinct failure modes.

• We provide dataset documentation, intended-use guidance, and a maintenance plan.

2 Related Work↩︎

MIRA-Math is positioned at the intersection of mathematical reasoning benchmarks, clarification and information-acquisition benchmarks, active reasoning under incomplete information, and data-centric benchmark design. Table ¿tbl:tab:positioning? summarizes the closest literature streams and the distinction between MIRA-Math and prior work.

3pt max width=

2.1 Final-answer mathematical reasoning is not enough↩︎

Mathematical reasoning benchmarks such as GSM8K and MATH have been central for measuring whether language models can produce correct final answers from complete problem statements [1], [2]. More recent evaluations increase difficulty, reduce contamination, or track competition-level performance [3]. These benchmarks are important, but they usually assume that the model receives all information required to solve the problem. As a result, they do not isolate the ability to notice that a problem is underdetermined, identify the missing fact, ask for it, and then use it correctly.

MIRA-Math targets this missing capability directly. It should therefore not be read as a replacement for full-information mathematical benchmarks. Instead, it complements them by turning information acquisition into an explicitly measured step. A model that is strong on ordinary math benchmarks may still fail in MIRA-Math if it cannot identify the missing atomic fact or if it asks for an irrelevant quantity.

2.2 Controlled diagnostic benchmarks↩︎

A second relevant stream uses controlled generation and exact evaluation to reveal hidden model weaknesses. Symbolic and scientific benchmark work has shown the value of generator-controlled tasks, exact verifiers, and diagnostic decompositions [9], [13]. Data-centric benchmark papers similarly emphasize that the dataset, generator, evaluation protocol, and documentation are part of the scientific contribution [10]–[12].

MIRA-Math follows this diagnostic tradition but uses controlled generation for a different purpose. Each instance is generated from a latent complete mathematical state that fixes a unique answer, while the solver’s private view omits a necessary atomic fact. This global-well-posed/local-underdetermined construction lets us evaluate not only whether the model eventually gets the answer right, but also whether it requested the correct missing information along the way.

2.3 Clarification and information acquisition↩︎

Clarification benchmarks study whether models ask useful questions when instructions, dialogue states, or user requests are ambiguous or underspecified [14]–[17]. This line of work is closely related in motivation in that practical assistants should not always guess when required information is absent. However, many clarification settings involve subjective ambiguity, natural human preferences, or dialogue-level uncertainty. These properties are valuable for realism but make it difficult to assign exact credit for a particular request.

MIRA-Math reduces this ambiguity by construction. The full latent state has a unique target answer, and the missing information is represented by typed atomic hints specified by the generator. The solver may still ask in free-form natural language, but the response channel is constrained. A fixed information-holder LLM sees only its private constraints and must choose between a structured offer and a structured decline. The accepted/declined events are therefore logged in an auditable schema, while the semantic matching decision itself remains LLM-mediated. The deterministic components are the generator invariants, canonical hint specification, instance validation, and final-answer verification.

2.4 Active and missing-information reasoning benchmarks↩︎

The closest recent work studies reasoning under incomplete information. QuestBench formalizes underspecified reasoning as missing variable assignments and evaluates whether models can identify the minimal necessary question in logic, planning, and math-derived tasks [18]. AR-Bench broadens the setting to active reasoning, where models must ask questions to acquire missing evidence in interactive environments [19]. CRITIC-math focuses directly on incomplete mathematical problems and evaluates whether large reasoning models proactively ask for information rather than hallucinating an answer [20].

MIRA-Math is closest to this family of benchmarks, but it makes a different design choice. QuestBench emphasizes selecting the right clarification from a finite set of options; MIRA-Math requires open-ended natural-language requests that must be precise enough to be accepted by a fixed constrained information-holder LLM. AR-Bench emphasizes broader active reasoning in interactive scenarios; MIRA-Math restricts the environment to a single controlled mathematical information channel. CRITIC-math studies whether models detect incompleteness in mathematical problems; MIRA-Math additionally evaluates whether the model identifies the exact missing atomic slot and integrates the returned value into an exact solution. These differences make MIRA-Math a narrower but more controlled diagnostic for minimal mathematical information requesting.

2.5 Agent, tool-use, and communication benchmarks↩︎

Broad agent benchmarks evaluate capabilities such as tool selection, API use, retrieval, web navigation, and long-horizon planning [4]–[6], [21]. LLM multi-agent frameworks and multi-agent reinforcement-learning work further study how agents coordinate and communicate under partial observability [22]–[28]. These settings are useful, but they conflate information acquisition with many other sources of difficulty.

MIRA-Math intentionally avoids this conflation. The second role is not an autonomous collaborator and does not solve, negotiate, explain, or strategically communicate. It is a fixed LLM-mediated information holder over dataset-provided atomic hints: the holder sees only its private constraints, must quote an explicit constraint when offering information, and otherwise returns a structured declination. This design makes MIRA-Math a benchmark of request precision and mathematical integration, not a benchmark of general multi-agent collaboration.

2.6 Data-centric benchmark design↩︎

Finally, MIRA-Math is a data-centric benchmark contribution. Work on dataset documentation, model cards, and responsible benchmark design emphasizes the need to specify intended uses, limitations, quality-control procedures, and maintenance plans [7], [8], [29], [30]. Following this view, the MIRA-Math release is intended to include generated JSONL files, family generators, exact verifiers, canonical hint specifications, prompt templates, run metadata, raw logs, and dataset documentation. The benchmark’s contribution is therefore not only a collection of problems, but a reproducible evaluation protocol for a specific capability: asking for the minimal missing mathematical fact needed to solve an underdetermined problem.

3 Benchmark Design↩︎

3.1 Task Formalization↩︎

A MIRA-Math instance is generated from a latent complete state \(z\). The full state determines a unique target answer \(y = f(z)\). The solver receives a private view \(v_A = \phi_A(z)\) that omits one necessary atomic fact. The hint source stores that atomic hint \(h \in H(z)\), each with a typed identifier such as a missing coefficient, boundary value, congruence, transition probability, or interpolation point. The benchmark enforces two conditions: \[\begin{align} \text{global well-posedness:} && \left|\{y: y \text{ is consistent with } v_A \cup H(z)\}\right| &= 1, \\ \text{local underdetermination:} && \left|\{y: y \text{ is consistent with } v_A\}\right| &> 1. \end{align}\]

3.2 Interaction Protocol↩︎

At evaluation time the solver is shown its private view, the target question, and the request budget \(B\). At each turn the solver may either issue a request or provide a final answer. Requests are free-form natural language. A fixed information-holder model receives its own private view containing the missing atomic fact and must respond using a structured schema. If the request semantically matches the fact it holds, the responder returns that fact; otherwise, it returns a declination and the solver may try again until the request budget is exhausted.

For example, in a recurrence instance the solver may see the recurrence rule and \(a(0)\) but not \(a(1)\). A request such as “What is the missing value of \(a(1)\)?” maps to the canonical initial-condition hint and receives the stored value. A vague request such as “Can you help me solve the recurrence?” is declined because it does not identify an allowed atomic hint. In a variable-slot family, such as a linear system with one missing coefficient, the request must identify the missing coefficient slot rather than merely ask for “the missing information.”

This constrained responder channel is a central design choice. In an unconstrained two-model dialogue, the responder could solve part of the problem, volunteer extra information, negotiate, or provide explanations. MIRA-Math removes these behaviors by giving the information holder only its private constraints and requiring a structured offer-or-decline response. The resulting request metrics should be interpreted as measuring the solver’s information-requesting behavior under a controlled LLM responder, not as a measure of general multi-agent teamwork or a purely deterministic request-matching oracle.

3.2.0.1 Responder implementation.

In the released reference runner, the information holder is implemented as a fixed LLM responder with a flat structured-output schema. Its only valid outputs are offer, with has_exact_match=true, a quoted private constraint, and a hint string; or decline, with a fixed declination message. If the structured output cannot be parsed, the runner attempts conservative raw-output extraction and otherwise records a declination. This design prevents the responder from solving the task, volunteering extra information, or giving partial clues, but it does not make semantic matching itself deterministic. Consequently, request metrics should be interpreted as solver performance under this fixed constrained responder channel. Exact mathematical correctness is still evaluated by deterministic family-specific verifiers.

3.3 Family Types↩︎

The 22 families are partitioned into two types. Type A families have a fixed hint slot: the missing information is structurally determined by the family. For example, a Bayes-rule instance always withholds the prior, and a recurrence instance always withholds one initial condition. Type B families have a variable hint slot: the missing position changes by instance, such as a missing transition probability, boundary value, coefficient, interpolation point, resistor, or grid cell. Type B therefore requires the solver to locate which slot is missing before phrasing the request.

This design supports diagnosis of two separate subskills. Type A primarily tests whether the model knows what kind of fact is needed and can compute after receiving it. Type B additionally tests whether the model can inspect its local view, identify the absent slot, and request that slot precisely.

3.4 Metrics↩︎

We report the following metrics.

• Accuracy: fraction of instances whose final answer normalizes to the ground truth under type-aware comparison.

• Request hit rate: mean per-instance ratio of accepted requests to fixed-responder messages, \(\text{offers} / (\text{offers} + \text{declines})\).

• First-request success: fraction of instances in which the solver’s first request receives a structured offer from the fixed responder.

• Average requests: mean number of requests the solver issues per instance.

• Average declinations: mean number of declinations from the fixed responder per instance.

• Average hints: mean number of accepted offers the solver receives per instance.

• Average tokens: approximate transcript cost, computed as mean transcript word-count × 1.3 per instance and summed across the solver and responder messages.

3.4.0.1 Trace-level decomposition.

To separate information-acquisition failures from downstream mathematical failures, we also classify each transcript using three binary indicators. Let \(O_i\) indicate that instance \(i\) received at least one structured offer; let \(C_i\) indicate that an offered hint matches the instance’s canonical atomic-hint specification under an offline audit of the quoted constraint and machine-readable hint fields; and let \(F_i\) indicate that the final answer is accepted by the exact verifier. We report: \[p_{\mathrm{no\text{-}canonical}}=\frac{1}{N}\sum_i (1-C_i), \qquad p_{\mathrm{integration\text{-}fail}}=\frac{1}{N}\sum_i C_i(1-F_i), \qquad p_{\mathrm{success}}=\frac{1}{N}\sum_i F_i.\]

The first quantity captures failures to acquire the canonical missing fact under the fixed responder channel. The second captures cases where the missing fact was acquired but the solver did not integrate it into a correct mathematical answer. The third is the usual final-answer accuracy. We separately log raw offers \(O_i\) so that any rare noncanonical offer can be distinguished from a genuine solver request success.

Accuracy measures mathematical completion, while request metrics measure information acquisition under the fixed responder channel. Their separation is important: a model may fail to elicit the canonical missing fact, may elicit it but still fail the mathematics, or may solve correctly after receiving it. The trace-level decomposition above makes these cases explicit rather than relying only on aggregate accuracy and request hit rate.

4 Dataset Construction↩︎

4.1 Current Release↩︎

The current MIRA-Math release is the 20/50 typed dataset: 20 instances per difficulty level for each Type A family and 50 instances per difficulty level for each Type B family. This yields 660 Type A instances, 1,650 Type B instances, and 2,310 instances in total. Each instance stores the family name, type, difficulty, random seed, solver prompt, target question, typed atomic-hint specification, canonical hint values, exact answer, and verifier metadata.

The 22 families (described in Appendix 12) cover Bayesian inference, Chinese remainder reconstruction, coordinate geometry, triangle path-sum inversion, linear systems with a separator variable, rank-one matrix completion, moment recovery, phase retrieval, piecewise functions, rank-deficient linear systems with a shared variable, recurrences, birth-death chains, series-parallel resistor networks, deconvolution, discrete tomography, eigenvector-constrained matrix questions, discrete Laplace equations, linear systems with a missing coefficient, Markov chains, polynomial interpolation, portfolio variance, and hidden Markov models.

4.2 Generator Invariants and Quality Control↩︎

Every generator enforces family-specific nondegeneracy conditions. Examples include rejecting singular linear systems, parallel lines, nonunique phase-retrieval instances, ambiguous tomography targets without a sufficient hint cell, and rational answers whose denominators exceed family-specific thresholds. All numeric answers are represented exactly as integers, tuples, or reduced rational values. Verifiers compare normalized exact values rather than surface strings.

For families where local underdetermination can be checked algebraically, the generator uses rank, uniqueness, or constraint-satisfaction tests. For finite discrete families, the generator can enumerate consistent completions. These checks are part of the dataset artifact, so users can regenerate the release, create larger splits, or audit any instance from its seed.

4.3 Data Format, Access, and Maintenance↩︎

The public release includes generated JSONL files, family generator code, exact verifiers, prompt templates, responder specifications, model-run scripts, and raw logs. The repository and dataset pages are:

The generated benchmark data are released under CC BY 4.0, and the generator, verifier, runner, and evaluation code are released under the MIT License.

The maintenance plan is to version releases by generator version, seed set, and prompt protocol. Future releases will preserve the current 20/50 typed split as a regression suite while adding larger held-out seed sets and optional multi-hint variants. Each release will include checksums, schema documentation, and a changelog documenting changes to generators, prompts, matching rules, and verifiers.

5 Experimental Protocol↩︎

We evaluate solver models in two prompt regimes: zero-shot (ZS) and four-shot (4S). The four-shot prompt provides examples of valid minimal requests and final-answer formatting without reusing test instances. In all reported experiments, the information-holder model is fixed to gpt-4o-mini; the solver model is the variable under evaluation. The responder receives only its private constraints and is prompted to return a flat structured output with one of two types: offer, when it judges the request to semantically match one of its private constraints, or decline, otherwise. A valid offer must include has_exact_match=true, the exact quoted private constraint, and the extracted hint string. A declination uses a fixed message and must not reveal what other information the responder holds. Thus the responder is fixed and constrained, but it is not a deterministic matching oracle.

The released runner stores the full transcript for each instance, including Agent A requests, Agent B offer/decline decisions, quoted constraints, extracted hints, parsing errors when present, final answers, and exact-verifier outcomes. This makes it possible to recompute aggregate metrics and the trace-level decomposition in Section 6 without rerunning model calls, provided the raw transcripts are retained.

The main results report six solver models. Exact decoding settings, model identifiers, timestamps, prompts, and raw transcripts are included with the release artifacts so that future users can reproduce the runs or update the baselines as model APIs and open-weight checkpoints change.

6 Results↩︎

6.0.0.1 Requesting and solving are distinct.

Figure 2 summarizes final-answer accuracy across solver models and prompting regimes, while Appendix 11.1 retains the full numeric request metrics. The aggregate results show that high request performance does not necessarily imply high final-answer accuracy. For example, in the zero-shot setting, gemini-2.5-flash has the highest request hit rate and first-request success among the completed models, but its final accuracy remains below gpt-5.1 and Gemma-4. Conversely, Gemma-4 obtains the highest zero-shot final accuracy while not achieving the highest request metrics. This validates the benchmark’s central diagnostic split: asking for the right fact and using it correctly are separable skills.

6.0.0.2 Trace snapshots make the diagnostic split concrete.

To make the aggregate metrics easier to interpret, Appendix 13 gives short, offline-audited transcript snapshots for three representative outcomes: no canonical hint acquired, canonical hint acquired but final answer wrong, and full request–resolve–solve success. These snapshots are not used to estimate prevalence and should not be read as representative samples. Instead, they show what the logged metrics mean at the transcript level. For example, a no-hint trace shows a solver cycling through plausible but wrong missing rates until the request budget is exhausted; a hint-but-wrong trace shows the solver receiving the prior needed for Bayes’ rule but still submitting an incorrect posterior; and a full-success trace shows the solver requesting the missing resistor value and then computing the equivalent resistance exactly. Table 1 summarizes the examples and their diagnostic role.

3pt

6.0.0.3 Request acquisition versus mathematical integration.

The transcript decomposition confirms that final accuracy should not be read as a pure requesting score. In the reference protocol, an instance can fail because the solver never obtains the canonical atomic hint, or because it obtains the hint but performs the subsequent computation incorrectly. We therefore report the disjoint trace-level categories in Table 2: no canonical hint acquired, canonical hint acquired but final answer wrong, and final answer correct. This decomposition is especially important for families such as Chinese-remainder reconstruction, phase retrieval, polynomial interpolation, and portfolio variance, where models can often obtain the relevant hint but still fail exact arithmetic, symbolic reconstruction, or answer normalization.

3pt

6.0.0.4 Variable-slot families are not uniformly harder.

Figure 3 shows that Type B families, despite requiring the solver to identify which slot is absent, are not uniformly harder in final-answer accuracy. Strong models often score higher on Type B than on Type A. This is not a contradiction: Type A contains several families where the request is easy but the downstream computation is brittle, such as exact Chinese-remainder reconstruction, path-sum inversion, moment recovery, and phase retrieval. The Type A/Type B distinction should therefore be interpreted as a missing-slot diagnostic, not as a total difficulty ranking. Appendix 11.1 gives the corresponding request-quality metrics, including hit rate, first-request success, and average number of requests by family type.

6.0.0.5 Difficulty interacts with family structure.

Figure 4 shows that smaller models can degrade sharply with difficulty, while stronger models are less monotonic. This reflects the benchmark design: difficulty scaling may increase coefficient ranges, dimensions, denominators, or target-index depth depending on the family. Per-family and per-type reporting remain essential, because a single aggregate score can hide whether a model fails at requesting, exact arithmetic, linear algebra, combinatorial search, or answer normalization. The full difficulty-by-type breakdown is provided in Appendix 11.1.

6.0.0.6 Four-shot prompting is not uniformly beneficial.

Figures 2–4 show that the tested four-shot prompt does not uniformly improve final-answer accuracy. The Type A/Type B split helps interpret this pattern. Demonstrations can help when the missing slot is structurally stable: in Type A families, the solver can often reuse a demonstrated pattern such as asking for a prior, initial condition, threshold, or missing equation. By contrast, Type B families require instance-level slot localization: the solver must inspect the current problem and identify which coefficient, transition, boundary value, grid cell, resistor, interpolation point, or emission entry is absent. A small four-shot prompt necessarily covers only a few request archetypes, and may therefore induce exemplar anchoring: the solver may imitate the demonstrated request forms instead of adapting to the specific missing slot in the current instance. We treat this as a prompt-sensitivity hypothesis rather than a causal claim. The main result is that demonstrations of the interaction protocol do not automatically improve minimal-information reasoning; in some families they improve first-request compliance while leaving mathematical integration unchanged or worse. Appendix 13 includes an illustrative prompt-anchoring snapshot (T4), and Appendix 11.1 gives the full per-family four-shot deltas.

Taken together, these results show that MIRA-Math should be read less as a single leaderboard and more as a diagnostic decomposition. The same aggregate accuracy can arise from different mechanisms: failure to localize the missing slot, successful acquisition followed by incorrect computation, or correct end-to-end request and solution. The trace-level logs, per-family metrics, and Type A/Type B split are therefore intended to be used jointly when comparing models or prompt strategies.

7 Failure Modes and Diagnostic Use↩︎

The benchmark exposes at least two recurring failure modes. Request-acquisition failures occur when the solver does not obtain the canonical missing fact under the fixed responder channel. These can happen because the solver asks for a wrong slot, asks too vaguely, or repeatedly probes quantities that are already present in its private view. Integration failures occur when the solver receives the canonical hint but still computes, formats, or normalizes the final answer incorrectly.

Several families make this distinction visible. In fixed-slot families such as bayes_missing _prior, the request can be easy while exact arithmetic remains brittle. In variable-slot families such as birth_death_missing_rate, the solver must first localize the absent rate before it can apply a standard computation. Other families, such as circuit_missing_resistance, show the intended successful pattern: identify the missing slot, request it precisely, and integrate the returned value into the exact verifier-compatible answer.

Appendix 13 gives compact transcript snapshots for these cases. The examples are illustrative rather than representative; aggregate claims are based on the tables above. Their purpose is to show how the same final incorrect score can correspond to different underlying failures: no canonical hint acquired, canonical hint acquired but wrong downstream computation, or successful request and solution. This is why the release includes both exact verifiers and raw interaction logs.

8 Intended Uses and Limitations↩︎

8.0.0.1 Intended uses.

MIRA-Math is intended for evaluating models, prompts, and solver loops that must operate under partial mathematical information. Appropriate uses include measuring request precision, comparing communication-efficiency strategies, auditing whether a model can identify missing facts, and studying the relationship between clarification success and final mathematical accuracy.

8.0.0.2 Out-of-scope uses.

The benchmark should not be used as evidence that a model can collaborate with another autonomous agent. The second role is a fixed, constrained information holder, not a teammate. The benchmark should also not be treated as a general measure of tool use, web navigation, social reasoning, or real-world decision-making.

8.0.0.3 Limitations.

The tasks are synthetic and mathematically structured. This improves controllability and exact verification but limits realism. The current release uses one required atomic hint per instance; many real tasks require multiple facts, or uncertain evidence. The typed hint schema and constrained responder protocol is useful for reproducibility but may understate the difficulty of communicating with humans or open-ended systems. The responder channel is controlled but not deterministic: in the reported experiments, a fixed LLM information holder performs the semantic offer-or-decline decision under structured-output constraints. This means request metrics can still reflect occasional responder matching errors, although the final-answer verifier and dataset validity checks are deterministic. Finally, model results can become stale as APIs and open-weight checkpoints change; the release is designed to support re-running baselines rather than treating the reported ranking as permanent.

9 Conclusion↩︎

MIRA-Math introduces a controlled benchmark for minimal information requesting in mathematical reasoning. Each instance is generated from a complete latent mathematical state with a unique answer, but the solver receives only a locally underdetermined view and must request exactly the missing atomic fact needed to complete the solution. This construction turns information acquisition into an explicit, measurable step rather than leaving it implicit inside final-answer accuracy.

The results show that this step is not redundant with ordinary mathematical solving. Across the tested models, request hit rate, first-request success, and final-answer accuracy diverge substantially: some models obtain the canonical hint but fail exact downstream computation, while others fail before acquiring the resolving fact. The Type A/Type B split and transcript-level decomposition further show that fixed missing slots, variable missing slots, arithmetic integration, and answer normalization can fail independently.

The benchmark is intentionally narrow. It does not claim to measure open-ended collaboration, tool use, or human clarification behavior. Instead, it provides a reproducible diagnostic setting with typed atomic hints, a fixed constrained LLM responder, deterministic validation, exact family-specific verifiers, and released transcripts. We hope this makes MIRA-Math useful both as a baseline suite for future models and as an analysis tool for studying when language models ask for missing mathematical information rather than guessing.

Dataset Certification↩︎

MIRA-Math is a synthetically generated benchmark. Each instance is produced from a recorded generator family, difficulty level, random seed, typed atomic-hint specification, exact answer, and verifier metadata. The release includes generated data, generator code, exact verifiers, prompt templates, responder specifications, model-run scripts, raw interaction logs, and documentation. The dataset contains mathematical problem text and synthetic numeric values rather than personal data. Intended uses and out-of-scope uses are described in Section 8.

10 Implementation and Reproducibility Details↩︎

10.0.0.1 Released artifacts.

The release contains the family generators, instance validator, exact family-specific solvers, prompt templates, structured-output schemas, reference LangGraph runner, scoring code, generated JSONL files, run metadata, and raw transcripts. Each generated instance records the family name, difficulty, random seed, agent-private constraints, machine-readable canonical atomic-hint specification, global solution, answer type, and verifier metadata.

10.0.0.2 Instance validation.

The validator checks three properties for each instance: global uniqueness of the full latent problem, local underdetermination of each agent’s private view, and sufficiency of the canonical atomic hint when applied to the solver’s view. These checks are family-specific and deterministic. The final-answer verifier also uses normalized exact values rather than surface-form string matching.

10.0.0.3 Responder protocol.

The reference responder is a fixed information-holder LLM. It receives only its private constraints and Agent A’s current request. It must emit a flat structured object with either offer or decline. An offer must set has_exact_match=true, quote the exact private constraint, and provide the extracted hint. A decline must not reveal alternative information. If structured parsing fails, the runner attempts conservative raw-output extraction; if that also fails, the response is recorded as a declination. Therefore, semantic matching is controlled and auditable but LLM-mediated.

10.0.0.4 Transcript scoring.

For each transcript, the scorer records final-answer accuracy, request attempts, offer count, declination count, first-request success, requests before first offer, token estimate, and exact expected/actual normalized answers. We additionally compute the trace-level indicators used in Table 2: whether a canonical hint was acquired, whether the solver produced a correct final answer after acquiring it, and whether a noncanonical offer occurred.

10.0.0.5 Versioning.

To make future reruns comparable, each release records the generator version, seed set, prompt-protocol version, responder model identifier, solver model identifier, decoding settings, code commit, dataset checksum, and transcript checksum. Model rankings should be interpreted as a snapshot under these settings because API-backed model behavior can change over time.

10.0.0.6 Exact run configuration.

Tables 3 and ¿tbl:tab:repro95models? record the concrete settings used for all reported runs so that the main reproducibility facts are available without consulting the release artifacts. All runs use greedy decoding (temperature \(=0\)) for both agents, a single pass per instance, and a fixed request seed of \(1234\); residual provider-side nondeterminism may remain where the backend does not honor the seed. Models are referenced by provider alias on the dates shown; the underlying API snapshots and open-weight checkpoints are not version-pinned and may change.

max width=

11 Additional Results and Full Numeric Tables↩︎

11.1 Full numeric tables for main-text figures↩︎

Tables ¿tbl:tab:full95overall95metrics?–¿tbl:tab:full95type95request95quality? provide the full numeric values underlying Figures 2–4. The main text visualizes the central accuracy patterns for readability, while these appendix tables retain the complete request-quality and token-usage metrics.

max width=

max width=

max width=

max width=

max width=

11.2 Per-family and prompt-effect results↩︎

The following tables provide additional per-family result summaries and four-shot effect analyses for each reported solver model and prompt regime.

max width=

max width=

max width=

max width=

max width=

max width=

max width=

max width=

max width=

max width=

max width=

max width=

max width=

max width=

max width=

max width=

max width=

max width=

max width=

max width=

max width=

max width=

max width=

max width=

12 Problem Family Generators↩︎

Each family below defines a parameterised class of two-agent reasoning problems. A generator samples an instance at random given a difficulty level \(d \in \{1,2,3\}\) and a random seed. For each instance, Agent A and Agent B each receive a disjoint subset of the constraints; neither subset alone determines the answer, but together they do, and exactly one atomic hint from B to A resolves A’s ambiguity.

The 22 families are split into two types:

• Type A — Fixed hint slot (11 families). The missing information is structurally determined by the family: it is always the same kind/position regardless of the specific instance. Agent A can predict what to request without scanning its constraints.

• Type B — Variable hint slot (11 families). The missing slot varies per instance — it could be any one of multiple possible positions. Agent A must first identify which slot is absent before formulating its request.

---

---

A1.Bayesian Inference with Missing Prior (bayes_missing_prior)↩︎

12.0.0.1 Setup.

Three rational parameters are sampled: \[P(H) = \tfrac{p_n}{p_d},\quad P(E \mid H) = \tfrac{s_n}{s_d},\quad P(E \mid \neg H) = \tfrac{f_n}{f_d},\] with denominators drawn from \(\mathrm{Unif}[2,\,3+2d]\) and numerators chosen so all three lie strictly in \((0,1)\) and \(P(E\mid H) \ne P(E\mid \neg H)\). The posterior is computed via Bayes’ rule: \[P(H \mid E) \;=\; \frac{P(E \mid H)\,P(H)}{P(E \mid H)\,P(H) \;+\; P(E \mid \neg H)\,(1 - P(H))}.\] Instances where the posterior denominator exceeds 200 are rejected.

12.0.0.2 Information split.

• Agent A: the likelihood \(P(E \mid H)\) and the false-positive rate \(P(E \mid \neg H)\).

• Agent B: the prior \(P(H)\).

B’s atomic hint is always the prior \(P(H)\).

12.0.0.3 Answer.

The posterior \(P(H \mid E)\) as a reduced fraction.

12.0.0.4 Difficulty scaling (\(d \in \{1,2,3\}\)).

Fraction denominators drawn from \([2,\, 3+2d]\) (max 5, 7, 9), controlling numerical complexity.

A2.Chinese Remainder Theorem Reconstruction (crt_reconstruction)↩︎

12.0.0.5 Setup.

Three distinct primes \((m_1, m_2, m_3)\) are drawn uniformly at random (without replacement) from the pool \(\{3, 5, 7, 11, 13, 17, 19\}\). An integer \(x \sim \mathrm{Unif}[0, M)\) is sampled, where \(M = m_1 m_2 m_3\), and the three residues \(r_i = x \bmod m_i\) are computed. By the Chinese Remainder Theorem the solution in \([0,M)\) is unique.

12.0.0.6 Information split.

• Agent A: congruences \(x \equiv r_1 \pmod{m_1}\) and \(x \equiv r_2 \pmod{m_2}\).

• Agent B: congruence \(x \equiv r_3 \pmod{m_3}\).

B’s atomic hint is always the third congruence \((m_3, r_3)\).

12.0.0.7 Answer.

The integer \(x\).

12.0.0.8 Difficulty scaling.

This family does not scale with difficulty: the prime pool and problem structure are identical for all \(d\). Accordingly, in mix generation it is always run at \(d = 1\) regardless of the requested difficulty.

A3.Coordinate Geometry with Missing Line (geometry_coordinates)↩︎

12.0.0.9 Setup.

An intersection point \((x, y)\) is drawn uniformly from \([-(4+d),\,4+d]^2\). Two non-parallel lines through \((x,y)\) are generated with integer coefficients in \([-(3+d),\,3+d]\): \[a_1 x + b_1 y = c_1, \qquad a_2 x + b_2 y = c_2,\] subject to \(a_1 b_2 - a_2 b_1 \ne 0\) (non-parallel). A reference point \(Q = (x + \Delta x,\; y + \Delta y)\) is placed at an integer Euclidean distance from \((x,y)\) using a Pythagorean pair \((\Delta x, \Delta y) \in \{(3,4),(4,3),(5,12),(12,5)\}\). The target is \(d((x,y), Q) = \sqrt{\Delta x^2 + \Delta y^2}\), guaranteed to be an integer.

12.0.0.10 Information split.

• Agent A: the first line equation and the coordinates of \(Q\).

• Agent B: the second line equation and the coordinates of \(Q\).

B’s atomic hint is always the second line equation.

12.0.0.11 Answer.

The integer distance \(d((x,y), Q)\).

12.0.0.12 Difficulty scaling (\(d \in \{1,2,3\}\)).

• Intersection point drawn from \([-(4+d),\,4+d]^2\) (range \(\pm\)​5, \(\pm\)​6, \(\pm\)​7).

• Line coefficients drawn from \([-(3+d),\,3+d]\) (range \(\pm\)​4, \(\pm\)​5, \(\pm\)​6).

A4.Triangle Path Sums (graph_path_sums)↩︎

12.0.0.13 Setup.

A triangle graph has three edges with positive integer weights \(e_{01}, e_{12}, e_{20} \sim \mathrm{Unif}[1,\,2+d]\). The three path sums (one per pair of edges sharing a vertex) are: \[s_1 = e_{01} + e_{12}, \qquad s_2 = e_{12} + e_{20}, \qquad s_3 = e_{01} + e_{20}.\] Any two path sums leave one degree of freedom, while all three path sums determine the three edge weights uniquely.

12.0.0.14 Information split.

• Agent A: path sums \(s_1\) and \(s_2\).

• Agent B: path sum \(s_3 = e_{01} + e_{20}\).

B’s atomic hint is always the equation \(e_{01} + e_{20} = s_3\).

12.0.0.15 Answer.

The triple \((e_{01}, e_{12}, e_{20})\).

12.0.0.16 Difficulty scaling (\(d \in \{1,2,3\}\)).

Edge weights drawn from \([1,\, 2+d]\) (max 3, 4, 5).

A5.Linear System with Separator Variable (linear_system_separator)↩︎

12.0.0.17 Setup.

Unknowns \((a, b, c)\) are drawn uniformly from \([-(5+d),\,5+d]^3\). Two linearly independent equations in \((a, b, c)\) are sampled with integer coefficients in \([-(3+d),\,3+d]\), subject to the constraint that, when \(c\) is substituted, the residual \(2 \times 2\) system in \((a, b)\) is uniquely solvable (i.e.the \(2 \times 2\) submatrix on columns \(a, b\) has rank 2).

12.0.0.18 Information split.

• Agent A: the two linear equations.

• Agent B: the exact value of \(c\).

B’s atomic hint is always \(c = \text{value}\).

12.0.0.19 Answer.

The triple \((a, b, c)\).

12.0.0.20 Difficulty scaling (\(d \in \{1,2,3\}\)).

• Solution drawn from \([-(5+d),\, 5+d]^3\) (range \(\pm\)​6, \(\pm\)​7, \(\pm\)​8).

• Coefficients drawn from \([-(3+d),\, 3+d]\) (range \(\pm\)​4, \(\pm\)​5, \(\pm\)​6).

A6.Rank-1 Matrix Completion (matrix_completion)↩︎

12.0.0.21 Setup.

A \(2 \times 2\) rank-1 matrix \(M = \mathbf{u}\mathbf{v}^\top\) is constructed from vectors \(\mathbf{u} = (u_0, u_1)^\top\) and \(\mathbf{v} = (v_0, v_1)^\top\) with \(u_i, v_i \sim \mathrm{Unif}[1,\,2+d]\): \[M = \begin{pmatrix} u_0 v_0 & u_0 v_1 \\ u_1 v_0 & u_1 v_1 \end{pmatrix} = \begin{pmatrix} a & b \\ c & d \end{pmatrix}.\] By the rank-1 identity, \(m_{11} = m_{01}\,m_{10} / m_{00}\) (valid since \(m_{00} = u_0 v_0 \ge 1\)).

12.0.0.22 Information split.

• Agent A: entries \(m_{00} = a\) and \(m_{01} = b\) (first row).

• Agent B: entry \(m_{10} = c\) (position \([1,0]\)).

B’s atomic hint is always \(m_{10} = c\).

12.0.0.23 Answer.

The integer \(m_{11} = bc/a\).

12.0.0.24 Difficulty scaling (\(d \in \{1,2,3\}\)).

Factor entries \(u_i, v_i\) drawn from \([1,\, 2+d]\) (max 3, 4, 5).

A7.Moment Problem on Three-Point Support (moment_problem)↩︎

12.0.0.25 Setup.

A discrete probability distribution on support \(\{0, 1, 2\}\) is specified by unnormalised weights \((q_0, q_1, q_2)\) summing to a common denominator \(D = 6 + 2d\). Three linear constraints determine the distribution uniquely: \[\text{(sum)}\;\; q_0 + q_1 + q_2 = D, \qquad \text{(1st moment)}\;\; q_1 + 2q_2 = m_1, \qquad \text{(2nd moment)}\;\; q_1 + 4q_2 = m_2.\] Values \(q_0, q_1 \sim \mathrm{Unif}[1, D-2]\) are drawn subject to \(q_2 = D - q_0 - q_1 > 0\), and \(m_1, m_2\) are derived accordingly.

12.0.0.26 Information split.

• Agent A: the sum constraint and the first-moment equation \(q_1 + 2q_2 = m_1\).

• Agent B: the sum constraint and the second-moment equation \(q_1 + 4q_2 = m_2\).

Each two-equation subsystem has rank 2 in three unknowns (ill-posed). B’s atomic hint is always the second-moment equation.

12.0.0.27 Answer.

The triple \((q_0, q_1, q_2)\).

12.0.0.28 Difficulty scaling (\(d \in \{1,2,3\}\)).

Common denominator \(D = 6+2d\) (values 8, 10, 12), controlling the magnitude of the weights and moments.

A8.Phase Retrieval via DFT Magnitudes (phase_retrieval)↩︎

12.0.0.29 Setup.

A signal \(\mathbf{x} = (x_0, x_1, x_2, x_3) \in \mathbb{Z}^4\) is drawn with \(x_0 \ne 0\) and \(|x_i| \le 2 + \min(d,2)\). The magnitude-squared spectrum is computed using closed-form expressions for a length-4 real DFT: \[\begin{align} |\hat{x}_0|^2 &= (x_0+x_1+x_2+x_3)^2, \\ |\hat{x}_1|^2 &= (x_0-x_2)^2 + (x_3-x_1)^2, \\ |\hat{x}_2|^2 &= (x_0-x_1+x_2-x_3)^2, \\ |\hat{x}_3|^2 &= (x_0-x_2)^2 + (x_1-x_3)^2. \end{align}\] Note that \(|\hat{x}_1|^2 = |\hat{x}_3|^2\) in this formulation. By rejection sampling, instances are kept only when (i) the magnitude vector alone admits multiple solutions (i.e.A is locally ill-posed), and (ii) the magnitude vector together with \(\mathrm{sign}(x_0)\) admits exactly one solution.

12.0.0.30 Information split.

• Agent A: the four magnitude-squared values \((|\hat{x}_0|^2, |\hat{x}_1|^2, |\hat{x}_2|^2, |\hat{x}_3|^2)\).

• Agent B: the sign bit \(\mathrm{sign}(x_0) \in \{-1, +1\}\).

B’s atomic hint is always \(\mathrm{sign}(x_0)\).

12.0.0.31 Answer.

The full signal \((x_0, x_1, x_2, x_3)\).

12.0.0.32 Difficulty scaling (\(d \in \{1,2,3\}\)).

Signal value range \(|x_i| \le 2+\min(d,2)\) (bounds 3, 4, 4 — caps at \(d = 2\), so \(d = 3\) is identical to \(d = 2\)).

A9.Piecewise Linear Function with Missing Threshold (piecewise_missing_threshold)↩︎

12.0.0.33 Setup.

A two-branch piecewise linear function is defined: \[f(x) = \begin{cases} a_0\,x + b_0 & \text{if } x < t, \\ a_1\,x + b_1 & \text{if } x \ge t, \end{cases}\] with slopes \(a_i \sim \mathrm{Unif}[-(2+d),\,2+d]\), intercepts \(b_i \sim \mathrm{Unif}[-(4+2d),\,4+2d]\), query point \(x \sim \mathrm{Unif}[-(5+3d),\,5+3d]\), and threshold \(t\) drawn uniformly from a symmetric interval, with \(x \ne t\) enforced and the two branch values at \(x\) required to be distinct (so the active branch is not degenerate).

12.0.0.34 Information split.

• Agent A: the query point \(x\) and both branch formulas, but not \(t\).

• Agent B: the query point \(x\) and the threshold \(t\).

Without \(t\), A cannot determine which branch is active. B’s atomic hint is always \(t\).

12.0.0.35 Answer.

The integer \(f(x)\).

12.0.0.36 Difficulty scaling (\(d \in \{1,2,3\}\)).

• Slope range \(\pm(2+d)\) (max \(\pm\)​3, \(\pm\)​4, \(\pm\)​5).

• Query and threshold range \(\pm(5+3d)\) (max \(\pm\)​8, \(\pm\)​11, \(\pm\)​14).

A10.Rank-Deficient Linear System, Shared Variable (rankdef_linear_shared)↩︎

12.0.0.37 Setup.

A solution \((x, y, z) \in \mathbb{Z}^3\) is drawn uniformly from \([-(5+d),\,5+d]^3\). Four integer-coefficient equations are generated: rows 1 and 2 form a rank-2 subsystem; rows 3 and 4 are each independently sufficient to complete the system to rank 3. Formally, all four rows have coefficients in \([-(3+d),\,3+d] \setminus \{0\}\) and are generated by rejection to satisfy: \[\mathrm{rank}(\text{rows }1,2) = 2, \quad \mathrm{rank}(\text{rows }1,2,3) = 3, \quad \mathrm{rank}(\text{rows }1,2,4) = 3.\]

12.0.0.38 Information split.

• Agent A: equations 1 and 2 (rank 2 in 3 unknowns, ill-posed).

• Agent B: equations 3 and 4 (either one suffices for A).

B’s atomic hint is always equation 3.

12.0.0.39 Answer.

The triple \((x, y, z)\).

12.0.0.40 Difficulty scaling (\(d \in \{1,2,3\}\)).

• Solution drawn from \([-(5+d),\, 5+d]^3\) (range \(\pm\)​6, \(\pm\)​7, \(\pm\)​8).

• Coefficients drawn from \([-(3+d),\, 3+d] \setminus \{0\}\) (range \(\pm\)​4, \(\pm\)​5, \(\pm\)​6).

A11.Recurrence Missing Initial Condition (recurrence_missing_init)↩︎

12.0.0.41 Setup.

A second-order linear recurrence over the integers: \[a(n+2) \;=\; r_1\,a(n+1) \;+\; r_2\,a(n), \qquad n \ge 0.\] Parameters are sampled as \(r_1 \sim \mathrm{Unif}[1,\,3+d]\), \(r_2 \sim \mathrm{Unif}[-2,2]\), and initial conditions \(a(0), a(1) \sim \mathrm{Unif}[-3-d,\,3+d]\). A target index \(n^* \sim \mathrm{Unif}[4,\,7+d]\) is drawn, and the target value \(a(n^*)\) is computed by forward iteration.

12.0.0.42 Information split.

• Agent A: the recurrence coefficients \(r_1, r_2\) and the initial condition \(a(0)\).

• Agent B: the recurrence coefficients \(r_1, r_2\) and the initial condition \(a(1)\).

Neither agent can compute \(a(n^*)\) alone. B’s atomic hint is always \(a(1)\).

12.0.0.43 Answer.

The integer \(a(n^*)\).

12.0.0.44 Difficulty scaling (\(d \in \{1,2,3\}\)).

• Recurrence coefficient \(r_1\) drawn from \([1,\, 3+d]\) (max 4, 5, 6).

• Initial conditions and coefficients drawn from \([-3-d,\, 3+d]\) (range \(\pm\)​4, \(\pm\)​5, \(\pm\)​6).

• Target index \(n^*\) drawn from \([4,\, 7+d]\) (max 8, 9, 10).

---

---

B1.Birth-Death Chain with Missing Rate (birth_death_missing_rate)↩︎

12.0.0.45 Setup.

A birth-death Markov chain on states \(\{0, 1, \ldots, n-1\}\) with \(n = d + 2\) is defined by birth rates \(\lambda_i\) (transitions \(i \to i+1\) for \(i = 0, \ldots, n-2\)) and death rates \(\mu_i\) (transitions \(i \to i-1\) for \(i = 1, \ldots, n-1\)), all drawn from \(\mathrm{Unif}[1,\,3+2d]\).

The unique stationary distribution satisfies detailed balance: \[\pi_i \;=\; \pi_0 \cdot \prod_{k=0}^{i-1} \frac{\lambda_k}{\mu_{k+1}}, \qquad i = 1, \ldots, n-1,\] normalised so \(\sum_{i} \pi_i = 1\). One rate—either some \(\lambda_i\) or some \(\mu_i\)—is chosen uniformly at random to be hidden. A target state \(s^* \sim \mathrm{Unif}[0, n-1]\) is drawn. Instances where \(\pi_{s^*}\) has denominator exceeding 300 are rejected.

12.0.0.46 Information split.

• Agent A: all birth and death rates except the one missing rate, in natural-language form.

• Agent B: the single missing rate (which rate it is varies per instance).

12.0.0.47 Answer.

The stationary probability \(\pi_{s^*}\) as a reduced fraction.

12.0.0.48 Difficulty scaling (\(d \in \{1,2,3\}\)).

• Number of states \(n = d+2\) (3, 4, 5 states).

• Rates drawn from \([1,\, 3+2d]\) (max 5, 7, 9).

B2.Series-Parallel Resistor Network (circuit_missing_resistance)↩︎

12.0.0.49 Setup.

A random binary tree with \(n = 2d + 1\) leaves is built by recursive splitting. Each leaf is a named resistor \(R_1, \ldots, R_n\) with a positive integer resistance drawn from \(\mathrm{Unif}[1,\,5+3(d-1)]\). Each internal node is independently labelled series or parallel with equal probability. The equivalent resistance is evaluated recursively: \[R_{\text{series}} = R_L + R_R, \qquad R_{\text{parallel}} = \frac{R_L\,R_R}{R_L + R_R}.\] One resistor \(R_k\) (chosen uniformly) is withheld. The circuit topology is described to both agents in natural language.

12.0.0.50 Information split.

• Agent A: the full circuit topology and all resistance values except \(R_k\) (which resistor varies per instance).

• Agent B: the value of \(R_k\).

12.0.0.51 Answer.

The equivalent resistance \(R_{\text{eq}}\) as a reduced fraction.

12.0.0.52 Difficulty scaling (\(d \in \{1,2,3\}\)).

• Number of resistors \(n = 2d+1\) (3, 5, 7).

• Resistance values drawn from \([1,\, 5+3(d-1)]\) (max 5, 8, 11).

B3.Deconvolution with Missing Measurement (deconvolution)↩︎

12.0.0.53 Setup.

A signal \(\mathbf{x} \in \mathbb{Z}^n\) and a kernel \(\mathbf{h} \in \mathbb{Z}^m\) are sampled with entries in \([-(2+d),\,2+d]\) and \(h_0 \ne 0\). The full output vector \(\mathbf{y} = \mathbf{x} * \mathbf{h} \in \mathbb{Z}^{n+m-1}\) is computed via linear convolution: \[y_k = \sum_{i=0}^{m-1} h_i\, x_{k-i}, \qquad k = 0, \ldots, n+m-2.\] The \(n+m-1\) equations in \(\mathbf{x}\) form an overdetermined linear system. By rejection, an index \(k^*\) is found such that removing \(y_{k^*}\) makes the system rank-deficient, while the full system uniquely determines \(\mathbf{x}\).

12.0.0.54 Information split.

• Agent A: the kernel \(\mathbf{h}\) and all output values \(y_k\) except \(y_{k^*}\) (which index varies per instance).

• Agent B: the missing output value \(y_{k^*}\).

12.0.0.55 Answer.

The signal \(\mathbf{x} = (x_0, \ldots, x_{n-1})\).

12.0.0.56 Difficulty scaling (\(d \in \{1,2,3\}\)).

• Signal length \(n = 3+\min(d,2)\) (4, 5, 5 — caps at \(d=2\)).

• Kernel length \(m = 2\) for \(d \le 1\), else \(m = 3\).

• Entry magnitudes bounded by \(2+d\) (max 3, 4, 5).

B4.Discrete Tomography with Ambiguous Cell (discrete_tomography)↩︎

12.0.0.57 Setup.

A \(3 \times 3\) binary grid \(G \in \{0,1\}^{3 \times 3}\) is sampled uniformly at random. Its row sums \(r_i = \sum_j G_{ij}\) and column sums \(c_j = \sum_i G_{ij}\) are computed. The row and column sums alone are typically consistent with multiple binary grids. A target cell \((i_t, j_t)\) is chosen from those that are ambiguous (i.e.their value differs across consistent completions). A hint cell \((i_h, j_h) \ne (i_t, j_t)\) is found by search: revealing its value must reduce the consistent completions to exactly one.

12.0.0.58 Information split.

• Agent A: all three row sums \(r_0, r_1, r_2\) and all three column sums \(c_0, c_1, c_2\).

• Agent B: the value of the hint cell \(G_{i_h j_h}\) (which cell varies per instance).

12.0.0.59 Answer.

The binary value \(G_{i_t j_t} \in \{0,1\}\).

12.0.0.60 Difficulty scaling.

This family does not scale with difficulty: the grid is always \(3 \times 3\) and the generation logic is identical for all \(d\). Accordingly, in mix generation it is always run at \(d = 1\) regardless of the requested difficulty.

B5.Matrix Entry Hidden from Eigenvector (eigenvector_missing_entry)↩︎

12.0.0.61 Setup.

An \(n \times n\) integer matrix \(M\) (with \(n = d + 1\)) is constructed such that a known integer vector \(\mathbf{v}\) is an eigenvector with eigenvalue \(\lambda\): \(M\mathbf{v} = \lambda\mathbf{v}\).

The key construction: one component \(v_{j^*} = 0\), so the eigenvector equation for row \(i^*\), \[\sum_{c=0}^{n-1} M_{i^* c}\, v_c = \lambda\, v_{i^*},\] contains no term involving \(M_{i^* j^*}\) (that term vanishes). Therefore \(M_{i^* j^*}\) is unconstrained by the eigenvector equations. A target column \(t^*\) is drawn ensuring \(M_{j^* t^*} \ne 0\), so that A cannot compute \((M^2)_{i^* t^*} = \sum_k M_{i^* k} M_{k t^*}\) without knowing \(M_{i^* j^*}\).

12.0.0.62 Information split.

• Agent A: all entries of \(M\) except \(M_{i^* j^*}\), and all components of \(\mathbf{v}\) (which entry is missing varies per instance).

• Agent B: the entry \(M_{i^* j^*}\).

12.0.0.63 Answer.

The integer \((M^2)_{i^* t^*}\).

12.0.0.64 Difficulty scaling (\(d \in \{1,2,3\}\)).

• Matrix dimension \(n = d+1\) (\(2 \times 2\), \(3 \times 3\), \(4 \times 4\)).

• Entry magnitudes bounded by \(3+2(d-1)\) (max 3, 5, 7).

• Eigenvalue \(\lambda\) drawn from \([1,\, 3+2(d-1)]\) (max 3, 5, 7).

B6.Discrete Laplace Equation on a Grid (laplace_grid)↩︎

12.0.0.65 Setup.

An \(n \times n\) grid has unknown values \(u_{ij}\) on every cell. Interior cells satisfy the discrete Laplace equation: \[4\,u_{ij} \;=\; u_{i-1,j} + u_{i+1,j} + u_{i,j-1} + u_{i,j+1}, \qquad (i,j) \text{ interior.}\] Boundary values are drawn as integer multiples of a scaling factor (24 for \(n=4\), 224 for \(n=5\)) chosen to ensure integer interior solutions. One boundary cell \((i^*, j^*)\) is chosen uniformly at random and withheld; it is verified that A’s system is rank-deficient without this value.

12.0.0.66 Information split.

• Agent A: all discrete Laplace equations and all boundary values except \(u_{i^* j^*}\) (which cell varies per instance).

• Agent B: the missing boundary value \(u_{i^* j^*}\).

12.0.0.67 Answer.

The integer value \(u_{i_t j_t}\) at a target interior cell.

12.0.0.68 Difficulty scaling (\(d \in \{1,2,3\}\)).

• Grid size \(n = 4\) for \(d \le 1\), \(n = 5\) for \(d \ge 2\) (\(4\times4\) grid with 4 interior cells; \(5\times5\) with 9 interior cells).

• Boundary value magnitudes scale linearly with \(d\) within each grid size.

B7.Linear System with Missing Coefficient (linear_system_missing_coeff)↩︎

12.0.0.69 Setup.

An \(n \times n\) system \(A\mathbf{x} = \mathbf{b}\) (with \(n = d + 1\)) is generated with a random non-singular integer matrix \(A\) and right-hand side \(\mathbf{b}\) (entries in \([-8,8]\)). The unique solution \(\mathbf{x}\) is computed by Gaussian elimination. One coefficient \(A_{i^* j^*}\) (chosen uniformly) is withheld. Instances where \(x_{k^*}\) has denominator exceeding 300 are rejected.

12.0.0.70 Information split.

• Agent A: all coefficients \(A_{ij}\) except \(A_{i^* j^*}\), and all right-hand-side values \(b_i\), in natural language (which coefficient is missing varies per instance).

• Agent B: the missing coefficient \(A_{i^* j^*}\).

12.0.0.71 Answer.

The solution component \(x_{k^*}\) as a reduced fraction.

12.0.0.72 Difficulty scaling (\(d \in \{1,2,3\}\)).

• System size \(n = d+1\) (\(2\times2\), \(3\times3\), \(4\times4\)).

• Coefficient magnitudes bounded by \(4+3(d-1)\) (max 4, 7, 10).

B8.Markov Chain with Missing Transition Probability (markov_missing_transition)↩︎

12.0.0.73 Setup.

An \(n\)-state ergodic Markov chain is generated with a random row-stochastic transition matrix \(P\) (each row sampled by drawing positive integer parts and normalising). One row index \(r^*\) and a column partition are chosen uniformly at random:

• A knows \((n-2)\) entries of row \(r^*\).

• B knows exactly one entry of row \(r^*\) at hint column \(c^*\).

• The remaining entry is derivable from \(\sum_j P_{r^*j} = 1\).

All other rows are fully known to A. The unique stationary distribution \(\boldsymbol{\pi}\) satisfies \(\boldsymbol{\pi}P = \boldsymbol{\pi}\), \(\sum_i \pi_i = 1\).

12.0.0.74 Information split.

• Agent A: all rows of \(P\) completely, except row \(r^*\) where only \(n-2\) entries are known (which row and column vary per instance).

• Agent B: entry \(P_{r^* c^*}\).

12.0.0.75 Answer.

The stationary probability \(\pi_{s^*}\) as a reduced fraction.

12.0.0.76 Difficulty scaling (\(d \in \{1,2,3\}\)).

• Number of states \(n = \min(d+1, 4)\) (2, 3, 4 — caps at \(d=3\)).

• Transition probability denominators up to \(n \cdot (2+d)\), controlling fractional complexity.

B9.Polynomial Interpolation with Missing Point (poly_interpolation)↩︎

12.0.0.77 Setup.

A polynomial \(p\) of degree \(\delta = 2 + \min(d, 2)\) is generated by sampling \(\delta + 1\) integer coefficients from \([-(3+d),\,3+d]\). A set of \(\delta + 1\) distinct integer evaluation points is drawn, and the values \(p(x_i)\) computed. By Lagrange interpolation, \(\delta + 1\) distinct points uniquely determine a degree-\(\delta\) polynomial, so \(\delta\) points leave it underdetermined. A fresh query point \(x^*\) (outside the interpolation set) is sampled and \(p(x^*)\) computed as the target.

12.0.0.78 Information split.

• Agent A: the \(\delta\) evaluation pairs \((x_0, p(x_0)), \ldots, (x_{\delta-1}, p(x_{\delta-1}))\) (which point is held back varies per instance).

• Agent B: the one remaining pair \((x_\delta, p(x_\delta))\).

12.0.0.79 Answer.

The integer \(p(x^*)\).

12.0.0.80 Difficulty scaling (\(d \in \{1,2,3\}\)).

• Degree \(\delta = 2+\min(d,2)\) (3, 4, 4 — caps at \(d=2\), so \(d=3\) is identical to \(d=2\) in structure but with wider coefficient range).

• Coefficient magnitudes bounded by \(3+d\) (max 4, 5, 6).

B10.Portfolio Variance with Missing Correlation (portfolio_variance_missing_corr)↩︎

12.0.0.81 Setup.

A portfolio of \(n = d + 1\) assets has weights \(w_i\), standard deviations \(\sigma_i \sim \mathrm{Unif}[1, 5]\), and pairwise correlations \(\rho_{ij} \in (-1, 1)\) for \(i < j\) (drawn as rational multiples of \(1/D\) for \(D \in \{2,3,4\}\)). Portfolio variance is: \[\sigma_p^2 = \sum_{i=0}^{n-1} \sum_{j=0}^{n-1} w_i\, w_j\, \sigma_i\, \sigma_j\, \rho_{ij}, \qquad \rho_{ii} = 1.\] One correlation pair \((a, b)\) with \(a < b\) is chosen uniformly at random and withheld. Instances where \(\sigma_p^2\) has denominator exceeding 500 are rejected.

12.0.0.82 Information split.

• Agent A: all weights \(w_i\), all standard deviations \(\sigma_i\), and all \(\binom{n}{2}\) pairwise correlations except \(\rho_{ab}\) (which pair varies per instance).

• Agent B: the correlation \(\rho_{ab}\).

12.0.0.83 Answer.

The portfolio variance \(\sigma_p^2\) as a reduced fraction.

12.0.0.84 Difficulty scaling (\(d \in \{1,2,3\}\)).

Number of assets \(n = d+1\) (2, 3, 4), giving \(\binom{n}{2}\) correlation pairs (1, 3, 6 — of which one is always missing).

B11.Hidden Markov Model with Missing Emission (steady_state_missing_emission)↩︎

12.0.0.85 Setup.

A Hidden Markov Model has \(n = d + 1\) states and \(m = \min(d+1, 3)\) observation symbols. The transition matrix \(P\) is fully specified (each row a random distribution over states), and the emission matrix has entries \(E[s][o]\) = probability of observing symbol \(o\) from state \(s\) (each row a random distribution over symbols).

The stationary distribution \(\boldsymbol{\pi}\) of \(P\) is computed by solving \(\boldsymbol{\pi}P = \boldsymbol{\pi}\). The marginal observation probability for a target symbol \(o^*\) is: \[P(\mathrm{observe}\; o^*) = \sum_{s=0}^{n-1} \pi_s\, E[s][o^*].\] One emission entry \(E[s^*][o^*]\) (chosen uniformly) is withheld.

12.0.0.86 Information split.

• Agent A: the full transition matrix \(P\) and all emission entries except \(E[s^*][o^*]\) (which state/symbol entry varies per instance).

• Agent B: the missing emission \(E[s^*][o^*]\).

12.0.0.87 Answer.

The observation probability \(P(\mathrm{observe}\; o^*)\) as a reduced fraction.

12.0.0.88 Difficulty scaling (\(d \in \{1,2,3\}\)).

• Number of states \(n = d+1\) (2, 3, 4).

• Number of symbols \(m = \min(d+1, 3)\) (2, 3, 3 — caps at \(d=2\)).

13 Illustrative Transcript Snapshots↩︎

This appendix gives compact snapshots of representative transcripts. The snapshots are included to make the qualitative failure modes in Section 7 concrete. They are not a sampling protocol and are not used to estimate prevalence. Aggregate conclusions should be drawn from the quantitative tables in the main text and Appendix A. The full transcripts and raw JSON logs are included with the released artifacts.

For each snapshot, Agent A is the solver and Agent B is the fixed constrained information holder. We report only the fields needed to interpret the trace: the instance, the canonical missing fact held by Agent B, the salient request/offer/decline sequence, and the exact-verifier outcome.

We exclude noncanonical offers from the integration-failure examples. If the responder offers a constraint that does not match the canonical atomic hint, the trace is categorized separately as a responder-matching anomaly rather than as a solver integration failure.

13.0.0.1 T1: No canonical hint acquired.

This trace illustrates a request-acquisition failure. The solver identifies the general family and realizes that a birth–death rate is missing, but repeatedly asks for wrong rate slots. Agent B never issues an offer, so the solver exhausts its request budget without the resolving constraint.

Model/mode: gpt-4o-mini, zero-shot 
Instance: bir-000768 
Family: birth_death_missing_rate 
Target: pi_2 
Canonical missing fact held by B: mu_4 = 2 

A requests: "the birth rate lambda_2" 
B declines. 
A requests: "the death rate mu_0" 
B declines. 
A requests: "the death rate mu_2" 
B declines. 
A requests: "the birth rate lambda_2" 
B declines. 
A requests: "the birth rate lambda_1" 
B declines. 
A requests: "the birth rate lambda_3" 
B declines. 
A requests: "the birth rate lambda_0" 
B declines. 
A requests: "the death rate mu_3" 
B declines. 
A requests: "the birth rate lambda_4" 
B declines. 

Outcome: no answer submitted. 
Ground truth: 1/9. 
Failure type: request-acquisition failure. 

This example shows why average request count and declination count are informative: the solver is not merely failing the final arithmetic, but failing to localize the hidden slot under the request budget.

13.0.0.2 T2: Canonical hint acquired, but final answer wrong.

This trace illustrates an integration failure. The solver asks for the correct missing prior and receives it, but then submits an incorrect posterior.

Model/mode: gpt-4o-mini, zero-shot 
Instance: bay-000044 
Family: bayes_missing_prior 
Target: P_H_given_E 
A's private facts: P(E|H) = 1/9, P(E|not_H) = 1/2 
Canonical missing fact held by B: P(H) = 5/8 

A requests: "the prior P(H)" 
B offers: P(H) = 5/8 

A final answer: 5/29 
Ground truth: 10/37. 
Failure type: integration failure after successful hint acquisition. 

The request was successful, so this transcript should not be counted as a gap-identification failure. The incorrect score comes from the downstream computation or simplification step.

13.0.0.3 T3: Canonical hint acquired and final answer correct.

This trace illustrates the intended successful loop. The solver identifies the absent resistor, receives the missing value, and computes the exact equivalent resistance.

Model/mode: gpt-4o-mini, zero-shot 
Instance: cir-000924 
Family: circuit_missing_resistance 
Target: R_eq 
Canonical missing fact held by B: R3 = 7 ohms 

A requests: "the resistance of R3" 
B offers: R3 = 7 ohms 
A computes: 
    G1 = R3 + R7 = 7 + 6 = 13. 
    G2 = R2 + R5 = 11 + 4 = 15. 
    G3 = R1 + G2 = 8 + 15 = 23. 
    G4 = G3 + R6 = 23 + 9 = 32. 
    G5 = G4 + R4 = 32 + 1 = 33. 
    R_eq = (13 * 33) / (13 + 33) = 429/46. 

A final answer: 429/46 
Ground truth: 429/46. 
Failure type: none; full request--resolve--solve success. 

13.0.0.4 T4: Optional prompt-anchoring snapshot for four-shot analysis.

This trace is included only to illustrate a prompt-sensitivity hypothesis. It should not be interpreted as a prevalence estimate. In this moment-recovery instance, the solver repeatedly asks for generic named moments rather than asking for the concrete missing equation held by Agent B.

Model/mode: gemini-2.5-flash, four-shot 
Instance: mom-000404 
Family: moment_problem 
Target: q0, q1, q2 
A's private equations: q0 + q1 + q2 = 12 q1 + 2q2 = 3 
Canonical missing fact held by B: q1 + 4q2 = 5 

A requests: "the 2nd moment" 
B declines. 
A requests: "the 1st moment" 
B declines. 
A requests: "the 0th moment" 
B declines. 
A requests: "the value of moment_0" 
B declines. 
A requests: "the value of moment_1" 
B declines. 
A requests: "the value of moment_2" 
B declines. 
A requests: "the 0th moment" 
B declines. 
A requests: "the 2nd moment" 
B declines. 
A requests: "the 3rd moment" 
B declines. 

Outcome: no answer submitted. 
Ground truth: q0 = 10, q1 = 1, q2 = 1. 
Interpretation: possible exemplar anchoring or mismatch between generic 
moment labels and the concrete constraint held by the responder. 

This example supports the qualitative interpretation in Section 6: four-shot demonstrations may encourage protocol-shaped requests without guaranteeing instance-specific slot localization. It is not used as evidence that this behavior is frequent.

References↩︎