1 Introduction↩︎

With the rapid adoption of large language models (LLMs) in scientific discovery, the expectation for model capability is shifting from simple knowledge retrieval to complex problem solving [1]. However, a fundamental gap exists in how we evaluate these foundation models: in scientific context, the validity of an answer is not intrinsic to the result itself, but is contingent upon whether it is produced within a specific framework of constraints. Unlike everyday user requests where a helpful answer is sufficient, scientific problems are governed by strict norms within the inputs (e.g., assumptions, boundary conditions, definitions, and procedures) that dictate whether a solution is scientifically meaningful or merely numerically plausible. Therefore, as models increasingly tackle rigor-grounded scientific tasks [2]–[4], evaluation must go beyond answer plausibility to assess whether they can strictly adhere to the constraints that establish the scientific validity of that result, namely scientific instruction-following.

However, current benchmarks fail to capture the critical interplay between scientific solutions and justifying constraints. Mainstream instruction-following benchmarks [5]–[8] primarily emphasize surface-level format compliance. For example, IFEval [5] evaluates properties like word counts or JSON validity. While effective for general-purpose dialogue, these criteria are insufficient for science tasks. A model may perfectly satisfy formatting rules yet violating critical scientific rigor, such as applying a formula where it is mathematically undefined or conflating incompatible unit systems. Conversely, prevailing scientific benchmarks [9], [10] focus almost exclusively on final-answer correctness. By treating the reasoning and constraint sanctification process as a black box, these evaluation systematically overlook “right-for-the-wrong-reasons” failures, failing to distinguish a rigorous scientific agent from one that merely arrives at the correct answer without adhering to the underlying scientific framework.

We address this issue by first defining scientific instruction following: the ability to solve a scientific problem while explicitly satisfying the constraints that govern correctness, meaning, and process. Unlike general instruction following, which often concerns stylistic preferences and fluency [11], this capability rests on three pillars of scientific validity. First, it requires adherence to scientific conditions: models must explicitly check applicability ranges, verify boundary conditions, and state assumptions to prove the solution is valid under the given parameters [12]. Second, it demands semantic stability: models must rigorously adhere to unit conventions, symbol definitions, and terminology standards to prevent “meaning drift”, where a model implicitly substitutes one quantity for another. Third, it necessitates adherence to specific scientific processes: when a specific numerical method or experimental protocol is requested, the model must provide auditable, executable steps rather than a generic conclusion.

In this paper, we introduce SciIF, a benchmark for scientific instruction following across multiple disciplines. Each instance contains: (i) a university-level scientific problem, (ii) an enabled subset of constraints drawn from a predefined catalog, and (iii) a reference solution that is validated to be correct and consistent with the enabled constraints. The catalog is designed around scientific practice and spans three families. Condition constraints regulate modeling validity (e.g., assumptions, boundary conditions, applicability range, unit conventions). Terminology constraints prevent meaning drift (e.g., cross-disciplinary disambiguation, in-domain term definitions, symbol and constant conventions, variable naming consistency). Process constraints require a specific scientific procedure (e.g., numerical methods or experimental methods). Because constraints are drawn from a fixed catalog and combined in controlled ways, SciIF  can test compositional instruction following: whether models can coordinate multiple scientific requirements in a single coherent answer [13].

A central design goal of SciIF  is auditability. For each enabled constraint, we specify required evidence that must appear in the model output; judges are instructed not to infer missing evidence. This makes compliance decisions depend on what the model explicitly commits to, rather than on judge guesswork. We report answer correctness, overall multi-constraint compliance, and per-constraint pass rates, enabling fine-grained diagnosis of where scientific instruction following breaks. Furthermore, we demonstrate that fine-tuning on SciIF  confers dual benefits, delivering a 2.8% boost on general instruction following (IFEval) and a remarkable 8.0% improvement on scientific tasks (MMLU-Physics). This indicates that the rigor required for scientific constraints generalizes effectively, sharpening the model’s ability to follow complex rules while deepening its domain expertise.

In summary, our contributions are as follows:

• We introduce SciIF, a multi-discipline benchmark that shifts the evaluation focus from mere numerical correctness to the rigorous execution of scientific instructions.

• We construct a predefined catalog of constraints that supports the generation of controlled problem mixtures for fine-grained failure diagnosis.

• We propose an auditable, evidence-based evaluation protocol that ensures compliance is measured by explicit logical commitments rather than implicit guesswork.

• We demonstrate the utility of SciIF  for model alignment, showing that learning strict scientific constraints generalizes to improve both general instruction following and domain-specific reasoning.

2 Related Work↩︎

2.0.0.1 Answer-Centric Scientific Benchmarks.

Most scientific and academic evaluations score models primarily by final-answer correctness. This includes both single-discipline benchmarks such as PhysUniBench [14] that target a specific domain and multi-discipline suites—such as MMLU [9] and BIG-Bench Hard [15]—as well as science-focused evaluations such as SciBench [10] and SciEvalKit[16]. While these benchmarks effectively quantify knowledge and reasoning under objective metrics, they often treat the derivation process as a black box. As a result, models can be “right for the wrong reasons”: producing a correct value while violating constraints that govern scientific validity and meaning (e.g., unstated assumptions, missing applicability checks, unit inconsistencies, or symbol drift). SciIF complements correctness-centric evaluation by explicitly measuring scientific constraint compliance as a separate axis. Concretely, it scores whether required scientific evidence is stated and checked in the output (e.g., explicit boundary substitutions or applicability-range validation), rather than assuming that a plausible derivation implicitly satisfies the rules.

2.0.0.2 General Instruction Following Evaluation.

Instruction following has been studied as cross-task generalization from natural language instructions [5]–[8], [17], [18], and improved through instruction tuning and synthetic instruction generation [19]–[21]. Human-centric evaluation and LLM-as-a-judge frameworks optimize for preference and overall assistant quality [22]–[28], but they typically do not require explicit evidence that scientific constraints are satisfied; fluent outputs can appear acceptable even when scientific conventions are violated. SciIF is complementary: it evaluates whether models both solve the task and explicitly satisfy the scientific constraints that justify the solution, and it enables tracking trade-offs between correctness and constraint compliance under controlled mixtures of constraints.

3 Benchmark Construction↩︎

SciIFis designed to evaluate scientific instruction following as a capability distinct from answer accuracy. In scientific problem solving, constraints such as unit discipline, symbol meaning, validity conditions, and method requirements are not cosmetic—they determine whether an answer is interpretable, reproducible, and even semantically correct. To surface these failures, SciIFseparates evaluation into two axes: (i) correctness of the scientific outcome, and (ii) explicit, auditable compliance with enabled scientific constraints.

3.1 Task Definition↩︎

SciIFevaluates scientific instruction following along two separate axes. Given a scientific problem, we score not only whether a model reaches the correct scientific result, but also whether it explicitly follows the enabled scientific constraints.

Each instance is a triple \((x, C, y^\star)\). Here \(x\) is a university-level scientific problem, \(C\) is a small set of enabled constraints drawn from a fixed catalog, and \(y^\star\) is a reference solution that is both correct and compliant with \(C\). A model receives \((x, C)\) and produces an output \(\hat{y}\).

3.1.0.1 Answer Correctness.

We score whether \(\hat{y}\) matches \(y^\star\) on the required scientific outcome, using task-appropriate equivalence criteria.

3.1.0.2 Constraint Compliance.

In parallel, we score whether \(\hat{y}\) provides explicit, problem-grounded evidence for each enabled constraint in \(C\). Judges check evidence presence rather than infer missing assumptions or validity checks, which cleanly separates correct-but-non-compliant outputs from compliant-but-incorrect ones.

3.2 Dataset and Constraint Catalog↩︎

SciIFuses a fixed test set of 334 university-level problems across Biology, Chemistry, Materials, and Physics, and we additionally release 910 training problems used for SFT and verifier-based RL. Each test instance enables a small set of constraints drawn from a fixed catalog of ten atomic constraints. The catalog spans three families: condition constraints (Assumptions, Boundary Conditions, Applicability Range, Units Standard), terminology constraints (Cross-disciplinary Term Disambiguation, Intra-discipline Term Definitions, Symbols & Constants Standardization, Variable Naming Consistency), and process constraints (Numerical Methods, Experimental Methods). Constraints are format-agnostic; we do not require a rigid template. A constraint passes only when the model states explicit, problem-grounded evidence tied to the instance’s symbols and values, rather than generic boilerplate.

3.3 Construction Pipeline↩︎

We construct candidate instances and apply a three-stage quality control pipeline. The stages are designed to separate three failure modes: ill-posed problems, incorrect references, and ungrounded constraints. Only instances that pass all stages are included in the final benchmark.

3.3.0.1 Stage 1: Problem Validity.

Goal: ensure the problem statement itself is well-posed. We check that the prompt provides sufficient information to determine the target quantity, that the scientific setting is internally consistent, and that the enabled constraints are relevant to the task described by the prompt. Instances fail this stage if the question is under-specified, contradictory, or if a constraint is enabled without any anchor in the problem statement that makes it applicable.

3.3.0.2 Stage 2: Reference Correctness.

Goal: ensure the reference solution provides the correct scientific outcome. Independently of constraint evidence, we solve or verify the problem and confirm that \(y^\star\) matches the requested target result and key conclusions. Instances fail this stage if the reference contains numerical errors, incorrect reasoning that changes the result, or mismatches the stated targets.

3.3.0.3 Stage 3: Constraint Grounding and Auditability.

Goal: ensure constraints are truly binding and auditable for this instance. We verify that every enabled constraint is instantiated by the prompt and that the reference explicitly provides the required evidence to satisfy that constraint. This stage is necessary because correctness alone does not guarantee that a constraint is meaningful or checkable. Instances fail this stage when a constraint is nominal rather than binding, or when the reference does not contain explicit, problem-grounded evidence that would allow a judge to audit compliance.

3.3.0.4 Stage 4: Domain Expert Review.

Goal: ensure the instance reflects realistic scientific practice and remains interpretable to practitioners. After passing automated and internal checks, each candidate is reviewed by a domain expert who assesses (i) whether the scientific context and assumptions are plausible for the stated domain, (ii) whether quantities, units, and parameter ranges are scientifically meaningful, and (iii) whether the reference solution communicates conclusions in a way consistent with field conventions. Instances fail this stage if an expert flags the setting as implausible, identifies domain-knowledge contradictions, or judges that the prompt or reference would mislead a practitioner despite being formally solvable.

3.4 Evaluation and Auditing↩︎

SciIFfollows a generate–then–audit protocol. Given a problem and its enabled constraint set, the model first produces an answer \(\hat{y}\). We then audit answer correctness and constraint compliance as two separate axes, so that being numerically correct can be distinguished from being scientifically auditable.

3.4.0.1 Constraint Auditing and Aggregation.

For each enabled constraint \(c \in C\), the auditor assigns PASS or FAIL by checking whether the model output contains explicit, instance-grounded evidence for that constraint. Evidence must be clearly stated and tied to the concrete symbols and values in the problem. If evidence is missing, vague, or contradicted by the solution, the constraint fails. For items with multiple constraints, we aggregate with a logical AND over enabled constraints: an item is compliant only if every \(c \in C\) passes. This makes failures directly attributable at the constraint level.

3.4.0.2 Correctness as an Independent Axis.

Correctness is audited independently by comparing \(\hat{y}\) to the reference \(y^\star\) under task-appropriate equivalence criteria for the final outcome and key conclusions. The correctness audit ignores constraint-following issues to avoid conflating scientific reporting failures with arithmetic or reasoning errors. This separation isolates two common regimes in scientific QA: correct-but-non-compliant outputs and compliant-but-incorrect outputs.

3.5 Hybrid Auditing↩︎

3.5.0.1 Rules First, Judges as Fallback.

To maximize reproducibility and minimize subjectivity, we use a hybrid auditing pipeline that prioritizes deterministic checks. We first apply a rule-based auditor for signals that admit stable verification, including parsing LaTeX math expressions, checking numerical equivalence under tolerances, validating required unit declarations, and matching constraint-specific evidence patterns. When a rule-based check is not applicable or yields an unreliable decision, we fall back to model-based judging under a strict evidence standard.

3.5.0.2 Dual Judging with Aligned Standards.

Model-based judging uses two independent judges, GPT-5.1 and Gemini-3-Flash, with a conservative agreement rule: a constraint passes only if both judges mark it PASS. In practice, different judges can apply systematically different strictness even on the same response. We therefore align judging standards using a held-out calibration set that is never used for reported results. We identify recurring disagreement modes and encode the corresponding decision thresholds into a fixed, standardized judging prompt. The standardized prompt instructs judges to evaluate evidence presence and consistency only, and to avoid reconstructing missing checks, assumptions, or derivations. After calibration, the judging prompts are fixed for all experiments.

4 Experiments↩︎

We evaluate whether scientific QA models can be both correct and scientifically auditable. SciIFreports two metrics per model: answer correctness and multi-constraint compliance.

4.1 Setup↩︎

We evaluate on the 334-item SciIFtest set spanning Biology, Chemistry, Materials, and Physics. We include a mix of frontier and open models, with the full list deferred to Appendix 9.1. All models use identical decoding settings and the same prompting template (Appendix 6). Compliance and correctness are scored with our dual-judge auditing protocol.

4.1.0.1 Post-Training Variants.

To test whether SciIF training improves both scientific QA and scientific auditability, we post-train Qwen3-8B on our 910-item training set in two stages. SFT targets stronger scientific problem solving and higher answer correctness. We then apply verifier-based RL that rewards explicit, problem-grounded constraint evidence, encouraging the model to proactively surface domain-relevant validity information such as assumptions, applicability limits, unit discipline, and boundary checks. We evaluate the base model, SciIF-SFT, and SciIF-RL on SciIFand on external benchmarks (Appendix 8).

4.2 Correctness is not Compliance↩︎

Our central finding is a persistent gap between getting the right answer and meeting scientific constraints. Across all evaluated models, answer correctness is substantially higher than strict multi-constraint compliance. Even strong models exceed \(80\%\) correctness, yet overall multi-constraint pass remains below \(30\%\) (best: \(29.6\%\)). This pattern holds across model families, indicating a systematic failure mode: models often reach the correct final result while omitting explicit evidence for one or more constraints that govern validity, meaning, and reproducibility.

Among closed-source models, GPT-5.2 and Grok-4 demonstrate superior performance, excelling in language understanding and computational reasoning respectively. In the open-source category, GLM-4.7 shows particularly strong capabilities across most dimensions.

We furthre visualize this gap in Figure 3. For most models, single-constraint pass is relatively strong, which indicates that many scientific requirements are individually achievable when they are the only thing to remember. The failure appears when constraints must be coordinated: multi-constraint overall pass collapses to a much lower regime even when correctness stays high. This separation shows that the dominant error is not inability to solve the underlying scientific problem, but inability to reliably produce a solution as a reproducible artifact with explicit, problem-grounded scientific discipline. In practice, this means an accuracy-only evaluation can label an output as “solved” even when it omits units, leaves assumptions implicit, fails to state validity limits, or drifts in symbol meaning—issues that can silently break interpretation or downstream reuse.

4.3 Compositional Effects↩︎

Correctness and compliance are related but asymmetric. Compliant answers are typically correct, while correct answers are frequently non-compliant, so accuracy-only evaluation overestimates scientific instruction-following ability. We also observe a consistent compositional collapse: compliance drops sharply as the number of enabled constraints increases from \(k{=}2\) to \(k{=}5\) (Appendix 9.2), revealing a coordination bottleneck in maintaining multiple scientific requirements within one derivation.

4.4 Generalizability↩︎

We test whether training on SciIF data improves performance outside SciIF. Starting from Qwen3-8B, we compare the base model to an SciIF-SFT checkpoint and an SciIF-RL checkpoint. Table 2 shows consistent gains on IFEval (Strict) (Appendix 7.6) and improved performance on MMLU science subjects (Appendix 8.7 Fig. 7), suggesting that constraint-oriented post-training can transfer to broader instruction adherence and scientific QA rather than only improving in-domain compliance (Appendix 6.1).

4.5 Where Models Break↩︎

Constraint-level analysis shows that failures concentrate on global semantic requirements such as Units Standard and Applicability Range, which demand explicit, problem-grounded statements and penalize silent assumptions. Full per-constraint breakdowns are reported in Appendix 9.

4.6 Explicit rubrics reduce judge variance↩︎

Because our evaluation uses LLM-as-a-judge, the judge prompt effectively defines the scoring rule. When constraints are under-specified, different judge models apply different implicit thresholds (e.g., treating \(0.1\) vs.\(0.01\) error as “correct”), leading to non-comparable compliance scores. We therefore make scoring points and decision thresholds explicit in the prompt (Appendix 6.4). This calibration sharply reduces inter-judge discrepancy: before clarification, GPT-5.1 vs.Gemini-3-Flash yields \(68.7\%\) vs.\(88.0\%\); after clarification, \(79.1\%\) vs.\(81.4\%\), reducing the gap from 19.3 to 2.3 points. This motivates treating prompt-level rubric specification as a first-class part of protocol.

4.7 Compositional Collapse: Patterns and Anomalies across Models↩︎

Beyond aggregate scores, the per-\(k\) breakdown reveals a consistent coordination bottleneck. As the number of enabled constraints increases, compliance drops steeply for nearly all models, even when answer correctness remains comparatively stable. Figure 4 visualizes this effect for ten representative models selected to cover typical and atypical curve shapes.

4.7.0.1 A shared baseline shape: smooth exponential-like decay.

Strong closed models such as GPT-5.2 and GPT-5.1 exhibit a similar high-start curve at \(k{=}2\), followed by a near-monotonic decline through \(k{=}5\). This pattern suggests that failures are rarely caused by a single missing skill. Instead, they arise from global constraint scheduling: maintaining unit discipline, symbol meaning, and required validity checks while completing the derivation.

4.7.0.2 Cliff-drop regimes: fragile constraint composition in open models.

Several models enter a “cliff” regime where compliance collapses rapidly once \(k{\ge}3\). For example, Qwen3-235b reaches low but non-zero compliance at \(k{=}2\) and \(k{=}3\), then drops to \(0\%\) for \(k{\ge}4\). This shape indicates that the model can sometimes satisfy isolated constraints, but fails to keep multiple constraints active across a longer solution trajectory.

4.7.0.3 Non-monotonic anomalies: brittleness and measurement effects.

A small set of models exhibit non-monotonic behavior, including partial rebounds at \(k{=}5\), or “zero-then-recovery” patterns. We treat these as diagnostics rather than evidence of improved compositional ability. Two factors can produce such anomalies: (i) small-sample variance at high \(k\) due to fewer items, and (ii) discrete rubric thresholds where a model occasionally “remembers” to add a missing evidence sentence that flips an AND-aggregated decision from FAIL to PASS. These anomalies reinforce a key point: compliance failures are often driven by missing explicit evidence, not necessarily by incorrect scientific reasoning.

4.8 Takeaway↩︎

SciIFmakes visible a gap that correctness-centric benchmarks systematically miss: scientific correctness and scientific auditability are distinct capabilities. Current models often produce correct results without reliably producing the explicit evidence needed for reproducible scientific reasoning, and this weakness becomes sharper under multi-constraint composition.

5 Conclusion↩︎

We introduce SciIF to evaluate scientific instruction following, addressing the limitations of current benchmarks that overlook the interplay between solution correctness and constraint satisfaction. Through a taxonomy of domain-specific constraints and an auditable evaluation protocol, SciIF distinguishes between rigorous reasoning and lucky guessing. In addition, our protocol emphasizes transparency and reproducibility, enabling consistent comparisons across models and settings. Crucially, we show that the rigor demanded by SciIF generalizes: models fine-tuned on our data exhibit dual improvements, boosting IFEval by 2.8 points and MMLU-Physics by 8.0 points. We will release SciIF to facilitate the development of LLMs that are not only knowledgeable but strictly compliant with the norms of scientific practice.

Limitation↩︎

We acknowledge that our dataset focuses on text-based problems, omitting multimodal scientific tasks (e.g., DNA, RNA sequence generation/understanding). Future work will extend SciIF to multimodal settings and explore iterative, multi-turn scientific agents.

Appendix Contents

AEvaluation Protocol and Prompt Templates

A.1Generate–then–audit protocol

A.2Rule-based verifier with judge fallback

A.3Answer generation prompts

A.4Judge prompts

A.5Decision policies

A.6Strict vs.Loose summary

A.7External consistency check on instruction following

A.8Constraint rubric summary

BBenchmark Details

B.1Source-type composition of the 1244-item pool

B.2Evidence-based rubrics

B.3Optional regeneration and stability filtering

B.4Judge scale alignment

B.5Case study: judge disagreement and prompt calibration

CPost-Training Details

C.1Case study: what verifier-based RL changes on IFEval

C.2Case study: SciIF SFT improves structured reasoning on MMLU

DAdditional Experimental Results and Diagnostics

D.1Models and inference settings

D.2Compositional collapse vs.number of constraints

EAI Assistant Use

FHuman Validation Protocol for Equivalence Between Model and Reference Answers

6 Evaluation Protocol and Prompt Templates↩︎

This appendix provides the information needed to reproduce our evaluation and training signals. We first document the generate–then–audit protocol and the rule-based verifier–judge pipeline, then list the answer-generation prompts, judge prompts, and decision policies used in Strict and Loose modes.

6.1 Generate–then–audit protocol↩︎

6.1.0.1 Two-axis evaluation.

Each model output is evaluated on two axes: answer correctness and constraint compliance. Correctness checks whether the required scientific outcome matches the reference under task-appropriate equivalence criteria. Compliance checks whether the output contains explicit, problem-grounded evidence for each enabled constraint. Judges evaluate evidence presence rather than infer missing checks, assumptions, or definitions.

6.1.0.2 Per-constraint auditing and aggregation.

Each enabled constraint is audited independently and receives PASS/FAIL. For multi-constraint items, item-level compliance is the logical AND across enabled constraints.

6.1.0.3 Dual-judge robustness.

We use two independent judges and apply a two-of-two rule. A constraint passes only if both judges return PASS, reducing sensitivity to single-judge drift and stylistic preference.

6.2 Rule-based verifier with judge fallback↩︎

We prioritize deterministic checks when feasible and fall back to LLM judges when a rule-based check is not applicable or fails to parse the output. Rule-based checks include numerical equivalence under task-defined criteria, expression parsing when needed, unit and dimensional consistency checks when units are stated, and mechanical substitution checks for boundary or limiting cases. When a rule cannot be executed reliably, the decision is delegated to the judge prompts below.

6.3 Answer generation prompts↩︎

6.4 Judge prompts↩︎

We audit correctness and constraint compliance separately. For compliance, judges evaluate one enabled constraint at a time against explicit evidence points and return a pointwise verdict.

6.5 Decision policies↩︎

6.5.0.1 Strict policy.

A point is YES only when the model output contains explicit evidence that is problem-grounded, correct, and consistent with the model’s own derivation. Missing evidence is NO. Incorrect claimed checks are NO.

6.5.0.2 Loose policy.

Loose mode keeps the same non-inference and truthfulness requirements, but only MAIN points are required for overall PASS. SECONDARY points may fail without failing the constraint.

6.6 Strict vs.Loose summary↩︎

We use two compliance modes to separate missing-evidence failures from substantive violations. Strict is the primary reporting setting and is designed for auditability. Loose is a diagnostic setting used to test whether low scores are driven mainly by omission of write-ups.

7 Benchmark Details↩︎

This section provides additional information about rubric structure, instance construction, and judge calibration.We provide a high-level overview of the ten constraints in Table 4 to clarify what constitutes auditable evidence at a glance.The full point-wise definitions, edge cases, and the Strict/Loose point split are provided in the supplementary rubric specification.

7.1 Source-type composition of the 1244-item pool↩︎

Our full data pool contains 1244 items and is intentionally dominated by textbook-style problems. We use web-derived items as a secondary source to broaden coverage, while keeping QA-style items as a smaller component. Overall, the source-type mixture follows an approximate 7:2:1 ratio: about 70% textbook, about 20% web, and about 10% QA. This design emphasizes academically grounded problem structures while still injecting topical diversity and alternative phrasing patterns.

7.2 Evidence-based rubrics↩︎

Each enabled constraint is accompanied by evidence points designed to be checkable without reconstruction or guesswork. A constraint passes only if all required points pass. Any missing, vague, or contradictory evidence causes failure. We release the full point-level rubric specification in machine-readable form as supplementary material.

7.3 Optional regeneration and stability filtering↩︎

For a small subset of items, generation can yield good problems but brittle references. When needed, we regenerate reference candidates and retain instances that fall within a target stability window, filtering out instances that are effectively trivial or near-impossible under the same constraints.

7.4 Judge scale alignment↩︎

Two-of-two voting reduces noise but does not guarantee that judges apply identical evidence thresholds. We align judge scales using a held-out calibration set that is never used for reported scores. We compare judge outputs per evidence point, identify recurring disagreement patterns, and refine judge prompts by making those thresholds explicit and executable. After calibration, prompts are fixed for all experiments.

7.5 Case study: judge disagreement and prompt calibration↩︎

We conducted a targeted case study to quantify and reduce judge arbitrariness. We sampled 50 instances and re-audited the same model outputs with two judge models, then compared pointwise PASS/FAIL decisions under the same rubric. We observed non-trivial disagreement: 19 out of 50 instances had at least one constraint-level mismatch, with 20 constraint-level mismatches in total. Disagreements concentrated on globally semantic constraints such as Symbols & Constants Standardization and Intra-discipline Definitions, where the effective threshold for “explicit evidence” differed across judges. These findings motivated prompt calibration that removes packaging-dependent expectations and rewrites evidence points into problem-grounded, executable checks, leaving minimal room for stylistic interpretation.

7.6 External consistency check on instruction following↩︎

As an external sanity check, we evaluated Qwen3-8B and its post-trained variants on IFEval under Strict and Loose settings. Verifier-guided reinforcement learning improves instruction following beyond supervised fine-tuning alone, with the RL variant achieving higher Strict performance than both the base and SFT variants. This supports the interpretation that constraint-oriented post-training strengthens general instruction discipline rather than only improving in-domain compliance on SciIF.

7.7 Constraint rubric summary↩︎

We define ten atomic constraints grouped into three families. Table 4 summarizes the main evidence requirements at a high level.

8 Post-Training Details↩︎

This section documents the SFT and verifier-based RL objectives, reward construction, and implementation notes.

8.1 SFT objective↩︎

We train a LoRA adapter on top of Qwen3-8B with the standard next-token likelihood objective: \[\mathcal{L}_{\mathrm{SFT}}(\theta) \;=\; - \sum_{t} \log \pi_\theta\!\big(y_t^\ast \mid q, y_{<t}^\ast\big), \label{eq:sft95objective95app}\tag{1}\] where \(y^\ast\) is the reference solution.

8.2 RL objective↩︎

We optimize a KL-regularized RL objective: \[\begin{align} J(\pi_\theta) &= \mathbb{E}_{(q,r)\sim \mathcal{D}} \Big[ \mathbb{E}_{o \sim \pi_\theta(\cdot \mid q)} \big[ R(q,o,r) \big] \\ &\qquad - \beta\, D_{\mathrm{KL}}\!\big(\pi_\theta(\cdot \mid q)\,\|\,\pi_{\mathrm{ref}}(\cdot \mid q)\big) \Big] \end{align} \label{eq:rl95objective95app}\tag{2}\]

where \(\pi_{\mathrm{ref}}\) is initialized from the SFT checkpoint and \(\beta\) controls deviation from the reference.

8.3 Constraint verifier↩︎

The verifier evaluates a response against the enabled constraints: \[V:(q,o,r)\mapsto \mathbf{v}\in\{0,1\}^d,\] where \(v_i=1\) iff the output provides the required evidence to satisfy constraint \(r_i\).

8.4 Reward construction with grouped severity↩︎

Let \(s_i\in\{0,1\}\) denote PASS/FAIL for constraint \(r_i\) from the verifier. We partition constraints into groups \(g\in\mathcal{G}\), assign within-group weights \(w_i\), and between-group weights \(W_g\).

For a group \(g\), the normalized group score is: \[R_g \;=\; \frac{\sum_{i\in g} w_i\, s_i}{\sum_{i\in g} w_i}.\] We combine group scores into an overall constraint-compliance score: \[R_{\mathrm{c}}(q,o,r) \;=\; \frac{\sum_{g\in\mathcal{G}} W_g\, R_g}{\sum_{g\in\mathcal{G}} W_g}.\] We add a binary answer-correctness signal \(R_{\mathrm{a}}(q,o)\) and compute the final scalar reward: \[R(q,o,r) \;=\; 0.7\,R_{\mathrm{c}}(q,o,r) \;+\; 0.3\,R_{\mathrm{a}}(q,o). \label{eq:final95reward95app}\tag{3}\]

8.5 Implementation notes and selected results↩︎

We perform PPO-style optimization with KL regularization to the SFT reference policy. On our evaluation, RL increases the single-constraint pass rate from \(5.0\%\) to \(11.7\%\) and answer correctness from \(24.3\%\) to \(25.7\%\). Among individual constraints, symbol and constant conventions improve from \(5.9\%\) to \(24.5\%\).

8.6 Case study: what verifier-based RL changes on IFEval↩︎

To understand what our verifier-based RL stage changes beyond in-domain compliance on SciIF, we run an external consistency check on IFEval under the Strict setting. We compare Qwen3-8B in a zero-shot setting against Qwen3-8B-RL. The RL variant improves Strict accuracy from 80.41% (435/541) to 83.18% (450/541), a gain of 2.77 points corresponding to 15 additional passing instances. To characterize the behavioral shift, we focus on the 48 cases where RL passes while zero-shot fails.

8.6.0.1 Where the gains concentrate.

The RL gains are not uniformly distributed across instruction types. They cluster in constraints that penalize extra “helpful” text and require precise surface-form control. The most frequent improved categories are sentence and word length constraints, case transformation constraints, forbidden keyword constraints, letter-frequency constraints, and punctuation bans. This pattern indicates that RL primarily reduces a consistent failure mode of general chat-style models: they often prefer conversational packaging and elaboration over strict compliance when the instruction demands a tight output envelope.

8.6.0.2 Behavioral shift: from conversational packaging to executable compliance.

Across improved cases, we observe three recurring corrections. First, the RL model reduces preambles and meta-commentary that violate strict formatting requirements. Second, it exhibits tighter control of quantitative length constraints, often showing implicit self-monitoring behavior such as stopping early and compressing content while preserving task intent. Third, it better coordinates multiple constraints simultaneously, avoiding partial satisfaction where one constraint is met but another is silently violated.

8.6.0.3 Representative examples.

We present three representative Strict-mode examples that illustrate the dominant error patterns. In all examples, the task content is easy for both models. The failures arise from instruction-following discipline rather than missing knowledge.

8.6.0.4 Implication for our training objective.

These results support an interpretation consistent with our SciIFfindings. Verifier-based RL strengthens a general notion of instruction discipline, especially in constraints that demand explicit surface-form control. This aligns with our benchmark design, where compliance is judged from written evidence. The IFEval case study suggests that the same training signal that improves auditable scientific constraint satisfaction also reduces format and length violations in a different instruction-following domain.

8.7 Case study: SciIF SFT improves structured reasoning on MMLU↩︎

We include a concrete MMLU-style example to illustrate a qualitative change we repeatedly observe after SciIF SFT 7: the model becomes more stable during solution writing and more reliable at making the final discrete decision that the task demands. In this item, all variants know the correct physics formula, yet they differ sharply in execution discipline. The SciIF SFT variant keeps exponent arithmetic explicit, performs a quick order-of-magnitude sanity check, and maps the estimate to the closest option without drifting into irrelevant text or misreading the scale.

None

9 Additional Experimental Results and Diagnostics↩︎

This section reports extended tables and diagnostic analyses that complement the main paper.

9.1 Models and inference settings↩︎

We evaluate: GPT-5.2, GPT-5.1, Gemini-3, Grok-4, Qwen3-max, GPT-o3, GPT-o4mini, Claude-4.5sonnet, Deepseek-v3.2, Minimax-M2, and GPT-4o. All models use identical inference settings: temperature \(=0\), max tokens \(=4096\), no tools or web.

9.2 Compositional collapse vs.number of constraints↩︎

Compliance drops sharply as the number of enabled constraints increases. We report the per-\(k\) breakdown in Table 5.

10 AI Assistant Use↩︎

We used GPT and Gemini for code refactoring/optimization and for polishing the manuscript’s language and formatting. All changes were reviewed by the authors, who take full responsibility for the final content and results.

11 Human Validation Protocol for Equivalence Between Model and Reference Answers↩︎

This appendix explains, for human readers, how we manually assess whether a model’s answer is equivalent to the reference (gold) answer. The process targets university-level problems and focuses on the equivalence of numerical results, key conclusions, and core reasoning, while keeping style or compliance issues out of scope.

1. What We Check↩︎

• Numerical agreement: When a numerical result is required, we verify that the final numbers match the reference. To ensure comparability, all reported numbers use exactly four significant figures; if the reference specifies a range or tolerance, we adhere to that.

• Conclusion agreement: Categorical outcomes (true/false, multiple-choice selection, sign/direction, inequality relations) must match the reference. If the reference states “undetermined” or “requires more information,” any definite conclusion in the model answer is considered non-equivalent.

• Conceptual and reasoning equivalence: Different wording is acceptable, but the core method or theorem employed should be equivalent (e.g., the same physical law or the same convergence criterion). Alternative methods are acceptable if they are logically equivalent for this problem and properly justified.

2. What We Do Not Penalize↩︎

• Minor formatting or wording differences, different paragraphing, or non-essential elaborations that do not affect the substance of the answer.

3. Step-by-Step Procedure↩︎

1. Preparation: Review the problem and the reference answer; extract key data, conditions, and the final conclusion.

2. Evidence tagging: Read the model answer and highlight explicit, problem-specific evidence (data, formulas, method statements), ensuring it ties to the symbols/conditions of this problem.

3. Numerical check: Compare all numerical conclusions item by item, reporting numbers with exactly four significant figures and verifying units and sign/direction where applicable.

4. Conclusion check: Verify that selections, truth values, and relational/directional statements match the reference; if the reference is “indeterminate,” ensure the model maintains the same stance.

5. Reasoning check: Confirm that the core concepts and methods align with the reference. If a different method is used, verify that it is logically equivalent and correct for this problem.

6. Final decision: If numerical results, key conclusions, and core reasoning align, we mark the answers as equivalent. Otherwise, we record the discrepancy with a brief reason (e.g., “different final value,” “method not equivalent,” “definite conclusion given where reference is indeterminate”).

4. Boundaries and Notes↩︎

• This process does not evaluate formatting templates, submission style, or other compliance requirements (handled separately).

• When the reference specifies units or measurement conventions, we follow them; if not specified, we use the conventions implied by the problem statement.

• For readability, we integrate evidence within a natural narrative rather than relying on checklists.

Using this protocol, we provide transparent, reproducible, and academically rigorous judgments of equivalence between model answers and the reference solutions.

References↩︎