1 Introduction↩︎

While Large Language Models (LLMs) [1]–[3] excel in general NLP, they struggle with "content effects", overestimating the logical validity of arguments aligned with world knowledge [4]–[6]. This entanglement hinders the development of trustworthy AI that prioritizes logical robustness [7]. We address this in SemEval-2026 Task 11 [8], aiming to assess the formal validity of syllogistic arguments across 12 diverse languages², independent of their plausibility (alignment with world knowledge). Furthermore, the task requires models to exhibit robustness by identifying relevant premises amidst irrelevant noise.

Our system utilizes mDeBERTa-v3 backbones [9] pre-trained on NLI tasks [10]. To ground the architecture in formal logic, we curate a synthetic dataset via a rule-based engine following syllogistic schemes [11], avoiding the noise of LLM-augmented data. We optimize via a multi-objective function integrating Group DRO [12], a scheduled bias-weighting parameter \(\lambda\), and KL Divergence to decouple formal reasoning from plausibility, and to enforce structural invariance across complex linguistic variations.

Our code and data generation scripts are publicly available to facilitate further research.³

2 Task Description↩︎

The task requires classifying the logical validity of a conclusion (\(C\)) given premises (\(P\)) across four subtasks. Evaluation targets the 2\(\times\)​2 intersection of Validity and Plausibility (Table 1).

We participated in all four subtasks: ST1 (English), ST2 (English with irrelevant premises), ST3 (12-language multilingual), and ST4 (12-language multilingual with irrelevant premises). ST2 and ST4 require joint validity classification and relevant premise selection. The primary metric is the Ranking Score (\(S_{rank}\)), which divides a performance component \(P\) by a logarithmic penalty on the Total Content Effect (TCE) [8]. \(P\) is Accuracy for ST1/ST3 and the average of Accuracy and F1 for ST2/ST4. TCE measures bias by comparing accuracy across the validity–plausibility quadrants; lower TCE indicates higher logical robustness.

The training dataset provided contained 960 English data points balanced across quadrants (VP: 25.0%, VI: 25.0%, IP: 24.4%, II: 25.6%).

3 Related Work↩︎

Recent work on logical reasoning in LLMs can be categorized along four directions, following the taxonomy of [1].

3.0.0.1 Neuro-Symbolic Approaches

[3] integrates external theorem provers to verify LLM-generated natural language explanations. By iteratively formalizing explanations and validating them with a logic solver, their feedback loop ensures logical soundness and completeness.

3.0.0.2 Prompt-Based Methods

[2] propose a Syllogistic-Reasoning Framework of Thought (SR-FoT), extending Chain-of-Thought prompting to a multi-stage syllogistic reasoning process. By generating major and minor premises before the conclusion, SR-FoT enforces strict information control, reducing errors and improving logical consistency.

3.0.0.3 Model Optimization

[5] proposes Activation Steering to disentangle content plausibility from logical validity within LLM hidden representations. Using probing techniques to identify content-sensitive layers, they apply conditional steering to guide the model toward formal reasoning. Similarly, [13] proposes a Logical Control Framework that separates content and logic subspaces via contrastive learning, nudging representations toward logical validity during inference without altering content.

3.0.0.4 Analytical Studies

[6] identifies distinct attention circuits responsible for formal reasoning versus content-dependent processing, revealing that certain attention heads are both necessary and sufficient for logical inference. On the evaluation side, [14] introduces Multi-LogiEval, demonstrating that LLM accuracy degrades as the number of premises and reasoning steps increases.

4 System Overview↩︎

Our system is designed to decouple semantic patterns from logical structure through synthetic data generation, bias-aware optimization, and leveraging the structural attention of mDeBERTa-v3 backbones.

4.1 Architecture↩︎

We utilize mDeBERTa-v3 [9] as our backbone, initialized from NLI-pretrained checkpoints [10]: mDeBERTa-v3-base-xnli-multilingual-nli-2mil7 for ST1 and ST3, DeBERTa-v3-large-mnli-fever-anli-ling-wanli for ST2, and microsoft/mdeberta-v3-base for ST4.⁴ The larger backbone for ST2 proved necessary for disentangling noisy premises. For ST1/ST3, a classification head consumes the [CLS] token for binary validity classification. For ST2/ST4, we use two specialized heads:

• Classification Head: Predicts logical validity from the [CLS] embedding.

• Premise Selection Head: Uses span-based pooling between [SEP] tokens to extract premise representations. We apply Masked Mean Pooling to each span, passing the resulting fixed-size vectors to a sigmoid layer for relevance probability.

4.2 Data Strategy↩︎

To overcome the scarcity of the provided dataset (\(N=960\)), we developed a rule-based engine to generate synthetic data based on the 256 Aristotelian moods and figures [11].

4.2.0.1 Vocabulary and Plausibility

We select nouns from WordNet [15] hypernym/hyponym chains using the NLTK library [16]. Plausible samples preserve hierarchical ordering (e.g., \(car \subset vehicle \subset artifact\)) for nouns, while implausible samples use cross-chain or random nouns. To further reduce content reliance, we introduce Symbolic Data (using gibberish/symbolic variables) and Complex Rephrasing (e.g., mapping "All \(x\) are \(y\)" to "Every \(x\) is, without exception, a \(y\).").

4.2.0.2 Relevant Premises Labeling

For invalid syllogisms in ST2 and ST4, we retain the indices of premises that provide partial entailment. This prevents the model from treating premise relevance as a purely binary consequence of overall validity, encouraging structural over-fitting to logical forms.

4.2.0.3 Multilingual Expansion

We first generated English samples, then translated nouns via DeepL⁵, and then translated the structure using language-specific rule-based templates. This ensures that the quantifiers (e.g., "No", "Some") are not modified across all 12 target languages.

4.2.0.4 Syntactic Robustness (HANS)

To ensure the models do not exploit surface-level syntactic heuristics (e.g., lexical overlap) during inference, we additionally integrate samples from the Heuristic Analysis for NLI Systems (HANS) dataset [17] into our robust training distributions.

4.3 Bias-Aware Optimization↩︎

To optimize for the Ranking Score, we use a multi-objective function: \[\mathcal{L}_{total} = \mathcal{L}_{DRO} + \lambda_{bias} \mathcal{L}_{bias} + \gamma \mathcal{L}_{KL}\]

4.3.0.1 Adaptive Group DRO and Scheduling

Instead of Cross Entropy Loss, which prioritizes global performance, we utilize Group Distributionally Robust Optimization (DRO) [12] to ensure that the model stays consistent by focusing more on low-performing subgroups. We define \(G=6\) subgroups: \(\{VP, VI, IP, II\}\) and two Symbolic groups. For multilingual tasks (ST3/ST4), each of the six subgroups is further divided into 12 subgroups, one for each language, resulting in a total of 72 subgroups. The model optimizes \(\mathcal{L}_{DRO} = \sum_{g=1}^{G} q_g \mathcal{L}_g\), where group weights \(q_g\) are updated adaptively: \[q_g^{(t+1)} \propto q_g^{(t)} \exp(\eta \mathcal{L}_g^{(t)})\] To ensure the model establishes a logical baseline before unbiasing, we implement Dynamic Scheduling: \(\lambda_{bias}\) and the DRO step-size \(\eta\) are initialized at 0 for the first epoch and incrementally increased. This prevents the optimization from collapsing into sub-optimal local minima where the model might ignore semantic content before mastering formal syllogistic structures.

4.3.0.2 Differentiable Bias Penalty

To align the training objective with the Content Effect (CE) evaluation metrics, we formulate a loss component using sigmoid confidence scores \(\hat{y}\). Let \(\hat{y}_{v,pl}\) be the model’s confidence scores for samples with validity \(v \in \{0, 1\}\) and plausibility \(pl \in \{0, 1\}\). We calculate: \[\begin{align} \bar{\hat{y}}_{Intra} &= \frac{1}{2} \sum_{p \in \{0, 1\}} | \text{mean}(\hat{y}_{1,p}) - \text{mean}(\hat{y}_{0,p}) | \\ \bar{\hat{y}}_{Cross} &= \frac{1}{2} \sum_{v \in \{0, 1\}} | \text{mean}(\hat{y}_{v,1}) - \text{mean}(\hat{y}_{v,0}) | \end{align}\] The final \(\mathcal{L}_{bias}\) is the mean of these effects. This approach allows the gradient to penalize uneven confidence distributions across quadrants, effectively "forcing" the model to maintain high logical certainty regardless of real-world plausibility.

4.3.0.3 Consistency via KL-Divergence

To ensure robustness to linguistic variation, we minimize the KL-Divergence (\(\mathcal{L}_{KL}\)) [18] between the output distributions of a simple syllogism (teacher) and its complex rephrasing (student). For example, mapping the simple form "All \(A\) are \(B\)" to the complex "If something is \(A\), then it is \(B\)". This \(\min \text{KL}(P_{simple} || P_{complex})\) optimization forces the model to align the semantic representation of complex conditionals with their fundamental logical forms, thereby preventing reliance on lexical shortcuts. This approach deliberately avoids converting complex syllogisms into a normalized abstract form, as such conversion would either require LLM-based paraphrasing, introducing probabilistic errors, or a deterministic rule-based normalizer that itself constitutes a non-trivial engineering overhead.

4.4 Data Splits and Preprocessing↩︎

We use a stratified 80/10/10 split (Train/Validation/Test) for both the training dataset provided by the curators [8], and our augmented dataset. Stratification is performed across to ensure balanced logical and semantic representation in every split.

4.5 Dataset Composition↩︎

Our augmentation pipeline expanded the fully balanced initial dataset (\(N=960\)) to include a more complete set of logical forms and variations: \(23,524\) total samples for ST1, \(27,988\) for ST2, 388,992 for ST3, and \(648,632\) for ST4. All subsets remain balanced across the four logic/plausibility quadrants and also across languages for multilingual subtasks.

4.6 Hyperparameters↩︎

We use the AdamW optimizer [19] with a linear schedule and a warmup ratio of 0.06. To account for the stability of the backbone, we apply a differential learning rate (\(\text{LR}_{backbone} = 0.1 \times \text{LR}_{head}\)). Specific subtask hyperparameter configurations are detailed in Appendix 9.

4.7 Implementation and Hardware↩︎

The system is implemented in PyTorch [20] using HuggingFace Transformers [21]. Training used NVIDIA Tesla P100 (16GB) and T4 Tensor Core (16GB) GPUs with Mixed Precision (FP16) and a Gradient Scaler.

5 Results and Analysis↩︎

5.1 Main Quantitative Findings↩︎

Table 2 provides the complete set of experimental configurations and ablation studies conducted across all four subtasks, with the best performing models per subtask highlighted. Our primary objective was to maximize the Ranking Score (\(S_{rank}\)) by minimizing the Total Content Effect (TCE) and maximizing the performance component \(P\). We additionally explored a couple of other experimental loss components, including Mean Squared Error (MSE) and FreeLB [23]; while these provided competitive stability, they were not included in the final system overview as they were not in our best results. Details on these experimental components can be found in Appendix 7.

As shown in the ST1 results, all configurations achieved perfect performance, suggesting that for ST1, the mDeBERTa-v3 backbone combined with our augmented dataset is sufficient.

However, the complexity of ST2 revealed performance gaps. Our final configuration using KL + DRO + Bias on the large model backbone achieved a perfect score of 100.0, successfully disentangling formal reasoning from noisy premises and content bias.

For ST3, using the bias alone produced near-perfect accuracy. Both Bias + HANS and Bias + HANS + DRO achieved a perfect score of 100.00. The addition of KL or MSE regularization failed to preserve this performance, suggesting that bias loss combined with HANS (with or without DRO) is sufficient to disentangle reasoning from content bias.

For the highly complex ST4, the Bias configuration achieved a Ranking Score of 37.78 during official evaluation. Notably, post-competition analysis revealed a drastic performance gap when evaluating on the original multilingual test set compared to its English translation generated via ChatGPT [22]. As shown in Table 2, the DRO + Bias model scored 25.50 on multilingual data but jumped to 40.35 on translated English data. A similar \(\approx 60\%\) rank increase occurs for the standalone Bias model. Interestingly, as we discuss further in Section 5.2, applying KL-Divergence consistency regularization significantly closes this cross-lingual gap.

5.2 Ablation and Error Analysis↩︎

For ST2, our experiments indicate that while the Vanilla model was able to achieve some success, it is susceptible to content effects (\(2.13\%\) TCE).

Surprisingly, adding only Differentiable Bias (40.13) or DRO + Differential Bias (36.42) degraded the performance compared to vanilla. DRO + Differential Bias showed the highest content effect (4.26%), which suggests that aggressive optimization can dominate the loss function and degrade performance. This finding underscores the importance of carefully designed multi-component systems rather than isolated regularization techniques. We found that an additional component, such as KL-Divergence or MSE, acted as a regularization anchor ensuring that even as the model is pushed to be unbiased, it remains grounded in the underlying formal syllogistic structure.

Our complete system (KL + DRO + Bias) achieved a 57.53 ranking score on the base model, representing a 25% improvement over vanilla. The FreeLB + MSE + DRO + Bias configuration achieved identical performance, indicating that both KL divergence and FreeLB provide similar consistency regularization benefits. However, FreeLB is computationally more expensive since it runs multiple forward passes per input.

Scaling the architecture from the Base to the Large model in the ST2 (KL + DRO + Bias) configuration yielded a perfect ranking score. This suggests that the capacity to abstract logical relationships from noisy premises scales directly with model parameters. Furthermore, while MSE and FreeLB provided stability during baseline experimentation, KL-Divergence consistency regularization proved more efficient and equally effective when generalizing to the Large backbone.

For ST3, the Vanilla model achieved a strong accuracy of 91.15%, but remained highly susceptible to content effect (8.29% TCE).

Introducing only the Differentiable Bias significantly improved the performance, accuracy increased to 98.96, while content effect dropped sharply to 1.04%, which suggests the effectiveness of bias-aware loss in mitigating the content effect. However, the most substantial jump occurred with the inclusion of the HANS dataset. The Bias + HANS configuration achieved a perfect 100.00 final score. Integrating DRO into this configuration maintained the perfect performance, indicating the model’s stability across different data distributions.

Interestingly, unlike the trends observed in ST2, additional consistency regularizers, KL-Divergence and MSE, failed to sustain the perfect score. Instead, they degraded the ranking scores and reintroduced the content effect. Both KL and MSE yielded identical results, suggesting that the model had already reached the optimal state and additional regularization may have introduced unnecessary constraints.

Furthermore, a critical takeaway from the ST3 results is the comparison between models trained without symbolic data and those trained with symbolic data. The models trained with symbolic data consistently outperformed the ones trained without it, suggesting that symbolic datasets contributed heavily to generalization. By training on abstract symbols rather than just real-world nouns, the model effectively disentangles logical reasoning from content bias.

For ST4, a pattern similar to ST2 emerges: DRO + Bias and Bias alone perform worse than Vanilla on multilingual test data (MT). To diagnose this, we translated the multilingual test set to English (ET) via GPT-5.2 [22]. The results reveal a significant cross-lingual gap: DRO + Bias scored 25.50 on MT but 40.35 on ET, and Bias alone scored 25.98 vs., a \(\approx 60\%\) rank increase, indicating that the core failure stems from (i) difficulty generalizing multilingual premise structures and/or (ii) “translationese” noise in mDeBERTa-v3’s pre-training data [24]. In contrast, the KL + DRO + Bias configuration achieved near-identical scores on both test sets (\(S_{rank}\) of 31.16 on MT vs. on ET), showing that KL-Divergence consistency regularization makes the model robust to cross-lingual variation by anchoring on simple English syllogisms and minimizing output divergence across all 12 language variants. However, this requires 24 forward passes per data point; due to compute constraints, training data was reduced to \(\approx 4\%\), suggesting further gains are achievable with more data.

6 Conclusion↩︎

We presented a system for disentangling formal logical reasoning from content effects in syllogistic argument classification. Our approach combines a rule-based synthetic data generation engine covering 256 Aristotelian syllogistic schemes across 12 languages with a multi-objective loss function integrating a scheduled differentiable bias penalty, Adaptive Group DRO, and KL-Divergence consistency regularization. The system achieved perfect Ranking Scores (100.0) with 0.00% bias on ST1, ST2, and ST3, confirming that bias-aware optimization with synthetic data can effectively isolate formal reasoning from real-world plausibility. On the most challenging subtask ST4, the system ranked 6th with 89.06% accuracy and 2.89% bias. The primary direction for future work is training the full KL + DRO + Bias pipeline on the complete ST4 dataset (648,632 samples), which was reduced to \(\approx\)​4% during the competition due to compute constraints, and which demonstrated the strongest cross-lingual robustness among all configurations.

7 Experimental Loss Components↩︎

As noted in Section 5.1, while our final winning configuration relied on KL-Divergence for consistency regularization and avoided adversarial perturbations to maintain computational efficiency on the Large backbone, our detailed ablation study (Table 2) reports metrics on two additional experimental loss components: Mean Squared Error (MSE) consistency and FreeLB (Free Large-Batch Adversarial Training).

7.1 Mean Squared Error (MSE) Consistency Regularization↩︎

Before adopting KL-Divergence, we experimented with Mean Squared Error (MSE) to enforce structural invariance. Under this configuration, every data point was structured to contain both a syllogism_simple and a syllogism_complex text variant. For the multilingual subtasks (ST3/ST4), this was expanded further such that a single data point could contain up to 24 language-specific simple and complex variations of the same underlying logical structure. During the forward pass, the model generated logits for both the simple constraint (\(z_{simple}\)) and the complex constraint (\(z_{complex}\)). The MSE consistency penalty was formulated as: \[\mathcal{L}_{MSE} = \frac{1}{N} \sum_{i=1}^{N} (z_{simple}^{(i)} - z_{complex}^{(i)})^2\] This loss component successfully anchored the model, ensuring that the confidence scores for semantically complex variations closely mapped to their fundamental, simple logical counterparts. While MSE provided competitive stability (Ranking Score 45.03 in ST2), we ultimately found that minimizing the KL-Divergence between the output probability distributions was more effective at preventing the model from relying on lexical shortcuts than minimizing the absolute difference between the raw logits.

7.2 Free Large-Batch Adversarial Training (FreeLB)↩︎

To further test the robustness of the model against grammatical noise and irrelevant premises, we implemented FreeLB [23]. FreeLB adds adversarial perturbations to word embeddings during training, forcing the model to make consistent predictions even when the input representations are subjected to small, continuous gradient-based attacks. We applied the standard FreeLB algorithm, executing multiple forward passes per input batch. In each inner step, we perturbed the embeddings by a constrained magnitude \(\epsilon\) and calculated the loss, updating the model parameters based on the accumulated gradients of the adversarially perturbed inputs. As shown in Table 2, the combination of FreeLB + MSE + DRO + Bias achieved a Ranking Score of 57.53 on the Base model, which was identical to the performance of our KL + DRO + Bias configuration. FreeLB successfully stabilized the aggressively debiased model. However, because FreeLB requires multiple forward passes per iteration, it is computationally expensive. When transitioning our architecture to the Large mDeBERTa-v3 backbone to maximize the Ranking Score, we dropped FreeLB in favor of the computationally lighter KL-Divergence approach, which yielded the same consistency benefits without multiplying the training time.

8 Extended Data Generation Procedure↩︎

As described in Section 3.2, our synthetic dataset was generated programmatically. To isolate the effects of structural noise, linguistic complexity, and consistency pairing, the data was generated via independent script executions rather than a monolithic pipeline.

8.1 Base Generation↩︎

The baseline dataset was generated using a 256-rule Aristotelian logic engine. This initial procedure outputs ‘Simple’ English syllogisms encompassing non-symbolic nouns, symbolic variables, and both 3-variable and 4-variable logical constraints. All subsequent augmentation scripts act upon this foundational subset.

8.2 Irrelevant Premise Injection↩︎

For robustness configurations (e.g., ST2/ST4), modifier scripts were executed over the base English data to inject structural noise:

1. Premise Removal: The modifier script discarded a structurally critical premise from a designated subset of valid chains, directly invalidating the logic.

2. Distractor Generation: The sequence then generated logically disjoint distractor premises (e.g., inserting a premise regarding "clouds" into a syllogism concerning "dogs") and concatenated them to the input text.

8.3 Paired Semantic Augmentation↩︎

Consistency optimization (e.g., KL-Divergence or MSE) requires identically paired "simple" and "complex" variants of underlying logical forms. For English datasets, a secondary complex-rephrasing script was applied to the base output, applying synonym replacement and structural permutations while maintaining parity with the corresponding simple input matrices.

8.4 Multilingual Translation↩︎

For the 11 non-English language tracks (ST3/ST4), translation followed two independent pathways depending on the required complexity constraint:

1. Simple Translation: The complete English input text (with or without injected noise) was translated into the target languages via the DeepL API.

2. Complex Translation: To execute complex rephrasing natively across languages, a specialized script isolated and translated only the semantic nouns from the English text. These localized noun-sets were subsequently passed to 11 language-specific generator scripts (e.g., ComplexSyllogisms_French.ipynb), which applied complex phrasing mappings independently. Because symbolic datasets lack semantic noun dependencies, they bypassed this translation step entirely.

9 Hyperparameter Configurations↩︎

9.1 Subtask 1 (ST1) Configurations↩︎

9.2 Subtask 2 (ST2) Configurations↩︎

9.3 Subtask 3 (ST3) Configurations↩︎

9.4 Subtask 4 (ST4) Configurations↩︎

References↩︎

---

1. Equal contribution.↩︎

2. English (en), German (de), Spanish (es), French (fr), Italian (it), Dutch (nl), Portuguese (pt), Russian (ru), Chinese (zh), Swahili (sw), Bengali (bn), and Telugu (te).↩︎

3. https://github.com/TahaZahid05/SemEval26-Task-11↩︎

4. All checkpoints are available on HuggingFace under the MoritzLaurer namespace, except for ST4 which uses a Microsoft checkpoint.↩︎

5. https://www.deepl.com/↩︎