1 Introduction↩︎

A logical fallacy is a common thinking error, especially one apt to mislead [1]. These arguments often appear rational and logically coherent on the surface, but deeper analysis reveals they are not [2]. Fallacies are traditionally classified into formal and informal types: formal fallacies violate the rules of logical structure regardless of content, while informal fallacies are patterns of mistakes that are made in the everyday uses of language and are related to contextual meaning [3], [4].

To evaluate the quality of an argument, it is helpful to reconstruct it into what is known as logical form, the structure that emerges when the specific content of a statement is replaced by variables [5]. For example, the argument If it rains, then the ground will be wet. It is raining. Therefore, the ground is wet has the logical form If P, then Q. P. Therefore, Q. Building on this formalization framework, [6] developed a structure-aware model for fallacy detection on the Logic dataset that compares arguments’ and fallacies’ logical forms. However, in their approach a single logical form is assigned to each fallacy, which might fail to capture the full spectrum of ways a fallacy can manifest in natural discourse. Another challenge is related to informal fallacies, where reasoning is often more nuanced and context-dependent than abstract representations suggest.

These limitations motivate the need to go beyond purely abstract representations, incorporating linguistic elements, such as lexical markers or rhetorical devices, to provide a more comprehensive characterization of how fallacies manifest in natural language. We argue that LLMs can inductively extract such representations from fallacious examples, capturing both the logical structure and linguistic cues that reveal the underlying mechanisms of deception. Unlike prior work that formalized fallacy logic through hand-crafted templates [7], our approach is data-driven and not restricted to a limited number of fallacies. Hereafter, we refer to these extracted structures collectively as structural patterns. Our goal is to investigate whether context-aware structural information is valuable for automated fallacy detection.

While existing supervised approaches require heavy computational resources for fine-tuning [8]–[12], to our knowledge, no prior work has explored fallacy classification from a structural perspective without any additional fine-tuning. Although we use labeled data for pattern extraction (resulting in a weakly supervised approach), our framework avoids fine-tuning costs and produces generalizable patterns that allow classification through prompting alone, enabling comparison with unsupervised methods.

We evaluate multiple prompting configurations to determine which components enhance performance and examine the impact of demonstrations on detection capabilities. Our approach, incorporating generated patterns, achieves noteworthy results among unsupervised methods on the dataset Logic. Finally, to validate the robustness and transferability of our patterns, we assess their performance across two further datasets spanning diverse domains and argumentative styles.

In summary, our contributions are threefold:

• We leverage LLMs to automatically extract patterns from fallacious examples and their explanations, which are then employed in inference-only classification.

• We evaluate different LLMs with various prompt designs outperforming competing approaches on the LOGIC dataset.

• We validate generalizability by testing our patterns on two different datasets with different domains and structures.

2 Related Work↩︎

Recent advances in fallacy detection have increasingly turned to LLMs, though few studies have relied exclusively on prompting-based techniques. Several works have employed fallacy detection to probe LLMs’ logical reasoning abilities [13]–[16]. Among these, [14] investigated self-verification capabilities and showed that LLMs face more challenges with structure-based (formal) fallacies with respect to content-based (informal) ones, and that fallacy definitions provide minimal improvements. [16] has shown that reasoning models have better performances with respect to non-reasoning ones for fallacy classification. Among studies relying exclusively on prompting techniques, [17] designed single-round and multi-round prompting schemes for zero-shot detection, while [18] introduced contextual prompting incorporating counterarguments, explanations, and goals with confidence-based ranking, showing that explanations particularly enhance performance. [19] assessed detection abilities on the Logic dataset using few-shot prompting, though their different taxonomy limits direct comparison with our work. Other research has examined the logical structure of argumentation. Most notably, [6] developed a structure-aware model based on Electra that distills arguments into logical forms and compares them against fallacy patterns sourced from logicallyfallacious.com. Another prominent framework in this field is Walton’s theory of argumentation [20], consisting in about 60 templates that capture common argument types, each associated with a set of critical questions to evaluate their validity. These schemes have been adopted in computational approaches for disinformation [21], misinformation [22] and fake news [23] detection. Regarding fallacies specifically, [24] introduced a dataset of sentences grounded in Walton’s argumentation schemes, labeling them as fallacies when the associated critical questions could not be successfully answered. However, Walton’s schemes do not hold a one-to-one correspondence with fallacies and cover only a limited number of them, limiting their direct applicability to fallacy classification. Of particular relevance is the work of [7], who re-designed four of [20] and [25]‘s schemes with the goal of expliciting fallacies’ implicit logic, introducing formal logical schemas with explicit variables and relationships. While this represents a meaningful step toward structural formalization, their patterns are manually designed and leave a large portion of fallacy types unrepresented. Our patterns, by contrast, are extracted automatically from data, allowing to adapt to intra-class variation.

3 Datasets↩︎

The Logic dataset is a collection of 2,449 examples across 13 fallacy types [6]. Instances are sourced from educational platforms about fallacies such as Quizziz and study.com. The dataset consists of brief dialogues and short statements. Given the educational intent behind these examples, sentences tend to have relatively straightforward syntactic structures, making the dataset particularly well-suited for the extraction of the patterns.

Although it contains 13 distinct classes, a thorough analysis revealed that some of the classes actually contain instances of different fallacies, that were grouped together. For instance, the class Hasty Generalization contains examples of actual Hasty Generalization as well as Slippery Slope (Table 1). While these grouped fallacies share common logical flaws and thus belong to the same conceptual group, they manifest through different structural patterns.

We experiment on two further datasets: Reddit [26], consisting of fallacious comments extracted from subreddits covering different topics and ElecDebate60to16 (hereafter ElecDebate) [27], a collection of televised debates of the presidential election campaigns in the U.S. from 1960 to 2016. Some fallacy classes contain sub-categories. In Table 2, we report a summary of the dataset, and a description of each taxonomy is provided in Appendix 11.

4 Pattern Generation↩︎

Natural arguments appear in several forms. Such variability manifests itself in Logic dataset as well as many others [28], [29]. For this reason, we address our research question by modeling patterns inductively from the training set of Logic. The choice of the dataset for pattern extraction is critical. It provides the required combination of structural clarity and fallacy diversity through its multiple sub-types per class. These properties make it especially suited for our purpose. The clean argumentative structure allows to formalize clear logical patterns while capturing intra-class variations.

Our pattern generation procedure features two steps:

Step 1: Explanation Generation Explanations have been shown to be instrumental in identifying and discrediting fallacious reasoning, as they make the logical structure of arguments explicit and open to scrutiny [30]. Furthermore, [18] has demonstrated that providing explanations constitutes valuable contextual information in zero-shot settings. We expected explanations to facilitate pattern extraction by breaking down the reasoning process and revealing shared reasoning flaws, particularly useful for informal fallacies.

Given a sentence from the training set and its fallacy label, we used llama-3.3-70B-Instruct [31] to generate an explanation that justifies why that sentence contains the specified fallacy.

Step 2: Pattern Extraction For each fallacy class, we used OpenAI’s reasoning model o4-mini [32] to extract patterns from all the sentences of that class and their explanations, requiring the model to preserve function words such as prepositions or adverbs and to abstract away from content words by replacing them with placeholders while keeping the original reasoning form. Additionally, summaries were extracted to derive new fallacy definitions.

We opted for llama-3.3-70B-Instruct for explanation generation as it provided high-quality explanations while remaining cost-effective for large-scale text generation. For pattern extraction, we employed o4-mini given its reasoning capabilities. The prompts used in our experiments are reported in the Github repository.¹

In the initial phase of our research, we aimed to cover two distinct logical aspects from our arguments and explanations, specific to formal and informal fallacies, respectively:

• arguments’ logical structure inspired by formal logic theory;

• recurring lexical schemes that frequently appear in both sentences and explanations, capturing specific information about the reasoning behind the fallacy as well as frequent syntactic particles, phrases, and examples that convey the fallacious intent.

Our patterns incorporate both of these aspects, as Table 3 shows. These logic-based patterns combine reasoning structure (variables X, Y, Z) with concrete linguistic features (specific phrases, loaded terms, rhetorical devices), occasionally retaining some definitions. The full list of patterns is available in the repository. The process resulted in approximately 3-6 patterns per fallacy class. Final patterns were obtained after providing different subsets to the model and selecting the best performing one on the validation set, in the attempt to retain only useful information and avoid redundancy. In section 3 we discussed how one class in the datasets could correspond to multiple fallacies. Although in some cases, e.g. a pattern for Tu quoque (a fallacy which is part of the class Ad Hominem in Logic), is correctly generated and selected, sometimes fails to select patterns when multiple fallacies are grouped under the same class label. This is expected because we include the fallacy class name in the prompt, which likely biases the model toward patterns that match its internal knowledge of that particular class name. To ensure a broader coverage of fallacies listed in Table 1, we manually isolated instances of frequent and undetected fallacies (such as Shifting the Burden of Proof) and repeated the procedure.

5 Experiments↩︎

This section describes our experiments for fallacy classification, including our patterns extracted by the procedure introduced in Section 4 and several competing prompting strategies. Additional experiments are reported in Appendix 13. We used the following LLMs for our experiments: gpt-4o, o4-mini, gpt-4.1-mini, LLama-3.3-70B, deepseek-r1 and Gemma-3-27B-it for a total cost of 75 USD. Our intent was to test LLMs from different providers and with different sizes and to compare reasoning and non-reasoning models.

5.1 Prompt Design↩︎

5.1.0.1 Baselines.

We compared our approach against several baselines that vary in the type and amount of information provided to the model. The simplest baseline (zero-shot) provides only the list of fallacy names in the dataset as a reference, establishing a minimal information condition. Our second baseline incorporates fallacy definitions to provide more comprehensive background knowledge (def). These definitions were initially sourced from [9] and subsequently refined based on our analysis to ensure clarity and consistency. Finally, we tested a baseline using standard logical forms, following the approach of [6] and sourcing these forms from logicallyfallacious.com. This final baseline (logical forms) allows us to assess the effectiveness of expert-made logical representations compared to our generated pattern-based approach.

5.1.0.2 LLM-derived Patterns and Definitions.

Beyond generating structural patterns, we leveraged the explanations from Section 4 to automatically create new fallacy definitions based on Logic training samples. We then replicated experiment def with these new definitions (new def). We also exploited the patterns extracted by adding them to the prompt (patterns) and by implementing a two-step approach where we first ask the LLM to identify the pattern and then to output the corresponding fallacy (pattern matching).

5.1.0.3 One-shot Prompting.

We further investigated the impact of providing examples to the model through several experimental configurations [33], with one-shot prompting proving most effective. Initially, we tested a static approach where one example per fallacy was randomly selected and shown to all test sentences (one-shot), establishing a baseline for example-based learning. To enhance this approach, we augmented the same examples with manually crafted explanations following our previously established definitions as guidelines (one-shot + exp). We sampled 5 different example sets and performance across all configurations was assessed over 5 runs to ensure statistical reliability.

More sophisticated was our dynamic one-shot prompting approach (dynamic one-shot), which computes embeddings for both training and test sentences to retrieve, for each test sentence, the most similar example per class in the training set. We used sentence-transformers/all-MiniLM-L6-v2² model and cross-encoder/stsb-roberta-base³ cross-encoder from SentenceTransformers [34] to compute embeddings and employed cosine similarity to evaluate similarity. We included the previously generated explanations of examples in the prompt as well (dynamic + exp).

Furthermore, we explored structure-focused similarity. Since [6] released a version of Logic with masked arguments (with content words replaced by placeholders), we conducted the same similarity-based procedure using these masked sentences (see an example in Table 4) in the attempt to force the embedding model to focus on structural rather than lexical similarities. For this configuration (syntax-based dynamic one-shot), we used sentence-transformers/all-MiniLM-L6-v2 from SentenceTransformers alongside a syntax-augmented version of RoBERTa-large extracted from [35] (see Appendix 14).

Finally, we incorporated the generated patterns into our dynamically retrieved examples and their explanations (dynamic + exp + patterns).

5.1.0.4 Multi-step Classification.

An alternative approach decomposes the classification task into three sequential steps within a single model call (multistep) using chain-of-thought prompting [36]. In the first step, the model is required to generate a structural pattern from the argument according to predefined structural rules. Subsequently, the model should match it to one of the patterns and, as a result, classify the argument.

5.2 Results and discussion↩︎

Table 5 summarizes all experimental configurations and results on Logic. It reveals a consistent improvement when the model leverages information about the underlying logic extracted through the LLMs, especially with reasoning models and gpt-4o. When using reasoning models, the model-generated definitions yield a 4.65% accuracy improvement over our manually corrected definitions. In the same way, including our generated patterns causes a 8.2% increase with respect to the logical forms extracted by the website logicallyfallacious.com and used in [6]. McNemar’s test proved statistical significance for all models using patterns against zero-shot and for all except llama and deepseek against logical forms method. When it comes to non-reasoning models, the new definitions do not really affect the performance, whereas using our patterns improves the accuracy by 5.8% on average. For comparison, we test our method against [7]’s templates: our patterns outperform said schemes by an average 10.7% across all models.

A notable result is the performance increase achieved through dynamic one-shot prompting. In particular, dynamic one-shot approach (using all-MiniLM-L6-v2) yields an average \(8.87\%\) increase in accuracy compared to one-shot, despite relying on semantic similarity for example selection. On the other hand, the syntax-oriented example retrieval strategy (syntax-based dynamic one-shot) does not outperform the semantic selection. This may be partially due to inaccuracies in the sentence masking process, which can negatively impact the retrieval of similar examples and the classification, consequently. The multistep approach shows weaker performance than pattern matching, especially for deepseek-r1 [37], implying that generating logical forms without explicit guidance constitutes the main challenge for the model in the request.

In summary, including context-aware logical patterns proves consistently beneficial for fallacy classification: patterns with gpt-4o reaches 73.5% accuracy, outperforming prior unsupervised methods (Table 6), while dynamic+exp+patterns with o4-mini achieves 74.2% when augmented with examples and patterns.

5.3 Error analysis↩︎

5.3.0.1 Pattern matching

Requesting the model to identify the closest pattern for each argument provides insight into the association process between sentences and patterns. For our analysis, we have split our fallacies into two groups in Table 7: (i) group 1, consisting of fallacies whose patterns include logical forms while still including additional contextual cues; (ii) group 2, consisting of fallacies that lack highly structured patterns and rely more on contextual and semantic features of the sentence.

Figure 1 shows consistently superior accuracy for Group 1, whose classes maintain relatively high performance across all experimental settings. The class Circular Reasoning emerges as the most accurately predicted class across all models. For what concerns Group 2, the overall accuracy is, on average, 22% lower with respect to Group 1. The classes Emotional Language, Red Herring and Extension Fallacy achieve moderate prediction accuracy, whereas only Evading the Burden of Proof’s patterns within the Intentional Fallacy category are correctly classified, and Equivocation remains entirely undetected by gpt-4.1-mini [38]. In summary, the models achieve better performance on logical fallacies that exhibit clearer structural characteristics but face difficulties with fallacies requiring more nuanced semantic understanding and contextual analysis.

Furthermore, matching patterns allows us to see that some instances can be deemed as fitting from a structural point of view, thus partially explaining the inherent difficulty of the classification task. While providing guidance through syntactic and logical structure proves beneficial for fallacy detection, this approach does not eliminate all sources of ambiguity, as some sentences may conform to multiple structural patterns. The critical point lies in context-aware pattern application. Models must not only identify logical forms but also evaluate their contextual validity in each sentence.

To quantify the degree of ambiguity inherent in pattern matching, we instructed the best-performing model o4-mini to return the five most similar patterns for each argument. This multi-candidate approach enables us to analyze whether lower-ranked patterns might also represent valid interpretations of the same argument. By examining the distribution of pattern similarities and evaluating classification accuracy when considering alternative matches, we can better understand the boundaries of pattern-based classification and identify instances where structural ambiguity genuinely complicates fallacy detection.

Table 8 shows that, when the model is prompted to return multiple matching patterns rather than a single best match, its confidence in the initial prediction decreases, resulting in a 3.4% drop in accuracy (see Table 5).

However, this apparent degradation is misleading when viewed in isolation. By incorporating the second-ranked pattern choice into our evaluation, performance recovers to \(75.1\%\), and continues to improve as we expand our candidate pool to include progressively lower-ranked options. Table ¿tbl:tab:example-correction-top2? illustrates a representative case where the model successfully identifies the correct pattern as its second choice, while its first-ranked selection remains structurally plausible. The model likely assigns one of the Ad Populum patterns because it closely matches the argument’s logic, while the Irrelevant Authority pattern does not fit the sentence since it requires discussion of an unrelated topic, which is not present in the sentence. These subtle distinctions likely make pattern matching more challenging than direct classification because it requires strict structural alignment as well as capturing broader content-related features.

5.3.0.2 Multistep classification.

The multistep approach fails to produce significant results. We conduct this experiment in a single passage to force the model to reason using both semantic and syntactic information. However, classification performance depends critically on the quality of the extracted logical forms, which proves inconsistent and model-dependent. For instance, o4-mini embeds classification-relevant contextual information directly into its generated logical forms (Table [tab:multi-step-output]). Furthermore, models demonstrate substantially weaker performance on Group 2 sentences compared to Group 1, showing an average decrease of 21.5% in F₁ score. Additionally, models frequently bypass the pattern matching phase entirely, arbitrarily assigning patterns despite clear misalignment with the extracted logical form. For example, given the argument People nowadays only vote with their emotions instead of their brains (an instance of Hasty Generalization), the model o4-mini first extracts the logical form All A only do B instead of C. The model then matches this form to the pattern Generalizing from a small sample or single event to an entire group or population, which correctly belongs to Hasty Generalization. While this produces an accurate classification, the assigned pattern does not precisely correspond to the extracted logical form. In summary, while humans naturally decompose pattern matching into multiple cognitive steps, this multi-stage process proves to be challenging for current LLMs. Models struggle to bridge the gap between abstract logical patterns and their content-dependent manifestations, often failing to identify the implicit premises and unstated logical connections that underlie the reasoning chain.

6 Experiments on Further Datasets↩︎

In order to further assess the quality of Logic-derived patterns, we conducted a subset of the experiments on Reddit and ElecDebate using the best performing model, o4-mini. We tested patterns extracted from Logic, restricted to the two datasets’ classes (first eight rows in Table 9) and patterns extracted from the datasets themselves (latest two rows in Table 9). Reddit patterns show a prevalence of linguistic markers over logical forms while ElecDebate ones emphasize stronger logical formalization, incorporating symbolic formalism.

Consistent with previous findings, logical pattern incorporation outperforms competing approaches on Reddit. Moreover, Logic-based and Reddit-based patterns yield comparable results. While taxonomy alignment prevents direct comparison, results from supervised and unsupervised methods [9], [17], [26], [39] are consistent with our findings (see Appendix 12). Only a comparison with [9] (Macro F₁=81.3%) is possible: patterns and same-dataset patterns outperform their results.

Regarding ElecDebate, Table 9 shows that dynamic one-shot yields the best performance, possibly due to the predominant presence of the class Emotional Language (62.5% of test set) whose detection may particularly benefit from similar worded examples. Indeed, same-dataset patterns achieve competitive results with respect to [17], [27] (see Appendix 12). These experiments showed a fair generalization of Logic-derived patterns on other datasets, with the additional advantage of not requiring labeled data to re-extract the patterns.

In order to prove the broader applicability of our approach beyond Logic-specific patterns, we tested patterns generated from the other two datasets on Logic.

Table 10 demonstrates solid results, validating findings on Logic and proving the robustness and transferability of our pattern-based methodology. Notice that accuracy is not directly comparable with values in Table 5 since Reddit and ElecDebate have a subset of the classes of Logic.

7 Conclusions↩︎

Fallacy detection is a challenging yet critical task to solve. Since fallacies often manifest in nuanced and context-dependent forms, purely abstract representations are insufficient to characterize the full spectrum of ways a fallacy can appear in natural language, thus motivating the need to combine logical structure with context-level linguistic cues. We present an experimental framework that inductively extracts context-aware structural patterns from fallacious arguments and their explanations, demonstrating that incorporating such patterns significantly enhances fallacy classification performance. Specifically, pattern-based classification achieves 73.5% accuracy on Logic, significantly outperforming prior unsupervised approaches, and 74.2% including one-shot examples. Being data-driven, these patterns are not bound to a fixed set of fallacies and can flexibly capture the diverse nuances through which each fallacy type manifests. Notably, reasoning models demonstrate consistently superior performance across all experimental configurations. Moreover, experiments on additional datasets confirm that the extracted patterns generalize effectively across domains, establishing data-driven pattern extraction as an effective method to generate generalizable logical representations.

8 Limitations↩︎

While this work demonstrates the efficacy of large language models in detecting logical fallacies by exploiting the underlying logical structure of sentences, it has several limitations. First, we intentionally generated patterns exclusively from the Logic dataset due to the quality and straightforward structure of its sentences. We are aware, however, that it does not fully cover the complex and multi-faceted spectrum of fallacies. Furthermore, our work is based on a small sample of LLMs. Nevertheless, we selected a diverse and representative subset, including models from different providers, with varying sizes and reasoning capabilities.

9 Ethics Statement↩︎

Logical fallacies can reinforce societal bias and facilitate the spread of misinformation, leading to harmful consequences for society. This work focuses on leveraging LLMs for detecting logical fallacies in argumentation and should not be employed to manipulate discourse by exploiting identified reasoning patterns. Furthermore, this approach risks amplifying existing LLM biases, potentially causing unfair detection. We acknowledge these limitations and encourage future bias mitigation research. We are aware of the environmental impact of large-scale LLMs usage. However, this study exclusively employs inference-only methods, significantly reducing computational requirements compared to training approaches. All datasets are used in accordance with their license and they have been checked for personally identifying and offensive content.

Acknowledgements↩︎

This publication is part of the project PNRR-NGEU, which has received funding from the MUR - DM 629/2024. We would like to thank the Qatar National Research Fund, part of Qatar Research Development and Innovation Council (QRDI), for also funding this work by grant NPRP14C0916-210015.

10 Implementation Details↩︎

In experiments where the task consisted of returning only the fallacy label, we set the temperature to 0, with the exception of o4-mini, gpt-4o and deepseek-r1. In all other experiments, the standard configuration was kept. Multiple prompt configurations were evaluated for each approach.

11 Fallacy Datasets↩︎

11.1 Logic↩︎

The dataset Logic [6] contains the following 13 fallacy classes: Faulty Generalization (Hasty Generalization), Ad Hominem, Ad Populum, Circular Claim (Circular Reasoning), False Cause (False Causality), Appeal to Emotion (Emotional Language), Fallacy of Relevance (Red Herring), Deductive Fallacy, Intentional Fallacy, Fallacy of Extension (Extension Fallacy), False Dilemma (Black-and-White Fallacy), Fallacy of Credibility (Irrelevant Authority) and Equivocation. The names in the parentheses are the actual names used in our experiments.

11.2 Reddit↩︎

The dataset Reddit [26] contains 8 fallacy classes: Appeal to Authority (Irrelevant Authority), Appeal to Majority (Ad Populum), Appeal to Nature, Appeal to Tradition, Appeal to Worse Problems, Black-and-White fallacy, Hasty Generalization and Slippery Slope. It contains the class No Fallacy as well. The names in parentheses are the actual labels used. In our experiments, only the classes included in Logic are retained (Table 11). We can keep the class Slippery Slope because two generated patterns for Hasty Generalization correspond to it.

11.3 ElecDebate↩︎

The dataset ElecDebate [27] contains the following 6 fallacy classes: Ad Hominem, Appeal to Emotion (Emotional Language), Appeal to Authority (Irrelevant Authority), Slippery Slope, False Cause and Slogan. The names in parentheses are the actual labels used. In our experiments, only the classes included in Logic are retained (Table 11).

12 Baselines↩︎

We consider only the classes of Reddit and ElecDebate in common to logic. For this reason, direct comparison with prior work is generally not possible. However, for reddit, [9] provide classwise F₁ scores, allowing us to compute Macro F₁ and compare our results. Tables 12 and 13 present the comparison with prior work for both datasets.

3pt

3pt

13 Additional Experiments↩︎

We are going to report some other experimental set-ups that have been explored, including some basic baselines that we have not included in Section 5.

13.1 Prompt design↩︎

• exp: to investigate whether explicit reasoning improves performance, we implemented a baseline that not only provides fallacy names but also requests the model to generate a two-sentence explanation for its classification decision, testing whether forcing the model to articulate its reasoning leads to better outcomes. The two-sentence constraint was intentionally designed to keep explanations concise and manageable for manual inspection of explanations.

• guidelines: to leverage the model’s classification errors for improvement, we develop guidelines derived from observed mistakes. We conduct pattern matching evaluation on the validation set and collect misclassified instances. For each class, we provide the model with incorrectly classified examples and prompt it to generate comprehensive detection guidelines (as can be seen from table 14), given our generated pattern as a reference. These guidelines are then adopted to evaluate the test set. Notably, only guidelines produced by o4-mini and partially by gpt-4.1-mini incorporate a little structural and logical information such as common connectors or logical forms while the majority of guidelines content across models focuses primarily on semantic characteristics rather than structural patterns.

13.2 Results↩︎

exp’s (Table 15) results show that requesting the model to articulate the reasoning does not really cause any improvement. Specifically, certain classes such as Intentional Fallacy and Extension Fallacy exhibit extremely low F₁ scores under the non-reasoning models (0.027 and 0.13 respectively on average), indicating performance deterioration compared to the zero-shot baseline. This proves that models process surface-level semantic patterns without being able to access the multi-layered intentional structures behind reasoning (Table 16).

Including guidelines yields only modest results. While these guidelines are designed to provide comprehensive fallacy knowledge, they appear to lack the appropriate type of information from which models can benefit. Indeed, providing explicit information about the underlying logical structure proves significantly more beneficial for model performance.

14 Syntax-augmented roBERTa↩︎

[35] introduces a syntax-augmented model that incorporates dependency tree information into pre-trained BERT-based [40] transformers through specialized Graph Neural Networks (GNNs) [41] that process dependency trees. The authors introduce two distinct fusion strategies to integrate syntactic structure into BERT representation. We adopted specifically roBERTa-large [42] in the attempt to perform a syntax-driven examples selection. Further details about the implementation are available in [35].

References↩︎

---

1. https://github.com/elenipapadopulos/fallacy-patterns↩︎

2. https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2↩︎

3. https://huggingface.co/cross-encoder/stsb-roberta-base↩︎