1 Introduction↩︎

Argumentation is fundamental to reasoning in various fields [1]: scientists publish discoveries together with supporting evidence, lawyers structure legal arguments to solve disputes, and political debaters rely on argumentation to gain the approval of the public. Early computational work focused on mining and evaluating arguments in well-structured texts such as essays [2] until more research shifted toward assessing argument quality [3]. A particularly effective approach within the field has been the use of pairwise comparisons, in which the annotators judge which of the two arguments is stronger based on specific criteria. These judgments can then be aggregated using statistical models, such as the Bradley-Terry (BT) model, to produce quality scores  [4] and obtain a ranking of evaluated arguments. This design reduces annotation cost and variance while providing reliable multidimensional quality scores for benchmarking automated evaluation methods. As online platforms grow, the ability to identify high-quality arguments becomes increasingly important  [5], especially for effective persuasion, decision-making, and participation in meaningful discourse. However, manually evaluating argument quality at scale remains a major bottleneck due to cost and inconsistency. Large Language Models (LLMs) offer a promising avenue for scaling argument quality assessment. LLMs demonstrated impressive capabilities in language understanding tasks, and a growing line of work explores the use of LLMs as annotators [6]—an example of LLM-as-a-judge—where the LLM is asked to evaluate text quality [7]. This strategy could enable large-scale assessments with minimal cost and time. However, whether LLMs can reliably replicate expert-level judgments is an open question. Existing research offers mixed results: some studies show that LLMs, such as GPT-3 and PaLM-2 can effectively assess certain quality aspects [6], while others found that GPT-3.5 underperformed specialized supervised models on tasks that involve ranking of arguments [8]. These contrasting findings motivate a deeper investigation into LLM-based argument quality evaluation. This study attempts to replicate expert-derived argument quality rankings from the Webis-ArgQuality-20 dataset [4]. Given that argument quality is multidimensional [3], this study will also examine the abilities of the LLMs in assessing the different quality dimensions. We aim to address the following research question (RQ):

RQ: How well can LLM pairwise comparisons reproduce expert rankings of argument quality?

We break it down through the following sub-research questions (SRQ):

1. SRQ1: How do LLMs of different sizes and families, when using various prompting strategies, compare in their ability to evaluate arguments on quality dimensions compared against human experts?

2. SRQ2: To what extent do LLMs of varying sizes and model families exhibit similar patterns of argument evaluation?

3. SRQ3: How consistent are LLM-based evaluations when repeated over multiple trials?

Our contribution is an extensive evaluation of 12 open-weight LLMs from five model families (7B–104B parameters) under zero-shot, few-shot, and chain-of-thought for pairwise argument assessment across logical, rhetorical, and dialectical dimensions. Using LLM pairwise judgments, we derive Bradley-Terry rankings, being, to the best of our knowledge, the first application of Bradley-Terry ranking to LLM-based comparative judgments in argument mining. Leveraging this contribution, we derive three important insights: (i) Our results show that LLM-derived BT rankings achieve moderate correlation with human judgments (up to 0.327–0.477 across Kendall, Pearson, and Spearman), with moderate expert-level pairwise agreement (up to Cohen’s \(\kappa\)=0.493), highlighting both the promise and current limitations of LLMs as evaluators of argument quality. (ii) Other LLMs show weak-to-high alignment with the best performing model, Llama-70B, while matching human-expert performance, indicating complementary understanding of quality dimensions across models. (iii) LLM predictions are stable, with label differences in at most 7.75% across three runs. Few-shot prompting further improves stability for larger models, whereas smaller models are inherently more stable but less accurate ¹.

2 Related Work↩︎

Argument quality assessment is relevant, as people tend to seek the strongest arguments to form opinions, write persuasive texts, foster agreement, and improve mutual understanding when presenting ideas to an audience [5]. To address this need, several frameworks conceptualize argument quality as a multidimensional construct. For example, the taxonomy proposed by [3] distinguishes logical, rhetorical, and dialectical dimensions of argument quality, providing a strong theoretical foundation for computational assessment. Complementing this perspective, [9], [10] emphasize deliberative norms that focus on the collaborative and constructive aspects of discourse,  [11], [12] focus on argument convincingness, and [13] operationalize argument quality through a recommendation criterion, namely whether an argument would be suitable for use, as-is, in a supporting or opposing speech.

Using these theoretical frameworks, several datasets for argument quality assessment have been proposed that vary in number of topics and arguments, ranking strategies, evaluation dimensions, research communities, number of annotators, and annotation methods [14]. Two main approaches have been adopted for evaluation: Likert or continuous scale judgments of individual arguments [3], [15]–[17] and, pairwise comparison setups in which annotators select the better of two arguments with respect to a given dimension [4], [11], [12], [18]. Pairwise comparisons avoid the problems of absolute ratings by not requiring assessors to assign fixed labels that are often treated as interval-scale, and instead asking them to simply choose between two items, which is easier, more consistent, and produces more reliable data [4], [19]. Subsequent scores are later obtained using a ranking function such as PageRank [20] (or variants of it [11]), Gaussian Process [21], or Bradley-Terry [22], [23].

As pairwise comparisons scale rapidly, recent studies have explored using LLMs to assess argument quality. [6], [8], [24] investigated agreement between LLMs and human annotators using models like GPT-3, GPT-3.5, and PaLM-2. [25] examined LLMs’ ability to assess argument convincingness, [26] studied the role of rationales in pairwise quality judgments, [27] proposed a multi-persona framework based on Llama models, and [28] used LLM-based ranking with medical arguments. Still, these works evaluate only a small set of LLMs, rely on Likert-scale, or focus on convincingness rather than multidimensional argument quality. In contrast, we systematically compare multiple LLM families and sizes across prompts using the broader theoretical framework of [3], operationalized through the Webis-ArgQuality-20 dataset. We assess argument quality via pairwise comparisons and aggregate judgments with the Bradley-Terry model, a well-established and computationally efficient approach [22], [23].

3 Methodology↩︎

The overall methodology consists of performing comparisons of argument pairs using zero-shot, few-shot, and chain-of-thought. Arguments are aggregated per topic (only topic-related arguments are compared), and the comparison is performed across three dimensions of argument quality.

3.1 Argument Quality Dimensions↩︎

Theoretical work in argumentation distinguishes three dimensions: logical, rhetorical, and dialectical [3], [29]. Logical quality, or cogency checks whether premises are relevant and sufficient to conclude. [30] defines logical quality as: local acceptability (are the premises worthy of acceptance?), local relevance (do the premises support or attack the conclusion?), and local sufficiency (are the premises enough to justify the conclusion?). Rhetorical quality or effectiveness measures whether the argument can persuade a target audience. Inspired by Aristotle’s rhetorical appeals, [3] further decomposes this dimension into: credibility (establishing the speaker’s authority), emotional appeal (engaging the audience), clarity (using correct and unambiguous language), appropriateness (aligning the tone and language with the topic), and arrangement (structuring the argument). Dialectic quality, or reasonableness measures the ability of the argument to help resolve a disagreement. It is further decomposed into acceptability (the audience’s acceptance of the argument), relevance (the ability of the argument to advance the discussion), and sufficiency (rebuttal to counter arguments) [30].

3.2 Pairwise Comparison via BT↩︎

Pairwise comparisons are a widely used method in statistical analysis. In argument quality assessment, this approach allows annotators to reduce cognitive load and annotation time by focusing on comparing two arguments at a time, instead of ranking a whole set. Once we obtain pairwise comparisons, we use the Bradley-Terry model to derive continuous quality scores. Given a pair of arguments \(d_i\) and \(d_j\) from the dataset, Bradley-Terry estimates the probability that the pairwise comparison \(d_i \succ d_j\) is true through the formula  [22]: \(P(d_i \succ d_j) = \frac{\gamma_i}{\gamma_i +\gamma_j}\). \(i>j\) denotes that argument \(d_i\) is preferred to \(d_j\).

4 Dataset↩︎

We use the Webis-ArgQuality-20 corpus, an argument quality dataset introduced by [4]. This gold-standard dataset comprises 1,610 texts across 20 debate topics, of which 1,271 are considered argumentative. Arguments were then subjected to 13952 pairwise comparisons (obtained via cyclic group sampling) analysis by experts (crowdsourcing), who annotated, for each topic and quality dimension (Logical, Rhetorical, or Dialectical Quality), which argument in each pair was better with respect to that specific quality dimension. These ground-truth labels were subsequently used to rank argument quality across dimensions using a Bradley-Terry model. We use the entire set of pairwise comparisons provided in the dataset in our experiments.

5 Experimental setup↩︎

We evaluate 12 open-weight models from five families: Mistral, Llama 3*, Qwen2.5, Olmo2, and Command-R, spanning parameter scales from 7B to 104B ². Each family includes models of different sizes, enabling a controlled analysis of the effects of both model scale and architecture on output variability and inter-model agreement. We focus exclusively on open-weight models to ensure reproducibility and transparency.

We evaluate whether LLMs can replicate expert-derived argument quality rankings published for the Webis-ArgQuality-20 corpus. The approach follows the original pairwise framework where each comparison task presents the two arguments—named Argument A and Argument B–and instructs the LLM to decide which argument is better with respect to a single quality dimension. The response is restricted to the following: ‘‘A’’ if A is better, ‘‘B’’ if B is better, or ‘‘Tie’’ to indicate a tie. No additional text is allowed.

We employed three prompting strategies: zero-shot, few-shot, and chain-of-thought. Zero-shot takes as input the dimension to judge and the pairwise comparison to assess. Chain-of-thought extends the zero-shot strategy by instructing the LLM to reason internally over sub-dimensions associated with each main dimension, following the framework of [3] described in Section 3.1. Logical quality is assessed through local acceptability, local relevance, and local sufficiency; rhetorical quality through credibility, emotional appeal, clarity, appropriateness, and arrangement; and dialectical quality through acceptability, relevance, and sufficiency. Each sub-dimension is reformulated as a comparative question asking which of the two arguments exhibits more of that property, accompanied by a brief definition. The model outputs only the final label for the target dimension, not intermediate sub-dimension judgments, as evaluation is conducted at the dimension level. Thus, the sub-dimensions function as structured guidance for the model’s internal reasoning process rather than as prediction targets in their own right. Finally, few-shot uses the same definitions as zero-shot but dynamically adds, for the pair of arguments under evaluation, three annotated examples within their topic: one where “A” is chosen, one where “B” is chosen, and one labeled as “Tie”. These examples remain the same across all comparisons within that topic.

Specifically, we consider arguments A and B, and for the label A, we select the pair with the largest score difference where the score of A is higher than that of B. Similarly, we choose an example for B by selecting the pair with the largest score difference where the score of B is higher than that of A. For the Tie example, we select, from the ground-truth comparisons labeled as Tie, the pair with the smallest quality score difference for that dimension and topic. In this way, we ensure that the few-shot examples are clear-cut and avoid ambiguous examples ³.

Regarding temperature, values in the 0–1 range show no statistically significant differences in performance, while values above 1 degrade it [31]. We set temperature to 1 to balance stability and stochasticity for measuring intra-model reliability, with gains expected to generalize to lower settings [31], [32].

Given the probabilistic nature of LLMs, we evaluated each model across three independent runs using the same pairing strategy to measure prediction variability. The final label was determined by majority vote. The argument (A or B) with the highest frequency was selected; in cases where no single argument secured a majority (e.g., a three-way split or a tie involving an explicit ‘Tie’ model response), the final label was Tie.

5.1 Evaluation metrics↩︎

To assess the reliability and consistency of LLM-based argument quality evaluation, we did three complementary analyses. First, in the LLM vs. Human Expert analysis, we compared LLM-generated pairwise preferences and the resulting Bradley-Terry scores against those derived from the Webis-ArgQuality-20 expert annotations across the logic, rhetoric, and dialectic dimensions. Second, in the Best LLM vs. Other LLMs analysis, we selected the highest-performing model and compared its pairwise annotations and resulting BT scores against those produced by other LLMs across the same argument pairs. Third, in the Prediction Variability analysis, we assessed the stability of each LLM by repeating the evaluation process across three independent runs and measuring the agreement between the resulting pairwise annotations.

We evaluated Bradley-Terry score alignment using Pearson (\(r\)), Spearman (\(\rho\)), Mean Absolute Error (MAE), and Root Mean Square Error (RMSE). Annotation agreement was measured using Cohen’s \(\kappa\), Krippendorff’s \(\alpha\), and percent agreement.

6 Results & Discussion↩︎

To address SRQ1, we begin by identifying the best-performing model and prompt configuration for evaluating argument quality. Table 1 compares rankings and annotations produced by experts with those generated by LLMs under zero-shot (zero), few-shot (few), and chain-of-thought (cot) prompting strategies, across 36 configurations comprising 12 models and 3 prompting strategies. Llama-70B with few-shot prompting achieves the highest average correlations across the Pearson, Spearman, and Kendall metrics over the logic, rhetoric, and dialect dimensions, with averaged scores of 0.473, 0.477, and 0.327, respectively. It is followed by Qwen2.5-72B under zero-shot prompting, which attains corresponding averaged scores of 0.463, 0.466, and 0.319. Both models also exhibit low variability across the three dimensions. Both configurations achieve the lowest error metrics, with MAE and RMSE of 0.811 and 1.027 for LLaMA-70B with few-shot, and 0.817 and 1.036 for Qwen2.5-72B with zero-shot, respectively. The highest Cohen’s \(\kappa\) is obtained also by LLama-70B with few-shot with 0.493, while the second best is the same model under both zero-shot and chain-of-thought. Similarly, these configurations achieve a percent agreement of 52.93%, 52.43%, and 52.33% for few-shot, chain-of-thought, and zero-shot, respectively. Performance by model size. Small and medium size models do not exhibit higher agreement with experts than large size ones. Across prompts, chain-of-thought may lead to lower correlation scores and agreement with expert rankings and annotations compared to zero-shot. Overall, zero-shot, few-shot, and chain-of-thought exhibit minimal variability across the evaluated dimensions.

Performance by model family. Command-R-7B and Command-R-104B achieve comparable performance, with Command-R-104B obtaining slightly higher correlation and agreement scores. In most other model families, smaller variants perform worse than their larger counterparts. The main exception is the Mistral family, where Mistral-7B achieves higher correlation with expert rankings across all prompting configurations than Mistral-22B, while also obtaining lower MAE and RMSE values. However, Mistral-22B achieves higher IAA scores.

Performance by prompt strategy. There is no systematic preference for a single prompting strategy across all models. Among small models, except for Qwen2.5-7B, few-shot prompting decreases correlation and agreement with experts relative to zero-shot prompting. A similar pattern is observed in medium-sized models, where few-shot prompting is generally detrimental, except for Mixtral-8x7B and Mixtral-8x22B. In contrast, among large models, few-shot prompting benefits Llama-70B and Command-R-104B.

Performance by quality dimension. Overall, configurations showed low variability across the logical, quality, and rhetorical dimensions, indicating that no specific dimension was harder to evaluate. Only Mixtral-8x7B and Command-R-7B under the few-shot setting presented higher variability compared to other configurations across Pearson, Spearman, Kendall, MAE, and RMSE. However, this variability was not reflected in Cohen’s \(\kappa\) or percent agreement. Table 2 shows the highest performance achieved by the best configuration, Llama-70B with few-shot, highlighting that the three dimensions had similar difficulty levels ⁴. Moreover, Figure 1 shows that, despite its performance, Llama-70B with few-shot produces a Bradley-Terry score distribution that differs from that of human experts, consistent with the moderate correlation observed. In particular, the LLM’s distribution appears slightly more right-skewed.

6.1 Cross-LLM Comparison↩︎

In order to address SRQ2, Table 3 compares the rankings and annotations produced by Llama-70B with few-shot against those generated by the remaining models. While configurations other than Llama-70B with few-shot show lower results with respect to the ground truth, we also evaluate how closely other LLMs agree with this best configuration.

Among all configurations, Llama-70B consistently shows the highest correlations and agreement scores across zero-shot and chain-of-thought. For zero-shot it achieves 0.892, 0.892, and 0.718 for Pearson, Spearman, and Kendall, respectively, while 0.831 and 84.45% for Cohen’s \(\kappa\) and percent agreement, respectively. Qwen2.5-72B follows closely under all prompting settings. Although Command-R-104B still shows substantial agreement, it consistently remains below both Llama-70B and Qwen2.5-72B, the other large-scale models.

Within the medium-sized models, Olmo2-32B is the closest to the best configuration. It surpasses Command-R-104B under zero-shot and chain-of-thought, while falling slightly behind under few-shot. The remaining medium-sized models show only moderate agreement. An exception is Mistral-22B, which obtains correlation scores close to 0 together with higher MAE and RMSE values.

The small models generally show moderate agreement and correlation scores. Qwen2.5-7B is the closest among the small models, reflecting the strong results of its larger 72B counterpart across all prompting settings. In contrast, Llama-8B achieves only moderate IAA and moderate-to-low correlation scores, together with higher MAE and RMSE values compared to the models with the strongest agreement. Similarly, Olmo2-7B does not align as well with the best configuration as its 32B counterpart, producing correlations close to 0. Command-R-7B, however, remains partially aligned as its larger version. The low variability across dimensions indicates that these findings are consistent for each of them⁵.

These results, together with those in Table 1, show that although Llama-70B is the best configuration and reaches moderate agreement, some groups of LLMs closely align with it and share its vision, while others are less aligned with it despite still agreeing with experts, reflecting differing but potentially accurate interpretations of the dimensions.

6.2 Prediction Variability↩︎

In order to address SRQ3, Table 4 shows how LLMs exhibit low prediction variability in the pairwise comparison annotation task. The worst-case scenario is Mistral-22B with few-shot prompting, where 7.75% of cases resulted in all three possible labels (“A”, “B”, and “Tie”) across runs. Mistral-22B also achieved the lowest Krippendorff’s \(\alpha\) scores overall (0.16, 0.16, and 0.11 for zero-shot, few-shot, and chain-of-thought, respectively). This is followed by Mixtral-8x22B, which obtained Krippendorff’s \(\alpha\) scores of 0.63, 0.39, and 0.50 under zero-shot, few-shot, and chain-of-thought, respectively. Among the less stable models, Command-R-104B achieved \(\alpha\) values of 0.67 and 0.82 with zero-shot and chain-of-thought, respectively, which improved to 0.86 with few-shot. All remaining models and prompting strategies exhibited strong agreement, achieving Krippendorff’s \(\alpha\) values above 0.78.

Although small models do not achieve the highest performance relative to human experts, they are generally the most stable in the pairwise comparison annotation task. They consistently attain perfect or almost perfect, Krippendorff’s \(\alpha\) scores, while producing few or no "2 equal" and "All unequal" cases. Large models show slightly lower stability. Still, benefiting from few-shot prompting, resulting in fewer "All unequal", "2 equal" cases, or better inter-annotator agreement. In contrast, few-shot prompting appears detrimental for medium-sized models, which consistently achieve lower Krippendorff’s \(\alpha\) scores than their zero-shot and chain-of-thought counterparts.

7 Conclusion & Future Work↩︎

This paper investigated the potential of 12 open-weight LLMs to evaluate argument quality across the logical, rhetorical, and dialectical dimensions using zero-shot, few-shot, and chain-of-thought. Through pairwise comparisons and Bradley–Terry modeling, we analyzed both the alignment of LLM-generated rankings with expert judgments and their consistency across runs.

We show that, although LLMs do not yet reach expert-level performance, they can approximate expert rankings to a meaningful degree. Among all evaluated models, Llama-70B with few-shot prompting achieved the strongest results, obtaining the highest (but moderate) agreement with expert annotations, with Pearson, Spearman, and Kendall correlations of 0.473, 0.477, and 0.327, respectively, as well as a Cohen’s \(\kappa\) of 0.493 and 52.93 percent agreement, across the three quality dimensions. Some models align more closely with Llama-70B, such as Qwen-2.5-72B, or smaller models like Qwen-2.5-7B and Olmo2-32B, while others are less consistent with it but still agree with experts. This suggests that different models may capture complementary perspectives, and that combining diverse LLMs may outperform relying on Llama-70B alone. Finally, we show that, across only three runs per model, prediction variability remains low. Disagreements ranged from cases where two out of three labels matched to instances where all labels differed (up to 7.75% of the total comparisons for the worst configuration Mistral-22B). While still low, this variability motivates the use of aggregation strategies such as majority voting. Few-shot prompting was also found to enhance stability, although its benefits were primarily observed in larger models.

Future work will explore fine-tuning [33] and ensembling LLMs for argument quality assessment, including dimension-specific models. Domain-specific training and ensembling may improve their ability to evaluate reasoning validity and detect fallacies. Second, we will broaden the scope of argument quality assessment by incorporating additional dimensions and working toward a comprehensive overall quality measure.

8 Limitations↩︎

In this section, the limitations of the study are discussed as well as the potential threats to the validity of the findings. The discussion is structured according to the common validity categories inspired by [34] guidelines, namely, internal validity, external validity, construct validity and conclusion validity.

8.1 Internal Validity↩︎

One limitation of this study is the inherent variability of LLM-based evaluations, as different runs may produce different pairwise judgments. While the results remained stable across runs and majority voting was used to improve reliability, stability itself remains an important factor to assess. In particular, we did not evaluate the models at lower temperature settings. Based on the hypothesis proposed by [31], which identifies a plateau beyond which model outputs may become unreliable, stability at a temperature of 1 may indicate comparable or greater stability at lower temperatures. Nevertheless, an ablation study across different temperature settings would be necessary to verify this assumption and confirm that all models produce comparable results for our task in line with the findings of [31].

Another limitation concerns prompt formulation. Although the same prompt templates were used for all models, differences in how individual LLMs interpret the instructions may have influenced the resulting judgments.

8.2 External Validity↩︎

First, the evaluation was based on a specific dataset consisting of arguments drawn from an online debate portal on a set of different topics. These arguments are mostly a few sentences. Therefore, the results achieved may not generalize to other more complex forms, such as essays. There is a possibility that it could be (more or less) challenging for the LLM. Another aspect is that the expert scores used as ground truth came from a specific annotation process and reflect the judgments of the annotators shaped by their own interpretation of the dimensions. In another context, the different dimensions could be understood differently, meaning the LLM’s alignment with one set of experts might not hold if used in another set. For this research, the zero-shot, few-shot, and chain-of-thought approaches were used where the model is given instructions to compare the arguments without fine-tuning or additional training. While these approaches are practical, it does not take into account the performance gains from fine-tuning the LLM. The results do not explain how well the models would do in that particular setting. Finally, LLMs are evolving rapidly. The LLM versions used in this study are not the latest and may have already been surpassed by newer releases. At the same time, to ensure a fair comparison, we accounted for model size (small, medium, and large) and family, while avoiding substantial version differences (e.g., comparing a version 3 model with a version 1 model from the same family). Although comparative and qualitative insights might remain relevant, metric values may shift even further with advances in LLM.

8.3 Construct Validity↩︎

Argument quality was split into three dimensions (logical, rhetorical, dialectical) according to the dataset’s scheme for annotation. There is a possibility that the LLMs comparisons do not align with the intended construct. For example the LLM might rate the argument’s logical quality on confident phasing even if the actual argument itself is logically flawed. Comparing them to human judgment without 100% certainty that they have the same understanding of the ‘quality’ dimensions is very challenging. Furthermore, while this approach provided insights that were valuable, it did not encompass every aspect related to argument quality such as precision and completeness. The dataset used for the gold standard comes from different experts who made pairwise comparisons. Any noise and/or bias could affect the results. The Webis-ArgQuality dataset is assumed as a valid measure for argument quality, but it is possible that there could have been inconsistencies in the annotation process or maybe the experts had their own biases. Another consideration is the separation of quality into three dimensions. Each dimension is treated independently for evaluation purposes, but there was no explicit reminder in the prompt to ignore the other dimensions. Therefore, there is a possibility that the LLM might try to bring in other dimensions despite the intention to isolate. If the LLM for example took rhetorical into account for logical dimension analysis, then results could be blurry.

8.4 Conclusion Validity↩︎

One limitation related to conclusion validity is seen through the number of pairwise comparisons generated for this study. Although 13952 comparisons across 1271 arguments provided a substantial dataset, prior work by [4] demonstrated the benefits of using far larger comparisons sets (over 41000). Larger datasets may increase the statistical robustness of the Bradley-Terry model, and improve the stability of resulting rankings. In this study, the comparison set was chosen to maintain consistency with the comparison corpus released as part of the Webis-ArgQuality-20, which relies on a cyclic group sampling strategy. Future research could explore the impact of scaling up the number of comparisons.

9 Ethical Considerations↩︎

The assessment of argument quality aims, in general, at the societal benefit since better, stronger, higher-quality arguments do not contain fallacies and spurious reasoning. Therefore, this work intends to promote the development of high-quality, grounded discussions. At the same time, two ethical risks are inherent to this work. The first is that, by strengthening the quality of arguments supporting them, unethical or harmful claims are spread. For instance, hateful messages could be supported by rhetorically high-quality arguments. One possible solution to this issue is to accompany argument quality assessment with hate speech detection or similar analyses involving other information quality aspects. The second risk is that, by demanding quality assessment to LLMs, intentional or unintentional distortions are introduced in the analysis and moderation of public discourse. For example, models could show biases that favor some types of arguments over others. For this reason, as indicated also above, it is important to include human oversight in these moderation systems and employ automated models in LLM-in-the-loop systems.

Acknowledgments↩︎

This work used the Dutch national e-infrastructure with the support of the SURF Coop-erative using grant no. EINF-15500. The authors would also like to thank Greta Damo for her helpful comments during the revision of this paper.

10 Specific Version of LLMs Evaluated↩︎

We use Mistral [35], Llama 3* [36], Qwen2.5 [37], Olmo2 [38], and Command-R [39] open-access Instruct-based models from Hugging Face ⁶. All models are publicly available and traceable through their repository identifiers:

Small Size Models:

• Mistral-7B: mistralai/
Mistral-7B-Instruct-v0.3

• Llama-8B: meta-llama/
Llama-3.1-8B-Instruct

• Olmo2-7B: allenai/
OLMo-2-1124-7B-Instruct

• Qwen2.5-7B: Qwen/Qwen2.5-7B-Instruct

• Command-r-7B: CohereLabs/``c4ai-command-r7b-12-2024

Medium Size Models:

• Mixtral-8x7B: mistralai/
Mixtral-8x7B-Instruct-v0.1

• Mistral-22B: mistralai/
Mistral-Small-Instruct-2409

• Olmo2-32B: allenai/
OLMo-2-0325-32B-Instruct

• Mixtral-8x22B: mistralai/
Mixtral-8x22B-Instruct-v0.1

Large Size Models:

• Llama-70B: meta-llama/
Llama-3.3-70B-Instruct

• Qwen2.5-72B: Qwen/
Qwen2.5-72B-Instruct

• Command-r-104B: CohereLabs/
c4ai-command-r-plus-08-2024

Small models were run on a single H100 GPU, medium-sized models on up to two H100 GPUs, and large models on up to four H100 GPUs.

11 Prompts↩︎

We tested three prompting strategies, zero-shot, few-shot, and chain-of-thought using a system-user prompt setup with structured outputs through the vLLM Python library ⁷.

11.1 Zero-shot Prompt↩︎

System Prompt: You are given two arguments: Argument A and Argument B. Decide which one is {dim_adverb} stronger: Reply with only one of the following options: A, B, or Tie.

User Prompt:
Argument A: {a}
Argument B: {b}

Output Labels: ["A", "B", "Tie"]

Where dim_adverb are replaced with the strings logically, dialectically, rhetorically, for the logic, dialectic, and rhetoric dimension, respectively. a, and b are replaced with the arguments A, and B, respectively, to be judged by the LLM.

11.2 Few-shot Prompt↩︎

System Prompt: You are given two arguments: Argument A and Argument B. Decide which one is {dim_adverb} stronger: Reply with only one of the following options: A, B, or Tie.

Examples:
Argument A: {ex_A_argA}
Argument B: {ex_A_argB}
A

Argument A: {ex_B_argA}
Argument B: {ex_B_argB}
B

Argument A: {ex_tie_argA}
Argument B: {ex_tie_argB}
tie

User Prompt:
Argument A: {a}
Argument B: {b}

Output Labels: ["A", "B", "Tie"]

Where dim_adverb, a, and b are instantiated as in the zero-shot prompt. ex_A_argA and ex_A_argB are few-shot examples where A is preferred on the evaluated dimension; ex_B_argA and ex_B_argB are examples where B is preferred; and ex_tie_argA and ex_tie_argB are examples where the preferred label is Tie.

We used 3 shots for each of the 20 topics in the Webis-ArgQuality-20 corpus, leading to 60 examples per dimension ⁸.

11.3 Chain-of-thought↩︎

System Prompt: You are given two arguments: Argument A and Argument B.
Decide which one is {dim_adverb} stronger considering these criteria:
{dim_criterias}
Let’s think step by step and reply with only one of the following options: A, B, or Tie.

User Prompt:
Argument A: {a}
Argument B: {b}

Output Labels: ["A", "B", "Tie"]

Where dim_adverb, a, and b are instantiated as in the zero-shot prompt. dim_criterias is replaced with the chain-of-thought questions for each quality dimension based on the theoretical work of [3]:

Criteria for logic:
- which is more acceptable? A premise of an argument is acceptable if it is rationally worthy of being believed to be true.
- which is more relevant to a conclusion? A premise of an argument is relevant if it contributes to the acceptance or rejection of the argument’s conclusion.
- which is more sufficient to justify a conclusion? An argument’s premises are sufficient if, together, they give enough support to make it rational to draw its conclusion.

Criteria for dialectic:
- which would be acceptable to the audience? Argumentation is acceptable if the target audience accepts both the consideration of the stated arguments for the issue and the way they are stated.
- which contributes more to constructive dialogue? Argumentation is relevant if it contributes to the issue’s resolution, i.e., if it states arguments or other information that help to arrive at an ultimate conclusion.
- which better anticipates or refutes counterarguments? Argumentation is sufficient if it adequately rebuts those counterarguments to it that can be anticipated.

Criteria for rhetoric:
- which appears more authorative/trust worthy? Argumentation creates credibility if it conveys arguments and similar in a way that makes the author worthy of credence.
- which makes a stronger emotional appeal? Argumentation makes a successful emotional appeal if it creates emotions in a way that makes the target audience more open to the author’s arguments.
- which has a clearer style? Argumentation has a clear style if it uses correct and widely unambiguous language as well as if it avoids unnecessary complexity and deviation from the issue.
- which has a more appropiate style? Argumentation has an appropriate style if the used language supports the creation of credibility and emotions as well as if it is proportional to the issue.
- Which is better arranged? Argumentation is arranged properly if it presents the issue, the arguments, and its conclusion in the right order.

12 Licenses and Intended Use↩︎

The dataset used in our experiments, Webis-ArgQuality-20, was created by [4] and is available under a Creative Commons Attribution 4.0 International (CC BY 4.0) license. It is publicly accessible at https://zenodo.org/records/3780049.

The models employed in our experiments, as listed in Appendix 10, were used in accordance with their respective licenses and applicable usage policies. The code used in our experiments, together with the LLM-generated pairwise judgments and the Bradley-Terry rankings derived from them, is released under a Creative Commons Attribution 4.0 International (CC BY 4.0) license. Redistribution of model-generated outputs is believed to be permitted under the applicable licenses of the models used; users remain responsible for complying with the original model licenses and usage policies.

References↩︎