Towards Scalable Oversight with Collaborative Multi-Agent Debate in Error Detection
October 23, 2025
Abstract
Accurate detection of errors in large language models (LLM) responses is central to the success of scalable oversight, or providing effective supervision to superhuman intelligence. Yet, self-diagnosis is often unreliable on complex tasks unless aided
by reliable external feedback. Multi-agent debate (MAD) seems to be a natural alternative to external feedback: multiple LLMs provide complementary perspectives and cross-checks for error detection. However, prior MAD protocols
frame debate as a zero-sum game, where the debaters compete to win the game instead of seeking the truth. Consequently, it leads to debate hacking: debaters tend to mislead the judge by misinterpreting the task or presenting overconfident
claims, which introduce more mistakes and underperform single-agent methods. To mitigate the issue, we introduce a new collaborative MAD protocol, termed ColMAD, that reframes MAD as a non-zero sum game.
Specifically, ColMAD encourages multiple agents to criticize each other in a supportive way, such that they can complement the missing points of each other. Therefore, the judge agent can make a more informative conclusion based on more
comprehensive evidence. Empirically, we show that ColMAD significantly outperforms previous competitive MAD by 19% and brings non-trivial improvements over single-agent methods in error detection.
1 Introduction↩︎
Large language models (LLMs) have gained huge success in solving tasks at various complexity levels spark_AGI?, o1?, r1?. As LLMs grow more powerful and capable of challenging tasks that require human expert-level knowledge, it becomes harder for humans to effectively understand and supervise LLMs amodei2016concrete?. Hence, it is essential to seek scalable oversight that provides effective supervision signals to powerful LLMs beyond human intelligence amodei2016concrete?, christiano2018supervising?, irving2018ai?, bowman2022measuring?, burns2023weaktostrong?, khan2024debating?, kenton2024on?. Detecting the errors in LLM responses is critical to the success of scalable oversight tyen2024cannotfind?, huang2024cannotselfcorrect?, kamoi2024self-correction?. However, LLMs are shown to be struggling to self-diagnose their own mistakes when without reliable external feedback kamoi2024self-correction?. In particular, realmistake? found that powerful LLMs like GPT-4 can not reliably detect errors in the responses of GPT-4 or Llama-2.
To complement the requirement for external feedback in error detection, it is natural to incorporate feedback from other LLMs to help with error detection. As LLMs differ in their knowledge and error tendencies kim2025correlated?, they are unlikely to commit the same mistakes simultaneously, thereby providing complementary signals for error
detection (see in Fig. 2). In fact, Multi-Agent Debate (MAD) is a promising scheme to realize this insight that incorporates the knowledge of multiple LLM agents to resolve complex reasoning tasks and
improve over single-agent methods Chen2025HarnessingML?, Feng2025WhenOL?, Buhl2025AnAS?. In particular, khan2024debating? and kenton2024on? showed that even a weak LLM can easily
identify the flaws and select the correct answer from the debate of powerful LLMs on complex tasks. Nevertheless, it remains underexplored whether MAD also facilitates the LLM error detection, and thus it raises an interesting research
question: Can we incorporate MAD to help with LLM error detection? In this work, we investigate the feasibility of using MAD to detect errors in LLM responses. Despite the high potential of the MAD, we find
that previous MAD approaches will even introduce more mistakes and underperform the use of a single LLM in error detection (see Fig. 3). Interestingly, as previous approaches often frame
MAD as a zero-sum game where the debaters compete with each other, we find that stronger LLMs will exhibit debate hacking behaviors in competitive MAD(CopMAD): as shown in Fig. 1, debaters will misinterpret the task requirements and present claims in an overconfident tone to mislead the judge agent. In other words, CopMAD debaters will try every possibility to persuade the
judge to win the game, instead of providing honest and evidential statements. Consequently, CopMAD can degenerate or even underperform single-agent methods (Proposition 1).
To mitigate the issue, we propose a new MAD protocol called Collaborative Multi-Agenet Debate(ColMAD) that reframes MAD as a non-zero sum game. In contrast to CopMAD, instead of prompting
the agents to win the game, ColMAD asks debaters to collaborate and complement each other’s missing points. Thus, the judge agent could make a more informative and objective decision based on the debating transcripts (Proposition 2). Empirically, across three benchmarks in ReaLMistake realmistake?, we demonstrate that ColMAD can lead to up to 4% improvements in identifying mistakes of LLMs compared to single-agent-based methods. In contrast, CopMAD will lead to
up to 15% performance decrease compared to single-agent methods. In addition to the increased correctness of the error detection, we also find that the explanations given by ColMAD to why there are errors in LLM responses are more aligned
to humans. The more human-aligned MAD protocol provides useful insights for scalable oversight.
2 Collaborative Multi-Agent Debate↩︎
In this section, we initiate an investigation of MAD for detecting errors in LLM responses.
2.1 Potential of Multi-Agent Collaboration↩︎
As realmistake? showed that a single LLM can hardly tell the errors in LLM responses when without no reliable external feedback,
a natural idea is to investigate whether incorporating multiple LLMs can mitigate the issue. Specifically, we consider the collaboration of two LLMs in error detection through MAD, and our discussion can also be generalized to two or more
LLMs. Specifically, we consider two agents \(A\) (Alice) and \(B\) (Bob), and denote the predictions of the error detections as \(\hat{{\boldsymbol{y}}}_A\)
and \(\hat{{\boldsymbol{y}}}_B\), with rationales (e.g., CoT reasoning) as \(x_A\) and \(x_B\), respectively. During the debate, they will emit messages
\(m_A\) and \(m_B\) to convince a judge \(J\).
Assumption 1 (Optimal Judge Strategy). Denoting the label as \(Y\in\{0,1\}\) with prior probability \(\pi\in(0,1)\), without debating, the judge will derive the final answer based on the initial responses \(X_0=(x_A,x_B)\). Assuming the judge \(J\) uses the Bayes test on the total log-likelihood ratio (LLR), we will have the LLR based on \(X_0\) as \[\label{eq:llr95x0} \Lambda_0(X_0)=\log\frac{p(X_0|Y=1)}{P(X_0|Y=0)}+\log\frac{\pi}{1-\pi}.\qquad{(1)}\] The contribution to LLR from the debating can be written as \[\label{eq:llr95contrib} l_i(m_i;X_0)=\log\frac{P(m_i|Y=1,X_0)}{P(m_i|Y=0,X_0)},\;i\in\{A,B\}.\qquad{(2)}\] Then, with debating, the LLR of the judge is \[\label{eq:llr95debating} \Lambda(X_0,m_A,m_B)=\Lambda_0(X_0)+l_A(m_A;X_0)+l_B(m_B;X_0)+\log\frac{\pi}{1-\pi}.\qquad{(3)}\]
Figure 2: Error reductions of prevalent LLMs in detecting errors of . The numbers refer to the reduced errors following the oracle collaboration via Eq. (1 ). It can be found that different LLMs are less likely to make mistakes simultaneously. When incorporating LLMs with higher heterogeneity, such as those from different companies, the error reduction rates will be higher..
Intuitively, if the elicited messages \(M=(m_A,m_B)\) bring additional information to the judge, i.e., \(I(Y;M|X_0)>0\) where \(I(\cdot;\cdot)\) denotes the mutual information, we will have \(\Lambda(X_0,M)>\Lambda(X_0)\). In order to provide additional information, we need to look into the cases where the predictions by \(A\) differ from those by \(B\). Under \(Y_A\neq Y_B\), if both agents are able to provide sufficient justifications and are more persuasive during the debate when they are debating for the correct answer. Hence, the judge can be convinced to take the correct answer when either of the agents is correct. The reduction of errors from the oracle collaboration can be calculated through \[\label{eq:oracle95collab} \min_{K\in\{A,B\}}\sum_{i}\mathbb{1}[\hat{{\boldsymbol{y}}}_K^{(i)}\neq{\boldsymbol{y}}^{(i)}]-\sum_{i}\mathbb{1}[\hat{{\boldsymbol{y}}}_J^{(i)}\neq{\boldsymbol{y}}^{(i)}],\tag{1}\] where \({\boldsymbol{y}}^{(i)}_K\) denotes the initial prediction of the agent \(K\in\{A,B\}\) on the \(i\)-th sample. Intuitively, Eq. (1 ) can be considered as the potential of the collaboration between \(A\) and \(B\).
Potential of multi-agent collaboration. We plot the error reductions for the prevalent LLMs benchmarked in ReaLMistake realmistake? in Fig. 2. We visualize the ratio of the reduced errors under the oracle collaboration protocol of Eq. (1 ), given the LLM
error detection results from ReaLMistake. The ReaLMistake benchmark contains \(3\) objective error detection tasks: (i) math word problem generation (Math problem) that requires LLMs to generate
math problems satisfying requirements; (ii) fine-grained fact verification (Fact verification) that requires LLMs to verify claims given fine-grained evidence; (iii) answerability classification (Answerability) that requires LLMs to leverage their
commonsense knowledge to examine whether a question is answerable. We calculate the potential of collaboration of multiple prevalent LLMs.
From Fig. 2, it can be found that two different LLMs tend to make fewer mistakes simultaneously, indicating a higher potential for oracle collaboration. The more different the two LLMs are in the collaboration,
especially for those from different companies, the less likely they are to make mistakes at the same time. For example, collaboration between Llama-2 and Llama-3.1 can lead to little-to-no error reduction. In contrast, the
collaboration between GPT-4 and Llama-2 can reduce more than 30% of errors. Furthermore, given the potential of test-time scaling capabilities of LLMs TTT_better?, muennighoff2025s1simpletesttimescaling?, collaboration of agents can have the potential of bringing additional information when both agents make mistakes and reduce beyond Eq. (1 ).
2.2 Pitfalls of Multi-Agent Debate↩︎
To gain more insights on incorporating MAD for error detection, we conduct an initial experiment with the Society of Minds (SoM) du2023improving?, as well as previous CopMAD methods khan2024debating?, kenton2024on?, which design sophisticated debater prompts along with a judge. Different from CopMAD, SoM
encourages LLMs to incorporate the good points from others’ responses, which deviates from the typical scheme of debate for scalable oversight irving2018ai?, khan2024debating?, kenton2024on?. We will
discuss the differences in Sec. 2.3 and focus more on CopMAD.
We consider the collaboration between Llama-3.1-70B (Llama-3.1 in short) and GPT4o-mini (4o-mini in short) . The results are given in Fig. 3. We report the precision, recall, and F1 results
following realmistake?. From the results, we can find that, although in most cases, SoM du2023improving? and CopMAD protocol khan2024debating?, kenton2024on? can improve the precision of the error detection compared with the single-agent methods, they also lead
to a severe decrease in the recall across all tasks. Hence, the resulting F1 decreases dramatically, as well.
Figure 3: Pitfalls of previous MAD protocols. Under previous MAD schemes, such as CopMAD or SoM, the debate results are lower than any of the LLMs involved in the debate in most cases. In contrast,
ColMAD significantly improves CopMAD and outperforms the use of a single LLM..
Essentially, CopMAD implements MAD into a zero-sum game, where the debaters compete and challenge each other’s claims, and explore the potential solutions ai_safety_debate?. A critical requirement for the success of MAD is the honesty. When the debaters are dishonest with the supporting evidence, the
zero-sum nature of CopMAD drives debaters to hack the debate, leading to a severe decrease in performance. Theoretically, in the binary classification case, assuming the optimal judge strategy, we can establish the following
formulation of CopMAD.
Proposition 1 (Pitfalls of dishonest competitive debating). Assuming a bounded LLR, i.e., \(l_i\leq L_i<\infty,\;i\in\{A,B\}\), let \(R(Z)\) denote the Bayes risk of the judge \(J\) when making decisions based on \(Z\), denote \(R_0=R(X_0)\), denote the outcome of zero-sum debating as \[V_\mathrm{comp}=\min_J\max_{e\in\mathcal{E}(J)} P(J(X_0,M_e)\neq Y),\] where \(M_e\) are the debating transcripts given by the optimal Nash equilibrium \(e\in\mathcal{E}(J)\) of the \(A\)–\(B\) subgame in convincing \(J\). Then, we have \(V_\textrm{comp}=R_0\).
The proof of Proposition 1 is given in Appendix 6.2. Intuitively, Proposition 1 illustrates a degenerated case of competitive debate among two dishonest agents: the two dishonest agents try to find any useful arguments to defend
for their answers. When they are assigned different answers as in CopMAD kenton2024on?, one of them is incorrect, and
thus the competitive debate can be formulated as the minmax problem in Proposition 1. Consequently, the optimal strategy for the judge is to consider the
original rationale by \(A\) and \(B\) instead of the debating transcripts. Given that the judge agent is usually imperfect and limited in the reasoning capabilities of the LLMs, the final
answer by the judge is usually dependent on the debate skills of the LLMs, which explains why MAD usually underperforms the single-agent method as shown in Fig. 3, as well as the empirical
observations from recent literature Smit2023ShouldWB?, Yang2025RevisitingMD?, Zhang2025IfMD?.
Empirically, we also find that the debating messages can be misleading. As shown in Fig. 4, the debater agent may come up with fake evidence or misinterpret the task requirements to mislead the judge. The debater agent may be overconfident or focus on persuasion instead of solving the questions, such that it may use an overconfident tone to mislead the judge. In other words, in a zero-sum debate game, the debating skill of the agent will significantly affect the choice of the judge.
2.3 Collaborative Multi-Agent Debate↩︎
To mitigate the debate hacking issue, we propose a new debate protocol, termed Collaborative Multi-Agenet Debate(ColMAD). Instead of performing a competitive debate, we instruct the agents to collaborate with each other, such
that they could explore all the critical information about the question, to enable the judge to make a well-informed decision. By turning the zero-sum game into a non-zero-sum game, ColMAD calibrates the dishonesty and provides
practical robustness to MAD when the debaters are not perfectly honest due to the limited capabilities of LLMs.
Proposition 2 (Collaborative debating). Under the same setting as Proposition 1, when the two agents are collaborating, we have the value as \[V_\mathrm{comad}=\min_J\min_{e\in\mathcal{E}_\mathrm{comad}(J)}P(J(X_0,M_e)\neq Y),\] where the \(M_e\) are the debating transcripts given by the optimal Nash equilibrium \(e\in\mathcal{E}_\mathrm{comad}(J)\) of the \(A\)-\(B\) subgame in collaboratively seeking the truth. Then, we have \(V_\mathrm{comad}\leq R(X_0)=V_\textrm{comp}\), where the strict inequality holds when \(I(Y;M_e|X_0)>0\).
The proof of Proposition 2 is given in Appendix 6.3. Intuitively, if the debater agents are able to provide additional information about the question, e.g., pointing out the missing information from the reasoning of the other agent, the collaborative scheme is provably better than the competitive scheme.
Degenerated collaboration. Despite the benefits of ColMAD, collaboration may also introduce biases and lead to degenerated collaborations. For example, SoM du2023improving? focuses on incorporating the useful information from other agents’ responses. However, as shown in Fig. 3, when an agent
collaborates with a less capable agent and treats the opinions from the other agent as correct information, even a strong agent can be misled and suffer from performance degeneration. It is also evident that the existing collaboration scheme, such as SoM,
only maintains an average performance of the agents involved in the collaboration Yang2025RevisitingMD?, Zhang2025IfMD?.
Practical implementations. In addition to clearly state the objective of collaboration for seeking the truthful answer, we implement the insights of ColMAD into specific prompting schemes: (i) Evidence verification:
ColMAD implements a quote-based system that asks the debaters to quote evidence from the context, and each quote will be verified if there is an exact match in the context, following kenton2024on?; (ii) Self-auditing: the debaters are required to self-audit if there exists one potential failure mode in the claim; (iii) Confidence calibration: the debaters are
required to provide a confidence estimate for their own claims. We provide a detailed description of the algorithm, as well as the prompts in Appendix 8.
3 Related Work↩︎
In this section, we discuss the related work and background of MAD and scalable oversight.
Multi-Agent Debate. MAD aims to imitate the cooperation of humans towards building a society of AI Minsky1987TheSO?. Recently, MAD has gained significant attention as one of the promising approaches to scale up the test-time computation for enhancing reasoning and alignment ai_safety_debate?, du2023improving?, khan2024debating?, kenton2024on?.
A typical MAD protocol involves two or more agents to explore a diverse set of solutions, provide evidence to support their respective solutions, and reach a consensus du2023improving?. A judge agent can also be incorporated to read the transcripts of the debate and to give a final answer khan2024debating?. MAD has demonstrated great potential in improving the reasoning and reducing the hallucinations of LLMs du2023improving?, liang2023encouraging?, yin-etal-2023-exchange?, chen-etal-2024-comm?. liang2023encouraging? further assigned
personalities to the debating agents to explore the potential answers more efficiently. yin-etal-2023-exchange? manually specified diverse roles to agents, and proposed a confidence-based mechanism to reduce the error propagation during reasoning. chen-etal-2024-comm? proposed a more fine-grained role assignment based on reasoning paths of agents.
Meanwhile, MAD also demonstrated great potential in facilitating the alignment ai_safety_debate?, khan2024debating?, kenton2024on?. khan2024debating? showed that LLMs tend to be more persuasive when debating for the correct answer. kenton2024on? generalized the study of khan2024debating? and showed that MAD can enable scalable oversight where a weak agent can easily tell the correctness of strong agents during debating.
Pitfalls of Multi-Agent Debate. Recent studies also showed the limitations of MAD wang-etal-2024-rethinking-bounds?, Smit2023ShouldWB?, Zhang2025IfMD?. wang-etal-2024-rethinking-bounds? found that single-agent methods can already perform competitively or outperform MAD when given sufficient information. Smit2023ShouldWB? showed that MAD can underperform single-agent when given sophisticated prompting methods such as
self-consistency wang2023selfconsistency?. Zhang2025IfMD?, Yang2025RevisitingMD? provided more
comprehensive evidence to support the findings by wang-etal-2024-rethinking-bounds? and
Smit2023ShouldWB?.
Different from previous MAD studies, in this work, we focus on evaluating the capabilities and MAD in detecting errors of LLM responses, which is a central task in scalable oversight realmistake?. Similar to wang-etal-2024-rethinking-bounds?, Smit2023ShouldWB?, Zhang2025IfMD?, and Yang2025RevisitingMD?, we find that the vanilla MAD protocol can often lead to degraded performances, significantly lower than single-agent approaches. In the meantime, we also find that
enabling collaboration improves the MAD performances, offering a new perspective on MAD for scalable oversight.
Scalable Oversight and Error Detection. Scalable oversight aims to provide supervision signals to AI models beyond the human capabilities, especially for tasks where it is hard to obtain ground-truth labels amodei2016concrete?, christiano2018supervising?, ai_safety_debate?, bowman2022measuring?. Scalable oversight can be implemented in a variety of forms, such as weak-to-strong generalization where weak LLMs provide direct supervision signals to elicit capabilities of stronger LLMs Burns2023WeaktoStrongGE?, and self-critique Saunders2022SelfcritiquingMF? where LLMs provide evaluation signals to further improve LLMs as an alternative to humans bowman2022measuring?. Following the latter protocol, kenton2024on? showed that weak LLMs can easily tell the correctness of the answers by strong LLMs when given the debating transcripts of the strong LLMs.
Hence, a central task in scalable oversight is detecting the errors in LLM responses realmistake?, tyen2024cannotfind?, huang2024cannotselfcorrect?, kamoi2024self-correction?, where LLMs are expected to self-correct their own responses or detect
the errors of responses from other LLMs kamoi2024self-correction?. A number of studies showed that LLMs
can struggle with correcting their own mistakes tyen2024cannotfind?, huang2024cannotselfcorrect?. Furthermore, realmistake? benchmarked using top LLMs such as GPT-4 and Claude 3 Opus to detect errors in responses by GPT-4 and Llama-2. The results by realmistake? showed that LLMs generically struggle to detect mistakes made by others, as well, while it is easier for humans. Different from prior works, we conduct an investigation
on whether MAD can help with error detection, offering a new perspective on MAD for scalable oversight.
4 Experiments↩︎
We conduct extensive experiments to demonstrate the effectiveness of ColMAD against CopMAD.
4.1 Experimental Setup↩︎
Datasets. We mainly use the ReaLMistake benchmark realmistake?, which focuses on objective LLM
error detection of three tasks: (a) Math Word Problem Generation: The original LLM is instructed to generate a math word problem that follows the given requirements. Mistakes of LLMs can be made in following the requirements, as well as in the
mathematical reasoning; (b) Fine-grained Fact Verification: The original LLM is instructed to check whether the claims in a sentence are well-supported. Mistakes of LLMs can be made due to the reasoning and use of the context information; (c)
Answerability Classification: The original LLM is instructed to classify whether a factual question is answerable or not. Mistakes of LLMs can be made due to hallucination and reasoning. The original LLMs are GPT-4-0613 and
Llama-2-70b. The statistics of the ReaLMistake benchmark can be found in Appendix 9.
Baselines. As our focus is to demonstrate the usefulness of ColMAD compared to CopMAD, hence we mainly adopt the scheme in kenton2024on? that demonstrated impressive capabilities in scalable oversight. In addition, we also consider a simple Ensemble baseline: if the two agents agree on an option, the prediction
will be the option; otherwise, the prediction will be randomly chosen. The Ensemble baseline can be considered as the simplest collaborative MAD approach.
LLM Backbones. Due to the massive updates of the state-of-the-art LLMs, the original LLMs benchmarked in ReaLMistake realmistake? are a bit outdated. Therefore, we incorporate new frontier LLMs, including GPT4o-mini openai2023gpt4omini?, Llama3.1-70B meta2024llama3.1?,
Mistral-7B-v0.3 mistral?, Qwen-2.5-72B qwen2.5? as well as the frontier reasoning LLMs including DeepSeek-R1 r1? and
QwQ-32B qwq32b?. For the pairing of collaborative agents, due to the limited spaces, we present a subset of paired LLMs. The
temperature of all LLMs is set to \(0\) to ensure reproducibility.
Evaluation Metrics. We report the F1 score following the common practice. In addition, noticing the nature of the task is the error detection, we additionally report and focus on the F2 score, which is an instantiation of the \(F_\beta\) score fbeta_score?: \(F_\beta=(1+\beta^2)\cdot\text{precision}\cdot\text{recall}/(\beta^2\cdot \text{precision}+\text{recall}),\) with setting \(\beta\) as \(2\). The F2 score emphasizes the recall rate, i.e., whether all errors in an LLM response can be detected, which is crucial for scalable oversight.
| Debater#1 | Debater#2 | Protocol | Math Problem | Fact Verification | Answerability | Avg. F1 | Avg. F2 | |||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| (l2ptr2pt)4-11 | F1 (\(\uparrow\)) | F2 (\(\uparrow\)) | F1 (\(\uparrow\)) | F2 (\(\uparrow\)) | F1 (\(\uparrow\)) | F2 (\(\uparrow\)) | F1 (\(\uparrow\)) | F2 (\(\uparrow\)) | ||||||||||||||||||
| Human | - | - | 90.00 | 84.91 | 95.45 | 95.45 | 90.48 | 87.96 | 89.44 | 91.98 | ||||||||||||||||
| GPT4o-mini | - | - | 78.70 | 89.10 | 75.76 | 82.78 | 70.10 | 74.73 | 74.85 | 82.20 | ||||||||||||||||
| Llama3.1-70B | - | - | 79.26 | 89.96 | 76.85 | 85.15 | 68.13 | 69.98 | 74.75 | 81.70 | ||||||||||||||||
| Mistral-7B-v0.3 | - | - | 55.21 | 53.07 | 73.24 | 83.33 | 60.71 | 59.44 | 63.05 | 65.28 | ||||||||||||||||
| Qwen-2.5-72B | - | - | 78.82 | 77.73 | 38.18 | 28.77 | 42.37 | 32.98 | 53.12 | 46.49 | ||||||||||||||||
| DeepSeek-R1 | - | - | 84.09 | 84.67 | 79.77 | 80.61 | 62.25 | 57.04 | 75.37 | 74.11 | ||||||||||||||||
| QwQ-32B | - | - | 84.44 | 86.17 | 73.62 | 71.77 | 61.74 | 56.10 | 73.27 | 71.35 | ||||||||||||||||
| GPT4o-mini | Llama3.1-70B | 78.38 | 90.06 | 75.12 | 85.47 | 75.36 | 83.33 | 76.29 | 86.29 | |||||||||||||||||
| 66.67 | 65.97 | 66.24 | 63.11 | 50.37 | 42.93 | 61.09 | 57.34 | |||||||||||||||||||
| Ensemble | 78.34 | 88.91 | 75.62 | 83.33 | 68.78 | 72.22 | 74.25 | 81.49 | ||||||||||||||||||
| Llama3.1-70B | GPT4o-mini | 78.38 | 90.06 | 77.42 | 88.98 | 72.91 | 79.74 | 76.24 | 86.26 | |||||||||||||||||
| 78.34 | 88.91 | 73.79 | 82.43 | 68.16 | 69.32 | 73.43 | 80.22 | |||||||||||||||||||
| Ensemble | 78.34 | 88.91 | 75.62 | 83.33 | 68.78 | 72.22 | 74.25 | 81.49 | ||||||||||||||||||
| GPT4o-mini | Mistral-7B-v0.3 | 77.13 | 88.84 | 77.14 | 87.10 | 74.40 | 82.26 | 76.22 | 86.07 | |||||||||||||||||
| 74.73 | 76.75 | 59.35 | 56.10 | 55.03 | 50.00 | 63.04 | 60.95 | |||||||||||||||||||
| Ensemble | 71.20 | 75.22 | 74.04 | 83.15 | 64.04 | 64.92 | 69.76 | 74.43 | ||||||||||||||||||
| Llama3.1-70B | Mistral-7B-v0.3 | 77.83 | 89.21 | 75.58 | 86.86 | 70.53 | 74.28 | 74.65 | 83.45 | |||||||||||||||||
| 74.64 | 82.98 | 70.53 | 75.28 | 64.37 | 64.37 | 69.85 | 74.21 | |||||||||||||||||||
| Ensemble | 69.11 | 73.01 | 73.93 | 83.69 | 62.50 | 62.93 | 68.51 | 73.21 | ||||||||||||||||||
| GPT4o-mini | DeepSeek-R1 | 82.76 | 90.52 | 76.77 | 83.89 | 70.79 | 71.75 | 76.77 | 82.05 | |||||||||||||||||
| 49.59 | 39.27 | 28.57 | 21.80 | 18.69 | 13.59 | 32.28 | 24.89 | |||||||||||||||||||
| Ensemble | 81.22 | 87.34 | 79.57 | 83.90 | 65.91 | 66.36 | 75.57 | 79.20 | ||||||||||||||||||
| Llama3.1-70B | DeepSeek-R1 | 81.95 | 90.13 | 78.26 | 87.66 | 73.91 | 76.40 | 78.04 | 84.73 | |||||||||||||||||
| 52.86 | 46.13 | 65.96 | 69.98 | 60.81 | 55.01 | 59.88 | 57.04 | |||||||||||||||||||
| Ensemble | 80.81 | 87.15 | 80.85 | 85.78 | 65.48 | 64.10 | 75.71 | 79.01 | ||||||||||||||||||
| QwQ-32B | Mistral-7B-v0.3 | 85.56 | 87.30 | 76.92 | 76.65 | 64.05 | 59.18 | 75.51 | 74.38 | |||||||||||||||||
| 64.79 | 57.07 | 62.67 | 58.02 | 29.75 | 23.56 | 52.40 | 46.22 | |||||||||||||||||||
| Ensemble | 72.83 | 72.58 | 76.19 | 81.08 | 58.60 | 55.02 | 69.21 | 69.56 | ||||||||||||||||||
| QwQ-32B | DeepSeek-R1 | 88.17 | 91.72 | 81.56 | 84.10 | 70.30 | 68.08 | 80.01 | 81.30 | |||||||||||||||||
| 65.28 | 58.02 | 60.40 | 55.69 | 29.75 | 23.56 | 51.81 | 45.76 | |||||||||||||||||||
| Ensemble | 83.62 | 84.47 | 78.82 | 78.82 | 58.67 | 53.53 | 73.70 | 72.27 | ||||||||||||||||||
| Debater#1 | Debater#2 | Protocol | Math Problem | Fact Verification | Answerability | Avg. F1 | Avg. F2 | |||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| (l2ptr2pt)4-11 | F1 (\(\uparrow\)) | F2 (\(\uparrow\)) | F1 (\(\uparrow\)) | F2 (\(\uparrow\)) | F1 (\(\uparrow\)) | F2 (\(\uparrow\)) | F1 (\(\uparrow\)) | F2 (\(\uparrow\)) | ||||||||||||||||||
| Human | - | - | 98.30 | 97.34 | 100.0 | 100.0 | 100.0 | 100.0 | 99.43 | 99.11 | ||||||||||||||||
| GPT4o-mini | - | - | 89.82 | 95.67 | 93.14 | 97.14 | 79.50 | 75.52 | 87.49 | 89.44 | ||||||||||||||||
| Llama3.1-70B | - | - | 90.78 | 96.10 | 91.45 | 93.75 | 82.87 | 81.12 | 88.37 | 90.32 | ||||||||||||||||
| Mistral-7B-v0.3 | - | - | 45.00 | 38.53 | 82.07 | 80.72 | 51.04 | 42.10 | 59.37 | 53.78 | ||||||||||||||||
| GPT4o-mini | Llama3.1-70B | 89.82 | 95.67 | 92.47 | 96.85 | 87.55 | 88.55 | 89.95 | 93.69 | |||||||||||||||||
| 82.07 | 81.10 | 78.18 | 70.84 | 51.34 | 41.59 | 70.53 | 64.51 | |||||||||||||||||||
| Ensemble | 90.46 | 95.95 | 93.43 | 96.82 | 81.30 | 78.62 | 88.40 | 90.46 | ||||||||||||||||||
| Llama3.1-70B | GPT4o-mini | 89.82 | 95.67 | 92.47 | 96.85 | 88.81 | 90.43 | 90.37 | 94.31 | |||||||||||||||||
| 88.89 | 93.51 | 89.47 | 91.12 | 76.99 | 73.13 | 85.12 | 85.92 | |||||||||||||||||||
| Ensemble | 90.46 | 95.95 | 93.43 | 96.82 | 81.30 | 78.62 | 88.40 | 90.46 | ||||||||||||||||||
| GPT4o-mini | Mistral-7B-v0.3 | 88.50 | 94.63 | 92.81 | 96.99 | 79.84 | 77.59 | 87.05 | 89.74 | |||||||||||||||||
| 85.71 | 86.31 | 80.53 | 74.23 | 59.70 | 50.76 | 75.31 | 70.43 | |||||||||||||||||||
| Ensemble | 64.76 | 57.24 | 59.51 | 51.52 | 63.81 | 55.83 | 62.69 | 54.86 | ||||||||||||||||||
| Llama3.1-70B | Mistral-7B-v0.3 | 90.14 | 95.81 | 90.91 | 94.41 | 84.50 | 84.10 | 88.52 | 91.44 | |||||||||||||||||
| 89.21 | 93.66 | 84.92 | 83.72 | 74.36 | 69.71 | 82.83 | 82.36 | |||||||||||||||||||
| Ensemble | 60.00 | 57.01 | 48.98 | 44.78 | 60.00 | 57.01 | 56.33 | 52.93 | ||||||||||||||||||
4.2 Empirical Results↩︎
The results of detecting errors in GPT-4 responses are given in Table 1, and the results for Llama-2 responses are given in Table 2. From the results, we have the following findings:
Single-agent LLMs still struggle with error detection. Although frontier LLMs have gained lots of improvements in the past year, when incorporated in error detection, even the powerful reasoning models like DeepSeek-R1 and
QwQ-32B, still suffer from a detection rate of the underlying errors.
Pitfalls of CopMAD. Aligned to our discussion, we can find that CopMAD protocol often leads to performance degeneration due to its zero-sum nature. For example, when adopting GPT4o-mini and
Llama3.1-70B as the debaters, CopMAD will decrease the F1 score by up to 13%, F2 score by up to 15%. Consequently, CopMAD even underperforms the simple Ensemble approach. Moreover, when coupling a relatively LLM with
a relatively weak LLM, such as Llama3.1-70B and DeepSeek-R1, Llama3.1-70B can be easily defeated by DeepSeek-R1 as DeepSeek-R1 has better debate skills, leading to a more significant
performance drop. The significant performance drops indicate that the CopMAD can not effectively realize the potential of the multiple LLMs.
Effectiveness of ColMAD. As shown in Table 1, across all settings, ColMAD significantly outperform CopMAD by a large margin under both F1 and F2 metrics.
Compared to single-agent performance, we can also find that ColMAD also brings non-trivial improvements (e.g., up to 4% when using GPT4o-mini and Llama3.1-70B), indicating an effective leverage of the diverse
knowledge of different LLMs. Furthermore, the improvements of ColMAD are general and robust across different combinations of LLMs that differ relatively large in their capabilities. For example, when combining Llama3.1-70B and
Mistral-7B-v0.3, as Mistral-7B-v0.3 is relatively weak, both CopMAD and especially the Ensemble method will be biased, while ColMAD remain bringing improvements over Llama3.1-70B; When
combining Llama3.1-70B and DeepSeek-R1, ColMAD effectively mitigates the degeneration led by debate hacking, and improves effectively over both Llama3.1-70B and DeepSeek-R1. Interestingly,
the knowledge of the reasoning model generically helps with the error detection when incorporated as Debater #2, while the reasoning model like QwQ-32B may not be as capable as the non-reasoning model like Llama3.1-70B in
leveraging the knowledge from the other debater.
Transferability of ColMAD to different candidate LLMs. Combining the results of Table 2, although detecting errors of Llama-2 responses is relatively easier than that of GPT-4, we can
also find that ColMAD outperforms CopMAD as well as the best single-agent performance by up to 4%. The results indicate the generality of the advantages of ColMAD over CopMAD.
Figure 5: (a) shows the results in F2 scores of ColMAD performance under different combinations of LLMs, where the diagonal line shows the single-agent performance. (b) shows the rate of alignment to the ground-truth
explanations given by CopMAD and ColMAD, where “ML-" refers to the combination of GPT4o-mini and Llama3.1-70B, and”LD-" refers to the combination of Llama3.1-70B and DeepSeek-R1.
ColMAD yields more reasonable explanations than CopMAD..
Figure 6: ColMAD is generically robust to the number of rounds for debate..
4.3 Ablation Studies↩︎
Different combinations of LLMs. In previous experiments, we set the first debater LLM as the judge LLM by default. To examine the influence of the judge implementation on the performance, we study different combinations of
Llama3.1-70B, Qwen-2.5-72B, and QwQ-32B that switch the orders. The results in F2 scores are given in Fig. 5 (a), where the diagonal line is the single-agent performance. It can be
found that the selection of judge LLM has a certain influence on ColMAD, while generically ColMAD maintains better improvements than single-agent.
Faithfulness of explanations. To examine whether ColMAD facilitates the identification of the correct predictions with correct explanations, we use LLM-as-a-judge to evaluate the alignment between the explanations given by
ColMAD and by CopMAD, respectively. The results are shown in Fig. 5 (b), it can be found that ColMAD yields more human-aligned explanations and reasoning in error detection, which
provides useful insights for scalable oversight.
Influence of debating rounds. In experiments, we set the number of debate rounds to \(2\) following the previous practice. In Fig. 6, we also examine the
sensitivity of ColMAD and CopMAD to the debate rounds. The results show that ColMAD is generically robust to different numbers of debate rounds.
5 Conclusions↩︎
In this work, we investigated using MAD to detect errors in LLM responses, which is a central task for scalable oversight. Our results show that previous CopMAD protocols can suffer from debate hacking due to the zero-sum
nature, where the persuasiveness of the debater agents can mislead the debating results. To mitigate the issue, we proposed ColMAD that asks the debaters to collaborate instead of combating. Empirical results with extensive experiments show
that ColMAD enables significantly better error detection capabilities, offering a new perspective for scalable oversight.
Acknowledgment↩︎
During the development of the project, we would like to acknowledge and thank Yu Mao for her insightful suggestions on prompt design and presentation.
LLM Use Statement↩︎
From the research side, this work studies the use of LLMs to perform multi-agent debate to detect errors in LLM responses. From the paper writing side, we use LLMs to assist with improving the writing of this work.
Ethics Statement↩︎
This work does not involve human subjects or personally identifiable information beyond public benchmarks used under their licenses. Our experiments evaluate error detection and decision protocols among LLMs, which benefits the oversight and prevents potential risks of superhuman intelligence in the future.
Reproducibility Statement↩︎
We will provide an anonymized link to our code upon the agreement of chairs during the discussion period. We also document the necessary details, including the prompts and experimental setups to reproduce our results.
6 More Details about Theories↩︎
6.1 Notations↩︎
A table of notations used in our work is given in Table 3.
| Notation | Meaning |
|---|---|
| \(y\in\{0,1\}\) | True label (e.g., error vs no_error). |
| \(Y\) | Random variable of the label predictions. |
| \(\vy\) | Predictions of the labels over a dataset. |
| \(\pi\) | Prior \(\Pr(y=1)\); prior log-odds \(\log\frac{\pi}{1-\pi}\). |
| \(X_0=(a,b)\) | Baseline signals from the two base models A, B (what the judge has without debate). |
| \(p(x\mid y)\) | Likelihood of baseline signal \(x\) under label \(y\). |
| \(\Lambda_0(x)=\log\frac{p(x\mid y=1)}{p(x\mid y=0)}\) | Baseline log–likelihood ratio (LLR), i.e., weight of evidence from \(X_0\). |
| \(m_A,\,m_B\) | Debate messages emitted by debater A and B. |
| \(M=(m_A,m_B)\) | Joint debate messages. |
| \(p_i(m_i\mid y,x)\) | Conditional likelihood model of debater \(i\)’s message given \(y\) and \(x=X_0\). |
| \(\ell_i(m_i;x)=\log\frac{p_i(m_i\mid y=1,x)}{p_i(m_i\mid y=0,x)}\) | Debater \(i\)’s additive LLR contribution given message and context. |
| \(|\ell_i|\le L_i\) | Bounded manipulability / persuasion budget for debater \(i\). |
| \(\Lambda(x,m_A,m_B)\) | Total LLR used by the judge: |
| \(\displaystyle \Lambda_0(x)+\ell_A(m_A;x)+\ell_B(m_B;x)+\log\frac{\pi}{1-\pi}\). | |
| \(J\) | Judge decision rule mapping \((X_0,m_A,m_B)\mapsto\{0,1\}\). |
| \(R(Z)\) | Bayes (minimum) \(0\)–\(1\) risk achievable when using signal \(Z\). |
| \(R_\mathrm{base}:=R(X_0)\) | Baseline Bayes risk using only the base signals (no debate). |
| \(V_\mathrm{adv}\) | Minimax error in adversarial (zero-sum) debating. |
| \(R_\mathrm{coop}\) | Bayes risk under cooperative, truth-seeking debating (messages add true evidence). |
| \(I(y;M\mid X_0)\) | Conditional mutual information: new information from messages beyond \(X_0\). |
| \(\eta(z)=\Pr(y=1\mid z)\) | Posterior probability under observable \(z\) (e.g., \(z=X_0\) or \(z=(X_0,M)\)). |
| \(\frac{1}{1+e^{|\Lambda|}}\) | Instantaneous Bayes error at balanced prior (\(\pi=\tfrac12\)) for total LLR \(\Lambda\). |
6.1.0.1 Debate setup.
We consider two agents \(A\) (Alice) and \(B\) (Bob), and denote the predictions of the error detections as \(\hat{{\boldsymbol{y}}}_A\) and \(\hat{{\boldsymbol{y}}}_B\), with rationales (e.g., CoT reasoning) as \(x_A\) and \(x_B\), respectively. During the debate, they will emit messages \(m_A\) and \(m_B\) to convince a judge \(J\).
Assumption 2 (Optimal Judge Strategy). Denoting the label as \(Y\in\{0,1\}\) with prior probability \(\pi\in(0,1)\), without debating, the judge will derive the final answer based on the initial responses \(X_0=(x_A,x_B)\). Assuming the judge \(J\) uses the Bayes test on the total log-likelihood ratio (LLR), we will have the LLR based on \(X_0\) as \[\label{eq-appdx:llr95x0} \Lambda_0(X_0)=\log\frac{p(X_0|Y=1)}{P(X_0|Y=0)}+\log\frac{\pi}{1-\pi}.\qquad{(4)}\] The contribution to LLR from the debating can be written as \[\label{eq-appdx:llr95contrib} l_i(m_i;X_0)=\log\frac{P(m_i|Y=1,X_0)}{P(m_i|Y=0,X_0)},\;i\in\{A,B\}.\qquad{(5)}\] Then, with debating, the LLR of the judge is \[\label{eq-appdx:llr95debating} \Lambda(X_0,m_A,m_B)=\Lambda_0(X_0)+l_A(m_A;X_0)+l_B(m_B;X_0)+\log\frac{\pi}{1-\pi}.\qquad{(6)}\]
Intuitively, if the elicited messages \(M=(m_A,m_B)\) bring additional information to the judge, i.e., \(I(Y;M|X_0)>0\) where \(I(\cdot;\cdot)\) denotes the mutual information, we will have \(\Lambda(X_0,M)>\Lambda(X_0)\). In order to provide additional information, we need to look into the cases where the predictions by \(A\) differ from those by \(B\). Under \(Y_A\neq Y_B\), if both agents are able to provide sufficient justifications and are more persuasive during the debate when they are debating for the correct answer. Hence, the judge can be convinced to take the correct answer when either of the agents is correct. The reduction of errors from the oracle collaboration can be calculated through \[\label{eq-appdx:oracle95collab} \min_{K\in\{A,B\}}\sum_{i}\mathbb{1}[\hat{{\boldsymbol{y}}}_K^{(i)}\neq{\boldsymbol{y}}^{(i)}]-\sum_{i}\mathbb{1}[\hat{{\boldsymbol{y}}}_J^{(i)}\neq{\boldsymbol{y}}^{(i)}],\tag{2}\] where \({\boldsymbol{y}}^{(i)}_K\) denotes the initial prediction of the agent \(K\in\{A,B\}\) on the \(i\)-th sample. Intuitively, Eq. (2 ) can be considered as the potential of the collaboration between \(A\) and \(B\).
6.2 Proof for Proposition 1↩︎
Proposition 3 (Restatement of Proposition 1). Assuming a bounded LLR, i.e., \(l_i\leq L_i<\infty,\;i\in\{A,B\}\), let \(R(Z)\) denote the Bayes risk of the judge \(J\) when making decisions based on \(Z\), denote \(R_0=R(X_0)\), denote the outcome of zero-sum debating as \[V_\mathrm{comp}=\min_J\max_{e\in\mathcal{E}(J)} P(J(X_0,M_e)\neq Y),\] where \(M_e\) are the debating transcripts given by the optimal Nash equilibrium \(e\in\mathcal{E}(J)\) of the \(A\)–\(B\) subgame in convincing \(J\). Then, we have \(V_\textrm{comp}=R_0\).
Proof. To show \(V_\textrm{comp}=R_0\), we need to show \(V_\textrm{comp}\leq R_0\), and \(V_\textrm{comp}\geq R_0\), and provide a condition that the equity holds.
(i) For \(V_\textrm{comp}\leq R_0\), given that the judge aims to choose the optimal strategy given any strategies of \(A\) and \(B\) that may degenerate the debate performance. We first consider a simple ignore strategy for the judge, \(J_\mathrm{ignore}\) that directly drops any additional debate transcripts between \(A\) and \(B\). We have \[P(J_\mathrm{ignore}(X_0,M_e)\neq Y)=P(J_\mathrm{ignore}(X_0)\neq Y)=R_0,\] which follows \[\max_{e\in\mathcal{E}(J)}P(J_\mathrm{ignore}(X_0,M_e)\neq Y)=R_0.\] Then, it suffices to know that \[V_\mathrm{comp}=\min_J\max_{e\in\mathcal{E}(J)} P(J(X_0,M_e)\neq Y)\leq \max_{e\in\mathcal{E}(J)}P(J_\mathrm{ignore}(X_0,M_e)\neq Y)=R_0,\] and \(V_\textrm{comp}\leq R_0\).
(ii) For \(V_\textrm{comp}\geq R_0\), we consider the strategies of the debaters. Without loss of generality, we assume \(A\) is debating for \(Y=1\) and \(B\) is debating for \(Y=0\). Since we do not impose any limits on the capabilities of the debater agents, they will try to present the evidence as most useful for the respective answer as they can. More formally, the optimal strategies for debaters \(A\) and \(B\) will yield the following \[M\mathrel{\perp\!\!\!\perp} Y |X_0.\] It follows that \[P(Y=1|X_0,M)=P(Y=1|X_0).\] Therefore, it suffices to know that \(V_\textrm{comp}\geq R_0\). That concludes our proof. ◻
6.3 Proof for Proposition 2↩︎
Proposition 4 (Restatement of Proposition 2). Under the same setting as Proposition 1, when the two agents are collaborating, we have the value as \[V_\mathrm{comad}=\min_J\min_{e\in\mathcal{E}_\mathrm{comad}(J)}P(J(X_0,M_e)\neq Y),\] where the \(M_e\) are the debating transcripts given by the optimal Nash equilibrium \(e\in\mathcal{E}_\mathrm{comad}(J)\) of the \(A\)-\(B\) subgame in collaboratively seeking the truth. Then, we have \(V_\mathrm{comad}\leq R(X_0)=V_\textrm{comp}\), where the strict inequality holds when \(I(Y;M_e|X_0)>0\).
Proof. The proof for Proposition 2 is relatively simple. Similar to the proof for Propositions 1, we can still establish that \(V_\mathrm{comad}\leq R_0\).
Furthermore, when \(I(Y;M_e|X_0)>0\), we know that with positive probability the posterior \(\eta(X_0,M_e)=P(Y=1|X_0,M_e)\) differs from \(\eta(X_0)=P(Y=1|X_0)\). Without loss of generality, if \(R\) is implemented as as \(0\)-\(1\) loss, \(R(X)=1-\mathrm{max}\{\eta(X),1-\eta(Z)\}\). As with a possible probability that there exists \((X_0,M_e)\) such that \[\max\{\eta(X_0,M_e),1-\eta(X_0,M_e)\}>\max\{\eta(X_0),1-\eta(X_0)\},\] then it follows that \[R(X_0,M)<R(X_0),\] which implies that there exists a better strategy for the judge to decrease the risk. ◻
7 More Results on Potential of Multi-Agent Collaboration↩︎
Given ReaLMistake, we provide more results on the error reduction of multi-agent collaboration under the oracle protocol as Eq. (2 ).
Figure 7: Error reductions of prevalent LLMs in detecting errors of . The numbers refer to the reduced errors following the oracle collaboration via Eq. (1 ). It can be found that different LLMs are less likely to make mistakes simultaneously. When incorporating LLMs with higher heterogeneity, such as those from different companies, the error reduction rates will be higher..
Figure 8: Error reductions of prevalent LLMs in detecting errors of . The numbers refer to the reduced errors following the oracle collaboration via Eq. (1 ). It can be found that different LLMs are less likely to make mistakes simultaneously. When incorporating LLMs with higher heterogeneity, such as those from different companies, the error reduction rates will be higher..
8 Details of the Debate↩︎
8.1 More details on ColMAD↩︎
The algorithm of ColMAD is given in Alg. 9, and the critical prompts of ColMAD are given as follows:
8.1.0.1 Debater prompt.
The debater agent is encouraged to collaborate with the other debater agent to provide comprehensive information to the judge to make a well-informed decision. In addition to clearly state the objective of collaboration for seeking the truthful answer,
we implement the insights of ColMAD into specific prompting schemes: (i) Evidence verification: ColMAD implements a quote-based system that asks the debaters to quote evidence from the context, and each quote will be verified if
there is an exact match in the context, following kenton2024on?; (ii) Self-auditing: the debaters are required to self-audit if
there exists one potential failure mode in the claim; (iii) Confidence calibration: the debaters are required to provide a confidence estimate for their own claims.
8.1.0.2 Judge prompt.
The Judge agent is instructed to analyze the debating transcripts to
8.1.0.3 Problem.
The problem statement is organized as follows:
8.1.0.4 Transcripts.
The transcripts are organized as follows:
9 More Details about Experiments↩︎
clccrrrrr|r & & & &
(l2ptr2pt)4-5 (l2ptr2pt)6-10 & & & & & & & & &
& & & & & & & & &
& Math Word Problem Generation & 140 & 252 & 151 & 25.0 & 57.1 & – & – & 62.1
& Fine-grained Fact Verification & 140 & 523 & 83 & 25.7 & 5.7 & 45.0 & – & 62.9
& Answerability Classification & 140 & 119 & 75 & 22.1 & – & 8.6 & 40.7 & 62.1
& Math Word Problem Generation & 160 & 235 & 163 & 51.2 & 67.5 & – & – & 80.0
& Fine-grained Fact Verification & 160 & 511 & 168 & 56.9 & 44.4 & 45.6 & – & 80.6
& Answerability Classification & 160 & 119 & 96 & 48.1 & – & – & 48.1 & 81.2