1 Introduction↩︎

Inference-time computation has emerged as a complementary scaling axis to parameter scaling for large language models [1]. Among inference-time strategies, multi-LLM collaboration leverages collective reasoning by aggregating outputs from multiple agents, with paradigms ranging from majority voting [2] to multi-agent debate [3], [4]. A more recent paradigm, Mixture-of-Agents (MoA) [5], organizes \(N\) proposer agents into \(L\) stacked layers followed by a final aggregator, enabling iterative refinement across depth. In principle, increasing depth and width should provide a natural path toward scalable collective reasoning. In practice, however, existing MoA systems fail to sustain gains as architectures deepen, often degrading, plateauing, or saturating after only a few layers.

Existing MoA variants attempt to sustain scaling through three broad strategies. The first strategy, information consolidation, curates the inputs of each layer to reduce redundancy, exemplified by RMoA [6], which selects a diverse subset of prior outputs and forwards only the inter-layer residuals between adjacent layers. The second, inter-agent interaction, exemplified by AttentionMoA [7], enables agents to refine each other within a layer through natural-language attention. The third, quality-guided selection, exemplified by VeriMoA [8], maintains a global cache of the highest-scoring solutions but requires domain-specific verifiers such as hardware description language simulators, limiting its applicability to specialized tasks.

While these methods differ architecturally, none sustains performance as MoA depth increases. Three mechanisms drive this scaling collapse [6], [7]. First, raw inter-layer concatenation propagates both useful reasoning and accumulated errors, allowing mistakes to compound across depth. Second, architectures with only local inter-layer communication fail to preserve strong reasoning trajectories generated earlier in the hierarchy. Third, as agents consume increasingly similar accumulated context, exploration diversity collapses and agents converge toward redundant solution paths, undermining the core benefit of multi-agent reasoning [9], [10]. More importantly, retaining memory alone is insufficient: exposing all agents to identical accumulated reasoning can itself accelerate convergence toward repetitive trajectories. Sustained MoA scaling therefore requires simultaneously preserving high-quality reasoning across layers while maintaining diversity among agents within each layer.

To address these scaling failures, we propose ReM-MoA, a Mixture-of-Agents framework centered on a Ranked Reasoning Memory that stores reasoning traces produced across the layers. Unlike episodic or retrieval-based memory systems that accumulate experience across tasks, ReM-MoA maintains a structured cross-layer reasoning memory within a single problem instance, enabling deeper layers to revisit prior reasoning paths. At each layer, proposer agents generate reasoning traces, and a Reviewer Agent [11] comparatively evaluates these traces by assigning scores together with targeted rationales. The ranked traces and rationale are then stored in memory [12]. Subsequent layers consult this memory through a Curated Diversified Memory Routing scheme that exposes different agents to distinct combinations of high-quality and low-quality reasoning trajectories, providing contrastive reasoning signals while preserving exploration diversity.

ReM-MoA therefore treats MoA scaling as a joint quality-diversity preservation problem rather than solely an aggregation or interaction problem. As an optional enhancement, the Reviewer Agent can be further improved through frontier-model distillation [13], where a strong teacher model supervises a smaller reviewer fine-tuned using parameter-efficient LoRA [14]. Multi-domain training produces a general-purpose Reviewer that transfers across reasoning tasks. Our contributions are as follows:

• We empirically characterize three scaling failure modes in existing MoA systems: degradation under raw concatenation, early plateauing under local residual propagation, and diversity collapse under intra-layer interaction mechanisms. These observations motivate structured cross-layer reasoning memory as a critical mechanism for sustained MoA scaling.

• We propose ReM-MoA, which sustains MoA scaling through a Ranked Reasoning Memory and a Curated Diversified Memory Routing scheme. At each layer, a comparative Reviewer Agent ranks reasoning traces with rationales, and subsequent agents receive distinct combinations of successful and failed traces, jointly preserving quality and diversity. An optional frontier-model distillation pipeline further upgrades the Reviewer Agent through multi-domain training.

• Across five reasoning benchmarks (math, formal logic, code, knowledge, and commonsense), ReM-MoA consistently outperforms prior MoA variants across all tested depth and width configurations, with the performance gap widening rather than diminishing as the architecture scales.

2 Related Work↩︎

2.0.0.1 Mixture-of-Agents variants.

Building on the original MoA framework [5], several variants extend the three paradigms from Section 1. Within information consolidation, beyond RMoA [6], SMoA [15] sparsifies agent interactions via dedicated judge and moderator agents, while Self-MoA [10] instead argues quality dominates diversity, aggregating samples from a single strong model. Within inter-agent interaction, beyond AttentionMoA [7], DEI [9] ensembles specialist agents with complementary expertise. The quality-guided paradigm is exemplified by VeriMoA [8] and related agentic frameworks for hardware description language generation [16]–[18], which rely on domain-specific feedback such as hardware simulators. No prior MoA variant maintains a structured cross-layer memory of reasoning traces accessible to subsequent layers.

2.0.0.2 Memory in LLM agents.

Explicit memory has been used for LLM agent self-improvement: Reflexion [19] stores verbal self-reflection from past failures, ReasoningBank [12] distills reasoning strategies from successful and failed trajectories across tasks, and AgentAuditor [20] augments LLM evaluators with retrieval-augmented examples for safety judgment. Our Ranked Reasoning Memory instead operates within a single problem across MoA depth, holding ranked traces from each layer for subsequent layers to consult.

2.0.0.3 LLM-as-Judge and judge distillation.

LLM-as-Judge [11] has emerged as a scalable proxy for human preference, with pairwise comparison generally more reliable than independent scoring [21]. JudgeLM [22] distills frontier-model judgments into open-source judges via knowledge distillation [13] and parameter-efficient fine-tuning [14]. Our Reviewer Agent extends this with a layer-wise comparative ranking that feeds a downstream reasoning memory.

3 Preliminaries: MoA Scaling↩︎

3.0.0.1 The standard MoA framework.

The original Mixture-of-Agents framework [5] defines the standard MoA structure: \(N\) proposer agents are organized into \(L\) sequential layers followed by a final aggregator. Layer 1 receives the original problem \(x\) and produces \(N\) candidate responses; each subsequent layer \(l \in \{2, \ldots, L\}\) receives the previous layer’s outputs together with \(x\) and produces \(N\) new responses; the aggregator then synthesizes the outputs of layer \(L\) into a final answer. Specific variants modify aspects of this structure (Section 2).

3.0.0.2 Quality and diversity drive scaling.

[10] model MoA accuracy through a linear regression \[t = \alpha q + \beta d + \gamma, \label{eq:moa-perf}\tag{1}\] where \(q\) is the average proposer accuracy and \(d\) is the Vendi Score of proposer outputs. With \(\alpha > \beta\) across datasets, MoA performance depends on both factors but is more sensitive to quality. Sustained scaling therefore requires mechanisms that preserve quality and diversity simultaneously, since gains in one cannot offset losses in the other.

3.0.0.3 Empirical scaling failures.

Figure 1 plots the accuracy of three representative MoA variants under two natural scaling axes: layer depth (top row, \(L\) from 1 to 9 with width fixed at \(N=3\)) and layer width (bottom row, \(N\) from 3 to 8 with depth fixed at \(L=3\)). The depth axis exposes three distinct failure patterns. Standard MoA [5] peaks at shallow depths and then degrades, with the drop reaching nearly 10 percentage points on MMLU-redux. RMoA [6] avoids this degradation but plateaus early, because its adjacent-layer residual cannot retrieve high-quality reasoning produced in non-adjacent earlier layers. AttentionMoA [7] improves but saturates before deeper layers (with slight late decline on MMLU-redux), since intra-layer attention promotes convergence among the layer’s agents. Width scaling exhibits the same hierarchy at smaller magnitude: Standard MoA still degrades on MMLU-redux, while RMoA and AttentionMoA improve more slowly with diminishing returns.

3.0.0.4 Design requirements.

The three patterns above instantiate the failure mechanisms outlined in Section 1 and show that existing variants violate the quality-diversity balance of Equation 1 as MoA scales. Sustaining scaling therefore requires preserving both factors of this equation:

Q (Quality Preservation): A scalable MoA should retain the quality of reasoning produced at each layer so that deeper layers build on it rather than dilute it. Both high-quality successful traces (positive guidance) and low-quality failed traces (counter-examples) enrich the input signal beyond positive examples alone. This calls for presenting traces in a quality-ranked form rather than raw concatenation, and keeping them accessible across all earlier layers.

D (Diversity Preservation): A scalable MoA should also preserve exploration diversity among agents within each layer, so that they do not collapse onto a single solution path as memory accumulates. To this end, different agents in the same layer should engage with different selections from the memory rather than sharing identical inputs.

ReM-MoA, introduced in Section 4, addresses both Q and D through a Ranked Reasoning Memory paired with a curated diversified memory routing scheme: the memory ranks and retains traces from every layer, and the routing scheme delivers quality-aware reference sets to within-layer agents.

4 ReM-MoA↩︎

ReM-MoA addresses the failure patterns observed in Section 3 through three interconnected components: a Ranked Reasoning Memory \(\mathcal{M}\), a curated diversified memory routing scheme \(\Pi\), and an optional reviewer distillation pipeline. Let \(\mathcal{A}_l = \{A_{l,1}, \ldots, A_{l,N}\}\) denote the \(N\) proposer agents at layer \(l \in \{1, \ldots, L\}\), and let \(r_{l,j} = A_{l,j}(P_{l,j})\) denote the reasoning trace produced by agent \(A_{l,j}\) given prompt \(P_{l,j}\). We write \(\mathbf{r}_l = (r_{l,1}, \ldots, r_{l,N})\) for the vector of all traces produced at layer \(l\). Figure 2 illustrates the framework: the memory ranks and retains \(\mathbf{r}_l\) at every layer (Section 4.1); the routing scheme delivers distinct curated reference sets to within-layer agents (Section 4.2); and the distillation pipeline optionally enhances the memory’s ranking module (Section 4.3). After layer \(L\), the aggregator consults the memory \(\mathcal{M}\) through routes drawn from the same three families as the proposer agents, and synthesizes the final answer.

4.1 Ranked Reasoning Memory↩︎

4.1.0.1 Memory subsystem.

The Ranked Reasoning Memory \(\mathcal{M}\) is a shared subsystem that records ranked traces at every proposer layer. After layer \(l\) has been processed, the memory accumulates as the union of per-layer entries: \[\mathcal{M}_{\leq l} = \bigcup_{k=1}^{l} \mathcal{M}_k, \label{eq:memory-set}\tag{2}\] where \(\mathcal{M}_k\) is the entry committed by layer \(k\) (defined below). The memory is append-only and supports read access from any subsequent layer: for any \(l' > l\), layer \(l'\) may consult \(\mathcal{M}_{\leq l}\). This subsystem addresses Q (Quality Preservation) because traces are stored in quality-ranked form rather than raw concatenated outputs, and remain accessible across arbitrary layer distances rather than only between adjacent layers.

4.1.0.2 The Reviewer Agent.

The ranking inside the memory is performed by an LLM-based Reviewer Agent \(\mathcal{R}\). After the \(N\) proposers at layer \(l\) produce their traces \(\mathbf{r}_l\), the Reviewer is invoked once with all \(N\) traces presented together; it compares the traces against each other and emits, for each trace, an integer score together with a targeted rationale specific to that trace: \[(\mathbf{s}_l, \boldsymbol{\rho}_l) = \mathcal{R}(x, \mathbf{r}_l), \label{eq:reviewer}\tag{3}\] where \(\mathbf{s}_l = (s_{l,1}, \ldots, s_{l,N})\) with each \(s_{l,j} \in \{1, \ldots, S\}\) being an integer score on a scale with maximum \(S\) (higher is better), and \(\boldsymbol{\rho}_l = (\rho_{l,1}, \ldots, \rho_{l,N})\) with each \(\rho_{l,j}\) being a rationale specific to trace \(r_{l,j}\) (rationales are independent across traces). We use comparative scoring rather than independent scoring because grounding each trace’s score in relative differences across candidates produces more reliable estimates than absolute scoring [11].

4.1.0.3 Memory entries and access.

Each layer commits one entry to the memory, formed by zipping the traces with their Reviewer-assigned scores and rationales: \[\mathcal{M}_l = \{(r_{l,j}, s_{l,j}, \rho_{l,j})\}_{j=1}^N. \label{eq:memory-entry}\tag{4}\] \(\mathcal{M}_l\) is a set of \(N\) trace-score-rationale triples. The accumulated memory \(\mathcal{M}_{\leq l}\) thus contains \(N \cdot l\) triples available for cross-layer retrieval by the routing scheme described next.

4.2 Curated Diversified Memory Routing↩︎

4.2.0.1 Curated reference delivery.

At every subsequent layer, each proposer agent receives, in addition to the problem \(x\), a curated set of reference traces drawn from the memory. The selection follows two design principles. First, the reference set must carry quality-aware information for the agent’s reasoning: it comprises high-scoring traces as positive examples, low-scoring traces as negative examples, or both. Second, the reference sets must differ across the \(N\) agents within the same layer, so the layer as a whole explores the memory’s content rather than redundantly consuming the same references. Formally, at layer \(l \geq 2\), the curated diversified memory routing scheme \(\Pi\) operates on the global pool of all previously committed entries, \(\mathcal{P}_l = \mathcal{M}_{\leq l-1}\): \[\mathcal{S}_{l, j} = \Pi_j(\mathcal{P}_l), \label{eq:routing}\tag{5}\] where \(\mathcal{S}_{l, j} \subset \mathcal{P}_l\) is the curated reference set routed to agent \(A_{l, j}\). Operating on \(\mathcal{P}_l\) rather than only the most recent layer realizes the cross-layer access promised by the memory subsystem (Section 4.1). The prompt for agent \(A_{l, j}\) is then: \[P_{l, j} = \begin{cases} x & \text{if } l = 1, \\ x \oplus \mathcal{S}_{l, j} & \text{if } l \geq 2, \end{cases} \label{eq:prompt}\tag{6}\] where \(\oplus\) denotes concatenation; each entry in \(\mathcal{S}_{l, j}\) contributes its trace together with its score and rationale.

4.2.0.2 Routing functions.

We implement the two design principles above through three families of routing functions operating on \(\mathcal{P}_l\). Let \(\text{Top}_n(\mathcal{M})\) denote the size-\(n\) subset of \(\mathcal{M}\) containing the highest-scoring entries, and \(\text{Bot}_n(\mathcal{M})\) the analogous subset for the lowest-scoring ones. The three families are: contrastive, of the form \(\text{Top}_a \cup \text{Bot}_b\) (denoted \(\text{Con}_n\) with \(n=a+b\)), pairing high- and low-scoring entries; positive reinforcement, of the form \(\text{Top}_a\), emphasizing successful patterns; and failure-mode focus, of the form \(\text{Bot}_a\), surfacing common pitfalls. For each layer, the \(N\) routing functions \(\Pi_1, \ldots, \Pi_N\) are distributed across these three families, with the distribution chosen to balance the three modes of reference. As one concrete instance, for \(N=3\) we use one route per family: \[\Pi_j(\mathcal{P}_l) = \begin{cases} \text{Top}_1(\mathcal{P}_l) \cup \text{Bot}_1(\mathcal{P}_l) & j = 1, \\ \text{Top}_2(\mathcal{P}_l) & j = 2, \\ \text{Bot}_2(\mathcal{P}_l) & j = 3. \end{cases} \label{eq:routes-N3}\tag{7}\] Appendix 8 details the distributions used for larger \(N\) in our scaling experiments. This design realizes both Q (Quality Preservation) and D (Diversity Preservation). For Q, the curated reference sets preserve the Reviewer-extracted quality signal as it propagates forward through the layers. For D, the \(N\) distinct routes within each layer sustain exploration diversity even as the memory grows.

4.3 Reviewer Distillation (Optional Enhancement)↩︎

4.3.0.1 Distillation pipeline.

The Reviewer Agent \(\mathcal{R}\) is implemented as an off-the-shelf LLM by default. It can be optionally enhanced via frontier-model distillation [13]. Crucially, the training data for distillation comes from ReM-MoA’s own runs: we execute the framework on a multi-domain set of training problems, collect the proposer traces \(\mathbf{r}_l\) generated at each layer, and query a strong external teacher \(\mathcal{T}\) on each (problem, traces) pair to obtain high-quality supervision targets: \[\mathcal{D}_{\text{train}} = \big\{(x_i, \mathbf{r}_i, \mathbf{s}_i^*, \boldsymbol{\rho}_i^*) \mid (\mathbf{s}_i^*, \boldsymbol{\rho}_i^*) = \mathcal{T}(x_i, \mathbf{r}_i)\big\}, \label{eq:train-data}\tag{8}\] where \((x_i, \mathbf{r}_i)\) is sampled from the framework’s intermediate outputs and \((\mathbf{s}_i^*, \boldsymbol{\rho}_i^*)\) is the teacher’s score-rationale labeling treated as the supervision target. A smaller base Reviewer \(\mathcal{R}_\theta\) is then fine-tuned via parameter-efficient LoRA [14] to imitate the teacher: \[\theta^* = \arg\min_{\theta} \sum_{(x, \mathbf{r}, \mathbf{s}^*, \boldsymbol{\rho}^*) \in \mathcal{D}_{\text{train}}} \mathcal{L}\big(\mathcal{R}_\theta(x, \mathbf{r}),\; (\mathbf{s}^*, \boldsymbol{\rho}^*)\big), \label{eq:lora-loss}\tag{9}\] where \(\mathcal{L}\) is the standard supervised loss on the concatenated score-rationale output. Because the training distribution matches the traces ReM-MoA actually produces at inference, the resulting \(\mathcal{R}_{\theta^*}\) learns to judge exactly the distribution of traces it will encounter at deployment.

4.3.0.2 Multi-domain training.

The training problems span multiple reasoning domains. Multi-domain training is essential to the design: it lets a single Reviewer generalize across all benchmarks we evaluate, including held-out domains not seen during training (Section 5). Full training configuration and hyperparameters are described in Section 5.1.

2pt

3pt

5 Experiments↩︎

We evaluate ReM-MoA against existing MoA variants to answer three research questions:

5.0.0.1 RQ1.

Does ReM-MoA continue to improve as depth and width scale, where prior MoA variants degrade, plateau, or saturate?

5.0.0.2 RQ2.

How much do the Ranked Reasoning Memory and the Curated Diversified Memory Routing each contribute to ReM-MoA’s overall gain?

5.0.0.3 RQ3.

Does multi-domain Reviewer distillation transfer to held-out reasoning domains?

5.1 Experimental Setup↩︎

5.1.0.1 Benchmarks.

We evaluate on five reasoning benchmarks, one per target domain: MATH [23] (mathematics), MMLU-redux [24] (knowledge), the formal_logic subject of MMLU [25] (formal logic), CRUX [26] (code understanding), and HellaSwag [27] (common-sense reasoning). We report accuracy on all evaluation domains, except for the code benchmarks (CRUX and, in the distillation experiment, MBPP), for which we report pass@1.

5.1.0.2 Proposer pool and aggregator.

A heterogeneous pool of three 7-8B open-source instruction-tuned models (Qwen2.5-7B-Instruct [28], Llama-3.1-8B-Instruct [29], and Qwen2.5-Coder-7B-Instruct [30]) serves as proposers. Qwen2.5-7B-Instruct is the aggregator in all configurations.

5.1.0.3 Reviewer Agent.

We evaluate two variants of our framework that differ only in the Reviewer: ReM-MoA uses the off-the-shelf Qwen2.5-7B-Instruct as the Reviewer, while ReM-MoA\(^*\) replaces it with a LoRA-distilled fine-tune of the same base model. For the distillation, we query GPT-5.5 as the teacher \(\mathcal{T}\) on traces generated by ReM-MoA itself, then fine-tune Qwen2.5-7B-Instruct with LoRA [14]. Training data consists of 100 problems each from MATH, ARC-Challenge [31], MedQA [32], MBPP [33], and HellaSwag (500 problems total); ReM-MoA is run on each at \(L=3, N=3\), yielding \(1{,}500\) (problem, layer-traces) training tuples. Full LoRA configuration is in Appendix 9.

5.1.0.4 Baselines.

We compare against three MoA variants representing the paradigms introduced in Section 2: standard MoA [5] (information concatenation), RMoA [6] (information consolidation), and AttentionMoA [7] (inter-agent interaction). All baselines share our proposer pool and aggregator, retaining each baseline’s original hyperparameters.

5.1.0.5 Implementation.

All experiments run on NVIDIA A100 GPUs via vLLM [34]. Proposer agents sample at temperature 0.8, the Reviewer at 0.3, and the aggregator at 0.5. We evaluate along two axes: depth, sweeping \(L \in \{1, \ldots, 9\}\) at \(N=3\), and width, sweeping \(N \in \{3, \ldots, 8\}\) at \(L=3\). Further implementation details are in Appendix 9.

5.2 Sustained Scaling and Cross-Benchmark Results (RQ1)↩︎

Tables 1 and 2 report depth and width scaling across the five benchmarks. Figure 3 plots the corresponding mean accuracy.

5.2.0.1 Baseline failure modes are confirmed on all five benchmarks.

Standard MoA degrades on all five benchmarks, peaking by \(L{=}3\) and losing \(9.1\) points on MMLU-redux (\(61.5 \to 52.4\)), \(4.0\) on MATH, \(3.0\) on CRUX, \(2.7\) on HellaSwag, and \(1.4\) on Formal Logic by \(L{=}9\). RMoA plateaus on MMLU-redux (its representative failure mode in Figure 1) and saturates on the other four benchmarks, with tail-3 gain below \(+0.2\) on every non-plateau cell. AttentionMoA saturates on four benchmarks and plateaus on MMLU-redux (peaking at \(L{=}5\) then slightly declining), with per-step gain shrinking to \(\leq 0.1\) at \(L{=}9\), indicating intra-layer convergence among agents.

5.2.0.2 ReM-MoA outperforms every baseline across all configurations.

ReM-MoA\(^*\) is column-best with ReM-MoA second across all \(45\) benchmark-\(L\) and \(30\) benchmark-\(N\) cells. The lead is largest on reasoning-intensive benchmarks: at \(L{=}9\), ReM-MoA exceeds AttentionMoA by \(+7.3\) on Formal Logic (\(67.2\) vs \(59.9\)), \(+6.3\) on CRUX, and \(+4.1\) on MATH. On knowledge-heavy MMLU-redux and common-sense HellaSwag, where headroom is smaller, ReM-MoA still leads at every \(L\) but saturates due to limited remaining ceiling.

5.2.0.3 The gap widens monotonically with scale.

Figure 3 renders the spectrum of scaling behaviors in one view and shows a clean monotonic fan-out among the five methods along the depth axis. Quantitatively, ReM-MoA’s gap over the strongest baseline grows from \(+0.84\) at \(L{=}1\) to \(+4.92\) at \(L{=}9\) (\(5.9{\times}\)); against Standard MoA it expands from \(+2.72\) to \(+18.44\) (\(6.8{\times}\)). Width yields a more modest expansion from \(+2.30\) at \(N{=}3\) to \(+3.68\) at \(N{=}8\), with the five methods following more closely-spaced trajectories than under depth (right panel of Fig. 3). The widening holds against all three baselines across the five benchmarks.

5.2.0.4 Distillation extends sustained scaling to all benchmarks.

ReM-MoA\(^*\) improves accuracy across all configurations, with gains over ReM-MoA of up to \(3.0\) points at \(L{=}9\) on MATH and Formal Logic. Distillation also flips the Trend label on the three cells where ReM-MoA still saturated (MMLU-redux and HellaSwag depth, MMLU-redux width), all moving to \(\uparrow\uparrow\). ReM-MoA\(^*\) thus sustains scaling on all five benchmarks under both axes.

2pt

5.3 Ablation Studies (RQ2)↩︎

We ablate three ReM-MoA components independently: cross-layer access (restrict reads to \(\mathcal{M}_{l-1}\) only), routing diversification (share one reference set across the layer’s \(N\) proposers), and rationale (remove rationales from memory entries). All variants use the off-the-shelf Reviewer (ReM-MoA, not ReM-MoA\(^*\)). Table 3 reports depth scaling.

5.3.0.1 Cross-layer access and routing drive sustained scaling; rationale refines magnitude.

Restricting reads to \(\mathcal{M}_{l-1}\) causes the largest drop (avg \(-4.28\) at \(L{=}9\), peak \(-6.2\) on Formal Logic), with all three previously sustained benchmarks collapsing from \(\uparrow\uparrow\) to \(\uparrow\rightarrow\) and mirroring RMoA’s adjacent-layer residual in Table 1. Sharing a single reference set across \(N\) proposers is the second-largest contributor (\(-2.50\) at \(L{=}9\)) and triggers the same regression on the three sustained benchmarks, confirming that within-layer diversification rather than memory richness alone drives sustained scaling. Removing rationales costs only \(-1.52\) at \(L{=}9\) but preserves the trend pattern fully (MATH, Formal Logic, and CRUX still sustain, MMLU-redux and HellaSwag still saturate), indicating that rationales act as refinement rather than scaffolding.

2pt

5.4 Reviewer Distillation (RQ3)↩︎

We compare three Reviewers on the five distillation domains (Section 5.1): Orig (off-the-shelf, used in ReM-MoA), Multi-FT (LoRA-distilled on all five domains, used in ReM-MoA\(^*\)), and LOO (distilled on only four domains, holding out the evaluation domain). Table 4 reports depth scaling at \(N{=}3\).

5.4.0.1 Distillation captures a transferable evaluation skill.

Multi-FT improves over Orig across all benchmarks and configurations (avg \(+1.92\) at \(L{=}9\), peak \(+3.0\) on MATH); on HellaSwag the gain even flips the trend from \(\uparrow\rightarrow\) to \(\uparrow\uparrow\), matching ReM-MoA\(^*\) in Table 1. LOO recovers \(44\%\) of Multi-FT’s average gain (\(+0.84\) at \(L{=}9\), ranging from \(67\%\) on MATH to \(21\%\) on HellaSwag) without seeing the held-out domain during training, while preserving Orig’s trend pattern. This transfer property explains why ReM-MoA\(^*\) in Table 1 generalizes to the three unseen evaluation domains (MMLU-redux, Formal Logic, CRUX).

6 Conclusion↩︎

We presented ReM-MoA, a Mixture-of-Agents framework that overcomes the scaling limitations of prior variants through a Ranked Reasoning Memory and a Curated Diversified Memory Routing scheme, jointly preserving quality and diversity across layers. Across five reasoning benchmarks under both depth and width scaling, ReM-MoA outperforms all baselines in every configuration, and the advantage over baselines widens, rather than diminishes, as depth grows. An optional multi-domain Reviewer distillation further extends sustained scaling to all benchmarks and transfers to held-out domains, establishing structured cross-layer memory as a critical missing mechanism for scalable multi-agent reasoning.

Limitations↩︎

6.0.0.1 Bounded width exploration.

We evaluate width scaling up to \(N{=}8\) at \(L{=}3\), matching the largest width studied in prior MoA work [5]–[7]. ReM-MoA still leads all baselines at \(N{=}8\) without visible saturation, but compute constraints prevented probing larger widths.

6.0.0.2 Single-scale proposer pool.

Our proposer pool consists of three 7-8B open-source instruction-tuned models, matching the scale most commonly studied in MoA literature. Whether the gains transfer to substantially larger proposers (e.g., 30B-70B) or smaller ones (e.g., 1-3B), where the quality-diversity trade-off may shift, remains to be empirically validated.

6.0.0.3 Reviewer Agent overhead.

ReM-MoA invokes one Reviewer call per layer, giving an inference cost of \(L \cdot N + L + 1\) LLM calls, on par with RMoA’s Residual Agent and substantially lower than AttentionMoA’s per-agent intra-layer attention (\(\sim 2L \cdot N\)). Reviewer distillation adds a one-time training cost and assumes access to a frontier-model teacher, but is optional: the off-the-shelf ReM-MoA without distillation already outperforms all baselines (Tables 1 and 2).

6.0.0.4 Inherited biases and computational cost.

ReM-MoA inherits the biases and failure modes of its constituent LLMs: the proposer agents may produce factually incorrect or socially biased reasoning, and the Reviewer Agent (itself an LLM) may exhibit known judge biases such as position bias or self-preference [11]. Our comparative scoring and contrastive routing reduce but do not eliminate these effects, and high-stakes deployment would require additional safeguards. The framework also incurs \(L{\cdot}N + L + 1\) LLM calls per query, with corresponding energy footprint; this overhead is on par with comparable MoA variants but higher than single-pass generation.

7 AI Usage↩︎

AI-based writing tools were used for limited language editing, paraphrasing, and polishing of the manuscript. All technical content, analysis, experimental results, and conclusions were developed and verified by the authors.

8 Routing Distributions for Larger Widths↩︎

Section 4.2 introduces the routing scheme and gives one concrete instance for \(N{=}3\) in Equation 7 . Table 5 extends this assignment to \(N \in \{4, \ldots, 8\}\), the range used in our width-scaling experiments (Table 2). The design principle is to keep the three route families (contrastive, positive reinforcement, failure-mode focus) represented at every layer while scaling the size of each route with \(N\), so that the reference sets cover more of the memory pool while remaining mutually distinct.

8.0.0.1 Handling sparse early layers.

At layer \(l\), the global pool \(\mathcal{P}_l = \mathcal{M}_{\leq l-1}\) contains \(N \cdot (l-1)\) entries. When a route requests more entries than are available (e.g., \(\text{Con}_4\) at \(l{=}2, N{=}3\), where only 3 entries exist), the route is truncated to the available entries. This truncation affects only the first one or two layers; once \(l \geq \lceil n/N \rceil + 1\), all routes operate at their nominal size.

9 Experimental Setup Details↩︎

This appendix expands the configuration referenced in Section 5.1, covering the Reviewer distillation pipeline, LoRA configuration, training hyperparameters, train/test isolation, and inference-time settings.

9.1 Distillation Variants↩︎

We evaluate three Reviewer configurations:

• Orig: the off-the-shelf base Reviewer (Qwen2.5-7B-Instruct), no fine-tuning. Used in ReM-MoA.

• Multi-FT: LoRA-distilled jointly on five domains (MATH, ARC-Challenge, MedQA, MBPP, HellaSwag; 100 problems each, 500 total). Used in ReM-MoA\(^*\).

• LOO: five variants, each LoRA-distilled on four of the five domains (400 problems per variant), holding out the evaluation domain. Used to assess cross-domain transfer in Section 5.4.

9.2 Trace Generation and Teacher Labeling↩︎

For each training problem, we run ReM-MoA at \(L{=}3, N{=}3\) to collect the proposer traces produced at every layer, yielding \(\sim\)​9 traces per problem. Each (problem, traces) tuple is then sent to a frontier-model teacher, GPT-5.5, which receives the problem, the proposer traces, and the ground-truth answer, and is constrained by a structured prompt (Appendix 10.5) to emit labels conforming to the Reviewer’s output schema — a per-trace integer score and a contrastive rationale. The ground-truth answer appears only in the teacher’s prompt and is never exposed to the student during training. Traces are filtered by length (50 to 2,000 tokens) before training to remove degenerate generations.

9.3 LoRA Configuration↩︎

LoRA [14] is applied to the attention projection matrices of Qwen2.5-7B-Instruct.

9.4 Training Hyperparameters↩︎

Each distillation run completes in approximately 2 hours on a single NVIDIA A100 (80GB) GPU. The full distillation pipeline—one Multi-FT plus five LOO variants (Appendix 9.1)—requires approximately 12 A100-hours of training compute in total.

9.5 Train/Test Isolation↩︎

Fine-tuning data is drawn exclusively from each dataset’s publicly available training split, while evaluation uses the held-out test split. For HellaSwag, we evaluate on the validation split since test labels are not public, following standard practice. Because train and test problems are disjoint by construction, label leakage from the GPT-5.5 teacher into evaluation is not possible at the problem level. The structured evaluation prompt (Appendix 10.5) further constrains the teacher’s output to fixed dimensions (score and rationale), focusing the student on the meta-skill of judging reasoning quality rather than memorizing canonical solutions. The LOO results in Section 5.4 additionally demonstrate that the student transfers to domains it never saw during training, confirming that the learned skill is general rather than domain-memorized.

9.6 Inference and Hardware↩︎

All experiments run on NVIDIA A100 (80GB) GPUs via vLLM [34]. Sampling temperatures follow Section 5.1: proposer agents at \(T{=}0.8\) (top-\(p\) 0.95), Reviewer at \(T{=}0.3\) (top-\(p\) 0.9), aggregator at \(T{=}0.5\) (top-\(p\) 0.9). Maximum new tokens per call: 2,048 for proposers and aggregator, 8,192 for the Reviewer (which must emit \(N\) score-rationale pairs in a single response).

10 Prompt Templates and Scoring Rubric↩︎

This appendix provides the prompt templates used by the proposer agents, the Reviewer Agent, the aggregator, and the frontier-model teacher (GPT-5.5) during distillation, together with the scoring rubric followed by the Reviewer.

10.1 Scoring Scale and Rubric↩︎

The Reviewer assigns each trace an integer score on a \(1\)-\(10\) scale (i.e., \(S{=}10\) in Equation 3 ). Following common LLM-as-judge practice, scores are organized into five tiers:

• 9–10 — Excellent: reaches the correct answer through a complete, coherent chain of reasoning with no logical errors.

• 7–8 — Good: reaches the correct answer with minor reasoning slips, redundancies, or under-justified steps that do not affect the final result.

• 5–6 — Partial: demonstrates valid reasoning on parts of the problem but reaches an incomplete or partially incorrect final answer.

• 3–4 — Weak: contains substantive reasoning errors propagating to an incorrect final answer; some intermediate steps may still be useful as counter-examples.

• 1–2 — Poor: final answer is wrong and the reasoning chain contains major flaws throughout.

Within each tier, the higher score indicates a stronger instance (e.g., a 10 is preferred to a 9). The accompanying rationale (one per trace) cites concrete strengths and weaknesses of the specific trace, anchored in its reasoning steps and conclusion. Rationales are independent across traces (Section 4.1).

10.2 Proposer Prompt↩︎

At layer 1, the proposer receives only the problem. At layer \(l \geq 2\), the proposer additionally receives the curated reference set \(\mathcal{S}_{l,j}\) — the problem concatenated with the reference traces, each tagged with its Reviewer score and rationale.

At layer 1, the "Reference Reasoning Traces" block is omitted and the prompt contains only the problem and the task instruction.

10.3 Reviewer Prompt↩︎

The Reviewer is invoked once per layer with all \(N\) proposer traces of that layer. It emits scores and rationales for all \(N\) traces in a single response, parsed into the memory entry of Equation 4 .

10.4 Aggregator Prompt↩︎

After layer \(L\), the aggregator receives the problem and three curated reference sets drawn from the memory through the same three families used by proposer agents (Section 4), and produces the final answer.

10.5 Teacher Prompt for Distillation↩︎

The frontier-model teacher (GPT-5.5) receives the problem, the proposer traces, and the ground-truth answer; it is constrained to emit labels in the same JSON schema as the Reviewer. The rationale is required to justify the score in terms of internal reasoning quality only, never restating the ground-truth answer, so that the student learns to judge without access to ground truth.

References↩︎