1 Introduction↩︎

Large language model (LLM)–based multi-agent systems (MAS) exhibit collective intelligence that can surpass the capabilities of a single LLM, achieving strong performance in mathematical computation [1], code generation [2], software development [3], and scientific discovery [4]. Prior studies [5]–[7] have demonstrated that carefully designed interaction topologies enhance the task-processing capability of MAS through effective collaboration.

Research on MAS with structured communication topologies has predominantly relied on dialogue-based interaction paradigms, as illustrated in Figure 1. In earlier frameworks [8]–[11], agent connections are homogeneous and restricted to a single interaction mode, with information flow governed by static communication graphs lacking dynamic adaptation. While recent studies [5], [6] have achieved performance gains by hybridizing intra- and inter-round dialogues within spatio-temporal graphs, this multi-round design inherently imposes additional computational and token costs. Although mitigation strategies such as edge pruning [12], dynamic agent selection [7], [13], [14], and proxy-based optimization [15] attempt to balance efficiency and performance, structural redundancy remains unavoidable. Specifically, each inter-round interaction triggers a subsequent intra-round dialogue phase, recursively compounding communication overhead [16], [17].

Single-round dialogue schemes offer the advantage of reduced communication overhead; however, maintaining high task performance within such a constrained framework remains a challenge [18]. Drawing on recent insights that evaluation and debate mechanisms can improve problem-solving in multi-agent systems [19], [20], we posit that explicitly modeling diverse interactions through a heterogeneous communication graph yields a balance between efficiency and performance. Thus, we propose TopoDIM (One-shot Topology Generation with Diverse Interaction Modes). Specifically, TopoDIM is characterized by the following key features:

Diverse Interaction Modes. TopoDIM leverages three efficient collaborative argumentation primitives [21], enabling agents to engage in complementary forms of information exchange beyond homogeneous message passing. One-shot Heterogeneous Topology Generation. TopoDIM employs a heterogeneous graph encoder to capture agent- and task-specific contexts, coupled with an autoregressive decoder that generates a communication topology with multiple interaction modes in a single inference step, eliminating iterative dialogue rounds. Decentralized Architecture. To facilitate adaptive decision-making and mitigate privacy risks [22], TopoDIM distills the global topology optimizer into lightweight local networks deployed on individual agents, enabling autonomous and fully decentralized topology construction.

Compared to existing approaches, TopoDIM substantially reducing computational overhead while promoting more diverse collaboration patterns, ultimately leading to improved task-solving performance in multi-agent systems. Our main contributions are summarized as follows:

• Observation. We suggest that replacing iterative topology generation in both intra- and inter-round dialogue processes with a one-shot heterogeneous topology formulation significantly improves communication efficiency without performance trade-offs.

• Framework. We propose TopoDIM, a decentralized framework that integrates a heterogeneous graph encoder with an autoregressive decoder to generate multi-relational communication topologies in one shot, enabling autonomous agent-level decision-making.

• Evaluation. Extensive experiments demonstrate that TopoDIM consistently enhances communication efficiency, task performance, and structural robustness across both homogeneous and heterogeneous multi-agent settings, surpassing strong task-adaptive cooperation baselines.

2 Methodology Overview↩︎

TopoDIM dynamically orchestrates conditioned, feedback, and dialectical behaviors within a single execution to leverage the complementary strengths of LLM agents and heterogeneous interactions. The schematic overview of TopoDIM is presented in Figure 2.

2.1 Communication Topology and Pipeline↩︎

Communication topology. We model LLM-based MAS as a directed heterogeneous graph \(\mathcal{G} = (\mathcal{V}, \mathcal{E}, r)\). The node set \(\mathcal{V} = \{v_1, \dots, v_N\}\) represents the collection of agents. As illustrated in Figure  2, each agent \(v_i\) is instantiated by a base language model \(\mathrm{LM}_i\), a role \(\mathrm{Role}_i\) (e.g., math-solver), a context \(\mathrm{Context}_i\) which includes the knowledge and dialogues history, and a set of external tools \(\mathrm{Tool}_i\) (e.g., file-reader). The edge set \(\mathcal{E}\) denotes agent interactions, where \(r: \mathcal{E} \to \mathcal{R}\) serves as an edge-type mapping function. Specifically, a directed edge \(e_{ij}\in\mathcal{E}\) signifies an interaction from \(v_i\) to \(v_j\) governed by a protocol \(r_{ij}:=r(e_{ij})\in\mathcal{R}\). The definition of \(\mathcal{R}\) will be discussed in Section 3.1.

Communication pipeline. Given a query \(q\), a scheduling function \(\psi\) maps \(\mathcal{G}\) to an execution sequence \(\psi \rightarrow\sigma = \langle v_{\sigma(1)}, \dots, v_{\sigma(N)} \rangle\). When activated, each agent \(v_i\) processes a prompt \(\mathcal{P}\) to produce a solution \(\mathcal{S}_i = v_i(\mathcal{P})\). The final output \(a\) is obtained by aggregating all agents’ solutions: \(a \leftarrow \mathrm{Aggregate}(\{\mathcal{S}_i\}_{i=1}^N)\). As a comparision, hybrid intra/inter-round dialogue paradigms framework rely on a \(T\)-round iterative simulation yielding sequential outputs \(\{a^{(t)}\}_{t=1}^{T}\), which imposes a potential overhead on execution efficiency.

2.2 Optimization and Decentralized Design↩︎

Centralized optimization. We employ reinforcement learning to optimize the interaction graph. Specifically, we formulate this task as learning a stochastic policy \(\pi_{\theta}\) that, given a task query \(q\), seeks to construct an optimal interaction graph \(\mathcal{G^*}= (\mathcal{V}^{*},\mathcal{E}^{*},r^{*})\). The objective of policy \(\pi_{\theta}\) is to maximize the expected reward over the distribution of possible graphs, guided by a task-related reward function \(R(\cdot)\) derived from the quality of the final answer: \[\begin{align} \mathcal{G}^*= \underset{\mathcal{G}}{\operatorname*{argmax}} \begin{bmatrix}\mathbb{E}_{\mathcal{G}\sim\pi_\theta(\cdot|q)}[R(\mathcal{G})] \end{bmatrix}. \end{align}\]

Decentralized design. To support adaptive decision-making while addressing potential privacy risks, we adopt a decentralized architecture where each agent \(v_i \in \mathcal{V}\) maintains an independent local policy network \(\pi^{(i)}_{\theta^\prime}\) to infer its connectivity with other nodes. Conditioned on the task embedding \(\mathbf{z}\) and the local states \(\mathbf{h}_i\) and \(\mathbf{h}_j\), the communication link between agent \(v_i\) and \(v_j\) is formulated as: \[p_{ij}^{(r)}=\pi_{\theta^\prime}^{(i)}(r_{ij}|\mathbf{h}_i,\mathbf{h}_j,\mathbf{z}).\]

3 Architecture Details↩︎

3.1 Diverse Interaction Modes↩︎

Recent studies indicate that diverse collaborative mechanisms, such as evaluation and debate, can significantly bolster the complex problem-solving capabilities of MAS [19], [20]. Motivated by these findings, we formalize the interaction space \(\mathcal{R}\) using three distinct edge types, as shown in Figure 3. Each edge type is instantiated via specific prompts to govern its interaction logic.

Conditioned edges. The classical edge type where agent \(v_j\) handles the query \(q\) conditioned with the outputs of agent \(v_i\).

Feedback edges. Encapsulate a supervisory mechanism where agent \(v_j\) critiques or validates the intermediate outputs generated by agent \(v_i\) and \(v_i\) re-handle the query \(q\) referring to the feedback of \(v_j\), simulating an evaluation and reflection process.

Debate edges. Model a debate process wherein agent \(v_j\) challenges \(v_i\)’s proposition for two rounds, ultimately \(v_j\) proceed the query \(q\) with the context of the debate.

Given a query \(q\), TopoDIM aims to learn a policy \(\pi_{\theta}\) that determines the optimal relation \(r^*_{ij} \in \mathcal{R}^\prime:= \mathcal{R}\cup \{\emptyset\}\) between the agent pair \((v_i, v_j)\), where \(\emptyset\) signifies the non-existence of an edge. \[r^*_{ij} = \underset{r \in \mathcal{R}^\prime}{\arg\max} \;\pi_{\theta}\left(r \vert v_i, v_j, q \right).\]

3.2 Heterogeneous Interaction Topology↩︎

We formulate the generation of heterogeneous topologies as a conditional autoregressive decision process. The learnable policy \(\pi_\theta\) defines a distribution \(p_{\theta}(\mathcal{G} \vert q)\) over feasible communication graphs.

Semantics-aware relational encoder. To establish a robust latent representation for each agent, we employ a relational graph convolutional network [23], [24] over a prior knowledge graph \(\mathcal{G}_\text{pri} = (\mathcal{V},\mathcal{E}_{\mathrm{pri}},r_{\mathrm{pri}})\), which is initialized by an advanced LLM (e.g., GPT-5). The node update rule at layer \(l+1\) is defined as \(\mathbf{h}^{\ell+1}_i = \sigma (\hat{\mathbf{h}}^{\ell}_i)\), where \[\hat{\mathbf{h}}^{\ell}_i = \sum_{r \in \mathcal{R}_{\mathrm{pri}}} \sum_{j \in \mathcal{N}_i^r} \frac{1}{c_{i,r}} \mathbf{W}_r^{(l)} \mathbf{h}_j^{(l)} + \mathbf{W}_0^{(l)} \mathbf{h}_i^{(l)},\] \(\mathcal{N}_{i}^{r}\) denotes the set of neighbors of node \(v_i\) under relation \(r\), and \(c_{i,r}=|\mathcal{N}_i^r|\). The initial node embeddings combine role and task information: \[\mathbf{h}_i^{(0)}=\mathbf{e}_i^\text{role}+\mathbf{W}_q\phi(q),\] where \(\mathbf{e}_{i}^{\mathrm{role}}\) is a learnable role embedding and \(\phi(q)\) is the feature representation of the task query \(q \in Q\) obtained from a pre-trained sentence encoder. The final set of node embeddings from the encoder, \(\{\mathbf{h}_1, \dots, \mathbf{h}_N\}\), constitutes the conditional representation \(\mathbf{H}\).

Autoregressive edge sampling decoder. Given the conditional representation \(\mathbf{H}\), a decoder generates topology \(\mathcal{G}\) sequentially. With a prespecified ordering of nodes, the joint probability of observing a graph \(\mathcal{G}\) reads: \[\begin{align} &p_\theta(\mathcal{G} \mid \mathbf{H}) = \prod_{j=2}^N \prod_{i=1}^{j-1} p_{\theta}(r_{ij} \mid \mathbf{H}, \mathcal{G}_{<ij}) \\ &= \prod_{j=2}^N \prod_{i=1}^{j-1} \frac{\exp\left(\mathbf{w}_{r_{ij}}^\top [\mathbf{h}_i \| \mathbf{h}_j] / \tau\right)}{\sum_{r \in \mathcal{R}^\prime} \exp\left(\mathbf{w}_r^\top [\mathbf{h}_i \| \mathbf{h}_j] / \tau\right)}, \end{align}\] where \(\|\) denotes concatenation, and \(\tau\) is the temperature. We sample the relation type \(r_{ij}\) using inverse transform sampling by generating \(u \sim \text{Uniform}(0,1)\) and setting \(F(r) = \sum_{r\in\mathcal{R}^\prime} p_\theta(r)\), where \(r_{ij} = \min\{r : F(r) \geq u\}\) represents the cumulative probability.

Structural constraints. To ensure logical consistency and prevent circular dependencies, we enforce an acyclic constraint via a dynamic mask \(\mathbf{M}\). Specifically, if the presence of edge \((v_i,v_j)\) violates the acyclic constraint in any existing path from \(v_i\) to \(v_j\), the probability of this node pair is masked as: \[p(r_{ij} \mid \cdot) \leftarrow p(r_{ij} \mid \cdot) \odot \mathbf{M}_{ij},\] where \(\mathbf{M}_{ij} = \mathbb{I}(v_i \not\rightsquigarrow v_j)\) indicates if \(v_i\) and \(v_j\) are not path-connected. Detailed discussions on structural constraint is shown in Appendix 8.

Adaptive sparsification. Inspired by connectivity sparsification methods [12], [14] that enhance task performance through structural reduction, we employ an adaptive pruning mechanism to retain only salient interactions. Specifically, we identify the subset of edges \(\mathcal{E}_{\mathrm{final}}\) based on confidence scores, subject to a sparsity budget ratio \(\alpha\): \[\begin{align} &\mathcal{E}_{\mathrm{final}}=\mathrm{TopK}\left(\{p(r_{ij})\}_{\forall i,j},1-\alpha\right). \end{align}\] To identify the most relevant contributors for each task, TopoDIM filters out inactive nodes that lack connectivity within the selected edge subset. The final node set \(\mathcal{V}_{\mathrm{final}}\) is derived by retaining only those vertices involved in at least one valid interaction: \[\mathcal{V}_{\mathrm{final}}=\{v\mid\exists(h,t,r)\in\mathcal{E}_{\mathrm{final}},v\in\{h,t\}\}.\]

Execution order. Upon establishing the final set of agents \(\mathcal{V}_{\mathrm{final}}\) and edges \(\mathcal{E}_{\mathrm{final}}\), the interaction process follows a breadth-first manner. Starting from the root node (characterized by zero in-degree), each agent interacts with its neighbors in a sequence determined by ascending node indices.

3.3 Diversity-Aware Optimization Strategy↩︎

We optimize a graph generation policy \(\pi_{\theta}\) via reinforcement learning, aiming to maximize a composite objective that balances task success with structural diversity. For each generated graph \(\mathcal{G} \sim \pi_{\theta}\), the agent ensembles execute the collaborative task to yield a binary performance metric \(R_{\mathrm{task}}(\mathcal{G})\in\{0,1\}\), where 1 indicates a successful solution and 0 indicates failure. To mitigate mode collapse and prevent the policy from converging to a narrow set of interaction patterns, we introduce a structural balance reward [25], [26]. This term is formulated as the Shannon entropy of the empirical edge-type distribution \(p(r) = |\mathcal{E}_r|/|\mathcal{E}|\), where \(\mathcal{E}_r\) denotes the set of edges with relation type \(r\): \[R_{\text{div}}(\mathcal{G}) = - \sum_{r\in \mathcal{R}} p(r) \log p(r).\] The holistic reward signal \(R(\mathcal{G})\) is a weighted interpolation of task performance and structural entropy: \[R(\mathcal{G}) = \lambda \cdot R_{\text{task}}(\mathcal{G}) + (1-\lambda) \cdot R_{\text{div}}(\mathcal{G}),\] where \(\lambda \in [0,1]\) is a hyperparameter governing the trade-off between performance exploitation and diversity exploration.

Following the policy gradient theorem [27], we maximize the expected objective \(J(\theta) = \mathbb{E}_{\mathcal{G}\sim\pi_\theta}[R(\mathcal{G})]\). To stabilize training and reduce variance, we employ a moving average baseline \(b\) [28], [29]. The overall loss function also includes an entropy regularization term for the policy’s output distribution to encourage exploration: \[\begin{align} \nabla_\theta J(\theta) = & \mathbb{E}_{\mathcal{G}\sim\pi_\theta} \Big[ \nabla_\theta \log \pi_\theta(\mathcal{G}) \cdot \left( R(\mathcal{G}) - b \right) \\ &+ \gamma \nabla_\theta H(\pi_\theta(\mathcal{G})) \Big], \end{align}\] where \(H(\pi_\theta)\) denotes the entropy of the policy distribution and \(\gamma\) is the regularization coefficient.

3.4 Decentralized Architecture Design↩︎

To facilitate scalable and autonomous deployment, each agent \(v_i\) employs a light local policy (e.g., MLP) \(\pi_{\theta^\prime}^{(i)}\) to predict a relation type with all other node \(\sum_{j\neq i}v_j\), taking each ordered pair of representations\(( \mathbf{h}_i^{(0)},\mathbf{h}^{(0)}_j)\). We align local policy with the global policy \(\pi_{\theta}\) via policy distillation, minimizing the Kullback-Leibler (KL) divergence \(D_{KL}\) to capture global structural dependencies: \[\begin{align} \mathcal{L}_{\text{distill}}&(\theta^\prime) = {\sum_{v_i \in \mathcal{V}}}{\sum_{v_j \in \mathcal{V} \setminus v_i}} D_{KL}\big( \\ &\pi_\theta (r_{ij}| \mathcal{G}_{\text{pri}},q) || \pi_{\theta^\prime}(r_{ij} | \mathbf{h}^{(0)}_i, \mathbf{h}^{(0)}_j)\big), \end{align}\] where \(\pi_\theta(\cdot|\cdot)\) is the centralized policy induced marginal distribution. During inference, agents autonomously sample connections from \(\pi_{\theta^\prime}^{(i)}\) following the structural constraints, thereby combining centralized training optimality with decentralized execution efficiency.

4 Experiment↩︎

4.1 Experimental Setups↩︎

Models and benchmarks. To evaluate the general adaptability of TopoDIM to different LLMs, we employ a diverse set of models including Gemma-3-it:12B [30], GPT-OSS-20B [31], GPT-OSS-120B [31], and DeepSeek-V3.2-251201:671B [32]. We test the performance of TopoDIM on three types of reasoning datasets: General Reasoning: MMLU-Pro [33]; Mathematics: MultiArith [34], GSM8K [35] , AIME\(_{(2023-2025)}\); Coding: HumanEval [36]; LiveCodeBench\(_{(202305-202403)}\)[37].

Baselines. We evaluate TopoDIM by comparing its task-solving performance against 6 typical methods, including Single-agent prompting methods: Vanilla, Chain-of-Thought [38] which augment few-shot exemplars with intermediate reasoning steps. Multi-agent topology methods: LLM-Debate [9], GPTSwarm [5], G-Designer [6], AgentDropout [14].

Implementations. In our experiments, Gemma-3 and GPT-OSS are deployed locally on an NVIDIA RTX 4090 via Ollama [39], while DeepSeek-V3.2 is accessed through its official APIs. We implement a temperature annealing schedule decaying from 2.0 to 0.5 and maintain a fixed learning rate of 0.01 for both centralized and decentralized training. Specific parameters include Top-K sparsity budget ratio \(\alpha \in \{0.3, 0.5, 0.7\}\), \(\lambda \in \{0.3, 0.5, 0.8\}\), and \(\eta \in \{0.01,0.03,0.05\}\). The feature extraction function \(\phi(\cdot)\) is instantiated with all-MiniLM-L6-v2 [40], yielding \(384\)-dimensional embeddings. Each experimental run is initialized with 5 agents. For all benchmarks, the query budgets for optimization (\(M\)) and and decentralized knowledge distillation (\(M'\)) are selected from \(\{40, 80\}\). Furthermore, we employ GPT-5[41] to generate prompt descriptions for three distinct interaction types, and the local policy is parameterized by an MLP. All remaining configurations and agent settings follow the protocols established in [6].

4.2 Main Results↩︎

Task performance. Table 1 demonstrates that TopoDIM outperforms various baselines across multiple datasets, including single models and state-of-the-art multi-agent systems. Specifically, with Gemma-3-it:12B and DeepSeek-V3.2-251201, TopoDIM yields average performance gains of 1.35% \(\uparrow\) and 1.38% \(\uparrow\) over the strongest baselines, respectively. Notably, TopoDIM also enhances the capabilities of the reasoning model (GPT-OSS:120B) on complex tasks, resulting in a 1.50% \(\uparrow\) increase in average predictive accuracy.

Computation efficiency. We compare the token consumption of TopoDIM against other baselines on MMLU-Pro and LiveCodeBench using GPT-OSS:120B, with the results illustrated in Figure 4. TopoDIM significantly reduces token expenditure, saving a total of 1.42M and 2.44M tokens on MMLU-Pro and LiveCodeBench, respectively, while maintaining competitive performance. A comprehensive efficiency analysis is provided in Appendix 10.

Heterogeneous agents adaptability. To validate the adaptability of TopoDIM within a heterogeneous MAS, we construct a collaborative framework comprising agents with varying capabilities: three GPT-OSS-20B agents and two GPT-OSS-120B agents, with the latter serves as the final decision-maker. As shown in Table 2, the results demonstrate that TopoDIM achieves SOTA performance across all datasets, yielding an average improvement of 1.86% \(\uparrow\) over existing homogeneous topology frameworks. TopoDIM leverages its adaptive sparsification design to effectively prune agents that make negligible contributions, thereby optimizing communication efficiency. A more comprehensive evaluation is provided in Appendix 11.

4.3 Framework Analysis↩︎

We conduct comprehensive experiments using GPT-OSS:120B to validate the framework design of our proposed TopoDIM.

Edge diversity. Figure 5 (a) demonstrates the impact of edge diversity on the collaborative problem-solving capabilities of TopoDIM. Specifically, Base. denotes the configuration with only conditioned edges, while + Fbk./Dbt. corresponds to the incremental addition of feedback and debate edge types. Our systematic investigation reveals a positive correlation between edge diversity and performance. Notably, integrating all available edge types improves the prediction accuracy/pass@1 by 2.21%\(\uparrow\) on MMLU-Pro and 2.18%\(\uparrow\) on LiveCodeBench, respectively.

Edge sparsity. The amount of edges, controlled by the TopK sparsity budget ratio \(\alpha \in \{0, 0.3, 0.5, 0.7, 1.0\}\), significantly impacts performance, as depicted in Figure 5 (b). The optimal accuracy/pass@1 (80.11% on MMLU-Pro and 87.28% on LiveCodeBench) is achieved at \(\alpha = 0.7\), which reveals a crucial trade-off: a dense graph risks in creating redundant communication channels, leading to model hallucination and ultimately impairing decision-making performance.

Training sample size. We evaluate the data efficiency of TopoDIM by varying the training sample size \(M\) (Figure 5 (c)). The results demonstrate a monotonic improvement in performance as the number of samples increases, with the most significant gains occurring within the first 40 samples. Decentralized sample size. To evaluate the effectiveness of the distillation process, we vary the number of decentralized samples \(M'\)(from 0 to 160, where 0 indicates the direct use of the centralized policy to generate topology). As illustrated in Figure 5 (d), TopoDIM achieves robust performance with as few as 40 samples, demonstrating the efficiency of the knowledge transfer.

Ablation studies. To analyze TopoDIM’s architecture and validate the necessity of its core components, we perform a series of ablation studies. First, to evaluate our heterogeneous interactions topology design, we consider two configurations: w/ Rand, which features randomly generated heterogeneous edges, and w/o Graph, which replaces \(\mathcal{G}_\text{pri}\) with a fully connected graph (using random edge types). These modifications result in average performance drops of 5.60%\(\downarrow\) and 1.99%\(\downarrow\), respectively, underscoring the significance of the graph prior and our topology design. Second, to evaluate the topology optimization strategy, we examine the contributions of the baseline and entropy regularization in optimization phases. Removing the baseline \(b\): w/o Baseline, which enhances training stability, leads to an accuracy decrease of 0.89%\(\downarrow\). Similarly, removing the entropy regularization term: w/o ER, which facilitates policy exploration, causes accuracy to fall by 1.38% \(\downarrow\). These studies demonstrate the critical role of each component in TopoDIM.

4.4 Case study↩︎

We conduct a case study on LiveCodeBench using GPT-OSS-120B to evaluate the performance and optimized communication structures of TopoDIM against existing methods. Figure 6 visualizes the topologies generated by different approaches for a representative query. The top panel depicts a typical intra-round dialogue graph. When handling complex tasks, such methods are often prone to information loss, leading to suboptimal performance in ambiguous scenarios. The middle panel illustrates an intra/inter-round paradigm that facilitates agent collaboration through multi-turn dialogues. However, this approach incurs additional token overhead and increases susceptibility to hallucination. In contrast, TopoDIM (bottom panel) dynamically constructs a sparse yet informative topology. By selectively activating conditional, feedback, and debate interaction patterns, it effectively preserves critical context without unnecessary token consumption, thereby successfully resolving complex reasoning challenges. Detailed dialogues are provided in Appendix 13.

5 Related Work↩︎

5.1 LLM-based MAS↩︎

Multi-agent systems(MAS) catalyze a paradigm shift from single-agent strategies [42], [43] to collective intelligence for complex problems [44]–[46]. Pioneering frameworks demonstrated the efficacy of communicative role-playing and social simulation [47], [48], while task-oriented architectures have formalized collaborative workflows in graph generation [49], software engineering [50], and logical reasoning [51] by integrating dynamic role assignment [52], iterative debate [17], [53], [54], and shared memory [55], [56]. Additionally, recent work has introduced benchmarks for evaluating MAS [57]. Recent advancements have corroborated decentralized architectures [22] and diversified interaction patterns [19] to improve task proficiency and adaptability. TopoDIM orchestrates decentralized structure and diverse interactions, demonstrating superior performance.

5.2 Topology Design of MAS↩︎

Communication topologies in LLM-based MAS are generally categorized into intra-round, inter-round, and hybrid structures [12]. Early research primarily organizes either intra- or inter-round dialogues, including chain [58], [59], tree [8], layered [9], [60], and filtered graphs [5], [10]. Recent advancements focus on generating task-specific adaptive topologies by integrating both intra- and inter-round interactions[7], [13], [15], [20], [61]. For instance, G-Designer [6] employs variational autoencoders to generate adaptive graph structures. While recent methods attempt to mitigate these costs via edge pruning [12] and dropout [14], TopoDIM eliminates redundant dialogue rounds while maintaining diverse interactions, making a promising performance.

6 Conclusion↩︎

In this paper, we aim at solving the structural redundancy of the hybrid intra/inter-round dialogue paradigm in MAS. We introduce TopoDIM, a novel decentralized framework for one-shot heterogeneous topology generation tailored adaptively to task queries. Experiments show that TopoDIM achieves SOTA performance in overall performance and cost-efficiency. Meanwhile, its adaptability to heterogeneous agent types allows to avoid the bucket effect. We hope our work will inspire future research on exploring synergy between cognitive mechanisms and interaction topology for scalable collective intelligence.

Limitations↩︎

Interaction constraints. While TopoDIM demonstrates competitive performance through conditioning, feedback, and debate mechanisms, our current focus prioritizes these direct interaction forms to maintain an optimal balance between predictive accuracy and token efficiency. Consequently, the scope of this work does not extend to complex organizational methods, such as dynamic coalition formation, where subsets of agents spontaneously align to address specific sub-problems.

Heterogeneous agents adaptability. We demonstrate TopoDIM outperforms baselines in heterogeneous agent settings, nevertheless finding the combination of high-performance and lightweight LLMs is non-trivial. In practice, we observed that simply introducing random agents yields small performance gains when tasks observably exceed agents capabilities. Furthermore, the deployment of heterogeneous LLMs imposes infrastructure challenges, as serving models with varying architectures and weights requires engineering support.

Computational Overhead. Compared to single-LLM approaches, MAS inevitably incurs more token consumption and latency due to extensive inter-agent communication [62]. While recent works attempt to mitigate this by organizing cooperation within the latent space [63], [64], we adhere to a topology-based paradigm to ensure compatibility with proprietary models. Future work will explore effective mechanisms to further optimize cost-efficiency while improving performance.

Ethical Considerations↩︎

This work relies exclusively on publicly available datasets and benchmarks, ensuring that no private or personally identifiable information is involved. While we did not observe emergent harmful behaviors or unintended autonomous actions during evaluation, we acknowledge that LLM-based multi-agent systems may inherit risks such as bias propagation or hallucinations. Crucially, our proposed framework serves as a coordination layer designed to enhance cooperative efficiency; it operates within the safety boundaries of the base LLMs without bypassing their existing guardrails. Finally, AI assistants were employed solely for grammatical error correction and rephrasing.

Acknowledgement↩︎

This work was supported by the National Natural Science Foundation of China (T2293771), the STI 2030–Major Projects (2022ZD0211400), and National Natural Science Foundation of China (No. 42230406).

7 Algorithm workflow↩︎

The algorithm workflow of TopoDIM is illustrated in Algorithm [Algo:topodim]. Specifically, TopoDIM comprises two training stages. In stage 1, TopoDIM iterates through all training queries and employs policy optimization to update the encoder and decoder parameters, incorporating structural constraints and adaptive sparsification. In stage 2, TopoDIM performs decentralized policy distillation using the same training set as in Stage 1, updating the parameters of lightweight networks which are deployed on each agent itself for a purpose of flexible and private decision for . During the inference phase, TopoDIM generates a communication topology and traverses it in a breadth-first manner. By facilitating agent cooperation through three semantically rich interaction modes, the method achieves competitive performance on complex tasks.

8 Acyclic Property↩︎

We impose acyclic structural constraints exclusively on conditioned and debate edges, leaving feedback edges unconstrained. Specifically, structural constraints are applied exclusively to conditioned and debate edges, which must remain acyclic to preclude circular dependencies (Figure 7 (a)). In contrast, feedback edges are allowed for arbitrary connectivity (Figure 7 (b)). During traversal, feedback edges are prioritized for immediate execution: they are excluded from degree calculations and removed upon visit. Meanwhile other edge types adhere to standard breadth-first scheduling.

9 Data Statistics↩︎

We summarized the data statistics in Table 4. In this paper we conduct experiments across three representative categories of datasets, incorporating challenging benchmarks such as MMLU-Pro, AIME, and LiveCodeBench to make a comprehensive evaluation of our proposed TopoDIM. For the AIME benchmark, given the limited volume of available problems, we utilize the entire collection of samples from 2023 to 2025 for training (40 samples) and evaluation (50 samples) [65]. For LiveCodeBench, we employ Release V1 version of for a moderately challenging evaluation on code generation tasks, which comprises data collected between May 2023 and March 2024. For MMLU-Pro, we adopt the same data processing pipeline as applied to MMLU in previous work [6], selecting 153 samples from the test set as our evaluation subset. The experimental settings for the remaining datasets, including MultiArith, HumanEval, and GSM8K, remain consistent with prior research [14] to ensure fair comparison.

10 Cost Efficiency↩︎

We provide a more detailed display for cost-efficiency of TopoDIM, shown in Table 6. TopoDIM cuts down the token expenditure, saving maximum prompt and completion token by 57.82% and 22.05% compared to the most efficient framework. Moreover, we conducted experiments to evaluate GPU resource overhead during the inference phase. By randomly instantiating 1,000 nodes, we demonstrated that our decentralized design, where each agent operates a lightweight network, requires merely 6.36 GB of memory, achieving significant cost efficiency.

11 Heterogeneous Agents Adaptability↩︎

To provide a more comprehensive evaluation of TopoDIM’s adaptability to heterogeneous agents, we conducted experiments on two additional configurations involving agents with distinct sources and capacities. As detailed in Table 5, we combined GPT-OSS:20B with DeepSeek-V3.2 and Gemma3-it:12b, respectively. In these settings, TopoDIM consistently outperformed other baselines across three representative datasets, achieving average improvements of 1.32% and 1.15%, respectively. It is worth noting that identifying effective combinations of high-performance and lightweight LLMs is non-trivial; naively introducing random agents with heterogeneous skills can sometimes be counterproductive (e.g., the performance degradation observed with G-Designer). In contrast, TopoDIM leverages its adaptive sparsification design to effectively prune agents that make negligible contributions, thereby optimizing communication efficiency.

12 Robustness Analysis↩︎

We evaluated the robustness of TopoDIM by conducting experiments on LiveCodeBench with GPT-OSS:120B. An agent is randomly selected and injected with prompts that drive it to generate failure outputs. As illustrated in Figure 8, thanks to its heterogeneous adaptability, TopoDIM exhibits remarkable robustness to these attacks, limiting the performance degradation to 0.3 points. Conversely, LLM-Debate incurs a drop of 1.2 due to its fixed communication topology. Meanwhile, dynamic topology frameworks like GPTSwarm, G-Designer, and AgentDropout also display competitive structural robustness, with an average drop of 0.43. This adaptive capability is vital for identifying malicious agents, underscoring the superiority of dynamic topologies in maintaining structural integrity.

13 Case Study↩︎

Task-specific Patterns. We conduct case studies with GPT-OSS-120B on AIME (math) and MMLU-Pro (general) to illustrate task-specific collaboration patterns, as shown in Figure 9. For an MMLU-Pro psychology question, TopoDIM generates an efficient topology that coordinates multiple agents to solve the problem. The psychologist and historian first engage in a debate. The critic then reviews the psychologist’s response and consults a knowledgeable expert for professional feedback. For an AIME problem, the math analyst first examines the question and passes the result to the math solver. Conditioned on this output, the math solver works out the solution and requests feedback from a programming expert for verification. Finally, the math solver and inspector enter a debate to further refine the solution.

Specific Cases. While the main text briefly compares the communication sequence of TopoDIM with two typical intra- and inter-round dialogue paradigms using GPT-OSS:120B on a LiveCodeBench question, this section presents the detailed cooperative dialogues among different agents, as illustrated in Figures 10 through 14. Upon receiving a question \(q\), TopoDIM effectively generates a communication graph to orchestrate agent collaboration. As shown in Figure 11, the algorithm designer outlines the solution strategy and provides the initial code. The bug fixer then generates a revised solution conditioned on the designer’s output. Subsequently, a programming expert engages in a debate with the bug fixer to produce robust code that efficiently solves \(q\), as depicted in Figure 12. Next, the project manager proposes a separate solution, which undergoes a detailed evaluation by the algorithm designer. Based on this critical feedback, the project manager refines the solution, as illustrated in Figure 13. Finally, a decision-maker reviews the collective contributions from all participants and delivers the final answer, as depicted in Figure 14. These interactions exemplify the iterative self-correction capability facilitated by the multi-type interactions.

References↩︎