1 Introduction↩︎

Large language models (LLMs), with differences in architecture design, training data, and optimization objectives, often excel at different tasks [1]–[4]. For example, DeepSeek-R1 achieves strong results in code generation [5], while Claude 3.7 Sonnet is particularly effective in factual retrieval and knowledge coverage [6]; our work also verifies this diversity of LLM performance (Figure 2 and 3). Due to this diversity, researchers tend to utilize multiple LLMs in their workflow, notably in LLM API server platforms and agentic workflows [7]; such usage pattern leaves us with valuable multi-LLM log data. Here, multi-LLM log data refers to structured serving records produced when multiple LLMs are invoked for user queries, rather than an ordinary collection of independent model responses. Such records preserve not only query-response pairs, but also model identity, task or domain information, response mode, inference cost, and quality feedback, which together characterize the behavior of different LLMs under aligned query settings. We point out that these multi-LLM log data are a valuable vault, leading to an important research question: can we fully leverage multi-LLM log data to fuse LLMs’ complementary capabilities?

To address this challenge, prior work has explored several different ways of integrating multiple LLMs. Some studies investigate multi-agent systems, where LLMs collaborate through division of labor [8]–[10]. Others develop LLM routers that dynamically assign queries to the most suitable model [11]–[15], while some existing model fusion approaches, including naive ensembles [16], [17] and model merging [18], aim to leverage multiple LLMs at the output and parameter levels, respectively. Despite their different formulations, these approaches share the same underlying goal of harnessing complementary strengths among LLMs. However, most existing methods study only one specific fusion stage in isolation. For example, routing methods mainly use multi-LLM records for query-level model selection, while distillation or model-fusion methods mainly focus on transferring responses or knowledge into a target model. As a result, they do not fully answer a practical question faced by users: given the same source of multi-LLM log data, should it be used for routing, thought-level prompting, or model-level distillation? While these explorations mark an important step forward, fundamental challenges remain in building LLM fusion approaches that are both broadly effective and practically feasible. These challenges motivate a closer examination of what properties an effective fusion framework should possess.

In this work, we argue that effective fusion methods should ideally satisfy two key requirements. First, effective fusion should be compatible with different LLM serving scenarios: Real-world LLM servicing, broadly speaking, can be categorized into local serving (when model weights are available) and API-based serving (when models are closed-source or expensive to run). While LLM distillation [19] has emerged as a popular paradigm for fusion, existing methods typically rely on logits or internal states, which are inaccessible in the generic LLM serving scenarios (e.g., API-based serving). In contrast, multi-LLM log data, which is stored in text format, offers the most practical, cost-effective, and scalable foundation for studying LLM fusion, and thus forms the focus of this work. Different from generic multi-response collections that mainly contain several outputs for the same input, the log data considered in this work contains structured metadata and feedback signals associated with LLM serving, including model identity, response mode, cost, task-specific performance, and LLM-based quality scores. These signals are directly useful for learning routing policies, retrieving reusable reasoning patterns, and selecting high-quality responses for distillation. Second, effective fusion should happen at different stages in the LLM pipeline to accommodate different users’ needs. In practice, users may or may not have the privilege of finetuning an LLM (due to cost and model availability considerations). Moreover, different tasks may require an LLM to fuse at different stages; for example, a RAG-based LLM system works more favorably with retrieval-based thought-level LLM fusion. In this work, we aim to systematically understand the trade-offs of LLM fusion at different stages. This stage-aware perspective is central to our motivation. API-only compatibility is not the sole novelty of our work; rather, we aim to provide a unified benchmark and framework for studying how the same structured multi-LLM logs can be exploited at different intervention points in the LLM pipeline. This motivates the fusion-stage matrix studied in this paper, which compares query-level, thought-level, and model-level fusion under a shared log source and evaluation protocol.

As illustrated in Table ¿tbl:Tab:FusionBench95intro?, the unique value of FusionFactory is not merely API-serving compatibility, since several prior routing methods also support API-only usage. Instead, FusionFactory differs by systematically studying how multi-LLM log data can support multiple fusion stages under a unified framework. Prior routing methods such as RouteLLM, GraphRouter, RouterBench, and LLM-Blender mainly focus on query-level fusion, while FuseLLM mainly studies model-level fusion. In contrast, FusionFactory jointly covers query-level, thought-level, and model-level fusion, which enables a cross-stage comparison of when and how the same log data should be used.

Based on the above observations, we propose LLMFusionBench, a large-scale benchmark designed to support comprehensive studies on LLM fusion. LLMFusionBench provides rich supervision by including both direct and reasoning-augmented responses from multiple LLMs, as well as reusable thought templates summarized from the top-performing models. Covering 14 tasks across 6 domains, LLMFusionBench is built from 20 open-source LLMs ranging from 8B to 671B, making it one of the most diverse resources for studying multi-LLM integration (Table ¿tbl:tab:2461? and [tbl:tab:2462]). In this sense, LLMFusionBench operationalizes the above notion of multi-LLM log data: for each query, it stores aligned responses from multiple LLMs together with their response modes, performance scores, inference costs, and LLM-judge scores. This structure distinguishes LLMFusionBench from ordinary multi-response datasets and provides the supervision signals needed for flexible capability fusion.

Building on LLMFusionBench, we introduce FusionFactory, a systematic framework for LLM fusion across three representative levels (comparison shown in Table ¿tbl:Tab:FusionBench95intro?). FusionFactory leverages different components of the structured log records at different fusion stages. Query-level fusion uses query and task information, model identity, performance, cost, and judge scores to learn routing decisions. Thought-level fusion uses historical queries, response modes, and top-ranked reasoning traces to construct reusable thought templates. Model-level fusion uses quality-ranked logged responses as distillation data. (1) Query-level fusion adapts to the varying strengths of different models and the diverse user preferences regarding token generation costs by constructing routers tailored to each query, enabling finer-grained capability fusion. Unlike traditional methods, we incorporate both direct outputs and reasoning-augmented responses [23]–[26] to better address complex user queries. (2) Thought-level fusion improves new responses by retrieving abstract thought templates (i.e., concise summaries distilled from top-performing model outputs) based on embedding similarity to past queries. This design avoids the noise of full past responses while still capturing useful reasoning patterns, which are then used as few-shot demonstrations for new queries [27]–[30]. (3) Model-level fusion takes a distillation view [31], [32], transferring complementary capabilities into a single base model by conducting supervised fine-tuning on high-quality responses aggregated across LLMs. Inspired by imitation learning in the LLM distillation domain [33], [34], for each query, we select the LLM responses with the top-k performance or the highest LLM judge scores as training data. This enables downstream deployment of a fused model without relying on multiple LLMs at inference time.

We systematically evaluate FusionFactory on LLMFusionBench and find that fusing LLMs with FusionFactory yields consistent benchmark performance improvement over the best individual LLM across 14 popular LLM benchmarks, where the optimal FusionFactory configuration varies across different benchmarks. Specifically, we find that: (1) thought-level fusion achieves the best overall performance, especially when using hybrid selection and large summaries; (2) model-level fusion performs the worst among the three levels, likely due to overfitting and difficulty generalizing across tasks; (3) query-level fusion offers a good balance between performance and efficiency, achieving decent results while requiring minimal computational overhead; (4) gains from fusing World Knowledge and Math domains are moderate, where these domains demand high factual accuracy or strict logical consistency, which are difficult to maintain through multi-LLM fusion. These results highlight the promise of multi-LLM log data as a practical foundation for developing fusion methods that flexibly adapt to diverse serving scenarios and fusion stages. Moreover, the cross-level comparison enabled by our fusion-stage matrix provides insights that cannot be obtained by studying routing or distillation alone. It reveals that different ways of using the same multi-LLM log data lead to different performance, cost, and generalization behaviors, thereby providing a practical decision framework for choosing among routing-based, thought-based, and training-based fusion under different deployment constraints.

2 LLMFusionBench: Benchmarking LLM Capability Fusion with Multi-LLM Log Data↩︎

In this section, we detail the construction of LLMFusionBench(Section 2.1) and its support for LLM capability fusion (Section 2.2), as illustrated in Figure 1.

2.1 LLMFusionBench Construction↩︎

As shown in Figure 1, we construct LLMFusionBench by sampling up to 500 training and 50 partial test queries (10:1 ratio) from each of 14 tasks across six domains (Table ¿tbl:tab:2461?). For each query, responses from multiple LLMs are collected to support comparison and exploration of query-, thought-, and model-level fusion (Sections 4.2, 5.2, and 6.2). For the final cross-level analysis (Section 7), we also use each task’s full test set to comprehensively evaluate fusion levels. Table 3 in the Appendix summarizes sample counts for training, partial test, and full test sets.

We formalize the resulting multi-LLM log data as structured serving records rather than ordinary response collections. Specifically, each log entry is represented as \((q,t,m,s,r,c,y,j)\), where \(q\) denotes the user query, \(t\) denotes the task or domain, \(m\) denotes the LLM identity, \(s\) denotes the response mode such as direct or reasoning-augmented prompting, \(r\) denotes the generated response, \(c\) denotes the inference cost, \(y\) denotes the task-specific evaluation result, and \(j\) denotes the LLM-judge score. Under this definition, LLMFusionBench is not simply a set of \((q,r)\) pairs, but an aligned multi-LLM serving log that records model behavior, response style, cost, and quality feedback for the same query.

As summarized in Table [tbl:tab:2462], we select 20 LLMs across varying scales. To elicit diverse responses, we apply distinct system prompts for direct and reasoning-based outputs. The 20-model pool contains both LLMs with native trained think-token behavior (e.g., DeepSeek-R1 and the Nemotron -think variants) and LLMs without it. We apply the same chain-of-thought elicitation procedure uniformly across all 20 models to enable apples-to-apples comparison across the pool; for models with trained think-token behavior the model’s native reasoning style is preserved when responding to our prompt, while for models without it the -think responses are produced via prompted CoT. Responses are evaluated with task-specific metrics (Table ¿tbl:app95tab:1?), but as shown in Figure ¿tbl:tab:performance95and95LLM95judge?, such metrics may favor short, low-information answers that hinder effective fusion. To address this, we introduce an LLM judge that scores the insightfulness of correct answers (prompt in Table 8). As shown in Table ¿tbl:tab:performance95and95LLM95judge?, the LLM judge better reflects informative value, offering greater utility for fusion.

To assess cost, we follow [15] by estimating total token usage via GPT-2 [35] and converting it into monetary cost using Together AI’s API pricing (Table 1). As shown in the right of Figure 1, collected data is stored in a DataFrame, with each row capturing task, query, LLM, response, performance, cost, and LLM judge score (details in Table 2).

This construction focuses on a practical subset of log signals that can be consistently collected in both local and API-based LLM serving scenarios. Richer production logs may additionally contain temporal order, explicit user feedback, error patterns, and multi-turn interaction history. These signals are valuable, but they are not always available across different serving settings, especially when using closed-source or API-based models. Therefore, LLMFusionBench emphasizes broadly accessible log attributes, including model identity, response mode, cost, task-specific performance, and LLM-based quality feedback, while leaving interaction-level signals such as temporal dynamics and user feedback as important extensions.

2.2 Support for LLM Capabilities Fusion↩︎

As introduced earlier, LLMFusionBench captures diverse LLM responses to the same query, enabling potential for LLM capability fusion. To illustrate this, we conduct a qualitative case study using a Math query (Figure 2). We compare responses from LLaMA-3.1 Nemotron-direct (51B), LLaMA-3.1 Nemotron-think (51B), and LLaMA-3-think (70B), where -think denotes CoT prompting. These responses exhibit distinct styles: direct and concise, analytical reasoning, and detailed explanation, respectively. These variations highlight opportunities to learn diverse reasoning patterns and enhance fusion effectiveness.

The structured log format is crucial for this capability-fusion analysis. Because different LLMs are evaluated on the same query with shared task-specific metrics, cost estimates, and judge scores, LLMFusionBench allows us to compare aligned model behaviors rather than unrelated responses. Model identity helps identify which LLM is suitable for which type of query, response mode distinguishes direct answering from reasoning-augmented generation, and performance or judge feedback provides supervision for selecting useful responses. These log properties are later used by query-level fusion for routing, thought-level fusion for retrieving and summarizing reasoning templates, and model-level fusion for selecting distillation data.

To complement our analysis, we conduct quantitative evaluations using LLM judge scores. As shown in Figure 3, we compute the average judge score for each model scale per query (based on Table [tbl:tab:2462]), and then calculate pairwise differences: small-medium, medium-large, and small-large. The resulting histograms reveal distinct performance gaps across scales, suggesting that models of different sizes exhibit complementary strengths and further motivating fusion via LLMFusionBench.

3 FusionFactory: Flexible Fusion of LLM Capabilities↩︎

As shown in Figure 4, we present FusionFactory across three representative levels of LLM capabilities fusion: query-, thought-, and model-level, corresponding to early, mid, and late fusion stages. Each level targets a distinct phase in the reasoning process.

This stage-aware organization provides the conceptual taxonomy used in this empirical study. We do not aim to provide formal theoretical guarantees or closed-form complexity bounds; instead, our goal is to systematically compare practical fusion strategies according to where they intervene in the LLM pipeline. Query-level fusion intervenes before generation by selecting a model or response mode for each query. Thought-level fusion intervenes during generation by retrieving and injecting reusable reasoning templates. Model-level fusion intervenes after response collection by consolidating selected logged responses into model parameters through supervised fine-tuning. This taxonomy clarifies that the three levels are not arbitrary engineering categories, but distinct intervention points in the LLM inference and learning process.

This stage-aware organization is also the basis of the fusion-stage matrix studied in this paper. Rather than treating API-only compatibility as the main novelty, FusionFactory aims to answer a broader question: given the same structured multi-LLM log data, should it be used before generation for routing, during generation for reasoning augmentation, or after response collection for model training? As illustrated in Figure 4, query-level fusion is an early-stage intervention before response generation, thought-level fusion is a mid-stage intervention during reasoning, and model-level fusion is a late-stage intervention that consolidates logged responses into model parameters. This matrix therefore provides a unified way to compare different uses of multi-LLM logs under the same benchmark and evaluation protocol.

Importantly, the three levels leverage different components of the structured multi-LLM log records introduced in Section 2.1. Query-level fusion uses query and task information, model identity, response mode, performance, cost, and LLM-judge scores to learn routing decisions. Thought-level fusion uses historical queries, response modes, generated responses, and quality-ranked reasoning traces to construct reusable thought templates. Model-level fusion uses task-specific performance or LLM-judge feedback to select high-quality logged responses as supervised fine-tuning data. Therefore, FusionFactory does not treat multi-LLM logs as ordinary text-only response collections; instead, it exploits their structured metadata and feedback signals to support different forms of capability fusion.

Query-level Fusion (Early Fusion) (Section 4) operates at the problem interpretation stage. A router analyzes the query and task characteristics to select the best LLM configuration, considering performance, cost, and LLM-based evaluations. The selected model then processes the query independently, enabling efficient task-specialized execution. Thought-level Fusion (Mid Fusion) (Section 5) intervenes during reasoning. An LLM summarizer distills abstract thought templates from top-k responses (ranked by performance or LLM judgment). For new queries, we retrieve similar historical queries and apply their thought templates as few-shot prompts, allowing dynamic coordination across reasoning styles. Model-level Fusion (Late Fusion) (Section 6) occurs after full responses are generated. A base LLM is fine-tuned using top-k high-quality outputs (selected via performance or LLM judgment) to internalize fused capabilities. This stage reflects post-hoc fusion via training rather than online collaboration. These three fusion stages offer complementary strengths at different points in the reasoning pipeline, allowing users to flexibly choose the most suitable level of integration based on their specific serving constraints, performance goals, and resource availability.

From a practical computational and deployment perspective, the three stages expose different trade-offs. Query-level fusion has the lowest inference overhead because it only requires a lightweight routing step before calling one selected LLM. Thought-level fusion introduces additional retrieval and prompt-token overhead, but it avoids model fine-tuning and remains compatible with API-based LLMs. Model-level fusion requires offline training and access to a trainable base model, but after training it only requires a single fused model at inference time. We present these as practical deployment trade-offs rather than formal complexity bounds, because exact theoretical costs are difficult to define uniformly across heterogeneous LLM APIs, model sizes, serving infrastructures, and pricing schemes.

This comparison is necessary because existing routing and distillation methods typically focus on one stage in isolation. Routing-based methods mainly study query-level model selection, while distillation-based methods mainly study model-level capability transfer. In contrast, FusionFactory uses the same multi-LLM log source to study all three stages together. This enables cross-stage analysis of performance, cost, and generalization behavior, and provides a practical decision framework for users who face different deployment constraints.

4 Query-level FusionFactory Fusion↩︎

4.1 Methodology: Query-level Fusion↩︎

The goal of query-level fusion is to assign the most suitable LLM for each user query, fusing LLMs’ capabilities through a router. This router selects models based on the query, task info, and LLM features, jointly considering performance, cost, and LLM judge scores. Under the structured log formulation in Section 2.1, this level directly uses the log fields related to the input and serving decision, including the query \(q\), task or domain \(t\), model identity \(m\), response mode \(s\), cost \(c\), task-specific performance \(y\), and LLM-judge score \(j\). To clarify the routing notation, we define each router action as \(a=(m,s)\in\mathcal{A}\), where \(m\) denotes the selected LLM and \(s\) denotes the selected response mode, such as direct or reasoning-augmented prompting. Thus, the router chooses an LLM-response-mode configuration rather than an unspecified model feature. As shown in the first panel of Figure 4, we follow [15] and define the reward for selecting action \(a\) for query-task pair \((q,t)\) as: \[\label{eq:Reward} \text{Reward}(q,t,a) = \alpha \cdot P(q,t,a) - \beta \cdot C(q,t,a) + \gamma \cdot J(q,t,a),\tag{1}\] where \(P(q,t,a)\) denotes the task-specific performance score obtained by applying the evaluation metric of task \(t\) to the response generated under action \(a\); \(C(q,t,a)\) denotes the inference cost, estimated from token consumption and API pricing; and \(J(q,t,a)\) denotes the LLM-judge score, which measures the informativeness or usefulness of the generated response beyond the task-specific metric. The coefficients \(\alpha\), \(\beta\), and \(\gamma\) control the relative importance of task performance, inference cost, and LLM-judge quality, respectively. The negative sign before \(C(q,t,a)\) reflects that lower inference cost is preferred, while higher performance and judge scores are preferred. Based on this, for a query \(q\) and its corresponding task \(t\), our goal is to learn a router model \(f_{\phi}\) such as a neural network that maps \((q,t)\) to an action \(a\in\mathcal{A}\) and maximizes the expected reward: \[\label{eq:router95obj95joint} \phi^{*} = \arg\max_{\phi} \; \mathbb{E}_{(q,t) \sim \mathcal{D}} \left[ \text{Reward}\left(q,t,f_{\phi}(q,t)\right) \right],\tag{2}\] where \(\mathcal{D}\) denotes the distribution of query-task pairs and \(\mathcal{A}\) denotes the set of candidate LLM-response-mode configurations. After obtaining \(\phi^{*}\), \(f_{\phi^{*}}\) selects the action \(a=f_{\phi^{*}}(q,t)\) for each new query-task pair, and the selected LLM configuration generates the final response. This revised formulation removes the ambiguous dot notation in the original reward function and explicitly connects the routing objective to the structured log fields used in query-level fusion.

From the perspective of the fusion-stage matrix, query-level fusion offers a lightweight way to exploit multi-LLM logs in API-based serving settings. It does not require access to model weights, logits, or internal states; instead, it only requires logged performance, cost, and judge signals to learn a routing policy. Thus, this stage is suitable when users want to improve the accuracy–cost trade-off while still calling only one selected LLM at inference time.

4.2 Experimental Settings: Query-level Fusion↩︎

Scenario. To explore query-level fusion, we define four routing scenarios aligned with Section 4: Performance First, Balance, Cost First, and LLM Judge. The first three examine trade-offs between performance and cost, while the last assesses routing under LLM-as-judge evaluation. These scenarios differ in their reward weights \((\alpha, \beta, \gamma)\) in Equation 1 , which control the emphasis on performance, cost, and LLM-judge scores and are set to \((1.0, 0.0, 0.0)\), \((0.5, 0.5, 0.0)\), \((0.2, 0.8, 0.0)\), and \((0.0, 0.0, 1.0)\), respectively.

Comparison Routers. We evaluate five representative routing methods for LLM fusion: RouterKNN [36], RouterSVM [20], RouterMLP [36], RouterBERT [14], and GraphRouter [15]. We also include several static baselines: Best LLM, which achieves the highest performance on the training set; LLM-Mean, which yields the average test performance across all LLMs; and Largest LLM, which denotes the test performance of the largest model. Detailed baseline description can be found in Appendix 12.

Evaluation Metric. Following [14], [15], we adopt three metrics: Performance, the average task accuracy over queries; Cost, the average inference expense of selected LLMs; and LLMScore, a learned evaluator’s quality rating over \(\langle\text{query},\;\text{ground truth},\;\text{ response}\rangle\) triples. Full LLMScore details are in Appendix 18.

4.3 Result Analysis: Query-level Fusion↩︎

We report results in Table ¿tbl:tab:query-level-results? with three key findings: (1) Query-level fusion outperforms the best single LLM across all four scenarios. As shown in Table ¿tbl:tab:query-level-results?, methods such as RouterMLP, RouterKNN, and RouterSVM consistently surpass the best individual LLM, with gains of at least 2% and up to 16% in performance or reward, depending on the scenario and metric. (2) GraphRouter achieves the strongest performance consistently. It brings over 10% relative reward gains in the performance first and balance settings, 16%+ in cost first, and a 5.7% LLMScore improvement under the LLM Judge scenario. Other routers are less stable, sometimes even underperforming the Best LLM baseline, showing the challenge of effective multi-task multi-LLM fusion. (3) GraphRouter effectively fuses smaller LLMs to outperform both the best and largest single models. We further assess GraphRouter’s ability across LLM scales (Table [tbl:tab:2462]). Under the performance-first setting, it consistently beats static baselines across small, medium, and large LLMs (Figure 5), highlighting both the strength of the method and the versatility of LLMFusionBench in LLM fusion.

5 Thought-level FusionFactory Fusion↩︎

5.1 Methodology: Thought-level Fusion↩︎

At the thought level, the core insight is to summarize the responses of multiple LLMs for each query into a thought template, thereby enabling the fusion of different LLMs’ capabilities to enhance few-shot responses. This level relies on the log fields that describe historical reasoning behavior, including historical query \(q_i\), response mode \(s\), generated response \(r\), task-specific performance \(y\), and LLM-judge score \(j\). These fields allow FusionFactory to select informative reasoning traces from aligned multi-LLM responses and abstract them into reusable templates, rather than directly retrieving arbitrary raw responses. Specifically, as shown in the second part on the right side of Figure 4, an LLM is employed to serve as a thought summarizer \(L_s\), distilling a generalized reasoning template \(T_i\) from the top-k responses \(R_i\) of query \(q_i\) selected according to performance or LLM judge: \[\label{eq:topk95selection} R_i = \{ r_1^{i}, r_2^{i}, \ldots, r_k^{i} \} = \text{Top-}k\left(\{r_j^i\}_{j=1}^{n} \text{Score}(r_j^i)\right),T_i = L_s(R_i)\tag{3}\] where \(\{r_j^i\}_{j=1}^{n}\) denotes the set of responses generated by \(n\) different LLMs for query \(q^i\), and \(\text{Score}(r_j^i)\) is given by task-specific metrics or an LLM judge.

Based on this, for new queries \(q^l\), we retrieve the top-d similar queries \(\mathcal{Q}^l=(q_1,q_2,...,q_d)\) via similarity search and use their corresponding thought templates \(\mathcal{T}^l=(T_1,T_2,...,T_d)\) to enable few-shot prompting of the LLM response \(R^l\): \[\label{eq:query95retrieval} \mathcal{Q}^l = \text{Top-}d\left( \mathcal{Q}, \text{sim}(q^l, q_i) \right),\mathcal{T}^l = (T_1, T_2, \ldots, T_d),R^l = \text{LLM}\left(q^l, \mathcal{T}^l \right)\tag{4}\]

Within the fusion-stage matrix, thought-level fusion occupies a distinct middle stage between routing and distillation. Unlike query-level fusion, it does not merely choose one model before generation. Unlike model-level fusion, it does not require fine-tuning a base model. Instead, it reuses high-quality reasoning traces from the logs as non-parametric guidance at inference time. This makes thought-level fusion useful when users want to exploit historical multi-LLM reasoning patterns while remaining compatible with API-based LLMs and avoiding offline model training.

5.2 Experimental Settings: Thought-level Fusion↩︎

Model and Implementation. We evaluate thought-level fusion across three model sizes and six task domains in LLMFusionBench. For each training query, LLaMA-3 (70B) generates a high-level thought template by summarizing the top-3 responses, selected by Performance and LLMScore, with cost as tiebreaker (prompt in Table 5). To assess effectiveness, we compare three paradigms: (1) Zero-Shot, using only the query; (2) Few-Shot, retrieving three similar queries (by cosine similarity) each with three top responses; (3) Thought Template, using the same three similar queries, but with distilled templates. For Thought Template, we try three strategies: (a) Performance-based, using top-3 responses by Performance; (b) LLMScore-based, by LLMScore; (c) Hybrid, selecting top-5 by Performance, then top-3 by LLMScore. We experiment with llama3-8b (small), llama-3.1-nemotron-51b-instruct (medium), and palmyra-creative-122b (large), chosen as top performers in their scale. Accuracy is averaged by domain and shown as radar plots in Figure 6.

5.3 Result Analysis: Thought-level Fusion↩︎

As shown in Figure 6, thought-level fusion consistently improves performance across all model sizes. Averaged over six task domains, small, medium, and large models improve by 12.7%, 4.8%, and 8.3%, respectively, compared to zero-shot. These improvements manifest differently across reasoning types and model sizes, as detailed below. (1) Strong gains on reasoning-intensive tasks (relative to zero-shot). Thought fusion is most effective on tasks like math and code. Measured against the zero-shot baseline, Hybrid fusion improves math by 21.3% (small) and 15.8% (medium); the large model (Palmyra Creative 122B) gains 57.6% in coding (from 0.33 to 0.52). Commonsense and reading comprehension also improve (14.9% and 16.9% for small models). In contrast, performance drops on world knowledge (-7.9% for small models) and varies on popular knowledge, likely due to these tasks emphasizing factual recall over reasoning. Relative to the stronger few-shot baseline (also shown in Figure 6), the best thought-fusion variant remains positive on math and code but the margin is smaller (within \(\sim\)​5% on several scale–strategy combinations); we therefore present thought-template fusion as a robust alternative to few-shot rather than a uniform improvement over it. (2) Large models still benefit. Thought-level fusion yields notable improvements even for large models: +57.6% in code and +6.8% in math. Medium models gain 11.9% (code) and 15.8% (math), showing the value of distilling diverse reasoning even at large model scales. (3) Hybrid strategy offers best trade-off. Here, accuracy refers to task-specific Performance (e.g., exact-match for QA, pass@1 for code) and quality refers to the LLM-judge score (LLMScore) over \(\langle\)query, ground-truth, response\(\rangle\) triples (Appendix 18); Hybrid selection picks top-5 by Performance, then top-3 by LLMScore as tie-breaker (Section 5.2). Averaged across the six domains for small models, Hybrid achieves the largest mean gain over zero-shot (+12.7%, vs Performance-based +3.6% and LLMScore-based +7.3%); at task granularity, however, no single strategy strictly dominates across all tasks in Figure 6, and the three are best understood as competitive alternatives whose relative ranking depends on the task. The corresponding overall (across-strategy) thought-fusion means for medium and large scales are +4.8% and +8.3% as reported in the opening paragraph of this section, with the per-strategy breakdown for those scales shown visually in Figure 6. See Appendix 13 and 16 for analysis of inference time and benefits of reasoning-augmented responses in thought-level fusion.

6 Model-level FusionFactory Fusion↩︎

6.1 Methodology: Model-level Fusion↩︎

The model-level aims to leverage different LLMs’ high-quality responses as training data to fuse LLM capabilities. In this level, the structured logs provide supervised distillation data by pairing each query \(q_i\) with quality-ranked responses \(R_i\), where the ranking is determined by task-specific performance \(y\) or LLM-judge score \(j\). Thus, model-level fusion uses the feedback signals stored in the logs to select which responses should be internalized by the base model. Specifically, as shown in the last part on the right side of Figure 4, we obtain the top-k responses \(R_i\) of query \(q_i\) selected according to performance or LLM judge as equation (3 ). Based on this, we can obtain a training data containing \(n\) samples \(Z=\{q_i,R_i\}_{i=1}^{n}\). Finally, we leverage \(Z\) to train an LLM \(f_{\theta}\) based on supervised fine-tuning: \[\label{eq:loss95minimization} \theta^{*} = \arg\min_{\theta} \frac{1}{n} \sum_{i=1}^{n} \mathcal{L}\left(f_{\theta}(q_i), R_i\right)\tag{5}\] Once \(\theta^{*}\) is obtained, the fine-tuned \(f_{\theta^{*}}\) can generate responses for different queries.

In the fusion-stage matrix, model-level fusion represents late-stage parametric consolidation. Compared with query-level and thought-level fusion, it requires access to a trainable base model and introduces offline fine-tuning cost. However, once training is completed, inference only requires a single fused model, which can be desirable when users need deployment simplicity or cannot call multiple LLMs during inference. Studying this stage together with query- and thought-level fusion allows us to compare whether multi-LLM logs are more effectively used as routing supervision, prompting guidance, or training data.

6.2 Experimental Settings: Model-level Fusion↩︎

Model and Implementation. To evaluate model-level fusion, we fine-tune the best-performing small LLM (LLaMA-3.1 (8B)) on LLMFusionBench. For each query, we rank and select LLM-generated responses using three strategies from Section 2.1: (1) Task Performance: top-\(K\) by performance; (2) LLM Judge: top-\(K\) by judge score; (3) Hybrid: first by performance, then tie-break with judge score. The selected responses from each strategy serve as training data.

Following knowledge distillation practices [37], [38], we augment training set by including each query’s ground truth along with the top-\(K\) LLM responses. This forms the Top-\(K\) SFT method. We compare it with two baselines: Zero-Shot: directly uses LLaMA-3.1 (8B) without fine-tuning; Top-\(K\) Label-only SFT: trains only on repeated ground truths (replicated \(K+1\) times to match training data volume). This setup tests whether model-level fusion effectively distills multi-LLM knowledge beyond supervised fine-tuning on ground truth alone.

6.3 Result Analysis: Model-level Fusion↩︎

We summarize results in Table ¿tbl:tab:comparison95results95model95fusion? with three key observations: (1) Model-level fusion yields modest gains that are consistent across our reported configurations. Computing the cross-domain mean of Table ¿tbl:tab:comparison95results95model95fusion? for each Top-\(K\) SFT configuration against the zero-shot baseline of \(0.530\), every Top-\(K\) SFT configuration improves the cross-domain mean by \(+0.012\) to \(+0.024\) (range: Performance-based \(+0.012\) at \(K{=}10\) to LLM-Judge \(+0.024\) at \(K{=}5\)). Per-domain strict-win counts vary more: counting wins across all six Top-\(K\) SFT configurations, model-level fusion beats zero-shot on Commonsense (6/6), Popular (6/6), Math (5/6), and Reading (4/6); the two domains where it usually loses are Code (2/6) and World Knowledge (1/6, with 2 additional ties at 0.300). Top-\(K\) Label-only SFT (rows 2 and 4 of Table ¿tbl:tab:comparison95results95model95fusion?) is reported as an ablation isolating the contribution of multi-LLM responses: at \(K{=}5\), multi-LLM Top-\(K\) SFT outperforms Label-only by \(+0.102\) on the cross-domain mean (\(0.549\) vs \(0.447\)); at \(K{=}10\), by \(+0.05\) (\(0.546\)–\(0.548\) vs \(0.496\)). The gap is largest on World Knowledge, where Label-only collapses to \(0.150\) at both \(K\) values, below the zero-shot baseline of \(0.300\). All model-level results are reported under a single SFT seed and a single response-sample draw; multi-seed variance estimation is left to future work. (2) Fusion may hurt on code; we conjecture data scarcity and task mismatch as contributing factors. All fusion strategies perform worse on code. We conjecture that data scarcity (MBPP and HumanEval together provide the smallest training pool of any domain — 494 samples / 6.97M tokens vs \(\geq 900\) samples / \(\geq 8.5\)M tokens for every other domain, Table 3) and the divergence between code generation and QA-style supervision contribute to this result; verifying this hypothesis is left to future work. (3) On the cross-domain mean, LLM-Judge or Hybrid selection dominates metric-based selection. Computing the cross-domain mean of Table ¿tbl:tab:comparison95results95model95fusion?, LLM-Judge selection achieves the highest mean at Top-\(5\) SFT (\(0.554\), driven primarily by Code), and Hybrid selection is best at Top-\(10\) SFT (\(0.548\)); Performance-based selection is dominated by at least one alternative at both \(K\) values. We therefore recommend LLM-Judge or Hybrid selection over rigid metric-based selection, particularly in domains such as Code where exact-match metrics undervalue informative responses.

7 Comparisons Across Three FusionFactory Fusion Levels↩︎

In this section, we compare the three FusionFactory levels on the full test set of 14 tasks (Table ¿tbl:tab:level95495results?), selecting the best-performing configuration for each level: Query-level: GraphRouter using task performance as the routing criterion; Thought-level: Hybrid strategy with thought summarization; Model-level: Fine-tuning with top-\(K\) responses selected by the LLM Judge. For the original cross-level comparison, we use a unified generation setting to compare query-, thought-, and model-level fusion. In the revised table, we further add the strongest individual LLM among all 20 models, representative response/model-level fusion baselines, and a near-duplicate-controlled thought-level setting to provide a more complete comparison. Table ¿tbl:tab:level95495results? further breaks down fusion settings by two factors: Fusion LLM Size (whether responses are fused from all or only small models) and Fusion Criteria (how responses are selected, e.g., task performance vs. LLMScore).

This cross-level comparison is central to the motivation of our fusion-stage matrix. Existing routing and distillation methods often study a single stage of the LLM pipeline in isolation, such as query-level model selection or model-level capability transfer. In contrast, our comparison asks how the same structured multi-LLM log data should be used across different intervention points: before generation for routing, during generation for thought-level reasoning augmentation, or after response collection for supervised fine-tuning. Therefore, Table ¿tbl:tab:level95495results? is not merely an engineering comparison, but a decision-oriented matrix for understanding the performance, cost, and generalization trade-offs among different ways of exploiting multi-LLM logs.

Consistent with the empirical nature of this work, our goal here is not to derive formal theoretical guarantees or closed-form complexity bounds. Instead, we use the unified benchmark setting to provide an empirical trade-off analysis of the three fusion stages. Query-level fusion tests whether logs are best used as routing supervision, thought-level fusion tests whether logs are best used as reusable reasoning guidance, and model-level fusion tests whether logs are best used as distillation data. This comparison helps characterize the practical behavior of different fusion stages under the same data source and evaluation protocol.

Using unified experimental settings, we now summarize and compare the results across all three fusion levels. (1) Thought-level fusion achieves the best overall performance. Thought-Full-Hybrid-70B outperforms all settings, demonstrating its ability to integrate reasoning from diverse LLMs. However, performance gains plateau when increasing model size or response count, suggesting diminishing returns relative to cost. After adding the strongest individual LLM and representative fusion baselines, Thought-Full-Hybrid-70B still achieves the best average performance, outperforming Best LLM (0.615 vs. 0.600), Self-Consistency (0.615 vs. 0.589), LLM-Blender (0.615 vs. 0.603), and FuseLLM-adapted (0.615 vs. 0.557). This indicates that abstract thought-template retrieval provides benefits beyond selecting the strongest single model or aggregating final responses. This result suggests that multi-LLM logs are especially useful when their high-quality reasoning traces are reused as non-parametric guidance at inference time. Compared with pure routing, thought-level fusion can expose the responding model to diverse reasoning patterns rather than selecting only one model. Compared with model-level fusion, it avoids compressing heterogeneous task behaviors into a single set of parameters. Empirically, this gives thought-level fusion a favorable balance between flexibility and effectiveness, especially for tasks where reusable reasoning patterns are more valuable than simply selecting a model. The near-duplicate-controlled setting further supports this interpretation. After removing retrieved training queries with embedding similarity above 0.9, thought-level fusion still achieves 0.604 average performance, remaining above zero-shot (0.567) and close to the original thought-level result (0.615). This suggests that the gain is not solely driven by retrieving near-duplicate training queries, but also reflects reusable reasoning patterns captured by the thought templates.

(2) Model-level fusion underperforms despite access to high-quality responses. Fine-tuned models outperform zero-shot in only 3 of 6 domains, and even fall behind on average. We hypothesize two contributing factors — overfitting to multi-response inputs and difficulty abstracting across heterogeneous task styles during SFT — and view rigorous verification (e.g., per-task SFT ablations) as future work. This observation highlights an important limitation of treating multi-LLM logs purely as distillation data. Although model-level fusion has deployment advantages after training, the offline consolidation process may introduce training bias, reduce adaptability, or overfit to the particular response styles selected from the logs. In an empirical bias-generalization interpretation, model-level fusion can reduce inference-time variability by using a single fine-tuned model, but it may increase training-induced bias when heterogeneous responses from different tasks and LLMs are compressed into one base model. This explains why API-compatible distillation alone does not fully address the broader question of how multi-LLM logs should be used. The adapted FuseLLM baseline shows a similar trend, achieving 0.557 average performance, which is close to our model-level fusion result but still below thought-level fusion. This further suggests that directly compressing heterogeneous multi-LLM logs into model parameters may be less effective than using them as retrieval-time reasoning guidance.

(3) Query-level fusion achieves moderate accuracy with the lowest cost. Query-level fusion via GraphRouter avoids running the backbone on most queries and therefore achieves the lowest per-query cost: under cost-aware routing, GraphRouter cost is \(0.0184\) vs \(0.1850\) for Best-LLM under Performance First, a roughly \(10\times\) reduction (Table ¿tbl:tab:query-level-results?). Thought- and model-level fusion in this section’s unified setting incur full LLaMA-3 (8B) inference cost per query, computed as the LLaMA-3 (8B) per-token price (Table 1) applied to the per-query token counts available in LLMFusionBench(Table 2). Thought-level fusion additionally incurs \(\sim\)​5% latency overhead and \(0.03\)s retrieval overhead over zero-shot (Appendix 13, Table 4). The cost ordering across fusion levels follows directly from this routing-vs-full-inference distinction. This result confirms the value of query-level fusion for cost-sensitive API-serving scenarios. Since query-level fusion only requires a lightweight router and calls one selected LLM at inference time, it provides a practical option when users prioritize efficiency. However, its performance ceiling can be limited because it exploits multi-LLM logs only through model selection, without reusing richer reasoning traces or consolidating capabilities into parameters. Empirically, query-level fusion therefore has a low-overhead profile, but may be biased toward the capability boundary of the selected LLM. Compared with response-level baselines, query-level fusion achieves lower average performance than LLM-Blender (0.585 vs. 0.603), but it requires only one selected model call rather than aggregating multiple candidate responses. This highlights its distinct role as an efficiency-oriented fusion strategy rather than a purely performance-oriented one.

(4) Fusion provides only marginal improvements on World Knowledge and Math. On these domains, the best fusion result exceeds zero-shot by at most 2 points absolute (Table ¿tbl:tab:level95495results?: Math thought-level \(0.612\) vs zero-shot \(0.602\); World Knowledge query-level \(0.338\) vs zero-shot \(0.334\)), suggesting these domains — which depend heavily on factual recall and strict logical sequencing — are less amenable to multi-LLM fusion under our current framework. These domain-level results further support the need for a fusion-stage matrix. Different tasks benefit from different ways of using multi-LLM logs. Knowledge-intensive tasks may require strict factual grounding and are sensitive to noisy fused information, while mathematical tasks require coherent step-by-step reasoning and may be harmed by inconsistent reasoning traces. Therefore, no single fusion level uniformly dominates across all settings, and a systematic comparison is needed to determine which fusion stage is appropriate under different domain requirements.

Taken together, these observations provide an empirical interpretation of the trade-offs among the three fusion levels. Query-level fusion is efficient and easy to deploy, but it mainly benefits from better model selection. Thought-level fusion is more flexible because it reuses reasoning traces without modifying model parameters, but it introduces retrieval and prompt-token overhead. Model-level fusion is attractive when a single deployable model is desired, but it requires offline training and may generalize poorly when the logs contain heterogeneous task patterns. The additional comparisons with Best LLM, response-level fusion baselines, and near-duplicate-controlled retrieval further show that the gains of thought-level fusion are not simply due to a stronger single model, final-response aggregation, or near-duplicate retrieval. This empirical trade-off analysis complements the stage-aware taxonomy in Section 3.

Overall, these findings validate our central claim: multi-LLM log data, when systematically structured by query / task / response triples and ranked by performance and LLM-judge scores, supports fusion strategies that outperform single-LLM baselines. On the full test set (Table ¿tbl:tab:level95495results?), thought-level fusion improves over zero-shot by \(4.8\) points on average across the six domains (\(0.567 \rightarrow 0.615\)); under cost-aware routing, query-level fusion matches the strongest single LLM at an order of magnitude lower cost (Table ¿tbl:tab:query-level-results?: GraphRouter \(0.602\) reward / \(0.0184\) cost vs Best-LLM \(0.546\) / \(0.1850\) under Performance First). The framework supports both local and API-served LLMs and applies across all three fusion stages, and LLMFusionBench provides a demanding benchmark for real-world multi-LLM integration. More importantly, the comparison demonstrates the unique value of our work beyond existing API-only routing or distillation methods. The main contribution is not simply that a method can operate with API-accessible models, but that the same multi-LLM log source can be systematically evaluated across query-level, thought-level, and model-level fusion. This matrix reveals when logs are better used as routing supervision, when they are better used as reasoning templates, and when they are less effective as fine-tuning data. Such cross-stage evidence would be difficult to obtain from studies that focus only on routing, response-level fusion, or distillation.

8 Additional Related Work↩︎

LLM fusion. Recent work explores fusing multiple LLMs or reasoning traces to enhance performance, robustness, and generalization. Query-level methods (e.g., LLMRouter [39], STaR [40]) treat routing as policy learning to select the best model per query. Thought-level methods (e.g., Thought Propagation [41], Tree of Thoughts [25]) aggregate reasoning paths for more consistent answers. Model-level methods (e.g., FrugalGPT [42], Self-Consistency [43]) combine multiple model outputs or samples to approximate stronger reasoning while controlling cost. LLM-as-judge approaches (e.g., LLM-as-Judge [44], RAG Fusion [45]) evaluate and select responses via a secondary LLM. However, most prior studies focus on a single level of fusion. We instead introduce LLMFusionBench, a large-scale benchmark for LLM fusion, and conduct a systematic comparison across three levels via FusionFactory, enabling more holistic and integrated insights.

LLM Router. Recent work on LLM routing focuses on improving efficiency and response quality by directing queries to the most suitable model. Early methods like RouteLLM [46] optimize preference-aligned model selection via learned policies, while RouterBench [20] benchmarks routing decisions but does not study routing logs for capability fusion. TO-Router [47] and Expert Router [48] emphasize parallel dispatch for efficiency, and Glider [49] integrates global and local signals to refine expert selection. In contrast, our work focuses on capability-oriented analysis. We leverage large-scale multi-LLM log data not only for model selection, but as a form of weak supervision to systematically study and fuse LLM capabilities across three flexible levels.

9 Conclusion↩︎

In this work, we revisit LLM fusion through the lens of API-based multi-LLM log data, arguing that practical fusion must (1) be compatible with real-world serving (local and API) and (2) support integration at different pipeline stages. We introduce LLMFusionBench, a large-scale benchmark spanning 14 tasks and 20 LLMs with diverse response modes and reusable thought templates, and propose FusionFactory, a three-level framework for stage-aware fusion. Experiments show that thought-level fusion delivers the strongest gains, query-level fusion offers the best accuracy–cost trade-off, and model-level fusion lags due to generalization limits. Overall, these results validate multi-LLM log data as a practical supervision source and position LLMFusionBench as a demanding, real-world testbed for flexible, serving-compatible LLM fusion methods. FusionFactory is model-agnostic by construction: extending the LLM pool to newer frontier models (e.g., GPT-5, Claude 4, Gemini 2) requires only re-running response collection on the existing query set, with no architectural change to the fusion pipeline.

10 Limitations and Future Work↩︎

While our work focuses on evaluating real-world task performance - particularly in domains like math and coding - future research should also investigate aspects of trustworthiness [50], [51] and justice [52], [53] in the fusion process. The act of fusing multiple responses may inadvertently introduce social biases, inaccuracies, or conflicting reasoning, which could impact the reliability and ethical soundness of the LLMs. Exploring safeguards or calibration techniques to ensure more robust and equitable fusion outcomes presents a valuable direction for further study.

Our study focuses on a practical and broadly accessible form of multi-LLM log data that can be collected in both local and API-based serving scenarios. Specifically, LLMFusionBench records aligned responses from multiple LLMs for the same query, together with model identity, response mode, inference cost, task-specific performance, and LLM-judge scores. These fields provide useful supervision signals for query-level routing, thought-level template construction, and model-level distillation. This design choice is intentional because, in many real-world deployments, especially when using closed-source or API-based LLMs, users may only have access to textual responses and serving metadata rather than model logits, internal states, or training-time information.

Richer production logs may additionally contain interaction-level signals, such as temporal order, explicit user feedback, error patterns, user revision behavior, and multi-turn interaction history. These signals can potentially enable more fine-grained fusion strategies. For example, temporal order may help track how model behavior changes over time, user feedback may provide direct preference supervision, and interaction history may support personalized or context-aware fusion. However, such signals are not always consistently available across public benchmarks, local models, and API-based serving systems. Incorporating these richer interaction-level log signals is therefore an important direction for future extensions of LLMFusionBench and FusionFactory.

A related concern is the privacy risk associated with log-based fusion. Multi-LLM logs may contain user queries, model responses, feedback, and interaction traces, which can include sensitive or personally identifiable information in real-world deployments. Although LLMFusionBench is constructed from public evaluation datasets and does not involve private user data, practical systems built on multi-LLM logs should adopt privacy-aware data handling procedures. These may include anonymization, personally identifiable information filtering, access control, retention policies, and aggregation or synthetic-log generation before logs are used for retrieval, training, or distillation.

Another limitation is the potential negative effect of low-quality logs. Since FusionFactory relies on logged responses and feedback signals, noisy, biased, hallucinated, or incorrectly scored responses may propagate into the fusion process. In query-level fusion, low-quality logs may lead to suboptimal routing policies. In thought-level fusion, misleading responses may be summarized into unhelpful thought templates. In model-level fusion, noisy or biased responses may be amplified through supervised fine-tuning. Future work should therefore investigate stronger log-quality control mechanisms, such as uncertainty-aware filtering, consistency checking, human or model-assisted verification, and robust training objectives that reduce the influence of unreliable logged responses.

In addition to log quality, API-based deployment of fusion systems may introduce reliability and transparency risks. A query-level router may select an unsuitable model, and a thought-level retriever may retrieve irrelevant or misleading templates, causing errors that are difficult to diagnose from the final response alone. Practical fusion systems should therefore expose routing decisions, retrieved templates, confidence estimates, and fallback mechanisms whenever possible, so that users and developers can better audit and control the fusion process.

Our current benchmark mainly focuses on single-turn benchmark queries, while real-world LLM logs often involve long documents, evolving user intents, tool-use traces, and multi-turn interaction histories. These settings can introduce additional failure modes. For long-text inputs, retrieved thought templates may omit important context or conflict with document-specific evidence. For multi-turn dialogue, query-level similarity alone may fail to capture changing user goals, and model-level distillation may compress incomplete or stale interaction states. Extending LLMFusionBench and FusionFactory to long-context reasoning, multi-turn dialogue, tool-augmented agents, and distribution-shifted deployment settings is an important future direction.

This work is primarily an empirical benchmark and framework study. We focus on constructing a large-scale multi-LLM log benchmark and systematically evaluating how the same log source can be used across query-level, thought-level, and model-level fusion. Therefore, our goal is not to provide formal theoretical guarantees or closed-form complexity bounds. A more formal theoretical treatment of multi-LLM fusion remains an important future direction. For example, future work could develop formal criteria for selecting fusion stages based on query difficulty, uncertainty, domain shift, cost constraints, or the expected reliability of retrieved logs.

Although several prior routing and distillation methods already support API-only settings, our study emphasizes a broader fusion-stage matrix rather than API compatibility alone. Existing routing methods mainly exploit logs for query-level model selection, while distillation methods mainly use model outputs for model-level capability transfer. In contrast, FusionFactory studies how the same structured multi-LLM log data can support routing, reasoning-template augmentation, and supervised fine-tuning under a unified benchmark and evaluation protocol. This matrix provides a practical decision framework for choosing how to use multi-LLM logs under different deployment constraints.

Future work could further extend this fusion-stage matrix by developing hybrid or adaptive fusion strategies. For example, a system may use query-level fusion for simple or cost-sensitive queries, invoke thought-level fusion for reasoning-intensive queries, and apply model-level fusion when repeated high-quality logs are available in a stable domain. Such adaptive strategies could dynamically choose the appropriate fusion level according to task difficulty, budget, domain type, or uncertainty estimates, further improving the utility of multi-LLM log data in real-world deployments.

11 More on LLMFusionBench↩︎

Here, we list all the tasks and the metrics used for each in Table ¿tbl:app95tab:1?, and all models along with their sizes and costs in Table 1.

6pt

12 Router Baseline Description for Query-Level Fusion↩︎

For the query-level fusion, we evaluate five representative routing methods to assess their LLM fusion capabilities: (1) RouterKNN [36], a non-parametric baseline that routes by nearest neighbors in query space and selects the majority LLM label; (2) RouterSVM [20], a support vector machine trained on query features and task labels; (3) RouterMLP [36], a multi-layer perceptron leveraging query embeddings and task context; (4) RouterBERT [14], a compact BERT classifier that encodes both query and task to predict the optimal LLM; and (5) GraphRouter [15], a graph-based model that treats routing as node classification over a heterogeneous graph of queries, tasks, and LLMs with learned edge interactions. All models use all-MiniLM-L6-v2 for embeddings, except GraphRouter, which employs Longformer [80] to obtain the embeddings for task/query/LLM. We also include several static baselines: Best LLM, which achieves the highest performance on the training set; LLM-Mean, which yields the average test performance across all LLMs; and Largest LLM, which denotes the test performance of the largest model.

13 Inference Time for Thought-Level Fusion↩︎

13.0.0.1 Setup and offline preprocessing.

As introduced in Table ¿tbl:Tab:FusionBench95intro? and Figure 1, the thought templates used by the thought-level fusion style are summarized offline on LLMFusionBench. For new queries, we do not re-generate summaries; we only retrieve the most relevant thought templates and prepend them to the prompt. Retrieval is implemented with FAISS, and responses are produced using the NVIDIA NIM API. This design minimizes online overhead and keeps inference practical for API-based serving.

13.0.0.2 Inference-time comparison.

Table 4 reports average inference times on the full test set with LLaMA-3 (8B) as the backbone across three paradigms: Zero-Shot, Few-Shot, and Thought-Hybrid. Thought-level fusion incurs only a minor increase relative to Zero-Shot and Few-Shot, which is attributable to the extra input tokens from the retrieved thought templates. The average lookup time for the thought-based method is 0.03 s, accounting for 0.7% of the total Thought-Hybrid inference time, which is essentially negligible. In return, the performance gains are substantial. We also expect the relative overhead to further diminish with larger base models, since total inference time increases while lookup time remains nearly constant.

14 Prompts for Thought-Level Fusion↩︎

Previous studies have shown that thought templates can enhance the performance of LLMs [81], [82]. However, existing templates are often either too general to a big category of questions [82] or too specific, tailored to only one particular case [81]. Our prompt 5 is designed to strike a balance between these extremes: it is specific enough to guide similar problems effectively, yet general enough to be applicable across a range of variations.

15 Detailed Results for Comparison Across Three FusionFactory Levels↩︎

In this section, we present detailed results for each task across all methods from the three aforementioned fusion levels, as shown in Table 6.

16 More Analysis: Does Reasoning Data Help for Reasoning-Oriented Tasks?↩︎

16.0.0.1 Motivation.

As discussed around thought-level fusion, LLMFusionBench includes both reasoning and non-reasoning responses (LLM-think vs.LLM-direct).

16.0.0.2 Analysis setup.

We analyze, under the thought-level hybrid selection setting, the composition of best-selected responses per task (reasoning vs.non-reasoning). This directly tests whether reasoning-style data is preferentially selected for tasks that demand more reasoning.

16.0.0.3 Findings.

Table 7 shows that (1) reasoning ratios are higher for most reasoning-oriented tasks—Math, Code, Commonsense Reasoning, and Popular—indicating that such tasks benefit from reasoning traces. (2) In contrast, World Knowledge and Reading Comprehension skew toward non-reasoning outputs, consistent with their stronger reliance on factual recall and concise evidence extraction. Overall, these results support that reasoning-level information in LLMFusionBench aids selection for reasoning-heavy tasks while preserving flexibility for knowledge-intensive ones.

17 Implementation details of FusionFactory↩︎

We utilize NVIDIA API¹ for our API calling in data generation and model inference. We also use a NVIDIA A6000 GPU to obtain all the embeddings for queries/tasks/LLMs described in section 4.2.

For model-level fusion, we adopt LLaMA-Factory² to fine-tune the model with LoRA, which is conducted on NVIDIA A6000 GPUs. For LoRA, we apply adaptation to all transformer layers with a rank of 8. Inputs are processed with a maximum context length of 2048 tokens unless otherwise specified. The training uses a per-device batch size of 8 and a gradient accumulation step of 4, resulting in an effective batch size of 32. We set the learning rate to 1e-4 and train the model for 3 full epochs. A cosine learning rate scheduler is used with a warmup ratio of 0.1 to ensure stable convergence. Training is conducted using bfloat16 precision, and Flash Attention 2 is enabled to accelerate attention computation. Note that, as shown in Table 3, the training data in the Code domain is relatively smaller and more challenging compared to other domains. Therefore, following the approach adopted in previous work [83], [84], we train a separate model for the Code domain, while a unified model is trained for the remaining domains.

18 Prompts for LLM-As-Judge Score Generation↩︎

We present the prompt for LLM-As-Judge Score Generation in Table 8.

19 Descriptions for tasks and LLMs↩︎

Using task and model descriptions generated by LLMs enhances the expressiveness and generalization capability of the router. In this work, we present descriptions of various tasks and LLMs generated by GPT-4o. Specifically, GPT-4o captures the unique characteristics and challenges associated with each task, as well as the size, cost, and distinct strengths of different LLMs. These detailed descriptions are provided in the tables below.

20 Case Study for All Levels↩︎

To better understand the behavior of LLMs under different fusion strategies, we conduct a series of case studies spanning diverse task types, including Math, Code, Commonsense Reasoning, World Knowledge, Reading Comprehension, and Popular questions. Each example is evaluated under four levels: Zero-shot, Query-level, Thought-level, and Model-level. These comparative analyses provide qualitative insights into how reasoning granularity and intervention level impact model performance, faithfulness, and error patterns. See Tables 10, 11, 12, 13, 14, and 15 for detailed comparisons across tasks.

21 The Use of Large Language Models (LLMs)↩︎

We used large language models (LLMs) solely for minor editing purposes, such as checking grammar and improving the readability of the text.

References↩︎

---

1. https://build.nvidia.com/nim↩︎

2. https://github.com/hiyouga/LLaMA-Factory↩︎