1 Introduction↩︎

Tool learning represents a crucial step in advancing the frontier of Large Language Models (LLMs), as it transforms them from passive language processors into proactive agents capable of interacting with dynamic environments [1], [2]. By bridging internal reasoning with external execution, tool-equipped LLMs are no longer constrained by static training data but can generalize to open-ended tasks and adapt to evolving user needs [3], [4]. This paradigm thus underscores not only the practical value of enhancing task coverage across diverse applications such as workflow automation and travel planning [5], [6], but also the theoretical importance of extending the fundamental boundaries of LLM capabilities [7], [8].

Despite the rapid progress in tool learning, the field still faces fundamental challenges that constrain its broader applicability. The effectiveness of function-calling (FC) relies heavily on high-quality training data, yet collecting and annotating real-world data is costly and labor-intensive [2]. Moreover, real-world function-calls are inherently complex and diverse, often characterized by ambiguous user intents, rapidly changing dynamic environments, multi-task requirements, and extended multi-turn interactions [3], [7], [9], [10]. These intertwined challenges underscore the urgent need for more robust data generation pipelines that can produce accurate, diverse, and broadly representative training data to support realistic function-calling scenarios.

Increasing research efforts have sought to design automated pipelines for synthesizing training data, which are then used to build tool-augmented LLMs [4], [8]. While progress has been made, fundamental limitations remain. Existing methods often rely on publicly available or manually constructed APIs, which are frequently unreliable and difficult to scale across broader tool sets, thereby constraining function-calling capabilities. In addition, the generated data often suffers from insufficient diversity and inadequate quality control, further weakening performance. From the perspective of scenario coverage, most approaches still emphasize single-turn or isolated function calls, leaving them unable to capture more realistic settings such as multi-turn dialogues or multi-task interactions. Together, these limitations suggest the insufficiency of current approaches and highlight the need for more effective and scalable solutions.

In this paper, we introduce GenesisFunc, a three-stage automated pipeline designed to generate function-calling training data that is high-quality, diverse, and broadly representative of real-world scenarios. To ensure the reliability of APIs and enable seamless extension to a wider range of tools, GenesisFunc begins by selecting a curated seed set from the widely adopted BFCL benchmark [11]. These APIs are verified to cover multiple domains, making them not only dependable but also practical for extension to downstream tools, since such functions are readily accessible in real applications [12]. Next, to improve dialogue quality, diversity, and coverage, GenesisFunc incorporates a multi-agent-assisted dialogue generation system. Through the coordinated interaction of agents, the framework leverages history-aware role differentiation and parameter-slot selection to expand diversity, while an agent-based scoring mechanism safeguards the quality of synthesized dialogues. Finally, to further guarantee correctness, GenesisFunc employs a multi-stage evaluation module that combines rule-based and model-based checks, followed by targeted human review, thereby ensuring both conformity and executability of the training data. Through this end-to-end process, GenesisFunc produces robust, diverse, and scenario-rich training data, which significantly enhances the function-calling capabilities of LLMs.

To validate the effectiveness of GenesisFunc, we conduct supervised fine-tuning on Qwen3-8B [13] using the training data generated by our pipeline and evaluate the zero-shot performance of the fine-tuned model across several public benchmarks. Since the tools are sourced from BFCL [11], we report results not only on the in-domain BFCL benchmark but also on the out-of-domain API-Bank [14] and ACEBench [15], thereby assessing both in-domain performance and generalization to unseen domains. Under a consistent evaluation protocol, our model consistently surpasses open-source baselines of comparable scales and remains highly competitive with API-based models [16]. Furthermore, we demonstrate that our approach can scale up to more downstream tools and verify that the data generated by our pipeline can also be used through reinforcement learning to enhance function-calling performance in multi-turn dialogue scenarios.

In summary, this paper makes three primary contributions: (1) We introduce GenesisFunc, an automated pipeline for generating high-quality function-calling training data for LLMs. The pipeline begins with selecting reliable APIs and integrates a Multi-agent Dialogue Generation Module and a Multi-stage Evaluation Module to ensure robustness, diversity, and scenario coverage. (2) Through extensive verification, GenesisFunc produces datasets that are accurate and diverse while covering real-world scenarios. Moreover, our approach demonstrates strong generalizability to downstream tools, underscoring its practical applicability. (3) We fine-tune Qwen3-8B with training data generated by our pipeline and evaluate the GenesisFunc-8B on three widely used benchmarks, BFCL [11], API-Bank [14], and ACEBench [15]. GenesisFunc-8B consistently outperforms open-source LLMs of comparable scales and remains highly competitive with API-based models.

2 Related Work↩︎

LLM Function-Calling Paradigms. Equipping LLMs with executable tools enables reliable and specialized problem solving [2]. Existing approaches mainly follow two paradigms: prompting and fine-tuning. Prompting leverages in-context tool specifications and demonstrations [17], [18]. ReAct [19] combines reasoning with API calls for multi-step tasks, but its performance is constrained by pretraining and degrades with increased tool complexity. These limitations motivate fine-tuning via supervised or reinforcement learning [9], [10]. Representative methods include ToolACE [8], which constructs large-scale tool-use corpora automatically, and RL-based approaches such as ToolRL [20] and AWPO [21], which enhance function-calling through policy optimization with reasoning rewards.

Data Synthesis for Function-Calling. As LLMs advance, reliance solely on human-authored corpora becomes insufficient for sustained progress [22]. To expand supervision without heavy annotation costs, recent works adopt prompt-driven transformations to augment existing datasets. For example, [23] proposes targeted prompting to elicit rare skills and long-tail behaviors from base models. However, purpose-driven data remain limited in tool-use scenarios. Prior efforts adapt resources from adjacent domains [24] or synthesize samples around public APIs [7]. Moreover, ToolACE [8] constructs large-scale function-calling corpora via automated pipelines, while ToolForge [25] introduces automated tool-use synthesis with reduced reliance on real APIs.

Our work is distinguished from the most relevant studies [2], [7], [8] in the following aspects. Prior works often rely on annotated or synthetic APIs, which lack reliability and struggle to scale across larger tool sets. These approaches also face limitations in diversity, quality, and coverage. In contrast, GenesisFunc is built upon reliable tools drawn from public benchmarks and offers stronger scalability. Furthermore, by leveraging a multi-agent dialogue generation framework and a multi-stage verification system, GenesisFunc produces tool-use training data with richer diversity, higher quality, and broader scenario coverage than prior works.

3 Preliminary↩︎

Given a user query \(q\) and a candidate tool set \(\mathcal{T}=\{t_1,\ldots,t_n\}\), where each tool \(t_i\) is defined by a name, a usage description, and a schema specifying required and optional parameters, the goal is to generate a valid sequence of tool calls by selecting tools and filling arguments with appropriate values and units. This process can be framed as follows: \[\mathcal{S}=\big[t_1(a_1),\ldots,t_k(a_k)\big]=g_\phi\!\big(q,\mathcal{T}\big), \label{eq:tool-call-reform}\tag{1}\] where \(g_\phi(\cdot)\) denotes an LLM with parameters \(\phi\), \(k\) is the number of invocations, and \(a_i\) denotes the argument payload for the \(i\)-th call (\(1\le i\le k\)), which is a set of parameter-value pairs, that is, \(a_i=[\,r_1\!:\!w_1,\,r_2\!:\!w_2,\ldots,r_\ell\!:\!w_\ell\,]\), with parameter names \(r_j\) and corresponding values \(w_j\). The query \(q\) may be a single turn or a full multi-turn history.

For fine-tuning, the pair \((q,\mathcal{T})\) is treated as input context, while the gold-standard tool-call sequence \(\mathcal{S}\) serves as the supervised target. Formally, the training samples can be represented as \(\{\,\langle q,\mathcal{T}\rangle,\mathcal{S}\,\}\), where each training instance connects a user query and the candidate tools with the corresponding sequence of tool invocations.

4 Data Generation Pipeline↩︎

Synthetic data plays a vital role in enhancing the function-calling capability of LLMs [8]. We propose an automated pipeline GenesisFunc that employs a structured, three-stage process to generate high-quality training data for tool-augmented LLMs, as illustrated in Figure 1. First, reliable and functionally diverse tools are selected from public datasets and placed into the Tool Pool. Next, the Multi-Agent Dialogue Generation module leverages a multi-agent-assisted framework to produce tool-use dialogues that are diverse, accurate, and broadly representative. Finally, the Multi-Stage Evaluation module systematically examines the correctness of the generated dialogues to guarantee training data quality.

4.1 Tool Pool↩︎

The quality of training data for enhancing LLM tool-use abilities depends critically on the reliability and coverage of the underlying APIs. However, existing datasets often rely on manually annotated APIs or handcrafted synthetic pipelines, which are either costly to scale or difficult to extend to real-world tools, limiting model generalizability.

To address these issues, GenesisFunc constructs a curated Tool Pool entirely sourced from the BFCL evaluation set [11]. The pool is built through a two-stage filtering process: GPT-4o [16] is first used to perform semantic clustering over all tools to remove redundant or highly similar ones, followed by light human verification to ensure correctness and usability. This process yields a curated Tool Pool of 1,000 tools. Since tools in BFCL are manually designed to cover a broad range of real-world usage scenarios, we further verify that the curated pool maintains high functional diversity across multiple domains. By relying on widely used and practically accessible tools, the Tool Pool balances reliability, diversity, and scalability, and serves as a critical foundation for the subsequent data generation modules in GenesisFunc.

4.2 Multi-Agent Dialogue Generation↩︎

For LLMs, the quality of fine-tuning data significantly affects downstream performance [26]. We propose a multi-agent dialogue generation module that leverages a multi-agent framework to assist the dialogue generation system in synthesizing function-calling dialogues. By leveraging agent interaction and collaboration, this module produces dialogues that are high-quality, diverse, and broadly representative, which is crucial to enhance tool-use capabilities of LLMs.

4.2.1 Multi-Agent Framework↩︎

The multi-agent framework includes four LLM-based agents: Sample Agent, Memory Agent, Function Agent, and Judge Agent. We adopt Gemini-2.5 Pro [27] as the backbone of all agents, which is an engineering choice due to Gemini-2.5 Pro’s stable API-following performance rather than an algorithmic requirement. Importantly, the multi-agent framework is model-agnostic; Appendix 7.1 shows that replacing proprietary models with strong open-source alternatives retains the overall workflow and leads to only moderate data quality degradation and slight downstream performance drops. System prompts and detailed agent descriptions and analyses are also in Appendix 7.1.

Sample Agent selects a subset \(\mathcal{T}_s \subseteq \mathcal{T}\) from the global Tool Pool, including target tools \(\mathcal{T}_{\text{target}}\) and distractors \(\mathcal{T}_{\text{dist}}\). Target tools are grouped by semantic or functional similarity, while distractors are chosen with varied relevance to increase realism and require the model to distinguish between strongly and weakly related tools.

Memory Agent records dialogue instances \(\mathcal{D}=\{d_1,\ldots,d_k\}\) and assigns each a type label \(\tau(d_i)\in\mathcal{C}\) reflecting the semantic context of tool usage, where the same math tool may address architectural geometry in one case and year calculations in another. It then summarizes past pairs \((d_i,\tau(d_i))\) and guides the generator to produce new dialogues with unseen \(\tau\), improving semantic diversity.

Function Agent selects one or more tools \(\{t_1, \ldots, t_k\} \subseteq \mathcal{T}_s\) to address the user query \(q\), and extracts both required and optional parameters for each tool \(t_i\). To enhance diversity and realism, a slot-selection strategy is applied: during parameter extraction, the agent randomly chooses a subset \(\mathcal{P}' \subseteq \mathcal{P}\) of optional parameters to instantiate, resulting in varied and realistic tool calls.

Judge Agent ensures rigorous quality control by selecting the best dialogue \(d^*\) from \(N\) candidates \(\{d_1,\ldots,d_N\}\) (default \(N=4\)) each generated round. Evaluation considers problem significance and tool appropriateness, with positional bias controlled as discussed in Appendix 7.1. \(N=4\) balances selection quality and generation efficiency. The selected \(d^*\) serves as the final output.

4.2.2 Dialogue Generation System↩︎

We design a dialogue generation system to produce three types of function-calling dialogues: single-turn, multi-turn, and special-case. The first two types cover both single-task and multi-task settings, enabling the simulation of simple requests and more complex interaction flows. The special cases, on the other hand, are designed to cover scenarios for relevance checking and error detection. The overall system is composed of two agents, a user agent \(U\) and an assistant agent \(A\), both instantiated with Gemini-2.5 Pro, that interact with each other to generate task-oriented conversations.

The detailed algorithm of the multi-agent dialogue generation module and dialogue examples are provided in Appendix 7.2. The dialogue generation system grounds each conversation in the sampled tool set \(\mathcal{T}_s\) drawn from the curated pool. Every dialogue contains user requests that can be handled by \(\mathcal{T}_s\) and tool call answers that explicitly specify the selected tools and the full argument payload. The assistant’s action space is \(\mathcal{A}_{\text{asst}}=\{\texttt{call},\,\texttt{ask},\,\texttt{answer}\}\). The assistant can issue a tool call \(t(a)\), request clarification, summarize tool outputs, or provide a direct non-tool reply when a call is unnecessary. The system records the chosen tool \(t\) and the payload \(a\) in the turn-level state and appends them to the turn-level trajectory for later evaluation and reuse. These recorded trajectories, after being categorized by the memory agent, are leveraged to guide subsequent dialogue generation, preventing the system from reproducing similar contexts and encouraging scenario diversity.

At turn \(t\), the user agent \(U\) issues a request \(q_t\) based on the tool set \(\mathcal{T}_s\). The assistant agent \(A\) receives the request \(q_t\) together with the dialogue history \(\mathcal{D}\), follows the system prompts, and generates the next action. In single-turn dialogues, \(A\) constructs a problem solvable with \(\mathcal{T}_s\), produces a tool call, and returns the final answer. In multi-turn dialogues, \(A\) alternates with \(U\) by requesting additional information when constraints are missing, and once the target length is reached or a stop signal triggers, the model outputs either call or answer. In special cases, prompts guide the generation of unsolved or erroneous samples, such as mismatched tools or invalid parameter values, to support relevance checking and error detection.

4.3 Multi-Stage Evaluation↩︎

We introduce a multi-stage evaluation system to assess the quality of synthesized dialogues, since inaccurate training data may significantly weaken the models’ ability to understand and execute functions. This system integrates a rule-based checker and a model-based checker, with final verification by human experts to ensure the accuracy of the resulting training data. More details can be found in Appendix 7.3.

(1) Rule Checker. Without executing tools, the Rule Checker performs basic compliance checks on synthesized dialogues to quickly filter out samples with format and alignment issues. It validates four aspects: completeness of the tool definition, compliance of the call format and parameters, soundness of the dialogue structure, and consistency between tool results and the assistant’s statements (rule details in Appendix 7.3) Each rule returns a pass or fail flag with a hint for correction, and outputs are aggregated for the downstream Model Checker and Human Validation.

(2) Model Checker. After format screening, the Model Checker employs GPT-4o [16] to evaluate semantic quality and task completion beyond the static rules (the prompt is detailed in Appendix 7.3). Unlike the Judge Agent that ranks candidate dialogues during generation, the Model Checker works post-hoc to verify finalized dialogues. Given the dialogue context and tool calls, it checks faithfulness, task satisfaction, and compliance, returning a rationale and a confidence score. Only samples with confidence scores above the threshold (\(\theta = 0.75\)) are retained.

(3) Human Validation. After the Rule Checker and the Model Checker complete automated screening, the remaining error rate is below 5%. Manual analysis shows that over 80% of the errors after automated screening stem from parameter extraction rather than function selection or intent understanding. Samples that fail the Rule Checker or obtain low-confidence from the Model Checker are routed to a human review, with higher priority given to errors that prevent correct function execution and errors in high-impact functions that are commonly used in core real-world scenarios. These samples account for about 5% of all samples. Human experts review each of these samples and provide revisions; the approved or revised dialogues are added to the training data after passing the second-pass human validation. Overall, the Human Validation stage only costs \(\sim\)​15 human-hours.

Overall comparison with other strong synthetic function-calling datasets. Appendix 8 provides detailed analyses of quantity, quality, coverage, diversity of the final training data. Compared to other strong synthetic function-calling datasets, our dataset achieves higher quality through richer multi-tool and multi-turn compositions, broader coverage across diverse scenarios, and balanced parameter utilization patterns, while maintaining efficient construction by reusing existing tool schemas and avoiding additional manual designs.

5 Experiments↩︎

5.1 Experimental Setup↩︎

We fine-tune LLMs on the training data produced by our pipeline and evaluate the resulting models in a broad range of settings. Unless otherwise noted, we conduct SFT [28] on Qwen3-8B [13] using our training data and denote the resulting model by GenesisFunc-8B. We compare GenesisFunc-8B against top-tier API-based and open-source foundation LLMs, as well as representative function-calling models trained with specialized function-calling training data including ToolACE-8B (fine-tuning LLaMA3.1-8B-Instruct on ToolACE-generated data), Hammer2.1-7B (based on Qwen 2.5 coder series), and Qwen-ToolRL-8B (fine-tuning Qwen3-8B with ToolRL datasets). Evaluations are conducted on three commonly used benchmarks: BFCL [11], API-Bank [14], and ACEBench [15], with all results averaged over three independent runs. More experimental details, including evaluation metric definitions and settings, are in Appendix 9.

5.2 Main Results↩︎

To comprehensively assess function-calling performance, we compare GenesisFunc-8B with baselines on both in-domain and out-of-domain datasets. The in-domain evaluation measures performance on data aligned with the training distribution, while the out-of-domain evaluation assesses its generalizability to unseen scenarios.

In-Domain Evaluation. Since the tools in pool are drawn directly from BFCL, we treat BFCL as the in-domain benchmark. As shown in Table 9.1.0.1, GenesisFunc-8B achieves 93.31 on the Non-Live split and 83.78 on the Live split, showing strong and robust performance, and outperforming the API-based models. GenesisFunc-8B achieves substantial gains over open-source models of similar scale and on some subsets matches or outperforms much larger models such as Qwen3-32B. Compared with the prior open-source function-calling SOTA ToolACE-8B, our model improves overall accuracy by 2.5% relative on Non-Live and 3.8% relative on Live. Overall, these results indicate that, for function-calling, strong and systematic alignment between training data distribution and real tool semantics can significantly narrow the performance gap between smaller models and much larger ones.

Out-of-Domain Evaluation. To evaluate the generalizability of our fine-tuned model, we conduct experiments on two out-of-domain benchmarks: API-Bank and ACEBench. As shown in Table 2, API-based models maintain a clear advantage over open-source ones, with GPT-4o reaching 85.10 on ACEBench. Open-source models fine-tuned for function calling achieve competitive performance but fall short. Compared with the prior open-source SOTA, our model achieves 64.79 on API-Bank and 78.64 (mean of Normal Overall and Special Overall) on ACEBench, yielding 7.3% and 9.4% relative gains. Notably, GenesisFunc-8B performs on par with GPT-4o-mini and Gemini-1.5-Pro. These results validate that the training data generated by our pipeline effectively enables model generalization to unseen scenarios.

General Abilities. We also find that fine-tuning on the training data generated by our pipeline preserves the model’s general abilities while substantially improving function-calling ability (more details in Appendix 10.1).

5.3 Ablation Study↩︎

Ablation on Multi-Agent Framework. Our multi-agent framework improves function-calling ability through agent collaboration, where the Memory Agent enhances dialogue diversity and the Judge Agent ensures accuracy. We conduct ablation studies by removing either agent and fine-tuning Qwen3-8B with LoRA, noting that the Sample Agent and Function Agent are indispensable to the workflow. As shown in Figure 2, BFCL results indicate that removing the Judge Agent causes a clear performance drop, while removing the Memory Agent leads to significant degradation, highlighting the importance of accuracy and diversity and validating the effectiveness of both agents. We further compare alternative slot selection strategies in the Function Agent in Appendix 10.2.

Impact of the Number of Samples. To assess how the number of generated samples influences function-calling ability, we use the same set of tools and only vary how many dialogues are produced for each tool. Besides the default setting of 5 dialogues per tool, we also construct datasets with 1 and 10 dialogues per tool. We fine-tune Qwen3-8B on these datasets and report BFCL results in Figure 3. Overall accuracy steadily consistently rises with more samples per tool, with a substantial gain from 1 to 5 samples but only modest improvement from 5 to 10, indicating eventually diminishing returns once sufficient scenario diversity is reached.

Ablation on Multi-Stage Evaluation. Detailed results are in Appendix 10.3. We find that models trained on data with the multi-stage evaluation module achieve higher accuracy across all conditions than those trained on non-evaluated data, verifying the effectiveness of this module.

5.4 Scalability and Reinforcement Learning↩︎

Scalable to More Tools. To assess scalability of our pipeline to more downstream tools, we add tools defined in ACEBench into the pool and use our pipeline to generate high-quality training data for fine-tuning. As shown in Table 4, performance on ACEBench improves markedly from 78.64 to 81.87 after adding these tools, due to better tool alignment. Meanwhile, aggregate results on BFCL remain comparable at 87.89 versus 88.55, and on API-Bank at 65.11 versus 64.79, indicating that introducing additional tool definitions does not notably degrade performance on other benchmarks. Overall, our pipeline scales well to broader tool inventories and yields consistent gains on targeted benchmarks with no observable degradations on other datasets, across different model sizes and different backbone architectures, as shown in Appendix 10.4 and Appendix 10.5.

Enhancing Function-Calling Ability in Multi-Turn Dialogues via RL. Inspired by prior works [20], [29], one research question is whether applying reinforcement learning (RL) on our data can further enhance the model’s function-calling capability, particularly in multi-turn dialogue scenarios where performance is less satisfactory. We conduct two sets of experiments: GenesisFunc-8B-RL(all) applies RL instead of SFT to Qwen3-8B using our training data; GenesisFunc-8B-RL(part) focuses on data in multi-turn scenarios, where the model is first SFT-trained on single-turn and special-case data and then RL-trained with multi-turn dialogues. To encourage deeper reasoning during training, we enable the built-in “thinking mode” of Qwen3-8B and augment the training samples with explicit reasoning traces. We use GRPO [30] for RL, with rewards designed around two dimensions: format compliance and functional correctness. More details are in Appendix 11.

Table 3 compares the performance of different training strategies on ACEBench. The model SFT-ed on our data achieves 83.67. Compared with the default SFT setup, GenesisFunc-8B-RL (part) improves Normal tasks substantially from 73.60 to 75.20, with multi-turn performance improved from 65.00 to 70.00, while maintaining 82.88 on Special subset, confirming that targeted RL training enhances long-context reasoning and handling complex interaction, improving generalization on multi-turn tasks while maintaining stability on Special cases. These results highlight the advantage of combining SFT and RL to improve function-calling abilities of the model.

Notably, both SFT and RL incur low training costs, as summarized in Appendix 9.3.

6 Conclusion↩︎

This paper introduces GenesisFunc, an automated data-generation pipeline for strengthening the function-calling abilities of LLMs. Beginning with a reliable tool set curated from open-source benchmarks, GenesisFunc leverages a coordinated multi-agent framework that assists a dialogue generator in producing high-quality, diverse, and representative function-calling training data, while remaining model-agnostic in design. In extensive experiments, models trained with GenesisFunc achieve state-of-the-art performance, marking a concrete advance in tool-augmented AI agents, and the methodology can be extended to more agentic and complex function-calling tasks using proprietary or open-source LLMs in future work.

Limitations↩︎

Despite the effectiveness of GenesisFunc, several important limitations still remain. First, GenesisFunc-8B achieves competitive tool-use performance but still falls short of API-based models like GPT-4 in broader reasoning and comprehension. Enhancing general abilities alongside function-calling remains an open challenge. Second, our training data does not yet fully encompass highly complex multi-turn scenarios that require tightly coupled tool sequences. In future work, we plan to focus more on challenging settings, such as agentic workflows and complex benchmarks. Nevertheless, we firmly believe that, in principle, these more demanding scenarios can also be addressed by extending the methodology developed in this work.

Ethical Considerations↩︎

6.0.0.1 AI Assistant Use

We used LLMs to assist with improving grammar, clarity, and wording in parts of this work. The use of LLMs was limited to language refinement, with all ideas, analyses, and conclusions solely developed by the authors.

7 Details of GenesisFunc↩︎

7.1 Multi-Agent Framework↩︎

7.1.0.1 System Prompts in Multi-Agent Framework

In our proposed multi-agent framework, we employ four distinct agents, Sample Agent, Memory Agent, Function Agent, and Judge Agent, to assist dialogue generation by producing diverse and high-quality conversations. Figure 4 presents the complete system-level prompts that define the roles and behaviors of the Sample Agent, Memory Agent, Function Agent, and Judge Agent. These prompts are used as system prompts throughout the generation process, with bracketed segments serving as dynamically filled placeholders.

7.1.0.2 Details of Sample Agent

To construct a sampled subset \(\mathcal{T}_s \subseteq \mathcal{T}\) from the global Tools Pool, the sample agent employs GPT-4o [16] to evaluate the semantic and functional similarity between tools. Given two tool descriptions that specify their name, purpose, and parameter schema, the model assigns a similarity score \(r \in [0,1]\), where \(r=1\) indicates nearly identical functionality and \(r=0\) indicates entirely unrelated purposes. Each tool pair is scored with a fixed prompt and temperature \(=0\) to ensure deterministic outputs. Tools with \(r>0.75\) or sharing similar functions are grouped as target tools \(\mathcal{T}_{\text{target}} = \{t_1,\ldots,t_m\}\). Distractors \(\mathcal{T}_{\text{dist}} = \{t_{m+1},\ldots,t_{m+n}\}\) are sampled from other clusters according to their relevance scores, categorized as high (\(r>0.6\)), medium (\(0.3<r\le0.6\)), and low (\(r\le0.3\)). All threshold values and sampling ratios were empirically tuned for consistency and reproducibility.

7.1.0.3 Details of the Judge Agent

The Judge Agent selects the best dialogue from multiple candidates based on problem significance and tool appropriateness. Prior literature has documented that LLM-based evaluators can exhibit positional bias in pairwise comparisons, where judgments may systematically favor specific positions within prompts [31]. To mitigate this, candidate dialogues are randomly shuffled before being evaluated by the Judge Agent. Additionally, we perform an A/B swap test on a subset of pairs, where dialogues are evaluated under swapped ordering. We observe no consistent directional preference across swapped conditions, indicating that positional bias is unlikely to be a major confounding factor in our selection process. This analysis enhances the reliability of the Judge Agent’s decisions and helps ensure that the data pipeline’s improvements are not driven by spurious positional effects.

7.1.0.4 Reproducibility

Our Multi-Agent Framework is model-agnostic and does not rely on any proprietary model capabilities or a specific combination of models. In practice, different stages of the pipeline impose different requirements on model behavior. In particular, tool-calling generation benefits from models with stable API-following performance, while other stages primarily require output consistency rather than advanced reasoning ability. In our implementation, Gemini-2.5 Pro [27] is adopted for tool-calling related agents due to its stable behavior, which reflects an engineering choice rather than an algorithmic requirement. Moreover, replacing Gemini-2.5 Pro with open-source models such as Qwen3-32B [13] preserves the overall workflow and produces coherent training data, although data quality and downstream tool-calling performance are moderately reduced. Specifically, on BFCL, the performance decreases by about 0.5% on the non-live split and 0.9% on the live split. These results indicate that stronger models primarily affect data quality, while the pipeline logic and functionality remain unchanged, supporting both reproducibility and practical accessibility.

7.2 Dialogue Generation System↩︎

7.2.0.1 Algorithmic Workflow of Multi-Agent Dialogue Generation

Algorithm [alg] illustrates the pseudocode workflow of our multi-agent dialogue generation system, highlighting the interactions among different agents, the data flow between successive stages, and the overall generation and selection process.

7.2.0.2 Case Study

We leverage a multi-agent framework to enable the dialogue generation system to produce three categories of dialogue: single-turn, multi-turn, and special cases. For both single-turn and multi-turn, we considered situations where users may accomplish either a single task or multiple tasks simultaneously. In addition, we incorporate special cases, such as when none of the available tools can address the user’s request, or when the user’s query prevents the model from filling in the tool parameters.

Figure 9 and Figure 10 illustrate single-turn scenarios in which the user intends to invoke one or multiple tools to complete either a single task or multiple tasks. Figure 11 and Figure 12 present analogous cases in multi-turn dialogues, where one or more tools are required to handle a single task or multiple tasks. Finally, Figure 13 demonstrates the special-case dialogue that we constructed.

7.3 Multi-Stage Evaluation↩︎

7.3.0.1 Rule Checker

Table 5 outlines the check rules we use, which consist of four complementary aspects: tool definition completeness, call format and argument compliance, dialog structure soundness, and consistency between tool outputs and the assistant’s responses.

7.3.0.2 Model Checker

The Model Checker verifies the correctness of function-calling dialogues using the following system-level prompt: “You are a Model Checker responsible for verifying the correctness of a function-calling dialogue. Given the dialogue, tool specifications, and tool-call outputs, determine whether the assistant selected the appropriate tool, used correct parameter formats, and provided a faithful answer. Identify any semantic errors and assign a confidence score between 0 and 1.” This prompt defines the Model Checker’s role and behavior across evaluations. We adopt GPT-4o [16] as the backend model due to its stable evaluative behavior, which improves robustness in automatic screening. This choice is an engineering consideration rather than an algorithmic requirement, and alternative models can be used without affecting the evaluation pipeline.

8 Details of Training Data↩︎

Our constructed function-calling dataset consists of 2,000 single-turn dialogue scenarios, 2,000 multi-turn dialogue scenarios, and 500 special-case dialogues. To further demonstrate the strengths of our dataset in terms of quality, coverage, and diversity, we provide a set of statistical analyses.

8.0.0.1 Relationship Between Training Data and BFCL Evaluation

Our training data uses the tool schemas provided by BFCL. During training, we do not use any BFCL test queries, test outputs, or instantiated evaluation samples. All dialogue are newly generated by our multi-agent framework, including user queries, dialogue contexts, and concrete parameter values. Although the tool schemas are shared between training and evaluation, this setting follows the standard in-domain evaluation protocol of BFCL and prior function-calling benchmarks. The BFCL evaluation focuses on generalization to unseen queries and novel argument combinations rather than memorization of specific tool invocations. As a result, the observed performance gains reflect improved tool selection and parameter grounding instead of exposure to evaluation instances.

8.0.0.2 Quality

In terms of quality, our constructed dataset features an average of 3.11 tools used per dialogue, 4.46 turns per multi-turn scenario, and 3.27 tasks per multi-task case. This design enables each dialogue to cover multiple tools, resulting in a substantially larger number of tool invocation instances than the number of dialogues, and the high number of tool calls, longer dialogue lengths, and richer task compositions expose the model to more complex tool-using situations, thereby better stimulating and enhancing its function-calling capability, while ensuring all data are newly generated and do not overlap with evaluation samples.

8.0.0.3 Coverage

In terms of coverage, our dataset incorporates a broader range of scenarios compared to previously constructed dataset. This expanded scenario coverage is a key factor in strengthening the model’s function-calling ability. An overview of the data statistics used in these representative tool-augmented LLMs is presented in Table 6.

8.0.0.4 Diversity

In terms of diversity, we further analyze the slot filling ratios to better demonstrate the diversity of our tool-call training data. Specifically, we sample 400 tools with concrete parameter values from the constructed dialogues. For each tool, we compute the slot filling ratio by dividing the number of filled optional parameters by the total number of available optional parameters, and then group the results into five equal-width intervals. Figure 5 reports the distribution across these ranges. The relatively balanced counts indicate that our dataset effectively captures heterogeneous slot utilization patterns, ranging from sparse filling to near-complete filling, highlighting its diversity.

8.0.0.5 Efficiency

We examine the efficiency of the data construction process and compare generation cost with other synthetic datasets. Unlike synthetic function-calling datasets such as ToolLLM [2] and ToolACE [8] that rely on manually designed or synthesized APIs, our approach operates directly on the tools already defined in BFCL [11]. This design avoids additional tool construction and manual schema engineering, thereby reducing the cost of dataset creation. The generation cost of our approach arises from model generation within a multi-agent framework, while the tool space remains fixed and reusable across samples. Consequently, the overall construction cost scales linearly with the number of generated samples and does not introduce dataset-specific tool design or manual annotation overhead. Since existing datasets differ in tool complexity, agent design, and model backbones, a direct numerical comparison of generation cost is difficult. Nevertheless, under a unified benchmark setting, our approach achieves a balance between the quality of the training data and construction efficiency.

9 Experimental Details↩︎

9.1 Benchmarks↩︎

9.1.0.1 BFCL

The Berkeley Function-Calling Benchmark (BFCL) [11] is a comprehensive framework for evaluating the function-calling abilities of LLMs across multiple languages, domains, and complex scenarios. It includes 4,951 test cases, with 3,951 single-turn dialogs and 1,000 multi-turn dialogs that emphasize dynamic and realistic settings. The tools pool is broad, combining Non-Live tools curated to cover diverse situations with Live tools that are continuously uploaded by users. BFCL supports two evaluation methods, one based on Python and the other not. In this work, we adopt the Python-based approach, which is shown below:

• Simple Function: This category represents the most basic yet also the most frequently encountered setting, where the input explicitly contains exactly one json function description and the model is expected to correctly invoke precisely that single function.

• Multiple Function: In this setting, the model receives 2 to 4 json function descriptions, but only one of them should be invoked. The task requires the model to identify and select the most suitable function call based on the user’s query and context.

• Parallel Function: Here, the model must invoke multiple function calls simultaneously in response to a single user query. The challenge lies in determining how many calls are required and executing them in parallel, whether the query is phrased as a short request or a longer description.

• Parallel Multiple Function: This combines the complexity of both multiple and parallel function settings. The model is given several function descriptions and must decide, for each one, whether it should be invoked, possibly multiple times or not at all.

For every category, BFCL provides both AST-based evaluation and a corresponding executable evaluation. In the executable setting, Python functions are manually implemented, inspired by publicly available REST API endpoints (such as retrieving weather data) as well as directly computable functions (such as linear regression). The purpose of this executable track is to assess whether the generated function calls can be reliably applied in real-world applications that depend on stable function execution.

9.1.0.2 API-Bank

The API-Bank [14] is composed of 314 dialogues involving a total of 753 API calls, specifically constructed to evaluate the abilities of LLMs in planning, retrieving, and invoking APIs in diverse scenarios. Within the dataset, 363 cases require only a single call, while 122 instances involve multiple calls, reflecting both simple and more complex usage patterns. Performance is systematically measured along three distinct dimensions, providing a comprehensive assessment of tool-use capability:

• Call: Evaluates whether the language model can correctly invoke a known API based on a given query.

• Retrieval+Call: Evaluates the model’s ability to first identify and accurately retrieve the correct API from context and then successfully perform the corresponding call when the API is not provided.

• Plan+Retrieval+Call: Assesses the capacity to plan a sequence of actions, retrieve multiple APIs, and invoke them when the APIs are initially unknown.

The primary evaluation metric for API-Bank is accuracy, formally defined as the ratio of correctly generated predictions to the total number of evaluation attempts.

9.1.0.3 ACEBench

ACEBench [15] is a bilingual Chinese and English benchmark for assessing LLMs tool use ability under realistic conditions. It contains about 2,000 annotated instances spanning a broad API set and organizes evaluation into three tracks:

• Normal: Single/multi-turn cases with similar-API and preference settings; calls are checked via AST matching against gold annotations.

• Special: Inputs with missing, malformed, or otherwise irrelevant parameters are included to thoroughly test the model’s robustness when handling imperfect or noisy instructions.

• Agent: Multi-turn, multi-step interactions in sandboxed scenarios, measuring process correctness and end-to-end task success.

Overall performance is computed from track-wise accuracies, providing fine-grained diagnostics of tool-use failures while avoiding reliance on live APIs or external LLM graders.

9.2 Baselines↩︎

Here we mainly introduce the two open-source models fine-tuned on function-calling data that we use as baselines, while details of the other open-source models and API-based models can be found in their publicly available technical reports.

• ToolACE-8B [8]: Obtained by fine-tuning LLaMA3.1-8B-Instruct with function-calling training data generated through data pipeline ToolACE.

• Qwen-ToolRL: Obtained by fine-tuning Qwen3-8B with the publicly available function-calling training dataset ToolRL [20], which is a hybrid corpus sampling 2K examples from ToolACE and 1K each from Hammer [32] and xLAM [33].

9.3 Compute Cost↩︎

We apply the parameter-efficient LoRA [28] strategy for fine-tuning and perform SFT using the LLaMA-Factory framework [34]. The primary computational overhead during generation arises from API calls, while post-training costs remain lightweight. Specifically, SFT is completed on a single A800 GPU in about 30 minutes, and the RL stage on four A800 GPUs in roughly 4 hours, making the overall compute requirements practical for reproduction. The training data follows the Alpaca format, where the instruction includes the system prompt, tool pool, and user query, and the output contains the tool call. Hyper-parameters are shown in Table 7.

9.4 Statistical Significance and Variance Analysis↩︎

All results reported in the main text are averaged over three runs with different random seeds. We also compute the standard deviation (reported as \(\pm\)SD in the tables) to assess model stability. Across all benchmarks, GenesisFunc-8B shows small standard deviations, typically below 0.5, indicating consistent performance across runs. The observed performance gains of GenesisFunc-8B exceed twice the standard deviations on all benchmarks, demonstrating that the improvements are well beyond random variation. To further verify robustness, we perform paired t-tests between GenesisFunc-8B and the strongest open-source baseline (ToolACE-8B), showing statistically significant gains (p < 0.05).

10 Additional Experiments↩︎

10.1 Study on General Abilities↩︎

To assess the broader impact of GenesisFunc-8B on general abilities, we evaluated it on five benchmarks: MMLU for knowledge [35], EvalPlus for code generation [36], GSM8K for mathematics [37], MGSM for multilingual reasoning [38], and GPQA for logical reasoning [39]. Baselines included LLaMA-3.1-8B-Instruct [40], Qwen3-8B [13], ToolACE-8B [8], and GPT-4 [16]. As shown in Figure 6, GenesisFunc-8B performs on par with Qwen3-8B on most benchmarks, indicating that our fine-tuning substantially improves function-calling ability while preserving the model’s general abilities. It also surpasses similarly sized open-source models on many tasks, demonstrating competitive performance at the 8B scale. The remaining gap to GPT-4 in reasoning and comprehension is expected and likely stems from differences in model size and the breadth of training data rather than negative transfer from function-calling training. Overall, these results highlight the promise of targeted specialization for function-calling while maintaining broad competence, and they leave open the challenge of jointly improving multiple capabilities together with function-calling performance in a single model.

10.2 Different Slot Selection Strategies in the Function Agent↩︎

Table 8 presents the results of different slot selection strategies in the function agent. Compared with two fixed strategies, filling all optional parameters (max) and leaving all optional parameters empty (min), our proposed dynamic slot selection approach achieves the best overall performance on both the Non-Live and Live settings. Specifically, GenesisFunc-8B(dynamic) reaches an overall accuracy of 93.31 on the Non-Live split and 84.83 on the Live split, surpassing both fixed strategies. These results demonstrate that dynamically selecting relevant slots enables the model to better capture diverse tool-use patterns while avoiding unnecessary noise, thereby improving the robustness and generalization of function-calling.

10.3 Ablation on Multi-Stage Evaluation Module↩︎

As discussed earlier, we employ a multi-stage evaluation module to ensure the correctness of the dialogues generated in the previous stage. To evaluate its effectiveness, we fine-tune models on two datasets: one that has passed through this evaluation module and another that has not. The fine-tuned models are then evaluated on the BFCL benchmark, with results presented in Table 9. The comparison clearly shows that models trained on data evaluated by the multi-stage module achieve higher overall accuracy than those trained on non-evaluated data, thereby demonstrating the effectiveness of our proposed evaluation module.

10.4 Scaling to Different Model Sizes↩︎

Scaling laws suggest a close relationship between model capacity and empirical performance. To examine how function-calling ability scales with model size, we evaluate the Qwen-3-xB model family [13], covering a range of parameter scales (4B, 8B, 14B, and 32B). Both raw models and models fine-tuned on our dataset are assessed on the BFCL [11], and the corresponding results are presented in Figure 7. Overall, the results show a steady improvement in function-calling performance as model size increases across both non-live and live evaluation settings, with larger models exhibiting stronger robustness and more consistent execution accuracy. In comparison to raw models, fine-tuned models consistently achieve higher performance at all scales, indicating that supervision from the GenesisFunc effectively enhances function-calling behavior. Importantly, the fine-tuned models maintain a smooth and stable scaling pattern, suggesting that GenesisFunc complements model capacity rather than altering the underlying scaling dynamics. Together, these observations demonstrate the effectiveness of GenesisFunc in supporting scalable performance gains for LLMs.

10.5 Generalization across Model Backbones↩︎

To analyze the impact of backbone choice on function-calling performance, we evaluate a set of representative LLMs, including Qwen3-14B, LLaMA-3-8B-Instruct, and LLaMA-3.1-8B-Instruct. All models are fine-tuned using our dataset and evaluated on the BFCL benchmark. In addition, to mitigate potential confounding effects introduced by different backbones and to more rigorously isolate the contribution of the data-generation pipeline itself, we conduct a controlled comparison on the same backbone. Specifically, since ToolACE is trained on LLaMA-3.1-8B-Instruct, we directly compare the two data-generation pipelines on this backbone. The comparative results are summarized in Figure 8. Figure 8 shows that fine-tuning with GenesisFunc consistently improves performance across all examined backbones under both non-live and live evaluation settings, demonstrating strong cross-backbone robustness. Despite differences in model architectures and pre-training objectives, fine-tuning yields stable and significant gains overall. Notably, models with lower initial function-calling accuracy tend to exhibit larger relative improvements, thereby narrowing the performance gap across backbones. More importantly, the controlled comparison on LLaMA-3.1-8B-Instruct indicates that, under the same backbone, GenesisFunc outperforms ToolACE, providing stronger evidence that the observed gains stem from the data-generation pipeline rather than backbone-specific characteristics.

11 Details of Reinforcement Learning↩︎

11.1 GRPO Algorithm↩︎

To fine-tune the model with structured rewards, we adopt Grouped Relative Policy Optimization (GRPO), a variant of PPO that normalizes advantages within groups of responses derived from the same input query. This design mitigates variance across samples under identical contexts, thereby stabilizing and accelerating training. Specifically, for each query \(Q\), the rollout responses form a group \(G_Q = \{(s_1,r_1), (s_2,r_2), \ldots, (s_n,r_n)\}\), where \(s_i\) denotes a candidate response and \(r_i\) its reward. Each reward is obtained as the sum of correctness and formatting scores relative to the reference annotation. Within each group, the mean \(\mu_Q\) and standard deviation \(\sigma_Q\) of rewards are computed as: \[\mu_Q = \frac{sec:1}{n}\sum_{i=1}^n r_i, \quad \sigma_Q = \sqrt{\frac{sec:1}{n}\sum_{i=1}^n (r_i - \mu_Q)^2},\] and the normalized advantage of response \(s_i\) is defined as: \[A(s_i|Q) = \frac{r_i - \mu_Q}{\sigma_Q + \eta},\] where \(\eta\) is a small constant added for numerical stability.

The policy \(\pi_\theta\) is then updated using the clipped PPO objective with group-normalized advantages: \[r_\theta(s_i|Q)=\tfrac{\pi_\theta(s_i|Q)}{\pi_{\text{old}}(s_i|Q)}.\]

\[\begin{align} J_{\text{GRPO}}(\theta) = \mathbb{E}\big[ \min( & r_\theta(s_i|Q)\,A(s_i|Q), \\ & \operatorname{clip}\big( r_\theta(s_i|Q),\, 1-\epsilon,\, 1+\epsilon \big) \\ & \qquad\qquad \times A(s_i|Q) ) \big]. \end{align}\] Unlike standard PPO, GRPO omits the KL penalty against a reference model, thereby allowing greater flexibility in adapting to diverse custom reward functions while still retaining training stability. This modification improves overall sample efficiency, encourages stronger alignment with structured reward signals, and leads to faster convergence as well as more robust policy performance.

11.2 Reward Design↩︎

Reward functions play a central role in reinforcement learning, guiding models to generate outputs that are both valid and useful. Following prior studies on rule-based reward signals [20], [41]–[43], we adopt a two-dimensional design that combines structural compliance with functional correctness, tailored to the demands of tool-augmented dialogue.

11.2.0.1 Structural Compliance

To ensure that generated outputs conform to the expected schema, we introduce a structural compliance reward. This reward evaluates whether each predicted tool call follows the prescribed format, including the presence of all mandatory fields and the correct logical ordering of elements. For single-tool cases, the reward reduces to a simple binary check defined as \[r_{\text{structural}}^{(i)} = \begin{cases} 1, & \begin{align} &\text{if the i-th output follows} \\ &\text{the required schema}, \end{align} \\ 0, & \text{otherwise}. \end{cases}\] For multi-tool cases, the score is computed for each tool call individually according to the above rule and then averaged across all calls: \[R_{\text{structural}} = \frac{sec:1}{N} \sum_{i=1}^{N} r_{\text{structural}}^{(i)},\] where \(N\) denotes the number of tools in the output.

11.2.0.2 Functional Correctness

In addition to structural validity, we further assess whether the predicted tool calls can successfully achieve the intended functionality. For this purpose, we introduce a functional correctness reward. This reward explicitly captures the degree of semantic and functional agreement between the predicted calls and the reference calls. For a single-tool case, the reward is defined as follows: \[r_{\text{correct}}^{(i)} = \begin{cases} 3, & \begin{align} &\text{if the tool name and all parameters} \\ &\text{match exactly}, \end{align} \\ 2, & \begin{align} &\text{if the tool name is correct and} \\ &\text{at least one parameter matches}, \end{align} \\ 1, & \text{if only the tool name is correct}, \\ 0, & \text{otherwise}. \end{cases}\] For multi-tool cases, the correctness score is first computed for each tool call using the above rule, and then the results are averaged across all calls: \[R_{\text{correct}} = \frac{sec:1}{N} \sum_{i=1}^{N} r_{\text{correct}}^{(i)},\] where \(N\) denotes the number of tools in the output.

Finally, we combine the two components to form the overall reward function. The structural compliance reward ensures that outputs remain syntactically valid, while the functional correctness reward encourages accurate execution of tool calls. By integrating both signals, the model receives feedback that jointly emphasizes format validity and semantic accuracy. The final reward is simply defined as follows: \[R_{\text{final}} = R_{\text{structural}} + R_{\text{correct}}.\] This unified formulation provides a balanced training signal, preventing the model from generating structurally invalid outputs and at the same time guiding it toward functionally reliable tool usage.

References↩︎