1 Introduction↩︎

Modern enterprise AI assistants are built as orchestrated agent systems where an LLM-based planner routes user queries to specialized skills [1], [2]. Routing decisions are driven by comparing the query against each skill’s natural language description. When descriptions are insufficiently discriminative, the planner misroutes queries, causing what we term skill collision: semantically overlapping skills compete for the same query population (Figure 1).

Skill collision is most acute during onboarding. Adding a new skill to a deployed agent forces its description to be precisely positioned relative to all incumbent skills. In practice, this requires iterative manual tuning: a developer writes a description, deploys the skill, observes routing errors from real or synthetic traffic, and edits accordingly. This loop is slow, requires expertise in both the skill’s functionality and the planner’s decision boundaries, and does not scale as agent capabilities grow.

We deploy an automated description optimization pipeline that replaces this manual loop. The pipeline initializes a candidate description with an LLM, then iteratively refines it using false-positive and false-negative cases from a labeled training set. On a production enterprise group chat agent (9 skills, 372 regression cases), the automated descriptions match manually tuned ones in routing quality (79.2% vs. 79.4% average F1, per-skill differences within the \(\pm\)​0.78% multi-seed noise floor) while reducing per-skill engineering effort from 120 minutes to 3.8 minutes (32\(\times\) speedup).

Given that the automated pipeline reaches manual quality, we ask which of its components actually matter. As originally implemented, the pipeline combined several mechanisms motivated by intuition: contrastive feedback presenting FP, FN, and TP cases simultaneously; iterative refinement up to a fixed budget; dual editing of the most-confused rival skill; tuned training set sizes. Through systematic ablation on the production system and at scale on ToolBench [3] (\(\sim\)​16k tools), spanning closed-world (per-query candidate pool) and open-world (retrieval over the full corpus) routing regimes, we find that most of these design choices change final routing F1 by less than 0.5%. A single LLM rewrite using any available FP/FN cases captures the bulk of available improvement: on production, single-shot achieves 79.2% F1 (per-skill -0.12% from human-written descriptions, matching iterative refinement within noise); on ToolBench open-world, single-shot achieves a +4.45% F1 gain, within 0.2% of iterative refinement.

Not all routing failures are description failures. Two further findings characterize where description optimization applies, and where it does not. First, closed-world (fixed candidate pool) and open-world (retrieval-based) routing require separate optimization: descriptions tuned in one regime transfer poorly to the other, and LLM initialization helps in one setting but hurts the other. Second, when two skills’ intended scopes genuinely overlap (as opposed to merely sharing under-discriminative wording), no description rewrite can resolve the collision; these skills exhibit large training-validation F1 gaps regardless of optimization signal quality. We identify this gap as a diagnostic signal that flags skills for architectural intervention (scope restructuring, intent-specific routing rules) rather than continued description refinement.

Taken together, these findings yield a compact operational picture for skill onboarding: optimize with a single-shot LLM rewrite, match the optimization regime to the deployment regime, and use the train-validation F1 gap to triage which skills need architectural intervention instead of more description tuning. We make three contributions: (1) a deployed production system that replaces manual description tuning at enterprise scale, validated as non-inferior to manually tuned baselines with a 32\(\times\) engineering speedup; (2) systematic ablation across production and ToolBench showing that pipeline complexity—feedback type, iteration budget, dual editing, training size—has negligible impact, and a single LLM rewrite suffices; (3) characterization of where description optimization applies, including a diagnostic signal for skills requiring architectural intervention and the separation of closed-world and open-world routing into distinct optimization regimes.

2 Related Work↩︎

Tool selection for LLM agents. LLM tool selection encompasses routing models and planners over large API collections [3]–[5], generating accurate calls from documentation [6], and retrieving candidate tools for orchestrators [7]–[9]. These upstream retrieval methods complement our work; we optimize the descriptions used within the established candidate pools.

Tool description optimization. Recent methods use LLMs and execution traces to rewrite tool documentation, aiming to improve downstream task completion [10]–[13]. Instead of targeting downstream execution success via execution traces, we target upstream routing decisions using objective false positive and negative cases to diagnose skill collisions. Furthermore, we rigorously evaluate against manually tuned descriptions in a production deployment and demonstrate that a single LLM rewrite captures the vast majority of available improvements.

Prompt optimization & self-refinement Numerous frameworks optimize overarching task prompts or refine outputs using textual gradients, demonstrations, or LLM self-criticism [14]–[22]. Instead of optimizing aggregate task prompts, we focus exclusively on per-skill descriptions evaluated via discrete routing errors. Furthermore, while many of these methods rely on multi-step evolutionary search or iterative closed-loop refinement, we demonstrate that a single-shot rewrite using objective routing feedback captures the vast majority of available improvements for tool description optimization.

3 Method↩︎

The skill onboarding pipeline takes as input a skill name and outputs an optimized routing description for use by the LLM planner. The pipeline has two stages: LLM initialization and error-feedback refinement. We additionally describe a single-shot variant which section 5.2 shows is sufficient to match the full iterative pipeline.

Stage 1: LLM initialization. We prompt an LLM with the skill name to generate a candidate routing description. The initialization prompt requests a description of what the skill does. The initialized description serves as both the starting point for refinement and a standalone baseline for evaluation.

Stage 2: Error-feedback refinement. The initialized description is evaluated on a labeled training set of queries with ground-truth skill labels. We collect false positives (FP: queries routed to the skill that should not be), false negatives (FN: queries the skill misses), and true positives (TP: correctly routed queries). At each iteration, we prompt the LLM with the current description, up to 5 FP and 5 FN cases (a practical token-budget choice; §5.2 confirms performance is insensitive to this limit), and a TP set matched to the number of negative cases (to balance the prompt context across positive and negative examples). The LLM is asked to identify routing failure patterns and revise the description. The refined description replaces the current one, and the process repeats up to a fixed iteration budget or until per-skill training F1 exceeds a \(90\%\) threshold (in practice, the iteration budget is the effective stopping criterion; see §5.2). At each iteration \(t\), the routing evaluation produces error cases \(\mathcal{E}_t = (\text{FP}_t, \text{FN}_t, \text{TP}_t)\); the description with the highest training F1 across iterations is selected. Algorithm 2 summarizes the loop.

Single-shot variant. We additionally evaluate a single-shot configuration where the LLM is given the initialized description and all available FP and FN cases in a single prompt, producing one revised description without further iteration. This simulates the approach a developer would take with a complete training set, and §5.2 shows it is sufficient to match iterative refinement.

Open-world adaptation. In the retrieval setting, all tools are indexed by description and candidates are retrieved per query via hybrid sparse-dense retrieval (BM25 [23] combined with text-embedding-ada-002 [24] cosine similarity, with per-query min-max normalization). The refinement loop runs identically, but FP cases now correspond to tools retrieved into the top-20 candidate pool that should not be invoked, providing a targeted disambiguation signal that reflects retrieval errors. Only tools that appear in at least one FP query’s retrieved pool are eligible for refinement.

4 Experimental Setup↩︎

Production setting. The production system is an enterprise group chat agent whose LLM-based planner routes queries to one of 9 skills spanning people search, web search, calendar scheduling, internal knowledge retrieval, organizational hierarchy lookup, email, and document generation. We use 372 synthetic test cases with ground-truth skill labels, created by product and engineering experts. Each query may target one or more skills; all targeted skills are labeled as positives. Per-skill training queries are synthesized separately from the 372-case test set; positive examples range from 10 to 119 per skill depending on skill scope, with negative examples sampled from queries targeting other skills to ensure label balance. We compare four conditions: HUMAN (currently deployed manually tuned descriptions, our primary comparison baseline), OLD (skill disabled, system-level pre-deployment baseline), INIT (LLM-initialized description, no error feedback), and Iter (optimized description after refinement). Training set is the full set of available examples per skill (referred to as trainmax); a smaller train20 variant is reported in Appendix 9.

ToolBench setting. ToolBench [3] covers \(\sim\)​16k RESTful API tools across I1, I2, I3 splits. I2 is the primary split for skill collision evaluation, because it requests multiple tools from the same API category to fulfill the query. ToolBench descriptions are developer-facing API stubs (e.g., “Search Book by its name”, “This endpoint will return back all news about Climate Change from all over the world”) rather than routing instructions, which makes description quality directly measurable but also inflates the absolute gain numbers relative to systems with already-curated descriptions. We frame the task as single-turn tool routing: given a query and a candidate pool, predict the correct tool, with no API execution. We treat the first non-terminal tool call from the official ToolBench trajectories as ground truth; the original trajectories are LLM-generated and we did not manually verify each label. We study two candidate pool regimes. Closed-world uses the per-query pre-defined available_tools list (3 to 15 tools) from the ToolBench dataset as the full candidate pool. Open-world retrieves the top-20 candidates per query from the full \(\sim\)​16k corpus via hybrid retrieval. For each split, we select the top-100 tools by positive sample count (requiring at least 25 positive and 25 negative queries); these subsets are fixed across all ablations. Per tool, we allocate 20 positive and 20 negative queries as the training set and hold out a balanced validation set of up to 50 positive and 50 negative queries, identical across all training-size ablation runs so that comparisons reflect only the change in training signal. We track three description states: ORIG (iteration 0, original placeholder), INIT (iteration 1, LLM-generated with no error feedback), and Iter (best description across up to 10 refinement iterations).

Ablation dimensions. On ToolBench we ablate: error signal type (FP+FN+TP / FP-only / FN-only / FP+FN without TP), iteration budget (\(\{5, 10, 30\}\)), per-iteration sampling (5 cases per category vs.all available), training size (\(\{5, 10, 20\}\) examples per class), dual editing (simultaneously refining the rival tool’s description), and three BM25 to dense retrieval weight combinations \((0.2, 0.8)\), \((0.5, 0.5)\), \((0.8, 0.2)\). Default settings unless specified are train\(=\)​20, max iterations\(=\)​10, BM25\(=\)​0.2, EMB\(=\)​0.8.

5 Results↩︎

5.1 Automated optimization matches manual tuning at production scale↩︎

Table 1 reports per-skill HUMAN, INIT, single-shot (SS), and iterative (Iter) F1 on the production agent. The average \(\Delta\)(HUMAN\(\to\)Iter) across the 9 skills is \(-0.20\%\) and \(\Delta\)(HUMAN\(\to\)SS) is \(-0.12\%\). To characterize run-to-run variance of the live regression API, we ran 3 independent seeds of the trainmax pipeline; the average per-skill standard deviation of \(\Delta\)F1 is \(\pm 0.78\%\), with 8 of 9 skills below \(\pm 1\%\) std (Appendix 11). Both averages are well below this per-skill noise floor; the largest per-skill differences are \(\pm 1.7\%\) for Iter and \(\pm 2.5\%\) for SS (Table 1). Per-skill outcomes split nearly evenly under both methods: 5 skills favor Iter and 4 favor manual tuning; SS splits 4 vs.. We interpret this as both single-shot and iterative automated optimization being non-inferior to manual tuning at production scale, but not exceeding it.

The value of automated onboarding is therefore operational. An applied scientist manually tuned descriptions for one skill using the standard process (write, regression test, observe failures, edit), recorded wall-clock time, and compared against the automated pipeline. Manual tuning required 120 minutes for this skill, whereas the automated pipeline produced comparable descriptions in 3.8 minutes, which is a 32× speedup.

Notably, the single-shot variant achieves this match without iteration or any of the additional pipeline mechanisms. The next section examines which components of the pipeline actually contribute to this result.

5.2 A single LLM rewrite captures most of the improvement↩︎

Table 2 summarizes ablations of five design choices originally motivating the elaborate pipeline: feedback signal composition, iteration budget, dual editing of confused tool pairs, per-iteration case sampling limit, and training set size. All five vary final F1 within \(0.5\%\) on ToolBench I2 open-world. Specifically: feedback signal type (FP+FN+TP vs.FP-only vs.FN-only vs.FP+FN without TP) yields gains within \(0.1\%\) of each other; dual editing provides negligible benefit; sampling 5 cases per category per iteration achieves similar gain as passing all available cases; training size effects (5 vs. vs. examples per tool) are within \(0.5\%\), with train=10 marginally best. The implication is that the per-example error feedback signal is the dominant driver of optimization quality. How that signal is packaged into the LLM rewrite prompt is has less effect. Detailed per-variant numbers are in Appendix 7.

Figure 3 shows the iteration budget sweep. On ToolBench open-world, gains increase monotonically with iteration budget: \(+4.24\%\) at max-5, \(+4.65\%\) at max-10, \(+5.07\%\) at max-30. Approximately \(90\%\) of tools hit the iteration cap without reaching the \(90\%\) stopping criterion at max-10, indicating optimization is still progressing when truncated. However, single-shot rewriting (one LLM call given all available training FP and FN cases) achieves \(+4.45\%\), matching max-10 within \(0.2\%\) and exceeding max-5. Iterative refinement with sampled feedback adds little when single-shot has access to the full training signal. The production pattern reported in Section 5.1 confirms this finding at production skill.

For practitioners trading compute against performance, single-shot is the dominant choice across both settings. Long iteration budgets capture marginal additional gains in open-world ToolBench but the additional cost is rarely justified.

5.3 Optimization regimes diverge between retrieval-based and fixed-pool routing↩︎

Closed-world baseline F1 is high (\(\sim\) \(91\%\) on I2) and gains from optimization are correspondingly modest (\(0.6\)-\(0.9\%\) across I1, I2, I3, Appendix 14). Open-world baseline F1 is lower (\(\sim\) \(70\%\) on I2) and gains are substantially larger (\(+4.5\%\), Appendix 8). More striking than the magnitude difference is a directional one: LLM-initialized descriptions hurt closed-world F1 by \(0.8\)-\(1.4\%\) comparing to the original placeholder text, while improving open-world F1 by \(1.5\)-\(2.1\%\).

The two regimes require separate optimization. We re-evaluated closed-world-optimized descriptions in the open-world setting without further refinement: gains are modest (\(0.4\)-\(1.0\%\)), well below in-setting open-world optimization (\(3.7\)-\(4.5\%\)). Descriptions tuned against a fixed candidate pool do not generalize to the dynamic retrieved pool: closed-world optimization improves routing precision among a small known set of competitors, while open-world optimization additionally improves retrieval recall.

Description optimization is qualitatively different in retrieval-based vs.fixed-pool routing. Practitioners should determine which regime their deployment matches and optimize within that regime: LLM initialization is a useful starting point for retrieval-based routing but should be avoided when the candidate pool is fixed and small.

5.4 Initial training F1 as a diagnostic signal↩︎

Figure 4 examines whether iter-0 training F1 (the routing F1 of the original placeholder description on the training set, before any optimization) predicts the value of optimization on held-out data. On ToolBench open-world, tools with iter-0 training F1 below \(65\%\) (\(n=33\)) achieve an average held-out gain of \(+6.27\%\), approximately \(10\) times the gain of tools with iter-0 training F1 at or above \(65\%\) (\(n=15\), \(+0.63\%\); t-test \(p < 0.001\)). The pattern is robust to threshold choice (Appendix 13), and all tools with validation gains above \(10\%\) fall in the low iter-0 region. Iter-0 training F1 is a useful prioritization signal: tools below \(65\%\) are the candidates where optimization adds substantial value.

On ToolBench, most training improvements transfer to held-out performance. Per-tool training and validation F1 gains on ToolBench are positively correlated (Spearman \(\rho = 0.66\)); only 2 of 48 tools (\(4\%\)) exhibit an overfitting signature where training gains substantially exceed validation gains (Appendix 15).

A subset of skills cannot be resolved by description optimization alone. On the production agent, two skills (IntKnowledge and WorkbackPlan) follow this pattern: low iter-0 training F1, large training gains, but failure to exceed the HUMAN baseline on the held-out test (Table 1; multi-seed confirmation in Appendix 11). Both skills carry inherent semantic overlap with other skills in the deployment: IntKnowledge retrieves person-related organizational data, overlapping PeopleSearch; WorkbackPlan produces structured plans, overlapping StatusReport in surface form. Description text alone cannot disambiguate skills whose intended scopes overlap, regardless of training signal quality. Practitioners encountering a skill with low iter-0 training F1 and a large \(\Delta\)train versus \(\Delta\)val gap should consider architectural intervention (e.g., intent-specific routing rules, restructured skill scopes) rather than description refinement.

6 Conclusion↩︎

We deployed an automated description optimization pipeline on a production enterprise group chat agent and validated findings at scale on ToolBench. Systematic ablation showed most of the originally designed mechanisms are unnecessary; a single-pass LLM rewrite given any FP/FN cases captures the bulk of available improvement and is non-inferior to manually tuned descriptions at production scale.

Limitations↩︎

We acknowledge several limitations. Our production system has 9 skills and 372 synthetic test cases, limiting statistical power for per-skill claims. The ToolBench original descriptions are placeholder text, which inflates the absolute gain numbers compared to systems with already-curated descriptions; the system’s value in such settings remains to be characterized. Open-world ablations are conducted on I2 only; I1 and I3 open-world experiments are left for future work. The semantic overfitting observation in production is based on 2 of 9 skills; while ToolBench data falsifies the original “low iter-0 predicts failure” hypothesis at scale, the production-specific pattern in low-data settings warrants validation on additional production deployments.

7 Ablation Tables↩︎

We ablate on the error signal type, iterative refinement, dual editing, and training size in Tables 3, 4, 5, 6, 7, and 8.

8 Open-World Detail (ToolBench)↩︎

Table 9 reports the full open-world results referenced in §5.3. Transfer re-evaluates closed-world-optimized descriptions in the open-world retrieval setting without further refinement. In-setting runs the full optimization loop using the retrieved candidate pool. Embedding-heavy retrieval (BM25=0.2, EMB=0.8) consistently outperforms BM25-heavy across both transfer and in-setting; BM25 fails to differentiate tools with generic placeholder names. \(n\) is the tools with at least one retrieval false positive; transfer evaluates all 100 tools (no FP requirement), while in-setting requires at least one FP per tool to be optimizable.

9 Production train20 Results↩︎

Trainmax (used in the main paper, §5.1) uses all available positive examples per skill (10 to 119, varying by skill). For comparison, train20 caps the training set at 20 positive plus 20 negative examples per skill (or fewer when the skill has fewer positives available; SendEmail, generate_status_report, WorkbackPlan fall back to all available data). Table 10 reports per-skill F1 for the train20-standard run. On average, train20 yields \(\Delta\)(OLD\(\to\)Iter) \(= +1.71\%\) versus trainmax-standard’s \(+3.65\%\), indicating that more training data helps in aggregate but not uniformly: 5 skills are within \(1\%\) of their trainmax result, while GetUserManager regresses sharply at train20 (an outlier we attribute to API non-determinism, since the multi-seed trainmax results show GetUserManager is otherwise stable; Appendix 11).

10 OLD Baseline (Production Agent)↩︎

OLD measures the system-level F1 with the new skill disabled, evaluated on the subset of test cases that do not target the skill. Because OLD uses a different test subset than HUMAN/INIT/Iter (which evaluate on the full 372 cases with the skill enabled), OLD\(\to\)Iter gains conflate skill availability with description quality and are not directly comparable to HUMAN\(\to\)Iter. We report OLD here for completeness; §5.1 uses HUMAN\(\to\)Iter as the primary comparison.

11 Multi-Seed Variance (Production Agent)↩︎

To characterize run-to-run variance, we ran 3 independent seeds of the trainmax-standard pipeline. Each seed includes a fresh optimization run (LLM calls at temperature \(> 0\)) and a fresh regression evaluation (live API calls at temperature \(> 0\)). The HUMAN baseline (\(79.4\%\)) was evaluated in a single regression run; we did not characterize HUMAN-specific run variance. The reported per-skill noise floor of \(\Delta\)F1 (\(\pm 0.78\%\) std on average; Table 12) bounds Iter variance directly; \(\Delta\)(HUMAN\(\to\)Iter) variance includes additional HUMAN evaluation noise we did not measure. The mean \(\Delta\)(OLD\(\to\)Iter) is \(+3.27 \pm 0.78\%\) across the 3 seeds, with 8 of 9 skills below \(\pm 1\%\) std. IntKnowledge regresses against OLD in all 3 seeds (\(-1.90\%\), \(-3.72\%\), \(-2.04\%\)), confirming this is a robust failure rather than a single-run artifact.

12 Production Training F1 Dynamics↩︎

Table 13 reports per-skill training-set F1 at iteration 0 (the initial LLM-generated description) and the best across iterations on the trainmax-standard production setting. Two skills exhibit large training F1 gains: IntKnowledge (\(+30.8\)pp) and WorkbackPlan (\(+28.1\)pp). Both also start with low iter-0 training F1 (\(\leq 67\%\)), and neither translates the training gain into a held-out F1 improvement above the HUMAN baseline (Table 1). The other 7 skills show no training F1 movement: each begins at or above the \(90\%\) stop criterion and the optimization loop terminates at iter-0. The pattern matches the small-data overfitting characterization in §5.4.

13 Iter-0 Training F1 as a Failure Predictor (ToolBench)↩︎

We test whether the production observation in §5.4, that low iter-0 training F1 predicts failure, generalizes to the 48 ToolBench open-world tools (I2, BM25=0.2/EMB=0.8, train_size=20, max10). For each tool, we extract iter-0 training F1 (train_f1@0) and the held-out validation gain \(\Delta\)val from the existing optimization logs; no new LLM calls are required.

The 65% threshold derived from production is highly significant on ToolBench (Welch’s \(t\), \(p = 0.0008\), \(N=48\)): tools below 65% iter-0 training F1 gain \(+6.3\%\) on validation on average, vs.\(+0.6\%\) for tools above (Table 14). However, the direction is opposite to the production failure mode: at ToolBench scale, low iter-0 F1 predicts larger held-out gains, not failure to generalize. Spearman correlation between \(\Delta\)train and \(\Delta\)val is \(+0.66\) (\(p < 10^{-4}\)): in the open-world retrieval setting, large training gains generalize. We interpret the production overfitting (IntKnowledge, WorkbackPlan) as a small-data-regime artifact of low-positive-sample skills (10–119 positives), not a general property of the pipeline.

14 Closed-World Results↩︎

Table 15 shows validation F1 in the closed-world setting, where the candidate pool is the per-query available_tools list (3–15 tools) rather than the full retrieved corpus. A consistent pattern emerges across all splits: LLM initialization (init) reduces F1 by \(0.8\)–\(1.4\%\) relative to the original placeholder descriptions (orig), while iterative refinement recovers and surpasses the original. Despite being completely uninformative text, placeholder descriptions perform comparably to zero-shot LLM-generated ones in this setting; the gains come entirely from error feedback, not initialization. This contrasts with the open-world setting (§5.3) where initialization helps, suggesting description quality matters more when retrieval recall is the bottleneck.

On I2, the training size sweet spot is 10 examples per class (\(+1.0\%\)); additional examples do not consistently improve the refinement signal. Iterative refinement (\(+0.9\%\)) modestly outperforms single-shot (\(+0.6\%\)), consistent with per-example signal being weaker in the fixed small candidate pool.

15 Training versus validation gain at ToolBench scale↩︎

Figure 5 plots per-tool training F1 gain against validation F1 gain for the 48 tools optimized in the open-world setting. Training and validation gains are positively correlated (Spearman \(\rho = 0.66\)); 2 tools (\(4\%\)) exhibit an overfitting signature (\(\Delta\)train \(> 20\%\), \(\Delta\)val \(< 5\%\)).

References↩︎