PORTS: Preference-Optimized Retrievers for Tool Selection
with Large Language Models
July 03, 2026
Abstract
Integrating external tools with Large Language Models (LLMs) has emerged as a promising paradigm for accomplishing complex tasks. Since LLMs still struggle to effectively manage large tool collections, researchers have begun exploring retrieval-based
methods to pre-select the most relevant options, addressing input length and latency constraints. However, existing retrievers are often misaligned with tool-calling LLMs due to their separate training processes. This paper presents PORTS, a
novel odds ratio preference optimization method for training retrievers aimed at tool selection. Using a perplexity-inspired preference signal from a frozen LLM, our approach fine-tunes a retriever to find helpful tools by optimizing the correlation
between the selection probabilities and the downstream performances while jointly enforcing a contrastive semantic loss between documentation strings. The versatility of PORTS and its ability to significantly improve tool selection accuracy
are demonstrated through extensive experiments on six datasets, two encoder models, and three LLMs with diverse prior knowledge. With low computational demands, our alignment process facilitates generalization to new queries and tools, proving valuable for
practical applications with evolving toolsets.1
1 Introduction↩︎
“The right tool for the right job.”—Proverb
Equipping Large Language Models (LLMs) with the capability to dynamically interact with external tools2 has garnered significant research attention. This integration not only improves the problem-solving potential of LLMs, but also dramatically expands their functional scope [2], [3]. When presented with a user query, tool-augmented LLMs can determine when and how to utilize specific tools to generate more accurate and informative responses. For example, tools can enable LLMs to use a calculator, set calendar events, and access real-time weather information. As the field continues to evolve, LLMs with tools are expected to play a pivotal role in shaping the future of Natural Language Processing (NLP) [1].
Fine-tuning LLMs with tool usage examples is expensive and confines the acquired knowledge to a predefined set of tools [4], [5]. The in-context learning paradigm alleviates these issues, but the limitations in input length and noise for lengthy prompts make it impractical to manage many descriptions or demonstrations directly [6], [7], introducing efficiency and accuracy challenges in tool-selection tasks, mainly when precise parameter specification and schema typing are paramount. Furthermore, when faced with hundreds of tool docstrings in the prompt, the language model alone struggles to identify the most suitable one, often resulting in suboptimal performance [1], increased computational needs, and high costs. Recently, the focus has shifted towards the use of retrievers to effectively support LLMs in the tool selection process [8]–[10]. Retrieval-enhanced pipelines filter the top-\(K\) most suitable tools for a given query, aiming to reduce noise and enhance the LLM’s ability to select the right tool and configure the necessary parameters for calling.
Although several publications have explored retrievers for tool selection, optimization of the retrieval component itself has received little consideration. Clustering approaches have proven successful in multi-dimensional representation spaces [11], but current methods predominantly employ non-parametric indexing techniques [12] or standard encoder models trained with supervised signals [8]. Whereas these methodologies can be effective in isolation, they often falter when integrated into a broader pipeline, primarily due to misalignment between the training criteria used for the retrieval and generation modules. A significant challenge arises when tools with descriptions similar to the user query are ultimately irrelevant or potentially misleading for the LLM [13], [14]. Furthermore, tools can exhibit subtle differences, such as variations in the names, numbers, and types of input parameters, which complicate effective selection–an issue that is increasingly prevalent given the rapid proliferation of publicly available tools and Model Context Protocol servers. In such scenarios, conditioning the encoder on the LLM output may provide additional training signals that benefit the selection process. However, most existing retriever adaptation techniques require the LLM to be trained from scratch [15]–[17], which can be prohibitively costly or unfeasible with closed-source solutions characterized by no access to internal representations. Recent research has explored an alternative approach: training encoder models with the support of LLMs, using them as supervision signals to better align representations with task-specific objectives [18]. We focus on this emerging class of methods, investigating how LLMs can be effectively leveraged and optimized to guide encoder alignment through feedback incorporation for tool retrieval.
In this paper, we propose a new method to train preference-optimized retrievers for tool selection (PORTS), aligning them with the needs of the LLM responsible for tool usage. Our training scheme adapts a pre-trained encoder model with
supervision signals from a black-box LLM, preferring retrieving tool docstrings that stimulate the downstream LLM to use the right tool. Introducing a novel contrastive preference loss enables a more accurate selection process in application domains where
multiple tools might adequately serve the task and yield coherent yet inaccurate results. We conduct experiments on six public datasets. The results are analyzed with various classes of encoders and LLMs. To gauge generalizability, tests are carried out
with in-domain and out-of-domain tools. We conclude that PORTS can effectively increase the tool selection performance of the baseline retriever, with low computational overhead. Figure 1 shows the averaged
metric gains of our alignment method in retrieving tools seen during training.
2 Related Work↩︎
2.0.0.1 Tool Learning
Recent studies in language modeling have explored the use of non-differentiable tools to supplement the knowledge stored in the model weights, offloading tasks to external modules. They broadly fall into two categories. Tuning-based methods train models to use one or a few tools in specific domains. Example works include TALM [19], Toolformer [20], ToolLLaMA [8], Gorilla [12], ToolkenGPT [21], and Granite [22]. These models are trained on datasets where the input text is augmented with tool calls. During inference, when such invocations are identified, the decoding process is paused, the corresponding tool is executed, and the result is incorporated before resuming text generation. Specialized tool-calling models mostly rely on synthetic instruction-tuning data generated from proprietary models such as GPT-4 [23]. Among the few exceptions intended for commercial applications is the NexusRaven series [24]. However, LLM fine-tuning is only applicable to open-source models and is generally hindered by expensive data collection and computing infrastructure, as well as poor flexibility in accommodating emergent or updated tools. Conversely, tuning-free methods are compatible with all LLMs and capitalize on in-context learning abilities, showing tool descriptions and demonstrations directly in the prompt [25]–[27]. As the tool arsenal grows (e.g., 16.000+ [8]), a retriever becomes essential. Retrieval-based and fine-tuning methods can be combined to achieve better performance [9].
2.0.0.2 Tool Retrieval
Merging Retrieval-Augmented Generation (RAG) and tool calling enables LLMs to evaluate a small subset of retrieved tools and select the most suitable for response formulation. Tool retrieval approaches can be classified into two main types: term-based and semantic-based. Term-based techniques, exemplified by TF-IDF [28] and BM25 [29], rely on exact term matching and utilize sparse representations for both tool docstrings and queries. For instance, Gorilla [12] implements tool retrieval by combining BM25 with GPT-Index. In contrast, semantic-based methods utilize neural networks to learn the relationships between queries and tool descriptions. CRAFT [30], for example, instructs LLMs to generate fictitious tool descriptions conditioned on the input queries, then uses pre-trained SimCSE for similarity computation. Few studies focus on training the retriever itself, while iterative refinement with cosine similarity has proven effective for adapting representations [31]–[33]. TPTU [34], ToolLLaMA [8] and Confucius [9] fine-tune a SentenceBERT model using contrastive learning objectives. ProTIP [10] fine-tunes BERT-base with a contrastive loss optimized for progressive tool retrieval. COLT [7] models collaborative relationships among multiple tools using graphs and implements tool retrieval through cross-view graph contrastive learning. ToolRerank [35] addresses the re-ranking stage of tool retrieval, proposing an adaptive and hierarchy-aware method. However, these studies do not consider LLM preferences in specializing the retriever.
3 Methodology↩︎
In this section, we introduce PORTS and elaborate on its theoretical motivations, design, and training losses. Figure 2 illustrates our architecture.
3.1 Preliminary↩︎
RAG is a widely used framework for augmenting LLMs with external knowledge sources. While models like RETRO [36] and FiD [37] achieve significant improvements through separate training of retrieval and generation components, end-to-end training approaches offer potential for enhanced relevance, coherence, and contextual awareness [38], [39]. However, end-to-end training poses challenges, including high computational demands, complex data handling, and the difficulty of maintaining dynamic search indexes with accurate, up-to-date embeddings. Techniques like batch negative sampling [40] have improved efficiency, but require careful selection of diverse negatives. Adapting RAG systems to new domains often necessitates simultaneous retraining of both the retriever and generator, underscoring the importance of efficient data management [41]. To address these challenges, methods such as RePlug LSR [42] use frozen language models as references to optimize retrieval without costly full-model fine-tuning. We argue that reducing output uncertainties alone is insufficient for aligning with users’ goals when selecting tools for tasks. Retrieval models must effectively navigate reference corpora by distinguishing between semantically similar but irrelevant options. Inspired by techniques from LLM fine-tuning, such as RLHF [43], DPO [44], and ORPO [45], retrieval models can leverage comparative loss to prioritize relevant information, mirroring approaches like triplet learning.
3.2 Task Definition↩︎
Given an input query \(q\), we aim to pair it with a candidate tool \(t_i\) from a predefined set \(\mathbf{T} = \{t_1, t_2, \ldots, t_{\lvert \mathbf{T}
\rvert}\}\) by maximizing their semantic alignment. PORTS is trained to prioritize the retrieval of tools’ docstrings \(d_{t_i}\) that most enhance tool calling accuracy when prompted to the LLM.
3.3 PORTS↩︎
Our approach optimizes a retrieval model \(\mathcal{R}\) through a dual training strategy accounting for query-docstring semantic similarity and tool support in correct answer prediction. We shape probabilities over available data by enforcing preferences on the top-\(K\) tools. An LLM serves as an indirect ranking agent, aligning selections with tool usage patterns, resulting in a context-aware algorithm tailored to downstream tasks.
3.3.0.1 Goal-Directed Retrieval
We formalize the retrieval process using an encoder \(\mathcal{E}\) and a generative LLM \(\mathcal{G}\). Each instance in the dataset \(\mathcal{D}\)
comprises a user query \(q\), the target tool call \(y\), the tool required \(t\) to solve the request (positive), and a set of \(n\) tools irrelevant to the task (negatives). Each tool \(t_i\) is associated with a description \(d_{t_i}\), which includes its general characteristics,
objectives, and parameters. PORTS first computes the alignment between \(q\) and the tool docstrings \(d_t\) using a cosine similarity function \(sim(q,d_t) = \mathcal{E}_\theta(q) \cdot \mathcal{E}_\theta(d_t)\), where \(\mathcal{E}: \mathbb{Z}^l \rightarrow \mathbb{R}^d\) denotes the retriever’s encoding function that maps input
sequences of \(l\) tokens to a \(d\)-dimensional vector space, parameterized by weights \(\theta\). These similarities are then normalized and converted into
retrieval probabilities using a softmax function with scaling factor \(\gamma\): \[\label{eq:retr-prob}
P_\mathcal{R}^\theta(t \vert q, d_t) = \frac{\exp \left ( \frac{sim(q,d_t)}{\gamma} \right )}{\sum\limits_{t_i \in \mathcal{T}} \exp \left ( \frac{sim(q,d_{t_i})}{\gamma} \right )}\tag{1}\] The retrieval distribution over the corpus of tools
in Eq. 1 is approximated by marginalizing over restricted triplet sets \(\mathcal{T} = (t_i^{+}, t_{i,1}^{-},\dots, t_{i,n}^{-}) \subseteq \mathbf{T}\) for efficiency. As in RePlug, we prompt each retrieved tool docstring independently with the query and then conduct \(K\) separate inferences. This enables a direct and noiseless correlation between tool selection
quality and confidence in the generated output. The output probability distribution \(Q_\mathcal{G}\) of the LLM, reflecting confidence in the final prediction, is computed as in Eq. 2 . It
applies a softmax function, parameterized by temperature \(\beta\), to the average log-likelihood \(P_\mathcal{G}\) of producing the correct tool call \(y\).
\[\label{eq:q95ppl} \begin{align} &Q_\mathcal{G}(t \vert q,d_t,y) = \frac{\exp \left ( \frac{P_\mathcal{G}(y \vert q,d_t)}{\beta} \right )}{\sum\limits_{t_i \in \mathcal{T}} \exp \left ( \frac{P_\mathcal{G}(y \vert q,d_{t_i})}{\beta} \right )} \\ & P_\mathcal{G}(y\vert q,d_t) = \frac{1}{l} \log \prod_{i=1}^l P_\mathcal{G}(y_i \vert q,d_t,y_{<i}) \end{align}\tag{2}\] The retriever is trained by optimizing the Kullback-Leibler divergence between the \(Q_\mathcal{G}(t \vert q,y)\) and \(P_\mathcal{R}^\theta(t \vert q)\) distributions: \[\begin{align} \label{eq:loss95replug} \mathcal{L}_\text{replug} = \mathop{\mathrm{\mathbb{E}}}\limits_{\mathcal{T},q,y \sim \mathcal{D}} \sum_{t_i \in \mathcal{T}} \mathrm{\small KL} \left( P_\mathcal{R}^\theta(t_i \vert q, d_{t_i}) \; \Vert \right. \\ \left. Q_\mathcal{G} (t_i\vert q,d_{t_i},y) \right) \end{align}\tag{3}\] During training, the model’s perplexity—which expresses the confidence in the prediction of the correct tool call—is leveraged by \(\mathcal{L}_\text{replug}\) in Eq. 3 to encourage the reshape of the retrieval distribution. As a result, the encoder model \(\mathcal{E}_\theta\) learns to assign lower ranks to tools that increase the probability of generating incorrect responses.
3.3.0.2 Preference-Aligned Retrieval
We introduce a contrastive loss signal that enforces a policy to favor selecting correct tools over incorrect ones (Eq. 4 ). \[\label{eq:orpo-pi} \pi_\theta (t\vert q, d_t) = \frac{P_\mathcal{R}^\theta(t \vert q, d_t)}{1 - P_\mathcal{R}^\theta(t \vert q,d_t)}\tag{4}\] For each positive-negative tool pair \((t_i^+, t^-_{i,j})\) in the input sample, we compute a ratio of their retrieval probabilities and apply the sigmoid function \(\sigma\) to derive a preference score (i.e., the relative likelihood of selecting one tool over the other). \[\label{eq:orpo-loss} \begin{align} &\mathcal{L}_\text{po} = - \sum_{i \in {1,n}} \log \sigma \left ( \log \pi_{+,-_i} \right) \\ &\pi_{+,-_i} = \frac{\pi_\theta(t^+ \vert q, d_{t^+})}{\pi_\theta (t_i^- \vert q, d_{t_i^-})} \end{align}\tag{5}\] The retriever incurs a penalty through the preference policies \(\pi\), as indicated in Eq. 5 , when it shows an increased likelihood of selecting erroneous tools.
3.3.0.3 Training Objective
PORTS combines the LLM-based retrieval proxy loss (\(\mathcal{L}_\text{replug}\)) and the preference optimization loss (\(\mathcal{L}_\text{po}\)) with a weighting factor
\(\lambda\): \(\mathcal{L}_\texttt{PORTS}= \mathcal{L}_\text{replug} + \lambda \cdot \mathcal{L}_\text{po}\). Our method synchronizes the encoder selections with the LLM tool-calling
patterns and imposes positive-negative embedding orientation constraints similar to deep metric learning [46].
3.3.0.4 Tool Triplets and Embeddings Asynchronous Update
The learning effectiveness in contrastive approaches is heavily influenced by the choice of negative examples [40]. We therefore implement a hard-negative sampling technique, choosing the \(n\) tools most semantically similar to the query as negative instances, where similarity is quantified using cosine similarity between embeddings computed by the encoder itself during training. To maintain computational efficiency while adapting to shifts in the embedding space during training, we periodically update both the tool embeddings and the selection of hard negatives every \(T\) training iterations [38].
3.3.0.5 Motivations for Contrastive Learning
We establish that, without fine-tuning, embeddings of tool docstrings are much more concentrated in space than traditional, general-domain text documents–due to recurring data types, keywords, and concise but distinctive functional signatures (see
Appendix 13). Without targeted supervision, these properties can lead the retriever to rely on superficial cues or converge to trivial matches. Our contrastive loss is designed to counteract this by steering retrieval
toward semantically relevant and challenging negatives, promoting fine-grained distinctions beyond what is captured by LLM likelihoods alone. More details about the theoretical foundations of PORTS are in Appendix 7.
4 Experimental Setup↩︎
4.1 Datasets↩︎
We evaluate PORTS on six popular tool-augmented datasets: ToolBench [8], API-Bank [47], APIBench [48], BFCL-v2 [49], ToolE [50], Octopus-v2 [51]. This
collection offers a heterogeneous testing ground characterized by varying scales, applications, and input modalities. When necessary, we adapt dataset instances to tool selection, which is the core task of our contributions. Extensive dataset documentation
is available in Appendix 9. Key information and statistics are reported in Table ¿tbl:tab:datasets?. For ToolBench, we focus on the most complex data split, G3, where queries demand the
interplay of tools with dissimilar features, functions, and objectives. Only for training, we decompose multi-tool instances from BFCL, API-Bank, and ToolBench into separate examples, each targeting a single tool. When handling conversational inputs, we
remove previous tool calls from the chat history of each fragment. We partition test-only benchmarks (Octopus-v2, ToolE, BFCL) into train and test sets using a 70/30 ratio. For Octopus-v2 and ToolE, which focus on single-tool selection without
incorporating heterogeneous levels of complexity (e.g., difficulty levels or programming languages), we design in-domain and out-of-domain variants to evaluate generalization abilities to seen and unseen tools. Out-of-domain variants are created with an
80/20 tools split, avoiding overlap between training and test.
width=
4.2 Evaluation Metrics↩︎
We quantify the retrieval performance using Recall@\(K\) [52] and NDCG@\(K\) [53], with \(K=\{1,3,5\}\) following [8]. Recall@\(K\) measures the proportion of cases where the positive tool appears in the top-\(K\) results, while NDCG@\(K\) also considers its ranking position within the top-\(K\).
4.3 Implementation Details↩︎
4.3.0.1 Models
For the embedding function of \(\mathcal{R}\), we test two prominent encoders: the 125M-parameter RoBERTa-base [54], a widely adopted baseline, and the 109M-parameter BGE-base [55], a top performer on the MTEB leaderboard.3 We evaluate each encoder’s architectural compatibility and its impact on retrieval precision. For \(\mathcal{G}\), we examine three open-source LLMs. (1) Codestral-22B-v0.1,4 a model specialized for code generation tasks with native tool calling, motivated by [24]. (2) Llama3-8B[56], a foundational model, and (3) Llama3-Groq-8B-Tool-Use,5 its fine-tuned variant specifically designed for advanced tool use. We employ 4-bit quantization for efficiency. Larger models were excluded due to computational limits, focusing on a selection balancing performance and efficiency. Prompt templates are in Appendix 8.
4.3.0.2 Hyperparameters
In our experiments, we used 3 negatives sampled every \(T=50\) training steps upon recalculating embeddings. We adopted training and evaluation batch sizes of 2 and 4, respectively, and set maximum sequence lengths of
512 and 1024 for the encoder and LLM inputs. We applied a loss weight \(\lambda\) of 0.3, with \(\gamma\) and \(\beta\) set to 0.5. We set the random seed to
42 for reproducibility and trained each configuration for 2 epochs using the AdamW optimizer, coupled with a cosine learning rate scheduler starting at \(1e^{-5}\). We kept LLMs frozen during training. To simulate
low-resource scenarios, we sampled \(10\)K unique instances per dataset, testing PORTS’ capacity to exploit few information more efficiently. Search spaces and hyperparameter insights are in Appendix 10.
4.3.0.3 Hardware Setup
Each run was performed on an internal workstation using a single Nvidia GeForce RTX3090 GPU with 24GB of dedicated memory, 64GB of RAM, and an Intel® Core™ i9-10900X1080 CPU @ 3.70GHz. The reference operating system is Ubuntu 20.04.3 LTS.
5 Results↩︎
We evaluate PORTS against RePlug, our primary baseline for goal-driven retrieval. Other training methodologies are omitted from direct comparison due to divergent optimization goals and architectural
assumptions. Our core findings are in Table ¿tbl:tab:results?, showing metric scores for all models, datasets, and a comparison of different losses for ablation. Given space constraints, we list tool selection outcomes only for the
top-performing LLM in each encoder-dataset-loss configuration. We refer the reader to Appendix 14 for exhaustive scores. We observe that PORTS consistently elevates baseline effectiveness in all
datasets (cf. the color gradients for \(\Delta_{avg}\) columns linked to \(\mathcal{L}_\texttt{PORTS}\) entries in Table ¿tbl:tab:results?). For seen tools,
PORTS yields substantial gains, boosting average Recall@\(\{1,2,3\}\) by up to 71.66 percentage points and average NDCG@\(\{1,3,5\}\) by 70.16. Even with unseen tools, the
improvements remain remarkable, reaching +61.24 and +59.79 points in Recall and NDCG, respectively.
width=.995
5.0.0.1 Baseline Impact
Both RoBERTa and BGE fine-tuned with PORTS significantly outperform their respective base models, demonstrating the broad applicability of our framework. RoBERTa exhibits a more pronounced response to
PORTS(\(\Delta_{avg}\) Recall of 47.28 compared to BGE’s 10.07), indicating that simpler and less specialized retrievers are more adaptable.
Table ¿tbl:tab:results? clearly illustrate that BGE begins with a notable disparity between positive and negative instances (average frozen Recall of 62.92 vs. RoBERTa’s 18.45). This pre-existing imbalance in
the retrieval distribution introduces a skew, which in turn attenuates the strength of our loss signal. Further robustness tests are reported in Appendix 12.
5.0.0.2 LLM Impact
The use of an LLM as a proxy during training requires careful scrutiny. Although the primary focus is not the direct accuracy of the downstream generative task, the distribution of perplexities between different generations raises questions about the
optimal LLM to maximize learning influence. Empirical data did not elucidate a definitive model preference but instead accentuated PORTS’ potential to transcend its intended tool usage scope. The performance delta across all experiments ranged
from +1 to +4 percentage points, with peaks attaining a +13 average recall improvement. Llama3-8B demonstrated superior performance when paired with RoBERTa, where–as previously discussed–there was markedly less resistance to
adaptation. This improvement can be attributed to the heightened uncertainty this model experiences in technical domains not anticipated during its pre-training regimen. Although elevated perplexity is generally undesirable for downstream task
optimization, it is advantageous in the PORTS framework due to the resultant richer and less skewed log-likelihood distribution, as elucidated in Eq. 2 .
5.0.0.3 Docstring Impact
The efficacy of contrastive retrieval is heavily contingent on docstring quality. APIBench illustrates this relationship, where vague descriptions result in smaller changes in the tool retrieval metrics. In fact, the tool documents in the dataset reference generic features of pre-trained models from HuggingFace, which often omit parameter names and types that could better guide the retriever at training time. Importantly, Llama3-8B tends to perform better in datasets with less technical tool documentation due to its superior management of perplexity scores. On the other hand, Llama3-Groq-8B-Tool-Use and Codestral-22B-v0.1, with their advanced tool usage capabilities, excel in datasets with more detailed docstrings (e.g., arguments, outputs, types, defaults), namely ToolBench and Octopus-v2.
5.0.0.4 Training Loss
Contrastive learning and RePlug have known simultaneous success in various application domains. Our assertions on the positive impact of preference-oriented learning have been substantiated through comprehensive ablations
presented in Table ¿tbl:tab:results?. PORTS outperforms RePlug in all evaluated domains, with substantial disparities in recall performance, with +15.8 and +12.3 in ToolE and ApiBench,
respectively. We underline that these datasets present unique challenges: ToolE necessitates precise tool decisions among similar options in complex scenarios, while ApiBench involves code generation primarily through pre-trained neural networks, described
only in broad application contexts. Such unique characteristics are conducive to showcase the benefits of our comparative loss approach, which likely contributes to PORTS’s superior performance in these settings. Zooming out, the statistics
outlined in Figure 1 reflect an average Recall@\(K\) improvement of 6.7 (RoBERTa) and 3.4 (BGE) across all datasets and LLMs, ultimately corroborating the
greater efficacy of our method over RePlug in the context of tool retrieval.
5.0.0.5 Out-Of-Domain Analysis
For domains with limited resources and scarce data, retrieval systems must be capable of effectively managing unfamiliar tools. Although previous research has developed systems that can adapt to new data [9], these systems are susceptible to overfitting, potentially due to biases in the distribution and usage patterns of tools. To evaluate
the robustness of our method in addressing these challenges, we investigated the performance of retrieval models trained with RePlug and PORTS when exposed to varying proportions of unseen tools from the ToolE
dataset. Starting with a 90/10 ratio of seen to unseen tools, we progressively reduced the training dataset and assessed the performance of RoBERTa guided by Llama3-8B on a consistent test distribution. Figure 4 illustrates the superior generalization capabilities of PORTS. Employing preference optimization loss, our contrastive learning techniques effectively derive semantic insights into query-tool
interactions without requiring extensive pairwise comparisons, thereby substantiating the enhanced low-resource capabilities of our solution and demonstrating its suitability for application areas with constantly evolving API documentation. Figure 3 reports the average recall across test datasets and encoder models for all PORTS’ variants fine-tuned with different LLMs.
6 Conclusion↩︎
We introduce PORTS, a novel training method to optimize encoders for tool retrieval tasks. Our goal is twofold: to align tool selection with the preferences of the calling LLM, and to maximize the odds ratio between correct and incorrect
tools. PORTS emphasizes low cost by leveraging LLMs’ prior knowledge to navigate the latent space of tool document similarities, focusing on the impact of retrieved samples. Experiments across multiple models and diverse datasets show that
PORTS achieves Recall@1 improvements of up to +72.5% and +58.7% over frozen baselines for in- and out-domain tools, with gains of +15.24% and +14.71% percentage points
compared to RePlug.
To further assess retrieval effectiveness in code generation scenarios, metrics like Pass@\(K\) could reflect the downstream impact on generative components. Incorporating relevance signals
based on actual output effects and message similarity may allow PORTS to integrate discounting mechanisms for more goal-directed retrieval. Future work could also investigate PORTS in biomedical discovery workflows [57]–e.g. helping an agent decide whether to use specialized function discovery models [58]–[60], call Gene Ontology tooling for term enrichment, query STRING for protein-protein interaction networks, or fetch data from the Unified Medical Language System–with dozens to hundreds of specialized data sources and
tools with different formats, coverage, and update frequency.
Acknowledgment↩︎
Research partially supported by: AI-PACT (CUP B47H22004450008, B47H22004460001); National Plan PNC-I.1 DARE (PNC0000002, CUP B53C22006450001); PNRR Extended Partnership FAIR (PE00000013, Spoke 8); 2024 Scientific Research and High Technology Program, project “AI analysis for risk assessment of empty lymph nodes in endometrial cancer surgery”, the Fondazione Cassa di Risparmio in Bologna; Chips JU TRISTAN (G.A. 101095947). LG Solution Srl for partially funding a PhD scholarship to L. Molfetta.
Limitations↩︎
Despite its strong results, PORTS has limitations that warrant further examination. First, its effectiveness is sensitive to the quality of tool documentation, with diminished gains in domains where docstrings are vague or underspecified.
Second, although PORTS maintains a memory efficiency comparable to RePlug LSR, it requires repeated querying of a frozen LLM to calculate the guidance signals. This dependence introduces additional computational overhead in
both time and memory, potentially limiting scalability in resource-constrained settings. Future work may address these challenges by reducing the reliance on LLM inference or developing efficient approximations of guidance signals, thus improving the
practicality of retrieval methods that use LLMs as proxies at training time.
7 In-Depth View on the Relevance of PORTS↩︎
Given an input query \(q\) and a set of tools \(t\) with description \(d_{t_i}\), PORTS combines RePlug’s
goal-directed retrieval with preference alignment, adapted explicitly for tool selection. This approach enhances both relevance and downstream performance, effectively addressing gaps in existing methods. The foundation lies in RePlug’s gradient structure, derived from minimizing the KL divergence between the retriever distribution \(P_\mathcal{R}(t|q,d_t)\) and the LM’s utility signal \(Q_\mathcal{G}(t|q,d_t,y)\).
7.0.0.1 RePlug Gradient Derivation
Starting from the KL divergence objective: \[\begin{align} \mathcal{L}_\text{replug} &= \text{KL}(P_\mathcal{R}^\theta \parallel Q_\mathcal{G}) \nonumber \\ &= \sum_t P_\mathcal{R}^\theta(t|q, d_t) \log \frac{P_\mathcal{R}^\theta(t|q,d_t)}{Q_\mathcal{G}(t|q,d_t,y)}, \end{align}\] we decompose it into entropy and cross-entropy terms: \[\begin{align} \text{KL} &= \underbrace{\sum_t P_\mathcal{R}^\theta \log P_\mathcal{R}^\theta}_{-H(P_\mathcal{R}^\theta)} - \underbrace{\sum_t P_\mathcal{R}^\theta \log Q_\mathcal{G}}_{-H(P_\mathcal{R}^\theta, Q_\mathcal{G})}. \end{align}\] To derive gradients with respect to retriever scores \(sim(q,d_t)\), we differentiate both terms. First, for the entropy term \(H(P_\mathcal{R}^\theta)\): \[\begin{align} \begin{aligned} \frac{\partial H(P_\mathcal{R}^\theta)}{\partial sim(q,d_t)}&= \frac{1}{\gamma}P_\mathcal{R}^\theta(t|q,d_t)\; \cdot \\ &\cdot\; \Bigl( 1 + \log P_\mathcal{R}^\theta(t|q,d_t) - H(P_\mathcal{R}^\theta) \Bigr), \end{aligned} \end{align}\] where \(\gamma\) is the retriever’s temperature parameter. For the cross-entropy term \(H(P_\mathcal{R}^\theta, Q_\mathcal{G})\): \[\begin{align} \begin{aligned} &\frac{\partial H(P_\mathcal{R}^\theta, Q_\mathcal{G})}{\partial sim(q,d_t)} = \frac{1}{\gamma}P_\mathcal{R}^\theta(t|q,d_t)\cdot \\&\cdot \left(\log Q_\mathcal{G}(t|q,d_t,y) - \mathbb{E}_{P_\mathcal{R}^\theta}[\log Q_\mathcal{G}]\right). \end{aligned} \end{align}\] Subtracting these gradients yields: \[\begin{align} \begin{aligned} &\frac{\partial \mathcal{L}_\text{replug}}{\partial sim(q,d_t)} = \frac{P_\mathcal{R}^\theta(t|q,d_t)}{\gamma} \cdot \\ &\;\;\;\;\;\; \cdot \left( \log\frac{P_\mathcal{R}^\theta(t|q,d_t)}{Q_\mathcal{G}(t|q,d_t,y)} \quad - \text{KL}(P_\mathcal{R}^\theta \parallel Q_\mathcal{G}) \right). \end{aligned} \end{align}\] Here, \(\log\frac{P_\mathcal{R}^\theta}{Q_\mathcal{G}}\) provides per-document alignment signals, while the \(\text{KL}\) term stabilizes training by serving as a baseline for global distribution shifts.
7.0.0.2 PORTS Gradient Extension
Our analysis of the full PORTS loss gradient highlights the limitations of RePlug alone: \[\begin{align} \begin{aligned} &\nabla_\theta \mathcal{L}_\text{PORTS} \propto
\nabla_\theta \mathcal{L}_\text{replug} + \nabla_\theta \mathcal{L}_\text{po} \\ &\nabla \mathcal{L}_\text{replug} = P_\mathcal{R}^\theta(t|q,d_t) \Biggl( \log\frac{P_R(t|q,d_t)}{Q_\mathcal{G}(t|q,d_t,y)} +\\& - \text{KL}(P_\mathcal{R}^\theta
\parallel Q_\mathcal{G}) \Biggr) \\ & \nabla_\theta \mathcal{L}_\text{po} = \left(1 + \frac{\pi_\theta(t|q,d_{t^+})}{\pi_\theta(t|q,d_{t^-})}\right)^{-1}. \end{aligned}
\end{align}\] While \(P_\mathcal{R}^\theta/Q\) compares the retriever confidence against the LLM’s docstring utility assessment, the KL term acts as a global stabilizer that prevents over-adjustments to individual
docstrings. Without our preference optimization component, the system simply minimizes differences between retrieval probabilities \(P_\mathcal{R}^\theta\) and downstream log-likelihoods \(\mathcal{Q}_\mathcal{G}\), overlooking LLM handling of imperfect descriptions. Specifically, pure RePlug gradients lack awareness of relative tool utility – a critical shortfall when
multiple tools have overlapping or ambiguous descriptions.
The preference optimization contrastive term addresses this by enforcing refinement of tool selection through pairwise comparisons. Unlike the KL penalty, which operates globally, the term \[\left( 1 + \frac{\pi_\theta(t|q,d_{t^+})}{\pi_\theta(t|q,d_{t^-})} \right)^{-1}\] explicitly rewards the retriever for distinguishing between semantically similar but functionally distinct tools (\(d_{t^+}\) and \(d_{t^-}\)). This creates implicit links between tools based on their downstream task performance rather than surface-level similarity. For example, when two API descriptions share terminology but differ in required parameters, the contrastive component amplifies gradients for the tool whose documentation better resolves this ambiguity in practice.
In so doing, PORTS benefits from negative examples not only through the downstream task signal (which can be noisy due to LLM approximation errors) but also through structured semantic comparisons. The retriever learns to associate subtle
linguistic cues in tool descriptions with their functional outcomes, even when \(Q_\mathcal{G}\) provides imperfect supervision. This dual mechanism proves critical in real-world scenarios where tool documentation
quality varies widely – the contrastive term compensates for sparse or ambiguous \(\mathcal{Q}_\mathcal{G}\) signals by reinforcing discriminative features across the toolset. PORTS’ triplet formulation also
enables flexible, state-independent negative sampling, unlike RePlug‘s iterative sampling, which may introduce bias. A qualitative example of PORTS’ retrieval and disambiguation capabilities is shown in
Appendix 15.
8 Prompt Templates↩︎
The prompt template for Large Language Models (LLMs) is typically divided into two distinct components: a system message and a user instruction. The system message serves to establish the model’s role and behavioral parameters. In contrast, the user
instruction delineates the specific task or query to be addressed. In cases where a model’s chat template does not inherently accommodate a discrete system message, this information is instead prepended to the user instruction. This ensures that the model
is primed with all necessary contextual and behavioral guidelines before processing the task at hand. Within the PORTS framework, a frozen LLM is prompted with the input query and the docstring of a retrieved tool to gauge the probability of
predicting the target call. During training, we mask the input up to the “Answer” tag and compute the next-token probability of the gold answer using the system and instruction sections as input with a causal attention approach. Our prompt templates are
reported in Listing 5 and Listing 6. To better recall the prior knowledge of the model, we describe tools as API functions. We use each model’s specific chat template,
omitting special tokens in the listings for clarity. We use a different prompt when working with Llama3-8B to better adhere to the chat template on which it was trained, as suggested in the HuggingFace model card.6
9 Software and Datasets: Details, Intended Use, and Impact↩︎
Despite the popularity of RePlug, no implementation code was publicly available. We dedicated significant effort to reconstructing the method from scratch, carefully clarifying its methodological
choices. To benefit the broader research community, we release our complete implementation–including PORTS–as fully open-source under a permissive MIT license. This ensures full reproducibility and establishes the first
open-source solution for goal-directed encoder fine-tuning.
Queries, tools, and docstrings can vary greatly depending on the dataset. Table 1, Table 2, Table 3, Table 4, Table 5, and Table 6 show representative input-output examples sampled from the test set of each dataset. Each dataset has been pre-processed to ensure compatibility with our tool-selection task by decoupling instances that require the use of multiple tools. Tool descriptions have been enhanced with detailed information about input and output parameter types, formats, and purposes, following Python-style docstrings to better define the scope of each tool and facilitate the retrieval process. Given the conversational and multi-tool nature of the API-Bank dataset, we have distinguished between inputs for the retriever and generative models. The retriever’s input excludes previous tool calls to avoid biases and inconsistencies in the similarity-based search, while generative models receive the full conversation history. This approach enhances tool selection accuracy while allowing generative models to leverage complete contextual information.
9.0.0.1 Licenses of Used Datasets
All datasets used in our experiments are publicly available and released under permissive open-source licenses. Specifically, ToolBench, ApiBench, Octopus, and BFCLv2 are distributed under the Apache 2.0 License, while ToolE and ApiBank are released under the MIT License. These licenses allow for both academic and commercial use, ensuring full compliance with open-source standards and enabling reproducibility of our experiments.
10 Hyperparameter Space↩︎
Table ¿tbl:tab:hyperparams? presents a comprehensive overview of the hyperparameters explored in our study. This extensive search space was designed to optimize the model’s performance across various dimensions, from basic configuration settings to more nuanced training parameters.
None
Figure 5: Prompt template for call generation with the retrieved tool for Gemma3-1B, Qwen3-4B, Llama3.2-3B, Llama3-8B, and Codestral-22B-v0.1..
None
Figure 6: Prompt template for call generation with the retrieved tool for Llama3-Groq-8B-Tool-Use..
| Field | Text |
|---|---|
| Query | Please provide me with the user information for the user with the username ’michaelbrown’. Also, fetch the order details for order ID 31415 and get the inventory status of the store. |
| Gold Tool’s Docstring | def petstore_blitz.getUserByName(): |
| Answer | petstore_blitz.getUserByName(username="michaelbrown") |
| Field | Text |
|---|---|
| Query | What’s the weather like in New York City for the next three days? |
| Gold Tool’s Docstring | def get_weather_forecast(): |
| Answer | get_weather_forecast(’New York City’) |
| Field | Text |
|---|---|
| Query | User: Can you give me a list of all the stress reduction technique categories you offer? |
| Gold Tool’s Docstring | User: Can you give me a list of all the stress reduction technique categories you offer? |
| Answer | def addTechnique(): |
| Answer | addTechnique(name=’Visualization’, description=’a relaxation exercise in which you create a peaceful mental image of a place or situation’, category=’Meditation’, difficulty=’easy’) |
| Field | Text |
|---|---|
| Query | Users want to engage in a conversation with a fictional character based on their persona. This conversation will be used as part of a script for an animation series. |
| Gold Tool’s Docstring | def AutoModelForCausalLM.from_pretrained(’pygmalion-6b’): |
| Answer | AutoModelForCausalLM.from_pretrained(’pygmalion-6b’) |
| Field | Text |
|---|---|
| Query | Search for a Chicken Noodle Soup recipe and a Vegan Salad recipe. |
| Gold Tool’s Docstring | def recipe_search.find(): |
| Answer | recipe_search.find(dish="Chicken Noodle Soup", diet="Vegan") |
| Field | Text |
|---|---|
| Query | Help me with a quick d20 roll, I’ve got a crucial decision to make in my game. |
| Gold Tool’s Docstring | def diceroller(): |
| Answer | diceroller() |
width=
11 Computational Budget↩︎
All experiments were performed on machines equipped with NVIDIA RTX 3090 GPUs (24GB VRAM). The total compute time required for training and evaluation across all PORTS’ variants amounted to approximately 500 GPU-hours. This includes
finetuning on multiple datasets, ablative experiments, running inference with large language models, and conducting retrieval evaluations.
12 Robustness↩︎
The efficacy of PORTS was evaluated through ablation studies to determine optimal parameter configuration and assess robustness across configuration variations, using the ToolE dataset with RoBERTa as encoder and Llama3-8B as generative models. To demonstrate the effectiveness of the contrastive loss, we examined the impact of using different numbers of negative examples in the learning process, with results in Figure 7 illustrating advantages of incorporating larger numbers of examples which better guide preference optimization. We investigated the effects of varying weighting factors \(\beta\) and
\(\gamma\), with Figure 8 showing higher \(\beta\) and lower \(\gamma\) values yield improved results, optimal when both
are set to 0.5. Additionally, we examined the influence of random seeds on our method, focusing on their impact on input data distribution and dropout layer behavior, with results in Table ¿tbl:tab:avg95recall95seed? demonstrating the
robustness and effectiveness of PORTS and its low variance in response to such configuration changes.
width=.85
13 Clustering Properties of Tool Embeddings↩︎
API docstrings exhibit skewed token distributions, dominated by recurring elements such as data types and keywords. Compared to general-domain retrieval corpora, tool-related datasets form well-separated semantic clusters, owing to their concise yet
distinctive functional signatures. Without targeted supervision, these structural properties can lead retrieval models to rely on superficial lexical cues or converge toward trivial matches. To characterize the clustering tendency of these representations,
we apply standard unsupervised algorithms–including K-Means (with \(K \in [2, 16]\)), Agglomerative Clustering (Ward linkage), and DBSCAN (with \(\epsilon = 0.3\))—on 20,000 randomly sampled
embedding vectors per dataset. We use four top-performing models from the MTEB leaderboard7 to extract these representations: BAAI/bge-m3,8 intfloat/multilingual-e5-large-instruct,9 answerdotai/ModernBERT-base,10 and Alibaba-NLP/gte-multilingual-base.11 The silhouette coefficient is
computed to quantify clustering quality, capturing both intra-cluster cohesion and inter-cluster separation. As summarized in Figure 9, tool-centric datasets consistently achieve higher silhouette scores than
general-domain corpora, including MSMarco [61], HotpotQA [62], and PubMedQA [63]. This high clustering tendency presents a fundamental limitation where semantically similar tools concentrate within the same dense clusters, making contrastive supervision essential to differentiate tools that share
similar descriptive features yet possess distinct functionalities and produce varying effects on LLM behavior, necessitating targeted intra-cluster contrastive learning to prevent training from optimizing merely for tool relevance rather than functional
effectiveness. This gap highlights the stronger intrinsic structure of tool embeddings and further motivates our contrastive supervision strategy to promote fine-grained, functionally meaningful distinctions beyond those induced by LLM likelihoods
alone.
width=
width=
14 Complete Results↩︎
Table ¿tbl:tab:results95roberta? and Table ¿tbl:tab:results95bge? complement the results of the main paper, listing the retrieval scores achieved by training encoders under the supervision signal of each LLM explored.
15 Qualitative Example↩︎
Table 10 presents a specific query from the ToolE test set, demonstrating the contrast between the top-3 tools retrieved by BGE with and without PORTS tuning. The results clearly illustrate
that our alignment process not only successfully positions the correct tool at the top rank, but also generates a significantly sharpened preference distribution.
References↩︎
Code, models, and datasets are publicly available at https://github.com/disi-unibo-nlp/ports↩︎
Consistent with [1], we argue that all external means of augmenting LLMs should be classified as tools. Accordingly, we regard individual APIs as separate tools.↩︎