1 Introduction↩︎

Code search is a fundamental process in software engineering, allowing developers to retrieve and query semantically relevant code snippets from large-scale codebases using natural language queries (NL-to-Code) [1]. This capability is crucial for code reuse, discovering relevant examples, and accelerating the learning and onboarding process for developers [2]. With the advancement of large language models (LLMs) such as ChatGPT [3] and DeepSeekCoder [4], Retrieval-Augmented Generation (RAG) has become a prominent approach to enhancing their capabilities in code-related tasks [5]. By retrieving and incorporating relevant code snippets, LLMs can significantly improve performance in software engineering tasks like code generation, among others [6]. However, the accuracy of code search remains a critical bottleneck [1], [7].

The prevailing approach to code search employs a pre-training and fine-tuning paradigm [8], typically using encoder-based models like CodeBERT [9] or [10]. In this method, natural language queries and code are converted into vector embeddings and fine-tuned with specific NL-to-Code datasets. This contrastive learning process helps capture semantic similarities, aligning related queries and code snippets in the vector space.

However, these pioneering encoder-based models face a fundamental bottleneck when compared to the current generation of LLMs: a significant gap in scale and semantic capability. Models like CodeBERT, typically with hundreds of millions of parameters, are orders of magnitude smaller than modern decoder-only LLMs [9]. This vast difference in scale, originating from billions of parameters and pre-training on trillions of tokens, endows LLMs with superior semantic understanding and reasoning abilities [11]. This raises a critical question: can these advanced capabilities address the inherent generalization limitations of smaller models and redefine the performance ceiling for code search?

Decoder-only LLMs excel in natural language processing and software engineering tasks, outperforming fine-tuned small-scale models in areas like code generation and program repair [12]. Their ability to process long and complex contexts [13], combined with rich pre-training data and large-scale parameters, enhances code and natural language understanding [2], [12], [14], potentially addressing the bottlenecks in code search tasks.

Despite their potential, the application of decoder-only LLMs, such as [15], CodeLlama [16], and DeepSeekCoder [4], to code search tasks remains underexplored. The suitability of these models for code search is still unclear. One key challenge is that code search is not a generative task like code generation or program repair, which better align with the pretraining tasks typically used for decoder-only LLMs. As a result, directly applying decoder-only LLMs to code search may not be the most effective approach. A deeper investigation is needed to determine whether these models can be effectively used for code search, how best to leverage them for this task, and whether they can outperform existing approaches based on smaller encoder-only pre-trained models. This paper aims to explore whether decoder-only LLMs could be the “silver bullet” for code search, offering new possibilities for improving efficiency and accuracy.

Study Design. To address this gap, we systematically evaluate the performance of state-of-the-art (SOTA) decoder-only LLMs in code search. Specifically, we investigate the following research questions (RQs):

• RQ1 (Zero-shot Performance): How well do decoder-only LLMs perform on code search without fine-tuning?

• RQ2 (Fine-tuning Improvement): To what extent does fine-tuning improve the performance of decoder-only LLMs over the zero-shot setting?

• RQ3 (Improvement Analysis): What factors contribute to performance gains from fine-tuning?

  • RQ3.a (Training Method): How do different fine-tuning strategies impact performance?

  • RQ3.b (Training Data): How does the quality and type of training data influence fine-tuning effectiveness?

  • RQ3.c (Single-Language Fine-Tuning): How the specific programming language used for fine-tuning influences a model’s ability to learn generalizable code representations?

  • RQ3.d (Model Size): How does the model size influence the effectiveness of fine-tuning in code search tasks?

  • RQ3.e (Query and Code Length): How do query and code lengths affect model performance?

• RQ4 (Computational Time): How does the computational time of decoder-only LLMs compare to that of smaller encoder-only models in code search tasks?

• RQ5 (Training Efficiency): How efficiently do decoder-only LLMs learn and generalize during fine-tuning compared to smaller encoder-only models in code search tasks?

These RQs systematically examine both the effectiveness and efficiency of decoder-only LLMs in code search. RQ1 establishes a baseline by assessing their zero-shot performance, while RQ2 investigates the impact of fine-tuning. RQ3 further explores the key factors driving performance improvements, considering fine-tuning methods, training data, model size, and input characteristics. Beyond accuracy, RQ4 examines computational cost, providing practical insights into model efficiency. Finally, RQ5 evaluates training efficiency, assessing how well these models adapt and generalize during fine-tuning. Together, these RQs provide a holistic evaluation of decoder-only LLMs for code search, balancing performance with resource efficiency.

Results and Key Findings. This study evaluates eleven state-of-the-art (SOTA) decoder-only LLMs for code search tasks, conducting a comprehensive analysis of their performance across two fine-tuning methods, two types of datasets, and five model sizes. Although decoder-only models initially underperform in zero-shot settings due to the mismatch between their code representations and the requirements of the code search task, fine-tuning significantly improves their performance, enabling them to better leverage their pre-trained code knowledge. Among the models evaluated, fine-tuned CodeGemma emerged as the top performer. On the CSN dataset, CodeGemma achieved a 4.8% improvement in average MRR over the leading encoder-only model, UniXcoder. On the dataset, where neither model was explicitly trained, CodeGemma demonstrated impressive gains by achieving a 40.4% increase in MAP compared to UniXcoder. These findings highlight the effectiveness of decoder-only LLMs, especially after fine-tuning, in generalizing across unseen datasets, with their larger size and richer pretraining enhancing their ability to adapt to different code search scenarios.

Our analysis also indicates that fine-tuning on code-specific datasets, employing supervised contrastive learning, and mid-sized model contribute to performance improvements. However, model architecture remains crucial, as larger models do not always guarantee better results. Decoder-only LLMs excel in long-code searches but struggle with ultra-short queries (fewer than 10 tokens) due to the curse of dimensionality and insufficient context. Although larger models lead to longer computational times, the costs are manageable, and they demonstrate superior training efficiency and generalization on limited data compared to smaller encoder-only models. In summary, this study highlights the significant potential of fine-tuned decoder-only LLMs for code search tasks, with strong performance and generalization across varying query lengths and datasets.

Our replication package, including code and results, is available online [17]. Furthermore, to facilitate further research, all fine-tuned models evaluated in this study are publicly available in a dedicated Hugging Face Collection [18]. The key contributions of this work are:

• First Systematic Study on Decoder-Only LLMs for Code Search: We systematically examine decoder-only LLMs in code search, comparing their zero-shot and fine-tuned performance.

• Comprehensive Benchmarking: We demonstrate that fine-tuned decoder-only LLMs, particularly CodeGemma, surpass encoder-only models like in code search, showcasing superior generalization.

• Optimization Strategies: We provide insights into model selection, fine-tuning techniques, training data quality, model size, query/code length impact, computational cost, and training efficiency.

2 Background↩︎

2.1 Code Search↩︎

Early code search engines were based on keyword matching between queries and code, primarily relying on text similarity methods[19]. While functional, these approaches struggled to accurately capture the semantics of code due to significant differences between programming and natural languages. Recent advancements in deep learning have led to more sophisticated models capable of extracting high-level semantic representations, significantly improving code search performance[20]. Deep learning models utilize neural networks to uncover hidden features from data, which aids in generating semantic representations of both natural language and code[21].

Gu et al. [20] were pioneers in applying deep neural networks to embed code and queries into a shared vector space, measuring their similarity through vector distances. Since then, various model architectures have been applied to code search, including sequence models [22], [23], convolutional neural networks (CNN) [24], [25], tree neural networks [26], graph models [27], and Transformer-based models [28], [29].

The development of pre-trained models on large-scale code datasets has also enhanced semantic understanding and search capabilities. For example, models like BERT [30], CodeBERT [9], GraphCodeBERT [8], and UniXcoder [10] have demonstrated impressive performance by training on bidirectional Transformers, where each token can attend to all others. Encoder-only architectures are typically favored for code search tasks due to their ability to handle code understanding better than decoder-only or encoder-decoder architectures [31]. However, these models still face critical challenges:

• Poor Generalization: Fine-tuned models often require task-specific adjustments, but fine-tuning datasets tend to be small, noisy, and narrowly sourced, which increases costs and can limit model performance on broader scenarios[32], [33]. Additionally, most models in code search, such as UniXcoder[10], have fewer than 100 million parameters, which restricts their generalization and performance on unseen examples.

Previous studies have shown that encoder-only models, such as UniXcoder, outperform smaller decoder-only models like CodeT5[34] in code search tasks [35]. However, the potential of decoder-only LLMs in this domain remains underexplored. This study aims to fill this gap by conducting the first systematic evaluation of decoder-only LLMs for code search. We compare their performance against traditional encoder-based models, offering insights into their strengths, limitations, and optimization strategies for improving code search performance.

2.2 Decoder-only LLMs for Information Retrieval↩︎

Recent research has applied decoder-only models to text embedding for information retrieval (IR), with notable improvements. For example, Ma et al. [36] fine-tuned LLaMA 2 [37] using S-BERT methods, and Wang et al. [38] created high-quality synthetic datasets for better IR. Springer et al. [39] proposed echo embeddings to address model robustness issues, while BehnamGhader et al. [40] used bidirectional attention and dual training sessions to improve performance.

However, these studies focus on general IR, with limited exploration in code search. No research has yet applied decoder-only LLMs like , CodeLlama, or DeepSeekCoder to code search tasks. This gap highlights the need to explore the performance of decoder-only models in code search specifically.

To the best of our knowledge, this paper is the first to systematically explore decoder-only LLMs for code search. We demonstrate that models like CodeGemma outperform encoder-only models such as , offering superior generalization. Additionally, we provide insights into optimizing these models for code search, including model selection, fine-tuning, training data, and model size.

3 Study Setup↩︎

In this section, we present the benchmark, metrics, and the selection/configuration of the LLMs used in our study.

3.1 Benchmark↩︎

To thoroughly assess the performance of decoder-only LLMs in code search tasks, we utilized the (CSN) [31] and  [41] datasets.

• The CSN dataset is a comprehensive benchmark specifically designed for evaluating code search tasks [31], frequently used by previous work [1], [7]. It encompasses a diverse range of programming languages, including Python, Java, JavaScript, Ruby, Go, and PHP. The dataset consists of millions of code snippets extracted from open-source repositories automatically, each paired with corresponding natural language documentation. This pairing facilitates the evaluation of code search models by measuring their ability to retrieve relevant code snippets in response to natural language queries.

• The dataset is the latest benchmark for Python code search [41]. is an enhanced version of the  [42] dataset, designed to address common challenges in existing code search datasets. It improves upon by pairing high-quality queries with multiple appropriate code snippets, blocks, and functions. These queries come from , and the code snippets are sourced from the filtered StaQC [43] and  [31] datasets. The candidate pairs are formed using multiple models and annotated automatically with the help of LLMs like Claude 3 Sonnet and GPT-4o, ensuring accurate matches. Using this dataset mitigates data leakage risk in two key ways: first, some of the generated responses do not originate from CSN, significantly reducing the potential for data leakage. Second, it differs from previous code search benchmarks in its task objectives by matching queries with multiple suitable code snippets. This divergence in dataset content and task goals further decreases the likelihood of data leakage.

We selected these two datasets due to their complementary strengths. CSN is widely used and covers multiple programming languages, facilitating alignment with previous research. , released in 2024, provides a realistic and challenging benchmark for Python code search, featuring multiple correct answers per query. This combination allows us to evaluate our models comprehensively across different languages and complexity levels. Table 1 provides more detailed information about these datasets, including the composition and number of datasets.

3.2 Metrics↩︎

We used Mean Reciprocal Rank (MRR) [44] and Mean Average Precision (MAP) [1] as our primary evaluation metrics. These metrics are widely adopted in information retrieval (IR) [45] and have been commonly used in previous code search research [1], [7]. Both MRR and MAP are designed to be maximized, with their values ranging from 0 to 1, where higher values indicate better performance.

MRR measures the mean of the reciprocals of the rank positions of the first relevant code snippet for each query. It is calculated as shown in Equation 1 , where \(\text{Rank}_i\) denotes the rank position of the first relevant code snippet for the \(i\)-th query, and \(N\) is the total number of queries. MRR is particularly useful in code search tasks because it emphasizes the rank of the first relevant result, reflecting the effectiveness of retrieving relevant code snippets early.

\[\label{eq:mrr} \text{MRR} = \frac{1}{N} \sum_{i=1}^{N} \frac{1}{\text{rank}_i}\tag{1}\]

MAP calculates the average precision score at each relevant item retrieved, averaged over multiple queries. It is computed as shown in Equation 2 , where \(N\) is the total number of queries, and \(\text{AP}_i\) is the Average Precision for the \(i\)-th query. The Average Precision (\(\text{AP}_i\)) is calculated as shown in Equation 3 , where \(Q_i\) is the total number of relevant code snippets for the \(i\)-th query, and \(\text{rank}(i,j)\) denotes the rank position of the \(j\)-th relevant code snippet for the \(i\)-th query. MAP provides a comprehensive measure of search performance by considering the precision of relevant results across the entire result set. It is especially suited for evaluating datasets like that involve multiple correct answers per query.

\[\label{eq:map} \text{MAP} = \frac{1}{N} \sum_{i=1}^{N} \text{AP}_i\tag{2}\]

\[\label{eq:ap} \text{AP}_i = \frac{1}{Q_i} \sum_{j=1}^{Q_i} \frac{j}{\text{rank}(i,j)}\tag{3}\]

3.3 Models↩︎

4pt

To thoroughly evaluate the effectiveness of decoder-only LLMs in code search tasks, we selected 11 state-of-the-art (SOTA) LLMs extensively examined in recent studies. We focused on open-source models released after 2023, excluding smaller models (with fewer than 1 billion parameters) due to their limited efficacy and larger models (with more than 7 billion parameters) due to computational resource constraints. Table 2 presents the details of the LLMs studied in our experiments, including model type, base models, version, publication year, model sizes and training data sizes. Our study encompasses a diverse range of decoder-only LLMs, varying across multiple dimensions such as (i) utilization of different base models, (ii) inclusion of various versions of series models (e.g., Llama2 and Llama3), and (iii) specialization in general-purpose versus code-specific training. To ensure readability throughout the paper, we refer to models by their common names (e.g., “CodeGemma") after specifying the exact version used in the table.

The selected LLMs are divided into two categories: general LLMs and code LLMs. General LLMs are trained for broad tasks, while code LLMs are specialized for software engineering, either trained from scratch on code corpora or fine-tuned with additional code data on top of general LLMs. We aim to investigate whether code LLMs outperform general LLMs in code search tasks, which require strong understanding of both natural language and code. For the general LLMs, we include their code-specific versions where available. Currently, a code-specific version of Llama3 is not available.

The details of the studied LLMs are as follows:

• Llama2 [37]: Employs a decoder-only transformer architecture with a 32K token vocabulary. While the larger 70B model in this family uses Grouped-Query Attention (GQA) for efficiency, the 7B and 13B versions used in our study utilize standard Multi-Head Attention. All versions feature Rotary Positional Embeddings.

• CodeLlama [16]: Developed by fine-tuning Llama2 with a higher sampling of code, featuring code infilling capabilities, support for large input contexts, and zero-shot instruction following for programming tasks.

• Mistral [46]: Incorporates Grouped Query Attention for faster inference and Sliding Window Attention to handle longer sequences efficiently, enhancing contextual understanding and reducing memory usage.

• CodeMistral [47]: Derived from Mistral, trained on a curated, mixed dataset of approximately 201,981 samples from six sources (including Open-Platypus[48], OpenOrca[49], and WizardLM[50]) specifically filtered for coding and reasoning tasks.

• DeepSeekLLM [15]: Follows the Llama model design with a Pre-Norm structure and RMSNorm function, using SwiGLU for feed-forward network activation.

• DeepSeekCoder [4]: Initially pre-trained with a diverse dataset consisting of 87% code, 10% code-related language (e.g., GitHub Markdown and StackExchange), and 3% non-code-related Chinese language, with further instruction fine-tuning.

• Gemma [51]: A lightweight, text-to-text, decoder-only LLM from Google, designed for various text generation tasks and optimized for deployment in resource-limited environments.

• CodeGemma [52]: Further trained based on Gemma, using a combination of open-source math datasets and synthetically generated code to enhance logical reasoning and problem-solving skills.

• Llama3 [53]: Utilizes a standard decoder-only transformer architecture with a 128K token vocabulary tokenizer and Grouped Query Attention (GQA) for improved efficiency and performance.

• StarCoder2 [54]: Trained on over 4 trillion tokens of code from The Stack v2 (covering 17 programming languages), it utilizes Grouped Query Attention (GQA), a 16,384-token context window with a 4,096-token sliding window, and a Fill-in-the-Middle (FIM) training objective to enhance code understanding.

• Qwen2.5-Coder [55]: A series of multilingual code models from Alibaba’s Qwen team, trained on a large-scale corpus of high-quality text and code, and supporting an extensive context window for better long-range dependency understanding.

Additionally, we include two encoder-only models for comparison: CodeBERT [9] and UniXcoder [10]. CodeBERT, based on the RoBERTa architecture, is pre-trained on a large code corpus using Masked Language Modeling (MLM) and Replaced Token Detection (RTD). UniXcoder, on the other hand, is pre-trained with a combination of masked language modeling, unidirectional language modeling, denoising autoencoding, and contrastive learning on multi-modal data, including code summaries and abstract syntax trees. We do not include smaller decoder-only models, such as CodeT5 [34], for comparison. Previous work [35] has shown that these models tend to perform worse than UniXcoder on code search tasks.

We obtained these models from their official repositories and used them for zero-shot inference or fine-tuning according to the provided guidelines. All evaluations were performed on an NVIDIA A800 80GB GPU.

4 RQ1: Zero-shot Performance↩︎

In this RQ, we investigate the performance of decoder-only LLMs on code search tasks in a zero-shot setting. This means applying the model directly to the code search task by obtaining embeddings of the query and the code in the code corpus without any task-specific fine-tuning. We compare the performance of eleven SOTA decoder-only LLMs and two SOTA encoder-only models under these conditions using the CSN and datasets.

4.1 Design↩︎

In this section, we outline the experimental design for evaluating the models using the CSN and benchmarks. Our approach involves encoding both queries and code snippets into vector representations, followed by ranking the code snippets based on their cosine similarity to the query. We evaluate two categories of models: decoder-only LLMs (for both general-purpose and code tasks) and encoder-only models (i.e., UniXcoder and CodeBERT). The embeddings for queries and code snippets are generated based on the model category. We set the consistent maximum input length to 512 tokens. This decision is based on a statistical analysis of the entire CodeSearchNet dataset (all six languages) using the UniXcoder tokenizer, which shows that this limit covers 99.46% of queries and 92.79% of code snippets.

For decoder-only LLMs, we employ mean pooling on the last hidden state to derive the sequence representation, a common practice for this architecture [40]. As illustrated in Figure 1 (b), the process begins by concatenating an instruction with the input (query or code snippet) and applying left-padding. The entire sequence is then processed by the transformer layers. Finally, we compute the element-wise mean of all token embeddings from the last hidden state (excluding padding tokens) to produce the final sequence embedding. Across all experiments, we prepend each query with the following consistent instruction:” Given a code search query, retrieve relevant passages that answer the query:". In contrast, code snippets are processed directly without any prefix. We adopt this approach of using a fixed instruction based on prior work, which indicates that model performance is not significantly sensitive to minor variations in [39].

For encoder-only models such as UniXcoder and CodeBERT, we use the standard approach of selecting the [CLS] token’s embedding as the representation for the entire sequence [9], [10]. Figure 1 (a) depicts this process. The input sequence is formatted with a special [CLS] token at the beginning and right-padding at the end. After the model’s transformer layers process the full sequence, the resulting output vector at the [CLS] position—which has aggregated information from the entire sequence—is taken as the final embedding.

Once the embeddings for the queries and code snippets are generated, we compute the cosine similarity between the query embedding and the embeddings of all code snippets in the dataset. The code snippets are then ranked based on their cosine similarity scores to the query. We use MRR and MAP (see Section 3.2) as our primary evaluation metrics, based on the ranking of ground truths for each query. MRR evaluates the effectiveness of retrieving relevant code snippets early in the ranking, while MAP provides a comprehensive measure of search performance by considering the precision of relevant results across the entire ranking. For the CSN dataset, which contains a single ground truth per query, we calculated MRR. Since CSN includes multiple programming languages, we also evaluated performance across different programming languages. For the dataset, we calculated both MRR and MAP, as it includes multiple ground truths per query. For both CSN and , we only considered the top-1000 results when calculating MRR and MAP. Our evaluation approach follows the methodologies outlined in the official repositories of the CSN [31] and [41] benchmarks.

4.2 Results↩︎

Table 3 presents the zero-shot performance of various models on the CSN and datasets. Bold indicates the best performance within each model category.

As shown in Table 3, UniXcoder achieves the highest zero-shot code search performance across multiple programming languages. It attains an average MRR of 0.706 on the CSN dataset, as expected, since UniXcoder was pre-trained on CSN and fine-tuned on NL-PL pairs. It also outperforms other models on , with MAP and MRR scores of 0.17214 and 0.21065, respectively.

UniXcoder’s strong performance is due to its effective code fragment embeddings, learned through multi-modal contrastive learning (MCL) and cross-modal generation (CMG) [10], making it well-suited for code search tasks. However, its zero-shot performance on , which was not part of its pretraining, drops to an MRR of 0.21065. This decline is likely due to differences in query style: CSN queries are derived from method docstrings, while queries are based on web search queries. These variations in query structure likely contribute to the performance gap across the datasets.

As shown in Table 3, UniXcoder, an encoder-only model, outperforms all decoder-only LLMs by a significant margin. Qwen2.5-Coder, the best-performing decoder-only LLM, achieves the highest performance on both CSN and . However, even Qwen2.5-Coder, which excels in general and code tasks, performs worse than UniXcoder on both datasets.

This advantage is expected, as UniXcoder is designed for code understanding. It uses pretraining tasks to learn semantic embeddings that represent code fragments effectively, allowing for direct similarity calculations. In contrast, decoder-only LLMs, trained for next-token prediction, are not optimized for code and produce embeddings that do not align well with the needs of code search, resulting in suboptimal performance. Springer et al. [39] note that causal attention in decoder-only LLMs hinders their ability to fully capture code context. Similarly, CodeBERT, despite being pretrained on the CSN dataset, struggles with code search tasks in zero-shot settings due to its embeddings not being fine-tuned for the task during pretraining [9]. On the other hand, because decoder-only LLMs benefit from large parameter sizes and extensive pretraining, their performance on code search in zero-shot settings is better than that of CodeBERT.

It is often assumed that code LLMs should outperform general LLMs in code search tasks because they are specifically designed to enhance the model of ability to understand code. However, the results show that code LLMs do not always perform better than their general LLM counterparts. For instance, Llama2 and benefited from additional training, which improved their performance on code search tasks. Conversely, the performance of Gemma and Mistral slightly decreased with additional training. This decline may be due to the fact that while additional training improved the models’ understanding of code, it may have compromised their ability to effectively understand and process search queries, which are crucial for code search tasks. Successful code search requires a strong understanding of both code and natural language queries. It is worth noting that the top two performers, Qwen2.5-Coder and Llama3, are the latest models evaluated. This suggests that continuous advancements in training data, scale, and architectural techniques are the primary drivers of state-of-the-art performance, rather than the model’s categorical label alone.

5 RQ2: Fine-tuning Improvement↩︎

In this section, we explore whether fine-tuning can help decoder-only LLMs bridge the gap between their pre-trained representations and the specific requirements of code search tasks. Specifically, we investigate how fine-tuning on the CSN dataset improves performance compared to the zero-shot setting.

5.1 Design↩︎

We fine-tuned the models using the CSN dataset and supervised contrastive learning (SupCon), following common practices [1], [10]. This method minimizes the distance between similar samples and maximizes the distance between different ones, ensuring relevant queries and code snippets are closely represented in the embedding space, while irrelevant ones are placed farther apart. Springer et al. [39] note that causal attention in decoder-only LLMs can hinder sentence-level understanding, leading to poor performance. To address this, we employed a bidirectional attention mechanism and incorporated Masked Next Token Prediction (MNTP) training prior to fine-tuning, as suggested by [40].

As shown in Figure 2, causal attention restricts the model to using information from the current token and its preceding context, limiting its ability to capture global semantic structures. In contrast, the bidirectional attention mechanism [30] enables each token to access both preceding and subsequent contexts, improving the model’s understanding of semantic relationships. MNTP training [30], [56] masks non-terminal tokens and predicts their content, helping the model learn complex syntactic structures and enhancing its compatibility with the bidirectional attention mechanism.

Since CSN only provides positive samples, we constructed negative samples for each query by randomly sampling non-corresponding code snippets from the training dataset, as random negatives have shown robust and acceptable performance in previous work[8], [9], [57]. We used this constructed dataset to conduct contrastive learning for all models.

We trained all decoder-only LLMs, as listed in Table 2, for 1000 steps with a batch size of 64, utilizing 2 NVIDIA A800 80GB GPUs for around 7.5 hours per model. For comparison, we applied the same data and approach to supervised contrastive learning for the CodeBERT and UniXcoder. To ensure fair comparison, we used the same random seed, training data, and hyperparameters for each model.

For the fine-tuning of our decoder-only LLMs, we employed Parameter-Efficient Fine-Tuning (PEFT) using the LoRA technique with a rank (\(\text{LoRA}_{r}\)) of 16 to manage computational demands. The models were optimized using AdamW with a learning rate of 2e-4 and a linear warmup over the first 300 steps. To ensure consistency, we maintained a maximum input length of 512 tokens (as justified in Section IV) and trained each model for 1,000 steps. This duration was primarily determined by our own empirical validation, as we observed that performance on a validation set ceased to improve significantly beyond this point. This protocol is also consistent with approaches in prior work [40]. To ensure computational efficiency, we utilized bfloat16 mixed-precision training, gradient checkpointing, and FlashAttention-2. The complete fine-tuning scripts, datasets, and all hyperparameter configurations are publicly available in our replication package [17].

The fine-tuned models are evaluated using the same metrics and test datasets as described in RQ1 (Section 4.1) to assess their performance.

5.2 Results↩︎

Table 4 presents the code search performance of the studied models after fine-tuning, with decoder-only LLMs sorted by average MRR across different programming languages on CSN.

Compared to the zero-shot setting, all decoder-only LLMs show substantial improvements after fine-tuning, with average MRR on CSN increasing by 453.1% to 727.4%. The improvement is even more pronounced on , a dataset with a style significantly different from the CSN training data. This suggests that fine-tuning enables LLMs to fully leverage their pre-trained code understanding capabilities, aligning their vector representations with the specific needs of code search tasks. By fine-tuning, models adjust their existing understanding of code, narrowing the gap and making them better suited for similarity-based retrieval. Importantly, this improved capability is transferable, indicating that fine-tuned models can generalize to different code search tasks.

The ranking of models after fine-tuning differs significantly from the zero-shot results, highlighting that the benefits of fine-tuning are not uniform across models. In general, code-specific LLMs outperform general-purpose LLMs in code search tasks across all models we evaluate. As shown in Table 4, CodeGemma and Gemma ranked first and second, respectively, on the CSN dataset, followed by DeepSeekCoder and DeepSeekLLM, while Mistral rank third to last. This is noteworthy because Mistral is commonly used as a baseline in information retrieval research [58]. It suggests that the code search domain may have unique characteristics that make models based on the Gemma architecture more effective.

After fine-tuning, decoder-only LLMs outperform encoder-only models on both CSN and . Notably, ’s performance declines post-fine-tuning, likely due to overfitting and the significant style difference between CSN and . This suggests limitations in the generalization of small encoder-only models like .

In contrast, decoder-only LLMs demonstrate notable improvements, with some achieving SOTA performance. For example, the fine-tuned DeepSeekCoder, Gemma, and CodeGemma show improvements of 2.5%, 2.6%, and 4.8% in average MRR on CSN compared to zero-shot UniXcoder. Additionally, CodeGemma exhibits a 40.4% improvement in MAP and a 34.6% improvement in MRR on over zero-shot UniXcoder. This highlights that the larger size and pre-training of decoder-only LLMs enable them to generalize better to unseen datasets like , even without specific training.

Among the models, CodeGemma demonstrates exceptional post-fine-tuning performance, surpassing both Gemma and other models across most metrics on CSN, and while its performance on ranks second, it also remains close to that of Gemma. This suggests that additional code pre-training enhanced its code understanding. Although this initially impacted its search task performance, fine-tuning significantly boosts its capabilities.

5.3 Qualitative Analysis↩︎

A puzzling result from our experiments is that fine-tuning UniXcoder on the CSN dataset, while improving its in-domain performance, leads to a significant performance drop on the benchmark. This is particularly intriguing as the same fine-tuning process significantly boosts the performance of decoder-only models like CodeGemma. To understand the root cause of these divergent outcomes, we conduct a comparative quantitative analysis to measure the change in discriminative power for both models.

Analysis Design. To measure the models’ ability to distinguish relevant from irrelevant code, we design an analysis on the benchmark. For each query in the benchmark, we define its corresponding code snippet as the positive sample (q, c\(^+\)) and a code snippet randomly sampled from the rest of the benchmark as a negative sample (q, c\(^-\)). We then compute the cosine similarity for a large population of positive and negative pairs for both UniXcoder and CodeGemma, before and after fine-tuning.

The results of this analysis are visualized in Figure 3 , which plots the density histograms of positive (Sim(q, c\(^+\))) and negative (Sim(q, c\(^-\))) samples similarities. To quantify the separation, we also compute the mean difference between positive and negative similarity scores.

Before fine-tuning (Figures 3a), UniXcoder exhibits strong discriminative power. The positive and negative distributions are visibly separated, with the positive pairs (yellow) centered at a higher similarity than the negative pairs (red). At this stage, the model has a mean similarity difference of 0.251. However, after fine-tuning on CSN (Figures 3b), the model’s ability to handle this out-of-domain (OOD) task appears to degrade. The two distributions move noticeably closer, with a larger area of overlap. This is confirmed quantitatively, as the mean similarity difference drops to 0.206. This objectively demonstrates that fine-tuning on CSN may have diminished UniXcoder’s ability to discriminate between relevant and irrelevant code. This result suggests that the model has likely overfitted to the in-domain CSN data, which could cause it to lose its discriminative power for the more diverse, OOD queries.

In contrast, CodeGemma shows a contrasting behavior. Before fine-tuning (Figures 3c), the base CodeGemma model has almost no discriminative power; the positive (orange) and negative (blue) distributions are nearly identical, with a negligible mean difference of 0.012. This helps explain its poor initial performance. However, after fine-tuning (Figures 3d), the model learns to organize the space effectively. The distribution for positive pairs (blue) shifts clearly to the right (higher similarity), while the negative distribution (orange) remains centered at a lower similarity. This creates a distinct separation, and the mean difference increases substantially from 0.012 to 0.201.

6 RQ3: Improvement Analysis↩︎

In this RQ, we explore the factors driving performance gains from fine-tuning decoder-only LLMs in code search tasks, focusing on four key aspects: training method, training data, model size, and query and code length.

6.1 Training Method↩︎

To understand the impact of training methods, we compare supervised contrastive learning, as used in RQ2 (refer to Section 5.1), with unsupervised contrastive learning, specifically SimCSE [59], and analyze their effectiveness.

SimCSE, an unsupervised contrastive learning approach, uses dropout masks to generate independent samples of the same input sentence. It aims to maximize the similarity between these samples while minimizing similarity with other sentences. We follow the approach in [40] and use the Wikitext-103 dataset for training. As Wikipedia data is included in the pre-training of all models, it does not impart new knowledge. For a fair comparison, models are adapted to a bidirectional attention mechanism and undergo MNTP training, similar to the setup in Section 5.1.

We select CodeGemma, Llama3, and Mistral for comparison. CodeGemma is chosen for its top test results, Llama3 for being the latest model, and Mistral for being the most widely used decoder-only language model in information retrieval applications [58].

To verify the statistical significance of our findings, we compare the performance of different approaches using the Wilcoxon signed-rank test, a non-parametric paired test. We report a result as statistically significant if the p-value is less than our chosen significance level of \(\alpha = 0.05\). To account for multiple comparisons across different models and datasets, we apply the Benjamini-Hochberg (BH) correction to the p-values. In Table 5, results marked with an asterisk (*) meet this significance criterion of \(p < 0.05\).

Table 5 shows the performance comparison of CodeGemma, Llama3, and Mistral on CSN and . The results indicate that supervised contrastive learning significantly outperforms unsupervised contrastive learning (SimCSE) across all three models. However, SimCSE also improves the performance of models comparing to the zero-shot setting in some cases.

To assess the impact of the training method on model performance, we select the top-performing models and focus on the Go language, which exhibits the highest MRR performance for CodeGemma in CSN. We analyze the cosine similarity of vectors generated by model embeddings across three methods: zero-shot, SimCSE, and supervised contrastive learning. Using all CSN test queries in Go, we treat matched code as positive samples and randomly select other code as negative samples. Details of this analysis for the Go language are presented in Fig. 4. In each subplot, the blue histograms represent the cosine similarity between the query and positive samples, while the orange histograms represent the similarity between the query and negative samples. Higher x-axis values correspond to greater similarity, and the height of the bars indicates the frequency of occurrences at each similarity level. Figure 4 visualizes CodeGemma’s ability to differentiate between relevant (positive) and irrelevant (negative) code snippets under our different experimental settings.

The results in Fig.4 reveal that while unsupervised contrastive learning improves performance, it does not significantly enhance the distinction between positive and negative samples. Supervised contrastive learning, however, greatly improves the correlation between queries and positive samples while reducing the correlation with negative samples. Note that we perform the same analysis for other programming languages in CSN, and the trends are consistent. However, due to space limitations, we do not present these results here.

6.2 Training Data↩︎

In this section, we analyze the impact of different training data on model performance. We used the E5 dataset and the CSN dataset. The E5 dataset [38] is a universal query dataset with approximately 1.5 million samples covering 93 languages, making it the most diverse dataset, focusing on general information retrieval tasks rather than code search.

For similar reasons to RQ3.a in Section 6, we select CodeGemma, Llama3, and Mistral for supervised contrastive learning. For a fair comparison, all the training settings are the same for both datasets, as detailed in Section 5.1. The statistical significance of the results is evaluated using the same protocol described in Section 6.1 (Wilcoxon signed-rank test with BH correction).

Table 6 confirms that fine-tuning on the specialized CSN dataset generally leads to a performance advantage over the general-purpose E5 dataset. This trend is most pronounced for CodeGemma, where the improvement is statistically significant across all metrics. However, our statistical analysis reveals that the magnitude of this advantage is highly dependent on the base model and evaluation task. For a powerful model like Llama3, the benefit of specialized data is not statistically significant on the in-domain CSN benchmark, though it remains significant for the more complex task. This suggests that while specialized data is beneficial, its impact can be modulated by the extensive knowledge already present in state-of-the-art pretrained models.

6.3 Single-Language Fine-Tuning↩︎

Our previous analysis establishes that fine-tuning on a specialized dataset yields superior performance. This finding, however, motivates a deeper investigation, as the specialized CSN corpus is itself multilingual. It remains unclear how the specific programming language used for fine-tuning influences a model’s ability to learn generalizable code representations. To explore this, we fine-tune our top-performing model, CodeGemma, on six single-language subsets derived from CSN. All other fine-tuning configurations, as detailed in Section 5.1, are kept identical to ensure a fair comparison.

The results, presented in Table ¿tbl:tab:mults95vs95single?, reveal that multilingual training provides superior generalization, challenging the straightforward assumption that monolingual fine-tuning is always the optimal path for language-specific specialization.

On one hand, the multi-language model demonstrates remarkable generalization and the power of positive knowledge transfer. It achieves a competitive average MRR across all languages on the CSN benchmark and exhibits a commanding, cliff-like performance advantage on the more comprehensive benchmark. This suggests that exposure to a diverse set of languages enables the model to learn more abstract and robust semantic representations of code. Furthermore, the multi-language model surprisingly outperforms single-language models on their own respective test sets for languages like Go, Python, and Ruby, indicating that the collective knowledge transferred from a diverse linguistic pool can provide a more powerful learning signal than specializing on a single language’s dataset alone.

On the other hand, monolingual fine-tuning exhibits significant instability and unpredictable results. Contrary to expectations, a model fine-tuned on a single language does not consistently achieve the best performance on its corresponding test set. In our experiments, the model fine-tuned solely on JavaScript not only achieves the top score on the JavaScript benchmark but also matches the best-performing specialized models on the Java and PHP test sets, all while securing the highest overall average MRR on CSN. This indicates that fine-tuning on JavaScript promotes a deep understanding that extends beyond JavaScript itself to the entire CSN corpus. In contrast, the model trained exclusively on Ruby achieves the best performance of all single-language models on the benchmark.

Furthermore, we observe that the choice of a single language for fine-tuning is a critical factor, as the effectiveness of each language as a data source varies dramatically. For instance, the model fine-tuned solely on JavaScript achieves the highest average MRR of any model, indicating its utility as a strong source for learning general code features. In stark contrast, the model trained only on Python performs poorly across all benchmarks, suggesting that Python’s prevalence in the initial training corpora may have contributed to diminishing returns during fine-tuning, leading to less competitive performance compared to other languages.

While the previous section confirms the benefits of multilingual training, it is crucial to understand how individual languages contribute to this overall success. This leads to a critical question regarding data composition: What is the performance impact of reducing or removing a language from the training mix? We address this through a new experiment that systematically discards data for selected languages from the CSN fine-tuning set.

We choose Java and Ruby for this study to represent two distinct cases: Java, a high-resource language, and Ruby, a comparatively lower-resource language. Apart from the modifications to the training corpus, all other fine-tuning parameters are kept identical to those detailed in Section 5.1.

The results are presented in Table ¿tbl:tab:discarding95language?, where Discard Language indicates the language being removed, and Discard Ratio specifies the proportion of its data that is discarded (0 for complete retention, 1 for complete removal). For clarity and to directly illustrate the impact of data removal, we have simplified the CSN evaluation metrics. On Discarded Lang. shows the model’s performance on the test set of the language being discarded (e.g., the Java test set when Java is the discard language). Avg. on Other Langs. represents the average performance on the remaining languages’ test sets. This presentation intuitively links the act of discarding a language’s data to its direct and indirect performance consequences.

As shown in the table, the baseline configuration (a discard ratio of 0) consistently yields the best performance on the benchmark and strong results on CSN. This reaffirms our earlier conclusion that the complete multilingual dataset is optimal for model generalization. However, a more nuanced, non-linear pattern emerges as we incrementally discard data. We observe that the poorest performance often occurs at intermediate discard ratios, rather than at the extremes. For instance, when discarding Java, the weakest CSN result appears at a ratio of 0.8, while the performance dips lowest at a ratio of 1.0. Similarly, for Ruby, both benchmarks show the worst performance at a 0.5 ratio.

We hypothesize that this may be due to a “critical mass" effect. When the training data for a particular language falls below a certain threshold but is not absent, it may be insufficient to form a coherent representation and is instead treated by the model as statistical”noise." This noise can disrupt the learning process for other languages more significantly than the language’s complete absence (a ratio of 1). Furthermore, the On Discarded Lang. column reveals a crucial distinction tied to pre-training knowledge. For a lower-resource language like Ruby, performance degrades substantially as its specialized fine-tuning data is removed. In contrast, for a high-resource language like Java, the model maintains strong performance even when its fine-tuning data is completely discarded, likely because the model has already been exposed to a vast amount of Java code during its initial pre-training phase.

6.4 Model Size↩︎

To investigate the impact of model size on code search performance, we conduct experiments across two different model families. We fine-tune models from the Llama2 family (using the 1.3B Sheared-LLaMA variant, 7B, and 13B) and the Qwen2.5-Coder family [60] (0.5B, 1.5B, 3B, 7B, and 14B) on the CSN dataset using our supervised contrastive learning setup. An identical training setup is used for all model sizes to ensure a fair comparison (see Section 5.1 for details).

As this analysis involves comparing more than two related groups (i.e., the different model sizes within a family), we employ a two-stage statistical procedure. First, we use the to determine if any statistically significant differences exist among the groups. If the result is significant (\(p < 0.05\)), we then conduct a post-hoc Wilcoxon test with Holm-Bonferroni correction for all pairwise comparisons to identify which specific pairs differed significantly. The significance level (\(\alpha\)) is set to 0.05. In our results tables, an asterisk (*) indicates that a model’s performance is statistically significantly different from that of the best-performing model within the same comparison group (\(p<0.05\)).

The results, summarized in Table 7, reveal a nuanced relationship between model size and performance, challenging the simple assumption that “bigger is always better".

Our key observations are as follows:

• Performance is not monotonic with size. Our results challenge the simple “bigger is better" scaling hypothesis within the context of code search. As shown in Table 7, the Qwen2.5-Coder family exhibits a clear performance peak with the 1.5B variant, which statistically significantly outperforms both its smaller and larger counterparts.

• An optimal size “sweet spot" appears to exist for tested architectures. The Llama2 family shows a mixed trend; specifically, its performance on the in-domain CSN dataset peaks at the 7B model, while the smallest 1.3B model achieves the best results on the out-of-domain benchmark. This, combined with the clear peak for Qwen2.5-Coder, suggests that for the model architectures we investigated, an optimal model size may exist for this task, beyond which performance can degrade.

• Model architecture remains a critical factor. The Qwen2.5-Coder architecture demonstrates a clear performance advantage, as its 1.5B model achieves substantially higher scores than any model from the Llama2 family on all metrics we tested.

6.5 Query and Code Length↩︎

We explore whether different model architectures influence a model’s preference between queries and code. We analyze this from two perspectives: the length of the query and code, and cases where the models match exactly on queries, i.e., top-1 ground truth hitting. We examine and CodeGemma after CSN fine-tuning, referring to them as and CodeGemma.

Some previous works have suggested that the past poor performance of decoder-only models in query results is due to the high-dimensional vector outputs causing curse of dimensionality which means that the data becomes sparse, making it difficult for the model to learn meaningful patterns [61]. This leads us to investigate the token length of queries and code encoded by the models. We analyze the query accuracy of and CodeGemma tokens in different length intervals. As shown in Fig 3 (a), both models perform poorly on ultra-short queries (queries with fewer than 10 tokens), but CodeGemma performs better on longer queries. This suggests that the larger size of the decoder-only model may lead to a better understanding of long queries.

We analyze the difference in perfectly matching queries between and CodeGemma, finding that 36% of the results matched perfectly only in CodeGemma.

However, this alone does not indicate bias. We further analyze the results in for these queries that only match perfectly in CodeGemma. Fig. 3 (b) shows that the matching code often ranks second in the corresponding query in . In fact, 82% of queries that match perfectly only in CodeGemma rank within the top 10 for the corresponding query in . This indicates that while the performance of decoder-only models is better than that of Encoder-only models, there is no inherent “preference" in the embedding vector. The improvement is more about the model’s enhanced understanding of code and search.

We analyze the search results based on CodeGemma and find that ultra-short queries (less than 10 tokens) often match longer code snippets. Fig. 3 (c) shows that for ultra-short Go language queries with code tokens less than 50, 93% of the top 10 matched codes are significantly longer, averaging 111 more tokens. This suggests that short query-short code combinations are weak spots for LLMs. Our case analysis identifies two main reasons for poor performance in ultra-short queries: 1. Sparse embedding vectors due to few tokens, leading to curse of dimensionality; 2. Low-quality matching results due to lack of context and unclear semantics.

To test the first reason, we conduct a small experiment by repeating queries to double their token length without changing semantics. We test ultra-short Go language queries on CSN using Llama3. Fig. [fig:repeated95Q] shows that this method effectively improves the cosine similarity between queries and code embeddings. The results of the repeated queries shift to the right, indicating that the high-dimensional vectors of the decoder-only model can lead to curse of dimensionality, which can be mitigated by increasing the query length without altering the meaning.

Regarding the second reason, our case study of Go data with fewer than 10 query tokens and poor results find that such ultra-short queries often lack context and clear semantics. Fig. 6 shows an example of such low-quality queries with its top-1 incorrect result. We observe that ultra-short queries tend to favor longer code snippets due to the higher likelihood of keyword matches. For instance, the top result for this ultra-short query is a code snippet of 80 tokens, while the corresponding code token length is 37.

This issue might be mitigated through query reformulation techniques. By expanding or rephrasing ultra-short queries, we can provide additional context and improve the clarity of the query semantics. This could potentially mitigate the sparse embedding vectors problem and enhance the quality of matching results. For instance, adding contextually relevant terms or rephrasing vague terms can help align the query better with the intended information, leading to more accurate and relevant search results.

7 RQ4: Computational Time Analysis↩︎

The trade-off between model size and computational time is a critical consideration in the design and deployment of code intelligence systems. As shown in RQ2, while decoder-only LLMs can outperform small-scale encoder-only models in terms of performance and generalization on code search tasks after fine-tuning, one potential drawback is the increased computational time due to their larger number of parameters. This could affect their practical usability.

In this research question, we focus on investigating the computational time required for decoder-only LLMs in comparison to small-scale encoder-only models when applied to code search tasks.

7.1 Design↩︎

To investigate the trade-off between model size and computational time, we compare two models with significantly different parameter sizes: UniXcoder (125M parameters) and CodeGemma (7B parameters).

The typical workflow for applying these models includes the following steps:

• Fine-tuning the model on the specific task (offline).

• Using the model to convert all code snippets in a codebase into code embeddings (offline).

• Indexing the code embeddings and creating an index using tools such as FAISS² or Milvus³ (offline).

• Converting the query into the query embedding using the model (online).

• Searching the top-k most relevant code snippets using the query embedding from the index (online).

For the purpose of this study, we focus on comparing the per-query embedding time, per-code embedding time, and fine-tuning time. Other steps in the workflow, such as indexing and search, are not directly influenced by the models themselves and are therefore excluded from our analysis. Both models are trained and evaluated using the CSN benchmark, consistent with the settings described in RQ2 (Section 5.1). The per-query and per-code embedding times are calculated by averaging the total embedding time across all queries and code snippets, respectively.

7.2 Results↩︎

Table 8 compares the computational costs of UniXcoder and CodeGemma across three key stages: fine-tuning, per-query embedding, and per-code embedding. The results highlight notable differences in computational demands, underscoring the trade-offs between model size and efficiency.

In the fine-tuning phase, the time required for CodeGemma is significantly higher than for UniXcoder, with a 98.64× increase. However, this increase is an acceptable trade-off, as fine-tuning is an offline process and does not need to be repeated frequently. The performance and generalization improvements that come with using a larger decoder-only LLM like CodeGemma justify the additional fine-tuning time. Moreover, once fine-tuned, these models can be reused across various domains without the need for retraining, making the long fine-tuning time less of an issue in practical scenarios.

For per-query embedding, CodeGemma takes 14.7 ms, which is 4.1× longer than UniXcoder’s 3.6 ms. While this increase in embedding time is evident, it is important to note that the per-query embedding phase is relatively lightweight compared to the other stages, and thus the increased computational cost remains manageable. Additionally, this increase in time is still justifiable when considering the superior performance that CodeGemma offers, especially in terms of generalization on code search tasks. As the query embedding occurs in the online phase, where embeddings are computed in real-time as queries are issued, the additional time cost is minimal relative to the benefits.

The per-code embedding time for CodeGemma (32.6 ms) is substantially longer than UniXcoder (4.2 ms), reflecting a 7.8× increase. This difference is a direct consequence of CodeGemma’s significantly larger and more complex architecture, featuring more layers and a wider hidden state than UniXcoder. Although both models process inputs truncated to the same 512-token length, CodeGemma’s deeper architecture requires more computation per token. However, this cost is acceptable for two main reasons. First, code embeddings are computed offline and typically only once for a given codebase. Second, techniques such as data parallelism can be employed to significantly reduce the overall wall-clock time required for embedding a large codebase with sufficient hardware. Thus, while the per-code embedding time is longer for CodeGemma, it is a reasonable cost when considering its one-time calculation and its potential for significant performance improvements.

In summary, while CodeGemma’s larger model size results in longer computational times than UniXcoder, these costs remain manageable. The increased fine-tuning time is a one-time expense justified by improved performance and generalization, while the per-query and per-code embedding times, though longer, are reasonable given that embeddings are computed only once. The longer embedding times are acceptable given that code indexing is an offline process that can be expedited with parallelization techniques.

8 RQ5: Training Efficiency↩︎

A key assumption behind the use of decoder-only LLMs is that their pretraining on large corpora and with numerous parameters has endowed them with a robust understanding of code. Consequently, these models may require fewer training samples to transfer their pre-existing knowledge to new tasks. To test this assumption, we compare the training efficiency and performance of decoder-only LLMs against smaller encoder-only models during fine-tuning on code search tasks.

8.1 Design↩︎

As in previous RQs, we compare CodeGemma (a decoder-only LLM) with UniXcoder, a leading encoder-only model. Both models are trained on the CSN training dataset, following the same training settings as described in RQ2 (see Section 5). We save the model checkpoints at regular intervals (every 200 training steps) and evaluate performance on the CSN test dataset as well as the dataset to assess generalization. This comparison helps us understand how quickly each model can leverage available training data and how performance evolves as training progresses.

8.2 Results↩︎

Figure 7 and Figure [fig:finetune95step95COSQA95MAP] illustrate the performance evaluations of the models at every 200 training steps (from 0 to 1000) on the CSN and datasets, respectively. Our experiments reveal that UniXcoder shows a slight decline in performance between 200 and 800 training steps on the CSN dataset (see Figure 7), which suggests potential overfitting. This indicates that the model may rely too heavily on specific patterns from the limited fine-tuning samples, hindering its ability to generalize effectively. A similar trend is observed on the dataset (Figure [fig:finetune95step95COSQA95MAP]), where UniXcoder’s performance plateaus after fine-tuning, with minimal improvement in adapting to new query styles and diverse tasks.

In contrast, CodeGemma shows significant improvements even after just 200 training steps (Figure 7 and [fig:finetune95step95COSQA95MAP]). As training progresses, CodeGemma continues to demonstrate consistent performance gains on both CSN and , indicating that larger models like CodeGemma can better generalize from limited data, likely due to their larger pre-trained models and greater representational capacity. These results suggest that decoder-only LLMs, particularly larger ones like CodeGemma, can leverage contrastive learning with fewer samples to activate the knowledge learned during pre-training. This capability allows them to quickly improve performance on new tasks with minimal fine-tuning, demonstrating greater training efficiency.

These findings emphasize the advantages of decoder-only LLMs in terms of training efficiency and generalization. Thanks to their pretraining on large corpora and their larger parameter sizes, these models require fewer task-specific examples and adapt more quickly to new tasks. This makes them highly efficient for fine-tuning, particularly in code search applications, where data is often limited.

9 Related work↩︎

9.1 Broader Applications of Decoder-Only LLMs↩︎

Recent research has begun to adapt decoder-only LLMs for various representation learning tasks, often achieving significant improvements. For example, in sequential recommendation, several frameworks like PAD [62] and LLM2Rec [63] have been proposed to enhance traditional recommenders with LLM embeddings, aiming to solve cold-start problems or blend semantic and collaborative data. In multilingual embedding, LUSIFER presents a zero-shot approach to adapt English-centric LLM embedders for multilingual tasks [64].

While these studies expand the application of decoder-only LLMs in related embedding-centric domains, their specific application and fine-tuning behavior for the code search task remain largely unexplored.

9.2 Empirical Studies in Information Retrieval↩︎

Concurrently, a growing body of empirical research has begun to dissect the performance, methodologies, and new challenges of LLMs in text retrieval. A recent comprehensive survey systematically overviews this paradigm shift from encoders to decoders for embedding generation, covering data, architecture, and pooling strategies [65]. Meanwhile, analytical studies have begun to explore the underlying model behaviors. For example, a recent analytical study [66] investigates how enabling bidirectional attention impacts the semantic representations of decoder-only LLMs for embedding tasks. Other studies analyze different facets, such as the safety risks of instruction-following retrievers [67].

To the best of our knowledge, there is still a significant gap in the literature regarding the systematic, empirical application of decoder-only LLMs to code search tasks. Our work addresses this gap by providing insights into model selection, data, and sizing for this specific domain.

10 Threats to Validity↩︎

One potential threat to internal validity is the selection of hyperparameters, which can significantly impact model performance. To address this, we followed the hyperparameter settings recommended in [40] for all model fine-tuning, ensuring consistency and reliability in our evaluations. Furthermore, the pretraining datasets for decoder-only LLMs may contain CSN data, which could undermine the reliability of model evaluations. To mitigate this concern, we also tested the models on a more recent dataset, , ensuring that the models had not been exposed to the relevant knowledge during pretraining. This additional measure helps ensure that the evaluation results reflect the model’s true capabilities rather than any bias introduced by prior exposure to the dataset.

The study used eleven decoder-only LLMs, which might limit generalizability. To address this, we selected models that rank high in code generation on MBPP [68] and have similar sizes, ensuring that our models are SOTA and representative. We prioritized using official models, and if unavailable, we chose those with high downloads and comprehensive evaluations to ensure model reliability.

In the analysis of RQ3, different aspects face distinct validity threats. For 6.1, our qualitative analysis presents results primarily for the Go language. This might limit the generalizability of these specific visual findings. However, we mitigate this by performing the same analysis on all other CSN languages and confirming that they all exhibit similar trends. For 6.2 and 6.4, our analyses are limited to the selected datasets and model families. To mitigate this threat, we employ rigorous statistical tests to ensure our findings are statistically significant.

Furthermore, the analyses for RQ4 and RQ5 were conducted exclusively on CodeGemma, our top-performing 7B model. This single-model analysis limits the generalizability of our findings on computational time and training efficiency, as these metrics are sensitive to model-specific factors like architecture, pre-training data, and hyperparameters.

11 discussion↩︎

In the session, we discuss some practical implications for researchers and developers. In addition, we discuss potential research directions for future work.

11.1 Practical Implications↩︎

Based on our systematic evaluation, we offer the following recommendations for those looking to apply language models to code search tasks:

For Zero-Shot Scenarios: When fine-tuning is not feasible due to resource or time constraints, we recommend using specialized encoder-only models like UniXcoder. Despite its smaller size, it offers a competitive balance of performance and efficiency, outperforming many recent, larger decoder-only LLMs in zero-shot settings.

For Fine-Tuning Scenarios: If resources for fine-tuning are available, we strongly recommend decoder-only LLMs, particularly models from the Gemma and Qwen2.5-Coder families. Our results show that their architectures demonstrate superior understanding and generalization capabilities after fine-tuning, surpassing even other recent and larger models.

For Fine-tuning Method: We recommend using supervised contrastive learning (SupCon). This method is highly effective at improving a model’s ability to distinguish between relevant and irrelevant code snippets, which is the core of the code search task.

For Training Data: Fine-tuning on a specialized code search dataset yields statistically significant improvements over using a general-purpose information retrieval dataset. Training on a multilingual dataset can bring stronger generalization ability and more stable performance compared to using a single-language dataset. Meanwhile, the data composition is critical, as a small amount of data from a specific language may interfere with the model’s training effectiveness, potentially acting as statistical noise.

For Model Size: Model size is not always proportional to performance. Our experiments indicate that a larger model does not guarantee better results for code search. We observe a non-monotonic trend where a mid-sized model (e.g., 1.5B) sometimes outperforms its larger counterparts. This suggests that practitioners may find a better cost-performance balance with moderately-sized models, but should determine the optimal size through targeted experiments.

For Resource Trade-offs: The cost of fine-tuning should be justified. While the offline cost of fine-tuning a large model is significant, it is a one-time investment that can yield a substantial and lasting improvement in the final system’s performance.

11.2 Future Work↩︎

Our findings not only provide practical guidance but also open up several avenues for future research:

• Developing Next-Generation Code Search Datasets: Our study confirms that fine-tuning on specialized code search data (like CSN, which uses code comments as queries) yields superior results compared to general-purpose text. However, the nature of code search is evolving. Modern applications, such as RAG and AI agents, will generate queries that are far more diverse than traditional human-authored comments. This indicates a critical need for new, high-quality, and varied fine-tuning datasets that reflect both complex human intent and AI-generated retrieval queries.

• Accelerating Fine-Tuning without Performance Loss: Although the current cost of full fine-tuning is a one-time investment, it remains substantial. For decoder-only LLMs, increasing the fine-tuning speed while ensuring no degradation in performance would significantly enhance their practical appeal. Future work should systematically evaluate the trade-offs of PEFT methods specifically for the code search task, aiming to optimize both training efficiency and final retrieval accuracy.

• Investigating the Non-Monotonic Scaling Effect: Our discovery that model size does not monotonically improve performance (with a 1.5B parameter model outperforming larger counterparts) warrants deeper investigation. Future research should explore why this occurs. Is it due to optimization difficulties, the nature of the code search task, or catastrophic forgetting during fine-tuning? Identifying the “sweet spot" for model scale in code search could lead to more efficient and effective models.

• Optimizing Data Composition: We note that small amounts of data from specific languages might act as “statistical noise" and hinder training. Future work could quantify this effect more precisely. What is the minimum data threshold required for a language to contribute positively to a multilingual model? Can curriculum learning or data re-weighting schemes mitigate the negative impact of low-resource languages in a multilingual dataset?

12 Conclusion↩︎

This study evaluates eleven state-of-the-art decoder-only LLMs for code search tasks. While these models initially underperform in zero-shot settings due to mismatched code representations, fine-tuning significantly boosts their performance, allowing them to better leverage pre-trained code understanding. Fine-tuned CodeGemma , for instance, achieves SOTA performance on CSN and top-tier performance on , underscoring the importance of specialized pre-training.

Our analysis shows that fine-tuning on code-specific datasets, utilizing supervised contrastive learning, and mid-sized models contribute to performance improvements. However, model architecture remains critical, as larger models do not always guarantee superior results. Decoder-only models excel in long-code searches but struggle with ultra-short queries due to the curse of dimensionality and insufficient context. Although the larger size of these models leads to longer computational times, the costs remain manageable. Moreover, decoder-only LLMs demonstrate superior training efficiency and generalization on limited data compared to smaller encoder-only models.

In summary, this study highlights the potential of fine-tuned decoder-only LLMs for code search tasks, demonstrating enhanced performance and generalization. Our findings offer a comprehensive guide for practitioners on model selection, data strategy, and performance trade-offs, while also highlighting key avenues for future research, such as developing next-generation datasets and investigating the non-monotonic scaling effects we identified.

References↩︎

---

1. This work was supported by National Natural Science Foundation of China (Grant No. 62402113), the General Program of the Natural Science Foundation of Guangdong Province, China (Grant No. 2025A1515011631), and Shenzhen Science and Technology Program (JCYJ20240813150107010).↩︎

2. https://github.com/facebookresearch/faiss↩︎

3. https://milvus.io/↩︎