1 sec:Introduction↩︎

In recent years, the problem of improving code execution efficiency has received sustained attention in the field of software engineering [1]–[6]. Traditional rule-based methods mainly focus on optimizing specific performance bugs, such as redundant loops and repeated computations, and have achieved promising results [7], [8]. However, designing such rules is often costly, and these methods lack adaptability to unseen scenarios [7]–[9]. More recently, new algorithms have been introduced to support code optimization [10], [11], but their performance often depends heavily on specific code structures and the definitions of target functions to be optimized. To overcome these limitations, researchers have explored machine learning and deep learning methods for code optimization [12]–[14]. These methods typically learn optimization heuristics from annotated code corpora and performance measurements, showing the potential to capture patterns beyond manually designed rules. Nevertheless, their effectiveness often depends heavily on specific program representations (e.g., LLVM Intermediate Representation, Abstract Syntax Trees) and carefully crafted training datasets, which are costly and limit the generalizability of these methods across diverse programming tasks [15]. With the rapid advancement of LLMs [16]–[20], which have profoundly influenced the field of software engineering, LLMs have demonstrated remarkable capabilities in code understanding and generation tasks, leading to their widespread adoption in software development and paving the way for LLM-based code optimization [21]. In this context, code efficiency is particularly critical in both research and real-world applications, as the computational efficiency of programs is a fundamental cornerstone of system performance and user satisfaction [22].

In this study, we focus specifically on the execution efficiency of code, which is primarily measured by execution time [23]. To improve the execution efficiency of code and address the limitations of existing approaches, we propose FasterPy, a low-cost and high-efficiency framework for optimizing code execution performance. Our work targets function-level optimization, aiming to improve the runtime performance of Python code without altering their semantics. We consider Python code as the target language in this study, as it is widely used in practice, especially among AI engineers, and its runtime overhead makes it a natural candidate for optimization [24]. By combining Retrieval-Augmented Generation (RAG) with Low-Rank Adaptation (LoRA) fine-tuning, the FasterPy framework adaptively enhances LLMs for code efficiency optimization tasks. It consists of two main components: (1) A Code Efficiency Optimization Knowledge Base, which stores low-efficiency (slow) code snippets along with their corresponding optimization suggestions. This component enables semantic similarity matching based on the input code and provides relevant suggestions as references for code improvement. (2) A Fine-Tuned Code Generation Model, obtained by applying LoRA fine-tuning to an existing LLM, which automatically generates an optimized version of the code based on the input code and the optimization suggestions retrieved in (1). These two components form a feedback loop system from semantic retrieval to efficiency-oriented code generation.

Our main contributions are twofold: (1) A Code Efficiency Optimization Knowledge Base. We constructed a retrieval-oriented knowledge base grounded in the Mercury [25] and PIE [3] benchmarks. Beyond integrating optimization instances from the two datasets, we augment them with LLM-generated summaries that capture the underlying optimization strategy and code transformation intent. We further reorganize the knowledge base into a unified retrieval-oriented structure, enabling effective retrieval of relevant optimization patterns and providing diverse optimization cases to support retrieval-augmented LLM-based code optimization. (2) A Code Execution Efficiency Optimization Framework. We proposed FasterPy, a cost-effective and efficient framework for optimizing code execution performance using LLMs. Designed with practicality in mind, FasterPy supports code optimization under standard hardware settings, without relying on specialized or high-cost computational resources. The framework’s components, including the optimization knowledge base and the fine-tuned model, are modular and extensible, making it potentially adaptable to other programming languages and promising for broader adoption. Preliminary results on C++ provide initial evidence of this potential.

Rather than introducing a new LLM architecture or fine-tuning paradigm, this work focuses on the practical challenge of code execution efficiency optimization. To this end, we construct a retrieval-oriented optimization knowledge base and integrate retrieval augmentation with parameter-efficient model adaptation, resulting in an effective and practical framework for LLM-based code optimization under standard hardware settings.

Organization: The rest of this paper is structured as follows: Section 2 presents the relevant background and covers the main concepts used in this study. Section 3 presents the related work. Section 4 provides a detailed description of FasterPy, including its core modules and implementation methods. Section 5 describes the experimental design, including the data processing pipeline, the experimental setup, and the improvements made to the PIE benchmarking program. Section 6 presents the evaluation methodology, benchmarks, metrics, and results. Section 7 presents the ablation studies, including component ablation, retrieved context representation ablation, and dataset ablation. The findings are further analyzed and discussed in Section 8. The potential threats to validity are clarified in Section 9. Finally, Section 10 concludes this work with future work directions.

2 sec:Background↩︎

This section provides the necessary background for our work, focusing on two key techniques: Retrieval-Augmented Generation (RAG), introduced in Section 2.1, which enhances model outputs by incorporating external knowledge, and Low-Rank Adaptation (LoRA), described in Section 2.2, a parameter-efficient fine-tuning method that adapts LLMs to specific tasks.

2.1 sec:Retrieval-Augmented32Generation↩︎

Retrieval-Augmented Generation (RAG) was first introduced by Lewis et al. [26]. This technique integrates LLMs with external information retrieval systems, incorporating scalable knowledge sources during the text generation process to enhance model performance on knowledge-intensive tasks. The retrieval mechanism effectively extends the model’s context window, making the generated content more targeted and contextually consistent. In knowledge-intensive tasks, RAG systems often outperform models that rely solely on internal knowledge [27]–[29].

To address the challenge that models struggle to effectively perceive efficiency differences in the task of code efficiency optimization, this paper adopts the principles of the RAG framework. Specifically, inefficient code inputs are first transformed into vector embeddings, which are then used to retrieve semantically relevant code samples and optimization suggestions from a vector database. These retrieved results serve as external knowledge support. By providing both the retrieval results and the original input to the LLM, the system significantly enhances the model’s understanding and generation capability for code efficiency optimization.

2.2 sec:Low-Rank32Adaptation↩︎

Parameter-efficient fine-tuning (PEFT) is a class of methods that adapts pretrained models to downstream tasks by introducing and training a small number of additional parameters while keeping the majority of the original parameters frozen [30]. Among these methods, Low-Rank Adaptation (LoRA) is a representative method proposed by Hu et al. in 2021 [31]. The core idea of LoRA is to insert a low-rank parallel bypass structure into the linear layers of the model. This bypass is constructed as the product of two trainable matrices. During training, the original model parameters remain frozen, and only the bypass matrices are optimized, which significantly reduces training costs. Compared with other fine-tuning methods, LoRA offers the following advantages:

• Preserves model architecture. Unlike Adapter Tuning [32], LoRA preserves the original model architecture, ensuring that inference speed remains unaffected.

• Greater generality. Compared with Prompt Tuning [33] and Prefix Tuning [34], LoRA can be applied to all linear layers rather than being limited to the input layers, offering broader applicability.

• Lower resource requirements. Only part of the parameters is trained, significantly reducing the demand for GPU memory and computational resources.

Therefore, in this work, we adopt LoRA to fine-tune pretrained models, enhancing their understanding of input-output formats specific to code optimization tasks and thereby improving their performance.

3 sec:Related32Work↩︎

In this section, we present the related work in two aspects, i.e., traditional approaches to code optimization (Section 3.1) and LLMs for code optimization (Section 3.2).

3.1 Traditional Approaches to Code Optimization↩︎

Early studies on code optimization primarily focused on rule-based methods, which leverage expert knowledge bases to optimize specific types of performance bugs and have proven effective in practice. Toffola et al. introduced MemoizeIt, which employs dynamic analysis to compare the inputs and outputs of method calls, thereby helping developers identify cases of repeatedly executed computations [7]. MemoizeIt was evaluated on 11 real-world Java programs, achieving speedups ranging from 1.04× to 12.93×. Complementarily, Olivo et al. proposed CLARITY, a static analysis approach that automatically detects redundant traversal bugs [8]. CLARITY was demonstrated on nine open-source Java code bases and achieved performance improvements of at least 2.45× on large inputs.

Optimization algorithms were also introduced into the field of code optimization. Lopez et al. enhanced code performance through the replacement of equivalent mutation operators [11]. Giavrimis et al. employed genetic algorithms to select implementations of abstract data structures in C++ code to improve performance [10]. It achieved up to 27.9% faster runtime, 16.1% lower CPU usage, and 2.7% lower memory usage in preliminary tests on three mainstream C++ libraries. Liu et al. significantly reduced the search space of the phase-ordering problem by modeling source code and performance dependencies between compiler optimization passes and using a clustering algorithm to automatically group these passes into efficient subsequences, ultimately achieving a 22% improvement in runtime performance and a 24% reduction in code size [35].

Recent years have witnessed the emergence of Machine Learning (ML) and Deep Learning (DL)-based methods demonstrating significant potential in code optimization. Deng et al. introduced CompilerDream, a model-based reinforcement learning framework for general code optimization, which learns a “compiler world model” to efficiently train an optimization policy that achieves strong zero-shot generalization and outperforms LLVM’s built-in optimizations on unseen programs [36]. Bendib et al. developed a reinforcement learning environment for the Multi-Level Intermediate Representation (MLIR) compiler, where they trained an agent to automatically optimize code that results in execution time faster than the optimized kernels generated by TensorFlow Just-In-Time compiler. [14]. Lamouri et al. introduced Pearl, a deep reinforcement learning framework that learns general code optimization strategies capable of generalizing to unseen programs, significantly outperforming top-tier compilers like Tiramisu by a geometric mean speedup of 2.02x [13].

3.2 LLMs for Code Optimization↩︎

With the rapid advancement of LLMs, the success of Transformer-based code models such as Codex [37] has opened up new possibilities for automated optimization of code execution efficiency. Garg et al. proposed DeepPERF, a Transformer-based model that is pretrained on mixed English-code data and fine-tuned using commit histories from GitHub repositories [38]. DeepPERF generates performance-enhancing patches for C# programs. Florath et al. utilized interactive human-LLM collaboration with GPT-4 to optimize critical functions in the open-source Python libraries Pillow [39] and Numpy [40], achieving performance speedups of up to 38x [41]. Chen et al. proposed SUPERSONIC, a sequence-to-sequence model fine-tuned from CodeBERT that achieves precise optimization of C/C++ source code by learning to generate patches in a diff format, with its performance on this specific task significantly surpassing GPT-3.5 and GPT-4 [2]. Shypula et al. explored multiple strategies to adapt LLMs for code optimization tasks, including prompt engineering, RAG, and fine-tuning, leveraging edit histories from competitive programming datasets [3]. Rahman et al. proposed Multi-Agent Reactive Code Optimizer (MARCO), which leverages a specialized agent architecture and an adaptive feedback mechanism to optimize high-performance computing (HPC) code [4]. Compared to Claude 3.5 Sonnet, MARCO achieved a 14.6% average runtime reduction. Giavrimis et al. proposed the Artemis AI framework, which collaboratively leverages multiple LLMs and structures the code optimization process into a systematic, search-based, multi-stage workflow [1]. Their approach seeks to find solutions that achieve maximum performance impact with minimal code changes. Artemis AI demonstrated concrete performance improvements on several real-world projects, achieving execution time reductions ranging from 15% to 52%. To overcome the limitation of Artemis where prompts optimized for one model perform poorly on another, Gong et al. proposed a framework named Meta-Prompted Code Optimization (MPCO) [42]. This framework enhances LLM-based code optimization by automatically generating prompts tailored to specific tasks and model characteristics for diverse LLMs, achieving performance improvements of up to 19.06% across five real-world codebases. Zhao et al. proposed a code optimization framework for C/C++ named SemOpt, which combines LLMs with rule-based static analysis [43]. SemOpt achieved performance improvements on test cases in real-world projects ranging from 5.04% to 218.07%, and manual evaluation verified that 89.86% of its generated optimization suggestions improved performance while preserving semantic correctness, demonstrating the practical utility of the method.

3.3 Conclusive Summary↩︎

While the aforementioned approaches have demonstrated promising results in code execution efficiency optimization, several limitations remain. First, rule-based code optimization methods demonstrate the effectiveness of expert-crafted rule libraries in addressing specific types of inefficiencies; however, they suffer from limited generalizability and scalability, and constructing and maintaining these knowledge bases incurs high costs. Second, algorithmic optimization approaches often rely heavily on specific code structures and the precise definition of objective functions, which restricts their applicability across diverse codebases. Third, ML and DL-based methods usually depend on specific program representations and meticulously crafted training datasets, making them costly to develop and difficult to scale. Finally, research on LLM-based code execution efficiency optimization also faces two challenges: (i) many models are tailored to specific programming languages or benchmark settings, limiting their flexibility. For example, DeepPERF [38] is designed for only C# programs; and (ii) some approaches suffer from suboptimal optimization efficiency or incur high computational costs, making them difficult to deploy in resource-constrained environments. For example, MARCO [4] incurs computational overhead through iterative communication between agents and the transmission of intermediate results. These limitations highlight the need for a more flexible and efficient optimization framework.

The components of our proposed FasterPy framework (e.g., the knowledge base and fine-tuned models) are designed to be replaceable and extensible, ensuring strong practicality and flexibility. Moreover, the framework enables efficient optimization without relying on costly hardware resources.

4 sec:Approach↩︎

In this section, we provide a detailed description of our proposed approach FasterPy. We begin with an overview of the entire framework in Section 4.1, outlining its workflow and key modules. The following sections provide detailed descriptions of each step, including the code preprocessing step in Section 4.2, the code embedding step in Section 4.3, the knowledge base retrieval step in Section 4.4, and the code generation step in Section 4.5.

4.1 Overview of FasterPy↩︎

FasterPy integrates a code optimization knowledge base and a LoRA-fine-tuned LLM, which together enable an end-to-end workflow: retrieving optimization guidance from the knowledge base and applying the guidance with the fine-tuned model to generate functionally equivalent code with improved execution efficiency. As shown in Figure 1, our framework begins with the user’s input, which includes a source code file containing inefficient implementations (i.e., slow code) and a set of functions to be optimized, identified by their names (i.e., target functions). In this work, we focus on improving the execution efficiency of Python code. The scope of inefficiency we target includes common inefficiency patterns such as suboptimal algorithmic complexity (e.g., brute-force algorithms), inefficient use of data structures and memory access patterns, unnecessary library imports, manual implementations of operations that could be efficiently performed using built-in functions, and the replacement of high-level optimized constructs (e.g., vectorized operations) with slower explicit loops. The overall workflow of the FasterPy framework consists of the following four steps, each of which is implemented by a dedicated and replaceable module.

Step 1. Code Preprocessing. In the first step, the framework extracts the target functions and their relevant contextual information from the original code (denoted as \(C_{origin}\)), including function bodies, global variables, class definitions, and other invoked functions. Irrelevant function declarations and definitions are removed, resulting in a cleaned code snippet referred to as \(C_{cleaned}\).

Step 2. Code Embedding. In this step, the cleaned code snippet \(C_{cleaned}\) is converted into a semantic representation vector \(E_{cleaned}\) through the code embedding process.

Step 3. Knowledge Base Retrieval. The vector \(E_{cleaned}\) is then fed into the knowledge base retrieval module, which retrieves the top \(m\) semantically similar slow code snippets from a preconstructed optimization knowledge base based on vector similarity. For each retrieved snippet, this step also obtains its associated \(n\) optimization suggestions along with their corresponding performance improvement measurements. It then selects the top \(k\) suggestions that balance semantic relevance and expected optimization benefit, and aggregates them into a suggestion set denoted as \(R_k\).

Step 4. Code Generation. After retrieving the candidate \(k\) suggestions, the framework integrates \(C_{cleaned}\) with the suggestion set \(R_k\) to construct a unified input prompt denoted as \(P\), which incorporates both the target code and the optimization guidance. The prompt \(P\) is then fed into a fine-tuned LLM to generate an optimized version of the code. The output \(C_{optimized}\) marks the end of the optimization pipeline.

4.2 sec:Code32Preprocessing↩︎

In practice, source code often exists within a complex contextual environment. The execution behavior of a function depends not only on its internal structure, but also on external factors such as global variables, class members, and calls to other functions. Consequently, directly providing only the target function as input to the LLM may fail to capture its actual execution semantics, while using the entire source file may introduce excessive irrelevant information. Such noisy or unrelated content can negatively affect both semantic understanding and subsequent optimization quality. Since the effectiveness of our framework ultimately depends on the quality of the input provided to knowledge base retrieval and code generation, an appropriate preprocessing strategy is necessary to preserve execution-relevant information while reducing unnecessary complexity.

Our preprocessing step follows the intuition of human engineers: it retains the body of the target function while preserving as much of its essential context as possible, such as related variables and invoked functions. Meanwhile, Cho et al. have shown that the performance of LLMs tends to degrade as the input sequence length increases [44]. In addition, excessively long or redundant inputs may also trigger hallucinations in LLMs [45], leading to unrealistic or unreasonable optimization outputs. Since our framework ultimately relies on an LLM for code optimization, it is important to limit the input length while retaining essential context. Therefore, beyond preserving the necessary contextual information, one of the key responsibilities of the preprocessing step is to control the complexity and length of the input code.

Based on the above considerations, our preprocessing step proceeds as follows. To illustrate the process, we provide an example of how the code evolves through each step. The original source code is shown in Figure 2 (a).

Step 1.1. Syntactic Parsing: We use Tree-sitter with its Python grammar, tree-sitter-python¹, to perform syntactic parsing on the original source code and construct its Abstract Syntax Tree (AST). As shown in Figure 2 (b), from the AST, we can clearly identify the functions, variables, and their dependencies.

Step 1.2. Context Extraction: Based on the set of target function names, we retain the body of each function along with its relevant variable declarations, class definitions, and all invoked functions, which are identified through the analysis of these functions. As shown in Figure 2 (c), we extracted the target function slow_function along with its relevant context, retaining the variables a, b, and c, as well as the function helper_function_1 that are related to its execution. Additionally, we keep the relevant content in main_function.

Step 1.3. AST-Guided Cleaning: Using line number information from the AST, we build a unified-style diff patch to remove unrelated functions. As shown in Figure 2 (c), we remove the irrelevant content from the target function, such as deleting helper_function_2 since it is not called.

Step 1.4. Code Formatting: We apply the Black formatter [46] to the retained code, producing a well-structured and human-readable standardized input text. As shown in Figure 2 (d), after applying the Black formatter, the indentation and spacing are in accordance with the standard, and unnecessary blank lines are removed.

The preprocessing step not only reduces the length of the input code and helps the LLM process the code more effectively, but also preserves as much semantic information as possible that is relevant to the target functions. This sets a solid foundation for the efficient operation of the subsequent embedding and optimization steps. The complete preprocessing step is detailed in Algorithm 3.

4.3 sec:Code32Embedding↩︎

The effectiveness of RAG largely depends on the ability to accurately retrieve highly relevant information from the knowledge base. In FasterPy, the core of this mechanism lies in whether the system can, based on the cleaned code snippet \(C_{cleaned}\), successfully retrieve the \(m\) most semantically similar slow code snippets and obtain their associated optimization suggestions and performance improvement measurements. Only when semantic alignment is ensured during the retrieval step can the subsequent code generation step be grounded in meaningful and effective references.

To achieve the above objective, this work adopts vector-based retrieval in place of traditional string matching. Although string-based exact matching can be efficient in certain scenarios, it often fails to capture code snippets that are semantically similar but differ in the syntax level, such as variable or function names [47]. This limitation can significantly degrade the quality of knowledge base retrieval. Therefore, we represent code in a distributed vector space and use vector similarity as the retrieval metric, enabling stronger semantic generalization during the retrieval process.

We adopt the Encoder-Only mode of the UniXcoder [48] (a multimodal code representation model) to obtain the vector representation of the cleaned code \(C_{cleaned}\), denoted as \(E_{code}\). The \(E_{code}\) is then normalized using L2 normalization [49], as shown in Equation 1 , to facilitate subsequent calculations of the inner product. The normalized embedding is denoted as \(E_{cleaned}\). Compared to purely text-based encoders, UniXcoder offers two major advantages: (1) Multimodal input: During encoding, UniXcoder incorporates not only the raw code text but also the comment information and the structure of the AST. This enables the model to capture syntactic, semantic, and documentation-level signals simultaneously during embedding, thereby improving the completeness and accuracy of code representations. (2) Sentence-level representation with fixed dimensionality: In the output stage, UniXcoder applies mean pooling over token-level embeddings, producing a fixed-dimensional vector representation for code inputs of varying lengths. \[\hat{\mathbf{E}}=\frac{\mathbf{E}}{\|\mathbf{E}\|^{2}}=\frac{\mathbf{E}}{\sqrt{\sum_{i=1}^{d} E_{i}^{2}}} \label{l2}\tag{1}\]

4.4 sec:Knowledge32Base32Retrieval↩︎

Although LLMs have demonstrated strong capabilities in code-related tasks, relying solely on model parameters may not always provide sufficient guidance for code performance optimization. Therefore, our framework retrieves relevant optimization examples together with their associated optimization suggestions and incorporates them as external knowledge to guide the subsequent optimization process. The core task of the knowledge base retrieval step is to efficiently retrieve the top-\(m\) records from the knowledge base whose slow code snippets are semantically most similar to the normalized vector representation \(E_{cleaned}\), which is produced by the code embedding module. For each retrieved record, the corresponding optimization suggestions and historical performance improvement measurements are also obtained. This retrieval process provides actionable references to build the subsequent optimization prompt and directly influences the final optimization quality.

We adopt Milvus², a vector database, as a retrieval engine to build a knowledge base for code execution efficiency optimization. This database supports high-dimensional vector search and provides Approximate Nearest Neighbor (ANN) capabilities, making it well-suited for large-scale semantic retrieval tasks. The schema of the knowledge base is shown in Table 1. The construction of the knowledge base and its data sources will be detailed in Section 5.3.

During the retrieval step, the input vector \(E_{cleaned}\) is used as the query vector. Milvus employs inner product similarity as the evaluation metric to compute the similarity between the query vector and the vector representations of slow code snippets stored in each record in the database - measuring how close the input code \(C_{cleaned}\) is to each stored slow code in the semantic space. The system returns the top-\(m\) records with the highest similarity scores above a predefined threshold, denoted as the set \(R_m\), in which each record contains its full metadata fields and the associated similarity score.

Since the knowledge base is primarily constructed from online sources, as detailed in Section 5.3, the effectiveness of the associated optimization suggestions can vary significantly. Although the previous retrieval step ensures the semantic relevance of the suggestions, relying solely on semantic similarity for ranking may fail to fully utilize suggestions that are slightly less similar but offer better optimization performance. For example, some slow code snippets with high semantic similarity to the input may correspond to suggestions that perform poorly in practice, while others with lower similarity scores may provide more effective optimizations. To improve the practical utility of the retrieved suggestions, we apply a filtration process to the initial candidate set \(R_m\) as detailed in Algorithm 4. This filtering process considers both semantic similarity and optimization effectiveness as joint criteria.

The filtration process proceeds as follows:

Step 3.1. Retain the most semantically similar record \(R_{closest}\) from the candidate set \(R_m\), with its similarity to \(C_{cleaned}\) denoted as \(S_{closest}\).

Step 3.2. Filter the remaining records by keeping only those that satisfy the condition \(S_i \geq S_{\text{closest}} - \Delta_s\), where \(\Delta_s\) is a predefined similarity tolerance threshold.

Step 3.3. Sort the remaining candidates in descending order according to their performance improvement scores (denoted as \(rate\)), and select the top \(k-1\) suggestions.

Step 3.4. Return the final set of \(k\) suggestions by combining the \(k-1\) selected suggestions with \(R_{closest}\).

This dual-criteria filtration mechanism not only ensures the relevance of the recommended suggestions through a similarity threshold, but also guaranties their practical effectiveness by selecting high-\(rate\) candidates. It enhances the ability of the subsequent generation module to produce efficient code while reducing the risk of hallucinations or performance degradation caused by inappropriate suggestions.

4.5 sec:Code32Generation↩︎

The primary task of the optimization module is to automatically generate an optimized version of the code, denoted as \(C_{optimized}\), with improved execution efficiency. This is achieved based on the cleaned code snippet \(C_{cleaned}\) produced by preprocessing module and the top-\(k\) high-quality optimization suggestions \(R_k\) retrieved from the knowledge base. This process can be divided into three steps: prompt construction, code generation, and post-processing.

Step 4.1. Prompt Construction. The cleaned input code \(C_{cleaned}\) is first combined with the top-\(k\) retrieved suggestions \(R_k\) to form a unified prompt input. The prompt template is shown below. Specifically, the prompt template adopts a role-playing instruction to place the LLM in a code optimization scenario, explicitly defines the objective of reducing runtime while maintaining correctness, and provides retrieved optimization suggestions as optional guidance rather than strict constraints.

It is worth noting that although each record in \(R_k\) typically contains the original slow code snippet, its corresponding optimization suggestion, and a performance score (i.e., \(rate\)), we deliberately exclude the slow code content during prompt construction and retain only the optimization suggestions. This design is based on our empirical findings: when using LLMs with fewer than 10 billion parameters, including both \(C_{cleaned}\) and multiple slow code examples in the prompt, the model can be easily misled. This often results in the generated \(C_{optimized}\) being contaminated with unrelated variable names, function structures, or execution paths from the slow code, leading to semantic inconsistencies or hallucinations. In contrast, providing only natural language suggestions helps the model focus more effectively on the semantic structure and performance bottlenecks of \(C_{cleaned}\), producing more reliable optimization results.

Step 4.2. Code Generation. After the prompt is constructed, it is fed into an LLM that has been explicitly fine-tuned for code execution efficiency optimization tasks. The LLM possesses both contextual understanding and instruction-following capabilities tailored to performance improvement. It leverages natural language suggestions to perform structural reorganization or semantically equivalent substitutions on \(C_{cleaned}\), to generate a more efficient version \(C_{optimized}\). The fine-tuning strategy will be described in detail in Section 5.6.2.

Step 4.3. Post-Processing. Since LLMs often generate mixed-format outputs - including natural language explanations, markdown formatting, and code snippets - extracting a clean and complete version of the optimized code \(C_{optimized}\) is essential for downstream analysis and evaluation. To address this, we design the prompt to follow a structured format (as provided in Step 4.1), instructing the model to enclose the generated code within a Markdown code block (i.e., using triple backticks ```). In the post-processing phase, we apply a regular expression to extract the contents of the code block. For example, given a model output wrapped in triple backticks (``` `````python````` ```), we use a regex such as ``` `````(?:python)?``\``s*([``\``s``\``S]*?)````` ``` to retrieve only the code. The regex enables efficient retrieval of a well-structured and semantically coherent optimized version.

The optimization module not only integrates optimization suggestions and generates improved code but also effectively controls the risk of hallucinations, making it a critical component in ensuring the final optimization quality.

5 sec:Experimental32Design↩︎

In this section, we present the design of the experiments used to prepare and conduct a comprehensive evaluation of FasterPy. We first describe the data collection in Section 5.1 and preprocessing processes in Section 5.2, followed by the construction of the knowledge base that supports optimization tasks in Section 5.3. Next, we detail the preparation of the training and evaluation datasets in Section 5.4 and Section 5.5, respectively, ensuring that the data is suitable and consistent for subsequent experiments. Finally, we introduce the experimental setup in Section 5.6, including experimental environment and fine-tuning parameter settings, as well as improvements made to the PIE benchmarking program in Section 5.7.

5.1 sec:Data32Collection↩︎

To support RAG-enabled optimization, a dataset that contains both code and its associated performance measurements is necessary, as knowledge base retrieval requires performance-aware examples to provide effective optimization guidance. Datasets commonly used for code generation tasks, such as HumanEval [37] and MBPP [50], contain only mappings between natural language descriptions and target code, but lack execution performance metrics. As a result, they fall short of meeting the requirements for code execution efficiency optimization. To this end, this work employs data sources derived from online judge websites (e.g., AIZU [51], AtCoder [52], and Leetcode [53]), which typically record each user submission along with corresponding execution status information (e.g., runtime and resource usage information). Such data provide performance measurements that enable the retrieval of effective optimization examples for improving code execution efficiency.

We select two representative datasets as our primary data sources: the Python split of the PIE dataset [3] and the Mercury dataset [25]. The PIE dataset is a benchmark designed for code performance optimization tasks, built upon IBM CodeNet. It contains approximately 35,000 code editing pairs, where each pair consists of two implementations for the same programming problem, with one version (typically the post-edit version) showing significantly better execution efficiency than the other. The Python split of PIE has been widely used in studies on code optimization [54]–[57]. Each record in the dataset includes fields such as the submitted code and its runtime, which makes it potentially suitable for fine-tuning LLMs targeting performance improvement. The key fields of the PIE dataset used in our study are shown in Table 2.

The Mercury dataset is primarily designed to train and evaluate the ability of LLMs to generate efficient implementations for code generation tasks. It contains a total of 1,889 Python tasks, each associated with multiple user-submitted solutions. The solutions field of the dataset is a list of solution objects, where each object includes not only the submitted code (in the solution subfield) but also metadata such as execution time and hash values. Other fields in the dataset, such as the original source information of the programming problems, are less relevant to our task and are therefore omitted in subsequent processing. Additionally, the task–submission structure of the Mercury dataset differs slightly from the optimization task addressed in this paper, and the detailed preprocessing steps are described in Section 5.2. The key fields of the Mercury dataset used in our study are shown in Table ¿tbl:tab:mercury95schema?.

It should be noted that these paired code samples are used only during the preparation stage, including optimization knowledge base construction and LoRA fine-tuning. During deployment and inference, the framework requires only the input code to be optimized and does not depend on the availability of corresponding optimized code pairs.

5.2 sec:Data32Preprocessing↩︎

To support the construction of the training set, test set, and optimization knowledge base for the code execution efficiency task addressed in this study, we build a dataset containing performance-improving code pairs and their performance measurement. Specifically, we preprocess and integrate the PIE and Mercury datasets to derive a combined dataset, referred to as \(Optimization-Dataset_{base}\) (\(OD_{base}\)). We strictly follow the official training/test splits of the PIE dataset, and ensure that the construction based on PIE are performed only on the training split, while the test split is reserved exclusively for evaluation. The standardized structure of \(OD_{base}\) contains four core fields: (1) input: low-efficiency code, (2) target: high-efficiency code, (3) summary: a natural language summary describing the optimization from input to target, and (4) rate: the effectiveness score of the optimization.

This section introduces the transformation process from the original datasets - PIE and Mercury - to the target dataset \(OD_{base}\), which consists of two main steps: (1) dataset normalization and alignment, and (2) automatic generation of optimization summaries. The overall pipeline is illustrated in Figure 5.

5.2.1 Dataset Normalization and Alignment↩︎

The original structure of the PIE dataset is close to the target dataset \(OD_{base}\) used for constructing our optimization knowledge base. Therefore, our preprocessing mainly focuses on field selection and semantic transformation. Specifically:

• Redundant fields irrelevant to execution efficiency optimization are removed;

• The following key fields are retained: \(input\) (inefficient code), \(target\) (optimized code), and \(diff\) (unified diff description);

• A new optimization effectiveness field is constructed as: \(rate = cpu\_time\_v0 / cpu\_time\_v1\). This metric measures the runtime improvement of the target implementation relative to the input implementation.

The resulting dataset is referred to as \(PIE_{v2}\), to distinguish it from the original version. The Mercury dataset is organized at the task level, where each record contains a programming task and multiple user-submitted solutions. This structure is evidently inconsistent with the pairwise format required by \(OD_{base}\). To address this, we convert the data into pairwise samples using the following steps:

Step 1. For each task, extract its corresponding \(n\) user submissions.

Step 2. Sort the submissions by execution time in ascending order and designate the fastest submission as \(Submission_{fastest}\), which serves as the reference for optimal performance.

Step 3. Pair each of the remaining \(n-1\) submissions with \(Submission_{fastest}\), resulting in \(n-1\) records. Each sample is structured as follows.

• \(input = Submission_i.solution\)

• \(target = Submission_{fastest}.solution\)

• \(rate = Submission_i.runtime/Submission_{fastest}.runtime\)

Step 4. Use difflib³ to generate a unified-style diff between the input and target, which is recorded as the \(diff\) field.

The processed samples form a new dataset \(Mercury_{v2}\), whose field structure is fully aligned with that of \(PIE_{v2}\) (including \(input\), \(target\), diff, and \(rate\)). Consequently, the two datasets were combined, resulting in a unified intermediate dataset, \(OD_{merged}\), which contains 60k+ samples.

5.2.2 Automatic Generation of Optimization Summaries↩︎

Since we aim to retrieve code optimization suggestions from the knowledge base, after the normalization and alignment of the dataset are completed, we use an LLM to generate optimization summaries that describe performance improvement. The summary is expected to accurately capture the key modifications made from the inefficient implementation (\(input\)) to its optimized counterpart (\(target\)), with a focus on code changes relevant to execution efficiency. Given the scale of our dataset (over 60k+ samples), we employ the Qwen-Max-0125 model [58], which offers a favorable balance between performance and inference cost to generate the optimization summary based on a carefully designed prompt. Qwen-Max-0125 is the most capable model in the Qwen2.5 series, particularly well-suited for complex, multi-step tasks. It demonstrates strong reasoning capabilities and a solid understanding of intricate instructions, achieving promising performance on challenging tasks [59]. The prompt template is shown below.

The LLM-generated natural language summaries are written into the \(summary\) field. Finally, by combining \(OD_{merged}\) with the generated summaries - while discarding the intermediate \(diff\) field — we construct the complete base dataset, \(OD_{base}\), which consists of the following four fields: \(input\), \(target\), \(summary\), and \(rate\), comprising over 60,000 instances.

5.3 sec:Knowledge32Base32Construction↩︎

To enable an efficient retrieval of optimization suggestions based on semantic similarity, we build a knowledge base for code execution efficiency optimization using the preprocessed dataset \(OD_{base}\) (see Section 5.2). The construction of the knowledge base is as follows.

First, for each \(input\) code snippet in \(OD_{base}\), a 768-dimensional vector representation is generated using the code embedding module. The resulting vectors are then appended to the dataset. Next, the augmented \(OD_{base}\), which contains both textual and vector representations, is imported into the knowledge base. The mapping of each field to the knowledge base schema is shown in Table 3.

It is worth noting that, due to the considerations discussed in Section 4.5, the diff and \(target\) fields are intentionally excluded from the knowledge base to avoid introducing potential noise or triggering hallucinations. The final constructed knowledge base supports vector-level Approximate Nearest Neighbor (ANN) similarity search, enabling rapid retrieval of the most semantically similar slow code given an input code snippet. The corresponding natural language optimization summaries are then returned as reference suggestions for the generation model.

5.4 sec:Training32Dataset32Preparation↩︎

The objective of the training step is to enhance the performance of the LLM employed in our framework for the final code generation step, aiming at code execution efficiency optimization with reference to natural language suggestions. Specifically, it focuses on the following two aspects:

• Task Adaptation. Guiding the LLM to understand the contextual prompt structure when optimization suggestions are included in the input, and to learn how to perform performance-oriented code optimization based on natural language instructions.

• Output Standardization. Improving the consistency of the generated outputs in terms of format and correctness, thereby enhancing the LLM’s practicality for optimization tasks.

Based on the above objectives, we randomly sampled 4,000 instances from the \(OD_{base}\) dataset to construct the training set. We adopt this training size as a practical configuration to provide sufficient task-specific examples for model adaptation while controlling training cost and time. This moderate-sized subset allows the model to learn the prompt structure and optimization patterns without requiring full-dataset fine-tuning. The sampled instances are formatted using the prompt template provided in Section 4.5 and formatting the dataset according to the messages structure⁴. The formatting details of the training instances, including the roles and content of each message, are as follows:

• ‘role’: ‘system’, ‘content’: System Prompt

• ‘role’: ‘user’, ‘content’: User Prompt, where the input code is taken from the \(input\) field of \(OD_{base}\), and the suggestions are two optimization suggestions retrieved from the knowledge base based on semantic similarity. We use a fixed number of two retrieved suggestions as a practical design choice to provide relevant optimization guidance while limiting excessive context length and potentially noisy retrieval results.

• ‘role’: ‘assistant’, ‘content’: the optimized version of the input code, i.e., the \(target\) field in \(OD_{base}\).

Furthermore, since both the training set and the knowledge base are derived from \(OD_{base}\), the retrieved suggestions may include the original optimization summary \(summary_i\) corresponding to the current training sample \(input_i\). If not handled properly, it can lead to the model overfitting to that specific suggestion, thereby reducing its ability to generalize to alternative optimization strategies.

To mitigate the risk of over-reliance on retrieved suggestions during training, improve the model’s generalization performance during inference, and reduce the possibility of potential data leakage caused by the trivial reuse of identical optimization pairs, we apply a further filtration step when constructing the training set: suggestions with a similarity score of 1 are explicitly removed. This ensures that the original optimization summary is not reused as training input for the same code snippet, thereby enhancing the model’s generalization capability under diverse suggestion conditions.

5.5 sec:Evaluation32Dataset32Preparation↩︎

We adopt the test dataset from the Python split of the PIE dataset as the basis for model evaluation. It originally contains fields such as \(problem\_id\), \(slow\_code\_col\) (inefficient code), \(reference\_code\_col\) (manual high-performance solution), and \(model\_generated\_potentially\_faster\_code\_col\) (model-generated result). Each task also comes with 3 to 4 test cases with expected outputs, as provided in the dataset, for systematic evaluation of correctness and efficiency.

Notably, some instances in the original test set fail to execute correctly in the current runtime environment due to several issues, including missing external dependencies, version incompatibilities, or I/O errors. To ensure the accuracy and reproducibility of the evaluation, we perform a filtering process on the test set. Specifically, we execute both the inefficient code and the reference implementations for each instance under a standard Python environment, and remove any samples exhibiting missing or incompatible packages, runtime errors, or mismatches between the program output and the expected results defined by the test cases. This process is automated using a Python script with multiprocessing to handle all instances efficiently.

After filtering, a total of 752 test instances that can stably run and produce correct outputs under the current runtime environment are retained to form the final test set used for evaluating the model’s optimization capabilities.

5.6 sec:Experimental32Setup↩︎

5.6.1 sec:Experimental32Environment↩︎

To ensure reproducibility, all experiments were conducted in a stable environment that was consistent and fixed across all examples and setups. Specifically, experiments were conducted on a server running CentOS 7.9.2009 (Core) (64-bit), equipped with dual AMD EPYC 7543 processors (2.8 GHz, 64 cores in total), 256 GB ECC DDR4 3200 MHz memory, and four NVIDIA Tesla A100 (40 GB NVLink) GPUs.

5.6.2 sec:Fine-tuning↩︎

We apply Low-Rank Adaptation (LoRA) to fine-tune LLMs using the ms-swift framework [60]. LoRA introduces trainable low-rank matrices into specific layers while freezing the original model parameters, thereby significantly reducing the number of trainable parameters. Although multiple reasoning-oriented (e.g., Qwen-Max-0125 [58]) and code-specialized models (e.g.,GPT-5.1-Codex [61], CodeLLaMA-7B-Instruct [20], Deepseek Coder-6.7B-Instruct [62], and Qwen2.5-Coder-7B-Instruct [63]) were considered in our evaluation (see Section 6.2), due to hardware limitations, only the latter two open-source models with billions of parameters were fine-tuned in this study. Fine-tuning is conducted on the dataset introduced in Section 5.4, with the goal of improving the model’s capability to generate optimized code with higher execution efficiency, guided by natural language suggestions.

We adopt the ms-swift framework [64] provided by the ModelScope community platform to perform LoRA fine-tuning, with the training hyperparameters listed in Table 4. Specifically, the LoRA rank \(r\) is set to 8, following prior LoRA studies [31], [65], which show that small rank values can provide a favorable trade-off between adaptation capacity and computational efficiency. The training results are shown in Figure 6 (Deepseek Coder-6.7B) and Figure 7 (Qwen2.5-7B), respectively.

5.7 sec:Improved32PIE32Benchmarking32Program↩︎

In the PIE benchmarking program provided by Shypula et al.  [3], each code snippet under evaluation is executed in an isolated environment by launching a separate subprocess. The main program is responsible for recording the runtime and output results. This approach offers several advantages, including isolating external context interference, providing a clean execution environment, and supporting the enforcement of timeouts to terminate abnormal programs, thereby ensuring the stability of the testing process. However, this design also has two limitations, as noted below:

High overhead and low efficiency. Each code snippet requires spawning a new subprocess for execution. However, creating a subprocess and allocating system resources incurs significant overhead, especially when conducting multiple testing iterations on large-scale datasets. As a result, the overall benchmarking time increases substantially.

Inaccurate measurement of execution time. In the original benchmarking program, timestamps are recorded before the subprocess is launched and after it finishes. This measurement includes the overhead of process startup and inter-process communication, leading to an overestimation of the actual execution time of the code. The measurement error is particularly pronounced for code snippets with very short execution time and may even obscure the true performance gains from optimization.

To address the above limitations, we refactor the original PIE benchmarking framework from the two key perspectives, aiming to improve code execution efficiency and enhance the accuracy of runtime measurement.

5.7.1 Execution Mechanism Based on exec()↩︎

Following prior code evaluation benchmarks such as HumanEval [37], EvalPlus [66], and CRUXEval [67], we adopt an exec()-based dynamic execution mechanism in our evaluation pipeline. Our implementation employs a hybrid design that combines a long-lived control subprocess with dynamic code execution using the exec() function in Python. The implementation consists of the following steps:

Step 1. A persistent control subprocess is created at the beginning of the main program, rather than spawning a new subprocess for each code snippet under test.

Step 2. The control subprocess receives the code snippet and the corresponding test input from the main process via a message queue.

Step 3. Before executing each snippet, the control subprocess redirects standard input to provide the test input, and standard output is redirected to a \(StringIO\) object in memory to capture the program output.

Step 4. To prevent the test code from interfering with the control flow, the code is executed within a new thread using exec(code), inside an isolated namespace. This ensures that even if the code contains termination commands (such as sys.exit()), they will only affect the thread but not the entire testing process.

This design maintains execution isolation while significantly reducing the overhead of repeatedly spawning new processes, thereby improving overall testing efficiency.

In our current evaluation setup, all benchmarked code snippets are single-threaded and do not involve asynchronous or parallel execution. Therefore, the adopted isolated execution mechanism is sufficient for accurate and stable measurement. Handling asynchronous or parallelized code execution will be explored in future work.

5.7.2 Runtime Instrumentation↩︎

To further improve the accuracy of execution time measurement, we adopt a runtime instrumentation approach. As illustrated in the timing code snippet below, timestamp recording statements are dynamically inserted at the start and end of the test code. Combined with a try-except-finally structure for exception handling, this ensures that execution time is accurately captured regardless of runtime errors.

5.7.3 Evaluation of Benchmarking Effectiveness↩︎

To validate the effectiveness of the improved benchmarking program, we conduct an evaluation focusing on the stability of runtime measurements. We perform the evaluation on the test dataset from the Python split of the PIE dataset mentioned in Section 5.5, which consists of 752 instances. For each instance, we use the reference code as the evaluation target to ensure functional correctness and eliminate confounding factors introduced by incorrect programs.

To assess measurement stability, we adopt the following protocol:

1. Randomized Execution Order. All test cases are randomly shuffled before each run to mitigate potential bias introduced by execution order (e.g., cache effects).

2. Repeated Measurements. Each benchmarking program executes the full dataset three times independently, producing three runtime measurements for every instance.

3. Recorded Metric. For each instance \(i\), we record the measured runtime \(t\). This results in three measurements \(t_{i1}\), \(t_{i2}\), \(t_{i3}\) per instance.

To comprehensively evaluate the effectiveness of the benchmarking programs, we consider both measurement stability and ranking consistency across repeated runs. Specifically, we adopt standard deviation and Kendall’s \(\tau\) coefficient [68] as evaluation metrics.

To quantify the stability of the benchmarking program, we adopt the standard deviation of runtime across repeated runs. For each instance \(i\), the standard deviation \(\sigma_{i}\) is calculated as shown in Equation 2 , where \(t_{ij}\) denotes the runtime measured in the \(j\)-th run and \(\mu_{i}\) is the mean runtime across the three runs. The calculation of \(\mu_{i}\) is shown in Equation 3 . A lower standard deviation indicates higher stability in runtime measurement.

\[\label{eqn:32runtime32std} \sigma_{i}=\sqrt{\frac{1}{3} \sum_{j=1}^{3}\left(t_{i j}-\mu_{i}\right)^{2}}\tag{2}\]

\[\label{eqn:32runtime32mean} \mu_{i}=\frac{t_{i 1}+t_{i 2}+t_{i 3}}{3}\tag{3}\]

In addition to stability, we further evaluate the consistency of relative performance rankings using Kendall’s \(\tau\) coefficient. Given two runs, Kendall’s \(\tau\) measures the agreement between the pairwise ordering of instances based on their runtime. Formally, it is calculated as shown in Equation 4 , where \(C\) and \(D\) denote the number of concordant and discordant pairs, respectively. Since Kendall’s \(\tau\) is defined between two rankings, we compute it in a pairwise manner across the three runs. Specifically, we calculate Kendall’s \(\tau\) for each pair of runs, i.e., (\(run_1\), \(run_2\)), (\(run_1\), \(run_3\)), and (\(run_2\), \(run_3\)). The final Kendall’s \(\tau\) score is then obtained by averaging over these three pairwise values. A higher value of Kendall’s \(\tau\) (closer to 1) indicates stronger agreement between rankings and thus higher reliability of relative performance comparison. \[\label{eqn:32Kendall8217s32Tau} \tau=\frac{C-D}{C+D}\tag{4}\]

Table 5 summarizes the results of the two benchmarking programs. Our improved benchmarking program achieves a significantly lower average runtime standard deviation of 0.0919, compared to the original PIE benchmarking program’s 0.1731, indicating improved measurement stability. Furthermore, our improved program obtains an average Kendall’s \(\tau\) of 0.9805, while the PIE benchmarking program only achieves -0.0478. This substantial difference shows that the improved program produces highly consistent runtime rankings across repeated runs, whereas the original PIE program yields nearly random rankings.

Based on the above analysis, we conclude that our improved benchmarking program not only stabilizes runtime measurements but also preserves consistent relative performance ordering across runs, suggesting reliable comparative evaluation.

6 sec:Evaluation↩︎

According to the experimental design in Section 5, we conduct a comprehensive evaluation of multiple LLMs on the proposed code execution efficiency optimization task. We first describe the evaluation methodology in Section 6.1. We then introduce the evaluation benchmarks in Section 6.2, followed by the evaluation metrics that quantify performance improvements in Section 6.3. Finally, we present the evaluation results in Section 6.4, highlighting FasterPy’s performance in code execution optimization.

6.1 sec:Evaluation32Method↩︎

To accurately assess the performance of the LLMs in terms of efficiency and correctness under the FasterPy framework, we evaluate three code versions for each problem from the test set mentioned in Section 5.5: \(slow\_code\_col\) (inefficient code), \(reference\_code\_col\) (manual high-performance solution), and \(model\_generated\_potentially\_faster\_code\_col\) (model-generated result), and record runtime and accuracy data. Each code snippet was assigned 3 to 4 test cases (\(test\_case_i\)) sourced from the PIE dataset. Each test case includes a predefined input (\(input_i\)) and the corresponding expected output (\(output_i\)). For every test case, the code was executed four times. To reduce the impact of cold-start effects - such as CPU scheduling, cache loading, and page faults - only the results from the last three executions were recorded, while the first run was excluded.

Runtime. The overall runtime for a code snippet is computed through a two-step process. First, for each individual test case, we determine its stable runtime by averaging the execution time of its last three runs. Then, the recorded runtime is the average of these stable runtimes across all \(n\) test cases of the problem (as shown in Equation 5 ), where \(n\) denotes the number of test cases, and \(Time_{i,j}\) represents the time taken for the \(j\)-th execution on the \(i\)-th test case.

Accuracy. We measure functional correctness using a Line-Level Accuracy score, which is based on comparing the execution results of the code against a ground-truth. Specifically, for each test case, this score is calculated as the percentage of generated lines in the code’s execution output that exactly match the corresponding lines in the expected execution output. The overall accuracy is the average of these scores across all test cases. The detailed computation is shown in Equation 6 , where \(n\) denotes the total number of test cases, \(L_i\) is the number of output lines in \(test\_case_i\), \(Output_{i,j}\) is the model’s output on the \(j\)-th line of \(test\_case_i\), and \(Expected_{i,j}\) is the corresponding expected output line. The indicator function \(\mathbb{I}\)(·) returns 1 if the lines are a perfect match and 0 otherwise.

\[\label{eqn:32runtime} \text{Runtime} =\frac{1}{n} \sum_{i=1}^{n}\left(\frac{1}{3} \sum_{j=2}^{4} \operatorname{Time}_{i, j}\right)\tag{5}\]

\[\label{eqn:32accuracy} \text{ Accuracy }=\frac{1}{n} \sum_{i=1}^{n} \frac{1}{L_{i}} \sum_{j=1}^{L_{i}} \mathbb{I}\left(\text{ Output }_{i, j}=\text{ Expected }_{i, j}\right)\tag{6}\]

6.2 sec:Benchmarks↩︎

To assess the robustness and general applicability of FasterPy, we benchmark it across LLMs of varying scales, ranging from billions to hundreds of billions of parameters. To assess how effectively FasterPy enhances LLMs’ code optimization performance, we evaluate the execution efficiency and functional correctness of the code generated by different LLMs on a unified code optimization dataset. Specifically, the models we select include Deepseek-Coder-6.7B-Instruct [62], Qwen2.5-Coder-7B-Instruct [63], CodeLLaMA-7B-Instruct [20], Qwen-Max-0125 [58], and GPT-5.1-Codex [61]. The five representative LLMs covering both open-source and proprietary systems with varying model scales. This selection enables evaluation across different architectures and parameter scales. Table 6 presents detailed information for each selected model. To ensure a fair comparison, all LLMs are prompted using the prompt template introduced in Section 4.5.

We apply the full FasterPy framework to Deepseek-Coder-6.7B-Instruct and Qwen2.5-Coder-7B-Instruct, including the knowledge base retrieval module (see Section 4.4) and LoRA fine-tuning (see Section 5.6.2). For Qwen-Max-0125 and GPT-5.1-Codex, due to computational constraints, only the knowledge base retrieval module is used in the evaluation. In total, we conduct evaluations and comparisons on the following models and their variants under a unified test set and experimental environment:

• Base models: Deepseek-Coder-6.7B-Instruct, Qwen2.5-Coder-7B-Instruct, CodeLLaMA-7B-Instruct, Qwen-Max-0125, and GPT-5.1-Codex.

• FasterPy-enhanced variants: Deepseek-Coder-6.7B-Instruct-FasterPy, Qwen2.5-Coder-7B-Instruct-FasterPy, Qwen-Max-0125-FasterPy (no LoRA fine-tuning applied), GPT-5.1-Codex-FasterPy (no LoRA fine-tuning applied).

6.3 sec:Evaluation32Metrics↩︎

To comprehensively evaluate the performance of LLMs on the task of optimizing code execution efficiency, we select four metrics. These metrics assess both runtime performance and functional correctness, and are used to systematically evaluate three types of code: the LLM-generated optimized code (\(model\_generated\_potentially\_faster\_code\)), the original slow implementation (\(slow\_code\)), and the human-written efficient implementation (\(reference\_code\)). The specific metrics are as follows:

• Runtime_Mean. This metric measures the average execution time of correctly executed samples, reflecting the overall execution efficiency of the generated code. It is defined as the average runtime across all test samples with an accuracy of 1 (Equation 7 ). \(N\) denotes the number of samples with full functional correctness (i.e., accuracy = 1), and \(Runtime_i\) represents the runtime of the \(i\)-th sample, which is computed as described in Equation 5 .

• Pass@1. This metric is a widely used correctness metric that measures the proportion of samples for which the model-generated code passes all test cases in a single attempt, thereby reflecting the model’s accuracy under one-shot generation (Equation 8 ). \(N\) denotes the total number of evaluated samples. Our evaluation focuses on the quality and execution efficiency of a single generated optimization result, which more closely aligns with the intended usage scenario of FasterPy. Therefore, we focus on Pass@1 rather than sampling-based metrics such as Pass@\(k\) (\(k>1\)).

• @Speedup. This metric measures the extent to which the LLM-generated code improves code execution efficiency compared to the original code (Equation 9 ). \(N\) denotes the number of samples with full functional correctness (i.e., accuracy = 1). \(\text{Runtime}_{i}^{origin}\) represents the runtime of the original code for the \(i\)-th sample, and \(\text{Runtime}{i}^{model}\) represents the runtime of the code for the \(i\)-th sample after being optimized by LLMs.

• Percent Optimized [%Opt]: The percentage of correctly optimized samples (%OPT) measures the proportion of functionally correct samples that are more runtime-efficient (at least 10% faster) than that of the original code (Equation 10 ). \(N\) denotes the total number of evaluated samples, and \(\mathbb{I}\)(·) is the indicator function that returns 1 if the condition is satisfied, and 0 otherwise.

\[\label{eqn:32runtime95mean} \text{Runtime\_Mean} =\frac{1}{N} \sum_{i=1}^{N} \operatorname{Runtime}_i\tag{7}\]

\[\label{eqn:32pass641} \text{Pass@1} =\frac{1}{N} \sum_{i=1}^{N} y_i,\text{ } yi=\mathbb{I}\left(\text{Accuracy}_i=1\right)\tag{8}\]

\[\label{eqn:32speedup} \textcolor{black}{\text{@Speedup} =\frac{1}{N} \sum_{i=1}^{N} \frac{\text{Runtime}_{i}^{origin}}{\text{Runtime}_{i}^{model}}}\tag{9}\]

\[\label{eqn:32OPT} \text{\%OPT} =\frac{1}{N} \sum_{i=1}^{N} \mathbb{I}\left(\text{Accuracy}_i=1 \wedge \text{Runtime}_{i}^{model} \times 1.1<\text{Runtime}_{i}^{origin}\right)\tag{10}\]

All the above metrics are computed only on functionally correct samples, i.e., test cases with an accuracy of 1. This is because if the code generated fails to pass all test cases, it cannot be regarded as a valid optimization regardless of its shorter runtime. Performance improvements are only meaningful when functional correctness is guaranteed.

6.4 sec:Evaluation32Results↩︎

The experimental results are visualized in the heatmap shown in Table 7, illustrating the performance of each model across three dimensions: code generation accuracy, execution efficiency improvement, and generalization of optimization capability. In the heatmap, darker colors indicate better performance. Overall, models integrated with the FasterPy framework consistently demonstrate superior performance across all evaluation metrics, significantly outperforming their original versions and other models of comparable scale.

6.4.1 sec:Code32Generation32Accuracy↩︎

In terms of code generation accuracy, as shown in Table 7, the original billion-parameter models - including Deepseek-Coder-6.7B-Instruct, CodeLLaMA-7B-Instruct, and Qwen2.5-Coder-7B-Instruct - all achieve Pass@1 scores below 70%, indicating their limited performance on the code execution efficiency optimization task. In contrast, after integrating the FasterPy framework, these models exhibit substantial improvements in accuracy. Specifically, Deepseek-Coder-6.7B-Instruct-FasterPy obtains an increase of approximately 37% in Pass@1 compared to its original version, while Qwen2.5-Coder-7B-Instruct-FasterPy receives a gain of around 23%.

These results suggest that billion-parameter LLMs may struggle to capture the structure and requirements of code execution efficiency optimization tasks without targeted adaptation. FasterPy significantly enhances their task-specific performance on code execution efficiency optimization by providing retrieved optimization suggestions, thereby improving the correctness of the generated code.

6.4.2 sec:Optimization32Effectiveness↩︎

In terms of runtime performance, the code generated by the billion-parameter models is generally comparable to, or even slower than, the original slow code. Specifically, Deepseek-Coder-6.7B-Instruct exhibits an average runtime of approximately 23 ms, while CodeLLaMA-7B-Instruct averages around 20 ms - both significantly slower than the input code, which has an average runtime of 16 ms. Qwen2.5-Coder-7B-Instruct performs slightly better, with an average runtime of about 13 ms.

After applying the FasterPy framework, the generated code from all models achieves better execution time than the input code, although they still lag behind the manually optimized reference code, which has an average runtime of 3 ms. Among them, Deepseek-Coder-6.7B-Instruct-FasterPy and Qwen2.5-Coder-7B-Instruct-FasterPy both achieve average runtime of less than 13 ms. In terms of improved code execution efficiency compared to the original code (@Speedup), all original billion-parameter LLMs achieve values below 1.80. This result suggests that current billion-parameter LLMs are unable to effectively perceive and generate performance-optimized implementations. In contrast, all LLMs augmented with the FasterPy framework demonstrate clear acceleration benefits, with @Speedup values exceeding 1.70. Among the billion-parameter LLMs, Qwen2.5-Coder-7B-Instruct-FasterPy achieves the best performance with a Speedup of 1.81, followed by Deepseek-Coder-6.7B-Instruct-FasterPy, which reaches approximately 1.71. The hundreds of billion-parameter models Qwen-Max-0125 and GPT-5.1-Codex, which already exhibit acceleration capabilities in their original versions, still show further improvement after incorporating FasterPy. Specifically, Qwen-Max-0125 improves by about 0.1 to reach approximately 2.14, while GPT-5.1-Codex improves by 0.3 to nearly 2.61.

These findings highlight that billion-parameter models, when not explicitly tailored for the task, exhibit limited sensitivity to code execution efficiency. In contrast, FasterPy provides an efficient solution for improving the adaptability of LLMs to code execution efficiency optimization and enabling practical deployment without reliance on expensive retraining or specialized hardware offering.

6.4.3 sec:Generalization32of32Optimization32Capability↩︎

The %OPT metric measures the proportion of code samples that can be optimized by the model while maintaining functional correctness, thereby reflecting the generalization capability of its optimization ability. As shown in Table 7, the original billion-parameter LLMs, including Deepseek-Coder-6.7B-Instruct, CodeLLaMA-7B-Instruct, and Qwen2.5-Coder-7B-Instruct show limited performance on this metric, with only about 20%~30% of the samples being successfully optimized. After incorporating the FasterPy framework, their performance improves. In particular, Qwen2.5-Coder-7B-Instruct-FasterPy achieves a %OPT of nearly 40%, indicating strong generalization capability in optimization. As for the hundreds of billion-parameter models, Qwen-Max-0125 achieves an improvement of about 10% compared to its original version, while GPT-5.1-Codex achieves an improvement of about 15%.

The improvement in %OPT indicates that FasterPy not only enhances the average optimization effect but also strengthens the model’s ability to identify and generalize across different types of performance bottlenecks, demonstrating strong practical adaptability.

In summary, our proposed FasterPy framework achieves significant improvements across all evaluation metrics, particularly demonstrating strong generalization and practicality in terms of improving code execution efficiency across a wide range of test cases.

6.4.4 Comparison with PIE↩︎

To further evaluate the effectiveness of FasterPy, we compare it with PIE [3], a representative approach that combines retrieval-based few-shot prompting with performance-conditioned generation to improve code efficiency. PIE is selected as a baseline due to its methodological relevance and its compatibility with our experimental setup.

Although prior methods introduced in Section  3 such as MARCO [4], SUPERSONIC [2], DeepPERF [38], and Artemis AI [1] are related, direct comparison is not feasible due to the limited availability of open-source implementations or their lack of support for Python workloads. We therefore include PIE as a closely related and reproducible baseline.

For a fair comparison, PIE is implemented using the same LLMs and evaluated on the same benchmarks as FasterPy. Due to the additional computational overhead introduced by retrieval-based prompting, we conduct the comparison using one representative open-source model (i.e., Deepseek-Coder-6.7B-Instruct [62]) and one representative closed-source model (i.e., GPT-5.1-Codex [61]). All configurations are aligned. The comparison results are shown in Table  8. Overall, FasterPy consistently outperforms PIE across all evaluation metrics. In particular, FasterPy achieves achieves lower Runtime and higher %OPT, Pass@1, and @Speedup, demonstrating its effectiveness in guiding the LLMs toward more efficient Python code optimization.

6.4.5 Cost Effectiveness↩︎

To analyze cost-effectiveness, we report the costs incurred by FasterPy, which originate from two stages: data preprocessing and the code optimization workflow.

Before the code optimization workflow, we employ the closed-source LLM Qwen-Max-0125 to generate optimization suggestions during the data preprocessing stage, as described in Section 5.2. This step incurs a total cost of 24.71$, corresponding to an average cost of 0.39$ per sample. Although this introduces an upfront overhead, the constructed dataset can be reused across code optimization tasks. Therefore, the cost can be amortized and becomes negligible in large-scale or repeated usage scenarios.

During the code optimization workflow, we report token consumption and monetary cost measured over the evaluation dataset. For open-source LLMs (Qwen2.5-Coder-7B-Instruct, Deepseek-Coder-6.7B-Instruct and CodeLLaMA-7B-Instruct), we report token usage as a proxy for computational cost, since their monetary cost depends on hardware and deployment configurations. For closed-source LLMs, we report monetary cost based on publicly available API pricing. Table 9 presents the average number of tokens and monetary cost per task across different models.

As shown in Table 9, FasterPy slightly increases the total token consumption compared to the benchmarks. Despite a marginal increase in token usage, our method consistently achieves better optimization effectiveness, resulting in improved cost-effectiveness in terms of performance per unit cost.

Overall, FasterPy achieves a favorable trade-off between cost and optimization performance. Despite a slight increase in token consumption, the improved optimization effectiveness and the amortized dataset construction cost make it a cost-efficient solution for real-world deployment.

6.4.6 Preliminary Evaluation on C++↩︎

We provide a preliminary evaluation on C++ to investigate whether FasterPy generalizes beyond Python. While our primary focus is Python due to its widespread use and unique optimization challenges in interpreted languages, it is important to understand whether the proposed framework can also benefit compiled languages such as C++.

We conduct experiments on the C++ split of the PIE dataset [3]. The data processing procedure and experimental settings are consistent with those described in Section 5. From the test set, 500 instances were randomly selected for experimentation and evaluation. For evaluation, the generated C++ code was compiled using GNU g++ (GCC) 11.4.1 with the C++17 standard. The experimental results in Table 10 show that FasterPy consistently improves performance over the benchmarks on C++ tasks. Although the magnitude of improvement is smaller compared to Python, the trend remains consistent, indicating that our framework is beneficial across languages.

Overall, these results provide preliminary evidence that FasterPy can generalize beyond Python, while broader evaluation on additional languages is left to future work.

7 sec:Ablation32Study↩︎

7.1 sec:Component32Ablation↩︎

To better understand the contribution of each component in our framework, we conduct an ablation study by isolating the effects of Retrieval-Augmented Generation (RAG) and Low-Rank Adaptation (LoRA). Due to the fact that closed-source LLMs (i.e., GPT-5.1-Codex and Qwen-Max-0125) do not support parameter fine-tuning, we conduct the ablation study on the open-source models introduced in Section 6.2 (i.e., Deepseek-Coder-6.7B-Instruct, Qwen2.5-Coder-7B-Instruct, and CodeLLaMA-7B-Instruct), which allow controlled modification of model parameters.

We include the base model (without RAG and LoRA) and the full framework (with RAG and LoRA) as references to facilitate a direct comparison with the ablated variants. The reported results are consistent with those in Section 6.4. All variants are evaluated under the same experimental settings described in Section 5.6 to ensure a fair comparison. Table 11 presents the performance of different variants on dataset described in Section 5.5.

As shown in Table 11, both RAG and LoRA contribute positively to the overall performance. The full framework consistently outperforms both individual components across all evaluation metrics. This suggests a complementary relationship between retrieval augmentation and fine-tuning. RAG provides high-quality contextual information, while LoRA enables the model to better utilize such information.

Overall, the results demonstrate that both components (RAG and LoRA) are essential, and their combination leads to the best performance.

7.2 Effect of Retrieved Context Representation↩︎

To investigate the impact of retrieved context representation, we conduct an additional ablation study on how retrieved knowledge is incorporated into the prompt introduced in Section 4.5. Specifically, we compare the following three settings:

For fair comparisons, all other settings are kept identical across experiments. We conduct the study on two representative models, including one open-source model DeepSeek-Coder-6.7B-Instruct [62] and one closed-source model GPT-5.1-Codex [61]. The evaluation follows the same metrics described in Section 6.3, including Pass@1, @Speedup, Runtime_Mean, and %OPT. Table 12 presents the comparison results under different retrieved context representations.

The results show that using natural language summaries alone achieves the best overall performance across most evaluation metrics. In particular, the Summary-only setting consistently achieves better Pass@1 and %OPT scores while obtaining lower Runtime Mean with substantially shorter prompts. Although the Summary + Code setting provides additional implementation details, its performance does not surpass the Summary-only setting. This suggests that including retrieved code snippets does not provide additional benefits, as natural language summaries alone already provide sufficient optimization guidance.

7.3 Dataset Ablation↩︎

To evaluate the impact of dataset composition on optimization performance, we conduct an additional ablation study by separately constructing the retrieval knowledge base and fine-tuning dataset using PIE [3] and Mercury [25]. Specifically, we compare three settings: (1) PIE-only, (2) Mercury-only, and (3) the combined PIE + Mercury dataset adopted in our main framework.

Since Mercury is collected from multiple users, its code diffs may contain heterogeneous coding styles, variable renaming, and formatting-related modifications. Such diversity may introduce noisy optimization signals compared with the relatively cleaner transformations in PIE. Therefore, this experiment aims to examine whether Mercury can still provide effective optimization knowledge despite the potential noise.

Following the previous ablation studies on the RAG and LoRA components in Section 7.1, this experiment focuses solely on the influence of dataset composition while keeping all other configurations unchanged. To provide a representative analysis while maintaining manageable experimental costs, we conduct this study on the open-source model Deepseek-Coder-6.7B-Instruct [62] and the closed-source model GPT-5.1-Codex [61]. The experimental results are summarized in Table  13.

The results show that the Mercury-only setting performs slightly worse than PIE-only across most evaluation metrics, which is expected given the presence of noisy edits and heterogeneous coding styles in Mercury. Nevertheless, Mercury-only still consistently achieves positive optimization gains, indicating that the dataset retains useful optimization patterns despite the noise. Furthermore, combining PIE and Mercury consistently yields the best overall performance, suggesting that the two datasets provide complementary optimization knowledge.

These observations demonstrate that the diversity of optimization transformations in Mercury does not significantly degrade optimization quality. Instead, the inclusion of heterogeneous real-world code modifications in Mercury improves the coverage and robustness of optimization suggestions when integrated with PIE.

8 sec:Discussion↩︎

8.1 Interpretation of Results↩︎

The evaluation results presented in Section 6.4 clearly demonstrate that the FasterPy framework significantly improves both the correctness and efficiency of the code across multiple base models. Notably, even instruction-tuned models without any explicit training for efficiency optimization exhibit meaningful performance gains after incorporating FasterPy, suggesting that LLMs possess a latent capacity for performance-aware code generation, which can be elicited through well-designed prompts that incorporate retrieved optimization suggestions.

Interestingly, LLMs with billion-parameter sizes (e.g., 6.7B and 7B) show greater relative improvements than hundreds of billion-parameter models, such as Qwen-Max-0125, possibly due to their weaker inherent optimization ability and greater dependence on external guidance. The consistent improvement in %OPT across all models indicates that FasterPy not only boosts average code execution efficiency but also expands the range of code instances where effective optimization is achieved. This result suggests that retrieving task-relevant optimization knowledge enables better generalization across diverse code inefficiencies.

Compared with traditional approaches such as manually crafted rule-based methods (e.g.,  [7], [8]) or algorithmic optimization approaches (e.g.,  [13], [36]), FasterPy is designed to provide the following two major advantages:

• Lower Cost. Rule-based methods rely on manual summarization and encoding by domain experts, which are costly to develop and difficult to maintain. Algorithmic optimization approaches, on the other hand, require the LLM to possess a deep understanding of code semantics and efficiency bottlenecks, often depending on hundreds of billion-parameter models with significantly higher training and deployment costs.

• Broader Applicability. Manually defined rules and fixed algorithms can cover only a limited range of code execution efficiency optimization scenarios, such as simple loop unrolling or constant folding, and often fail to capture more complex or implicit strategies, such as efficient API substitution or execution path restructuring. In contrast, FasterPy learns from historical optimization examples and leverages a retrieval mechanism to generate more diverse optimization suggestions. In addition, FasterPy’s components are modular and extensible, making our framework potentially adaptable to other programming languages.

Our low-cost and flexible approach FasterPy could be potentially suitable for optimizing code execution efficiency in industrial codebases, especially in environments without specialized computational resources. More complex and higher-cost approaches can be introduced incrementally when initial optimization objectives cannot be fully met.

8.2 Implications↩︎

8.2.1 Toward Automatic Code Optimization↩︎

The experimental results of our study demonstrate that FasterPy, our proposed framework combining RAG and LoRA, exhibits strong performance and generalizability in automatic code efficiency optimization tasks. This finding not only validates the potential of LLMs in enhancing code efficiency but also suggests a broader reflection on their unique value compared to traditional optimization methods and their future trajectory.

Compared to traditional approaches, LLMs offer several distinct advantages: 1) Understanding Code Semantics and Intent: Unlike traditional methods that rely on formal rules, LLMs, through pre-training on vast code corpora, can learn the underlying semantics of code and the developer’s intent [69]. This semantic understanding enables LLMs to identify higher-level optimization opportunities that are often imperceptible to traditional methods, such as refactoring complex code logic or substituting entire algorithms with more efficient alternatives. 2) Moving Beyond Heuristics: LLMs are not bound by fixed heuristic rules. Instead, LLMs learn the correlation between code patterns and performance metrics, which allows the LLMs to potentially discover novel and counter-intuitive optimization strategies, such as eliminating redundant intermediate computations. 3) Handling Ambiguity and Diversity: LLMs can process natural language comments, unstructured code snippets, and even incomplete code. This capability allows them to provide optimization suggestions throughout the entire software development lifecycle, not just during the compilation stage.

With the evolution of model architectures, a key future direction is the integration of static and dynamic information in program execution. This involves empowering models to combine static code features (e.g., Abstract Syntax Trees, Control Flow Graphs) with dynamic features (e.g., runtime performance measurements, profiling information) to form a more holistic understanding of program behavior. Furthermore, with the expansion of the context window, LLMs could gain the ability to analyze and refactor entire repositories. This would enable cross-file and cross-module optimizations at the repository level that go far beyond the current scope of single functions or files.

8.2.2 Leveraging Historical Code Data with LLMs for Code Optimization↩︎

During the construction of the optimization knowledge base and training dataset, this study conducts systematic cleaning and structural processing of the PIE and Mercury datasets. By utilizing the performance-improving code pairs present in the datasets, we summarize code optimization strategies to build the knowledge base. Our approach attempts to leverage Retrieval-Augmented Generation (RAG) to use historical code optimization records to guide future code optimization. The results indicate that historical code optimization records have the potential to guide code optimization efforts. Based on this finding, we suggest that code optimization systems could further mine and structurally organize the version evolution histories of projects on open-source platforms such as GitHub to extract performance-improving code edits that can be leveraged for retrieval-augmented code optimization. Additionally, leveraging LLMs to annotate and summarize historical commits can significantly reduce manual labor costs while greatly expanding the coverage of optimization knowledge for code execution efficiency.

Although the optimization suggestions generated by current LLMs may fall short of human-level granularity and accuracy [70], their efficiency and scalability make them a cost-effective solution for industrial applications. In practice, achieving good enough automatic optimization for the majority of code is often more valuable than pursuing perfect optimization for a few samples. Therefore, LLM-driven mechanisms for automatic annotation, retrieval, and generation of code optimization knowledge are expected to become core components of future code optimization.

9 sec:Threats32to32Validity↩︎

The potential threats to the validity of this study are discussed in accordance with the guidelines proposed in [71]. To ensure the reliability and reproducibility of the results, this study was carefully designed across multiple stages, including data construction, model training, and evaluation methods. However, we acknowledge that certain factors may still affect the validity of our findings. In this section, we discuss potential threats that may affect the results of this study.

Construct Validity: Construct validity concerns whether the evaluation metrics accurately measure the intended capabilities of this study. We primarily adopt metrics such as average runtime (Runtime_Mean), Pass@1, @Speedup, and %OPT to assess model performance on the task of code execution efficiency optimization. While these metrics are widely used and representative in existing literature [3], [57], [72], they may not fully capture the improvements in real-world scenarios. This study has two threats to construct validity: (1) Execution time is sensitive to external factors such as the runtime environment and cache state. (2) Pass@1 only evaluates the performance of a single generation, without accounting for potential accuracy improvements through retries or reranking.

Internal Validity: Internal validity refers to whether any confounding factors in the experimental design may compromise the validity of causal inferences. In this study, the observed improvements in model performance may be influenced by both the quality of the fine-tuning dataset and the relevance of the retrieved suggestions. While multiple controlled settings and a standardized evaluation pipeline were employed, a slight risk of data leakage remains due to partial sample overlap between the knowledge base and the fine-tuning data. This overlap may impact the model’s ability to generalize its understanding of prompt structures beyond the training set. In addition, due to computational constraints, only the open-source LLMs with billions of parameters were fine-tuned in this study, and consequently, we cannot provide a full assessment and comparison between all LLMs. This may limit the fairness and completeness of the comparison across all LLMs, as the full FasterPy framework was not uniformly applied.

External Validity: The optimization tasks and evaluation datasets used in this work are primarily constructed from PIE and Mercury, which are derived from competitive programming and open-source problem repositories. While these datasets provide a valuable benchmark, they may not fully capture the characteristics of production-level industrial code; thus, the effectiveness of the proposed FasterPy framework in real-world application scenarios remains to be further examined. Moreover, this study evaluates only a limited set of mainstream LLMs, and its evaluation focuses exclusively on Python programs. Although FasterPy is designed to be language-agnostic and potentially applicable to other languages such as C++, Rust, and Go, language-specific characteristics may influence the optimization outcomes. Future work should therefore explore a broader range of models, programming languages, and task domains to further validate the framework’s general applicability.

Reliability: The experimental design, including baseline model selection, evaluation metrics and computation methods, and an improved PIE evaluation program, has been provided to ensure the scientific rigor and reproducibility of the evaluation results. While the improved PIE evaluation program enhance the accuracy of runtime measurements, potential sources of variability remain. Specifically, Python’s interpreted execution model and its built-in garbage collection mechanisms may introduce minor fluctuations in timing, particularly for ultra-short programs where such overheads become non-negligible. To improve the reliability and precision of performance measurements, future work may consider adopting lower-level instrumentation techniques, such as instruction-level profiling or hardware-based counters, instead of relying solely on high-level timestamp measurements.

10 sec:Conclusions↩︎

In this study, we introduce and implement FasterPy, an LLM-based code execution efficiency optimization framework, with the goal of improving the execution efficiency of code. Existing approaches for code execution efficiency optimization are mostly based on manually crafted rule libraries or iterative algorithms, which are often costly. To address this issue, we propose an automated framework for code execution efficiency optimization driven by LLMs. The framework consists of four steps: code preprocessing, code embedding, knowledge base retrieval, and code generation. In addition, a knowledge base for code execution efficiency optimization was constructed based on the PIE [3] and Mercury [25] datasets. Our dataset contains over 60,000 entries, each including slow code, optimized code, a summary of the optimization from slow to fast code, and the embedding vector of the slow code. The experimental results show that FasterPy significantly improves both code execution efficiency and correctness, demonstrating its effectiveness and applicability in code efficiency optimization tasks.

In the next step, we plan to: (1) Collect real-world code from open-source platforms such as GitHub to further evaluate and enhance the effectiveness of FasterPy in industrial contexts. (2) Explore the development of code embedding models specifically designed for execution efficiency optimization tasks, or apply rule-based code slicing techniques to more precisely locate performance bottlenecks, thereby improving the relevance and precision of retrieved optimization suggestions. The current retrieval module in FasterPy relies on code semantic similarity using the UniXcoder model, which encodes source code, comments, and abstract syntax trees. While the UniXcoder-based retrieval approach provides a reasonable level of semantic understanding, the model itself is not tailored to the specific requirements of code execution efficiency optimization. For example, two code segments may share the same type of performance issue (e.g., redundant computation within loops) but differ entirely in functionality, leading to low semantic similarity and thus failing to retrieve useful optimization suggestions. (3) Introduce more stable and fine-grained measurement techniques to further improve the evaluation framework. Although we have redesigned the original PIE testing framework to significantly improve testing efficiency and reduce timing errors, runtime measurements may still be affected by system-level factors such as server load, kernel scheduling, and caching. In future work, we will consider incorporating hardware-level performance counters to directly measure low-level metrics such as the number of executed instructions and CPU cycles, which can further enhance the accuracy and reliability of the evaluation results.

Data Availability↩︎

The replication package, including the dataset used in this work, has been made available at [73].

References↩︎

---

1. https://github.com/tree-sitter/py-tree-sitter↩︎

2. https://milvus.io↩︎

3. https://docs.python.org/3/library/difflib.html↩︎

4. For reference, see the standard Dataset Formats and Types.↩︎