1 Introduction↩︎

Large language models (LLMs), such as GPT-4 [1] and Copilot [2], have become increasingly popular for assisting software developers with various tasks such as program repair [3], [4], automated testing [5], [6], and code translation [7], [8]. LLMs generate code based on instructions and offer intelligent recommendations, boosting developers’ productivity. Various benchmarks have been proposed to evaluate the correctness of code generation. Notable examples include HumanEval [9], APPS [10], BigCodeBench [11], and DS-1000 [12], which cover basic programming, competition-level, and data science tasks. These benchmarks have been widely used to assess the code generation capabilities of LLMs.

Despite advancements in ensuring code correctness, there remains a significant gap in the literature regarding the efficiency of code produced by LLMs [13], [14]. The importance of efficiency cannot be understated, as it directly impacts the speed of execution and the utilization of memory, which is especially important in resource-constrained environments such as mobile devices or embedded systems [15]. Efficiency of code is crucial for building scalable and sustainable software to meet the growing demands of the digital world. Furthermore, efficient code plays a pivotal role in green computing and sustainability efforts. By optimizing algorithms and reducing computational overhead, we can significantly lower energy consumption and carbon footprint. This is particularly relevant as the global demand for digital services increases.

The efficiency of two correctly generated code snippets for the same task can vary significantly. Consider the example in Figure 1, where Copilot and GPT-4 are tasked with merging two sorted arrays. Copilot generates a function that concatenates the arrays and then applies a basic Bubble Sort algorithm. While functionally correct, this approach suffers from sub-optimal time complexity of \(O((n + m)^2)\) and space complexity of \(O(n + m)\), where \(n\) and \(m\) are the array lengths. In contrast, GPT-4 generates a function that efficiently merges the arrays by systematically comparing and appending elements from each array in a single pass. This method achieves a time complexity of \(O(n + m)\), exhibiting a linear relationship with the combined lengths of the arrays. Its space complexity remains \(O(n + m)\). The disparity in efficiency highlighted in Figure 1 underscores the critical need to benchmark code generation from the perspective of code efficiency.

While being intuitive, using existing code generation benchmarks like HumanEval [9] and MBPP [16] to assess code efficiency has several limitations. These efforts primarily focus on correctness, often featuring simple tasks solvable with short code snippets. This simplicity can lead to indistinguishable efficiency across different LLMs, making it difficult to discern meaningful differences in their performance. Furthermore, most tasks are not inherently efficiency-critical, making any observed efficiency discrepancies less significant. Finally, these benchmarks lack comprehensive and diverse test cases that can thoroughly evaluate code efficiency under varying and substantial computational loads. Consequently, they are inadequate for assessing the efficiency of code generation.

This paper introduces EffiBench, a benchmark specifically designed for evaluating the efficiency of the code that is automatically generated. EffiBench comprises 1,000 efficiency-critical code generation problems selected from LeetCode. Each coding problem is paired with an executable manually-written canonical solution which has been awarded the highest rating on LeetCode for its optimal time and space efficiency. We also develop a test case generator to produce a vast number of test cases for each problem to allow for an in-depth and comprehensive analysis of the code efficiency. Moreover, EffiBench integrates a diverse set of efficiency metrics, such as execution time, maximum memory usage, and total memory usage during execution.

We conduct a comprehensive study to evaluate the efficiency of code generated by 42 LLMs. Our findings reveal that among both open- and closed-source LLMs, StarCoder2-15B [17] and GPT-4 consistently produced the most efficient code. Nevertheless, even these top performers still lag behind the efficiency of human-written canonical solutions. For instance, GPT-4 generated code exhibits an average execution time that is times that of the human-written canonical solutions. In the most extreme cases, the execution time and total memory usage of GPT-4 code are 13.89 and 43.92 times that of the canonical solutions, respectively. Furthermore, our analysis reveals that a high pass@1 score (indicating the LLM’s ability to generate correct code on the first attempt) does not necessarily translate to more efficient code. For example, GPT-4-turbo-preview has a higher pass@1 score than GPT-4, but lower code efficiency.

To conclude, this paper makes the following contributions:

• We introduce EffiBench, the first benchmark specifically designed to assess the efficiency of code generated by LLMs.

• We conduct an extensive evaluation of 42 LLMs on EffiBench, revealing that even state-of-the-art LLMs (e.g. GPT-4) exhibit significant inefficiencies compared to optimal human-written solutions.

• We release an efficiency testing framework³, which enables evaluating the efficiency across various code generation benchmarks (See 8.6).

2 Related Work↩︎

2.1 LLMs for Code↩︎

The burgeoning interest in LLMs for code has coincided with the profusion of openly available code repositories and the pressing need to enhance the productivity of software developers. Initial models predominantly focused on code generation tasks have included AlphaCode [18], CodeGen [19], CodeT5+ [20], InCoder [21], StarCoder [22], SantaCoder [23] and DeepSeek Coder [24], all of which were trained on code. Contrastingly, models such as Codex [9], Astraios [25], and CodeLLaMA [26] represent a subsequent stride, having been fine-tuned from foundation models [27], [28]. The evolution continued as LLMs leveraged instruction-like datasets derived from GPT [1], [29] for fine-tuning. Among these, WizardCoder [30] and Phi-3 [31] are notable examples. Across various coding applications, these code LLMs have set new standards of excellence, showcasing their prowess in domains including program repair [3], [4], automated testing [5], [6], [32]–[34], code translation [7], [8], type prediction [35], [36], and code summarization [37], [38].

2.2 Code Generation Benchmarks↩︎

Code generation [9], [16], [39]–[41] has emerged as a vital domain for evaluating LLMs, where models generate code snippets based on natural language descriptions, often given in the form of docstrings. Recent works try to improve HumanEval and MBPP from different perspectives. For example, HumanEval+ [42] enhances HumanEval with improved test cases, remedying the issue of mistakenly accepted faulty solutions. Meanwhile, ReCode [43] takes a different approach by altering function names and docstrings within the HumanEval structure. Expanding the scope beyond Python, HumanEval-X [44], MultiPLe [45], and MBXP [46] extend the HumanEval and MBPP benchmarks to incorporate a variety of programming languages. The universe of code generation benchmarks widens further when we consider the specialized needs of data science. DS-1000 [12], ARCADE [47], NumpyEval [48], and PandasEval [49] focus on the generation of code within this context. Beyond mere code creation, there are benchmarks like APIBench [50], MTPB [51], RepoBench [52], ODEX [53], SWE-Bench [54], GoogleCodeRepo [55], RepoEval [56], and Cocomic-Data [57], which ratchet up the complexity by evaluating a model’s prowess in utilizing APIs or completing broader software engineering tasks. Recent studies [13], [15], [58], [59] have indicated that code generated by LLMs tends to be less efficient in terms of execution time and memory usage compared to canonical solutions⁴. To bridge this gap, our benchmark EffiBench is specifically designed to evaluate the efficiency of code generation⁵.

3 Benchmark Construction↩︎

3.1 Efficiency-critical Problem Collection↩︎

3.1.0.1 Coding problem collection

Inspired by the common practice [61]–[63] of using LeetCode problems to evaluate human developers’ abilities in writing efficient algorithms, we collect the coding problems that appear on LeetCode. Specifically, we collect all problems tagged with “LeetCode” on the HuggingFace platform. We remove duplicate problems with identical problem IDs (each project has a unique ID in LeetCode). We also remove problems whose interview frequencies are lower than 40% at LeetCode. In the end, we obtain 2,605 problems as initial problem candidates.

3.1.0.2 Efficiency-critical problem filtering

This step selects efficiency-critical problems from the initial 2,605 problem candidates. The problems collected from HuggingFace are not tagged with algorithm topics. Therefore, we map each problem in LeetCode and label the problem with the “Topic” tag provided by LeetCode. We then choose typical algorithms (1) that are introduced in common algorithm textbooks [64], which are also the most widely covered in Leetcode. This yields 1,146 problems altogether.

3.2 Canonical Solution Construction↩︎

For each coding problem, EffiBench provides an executable canonical solution to serve as a baseline to calculate the normalised efficiency. Drawing inspiration from DS-1000 [12], which collects canonical solutions based on the most starred responses on Stack Overflow, we begin with collecting the top-starred solutions for each problem from the LeetCode Discussion Forum. For each collected solution, we need to guarantee that they are executable in a non-Leetcode environment. To this end, we manually fix the solutions that need to import extra classes such as TreeNode and ListNode as well as extra packages such as List and Bisect. We also remove the solutions that require specialized packages implemented only by LeetCode. In the end, we managed to map executable canonical solutions for 1,000 coding problems, which then be regarded as our final efficiency dataset.

3.3 Test Case Generation↩︎

It is essential to have adequate and diverse test cases to evaluate a program’s efficiency across various scenarios. Since directly generating test cases with LLMs (e.g., GPT-3.5) requires large token overhead and has a low accuracy (See 8.26), we develop a test case generator for each coding problem as an integral part of our benchmark construction. In particular, we require GPT-3.5-turbo to produce the test case generator, which is prompted to generate massive test cases with different input sizes, data distribution, and edge cases. Users can decide how many tests they would like to generate for each problem. We also provide 100 tests within EffiBench for users to use directly, which also serve as the tests in our evaluation in this paper (Results with 10 tests and 1,000 tests are shown in Appendix 13).

3.4 Efficiency Metrics↩︎

Efficiency metrics are crucial for benchmarking code generation models automatically. Following LeetCode, we design automatic efficiency metrics from two aspects: execution time and memory usage. Specifically, we use the following metrics: Execution Time (ET), Normalized Execution Time (NET), Max Memory Usage (MU), Normalized Max Memory Usage (NMU), Total Memory Usage (TMU), and Normalized Total Memory Usage (NTMU) to measure the overall capability of a code generation model in generating efficient code.

3.4.0.1 Execution Time (ET)

Execution time (ET) measures the average time taken for code execution. Mathematically, ET is defined as: \[ET = \frac{1}{N}\sum^{N}T_{\text{code}}\] where \(ET\) is the execution time metric, \(T_{\text{code}}\) is the execution time of the code (with all the test cases), and \(N\) is the number of codes generated by code generation models used for evaluation.

3.4.0.2 Normalized Execution Time (NET)

Normalized Execution Time (NET)⁶ measures the execution time required by generated code relative to that of a canonical solution. We define NET as: \[NET = \frac{1}{N}\sum^{N}\frac{T_{\text{code}}}{T_{\text{canonical}}}\] where \(T_{\text{code}}\) is the execution time of the generated code and \(T_{\text{canonical}}\) is the execution time of the canonical solution. A NET value greater than 1 indicates that the generated code is slower than the canonical solution, while a value less than 1 suggests the generated code is faster.

3.4.0.3 Max Memory Usage (MU)

Max Memory Usage (MU) measures the average max memory consumption during code execution. Mathematically, MU is defined as: \[MU = \frac{1}{N}\sum^{N}M_{\text{code}}\] where \(MU\) is the memory usage metric, \(M_{\text{code}}\) is the max memory consumption of the generated code among all the test cases, and \(N\) is the number of code instances generated by code generation models used for evaluation. This metric is critical to assess the resource efficiency of generated code, particularly in environments with limited maximum memory capacity.

3.4.0.4 Normalized Max Memory Usage (NMU)

Normalized Max Memory Usage (NMU) quantifies how the max memory efficiency of the generated code compares to the canonical solution. We define NMU as: \[NMU = \frac{1}{N}\sum^{N}\frac{M_{\text{code}}}{M_{\text{canonical}}}\] where \(NMU\) is the normalized max memory usage metric, \(M_{\text{code}}\) is the max memory usage of the generated code, and \(M_{\text{canonical}}\) is the max memory usage of the canonical solution. An NMU value less than 1 indicates that the generated code is more memory-efficient than the canonical solution, whereas a value greater than 1 suggests it is less efficient in terms of memory usage. This metric provides a relative measure of the memory optimization in the generated code in comparison to a standard baseline.

3.4.0.5 Total Memory Usage (TMU)

Total Memory Usage (TMU) assesses the efficiency of memory usage throughout the execution of code, taking into account both the magnitude and duration of memory utilization. To calculate TMU, first, monitor and record the memory usage at discrete time intervals during the execution, resulting in a memory usage profile \(M(t)\), where \(t\) represents time. Then, compute the area under the curve of \(M(t)\) over the total execution time, \(T_{\text{total}}\), using numerical integration methods such as the trapezoidal rule: \[TMU = \frac{1}{N}\sum^{N}\int_0^{T_{\text{total}}} M(t) \, dt\] A lower TMU value indicates higher memory efficiency, reflecting an optimized balance between the amount of memory used and the duration of its usage.

3.4.0.6 Normalized Total Memory Usage (NTMU)

The Normalized Total Memory Usage (NTMU) offers a comparison of the dynamic memory efficiency between the generated code and the canonical solution. To determine NTMU, calculate the TMU for both the generated code and the canonical solution. Normalize the TMU of the generated code by dividing it by the TMU of the canonical solution: \[NTMU = \frac{1}{N}\sum^{N}\frac{TMU_{\text{code}}}{TMU_{\text{canonical}}}\] where \(TMU_{\text{code}}\) is the TMU of the generated code and \(TMU_{\text{canonical}}\) is the TMU of the canonical solution. An NTMU value less than 1 signifies that the generated code manages dynamic memory more efficiently compared to the canonical solution, while a value greater than 1 indicates less efficient management of dynamic memory. This metric provides insight into the relative use of dynamic memory of generated code compared to an established benchmark.

4 Benchmark Statistics↩︎

We provide the detailed statistics of the dataset in 1. The coding problems in EffiBench have three difficulty levels (171 easy-level, 589 medium-level, and 240 hard-level problems), where the difficulty of each problem is defined by LeetCode [65]. The table lists the number of problems for each algorithm. Specifically, EffiBench contains 243 problems for the greedy algorithm, 277 for dynamic programming (DP), 48 for backtracking, 21 for divide and conquer, 108 for depth-first search (DFS), 86 for breadth-first search (BFS), 148 for binary search, 105 for two pointers, 70 for sliding window, 102 for bit manipulation and 238 for sorting algorithm. The sum of problems in different algorithms can be larger than the number of total problems because one problem in our dataset may belong to two algorithm classes. On average, a problem description in EffiBench contains 212.0 words. The canonical solutions, which represent the baseline code against which the generated code is compared, have 14.6 lines on average.

We provide a comparison of EffiBench and other code generation datasets in 2. Specifically, we compare EffiBench with the five most widely used code-related datasets (i.e., HumanEval, MBPP, APPS, DSP, and DS-1000). Different from the previous dataset that focuses on analyzing whether the code passes all test cases, EffiBench also analyzes the efficiency during the code execution procedure. Although EffiBench is primarily designed to assess the efficiency of generated code, it can also serve to evaluate code correctness, akin to other code generation datasets.

5 Evaluation↩︎

By default, the experiments are conducted in an edge server with an Intel Xeon Platinum 8336C CPU with 128 cores, 8 * NVIDIA A100-SXM GPUs, and a total memory capacity of 2.0TiB. We set the timeout for each code execution as 10 (s). The main goal of our work is to provide a benchmark that evaluates the efficiency of LLM-generated code within an identical environment, and we do expect that with different environments, the absolute values of the efficiency metrics would be different. We report results with different environments in 15, where our evaluation results demonstrate that despite the differences in absolute values, the ranking of LLMs is rather stable (p-value\(>>0.05\) based on Kruskal-Wallis H tests). Besides, to provide a more reliable evaluation framework, we have also provided a server in the Hugging Face Space, where users can directly upload the code generation JSON file and then the server will execute the code locally and report the efficiency results with the same environment in the future.

5.0.0.1 Models:

We evaluate both open- and closed-source LLMs in code generation. For open-source models, we evaluate EffiBench with CodeLlama-hf family (i.e., 7B, 13b, 34b, and 70B), CodeLlama-Instruct-hf family (i.e., 7B, 13b, 34b, and 70B), deepseek-coder-instruct (i.e., 1.3B and 6.7B) and base models (i.e., 6.7B and 33B), Phind-CodeLlama-34B (i.e., v1 and v2), starcoder, starcoderbase, and starcoder2 (i.e., 3B, 7B, and 15B), WizardCoder (i.e., 13B and 15B), XwinCoder (i.e., 13B and 34B), Yi models (34B, 34B-Chat, and 200K version), and five widely proposed SOTA models, i.e., Magicoder-6.7B, Mistral-7B, octocoder, Artigenz-6.7B, CodeFuse-33B, and codegemma-7b⁷ since these open-source models have obtained SOTA pass@1 in the HumanEval and MBPP datasets. For closed-source models, we evaluated EffiBench with GPT-3.5, GPT-4 [1], and claude-3, since we observe that these models obtain high pass@1 in code generation datasets (e.g., HumanEval [9], MBPP [16]). For GPT-3.5 models, we experiment with GPT-3.5-turbo-0301, GPT-3.5-turbo-0613, and GPT-3.5-turbo-1106 which represent three different versions of the GPT-3.5. For GPT-4 models, we experiment with GPT-4-turbo and GPT-4 (GPT-4-0613). For the claude-3 model, we evaluate the sonnet and haiku versions. For each LLM, we first collect the code that is correctly generated for each coding problem (i.e., they can pass all test cases provided by the dataset), then execute these correct code and calculate the efficiency metrics (See 3.4).

5.0.0.2 Prompt:

Our prompt follows the MBPP code generation prompt, where the prompt first provides the task description and then provides a few examples with input and output pairs. Each example has an explanation of the rationality of the output. The prompt also has the assertion part, which intends to constrain the function signature with the input and output format.

5.1 End2End Results↩︎

5.1.0.1 Open-source models

The evaluation results of open-source models are illustrated in 3. Our evaluation results demonstrate that all open-source models’ generated code requires more overhead than the human-written canonical solutions. For example, StarCoder2-15B, the most efficient open-source model in terms of NET, NMU, and NTMU, on average still needs 2.59x execution time, 1.71x max memory usage (i.e., memory peak), and 4.83x total memory usage during the code execution compared with the canonical solutions. We suspect that this is because human-written canonical solutions, while optimal, are in the minority within the training data of these LLMs. Consequently, the LLMs tend to learn non-optimal solutions, which are more frequently distributed in the training data. In addition, our results demonstrate that open-source LLMs with lower pass@1 tend to have better efficiency. The key reason is that these LLMs can only generate correct code on relatively simple problems, which makes it easier to achieve efficiency compared to more complex and challenging problems (see 27-29).

5.1.0.2 Closed-source models

The evaluation results of closed-source models are demonstrated in the bottom part of 3. Our results illustrate that similar to open-source models, all closed-source models generated code still need more overhead than the canonical solution on average. Despite GPT-4 generated code obtaining the most efficient results for closed-source models, its generated code still needs on average 3.12x execution time and 6.36x total memory usage during the code execution compared with the canonical solution. In the worst case, the execution time is almost 14x that of the canonical solution. In addition, although consistent training can improve the correctness of LLM-generated code, the efficiency of LLM-generated code may not improve. For example, the pass@1 for GPT-3.5-turbo increases from 42.3% to 49.3% when the model version is updated from 0301 to the 1106 version, the execution time of the code generated by GPT-3.5-turbo increases from 3.18x to 3.40x.

Consistency of different metrics: When we compare the benchmarking results from different efficiency metrics, we can observe that the rankings of different LLMs from the basic metrics (highlighted in bold in the head row) maintain a general consistency. For example, in closed-source models, GPT-4 obtains the most efficient results in the majority of metrics. Yet, for other metrics where GPT-4 does not get the highest efficiency, the code generated by GPT-4 is also close to the most efficient LLM-generated ones. This consistency across metrics reinforces their credibility in assessing a model’s capability to generate efficient code.

Correctness: Although EffiBench is designed to focus on benchmarking efficiency of LLM-generated code, it can also be adapted to benchmark code correctness, as shown by pass@1 in the last column of 3. For open-sourced LLMs, our results demonstrate that they have low pass@1: many of their pass@1 are lower than 10% (i.e., 23 out of 35 models), which indicates that open-source models still need to put a lot of effort into improving code generation correctness. For closed-sourced LLMs, GPT-4-turbo-preview has the highest pass@1 of 65.4%.

5.2 Results with Identical Coding Problems↩︎

In 3, we directly calculate the efficiency of the correct code for each model. However, different LLMs may have different correctness for the same coding problem. As a result, the results for different LLMs in 3 are based on different coding problems. In this section, we mitigate such threats by by analyzing the efficiency results with identical coding problems. In other words, we focus on analyzing problems correctly addressed by all LLMs. Since open-source LLMs do not have overlap for the tasks that are correctly generated, we only report results on closed-source LLMs only. The evaluation results are shown in 4, which contains 210 problems that have been correctly addressed by all closed-source LLMs. The evaluation results demonstrate that the results of each metric are slightly different from those shown in 3. Overall, GPT models outperform Claude models in code efficiency, with GPT-4 achieving the highest efficiency as measured by most efficiency metrics.

5.3 Results for Different Algorithms↩︎

As shown in 1, EffiBench is constructed with 11 different algorithms⁸. In this section, we explore whether the LLMs have different code efficiency across different algorithm subsets. 5 reports the results of three closed-source LLMs for different algorithm subsets. Our results demonstrate that LLMs have different code efficiency for different algorithm subsets. For example, GPT-3.5-turbo-0301 is less efficient for dynamic programming (DP), which requires 7.73x total memory usage during the code execution procedure. In contrast, GPT-3.5-turbo-0301 demonstrated higher efficiency in the DFS and binary search subset, which only requires 5.68x and 5.83x NTMU compared with the canonical solution. We indicate that the observed differences come from the availability of training data. Specifically, models tend to perform better on tasks for which their training corpus contains abundant and varied examples with efficient solutions.

5.4 Worst Case Analysis↩︎

In this section, we conduct a study to analyze the inefficient code generated by GPT-3.5-turbo-0301 (similar to the analysis in 5.3). Specifically, we collect the 10 most inefficient pieces of code for NET, NMU, and NTMU metrics and then manually analyze the implementation algorithm used by each code. The evaluation results are demonstrated in 6. The evaluation results demonstrate that the majority of the inefficient pieces of code are associated with DP and backtracking algorithms, with these categories showing the highest occurrences across the metrics. In particular, DP and backtracking algorithms show the highest counts in NTMU, indicating that these algorithms tend to generate code with higher memory consumption inefficiency, which highlights the areas where GPT-3.5-turbo-0301 struggles the most, suggesting a need for further optimization in generating code for complex algorithmic tasks.

To further understand the reasons for inefficiency in the LLM-generated code, we conduct a case comparison of GPT-3.5-turbo-0301 generated code and canonical solution in DP subset to analyze why LLM-generated code is inefficient. As shown in 2, we can observe that the key reason for GPT-3.5-turbo-0301 being less efficient than the canonical_solution is due to the code generated by GPT-3.5-turbo-0301 first generating a 2-dimensional matrix which requires large overhead for memory usage when the parameters \(n\) and \(k\) are very large. However, the canonical_solution generates two lists, which significantly reduces the memory usage for the code. GPT-3.5-Turbo-0301 implements a straightforward dynamic programming approach with a complete matrix to keep track of results for every possible pair of \(n\) and \(k\), while the canonical solution optimizes by maintaining a rolling sum, which helps to reduce the space complexity from \(O(n \times k)\) to \(O(k)\), leading to a more memory-efficient implementation. This optimization in the canonical solution results in a significant performance improvement. Specifically, GPT-3.5-turbo-0301 generated code has 70.62x memory usage during the code execution compared with canonical_solution.

6 Conclusion and Future work↩︎

In this paper, we introduce EffiBench, a benchmark designed to evaluate the efficiency of code generated by various code generation models. EffiBench encompasses 1,000 problems and consists of 11 distinct algorithmic subsets. Unlike previous benchmarks that primarily emphasize the correctness of code generation, EffiBench extends the evaluation criteria to include both execution time analysis and memory usage analysis. We also provide the evaluation server in Hugging Face to allow researchers to evaluate their methods with the same hardware and software. By incorporating these metrics and the Hugging Face server, EffiBench aims to inspire the research community’s focus towards not only the correctness but also the efficiency and sustainability of code generated by code generation models. In the future, we will consider extending EffiBench with other programming languages (e.g., C++, Java, JS, and Go).

7 ACKNOWLEDGMENT↩︎

The work is supported in part by National Key R&D Program of China (2022ZD0160201), HK RGC RIF (R7030-22), HK ITF (GHP/169/20SZ), a Huawei Flagship Research Grant in 2023, HK RGC GRF (Ref: 17208223 & 17204424), and the HKU-CAS Joint Laboratory for Intelligent System Software.

8 Appendix↩︎

8.1 Limiations↩︎

While EffiBench represents a significant step towards evaluating code efficiency in code generation models, it currently has several limitations:

Language Focus: The benchmark is currently limited to Python and does not encompass other programming languages. This restricts the scope of the evaluation and prevents a comprehensive understanding of efficiency across different language paradigms.

Dataset Scope: EffiBench focuses solely on LeetCode problems, which primarily involve algorithmic challenges. This excludes real-world applications and other coding scenarios that might necessitate different efficiency considerations.

Environment Dependency: The efficiency results obtained using EffiBench may vary across different hardware and software environments. This highlights the need for standardized testing environments to ensure consistent and reliable comparisons between models. To address this limitation, we provide the request link in our Hugging Face Leaderboard for researchers to evaluate their pre-trained LLMs generated code efficiency, which uses the same environment for efficiency testing. In the future, we will also set up an efficiency testing server in Hugging Face Space for researchers to automatically get the efficiency metrics for LLM-generated code.

8.2 Improvement Strategies↩︎

To address the limitations of EffiBench, we propose several improvement strategies as follows:

Broadening Language Coverage: Recognizing the importance of a diverse range of programming languages, we aim to expand the benchmark beyond Python in the future. This allows for a more comprehensive evaluation of code efficiency across different language paradigms, ultimately providing a more holistic understanding of the performance of code generation models.

Enhancing Dataset Diversity: To ensure that EffiBench is representative of a wide array of coding scenarios, we plan to incorporate more diverse datasets into our evaluation framework. While LeetCode problems offer valuable insights into algorithmic efficiency, we understand the need to consider real-world applications and other coding contexts. As a starting step, we have provided an efficiency testing framework that can be used with other datasets, such as HumanEval [9] and MBPP [16]. Moving forward, we will continue to seek out and integrate datasets that can enrich our understanding of code efficiency.

Standardizing Testing Environments: To address the variability in efficiency results due to different hardware and software environments, we are committed to establishing more standardized testing conditions. We have already taken a step in this direction by providing a request link in our Hugging Face Leaderboard for researchers to evaluate their LLMs generated code efficiency, which ensures that the same environment is used for testing. We also plan to set up an efficiency testing server, potentially hosted on Hugging Face Space, where developers can automatically obtain efficiency metrics for their LLM-generated code, which not only promotes consistency and reliability in our results but also makes the testing process more convenient and accessible for our users.

8.3 Broader Impacts↩︎

We list the potential positive societal impacts as follows:

Improved Software Efficiency By benchmarking and improving the efficiency of code generated by LLMs, we can develop software that runs faster, consumes less memory and processing power. This can lead to more responsive applications, reduced operational costs, and a better user experience.

Environmental Sustainability More efficient code can contribute to reduced energy consumption, which is beneficial for the environment. This aligns with global efforts to reduce carbon emissions and promote sustainability.

Enhanced Developer Productivity LLMs can significantly augment developer productivity by generating code snippets based on coding instructions and offering intelligent recommendations. This can free up developers’ time to focus on more complex tasks.

Scalable Software Development Efficient code is crucial for building scalable software to meet the growing demands of the digital world. By improving the efficiency of code generated by LLMs, we can develop software that can handle larger volumes of data and users.

On the other hand, we summarize the potential negative societal impacts as follows:

Job Displacement The increased use of LLMs in code generation could potentially lead to job displacement for some software developers in the future, particularly those involved in more routine coding tasks.

Over-reliance on AI Developers may become overly reliant on LLMs, which could lead to a lack of understanding of the generated code and potential security or functionality issues.

Security Risks If not properly managed, the use of LLMs could introduce security risks. For example, LLMs might generate code with vulnerabilities that could be exploited by malicious actors.

Quality Concerns While LLMs can generate efficient code, the quality of the code in terms of readability, maintainability, and adherence to coding standards may not always meet the desired levels. This could lead to difficulties in code maintenance and development in the long term.

8.4 Efficiency Metrics↩︎

8.4.0.1 Execution Time (ET)

Execution time (ET) measures the average time taken for code execution. Mathematically, ET is defined as: \[ET = \frac{1}{N}\sum^{N}T_{\text{code}}\] where \(ET\) is the execution time metric, \(T_{\text{code}}\) is the execution time of the code (with all the test cases), and \(N\) is the number of codes generated by code generation models used for evaluation.

8.4.0.2 Normalized Execution Time (NET)

Normalized Execution Time (NET)⁹ measures the execution time required by generated code relative to that of a canonical solution. We define NET as: \[NET = \frac{1}{N}\sum^{N}\frac{T_{\text{code}}}{T_{\text{canonical}}}\] where \(T_{\text{code}}\) is the execution time of the generated code and \(T_{\text{canonical}}\) is the execution time of the canonical solution. A NET value greater than 1 indicates that the generated code is slower than the canonical solution, while a value less than 1 suggests the generated code is faster.

8.4.0.3 Max Memory Usage (MU)

Max Memory Usage (MU) measures the average max memory consumption during code execution. Mathematically, MU is defined as: \[MU = \frac{1}{N}\sum^{N}M_{\text{code}}\] where \(MU\) is the memory usage metric, \(M_{\text{code}}\) is the max memory consumption of the generated code among all the test cases, and \(N\) is the number of code instances generated by code generation models used for evaluation. This metric is critical to assess the resource efficiency of generated code, particularly in environments with limited maximum memory capacity.

8.4.0.4 Normalized Max Memory Usage (NMU)

Normalized Max Memory Usage (NMU) quantifies how the max memory efficiency of the generated code compares to the canonical solution. We define NMU as: \[NMU = \frac{1}{N}\sum^{N}\frac{M_{\text{code}}}{M_{\text{canonical}}}\] where \(NMU\) is the normalized max memory usage metric, \(M_{\text{code}}\) is the max memory usage of the generated code, and \(M_{\text{canonical}}\) is the max memory usage of the canonical solution. An NMU value less than 1 indicates that the generated code is more memory-efficient than the canonical solution, whereas a value greater than 1 suggests it is less efficient in terms of memory usage. This metric provides a relative measure of the memory optimization in the generated code in comparison to a standard baseline.

8.4.0.5 Total Memory Usage (TMU)

Total Memory Usage (TMU) assesses the efficiency of memory usage throughout the execution of code, taking into account both the magnitude and duration of memory utilization. To calculate TMU, first, monitor and record the memory usage at discrete time intervals during the execution, resulting in a memory usage profile \(M(t)\), where \(t\) represents time. Then, compute the area under the curve of \(M(t)\) over the total execution time, \(T_{\text{total}}\), using numerical integration methods such as the trapezoidal rule: \[TMU = \frac{1}{N}\sum^{N}\int_0^{T_{\text{total}}} M(t) \, dt\] A lower TMU value indicates higher memory efficiency, reflecting an optimized balance between the amount of memory used and the duration of its usage.

8.4.0.6 Normalized Total Memory Usage (NTMU)

The Normalized Total Memory Usage (NTMU) offers a comparison of the dynamic memory efficiency between the generated code and the canonical solution. To determine NTMU, calculate the TMU for both the generated code and the canonical solution. Normalize the TMU of the generated code by dividing it by the TMU of the canonical solution: \[NTMU = \frac{1}{N}\sum^{N}\frac{TMU_{\text{code}}}{TMU_{\text{canonical}}}\] where \(TMU_{\text{code}}\) is the TMU of the generated code and \(TMU_{\text{canonical}}\) is the TMU of the canonical solution. An NTMU value less than 1 signifies that the generated code manages dynamic memory more efficiently compared to the canonical solution, while a value greater than 1 indicates less efficient management of dynamic memory. This metric provides insight into the relative use of dynamic memory of generated code compared to an established benchmark.

8.5 Model↩︎

We study both open- and closed-source LLMs in code generation. For open-source models, we evaluate¹⁰ EffiBench with CodeLlama-hf family (i.e., 7B, 13b, 34b, and 70B), CodeLlama-Instruct-hf family (i.e., 7B, 13b, 34b, and 70B), deepseek-coder-instruct (i.e., 1.3B and 6.7B) and base models (i.e., 6.7B and 33B), Phind-CodeLlama-34B (i.e., v1 and v2), starcoder, starcoderbase, and starcoder2 (i.e., 3B, 7B, and 15B), WizardCoder (i.e., 13B and 15B), XwinCoder (i.e., 13B and 34B), Yi models (34B, 34B-Chat, and 200K version), and five widely proposed SOTA models, i.e., Magicoder-6.7B, Mistral-7B, octocoder, Artigenz-6.7B, CodeFuse-33B, and codegemma-7b¹¹ since these open-source models have obtained SOTA pass@1 in the HumanEval and MBPP datasets. For closed-source models, we evaluated EffiBench with GPT-3.5, GPT-4 [1], and claude-3, since we observe that these models obtain high pass@1 in code generation datasets (e.g., HumanEval [9], MBPP [16]). For GPT-3.5 models, we experiment with GPT-3.5-turbo-0301, GPT-3.5-turbo-0613, and GPT-3.5-turbo-1106 which represent three different versions of the GPT-3.5. For GPT-4 models, we experiment with GPT-4-turbo and GPT-4 (GPT-4-0613). For the claude-3 model, we evaluate the sonnet and haiku versions. For each LLM, we first collect the code that is correctly generated for each coding problem (i.e., they can pass all test cases provided by the dataset), then execute these correct code and calculate the efficiency metrics (See 3.4).

8.6 Generalizability for other Benchmarks↩︎

Since one of our contributions is that we provide an efficiency evaluation framework, in this section we provide the generalizability of our framework on other benchmarks. Specifically, we evaluate the efficiency of LLM-generated code on HumanEvalPlus and MBPPPlus¹² [42]. The evaluation results are demonstrated on 8. We can observe that EffiBench’s framework can integrate with other benchmarks and then be used to evaluate the efficiency of LLM-generated code. Besides, we can also observe that the efficiency of LLM-generated code in this benchmark is close to the canonical solutions and sometimes even better than the canonical solutions. For example, the NET of OpenCodeInterpreter-DS-1.3B is 0.86, which is even lower than the canonical solutions. We can also observe that this behavior also exists in the MBPPPlus, while different from these benchmarks, we can observe that most of the code generated by LLMs is less efficient than the canonical solutions in EffiBench.

8.7 Efficiency metrics distribution↩︎

As shown in 3, we report the ratio of correct code with 5x efficiency metrics (i.e., NET, NMU, and NTMU) in total correct code generated by LLMs. In this section, we further analyze the distribution of normalized efficiency metrics, i.e., whether there are cases where LLMs yield more efficient code than the canonical solutions. The evaluation results are demonstrated in 9, where we evaluated 7 LLMs based on following the setup of 4. We can observe that for all evaluated LLMs, there are only a small of code generated by LLMs in 9 are more efficient than the canonical solutions, while most of the code is less efficient. For example, we can observe that only 0.23% code in Claude-3-sonnet generated correct code is more efficient than the canonical solution, while 99.77% code’s NET is large or equal to the canonical solution generated code. We suspect that the overall inefficiency of the code produced by LLMs when compared to canonical solutions may be attributed to the distribution of the training data. Typically, these datasets prioritize the correctness of code and collect code from repositories like GitHub where code is often correct but not necessarily optimized for efficiency. Focusing primarily on correctness without adequate attention to efficiency could result in neglecting efficiency in the code generated by LLMs.

8.8 Case illustration for worst case↩︎

As shown in 5, we can observe that most of the three most inefficient pieces of code are implemented by DP, backtracking, and BFS. In this section, we provide the comparison of GPT-3.5-turbo-0301 generated code and canonical solution to analyze why LLM-generated code is inefficient¹³.

BFS We provide the worst-case illustration for BFS in 4. We can observe that the code completed by GPT-3.5-turbo-0301 is less efficient in terms of memory usage compared to our canonical_solution. Specifically, GPT-3.5-turbo-0301’s code employs a standard BFS with a list-based queue, alongside a set for tracking visited states and deadends. The space complexity for this solution includes O(N) for deadends and visited states, and potentially O(\(10^4\)) for the queue, as it may store all possible lock combinations in the worst-case scenario. The breadth of the search linearly expands with the number of steps, as each step introduces multiple neighbors into the queue. Conversely, canonical_solution adopts a more sophisticated approach with a two-way BFS, utilizing two dictionaries for tracking the search from both ends and two deques for managing the queues. The space complexity remains O(N) for deadends, similar to Solution 1, but each dictionary and deque can grow up to O(\(10^4\)) in the worst-case scenario. However, the two-way BFS approach significantly condenses the search breadth by converging from both ends, reducing the overall memory consumption.

Backtracking We provide the worst-case illustration for Backtracking in 5. We can observe that GPT-3.5-turbo-0301 implementation requires substantially higher memory usage due to its less optimized recursive exploration strategy. This version systematically checks every possible combination of segments that could form an IP address by recursively calling the validation and appending results for each possible segment split. This approach accumulates a significant memory overhead as every recursive call consumes stack space and each path’s state is saved until the recursion unwinds. Conversely, the canonical solution leverages a more refined backtracking mechanism that strategically prunes invalid paths earlier through its check function and reduces unnecessary recursive depth by verifying conditions upfront. Additionally, the canonical method uses a dynamic list \(t\) to store temporary segments, effectively managing memory by adding and removing segments as needed without redundantly holding onto unsuccessful paths, leading to a drastically reduced memory footprint during execution. This optimization in the canonical solution translates into a significant performance improvement. Specifically, GPT-3.5-turbo-0301 generated code has 34.36x memory usage during the code execution compared with canonical_solution.

8.9 Generalizability for other Benchmarks↩︎

Since one of our contributions is that we provide an efficiency evaluation framework, which raises one question about whether we can use the framework of EffiBench to measure the efficiency of LLM-generated code for other benchmarks. In this section, we provide the generalizability of our framework on other benchmarks. Specifically, we evaluate the efficiency of LLM-generated code on HumanEval+ and MBPPP+¹⁴ [42]. The evaluation results are demonstrated on 8. The evaluation results demonstrate that EffiBench’s framework can integrate with other benchmarks and then be used to evaluate the efficiency of LLM-generated code. In addition, our results also demonstrate that the efficiency of LLM-generated code in these two datasets is close to the canonical solutions and sometimes even better than the canonical solutions. For example, the NET of OpenCodeInterpreter-DS-1.3B is 0.86 in the HumanEval+ dataset, which is even lower than the canonical solutions.

8.10 Efficiency metrics distribution↩︎

As demonstrated in 3, the efficiency of LLM-generated code are lower than the efficiency of the dataset provided canonical solution. To measure the ratio of the inefficient code generated by LLMs in the total LLM-generated code, we provide the ratio of the code higher / lower than the efficiency of the canonical solution provided by the dataset. The evaluation results are demonstrated in 9, where we evaluated 7 LLMs based on following the setup of 4. The evaluation results demonstrate that for all evaluated LLMs, there are only a small of code generated by LLMs in 9 are more efficient than the canonical solutions, while most of the code is less efficient. For example, only 0.23% code in Claude-3-sonnet generated correct code is more efficient than the canonical solution, while 99.77% code’s NET is large or equal to the canonical solution generated code. We suspect that the overall inefficiency of the code produced by LLMs when compared to canonical solutions may be attributed to the distribution of the training data. Typically, these datasets prioritize the correctness of code and collect code from repositories like GitHub where code is often correct but not necessarily optimized for efficiency. Focusing primarily on correctness without adequate attention to efficiency could result in neglecting efficiency in the code generated by LLMs.

8.11 Case study for efficient solution↩︎

8.12 Calculating the normalized metrics with task level↩︎

In 3.4, we define the normalized efficiency metrics at the dataset level. For example, NET is defined as: \[NET = \frac{1}{N}\sum^{N}\frac{T_{\text{code}}}{T_{\text{canonical}}}\] In this section, we further discuss the normalized efficiency metrics for LLM-generated code at the dataset level. For example, we set NET* as the dataset-level normalized execution time metric. The NET* is defined as: where \(T_{\text{code}}\) is the execution time of the generated code, and \(T_{\text{canonical}}\) is the execution time of the canonical solution. \[NET = \frac{\sum^{N}T_{\text{code}}}{\sum^{N}T_{\text{canonical}}}\]

We follow the setup of 4 to evaluate the efficiency of LLM-generated code in 9 open- and closed-source models. The evaluation results are demonstrated in 12. We can observe that with the dataset-level normalized metric calculation, the efficiency of LLM-generated code is closer to the canonical solution. For example, GPT-3.5-turbo-0301 generated code required execution time decreases from 3.18x to 2.92x compared to the canonical solution. The key reason is that the dataset-level normalization aggregates the performance across all tasks, potentially masking significant variations in efficiency on individual tasks. While the dataset-level normalized metric, such as NET*, provides a broad overview of the model’s performance, it can obscure important details about how well the model handles specific tasks. For example, this dataset-level calculation ignores the metrics evaluated in 9. This aggregation can lead to a situation where poor performance on a few tasks is averaged out by better performance on others, giving a potentially misleading impression of overall efficiency.

8.13 Efficiency distribution for the normalized metrics↩︎

As shown in 9, we report the efficiency distribution for normalized metrics of the LLM-generated code. In this section, we further break down the efficiency distribution of GPT-3.5-turbo-0301 generated code. Specifically, for each normalized metric, we collect all GPT-3.5-turbo-0301 generated code’s efficiency metric. Then we divide them into 100 buckets. Then, we report the accumulated figures in 7. We can observe that most of the GPT-3.5-turbo-0301 generated code is less efficient than the canonical solution (i.e., value = 1).

8.14 Efficiency of Code with different number of tests↩︎

Our experiments in 3 only consider 100 tests for each problem, which inspires us to consider how different numbers of tests affect the efficiency of code generated by code generation models. To answer this question, we investigate how does different number of tests affects the efficiency score for each metric. The evaluation results are shown in 13, where we can observe that once we increase the tests from 10 to 1,000, the efficiency score for NET, NMU, and NTMU increase for GPT-3.5-turbo-0301. For example, the GPT-3.5-turbo-0301’s NTMU increases from 4.75 to 10.08. We indicate that the key reason is once we increase the number of tests, more edge cases would be covered  (e.g., more length, data distribution). However, since the tests for the efficiency experiments, the overhead such as memory usage increases largely. For example, when we increase the tests from 100 to 1,000, the TMU increases from 8.84 MB*s to 340.51 MB*s, which requires more computation resources for experiments. So in our experiments and Leaderboard, we focus on studying the LLM-generated code efficiency in 100 tests.

8.15 Randomness↩︎

8.15.0.1 Seed

We also evaluated the efficiency of the code generated by GPT-3.5-turbo-0301 five times in the same environments to ensure the reliability of our results. As demonstrated in 14, performance metrics such as ET, MU, and TMU show remarkable consistency across different executions. Specifically, the standard deviations (std) for these metrics are exceptionally low, demonstrating minimal variability and highlighting the stability of the code execution in our testing environment. For example, the mean of the ET is 0.39 (s), while the std of the ET is 0 for the five times results. This consistent performance underpins the robustness of our experimental approach, providing a solid foundation for further analysis of the model’s operational characteristics.

8.15.0.2 Environment

We also provide an analysis of the efficiency of the code generated by closed-source models in different local environments. The results are shown in 15, where we can observe that in different environments, the efficiency changed slightly, which pushes us to consider how can we avoid the bias for different users to use EffiBench to quantify the efficiency of their pre-trained code generation models. To avoid this problem, we provide two different solutions that can maintain the same code execution environment. First, we provide Request efficiency evaluation form in our Leaderboard and Github, by filling the request we will evaluate the efficiency of the request pre-trained code generation model and then report it to the user. Second, we also provide a server in the Hugging Face Space where users can directly upload the code generation JSON file and then the server will execute the code locally and then report the efficiency results. The testing time in the server only requires less than one minute for each model (See 8.16).

8.16 Overhead↩︎

The overhead of the efficiency evaluation is important as if the overhead of the evaluation is very long, the validity of the results will be questionable. To address this concern, we provide the overhead report for the closed-source models in 16. We can observe that the overhead required by each model for efficiency testing is lower than 1 minute. For example, the source code generated by GPT-3.5-turbo-0301 only requires 32 (s) to finish the efficiency testing.

8.17 Discussion on Time and Space Complexity↩︎

In our experiment, we aim to quantify the efficiency of code generated by code generation models with our efficiency metrics. While time and space complexity are conventional metrics in software development for assessing code efficiency, we opted not to rely solely on these for several reasons. Firstly, identical time and space complexity annotations do not guarantee equivalent performance across different implementations. For instance, two algorithms with time complexities expressed as \(T(2n)\) and \(T(n)\) might both be classified under the same complexity order \(O(n)\). However, their practical execution times and resource utilization can vary significantly, underscoring the limitations of using complexity classes as the sole measure of efficiency. Secondly, accurately determining the time and space complexity of a given piece of code typically requires manual analysis and labeling. This process is inherently subjective and prone to human error, making it less suitable for automated, large-scale evaluation of code generation models. The necessity for manual intervention contradicts our goal of automating the efficiency evaluation process as much as possible. Thirdly, although there are models designed to predict the time and space complexity of code, these predictions are often sub-optimal and can be inaccurate¹⁵. Relying on such models for critical evaluations might introduce significant errors, leading to misleading conclusions about a code generation model’s efficiency. Given these considerations, we chose to focus on direct measurements of execution time and memory usage through our specified metrics. These measurements provide a more accurate, objective, and practical assessment of the generated code’s efficiency, reflecting real-world performance more closely than theoretical complexity classes. This approach allows for a nuanced analysis of the models’ output, enabling a comprehensive evaluation of their practical utility in software development scenarios.

8.18 Algorithm subsets↩︎

8.19 Calculating the normalized metrics with task level↩︎

In 3.4, we define the normalized efficiency metrics at the dataset level. For example, NET is defined as: \[NET = \frac{1}{N}\sum^{N}\frac{T_{\text{code}}}{T_{\text{canonical}}}\]. In this section, we further discuss the normalized efficiency metrics for LLM-generated code at the dataset level. For example, we set NET* as the dataset-level normalized execution time metric. The NET* is defined as: where \(T_{\text{code}}\) is the execution time of the generated code, and \(T_{\text{canonical}}\) is the execution time of the canonical solution. \[NET = \frac{\sum^{N}T_{\text{code}}}{\sum^{N}T_{\text{canonical}}}\]

We follow the setup of 4 to evaluate the efficiency of LLM-generated code in 9 open- and closed-source models. The evaluation results are demonstrated in 12. We can observe that with the dataset-level normalized metric calculation, the efficiency of LLM-generated code is closer to the canonical solution. For example, GPT-3.5-turbo-0301 generated code required execution time decreases from 3.18x to 2.92x compared to the canonical solution. The key reason is that the dataset-level normalization aggregates the performance across all tasks, potentially masking significant variations in efficiency on individual tasks. While the dataset-level normalized metric, such as NET*, provides a broad overview of the model’s performance, it can obscure important details about how well the model handles specific tasks. For example, this dataset-level calculation ignores the metrics evaluated in 9. This aggregation can lead to a situation where poor performance on a few tasks is averaged out by better performance on others, giving a potentially misleading impression of overall efficiency.

8.20 Efficiency distribution for the normalized metrics↩︎

As shown in 9, we report the efficiency distribution for normalized metrics of the LLM-generated code. In this section, we further break down the efficiency distribution of GPT-3.5-turbo-0301 generated code. Specifically, for each normalized metric, we collect all GPT-3.5-turbo-0301 generated code’s efficiency metric. Then we divide them into 100 buckets. Then, we report the accumulated figures in 7. We can observe that most of the GPT-3.5-turbo-0301 generated code is less efficient than the canonical solution (i.e., value = 1).

8.21 Efficiency of Code with different number of tests↩︎

Our experiments in 3 only consider 100 tests for each problem, which inspires us to consider how different numbers of tests affect the efficiency of code generated by code generation models. To answer this question, we investigate how does different number of tests affects the efficiency score for each metric. The evaluation results are shown in 13, where we can observe that once we increase the tests from 10 to 1,000, the efficiency score for NET, NMU, and NTMU increase for GPT-3.5-turbo-0301. For example, the GPT-3.5-turbo-0301’s NTMU increases from 4.75 to 10.08. We indicate that the key reason is once we increase the number of tests, more edge cases would be covered  (e.g., more length, data distribution). However, since the tests for the efficiency experiments, the overhead such as memory usage increases largely. For example, when we increase the tests from 100 to 1,000, the TMU increases from 8.84 MB*s to 340.51 MB*s, which requires more computation resources for experiments. So in our experiments and Leaderboard, we focus on studying the LLM-generated code efficiency in 100 tests.

8.22 Difficulty↩︎

We also provide the efficiency results of all open- and closed-source models in the different difficulty in 27-29. We can observe that the pass@1 of open-source LLMs is very low.

8.23 Randomness↩︎

8.23.0.1 Seed

We also evaluated the efficiency of the code generated by GPT-3.5-turbo-0301 five times in the same environments to ensure the reliability of our results. As shown in 14, we can observe that performance metrics such as ET, MU, and TMU show remarkable consistency across different executions. Specifically, the standard deviations (std) for these metrics are exceptionally low, demonstrating minimal variability and highlighting the stability of the code execution in our testing environment. This consistent performance underpins the robustness of our experimental approach, providing a solid foundation for further analysis of the model’s operational characteristics.

8.23.0.2 Environment

We also provide an analysis of the efficiency of the code generated by closed-source models in different local environments. The results are shown in 15, where we can observe that in different environments, the efficiency changed slightly, which pushes us to consider how can we avoid the bias for different users to use EffiBench to quantify the efficiency of their pre-trained code generation models. To avoid this problem, we provide two different solutions that can maintain the same code execution environment. First, we provide Request efficiency evaluation form in our Leaderboard and Github, by filling the request we will evaluate the efficiency of the request pre-trained code generation model and then report it to the user. Second, we also provide a server in the Hugging Face Space where users can directly upload the code generation JSON file and then the server will execute the code locally and then report the efficiency results. The testing time in the server only requires less than one minute for each model (See 8.16).

8.24 Overhead↩︎

We provide the overhead report for the closed-source models in 16. We can observe that the overhead required by each model for efficiency testing is lower than 1 minute.

8.25 Discussion on Time and Space Complexity↩︎

In our experiment, we aim to quantify the efficiency of code generated by code generation models with our efficiency metrics. While time and space complexity are conventional metrics in software development for assessing code efficiency, we opted not to rely solely on these for several reasons. Firstly, identical time and space complexity annotations do not guarantee equivalent performance across different implementations. For instance, two algorithms with time complexities expressed as \(T(2n)\) and \(T(n)\) might both be classified under the same complexity order \(O(n)\). However, their practical execution times and resource utilization can vary significantly, underscoring the limitations of using complexity classes as the sole measure of efficiency. Secondly, accurately determining the time and space complexity of a given piece of code typically requires manual analysis and labeling. This process is inherently subjective and prone to human error, making it less suitable for automated, large-scale evaluation of code generation models. The necessity for manual intervention contradicts our goal of automating the efficiency evaluation process as much as possible. Thirdly, although there are models designed to predict the time and space complexity of code, these predictions are often sub-optimal and can be inaccurate¹⁶. Relying on such models for critical evaluations might introduce significant errors, leading to misleading conclusions about a code generation model’s efficiency. Given these considerations, we chose to focus on direct measurements of execution time and memory usage through our specified metrics. These measurements provide a more accurate, objective, and practical assessment of the generated code’s efficiency, reflecting real-world performance more closely than theoretical complexity classes. This approach allows for a nuanced analysis of the models’ output, enabling a comprehensive evaluation of their practical utility in software development scenarios.

8.26 Discussion Automatically-generated Test Cases↩︎

As discussed in 3.3, EffiBench generated test cases by first developing a test case generator for each coding problem, where we modify the test case generator to make sure the test cases generated by the generator are correct. Then, we use the test case generator to generate test cases for each task. In this section, we discuss why do we not directly require LLM (e.g., GPT-3.5-turbo) to generate test cases for each task. Specifically, we provide the experiment results of four closed-source LLMs generated test cases’ accuracy. The evaluation results are demonstrated in 31, where we can observe that the accuracy of the test cases generated by four LLMs is lower than 15%, which explains why do we not use LLM to generate test cases for each task, i.e., the accuracy of test cases are very low.

8.27 Case illustration of test case generator↩︎

We provide a case example to illustrate that how does test case generator generate test cases for EffiBench. Specifically, as shown in 9, we can observe that the script is used to generate 100 tests for the function lengthOfLongestSubstring, where the test case generator randomly generates input and then feeds into the canonical solution. Then, the canonical solution returns the output for the given input.

Checklist↩︎

The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default to , , or . You are strongly encouraged to include a justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:

• Did you include the license to the code and datasets?

• Did you include the license to the code and datasets?

• Did you include the license to the code and datasets?

Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.

1. For all authors...

   1. Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

   2. Did you describe the limitations of your work?

   3. Did you discuss any potential negative societal impacts of your work?

   4. Have you read the ethics review guidelines and ensured that your paper conforms to them?

2. If you are including theoretical results...

   1. Did you state the full set of assumptions of all theoretical results?

   2. Did you include complete proofs of all theoretical results?

3. If you ran experiments (e.g. for benchmarks)...

   1. Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?

   2. Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?

   3. Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?

   4. Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?

4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

   1. If your work uses existing assets, did you cite the creators?

   2. Did you mention the license of the assets?

   3. Did you include any new assets either in the supplemental material or as a URL?

   4. Did you discuss whether and how consent was obtained from people whose data you’re using/curating?

   5. Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?

5. If you used crowdsourcing or conducted research with human subjects...

   1. Did you include the full text of instructions given to participants and screenshots, if applicable?

   2. Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?

   3. Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?

References↩︎

---

1. Equal Contribution.↩︎

2. Corresponding Author.↩︎

3. We also make Github Repo public and then researchers can create issues in Github to evaluate the efficiency. Or they can directly use the docker and our public Hugging Face Server for efficiency calculation.↩︎

4. Liu et al. [58] focus on collecting tasks that evaluate performance and ensuring reliable assessments, while EffiBench concentrate on developing the dataset construction pipeline and benchmarking the efficiency of LLM-generated code for efficiency-critical tasks.↩︎

5. A parallel work, Mercury [60], is also used to measure the efficiency of LLM-generated code.↩︎

6. To demonstrate code-level efficiency, we evaluate the normalized efficiency metrics in task level, rather than total LLM-generated code / total canonical solutions. For the second calculation strategy, we also provide the scripts in our Github Repo.↩︎

7. The model names are extracted from Hugging Face model card.↩︎

8. Note that the task is classified as a specific algorithm but the code generated by LLMs may consider addressing the task with other algorithms.↩︎

9. To demonstrate code-level efficiency, we evaluate the normalized efficiency metrics in task level, rather than total LLM-generated code / total canonical solutions. For the second calculation strategy, we also provide the scripts in our Github Repo.↩︎

10. The full evaluated model lists can be seen in our Hugging Face leaderboard.↩︎

11. The model names are extracted from Hugging Face model card.↩︎

12. HumanEval and MBPP datasets have a limited number of test cases (fewer than 10) for each task, which can lead to highly random efficiency testing results due to the rapid execution of the code. To mitigate the impact of randomness, we utilize the test cases provided by EvalPlus to ensure sufficient testing time.↩︎

13. We demonstrate DP example in 2.↩︎

14. HumanEval and MBPP datasets have a limited number of test cases (fewer than 10) for each task, which can lead to highly random efficiency testing results due to the rapid execution of the code. To mitigate the impact of randomness, we utilize the test cases provided by EvalPlus to ensure sufficient testing time.↩︎

15. https://community.ibm.com/community/user/ai-datascience/blogs/sepideh-seifzadeh1/2021/10/05/ai-for-code-predict-code-complexity-using-ibms-cod↩︎

16. https://community.ibm.com/community/user/ai-datascience/blogs/sepideh-seifzadeh1/2021/10/05/ai-for-code-predict-code-complexity-using-ibms-cod↩︎