1 Introduction↩︎

Large Language Models (LLMs) are increasingly evaluated on code benchmarks such as SWE-Bench [1] to measure their code generation, code reasoning, and debugging capabilities. These benchmarks play a critical role in shaping the understanding of LLMs’ true capabilities and limitations. However, the validity of conclusions drawn from these code benchmarks depends on the rigor, reliability, and reproducibility of the benchmarks themselves. When the benchmark construction is flawed, the validity of conclusions may not hold.

Encouragingly, awareness of benchmark quality has grown. Guidelines [2] and enhancing approaches [3]–[5] were proposed to provide the good practices and enhance the quality in certain aspects (e.g., code coverage [3], data contamination [6]). As a result, one might expect that benchmark construction practices have already become more rigorous over time.

However, our findings suggest the opposite. Through a large-scale survey of 672 code benchmarks across the last decade (2014 – 2025), we uncover a concerning trend: despite rising awareness of benchmark quality, the number of flawed benchmarks has continued to grow. For example, in 2025 alone, the number of benchmarks that ignore code coverage when providing test cases nearly matches the total count accumulated across the previous ten years. More empirical evidence includes:

46% benchmarks did not go through quality assurance check; 79.8% did not consider data contamination; 67.1% did not deduplicate the data points;

85.0% did not ensure a reliable judgement, such as ensuring a high code coverage when test suites are provided; 66.1% benchmark evaluation was one-pass, without repeating the experiment to avoid randomness;

38.2% of the benchmarks did not provide the essential information (e.g., prompts) for reproducibility; 80.0% did not provide the log information on the benchmarks; 14.7% are not open source, 2.2% only partially released;

In response, we take a clear position: Code benchmarks must prioritize rigor in benchmark construction, reliability in evaluation, and reproducibility in release as first-class objectives.

To operationalize this position, we introduce a comprehensive code benchmark guideline, How2Bench, comprising 55 rigor-oriented checklists, each annotated with its relative threat level to the benchmark validity. This checklist covers the entire lifecycle of benchmark development, from design and construction to evaluation, analysis, and release, as shown in Figure 1.

Upon the 55-item checklist, we revisited 672 code benchmarks, and quantified longitudinal trends in how benchmark practices have evolved. We observe a lag between growing awareness and actual practice. Although the yearly proportion of benchmarks that prioritize data quality and evaluation credibility has increased, the absolute number of flawed benchmarks continues to rise, largely due to the rapid overall growth in benchmark production.

In addition to these concerning trends, we also identify several positive signals:

During Benchmark Design: recent benchmarks increasingly focus on practical, real-world problems; notably, the number of project-level benchmarks in 2025 nearly tripled compared to 2024, and the distribution of code task types has become more diverse;

During Benchmark Construction: creators are showing stronger awareness of manual quality assurance: the number of benchmarks incorporating human verification more than doubled from 2024 to 2025;

During Benchmark Evaluation: benchmark creators increasingly include a larger and more diverse set of studied LLMs, improving the generalizability and robustness of reported findings;

During Benchmark Release: growing open-science engagement, with an upward trend in publicly released artifacts, prompts, and evaluation resources, reflecting stronger community commitment to transparency and reuse.

Human Study – To better understand whether the current situation stem from lack of awareness, resource constraints, or incentive misalignment, we conducted a human study involving 49 participants through questionnaires. All participants concurred on the necessity of having a checklist for benchmark construction to enhance quality. Interestingly, beyond admitting the its importance, the collected feedbacks also exposed gaps in awareness: 16% of participants were unaware of the necessity for data denoising; over 40% were not aware that the experimental setup and environment could impact the reproducibility and transparency.

The questionnaires results revealed that the current issues in flawed benchmark not only stems from the significant effort required to construct a rigorous and reliable benchmark, but also from a lack of awareness regarding how crucial a rigorous development process is to the quality of benchmarks and the reliability of evaluations. Without being aware of and addressing the quality of code-related benchmarks (as well as other kinds of LLM benchmarks), the reliability and reproducibility of benchmark results remain compromised, misleading research directions, and hindering meaningful progress.

This paper makes contributions in five aspects:

• Novelty. We introduce How2Bench, a comprehensive set of guidelines packaged as a 55-criteria checklist that covers the lifecycle of code-related benchmark development.

• Significance. How2Bench presents the first comprehensive set of actionable guidelines for developing high-quality benchmarks, striving to create a more reliable and transparent environment. The human study also highlighted the demand for such a detailed guideline.

• Usefulness. How2Bench serves as a guideline for practitioners before/during developing code-related benchmarks, and a checklist for evaluating existing benchmarks after their release. For ease of use, we also provide a printable version of How2Bench on Appendix 15.

• Generalizability. Most criteria listed in How2Bench can be adopted or adapted to other benchmarks such as Question-answering, mathematical reasoning, and multimodal benchmarks.

• Long-term Impact. Our statistics alert the community to the severity and prevalence of non-standard practices in benchmark development. It ultimately improves the overall quality of benchmarks through propagation effects among them.

2 Related Work↩︎

2.1 Code Benchmarks↩︎

Benchmarks for coding tasks like code generation [7], [8], defect detection [9]–[11], and program repair [1], [12] are increasingly common, reflecting the growing needs for using LLMs for coding tasks. Recent studies have highlighted various issues with these benchmarks, ranging from design inconsistencies to scope and applicability limitations. For example,  [3] found that even some widely used benchmarks, such as HumanEval [7] and MBPP [8], contains a non-trivial proportion of bugs in implementation, documentation, and test cases. Our work, in comparison, introduces a detailed guideline that guides the benchmark development during the entire lifecycle.

2.2 Benchmark Surveys and Studies↩︎

Several recent surveys [13] and empirical studies have profiled the status quo of LLM development. These studies either explore the overall performance for certain areas, such as software engineering [14], [15], or investigate the capabilities of LLMs on specific tasks, such as code generation [16], [17] and test generation [18]–[20]. A survey [21] about how to evaluate LLMs was proposed to answer what/where/how to evaluate LLMs. This paper differs from these studies in its purpose and perspectives.

Assessments or guidelines for benchmarks [13], [22]–[24] are also constantly released. For example, BetterBench [24] is a related work assessing the AI benchmarks against 46 criteria. Then, it scored 24 AI benchmarks in various domains and ranked them. BetterBench differs from this paper in several key aspects: scope (general benchmarks vs. code-related benchmarks), lifecycle division (it addresses benchmark retirement, while How2Bench focuses on benchmark evaluation, analysis, and release), and objectives (scoring benchmarks vs. offering comprehensive guidelines for future benchmark development). Additionally, the study in this paper was conducted on a much larger scale (24 vs. 572 benchmarks), statistically highlighting the prevalent issues in existing benchmarks.

Unlike these surveys and guidelines, our work reveals a longitudinal trends before and after them came out.

3 Guideline Design↩︎

3.1 The Lifecycle of Benchmark Development↩︎

Code-related benchmark development comprises five typical phases (Phase 1 - 5), as shown in Figure 1, explained in detail as follows.

Phase 1. Design. At the beginning of benchmark development, it is vital to identify the motivation, the scope and the capabilities required by the application scenario of interest. To achieve this objective, one needs to carefully consider the application scenarios, making sure these scenarios align with real-world demands [25]. Also, it is necessary to assess whether other benchmarks already exist that address similar tasks, and to identify any shortcomings they may possess [26], [27]. Furthermore, this new benchmark should be designed to evaluate specific LLMs’ capabilities; the crafted tasks are expected to reflect these capabilities [28].

Phase 2. Construction. Benchmark Construction phase moves from design to execution. Typically, data is collected from public coding websites such as GitHub, LeetCode, and StackOverflow. This is followed by preprocessing, which includes filtering, cleaning (e.g., deduplication, denoising), and curation (e.g., aligning tests with corresponding code). The phase usually ends with a validation process, which can be manual or automated [26].

Phase 3. Evaluation. Once the benchmark is available, the next step is to apply it to LLMs, validating if it can effectively measure the intended LLM capabilities. Essential considering factors include selecting a representative array of LLMs, configuring settings like prompts and hyperparameters for consistency, choosing appropriate experimental environments to meet LLM requirements, and implementing thorough logging to ensure dependable and reproducible results.

Phase 4. Analysis. After evaluation, experimental results are analyzed, drawing conclusions on LLMs’ capabilities. This phase involves comparing each LLM’s performance to identify standout or underperforming models. Then, proper visual aids such as bar charts and tables can be used to display the experimental results, presenting clearer observation and deeper inspiration, such as the correlations between models, the correlations with related benchmarks, or performance in upper-/down-stream tasks [29]. Indeed, a thorough analysis helps pinpoint areas for improvement and guides future LLM enhancements.

Phase 5. Release. The final phase is to make the benchmark open-accessible. This phase involves meticulously preparing all materials associated with the benchmark, ensuring they are ready for open access to foster widespread adoption and collaboration. Clear, comprehensive documentation is provided to guide users on effectively utilizing the benchmark. Additionally, all logged experiment details are made available, enhancing the reproducibility and transparency of the benchmark.

3.2 Study Design↩︎

Our study consists of four steps (Figure 2). All steps are explained as follows.

Step 1. Guideline Construction. To begin with, we sketched the initial guidelines for each phase in the benchmark development lifecycle (Section 3.1, Figure 1) by reviewing existing literature [18], [24], [30], [31] and brainstorming. After that, we refined the guidelines through a series of interviews with various stakeholders, including model developers and benchmark builders, allowing for the addition, deletion, or modification of criteria based on expert feedback and practical insights. In order to allow more flexibility and increase the practicality of the guideline, we prioritized them with \(\bigstar\) \(\bigstar\) \(\bigstar\) (Highly important), \(\bigstar\) \(\bigstar\) (Important), and \(\bigstar\) (Optional). By doing so, the highly-recommended criteria are essential to comply with when setting up a new benchmark, while other criteria allow compromise, making the guideline simpler for benchmark developers to follow.

Step 2. Literature Profiling. This step begins by collecting related benchmarks according to their publication time, venue, and coding tasks, then employing techniques like snowballing to ensure a comprehensive collection. This step leads to 672 code-related benchmarks for study. The detailed statistics are available in Appendix 14. This step is followed by profiling each selected benchmark through a thorough review of corresponding papers and examination of the released artifacts or homepages associated with these benchmarks. The phase is completed by reporting statistics that highlight overall trends, pros, and cons identified during the profiling, providing a structured overview of existing benchmarks.

Step 3. Focused Case Study. After obtaining an overall impression of existing benchmarks, we selected 30 (= 5 * 6) representative benchmarks from the top-5 most frequent coding tasks (see Figure 9), with top-5 highly-cited benchmarks plus the latest 1 benchmark (Appendix 13). Each selected benchmark is then analyzed against How2Bench, examining how well they meet the established criteria, studying their overall statistics, and identifying both exemplary and poor cases. Insights and references from existing literature are also incorporated to enrich the analysis, providing a deeper understanding of the benchmarks’ performance and areas for improvement.

Step 4. Human Study. The final step is a human study that evaluates the importance and practicality of How2Bench. This involves designing a questionnaire by first initiating and iterating to gather diverse, logical insights, which is then distributed to a targeted audience. After collecting and filtering responses for quality, the data is analyzed to derive insights. See Appendix 12 for details.

4 Guideline and Key Statistics↩︎

The completed guideline How2Bench with 55 criteria can be found in the Appendix. Each guideline contains: an actionable check criterion with necessary explanations; a priority indicator, divided into three levels in total, marking the importance of the checklist; and a checkbox for convenience.

4.1 Guideline for Benchmark Design↩︎

Explanation – For benchmark design, we listed four essential criteria, as shown in Figure 3. In particular, the guideline starts by recommending that benchmarks should initially assess if they are addressing a significant gap in existing research, ensuring the relevance and necessity of the benchmark. The scope of the benchmark is expected to be well-defined, clarifying the capabilities or characteristics being tested, how these relate to practical scenarios such as programming assistance or automated testing, and the relevance of these capabilities in real-world applications.

Key Statistics – According to our statistics among 672 benchmarks, apparent research bias can be observed in terms of coding tasks, programming languages, and code granularities (Appendix 11.1). For example, Code Generation is most prevalent coding task, with 235 benchmarks focusing on this area, according to 34.97% (235/672) of studied benchmarks, indicating a significant interest in generating code automatically. The second most prevalent is code reasoning 12.64% (85/672), followed by Program Repair and Defect Detection.

When examining the distribution of coding tasks by year (see Figure 10), we observe that benchmark growth accelerated most rapidly in the past two years (2024–2025), with code generation remaining the most frequently benchmarked task. At the same time, demand for code reasoning has increased substantially, rising from 19 benchmarks in 2024 to 61 in 2025. Benchmarks for program repair also increased over the same period, from 22 to 33.

Regarding the bias in programming language, Figure 11 shows that 409 (71.50%) benchmarks are in Python, followed by Java and C++, with 229 and 160, respectively. This observation consolidates the observation from previous works [14], [32] on a larger scale. When analyzed by year (Figure 12), the number of benchmarks for C/C++ and JavaScript has increased noticeably between 2024 and 2025. We also observe a sharp rise in Rust-related benchmarks: although only a few benchmarks existed for Rust since its release in 2015, 57 benchmarks were published in the past two years (27 in 2024 and 29 in 2025). These trends in programming language coverage reflect the evolving demand for LLMs across different languages, highlighting particularly the growing reliance on large models to generate, reason about, and maintain code in these languages.

Regarding the bias in granularity, a dramatic upward trend was observed in project-level benchmarks (Figure 16): between 2024 and 2025, the number of project-level benchmarks surged, increasing from 35 to 97 new benchmarks. Importantly, these numbers refer to benchmarks newly introduced in each year, rather than cumulative totals. This surge indicates a growing community focus on the real-world applicability and large-scale practical utility of LLMs.

Regarding the LLMs’ capability evaluated, during the focused case study (listed in Appendix 13), we identified that 20% benchmarks have not explicitly specified the capabilities (e.g., intention understanding, program synthesis) to be evaluated, and 23.3% have not specified application scenarios the benchmark targets.

Besides, we also identified a case in MBPP [8] where a case fell out of the target evaluation capabilities (Appendix 11.2). Indeed, clearly defining the application scenarios/scopes/capabilities could help benchmark constructors establish precise goals for the design and development of the benchmark, ensuring accuracy in the evaluation.

4.2 Guideline for Construction↩︎

Explanation – Figure 4 shows 19 criteria for benchmark construction. Essentially, for data source, the key considerations include verifying the traceability and quality of the data source, addressing potential data contamination [33], and ensuring that the data sampling processes are scientifically robust and rigorous. Also, for data representativeness, it also guides through specific checks to ensure the benchmark’s scope is strictly adhered to, such as making sure every data point falls within the targeted scope and that the data can cover all studied capabilities, domain knowledge, and application scenarios.

For data preprocessing and cleaning, it also stresses handling code-specific aspects, such as compilability and execution, along with cleaning and manually reviewing data for quality assurance. Output validation methods and evaluation metrics must be carefully designed and reviewed to ensure they effectively measure the benchmark’s goals. Lastly, it suggests considering additional evaluation perspectives, such as safety [34], [35] checks, ensuring the code does not contain sensitive information.

Key Statistics – According to our statistics (Appendix 11.3), the 572 benchmarks exhibit numerous irregularities in their implementation, which could significantly threaten the reliability of the benchmarks. Surprisingly, 67.1% of benchmarks did not deduplicate or did not mention. 79.8% benchmarks did not consider or handle data contamination threats. About 46.0% of the benchmarks did not go through any quality assurance checks such as manual checks and code execution. In particular, we summarized the commonly-used data quality assurance metrics and their frequency: manual check (45.8%), code execution (1.2%),others (e.g., heuristic rules, 6.4%).

Examining the trends by year (Figure 24), however, one positive development is the increasing prevalence of manual quality checks. The number of benchmarks with manual check guarantees doubled from 84 in 2024 to 193 in 2025, indicating that a growing share of code benchmarks now undergo partial or full human verification.

Also, since we focus on code-related benchmarks, which usually accompany test cases, code coverage also needs to be considered. According to the statistics over oracles (Figure 31), passing test cases (257 / 672 = 38.2%) and the exact match (193 / 672 = 28.7%) are the most common oracles used in code benchmarks. However, we observed that only 15.9% considered and reported code coverage (i.e., line coverage, branch coverage) explicitly in their papers, while 82.5% did not consider the code coverage when constructing the test suites for benchmark evaluation. The annual distribution (Figure 33) makes this trend even clearer: although many benchmarks in the past three years (2023–2025) did not consider code coverage, the sheer volume of benchmarks in 2025 means that the absolute number of such benchmarks is high, with 24, 77, and 123 benchmarks respectively for 2023, 2024, and 2025. This underscores that, despite growing awareness of evaluation rigor, a substantial number of benchmarks continue to provide incomplete testing, highlighting a persistent threat to the validity and reliability of benchmark-driven assessment. It severely affects the reliability of findings on these benchmarks, potentially misguiding future research and applications based on these flawed assessments.

4.3 Guideline for Evaluation↩︎

Explanation – Guidelines for benchmark evaluation focus on the rigorousness and reliability of the evaluation. How2Bench provides 12 criteria for benchmark evaluation, as shown in Figure 5. It mainly focuses on the comprehensive evaluation processes for benchmarks involving LLMs. For evaluation design, it stresses the importance of assessing a sufficient and representative range of LLMs to ensure the benchmark’s applicability across various model families and configurations, both open and closed-source. Figure 35 and Figure 37 show the distribution of numbers of LLMs studied and the most exercised LLMs.

Also, prompting has a direct impact on the quality of the LLMs’ output results [36]–[39]. As pointed out by a recent study, up to 40% performance gap could be observed in code translation when prompts vary [40].

Additionally, the experiment environment is essential for reproducibility and transparency. Indeed, the hardware, software, and platform environments used during experiments might influence the outcomes [41]. Furthermore, because of the nondeterministic nature of LLMs, experiments should be repeated, and randomization strategies should be used to mitigate the effects of randomness and parameter configuration biases. Lastly, meticulously documented logs of the experimental process is advised to facilitate transparency and reproducibility, detailing everything from parameter settings to the specific LLM pipelines such as vLLM [42] used.

Key Statistics – Among the 672 benchmarks, 585 of them are evaluated over LLMs. As shown in Figure 35, most benchmarks were evaluated against six LLMs (10.0% = 59 / 585), followed by six LLMs. Encouragingly, increasing LLMs are being studied for code benchmarks. As shown in Figure 36, in 2024, 39.5% of benchmarks were evaluated against fewer than five LLMs, whereas in 2025, 64.2% of benchmarks (36.7% + 27.5%) were evaluated against 5 - 20 models. This shift indicates that benchmark studies are increasingly aiming for more comprehensive and generalizable evaluations. However, it also introduces substantially higher computational and financial costs, highlighting the trade-off between evaluation rigor and resource efficiency.

For reference, we listed the top 10 most studied LLM families in Figure 37. Among them, the GPT series from OpenAI is the most extensively studied, accounting for 76% (446/585), followed by Deepseek and Qwen.

The prompt quality also significantly impacts the LLM evaluation [40]. According to a recent study, up to 40% of performance variation could be observed in the code translation task [40]. So, carefully designing a prompt needs consideration. However, 76.7% representative benchmarks (Appendix 13) do not validate whether the prompts they used are well-designed (Appendix 11.4). Similarly, though 89.4% benchmarks were evaluated in a zero-shot manner, only 18.6% benchmarks were evaluated under few-shot, 3.7% under Chain-of-Thought and 1.0% under RAG (Figure 40). However, as shown in Figure 41, 76.7% representative benchmarks (Appendix 13) do not validate whether the prompt they used is well-designed.

Regarding the evaluation process, our statistics exposed that only 33.4% of benchmark evaluations have been repeated (Appendix 11.4). Also, regarding the transparency and matriculated documents, the observation is not optimistic – Only 6.3% benchmarks provided their experiment environment. More than 38% of benchmarks did not provide reproducible instructions such as prompts, examples for few-shot learning, or content for retrieval (Figure 48). Only half (50.5%) provide hyperparameters such as temperature for reproduction.

4.4 Guideline for Evaluation Analysis↩︎

Explanation – The analysis of the experiment results is expected to be objective and comprehensive, hopefully providing insights or actionable advice. So, we listed 10 criteria for the evaluation analysis phase, as shown in Figure 6. Regarding the perspectives of analysis, inspired by classic measurement theory [30], we suggest four essential perspectives, including difficulty (whether a benchmark is appropriately challenging for LLMs), stability (whether the results are consistent through repeated trials), differentiability (whether benchmarks can differentiate the strengths and weaknesses of various LLMs), and inspiration (e.g., the correlations between the upper-/down-stream coding tasks and LLM scores).

Moreover, effective presentation of results using clear visual and textual descriptions could ensure the findings are understandable and actionable. The phase concludes with the suggestion to interpret and explain the results comprehensively, providing a basis for future enhancements.

Key Statistics – Because experimental analysis is relatively subjective and cannot be obtained through mechanical scanning, we focus on 30 representative focus benchmarks (Appendix 13), covering the highest cited and latest benchmarks in top-5 tasks. Figure 45 shows an example from CruxEval [43] where the experimental scores can hardly be read from the figures.

Also, explaining experiment results is crucial for other practitioners to understand what the outcomes mean in the context of the research questions. According to our statistics (Appendix 11.5), 70% of benchmarks have detailed explanations and analyses of their evaluation results, while still 30% have not. Indeed, an explanation contributes to the body of knowledge by making it possible to understand and compare results with previous studies, promoting transparency within the community.

4.5 Guideline for Benchmark Release↩︎

Explanation – Finally, releasing a benchmark for open access also needs careful consideration. We offered 10 suggestions for this step, as shown in Figure 7, to highlight essential steps for public release preparation, emphasizing accessibility and ethical compliance. This includes setting an appropriate license to clarify usage rights, conducting a thorough review to eliminate sensitive or harmful content such as the API keys to access LLMs, the personal emails or toxic code comments [44] unless they are a part of the benchmark, and ensuring reproducibility by making all related materials openly available. Detailed prompts and clear descriptions of the experimental setup are advised to facilitate replication. Additionally, providing user manuals and evaluation interfaces is crucial for effective user engagement with the benchmark, enhancing its reliability and value for the research community.

Key Statistics – The final step involves the release of the benchmark. The fundamental requirement for releasing a benchmark is that it must be open-sourced. However, surprisingly, we observed that 2.2% of the benchmarks are only partially open-sourced (e.g., missing some subjects or tests), and 14.7% are not open-sourced at all (e.g., links/web pages are no longer active). Furthermore, prompts, which are necessary for reproducibility, are not disclosed in 38.2% of the benchmarks (Figure 48), not to mention the lack of public information on experimental settings (Figure 39 and Figure 38) and experimental parameters (Figure 53). What is worse, 19.3% benchmarks do not set up licenses (Figure 54). The absence of licensing may lead to severe legal and ethical issues, potentially resulting in unauthorized use and distribution of proprietary technologies. Additionally, only 16.7% of the benchmarks make their logged experimental results publicly available (Appendix 11.6). Note that this conclusion about log availability are based on publicly accessible materials; logs are often missing for a combination of policy, privacy, and practicality reasons.

Fortunately, the recent years show a positive shift toward openness (Figure 47). From 2024 to 2025, the number of open-sourced benchmarks increased substantially, rising from 169 to 266, indicating growing community commitment to transparency, accessibility, and reproducibility.

5 Human Study↩︎

To delve deeper into the integration of knowledge and action, we surveyed 49 global researchers in AI (42.6%) and Software Engineering (57.14%), as shown in Figure 61. Each participant had published at least one research paper to ensure their research maturity, and half had constructed code benchmarks. See Appendix 12 for more details about the participants’ demographics and questions in questionnaires.

First, all participants agreed that having a checklist for benchmark construction would contribute to the quality of the benchmark. 47/55 criteria in How2Bench are deemed important by more 80% participants. Additionally, among the 21 participants who have constructed code-related benchmarks, 53 out of 55 criteria were deemed important by all benchmark developers; only two criteria (criteria 3 and 4 in Section 4) were considered unimportant by a few individuals (3 and 2 participants, respectively). Additionally, we received two valuable suggestions that draw importance to recording the time/monetary costs of constructing the benchmark and conducting the experiments.

However, we also identified some notable gaps in awareness. First, regarding the data preparation, more than 15% of participants were not aware that the selection of data should consider the target scope of the evaluation set (i.e., the data must be representative). 16% of participants were unaware of the need for data denoising, while half (8%) of these have already published at least one paper on benchmarking construction. This oversight can significantly affect the validity and generalizability of experimental results, underscoring the importance of a comprehensive understanding of data handling for reliable research outcomes.

Second, regarding evaluation replicability and reliability. Over 40% of participants believe that recording and publicizing the hardware and software environments, software versions, and libraries used in experiments is not important, with more than 20% still considering it unimportant despite already done so. This reveals a significant lack of awareness about the impact that experimental environments can have on the reliability, reproducibility, and stability of evaluation results. In fact, various studies have demonstrated that different experimental environments, parameters, and prompts can lead to substantial variations in outcomes [45]–[47].

6 Alternative Views↩︎

We clarify several seemingly natural but ultimately misleading ways to frame our work. Alternative View 1: Frame our work as a survey of code benchmarks. Our decade-scale analysis of 672 benchmarks may suggest that this paper is primarily a survey. While we do provide quantitative and qualitative observations, our main contribution is normative and prescriptive, not purely descriptive. Rather than cataloging existing work, we (i) expose systematic gaps between awareness and practice, and (ii) propose actionable standards and tooling directions for future benchmark design. Alternative View 2: Frame our work as a criticism of specific benchmarks or communities. Our intent is not to single out particular benchmarks or communities as bad. The issues we study are widespread and systemic. We emphasize structural incentives and use HOW2BENCH to offer a constructive, forward-looking path for community-wide improvement. Alternative View 3: Frame our work as a call for perfectionism or over-engineered benchmarks. Prioritizing rigor, reliability, and reproducibility does not mean that all checklists must be satisfied for a benchmark to be “acceptable”. HOW2BENCH is a structured checklist and prioritization tool, not a rigid standard. It makes trade-offs explicit, helps authors identify the most critical gaps, and supports transparent choices. Our goal is to shift the default toward more principled practices and clearer documentation, not to demand maximal rigor in every dimension.

7 Discussion↩︎

7.1 Trade-offs Between Benchmark Rigor and Development Efficiency↩︎

Despite all the above arguments, we admit that constructing rigorous benchmarks often entails a significant investment of time and human effort, which can lead to reduced efficiency in the development and evaluation process. Indeed, ensuring data quality, implementing thorough validation procedures, and designing comprehensive evaluation protocols requires non-trivial effort and may slow down the pace of research. It also echoed our human study. Through our human study, we found that researchers are often aware of the importance of several criteria (e.g., data denoise and repeating the experiments), but did not implement them due to time constraints or other limitations.

However, this trade-off between rigor and efficiency is necessary to guarantee the reliability, reproducibility, and scientific value of benchmark results. While faster, less rigorous benchmarks might accelerate short-term experimentation, they risk producing misleading or non-generalizable findings that ultimately hinder long-term progress. Therefore, each checklist in How2Bench is labeled with a priority: \(\bigstar\) \(\bigstar\) \(\bigstar\) (Highly important), \(\bigstar\) \(\bigstar\) (Important), and \(\bigstar\) (Optional). We hope the design of this indicator may better help benchmark developers balance efficiency with rigor.

7.2 Awareness and Action↩︎

It is also worth recapping the notable gaps in awareness of the importance of data preparation and reproducibility (Section 5). For example, 16% of participants were unaware of the need for data denoising; 40% of participants believe that recording and publicizing the hardware and software environments, software versions, and libraries used in experiments is not important. However, without being aware of and addressing the quality of LLM benchmarks, the reliability and reproducibility of benchmark results remain compromised. This observation should be a call to action for the research community to strengthen education and awareness around best practices in benchmark development.

7.3 Criteria on Literature Labeling↩︎

Labeling Process: Because the literature scanning and annotation (i.e., Step 2 in the study workflow) involve manual labeling, we summarize the concrete steps we followed when labeling the 672 benchmarks. First, Independent annotation by two primary annotators. For each benchmark paper, two primary annotators independently: (1) Skimmed the full paper and then carefully read sections likely to contain relevant information (e.g., Dataset/Data, Data Collection, Data Curation/Cleaning, Quality Control/Assurance). (2) Assigned each attribute a label from present, not mentioned. (3) Recorded textual evidence (exact sentences or locations) whenever an attribute was labeled as “present”, storing all evidence in a shared annotation sheet. Second, Conflict detection and resolution with additional annotators. After both primary annotators completed their labels, we automatically identified conflicts (e.g., one annotator labeled “has contamination check” as present while the other labeled it as absent). For each conflicted benchmark–attribute pair: The two primary annotators first re‑checked the paper and their evidence. If disagreement remained, two additional annotators joined the discussion to reach a consensus. Finally, Finalization and quality control. Only consensus labels were used in our descriptive statistics. We calculated inter-annotator agreement using Cohen’s kappa; it ranged from 0.76 to 0.92 across attributes.

Labeling Criteria: The manual annotation (e.g., determining whether a paper considers data denoising) follows the annotation criteria: (1) Criteria for checking whether data quality assurance is present: We labeled a benchmark as “considers data quality assurance” only if we identified a dedicated section, subsection, or explicit sentences describing how data quality was ensured. In the absence of such textual evidence, we labeled the benchmark as not considering data quality assurance. Annotators look for explicit discussion of how data quality is ensured. Typical evidence includes sections or sentences tagged as “Data Curation / Cleaning / Preprocessing”, “Quality Control / Quality Assurance”, etc., or detailed descriptions of construction and filtering procedures. (2) Criteria for data quality assurance by manual check: We mark a benchmark as using manual checks if it explicitly mentions human involvement in inspecting or validating data/labels, with keywords such as “manual”, “human”, “inspection”, “verification”, “review”, “spot check”, “random sampling”, “audt”, “sanity check”, “hand‑curated”, “human‑in‑the‑loop”, “expert”, “label correction”, “error analysis”, “case‑by‑case”. (3) Criteria for data quality assurance by code execution: We mark a benchmark as using code execution checks if it describes validating items by actually running code or tests, with keywords such as “execute/execution”, “run”, “compile”, “build”, “unit tests”, “test suite”, “pass/fail”. (4) Criteria for data quality assurance by LLM‑as‑judge: We mark a benchmark as using LLM‑as‑judge if it delegates evaluation or filtering to a model as a judge, with phrases like “LLM‑as‑a‑judge”, “LLM judge”, “model judge”, “[model name] as judge”. (5) Criteria for data contamination check: We mark a benchmark as checking for contamination if it explicitly examines or mitigates training–test overlap. We rely on terms such as: “data contamination”, “leakage”, “overlap”, “Jaccard similarity”, “time‑based split”, “cutoff date”, “post‑YYYY”. (6) Criteria for checking Data deduplication: We mark deduplication as present if the benchmark explicitly states with cues like: “deduplication”, “de‑dup”, “remove duplicates”, “duplicate removal”, “redundant”, “repetitive”. (7) Criteria for checking coverage considerations: For code coverage, we look for mentions of: “coverage”, “test adequacy”, “test strength”, “test suite quality”, ‘corner case(s)’’ or related phrasing.

8 Conclusion↩︎

Awareness of benchmark quality has grown. Yet, we observed a lag between growing awareness and actual practice after surveying 672 code benchmarks. This position paper argues that code benchmarks should prioritize rigor in benchmark construction, reliability in evaluation, and reproducibility in release as first-class objectives; users should actively scrutinize the quality and credibility of the evidence the benchmark designer provides for each claimed category. To operationalize this position, we introduce a code benchmark guideline How2Bench with 55 checklists. We hope this work serves as both a warning and a catalyst, encouraging the community to adopt stronger shared standards.

Acknowledgements↩︎

This work was supported by the National Natural Science Foundation of China (Grant No. 92582201, No. 62402113), the Hong Kong SAR Research Grant Council/General Research Fund (Ref No. 16210725), the Hong Kong SAR Research Grant Council/Theme-based Research Scheme (Ref No. T41-517/25-N), Guangdong Basic and Applied Basic Research Foundation (Grant No. 2024A1515010145), Taighde Éireann – Research Ireland under Grant Number. 13/RC/2094_2, and Research Grants Council of the Hong Kong Special Administrative Region, China (RGC Ref. No. SRFS2425-4S03 of the Senior Research Fellow Scheme).

Impact Statement↩︎

This position paper argues that code benchmarks should prioritize rigor in construction, reliability in evaluation, and reproducibility in release.

Reframing Benchmark Quality By analyzing a decade of code benchmark development (2014–2025), we present evidence that awareness of benchmark quality has not yet fully translated into a more rigorous practice. We argue that benchmark quality should not be treated as an informal best effort, but as a first-class obligation. Our proposed lifecycle-aware framework and 55-item checklist articulate a concrete, actionable standard for what constitutes a methodologically rigorous, reliable, and reproducible benchmark, providing a shared reference point for researchers, reviewers, and benchmark creators.

Shaping Community Norm This work advocates for a shift in community norms away from speed-driven benchmark releases and leaderboard-centric evaluation toward long-term reliability. By making evaluation rigor more explicit and measurable with the checklist (How2Bench), we aim to influence how benchmark papers are reviewed and how research contributions are rewarded.

Broader Influence on Machine Learning Evaluation. Although our empirical analysis focuses on code-related benchmarks, the core position advanced in this paper, i.e., benchmark progress should be constrained by methodological rigor, reliability, and reproducibility, extends to evaluation practices across machine learning, including vision, natural language processing, robotics, and multimodal AI. We hope this work advances a broader shift and serves as a warning toward rigorous evaluation, strengthening the transparency, comparability, and trustworthiness of benchmarks.

9 Appendix↩︎

The appendix is organized as follows:

• Appendix 10 lists the checklist items in each benchmark development phase.

• Appendix 11 lists all the statistics in the survey.

• Appendix 12 explains the details of human study.

• Appendix 13 lists all the benchmarks in focused study.

• Appendix 14 lists all the surveyed benchmarks in a chronological order.

• Appendix 15 shows the complete How2Bench with 55 checklists.

10 Detailed Guidance in How2Bench↩︎

We present the detailed guidance in each phase.

11 Statistics of studied benchmarks↩︎

In this section, we conducted a comprehensive and detailed statistical analysis of the 572 benchmarks collected.

11.1 Profile of Studied Benchmarks↩︎

We first show the trend in the development of benchmarks from 2014 to 2025. As shown in Figure 8, the data shows a modest beginning, with only a handful of benchmarks created annually until 2017. From 2018 onwards, there has been a noticeable uptrend in benchmark creation, culminating in a significant jump to 210 benchmarks in 2024, and 316 benchmarks in 2025. This sharp increase indicates a recent heightened interest and demand for comprehensive code-related benchmarks for LLMs, reflecting the evolving complexities and expanding requirements of automated software engineering.

Hierarchy of Benchmarks. Figure 59 visualizes the inheritance relationships among benchmarks, indicating that the benchmarks on the left serve as sources for those on the right. It highlights that 18% of benchmarks act as data sources, continuously benefiting the construction of subsequent benchmarks.

Figure 59 reveals that HumanEval [7], as the most significant source benchmark, benefits at least 15 downstream benchmarks, followed by MBPP [8] and CodeSearchNet [48]. From the right side of the figure, some benchmarks, like VulBench [10], incorporate methodologies or data from 4 previous benchmarks, and codeRagBench [49] integrates elements from 8 prior benchmarks.

This hierarchical structure among benchmarks also alerts us that the data quality of a benchmark not only affects its own credibility but can continue to impact others if it serves as a source. This underscores the importance of adhering to stringent guidelines during benchmark development and highlights the crucial role of establishing standards to ensure the integrity and utility of benchmark data across research and development efforts.

Coding Task. Regarding the coding tasks, Figure 9 illustrates the distribution of various coding tasks across benchmarks. It is clear that the task of Code Generation is most prevalent, with 235 benchmarks focusing on this area, according to 34.97% (235/672) of studied benchmarks, indicating a significant interest in generating code automatically. The second most prevalent is code reasoning 12.64% (85/672), followed by Program Repair and Defect Detection.

When examining the distribution of coding tasks by year (see Figure 10), we observe that benchmark growth accelerated most rapidly in the past two years (2024–2025), with code generation remaining the most frequently benchmarked task. At the same time, demand for code reasoning has increased substantially, rising from 19 benchmarks in 2024 to 61 in 2025. Benchmarks for program repair have also nearly doubled over the same period, increasing from 22 to 33.

These trends suggest a growing emphasis on LLMs’ reasoning capabilities as well as their role in code development and software maintenance, reflecting an evolving expectation of model competence beyond surface-level code synthesis.

Programming Languages. Figure 11 shows the distribution of benchmarks across various programming languages. The overall trend indicates a strong preference for benchmarking Python, which leads with 409 (71.50%) benchmarks, followed by Java and C++, with 229 and 160, respectively. The graph also reveals a diverse range of languages being used. In total, 67 programming languages are studied by these 672 benchmarks. Though some programming languages, such as Kotlin, Swift, and Scala, are less frequently exercised, the benchmarks involving them are tailored to different application needs and technology environments. This distribution shows the existing benchmarks are dominated by three mainstream programming languages, leaving other programming languages less studied and benchmarked.

When analyzed by year (Figure 12), the number of benchmarks for C/C++ and JavaScript has increased noticeably between 2024 and 2025. We also observe a sharp rise in Rust-related benchmarks: although only a few benchmarks existed for Rust since its release in 2015, 57 benchmarks were published in the past two years (27 in 2024 and 29 in 2025). These trends in programming language coverage reflect the evolving demand for LLMs across different languages, highlighting particularly the growing reliance on large models to generate, reason about, and maintain code in these languages.

Natural Language. Figure 13 illustrates the distribution of benchmarks for different natural languages. The bar chart overwhelmingly shows that English is the dominant language, with 577 (85.9%) benchmarks highlighting its ubiquity in research and development. Other languages have significantly fewer benchmarks, with 16 in Chinese and 15 in Russian. The category labeled “Other” includes 24 benchmarks spread across other natural languages, indicating some diversity but limited attention to non-English benchmarks. This distribution highlights the prominence of English in the global research community and also demonstrates the uneven representation of natural languages in the studied benchmarks.

Modals in the benchmarks. Figure 14 presents the distribution of benchmarks according to the type of language used in their prompts. The chart shows that the majority, at 66.2%, of the benchmarks use a combination of natural language and programming Language, followed by PL only (13.7%) and NL only (18.2%).

Granularity. The code snippet in a code-related benchmark varies from statement-level (i.e., one line of code. For example, CoNaLa [50] and Math-QA [51]), function-level (i.e., a function unit of code. For example, HumanEval [7] and MBPP [8]), class-level (i.e., a class with multiple function units of code. For example, ClassEval [52]) and project-level (i.e., a project with multiple classes or modules. For example, DevEval [53] and JavaBench [32]).

Figure 15 illustrates the granularity levels at which benchmarks are typically conducted. The chart shows that the majority of benchmarks, comprising 66.9%, focus on the function level. Projects constitute 21.5% of the benchmarks. Class-level granularity is the least represented at only 3.4%.

When analyzed by year, an interesting phenomenon emerges (Figure 16): between 2024 and 2025, the number of project-level benchmarks surged, increasing from 35 to 95 new benchmarks. Importantly, these numbers refer to benchmarks newly introduced in each year, rather than cumulative totals. This surge indicates a growing community focus on the real-world applicability and large-scale practical utility of LLMs.

11.2 Statistics about Benchmark Design↩︎

Design of Studied Capabilities. To understand whether benchmark developers recognize the capabilities of LLMs they aim to evaluate, we carefully analyzed 30 representative benchmarks (Appendix 13) to see if they clearly specify the capabilities being assessed by their benchmarks. As shown in Figure 17, 80% of benchmarks explicitly specify the capabilities (e.g., intention understanding, problem solving, testing, debugging capabilities)to be evaluated, while 20% do not. The statistics show that the most highly cited benchmarks clearly define the assessment capabilities.

Furthermore, we investigated the 30 focused benchmarks and identified a case (Figure 18) from MBPP [8] where the case is likely to fall outside of the targeted capability of the benchmark. In particular, MBPP [8] aims to “measure the ability of these models to synthesize short Python programs from natural language descriptions” for “entry-level programmers”. As we can see from Figure 18, the prompt requires LLMs to “Write a function to calculate the dogs’ years.” Simply from this description, an entry-level programmer is unlikely to write a correct program without knowing the conversion equation from dogs’ year to dogs’ age. In other words, this case is more about assessing whether LLMs have acquired this specific knowledge rather than evaluating the most fundamental programming skills.

Design of Studied Application Scenarios. Similarly, to understand whether benchmark developers scoped the application scenarios of LLMs they aim to evaluate, we carefully analyzed 30 representative benchmarks (Appendix 13) to see whether they explicitly specify the application scenarios their benchmarks target. As shown in Figure 19, 76.7% representative benchmarks have clearly specified application scenarios (e.g., programming assistant), while the rest do not. Indeed, clearly defining the application scenarios could help benchmark constructors establish precise goals for the design and development of the benchmark, ensuring accuracy in the evaluation.

11.3 Statistics about Data Preparation↩︎

11.3.1 Data Preprocessing↩︎

Data Deduplication. During benchmark preparation, data cleaning and preprocessing are necessary. However, as shown in Figure 20, only 32.9% benchmarks have deduplicated the collected data. More than half of them didn’t mention this process.

The yearly trend (Figure 21) shows that, although the number of benchmarks that applied deduplication increased slightly from 71 in 2024 to 93 in 2025, the total number of benchmarks in 2025 more than doubled compared to 2024. As a result, the absolute number of benchmarks that ignored or did not apply deduplication nearly doubled, highlighting a growing risk of duplicated data despite increased awareness.

To investigate the situation, we went through the 30 representative benchmarks (Listed in Appendix 13) and found two duplicated subjects in MBPP [8]. Tasks with id 71 and 141 examined the same functionality, i.e., “Write a function to sort a list of elements.”, collected from the same source.

Data Quality Assurance. Ensuring data quality for the benchmark is essential. However, our statistics (Figure 23) show disappointing results. 46.0% of benchmarks do not take any measures for data quality assurance. Among those benchmarks that do incorporate data quality measures, the majority rely on manual checks, which account for 45.8%. Other countermeasurements, such as code execution, constitute only 1.2%. Additional methods, such as using download counts as a basis, are also employed (6.4%) .

Examining the trends by year (Figure 24), one positive development is the increasing prevalence of manual quality checks. The number of benchmarks with manual check guarantees doubled from 84 in 2024 to 193 in 2025, indicating that a growing share of code benchmarks now undergo partial or full human verification.

Additionally, we dived into the 30 representative benchmarks (Listed in Appendix 13) and identified an example where the code cannot be executed successfully. As shown in Figure 25, the function swap() in line 7 has not been defined, so the execution of the code would fail if the code has been executed. This highlights a significant gap in ensuring the reliability and validity of benchmark data, underscoring the need for more rigorous and automated data quality assurance practices.

Data Contamination Resolution. Data contamination [6], [54] threat has been widely discussed. A benchmark with contaminated data may yield overclaimed results, misleading the understanding of the LLMs’ capabilities. According to our statistics (Figure 26), most (79.8%) benchmarks were not aware of and have not taken any measures to alleviate data contamination, being vulnerable to data contamination threats. The yearly trend (Figure 27) further highlights the persistent neglect of data contamination in benchmark construction. From 2024 to 2025, the number of benchmarks that did not account for data contamination nearly doubled, increasing from 165 to 236, indicating that this risk persists or has become more widespread.

11.3.2 Statistics about Data Curation↩︎

Ground truth solutions. Figure 28 shows that although the majority (89.1%) of benchmarks provide reference code as ground truth, there are 9.8% of benchmarks without reference code. Although it is not compulsory as long as object measurements (e.g., test cases) are provided, a reference code is still recommended. Indeed, if a benchmark provides reference code, its reliability tends to be better because it ensures that there are feasible solutions for the tasks involved. This guarantees that the tasks are theoretically and practically solvable, enhancing the benchmark’s usefulness and credibility in real-world applications.

From the year-by-year trend (Figure 29), the number of benchmarks without ground truth tripled in 2025, reaching 44 benchmarks. This may reflect a shift toward more open-ended and challenging benchmarks, where standard solutions do not yet exist or remain to be explored, highlighting the evolving nature of tasks designed for LLM evaluation.

Additionally, the correctness of the ground truth solution should also be noted. Figure 30 shows an incorrect code solution provided in HumanEval [7]. This should draw benchmark constructors’ attention to the correctness of the benchmark reference code.

Oracle. An oracle [55] is a way to determine whether the output is correct or not. For example, assume the output of LLMs is in the form of code, then an oracle could be running tests against the code and see whether the code can pass all the tests. Figure 31 shows the distribution of types of oracle that are used in these benchmarks. Clear that passing test cases (257 / 672 = 38.2%) and the exact match (193 / 672 = 28.7%) are the most common oracles used in code benchmarks, followed by thresholds (i.e., similarities smaller than a specific threshold).

Code coverage [56], as a common oracle for code-related benchmarks, measuring the ratio of covered code by tests, has been widely adopted to determine the output correctness. It should be considered whether a benchmark uses test case passing as a criterion for the correctness of the generated code. Otherwise, a test could be too weak to detect the existence of a defect in the generated code. For example, as pointed out by prior work [3], existing benchmarks such as HumanEval [7] and MBPP [8] still suffer from “insufficient tests”, allowing incorrect code to pass all the tests without capturing the bugs.

Despite its importance, as shown in Figure 32, among the benchmarks that use test cases as the oracle, only 13.6% considered and reported “test coverage” explicitly in their papers, while 85.0% ignored the test coverage.

The annual distribution (Figure 33) makes this trend even clearer: although many benchmarks in the past three years (2023–2025) did not consider code coverage, the sheer volume of benchmarks in 2025 means that the absolute number of such benchmarks is high, with 24, 77, and 123 benchmarks respectively for 2023, 2024, and 2025. This underscores that, despite growing awareness of evaluation rigor, a substantial number of benchmarks continue to provide incomplete testing, highlighting a persistent threat to the validity and reliability of benchmark-driven assessment.

Furthermore, we dived into 30 representative benchmarks (Listed in Appendix 13) and identified an example (Figure 34) from MBPP [8] where the test is incorrect. It alerts us that both the quality of the test and the test adequacy (e.g., code coverage) should be considered.

11.4 Statistics about Evaluation↩︎

Studied LLMs. We summarize the number of LLMs that have been evaluated in each benchmark evaluation. Among the 672 benchmarks, 585 of them are evaluated over LLMs, so we show the statistics over them. As shown in Figure 35, most benchmarks were evaluated against six LLMs (10.0% = 59 / 585), followed by three LLMs.

A positive trend also emerges when examining the year-by-year distribution (Figure 36). In 2024, 39.5% of benchmarks were evaluated against fewer than five LLMs, whereas in 2025, 64.2% of benchmarks (36.7% + 27.5%) were evaluated against 5–20 models. This shift indicates that benchmark studies are increasingly aiming for more comprehensive and generalizable evaluations. However, it also introduces substantially higher computational and financial costs, highlighting the trade-off between evaluation rigor and resource efficiency.

Additionally, we listed the top-10 LLMs by the number of code-related benchmarks they have been evaluated, as shown in Figure 37. GPT series leads significantly with 446 benchmarks, suggesting its widespread adoption and possibly its versatility or superior performance in handling code-related tasks. The rest, including DeepSeek, Qwen, and others, show varying degrees of involvement, with numbers ranging from 269 down to 64 benchmarks for Claude. This figure may provide a reference for choosing which model to evaluate. In addition, it is worth mentioning that different LLMs should be considered for different coding tasks.

Experiment Environments. The experimental environment (such as the operating system and hardware) is important for the reproduction of the experiment. However, Figure 39 and Figure 38 highlight a significant gap. Only 24.6% of benchmarks document the devices used in their experiments, leaving a substantial 75.4% that do not. The situation appears even more dire when considering os, with only 6.3% of benchmarks documenting the OS used, while a staggering 93.7% neglect to record this information.

Prompting and Prompting Strategies Prompting has a direct impact on the quality of the LLMs’ output results [36]–[38]. So, we summarized whether different prompting strategies have been evaluated and statistics the distribution. Figure 40 shows the usage of four kinds of prompts: zero-shot, few-shot, chain-of-thought, and retrievals (RAG). From Figure 40, we can see that a vast majority (89.4%) of benchmarks were evaluated in a zero-shot context setting, while only 18.6% of benchmarks were evaluated in a few-shot manner. Even fewer benchmarks were evaluated under the Chain-Of-Thought (CoT) and RAG settings, utilized by only 3.7% and 1.0%.

Prompt Quality The prompt quality also greatly impacts the LLM evaluation [40]. So, carefully designing a prompt needs consideration. However, as shown in Figure 41, 76.7% representative benchmarks (Appendix 13) do not validate whether the prompt they used is well-designed.

Repeated Experiment Given the random nature of LLMs, the experiments are expected to repeat, ensuring the stability and reliability of the results. However, as shown in Figure 42, only 33.4% benchmarks went through a repeated experiment, while a majority of 66.6% opted against repeating the experiment. This reflects a lack of awareness regarding the stability and reproducibility of evaluations.

Examining trends year by year (Figure 43), the number of benchmark developers who repeated experiments increased in 2025 compared to 2023 and 2024 (31 in 2023, 79 in 2024, and 111 in 2025). However, because the total number of benchmarks grew substantially in 2024 and 2025, the absolute number of benchmarks without repeated experiments remains the highest, totaling 205 benchmarks, which accounts for 30.5% (205/672) of all benchmarks. This indicates that, despite increasing awareness of reproducibility, a significant fraction of benchmarks still lack repeated verification, posing a persistent risk to evaluation reliability.

11.5 Statistics about Analysis↩︎

Experiment Explanation. Explaining experiment results is crucial for other practitioners to understand what the outcomes mean in the context of the research questions. So, we investigate whether the representative benchmarks (Appendix 13) have explained the experiment results. As shown in Figure 44, 63.3% benchmarks have detailed explanations and analyses of their evaluation results, while still 36.7% have not. Indeed, an explanation contributes to the body of knowledge by making it possible to understand and compare results with previous studies, promoting transparency within the community.

A clear and precise presentation of experimental results is important for enabling robust interpretation and comparison across benchmarks. However, further examination of the 30 representative benchmarks (listed in Appendix 13) revealed notable deficiencies in result visualization. As shown in Figure 45, CruxEval [43] exhibits unclear experimental result presentation. Specifically, the scatter plot suffers from ambiguous labeling, poor readability of axis values, and inconsistent marker encoding, making it difficult for researchers to extract meaningful insights. Such presentation shortcomings obscure the performance relationships between methods and compromise the benchmark’s usability for fair evaluation. To address these issues, benchmarks should adopt standardized and well-documented visualization practices, ensuring results are interpretable, accessible, and reproducible.

11.6 Statistics about Release↩︎

Data Accessibility. The fundamental requirement for releasing a benchmark is that it must be open-sourced. However, surprisingly, as shown in Figure 46, we observed that 2.2% of the benchmarks are only partially open-sourced (e.g., missing some subjects or tests), and 14.7% are not open-sourced at all (e.g., links/web pages are no longer active).

Fortunately, the year-by-year trend (Figure 47) shows a positive shift toward openness. From 2024 to 2025, the number of open-sourced benchmarks increased substantially, rising from 169 to 266, indicating growing community commitment to transparency, accessibility, and reproducibility.

Prompt Accessibility. Detailed prompts are essential for ensuring the reproducibility and transparency of code-related benchmarks. Yet, Figure 48 indicates that 38.2% of benchmarks do not provide detailed prompts, limiting the ability to accurately replicate and evaluate the performance of LLMs. Lack of prompt disclosure highlights a gap in benchmark design practices, as prompts are often indispensable for understanding model performance under specific conditions, raising concerns about the consistency and reproducibility of reported results.

When examined on a year-by-year basis Figure 49, from 2024 to 2025, although the number of benchmarks that release prompts increased from 140 to 233, the number of benchmarks that do not release prompts also grew, from 64 to 82. This indicates that transparency has improved in absolute terms, but has not kept pace with the rapid growth in benchmark releases.

Logging Info Accessibility. Providing detailed logging information, including comprehensive experimental results, is essential for ensuring transparency, verifiability, and reproducibility in benchmarking research. However, as shown in Figure 50, only 16.7% of the benchmarks make their experimental results publicly available, while 80.0% fail to disclose this critical information. Alarmingly, an additional 3.3% provide only partial logging details, further complicating result verification. The absence of complete logging information creates significant barriers for researchers attempting to reproduce experiments or validate reported findings, thereby undermining the reliability of benchmarks. To address this, we emphasize the necessity of making detailed logging information, including intermediate results and metrics, publicly accessible to uphold rigorous scientific standards and foster trustworthy comparisons across models.

User Manual Accessibility. A high-quality user manual, such as a well-documented README file, is crucial for enhancing benchmark usability, enabling users to understand the dataset, execute provided scripts, and reproduce results efficiently. However, our analysis revealed that a significant number of benchmarks lack comprehensive user manuals, hindering accessibility and adoption. As depicted in Figure 51, poorly structured or incomplete manuals often omit essential components such as benchmark introductions, usage instructions, and evaluation scripts. This creates unnecessary barriers for researchers who rely on these manuals for setup and experimentation. To address this, we advocate for benchmarks to include clear, standardized user manuals that provide an overview of the benchmark, step-by-step execution guides, and troubleshooting instructions, ensuring a seamless and reproducible user experience.

Convenient Evaluation Interface Availability.

Providing convenient evaluation interfaces is essential for enhancing the usability and accessibility of benchmarks, enabling researchers to easily reproduce results and compare models. As shown in Figure 52, 20% of benchmarks fail to offer any evaluation interfaces, imposing significant barriers to usability. While a majority of benchmarks (80%) provide such interfaces, including command-line tools, Docker images, or scripts, the absence of standardized and user-friendly evaluation tools in a notable minority of cases highlights an area for improvement. Benchmarks without convenient evaluation interfaces require users to spend additional effort in setup and result verification, which can discourage adoption and hinder reproducibility. To address this, we emphasize the importance of releasing benchmarks with well-documented, ready-to-use evaluation pipelines to promote efficient, reliable, and fair benchmarking practices.

Temperature Records. One critical parameter for benchmarking is the temperature setting, which influences stochasticity in LLMs. As shown in Figure 53, we observed that 49.5% of benchmarks fail to record the temperature setting, hindering reproducibility and fair evaluation. While 50.5% of benchmarks do document this parameter, the majority omission highlights an overlooked yet essential aspect of benchmark transparency.

License Provision. Releasing benchmarks under a clear and accessible license is fundamental for legal compliance and ensuring open collaboration. Figure 54 reveals that 19.3% of benchmarks do not provide a license, limiting their usability and distribution. Encouragingly, 80.7% of benchmarks do include a license, but the lack of licensing in nearly one-fifth of the benchmarks raises concerns about widespread adoption and usage. These findings emphasize the need for standardized practices in benchmark releases to promote legal clarity and accessibility.

Data Security. Ensuring data security is a critical yet often overlooked aspect of benchmark development. Sensitive information, such as API keys, credentials, or private tokens, should never be included in benchmark releases. However, further investigation into 30 representative benchmarks (listed in Appendix 13) revealed instances of sensitive data leakage. As shown in Figure 55, XSemPLR [57] inadvertently included an API key in its release, a critical oversight that can expose resources to external exploitation.

Similarly, Figure 56 highlights an example from CrossVul [58], where personal names and email addresses were unintentionally disclosed. Such leakage poses risks of unauthorized access and resource misuse, potentially compromising systems and research integrity.

Usability. Clear and comprehensive documentation is crucial for ensuring the usability of benchmarks, as poorly written instructions can significantly hinder adoption and reproducibility. We dived into the 30 representative benchmarks (listed in Appendix 13) and identified an example where the README file provided insufficient and unclear information. As shown in Figure 57, VulDeePecker [59] includes a less-than-ideal ReadMe file, which lacks essential usage instructions and evaluation guidelines, making the benchmark difficult to understand and deploy.

In contrast, Figure 58 highlights APPS [60], which provides well-structured and easy-to-follow documentation. The APPS benchmark includes step-by-step instructions for generating, evaluating, and analyzing results, enabling users to efficiently reproduce experiments.

These observations emphasize the importance of high-quality documentation for benchmarks to enhance accessibility, reduce friction in usage, and foster reproducible research.

12 Details of Human Study↩︎

12.1 Interviewee Selection↩︎

The selection of interviewees is pivotal to ensuring the representativeness and relevance of the data collected. This involves identifying individuals with the knowledge or experience pertinent to the research theme.

To this end, we chose graduate students from SE or AI fields who have published at least one paper. This criterion ensures that participants have research experience and judgment capabilities. The focus on SE and AI fields is due to their likely interest in code benchmarks. Particularly, we aimed to recruit individuals who have published papers on code benchmarks to obtain firsthand feedback from experienced benchmark developers.

12.2 Survey Question Design↩︎

Questions. The body of the survey was divided into five stages of benchmark development (following Figure 1), with necessary background information provided for each stage. Each criterion in How2Bench was slightly modified to be in the first-person perspective, making it easier for interviewees to empathize and answer the questions from their own viewpoint. Finally, to facilitate comprehension, questions and instructions were translated into both English and Chinese.

Question Setting. To minimize the effort required from respondents, we designed single-choice questions with four options:

I found it important, and I have done it.

I found it important, although I haven’t done it.

I found it not important, but I have done it.

I found it not important, and I wouldn’t do it.

This format is intended to orthogonally explore the correlation between awareness and behavior.

12.3 Interview Process↩︎

Questionnaire Distribution The questionnaire was distributed via online platforms, targeting academic and professional networks related to SE and AI. The distribution started on October 27, 2024, and ended on November 27th, 2024, lasting one month.

Results Collection The responses were automatically collected through the online platform used for distribution.

Survey Screening Since the requirement was for participants who have published papers, responses from those selecting “No” to having published a paper were excluded. Also, incomplete surveys where not all questions were answered were also considered invalid and excluded from the analysis.

12.4 Interview Result Analysis↩︎

In total, we collected 50 responses. The respondents were from seven regions, including the United States, the United Kingdom, Germany, Australia, China, and others, as shown in Figure 60. Only one survey was invalid due to the respondent selecting “have not published a paper”, leaving 49 valid surveys for analysis. A breakdown of the respondents’ demographics is shown in Figure 61. The detailed responses for all 55 criteria in How2Bench are shown in Figure 62 and Figure 63.

13 List of Studied Benchmarks (Focused Ones)↩︎

Code Generation: Five with top citations:

• HumanEval [7]

• MBPP [8]

• CodeContest [61]

• leetcodehardgym [62]

• APPS [60]

The latest one as of 31/12/2025:

• CIFE [63]

Defect Detection: Five with top citations:

• VulDeePecker [59]

• Devign [64]

• Chromium and Debian [65]

• \(\mu\)VulDeePecker [66]

• Synthetic Dataset [67]

The latest one as of 31/12/2025:

• Pushkar et al. [68]

Program Repair: Five with top citations:

• Defects4J [9]

• BFP [69]

• MANYBUGS, INTROCLASS [70]

• HumanEval-Java [71]

• QuixBugs [72]

The latest one as of 31/12/2025:

• BackportBench [73]

Code Summarization: Five with top citations:

• CODE-NN [74]

• Java-small/med/large [75]

• code-summarization-public [76]

• HumanEvalPack [77]

• Shrivastava et al. [78]

The latest one as of 31/12/2025:

• ContextCRBench [79]

Code Reasoning: Five with top citations:

• LiveCodeBench [80]

• CRUXEval-X [81]

• xCodeEval [82]

• CodeQA [83]

• RepoQA [84]

The latest one as of 31/12/2025:

• RE2-Bench [85]

14 List of Studied Benchmarks (Full)↩︎

We collected and studied 672 code-related benchmarks. We then listed and grouped them by year.

2025:

• Web-Bench [86]

• EFFIBENCH-X [87]

• EDIT-Bench [88]

• CIRCLE [89]

• CS101-Gold [90]

• OSS-Bench  [91]

• CodeArena  [92]

• RefactorCoderQA  [93]

• SwiftEval  [94]

• CoRe  [95]

• CodeElo  [96]

• LongCodeBench  [97]

• DSCodeBench  [98]

• RustEvo2̂  [99]

• JITVUL  [100]

• UA-Code-Bench  [101]

• CodeJudgeBench  [102]

• HumanEval-Judge, MBPP-Judge, BigCodeBench-Judge  [103]

• ContextCRBench  [79]

• LoCoBench  [104]

• CodeMixBench  [105]

• VERINA  [106]

• SecureAgentBench  [107]

• ClassEval-T  [108]

• ProjectEval  [109]

• IFEvalCode  [110]

• CodeARC  [111]

• TestGenEval  [112]

• CodePromptEval  [113]

• FreshBrew  [114]

• CLOVER  [115]

• THROWBENCH  [116]

• QuanBench  [117]

• FrontendBench  [118]

• SolEval  [119]

• SWE-Bench-Live  [120]

• SWE-Bench++  [121]

• Defects4Log  [122]

• SecVulEval  [123]

• PATCHEVAL  [124]

• VulnRepairEval  [125]

• FullStackBench  [126]

• CodeIF  [127]

• AutoCodeBench  [128]

• TransLibEval  [129]

• SWE-Perf  [130]

• TRACY  [131]

• UnLeakedTestbench  [132]

• TypyBench  [133]

• LogicCat  [134]

• BIRD-INTERACT  [135]

• SQL-Synth  [136]

• Falcon  [137]

• CORGI  [138]

• GITS-Eval  [139]

• CWEval  [140]

• DI-BENCH  [141]

• LINUXFLBENCH  [142]

• ClassEval  [52]

• Deep-Bench  [143]

• MCMD  [144]

• HumanEvalNext  [145]

• FEA-Bench  [146]

• SolBench  [147]

• DEFECTS4J-TRANS  [148]

• CASTLE  [149]

• CodeIF-Bench  [150]

• Multi-SWE-bench  [151]

• SciReplicate-Bench  [152]

• SWE-PolyBench  [153]

• APIRAT  [154]

• LeetCodeDataset  [155]

• CoCo-Bench  [156]

• SecRepoBench  [157]

• CodeFlowBench  [158]

• SWE-smith [159]

• WebGen-Bench [160]

• SWE-rebench [161]

• ResearchCodeBench [162]

• WebUIBench [163]

• SWE-Factory [164]

• SafeGenBench [165]

• Zeng et al.  [166]

• SWE-MERA [167]

• F2STRANS [168]

• VulCoCo [169]

• NoCode-bench [170]

• MRG-Bench [171]

• STEPWISE-CODEX-Bench [172]

• CodeFuse-CR-Bench [173]

• SWE-Mirror [174]

• SWE-Bench-Pro [175]

• MultiSpider-2.0 [176]

• MULocBench [177]

• TC-Bench [178]

• Defects4C  [179]

• GDPR-Bench-Android [180]

• PRDBench [181]

• CodeAlignBench [182]

• Go-UT-Bench [183]

• RGym [184]

• CodeFuse-CommitEval [185]

• PACIFIC [186]

• NL2Repo-Bench [187]

• RE2-Bench [85]

• CIFE [63]

• SWE-EVO [188]

• Multi-Docker-Eval [189]

• CodeCriticBench [190]

• Cui et al.  [191]

• CodeReviewQA [192]

• BigO(Bench) [193]

• SWR-Bench [194]

• UniCode [195]

• SusVibes [196]

• COFFE [197]

• DependEval [198]

• SWA-Bench, SWEE-Bench  [199]

• CVE-Bench [200]

• Obscura [201]

• BinaryLLMs-Eval [202]

• CodeAssistBench [203]

• SWE-QA [204]

• RECODE-H [205]

• CoReQA [206]

• HumanEval-V [207]

• CAMA  [208]

• C2RUST-BENCH [209]

• CRUST-Bench [210]

• OSVBench [211]

• VADER [212]

• CodeSense [213]

• CETBench [214]

• DesignBench [215]

• CoQuIR [216]

• MultiCodeIF [217]

• CoreCodeBench [218]

• WebMMU [219]

• ProjectAnalyzer [220]

• VulGate [221]

• text2SQL4PM [222]

• E2EDev [223]

• Chart2Code [224]

• RealClassEval [225]

• BackPortBench [73]

• Sun et al. [226]

• Pushkar et al. [227]

• RepoTransBench [228]

• DomainCodeBench [229]

• UniVul [230]

• mHumanEval [231]

• WebCode2M [232]

• CodeJudge-Eval [233]

• HumanEval-XL [234]

• Visual-SWE-bench [235]

2024:

• CodeEditorBench [236]

• MHPP [237]

• LiveCodeBench [80]

• CodeAgentBench [238]

• CruxEval [43]

• BigCodeBench [239]

• OOPEval [240]

• DevEval [53]

• Long Code Arena [241]

• CodeRAGBench [49]

• ScenEval [242]

• AICoderEval [243]

• VersiCode [244]

• VHDL-Eval [245]

• NaturalCodeBench [246]

• CodeGuard+ [247]

• PECC [248]

• USACO [249]

• ParEval [250]

• MxEval [251]

• MMCode [252]

• Plot2Code [253]

• ChartMimic [254]

• DebugBench [255]

• PythonIO [256]

• StaCCQA [257]

• RepoQA [84]

• PRIMEVUL [258]

• VulDetectBench [11]

• ProCQA [259]

• CoSQA+ [260] [261]

• JavaBench [32]

• HumanEvo [262]

• REPOEXEC [263]

• EHR-SeqSQL [264]

• BookSQL [265]

• AMBROSIA [266]

• WUB, WCGB [267]

• RES-Q [268]

• PythonSaga [269]

• Mercury [270]

• ENAMEL [271]

• RealHumanEval [272]

• CoderUJB [273]

• EvoEval [274]

• ML-Bench [275]

• VerilogEval [276]

• CodeApex [277]

• HumanEvalPack [77]

• HumanEval+ [278]

• HumanEval-X [279]

• XCodeEval [82]

• CoderEval [280]

• CodeXGLUE [281]

• VulnPatchPairs [12]

• WikiSQL [282]

• CrossCodeEval [283]

• SWE-bench [1]

• BAIRI et al. [284]

• BioCoder [285]

• RepoBench [286]

• NoFunEval [287]

• CoCoMIC [288]

• Java-small/med/large [75]

• FixEval [289]

• CommitBench [290]

• InfiAgent-DABench [291]

• InfiBench [292]

• Design2Code [293]

• MatPlotBench [294]

• EditEval [295]

• D1, D2, D3 [296]

• RepoEval [297]

• BetterTypes4Py, InferTypes4Py [298]

• HumanEval-Java [71]

• PIE [299]

• EvalGPTFix [300]

• EHRSQL [301]

• Spider2-V [302]

• TESTEVAL [303]

• ChatTester [304]

• Code Lingua [305]

• EffiBench [306]

• CRUXEval-X [81]

• DomainEval [307]

• SWE-Bench+  [308]

• eyeballvul  [309]

• ComplexCodeEval  [310]

• BabelBench  [311]

• CRQBench  [312]

• R2C2-Coder  [313]

• SAFIM  [314]

• McEval  [315]

• PolyHumanEval  [316]

• RustRepoTrans  [317]

• RAPID  [318]

• BEAVER  [319]

• Spider2.0  [320]

• Wu et al. [321]

• BigTable-0.2k [322]

• SWT-Bench [323]

• CodeSecEval [324]

• SWE-bench-java [325]

• TypeEvalPy/PyCG [326]

• M²RC-EVAL [327]

• MdEval [328]

• CPP-UT-Bench [329]

• OBFUSEVAL [330]

• ExecRepoBench [331]

• VulnLLMEval [332]

• TestBench [333]

• Codev-Bench [334]

• SWE-bench Multimodal [335]

• HumanEval-V [336]

• HumanEval_T [337]

• TDD-Bench Verified [338]

2023:

• MCoNaLa [339]

• MultiPL-E [340]

• ODEX [341]

• TACO [342]

• DOTPROMPTS, MGDMICROBENCH [343]

• StudentEval [344]

• CodeTransOcean [345]

• G-TransEval [346]

• AVATAR [347]

• RunBugRun [348]

• VulBench [10]

• DiverseVul [349]

• Hellendoorn et al. [67]

• XSemPLR [57]

• BIRD [350]

• Stack-Repo [351]

• RepoEval [297]

• MTPB [352]

• ARCADE [353]

• Shrivastava et al. [78]

• Grag et al. [354]

• GSM-HARD [355]

• InferredBugs [356]

• LeetcodeHardGym [62]

• APIBench [357]

• ClassEval [52]

• CommitChronicle [358]

• TeCo [359]

• TESTPILOT [360]

2022:

• AixBench [361]

• TypeBugs [362]

• XLCoST [363]

• CS1QA [364]

• Chromium and Debian [65]

• Spider-Realistic [365]

• Spider-SS [366]

• DSP [367]

• CodeContest [61]

• PandasEval, NumpyEval [368]

• TorchDataEval, MonkeyEval, BeatNumEval [369]

• DS-1000 [370]

• MCMD [144]

• ExeDS [371]

• QuixBugs [72]

• ManyTypes4Py v0.7 [372]

• Lyra  [373]

2021:

• SySeVR [374]

• Ling&Wu et al. [375]

• Chen et al. [376]

• MBPP, MathQA-Python [8]

• HumanEval [7]

• APPS [60]

• Berabi et al. [377]

• CrossVul [58]

• PYPIBUGS, RANDOMBUGS [378]

• D2A [379]

• CodeQA [83]

• Spider-DK [380]

• KaggleDBQA [381]

• SEDE [382]

• Spider-Syn [383]

• CoDesc [384]

• Methods2Test [385]

• Rozière et al. [386]

2020:

• Lachaux&Roziere et al. [387]

• \(\mu\)VulDeePecker [66]

• CosBench [388]

• PACS [389]

• Criteria2SQL [390]

• SQUALL [391]

• Hu et al. [392]

• CodeSearchNet Challenge [48]

• MIMICSQL [393]

• Atlas [394]

• Liu et al. [395]

• Android [396]

• CCSD [397]

2019:

• BFP [69]

• SARD [398]

• Spider [399]

• JuICe [400]

• Nguyen et al. [401]

• Lin et al. [402]

• Zhou et al. [64]

• CoSQL [403]

• SParC [404]

• Malik [405]

• LeClair [406]

2018:

• CoNaLa [50]

• DeepCom [407]

• TL-CodeSum [408]

• code-summarization-public [76]

• Russell et al. [409]

• VulDeePecker [59]

• Lin et al. [410]

• StaQC [411]

• Advising [412]

• ConCode [413]

• NNGen [414]

• Gu et al. [415]

2017:

• QuixBugs [416]

• the DeepFix dataset [417]

• Barone et al. [418]

2016:

• CODE-NN [74]

• Mou et al. [419]

2015:

• MANYBUGS, INTROCLASS [70]

2014:

• Defects4j [9]

• BigCloneBench [420]

15 Guideline↩︎

Finally, for ease of printing and use, we organized the guideline How2Bench into a clear, color-coded checklist (4 pages in total) that is easy to print, attached at the end of the paper.

References↩︎