1 Introduction↩︎

During the past decade, automated program repair (APR) has made remarkable progress, further accelerated by the advent of agentic coding frameworks [1]–[3]. Benchmarks such as Defects4J [4], QuixBugs [5], and SWE-Bench [6] have been instrumental in driving these advances, providing standardized tasks and reproducible environments for evaluating APR techniques. However, these benchmarks are almost exclusively focused on functional bugs, leaving non-functional issues, and performance issues in particular, underrepresented.

Recently, researchers have turned their attention to automated fixing of performance issues [7]–[10]. This new trend has led to the creation of benchmarks of execution time improvement patches, such as PIE [11], SWE-Perf [12], GSO [13], and PerfBench [14]. These benchmarks target languages such as Python, C++, and .NET. Despite being one of the most widely used programming languages, Java remains underrepresented in this area. Note that Java’s reliance on just-in-time compilation and garbage collection makes execution time measurements volatile, demanding more rigorous benchmarking methodology than is needed for languages like Python or C++. The only existing benchmark of execution time improvement patches in real-world Java projects is recently proposed by Yi et al. [15], which includes 65 tasks from four repositories. This dataset presents a fixed set of patches without containerized environments, making it difficult to reproduce results as project dependencies evolve over time. Beyond the language gap, to the best of our knowledge, existing performance improvement benchmarks have a common limitation: these datasets cannot be extended to collect new real-world patches according to user-defined configurations.

In this paper, we present JETO-Mine, the first configurable and reusable tool for creating reproducible benchmarks of real-world execution time improvement patches (ETIPs) in Java. JETO-Mine provides a fully automated pipeline that researchers can run with their own desired filters and configurations to continuously collect new benchmarks. The pipeline consists of three phases. First, a static analysis phase crawls GitHub repositories and identifies commits and PRs that address execution time issues, using user-defined filters and an LLM-based issue classifier. Second, a dynamic analysis phase wraps the identified ETIPs in Docker images, building and executing the project tests on both original and patched versions to create fully reproducible environments. In this phase, JETO-Mine also runs the existing tests in the project multiple times and conducts statistical testing with user configurable parameters to find objective evidence for improvement of execution time. Third, an evaluation harness enables the quantitative assessment of generated patches and tests by running them inside Docker images and performing statistical testing on execution times.

Using JETO-Mine, we build JETO-Bench, a benchmark of 660 identified ETIPs and 91 manually verified executable ETIPs collected from 174 open-source Java repositories spanning 11 years of development history. To demonstrate the usefulness of JETO-Bench, we run OpenHands [2], a leading open-source coding agent, on the 91 verified executable ETIPs. OpenHands correctly fixes 14.3% (13/91) of execution time issues, aligning with the results reported by similar studies on other programming languages [13], [14]. We also find that the evaluation harness automatically detects most incorrect patches, confirming the value of execution-based feedback for patch assessment. Furthermore, our analysis reveals that open-source Java projects largely lack tests that quantitatively demonstrate execution time improvements, highlighting a significant gap in current testing practices and a promising opportunity for future research.

In summary, this paper makes the following contributions:

• JETO-Mine, the first configurable and reusable tool for automatically creating reproducible benchmarks of execution time improvement patches in Java. JETO-Mine consists of a three-phase pipeline with user-defined configurations and statistical testing.

• JETO-Bench, a benchmark of 660 identified and 91 manually verified executable ETIPs collected from 174 real-world Java repositories, accompanied by Docker images for full reproducibility.

• An empirical evaluation of OpenHands on JETO-Bench. This empirical investigation provides insight into the current capabilities and limitations of advanced coding agents on Java execution time improvement tasks, and reveals the scarcity of execution time improvement tests in open-source Java projects.

2 Execution Time Improvement Patches↩︎

With recent developments in automated bug fixing, researchers are creating new techniques to fix non-functional bugs, such as memory bugs [16], [17], vulnerabilities [18], code quality deficiencies [19], and execution time issues [20]. To enable further research in this area, we propose JETO-Mine, the first configurable and reusable tool for creating benchmarks of real-world execution time improvement patches in Java. Execution time improvement patches differ from functional bug fixes in an important way: a correct patch preserves the functionality of the original version while improving its performance w.r.t. the input under which it is tested. These nuances require precise definitions, which we provide below.

Note that in our definitions of both ETIP and ETIP detector test, we only consider significant improvements of execution time. We emphasize significance in order to dismiss flaky changes due to randomness or execution environment variability. JETO-Mine lets users determine which level of difference is significant by choosing the number of times a test is run, the minimum execution time improvement, and the p-value that entails statistical significance. We explain this user-driven configuration further in 3.3.

Benchmarking execution time in Java is particularly challenging due to the characteristics of the Java Virtual Machine (JVM). The JVM’s just-in-time (JIT) compiler progressively optimizes frequently executed code paths at runtime, meaning that the first several executions of a method can be significantly slower than subsequent ones [21]. Garbage collection (GC) introduces non-deterministic pauses that can distort timing measurements, especially for short running tests [22]. Class loading and bytecode verification also add overhead to early executions. These effects make measurements unreliable for detecting genuine execution time improvements. In contract with benchmarks targeting Python or C++, where execution time is more directly determined by the source code, a Java ETIP benchmark must explicitly account for JVM warm-up, GC interference, and JIT compilation variability. These JVM-specific concerns motivate several design decisions in JETO-Mine, which we describe in 3.3.

JETO-Mine identifies ETIPs in open-source Java projects, creates Docker images that enable reproducible execution of these ETIPs, collects test execution time to help quantitative comparison, and provides tools for evaluating generated patches and tests. This framework facilitates future research in the area of execution time improvement in Java programs.

3 System Design↩︎

3.1 Overview↩︎

1 illustrates an overview of how JETO-Mine works. The workflow consists of three main phases: 1) Static analysis: JETO-Mine goes over GitHub commits and PRs and identifies ETIPs based on the filters provided by the user, 2) Dynamic analysis: JETO-Mine wraps the identified ETIPs in docker images and creates a fully reproducible environment for their execution, and 3) Evaluation harness: using the reproducible environments generated in the second phase, JETO-Mine provides the tools for the quantitative evaluation of generated patches and tests. In the remainder of this section, we explain each phase in detail.

3.2 Static Analysis↩︎

2 shows how the static analysis phase of JETO-Mine works. The inputs are the user-defined filters that determine which patches should be collected and a GitHub token that enables access to public repositories. After going through patches in the form of both commits and PRs, JETO-Mine identifies and outputs ETIPs and their associated issues. This output provides the necessary data for a performance improvement patch generation task: the original version of the project, the issue that describes the performance problem, and the ground-truth patched version that resolves it.

Static analysis starts by crawling repositories (step 1) that meet four criteria: 1) it is a Java project that uses Java 8 or newer and builds with mvnw, 2) the repository has been updated as recently as the user specified date, 3) it has at least the number of stars determined by the user, and 4) the project is not archived. These criteria ensure that the project is active and popular according to user definition, and uses a build tool that enables reproducibility.

After the desired repositories are selected, JETO-Mine checks the commit diffs on the default branch over the timeline specified by the user (step 2). A commit diff is considered valid if it 1) only changes .java files, 2) does not change test files, and 3) modifies fewer files than the user specified limit. The commits that meet these requirements are then checked to find their associated issues (step 3). JETO-Mine then prompts an LLM with the issue title, description, and commit message to determine if the issue is primarily focused on improving execution time (step 4). Only valid commits linked to execution time improvement issues are kept.

In the fifth and final step, JETO-Mine cleans up the filtered commit-issue pairs. First, for commits that are part of a PR, we replace the commit with the corresponding PR to capture the complete change intended by the developer. This ensures that the considered code change fully addresses the execution time issue. We also verify that the PR’s changed files pass all the considered criteria. Second, we deduplicate the set of all commits and PRs based on their code diff, as there can be multiple commits on the default branch with the same diff depending on the PR merge strategy used in the project.

The result of static analysis is the set of identified ETIPs, each accompanied by an issue describing the execution time problem resolved by the patch. Each ETIP provides three essential pieces of information: an original version of the program with an execution time defect, a GitHub issue describing that defect, and a ground-truth patch showing how the issue can be resolved. Traditionally, such data has been successfully used for the evaluation of patch generation tools [23], which confirms the usability of the identified ETIPs.

Identified ETIPs can be used for patch generation evaluation by checking if a generated patch exactly matches the ground truth [23], [24]. However, a generated patch can still correctly improve execution time without exactly matching the ground truth, e.g. by applying the same algorithmic optimization with minor differences in boundary checks or operation ordering. To provide more fine-grained assessment beyond syntactic similarity, JETO-Mine makes ETIPs executable in a fully reproducible manner. By executing a generated patch, we can determine if it compiles, passes tests, or improves test execution times. Such feedback enables more precise evaluation and can also serve as a signal to iteratively improve the generated patch [25]. To guarantee executability, the identified ETIPs are passed to the dynamic analysis phase of JETO-Mine.

3.3 Dynamic Analysis↩︎

The dynamic analysis phase of JETO-Mine, as shown in 3, takes the identified ETIPs and user configurations as input and outputs executable ETIPs in the form of Docker images. The resulting images contain the environment needed for building the project and running relevant tests, including all required dependencies, enabling full reproducibility for running generated patches or tests.

The user can configure three parameters that determine whether a patch shows significant execution time improvement: the number of executions \(num\_exec\), the minimum improvement percentage \(min\_impr\), and the target p-value \(p\_val\). These parameters let the user control the statistical rigor of the results.

The dynamic analysis phase of JETO-Mine is designed to address the JVM-specific challenges described in 2. To mitigate the effect of JIT compilation warm-up, JETO-Mine executes the test suite once as a dedicated warm-up round before beginning timed executions. The results of the warm-up round are discarded. To reduce the impact of garbage collection pauses and other sources of timing variation, JETO-Mine runs each test suite \(num\_exec\) times (30 by default) and applies statistical testing. Moreover, JETO-Mine runs original and patched versions inside the same docker image with identical JVM versions and configurations, ensuring that JIT compilation behavior, GC settings, and class loading conditions are the same for both versions. These measures are essential for producing reliable execution time comparisons on the JVM.

Given the inputs, the dynamic analysis starts by analyzing the code diff between the original and patched versions to identify the modified files and modules (step 1). Next, JETO-Mine creates a Docker image with the correct Java version and builds and runs the JUnit tests of the modified modules on both versions (step 2). The tests are executed using Maven Wrapper multiple times according to the user given configuration. The build and test logs are saved in the image for later investigation and analyzed to ensure that across all executions of both versions: 1) the project successfully builds, 2) the modified modules contain tests, and 3) all tests pass. If any of these requirements is not met, the ETIP is excluded from the final set of executable ETIPs. This ensures that the tests can later be used to assess the functional correctness of generated patches.

After confirming that both versions pass all tests, JETO-Mine extracts test execution times from the logs (step 3). The extracted data consists of times \(T_{ijv}\), where \(1 \leq i \leq n\) refers to the \(i\)th test class in the modified modules, \(1 \leq j \leq num\_exec\) indicates the execution round, and \(v \in \{original, patched\}\) indicates the version.

At step 4, JETO-Mine performs statistical testing on the collected times. For each test class \(1 \leq i_{0} \leq n\), JETO-Mine conducts a paired one-sided binomial test between the lists of \(T_{i_{0},j,original}\) and \(T_{i_{0},j,patched}\) for \(1 \leq j \leq num\_exec\). If the test shows that one version is \(min\_impr\) percent faster than the other with a p-value below \(p\_val\), JETO-Mine considers the \(i_{0}\)th test class an ETIP detector test. Note that for a given ETIP, some tests may favor the original while others favor the patched version.

Finally, JETO-Mine summarizes the execution results in a report (step 5). For each ETIP, the report indicates whether it successfully builds and passes tests, which test classes detect a significant execution time difference, and whether there is a significant difference in overall execution time. JETO-Mine considers an ETIP executable and creates a Docker image for it if both versions build and pass tests in all \(num\_exec\) rounds. Even if no existing test detects an execution time improvement, the Docker-based executability still provides valuable feedback on build and test success that patch generation tools can leverage.

3.4 Evaluation Harness↩︎

The third and final phase of JETO-Mine provides an evaluation harness. This harness takes the ID of an executable ETIP, a generated patch, and an execution configuration as input, and evaluates whether any test class in the modified modules detects a significant execution time improvement by the generated patch.

The evaluation harness works in five steps. First, the Docker image corresponding to the selected ETIP is pulled if not already present and a container is created. Second, the generated patch is applied to the original version and saved in a separate directory inside the container. Third, the tests are executed on both original and patched versions as many times as specified in the configuration. Fourth, the logs are analyzed to verify that both versions pass all tests in every execution round and to extract test execution times. Finally, a statistical test is performed to determine if any test class detects a statistically significant difference between the execution times of the two versions. The logs and statistical test results are presented to the user as the final report.

In addition to patch evaluation, JETO-Mine also supports the evaluation of test generation tools. For this purpose, JETO-Mine takes a generated test class and runs it on the original and ground-truth patched versions, then conducts statistical testing to determine if the generated test is an ETIP detector test. This enables research in a rarely studied area: the generation of tests that detect execution time improvement.

3.5 Implementation↩︎

We implement JETO-Mine in Python. In its default configuration, the static analysis phase targets projects with at least 20 stars and commits that change at most 20 files since 2015, and uses gpt-5.1-codex-mini for the LLM-based issue filter. In the dynamic analysis phase, the tests are executed once for warm-up and 30 times for collecting execution times. Following previous work [12], JETO-Mine considers an improvement of 5% with a p-value of 10% significant. JETO-Mine, JETO-Bench, and all data relevant to this paper are publicly available [26].

4 Experimental Methodology↩︎

4.1 Building JETO-Bench↩︎

We start our experiments by running JETO-Mine to build JETO-Bench, the first benchmark of execution time improvement patches in diverse real-world Java projects.

We build JETO-Bench by running JETO-Mine with its default configuration (see 3.5), collecting commits and PRs until 2025-11-28. JETO-Mine checks 3,686 repositories and 1,769,958 commits spanning 11 years of open-source development history. This shows the large scale of JETO-Bench and our study, which provides unprecedented insight into execution time improvement in open-source Java projects.

1 reports the distribution of commits and stars across the considered repositories. The number of commits per repository ranges from 1 to 55,824 with a median of 58, while stars range from 20 to 93,448 with a median of 53, showing notable diversity in both activity and popularity. The result of running JETO-Mine on this large and diverse set of repositories and commits is JETO-Bench.

JETO-Bench includes a set of ETIPs identified by the static analysis phase and a set of executable ETIPs in the form of Docker images built by the dynamic analysis phase. We conduct experiments to study the characteristics of JETO-Bench, the effectiveness of advanced coding agents on JETO-Bench, and the performance of JETO-Mine in building Java ETIP benchmarks.

4.2 Research Questions↩︎

Our experiments study the following research questions:

• RQ1 (JETO-Bench characteristics): What are the characteristics of the ETIPs collected by JETO-Mine in terms of diversity, scope, and test coverage?We assess the characteristics of the identified and executable ETIPs in JETO-Bench to show the diversity of real-world execution time improvements collected by our approach.

• RQ2 (using JETO-Bench in practice): How effectively can advanced coding agents fix execution time issues in JETO-Bench?We run OpenHands, a leading open-source coding agent, on the executable ETIPs in JETO-Bench to evaluate the current state of automated ETIP generation.

• RQ3 (JETO-Mine performance): How precise and practical is JETO-Mine for building Java ETIP benchmarks?We investigate the precision and accuracy of JETO-Mine and measure the cost of benchmark building in terms of time, space, and financial expenses.

4.3 Protocol for RQ1 (JETO-Bench characteristics)↩︎

To answer RQ1, we count the number of identified ETIPs, executable ETIPs, and the repositories from which they are collected. These numbers indicate the overall applicability of JETO-Mine. We then characterize the ETIPs along four dimensions: the number of files changed, the number of modules modified, the year of creation, and the popularity of the corresponding repository. A diverse set of ETIPs enables the evaluation of various aspects of patch generation tools.

We also examine ETIP detector tests using two statistical testing configurations: 1) the default configuration, following previous work [12], with \(p\_val=0.1\) and \(min\_impr=0.05\), and 2) a conservative configuration with \(p\_val=0.05\) and \(min\_impr=0.1\). For each configuration, we check how many executable ETIPs have a test that detects a significant execution time improvement. Cases where no such test exists indicate that the open-source project lacks tests demonstrating execution time improvements, highlighting an opportunity for future research in test generation.

4.4 Protocol for RQ2 (using JETO-Bench in practice)↩︎

To answer RQ2, we use OpenHands to generate patches for the execution time issues related to in executable ETIPs. OpenHands provides isolation via Docker sandboxing; we leverage this by letting OpenHands use the Docker images created by JETO-Mine as its sandbox environment with all the dependencies and settings needed to run the project. Based on preliminary experiments, we opt for gpt-5-mini, which provides strong effectiveness at reasonable cost compared to both older and larger models. To ensure a high-quality evaluation set, we manually filter the executable ETIPs according to two criteria: 1) the linked issue is descriptive enough for a developer to implement the fix, and 2) the issue is genuinely focused on execution time improvement. This yields a manually verified set of executable ETIPs that we recommend for future research.

After running OpenHands, we manually classify each generated patch into four categories: 1) exact match: the AST is identical to the ground truth, 2) semantically equivalent: functionally equivalent to the ground truth and applying similar optimization but syntactically different, 3) correct location: modifies the right code location but differs semantically from the ground truth, and 4) wrong location: modifies an incorrect location. The first two categories constitute correct fixes. Although correct location patches fall short of correctness, they demonstrate accurate fault localization, a key step in patch generation [27]. We also track multi-file and multi-module patches in each category to assess performance on issues that span beyond a single file.

We use the evaluation harness to build and run tests on all generated patches. The harness automatically identifies patches that have incorrect format, do not compile, or fail tests, allowing them to be dismissed without manual assessment. We verify that all correct and semantically equivalent patches pass the tests, which reaffirms the soundness of the harness. With this experiment, we check that the evaluation harness of JETO-Mine successfully employs the executable ETIPs in JETO-Bench to provide execution-based feedback on generated patches.

Note that the goal of this experiment is not to build a state-of-the-art ETIP generation tool. The goal is to investigate whether JETO-Bench and the evaluation harness can be successfully used to evaluate patch generation tools, and to gain insight into the capabilities of an advanced off-the-shelf agent on Java execution time issues.

4.5 Protocol for RQ3 (JETO-Mine performance)↩︎

To answer RQ3, we randomly select and manually analyze 50 identified ETIPs to evaluate the precision of ETIP identification, checking whether each is linked to an issue that focuses on execution time improvement. To assess overall accuracy, we separately sample 50 issues passed to the LLM-based Issue Filter (see 2) and manually check whether this filter correctly labels them, regardless of whether they are execution time related or not.

We also measure the time needed for identifying ETIPs, the cost of LLM-based issue filtering, the time the dynamic analysis phase takes to build Docker images, and the size of the resulting images. These numbers indicate the feasibility of using JETO-Mine on a large corpus of open-source commits.

4.6 Experiment Resources↩︎

We run all experiments on a machine with one AMD EPYC 7742 processor (64 cores, up to 2.25 GHz) and 256 GB DDR4 RAM (3200 MT/s). In the dynamic analysis phase, we dedicate 32 CPU cores and 80 GB RAM for building each Docker image.

5 Experimental Results↩︎

5.1 Results for RQ1 (JETO-Bench characteristics)↩︎

JETO-Bench contains 660 identified ETIPs and 102 executable ETIPs¹. Compared to similar benchmarks for other programming languages [12], [14], 102 executable patches provide sufficient data for evaluating advanced patch generation techniques. Additionally, 21 ETIPs build successfully but are excluded from the executable set because their modified modules contain no tests, showing that the dynamic analysis effectively filters out patches lacking functional correctness validation.

The identified ETIPs are collected from 174 distinct repositories, while the executable ETIPs come from 40. 4 shows the distribution across the top 15 repositories, which cover topics such as distributed systems, data management, cloud-native frameworks, and software infrastructure, demonstrating the topical diversity of JETO-Bench.

2 presents the modification scope of ETIPs in JETO-Bench. While most patches modify a single module (average close to 1), the benchmark includes patches spanning up to 13 modules and 20 files. Notably, 270 identified ETIPs and 40 executable ETIPs modify multiple files, making JETO-Bench suitable for evaluating repo-level patch generation tools [6].

5 shows that most ETIPs come from repositories with at least 10,000 stars, indicating maturity and reliability. 6 shows a steady distribution of ETIPs from 2019 onward, with fewer from earlier years due to the Java 8 minimum requirement. This temporal diversity ensures the benchmark is not biased toward a particular development period.

Finally, we check if existing JUnit tests serve as ETIP detector tests. With the default configuration (\(p\_val=0.1\), \(min\_impr=0.05\)), only 19 ETIPs have a test class showing statistically significant improvement. With the conservative configuration (\(p\_val=0.05\), \(min\_impr=0.1\)), only 8 do. Our manual assessment reveals that many of these tests show significant improvement due to noise in time collection rather than actual performance testing. Our takeaway is that open-source Java projects largely lack tests that verify execution time improvements, presenting an opportunity for research on automated ETIP detector test generation.

5.2 Results for RQ2 (using JETO-Bench in practice)↩︎

As explained in 4.4, before running OpenHands on JETO-Bench, we first manually check the executable ETIPs in JETO-Bench and their corresponding issues. We only keep the issues that are focused on execution time improvement and have informative description. The curated set contains 91 issues and their executable ETIPs. We run OpenHands on these 91 cases and prompt it to generate patches to fix the execution time issue in the original version. We then manually assess the generated patches to gain a deep understanding of OpenHands performance on JETO-Bench.

3 shows the results of fixing the 91 manually verified execution time issues, broken down by issue scope (single-file vs.multi-file and single-module vs.multi-module). Each generated patch is classified as a correct fix (exact match or semantically equivalent), a modification at the correct location but with semantic differences, a modification at the wrong location, or not generated. The column “Test Success” reports how many generated patches successfully build and pass the test suite according to the evaluation harness of JETO-Mine.

private void readClassFileNames() throws IOException {
  URL jarUrl = jar.toUri().toURL();
  URLClassLoader classLoaderFromJar = new URLClassLoader(new URL[] {jarUrl}, null);
  ClassPath classPath = ClassPath.from(classLoaderFromJar);
  ImmutableSet<ClassInfo> allClasses = classPath.getAllClasses();
  classFileNames = allClasses.stream().map(ClassInfo::getName).collect(toImmutableSet());
}
private void readClassFileNames() throws IOException {
  try (JarFile jarFile = new JarFile(jar.toFile())) {
    ImmutableSet.Builder<String> classNames = ImmutableSet.builder();
    Enumeration<JarEntry> entries = jarFile.entries();
    while (entries.hasMoreElements()) {
      JarEntry entry = entries.nextElement();
      String name = entry.getName();
      if (name.endsWith(".class")) {
        classNames.add(className);
      }
    }
  }
  this.classFileNames = classNames.build();
}

As presented in 3, OpenHands generates a correct fix for 14.3% (13/91) of the issues. This aligns with state-of-the-art studies on fixing performance issues in other programming languages [14]. This suggests that JETO-Bench is a reliable benchmark for evaluating advanced patch generation tools on Java execution time issues. We also see that 10 of the correct fixes exactly match the ground-truth, while 3 of them are semantically equivalent to the ground-truth but do not match it. [lst:semantic-equivalence] is an example of a semantically equivalent patch generated by OpenHands for issue #1320 of “cloud-opensource”. This listing shows the original code, the ground-truth patched version and the patched version generated by OpenHands. To save space, we show the ground-truth and OpenHands patches in a single snippet as a diff. The issue states that in the original version all the classes are loaded from the Jar file and then the class names are taken from the loaded classes. In the patched version, the class names are directly read from the Jar file. The difference between the ground-truth and generated patches appears at lines 19-21, where they build the class name. Both versions read the filename, remove .class from the end of it, and replace / with ., but do this in a different order. Also, the ground-truth uses 6 as a literal, while OpenHands computes this literal with ".class".length(). This example shows how a generated patch can apply the same improvement as the ground-truth with a different syntactical modification. Consequently, we cannot solely rely on statically checking the changes to the AST. This demonstrates the importance of having more fine-grained and execution-based feedback on generated patches. The JETO-Mine evaluation harness gives such feedback by running generated patch in the docker-based reproducible environment of executable ETIPs.

The column “Test Success” of 3 shows that only for 22% (20/91) of issues the generated patch successfully applies, compiles, and passes the tests. This means that most incorrect generated patches can be automatically labeled as incorrect using the evaluation harness of JETO-Mine, which reaffirms the significance of the execution-based feedback provided by the harness. We also find that for 36.3% (33/91) of issues, OpenHands does not generate a patch. Looking at OpenHands logs, we find that in most of these cases OpenHands fails to generate a reasonable patch because it cannot understand the context of the issue and find the exact location of the fault. This indicates the importance of fault localization for fixing execution time issues. In 3, we see that in 27.5% (25/91) of cases, OpenHands fails to generate a correct patch, however it localizes the fault and modifies the correct location. This is a promising result that shows the feasibility of improving the effectiveness of OpenHands on JETO-Bench.

The comparison between issues with different scopes in 3 shows that OpenHands can fix issues at all scopes: 18.5% of single-file issues, 8.1% of multi-file issues, 14.3% of single-module issues, and 14.3% of multi-module patches. This indicates the ability of OpenHands to fix repository level bugs and the usefulness of JETO-Bench in evaluating repository level bug fixing, which is becoming the focus of patch generation research [6]. Examining the two multi-module correct fixes, we find that both involve small, repetitive changes across modules (e.g. changing a constant from RefreshPolicy.WAIT_UNTIL to RefreshPolicy.IMMEDIATE), suggesting that complex multi-module performance issues remain an open challenge.

5.3 Results for RQ3 (JETO-Mine performance)↩︎

As explained in 4.5, to assess the precision of JETO-Mine, we manually analyze 50 randomly selected identified ETIPs. Note that the LLM-based filter of JETO-Mine considers all issues corresponding to these ETIPs to be focused on execution time. Our manual analysis confirms that 48 of these issues are focused on execution time improvement. This means that according to this investigation, the precision of JETO-Mine is 96% (48/50). Compared to previous study on issue classification [28], this is a high precision that makes JETO-Mine a reliable tool for collecting ETIPs.

A more detailed analysis reveals that beyond the 2 non-relevant issues, 5 additional issues focus on execution time but also require new functionality, which falls outside our ETIP definition (2). We exclude these from our manually verified set of 91 executable ETIPs to ensure all patches preserve functional equivalence.

To assess overall accuracy, we also sample 50 commits passed to the “LLM-based Issue Filter” step, of which 3 are manually confirmed as execution-time-related and 47 are not. JETO-Mine correctly labels 48 of 50 issues (96% accuracy), confirming that the filter is reliable across both positive and negative cases.

Measuring the time dedicated to running the static analysis phase of JETO-Mine on the 1,769,958 target commits, we find that JETO-Mine takes 478 hours to identify the 660 ETIPs. This shows the value of JETO-Bench for future research on fixing execution time issues, as it provides filtered data that requires almost 20 days of commit inspection. The static analysis of the commits also spends $5.33 to filter the issues with the help of LLMs, which shows the affordability of our approach.

The dynamic analysis phase of JETO-Mine takes 152 hours to process the 660 identified ETIPs and create 102 executable ETIPs in form of Docker images. This means that processing each identified ETIP takes only 13.8 minutes, which indicates the feasibility of building large benchmarks with JETO-Mine. 7 shows the characteristics of the Docker images built by JETO-Mine. The median time it takes to process an identified ETIP and build the Docker image is 34 minutes. Also, the median size of built docker images is 790 MB. Again, this shows that JETO-Mine can be used in practice to create Docker images for reproducible execution of ETIPs.

6 Discussion↩︎

6.1 JMH Tests↩︎

As explained in 3.3, JETO-Mine runs the JUnit tests multiple times and performs statistical testing to see if any test detects a significant execution time improvement. Our experiments show that such JUnit tests are rare (see 5.1). This leads us to another question:

Is there a framework that open-source Java projects commonly use to demonstrate execution time improvements?

The standard and most widely used framework for performance testing in Java is the Java Microbenchmark Harness (JMH)² [15]. We analyze the 91 manually verified executable ETIPs in JETO-Bench to assess the use of JMH in open-source Java projects. We find that only 15% (14/91) of the executable ETIPs have JMH benchmarks in their code. This shows that even considering JMH benchmarks does not solve the lack of tests that demonstrate execution time improvements. Therefore, we emphasize the importance of adding more ETIP detector tests in open-source Java projects.

6.2 Test Generation↩︎

Given the lack of tests that show execution time differences in open-source projects, automated ETIP detector test generation is a valuable task for future research. As explained in 3.4, the evaluation harness of JETO-Mine enables the evaluation of tools that generate ETIP detector tests. In a preliminary study, we employ OpenHands to generate tests that show a statistically significant difference between the execution times of original and patched versions for our 91 manually verified executable ETIPs. We find that OpenHands is able to generate an ETIP detector test for only one of the 91 ETIPs (the generated test and the corresponding ETIP are available in our replication package [26]). This indicates the difficulty of generating ETIP detector tests and presents a promising opportunity for future studies.

The challenge of generating ETIP detector tests lies in the fact that such tests must not only exercise the modified code paths, but must do so with inputs large or representative enough to reveal measurable performance differences. A test that calls the improved code with a small input may pass on both versions with little timing difference, even if the patch provides orders of magnitude improvement on realistic workloads. This contrasts with functional test generation, where a single well-chosen input can definitively distinguish correct from incorrect behavior. Future work on ETIP detector test generation will likely need to combine code coverage techniques with workload synthesis to produce inputs that stress the performance-critical paths identified by the patch.

6.3 Threats to Validity↩︎

6.3.1 Threat to External Validity↩︎

JETO-Mine collects ETIPs from projects that use Maven Wrapper to build. This means Java projects that use other build tools, such as Gradle, are not considered. Hence, we cannot be certain that our approach generalizes to Java projects that do not use Maven Wrapper. However, as reported in 5.1, the number of ETIPs identified by JETO-Mine aligns with similar benchmarks for other programming languages [12], [14], which indicates the usefulness of our tool for evaluating advanced patch generation tools.

6.3.2 Threat to Internal Validity↩︎

The dynamic analysis and evaluation harness of JETO-Mine run project tests to determine if they detect an execution time improvement. Since the execution environment can be volatile, the results of this step may not be fully robust and reproducible. To address this concern, we run the tests multiple times and use statistical testing to report how confidently we can claim a test detects an improvement. However, in our experiments, we noticed that the order of running tests can affect execution time. This is an issue because by default, Maven does not guarantee a deterministic order for running tests. We resolve this with two techniques. First, we always configure Maven to run the tests in alphabetical order on both versions. Second, over the \(num\_exec\) rounds of executing tests on original and patched versions, at odd rounds we first run the patched version and at even rounds we first run the original version. These two techniques minimize the effects of test execution order and environment volatility to maximize confidence in our statistical test results.

7 Related Work↩︎

Recently, many researchers are working on automated repair of various non-functional bugs, including performance issues [7]–[9], [29]–[31]. This line of research heavily depends on having benchmarks for the evaluation of patches generated by repair tools. Consequently, researchers are also working on new performance improvement benchmarks and tools to collect such benchmarks [12]–[14]. In this section, we review the literature on performance issue benchmarks, automated fixing of performance issues, and studies on the performance optimization capabilities of AI agents, which are the closest research areas to our work.

7.1 Performance Issue Benchmarks↩︎

There are established bug benchmarks, such as Defects4J [4], Bugs.jar [32], IntroClassJava [33], QuixBugs [5], and Bears [34], which have been used for evaluating automated program repair (APR) tools for more than a decade. However, these benchmarks are particularly focused on functional bugs and rarely updated. In a more recent effort, Jimenez et al. [6] have proposed SWE-Bench, which has become one of the most prominent benchmarks for APR evaluation. SWE-Bench provides pairs of GitHub issues and their resolving PRs as well as Docker containers for reproducing the issues and executing generated patches. In contrast to JETO-Bench, SWE-Bench is focused on Python and functional bugs.

After years of progress in automatically fixing functional bugs, researchers have recently paid more attention to building APR tools for performance bugs [35]. Consequently, new benchmarks of performance bugs are also created to support this effort [36]. Several benchmarks are collected from code competition submissions, including PIE [11] (77K C++ modifications), EffiBench [37], EvalPerf [38], and Mercury [39]. While competition submissions are widely used for evaluating the performance of generated code [40], these programs are usually small and different from real-world projects. To address this limitation, researchers have created more realistic benchmarks from real-world repositories: SWE-Perf [12] (140 tasks from 12 Python repositories), SWE-fficiency [35] (performance PRs from nine data-science and HPC repositories), GSO [13] (102 tasks from 10 codebases in Python, C, and SIMD), PeacExec [8] (146 tasks from 47 Python repositories), and PerfBench [14] (81 tasks from .NET repositories with an evaluation harness for test generation). Evaluations on these benchmarks consistently show the difficulty of automatically fixing performance issues, with OpenHands achieving success rates of 4.9% on GSO and 3% on PerfBench. Automated performance improvement tools have also been tested on more specialized programs, such as GPU kernels [41] and compilers [42].

Compared to these benchmarks, JETO-Bench is the first benchmark of execution time improvement patches specifically targeting Java programs. While the existing benchmarks primarily focus on Python [8], [12], [35], C++ [11], .NET [14], or multiple languages [13], Java remains underrepresented despite being one of the most widely used programming languages. The most closely related benchmark that focuses on Java is the dataset used by Yi and Leitner [15], which contains 65 tasks from four open-source Java projects. However, their dataset does not provide an automated pipeline for collecting new tasks or Docker-based environments for reproducible evaluation. JETO-Mine provides a fully automated pipeline for creating new benchmarks with user-defined filters, Docker-based reproducible environments, and an evaluation harness that supports both patch and test generation evaluation.

7.2 Fixing Performance Issues↩︎

The existing benchmarks have enabled many studies on automated fixing of performance issues. Several tools target function-level optimizations: RAPGen [7] uses retrieval-augmented prompt generation, DeepDev-PERF [29] fine-tunes a transformer model for repairing performance bugs, PerfCoder [9] fine-tunes an LLM on curated real-world optimizations, SBLLM [43] combines LLMs with evolutionary search, and EFFI-LEARNER [44] uses execution overhead profiles to improve efficiency of LLM-generated code. At the repository level, Ren et al. propose PEACE [8], a hybrid framework for code efficiency optimization through automatic code editing, achieving a 69.2% correctness rate on PeacExec.

Researchers have also evaluated the performance of patches generated by agentic frameworks, such as OpenHands [2], Agentless [3], SWE-Agent [1], and AutoCodeRover [45]. Fan et al. [10] introduce new metrics to compare the performance of programs generated by these frameworks, while Peng et al. [46] present an empirical study comparing agent-authored and human-authored optimizations, finding that the optimization patterns largely overlap.

The work of Yi and Leitner [15] presents an investigation on LLM-based performance improvements using their dataset of 65 real-world tasks mined from four open-source Java programs. They generate patches using two leading LLMs under four prompt variations and use JMH benchmarks to compare the results with human-authored solutions. Their study demonstrates that human developers outperform LLM fixes by a statistically significant margin.

Our evaluation of OpenHands on the 91 manually verified executable ETIPs in JETO-Bench aligns with the findings reported by these studies. Similarly to previous work, we observe that state-of-the-art agents achieve limited success rates on performance optimization tasks, motivating the need for dedicated benchmarks like JETO-Bench.

8 Conclusion↩︎

In this paper, we present JETO-Mine, the first configurable and reusable tool for creating reproducible benchmarks of execution time improvement patches (ETIPs) in real-world Java projects. JETO-Mine combines three phases: 1) static analysis for identifying ETIPs from GitHub repositories, 2) dynamic analysis for building Docker-based reproducible environments, and 3) evaluation harness for quantitative assessment of generated patches and tests.

Using JETO-Mine, we built JETO-Bench, a benchmark of 660 identified ETIPs and 91 manually verified executable ETIPs collected from 174 open-source Java repositories. Our manual analysis shows that JETO-Mine identifies ETIPs with a precision of 96%. Running OpenHands on the 91 verified executable ETIPs, we find that it correctly fixes 14.3% (13/91) of the issues, aligning with results reported by similar studies on other programming languages. The evaluation harness automatically detected most incorrect patches, confirming the value of its execution-based feedback.

Our analysis also reveals that open-source Java projects largely lack tests that demonstrate execution time improvements: only 15% of verified executable ETIPs contain JMH benchmarks, and existing JUnit tests rarely detect statistically significant execution time differences. Therefore, the evaluation harness of JETO-Mine presents an opportunity for future research on automated fixing of execution time issues and automated generation of ETIP detector tests.

Acknowledgements↩︎

We used Cursor as a coding assistant during the development of JETO-Mine. We also used Claude to help polish the text and figures of this paper.

Data Availability Statement↩︎

JETO-Mine, JETO-Bench, and all data and scripts needed to reproduce the results reported in this paper are publicly available at: https://github.com/khesoem/JETO-Bench

The replication package includes the source code of JETO-Mine, the full list of 660 identified ETIPs and their associated issues, the Docker images for the executable ETIPs, and manually verified set of 91 executable ETIPs. The data and scripts needed for reproducing the tables and charts are also presented.

References↩︎