1 Introduction↩︎

Automated unit test generation has relied on random testing, search-based testing, symbolic execution, and, more recently, large language models (LLMs). Despite this progress, generating high-quality tests for complex Java methods remains challenging. The difficulty often lies not in the size of the focal method itself, but in satisfying branch-specific testing requirements that are only partially visible from the method’s source code. Reaching a target behavior may require establishing a prerequisite program state, exercising a particular switch branch, triggering an exception-translation path, invoking the method through a valid entry route, or constructing assertions strong enough to detect semantic regressions. Some of these requirements can be inferred statically, while others only become visible after generated tests are compiled, executed, and evaluated.

Recent LLM-based test generation approaches have moved beyond focal-method-only prompting by incorporating project context, static analysis, coverage feedback, and mutation guidance [1]–[9]. For example, KTester injects project structure and usage knowledge [2], PANTA uses coverage feedback to target under-tested paths [3], and MUTGEN leverages mutation feedback to strengthen generated tests [9]. However, these signals are typically delivered to the LLM as prompts, hints, path descriptions, or post-execution feedback. Existing approaches do not maintain an explicit representation of individual testing requirements and their status across iterations. Consequently, later rounds may revisit already-satisfied behaviors while unresolved branches, or weak oracles, remain unaddressed. Furthermore, coverage-oriented and mutation-oriented feedback are usually handled separately, limiting their ability to jointly improve reachability and fault-detection strength.

We present TATG, a framework that unifies static program analysis and dynamic evaluation feedback through a unified objective representation. TATG represents each testing requirement as a tracked objective containing fields such as supporting evidence, required setup, and target action. Static analysis generates structural objectives for reachability, branch coverage, exception handling, object construction, boundary behavior, and entry routes. During generation, compilation/execution, coverage, oracle-quality, and mutation feedback are converted into feedback objectives and integrated into the same tracking mechanism. This representation enables the framework to reason explicitly about which requirements have been satisfied, which remain unresolved, and which require further refinement.

Building on this objective representation, TATG employs a two-stage generation strategy. The structural stage focuses on producing executable tests, improving reachability, and increasing line/branch coverage using static objectives, along with compilation/execution, and coverage feedback. Once structural progress stabilizes, the hardening stage uses mutation feedback to improve fault-detection capability.

We evaluate TATG on an extended benchmark of 141 focal methods, comprising complex methods with rich control-flow structures and project dependencies, collected from all projects used by recent LLM-based baselines, as well as additional modern multi-module projects. We compare TATG against six representative approaches across three categories: traditional, hybrid (traditional + LLM), and LLM-based. The evaluation is conducted using two recently released open-source LLMs, Qwen3.6-35B-A3B and DeepSeek-V4-Flash-284B, representing different model sizes. In addition, we compare TATG against a proprietary industrial test-generation tool on a randomly selected subset of 50 focal methods. The results show that TATG consistently outperforms all baselines in terms of structural coverage and mutation score under the same iteration budget. Furthermore, TATG achieves performance comparable to the industrial tool while achieving higher line coverage and a higher mutation score.

Contributions. This paper makes the following contributions:

• Tracking-aware objective representation. We introduce a unified objective representation for LLM-based unit test generation that captures branch-level testing requirements together with supporting evidence, required setup, target actions, expected oracles, acceptance predicates, and execution status. The representation enables explicit tracking of which testing objectives have been satisfied and which remain unresolved across generation iterations.

• Two-stage objective-guided generation. We propose a two-stage objective-guided framework that coordinates structural coverage improvement and mutation-based oracle strengthening within a unified generation process. Static-analysis results, compilation/execution diagnostics, coverage feedback, and mutation feedback are all represented and managed through the same objective-tracking mechanism.

• Comprehensive evaluation on challenging Java methods. We evaluate TATG on 141 complex Java methods, including the 110 KTester subjects and 31 additional challenging methods. Results show that TATG consistently outperforms strong state-of-the-art baselines, while achieving a performance comparable to a proprietary industrial test generation tool. Ablation studies further confirm the effectiveness of the two-stage generation strategy.

2 Motivating Example↩︎

We use a single running example to illustrate the testing problem that TATG addresses. Listing [lst:running] presents a simplified, representative token-stream reader that abstracts recurring testing challenges from real-world code, rather than reproducing a concrete class from an existing project. The field currToken records the reader’s current state. The collaborator stream provides three operations: advance() moves to the next token and may throw StreamException; getText() returns the current string value; and getInt() returns the current integer value. The method under test, nextTextValue(), returns a textual representation of the current token when possible and throws IOException when stream processing fails.

class StreamTokenReader {
  private Token currToken;
  private TokenStream stream;

  String nextTextValue() throws IOException {
    if (currToken == Token.FIELD_NAME) {      // G
      currToken = stream.advance();
      if (currToken == Token.STRING)          // P1
        return stream.getText();
      if (currToken == Token.NULL)            // P2
        return null;
      if (currToken == Token.INT)             // P3
        return String.valueOf(stream.getInt());
    }
    if (currToken == Token.NULL)              // P4
      return null;
    try {
      stream.advance();
    } catch (StreamException e) {             // P5
      throw new IOException("Stream error", e);
    }
    throw new IOException("Not a text value"); // P6
  }
}

A useful test suite for Listing [lst:running] must satisfy requirements that are not explicit. First, most paths are state-dependent. Paths P1–P3 are reachable only when currToken is FIELD_NAME before the call and stream.advance() then returns a particular next token. A test that simply constructs a fresh reader and calls the target may reach only the fall-through exception. Second, the number of nested branches is finite: P1, P2, and P3 correspond to distinct token constants, and each needs a distinct input or mock behaviour. Third, P5 requires exceptional predecessor behaviour: the collaborator must throw StreamException at the right call site so that the method executes the catch-rethrow block. Finally, if the method is package-private or is normally accessed through a facade, the test must also select an executable entry route.

These requirements also explain why coverage should precede mutation refinement. If tests do not reach P2, P3, or P5, mutants inside those branches provide little actionable information: mutation analysis mainly reports that the mutated statements were not covered. In contrast, coverage feedback reveals unresolved coverage obligations and directs subsequent test generation toward the prerequisite state or collaborator behaviour needed to satisfy them. Once those branches are reached, mutation feedback becomes useful: covering the Token.STRING branch is insufficient if the test does not verify the returned text. A mutant that changes return stream.getText() to return "" may not be killed. At that point, the useful objective is no longer reachability, but oracle strengthening, e.g., asserting the exact text returned by the stream.

Existing generators struggle with this example because the information is not represented at the right granularity. Search-based tools optimize generated call sequences and coverage feedback, but the required setup for this method is highly specific: the test must place the reader in a particular token state or configure the stream to fail when advance() is executed. LLM-based tools can reason about the source and often generate readable tests, yet a generic branch- or exception-oriented prompt does not translate P1–P6 into distinct testing goals with required setup and measurable acceptance criteria. KTester, a recent state-of-the-art approach, combines project knowledge for test generation, however, the context remains coarse-grained (e.g., it does not capture predicate-specific information such as condition variables) and is not maintained as a per-path objective that records whether each branch or exception path has been attempted, satisfied, or left open. As a result, a passing test suite may still leave deep paths uncovered, while subsequent repair or regeneration lacks an explicit objective that identifies the unresolved branch or exception behavior requiring attention, such as a specific token-dispatch branch or an exception-translation path.

TATG addresses this example through two-stage objective-guided generation. Before generation, static analysis translates branch-relevant facts into explicit objectives, such as “exercise the branch where currToken == Token.STRING”, “exercise the branch where currToken == Token.NULL after entering the FIELD_NAME state”, and “configure stream.advance() to throw StreamException and assert the resulting IOException”. The structural stage first uses compilation and execution diagnostics, together with coverage feedback, to improve target reachability and code coverage. After each round, TATG updates the objective state to record which objectives were satisfied, which failed compilation/execution, and which remain open. Only after sufficient structural progress does mutation feedback become actionable: mutants in unreached code mainly indicate missing reachability, whereas live mutants in reached code may indicate either insufficient test inputs or weak assertions. Therefore, the hardening stage converts mutation testing feedback into semantic objectives that strengthen oracles and kill live mutants. Both stages share the same objective schema, but the generation focus shifts from improving reachability to strengthening oracles.

3 Approach↩︎

3.1 Overview↩︎

TATG is a two-stage, objective-guided framework for LLM-based unit test generation. Its central abstraction is an objective state: a set of testing goals that records what should be tested, what evidence supports each goal, how a test should reach it, how success will be checked, and whether previous attempts have made progress. An objective may ask the model to reach a branch, construct a required state, repair an invalid setup, strengthen a weak assertion, or kill a live mutant. Across iterations, TATG updates the status of each tracked objective, unifying compilation/execution diagnostics, coverage gaps, and mutation results into a single objective rather than handling them independently.

Figure 1 shows the workflow. Given a focal method and its corresponding source, TATG first constructs a compact target context that includes the method body, relevant condition values, construction information, and accessible entry routes. Static objective synthesizers then create the initial objective state from source and type facts, such as state-dependent setup, branch-relevant facts, exception paths, construction constraints, boundary inputs, and entry routes. The first stage performs structural generation, using compilation/execution diagnostics to produce executable tests and coverage feedback to improve branch reachability and coverage. The second stage performs hardening: once the relevant code is reached, mutation testing feedback creates semantic objectives that guide the LLM to strengthen assertions and kill live/uncovered mutants.

The two stages share the same objective schema and the same objective-guided prompt structure, but they differ in focus. The structural stage prioritizes executable tests and hard-to-reach paths. The hardening stage prioritizes strengthening oracles for code that is already reachable. This order is important for complex methods: if the test suite does not reach deep branches, mutation feedback often only reports that mutants are not covered, which provides little useful oracle guidance.

The remainder of this section follows the lifecycle of an objective in TATG. Section 3.2 defines the objective; Section 3.3 describes how initial objectives are derived from the source code and how they are updated from compilation/execution diagnostics, coverage, and mutation feedback; Section 3.4 explains the prompt structure and how the objective state is rendered into the prompt; and Section 3.5 describes the budget allocation and termination rules.

3.2 Objective definition↩︎

An objective state \(\mathcal{G}\) is a set of objectives: \[\mathcal{G}=\{G_1,G_2,\ldots,G_n\},\] where each objective \(G\) is represented as a tuple: \[G = \langle id, src, cat, E, S, A, Q, \alpha, status, key \rangle .\] The tuple’s fields have the following definitions.

Identifier. The field \(id\) is an identifier for the objective, which allows TATG to track the same testing goal across iterations. It is derived from the objective source and its evidence. For example, an objective created from a missed branch may use the source line and branch index, while a mutation objective may use the mutant identifier reported by mutation testing.

Source. The field \(src\) records where the objective comes from: \[src \in \{ static, coverage, validation, oracle, mutation \}.\] Static objectives are created before generation. Coverage objectives are created from missed lines or branches. Validation objectives are created from compilation or execution failures. Oracle objectives are created when generated tests execute code, but contain weak or missing assertions. Mutation objectives are created from mutation testing feedback.

Category. The field \(cat\) specifies the objective category, which determines how the objective is rendered in the prompt.

Evidence. The field \(E\) contains the evidence that justifies the objective. Evidence may include a source line, a branch condition, an enum constant, a caught exception path, a constructor signature, a validation error, a weak assertion, or a live mutant description. For example, for a catch-rethrow path, \(E\) may contain the source snippet catch (StreamException e) and the wrapper exception IOException. For a mutation objective, \(E\) may contain the mutator name, mutated line, and live mutant id.

Required setup. The field \(S\) describes what must be arranged before executing the target action. This may include constructing an object, selecting an input value, positioning an internal state through public APIs, configuring a mock collaborator, or routing through a facade method. For example, covering a Token.NULL branch under a FIELD_NAME guard may require the setup “advance the stream to FIELD_NAME, then advance to NULL.”

Target action. The field \(A\) describes the action that the test must execute. Usually, this is a direct call to the focal method. For non-public targets, \(A\) may be an accessible entry route, such as a public facade method that eventually reaches the target. For mutation objectives, \(A\) identifies the call intended to exercise the mutated code.

Oracle shape. The field \(Q\) describes the expected form of the assertion, not necessarily the final assertion text. Oracle shapes include exact-value assertions, null assertions, exception assertions, state-change assertions, interaction assertions, and differential assertions. For example, an objective may specify that the test should use assertThrows(IOException.class, ...) for an exception path, or assertEquals("42", result) for a mutant that changes an integer-to-string return value.

Acceptance predicate. The field \(\alpha\) is a measurable predicate that determines whether the objective has been satisfied after executing generated tests. It maps a feedback record to a Boolean value: \[\alpha(F) \rightarrow \{true,false\}.\] For a coverage objective, \(\alpha\) checks whether the corresponding line or branch is covered in the JaCoCo report. For a validation objective, \(\alpha\) checks whether the candidate compiles, executes, and reaches the target. For an oracle objective, \(\alpha\) checks whether the weak assertion has been replaced by a stronger oracle shape. For a mutation objective, \(\alpha\) checks whether a live mutant is killed.

Status. The field \(status\) records the objective lifecycle: \[status \in \{open, attempted, satisfied, blocked\}.\] An objective is open when it has not yet been addressed. It becomes attempted when a generated test claims to target it. It becomes satisfied when its acceptance predicate evaluates to true. It becomes blocked when generated tests repeatedly fail validation or cannot reach the target action.

De-duplication key. The field \(key\) records the behavioral signature of an objective attempt. It is not a test name or a hash of the generated code. Instead, it summarizes the behavior being attempted: \[key = \langle cat,\;S,\;A,\;Q \rangle .\] For example, two tests that both exercise the Token.NULL arm after entering the FIELD_NAME state and both assert a null result share the same key, even if their code differs syntactically. When an objective repeatedly fails to make progress, either due to execution failures or inability to reach the target behavior, TATG retires the objective from \(\mathcal{G}\) to avoid repeatedly exploring redundant generation strategies.

3.3 Objective Projection↩︎

Table 1 summarizes how the objective fields (cat, \(S\), \(A\), and \(Q\)) are instantiated from different types of evidence (\(E\)). We consider five evidence sources: static analysis, compilation/execution diagnostics, coverage feedback, oracle feedback, and mutation feedback. The first six rows correspond to structural objectives derived from source code, whereas the remaining rows correspond to feedback-driven objectives derived during iterative generation. For each evidence type, the table specifies how the setup (\(S\)), target action (\(A\)), and oracle shape (\(Q\)) are constructed.

Structural objectives are initialized before the first generation round. Most structural objectives are derived through static analysis, including StateSetup, DispatchArm, ExceptionPath, Construction, BoundaryInput, and EntryRoute. However, some testing requirements cannot be reliably inferred through static analysis alone. Therefore, TATG performs additional LLM-based objective inference, using the objective definitions and the focal method source code to identify higher-level testing objectives. For example, from the code fragment in Lines 17–22 of Listing [lst:running], the LLM infers an ExceptionPath objective corresponding to the try–catch–rethrow path. It may further infer a StateSetup objective indicating that the collaborator state must be initialized to a specific value before invoking the focal method. This additional objective subsequently guides test generation toward triggering catch block P5.

Feedback-driven objectives are generated incrementally throughout iterative generation. Compilation and execution diagnostics, along with coverage reports, generate ValidationRepair objectives, weak assertions identified in generated tests generate OracleStrengthening objectives, and mutation testing generates MutationReachability and MutationAssertion objectives for uncovered and live mutants, respectively.

Table 1 lists all structural objectives that can be obtained through static analysis. The additional LLM-based objective inference complements static analysis by identifying objectives that require semantic reasoning beyond what syntactic program analysis can provide. In contrast, feedback-driven objectives are introduced dynamically as new evidence becomes available after each generation round. Together, these objectives provide a unified representation that accumulates structural knowledge and execution feedback throughout the iterative generation process.

3.4 Prompt Construction↩︎

Listing [lst:prompt-structure] shows the prompt structure. The central part of the prompt is the set of objective cards. Each objective card is a textual rendering of one active objective. It contains the same fields as the objective state defined in Section 3.2. If the objective was updated in a previous round, the card also includes a brief prior observation explaining what happened, such as “attempted but did not reach the branch”, “compiled but failed at runtime”, or “produced a live mutant”.

Task:
  Generate or revise JUnit tests for the focal method.

Focal method context:
  focal_method:
    <method source>
  object_context:
    <relevant fields and helper methods>
  construction_context:
    <constructors, factories, builders, mocks, or subclasses>
  entry_context:
    <direct call or accessible caller routes>
  test_stack:
    <JUnit version, Mockito availability, Java version>

Active objective cards:
  Objective <G_i>:
    source:
      static | coverage | validation | oracle | mutation
    category:
      StateSetup | DispatchArm | ExceptionPath |
      Construction | BoundaryInput | EntryRoute |
      ValidationRepair | OracleStrengthening |
      MutationReachability | MutationAssertion
    status:
      open | attempted | blocked
    evidence:
      <source lines, branch condition, diagnostic,
       weak assertion, or mutant description>
    prior_observation:
      <what happened in the previous round, if any>
    required_setup:
      <object state, input values, mock behavior,
       construction strategy, or entry route>
    target_action:
      <direct target call or accessible caller route>
    oracle_shape:
      <exact value, null, exception, state change,
       interaction, or differential assertion>
    acceptance_condition:
      <how TATG will check whether G_i is satisfied>
    de_duplication_key:
      <cat, setup, action, oracle shape>

Global constraints:
  - compile under the project build
  - do not modify production code
  - use only available project dependencies
  - preserve already passing tests when repairing
  - avoid duplicating existing objective keys
  - return tests in the required format

Output contract:
  new_tests: [
    {
      objective_id: "<G_i>",
      test_name: "<name>",
      input_family: "<declared input partition>",
      setup: "<state/mocking/construction>",
      oracle_shape: "<declared oracle shape>",
      test_code_lines: ["<JUnit code lines>"]
    }
  ]

This structure allows each testing goal to receive its own feedback. For example, if a previous test claimed to target the branch P2 in Listing [lst:running] but failed, TATG does not simply append “missed branch at line 10” to the next prompt. Instead, it updates the corresponding DispatchArm objective: the evidence records the missed branch condition, the prior observation records the failed attempt, the required setup states how to reach the NULL state, the oracle shape specifies the expected assertion, and the acceptance condition states that P2 must be covered in the next run.

When a generated test fails to compile or execute, TATG creates or updates a ValidationRepair objective. The evidence field records the compiler diagnostic, execution, or target-reach failure; the required setup field describes the suspected fix; the target action maintains the intended call or entry route; and the acceptance condition requires that the revised test compile, execute, and reach the target. Thus, repair is not handled by a separate prompt. It is represented as another objective category in the same objective state.

The LLM response fields are used to associate generated tests with objectives during validation. After execution, TATG updates each objective according to its acceptance condition and de-duplication key. This is the mechanism that turns multi-round prompting into explicit objective-state tracking.

3.5 Budgeting and Termination↩︎

TATG allocates the per-iteration generation budget according to the number of active objectives, rather than using a fixed number of tests for every round. The number of requested tests is computed as \(B_r=\mathrm{clip}(B_{\min}, B_{\max}, \lceil \lambda |\mathcal{A}_r| \rceil)\), where \(\lambda\) controls how many candidates are allocated per active objective, and \(B_{\min}\) and \(B_{\max}\) bound the budget. The concrete values of these parameters are reported in the evaluation setup. A stage terminates when its configured round limit is reached, when no active objective remains, or when recent rounds do not satisfy any new objectives.

4 Experimental Design↩︎

We evaluate TATG by addressing the following research questions (RQs):

• RQ0: Overall performance. How does TATG compare with representative baselines across effectiveness and efficiency metrics?

• RQ1: Comparison with coverage- and mutation-oriented approaches. How effective are the structural stage and the hardening stage compared with representative coverage-oriented and mutation-oriented approaches, respectively?

• RQ2: Ablation study. How does each stage contribute to the overall effectiveness of TATG?

RQ0 evaluates the overall effectiveness and efficiency of TATG as a complete test-generation framework. RQ1 compares each stage with recent approaches that target the same objective, namely, structural coverage and mutation score, respectively. RQ2 isolates the contributions of the two stages through ablation analysis. Specifically, the structural stage focuses on producing executable tests and improving structural coverage, whereas the hardening stage leverages mutation feedback to strengthen assertions and improve fault-detection capability.

We evaluate TATG on 141 complex Java methods, including all 110 subjects from the KTester benchmark and 31 additional challenging methods drawn from larger multi-module projects. The remainder of this section presents the datasets, baselines, evaluation metrics, and implementation.

4.1 Datasets↩︎

We adopt all focal methods from the KTester benchmark [2], which contains 110 complex Java methods from 10 open-source projects. To further evaluate TATG on even more challenging real-world software systems, we collect 31 additional methods from 10 open-source projects. Compared with the projects used by KTester, the additional projects are selected from the MocklessTester benchmark [10]. They include both Defects4J projects and modern multi-module Java projects (cutoff date: March 2025), such as JQuick-Curl [11], ADK Java [12], AgentScope Java [13], and A2A Java [14]. Given the high computational cost of evaluation, these subjects are carefully selected through purposive sampling. We first prioritize projects not included in KTester to increase project diversity. Within these projects, we favor methods with relatively high cyclomatic complexity (CC) and non-comment lines of code (NLOC), as these characteristics generally indicate greater testing complexity. We further select methods that exhibit features known to challenge automated test generation, including package-private members (i.e., methods without an explicit access modifier), private methods, and dependencies that require constructing objects from external classes. Consequently, the additional subjects cover a broader range of access-control constraints and dependency patterns commonly encountered in real-world Java projects.

Table 2 summarizes the benchmark subjects, including the number of methods, cyclomatic complexity (CC), and non-comment lines of code (NLOC). CC and NLOC are computed using Lizard [15]. Across all 141 methods, the mean CC is 18.5, the median CC is 16, and the mean NLOC is 59.8, indicating substantial structural complexity [9], [16]. All methods are extracted from real-world open-source projects, ensuring that the evaluation is conducted on realistic testing tasks.

4.2 Baselines↩︎

We compare TATG against representative traditional, hybrid, and LLM-based test generation approaches.

• Randoop [17]: a feedback-directed random testing tool that generates unit tests through randomized sequences of method and constructor invocations.

• EvoSuite [18]: a search-based test generation tool that optimizes generated test suites toward structural coverage objectives.

• EvoObj [19]: an extension of EvoSuite that improves object-oriented test generation by addressing object construction challenges.

• KTester [2]: a recent LLM-based test generation approach that injects project-structure and project-usage knowledge into the generation process.

• PANTA [3]: a recent LLM-based approach that combines static analysis and coverage-guided feedback to target under-tested execution paths.

• MUTGEN [9]: a recent LLM-based approach targeting mutation score through mutation testing feedback, used for comparison with the hardening stage.

4.3 Evaluation Metrics↩︎

Effectiveness metrics. TATG aims to improve both the structural coverage and the fault-detection capability of generated tests. Following prior work [2], we report line coverage and branch coverage at the method level measured using JaCoCo [20]. Line coverage measures the percentage of executable lines in the focal method covered by the generated tests, while branch coverage measures the percentage of conditional branches exercised.

To evaluate fault-detection capability, we use mutation score as the primary effectiveness metric. Mutation score measures the percentage of generated mutants detected by the generated test suite and is computed as:

\[\text{Mutation Score} = \frac{\text{\# killed mutants}}{\text{\# total mutants}} \times 100\label{eq:mutation95score}\tag{1}\]

Following prior work [2], [3], [9], mutants are produced by PITest [21] with its default mutator set. A mutant is considered killed if at least one generated test distinguishes the behavior of the mutated program from that of the original program, typically through a failed assertion or an unexpected exception. A higher mutation score indicates stronger fault-detection capability.

Efficiency metrics. We report the average number of generated test cases and the average end-to-end execution time per focal method. The former measures the number of test methods contained in the final generated test class, while the latter measures the average time required to complete the entire testing workflow of each approach for a focal method.

4.4 Implementation↩︎

All benchmark subjects are built using Maven. Following the official documentation of each tool, we integrate EvoSuite, Randoop, EvoObj, JaCoCo, and PITest into the Maven pipeline for automated test generation, coverage measurement, and mutation testing, respectively. We used a per-focal-method generation budget of 900 seconds and fixed the random seed to 3 for EvoSuite, Randoop, and EvoObj. For approaches requiring multiple random seeds, we report the average results across runs. Each of the 141 focal methods was evaluated independently. Generated tests were installed in isolated, copied workspaces and then evaluated using JaCoCo and PITest for method-level coverage and mutation scores. Among the baselines, as neither Randoop nor PANTA natively supports method-level generation, both are executed independently for each focal method, with PANTA additionally prompted to generate tests only for the target focal method.

TATG is implemented in Python. Static analysis is implemented using SootUp [22], including call-graph construction, type analysis, constructor discovery, branch analysis, exception analysis, and entry-route extraction. As described in Section 3.5, we set the maximum number of generated tests per focal method \(B_{\max}\) to 50, the minimum number \(B_{\min}\) to 1, and the factor \(\lambda\) to 1. Static analysis is limited to 300 seconds per method, and the overall test-generation process to 1800 seconds. We use five iterations for each of the structural and hardening stages, for a total of 10 test-generation iterations.

We deploy Qwen3.6-35B-A3B locally using vLLM [23] with its default configuration and a temperature of 0.2. The same configuration is used throughout all experiments. To assess the generality of the results obtained with TATG, we also evaluate it on the DeepSeek-V4-Flash-284B model served via Fireworks AI [24], using the same configuration. Both models were released around April 2026.

To ensure a fair comparison, all approaches are executed using the same project-specific Java version and JUnit 5 framework. Coverage is measured with JaCoCo, and mutation testing is performed with PITest using its default mutator set. We run KTester and PANTA for 10 iterations. We run MUTGEN for five iterations, the same as in the hardening stage.

All experiments are conducted on a Linux server running Ubuntu 24.04.1 LTS with a 56-core CPU, 125 GB of RAM, and two NVIDIA RTX 6000 Ada GPUs.

The implementation of TATG and the benchmark subjects will be made publicly available.

5 Experimental Results↩︎

4pt

5.1 RQ0: Overall Performance↩︎

We compare the end-to-end performance of the two-stage TATG with the first five baselines listed in Section 4.2. All LLM-based and hybrid approaches are evaluated using two recently released open-source LLMs, Qwen3.6-35B-A3B (35B) and DeepSeek-V4-Flash (284B), which represent two different model scales and have demonstrated strong code-generation performance. Our approach is the first LLM-based test generation approach that jointly targets structural coverage and mutation score. Since no existing LLM-based approach optimizes both objectives simultaneously, the end-to-end comparison is limited to recent coverage-oriented approaches: KTester and PANTA.

The complete results are reported in Table 3. Across both models, TATG consistently achieves the best overall performance in terms of line coverage, branch coverage, and mutation score while generating fewer test cases than the baselines. Specifically, TATG improves both structural coverage and fault-detection capability over existing approaches on both models, demonstrating that the proposed two-stage objective-tracking framework is effective regardless of the underlying LLM. We further conducted paired-sample Wilcoxon signed-rank tests on line coverage, branch coverage, and mutation score. The results show that TATG achieves statistically significant improvements over both PANTA and KTester across all three metrics on both LLMs (\(\alpha = 0.05\)).

In terms of generation time, the primary overhead comes from invoking PITest during each hardening iteration to obtain mutation feedback. Although mutation testing accounts for most of the additional runtime, this overhead is acceptable in practice because it further improves mutation score and structural coverage, yielding higher-quality test suites at the cost of only about 8 additional minutes.

For traditional approaches, EvoSuite and Randoop achieve nearly complete coverage on relatively easy subjects. However, their overall performance is substantially reduced because they fail to generate valid tests for many of the more challenging subjects, especially those with rich project dependencies. Although EvoObj enhances EvoSuite with an LLM, its overall performance is still limited by the capabilities of the underlying EvoSuite framework.

In addition, we collaborated with an industrial partner to evaluate their proprietary test generation tool, which is based on the GLM-5.2-753B model. Due to the high computational cost, the comparison was conducted on 50 focal methods, including 39 randomly selected methods from the 110 KTester subjects and 11 methods from our additional benchmark, preserving the original ratio of 110:31. The industrial tool achieved 68.23% line coverage, 66.46% branch coverage, and a mutation score of 46.14%. In comparison, TATG with the same model achieved 75.01% line coverage, 65.12% branch coverage, and a mutation score of 53.72%. These results indicate that TATG achieves comparable branch coverage while substantially improving line coverage and mutation score. Moreover, TATG successfully generated valid tests for six additional focal methods for which the industrial tool failed. Due to the high inference cost of GLM-5.2-753B, we do not use this model for the full benchmark evaluation or for comparisons with all baselines. Instead, the comparison with the industrial tool is conducted on the selected subset using the same underlying model to ensure a fair evaluation.

5.2 RQ1: Comparison with Coverage- and Mutation-Oriented Approaches↩︎

To demonstrate the effectiveness of the proposed two-stage framework, Table 3 reports not only the end-to-end performance of TATG, but also the performance of its structural and hardening stages separately. Since TATG is the first LLM-based approach that jointly targets structural coverage and mutation score, we compare the structural stage with recent coverage-oriented approaches and the hardening stage with the mutation-oriented baseline MUTGEN.

Across both LLMs, the structural stage consistently outperforms all coverage-oriented baselines. Specifically, compared with the best baseline PANTA, as shown in Table 3, the structural stage of TATG improves line coverage by 11.48% and branch coverage by 4.97% on Qwen, and by 6.06% and 5.72% on DeepSeek. Notably, these improvements are achieved using only five iterations, whereas PANTA is configured with ten iterations. For the hardening stage, TATG consistently achieves higher mutation scores than MUTGEN, improving the mutation score by 2.88 % on Qwen and by 2.93 % on DeepSeek. Moreover, the hardening stage further improves the line and branch coverage by 13.38% and 14.79% on average, respectively, compared with 10.10% and 10.59% achieved by MUTGEN.

The improvements achieved by both stages originate from the unified objective representation. Instead of repeatedly prompting the LLM with newly obtained feedback at each iteration, TATG maintains explicit objective states throughout the iterative generation process. Completed objectives are not revisited, allowing subsequent iterations to focus exclusively on unresolved testing goals. Furthermore, each objective provides fine-grained guidance beyond less informative coverage or mutation feedback, including input constraints, dependency construction, and mutation-specific requirements. This objective-tracking mechanism enables the LLM to make more effective use of each generation iteration, resulting in improvements in both structural coverage and mutation score.

5.3 RQ2: Ablation Study↩︎

Table 3 shows that the hardening stage makes a substantial contribution to the overall performance by consistently improving the mutation score over the structural stage alone.

To evaluate the contribution of the structural stage, we conduct an ablation study by removing it and running only the hardening stage for 10 iterations. Since PITest requires at least one executable test to produce mutation feedback, the first iteration is performed without mutation feedback to bootstrap test generation. Without the structural stage, TATG achieves only 26.90% line coverage, 20.45% branch coverage, and a mutation score of 23.45% on Qwen. The corresponding results on DeepSeek are 24.38%, 18.37%, and 20.91%, respectively. Performance drops substantially in both structural coverage and mutation score. This result supports our design rationale that effective mutation-guided hardening relies on establishing sufficient reachability and structural coverage first.

5.4 Threats to Validity↩︎

Internal validity. LLM outputs are stochastic and may vary across model releases, inference settings, and execution environments. To reduce this threat, we evaluate all LLM-based approaches with the same model version, configuration, prompting budget, and execution environment, and apply the same validation, coverage, and mutation testing pipeline to all generated tests.

Construct validity. Construct validity may be affected by whether the selected metrics accurately reflect the objectives of the proposed approach. Since TATG explicitly targets structural coverage and mutation score, we evaluate it using method-level line coverage, branch coverage, and mutation score, following prior work.

External validity. External validity is limited by the benchmark and implementation scope. We reuse benchmark tasks, project versions, and evaluation protocols from prior work [2] and add 31 challenging methods from larger multi-module projects to enable broader assessment. The current implementation targets Java/JUnit. While TATG’s objective representation is language-independent, porting TATG to another ecosystem would simply require replacing the Java-specific providers used for static analysis, test execution, coverage measurement, mutation testing, and validation diagnostics, such as SootUp, JaCoCo, and PITest.

6 Related Work↩︎

We organize existing test generation approaches by the type of guidance used during the generation process. Recent approaches increasingly integrate multiple sources of guidance, including static analysis, coverage and mutation feedback, to iteratively refine generated tests. The key distinction between existing work and TATG lies in how these guidance signals are represented, managed, and propagated across iterations.

6.0.0.1 Traditional automated test generation

Search-based, random, and symbolic techniques predate LLMs and demonstrate the power of explicit coverage objectives and execution feedback. Tools such as EvoSuite [18] and its many‐objective extensions (e.g., MOSA [25], DynaMOSA [26]) optimise towards structural coverage targets, while Randoop [26] uses feedback-directed random exploration; symbolic executors like KLEE [27] and JPF [28] solve path constraints to synthesise inputs. These techniques have strong coverage and fault-detection potential, but they rely primarily on search heuristics, branch-distance functions, and path constraints rather than explicit representations of testing objectives, making it difficult to capture the complex project dependencies required to generate executable tests.

6.0.0.2 LLM-based and knowledge-guided generation

The emergence of large language models has prompted recent techniques that treat test generation as a code understanding and synthesis task. Approaches such as AthenaTest, TestSpark, ASTER, HITS, RAGTest, ChatUniTest, TestSpark and KTester guide LLMs by incorporating additional information into prompts through program analysis or retrieval techniques [1], [29], [30], [30]–[33]. These approaches highlight the importance of contextual guidance. However, testing intent and supporting knowledge are typically embedded in prompts in an ad hoc manner, making it difficult to represent, organize, and systematically reuse testing objectives.

6.0.0.3 Feedback‑guided improvement

Several approaches iteratively refine tests using execution or coverage feedback [4]–[6], [34], [35]. Among them, PANTA alternates static control‑flow analysis with dynamic coverage measurement to drive an iterative generate–execute loop and systematically explore uncovered branches [3]. These approaches underscore the value of feedback loops but primarily treat feedback as prompt text, coverage summaries, or repair requests without maintaining a structured state across iterations. In contrast, TATG normalises static and dynamic signals into a unified objective representation. Feedback thus becomes explicit and traceable, enabling unified management of structural and feedback-driven objectives across iterations.

6.0.0.4 Mutation‑guided generation

Mutation testing provides a stronger adequacy signal than structural coverage [9], [21]. Recent approaches, such as MuTAP [7], ACH [34], and MUTGEN [9], integrate mutation feedback into LLM‑based generation to strengthen assertions. These approaches demonstrate the effectiveness of mutation signals but treat mutants as separate targets. TATG includes mutation objectives in the same tracking model as structural and feedback-driven objectives; once reachability and basic assertions are achieved, mutation objectives trigger oracle strengthening within the two-stage generation process.

7 Conclusion↩︎

This paper presents TATG, an LLM-based unit test generation framework based on a unified objective representation. By representing static analysis results, execution feedback, coverage gaps, and mutation feedback as tracked objectives, TATG enables iterative guidance across generation rounds, allowing the LLM to focus on unresolved testing objectives.

We evaluated TATG on 141 complex Java methods against representative traditional, hybrid, and LLM-based test generation approaches. Experimental results show that TATG consistently achieves higher structural coverage and mutation score than existing baselines. On a randomly selected subset of 50 methods, TATG also achieves performance comparable to that of an industrial proprietary test generation tool while achieving higher line coverage and a higher mutation score. These results demonstrate that unified objective tracking enables more effective iterative guidance for LLM-based unit test generation.

Acknowledgment↩︎

This work has emanated from research jointly funded by Taighde Éireann – Research Ireland under Grant number 13/RC/2094_2 and by Huawei Technologies Co., Ltd. Lionel Briand is also supported by the Natural Sciences and Engineering Research Council of Canada. For the purpose of Open Access, the authors have applied a CC BY public copyright licence to any Author Accepted Manuscript version arising from this submission.

References↩︎