1 Introduction↩︎

Recent evaluations of coding agents are moving beyond isolated code generation toward more realistic software delivery tasks. Benchmarks such as HumanEval, SWE-bench, and LiveCodeBench have been effective for measuring function-level programming, general bug fixing, and repository-level code modification [1]–[3]. In real development workflows, however, many tasks require more than locally correct code: an agent must understand the business scenario, choose an appropriate domain solution, coordinate changes across components, and preserve safety and state consistency after implementation.

Payment integration is a typical example of this shift. For many applications, payment capability is a key step from product workflow to commercial operation. Yet integrating a payment product is not simply adding an API call. A correct implementation must select a suitable payment product, keep credentials and signing logic on the server side, coordinate frontend and backend payment flows, verify asynchronous notifications, and update local order or service state only after reliable payment confirmation. If an agent only makes the code appear runnable while trusting client-side payment status, skipping signature verification, mishandling repeated notifications, or causing inconsistency between payment state and business fulfillment, the result may introduce real fund-safety and business-consistency risks.

These properties make payment integration a challenging domain-specific setting for evaluating coding agents. It jointly tests code generation, cross-component integration, payment product understanding, risk handling, and business-state maintenance. Existing general-purpose benchmarks are not designed to systematically measure these capabilities, because the correctness of payment integration depends not only on local code behavior, but also on payment-domain rules, security boundaries, and consistency across the business workflow.

To address this setting, we propose Alipay-PIBench, an Alipay-led repository-level benchmark for evaluating AI coding agents on realistic Alipay payment integration tasks. Built on Alipay Open Platform products and business-style repositories, Alipay-PIBench evaluates not only whether an agent can complete a payment integration, but also whether the implementation is secure, reliable, convenient to integrate, and consistent with local business state.

Alipay-PIBench follows a systematic evaluation pipeline. First, we use Product-Grounded Task Construction to connect real payment products, business scenarios, initial repositories, and natural-language integration requests, so that each task evaluates both product understanding and repository-level implementation. Second, we divide payment integration into two progressive scenarios: Basic (Functional Payment Completion), which focuses on the core payment loop including payment creation, payment interaction, backend confirmation, and local state update; and Advanced (Risk-Aware Payment Hardening), which focuses on risk hardening such as signature verification, idempotency, abnormal transaction states, refunds, and fund-safety boundaries. Third, we adopt Rubric-Derived Evaluation, deriving deterministic checks and LLM-assisted assessment criteria from scenario-specific rubrics so that evaluation items remain traceable to payment-domain requirements rather than ad hoc tests or generic judge prompts. Finally, through a paired without-skill/with-skill setting, Alipay-PIBench evaluates whether the alipay-payment-integration skill helps agents complete payment integration more efficiently, conveniently, and safely.

This paper makes the following contributions:

• We propose the first Alipay-led repository-level benchmark for evaluating coding agents on realistic Alipay payment integration tasks.

• We propose Product-Grounded Task Construction, which organizes payment products, business scenarios, initial repositories, and natural-language integration requests into a unified task format.

• We introduce a progressive capability decomposition of payment integration into Basic (Functional Payment Completion) and Advanced (Risk-Aware Payment Hardening), distinguishing end-to-end payment completion from the ability to harden payment integrations against payment-specific risks and boundary failures.

• We develop Rubric-Derived Evaluation, which derives deterministic checks and supplementary large-language-model (LLM)-assisted rubric assessment criteria from scenario-specific payment rubrics, making evaluation evidence traceable to domain requirements.

• We establish a paired without-skill/with-skill intervention protocol to measure how access to structured Alipay payment guidance changes adherence to product-specific APIs and integration patterns, payment safety, payment–business state consistency, and output-token efficiency.

Across the 18 task instances and six evaluated models, mean RPR under the with-skill condition ranged from 68.58% to 91.37%. Compared with the corresponding without-skill condition, the with-skill condition improved mean RPR by 10.31 percentage points on average, with gains in 101 of 108 model–product–scenario comparisons. The average improvement was larger for Basic tasks than for Advanced tasks (+11.27 versus +9.35 percentage points). Method-level results further distinguish source-level completion, executable payment behavior, and payment-domain requirements, motivating the use of progressive scenarios and complementary rubric-grounded evaluation signals.

2 Related Work↩︎

2.0.0.1 Repository-level benchmarks for coding agents.

Coding benchmarks have expanded from function-level generation toward realistic software engineering. HumanEval, APPS, and LiveCodeBench evaluate isolated programming and algorithmic problem solving [1], [3], [4], while SWE-bench evaluates issue resolution in existing repositories through executable tests [2]. Recent benchmarks further cover repository-level feature implementation, project construction, and longer-horizon development. FEA-Bench studies feature addition [5]; ProjDevBench and NL2Repo-Bench evaluate project or repository construction from natural-language requirements [6], [7]; and BeyondSWE broadens evaluation to cross-repository reasoning, domain-specific repair, dependency migration, and repository generation [8]. DeepSWE further introduces original repository-scale tasks and behavior-oriented verifiers to reduce contamination and implementation-specific grading [9]. Domain-specific benchmark design is also exemplified by ABench-Physics, which evaluates model performance on high-difficulty, dynamic physics problems [10]. These benchmarks reflect a transition toward realistic software delivery, but mostly remain organized around general software-engineering task forms.

2.0.0.2 Production and payment-integration evaluation.

Production-oriented evaluations increasingly emphasize realistic repositories, deployment constraints, and reliable verification. ProdCodeBench derives executable tasks from production developer-agent sessions and applies test-relevance and stability checks during benchmark construction [11]. Security-oriented evaluations further examine whether functionally successful implementations satisfy deployment-critical security requirements [12]. Most closely related to the payment setting, Stripe reports an industry evaluation suite for coding agents on realistic Stripe integrations, using full coding environments and deterministic graders over APIs, user interfaces, and Stripe-side artifacts [13]. Together, these efforts demonstrate a broader shift from generic code correctness toward production-grounded evaluation under domain and platform constraints. Alipay-PIBench follows this direction by studying repository-level payment integration in the Alipay ecosystem, with additional attention to payment-specific risks and business-state consistency.

2.0.0.3 Rubric-grounded evaluation and domain guidance.

Executable tests provide reproducible evidence of software behavior, but may not fully capture product selection, security placement, or cross-component business consistency. Some recent benchmarks therefore combine execution-based evaluation with LLM-assisted review, although RuVerBench shows that LLM rubric verification in agentic coding remains noisy [6], [14]. In Alipay-PIBench, scenario-specific payment rubrics provide a common specification from which deterministic checks and supplementary semantic criteria are derived, while executable evaluation remains the primary correctness signal. Separately, Stripe’s agent-steering experiments indicate that explicitly loaded, structured skills can guide integration behavior more effectively than passive warnings or documentation [15]. Motivated by this practical question, Alipay-PIBench performs paired without-skill/with-skill trials under the same task, environment, and evaluation criteria to measure the effect of the Alipay payment skill.

3 Alipay-PIBench Overview↩︎

3.1 Benchmark Composition↩︎

Alipay-PIBench is a benchmark suite for repository-level payment integration. It is organized around product-specific projects, each corresponding to one Alipay payment product and an associated business setting. The selected products span multiple common Alipay integration patterns, including mobile-app payments, web-based payments, in-person code payments, pre-authorization, and recurring or subscription-style payments. These products differ in user entry points, payment-initiation flows, backend confirmation paths, and risk profiles, allowing the benchmark to evaluate both product understanding and repository-level implementation.

Each product-specific project contains two progressive scenarios. Basic (Functional Payment Completion) starts from a business workflow in which the target Alipay product has not yet been integrated, although another payment capability may already exist. Depending on the project, the agent may connect an existing payment entry point or add a product-specific entry point. Advanced (Risk-Aware Payment Hardening) starts after the core Alipay integration is present and evaluates payment-specific safety and boundary handling. The two scenarios are therefore progressive rather than independent: Basic evaluates whether the agent can establish the end-to-end payment loop, whereas Advanced evaluates whether it can harden that loop against risks such as invalid or duplicated notifications, non-idempotent state updates, abnormal transaction states, and unsafe refund behavior.

We define a task instance as a project–scenario pair. Each task instance contains agent-facing inputs and evaluator-facing materials. The agent-facing inputs consist of an initial repository and a scenario-specific natural-language instruction, and the expected output is a modified repository. The evaluator-facing materials consist of scenario-specific rubrics, deterministic checks derived from those rubrics, and supplementary LLM-assisted criteria for rubric items requiring semantic or payment-domain judgment. Pairing each of the nine product-specific projects with Basic and Advanced scenarios yields 18 task instances. This shared format supports consistent evaluation across products, scenarios, agents, and without-skill/with-skill conditions.

4pt

Table 1 summarizes the product-level coverage of the current release. The project column identifies the corresponding repositories or project sites [16]–[23]. For each product, the table records the project, backend technology stack, initial payment status, and the number of rubric items in the Basic and Advanced scenarios. Although product details differ, Basic tasks typically require coordinated changes to payment configuration, backend payment creation and confirmation, local business state, and application-facing payment entry points where applicable. Advanced tasks build on this foundation and require the agent to address payment-specific risks such as duplicate notifications, repeated confirmation calls, inconsistent local states, unsafe refund behavior, and insufficient verification of asynchronous notifications.

3.2 Product-Grounded Task Construction↩︎

3.2.0.1 Project and scenario initialization

We construct task instances from business-style applications and payment-specific integration requirements. For each Alipay payment product, we select a repository in which the surrounding business workflow is already implemented. The repository’s existing payment state is product-dependent: some projects contain no payment capability, whereas others contain an existing payment integration. These differences are retained as part of the task context and are summarized in Table 1.

For each project, we prepare two independently initialized scenario states. The Basic state is a repository snapshot taken before the target Alipay product is integrated. The Advanced state is a separately prepared snapshot in which the core target-product payment path is already present, so that the task can focus on product-specific safety, fund-risk, and boundary requirements. Basic and Advanced are evaluated as separate controlled task instances with their own scenario-specific instructions, rubrics, and starting repositories.

3.2.0.2 Instruction and rubrics

For each product–scenario pair, we prepare a payment-specific natural-language instruction and a corresponding set of rubrics. The instruction defines the task presented to the agent. It specifies the business objective, repository context, payment behavior, and constraints that the solution must satisfy. Written as a developer-facing integration request, it generally leaves internal code organization open while specifying externally observable interfaces, payment semantics, security constraints, and, where necessary, required API, SDK, or configuration behavior. The target payment product is explicitly specified through the task metadata and corresponding instruction, which may additionally provide product-specific API and business-context details. Where relevant, the instruction also identifies the interfaces and observable behaviors used to evaluate the completed integration.

The instruction reflects the starting state of each scenario. A Basic instruction asks the agent to add or complete the target payment capability within an existing business workflow. An Advanced instruction addresses safety-sensitive or boundary requirements in a separately prepared payment-integration context. The task description and interface requirements remain specific to each product and project, with product-specific API and integration requirements grounded in the Alipay Open Platform documentation [24].

The rubrics decompose each instruction into checkable requirements. Their composition varies across products and projects. Basic rubrics may assess payment creation, signing and credential handling, application-facing invocation, payment-result processing, and consistency between payment and business states. Advanced rubrics may assess safety-sensitive properties such as idempotent state transitions, duplicate-notification handling, abnormal transaction states, notification authenticity, or refund boundaries, where relevant to the task.

3.2.0.3 Running example

Figure 2 illustrates the construction and evaluation of the LaravelGymie-JSAPIPayment-basic task. The initial Laravel repository provides a gym-membership workflow, including membership plans, order creation and query APIs, and an Alipay Mini Program page, but stops after creating a pending order. The instruction asks the agent to complete the existing workflow with Alipay Mini Program payment rather than introduce a replacement API. A valid solution must create the payment trade on the server, return the case-sensitive tradeNO field for native Mini Program invocation, and advance the order and membership fulfillment only after a trusted server-side result. The task rubrics further require invalid notifications to be rejected, repeated success notifications to be handled idempotently, and client-side callbacks not to determine the final payment state.

The same scenario-specific rubrics are instantiated as task-specific deterministic checks, including static, unit, integration, and E2E tests where applicable, with product-specific client checks where applicable, together with supplementary LLM-assisted criteria for properties that require semantic code inspection. The resulting evaluators are then reused for the paired without-skill/with-skill comparison.

3.2.0.4 Rubric-Derived Evaluation

The scenario-specific rubrics serve as the source specification for evaluation. For rubric items that can be operationalized as executable checks, we derive deterministic evaluators, including static checks, unit tests where applicable, integration tests, and E2E or environment-level validation. Static checks examine structural properties such as the presence and placement of payment-related code, configuration, and credential handling. Unit and integration tests examine local payment logic, backend API behavior, state transitions, and interactions between payment and business modules.

When evaluation requires Alipay SDK calls, we use the Alipay sandbox environment where reliable support is available. When a product is not supported by the sandbox, or when rare abnormal states and risk boundaries cannot be triggered reproducibly, we use deterministic mock responses or fixtures. These cases are derived from the same scenario-specific rubrics and manually reviewed to ensure that they evaluate payment behavior rather than incidental implementation details.

For rubric items that cannot be fully captured by executable checks, we derive supplementary LLM-assisted assessment criteria from the same rubrics. These criteria examine properties such as payment-product fit, end-to-end flow consistency, security-sensitive logic placement, and payment–business state consistency. We manually review both deterministic checks and LLM-assisted criteria for consistency with the task instruction and the intended rubric item. This procedure makes the resulting evidence traceable to the same task specification rather than defining executable and semantic evaluators independently.

3.2.0.5 Paired skill intervention

Finally, we define the paired intervention used to study the practical value of structured payment guidance. In the without-skill condition, the agent receives the project and task instruction without access to alipay-payment-integration. In the with-skill condition, the same agent also receives the official alipay-payment-integration skill, which provides reusable integration guidance rather than task-specific solutions. Its scope includes payment-product guidance, integration procedures, interface explanations, SDK and sandbox guidance, code examples, and common payment-integration pitfalls.

For each pair, we keep the agent, project, task instruction, and execution environment fixed, and evaluate both outputs using the same scenario-specific rubrics and evaluators. The paired protocol supports analysis of how access to structured Alipay payment guidance affects payment-product selection, integration efficiency, payment safety, and payment–business state consistency.

3.3 Evaluation and Scoring↩︎

3.3.1 Evaluation Signals↩︎

We evaluate each generated solution using test items aligned with its scenario-specific rubrics. The evaluator executes these checks on the modified repository. Each item is assigned to static analysis, unit testing where applicable, integration testing, E2E validation, or LLM-assisted review.

3.3.1.1 Static checks.

Static checks examine source-level and structural signals, including SDK or OpenAPI usage, credential handling, payment entry points, signature-verification capability, field binding, and state or refund models where applicable. For safety tasks, they also inspect notification-verification hooks, fake-success bypasses, idempotency or terminal-state guards, refund or query capability, and secret-management safeguards. These checks provide lightweight evidence of the expected implementation structure, but do not establish end-to-end runtime correctness.

3.3.1.2 Unit tests.

Unit tests are used where a task provides applicable local test coverage, such as project-specific application tests.

3.3.1.3 Integration tests.

Integration tests verify application startup, product-specific payment flows, backend state transitions, and interactions with payment gateways or deterministic fixtures. Safety tasks additionally cover duplicated notifications, repeated requests, invalid signatures, wrong amounts, exceptional transaction states, refund boundaries, and related risks where applicable. Some tasks use local mock Alipay gateways or deterministic fixtures for reproducibility rather than relying uniformly on live sandbox behavior.

3.3.1.4 E2E tests.

End-to-end (E2E) tests exercise the user-facing or integration-facing entry points required by each task. They verify endpoint reachability, backend invocation, and observable states such as success, failure, cancellation, and pending where applicable. Some combined business-and-payment logic is also exercised by integration tests.

3.3.1.5 LLM-assisted domain assessment.

LLM-assisted domain assessment examines rubric-defined semantic properties, including product mapping where applicable, server-side signing and confirmation, state-machine coherence, security-sensitive logic placement, and cross-component linkage. For safety tasks, it further assesses whether the implementation addresses the requested risk class rather than merely preserving the happy path. This signal helps identify plausible but unsafe implementations that deterministic checks may miss.

3.3.2 Rubric-Level Metric and Aggregation↩︎

We use Rubric Pass Rate (RPR) as the primary metric. RPR is the fraction of rubric requirements satisfied by a generated solution. A configuration \(c\) denotes a model evaluated under one skill condition, either without-skill or with-skill. For task instance \(t\) and trial \(r\), let \(\mathcal{M}_t\) denote the evaluation methods used for \(t\). Let \(\mathcal{I}_{t,m}\) denote the rubric items evaluated by method \(m\). We encode each outcome as \(y_{c,t,r,i}\in\{0,1\}\): pass is 1, and fail or evaluator error is 0. The weighted RPR is \[\mathrm{RPR}_{c,t,r} = \frac{ \sum_{m \in \mathcal{M}_t} \alpha_m \left( \frac{1}{|\mathcal{I}_{t,m}|} \sum_{i \in \mathcal{I}_{t,m}} y_{c,t,r,i} \right)}{\sum_{m \in \mathcal{M}_t} \alpha_m},\] where \(\alpha_m\) is the relative weight of method \(m\). We use conservative relative weights: \[\alpha_{\mathrm{static}}=\alpha_{\mathrm{unit}}=\alpha_{\mathrm{llm}}=1, \qquad \alpha_{\mathrm{integration}}=\alpha_{\mathrm{e2e}}=2.\] The denominator normalizes over the methods available for \(t\). Thus, RPR remains in \([0,1]\) when task instances use different evaluator sets. We assign higher weight to integration and end-to-end checks because they more directly test whether the generated repository preserves a coherent payment flow across backend services, application state and user-facing entry points. Static, unit and LLM-assisted checks provide complementary but more localized evidence.

Reported scores are macro-averages over project-scenario task instances. For repeated trials of the same task instance, we first compute its mean RPR, so each task instance contributes equally regardless of its rubric count. LLM-assisted domain assessment enters the same aggregation as a rubric-aligned method. Evaluator errors are also retained separately for diagnostic analysis.

4 Experiments and Results↩︎

To assess the performance of coding agents on realistic Alipay payment integration tasks, we evaluated six models on all 18 task instances in Alipay-PIBench. Each instance paired a project with a scenario and was evaluated under paired conditions with and without access to alipay-payment-integration. The experiments were organized around three research questions:

• RQ1: Model capability. How capable are current models at completing Alipay payment integration across products and scenarios?

• RQ2: Skill intervention. How does access to alipay-payment-integration change performance?

• RQ3: Evaluation-method diagnostics. What aspects of Alipay payment integration capability are revealed by different evaluation methods?

4.1 Experimental Setup↩︎

We conducted all experiments on Alipay-PIBench using Claude Code version 2.1.200. The evaluation covered six high-performing model variants available in our environment at the time of the study: Claude Opus 4.8, GLM-5.2, Kimi K2.7 Code, DeepSeek-V4-Pro, MiniMax M3, and Qwen3.7-Max. Official documentation is available for the Claude Opus 4.8, GLM-5.2, Kimi K2.7 Code, DeepSeek V4, and MiniMax M3 variants used in this study [25]–[29]. Each trial started from a fresh Docker container initialized with the repository and evaluation environment for the corresponding task. Consequently, files, dependency caches, service states, and evaluator outputs generated during one trial did not affect any other trial. For each combination of model, project, scenario, and skill condition, we kept the task configuration and evaluation procedure fixed.

4.2 RQ1: Model Capability↩︎

RQ1 characterizes model capability across Alipay payment integration products and scenarios. The results reported here come from configurations in which alipay-payment-integration was installed in the coding agent. Under this setting, the comparison focuses on how models use repository context, payment-domain instructions, and project-specific code structure. Figure 3 reports mean RPR for each model, product, and scenario. The top and bottom panels correspond to Basic and Advanced tasks, respectively, with an independent color scale in each panel.

First, model capability varies substantially across Alipay payment integration tasks. Claude Opus 4.8 achieves the highest overall mean RPR (91.37%), followed by GLM-5.2(87.12%) and Kimi K2.7 Code(82.18%). Qwen3.7-Max and DeepSeek-V4-Pro obtain lower intermediate scores of 75.78% and 75.23%, respectively, while MiniMax M3 achieves 68.58%. The 22.79 percentage-point gap between the highest and lowest scores indicates substantial variation in model capability.

Second, this variation is also reflected in performance consistency across products and scenarios. Claude Opus 4.8 achieves at least 90% mean RPR in 13 of the 18 product–scenario cells, and GLM-5.2 reaches this level in 10 cells. Kimi K2.7 Code reaches the same level in 6 cells, while Qwen3.7-Max and DeepSeek-V4-Pro do so in 2 cells each. At the same time, even Claude Opus 4.8 falls to 65.73% on Advanced In-App Payment, whereas MiniMax M3 ranges from 46.54% on Advanced Authorization Hold to 98.00% on Basic QR Code Payment. These results show that model capability is expressed through both overall performance and consistency across payment products and scenarios.

Third, product context shapes how model capability is expressed. App has the lowest overall score (58.65%), and only Claude Opus 4.8 exceeds 65% mean RPR on this product (71.30%). The source application originally supported online booking followed by offline settlement, rather than an existing online payment flow. Agents therefore had to introduce a payment entry point, backend payment state, and confirmation logic into a workflow not organized around online settlement. The other models score lower on App, including GLM-5.2 at 62.71%, Qwen3.7-Max at 55.39%, and DeepSeek-V4-Pro at 51.12%. Pre-Authorization exhibits a different profile. Unlike In-App Payment, which is challenging for nearly all models, Pre-Authorization shows substantial variation across models and scenarios. Its overall mean RPR is 69.76%, while individual scores range from 42.96% to 98.67%. These results show that some products are difficult for nearly every model, whereas others differentiate models by their ability to maintain payment state, confirmation, and risk-handling logic across a longer code path.

Finally, Basic and Advanced tasks provide complementary views of model capability. Basic tasks primarily test whether a model can establish the initial Alipay payment loop. Advanced tasks focus on fund safety, risk handling, and boundary conditions after a basic integration exists. The scenario split therefore reveals different model profiles rather than a single difficulty ordering. Claude Opus 4.8 remains strong in both scenarios, with scores of 92.58% on Basic tasks and 90.17% on Advanced tasks. GLM-5.2 is similarly stable, scoring 87.05% and 87.19%, respectively. By contrast, Qwen3.7-Max scores lower on Basic tasks (70.88%) than on Advanced tasks (80.67%), whereas MiniMax M3 performs better on Basic tasks than on Advanced tasks (70.20% versus 66.97%). These results indicate that constructing a payment flow and preserving safety-related payment behavior should be examined as distinct capabilities.

RQ1 Summary. Model capability explains much of the observed performance spread and is expressed through overall performance, cross-product consistency, and adaptation to scenario-specific requirements. Alipay-PIBench therefore supports a fine-grained characterization of current models, showing where their Alipay payment integration capability is strong and where it remains limited.

4.3 RQ2: Skill Intervention↩︎

RQ2 examines how access to alipay-payment-integration changes coding-agent performance on Alipay payment integration tasks. We compare paired without-skill and with-skill trials under the same task, repository, and evaluation procedure. Table ¿tbl:tab:rq2-skill-effect-performance? reports mean RPR for each model, product, and scenario, together with the percentage-point change associated with access to alipay-payment-integration.

Across all 108 model–product–scenario cells, access to alipay-payment-integration is associated with a 10.31 percentage-point increase in mean RPR. The paired change is positive in 101 cells, lower in 4 cells, and effectively unchanged in 3 cells. The gain is larger for Basic tasks than for Advanced tasks (+11.27 versus +9.35 points), but both scenarios benefit. Every evaluated model also improves on average, with gains ranging from +6.81 points for Claude Opus 4.8 to +15.51 points for Kimi K2.7 Code. These results show a clear positive association under the paired evaluation protocol, with the magnitude varying by model, product, and scenario.

The gain is larger when the without-skill baseline is low. Cells with a without-skill mean RPR below 40% improve by 17.64 points on average, compared with 4.52 points for cells with baseline scores between 80% and 100%. This pattern suggests that alipay-payment-integration is particularly useful when the baseline solution has not recovered the payment-flow structure, although the smaller gains at high baseline scores may also reflect a ceiling effect. Representative examples include Kimi K2.7 Code on Advanced Pre-Authorization, which improves from 31.43% to 79.43% (+47.99 points), and DeepSeek-V4-Pro on Basic Barcode, which improves from 58.33% to 96.67% (+38.33 points). These examples indicate that alipay-payment-integration is especially useful when models must recover product selection, payment-state transitions, payment confirmation, and callback or notification handling from repository context.

The few lower-scoring cells are concentrated in Advanced tasks, where evaluation emphasizes safety and boundary behavior rather than simply adding a missing payment entry point. The remaining decreases are isolated across products and models, rather than indicating a consistent degradation under with-skill. Manual inspection of the corresponding execution traces indicates that these cases are associated with runtime interruptions or implementation failures in individual trials.

As a secondary analysis, we examined output tokens normalized by mean RPR to assess generation efficiency. For each model and scenario, we averaged the paired results across the nine payment products, yielding 12 model–scenario averages in total (6 models \(\times\) 2 scenarios). The relative output/RPR ratio was below 1.0 for 11 of the 12 averages, ranging from 0.72 to 1.03, with a mean of 0.86. This result indicates that alipay-payment-integration generally reduced the amount of generated output required for each unit of rubric completion, although the effect was not universal. Because the ratio captures only output tokens relative to rubric completion, it should be interpreted as a relative efficiency measure rather than an end-to-end cost estimate.

RQ2 Summary. Access to alipay-payment-integration helps coding agents complete Alipay payment integration more effectively and efficiently. It improves mean RPR across models and products, with the largest gains occurring when models cannot reconstruct product-specific payment logic from repository context alone. It also tends to reduce output tokens per unit RPR, suggesting that structured Alipay guidance can reduce trial and error and support a more efficient integration process for developers using coding agents.

4.4 RQ3: Evaluation-Method Diagnostics↩︎

RQ3 examines what different evaluation methods reveal about Alipay payment integration capability. The methods target different verification layers: Static checks inspect source-level structure, Unit tests provide local behavioral evidence where available, Integration tests exercise backend interactions and state transitions, E2E tests inspect externally visible workflow behavior, and LLM-assisted assessment examines payment-domain semantics. Figure 4 reports the unweighted within-method RPR under the with-skill condition.

The first pattern is a gap between structural completion and executable behavior. Static RPR averages 92.79% for Basic tasks, whereas Integration and E2E average 73.16% and 78.24%, respectively. For example, Basic In-App Payment obtains 83.0% Static RPR but only 38.6% Integration RPR. These results show that agents can introduce expected SDK usage, payment entry points, fields, and state models without fully connecting them into an executable payment workflow.

The second pattern is a gap between executable paths and payment-domain requirements. In Advanced tasks, E2E RPR averages 97.03%, whereas LLM-assisted domain assessment averages 61.05%. Advanced Order QR, for example, reaches 100% E2E RPR but only 58.6% LLM-assisted RPR. This result indicates that an implementation can execute through the tested entry points while still lacking evidence for payment-safety and state-consistency requirements.

The reverse pattern also occurs. Basic Pre-Authorization obtains 96.4% under LLM-assisted assessment but only 39.3% under E2E testing. An implementation may therefore appear semantically consistent with product requirements while failing through the required application-facing workflow. Together, these patterns show that structural, executable, and domain-level evaluation expose different forms of incomplete payment integration.

Because the methods cover different rubric subsets, their scores should not be interpreted as direct comparisons of evaluator accuracy or strictness. Their value lies in failure localization: the combined signals help determine whether an integration is missing expected structure, fails during execution, or remains incomplete in payment safety and business-state consistency.

RQ3 Summary. Payment integration cannot be reliably assessed by checking only whether the expected code is present or whether a tested path executes. Static, execution-based, and LLM-assisted evaluations reveal structural completion, executable workflow behavior, and payment-domain correctness from different perspectives. Multi-method evaluation is therefore necessary not only to determine whether an integration fails, but also to identify where it fails.

5 Conclusion↩︎

Taken together, the results show that Alipay payment integration requires an agent to select an appropriate product, modify a repository across components, preserve payment state, and satisfy payment-specific safety requirements.

We developed Alipay-PIBench to measure these demands in realistic project repositories. The benchmark connects payment products with business workflows and existing codebases, and separates Basic functional completion from Advanced risk-aware hardening. Its rubric-derived framework combines deterministic checks with supplementary LLM-assisted assessment, while the paired without-skill/with-skill protocol measures the effect of structured payment guidance.

In our evaluation of 18 task instances and six models, mean RPR under the with-skill setting ranged from 68.58% to 91.37%. Relative to the corresponding without-skill condition, mean RPR increased by 10.31 percentage points on average, with gains in 101 of 108 model–product–scenario comparisons. The average improvement was larger for Basic tasks than for Advanced tasks (+11.27 versus +9.35 percentage points). The method-specific results also separated source-level completion from executable payment behavior and payment-domain requirements, supporting the use of progressive scenarios and complementary rubric-grounded signals.

Our results are grounded in the Alipay Open Platform products, task instances, and coding-agent configurations studied here. Future work can extend the benchmark with additional models and agent frameworks, broader repository coverage, and longer-horizon executable payment workflows, allowing the robustness of these findings to be examined across a wider range of settings.

References↩︎