1 Introduction↩︎

Interactive language systems—from task-oriented and open-domain dialog assistants to LLM-based tool-use and planning agents—increasingly operate over multi-turn trajectories where failures emerge over time [1]–[4]. In many deployments, failing to detect risky outcomes early can lead to user frustration, wasted computation, or unsafe behavior. This motivates early failure alerting: deciding, as a dialog or agent rollout unfolds, whether to raise an alert now or wait for more information so that interventions (e.g., escalation to a human, strategy revision, or halting costly computation) can occur in a timely manner.

Early alerting in such multi-turn trajectories poses two distinct challenges. First, supervision is typically available only as a final outcome—for example, whether a support case was resolved, a donation was made, or an agent completed the task—whereas alerts must be raised while the dialog or rollout is still unfolding. This leaves the alerter to infer which turns drove the outcome from only the outcome itself, a credit-assignment problem that is harder when many turns reflect ordinary interaction progress rather than failure evidence (Figures 9–13). Second, deployments rarely have a single fixed notion of “early enough” or “accurate enough”: acceptable accuracy targets and delay budgets vary across products, users, and safety regimes, and they can change over time. In practice, this means systems need multiple accuracy–earliness operating points, and ideally the ability to traverse these trade-offs in a predictable way, without needing to retrain a new alerting policy for each preference.

Existing approaches address parts of these challenges, but each leaves a key gap. For the first challenge, the most common approach is to train a naive failure predictor by broadcasting the terminal label to every prefix [5], [6]; because every prefix of a failed trajectory is supervised as positive—including early prefixes that contain no failure-indicating content—the predictor is biased toward early triggers and attains lower maximum accuracy (Section 5.2). Multiple instance learning (MIL) [7], [8] more directly targets the weak-supervision problem by identifying which turns drove a trajectory-level label, but operates on full trajectories and is not directly usable for online alerting. LLM-based judges [9], [10] sidestep weak supervision by scoring trajectories online from prompts alone, but exhibit systematic biases.

For the second challenge, learned early-alerting policies such as ALERT\(^*\) [11] and FIRMBOUND [12], as well as end-to-end RL approaches, can produce stopping decisions for a fixed accuracy–earliness preference, but obtain different operating points by training/fitting a separate model or policy per preference. This makes them inflexible to deployment-time changes in accuracy or earliness preferences, since adapting to a new trade-off requires retraining.

Our approach. We address both challenges with two complementary components (Figure 1). We first train an attention-based failure predictor that exploits the sparse turn-level failure evidence—rather than treating every prefix of a failed trajectory as equally failure-indicative—to produce more accurate online failure estimates. We then learn \(\alpha\)-STOP (\(\alpha\)-conditioned Sequential Triggering Optimization Policy), a single alerting policy whose accuracy–earliness operating point is selected at inference time by varying \(\alpha\in(0,1]\), without retraining. We evaluate across a diverse set of interactive NLP benchmarks spanning dialog—customer-support resolution (PCS), task-oriented dialog breakdown (BETOLD), and persuasion (P4G)—and LLM-agent trajectories—tool/API use (AppWorld) and text-environment planning (ALFWorld). To summarize, our main contributions are:

• Empirical characterization of sparse failure evidence. Across a diverse set of dialog and LLM-agent benchmarks, we show that turns most indicative of eventual failure are sparse and tend to appear later in the trajectory rather than uniformly throughout.

• Weakly supervised online failure prediction. Building on this, we develop a failure predictor that leverages this sparse turn-level evidence for more accurate online failure prediction, improving Pareto-frontier quality (hypervolume) by 1–10% over baseline.

• Preference-conditioned early alerting. We then turn the resulting failure predictions into alert decisions with a single stopping policy that can be tuned at inference time to balance earliness against accuracy, improving Pareto-frontier quality (hypervolume) by 3–42% over state-of-the-art trigger baselines (ALERT\(^*\), FIRMBOUND) while reducing training cost per operating point by 1–3 orders of magnitude.

2 Related Work↩︎

Dialog breakdown and quality monitoring. A line of work in NLP studies when interactive dialogs go wrong, including chat-oriented dialog breakdown [2], task-oriented breakdown prediction [3], conversation-quality estimation [13], [14], and frustration detection [1]. These works focus on estimating breakdown, satisfaction, or quality signals in dialogs. We instead study controllable early alerting from partial interactions under an accuracy–earliness trade-off, and evaluate across both dialogs (PCS, BETOLD, P4G) and LLM-agent trajectories.

Monitoring LLM-agent trajectories. Recent work studies intermediate supervision and monitoring in multi-step reasoning, agentic tasks, and dialog workflows: process reward models provide step-level feedback for reasoning [15], [16], agent process-refinement methods derive step-level signals for LLM agents [17], and workflow-action prediction models future actions in task-oriented dialogs [18]. LLM-as-a-judge methods [9], [10], [19] score open-ended outputs with minimal labeling, but exhibit known biases. We instead learn turn-level evidence from trajectory-level outcome labels alone and pair it with an adaptable stopping policy.

Weak supervision via multiple-instance learning. MIL learns instance contributions from bag-level labels and yields interpretable attributions [7], [8], with applications to time series [20], [21] and prompt-based selection [22]. However, MIL operates on full sequences, while early alerting requires a signal on partial prefixes. We provide a method that learns online evidence from these offline attributions and combines it with a prefix-level failure prediction, enabling early alerting from weak trajectory-level supervision.

Early classification and preference-conditioned stopping. ECTS studies early decision-making over time series [5], commonly supervising each prefix with the terminal label; in sparse-failure dialogs and agent trajectories, this can assign positive supervision to prefixes that contain little or no failure evidence. We instead use trajectory labels to infer a sparse turn-level evidence signal and combine it with the prefix prediction for online failure estimation. ECTS also provides stopping rules, including learned RL triggers [11], [23] and adaptive threshold methods [12], but these typically train separate models per operating point. Building on goal- and preference-conditioned RL [24]–[26], we train a single \(\alpha\)-conditioned stopping policy that sweeps a wide range of accuracy–earliness operating points at inference time, without retraining per preference.

3 Method↩︎

The method is organized around a simple question: at each turn, how likely is this trajectory to fail, and should we alert now? We answer the first part with an attention-based failure predictor that starts from a naive failure predictor but incorporates a sparse turn-level evidence signal learned via MIL from trajectory-level labels. This produces an online, turn-level failure probability \(\hat{p}_t\) from trajectory-level labels alone (Section 3.1). We answer the second part with \(\alpha\)-STOP, a stopping policy that decides at each turn whether to alert based on the running failure prediction \(\hat{p}_t\), with \(\alpha\) selecting the desired accuracy–earliness operating point at inference time (Sections 3.2–3.3).

3.1 Failure Predictors↩︎

Let \(H_t=(h_1,\dots,h_t)\) denote the prefix history of an interaction and \(Y\in\{0,1\}\) its terminal failure label. We represent each prefix by a fixed-dimensional embedding \(z_t=\mathrm{Enc}(H_t)\) produced by a frozen pre-trained encoder (Appendix 8.1) and write \(\tau_t\) for the normalized time. All predictors below operate on \((z_t,\tau_t)\) and are trained from trajectory-level labels alone.

3.1.1 Naive Failure Predictor↩︎

Following the standard early-classification recipe [5], we train a prefix predictor \[\label{eq:naive-belief} b_t = f_{\text{base}}(z_t,\tau_t)\in[0,1]\tag{1}\] by treating each prefix as a training example with the terminal label \(Y\), using binary cross-entropy (Appendix 8). This is the natural baseline approach in this setting, but it can be ill-suited when failure-indicating evidence is sparse and concentrated late: for a failed trajectory, the loss assigns the failure label to every prefix, including many early prefixes with no failure-indicating turns, which we find encourages premature alerts (Table 2, Figure 3).

3.1.2 Attention-Based Failure Predictor↩︎

To address this weakness, we introduce an attention-based failure predictor that uses MIL to infer sparse turn-level failure evidence from trajectory-level labels and combines this evidence with the naive prediction into a single failure score \(\hat{p}_t\). The predictor has two stages: (i) an attention-based MIL stage that produces online, turn-level evidence from trajectory-level labels, and (ii) a fusion stage that combines this evidence with the naive prefix prediction to produce the final failure probability \(\hat{p}_t\).

3.1.2.1 Stage 1: MIL-based turn-level evidence.

We use attention-based MIL to learn, from the same trajectory-level labels, which turns are most responsible for the outcome [7], [8]. For turns \(k=1,\dots,T\), an expert network produces per-turn risk scores \(s_k=\sigma(f_\theta(z_k,\tau_k))\in[0,1]\) and a gating network produces attention logits \(a_k=g_\gamma(z_k,\tau_k)\), where \(\sigma\) is the logistic sigmoid and \(f_\theta,g_\gamma\) are MLPs. The trajectory-level prediction is the attention-weighted sum \[\label{eq:mil-prediction} r_T = \sum_{k=1}^{T} \pi_k^{(T)}\, s_k, \qquad \pi_k^{(T)} = \frac{\exp(a_k)}{\sum_{j=1}^{T}\exp(a_j)},\tag{2}\] trained with binary cross-entropy against \(Y\).

Since \(\pi_k^{(T)}\) is computed from the full trajectory and is unavailable online, we train an online gating predictor \(\hat{\pi}_k\) to approximate \(\pi_k^{(T)}\) using only the prefix up to turn \(k\) (Appendix 8), and form per-turn contributions \(c_k=\hat{\pi}_k s_k\). We represent prefix-level evidence by the top-\(K\) contributions observed so far, \[\label{eq:topk-evidence} \mathbf{E}_t = \big(c_{(1)},\dots,c_{(K)}\big).\tag{3}\] A scalar summary \(\hat{E}_t\in[0,1]\) used for diagnostics (refinement analysis in Section 5.2, Figure 5) and as the standalone MIL-only signal in the ablation (Appendix 13) is described in Appendix 8.

3.1.2.2 Stage 2: Fusion with the naive prefix prediction.

The naive prediction \(b_t\) and the turn-level evidence \(\mathbf{E}_t\) are complementary in timing and accuracy: \(b_t\) rises early but achieves lower maximum accuracy, while \(\mathbf{E}_t\) rises much later but is more accurate when it does (Section 5.2). Using the scalar summary \(\hat{E}_t\) as a diagnostic, we further find that within bins of \(b_t\), the observed failure rate still varies with \(\hat{E}_t\) (Figure 5), confirming that turn-level evidence carries information about the label beyond what \(b_t\) already encodes. Motivated by this refinement, we fuse the two signals into the attention-based predictor’s final output \[\label{eq:attention-based-predictor} \hat{p}_t = \sigma\!\big(g_\psi(b_t,\mathbf{E}_t,\tau_t)\big),\tag{4}\] where \(g_\psi\) is an MLP trained with binary cross-entropy on the terminal label (Appendix 8). We provide a theoretical justification for why this fusion can strictly improve over the naive predictor under log loss in Appendix 7.3. Section 5.2 verifies that the attention-based predictor yields better accuracy–earliness trade-offs than the naive predictor.

3.2 Stopping Objective↩︎

Any predictor must be coupled with a stopping rule to produce alerts. We adopt the standard accuracy–earliness scalarization with a preference parameter \(\alpha\in(0,1]\): \[\label{eq:scalarized-objective} J_\alpha(\pi) = \mathbb{E}_\pi\!\Big[\alpha\, R_{\mathrm{acc}}(p_\tau) - (1-\alpha)\, C_{\mathrm{delay}}(\tau)\Big],\tag{5}\] where \(\tau\) is the stopping time chosen by the policy, \(R_{\mathrm{acc}}\) rewards a correct trajectory-level prediction at \(\tau\), and \(C_{\mathrm{delay}}(\tau)\) is a per-step delay cost. Larger \(\alpha\) values weight accuracy more and are intended to induce later stopping.

3.3 Trigger Policy: \(\alpha\)-STOP↩︎

The simplest stopping rule under this objective is a plug-in threshold trigger that stops when \(\hat{p}_t\) exceeds a tuned cutoff; we refer to this as the Plug-in Threshold trigger. This method already yields strong trade-offs, but is restricted to a one-dimensional stopping rule that ignores contextual signals beyond the current score. We therefore learn a preference-conditioned trigger policy that operates on a richer state representation \[\label{eq:state-representation} s_t = (\hat{p}_t,\, b_t,\, \mathbf{E}_t,\, \tau_t,\, z_t),\tag{6}\] and conditions on the user-specified preference \(\alpha\): \[\label{eq:trigger-policy} a_t \sim \pi_\phi(\cdot \mid s_t, \alpha), \qquad a_t\in\{\textsf{CONT},\textsf{STOP}\}.\tag{7}\] A single \(\alpha\)-conditioned policy can represent a continuum of accuracy–earliness operating points at inference time, allowing a practitioner to adapt to different deployment trade-offs (e.g., shifting to more conservative alerts during a high-stakes period) efficiently by simply varying \(\alpha\), without retraining. For \(\alpha\) to be an effective control knob, however, its effect must be ordered: larger \(\alpha\) values, by putting more weight on accuracy, should induce later, more conservative stopping. We validate this empirically in Section 5.3 (we also provide a theoretical justification in Appendix 7.4).

Behavior cloning then PPO. We initialize \(\pi_\phi\) by behavior cloning from a plug-in threshold controller fit on validation data and refine it with PPO [27] on \(J_\alpha\) (Appendix 8.3). The role of each component (single \(\alpha\)-conditioned policy, richer state, attention-based predictor input, and BC initialization) is examined in the ablations in Section 5.3.

4 Experiments↩︎

4.1 Datasets↩︎

We evaluate early failure alerting in multi-turn interaction trajectories, where the system raises a one-sided alert by triggering only on the positive (failure) class. Table 10 summarizes dataset statistics and splits. Detailed dataset descriptions are provided in Appendix 9. Sample trajectories are provided in Figure 9-13.

Dialog datasets. We evaluate early failure alerting on three dialog benchmarks. PCS consists of long, open-ended customer-support interactions where failures primarily reflect non-resolution: incomplete troubleshooting, escalation without follow-through, verification delays, miscommunication, and user disengagement/premature termination. BETOLD [3] contains task-oriented appointment-booking dialogs with explicit breakdown outcomes. Failures are dominated by intent/slot misalignment and dialog-management loops (repetition, unresolved ambiguities/negations, infeasible options) that prevent task completion. Persuasion for Good (P4G) [28] contains donation-persuasion chats where a persuader tries to get the persuadee to donate to a charity. Failures often stem from insufficient trust in the charity, overly forceful pressure, or failing to secure a clear commitment.

Agentic reasoning and planning datasets. AppWorld [29] and ALFWorld [30] evaluate early failure alerting in LLM-agent language trajectories. AppWorld tasks are specified through natural-language goals and solved through reasoning traces, API documentation, tool calls, and textual environment feedback. Failures are dominated by tool-use breakdowns such as authentication/permission issues, incorrect API calls/parameters, missing dependencies, and poor recovery after errors. ALFWorld is a text-based embodied environment in which agents condition on language goals, textual observations, and natural-language-style actions. Failures largely arise from planning/execution mistakes including object misidentification, action failures without recovery, inefficient exploration, and premature termination before satisfying goal conditions. Since neither benchmark provides offline rollouts, we generate rollouts under standard task setups and derive trajectory-level success/failure labels from task-completion signals (see Appendix 9 for details).³

4.2 Stopping Decision Baselines↩︎

We compare against baselines spanning (i) LLM judges for language-centric online scoring, (ii) end-to-end RL triggering, and (iii) SOTA learned-trigger methods from the ECTS literature, ALERT\(^*\) [11] and FIRMBOUND [12]. We additionally include an internal sanity check, Plug-in Threshold—a threshold-based rule applied directly to our attention-based predictor’s output \(\hat{p}_t\)—to isolate the contribution of the learned trigger policy from that of the predictor. For all trigger comparisons, we follow the standard separable protocol [5]: we fix the underlying prefix predictor outputs and compare only the stopping rule on top, isolating the decision component. Full details are in Appendix 11.

Trigger baselines. We compare \(\alpha\)-STOP to ALERT\(^*\) and FIRMBOUND, SOTA learned-trigger methods for early classification: ALERT\(^*\) learns a non-myopic stopping policy via offline RL [11], while FIRMBOUND extends SPRT to finite horizons by learning time-dependent (adaptive) decision thresholds via backward induction [12]. Both learned methods fix the accuracy–earliness trade-off at training time: ALERT\(^*\) optimizes a single penalty-weighted objective during RL training, and FIRMBOUND’s backward-induction solver requires re-fitting its time-dependent thresholds (and accompanying GP-regression value estimates) for each new penalty. Consequently, sweeping operating points with these baselines requires re-training/re-fitting rather than a post-hoc adjustment, motivating the comparison with \(\alpha\)-STOP’s inference-time control (see Table 4). As an internal sanity check, we additionally apply a Plug-in Threshold rule on top of our attention-based predictor: a parameter-free myopic rule that stops as soon as the predicted failure probability exceeds a tuned threshold. This isolates how much of \(\alpha\)-STOP’s performance comes from the learned trigger versus the underlying predictor itself.

End-to-end RL. We include this baseline to test the value of our modular failure-predictor + trigger design. It maps prefix features directly to action under the same accuracy–earliness objective, jointly learning prediction and control. Because supervision is only available at the trajectory level, end-to-end RL must perform both credit assignment over turns and trade-off learning from a single sparse, delayed reward, which is well-known to be high-variance and sample-inefficient. Our modular design instead anchors policy learning to a separately trained failure predictor.

LLM judge. We include an LLM-judge baseline that prompts large models to predict whether a prefix will eventually fail. We evaluate GPT-5.2 [31], Gemini 3 Pro [32], and Claude Opus 4.5 [33], reporting the best per dataset in the main text (all prompts/results in Appendix 11; full results in Table 7). While strong on language trajectories, LLM judging can be biased [10] and does not provide a principled mechanism for adapting to different accuracy–earliness preferences.

4.3 Evaluation metrics↩︎

We evaluate methods in the accuracy–earliness plane because a deployed alerting system may need different operating points, from early conservative alerts to later high-confidence alerts. Each method produces a set of operating points by sweeping its control parameter (e.g., threshold, \(\alpha\), or penalty). We define earliness as the normalized trigger time \(e \in [0,1]\) (lower is earlier; [34]). We report three complementary summary metrics that capture, respectively, peak detector quality and multi-objective frontier quality (see Appendix 12; [5], [35], [36]):

Maximum accuracy (Max Acc). Max Acc is the best accuracy achieved over the sweep, ignoring earliness. This metric provides a sense of the upper bound on detector quality when delay is not penalized.

Hypervolume (HV). HV is a standard multi-objective metric [35] that measures the area of the accuracy–earliness region dominated by a method’s operating points, relative to a fixed worst-case reference. Higher HV indicates operating points that are both more favorable (high accuracy, low delay) and that cover a wider range of trade-offs.

Inverted Generational Distance Plus (IGD+). IGD+ is a standard multi-objective metric [36] that measures the average distance from a method’s operating points to a reference Pareto front built from the union of non-dominated points across all compared methods. Lower IGD+ indicates closer approximation to this best observed front.

Because HV and IGD+ depend on the number of sampled operating points, we compare methods by subsampling their nondominated set to 11 points using a commonly used heuristic, HSSP (a greedy hypervolume-maximizing subset; [37]), matching the maximum available number of operating points for ALERT and End-to-End RL.

4pt

5 Results and Discussion↩︎

5.1 Failure evidence concentrates in a few turns↩︎

We first examine how failure evidence appears across turns in the dialogs and agentic interactions we study: whether it is broadly distributed across the interaction, or concentrated in a small number of turns. We find that failed trajectories are not uniformly informative. Many turns reflect ordinary interaction progress—a greeting, a slot-filling exchange, an API lookup, a navigation step, or an exploratory plan—and would not by themselves distinguish a future failure from a future success. The failure evidence is concentrated in a much smaller set of turns, as the examples in Figure 9–13 illustrate qualitatively.

To characterize this pattern across the five datasets, we use three strong LLM judges (GPT-5.2, Gemini 3 Pro, Claude Opus 4.5) to score each turn on a 1–5 relevance scale, where 1 is least relevant and 5 is most relevant to the final failure outcome (Appendix 10). Each judge observes the full trajectory before assigning retrospective turn-level scores. We use these ratings only to describe the empirical structure of failure-relevant evidence (its sparsity and timing); they are not used as supervision for our models.

Table 1 supports two related observations. First, failure evidence is sparse: high-rated turns account for only 4.7–11.3% of turns across datasets, while the remaining mass falls predominantly on low ratings (low-rated turns, mean rating \(\leq 2\), account for 65.5–75.7% of turns; Appendix 10, Table 12). Second, high-rated turns tend to appear after substantial context has accumulated: the first high-rated turn occurs, on average, after 59.0–83.6% of the trajectory has elapsed. The pattern is consistent across judges: pairwise high-rated agreement (Fleiss’ \(\kappa\)) is moderate to substantial (Appendix 10, Table 15), and the qualitative picture is unchanged under alternative aggregation: majority-vote thresholding (Table 13) and leave-one-judge-out aggregation (Table 14).

We next ask how this structure affects online failure prediction, comparing the naive predictor that assigns the terminal trajectory label to every prefix against our proposed attention-based failure predictor (Section 3.1.2), which fuses the naive prediction with a sparse turn-level evidence signal.

5.2 The attention-based failure predictor yields superior accuracy–earliness trade-offs↩︎

Given the sparse, often-late evidence structure documented in Section 5.1, we ask which predictor best estimates the probability of eventual failure from an unfolding trajectory. To answer this cleanly, we evaluate each predictor under the same simple decision rule: Plug-in Threshold, which stops as soon as the predicted failure probability exceeds a tuned threshold (and otherwise predicts no failure at the horizon). A simple threshold rule on the predicted probability lets predictor quality be read off directly from the accuracy–earliness trade-off, without entangling it with the behavior of a more complex stopping policy. Richer stopping policies are studied in the next section.

Table 2 and Figure 3 show a consistent pattern across datasets. The naive failure predictor exhibits early triggers but attains lower maximum accuracy, consistent with the training setup described in Section 3.1: broadcasting the terminal label to every prefix encourages it to assign high failure probability before strong evidence appears. The attention-based failure predictor consistently improves Pareto quality (higher HV / lower IGD\(^+\)) and achieves higher maximum accuracy by combining the naive prediction \(b_t\) with the learned turn-level evidence \(\mathbf{E}_t\). In practical terms, a better Pareto frontier means that whatever balance of accuracy and earliness a deployment requires, the attention-based predictor offers a stronger operating point than the naive baseline.

Why the attention-based predictor helps. The attention-based predictor improves over the naive predictor by adding a sparse turn-level evidence signal (Figure 4) learned from trajectory labels. This signal is complementary to \(b_t\): within bins of the naive prediction, the observed failure rate still increases with increasing evidence \(\hat{E}_t\) (Figure 5), indicating that the evidence signal stratifies failure risk among prefixes that look similar under \(b_t\). The MIL-only ablation (Appendix 13) shows why fusion is needed: thresholding \(\hat{E}_t\) alone is accurate but yields worse frontier quality than the full attention-based predictor (Figure 3; Table 6), so the gain comes from combining the early but noisy naive prediction with the more conservative evidence signal.

5.3 Preference-controlled RL trigger (\(\alpha\)-STOP) improves the Pareto frontier↩︎

The previous section established the attention-based failure predictor as a strong method for estimating eventual failure. The Plug-in Threshold rule is useful for diagnosing failure-predictor quality, but it reduces stopping to a one-dimensional cutoff on the current score. We therefore learn \(\alpha\)-STOP, a preference-conditioned RL trigger (Section 3.3) that learns a richer stopping policy and uses \(\alpha\) to sweep the Pareto frontier at inference time.

Improved Pareto frontier quality. Table 3 and Figure 2 compare \(\alpha\)-STOP to the baselines described in Section 4.2. All trigger baselines are trained on the attention-based failure predictor for fair comparison. \(\alpha\)-STOP achieves stronger Pareto-front quality (higher HV / lower IGD\(^+\)) and higher maximum accuracy on PCS, BETOLD, P4G, and AppWorld, while remaining competitive on ALFWorld where performance is near-saturated. This means that across earliness budgets, \(\alpha\)-STOP provides higher-accuracy alerting options than competing trigger policies.

Efficient Pareto frontier coverage. \(\alpha\)-STOP produces the Pareto frontier from a single trained policy by sweeping \(\alpha\) at inference time. In contrast, ALERT and end-to-end RL require separate training runs for different preferences, and FIRMBOUND requires separate runs with distinct GP-regression fits (used by its backward-induction solver) for different penalty values. Table 4 quantifies this gap: across the reported runs, \(\alpha\)-STOP achieves orders-of-magnitude lower GPU-hours per operating point while enabling substantially denser frontier coverage. In deployment terms, this means an operator can shift to a new accuracy–earliness target by varying \(\alpha\) at inference, without launching a new training job.

Monotone control. A key goal of \(\alpha\)-STOP is predictable control: increasing \(\alpha\) should induce later stopping (higher earliness), consistent with the structural analysis in Appendix 7.4. Figure 6 confirms this empirically, and Table 5 shows consistently high rank correlations between \(\alpha\) and earliness across all datasets, indicating that \(\alpha\) provides predictable control.

5.3.1 Ablations↩︎

In this section, we ablate the key design choices in \(\alpha\)-STOP to understand what drives its performance.

\(\alpha\)-conditioning vs. single-\(\alpha\) policies. A single \(\alpha\)-conditioned policy matches or improves independently trained fixed-\(\alpha\) policies in nearly all reported Max Acc / HV / IGD\(^+\) comparisons while being substantially more efficient (Table 8), and it yields more reliable monotone control in \(\alpha\) (Table 5).

State richness. Conditioning on a richer state representation that includes \((b_t,\mathbf{E}_t,z_t)\), rather than only \(\hat{p}_t\), consistently leads to better performance (Table 8), suggesting the trigger benefits from additional context beyond \(\hat{p}_t\).

Prediction source. Replacing the attention-based predictor with the naive failure predictor \(b_t\) reduces performance across nearly all trigger choices (Tables 3, 9), indicating that the gain from the attention-based predictor is largely orthogonal to the choice of trigger. The resulting \(\alpha\)-STOP variant remains the strongest trigger on top of \(b_t\) (Table 9), indicating that \(\alpha\)-STOP is still effective even with a weaker prediction source.

BC\(\rightarrow\)PPO training recipe. BC initialization gives \(\alpha\)-STOP a strong starting policy before PPO fine-tuning. BC trains \(\pi_\phi\) to mimic a threshold rule fit on validation data, so the BC-only checkpoint approximates the Plug-in Threshold baseline on the same predictor, providing a performance floor at initialization. PPO then refines this initialization and consistently improves HV and IGD\(^+\) (Table 3), showing that the learned policy adds value beyond imitation. This initialization is also important in practice: learned-trigger baselines such as ALERT\(^*\) and FIRMBOUND do not provide a mechanism for threshold-imitation initialization, and often fail to outperform the threshold baseline (Table 3). Without BC, training PPO from scratch degrades substantially (Table 8), particularly on longer trajectories, indicating that the BC initialization is crucial for performance.

Takeaway. Taken together, these comparisons separate the sources of improvement. In Section 5.2, we hold the stopping rule fixed and vary the predictor, showing the benefit of moving from the naive to the attention-based failure predictor. In Section 5.3, we hold the predictor fixed and vary the stopping rule, showing the additional gain from \(\alpha\)-STOP over Plug-in Threshold, ALERT\(^*\), FIRMBOUND, and End-to-End RL (Table 3). The prediction-source ablation then crosses these choices and probes error propagation across stages: replacing the attention-based predictor with the weaker naive predictor reduces most trigger results, but \(\alpha\)-STOP remains the strongest trigger on that weaker signal (Table 9). Thus, the full improvement comes from two separable effects: the attention-based predictor provides a better estimate of eventual failure, and \(\alpha\)-STOP learns when to act on that estimate for different accuracy–earliness preferences. This separation is useful operationally: the predictor improves the risk signal available at each prefix, while \(\alpha\)-STOP controls how aggressively the system turns that signal into an alert. Notably, the End-to-End RL baseline that jointly learns prediction and triggering underperforms \(\alpha\)-STOP, suggesting that in this weakly supervised setting, decoupling failure prediction from trigger optimization is more effective than learning both from a single sparse reward.

6 Conclusion↩︎

We presented a practical weakly supervised framework for early failure alerting in dialogs and LLM-agent trajectories. Leveraging the sparse nature of failure-indicating evidence, we developed an attention-based failure predictor and a single trigger policy, \(\alpha\)-STOP, with deployment-time control over the accuracy–earliness trade-off. Across benchmarks, our method improves frontier quality against SOTA baselines while supporting different accuracy–earliness preferences at inference time, without requiring retuning.

Limitations↩︎

Datasets and reproducibility. Our evaluation spans four public benchmark families (BETOLD, P4G, AppWorld, ALFWorld) and one proprietary customer-support dataset (PCS). For the public benchmarks we will release code, prompts, splits, generated agent rollouts, and implementation details. AppWorld and ALFWorld trajectories are produced with specific ReAct-style agents and model backbones, so the reported failure modes should be read as representative of these rollout settings. However, our method is backbone-agnostic—it operates on trajectory-level labels and a fixed prefix encoder—so we expect it to naturally extend to other agent implementations for AppWorld and ALFWorld.

Coverage of the sparse-evidence claim. The five benchmarks above are diverse but not exhaustive of multi-turn NLP tasks, so the sparse, often-late structure of failure-indicating turns may not characterize every setting. However, the evaluation spans both dialog and LLM-agent trajectories—customer support, task-oriented dialog, persuasion, tool/API use, and text-environment planning—and we expect this structure to be common in interactive NLP, where most turns carry routine content and only a few contain decisive cues for the eventual failure outcome.

LLM-judge turn relevance. Our turn-relevance analysis relies on retrospective LLM judgments: judges see the full failed trajectory and may be sensitive to model-specific biases or prompt wording, so their ratings should not be interpreted as causal annotations of failure. However, these ratings are used only as an auxiliary diagnostic of sparsity and timing, not as training supervision, and we aggregate across three strong judges with robustness and inter-judge agreement checks in Appendix 10.

Adaptation and compute. Our predictors depend on the quality and context window of the fixed pretrained encoder, and new deployment domains may require refitting the lightweight prediction and trigger heads. However, this adaptation burden is modest in our setup: the failure predictors and \(\alpha\)-STOP are small MLPs over frozen prefix embeddings, per-turn inference is cheap, and changing the desired accuracy–earliness preference only requires varying \(\alpha\) at inference time rather than retraining a separate trigger.

Ethical Considerations↩︎

This paper develops methods for early failure alerting in interactive AI systems. A primary benefit is improved safety and reliability by enabling earlier escalation to human oversight or safe fallback behaviors. Potential risks include misconfigured trade-off settings (over/under-alerting) and biased failure labels leading to uneven alerting across users or domains. We recommend domain-specific calibration and monitoring, including audits of labels/alerts and human oversight when alerts affect consequential decisions.

7 Theoretical Motivation↩︎

This section provides a more formal treatment of the assumptions and results underlying our methods and experiments:

• Section 7.1 (belief-MDP reduction and threshold structure) supports the use of plug-in threshold rules on learned failure predictors as the simplest stopping rule under the accuracy–earliness objective in Section 3.2.

• Section 7.2 (scalar evidence score as a proxy for belief) supports using \(\hat{E}_t\) as an interpretable diagnostic and as the standalone MIL-evidence-only ablation in Appendix 13.

• Section 7.3 (Bayes-optimal estimator under log loss) supports fusing the naive prefix prediction with the learned turn-level evidence inside the attention-based failure predictor (Section 3.1.2).

• Section 7.4 (monotone ordering of the optimal stopping time) supports treating \(\alpha\) as an ordered control knob, which the trigger policy in Section 3.3 exploits via \(\alpha\)-conditioning and ordered behavior-cloning initialization.

Throughout, we cite canonical references for full proofs and provide brief intuition.

7.1 Belief-MDP Reduction and Threshold Structure↩︎

This subsection records standard results used to motivate plug-in threshold rules on the learned predictors of Section 3.1.

7.1.0.1 Belief-MDP reduction (standard).

In a POMDP, the posterior belief state is a sufficient statistic: conditioned on \(p_t\), the distribution of future observations (and thus future beliefs and rewards under any action) is independent of the full history. This yields an equivalent fully observed MDP on belief space; see [38].

7.1.1 Threshold Structure for Finite-Horizon Stopping in Belief Space↩︎

Consider a finite-horizon stopping problem on belief space with actions \(\{\textsf{STOP},\textsf{CONT}\}\) and per-step continuation penalty \(-\lambda\). Let the stop payoff at time \(t\) be \(U_t(p)\), and define the value recursion \[\begin{align} V_{T,\lambda}(p) &= U_T(p),\\ V_{t,\lambda}(p) &= \max\Big\{U_t(p),\;-\lambda + \mathbb{E}\big[V_{t+1,\lambda}(p')\mid p\big]\Big\}, \nonumber\\ &\qquad p'\sim Q_t(\cdot\mid p), \end{align}\] where \(Q_t(\cdot\mid p)\) is the belief transition kernel under continuation.

Define the continuation value, stopping advantage, and stopping region \[\begin{align} C_{t,\lambda}(p) &\triangleq \mathbb{E}\big[V_{t+1,\lambda}(p')\mid p\big], \\ \Delta_{t,\lambda}(p) &\triangleq U_t(p)+\lambda - C_{t,\lambda}(p), \\ S_t(\lambda) &\triangleq \{p:\Delta_{t,\lambda}(p)\ge 0\}. \end{align}\]

See [39] for monotone-structure results in POMDPs and their implication for threshold policies in belief space. Intuitively, under monotone dynamics, higher belief \(p\) stochastically leads to higher future beliefs, so \(C_{t,\lambda}(p)\) is nondecreasing in \(p\). With a nondecreasing stop payoff, the advantage of stopping crosses zero once, yielding an upper-set stopping region and hence a single belief threshold.

This justifies the Plug-in Threshold rule used in the main text: under these conditions, the Bayes-optimal stopping policy is itself a threshold on the belief \(p_t\), so thresholding a good belief proxy (the learned failure probability \(\hat{p}_t\)) is the simplest stopping rule consistent with the belief-space analysis and serves as a natural reference point for any more expressive trigger policy.

7.2 Scalar Evidence Score as a Proxy for Belief↩︎

Evidence as a proxy for belief. Recall from Section 3.2 that the posterior belief \[p_t=\Pr(Y=1\mid H_t),\] is a sufficient statistic for optimal stopping, and that under the conditions of Theorem 1 the optimal policy is a threshold rule in \(p_t\).

We now formalize when the learned evidence statistic \(\hat{E}_t\) can be used in place of \(p_t\).

Assumption 2 states that \(\hat{E}_t\) preserves the ordering of posterior beliefs. In simple terms, this means that as evidence increases, the probability of the outcome \(Y=1\) increases. In practice, \(p_t\) is unobserved; we empirically validate this assumption by verifying that the empirical outcome rate \(\Pr(Y=1\mid \hat{E}_t\in\text{bin})\) increases monotonically across bins of \(\hat{E}_t\) (Figure 7).

Corollary 1 shows that, provided the evidence statistic is order-preserving in the belief, threshold-optimal stopping decisions in belief space transfer directly to threshold decisions in \(\hat{E}_t\).

7.3 Bayes-optimal estimator under log loss (standard result)↩︎

Let \(\mathcal{F}(X)\) denote measurable predictors mapping \(X\) to \((0,1)\), and define the Bayes risk under log loss \[\label{eq:bayes-risk-app} \mathcal{R}^\star(X) \;=\; \inf_{f\in\mathcal{F}(X)} \mathbb{E}[\ell_{\log}(f(X),Y)].\tag{8}\]

Proof. Under log loss, the Bayes act is the conditional posterior \(p^\star(X)=\Pr(Y=1\mid X)\) (log score is strictly proper; [40]), yielding \(\mathcal{R}^\star(X)=H(Y\mid X)\). The identity \(H(Y\mid X)-H(Y\mid X,Z)=I(Y;Z\mid X)\ge 0\) gives the result.

7.4 Monotone ordering of the optimal stopping time↩︎

We provide a complete proof of Theorem 3.

7.4.0.1 Assumptions.

We assume (i) the conditions of Theorem 1 (so that the optimal stopping region is an upper interval in belief) and (ii) that delay enters additively via a per-step continuation penalty \(-\lambda\).

Step 1: The value function is nonincreasing in \(\lambda\). Recall the finite-horizon Bellman recursion defining \(V_{t,\lambda}\) in Section 3.2. We first show that the value is pointwise nonincreasing in the delay weight \(\lambda\).

Step 2: Stopping regions expand with \(\lambda\). Recall the continuation value \(C_{t,\lambda}(p)\), stopping advantage \(\Delta_{t,\lambda}(p)\), and stopping region \(S_t(\lambda)\) from Section 3.2. We show \(S_t(\lambda)\) expands with \(\lambda\).

Step 3: Monotone thresholds in \(\lambda\) and \(\alpha\). By Theorem 1, \(S_t(\lambda)=[\theta_t(\lambda),1]\). By Lemma 2, \(S_t(\lambda_1)\subseteq S_t(\lambda_2)\) whenever \(\lambda_2\ge\lambda_1\), which implies \[\theta_t(\lambda_2)\;\le\;\theta_t(\lambda_1)\quad\text{for all }t.\] Since \(\lambda(\alpha)\) is decreasing in \(\alpha\), this is equivalent to: if \(\alpha_2\ge\alpha_1\), then \(\theta_t(\lambda(\alpha_2))\ge\theta_t(\lambda(\alpha_1))\) for all \(t\).

Step 4: Pathwise monotonicity of the stopping time. Fix a realized belief path \((p_t)_{t=0}^T\) under the continue-until-stop evolution, and define the induced (forced-horizon) stopping time \[\tau(\alpha)\triangleq \min\{t\le T:\; p_t\ge \theta_t(\lambda(\alpha))\}.\] If \(\alpha_2\ge\alpha_1\), then Step 3 gives \(\theta_t(\lambda(\alpha_2))\ge\theta_t(\lambda(\alpha_1))\) for all \(t\), hence \[\{p_t\ge\theta_t(\lambda(\alpha_2))\}\subseteq \{p_t\ge\theta_t(\lambda(\alpha_1))\}\quad\text{for all }t,\] which implies \(\tau(\alpha_2)\ge\tau(\alpha_1)\) for every sample path (and therefore almost surely). Taking expectations yields that \(\mathbb{E}[\tau(\alpha)]\) is nondecreasing in \(\alpha\). This completes the proof of Theorem 3.

7.4.1 Empirical support for the monotone-control assumptions.↩︎

The monotone-control result (Theorem 3) is stated for the (unobserved) true posterior belief \(p_t=\Pr(Y=1\mid H_t)\). We therefore validate, to the extent possible with learned prediction proxies, that the key monotone-structure conditions are consistent with our attention-based failure predictor \(\hat{p}_t\) and experimental setup.

7.4.1.1 (1) Monotone stop payoff (holds exactly for our one-sided reward).

In the early alerting setting, STOP corresponds to triggering an alert, i.e., predicting \(Y=1\). We use reward \(0\) for a correct prediction and \(-1\) for a misclassification. Under belief \(p=\Pr(Y=1\mid H_t)\), the expected stop payoff is \[\begin{align} U_t(p) &~=~\mathbb{E}[R_{\mathrm{acc}}\mid p,\textsf{STOP}] \nonumber\\ &~=~0\cdot \Pr(Y{=}1\mid p) + (-1)\cdot \Pr(Y{=}0\mid p) \nonumber\\ &~=~-(1-p) ~=~ p-1, \end{align}\] which is linear and (strictly) nondecreasing in \(p\). Thus higher failure probability always makes stopping weakly more attractive, satisfying the monotone stop-payoff condition.

7.4.1.2 (2) MLR-preserving belief dynamics (standard structural condition).

The MLR-preservation condition is a property of the underlying belief update induced by the environment’s observation dynamics. Since interactive benchmarks do not expose the latent generative model, we adopt this as a standard monotone-structure assumption, consistent with classical threshold-optimality results in belief-space stopping (Appendix 7.1).

7.4.1.3 (3) Stochastic monotonicity (validated for \(\hat{p}_t\) via one-step lookahead).

A practical implication of stochastic monotonicity is that larger current belief should imply larger expected next-step belief under continuation. We validate an empirical analogue for our learned proxy by estimating the one-step lookahead curve \[m(p) ~:=~ \mathbb{E}[\hat{p}_{t+1}\mid \hat{p}_t=p],\] using binning/smoothing over prefixes. Figure 8 plots \(m(p)\) against \(p\) across datasets; the resulting curves are monotone with high rank correlations, indicating that \(\hat{p}_t\) exhibits the temporal consistency implied by stochastic monotonicity.

7.4.1.4 (4) Additive per-step penalty \(-\lambda\) (by design).

Our objective includes a constant per-step delay cost (Eq. 5 ), so continuation incurs an additive penalty \(-\lambda\), matching the theorem’s requirement.

7.4.1.5 Summary.

For one-sided alerting, the monotone stop-payoff condition holds exactly (\(U_t(p)=p-1\) is nondecreasing). We adopt the standard MLR-preservation assumption on the underlying belief dynamics, and empirically validate a stochastic-monotonicity analogue for the learned proxy \(\hat{p}_t\) via one-step lookahead. Together with the additive per-step delay penalty, these checks support using \(\alpha\) as an ordered control knob and motivate our monotone-control analysis and empirical monotonicity results.

8 Training Details↩︎

8.1 Input Representations↩︎

We represent each prefix history \(H_t\) by a fixed-dimensional embedding \(z_t=\mathrm{Enc}(H_t)\) produced by a frozen, pre-trained encoder. For all datasets, we use 1024-dimensional embeddings.

For all datasets (PCS, AppWorld, BETOLD, P4G, and ALFWorld), we use Qwen3-Embedding-0.6B [41], which supports up to 32k tokens of context and produces embeddings up to 1024 dimensions [41]. At each turn \(t\), we encode the prefix \(H_t\) by concatenating prior turns (truncated to the most recent 20 turns and to the model context limit). We follow the model’s instruction-style encoding interface [41] and format inputs as instruction-query pairs: the instruction contains a domain-specific task description (e.g., identifying negative turns in task-oriented interactions) along with the preceding context, while the query contains only the current turn to be embedded. We extract embeddings using last-token pooling and apply \(\ell_2\) normalization.

8.2 Belief Estimation↩︎

All failure predictors are trained with Adam and a cosine annealing learning-rate schedule. We select the best checkpoint by validation loss and perform a log-space grid search over learning rates in \([10^{-4},10^{-2}]\). We define normalized time as \(\tau_t \triangleq t/T \in (0,1]\).

8.2.1 Naive Failure Predictor↩︎

The naive failure predictor \(f_{\text{base}}\) maps \((z_t,\tau_t)\) to a scalar \(b_t\in[0,1]\) using a 2-layer MLP (1024 hidden units, ReLU). We train for 100 epochs with batch size 256 using binary cross-entropy.

8.2.2 Attention-Based Failure Predictor↩︎

8.2.2.1 MIL stage.

The offline sparse attribution model consists of two 2-layer MLPs (1024 hidden units, ReLU) operating on \((z_t,\tau_t)\): (i) an expert network producing per-turn scores \(s_t\), and (ii) a gating network producing attention logits \(a_t\). The sequence-level prediction is the attention-weighted sum of sigmoid-activated scores (cf.Section 3.1.2). We train for 100 epochs with batch size 256 using binary cross-entropy.

Because the offline attention \(\pi_k^{(T)}\) is computed from the full trajectory and is unavailable at inference time, we train an online gating predictor \(\hat{\pi}_k\) that matches it from prefix features alone, following Section 3.1.2. The online gating predictor is a 2-layer MLP (1024 hidden units) that maps \((z_t,\tau_t)\) to \(\hat{\pi}_t\in[0,1]\), trained to match \(\pi_t^{(T)}\) with weighted binary cross-entropy; turns with \(\pi_t^{(T)}>0.01\) are weighted by 0.9 and the remainder by 0.1 to mitigate the imbalance between non-zero and zero attention weights. The resulting per-turn contributions \(c_k=\hat{\pi}_k s_k\) are aggregated into the top-\(K\) evidence vector \(\mathbf{E}_t\) used by the fusion stage.

For simplicity, we set the evidence budget as \(K{=}10\) for all datasets since the top-10 contributions capture most of the evidence mass (>90% coverage; see Figure 4).

8.2.2.2 Fusion stage.

The fusion MLP \(g_\psi\) inside the attention-based failure predictor operates on derived features rather than embeddings: it takes \((b_t,\mathbf{E}_t,\tau_t)\) and outputs \(\hat{p}_t\in[0,1]\) via a 2-layer MLP (512 hidden units, ReLU) followed by a sigmoid. We train for 100 epochs with batch size 256 using binary cross-entropy.

8.2.2.3 Scalar evidence score (diagnostic / MIL-evidence-only ablation).

For the diagnostic refinement analysis (Section 5.2, Figure 5) and the standalone MIL-only ablation (Appendix 13), we additionally train a scalar evidence predictor \(\hat{E}_t\in[0,1]\). It is a 2-layer MLP (1024 hidden units) with sigmoid output that maps \((\mathbf{E}_t,\tau_t)\) to a scalar trained to predict the offline cumulative-evidence target \(E_t^{\mathrm{off}}=\sum_{k=1}^{t}\pi_k^{(T)} s_k\in[0,1]\), using the top-\(K\) contributions with \(K{=}10\) (zero-padded when \(t<K\)). This predictor is not part of the runtime attention-based failure predictor and is used only as an interpretable scalar summary of the evidence.

8.3 Trigger Learning (\(\alpha\)-STOP)↩︎

We train an \(\alpha\)-conditioned trigger policy to decide when to stop and predict using a two-stage procedure: behavior cloning (BC) initialization followed by PPO fine-tuning. Our PPO implementation is based on CleanRL [42].

8.3.1 Environment↩︎

We formulate early alerting/classification as a finite-horizon decision problem. At each turn \(t\), the agent chooses between continuing (CONT) or stopping to predict (STOP).

8.3.1.1 Action space.

Actions are CONT (wait) or STOP (trigger a positive-class alert and terminate). If the agent never triggers, the episode terminates at \(T\) with an implicit negative prediction.

8.3.1.2 Reward Structure.

The reward trades off prediction delay against misclassification: \[\label{eq:reward} r_t = -\!\left(\alpha \cdot c_{\text{misclf}} \cdot \mathbb{I}[\text{t is terminal}] + (1-\alpha)\cdot \tfrac{1}{T}\right)\tag{11}\] where \(r_t\) is the reward at turn \(t\), \(\alpha \in [0,1]\) is a trade-off parameter, \(T\) is the episode horizon, \(c_{\text{misclf}}\) is the misclassification cost (1 for false positives/negatives, 0 otherwise), and \(\mathbb{I}[\cdot]\) indicates whether the current step is terminal (i.e., when the agent makes a prediction, or when the episode ends at the final turn). The second term applies a constant per-step delay penalty of \((1-\alpha)/T\), so the cumulative delay cost after \(t\) steps is \((1-\alpha)\cdot t/T\). On termination, the agent additionally receives the misclassification penalty \(\alpha \cdot c_{\text{misclf}}\). Setting \(\alpha = 1\) prioritizes accuracy (no delay penalty), while \(\alpha = 0\) prioritizes earliness (no misclassification penalty). Note that this equation matches our objective in Equation 5 and is the per-step decomposition of Equation 5 .

8.3.1.3 State representation.

At each turn, the policy observes \[s_t = (\alpha,\; \hat{p}_t,\; b_t,\; \mathbf{E}_t,\; \tau_t,\; z_t),\] where \(z_t=\mathrm{Enc}(H_t)\) is the prefix embedding, \(\tau_t=t/T\), \(\mathbf{E}_t\) denotes the top-\(K\) evidence contributions (with \(K=10\)), and \((b_t,\hat{p}_t)\) are the naive and attention-based failure predictions.

8.3.2 Policy architecture↩︎

We use a feedforward actor–critic with separate heads. Both actor and critic use three fully-connected layers (256 \(\rightarrow\) 512 \(\rightarrow\) 128) with ReLU activations and orthogonal initialization [43]. The actor outputs a categorical distribution over valid actions (with action masking where applicable), and the critic estimates \(V(s_t)\).

8.3.3 Behavior cloning (BC)↩︎

We initialize the policy by behavior cloning from a plug-in threshold controller applied to the attention-based failure prediction \(\hat{p}_t\). For each \(\alpha\in\{0.0,0.1,\ldots,1.0\}\), we select a threshold on the validation set that maximizes the shaped objective above, then generate expert rollouts for all \((\text{episode},\alpha)\) pairs.

We train actor and critic jointly with \[\mathcal{L} = \mathcal{L}_{\text{BC}} + \lambda_v \mathcal{L}_{\text{value}}, \qquad \lambda_v = 1.0,\] where \[\mathcal{L}_{\text{BC}} = -\mathbb{E}\big[w(a)\log \pi_\theta(a\mid s)\big]\] uses \(w(a)=0.9\) for terminal (prediction) actions and \(w(a)=0.1\) for CONT to mitigate class imbalance, and \[\begin{align} \mathcal{L}_{\text{value}} &= \tfrac{1}{2}\mathbb{E}\big[(V_\phi(s)-G_t)^2\big], \\ G_t &= \sum_{k=0}^{T-t}\gamma^k r_{t+k}, \quad \gamma=1. \end{align}\] We train for 100 epochs with batch size 256 and learning rate \(5\times 10^{-4}\) (linear annealing), selecting the best checkpoint by validation loss.

8.3.4 PPO fine-tuning↩︎

We fine-tune the BC-initialized policy using PPO [27], following the CleanRL implementation [42]. We train for 10M environment steps using 128 parallel environments and rollout length 64. Unless stated otherwise, we use CleanRL PPO defaults: learning rate \(2.5\times 10^{-4}\), clip coefficient 0.2, value coefficient 0.5, entropy coefficient 0.01, GAE \(\lambda=0.95\), 4 PPO epochs per update with 4 minibatches, advantage normalization, and gradient norm clipping. We use \(\gamma=1\) and a cosine annealing warm-restarts scheduler (min LR ratio 0.1).

To encourage generalization across trade-offs, each environment samples \(\alpha\) independently: 30% use \(\alpha=1.0\), and 70% sample from \(\mathrm{Beta}(5,2)\). We select the best checkpoint by best average validation performance across all \(\alpha \in \{0,0.1,\ldots,1.0\}\) values. At inference, we sample 100 values of \(\alpha\in[0,1]\) to densely trace a Pareto frontier.

8.3.5 Ablations↩︎

8.3.5.1 State richness.

We train variants with the same BC+PPO procedure but different state inputs (all include \(\alpha\) and \(\tau_t\)):

• Attention-based prediction \(\hat{p}_t\) only: \((\alpha,\hat{p}_t,\tau_t)\).

• Attention-based prediction \(\hat{p}_t\) and prefix embedding \(z_t\): \((\alpha,\hat{p}_t,z_t,\tau_t)\).

• Attention-based prediction \(\hat{p}_t\), naive prediction \(b_t\), and evidence vector \(\mathbf{E}_t\): \((\alpha,\hat{p}_t,b_t,\mathbf{E}_t,\tau_t)\).

8.3.5.2 Single-\(\alpha\) policies.

We train separate fixed-\(\alpha\) policies for each \(\alpha\in\{0,0.1,\ldots,1.0\}\), omitting \(\alpha\) from the input. Each policy is evaluated at its training \(\alpha\).

8.3.5.3 Belief source ablation.

We keep the pipeline unchanged but replace \(\hat{p}_t\) with the naive score \(b_t\): BC uses a threshold policy on \(b_t\), and PPO observes \((\alpha,b_t,\tau_t,z_t)\).

8.3.5.4 No BC.

We remove BC and train PPO from scratch. Since PPO is no longer initialized from expert demonstrations, we train for 100M steps (instead of 10M).

9 Dataset Details↩︎

Here we provide detailed descriptions of the datasets used in our experiments. Note that we use positive labels for failure and negative labels for success, since we are interested in early failure alerting. Table 10 summarizes dataset statistics and splits. Sample trajectories are provided in Figure 9-13.

We checked all datasets for personally identifying information (PII). The only dataset requiring attention is PCS, which consists of proprietary customer service transcripts; all identifying fields (names, contact details, account identifiers) were removed and replaced with generic placeholders prior to use. The other four datasets do not contain real personal data: BETOLD represents dialogs via intent and entity annotations rather than raw text, Persuasion for Good consists of crowdsourced dialogs collected without personally identifying metadata, and AppWorld and ALFWorld are fully synthetic environments.

9.1 Dialog Datasets↩︎

9.1.0.1 PCS (Product Customer Service).

We use a proprietary customer service chat transcripts dataset. Each dialog comes with a definitive resolution label, where we label a dialog as positive if it is resolved and negative if it is not resolved. Each dialog has two speakers: the customer and the agent. The dataset is split 80/10/10 (train/val/test) using stratified sampling to preserve class balance.

9.1.0.2 BETOLD.

This dataset contains task-oriented dialogs annotated for breakdown detection [3]. We use the original train/valid/test splits provided by the dataset. Each dialog is represented by a sequence of utterances with intent and entity annotations. We map the LUHF (Lack of User-Helpful Feedback) labels: “not LUHF” dialogs receive a negative label (-1), while “LUHF” dialogs (indicating breakdown) receive a positive label (+1).

9.1.0.3 Persuasion for Good.

This dataset contains dialogs between a persuader and persuadee regarding charitable donations [28]. Dialogs where the persuadee declined to donate are labeled as positive (+1), while successful persuasion attempts are labeled as negative (-1). The dataset is split 80/10/10 using stratified sampling.

9.2 Agentic Reasoning and Planning Datasets↩︎

For both AppWorld [29] and ALFWorld [30], pre-collected offline trajectories were not publicly available, so we generated all agent trajectories ourselves by running LLM-based agents in the respective environments. We utilized public ReAct agent implementations (https://github.com/stonybrooknlp/appworld and https://github.com/lajiman/alfworld-with-llm) for both environments.

9.2.0.1 AppWorld.

AppWorld [29] is a simulated environment for evaluating autonomous agents on day-to-day digital tasks across nine applications with 457 APIs. We label trajectories based on task completion provided by the environment: successful completion receives a negative label (-1), while failure receives a positive label (+1).

Environment and Task Splits. The AppWorld benchmark organizes tasks into scenarios, with 3 task instances per scenario. We use three splits: Train (30 scenarios \(\times\) 3 = 90 tasks), Dev (19 scenarios \(\times\) 3 = 57 tasks), and Test_Normal (56 scenarios \(\times\) 3 = 168 tasks).

Agent Implementation. We use a ReAct (Reasoning + Acting) agent [44] based on the reference implementation provided by AppWorld (https://github.com/stonybrooknlp/appworld). The agent generates Python code blocks as actions, receives execution outputs from the environment, and continues until task completion or the maximum number of interactions (50) is reached. The benchmark is publicly available at https://github.com/stonybrooknlp/appworld.

LLMs Used. To obtain varied behavior among the trajectory samples, we employed seven large language models spanning two model families (GPT-4.1 [45] and GPT-5 [31]) with different parameter sizes. Specifically, we used: gpt-41-nano-2025-04-14, gpt-41-mini-2025-04-14, gpt-41-2025-04-14, gpt-5-nano-2025-08-07, gpt-5-mini-2025-08-07, gpt-5-chat-2025-08-07, and gpt-5-2025-08-07. The diversity in model families and sizes ensures heterogeneous reasoning patterns and action sequences across trajectories, which is beneficial for training downstream models on this data.

Trajectory Statistics. We generate trajectories as follows: Train: 90 tasks \(\times\) 7 models \(\times\) 5 runs = 3,150 trajectories; Val: 57 tasks \(\times\) 7 models \(\times\) 1 run = 399 trajectories; Test: 168 tasks \(\times\) 7 models \(\times\) 1 run = 1,176 trajectories. Total: 4,725 trajectories. For training, we use 5 runs per task-model pair with temperature 0.3 to increase diversity.

9.2.0.2 ALFWorld.

ALFWorld [30] is a text-based embodied environment where agents perform household tasks (e.g., finding objects, heating food) by issuing natural-language actions. We label trajectories based on task completion label provided by the environment: successful completion receives a negative label (-1), while failure receives a positive label (+1).

Environment and Task Splits. We use the standard ALFWorld environment splits: Train (3,553 tasks), Eval in-distribution (140 tasks), and Eval out-of-distribution (134 tasks). Each task corresponds to one ALFWorld episode. For every episode, we recorded the full agent–environment interaction trace (interleaved reasoning, actions, and environment observations), along with episode metadata (goal, success flag, and episode length).

Agent Implementation. We use the ReAct agent implementation from the public repository (https://github.com/lajiman/alfworld-with-llm). The agent alternates natural-language reasoning (think) steps with environment actions, conditioned on the scene description and task goal. The agent uses in-context demonstrations (few-shot examples) to induce the desired Think/Action formatting and to stabilize behavior across task instances.

LLMs Used. To obtain varied behavior among the trajectory samples, we use three LLMs from the GPT-4o and GPT-4.1 families: GPT-4o [46] (gpt-4o-2024-11-20), GPT-4o-mini (gpt-4o-mini-2024-07-18), and GPT-4.1-nano (gpt-41-nano-2025-04-14). We intentionally avoided frontier models to ensure a meaningful number of failure trajectories, as frontier models tend to produce mostly successful trajectories.

Trajectory Statistics. We run each task once per model (3 models). Train: \(3{,}553 \times 3 = 10{,}659\) trajectories; Eval in-dist: \(140 \times 3 = 420\); Eval out-of-dist: \(134 \times 3 = 402\). Total: 11,481 trajectories.

9.3 Artifact Licenses↩︎

Table 11 lists the licenses for all publicly released scientific artifacts used in this work.

PCS is proprietary customer-support data and is not publicly released. The ALERT\(^*\) baseline code [11] was made available via an anonymous submission repository and does not carry a stated open-source license; we used it solely for experimental comparison. The ALFWorld ReAct agent implementation does not carry a stated license in its repository. The proprietary LLM APIs used in this work—GPT-4o, GPT-4.1, and GPT-5 (OpenAI), Gemini 3 Pro (Google), and Claude Opus 4.5 (Anthropic)—are accessed under their respective commercial terms of service and are not open-source. All artifacts listed above were used in a manner consistent with their intended use and the terms under which they were released.

None

None

None

None

None

10 Retrospective LLM Analysis of Failure-Indicating Turns↩︎

This appendix provides the details and robustness checks for the LLM-judge analysis summarized in Section 5.1. We use an LLM judge as an independent qualitative sanity check on the temporal distribution of failure-indicating information. The judge is given each full failure trajectory and asked to rate every turn’s relevance to the eventual outcome; this is a retrospective attribution task, distinct from online alerting, and we therefore use it only as a diagnostic rather than as a quantitative measure of evidence or as a replacement for our learned estimators. The analysis is conducted only on failure trajectories.

10.0.0.1 Methodology.

For each trajectory, we provide the LLM with the full dialog as context and five few-shot reference trajectories (both successes and failures), then ask it to rate every turn independently on a 1–5 relevance scale (1 = no apparent contribution, 5 = critical contribution). To reduce prompt-induced artifacts, we instruct the model to rate each turn based on its content rather than to force a particular distribution of scores. We use GPT-5.2, Gemini 3 Pro, and Claude Opus 4.5 at temperature 0 with a fixed few-shot set, and evaluate 100 held-out failure trajectories per dataset.

10.0.0.2 Results.

The main-body summary (Table 1) reports the percentage of high-rated turns and the position of the first high-rated turn, averaged across the three judges. We additionally provide: (i) the percentage of low-rated turns under the same aggregation (Table 12), which accounts for 65.5–75.7% of turns across datasets, indicating that the non-high mass is concentrated on low rather than neutral ratings; (ii) two aggregation-robustness checks—majority-vote thresholding (Table 13) and leave-one-judge-out (Table 14)—both of which preserve the sparse-and-late pattern; (iii) inter-judge agreement via Fleiss’ \(\kappa\) (Table 15), which is fair to substantial on the binarized high/low indicators (\(\kappa\in[0.45,0.65]\)) and fair on the full 5-category rating (\(\kappa\in[0.32,0.45]\)). Together, these observations indicate that failure-indicating information is sparse and tends to appear late in the trajectory, consistent with the evidence-mass concentration in our top-\(K\) analysis (Fig. 4) and with the empirical behavior that the attention-based predictor can peak late while the naive predictor often peaks earlier on low-evidence prefixes.

10.0.0.3 Prompts.

Below we provide the exact prompts used for each dataset to ensure reproducibility and transparency in our LLM-based evidence analysis.

PCS:

BETOLD:

Persuasion for Good:

AppWorld:

ALFWorld:

11 Baseline Implementation Details↩︎

11.1 Trigger Baselines↩︎

11.1.0.1 Plug-in Threshold.

Plug-in Threshold is an internal sanity check that applies a myopic threshold rule directly to our attention-based predictor: it stops when the predicted failure probability \(\hat{p}_t\) exceeds a fixed threshold. It serves as a reference point that isolates the contribution of the learned \(\alpha\)-STOP policy from that of the underlying predictor. For each \(\alpha\) value, we learn the optimal threshold on the validation set by exhaustively sweeping threshold values in \([0,1]\) with step size 0.001 and selecting the threshold that maximizes the reward objective in Eq. 11 . We repeat this procedure for 100 \(\alpha\) values sampled uniformly from \([0,1]\). We then evaluate the resulting model at its corresponding \(\alpha\) value.

11.1.0.2 ALERT\(^*\).

We use the ALERT\(^*\) [11] implementation from the original codebase (https://anonymous.4open.science/r/ALERT). ALERT\(^*\) trains a DQN-style offline RL triggering policy on trajectories constructed from per-turn class probabilities. Concretely, at each turn \(t\) we take a scalar score \(s_t \in [0,1]\) and convert it into a two-class probability vector \(\bigl[P(y{=}0), P(y{=}1)\bigr] = \bigl[1-s_t,\, s_t\bigr]\), build state features using the same feature set as in the authors’ codebase (e.g., max probability, margin, predicted label, confidence bins, and normalized time), and train with the same training procedure and hyperparameters as in the authors’ codebase. In our experiments, we set \(s_t=\hat{p}_t\) (attention-based failure prediction) for Fig. 2 and Table 3, and set \(s_t=b_t\) (naive failure prediction) for Table 9. ALERT\(^*\) is trained for a single trade-off setting, so we train 11 models, one for each \(\alpha \in \{0.0, 0.1, \ldots, 1.0\}\), and evaluate each model at its corresponding \(\alpha\) using the authors’ protocol.

ALERT\(^*\) is originally implemented for early classification, so we made minor modifications to adapt it to the early alerting setting. In our setting, triggering corresponds to predicting class \(1\) (rather than predicting the argmax class as in early classification), and reaching the end without triggering corresponds to predicting class \(0\) (rather than being forced to make a final class-0 vs.class-1 prediction). Rewards are computed using the same objective as Eq. 11 , which is the same reward objective as in the original ALERT\(^*\) codebase.

11.1.0.3 FIRMBOUND.

We use FIRMBOUND [12] via the authors’ original codebase (https://github.com/Akinori-F-Ebihara/FIRMBOUND). FIRMBOUND learns time-dependent stopping boundaries via backward induction to optimize the accuracy–earliness tradeoff. Following the standard FIRMBOUND interface, we convert per-turn probabilities (\(s_t\)) to log-odds and then to an LLR-matrix representation (used as the sufficient statistic), and learn a boundary model for each time step using Gaussian process (GP) regression as it trains much faster than convex function learning (CFL). In our experiments, we set \(s_t=\hat{p}_t\) (attention-based failure prediction) for Fig. 2 and Table 3, and set \(s_t=b_t\) (naive failure prediction) for Table 9.

FIRMBOUND is originally implemented for early classification, so we made minor modifications to adapt it to the early alerting setting. In our setting, early stopping corresponds to predicting only the positive class and if no stop occurs by the deadline, the prediction is class \(0\).

Accordingly, the stopping risk becomes the risk of stopping-and-predicting class\(1\) (proportional to \(1-p_1\)), while the deadline risk corresponds to predicting class \(0\) at the final available step (proportional to \(p_1\)).

Penalty sweep protocol. FIRMBOUND does not optimize the same \(\alpha\)-weighted reward objective as Eq. 11 , so there is no direct mapping from FIRMBOUND’s cost parameters to a specific \(\alpha\) value. Nonetheless, we can trace out a Pareto frontier by sweeping the misclassification penalty parameter (with fixed per-step sampling cost 0.1): for each penalty value, we retrain the time-dependent stopping boundary and then evaluate the resulting model. We use a log-spaced sweep of 100 misclassification penalty values in \([1, 1000]\); we chose 100 points because the mapping to \(\alpha\) is unknown and we wanted to cover the Pareto frontier as thoroughly as possible. As shown in Fig. 2, we find that FIRMBOUND is able to cover the Pareto frontier.

Note. Both ALERT\(^*\) and FIRMBOUND do not produce the full Pareto frontier with a single trained model (as opposed to \(\alpha\)-STOP). Instead, each point on their Pareto front is obtained by retraining (or re-fitting) the method under a different trade-off setting (e.g., different \(\alpha\) for ALERT\(^*\), and different misclassification penalty for FIRMBOUND).

11.2 End-to-End RL Baseline↩︎

We use the same PPO implementation and environment setup as \(\alpha\)-STOP. Unlike \(\alpha\)-STOP, end-to-end RL does not use a classifier, so the policy state excludes belief/evidence scores and instead consists only of the turn embedding \(\mathbf{h}_t\) and normalized time \(\tau\). Since there is no expert classifier to generate demonstrations, we do not use behavior cloning initialization. We therefore train PPO from scratch for 100M steps (instead of 10M for \(\alpha\)-STOP).

11.3 LLM Judge Baseline↩︎

We implement an LLM-based online early failure detector that reads each dialog turn-by-turn and decides whether to trigger a failure prediction now or continue monitoring. Concretely, at turn \(t\) we provide the LLM with the conversation prefix (up to turn \(t\)) and ask whether the eventual failure is already evident. If the LLM triggers at any turn, we predict FAILURE and record the trigger step; if the LLM never triggers before the end, we predict SUCCESS. We treat FAILURE as the positive class prediction.

11.3.0.1 LLMs.

We evaluate three frontier large reasoning models:

• Claude Opus 4.5 (anthropic.claude-opus-4-5-20251101-v1-0)

• GPT-5.2 (gpt-5.2-2025-12-11)

• Gemini 3 Pro (gemini-3-pro-preview)

11.3.0.2 Decoding and structured output.

For each decision we use deterministic decoding (temperature \(=0\)) and request a structured output with fields: should_trigger_failure (bool), confidence (\(\in[0,1]\)), reasoning (1–2 sentences), and failure_indicators (list). We retry up to 5 times on transient failures.

11.3.0.3 Few-shot calibration.

We use balanced few-shot prompting with \(K=10\) success examples and \(K=10\) failure examples, sampled from the validation split.

11.3.0.4 Prompt template (shared across datasets).

At each turn \(t\), we instantiate the following template, where the SYSTEM_PROMPT and GUIDELINES blocks are dataset-specific and the few-shot block contains \(K\) labeled success and \(K\) labeled failure dialogs:

11.3.0.5 Dataset-specific prompts.

We use tailored system prompts and guidelines for each dataset:

PCS

SYSTEM PROMPT

GUIDELINES

BETOLD

SYSTEM PROMPT

GUIDELINES

Persuasion for Good (P4G)

SYSTEM PROMPT

GUIDELINES

AppWorld

SYSTEM PROMPT

GUIDELINES

ALFWorld

SYSTEM PROMPT

GUIDELINES

Note. Unlike threshold-based or RL-based triggers, LLM judges do not provide a mechanism to sweep the accuracy–earliness trade-off. Each LLM model produces only a single point on the accuracy–earliness curve; there is no knob to trade off more accuracy for later stopping or vice versa.

12 Metric Computation Details↩︎

We evaluate early-failure detectors in the accuracy–earliness plane, where each method produces a set of operating points by sweeping its control parameter (e.g., threshold, \(\alpha\), or penalty). We report (i) Max Acc (best achievable accuracy over the sweep), (ii) HV (hypervolume; higher is better; [35]), and (iii) IGD+ (inverted generational distance plus; lower is better; [36]). For HV and IGD+ we use PyMOO’s [47] indicator implementations.

12.1 Earliness↩︎

We define earliness as the ratio of the trigger turn to the total number of turns, a standard choice in early-risk / early detection evaluation [34]. We compute it over the subset of successfully triggered samples, following common practice in early-risk evaluation: \[\text{earliness} = \frac{1}{|\mathcal{S}|}\sum_{i \in \mathcal{S}} \frac{t_i^{\text{trigger}}}{T_i},\] where \(\mathcal{S}\) is the set of samples on which the model successfully triggers, \(t_i^{\text{trigger}}\) is the turn at which the model triggers for sample \(i\), and \(T_i\) is the total number of turns for sample \(i\). Earliness is bounded in \([0,1]\), where lower values indicate earlier stopping. If the model never triggers on any sample, we set earliness to \(1.0\) (i.e., equivalent to waiting until the end).

Because earliness is computed over triggered samples, improvements in earliness could in principle come from reduced coverage (i.e., triggering on fewer positives). We therefore also report Recall–Earliness curves, where recall is the fraction of positives that trigger before the horizon. In the operating region relevant to the reported Pareto fronts, \(\alpha\)-STOP’s gains are not explained by sacrificing recall; its recall is comparable to the best non-degenerate baselines (see Figure 14).

12.2 Pareto Frontier Construction↩︎

Given an operating point \(p=(a,e)\) with accuracy \(a\in[0,1]\) and earliness \(e\in[0,1]\), we define Pareto dominance (maximize accuracy, minimize earliness) as: \[\begin{align} (a_1,e_1) \succ (a_2,e_2) &\iff (a_1 \ge a_2)\wedge(e_1 \le e_2) \\ &\quad\wedge\big((a_1>a_2)\vee(e_1<e_2)\big). \end{align}\] For plotting and for IGD+ reference construction, we form a reference Pareto set by pooling all operating points from all compared methods and discarding dominated points under the above relation.

12.3 HV↩︎

We compute hypervolume using PyMOO’s Hypervolume indicator. Since PyMOO assumes minimization, we transform each operating point \((a,e)\) into minimization space: \[x = (-a, e).\] We use the fixed reference point \[r = (0,1),\] which corresponds to accuracy \(=0\) and earliness \(=1\) (i.e., the worst possible point in our bounded metric space). This choice (i) dominates all feasible transformed points (since \(-a\in[-1,0]\) and \(e\in[0,1]\)), (ii) is dataset- and method-independent, and (iii) makes HV comparable across methods on the same task.

12.4 IGD+↩︎

We compute IGD+ using PyMOO’s IGDPlus indicator in the same transformed (minimization) space \(x=(-a,e)\). The reference Pareto front for IGD+ is computed as follows:

1. Collect the union of all evaluated operating points \((a,e)\) across all compared methods.

2. Filter the union to its non-dominated subset under the dominance definition above, yielding \(\mathcal{P}_{\mathrm{ref}}\).

3. Transform \(\mathcal{P}_{\mathrm{ref}}\) into minimization form \(\mathcal{P}_{\mathrm{ref}}^{\min} = \{(-a,e) : (a,e)\in\mathcal{P}_{\mathrm{ref}}\}\).

IGD+ for a method is then computed as the distance from that method’s point set (in minimization form) to \(\mathcal{P}_{\mathrm{ref}}^{\min}\), where lower values indicate closer approximation to the pooled non-dominated trade-off frontier.

12.5 HSSP Subsampling↩︎

To ensure fair comparison when methods produce different numbers of operating points, we compute HV and IGD\(^+\) after selecting a fixed-size subset of \(K\) points from each method. We use the hypervolume subset selection problem (HSSP) framework [37] and apply a standard greedy approximation that selects points to (approximately) maximize hypervolume.

12.5.0.1 Protocol.

For each method:

1. Collect all operating points \((a,e)\) (accuracy \(a\in[0,1]\), earliness \(e\in[0,1]\)).

2. Filter to the non-dominated subset under our dominance definition.

3. Select \(K\) points via greedy HSSP:

   • Initialize \(\mathcal{A}\leftarrow\emptyset\).

   • While \(|\mathcal{A}|<K\) and candidates remain, add the point \(p^\star\) that maximizes the hypervolume gain \(\mathrm{HV}(\mathcal{A}\cup\{p\})\).

We compute HV on the selected set in a minimization space (we minimize \((-a,e)\)) using reference point \(r=(0,1)\).

For IGD\(^+\), we compute distances from each method’s selected set to a common reference front constructed as the non-dominated union over all methods’ operating points (before subsampling).

This approach ensures that: (i) each method contributes exactly \(K\) points (or fewer if \(K\) exceeds the method’s non-dominated set size) and (ii) comparisons are independent of how densely each method samples the accuracy-earliness trade-off space.

13 MIL-Evidence-Only Ablation↩︎

To isolate the role of fusion in the attention-based failure predictor, we ablate the fusion stage and threshold the scalar MIL evidence score \(\hat{E}_t\) (Section 3.1.2) alone. Table 6 compares this MIL-only signal against the full attention-based predictor \(\hat{p}_t\) under the same Plug-in Threshold rule.

The MIL-only signal achieves higher maximum accuracy on every dataset, but yields lower hypervolume and substantially worse IGD\(^+\) than the full attention-based predictor. This is because \(\hat{E}_t\) rises only after evidence accumulates and so triggers late: it produces accurate but conservative alerts and therefore covers a narrow region of the accuracy–earliness frontier. The attention-based failure predictor fuses \(b_t\) and \(\mathbf{E}_t\) via the MLP \(g_\psi\) and achieves stronger Pareto-frontier quality on all five datasets, confirming that the fusion stage is what turns the selective MIL evidence into a useful online predictor.

14 AI Assistance↩︎

LLMs (ChatGPT ⁴ and Claude ⁵) were used for coding assistance and polishing the writing of this manuscript. All the content, experimental design, results, and claims were produced and verified by the authors.

References↩︎

---

1. Code and data will be released upon publication.↩︎

2. Work done during a research internship at Intuit AI Research.↩︎

3. We use public ReAct agent implementations for both environments and provide full dataset construction details in Appendix 9. We will release the datasets upon publication.↩︎

4. https://chatgpt.com/↩︎

5. https://www.anthropic.com/claude↩︎