1 Introduction↩︎

LLM-based agents increasingly tackle long-horizon tasks (web navigation, tool use, embodied simulation) in which a single episode spans many rounds of interaction and consumes thousands of generated tokens. A large fraction of this compute is wasted: when an agent has misunderstood the task, entered an unrecoverable state, or begun to loop, the episode is already doomed long before it formally times out or returns a wrong answer. If we could detect such doomed episodes early and abort them, the saved compute could be reallocated to retries, sampling additional trajectories, or simply reducing serving cost.

Three obstacles stand in the way. First, we need a signal that distinguishes doomed episodes from eventually-successful ones early in the trajectory, and early is precisely when behavioral evidence is scarcest. We find that scorers reading only the agent’s observable behavior are barely better than chance in the first round and become informative only around rounds 3–4, by which time over a third of episodes have already finished and much of the useful remaining compute is gone. Lightweight probes on the agent’s internal activations show the opposite pattern: at the very first round they already match or exceed the surface scorer’s eventual peak, attained only two to three rounds later, and they reach their own peak at round 2 (Figure 4).

Second, any abort policy is only usable in deployment if it comes with a controllable guarantee on the harm it causes: aborting an episode that would have succeeded silently destroys task reward, so practitioners need to bound the rate of such false aborts before deployment, not observe it afterwards. Third—and specific to the sequential nature of agent episodes, a monitor that re-evaluates the episode at every round faces accumulating risk: even if each individual check rarely kills a good episode, a successful trajectory must survive all of them, so per-round guarantees do not compose into the episode-level guarantee that matters.

We address these with a recall-controlled cascade (Figure 1): at each of an episode’s first \(R_g\) rounds, a gate aborts the episode if its probe score exceeds a threshold \(\tau_r\). Each threshold is calibrated so that an exact binomial (Clopper–Pearson) lower confidence bound on the gate’s survival rate for successful episodes meets a per-round recall budget \(t_r\), and the budget vector \((t_1, \dots, t_{R_g})\) is searched on a disjoint validation split to maximize compute savings subject to the global recall (the fraction of eventually-successful episodes that survive every gate) meeting a user-chosen target. A safety margin makes the search robust to validation noise; a stricter variant replaces the margin with a certificate on the global recall itself, yielding a distribution-free, a priori verifiable guarantee.

The cascade is strongly compute-positive. At a 90% global recall target it saves \(47.1\% \pm 10.3\%\) of inference compute for a Qwen-2.5-7B agent and \(37.2\% \pm 8.8\%\) for a Llama-3.2-3B agent, \(1.6\)–\(1.7\times\) the best single-gate policy at the same target, confirming that the freedom to distribute recall budget across rounds, rather than spend it all at one decision point, is where much of the practical value lies. At a conservative 95% target the cascade still roughly doubles the single-gate savings, and its achieved recall matches its target in every configuration tested. Swapping the probe for a behavior-only scorer inside the identical cascade cuts savings roughly in half on the stronger agent, and stacking behavioral features onto the probe adds nothing, indicating the internal signal already captures the behavioral one.

Finally, we give an honest account of what certified recall control costs in data. With \(n\) successful validation episodes, a one-sided certificate at confidence \(95\%\) can support recall targets only up to \(0.05^{1/n}\); at our scale this caps certifiable targets near \(0.974\), and indeed the certified variant abstains at targets \(0.98\) and \(0.99\), which would require roughly \(1.3\times\) and \(2.6\times\) more successful episodes than we have (Figure 8). We view this not as a weakness of the method but as its point: the same machinery that saves compute also tells the practitioner, before deployment, which promises the available data can and cannot back.

Our contributions are:

• We demonstrate across two agent models that eventual task failure of an LLM agent is predictable from internal activations within the first interaction rounds, whereas behavior-only scorers become informative only at rounds 3–4, after most recoverable compute is spent.

• We propose the first abort policy for LLM agents that controls episode-level success recall across multiple sequential decision points, via per-round distribution-free gates whose recall budgets are jointly optimized under a global constraint, with an optional certificate.

• We show the cascade saves up to \(47.1\%\) of inference compute at 90% global recall (\(1.6\)–\(1.7\times\) the best single-gate policy) and characterize the data requirements for certifying stricter targets.

2 Method↩︎

2.1 Problem Setup↩︎

An agent episode is a sequence of interaction rounds \((s_1, a_1, s_2, a_2, \dots)\) between an LLM policy and an environment, terminating with a binary outcome \(y \in \{0, 1\}\). We index rounds from one; episodes run up to \(R_{\mathrm{full}} = 20\) rounds. Let \(c_r\) denote the cumulative inference cost incurred through round \(r\) and \(C\) the total cost of the full episode.

An abort cascade places a gate at each of the first \(R_g\) rounds (\(r = 1, \dots, R_g\); we use \(R_g = 6\)). At gate \(r\), every episode still running is scored by a per-round scorer \(f_r\) and aborted if \(f_r(x) > \tau_r\), losing any potential success but saving the remaining cost \(C - c_r\). Episodes terminate at different rounds; in our data, over a third of episodes finish within the first two rounds (Figure 3), so a gate at a late round sees only the episodes still alive there: late gates are cheap in recall but also guard less remaining compute. This trade-off is exactly what the cascade optimizes.

Aborting an episode that would have succeeded is the critical error mode. We target global success recall: among episodes with \(y = 1\), the fraction that survive every gate and run to completion. The design goal is to maximize expected cost savings subject to a user-specified global recall floor \(\rho^\star\). Note that controlling the recall of each gate in isolation does not control the global recall: false-abort probability accumulates across gates, and the accumulation depends on how many successes each gate sees.

2.2 Per-Round Failure Scorers↩︎

At each gate round \(r\), we extract a feature vector \(x \in \mathbb{R}^d\) from the agent LLM’s internal activations: the residual-stream hidden state at the final token of the agent’s generated action in round \(r\), extracted via a teacher-forced forward pass over the trajectory, at a single layer per model (layer 14 of 28 for Llama-3.2-3B, layer 20 of 28 for Qwen-2.5-7B). Layers were fixed once per model by preliminary per-layer probe-AUC sweeps: for Qwen-2.5-7B over layers \(\{0, 2, \dots, 28\}\) on an independent pilot set of rollouts collected before the main experiments (post-generation AUC peaks at layer 20), and for Llama-3.2-3B over layers \(\{6, 10, 14, 18, 22, 26\}\), where the post-generation sweep selects layer 14 (see the technical appendix, “Per-Layer Probe AUC Sweeps”). A per-round probe \(f_r\), a logistic regression on standardized features (\(L_2\) regularization, \(C = 1\)), is trained to predict eventual failure \(1 - y\) from episodes alive at round \(r\).

We compare against two alternative scorers, holding the entire downstream calibration and search pipeline fixed. The surface scorer observes only the agent’s behavior: a logistic model over trajectory features observable from the serving API alone: the mean action-token log-probability of the current round, the mean over preceding rounds, the number of generated tokens, the prefix length, and the count of preceding rounds whose environment feedback contains error keywords (“error”, “invalid”, “fail”, etc.). The stacking scorer concatenates these surface features onto the probe’s activation features, testing whether behavioral evidence adds anything beyond the hidden state.

2.3 Per-Round Recall-Calibrated Gates↩︎

Each gate’s threshold is set on a calibration split so that the gate provably passes at least a \(t_r\) fraction of successful episodes. Given a per-round recall budget \(t_r\), let \(\mathcal{S}_r = \{f_r(x_i) : y_i = 1,\;i \text{ alive at } r\}\) be the scores of the \(n_r\) successful calibration episodes alive at round \(r\). For a candidate threshold \(\tau\), let \(k(\tau) = |\{s \in \mathcal{S}_r : s \le \tau\}|\) be the number of survivors; the exact binomial (Clopper–Pearson) lower confidence bound on the gate’s true survival rate is the Beta quantile \[\underline{p}(\tau) \;=\; \mathrm{Beta}^{-1}\!\left(\alpha;\; k(\tau),\; n_r - k(\tau) + 1\right),\] and we set \(\tau_r\) to the smallest calibration score with \(\underline{p}(\tau_r) \ge t_r\), at per-gate confidence level \(1 - \alpha\) with \(\alpha = 0.05\). This is deliberately stronger than a marginal conformal quantile: the per-round guarantee holds with high confidence over the calibration draw, at the price of conservatism when \(n_r\) is small (a gate whose \(n_r\) cannot support \(t_r\) abstains, i.e., aborts nothing). A conformal-quantile variant (\(\tau_r = \mathrm{Quantile}_{\lceil (n_r+1) t_r \rceil / n_r}(\mathcal{S}_r)\)) is compared in the technical appendix (“Conformal-Quantile Gates”). A budget of \(t_r = 1\) disables the gate.

2.4 Recall Budget Search Under a Global Constraint↩︎

Per-round guarantees do not compose multiplicatively in any useful way: a union bound over gates is valid but hopelessly conservative, because it ignores that late gates expose few successes and that per-gate false-abort events are far from disjoint. We therefore treat the budget vector \(\mathbf{t} = (t_1, \dots, t_{R_g})\) as a hyperparameter and select it empirically on a validation split disjoint from calibration.

Concretely, probe scores are produced by task-grouped cross-fitting (stratified group \(k\)-fold), so every episode is scored by a probe that never saw its task during training. Tasks are then partitioned into a calibration set for gate thresholds (\(20\%\) of tasks), a validation set for budget search (\(20\%\)), and a held-out test set (\(60\%\)). Grouping by task throughout ensures no task contributes episodes to two sides of any split.

For each candidate \(\mathbf{t}\) on the grid \(t_r \in \{0.85, 0.90, 0.95, 0.98, 0.99, 1.0\}\) (\(6^6 = 46{,}656\) candidates), we calibrate all gates on the calibration split, simulate the full cascade on the validation split, and record its global recall \(\hat{\rho}_{\mathrm{val}}(\mathbf{t})\) and compute savings. The deployed budget maximizes validation savings subject to a feasibility condition, for which we study two variants:

• Margin (default): require \(\hat{\rho}_{\mathrm{val}}(\mathbf{t}) \ge \rho^\star + \delta\) with a fixed safety margin \(\delta = 0.02\). This is a heuristic guard against selection bias from searching over many candidates; it carries no formal guarantee, and we validate it empirically.

• Certificate: require that the Clopper–Pearson lower bound at level \(1 - \alpha_m\) (\(\alpha_m = 0.05\)) on the global recall, computed from the \(n_{\mathrm{pos}}\) successful validation episodes, exceed \(\rho^\star\). This yields a distribution-free guarantee that holds despite the search, at the price of conservatism analyzed below.

If no candidate is feasible, the policy abstains and aborts nothing. Two fixed budget allocations serve as structural baselines: single-gate, which spends the entire recall budget at a single round, with both the round and its budget selected by the same validation search (recovering the single-decision-point policies of prior monitoring work as a special case of our framework), and uniform, which sets the same \(t_r\) at every gate.

2.4.1 Sample complexity of certification.↩︎

The certificate makes visible a fundamental data requirement. Even a candidate that aborts nothing has a validation lower bound of \(\mathrm{Beta}^{-1}(\alpha_m; n_{\mathrm{pos}}, 1) = \alpha_m^{1/n_{\mathrm{pos}}}\) (the “rule of three”), so targets above this level are unattainable regardless of scorer quality. Our splits yield \(n_{\mathrm{pos}} = 113\) (Llama) and \(115\) (Qwen), capping certifiable targets near \(0.974\); the resulting sample complexity of stricter targets is analyzed in the Results (Figure 8) and extends directly when recalibrated on more data.

3 Experimental Setup↩︎

3.0.1 Environment and Agents.↩︎

We use TextCraft [1], a text-based crafting environment from the AgentGym suite [2] in which an agent must synthesize a target item by navigating multi-step crafting recipes, with binary task success. We collect \(800\) episodes over \(100\) tasks for each of two agent policies: Llama-3.2-3B [3] (task success rate \(63.0\%\)) and Qwen-2.5-7B [4] (\(74.8\%\)), sampling at temperature \(1.0\) (8 episodes per task). Episodes run up to \(R_{\mathrm{full}} = 20\) rounds; gates are placed at rounds \(1\)–\(6\). The population still alive at each gate decays quickly (Figure 3): \(35\%\) (Llama) and \(39\%\) (Qwen) of episodes already finish within two rounds, and \(53\%\)/\(69\%\) before the last gate, quantifying how sharply late gates’ exposure decays. Compute savings are reported as the fraction of total generated tokens saved by aborts, counting only each aborted episode’s remaining computation.

3.0.2 Protocol.↩︎

Probes are per-round logistic regressions (\(C = 1\), standardized features) on layer-14 (Llama) / layer-20 (Qwen) activations. The entire pipeline (task-level splitting, probe cross-fitting, gate calibration, budget search, evaluation) is repeated over \(20\) random seeds; we report test-split mean \(\pm\) standard deviation. All scorers and all budget-allocation baselines share identical splits, calibration machinery, and search procedure, so differences isolate the quality of the underlying signal (scorer comparisons) or the value of distributing the recall budget (allocation comparisons).

4 Results↩︎

4.1 Internal States Predict Failure Before Behavior Does↩︎

Figure 4 plots the cross-fitted AUC of each per-round scorer for predicting eventual failure among episodes alive at that round. The behavior-only surface scorer is barely better than chance at the first gate and becomes informative only around rounds 3–4, consistent with behavioral evidence (repetitions, lack of progress, invalid actions) needing time to accumulate. The probe shows the opposite timing pattern: its first-round AUC already matches or exceeds the surface scorer’s peak, which the latter attains only two to three rounds later, and the probe peaks at round 2, an absolute advantage of \(0.12\)–\(0.21\) AUC over the surface scorer in precisely the rounds where most of the episode’s compute is still unspent. The two signals converge only at round 6, by which point the majority of episodes have already ended (Figure 3). The probe’s decline after round 2 reflects population filtering rather than fading signal: episodes that terminate early leave the alive-set, concentrating the harder, longer-running cases at later gates. This timing difference matters more than peak values: discriminative power at rounds 1–2 protects far more useful remaining compute than the same power at rounds 3–4, a point the cascade results below make quantitative.

4.2 Main Results: Cascade vs.Single Gate↩︎

Table ¿tbl:tab:main? reports the compute savings of the abort cascade at four global recall targets for both agents, against the single-gate and uniform budget allocations; Figure 5 plots the recall each cascade actually achieves against its target, and Figure 6 traces the savings–target frontier. Three observations stand out.

First, achieved recall matches its target in every configuration (Figure 5): across all eight agent–target pairs the cascade’s mean test recall deviates from its target by at most \(0.027\), always within one standard deviation, and errs above the target in seven of eight cases. The savings in Table ¿tbl:tab:main? are thus attained under honest risk control rather than post-hoc threshold tuning.

Second, distributing the recall budget across rounds is worth \(1.6\)–\(2.3\times\) over the best single decision point, at every target and for both agents (Figure 6); for Llama the ratio widens as the target tightens (up to \(2.3\times\) at \(0.97\)), while for Qwen it holds near \(1.7\)–\(1.8\times\). The searched budgets explain why: the modal allocation at Llama’s \(0.90\) target is \((0.95, 0.85, 0.85, 1.0, 1.0, 1.0)\): it spends recall aggressively at rounds 1–3, where doomed episodes are identifiable and most of their cost is still unspent, and disables late gates entirely. The uniform allocation, forced to be equally strict at the recall-expensive early gates and the low-yield late ones, saves almost nothing, confirming that where the budget is spent, not just how much, drives the savings.

Third, the stronger agent yields larger savings. Qwen’s cascade saves more than Llama’s at every target despite Qwen’s higher success rate leaving fewer doomed episodes to abort. Two factors contribute: Qwen’s failure signal is simply stronger at every gate (Figure 4), and its episodes finish earlier (Figure 3), concentrating surviving failures into an increasingly separable population at mid-trajectory gates.

4.3 Does the Signal Require Internal Access?↩︎

Figure 7 fixes the target at \(0.95\) and swaps the scorer inside the otherwise-identical pipeline, for both the cascade and single-gate allocations. Three conclusions emerge.

First, internal access roughly doubles the savings of the strongest configuration on Qwen (\(32.0\%\) for the probe cascade vs.\(17.0\%\) for the surface cascade); on Llama the gap is present but narrower. Second, stacking behavioral features onto the probe adds nothing: savings land within noise of the probe alone, indicating that whatever the surface scorer knows, the hidden states already encode; the converse is clearly false. Third, the cascade structure amplifies weak scorers even more than strong ones: moving from single gate to cascade multiplies surface savings by roughly \(3.6\times\), versus about \(1.9\times\) for the probe. The per-round AUCs of Figure 4 explain this asymmetry: the surface signal only matures at rounds 3–4, so a single early gate cannot exploit it at all, whereas the cascade can defer part of the budget to the rounds where the behavioral evidence finally arrives. Notably, the surface single gate remains weak at every round—even at its best placement it saves less than the probe single gate at round 1, so the cascade is a complement to, not a substitute for, a signal that arrives early.

4.4 The Limits of Certification at High Recall↩︎

Replacing the empirical margin with the global certificate (\(\alpha_m = 0.05\)) makes the guarantee formal, and its cost explicit. Figure 8 plots the minimum number of successful validation episodes required to certify a given target, the “rule of three” bound \(n_{\mathrm{pos}} \ge \ln \alpha_m / \ln \rho^\star\), against the \(n_{\mathrm{pos}} = 113\)/\(115\) our splits provide. Targets up to \(\approx 0.974\) lie below the curve and are certifiable; \(0.98\) and \(0.99\) lie above it, so no candidate, including the no-op, can be certified, and the policy abstains with zero savings by construction. This is not the scorer’s failure but the target’s data cost: even the margin-based single gate at \(0.98\) (with the margin in mean-minus-one-\(\sigma\) form at these targets) saves only \(1.5\%\)–\(5.5\%\) across the two agents, confirming that near-perfect recall leaves little room for any abort policy at this data scale. We regard this transparency as a feature: the method reports, before deployment, that \(800\) episodes suffice to back a \(0.95\)–\(0.97\) promise but not a \(0.99\) one, and Figure 8 prescribes exactly how much additional data the stricter promise costs.

4.5 Does the Margin Actually Protect the Target?↩︎

The margin rule is a heuristic, so we test it directly. Across the eight main configurations, the searched budget’s validation recall exceeds its test recall by \(0.008\)–\(0.030\), precisely the selection optimism the \(\delta = 0.02\) margin is sized to absorb. The margin absorbs it almost exactly and no more: in the worst case (Llama at \(0.92\)) the mean test recall lands \(0.001\) under the nominal target, well within seed noise, while in all other configurations it lands above. Figure 9 visualizes this per seed. Removing the margin entirely (\(\delta = 0\)) confirms it is load-bearing: at the \(0.95\) target the same search lands at test recall \(0.943 \pm 0.032\) (Qwen) and \(0.942 \pm 0.024\) (Llama), both below target, while saving more (\(41.0\%\)/\(26.3\%\)), quantifying exactly the savings the margin trades for reliability.

5 Related Work↩︎

5.0.1 Predicting success from internal representations.↩︎

Lightweight probes on frozen activations decode properties that a model does not reliably verbalize [5], [6]. For LLMs, such probes predict whether the model knows an answer [7], whether a statement is true [8]–[10], and whether a generation is hallucinated [11]–[13], with the relevant signal repeatedly found to be linearly decodable [9], [10]. Closer to our setting, internal states anticipate chain-of-thought success before completion [14] and encode self-verification signals in reasoning models [15]. For agents specifically, [16] recover linearly separable success/failure directions in the activations of LLM agents in ScienceWorld and AlfWorld, using conformal prediction upstream to label step-level representations from sparse episode rewards, with representation steering as the downstream application. [17] probe a related but distinct quantity: representational commitment, the early convergence of an agent’s hidden states across resampled continuations, and find it deliberately orthogonal to correctness, with committed-correct and committed-wrong runs indistinguishable, whereas our probes predict the outcome itself. These results establish that outcome-relevant signal exists in the residual stream and is linearly accessible; none of them ask when in an episode the signal becomes actionable, nor turn it into a stopping rule with a controllable error rate. Both questions are the subject of this paper.

5.0.2 Failure detection and monitoring of LLM agents.↩︎

Failure analyses consistently find that decisive errors arise early and cascade through the remaining trajectory [18], [19], motivating online monitoring over post-hoc attribution; AgentRx [20] localizes the decisive failure step from completed execution trajectories, a source of step-level supervision that per-round probes like ours could in principle exploit. Existing online monitors read trajectory text or derived trajectory embeddings: AgentForesight [21] trains a separate 7B auditor with reinforcement learning to localize the earliest decisive error in multi-agent systems, [22] fit a weakly supervised text-encoder alerter with a tunable accuracy–earliness trade-off, and MASC [23] anomaly-scores step embeddings against prototype-guided next-execution reconstructions to trigger self-correction in multi-agent systems. [24] compute inexpensive behavioral statistics over agent interactions, but for offline triage of logged trajectories rather than online intervention. An alternative is to ask the agent itself: BAGEN [25] formalizes budget awareness as verbalized interval prediction of remaining cost, but finds that frontier agents are systematically overoptimistic on doomed tasks and that even after SFT and RL the intervals reach at most 47% coverage, leaving calibration as an open problem; prompted early exit likewise trades success for cost without controlling the loss [26]. Our monitor differs on the two axes that matter for deployment: the signal is read from the agent’s own activations at negligible cost, requiring neither an auxiliary LLM pass per step nor trust in the agent’s self-reports, and the abort decision carries an episode-level, a priori verifiable bound on the rate of falsely terminated successes, which no existing monitor provides; existing alerters control at best a per-check error rate, which degrades over the length of the episode.

5.0.3 Conformal prediction and risk control for LLMs.↩︎

Split conformal prediction and related distribution-free calibration give finite-sample guarantees under exchangeability [27]–[30], with extensions to general risks [31]–[33]. For LLMs, these tools calibrate prediction sets and factuality filters [34]–[36] and let planners ask for help with statistical guarantees on task completion [37]. [38] give a complementary formal treatment of when to quit, casting abstention as an explicit action in a regularized reinforcement-learning objective and deriving value-function conditions under which quitting is optimal; their guarantee is tied to the quality of a learned value estimate, whereas ours is finite-sample and distribution-free, and controls an episode-level error rate across a sequence of gates. Closest in spirit is CALM [39], which calibrates per-token early-exit thresholds so that sequence-level quality constraints hold with high probability; this is the same pattern of composing many local decisions under one global constraint that our cascade instantiates, at a different granularity and for a different risk: rather than skipping decoder layers under a text quality constraint, we terminate entire agent episodes under a constraint on global success recall.

5.0.4 Adaptive allocation of inference compute.↩︎

Reinforcement learning with verifiable rewards (RLVR) is an adjacent source of motivation: modern reasoning systems optimize sampled rollouts from task-level correctness signals. DeepSeekMath introduced Group Relative Policy Optimization (GRPO) for mathematical reasoning [40], and DeepSeek-R1 showed that large-scale RL on verifiable tasks can elicit self-reflection, verification, and strategy adaptation [41]. Our setting uses the same kind of episode-level success signal, but at inference time: rather than updating the policy, we decide whether a running rollout should continue while preserving global success recall.

A complementary literature adapts compute to instance difficulty: adaptive and early-stopping self-consistency truncate sampling once answers agree [42]–[44], cascades route queries from cheap to expensive models [45], optionally abstaining early when the large model is also likely to fail [46], and overthinking mitigation stops chain-of-thought once further reasoning is redundant [47], [48], terminating computation that has already succeeded, the mirror image of our problem. Atropos [49] is closest to ours in goal: a graph network over a semantic flow graph of sampled reasoning paths predicts whether an ongoing self-consistency run will succeed, triggering early termination and a hotswap to a stronger model; the predictor reads generated text rather than internal states and provides no control on the rate of falsely terminated successes. At the systems level, early-exit architectures realize per-token adaptive depth and expose the serving-stack considerations any activation-reading monitor inherits [50], [51]. In multi-step settings, process reward models score partial trajectories to prune search branches [52]–[54], precisely because waiting for complete rollouts is costly. These methods allocate compute across samples, models, or branches, generally without guarantees on the induced loss of task reward; we address the orthogonal single-trajectory decision of whether a running episode is worth finishing at all, with recall control built in, so that the reported compute savings are honest by construction.

6 Discussion and Limitations↩︎

6.0.1 Margin vs.certificate.↩︎

Our default policy controls global recall through an empirically validated margin, not a theorem; the certified variant closes this gap but, at our data scale, only up to targets of \(\approx 0.97\). This mirrors the state of practice in selective prediction: formal episode-level guarantees over sequential gates with searched budgets require either more calibration data or sharper multiple-testing machinery (e.g., fixed-sequence or e-value methods) than the one-shot bound we certify with. We see tightening this loop as the main methodological open problem our results motivate.

6.0.2 Search granularity and margin size.↩︎

The budget grid is deliberately coarse. A finer or continuous search could only improve validation savings, and its failure mode is conservative (an infeasible or suboptimal grid point gives up savings, never recall), so grid resolution trades efficiency, not safety. The margin \(\delta = 0.02\) was fixed a priori and happens to match the binomial standard error of validation recall at our scale, \(\sqrt{\rho^\star (1 - \rho^\star) / n_{\mathrm{pos}}} \approx 0.02\) for \(\rho^\star = 0.95\) and \(n_{\mathrm{pos}} \approx 114\); this gives a data-driven recipe for sizing \(\delta\) in a new domain, with the certificate available whenever a heuristic margin is unacceptable.

6.0.3 Scope.↩︎

Our evidence comes from one environment (TextCraft) with two agent models and \(800\) episodes each; the exchangeability assumption underlying the gates can be violated under distribution shift (e.g., new task types at deployment). Recalibration on a small labeled sample restores the guarantee at modest cost. Gates are placed at the first six rounds: beyond round 6 only a minority of episodes remain alive (Figure 3), per-round AUC recedes toward its round-1 level, and the searched budgets at strict targets already disable rounds 5–6, indicating little headroom further out. Finally, our activations are extracted via an offline teacher-forced re-run; a deployed cascade would read them from the serving stack at generation time. Since the feature is the activation at the final generated token of the round, no extra forward pass is required, but exposing intermediate activations inside optimized serving stacks (fused attention kernels, paged KV caches) is a real engineering task in its own right, as the early-exit inference literature documents [50], [51]. Relatedly, we measure savings in generated tokens; realized wall-clock or dollar savings depend additionally on batching, prefill–decode asymmetry, and whether the freed capacity is actually reused.

6.0.4 What aborted compute buys.↩︎

We report savings, not what to do with them. The natural next step is closing the loop: reallocating aborted compute to retries of the same task turns recall-controlled abort into a test-time-scaling policy whose reward effect can be measured end-to-end.

7 Conclusion↩︎

We showed that eventual failure of an LLM agent episode is predictable from internal activations within the first interaction rounds (before behavior-only monitors become informative) across two agent models, and that a cascade of recall-calibrated gates, with per-round budgets searched under a global constraint, converts this signal into compute savings of up to \(47.1\%\) at a \(90\%\) global success-recall target, \(1.6\)–\(1.7\times\) what any single decision point achieves. The same framework tells practitioners which recall promises their data can certify and which it cannot. Together these suggest a practical, honest route to cutting the inference cost of agentic LLM systems without silently sacrificing task success.

A. Per-Layer Probe AUC Sweeps↩︎

Probe layers were fixed once per model and used unchanged in all experiments. For Qwen-2.5-7B, a per-layer sweep of the same logistic probe over layers \(\{0, 2, \dots, 28\}\) was run on an independent pilot set of rollouts collected before the main experiments; the post-generation AUC peaks at layer 20. For Llama-3.2-3B, the sweep covered layers \(\{6, 10, 14, 18, 22, 26\}\); the post-generation sweep selects layer 14.

B. Conformal-Quantile Gates↩︎

Table 1 compares the Clopper–Pearson gate of the main text against the conformal-quantile variant, for a single probe-based gate at round 1 with per-round target \(0.95\) (20 seeds). The quantile gate controls recall only on average over the calibration draw, so with the small per-round calibration sets available here its realized recall fluctuates below target, violating it for Llama (\(0.933\) vs.\(0.95\)) while saving more compute. The Clopper–Pearson gate is conservative (recall \(0.972\)–\(0.977\) at target \(0.95\)) but never violates. This is the trade the main text accepts: the cascade’s budget search recovers much of the conservatism by spending recall where it is cheap, while retaining a high-confidence per-gate guarantee.

References↩︎