1 Introduction↩︎

The deployment of LLM-based agents with tool-calling capabilities has accelerated rapidly, with production systems now using agents to search databases, send emails, manage calendars, and orchestrate multi-step workflows through structured function calls [1]–[3]. As these systems mature, a reliability question that has received surprisingly little systematic attention comes to the fore: if you run the same agent on the same task twice, do you get the same behavior?

This question has immediate practical consequences:

• Testing. If agents behave differently across runs, unit tests asserting on outputs are flaky by design. Reliable behavioral invariants are needed to test that an agent “does the right thing” [4].

• Debugging. Failure reproducibility is a prerequisite for root-cause analysis. Behavioral variance makes production failures intermittent and difficult to trace.

• Safety and auditability. High-stakes deployments require guarantees that the agent will not take an unexpected action on a re-run (e.g., sending duplicate emails, creating conflicting calendar events) [5].

• Cost optimization. If behavioral variance is predictable from task characteristics, consistency-aware routing can assign tasks to cheaper models when variance is tolerable and reserve high-consistency models for critical workflows.

1.0.0.1 The gap in prior work.

[6] studied behavioral consistency in ReAct-style agents [7] on HotpotQA, finding that agents produce 2.0–4.2 distinct action sequences per 10 runs and that inconsistency predicts failure. This is important but limited to search-only actions in a question-answering setting. Real-world agents operate over structured tool-calling interfaces with typed parameters, multiple heterogeneous tools, and multi-step pipelines. The distinction matters: tool calls are discrete typed objects (not free-form text), have observable side effects (emails sent, events created), and compose in sequences where early divergence propagates through subsequent steps. It is unclear whether consistency patterns from ReAct agents transfer to this richer action space.

1.0.0.2 This work.

We extend the study of behavioral consistency to multi-step tool-calling agents across diverse task types, making the following contributions:

1. Benchmark. 19 tasks spanning five categories, paired with 10 deterministic simulated tools that isolate LLM variance from environmental non-determinism (Section 4.1).

2. Formal metric framework. Formal definitions of Tool Sequence Similarity (\(\mathrm{TSS}\)), Argument Consistency (\(\mathrm{AC}\)), divergence point, and output agreement, targeting distinct behavioral layers (Section 3.2).

3. The structural/parametric distinction. \(\mathrm{TSS}= 0.87\) substantially exceeds \(\mathrm{AC}= 0.69\) across all models and categories (\(d = 0.75\), \(p < 10^{-13}\)); this is a novel finding not captured by prior single-metric consistency studies (Section 5.1).

4. Correctness validation. \(\mathrm{TSS}\) predicts task correctness (\(d = 0.81\), \(p<0.001\)) while \(\mathrm{AC}\) does not (\(r = 0.12\), n.s.), ruling out trivial “consistently wrong” explanations and establishing \(\mathrm{TSS}\) as a reliability proxy (Section 5.5).

5. Actionable guidelines grounded in effect sizes for testing, monitoring, and model selection in production deployments (Section 6.3).

2 Related Work↩︎

2.0.0.1 Behavioral consistency in LLM agents.

[6] measured consistency in ReAct agents on HotpotQA using search-only actions, finding inconsistency predicts failure. We extend to typed tool calls with diverse tool sets and task types, and introduce the structural/parametric distinction absent from prior work. [8] showed that sampling multiple reasoning chains and marginalizing improves accuracy; this leverages variance rather than characterizing its structure, and focuses on reasoning rather than tool calls. [9] studied self-reflection in LLM agents and found mixed effects, suggesting that deliberate consistency-improving strategies do not uniformly help.

2.0.0.2 LLM reliability and robustness.

[10] quantified LLM sensitivity to prompt formatting, finding large swings from superficial changes. [11] demonstrated order sensitivity in few-shot prompts. [12] showed that standard NLP evaluations underestimate robustness failures. Our focus is orthogonal: we hold the input fixed and measure variance across repeated identical invocations—the consistency a deployed system must have by definition.

2.0.0.3 Agent capability evaluation.

AgentBench [13], ToolBench [1], API-Bank [2], and Gorilla [14] evaluate whether agents can solve tasks. [3] showed LLMs can learn tool use from self-generated demonstrations. [4] surveyed pitfalls in AI agent evaluations, noting that variance across runs is rarely reported. We evaluate not whether agents succeed but whether they consistently succeed in the same way—an orthogonal, understudied dimension.

2.0.0.4 Reliability in ML systems.

The broader ML reliability literature addresses distribution shift, uncertainty quantification [15], and out-of-distribution detection—all concerned with performance under changed inputs. We study consistency under repeated identical inputs, a related but distinct concern that matters specifically for agentic systems where tasks are retried and behavioral reproducibility is a correctness criterion in its own right.

3 A Framework for Agent Behavioral Consistency↩︎

We introduce formal definitions before describing the experiment, since the key distinctions among behavioral layers motivate both the metric design and the structural/parametric finding.

3.1 Agent Execution Model↩︎

This framing reveals a natural hierarchy of behavioral layers:

1. Structural layer: the sequence of tool names \((\mathtt{name}_1, \ldots, \mathtt{name}_k)\)—the agent’s procedural choice.

2. Argument layer: the argument maps \(\mathbf{a}_i\) at each step—how the procedure is parameterized.

3. Output layer: the final response \(r\)—the surface-level text the user sees.

Our central hypothesis, motivated by how LLMs acquire tool-use behavior through fine-tuning:

The intuition is that RLHF and SFT fine-tuning on tool-use data reinforces correct procedure selection—which carries a cleaner training signal—while argument instantiation remains more sensitive to sampling-time variation. Figure 1 illustrates the pattern concretely.

3.2 Formal Metric Definitions↩︎

Additional metrics: Unique Sequences counts distinct tool-name sequences across \(N\) runs; Divergence Point is the mean step at which a trace pair first differs; Output Agreement is the exact-match rate of final responses.

4 Methodology↩︎

4.1 Task Benchmark↩︎

We design 19 tasks across five categories of increasing ambiguity (Table [tab:full95tasks], Appendix [app:tasks]):

• Data Retrieval (4 tasks): Contact lookup, email search, aggregation. Clear instructions with deterministic correct tool sequences.

• Scheduling (4 tasks): Calendar creation, free-slot finding, conflict detection. Require temporal reasoning with well-defined procedures.

• Computation (3 tasks): Inventory valuation, revenue projection. Require numeric tool calls with specific arguments.

• Multi-Tool Composition (4 tasks): Chains of 3–5 different tools (e.g., “find email \(\to\) look up sender \(\to\) schedule meeting”).

• Ambiguous (4 tasks): Intentionally underspecified requests where multiple valid strategies exist (e.g., “Help me prepare for my meetings tomorrow”).

Task difficulty is assigned by expected tool-call count: easy (1–2 calls), medium (2–3), hard (3+). We validate this: hard tasks require significantly more tool calls (mean \(3.0 \pm 2.4\)) than easy tasks (\(1.9 \pm 0.7\); Spearman \(\rho = 0.26\), \(p = 0.004\)).

4.2 Tool Environment↩︎

We implement 10 deterministic simulated tools spanning 7 domains (contacts, calendar, email, products, weather, calculations, notes; see Appendix [app:tools]). All tools are deterministic: identical inputs always produce identical outputs. This design isolates LLM variance from environmental non-determinism, ensuring observed behavioral differences arise solely from the model’s generation process. Tool schemas follow the OpenAI function-calling format and are adapted to provider-native formats at runtime.

4.3 Agent Framework↩︎

Each run follows a standard tool-calling loop: (1) the model receives a system prompt¹ and the user task; (2) the model responds with structured tool calls; (3) tools execute deterministically; (4) the model may call additional tools or produce a final response; (5) the loop continues for up to 10 iterations. We use each provider’s native tool-calling API.

Temperature is set to 1.0 (the default for most providers). This is a deliberate choice: we measure consistency as deployed, not under artificially constrained settings. Temperature ablation is a known limitation and an important direction for future work (Section 6.5).

4.4 Models↩︎

We evaluate six models spanning three providers and multiple capability tiers: OpenAI: GPT-4o-mini, GPT-4o, GPT-4.1-mini, GPT-4.1; Anthropic: Claude Sonnet 4; Meta/Together AI: Llama 3.3 70B Instruct Turbo. This covers flagship models (GPT-4.1, Sonnet 4), cost-optimized variants (GPT-4o-mini, GPT-4.1-mini), and an open-source model (Llama 3.3). Each model runs each of the 19 tasks 10 times, yielding 1,140 agent traces.²

4.5 Correctness Evaluation↩︎

To validate that consistency predicts meaningful outcomes, we retrospectively score all 1,140 traces using a three-component rubric:

1. Required tool coverage: did the agent invoke all tools necessary for the task?

2. Argument validity: do key arguments match expected patterns (e.g., send_email.to \(\sim\) /alice@exampleċom/, create_calendar_event.date = 2026-03-02)?

3. Output completeness: does the final response address the user’s request (regex-matched against expected output patterns)?

A trace is scored correct iff all applicable criteria are satisfied. Full per-task criteria are in Appendix [app:correctness] and the code release.

4.6 Statistical Analysis↩︎

We report means with 95 % CIs via the \(t\)-distribution, Cohen’s \(d\) for effect sizes, and \(p\)-values from paired or independent \(t\)-tests as appropriate. Cross-model comparisons use one-way ANOVA with \(\eta^2\). We acknowledge that a linear mixed-effects model (treating tasks as random effects) would be more appropriate given the shared task structure across models; the simple ANOVA is conservative but adequate given our sample sizes. We test five primary hypotheses; all survive Bonferroni correction at \(\alpha = 0.01\). Split-half reliability (TSS: \(r = 0.66\), \(p < 10^{-16}\), \(n = 125\)) confirms moderate metric stability at \(N = 10\).

5 Results↩︎

5.1 Structural Consistency with Parametric Variance↩︎

Hypothesis 1 is confirmed. Agents exhibit structural consistency with parametric variance: mean \(\mathrm{TSS}= 0.87\), 95 % CI \([0.84, 0.90]\), versus mean \(\mathrm{AC}= 0.69\), \([0.64, 0.74]\) (paired \(t\)-test: \(t = 8.41\), \(p < 10^{-13}\); Cohen’s \(d = 0.75\)). The pattern holds across all models and categories (Table 1, Figure 2).

5.2 Ambiguity Is the Dominant Driver of Inconsistency↩︎

Task category strongly affects consistency (Table 2, Figure 3). Ambiguous tasks have substantially lower AC (\(0.52\) vs.\(0.72\) for structured; Cohen’s \(d = 0.74\), \(t = 3.34\), \(p = 0.001\)) and lower TSS (\(0.79\) vs.\(0.89\); \(d = 0.58\), \(p = 0.010\)). The ambiguity effect on AC (\(d = 0.74\)) exceeds the between-model effect (\(\eta^2 = 0.08\), n.s.), establishing that task specification quality is a stronger lever on consistency than model selection.

5.3 Divergence Concentrates in Early Steps↩︎

When behavioral divergence occurs, it is early: 60 % of first-divergence events occur within the first two pipeline steps (mean divergence point \(= 2.2\); Figure 4). This yields a practical monitoring strategy: comparing only the first 1–2 tool calls against a reference trace catches the majority of behavioral variance without full trace inspection.

5.4 Output Text Is Not a Reliability Signal↩︎

Final natural language responses exhibit near-zero exact-match rates (\(<\)​5 % overall) even when the underlying tool-calling sequences are identical. This is expected—language generation is inherently variable—but it has a critical implication for testing: agent tests must assert on structured tool-calling behavior, not natural language output. Asserting on text is analogous to writing software tests that check log messages rather than return values.

5.5 Consistency Predicts Correctness↩︎

We score all 1,140 traces using the rubric in Section 4.5. Overall correctness is 77.1 %. More importantly, \(\mathrm{TSS}\) is a significant predictor: Pearson \(r = 0.32\) (\(p = 0.005\)), Spearman \(\rho = 0.42\) (\(p < 0.001\)). A median split reveals that high-\(\mathrm{TSS}\) conditions (\(\mathrm{TSS}\geq 0.90\)) achieve 90.2 % correctness versus 61.2 % for low-\(\mathrm{TSS}\) (Cohen’s \(d = 0.81\), \(p < 0.001\); Figure 5).

Crucially, \(\mathrm{AC}\) does not predict correctness (\(r = 0.12\), \(p = 0.31\), n.s.). Agents may phrase search queries differently, format dates differently, or vary message bodies across runs—none of this impairs task success. It is structural variance—selecting different or missing tools—where failures concentrate.

5.6 Cross-Model Differences↩︎

Models differ significantly in structural consistency (ANOVA on \(\mathrm{TSS}\): \(F = 3.52\), \(p = 0.003\), \(\eta^2 = 0.15\)) but not in argument consistency (\(F = 1.61\), \(p = 0.15\), \(\eta^2 = 0.08\), n.s.), indicating that AC variation is dominated by task-level factors.

GPT-4.1-mini achieves the highest \(\mathrm{TSS}\) (0.92) and \(\mathrm{AC}\) (0.81). Llama 3.3 70B is significantly below all proprietary models in \(\mathrm{TSS}\) (0.71, \([0.61, 0.82]\)), with 3.3 unique sequences per task versus 1.6–2.2 for proprietary models (post-hoc \(p < 0.05\) vs.all others). Claude Sonnet 4 achieves the second-highest \(\mathrm{AC}\) (0.76) while exploring more unique sequences (2.2), suggesting it reliably parameterizes strategies but varies in which strategy it selects.

The Model \(\times\) Category interaction (Figure 6) is notable: Llama 3.3 70B drops most sharply on ambiguous (\(\mathrm{TSS}= 0.54\)) and composition (\(\mathrm{TSS}= 0.63\)) tasks, while proprietary models maintain \(\mathrm{TSS}> 0.80\) even on ambiguous tasks.

6 Discussion↩︎

6.1 Why Structural Consistency Exceeds Argument Consistency↩︎

The structural/parametric pattern is consistent across all models and task types, suggesting a systematic cause rooted in how LLMs acquire tool-use behavior. Training corpora for tool-calling likely contain many demonstrations of the same high-level task type solved with the same tool sequence, but with varied argument values (different names, dates, query strings) across instances. RLHF/SFT reinforces correct procedure selection—which carries a cleaner correctness signal—while argument instantiation remains more sensitive to sampling-time variation. This “procedural schema” interpretation is directly supported by our correctness analysis: argument variance (\(\mathrm{AC}\)) does not predict failure (\(r = 0.12\), n.s.), meaning the variations are largely semantically equivalent (e.g., “Alice” vs.”alice” in a case-insensitive lookup). Structural variance (\(\mathrm{TSS}\)) predicts failure strongly (\(d = 0.81\)), because missing or wrong tools are not semantically equivalent to correct ones.

6.2 TSS as a Practical Reliability Proxy↩︎

The practical value of \(\mathrm{TSS}\) as a reliability proxy rests on three properties established here: (1) it predicts correctness (\(d = 0.81\)) without requiring correctness labels; (2) it can be estimated at deployment time by comparing a run’s tool sequence against a reference; and (3) the early-divergence result (60 % in steps 1–2) means monitoring only the first two tool calls suffices.

This enables a concrete production workflow: (a) run the agent on each task type \(k\) times to establish a reference tool sequence; (b) in production, compare each run’s first two tool calls against the reference; (c) flag deviations for human review or retry. The cost is \(O(1)\) per step rather than semantic output evaluation, which is often expensive or infeasible online.

6.3 Implications for Agent Deployment↩︎

1. Reduce ambiguity before switching models. The ambiguity effect on consistency (\(d = 0.74\)) outweighs the between-model effect (\(\eta^2 = 0.08\), n.s.for \(\mathrm{AC}\)). Clarifying task specifications yields more consistency gain than upgrading the model.

2. Monitor early steps. 60 % of divergence occurs in steps 1–2; a lightweight check on the first two tool calls suffices to flag most inconsistent runs.

3. Test tool calls, not text. Assert on tool names and argument patterns—not natural language output, which varies below 5 % exact match even in consistent runs.

4. Use \(\mathrm{TSS}\) as an online reliability signal. A run whose tool sequence deviates from a reference trace is nearly three times more likely to fail; no correctness labels are needed to compute this signal.

5. Match model to consistency requirement. GPT-4.1-mini offers the best structural consistency for consistency-critical workflows. Llama 3.3 70B introduces substantially higher behavioral diversity and may require additional guardrails.

6.4 Relation to Prior Work↩︎

[6] reported 2.0–4.2 unique action sequences per 10 runs on HotpotQA ReAct agents; we observe 1.6–3.3 unique tool-call sequences, broadly consistent. Our finer-grained analysis reveals that the unique-sequence count masks the structural/parametric distinction: even conditions with \(>\)​2 unique sequences often share high \(\mathrm{TSS}\) because sequences agree on most tool names with only minor reordering. Prior consistency studies also did not establish a connection to correctness; our correctness analysis (\(d = 0.81\)) grounds the metric in task outcomes.

6.5 Limitations↩︎

• Sample size. \(N = 10\) runs per cell yields split-half reliability \(r = 0.66\). Aggregate results are well-powered (1,140 traces), but per-cell CIs are wide. Future work should use \(N \geq 30\).

• Single temperature. We measure at \(T = 1.0\) only. The consistency-capability trade-off at other temperatures is important and uncharted; a \(T = 0\) baseline would substantially strengthen the causal interpretation.

• Structured correctness proxy. Our rubric is not a full semantic evaluation. Human judges on a random 200-trace subset would provide stronger validity. The significant rank correlation (\(\rho = 0.42\), \(p < 0.001\)) provides initial support.

• Limited tasks per category. 3–4 tasks per category makes category-level estimates imprecise. \(\geq\)​10 tasks per category are needed for reliable inference.

• Single system prompt. Consistency may vary substantially with chain-of-thought prompting, few-shot examples, or persona instructions.

• Simulated tools. Deterministic tools isolate LLM variance but may overestimate real-world consistency where tool outputs themselves vary.

• Provider coverage. Google Gemini and Mistral are absent, limiting generalizability.

• AC alignment. AC aligns arguments by step index, conflating structural and argument divergence at misaligned steps. A disentangled metric would be more informative.

6.6 Future Directions↩︎

1. Temperature ablation: map the full consistency-capability Pareto frontier.

2. Larger benchmark: \(\geq\)​10 tasks per category, \(\geq\)​50 total.

3. Human correctness evaluation: human judges on 200 traces.

4. Consistency-aware routing: route tasks to models based on predicted \(\mathrm{TSS}\).

5. Multi-turn and long-horizon agents: web agents, coding agents.

6. Fine-tuning effects: does RLHF explicitly optimize consistency?

7. Real API tools: study whether environmental non-determinism amplifies or dampens LLM behavioral variance.

7 Conclusion↩︎

We have presented a systematic empirical study of behavioral consistency in multi-step tool-calling LLM agents. Across six models, 19 tasks, and 1,140 agent traces, we identify a “structural consistency, parametric variance” pattern that is large (\(d = 0.75\)) and highly significant (\(p < 10^{-13}\)): agents reliably select the same tools in the same order but vary in argument details. Critically, only the structural layer predicts task success (\(d = 0.81\), \(p < 0.001\)), while argument variance is benign (\(r = 0.12\), n.s.)—validating the pattern and ruling out trivial consistency. Task specification quality outweighs model selection as a consistency lever, divergence concentrates early in the pipeline, and \(\mathrm{TSS}\) serves as a lightweight, correctness-free reliability proxy.

These findings suggest that LLMs acquire robust procedural schemas for tool use through training, while argument instantiation remains sensitive to sampling variation. Understanding and exploiting this structure—testing on tool calls not text, monitoring early steps, flagging low-\(\mathrm{TSS}\) runs—is a practical path toward more reliable agentic systems.

Reproducibility↩︎

All code, benchmark definitions, tool implementations, raw traces, and analysis scripts are available at https://github.com/Abelo9996/agent-consistency. The full system prompt, task list, and per-task correctness criteria are in Appendices [app:tasks]–[app:correctness].

Appendix A. Full Task Benchmark↩︎

Table [tab:full95tasks] lists all 19 benchmark tasks used in the study. Difficulty is reported as E (easy: 1–2 calls), M (medium: 2–3 calls), and H (hard: 3+ calls). Expected tools denote the minimal correct solution pattern.

Appendix B. Tool Descriptions↩︎

Table 3 summarizes the 10 deterministic simulated tools used in our environment. Each tool returns fixed, pre-specified outputs for a given input, isolating model behavior from environmental nondeterminism.

Appendix C. System Prompt↩︎

The following system prompt was used verbatim for all models and all runs:

You are a helpful assistant with access to a set of tools. Use the provided tools to complete the user’s request as accurately and efficiently as possible. Only call tools when necessary. If you have all the information you need to respond, do so directly. When you have completed the task, provide a concise summary of what you did and the results.

This prompt is intentionally minimal so that we do not anchor models to a particular tool-selection strategy and thereby artificially inflate measured consistency.

Appendix D. Correctness Criteria (Illustrative Subset)↩︎

Table 4 provides illustrative examples of the correctness rubric used to score traces. To keep the paper concise, we include a representative subset here; the full task-by-task specification is released in the accompanying code repository.

References↩︎