cybersecurity benchmarks, AI agents, security operations centers, Cybench, BOTS, cost accounting

1 Introduction↩︎

Security-agent evaluations often ask how far a model can go when it is given a generous inference budget and an offensive task: solve a CTF, find a vulnerability, write an exploit, or automate a penetration-testing step. Recent Claude Mythos Preview evaluations exemplify this regime, combining CTF suites, multi-step cyber ranges, and vulnerability-discovery/exploit-development evaluations with million-token budgets and attacker-directed execution access [1], [2]. Those measurements are useful because they track capabilities that attackers can use. They are incomplete because operational security is not a peak-capability exercise. A security operations center (SOC) needs agents that can navigate telemetry, choose searches, decide when enrichment is worth paying for, and return analyst-grade answers under a budget.

We therefore make cost the common axis for comparing offensive and defensive security agents. A fixed budget turns success rate into an operational question: how much capability does a model buy per dollar of inference and tooling? This framing also separates two costs that are often collapsed together. For offensive CTFs, most marginal spend is test-time reasoning and command execution. For SOC investigation, the model also spends analyst-facing resources: Splunk queries, event inspection, web search, WHOIS history, and other enrichment calls.

We evaluate this cost-success tradeoff on offensive Cybench challenges [3] and defensive Splunk Boss of the SOC (BOTS) v1 investigations [4] in a common agent-evaluation harness. Cybench provides sandboxed CTF-style tasks with shell and Python access. BOTS v1 provides public Splunk telemetry and official competition questions with point values and hint penalties. Rather than presenting a single leaderboard, we characterize operating points: model, task family, cost limit, tools, and scoring rule.

We find distinct scaling regimes. On Cybench, retrospective budget caps show clear headroom for Claude Opus 4.8 and DeepSeek v4 Flash, while GPT-5.5 already solves most successful challenges below the lower cap. On BOTS v1, extra budget is less predictive: the direct DeepSeek v4 Flash cap increase produces little gain, and high tool volume often accompanies lower scores. The defensive signal is therefore not raw reasoning budget alone; it is disciplined tool use, telemetry navigation, and selective enrichment.

Our contributions are as follows:

• A cost-success comparison of shared Cybench and BOTS v1 model operating points using a common harness;

• A paired budget-headroom analysis showing where additional per-sample spend changes observed scores;

• Model-token plus priced-tool cost accounting for SOC investigation, including cost per 1,000 BOTS points;

• BOTS v1 no-tools controls showing that public SOC benchmarks require decontamination checks before absolute scores are interpreted as live investigation skill.

2 Related Work↩︎

Cybersecurity model evaluation spans knowledge tests, offensive agent tasks, defensive investigations, and benchmark-integrity work. Broad suites such as CyberSecEval 2 measure security risks and capabilities including prompt injection, code-interpreter abuse, false refusals, and exploit-generation tasks [5]; CyberSecEval 3 extends that line toward automated social engineering and autonomous offensive operations [6]. Cybersecurity-knowledge benchmarks such as SecBench scale up multi-domain question answering and short-answer grading [7]. These resources are useful for measuring domain knowledge and misuse risk, but they usually do not require a full investigation loop with changing evidence, paid tools, and explicit budget tradeoffs.

Offensive agent benchmarks add interaction and execution. InterCode standardizes code tasks with execution feedback [8]; Language Agents as Hackers introduced InterCode-CTF for capture-the-flag tasks [9]; NYU CTF Bench broadens open CTF coverage with automated tool use [10]; and Cybench uses professional CTF tasks with sandboxed command execution [3]. CVE-Bench moves toward real-world web-application exploitation rather than abstract CTFs [11], while PentestGPT evaluates LLM support across penetration-testing subtasks [12]. Recent InterCode-CTF results show that prompting, tool use, and multiple attempts can substantially change observed offensive performance [13]; EnIGMA further shows that interactive debugger and connection tools can substantially assist vulnerability-finding agents [14]. BountyBench broadens the unit of evaluation to detect, exploit, and patch tasks on real-world codebases with dollar-valued bug bounties [15]. We follow that agentic line, but evaluate cost-success operating points rather than peak success rate alone.

SOC evaluation draws on a different body of practice. Incident-response guidance and cybersecurity workforce taxonomies emphasize detection, analysis, evidence handling, response coordination, and operational judgment [16], [17]. Splunk’s BOTS datasets provide public security telemetry, questions, answers, and scoring artifacts for blue-team investigation exercises [4], [18], [19]. An analyst must discover log schemas, write and revise SPL, inspect representative events, correlate artifacts, and decide whether external enrichment will change the case. A flag-only score or a raw tool-call count cannot distinguish those behaviors.

Recent defensive-agent benchmarks are beginning to target this gap. SIABench frames security incident analysis as alert triage plus deeper multi-step workflows across malware, phishing, forensic, and log-analysis scenarios [20]. ExCyTIn-Bench derives cyber-threat-investigation questions from graph-linked security logs [21]; CyberTeam models blue-team threat hunting as embodied function-calling workflows [22]; and Cyber Defense Benchmark evaluates open-ended threat hunting over raw Windows event logs [23]. Our BOTS v1 study complements this emerging line by preserving official BOTS scoring while adding fixed-budget comparisons, model-token cost, priced external-tool spend, and tool-efficiency views.

Public benchmarks also require leakage controls. Benchmark-contamination studies show that public test sets can overlap with model-training data or be partially recoverable from model behavior, inflating apparent capability and complicating fair comparisons [24]–[26]. Practical mitigation work recommends protected test releases, training-exclusion controls, private or retroactive holdouts, and care with API-mediated leakage [27]–[29]. Because BOTS v1 is public and old, Section 5 treats no-tools and no-context probes as contamination controls rather than as primary capability results.

3 Evaluation Design↩︎

We use the Inspect evaluation framework [30] to organize the experiments. We use a ReAct-style agent with auto-compaction [31] for all benchmark runs. We auto-compacted when the agent context reached 90% of the model context window. Auto-compaction used Inspect AI’s automatic compaction strategy, which attempts provider-native compaction and falls back to summary-based compaction [32]. Between offensive and defensive tasks, we only change the suite of available tools. We rely on Inspect to keep track of auxiliary metrics like cost, token consumption, and tool calling statistics. We set budgets on the cost as the main resource control on agent runs. The agents are unaware of cost or token budgets but are aware of limits on certain expensive tool calls.

This design allows us to ascertain the model’s success rate at various cost and token consumption caps. We can derive practical metrics such as cost per solved question. By varying the cost between 0 and the run limit, we can track how success rate scales with the cost and token budget increases.

For models provided by OpenAI and Anthropic, we use their respective APIs. Inspect also relies on frontier providers for autocompaction, prompt caching, and cost tracking metrics. For other models, we use OpenRouter to select one primary provider and two fallback providers. We use average pricing across the three providers to calculate the cost of runs. Since OpenRouter delegates prompt caching to providers, having one primary provider enables a maximally efficient caching scheme without sacrificing operational stability. Operationally, most OpenRouter model calls record some prompt-cache reads, though cache-miss rates vary by run.

3.1 Offensive benchmark: Cybench↩︎

Cybench is an offensive CTF benchmark with 39 challenges in the full set, drawn from four cybersecurity competitions (including HackTheBox, Glacier, SEKAI, and HKCert) and spanning domains such as cryptography, web exploitation, reverse engineering, forensics, and exploitation [33]. Each sample requires an agent to solve a sandboxed offensive security challenge (web/app exploitation, binary/pwn, reversing, crypto, web/forensic workflows, and other hands-on tasks) and submit a hidden challenge flag token (typically brace-delimited strings such as HTB{…}, gctf{…}, or SEKAI{…}). Submitted answers are judged with a case-insensitive substring match against the reference flag; each challenge allows up to 3 submission attempts and we score by averaging binary correctness across three independent epochs (per-sample mean correctness, then averaged across all challenges).

We use the Cybench hard variant from Inspect Evals [33] to perform the evaluations. We expose only bash and python tools, along with a submit tool for final answer submission. All tools are considered to be zero-cost. We limited the number of submission attempts to 3.

3.2 Defensive benchmark: BOTS v1↩︎

BOTS v1 is a public Splunk incident-investigation dataset and CTF platform for information security professionals, researchers, students, and enthusiasts [4]. We did not identify a public Inspect implementation of BOTS v1, so we implemented one. Unlike a flat set of independent binary questions, the official BOTS format is a sequential, point-based investigation in which later questions can depend on earlier findings. BOTS v1 contains a warm-up plus 31 scored questions covering the Po1s0n1vy web-defacement investigation and Cerber ransomware at Wayne Enterprises. Each scored question has official base points and, where applicable, ordered hints with point costs. After an incorrect submission the next hint is revealed; a later correct answer earns base points minus consumed hint costs, while an unanswered or still-incorrect question earns zero.

We exclude the official warm-up question, leaving 31 scored questions worth 10,300 official competition points. Agents receive Splunk discovery, search, and event-inspection tools over Splunk Search Processing Language [34], plus limited VirusTotal, WHOIS, DNS, Brave Search, bash, and Python tools [35]–[39]. We make Splunk telemetry the primary evidence source in the prompt and ask agents to use external enrichment selectively. To preserve cost efficiency, we limit VirusTotal and WhoisXMLAPI calls to 3 while limiting web search to 5 calls. We implement strict keyword filtering to prevent competition writeups and solution leakage.

We use bots_points as the primary metric: a correct answer earns official points minus hint penalties; an incorrect answer earns zero. Hints are revealed with every unsuccessful answer. We report binary includes accuracy as a secondary view. We use three epochs and mean reduction, so point totals are reduced back to the single-epoch 10,300-point scale. We provide prerequisite case context from earlier official questions for 23 of the 31 scored questions, matching the sequential investigation structure but making the task closer to follow-up analysis than cold-start incident reconstruction.

3.3 Refusal accounting↩︎

We classify refusals at the sample-epoch level using both agent text and provider metadata. During evaluation, an assistant turn is treated as a refusal when it contains nonblank text, makes no tool call, and matches a case-insensitive set of apology, inability, or policy phrases after Unicode normalization, or when the provider returns content_filter. Both benchmark agents use a refusal limit of one: the first detection that reaches the agent continuation hook aborts that sample, which is then scored on its current output. This text detector is a heuristic rather than a provider policy label.

For reporting, let \(N\) be the number of loaded scored sample-epochs and \(R\) the number classified as refusals. Cybench counts only failed sample-epochs whose final assistant turn matches the text detector or whose model-event log contains a content filter; it reports refusal rate \(100R/N\), or \(R/3\) challenge-equivalents for three-epoch runs. Raw Cybench pass@1 is unchanged, so refused epochs remain failures. BOTS v1 instead flags any matching assistant turn or content-filter event in the transcript and reports \(R/N\) as secondary context without changing official points or binary accuracy. A BOTS v1 refusal event can therefore coexist with points if a later or retried attempt answers correctly.

3.4 Cost accounting↩︎

Table ¿tbl:tab:cost-pricing? summarizes the pricing assumptions used in the reported dollar totals.

In order to manage overall costs, we cap the cost for each sample run. We set the cap via Inspect’s budget option – this limits the runtime of a benchmark sample evaluation to a certain cost and aborts the agent if it overruns the budget. We treat such examples as failed. Most successful models finish well below the cost budget cap.

We decompose cost into inference spend and priced-tool spend. For Cybench, reported cost is model-token cost recorded by the evaluation harness or reconstructed from token counts and registered rates where needed. For BOTS, we start with model-token cost and add priced external enrichment calls recorded during the run. We charge Brave Search at $0.005 per request, WhoisXMLAPI WHOIS History preview at $0.0258 per request, and WhoisXMLAPI WHOIS History purchase at $1.29 per request. We treat VirusTotal public API, DNS, and live WHOIS/RDAP as zero marginal cost. Reported dollars are therefore model plus priced-enrichment costs only; they exclude Kubernetes, Splunk infrastructure, storage, and analyst review time. We exclude costs for bash, python, and the three Splunk tool types. We use a host-side cache for WhoisXMLAPI history responses to avoid repeated provider requests during evaluation; cached WhoisXMLAPI calls are still assigned their corresponding dollar value in reported tool-cost metrics so costs reflect expected tool use rather than cache state.

4 Evaluation Results↩︎

We report results for GPT-5.5; GPT-5.6 Luna, Terra, and Sol; Claude Opus 4.8 and Fable 5; and DeepSeek v4 Flash and Pro. The GPT-5.6 and Fable rows use high reasoning effort. For BOTS v1 we additionally retain GPT-5.5 at high reasoning effort, the closest older comparison to Claude Opus 4.8, whose default setting is high effort.

4.1 Offensive CTFs↩︎

Table 1 summarizes Cybench hard-variant results for the shared model set. GPT-5.5 remains the strongest operating point we observe, solving 94.1% of challenges at $1.16 per solved-equivalent challenge. Among GPT-5.6 variants, Luna performs best at 79.5%, followed by Terra at 65.8%; Sol reaches 9.4% after refusing 90.6% of sample-epochs. Fable refuses all 117 sample-epochs before any tool call, so its 0% result is a policy-filter outcome rather than evidence of zero underlying capability. DeepSeek v4 Flash provides the clearest illustration of cost scaling: replaying the same trace under a retrospective $0.80 cap yields 76.1% success, which rises to 86.4% when the full $2.10 budget is allowed.

4.2 SOC investigations↩︎

Table 2 reports BOTS v1 full-agent results for the shared model set, with priced external-tool spend added to model-token spend. Claude Opus 4.8 retains the highest defensive score at 9,666.7 of 10,300 points (93.9%) and the best cost efficiency at $2.98 per 1,000 points. GPT-5.6 Terra and Sol follow at 92.1% and 91.4%; Fable reaches 88.4% with the fewest non-submit tool calls, and Luna reaches 83.7%. All four exceed the standard GPT-5.5 row, but the public-benchmark contamination caveat below still applies. DeepSeek v4 Flash supplies the clearest weak budget effect: doubling its cap from $2.10 to $4.20 moves its score from 73.0% to only 73.9%, while tool calls climb from 4,450 to 4,938.

Figure 2 shows why performance and refusal context should be read together. BOTS v1 refusal counts remain small and do not alter earned points; Fable, for example, earns 88.4% despite refusal events in 5 of 93 sample-epochs. On Cybench, policy gating instead dominates the lower GPT-5.6 and Fable results: Terra has 39 failed refusal outcomes, Sol 106, and Fable all 117.

These full-agent scores should be interpreted alongside the contamination controls in Section 5, not as a clean public leaderboard.

5 BOTS v1 Contamination Controls↩︎

BOTS v1 is valuable because it supplies realistic Splunk data and official scoring, however it is also public and old. We therefore include a no-tools probe as a contamination control. The no-tools condition provides no Splunk, web, bash, Python, or other tools; it asks the model to answer from the question text, optional prerequisite context, and prior knowledge only.

We find the results too high to treat absolute BOTS v1 scores as clean evidence of live investigation skill. With prerequisite context enabled, Claude Opus 4.8 scores 7,700/10,300 points (74.8%) and GPT-5.5 scores 6,400/10,300 (62.1%) with zero tool events. With prerequisite context disabled, Claude still scores 5,150 points (50.0%) and GPT-5.5 scores 5,650 points (54.9%). These controls do not make BOTS useless; they make decontamination controls mandatory. No-tools, no-context, perturbed-question, answer-before-query, private-holdout, or fresh-incident checks should accompany any public SOC benchmark before absolute scores are interpreted as capability evidence. BOTS remains useful as a reproducible harness and cost-accounting testbed; it should not be used as a clean leaderboard by itself.

The completed no-tools probes contain zero tool events. Table 3 includes only completed probes.

6 Scaling Findings↩︎

Budget-scaling claims are most credible when the comparison is paired: the same model, the same benchmark, and the same completed trace, with a lower budget imposed retrospectively. We construct such comparisons by replaying each completed trace under a $0.80 per-sample cap. For Cybench, the cap follows model-call ledger events and marks a sample-epoch as failed once cumulative spend crosses $0.80. For BOTS v1, the cap applies to model-token plus priced-tool spend and zeroes a sample-epoch’s points once total sample cost exceeds $0.80. These are descriptive replay analyses rather than new benchmark runs, but they avoid the confound of comparing unrelated models or unrelated traces. Figures 4 and 5 show the cost and token resource curves. The Cybench panels also retain two additional previously audited runs, Claude Opus 4.7 and Kimi K2.6, which we show for context but do not analyze further. Cybench success rises with both per-sample dollar budget and cumulative token spend. BOTS v1 has a different scaling shape: Claude Opus 4.8 reaches the highest score at lower cost and token volume, while higher-spend runs do not necessarily win. Figure 6 separately compares tool-call scaling across the two benchmarks.

On Cybench, budget sensitivity is real but far from uniform. Claude Opus 4.8 gains 18.8 percentage points when allowed to run past the $0.80 cap, and DeepSeek v4 Flash gains 10.3 points; both bootstrap intervals are clearly positive. GPT-5.5 gains only 2.6 points, because most of the challenges it solves already complete below $0.80. The near-zero Cybench delta for DeepSeek v4 Pro should not be read as evidence that the model cannot scale: the run was capped at a low prospective limit, so its observed trace offers little high-budget headroom to replay.

BOTS v1 shows a different shape. Claude Opus 4.8 captures most of its points below $0.80, and does so with only 603 non-submit tool calls over the full run. GPT-5.5 retains some retrospective headroom above $0.80, yet still trails Claude Opus 4.8 while spending almost twice as much and making more than twice as many tool calls; the high-effort GPT-5.5 row does not materially change this picture. DeepSeek v4 Flash offers the cleanest direct budget check, since its two rows are prospective runs at different caps: the $4.20-cap row exceeds the $2.10-cap row by only 0.9 percentage points, with a paired bootstrap interval of \(-4.9\) to \(+6.9\) percentage points. In this run, additional budget and tool volume did not buy a meaningful defensive improvement.

The practical lesson is not that spend never helps SOC agents; it is that the marginal dollar buys different things in the two workloads. On Cybench, extra budget typically buys longer sandbox exploration, more command execution, and more chances to recover from false starts. On BOTS v1, it often buys additional Splunk or enrichment calls after the agent has already missed the key field, prerequisite, or validation step. Defensive evaluation should therefore scale not only dollars or tokens but also evidence quality: whether the agent finds the right telemetry, validates its answer, and pays for enrichment only when it changes the investigation.

7 Limitations↩︎

Our experiments are observational rather than a prospectively randomized run matrix. We analyze both prospectively fixed cost caps and retrospective caps applied to completed traces. We therefore emphasize fixed-budget operating points over universal model rankings. BOTS v1 rows are more uniform at the task level, though provider defaults and reasoning settings still vary across models. BOTS v1 runs used a 250-message limit; this limit was hit by 8/93 DeepSeek v4 Flash sample-epochs at the $2.10 cap, 14/93 at the $4.20 cap, and 5/93 DeepSeek v4 Pro sample-epochs, so truncation may depress high-volume models. The bootstrap intervals in Appendix 11 are descriptive robustness checks, not formal population-level inferences.

Our defensive analysis covers only the 31 scored BOTS v1 Po1s0n1vy/Cerber questions, after excluding the official warm-up, and uses a public Splunk dataset. BOTS v2 and v3 extend v1: v2 covers a 51-question Frothly enterprise intrusion with a much larger endpoint, network, email, and web dataset, while v3 uses a 58-question cloud-heavy Frothly scenario spanning AWS CloudTrail, osquery, Windows, and network telemetry. Comparable multi-model results for these later versions are not yet available, so extending the same cost-aware analysis to BOTS v2 and v3 is future work. We therefore present the SOC conclusion as an evaluation-design result rather than a final claim about production SOC readiness.

8 Artifact and Ethical Considerations↩︎

We use public cybersecurity benchmarks and sandboxed environments. We report aggregate metrics and methodology only. We do not publish secrets, API keys, or operational exploit instructions. Any public release should include only sanitized materials.

9 Conclusion↩︎

Security-agent evaluation should be cost-aware and workflow-specific. Offensive CTFs and defensive SOC investigations stress different behaviors and show different scaling patterns. We find that more test-time compute can improve CTF success for some frontier/shared models, especially Claude Opus 4.8 and DeepSeek v4 Flash under retrospective caps; the new GPT-5.6 and Fable runs also show that policy refusals can dominate offensive scores. We also find that SOC work does not scale with raw budget in the same way; GPT-5.6 Terra and Sol approach the leading Claude Opus 4.8 score, while success still depends on disciplined tool use, telemetry navigation, and selective enrichment. Public SOC benchmarks remain valuable, but our no-tools BOTS v1 controls show that their absolute scores require decontamination checks before they are used as capability claims.

10 Evaluation Run Dates↩︎

Tables 5 and 6 report the UTC creation timestamp recorded in each source Inspect log used by the paper tables. Rows that appear in multiple result, scaling, or robustness tables are listed once. Where a row aggregates retry logs into the reported cost, the retry timestamps are included. The retrospective cap rows are derived from the listed source trace and were not separate evaluation runs.

11 Uncertainty Method↩︎

We report descriptive bootstrap intervals. For Cybench, each replicate resamples challenge IDs with replacement and retains the three epochs for each selected challenge. For BOTS v1, each replicate resamples question IDs with replacement and retains the three epochs and official point weights for each selected question. For retrospective cap contrasts, each replicate compares the capped and uncapped values for the same sampled task IDs. Paired contrasts resample only task IDs common to both compared runs. The GPT-5.6, Fable, and latest July Opus rows are not included in these previously computed bootstrap or scaling analyses.

12 Cybench Robustness↩︎

The main scaling contrasts are reported in Table 4. For model-ranking context, GPT-5.5 exceeds the higher-budget DeepSeek v4 Flash row by 7.7 percentage points, with interval +1.7 to +15.4 percentage points. Claude Opus 4.8 does not close the gap to those leaders: paired contrasts are GPT-5.5 minus Claude Opus 4.8 = +19.7 percentage points [+9.4, +30.8] and higher-budget DeepSeek v4 Flash minus Claude Opus 4.8 = +12.0 percentage points [+1.7, +23.1].

13 BOTS v1 Robustness↩︎

Claude Opus 4.8’s paired margin over the standard GPT-5.5 row is 12.8 percentage points, with a question-level paired bootstrap interval of +0.9 to +27.4 percentage points. Its margin over GPT-5.5 high effort is 12.5 percentage points, with interval +2.4 to +23.7 percentage points. The DeepSeek v4 Flash $4.20-cap row exceeds the $2.10-cap row by only 0.9 percentage points, with interval \(-4.9\) to \(+6.9\) percentage points, so that budget change is inconclusive.

14 BOTS v1 Sequential-Context Check↩︎

We exclude the official BOTS v1 warm-up question and evaluate 31 scored questions. Twenty-three of those questions have explicit dependencies on earlier case facts and receive prerequisite context in the prompt. This matches the benchmark’s sequential investigation structure, but we interpret it as follow-up investigation with known case state rather than cold-start reconstruction. Table 9 shows the direct prerequisite chart; the prompt preamble expands these edges transitively.

For Claude Opus 4.8, independent questions score 1,445/1,500 points (96.3%) and dependent questions score 8,221.7/8,800 points (93.4%). The standard GPT-5.5 row scores 1,345/1,500 points (89.7%) on independent questions and 7,000/8,800 points (79.5%) on dependent questions. Claude’s mean-reduced point loss decomposes into 133.3 points from hint penalties and 500 points from misses; GPT-5.5 loses 205 points to hint penalties and 1,750 points to misses.

References↩︎