1 Introduction↩︎

Organizations deploying large language models (LLMs) in business contexts invest heavily in evaluation—comparing models on quality metrics, measuring compliance with domain requirements, and validating outputs against business rules. Yet the dominant framing of LLM evaluation, in both academic benchmarks and enterprise practice, treats it as a model selection exercise: run a suite of tests, rank the models, deploy the winner [1]–[3]. This framing captures only a fraction of the value evaluation can provide.

In production systems, the model is rarely the only—or even the primary—variable determining output quality. The prompt, retrieval pipeline, configuration parameters, and input data formatting all shape the result. When a system underperforms, the question is not “which model should we switch to?” but “what’s wrong and what should we change?” Evaluation systems designed solely for model ranking cannot answer this question: they produce a score, not a diagnosis.

1.1 Motivation↩︎

Two developments motivate a rethinking of evaluation’s role. First, continuous evaluation advocates observe that fixed benchmarks fall short for enterprise-scale agents where requirements evolve continuously [4], and that point-in-time analyses do not address companies’ need to continuously assess tool reliability [5]. These observations establish that evaluation must be ongoing—but ongoing measurement alone is insufficient if it does not produce actionable signals.

Second, the prompt engineering literature demonstrates that systematic, metric-driven iteration consistently outperforms one-shot design. [6] show that minor prompt formatting changes can swing model performance by up to 76 percentage points. APE [7] and DSPy [8] demonstrate that automated prompt refinement, guided by evaluation metrics, produces prompts that outperform human-designed alternatives. These findings suggest that evaluation’s primary value lies not in selecting between models but in guiding the iterative improvement of the system around a model.

1.2 Gap↩︎

Despite these converging insights—that evaluation should be continuous [4], [5] and that iterative refinement works [6]–[8]—no existing methodology combines them into a coherent workflow. The continuous evaluation literature focuses on when to re-evaluate but not on what to do with the results beyond tracking quality over time. The prompt engineering literature optimizes prompts against aggregate metrics but does not address how to diagnose which aspect of a prompt is causing failures. Multi-dimensional evaluation frameworks like HELM [1] and DecodingTrust [9] demonstrate that models have heterogeneous strength profiles across dimensions, but frame this as a reporting concern rather than a diagnostic tool for iterative improvement.

The result is a gap between evaluation-as-measurement and evaluation-as-improvement. Practitioners who want to use evaluation diagnostically must improvise: manually inspecting failure cases, guessing which system variable to change, and running ad hoc experiments without a structured workflow.

1.3 Contributions↩︎

We present EvalLoop, a methodology that reframes evaluation as a feedback loop for iterative system improvement. EvalLoop is organized around three core mechanisms:

1. Dimensional metric grouping. Metrics are grouped by business-relevant quality dimensions, enabling diagnosis of orthogonal failure modes. When Structural Compliance is 96% but Hallucination Free Rate is 42%, these are different root causes requiring different interventions—a distinction invisible in aggregate scoring.

2. Failure mode classification. For judge-evaluated dimensions, the evaluation system classifies why outputs fail—not just that they fail. This bridges the gap between dimensional diagnosis and actionable intervention.

3. Iteration workflow. A structured diagnose-hypothesize-intervene-measure cycle that treats evaluation runs as experiments, making the impact of each change visible and attributable.

We validate EvalLoop through a case study on sales intelligence briefing generation (10 models, 3 providers, 18 metrics, 3 iterations). The methodology enabled a targeted prompt fix that improved the best model from 82.6% to 94.6% overall. We package EvalLoop as a reusable artifact bundle (practitioner playbook, coding agent specification, template repository).

1.4 Paper Organization↩︎

Section 2 positions this work relative to LLM evaluation frameworks, judge reliability research, and prompt engineering. Section 3 defines the problem. Section 4 presents the EvalLoop methodology. Section 5 validates through our case study. Section 6 discusses generalizability and threats to validity. Section 7 concludes.

2 Background and Related Work↩︎

2.1 LLM Evaluation Frameworks↩︎

The evaluation of large language models has evolved from single-metric benchmarks to multi-dimensional assessment frameworks. HELM [1] established the principle of holistic evaluation across seven metric categories, demonstrating that models exhibit heterogeneous performance profiles. [2] provide a comprehensive taxonomy organizing evaluation along three axes: what to evaluate, where, and how. DecodingTrust [9] extends multi-dimensional evaluation to trustworthiness, assessing GPT models across eight dimensions and finding that high capability does not guarantee trustworthiness.

These frameworks share a common limitation: they evaluate models on dimensions but do not prescribe how to use dimensional results to improve the system. Our methodology addresses this gap by connecting dimensional evaluation to iteration.

For domain-specific NLG evaluation, G-Eval [10] demonstrates that structured rubrics improve LLM judge correlation with human judgments. FActScore [11] introduces atomic decomposition for factuality. We extend this principle into failure mode classification: not just identifying which claims are unsupported, but categorizing why they are unsupported to guide prompt fixes.

2.2 LLM-as-Judge Reliability↩︎

LLM judges have become the primary evaluation mechanism for semantic quality dimensions [12]. However, their reliability is contested. [13] demonstrate position bias where response ordering affects rankings. [14] show systematic self-preference: LLM evaluators favor their own family’s generations. [15] show that diverse judge panels outperform homogeneous ones. [16] raise the meta-evaluation question of judge validation protocols.

Our methodology incorporates these findings through cross-provider judge panels (Section 4.4): judges from at least two different model providers, with rubric-based prompts and multi-judge aggregation.

2.3 Domain-Specific Enterprise Evaluation↩︎

[3] evaluate LLMs across enterprise-specific tasks and find that “no model dominates across all tasks.” The Sales Research Bench [17] evaluates sales AI across eight customer-weighted quality dimensions. Two recent papers argue for continuous evaluation: [4] propose continuous benchmark generation, and [5] present a framework for tracking evaluation as a “moving target.”

Our work shares the continuous evaluation premise but extends it: evaluation should not only be ongoing but diagnostic—producing signals that drive specific system changes.

2.4 Prompt Engineering as Iterative Process↩︎

The prompt engineering literature demonstrates that systematic optimization outperforms one-shot design. APE [7] shows that LLMs can generate prompts matching human engineer performance. DSPy [8] compiles declarative LLM programs into optimized prompts via metric-driven iteration. [6] quantify the stakes: minor formatting changes can swing performance by up to 76 percentage points.

These works optimize prompts against aggregate metrics. None addresses the diagnostic question: which aspect of the prompt is causing which type of failure? EvalLoop fills this gap by connecting dimensional evaluation with failure mode classification to produce targeted prompt modifications.

2.5 Positioning↩︎

Table 1 positions our contribution relative to the most closely related work.

3 Problem: From Measurement to Diagnosis↩︎

3.1 Static Evaluation Misses the Primary Value↩︎

Current LLM evaluation practice operates predominantly in what we term model selection mode: the system design is fixed, evaluation compares models, and the output is a ranking. Enterprise evaluation papers frame their contribution as helping organizations “select the right model” [3], [18]. This framing misses the primary value of evaluation for deployed systems. Model selection is a one-time decision; system improvement is the continuous work.

3.2 What Evaluation-Driven Improvement Requires↩︎

For evaluation to serve as an improvement tool, three capabilities are necessary:

Dimensional decomposition. The evaluation must report where the system is failing. An aggregate score of 82.6% provides no diagnostic signal. A dimensional profile showing [Structural: 96%, Content: 79%, Hallucination: 85%, Business Logic: 87%, Synthesis: 66%] immediately identifies improvement targets.

Failure mode classification. Within a weak dimension, the evaluation must report why outputs fail. “Hallucination rate is 85%” does not suggest a fix. “69% of hallucinations are inference-beyond-stated-facts” directly implies a specific intervention.

Iteration support. The evaluation system must make re-evaluation cheap. Configuration-driven architecture, experiment tracking, and checkpoint recovery are infrastructure prerequisites.

3.3 Why Dimensional Grouping Enables Diagnosis↩︎

The key insight is that different quality failures have different root causes and require different interventions. Structural failures are caused by unclear format instructions. Hallucination failures are caused by missing grounding constraints. Synthesis failures are caused by absent paraphrasing requirements.

Aggregate scoring conflates these orthogonal failure modes. Dimensional grouping makes them distinguishable: if metrics are grouped by the intervention that would fix them, then identifying the weakest dimension is equivalent to identifying the most impactful next intervention.

This aligns with findings from multi-dimensional evaluation frameworks. [3] demonstrate that models have heterogeneous profiles. HELM [1] reports results across seven dimensions precisely because aggregate rankings obscure important distinctions. Our contribution connects this observation to a workflow: dimensional profiles are not just informative but actionable.

4 EvalLoop: A Design Methodology↩︎

EvalLoop is organized around six principles. Each is articulated independently of our case study and grounded in prior literature.

4.1 Evaluation as Feedback Loop↩︎

An evaluation system operates in one of two modes: (1) Model selection (static)—the system design is fixed; evaluation compares models and picks the best one; and (2) System improvement (iterative)—evaluation measures the impact of changes to system variables, enabling a diagnose-fix-measure cycle.

The evaluation literature overwhelmingly supports mode 1. Our methodology addresses mode 2. The iteration workflow proceeds as:

1. Baseline evaluation. Run all target models against the full metric suite. Obtain dimensional profiles.

2. Diagnosis. Identify the weakest dimension(s). Classify failure modes.

3. Hypothesis. “Failures in dimension X are caused by system variable Y.”

4. Intervention. Change one system variable. Re-evaluate.

5. Comparison. Did the target dimension improve? Did others regress?

The prompt is frequently the most productive variable to iterate on. [6] demonstrate that minor prompt changes can swing performance by 10–70+ percentage points.

Final-stage human gate. After the iteration loop plateaus and 3–5 finalists are short-listed by dimensional and operational criteria, we recommend a one-time blind review by a domain expert before deployment. The gate confirms that dimensional improvements correspond to perceived quality and resolves multi-criteria deployment trade-offs (cost, latency, provider diversity) that automated metrics cannot decide. Critically, the gate is a terminal step, not a per-iteration check—automated metrics drive the hot loop, humans gate the cold one. This preserves iteration speed while preventing dimensional ceiling artifacts from masking residual quality issues. Section 5 (§5.6) instantiates this on the case study. Figure 1 illustrates the full cycle and its terminal human gate.

4.2 Dimensional Metric Grouping↩︎

We recommend grouping metrics by business-relevant quality dimension rather than by failure severity or measurement technique. The rationale: dimensions aligned with distinct failure modes make diagnosis actionable.

Definition. A dimension is a named set of metrics satisfying two criteria: (1) the metrics test a common underlying quality aspect recognizable to stakeholders (communicational validity), and (2) the metrics plausibly share an intervention path (interventional validity).

We recommend validating criterion 2 by checking intervention coherence: when the system changes, do metrics within the dimension move in the same direction? In our case study, Content Accuracy showed 75% intervention coherence; Business Logic showed 71%.

Static within-dimension correlation (Pearson, phi coefficient) is not a reliable validation criterion. Our empirical analysis found near-zero correlation within all dimensions—because many metrics are binary at ceiling (\(>\)​88% pass rates). Data-driven clustering produced groupings with no correspondence to expert dimensions (Adjusted Rand Index = \(-\)​0.09). Dimensions serve diagnostic and communicational purposes, not statistical ones.

Recommended dimension count: 5–8. Based on precedent from HELM (7), DecodingTrust (8), Sales Research Bench (8), and our case study (5).

4.3 Prompt Iteration as Primary Workflow↩︎

In our case study, the prompt—not the model—was the primary source of quality issues. The same model improved from 82.6% to 94.6% through a single targeted prompt change, while an undirected configuration change in a prior iteration produced zero quality impact.

Failure mode classification bridges diagnosis and action. Dimensional profiles tell a practitioner where the system is weak; failure mode classification tells them why. Categorizing 4,218 hallucination instances revealed: 41% were inferences beyond stated facts, 28% were claims neither confirmed nor denied, 20% were misattributions, 2% were direct contradictions. The dominant failure mode (69%) pointed to a specific prompt weakness: encouraging synthesis without grounding constraints.

This extends FActScore’s atomic decomposition [11] and CritiqueLLM’s critique generation [19] into a systematic classification step.

4.4 Measurement Patterns: Deterministic Metrics and LLM Judges↩︎

We recommend a deterministic-first heuristic: when a requirement can be checked by a rule, prefer the rule over an LLM judge. Rules are cheap, reproducible, and immune to judge biases [13]. Figure 2 illustrates the decision heuristic.

When LLM judges are necessary, their unreliability must be managed through: (1) cross-provider panels [13], [15]; (2) rubric-based prompts [10], [20]; (3) multi-judge aggregation reporting mean, median, and consensus; (4) failure mode classification in judge outputs; and (5) meta-evaluation honesty—framing judges as heuristic, not ground truth [16].

4.5 Architectural Patterns for Iteration↩︎

The evaluation system’s architecture should make iteration cheap:

1. Configuration-driven design. Model lists, prompt versions, and thresholds live in YAML/JSON—not hardcoded. Following DSPy’s separation principle [8].

2. Protocol-based evaluators. Each metric implements a common interface. New dimensions require new evaluator files, not orchestrator changes.

3. Provider-agnostic inference. An abstraction layer isolates provider-specific API logic.

4. Batch processing with checkpoint recovery. Long evaluations are resumable.

5. Experiment tracking. Every run logs to MLflow with full configuration and per-metric scores.

4.6 Developer vs.Agent Responsibilities↩︎

Decisions the developer must make: Dimension definitions, metric thresholds, rubric content, when to stop iterating.

Research the developer should conduct: Available judge models, provider rate limits, existing benchmarks.

Scaffolding a coding agent can generate: Orchestrator skeleton, batch processor, experiment tracking integration, evaluator templates.

This three-way split enables a reusable artifact bundle: a playbook guides developer decisions, an agent specification tells a coding agent what to scaffold, and a template repository provides the starting point.

5 Case Study: Sales Intelligence Briefing Generation↩︎

5.1 Domain, Task, and Setup↩︎

Task. Given a structured fact set describing a customer account, generate a concise bulleted briefing for a sales representative. The output must synthesize across facts, respect domain terminology rules, and avoid stating anything not grounded in the provided facts.

Test corpus. 100 synthetic account fact sets generated through a controlled pipeline: entity definitions are expanded into a matrix of candidate data points, populated with mock values, filtered via explicit rules, and emitted as normalized atomic key-value claims. The resulting fact sets exhibit realistic sparsity and domain variation while avoiding disclosure of proprietary data.

Models evaluated. 10 models across 3 providers: GPT-5.4, GPT-5.4-mini, GPT-5.4-nano (Azure OpenAI); Claude Opus 4.6, Claude Sonnet 4.6, Claude Haiku 4.5 (AWS Bedrock); Gemini 2.5 Pro, Gemini 2.5 Flash Lite, Gemini 3.1 Pro, Gemini 3.1 Flash Lite (Google Vertex AI).

Configuration. Temperature 0.0 (except Gemini 3.1 at provider default of 1.0). Max tokens: 2,000. All models received identical prompts via a shared prompt configuration. Judge panel: Claude Sonnet 4.6, GPT-5.4, Gemini 3.1 Pro.

Metrics. 18 metrics grouped into 5 dimensions: Structural Compliance (3 deterministic), Content Accuracy (4 deterministic + 1 judge), Hallucination Free Rate (1 judge, 3-judge panel), Business Logic (7 deterministic), Synthesis Power (2 deterministic).

5.2 Applying the EvalLoop Methodology↩︎

We executed three iterations:

Iteration 1: Baseline (prompt v2.0). The best model (gpt-5.4-nano, 87.4% overall) scored unevenly across dimensions. gpt-5.4 ranked 4th at 82.6%, with Synthesis Power at only 65.9%.

Iteration 2: Configuration experiment. Disabled reasoning tokens for Gemini models. Result: no significant change in any dimension—illustrating undirected iteration.

Iteration 3: Prompt refinement (v2.0 \(\to\) v3.0). Failure mode classification identified three actionable problems: (1) forbidden-term ambiguity causing Content Accuracy failures; (2) missing synthesis instruction causing high regurgitation; (3) prompt encouraging inference without grounding constraints (69% of hallucinations).

5.3 Cross-Run Results↩︎

Table 2 shows the dimensional impact on gpt-5.4 with paired statistical tests.

Statistical methodology. Because the same 100 test cases drive both runs, we use paired tests on per-case dimensional scores. We report two-sided paired \(t\)-tests (Bonferroni-corrected over the 5 dimensions, \(\alpha_\text{adj}=0.01\)), 95% percentile bootstrap confidence intervals on the per-case mean difference (10,000 resamples, fixed seed for reproducibility), Wilcoxon signed-rank as a non-parametric robustness check, and Cohen’s \(d\) for paired samples as the effect size. Although Shapiro–Wilk rejects strict normality of the paired differences for several dimensions, the Central Limit Theorem applies at \(n=100\), and bootstrap CIs match parametric t-CIs to within \(0.4\)pp on every dimension; we report both for transparency. Cross-dimension correlation of paired differences is small (\(|r| \leq 0.21\)), so Bonferroni’s independence assumption is met and Holm–Bonferroni produces identical conclusions. Per-cell sample sizes vary slightly in the appendix (n = 97–100): rows where every judge call failed for a metric are dropped from that dimension’s test only.

Improvement concentrated in dimensions targeted by the prompt change. Four of five dimensions show statistically significant gains after Bonferroni correction (Content Accuracy, Hallucination Free Rate, Business Logic, Synthesis Power; all \(p<0.001\)), with very large effect sizes for Content Accuracy (\(d=1.21\)) and Synthesis Power (\(d=1.51\)). The overall improvement of \(+12.0\)pp (95% CI \([+10.4, +13.5]\)) is highly significant with very large effect size (\(d=1.53\)).

Structural Compliance: practical equivalence, not just non-significance. Structural Compliance was not a target of the prompt revision. Its paired difference is statistically indistinguishable from zero (\(p=0.37\), Wilcoxon \(p=0.37\), Cohen’s \(d=-0.09\)), and 89% of per-case differences are exactly zero. The 95% CI of \([-3.0, +1.0]\)pp lies entirely within a \(\pm 5\)pp practical-equivalence band, so we read this as evidence of equivalence, not inconclusiveness. This is the result the dimensional-profiling claim predicts: targeted prompt changes should improve diagnosed dimensions while leaving non-targeted dimensions practically unchanged.

Figure 3 visualizes the per-dimension impact. No single model dominates all dimensions (Table 3): Gemini 2.5 Pro leads Synthesis (98.9%) but trails Hallucination (85.0%); gpt-5.4-mini leads Hallucination (95.0%) with a 10pp advantage for hallucination-sensitive deployments.

5.4 What Dimensional Analysis Revealed↩︎

Provider-level patterns. Claude models averaged 84.3% Hallucination but 95.8% Synthesis; GPT models averaged 91.3% Hallucination but 89.8% Synthesis—suggesting systematic differences in prompt interpretation invisible to aggregate scoring.

Framing comparison. For hallucination-sensitive deployment, the aggregate-ranked winner (Gemini 2.5 Pro, 93.4%) has only 85.0% hallucination score. The dimensionally-informed choice (GPT-5.4-mini, 95.0% hallucination) provides a 10pp safety advantage at lower cost.

Inter-judge agreement. Mean pairwise \(r\)=0.51 on hallucination, 67.4% unanimous agreement. Recommendation language achieved \(r\)=0.65, 99.4% unanimous—consistent with the deterministic-first heuristic.

5.5 Dimensional Validation↩︎

We validated the 5-dimension grouping using phi coefficients, hierarchical clustering, cross-run delta coherence, and mutual information. Content Accuracy showed moderate validation (75% intervention coherence, 3.3\(\times\) MI ratio). Business Logic showed no statistical cohesion (ARI=\(-\)​0.09 vs.data-driven clustering). This finding is itself diagnostic: Business Logic is communicationally valid but not interventionally valid—its metrics require per-rule interventions rather than a single fix. Three metrics saturate at 100% pass in this run (bullet_format_compliance, contact_age_filter, length_final) and are excluded from the binary-association tracks because phi and mutual information are undefined for zero-variance variables; they remain part of their respective dimensions and re-enter the intervention-coherence analysis.

5.6 Human Validation: SME Deployment Gate↩︎

After Iteration 3, four finalists were short-listed from the 10 evaluated models by combining dimensional scores, cost, latency, and provider diversity: gpt-5.4, gpt-5.4-mini, gemini-2.5-flash-lite, and Claude Opus 4.6. A subject-matter expert with experience in the target sales role then performed a blind review on a 16-case sample chosen to span all business segments and contract types. Per case, the SME received four outputs labeled “Model 1”–“Model 4” with the model-to-label mapping randomized, and selected the best and 2nd-best. Per-model preference was summarized via a weighted count (\(0.7 \times \text{best} + 0.3 \times \text{2nd-best}\)).

Claude Opus 4.6 was the SME’s top weighted choice (top-2 in 13 of 16 cases, 81%) and was selected for deployment after combining the SME signal with cost and latency. The dimensional ranking and the human gate converged: gemini-2.5-flash-lite, weakest on dimensional scores, was also weakest in human review, while opus and gpt-5.4-mini—both strong dimensionally—were the two human-preferred finalists.

Convergence with EvalLoop dimensions. SME free-text notes (12 of 16 cases) clustered cleanly into the existing dimensional taxonomy: length/concision concerns (Synthesis Power), role-inappropriate terminology such as “Active” used in Perm contracts (Content Accuracy), priority-ordering observations (Business Logic), and spurious content for absent facts (Hallucination Free Rate). No novel failure mode emerged in human review—the dimensional taxonomy held up under blind expert scrutiny.

Iteration economy. Human review focused on a 4-model \(\times\) 16-case sample (64 observations) instead of the full 10-model \(\times\) 100-case design (1,000 observations)—a 94% reduction in review burden—without losing validation strength: the dimensional profiles had already filtered out clearly-failing candidates (no finalist received a “None Are Great” verdict on any case).

Limitations. Single-reviewer design with no inter-rater agreement; 16 of 100 cases reviewed; rankings rather than per-dimension numeric scores. The gate is positioned as a deployment checkpoint, not a calibration of LLM judges against human ground truth—that is a separate study.

6 Discussion↩︎

6.1 Generalizability↩︎

EvalLoop’s principles apply to any LLM-powered system where output quality decomposes into distinguishable dimensions, different failure modes suggest different interventions, and the practitioner can re-evaluate after changes. The specific dimensions are task-dependent; the methodology prescribes the process of deriving, validating, and iterating on dimensions.

The case study’s finding that the prompt was the primary bottleneck may not generalize to all domains. For reasoning-heavy tasks, model capability may be the limiting factor. For retrieval-augmented systems, retrieval quality may dominate. EvalLoop supports iterating on any system variable; the prompt-first heuristic is a recommended starting point, not an invariant.

6.2 Boundary Conditions↩︎

EvalLoop is not appropriate when: output has no decomposable structure (open-ended creative writing); the system is already at ceiling on all dimensions; the bottleneck is outside the evaluation loop (training data, retrieval quality); or evaluation cost is prohibitive for rapid iteration.

6.3 Threats to Validity↩︎

Internal validity. The iteration improvement was measured on the same 100-account test corpus. Overfitting to the test distribution remains a risk absent a held-out validation set. Statistical testing on per-case paired deltas (Table 2) confirms that four of five dimensional improvements survive Bonferroni correction with medium-to-very-large effect sizes; the Structural Compliance change (\(-\)​1.0pp) is reported transparently as non-significant.

External validity. Results derive from a single task and domain. Generalization is unvalidated, though methodology principles are grounded in multi-domain literature.

Construct validity. The 5-dimension grouping was expert-defined, not data-derived. Two dimensions showed genuine statistical structure; others group independent metrics. We report this transparently.

Judge reliability. Inter-judge agreement (\(r\)=0.51, 67.4% unanimous) indicates moderate reliability. Judges remain heuristic evaluators whose agreement with human ground truth was not validated quantitatively in this study. The SME gate (§5.6) provides qualitative confirmation—human-preferred finalists matched dimensional rankings and SME concerns mapped onto the existing dimensions—but a single reviewer on 16 cases is not a calibration of judges against humans.

Human gate scope. The SME gate complements LLM-judge evaluation rather than replacing it: judges scale to 1,000-evaluation iterations; humans gate before deployment. We position human review as a one-time terminal step to preserve iteration speed. Higher-stakes domains (regulated communications, medical, legal) may need a larger panel and per-iteration sampling—the methodology supports this, but our case study does not validate it.

7 Conclusion↩︎

We presented EvalLoop, a methodology for evaluation-driven iterative improvement of business AI systems. EvalLoop reframes evaluation from static model selection to a diagnostic feedback loop: dimensional metric grouping surfaces where the system is failing, failure mode classification reveals why, and a structured iteration workflow translates diagnosis into targeted fixes with measurable impact.

Our case study demonstrated the methodology’s value: dimensional diagnosis identified prompt-induced interpretation errors as the dominant hallucination source, enabling a targeted fix that improved the best model from 82.6% to 94.6%. The empirical validation of dimensional grouping produced a finding of independent interest: expert-defined dimensions serve communication and intervention-targeting purposes but do not necessarily correspond to statistical structure in the data.

Future work. Longitudinal validation across multiple domains; automated dimension discovery from requirements; integration with automated prompt optimization where dimensional diagnosis guides the optimizer’s objective function.

Acknowledgements↩︎

This work was conducted at Robert Half, where the Data Science team builds, deploys, and evaluates production AI systems for enterprise use. We thank Robert Half for encouraging the publication of applied research methodologies developed in the course of this work and for fostering an environment that supports experimentation, evaluation, and continuous improvement of AI systems.

We are grateful to the colleagues, practitioners, and business stakeholders whose domain expertise and feedback helped shape the evaluation framework and case study presented in this paper. Their collaboration helped ensure that the methodology addressed practical challenges encountered in real-world enterprise AI deployments.

8 All-Models Significance Results↩︎

Table 1 extends the gpt-5.4 analysis (Table 2) to all 10 evaluated models, using the same paired-test methodology (paired \(t\)-tests with Bonferroni correction over 5 dimensions, 95% bootstrap CIs with 10,000 resamples and fixed seed, Wilcoxon as a robustness check, and Cohen’s \(d\)). Per-cell sample sizes vary by 0–3 because rows where every judge call failed for a metric are dropped from that dimension’s test only: the GPT and Gemini 2.5 families retain \(n=100\) on every dimension; gemini-3.1-pro drops to \(n=99\) on Hallucination Free Rate; Claude Haiku and Claude Sonnet retain \(n=99\) across all dimensions; Claude Opus retains \(n=97\). The pattern is consistent across providers: Content Accuracy and Synthesis Power show large, highly significant gains for nearly every model; Hallucination Free Rate gains are largest for the Claude family (whose v2.0 baselines were lowest); Structural Compliance changes are mixed in sign and small in magnitude.

References↩︎