1 Introduction↩︎

Fine-tuning and evaluating large language models (LLMs) for e-commerce applications demands massive volumes of high-quality labeled data. Specifically, predicting and evaluating product attribute values from unstructured catalog text (e.g., product titles, descriptions, bullet points) requires labeled examples that maintain representativeness across three key dimensions: product categories (e.g., electronics, clothing, furniture), attributes (e.g., color, material, dimensions), and languages. At catalog scale—thousands of product categories, thousands of attributes, and multiple languages—achieving reliable model performance requires hundreds of labeled examples per (product category \(\times\) attribute \(\times\) language) triplet. This combinatorial requirement translates to millions of annotations, making human labeling prohibitively costly.

Prior work demonstrates the feasibility of using LLMs to generate synthetic labels for e-commerce attribute value extraction [1]. However, deploying such approaches at industrial scale introduces a critical challenge: how to validate label quality efficiently across heterogeneous attributes and languages without exhaustive manual review. A scalable solution must integrate synthetic label generation with automated quality control mechanisms.

This paper presents SynthAVE (Synthetic data for Attribute Value Extraction), a large-scale human-validated benchmark augmenting prior generation methodology [1] with scalable quality assurance. We introduce a multi-LLM auditing framework where each generated label is independently evaluated by 21 judge configurations (7 model families \(\times\) 3 prompts), with final labels determined through majority voting. To mitigate systematic biases, we employ diverse models from different providers alongside varied prompting strategies. Ground truth for calibrating this system is established through human validation of 12,726 products.

Empirical evaluation demonstrates that diverse models with different biases aggregate into highly reliable predictions: the ensemble achieves 95.2% agreement with human experts (Cohen’s \(\kappa = 0.92\)) while enabling cost-effective validation at scale.

Our main contributions are:

• SynthAVE: A human-validated benchmark of 12,726 products spanning 229 product categories, 792 attributes, and 4 languages for multilingual attribute extraction research.¹

• Multi-LLM Validation Framework: An auditing approach employing majority voting across diverse judge configurations, achieving 95.2% agreement with human evaluation while nabling cost-effective validation at scale.

2 Related Work↩︎

Information extraction (IE) from unstructured web text is a foundational capability for large-scale e-commerce systems and knowledge base construction. Common applications include product attribute extraction, entity population, catalog enrichment, and fact acquisition from heterogeneous online sources [2]–[4]. Recent advances in large language models (LLMs) have substantially lowered the barrier to deploying flexible extraction systems across domains and schemas [5]. However, evaluating the quality of extracted information remains a central challenge, primarily due to the scarcity of high-quality human-annotated data and the scale at which modern IE systems operate.

Classical evaluation methodologies for IE depend on manually annotated ground truth datasets and report performance using metrics such as precision, recall, and F\(_1\) score [5], [6]. While effective in controlled research settings, this paradigm does not scale to real-world web and e-commerce applications. Annotation is expensive, time-consuming, and often requires domain expertise, particularly when extraction targets complex or evolving schemas [7], [8]. As a result, many production pipelines operate with limited or no gold-standard evaluation data, motivating the development of evaluation methods that function under weak or missing supervision.

Existing benchmarks for product attribute extraction illustrate this tension between scale and quality. The Multi-source Attribute Value Extraction (MAVE) dataset [9] provides 2.2 million attribute-value annotations across 1,257 attributes from Amazon product profiles—achieving impressive scale through automated pipelines. However, MAVE’s quality assurance relies primarily on classifier-based filtering rather than comprehensive human validation of attribute-value annotations. Additionally, MAVE is limited to English and addresses extraction rather than verification tasks. These limitations motivate both the exploration of synthetic evaluation methods and the development of more rigorous validation frameworks.

In response to these constraints, recent work has explored evaluation strategies that reduce or eliminate dependence on human-labeled ground truth. [10] propose an automatic IE evaluation framework based on synthetic ground truth generation, in which artificially constructed structured facts are injected into natural documents and systems are evaluated on their ability to recover them. Despite their scalability, synthetic evaluation methods introduce limitations related to realism—artificially injected facts may not fully capture the ambiguity, inconsistency, and noise present in naturally occurring e-commerce text [1].

Another line of work investigates the use of LLMs to generate annotated datasets for downstream evaluation. By leveraging LLMs as annotators, it becomes possible to rapidly construct large quantities of labeled data where human annotation is infeasible [11], [12]. In e-commerce contexts, this supports approximate evaluation of extraction pipelines across diverse product categories [13]. However, automatically generated annotations may reflect model-specific biases or systematic errors, necessitating careful validation and selective human oversight [14]. Pipeline design choices—decomposing extraction into multiple stages, structured output formats, and normalization rules—further improve performance [15]–[17], but complicate evaluation when standardized benchmarks are unavailable.

3 Task and Dataset↩︎

Attribute Value Extraction Task. The attribute value extraction (AVE) task requires determining whether a given attribute value can be verified from unstructured product text. Given a product’s unstructured text (title, bullet points, description) and an attribute-value pair, the task is to assign one of three labels: CORRECT, INCORRECT, or UNKNOWN.

We illustrate with a concrete example. Given the product title ‘‘Portable Laptop Stand, Adjustable Height, Aluminum Construction, Silver Finish, Compatible with 10-17 inch Devices’’, the label depends on the attribute-value pair: \((\texttt{color}, \texttt{Silver})\) is correct since the text states “Silver Finish”; \((\texttt{color}, \texttt{Black})\) is incorrect as it contradicts the text; and \((\texttt{weight}, \texttt{1.2kg})\) is unknown since no weight information is present.

Training models to perform this three-way classification reliably requires labeled examples of all cases across diverse product categories, attributes, and languages. Our synthetic data generation pipeline produces such examples at scale, and our validation framework ensures their quality.

Synthetic Data Generation. We constructed the dataset in two stages. First, we sampled real products from a large e-commerce catalog spanning 4 languages (Spanish, French, Italian, German), ensuring categories exist in all languages with minimum 30 products per category. Second, for each sampled product, we generated synthetic variants using the controlled generation methodology of [1]. This process creates a strict one-to-one correspondence: each synthetic product instance is paired with exactly one target attribute and one value slot. The generation pipeline produces all three label types: correct (value explicitly or implicitly supported by text), incorrect (text contradicts the value), and unknown (attribute cannot be determined from available text).

\begin{table}[t]

\vskip

-0.1in \end{table}

The synthetic dataset comprises 12,726 products across 229 product categories, covering 792 unique attributes and 2,607 distinct product-attribute combinations. The dataset spans major product domains including electronics, apparel, home & furniture, and sporting goods. Each category contains a minimum of 30 products, with larger categories such as NOTEBOOK_COMPUTER reaching 209 products. Categories are evaluated on domain-specific attributes (e.g., connectivity_technology for electronics, closure.type for apparel). Full category and attribute distributions are provided in Appendix 8.

Validation Challenge. While the synthetic pipeline achieves 92.6% accuracy in preliminary human analysis, this error rate is problematic for benchmark evaluation—metrics computed against noisy ground truth yield misleading conclusions. Yet exhaustive human review of 12,726 products across 4 languages is prohibitively expensive. Section 4 presents our solution: a multi-LLM validation framework that achieves human-level accuracy at a fraction of the cost.

4 Multi-LLM Validation Framework↩︎

Having defined the attribute value extraction task and dataset construction process, we now address the core challenge: validating synthetic labels at scale without exhaustive human review. Building upon established generation methods [1], we present a multi-LLM auditing framework that aggregates diverse model judgments into reliable quality estimates.

Our framework treats LLM judges as independent evaluators whose collective judgment, when properly aggregated, can approximate human expert consensus. This requires careful attention to two foundational aspects: ensuring independence across judges, and establishing qualification criteria that parallel traditional human auditor standards.

Judge Selection and Configuration. For aggregated judgments to be statistically meaningful, individual judges must provide genuinely independent assessments. We enforce this through diversity along two dimensions.

Model family diversity. We selected 7 LLM families from different providers (Table ¿tbl:tab:model95details?): Claude [18], Nova [19], GPT [20], Mistral [21], DeepSeek [22], Qwen [23], and Gemma [24]. This ensures diversity across model scales and training objectives (general-purpose, reasoning-focused, multilingual), with outputs independent of family-specific biases.

\begin{table}[t]

\vskip

-0.1in \end{table}

Prompt diversity. For each model, we developed three substantially different prompt versions (detailed in Appendix 10). This prevents model outputs from reflecting prompt over-fitting or artificial consistency due to identical phrasing.

This design yields \(7 \times 3 = 21\) independent judge configurations. Each configuration evaluates all 12,726 products across 4 languages (Spanish, French, Italian, German), producing 267,246 total judgments.

Qualification Criteria. Beyond independence, each LLM judge must meet qualification criteria that parallel traditional standards for human auditors. General competency establishes baseline capability: models must meet minimum performance thresholds on established benchmarks, such as Global Average \(\geq 70\%\) on LiveBench [25], paralleling education or certification requirements for human auditors. Task-specific competency ensures domain relevance: models must demonstrate sufficient understanding of attribute verification through internal benchmark evaluation, analogous to subject matter expertise. Self-consistency addresses output reliability: given known variability in LLM outputs due to floating-point precision issues [26], judges must achieve minimum consistency when generating responses for identical prompts. Instruction adherence ensures usable outputs: models must reliably follow specified output formats, paralleling structured reporting requirements for human evaluators.

Evaluation Task. Judges perform the attribute consistency verification task defined in Section 3, classifying each sample as CORRECT, INCORRECT, or UNKNOWN with brief justification.

Aggregation through Majority Voting. Given a panel of qualified, independent judges, we aggregate their outputs via majority voting across all 21 configurations. This aggregation mechanism yields several important properties. Higher inter-judge agreement correlates with higher prediction accuracy, as consensus signals clearer cases. Unanimous agreement indicates highest-confidence predictions, while split decisions flag samples warranting closer inspection. Most importantly, diverse judges with moderate individual agreement can achieve high ensemble accuracy through error cancellation.

We therefore deliberately seek diversity among judges: moderate inter-model agreement (as measured by Fleiss’ \(\kappa\)) combined with high ensemble-human agreement (Cohen’s \(\kappa\)) indicates successful diversity—independent judges with different biases combining into robust collective predictions.

Ground Truth Establishment. To evaluate our framework, we established ground truth labels through a disagreement-based annotation strategy that leverages the LLM arena to minimize human effort. Each synthetic sample has an original label assigned by the generation algorithm. We compared this label against the LLM arena majority vote and applied the following protocol:

• Agreement cases: When the majority vote matched the original synthetic label, the sample was accepted as correct without human review.

• Disagreement cases: When the majority vote contradicted the original label, the sample was flagged for manual review by domain experts, who determined the final ground truth label.

To validate the agreement assumption, an expert annotator independently triaged a stratified random sample of 400 agreement cases (100 per language, comprising 100 unanimous and 300 mixed-agreement samples). The annotator overturned the majority vote in only 12/400 cases (3.0%, 95% CI [1.3%, 4.7%]). Critically, unanimous cases—where all 21 judges agreed—were never overturned (0/100), and all 12 disagreements occurred exclusively in mixed-agreement samples (4.0% overturn rate). Among majority labels, UNKNOWN showed the highest overturn rate (5.0%) compared to CORRECT (2.2%) and INCORRECT (0.0%), consistent with UNKNOWN being the “hedge” vote that occasionally masks determinable cases. This confirms that agreement between the generation algorithm and diverse LLM judges serves as a reliable signal of label correctness, with the majority panel achieving \(\sim\)​97% accuracy (full triage analysis in Appendix 12).

5 Results↩︎

We first establish that our validation framework achieves human-level reliability, then present the final dataset features after cleaning.

5.0.0.1 Overall Arena Performance.

Table ¿tbl:tab:arena95performance? summarizes agreement between LLM Arena majority voting and human expert evaluation. The arena achieves 95.2% overall agreement with human evaluators, with Cohen’s \(\kappa = 0.92\) indicating almost perfect inter-rater reliability [27]. Performance is consistent across languages (94.0–96.4%). Individual judges show substantial inter-model agreement (Fleiss’ \(\kappa\) = 0.757), yet the majority vote ensemble achieves almost perfect agreement with humans—validating that diverse models with different biases aggregate into highly accurate predictions.

\begin{table}[t]

\vskip

-0.1in \end{table}

5.0.0.2 Data Cleaning Performance.

Table ¿tbl:tab:cleaning95performance? summarizes the Arena’s effectiveness as a data cleaning mechanism. The original synthetic pipeline achieves 92.6% accuracy (946 errors across 12,726 samples). The arena corrects 786 of these errors (83.1% error correction rate), improving accuracy to 95.0%. When framed as binary bad-label detection, the Arena achieves 98.0% precision and 95.2% recall (F1 = 96.6%)—when it flags a label as incorrect, it is almost always right.

\begin{table}[t]

\vskip

-0.1in \end{table}

5.0.0.3 Per-Class Performance.

Table ¿tbl:tab:per95class? presents per-label metrics. The arena performs best on CORRECT (F1 = 97.3%), followed by UNKNOWN (F1 = 94.5%). INCORRECT labels prove most challenging (F1 = 89.2%), primarily due to lower recall—the arena tends to classify borderline INCORRECT cases as UNKNOWN, a conservative behavior preferable in production where false negatives are less costly than false positives.

\begin{table}[t]

\vskip

-0.1in \end{table}

5.0.0.4 Performance by Original Label Type.

Stratifying by original synthetic label reveals where the arena adds most value (Table ¿tbl:tab:by95original95type?). Originally CORRECT labels already achieve 97.3% accuracy, with modest arena improvement (+1.2%). Originally INCORRECT labels show the largest gap: synthetic accuracy is only 79.3%, but the arena recovers to 88.7% (+9.4%), demonstrating particular effectiveness at correcting the most error-prone category.

\begin{table}[t]

\vskip

-0.1in \end{table}

5.0.0.5 Agreement Level and Accuracy.

Accuracy correlates strongly with agreement level: unanimous consensus (36% of products) yields 100% accuracy, very high agreement (\(>\)​85%) yields 98.8%, while low-agreement cases (\(<\)​50%, 1.4% of products) achieve only 65.1%, suggesting these warrant human review. Full stratification is provided in Appendix 7.

5.0.0.6 Additional Analyses.

No individual judge configuration outperformed the majority vote ensemble (95.0%), and performance varied more across model families than across prompt variations, underscoring the importance of model diversity. Figure 2 illustrates this: certain model families consistently form high-agreement clusters (\(\kappa \approx 0.80\)–\(0.90\)) while others show lower mutual agreement (\(\kappa \approx 0.57\)–\(0.65\)), with patterns invariant across languages. Language-specific cleaning shows Italian achieving the highest fix rate (90.2%) and French the lowest (76.1%). Error analysis reveals 80.5% of errors involve confusion with UNKNOWN, while direct CORRECT \(\leftrightarrow\) INCORRECT misclassification is rare (19.5%). Full details are in Appendix 7.

5.0.0.7 Cost Analysis.

Total validation cost was $290.50 for 12,726 products ($22.83 per 1,000 products, 267,246 API calls).² Detailed breakdowns are in Appendix 13.

5.0.0.8 The SynthAVE Dataset.

The resulting dataset comprises 12,726 human-validated products spanning 229 product categories, 792 attributes, and 4 languages (47.7% CORRECT, 15.8% INCORRECT, 36.5% UNKNOWN). All samples carry quality guarantees through LLM-human consensus (97% verified accuracy via expert triage of 400 agreement cases; Appendix 12) or direct expert annotation. Unlike MAVE [9], SynthAVE provides comprehensive human validation, explicit UNKNOWN labels, and multilingual coverage. The dataset and per-judge predictions will be publicly released.

6 Conclusion↩︎

We presented SynthAVE, a large-scale human-validated dataset and validation framework for synthetic e-commerce data. Building on prior work in synthetic data generation [1], we introduced an LLM-arena framework that enables scalable quality assurance through multi-model evaluation.

Our evaluation yields three key findings. First, the LLM arena achieves 95% agreement with human evaluation (Cohen’s \(\kappa\) = 0.92) across 12,726 products, 229 product categories, 792 attributes, and 4 languages. Second, diversity enables accuracy: individual judges show only substantial agreement (Fleiss’ \(\kappa\) = 0.76), yet the ensemble outperforms any single model—with unanimous consensus reaching 100% accuracy. Third, expert triage of 400 agreement cases confirms that the majority panel is overturned only 3% of the time (0% for unanimous cases), validating the disagreement-based annotation strategy. The arena corrects 83.1% of synthetic labeling errors at $0.02 per product, reserving human review only for low-agreement cases.

These results support adopting multi-LLM evaluation protocols that incorporate prompt and model diversity, moving beyond single-model approaches. The SynthAVE dataset will be publicly released to support future research in scalable data quality assurance for attribute value extraction.

Limitations↩︎

Several limitations warrant discussion. First, our evaluation covers four European languages with shared Latin script; generalization to languages with different scripts (e.g., Chinese, Arabic), morphological structures, or lower LLM proficiency requires additional validation. Similarly, the framework was evaluated on e-commerce attribute verification—a task with relatively objective ground truth—and may perform differently on more subjective evaluation tasks.

Second, the framework depends on the availability of diverse, high-capability LLMs. As model families evolve, retire, or change pricing, the specific judge configurations may require updating. The finding that some model families significantly underperform others (accuracy ranging from 76% to 93%) raises questions about minimum capability thresholds for judge inclusion.

Third, our ground truth construction relied on a disagreement-based annotation strategy where human review was triggered only when the LLM ensemble disagreed with synthetic labels. Expert triage of 400 stratified agreement cases validated this approach (3.0% overturn rate, 0% for unanimous cases), though it does not provide traditional inter-annotator agreement metrics among human experts.

Finally, while the framework is designed for offline validation rather than real-time inference, the latency of running 21 judge configurations may still be prohibitive for rapid iteration cycles. Future work could explore distilling ensemble judgments into smaller, specialized models for faster evaluation.

Ethics Statement↩︎

SynthAVE is constructed from synthetic product data generated using the methodology of [1], with seed products sampled from commercial e-commerce catalogs. All brand names, model identifiers, and potentially identifying product information were anonymized during generation to prevent privacy concerns and mitigate LLM biases from prior knowledge of specific products—no real customer data or personally identifiable information is present in the released dataset. Human validation was performed by domain experts specialized in e-commerce catalog systems, who established ground truth labels through careful review of disagreement cases between the synthetic generation pipeline and LLM arena predictions. SynthAVE is designed as a test set optimized for evaluation coverage: it spans 2,598 product type-attribute combinations with a minimum of 30 samples per combination, prioritizing breadth over the volume typically required for training. We encourage researchers to leverage this benchmark for downstream applications including validating catalog quality assurance systems, improving attribute coverage in product catalogs, developing robust multilingual extraction methods, and advancing LLM-as-judge evaluation methodologies. We hope SynthAVE contributes to more accurate and reliable e-commerce information systems that benefit both businesses and consumers.

7 Detailed Results↩︎

7.1 Agreement Level and Accuracy↩︎

Table ¿tbl:tab:agreement95accuracy? shows arena accuracy stratified by the proportion of judges agreeing on the majority vote.

7.2 Data Cleaning Performance by Language↩︎

Table ¿tbl:tab:language95cleaning? presents detailed cleaning performance by language, including fix rates and regression rates.

7.3 Error Pattern Distribution↩︎

Table ¿tbl:tab:error95patterns? breaks down the arena’s 630 errors by confusion pattern. The dominant pattern is INCORRECT \(\rightarrow\) UNKNOWN (27.5%), indicating conservative abstention. Confusion involving UNKNOWN accounts for 80.5% of all errors, while direct CORRECT \(\leftrightarrow\) INCORRECT misclassification is rare (19.5%).

8 Dataset Statistics↩︎

This appendix provides comprehensive statistics for the SynthAVE test set we release with this paper.

8.1 Overall Scale↩︎

• Total products: 12,726 unique products

• Product categories: 229 distinct types

• Unique attributes: 792 across all languages

• Product-attribute combinations: 2,607 unique pairs

• Average entries per combination: 4.9

• Languages: 4 (Spanish, French, Italian, German)

• Products per language: 3,007–3,501

• Judge configurations: 21 (7 models \(\times\) 3 prompts)

8.2 Label Distribution↩︎

Table ¿tbl:tab:label95distribution? presents the overall label distribution based on majority vote across all judges.

8.3 Product Category Distribution↩︎

Table ¿tbl:tab:category95diversity? presents the top 20 product categories by volume.

8.4 Attribute Distribution↩︎

Table ¿tbl:tab:attribute95diversity? presents the top 20 attributes by frequency.

8.5 Top Product-Attribute Combinations↩︎

Table ¿tbl:tab:top95combinations? presents the most frequent product-attribute combinations in the dataset.

8.6 Per-Language Statistics↩︎

Table ¿tbl:tab:language95detailed? provides detailed per-language statistics including unique product types, attributes, and combinations.

8.7 Ground Truth Distribution↩︎

For human validation, we collected ground truth labels for all entries. Table ¿tbl:tab:ground95truth? shows the distribution.

9 Inter-Judge Agreement Heatmaps by Language↩︎

This appendix presents pairwise agreement heatmaps for all four languages. Figure 2 in the main text shows German (Cohen’s \(\kappa\)). Below we provide the complete set for all languages and both agreement metrics.

9.1 Cohen’s Kappa Heatmaps↩︎

Figures 3, 4, 5

9.2 Raw Agreement Heatmaps↩︎

Figures 6, 7, 8, 9

10 Evaluation Prompt Templates↩︎

This appendix provides the complete prompt templates used for our LLM Arena evaluation across all three prompt versions. Each version maintains the same evaluation task (attribute consistency verification) while employing different structural approaches and reasoning strategies to ensure IID generation requirements.

10.1 Prompt Version 1: Direct Instruction↩︎

10.2 Prompt Version 2: XML-Structured with Examples↩︎

10.3 Prompt Version 3: Role-Based with Systematic Guidelines↩︎

11 Evaluation Settings↩︎

11.1 Input Format↩︎

Figure 13 shows the input format provided to each judge.

11.2 Model Configuration↩︎

All models use a low temperature (0.1) to ensure consistent, reproducible outputs. We leave default settings for Top-p and Top-k parameters.

For OpenAI’s GPT OSS reasoning model, we set
reasoning_effort="low" to balance computational cost with performance. This parameter controls the depth of the model’s internal chain-of-thought reasoning before producing a final answer. Even at the “low” setting, the model demonstrates strong performance on our classification task while significantly reducing inference costs compared to higher reasoning effort levels.

Model-Specific Formats. Each model family requires different API message formats:

• Claude: Uses Anthropic’s message format with
  anthropic_version specification

• Nova: Requires nested content structure with explicit text fields

• OpenAI: Standard chat completion format with with
  reasoning_effort parameter

• Mistral: Uses instruction-tuned format with [INST] tags

• DeepSeek/Qwen3/Gemma: Standard chat message format

11.3 Response Parsing↩︎

Models return responses in different formats requiring specialized parsing:

• Standard format (Claude, Nova, Mistral, DeepSeek, Qwen3, Gemma): First line contains the classification label (CORRECT, INCORRECT, or UNKNOWN), followed by reasoning on subsequent lines.

• OpenAI reasoning format: Response contains
  <reasoning>...</reasoning> tags encapsulating the model’s chain-of-thought, followed by the final classification label. The parser extracts both components separately.

Invalid or unparseable responses default to UNKNOWN with the full response content logged for debugging.

11.4 Evaluation Pipeline↩︎

For each product and judge configuration:

1. Input formatting: Product data formatted according to model-specific API requirements

2. API invocation: Model called via AWS Bedrock with configured parameters

3. Response parsing: Extract classification and reasoning from model output, handling format variations

4. Validation: Verify response contains valid classification (CORRECT, INCORRECT, or UNKNOWN); invalid responses default to UNKNOWN

5. Error handling: API failures and malformed responses are caught and logged, with graceful degradation to UNKNOWN

6. Storage: Save classification, reasoning, model ID, prompt version, and timestamp

All models were accessed through AWS Bedrock’s unified API, ensuring consistent infrastructure across different model providers. The pipeline processes 21 judge configurations (7 models \(\times\) 3 prompt variants) per product systematically.

12 Human Triage of Agreement Cases↩︎

To validate the assumption that agreement between the synthetic generation pipeline and the LLM arena majority vote implies label correctness, an expert annotator independently triaged a stratified random sample of 400 agreement cases. The sample was balanced across locales (100 per language) and stratified by agreement level (100 unanimous, 300 mixed-agreement).

For each sample, the annotator reviewed the product text, attribute-value pair, and majority vote label, then recorded whether they agree with the majority, disagree (i.e., would overturn it), or are unsure. Only clear disagreements count as overturns in the analysis below; unsure cases (6/400) are excluded from the overturn rate.

12.0.0.1 Overall Results.

The annotator disagreed with the majority vote in only 12 out of 400 cases (3.0%), with a 95% confidence interval of [1.3%, 4.7%] (margin of error \(\pm\)​1.7%).

12.0.0.2 Breakdown by Locale.

12.0.0.3 Key Findings.

• The majority panel achieves a 97% accuracy rate (12/400 overturns) with a tight confidence interval of [1.3%, 4.7%].

• Unanimous cases have a 0% overturn rate (0/100): when all 21 judges agree, the label is always correct in our sample.

• UNKNOWN is the least reliable majority label at 5.0% overturn rate, compared to CORRECT at 2.2% and INCORRECT at 0.0%. This is consistent with UNKNOWN being the conservative vote that can mask determinable cases.

• All 12 overturns came from mixed-agreement samples (4.0% rate); no unanimous sample was overturned.

• Locale differences range from 1.0% (FR) to 5.0% (IT), but with \(n\)=100 per locale the margins of error (2–4%) overlap substantially. Distinguishing locale-level effects would require \(\sim\)​1,000+ samples per locale.

12.0.0.4 Implications.

The 3% \(\pm\) 1.7% overturn rate supports the claim that the majority panel correctly labels \(\sim\)​97% of agreement cases. Combined with the 0% overturn rate for unanimous cases, this validates the disagreement-based annotation strategy: agreement between the generation pipeline and a diverse 21-judge panel is a reliable signal of label correctness, with residual error concentrated in inherently ambiguous mixed-agreement cases.

13 Cost Analysis Details↩︎

This appendix provides detailed cost breakdowns for the LLM Arena validation framework. All costs are based on AWS Bedrock on-demand pricing as of January 2026.³

13.1 Token Statistics↩︎

Each API call consists of input tokens (the prompt with product data) and output tokens (the model’s classification and reasoning). Input token counts vary based on prompt version and product text length; output tokens are estimated at 50 tokens per response (classification label plus brief reasoning).

Table ¿tbl:tab:token95stats? presents input token statistics. The three prompt versions differ in complexity: V1 provides basic instructions, V2 adds attribute descriptions, and V3 includes worked examples. This is reflected in token counts, with V3 prompts averaging 70% more tokens than V1.

Token counts vary across languages due to differences in product text length and linguistic characteristics. German (DE) has the highest average token counts, while Spanish (ES) has the lowest. The standard deviation of approximately 285–308 tokens reflects the diversity in product description lengths across the dataset.

13.2 Full Arena Costs↩︎

Table ¿tbl:tab:arena95costs95language? shows the per-language cost breakdown for running the full 21-judge arena (7 models \(\times\) 3 prompt versions). Costs scale linearly with dataset size, with French being most expensive due to having the most products.

13.3 Scaling Projections↩︎

Table ¿tbl:tab:scaling95costs? projects arena costs at various scales. At production scale (1M products), the full arena would cost approximately $22,825—substantially less than equivalent human annotation, which would cost $100,000–500,000 at typical rates of $0.10–0.50 per label.

13.4 Cost-Efficient Configurations↩︎

For budget-constrained deployments, reduced ensemble configurations offer significant savings with potential accuracy trade-offs.

Table ¿tbl:tab:reduced95ensembles? presents alternative ensemble configurations. Cost varies substantially across model families, with open-weight models typically 10–30\(\times\) cheaper than commercial APIs. Using a single model family with all three prompt versions represents 3–49% of full arena cost depending on family choice.

The trade-off between cost and accuracy presents optimization opportunities. Unanimous agreement cases (36% of products) could use a smaller initial ensemble, with full arena deployment reserved for uncertain cases. Such adaptive strategies could reduce average cost by 30–50% while maintaining high accuracy on confident predictions. However, ablation studies validating accuracy of reduced configurations are required before deployment.

References↩︎

---

1. The public release will additionally include English-language annotations produced using the same pipeline, extending coverage to 5 languages.↩︎

2. Pricing based on commercial API rates as of June 2025.↩︎

3. https://aws.amazon.com/bedrock/pricing/↩︎