1 Introduction↩︎

Large Language Models (LLMs) and general-purpose AI assistants, such as ChatGPT and Gemini, are fundamentally reshaping how users interact with information by providing direct, conversational, and context-aware answers. This shift is disrupting traditional search engine marketing (SEM) that relies on clicking sponsored page on the search engine. Notably, [1] reports that 35% of casual users find LLMs more helpful for obtaining information than search engines. A recent study [2] confirms that Google users’ click-through rate (CTR) is 8% when AI summaries are present, compared to 15% without them, representing nearly a 47% decline in CTR on traditional links when AI summaries appear. Meanwhile, to monetize LLM models, service providers such as Bing Chat [3] and Perplexity [4] have begun experimenting with integrating advertisements directly within LLM-generated responses, referred to as generative engine marketing (GEM).

This work focuses on a linchpin problem of GEM, called the ad-injected response (AIR) generation. Given an advertisement database \(\mathcal{D}\) and an LLM \(\mathcal{M}\), the AIR generation problem takes as input a user query and optional context (e.g., user profile or chat history) and aims to seamlessly integrate relevant ad descriptions from \(\mathcal{D}\) into the \(\mathcal{M}\)-generated response to this query. As a new research direction, datasets, evaluation metrics, and solutions for AIR generation remain limited. Existing LLM benchmark datasets [5]–[8] contain many questions from domains that are fundamentally unsuitable for ad integration, such as mathematics, coding, and data labeling. Moreover, common text quality metrics such as ROUGE [9] and BLEU [10] evaluate responses against reference texts, which introduces bias when ads are injected into pre-generated templates or responses. At the same time, recent LLM-as-a-Judge approaches [7], [8] are not tailored for evaluating AIR, since they fail to capture nuanced aspects of user experience such as trust or jump-out feeling [11]. Regarding existing solutions, [12] propose the first conceptual framework for LLM-based advertising. In this framework, the authors focus on formulating the stages of this emerging advertising ecosystem. Although they mention the idea of first generating an ad-free response and then modifying it by inserting suitable ads for AIR generation, their main focus is on the broader mechanism design, e.g., whether the platform or the advertiser performs this modification and the associated challenges, whereas the specific strategy for how to perform the modification remains an open question. [11] design a chatbot for AIR generation in which relevant ads are inserted into the system prompt to personalize responses and promote products or brands, yet its performance has not been extensively studied.

To address these issues, we introduce GEM-Bench, the first comprehensive benchmark for AIR generation in GEM. GEM-Bench consists of curated datasets, a measurement ontology, and several vanilla solutions for evaluating AIR generation in GEM. Specifically, we curate two datasets, MT-Human and LM-Market, for chatbot scenarios, containing user queries to ChatGPT that are suitable for ad injection, and one dataset, CA-Prod, that simulates the AI overview feature in search engines. To evaluate the quality of AIR, we propose a metric ontology that captures multiple aspects of user satisfaction and engagement. This ontology can also be applied to other stages of GEM, such as estimating response quality during bidding and auction. Finally, we propose a multi-agent paradigm, called Ad-LLM, for AIR generation, which first produces ad-free responses and subsequently injects advertisements with refinement. It decouples agents and workflows, enabling flexible development of future solutions. We also instantiate several baseline solutions under this paradigm. Our preliminary experiments reveal clear trade-offs among different solutions. We find that the simple system prompt-based approach [11], while achieving relatively high engagement (e.g., 6.5% higher CTR), often compromises user satisfaction by reducing response accuracy and trust. In contrast, Ad-LLM improves user satisfaction and engagement, achieving relative gains of up to 39.8% in accuracy, 89.3% in personality, and 62.9% in trust across multiple datasets. These improvements are consistently validated by different LLMs and human participants, reaching a strong consensus. However, this baseline incurs additional overhead, consuming about 2\(\times\) more output tokens than the system prompt-based solution.

To summarize, we make the following contributions in this work:

• We introduce GEM-Bench, the first comprehensive benchmark for evaluating ad-injected response (AIR) generation in generative engine marketing (GEM).

• We curate and preprocess three datasets from two real-world scenarios, chatbots and search engines, where AIR generation can be effectively applied.

• We design an evaluation ontology for assessing the quality of AIR, considering multiple aspects of user satisfaction and engagement.

• We introduce a multi-agent framework Ad-LLM with several baseline implementations for AIR generation, enabling researchers to easily experiment with and develop new solutions.

2 Preliminaries↩︎

This section first introduces the existing frameworks for search engine marketing (SEM) and generative engine marketing (GEM), followed by formulating the Ad-Injected Response (AIR) Generation problem and briefly reviewing the existing solution, named Ad-Chat [11].

2.1 SEM and GEM↩︎

The online advertising ecosystem consists of three stakeholders: platforms, users, and advertisers. Platforms connect advertisers with users by delivering targeted ads, aiming to balance user experience with sustainable monetization. Users consume content and may interact with ads, expecting minimal disruption. Advertisers pay for placements to reach target audiences and drive conversions.

As a primary channel of online advertising, SEM displays sponsored results in response to user queries on a search engine. The standard paradigm [13] typically consists of the following core stages:

• Stage 1: Offline bidding. Advertisers first configure their campaigns by placing monetary bids on one or more keywords relevant to their ads.

• Stage 2: Query processing. When a user submits a query, the platform retrieves a ranked list of organic results and, in parallel, collects a set of candidate ads that match the query.

• Stage 3: Slot identification. Based on the query and result list, the platform determines how many ad slots will be displayed.

• Stage 4: Performance prediction. For each advertiser–slot pair in the candidate set, the platform predicts the likelihood of a user clicking the ad, i.e., the click-through rate (CTR).

• Stage 5: Auction and allocation. Finally, the platform runs an auction to allocate ads across the available slots, aiming to maximize overall expected value by considering bid amounts and CTR.

Analogous to SEM, [12] propose a conceptual framework for generative engine marketing (GEM), in which an LLM integrates native advertisements directly into its generated responses. This framework mirrors the five stages of SEM but extends them to the context of LLMs. In particular, during query processing (stage 2), the system first generates an ad-free response to the user query and then performs an additional modification step to inject a relevant ad candidate into the output.

Although the overall pipeline can be adapted from SEM, each stage of GEM introduces unique challenges and open questions. In the bidding stage, a central issue is determining what exactly advertisers would bid for. In the slot identification stage, the absence of explicit ad slots raises two key problems: deciding whether a query is suitable for ad inclusion and, if so, identifying insertion points within unstructured text that may contain multiple candidates. For performance prediction and auction, a fundamental challenge is estimating click-through rates (CTR) from unstructured responses. Beyond explicit engagement signals, the system must also evaluate and predict user satisfaction with the generated content. Finally, in the ad-injected response (AIR) generation stage, the central problem is how to produce output that (i) incorporates ads seamlessly without degrading the user experience, and (ii) satisfies the advertiser’s objectives.

2.2 Ad-Injected Response Generation↩︎

In this work, we focus on the Ad-Injected Response (AIR) Generation problem. It is the linchpin step of GEM, since regardless of how the overall framework evolves, the AIR is eventually what the system delivers to the user.

We represent the results of the offline bidding stage as an advertisement database (AdDB) \(\mathcal{D}\), where each row \(d_i \in \mathcal{D}\) contains the bidding string, its associated embedding, bidding price, and product meta-information (such as advertiser, description, URL, and ad creatives). Given an AdDB \(\mathcal{D}\), an LLM \(\mathcal{M}\), and an integer \(k\), the AIR generation problem takes as input a user query and possible context (e.g., user profile or chat history) and aims to return an ad-injected response (AIR) that seamlessly integrates \(k\) unstructured and relevant ads from \(\mathcal{D}\) into the \(\mathcal{M}\)-generated response to the query. The objective is to maximize both user satisfaction and user engagement. Satisfaction reflects the user experience when interacting with the model \(\mathcal{M}\) and can be measured by factors such as response latency and content quality. Engagement, on the other hand, captures whether the user notices the ad, has a positive attitude toward it, and potentially clicks the in-text ad link, which are the dimensions of primary interest to advertisers. For simplicity, we assume that every query discussed here has already passed the slot identification stage and is therefore suitable for ad insertion, and we set \(k = 1\) in the subsequent discussion.

Beyond the idea of inserting a relevant ad into an original response [12], [11] propose Ad-Chat, which integrates native ads into the system prompt when responding to user queries. As illustrated in Figure 1, given a user query and possible user profile and chat history, Ad-Chat first uses the LLM \(\mathcal{M}\) to assign a topic from Google Topics [14] to the conversation. It then instructs \(\mathcal{M}\) to select the most suitable product from the bidding products associated with that topic. Finally, Ad-Chat inserts the ad with its description and URL into the system prompt, aiming to appeal to the user and personalize the response while promoting products, ultimately generating the AIR.

3 GEM-Bench↩︎

This section introduces the main components of GEM-Bench: curated datasets for both chatbot and search engine scenarios, a measurement ontology for evaluating AIR, and a modular framework implementation. Detailed descriptions of each component are provided in Appendix.

3.1 Datasets↩︎

For the chatbot scenario, we construct two query sets. MT-Human is curated from MT-Bench [7], where each query is manually examined for suitability for ad insertion. We retain all 10 first-turn queries from the humanities category to form the MT-Human dataset, including examples such as “What are some business etiquette norms when doing business in Japan.” Due to the large scale of LMSYS-Chat-1M [8], LM-Market is derived through a multi-stage pipeline: we filter for single-turn English queries, cluster them into topics, and then apply LLM-assisted scoring followed by manual verification to retain queries with high ad-injection potential. In total, 1,701 queries are selected across three domains: travel planning, recipe recommendation, and software tools comparison. The associated AdDB \(\mathcal{D}\) is adopted from Ad-Chat [11], which contains 6,556 manually verified products, brands, and organizations spanning 25 main topics and 576 subtopics.

For the search engine scenario, we construct CA-Prod from a commercial dataset of query–ad pairs with human-annotated relevance labels [15]. Following a procedure similar to LM-Market, we cluster ads into six representative domains, such as nutrition supplements, women’s dresses, and Android devices, and assign topics to queries according to their relevant ads. After filtering out extremely sparse or dense cases, we retain 120 queries associated with 2,215 unique products. To mimic the AI overview feature in modern search engines, the inference model \(\mathcal{M}\) is set to generate a concise overview for each keyword query, helping users quickly grasp the topic. The statistics of the curated datasets are summarized in Table 1.

3.2 Measurement Ontology↩︎

Consider a response \(r\) with \(\ell = |r| > 1\) sentences, where \(s_i\) denotes the \(i\)-th sentence and \(\mathbf{s}_i\) its embedding. Let \(\text{sim}(\cdot,\cdot)\) denote cosine similarity, and \(\bar{\mathbf{s}}\) the mean embedding of all sentences. To capture user satisfaction, we define two global metrics response flow and response coherence, and two corresponding local metrics ad flow and ad coherence. Specifically, response flow measures semantic continuity across adjacent sentences: \[f(r) = \frac{1}{\ell-1}\sum_{i=1}^{\ell-1}\text{sim}(\mathbf{s}_i,\mathbf{s}_{i+1}),\] while response coherence evaluates how consistently each sentence aligns with the overall theme: \[c(r) = \frac{1}{\ell}\sum_{i=1}^{\ell}\text{sim}(\mathbf{s}_i,\bar{\mathbf{s}}).\] Ad flow measures whether semantic transitions around an injected ad remain smooth, and ad coherence captures how well the ad aligns with the non-ad content. For engagement, we adopt injection rate to record whether a response contains an ad, and click-through rate to estimate further user conversion.

Beyond similarity-based measures, we include six LLM-as-a-Judge metrics that capture user satisfaction by accuracy, naturalness, personality, and trust, and user engagement by notice and click. Each metric is evaluated along two dimensions. Accuracy evaluates whether the response directly answers the user’s question based on relevance and factual correctness. Naturalness measures how ad insertion affects conversational flow through interruptiveness and authenticity impact. Personality assesses whether the assistant remains user-centered by examining helpfulness and the degree of sales-driven behavior. Trust evaluates users’ confidence in the response by assessing holistic credibility and perceived bias introduced by ads. Notice simulates whether users recognize mentioned products using product awareness and their resulting attitude. Click estimates link-level engagement using link awareness and the likelihood of clicking based on relevance and appeal. For each dimension, the judge assigns one of three quality levels: bad, moderate, or good. These two dimension ratings are then aggregated into the final discrete score using a unified rubric: 0 for both dimensions rated bad; 30 for one bad and one moderate; 60 for both moderate or a combination of bad and good; and 90 for both good. This explicit rubric, together with dimensions grounded in a codebook from real user studies [11], ensures both validity and practical relevance.

3.3 Ad-LLM Framework↩︎

We introduce Ad-LLM, a modular multi-agent framework for AIR generation. As shown in Figure 1, it follows a chain-like workflow with four agents: an ad-free response generator, an ad retriever, an ad injector, and a response rewriter. Given a query \(q\), an ad database \(\mathcal{D}\), and an LLM \(\mathcal{M}\), the framework operates as follows. The response generator first produces an ad-free response \(r\). The ad retriever then selects a small set \(\mathcal{D}_t\) of relevant ads by measuring semantic similarity between either the query or the generated response and the ad contents. The ad injector determines both which ad to insert and where to place it by minimizing the flow disturbance introduced by the insertion. For a response \(r\) with \(\ell\) sentences, inserting an ad sentence \(s_d\) after \(s_i\) yields \(r_{i,d}\). Given retrieved candidates \(\mathcal{D}_t\), we choose \[r' = \arg\min_{i \in [\ell-1],\, d \in \mathcal{D}_t} \Psi(r_{i,d}),\] where \[\Psi(r_{i,d}) = \mathrm{sim}(\mathbf{s}_i,\mathbf{s}_{i+1}) - \frac{\mathrm{sim}(\mathbf{s}_i,\mathbf{s}_d) + \mathrm{sim}(\mathbf{s}_d,\mathbf{s}_{i+1})}{2}.\] This objective quantifies the disturbance at the injection point and selects the ad–position pair that least disrupts the original flow. The response rewriter then refines \(r'\) using \(\mathcal{M}\) to ensure that the injected ad integrates smoothly with the surrounding context. Notably, Ad-LLM also supports multi-\(k\) insertion. After inserting the best ad according to the disturbance objective, the updated response becomes the input for the next selection and insertion step. This iterative procedure continues until \(k\) ads have been inserted, applying the same criterion at each iteration.

4 Experiments↩︎

GEM-Bench aims to answer the following three research questions:

• RQ1: How effective are solutions in terms of quantitative and qualitative evaluation ontologies?

• RQ2: How do humans react to AIR, and how well their evaluations align with LLMs?

• RQ3: How do the core agents and their implementations impact the effectiveness of Ad-LLM?

• RQ4: How much does each solution cost in terms of waiting time and monetary cost?

GEM-Bench evaluates the performance of four solutions: the existing solution Ad-Chat [11] and three instances, GI-R, GIR-R, and GIR-P, of the implemented framework Ad-LLM across the aforementioned datasets and evaluation metrics. For LM-Market, we sample 100 queries from the three domains above for evaluation. We additionally scale the dataset to 1,000 queries to assess robustness, and the results remain consistent (see Appendix). For the three Ad-LLM instances, GIR represents completing all three agents to Generate, Inject, and Rewrite, but GI skips the final rewriting step. The suffixes -P and -R indicate whether the ads are retrieved based on the user prompt or the original response, respectively. Unless otherwise specified, all solutions use doubao-1.5-lite-32k as the base LLM, and their results are evaluated using gpt-4.1-mini. Additionally, for quantitative evaluation and Ad-LLM instances, we set the embedding model to text-embedding-3-small by default. For Ad-Chat, we follow its original implementation and retain the default configuration, including the system prompts. In addition, we examine the impact of varying system prompts and show in Appendix that the performance is insensitive to these prompt changes. All datasets, execution snapshots, and source code for GEM-Bench are available at .

4.1 Effectiveness Evaluation (RQ1)↩︎

Ad-LLM outperforms Ad-Chat in user satisfaction and user engagement. Table 2 and Table 3 report the performance of Ad-Chat and three Ad-LLM instances in terms of both quantitative and qualitative evaluation ontologies across three datasets. Overall, the best Ad-LLM instance improves Ad-Chat by 14.6%, 1.8%, and 6.3% in terms of overall quantitative metrics, and by 14.8%, 15.6%, and 15.64% in terms of overall qualitative metrics on MT-Human, LM-Market, and CA-Prod, respectively. These results demonstrate that generating a raw response and subsequently injecting ads yields better response quality compared to the simpler approach of relying solely on system prompt injection. For specific user satisfaction and engagement dimensions, Ad-Chat consistently shows a substantial performance gap compared to Ad-LLM solutions across all three datasets, particularly in dimensions such as accuracy, personality, and trust. Notably, the best Ad-LLM instance improves Ad-Chat by up to 39.8%, 89.3%, and 62.9% across all datasets in terms of accuracy, personality, and trust, respectively. This can be explained by the fact that the system prompt of Ad-Chat is designed to appeal to users and personalize responses when promoting products. Consequently, this often causes the LLM to adopt a salesman-like role, which can compromise response accuracy and diminish user trust. Moreover, we observe that the injection rate (IR) of Ad-Chat is significantly lower than that of Ad-LLM solutions, even though only queries suitable for ad injection are considered. This is because the injection timing of Ad-Chat is determined by instructions in system prompts, which are difficult to measure and thus uncontrollable. Finally, we find that Ad-Chat achieves a higher click-through rate (CTR) in the search engine scenario, outperforming the best Ad-LLM instance by 6.5%. This suggests that retrieving relevant products using an LLM performs better than relying solely on the similarity of semantic embeddings.

Table [tab:consistent-evaluation] presents the overall qualitative scores of each solution as evaluated by four LLM judges: gpt-4.1-mini, qwen-max, claude-3.5-haiku, and kimi-k2. These judgment models are distinct from the base LLM doubao-1.5-lite-32k, which helps mitigate the alignment issue wherein a judge model may favor outputs generated by models from the same family [16]. Notably, the Ad-LLM solutions consistently achieve the highest scores, with GIR-R ranking either first or second across all datasets according to all four LLM judges. This consistency indicates strong agreement among different LLMs regarding the effectiveness of Ad-LLM. The scores for individual qualitative dimensions closely mirror those reported in Table 3. We also evaluated the performance by fixing the LLM judge to gpt-4.1-mini and varying the base LLM among doubao-1.5-lite-32k, llama3.3-70B, qwen3-30b-a3b, gemini- 2.0-flash-lite, and gemini-2.5-flash-lite. We observe that GIR-R consistently outperforms Ad-Chat in terms of overall qualitative score, and the comparison of solutions in each individual dimension exhibits similar results across different base LLMs. Due to space constraints, we refer interested readers to Appendix for detailed results about each LLM judge and base LLM, respectively.

As shown in Figure 2, Ad-Chat suggests using Megabus, a long-distance coach service that does not operate in Zürich. This response is not only unhelpful but factually incorrect. The Ad-LLM instance GIR-R, on the other hand, provides accurate and actionable directions: take tram line 13 or 14 from Zürich Main Station to the Sihlcity stop, a trip that takes about 10–15 minutes. The promotional element, a recommendation to use Citymapper, is light and relevant, since Citymapper is a tool for urban travel planning. Beyond this example, we observe consistent results in other scenarios, including queries about Seattle attractions, detox meal recommendations, and coworker gift suggestions. Detailed case studies are provided in Appendix.

Figure 5 reports the performance of the best-performing Ad-LLM variant GIR-R on the LM-Market dataset for \(k \in \{1,2,3,4,5\}\). Overall, we observe a clear decrease in response quality as \(k\) increases. In particular, all user-satisfaction metrics decline monotonically under denser ad placement. The Notice metric also decreases, as it measures not only whether users see the injected ads but also their attitude toward noticing them, which becomes more negative when more ads are inserted. In contrast, the Click metric shows a non-monotonic pattern: it initially increases because more injected ads raise the chance that users notice and click at least one link, but it slightly decreases when \(k\) becomes larger, as the overall response becomes less relevant and appealing, reducing users’ willingness to click. This divergence aligns with real-world advertising behavior, where higher ad density increases exposure but reduces user experience.

4.2 User Study (RQ2)↩︎

We recruited 118 participants with diverse demographic backgrounds. The evaluation pool contained 20 queries from three datasets, each paired with two model responses and evaluated on six qualitative dimensions. Each participant rated 10 randomly sampled (query, response) pairs under a randomly assigned dimension. Since some participants contributed multiple submissions evaluating different qualitative dimensions, we obtained 157 valid questionnaires and 3,140 human ratings. On average, each of the 40 (query, response) items received around 65 independent judgments. Further details are provided in Appendix.

We evaluate both the internal consistency of human annotators and their alignment with the LLM-as-a-judge using Spearman correlations. For human reliability, we compute split-half correlation \(\rho_{hh}\) by repeatedly partitioning annotators into two random halves and correlating the resulting item-level average scores. The average correlation across 100 splits is \(\rho_{hh}=0.84\), indicating strong internal agreement among annotators. To assess human-LLM alignment, we correlate human item-level averages with the scores produced by the default judge gpt-4.1-mini. The resulting correlation \(\rho_{hl}=0.73\) reflects substantial agreement between human evaluations and LLM-based judgments.

We also compare the relative preference across the six qualitative metrics. As shown in Figure [fig:human-llm-radar], both humans and the LLM consistently assign higher scores to GIR-R than to Ad-Chat on nearly all dimensions. GIR-R outperforms Ad-Chat in Accuracy, Naturalness, Personality, Trust, and Notice, with an even larger advantage under LLM judgments. This consistent pattern indicates a shared preference for GIR-R and strong alignment between human and LLM assessments.

4.3 Ablation Study (RQ3)↩︎

Based on the results in Table 2, Table 3, and Table [tab:consistent-evaluation], we derive the following insights regarding the three instances of Ad-LLM.

Specifically, GI-R consistently outperforms refinement-based approaches in quantitative overall scores by relative improvements of up to 5.5%, 4.7%, and 2.0% on MT-Human, LM-Market, and CA-Prod. This is because GI-R explicitly determines the optimal injection position based on response flow, where flow is grounded in semantic similarity and thus directly benefits metrics such as RF, RC, and AC. Correspondingly, GI-R suffers relative degradations of up to -6.7%, -4.4%, and -2.0% in qualitative overall scores. The gap is especially pronounced in naturalness, where GI-R falls behind by as much as –25.8%. This degradation occurs because GI-R directly injects ad sentences without rewriting, making the insertion abrupt and less coherent with the context. By contrast, GIR-R and GIR-P incorporate rewriting to smooth the transition, thereby improving naturalness, trust, and personality, which explains their superior qualitative outcomes.

For the MT-Human and LM-Market datasets, response-based retrieval GIR-R consistently outperforms prompt-based retrieval GIR-P. Notably, GIR-R achieves the best overall qualitative score, surpassing GIR-P by 1.3% as reported in Table 3. This superiority is consistently observed across different judge models. For example, under kimi-k2, GIR-R outperforms GIR-P with a relative gain of 3.9%. This advantage stems from the nature of chatbot interactions, where questions and responses often involve complex semantic relationships. For instance, when a user asks, “I often feel tired at work, what would you recommend”, the model’s response provides nuanced context that helps anchor ad injection in a meaningful way. In contrast, prompt-based retrieval GIR-P achieves superior qualitative performance on the CA-Prod dataset. As shown in Table 3, GIR-P achieves a qualitative overall score of 58.41%, improving upon GI-R by 7.2%. This trend is also consistent across judge models. For example, under kimi-k2, GIR-P surpasses GIR-R by 0.5%. This is because, in the search engine scenario, user queries tend to be keyword-oriented, such as “best wireless headphones” or “cheap running shoes”. In such cases, the ad sentence acts almost like a direct extension of the query, making prompt-based retrieval naturally more aligned with user intent.

We also evaluate the impact of different embedding models during product retrieval by taking the GI-R solution as an example and replacing text-embedding-3-small with all-MiniLM-L6-v2. The results are shown in Table 6.

Specifically, text-embedding-3-small achieves an overall qualitative score of 69.70% and 70.46% on the MT-Human and LM-Market datasets, representing relative improvements of 1.3% and 2.8% over all-MiniLM-L6-v2, respectively. This advantage is especially notable in key dimensions such as naturalness, where the improvement is 7.03% on MT-Human and 4.38% on LM-Market. These results indicate that a stronger embedding model is more effective at capturing nuanced semantic relationships within chatbot-style queries and responses. In contrast, on the CA-Prod dataset, the two models perform almost identically, with differences within 0.04%. This suggests that in keyword-oriented search scenarios, the choice of embedding model has limited impact, as retrieval is primarily driven by surface-level lexical matches.

4.4 Cost Study (RQ4)↩︎

In the final set of experiments, we evaluate the time and monetary costs of different solutions using the metrics Extra Input Tokens To First Token (ITTFT), Extra Output Tokens To First Token (OTTFT), and their aggregation (Overall). Specifically, ITTFT and OTTFT measure the average number of input tokens and generated output tokens, respectively, before returning the final response, which indicates the additional overhead for ad injection. Moreover, ITTFT and OTTFT fundamentally reflect the waiting time before a user sees the first response token, independent of latency introduced by different LLM API requests. We do not compute the average price by combining both ITTFT and OTTFT directly, as their unit prices vary across platforms. However, we observe that the ratio of input-to-output unit prices is approximately 1:2 for doubao-1.5-lite-32k. Therefore, we define the aggregated cost (Overall) as \(0.5 \times \text{ITTFT} + \text{OTTFT}\).

Table 7 reports the cost of each solution across different datasets. We observe that GI-R consistently incurs the lowest cost. In contrast, GIR-R and GIR-P consume a similar number of input and output tokens, around 10 times and 2 times more than GI-R, respectively. This is because both GIR-R and GIR-P include the final Response-Rewriter stage, which refines the ad-injected content to ensure high response quality. Another notable observation is that although Ad-Chat and GI-R generate a similar number of additional output tokens, Ad-Chat consumes around 5 to 25 times more input tokens than GI-R. This is attributed to Ad-Chat’s use of LLM agents throughout its workflow, which introduces substantial input token overhead during topic assignment and product selection stages. This overhead is particularly pronounced in the CA-Prod dataset, where Ad-Chat first uses an LLM to select the best-fit topic from six candidates and then selects one product from an average of 370 candidates per topic. Despite incurring the highest overhead on CA-Prod, Ad-Chat achieves the highest CTR, benefiting from its LLM-based product selection strategy.

5 Additional Related Works↩︎

Existing LLM benchmarks focus on evaluating models across different dimensions, including software engineering abilities [7], [17]–[20], tool usage [21]–[23], and reasoning in challenging real-world scenarios such as online shopping and web browsing [19]–[21], [24]–[27]. Some recent benchmarks specifically target human-interactive tasks [23], [28], including social and task-oriented dialogue. Other efforts further emphasize multi-turn conversational settings [7], [22], [23], [28]–[30]. For example, MT-Bench [7] provides a multi-turn question set that covers diverse tasks such as roleplay, extraction, reasoning, and knowledge retrieval. It leverages LLM-as-a-Judge to approximate human preferences, showing over 80% agreement with human evaluators. Their judging method considers general response quality factors such as helpfulness, relevance, and accuracy. However, these are not specifically designed for AIR.

[31] propose a benchmark for evaluating ad creatives using reference-based metrics such as ROUGE and BLEU. However, such metrics are sensitive to reference choice and may yield unfair results. For instance, Ad-LLM solutions typically insert or refine content based on the original response, often resulting in disproportionately high scores. [32] introduce a benchmark for assessing the quality of detailed landing pages in SEM, which is beyond the scope of this work. Similarly, [33] address the complementary task of detecting ads inserted into text, constructing an ad-injected dataset using an approach similar to Ad-Chat [11]. Despite these efforts, no benchmark currently exists for AIR evaluation.

Early solutions for SEM [34] propose semantic embedding methods to match queries and ads beyond exact keyword overlap. More recently, LLM-based methods such as VALUE [35] leverage generative rewriting to align user queries with advertiser bidwords while optimizing both relevance and monetization. Beyond query rewriting, recent works study how LLM-native content should be auctioned in GEM. At the token level, [36] design auctions over LLM token outputs to ensure incentive compatibility. At the passage level, [37] introduce retrieval-augmented segment auctions for ad placement, while [38] study auctions for LLM-generated summaries that generalize traditional position auctions. Together, these efforts highlight the growing role of LLMs in advertising, but they address different aspects of the problem and remain orthogonal to our focus.

6 Conclusions↩︎

In this work, we introduce GEM-Bench, the first comprehensive benchmark for ad-injected LLM response generation. It consists of three curated datasets, a multi-faceted measurement ontology, and a highly extensible multi-agent framework designed to support solution development and reproducible evaluation. Our preliminary experiments highlight a clear trade-off between prompt-based methods and ad-insertion strategies in terms of effectiveness and computational overhead. At present, GEM-Bench is limited by the relatively small size of its datasets and its human evaluation. We plan to address these limitations in future work and release updated versions of the benchmark.

7 Ethics Statement↩︎

During the development of GEM-Bench, we identified several ethical considerations. First, ad-injected generative models may influence user decision-making and potentially manipulate behavior. To mitigate this, our benchmark emphasizes user satisfaction metrics beyond engagement, such as trust, personality, and naturalness, thereby discouraging deceptive or harmful advertising practices. Second, dataset construction and experimental studies were conducted without collecting sensitive data, ensuring compliance with privacy and data protection standards. Third, we recognize the potential for commercial bias, as generative models exposed to ad injection could reinforce discrimination, unfair targeting, or misinformation. During the preparation of this manuscript, a large language model (ChatGPT) was used only to polish the writing. All research ideas, methods, and results are original to the authors. By making these risks explicit, GEM-Bench is intended as a diagnostic tool to encourage safeguards and support responsible deployment, rather than to optimize advertising in user-facing systems.

In addition, we position our work from the platform’s perspective and emphasize that ad-injected generation requires transparent disclosure practices similar to the “Sponsored” tag used in search engines. The specific design of disclosure and the study of user acceptance remain open research questions that call for collaboration across academic and industrial communities, as well as updated advertising regulations to support fairness and accountability in generative systems. From the same platform-centered viewpoint, ad safety control is a fundamental requirement. Our framework assumes ads originate from a verified and policy-compliant database curated by the platform, and content moderation and authenticity screening are applied before both the bidding and retrieval stages. Quality-based bidding scores [39]–[41] that combine relevance, trust, and historical reliability can further down-rank or exclude low-trust advertisers, ensuring that only credible ads are eligible for injection.

8 Reproducibility Statement↩︎

The detailed processes of data curation, metric design, and multi-agent implementation are provided in Appendices 9, 10, and 11. The prompt used for LLM-as-a-Judge evaluation is illustrated in Appendix 18. To ensure reproducibility, we set the temperature of all generation models and judgment models to 0. Furthermore, we repeat the generation and judgment nine times and preserve the snapshots of latest three runs in the repository.

9 Datasets↩︎

9.1 Chatbot: MT-Human and LM-Market↩︎

The MT-Human dataset is extracted from MT-Bench [7], a benchmark for evaluating LLMs through multi-turn dialogues. In MT-Bench, the dataset consists of 80 user queries across 10 categories (writing, roleplay, math, coding, extraction, STEM, and humanities), with 10 queries in each category. We manually examine each query for its suitability for ad insertion and retain all 10 first-turn queries from the humanities category to form the MT-Human dataset. It includes queries such as “What are some business etiquette norms when doing business in Japan”.

LM-Market is processed based on LMSYS-Chat-1M [8], a dataset containing 1 million real user–LLM chats collected from the Vicuna demo and Chatbot Arena. Each conversation can have multiple turns and be in different languages. Due to its scale, manual curation is infeasible. To address this, we design a multi-stage filtering and clustering pipeline. First, we filter for single-turn, English conversations, yielding 540,686 queries. From these, we further select those labeled under the Marketing category, resulting in 36,605 queries. Second, we cluster the remaining queries into topic clusters. We compute text embeddings for each query using OpenAI’s text-embedding-3-small model and apply BIRCH [42] to group them into 3,758 leaf-level clusters. To create higher-level groupings, we represent each leaf-level cluster by its centroid embedding, and then apply hierarchical linkage clustering to organize these clusters into a bottom-up tree structure. From this hierarchy, we select the level that balances the clustering granularity and human reviewing workload, yielding 20 clusters that together cover all queries. Third, to understand the topic of each cluster, we select the 100 queries closest to the cluster centroid and use GPT-4o to summarize the topic and provide a description. Among the 20 clusters, we retain Travel Planning, Recipe Recommendation, and Software Tools Comparison, as they exhibit higher potential for ad insertion, yielding 1,701 queries. To further refine this set, we employ the LLM doubao-1.5-lite-32k to assign each query a score on a 0–10 scale representing the likelihood that an advertisement can be naturally integrated into its response. From the queries with scores above 5, we randomly sample 100 and then manually review them, selecting 100 queries for inclusion in LM-Market.

In addition to the user query set, a predefined advertisement database \(\mathcal{D}\) is required for the chatbot scenario. We adopt the product set constructed and curated in Ad-Chat [11]. Specifically, Ad-Chat first leverages Google’s Topics API list of ad interest categories [14], which provides a two-level taxonomy consisting of 25 main topics and 576 subtopics. For each subtopic, GPT-3.5-Turbo is used to generate 10 advertisements, resulting in a database \(\mathcal{D}\) of 6,556 distinct products, brands, or organizations. Each advertisement entry includes a brand name, description, and URL, and has been manually verified by the authors.

9.2 Search Engine: CA-Prod↩︎

For the search engine scenario, we construct the CA-Prod dataset using a commercial advertising dataset [15], which contains 300,000 query–ad pairs from a commercial search engine. Each pair consists of a keyword query, ad metadata, and a manual label indicating whether the ad is relevant to the query. An ad is considered positive if labeled relevant, and negative otherwise. Each query corresponds to a list of ads displayed on the results page along with their labels. We filter out records with missing fields and retain only queries that have both positive and negative ads.

To further refine representative queries and their product lists, we adopt a three-step process: (i) identify product topics, (ii) assign topics to queries, and (iii) sample queries and product lists. In the first step, we follow the same processing method as in LM-Market and cluster the ads into six topics: lawn and garden equipment, slip-on shoes, modern household items, nutrition supplements, Android tablets and smartphones, and women’s dresses. Specifically, we concatenate each ad’s metadata (title, description, URL, and advertiser) and generate a semantic embedding for each ad. Based on these embeddings, we apply K-means clustering and search for the number of clusters that yields the best silhouette score, which results in six final clusters. To obtain human-readable labels, we select the 100 ads closest to each cluster centroid and use GPT-4o to summarize their names and descriptions. In the second step, we assign a topic to each query based on the majority topic of its positive ads. To remove extremely sparse or dense query–ad lists, we retain only queries with 6–50 ads and a positive-to-negative ratio between 10% and 80%. In the final step, for each topic cluster we randomly sample 20 queries along with all their associated ads from the remaining pool. This process yields 120 queries and 2,215 unique products.

10 Measurement Ontology↩︎

10.1 Quantitative Metrics↩︎

To capture user satisfaction, we focus on response quality and propose four similarity-based metrics: two global metrics (response flow and response coherence) and two local metrics related to the inserted ad (ad flow and ad coherence). We also consider injection rate and click-through rate to reflect advertiser needs. Consider a response \(r\) with \(\ell = |r| > 1\) sentences, where \(s_i\) denotes the \(i\)-th sentence (\(i \in [\ell]\)). Let \(r_{i,d}\) denote the response obtained by inserting an ad sentence \(s_d\) immediately after \(s_i\).

For a response \(r\) with \(\ell\) sentences, response flow is defined as the average similarity between all consecutive sentences: \[\label{eq:res-flow} f(r)=\sum_{i\in[\ell-1]}\frac{\text{sim}(\boldsymbol{s}_i,\boldsymbol{s}_{i+1})}{\ell-1}.\tag{1}\] Here, \(\text{sim}(\boldsymbol{s}_i,\boldsymbol{s}_{i+1})\) is the cosine similarity between the embeddings of \(s_i\) and \(s_{i+1}\). Intuitively, response flow captures semantic continuity, reflecting how smoothly ideas transition. A higher value indicates more natural flow.

Given a response \(r\), it measures the average similarity of each sentence to the overall semantic center of the response: \[\label{eq:res-coh} c(r)=\sum_{i\in[\ell]}\frac{\text{sim}(\boldsymbol{s}_i, \bar{\boldsymbol{s}})}{\ell},\tag{2}\] where \(\bar{\boldsymbol{s}}\) is the mean embedding of all sentences, indicating the embedding of the main idea of \(r\). This metric captures how consistently the sentences relate to a central topic. Higher values indicate stronger coherence.

However, these global metrics may be insensitive to ad injection and be dominated by non-ad content, since responses typically contain only one or a few ad sentences. To address this, we introduce local metrics for flow and coherence.

For a response \(r_{i,d}\) where a single ad sentence \(s_d\) follows \(s_i\), ad flow is defined as \(\exp\!\left(-\left|\text{sim}(\boldsymbol{s}_i, \boldsymbol{s}_d) - \text{sim}(\boldsymbol{s}_d, \boldsymbol{s}_{i+1})\right|\right)\). The term captures the imbalance of flows on both sides of the ad. Smaller differences indicate smoother integration, while larger ones reduce the score via exponential decay.

Compared to Eq. (2 ) measuring coherence to the overall response, ad coherence focuses on how ad sentences align with the non-ad content. It takes the average similarity between each ad and the mean embedding of all non-ad sentences, reflecting how well the ad fits the main topic. Higher values indicate better alignment.

This metric measures whether an ad is inserted into the response \(r\), with a value of 1 if an ad is present and 0 otherwise. Injection rate serves as a proxy for the system’s ability to deliver promotional content and represents the upper bound of the probability that a user notices an ad. Higher values indicate more consistent ad delivery.

Click-through rate (CTR) measures user engagement with ads, defined as the proportion of ad impressions that result in a click. A higher CTR reflects ads that are more attractive, relevant, or persuasive to users.

10.2 Qualitative Metrics↩︎

We further propose the metrics of accuracy, naturalness, personality, and trust for measuring user satisfaction, and the metrics of user notice and click for user engagement. These metrics are question-based evaluations implemented with the LLM-as-a-Judge framework [7]. For each metric, the judge evaluates a response along two dimensions, assigning a quality level of bad, moderate, or good for each. The two ratings are then aggregated into a final score of 0, 30, 60, or 90: a score of 0 indicates both dimensions are bad; 30 indicates one is bad and the other moderate; 60 indicates both are moderate or a combination of bad and good; and 90 indicates both are good. In the following, we elaborate on each metric and its corresponding dimensions.

The first metric evaluates whether a response resolves the user query in terms of both relevance and accuracy, which should be the user’s primary need. Relevance refers to how directly and appropriately the response addresses the user’s specific question or request. Accuracy considers whether the information provided is factually correct, precise, and reliable.

This metric measures the extent to which ad insertion disrupts the flow and naturalness of the conversation, based on interruptiveness and authenticity. Interruptiveness examines whether the ad creates a “jump out” or “abrupt” feeling during reading, breaking the user’s continuous focus on the topic. Authenticity evaluates whether the ad undermines the “human touch” or “natural flow” of the conversation, making the response seem rigid, formulaic, and less authentic.

This metric evaluates the response along two core dimensions: helpfulness and salesmanship. Helpfulness indicates the extent to which the chatbot’s response truly solves user problems and meets their needs, rather than merely guiding purchases or promotions. Since the primary role of most LLMs is to act as helpful AI assistants, this dimension is crucial. Salesmanship, by contrast, assesses whether the chatbot is overly focused on promoting or advertising products at the expense of being user-centered.

This metric measures users’ perception of the credibility and fairness of the response after ad insertion. Credibility refers to the strength of the user’s trust in the response: do the ads make users suspect hidden motives or commercial bias behind the content? Perceived bias evaluates whether the ads create a sense of “sponsor preference” or “promotional tendency” that undermines the neutrality or fairness of the answer.

This metric evaluates whether users notice the products in responses and how they perceive them, considering two dimensions: notice and attitude. The notice dimension captures whether users recognize any products, brands, or sponsored content in the response. The attitude dimension reflects how users are likely to react to the noticed products or brands, which may be positive, negative, or neutral.

This metric simulates whether users will click on products or brands in chatbot responses, considering both notice and click behavior dimensions. Notice refers to whether users are aware of any sponsored links in the response. Click measures how likely users are to click on the noticed sponsored links based on their relevance, appeal, and context.

11 Ad-LLM Implementations↩︎

Algorithm 3 illustrates the workflow of Ad-LLM. Specifically, it takes as input a user query \(q\) along with other possible context, an ad database \(\mathcal{D}\), and an LLM \(\mathcal{M}\) for response generation. During the offline indexing stage, Ad-LLM first serializes each row of ad \(d \in \mathcal{D}\) into natural language in the format column name: entity value. It then computes the semantic embedding \(\mathbf{d}\) of this textual representation of \(d\) using a predefined text embedding model. These embeddings are indexed for online stages. The online AIR generation (Lines 2–5) can be regarded as an application of retrieval-augmented generation (RAG), which sequentially invokes the following four agents and ultimately outputs the AIR\(r_d\). Ad-LLM first invokes the Response-Generator agent to generate an original ad-free response \(r\) based on \(q\) and \(\mathcal{M}\), where the system role is set to a helpful AI assistant by default. The query \(q\) and response \(r\) are then passed to the Ad-Retriever agent to retrieve a set \(\mathcal{D}_t\) of the top-\(t\) relevant ads from \(\mathcal{D}\). Notably, retrieval can be performed using either the text embedding of the query \(q\) or the raw response \(r\), with \(t\) set to 5 by default. Next, the Ad-Injector agent selects the best ad from \(\mathcal{D}_t\) and its injection position within response \(r\) such that the flow disturbance after injection is minimized. We denote \(r'\) as the response after injecting the natural language of the selected ad at its corresponding position. Finally, the Response-Rewriter agent refines \(r'\) using \(\mathcal{M}\), with the goal of adjusting the surrounding context to ensure that the ad snippet fits naturally into the flow, without altering other unrelated content.

At the core of Ad-LLM is selecting an appropriate ad and its insertion position in Ad-Injector. We formulate this as minimizing flow disturbance. As defined in Section 10.1, let \(r_{i,d}\) denote the response obtained by inserting an ad sentence \(s_d\) after \(s_i\) and \(\mathcal{D}_t\) be a set of \(t\) ad candidates. The objective is \[\label{eq:injection} r' = \arg\min_{i \in [\ell-1],\, d \in \mathcal{D}_t}\Psi(r_{i,d}),\tag{3}\] where \(\Psi(r_{i,d})\) measures the change in flow caused by the insertion. A simple disturbance function compares the local flow before and after insertion: \[\Psi(r_{i,d}) = \text{sim}(\boldsymbol{s}_i,\boldsymbol{s}_{i+1}) - \frac{\text{sim}(\boldsymbol{s}_i,\boldsymbol{s}_d) + \text{sim}(\boldsymbol{s}_d,\boldsymbol{s}_{i+1})}{2}.\] Intuitively, this captures the drop in response flow at the injection point, since the rest of the response remains unchanged.

12 Effectiveness Evaluation with Increased Query Volume↩︎

To further verify robustness, we followed the default configuration and scaled the LM-Market dataset to 1,000 queries. We compared both the baseline Ad-Chat and the best-performing solution Ad-LLM(GIR-R) for AIR generation, resulting in average injection rates of 0.853 and 1.000, respectively. Table 8 reports performance across the six proposed qualitative dimensions. GIR-R consistently outperforms Ad-Chat across all metrics, confirming that the benchmark conclusions remain valid and stable at larger scales.

13 Effectiveness Evaluation with Different System Prompts↩︎

Besides the default system prompt (called interest-based) used in the original Ad-Chat, the authors also provide additional system prompts in the released implementation. Specifically, the default serves as a neutral configuration that instructs the model to generate helpful responses while smoothly inserting the product, its URL, and key features when relevant. In contrast, the alternative prompt, named user-centric, is a user-oriented variant that asks the model to present the product in a more personalized and appealing way, incorporating user-centric cues while inserting the product and URL when appropriate. As reported in Table 12, the performance of Ad-Chat under different system prompts is comparable across all three datasets and six qualitative metrics. This indicates that the behavior of Ad-Chat is relatively insensitive to prompt-level variations, and its ad-injection effectiveness remains stable regardless of the system prompt used.

14 Effectiveness Evaluation across Different Judge LLMs↩︎

Table 9, Table 10, and Table 11 present the qualitative results across all datasets and solutions by using LLM qwen-max, kimi-k2, and claude-3.5-haiku as judges, respectively. Across different evaluators, the relative ranking of solutions remains consistent: GIR-R and GIR-P generally achieve the strongest overall performance, followed by GI-R, while the baseline Ad-Chat lags behind.

15 Effectiveness Evaluation across Different Base LLMs↩︎

In addition to doubao-1-5-lite, we further evaluate both the baseline Ad-Chat and the best-performing solution GIR-R on the LM-Market and CA-Prod datasets across four additional base LLMs: qwen3-30b-a3b, llama3.3-70B, gemini-2.0-flash-lite, and gemini -2.5-flash-lite.

Table 13 reports the overall qualitative scores of different base LLMs. We observe that GIR-R consistently outperforms Ad-Chat across all models and both datasets. In particular, GIR-R achieves an average relative improvement of around 10% on LM-Market and around 16% on CA-Prod over Ad-Chat. Furthermore, comparing gemini-2.0-flash-lite and    gemini-2.5-flash-lite, we observe that although the superiority of GIR-R remains consistent, stronger base LLMs naturally yield higher absolute generation quality for both solutions, indicating that AIR generation method benefits from and scales with stronger underlying models.

Table 14 and Table 15 provide detailed per-dimension results. To summarize, across all evaluated base LLMs, GIR-R consistently outperforms Ad-Chat on the four user-satisfaction dimensions (accuracy, naturalness, personality, trust). The two methods show comparable performance on engagement metrics (notice, click), with Ad-Chat occasionally showing slightly higher scores because it tends to expose ads more directly, whereas GIR-R focuses on flow-preserving insertion that prioritizes user satisfaction. These results confirm that our benchmark conclusions are stable and robust across diverse and stronger base LLMs.

16 Case Study↩︎

The comparison between Ad-Chat and GIR-R shows that GIR-R consistently produces responses that are both relevant and useful, whereas Ad-Chat often allows advertisements to overshadow or even replace the actual answer. Ad-Chat tends to insert heavy, off-topic promotions that make the response feel more like an advertisement than a helpful reply. In contrast, GIR-R maintains response quality by first addressing the user’s question in detail and then adding light, contextually related advertising. In the following examples, the ad-related content is highlighted in blue.

In Figure 4, Ad-Chat begins by listing a few relevant landmarks but then abruptly pivots to promoting Sydney, Australia, even including a tourism website link. This shift makes the response misleading and distracts from the user’s request about Seattle. GIR-R, by contrast, provides a thorough list of attractions actually in Seattle, such as the Space Needle, Pike Place Market, and Chihuly Garden and Glass, along with local highlights like Discovery Park and Gas Works Park. The only advertising comes in the form of a mention of Sound Transit, which is relevant to the travel context and does not detract from the main answer.

In Figure 5, Ad-Chat briefly mentions healthy ingredients but quickly shifts into marketing a branded “Quick Cleanse Kit,” describing teas and supplements rather than meals. This fails to answer the user’s request for detox meals and instead turns the response into a product pitch. GIR-R, by contrast, offers a clear explanation of detox meals, including fruits, vegetables, lean proteins, and healthy fats, along with their benefits. It even provides a sample daily menu, directly addressing the user’s query. While GIR-R does close with a product promotion, the ad appears only after a strong and relevant answer, making it less disruptive.

In Figure 6, Ad-Chat recommends buying a Dell laptop or gaming system. While related to IT, these are unrealistic and expensive choices for a coworker gift, making the answer impractical. GIR-R instead offers a broad range of thoughtful and affordable options, organized into categories: peripherals such as a mouse or headset, gadgets like a portable SSD or smartwatch, IT-related books and magazines, office décor, and even experience-based gifts such as courses or conference tickets. These suggestions are more appropriate, practical, and consistent with workplace gifting norms. Advertising is also lighter: Ad-Chat reads like a direct Dell promotion, whereas GIR-R includes only subtle brand mentions, such as Logitech, embedded within otherwise useful recommendations.

17 User Study Configuration↩︎

We recruited 118 participants with diverse demographic backgrounds, including variation in age, gender, education level, industry, and frequency of AI usage. Detailed demographic statistics are presented in Figure 7. Participants were invited to evaluate responses produced by two models, the baseline Ad-Chat and the best-performing solution Ad-LLM(GIR-R). Each participant assessed the model outputs along one qualitative dimension selected from accuracy, naturalness, personality, trust, notice, and click. All ratings followed a four-point Likert scale, using the same dimension-specific descriptions employed in the system prompts of our LLM-as-a-judge setup, which ensures full comparability. We selected 20 queries through stratified sampling across three datasets, including 2 queries from MT-Human, 13 from LM-Market, and 5 from CA-Prod. To avoid participant fatigue, because a full evaluation would require 240 ratings per person, each participant was assigned a simplified task consisting of 10 randomly sampled queries, one randomly selected qualitative dimension, and two responses per query. The responses from Ad-Chat and Ad-LLM were presented side by side, and the left or right placement of each model’s output was randomized. Each resulting judgment corresponds to a tuple containing the query, the method response, the evaluation dimension, and the assigned score.

In total, 184 questionnaire submissions were collected, containing 3,680 human rating records. Some participants contributed multiple submissions because they evaluated different qualitative dimensions. We removed low-quality submissions whose user–group Spearman correlation was no greater than zero. The user–group Spearman correlation measures how closely a participant’s rating scores follow the consensus trend formed by all other raters on the same set of items. A non-positive value indicates that the participant provided either random noise or systematically opposite judgments, both of which suggest unreliable or inattentive responses. After preprocessing, we obtained 157 valid questionnaires and 3,140 corresponding rating records. Across the 40 combinations of query and method’s response, each item received an average of 65 independent human judgments. The resulting data volume provides sufficient coverage for reliable split-half estimation of human consistency and supports stable item-level correlation analyses with LLM-as-a-judge scores.

18 LLM-as-a-Judge Implementations↩︎

The prompt templates designed for each judge are shown in Figure 8, Figure 9, Figure 10, Figure 11, Figure 12, and Figure 13.

References↩︎