1 Introduction↩︎

Responsible AI (RAI), which includes fairness and bias, safety, privacy, veracity, robustness, explainability, security, transparency, and governance, is increasingly important due to growing focus on regulating the development and use of AI [1]. There are many valuable public benchmarks to evaluate and compare LLMs for answer quality and RAI dimensions such as veracity, toxicity and fairness in a generic way [2]–[4]. However, a limitation of these benchmarks is that they are intentionally general, designed for a broad use case. When evaluating a system for RAI dimensions it is necessary to first define an application, and based on that application, establish suitable criteria for each dimension. For example, an application that writes product descriptions for children’s Halloween costumes will have different fairness and safety requirements than an application that writes summaries of horror films. Designing an evaluation dataset that is specific enough to measure RAI in a meaningful way, but not so specific that the dataset is only useful for a niche application, is a challenge.

Application-based assessments answer two critical user questions: 1.)

2.)

To fill this gap we present data designed for a text generation use case, specifically the generation of a paragraph description from a bulleted list of attributes. We curate the data with an e-commerce seller in mind, using an LLM to generate product descriptions given a set of product features as input.

We construct query templates representing demographic identity groups, product adjectives and categories. These attributes support the downstream assessment of fairness in systems that use the dataset as input. We formulate queries from the templates and submit them to a large e-commerce search engine. We retrieve the top \(k<=40\) results for each query and collect the product details for each search result. After cleaning, the resulting dataset contains rows, each with a product and its features, labeled for fairness attributes, along with the query template used to retrieve the product. The dataset is available for download under the Creative Commons BY 4.0 license at https://github.com/amazon- science/application-eval-data.

In this paper we review similar datasets (Section 2), describe the construction of the dataset in detail (Section 3), and demonstrate how to use this data in a responsible AI evaluation of the overall quality of the generated text, as well as veracity, toxicity and fairness (Sections 4 and 5). Finally, we discuss the limitations of the dataset, and future directions (Section 6).

2 Related Datasets↩︎

Existing large public benchmarks (e.g., HELM [3], FAIR Enough [5], and Decoding Trust [2]), standardize evaluation across use cases, and provide a high-level summary of model performance including RAI dimensions for different tasks, but do not capture RAI requirements specific to an application.

Kaggle¹ offers a large number of datasets, including data labeled for RAI dimenions. Three commonly used datasets for evaluating safety were curated for the Jigsaw challenge [6]–[8], and include multilingual data, and a dataset labeled with protected attributes (gender, ethnicity, religion, sexual orientation and disability) to measure unintended bias in offensive content detection. A limitation of the data is that it reflects a definition of offensive content that may not be appropriate for a given application. For example, descriptions of adult products may include sexual terms inappropriate for a general setting, and labeled as "toxic" in these datasets.

The Jigsaw corpora are included, along with other hate speech corpora, in the MetaHate corpus [9], which has 1.2M social media comments labeled for hate speech. The data does not include labels for the target of the hate speech (although some of the corpora included are focused on specific target groups).

The Amazon Reviews Datasets [10]–[12] include eCommerce reviews and product meta-data scraped from Amazon.com. The original collection contains over 200M reviews of products from 29 categories. While the data could be used for an application-specific evaluation of text generation quality, it does not contain attributes needed for fairness evaluations. It does not contain high-risk product categories (such as adult products), and the reviews have been filtered by Amazon for toxic language, making it inadequate to evaluate safety. Derived versions on Kaggle contain the review data but omit the product meta-data completely (c.f. [13]).

Zhang et al. [14] constructed a collection of 31,000 product Questions paired with 60,000 answers sampled from the 2016 Amazon Reviews Dataset [11] for veracity measurement. The data includes product categories Electronics, Home and Kitchen, Sports and Outdoors, Health and Personal Care, and Cell Phones and Accessories. The QA pairs are labeled on a 5-point scale from True to False, according to community votes. The data is not labeled for fairness evaluation, and does not include adult or sensitive content, making it unsuitable for evaluating safety.

3 Dataset Construction↩︎

We constructed the dataset to evaluate the RAI dimensions of highest risk for our use case. For a seller generating product descriptions, these are: Quality (well aligned with what a human would write); Veracity (true and complete product facts, avoiding untrue claims); Safety (no harmful or toxic language); Fairness( generated descriptions score well for a variety of product types and target customers, with no large discrepancies).

The dataset includes ground truth product descriptions for quality and veracity, benign and sensitive categories for safety evaluation, and product categories associated with men and women for fairness evaluation. To collect a diverse set of products, we constructed a set of product search queries. Queries are composed of pairwise combinations of a product adjective, a product category, and an identity group, as in “\(\texttt{<adjective>}\) products for \(\texttt{<identity\_group>}\) people”, or “products for \(\texttt{<identity\_group>}\) people in \(\texttt{<category>}\)”.

We employed 13 identity groups from the Toxigen dataset [15], identified in a bottom-up data labeling approach. They include attributes such as race, ethnicity, age, religion, disability status, sexual orientation, and gender identity, which are critical demographic cohorts for studying fairness (e.g., [16] [17], [18]).

We selected a small set of gendered adjectives based on the analysis in [19]. In that work, gendered word lists were identified using distance in embedding space to conceptual clusters around the terms man and woman. We used these word associations and word lists to find adjectives that can modify a product search, for example cute, strong or sexy. Gendered word clusters from [19] align well to product categories from the Amazon.com catalog. We selected eight categories associated with man and eight associated with woman .

The full list of query modifiers is shown here, with high-risk categories marked with asterisk (\(^\ast\)). Adjectives: {any, superior(m), essential(m), solid(m), adorable(w), unique(w), inexpensive(w)};
Categories: {any, Automotive(m), Electronics(m), Sports & Outdoors(m), Appliances(m), Industrial & Scientific(m), Shooting(m)\(^\ast\), Knives, Parts, & Accessories(m)\(^\ast\), Weapons(m)\(^\ast\), Beauty & Health(w), Clothing, Shoes, Jewelry, & Watches(w), Kitchen & Dining(w), Arts, Crafts & Sewing(w), Gardening & Lawn Care(w), Sexual Wellness(w)\(^\ast\), Tobacco-Related Products(w)\(^\ast\), Lingerie(w)\(^\ast\)};
Identity Groups: {any, African, Asian, Native American, Latino, Chinese, Mexican, Middle Eastern, LGBTQ+, Women, Mental Disabilities, Physical Disabilities, Jewish, Muslim}.

We also balanced the product categories with high and low risk of toxic language in the LLM output. We selected six high-risk categories and labeled each category with its gender association. The risk of each category was assigned through consensus among the team designing the dataset (5 people), and preliminary experiments confirmed that the rate of toxic model output can be higher for these six categories.

These methods generated 382 source queries, which we submitted to the Amazon.com website to retrieve \(k<=40\) products associated with each query. Products may be repeated among multiple queries, and some queries yielded fewer than 40 search results. The resulting dataset contains rows, with unique products. Table 1 shows an overview of the dataset size.

Each row contains the fields shown in Table 2. The title, description, and feature bullets are all provided by the product seller. We take these fields as ground truth, given that they have been approved for publication by both the seller and the platform. Even in cases where the seller may have used a model to generate these fields, they represent desirable outputs for comparison.

4 sec:Experimentation↩︎

To show how the dataset can be used for LLM evaluation, we present a sample analysis of quality, safety, veracity, and fairness of the Llama 3.2 11B model [20]. For each row in the dataset, we constructed a zero-shot prompt (Table 3) asking the model to generate a short description of a product (fewer than 125 words which is near the median length of human-written product descriptions), given the product category, title and feature bullets, and the adjective used to retrieve the product. We define the metrics for quality, veracity, safety, and fairness as follows:

Quality and Veracity depend on the similarity of LLM output to the ground truth product description (described in Section 3). To measure quality, we compute semantic accuracy as the overall semantic similarity (BertScore F1 [21] rescaled to [0,1]) of the LLM output compared to the ground truth. For veracity, we apply BertScore components to informational elements, calculating precision and recall.

Safety: We use the toxicity metric from the unbiased detoxify toxicity classifier [22], which assigns each LLM-generated description a score in the range [0,1], where higher values indicate greater likelihood of toxic content.

Fairness: We apply the meta-metric cohort disparity for both toxicity and accuracy scores. For a given metric, we report the ratio of best-performing cohort on that metric to the worst-performing cohort. Using the query templates from Section 3, we define cohorts by identity group, product category, and query adjective.

We choose a simple set of metrics to demonstrate the utility of the dataset. System developers will apply their own metrics of interest, which may change over time. However the dataset is structured to support a variety of RAI dimensions, as shown here.

5 Results↩︎

Table 4 shows results for basic metrics from Section [sec:metrics].

Quality, measured by BertScore accuracy, has a mean of . It varies little across the dataset. This indicates high overall similarity between human and LLM-generated descriptions.

Veracity, measured by BertScore precision and BertScore recall, shows more variation. Some LLM outputs include hallucinated words that bring precision down to a low of , or omit information for a minimum recall score of . For example, we observe products in the data where ground-truth descriptions focus on the benefits of the product ("help active thinking") while the LLM output adheres strictly to the product features ("made with safe parts").

Safety, measured by detoxify scores, shows low overall toxicity. Mean toxicity over all examples is . Very low toxicity is normal for datasets that are not designed to include high-risk inputs. For example, the HELM classic leaderboard for toxic-fraction scores² is in the range of \([.001,.01]\) on some datasets. However, the maximum toxicity of our dataset is , indicating that high-risk categories are an important feature to include. LLM descriptions from the Sexual Wellness category scored very high in the detoxify sexually explicit sub-type, while the Shooting category scored highest in the threat sub-type. This highlights the need to align expectations for toxicity with the use case; accurate descriptions for sexual wellness require language that .

Fairness results are shown in Table 5. This table shows differences among cohorts, i.e. how identity groups, categories, and adjectives compare to each other. While the differences in accuracy are small, there is a 21-fold increase in toxicity between the least-toxic category (Appliances) and the most toxic (Sexual Wellness). This is to be expected as the detoxify classifier will identify terms related to Sexual Wellness as sexually explicit.

Adjective cohorts showed no significant disparity. However the identity groups reveal striking fairness differences. For example in Figure 1, by comparing detailed toxicity sub-types for each group, we see how the language used by the model varies. Products associated with the group Women resulted in significantly higher scores for sexually explicit language, even though this group was near the middle range for overall toxicity. Note that the detoxify classifier uses a general definition of toxicity, which would need to be customized for this specific application.

The dataset can also be used to compare models and make design choices. from leaderboards [20], [23] that larger models perform better overall. However, testing on our dataset shows a much smaller gap between 1B and 11B, compared to the leaderboards. This

6 Discussion and Limitations↩︎

We have demonstrated a method to construct a dataset specific to an application use case, and showed that the resulting dataset is sufficient to reveal disparities in model performance among demographic cohorts. The data supports safety testing of models, depending on a customizable safety definition. Unlike existing responsible AI benchmarks that are often generic, our dataset supports a fine-grained evaluation specific to the application context, offering insights for designing better user experiences in realistic settings. Our sample evaluation shows how the data can be used, assessing the cost-performance tradeoff among models.

We also recognize limitations and opportunities for improvement. Quality and veracity metrics rely on ground truth data, which is derived from human-written product descriptions. These descriptions contain natural imperfections and biases, according to the seller’s goals. For example, the ground-truth descriptions for Women’s products contain more sexually explicit language. However, the dataset supports a variety of evaluation metrics. Downstream consumers of the data could apply LLM-based judges, to reduce the reliance on ground truth.

Although we capture some diversity in gender associations among product cohorts, binary associations explicitly, non-binary associations with the catchall term “any". rather than an explicit, verified label. This a realistic user experience on e-commerce websites, where consumers

One important extension of the work would be to cover multi-modal or multi-lingual components from the online product listings, or to generate images, which can be scored using automatic quality metrics like Human Preference Scores [24].

7 Conclusion↩︎

In this work, we introduce a dataset representing a real-world application. The data methodology aligns application-specific risks (Safety, Veracity, Fairness) with metrics and data attributes. We show an example of how the data can be used for model assessment, revealing significant differences among LLM-generated descriptions for products marketed to different shopper cohorts (e.g., Women, Latino, LGBTQ+). We look forward to future experiments on this dataset from the broader research community, expanding our understanding of language model performance in realistic end-user applications.

8 GenAI Usage Disclosure↩︎

In this work LLMs were used to generate synthetic product descriptions as described in Section 3. The generated product descriptions are paired with human-sourced product descriptions, and synthetic queries constructed from a template. Since the human-sourced product descriptions were scraped from the Amazon.com website, it is not possible to know whether they were hand written or written by generative AI. Nonetheless, as the descriptions were attached to the product listing by the seller, we consider it reasonable to assume the seller endorsed the description as representative. Generative AI was not used in the writing of this paper.

References↩︎