1 Introduction↩︎

Language models are increasingly deployed as agents that interact with users to solve some task. For example, agentic recommender systems, like ChatGPT Shopping Research [1], Perplexity Shopping [2], and Amazon’s Rufus [3], elicit a user’s shopping intent via chat, search for relevant items using tools, and recommend items in reports for users to select from (Figure 1 top). A quiet assumption underlying most such systems — and the interactive benchmarks that evaluate them — is that agents are interacting with expert users: users who can specify their preferences fully when pressed [4]–[6]. In this work, we challenge this assumption.

Imagine asking an agentic recommender system for hiking boot recommendations. You make a good faith attempt to describe your requirements (size, price, preferred color), and the agent returns a written report with a set of recommendations. Whether you end up with the right boot depends not on the agent’s search skill, but on whether it helped you develop your preferences for features that matter — even ones you didn’t know to ask about. As a non-expert hiker, you may not realize that boots can run narrow or wide, or that lug depth affects what terrain you can safely hike on — but once you do, these features change both whether you can give the agent useful information to search with, and whether you can evaluate its recommendations. A truly helpful agent surfaces important, originally unknown features unprompted, before they become a bad purchase.

This example illustrates that real users are often not experts. Users may lack the domain knowledge to have well-formed preferences in the first place — they not only underspecify their initial requests, but are frequently unaware of what features they could form preferences over in the first place [7]–[9]. A user who has never heard of lug depth cannot answer a question about it; agents must instead teach this domain knowledge — through intentional examples and proactive explanations, not just clarifying questions — before users can form a preference over it. The same gap also limits a user’s ability to act on recommendations: even if a report contains the ideal pair of boots, a user who hasn’t yet formed a preference about width can’t reliably tell it apart from similar-looking options.

To formalize these principles, we propose CoPref(Co-constructed Preferences), a model of how non-expert users construct — rather than simply reveal — preferences based on agent dialog actions [10], [11]. In CoPref, users maintain a preference state — a subset of their task preferences they are aware of, which begins incomplete and evolves each turn depending on agent actions (Figure 1 bottom). Critically, not all features are equally easy to construct preferences over, a distinction we formalize using the Search-Experience-Credence (SEC) framework from Information Economics [12], [13]: some features can be elicited by a simple question, some can only be recognized once the user has seen or compared real options, and some require the agent to provide a technical explanation that increases the user’s domain knowledge. This means an agent cannot rely on clarifying questions alone — it must learn which features typically demand which dialog action across users. A user’s final preference state, accumulated over the course of the interaction, also determines how well they can evaluate solutions.

We study these ideas concretely in the setting of agentic recommender systems, where we introduce CoShop, a benchmark evaluating agents with CoPref users across fashion, movie, and book recommendation domains. CoShop measures performance by the user-agent team’s item accuracy: in each instance, an agent converses with a CoPref user to learn about their preferences, searches for relevant items, and writes a report of \(k\) recommendations (Figure 1 top). The user then selects one of the \(k\) items based on their preference state at that point in the conversation. Team accuracy requires two things to go right: the target item must be among the \(k\) recommendations, and the user’s preference state must be rich enough for them to identify it. An agent that searches well but teaches the user nothing leaves the task unspecified and the user unable to select the correct item.

In evaluations of RAG agents built from five frontier language models, we find that all models significantly drop in performance when faced with CoPref users — e.g., while claude-sonnet-4.6 scores 94.3% accuracy given fully-specified preferences upfront, it scores only 27.6% team accuracy on CoShop after 5 turns of conversation. No model exceeds 56% on any domain in CoShop, and all exhibit gaps between standalone agent performance and team accuracy. An error analysis reveals two failure modes. The first is poor report quality: agents omit or misrepresent item features, leaving users unable to evaluate recommendations even when the right item is present. The second is underdeveloped user preferences: agents fail to take the dialog actions needed to expand what users know, leaving them unable to recognize the correct item.

In summary, our contributions are:

• CoPref, a model of preference construction in which different features — depending on how they’re learned — demand different agent dialog actions.

• CoShop, an interactive benchmark of agentic recommender systems, which is open-sourced at this URL.

2 CoPref: Modeling users with developing preference states↩︎

We study agents that converse with a user over a bounded number of turns before proposing a solution. Formally, an agent interacts with a user for \(T \le B_\text{turns}\) turns,¹ then returns a solution \(x\) from a solution space \(\mathcal{X}\) (e.g., a set of hiking boots). Solutions vary along a set of features \(\mathcal{F}\) that users may care about, such as price or lug depth; each solution’s values on these features are given by a feature map \(\phi: \mathcal{X} \to \mathbb{R}^{|\mathcal{F}|}\).

We assume the user has some latent target item \(x^\ast\) in mind, and that their preferences are determined by its feature values: for each feature \(f \in \mathcal{F}\), the preferred value is \(\phi_f(x^*)\), i.e. the value the target takes on that feature. For instance, \(\phi_\text{price}(x^*) = ``{\le}\$50''\) is a budget preference: the user wants items priced at or under $50. Critically, however, a non-expert user need not be aware of — or hold an opinion on — every feature in \(\mathcal{F}\) at once. We capture this with a preference state \(S_t \subseteq \mathcal{F}\) the subset of features over which the user has actually formed a preference by turn \(t\). This state determines what the user can ultimately evaluate: at the end of the conversation, the user judges a proposed solution \(\hat{x}\) based on how well it matches their preferences over the features in their final state \(S_{T+1}\).

2.0.0.1 Beyond expert users: modeling preference construction.

Prior work models users as experts — their preference state at every turn includes all features, i.e. \(S_t = \mathcal{F}\) for all \(t\). This means the user can answer any question about any feature \(f\) with \(\phi_f(x^*)\) perfectly (Figure 1 bottom left). Not only is this assumption unrealistic, it rewards undesirable agent behavior: since the user can answer any question, an agent’s best strategy is to bombard users with questions upfront. We observe exactly this in Section 4.4, where models like gpt-5.2 ask a median of 9 clarifying questions. The social science literature offers a richer model: [11] and [10] argue that preferences are not pre-formed and waiting to be retrieved — they are constructed through interaction, shaped by what the user encounters and learns from the agent. [14]–[17] further document that users lack confident, fully-formed preferences, particularly in domains where they have limited expertise. The agent’s role is thus not just to query a static preference state, but to actively participate in constructing it.

We operationalize this by modeling preference states as dependent on agent actions. \(S_1\) is initialized to be sparse, matching empirical studies suggesting consumers start with few predefined preferences [9]. At turn \(t\), the user updates their preference state based on the previous agent message (Figure 1 bottom right).

2.0.0.2 Not all preferences are equally easy to construct.

Information Economics offers a relevant distinction: not all product attributes require the same kind of information before a consumer can form a judgment about them. The Search-Experience-Credence (SEC) framework [12], [13], [18] formalizes this by classifying features according to what a consumer needs in order to assess them, and empirical work testing the framework [19] confirms that consumer behavior really does differ systematically across these types. We adopt this typology and partition \(\mathcal{F} = \mathcal{F}_\text{s} \cup \mathcal{F}_\text{e} \cup \mathcal{F}_\text{c}\) accordingly.

1. Search features \(\mathcal{F}_\text{s}\) are easy for the user to form preferences over; they do not require additional information for the user to assess what they prefer (e.g., price, size).

2. Experience features \(\mathcal{F}_\text{e}\) require grounding in exemplar solutions \(x \in \mathcal{X}\) for a user to decide what their preferred value is. A user knows what they like or don’t like when they see it, but not beforehand.

3. Credence features \(\mathcal{F}_\text{c}\) require domain knowledge to form a preference for. For example, a preference for viscose vs. lyocell fabric requires the user to do additional research to understand.

Each SEC type suggests a different way an agent can help a user construct a preference over it (Figure 2). A clarifying question is enough to surface a search feature, since the user already knows their preference and just needs to be asked. Example options can ground an experience feature — and, trivially, a search feature too — by letting the user react to something concrete rather than an abstract attribute. A technical explanation can unlock a credence feature — and, again, a search feature — by supplying the domain knowledge the user was missing. Concretely, a feature is added to the user’s preference state \(S_t\) once the agent’s turn-\(t\) message references it through an action capable of unlocking its type.

In the classical SEC literature, the search, experience, and credence partitions are fixed across all users. However, we hypothesize that SEC splits should vary by the individual; two users may bring different domain knowledge to the same feature.

2.1 Human study: validating CoPref↩︎

CoPref rests on three claims: that non-expert users have a mix of search, experience, and credence features, rather than just search features; that the dialog actions we propose — questions, examples, explanations — succeed in unlocking preferences over their corresponding feature type; and that a feature’s SEC type may be a property of the individual rather than fixed across users. We validated these with a human study.

We recruited \(n=25\) participants for a simulated clothing shopping task. Participants imagined shopping for a sweater with an AI agent that needed to understand their preferences before recommending items. Participants self-reported a range of expertises with sweaters (28% claimed to be very familiar with sweaters). To elicit each participant’s SEC split, we administered 10 rounds, one per sweater feature drawn from the H&M dataset in Section 3 (e.g., price, neckline, material). In each round, participants saw a clarifying question about the feature and could either answer directly, in which case we classified the feature as a search feature for that participant; or indicate uncertainty by requesting examples or a technical explanation, in which case we classified the feature as an experience or credence feature, respectively. Those requesting examples or explanations were then shown the corresponding content and asked again to state a preference. Additional details are in Appendix 7.

2.1.0.1 Non-expert users have heterogeneous preferences, and dialog actions unlock them.

Our results validate the first two claims. Individuals have a mix of search, experience, and credence features: 0 participants had all search features, as every participant asked for examples and/or an explanation on at least one feature; 56% asked only for examples or only for explanations, suggesting they had search-and-experience or search-and-credence features among the presented set, while the remaining 44% had all three kinds (Figure 3). Appropriate dialog actions also do unlock preferences for experience and credence features: of (participant, feature) pairs where the participant requested examples after indicating uncertainty, 85% were subsequently "unlocked" (i.e., the participant went on to provide a non-empty free-text preference); of pairs where the participant requested an explanation, 75% were unlocked.

2.1.0.2 SEC type varies across individuals.

Aside from gender, no feature was uniformly classified as search, experience, or credence across participants. At the same time, credence classifications were sparse and concentrated in a small number of features (shoulder style, belt/tie presence), suggesting that while credence features are not universal, they cluster around a handful of technically demanding attributes. This suggests that features carry some base rate of difficulty, but their SEC classifications vary across individuals based on individual domain knowledge.

2.2 Limitations of CoPref↩︎

CoPref makes two simplifying assumptions about preference construction. First, users form a preference over a feature based on a single agent action, rather than requiring multiple interactions. Second, once formed, preferences are stable and never forgotten: we assume that \(S_t\) is monotonically non-decreasing. Real preference construction may be more iterative and less stable than this model assumes — a user might need to see examples before realizing they need a technical explanation, or might revise a preference they had previously settled on. These simplifications make CoPref users easier to collaborate with — however, we find in Section 4 that frontier language models still fail to help users develop preferences, even under these favorable simplifications.

3 CoShop: An interactive benchmark for agentic recommender systems↩︎

As a case study of how well agents work with CoPref users, we instantiate an interactive benchmark for agentic recommender systems. Shopping is a natural fit for our problem setting: product attributes like price and color give us a well-defined feature set \(\mathcal{F}\), and a first-time buyer rarely knows all features that will matter to them until they review real options while shopping.

3.0.0.1 Problem definition.

In recommendation, the solution space \(\mathcal{X}\) is a catalog of items, the target \(x^\ast\) is the item the user has in mind, and the features \(\mathcal{F}\) are catalog attributes such as price or material. Not all features apply to all items; we use \(\mathcal{F}(x)\) to denote the features relevant to \(x\) (e.g., sleeve length isn’t a preference one can hold about shoes). As in Section 2, the preference state \(S_t \subseteq \mathcal{F}(x^*)\) tracks which features the user has formed an opinion on by turn \(t\). Each rollout proceeds in four stages (Figure 1, top):

1. Conversation. The agent converses with the user for up to \(B_\text{turns} = 5\) turns or until the agent emits a special “end conversation” token. The agent can call a catalog search tool, which takes a query string \(q\) and returns the \(m\) most relevant items with features \(\phi(x_1), \dots, \phi(x_m)\). This stage is limited to \(B_\text{preview} = 50\) retrievals, 20 clarifying questions, and 10 unique items shown to the user, based on prior estimates of how many questions users tolerate before dropping out of elicitation [20] (we provide a sensitivity analysis in Appendix 11.2).

2. Research. The agent searches more thoroughly, with an expanded budget of \(B_\text{search} = 250\) retrievals.

3. Report. The agent writes up its final \(k = 5\) recommendations.

4. Selection. The user picks one item \(\hat{x}\) from the report, evaluating it against their final preference state \(S_{T+1}\).

Performance is measured by the user-agent team’s accuracy: whether \(\hat{x}\) matches \(x^\ast\). Evaluating by team accuracy stresses the importance of preference construction: without a preference state \(S_{T+1}\) rich enough to distinguish the target from its \(k-1\) distractors, the user may be unable to break ties well.

3.0.0.2 Product domains.

We source CoShop data from three existing datasets of real human-product interactions (Table 4): fashion purchases from the H&M dataset [21], movie recommendations from MovieLens[22], and books from Goodreads[23], [24]. Each dataset provides a product catalog \(\mathcal{X}\) with domain-specific columns \(\mathcal{F}\); to increase the size of \(\mathcal{F}\), we enriched the original datasets’ columns using a mix of regex and LLM-based methods (Appendix 8).

3.0.0.3 Constructing test users.

For each domain, agents are evaluated in interactions with 100 test users. Each user is anchored to a real human from the original dataset. The user’s target item \(x^\ast\) is the last item that the user purchased or rated at least 4 out of 5 stars. Motivated by our human study results in Section 2.1, each user’s SEC split is personalized based on their purchase history (Appendix 8.4). Concretely, for each feature \(f\) and target value \(v = \phi(x^*)_f\), we compute the pointwise mutual information between the catalog marginal \(p_\text{catalog}(f{=}v)\) and the user’s empirical purchase frequency \(p_\text{user}(f{=}v)\). Features the user consistently over-selects relative to the catalog are classified as search; features selected at roughly the catalog rate as experience; and features that are either rare in the catalog or the user under-selects, as credence. The most common search, experience, and credence features for each dataset are given in Table ¿tbl:tab:sample95sec?. Each user’s initial state \(S_{1}\) is a subset of their search features \(\mathcal{F}_S\). We construct \(S_1\) so that it is one feature short of identifying \(x^\ast\): we build \(S_{1}\) for each user by randomly adding features from the user’s search split, stopping just before gpt-oss-120b can recall \(x^\ast\) directly from the research stage at a budget of 250. This is to ensure challenging, but not overly adversarial initializations.

Unlike typical LLM-based user simulators, which prompt a model once with a persona and let it improvise an entire conversation, we explicitly anchor our simulator LLM gpt-oss-120b to the current preference state \(S_t\). Concretely, we generate natural language messages conditioned only on the current preference state \(S_t\), and we have gpt-oss-120b select a final item conditioned only on the final preference state \(S_{T+1}\) and the agent’s report (with item order randomized to prevent position bias).

3.0.0.4 Parsing dialog actions from agent messages.

In order to study agents in as natural a setting as possible, agents emit natural language messages. To compute state updates, we used gpt-oss-120b to parse the agent’s message into dialog actions and, critically, identify the features being referenced by each parsed action. This parsing step introduces some error, mainly around identifying the referenced features: in Appendix 9.2, we found that gpt-oss-120b has question parsing accuracy of 74.0% and item parsing accuracy of 94.0%. In Appendix 11.3, we also experimented with removing this parsing step: we required agents to emit structured dialog actions during the conversation stage (ask_question, show_item_for_feedback, or explain), each of which must explicitly reference the target feature name. This removes the confound of parsing error at the cost of realism, and it may change the distribution of what agents say. We found that removing the parsing step dramatically degraded performance for tested models, which tended to hallucinate column names (e.g., color instead of perceived_colour_master_name), even when the model’s natural language output was coherent.

4 Benchmark results on CoShop↩︎

We evaluated five frontier language models on CoShop: one smaller and one larger model from the GPT and Claude families (gpt-4.1-mini, gpt-5.2, claude-haiku-4.5, claude-sonnet-4.6), as well as one open-source model gpt-oss-120b. We use a simple RAG agent harness which gives the model access to a catalog search tool, a private scratchpad, a reminder to be cognizant of the user’s limited attention, and a summary of the SEC theory along with the various budgets \(B_\text{turn}, B_\text{preview}, B_\text{search}\) (prompts in Appendix 10). We also compared agents to non-LLM baselines which retrieve 500 items using a query based on \(S_1\) and then rerank either randomly or by catalog popularity.

4.1 Agents excel at search but fail to resolve underspecification↩︎

We begin by isolating agents’ task execution ability — how well can agents find and recommend items given the user’s full preference set \(\phi(x^*)\) upfront? In Figure 5A, the leftmost fully-specified setting shows that models score highly in this fully-specified setting, with claude-sonnet-4.6 scoring an average of 94.3% across datasets. There are two exceptions to this rule. claude-haiku-4.5 scored 98% on H&M, but on both MovieLens and Goodreads hit context limit errors on 96 of 100 runs. gpt-4.1-mini scored 86% on H&M but only 17% on MovieLens and Goodreads; on these datasets, in nearly half of runs it failed to terminate after exceeding the research budget, instead issuing tool calls indefinitely (Appendix Table 6).

Next, we ask how performance changes as users become more underspecified. In the search features only setting, users develop preferences over the course of the conversation as in CoPref, but all features are search features (\(\mathcal{F}_\text{s} = \mathcal{F}\)). This means that any dialog action can add any feature to the preference state; preferences are simply retrieved, and clarifying questions can recover all preferences. This mimics the setups of [25]–[27]. Finally, we use CoPref users from Section 2, where features vary in construction difficulty depending on the user’s SEC split.

Figure 5A traces final accuracy across these three user types. All models drop in performance when non-search features are introduced. For example, while gpt-5.2 scores 57.0% accuracy when all features are search features, it scores only 27.7% accuracy when some features require actions besides clarifying questions.

4.2 Team accuracy reveals human-facing failures↩︎

We now report detailed results on CoShop using three families of metrics, each isolating a different component of the pipeline. State completeness measures the fraction of the user’s full preference set recovered by end of conversation: \(|S_{T+1}| / |\mathcal{F}(x^*)|\). Agent recall measures whether \(x^\ast\) appears in the agent’s top-\(k\) recommendations after research. Team accuracy — our primary metric — measures whether the user ultimately selects \(x^\ast\) from the agent’s report, and is upper bounded by agent recall at \(k\). Table 1 (RAG Agents section) reveals two things: first, no model recovers more than 54% of the user’s preferences over the course of the conversation. This means agents are leaving a substantial portion of users’ preferences undeveloped. Second, team accuracy is consistently and substantially lower than agent recall: even when agents surface \(x^\ast\) in their top \(k\), users often cannot identify it as better than the other \(k-1\) presented options. For example, gpt-5.2 scores 65% recall at \(k\) on H&M but only 56% team accuracy, a 9-point gap.

We isolate the sources of this gap experimentally in Figure 5B. The first source is poor report quality. Since users select \(\hat{x}\) based solely on the agent’s generated report, if an agent misrepresents an item (e.g., omits or hallucinates features), or if their report is strongly biased towards a particular item, this directly impairs team accuracy. To measure the extent to which this happens, we replayed the user selection step with complete reports — programmatically generated descriptions of \(\phi(x)\), which includes all features in \(\mathcal{F}\) — in place of the agent’s report. Team accuracy increases significantly for gpt-oss-120b, and slightly for gpt-4.1-mini and claude-sonnet-4.6(blue bars).

The second source for the team-agent gap is underdeveloped preference states. Even with complete reports, users sometimes cannot distinguish between items because the final state \(S_{T+1}\) lacks the discriminative features needed to break apparent ties — a direct consequence of failing to develop the user’s preference state during conversation. Replaying selection with both complete reports and the full preference set \(\phi(x^*)\) improves accuracy further (green bars). For all models outside of gpt-oss-120b, underdeveloped preference states explain most of the team-agent gap. Note that underdeveloped preference states are also responsible for low agent recall@\(k\) in the first place. Agents that fail to grow the user’s preference state leave the search task greatly underspecified.

Further, with complete reports and full preference states, team accuracy substantially exceeds agent recall@\(1\) for all models (Figure 5B). This shows that a well-informed user is a better selector than the original agent — teams with well-informed users can outperform the agent alone.

4.3 Interventions on agent knowledge and communication↩︎

The agents above interact with users zero-shot, with no knowledge of their prior rating history. In principle, history access could help in two ways: since our SEC splits are derived from purchase history, models might better anticipate whether a given feature is likely to be search, experience, or credence for a particular user, and might also infer preferences directly from past behavior without needing to ask. Table 1 (history access) and Figure 5C show that for the models we tested, neither benefit materializes significantly.

A more direct lever is how agents communicate during conversation. Most models show example items frequently, but describe them incompletely — rather than proactively surfacing features the user may not have considered, agents tend to rehash already-known ones, missing the key opportunity to grow \(S_t\) that examples afford. In Table 1 (override item descriptions) and Figure 5C, we overrode agent messages during conversation crudely, replacing their item descriptions with programmatic ones that mention all catalog features. We found substantial gains: gpt-4.1-mini’s state completeness jumps from 44% to 61% and team accuracy from 36% to 46%. This suggests that improving how agents use dialog actions — particularly examples and explanations — is a tractable path to better performance with CoPref users, even holding model capability fixed.

4.4 Analysis of interaction strategies on CoShop↩︎

To understand what drives underdeveloped preference states, we turn to Table 2, which reveals that models cluster into qualitatively different interaction strategies.

4.4.0.1 Conversation vs. research.

We observe two groups of models when looking at the balance of the conversation and research stages (Table 2). Smaller models (gpt-oss-120b, claude-haiku-4.5, gpt-4.1-mini) exhaust the conversation turn budget (median 5.00 turns) while using relatively little of the research budget. The larger models (claude-sonnet-4.6, gpt-5.2) prematurely end the conversation, but use more of the 250 item research budget. In 37% of conversations, gpt-5.2 ends the conversation after two turns. This suggests an “execution bias” for the larger models — these are overly cautious about using user time and try to do more work offline, which fails on CoShop because the task remains underspecified.

4.4.0.2 Distribution of dialog actions.

All models ask questions and show example items, but no model proactively provides explanations for technical features. As a result, no model unlocks credence features during conversation.

When showing examples, some models frequently hallucinate items that do not exist in the catalog: claude-haiku-4.5 in 47% of conversations and gpt-oss-120b in 31%. In these cases the user simulator prompts the model to verify against the catalog, after which it corrects itself, but the opportunity to surface genuine features from a real item is lost. Even when models show real items, they often omit features in their descriptions, such that the examples fail to grow \(S_t\). The loops column in Table 2 captures this: a loop occurs when the model shows an item satisfying the user’s known preferences and the user simulator — programmed to always request more detail in this case — receives a response in which the model just restates the same features rather than introducing new ones. All models except gpt-5.2 loop frequently — in 35% of gpt-4.1-mini’s conversations, a request for more detail yields nothing new.

Models also vary in the types of questions they ask. 20% of the questions that gpt-5.2 issues are confirmation questions: they ask the user to confirm preferences which have already been clearly stated in a prior turn. For example, when a user requests a popular 1970s–80s ensemble film with “a strong adult female lead” and an “ensemble team vibe with only mild action,” gpt-5.2 spends successive turns probing how strict each preference is rather than searching for items that satisfy them:

(6) Is “mystery/thriller” a hard requirement (primary genre), or just a strong preference? (7) Is “survival plot” a hard requirement (characters trapped / disaster / hunted / must endure), or were you using “survival” more loosely (career/social survival, etc.)? (8) For the “ensemble team vibe,” do you mean a group solving the central mystery together, or is it okay if the lead drives the case and there’s just a strong supporting cast? (9) Is comedy actually desired (e.g., “mystery-comedy” / “comedic thriller”), or are you fine with a straight thriller as long as the ensemble mystery-solving is present? (10) How strict is the “strong adult female lead” requirement versus simply having an ensemble with at least one prominent adult woman?

We provide the full list of model questions from this interaction in Appendix Figure 10; Appendix Figure 9 shows an analogous H&M example.

Together, these patterns — confirmation questions that retrieve rather than construct preferences, hallucinated items, and loops that fail to introduce new features — explain why no model recovers more than 54% of the user’s preference state despite using the full turn budget. Appendix 12 provides an extended qualitative analysis of transcripts.

5 Related Work↩︎

(Appendix 13 surveys additional related work.)

A body of work evaluates how language models resolve underspecified requests in task-oriented settings [4]–[6], [28]–[32], but these works make the expert user assumption that we argued against in Section 2.

Most shopping agent evaluations are non-interactive: agents are prompted with fully-specified preferences \(\phi(x^*)\) upfront and evaluated solely on recall@\(k\) [33]–[37]. [25] and [26] evaluate conversational recommender systems in underspecified settings, but both still simulate users with the expert user model, where all features are search features. [25] additionally measure how much the agent teaches the user about the domain; we similarly emphasize the importance of user learning, but we evaluate this more directly via the user’s state completeness and center it by focusing on team accuracy.

In CoShop, agents are evaluated based on team accuracy: if agents fail to build up the user’s domain knowledge, the user is too weak to verify the recommended slate. These ideas relate to the problem of scalable oversight [38], [39], which studies how a non-expert human can effectively supervise a stronger model. Our work considers a setting where the agent can augment the human’s expertise so that the human might be better able to supervise.

A new line of work studies how to simulate users for training and evaluating task-oriented agents [40]–[44]. These works are split by whether they take a bottom-up, data-driven approach (i.e., finetuning language models on user utterances) or a top-down, model-based approach (i.e., defining a model of human behavior to condition a language model on). Because of the limited amount of multi-turn user data, the former community tends to emphasize simulating human utterance styles, while the latter emphasizes simulating human behaviors. Our work falls in the latter camp, where we specifically model preference construction.

Concurrent work by [27] models user preferences as hierarchical. CoPref instead emphasizes heterogeneity: some features (e.g., experience and credence features) require non-question dialog actions to construct. [27] train agents against synthetic tasks with their user model and show this improves real user satisfaction, whereas we benchmark agents against CoPref users on recommendation tasks.

6 Conclusion↩︎

We introduced CoPref, a model of how users construct rather than retrieve preferences over the course of an interaction, and CoShop, a benchmark instantiating this model in agentic recommendation. Our results show that current agents systematically underdevelop user preference states. We hope future work (1) explores methods to improve agent quality on CoShop, such as training with CoPref users, and (2) extends CoPref beyond the recommendation setting.

6.0.0.1 Using the CoShop package.

CoShop is open-sourced as a package to support future agent and model evaluations. It can be accessed at this site.

Acknowledgements↩︎

The authors thank Theodora Worledge, Nicole Meister, Qinan Yu, Ed Chen, and other members of the Guestrin lab, as well as Miguel Liu-Schiaffini, Vignesh Kothapalli, and Marco Tulio Ribeiro, for discussions and feedback on this project.

7 Human study↩︎

Below we provide additional details on the study design, participant demographics, and results for each hypothesis validated in Section 2.1. This study was approved by our institution’s IRB.

7.1 Study design↩︎

We administered a Google Form structured around a simulated clothing shopping task. Participants were asked to imagine they were shopping for a sweater and that an AI shopping agent needed to understand their preferences before making recommendations. The form proceeded in two stages.

7.1.0.1 Stage 1: Pre-survey.

Participants first answered questions about their demographics, shopping habits, AI tool usage, and familiarity with sweaters.

7.1.0.2 Stage 2: Preference elicitation.

Participants completed 10 rounds of questions, one per sweater feature drawn from the H&M dataset (Section 3). The features were drawn from the H&M catalog; all participants saw the same 10 features: price, gender, target style intent, belt or tie presence, shoulder style, collar style, graphic placement, material, color family, and neckline.

In each round, the participant was shown a direct clarifying question about the feature (e.g., “What neckline do you prefer for the sweater?”) and asked to respond in one of four ways:

1. Answer directly in a free-text box.

2. “I’m not sure — show me examples.”

3. “I’m not sure — show me an explanation of what this means.”

4. “I don’t care.”

Responses (1) and (4) were interpreted as settled preferences; the feature was classified as a search feature for that participant. Responses (2) and (3) indicated that the participant needed additional grounding; the feature was classified as requiring experience or credence grounding, respectively.

Participants selecting option (2) were shown a set of example items illustrating variation along that feature. Participants selecting option (3) were shown a brief technical explanation of the feature generated by Claude Sonnet 4.6. In both cases, participants were then asked to state their preference in free text or select “I don’t care.” A non-empty free-text response was interpreted as evidence that the participant successfully formed a preference after seeing the provided content.

7.2 Participants↩︎

We recruited \(n=25\) participants via word of mouth. The sample skewed young; we estimate the majority of participants were under 30. 16 participants identified as female and 9 as male. Most were frequent users of AI tools such as ChatGPT, Claude, and Gemini, though over 50% had never used AI for shopping or gift recommendations. Participants self-reported a range of sweater-shopping expertise: only 7/25 (28%) claimed to be very familiar with sweaters, consistent with our focus on non-expert users. Figure 7 summarizes participant demographics.

7.3 Limitations↩︎

Our study design asks participants to decide in a single shot whether a non-search feature is experience or credence. In practice, a user might request examples and then realize they need a technical explanation after seeing those examples — a more iterative process than our one-shot classification captures. Additionally, our sample is small (\(n=25\)) and skews young, which may limit generalizability. These limitations are consistent with the simplifying assumptions of CoPref discussed in Section 2.

8 CoShop dataset details↩︎

CoShop tests agents on three datasets, each representing a separate product domain: fashion recommendations (H&M), movie recommendations (MovieLens), and book recommendations (Goodreads). For each dataset, we source the catalog \(\mathcal{X}\) and user profiles (including a target item \(x^\ast\)) from the original datasets. We enrich the domain feature set \(\mathcal{F}\) using regex-based feature extraction. Section 8.4 describes how we compute the search-experience-credence (SEC) split and the initial known feature set \(S_1\) for each user. In each per-dataset feature table below, the S, E, and C columns report the percentage of the 100 test users for whom that feature is classified as search, experience, or credence, respectively (these need not sum to 100%, since a feature is excluded for users whose target value is missing or expresses no preference).

8.1 H&M preprocessing↩︎

8.1.0.1 Overview.

The agent’s goal is to help a user find a fashion item they would like from the H&M catalog [21]. There are catalog items and \(|\mathcal{F}| = 77\) features per item. Across users, the average SEC split is \(|\mathcal{F}_\text{s}| \approx 6.1\) search, \(|\mathcal{F}_\text{e}| \approx 21.5\) experience, and \(|\mathcal{F}_\text{c}| \approx 9.1\) credence features, with an average initial known feature set size of \(|S_1| \approx 3.9\).

8.1.0.2 Users.

We start from the original dataset’s catalog and transaction log. For each customer, we take their newest transaction as the target item \(x^\ast\). All remaining in-catalog purchases by the same customer form their purchase history. We keep users who have at least 30 history items and retain the first 100 valid users.

8.1.0.3 Catalog.

Each article’s price is imputed as the maximum observed transaction price; articles without a price are dropped. We exclude catalog items that are missing an image, belong to the undergarment section, or are not purchased by any of the 100 users in our evaluation set. To enrich the catalog, we use Qwen/Qwen3-VL-4B-Instruct to generate a visual description from each product image and then use regular expressions to extract structured features. The \(|\mathcal{F}| = 77\) features span: basic attributes (product category, pattern, color, price); fit and cut (fit, sleeve length, neckline, waist rise, leg silhouette, dress length, skirt shape, silhouette shape); construction details (lining, pockets, ribbing, drawstring, woven vs.knit fabric, seam type, wrap style); outerwear and performance (jacket type, waterproof, windproof, functional finish); surface and decoration (fabric texture, graphic type/theme/placement/scale, embroidery, sequins, trim, logo); part-specific details (sleeve style, cuff, shoulder exposure, strap, armhole, hem, back, side features); ethical attributes (sustainable, vegan, toxin-free); and demographics (gender, age group, sportswear, formality). Table 1 lists all features with their descriptions.

To avoid a situation where the user simulator asks for too specific of a price (e.g., “Can I have a shirt priced at $15.93?”), the simulator view of \(\phi(x^*)\) coarsens the price feature: it is rounded up to the nearest $10 and displayed as an upper bound (e.g.a price of $15.93 becomes “\(\le\) $20”).

8.2 MovieLens preprocessing↩︎

8.2.0.1 Overview.

The agent’s goal is to help a user find a movie they would like from the MovieLens catalog [22]. There are catalog items and \(|\mathcal{F}| = 92\) features per item. Across users, the average SEC split is \(|\mathcal{F}_\text{s}| \approx 8.6\) search, \(|\mathcal{F}_\text{e}| \approx 70.2\) experience, and \(|\mathcal{F}_\text{c}| \approx 8.8\) credence features, with an average initial known feature set size of \(|S_1| \approx 1.7\).

8.2.0.2 Users.

We start from the original dataset’s rating log. For each user, we sort ratings from newest to oldest, keeping only users who rated movies on at least five distinct calendar days. The target item \(x^\ast\) is the first movie that receives a rating of at least 4 out of 5 stars in that order. All other rated in-catalog movies count as history; we drop users with fewer than 50 history entries and retain the first 100 valid users.

8.2.0.3 Catalog.

We start from the MovieLens movie table linked to The Movie Database (TMDB) metadata: titles, release years, plot overviews, genres, vote averages and counts, popularity, runtime, budget, revenue, collection membership, production countries and companies, spoken languages, and user-contributed tags. We exclude movies that do not appear in any of the 100 users’ rating histories. To enrich the catalog, we use regular expressions to infer additional narrative, tone, and theme attributes. The \(|\mathcal{F}| = 92\) features include: core metadata (release year, rating, vote count, popularity, runtime, budget, revenue, genres, collection, production companies/countries, spoken languages); production attributes (MPAA rating, animation style, award profile, source material, cinematographic format, home media availability); content flags (nudity, remake, sequel, Bechdel test, IMDB top list, national film registry, black-and-white, era); and a large set of thematic and genre indicators (83 binary features covering tones such as melancholic, slow burn, or dark comedy; narrative elements such as twist ending, road trip, or time travel; demographics such as female-led; and setting/subject tags such as high school, prison, or urban setting). Table 2 lists all features with their descriptions.

To avoid a situation where the user simulator asks for too specific of a feature (e.g., a movie with exactly a $1.3M revenue), the simulator view of \(\phi(x^*)\) coarsens several fields. original_language ISO 639-1 codes are replaced with English language names. popularity is replaced by a discrete label derived from the catalog distribution (e.g.very popular, below average). vote_average is floored to the nearest half-point and prefixed with “\(\ge\)”; vote_count is floored to the nearest 100; budget and revenue are floored to the nearest $100,000; runtime is rounded up to the next multiple of 30 and prefixed with “\(\le\)”; year is mapped to a two-decade phrase (e.g.”1980s or 1990s”).

8.3 Goodreads preprocessing↩︎

8.3.0.1 Overview.

The agent’s goal is to help a user find a book they would like from the Goodreads catalog [23], [24]. There are catalog items and \(|\mathcal{F}| = 102\) features per item. Across users, the average SEC split is \(|\mathcal{F}_\text{s}| \approx 7.8\) search, \(|\mathcal{F}_\text{e}| \approx 67.0\) experience, and \(|\mathcal{F}_\text{c}| \approx 13.5\) credence features, with an average initial known feature set size of \(|S_1| \approx 3.8\).

8.3.0.2 Users.

We start from the original dataset’s reading log, keeping interactions that have a marked read date. For each user, we sort reads from newest to oldest. The target item \(x^\ast\) is the first book in that order with usable genre labels. All other reads form the user’s history; we drop users with fewer than 5 history entries and retain the first 100 valid users.

8.3.0.3 Catalog.

We start from the public Goodreads book dump, keeping only books with non-empty titles and descriptions. We attach series metadata, author name, and aggregated genre labels per work where available, and add a small random sample of review text per book. We exclude works that do not appear in any of the 100 users’ reading histories. To enrich the catalog, we use regular expressions to infer additional plot, tone, and trope-level features. The \(|\mathcal{F}| = 102\) features include: bibliographic metadata (author, genres, average rating, publication year, page count, publisher, series name/length/position, review and rating counts); format and publication (ebook, audiobook, physical format, indie/self-published, debut); content descriptors (world type, setting type, age group, emotional tone, violence level, NSFW, happy ending, cliffhanger, magic system, technology level, paranormal creatures); award and cultural (award status, book club pick, classic, associated cultures, summer read, chick lit, art/music focus, based on true story, illness theme, LGBTQ themes, cookbook details, nonfiction subtype); narrative and plot devices (narrative devices, plot devices, fantasy/mystery/sci-fi subgenre, fanfiction origin); themes (24 binary theme flags covering death, love, family, war, identity, crime, etc.); tropes (21 binary romance and narrative trope flags); and romance subgenres (10 binary flags). Table 3 lists all features with their descriptions.

The simulator view of \(x^\ast\) coarsens several fields. The genres list is truncated to at most the first three labels. series_works_count is bucketed (0, 1, 2–3, 4–10, 10+); series_position maps to either first-in-series or a coarse mid-series label. average_rating is floored to the nearest half-point and prefixed with “\(\ge\)”; ratings_count and text_reviews_count are floored to the nearest 100; publication_year is floored to the nearest multiple of 100 (e.g. \(\rightarrow\) “\(\ge\) 1900”) and prefixed with “\(\ge\)”; num_pages is rounded up to the next multiple of 100 and prefixed with “\(\le\)”.

8.4 Computing user SEC splits↩︎

For each user \(i\) and target item \(x^\ast\), we classify every feature \(f\) as search, experience, or credence based on how well-established the user’s preference for \(x^\ast\)’s value was before the purchase.

Features are scored using a base-rate-adjusted surprise metric. Let \(v = \phi(x^*)_f\) be the target value and let \(p_\text{user}(f{=}v)\) and \(p_\text{catalog}(f{=}v)\) be the empirical frequency of the value \(v\) for feature \(f\) across the user’s purchase history and catalog, respectively. The score is the pointwise mutual information between the catalog and user distributions: \[s(f, x^*) \;=\; \log_2\!\frac{p_\text{catalog}(f{=}v)}{p_\text{user}(f{=}v)} \;=\; -\log_2 p_\text{user}(f{=}v) \;+\; \log_2 p_\text{catalog}(f{=}v).\] Features are then classified as:

• Search (\(s < -1\)): the user over-selects this value relative to the catalog, indicating a settled prior preference.

• Experience (\(-1 \le s < 1\)): the user selects this value at roughly the catalog rate; no strong directional preference signal.

• Credence (\(s \ge 1\), or value never seen in history, or feature almost always missing in the catalog): the user substantially under-selects this value or has never purchased it.

Three edge cases are handled before scoring. (1) If \(\phi(x^*)_f\) is missing or indicates no preference, the feature is excluded entirely. (2) If value \(v\) never appears in the user’s history (\(p_\text{user}(f{=}v) = 0\)), the score is undefined; we assign a large sentinel value and classify the feature directly as credence. (3) If the feature is structurally unobserved for nearly all catalog items (over 95% missing), the catalog marginal is unreliable and the feature is classified as credence.

8.5 Computing user initial preference states↩︎

The initial feature set \(S_1\) for each user contains the search features the agent knows at the start of the conversation. To compute \(S_1\), we simulate an incremental catalog search with a fixed retrieval budget: starting from a seed of always-known search features (e.g.genre), we iteratively add one search feature at a time and generate a natural-language search query from the currently known features using gpt-oss-120b. The query is issued to a vector search index over catalog item descriptions with a retrieval budget of \(k = 250\) items. We stop adding features once \(x^\ast\) appears in the top-\(k\) results, or once all search features have been included. The resulting feature set—the minimal set of search features sufficient for the target item to be retrievable by a search query—becomes \(S_1\).

9 User simulator details↩︎

Our user simulator plays the shopper in multi-turn dialogues. The user simulator has two main functions: to exchange messages with the agent during elicitation, and to select the final \(x^\ast\) from the agent’s report at the end of the interaction.

Following Section 2, the simulator maintains a preference state \(S_t := \phi_{S_t}(x^*)\) as a growing set of known catalog columns. Unlike typical LLM-based user simulators, which prompt a model once with just a persona and let it improvise an entire conversation, we explicitly anchor our simulator LLM gpt-oss-120b to the current preference state \(S_t\). Concretely, we generate natural language messages conditioned only on the current preference state \(S_t\), and we have gpt-oss-120b select a final item conditioned only on the final preference state \(S_{T+1}\) and the agent’s report (with item order randomized to prevent position bias).

9.0.0.1 Overview.

Upon receiving a message, the simulator first checks that any item recommendations use the expected structured JSON format; a lightweight classifier may ask the agent to fix formatting if items are clearly recommended but not tagged. It then parses the message into atomic clarifying questions, recommended items, and credence-style explanations, maps each to relevant catalog features, adds referenced features unlocked via the correct action to the preference state as described in Section 2, and then generates a natural-language reply from independent LM calls for each component (Figure 8). If nothing structured is extracted, the simulator falls back to a single freeform LLM turn with the full persona and conversation history. At most five features are revealed per turn in total (across all components), limiting how much preference information any single exchange can transfer. We parallelize LM calls where possible using LOTUS [45].

Each simulator instance is also given a lightweight persona derived from that user’s purchase or rating history, used only in the fallback freeform turn and in the question-answering prompt to provide conversational grounding. The persona does not contain the target item \(x^\ast\) or any of its feature values. The persona content differs by dataset:

• H&M: age, club membership status, newsletter frequency, number of prior purchases, average and maximum historical spend, and clothing size.

• Goodreads: total books rated, count of high ratings (\(\ge 4\) stars), and top-3 genres and top-3 authors by frequency.

• MovieLens: total movies rated, count of high ratings (\(\ge 4\) stars), and top-3 genres and top-3 spoken languages by frequency.

9.1 Prompts↩︎

9.1.0.1 Clarifying questions.

For each extracted question, a semantic filter identifies which catalog features are relevant to answering it. The LM is then asked to answer briefly in the first person, conditioned on the current preference state \(S_t\) and the relevant features. It is instructed to defer (e.g., “I don’t know yet” or “I’m open to anything”) whenever those features are not yet in \(S_t\).

Extracting atomic questions. This step turns the raw agent message into a list of atomic, self-contained questions that ask the shopper about preferences or constraints. It deliberately drops questions about choosing among shown items and meta-requests.

You are a question extractor. Given a message from a customer service agent to a shopper, extract ONLY the clarifying questions FROM THE MESSAGE.

Clarifying questions ask about the user's underlying preferences, requirements, background, or constraints for the product search. These questions gather missing information the assistant needs in order to better tailor future recommendations. Examples:
- "Have you ever purchased wool?"
- "What size do you prefer?"
- "Do you have a budget in mind?"
- "Are you willing to relax any constraints?"
- "You said X. Could you clarify what you mean?"
- "Let me know if you have other preferences (e.g., sustainability)" -> "Do you have a preference for sustainability?"

Do NOT include questions that:
- ask for feedback or opinions on specific items or sets of items that were recommended,
- ask the user to choose between items already shown,
- ask what the assistant should do next (e.g., see more details, restart the search, show similar items),
- check whether an explanation was helpful,
- or otherwise talk about the conversation/meta-strategy without asking for new constraints or preferences.

These are NOT clarifying questions and must be EXCLUDED:
- "What do you think of item <item><id>123</id><information>Blue dress.</information></item>?"
- "How do you feel about this one?"
- "Would you like me to start my search over again?"
- "Do you want to see more details?"
- "Do you have any questions?"
- "Would you like to see more colors?"
- "Would you like to see items similar to this one?" 
- "Let me know if you'd like to narrow the list down."
- "Which of these is your favorite?"
- "Which of these three catches your interest the most?"
- "Which of these sounds like the best fit for you?"
- "Do any of these items look interesting to you?"
- "Does that explanation help?"
- "Do you have a preference among these options?"

Breaking Down Questions into Atomic Units

Split compound questions into individual atomic questions, where each question asks about exactly one attribute. For example:
"To narrow it down, could you let me know if you have any preferences for color, material (e.g., cotton, linen, silk), or a price range you'd like to stay within?"
becomes:
1. "Do you have any preferences for color?"
2. "Do you have any preferences for material (e.g., cotton, linen, silk)?"
3. "Do you have any preferences for price range?"

Likewise, any question that bundles multiple attributes with "or" must be split. For example:
"Do you have any preferred colors or patterns?"
becomes:
1. "Do you have any preferred colors?"
2. "Do you have any preferred patterns?"

Exception (binary choice questions). Do NOT split a question that asks the user to choose between exactly two options. For example:
"Do you prefer a standalone novel, or are you open to reading a book that's part of a series?"
This remains a single question.

Making Each Question Standalone
Every question must be fully self-contained with no unresolved references.
- Rewrite follow-ups that reference a prior question. For example, "Do you want red or blue? And how much more do you like one than the other?" becomes:
  1. "Do you want red or blue?"
  2. "How much more do you like red than blue, or vice versa?"
- Omit questions that cannot be rewritten because their reference cannot be resolved. For example, "Are you truly open to either style?" must be dropped if "either style" is undefined.
- Expand vague references to options. For example, "Do you have a preference among these fabric options?" must be rewritten to name the options explicitly: "Do you have a preference among these fabric options (cotton / polyester)?"

Return each clarifying question on its own line. If there are no clarifying questions, return exactly: NONE
Do not include any other text, numbering, or explanation. DO NOT MAKE UP QUESTIONS: ONLY EXTRACT THEM FROM THE FOLLOWING MESSAGE.

Identifying features relevant to the question. For each extracted question, we use LOTUS’s semantic filter operation to decide which features are relevant to answering the question. Each feature is filtered using the following prompt:

The user has a preference for feature {column_name} with description '{description}'. The preferred value is {values}. Decide if this preference is directly relevant to answering this clarification question from a customer service agent: {question}

Generating question answer. The LM generates a short first-person reply conditioned on the string rendering of the current preference state \(S_t\), the question text, and a natural-language hint built from the relevant features and their values in \(S_t\).

You are the SHOPPER (customer) answering a question from the customer service agent.

Your background: {simulator_persona}

Your current preferences (use only these to answer):
{z_context}

Make sure you provide the preference for this feature when answering the question: {feature_hint}
Try not to directly quote this and paraphrase it instead.

Question from the agent: {question}

Answer in 1-2 sentences. Be brief and conversational, but not too confident (e.g., 'Red would be nice' rather than 'I want red and nothing else.'). If your preferences above do not contain the answer, say "I don't know yet" or "I'm open to anything". Do not invent details.

9.1.0.2 Item examples.

For each recommended item, the simulator uses the agent’s textual description to evaluate, per known feature, whether the item matches the target preference (match / mismatch / not enough information). The LM is then prompted to write two sentences of feedback conditioned on the current state \(S_t\), emphasizing mismatches and surfacing newly realized preferences.

Evaluating whether items match preferences. For each item, we use LOTUS’s semantic map operation to identify if the item matches user preferences. The operation classifies whether the agent’s description implies a value that matches, contradicts, or leaves undetermined the target, yielding YES, NO, not_mentioned, or agent_uncertain.

Evaluate whether the text supports the following requirement.

Feature: {column_name}
Meaning: {column_description}
Required value: {true_value}

Decision process:
1. Look for text that provides clear evidence about the feature {column_description}.
2. Infer the value implied by the text, or if there is not enough information, return not_mentioned.
3. Compare the inferred value to {true_value}. Determine whether the implied value semantically matches the user's requirement. Be generous: treat synonyms, paraphrases, compatible values, and values that fall within the user's stated range as matching. Only say NO if the implied value clearly contradicts or does not satisfy the user's requirement.

Be generous with matching. For example, if the feature name is "buttoned-cuffs" and the presented value is "buttons" and the requirement is True, these are the same: the presented value is that the buttoned cuffs have buttons. If the feature name is "product type" and the presented value is "dress" and the actual value is "garment full body" these are also the same: a dress is a kind of full body garment. Another example: a series with 0 books in it is the same as a series with 1 book in it (both values mean 'standalone book'). A movie rated PG-13 or G is the same as being "not rated R." If a movie has Latin dub and the requirement is "Spoken language: Swedish or Latin", then this movie matches the requirement. If a movie is mentioned as having two protagonists (a male and a female), and the requirements is "Protagonist gender: female", then this movie matches the requirement: at least one protagonist has the right gender.

Only say NO if the implied value clearly contradicts the user's requirement. If a book described as a biogrpahy, then a feature like 'genre = fantasy' is NO.

Output rules:
- YES -> the text implies a value that does not contradict {true_value} or semantically matches it.
- NO -> the text implies a value that contradicts {true_value}.
- agent_uncertain -> the text mentions {column_description} but says the value is unknown or not in the catalog.
- not_mentioned -> the text contains no evidence about {column_description}.

Return exactly one of:
YES
NO
agent_uncertain
not_mentioned

Only return YES or NO if there is enough information presented to make a clear inference. If the feature is not explicitly mentioned, favor not_mentioned.

Text:
{message}

Shopper feedback on a recommended item. Given \(S_t\), the agent’s description, and the per-feature match summary, the model writes brief, conversational feedback: emphasize failures and uncertainty, allow at most one positive aside, and surface newly realized preferences when the item interaction triggers them.

You are the SHOPPER (customer) evaluating an item the agent recommended.

My current preferences (what I'm looking for):
{z}

Item ID: {item_id}

Description from the agent:
{raw_description}

=====================

How this item compares to my requirements:
{comparison_section}

Features I care about that the agent didn't clarify for this item:
{missing_section}

New preferences I just realized matter (and how this item fits):
{newly_revealed_section}

=====================

Write exactly 2 sentences of brief, natural feedback on how the item matches or doesn't match your requirements. YOU DO NOT CARE ABOUT ANY FEATURES OTHER THAN THE REQUIREMENTS ABOVE. Keep it conversational, not formal, like a text message. You do NOT need to list every preference that the item satisfies -- at most, mention one thing that looks good. Clearly point out the ways the item fails your requirements. If any mismatches are because you just realized you care about something new, say that explicitly (e.g., "Oh, I actually do care about color and I want it to be blue, so this red one doesn't quite fit"). For any feature that is missing or unclear, especially newly realized ones, phrase it as an uncertainty statement to the agent (e.g., "I'm not sure if it has adjustable shoulder straps, which I'd like."). Write in the FIRST PERSON as the shopper. Paraphrase, don't quote the preferred values.

9.1.0.3 Explanations.

The simulator detects paragraph-length educational spans in the agent’s message and maps each to a catalog feature. Only explanations that cover the target value allow the corresponding credence feature to enter \(S_t\). The LM then generates a response acknowledging the agent’s wording and, where the feature is now understood, articulating the desired value.

Detecting explanations. The LM identifies paragraph-length educational spans, maps each to a catalog column, and judges whether the explanation’s options or ranges plausibly include the simulator’s target value.

You are a helper that identifies and extracts extended educational explanations from a customer service agent message.

An EXPLANATION is when the agent dedicates AT LEAST A PARAGRAPH of text to teaching the user about a feature: what it means or what values/options it can take (e.g. "Sustainable items are made with...", "Some lengths include floor-length, ankle-length, calf-length...", "Some patterns include stripes & florals."). Do NOT treat questions as explanations: e.g. "What skirt length are you looking for?" is a question, not an explanation; return no explanation for that. Do not treat short lists inside questions ("What colors (e.g. blue, red) are you looking for?") as explanations. This is not a paragraph-length explanation.

You are given:
1. The assistant message.
2. A JSON object whose KEYS are catalog column names and whose values describe each column and give example values. This JSON is restricted to the small subset of catalog features we are currently interested in (the CREDENCE features).
3. A JSON object "target_values" whose KEYS are the same catalog column names and whose values are lists of the TARGET value(s) for that feature (the item(s) we care about). Use this to set "target_value_mentioned_or_in_range".

Your task: If the agent explains a feature (describes what the feature is or what options/values it can take), return an object with:
- "relevant_column": the exact catalog column name from the JSON keys that is being explained (e.g. dress_length for skirt/dress length, graphical_appearance_name for patterns).
- "target_value_mentioned_or_in_range": boolean. True if the explanation mentions or includes the target value for this feature. The explanation may describe a range or list of options; if the target value (or a broader category that includes it) is mentioned, set true. Examples: if the explanation says "Dress patterns include stripes and polka dots" and the target value is "large stripes", set true (stripes is mentioned). If the target value is "chevron" or "solid", set false (not mentioned). If the explanation lists "floor-length, ankle-length, calf-length" and the target value is "calf-length", set true.
- "justification": a short sentence explaining why the target value is mentioned or in range.
There should only be one object per feature in the list.

Examples:
- Message: "A dress's skirt length refers to how long it is on the knee. Some common lengths include floor-length, ankle-length, calf-length, knee-length, and mini, meaning above the knee. Typically longer lengths are considered more modest. Shorter lengths can sometimes elongate the leg." with target value "calf-length" -> {"relevant_column": "dress_length", "target_value_mentioned_or_in_range": true, "justification": "The explanation lists 'calf-length' as a common length."}
- Message: "What skirt length are you looking for?" -> [] (no explanation; this is a question)
- Message: "Some patterns for dresses include stripes & florals." with target value "large stripes" -> {"relevant_column": "graphical_appearance_name", "target_value_mentioned_or_in_range": true, "justification": "The explanation lists 'stripes' as a pattern from which an average internet user could derive the target value 'large stripes'."}
- Message: "Some patterns for dresses include stripes & florals." with target value "chevron" -> {"relevant_column": "graphical_appearance_name", "target_value_mentioned_or_in_range": false, "justification": "The explanation does not mention the target value 'chevron'."}
- Message: "Price refers to the cost of the dress. A typical price for a dress is $50 to $100." with target value "under $70" -> {"relevant_column": "price", "target_value_mentioned_or_in_range": true, "justification": "The target range (0-70) overlaps with the explanation range (50-100)."}
- Message: "Price refers to the cost of the dress. A typical price for a dress is around $100." with target_value "under $20" -> {"relevant_column": "price", "target_value_mentioned_or_in_range": false, "justification": "The target range (0-20) does not overlap with the explanation range, which is around 100. 'Around 100' can be treated as (80-120)."}
- Message: "What kind of material a dress is made of affects its look. For the specifications you gave, I would recommend materials like cotton and polyester." with target value "viscose" -> {"relevant_column": "material", "target_value_mentioned_or_in_range": false, "justification": "The explanation doesn't list the target value 'viscose', and an average internet user could not derive viscose from the other presented options."}

Catalog columns (JSON):
{features_json}

Target values per column (JSON):
{target_values_json}

Assistant message:
{message}

If there are no such explanations, return an empty list.

REMEMBER: an explanation is at least a paragraph of text dedicated to teaching the user about a single feature.

Output format: Return ONLY a JSON array of objects, each with exactly: "relevant_column" (string) and "target_value_mentioned_or_in_range" (boolean). No other keys or text.

Shopper reply after the agent explains credence features. The model acknowledges the agent’s wording, then paraphrases the desired values for features so the turn reads like a natural shopper reaction to a tutorial.

The customer service agent just provided technical explanations for the following features:

{credence_to_answer} {credence_to_acknowledge}

STRICT RULE: If a technical term or hidden feature name does not appear in the agent's own wording, you must NEVER use it in your response -- not even to refer back to what they said. Only use the language and concepts the agent actually used when talking to you, in plain, everyday words.

Write a response in first person acknowledging these explanations and then communicating the desired values for these features: {credence_to_answer}

Here are our desired values:

{z}

Write in first person as the shopper with these preferences, and use natural, conversational language that avoids technical jargon and column or tag names. Do not directly quote any line from the preferences specification (for example, never say things like 'Has stretch: True'); instead, paraphrase it in natural, conversational language.

Example 1 (materials): Agent explains different materials after asking "Do you care about the material or fabric?" -> Shopper: "Thanks, that helps a lot. Based on what you said, I'd really like something in a softer, breathable fabric that feels comfortable to wear for a long time."

Example 2 (decades): Agent talks about different decades after asking "Is there a decade for a movie you're interested in?" -> Shopper: "Got it, that makes sense. From your explanation, I think I'd most enjoy something set around the 80s vibe you described."

Example 3 (waist type): Agent explains waist styles after asking "What kind of waist would you like?" -> Shopper: "Thanks for breaking that down. From what you described, I'd prefer the kind of waist that feels more structured and flattering rather than super loose."

9.2 Evaluating internal validity: parsing quality↩︎

To assess the simulator’s internal validity, we evaluated whether the two core parsing steps — mapping questions to relevant features, and judging item-preference matches — produce outputs consistent with human judgment on random samples drawn from test runs (Table ¿tbl:tab:message-parser-questions? and Table ¿tbl:tab:message-parser-features?). All evaluations use gpt-oss-120b as the underlying language model.

Note that in Appendix 11.3, we also run CoShop-structured, a variant of the benchmark that requires agents to explicitly name which features their dialog actions target. This reduces the simulator parsing steps evaluated here to just two-way item-feature matches (match / contradict the target values). We observe that removing the parser dramatically drops agent performance.

For clarifying questions (Table ¿tbl:tab:message-parser-questions?), annotators judged which catalog columns are relevant to answering each question; the simulator may output a set of columns or abstain (empty set). We score question-level accuracy as the fraction of questions on which either (i) both simulator and human abstain, or (ii) the human does not abstain and the simulator names at least one column the human selected. On \(n{=}50\) such questions, accuracy is 74.0%, and when both sides commit to at least one column, agreement rises to 91.3%. Treating simulator abstention as the positive class, abstention precision is 59.3% and recall is 100%: whenever the simulator abstains, a human would also abstain in only about three fifths of cases, so the simulator over-abstains relative to humans. The feature-level rows aggregate micro-precision, recall, and F1 over every (question, column) pair; compared to a random baseline that flips a fair abstention coin per question (and, at the feature level, resamples as many predicted columns per question as the simulator, uniformly from the allowlist), the simulator remains far stronger (e.g.micro-F1 0.711 vs.).

For item–feature judgments (Table ¿tbl:tab:message-parser-features?), humans labeled whether the agent’s description of an item, relative to a known preference, matches the target value, contradicts it, or leaves the feature underspecified. On \(n{=}50\) such (item, feature) pairs, the simulator’s 3-way classification matches humans with 94.0% accuracy, versus 33.3% when guessing one of the three labels uniformly at random and 48.0% under a per-judgement majority baseline. Separately, collapsing to whether the simulator claims enough evidence to judge (non-missing vs.missing), precision for non-missing is 92.9% with recall 100% (F1 96.3%), indicating reliable detection of when the text supports a firm match or mismatch.

10 RAG agent details↩︎

In Section 4, we benchmark language models on CoShop. These models use a simple RAG agent harness, which we implement using langchain.

To keep context length manageable across multi-step agentic rollouts, we summarize accumulated tool results rather than carrying their full text forward. Every tool message is left intact for the first action that immediately follows it; only once a second unsummarized result arrives does compression fold the stale result into a summary. Summaries are generated by gpt-oss-120b, and each tool result is summarized at most once.

10.1 Baseline RAG agent prompts↩︎

This is the primary harness we benchmark with. The agent is prompted with a high-level description of the SEC model and the three dialog actions.

Conversation system message

You are a helpful assistant working with a user to find the right {item_name}. This conversation is a preference-development phase: your goal is not to find the answer yet, but to understand and help the user develop their preferences so the search that follows can succeed. You have a small budget to search the catalog to find example items for discussion during this phase. After the conversation, you will have a separate, larger budget to search the catalog thoroughly. Use the conversation to prepare for that step — not to solve it.

Note that the only preferences we care about for the user are features relevant to available items. The user will be able to reason best over actual catalog features, rather than general questions about their context.

Communicate with the user in natural language. Whenever you reference a specific catalog item — whether introducing it, answering a follow-up question about it, or adding new details — you MUST include a JSON line for it. Each JSON line must appear on its own line, contain an 'id' field with the item's id from the web results, and include all item-specific details as additional keys. Use double quotes for all keys and string values. Do not put any other text on that line. Example: {"id": 1234, "color": "black", "pattern": "black-and-white floral"}.
CRITICAL: If the user asks a follow-up question about a specific item, your text answer is fine, but you MUST still include a JSON line for that item in the same response — even if you already showed it before. All item-specific details belong in the JSON; do not reference them only in prose.To show the user a message, do not make tool calls in that message.

You have a budget of 5 turns for this conversation. One turn consists of a user message followed by your response. You can ask at most 20 clarifying questions. The user can review at most 10 unique items. Use your turns wisely to understand the user's preferences, constraints, and context.

Keep your messages brief to avoid overburdening the user. You do not need to find the item during this conversation — you only need to understand and develop the user's preferences. When you feel confident about their preferences, generate <END_CONVERSATION> to move to the research phase. Do not generate <END_CONVERSATION> in any other message, or the interaction will end prematurely.

People vary in how much they already know about their own preferences, and not every preference can be unlocked by asking a direct question.

A useful framing is to think about three situations a user might be in for any given feature:

1. The user already has a settled preference. They can answer a direct question accurately — they have reliable past experience with this feature and know what they want. Asking directly is the right move.

2. The user can form a preference by seeing examples. They have no strong prior opinion and cannot answer abstractly, but will react clearly when shown a concrete item. Asking abstractly may not work here — they need to see it first. The right move is to show items that highlight this feature, then invite the user to react.

3. The user needs domain knowledge before they can form any preference. They have not encountered this feature before, or the terminology is unfamiliar. Even seeing examples may not help without first understanding what the feature means. The right move is to briefly explain the feature and why it matters, then ask.

You will need to infer which situation applies — it is not told to you. Some signals: confident direct answers suggest a settled preference; hesitation or uncertainty suggests the user may need examples or explanations.

When multiple features fall into situations 2 or 3, be selective: surface only the most impactful unknown(s) at a time. Addressing every underspecified feature at once creates cognitive overload.

Research system message

========= FINAL RECOMMENDATION TIME =========
Your task is to thoroughly explore the web using the search_web tool and return the top {k} items that best match the user's preferences.

Based on your conversation with the user and your understanding of their preferences, constraints, and context:
Use the search_web tool to explore items. You can make a total of 250 items across all queries.

The moment you don't issue a tool call, this step will end.

After exploring, return a JSON object with a single key "top_k_items" containing a list of exactly {k} item IDs in ranked order (best match first).

The JSON object should have this format:
{
    "top_k_items": ["item_id_1", "item_id_2", ..., "item_id_{k}"]
}

Forget all other formatting instructions from before this. You are no longer talking to the user, so return ONLY the JSON object as a string without any other text or formatting. The item IDs should be the "id" field from the search_web results

Report generation system message

You are given a set of candidate items to recommend to a user. You have already seen the conversation with the user in your existing context.

Your task is to produce one JSON object per catalog id (in the order shown below) explaining fit for the user, using the SAME output convention as when you describe items in this task: each value MUST be a JSON object (not a prose string) with an "id" field matching that catalog id, feature keys with updated values where helpful, and any extra keys you need to justify fit (e.g. rationale in string fields). Use double-quoted keys and string values. Do not wrap item details in free-floating prose outside those objects. Aim to make each object self-contained so the user can decide from these JSON lines alone.

Items (in recommended order):
[one line per id: id and catalog_json value, inserted at runtime]

Return ONLY a JSON object mapping item ids (strings) to JSON objects.
Example:
{
  "id_1": {"id": "id_1", "color": "navy", "why_it_fits": "Matches your preference for dark solids."},
  "id_2": {"id": "id_2", "pattern": "striped", "why_it_fits": "Conflicts with your dislike of busy patterns."}
}

Principles:
- You are an impartial, comprehensive, and honest advisor. Your main job is to provide the user with the information they need to make a decision on what is best for themselves.
- Write to the user in second person, and reflect their preferences and constraints in your response.
- Make sure your descriptions are entirely self-contained.

Include EVERY item. Do not include any other text.

10.2 History-aware agent prompts↩︎

In Section 4.3, we gave agents access to a tool that allows it to access user history. This design extends the baseline RAG agent with access to the user’s historical purchase/rating data via a pandas REPL tool. The tool description below is what the model sees in its tool list.

Run a single pandas expression over the user's history, exposed as a pandas DataFrame named `df` and indexed by item id. Only `df` and `pd` are available; the expression must be a single pandas/Python expression with no imports, assignments, or statements. Each call is independent: `df` is reset to the user's full, original history every time, so results do not carry over between calls — express each query against the complete history. The available columns are not listed here; inspect them yourself (e.g. "df.columns.tolist()" or "df.head()") before relying on specific column names. Examples: "df[df.user_rating_of_5 >= 4].sort_values('year').head(10)"; "df.groupby('genre')['user_rating_of_5'].mean()"; "len(df)". At most 10 rows are returned for table/series results, and the returned text is truncated to 2000 characters — prefer expressions that select only the columns/rows you need.

10.3 Structured-action agent prompts↩︎

In Appendix 11.3, we experimented with using a structured-action agent. The conversation system message is identical to the baseline agent except that the item-format instructions are replaced with the structured-action block below (all other text is unchanged).

Communicate with the user in natural language when appropriate, but you MUST encode specific dialog actions as separate lines using this exact format: one keyword, a single space, then one JSON object (double-quoted keys and strings).

1) ASK_QUESTION — ask the user about a preference or constraint.
ASK_QUESTION {"question": "What is your budget?", "relevant_features": ["price"]}
relevant_features: catalog feature names this question is about (may be multiple).

2) SHOW_ITEM_FOR_FEEDBACK — proactively present an item for the user to react to.
SHOW_ITEM_FOR_FEEDBACK {"item_id": "12345", "features_for_feedback": {"color": "navy", "material": "cotton"}}
item_id: id from search/web results. features_for_feedback: map from catalog feature name to the value you are showing the user (all item-specific details must appear here).

3) ITEM_FOLLOWUP — use this whenever you answer a user question about a specific item, or need to surface additional details about an item you already showed. Your prose answer is fine, but you MUST also emit this action so the item details are captured.
ITEM_FOLLOWUP {"item_id": "12345", "features_for_feedback": {"color": "navy", "material": "cotton"}}
Use the same field names as SHOW_ITEM_FOR_FEEDBACK. Include all item-specific details the user needs inside features_for_feedback — not in the surrounding prose.

4) EXPLAIN — a paragraph or more teaching the user about a feature or concept.
EXPLAIN {"explanation_text": "Dress length usually means...", "relevant_features": ["dress_length"]}
relevant_features: catalog feature names the explanation addresses.

Actions can be freely combined within the same message. For example, you can show an item highlighting certain features with SHOW_ITEM_FOR_FEEDBACK, then immediately follow it with an ASK_QUESTION about one of those features — this lets you ground a question in something concrete the user just saw. Similarly, an EXPLAIN can be paired with an ASK_QUESTION to teach and then probe in a single turn.

Rules:
- One action per line; freely mix action lines with normal prose on other lines.
- If you ask a question in prose without ASK_QUESTION, the user cannot respond to it.
- If you discuss an item without SHOW_ITEM_FOR_FEEDBACK or ITEM_FOLLOWUP, the item details will not be captured.
- All feature names must exactly match catalog column names.

10.4 Catalog search tool↩︎

Agents are given a search_web tool backed by HNSW vector search [46] over Qwen-3-Embedding-8B embeddings of featurized catalog items \(\phi(x)\) for \(x \in \mathcal{X}\) [47]. Each call takes a natural-language query string, a max_items count, and an optional filters dict mapping catalog column names to lists of exact allowed values. The tool returns the top-\(m\) items ranked by embedding similarity, restricted to items whose column values match all filter conditions. The agent thus has two complementary handles on the catalog: free-form semantic query and explicit column-value pinning.

The following is the tool description the agent sees in its tool list.

Search the web for products. The first argument is a string used to search the web. The second argument is the maximum number of items to return for this query. The more specific the query, the more likely you are to find a particular product. If the query is too specific, you may get no results. An empty query will return the entire web. The search query should be in natural language.

Optionally, supply a 'filters' dict mapping column names to a list of EXACT allowed values to restrict results to items matching those column values. This can help denoise your results. Use 'NA' as a value to match rows where that column is missing/null. 
IMPORTANT: there is no support for negation (no NOT) or inequality (no <, <=, >, >=).

Budget constraints: Total budget of {budget} items across all queries.

Automatic prefilter. To increase the catalog search tool’s recall, every query also passes through an automatic prefiltering step before vector search. A language model (gpt-oss-120b, temperature 0) reads the raw query string and the list of filterable catalog columns and generates a list of simple pandas DataFrame.eval() boolean expressions. These are applied greedily in a round-robin over multiple orderings: each expression is kept only if it does not reduce the matching set to zero; kept expressions intersect to form the filtered candidate pool. The vector search then runs over the intersection of this pool and any hard-filter ids; if the result is still fewer than max_items, the system supplements from an unrestricted baseline query. The prompt used to generate prefilter expressions is below.

[system]
You produce a list of SIMPLE pandas DataFrame.eval() expressions. Each expression must evaluate to a boolean Series. We will apply them one by one and take the intersection of matching rows; if an expression would leave zero rows, we skip it. So each expression should be ONE simple clause (e.g. one column comparison). Do NOT combine multiple conditions with & or | in a single expression — use separate list items instead. Use only column names provided. For string literals always use single or double quotes (e.g. 'Comedy', "Drama"). Be case sensitive. Use backticks ONLY for column names that contain spaces or are not valid Python identifiers (e.g. `Area (cm^2)`), never for string values. When the user lists multiple values for the same attribute, use one expression per value (e.g. one item "genres.str.contains('Comedy')", another "genres.str.contains('Drama')"). Return a JSON object with a key 'expressions' containing a list of expression strings. No explanation, no code block. Try not to read into semantic meaning. Filter based on obvious intent. Avoid filtering on titles or descriptions. Fewer, simpler expressions are better than one long combined expression. DO NOT add any filters that are not clearly implied by the string, in particular do NOT add an is_ebook filter. For example, 'book popular' does not mean you should threshold on avg_rating or num_ratings because that is reading too much into the string, and 'recent' does not mean you can safely filter on a year because that is reading too much into the string.

[user]
Query: {query}

Filterable columns (name -> type): {column_types}

Example values by column: {examples_by_column}

Give a list of simple pandas eval expressions (booleans) to filter rows. Return JSON: {"expressions": ["expr1", "expr2", ...]}

11 Additional CoShop results↩︎

11.1 Detailed results↩︎

11.1.0.1 Agent performance across user types, by dataset.

Figure 5A in the main text shows the performance drop across user types averaged across datasets. Table 6 shows the same comparison (fully-specified, search-only, CoShop) broken out per dataset.

11.1.0.2 Utility metrics by dataset.

Table 1 reports exact-match item accuracy. Table 7 reports utility-based results instead. Unlike exact match, utility rewards recommendations that are close to \(x^\ast\) even when they do not match it exactly. It is computed as \[u(x, x^*) := \frac{1}{|\mathcal{F}(x^*)| + 1} \left( \text{EmbeddingSimilarity}(x, x^*) \sum_{j \in \mathcal{F}(x^*)} 1\{\phi_j(x^*) = \phi_j(x)\} \right)\] where EmbeddingSimilarity is rescaled cosine similarity between Qwen-3-Embedding-8B embeddings of the \(x\) and \(x^\ast\) catalog entries.

Agent utility under complete reports (“oracle reports”) shows how much team utility is lost to report quality alone.

11.2 Sensitivity analysis to budget parameters↩︎

To verify that our results are not artifacts of these particular budget choices, we ran a sensitivity analysis on a subset of users comparing CoShop performance under the full budget constraints against a relaxed setting with only a turn budget. As shown in Table 8, team accuracies are not meaningfully sensitive to the question and item budgets, with the exception of gpt-oss-120b on H&M, suggesting these limits were already non-binding for most agents.

11.3 Structured dialog actions↩︎

In order to study agents in as natural a setting as possible, agents emit natural language messages in CoShop. To compute state updates, we used gpt-oss-120b to parse the agent’s message into dialog actions and, critically, identify the features being referenced by each parsed action. This parsing step introduces some error, mainly around identifying the referenced features: in Appendix 9.2, we found that gpt-oss-120b has question parsing accuracy of 74.0% and item parsing accuracy of 94.0%, with a tendency to overabstain on questions. This parsing step introduces a potential confound: performance may suffer because the parser misidentifies which features are relevant to what the agent is asking about / showing / explaining.

To isolate this, we ran a variant in which agents emit structured dialog actions rather than free-form text, each of which must explicitly reference the target feature by its catalog column name. In this setting, each agent turn consists of one of three typed actions ask_question, show_item_for_feedback, or explain. This eliminates parsing error entirely, at the cost of realism: agents can no longer use natural language freely, and the constraint may itself change what agents choose to say.

Table 9 compares natural and structured formats across all metrics, averaged across datasets. Removing the parsing step dramatically degraded performance for smaller models, which tended to hallucinate column names even when their underlying natural language was coherent — indicating that parsing error is not the main driver of failures observed in the natural setting.

12 Extended qualitative analysis↩︎

12.1 Example clarifying questions generated by gpt-5.2↩︎

Figure 9 provides a sample of questions asked by gpt-5.2 on a H&M instance. gpt-5.2 asks the most questions in interactions, including the most ungrounded questions (questions asked based solely on pretraining knowledge, generated before reviewing the catalog).

12.2 Example of premature conversation ending by gpt-5.2↩︎

Figure 11 shows a representative transcript in which gpt-5.2 ends the conversation early to move to the research stage. After a single round of clarifying questions, the user volunteers that they are “open” or “flexible” on most remaining attributes, and gpt-5.2 treats this as license to stop eliciting — summarizing a target with several dimensions still unspecified and proposing to move on, rather than probing the dimensions the user left open. This illustrates the “execution bias” discussed in Section 4.4: the model is reluctant to spend additional user turns even when the preference state is still underdeveloped.

12.3 Annotated walkthrough↩︎

In this section, we do an extended qualitative analysis of sample runs from CoShop, walking through how different models respond to the same user.

User persona:

12.3.1 Walkthrough: gpt-5.2(successly finds \(x^\ast\) in 10 turns, 20 questions)↩︎

TL;DR. gpt-5.2 asks the user 20 clarifying questions, only some of which the user can answer, before ever reviewing the option set \(\mathcal{X}\). Several of these questions are repeated, e.g. confirming a "no pockets" preference multiple times. gpt-5.2 ultimately gets in two rounds of item reviews; its descriptions of items are detailed, leading the user to acquire many experience features. The agent learns quickly from these new experience features and finds the target item on the last turn. The conversation ends due to exceeding the turn budget.

gpt-5.2 immediately asks eight clarifying questions before making any tool calls to review what features are available in the catalog.

Questions 1 and 6 add search features Formality, Pattern, and Color family to the user state. The user has no defined preference for sleeve length (Question 2), and so abstains. Question 4 should map to the Neckline feature, which is also a search feature; this should have been added to the user state, but a simulator error (incorrect abstention on a clarifying question) causes this feature to remain unadded (see Appendix 9.2). Question 8 maps to the Price feature, giving the user budget, but this is an experience feature and is not added since no items are shown. Question 8 also asks about concepts that seem reasonable according to pretraining knowledge, but are not relevant to the actual catalog available, such as size and shipping information. Questions 5 and 7 touch on the credence Fabric and Waist shaping method features respectively, which triggers the simulator to ask for an explanation of these features.

The agent’s fabric explanation highlighted the user’s target preference of cotton, so this feature is added to the state. gpt-5.2 follows up with several more clarifying questions, including a redundant question about pockets and more technical questions about fabric properties. The question about machine-washability touches on a credence feature “Functional performance feature”, which was omitted from Table 4 from brevity. Similarly, the question about pocket details touches on another credence feature “Pocket configuration.”

We omit the next five turns, in which gpt-5.2 asks additional clarifying questions without making any tool calls or showing the user any example items. We list these additional questions across all five turns here as an illustration:

Notice that many questions are redundant, e.g. confirming the no pockets requirement multiple times, and probing the price feature multiple times.

gpt-5.2 ends up asking 20 total clarifying questions like this without reviewing the catalog, exhausting the question budget.

The next agent turn searches the catalog and shows example items.

The item examples add experience features to the user state depending on what features they mention. Since gpt-5.2 uses detailed item descriptions, it adds many experience features, such as the user’s budget, preferred neckline, and preferences for whether the fabric has stretch and whether the dress has lining. Although from context gpt-5.2 has probably retrieved dress results, the product type is not noted in the item descriptions, prompting the simulator to clarify that the results actually are dresses.

At this point the conversation ends due to exceeding the turn budget.

12.3.2 Walkthrough: claude-sonnet-4.6(fails to find \(x^\ast\) in 4 turns and 8 questions)↩︎

TL;DR. claude-sonnet-4.6 asks the user 6 clarifying questions, explains two credence features, and then ends the conversation prematurely on turn 4. It never shows the users any items during the conversation.

claude-sonnet-4.6 immediately asks six clarifying questions before making any tool calls to review what features are available in the catalog.

Questions 1 and 3 add search features Formality, Pattern, and Color family to the user state. Question 5 maps to the Price feature, giving the user budget, but this is an experience feature and is not added since no items are shown. Question 4 touches on the credence Fabric feature respectively, which triggers the simulator to ask for an explanation of these features.

The agent’s fabric explanation highlighted the user’s target preference of cotton, so this feature is added to the state. The user doesn’t have a target value for the sleeve length question.

We omit two turns of additional clarifications. On Turn 5, claude-sonnet-4.6 issues the preference summary below and then emits the “End Conversation” token, terminating the conversation without searching the catalog.

13 Additional related work↩︎

13.0.0.1 Strategies for resolving underspecification.

A growing body of work studies agents that must resolve underspecified requests through interaction [6], [31], [48]–[52]. In the shopping domain specifically, most benchmarks are non-interactive [33]–[37], and the interactive exceptions, such as [25], [26], still assume a user who can provide any preference upon questioning. The common thread is that specification is treated as retrieval of a preexisting preference, done primarily through clarifying questions. Our contribution is to model specification as construction: the user’s preference state does not exist in complete form at the outset but is built through the agent’s choice of clarifying questions, examples, and explanations.

13.0.0.2 Conversational recommender systems.

While traditional recommender systems rely on user-item interactions, like ratings, in order to elicit user preferences, conversational recommendation uses language to elicit user intent and preferences [53]–[56]. A substantial line of work studies how to elicit preferences, such as question-generation strategies [57]–[60] and strategies for selecting items to elicit user feedback [61], [62]. A recurring finding is that users cannot simply answer clarifying questions about catalog features, as if filling in slots in the system’s mental model; instead, some features are easier for users to answer questions about than others [16], [17]. By using CoPref users with an SEC decomposition, CoShop is a benchmark aligned with this strain of thinking: rather than simply evaluating if the agent can identify what catalog slots to ask about, we evaluate if the agent can provide the necessary context to help the user construct a preference.

13.0.0.3 User simulators.

A body of work studies how to simulate users for training and evaluating task-oriented agents, both for conversational recommendation and in broader tasks [40]–[44], [63]. Naive LLM-based simulators have been shown to underrepresent behavioral diversity [64]. To improve simulator faithfulness, some works take a bottom-up, data-driven approach (i.e., finetuning language models on user utterances). We instead take a top-down, model-based approach (i.e., defining a model of human behavior to condition a language model on).

Prior work also finds that if shown \(x^\ast\), simulators may leak titles [65] or extra features that were not asked about. We avoid these problems by restricting the context of the LLM to exactly the current preference state \(S_i\), which does not include features like titles that uniquely identify the target item.

13.0.0.4 Mixed-initiative systems.

The question of who drives an interaction—and when control should pass between human and system—predates conversational LLMs. [66]–[68] frame mixed-initiative interaction as the interleaving of contributions so that each agent does what it is best suited to at the appropriate moment. Within recommendation specifically, [69] study initiative transfer and find that the most effective starting mode depends on the user, and [70] show that conversational search strategies—clarify-then-execute versus execute-then-clarify—have nuanced, user-dependent payoffs. Our three dialog actions (clarifying questions, example items, and explanations) are precisely initiative-taking moves.

References↩︎

---

1. We count a turn as one user message followed by an agent message.↩︎