1 Introduction↩︎

Large Language Model (LLM)-powered agents extend foundation models with additional modules for memory, planning, and execution, enabling multi-step, goal-directed reasoning over long contexts [1]–[4]. As shown in Figure 1, a central challenge for LLM-powered agents is personalization: user objectives are not explicitly specified; preferences are implicit and partially observed through histories; and correctness is user-centric rather than globally optimal [5]. In practice, users implicitly reason over their own past experience, translate vague intent into specific criteria, and discriminate between plausible options [6]–[8]. Accordingly, a personalization agent should inherently align its planning and action to one particular user’s intents rather than generate generic responses following a fully specified goal [3]. Therefore, we ask How can we benchmark the agentic personalization capability of LLMs? To our knowledge, existing benchmarks individually address one or two of these axes but never combine noisy heterogeneous histories, vague-to-refined search trajectories, and closely competing candidates. Recommendation benchmarks cast personalization as next action prediction from historical signals, emphasizing preference deduction over objective understanding [9], [10]. Rationale prediction benchmarks evaluate localized reasoning, focusing on generating intermediate justifications in isolation rather than reaching a user-aligned outcome [11], [12]. Multi-turn Memory benchmarks model personalization over largely homogeneous conversations, evaluating relevant information extraction instead of intent inference under ambiguity [4], [13]. In all, these benchmarks sidestep the need to jointly reason over noisy histories, unrefined user queries, and closely competing candidates. As a result, they also fail to evaluate whether agentic pipelines (e.g., ReAct) contribute to intent inference and preference-aligned actions [14].

Motivated by these limitations, we construct APeB(Agent Personalized Benchmark) to evaluate agents’ personalization ability through the Personalized Product Search (PPS) task. PPS frames personalization as a user-specific choice among competing products conditioned on a shopping intent and long-term behavioral preference [15], [16]. On online platforms, user behavior is noisy across domains, shopping intent is often vague and early-stage, and purchases occur only after comparing similar products [6], [17]–[19]. However, prior PPS benchmarks rely on sanitized histories, clarified queries, and easily separable candidates [8], [20], thereby failing to evaluate personalization under ambiguous, reasoning-intensive conditions. The preference-aligned purchase also contrasts with generic shopping benchmarks that focus on isolated sub-tasks (e.g., product matching) without grounding decisions in user-specific intents and histories [21], [22]. Comparison with related benchmarks is shown in Table ¿tbl:tab:compare?. Full related work is in Appendix 6.

In turn, APeB contains intricate cases drawn from a leading content platform with shopping functionality. It includes 5,648 cases from 10+ categories collected from a 30-day retained-session window with up to 60 days of pre-session history (Section 3). Users primarily consume videos and livestreams, yet can seamlessly transition to product search, browsing, and purchase within the same platform. This end-to-end flow yields rich, heterogeneous interaction histories that capture real-world personalization challenges (see Appendix 7.1). We disentangle user behavior into a long-term history period and a personalization shopping session, delimited by the timestamp of a user-issued search (Section 2.1.1). Sessions are retained only if the search leads to a query-aligned purchase following reasoning-intensive browsing (Section 2.1.2). The history period comprises diverse interactions with products and contextual content (e.g., videos and livestreams), providing long-term personalization signals (Section 2.2.1). The shopping session contains user searches spanning vanilla intent to refined keywords (Section 2.2.2), together with a set of heavily viewed competitive candidates (Section 2.2.3) and a final observed purchase, enabling intent evaluation and verifiable user-centric assessment. We evaluate whether personalization agents can first infer a user’s latent preferences from vague intents and history hints (Planning) and select among close alternatives (Action) on the user’s behalf (the pipeline defined in Figure 1).

We test latest LLMs with agentic workflows on . Evaluation spans and queries with random and hard candidates, and includes comparison against a PPS model UniSAR [23]. Results show that (1) in settings dominated by semantic alignment rather than personalization ( queries or random candidates), LLMs substantially outperform UniSAR, and ReAct workflows further improve performance; (2) for tasks requiring user modeling and intent refinement ( with hard candidates), LLMs fall to the level of zero-shot UniSAR, with limited gains from ReAct workflows; (3) rubric analysis shows that queries expose weak intent and history attribution, with ReAct further harming performance via poor multi-step planning; (4) we find that a simple history-grounded ague uery efinement gent() pipeline improves intent inference and Hit@1 over single-prompt under .

Overall, serves as a testbed for evaluating agentic systems under realistic personalization constraints.

2 APeB Construction↩︎

We introduce APeB, a benchmark designed for evaluating agentic personalization capability on the PPS task. To retain cases in which the personalized decisions are time-consuming and interaction-heavy (Section 2.1.2), we collect non-trivial user shopping sessions with their heterogeneous histories mined from holistic user behavior logs (Appendix 7). As shown in Figure 2, a session is defined as a time-bounded search-to-order window that begins with a search and ends with a purchase (Section 2.1.1). Non-trivial sessions involve query reformulation and extensive product comparison before an intent-aligned order (Section 2.1.2), reflecting how user history shapes query formulation, exploration, and decision-making [24], [25]. These sessions inherently require an agent to jointly reason over long-term user preferences, vanilla query intent, and fine-grained candidate differences, rather than relying on surface-level semantic matching. For filtered sessions, we construct three benchmark components: (i) Heterogeneous histories–preference signals (Figure 2 Left, Section 2.2.1); (ii) User-issued (vague, early-stage) and (well-specified, late-stage) queries–different levels of query specificity (Figure 2 Right, Section 2.2.2); and (iii) Hard candidate set–products the user considered, together with the observed purchased item (Figure 2 Right, Section 2.2.3).

2.1 Non-trivial Shopping Session Discovery↩︎

To identify non-trivial shopping sessions from behavior logs, we first retain users with sufficient interaction density over a 60-day window (see Appendix 7.2). Following the long–short period division paradigm [10], we then extract search-to-order sessions by anchoring boundaries at search timestamps. From these, we further select non-trivial cases characterized by query evolution, extensive content browsing, and cross-product comparison prior to a personalized purchase decision.

2.1.1 Search-to-Order Session Disentangle↩︎

We disentangle each user’s interaction sequence by the search action timestamp (‘{’ in Figure 2). For a user \(u\), let \(\mathcal{I}_u = \{(x_u^k, t_u^k)\}_{k=1}^{N_u}\) denote the chronologically ordered interaction log, where \(x_u^k\) is an action and \(t_u^k\) its timestamp. We consider an initial search action \(x_u^m=\mathrm{query}(q)\) at time \(t_u^m\) and a final purchase action \(x_u^n=\mathrm{order}(p)\) at time \(t_u^n\). A search-to-order session is identified if the purchase occurs within a fixed time window \(\Delta=30\) minutes after the search: \(t_u^n - t_u^m \le \Delta.\) Users without any such session are removed; if multiple sessions exist, we retain one per user. For each retained session, the search time \(t_u^m\) and the purchase time \(t_u^n\) define the disentanglement boundary: \[\begin{align} \mathcal{H}_u & = \{(x_u^k, t_u^k)\in\mathcal{I}_u : t_u^k < t_u^m\}, \tag{1}\\ \mathcal{S}_u & = \{(x_u^k, t_u^k)\in\mathcal{I}_u : t_u^m \le t_u^k \le t_u^n\}. \tag{2} \end{align}\] Here, \(\mathcal{H}_u\) captures long-term preference signals, while \(\mathcal{S}_u\) contains goal-driven queries and candidates. This disentanglement separates preference inference from task execution , enabling reliable evaluation of personalization reasoning.

2.1.2 Non-trivial Session Filtering↩︎

For each user \(u\) with a session of ordered events \(\mathcal{S}_u\) (Section 2.1.1), each event is one of search, product browsing, media (video/livestream) browsing, or purchase. It can be represented as \[\mathcal{S}_u = \big( \mathcal{Q}_u^{\mathrm{search}},\; \mathcal{P}_u^{\mathrm{view}},\; \mathcal{V}_u^{\mathrm{view}},\; \mathcal{P}_u^{\mathrm{order}} \big),\] where \(\mathcal{Q}_u^{\mathrm{search}}\) is the ordered list of search queries, \(\mathcal{P}_u^{\mathrm{view}}\) the product-interactions, \(\mathcal{V}_u^{\mathrm{view}}\) the media-interactions, and \(\mathcal{P}_u^{\mathrm{order}}\) the purchased products. To obtain non-trivial shopping sessions, we formulate these three conditions:

(1) Iterative query refinement. The user progressively clarifies intent through multiple searches: \[\label{eq:query-refine} |\mathcal{Q}_u^{\mathrm{search}}| \ge 2.\tag{3}\]

(2) Extensive-browsing behavior. The user exhibits substantial exploration prior to purchase: \[\label{eq:extensive-browsing} |\mathcal{P}_u^{\mathrm{view}}| + |\mathcal{V}_u^{\mathrm{view}}| \ge \tau_v ,\tag{4}\] where \(\tau_v\) is a minimum interaction threshold.

(3) Intent–order alignment. The final purchase must be semantically consistent with both the user’s expressed intent and explored candidates. Specifically, there exists a purchased product \(p_u^{o} \in \mathcal{P}_u^{\mathrm{order}}\) such that \[\label{eq:intent-align} \begin{align} &\exists\, q \in \mathcal{Q}_u \;\text{s.t.}\;\mathbb{I}[\mathrm{Align}(q, p_u^{o})] = 1, \\ &\left|\left\{\, v \in \mathcal{P}_u^{\mathrm{view}} : \mathbb{I}[\mathrm{Align}(v, p_u^{o})] = 1 \right\}\right| \ge \tau_a , \end{align}\tag{5}\] where \(\tau_a\) is a minimum alignment count. If multiple purchases satisfy, we retain only one.

Criteria are primarily enforced via rule-based filtering (Appendix 7.3.1). Semantic alignment in (3) is assessed using GPT-4.1 with human verification (Appendix 7.3.2). We remove unsatisfied sessions and retain one session per user. These criteria target a common high-effort shopping regime, 67% of users exhibit multi-step shopping behavior in a long session, defined as issuing at least two queries and comparing at least ten products before purchase (Appendix 7.5.2).

2.2 History & Query-Candidate Identification↩︎

For users with retained sessions, we construct their heterogeneous histories and aligned query–candidate pairs.

2.2.1 Heterogeneous History Curation↩︎

Heterogeneous long-term interactions provide rich signals for inferring user persona. For a user \(u\) with historical interaction log \(\mathcal{H}_u\) (Eq. 1 ), we first filter out low–dwell-time records (Appendix 7.4.1). We then consider two complementary types for the remaining interactions.

(1) Product interactions. Each product interaction is defined as \[\label{eq:his-product-interact} x_{u,\mathrm{product}}^k = (\mathrm{meta}_p^k,\;\mathrm{type}^k,\;\Delta t^k),\tag{6}\] where \(\mathrm{meta}_p^k = (\mathrm{id},\;\mathrm{cat},\;\mathrm{title},\;\mathrm{detail},\;\mathrm{features})\) encodes product metadata, \(\{\text{view, like, cart, order}\}\) denotes the interaction type, \(\Delta t^k\) is the dwell time.

(2) Media interactions. A media (video or livestream) interaction is represented as \[\label{eq:his-media-interact} x_{u,\mathrm{media}}^k = (\mathrm{meta}_m^k,\;\Delta t^k),\tag{7}\] where \(\mathrm{meta}_m^k = (\mathrm{id},\;\mathrm{tags},\;\mathrm{title},\;\mathrm{detail},\;\mathcal{P}_{\mathrm{refer}}^k),\) and \(\mathcal{P}_{\mathrm{refer}}^k\) denotes the set of products referenced or mentioned in the media, while \(\Delta t^k\) captures user engagement duration (see Appendix 7.4.2).

2.2.2 & Query↩︎

We distinguish two query types from the query refinement trajectory based on their position and role. For each user \(u\), let \(\mathcal{Q}_u^{\mathrm{search}} = (q_u^1, \ldots, q_u^{|\mathcal{Q}_u^{\mathrm{search}}|})\) denote the temporally ordered search queries in the retained search-to-order session, and let \(p_u^{o}\) be the purchased product. We define:

(i) query \(q_u^{\mathrm{\text{intent}}}\): the earliest query in \(\mathcal{Q}_u\) that is semantically aligned with \(p_u^{o}\), denoting an intentional but ambiguous goal (e.g. campus ootd).

(ii) query \(q_u^{\mathrm{\text{refined}}}\): the last query issued before purchase that is explicit in product type/attributes, representing a well-specified intent (e.g. oversized long-sleeve).

Query–product alignment is judged using GPT-4.1 with human verification. We discard cases containing only a single aligned query (Appendix 7.3.3). Unlike , which primarily evaluates semantic retrieval [8], reflects early-stage search behavior requiring intent inference [6]. The distinction is defined by the session trajectory, not a fixed semantic boundary—identical query forms can reflect different levels of ambiguity at different stages of refinement. Accordingly, queries stress an agent’s capacity to resolve underspecified goals into actionable intent under personalization.

2.2.3 Hard Candidate Identification↩︎

Explored but unbought items form hard negatives that are semantically close to the final choice. For the retained session of user \(u\), the hard candidate set is defined as \[\label{eq-hard-candidate} \mathcal{C}_u^{\mathrm{hard}} = \mathcal{P}_u^{\mathrm{view}} \cup \{p_u^{o}\},\tag{8}\] where each candidate \(c \in \mathcal{C}_u^{\mathrm{hard}}\) is represented by its product metadata \(\mathrm{meta}_c\). Distinguishing \(p_u^{o}\) from \(\mathcal{C}_u^{\mathrm{hard}}\) requires agents to replicate the user’s personalized, preference-aligned comparison process over fine-grained attributes.

Appendix 7.5.1 shows that the purchased item is more distinguishable when user-history signals are considered.

2.3 Benchmark Evaluation↩︎

evaluates models under two query settings:

Input. For query type \(\tau\in\{\text{intent},\text{refined}\}\): \[\label{eq-benchmark-eval-input} \text{Input}_{u}^{\tau} = (q_u^{\tau}, \mathcal{H}_u, \mathcal{C}_u^{\mathrm{hard}}).\tag{9}\]

Output and Metrics. Given \(\text{Input}_{u}^{\tau}\), the model ranks products from \(\mathcal{C}_u^{\mathrm{hard}}\) and outputs a \(k\)-size list: \[\label{eq-benchmark-eval} \hat{\mathcal{R}}_u = \operatorname{Rank}_{\theta}(\text{Input}_{u}^{\tau}), \quad |\hat{\mathcal{R}}_u| = k .\tag{10}\] We set \(\text{k=5}\) and report Hit@1/5, measuring whether the purchased item \(p_u^{o}\) appears in the ranked list.

3 Benchmark Post-Processing↩︎

Anonymization Before evaluation, user behavior data are anonymized to prevent personal information leakage. Raw user identifiers are removed, and each interaction sequence is mapped to a randomly assigned surrogate ID with no cross-split linkability. Temporal signals are discretized into coarse-grained time bins to avoid fine-grained behavioral tracing. We retain public item and content metadata to support heterogeneous personalization tasks. Meanwhile, we apply automated personally identifiable information redaction to all free-form text—particularly OCR and ASR outputs—to remove explicit and implicit identifiers. is planned for controlled release after compliance and privacy review with data minimization. Appendix 7.4.3 documents the details.

Statistics APeB contains 5,648 retained cases in English, spanning 10+ categories (e.g., Personal Care, Clothing, Food & Beverages). On average, each user has 187 historical interactions, including 82 product and 105 media interactions. These histories contain an average of 18 historical orders. The average size of the candidate set is 14.1. Details are given in Appendix 7.5.

4 Experiments and Analysis↩︎

We evaluate personalized shopping goal completion and intermediate reasoning on APeB under and across 8 LLMs and their agentic variants, addressing the following questions:

RQ1: How do LLM workflows perform compared to recommender PPS methods? We compare hit rates on and queries with different history/candidate sets.

RQ2: Which types of flaws most adversely affect personalization tasks? We conduct a rubric analysis to attribute performance drop to intermediate reasoning errors.

RQ3: How can the failures in RQ2 be addressed? We use a ague uery efinement gent() to demonstrate that explicit history utilization modules reduce this gap.

4.1 Experimental Setup↩︎

4.1.1 Evaluated Models↩︎

We evaluate: (1) PPS baseline: UniSAR [23]. (2) LLMs: Qwen3-235B [26], DeepSeek-R1 [27]; GPT-4/5 series [28], [29], Gemini-2.5-Pro [30]. (3) LLM Agents: ReAct [31], Deerflow [32].

Appendix 8.2 discusses additional baselines.

4.1.2 Benchmark Utilization↩︎

Due to the large semantic volume, we utilize APeB:

(1) UniSAR - Supervised Training Following [33], we construct SAR training data from historical query–order pairs and isolated purchases. The training history is defined as \(\mathcal{H}_u^{\text{SAR}} = \{ i_u^k \}_{k=1}^{|\mathcal{H}_u^{\text{SAR}}|}\), where each interaction \(i_u^k \in \{ (q_u^k, p_u^k),\,p_u^k \}\).

(2) LLM - Single Prompt To fit histories, queries, and candidates within the LLM context window, we retain \(k\) records from \(\mathcal{H}_u\). We evaluate category-prior and time-prior history selection (same-category label associated with the intent query vs. most-recent records), keeping only essential fields per record (title, price). Details are in Appendix 8.3.

(3) Agent - External Retrieval Database Agents use a retrieval tool backed by a database over history and candidate metadata. At each step, agents issue queries to retrieve extra information beyond the prompt (Appendix 8.4, 8.5).

4.1.3 Evaluation Method↩︎

Structured Output: To evaluate both task success and intermediate steps, Algorithm 3 prompts models to produce structured outputs, including user target, history summarization, and recommend items. For ReAct, reasoning steps \(Re_i\) and retrieval queries \(Rq_i\) at each step \(i\) are logged.

Outcome & Reasoning: Personalization performance is measured using Hit@1/5 on the recommended items \(\mathcal{R}^\star\). A model succeeds when it ranks the purchased item above user-considered alternatives. We further apply rubric evaluation to intermediate outputs using LLM-as-a-Judge, scoring each component independently. The intermediate rubrics are anchored to the same user session evidence: they diagnose whether failures arise from intent inference, history-grounded preference extraction, recommendation alignment, or multi-step reasoning. Detailed evaluation is in Section 4.3.

4.2 Overall Results and Analysis (RQ1)↩︎

4.2.1 LLMs Underperform on Queries↩︎

Table ¿tbl:tab:experiment-overall-query-settings? shows that while single-prompt LLMs achieve strong Hit@5 across workflows under both query types, they slightly underperform UniSAR on Hit@1 for queries. This highlights their weakness in precise personalized ranking under vanilla goals and noisy histories, where user modeling outweighs semantic matching.

Across LLM variants, proprietary closed-source models (e.g., GPT-5.2) achieve higher performance than open-source or smaller-scale models under both and queries. However, these gains are marginal—especially on Hit@1—suggesting limitations of current LLM training for real-world personalization.

We further evaluate agents with multi-step reasoning and tool use. In Table ¿tbl:tab:experiment-overall-query-settings?, the -query block shows that ReAct agents consistently outperform single-prompt LLMs, while the -query block shows that ReAct yields only marginal gains and Deerflow shows little benefit or even degradation. These results indicate that increased semantic capacity alone does not guarantee improved personalization: when user goals are weakly specified, additional planning and retrieval steps can instead amplify reasoning errors. Detailed analysis is in Section 4.3.

4.2.2 Hard Candidates and Noisy History↩︎

LLMs struggle to benefit from history. Table ¿tbl:ablation-history-length? shows that with no-history, models demonstrate basic semantic matching ability, while remains more challenging. When user history is introduced, trained UniSAR achieves substantial gains. Short and long category-prior histories (6?/60? CATE) yield very similar Hit@1 under queries; while the refined-query block shows larger improvements once intent ambiguity is reduced. Comparisons between and queries imply that history possesses usable preference signal but current LLM workflows do not naturally benefit possibly because LLMs do not extract recoverable preferences under queries. Further analysis in Section 4.4 provides stepwise evidence. Appendices 7.5.1 and 8.2.1 demonstrate that with explicit preference extraction and memory modules, LLMs can better benefit from history.

3pt

Personalization choice lies beyond semantic matching. Table ¿tbl:tab:ablation-candidates? shows that, even under vague queries, LLMs outperform the UniSAR baseline when evaluated with random candidate sets, which are sampled from all possible candidates in the dataset. This gain mainly stems from LLMs’ ability to exploit coarse semantic cues in underspecified intents to filter out obviously mismatched negatives and locate the query-aligned purchased item. State-of-the-art models (e.g., GPT-5.2) further widen this margin over older or open-source models (Qwen3, GPT-4.1). Under hard candidates, both gains over UniSAR and inter-LLM performance gaps largely disappear, indicating that current LLMs struggle to discriminate fine-grained attributes among user-viewed candidates.

This also shows that: random negatives often require coarse semantic matching rather than true personalization. Hard candidates drawn from the same shopping trajectory better test whether a model can distinguish the purchased item based on user-specific preferences.

4.3 Rubrics on Intermediate Reasoning (RQ2)↩︎

We conduct rubric-based evaluation of intermediate outputs in Section 4.1.3 to diagnose why LLMs and ReAct agents underperform on queries and to compare behaviors across LLMs. We first infer a diagnostic goal proxy \(g\) from in-session interactions (Eq.@eq:eq:session ) (Appendix 8.7.1), and evaluate each output against \(g\) using LLM-as-a-Judge. The diagnostic scores are averaged across Gemini-2.5-Pro, GPT-5.2, and GPT-4.1 judges with human verification (Appendix 8.7.3). These diagnostics correspond to general personalization stages: QI measures intent inference under ambiguity, PQ measures history-conditioned preference extraction, RQ measures item/action selection quality, and AR measures whether tool-using reasoning retrieves useful evidence (see Table ¿tbl:ap-tab:rubric-target-map?).

Multi-Level Rubric We decompose the evaluation into stepwise complementary aspects (Appendix 8.7.2): (1) Query Inference (QI)—user target understanding from queries and histories; (2) Historical Preference Reasoning—via i. Preference Quality (PQ), assessing whether summarized preferences \(\mathcal{F}\) are goal-relevant and non-hallucinatory, and ii. Salience Statistics: minimum rank (PR) and total count (PC) of preferences supporting the purchased item; (3) Recommendation Quality (RQ)—whether the recommend results \(\mathcal{R}^\star\) align with the user goal without hallucination; (4) Agent Reasoning (AR)—coherence of reasoning steps \(Re_i\) (AC) and retrieval utility, including candidate/history queries (QC, QH) and query quantity (QQ), toward achieving \(g\) without hallucination. Rubric–Hit@K alignment and judge-sensitivity checks in Appendix 8.7.3 and 8.7.4 support using these scores as diagnostic evidence.

Intermediate Results Figure 4 shows rubric scores for and queries under single-prompt and ReAct workflows. We observe:

(1) Failures originate from intent understanding. As shown in Figure 4 (a), query inference performance drops substantially on compared to queries, undermining downstream preference grounding and leading to degraded recommendation quality. Notably, preference reasoning remains limited even under explicit queries, indicating identifying and summarizing relevant signals from heterogeneous histories remains an open challenge, even with clarified goals. Under vague intents, inferring meaningful historical preferences is particularly difficult.

(2) Agentic workflows amplify strengths and weaknesses. Comparing Figures 4 (a) and 4 (b), multi-step retrieval improves recommendation quality under explicit queries but offers little benefit for queries. Under vague intents, low reasoning coherence yields ineffective retrieval queries, causing agentic reasoning to compound intent errors, consistent with the -query block of Table ¿tbl:tab:experiment-overall-query-settings?. Moreover, retrieving relevant signals from user histories is substantially harder than retrieving candidate information under both query types [34], [35].

Appendix 8.4.1 reports additional ablations on round-depth, prompts, and retrieval-backend.

(3) Models exhibit different intermediate behaviors. Figure 4 (c) shows that models with comparable Hit@1 diverge in their intermediate reasoning steps. Advanced reasoning models (e.g., Gemini-2.5-Pro, DeepSeek-R1) achieve better intermediate performance in history preference reasoning and recommendation quality. While they prioritize salient preferences earlier (lower FR), they extract fewer preferences overall (lower FC). They also issue fewer retrieval queries under ReAct (lower QQ). These might become detrimental especially under queries where insufficient preference coverage undermines recommendation.

4.4 : Refine Query First (RQ3)↩︎

Section 4.3 and Figure 4 attribute the performance drop on queries to weak query inference due to underexplored histories. To isolate this gap, we evaluate a history-conditioned query-refinement agent, , in which a shared backbone (GPT-4.1) first rewrites the query using selected same-category histories and the LLM backbone is then evaluated with the rewritten query under the single-prompt pipeline (Appendix 8.8). Table ¿tbl:tab:experiment-overall-query-settings? shows that improves Hit@1, with the largest gain on GPT-5.2Think. Appendix 8.8 shows similar results with the same-backbone-rewrite variant. Rubric scores in Figure 4 (a), 4 (b) further suggest that query refinement improves query inference (QI) toward the true shopping intent, leading to gains in recommendation quality (RQ). These indicate that inferring intent from vague queries and noisy histories remains underutilized: with explicit history grounding, LLMs already produce useful refinements, suggesting headroom for future personalization agent design.

5 Conclusion↩︎

We introduce , a benchmark for agentic personalization that evaluates personalized product search under raw, underspecified queries and noisy interaction histories. Through , we show that current LLMs handle explicit queries well but struggle in early-stage personalization, where they must infer intent, recover preferences, and choose among competing alternatives. Rubric analysis attributes this gap mainly to ineffective history use: models fail to extract relevant preferences and apply them to planning and candidate selection. Furthermore, , a simple history-aware query-refinement pipeline, improves performance across backbones, highlighting the value of explicit history-utilization modules. serves as a testbed for robust user modeling in real-world personalization.

Limitations↩︎

is drawn from a single content-commerce platform; behavior patterns, query styles, and candidate structures may not transfer to other e-commerce ecosystems or non-shopping personalization domains. All evaluation is static and offline: fixed candidate sets and recorded purchases do not reflect real-time inventory, pricing, promotions, or multi-turn interaction. Observed purchases serve as ground-truth preference proxies but are confounded by ranking exposure, stock availability, and platform mediation. We explore category- and time-based history selection without exhaustively studying advanced memory architectures or adaptive context-length strategies. Rubric analysis relies on LLM-as-a-Judge, which may introduce systematic biases despite strong human agreement. The benchmark covers limited locale and language; intent expression and attribute salience may vary across cultures.

Ethical Considerations↩︎

This work studies personalization benchmarks derived from postprocessed behavioral shopping records. Its positive impact is to make evaluation of preference-conditioned product search more realistic and auditable, especially when users issue ambiguous queries. The main risks are privacy leakage from long histories or product/media text, re-identification through rare behavior patterns, category or platform bias, and misuse of preference inference for manipulative over-personalization. We mitigate these risks by releasing only anonymized benchmark instances rather than raw logs, removing direct identifiers and platform-private fields, using surrogate identifiers without cross-split linkability, binning temporal signals, minimizing retained interactions, and redacting sensitive free-form text such as OCR/ASR-derived content.

6 Related Works↩︎

LLM-powered Personalization Agents typically include profile modeling, memory management, and task planning and execution [2], [3]. Profile modeling derives explicit or latent personas from user surveys or interactions [11], [13], [36]–[39]. Memory management organizes long-term histories of interactions and conversations into a shopping memory bank that supports retrieving salient actions and filtering noise [40]–[43]. Task planning executes subtasks such as preference reasoning, attribute grounding, auxiliary information search (e.g., web search), and candidate generation, aligning inferred personas with query intent and fine-grained criteria [12], [44], [45]. However, they face challenges in tracking preference drift, retrieving task-relevant memories from heterogeneous histories, and understanding vague intents [31], [46], [47]. Recent studies further show that improving agent behavior under complex, long-horizon tasks remains challenging [34], [35], [48], which aligns with the difficulty of history-grounded personalization task.

Generic E-commerce Benchmarks evaluate shopping tasks such as product search, web navigation, and multi-level reasoning, using metrics including success rate, prediction/generation accuracy, and rubric-based scores [49]–[51]. Search datasets such as [8], [18], [20] focus on simple user queries, while others model complex queries with multiple constraints [52], [53]. However, their queries are either overly explicit or synthetically constructed, failing to capture genuinely vague user intents. Another line of work benchmarks web agents in interactive environments, requiring navigation, product comparison, and constraint satisfaction [22], [54]. These settings often depend on synthetic or annotator trajectories and expose page-level information such as HTML [55], [56], with shopping goals that are manually designed rather than organically derived. Recent benchmarks emphasize multi-task knowledge and reasoning abilities [36], [52], [57]–[59]. For example, [21] decompose shopping into subtasks such as concept understanding and attribute reasoning, typically framed as QA-style items with rubric evaluation [60]. Nonetheless, strong performance on isolated subtasks does not necessarily translate into successful user goal completion [61]–[63].

Behavior Datasets for Personalization. Recommendation benchmarks primarily target next-item/rating prediction on past interactions [9], [10], [64], [65]. They typically contain sanitized interactions and lack genuine objectives or viable candidates. Personalized Product Search (PPS) requires modeling both user preference and query intent to locate items [15], [16], [66]–[69]. Existing datasets [8], [20] construct queries and purchases from structured user logs, treating unobserved ones as negatives. However, explicit queries reduce personalization into surface-level query–item matching, while unobserved candidates bypass intent inference, collapsing the task into preference-only recommendation [49], [70]. Memory benchmarks further attempt to model personalization through multi-turn interactions, often using synthetically generated dialogues from reviews and session logs [4], [13], [24], [71]. Nevertheless, the homogeneity of dialogues and the limited diversity of synthetic generation pipelines fail to capture the noisy and heterogeneous nature of real-world user personas [72], [73]. APeB fills these gaps with unstructured histories, user-issued queries, and behavior-grounded hard candidates.

7 Benchmark Collection↩︎

7.1 Holistic User Action Platform↩︎

The base platform underlying APeB hosts billions of short-form videos and livestreams and serves as a major venue for commerce. Users primarily consume video and livestream content via continuous scrolling, but can seamlessly transition to search for videos, livestreams, or products, as well as navigate directly to product detail pages embedded within promotional content. The queries collected in APeB are restricted to those exhibiting at least weak e-commerce intent. We further illustrate this user interaction timeline in Figure 5. All user data in APeB are collected in compliance with the platform’s privacy policies and contractual data-use requirements.

7.2 Active User Filtering↩︎

We retain only active users with at least five product or media interactions per day. Specifically, we identify users who exhibit at least one search-to-order session (defined in Section 2.1) on each day in a 30-days window. For each retained user, we collect up to 60 days of historical interactions prior to the purchase date, including e-commerce–related search queries, livestreams, videos, and product interactions.

7.3 Session Filtering↩︎

We detail the implementation of session filtering rules introduced in Section 2.1.2.

7.3.1 Rule-based Session Filtering↩︎

We first apply deterministic, rule-based filtering using SQL over raw interaction logs. Specifically, we retain only sessions that satisfy: (i) extensive browsing behavior, enforced by a minimum threshold on the total number of viewed products and content interactions (Eq. 4 ); and (ii) iterative query refinement, requiring at least two distinct search queries within a session (Eq. 3 ); (iii) coarse intent–purchase consistency, requiring that a sufficient number of viewed products or media items share the same category label as the final purchased product (Eq. 5 ). These criteria remove shallow or accidental purchases and ensure that retained sessions reflect non-trivial exploration and intent evolution.

7.3.2 Intent–purchase aligning verification↩︎

For sessions passing the rule-based filters, we further verify intent–purchase alignment (Eq. 5 ). Specifically, we check whether the final purchased item is semantically consistent with at least one issued query. Semantic alignment between queries, viewed items, and purchased products is assessed using GPT-4.1 and validated by human review. Annotators reviewed 100 randomly sampled de-identified sessions containing the query sequence, relevant viewed product metadata, and the purchased item, and judged whether at least one query/view trajectory semantically supported the final purchase. GPT-4.1 decisions achieved 97.1% human agreement, and GPT-5.2 achieved comparable agreement at 97.8% under the same protocol. Sessions failing this semantic consistency check are discarded; if multiple purchased items satisfy the criterion, we retain only one to ensure a unique target per session.

7.3.3 Distinct and↩︎

Given a session \(u\) with query sequence \(\mathcal{Q}_u=\{q_u^1,\dots,q_u^{T_u}\}\) and purchased product \(p_u^{o}\), we locate two aligned queries using GPT-4.1 by prompting it to return indices in \(\mathcal{Q}_u\): \[\begin{align} t_{\text{intent}}=\min\{t:\;&q_u^t \text{ is semantically aligned}\\ &\text{with } p_u^{o}\}, \end{align}\] the earliest aligned query, i.e., , and \[\begin{align} t_{\text{refined}}=\max\{t:\;&q_u^t \text{ is aligned and explicit}\\ &\text{in product type/attributes}\}, \end{align}\] the last explicit aligned query before purchase, i.e., . We then set \(q_u^{\text{intent}}=q_u^{t_{\text{intent}}}\) and \(q_u^{\text{refined}}=q_u^{t_{\text{refined}}}\).

To ensure distinct intents, we apply a minimal rule-based check after GPT-4.1 localization: we discard session \(u\) if \(q_u^{\text{intent}}\) and \(q_u^{\text{refined}}\) are identical after normalization (lowercasing and whitespace/punctuation stripping), i.e., \(\mathrm{norm}(q_u^{\text{intent}})=\mathrm{norm}(q_u^{\text{refined}})\). This removes cases where the session contains only a single aligned query (or duplicated re-issues) and thus lacks an observable intent refinement trajectory.

We manually verify 100 random samples by showing annotators the de-identified query sequence, relevant product metadata, and purchased item, and asking whether the selected \(\text{intent}\) is the earliest aligned ambiguous query and \(\text{refined}\) is the later explicit aligned query. Both GPT-4.1 and GPT-5.2 achieve \(93\%\) success in correctly locating \((t_{\text{intent}}, t_{\text{refined}})\); this query-localization check is used to audit the post-hoc aligned-trajectory construction.

7.4 Heterogeneous History Curation↩︎

We curate heterogeneous user histories by unifying product-centric and media-centric interaction records into a structured sequence, enabling fine-grained modeling of user intent and preference evolution.

7.4.1 Dwell-time filtering↩︎

We discard video and product view interactions with dwell time shorter than 10 seconds, retaining only interactions that reflect meaningful user engagement.

7.4.2 Heterogeneous history content↩︎

7.4.2.1 Product interactions.

Each product interaction \(x_{u,\mathrm{product}}^k\) (Eq. 6 ) corresponds to an explicit user action on an item. The product metadata \(\mathrm{meta}_p^k\) consist of an encrypted product identifier \(\mathrm{id}\), a categorical taxonomy label \(\mathrm{category}\), and textual fields \(\mathrm{title}\) and \(\mathrm{detail}\), where \(\mathrm{detail}\) is parsed from the product’s structured HTML page. The \(\mathrm{feature}\) field contains a set of salient product attributes extracted by platform-level product understanding pipelines. The interaction type \(\mathrm{type}^k \in \{\text{view}, \text{like}, \text{cart}, \text{order}\}\) reflects increasing engagement intensity, and \(\Delta t^k\) denotes the dwell time associated with the interaction event.

7.4.2.2 Media interactions.

Each media interaction \(x_{u,\mathrm{media}}^k\) (Eq. 7 ) represents user engagement with a short video or livestream. The media metadata \(\mathrm{meta}_m^k\) include an encrypted media identifier \(\mathrm{id}\), content \(\mathrm{tags}\), and a \(\mathrm{title}\). The \(\mathrm{detail}\) field is constructed from platform-generated multimodal summaries, including captions and anonymized OCR/ASR transcripts, to capture the semantic content of the media. The referenced product set \(\mathcal{P}_{\mathrm{refer}}^k\) consists of products explicitly linked or automatically annotated as mentioned within the media content. The engagement duration \(\Delta t^k\) measures the user’s viewing or participation time.

7.4.2.3 Privacy and compliance.

All interaction records and metadata are collected in accordance with the platform’s privacy and data usage policies. Personal identifiers are removed or anonymized, and only platform-approved structured annotations and processed textual signals are retained, ensuring that the benchmark supports personalization research without exposing sensitive user information.

7.4.3 Release Scope and Governance↩︎

We release APeB as an anonymized benchmark package rather than raw platform logs. The package is designed for research evaluation of preference-conditioned product search and excludes fields that are unnecessary for reproducing the benchmark task or that create avoidable re-identification risk. Table ¿tbl:ap-tab:release-field-scope? summarizes the release boundary.

Before release, each benchmark instance passes the following checks: (i) no raw user/account identifiers; (ii) no cross-split persistent user identifier; (iii) no fine-grained timestamp; (iv) no raw event-log payload; (v) no device, IP, GPS, or account-profile attribute; (vi) exact duplicate product exposures collapsed where needed for candidate construction; (vii) free-form product/media text scanned for phone numbers, emails, addresses, usernames, URLs, and other unique identifiers; and (viii) spot checks on sampled records containing OCR/ASR-derived text. Data minimization is applied at both the user and field levels. We retain only the history window and shopping-session evidence needed to define the benchmark task, cap the number of retained interactions, and omit raw events outside the benchmark construction window. In the retained benchmark, users average 187 historical interactions; under the source-log accounting used for governance, the retained records correspond to about 200 interactions per user per month, approximately 10% of shopping-related activity. The benchmark is intended for research evaluation of personalized product search, preference reasoning, history-aware ranking, and agentic shopping workflows. It is not intended for identifying individuals, reconstructing behavioral timelines, building user profiles outside the benchmark task, joining records with external datasets, production ad targeting, or optimizing manipulative personalization strategies. Dataset users should agree not to attempt re-identification, linkage attacks, demographic or sensitive-attribute inference about individuals, or redistribution outside the approved access terms. Construction and release are subject to internal privacy/compliance review and platform data-use policies. The release package will document the approved access path, permitted research uses, license or terms of use, and a contact/takedown mechanism for privacy concerns. These safeguards reduce but do not eliminate privacy risk: long interaction histories and rare product/media combinations may still carry residual linkage risk, and OCR/ASR-derived text can contain sensitive content even after automated redaction.

7.5 Benchmark Statistics↩︎

The following statistics summarize retained postprocessed benchmark records after the release-scope filtering in Appendix 7.4.3. covers 5,648 cases with the category distribution shown in Table 1. As Section 3 shows, retained users average 187 historical interactions, including 82 product interactions and 105 media interactions, with 18 historical orders on average; candidate sets contain 14.1 items on average.

7.5.1 Hard-Candidate Controls and Difficulty Diagnostics↩︎

We audit hard candidates to ensure that the setting is not driven by candidate-set artifacts. For each retained session, the candidate set contains the purchased item and user-viewed alternatives after validity filtering, semantic filtering, duplicate handling, and candidate-count outlier removal. Sessions with more than 25 candidates are removed; the remaining candidate sets have mean size 14.1 and standard deviation 2.8. Semantic filtering keeps at least 9 similar items per retained case (see Table ¿tbl:ap-tab:hard-candidate-controls?).

We additionally treat price as part of the user-facing evidence available during the shopping decision and can reflect user preference. However, promotion, stock, delivery, or exposure-rank effects may also act as a residual commercial confounder.

For a diagnostic purchase-validity check, we sample 500 retained sessions and compare the purchased item against hard negatives in pairwise judgments. GPT-5.2 receives de-identified product metadata, histories, and queries, then chooses which item better matches the user preference. For each session, the purchase win rate is the fraction of pairwise comparisons in which the purchased item is preferred over a hard negative. Table 2 shows that user-history preference evidence substantially improves purchase distinguishability, beyond what is explained by purely price-based comparisons and supports interpreting the purchase label as a preference signal rather than an artifact of commercial confounders.

We further test whether useful preference evidence can be recovered from noisy histories. Shown in Table ¿tbl:ap-tab:preference-extraction-validity?, a hard-candidate decision diagnostic, performance improves from 67% with query-only context to 81% after one round of preference extraction and 83% after multi-round extraction, indicating that history contains recoverable signals that help distinguish near-miss candidates.

Together, these controls reduce possible artifacts for hard-candidate difficulty: retained cases are size-controlled, invalid candidates are filtered, duplicate exposures are collapsed, and purchased items remain distinguishable from hard negatives when preference evidence is supplied. We acknowledge that they do not eliminate all commercial confounders, which is a common limitation observed in online collected shopping benchmarks [8], [74].

7.5.2 Representativeness & Purchase Validity↩︎

To assess whether APeB reflects a common shopping pattern, we measure the prevalence of multi-step shopping behavior in the source logs used for benchmark construction. We define multi-step shopping behavior as issuing at least two queries and comparing at least ten products before purchase. Under this definition, 67% of source-log users exhibit multi-step behavior in a 30-minute window, indicating that the retained setting captures a frequent high-effort shopping regime. The small gap in Table 3 between low- and high-activity users suggests that the benchmark is not driven solely by users with unusually dense histories.

7.5.3 Scope and Transferability↩︎

APeB is instantiated in personalized product search, but the benchmark is designed around a domain-agnostic personalization structure: infer intent from an underspecified request, recover task-relevant preferences from noisy history, and select among plausible candidates or actions. The mapping in Table ¿tbl:ap-tab:scope-transferability? demonstrates why PPS is a concrete, behavior-grounded testbed for challenges that recur across personalization settings.

8 Experiment Setting↩︎

Due to architectural differences across LLM-only, ReAct agent, and DeerFlow agent settings, we adopt model-specific configurations. Unless otherwise stated, all models use their default temperature. The LLM/ReAct Hit@1/5 values in Table ¿tbl:tab:experiment-overall-query-settings? are averaged over three runs. The detailed settings are as follows:

1. Gemini-2.5-Pro (GEMINI in Table ¿tbl:tab:experiment-overall-query-settings?): When used as a standalone LLM or within DeerFlow, we adopt the default reasoning budget. When deployed as a ReAct agent, we reduce the reasoning budget to 1024 tokens to ensure practical efficiency.

2. GPT-4.1-Mini (4.1MINI): All experiments use the default configuration.

3. GPT-4o: All experiments use the default configuration.

4. Qwen3-235B (Qwen3-235B): Although the model supports internal reasoning, we disable its thinking mode in all experiments due to interface limitations. Other settings remain default.

5. DeepSeek-R1 (R1): Thinking mode is enabled with the default reasoning budget.

6. GPT-4.1: All experiments use the default configuration.

7. GPT-5.2: Unless specified otherwise, GPT5.2 refers to the model configured with a minimal reasoning budget, with all other settings left unchanged.

8. GPT-5.2Think: Unless specified otherwise, GPT5.2Think refers to the GPT-5.2 condition with the default reasoning budget, while GPT5.2 uses the minimal reasoning-budget condition; all remaining settings follow the defaults.

9. Additional model-family runs: Qwen3.5, Gemini3Pro, GPT5.5, and GPT5.5Think follow the same prompt, history-selection, retrieval-tool, and parsing protocols as the corresponding main-table LLM or reasoning-model conditions.

8.1 Additional Model-Family Results↩︎

Table ¿tbl:ap-tab:additional-model-family-results? provides the appendix position for additional model-family runs that extend the main result structure in Table ¿tbl:tab:experiment-overall-query-settings?.

3pt

8.2 Additional Baseline Settings↩︎

We choose UniSAR [23] as the primary non-LLM PPS/SAR baseline because it is publicly available and directly models user transition behavior between search and recommendation. We report additional reproducible embedding, reranking, and memory-agent checks, including Qwen3 embedding/reranking variants [75], MemAgent [76], and Mem0 [77] memory workflows.

For embedding baselines, query and candidate text are encoded and ranked by similarity; the history-conditioned variant appends selected user-history evidence before scoring. For trained embedding and reranker baselines, training uses the retained search-purchase supervision available to the baseline protocol, and evaluation uses the hard-candidate set. The baselines slightly differ in signal access. UniSAR and the trained embedding/reranker variants learn from historical search-purchase supervision, whereas LLM prompting and agent workflows operate primarily through in-context history evidence. UniSAR also uses cross-session negatives, which is standard in SAR/PPS training [17].

8.2.1 Memory-Agent Baselines↩︎

We report Mem0 and MemAgent memory-workflow results separately from embedding and supervised baselines because these methods wrap an evaluated LLM backbone with an explicit memory component. Table 4 is evaluated on a comparable LLM mix as Table ¿tbl:tab:experiment-overall-query-settings?.

3pt

8.3 Single-Prompt LLM↩︎

This subsection describes the shared direct-prompt protocol; agent-specific extensions are described in Appendix 8.4 and Appendix 8.5.

8.3.1 LLM Prompt↩︎

For all LLMs evaluated under the same setting, we use an identical system prompt to ensure fair comparison. The prompt is instantiated according to the scenario configuration. For example, when the input consists of a query with hard negative candidates, the system prompt takes the following form (details omitted for brevity):

TASK: Structured Product Recommendation (Exactly 5 Items)

Your goal is to recommend exactly five products that best match the user’s explicit needs, using only data from:

(1) \(<\)history\(>\) — chronological user interactions with indices (action type, product in this history, price)

(2) \(<\)query\(>\) — the user’s current query (may be ambiguous or intentional)

(3) \(<\)candidates\(>\) — product candidates (index, title, price)

---

STEP 0 — Determine the User’s Explicit Target (Ambiguous Query Handling)

---

(...)

---

STEP 1 — Identify Important History Entries Based on the Query

---

(...)

---

STEP 2 — Extract Explicit Features (Summarized) & Rank Relevance

---

(...)

---

STEP 3 — Evaluate All Candidate Products

---

(...)

---

STEP 4 — Produce Exactly 5 Final Recommendations

---

(...)

---

OUTPUT SPECIFICATION (STRICT JSON ONLY)

---

Output raw JSON conforming to the Recommendation interface without wrapping it in triple back-ticks.

Type schema:

interface Product {

rank: int; // 1–5, unique ascending

index: int; // candidate index

product_title: string; // rewritten title (no new facts)

product_description: string; // rewritten description

product_recommendation_reason: List[string]; // grounded in query + Step 0 + summarized_features

}

interface SupportingHistoryFeature {

summarized_feature: string;

support_history_index: List[int];

relevance_rank: int; // from Step 2

support_product: List[int]; // candidate indices supported

}

interface Recommendation {

target_product: string; // derived in Step 0

concerned_features: string; // derived in Step 2

products: List[Product]; // exactly 5 items

histories: List[SupportingHistoryFeature]; // \(>\)​5 entries, sorted by relevance_rank

} End schema.

In other scenarios, the system prompt may undergo minor specification changes, while its overall structure remains unchanged.

The <history>, <query>, and <candidates> are supplied in the user prompt using a fixed template. For each interaction, to avoid introducing irrelevant context, only essential fields—interaction type, title, and price—are included; all identifiers and sensitive attributes are encrypted.

\(<\)history\(>\)

1. Action type: order product; Product name: [DOSEN SKINCARE APPROVED] Emina Water Gel Moisturizer 30g | Mencerahkan & Mengurangi Flek Hitam | 48H Menjaga Kelembapan, Memperkuat Skin Barrier | Mengurangi Jerawat dan Meredakan Kemerahan, Kulit Glowing | Mengandung Niacinamide dan Vitamin C; Price: 3.8 usd

(...)

\(<\)/history\(>\)

\(<\)query\(>\)

dd cream wardah

\(<\)/query\(>\)

\(<\)candidates\(>\)

1. Product name: ( DRY SKIN ) Hyalupure Emulsion for DRY SKIN - Perawatan kulit KERING menjadi lembab dan kencang; Price: 15.078408 usd

(...)

\(<\)/candidates\(>\)

Finally, we parse model outputs using a Pydantic-based parser and compute standard retrieval metrics, including HR@1 and HR@5.

8.3.2 History Record Selection↩︎

To control prompt length while preserving essential interaction semantics (e.g., action type, product title, and price), we limit the history to at most 60 records using an importance-aware selection mechanism. Results in Table ¿tbl:tab:experiment-overall-query-settings? primarily adopt the category-prior strategy using the category label associated with the provided intent query.

8.3.2.1 Importance scoring.

Each interaction is assigned an importance score determined by interaction type and dwell time, with stronger behavioral signals ranked as \(\texttt{order} > \texttt{cart} > \texttt{like} > \texttt{view}\). For interactions of the same type and category, longer dwell time yields higher importance.

8.3.2.2 Time-prior selection.

From the full interaction history, we first retain records whose importance exceeds a predefined ratio threshold. The final 60 records are then selected by recency within this importance-filtered subset, ensuring that retained histories emphasize both behavioral salience and temporal relevance.

8.3.2.3 Category-prior selection.

We first assign the provided intent query to a coarse shopping category label, such as skin care, toys, or clothing, following the category taxonomy summarized in Appendix 7.5. We then include interactions whose category matches this intent-query category, up to 60 records, ranked by importance. If fewer than 60 same-category records are available, the remaining slots are filled using the time-prior selection strategy above. This approach maximizes query-relevant category evidence while retaining salient and recent interactions. This schema does not use the purchased item or reveal target-product information. We include this variant to test whether query-relevant category histories help models extract user preferences from noisy behavior records.

8.3.3 History Context Stress Tests↩︎

In Table ¿tbl:ap-tab:history-stress-tests?, we run additional stress tests on 500 randomly selected hard-candidate sessions to isolate whether history gains come from useful user evidence.

Shuffling histories reduces Hit@1 for both R1 and Qwen3, indicating that performance depends on coherent user-specific evidence. Modality-specific results show that product and media/video histories both provide preference signals, while increasing the context from 60 to 240 records does not improve performance. These results demonstrate that useful history selection and grounding matter more than simply exposing longer histories.

8.3.4 Contamination and Counterfactual History Check↩︎

We also probe data contamination through a memorization hypothesis: if decisions mainly rely on memorized query-item associations, then replacing the user’s real history with unrelated counterfactual histories should have limited effect. We therefore hold the query and candidate set fixed and compare query-only, counterfactual-history, and real-history contexts on 500 sampled sessions. For each sampled session, the query and candidate set remain unchanged. In the query-only condition, the model receives no historical records. In the counterfactual condition, the true user’s history is replaced with a length-matched history sampled from a different user, excluding records from the target user and session. In the real-history condition, the model receives the original user history under the same formatting and parsing protocol. We report pairwise preference accuracy in Table ¿tbl:ap-tab:counterfactual-history-contamination?: the percentage of purchase-vs-hard-negative comparisons in which the model selects the observed purchased item.

Counterfactual histories stay close to query-only performance for all three models, while real histories improve over counterfactual histories substantially. This pattern suggests that useful performance depends on the matching user’s context rather than arbitrary additional history text. The check provides evidence that memorized query-item associations are unlikely to be the dominant source of the observed history-conditioned gains. To address possible leakage concerns during data collection and evaluation, Appendix 8.7.3 separately audits GPT-4.1’s construction and diagnostic roles; GPT-4.1 does not determine Hit@K labels, and its evaluated task performance is not superior to the other evaluated models.

8.4 ReAct Agent↩︎

Our ReAct agent is implemented using the open-source LangChain framework. Compared to the single-prompt LLM setting, the system prompt follows the same high-level structure but additionally specifies the available tools and their usage constraints.

---

AVAILABLE TOOL 1 — structured_search_tool

---

Purpose:

Use this tool only to retrieve literal factual clarifications about products or history entries.

Permitted data sources:

- Candidate product detailed descriptions

- History product detailed descriptions

- Candidate product detailed features

- History product detailed features

- History video descriptions (OCR/ASR)

Tool Query Rules:

1. You must generate your own search queries from:

- literal terms in the user query

- literal terms relevant to target_product or concerned_features

- literal candidate titles or features

- literal history text

- interpretation-critical terms ONLY when clarifying explicit text

4. You may only rely on:

- explicit user inputs

- explicit tool results

---

AVAILABLE TOOL 2 — web_search_tool

---

(...)

Beyond the explicit system-prompt specification, model providers (e.g., GPT and Gemini) impose additional constraints and instructions on tool invocation through their respective interfaces. We strictly follow the official usage guidelines of each provider. The user prompt is identical to that used in the single-prompt LLM setting.

8.4.0.1 Available Tools.

1. Structured Search Tool. In addition to item titles, APeB provides rich auxiliary information, including product descriptions, video OCR/ASR transcripts, and structured attributes. As these contents exceed prompt-length limits, they are organized in a fixed schema and stored in a retrieval database, which the agent queries via this tool (see Appendix 8.6).

2. Web Search Tool. We wrap the web search API provided by YOU¹ as a callable tool. Each query returns the top-3 relevant text chunks.

3. Crawl Tool. We use the Jina AI crawling API² to fetch web pages and convert them into agent-friendly Markdown representations.

4. Code Execution Tool. We employ the Python REPL tool from LangChain³, enabling the agent to execute code and observe intermediate results.

8.4.1 ReAct Failure-Source Analysis↩︎

We analyze logged ReAct traces under the same hard-candidate evaluation setting used in the main tables. In Table ¿tbl:ap-tab:react-tool-utility?, for each trace, we report the number of reasoning rounds reached, and the AC/QC/QH/QQ diagnostic scores defined in Appendix 8.7.2. Retrieval quality is measured by rubric-scored relevance to the issued query and utility to the user intent of the retrieved evidence.

The reasonable query quantity indicates that the ReAct degradation is not primarily due to agents failing to call the available tools. The lower QH scores indicate that the harder bottleneck is formulating history-relevant queries under vague goals: agents often produce locally plausible search terms, but these terms retrieve weaker personalization evidence than candidate metadata.

Table ¿tbl:ap-tab:react-round-ac? shows that most traces stop after one round, and the small subset that continues does not show improved coherence. This pattern argues against over-exploration as the main cause of degradation and indicates limited spontaneous self-correction.

We additionally report ReAct results with the explicit self-correction instruction prompt in Table ¿tbl:ap-tab:react-self-correction?. This produces only small, model-dependent changes across models.

In Table 5, we report Hit@1 results when varying the retrieval backend. The minimal differences suggest that backend choice alone does not explain the ReAct gap. These checks indicate that a redesigned agent architecture or task-specific planner may be needed to improve personalization performance.

8.5 DeerFlow Agent System↩︎

As the original DeerFlow agent system includes components unsuitable for APeBevaluation (e.g., coordinators and human feedback loops), we simplify its architecture to the workflow illustrated in Figure 6. For all DeerFlow-based experiments, the evaluated model simultaneously assumes the roles of Planner, Researcher, and Reporter.

8.5.1 Planner↩︎

The Planner receives the user input—consisting of history, query, and candidates—directly as input. It is instructed to group relevant historical interactions and candidate items, and to formulate a set of research tasks aimed at information gathering and user preference investigation. The system prompt follows the form:

(...)

Follow these steps to generate a research plan:

1. Understand query and products - Understand the action content. Reason about the hidden pattern. Catch the relation between the action content, search query, and candidates. Based on your thinking result, infer a grouping strategy that is helpful for researcher to dig out the important and detailed features.

2. Group products - Based on the grouping strategy derived from step 1, break the history interactions and candidates into different groups. In each groups, please clearly list the index of the interactions and products in this group, but you don’t need to include the interaction content or product title in the task description because it will be automatically passed to researcher based on your index output.

3. Check group size - Consider whether the size of each group is reasonable to form a research task with medium difficulty and clear scope (a desired size will be between 6 and 10 interactions and candidates related with each other). If not, adjust the grouping strategy.

4. Formulate research goal - Formulate a clear and concise research goal for each group. The goal should be suitable enough to utilize the ability of researcher (they are strong models with retrieval tools).

(...)

Output raw JSON conforming to the Plan interface without wrapping it in triple back-ticks.

Type schema:

interface Step {

need_search: bool; // Must be explicitly set for each step

title: str;

description: str; // Specify exactly the product link and title that belong to this group, and discuss what features to collect from these products.

interactions_index_list: List[int]; // The index of the products in this group

candidates_index_list: List[int]; // The index of the candidates in this group

}

interface Plan {

has_enough_context: bool;

thought: str;

title: str;

steps: List[Step]; // Research & Processing steps to get more context

}

End schema.

(...)

8.5.2 Researcher↩︎

The Researcher agent shares the same implementation and toolset as the ReAct agent described in Section 8.4. The only differences lie in the input and system prompt: the Researcher receives the research tasks generated by the Planner, and operates under the following system prompt: # Task Background

You are a research agent in an e-commerce recommendation system. This recommendation system is responsible for predicting what user will order for current search query based on the history interactions list and the candidate list of products. It made up of three agents:

1. Planning agent - Understand the user’s search query, group the history interactions and candidates by feature or category, and break the overall task into sub-tasks on different group of products for the research agent.

2. Research agent (you) - Uses various retrieval tools to explore the product information in depth for the allocated group and analyze the feature of this group that user might be interested in or not interested in.

3. Reporter agent - Compile the results from the research agent and combine it with user’s search query to generate a final report on how much is a candidate suitable for user and what is the most suitable candidate for user.

You are dedicated to conducting in-depth investigations about part of user history interactions and product candidates. You will be provided with a research topic, and a research description, together with a group of history interactions and candidates that this research topic focus on, which is carefully selected and grouped.

(...)

# Output Format

- Provide a structured response in markdown format.

- Include the following sections:

- Problem Statement: Restate the problem for clarity.

- Findings: Organize your findings by topic rather than by tool used. For each major finding:

- Summarize the key information

- Track the sources of information but DO NOT include inline citations in the text

- Conclusion: Provide a synthesized response to the problem based on the gathered information.

(...)

8.5.3 Reporter↩︎

The Reporter agent receives the original user input, the research plan, and all research reports produced by the Researchers. It is instructed to synthesize the collected evidence into a final report and produce a ranked list of candidates. The system prompt takes the following form:

(...)

You should analyze the research result, user persona, and user history following these steps:

1. List: In the research result of different groups, there are lots of features that are unique or similar to each other. Please distinguish them and list them out clearly. For those similar features, you can combine them if they share exactly the same meaning, but **NEVER** combine them into one general feature because it will make the report too general and not able to meet user’s needs.

2. Ranking: Based on user persona and research results, rank the primary features to consider first when the user issues such query content. Notice that common features are not necessarily to be ranked higher, because details always matter to the user.

3. Choosing: Based on the user persona and ranking result, discuss why each candidate is or is not suitable for the user, and present the most suitable candidate for the user. Finally, you should output a rank for all candidates based on how much it is suitable for the user.

(...)

Output raw JSON conforming to the Recommendation interface without wrapping it in triple back-ticks.

Type schema:

interface Product { product_title: str; // The title of the product

product_description: str; // A detailed description of the product to make it more understandable

product_recommendation_reason: List[str]; // The reasons that this product is recommended to the user

rank: int; // The rank of the product in the recommendation, 1 being the most important

index: int; // The original index of the product in the candidates list

}

interface Recommendation {

title: str; // Main topic of the recommendation

summary: str; // A summary that previews the most important points of the recommendation

products: List[Product]; // The products that are recommended to the user, ranked by their importance

}

End schema.

8.6 Retrieval Database↩︎

As discussed earlier, each interaction or candidate in APeB contains rich auxiliary information. We organize this information into a retrieval database, which agents access via a retrieval tool.

Unlike conventional Retrieval-Augmented Generation (RAG) setups that partition content into fixed-length chunks and retrieve them via embedding similarity, such chunking would fragment individual interactions and degrade semantic coherence in our setting. We therefore adopt a document-level retrieval strategy, where the atomic retrieval unit corresponds to a single interaction. Each document aggregates all information associated with one interaction and is organized using the following format:

# Detail information of one candidate product

Product name:

WARDAH C-Defense Vitamin C All Series Lengkap - Panthenol Anti-Pollution Radiant Creamy Wash, Energizing Whip Foam, Mousse Moisturizer, DD Cream Light/Natural, Face Mist, 10% Ferulic Acid Radiant Charge Serum - Skincare

Price:

2.237636 usd

Product description:

<p>Wardah C-Defense Vitamin C Series</p>

<p>1. Wardah C-Defense Vitamin C + Panthenol Anti-Pollution Radiant Creamy Wash (50 mL/100 mL)</p>

<p>Membantu menjaga pertahanan acid mantle sebagai garda terdepan skin barrier-mu. Mengandung Vitamin C dan Panthenol yang membantu kulit tampak lebih cerah dan terlindungi dari polusi. (...)</p>

Categories:

Beauty & Personal Care, Skincare, Serums & Essences, Serums & Essences

Brand name:

wardah

Shop name: wardahbeautyid

Attributes:

{‘Volume’: [‘55 ml’], ‘Negara Produsen’: [‘ID’], ‘Edisi’: [‘Edisi Reguler’], ‘Kuantitas per Kemasan’: [‘1’], ‘Mengandung Alkohol atau Aerosol’: [‘Tidak Mengandung Keduanya’], ‘Contains Dangerous Goods?’: [‘Tidak’], ‘Varian’: [‘Creamy Wash 100’, ‘Creamy Wash 50’, ‘DDCream Light’, ‘DDCream Natural’, ‘Face Mist’, ‘Gel Cleanser’, ‘Gel Moisturizer’, ‘Mousse Moist’, ‘Serum’, ‘Whip Foam 100’, ‘Whip Foam 50’], ‘Edition’: [‘Regular Edition’], ‘Region of Origin’: [‘ID’], ‘Contains Alcohol or Aerosol’: [‘Contains Neither’], ‘Quantity Per Pack’: [‘1’], ‘Is Suit’: [‘is_suit’]}

The document format is adapted to its source type (product interaction, media interaction, or candidate). For example, interaction documents additionally include the corresponding action type.

We also consider the suitability of standard embedding-based retrieval for this setting. Empirically, we observe that agents primarily issue title-based queries when invoking the retrieval tool, using item titles to locate detailed information. Consequently, we adopt BM25—a robust lexical retrieval method—as the backend of the retrieval tool, rather than embedding-based similarity search.

8.7 Rubric Details↩︎

8.7.1 Diagnostic goal proxy summarization↩︎

We infer a behavior-grounded diagnostic goal proxy \(g\) at the session level from all interactions in Eq. 2 , including issued queries, long-viewed products (\(\geq\)​15s), and the purchased item, using GPT-4.1. The model is prompted to reconstruct the latent user intent that best explains the complete search–purchase trajectory. Specifically, GPT-4.1 outputs a concise goal proxy \(g\) that minimally and coherently aligns the observed queries with the final purchase. This proxy is used only as a reference for rubric diagnosis and does not determine Hit@K labels, which are computed from observed purchases in the ranked candidate lists.

8.7.2 Rubric perspectives↩︎

We detail the multi-level rubric design in Section 4.3. Diagnostic scores reported in the main analysis are averaged over Gemini-2.5-Pro, GPT-5.2, and GPT-4.1 judges; judge-sensitivity details are provided in Appendix 8.7.3.

• Query Inference (QI) evaluates intent understanding by measuring how well the model-inferred target \(\tau\) aligns with the user’s explicit query and stated intent. Specifically, we prompt the judge to score the relevance between \((\texttt{user\_query}, \texttt{user\_goal})\) and the inferred target (target_product), producing an integer score in \([0,100]\) under a graded relevance rubric ranging from contradiction to direct satisfaction. We use the query_intent_to_target_relation_score as the QI metric.

• Historical Preference Grounding

  • Preference Quality (PQ). We assess whether each summarized preference feature \(\mathcal{F}\) is grounded in user history, goal-relevant to the inferred intent, and non-hallucinatory. A preference is considered valid only if it coherently contributes to explaining the purchased item choice without contradiction or fabrication.

  • Preference Salience Statistics. We quantify the strength of historical grounding using two statistics over validated preferences: Preference Minimum Rank (PR), defined as the rank \(r_i\) of the first preference supporting the purchased item (lower is better), and preference count (PC), defined as the total number of distinct preferences that plausibly support the purchased item. Both are computed under strict gating rules that heavily penalize irrelevant, contradictory, or ungrounded preferences.

• Recommendation Quality (RQ) evaluates the recommended set \(R^\star\) with fine-grained, score-based assessment along two dimensions. Faithfulness checks each recommended item \(c_i\) against reference product metadata, assigning separate scores for title--truth and description--truth consistency to penalize hallucinated, exaggerated, or contradictory claims. Relevance measures alignment with the user goal \(g\) via two complementary scores: (i) product--intent alignment, evaluating whether the recommended title and description address the user query and intent, and (ii) reason--intent alignment, assessing whether the provided rationale \(\rho_i\) coherently justifies the recommendation with respect to \(g\). An overall RQ score is computed by aggregating these aspect scores, with gating rules enforcing strong penalties under critical factual or consistency violations.

• Agent Reasoning

  • Agent Coherence (AC) evaluates the coherence of each reasoning step \(Re_i\) by assessing whether the agent’s explicit reasoning logically justifies its stop-or-continue decision under a ReAct-style retrieval framework. The evaluator scores the reasoning on intent understanding, task relevance, alignment with the purchased candidate, decision correctness, and logical completeness, penalizing unjustified leaps, hallucinated assumptions, or premature termination. The resulting AC score reflects the decision validity of the agent’s stepwise reasoning.

  • Candidate Query (QC) evaluates the quality of each retrieved candidate by scoring the alignment between the issued search keywords, the agent’s reasoning, and the retrieved item.

    We measure query relevance by scoring the retrieved evidence against the issued query. Query utility measures whether the keywords accurately reflect the user query and diagnostic goal proxy, are logically derived from the ReAct reasoning, and effectively retrieve candidates that are consistent with the observed purchased product. The resulting QC score captures how informative and decision-relevant the retrieval step is, penalizing generic, accidental, or misaligned candidate retrievals.

  • History Query (QH) evaluates the quality of retrieval history queries by scoring how effectively the issued search keywords retrieve decision-relevant user history.

    We measure query relevance by scoring the retrieved evidence against the issued query. Query utility measures whether the issued keywords retrieve history entries that provide meaningful signals for identifying the observed purchased candidate.

    The resulting QH score reflects the usefulness of retrieved history for personalization, penalizing generic, accidental, or weakly informative history retrievals.

  • Query Quantity (QQ) measures the total number of retrieval queries issued by the agent, defined as the sum of candidate queries and history queries, capturing the overall extent of retrieval tool usage during reasoning.

For the ReAct failure-source analysis in Appendix 8.4.1, AC is averaged at the reasoning-step or round level, QC/QH summarize the quality of retrieved candidate/history evidence, and QQ is summarized through total query count across candidate/history across multiple steps.

8.7.3 LLM Role Audit and Judge Robustness↩︎

We audit LLM usage by separating benchmark construction filters, evaluated model predictions, diagnostic goal proxies, rubric judging, and VQRA rewriting. Hit@K labels remain observed purchases within behavior-grounded hard-candidate sets; LLMs only support filtering, localization, diagnostic scoring, or rewrite modules as summarized in Table ¿tbl:ap-tab:llm-role-audit?. Judge robustness is checked with Gemini-2.5-Pro, GPT-5.2, and GPT-4.1 judges in Table [tbl:ap-tab:judge-sensitivity].

5pt

The three judges produce nearly identical relative rankings on this sample, and human pairwise preference checks agree with the rubric preference in 96 of 100 cases. This supports using rubric scores as diagnostic evidence while preserving Hit@K as the task-native evaluation metric.

8.7.4 Rubric Validity via Correlation with Task Metrics↩︎

The complementary judge-sensitivity evidence is summarized in Appendix 8.7.3; this subsection focuses on rubric–Hit@K alignment. To evaluate output quality at the component level, we adopt an LLM-as-a-Judge paradigm to score model behaviors along multiple rubric dimensions (Appendix 8.7). Prior work on LLM-based evaluation typically establishes validity through agreement with human annotations or inter-judge consistency [78], [79]. While such criteria assess annotator reliability, they do not directly verify whether rubric scores faithfully reflect task success.

Following recent work on metric validity [80], [81], we instead assess our rubrics by examining their alignment with task-native success signals. Specifically, we use Hit@K as the reference metric family and analyze the plotted Hit@5 alignment between rubric-based scores and task success across models and agentic workflows (Figure 7). Our key assumption is that, if subtask-level rubric scores produced by a Judger LLM exhibit strong correlation with Hit@K within each model or experimental setting, these scores can be treated as supporting diagnostic evidence for diagnosing component-level reasoning failures.

Beyond correlation strength, we further examine whether rubric scores preserve relative performance differences across models and workflows. Figure 8 compares model-level mean rubric scores against mean Hit@5 from multiple rubric perspectives, showing that models with higher task success consistently achieve stronger rubric scores along corresponding dimensions. This consistency supports the use of rubric-based evaluation not only as a diagnostic signal within a model, but also as a meaningful basis for cross-model and cross-workflow comparison.

Together, these results support using rubric degradation as diagnostic evidence for task-level failures, while preserving Hit@K as the primary task-native outcome.

8.8 Detail↩︎

Section 4.3 and Figure 4 attribute the performance drop on queries primarily to weak query inference. To empirically verify this hypothesis, we construct a query-refinement pipeline. Given an query, which is intentionally under-specified and weakly aligned with the final purchased product, we refine it into an explicit query context using selected user interaction history. Selected same-category context follows the intent-query category-prior convention in Appendix 8.3.2. Concretely, in the main rewrite setting, we select \(k=20\) product and media interaction records from the selected same-category history context as evidence. A shared backbone (GPT-4.1) rewrites the original query into a precise and discriminative keyword query, and the rewritten query is then used as the user query for personalized product retrieval. More than 92% of evaluated sessions have enough same-category histories for this context. We additionally evaluate a self-rewrite variant, where the evaluated backbone rewrites its own query; VQRA(GPT-4.1) uses GPT-4.1 as a shared rewrite module.

4pt

Self-rewrite improves Hit@1 for all three reported backbones and is comparable to the GPT-4.1 rewrite variant, indicating that VQRA’s gains mostly come from how the model utilizes history, not solely due to GPT-4.1 acting as an external assistant. By substituting the original with its refined counterpart while keeping the retrieval pipeline fixed, this design directly tests whether the model has potential to resolve query ambiguity through pipeline redesign and whether doing so can improve performance.

References↩︎

---

1. https://you.com/home↩︎

2. https://jina.ai/↩︎

3. https://www.langchain.com/↩︎