1 Introduction↩︎

Personalized assistants are increasingly expected to account for a user’s long-term interests, recent activities, and implicit preferences [1]–[3]. Existing benchmarks, however, mainly test whether models remember preferences explicitly stated in dialogue, emphasizing memory rather than insight. In practice, preferences are often revealed indirectly through what users create, share, photograph, discuss, and repeatedly engage with [4], [5]. Social-media timelines offer a rich source of such signals, but recovering them requires aggregating multimodal evidence over time, distinguishing stable hobbies from recent fixations, and applying the inferred profile in personalized interaction. Figure 1 illustrates how timeline evidence can be transformed into stable and recent interests for dialogue.

Prior benchmarks mostly represent user context as dialogue-derived stated preferences [6]–[9]. Although recent work incorporates longer behavioral histories [5], it still relies on synthetic or structured textual logs. These settings bypass a key challenge for MLLMs: inferring user interests from noisy, unstructured, longitudinal social-media traces, where evidence is weak, distributed across posts, and often available only through images, timestamps, or cross-post patterns.

We introduce SocialPersona , a benchmark for evaluating MLLM personalization from multimodal social-media context. Built from real timelines of everyday, non-promotional users, SocialPersona contains chronologically organized text, images, and timestamps. From these timelines, we construct human-validated interest profiles across seven domains: sports and outdoor activities, entertainment, gaming, food and drink, travel and city exploration, photography and creation, and pets. Each profile separates stable interests from recent interests and grounds them in supporting evidence.

SocialPersona supports two evaluation settings. In profile construction, models infer active domains and fine-grained interest tags from raw multimodal timelines. In personalized dialogue generation, models receive social-media context with a current request and are evaluated on whether their responses align with the user’s stable or recent interests. Together, these tasks test whether MLLMs can both recover implicit preferences and use them in downstream interaction.

Concretely, SocialPersona contains timelines from 171 real users, with an average of 176.81 posts and 130.38 images per user. A semi-automated pipeline followed by human verification yields 2,597 preference tags grounded in textual, visual, and temporal evidence. Experiments with proprietary and open-weight MLLMs show that current models still struggle to infer fine-grained and recent interests, and to consistently use inferred profiles in personalized responses.

Our contributions are three-fold:

1. We introduce a new task formulation that challenges MLLMs to infer user preferences from longitudinal, multimodal social-media behavior—aggregating sparse textual, visual, and temporal signals across long horizons—and to apply the inferred preferences in personalized dialogue generation.

2. We construct SocialPersona , a real-user benchmark with long-horizon timelines, multimodal evidence, timestamps, and human-validated profiles across seven domains, and publicly release benchmark code with a de-identified evaluation subset².

3. We evaluate proprietary and open-weight MLLMs on profile construction and personalized dialogue generation, revealing gaps in cross-modal evidence aggregation and user-aligned response generation.

2 Related Work↩︎

4pt

2.1 Personalization Benchmarks↩︎

Recent personalization benchmarks mainly construct user context from dialogue histories or structured behavior logs. LaMP [6] evaluates personalized language tasks from user-specific textual histories, while PrefEval [7], PersonaMem [8], and PersonaLens [9] study preference recognition, user memory, and personalized response generation from conversational context. More recent benchmarks move toward longer-term behavioral modeling, including Mem-PAL [5] for behavioral-log-grounded dialogue and ALPBench [11] / GISTBench [12] for e-commerce or short-video interest inference. Agent-oriented benchmarks further extend personalization to search, web, and mobile environments [13]–[17].

As summarized in Table 1, SocialPersona differs from prior benchmarks by combining multimodal input, real-user data, revealed-preference signals, profile evaluation, and dialogue evaluation in one setting.

2.2 Multimodal Social-media Understanding↩︎

Prior multimodal social-media datasets study content-level tasks such as sentiment and affect analysis [18]–[20], sarcasm and humor detection [21], crisis response [22], misinformation verification [23]–[27], harmful-content recognition [28], [29], and broad MLLM evaluation on social-networking scenarios [30]–[32]. However, these benchmarks primarily label individual posts or interactions for predefined tasks. User-related signals, when included, are usually treated as demographic attributes, engagement prediction, or recommendation targets. SocialPersona instead treats a user’s timeline as external personalization context: models must aggregate sparse textual, visual, and temporal evidence across many posts, distinguish stable from recent interests, and generate responses aligned with the inferred profile.

3 SocialPersona Construction↩︎

3.1 Problem Setting↩︎

We study whether MLLMs can infer and use preferences from social-media timelines. For each user \(u\), a temporally ordered timeline \(\mathcal{S}_u = \langle p_1, \dots, p_n \rangle\) consists of posts \(p_i = (x_i, v_i, \tau_i)\) with text \(x_i\), visuals \(v_i\), and timestamp \(\tau_i\), spanning at most 200 posts over two years.

We define profiles over seven interest domains adapted from prior preference taxonomies [7] and platform-level interest categories: \(\{\)sports_outdoor, entertainment, gaming, food_drink, travel_city_exploration, photography_creation, pets\(\}\). For each active domain, the gold profile contains stable interests (recurring patterns across the timeline), recent interests (emerging or time-local signals near the end of the observation window), and supporting evidence links retained for auditability. We exclude demographic, identity-related, health, political, and other sensitive attributes.

SocialPersona supports two tasks. In profile construction, a model predicts stable and recent interest tags from the user timeline. In personalized dialogue generation, a model receives the timeline together with a natural user request and generates a response aligned with the user’s stable or recent interests. The overall construction and evaluation pipeline is shown in Figure 2: SocialPersona first converts raw social-media timelines into human-verified stable and recent interest profiles, and then evaluates whether MLLMs can recover these profiles and use them in personalized dialogue.

3.2 User and Timeline Collection↩︎

We construct SocialPersona from real social-media timelines of long-tail organic users, rather than celebrities, brand accounts, or highly curated public profiles. This design choice is intended to capture relatively natural, self-expressive preference traces instead of broadcast-oriented content. Starting from 8,000 candidate accounts, we apply automatic filters based on follower count, follower–followee ratio, and image trace density, retaining accounts with 5–5,000 followers, FFR in \([0.5,2]\), and ITDR \(\geq 0.3\). These filters remove extremely sparse, highly public, or insufficiently multimodal accounts while reducing the presence of broadcaster-style users [33], [34]. We then manually inspect the remaining accounts to exclude commercial, repost-heavy, or otherwise low-quality cases, resulting in 250 candidate users. Detailed definitions of FFR, ITDR, and the manual filtering criteria are provided in Appendix 7.

For each selected user, we collect up to 200 posts from the most recent two-year window. Each post is stored as a structured multimodal record containing its timestamp, textual content, hashtags, URLs, and attached visual content, including images or video cover frames. As the original posts come from users across multiple countries and languages, we standardize all textual content by translating it into English, thereby enabling consistent profile construction and evaluation. After profile construction and verification, we further remove users with fewer than three active interest domains, as such profiles provide insufficient personalization signals for reliable evaluation. This yields the final benchmark of 171 users.

3.3 Gold Profile Construction↩︎

Given each user’s multimodal timeline, we construct gold profiles with an LLM-assisted but human-verified pipeline. First, we use Gemini-3-Flash [35], [36] to perform conservative post-level extraction from the original text and visual content. The extractor is instructed to identify only observable, evidence-grounded interest signals, record modality attribution, and avoid demographic, identity-related, or speculative claims.

Second, we aggregate post-level candidates across each timeline. Near-duplicate posts are down-weighted, semantically equivalent tags are merged into canonical interests, and each user-domain pair is represented as an evidence pack. Third, we compute preliminary stable and recent assignments using duplicate-adjusted support, temporal dispersion, recency, and confidence. Stable interests require repeated support across the timeline, while recent interests emphasize evidence concentrated in the most recent 90 days.

Fourth, we apply an LLM-based calibration stage using Gemini-3.1-Pro [37], [38]. This stage checks the aggregated evidence packs, canonical candidates, scores, and preliminary temporal buckets for weak support, over-generalization, speculative labels, and bucket errors. Finally, trained annotators manually verify each calibrated profile against its supporting posts. Accepted interests must be concrete, domain-appropriate, sufficiently supported, and assigned to the correct temporal bucket.

3.3.0.1 Quality assurance.

Five trained annotators independently verified each user profile, with disagreements resolved by an additional adjudicator. The process required approximately 350 annotator-hours. On a 40-user overlap subset, inter-annotator agreement reached Krippendorff’s \(\alpha=0.72\) for tag acceptance and \(\alpha=0.63\) for stable/recent bucket assignment. Human verification modified about 12% of pipeline-proposed tags: 8% removed for insufficient evidence, 3% refined to more concrete labels, and 1% added as missed but supported interests, confirming that human oversight is essential for benchmark quality.

4 Evaluation and Experiments↩︎

4.1 Evaluation Setup↩︎

All experiments use a fixed 100-user subset to ensure comparable cost and conditions. We evaluate profile construction and personalized dialogue generation using a timeline representation consisting of post text, image captions, and timestamps.

4.1.0.1 Evaluated models.

Our main model suite is designed to cover both proprietary and open-weight MLLMs. The initial suite contains six models: Gemini-2.5-Flash [39], GPT-4o-mini [40], GPT-5.4 [41], Qwen2.5-VL-7B-Instruct [42], Qwen3-VL-8B-Instruct [43], and Qwen3.5-35B-A3B [44].

4.1.0.2 Profile construction evaluation.

Given \(\widetilde{\mathcal{S}}_u^{M}\), each method predicts stable and recent interests over seven domains. A domain is active if it has at least one gold interest, and predicted active if the method outputs any interest in it. For interest-tag recovery, we evaluate each user, domain, and bucket \(b \in \{\mathrm{stable}, \mathrm{recent}\}\) using normalized exact match, followed by o3 [45]-based semantic matching for unmatched tags as in Appendix 6.3. Matches are one-to-one and define true positives; unmatched predictions and gold tags are false positives and false negatives.

4.1.0.3 Personalized dialogue evaluation.

The dialogue task evaluates whether a model can generate personalized but not over-personalized recommendations from social-media-derived user information. We evaluate four dialogue input settings. The first is a timeline-conditioned setting, where the model receives the user’s post text, image captions, and timestamps directly. The remaining three are two-stage profile-conditioned settings: the model first constructs a profile using one of the three profile-construction settings described below, and then generates a dialogue response using only that generated profile as personalization context. These settings test whether a model-generated profile can serve as an effective intermediate memory representation, rather than only being evaluated as a structured prediction.

For each dialogue setting, we consider two intents: stable-interest recommendation, which targets the user’s long-term preferences, and recent-interest exploration, which introduces mildly novel yet personally suitable items. For each setting, we prepare ten natural requests and randomly sample one for each user–setting instance; the full pools are given in Appendix 6.5. We use GPT-5.5 [46] and Qwen3.7-Max [47] as judges, following the prompt in Appendix 6.6. Given the gold profile, the judge scores each response on interest coverage, concreteness, and fluency.

4.2 Main Results: Profile Construction Across Settings and Models↩︎

We evaluate profile construction across three settings and six models (Table 2), separating model capability from the strategy used to handle long timelines.

4.2.0.1 Profile-construction settings.

Direct feeds the full timeline (text, image captions, timestamps, up to 200 posts) in one pass. Hierarchical splits the timeline into 20-post chunks, summarizes each independently, then aggregates chunk summaries into the final profile. Extractive–abstractive proceeds in two stages as detailed in Appendix 6.9. First (extraction), the LLM is prompted to select up to \(K=12\) representative posts per domain, guided by relevance, specificity, recurrence, recency, and multimodal grounding. Second (abstractive synthesis), the LLM generates the profile using only the selected posts as evidence, with no access to the original full timeline.

6pt

4.2.0.2 Over-generalization trumps recall.

The dominant failure mode across all models and settings is systematic over-generalization: models reduce 4–7 gold interest tags to 1–2 broad categories per active domain. Averaged over the direct setting, gold profiles contain 3.4 tags per active domain, while models predict only 1.5. This gap is consistent across all evaluated models and represents a fundamental limitation of current MLLMs in fine-grained preference elicitation from social-media evidence. A prompt analysis in Appendix 10 confirms that this over-generalization is not an artifact of the evaluation prompt: removing the conservative instruction to “prefer fewer, broader tags” does not materially change Interest F1, indicating that the recall gap reflects genuine model limitations rather than benchmark design.

4.2.0.3 Domain blindness follows evidence modality.

Domain detection varies with evidence modality. Text-dispersed interests, such as music habits, city exploration, and meals, are often missed: in Appendix 12, most models incorrectly mark text-heavy travel and food-drink domains as inactive. In contrast, visually concentrated domains, such as pets and gaming screenshots, obtain the highest F1 scores across settings (Table 13). This asymmetry indicates a reliance on visual salience for domain activation: models reliably activate domains when images directly show relevant content, but often default to inactive when evidence is diffuse and textual. Even GPT-5.4, despite the best overall tag F1, misses 12.5% of gold-active domains under direct profiling.

4.2.0.4 Recent-interest blindness is a temporal reasoning gap.

Stable F1 exceeds Recent F1 across every model, setting, and input mode. This is not simply because recent interests are “harder”—it reflects a fundamental limitation in how models process timelines. Distinguishing an interest that recurs across 18 months (stable) from one concentrated in the most recent 90 days (recent) requires tracking temporal dispersion across posts. Current models receive posts as a flattened sequence; timestamps are present in the input but models show no evidence of computing temporal distribution patterns from them. Gemini-2.5-Flash and Qwen3.5-35B-A3B assign nearly all predicted tags to the stable bucket regardless of the actual temporal evidence, while GPT-4o-mini—the only model with non-trivial Recent F1 (0.086)—distributes tags more evenly between buckets but without temporal alignment, suggesting its higher Recent F1 reflects a flatter prior over buckets rather than genuine temporal discrimination.

4.2.0.5 Hierarchical profiling amplifies model-specific behaviors.

Hierarchical profiling improves Interest F1 for strong models (GPT-5.4, GPT-4o-mini) but causes conservative models to collapse, as sparse chunk-level summaries leave the aggregation step with insufficient evidence. The same pattern recurs in the extractive setting: capable models benefit from the two-stage decomposition while weaker models degrade further. Detailed per-model breakdowns are provided in Appendix 12.

4.3 Main Results: Personalized Dialogue Generation↩︎

Table 3 reports dialogue results across four input settings and six models. The timeline-conditioned setting provides the raw timeline directly. The other three are profile-conditioned: the model first constructs a profile using one of the three settings from Table 2, then generates a response using only that profile. We report three judge-scored dimensions (0–5 scale) and their unweighted average \(\mathrm{Avg} = (\mathrm{Coverage} + \mathrm{Concreteness} + \mathrm{Fluency}) / 3\).

4.3.0.1 Profile conditioning is a double-edged filter.

Profile conditioning filters timeline noise but also propagates profiling errors. When generated profiles are sparse, personalization coverage and concreteness drop; when profiles are richer, the intermediate profile can improve dialogue quality. For example, Gemini-2.5-Flash loses coverage under profile conditioning, while GPT-5.4 slightly improves coverage, suggesting that profile quality determines whether the profile acts as a useful memory representation or an information bottleneck. The profile–dialogue correlation analysis (Section 4.4) confirms this dependency quantitatively, with 16 of 18 Interest F1–Avg.correlations positive and stronger under the tighter information bottleneck of hierarchical and extractive profiling. Fluency remains relatively stable across settings and models (3.0–4.3 for most), indicating that the core challenge is not generating readable text but producing responses that engage the correct interests.

4.3.0.2 Human–LLM agreement.

A 30-response human agreement study confirms that the three dialogue dimensions are reliably judgeable and that both LLM judges track human rankings. Inter-annotator agreement is substantial for fluency, concreteness, and coverage (\(\alpha=0.79/0.71/0.65\)), and GPT-5.5 correlates most strongly with human mean scores (\(\rho=0.76/0.65/0.58\), all \(p<0.01\)). Full sampling and annotation details are in Appendix 15.

4.4 Additional Analyses↩︎

4.4.0.1 Input-modality analysis.

Input modality analysis on Qwen3.5-35B-A3B and GPT-4o-mini shows that text provides the main profiling signal, while image captions supply complementary tacit cues: adding captions raises GPT-4o-mini’s Active F1 from 0.503 to 0.737. Timestamps further improve Active F1 but leave Tag F1 largely unchanged, suggesting that temporal structure helps domain detection more than fine-grained tag recovery.

4.4.0.2 Per-domain difficulty.

Per-domain breakdown (Table 13) confirms that domains with visually distinctive evidence (pets, gaming) are consistently easier across settings, while domains requiring synthesis of dispersed evidence (travel, entertainment) are hardest.

4.4.0.3 Profile–dialogue correlation.

The two-stage design enables a direct test of whether profile quality translates to dialogue quality. Per-user Spearman rank correlations show that profile Interest F1 positively correlates with dialogue quality: 16 of 18 Interest F1–Avg.correlations are positive (6 significant at \(p<0.05\)), with stronger correlations under Hierarchical and Extractive profiling, where the profile acts as a tighter information bottleneck. Interest F1 correlates most strongly with coverage (15 of 18 pairs significant), confirming that better profiling primarily improves interest engagement.

5 Conclusion↩︎

We introduced SocialPersona , a benchmark for personalized user profiling and response generation from multimodal social-media timelines, built from real social-media user data with human-validated interest profiles across seven domains. Our experiments reveal three central findings. First, current models systematically over-generalize, collapsing specific interests into broad categories—a genuine capability gap, not an artifact of conservative evaluation instructions (Appendix 10). Second, models exhibit pronounced modality asymmetry: visually salient domains are reliably detected while text-distributed domains are frequently missed, and this bias propagates into dialogue. Third, distinguishing stable from recent interests remains beyond current capabilities; models lack the cross-post temporal reasoning needed to separate persistent patterns from emerging ones. The profile–dialogue correlation validates the two-stage design but reveals that sparse profiles compound these failures downstream.

Limitations↩︎

Our evaluation uses image captions rather than raw images because full user timelines can contain hundreds of images, making raw-image evaluation difficult under heterogeneous API limits, visual-token budgets, and upload interfaces. Captions provide a unified cross-model input format but may lose pixel-level details. The modality analysis shows that captions still add substantial signal alongside text for the interest categories studied here.

Ethical Considerations↩︎

SocialPersona is designed as an aggregate benchmark for studying personalized modeling from publicly accessible social-media content under a restricted research protocol. Annotation and evaluation are limited to non-sensitive, evidence-grounded interests, and explicitly avoid demographic, identity-related, health, political, or other sensitive inferences. We release only a de-identified text-plus-caption subset, exclude original images, and provide the full benchmark only through controlled research access. The benchmark is intended solely for aggregate model evaluation and research purposes, and must not be used for identification, surveillance, targeting, or consequential decision-making.

6 Prompt Templates↩︎

6.1 Single-Post Analysis Prompt↩︎

6.2 Profile Evaluation Model Prompt↩︎

6.3 Profile Evaluation Matching Judge Prompt↩︎

6.4 Dialogue Generation Prompt↩︎

6.5 Dialogue User Request Pool↩︎

For dialogue evaluation, we sample one user request from the corresponding setting-specific pool. The stable-interest pool targets stable preference use, while the recent-interest pool targets recent exploration that remains compatible with the user’s stable preferences.

6.5.0.1 Stable-interest recommendation requests.

1. I want something that fits my usual taste. Could you recommend one option for me?

2. Could you suggest one thing I’d probably enjoy based on what I usually like?

3. I’m looking for a recommendation that feels very me. What’s one good option?

4. Choose one option that fits what I’ve liked for a while.

5. I want a safe choice that matches my usual preferences. What should I try?

6. Recommend one activity or item that seems close to my regular taste.

7. Based on what I tend to enjoy, what is one practical suggestion?

8. I’m not trying to branch out today; give me one recommendation that fits my normal style.

9. What’s one personalized option that would likely suit my everyday interests?

10. Give me one recommendation grounded in what I’ve consistently liked before.

6.5.0.2 Recent-interest exploration requests.

1. I want to try something a bit new, but still something that feels like me. Any suggestion?

2. Could you recommend one fresh option that connects to what I’ve been into lately?

3. I’m open to exploring something new. What’s one suggestion that still matches my taste?

4. Choose one option that builds on what has caught my attention recently, without feeling random.

5. I’d like a small change from my usual choices. What should I try?

6. Recommend one new-ish activity or item that fits what I seem to be into right now.

7. What’s one recommendation that reflects what I’ve been paying attention to lately?

8. I want something slightly outside my routine, but not totally unfamiliar. Any idea?

9. Suggest one option that feels current for me while still matching my usual taste.

10. Give me one practical recommendation that feels timely for me, not just my old favorites.

6.6 Dialogue Evaluation Judge Prompt↩︎

6.7 Caption Prompt↩︎

6.8 Hierarchical Profile Construction Prompts↩︎

6.9 Extractive Profile Construction Prompts↩︎

The extractive–abstractive method proceeds in two stages. First, the LLM receives the full user timeline and is prompted to select up to \(K\) representative posts per domain. The selection criteria include relevance, specificity (concrete interest signals rather than vague topics), recurrence (preferring posts consistent with repeated behavior), recency (capturing potential recent interests), and multimodal grounding (using image captions as evidence). Second, the LLM receives only the selected posts and synthesizes the final profile, without access to the original full timeline. The prompts for both stages are shown below.

6.10 Calibration and Gold Rewrite Prompts↩︎

7 User Filtering and Profile Verification Details↩︎

This appendix provides implementation details that are summarized in Section 3.2 and Section 3.3.

7.0.0.1 Automatic account filtering.

We use three account-level heuristics before manual inspection. First, we retain accounts with 5–5,000 followers, excluding extremely inactive accounts and highly public accounts. Second, we compute the follower–followee ratio (FFR) as \[\mathrm{FFR} = \frac{N_{\mathrm{follower}}}{N_{\mathrm{followee}} + \epsilon},\] where \(\epsilon\) avoids division by zero. We retain accounts with \(\mathrm{FFR}\in[0.5,2]\), which favors relatively reciprocal social neighborhoods and filters highly asymmetric broadcaster-style accounts. Third, we compute the image trace density ratio (ITDR) as \[\mathrm{ITDR} = \frac{N_{\mathrm{posts\;with\;images}}}{N_{\mathrm{total\;posts}}},\] and retain users with \(\mathrm{ITDR}\geq0.3\) to ensure sufficient visual evidence for multimodal profiling.

7.0.0.2 Manual account inspection.

After automatic filtering, annotators remove accounts that are commercial, celebrity-like, organization-operated, repost-heavy, dominated by low-information content, or lacking sufficient personal and preference-relevant signals. This stage yields 250 candidate users for timeline collection and profile construction.

7.0.0.3 Cross-post aggregation.

Post-level interest candidates are aggregated into canonical interests before temporal scoring. Near-duplicate posts are clustered and down-weighted so that repeated captions, repost-like content, or bursty discussions do not count as independent evidence. Semantically equivalent tags are normalized and merged into canonical interests. For each canonical interest, we retain supporting posts, modality attribution, duplicate-adjusted support, temporal distribution, and extraction confidence.

7.0.0.4 Human profile verification.

Human verification is conducted over calibrated domain-level profiles and their supporting evidence. Annotators remove unsupported interests, revise overly broad labels into more concrete tags, and add missing tags when the evidence clearly supports them. Accepted labels must be concrete, non-sensitive, domain-appropriate, and supported by the timeline. After this verification stage, users with fewer than three active domains are removed to ensure enough positive personalization signals for evaluation. Our annotators are trained to be conservative and evidence-grounded, avoiding over-interpretation or inference beyond what the timeline supports. This process yields the final set of 100 users for the benchmark.

8 Annotator Recruitment, Instructions, and Data Consent↩︎

This section supplements the profile verification (Section 3.3) and dialogue evaluation (Appendix 15) with details on annotator recruitment, compensation, instructions, and data consent.

8.0.0.1 Recruitment and payment.

All annotators were recruited from the undergraduate and graduate student population at the authors’ institution. Annotators were compensated at 50 CNY per hour. This rate exceeds the typical student hourly wage at the authors’ university and is consistent with compensation for comparable annotation tasks in the region.

8.0.0.2 Instructions to participants.

All annotators received written annotation guidelines before beginning work, covering task definitions, quality rubrics, and annotated examples. For profile verification, annotators were instructed to be conservative and evidence-grounded, accepting interests only when directly supported by timeline evidence (see the criteria in Appendix 7). For dialogue evaluation, annotators received the full 0–5 scoring rubrics shown in Appendix 6.6 and completed a calibration round before the agreement study. The complete instruction documents are available upon request.

8.0.0.3 Data consent.

The social-media posts used in this benchmark were collected from publicly accessible accounts. We exclude private or restricted-access content. The released benchmark subset contains only de-identified text and captions; original images are excluded to prevent re-identification, and the full dataset is provided only through controlled research access (see Ethical Considerations). No annotator personal information was collected or retained.

9 Profile Scoring Details↩︎

This appendix provides the detailed scoring rules used in the algorithmic temporal scoring stage of profile construction. These scores are used only to produce preliminary stable/recent assignments before LLM-based calibration and human verification.

9.0.0.1 Canonical interest evidence.

After post-level extraction and cross-post aggregation, each canonical interest \(c\) in domain \(d\) is associated with an evidence set \[\mathcal{E}_{u,d,c} = \{ e_1, e_2, \dots, e_m \},\] where each evidence item corresponds to a supporting post. Each evidence item records the post timestamp, evidence modality, duplicate-adjusted weight, and post-level extraction confidence. If a post belongs to a near-duplicate cluster \(C\), its support weight is discounted by \(1/|C|\), so that repeated or highly similar posts do not artificially inflate an interest.

Let \(w_j\) denote the duplicate-adjusted weight of evidence item \(e_j\), and let \(c_j \in [0,1]\) denote its extraction confidence. The effective support count of a canonical interest is defined as \[S_{\mathrm{eff}} = \sum_{e_j \in \mathcal{E}_{u,d,c}} w_j.\] The average confidence is computed as a weighted average: \[\bar{c} = \frac{\sum_{e_j \in \mathcal{E}_{u,d,c}} w_j c_j}{\sum_{e_j \in \mathcal{E}_{u,d,c}} w_j}.\]

9.0.0.2 Temporal bins.

To estimate whether an interest is persistent over time, we divide each user’s timeline into monthly bins. Let \(B_{\mathrm{distinct}}\) be the number of distinct monthly bins that contain at least one supporting evidence item for the canonical interest. Let \(B_{\mathrm{span}}\) be the total number of monthly bins covered by the user’s collected timeline. We set \[B_{\mathrm{req}} = \min(6, B_{\mathrm{span}}),\] so that long timelines require evidence spread across multiple periods, while shorter timelines are not penalized excessively.

9.0.0.3 Stable-interest score.

The stable score estimates whether a canonical interest reflects a stable preference. It combines three factors: effective support, temporal dispersion, and extraction confidence: \[\begin{align} \mathrm{score}_{\mathrm{stable}} &= 0.60 \min\!\left(1, \frac{S_{\mathrm{eff}}}{3}\right) \\ &\quad + 0.20 \bar{c} + 0.20 \min\!\left(1, \frac{B_{\mathrm{distinct}}}{m}\right), \end{align}\] where \(m = \max(2,B_{\mathrm{req}}-1)\). The first term rewards repeated support after duplicate discounting. The second term incorporates the confidence of post-level extraction. The third term rewards evidence distributed across multiple time periods. This score is designed to favor interests that appear repeatedly and persistently across the user’s timeline.

A canonical interest is marked as a preliminary stable interest if it satisfies \[\mathrm{score}_{\mathrm{stable}} \geq \theta_{\mathrm{stable}}\] and has sufficient temporal support: \[S_{\mathrm{eff}} \geq 3, \qquad B_{\mathrm{distinct}} \geq 2.\] In our implementation, we set \[\theta_{\mathrm{stable}} = 0.65.\]

9.0.0.4 Recent-interest score.

The recent score estimates whether a canonical interest reflects a recent or emerging preference. We focus on the most recent 90 days of the user’s timeline. Let \(\mathcal{E}_{u,d,c}^{90}\) denote the subset of evidence items whose timestamps fall within this window. We define \[S_{90} = \sum_{e_j \in \mathcal{E}_{u,d,c}^{90}} w_j,\] and \[\bar{c}_{90} = \frac{\sum_{e_j \in \mathcal{E}_{u,d,c}^{90}} w_j c_j}{\sum_{e_j \in \mathcal{E}_{u,d,c}^{90}} w_j}.\] If no evidence appears in the most recent 90 days, we set \(S_{90}=0\) and \(\bar{c}_{90}=0\).

The recent score is defined as \[\mathrm{score}_{\mathrm{recent}} = 0.65 \min\!\left(1, \frac{S_{90}}{2}\right) + 0.35 \bar{c}_{90}.\] Compared with the stable score, the recent score places more emphasis on recent support and does not require broad temporal dispersion across the full timeline.

A canonical interest is marked as a preliminary recent interest if it does not satisfy the stable-interest condition but satisfies \[\mathrm{score}_{\mathrm{recent}} \geq \theta_{\mathrm{recent}}\] and has sufficient recent evidence: \[S_{90} \geq 2.\] In our implementation, we set \[\theta_{\mathrm{recent}} = 0.60.\]

10 Prompt Analysis: Conservative vs.Neutral Evaluation Instructions↩︎

The profile evaluation prompt (Appendix 6.2) instructs models to “prefer fewer, broader tags” and limit most active domains to “1–2 reliable tags.” This conservative design intentionally prioritizes precision over recall: without such guidance, models in pilot experiments produced noisy tag lists containing near-duplicates and one-off mentions that did not reflect genuine user interests.

We note a potential concern: if the prompt constrains output volume, the benchmark may measure prompt compliance rather than profiling capability, creating an artificial ceiling on recall. To rule this out, we conduct a prompt analysis replacing the conservative instructions with a neutral variant that removes all constraints on tag count and breadth. The full neutral system prompt is:

The neutral user prompt mirrors the same changes:

We evaluate Qwen3.5-35B-A3B under the direct profile construction setting on the full 100-user evaluation subset, using identical gold profiles, anchor-matching, and evaluation protocol across both prompt variants.

6pt

Removing the conservative constraints changes metrics only marginally. Interest Recall improves slightly (+0.023), but Interest Precision drops by more (–0.073) as models produce specific tags that misalign with gold labels. The net effect on Interest F1 is negligible (+0.014). These results confirm that the conservative prompt is not the primary driver of low recall—the gap reflects genuine model limitations in recovering fine-grained interests from behavioral traces, not an artifact of benchmark design.

11 Input-Modality Analysis↩︎

To quantify the relative contribution of each input modality to profiling performance, we evaluate two models under progressively richer input configurations: text only, image captions only, text plus image captions, and the full setting adding timestamps. Table 5 reports active-domain detection and interest-tag recovery across all four conditions. Text provides the primary profiling signal, while image captions and timestamps contribute complementary gains, as discussed in Section 4.4.

6pt

12 Case Studies: Error Analysis↩︎

This appendix presents detailed case studies supporting the error analysis in Sections 4.2 and 4.3. We examine predictions from representative users across all seven domains, comparing gold profiles against model outputs under the direct, hierarchical, and extractive settings. Tables 7–10 organize examples by failure pattern.

Notation. In all case-study tables that follow: S = stable interest tags; R = recent interest tags; D = direct, H = hierarchical, E = extractive–abstractive profiling.

5pt

12.1 Over-Generalization and Cross-Category Confusion↩︎

Table 7 illustrates the most pervasive failure mode: models collapsing multiple specific gold tags into one or two broad categories, and confusing semantically adjacent but factually incorrect categories.

6pt

The User A food_drink case exhibits three distinct but related failure modes in a single domain. First, over-generalization: GPT-5.4 collapses seven specific gold tags—dining out, birthday cake, dessert, coffee, restaurant dining, casual dining, and confectionery—into just two broad labels (sweets and desserts, restaurants and dining out). While these are not factually wrong, they discard the granularity needed for downstream personalization: recommending a confectionery shop is qualitatively different from recommending a birthday cake bakery, and a coffee shop recommendation differs from a casual-dining suggestion. Second, cross-category confusion: Gemini-2.5 predicts Baking despite the user having zero baking activity anywhere in the timeline. The gold profile explicitly includes home cooking as a negative interest—this user purchases prepared food and eats at restaurants, but does not cook. The model appears to conflate “engages with food content” with “prepares food at home,” a category error analogous to confusing “attends concerts” with “plays an instrument.” Third, false inactive: GPT-4o-mini and Qwen3-VL-8B classify the entire domain as INACTIVE, missing all seven gold tags. This is a severe false-inactive error: the domain is supported by posts spanning the full four-month observation window across multiple modalities (text, images, and mixed), yet two models fail to activate it at all.

The User B entertainment case exhibits the same over-generalization pattern. Gemini-2.5 predicts music, movies and TV, adding a film/television interest absent from the gold profile, while GPT-5.4 reduces five tags to one (Arabic music). Across both cases, the pattern is consistent: models default to broad, safe hypernyms and resist committing to the specific subcategories that make personalized recommendations actionable.

12.2 Domain Blindness: False Inactive Errors↩︎

Table 8 shows cases where models incorrectly classify an active domain as inactive, missing all gold interest tags. These errors concentrate in domains whose evidence is primarily textual and dispersed across many posts.

6pt

The User B travel case is illustrative: all four models predict inactive for a domain where gold lists city exploration, city walks, and urban exploration. These interests are expressed through casual text mentions across posts (e.g., “went for a walk downtown,” “exploring a new neighborhood”) rather than through prominent images or hashtags. Without a visually anchoring post, the evidence fails to reach the model’s activation threshold.

12.3 Hallucinated Domains: False Active Errors↩︎

The inverse failure—activating a domain the user does not actually engage with—occurs when models over-interpret incidental posts as preference signals. Table 9 presents two representative cases. The first involves a user whose timeline is dominated by professional-wrestling content (187 posts, gold-active domains: sports_outdoor and entertainment with wrestling-related interests). GPT-5.4 under hierarchical profiling activates the gaming domain with tags that explicitly name wrestling—“video game references in wrestling-related memes” and “Pokémon-themed wrestling events.” The model’s own labels concede the content is wrestling-related, yet it places them in gaming. Five posts out of 187 mention video games at all, and all five are wrestling-context posts: three document a CMLL\(\times\)Pokémon crossover wrestling show, one jokes about a wrestler taking time off to play a game, and one is an incidental mention. The model conflates wrestling content that references gaming with the user having a gaming interest.

4pt

The User B photography case reveals a broader systemic pattern: across the full benchmark, posting selfies—a common behavior on social media—is frequently misinterpreted as photography enthusiasm. This mirrors the over-generalization pattern from Section 12: models lack the pragmatic judgment to distinguish “taking a photo to document an experience” from “photography as a sustained interest.” The User C case adds a further dimension: even when the model correctly identifies the topic of a post (wrestling), it can assign it to the wrong domain, revealing a category-boundary problem in how models map post content to interest domains.

12.4 Recent-Interest Blindness and Temporal Bucket Confusion↩︎

Table 10 illustrates a pervasive failure: models cannot distinguish long-standing interests from recently emerged ones, assigning nearly all predictions to the stable bucket regardless of when evidence appears in the timeline. User D provides a representative case: a frequent traveler with a clear pattern of general, recurring travel behavior (cruises, city walks, sightseeing) punctuated by specific recent destinations (Volendam, the Netherlands, the Sinai Desert).

6pt

The failure pattern extends beyond User D. Across the benchmark, models assign 70–90% of predictions to the stable bucket regardless of temporal evidence distribution. When recent tags are predicted, they often correspond to interests that the gold profile classifies as stable, and vice versa. GPT-5.4’s Amsterdam/Netherlands contradiction is especially revealing: the model recognizes that “the Netherlands” is a recent topic but places the capital city of that same country in the stable bucket, demonstrating that these models perform surface-level topic labeling rather than genuine temporal reasoning about when and how frequently evidence appears.

12.5 Dialogue Case Studies↩︎

The following examples illustrate how profiling failures propagate into downstream dialogue (Section 4.3).

12.5.0.1 Profile under-generation case (food-drink user).

In the direct-profile-conditioned setting, a user’s gold food-drink profile contains seven specific interests spanning dining out, regional cuisines, and holiday meals. Gemini-2.5-Flash’s profile reduces this to two tags (Dining out, Asian cuisine); its dialogue response ignores food entirely, instead recommending astrophotography based on the user’s photography interests. The judge assigns coverage \(=\) 1.0/5, noting the model “ignores the required interests.” GPT-5.4’s profile for the same user captures restaurant dining and sushi; its dialogue recommends a specific Japanese restaurant, earning coverage \(=\) 3.0/5. The difference illustrates a compounding failure: conservative profiling removes the very tags that would enable diverse, personalized responses.

12.5.0.2 Visual dominance case (astrophotography fixation).

In the timeline-conditioned setting, models can access all posts directly, yet they exhibit the same modality bias observed in profile construction. For the user described above, Gemini-2.5-Flash overlooks seven supported food interests and a hiking interest to recommend astrophotography across both dialogue turns, fixating on the domain with the most visually prominent evidence (sky and landscape photography). This pattern recurs across users: when a domain generates abundant images, it crowds out text-supported interests in downstream dialogue, even when those text-supported interests are equally or more relevant to the user’s request.

These two cases illustrate a compounding failure chain. Conservative profiling strips away fine-grained interest tags (profile under-generation); the resulting sparse profile provides insufficient hooks for the dialogue model, which then defaults to visually dominant domains regardless of the user’s actual request. The chain can be broken at either stage—better profiling yields richer profiles, and direct timeline access bypasses profile sparsity—but the modality bias persists in both paths, suggesting that balanced cross-modal attention is a prerequisite for either approach to succeed.

13 Profile–Dialogue Correlation Tables↩︎

This appendix provides the full correlation tables referenced in Section 4.4.

6pt

3pt

14 Per-Domain Difficulty Breakdown↩︎

Table 13 breaks down Interest Tag F1 by domain for GPT-5.4 across the three profile-construction settings. The seven domains form a clear difficulty spectrum. Pets and Gaming sit at the top (Direct F1: 0.522 and 0.385), benefiting from concentrated, visually distinctive evidence—pet photos and game screenshots are unambiguous signals that models detect reliably. Travel & City Exploration and Entertainment anchor the bottom (Direct F1: 0.272 and 0.313), reflecting the challenge of synthesizing evidence dispersed across many posts: a user’s travel interests may be scattered across dozens of posts mentioning different destinations, cuisines, and activities, with no single post fully defining the interest.

Hierarchical profiling improves Interest Tag F1 on six of seven domains, with the largest gains on difficult domains that require cross-post aggregation. Sports & Outdoor gains +0.164 (0.324 \(\rightarrow\) 0.488) and Travel gains +0.122 (0.272 \(\rightarrow\) 0.394), confirming that chunking helps aggregate weak but recurrent signals that the direct setting misses. The exception is Pets, where Extractive profiling achieves the best result (0.575 vs.Hierarchical 0.497). Pets evidence tends to be concentrated in a small number of high-signal posts (photos of the user’s own pets), so selecting the top \(K\) posts per domain preserves nearly all available evidence while filtering noise. For text-dispersed domains, however, Extractive profiling underperforms Hierarchical, as the selection stage must commit to a fixed set of posts before knowing which evidence will prove relevant.

The difficulty ranking is consistent across all three settings, suggesting that per-domain hardness is an intrinsic property of how evidence is distributed across modalities and posts rather than an artifact of any particular profiling strategy.

15 Human–LLM Dialogue Evaluation Agreement Study↩︎

This appendix provides the detailed setup and full results of the human–LLM agreement study referenced in Section 4.3.

15.1 Sampling and Annotation Setup↩︎

We sample 30 model responses from the full dialogue evaluation pool (100 users \(\times\) 4 settings \(\times\) 6 models) using stratified sampling across three dimensions to ensure coverage of the full quality and setting space:

1. Dialogue setting: 7–8 responses from each of the four settings (Timeline-conditioned, Direct, Hierarchical, Extractive–abstractive), ensuring representation of both raw-timeline and profile-conditioned generation paths.

2. User intent: 15 stable-interest recommendation responses and 15 recent-interest exploration responses.

3. Quality tier: 10 responses each from low (GPT-5.5 Avg.\(\leq 2.5\)), medium (\(2.5 < \text{Avg.} \leq 3.5\)), and high (Avg.\(> 3.5\)) tiers, ensuring annotators see the full quality range.

Three annotators with prior experience on the SocialPersona annotation team (see Section 3.3) independently score each response. Each annotator receives a sheet containing, for every response: (1) the user’s gold profile summary (stable and recent interests per domain), (2) the user request text and intent type, (3) the model response text, and (4) the identical 0–5 scoring rubrics for coverage, concreteness, and fluency used by the LLM judges (Appendix 6.6). Annotators do not see LLM judge scores, model identities, or dialogue settings. Total annotation time is approximately 5 annotator-hours.

15.2 Agreement Metrics↩︎

We report two complementary agreement measures:

Human–human agreement. We compute Krippendorff’s \(\alpha\) for ordinal data across the three annotators on each dimension. This validates whether the evaluation task itself is reliably human-judgeable. We interpret \(\alpha > 0.80\) as strong agreement, \(\alpha \in [0.67, 0.80]\) as moderate, and \(\alpha < 0.67\) as tentative.

Human–LLM agreement. For each response and dimension, we average the three annotator scores to form a human reference. We then compute per-dimension Spearman rank correlation \(\rho\) between this reference and each LLM judge (GPT-5.5 and Qwen3.7-Max). Spearman \(\rho\) captures monotonic ranking agreement without assuming linearity or interval-scale properties of the 0–5 scores.

15.3 Results↩︎

Table 14 reports the full results. Three findings emerge:

First, human–human agreement follows the expected difficulty ordering: fluency is most objective (\(\alpha=0.79\)), followed by concreteness (\(\alpha=0.71\)), with coverage being most subjective (\(\alpha=0.65\)). All three values fall within the moderate-to-substantial agreement range, confirming that the evaluation dimensions are reliably applicable by trained annotators.

Second, both LLM judges correlate positively and significantly with human judgments across all dimensions (\(p < 0.01\) for all \(\rho\)), validating their use as automated proxies for dialogue quality assessment. GPT-5.5 consistently outperforms Qwen3.7-Max in human alignment, consistent with its role as the primary judge in our main results (Table 3).

Third, the dimension-level gap mirrors the human–human pattern: coverage shows the lowest agreement in both settings. This is expected for a task requiring judges to assess whether a response meaningfully engages with specific user interests; borderline responses that mention a domain tangentially without clearly centering on a gold interest are inherently ambiguous. The residual disagreement on coverage (\(\rho=0.52\)–\(0.58\)) sets a plausible upper bound on automated coverage evaluation precision, but the positive and significant correlations confirm that LLM judges capture the correct ranking signal for model comparison—sufficient for the benchmarking conclusions drawn in Section 4.3.

References↩︎