1 Introduction↩︎

Personalization has emerged as a fundamental capability for modern large language models (LLMs) [1]. As LLMs are deployed across recommendation [2], search [3], dialogue [4], writing assistance [5], and content generation [6], the same query is expected to produce different and contextually appropriate responses for different users. Traditional tools for representing users (e.g., collaborative-filtering embeddings) [7], [8] capture statistical regularities but do not interface naturally with instruction-tuned LLMs, whose only conditioning surface is the prompt itself  [9]. As there are increasingly more closed-weight model and agent experiences which users can only personalize via inputs, personalization through a representational view also becomes infeasible. This mismatch has driven a wave of interest in LLM-native natural-language user profiles that can be organically consumed by a frozen (and possibly private) backbone [6], [10], [11], injecting per-user signal without model retraining. Such natural-language profiles are also alluring for cross-domain and zero-shot transfer capabilities.

The simplest baseline for natural-language profiles is to concatenate a user’s raw interaction history into a prompt and let the LLMs attend over it directly [6]. In practice, this is prohibitive at deployment scale. Real users accumulate hundreds to thousands of interactions over months or years [8], [12], and stuffing the full history into every request forces the model to prefill an extremely long context on each call, inflating latency, KV-cache memory, and per-query inference cost, and pushing many users past the model’s context window entirely [11], [13].

Even when the window nominally fits, long-context attention degrades: salient signal gets “lost in the middle” [14], [15] and irrelevant interactions dilute the personalization quality. A natural-language profile sidesteps both failure modes: it compresses raw behavior into a compact, fixed-size representation that can be precomputed offline once per user, cached, and reused across arbitrarily many downstream queries, amortizing the prefill cost over an entire user session rather than paying it per request [13], [16].

A growing body of work moves beyond prompting and explicitly learns the profile generator. [13] fine-tune an LLM to produce task-aware textual summaries of user history. [16] train a 1.7B-parameter policy with GRPO, rewarding profiles that downstream engagement-prediction oracles can use to recover click signals. [17] close the loop end-to-end, treating a downstream recommender as the reward function for a profile/query generator. Profile-augmented variants of per-user PEFT methods [18] likewise distill “good” profiles by reference to a labeled downstream task. All these methods share a single supervision recipe: a labeled downstream task defines profile quality. While effective, this recipe has severe limitations: rewards are expensive and not scalable – each target task requires an annotated corpus and fresh label collection for policy retraining, resulting in expensive task-specific profiles which undercut generality and shared latent user interests.

In light of this challenge, we introduce Bidirectional User Modeling via Profiles (BUMP), a self-supervised framework that trains a profile generator without any downstream task labels. Our intuition derives from two principles: a good profile should be predictive of future behaviors, and conversely identifiable from these behaviors. These properties can be measured directly against the user’s own held-out interactions, without external annotation. Hence, we turn this self-consistency principle into a powerful training signal.

Specifically, given a batch of users, we ask an LLM policy to emit a free-form textual profile, or user summary, and a small frozen LLM judge scores it in two complementary directions: a forward direction in which the generated profile is used as a query to rank the user’s held-out interactions against in-batch interactions from other users, and a backward direction in which a held-out interaction is used as a query to rank the user’s profile against in-batch profiles from other users. Both directions are scored with multi-positive NDCG and combined into a single dense per-rollout reward that drives policy optimization with GRPO [19].

In BUMP, because every other user in the batch supplies free negatives, each training example yields supervision from raw interaction logs alone – no downstream task is invoked, no annotation is collected, and the supervision budget grows automatically with the user population rather than the labeling budget. Self-supervision also frees the profile from any single downstream task’s bias: the user profile is optimized to be a faithful description of the user and not a feature engineered for any one particular classifier, so the same profile transfers across tasks without retraining the generator. Our contributions are threefold:

• A novel self-supervised objective for user profile generation. We formulate user profile generation as a bidirectional in-batch ranking problem, deriving dense supervision from raw interaction logs without any downstream task label.

• BUMP and BUMP+. We instantiate the objective as BUMP, a GRPO-trained profile generator scored by a frozen LLM judge with multi-positive NDCG over in-batch negatives, and propose BUMP+, which strengthens the signal via hard-negative mining in both directions.

• Strong empirical results. On the LaMP benchmark, our proposals match or outperform closed-source API baselines packed with more parameters and prior methods trained with explicit labeled downstream rewards.

2 Related Work↩︎

2.1 LLM Personalization↩︎

Personalization for instruction-tuned LLMs has received substantial recent attention, summarized in several surveys covering taxonomy, evaluation, and applications [1], [20]–[23]. Existing methods fall into three broad families. A first line retrieves a small set of user-history items at inference time and concatenates them into the prompt, leaving the LLM frozen but paying a per-query retrieval and prefill cost that remains bounded by the model’s context window [5], [6], [24], [25]. A second line assigns each user a lightweight adapter trained on their history, or fine-tunes the backbone with user-specific objectives, achieving strong task-specific accuracy at the cost of per-user training and opaque, non-interpretable weight deltas [2], [18], [20], [26]–[30]. A third line operates below the prompt surface, encoding user history as soft prompts, low-rank hidden-state shifts, or activation-steering directions that are parameter-efficient but neither interpretable nor portable across backbones [31]–[35]. BUMP falls into none of these families: like retrieval methods it produces a textual artifact in the prompt and like representation methods it compresses behavior into a fixed-size cache, but unlike all of the above the profile generator itself is trained without per-user adaptation, retrieval supervision, or downstream task labels.

2.2 User Profile Generation↩︎

A directly relevant line of work focuses on generating a natural-language profile, persona, or summary of a user that downstream systems can consume; recent empirical studies show that the form and content of these profiles materially affect personalization quality [36], [37]. The simplest approach prompts a strong LLM to summarize a user’s history – as a static paragraph, a structured object updated over time, or as input to multi-agent simulation – but profile quality remains bounded by zero-shot prompting and task-aware prompts tie each profile to a single application [13], [18], [38], [39]. A more recent line trains the profile generator itself, optimizing it against engagement-derived rewards, downstream recommenders, distilled supervision, or controllable per-dimension objectives – all of which share a common dependency on a labeled downstream task to define what makes a profile good [10], [11], [16], [17], [23], [26], [40].

3 Bidirectional User Modeling via Profiles (BUMP)↩︎

3.1 Preliminaries↩︎

We denote a set of users as \(\mathcal{U}\), each with a chronologically ordered interaction history \(H_u = (h_{u,1}, h_{u,2}, \ldots, h_{u,T_u})\), where each \(h_{u,t}\) is a textual record of an interaction, i.e. a dialogue turn, item interaction with metadata, or a written post or paper. Our goal is to train a profile generation LLM \(\pi_\theta\) which maps a user’s history to a free-form textual profile \(s_u \sim \pi_\theta(\cdot \mid H_u^{\text{vis}})\), which we call the user’s profile. Crucially, no labeled downstream task is available at training time; the only supervision we may exploit is the user’s own interaction stream.

3.2 Overview↩︎

To enable self-supervised evaluation of \(s_u\), we partition each user’s history into a visible prefix \(H_u^{\text{vis}}\) that is shown to the policy and a held-out portion \(H_u^{\text{ho}} = (h_u^{(1)}, \ldots, h_u^{(P)})\) that is reserved as supervision. A faithful profile of \(u\) should make \(H_u^{\text{ho}}\) recognizable as belonging to \(u\) in two complementary senses: (i) using \(s_u\) as a query, the judge should rank items in \(H_u^{\text{ho}}\) above items belonging to other users; and (ii) using any one item in \(H_u^{\text{ho}}\) as a query, the judge should rank \(s_u\) above profiles belonging to other users.

We turn this two-sided self-consistency principle into a bidirectional reward computed by a small frozen LLM judge \(J\), and use it to train \(\pi_\theta\) with reinforcement learning. Concretely, the training pipeline operates over a batch \(\mathcal{B}\) of \(B\) users: (i) \(\pi_\theta\) emits one or more candidate profiles from each \(H_u^{\text{vis}}\); (ii) every user’s profile and every user’s held-out items collectively form a pool of anchors, positives, and in-batch negatives; (iii) \(J\) produces ranking signals in two directions, scored with multi-positive NDCG; and (iv) GRPO [19] updates \(\pi_\theta\) to maximize the resulting dense per-rollout reward. The training reward is composed of two complementary ranking signals computed by the frozen judge \(J\) over the batch \(\mathcal{B}\), as shown in 1 and 2.

3.3 Bidirectional Reward Design↩︎

Forward direction (profile \(\to\) history). For each user \(u \in \mathcal{B}\), we form a candidate pool by combining \(u\)’s \(P\) held-out items \(H_u^{\text{ho}}\) (positives) with \(K\) items sampled from other users’ held-out items (in-batch negatives). The judge is prompted with \(s_u\) as a query and the \(P+K\) items as a list of candidates, and is asked to return a ranked permutation. Letting \(r_{p,\text{fwd}}^{u}\) denote the predicted rank of the \(p\)-th positive, the forward reward is multi-positive NDCG with binary relevance: \[R_{\text{fwd}}^u = \frac{1}{Z_P}\sum_{p=1}^{P} \frac{1}{\log_2\big(r_{p, \text{fwd}}^{u} + 1\big)},\] where \(Z_P = \sum_{j=1}^{P} \frac{1}{\log_2(j+1)}\). In this formulation, \(R_{\text{fwd}}^u = 1\) precisely when every positive lands in the top \(P\) slots of the judge’s ranking, i.e. when the summary \(s_u\) has successfully identified \(u\)’s held-out behavior against the in-batch distractors.

Backward direction (history \(\to\) profile). For each held-out item \(h_u^{(p)}\), we form a pool by combining \(u\)’s generated summary \(s_u\) (sole positive) with \(K\) profiles sampled from the other users in the batch (in-batch negatives). The judge ranks the \(1+K\) profiles given \(h_u^{(p)}\) as a query, yielding a single-positive NDCG; averaging across the \(P\) held-out items of \(u\) gives \[R_{\text{bwd}}^u = \frac{1}{P}\sum_{p=1}^{P} \frac{1}{\log_2\big(r^{u}_{p,\text{bwd}} + 1\big)},\] where \(r^{u,}_{p,\text{bwd}}\) is the rank assigned to \(s_u\) by \(J\) given the \(p\)-th held-out item as query.

Intuitively, the two terms enforce complementary properties: the forward term requires \(s_u\) to be predictive of \(u\)’s held-out behavior, while the backward term requires \(s_u\) to be identifiable from that behavior. Optimizing the forward term alone would tolerate generic profiles that match many users’ items; optimizing the backward term alone would tolerate idiosyncratic profiles that fail to capture actual behavior. Their conjunction is strictly tighter than either alone.

3.4 Additional Regularization↩︎

Position Debiasing. Single-pass LLM list-rankers are known to exhibit position bias [41], [42]: items shown earlier in the candidate list are systematically rewarded over those shown later, independently of their relevance. We mitigate this by averaging over \(M\) independent random presentation orders. Specifically, let \(\sigma\) denote a permutation of the candidate indices \({1,\ldots,N}\), where \(N = P+K\) (forward) or \(1+K\) (backward), and let \(\mathrm{NDCG}(\sigma)\) denote the NDCG that results when the judge is shown the candidates in order \(\sigma\) and the resulting ranking is scored against the ground-truth positives. Sampling \(M\) permutations \(\sigma_1,\ldots,\sigma_M\), the debiased per-direction reward is \(\bar{R}^{u}_{\mathrm{dir}} = \frac{1}{M}\sum_{m=1}^{M}\mathrm{NDCG}\left(\sigma_m\right)\) where \(\mathrm{dir}\in \{\mathrm{fwd},\mathrm{bwd}\}\) which is an unbiased Monte-Carlo estimator of the expectation \(\mathbb{E}_{\sigma\sim\mathrm{Unif}(\mathcal{S}_N)}\left[\mathrm{NDCG}(\sigma)\right]\). Increasing \(M\) reduces estimator variance at linear cost in judge calls; we find \(M \leq 3\) sufficient.

Profile Length Penalty. Without an explicit length constraint, an LLM trained against NDCG-style rewards quickly discovers a reward-hacking strategy: pad the profile with broadly relevant boilerplate so that any held-out item becomes superficially plausible, inflating the judge’s ranking signal at the expense of representational specificity [16]. We suppress this failure mode with a soft length penalty that activates only when the profile exceeds a per-task threshold \(T\): \[R_{\mathrm{len}}(s_u) = \Big[\exp\Big(\alpha,\tfrac{\max(0,; \mathrm{ntok}(s_u)-T)}{T}\Big) - 1\Big],\] where \({\mathrm{ntok}(s_u)}\) denotes the token count of \(s_u\) as produced by the policy’s tokenizer, and \(\alpha\) controls how sharply the penalty ramps beyond \(T\). Below the threshold, \(R_{\mathrm{len}}=0\) and the policy is free to choose any length; above it, the exponential ramp grows fast enough to dominate any further NDCG gain a longer profile could buy.

Combining all three reward components, the per-rollout reward optimized by GRPO is \[R(s_u)=\bar{R}^{u}_{\mathrm{fwd}}+\bar{R}^{u}_{\mathrm{bwd}}-R_{\mathrm{len}}(s_u).\]

3.5 BUMP+: Hard-negative Mining↩︎

Random in-batch negatives are often easy: an item drawn from a topically unrelated user is trivially separable, producing a saturated reward that delivers little gradient. We further propose BUMP+, which strengthens the signal along both directions using a frozen embedding model \(\phi\) – in practice, we use BGE  [43]. In the forward direction, we extend each user’s candidate pool to a wider window \(H_u^{\text{pool}} \supseteq H_u^{\text{vis}}\) and select, from \(H_u^{\text{pool}} \setminus H_u^{\text{vis}}\), the items whose \(\phi\)-similarity to the visible-history centroid is lowest; these are the user’s own items that are topically distinct from the visible window, and they share the user’s identity while being unaligned with the profile’s training distribution. In the backward direction, we identify the top-\(J\) users in the batch whose visible-history centroids are most similar to \(u\)’s under \(\phi\), and use their generated profiles as negatives – forcing the judge to distinguish profiles of topically similar users rather than random ones. Hard-negative mining adds a single \(\phi\) pass per item, requires no labels, and is orthogonal to the rest of the pipeline.

3.6 Optimization and Downstream Use↩︎

Given the reward formulation for each rollout, we optimize \(\pi_\theta\) with Group Relative Policy Optimization (GRPO) [19]. Once \(\pi_\theta\) is trained, deploying the profiles requires no further optimization. For each user, the policy generates a single profile from their history offline; the resulting string is cached and reused. At serving time the profiles is prepended to any downstream prompt – classification, ranking, dialogue, generation – and consumed by an arbitrary backbone.

3.7 Connections to Self-Supervised Learning↩︎

BUMP is a direct instantiation of the contrastive self-supervised learning (SSL) recipe, lifted into the space of natural language. Contrastive SSL treats each datum as an implicit class and learns by pulling together different “views” of the same datum while pushing apart views of distinct data [44], [45]. In our setting, the implicit class is a user; the two views are the generated profile \(s_u\) and any held-out \(h_u^{(p)}\); and the negatives are supplied by the other users in the batch, mirroring the in-batch-negative trick that has scaled contrastive learning [44]–[46].

The bidirectional structure of our reward then plays the role that symmetric losses play in contrastive SSL, where one-sided objectives are well known to admit degenerate, collapsed solutions [47], [48]. Our forward and backward terms are the analogue for textual user profiles: the forward term alone admits profiles predictive of the user’s behavior but too generic to be identifying; the backward term alone admits profiles that are uniquely identifying but uninformative. Their conjunction enforces both descriptive sufficiency and discriminative identity.

4 Experiment↩︎

4.1 Experimental Setup↩︎

Dataset. We utilize the LaMP benchmark [6] as the standard testbed for prompt-level LLM personalization [18], [26]. We cover six of the seven public LaMP tasks; LaMP-6 (personalized email writing) is omitted as its corpus is not publicly released. The six tasks together span five domains and three task types – binary classification, multi-class classification, ordinal regression, and free-form generation, with additional details explained in 6.1. We adopt the time-based split released with LaMP.

Setting. For each LaMP task, we train \(\pi_\theta\) on the training-split users using only their interaction histories — no LaMP labels are consulted to construct the reward. Each user’s history is partitioned chronologically into a visible prefix \(H_u^{\mathrm{vis}}\) (the most recent 15 items, prepended to the generation prompt) and a held-out positive \(H_u^{\mathrm{ho}}\) (the single subsequent item). The bidirectional NDCG reward of 3.3 is computed by the frozen judge against \(K{=}8\) in-batch negatives and averaged over \(M{=}2\) random presentation orders for position debiasing. GRPO runs for two epochs with group size \(G{=}8\) rollouts per prompt.

We then freeze \(\pi_\theta\), greedily decode one profile \(s_u\) per user (up to 500 tokens) for both splits, and cache it. To read out downstream personalization quality we fine-tune another Qwen3-4B-Instruct backbone with a LoRA adapter [49] on the LaMP pairs \((\texttt{input}, \texttt{gold})\), conditioning each input on the cached profile via templates reported in 6.2. Across all compared methods the only difference is this textual prefix, isolating the contribution of the user representation. We run all experiments for 3 times and reported their average metric values.

Backbone Models. The profile generator \(\pi_\theta\) and the frozen LLM judge \(J\) are both initialized from Qwen3-4B-Instruct-2507 [50]. We train in bfloat16 mixed precision with FlashAttention-2 [51] and gradient checkpointing. The judge is served via vLLM [52] on a dedicated set of GPUs through a batched /v1/completions endpoint, and is constrained to emit only parseable rankings using vLLM’s guided-regex decoding. The judge is never updated during training. For BUMP+, hard negatives are mined with BGE-base-en-v1.5 (768-dim) [43]. Embeddings are precomputed once per dataset and cached, so mining adds negligible overhead at training time. We further run ablation studies on the size of \(\pi_\theta\) and \(J\).

Compute. All experiments are run on a single 8-GPU H100 node. For GRPO we partition the GPUs as \(4\) for training + \(2\) for rollout (vLLM policy) + \(2\) for judge (vLLM judge); for SFT we uses all eight GPUs for LoRA-SFT training.

Baselines. We compare BUMP against four families of baselines, all evaluated under the same LoRA-SFT protocol so that the textual user prefix is the sole point of variation:

No personalization. The Qwen3-4B backbone is LoRA-fine-tuned on the LaMP \((\texttt{input}, \texttt{gold})\) pairs with no user prefix, no history, and no profile — a strict lower-bound that measures what the backbone can do without any user signal.

Raw history. The user’s most recent 15 interactions are concatenated and prepended to the input verbatim, with no summarization, following the prompt-based recipe of [6].

Zero-shot profiles. A frozen profile generator is prompted to summarize the user’s visible history. We instantiate this with three backbones: Qwen3-4B-Instruct (same family as our policy), and the closed-source APIs Gemini-3-Flash and Gemini-3-Pro [53]. None of these generators receive task-specific training; together they probe how far a strong prompted summarizer can carry the downstream task without RL.

Direct Reward. A trained profile generator following the recipe of LettinGo [10]: the same Qwen3-4B policy is optimized with GRPO, but the reward is the downstream LaMP signal (detailed in 6.3). This is the closest task-supervised counterpart to BUMP and isolates the contribution of self-supervision from the contribution of GRPO itself.

For all profile-based methods, we use an identical profile-generation prompt and an identical SFT input template for fair comparison. The exact prompts are listed in 6.2.

4.2 Overall Result↩︎

1 reports performance on all six LaMP tasks. Our takeaways are summarized as:

RL on Qwen3-4B substantially lifts the prompted baseline. Averaging relative improvement over the 11 metric rows, every trained variant beats the zero-shot Qwen3-4B profile: Direct Reward by \(10.8\%\), BUMP by \(10.3\%\), and BUMP+ by \(11.3\%\). The lift concentrates on classification (e.g., LaMP-2 F1 moves from \(65.5\) to \(84.7\), \(+29.3\%\) rel.), confirming that a 4B backbone becomes a competitive profile generator once tuned with the right reward.

Self-supervised matches or beats task-supervised. BUMP+ ties or wins against Direct Reward on 9 of 11 rows — losing only on LaMP-4 ROUGE-1/L — with the largest wins on classification (LaMP-2 Acc \(89.9\) vs.\(88.9\), F1 \(84.7\) vs.\(84.0\)) and LaMP-3 MAE (\(0.214\) vs.\(0.218\)). BUMP alone is more uneven, underscoring that the hard-negative mining is what makes self-supervision competitive. Together these support our central thesis: a well-designed self-supervised reward attains the same or better outcomes than labeled downstream rewards.

A 4B open model beats Gemini-3-Pro across the board. BUMP+ wins on all 11 metric rows against the strongest closed-source baseline (e.g., LaMP-2 F1 \(84.7\) vs.\(83.4\), LaMP-3 MAE \(0.214\) vs.\(0.222\), LaMP-7 ROUGE-L \(0.482\) vs.\(0.472\)), and BUMP wins on \(9\) of \(11\). Given the scale asymmetry, this strengthens the case for BUMP as a deployable alternative to per-user API calls.

Raw history wins on LaMP-4/5/7 generation. On the three ROUGE-scored generation tasks, Raw Hist. is the best column on every row, while BUMP+ remains the best profile-based method. We attribute this to a metric–representation mismatch: ROUGE rewards \(n\)-gram surface overlap with the user’s prior outputs, which raw history preserves verbatim, while any natural-language profile — ours or otherwise — abstracts away exactly that idiolect. On classification and rating tasks the surface signal is irrelevant and the user’s topical preferences — which a good profile does preserve — dominate; BUMP+ then flips the ranking and beats raw history on LaMP-1, LaMP-2, and LaMP-3 MAE.

4.3 Analysis on the Bi-directional Design↩︎

4pt

2 drops one direction at a time, with all other settings held fixed. Two patterns emerge: (i) Removing either direction degrades performance on every metric row, confirming that descriptive sufficiency (forward) and discriminative identity (backward) are non-redundant: the full reward is best on \(11/11\) rows. (ii) “w/o Backward” (forward-only) consistently beats “w/o Forward” (backward-only). The forward term directly optimizes the property downstream tasks need whereas the backward term primarily regularizes against generic, non-identifying profiles, refining rather than replacing the forward objective.

4.4 Perf. w.r.t. Profile Length↩︎

4pt

3 sweeps the length threshold \(T\in\{100,300,400\}\). The trend splits by task type. On the two classification tasks (LaMP-1/2), performance climbs monotonically. Both tasks are information-intensive — author attribution and topic tagging depend on the breadth of a user’s topical footprint, which a longer profile preserves. LaMP-3 through LaMP-7 plateau or slightly regress past \(T{=}300\). These tasks turn on a few coarse preference axes and extra tokens add little new signal and risk diluting the profile. \(T{=}100\) is the weakest setting on every row, confirming that the length penalty must still leave enough budget for the profile to be useful.

4.5 Perf. w.r.t. the Number of Negatives↩︎

4pt

4pt

4 sweeps the number of in-batch negatives \(K\in\{4,8,16\}\). Performance forms an inverted-U around \(K{=}8\), which is best on every row. At \(K{=}4\), a random batch rarely contains confusable distractors, so NDCG quickly hits its ceiling and the per-rollout gradient vanishes. At \(K{=}16\), the candidate pool exceeds the judge’s reliable list-ranking capacity [42]; orderings become noisy and the reward signal turns biased rather than merely sparse, suggesting judge hallucination is a stricter bottleneck than reward saturation.

We next study how many of the \(K{=}8\) negatives should be mind for difficulty rather than drawn uniformly. 5 sweeps the hard-negative count \(J_{\mathrm{hard}}\in\{2,4,6\}\). Going from \(J_{\mathrm{hard}}{=}2\) to \(J_{\mathrm{hard}}{=}4\) yields the bulk of the gain on every task. Pushing further to \(J_{\mathrm{hard}}{=}6\) produces no additional improvement and a marginal regression on most rows: once half the negatives are mined, the task is challenging enough that additional mining begins to introduce embedding noise — BGE-derived “hardness” is only a proxy for true confusability — without compensating gain in gradient quality.

Additional Experiments: Due to the space limitation, we include additional experiments in the appendix (i.e., perf. w.r.t. different judges and backbones in 6.4, reward learning curves in 6.5, ablation on the judge debias in 6.6, and complementarity of BUMP with the downstream reward in 6.7).

5 Conclusion↩︎

We introduced BUMP, a self-supervised framework that trains a textual profile generator with no downstream task labels. A bidirectional in-batch ranking reward — measuring whether a generated profile predicts a user’s held-out behavior and whether that behavior identifies the profile — supplies dense supervision from raw interaction logs alone, and embedding-based hard-negative mining (BUMP+) further sharpens the signal. On the LaMP benchmark, our 4B self-supervised policy matches or beats task-supervised baselines on \(9/11\) metric rows and outperforms Gemini-3-Pro on all of them, demonstrating that self-supervision can substitute for labeled downstream rewards in textual user modeling.

Limitation: Our framework inherits known LLM-as-judge biases (verbosity, stylistic, self-preference) [41], [42] and judge hallucinations grow as the candidate pool expands (4), capping policy quality. BUMP+ also relies on BGE similarity as a proxy for confusability; our \(J_{\mathrm{hard}}\) ablation shows aggressive mining injects noise — task-aware hardness measures are a natural next step.

6 Appendix↩︎

6.1 Datasets↩︎

We evaluate BUMP on the LaMP benchmark [6], the standard testbed for prompt-level LLM personalization. We cover six of the seven public LaMP tasks; LaMP-6 (personalized email writing) is omitted because its corpus is not publicly released. The six tasks together span five domains and three task types – binary classification, multi-class classification, ordinal regression, and free-form generation:

• LaMP-1: Personalized Citation Identification. Binary classification of which of two candidate papers a user is more likely to cite, given their publication history.

• LaMP-2: Personalized Movie Tagging. 15-way classification of a movie tag (e.g.”sci-fi”, “dark comedy”) consistent with the user’s prior tagging behavior; we use the updated “LaMP-2 (new)” split.

• LaMP-3: Personalized Product Rating. 1–5 ordinal rating prediction on Amazon reviews.

• LaMP-4: Personalized News Headline Generation. Generate a headline for a given article in the user’s headline-writing style.

• LaMP-5: Personalized Scholarly Title Generation. Generate a paper title from an abstract in the user’s authoring style.

• LaMP-7: Personalized Tweet Paraphrasing. Paraphrase a given tweet in the user’s tweeting style.

Statistics. 6 reports the number of training and development users, the average number of historical interactions per user, and the official evaluation metric for each task. The six tasks cover a wide range of regimes: average profile lengths vary from \(\sim\)​16 tweets (LaMP-7) to \(>\)​180 reviews (LaMP-3), and training sizes range from \(\sim\)​5K to 20K users.

Training Split and Evaluation Protocol. For each dataset,we randomly sample 10% of the training set as dev set for early stopping and hyper-parameter tuning and use the publicly released evaluation set as the testing set, as LaMP’s testing sets are not publicly available. With respect to evaluation metrics, for LaMP-1, we use accuracy; for LaMP-2, we use accuracy as well as F1 score; for LaMP-3, we use MAE and RMSE ; and for LaMP-4, LaMP-5, and LaMP-7, we use ROGUE-1 and ROGUE-L.

6.2 Prompts↩︎

The forward and backward judge prompts of the BUMP reward are already shown in 1 and 2; below we provide (i) the profile-generation prompts consumed by \(\pi_\theta\) at training and inference time (8), (ii) the parallel prompts used by the zero-shot Gemini baselines (9), and (iii) the downstream SFT input template that prepends the cached profile to every LaMP query (10). All baselines that consume a textual user profile share the same SFT template, so the prefix string is the sole point of variation across methods.

6.3 Direct Reward Formulation↩︎

The Direct Reward baseline (1) replaces our bidirectional NDCG reward with a downstream LaMP-task signal while keeping every other component fixed: identical policy (Qwen3-4B-Instruct), frozen judge \(J\), length penalty \(R_{\mathrm{len}}\), and GRPO hyperparameters. This isolates the contribution of self-supervision from the contribution of GRPO itself. For each rollout the judge is conditioned on the candidate profile \(s_u\) together with the LaMP question \(q_u\), and is asked to solve the downstream task directly from \(s_u\). The per-rollout reward is task-specific:

Classification (LaMP-1, LaMP-2).
The judge predicts one of \(L\) allowed labels — the two candidate papers for LaMP-1, the 15 movie tags for LaMP-2 — with the output space enforced by vLLM’s guided_choice constrained decoding to eliminate hallucination. The reward is exact match: \[R_{\mathrm{cls}}(s_u) = \mathbb{1}\!\left[J(s_u, q_u) = y_u\right],\] where \(y_u\) is the gold label.

Ordinal regression (LaMP-3).
The judge predicts a rating \(\hat{y}\in\{1,\dots,5\}\), again under guided decoding. The reward is graded by distance to the gold rating \(y_u\): \[R_{\mathrm{ord}}(s_u) = \max\!\Big(0,\; 1 - \tfrac{|J(s_u, q_u) - y_u|}{4}\Big).\]

Generation (LaMP-4, LaMP-5, LaMP-7).
The judge freely decodes the target text (a headline / paper title / tweet paraphrase). The reward is ROUGE-L F1 against the gold target: \[R_{\mathrm{gen}}(s_u) = \mathrm{ROUGE\text{-}L}\!\left(J(s_u, q_u),\; y_u\right).\]

In all three cases the task reward is combined with the same length penalty used by BUMP, yielding \(R(s_u) = R_{\mathrm{task}}(s_u) - R_{\mathrm{len}}(s_u)\), and is optimized with GRPO under the same group size \(G{=}8\) and per-device batch size as BUMP. Every other knob (epochs, learning rate, threshold \(T\)) is identical, so the only point of variation between Direct Reward and BUMP is the reward function itself.

6.4 Perf. w.r.t. Different Judges and Backbones↩︎

7 and 8 sweep the size of the frozen judge \(J\) and the GRPO policy \(\pi_\theta\) independently over \(\{0.6\text{B}, 4\text{B}, 8\text{B}\}\) Qwen3-Instruct variants. In both cases the 4B configuration is the default and matches the BUMP row of 1; we vary one component and hold the other at 4B.

4pt

4pt

Judge size saturates at 4B. Scaling the judge from 0.6B to 4B yields the bulk of the gain on every task (e.g.,LaMP-1 \(77.5\!\to\!80.9\), LaMP-3 MAE \(0.232\!\to\!0.226\)): a sub-1B judge produces noisy rankings that propagate as noisy gradients. Pushing further to 8B brings no consistent improvement — differences fall within seed-level noise on all \(11\) rows. The judge’s task (list-ranking \(1+K\) candidates) appears bounded in difficulty: once \(J\) is capable of producing low-variance rankings, additional scale is wasted.

Policy size keeps scaling. The GRPO policy, in contrast, benefits from every increment. The 0.6B policy is sharply worse across the board (e.g.,LaMP-2 F1 drops \(84.5\!\to\!76.5\)), consistent with \(\pi_\theta\) needing enough capacity to emit a coherent, on-distribution profile. Scaling to 8B yields a further consistent lift over the 4B default (e.g.,LaMP-1 \(80.9\!\to\!82.5\), LaMP-4 ROUGE-L \(0.179\!\to\!0.184\)). Profile generation is the harder, less bounded side of the system, and self-supervision evidently scales with policy capacity — a useful property given that the cost of serving a larger judge is what otherwise limits joint scaling.

6.5 Reward Trajectory of BUMP↩︎

11 plots the per-step mean of the two reward components across the six LaMP tasks. Two patterns hold uniformly.

Both directions improve monotonically on every task. Across all six tasks, both \(R_{\mathrm{fwd}}\) and \(R_{\mathrm{bwd}}\) rise smoothly throughout training, including LaMP-1 ( \(R_{\mathrm{fwd}}\!:\;0.65\!\to\!0.66\), \(R_{\mathrm{bwd}}\!:\;0.45\!\to\!0.50\)), where the absolute range is small but the trend is clean. The bidirectional reward thus delivers usable gradient over the full training trajectory rather than collapsing once one direction saturates.

The backward term carries more of the visible motion. On every task, \(\Delta R_{\mathrm{bwd}}\) exceeds \(\Delta R_{\mathrm{fwd}}\), often by a factor of two or more (e.g.LaMP-7: \(\Delta R_{\mathrm{bwd}} \approx 0.07\) vs.\(\Delta R_{\mathrm{fwd}} \approx 0.03\); LaMP-1: \(\Delta R_{\mathrm{bwd}} \approx 0.05\) vs. \(\Delta R_{\mathrm{fwd}} \approx 0.01\)). \(R_{\mathrm{fwd}}\) starts close to ceiling — LaMP histories carry strong topical signal that even a zero-shot profile already preserves, so the forward ranking task is already partially solved at step 0. \(R_{\mathrm{bwd}}\) instead requires the profile to be uniquely identifying among other users’ profiles, a harder property that the policy must actively learn, which is where most of the visible training-time progress shows up.

6.6 Analysis on Debiasing the Judge↩︎

4pt

9 sweeps the number of debiasing permutations \(M\in\{1,2,3,4\}\). At \(M{=}1\) the per-rollout reward is computed from a single arbitrary candidate ordering and inherits the judge’s position bias directly; performance drops on every row (e.g.LaMP-1 \(80.9\!\to\!79.0\), LaMP-2 F1 \(84.5\!\to\!82.8\), LaMP-4 ROUGE-L \(0.179\!\to\!0.172\)). \(M{=}2\) already captures the bulk of the benefit — the variance of the Monte-Carlo NDCG estimator falls as \(1/M\), so two permutations halve the dominant noise term — and further increases to \(M{=}3\) or \(M{=}4\) yield only seed-level differences. Since each additional permutation costs one extra judge call per rollout, \(M{=}2\) is the natural operating point and matches the default used in 1.

6.7 Complementarity of BUMP↩︎

Self-supervised and task-supervised rewards capture different aspects of profile quality: the bidirectional NDCG reward of BUMP optimizes the profile to be a faithful, identity-preserving description of the user, while the Direct Reward of 6.3 optimizes it to maximize a single labeled task signal. These objectives are not redundant. To test whether they combine cleanly, we train a third variant that augments BUMP+’s reward with the per-task Direct Reward at unit weight, yielding the per-rollout reward \[R(s_u) = \bar{R}^{u}_{\mathrm{fwd}} + \bar{R}^{u}_{\mathrm{bwd}} + R_{\mathrm{task}}(s_u) - R_{\mathrm{len}}(s_u),\] optimized with the same GRPO settings as the rest of the paper.

4pt

Combined reward wins on every row. 10 shows that the combined variant matches or beats both parents on all \(11\) metric rows (e.g.LaMP-1 \(81.0\) vs.\(80.1\,/\,79.3\); LaMP-2 F1 \(85.0\) vs.\(84.7\,/\,84.0\); LaMP-3 MAE \(0.211\) vs.\(0.214\,/\,0.218\); LaMP-4 ROUGE-L \(0.186\) vs.\(0.180\,/\,0.182\)). The two reward formulations capture complementary signal: the self-supervised term enforces broad user fidelity that transfers across tasks, while the task-supervised term sharpens the profile toward whatever idiosyncrasy the current LaMP task happens to need.

Practical implication. Our central thesis is that self-supervision is a viable replacement for downstream labels (1). This experiment shows the stronger complementary claim: when labels are available, BUMP can be layered on top of the task reward at no architectural cost and acts as a strict regularizer, rather than a competing alternative.

6.8 Additional Hyperparameters↩︎

We report the hyperparameters used by the two training stages in our pipeline: GRPO for the profile generator \(\pi_\theta\) (3.6) and LoRA-SFT for the downstream backbone that consumes the cached profile (3.6). The same configuration is used across all six LaMP tasks unless otherwise noted; per-task differences are limited to dataset paths and the length threshold \(T\) (3).

GRPO (profile generator \(\pi_\theta\)). The policy is optimized with TRL’s GRPO trainer in bfloat16 with gradient checkpointing.

• Epochs: \(2\)

• Per-device train batch size: \(4\)

• Gradient accumulation steps: \(20\) (effective batch \(4 \times 20 \times 4 = 320\) over 4 training GPUs)

• Learning rate: \(1.0\!\times\!10^{-5}\), warmup ratio \(0.05\)

• KL coefficient \(\beta\): \(0.01\)

• Group size (rollouts per prompt) \(G\): \(8\)

• Max completion length: \(500\) tokens

• Save strategy: every \(50\) steps, keep last \(10\) checkpoints

The bidirectional reward is computed against \(K{=}8\) in-batch negatives and averaged over \(M{=}2\) random presentation orders. For BUMP+, \(J_{\mathrm{hard}}{=}4\) of the \(K{=}8\) negatives are hard-mined with BGE; the rest are sampled uniformly in-batch.

LoRA-SFT (downstream backbone). Once \(\pi_\theta\) is frozen and each user’s profile is cached, we fine-tune a separate Qwen3-4B-Instruct backbone with a LoRA adapter on the LaMP \((\texttt{input}, \texttt{gold})\) pairs, conditioned on the cached profile via the template of 10.

• Epochs: up to \(10\) with early stopping (patience \(5\), threshold \(0\)) on dev marginalized accuracy

• Per-device train / eval batch size: \(4\,/\,4\)

• Gradient accumulation steps: \(4\) (effective batch \(4 \times 4 \times 8 = 128\) over 8 GPUs)

• Learning rate: \(1.0\!\times\!10^{-5}\), \(10\) warmup steps

• Max sequence length: \(4200\) tokens

• Evaluation / save strategy: every \(50\) steps, keep last \(3\) checkpoints

• Precision / memory: bfloat16, gradient checkpointing, \(16\) data-loader workers with pinned memory

LoRA configuration: rank \(r{=}32\), \(\alpha{=}32\), dropout \(0.05\), applied to all linear projections (target_modules=all-linear), task type causal LM. We disable load_best_model_at_end under DDP because PEFT’s adapter reload places every rank’s adapter on cuda:0, which OOMs on an 8-GPU run; instead, we select the best checkpoint post-hoc using dev metrics.

Shared across stages. Both stages use the same backbone family (Qwen3-4B-Instruct-2507) and the same dev-set early-stopping discipline (10% random split of training users; 6.1). The frozen judge \(J\) is served via vLLM with guided-regex decoding on a dedicated pair of GPUs and is never updated. All reported numbers are averaged over \(3\) random seeds sharing these settings.

References↩︎