1 Introduction↩︎

Search in large-scale video catalog applications differs from classical web search in a way that makes personalization especially important: ranking is incremental. The system must return useful results after every keystroke, including short prefixes where lexical evidence is weak but user intent already matters. In this regime, the first 1–3 characters create a signal vacuum: many plausible candidates share nearly indistinguishable text evidence, so non-personalized ranking tends to fall back to global popularity and broad behavioral priors rather than the current user’s taste.

This short-prefix regime is the core problem addressed in this paper. Our central empirical finding is that personalization matters far more on these ambiguous incremental-search inputs than on longer, more fully specified queries: offline, the proposed personalized ranker improves NDCG@10 by +8.63% on prefix queries of length 1–3, compared with only +1.46% on longer queries. We therefore frame the problem not simply as “building a better personalized ranker,” but as identifying the right personalization signals for the low-lexical-information regime of incremental search.

Figure 1 illustrates non-personalized incremental search behavior using three prefixes of increasing length from the same query (t \(\rightarrow\) th \(\rightarrow\) the s). In a non-personalized search experience, users may need to type several additional characters before the intended item becomes easy to find.

We compare against a non-personalized baseline ranker driven mainly by aggregated behavioral and lexical signals, so users in the same geography would see a shared default ranking for a query. While this baseline is robust for fully specified navigational queries, it underperforms precisely where incremental search is hardest: early prefixes with many plausible matches and insufficient lexical separation.

Our thesis is that hybrid embeddings are the right answer for this regime. Text-based embeddings provide semantic coverage and cold-start robustness when interaction data are sparse, while ID-based embeddings contribute high-precision collaborative structure when co-engagement signals are dense. Injecting both signals into a ranker lets the system personalize exactly where lexical matching is least informative, without sacrificing coverage or latency.

We target five constraints that make this problem challenging in practice:

• Ambiguity at short prefixes. Candidate sets are large while lexical discriminative power is low.

• Broad-intent queries with many valid matches. Generic queries (e.g., “comedy”) can return a large set of plausible candidates, making personalization critical for surfacing the most relevant results for each user.

• Coverage across the catalog. Personalization should work for both popular and long-tail content.

• Cold-start resilience. Newly launched items lack interaction history and cannot rely on collaborative signals alone.

• Strict serving budgets. Personalization must satisfy low-latency constraints at scale.

To address these constraints, we introduce a hybrid personalization stack that combines semantic and collaborative embeddings within a learning-to-rank pipeline. Our core contributions are

1. Problem framing for incremental-search personalization. We show that personalization has disproportionate value on short-prefix queries, with NDCG@10 improving by +8.63% on 1–3 character queries versus +1.46% on longer queries.

2. Hybrid embedding formulation for the short-prefix regime. We learn two complementary item spaces: a text-semantic encoder (TextEmb) and an ID-collaborative model (IdEmb), and show through ablations that each contributes non-redundant ranking signal.

3. Online user representation for low-latency ranking. We compute user embeddings from previously watched items with bounded-memory aggregation, and expose per-candidate text/ID cosine features to the ranker with minimal latency overhead.

4. Ranker integration. We inject hybrid personalization signals into a pairwise XGBoost model trained on large scale user engagement dataset, preserving compatibility with an existing multi-stage retrieval and ranking stack.

5. Evaluation across offline and online regimes. We report temporal hold-out metrics (NDCG@10, MRR@10, converted-item position) and a three-week online A/B test demonstrating statistically significant engagement gains.

6. Embedding-quality analysis with reduced click bias. We construct a held-out evaluation corpus via a large language model (LLM), decoupling embedding quality assessment from exposure bias in session-derived data.

Table 1 summarizes the headline results. The prefix-query lift is the central result: it shows that incremental search is the regime where personalization contributes the most value.

2 Related Work↩︎

2.1 Learning to Rank↩︎

Learning-to-rank methods for search include pointwise, pairwise, and listwise formulations [1]. Pairwise approaches such as RankNet/LambdaRank remain strong practical choices in industrial systems [2], [3]. In high-throughput applications, tree-based models with pairwise objectives (e.g., LambdaMART, XGBoost with rank:pairwise) provide an effective trade-off between ranking quality, feature flexibility, and serving efficiency [4]. Our work builds on this line by adding hybrid embedding signals to an existing pairwise ranker.

2.2 Personalized Search↩︎

Personalized web search has a long history [5], [6]. Early work used long-term user interest profiles derived from query logs; later work distinguished short-term and long-term behavioral signals [7]. Recent industrial evidence from streaming search at Netflix further highlights the practical impact of personalization in large-scale media systems [8]. A key challenge in this line of work—also central to our setting—is balancing personalization strength against degradation for navigational queries where all users share the same intent. Our system explicitly addresses this by treating personalization signals as additive features within an existing relevance-first ranker.

2.3 Embedding-based Retrieval and Dense Bi-Encoders↩︎

Dual-encoder architectures such as DSSM [9] established independent query/document encoding for scalable relevance modeling. Later two-tower systems improved industrial retrieval through large-batch training and ANN serving [10]. Dense retrieval methods (DPR [11] and large dual encoders [12]) demonstrated that bi-encoder models pre-trained on natural language can generalize well to new domains via contrastive fine-tuning. Sentence-BERT [13] extended this to sentence-level semantic similarity; we adapt this line of work to multilingual media metadata with a domain-specific fine-tuning objective.

2.4 Collaborative and Hybrid Recommendation↩︎

Collaborative methods from matrix factorization [14] to neural collaborative filtering [15] and item2vec-style representations [16] capture behavioral proximity effectively for head items. Hybrid recommenders combine collaborative and content-based signals to improve robustness under sparsity and cold-start [17]. Our approach instantiates this hybrid principle in search ranking. Contrastive self-supervised objectives have also been applied to recommendation [18], [19], motivating our use of a pairwise hinge loss for embedding training.

2.5 Session-based and Sequential Recommendation↩︎

Session-based models [20], [21] capture short-term intent from recent interactions without requiring long-term user profiles. Sequential models such as SASRec [22] and BERT4Rec [23] apply self-attention over the history to model temporal dynamics. Our current system uses mean pooling over watch history, an approach that has also been used in prior large-scale industrial recommendation systems to build user embeddings by averaging watch-history item embeddings [24]. We identify this approach of mean pooling as a limitation and target for future work; however, mean pooling provides a simple, low-latency baseline that already yields substantial gains (Section 6).

2.6 Personalization in Large-Scale Media Systems↩︎

Large-scale industrial recommender stacks combine user history with multi-stage ranking and deep representation learning [24]–[26]. One especially relevant prior system [24] builds a user embedding by averaging item embeddings from watch history and uses it in a two-stage retrieve-then-rank pipeline. Recent industrial search work likewise demonstrates the value of personalization at scale [8]; our work differs in focusing specifically on the per-keystroke incremental regime, where lexical signal is weakest and personalization most needed. Our contribution further complements behavioral embeddings with text-semantic embeddings to handle cold-start and long-tail content.

3 System Overview↩︎

Figure 2 shows the end-to-end architecture. The system comprises three phases.

Offline Embedding Generation. Item embeddings are generated offline over the full video catalog at a fixed refresh cadence using item identifiers and metadata signals. We materialize these embeddings in two serving surfaces: (i) a key–value store indexed by item ID for low-latency online lookup, and (ii) the search index document metadata so embeddings are retrieved directly during candidate recall.

User History. User watch history is maintained in a low-latency key-value store, refreshed on a regular cadence from interaction logs. Per user, we retain a bounded set of the most recent play events over a long-horizon window; episode-level interactions are mapped to the parent show, and duplicate items are deduplicated to the most recent play.

Online serving. At query time, the search stack retrieves user history from the key-value store and forms user representations via mean pooling over historical item embeddings. The ranker consumes both user vectors and per-candidate item embeddings (available from index metadata), and computes user–item cosine similarity features in both embedding spaces (ID-based and text-based). These personalization features are combined with lexical and behavioral signals in the final ranking model. The end-to-end personalization adds minimal latency to the non-personalized baseline.

4 Embedding Models↩︎

4.1 Problem Formulation↩︎

Let \(\mathcal{V}\) denote the set of all video items (movies and TV shows), and let \(\phi(v) \in \mathbb{R}^d\) be the embedding of item \(v \in \mathcal{V}\). Training instances are constructed from search-session sequences. For a current item \(i_t\), we define a positive item \(i_{t+1}^{+}\) as the next watched/converted item from the same user within a specified period of time, and a negative item \(i^{-}\) from in-batch negative sampling. The objective is to make the positive pair more similar than the negative pair by a margin \(m>0\), implemented via the pairwise hinge loss \[\mathcal{L}=\max\!\Big(0,\, m - \mathrm{cos\text{-}sim}\!\left(\phi(i_t), \phi(i_{t+1}^{+})\right) + \mathrm{cos\text{-}sim}\!\left(\phi(i_t), \phi(i^{-})\right)\Big), \label{eq:obj}\tag{1}\] which serves as the canonical training objective for both embedding models.

4.2 Training Data Generation↩︎

Training data for both embedding models is derived from search logs, a dataset of individual user search interactions recording queries, impressed items, clicked items, and converted (purchased/played/subscribed) items alongside temporal metadata.

Positive pairs. A positive (anchor, positive) pair is formed when two items, \(v_a\) and \(v_p\), appear in the same user’s search-driven engagement history within a specified time window. We do not use general playback-log data for this step. Although the broader playback corpus is substantially larger, it is noisier (e.g., it includes recommendation and homepage exposures), whereas search-log-derived pairs are smaller but better aligned with the search use case.

Negative sampling. For each positive pair, negative items are sampled uniformly from the current mini-batch.

LLM-based evaluation data. Session-derived evaluation is biased by the same factors that confound training: shared-account co-engagement spans divergent genres, promoted and free-tier content drives co-viewing independent of similarity, and head items dominate the co-engagement graph. We construct a held-out evaluation corpus using a large instruction-tuned language model (LLM) to decouple embedding quality assessment from these exposure biases. Anchor items are sampled uniformly across popularity deciles, and each item appears as an anchor a bounded number of times to ensure balanced representation across the catalog—a property unattainable with session-derived pairs. The LLM is prompted with a role specification, a one-shot contrastive example, and structured metadata (title, media type, genre, principal cast) for each candidate pair, and returns a binary similarity judgment under the criterion: “two items are similar if a viewer of one is likely to enjoy the other.” Positive judgments form the positive set for triplet evaluation; negatives form high-quality hard negatives. The corpus is used exclusively for evaluation; no LLM call is made at serving or training time.

4.3 Text-Based Embedding Model: TextEmb↩︎

The text-based embedding model (TextEmb) is a Siamese encoder trained on co-played item pairs. For each item, metadata fields are encoded with a shared text encoder and then combined through reshape/concatenation plus an MLP projection head to produce a shared embedding. Figure 3 illustrates the metadata encoding and projection pipeline used to produce the final item vector.

Backbone. The text encoder is a multilingual transformer-based sentence encoder initialized from a strong multilingual backbone and further adapted for semantic similarity over multilingual text pairs. Using a multilingual encoder naturally handles non-English locales without requiring separate models per language.

Item Tower. For each item, we tokenize multiple text fields (genre, title, artists, entity type) independently at fixed length. Each field is encoded by the shared multilingual transformer backbone. The resulting field embeddings are reshaped/concatenated and passed through a projection head (batch normalization, linear layers, and ReLU), then L2-normalized to produce the final item vector. During fine-tuning, the encoder remains trainable so the representation adapts to domain-specific semantics in media search.

Training objective. Both embedding models optimize the pairwise hinge loss in Eq. 1 . In implementation, we compute similarity with a dot product between L2-normalized item vectors; under this normalization, dot product is equivalent to cosine similarity, so Eq. 1 and the training-time implementation are identical. Here \(i_t\) denotes the anchor item, \(i_{t+1}^{+}\) the positive item, \(i^{-}\) the negative item, \(\phi(\cdot)\) the embedding function, and \(m\) the margin selected on a validation split.

Training Setup. The model is trained for multiple epochs using multi-GPU mixed-precision training, with tuned batch size and learning-rate schedule selected on a validation split.

4.4 ID-Based Collaborative Embedding Model: IdEmb↩︎

The ID-based model (IdEmb) is also a Siamese encoder trained on co-played item pairs, using only discrete item identifiers (no content metadata). This design captures behavioral co-play patterns that may not be reflected in textual signals e.g. two genre-distinct films that attract the same audience demographic.

Architecture. A trainable ID-embedding lookup maps each item identifier to a high-dimensional latent vector, which is then refined by a compact multilayer projection network into the shared embedding space used for scoring. This design preserves item-level behavioral diversity while producing representations compatible with downstream ranking features.

Training. The same co-played-item pair construction, Siamese training objective, and infrastructure as TextEmb are used, but the encoder tower replaces text encoding with ID embedding lookup. The model is trained for multiple passes over the data, and optimization settings are aligned with those used for TextEmb to enable consistent comparison.

Coverage limitation and cold-start fallback. Because the ID table is learned from historical interactions, items not observed during IdEmb training do not receive an ID embedding at all, and very low-frequency items may receive less reliable representations. For such items, the system falls back to the TextEmb text embedding features at serving time; coverage is summarized in Table 2.

Table 2 motivates the hybrid design: TextEmb provides full catalog/user coverage, while IdEmb contributes high-precision collaborative signal for the majority of sessions and users where interaction-derived items are available. The missing 12–13% of session/user coverage for IdEmb comes primarily from items that were not seen during item-ID model training, and therefore have no learned ID vector at serving time; this is exactly the gap that TextEmb covers.

4.5 Embedding Evaluation↩︎

Table 3 first reports the training and validation losses tracked during model training for both models.

Table 4 summarizes triplet evaluation metrics for both models on the held-out LLM-annotated evaluation corpus.

The results highlight complementary strengths: IdEmb has lower pairwise-loss values, while TextEmb achieves higher triplet accuracy and a larger A-P/A-N margin on the LLM-labeled set; we defer detailed interpretation to Section 6.3.

5 Personalization Ranking Model↩︎

5.1 Feature Engineering↩︎

The personalized ranker extends the existing non-personalized feature set with two embedding-derived affinity features. In practice, features fall into few core groups: long and short-horizon behavioral signals (e.g., click/watch tendencies), lexical text-match signals for query title/cast/genre overlap, catalog-quality signals (e.g., popularity and quality priors), and personalization signals. This preserves the relevance-first behavior of the ranker while adding user-specific evidence.

The two added personalization features encode user–item affinity from collaborative and text-semantic spaces, respectively. They contribute information that is complementary to behavioral aggregates and lexical matching, enabling meaningful per-user reranking even when many candidates have similar global popularity or text evidence, especially for ambiguous and short-prefix queries.

5.2 Personalization Feature Computation↩︎

At serving time, the affinity score between a user and an item is computed as \[\begin{align} s_t(u, v) &= \mathrm{cos\text{-}sim}\!\left(\bar{\phi}_t(u),\; \phi_t(v)\right) \\ s_{id}(u, v) &= \mathrm{cos\text{-}sim}\!\left(\bar{\phi}_{id}(u),\; \phi_{id}(v)\right), \end{align}\] where \(\phi_t\) and \(\phi_{id}\) are the text and ID embeddings, respectively, and \[\bar{\phi}(u) = \frac{1}{|\mathcal{H}_u|}\sum_{v_i \in \mathcal{H}_u} \phi(v_i), \label{eq:mean95pool}\tag{2}\] is the mean-pooled user embedding over watch history \(\mathcal{H}_u\).

Aggregation strategy. Mean pooling (Eq. 2 ) was chosen as a deliberate baseline because it is simple, compatible with strict serving-latency requirements, and consistent with a user-representation strategy used in prior large-scale industrial recommendation systems [24]. These properties make mean pooling a natural starting point before introducing more complex sequential user models.

Fallback. When a user has no history or has opted out of personalization, \(s_t\) and \(s_{id}\) are set to “missing,” effectively signaling the ranker to disregard these features while preserving model structure for all other users.

5.3 Ranker Architecture and Training↩︎

The ranking model is a gradient-boosted decision tree (GBDT) trained with XGBoost [4] using a pairwise ranking objective: \[\mathrm{Obj} = \sum_{(i,j) \in \mathcal{P}} \log\!\left(1 + e^{-(s_i - s_j)}\right) + \Omega(F)\] where \(\mathcal{P}\) is the set of preference pairs derived from relevance labels, \(s_i\) is the score for item \(i\), and \(\Omega(F)\) is XGBoost’s regularization term (L1 + L2 on leaf weights).

Training data. The ranker is trained on a large, temporally split corpus of labeled \((\mathrm{query}, \mathrm{user}, \mathrm{candidate})\) examples spanning millions of interactions and a held-out validation set, which supports stable generalization estimates. Labels are weighted to prioritize high-intent outcomes, i.e., \(w_{\mathrm{conv}} > w_{\mathrm{click}} > w_{\mathrm{imp}}\), aligning optimization with business impact rather than raw engagement volume. Negative examples are drawn from items in the recall set that received no engagement. The model uses a rich feature set spanning behavioral, lexical, catalog-quality, and personalization signals, including prefix-query traffic.

6 Offline Evaluation↩︎

6.1 Evaluation Protocol↩︎

Offline evaluation is conducted on a strictly temporally held-out test split that follows the training window, preventing label leakage. The evaluation set comprises large-scale search sessions across major device families.

Metrics. We report:

• NDCG@10: Normalized Discounted Cumulative Gain at cut-off 10, where items are ordered by personalized score and ground truth is derived from engagement labels.

• MRR@10: Mean Reciprocal Rank of the first converted item.

• Avg.Converted Position (ACP): Mean rank of the converted item under the model’s scoring.

6.2 Ranker Comparison↩︎

Tables 5 and 6 present the offline evaluation comparing four conditions: (1) the non-personalized baseline (No P13N), (2) the proposed personalized ranker (Ours: TextEmb+IdEmb), (3) a text-only ablation (ranker with TextEmb features only, IdEmb removed), and (4) an ID-only ablation (ranker with IdEmb features only, TextEmb removed).

Key observations.

1. Hybrid is best overall. The Text + ID model achieves the strongest results across ranking quality and converted-item position, outperforming both single-signal ablations.

2. Complementarity of text and collaborative signals. Only Text and Only ID both improve substantially over No P13N, but combining them yields the highest NDCG@10 and MRR@10, indicating complementary signal value.

3. Better conversion placement. Average converted-item position improves from 4.7306 (No P13N) to 3.9802 (Text + ID), moving converted items meaningfully closer to the top of the ranked list.

4. Generalization quality. Train/test losses are lowest for the hybrid model, consistent with stronger ranking quality on held-out data.

Aggregate summary. Relative to No P13N, the hybrid model improves NDCG@10 and MRR@10 while reducing average converted-item position, with gains beyond either single-signal ablation.

6.3 Embedding Quality Comparison↩︎

Table 7 reports standalone retrieval quality: the average rank of the converted item when candidates are ordered by user–item cosine similarity alone, without the downstream ranker. IdEmb marginally outperforms TextEmb on session data (10.81 vs. all-recalled; 1.89 vs. impressed). Table 4, in contrast, shows TextEmb winning on LLM-judged triplet accuracy (89.50% vs.%) and A-P/A-N margin (0.4124 vs.).

This inversion is not a contradiction but the signature of two signals optimized for different notions of similarity, measured by evaluations with different biases. TextEmb’s contrastive objective over metadata pushes it toward text-grounded semantic proximity, which closely mirrors what the LLM is asked to judge—the LLM has no visibility into user behavior and evaluates pairs purely from structured metadata. IdEmb, trained only on co-engagement, encodes behavioral proximity: items that attract the same audience regardless of textual overlap. Session-based retrieval naturally favors this behavioral signal because its scoring target (the next converted item) is itself behavioral. Each evaluation is biased toward the embedding whose training objective most closely matches its scoring criterion.

Qualitatively, IdEmb excels on head and mid-tail content where co-engagement is dense (franchise loyalty, cross-genre audience affinity), while TextEmb generalizes better to long-tail and newly released items where behavioral signal is sparse. The ranker ablation in Table 6 confirms these are complementary rather than substitutable: Text+ID outperforms both single-signal variants on every ranker metric, with non-redundant contributions from each feature. We view the LLM corpus and session-based retrieval as two lenses on embedding quality—neither dominates—and the decision to deploy both models is ultimately justified by downstream ranker performance, not by either evaluation alone.

6.4 History Length Sensitivity↩︎

Figure 5 shows relative improvement as a function of user history length, partitioned into buckets. The figure makes the monotonic relationship visually explicit: richer user history strengthens personalization effectiveness.

Gains increase with history depth. Both NDCG and MRR lifts are positive across all buckets and rise from the 1–5 cohort to the 51–100 cohort, indicating that richer watch histories enable stronger personalization while still yielding measurable improvements for sparse-history users. Notably, baseline relevance is lower for longest-history cohorts (NDCG@10 0.733 → 0.680), so the larger lift suggests personalization adds the most value where the default ranker underperforms.

6.5 Query-Length Sensitivity↩︎

Table 8 compares non-personalized and personalized ranking quality on short-prefix queries (1–3 characters) versus longer queries (>3 characters).

The gains are larger on short-prefix queries than on longer queries, consistent with higher ambiguity in the earliest incremental-search inputs.

7 Online A/B Experiment↩︎

7.1 Experiment Setup↩︎

We conducted a three-week online A/B experiment, with treatment and control randomized at the user level.

Eligibility. Users are eligible for personalization if they have opted in to personalized recommendations.

Statistical testing. Randomization is performed at the user level. We report relative lift with two-sided significance testing with multiple-testing correction across the experiment’s metric suite (Section 7.2) and monitor metric stability daily to guard against transient novelty effects.

7.2 Results↩︎

Both primary online metrics cross their corrected significance thresholds with confidence intervals entirely above zero: Tap-Through Rate improves by +1.14% and Conversion Rate by +1.23%. Average Converted Position also improves by \(-\)​2.91% (lower is better), reported as a descriptive ranking-quality indicator. Because most users in both arms were eligible for personalization, these top-line lifts are averaged over the full user population and likely understate the effect for users with usable watch history.

8 Discussion and Limitations↩︎

Incremental search is the main personalization regime. The concentration of lift on short-prefix queries is the paper’s strongest result: Table 8 shows +8.63% NDCG@10 on 1–3 character queries versus +1.46% on longer queries, consistent with the idea that early prefixes create a low-lexical-information regime where personalization must supply the missing discrimination.

Hybrid embeddings are complementary. Table 6 shows that neither embedding family alone is sufficient: IdEmb is strongest where behavioral signal is dense, while TextEmb broadens catalog coverage and supports long-tail or newly launched content. The LLM-based evaluation provides a second lens on this complementarity by separating semantic similarity from exposure and click bias.

Limitations. Online results are from a single regional deployment; broader generalization should be validated separately. Offline evaluation remains subject to exposure bias; users with sparse history benefit less than richer-history cohorts; and the current mean-pooled user representation does not model temporal dynamics or session intent. Training/serving skew and embedding freshness may further attenuate measured online gains, and this is likely the most important caveat when interpreting offline-to-online fidelity because serving-time history updates and embedding snapshots are not perfectly synchronized with the materialized offline data.

9 Conclusion and Future Work↩︎

We presented a personalization framework for incremental video search and argued that this setting is a distinct retrieval-and-ranking regime: at 1–3 characters, lexical evidence is too weak to separate plausible candidates, and user-specific signals become disproportionately valuable. Our results support this framing, with the largest gains occurring exactly in that short-prefix regime, and they show that the strongest solution is hybrid: text embeddings provide semantic coverage and cold-start robustness, while ID embeddings contribute high-precision behavioral affinity where interaction data are rich. Future work includes sequential and query-aware user models, multimodal item embeddings, and explicit exploration mechanisms.

References↩︎