1 Introduction↩︎

In sequential CTR prediction, the central design question is not only which architecture to use, but at what granularity the target should interact with the user history. At a coarse level, raw-item architectures let the target score history directly, while latent-query architectures mediate this interaction through learned query representations. We refer to this design axis as interaction granularity. Our starting point is that interaction granularity should be treated as an explicit design dimension rather than as a fixed side effect of architecture choice.

At one extreme, DIN [1] attends from the target to each history token. This fine-grained interaction depends on reliable item embeddings, so matching quality degrades on long-tail items with sparse training signal. At the other extreme, latent-query architectures—HyFormer [2], MixFormer [3], OneTrans [4]—build query representations by combining the target with other information. This is more robust across density regimes but blunter: the target’s influence is diluted, and the resulting queries are less target-specific (Figure 1). Existing backbones therefore expose different interaction granularities, but they do not make that granularity easy to design or enrich explicitly once the architecture is fixed.

We address this gap with a staged design view of target–history interaction for latent-query backbones. Stage 1 selects the interaction bottleneck used by the backbone. Stage 2 adds a position-aware reference through a static position prior. Stage 3 adds target-conditioned residuals that recover target-specific control without changing the backbone’s basic computation pattern. Under this view, interaction granularity is not a monolithic architecture decision inside the latent-query family; it can be decomposed into separable design layers that can be analysed and validated incrementally.

We instantiate Stage 2 with Query-Specific Position Bias (QueryPos), a per-query learnable position prior that independently improves latent-query backbones. We instantiate Stage 3 with LENS, a pair of zero-initialised residual modules: a Target-Conditioned Query Gate (TCQG) that controls which latent queries are active for a given candidate, and a Target-Conditioned Position Bias (TCPB) that controls where those active queries retrieve from history. Zero initialisation guarantees that the model starts as the unmodified backbone and can be added incrementally. The primary evidence for this staged view is portability: across four datasets spanning dense to extremely sparse item regimes (\(\sim 10^3\) to \(\sim 1.1\) samples per item), the four HyFormer cells in the staged validation and the eight external portability cells on MixFormer and OneTrans all show positive total-gain point estimates. This result suggests that the same Stage 2 + Stage 3 additions transfer across multiple latent-query backbones, supporting a portable design recipe within this family.

Beyond the modules themselves, we identify a density-dependent mechanism governing how target conditioning should be formed. The optimal condition source shifts from item-only to item\(+\)sequence as item density decreases, with a crossover near \({\sim}50\) samples per item. This reveals that the right interaction granularity is not fixed, but adapts to the data regime.

The contributions are: (1) we use interaction granularity as an organizing perspective for latent-query sequential CTR models and show that target–history interaction can be analysed as a staged design problem rather than treated only as a fixed consequence of backbone choice; (2) we introduce QueryPos, a position prior that provides a stable position-aware reference across latent-query backbones; (3) we introduce LENS, a pair of target-conditioned query-routing modules that restore target-specific control on top of fixed latent-query bottlenecks, and validate them across three backbones and four density regimes; and (4) we identify a density-driven condition-source rule showing when target-only conditioning is sufficient and when sequence context should be added.

2 Background↩︎

2.1 Problem Setup↩︎

Each training sample contains a user, a candidate item, a binary click label, a behaviour sequence, and categorical/dense metadata fields (user attributes, item metadata, context). All experiments use the full-side feature protocol: all available non-sequential fields are enabled by default.

\[\hat{y}_{u,i}=f_{\Theta}\!\left(i,\mathcal{S}_u,\mathcal{X}_{u,i}\right) = P_{\Theta}\!\left(y_{u,i}=1 \mid i,\mathcal{S}_u,\mathcal{X}_{u,i}\right), \label{eq:problem95setup}\tag{1}\] where \(\mathcal{S}_u\) denotes the user behaviour sequence and \(\mathcal{X}_{u,i}\) denotes the non-sequential fields for the user–candidate pair and context.

2.2 Latent-Query Backbones and the HyFormer Reference↩︎

We use HyFormer as the reference latent-query backbone for staged validation and later test portability to MixFormer and OneTrans. HyFormer-style models [2] aggregate behaviour sequences through cross-attention from a small set of latent queries to sequence key/value embeddings. With \(q\) latent queries and sequence length \(L\), the cost is \(O(qL)\) rather than \(O(L^2)\). The basic HyFormer block alternates Query Decoding (cross-attention over the sequence) and Query Boosting (token mixing among queries and optionally non-sequential feature tokens). Queries are initialised by a QueryGen projection from the full-side feature vector.

2.3 Interaction Granularity↩︎

We define interaction granularity as the level at which the candidate modulates history aggregation. This is a design dimension that can be separated from many other backbone choices: an architecture determines an initial interaction bottleneck, but position structure and target-conditioned control need not remain implicit consequences of that bottleneck. In this paper, we only distinguish two coarse regimes: raw-item interaction (item\(\times\)seq) and latent-query interaction (query\(\times\)seq). The table below summarises four representative architectures under this view.

The key limitation is not that these bottlenecks are wrong, but that they are largely fixed by architecture once the backbone is chosen. DIN grants the candidate direct per-item influence. HyFormer, MixFormer, and OneTrans instead mediate the candidate through compressed query representations. Despite their different entry points, these latent-query backbones do not expose dedicated, separable controls for two aspects of candidate-specific behaviour modelling: which query representations should be active for the current candidate, and where those active representations should retrieve from the history. Our staged design in Section 4 treats these missing controls as separable layers rather than as architecture-specific details.

3 Related Work↩︎

3.0.0.1 Sequential and Target-Aware CTR.

Sequential recommendation has progressed through RNN-based models (GRU4Rec [5]), convolutional approaches (Caser [6]), and transformer architectures: SASRec [7] adopted causal self-attention, and BERT4Rec [8] extended this with bidirectional masked-token training. These backbones are target-agnostic at the encoding stage. DIN [1] introduced target-aware attention where the candidate queries each historical interaction. Extensions include DIEN [9] (interest evolution via GRU), DSIN [10] (session structure), and BST [11] (Transformer encoder over the sequence). Lifelong-history approaches scaled this paradigm: MIMN [12], SIM [13], ETA [14], and SDIM [15]. A common thread is that target awareness operates at individual history items. LENS instead operates at the query level, providing target specificity without competing with item-level representations.

3.0.0.2 Latent-Query Architectures and Position Mechanisms.

The query-decoding paradigm compresses variable-length inputs through learned queries. Perceiver [16] demonstrated iterative cross-attention from a latent array; Set Transformer [17] introduced inducing points; Slot Attention [18] applied iterative slot refinement. In recommendation, MIND [19] and ComiRec [20] extract multiple interests but are target-agnostic retrieval encoders. For CTR ranking, HyFormer [2] uses latent queries with cross-attention and token mixing; MixFormer [3] co-scales dense interaction and sequence modelling with per-head target projections; OneTrans [4] tokenizes all inputs into a single Transformer stream. Although these backbones differ in where the candidate enters the computation, the resulting interaction bottleneck is effectively fixed once the architecture is chosen. LENS extends them by adding explicit target-conditioned controls not present in these architectures. On position encoding, approaches have evolved from sinusoidal encodings [21] through relative representations [22], T5 bucketed biases [23], ALiBi [24], RoPE [25], and Transformer-XL [26]. In CTR, position is typically injected via learned embeddings added to sequence tokens [7], [8]. QueryPos differs in being per-query and applied in cross-attention rather than self-attention; TCPB further generates a candidate-specific position residual.

3.0.0.3 Conditional Modulation and Feature Interaction.

Feature-wise modulation includes FiLM [27], Squeeze-and-Excitation [28], GLU variants [29], [30], and HyperNetworks [31]. TCQG resembles SE-Net’s channel recalibration but conditions on the target-item embedding rather than global context, and the gated representations are latent queries that subsequently attend over the behaviour sequence. Zero initialisation preserves the backbone as a strict subcase. Feature interaction methods—Wide&Deep [32], DeepFM [33], DCN [34], [35], AutoInt [36], DLRM [37]—are orthogonal to sequence aggregation. On action types, HSTU [38] models heterogeneous actions in a trillion-parameter generative framework. Our typed-exposure approach is simpler: additive action-type embeddings at the input level, requiring no architectural changes, motivated by the embedding-coverage problem in sparse item-vocabulary settings.

4 Method: LENS as a Staged Design Framework↩︎

We develop our method as a three-stage design framework for target–history interaction. Stage 1 selects the interaction bottleneck used by the backbone. Stage 2 establishes a position-aware reference that is compatible with that bottleneck. Stage 3 adds LENS target-conditioned query routing to restore candidate-specific control without changing the backbone’s basic computation pattern. Figure 2 visualizes this staged decomposition directly: blue for the Stage 1 backbone bottleneck, green for the Stage 2 static position-aware reference, and red for the Stage 3 target-conditioned routing. We instantiate the framework on HyFormer as the reference backbone and evaluate its portability to MixFormer and OneTrans in Section 6.4.

Each stage addresses an orthogonal design dimension: bottleneck selection, static position structure, and target-conditioned modulation. This decomposition lets the gains from different design choices be validated independently rather than conflated inside a single architecture change.

4.1 Stage 1: Selecting the Interaction Bottleneck↩︎

The backbone choice fixes the structural granularity at which target–history interaction occurs. For a query-decoding backbone such as HyFormer, let \(\mathbf{t} \in \mathbb{R}^D\) denote the target item embedding, \(\mathbf{f} \in \mathbb{R}^{F}\) the concatenated full-side feature representation, and \(\mathbf{S} \in \mathbb{R}^{L \times D}\) the behaviour-sequence key/value embeddings. The decoder represents history with \(q\) latent queries: \[\mathbf{Q}_0 = \operatorname{QueryGen}(\mathbf{f}) \in \mathbb{R}^{q \times D}.\] Each layer \(\ell\) cross-attends from the latent queries to the behaviour sequence and applies query-token mixing: \[\begin{align} \mathbf{Q}_\ell &\leftarrow \mathbf{Q}_\ell + \operatorname{CrossAttn}_\ell(\operatorname{LN}(\mathbf{Q}_\ell),\; \mathbf{S},\; \mathbf{S}), \\ \mathbf{Q}_\ell &\leftarrow \mathbf{Q}_\ell + \operatorname{Boosting}_\ell(\operatorname{LN}(\mathbf{Q}_\ell),\; \mathbf{X}_{\mathrm{ns}}). \end{align}\]

HyFormer exposes three optional capacity switches that define the Stage 1 backbone-selection space in Table 2: Seq Pooling Tokens, NS Tokens in Query Boosting, and Per-query FFN. We compare these individual switches, along with the Full Switches combination, and adopt the Seq Pooling Tokens variant as the downstream reference for its minimal cross-dataset variance. Once the backbone is fixed, QueryPos and LENS become portable.

4.2 Stage 2: Position-Aware Reference↩︎

The query decoder is efficient but does not prescribe a position-aware cross-attention design, even though the recency of a behaviour in the history is informative for CTR. QueryPos addresses this gap by injecting a static, candidate-independent position bias into the cross-attention logits.

We evaluate three candidate mechanisms: Global Position Bias, which adds a single learnable recency curve shared across all latent queries; Absolute Position Embedding, which adds a learned position embedding to each sequence token before key/value projection; and Query-Specific Position Bias (QueryPos), which gives each query its own learnable position curve. Part II of Table 2 reports the comparison: Global Pos Bias is simple and competitive, Abs Pos Emb is strong on KuaiRec and TAAC but collapses on KuaiRand, and QueryPos stays within \(0.0021\) AUC of the per-dataset optimum while maintaining the lowest cross-dataset variance. We adopt QueryPos as the position-aware reference.

QueryPos adds a learnable per-query, per-position prior to cross-attention logits: \[\operatorname{AttnLogits}(\mathbf{q}_i, \mathbf{k}_j) = \frac{\mathbf{q}_i \mathbf{k}_j^\top}{\sqrt{d_h}} + B_{\mathrm{pos}}^{(\ell)}[i, j], \label{eq:querypos}\tag{2}\] where \(B_{\mathrm{pos}}^{(\ell)} \in \mathbb{R}^{q \times L_{\max}}\) is a layer-specific learnable bias, zero-initialised. The bias is indexed over right-aligned sequence positions: the last valid index corresponds to the most recent behaviour. Each row of \(B_{\mathrm{pos}}\) is therefore a position-preference curve for one latent query. Unlike a shared ALiBi-style bias [24], each query learns its own curve, reflecting the assumption that different interest dimensions can favour different sequence-index regions: brand preference is often recent, whereas broader category affinity may span a longer prefix.

QueryPos requires only a query/key cross-attention pathway and is therefore backbone-agnostic. We verify in Section 6.4 that adding QueryPos to MixFormer and OneTrans—neither of which includes a dedicated query-specific position component—yields immediate gains without any target conditioning. This shows that static position awareness is a meaningful addition to these backbones, not merely a stepping stone to the target-conditioned modules introduced next.

4.3 Stage 3: Target-Conditioned Query Routing (LENS)↩︎

The backbone with QueryPos produces a position-aware query decoder whose behaviour is identical for every candidate item: the same queries attend with the same position biases regardless of what is being scored. LENS introduces candidate specificity by adding two zero-initialised residual modules conditioned on a single vector \(\mathbf{c}\). By default \(\mathbf{c} = \mathbf{t}\) (target embedding); for sparse target regimes \(\mathbf{c} = [\mathbf{t}; \bar{\mathbf{s}}]\), where \(\bar{\mathbf{s}}\) is the masked mean of the raw sequence-item embeddings (averaging over non-padding positions), computed from the shared embedding table before projection into the key/value pathway.

The Target-Conditioned Query Gate (TCQG) modulates which latent queries are active for the current candidate. The Target-Conditioned Position Bias (TCPB) modulates where those active queries retrieve information from the indexed history. The two modules are symmetric: \(\mathbf{c}\) controls query activation in TCQG and position retrieval in TCPB.

4.3.1 Target-Conditioned Query Gate (TCQG).↩︎

The gate replaces the raw query initialisation with a condition-modulated query tensor: \[\begin{align} \mathbf{Q}_{\mathrm{raw}} &= \operatorname{QueryGen}(\mathbf{f}), \\ \mathbf{g}(\mathbf{c}) &= 2\,\sigma\!\left(\operatorname{reshape}(\mathbf{W}_t \mathbf{c},\; q \times D)\right), \label{eq:tcqg95gate} \\ \mathbf{Q}_0 &= \mathbf{Q}_{\mathrm{raw}} \odot \mathbf{g}(\mathbf{c}), \end{align}\tag{3}\] where \(\mathbf{W}_t \in \mathbb{R}^{qD \times d_c}\) is zero-initialised and \(d_c\) is the condition dimension. At initialisation, \(\mathbf{W}_t \mathbf{c} = \mathbf{0}\) and \(\mathbf{g}(\mathbf{c}) = 2\sigma(\mathbf{0}) = \mathbf{1}\), so the model reduces exactly to the QueryPos reference. During training, the condition learns to amplify or suppress individual query dimensions, producing a candidate-specific query-activation profile. The factor of \(2\) is a centring convention: a plain sigmoid starts the gate at \(0.5\) and shrinks the query magnitude, whereas \(2\sigma(\cdot)\) preserves the reference scale.

Unlike DIN, which achieves target specificity at the item level by attending from the target to each behaviour item, TCQG operates at the query level: it modulates which abstract interest dimensions are active before the sequence is aggregated.

4.3.2 Target-Conditioned Position Bias (TCPB).↩︎

QueryPos is query-specific but static: after training, the same position curve is used for every candidate item. TCPB makes this position-bias residual a function of the target condition. For each layer \(\ell\), we project \(\mathbf{c}\) into a low-rank query-position mixture: \[\begin{align} \mathbf{M}^{(\ell)}(\mathbf{c}) &= \operatorname{reshape}(\mathbf{W}_{\tau}^{(\ell)}\mathbf{c},\; q \times r), \tag{4}\\ \mathbf{P}^{(\ell)} &= \operatorname{PosEmb}^{(\ell)}(1{:}L) \in \mathbb{R}^{L \times r}, \tag{5}\\ B_{\mathrm{tc}}^{(\ell)}(\mathbf{c})[i,j] &= \langle \mathbf{M}^{(\ell)}(\mathbf{c})_{i,:},\; \mathbf{P}^{(\ell)}_{j,:} \rangle. \tag{6} \end{align}\] The resulting \(B_{\mathrm{tc}}^{(\ell)}(\mathbf{c}) \in \mathbb{R}^{q \times L}\) is added to the cross-attention logits alongside the static QueryPos bias: \[\operatorname{AttnLogits}(\mathbf{q}_i, \mathbf{k}_j) = \frac{\mathbf{q}_i \mathbf{k}_j^\top}{\sqrt{d_h}} + B_{\mathrm{pos}}^{(\ell)}[i,j] + B_{\mathrm{tc}}^{(\ell)}(\mathbf{c})[i,j]. \label{eq:full95logits}\tag{7}\] We use rank \(r=8\). Each layer’s condition projection \(\mathbf{W}_{\tau}^{(\ell)}\) is zero-initialised, so \(B_{\mathrm{tc}}^{(\ell)}(\mathbf{c}) = 0\) at initialisation and the model again reduces to the QueryPos reference. The condition can therefore steer a latent query toward recent-index, mid-history, or longer-range positions on a per-candidate basis, while the static QueryPos bias provides the shared positional scaffold.

4.3.3 Combined Effect, Initialisation, and Condition Source.↩︎

TCQG selects which queries are active; TCPB steers where those queries read. All LENS projections are zero-initialised, so the model starts from the QueryPos reference and is compatible with warm-starting from a pre-trained checkpoint.

The choice of condition source \(\mathbf{c}\) is determined by the data regime rather than by model capacity. When the number of training samples per item exceeds approximately 50, the target ID embedding has received enough direct gradient supervision to act as a stable control signal, and \(\mathbf{c} = \mathbf{t}\) suffices. When samples per item falls below this threshold, the target ID embedding is under-trained and we concatenate the masked sequence mean: \(\mathbf{c} = [\mathbf{t};\, \bar{\mathbf{s}}]\). The architecture is unchanged across regimes; only the condition source switches. In our evaluation suite, the crossover falls between TaobaoAd (\({\sim}59\) samples/item, where \(\mathbf{c}{=}\mathbf{t}\) is selected and performs on par with item\(+\)sequence conditioning) and TAAC (\({\sim}22\) samples/item, where \(\mathbf{c}{=}[\mathbf{t};\bar{\mathbf{s}}]\) is optimal).

4.4 Parameter and Compute Cost↩︎

The combined QueryPos and LENS additions are negligible relative to the model’s embedding tables: the total of \({\sim}90\)K parameters amounts to less than 2% of a typical embedding table budget (5–50M). The per-component breakdown is given in Appendix Table 5. The compute overhead is similarly minimal: one matrix multiply of size \(d_c \times qD\) for the gate, a small per-layer projection for TCPB, and \(q \times L\) scalar additions to existing attention logits per layer. Neither module introduces new activation functions, normalisation layers, or sequential dependencies beyond what already exists in the cross-attention pathway.

5 Experimental Setup↩︎

5.1 Datasets↩︎

We evaluate on four datasets that span different density regimes, summarised in Table 1. The bolded samples-per-item row is the primary axis for the analysis. It spans nearly three orders of magnitude across the four datasets (\(\sim\!1000 \to \sim\!1.1\)) and serves as a proxy for target-ID embedding maturity. KuaiRec provides enough samples per item to learn a reliable target embedding, while KuaiRand does not; this contrast motivates the condition-source rule used by LENS.

3pt

KuaiRec [39] has \(\sim\!1000\) samples per item, so item embeddings are densely trained and target-history matching is reliable. We use positive-only histories capped at 200 items together with the full-side feature scope. The dataset is used as the dense reference for the robustness analysis because DIN’s native sequence path is already strong here.

TaobaoAd [40] consists of 21.9M training samples with dual behaviour sequences and medium item density, and serves as a transition-zone calibration benchmark.

TAAC [41] is a sparse-vocabulary industrial dataset from the Tencent Advertising Algorithm Competition 2025, with \(\sim\!22\) training samples per item. We use the time-split protocol with sequence length 100. In this regime item embeddings alone are unreliable, and non-sequential metadata becomes useful.

KuaiRand [42] is an exposure-log dataset with \(\sim\!4.4\)M items and \(\sim\!1.1\) training samples per item. Its histories contain exposed-but-skipped items, clicks, strong engagement, and negative feedback. We adopt typed exposure histories with action-type embeddings as the default protocol; click-only filtering would discard many long-tail exposure tokens that are needed as sequence-side coverage for later target items. The standard-log train split and the random-log evaluation split have different positive rates, which reflects the exposure-log selection bias analysed in Section 7.1.

5.2 Models and Controls↩︎

We compare four model families: DIN [1] as a target-aware attention baseline, MixFormer [3] and OneTrans [4] as unified sequential/non-sequential Transformer baselines, a selected HyFormer query-decoding reference, and the proposed QueryPos\(+\)LENS extensions applied to the HyFormer backbone. For each family, configurations are selected on seed 42 and then frozen for the three-seed confirmation; Appendix 10.1 summarises the selected settings. HyFormer contains four optional switches that are ablated in Part I of Table 2; we fix the Seq Pooling Tokens variant as the downstream HyFormer reference. All comparisons share the input protocol and use matched or selected capacity for each backbone. We also report two input scopes: the controlled scope uses only item IDs and sequence features, while the full-side scope adds all non-sequential metadata. The pair of scopes is used in Section 7.1 to measure each model’s sensitivity to non-sequential features.

For the cross-backbone portability study (Section 6.4), we port both QueryPos and LENS to MixFormer and OneTrans, the two external baselines from Part I. Neither backbone includes a dedicated query-specific position component, so QueryPos is the first such component they receive. We fix all backbone hyperparameters at the values used in Part I and apply the same density-driven condition-source rule.

5.3 Protocol↩︎

All results use AUC as the primary metric. Main comparison tables report three-seed summaries (seeds 42, 123, 456). We use a two-phase protocol: Phase 1 explores configurations with seed 42; Phase 2 confirms selected comparisons over three seeds.

The LENS architecture is fixed as TCQG plus TCPB. The only variable is the condition source, which follows a pre-specified rule based on samples per item (Table 1): the target item embedding is used in dense and transition regimes, and an item\(+\)sequence condition (the target embedding concatenated with the masked sequence mean) is used in sparse regimes. Part III of Table 2 reports the condition-source ablation behind this rule.

5.4 Implementation Details↩︎

All models are implemented in PyTorch and trained on NVIDIA H20 GPUs with Adam optimizer (learning rate \(10^{-3}\)). Batch size is 4096 unless memory requires otherwise. Each baseline uses its own best-performing hyperparameters (layer count, hidden size, head count, etc.) rather than a shared configuration; Appendix 10.1 lists the selected settings per backbone per dataset.

DIN uses the standard target-attention architecture with matched embedding dimension and a 3-layer MLP head; the “full-side” variant concatenates all non-sequential feature embeddings with the attention output before the MLP, creating the competing feature path analysed in Section 7.1. HyFormer uses 4 Transformer layers with 4 attention heads and a 3-layer MLP prediction head (\([256, 128, 1]\), ReLU); the Seq Pooling Tokens variant is selected as the reference (Section 6.1). MixFormer uses the selected capacity per dataset with target-in-dense enabled. OneTrans uses the bidirectional/no-causal-mask variant. On TaobaoAd, both MixFormer and OneTrans consume the primary behaviour sequence and target only; we treat these rows as conservative baselines.

5.4.0.1 KuaiRand typed-exposure protocol.

Action types are mapped to 4 categories (exposed/non-click, click/valid-play, strong engagement, hate) and embedded with a learnable action embedding table of dimension \(D\). The action embedding is added to the item embedding before entering the sequence encoder: \(\mathbf{e}_{\mathrm{token}} = \mathbf{e}_{\mathrm{item}} + \mathbf{e}_{\mathrm{action}}\).

6 Results and Analysis↩︎

Table 2 presents the staged validation results across all four datasets, organised into five parts. Part I validates Stage 1 by selecting the backbone reference. Part II validates Stage 2 by establishing a position-aware reference. Parts III–V validate Stage 3 by adding LENS target-conditioned query routing, selecting the condition source, and attributing the gains. We analyse each stage in turn below.

3.5pt

6.1 Stage 1: Backbone Reference Selection↩︎

Part I of Table 2 shows that no single HyFormer switch configuration dominates all four datasets. The absolute spread among variants is modest (at most 0.0075 AUC on KuaiRec). We select HyFormer + Seq Pooling Tokens as the backbone reference: its mean gap to the per-dataset optimum is only 0.0026 AUC, with the lowest worst-case deficit and the lowest cross-dataset variance (std \(\leq 0.0037\) on all benchmarks).

Figure 3 shows that all latent-query models exhibit increasing advantage over DIN as item density decreases. On dense KuaiRec, DIN remains competitive; the benefit of richer cross-attention emerges in sparser regimes where item embeddings are unreliable.

6.2 Stage 2: Position Mechanism Validation↩︎

Part II compares three static position mechanisms on the backbone reference. All three provide substantial gains over the unpositioned backbone (compare to Part I), confirming that explicit position structure is a first-order contributor in latent-query architectures. Among the three, no mechanism uniformly dominates. Abs Pos Emb achieves the highest AUC on KuaiRec and TAAC but drops substantially on KuaiRand (std 0.0039). Global Pos Bias is competitive on KuaiRand but has higher variance (0.0055). Query-Specific Pos Bias (QueryPos) trails the per-dataset optimum by at most 0.0021 AUC on every benchmark while maintaining the lowest cross-dataset variance. We adopt QueryPos as the position-aware reference.

6.3 Stage 3: Target-Conditioned Query Routing↩︎

Parts III–V of Table 2 evaluate LENS target-conditioned query routing atop the QueryPos reference.

6.3.0.1 Part III: Condition source selection.

The condition source determines what information is injected into the position bias and query gate. Part III compares two sources—item-only (the target item embedding) and item+sequence (the target embedding concatenated with the masked sequence mean)—and reports the density-driven selection rule. On KuaiRec (\(\sim\)​1000 samples/item), the item-only condition exceeds item+sequence; on TaobaoAd (\(\sim\)​59), the two are tied and item-only is selected under the rule, yielding the superscript ⁱ. On TAAC (\(\sim\)​22) and KuaiRand (\(\sim\)​1.1), item+sequence provides the gain, yielding ^i+s. The transition boundary therefore falls near \(\sim\)​50 samples/item, which aligns with the intuition that reliable item embeddings remove the need for additional sequence context in the conditioning signal.

Figure 4 plots the condition-source AUC gain (item+sequence minus item-only) against item density. The crossover near \(\sim\)​50 samples/item separates datasets where the enriched condition helps (TAAC, KuaiRand) from those where it is neutral or slightly harmful (KuaiRec, TaobaoAd), with TaobaoAd lying near the boundary. This empirical boundary motivates the pre-specified rule used throughout the paper: apply item-only conditioning when samples/item exceeds \(\sim\)​50, and item+sequence conditioning otherwise.

6.3.0.2 Part IV: Module attribution.

Part IV decomposes the full LENS system into its two components—Target-Conditioned Position Bias (TCPB) and Target-Conditioned Query Gate (TCQG)—by removing each in turn. On TaobaoAd and KuaiRand, the system without TCQG (i.e., TCPB alone) slightly exceeds the full model: 0.6409 and 0.6952 respectively, versus 0.6403 and 0.6945 for Full LENS. This suggests that TCPB captures most of the Stage 3 gain in medium-to-sparse density regimes, where position-dependent target modulation provides the primary benefit and the additional gate contributes less. Conversely, on KuaiRec and TAAC, removing either module degrades performance relative to Full LENS (0.8388 and 0.7575 without TCPB; 0.8383 and 0.7565 without TCQG), indicating both modules contribute. The complementary pattern confirms that TCPB and TCQG address different aspects of the target-conditioned computation: TCPB modulates where each query retrieves history, while TCQG modulates which queries are active for a given candidate.

6.3.0.3 Part V: Delta summary.

The three delta rows quantify cumulative gains at different reference points. The module delta (\(\Delta\) modules over QueryPos base) isolates the value of LENS modules alone, ranging from +0.0011 on TaobaoAd to +0.0045 on KuaiRand. The total LENS delta versus the HyFormer reference (without position or conditioning) captures the joint QueryPos\(+\)LENS contribution: +0.0090 on KuaiRec, +0.0009 on TaobaoAd, +0.0036 on TAAC, and +0.0266 on KuaiRand. Finally, the delta versus DIN demonstrates the end-to-end advantage of the full QueryPos\(+\)LENS system over a conventional target-attention baseline, reaching +0.0434 AUC on KuaiRand—the sparsest regime where latent-query modelling with target conditioning yields the largest benefit.

Figure 5 provides qualitative evidence for the conditioning mechanism. On TAAC, the static QueryPos bias produces a position-dependent but target-independent attention pattern. After target conditioning, the heatmap shows clear redistribution of attention mass: for one target category the model concentrates on recent positions (80–100), while for a different target the attention spreads across mid-sequence positions (40–70). This qualitative shift confirms that LENS enables the model to adaptively route attention in a target-specific manner, which the static position bias alone cannot achieve.

6.4 Cross-Backbone Portability↩︎

The portability results are the primary evidence that the proposed additions transfer within the latent-query family. If the same Stage 2 and Stage 3 additions improve multiple latent-query backbones, then the design is better understood as a portable recipe than as a single-backbone variant. The QueryPos+LENS design is therefore tested beyond HyFormer. Both components require only that the backbone exposes a query/key or cross-attention pathway where a position bias and a target-conditioned routing module can be injected. We port both QueryPos and LENS to MixFormer [3] and OneTrans [4], the two external full-strength baselines from Part I. Neither backbone includes a dedicated query-specific position component: MixFormer uses raw content attention in its cross-attention blocks, and OneTrans relies solely on self-attention over a concatenated token stream without position embeddings.

4pt

Table 3 reports the per-component decomposition across all eight backbone\(\times\)dataset cells. All eight cells achieve positive total-gain point estimates (\(\Delta\) Total). The largest gain appears on OneTrans\(\times\)KuaiRec (\(+0.0139\)), where QueryPos and LENS contribute nearly equally (\(+0.0070\) and \(+0.0069\)). This balanced decomposition reflects OneTrans’s lack of native position encoding: both the static position prior and target-conditioned modulation address genuine architectural gaps.

On TAAC, a consistent pattern appears across both backbones: QueryPos contributes near-zero deltas (\(-0.0003\) for MixFormer, \(-0.0002\) for OneTrans), while LENS accounts for the positive total-gain point estimates (\(+0.0010\) and \(+0.0014\)). This is consistent with TAAC’s shorter history truncation (\(L{=}100\)): when sequences are shorter, explicit position structure adds less marginal information, but target conditioning remains valuable for routing attention based on the candidate.

MixFormer shows a similar QueryPos-weak, LENS-strong pattern on KuaiRand and TAAC. This reflects MixFormer’s per-head dense architecture, where the target embedding is already integrated into every attention head through the initial projection [3]. The pre-existing target-content coupling limits the marginal value of an additive position prior, but does not subsume the benefit of explicit target-conditioned modulation. On KuaiRec, where item embeddings are well-trained, QueryPos does contribute meaningfully (\(+0.0050\)), consistent with position structure being more informative when content embeddings are reliable.

The practical implication is that QueryPos and LENS function as a modular design recipe: apply QueryPos as a low-risk position baseline, then optionally layer LENS with the density-driven condition-source rule. Both are additive in parameter cost (\(<\)​2% of embedding tables) and preserve the pre-existing attention pattern at training start through zero-initialised residual connections.

Taken together, Table 3 and the HyFormer staged analysis support the same conclusion: interaction granularity is not exhausted by selecting a single reference backbone. The positive total-gain point estimates in all twelve backbone\(\times\)dataset cells show that the Stage 2 and Stage 3 additions transfer across multiple latent-query backbones, which is why we view portability as primary evidence that the design is not tied to one architecture.

7 Supporting Diagnostics↩︎

The preceding sections present the main LENS architecture result. This section collects two diagnostics that are relevant for deploying sequential CTR models but do not themselves belong to the LENS architecture. The first concerns non-sequential features in DIN-like models on dense item vocabularies, where they can create a shortcut that competes with the sequence-attention path. The second concerns history protocols on sparse-vocabulary logs such as KuaiRand, where positive-only filtering removes the exposure tokens needed for embedding coverage.

7.1 More Features Are Not Always Better for DIN-like Models↩︎

Figure 6 gives a simple diagnostic on whether additional feature paths help DIN-like target-attention models. On KuaiRec, DIN’s native target–history path is already strong because the item vocabulary is dense (\(\sim\)​1000 samples per item) and the model can learn reliable item-to-history matching from the sequence alone. Adding the two-ID sparse path lowers AUC from \(0.8382\) to \(0.8326\), and adding full-side metadata lowers it further to \(0.8281\). The extra features do not act purely as information; they introduce an MLP path that can memorise coarse user–item co-occurrence and weakens the pressure to use the sequence-attention path.

TAAC reverses the situation. With \(\sim\)​22 samples per item, DIN’s native path is much weaker: native DIN reaches \(0.7162\) and the two-ID sparse path is statistically tied. Full-side metadata raises AUC to \(0.7480\) with low variance. Metadata is therefore informative when item embeddings are sparse and shared side fields generalise across items, and harmful when item embeddings are already mature and the sequence path is reliable.

Table 4 asks whether query-decoding models share the full-side shortcut behaviour on KuaiRec. DIN drops by \(-0.0101\) AUC from the controlled to the full-side input, HyFormer drops by \(-0.0010\), and LENS without TCPB reaches the same rounded mean AUC in both settings. We isolate TCQG in this probe because Figure 6 diagnoses a feature-path shortcut: the question is whether target-conditioned query filtering prevents non-sequential metadata from overwhelming the sequence path. TCPB is a position-retrieval module and would add a second, position-level conditioning effect orthogonal to the diagnostic.

Among the three rows, the gate-only LENS variant attains the highest AUC in both settings (\(0.8388\) controlled, \(0.8388\) full-side), consistent with TCQG acting as a relevance filter on the metadata path. The corresponding numbers for Full LENS are reported in Appendix 10.2; they are not used as the main robustness claim because, on dense KuaiRec, TCPB strengthens the mature target-ID conditioning path and increases sensitivity to the controlled item-ID-only setting, conflating the two effects.

HyFormer also degrades, although less than DIN. Its non-sequential features enter through QueryGen and mix into the query initialisation; without a target-specific filter, the sequence-derived query representation is diluted by metadata that may be irrelevant to the current candidate. The mean-pooled sequence injection in QueryGen reduces but does not remove this dilution, since it cannot suppress individual metadata dimensions per candidate. TCQG addresses the dilution directly: it conditions the query initialisation on the target embedding before any sequence aggregation, leaving metadata dimensions aligned with the candidate amplified and unrelated dimensions damped toward the initialisation value. This is consistent with the ordering in Table 4: DIN has no filter (\(-0.0101\)), HyFormer has partial mitigation through sequence injection (\(-0.0010\)), and TCQG provides explicit target-conditioned filtering (\(0.0000\)).

7.2 For Long-Tail Items, Do Not Use Positive-Only Histories↩︎

Figure 7 compares four KuaiRand history protocols. Click-only histories are substantially worse than all-exposure histories: LENS falls from \(0.6736\) AUC with all exposures to \(0.6126\) with click-only, and DIN and HyFormer show the same direction. The drop is large enough to be consistent with a protocol-level coverage problem rather than a modelling fluctuation.

A plausible dominant factor is embedding coverage. KuaiRand has \(\sim\)​4.4M items and \(\sim\)​1.1 training samples per item. Restricting the history to clicked positives removes most long-tail items from the sequence side during training, leaving their embeddings without sequence-side gradient. Those items still appear as negative targets in evaluation, but the model cannot score them against latent queries because the corresponding key embeddings are essentially untrained. All-exposure histories recover much of the lost AUC by retaining displayed-but-skipped items as sequence tokens.

All-exposure histories conflate behaviour, however: skipped and clicked items carry identical representations, so the sequence cannot expose interest signals. Typed exposure histories combine the two requirements. They keep all exposure tokens, so embedding coverage is preserved, and they add a lightweight action-type embedding so that exposed-but-skipped tokens are distinguishable from positively engaged ones. Under this protocol LENS reaches \(0.6971\), well above HyFormer (\(0.6583\)) and DIN (\(0.6509\)). The gap over HyFormer (\(+0.0388\)) is the largest among our four datasets, consistent with target-conditioned query and position modulation being most useful when histories contain heterogeneous behaviours.

The fusion comparison is also informative. LENS prefers additive fusion (\(e_{\mathrm{item}}+e_{\mathrm{action}}\)), while DIN benefits from concat-MLP fusion. A plausible explanation is that query decoders rely on a stable token space in which action type modulates the item embedding smoothly, whereas DIN’s item-level attention can absorb a richer per-token representation; the concat-MLP projection produces a token space that the query decoder does not fully recover.

8 Discussion↩︎

8.0.0.1 Evidence for separable design layers.

A useful way to interpret the experimental results is to view target–history interaction in latent-query sequential CTR as a stack of design layers rather than as a single architecture choice. In our experiments, the backbone bottleneck explains the largest variance, QueryPos adds a static position prior orthogonal to that bottleneck, and LENS adds candidate-specific modulation on top. The gains from Stage 2 and Stage 3 are approximately additive across the portability cells, which is why we interpret the results as evidence for a portable design recipe rather than for a single architecture-specific trick.

8.0.0.2 QueryPos as a portable position prior.

The latent-query CTR backbones evaluated here do not explicitly model query-specific position preferences. Adding a per-query learnable position bias yields consistent improvements, especially on datasets with longer sequences. QueryPos adds negligible parameters (\(N_L \cdot q \cdot L_{\max}\), typically \(<\)​10K) and requires only a single additive term in the attention logits, making it a natural Stage 2 baseline before any target conditioning is introduced.

8.0.0.3 Stage 3 gains are portable even when their balance shifts.

Across both MixFormer and OneTrans, the Stage 3 gain from LENS remains positive in every reported cell, and on MixFormer it is often larger than the Stage 2 gain from QueryPos. MixFormer’s per-head projection already incorporates the target into every dense head [3], limiting the marginal value of an additional query gate. None of the evaluated backbones natively supports target-conditioned position retrieval, so the Stage 3 residual still fills a gap within the tested latent-query family. This helps explain why the target-conditioned gain remains portable even when the backbone’s native target pathway changes.

8.0.0.4 Density-driven condition source.

The crossover near \({\sim}50\) samples per item separates item-only from item\(+\)sequence conditioning. Above this threshold the target-ID embedding is reliable; below it the masked sequence mean supplements the undertrained embedding. We do not claim that this boundary is universal, but our experiments suggest that samples-per-item is a useful one-dimensional proxy for deciding how much sequence context should enter the conditioning signal.

8.0.0.5 Implications for future architecture design.

The staged view is useful beyond the particular LENS instantiation studied here, especially for latent-query CTR backbones. Rather than treating interaction granularity as a fixed architectural side effect, these models can choose a bottleneck, add a position-aware reference, and then evaluate target-conditioned modulation as a separate layer. More broadly, the staged analysis offers a way to evaluate new components in isolation instead of attributing all gains to a monolithic architecture revision.

8.0.0.6 Practical guidelines.

A practical deployment recipe follows directly from the stages. First, tune backbone capacity switches and select the interaction bottleneck. Second, add QueryPos as a low-risk position prior; it helps most on longer sequences (\(L \geq 200\)) and is often neutral rather than harmful on shorter ones. Third, optionally layer LENS with zero initialisation and the density-driven condition-source rule. On MixFormer-like architectures, TCPB alone may already capture most of the Stage 3 gain. We nevertheless treat the full LENS system as the default portable recipe: TCQG remains useful on dense regimes and underlies the target-conditioned filtering effect highlighted by the feature-path diagnostic in Section 7.1. For sparse-vocabulary exposure logs, retain all exposure tokens with action-type embeddings rather than filtering to positives.

9 Conclusion↩︎

We showed that interaction granularity—the level at which the candidate modulates history aggregation—is a designable, separable dimension within latent-query sequential CTR backbones rather than a fixed consequence of backbone choice. Within this staged view, QueryPos provides a portable position-aware reference and LENS provides target-conditioned residuals that restore candidate-specific control through query activation and position retrieval. Across HyFormer, MixFormer, and OneTrans on four datasets spanning \({\sim}10^3\) to \({\sim}1.1\) samples per item, all twelve backbone\(\times\)dataset cells show positive total-gain point estimates, supporting portability within the latent-query family as primary evidence that the design is not tied to one architecture. We further identify a density-driven condition-source rule—item-only above \({\sim}50\) samples per item, item\(+\)sequence below—and two practical diagnostics on feature-path shortcuts and typed-exposure histories. Each added component costs \(<\)​2% of embedding tables, making the approach practical for incremental adoption.

10 Extended Results and Configurations↩︎

This appendix collects baseline configurations, parameter cost details, supporting visualisations, and per-seed details for the DIN feature-path diagnostic.

10.1 Baseline Configuration Summary↩︎

Table 6 summarises the per-dataset configurations used for the four backbones in the main comparison. The settings are selected during seed 42 exploration under the current protocol and then fixed before three-seed confirmation. “No causal mask” denotes the bidirectional OneTrans variant, selected because the causal-mask setting was worse in offline CTR evaluation. TaobaoAd MixFormer and OneTrans currently use the primary behaviour sequence and target only. Complete configuration files and training records will be released as supplementary artifacts.

4pt

10.2 KuaiRec Full-LENS Controlled vs Full-side Diagnostic↩︎

Table 7 contrasts Full LENS under two input scopes on dense KuaiRec: controlled uses sequence and target only (no non-sequential side fields), while full-side matches the main-table configuration. Full LENS is the strongest model under both scopes, but its full-side AUC is \(0.0055\) lower than the controlled scope. The interpretation is that on a dense item vocabulary the controlled inputs already saturate the target-ID and sequence signals; injecting additional metadata through QueryGen can dilute the TCPB-conditioned position retrieval. This is consistent with the gate-only feature-path diagnostic in Section 7.1, where DIN exhibits the same density-regime behaviour more sharply. The same TCPB module remains beneficial under the main full-side LENS on the other three datasets, where the target-ID is less reliable on its own.

10.3 LENS Component Attribution Visualization↩︎

Figure 8 visualizes Part IV of Table 2 as per-dataset bar charts. Starting from the HyFormer+QueryPos baseline, adding either TCQG or TCPB alone produces partial gains; combining them matches or recovers the best variant. TCPB is the dominant contributor on TaobaoAd and KuaiRand, while Full LENS is strongest on KuaiRec and TAAC.

10.4 Full Position-Bias Visualization↩︎

Figure 9 shows the full learned Query-Specific Position Bias matrices on KuaiRec and TAAC. The two datasets behave differently. On KuaiRec, the 12 latent queries largely converge to a shared recent-position preference, so a single shared position curve already captures most of the position-aware signal. On TAAC, the 16 queries differentiate into more diverse temporal profiles, with some queries concentrated on recent positions and others retaining a broader historical window. This contrast is why the main text keeps only the TAAC view: the heterogeneity of TAAC queries is what makes target-conditioned query activation a meaningful complement to a static position prior.

10.5 DIN Feature-Path Per-Seed Details↩︎

This subsection expands the feature-path analysis from Section 7.1. The key diagnostic on KuaiRec is that DIN recovers its native accuracy when the sparse branch parameters are frozen at zero, but drops when the branch is trainable; the sparse-feature MLP branch creates a competing gradient path that degrades the native target-attention mechanism. Table 8 reports the per-seed AUCs underlying this conclusion: native DIN is the best variant on every seed, and adding full-side metadata is consistently worst. On TAAC, by contrast, the two-ID sparse path alone does not help (native and +two-ID are tied at \(\sim\)​0.716), but full-side metadata improves DIN to \(0.748\). The feature-path effect is therefore density-regime dependent rather than architecture-specific.

11 HyFormer+ Switch Ablation: Per-Seed Details↩︎

This appendix provides per-seed values supporting the HyFormer switch ablation in Part I of Table 2. The three-seed mean and standard deviation already appear in the main table; the values below are the underlying per-seed AUCs.

References↩︎