1 Introduction↩︎

Large-scale financial services companies serve their users through digital platforms, including web and mobile applications, generating massive volumes of clickstream data, such as timestamps, page views, and user actions. These behavioral logs capture what users did, yet not why: the underlying intent remains latent. Moving beyond surface-level interactions to understand user intent would enable recommender systems to deliver more relevant personalized experiences [1]–[3]. In financial services, the user intent in the pre-login setting carries significant weight: unlike e-commerce or social platforms, where the richest behavioral signal is generated post-login, most financial product exploration, comparison, and even application initiation occurs on the public website before authentication, making pre-login browsing one of the highest-leverage signals available. However, two challenges prevent this signal from being used:

• Pre-login context is lost at login: recommender systems treat users as if they have no prior unauthenticated browsing history, discarding the intent they have already expressed on the website. This can happen because identifying users pre-login requires entity resolution methods and connecting this information post-login requires cross-platform event capture to be centralized.

• Existing intent representations tend to be ad-hoc and overly narrow (e.g., simply logging that “a user visited credit card pages”), thereby failing to capture more nuanced behavioral signals. Due to both the regulatory nature of the financial industry and the analytical needs to provide a better user experience, qualitative analysis and explainability are crucial. However, existing approaches lack an explainability-aware intent representation that serves both quantitative tasks (e.g., ranking, recommendation) and these essential qualitative use cases (e.g., segmentation, analytics) at scale.

A semantic intent engine that transforms raw clickstreams into structured intent representations would address these challenges, improving dynamic user segmentation, personalized content ranking, marketing arbitration, and much more. To connect pre-login browsing with post-login experiences, we leverage two straightforward identity signals: within-session linkage via the first-party cookie¹, and cross-device linkage from web to mobile app via the shared login credential. Building on this linkage to address the gap of discarded pre-login signals in financial services, we present a solution with three core contributions that demonstrate the practical usefulness of our approach:

• Dual-purpose intent prediction system. A self-supervised Transformer encodes multi-modal web clickstream into a session embedding, while an LLM-generated intent taxonomy is distilled into a lightweight classifier to provide interpretable intent labels from the same embedding space. This yields one shared representation with two complementary outputs: a dense embedding for quantitative tasks and human-readable labels for qualitative understanding.

• Clustering-enhanced LLM taxonomy generation. A clustering layer over session embeddings enables stratified sampling, improving the iterative taxonomy generation approach [4], which generates and refines an intent taxonomy through successive minibatches of samples, by ensuring the LLM sees the full diversity of user behavior rather than a random subset.

• Embedding-space intent distillation. LLM-generated intent labels are distilled into a lightweight classifier that operates directly on the compact fixed-dimensional vector as input rather than raw text sequences, allowing it to scale to 10M+ daily sessions at inference without LLM calls or separate text encoding.

2 Related Work↩︎

2.1 User Behavior Representation Learning and Sequential Recommendation↩︎

Modeling user behavior sequences is central to modern recommender systems, and methods differ primarily in how sequential dependencies are captured.

Early approaches apply traditional machine learning techniques to study user behavior, such as manual rule mining [5], Markov Chain models [6], and unsupervised clustering [7], [8]. These methods capture only static or short-range patterns and lack the capacity to model long behavior sequences. Recurrent architectures such as GRU4Rec [9] advance beyond these with gated sequential encoders, but are inherently non-parallelizable and suffer from decaying sensitivity to long-range dependencies.

Transformer-based architectures address these limitations with self-attention over behavior sequences. SASRec [10] applies unidirectional self-attention to capture both short- and long-term sequential dependencies. BERT4Rec [11] extends this with bidirectional attention and masked-item prediction to leverage context from both directions. Industry systems scale these to multi-feature settings: BST [12] at Alibaba and TransAct [13] at Pinterest enrich item sequences with action-level signals; TRACE [14], STEP [15], and TIMeSynC [16] extend further to tabular event streams where each event is a heterogeneous row of categorical and numerical attributes. However, these systems fuse raw categorical and numerical features without incorporating enriched semantic representations, such as pretrained text embeddings derived from page content. Additionally, none provides an interpretability layer over the modeled behavior sequences: the resulting representations cannot be inspected, summarized, or grounded in human-readable form.

2.2 LLM-Based Taxonomy Generation↩︎

In recommender systems, intent representations have been used to structure candidate generation [17], model multiple user interests [1], and encode latent intents in sequential models [2]. While these methods encode intent implicitly, an explicit, human-readable taxonomy that organizes intents into structured categories would be a valuable complement when goals extend to behavioral analysis, user segmentation, and directly guiding LLM-based ranking [3].

Pre-LLM taxonomy generation relied on hand-crafted patterns [18], hierarchical topic modeling [19], and embedding-based term clustering [20], but either lacked scalability or failed to capture nuanced semantics. Large language models bridge this gap by producing fluent, interpretable labels over groups of documents, as demonstrated by methods such as TopicGPT [21], GoalEx [22], and ClusterLLM [23]. TnT-LLM [4] shows that iterative minibatch refinement paired with human-in-the-loop validation [24] is needed for large datasets that do not fit into a single prompt. To make LLM-derived taxonomies deployable at production scale, these systems distill LLM-generated labels into lightweight student classifiers [4], [25], consistent with broader evidence that LLM teachers can train compact models at a fraction of the inference cost [26], [27]. However, all prior LLM-based taxonomy systems operate on raw text inputs. Applying iterative LLM-based taxonomy generation and distillation based on the intent embeddings derived from multi-modal clickstream sequences in financial services remains unexplored.

In summary, to our knowledge, prior work addresses user behavioral representation learning (Section 2.1) and LLM-based taxonomy generation and distillation separately, but none combines the two.

3 Method↩︎

3.1 Overview↩︎

Our system produces two complementary outputs from pre-login web clickstream data: (1) a dense session embedding for downstream tasks, and (2) human-readable intent labels for interpretability.

In the first stage (Section 3.2), raw clickstream events are encoded into session embeddings via a self-supervised Transformer. In the second stage (Section 3.3), an LLM-driven pipeline constructs an intent taxonomy from representative sessions and assigns each session to multiple categories. The resulting labels are distilled into a lightweight classifier that consumes the embeddings and produces interpretable intent labels for scalable inference.

3.2 Multi-Modal Session Embedding↩︎

The input to our model is pre-login web clickstream sessions: chronologically ordered sequences of events. Let a session be \(s = (x_1, \ldots, x_T)\) of length \(T\) (the maximum session length, with shorter sessions padded), where each event \(x_i\) carries \(f\) heterogeneous features, including page information, dwell time, event type (click, scroll, view, etc.), and semantically enriched features such as page HTML content. All features are encoded into a common \(d\)-dimensional space.

Discrete features via learned embedding tables. Each discrete feature is embedded via a learned lookup table. A feature with vocabulary size \(V\) has a learned embedding matrix \(E \in \mathbb{R}^{V \times d}\), and its token index is mapped to a \(d\)-dimensional vector by row lookup.

Page content via a pretrained sentence encoder. To inject semantic content from the rendered page rather than just the URL string, we pre-compute a vector for every URL by fetching its HTML, extracting the main text, and encoding it with a pretrained sentence transformer [28] into a \(d_\text{url}\)-dimensional vector. The resulting vector is then projected to \(d\) dimensions through a learned linear layer \(W_\text{url} \in \mathbb{R}^{d \times d_\text{url}}\) with bias \(b_\text{url} \in \mathbb{R}^{d}\).

Sequence position via a learned positional embedding. A learned positional embedding matrix \(P \in \mathbb{R}^{T \times d}\) supplies one \(d\)-dimensional vector per position, where the \(i\)-th row of \(P\) is the embedding for position \(i \in \{1, \ldots, T\}\).

Per event, the feature vectors are concatenated into an \(fd\)-dimensional vector and fused by an \(\text{MLP}: \mathbb{R}^{fd} \to \mathbb{R}^d\), after which the positional embedding is added and a Transformer encoder \(\mathcal{T}\) attends over the full sequence: \[\begin{align} h_i &= \text{MLP}\bigl(e_i^{(1)} \oplus \cdots \oplus e_i^{(f)}\bigr) \in \mathbb{R}^d, \\ z_i &= h_i + P_i \in \mathbb{R}^d, \\ (o_1, \ldots, o_T) &= \mathcal{T}(z_1, \ldots, z_T), \quad o_i \in \mathbb{R}^d, \end{align}\] where \(\oplus\) denotes vector concatenation. Because the complete pre-login session is available at inference time rather than streamed incrementally, the encoder \(\mathcal{T}\) is free to use either unidirectional or bidirectional self-attention, and the final session embedding is obtained by either last-hidden-state or mean pooling over \((o_1, \ldots, o_T)\). The model is trained self-supervised: depending on the attention setting, it predicts the page title and URL of either next event (causal) or masked events (bidirectional) from context. This requires no labeled data and encourages the embedding to capture the semantic content and sequential structure of the session.

Depending on the model and downstream performance (see Section 4.2), the final model is a multi-layer Transformer encoder trained with bidirectional masked language modeling and mean pooling.

3.3 Intent Taxonomy Generation and Distillation↩︎

While the session embeddings capture user behavior in a dense vector, they lack interpretability. To bridge this gap, we construct an intent taxonomy using LLMs and distill it into a lightweight classifier that maps embeddings to human-readable intent labels. The pipeline has three phases: clustering-based sampling, taxonomy generation, and label assignment and distillation.

Clustering-based sampling. We compare K-Means and HDBSCAN for clustering the session embeddings, tuning \(K\) for K-Means and minimum cluster size for HDBSCAN via silhouette score, Davies-Bouldin Index (DBI), and Calinski-Harabasz Index (CHI). We adopt K-Means with an optimal \(K\) based on these metrics. We then sample \(N\) sessions per cluster, yielding \(K \times N\) representative sessions. This ensures the LLM sees the full diversity of user behavior rather than a random or skewed subset. Note that clustering serves only as a stratified sampling mechanism: clusters do not become taxonomy categories since individual sessions, particularly those farther from cluster centroids, may carry multiple intents. Each sampled session’s raw clickstream is formatted as a sequence of [dwell_time] [event_type] page_title : page_url entries for LLM consumption.

Taxonomy generation. Because page metadata changes frequently as content is produced dynamically, manually curating and maintaining an intent taxonomy is impractical; an automated generation approach is needed to keep pace with evolving content while reducing manual overhead. Inspired by TnT-LLM [4], we adopt the iterative minibatch approach to construct a fixed intent taxonomy through three stages: generate, update, and review. Unlike free-form labeling, where an LLM may produce semantically equivalent but inconsistently worded intents, this approach constrains the label set and refines it iteratively as more data is observed.

In the first step, a generation prompt prompts the LLM to take the first minibatch of sessions and produce an initial taxonomy of \(I\) intent categories, each with a name, description, and confidence score. In subsequent steps, an update prompt asks the LLM to refine the taxonomy based on the new minibatches and the confidence scores of existing categories by adding, removing, merging, or splitting categories, while maintaining the target category count. A final review prompt checks for completeness, mutual exclusivity, and clarity. The generate–update–review cycle is repeated for \(E\) epochs over the sampled sessions. With \(K \times N\) sampled sessions, a minibatch size of \(B\), and \(E\) epochs, this requires \(\mathcal{O}(K \cdot N \cdot E / B)\) LLM calls. Each prompt includes background context, task instructions, formatting constraints (e.g., target number of categories, maximum intent label length), along with a minibatch of formatted clickstream sessions. After each taxonomy is produced, we perform a manual human review and refine the prompts as needed, ensuring the final taxonomy reflects domain expertise rather than LLM output alone.

Label assignment and distillation. With the taxonomy fixed, we assign intent labels to each sampled session via zero-shot LLM prompting and then train a distillation model. Following findings that LLMs achieve strong performance on annotation tasks [29], we use LLMs to classify each session into multiple intents from the taxonomy, each with a confidence score. Micro intents (specific sub-goals) and chain-of-thought reasoning are additionally collected to enable human review of the assigned labels, iteratively refining the prompt as needed.

To enable scalable inference, we distill the LLM-generated labels into a lightweight MLP classifier [4], [25]. The student model takes the session embedding as input and predicts intent probabilities independently for each label. We use the LLM’s confidence scores as soft targets via per-label binary cross-entropy, preserving the teacher’s uncertainty and producing better-calibrated predictions. At inference time, the distilled MLP is able to produce intent labels at scale without separate text encoding or LLM calls.

4 Experiments↩︎

4.1 Experimental Setup↩︎

Web Clickstream Data. Our primary data source is internal clickstream data from our company’s financial services website. All activity occurs within a single first-party domain with no cross-site tracking involved. Each session consists of a chronologically ordered sequence of user interaction events, beginning with the user’s first page visit and ending with idle timeout or logout; users may or may not log in during a session. Each event carries timestamps, page information, event type, cookies, device metadata, etc. For our study, we select the pre-login portion of each session that captures user intent before authentication. The full website receives approximately 10M sessions per day. We filter to sessions containing both pre-login activity (e.g., visits to product pages and servicing information) and post-login activity (e.g., account management, payments, product enrollment), yielding approximately 100K sessions per day.

To evaluate the session embeddings and intents on downstream tasks, we join pre-login sessions with post-login signals (mobile homepage tile clicks and product conversions) using a temporal join to link pre-login browsing activity to authenticated outcomes. We select four months of data for our experiments. All train, validation, and test splits are partitioned by time to prevent data leakage.

Mobile Homepage Tile Ranking. The internal mobile home screen presents various product and content tiles (Table 1) after users log in. Each tile is powered by its own recommender model that produces its content, while a global ranking model orders the tiles on the screen. This downstream validation dataset records which tiles each user taps, providing a direct signal of cross-platform post-login intent. We frame this as a multi-label classification task: given a user’s features \(\mathbf{x}\), predict which tiles the user will tap, i.e., \(P(\text{click on tile } t \mid \mathbf{x})\) for each tile \(t\) in the candidate set. The production model is a multi-label classifier that ranks tiles using features derived from post-login behavior (e.g., sequences of page views) and user context, with click-through rates per tile as a simple baseline. None of the existing features capture pre-login signals.

User Conversion. The other validation dataset focuses on user conversions across both mobile and web platforms, specifically tracking which products or features users ultimately apply for or enroll in after pre-login browsing (Table 2). We frame this as a multi-label classification task: given the pre-login intent signals, predict which post-login actions a user will take; multi-label classification is appropriate because a single session may lead to multiple conversions. This task is motivated by a key constraint in financial services: for unauthenticated users, especially prospects who have no prior relationship with the institution, pre-login clickstream activity is the only available signal, as static user features and post-login behavioral data do not yet exist. We therefore evaluate how well session embeddings and distilled intent labels perform in this data-scarce setting.

4.2 Mobile Homepage Tile Ranking Results↩︎

We evaluate three session embedding variants, each added as an additional feature set on top of a baseline model:

1. Causal attention with last-hidden-state pooling;

2. Causal attention with mean pooling;

3. Bidirectional attention with mean pooling.

The baseline model is a LightGBM classifier trained one-vs-rest over tile categories, using click-through rate (CTR) and site retargeting features (SRF) as inputs. Both site retargeting features and session embeddings are derived from pre-login browsing activities: the former provides coarse-grained intent signals by aggregating page visit counts by product category over rolling time windows (9 dimensions), while the latter encode sequential patterns and content-level semantics. The evaluation metrics are macro Recall@K, which measures the fraction of users whose clicked tile appears in the top \(K\) predictions, and Log Loss, which measures the calibration of predicted probabilities. We report macro rather than micro metrics because of the heavy class imbalance in tile clicks: the most-clicked tile is a low-value servicing widget, while less-frequent tiles such as product offers and travel rewards are clicked less but drive more significant revenue.

As shown in Table 3, all three variants improve recall over the baseline, confirming that pre-login session representations carry signal beyond coarse page-visit counts. The bidirectional + mean pooling variant achieves the best overall performance, with a 13.38% Log Loss reduction and 1.88% macro Recall@1 gain. The advantage of bidirectional attention is intuitive: because the complete pre-login session is available at inference time, each event can attend to both preceding and subsequent actions, producing richer contextual representations than the causal setting where each event only sees its past. Similarly, mean pooling outperforms last-hidden-state pooling because the final event in a session can be a low-signal action such as a redirect or bounce; averaging over all event representations avoids the potential last-token bias.

4.3 Intent Taxonomy and Conversion Prediction Results↩︎

Table 4 shows example intents generated by the taxonomy pipeline at two granularity levels: \(I{=}25\) and \(I{=}35\). The iterative process produces semantically coherent categories, each grounded in observed clickstream patterns rather than predefined rules. After human review, we select \(I\)=35 for the following experiments. For clustering-based sampling, we select K-Means with \(K{=}50\) based on silhouette score (0.535), Davies-Bouldin Index (1.478), and Calinski-Harabasz Index (19,250), indicating well-separated clusters suitable for stratified sampling.

To validate LLM label quality, we conduct a manual review on a stratified random sample spanning all intent categories. A human annotator independently assesses whether the LLM-assigned labels correctly capture the session’s browsing intent, rating each as agree, partially agree, or disagree. The strict agreement rate (full agree) is 97.0% and the lenient rate (agree + partially agree) is 99.5%, with the single disagreement involving a low-confidence edge case where no suitable taxonomy label existed. These results confirm that LLM-based label assignment is reliable enough to serve as a teacher signal for distillation without requiring exhaustive human annotation.

We report results using micro metrics for the following experiments. Unlike tile ranking, where rare clicks can drive high revenue and thus necessitate macro metrics, conversion frequency directly correlates with business impact, making frequency-weighted micro metrics the ideal evaluation for high-volume actions like credit card applications. The distilled student MLP achieves micro F1 of 0.822 against LLM-generated labels on held-out test data. This confirms that the lightweight classifier reproduces the teacher’s intent assignments while operating directly on the 64-dimensional embedding.

To validate the utility of our representations, we evaluate four feature sets on the user conversion prediction task described in Section 4.1. All models use the same MLP architecture with binary cross-entropy loss, per-class positive weighting, and early stopping on validation loss, so differences in performance are attributable solely to input features:

1. Site retargeting features (SRF): 9 hand-crafted page-category count features aggregated over rolling time windows.

2. Distilled intents (student): 35-dimensional probability vector from the distilled intent classifier (Section 3.3).

3. LLM intents (teacher): 35-dimensional raw intent confidence vectors produced by the LLM during label assignment (Section 3.3).

4. Session embedding: 64-dimensional embedding (Section 3.2).

Hand-crafted features are insufficient. SRF (A) shows a 35.8% degradation in micro F1 relative to the LLM teacher, confirming that coarse page-category counts over rolling time windows fail to capture the behavioral nuances needed for conversion prediction. This motivates learning richer representations from the raw clickstream.

LLM teacher validates the taxonomy quality. The LLM teacher (C) serves as the reference baseline for intent-based representations. Its 35-dimensional confidence vector, despite being highly sparse, encodes meaningful predictive information derived from the full clickstream context.

Distillation effectively transfers teacher knowledge. The distilled student (B) retains 93% of the LLM teacher’s predictive power, with only a 7.0% relative decrease in micro F1. This validates that knowledge distillation successfully transfers the majority of the teacher’s signal into a lightweight MLP operating directly on the 64-dimensional embedding. The remaining gap reflects the inherent compression from per-session LLM judgments to a fixed classifier learned from a limited labeled sample, a necessary tradeoff for scaling from thousands of LLM calls to millions of MLP inferences (Section 4.4). Notably, combining distilled intents with SRF (A+B) yields +37% relative F1 improvement over SRF alone, confirming that intent labels capture substantially richer signal than count-based features.

Session embedding contains the richest signal. The session embedding (D) achieves the best micro F1, a 4.3% improvement over the LLM teacher, despite being only 64-dimensional. This is expected: the session embedding is the input to the distillation pipeline, and it encodes the full multi-modal clickstream in a dense representation. The distillation pipeline then extracts interpretable intent labels from this same embedding space, necessarily losing some information in exchange for human readability.

Together, these results validate the dual-output design: the session embedding serves quantitative downstream tasks with maximum predictive power, while the distilled intent labels, derived from the same embedding space, add interpretability, segmentation capability, and stakeholder-facing explanations at minimal cost to accuracy.

4.4 Inference Efficiency↩︎

A key motivation for knowledge distillation is production scalability. Assigning intent labels to raw clickstreams via direct LLM inference, either through an API or a self-hosted model, requires generating a full response per session, each incurring 2–4 seconds of latency. For example, at a scale of 100K sessions, this amounts to over 80 hours of serial inference time, which is infeasible for deployment.

By contrast, our pipeline from raw clickstreams to intent labels is extremely fast: the Transformer encodes 100K sessions into embeddings in 60 seconds, and the distilled MLP produces intent labels from those embeddings in 0.009 seconds—totaling under 61 seconds per 100K sessions on average. This represents a throughput improvement of more than 3,000\(\times\) with only a 7% drop in labeling performance, reducing the high cost of large-scale data batch processing and retaining a high-quality production feature without separate text encodings or LLM inference.

5 Conclusion and Future Work↩︎

We presented a scalable system that produces both session intent embeddings and interpretable intent labels from web clickstream data, bridging the gap between raw behavioral signals and semantic understanding of user intent in financial services. The session embeddings outperform baselines on the home screen tile ranking task, demonstrating that they capture richer signals than aggregation-based features. The intent taxonomy provides actionable categories that can be used for user segmentation, personalization, and marketing strategy. The distilled classifier also outperforms the hand-crafted features while producing human-readable intent labels and enabling scalable inference at serving time, though it achieves slightly lower performance than the raw session embeddings, which is an expected tradeoff.

Limitations. We acknowledge that the system is built and evaluated on financial services clickstream data from a single institution; generalizability to other domains and downstream tasks remains to be validated.

Future work. First, we plan to conduct online A/B testing to measure the impact of session embeddings and intent labels on the home tile ranking task. Second, we aim to automate taxonomy refinement over time, enabling the intent catalog to adapt as user behavior and product offerings evolve. Finally, we plan to extend from session-level to user-level intent prediction by aggregating embeddings and intent labels across multiple sessions per user over a longer time period.

References↩︎