1 Introduction↩︎

In modern digital services virtually every user interaction is recorded as a timestamped event. Sequences of such events—purchases, page views, financial transactions—encode rich behavioral patterns commonly referred to as transactional event sequences. Accurate modeling of these sequences is critical for fraud detection, credit scoring, churn prediction, and personalized recommendation [1], [2].

A particularly valuable objective is learning user-level representations: a compact profile that summarizes behavioral history can serve as a powerful prior for multiple downstream tasks without reprocessing the full sequence at inference time. Contrastive Learning for Event Sequences (CoLES) [3] is a prominent self-supervised paradigm for this purpose, constructing positive pairs through sequence augmentations (masking, time jitter, within-window permutation) and optimizing a contrastive objective. The resulting embeddings capture stable user profiles without label supervision. However, CoLES depends on a sequential encoder backbone, in practice an RNN or Transformer, both of which have fundamental limitations for long sequences.

Table 1 summarizes the trade-offs among encoder architectures. RNNs process events strictly sequentially and suffer from vanishing gradients, preventing reliable long-range dependency capture. Transformers overcome this via self-attention but at \(\mathcal{O}(L^2)\) complexity, which is prohibitive for banking sequences averaging 881 events/user (Age dataset, 21M total events) or 156M total events (MBD). State Space Models (SSMs)—specifically Mamba [4]—offer \(\mathcal{O}(L)\) complexity, stable long-range memory via HiPPO initialization [5], and a parallelizable convolutional training form.

Despite these advantages, SSMs remain underexplored in the context of user-centric personalized modeling. This work closes the gap with two hybrid architectures that inject CoLES user embeddings into Mamba: (1) hidden-state initialization—the CoLES embedding is projected into Mamba’s initial hidden state \(\mathbf{h}_0\); and (2) prefix concatenation—the projected embedding is prepended as an extra context token (Figure 1). Both strategies are evaluated with per-pair Optuna [6] hyperparameter tuning, and model decisions are interpreted with discretization-step maps and Integrated Gradients [7].

2 Related Work↩︎

Event sequence modeling. EBES [1] provides a large-scale benchmark showing that GRU-like models [8] dominate on short sequences while long-range architectures become competitive as sequence length grows. Transformer-based encoders such as BST [9] and BERT4Rec [10] achieve strong results on behavioral sequences from e-commerce platforms but incur quadratic attention overhead that is prohibitive for banking sequences with hundreds to thousands of events per user.

LLM-based and hybrid approaches. LLM4ES [11] learns user embeddings by converting event sequences into textual descriptions fed to a language model, demonstrating strong zero-shot transfer to downstream tasks. LATTE [12] reduces inference costs by aligning compact transaction embeddings with LLM-generated behavioral summaries via contrastive training, showing that frozen LLM semantic priors improve downstream quality at lower latency. EAFD [13] uses a pretrained-embedding-guided LLM agent to discover interpretable features complementary to the latent representation, achieving up to \(+5.8\%\) relative gains over state-of-the-art embeddings on transaction benchmarks. FinTRACE [14] proposes a retrieval-first pipeline that converts raw transaction histories into behavioral features stored in a knowledge base, enabling grounded LLM reasoning for financial analytics in low-supervision settings. Despite competitive accuracy, all LLM-based approaches impose substantial inference overhead, motivating lightweight SSM-based alternatives [2].

Contrastive learning for event sequences. CoLES [3] pioneered self-supervised contrastive learning for event sequences. MLEM [15] combines contrastive and generative objectives; Yugay and Zaytsev [16] study joint training strategies, both showing gains over pure contrastive baselines. Our work is complementary: we keep CoLES fixed as a pretrained user-profile module and focus on integrating its embeddings into a more expressive SSM-based downstream encoder.

State Space Models. S4 [17] introduced structured SSMs with near-linear complexity, setting competitive Long Range Arena [18] scores. Mamba [4] added selective state gating via input-dependent \(B_t\), \(C_t\), \(\Delta_t\); Mamba-2 [19] simplified the state matrix to scalar-diagonal form, yielding faster training and improved long-range retention. ReMamba [20] explores selective compression for very long contexts. A comprehensive survey of SSM variants is provided in [21]. At scale, Mamba-based language models match Transformers on most tasks while offering up to \(8\times\) inference speedup [22].

Explainability of Mamba. Ali et al. [23] proved that selective SSMs are mathematically equivalent to linear-attention models, enabling derivation of implicit attention matrices and direct application of gradient-based XAI methods. Yang and Wang [24] proposed discretization-step maps and cross-modal attention weights for Vision Mamba; these maps generalize to any sequential domain and constitute the primary interpretability tool in this work alongside Integrated Gradients [7].

3 Methodology↩︎

3.1 Datasets↩︎

Experiments are conducted on three public benchmarks (Table 2). Age contains real banking transactions labeled with one of four age groups (balanced, 25% each; evaluated by Accuracy). MBD contains banking transactions with four binary product-acquisition labels (severe class imbalance: \(\approx\!0.3\%\) positive per label; evaluated by mean ROC-AUC). Taobao captures e-commerce user behavior; the target is whether a user makes a purchase within 7 days (43/57 class split; evaluated by ROC-AUC).

3.2 Feature Preprocessing↩︎

Numerical features are transformed via \(\log(1+|x|)\!\cdot\!\text{sign}(x)\), normalized with per-sequence \(z\)-score, and projected with a linear layer. Categorical features are mapped to learned embedding tables. Timestamps are encoded as successive differences \(\delta t_i = t_i - t_{i-1}\) and projected analogously to numerical features. All per-feature vectors are concatenated to form the per-event input token \(\mathbf{x}_t \in \mathbb{R}^{d_\text{model}}\). The timestamp encoding strategy (difference, categorical bucketing, or none) is treated as a tunable hyperparameter.

3.3 Baseline Models↩︎

Plain Mamba. Transaction tokens are projected into the model dimension \(d_\text{model}\), processed by a stack of Mamba blocks (each with RMSNorm and residual connection), and the last hidden state is fed to a linear classifier. The initial hidden state \(\mathbf{h}_0\) is the zero vector.

CoLES + linear classifier. CoLES is trained self-supervisedly on the full dataset (train + test union; no label leakage) for 100K gradient steps. User embeddings are extracted and cached. A linear classifier trained on these embeddings isolates the quality of the contrastive representation as a standalone predictor.

3.4 Proposed Approaches↩︎

Figure 1 illustrates both proposed hybrid methods. In both cases CoLES is pretrained independently and its weights are frozen during the downstream phase.

Hidden-state initialization. We replace the zero initial state with a two-step projection of the user embedding: \[\mathbf{h}_0 = W_2\,\sigma(W_1\,\mathbf{e}_\text{CoLES}),\] where \(W_1\!\in\!\mathbb{R}^{d_\text{mid}\times d_\text{coles}}\) and \(W_2\!\in\!\mathbb{R}^{(d_N \cdot L_\text{ssm})\times d_\text{mid}}\) are learnable. This initialization is applied only to the first Mamba layer; subsequent layers receive the preceding layer’s output state. The user profile is thus available as a prior before the first transaction is processed.

Prefix concatenation. The CoLES embedding is projected into the transaction token space and prepended to the input sequence: \[\tilde{\mathbf{X}} = [\,P\,\mathbf{e}_\text{CoLES}\;;\;\mathbf{X}\,],\] where \(P\!\in\!\mathbb{R}^{d_\text{model}\times d_\text{coles}}\). Mamba can learn at each subsequent position how strongly to condition on the user prior in the prefix.

3.5 Hyperparameter Optimization↩︎

For each (architecture, dataset) pair we run Optuna [6] with Tree-structured Parzen Estimation (TPE) for 50 trials. To keep tuning tractable: epochs are capped at 10, sequence lengths are halved, and for MBD a 1M-sequence subsample is used. Key search ranges: learning rate \([10^{-5},\,10^{-1}]\) (log-uniform), Mamba layers \(\{1,2,3\}\), state-space dimension \(\{16,32\}\), timestamp encoding \(\{\text{diff},\text{cat},\text{none}\}\), and categorical/numerical embedding dimensions \([4,64]\). The first 5 trials use random sampling; TPE guides the remaining search. Results are stored in a SQL database for resumable, reproducible tuning.

3.6 Training Protocol↩︎

All models are optimized with AdamW. CoLES pretraining runs on the train\(\cup\)test union for 100K gradient steps with batch sizes in \([32,\,256]\); embeddings are extracted and cached for all users. Plain Mamba and both hybrid models are then trained on the training split with early stopping (patience \(= 3\), max 100 epochs) monitored on the held-out validation metric. For users without a precomputed embedding at inference time, a zero vector is substituted, reducing the hybrid model exactly to plain Mamba and providing a natural cold-start fallback.

4 Results↩︎

4.1 Main Results↩︎

Table 3 reports test-set performance. Both hybrid methods consistently outperform all baselines on all three datasets. The largest absolute gains appear on Age (\(+3.2\) pp for hidden-state initialization over plain Mamba) and MBD (\(+2.6\) pp). On Taobao improvements are smaller (\(+0.7\) pp) but consistent across both methods. CoLES with a linear classifier underperforms plain Mamba on all three datasets, confirming that sequential context provided by Mamba is important and that CoLES embeddings are most effective as a complementary user-level prior rather than a standalone predictor.

4.2 Convergence Analysis↩︎

Learning-curve analysis reveals a second advantage of the hybrid architectures beyond final metric values. Both CoLES-augmented models reach peak validation performance at epochs 2–3, while plain Mamba requires 6 epochs to converge. This accelerated convergence indicates that the user prior injected via CoLES—whether as an initial hidden state or as a prefix token—provides the model with a meaningful starting point that reduces the number of gradient steps needed to learn user-level behavioral patterns from the raw transaction stream. Faster convergence is practically relevant for large-scale industrial deployments where training time and compute costs are critical constraints.

4.3 Hyperparameter Sensitivity↩︎

Tuning across all (model, dataset) pairs reveals stable patterns: (i) optimal learning rates cluster around \(10^{-4}\); higher values cause training instability for all architectures; (ii) three Mamba layers consistently outperform shallower configurations; (iii) categorical and numerical embedding dimensions in \([15,40]\) balance expressiveness and parameter cost; (iv) for long sequences (Age, avg. 881 events), a state-space dimension of 32 outperforms 16, while short-sequence datasets (MBD: 21 events/window) show less sensitivity to this parameter.

4.4 Explainability Analysis↩︎

We apply two complementary XAI techniques to the hidden-state initialization model, sampling 16 test clients per dataset. Figure 2 summarizes the results.

Discretization-step maps. The scalar \(\Delta_t\) controls how strongly the hidden state is updated at position \(t\). On Age and MBD, \(\Delta_t\) lies uniformly in \([0.75,\,1.0]\) across all positions, indicating that the model updates its state equally on every transaction without selective filtering. On Taobao, \(\Delta_t\) exhibits pronounced dips at specific positions: the model learns to suppress low-intent events (item browsing) and amplify high-intent ones (add-to-cart, purchase), reflecting the qualitatively distinct interaction types present in e-commerce data.

Integrated Gradients. IG [7] attributes output logits to input features via path-integral gradients from a zero baseline. On Age, importance is nearly uniform across transaction amount (num_0\(\approx\)​0.345), timestamp (num_1\(\approx\)​0.328), and category (cat_0\(\approx\)​0.328). On MBD, the currency feature (cat_0) dominates with importance 0.154, substantially exceeding all other features, implying that foreign-currency transactions correlate strongly with holding specific bank products. On Taobao, importance is split between item identifier (cat_1\(\approx\)​0.524) and behavior type (cat_0\(\approx\)​0.476).

Taken together, selective event filtering emerges only when transaction types carry qualitatively different behavioral signals (Taobao), while banking datasets elicit broad, uniform reliance on all transaction features—consistent with the less structured nature of financial event streams compared to e-commerce interaction logs.

5 Conclusion↩︎

We presented two hybrid user-centric architectures that integrate frozen CoLES user embeddings into the Mamba SSM via hidden-state initialization and prefix concatenation. Experiments on three diverse transactional datasets demonstrate consistent improvements over standalone Mamba and CoLES baselines, with notably faster convergence (peak validation at epochs 2–3 vs. 6 for plain Mamba) attributable to the user-level prior. Explainability analysis confirms that the models develop selective event filtering on e-commerce data while processing banking sequences more uniformly—findings that are interpretable in terms of domain knowledge about financial vs.behavioral event types.

The moderate magnitude of metric gains suggests that CoLES and Mamba extract partially overlapping information from the same sequence. Contributing factors include: (1) CoLES trains on short random sub-sequences and ignores global event chronology; (2) the linear projection from CoLES space into Mamba’s state space may be insufficiently expressive; (3) separate optimization of CoLES and Mamba misses potential synergies from joint training.

Promising directions for future work include: (i) joint end-to-end training of CoLES and Mamba so the contrastive representation adapts to the SSM encoder’s needs; (ii) replacing the linear projection with a multi-layer or cross-attention module; (iii) evaluation with Mamba-2 [19], which offers improved long-range retention at lower cost; (iv) scaling to sequences with \(L\!>\!1000\) events to fully leverage the \(\mathcal{O}(L)\) complexity advantage over Transformers; and (v) combining CoLES-initialized SSMs with LLM-based pipelines [12], [13] to unite user-level priors with language grounding and interpretable feature discovery.

References↩︎