1 Introduction↩︎

Fraud detection in digital banking has evolved from rule-based systems to machine learning models operating on aggregated features. However, modern fraud—account takeover, card-not-present fraud, money mule schemes, and first-party fraud—manifests as subtle temporal patterns distributed across multiple data streams. A fraudulent account takeover, for example, may involve a failed login from a new device, followed by a password reset, a P2P transfer to an unfamiliar recipient, and a risk system denial—all within a 48-hour window. No single event is anomalous in isolation; the signal emerges from the temporal correlation across streams.

Traditional approaches aggregate these streams into fixed feature vectors (e.g., “number of failed logins in 7 days,” “total transfer amount in 30 days”) and feed them to gradient-boosted trees [1]. While effective for simple patterns, this approach discards the sequential structure and cross-stream temporal relationships that characterize sophisticated fraud.

We address this gap with the Multi-Stream Fraud Transformer (MSFT), which:

• Processes each event stream (transactions, logins, risk events) as a variable-length sequence with its own Transformer encoder

• Preserves temporal information through time-aware positional encodings derived from actual event timestamps

• Learns cross-stream correlations through bidirectional attention between encoded stream representations

• Combines stream-level and cross-stream signals through a gated fusion mechanism

We evaluate MSFT through a controlled ablation study that isolates the contribution of each architectural component, demonstrating clear performance gains from temporal awareness and multi-stream fusion.

2 Related Work↩︎

2.1 Feature-Based Fraud Detection↩︎

Gradient-boosted decision trees (XGBoost [1], LightGBM [2]) remain the industry standard for fraud detection, operating on hand-engineered aggregated features computed over fixed time windows. These models excel at capturing non-linear relationships between static features but cannot model sequential dependencies or cross-stream temporal correlations.

2.2 Sequence Models for Fraud Detection↩︎

Recent work has applied recurrent neural networks [3] and Transformers [4] to transaction sequences, treating fraud detection as a sequence classification problem. These approaches capture temporal patterns within a single stream but typically process streams independently or concatenate them without explicit cross-stream reasoning.

2.3 Multi-Modal and Multi-Stream Learning↩︎

Multi-modal fusion has been extensively studied in vision-language models [5] and multimodal sentiment analysis. Techniques include early fusion (concatenation), late fusion (ensemble), and attention-based fusion. Our work adapts these ideas to the financial domain, where streams are heterogeneous event sequences with shared temporal structure.

2.4 Temporal Encoding↩︎

Standard positional encodings capture ordinal position but not actual time gaps between events. Time-aware encodings that incorporate continuous timestamps have been explored in temporal point processes [6] and event sequence modeling. We adopt learnable frequency-based temporal encodings that allow the model to distinguish events minutes apart from events days apart.

3 Problem Formulation↩︎

Given a user \(u\) with \(K\) event streams \(\{S_1, S_2, \ldots, S_K\}\), where each stream \(S_k = \{(e_1^k, t_1^k), (e_2^k, t_2^k), \ldots, (e_{n_k}^k, t_{n_k}^k)\}\) consists of events \(e_i^k\) with timestamps \(t_i^k\), and a set of aggregated features \(\mathbf{x}_u \in \mathbb{R}^d\), we aim to predict the binary label \(y_u \in \{0, 1\}\) indicating whether the user is fraudulent.

Each event \(e_i^k\) is a heterogeneous record containing categorical fields (merchant category, entry type, session event) and numerical fields (amount, time between events). The key challenge is that the discriminative signal lies not in individual events or aggregated statistics, but in the temporal patterns and cross-stream correlations.

4 Architecture↩︎

4.1 Overview↩︎

The Multi-Stream Fraud Transformer consists of four stages: (1) per-stream token embedding, (2) positional encoding, (3) per-stream Transformer encoding, and (4) fusion and classification.

4.2 Per-Stream Token Embedding↩︎

For each stream \(k\), an event \(e_i^k\) contains categorical features \(\{c_1, \ldots, c_m\}\) and numerical features \(\{n_1, \ldots, n_p\}\). Categorical features are embedded via learned embedding tables (or hashing for high-cardinality fields): \[\mathbf{h}_{\text{cat}} = [\text{Emb}_1(c_1); \text{Emb}_2(c_2); \ldots; \text{Emb}_m(c_m)]\] Numerical features are projected through a small MLP: \[\mathbf{h}_{\text{num}} = \text{MLP}_{\text{num}}([n_1; n_2; \ldots; n_p])\] The token representation is: \[\mathbf{h}_i^k = W_k [\mathbf{h}_{\text{cat}}; \mathbf{h}_{\text{num}}] + b_k\]

4.3 Positional Encoding↩︎

We compare two positional encoding strategies:

Sinusoidal (ordinal): Standard sinusoidal encoding based on position index.

Time-aware: Learnable frequency-based encoding using actual timestamps: \[\text{PE}(t) = W_{\text{proj}} [\sin(\omega_1 t + \phi_1); \cos(\omega_1 t + \phi_1); \ldots]\] where \(\omega_f\) and \(\phi_f\) are learnable frequency and phase parameters. This allows the model to understand that events 5 minutes apart are more related than events 5 days apart.

4.4 Per-Stream Transformer Encoder↩︎

Each stream is processed by an independent Transformer encoder with a prepended learnable [CLS] token: \[[\text{CLS}_k; \mathbf{h}_1^k; \ldots] \xrightarrow{\text{Encoder}_k} [\mathbf{z}_{\text{CLS}}^k; \mathbf{z}_1^k; \ldots]\] The [CLS] output \(\mathbf{z}_{\text{CLS}}^k\) serves as the stream-level representation.

4.5 Fusion Mechanisms↩︎

We study five fusion strategies:

Concatenation (concat): Concatenates stream CLS tokens and static features: \[\mathbf{f} = [\mathbf{z}_{\text{CLS}}^1; \ldots; \mathbf{z}_{\text{CLS}}^K; \text{MLP}_{\text{static}}(\mathbf{x}_u)]\] This is our baseline. It treats every stream representation as equally informative for every user. The downstream classifier must implicitly learn which dimensions of the concatenated vector are relevant per sample.

Gated Fusion. Different fraud types are dominated by different streams: account takeover is primarily a login phenomenon, card-not-present fraud is primarily a transaction phenomenon, and money mule activity manifests in both. Concatenation forces the classifier to discover this routing implicitly. Gated fusion makes it explicit by introducing a small gating network that produces a soft importance weight \(g_k \in [0,1]\) for each stream conditioned on the full multi-stream context: \[\label{eq:gated} g_k = \sigma\!\left(W_g\,[\mathbf{z}_{\text{CLS}}^1; \mathbf{z}_{\text{CLS}}^2; \ldots; \mathbf{z}_{\text{CLS}}^K; \mathbf{x}_u]\right)_k\tag{1}\] \[\mathbf{f} = \sum_{k=1}^{K} g_k \cdot \mathbf{z}_{\text{CLS}}^k\] The gating network is a two-layer MLP with LayerNorm; its final layer is initialized to zero so all gates start at \(\sigma(0) = 0.5\), preventing one stream from dominating early in training. Crucially, the fused representation \(\mathbf{f}\) has dimension \(D\) (not \(K \cdot D\) as in concat), which acts as a regularizer: the model cannot rely on having \(K\) copies of the signal and must commit to a single weighted combination. We observed empirically that this leads to substantially higher precision (Section 7), at the cost of slightly lower AUROC than time-aware PE alone.

Temporal. Uses time-aware PE (Eq. 4) at the encoder input, with concatenation fusion at the output. The architectural change is minimal but the inductive bias is significant: events are now indexed by elapsed time rather than ordinal position, so the model can recognize “three failed logins within an hour” as different from “three failed logins over a week.”

Cross-Stream Attention. The previous mechanisms treat each stream’s CLS token as a self-contained summary. Cross-stream attention relaxes this assumption: it lets every token in one stream attend to every token in another stream after per-stream encoding. Concretely, for each ordered pair of streams \((k, j)\) we maintain a multi-head attention layer: \[\tilde{\mathbf{z}}^k = \text{CrossAttn}(\mathbf{z}^k, \mathbf{z}^j) = \text{softmax}\!\left(\frac{\mathbf{z}^k W_Q (\mathbf{z}^j W_K)^\top}{\sqrt{d}}\right) \mathbf{z}^j W_V\] where \(\mathbf{z}^k \in \mathbb{R}^{L_k \times D}\) is the full encoded sequence (not just CLS) for stream \(k\). Each cross-attention layer is followed by a residual connection, layer normalization, and a feed-forward block, mirroring the standard Transformer block design. This allows, for example, a transaction token at \(t=102\text{h}\) to attend to a login token at \(t=100\text{h}\) and condition its representation on whether that login was successful or denied. The CLS token of each stream after cross-attention is then used in the downstream concatenation. With three streams we instantiate six directed cross-attention layers, which adds approximately 19M parameters compared to concat at the \(D{=}512\) scale and substantially increases the inductive bias toward multi-stream temporal correlations.

Full: Combines gated fusion + time-aware PE + cross-stream attention. Time-aware PE is applied at the encoder input, cross-attention is applied after per-stream encoding, and gated fusion is applied to the resulting CLS tokens.

4.6 Classification Head↩︎

The fused representation passes through a two-layer MLP: \[\hat{y} = \sigma(W_2 \cdot \text{ReLU}(W_1 \mathbf{f} + b_1) + b_2)\] Training uses binary cross-entropy with positive class weighting.

5 Dataset↩︎

5.1 Synthetic Data Generation↩︎

We generate a large-scale synthetic dataset designed to mimic real-world digital banking fraud patterns. The dataset contains 10,000,000 users with a 1.5% fraud rate distributed across four fraud types (Table 1).

5.2 Event Streams↩︎

Each user has three event streams:

Transactions (50–120 events): Purchase, deposit, transfer, and ATM withdrawal events with merchant category, amount, entry type, and network information.

Logins (12–40 events): Authentication events with device information, risk band, outcome, IP/device novelty flags.

Risk Events (4–15 events): Platform-level risk signals including login attempts, password resets, device changes, PII changes, and disputes with allow/deny outcomes.

5.3 Design Principles↩︎

The dataset is designed so that:

1. Aggregated features are weakly discriminative—fraud amounts overlap with legitimate spending

2. Temporal patterns are discriminative—fraud events cluster within a \(\pm\)​96h window

3. Cross-stream correlation is the strongest signal—login anomaly + transaction anomaly + risk denial in the same time window

5.4 Aggregated Features↩︎

We compute 64 aggregated features including windowed transaction counts and amounts (7d, 30d, 90d, 365d), login statistics, P2P ratios, percentile amounts, and balance metrics.

6 Experimental Setup↩︎

6.1 Data Splits↩︎

6.2 Baselines↩︎

XGBoost: Gradient-boosted trees on 64 aggregated features. 500 estimators, max depth 6, scale_pos_weight adjusted for class imbalance.

Flat Transformer: A single-stream baseline that merges events from all streams into one sequence, sorted by ascending time_to_snapshot. Each token receives a learned stream-type embedding so the model knows whether the event is a transaction, login, or risk event. The merged sequence is processed by a 24-layer Transformer encoder (versus 8 layers \(\times\) 3 streams in multi-stream models), giving the same total parameter budget (83.7M). This isolates the contribution of the multi-stream architectural choice from raw model capacity.

LSTM: Multi-stream bidirectional LSTM with the same per-stream embedding as the Transformer. Concatenation of final hidden states for classification.

6.3 Model Configuration↩︎

6.4 Training↩︎

All Transformer models are trained with distributed data parallel (DDP) across 8 NVIDIA A10G GPUs. Gradient clipping at norm 1.0. Best model selected by validation AUPRC.

6.5 Evaluation Metrics↩︎

We report AUROC, AUPRC (primary metric for imbalanced data), F1 score, precision, and recall at threshold 0.5.

7 Results↩︎

7.1 Main Results↩︎

Table 4 presents the main results. All Transformer variants dramatically outperform XGBoost, demonstrating that sequence modeling captures fraud patterns invisible to aggregated features. Figure 3 visualizes the performance gap.

7.2 Analysis↩︎

XGBoost vs.Sequence Models. XGBoost achieves only 0.74 AUROC on the aggregated features, confirming that the fraud signal is primarily temporal and cross-stream. All Transformer variants outperform XGBoost by over 25 points in AUROC.

Single-Stream vs.Multi-Stream. The flattened-stream baseline merges all events from all three streams into a single time-sorted sequence, processed by a 24-layer Transformer with the same total parameter budget (83.7M) as the multi-stream models. Despite parameter parity, this baseline reaches only 0.8174 AUROC and 0.4253 AUPRC—a drop of 18 AUROC points and 54 AUPRC points relative to multi-stream concat. The flattened model also has 91,526 false positives compared to 2,417 for concat, a 38\(\times\) degradation in production-relevant precision. This confirms that the inductive bias of per-stream encoding is essential: a stream-type embedding alone does not allow a single shared encoder to disambiguate events of fundamentally different semantics (transactions vs.logins vs.risk events). Multi-stream architectures are therefore not a convenience choice—they are necessary for this problem class.

Temporal Encoding. Time-aware positional encoding achieves the highest AUROC (0.9961) and AUPRC (0.9692). By encoding actual timestamps rather than ordinal positions, the model distinguishes events occurring minutes apart from events days apart—critical for detecting fraud velocity.

Gated Fusion. Gated fusion achieves the highest F1 (0.9639) and precision (0.9887) among Transformer variants, with only 241 false positives out of 1.5M test users. The learned importance weights allow dynamic emphasis on the most informative stream per user. We probed the learned gates by computing the average gate values per fraud type. Account takeover and money mule samples assigned the highest weight to the login stream (0.62 and 0.58 respectively), while card-not-present and first-party fraud assigned the highest weight to the transaction stream (0.71 and 0.66). Legitimate users had relatively uniform gates centered around 0.5, indicating no strong stream preference. This routing behavior is consistent with the underlying fraud patterns and is impossible for concat to express directly; the classifier must instead learn equivalent routing through linear weights on a \(K \cdot D\) dimensional vector, which is harder to optimize on imbalanced data.

Cross-Stream Attention. Cross-attention alone (0.9884 AUROC) underperforms other Transformer variants in our ablation, but the underperformance is structural rather than fundamental. Adding bidirectional cross-attention between three streams introduces six new attention layers with roughly 19M additional parameters. With only three training epochs, these layers are still learning the cross-stream alignment that simpler mechanisms (gated fusion, time-aware PE) capture more directly. The pattern in the metrics supports this: cross-attention has the highest recall (0.9144) but the lowest precision (0.6104) among Transformer variants, which is the signature of an under-fit model that has learned “something is suspicious” but not yet “what specifically” to attend to. The full model resolves this by combining cross-attention with gated fusion and temporal encoding, recovering most of the precision (0.9763) while preserving recall (0.9319).

Full Fusion. The full model (0.9950 AUROC, 0.9536 F1) outperforms cross-alone substantially, showing that gated fusion and temporal encoding stabilize the cross-attention component. With longer training budgets we expect cross-attention and full fusion to surpass the simpler variants, as cross-stream temporal correlation is structurally the strongest signal in the data (Section 7.3).

7.3 Stream Ablation↩︎

The stream ablation reveals that the risk event stream provides the dominant fraud signal. Adding risk events to transactions yields a 14-point AUROC improvement (0.857 \(\rightarrow\) 0.998), while adding login events alone provides minimal benefit. Risk events encode the platform’s existing fraud detection decisions (allow/deny) as a temporal sequence, which the model leverages for pattern recognition. Notably, the three-stream combination (txn + login + risk) achieves the highest precision (0.9991) despite slightly lower AUROC than txn + risk alone, suggesting that login events help filter out false positives even when they do not add ranking discrimination. Figure 4 visualizes this contribution pattern.

7.4 LSTM Comparison↩︎

The LSTM baseline (84.7M parameters, 4-layer bidirectional, hidden size 1024) uses the same per-stream embedding and concatenation fusion as the Transformer, replacing only the encoder. At 0.9979 AUROC, the LSTM is competitive with Transformer variants on this benchmark, which is consistent with the observation that bidirectional LSTMs are strong baselines on sequences shorter than a few hundred tokens [3], [7]. The competitiveness of LSTM here should not be interpreted as evidence that it is preferable in deployment. Three considerations limit LSTM scalability for production fraud detection:

Sequence length. Our experimental setup truncates user histories at 120 events per stream. Production fraud detection systems typically require histories of several thousand events spanning months to years [8]. LSTMs suffer well-documented vanishing-gradient and long-range modeling issues at such lengths, while Transformers have been shown to maintain performance into the 10K+ token regime [9]. We expect the LSTM-vs-Transformer gap on our benchmark to widen substantially at production-realistic context lengths.

Parallelism and scaling. LSTMs are inherently sequential along the time dimension during both training and prefill, while Transformers can parallelize across the entire sequence. Empirical scaling studies [10], [11] show that Transformers exhibit cleaner power-law scaling with parameters, data, and compute. As fraud detection systems move toward foundation-model-style architectures with hundreds of millions to billions of parameters, the Transformer’s parallelism advantage compounds.

Cross-stream reasoning. The LSTM baseline uses the same concat fusion as our Transformer-concat variant. It cannot natively support the cross-stream attention mechanism that we ablate in Section 7, which limits its ability to express fine-grained temporal alignment between streams—a key direction for future work as we scale the architecture.

In summary, the multi-stream architectural choice (Section 7.3) is the dominant factor in our results, while the choice of sequence backbone (LSTM vs.Transformer) is secondary on this synthetic benchmark. We adopt the Transformer as our primary architecture based on its better-documented scaling behavior and explicit support for cross-stream attention.

7.5 Feature Importance↩︎

Feature importance is relatively flat (max 5.6%), indicating no single aggregated feature is strongly predictive—a hallmark of problems where the true signal is sequential.

7.6 Real-World Validation↩︎

We validate on proprietary production data from a digital banking platform with similar event streams (transactions, logins, risk events). The multi-stream Transformer achieves over 22% relative improvement in AUROC compared to the XGBoost aggregated-feature baseline, consistent with the magnitude of improvement observed on synthetic data. Figure 5 shows the cross-domain consistency.

7.7 Scale Consistency↩︎

To demonstrate consistency across scales, we report results on a 1M user dataset with 3–5M parameter models (Table 7).

The relative ordering is consistent: temporal and gated fusion provide the strongest improvements over concatenation at both scales.

8 Discussion↩︎

8.1 Why Aggregated Features Fail↩︎

The aggregated feature approach fundamentally loses temporal ordering and cross-stream correlation. A user with 3 failed logins in 7 days and 2 P2P transfers may be fraudulent or legitimate—the discriminative signal is whether these events co-occur within hours, which aggregation destroys.

8.2 Temporal Encoding Benefits↩︎

Time-aware positional encoding provides two advantages: (1) it captures event velocity (rapid succession vs.spread over days), and (2) it enables implicit cross-stream temporal alignment without explicit timestamp matching.

8.3 Production Considerations↩︎

The gated fusion model is most suitable for production deployment due to its exceptional precision (0.989). In fraud detection, false positives directly impact customer experience through account freezes and declined transactions. The architecture supports variable numbers of streams and gracefully handles missing data.

8.4 Limitations↩︎

• Synthetic data may not capture all real-world fraud patterns

• Cross-stream attention requires more training epochs at this scale

• Evaluation is offline; production deployment requires latency considerations

• The model assumes all streams are available at inference time

9 Conclusion↩︎

We presented the Multi-Stream Fraud Transformer, a configurable architecture for financial fraud detection that processes heterogeneous event streams through independent Transformer encoders with temporal awareness and multi-stream fusion. Our experiments on 10M users with 85M parameter models demonstrate that:

1. Per-stream encoding is essential. At matched parameter budget, a single-stream Transformer baseline that merges all events into one time-sorted sequence reaches only 0.82 AUROC, versus 0.99+ for multi-stream variants—an 18-point gap. The architectural inductive bias of per-stream encoding is the dominant factor in our results.

2. Sequence models substantially outperform aggregated features (0.74 \(\rightarrow\) 0.99 AUROC), consistent with prior fraud detection literature [3], [7].

3. The risk event stream provides the dominant individual signal, contributing a 14-point AUROC improvement when added to transactions; combining all three streams achieves the highest precision (0.9991, only 19 false positives in 1.5M test users).

4. Among Transformer fusion variants: gated fusion achieves the best precision (0.989), making it most suitable for production deployment where false-positive cost is high; time-aware positional encoding achieves the best AUROC (0.9961).

5. LSTM and Transformer are both strong sequence backbones on our benchmark (sequence length \(\sim\)​120). The Transformer’s documented advantages in long-context modeling [9] and parameter scaling [10], [11] make it the preferred backbone as production deployments move to longer histories [8].

6. Results validated on real production data: over 22% relative AUROC improvement over XGBoost on a US digital banking platform, consistent with synthetic-data findings.

The architecture is designed for production deployment with configurable stream inputs, making it applicable to any multi-stream event classification problem in financial services.

References↩︎