1 Introduction↩︎

Tabular transaction records contain many high-cardinality categorical variables (card identifiers, billing addresses, email domains, device fingerprints) with cardinalities from four to tens of thousands. How to represent such categories is an important modeling decision, but evidence is inconsistent: encoder benchmarks stay within the classical family [1], [2], while tabular deep-learning studies fix the encoding and vary the model [3]–[5]. A banking practitioner cannot determine whether neural entity embeddings [6] or a leakage-safe target encoder [7], [8] will serve best in a fraud model when model-transparency requirements (such as SR 11-7 in the US, or analogous audit standards elsewhere) demand that scores be explainable.

The question spans three axes. Accuracy margins are narrow, so isolating the encoder from the learner matters. Compute varies widely: end-to-end deep models can be an order of magnitude slower than tree-plus-embedding pipelines. Interpretability is critical under SR 11-7, which requires model-risk owners to justify scores, so a slightly less accurate but more interpretable encoder may be preferable. We report all three together.

We address: what are the accuracy, interpretability, and computational trade-offs of seven encoders on a large fraud dataset with a fixed downstream learner, and where does lightweight statistical grouping sit on this continuum? Our contributions:

1. Cross-paradigm controlled experiment. We compared five encoders under a single fixed LightGBM learner on the IEEE-CIS dataset with stratified CV, plus CatBoost and TabNet as cross-paradigm reference points. Previous studies [3], [4] varied the model family with a default encoder; we varied only the encoder, finding \(\approx\)​0.01 AUC-ROC attributable to encoding alone.

2. Auditor-readable encoding operationalized and benchmarked. We operationalized and benchmarked target-aware tier grouping (Bayesian-smoothed fraud rates binned into \(K\) ordinal tiers [7], [9]) and evaluated it against learned representations.

3. Interpretability-first framing for regulated finance. We scored interpretability, compute, and metric robustness as primary outcome dimensions alongside SR 11-7 model-risk-management expectations.

2 Related Work↩︎

Categorical encoding. Target encoding with Bayesian smoothing [7] underpins leakage-safe variants in CatBoost [8]. Other methods include feature hashing [10], similarity encoding [2], [11], and conjugate Bayesian encoders [9]. Pargent et al. [1] find regularized target encoding best among classical encoders; Guo and Berkhahn [6] established entity embeddings as the deep-learning default. Our tier grouping applies the smoothing principles of [7], [9] and bins smoothed rates into ordinal tiers; we evaluate it as an auditor-readable encoder.

Trees versus deep learning on tabular data. Recent deep models include TabNet [12], NODE [13], and FT-Transformer [14]. Grinsztajn et al. [3] and Shwartz-Ziv and Armon [4] show GBDTs (XGBoost [15], LightGBM [16]) still match or exceed them; Borisov et al. [5] identify interpretability and compute as unresolved trade-offs. None of these studies compare encoders across both paradigms.

Closing the divide. Encoder benchmarks use classical learners [1], [2]; tree-vs-deep studies fix the encoding [3], [4]. No study has treated the encoder as the controlled variable across both. We close this gap on IEEE-CIS [17], a 590K-transaction fraud task (\(\approx\)​3.5% base rate) [18]–[21], keeping the learner, folds, and protocol constant [22].

3 Methods↩︎

We defined seven encoders on the IEEE-CIS fraud dataset (590,540 records, 3.50% positive, eight high-cardinality columns, cardinalities 4–13,553). E1–E4 and E6 feed the same downstream LightGBM [16] with frozen hyperparameters; E5 uses CatBoost [8] as both encoder and learner; E7 feeds target-encoded input to TabNet [12]. All encoders receive identical raw features (eight high-card columns, eleven low-cardinality one-hot columns, 395 numerical columns). We partition results into controlled comparisons (E1–E4, E6: same learner) and cross-paradigm observations (E5, E7: different learner).

The eight high-cardinality target columns (with cardinalities) are: card1 (13,553), DeviceInfo (1,786), addr1 (332), addr2 (74), R_emaildomain (60), P_emaildomain (59), card4 (4), card6 (4).

E1: One-hot. Each value becomes an indicator; values that occurred fewer than fifty times in the training fold are dropped (\(\approx\)​2,912 features per fold).

E2: Target encoding. Each value is replaced with its out-of-fold mean target via five-fold inner cross-fitting with Bayesian smoothing (\(m{=}30\)); a permutation test confirms leakage invariance.

E3: Frequency. Each value maps to its relative frequency in the training fold.

E4: Tier grouping (interpretable). Per high-cardinality column on each fold’s training partition: (1) compute Bayesian-smoothed fraud rate \(\hat{r}_v = (n_v r_v + m\bar{r})/(n_v + m)\), \(m{=}30\); (2) sort by \(\hat{r}_v\) and bin into \(K \in \{3,5,7\}\) ordinal tiers via equal-frequency, KS-greedy, or chi-square binning; (3) replace raw values with tier IDs, monotone in fraud rate. Unseen test values map to the tier closest to \(\bar{r}\). Tiers are auditor-readable: an inspector sees “tier 5 \(=\) outlook.com at 16.5%, tier 1 \(=\) comcast at 1.2%” from bin boundaries. Because several columns are skewed (addr2 is 88% one value), \(K\) is an upper bound on populated tiers.

E5: CatBoost. CatBoost [8] consumes raw categoricals natively via a permutation-based ordered target statistic, avoiding leakage without explicit inner CV. Hyperparameters mirror LightGBM (iterations=500, learning_rate=0.05, depth=7, early_stopping_rounds=50).

E6: Entity embeddings \(\to\) LightGBM. Following Guo and Berkhahn [6], a shallow network learns dense embeddings per high-card column; trained embeddings are extracted and fed to the same frozen LightGBM. Per-column dimension is \(\min(50, (n_\text{unique}+1)//2)\) (255 total). Classifier head: two Linear–ReLU–Dropout(0.3) blocks (256, 128), Adam (\(lr{=}10^{-3}\)), batch 4096, 20 epochs, patience 3. The extracted matrix averages \(\approx\)​1,824 columns. For each outer fold, the embedding network is fit only on the training partition; validation labels enter solely via early-stopping. TabNet (E7) follows the same rule.

E7: Target encoding \(\to\) TabNet. TabNet [12] with published defaults (n_steps=3, n_a=n_d=64, \(\gamma{=}1.5\), batch 4096, 50 epochs, patience 5). E7 receives the same pre-encoded numerical feature matrix as E2 (low-cardinality columns one-hot, high-cardinality columns target-encoded) and does not consume raw categorical indices; it differs from E2 only in the downstream learner (TabNet vs.LightGBM), so the E7–E2 contrast isolates the learner with encoding held fixed. No architecture or hyperparameter search was performed.

4 Experimental Setup↩︎

Dataset. IEEE-CIS Fraud Detection [17]: 590,540 transactions, 3.50% positive, 432 features. Eight high-cardinality columns (card1, DeviceInfo, addr1, addr2, R_emaildomain, P_emaildomain, card4, card6) are the encoding targets; eleven low-cardinality columns are always one-hot encoded identically.

Protocol. Stratified 5-fold CV \(\times\) 3 repeats \(=\) 15 runs per encoding. LightGBM hyperparameters fixed across E1–E4, E6 (n_estimators=500, learning_rate=0.05, max_depth=7, early_stopping_rounds=50), with no per-encoding tuning. The robustness experiment retrains all encoders on 50/25/10% subsamples (5 folds per cell, 105 runs); the per-column experiment trains E2, E4, E6 on one high-card column at a time (5 folds per cell, 120 runs).

Metrics. We fixed AUC-ROC as the primary metric in advance of running the experiments and report AUC-PR as a secondary, imbalance-aware metric. Fixing the primary metric beforehand blocks ex-post metric-shopping; we keep AUC-ROC primary precisely because the two metrics disagree on the winner (Section V-D).

Statistical tests. Our \(R{=}15\) resamplings have correlated folds, so ordinary paired tests understate variance. We take the Nadeau–Bengio corrected resampled \(t\)-test [23] (\(n_\text{test}/n_\text{train}{=}0.25\), df\({=}14\)) as primary pairwise inference. For an illustrative omnibus ranking we run Friedman + Nemenyi [24] (\(\chi^2{=}80.9\), \(p{=}2.3\times10^{-15}\), CD\(_{0.05}{=}2.325\)); the CD diagram serves as a ranking aid alongside the NB-corrected test.

Hardware and reproducibility. All encoders ran on the same Apple-Silicon machine (neural models on MPS GPU, tree models on CPU). MPS is low-throughput relative to NVIDIA, so absolute wall-clock times overstate DL cost; the cost ranking E4\(<\)E6\(<\)E5\(<\)E7 is architectural. Seeds: fold splits {42,43,44}, models 42. Software: LightGBM 4.6, CatBoost 1.2.10, PyTorch 2.12.1, pytorch-tabnet 4.1.0. The IEEE-CIS dataset is public [17]; all hyperparameters and splits are in Sections III–IV. Reproducibility of execution assumptions is an active concern in financial ML [25].

5 Results↩︎

5.1 Main results and significance↩︎

Table 1 reports mean \(\pm\) std over 15 runs, sorted by AUC-ROC. Scores span only 0.9515–0.9612 (\(\approx\)​0.97 pp), with entity embeddings (E6) leading at 0.9612, narrowly ahead of CatBoost (E5, 0.9602).

4pt

Table 2 provides the NB-corrected test results that settle the anchor contrasts. E6 significantly outperforms both E4 (\(+0.0064\), \(t{=}11.74\), \(p{<}0.001\)) and E7 (\(+0.0091\), \(t{=}6.95\), \(p{<}0.001\)). Conversely, E4 vs.E7 (\(p{=}0.089\)) and E6 vs.E5 (\(p{=}0.23\)) are statistical ties. It is also noteworthy that E2 outperforms E4 (\(-0.0023\), \(p{=}0.0003\)), meaning tier grouping is competitive with but slightly below the classical champion.

3pt

The Friedman/Nemenyi CD diagram (Figure 1) is illustrative only, since CV folds violate the cross-dataset independence the test assumes. The fold-level CD (\(N{=}5\), \(k{=}7\)) does not account for repeat-level pairing, limiting its power; it reads E6–E4 as a tie (\(\Delta\)rank 3.00 vs.CD\({=}4.03\)) while the properly powered NB-corrected test finds significance. The cliques are consistent with Table 2: E6/E5/E2 in the top band, E4 just below (E2 statistically ahead at \(p{=}0.0003\)), and E7/E3/E1 in the bottom band with E7 indistinguishable from E4 (\(p{=}0.089\)).

5.2 Accuracy–compute frontier↩︎

Compute cost differences are much larger than accuracy differences. With tier grouping (E4) as the cheapest encoder (24.5 s/fold, clearing 0.954 AUC), embeddings (E6) gain \(+0.0064\) AUC at 4.3\(\times\) the time (104.6 s). TabNet (E7) costs approximately 67\(\times\) more (1654.3 s/fold) yet scores 0.0091 below E6. E7’s fit time varies (1263–2255 s) due to the sample-size sensitivity of the sparse-attention architecture; our comparisons use default published parameters with no architecture or hyperparameter search. Given the total AUC-ROC spread is \(\approx\)​0.97 pp, practical encoder selection depends on compute cost, interpretability, and metric choice.

5.3 Interpretability and what the embeddings encode↩︎

Figure 2 projects E6’s P_emaildomain embeddings (30-dim) to 2-D via t-SNE, colored by E4’s tiers. K-means on the 2-D projection vs.E4’s tier assignment gives ARI 0.051 and NMI 0.175; because t-SNE does not preserve distances, these values are illustrative of weak visual alignment only and do not quantify representational similarity. E4’s tiers encode single-column fraud-rate ordering; the per-column probe (Section V-E) provides the sound evidence that E6’s embeddings capture cross-column structure the tiers discard.

5.4 AUC-ROC vs AUC-PR: the metrics disagree↩︎

Because fraud detection is highly imbalanced (3.5% positive), AUC-PR is the more deployment-sensitive metric. Here CatBoost (E5) leads at 0.8216, well above E6 (0.7928), E7 (0.7828), and E4 (0.7704); the E5–E6 AUC-PR gap (0.029) is an order of magnitude larger than their AUC-ROC gap (0.001). None of the five LightGBM-routed encoders approach E5’s precision-recall performance, and no encoder dominates both metrics. We fixed AUC-ROC as the primary metric in advance to prevent metric-shopping.

5.5 Robustness and sensitivity↩︎

Data scarcity. We conducted subsampling at 50/25/10% (105 runs) for the six tree-routed encoders (E1–E6). All six exhibited nearly parallel degradation from 50% to 10%, with \(-0.031\) to \(-0.043\) AUC-ROC loss over that interval, and E6 remaining the highest performer at every fraction (0.9527/0.9391/0.9157 at 50/25/10%). Over the same 50%-to-10% interval, E4 lost slightly more than E6 (\(-0.043\) vs.\(-0.037\)), so the hypothesis that learned embeddings degrade faster than tier grouping is not supported. The contrast is driven by E7: off-the-shelf TabNet collapsed to 0.8108 at 10%, a loss roughly 3\(\times\) any tree-routed encoder’s, with variance exploding (std \(=\) 0.061 vs.\(\leq\)​0.004). This finding is scoped to stock TabNet; tuning or regularization may improve results.

Per-column localization. E2, E4, and E6 were evaluated (120 runs) with training on one high-cardinality column at a time. Results indicate E6 underperformed classical encoders on all eight columns (\(-0.0009\) to \(-0.0045\) AUC-ROC), confirming that E6’s edge is attributable to joint multi-column embedding: latent co-occurrence across card1, addr1, and DeviceInfo, rather than any single column.

Tier-grouping sensitivity. As shown in Table 3, E4 is largely insensitive to its hyperparameters: AUC-ROC moves less than 0.0019 across \(K{=}3\!\to\!7\) for equal-frequency binning, the best sub-variant at every \(K\). The \(K{=}5\) headline config (0.9548) is virtually equal to the best (\(K{=}7\), 0.9556) within one standard deviation.

6 Discussion↩︎

Encoding as a confounder. The confounding role of encoding is evident in the work of Grinsztajn et al. [3] and Shwartz-Ziv and Armon [4], who varied the model family along with default encodings. Our approach fixes the learner and varies only the encoder, finding a \(\approx\)​0.01 AUC-ROC swing attributable to encoding alone. We do not claim this matches the tuned, cross-dataset GBDT-vs-DL gaps those authors report, but our structural point stands: encoder choice is an independent degree of freedom that the trees-versus-deep comparisons controlled for implicitly and could not observe.

Extracted embeddings versus TabNet. The two deep-network conditions sit at opposite rank extremes: E7 (target-encoded input to TabNet) is worst and E6 (extracted embeddings to LightGBM) is best. Because E7 shares E2’s target encoding, the E7–E2 gap (\(-0.0049\), 1579 features each) isolates the downstream learner, showing untuned TabNet trails LightGBM on identical input. Therefore, this E6–E7 gap illustrates a richer encoding of the data, as well as a more capable learner, and whether tuned TabNet, FT-Transformer [14], or other deep tabular architectures will help close this gap remains open. In addition, task-specific evaluations have the potential to reverse the expected complexity rank order of conditions in other domains. For example, Cao et al. [26] found that smaller language models perform better than larger models on targeted tasks, which also reflects the result in this study where the most complex pipeline did not outperform all others. A parallel observation holds for representation learning: Lai et al. [27] report that transformer-based embeddings do not consistently outperform simpler semantic encoders on short-text classification, echoing our finding that a more elaborate encoder is not universally superior. Using the controlled LightGBM comparisons, we can cleanly isolate E6 compared to E4 (\(p{<}0.001\)) and E4 compared to E2 (\(p{=}0.0003\)).

Transparency costs. E4’s transparency costs \(+0.0064\) AUC-ROC versus E6 and 0.0023 versus E2; both gaps are significant but relatively small. E4 ties E7 (\(p{=}0.089\)) at 1/67 the cost and trails E2/E5/E6 by at most 0.0064. Under SR 11-7, an auditor can directly inspect tier boundaries (e.g., card6: debit-or-credit at 2.0% in tier 1, credit at 6.6% top tier); a 30-dim embedding vector offers no such reading. The interpretability gap that Borisov et al. [5] identify can, for the encoding axis, be addressed by tier grouping.

Metric sensitivity. AUC-ROC and AUC-PR disagree on the top performer: E6 leads AUC-ROC (narrowly, tying CatBoost), while E5 is far ahead on AUC-PR (0.8216 vs.). This lead reflects CatBoost’s bundled encoder+learner, not encoding alone, since E5 is a cross-paradigm observation (Section III). Sun et al. [28] similarly find, in bank-account fraud, that training objective and class-imbalance handling can matter more than architecture. Our result shows that the encoding decision reshapes rankings independently of the learner. The same pattern recurs across financial modeling, from cross-market volatility forecasting [29] to robustness–precision trade-offs in financial retrieval [30]. The two metrics thus produce different winners, and the choice depends on deployment objectives.

Practical recommendation (on IEEE-CIS). Table 4 maps deployment priorities to encoder choice. Maximum accuracy is achieved with E6 and E5; E4 is the preferred choice under model-risk constraints. Off-the-shelf TabNet provides no advantage under this protocol, though a tuned TabNet or alternative deep tabular architecture (e.g., FT-Transformer [14]) could produce different results.

7 Conclusion↩︎

On the IEEE-CIS fraud benchmark, a controlled comparison of seven categorical encoding strategies yielded measurable differences in accuracy, computational cost, and interpretability. Learned entity embeddings fed to LightGBM produced the highest AUC-ROC (0.9612), slightly ahead of CatBoost and \(+0.0064\) above tier grouping, an edge limited to joint multi-column embedding. The tier grouping method proved competitive and fully auditable, trailing target encoding by only 0.0023 while offering tier boundaries suitable for SR 11-7 review.

The off-the-shelf version of TabNet did not outperform tree-based pipelines and collapsed under data scarcity. On AUC-PR, the more deployment-sensitive metric, CatBoost leads by a wide margin; no single encoder dominated both metrics. Whether tuned deep tabular architectures close the gap remains to be tested.

Limitations. (1) Single dataset. We used IEEE-CIS only; we chose depth over breadth and scope all claims accordingly. (2) TabNet not exhaustively tuned. E7 uses default published parameters; a tuned TabNet, FT-Transformer [14], or NODE [13] could produce different results. (3) No temporal split. We applied stratified random CV; a time-ordered split could change rankings, particularly for target-derived encoders (E2, E4) whose smoothed rates may be inflated by future-leaking folds. (4) Anonymized features. V1–V339 are anonymized, so E4’s tier boundaries can be inspected numerically but not mapped to business semantics. (5) Fixed downstream model. LightGBM hyperparameters remain unchanged across E1–E4/E6; per-encoding tuning might reorder classical encoders. (6) Small AUC deltas (\(\approx\)​0.97 pp spread); the practical signal lies in the compute, interpretability, and data-scarcity axes.

References↩︎