1 Introduction↩︎

Relational databases (RDBs) are the backbone of structured data management, where multiple interconnected tables are linked via primary-foreign key relationships. Countless services are performed in e-commerce, healthcare, finance, banking and scientific research by managing data in dozens of tables, millions of rows, and hundreds of columns. [1], [2]. RDBs are designed and optimized to support complex data queries and management, but they do not take into account the requirements of predictive modeling. Many consequential predictive problems, including customer churn forecasting, clinical risk scoring, supply-chain demand estimation, and recommendation, can benefit from learning RDBs [2], [3]. However, the sheer volume and breadth of RDB data spread across dozens of interrelated tables and millions of rows pose a unique challenge for developing big-data-driven predictive models. The fundamental challenge in applying conventional machine learning paradigms to RDBs stems from the structural distinctions between “tabular” and “relational” learning. In a typical tabular structure, each data sample is a self-contained row where samples are assumed to be independent and identically distributed (i.i.d.). RDBs are different from typical tabular data sets in three different ways. First, an RDB includes multiple related tables linked by constraints of the primary-key and foreign-key (PK–FK), which also contribute the necessary predictive signals to learn an entity type [4]. For example, predicting an entity type, such as customers, would benefit from customer-related information spread across other tables that store transactions, orders, and reviews. Second, RDBs commonly include timestamps and historical records, unlike a data table with static information. Time information must be properly considered to avoid temporal leakage in predictive modeling tasks [1]. Third, samples in RDB are inherently dependent, unlike the i.i.d. assumption made in traditional machine learning. For example, two customers can order similar products and services or the same patient appears in multiple follow-up visits, introducing statistical dependencies between samples [3]. These three distinctions have motivated recent research efforts in introducing deep learning methods that operate directly on RDBs leveraging the relational information across tables. However, deep learning methods for RDBs have been evaluated under varying training configurations and computational budgets against some selective baselines. Therefore, it is unclear whether the performance gain reported for the existing methods is significant compared to other competitive baselines. The tabular learning community has benefited from extensive benchmarking studies [5]–[8]. We argue that recent deep RDB methods have achieved a level of maturity that warrants a first of its kind benchmarking study.

In this work, we present a systematic evaluation of state-of-the-art deep learning methods for relational databases. Our controlled experiments with a common protocol compare across representative architectures that span GNN-based methods, graph transformers, and foundation model approaches. We do not intend to propose a new method, but to inform the community with an empirical characterization of state-of-the-art methods to further guide future research and practical deployment decisions.

2 Related Work↩︎

2.1 Single-Table Tabular Methods↩︎

Deep learning for tabular data has long struggled to demonstrate robust performance compared to traditional machine learning methods [9], [10]. Unfortunately, there is no single best learning algorithm for all tabular datasets due to heterogeneity in domains and tabular structures [11], unlike image and language models. Gradient-boosted decision trees, such as LightGBM, remain one of the dominant methods for tabular prediction tasks due to their robust performance and fast training without the requirement of feature engineering [10]. RealMLP has emerged as a competitive deep learning model for tabular data, demonstrating that improved MLP architectures can achieve performance comparable to gradient-boosted decision trees on large tabular benchmarks [12]. Recently, TabPFN 2.5 [13] has topped the tabular model leaderboard, which uses a transformer model trained on millions of synthetic tabular datasets. Training examples, such as synthetic data for TabPFN 2.5, are generated by simulating millions of tabular datasets using structural causal models that create diverse relationships between features and targets and include realistic challenges such as noise, nonlinear transformations, and categorical variables.

The final prediction on the test data is achieved using a single forward pass without any gradient-based finetuning. The recent TabPFN model has shown competitive performance, strongly outperforming traditional machine learning baselines in small-to moderate-sized data sets (around 10,000 samples and 500 features). However, a recent study [14] also shows that the TabPFN 2.5 model struggles against tree-based models on large-scale data sets. Because relational databases involve millions of records across multiple tables, they may pose a real challenge for TabPFN 2.5.

All learning methods for tabular data assume a single table with self-contained rows and columns. However, many data tables, namely relations, from business, finance, and medical enterprises appear in RDBs, connected by referential integrity, as shown in Figure 1. Therefore, a learning task (e.g., classification or regression) related to a single table in an RDB can benefit from incorporating data drawn from its neighboring tables linked through referential keys. The deep learning methods proposed for single tables can be used by joining multiple related tables into a single table. However, the joining process inevitably loses important relational information [15], [16] and generates missing values when the rows between tables do not have one-to-one mappings. Over the past few years, a number of deep architectures have been proposed, tailored to learning the multi-table referential structure of RDBs. Among the proposed architectures, graph neural networks are the most common, where RDBs are modeled using heterogeneous graphs [1], [3], [17]. The attention mechanism on the relational structure is introduced in the graph transformer [18]. More recently, graph-based [19] and transformer-based [4] foundation models for relational databases have been proposed to generalize across different database schemas.

2.2 Graph-based learning methods↩︎

Joining multiple tables into a single table can compromise useful relationships between tables, which can be preserved by modeling multi-table structures as graphs for processing with neural networks. Griffin [19] models an RDB as a graph with rows as nodes and their relationships (primary-foreign key links between rows across tables) as edges. Griffin models RDBs in graph structures following two steps. Griffin first obtains an initial representation for each row by aggregating its column-level information. In other words, attention between the column-level cell embeddings of each row is used to obtain the corresponding row embedding. The second step in Griffin involves information exchange between rows across tables in RDBs. A message-passing mechanism is applied to update the row embeddings (representations) by propagating messages across connected rows. The message passing follows the edges in a graph representing the primary-foreign key relationship: a row in one table sends its embedding to connected rows in other tables. Each row updates its embedding by aggregating the incoming messages from connected rows. A special feature of Griffin is that it is designed as a foundation model. It includes pretrained text and numerical encoders to handle diverse data types in RDBs. Griffin is pretrained on both single-table and multi-table databases to learn general representations that transfer across databases and tasks, enabling a single pretrained model to be adapted through task-specific fine-tuning for various downstream applications.

2.3 Transformer-based learning methods↩︎

The Relational Transformer (RT) [4] introduces a fine-grained representation of individual tabular cells in RDB. For each table, every cell value is encoded as a vector representation, including the row, column, and table identifiers of the cell. For a given prediction task, the model uses the target row sample whose label is to be predicted and combines all cell embeddings of that row and those from rows in other tables linked by primary-foreign key relationships. The combined cell representations from multiple tables are used to learn a between-cell attention-weighted representation using a transformer. However, sparse attention is maintained by restricting cell pairs in attention computation. Cell pairing is regulated using a fixed binary self-attention mask. The binary mask allows a cell representation to attend to other cells if they belong to the same row (capturing interactions between columns), the same column within a table (capturing interactions between rows), or cells belonging to rows of different tables that are connected via primary-foreign key relationships. All other cell pairs are masked out in computing attention and do not attend to each other.

2.4 Hybrid graph-transformer learning methods↩︎

DBFormer [20] uses a hybrid graph-transformer architecture. Similarly to graph-based learning methods such as Griffin [19], the relational structure of the database is represented by rows as nodes and primary-foreign-key relationships as edges. However, DBFormer differs from Griffin in two important ways: 1) how it processes information within a table through the attention mechanism and 2) how it exchanges information between tables through graph modeling. Within each table, DBFormer [20] uses a transformer encoder to learn each row representation using the self-attention mechanisms. The self-attention mechanisms learn attention between columns to capture feature interactions in row representation. To learn between tables, DBFormer connects rows between tables via primary-foreign key (PK-FK) links. However, instead of using message passing via standard GNN aggregation, DBFormer uses cross-attention between the cells (features) of connected rows to learn PK-FK representation. This process is repeated across multiple transformer layers , allowing information to propagate across multi-hop relational neighborhoods. One point to note is that DBformer is not designed as a foundation model, unlike its counterparts RT and Griffin models. Therefore, DBformer must be trained on individual RDBs data from scratch.

2.5 Need for systematic benchmarking↩︎

Existing deep RDB methods are evaluated using varying experimental protocols, computational budgets, and against selective baselines [4], [19], [20], which makes a fair comparison of current models challenging. Deep RDB models learn from multiple tables by expanding the relational neighborhood of a target row outward through primary-foreign key links, as illustrated in Figure 1. Practical factors beyond predictive performance, such as training time and memory usage, are critical to processing large RDBs with up to millions of samples. However, computational time and complexity are rarely reported in the deep RDB learning literature. For example, the training time reported in RT [4] is for a specific large-scale hardware configuration (8 GPUs/320GB of memory), which is not directly comparable to the hardware used in other studies. Moreover, the performance of a learning algorithm can vary significantly for different databases and tasks, since there is no universal best performer for all data problems [21]. Therefore, the selection of databases and learning tasks is a critical step in evaluating the learning algorithms. For example, the study-outcome classification task in the rel-trial RDB shows superior performance on the pretrained Griffin [19] model compared to its non-pretrained counterpart. The opposite performance trend is observed for the user-clicks classification task in the rel-avito RDB. Consequently, a standardized benchmarking protocol is needed to allow a fair and independent assessment of the deep RDB models proposed in the literature. The tabular learning community has benefited substantially from this kind of controlled evaluation [9], [10]. Relational deep learning, despite its rapidly expanding model landscape, lacks a comparable body of research. Our work aims to fill this gap by evaluating all models in a unified protocol with matched computational budgets, consistent data splits, and standardized evaluation metrics.

3 Methodology↩︎

3.1 Customizing for benchmarking↩︎

Individual deep RDB methods have been implemented using distinct protocols, each requiring adjustments to ensure fair benchmarking in a controlled setting. We have developed a unified benchmarking protocol that standardizes data representation, relational context construction, and evaluation for all methods, as shown in Table 2.

3.1.1 Relational mapping in hops↩︎

First, recent studies differ in how they manage and exploit relationships between tables in an RDB. Relational databases store data in multiple connected tables, where relationships between tables (via primary–foreign keys) allow information to be shared across them. Relational models, such as RT [22], Griffin [19], and DBFormer [20], are designed to use connections between tables. However, these methods rely on predefined sampling strategies, e.g., randomly selecting a limited number of rows from neighboring tables. There is no clear control mechanism for how far the referential chain of tables in an RDB is considered when sampling rows. To address this inconsistency, we introduce a hop-based formulation, where a hop represents a one-step referential connection (PK-FK) between two neighboring tables. For a target row or table, the 1-hop neighborhood consists of rows of tables directly linked to the target through a PK-FK relationship. The 2-hop neighborhood consists of rows from tables in the second neighborhood that are connected to the first neighborhood but not directly to the target, as shown in Figure 1. The hop setting can be used to control how deeply the model retrieves information from an RDB for a predictive task. By the definition of hop, a hop=0 configuration uses only the target table without including rows from the neighborhood tables. For a given target row \(u\), we define its relational neighborhood using hop-based expansion, where \(\mathcal{N}_h(u)\) denotes the set of rows that belong to the \(h\)-hop neighborhood of \(u\), with \(h \in \{0,1,2\}\) in this study.

The 0-hop neighborhood contains only the target row: \[\mathcal{N}_0(u) = \{u\}\]

The 1-hop neighborhood consists of all rows directly connected to \(u\): \[\mathcal{N}_1(u) = \{ v \mid v \text{ is directly linked to } u \}\]

The 2-hop neighborhood represents rows that are reached by following two consecutive primary–foreign key connections starting from the target row (u). \[\mathcal{N}_2(u) = \{ w \mid w \text{ is linked to some } v \in \mathcal{N}_1(u) \}\]

In general, the relational context of \(u\) up to hop \(H\) is: \[\mathcal{N}_{\le H}(u) = \bigcup_{h=0}^{H} \mathcal{N}_h(u)\] which includes all rows reachable within \(H\) relational steps. This neighborhood is constructed iteratively by expanding the target row through PK-FK connections for \(H\) steps and restricting expansion beyond \(H\) hops.

3.1.2 Data encoding mechanism↩︎

Second, deep RDB methods use different data encoding methods to convert and combine a heterogeneous feature space (numerical, categorical, or ordinal) of RDB tables into an effective representation vector. For example, DBFormer uses GloVe embeddings [23], Griffin uses Nomic Embed [24], and RT uses MiniLM-v2 [25]. These encoders differ substantially in their embedding dimensionality and semantic capabilities. The choice of data encoder can influence the performance of deep RDB models. For example, GloVe embeddings are outdated because they do not provide context-aware embeddings, unlike transformer-based encoders. The Nomic Embed encoder contains roughly 137 million parameters and outputs 768-dimensional embeddings, whereas MiniLM-v2 has about 33 million parameters and generates 384-dimensional embeddings—approximately one-quarter of the parameters and half the embedding dimensionality. Since our benchmarking protocol involves training each model multiple times on several large RDBs, tasks, and hop settings, using a lighter encoder significantly reduces computational cost. Therefore, to enable a fair comparison, we employ MiniLM-v2 [25] for all models, which generates 384-dimensional dense embeddings. We use MiniLM-v2 [25], particularly for encoding categorical and textual features, from SentenceTransformers [26]. MiniLM-v2 is a compact transformer model obtained via knowledge distillation from larger pretrained model, BERT. Specifically, it learns to mimic the self-attention behavior of the large BERT model while using significantly fewer parameters, resulting in a lightweight yet high-quality contextual encoder.

Third, different encoding strategies for numerical features in RDBs have been adopted in the literature. For example, DBFormer uses a simple linear layer to convert each numeric value into a vector, while RT uses a similar linear transformation to map each value to the hidden dimension of the model. In short, both DBFormer and RT use a simple linear layer to project each numerical value directly into their respective model. In comparison, Griffin employs a more advanced strategy. It begins by normalizing numerical values with a quantile transformer. This transformer, provided by scikit-learn [27], applies a statistical procedure that maps numerical inputs onto a normal distribution. After this step, the normalized values are fed into a pretrained encoder–decoder pair, both realized as Multi-Layer Perceptrons (MLPs).

3.1.3 Customizing non-relational baselines↩︎

In this study, we consider state-of-the-art non-relational baseline methods for deep learning on tabular data. These methods need to be extended for multi-hop learning to ensure a fair comparison. For TabPFN 2.5 [13] and LightGBM, we convert multi-hop relational information into tabular features by aggregating data from connected tables, similar to how RelBench [1] compares traditional machine learning models against its proposed GNN-based Relational Deep Learning (RDL) model. However, unlike RelBench [1], we control the relational context by limiting the hop distance (H) and enforcing temporal consistency to avoid information leakage.

3.2 Effect of multi-hop deep RDB learning↩︎

Deep RDB learning uses a task-specific target table. A learning task that uses only the target table is defined as having hop = 0, meaning that no information from neighboring tables connected to the target is used. As shown in Figure 1, the hop = 1 scenario incorporates the tables directly adjacent to the target table. Similarly, in the hop=2 scenario, the network is expanded to include the second-tier tables that are directly linked to the adjacent tables. The computational complexity increases as we expand the network of tables from hop =0 to hop=2. The effect of network expansion on overall learning performance has not been well studied in the deep RDB literature.

4 Results↩︎

4.1 Relational databases and computing platform↩︎

In recent work on deep learning for RDB [19], [22], RelBench appears as the primary benchmarking repository [1]. We use five databases from RelBench, assigning two tasks to each: one classification task and one regression task. The selected databases and tasks are summarized in Table 1. For each database, Table 1 reports the total number of rows in all tables (database rows), the number of samples available for the prediction task at hop = 0 (task rows), and the number of training, validation and test samples derived from the task rows following the RelBench split. The tasks are further grouped into three categories based on the scale of the task rows. Small tasks contain fewer than 100,000 rows, medium tasks contain fewer than 1 million task rows, and large tasks contain at least 1 million task rows.

Deep relational models (DBFormer, Griffin and RT) are trained and evaluated on an NVIDIA DGX Spark GB10, a personalized AI supercomputer with a 20-core ARM CPU, 128 GB of unified memory, and Ubuntu 24.04 LTS operating system. The baseline methods for learning tabular data (LightGBM and TabPFN 2.5) are run on an Ubuntu 22.04 workstation with a 13th Gen Intel Core i9-13900F 24-core processor, 64, GB of RAM, and an NVIDIA GeForce RTX 4090 GPU with 24 GB of VRAM.

4.2 Classification performance↩︎

The single table classification task (hop = 0) reveals that RT is overall the best method (average rank 1.6) followed by LightGBM. Griffin places third, while TabPFN 2.5—arguably the best-reported method for learning tabular data, ranks fourth. The underperformance of TabPFN 2.5 may be surprising, but note that relational databases typically contain far more samples than traditional tabular datasets. It is worth mentioning that TabPFN 2.5 claims to outperform other state-of-the-art approaches on small datasets, specifically those with fewer than 10,000 samples [14]. DBformer appears as the worst of all methods. Furthermore, the DBformer authors have benchmarked it only on databases from the CTU repository, underscoring the need for this benchmarking study. An interesting exception is the rel-trial database, where RT ranks fourth. The rel-trial database contains clinical trial records, which are highly specialized and domain-specific. With only 13,779 task-specific samples and no relational context at hop = 0. Therefore, RT cannot fully demonstrate its capacity, as it is inherently designed to exploit cross-table dependencies.

The classification task that incorporates rows from tables in the immediate neighborhood (hop = 1) exhibits distinct performance rankings. RT outperforms all methods with an average rank of 1.2 followed by Griffin (2.2) and TabPFN 2.5 (3.0). An exception is TabPFN 2.5, which slightly edges out RT on the rel-avito user-visits classification task. The classification tasks in the hop = 2 experiment demonstrate the superiority of the RT method over other baselines, with an average rank of 1.0. Griffin is the second best (2.4), followed by DBformer and LightGBM. TabPFN 2.5 delivers the weakest results, underscoring that single-table learning methods fall short when tackling relational databases with rich, complex referential integrity. The superiority of RT is evident when we consider fifteen experimental cases that make up three hops and five datasets. Figure  2 (a) shows pairwise model comparisons where RT wins against LightGBM (14/15), TabPFN 2.5 (13/15), and Griffin (14/15). Griffin is the second most competitive method overall, winning in 11/15 cases against LightGBM and 12/15 against DBFormer. TabPFN 2.5 outperforms RT in two cases. The first is the rel-trial study-outcome task at hop = 0, where the small number of task-specific samples (13,779) favors TabPFN 2.5, which is specifically designed for small datasets with fewer than 10,000 samples [14]. The second is the rel-avito user-visits task at hop = 1, where the performance gap is negligible (TabPFN 2.5: 64.5 vs RT: 63.9), suggesting that this may reflect random variance rather than a systematic advantage.

4.3 Regression performance↩︎

The single table regression task (hop = 0) reveals that RT is the best method (average rank 2.0), followed by Griffin (2.8) and TabPFN 2.5 (3.1). Specifically, RT ranks 3rd and 4th on rel-avito ad-ctr and rel-stack post-vote regression tasks, respectively. Despite the worst rank, DBformer results in the best RMSE scores on the rel-avito ad-ctr regression task when RT ranks 3rd. Regression tasks that incorporate rows from adjacent tables (hop = 1) show a shift in ranking. TABPFN 2.5 matches RT in performance, with both methods achieving an average rank of 2.4. DBFormer (3.6) performs better than LightGBM (3.8) in the hop = 1 setting. In the hop = 2 configuration, RT gains the best average rank of 2.0, while TabPFN 2.5 (2.6) and Griffin (2.8) follow. The win matrix in Fig. 2 (b) summarizes pairwise model comparisons across 15 regression tasks, three hops, and five databases. RT wins on 9/15 tasks against LightGBM and DBFormer and on 12/15 against TabPFN 2.5 and Griffin. TabPFN 2.5 shows a notably stronger performance in regression than in classification, outperforming LightGBM and DBFormer on 11/15 tasks. Interestingly, TabPFN 2.5 outperforms RT on the rel-stack post-votes regression task for all three hop settings. TabPFN 2.5, which was pretrained on 130 million synthetic datasets [13], may have seen comparable distributional structures during this pretraining, which could give TabPFN 2.5 an advantage over this specific regression task.

4.4 Effects of relational hops↩︎

Intuitively, hop = 2 should outperform hop = 1, since it pulls in richer contextual row samples from related tables via referential links. However, all methods, except RT, perform worse under hop = 2 than hop = 1 in the rel-trial study-outcome classification task. The unexpected underperformance of hop=2 may be due to the uniqueness in how health records data are spread across relational tables within limited samples. RT is the only model that consistently improves across all hop settings on rel-trial, indicating its attention mechanism better exploits deeper relational context even with limited samples.

The regression results exhibit a pattern comparable to the classification results. The results of the hop=2 setting do not consistently outperform those of the hop=1 setting. The performance gap between these two hop configurations remains small for relational models (DBFormer, Griffin, and RT). For example, Griffin’s RMSE remains unchanged in rel-avito (0.82 \(\rightarrow\) 0.82) and rel-f1 (0.68 \(\rightarrow\) 0.68) from hop = 1 to hop = 2. Similarly, DBFormer shows no improvement in rel-stack (0.36 \(\rightarrow\) 0.36), and RT’s improvement in rel-amazon is minimal (0.47\(\rightarrow\) 0.45). In many cases, the improvement from hop = 1 to hop = 2 is minimal, raising the question of whether the performance gain justifies the high computational cost, as evidenced by the training times reported in Table 5. For example, Griffin’s training time increases from 202 minutes at hop = 1 to 403 minutes at hop = 2 on the rel-f1 driver-dnf task, a 2\(\times\) increase in training cost, while the AUC improvement is only 2%.

5 Conclusions↩︎

This paper presents one of the first systematic benchmarking studies of state-of-the-art deep learning methods for relational databases. Using a unified evaluation protocol with consistent data splits, computational budgets, and standardized metrics, we demonstrate that the RT method is the strongest in both classification and regression tasks. In other words, transformer-based methods are superior to conventional graph-based modeling of relational databases. Although RT and Griffin are built for relational data tables connected by primary and foreign keys, they still outperform leading state-of-the-art tabular learning approaches, such as TabPFN 2.5, in single-table (hop = 0) classification and regression benchmarks. The gain from increasing hop from 1 to 2 is often minor relative to the substantial increase in computation, casting doubt on whether expanding relational neighborhoods is worth the cost. Developing relational deep learning methods that can reliably exploit multi-hop relational context with lower computational overhead remains an important open problem for future research.

References↩︎