1 Introduction↩︎

Ensemble methods improve the performance of individual models, called base learners, by combining them. The set of base learners of an ensemble model, its pool, is homogeneous if every base learner is of the same family, and heterogeneous otherwise. Homogeneous tree-based ensemble models have been considered some of the best performing models in tabular classification for years [1]. If much has been written about their performance, the capabilities of heterogeneous ensemble models’ have been less studied (see Section 2), despite their prevalence in machine-learning competitions such as on Kaggle.

In this work, we provide the following contributions:

• a thorough experimental study of the performances of 9 different parallel ensemble methods using a diverse and heterogeneous pool;

• a new set of “best practice” recommendations on the use of such ensemble methods, for small to large datasets;

• a validation of these recommendations using TabArena’s precomputed data [2];

• and two key observations, namely (1) that Blending and Stacking are inconsistent, but their inconsistencies are independent and happen on different tasks and (2), while Hard Voting’s probabilistic classification is rather weak, Robust Soft Voting’s own probabilistic classification is particularly successful, especially in the multiclass case.

The structure of the paper is as follows. After surveying the related work (Section 2), we evaluate ensemble methods in a first exploratory study on a simple pool (Section 3). Then, we make recommendations (Section 4) that we validate in a second study (Section 5), closer to real-life conditions, using TabArena’s pre-computed results, before concluding (Section 6).

2 Related Work↩︎

Previous studies have often focused on homogeneous ensemble models [3] [4], but works comparing heterogeneous ensemble models do exist. Creators of ensemble methods and AutoML researchers interested in post-hoc ensembling have compared the results of different ensemble methods in different ways.

In the first case, studies have focused on a specific subset of ensemble methods, such as Stacking [5], or were done using a few datasets [6]. The work most similar to ours’ is [5], which compared many types of Stacking techniques with both Hard Voting and Single Best on \(30\) different datasets. Our work differs in terms of ensemble methods, pools, datasets, and objective, as we have access to more modern classifiers, test a larger curated set of datasets [[7]][2] and don’t focus only on Stacking.

In the second case, the comparison of different ensemble methods was often not the focus. The closest work to ours lies in the appendix of the Assembled OpenML text [8], which provided a quick comparison of different ensemble methods based on base learners found by a CASH method, optimizing both the choice of the family of learner as well as their hyperparameters. This represented well the results of an AutoML method but left in some cases a homogeneous pool. In [9], Ensemble Selection [6] and Stacking were quickly compared in the context of post-hoc ensembling with the first being considered faster and more robust. Recently, [10] offered a comparison of several modern post-hoc ensembling methods. Our approach deals with a wider spectrum of ensemble methods, while the resulting recommendations have been shown to generalize to large datasets.

Taking together both perspectives, prior comparisons either focused on a restricted set of methods, typically Stacking versus Hard Voting, or used homogeneous pools, outdated base learners, or small dataset collections. Our work extends this landscape in three ways: (1) we evaluate a broader set of parallel ensemble methods, including cluster-based dynamic selection and the median aggregation rule; (2) we use a deliberately diverse heterogeneous pool that includes modern classifiers (XGBoost, multiple MLP architectures); and (3), more relevant for the end-user perspective, we translate empirical observations into a concrete decision procedure and validate it on an independent benchmark. To our knowledge, no prior work simultaneously provides recommendations and validates them empirically on a separate benchmark suite.

3 Exploratory Experiments↩︎

We select a set of widely used heterogeneous ensemble methods and apply them on tasks from OpenML CC18 [7].

3.1 Ensemble Methods and Base Learners↩︎

3.1.1 Ensemble Methods↩︎

Our approach assumes that no estimation of the performance of base learners on the given tasks is available. This differs from cases such as post-hoc ensembling [9] where base learners are first trained, evaluated, and then selected for an ensemble based on their evaluation. We also focus on ensemble methods whose base learners’ training is independent, to allow a base learner to be reused across ensemble methods with the same training scheme.

We studied three groups of model-agnostic ensemble methods: voting, stacked generalization and dynamic selection.

• We chose Soft Voting [11], Hard Voting [11] and Bagging [12] because of their wide use. For Hard Voting’s probabilistic classification, we use the proportion of voters for each class, a known yet rarely used technique. We also added Robust Soft Voting - the median rule - for its robustness.

• We also added Stacking [13] and Blending [14] for their popularity. Both use base learners’ predictions to generate a meta-dataset on which they train a final estimator, a logistic regression in our case. Their difference lies in the base learners’ training. In Blending, they are trained once through a \(90/10\) training/holdout scheme. In Stacking, base learners follow a \(5\)-fold cross validation to create the meta-dataset from out of fold predictions, then the base learners used for predictions are trained on the whole training set.

• Finally, we included a Dynamic Classifier Selection (DCS) method and a Mixture of Experts (MoE) method, which part the feature space using clustering techniques.

Details about the methods and their implementation may be found in Appendix A.

3.1.2 Base Learners↩︎

We chose a diverse set of well-known classifiers with different inductive biases, representative of typical practical uses: two Decision Trees (one with a limited depth, the other, without), two SVM (with linear and RBF kernels), a very simple model giving the Most Common class, a Gaussian Naïve Bayes model, a Logistic Regression, a Random Forest, a XGBoost, and three MLPs (one with the default size (\(5 \times 10\)), one with only one long hidden layer (\(1 \times 100\)) and one with four times the size of the default (\(10 \times 20\))).

These models come from the scikit-learn library [15], with the exception of the popular XGBoost [16] and Most Common. In most cases, we kept the default values instead of biasing the results with the authors’ choice of hyperparameter values. We only chose different hyperparameter values to avoid overly long training times, and to make variations, if we expected them to have a major impact on the results.

Each dataset was preprocessed as follows: (1) missing features were imputed; (2) we used a one-hot encoder on categorical features and dropped constant features; and (3) each feature’s mean and variance were scaled between \(0\) and \(1\).

Implementation details may be found in Appendix B.

3.2 Methodology↩︎

The performances of ensemble methods were measured using mainly their mean ROC AUC OVO from five repetitions of a \(10\)-fold cross validation.

As mentioned above, we used the OpenML CC18 [7] suite of classification tasks, but removed the \(16\) biggest datasets - those with more than \(500,000\) cells - due to limited computational resources. This left us with \(28\) binary tasks and \(28\) multiclass tasks.

3.3 Results↩︎

3.3.1 General Results↩︎

1 shows the mean score (from low, 0, to high, 1), mean rank (lower is better) and ratio of victories (being the best method) of the ensemble methods for the ROC AUC and two classification metrics, Matthew’s Correlation Coefficient (MCC) and accuracy.¹

1 reveals a clear metric-dependent split. In probabilistic classification (ROC AUC), Robust SV leads, followed by Soft Voting and Stacking; the spread across methods is modest (\(0.026\) between best and worst). In discriminative classification (MCC, accuracy), Stacking and Blending lead, while Robust SV drops to rank \(4.5\) and \(4\). This divergence arises because the median aggregation rule smooths probability estimates toward the center of the simplex, which benefits calibration and AUC but can hurt hard-decision boundaries. Hard Voting performs poorly in probabilistic classification because its class probabilities are estimated by the vote proportion, a coarse approximation of the true posterior[17].

3.3.2 Specific-Task Results↩︎

We make a more fine-grained study of the models (see 1), using heat maps of the ROC AUC OVO mean rank of each ensemble method for each dataset. Datasets are referenced by their OpenML ID [18], and ordered, in the figure, according to their number of samples (smaller to larger from left to right).

First, we see that Stacking and Blending have independent consistency problems. While having the best overall performances, each of them is one of the two worst methods in \(11\) out of \(56\) tasks. Robust SV and Soft Voting are more consistent.

Secondly, results change as datasets increase in size. We see this better in 2. Voting-based methods relatively worsen as datasets become larger, while other methods improve.

3.3.3 Importance of Base Learners↩︎

While voting methods give the same importance to every base learner, others do not. Stacked generalization methods’ final estimators give a different weight to each base learner while selection methods only select one base learner for each sample. We show in 2 a measure of the importance given to each base learner by each ensemble method, a ratio between \(0\) and \(1\). For stacked generalization, we use the average absolute weight given to each base learner by the final estimator. For selection methods, we use the percent of samples for which a base learner was selected over the others.

In the first case, correlated base learners’ results could lead to arbitrarily high weights. We thus monitor the proportion of weights with an illogical sign, such as a negative weight for the probability of class A given by a base learner when computing the probability of class A by the ensemble model. They are in light green in 2.

4 Analysis↩︎

4.1 Stacking↩︎

Stacking has the best classification performance (1) and is able to ignore lower-performing base learners (2). However, its training scheme is time-consuming and different from all the ensemble methods studied here, meaning most of its base learners aren’t usable by other ensemble methods. Finally, while its results are usually good, they are inconsistent, the method performing badly on some tasks (1).

Blending is fairly similar. It has the second best classification performance (1) and the same ability to ignore the worst base learners (2). It performs slightly worse on smaller datasets, likely not having enough samples to train its final estimator. It’s faster to train, and its results are slightly less inconsistent (1). Its base learners’ simple training scheme allows other ensemble methods to opportunistically use them.

Interestingly, while Stacking and Blending both occasionally performed badly, with every repetition showing the same low performance, they didn’t do so on the same tasks. Their inconsistencies appeared independent, which implies they don’t come from the base learners, but from the final estimator’s training.

4.2 Voting↩︎

Voting methods performed well on smaller datasets, but their performance decreased as the size of datasets increased (2).

This behavior is consistent with a bias-variance analysis of ensemble methods [19]. In voting ensembles, all base learners contribute equally to the final prediction regardless of their individual quality. On small datasets, variance dominates for most learners and uniform averaging provides genuine variance reduction. As dataset size grows, the bias of weaker learners (e.g., Shallow Decision Tree, Most Common) becomes the limiting factor, their error is irreducible given the model class and continues to be weighted equally with stronger learners. Methods with learned weights (Stacking, Blending) and those performing implicit selection (Cluster DCS, MoE) are less susceptible to this regime change, though they introduce their own instabilities. Pruning the pool to retain only competitive base learners would be expected to restore the advantage of voting methods on larger datasets, a direction which could be explored in future work.

4.3 Dynamic Selection Methods↩︎

Cluster DCS and MoE didn’t perform well (1), each only being the best method in one case (1). Though they are able to ignore the worse base learners, they have a tendency to overly rely on a single model in many parts of the feature space (2). Of note, the single time Cluster MoE was the best method, it used XGBoost for \(99.6\%\) of its predictions.

4.4 Recommendations↩︎

Based on the previous experiment and analysis, we provide evidence-based, easy-to-use recommendations regarding which ensemble method and training scheme to use in practice (see Table 3). They can all be read and used simultaneously, enabling people using multiple methods to either choose the best-performing one after testing them or to combine their results.

5 Validation on TabArena↩︎

We validate our recommendations through a set of experiments using TabArena’s precomputed results [2].

5.1 Methodology↩︎

TabArena provides the predicted probabilities of many modern classifiers for each sample of a curated list of datasets [2]. It’s thus possible to simulate ensembling these classifiers as well as running AutoML optimizers. Predictions come from the repetitions of \(3\)-fold cross validations. To ensemble them and train the ensemble methods we implemented (see above), we further nested a \(3\)-fold cross validation in each test fold, leaving \(11\%\) of samples for ensemble training.

We assess the quality of our recommendations by comparing its results to the subset of ensemble methods implementable on TabArena, using a different diverse set of base learners to ensure our recommendations are model-agnostic: LightGBM, ExtraTrees, Modern NCA, RealMLP, Logistic Regression, ExplainableBM, xRFM and RealTabPFN 2.5. When multiple ensemble methods were recommended, we averaged their predictions. Stacking was not simulated.

To get close to real conditions, we performed a CASH optimization through the simulation of a \(250\)-iteration random search for every task and used the top \(50\) model instances as base learners, reducing diversity but selecting only well performing base learners. Both this random search and the following experiments use ROC AUC, for the \(22\) binary tasks, and negative log loss (NLL), for the \(6\) multiclass tasks. We ensured no dataset from the exploratory study was present.

We had two testable questions: (1) is each ensemble method better than Single Best, and (2) are our recommendations better than the existing ensemble methods? We used Wilcoxon signed rank tests to test them, with a significance threshold \(\alpha = 0.05\) and a Holm-Bonferroni correction [20].

5.2 Results↩︎

4 and 5 show the general results we obtained, as well as the results of the significance tests². Results differ from the previous experiment (Section 3) in a few ways. Soft Voting and Robust SV, on one hand, show even stronger results on the multiclass tasks. Single Best, on the other hand, performed badly, being significantly worse than every method, except for Hard Voting.

Our set of recommendations (Recs) performed well, with slightly better results than Blending, especially in the multiclass case. Though it wasn’t significantly better than the best methods, its results were more consistent than their’s.

However, it may be necessary to update our recommendation \(5.\) In our exploratory study, Single Best achieved a mean multiclass ROC AUC rank of \(3.0\), comparable to Robust SV (\(3.1\)) and Stacking (\(3.1\)). However, this parity disappears in the validation study, where Single Best is significantly worse than all ensemble methods except Hard Voting. We therefore recommend Single Best only when a dominant base learner has been identified through cross-validation, as its otherwise weaker average behavior does not justify its use as a default.

5.3 Stacking results↩︎

Implementing Stacking in TabArena required more adaptations than the other methods. We had to use two independent \(3\)-fold cross-validations for the base learners and for the meta-dataset. However, data leaked. Specifically, the learners used to create the meta-dataset (but not for predictions) were each trained on part of the test set. For this reason, we separated its results from the rest.

6 shows the results after including Stacking. Despite data leaks benefiting it, the performance Stacking showed here were considerably more mixed than in the previous study. While it was the best method in \(8\) out of \(28\) cases, it was also the worst or second worst method in another \(10\) out of \(28\) cases.

Surprisingly, all of Stacking’s wins on binary tasks occurred on datasets where our recommendations would have otherwise won. Stacking seems at its best when the rest of our recommendations are already strong, and generally worse on other tasks. Considering its mediocre results here, it might be wise to avoid using it when combining recommendations by averaging them.

6 Conclusion↩︎

In this work, we assessed the performance of several heterogeneous ensemble models and introduced new recommendations for their use, which we validated empirically. To enable this validation, we extended TabArena to allow the use of different ensemble methods.

Specifically, we noted that Hard Voting shouldn’t be used for probabilistic classification. Stacking is a strong ensemble method that suffers from inconsistencies, particularly in the multiclass case and when base learners have correlated results. Blending is a version of Stacking with lower performance in the best case but with less inconsistencies. Interestingly, they perform badly in different cases. Soft Voting is a good alternative on multiclass cases, although it suffers from bad base learners; Robust SV suffers slightly less from this problem. Single Best is a good alternative if a base learner is much stronger than the others; otherwise, it is significantly worse than ensemble methods.

Further work on optimizations for ensemble models such as pruning and weight optimization techniques would compliment the current work. A larger-scale comparison using the base learners of the second experiment with all methods of the first experiment would also allow a more thorough comparison.

7 Ensemble methods↩︎

We briefly describe the ensemble methods used and the implementation for the exploratory experiments.

7.1 Description↩︎

We use the following notations for all definitions. Let \(\mathcal{X}\) be the feature space for a classification problem with a set of classes \(C = (c_k)_{k \leq n_C}\). Let \(B = (b_i)_{i \leq n_B}\) be a set of trained base learners whose respective error is \(\epsilon^{(i)}_{A} \in \mathbb{R}^{+}, \forall A \subset \mathcal{X}\), whose class predictor is \(h^{(i)}: \mathcal{X} \mapsto C\), and whose probability predictor is \(f^{(i)}: \mathcal{X} \mapsto \Delta^{n_C}\), with \(\Delta^{n_C}\) the standard \(n_C\)-simplex. \(\forall i \leq n_B, \forall k \leq n_C,\) let \(f_k^{(i)}: \mathcal{X} \mapsto [0;1]\) be the probability predictor for \(c_k\), such that \(f^{(i)} = [f_k^{(i)}]_{k=1}^{n_C}\). \(\forall i, j \in \mathbb{N}^2\), let \(\delta_{ij}\) be the Kronecker delta. Let \(x \in \mathcal{X}\) be an unknown sample.

Soft Voting uses the arithmetic mean of the base learners’ class probabilities for its predictions. Fumera and Roli [21] analyzed its decision boundary [22], and showed the method is able to reduce its base learners’ added error - the non-intrinsic part of the error - by a factor equal to the size of its pool in the best case. Its probability predictor \(f^{sv}\) is defined as

\[f^{sv}(x) \triangleq [\operatornamewithlimits{mean}_{i \leq n_B}({f_k^{(i)}(x)})]_{k=1}^{n_C} = \frac{1}{n_B}\sum_{i = 1}^{n_B}{f^{(i)}(x)}~.\]

Robust Soft Voting uses the median of the base learners’ class probabilities for its predictions, normalized to remain a probability. It’s a variant of Soft Voting that uses a more robust measure of central tendency. Its probability predictor \(f^{rsv}\) is defined \(\forall x \in \mathcal{X}\) as the only value in \(\Delta^{n_C}\) that satisfies

\[f^{rsv}(x) \propto [\operatornamewithlimits{median}_{i \leq n_B}({f_k^{(i)}(x)})]_{k=1}^{n_C}~.\]

Hard Voting or Plurality Voting uses its base learners’ predictions as votes and chooses the class with the most votes. The jury theorem [23] states, under several strict hypotheses, that its accuracy should increase with the size of the pool and converge to the best possible predictor with an infinite pool for binary classification. Its class predictor \(h^{rsv}\) and probability predictor \(f^{rsv}\) are defined as

\[h^{hv}(x) = \arg\!\max_{c \in C}(\left|\{i \leq n_B, h^{(i)}(x) = c\}\right|)~\text{and}\] \[f^{hv}(x) = [\frac{1}{n_B}\left|\{i \leq n_B, h^{(i)}(x) = c_k\}\right|]_{k=1}^{n_C}~.\]

Stacking and Blending use the predictions of its base learners as features to train a logistic regression. It leads to the creation of an unconstrained weighted soft-voting model by class with weights \((\beta^{(i)})_{1 \leq i \leq n_B} \in (\mathbb{R}^{n_C})^{n_B}\) as well as an intercept \(\beta^{(0)} \in \mathbb{R}^{n_C}\), both given to the non-linear bijective logistic function \(\sigma\). If properly optimized and in the best case, a weighted soft-voting model should theoretically completely remove its base learners’ added error [21]. Its probability predictor \(f^{st}\) is defined as

\[f^{st}(x) = [\sigma(\beta^{(0)}_k + \sum_{i = 1}^{n_B}{\beta^{(i)}_kf^{(i)}(x)})]_{k=1}^{n_C}~.\]

Dynamic Classifier Selection first trains the base learners on all samples, then partitions the feature space into a disjoint union \(\mathcal{X} = \bigsqcup_{j = 1}^{n_X}{\mathcal{X}_j}\) and selects the best performing base learner on each part. During prediction, only the base learner corresponding to the part the sample falls in is used for prediction. As the best performing base learner is selected for each part, the resulting ensemble model is expected to perform better than any of its base learners. Its probability predictor \(f^{dcs}\) is defined \(\forall j \leq n_X\), with \(a_j \triangleq \arg\!\min_{l \leq n_B}{\epsilon^{(l)}_{\mathcal{X}_j}}\), if \(x \in \mathcal{X}_j\), by

\[f^{dcs}(x) \triangleq {\sum_{i = 1}^{n_B}{\mathbb{\delta}_{a_j i}f^{(i)}(x)}}~.\]

Mixture of Experts first partitions the feature space, then trains different instances of all base learner on each part \(\forall j \leq n_X, f^{(i, j)} : \mathcal{X}_j \mapsto \Delta^{n_C}\) , and selects the best performing base learner on each part. Similarly to spline interpolation, base learners are expected to model simple parts of the true distribution. Its probability predictor \(f^{moe}\) is defined \(\forall j \leq n_X\), with \(a_j \triangleq \arg\!\min_l{\epsilon^{(l)}_{\mathcal{X}_j}}\), if \(x \in \mathcal{X}_j\), by

\[f^{moe}(x) \triangleq {\sum_{i = 1}^{n_B}{\mathbb{\delta}_{a_j i}f^{(i, j)}(x)}}~.\]

7.2 Exploratory experiments↩︎

While we created custom ensemble methods for the validation experiments, we were able to use and customize existing implementations for the exploratory experiments.

8 Base learners↩︎

8.1 Exploratory experiments↩︎

We note the hyperparameters used for each base learner in the exploratory experiment. Hyperparameters that aren’t explicitly given were left to their default values in their given library.

The libraries we used were scikit-learn 1.7.1 [15] and xgboost 3.0.4 [16].

Note that the logistic regression used as a final estimator in Blending and Stacking uses the same hyperparameters described above.

8.2 Validation experiment↩︎

For the validation experiments, we used TabArena’s results [2] and their hyperparameter ranges. For most base learners, they are noted in the appendix C of the TabArena paper. The ranges for xRFM and RealTabPFN are not noted; we show their hyperparameter values and ranges, taken from TabArena’s code, below:

References↩︎

---

1. Measures and further details may be found at .↩︎

2. Measures may be found at .↩︎