1 Introduction↩︎

Conditional inference separates feature selection from threshold optimization to reduce split selection bias. Classical CART trees choose both a feature and a threshold from the same label-dependent search, which helps explain the well-documented preference for features with many admissible cutpoints [1]–[7]. In the conditional inference framework [8], [9], Stage A tests features for association with the response at the current node and selects among them; Stage B then optimizes a threshold within the selected feature. That separation addresses split selection bias, but the classical approach can require substantial computation through large permutation budgets and exhaustive threshold scans at every node. The same conditional inference framework also motivated later work on conditional variable importance [10]. A related line of work uses forests directly for variable screening and ranking, rather than only for final prediction [11].

This paper evaluates conditional inference trees (CIT) and conditional inference forests (CIF) as feature selection methods. For each seed and cross-validation fold, a method produces a ranking from the training fold. Fixed downstream learners are then fit on top-\(k\) prefixes of that ranking and evaluated on held-out data, with results summarized across datasets. The benchmark asks whether tree- and forest-derived rankings put predictive variables early enough to improve fixed downstream models. We also ask whether CIT/CIF runtime hyperparameters reduce computation while preserving ranking quality, and where CIF recovery is weaker in high-\(p\) or synthetic settings. This feature-selection benchmark is separate from work on trees and forests as direct predictors or causal estimators [12]–[14]. Because several methods have tunable configurations, the benchmark chooses one configuration per method and task. As a sensitivity check, leave-one-dataset-out (LODO) analyses repeat the configuration choice after holding out each dataset, using the complete-case panels defined in 4.2.

The theorem studies the Stage A feature test at a fixed node. It assumes that the features tested at the node and the resample budget are fixed independently of the node responses, so exhaustive fixed-\(B\) \(p\)-values have a clear permutation reference distribution under the nodewise complete permutation null. The benchmarks then evaluate the tree and forest feature rankings empirically.

1.0.0.1 Contributions.

This paper contributes a real-data benchmark of conditional-inference feature rankings for top-\(k\) downstream prediction. The benchmark compares CIF with ctree, cforest, CIT, and other baselines in classification and companion regression analyses, and includes ablations of the main CIT/CIF runtime hyperparameters. The paper also states the fixed-node, exhaustive fixed-\(B\) Stage A guarantee (1 1); the full tree and forest rankings are evaluated in the benchmarks. Finally, the high-\(p\) and synthetic analyses show cases with weaker top-\(k\) recovery and how forest feature sampling can reduce how often forest splits use informative features (6).

2 Tree-growing method↩︎

At each internal node, conditional inference trees make two decisions. Stage A tests features for association with the response and chooses a feature to split. Stage B then searches thresholds only for that selected feature; if no valid split is accepted, the node becomes terminal. In this paper, CIT denotes the resulting single-tree feature selection method. CIF denotes a bootstrap ensemble of such trees with feature sampling at each node and feature rankings formed by aggregating split importances across trees. CIF-all disables feature subsampling: every nonconstant available feature can enter Stage A at each node. This changes the Stage A feature set, the Stage A multiplicity correction, and the per-node computational cost.

2.1 Stage A and Stage B↩︎

At a node \(t\), write \((X_t,Y_t)\) for the samples reaching that node. Let \(F_{t,\mathrm{avail}}\) be the feature set carried into the node by the tree-growing algorithm, including any feature muting inherited from ancestors. Let \(F_{t,\mathrm{nonconst}} \subseteq F_{t,\mathrm{avail}}\) be the features that are nonconstant on the node samples. After any feature subsampling, Stage A considers a set \(F_t \subseteq F_{t,\mathrm{nonconst}}\); its size \(m_t=|F_t|\) is the Bonferroni denominator for that node. Any label-independent auxiliary randomness used by a selector is denoted by \(U\) and is held fixed in the fixed-node guarantee.

When adaptive stopping is enabled during tree growing, Stage A can evaluate features sequentially and stop before every feature in \(F_t\) has been tested. Let \(E_t \subseteq F_t\) denote the evaluated subset; in the exhaustive fixed-\(B\) reference rule, \(E_t=F_t\). The fixed-node theorem in 3 applies to that reference rule: the features in \(F_t\) and the budget \(B\) are fixed independently of the node responses, and every \(p_{t,j}\) for \(j\in F_t\) uses that budget. Adaptive tree growing adds response-dependent node selection, inherited feature muting, and stopping-time scores used for tree construction.

2.1.0.1 Stage A (feature selection).

In the exhaustive fixed-\(B\) setting, Stage A follows the feature-selection and threshold-search separation of conditional inference trees [8]. The original conditional inference tests use the permutation-statistic framework of [15]; here, Monte Carlo label permutations calibrate the configured selector statistic. Stage A computes a selector statistic \(T^{\mathrm{sel}}_j(X_{t,j},Y_t;U)\) and permutation \(p\)-value \(p_{t,j}\) for every feature \(j \in F_t\), then selects \[p_t^{\star,\mathrm{ref}} := \min_{j \in F_t} p_{t,j}, \qquad M_t^\star := \{j \in F_t : p_{t,j}=p_t^{\star,\mathrm{ref}}\}.\] Let \(\tau_t=\alpha_{\mathrm{sel}}/m_t\) when Bonferroni is enabled and \(\tau_t=\alpha_{\mathrm{sel}}\) otherwise. If \(p_t^{\star,\mathrm{ref}} \ge \tau_t\), Stage A returns no feature and the node becomes terminal. If \(p_t^{\star,\mathrm{ref}} < \tau_t\), the selected feature \(j_t^{\star,\mathrm{ref}}\) is drawn uniformly from \(M_t^\star\) using reservoir sampling, and the node enters Stage B.

During adaptive tree growing, Stage A uses the same decision structure but can order features by a preliminary score and evaluate only the subset \(E_t\). The returned value \(q_t^{\mathrm{feat}}\) is used to grow the tree; it is not an exhaustive fixed-\(B\) \(p\)-value covered by 1. Under the exhaustive reference rule with Bonferroni correction, \(\min(1,m_t p_t^{\star,\mathrm{ref}})\) is a conservative complete-null \(p\)-value for the fixed-node Stage A test. When feature muting is enabled, features with non-rejecting Stage A \(p\)-values can be removed from descendant feature pools. When muting is disabled, descendants keep \(F_{t,\mathrm{avail}}\) except for deterministic constant feature pruning.

2.1.0.2 Stage B (threshold search).

Given a selected feature \(j_t^\star\), Stage B starts from midpoints between ordered unique values of \(X_{t,j_t^\star}\). The exact method uses all such midpoints. The random method samples a bounded subset of midpoints, while the percentile and histogram methods choose bounded representative thresholds derived from the midpoint distribution. Thresholds that would leave either child below the minimum leaf size are filtered out. If no threshold remains, Stage B returns no split.

For the remaining finite threshold set \(C_{t,j_t^\star}\), Stage B selects the threshold used by the tree. The tested statistic at threshold \(c\) is the weighted child impurity \[\frac{|Y_t^L(c)|}{|Y_t|}\mathrm{Imp}(Y_t^L(c)) + \frac{|Y_t^R(c)|}{|Y_t|}\mathrm{Imp}(Y_t^R(c)),\] where \(\mathrm{Imp}\) is the chosen node impurity measure. This statistic is used in a left-tail test, and the exhaustive fixed-\(B\) rule chooses the threshold with the smallest Stage B \(p\)-value, with uniform tie breaking. During adaptive tree growing, threshold scanning may reorder thresholds and stop after the first rejecting threshold, so the returned Stage B score is a tree-construction score rather than a global minimum over all retained thresholds. Let \(\tau_t^{\mathrm{split}}=\alpha_{\mathrm{split}}/|C_{t,j_t^\star}|\) when Bonferroni is enabled for Stage B and \(\tau_t^{\mathrm{split}}=\alpha_{\mathrm{split}}\) otherwise. If no evaluated Stage B \(p\)-value is below \(\tau_t^{\mathrm{split}}\), the node becomes terminal. After a threshold passes the Stage B test, the final minimum required-improvement check uses the corresponding weighted impurity decrease. Stage B chooses the split used by the tree; the returned Stage B score is not a valid post-selection \(p\)-value. The training skeleton in 1 summarizes this flow and, for the benchmarked CIT and CIF configurations, stores two values at each split: the Stage A permutation \(p\)-value for the selected feature and the Stage B permutation \(p\)-value for the selected threshold.

2.2 Monte Carlo permutation \(p\)-values↩︎

Let \(T_{\mathrm{obs}}\) be the observed statistic and \(T_1,\dots,T_B\) the same statistic computed on \(B\) random label permutations. For a right tail test we use the Phipson–Smyth \(+1\) \(p\)-value [16], [17], where \(\mathbf{1}\{\cdot\}\) denotes the indicator function: \[p_{\mathrm{MC}}^{\ge} := \frac{1 + \sum_{b=1}^B \mathbf{1}\{T_b \ge T_{\mathrm{obs}}\}}{B+1}.\] For a left tail test we analogously define \[p_{\mathrm{MC}}^{\le} := \frac{1 + \sum_{b=1}^B \mathbf{1}\{T_b \le T_{\mathrm{obs}}\}}{B+1}.\] Stage A uses the right tail convention and Stage B the left tail convention; ties count as exceedances in both cases.

2.3 Forests and runtime hyperparameters↩︎

CIF combines bootstrap trees and averages their predictions. For feature selection, it sums each fitted tree’s normalized split importances and renormalizes the forest vector. Each CIF tree uses the same Stage A and Stage B node rules as CIT [12], [18]. The fixed-node theorem applies only to the Stage A reference test described above. Across the CIT/CIF benchmark configurations, we hold Bonferroni correction, adaptive stopping, feature muting, feature scanning, threshold scanning, histogram-256 thresholding, and CIF bootstrap sampling fixed. The varying settings are selector choice and honesty.

Runtime depends on the realized work at each node: features tested in Stage A, thresholds generated and retained in Stage B, permutations evaluated before stopping, and features carried into the node. Adaptive stopping changes the number of permutations evaluated for each test [19], [20]. Histogram thresholding reduces the Stage B threshold set by replacing exhaustive cutpoints with binned thresholds. Feature and threshold scanning change the order in which features or thresholds are tested under early stopping; feature muting can reduce the feature pool inherited by descendants. Appendix 12 gives algorithm and reproducibility details, Appendix 13 gives runtime and memory accounting, and 5.2 reports the measured timing effects.

2.4 Feature rankings↩︎

The experiments evaluate each method through the ranking it produces on the training fold. CIT and CIF rankings are fitted split-importance rankings, not rankings of Stage A \(p\)-values. For a fitted CIT, feature \(j\) receives the sum, over internal nodes that split on \(j\), of the impurity decrease at that node: parent impurity minus the weighted child impurity. If the total importance is positive, the vector is normalized to sum to one; if the tree contributes no positive impurity decrease, the vector remains zero. CIF sums these tree importance vectors over trees and then normalizes the forest vector in the same way. Stage A determines which features enter threshold optimization, while impurity decrease scores only the accepted splits by their local reduction in impurity. Larger importances are ranked earlier, and features never used by a split receive zero importance.

This split-importance ranking differs from standard random-forest impurity importance in how a feature enters threshold search. In CART-style trees, the feature and threshold are chosen together by an impurity search, so a feature with more possible thresholds has more chances to produce a large apparent impurity decrease [7]. In CIT and CIF, Stage A first selects a feature by a permutation test; Stage B then searches thresholds only for that feature. This reduces the advantage that high-cardinality features get from having more possible thresholds. We use these fitted split-importance scores as rankings and evaluate them through downstream top-\(k\) prediction and synthetic feature recovery.

3 Fixed-node Stage A control↩︎

This section analyzes the exhaustive fixed-\(B\) Stage A feature selection rule at a fixed node. The theorem proves superuniformity for a single \(+1\) Monte Carlo permutation \(p\)-value, and the corollary applies Bonferroni correction across the fixed-node Stage A tests. In the notation of 2.1, this is the reference case where every feature in \(F_t\) is tested with the same fixed nodewise budget \(B\). All probabilities below condition on \((X_t,U)\), where \(U\) collects label-independent auxiliary randomness other than the Monte Carlo permutation draws.

This is the standard exchangeable-rank result for Monte Carlo permutation tests [16], [21], [22]. [app:permutation-exchangeability,app:proof-plusone] state the exchangeability condition and give the proof. The left-tail version follows by applying the same result to \(-T\).

Combining 1 with the Bonferroni union bound over the \(m_t\) Stage A tests gives fixed-node Stage A control under the complete null. The result is local: it does not cover partial-null familywise error control, \(p\)-values for selected features, or error control for fitted trees and forests.

4 Experimental design↩︎

The experiments ask whether rankings produced by CIT and CIF place predictive features early enough for fixed downstream learners. For each training fold, a method produces a feature ranking, and fixed downstream learners evaluate the selected features on the held-out fold. We then summarize overall ranks, how often CIF performs well across datasets, high-dimensional behavior, recovery on synthetic data with known informative features, CIF ranking ablations, and feature sampling in forests.

4.1 Benchmark setup↩︎

The benchmark uses 5 random seeds, with 5-fold cross-validation within each seed. Classification fits logistic regression (LR), support vector machine (SVM), and k-nearest neighbors (KNN) as downstream models and reports balanced accuracy, defined as mean recall across classes. Regression fits ridge regression, support vector regression (SVR), and KNN and reports \(R^2\). The main tables use the standard values \(k \in \{5,10,25,50,100\}\). Downstream learner settings are fixed. Logistic regression uses balanced class weights and a 1,000-iteration limit; SVM uses an RBF kernel with unit regularization, \(\gamma=\texttt{scale}\), and balanced class weights; KNN uses 5 distance-weighted neighbors; ridge uses unit penalty strength; and SVR uses an RBF kernel with unit regularization, \(\epsilon=0.1\), and \(\gamma=\texttt{scale}\).

For each seed and fold, feature-standardization parameters are estimated on the training fold and then applied to the held-out fold. The ranking method uses only the standardized training fold and produces the ordered feature list described in 2.4. For each \(k\), we take the top-\(k\) features from that ranking, refit standardization on the selected training columns, apply the same transform to the selected held-out columns, fit the downstream model on the selected training data, and evaluate it on the held-out fold.

The downstream evaluation also records performance using the full feature set, \(k=p\), for every dataset. For datasets with more than 100 features, the separate high-\(p\) analysis adds fixed values \(k \in \{150,200,300,500,750,1{,}000\}\) and fractional values \(\lceil 0.25p\rceil,\lceil 0.5p\rceil,\lceil 0.75p\rceil\), keeping only values no larger than \(p\). The main tables average only the standard \(k\) values.

4.2 Benchmark comparisons↩︎

The benchmark averages scores in three steps. First, for each \((\text{dataset}, \text{downstream model}, k)\) combination, scores are averaged over folds and seeds. Second, each dataset score averages those fold-and-seed means over downstream models and over the standard \(k\) values available for the methods in that comparison. Third, the tables average dataset scores over datasets.

The real-data classification benchmark contains 23 datasets. Comparisons involving partykit ctree and cforest use the 22-dataset subset that excludes dexter; comparisons among methods with dexter results use all 23 datasets. The complete-case classification analysis, used for bootstrap intervals and omnibus rank testing, contains 14 datasets with all 17 methods, all three downstream learners, and all five standard values of \(k\). Regression contains 8 datasets; the analogous rank summary uses the 6 complete-case datasets with all methods, downstream learners, and standard \(k\) values available.

By-\(k\) results are reported separately for \(k=5,10,25,50,100\). The corresponding classification dataset counts are \(22, 21, 15, 15, 14\); the regression counts are \(8, 8, 7, 7, 6\). Pairwise CIF comparisons stratified by \(k\) use the dataset counts reported with the corresponding rows. The 14-dataset classification and 6-dataset regression summaries are complete-case analyses, retaining only datasets with every method, downstream learner, and standard \(k\) value.

4.3 Datasets↩︎

The real-data benchmark includes 23 classification datasets and 8 regression datasets. The separate high-\(p\) analysis restricts attention to the 15 classification datasets and 6 regression datasets with more than 100 features. The full inventories of the real and synthetic datasets are listed in 14, which also summarizes the synthetic designs and their parameter choices.

Feature recovery uses 8 synthetic classification datasets and 8 synthetic regression datasets with known informative features and controlled noise structure. A separate study of feature sampling uses \(n=250\) and varies \(p \in \{100, 500, 1{,}000\}\) and \(n_{\mathrm{informative}} \in \{1,2,5,10\}\). That study compares CIT, a CART decision tree (DT), a random threshold tree (RT), CIF, CIF-all, random forests (RF), and ExtraTrees (ET), using 1,000 trees for forest methods. CIF-all denotes the CIF variant that evaluates all nonconstant features at each split.

4.4 Compared methods↩︎

The benchmark includes three types of feature selectors: permutation-test filters, embedded tree or ensemble methods, and wrapper baselines. Permutation-test filters compute univariate dependence statistics and rank features by their permutation \(p\)-values: multiple correlation (MC) and randomized dependence coefficient (RDC) for classification, and Pearson correlation (PC), distance correlation (DC), and RDC for regression. The RDC filter keeps the empirical-CDF random feature map from [23] but replaces the canonical-correlation solve with the maximum absolute pairwise correlation across projected columns. Distance correlation follows [24].

Embedded methods rank features using tree or ensemble importance scores. These methods are CART decision trees (DT), randomized trees (RT), CIT, CIF, random forests (RF) [12], ExtraTrees [25], XGBoost [26], LightGBM [27], CatBoost [28], and the partykit reference implementations ctree and cforest [8], [29], [30]. Wrapper baselines score features through auxiliary random-forest fits: Boruta [31], permutation importance (PI) [7], [12], conditional permutation importance (CPI) [32], and random-forest recursive feature elimination (RF-RFE) [33], [34].

In this benchmark, CPI uses stratified conditional permutation with correlation-based strata. For each target feature, it trains a 100-tree random forest on a training split and scores a 20% validation split. It then identifies validation-set features with absolute correlation at least 0.5 with the target feature, forms five-bin percentile strata from those conditioning features, and permutes the target feature within strata. If no conditioning feature meets the correlation threshold, the target feature is permuted globally.

Ranking rules use each method’s natural score direction. Embedded tree and ensemble baselines rank features by decreasing fitted importance: DT, RT, RF, ExtraTrees, XGBoost, LightGBM, and CatBoost. Permutation-test filters rank by increasing \(p\)-value, with dependence score as the secondary key. The ctree reference ranks features by split-use counts. For cforest, we use partykit’s unconditional variable-importance scores with one permutation; CIF uses split-importance rankings from accepted conditional-inference splits. The cforest results therefore refer to this benchmarked partykit importance setting. Boruta and recursive feature elimination rank by their fitted rank variables, with smaller ranks first. PI and CPI rank by decreasing importance; the CPI row uses the stratified conditional-permutation procedure described above. Ties are broken by deterministic, method-specific order for reproducibility.

4.5 Configuration selection↩︎

Several methods, including CIT, CIF, XGBoost, LightGBM, ctree, and cforest, were evaluated under multiple configurations. Permutation-test filters used one configuration per statistic. For each task, the paper reports one configuration per method, selected by mean score over real datasets, downstream models, and the standard values \(k \in \{5,10,25,50,100\}\). Methods with only one configuration are treated as fixed. The selected configuration is then held fixed across datasets for that task. Because configuration selection and reporting use the same real-data benchmark, the reported scores, intervals, and rank summaries are descriptive. As a sensitivity check, we repeat configuration selection in leave-one-dataset-out (LODO) form for both tasks. In classification, each of the 14 complete-case datasets is evaluated using configurations chosen from the other 13 datasets. In regression, each of the 6 complete-case datasets is evaluated using configurations chosen from the other 5 datasets. The downstream models, \(k\) values, and averaging within each dataset are unchanged.

For the synthetic analysis, configurations are selected separately from the real-data benchmark. Each method is assigned the configuration with the highest average recovery of the true informative features over \(k = 5, 10, 25, 50,\) and \(100\). The benchmarked CIT and CIF configurations use adaptive sequential stopping, Bonferroni correction, feature muting, feature scanning, threshold scanning, and 256-bin histogram thresholding. The benchmarked CIT and CIF variants differ in selector and honesty; CIF is run as a bootstrap forest. Each CIF runtime ablation changes one setting.

To isolate which parts of the CIF ranking matter, we run a ranking ablation on the real-data benchmark datasets. The ablation holds the selected CIF configuration fixed and changes one component at a time: replacing fitted split importance with split counts, disabling feature muting, disabling bootstrap sampling, or reducing the forest to one tree. Each variant is evaluated with the same downstream learners, folds, seeds, and standard top-\(k\) values as the main benchmark.

5 Results↩︎

5.1 Real-data benchmark↩︎

1 reports the main real-data comparison. Classification uses the 22 datasets with ctree and cforest results; the full classification collection has 23 datasets, with dexter reserved for analyses that compare methods available on that dataset. Regression uses all 8 datasets. Within each dataset, scores are first averaged over downstream learners and the standard \(k\) values shared by the compared methods, then converted to ranks. Lower mean rank is better.

With selected configurations fixed, CIF ranks 4th of 17 classification methods and 3rd of 18 regression methods. 2 compares CIF directly with ctree, cforest, and CIT. 15 gives the learner-by-\(k\) tables (16 17), pairwise CIF-comparison heatmaps (7 8), and seed-sensitivity tables (18 19); each analysis reports its dataset count.

5pt

2 shows CIF ahead on average in all six direct pairings. On the complete-case datasets, CIF is also ahead of ctree, cforest, and CIT in both tasks. The 95% bootstrap CIs for mean CIF-minus-comparator differences are as follows. Classification (14 datasets): ctree [+0.068, +0.154], cforest [+0.067, +0.160], CIT [+0.023, +0.060]. Regression (6 datasets): ctree [+0.113, +0.529], cforest [+0.040, +2.570], CIT [-0.007, +1.224]. The regression interval for CIF versus CIT crosses zero; the other complete-case intervals are positive. The regression intervals are wider because fewer datasets are available and some comparator mean \(R^2\) values are negative.

5.1.0.1 CIF ablations.

3 compares the selected CIF configuration with four targeted changes: split-count ranking, disabling bootstrap sampling, disabling feature muting, and using one tree.

4pt

Reducing CIF to one tree has the largest effect: \(-0.061\) balanced accuracy in classification and \(-0.552\) \(R^2\) in regression. Replacing fitted split importance with split counts changes the mean score by \(-0.003\) in classification and \(-0.022\) in regression, with intervals that include zero. Disabling feature muting changes both tasks by \(+0.000\), while disabling bootstrap sampling changes classification by \(+0.001\) and regression by \(-0.208\).

The complete-case Friedman tests use dataset-level method means. Classification: \(\chi^2(16)=141.32\), \(p<0.001\), Kendall’s \(W=0.63\). Regression: \(\chi^2(17)=33.79\), \(p=0.0089\), Kendall’s \(W=0.33\). The regression panel has 6 datasets, so the \(p\)-value should be read with the effect sizes, intervals, and LODO results. These intervals and rank tests summarize the benchmark-selected configurations. The LODO analysis below checks whether those configurations are stable under one-dataset holdout.

In the 14-dataset classification panel, LODO ranks CIF fifth of 17 on average (mean rank 5.64), and CIF reselects the main-benchmark configuration in all 14 holdouts. In the 6-dataset regression panel, LODO ties CIF with CatBoost for second on average (mean rank 5.50), behind ExtraTrees; CIF reselects the global regression configuration in 5 of 6 holdouts. [tab:cif-ci-comparisons,tab:lodo-config-sensitivity] give the confidence intervals and LODO details.

4 reports how often CIF appears in the top half or top 3 and how often it beats tree baselines.

6pt

In classification, CIF is in the top half on 21 of the 22 datasets with ctree and cforest results, and in the top 3 on 5 of 22. In regression, CIF is in the top half on 6 of 8 datasets and in the top 3 on 2 of 8. Against the greedy single-tree baselines, CIF beats DT on 14/23 classification datasets and 6/8 regression datasets, and beats RT on 21/23 classification datasets. The RT regression comparison is close: CIF is ahead on 5/8 datasets, with a mean difference near zero.

[fig:k-trajectory,fig:regression-k-trajectory] report mean rank separately by \(k\). In classification, CIF’s mean rank is 6.2 at \(k=5\), 7.4 at \(k=10\), 5.5 at \(k=50\), and 5.0 at \(k=100\). In regression, CIF’s mean rank is 8.9 at \(k=5\), 5.6 at \(k=50\), and 7.3 at \(k=100\). These by-\(k\) results add context to 1: CIF is strongest at larger displayed \(k\) values in classification and at \(k=50\) in regression. The aggregate ctree and cforest comparisons use 22 classification datasets. For \(k=5,10,25,50,100\), the classification columns use 22, 21, 15, 15, and 14 datasets, respectively; the regression columns use 8, 8, 7, 7, and 6 datasets, respectively. Complete-case analyses use 14 classification datasets and 6 regression datasets.

5.2 Runtime ablations↩︎

The runtime ablations identify which benchmark settings have the largest effect on fitting time and report how much the ranking scores move when those settings change. Each row changes one setting and reports elapsed wall-clock fit time relative to the selected configuration for the same estimator. Ratios above 1 are slower than that reference, and ratios below 1 are faster. CIT and CIF are shown separately because the CIT run records fit time and synthetic top-10 feature recovery, while the CIF runs also record downstream score changes. The main benchmark remains the primary performance comparison. 13 links each ablation to the affected work terms.

6pt

5.2.0.1 CIT runtime ablations.

5 shows that CIT timing effects vary by data regime. Disabling adaptive stopping reduces runtime in the synthetic runs and has less consistent timing on real datasets. For a single fitted tree, changing this option can also change the realized sequence of features, thresholds, and nodes evaluated. These rows focus on timing and synthetic feature-recovery changes.

6pt

5.2.0.2 CIF runtime ablations.

6 shows consistent slowdowns for two CIF changes. Disabling adaptive stopping makes fits 4.0–8.4\(\times\) slower by median dataset-level ratios. In the separate threshold-search ablation, exact threshold search is 1.9–10.8\(\times\) slower than histogram-256 by ratios of group mean elapsed times. Downstream score changes and synthetic feature-recovery changes are at most 0.011 and 0.017. The feature-scanning effect is smaller and less consistent once adaptive stopping is enabled. Removing Bonferroni correction is faster for both CIT and CIF, but it changes the statistical rule and yields trees with more selected features; the selected setting keeps Bonferroni correction.

6 High-dimensional and synthetic analyses↩︎

6.0.0.1 High-dimensional real datasets.

These analyses study where CIF’s scores peak as \(k\) grows, how feature recovery on synthetic datasets changes across values of \(k\), and how feature sampling inside the forest affects how often splits use informative features. Across the 15 classification datasets with more than 100 features, CIF first reaches its best score at an intermediate value of \(k\) (\(100 < k < p\)) in 30 of 45 dataset-learner combinations. It first reaches its best score at the full feature set (\(k=p\)) in only 5 of 45 combinations. 4 shows this pattern by downstream learner; regression is summarized next. Classification scores most often peak at intermediate \(k\): 11 of 15 LR datasets, 9 of 15 SVM datasets, and 10 of 15 KNN datasets. The full feature set is mainly an LR exception. Moving from \(k{=}100\) to \(k{=}p\) gives a mean \(k{=}p\) minus \(k{=}100\) score change of \(+0.022\) for LR, and \(k=p\) matches the best observed LR score in 6 of 15 datasets. For SVM and KNN, the corresponding mean changes are \(-0.058\) and \(-0.075\), and \(k=p\) is best in only 1 of 15 datasets for each learner.

In the 6 high-\(p\) regression datasets, full-feature evaluation is rarely best. Across the 18 dataset-learner combinations, 11 peak at or below \(k{=}100\), only 2 peak at \(k=p\), and the mean \(k{=}p\) minus \(k{=}100\) score change is \(-0.308\). The main full-feature exception remains the classification LR result in 4.

6.0.0.2 Synthetic feature recovery.

Across synthetic classification datasets, 34% of the features in CIF’s top-\(k\) list are truly informative on average over the evaluated \(k\) values, ranked 8th of 17, and its top-1 hit rate is 64%, ranked 12th of 17. Across synthetic regression datasets, the corresponding values are 39%, ranked 10th of 18, and 81%, ranked 15th of 18. In both tasks, CIF recovers true features within the top-\(k\) list more reliably than it places one first. 7 summarizes this contrast, and 5 shows recovery as \(k\) grows. The percentage of selected features that are informative falls as \(k\) grows. On redundant-feature synthetic designs, most selected features correspond to true signals or redundant proxies: 83% for classification and 84% for regression.

6pt

6.0.0.3 Feature sampling in forests.

A sparse simulation asks whether forest feature sampling makes splits less likely to use informative features in classification and regression settings. 6 shows one sparse classification design comparing CIF with CIF-all. CIF samples a subset of features at each split; CIF-all considers every nonconstant feature in Stage A. 16 gives the classification and regression results over \(p \in \{100,500,1{,}000\}\).

In the single-tree simulations, CIT uses informative features in 100.0% of classification splits and 93.5% of regression splits. In this fixed design, informative features are usually selected when every nonconstant feature is considered. In forests, sampling a subset of features reduces how often informative features are used for splits. Across the classification forest simulations with 1,000 trees, CIF uses informative features in 33.2% of splits, compared with 91.7% for CIF-all, 8.4% for RF, and 6.5% for ET. CIF also uses more distinct noise features than CIF-all, 265 versus 93 on average. For the \(p{=}1{,}000\) classification setting with two informative features, CIF uses informative features in 9.0% of splits, while CIF-all uses them in 100.0%. In regression, CIF-all uses informative features more often than CIF, 42.3% versus 27.8%, but uses more distinct noise features, 528.8 versus 367.7. CIF-all isolates feature subsampling because feature muting and the other tree settings are held fixed. In these sparse high-\(p\) simulations, splits use informative features most often when every nonconstant feature can enter Stage A. When the sampled feature subset rarely includes an informative feature at a node, Stage A selects from that subset.

7 Discussion↩︎

In these experiments, CIF is best supported as a top-\(k\) feature-selection method for a downstream model. The benchmark asks whether a ranking places predictive features early enough for fixed downstream learners. Classification gives the clearest support; the smaller regression panel is positive but less precise. The fixed-node Stage A theorem provides the local guarantee; fitted tree and forest rankings are assessed empirically.

7.0.0.1 Guidance.

A CIF ranking should be judged through the downstream model that will use the selected features. In practice, choose \(k\) on held-out data over a small set of values that includes the intended feature budget and, when feasible, the full feature set as a sanity check. When prediction accuracy is the main target, tree-ensemble baselines such as random forests, ExtraTrees, and boosted trees should remain in the comparison set. When the target is variable screening or a ranked feature list for later modeling, CIF should also be compared with ctree, cforest, and forest-based screening methods such as Boruta, permutation or conditional permutation importance, and RF-RFE [11].

The ablations show that adaptive stopping and histogram-256 thresholding account for the largest measured CIF runtime changes, with small downstream score changes in these experiments. Bonferroni correction is part of the Stage A testing rule and is not treated as a runtime shortcut. For ranking quality, forest size matters more than the split-count and feature-muting variants tested here, so the selected CIF configuration is the relevant ranking reference when ranking quality is the priority.

In sparse high-\(p\) settings, feature use should be monitored directly. The simulations show that CIF feature sampling can leave informative features out of many forest split decisions, especially in sparse classification designs. Useful diagnostics include top-\(k\) stability across seeds, the share of splits using the final top-\(k\) features or any prespecified positive control features, the number of distinct low-ranked or known null features used in splits, and sensitivity to larger sampled feature sets or to a small CIF-all run with fewer seeds or trees.

7.0.0.2 Limitations.

The fixed-node result is a reference guarantee for exhaustive fixed-\(B\) Stage A testing at one node, with the tested features and permutation budget fixed independently of the node responses. The benchmarked trees and forests add adaptive stopping, feature and threshold ordering, feature muting, bounded threshold sets, bootstrap samples, and forest aggregation, so their behavior is assessed through the ranking benchmark.

The benchmark coverage is uneven across analyses. Some comparisons use broader panels where both methods are available, while complete-case rank and interval summaries use smaller panels that include every method. These summaries are therefore descriptive rather than confirmatory.

The sampled-forest analysis isolates a sparse high-\(p\) setting in which forest feature sampling can limit how often splits use informative features: larger sampled feature sets increase informative-feature use, but they also change runtime and the mix of noise features used by splits. Track how often splits use informative features alongside downstream performance and runtime before changing the number of features sampled at each split.

8 Conclusion↩︎

This paper evaluates conditional-inference tree and forest rankings for top-\(k\) downstream prediction and proves a fixed-node guarantee for Stage A permutation testing. The experiments support CIF as a top-\(k\) feature-ranking method, especially in classification: its rankings compare favorably with ctree, cforest, and CIT, and they remain competitive among stronger tree-ensemble baselines.

The ablations separate runtime and ranking effects. Adaptive stopping and bounded threshold search avoid large runtime increases, while the one-tree ablation has the largest ranking-quality loss. The high-\(p\) and synthetic analyses add two diagnostic cautions: sparse forests should be checked for whether informative features are used in enough splits to enter the top-\(k\) ranking, and CIF should not be treated as a first-feature discovery method when the goal is to identify a single driver. Overall, these results support CIF as a feature-ranking method for downstream prediction, backed by a local Stage A result and benchmark evidence.

References↩︎