1 Introduction↩︎

Collaborative inference across multiple agents can improve prediction quality, but it also raises a practical question: when is collaboration actually worth its cost? In many multi-agent systems, different clients observe complementary views of the same sample, and a stronger prediction can be obtained by combining their information at a server. Vertical federated learning (VFL) provides a natural framework for this setting: each client processes its own local view and only selected information is communicated for joint prediction [1], [2]. The difficulty is that this more accurate collaborative route is also the most expensive one, as it may require transmitting intermediate representations rather than lightweight local predictions. This trade-off is especially relevant at inference time. A system may be able to form a quick prediction from local client outputs alone, while a stronger collaborative prediction is available if additional communication is allowed. In practice, it is rarely necessary to invoke that stronger route on every sample. Some inputs are already easy for the local route, while others are precisely the cases where escalation to collaborative inference is most useful. The challenge is therefore to decide, on a per-sample basis, whether the extra communication is justified.

We study this question in a two-round VFL protocol. In Round–1, each client sends a local posterior and the server forms a pooled prediction. In Round–2, if the sample is escalated, clients send higher-dimensional embeddings that are fused by a stronger collaborative model. The routing problem is to decide, after Round–1, whether switching to the Round–2 predictor is likely to improve correctness enough to justify the additional cost.

Our approach is based on a simple idea: treat escalation as an expected-gain decision. Rather than learning a separate router from routing labels, we estimate the utility of escalation analytically from quantities that can be computed on held-out calibration data. The resulting score depends on two ingredients: a calibrated version of the pooled Round–1 posterior, which estimates the plausible class labels after the first round, and classwise reliability statistics of the Round–2 VFL model, which capture how often the collaborative route is correct for each class. Together, these quantities yield a threshold-based routing rule that is easy to interpret and straightforward to deploy.

This work makes the following contributions:

• We study inference-time selective escalation in a two-round collaborative inference setting, where a lightweight initial prediction route can optionally invoke a costly collaborative VFL model.

• We propose an analytical routing rule based on expected gain, using only the pooled Round–1 posterior and classwise reliability statistics of the collaborative model.

• We empirically validate the proposed approach across multiple multi-view classification tasks and deployment conditions, showing that it achieves a favorable accuracy–escalation trade-off and consistently outperforms learned-gain-, confidence-, and deferral-based baselines.

2 Related Work↩︎

Our work sits at the intersection of communication-efficient vertical federated learning, selective prediction and deferral, and adaptive inference.

A substantial body of work in VFL studies how to reduce the communication and computation cost of collaborative training and inference. Existing directions include compression and quantization, sparse communication, feature selection, limited-overlap protocols, and caching or reuse of intermediate representations [3]–[7]. These approaches are primarily concerned with making collaborative inference cheaper once it is performed. By contrast, our focus is on a different question: given a trained collaborative route, when should it be invoked at all at test time?

Our work is also related to selective prediction and reject-option classification, where a model may abstain when it is uncertain [8], [9], and more closely to learning-to-defer methods, where a predictor decides whether to keep its own prediction or hand the example to another model or expert [10]–[13]. These lines of work provide the broader decision-theoretic context for our setting: a weak route is available immediately, while a stronger alternative may be used selectively.

The closest conceptual connection is to adaptive inference and classifier cascades, where additional computation is used only when its expected benefit justifies the extra cost [14], [15]. Our two-round protocol fits naturally into this perspective: Round–1 provides a cheap pooled prediction, and Round–2 provides a stronger collaborative one. The main difference is in how the routing signal is constructed. Rather than learning a separate router or explicitly modeling the unseen output of the stronger route, we propose a plug-in approximation to the expected gain of escalation based on two ingredients estimated on held-out data: a calibrated posterior from the pooled Round–1 prediction and classwise reliability statistics of the Round–2 model.

3 Problem Setup↩︎

We consider a multi-agent classification problem with \(M\) clients and \(K\) classes in a vertical federated inference setting, where different clients observe complementary partial views of the same underlying sample and a server combines the communicated information to produce a prediction [1]. A sample is drawn from an unknown distribution over \[(\boldsymbol{x}_1,\ldots,\boldsymbol{x}_M,y)\in \mathcal{X}_1\times\cdots\times\mathcal{X}_M\times\{0,\ldots,K-1\},\] where \(\boldsymbol{x}_i\) denotes the observation available to client \(i\) and \(y\) is the ground-truth class label.

Each client \(i\in\{1,\ldots,M\}\) is equipped with:

• a local classifier \[f_i:\mathcal{X}_i\to \Delta^{K-1},\] which outputs a local posterior \[\boldsymbol{p}_i = f_i(\boldsymbol{x}_i)\in \Delta^{K-1},\] where \(\Delta^{K-1}\) denotes the probability simplex over \(K\) classes;

• a feature encoder \[E_i:\mathcal{X}_i\to \mathbb{R}^{d_i},\] which outputs an embedding \[\boldsymbol{e}_i = E_i(\boldsymbol{x}_i).\]

3.1 Two-Round Inference↩︎

We study a two-round inference protocol in which a cheap Round–1 local route prediction may optionally be replaced by a stronger but more communication-intensive collaborative prediction.

3.1.0.1 Round–1: local posterior communication.

Each client sends only its local posterior \(\boldsymbol{p}_i\) to the server. The server constructs an aggregate posterior by average pooling: \[\bar{\boldsymbol{p}} = \frac{1}{M}\sum_{i=1}^M \boldsymbol{p}_i . \label{eq:avg95pool}\tag{1}\] The corresponding cheap prediction route is \[\hat{y}_{\mathrm{loc}} = \mathop{\mathrm{arg\,max}}_{k\in\{0,\ldots,K-1\}} \bar{\boldsymbol{p}}(k). \label{eq:y95loc}\tag{2}\]

3.1.0.2 Round–2: collaborative embedding fusion.

If the sample is escalated, each client computes an embedding \(\boldsymbol{e}_i=E_i(\boldsymbol{x}_i)\) and transmits it to the server. A server-side VFL fusion model \[g_{\mathrm{VFL}}:\mathbb{R}^{d_1}\times\cdots\times\mathbb{R}^{d_M}\to \Delta^{K-1}\] produces the collaborative posterior \[\boldsymbol{q} = g_{\mathrm{VFL}}(\boldsymbol{e}_1,\ldots,\boldsymbol{e}_M), \label{eq:vfl95posterior}\tag{3}\] with prediction \[\hat{y}_{\mathrm{VFL}} = \mathop{\mathrm{arg\,max}}_{k\in\{0,\ldots,K-1\}} \boldsymbol{q}(k). \label{eq:y95vfl}\tag{4}\]

Round–1 is intentionally lightweight: it requires only communication of \(K\)-dimensional posterior vectors and uses low-complexity local predictors. Round–2 is generally more expressive but more expensive, because it requires transmitting higher-dimensional intermediate representations and applying a collaborative server-side fusion model.

3.2 Routing Policy and Objective↩︎

A routing policy decides, after observing the pooled Round–1 posterior, whether the sample should be escalated to Round–2. Let \[\pi:\Delta^{K-1}\to\{0,1\}\] be a binary routing rule, where \(\pi(\bar{\boldsymbol{p}})=1\) means escalate and \(\pi(\bar{\boldsymbol{p}})=0\) means adopt the Round–1 prediction.

Given \(\pi\), the final prediction is \[\hat{y}^{\pi} = \begin{cases} \hat{y}_{\mathrm{VFL}}, & \text{if } \pi(\bar{\boldsymbol{p}})=1,\\ \hat{y}_{\mathrm{loc}}, & \text{otherwise.} \end{cases} \label{eq:final95prediction}\tag{5}\]

The system-level goal is to maximize prediction accuracy under an average escalation budget:

\[\begin{align} \max_{\pi} \quad & \mathbb{E}_{(\boldsymbol{x}_1,\ldots,\boldsymbol{x}_M,y)\sim \mathcal{P}} \left[ \mathbf{1}\!\left[\hat{y}^{\pi}=y\right] \right] \\ \text{s.t.} \quad & \mathbb{E}_{(\boldsymbol{x}_1,\ldots,\boldsymbol{x}_M,y)\sim \mathcal{P}} \left[ \pi(\bar{\boldsymbol{p}}) \right] \le B \end{align} \label{eq:system95objective}\tag{6}\] where \(\mathcal{P}\) denotes the test-time distribution and \(B\in[0,1]\) is a prescribed escalation budget. The constraint is equivalent to controlling the average communication cost up to a constant factor¹.

3.3 Escalation Gain↩︎

For a sample with true label \(y\), we define the gain of escalation as \[G = \mathbf{1}\!\left[\hat{y}_{\mathrm{VFL}}=y\right] - \mathbf{1}\!\left[\hat{y}_{\mathrm{loc}}=y\right] \in\{-1,0,+1\}. \label{eq:gain}\tag{7}\] This quantity distinguishes three cases:

• \(G=+1\): escalation is beneficial (Round–1 is wrong, Round–2 is correct);

• \(G=0\): escalation is neutral (both routes agree in correctness);

• \(G=-1\): escalation is harmful (Round–1 is correct, Round–2 is wrong).

This formulation follows the general decision-theoretic view that selective escalation should depend on the conditional benefit of invoking a stronger but more expensive prediction route. Closely related ideas appear in value-of-information-based active classification and in recent analyses of classifier cascades, where the routing decision is expressed through the relative correctness of a weak and a strong predictor [15]. In our setting, the weak route is the pooled Round–1 predictor and the strong route is the collaborative Round–2 VFL predictor.

Since the escalation decision is made after Round–1, the pooled posterior \(\bar{\boldsymbol{p}}\) is the natural observable available to the router. We therefore consider threshold-based policies of the form \[\mathbb{E}[G\mid \bar{\boldsymbol{p}}] > \tau, \label{eq:decision95rule95general}\tag{8}\] where \(\tau\) controls the communication–accuracy trade-off. Using the definition of \(G\), the conditional expected gain can be written as \[\begin{align} \mathbb{E}[G\mid \bar{\boldsymbol{p}}] &= \mathbb{P}(\hat{y}_{\mathrm{VFL}}=y\mid \bar{\boldsymbol{p}}) - \mathbb{P}(\hat{y}_{\mathrm{loc}}=y\mid \bar{\boldsymbol{p}}). \label{eq:gain95decomposition} \end{align}\tag{9}\]

4 Method↩︎

4.1 A Plug-in Approximation of the Expected Escalation Gain↩︎

Conditioning on \(\bar{\boldsymbol{p}}\), the first term in 9 can be expanded as \[\begin{align} \mathbb{P}(\hat{y}_{\mathrm{VFL}}=y \mid \bar{\boldsymbol{p}}) &= \sum_{k=0}^{K-1} \mathbb{P}(\hat{y}_{\mathrm{VFL}}=y, y=k \mid \bar{\boldsymbol{p}}) \nonumber\\ &= \sum_{k=0}^{K-1} \mathbb{P}(\hat{y}_{\mathrm{VFL}}=k \mid y=k,\bar{\boldsymbol{p}})\, \mathbb{P}(y=k \mid \bar{\boldsymbol{p}}). \label{eq:vfl95term95exact} \end{align}\tag{10}\]

The quantity \(\mathbb{P}(\hat{y}_{\mathrm{VFL}}=k \mid y=k,\bar{\boldsymbol{p}})\) is generally sample-dependent and not directly available at test time. We therefore adopt the following approximation.

4.1.0.1 Classwise reliability approximation.

We approximate the conditional correctness of the Round–2 route given the class by a classwise average reliability term: \[\mathbb{P}(\hat{y}_{\mathrm{VFL}}=k \mid y=k,\bar{\boldsymbol{p}}) \approx \mathbb{P}(\hat{y}_{\mathrm{VFL}}=k \mid y=k) = C_{\mathrm{VFL}}[k,k], \label{eq:classwise95reliability95assumption}\tag{11}\] where \(C_{\mathrm{VFL}}[k,k]\) denotes the diagonal entry of the Round–2 confusion matrix for class \(k\). In other words, once the true class is fixed, we assume that the remaining dependence of Round–2 correctness on \(\bar{\boldsymbol{p}}\) is small enough to ignore in the analytical model. This makes the score easy to estimate from held-out data: the behavior of the collaborative route is summarized by one reliability coefficient per class instead of a fully sample-dependent conditional model. The trade-off is that the approximation cannot capture within-class variation in Round–2 behavior. If the collaborative model behaves differently for samples from the same class depending on the pooled Round–1 posterior, then the analytical score may systematically overestimate or underestimate the true benefit of escalation. This issue can become more pronounced under distribution shift, when the calibration data are not representative of test-time conditions, or when some classes are too rare to estimate their reliability coefficients accurately. Under the approximation in 11 , the first term in 9 becomes \[\mathbb{P}(\hat{y}_{\mathrm{VFL}}=y \mid \bar{\boldsymbol{p}}) \approx \sum_{k=0}^{K-1} C_{\mathrm{VFL}}[k,k]\, \mathbb{P}(y=k \mid \bar{\boldsymbol{p}}). \label{eq:vfl95term95plugin}\tag{12}\]

For the Round–1 route, \(\hat{y}_{\mathrm{loc}}\) is a deterministic function of \(\bar{\boldsymbol{p}}\). Therefore, \[\mathbb{P}(\hat{y}_{\mathrm{loc}}=k \mid y=k,\bar{\boldsymbol{p}}) = \mathbf{1}[\hat{y}_{\mathrm{loc}}=k],\] and we obtain: \[\begin{align} \mathbb{P}(\hat{y}_{\mathrm{loc}}=y \mid \bar{\boldsymbol{p}}) &= \sum_{k=0}^{K-1} \mathbb{P}(\hat{y}_{\mathrm{loc}}=k \mid y=k,\bar{\boldsymbol{p}})\, \mathbb{P}(y=k \mid \bar{\boldsymbol{p}}) \nonumber\\ &= \sum_{k=0}^{K-1} \mathbf{1}\!\left[\hat{y}_{\mathrm{loc}}=k\right]\, \mathbb{P}(y=k \mid \bar{\boldsymbol{p}}). \label{eq:loc95term95exact} \end{align}\tag{13}\]

Substituting 12 and 13 into 9 yields the plug-in approximation \[s(\bar{\boldsymbol{p}}) := \sum_{k=0}^{K-1} \mathbb{P}(y=k \mid \bar{\boldsymbol{p}}) \left( C_{\mathrm{VFL}}[k,k] - \mathbf{1}\!\left[\hat{y}_{\mathrm{loc}}=k\right] \right), \label{eq:analytical95score}\tag{14}\] which we use as the analytical routing score. The score is high when the calibrated posterior assigns low probability to the Round–1 predicted class while assigning probability mass to classes on which the Round–2 model is reliable. It becomes negative when the pooled posterior is concentrated on the current Round–1 label or when the Round–2 model is weak on the plausible classes, indicating that escalation is unlikely to help.

4.2 Estimating the Score from Held-Out Calibration Data↩︎

To evaluate the analytical score in 14 at test time, two quantities must be estimated from held-out calibration data. The first is the conditional class posterior \(\mathbb{P}(y=k \mid \bar{\boldsymbol{p}})\), which captures how the pooled Round–1 posterior relates to the true label. The second is the classwise reliability of the Round–2 collaborative predictor, summarized by the diagonal entries \(C_{\mathrm{VFL}}[k,k]\) of its confusion matrix. We estimate these two components separately and then combine them into a deployable routing score.

4.2.0.1 Estimating Round–2 classwise reliability.

Let \[\mathcal{D}_{\mathrm{calib}} = \left\{(\boldsymbol{x}_1^{(n)},\ldots,\boldsymbol{x}_M^{(n)},y^{(n)})\right\}_{n=1}^{N_{\mathrm{calib}}}\] be a held-out calibration set. Using the Round–2 VFL predictor on this set, we estimate the confusion matrix by \[C_{\mathrm{VFL}}[k,c] = \frac{ \sum_{n:\,y^{(n)}=k} \mathbf{1}\!\left[\hat{y}_{\mathrm{VFL}}^{(n)}=c\right] }{ \sum_{n:\,y^{(n)}=k} 1 }, \label{eq:cvfl95estimator}\tag{15}\] so that the diagonal term \(C_{\mathrm{VFL}}[k,k]\) estimates the empirical probability that the collaborative route predicts class \(k\) correctly conditional on \(y=k\).

4.2.0.2 Calibrating the pooled Round–1 posterior.

The remaining ingredient is the conditional distribution \(\mathbb{P}(y=k \mid \bar{\boldsymbol{p}})\). A natural baseline is to use the pooled posterior entry \(\bar{\boldsymbol{p}}(k)\) itself as an estimate of \(\mathbb{P}(y=k \mid \bar{\boldsymbol{p}})\). In practice, however, pooled posteriors are not perfectly calibrated. We therefore apply a multiclass Dirichlet calibration map [16]² fitted on the same held-out calibration data: \[\hat{\boldsymbol{m}}_{\phi}(\bar{\boldsymbol{p}}) = \mathrm{softmax}\!\left( W\log(\bar{\boldsymbol{p}}+\boldsymbol{\epsilon})+\boldsymbol{b} \right), \label{eq:dirichlet95map}\tag{16}\] where \(\phi=(W,\boldsymbol{b})\), \(W\in\mathbb{R}^{K\times K}\), and \(\boldsymbol{b}\in\mathbb{R}^K\).

The calibration parameters are obtained by minimizing a regularized negative log-likelihood objective: \[\scriptsize \hat{\phi} = \arg\min_{W,\boldsymbol{b}} \left[ - \sum_{n=1}^{N_{\mathrm{calib}}} \log \hat{\boldsymbol{m}}_{\phi}\!\left(\bar{\boldsymbol{p}}^{(n)}\right)[y^{(n)}] + \lambda_{\mathrm{id}} \left( \|W-I\|_F^2+\|\boldsymbol{b}\|_2^2 \right) \right]. \label{eq:dirichlet95fit}\tag{17}\] We then approximate the conditional class posterior by \[\mathbb{P}(y=k \mid \bar{\boldsymbol{p}}) \approx \hat{\boldsymbol{m}}_{\hat{\phi}}(\bar{\boldsymbol{p}})[k]. \label{eq:posterior95estimate}\tag{18}\]

Substituting the calibrated posterior estimate 18 and the classwise reliability coefficients \(C_{\mathrm{VFL}}[k,k]\) into 14 yields the test-time score \[\hat{s}(\bar{\boldsymbol{p}}) = \sum_{k=0}^{K-1} \hat{\boldsymbol{m}}_{\hat{\phi}}(\bar{\boldsymbol{p}})[k] \left( C_{\mathrm{VFL}}[k,k] - \mathbf{1}\!\left[\hat{y}_{\mathrm{loc}}=k\right] \right). \label{eq:deployable95score}\tag{19}\] The final routing policy is therefore \[\pi_{\tau}(\bar{\boldsymbol{p}}) = \mathbf{1}\!\left[\hat{s}(\bar{\boldsymbol{p}})>\tau\right], \label{eq:final95policy}\tag{20}\] that is, the sample is escalated whenever the estimated gain of switching to the collaborative Round–2 route exceeds the threshold \(\tau\).³

5 Evaluation Setting↩︎

We evaluate the proposed routing rule on three classification benchmarks: CIFAR-10, CIFAR-100, and ModelNet40. Together, these datasets cover complementary settings, from standard 2D image classification to synthetic multi-view 3D object recognition. In each case, we construct a multi-client inference problem in which several clients observe partial views of the same underlying sample, while the server aggregates the communicated information and, when needed, applies the collaborative Round–2 predictor.

5.1 Datasets↩︎

5.1.0.1 CIFAR-10 and CIFAR-100

For CIFAR-10 and CIFAR-100 [18], we create a four-client benchmark by partitioning each image into four non-overlapping quadrants taken from the same original image. Each quadrant is assigned to one client, so that the four clients receive complementary observations associated with the same class label. This provides a simple vertically partitioned setting in which no client has access to the full image.

5.1.0.2 ModelNet40

For ModelNet40, we construct a two-client benchmark from the original 3D object models. Each object mesh is first normalized in scale and then rendered from two fixed viewpoints, producing a paired multi-view sample for the same underlying object. In the two-view setting used here, the rendered images correspond to a top view and a diagonal view, which are assigned to the two clients. The resulting pair provides complementary observations of the same 3D shape while preserving a shared class label.

5.2 Baselines↩︎

We compare against four baselines overall. The learned, confidence, and oracle baselines are shown in the main trade-off curves, while the L2D baseline is reported separately at matched escalation budgets.

5.2.0.1 Learned (\(\textsf{\mathrm{\small Diff-01}}\)) gain router

We compare against a learned router that predicts the realized escalation gain. The router is a two-layer MLP trained on the calibration set to regress \[G=\mathbf{1}[\hat{y}_{\mathrm{VFL}}=y]-\mathbf{1}[\hat{y}_{\mathrm{loc}}=y]\] using mean squared error. Its input consists of features \(\boldsymbol{z}\) available after Round–1. \(\boldsymbol{z}\) includes the pooled posterior \(\bar{\boldsymbol{p}}\) together with uncertainty and disagreement summaries derived from the local and pooled predictions, such as the local entropies \[H(\boldsymbol{p}_i)=-\sum_{k=0}^{K-1} p_i(k)\log p_i(k), \qquad i=1,\dots,M,\] the entropy of the pooled posterior \[H(\bar{\boldsymbol{p}})=-\sum_{k=0}^{K-1} \bar p(k)\log \bar p(k),\] and the KL divergences between each local posterior and the pooled posterior \[D_{\mathrm{KL}}(\boldsymbol{p}_i \,\|\, \bar{\boldsymbol{p}}) = \sum_{k=0}^{K-1} p_i(k)\log\frac{p_i(k)}{\bar p(k)}, \qquad i=1,\dots,M.\] These features provide a compact summary of Round–1 confidence and cross-client disagreement for training a gain predictor.

5.2.0.2 Confidence-based router

This baseline escalates when the pooled Round–1 posterior is insufficiently confident, using the maximum pooled posterior mass as a scalar confidence score.

5.2.0.3 Learning to Defer-based router

We also compare against a learning-to-defer (L2D) formulation in [11]. Each client trains a \(K+1\)-class classifier, where the first \(K\) outputs correspond to the original task classes and the additional output corresponds to a defer/escalate action. The VFL model is treated as the downstream expert. After training the models of the Round–2 VFL route, the client models (i.e. the \(K+1\)-class classifiers) are trained on the same training split using the VFL correctness indicator as a signal to tailor the loss. Specifically, for each training sample, the frozen Round–2 VFL route predicts \(\hat{y}_{\mathrm{VFL}}\), and the indicator \(\mathbf{1}[\hat{y}_{\mathrm{VFL}}=y]\) determines whether deferral should be rewarded. When the Round–2 VFL route is correct, the client is encouraged to assign probability both to the true class and to the defer output; when the Round–2 VFL route is incorrect, the client is trained only to predict the true class locally. This corresponds to the loss \[\mathcal{L}_{\mathrm{L2D}} = -\alpha \log p_y^{+} - \mathbf{1}[\hat{y}_{\mathrm{VFL}}=y]\log p_{\mathrm{def}}^{+},\] where \(p_y^{+}\) is the client probability assigned to the true class, \(p_{\mathrm{def}}^{+}\) is the probability assigned to the defer output, and \(\alpha\) controls the classification term. At inference time, the server averages the \(K+1\)-dimensional posteriors. Let \[\bar{\boldsymbol{p}}^{+}\in\mathbb{R}^{K+1}\] denote this averaged posterior, and let \(\bar p_{\mathrm{def}}^{+}\) be its defer component. The routing score is \[s_{\mathrm{L2D}} = \bar p_{\mathrm{def}}^{+} - \max_{k=0,\dots,K-1}\bar p^{+}(k),\] and escalation occurs when \(s_{\mathrm{L2D}}>\tau\). The canonical decision here corresponds to taking the argmax over the \(K+1\) outputs and escalating whenever the defer class is selected. We compare our analytical model with this baseline separately at matched escalation budgets for different values of \(\alpha\) in Table 2.

5.2.0.4 Oracle (\(\textsf{\mathrm{\small Diff-01}}\)) router.

This reference policy has access to the realized gain \(G\) and therefore represents an upper bound on routing performance under a given escalation budget. It is not deployable in practice.

Table 1 summarizes the scalar routing score used by each method and the corresponding escalation rule.

5.3 Training and Calibration Protocol↩︎

For each dataset, the available training data is split into two disjoint subsets. The first is used to train both the client-side local predictors and the Round–2 collaborative VFL model. The second is held out for calibration and router construction. This calibration subset is used to fit the Dirichlet calibration map for the pooled Round–1 posterior and to estimate the classwise Round–2 reliability coefficients used by the analytical score. The same subset is also used to fit the learned \(\textsf{\mathrm{\small Diff-01}}\) gain router, so that the analytical router and learned routing baseline rely on the same data split.

The Round–1 predictor is built from local classifiers that operate only on the client’s own partial observation. The Round–2 predictor uses client-specific encoders together with a server-side fusion head. In our implementation, the Round–2 encoders are based on ResNet-18 backbones [19], followed by a prediction head at the server. Since both the local predictors and the collaborative model are trained on the same training subset, differences between the two routes reflect their modeling and communication capabilities rather than unequal access to training data.

5.4 Inference-Time View Degradation↩︎

To examine how routing behavior changes as the quality of client views degrades, we introduce controlled input corruption at inference time. Specifically, we add zero-mean Gaussian noise with fixed standard deviation \(\sigma=0.25\) to the normalized inputs of each client for a selected subset of classes during test-time evaluation. No corruption is applied during training or calibration: the local predictors, the Round–2 collaborative model, and the posterior calibration map are all learned from clean data. This setup therefore introduces a controlled mismatch between model construction and deployment conditions.

The noise is applied consistently across the inference pipeline, affecting both the Round–1 local predictors and the client encoders used by the Round–2 VFL model. As the number of affected classes increases, a larger fraction of the input space is subject to degraded local observations, making the routing problem progressively more challenging.

This is intended to emulate test-time heterogeneity, where the informativeness of clients views may degrade unevenly across the data–for example due to occlusion, blur, illumination variation, truncation, or sensor noise. The goal is to assess whether the proposed routing rule remains effective under increasingly adverse deployment conditions.

5.5 Model Architectures↩︎

5.5.0.1 Round–1 Local Models

In all experiments, clients use the same lightweight local classifier⁴ in Round–1: a small convolutional network with four strided \(3\times 3\) convolutional blocks, one additional same-resolution refinement block, global average pooling, dropout, and a final linear layer. This design keeps the first route intentionally simple.

5.5.0.2 Round–2 VFL Local and Server Models

Round–2 uses one ResNet-18 encoder at each client to produce the embeddings, followed by a fusion head at the server. The fusion head projects client embeddings into a common latent space, computes sample-dependent client gates, and classifies from a representation that combines the gated fusion with explicit interaction features.

5.5.0.3 Posterior Calibration

Across all datasets, the pooled Round–1 posterior is post-hoc calibrated with a multiclass Dirichlet calibrator using a held-out calibration dataset⁵.

Dataset-specific training, calibration, and routing hyperparameters are summarized in the Appendix.

5.6 Evaluation Metrics↩︎

Let \[\pi_{\tau}(\bar{\boldsymbol{p}}) = \mathbf{1}\!\left[\hat{s}(\bar{\boldsymbol{p}})>\tau\right]\] denote the analytical routing policy under threshold \(\tau\), where \(\hat{s}(\bar{\boldsymbol{p}})\) is the deployable score defined in 19 . Here, \(\pi_{\tau}(\bar{\boldsymbol{p}})=1\) means that the sample is escalated to the VFL route, while \(\pi_{\tau}(\bar{\boldsymbol{p}})=0\) means that the Round–1 local prediction is retained. The final prediction is \[\hat{y}^{\pi_\tau} = \begin{cases} \hat{y}_{\mathrm{VFL}}, & \pi_\tau(\bar{\boldsymbol{p}})=1,\\ \hat{y}_{\mathrm{loc}}, & \pi_\tau(\bar{\boldsymbol{p}})=0. \end{cases}\] Varying \(\tau\) traces different communication–accuracy operating points. We report the following metrics:

5.6.0.1 System accuracy.

\[\mathrm{Acc.}(\tau) \triangleq \mathbb{E}\!\left[ \mathbf{1}\!\left[\hat{y}^{\pi_\tau}=y\right] \right].\]

5.6.0.2 Escalation rate.

\[\mathrm{ER}(\tau) \triangleq \mathbb{E}\!\left[ \pi_\tau(\bar{\boldsymbol{p}}) \right].\]

Sweeping \(\tau\) on the test set yields the trade-off curve \[\bigl(\mathrm{ER}(\tau),\mathrm{Acc.}(\tau)\bigr).\] For reference, we also report the two endpoint policies: never escalating, which corresponds to local-only prediction, and always escalating, which corresponds to Round–2 VFL route prediction.

To assess whether escalations are well targeted, we report \[\begin{align} \mathrm{EP}(\tau) &\triangleq \mathbb{E}\!\left[ \mathbf{1}[G=+1] \mid \pi_\tau(\bar{\boldsymbol{p}})=1 \right], \nonumber\\ \mathrm{NE}(\tau) &\triangleq \mathbb{E}\!\left[ \mathbf{1}[G=0] \mid \pi_\tau(\bar{\boldsymbol{p}})=1 \right]. \nonumber \end{align}\]

\(\mathrm{EP}\) is the escalation precision: among escalated samples, it measures the fraction for which VFL corrects a Round–1 error. \(\mathrm{NE}\) is the neutral escalation share: among escalated samples, it measures the fraction for which escalation does not change correctness. Thus, high \(\mathrm{EP}\) indicates that escalations are often beneficial, whereas high \(\mathrm{NE}\) indicates that many VFL calls are unnecessary.

6 Results↩︎

We first analyze ModelNet40 in detail, then summarize the CIFAR results. 2 summarizes average (over 5 runs) test performance on ModelNet40 across the three inference-time Gaussian-noise regimes. Among the practical routing rules, the analytical expected-gain router performs best overall. Across operating points, it reaches a higher peak accuracy than both endpoint policies—local-only and always-escalate VFL—in all three noise regimes. By contrast, the \(\textsf{\mathrm{\small Diff-01}}\) learned router is consistently weaker. This suggests that the calibrated expected-gain estimate already captures most of the useful routing signal, allowing the router to prioritize samples that are likely to benefit from escalation while avoiding those that are neutral or harmful. This pattern is reflected in the EP and NE curves.

In particular, in the noise-free setting, the pooled Round–1 local predictor reaches roughly \(67\%\) accuracy; this drops to about \(59\%\) and \(56\%\) as noise is applied to 10 and 20 classes, respectively. The always-escalate Round–2 VFL route follows a similar trend, decreasing from about \(79\%\) in the noise-free setting to around \(71\)–\(74\%\) under the two noisy regimes, while remaining consistently stronger than Round–1. This confirms that escalation remains beneficial in principle, even under substantial degradation of the clients views.

A second consistent pattern is that most test samples are neutral with respect to escalation: the Oracle curve saturates early (around \(20\)–\(25\%\) on average), indicating that only a subset of examples truly benefit from invoking Round–2. As a consequence, very low routing thresholds lead to over-escalation, since many Round–2 calls do not change the final prediction and may occasionally even hurt it.

The most effective operating points for our method occur at moderate escalation rates, approximately \(40\)–\(60\%\) in the noise-free setting and when noise is applied to 10 classes. This corresponds to escalating roughly half of the test samples rather than operating near either endpoint. This behavior is consistent with the intended role of routing: the router filters out many neutral escalations, preserves most beneficial escalations, and avoids harmful escalations until high escalation rates force their inclusion. As the escalation rate increases further, the accuracy of our method gradually declines and eventually converges to that of the always-escalate VFL baseline, both in the noise-free case and under noise affecting \(10\) classes. This decline reflects the point at which the router is forced to escalate samples that were previously held out precisely because they are harmful.

Importantly, the same qualitative behavior is observed for the Oracle strategy: once all beneficial samples have already been escalated and harmful samples can no longer be avoided, additional escalation inevitably includes detrimental cases, leading to a degradation in overall accuracy.

The confidence-based baseline improves over local-only prediction, but remains consistently less effective than the proposed analytical router, indicating that confidence alone is not a sufficiently informative signal for escalation. The Oracle performance remains better than all deployable methods. The remaining gap between the oracle and the analytical router indicates clear room for improving sample selection.

We further compare against the L2D baseline of [11]. We report the comparison separately at matched escalation budgets in 2 while accounting for the baseline performance for different values of \(\alpha\). The 0% escalation (local-only) point shows similar performance among Round–1 predictions. As expected, EP and NE are both \(0\%\) at this point because no samples are escalated.

The difference in performance becomes clearer once escalation is allowed. At \(20\%\) escalation, the proposed router reaches \(75.40\%\) accuracy, compared with \(70.92\%\) for \(\alpha=2\) in the L2D baseline. Equivalently, our proposed router captures a much larger fraction of beneficial escalations, with an absolute EP improvement of about 18 percentage points over the strongest L2D variant at that budget. At \(40\%\) escalation, the same pattern persists. At higher escalation rates, the gap naturally narrows as more samples are sent through the VFL-based route regardless of the router score (i.e. threshold \(\tau\) becomes small). Still, the proposed router keeps the better end-to-end accuracy at \(60\%\). Finally, at \(80\%\), some L2D variants match or slightly exceed the analytical router on these metrics. However, this occurs only at a much larger communication budget, whereas the analytical router already reaches comparable or better accuracy at \(40\%\)–\(60\%\) escalation.

At \(100\%\) escalation, both methods reduce to pure VFL inference, and therefore obtain the same final accuracy, \(74.67\%\). The EP and NE values, however, do not have to match at this endpoint. They are computed from the realized gain with respect to each method’s own Round–1 local prediction models performance. Unlike other experiments which differ only in the routing strategy and share the same Round–1 and Round–2 models; here, the analytical router uses the pooled \(K\)-class local predictors as its local baseline, whereas the L2D baseline uses the prediction of its trained \(K+1\)-class client model. Thus, even though all final predictions come from the same VFL-based route at \(100\%\) escalation, the underlying per-sample realized gain can differ, which explains why EP and NE are different.

Finally, 3 shows analogous behavior on CIFAR-10 and CIFAR-100 in the noise-free setting and under inference-time noise. Most examples are again neutral with respect to escalation, so aggressive thresholds over-escalate without commensurate accuracy gains. Consistent with the ModelNet40 findings, the analytical router provides the most favorable trade-off across operating points, improving over both endpoint policies and outperforming confidence-based and learned-gain routing, while still leaving a gap to the oracle that highlights remaining headroom.

7 Limitations↩︎

The proposed router is intentionally simple and interpretable, but it has several limitations. First, the analytical score relies on a classwise reliability approximation for the Round–2 VFL model. In particular, it replaces the sample-dependent quantity \(P(\hat{y}_{\mathrm{VFL}}=k \mid y=k,\bar{\boldsymbol{p}})\) with the class-level estimate \(P(\hat{y}_{\mathrm{VFL}}=k \mid y=k)\). This makes the score easy to estimate from held-out data, but it ignores within-class variation in the behavior of the collaborative model. If some samples from the same class benefit strongly from escalation while others do not, the classwise approximation may overestimate or underestimate the true gain.

Second, the method depends on the quality and representativeness of the calibration set. The calibrated posterior and the Round–2 reliability coefficients are both estimated offline. If the calibration distribution differs from the test distribution, the expected-gain score may become less reliable as observed in the simulations once noise is injected to the client views during inference. This is especially relevant under distribution shift or when some classes are rare, since the corresponding reliability estimates may have high variance.

8 Conclusion↩︎

We considered inference-time selective escalation in a two-round collaborative inference setting, where a cheap Round–1 local route prediction can optionally be replaced by a stronger but more expensive collaborative prediction. We formulated this routing problem as an expected-gain decision under a communication budget and proposed a simple analytical routing rule based on post-hoc calibration of a Round–1 posterior together with reliability estimates of the Round–2 route predictor. The resulting method is lightweight, interpretable, and does not require training a dedicated routing model. Empirically, the proposed rule improves the communication–accuracy trade-off over confidence-based, learned-gain, and learning-to-defer baselines across the studied benchmarks.

Biography↩︎

References↩︎

---

1. While our focus is not on optimizing the communication layer itself through compression, quantization, or coding techniques, we explicitly study selective escalation of the collaborative route as a mechanism for reducing communication cost at inference time. A more detailed discussion of the communication cost in this work is provided in Appendix.↩︎

2. We compare against temperature scaling [17] and a no-calibration setting in the Appendix. In our experiments, Dirichlet calibration yields the best routing performance.↩︎

3. Because the additional Round–2 communication cost is fixed per escalated sample in our protocol, this cost can be absorbed into the threshold \(\tau\). Thus, increasing \(\tau\) corresponds to requiring a larger expected accuracy gain before paying the communication cost. For a target escalation budget \(B\), \(\tau\) can be chosen on calibration data as the empirical \((1-B)\)-quantile of the routing scores, so that approximately a fraction \(B\) of samples is escalated.↩︎

4. Exception applies to the [11] baseline experiment, as clients’ local models differ in their output layer which outputs \(K\) label posteriors in addition to a defer posterior, as described in 5.2.↩︎

5. For fairness, the same calibration split is used to train the learned \(\textsf{\mathrm{\small Diff-01}}\) router.↩︎