1 Introduction↩︎

Federated Learning (FL) has become a popular distributed learning paradigm that enables model training on distributed data while ensuring the data remains on client devices [1]–[3]. In a typical FL system, a central server coordinates training by aggregating model updates (e.g., gradients or weights) that clients compute on their local data.

In this work, we focus on FL applications where clients are edge devices, such as smartphones or Internet-of-Things (IoT) devices. These devices are often constrained by limited battery life, computational power, and network bandwidth, making it feasible for only a fraction of clients to participate in each training round [4], [5], a setting known as partial client participation. Compounding this challenge is the statistical heterogeneity of client data; since the data distributions are often non-IID across clients, their local model updates can vary significantly [6]. When combined with partial client participation, this heterogeneity means the aggregated update in any given round may not be representative of the true global data distribution, leading to high variance and instability in the training process [7], [8]. Furthermore, participation may be systematically biased, as clients with greater computational or network resources may participate more frequently, introducing a convergence bias [9]. While this bias can be mitigated by techniques like importance sampling [7]–which re-weights updates to ensure all clients contribute equally in expectation—such methods, despite ensuring unbiased convergence, often introduce even higher variance into the training process [10], [11].

To mitigate the training instability caused by partial client participation, several prior works reuse stale information to incorporate updates from inactive clients [9], [12]–[14]. These methods can be broadly categorized by how they leverage this stale information. (1) Direct Reuse of Stale Updates: One line of work directly reuses the last known update from an inactive client as a surrogate for its current one. MIFA [12] and FedVARP [13] both maintain the most recent update received from each client at the server. In each round, this memory is updated with fresh updates from participating clients, while the stale updates for non-participating clients are retained. MIFA then updates the global model by averaging all stored updates (fresh and stale), while FedVARP employs a SAGA-like variance reduction scheme that uses the stale updates as control variates [13], [15]. Similarly, SCAFFOLD [14] corrects for client-drift using stateful client-specific control variates. When a client is inactive, its stale control variate is implicitly reused during the server-side aggregation, thus incorporating its past state into the global update. ConFREE [16] mitigates gradient conflicts in personalized FL by projecting active clients’ updates to remove opposing components, but it does not address the misalignment of stale gradients arising from inactive clients. (2) Weighted Stale Updates: Acknowledging that highly stale information can be detrimental, another line of work proposes down-weighting stale updates when aggregating them with fresh ones. FedStale [9] introduces a hyperparameter \(\beta\) to form a convex combination of an update using only fresh information (akin to FedAvg) and one using both fresh and stale information (akin to FedVARP). This allows the algorithm to control the influence of stale updates. This strategy is also common in asynchronous FL, where the updates from slower devices are reweighted to ensure training stability [17]–[19]. Although these methods can handle the variance from partial participation, their main limitation is that they treat stale gradients as fixed vectors. For example, for less active clients, these approaches ignore the fact that the direction of their gradients becomes increasingly misaligned as the global model evolves. This misalignment can destabilize the training process. Crucially, these approaches lack a mechanism to actively correct the stale gradient’s direction by leveraging the more current information provided by active clients.

To address this issue, we propose FedSteer, a novel corrective mechanism that replaces a client’s stale gradient with a more accurate estimate derived from an evolving gradient subspace. FedSteer constructs this subspace from a cache of recent gradients provided by a small core set of clients \(\mathcal{X}\). Although each gradient vector exists in the high-dimensional space \(\mathbb{R}^d\), the subspace’s dimensionality is no more than the size of the core set \(|\mathcal{X}|\). When a client (indexed by \(i\)) is active, we project its gradient onto the current subspace to find low-dimensional coordinates (\(\mathbf{s}_i\)). The coordinates \(\mathbf{s}_i\) can be interpreted as stable similarity coefficients, which capture the relationship between client \(i\) and clients in the core set. When a client becomes inactive, FedSteer reuses these coordinates to the newly evolved subspace drifted by active clients from the core set to reconstruct a corrected gradient estimate. This process effectively “steers” the client’s outdated information to approximate its true local gradient for the current model.

We complement FedSteer with a selective caching strategy that significantly reduces the server’s memory overhead. The server only stores the gradient vectors (each in \(\mathbb{R}^d\)) for the small core set \(\mathcal{X}\), and the low-dimensional coordinate vectors are cached for the entire client population (\(N\)). FedSteer’s memory cost is therefore \(O(|\mathcal{X}|d+N|\mathcal{X}|)\), whereas prior methods [9], [12], [13] that cache all stale gradients incur a much higher cost of \(O(Nd)\).² To our knowledge, FedSteer is the first approach that corrects the direction of a client’s stale update by leveraging collective, timely information from other clients in FL.

Our Contributions:

• We propose FedSteer, a novel algorithm that introduces a directional correction mechanism for stale updates. FedSteer reconstructs a corrected update gradient for inactive clients by applying a client’s stable, cached projection coordinates to a dynamic gradient subspace that is drifted by other active clients. FedSteer effectively steers outdated gradients toward the current global objective. We prove that this corrective projection minimizes the global aggregation variance, providing theoretical support for its performance.

• We introduce a selective client caching strategy to make FedSteer memory-efficient. This method iteratively optimizes a small, representative core set of clients during a warm-up phase. The updated gradients from this core set are sufficient to construct an effective gradient subspace, significantly reducing the server’s storage cost.

• We provide a rigorous convergence analysis, proving that FedSteer achieves a tighter convergence upper bound compared to prior methods that reuse stale information without a directional correction mechanism.

• Our extensive empirical evaluation on multiple datasets under different settings of data and system heterogeneity demonstrates that FedSteer significantly outperforms state-of-the-art baselines. It achieves an accuracy gain of at least 7.9% over the next-best method and prevents training collapse in the most challenging scenarios.

The rest of the paper is organized as follows. In Section 2, we formulate the problem of aggregation variance and introduce our proposed method, FedSteer, detailing its corrective projection mechanism and selective caching strategy. We then provide a convergence analysis in Section 3 and present extensive experimental results in Section 4. Finally, we conclude the paper in Section 5. Detailed proofs for the convergence analysis are provided in Supplementary Material 7, along with additional experimental settings and results in Supplementary Material 8.

2 Methods↩︎

2.1 Problem Formulation↩︎

We consider an FL system with a set of \(N\) clients, indexed by \(i\in\mathcal{N} = \{1, \dots, N\}\). The global objective is to minimize a weighted average of the clients’ local loss functions: \[\min_{\mathbf{w} \in \mathbb{R}^d} F(\mathbf{w}) \triangleq \sum_{i=1}^{N} d_i F_i(\mathbf{w}),\] where \(\mathbf{w} \in \mathbb{R}^d\) is the global model and \(F_i(\mathbf{w}) \triangleq \mathbb{E}_{\xi_i \sim \mathcal{D}_i}[\ell(\mathbf{w}, \xi_i)]\) is the local objective for client \(i\), representing the expected loss over its data distribution \(\mathcal{D}_i\). Each client’s contribution is scaled by a weight \(d_i\), typically set in proportion to its dataset size, such that \(d_i = \frac{|\mathcal{D}i|}{\sum{j\in\mathcal{N}} |\mathcal{D}_j|}\). We model each client’s participation as an independent Bernoulli trial with a client-specific participation probability \(p_i\).

FL training process: In each global round \(t=1,2,\dots,T\), a subset of clients \(\mathcal{A}_t\) is active, with client \(i\) participating with probability \(p_i\). Active clients perform \(E\) local training epochs (e.g., using SGD) starting from \(\mathbf{w}_i^{0}=\mathbf{w}^t\). After training, each active client \(i \in \mathcal{A}_t\) computes its total update \(\mathbf{g}_i^t=\mathbf{w}_i^{0}-\mathbf{w}_i^E\) and sends it to the server. Define the stale gradient of client \(i\) as: \(\text{if } i \in \mathcal{A}_{t}\text{: } \mathbf{h}_i^{t+1}= \mathbf{g}_{i}^{t}, \text{ otherwise: } \mathbf{h}_i^{t+1}=\mathbf{h}_{i}^{t}.\)

2.2 Corrective Projections for Stale Gradients↩︎

To address the issue of extreme staleness, we propose FedSteer, a method that corrects a stale client update by projecting the new gradient onto a dynamically evolving gradient subspace, and reuse the subspace coordinates instead. As shown in Fig. 1, this process steers the outdated information toward the current optimization landscape, making it relevant for the global update.

2.2.1 The Dynamic Gradient Subspace↩︎

The foundation of our method is a low-dimensional gradient subspace constructed from recent client updates. The server maintains a cache of the most recent raw gradients, denoted as \(\mathbf{h}_i^t \in \mathbb{R}^d\), from a small, representative subset of clients \(\mathcal{X} \subseteq \mathcal{N}\), referred to as the core set (the selection process is described in Section 2.4). To ensure these gradients can form a well-defined basis, we make the following assumption:

The assumption allows us to safely normalize each raw gradient, improving numerical stability. The normalized basis vector \(\bar{\mathbf{h}}_i^t\) is defined as: \[\begin{align} \bar{\mathbf{h}}_i^t=\frac{\mathbf{h}_i^t}{\|\mathbf{h}_i^t\|_2}. \end{align}\]

The dynamic subspace is then formally defined by a matrix \(\mathbf{Q}_t \in \mathbb{R}^{d \times |\mathcal{X}|}\), whose columns are these normalized basis vectors: \[\mathbf{Q}_t = \begin{bmatrix} \bar{\mathbf{h}}_{j_1}^t & \dots &\bar{\mathbf{h}}_{j_k}^t &\dots & \bar{\mathbf{h}}_{j_{|\mathcal{X}|}}^t \end{bmatrix}_{j_k \in \mathcal{X}}\label{eq:Q}.\tag{1}\] While the core set \(\mathcal{X}\) can be fixed, the subspace itself is dynamic, as the basis vectors \(\bar{\mathbf{h}}_j^t\) are updated whenever a client \(j \in \mathcal{X}\) participates in a training round. Even if some core set clients are temporarily inactive, the subspace \(\mathbf{Q}_t\) as a whole continues to evolve through the participation of its other members. Consequently, \(\mathbf{Q}_t\) serves as a continuously updated, low-dimensional representation of the dominant directions of recent client update gradients. This set of dominant directions provides a basis for estimating any client’s update. Since \(\mathcal{X}\) is chosen to be representative, projecting a client’s gradient onto \(\mathbf{Q}_t\) effectively decomposes its update into a combination of these core directions. The resulting coordinates, which capture the client’s similarity to the core set \(\mathcal{X}\), are then reused to reconstruct an estimated gradient for any inactive client.

2.2.2 Corrective Projection via Cached Coordinates↩︎

Rather than directly reusing a stale gradient \(\mathbf{h}_i^t\), which risks being outdated and destabilizing the training process, FedSteer computes a more relevant approximation, \(\hat{\mathbf{g}}_i^t\), that lies within the dynamic subspace \(\mathbf{Q}_t\). This approximation is defined as a linear combination of the subspace’s basis vectors: \[\hat{\mathbf{g}}_i^t = \mathbf{Q}_t \mathbf{s}_i.\] Here, the vector \(\mathbf{s}_i \in \mathbb{R}^{|\mathcal{X}|}\) denotes the coordinates of the gradient approximation within the subspace. The optimal coordinates are those that make \(\hat{\mathbf{g}}_i^t\) the best possible estimate of the true gradient \(\mathbf{g}_i^t\). A standard projection, however, can yield coordinates with large magnitudes that overfit to the specific basis vectors of \(\mathbf{Q}_t\) at a single time step. To find a more stable and generalizable coordinate representation, we introduce a regularization term into the optimization.

For an active client \(i \in \mathcal{A}_t\), the server computes its optimal coordinates \(\mathbf{s}_i^*\) by solving the following regularized least-squares (Ridge Regression) problem: \[\min_{\mathbf{s}_i \in \mathbb{R}^{|\mathcal{X}|}} \| \mathbf{g}_i^t - \mathbf{Q}_t \mathbf{s}_i \|^2 + \lambda \|\mathbf{s}_i\|^2, \forall\; i\in\mathcal{A}_t, \label{eq:problem}\tag{2}\] where the regularization parameter \(\lambda > 0\) penalizes large coefficient values. This prevents the projection from relying too heavily on any single cached gradient in the subspace, promoting a more robust representation. The closed-form solution is: \[\mathbf{s}_i^* = (\mathbf{Q}_t^\top \mathbf{Q}_t + \lambda \mathbf{I})^{-1} \mathbf{Q}_t^\top \mathbf{g}_i^t. \label{eq:solution}\tag{3}\]

Once computed, \(\mathbf{s}_i^*\) are stored in the server’s cache for client \(i\): \(\mathbf{s}_i^{\mathrm{cache}}=\mathbf{s}_i^*\), and its gradient estimate is written as \(\hat{\mathbf{g}}_i^t = \mathbf{Q}_t \mathbf{s}_i^{\mathrm{cache}}\), which is used in global aggregation.

For an inactive client \(i \notin \mathcal{A}_t\), its true gradient is unknown. FedSteer avoids directly reusing its stale gradient \(\mathbf{h}_i^t\). Instead, it retrieves the client’s cached coordinates \(\mathbf{s}_i^{\mathrm{cache}}\), which encode the stable relationship between client \(i\)’s update and the core set’s basis vectors. It then applies these coordinates to the current subspace \(\mathbf{Q}_t\). As illustrated in Fig. 1, because this subspace has been updated by active clients, the resulting reconstruction \(\hat{\mathbf{g}}_i^t=\mathbf{Q}_t \mathbf{s}_i^{\mathrm{cache}}\) effectively “steers” the inactive client’s historical information to align with the current optimization path.

2.2.3 The FedSteer Aggregation Rule↩︎

The final server update uses the projected gradients \(\hat{\mathbf{g}}_i^t=\mathbf{Q}_t \mathbf{s}_i^{\mathrm{cache}}\) as a baseline estimate for all clients and applies a variance reduction correction using the true gradients from the active set: \[\begin{align} \mathbf{w}_{t+1} &= \mathbf{w}_t - \eta_s \Delta_t \\ \Delta_t &= \underbrace{\sum_{i\in\mathcal{N}} d_i \hat{\mathbf{g}}_i^t}_{\text{Projection Baseline}} + \underbrace{\sum_{i\in\mathcal{A}_t} \frac{d_i}{p_{i}} (\mathbf{g}_i^t - \hat{\mathbf{g}}_i^t)}_{\text{Residual Correction}}\label{eq:aggregation} \end{align}\tag{4}\]

The unbiased estimator in Eq. 4 reduces variance by leveraging a baseline estimate (\(\hat{\mathbf{g}}_i^t\)) which allows the active clients to estimate only the small, low-variance residual error (\(\mathbf{g}_i^t-\hat{\mathbf{g}}_i^t\)).

The pseudocode of FedSteer is provided in Algorithm 2.

Complexity analysis: FedSteer incurs no additional communication cost over standard FL (e.g., FedAvg/FedVARP): only active clients transmit updates, and client-side computation remains unchanged. The server solves a small ridge regression with complexity \(O(k^2 d + k^3)\), which is modest since \(k \ll N\) and comparable to standard aggregation. FedSteer is also more memory-efficient than stale-gradient methods (FedVARP/FedStale/MIFA), reducing storage from \(O(Nd)\) to \(O(Nk + kd)\) by keeping low-dimensional coordinates for all clients and full gradients only for a small core set. Detailed complexity analysis is provided in the Supplementary Material 8.2.

2.3 Variance Minimization via Projection↩︎

We show that having each client \(i\) minimize its local projection error (Eq. 2 with \(\lambda=0\)) over its own variable vector \(\mathbf{s}_i\) is equivalent to minimizing the variance of the global update \(\Delta_t\) over the collective set of variables \(\{\mathbf{s}_i\}_{i\in\mathcal{N}}\).

2.4 Selective Client Caching↩︎

We propose a method to select an effective core set \(\mathcal{X}\) during an initial warm-up phase. After the warm-up phase, the optimized core set \(\mathcal{X}\) is fixed for the formal training.³ The warm-up phase consists of \(T_0\) selection cycles. In each selection cycle, the server holds the global model constant and asynchronously collects available gradient from each client as they naturally become active. Once gradients from all \(N\) clients have been gathered, the server has a complete snapshot of the gradient landscape and proceeds with the optimization step for \(\mathcal{X}\). It then performs an aggregation including all clients to update the global model for the next selection cycle.

The core set selection within a selection cycle is performed via a greedy local search that iteratively improves an initially random subset \(\mathcal{X}\). The objective is to find a core set whose gradient subspace can accurately reconstruct the gradients of the entire client population. Specifically, the algorithm seeks to find a set \(\mathcal{X}\) that minimizes the total regularized projection error, \(J(\mathcal{X})\), defined as: \[J(\mathcal{X})=\sum_{i\in\mathcal{N}}d_i \left( \| \mathbf{G}_i^t - \mathbf{Q}_t(\mathcal{X}) \mathbf{s}_i \|_2^2 + \lambda \|\mathbf{s}_i\|_2^2 \right),\] where \(\mathbf{Q}_t(\mathcal{X})\) is the subspace matrix generated from the gradients of clients in \(\mathcal{X}\), following the same definition in Eq. 1 . Since an exhaustive search for the optimal \(\mathcal{X}\) is computationally intractable, we refine the set through single-client swaps. In each iteration, it evaluates exchanging one client in the core set with one outside of it and executes the single swap that yields the greatest reduction in the objective \(J(\mathcal{X})\). To manage the computational cost, we limit this search to a small number of iterations (e.g., at most five) per selection cycle. For each selection cycle, by updating the model and re-optimizing the core set, we ensure \(\mathcal{X}\) is robust because its selection is based on performance across several different versions of the model. This prevents the choice of \(\mathcal{X}\) from overfitting to a specific model state, ensuring \(\mathcal{X}\) remains representative as the model continues to train. The pseudocode of the algorithm is provided in Algorithm 3.

Complexity analysis: The selective caching algorithm has complexity \(O(T_{0} I_{\max} N k)\), where \(T_{0}\) is the number of selection cycles and \(I_{\max}\) is the number of swap iterations per cycle. This cost is incurred only during a short warm-up phase on the server (typically the first \(T_{0}=5\) rounds), after which the core set is fixed and no further selection overhead is introduced for the remaining training rounds. To scale to very large client populations \(N\), we also employ a subsampled selection strategy, which achieves comparable performance (Supplementary Material 8.3).

3 Convergence Analysis↩︎

We make the following standard assumptions.

We provide the proof of Theorem 2 in the Supplementary Material 7. The convergence bound consists of several terms. Some, like the iterate initialization error, vanish as the number of global rounds \(T\) grows. Others, including the partial update variance error, stochastic gradient error, and error from data heterogeneity contribute to a non-vanishing error floor, which is characteristic of stochastic, non-convex optimization in FL. As stated in Theorem 1, FedSteer can explicitly minimize the partial update variance term \(\mathbb{E}\left[\|\Delta^t - \mathbb{E}[\Delta^t]\|^2\right]\). By reducing the magnitude of this dominant, non-vanishing term, FedSteer achieves a tighter convergence bound compared to prior methods including FedVARP [13] and FedStale [9]. As shown in Remark 2, methods like FedVARP and FedStale can be viewed as constrained versions of FedSteer where the coordinates \(\mathbf{s}_i\) are restricted to be one-hot vectors. This suboptimal choice precludes the minimization of the partial update variance. Consequently, these methods, with a larger variance in the global update, result in a looser theoretical convergence bound.

4 Experiments↩︎

4.1 Experimental Setup↩︎

FL system: We simulate an FL system with \(N=100\) clients under independent system and data heterogeneity. For system heterogeneity, we create a weak group (50 clients with participation probability \(p_i=0.04\)) and a powerful group (50 clients with \(p_i=0.16\)), resulting in a network-wide average participation rate of \(0.1\). For data heterogeneity, a \(\gamma\) fraction of clients (the common-label group) are assigned data from the first half of the \(L\) total labels, while the remaining \(1-\gamma\) fraction (rare-label group) receive data from the second half. To ensure a uniform global label distribution while maintaining an average of 100 datapoints per client, we allocate \(\lfloor50/\gamma\rfloor\) datapoints to each common client and \(\lfloor50/(1-\gamma)\rfloor\) to each rare client. We conduct experiments under moderate (\(\gamma=0.7\)) and extreme (\(\gamma=0.9\)) heterogeneity settings.

Datasets & Models: Fashion-MNIST, EMNIST, and CIFAR-10 datasets are used [21]–[23]. The model for Fashion-MNIST features two convolutional, two max-pooling, and two fully-connected layers, and the EMNIST model uses a deeper architecture with three of each layer type. We apply a PreAct ResNet-18 [24] for the CIFAR-10 task. Experiments are conducted on a GeForce 1080Ti with 8 different random seeds. We report the average accuracy and variance across these runs. We provide more details of experimental settings in the Supplementary Material 8.

Baselines: We compare FedSteer against a suite of baseline methods that employ diverse strategies for handling client stale updates. 1) FedAvg serves as the fundamental baseline; it aggregates updates only from active clients in each round and discards any stale information. 2) FedVARP and FedStale both reuse stale updates using a SAGA-like aggregation rule for variance reduction. FedStale further introduces a down-weighting mechanism (\(\beta\)) to mitigate the negative impact of excessively old gradients. We report FedStale’s best result from the selection of \(\beta\in\{0.5,0.6,0.7,0.8,0.9\}\). 3) SCAFFOLD corrects for client drift using control variates, which implicitly leverages historical information. 4) MIFA takes a direct approach by substituting updates from inactive clients with their most recent stale gradients. 5) FedProx (\(\mu=0.1\)) applies the regularization to limit the local drift caused by client data heterogeneity. 6) We also benchmark against a Full Participation setting, where the participation rate is \(1.0\), to establish a performance upper bound for the given data distribution.

4.2 Comparison with baseline methods↩︎

The results in Fig. 4 and Table 1 are reported using a core set size of \(|\mathcal{X}|=10\) for EMNIST and CIFAR-10, and \(|\mathcal{X}|=40\) for Fashion-MNIST, with a regularization factor of \(\lambda=0.5\). The core set \(\mathcal{X}\) was optimized for 5 selection cycles (\(T_0=5\)). We also compare two schemes for updating \(\mathbf{s}_i\): the default method (update only when active) and an enforced update every 5 rounds (enforce=5).

Our experiments reveal the vulnerability of existing baselines to highly stale updates under extreme heterogeneity; FedSteer, however, successfully overcomes this issue to deliver a significant performance advantage. As we mentioned, the experiment setting features a rare-label client group (\(1-\gamma\) fraction) that provides critical and unique information. However, half of these clients have a very low participation probability (weak group, \(p_i=0.04\)), which introduces severely stale updates into the training process. As shown in Fig. 4 and quantified in Table 1, existing methods are highly susceptible to these highly stale updates, especially under the extreme heterogeneity setting (\(\gamma=0.9\)) on EMNIST. For FedVARP, FedStale, and MIFA, the stale gradients fail to align with the current optimization objective, resulting in final accuracies (0.281, 0.309, and 0.243, respectively) that are on par with or worse than a standard FedAvg baseline (0.314). The failure of SCAFFOLD and FedProx is pronounced: their drift correction mechanisms are fundamentally compromised by extreme staleness, leading to a collapse of the training process and a final accuracy of just 0.048 and 0.051.

FedSteer, however, achieves stable convergence by avoiding the direct use of stale updates. Instead, it leverages their projection coordinates on a dynamic subspace \(Q_t\), which is continuously steered by more active clients. On EMNIST (\(\gamma=0.9\)), the enforced-update version of FedSteer achieves a final accuracy of 0.620, nearly doubling the performance of the best-performing baseline. On Fashion-MNIST (\(\gamma=0.9\)), FedSteer provides an 8.8% higher accuracy (0.554) compared to the next-best method, FedAvg (\(0.509\)). Under moderate heterogeneity (\(\gamma=0.7\)), where baselines are more competitive, FedSteer consistently exceeds the top-performing methods by 7.8%, approaching the ideal accuracy of full participation. FedSteer maintains this advantage on the larger and more complex CIFAR-10 dataset, achieving accuracies of \(0.622\) and \(0.660\) under \(\gamma=0.9\) and \(\gamma=0.7\), respectively.

4.3 Impact of Core Set Size and Regularization↩︎

We evaluate the impact of the core set size \(|\mathcal{X}|\) and regularization on FedSteer’s performance on EMNIST, with results presented in Fig. 5. In this experiment, projection coordinates are recomputed every five rounds, which requires them to be robust enough to remain effective as the subspace evolves. The results reveal a critical trade-off between the representational capacity of the subspace and the coordinates’ generalization ability. For a small core sets (\(|\mathcal{X}| \le 30\)), the subspace provides a limited representation of the gradient landscape. Strong regularization is crucial here, as it prevents the coordinates from overfitting to the small basis, producing more robust estimates and leading to higher accuracy. For medium core sets (\(40 \le |\mathcal{X}| \le 80\)), as the core set grows, the subspace’s capacity to represent client gradients increases. In this range, the unregularized approach starts to outperform the regularized one by leveraging the richer, more representative basis without the constraint of a penalty. For large core sets (\(|\mathcal{X}| > 80\)), the performance of both methods degrades. This decline can be attributed to the coordinates becoming over-specialized. When the basis becomes too large, the subspace’s high dimensionality allows it to fit client gradients too closely at the moment of computation. These over-specialized coordinates are highly sensitive to the subspace’s evolution across subsequent rounds; when they are reused, their effectiveness diminishes, leading to inaccurate gradient reconstructions and a drop in model accuracy.

In Appendix 8, we provide the detailed experimental setup (8.1), complexity analysis against standard baselines (8.2), and additional results including a subsampled core set selection strategy for large-scale deployment with reduced complexity (8.3), convergence plots on Fashion-MNIST and CIFAR-10 (8.4), and the stability of coordinate reuse under evolving subspaces (8.5).

5 Conclusion↩︎

In this work, we introduce FedSteer, a novel algorithm that creates a dynamic, low-dimensional gradient subspace from a small client core set. It projects a client’s gradient onto this subspace, computing and caching stable, low-dimensional coordinates for reuse. For inactive clients, these coordinates “steer” outdated information toward the current global objective by being applied to the newly evolved subspace. Complemented by a memory-efficient selective caching strategy, this mechanism is proven to minimize global update variance and achieve a tighter convergence bound. Experiments confirm FedSteer significantly outperforms baselines, prevents training collapse under extreme heterogeneity, and delivers significant accuracy gains.

6 Acknowledgements↩︎

This work was supported by NSF CNS-2106891, TREES: ANR-24-TSIA-0004, and A*STAR under its IAF-ICP programme (H25-MCP3438).

7 PROOFS↩︎

To prove Theorem 1, we first formalize the sources of randomness inherent in the training process. Source of Randomness: The system’s stochasticity arises from two primary sources in each global round \(t\).

• Client sampling: A subset of clients, denoted by the random set \(\mathcal{A}_t\) is selected to participate in the training. This selection is determined by the sampling distribution \(\mathbf{p}=\{p_i\}_{i\in\mathcal{N}}\).

• Data sampling: Each selected client \(i\in\mathcal{A}_t\) computes stochastic gradients using mini-batches of data randomly sampled from its local dataset.

To formalize our analysis, we define the random variables corresponding to these processes. Let \(\xi_{i}^{(t,k)}\) denote the mini-batch of data points sampled by client \(i\) at its \(k\)-th local step during round \(t\).

To track the evolution of randomness across multiple rounds, we use the following notation:

• \(\mathbb{1}_{i\in\mathcal{A}_t}\) is the indicator of client participation.

• \(\mathcal{A}^{(s:q)}=\{\mathcal{A}_{s}, \mathcal{A}_{s+1}, \dots, \mathcal{A}_{q}\}\) represents the sequence of client sets sampled from round \(s\) to round \(q\).

• \(\xi_{i}^{t}=\{\xi_{i}^{t,k}\}_{k=0}^{K-1}\) denotes the collection of all data mini-batches sampled by client \(i\) during its local training in round t.

• \(\xi^{t}=\{\xi_{i}^{t}\}_{i\in\mathcal{A}_t}\) denotes the collection of all data mini-batches sampled by active client \(i\) at round \(t\).

• \(\xi^{(s:q)}=\{\xi^{s},\dots,\xi^q\}\) denotes the collection of all data mini-batches sampled by active clients from round \(s\) to round \(q\).

The stochastic progression of the algorithm from the first round to the current round \(t\) can be expressed as: \[\begin{align} \mathcal{H}^t=\{\mathcal{A}_1,\mathcal{A}_2,\dots, \mathcal{A}_{t-1},\xi^1, \xi^2, \dots, \xi^{t-1}\}. \end{align}\]

We define the following pseudo-gradients for the proof: \[\begin{align} \text{Local stochastic pseudo-gradient: } & g_i^t=\frac{1}{E} \sum_{e=1}^{E-1} \nabla F_i(w_i^{t,e},\xi_i^{t,e}), \\ \text{Local pseudo-gradient: }& \overline{g}_i^t=\frac{1}{E} \sum_{e=1}^{E-1} \nabla F_i(w_i^{t,e}), \\ \text{Global stochastic pseudo-gradient: }& g^t=\sum_{i\in\mathcal{N}} d_i g_i^t,\\ \text{Global pseudo-gradient: }& \overline{g}^t=\sum_{i\in\mathcal{N}} d_i \overline{g}_i^t, \\ \text{Global stale pseudo-gradient: }& h^t=\sum_{i\in\mathcal{N}} d_i h_i^t, \\ \text{Global estimate pseudo-gradient: }& \hat{g}^t=\sum_{i\in\mathcal{N}} d_i \hat{g}_i^t, \end{align}\] where \(E\) is the number of local epochs, following the same definition of the paper. In the proof, we also use \(K\) to denote the number of local epochs, in the case where \(E\) may be misunderstood as the compute of expectation.

The bound in Eq. (56) captures that, when the number of local iterations \(K\) equals 1, \(\bar{g}_i^{t}\) and \(\nabla F_i(\mathbf{w}^{t})\) become equivalent.

Per Round Progress

Define Lyapunov function, for any \(\frac{\eta_s^2 L}{2} <\delta <\frac{\eta_s}{2}\) and \(\alpha\geq0\): \[\begin{align} \psi^{t+1}=F(w^{t+1})+(\delta-\frac{\eta_s^2 L}{2})\|\Delta^t\|^2+\alpha \underbrace{\frac{1}{N} \sum_{i=1}^N \|\nabla F_i(w^t)-h_i^{t+1}\|^2}_{H^{t+1}} \end{align}\] Considering expectation over the randomness at the \(t\)-th round and invoking the standard descent lemma for smooth objective: \[\begin{align} &\mathbb{E}_{\mathcal{A}_t, \xi^{t} \mid \mathcal{H}^{t}} \left[ \psi^{t+1} \right] \\ &= \mathbb{E}_{\mathcal{A}_t, \xi^{t} \mid \mathcal{H}^{t}} \left[ F(w^{t+1}) + \left(\delta - \frac{\eta_s^2 L}{2}\right) \left\| \Delta^{t} \right\|^2 + \frac{\alpha}{N} \sum_{i=1}^{N} \left\| \nabla F_i(w^{t}) - h_i^{t+1} \right\|^2 \right]\\ &\leq F(w^{t}) - \frac{\eta_s}{2} \left\| \nabla F(w^{t}) \right\|^2 - \frac{\eta_s}{2} \, \mathbb{E}_{\xi^{t} \mid \mathcal{H}^{t}} \left\| \bar{g}^{t} \right\|^2 + \frac{\eta_s}{2} \, \mathbb{E}_{\xi^{t} \mid \mathcal{H}^{t}} \left\| \bar{g}^{t} - \nabla F(w^{t}) \right\|^2\\ &+ \frac{\eta_s^2 L}{2} \, \mathbb{E}_{\mathcal{A}_t, \xi^{t} \mid \mathcal{H}^{t}} \left\| \Delta^{t} \right\|^2 + \left(\delta - \frac{\eta_s^2 L}{2}\right) \mathbb{E}_{\mathcal{A}_t, \xi^{t} \mid \mathcal{H}^{t}} \left\| \Delta^{t} \right\|^2 + \frac{\alpha}{N} \sum_{i=1}^{N} \mathbb{E}_{\mathcal{A}_{t}, \xi_i^{t} \mid \mathcal{H}^{t}} \left\| \nabla F_i(w^{t}) - h_i^{t+1} \right\|^2\\ &\leq F(w^{t}) - \frac{\eta_s}{2} \left\| \nabla F(w^{t}) \right\|^2 - \frac{\eta_s}{2} \, \mathbb{E}_{\xi^{t} \mid \mathcal{H}^{t}} \left\| \bar{g}^{t} \right\|^2 + \frac{\eta_s}{2N} \sum_{i=1}^{N} \mathbb{1}_{i\in\mathcal{A}_t} \, \mathbb{E}_{\xi_i^{t} \mid \mathcal{H}^{t}} \left\| \bar{g}_i^{t} - \nabla F_i(w^{t}) \right\|^2\\ &+ \delta \, \mathbb{E}_{\mathcal{A}_t, \xi^{t} \mid \mathcal{H}^{t}} \left\| \Delta^{t} - \bar{g}^{t} + \bar{g}^{t} \right\|^2 + \frac{\alpha}{N} \sum_{i=1}^{N} \mathbb{E}_{\mathcal{A}_{t}, \xi_i^{t} \mid \mathcal{H}^{t}} \left\| \nabla F_i(w^{t}) - h_i^{t+1} \right\|^2\\ &\leq F(w^{t}) - \frac{\eta_s}{2} \left\| \nabla F(w^{t}) \right\|^2 + \left(\delta - \frac{\eta_s}{2}\right) \mathbb{E}_{\xi^{t} \mid \mathcal{H}^{t}} \left\| \bar{g}^{t} \right\|^2 + \frac{\eta_s}{2N} \sum_{i=1}^{N} \mathbb{E}_{\xi_i^{t} \mid \mathcal{H}^{t}} \left\| \bar{g}_i^{t} - \nabla F_i(w^{t}) \right\|^2\\ &+ \delta \, \mathbb{E}_{\mathcal{A}_t, \xi^{t} \mid \mathcal{H}^{t}} \left\| \Delta^{t} - \bar{g}^{t} \right\|^2 + \alpha \, \mathbb{E}_{\mathcal{A}_t, \xi^{t} \mid \mathcal{H}^{t}} \left[ \frac{1}{N} \sum_{i=1}^{N} \left\| \nabla F_i(w^{t}) - h_i^{t+1} \right\|^2 \right]\\ &\leq F(w^{t}) - \frac{\eta_s}{2} \left\| \nabla F(w^{t}) \right\|^2 + \frac{\eta_s}{2N} \sum_{i=1}^{N} \mathbb{E}_{\xi_i^{t} \mid \mathcal{H}^{t}} \left\| \bar{g}_i^{t} - \nabla F_i(w^{t}) \right\|^2\\ &+ \delta \, \mathbb{E}_{\mathcal{A}_t, \xi^{t} \mid \mathcal{H}^{t}} \left\| \Delta^{t} - \bar{g}^{t} \right\|^2 + \alpha \, \mathbb{E}_{\mathcal{A}_t, \xi^{t} \mid \mathcal{H}^{t}} \left[ H^{t+1} \right] \end{align}\] We simplify some notations: \[\begin{align} \mathbb{E}[\psi^{t+1}] &\leq F(w^t)-\frac{\eta_s}{2}\|\nabla F(w^t)\|^2 +\frac{\eta_s}{2N}\sum_{i=1}^N \mathbb{E}\|\overline{g}_i^t - \nabla F_i(w^t)\|^2 \label{eq:psi95ineq} \\ &+ \delta \underbrace{\mathbb{E}\|\Delta^t-\overline{g}^t\|^2}_{\text{variance of update}} +\alpha \mathbb{E}[H^{t+1}] \end{align}\tag{31}\] Next we apply Lemma 6, \[\begin{align} \mathbb{E}[\psi^{t+1}] &\leq F(w^t)-\frac{\eta_s}{2}\|\nabla F(w^t)\|^2 +\frac{\eta_s}{2N}\sum_{i=1}^N \mathbb{E}\|\overline{g}_i^t - \nabla F_i(w^t)\|^2+ \delta\mathbb{E}\|\Delta^t-\overline{g}^t\|^2 \\ &+\alpha [p_{avg} \frac{\sigma^2}{K}+\frac{1}{N} \sum_{i=1}^N p_i^t\mathbb{E}\|\overline{g}_i^t-\nabla F_i(w^t)\|^2\\ &+\eta_s^2L^2(1+\frac{1}{C})(1-p_{avg})\|\Delta^{t-1}\|^2+(1+C)(1-p_{min})H^t]\\ &=F(w^t)\\ &+\alpha \eta_s^2L^2(1+\frac{1}{C})(1-p_{avg})\|\Delta^{t-1}\|^2+\alpha (1+C)(1-p_{min})H^t \\ &-\frac{\eta_s}{2}\|\nabla F(w^t)\|^2+\frac{\eta_s}{2N}\sum_{i=1}^N \mathbb{E}\|\overline{g}_i^t - \nabla F_i(w^t)\|^2+ \delta\mathbb{E}\|\Delta^t-\overline{g}^t\|^2 \\ &+\alpha p_{avg} \frac{\sigma^2}{K}+\frac{\alpha}{N} \sum_{i=1}^N p_i^t\mathbb{E}\|\overline{g}_i^t-\nabla F_i(w^t)\|^2 \label{eq:psi} \end{align}\tag{32}\] For bounding within the Lyapunov recursive framework, the conditions for this recursion step are: \[\begin{align} \alpha \eta_s^2L^2(1+\frac{1}{C})(1-p_{avg})&\leq \delta -\frac{\eta_s^2 L}{2} \tag{33}\\ \alpha (1+C)(1-p_{min}) &\leq \alpha \tag{34} \end{align}\] From Eq. 34 , we know \[\begin{align} C\leq \frac{p_{min}}{1-p_{min}} \end{align}\] We select \(C=\frac{p_{min}}{2(1-p_{min})}\). From Eq. 33 , we can have many selections of \(\alpha, \delta\). For the convenience of writing, we select \(\delta=\eta_s^2L\). Then, \(\alpha\) needs: \[\begin{align} \alpha \leq \frac{1}{2L(1+\frac{1}{C})(1-p_{avg})}=\frac{p_{min}}{2L(2-p_{min})(1-p_{avg})} \end{align}\] We select \(\alpha = \frac{p_{min}}{4L(2-p_{min})}\). Using the values of \(\delta\) and \(\alpha\) we select, Eq. 32 can be written as: \[\begin{align} \mathbb{E}[\psi^{t+1}]&\leq \psi^t -\frac{\eta_s}{2}\|\nabla F(w^t)\|^2\\ &+\frac{\eta_s}{2N}\sum_{i=1}^N \mathbb{E}\|\overline{g}_i^t - \nabla F_i(w^t)\|^2+ \delta\mathbb{E}\|\Delta^t-\overline{g}^t\|^2 \\ &+\alpha p_{avg} \frac{\sigma^2}{K}+\frac{\alpha}{N} \sum_{i=1}^N p_i^t\mathbb{E}\|\overline{g}_i^t-\nabla F_i(w^t)\|^2\\ &=\psi^t -\frac{\eta_s}{2}\|\nabla F(w^t)\|^2\\ &+\frac{\eta_s}{2N}\sum_{i=1}^N \mathbb{E}\|\overline{g}_i^t - \nabla F_i(w^t)\|^2+ \eta_s^2L \mathbb{E}\|\Delta^t-\overline{g}^t\|^2 \\ &+\frac{p_{min} p_{avg} \sigma^2}{4L(2-p_{min})K} \\ &+\frac{p_{min}}{4L(2-p_{min})N} \sum_{i=1}^N p_i^t\mathbb{E}\|\overline{g}_i^t-\nabla F_i(w^t)\|^2 \end{align}\] Use Lemma 5 to bound \(\mathbb{E}\|\overline{g}_i^t-\nabla F_i(w^t)\|^2\), we can further have: \[\begin{align} \mathbb{E}[\psi^{t+1}] &\leq \psi^t -\frac{\eta_s}{2}\|\nabla F(w^t)\|^2\\ &+\frac{\eta_s}{2N}\sum_{i=1}^N \mathbb{E}\|\overline{g}_i^t - \nabla F_i(w^t)\|^2+ \eta_s^2L \mathbb{E}\|\Delta^t-\overline{g}^t\|^2 \\ &+\frac{p_{min} p_{avg} \sigma^2}{4L(2-p_{min})K} \\ &+\frac{p_{min}}{4L(2-p_{min})N} \sum_{i=1}^N p_i^t\mathbb{E}\|\overline{g}_i^t-\nabla F_i(w^t)\|^2\\ &\leq \psi^t+ \eta_s^2L \mathbb{E}\|\Delta^t-\overline{g}^t\|^2 -\frac{\eta_s}{2}\|\nabla F(w^t)\|^2 \\ &+[\frac{\eta_s}{2}+\frac{p_{min} p_{avg}}{4L(2-p_{min})}] 8\eta_c^2L^2K(K-1) \|\nabla F(w^t)\|^2\\ &+\frac{p_{min} p_{avg} \sigma^2}{4L(2-p_{min})K} \\ &+[\frac{\eta_s}{2}+\frac{p_{min} p_{avg}}{4L(2-p_{min})}] 2\eta_c^2L^2K(K-1) \frac{\sigma^2}{K}\\ &+[\frac{\eta_s}{2}+\frac{p_{min} p_{avg}}{4L(2-p_{min})}] 8\eta_c^2L^2K(K-1) \sigma_g^2 \end{align}\] Here we want the coefficient for the gradient squared norm not exceed \(\frac{-\eta_s}{4}\). To achieve this, we bound learning rates as: \[\begin{align} 8\eta_c^2L^2K(K-1)&\leq \frac{1}{4}\\ \frac{p_{min} p_{avg}}{4L(2-p_{min})} &\leq \frac{\eta_s}{2} \end{align}\] Requirements for learning rates can be easily derived here. Then, we have \[\begin{align} \mathbb{E}[\psi^{t+1}] &\leq \psi^t+ \eta_s^2L \mathbb{E}\|\Delta^t-\overline{g}^t\|^2 -\frac{\eta_s}{4}\|\nabla F(w^t)\|^2 \tag{35} \\ &+\frac{p_{min} p_{avg} \sigma^2}{4L(2-p_{min})K} \\ &+[\frac{\eta_s}{2}+\frac{p_{min} p_{avg}}{4L(2-p_{min})}] 2\eta_c^2L^2K(K-1) \frac{\sigma^2}{K}\\ &+[\frac{\eta_s}{2}+\frac{p_{min} p_{avg}}{4L(2-p_{min})}] 8\eta_c^2L^2K(K-1) \sigma_g^2 \tag{36} \end{align}\] Next, we provide a bound for the initial progress following similar steps before.

The initial progress can be bounded following similar steps above. Simlarly as Eq. 31 , \[\begin{align} \mathbb{E}[\psi^2]&\leq F(w^1)-\frac{\eta_s}{2} \|\nabla F(w^1)\|^2+\frac{\eta_s}{2N} \sum_{i=1}^N \mathbb{E}\|\overline{g}_i^t - \nabla F_i(w^1) \|^2\\ &+\delta \mathbb{E}\|\Delta^1-\overline{g}^1\|^2+\alpha \mathbb{E}[H^2] \end{align}\] where \(H^2=\frac{1}{N}\sum_{i=1}^N \|\nabla F_i(w^1)-h_i^2\|^2\). Similarly, using Lemma 5, we can bound this as: \[\begin{align} \mathbb{E}[\psi^2]&\leq F(w^1)-\frac{\eta_s}{4} \|\nabla F(w^1)\|^2 +\eta_s^2 L \mathbb{E}\|\Delta^1-\overline{g}^1\|^2\\ &+\frac{p_{min} p_{avg} \sigma^2}{4L(2-p_{min})L}\\ &+[\frac{\eta_s}{2}+\frac{p_{min} p_{avg}}{4L(2-p_{min})}] 2\eta_c^2 L^2 K(K-1) \frac{\sigma^2}{K}\\ &+[\frac{\eta_s}{2}+\frac{p_{min} p_{avg}}{4L(2-p_{min})}] 8\eta_c^2 L^2 K(K-1) \sigma_g^2\\ &+\frac{p_{min}(1-p_{min})}{4L(2-p_{min})} \frac{1}{N} \sum_{i=1}^N \|\nabla F_i(w^1)-h_i^1 \|^2 \end{align}\] Finally, we can get the convergence conclusion. From Eq. 35 , unfolding he recursion for \(t=2,3,\cdots, T\), it yields: \[\begin{align} \mathbb{E}[\psi^{t+1}] &\leq \psi^2 + \eta_s^2 L \sum_{t=2}^T \mathbb{E}\|\Delta^t-\overline{g}^t\|^2 -\frac{\eta_s}{4}\sum_{t=2}^T \mathbb{E}\|\nabla F(w^t)\|^2\\ &+\sum_{t=2}^T \frac{p_{min} p_{avg} \sigma^2}{4L(2-p_{min})K} \\ &+\sum_{t=2}^T[\frac{\eta_s}{2}+\frac{p_{min} p_{avg}}{4L(2-p_{min})}] 2\eta_c^2L^2K(K-1) \frac{\sigma^2}{K}\\ &+\sum_{t=2}^T[\frac{\eta_s}{2}+\frac{p_{min} p_{avg}}{4L(2-p_{min})}] 8\eta_c^2L^2K(K-1) \sigma_g^2 \end{align}\] \(\psi^2\) can be bounded as Eq. 32 , therefore, we can get \[\begin{align} \mathbb{E}[\psi^{t+1}] &\leq F(w^1) + \eta_s^2 L \sum_{t=1}^T \mathbb{E}\|\Delta^t-\overline{g}^t\|^2 -\frac{\eta_s}{4}\sum_{t=1}^T \mathbb{E}\|\nabla F(w^t)\|^2\\ &+\sum_{t=1}^T \frac{p_{min} p_{avg} \sigma^2}{4L(2-p_{min})K} \\ &+\sum_{t=1}^T[\frac{\eta_s}{2}+\frac{p_{min} p_{avg}}{4L(2-p_{min})}] 2\eta_c^2L^2K(K-1) \frac{\sigma^2}{K}\\ &+\sum_{t=1}^T[\frac{\eta_s}{2}+\frac{p_{min} p_{avg}}{4L(2-p_{min})}] 8\eta_c^2L^2K(K-1) \sigma_g^2 \\ &+\frac{p_{min}(1-p_{min})}{4L(2-p_{min})} \frac{1}{N} \sum_{i=1}^N \|\nabla F_i(w^1)-h_i^1 \|^2 \label{eq:sumT} \end{align}\tag{37}\] Notice that \(\min_{t\in[1,T]} \mathbb{E}\|\nabla F(w^t)\|^2 \leq \frac{\sum_{t=1}^T\|\nabla F(w^t)\|^2 }{T}\) Dividing both sides of Eq. 37 by T, after rearranging the terms, we can get \[\begin{align} \min_{t\in[1,T]} \mathbb{E}\|\nabla F(w^t)\|^2 &\leq \frac{4(F(w^1)-\mathbb{E}[\psi^T])}{\eta_s T} + 4 \eta_s L \frac{\sum_{t=1}^T \mathbb{E}\|\Delta^t-\overline{g}^t\|^2}{T} \\ &+ \frac{1}{T}\frac{p_{min} p_{avg} \sigma^2}{\eta_s L(2-p_{min})K} \\ &+[2+\frac{p_{min} p_{avg}}{\eta_s L(2-p_{min})}] 2\eta_c^2L^2K(K-1) \frac{\sigma^2}{K}\\ &+[2+\frac{p_{min} p_{avg}}{\eta_s L(2-p_{min})}] 8\eta_c^2L^2K(K-1) \sigma_g^2 \\ &+\frac{1}{T} \frac{p_{min}(1-p_{min})}{\eta_s L(2-p_{min})} \frac{1}{N} \sum_{i=1}^N \|\nabla F_i(w^1)-h_i^1 \|^2 \end{align}\] Notice \(\psi^T \geq F(w^T) \geq F(w^*)\). Therefore, we can have \[\begin{align} \min_{t\in[1,T]} \mathbb{E}\|\nabla F(w^t)\|^2 &\leq \underbrace{\frac{4(F(w^1)-F(w^*))}{\eta_s T}}_{\text{iterate initialization error}} + \underbrace{4 \eta_s L \frac{\sum_{t=1}^T \mathbb{E}\|\Delta^t-\overline{g}^t\|^2}{T}}_{\text{partial update variance error}} \\ &+ \underbrace{\frac{1}{T}\frac{p_{min} p_{avg} \sigma^2}{\eta_s L(2-p_{min})K}}_{\text{stochastic gradient error (will disappear)}} \\ &+\underbrace{[2+\frac{p_{min} p_{avg}}{\eta_s L(2-p_{min})}] 2\eta_c^2L^2K(K-1) \frac{\sigma^2}{K}}_{\text{stochastic gradient error}}\\ &+\underbrace{[2+\frac{p_{min} p_{avg}}{\eta_s L(2-p_{min})}] 8\eta_c^2L^2K(K-1) \sigma_g^2}_{\text{error from data heterogeneity (non-iid level)}} \\ &+\underbrace{\frac{1}{T} \frac{p_{min}(1-p_{min})}{\eta_s L(2-p_{min})} \frac{1}{N} \sum_{i=1}^N \|\nabla F_i(w^1)-h_i^1 \|^2}_{\text{memory initialization error}} \end{align}\]

8 Additional Experimental Details↩︎

8.1 Detailed Hyperparameters and Model Structures↩︎

For reproducibility, we provide a detailed table (Table 2) specifying the exact model architectures and hyperparameters that we used in the experiments.

8.2 Detailed Comparison of Communication and Compute Overhead↩︎

FedSteer is designed specifically for resource-constrained edge devices. We provide a detailed comparison below:

Communication Cost (Identical to FedVARP/FedAvg): FedSteer does not incur any additional communication costs compared to standard FL. Active clients send new updates; while inactive clients send nothing (they do not perform local training when they are inactive). No additional bits or control variates are transmitted during training. The dynamic subspace optimization happens on the server only.

Client Compute (Zero Additional Cost Compared to FedAvg): FedSteer’s projection logic (solving \(s_i\)) happens entirely on the server. Clients only perform standard local training when it participates.

Server Compute (Negligible): The server solves a Ridge Regression problem for active clients (Eq. (5) in the paper) to derive the coordinates \(\mathbf{s}_i\). The cost of directly computing the closed-form solution (Eq. (6)) is \(O(k^2d+k^3)\), where \(d\) is the model dimension and \(k=|\mathcal{X}|\) is the core set size. Compared to the cost of standard weight aggregation (\(O(N d)\)), directly computing the closed-form may yield comparable cost (for example, N=100, k=10), normally the server is computationally powerful, so they can handle it. In the case that the server is also computation limited, it can alternatively solve it via gradient descent to avoid matrix operations entirely, as the problem is strictly convex. The core set selection (Algorithm 2) is restricted to a short "warm-up" phase, then the core set is fixed – this overhead drops to zero.

Server Memory (Superior Efficiency): FedSteer is significantly more memory-efficient than FedVARP/FedStale/MIFA. FedVARP/FedStale/MIFA must store full stale gradients for all \(N\) clients (\(O(N \cdot d)\)). FedSteer stores projection coordinates (\(\mathbf{s}_i \in \mathbb{R}^{k}\)) for all clients, and full stale gradients only for the small core set (\(k \ll N\)). Therefore the cost is \(O((N \cdot k + k \cdot d)\). For large models (\(d\)) and many clients (\(N\)), this is a massive reduction. In experiments, we set \(k=10, N=100\), resulting in reduced memory usage by around 10 times.

8.3 Scalability via Subsampling↩︎

To ensure scalability for very large \(N\), we introduce a random subsampling strategy during core-set selection. Instead of searching over the entire client population, we randomly sample a subset \(N_{\text{sub}} \ll N\) (e.g., 20 candidates out of 100) and perform the greedy swap search within this subset. This reduces the selection complexity to \(O\!\left(T_{0} I_{\max} N_{\text{sub}} k \right).\) Since \(N_{\text{sub}}\) is small and fixed (e.g., \(N_{\text{sub}}=20\)), the computational cost becomes independent of the total population size \(N\), enabling scalability to large-scale FL systems. We further evaluate this subsampling strategy. As shown in Table 3, subsampling incurs only a minor accuracy drop (approximately 2.5% on average across two settings) while still outperforming all baselines.

8.4 Additional Experimental Results↩︎

Figures 6 and 7 empirically demonstrate the convergence behavior of FedSteer against multiple baseline algorithms over 150 global training iterations under extreme heterogeneity (\(\gamma=0.9\)). Across both the Fashion-MNIST and CIFAR-10 tasks, FedSteer achieves stable convergence and significantly outperforms all evaluated baseline methods, obtaining a final accuracy superseded only by the theoretical “full participation” upper bound.

8.5 Effectiveness of reusing \(\mathbf{s}_i\) across rounds↩︎

Figure 8 illustrates the evolution of the projection coefficients, \(s_i\), for three representative clients (5, 55, and 95) under different experimental conditions. We present a 2x2 comparison varying the regularization strength (\(\lambda\)) and the core set size (\(|X|\)). Each heatmap plots the coefficient vector \(s_i\) (y-axis) against the communication round (x-axis). The most prominent observation across all settings is the stability of these coefficients over time. The vertical patterns, which represent the distribution of \(s_i\) at a given round, exhibit slight changes as training progresses. This indicates that the optimal projection of a client’s data onto the shared base vectors remains largely consistent. The result is conducted with FedSteer (enforce=5) under EMNIST. The observed stability supports FedSteer to reuse projection coefficients instead of the stale gradients, as the coefficients do not change significantly between rounds. Furthermore, the distinct patterns among clients (e.g., the contrast between client 55 and the others) highlight the method’s ability to effectively capture stable, yet heterogeneous, client-specific representations.

References↩︎

---

1. The work was performed while the author was affiliated with Carnegie Mellon University.↩︎

2. In experiments, we set \(N=100\), \(|\mathcal{X}|=10\), and the model dimension \(d\) exceeds \(1.6\times 10^6\). With these values, our method reduces the gradient-caching memory overhead by nearly an order of magnitude.↩︎

3. To ensure a fair comparison with baselines, the global model’s weights are re-initialized before the formal training process begins.↩︎