1 Introduction↩︎

Federated learning (FL) [1] enables multiple clients to jointly train a model without centralizing raw data, making it attractive for privacy-sensitive applications and for the distributed adaptation of large language models. As foundation models grow, communicating and locally updating full model weights becomes prohibitive for FL clients, so parameter-efficient fine-tuning has become essential to federated adaptation. Low-Rank Adaptation (LoRA) [2] is the dominant choice, where each client trains and communicates only a low-rank update \(\Delta W = BA\) instead of full model weights, reducing both local compute and communication cost by orders of magnitude.

LoRA-based FL, however, suffers heavily from data heterogeneity. When clients hold different data distributions or task semantics, their local updates conflict after aggregation, causing client drift, unstable optimization, and poor global performance. Some literature addresses this problem by modifying the optimization dynamics through proximal regularization, momentum, normalized updates, or control variates [1], [3]–[5], or by introducing auxiliary data, personalized models, and knowledge distillation [6]–[13]. Subspace constraints have also proven effective in centralized settings for continual learning, multi-task learning, parameter-efficient adaptation, and model editing [14]–[21], and recent federated LoRA work studies how adapters should be personalized, partitioned, or aggregated under heterogeneity [22]–[29]. Most literature attempts to resolve conflicts among model parameters and align them before or after aggregating local models.

In this paper, however, we trace the degradation to a structural mechanism that these approaches leave unaddressed. Aggregating LoRA adapters allows one client’s adapter to act on another client’s representations: for two clients aggregated by \(W_{\mathrm{merge}} = W + B_1 A_1 + B_2 A_2,\) evaluating the merged model on an activation \(h_1\) from client 1 produces the cross-client term \(B_2 A_2 h_1\), which applies client 2’s adapter to client 1’s features and can perturb client 1’s predictions after aggregation. We call this effect data-parameter interference. Importantly, the interference depends on the alignment between the row space of \(A_2\) and the activation subspace of client 1, so mitigating it requires controlling the subspaces in which clients update. No existing line of work exercises this control. Optimization-level remedies manipulate the training dynamics and gradients, while federated LoRA designs reorganize the adapters to be shared. Centralized subspace methods propose to constrain update subspaces, but they either violate the FL protocol or rely on a fixed subspace which can be stale during training. Removing interference therefore calls for a new approach, in which each client should update in a subspace that is expressive for its own data yet insensitive to other clients’ data. Such subspace allocation must be federated, using only local information, and dynamic, adapting as representations evolve.

To address the data-parameter interference under the two requirements, we propose Dynamic Subspace Boosting (Dysco), a plug-in method that treats federated LoRA aggregation as a subspace-allocation problem. In each communication round, every client first computes an activation-insensitive subspace from its local hidden representations by solving an orthonormal projection problem with a closed-form solution. Only the resulting basis is transmitted to the server, so raw data and intermediate activations remain local and the procedure stays federated. The server then constructs a client-specific merged subspace by maximizing its overlap with the insensitive subspaces of the other clients. Finally, to remain dynamic under representation drift, Dysco performs multi-round subspace boosting to preserve useful directions from earlier rounds while adapting future updates to the current representation geometry.

Theoretically, we provide a convergence analysis by examining the data-parameter interference as an aggregation-error term, and show that Dysco yields a tighter bound on this error under data heterogeneity. Empirically, we evaluate Dysco on controlled synthetic federated tasks and on MIMIC-IV clinical-note classification in the main body, and report additional GLUE benchmark experiments in the appendix. On synthetic federated regression tasks under the orthogonal-subspace partition that the theory identifies, Dysco drives the training loss down substantially faster than FedAvg-style baselines, reducing the final-round training loss by a factor of up to \(9\) across Transformer depths and LoRA ranks. On MIMIC-IV clinical note classification [30] with Llama-3.2-1B [31], Dysco improves all five baselines, with a headline gain of \(4.3\) points on FedAvgM [32] and a near-flat \(0.5\) on Scaffold [5] from a degenerate \(50\%\) baseline. It also outperforms recent federated LoRA baselines, including FFA-LoRA [33] and FedSA-LoRA [34], while adding only \(0.9\%\) wall-clock overhead on average. Consistent average-accuracy improvements across baselines are observed on the GLUE benchmark [35] with RoBERTa-large [36], with per-task variability reflecting the higher cross-task interference of an eight-task federation.

1.0.0.1 Contributions.

We summarize our main contributions as follows:

• We show that aggregating LoRA adapters creates cross-client terms, revealing a concrete mechanism by which heterogeneity destabilizes LoRA-based FL, and formulate the model aggregation as a subspace-allocation problem.

• We propose Dysco, which combines local activation-insensitive subspace selection, closed-form server-side subspace merging, and multi-round subspace boosting to adapt to representation drift. We analyze the convergence of Dysco and prove that it yields a tighter upper bound on the aggregation error.

• Through extensive experiments under synthetic settings and real benchmarks such as GLUE with RoBERTa-large and MIMIC-IV-note with Llama-3.2-1B, we show the significant improvement of global performance, fewer rounds to converge, and negligible computational overhead against baselines, across various client scales, LoRA ranks, and block selections.

2 Preliminaries↩︎

2.1 Federated Learning↩︎

We consider a federated learning setting with \(N\) clients indexed by \(i\in\{1,\dots,N\}\), trained over \(R\) communication rounds indexed by \(t\in\{0,\dots,R-1\}\). Client \(i\) holds a dataset \(\mathcal{D}_i\) drawn from its local distribution, which may differ substantially from other clients’ data distributions. The tasks can be different across clients, or they could be the same task but with differently distributed inputs. We denote the model parameters as \(W\). Each client \(i\) has a population objective \(f_i\), and the global objective is \(f(W) = \frac{1}{N}\sum_{i=1}^{N} f_i(W)\), with minimum value \(f^\star = \inf_W f(W)\). In standard FL, all clients share the same model architecture \(W\) and aim to learn a single global \(W\) that is effective for all \(\mathcal{D}_i\).

2.2 Low-Rank Adaptation↩︎

In this paper, we mainly focus on training language models with low-rank adaptation (LoRA). Rather than training the full model \(W\) on each client, which can be large and costly to communicate, each client only learns a low-rank update to \(W\). Specifically, at round \(t\) client \(i\) holds trainable matrices \((B_i^t, A_i^t)\in\mathbb{R}^{d\times r}\times\mathbb{R}^{r\times d}\) for a small rank \(r \ll d\), where \(d\) is the dimension of the weight matrix, and modifies the model as \(W + B_i^t A_i^t\). Initially, \(B_i^0 A_i^0\) is zero, and each client trains its LoRA factors on local data while keeping \(W\) fixed. This low-rank training drastically reduces communication, as each client only sends the \(B_i^t\) and \(A_i^t\) matrices of size \(O(dr)\) instead of the full \(W\) of size \(O(d^2)\). The canonical FedAvg-LoRA aggregation yields the global model \(W + \frac{1}{N}\sum_{i=1}^NB_i^t A_i^t\) at the end of round \(t\). We suppress the round superscript and write \((B_i, A_i)\) when the round index is not essential. In our approach, these LoRA matrices also serve as the foundation for coordinating subspaces.

3 Our Method↩︎

We address data heterogeneity in LoRA-based federated learning by controlling the update subspaces in which client adapters operate. Rather than first averaging client parameters and then correcting the resulting conflicts, we allocate client-specific subspaces that reduce cross-client interference during aggregation. We begin by showing that interference arises when an adapter learned from one client acts on the representations of another client, which motivates a subspace-allocation view of federated LoRA (3.1). This view naturally suggests using activation-insensitive directions. However, making this principle work in FL is challenging because client representations cannot be pooled and the relevant subspaces evolve throughout training. We then present Dynamic Subspace Boosting (Dysco), which implements subspace allocation in a federated and dynamic manner through local subspace selection, server-side subspace merging, and multi-round subspace boosting (3.2). Finally, we briefly summarize the theoretical analysis in 3.3, deferring the full statement and proofs to 9. The complete algorithm is summarized in 2, and an overview is shown in 1.

3.1 From Interference to Subspace Allocation↩︎

3.1.0.1 Interference in LoRA Aggregation.

We first identify a geometric source of negative transfer in LoRA-based FL. Consider two clients whose LoRA adapters are aggregated into \[W_{\mathrm{merge}} = W + B_1A_1 + B_2A_2 .\] When this merged model is evaluated on an activation \(h_1\) from client 1, the forward computation decomposes as \[\begin{align} W_{\mathrm{merge}}h_1 &= (W + B_1A_1 + B_2A_2)h_1 \notag \\ &= \underbrace{(W+B_1A_1)h_1}_{\text{client-1 adaptation}} + \underbrace{B_2A_2h_1}_{\text{cross-client interference}} . \label{eq:interference} \end{align}\tag{1}\] The second term applies the adapter learned from client 2 to client 1’s representation. When client distributions are heterogeneous, \(A_2h_1\) need not be small, so the update from one client can alter the predictions of another in an uncontrolled way. This effect is particularly pronounced in LoRA because the right factor \(A_i\) selects the input directions along which client \(i\) is allowed to update the model. For brevity, we omit layer indices unless otherwise specified.

3.1.0.2 Aggregation as Subspace Allocation.

Equation 1 suggests that the role of the server is not only to average client updates, but also to decide where those updates should live. The interference term disappears whenever the update subspace of client 2 is orthogonal to client 1’s activations, namely \(A_2h_1 \approx 0\), regardless of the learned left factor \(B_2\). Thus, controlling the row space of \(A_i\) provides a direct mechanism for controlling cross-client interference.

We therefore view federated LoRA aggregation as a subspace allocation problem. For each client, the server should allocate an update subspace that is expressive for that client’s local task while being insensitive to the representations of other clients. Standard FedAvg-LoRA aggregation corresponds to the degenerate allocation in which every client may update the same ambient space, leaving the interference term in 1 unconstrained. Our goal is instead to allocate client-specific low-dimensional subspaces before local training, so that subsequent updates are less likely to activate other clients’ representations.

3.1.0.3 Learning Insensitive Subspaces.

The basic building block is an insensitive subspace, a set of directions that produces minimal response on a client’s activations. Let \(H_i \in \mathbb{R}^{n_i \times d}\) denote the matrix of latent representations collected from client \(i\), with each row corresponding to one activation vector. For a target rank \(r\), we compute an orthonormal basis \[\label{eq:compute32subspace} A_i^* = \arg\min_{A \in \mathbb{R}^{r \times d}} \| A H_i^\top \|_F^2 \qquad \text{s.t. } AA^\top = I_r .\tag{2}\] The objective measures the total energy of client \(i\)’s activations after projection onto the row space of \(A\). Minimizing it selects directions to which client \(i\) is least sensitive, while the orthonormality constraint prevents the trivial zero solution and fixes the subspace dimension.

This problem has a closed-form solution. Let \[C_i = H_i^\top H_i\] be the empirical activation Gram matrix, and write its eigendecomposition as \(C_i = Q_i \Sigma_i Q_i^\top\), with eigenvalues \(\sigma_1 \ge \sigma_2 \ge \cdots \ge \sigma_d \ge 0\). Then \(A_i^*\) is given by the \(r\) eigenvectors associated with the smallest eigenvalues of \(C_i\). Equivalently, \(A_i^*\) spans the approximate null space of client \(i\)’s representation distribution. If another client’s update is constrained to this subspace, then its right LoRA factor produces little response on client \(i\)’s data, thereby reducing the interference term in 1 .

3.1.0.4 Toward Federated and Dynamic Allocation.

The insensitive-subspace criterion above suggests a natural way to suppress interference by updating the model along directions that are weakly activated by the data on which the update should have minimal effect. In a centralized setting, such directions could be computed from pooled representations by constructing \(H_{\neg i}\) from the activations of clients other than \(i\) and solving an objective of the form \[\arg\min_A \|A H_{\neg i}^{\top}\|_F^2 .\] This centralized variant subsumes prior null-space-constrained methods such as Null-LoRA [16], AlphaEdit [18], and related work on knowledge editing and model merging [17], [20], which solve the same insensitive-subspace objective on pooled centralized activations and treat the resulting basis as fixed. In this sense, centralized null-space-LoRA is the static, data-pooled special case of the subspace-allocation problem we study here. However, this direct construction is not suitable for federated learning. First, it requires access to cross-client activations, whereas FL is designed to avoid sharing client data or intermediate representations. Second, the relevant activation geometry is not fixed. As local training and global aggregation update the model, the representations seen by each client also change, so a subspace computed once can gradually become misaligned.

These two constraints make subspace allocation in FL fundamentally different from its centralized counterpart. The allocation must be federated, using only locally computed information, and dynamic, adapting as client representations drift across communication rounds. In the next section, Dysco is proposed to realize this federated and dynamic form of subspace allocation.

3.2 Dysco for Dynamic Subspace Boosting↩︎

Dysco solves the subspace allocation problem in the federated and dynamic setting identified above. It consists of three steps aligned with the two constraints of FL. Local subspace selection and server-side subspace merging address the information constraint by allowing clients to compute insensitive subspaces locally and allowing the server to allocate subspaces using only transmitted bases, without accessing raw features or intermediate representations. Multi-round subspace boosting addresses the temporal constraint by recomputing and accumulating subspaces as representations evolve during federated training.

3.2.0.1 Local Subspace Selection.

Each client \(i\) solves 2 using only its local data and obtains an insensitive subspace \(A_i^* \in \mathbb{R}^{r \times d}\). The client then sends only the orthonormal basis \(A_i^*\) to the server. This preserves the FL protocol because no raw samples or intermediate activations are shared. However, \(A_i^*\) captures directions that are insensitive to client \(i\)’s own data, while the update subspace assigned to a client should ideally be insensitive to the data of the other clients. The server therefore needs to combine the locally computed subspaces into client-specific allocations.

3.2.0.2 Server-Side Subspace Merging.

After collecting \(\{A_1^*, A_2^*, \dots, A_N^*\}\) from all clients, the server constructs a merged subspace \(A^{\mathrm{merge}}_i\) for each client \(i\). The goal is to allocate to client \(i\) directions that align well with the insensitive subspaces of the other clients. We formulate this merging step as \[\label{eq:merge95obj} A^{\mathrm{merge}}_i = \arg\max_{A \in \mathbb{R}^{r \times d}} \sum_{j=1;\, j \ne i}^{N} \| A A_j^{* \top} \|_F^2 \qquad \text{s.t. } A A^\top = I_r .\tag{3}\] The quantity \(\|A A_j^{*\top}\|_F^2 = \operatorname{tr}(A A_j^{*\top} A_j^* A^\top)\) measures the overlap between the candidate subspace \(A\) and the insensitive subspace of client \(j\). Maximizing the aggregate overlap selects directions that are broadly compatible with the clients whose predictions should not be disturbed.

Problem 3 also admits a closed-form solution. Define \[S_i = \sum_{j=1;\,j \ne i}^N A_j^{* \top} A_j^* \in \mathbb{R}^{d \times d},\] which aggregates the projectors onto the insensitive subspaces of all clients except \(i\). Let \(S_i = V \Lambda V^\top\), where the eigenvalues satisfy \(\lambda_1 \ge \lambda_2 \ge \cdots \ge \lambda_d \ge 0\) and the corresponding eigenvectors are \(V = [v_1, \dots, v_d]\). The optimal merged subspace is given by the top-\(r\) eigenvectors of \(S_i\), \[\label{eq:solve32merged32subspace} A^{\mathrm{merge}}_i = [\, v_1^\top;\; v_2^\top;\; \cdots;\; v_r^\top \,],\tag{4}\] up to an arbitrary \(r \times r\) rotation. The server then broadcasts \(A^{\mathrm{merge}}_i\) back to client \(i\).

The matrix \(S_i\) summarizes how strongly each direction is supported by the other clients’ insensitive subspaces. When these subspaces share common directions, those directions become dominant eigendirections of \(S_i\). When the subspaces differ, the top eigenspace balances directions shared by many clients against directions that are strongly preferred by a smaller subset.

3.2.0.3 Multi-Round Subspace Boosting.

After receiving \(A^{\mathrm{merge}}_i\), each client initializes its right LoRA factor with the merged subspace and trains the left factor \(B_i\) on local data. Because the model changes after local training and server aggregation, the latent representations also change across communication rounds. Let \(H_i^t\) denote the feature matrix of client \(i\) at round \(t\). Solving 2 with \(H_i^t\) yields a local subspace \(A_{i,t}^*\), and the subsequent merging step produces \(A_i^{\mathrm{merge},t}\). After one round of training and aggregation, the feature matrix shifts to \(H_i^{t+1}\), so the previously merged subspace may no longer match the current representation geometry.

To handle this representation drift, Dysco progressively boosts subspaces across rounds instead of overwriting them. At round \(t+1\), clients recompute local subspaces from the updated features \(H_i^{t+1}\), and the server constructs new merged subspaces \(A_i^{\mathrm{merge},t+1}\). Each client freezes the previously learned adapter pair \((B_i^t, A_i^{\mathrm{merge},t})\) and adds a new LoRA block \((B_i^{t+1}, A_i^{\mathrm{merge},t+1})\), where \(B_i^{t+1}\) is trained from scratch. The frozen adapters preserve previously learned update directions, while the newly added block adapts to the current representation geometry. The final model parameters take the form \[\label{eq:boosted95param} W = W_0 + \sum_{t=0}^{R-1} B_i^t\, A_i^{\mathrm{merge},t},\tag{5}\] where \(R\) is the total number of communication rounds. Following [20], we relax the orthonormality constraint of \(A_i^{\mathrm{merge},t}\) during local training, allowing the subspace to be fine-tuned jointly with \(B_i^t\) for improved performance.

3.3 Optimization Comparison with FedAvg-LoRA↩︎

The data-parameter interference of 1 enters the standard nonconvex local-SGD convergence bound as an additive aggregation-error term. For FedAvg-LoRA this term is bilinear in the average cross-client drifts of both LoRA factors \(\Delta_A^{t+1}\Delta_B^{t+1}\), because the server averages both \(A\) and \(B\) across clients whose locally trained subspaces disagree under data heterogeneity. Dysco fixes the right factor on a per-client basis to the server-merged \(A_i^{\mathrm{merge},t+1}\), both during local training and at global application, which eliminates the \(A\)-drift contribution by construction and collapses the interference to a linear function of the left-factor drift alone, scaling as \(\rho_A^{t+1}\Delta_B^{t+1}\) with \(\rho_A^{t+1}\) the maximum squared spectral norm of the merged right factors and \(\rho_A^{t+1}=1\) for orthonormal merged subspaces. This structural collapse yields a \(\Theta(\Delta_A/\rho_A)\) tightening of the per-round stationarity bound and a \(\Theta(K^2)\) widening advantage in the local-step count \(K\), under standard data-heterogeneity conditions. The full statement, assumptions, and proofs are deferred to 9.

4 Synthetic Experiments↩︎

4.1 Experimental Setup↩︎

We construct a controlled federated scenario in which the subspace-allocation view can be tested directly. There are \(N\) clients, each with a vector-valued task: for client \(i\) we draw a column-orthonormal subspace \(U_i \in \mathbb{R}^{d_{\mathrm{model}} \times r}\) and generate input tokens \(x = Z_i U_i^\top\) that lie entirely in \(U_i\), with labels produced by a per-client target operator \(W_i^\star = (1-\alpha)\,W_{\mathrm{shared}} + \alpha\,W_{\mathrm{perturb},i}\) at the maximum-heterogeneity setting \(\alpha=1\). Local training uses a Transformer encoder with LoRA adapters of rank \(r\) attached to selected attention projections from \(Q\), \(K\), \(V\), and all non-LoRA parameters are frozen. We report the merged global model’s average training loss and the corresponding top-1 test accuracy to measure the generation capability of the Transformer. Full data-generation details, architecture specification, and training hyperparameters are deferred to 10.1.

4.2 Main Results↩︎

3 reports training loss and test accuracy for stacked Transformers of depth \(L\in\{1,2,5\}\). Dysco substantially outperforms FedAvg at every depth: its final training loss is \(0.11\) against FedAvg’s \(0.57\) for \(L=1\), \(0.06\) against \(0.51\) for \(L=2\), and \(0.05\) against \(0.46\) for \(L=5\), with each gap exceeding the seed standard deviation. By round \(20\), Dysco’s training loss has already fallen below FedAvg’s round-\(100\) loss at every depth, showing an order-of-magnitude reduction in communication cost at matched performance. The same pattern holds on test accuracy: Dysco reaches \(0.53\), \(0.59\), \(0.44\) for \(L=1,2,5\), respectively, while FedAvg remains at \(0.15\), \(0.18\), \(0.24\), respectively. By fixing the right LoRA factor at the data-aligned subspace, Dysco collapses the bilinear cross-client interference in FedAvg-LoRA to a linear function of the drift in \(B\) alone, and this advantage persists as the network deepens.

4.3 Effect of Learnable LoRA Blocks↩︎

4 compares LoRA on \(V\) alone with LoRA on \(Q\!+\!K\!+\!V\); the underperforming \(Q\)-only and \(K\)-only ablations are reported in 10.2. Restricting LoRA to \(V\) alone recovers most of the gain of the full \(Q\!+\!K\!+\!V\) configuration. Dysco’s final training loss is \(0.08\) against \(0.06\), and its test accuracy is \(0.58\) against \(0.59\), while FedAvg stays above \(0.51\) loss and at most \(0.18\) accuracy in either column. The mechanism is that \(V\) writes content into the residual stream, so a client-aligned LoRA basis lands one client’s value updates in directions orthogonal to other clients’ representations, whereas \(Q\) and \(K\) enter only through a softmax that is largely insensitive to the LoRA basis. Aligning the subspace of the projection that writes into the residual stream is therefore necessary and nearly sufficient for the Dysco lift in this controlled setting.

4.4 Scaling with Number of Clients↩︎

5 sweeps \(N\in\{4,16,100\}\) at rank \(r=4\), and the boundary case \(N=64\) is deferred to 10.3. At \(N=4\) where \(N\!\cdot\!r=16\) is far below \(d=128\), the strict orthogonal partition is feasible and Dysco drives the training loss to \(0.05\) while FedAvg plateaus at \(0.36\). At \(N=16\), FedAvg stagnates near its initial loss for the full \(100\) rounds, whereas Dysco still cuts the loss by an order of magnitude to \(0.10\). At \(N=100\), the strict partition is no longer feasible and Dysco partially degenerates toward FedAvg by construction; its final loss is \(0.30\) against FedAvg’s \(0.55\). Test accuracy follows the same trend. Dysco yields \(0.55\), \(0.41\), \(0.17\) for \(N=4,16,100\), respectively, while FedAvg collapses to the random-chance floor of \(0.10\) as soon as \(N\) leaves the strict-orthogonal regime. The data-aligned allocation therefore continues to extract benefit from the per-client random orthonormal fallback, since independent random orthonormal \(U_i\) still suppress the cross-client bilinear interference term in expectation even without strict mutual orthogonality.

4.5 Effect of LoRA Rank↩︎

6 sweeps the LoRA rank \(r\in\{8,16,32\}\), and the small-capacity case \(r=4\) is deferred to 10.4. Higher ranks amplify cross-client interference for FedAvg, whose loss grows from \(0.51\) at \(r=8\) to \(0.76\) at \(r=16\) and \(1.21\) at \(r=32\), while Dysco’s loss is \(0.06\) at \(r=8\), \(0.11\) at \(r=16\), and \(0.26\) at \(r=32\), with the gap widest at \(r=8\). The test accuracy has a similar trend, where Dysco’s accuracy is \(0.59\) at both \(r=8\) and \(r=16\) and \(0.53\) at \(r=32\), while FedAvg’s accuracy never exceeds \(0.24\). Higher-dimensional adapters are thus useful only when each client’s update is allocated to a subspace insensitive to the others’.

5 Real-World Experiments↩︎

5pt

5.1 Experimental Setup↩︎

5.1.0.1 Datasets.

We evaluate Dysco’s per-task accuracy on the MIMIC-IV clinical-note benchmark in the main body and defer the corresponding per-task GLUE results to 11.1. MIMIC-IV-note [30], [37] is a dataset consisting of de-identified free-text clinical notes for patients in the MIMIC-IV database [38], [39], containing diverse note types, including discharge summaries, radiology reports, nursing notes, and physician progress notes, spanning multiple hospital departments and care settings. This dataset provides a realistic and challenging benchmark for evaluating models on clinical language understanding tasks due to its heterogeneity, domain-specific terminology, and unstructured nature. Following standard practice, we preprocess the notes by removing protected health information and normalizing text. We select four diseases, including neurological disorders (NEURO), liver diseases (LIVER), coagulation deficiency (COAG), and weight loss (WGHTLOSS). Each client is assigned a binary classification of a disease.

5.1.0.2 Models.

For the MIMIC-IV-note dataset, we use Llama-3.2-1B [31]. Llama-3.2-1B is a compact decoder-only large language model that yields strong performance per parameter. We use Llama-3.2-1B to evaluate the behavior of our method on a modern decoder-only LLM backbone in a federated clinical-text setting.

5.1.0.3 Baselines.

As Dysco is a plug-in method that enhances existing FL algorithms, we select five commonly used algorithms in FL and compare the performance w/ and w/o Dysco, including FedAvg [1], FedAvgM [32], Scaffold [5], FedProx [3], and FedNova [4].

5.1.0.4 Configurations.

During training, we use the AdamW optimizer [40] with a warmup ratio of \(0.06\) and a linear learning rate schedule. Following [2], LoRA is set with a rank of \(r=8\), a scaling factor of \(\alpha=16\), and is only applied to the query and value blocks. The number of local epochs is \(5\), and the total number of communication rounds is \(30\).

5.2 Main Results↩︎

5.2.0.1 Training Loss Convergence.

7 plots the average training loss versus communication round for both benchmarks under the standard configuration described in 5.1. On GLUE, Dysco’s loss drops below FedAvg’s after round \(3\) and remains lower for the rest of training, reaching \(0.06\) at round \(30\) while FedAvg plateaus at \(0.10\). On MIMIC-IV-note, the gap opens earlier and persists for the full \(30\) rounds, with Dysco’s loss settling at \(0.08\) against FedAvg’s \(0.11\). The speed improvement on MIMIC-IV-note is less significant than that on GLUE, since clinical notes have more similar underlying semantics and the domain gap is smaller. It implies the benefit of Dysco under heavy data heterogeneity. For brevity, in the following sections, we only show the results of MIMIC-IV-note in the main body and defer those of GLUE in 11

5.2.0.2 Final Accuracy.

1 summarizes the classification accuracies (%) on the MIMIC-IV-note dataset across the four clinical prediction tasks as well as the averaged performance over all tasks. Dysco leads to consistent and substantial improvements in global model performance across nearly all tasks and FL settings. When averaged over tasks, enabling Dysco improves accuracy for every baseline considered, with gains ranging from \(0.5\%\) for Scaffold up to \(4.3\%\) for FedAvgM. This indicates that the benefits of Dysco are robust and not tied to a specific optimization strategy or aggregation rule. At the task level, Dysco yields clear improvements for the majority of method-task combinations, with peak per-task gains of \(3.0\%\) for FedProx on COAG and \(6.7\%\) for FedAvgM on NEURO. The contribution of each Dysco component is empirically validated in 11.2. The corresponding analysis on the GLUE benchmark is provided in 11.1.

5.2.0.3 Comparison with State-of-the-Art Federated LoRA Methods.

We compare Dysco against two recent federated LoRA methods, FFA-LoRA [33] and FedSA-LoRA [34], which address heterogeneity by freezing or structurally partitioning LoRA adapters. 3 reports the performance on the MIMIC-IV-note dataset. FedAvg with Dysco outperforms both FFA-LoRA and FedSA-LoRA, demonstrating that dynamically resolving subspace interference is more effective than static structural constraints.

5pt

5.2.0.4 Computational Overhead.

We measure the wall-clock time per federated round for three representative FL methods, including FedAvg, FedProx, and Scaffold, across five settings spanning MIMIC-IV-note with Llama-3.2-1B and the GLUE configurations evaluated in 11.1, each running a single outer round. 2 reports the average wall-clock time per round for each method. Across all three methods and five settings, Dysco adds only \(0.9\%\) wall-clock overhead on average, with per-method averages ranging from \(0.6\%\) for Scaffold to \(1.5\%\) for FedAvg. The subspace computation and cross-client aggregation introduced by Dysco are dominated by the LoRA training cost, making the additional overhead negligible in practice. A per-setting breakdown is provided in 11 in the appendix.

6 Related Work↩︎

6.0.0.1 Handling Data Heterogeneity in FL.

Data heterogeneity is a central challenge in federated learning because local optimization on client-specific distributions can induce client drift and degrade the aggregated global model [41], [42]. A large body of work improves robustness to non-IID data by modifying the federated optimization procedure. FedProx constrains local updates with a proximal term [3], Scaffold uses control variates to correct biased local directions [5], and FedNova normalizes client updates to reduce objective inconsistency caused by heterogeneous local training [4]. Other approaches use auxiliary data, generated data, or distillation to reduce distribution mismatch [6]–[9], while personalized FL learns client-specific models or adaptations for heterogeneous populations [10]–[13]. These methods primarily address heterogeneity through optimization, calibration, auxiliary information, or personalization. Our work is complementary. We retain a single shared global model, but control the LoRA update subspaces in which client updates are learned, thereby reducing cross-client interference before aggregation.

6.0.0.2 Subspace and Null-Space Methods.

Subspace constraints have been widely used to reduce interference among tasks, updates, or model edits. In continual learning, gradient projection methods restrict new updates to directions that minimally affect previous tasks [14]. In multi-task learning, gradient surgery removes conflicting components across task gradients [15]. Related ideas also appear in parameter-efficient adaptation, where Null-LoRA constrains LoRA updates to the null space of the pre-trained model [16], as well as in knowledge editing and model merging, where null-space or orthogonal-subspace constraints are used to preserve existing behavior while incorporating new information [17]–[21], [43], [44]. Most of these methods compute the relevant subspace from centralized information or use a fixed subspace throughout the procedure. In contrast, Dysco targets the federated setting where raw data and intermediate activations cannot be pooled, and where the activation geometry changes across communication rounds. It therefore performs subspace selection locally, merges only transmitted bases on the server, and dynamically refreshes the allocation through multi-round boosting.

6.0.0.3 Federated LoRA.

LoRA has become a common tool for parameter-efficient fine-tuning of large language models by learning low-rank adapters while freezing the backbone [45]. Recent work extends LoRA to FL to reduce communication and computation in distributed fine-tuning. pFedLoRA and FDLoRA introduce personalized LoRA designs for model or data heterogeneity [22], [23]. FFA-LoRA and FedSA-LoRA improve federated adapter aggregation by freezing or selectively sharing LoRA factors [24], [25]. Other methods, including HetLoRA, FlexLoRA, FLoRA, and LoRA-A\(^2\), support heterogeneous ranks, tasks, or client resources through specialized aggregation and rank-allocation strategies [26]–[29]. These methods focus on how LoRA adapters should be shared, personalized, or aggregated under federated constraints. Dysco addresses a different and complementary issue: the data-parameter interference that arises when an aggregated adapter from one client acts on another client’s representations. By allocating, merging, and boosting client-specific update subspaces, Dysco can be plugged into standard FL algorithms to improve LoRA-based federated fine-tuning under heterogeneity.

7 Conclusion↩︎

We studied LoRA-based federated fine-tuning under client heterogeneity and traced unstable aggregation to data-parameter interference, where client updates collide when projected into a shared subspace. This motivates a subspace-allocation view, in which client updates should occupy subspaces that are locally expressive yet insensitive to other clients’ representations. We proposed Dysco, a plug-in method that instantiates this view through local activation-insensitive subspace selection, closed-form server-side subspace merging, and multi-round subspace boosting that re-allocates as representations evolve. Empirically, Dysco shrinks the final-round training loss by roughly \(5\)–\(9\times\) over FedAvg-style baselines on controlled synthetic federated tasks, and on MIMIC-IV clinical-note classification it improves baselines with average gains up to \(+4.3\) points while adding negligible computational overhead.

7.0.0.1 Limitations.

Dysco assumes all clients share the same base model architecture, targeting data rather than model heterogeneity; handling heterogeneous architectures would require extra mechanisms such as distillation, module-wise merging, or architecture-aware client clustering. Dysco also relies on synchronous aggregation, since the server must collect all client subspace bases before merging, making it unsuitable for asynchronous settings.

Acknowledgment↩︎

This material is based in part upon work supported by the National Science Foundation under Grant IIS-2212174, National Institute of Aging (NIA) R01AG072449, and National Institute of General Medical Sciences (NIGMS) R01GM145700. Fei Wang would also like to acknowledge the support from NSF 2212175, NIH RF1AG072449, RF1AG084178, R01AG080991, R01AG080624, R01AG076448, R01AG076234, OT2OD038083, R01AG072449, R01NS140142 and R01NS148533.

8 Broader Impact↩︎

Dysco aims to improve the reliability of federated adaptation for large language models in privacy-sensitive domains by reducing instability induced by heterogeneity. Promising directions include: (i) tighter privacy analyses for communicating subspace bases, (ii) principled subspace scheduling guided by interference diagnostics, and (iii) extending dynamic subspace control to other PEFT forms and to personalized or clustered FL settings. Overall, dynamic subspace control provides a practical and theoretically grounded tool for making LoRA-based FL more stable under real-world non-IID data.

9 Comparison between Dysco and FedAvg Under Data Heterogeneity↩︎

This appendix provides a direct optimization-side comparison between Dysco and FedAvg-LoRA. We adopt the aggregation-error framework of [46], embedding the data-parameter interference as an explicit term \(\|\bar E^{t+1}\|_F^2\) in a standard local-SGD convergence bound, and show that Dysco’s server-fixed per-client right factor structurally collapses the bilinear cross-client interference of FedAvg-LoRA to a linear one, yielding a \(\Theta(\Delta_A/\rho_A)\) tightening of the iterate’s optimization error across two convergence regimes, including per-round nonconvex stationarity and \(K\)-step local SGD.

9.1 Setup and Notation↩︎

We focus on a single LoRA-adapted layer of a fixed network; the statements apply per layer and per LoRA block and can be summed over layers. For each of \(N\) clients \(i\in\{1,\dots,N\}\), let \(f_i:\mathbb{R}^{d\times d}\to\mathbb{R}\) denote client \(i\)’s population objective and \(f\triangleq\tfrac1N\sum_i f_i\) the global objective. We write \(W_0\) for the pretrained initialization, \(\Delta W = W - W_0\) for the update, and \(W^\star\in\mathop{\mathrm{arg\,min}}_W f(W)\), \(f^\star=f(W^\star)\) for the (unconstrained) global minimum. The LoRA factors at client \(i\) and round \(t\) are denoted \((B_i^t,A_i^t)\in\mathbb{R}^{d\times r}\times\mathbb{R}^{r\times d}\), and the corresponding adapter contribution is \(B_i^t A_i^t\). In the theoretical analysis, we consider a more general aggregation form, written as \(\bar B^t\triangleq\tfrac1N\sum_i B_i^t\) and \(\bar A^t\triangleq\tfrac1N\sum_i A_i^t\). Throughout we assume \(f\) is \(\beta\)-smooth in the Frobenius norm:

9.2 Data-Parameter Interference in the Convergence Bound↩︎

We develop an optimization-side convergence analysis in four steps: (i) define the data-parameter interference as an explicit per-round quantity, (ii) prove Dysco’s per-client server-fixed right factor structurally reduces it, (iii) embed this term inside a nonconvex stationarity bound, and (iv) quantify the resulting tightening.

The data-parameter interference measures the gap between the model and algorithm actually applies to client data and the idealized aggregate that averages locally-computed rank-\(r\) updates. We insert this gap as an additive perturbation \(E^{t+1}=W^{t+1}-W^{t+1}_{\mathrm{ideal}}\) inside the standard nonconvex SGD descent inequality. Our contribution is the structural observation that Dysco collapses \(E^{t+1}\) from a bilinear cross-client interference (drift in both LoRA factors) into a linear one (drift in the left factor \(B\) only). We quantify the resulting tightening in two regimes: per-round nonconvex stationarity (1) and \(K\)-step local SGD (2).

9.2.1 Regime I: Per-round Nonconvex Stationarity↩︎

9.2.1.1 Additional optimization assumptions.

We add two mild assumptions that are standard in federated LoRA convergence analysis.

These mirror Assumptions 4.2 and 4.3 of the FedRot-LoRA analysis and are non-restrictive in fine-tuning regimes.

9.2.1.2 Defining the data-parameter interference.

At round \(t+1\), the per-client data-parameter interference is the gap between the algorithm’s actual update at client \(i\) and the ideal update obtained by averaging client-level rank-\(r\) updates rather than averaging factors: \[\label{eq:aggerr95def} E_i^{t+1,\mathrm{ALG}} \;\triangleq\; W_i^{t+1,\mathrm{ALG}} \;-\; W_i^{t+1,\mathrm{ideal,ALG}},\tag{6}\] where \(W_i^{t+1,\mathrm{ALG}}\) is the model the algorithm applies to client \(i\)’s data and \(W_i^{t+1,\mathrm{ideal,ALG}}\) is the corresponding ideal aggregate of local update matrices. Concretely:

FedAvg-LoRA. The aggregated global model at every client is \(W^{t+1,\mathrm{FA}}=W_0+\bar B^{t+1}\bar A^{t+1}\) with \(\bar B^{t+1}=\tfrac1N\sum_i B_i^{t+1}\), \(\bar A^{t+1}=\tfrac1N\sum_i A_i^{t+1}\). The ideal aggregate is \(W^{t+1,\mathrm{ideal,FA}}=W_0+\tfrac1N\sum_i B_i^{t+1}A_i^{t+1}\). The polarization identity \(\bar X\bar Y-\tfrac{1}{N}\sum_i X_iY_i=-\tfrac{1}{2N^2}\sum_{i,j}(X_i-X_j)(Y_i-Y_j)\) then yields \[\label{eq:E95fa} E^{t+1,\mathrm{FA}} \;=\; \bar B^{t+1}\bar A^{t+1}-\tfrac1N\sum_{i=1}^N B_i^{t+1}A_i^{t+1} \;=\; -\frac{1}{2N^2}\sum_{i,j=1}^N (B_i^{t+1}-B_j^{t+1})(A_i^{t+1}-A_j^{t+1}).\tag{7}\] This is the precise data-parameter interference that appears in 1 of 3.1: it is bilinear in the cross-client drifts of both LoRA factors.

Dysco. At round \(t+1\), Dysco fixes the right factor on a per-client basis to the server-merged \(A_i^{\mathrm{merge},t+1}\) (computed before local training and used identically in both local training and global application that round), so the model applied to client \(i\)’s data is \(W_i^{t+1,\mathrm{Dy}}=W_0+\bar B^{t+1}A_i^{\mathrm{merge},t+1}\) with \(\bar B^{t+1}=\tfrac1N\sum_j B_j^{t+1}\); the ideal update at client \(i\) uses its own locally trained \(B_i^{t+1}\) under the same \(A_i^{\mathrm{merge},t+1}\), giving \(W_i^{t+1,\mathrm{ideal,Dy}}=W_0+B_i^{t+1}A_i^{\mathrm{merge},t+1}\). (The multi-round boosting in 5 adds previously frozen blocks to this round’s contribution; the analysis below treats one boosting block per round, which is the granularity at which the interference is incurred.) Hence \[\label{eq:E95dy} E_i^{t+1,\mathrm{Dy}} \;=\; \big(\bar B^{t+1}-B_i^{t+1}\big)\,A_i^{\mathrm{merge},t+1}.\tag{8}\] The structural identity here is sharper than “smaller”: Dysco enforces \(A_i^{\mathrm{train}}\equiv A_i^{\mathrm{agg}}\equiv A_i^{\mathrm{merge},t+1}\), so the right-factor drift across clients is identically zero by construction (not just upper-bounded). The matrix-product inequality on the surviving residual then makes 8 linear in the cross-client drift of the left factor alone.

9.2.1.3 Cross-client drift quantities.

The interferences in 7 –8 are naturally controlled by the post-local-training client-drift of the LoRA factors: \[\label{eq:drift95def} \Delta_B^{t+1}\triangleq\frac{1}{N^2}\sum_{i,j=1}^N\|B_i^{t+1}-B_j^{t+1}\|_F^2, \qquad \Delta_A^{t+1}\triangleq\frac{1}{N^2}\sum_{i,j=1}^N\|A_i^{t+1}-A_j^{t+1}\|_F^2.\tag{9}\] \(\Delta_A^{t+1}\) and \(\Delta_B^{t+1}\) are data-derived, algorithm-agnostic quantities that measure cross-client divergence of the locally trained factors; they depend on the underlying data heterogeneity but not on which aggregation rule is applied.

9.2.1.4 Dysco’s structural reduction of the interference.

The following lemma is the central inequality of this section: Dysco’s interference is upper-bounded by a linear function of \(B\)-drift alone, while FedAvg-LoRA’s is bounded only by the bilinear product \(\Delta_A^{t+1}\Delta_B^{t+1}\).

9.2.1.5 Embedding the interference in a convergence bound.

With Dysco’s structural reduction (1) in hand, we now insert \(\|\bar E^{t+1}\|_F^2\) into a standard nonconvex stationarity bound; specializing the bound via 1 then yields algorithm-specific rates.

The bound 10 decomposes the stationarity rate into (i) the initial-gap term, (ii) the cumulative-interference term, and (iii) a learning-rate residual. Any algorithm that decreases \(\|\bar E^{t+1}\|_F^2\) obtains a strictly tighter stationarity bound at the same learning rate—which, by 1, is precisely what Dysco achieves.

9.2.1.6 Interpretation.

By fixing \(A_i^{\mathrm{merge}}\) both during local training and global application, Dysco eliminates the \(A\)-factor cross-client drift from the interference, making it linear in \(B\)-drift rather than bilinear in the product of \(B\)- and \(A\)-drifts; 1 translates this structural reduction into a \(\Theta(\Delta_A/\rho_A)\) tightening of the stationarity rate.

9.2.2 Regime II: Multi-step Local SGD with \(K\) Inner Steps↩︎

The per-round bound of 1 treats one local SGD step per round. Practical federated LoRA fine-tuning uses \(K\gg 1\) local steps to amortize communication; we extend the analysis to this regime and show that Dysco’s structural advantage strengthens with \(K\).

9.2.2.1 Multi-step drift accumulation.

Each client performs \(K\) inner SGD steps per round starting from the synchronized state \((\bar B^t,\bar A^t)\), so the trajectory is \(B_i^{t,k+1}=B_i^{t,k}-\eta\,\widehat\nabla_B f_i(W_i^{t,k},\xi_i^{t,k})\), \(k=0,\dots,K-1\), and similarly for \(A_i^{t,k}\). Under 2 3, applying the triangle inequality and the a.s.stochastic-gradient bound across the \(K\) inner steps gives \[\label{eq:Kdrift95bound} \big\|B_i^{t,K}-B_j^{t,K}\big\|_F \;\le\; 2\eta K\,G_B, \qquad \big\|A_i^{t,K}-A_j^{t,K}\big\|_F \;\le\; 2\eta K\,G_A,\tag{11}\] so the cross-client drifts at round-end satisfy \(\Delta_B^{t+1}\le 4\eta^2 K^2 G_B^2\) and \(\Delta_A^{t+1}\le 4\eta^2 K^2 G_A^2\).

9.2.2.2 Interpretation.

Increasing \(K\) widens Dysco’s relative advantage as \(\Theta(K^2)\), consistent with the empirical trend observed in 3 where Dysco’s accuracy gap over FedAvg compounds as deeper Transformers absorb more effective local update structure within each round.

9.3 Summary of the Comparison↩︎

Embedding the data-parameter interference \(\|\bar E^{t+1}\|_F^2\) as an explicit term in the convergence rate, Dysco’s server-fixed merge step collapses its bilinear \(\propto\Delta_A\Delta_B\) form to a linear \(\propto\rho_A\Delta_B\) form by eliminating the \(A\)-drift contribution. This yields a \(\Theta(\Delta_A/\rho_A)\) tightening of the iterate’s optimization error across both regimes—per-round nonconvex stationarity (1) and \(K\)-step local SGD (2). At the standard FL operating point \(\eta K G_A=\Theta(\sqrt r)\), this tightening reduces to \(\Theta(r)\), which is what Dysco’s server-fixed subspace allocation is engineered to exploit and what FedAvg-LoRA’s factor averaging cannot.

10 Additional Synthetic Experiments↩︎

This appendix collects additional details and ablations for the synthetic federation introduced in 4.1.

10.1 Synthetic-Experiment Setup Details↩︎

This appendix completes the description of the controlled federation introduced in 4.1.

10.1.0.1 Data generation.

The hidden dimension is \(d_{\mathrm{model}}=128\), the per-client target is \(d_{\mathrm{out}}=10\)-dimensional, and the sequence length is \(S=8\). For each client \(i\) we draw a column-orthonormal subspace \(U_i \in \mathbb{R}^{d_{\mathrm{model}} \times r}\) and sequence latents \(Z_i \in \mathbb{R}^{S \times r}\), so that the input tokens \(x = Z_i U_i^\top \in \mathbb{R}^{S \times d_{\mathrm{model}}}\) lie entirely in \(U_i\). Labels are produced as \(y = \overline{x}\,W_i^{\star\top} \in \mathbb{R}^{d_{\mathrm{out}}}\), where \(\overline{x}\) is the token-mean of the sequence \(x\) and \(W_i^\star = (1-\alpha)\,W_{\mathrm{shared}} + \alpha\,W_{\mathrm{perturb},i}\) with \(\alpha=1\) to induce the maximum cross-client heterogeneity. When \(N\!\cdot\!r \le d_{\mathrm{model}}\) the subspaces \(\{U_i\}\) form a mutually orthogonal partition of \(\mathbb{R}^{d_{\mathrm{model}}}\); when \(N\!\cdot\!r > d_{\mathrm{model}}\) each client receives an independent random orthonormal \(r\)-dim subspace, so inter-client orthogonality is no longer strict. Each client has \(100\) training and \(100\) test sequences.

10.1.0.2 Model architecture.

Local training uses a depth-\(L\) Transformer encoder with \(d_{\mathrm{model}}=128\), \(h=4\) attention heads, and a feed-forward block of width \(256\). LoRA adapters of rank \(r\) are attached to the selected attention projection(s) (any combination of \(Q\), \(K\), \(V\)) at every encoder layer. The trainable factors are the LoRA \(A_i,B_i\) matrices; all non-LoRA parameters of the Transformer (FFN, LayerNorm, attention output projection, prediction head) are frozen. For FedAvg, \(A_i\) is resampled i.i.d.from \(\mathcal{N}(0, 1/d_{\mathrm{model}})\) each round, matching the standard LoRA initialization used in our MIMIC experiments; for Dysco, \(A_i\) is set to \(U_i^\top\), so Dysco’s \(A_i\) is fixed across rounds while \(B_i\) is trained locally and merged on the server.

10.1.0.3 Optimization and evaluation.

Every configuration is trained for \(T=100\) communication rounds with \(K=10\) local SGD steps per round, learning rate \(0.01\), and is repeated for three random seeds (\(\{42,43,44\}\)). For each setting we report the merged global model’s average MSE loss \[F(W) = \tfrac{1}{2N}\sum_{i=1}^N \tfrac{1}{m_i}\sum_{x \in \mathcal{D}_i} \|W(x) - y\|_2^2,\] where \(W(x) \in \mathbb{R}^{d_{\mathrm{out}}}\) is the Transformer’s output for input sequence \(x\) and \(\mathcal{D}_i\) is client \(i\)’s training set, and the corresponding top-\(1\) test accuracy obtained by comparing \(\arg\max\) of \(W(x)\) to \(\arg\max\) of the target \(y\).

10.2 Block Ablation: Q-only and K-only↩︎

4.3 reports the two settings (\(V\)-only and \(Q\!+\!K\!+\!V\)) where attaching LoRA produces a substantial Dysco lift. For completeness, this appendix shows the other two settings in the same controlled federation: \(Q\)-only and \(K\)-only.

In both settings, neither method drives the training loss below \(0.67\), in sharp contrast to the \(V\)-only result (\(0.08\)). Dysco’s final test accuracy is \(0.15\) (Q-only) and \(0.14\) (K-only) compared to FedAvg’s \(0.11\) in both, leaving both methods within a small margin of the random-chance floor. This is consistent with the mechanism described in 4.3: \(Q\) and \(K\) only shape attention weights through a softmax that is largely insensitive to the LoRA basis, so a client-aligned subspace allocation has little leverage there. The data-parameter interference that Dysco neutralizes is dominated by the value projection \(V\), which writes into the residual stream.

10.3 Scaling with Number of Clients: \(N=64\)↩︎

4.4 reports the main-text settings \(N\in\{4,16,100\}\), which span the strict-orthogonal regime and its breakdown. For completeness, we report here the intermediate setting \(N=64\), where \(N\!\cdot\!r=256\) is twice the ambient dimension \(d=128\) and the random orthonormal fallback is fully engaged. Dysco’s final training loss is \(0.25\), compared with \(0.55\) for FedAvg, and its test accuracy reaches \(0.21\) while FedAvg remains at the random-chance floor near \(0.10\).

10.4 Effect of LoRA Rank: \(r=4\)↩︎

4.5 reports the main-text settings \(r\in\{8,16,32\}\), which expose how rank-driven over-parameterization amplifies FedAvg’s cross-client interference. For completeness, we report here the small-capacity setting \(r=4\), where the LoRA adapter dimension matches the per-client subspace rank used as the default in the depth, blocks, and clients ablations. Dysco’s final training loss is \(0.05\), compared with \(0.36\) for FedAvg (a roughly \(7\times\) ratio), and its test accuracy reaches \(0.55\) while FedAvg sits at \(0.20\).

11 Additional Real-World Experiments↩︎

This appendix collects additional results on the real-world benchmarks (GLUE and MIMIC-IV-note) introduced in 5.

11.1 GLUE Experiments↩︎

In this appendix we present the GLUE-benchmark counterpart to the MIMIC-IV results in the main body. The GLUE setting federates eight semantically diverse NLU tasks (sentiment, paraphrase, entailment, similarity, etc.) over a smaller backbone (RoBERTa-large, \(0.4\)B), which makes cross-task interference substantially more severe than in the MIMIC setting that shares a common note pool and a larger LLaMA-3.2-1B backbone. We retain the GLUE experiment here both as a stress-test of Dysco’s generality and as a controlled comparison with the same FL algorithms used in 5.

11.1.0.1 Dataset and model.

The GLUE benchmark [35] consists of eight datasets, including MRPC [47], QQP¹, QNLI [48], MNLI [49], SST-2 [50], CoLA [51], STS-B [52], and RTE [53]. The tasks involve different types of sentence classification or similarity judgments, so their label spaces and data distributions differ, but they all share some underlying knowledge of language. Following [20], we measure the Matthews correlation coefficient for CoLA and the average of the Pearson and Spearman correlation coefficients for STS-B; accuracy is measured for the remaining tasks. Each GLUE task is assigned to a different client in a federated setup. We use RoBERTa-large [36] as the backbone—a high-capacity encoder-only Transformer that builds on BERT [54]—to serve as a representative large encoder-only model.

11.1.0.2 Results.

4 reports the performance of the global models evaluated on the GLUE benchmark under different federated optimization methods. Across all five methods, Dysco improves the average GLUE accuracy, with absolute gains of \(+2.1\) (FedAvg), \(+11.4\) (FedAvgM), \(+0.6\) (FedProx), \(+1.2\) (Scaffold), and \(+8.3\) (FedNova) points. The largest lifts are concentrated on FedAvgM and FedNova; the smallest is on FedProx, whose average-level gain is partially offset by per-task regressions on STS-B and QNLI. At the per-task level, the picture is correspondingly varied: Dysco wins on \(7/8\) tasks under FedAvgM, \(5/8\) under FedAvg, \(4/8\) under Scaffold (with two ties), \(3/8\) under FedNova (with five ties), and \(2/8\) under FedProx, reflecting that an eight-task federation with semantically heterogeneous tasks induces more cross-client interference than the four-task MIMIC federation. The overall accuracy on GLUE is lower than on MIMIC because GLUE federates eight semantically diverse tasks (entailment, similarity, sentiment, etc.) over a smaller backbone (RoBERTa-large, 0.4B), whereas MIMIC’s four related clinical tasks share a common note pool and benefit from a larger model (LLaMA-3.2-1B). This more severe cross-task interference makes GLUE a substantially harder aggregation problem. Beyond the final accuracy reported in the table, 11 traces the average GLUE accuracy across communication rounds and shows that Dysco consistently delivers faster and better convergence than the baseline. The Dysco curve rises more steeply in the early rounds, reaches the baseline’s final accuracy within roughly half the communication budget, and continues to climb to a noticeably higher plateau by the end of training. This combination of accelerated convergence and improved final accuracy indicates that suppressing the data-parameter interference both shortens the number of rounds needed to reach a target accuracy and lifts the accuracy ceiling that factor averaging can attain.

11.2 Ablation Study↩︎

We ablate the two core components of Dysco—subspace merging and subspace boosting—by disabling each one individually while keeping the other active. Specifically, w/o Merging skips cross-client subspace aggregation so that each client initializes LoRA \(A\) from its own SVD-computed subspace, and w/o Boosting computes and aggregates subspaces only at round 0 and reuses them in all subsequent rounds without recomputation. All experiments use FedAvg as the FL algorithm and report the global model performance at round 10.

11.2.0.1 Discussion.

On both benchmarks, the full Dysco achieves the best average performance on GLUE (53.9%) and is competitive on MIMIC (82.5% vs.% for w/o Boosting), while the two ablation variants and FedAvg perform worse on average. Removing merging or boosting individually still improves over vanilla FedAvg on MIMIC, indicating that each component provides independent benefit. However, on GLUE the ablation variants underperform FedAvg in average accuracy despite excelling on individual tasks (e.g., SST-2, QNLI), because the missing component causes severe degradation on STS-B. This confirms that subspace merging and boosting are complementary: merging aligns cross-client subspaces to reduce interference, while boosting adaptively refines them across rounds to maintain representational quality.

12 Additional LoRA-Rank Experiments on Real Data↩︎

We further investigate the effect of LoRA ranks in the two real-world benchmarks.

12.1 Detailed Computational Overhead↩︎

11 provides the per-setting wall-clock time breakdown for the overhead experiment summarized in 2. All experiments use a single outer round to isolate per-round cost. Settings span the MIMIC-IV-note benchmark (4-task, Llama-3.2-1B) and the GLUE benchmark (2/4/6/8-task, RoBERTa-large).

References↩︎