1 Introduction↩︎

Parameter-Efficient Fine-Tuning (PEFT) is widely used to adapt large language models (LLMs) [1]–[3] when full fine-tuning is computationally and operationally expensive. Among PEFT strategies, Low-Rank Adaptation (LoRA) [4] is a standard choice due to its simplicity, strong empirical performance, and deployment-friendly mergeability. LoRA is motivated by the observation that task-specific weight updates often exhibit low effective dimensionality, consistent with evidence on low intrinsic dimension in fine-tuning [5]. Concretely, LoRA freezes pretrained weights \(W_0\) and learns a rank-\(r_{\text{tot}}\) update \(\Delta W = \frac{\alpha}{r_{\text{tot}}}AB\), where \(A \in \mathbb{R}^{m \times r_{\text{tot}}}\) and \(B \in \mathbb{R}^{r_{\text{tot}} \times n}\), yielding the adapted weight \(W_0 + \Delta W\). Since the update is additive and linear, it can be merged into \(W_0\) after training, preserving the backbone architecture and inference efficiency [4].

Despite its success, standard LoRA assigns each adapted matrix a single low-rank update pathway. This raises a natural question: under a fixed rank budget, can one shared low-dimensional pathway adequately support the diverse behaviors required by modern LLMs? In the standard factorization, the adaptation signal depends on the input only through a single projection \(XA\), i.e., a single set of input-side directions shared by all adaptation effects. Consequently, heterogeneous behaviors may be forced to share and compete for the same input-side directions under a fixed rank budget, which can induce interference and optimization coupling. Increasing the rank can partially alleviate this issue, but it also increases trainable parameters and still provides no explicit mechanism for structured adaptation under a fixed budget.

In this work, we mitigate this coupling by structuring the low-rank update into multiple expert components without increasing the total rank budget. We propose Static Orthogonal Subspace LoRA (SOS-LoRA), a drop-in replacement for standard LoRA that reparameterizes a rank-\(r_{\text{tot}}\) update as a sum of \(K\) static (always-on, non-routed) low-rank experts. Each expert uses its own input-side directions, is assigned a fixed scale, and is explicitly encouraged to remain directionally distinct from other experts. Importantly, SOS-LoRA does not enlarge the hypothesis class beyond rank-\(r_{\text{tot}}\) updates; rather, it introduces an optimization-oriented inductive bias that promotes directionally diverse adaptation under the same low-rank budget.

SOS-LoRA instantiates this inductive bias through three components. First, a parallel expert decomposition splits the total rank budget into \(K\) experts of rank \(r' = r_{\text{tot}}/K\), matching the parameter budget of standard LoRA at the same \(r_{\text{tot}}\). Second, a fixed multi-scale scaling scheme assigns each expert a static scale spanning small-to-large magnitudes; this scales the raw backpropagated gradients entering each expert and encourages scale-separated learning dynamics. Third, orthogonal input-side diversification combines cross-expert orthogonal initialization (with zero initial update) and an orthogonality regularizer that penalizes cross-expert correlations in input-side directions.

Crucially, SOS-LoRA preserves LoRA’s key deployment property: after training, all expert updates can be merged into the pretrained weights as a single effective update matrix. Deployment therefore uses the same backbone architecture and per-token FLOPs as the base model, incurring no additional inference latency after merging [4].

Our contributions are threefold:

• We identify a key limitation of standard LoRA under a fixed rank budget: a single shared input-side projection can couple heterogeneous behaviors through a common low-dimensional update pathway, which can induce interference and optimization coupling.

• We propose SOS-LoRA, which reparameterizes a rank-\(r_{\text{tot}}\) LoRA update as a sum of \(K\) static low-rank experts with fixed multi-scale scaling and explicit cross-expert diversification of input-side directions, while preserving LoRA’s mergeability and inference efficiency.

• We empirically evaluate SOS-LoRA on reasoning and knowledge-intensive benchmarks, encoder-based NLU tasks (GLUE), and math reasoning (GSM8K/MATH), showing consistent improvements over matched-budget LoRA baselines and recent LoRA variants under a unified training and evaluation pipeline.

2 Related Work↩︎

Parameter-Efficient Fine-Tuning (PEFT) adapts large pretrained models by updating only a small subset of parameters [3]. Representative approaches include adding lightweight modules such as Adapters [6], prompt-based tuning that optimizes continuous prompts or prefixes [7], [8], and low-dimensional reparameterizations of weight updates. Among these, Low-Rank Adaptation (LoRA) [4] and its subsequent extensions [9], [10] freeze pretrained weights and learn efficient low-rank updates, making LoRA-based methods a widely used PEFT family.

A growing body of work extends LoRA along multiple axes. Optimization- and training-focused variants aim to improve convergence or update quality (e.g., LoRA+ [11], LoRA-GA [12], LoRA-Pro [13]), while orthogonality-based methods encourage less interfering low-rank updates (e.g., O-LoRA [14], primarily studied in continual learning). Other directions explore multiple LoRA experts through token-dependent routing (e.g., MixLoRA [15]), combine multiple trained LoRAs via learnable gating (e.g., MoLE [16]), or introduce multi-scale designs across layers (e.g., MSPLoRA [17]). Related ideas on enhancing specialization or expressiveness have also been explored in nonlinear LoRA designs and neuron-level mixture frameworks [18], [19]. In contrast, SOS-LoRA preserves LoRA’s mergeability and a fixed total rank budget, but reparameterizes the update within each adapted weight as a sum of \(K\) static (always-on, non-routed) low-rank experts. It further employs a fixed multi-scale scaling scheme and explicitly promotes cross-expert diversity of input-side directions via an orthogonality regularizer, while remaining expressively equivalent to standard LoRA under the same total rank.

3 Methodology↩︎

We propose Static Orthogonal Subspace LoRA (SOS-LoRA), a parameter-efficient fine-tuning (PEFT) method that aims to improve effective adaptation under a fixed low-rank budget. SOS-LoRA does so by: (i) decomposing a single rank-\(r_{\text{tot}}\) update into multiple static (always-on, non-routed) low-rank experts; (ii) using a pre-specified multi-scale scaling schedule (optionally with mild input-independent calibration) to promote stable, scale-separated optimization dynamics; and (iii) explicitly encouraging cross-expert diversity of input-side directions through initialization and a lightweight regularizer. Like standard LoRA [4], SOS-LoRA is mergeable into linear layers: after fine-tuning, the learned low-rank updates can be merged into the base weights, yielding zero additional inference-time parameters and latency after merging. Figure 1 provides an overview of the standard LoRA parameterization and the SOS-LoRA design.

3.1 Problem Formulation: Input-Side Coupling Under a Fixed Rank Budget↩︎

Let \(W_0 \in \mathbb{R}^{m \times n}\) be a pre-trained weight matrix in a Transformer linear layer. Let \(X \in \mathbb{R}^{N \times m}\) denote the input activations for \(N\) token instances (flattened across batch and sequence), and let \(Y \in \mathbb{R}^{N \times n}\) denote the corresponding outputs. Standard Low-Rank Adaptation (LoRA) [4] freezes \(W_0\) and parameterizes an additive update \(\Delta W\) with rank at most \(r_{\text{tot}}\): \[Y = X(W_0 + \Delta W) = X\!\left(W_0 + \frac{\alpha}{r_{\text{tot}}} A B\right), \label{eq:lora}\tag{1}\] where \(A \in \mathbb{R}^{m \times r_{\text{tot}}}\), \(B \in \mathbb{R}^{r_{\text{tot}} \times n}\), and \(\alpha\) is a scaling scalar.

3.1.0.1 Shared input-side pathway.

Eq. 1 implies that the LoRA path depends on the input only through a single low-dimensional projection. Writing \(Z = X A \in \mathbb{R}^{N \times r_{\text{tot}}}\), we have \(XAB = ZB\), hence the LoRA contribution depends on \(X\) only via \(Z\). In particular, for any \(X_1, X_2\) such that \(X_1A = X_2A\), the LoRA path yields identical outputs \(X_1AB = X_2AB\). Equivalently, for a single token row \(x \in \mathbb{R}^{1 \times m}\), the LoRA path depends on \(x\) only through \(xA\) (or \(A^\top x^\top\)). Under a fixed rank budget, this means that all input-sensitive adaptation effects are mediated through the same shared rank-\(r_{\text{tot}}\) input projection.

3.1.0.2 Optimization coupling under a fixed rank budget.

Increasing \(r_{\text{tot}}\) increases the maximum rank of \(\Delta W\) but also increases the number of trainable parameters. Under a fixed \(r_{\text{tot}}\), a single factorization \(AB\) can couple heterogeneous adaptation signals by forcing them to share and compete for the same input-side directions in \(A\). SOS-LoRA introduces a structured parameterization and explicit cross-expert direction regularization to bias optimization toward more diverse input directions, while preserving the same overall rank budget.

3.2 SOS-LoRA: Architecture Design↩︎

SOS-LoRA replaces a single rank-\(r_{\text{tot}}\) adapter with \(K\) parallel rank-\(r'\) experts, where \(r' = r_{\text{tot}}/K\) (assuming \(K\) divides \(r_{\text{tot}}\)). The design has three components: (i) parallel expert decomposition, (ii) multi-scale scaling, and (iii) input-side diversification via orthogonal initialization and regularization.

3.2.0.1 Expressivity equivalence (hypothesis class).

SOS-LoRA does not enlarge the set of representable rank-\(r_{\text{tot}}\) updates compared to standard LoRA; instead, it provides an optimization-oriented inductive bias through scale-separated optimization and cross-expert direction diversity. Formally:

3.2.1 Parallel Experts and Multi-Scale Scaling↩︎

We denote expert parameters by \(\{(A_k, B_k)\}_{k=1}^K\), where \(A_k \in \mathbb{R}^{m \times r'}\) and \(B_k \in \mathbb{R}^{r' \times n}\). The SOS-LoRA forward pass is \[Y_{\text{SOS}} = XW_0 + \sum_{k=1}^{K} s_k \, X A_k B_k, \label{eq:sos95forward}\tag{2}\] where \(s_k > 0\) controls the contribution of expert \(k\). Here, “static” means that (i) all experts are active for all tokens, (ii) there is no token-dependent routing, and (iii) the decomposition introduces no conditional computation.

3.2.1.1 Fixed multi-scale base scaling.

We initialize expert scales as \[\label{eq:multi95scale} \begin{align} s_k &= \frac{\alpha}{r_{\text{tot}}} \cdot \frac{\gamma_k}{\bar{\gamma}}, \\ \gamma_k &= 1 + \frac{k-1}{\max(1, K-1)}(\gamma_{\max}-1), \\ \bar{\gamma} &= \frac{1}{K}\sum_{k=1}^K \gamma_k, \end{align}\tag{3}\] so that the average scaling matches standard LoRA, \(\frac{1}{K}\sum_{k=1}^K s_k = \alpha/r_{\text{tot}}\), while experts receive linearly spaced relative scales \(\gamma_k \in [1,\gamma_{\max}]\). Unless stated otherwise, we use \(\gamma_{\max}=2.5\).

3.2.1.2 Scale-separated gradient signals.

Let \(\Delta = \partial \mathcal{L} / \partial Y_{\text{SOS}} \in \mathbb{R}^{N \times n}\) denote the upstream gradient for loss \(\mathcal{L}\). From Eq. 2 , the per-expert gradients satisfy \[\frac{\partial \mathcal{L}}{\partial B_k} = s_k \, (X A_k)^\top \Delta, \qquad \frac{\partial \mathcal{L}}{\partial A_k} = s_k \, X^\top (\Delta B_k^\top), \label{eq:grad95scale}\tag{4}\] so \(s_k\) linearly scales the raw backpropagated gradients entering expert \(k\). With adaptive optimizers, the resulting parameter updates are not strictly linear in \(s_k\) due to normalization effects; nevertheless, fixed multi-scale scaling provides a simple and stable mechanism to bias experts toward different update magnitudes.

3.2.1.3 Optional input-independent calibration (training-time only).

In practice, we optionally allow a mild input-independent reweighting across experts, initialized from the multi-scale scaling in Eq. 3 . This preserves the always-on structure (no routing), can be absorbed into the low-rank factors (hence remains mergeable), and is used purely as a stabilization mechanism during training. In our experiments, unless stated otherwise, we adopt a LoRA+ optimizer setting that uses different effective learning rates for the LoRA factors [11]; this is an optimizer-side choice and orthogonal to the SOS-LoRA parameterization.

3.2.2 Orthogonal Input-Side Diversification↩︎

To reduce redundancy and training-time interference among experts, we encourage diversity of input-side directions across \(\{A_k\}\) through initialization and a lightweight regularizer.

3.2.2.1 Cross-expert orthogonal initialization with zero initial update.

Assuming \(r_{\text{tot}} = K r' \le m\), we initialize experts to span diverse input-side directions while keeping the initial update exactly zero. Concretely, we sample \(G \in \mathbb{R}^{m \times (K r')}\) with i.i.d.Gaussian entries, compute a thin QR factorization \(G = QR\) with \(Q^\top Q = I_{K r'}\), and partition \(Q\) into \(K\) blocks \(Q = [Q_1,\ldots,Q_K]\) with \(Q_k \in \mathbb{R}^{m \times r'}\). We set \(A_k = Q_k\) and initialize \(B_k = 0\) for all \(k\). This yields \(\Delta W = \sum_{k=1}^K s_k A_k B_k = 0\) at initialization, preserving the backbone behavior at the start of fine-tuning.¹ Under this scheme, \(B_k\) receives gradients immediately (Eq. 4 ), while \(\partial \mathcal{L} / \partial A_k = 0\) at initialization since \(B_k = 0\). Thus, \(A_k\) starts updating only after \(B_k\) becomes non-zero, which stabilizes early training while starting from cross-expert-diverse input directions.

3.2.2.2 Cross-expert input-direction regularization.

We regularize the input-side directions by penalizing cross-expert correlations between the column spaces of \(\{A_k\}\). Let \(\tilde{A}_k \in \mathbb{R}^{m \times r'}\) denote the column-wise \(\ell_2\)-normalized version of \(A_k\), i.e., \(\tilde{a}=a/(\|a\|_2+\varepsilon)\) for each column \(a\) and a small \(\varepsilon>0\). We define \[\mathcal{L}_{\text{orth}} = \frac{\lambda}{r'^2} \sum_{1 \le k < \ell \le K} \left\| \tilde{A}_k^\top \tilde{A}_{\ell} \right\|_F^2, \label{eq:l95orth}\tag{5}\] which equals the sum of squared cosine similarities between all pairs of normalized columns across different experts. This directly discourages experts from collapsing to the same input-side directions while leaving within-expert structure unconstrained. In implementation, \(\lambda\) can be scheduled (e.g., delayed ramp-up), and Eq. 5 can be implemented either as an explicit loss term or via an equivalent gradient form without changing the underlying regularization objective.

4 Experiments↩︎

4.0.0.1 Models and Datasets.

We consider decoder-only LLMs (Llama 2-7B [20], Llama 3-8B [21]) and an encoder-only model (RoBERTa-base [22]). Our benchmarks cover three capability groups: (i) reasoning and knowledge-intensive QA: BoolQ [23], PIQA [24], SocialIQA [25], HellaSwag [26], WinoGrande [27], ARC-E/ARC-C [28], and OpenBookQA [29]; (ii) natural language understanding: GLUE [30]; and (iii) math reasoning: GSM8K [31] and MATH [32].

4.0.0.2 Compared Methods.

We compare SOS-LoRA with full fine-tuning (Full FT) as a reference and standard LoRA [4] as the primary PEFT baseline. We additionally include representative LoRA variants that appear in the main results tables: DoRA [33] (decomposing weights into magnitude and direction, with low-rank adaptation on the directional component), AdaLoRA [34] (adaptive rank allocation), DyLoRA [35] (supporting a range of effective ranks), MELoRA [36] (a mini-ensemble of low-rank adapters), PiSSA [37] (SVD-based initialization), Delta-LoRA [38] (propagating updates via deltas of successive low-rank products), MixLoRA [15] (a routing-based sparse mixture of LoRA experts inserted into feed-forward blocks), and MSP-LoRA [17] (a multi-scale pyramid LoRA with shared and layer-specific components). For the cost analysis in Table 3, we report LoRA and Full FT as references; additional PEFT baselines are included where noted.

4.0.0.3 Implementation Details.

All experiments are implemented in PyTorch with Hugging Face Transformers. Unless otherwise noted, SOS-LoRA is applied to all linear projections: for decoder-only LLMs (the Llama family), we adapt the attention projections (q,k,v,o) and MLP projections (up,gate,down); for RoBERTa-base, we adapt the self-attention and feed-forward linear layers. Unless otherwise specified, we use a fixed total rank \(r_{\text{tot}}=8\) per adapted matrix, decomposed into \(K=4\) static experts (\(r'=2\)), with LoRA dropout rate 0.05. We use fixed multi-scale scaling with \(\gamma_{\max}=2.5\) (Eq. 3 ) and orthogonal initialization with cross-expert regularizer weight \(\lambda=0.01\) (Eq. 5 ). We optimize with AdamW [39] and a linear learning-rate schedule with warmup ratio 0.1. Within each benchmark, we match the training data and training budget across methods. For RoBERTa-base on GLUE, we train for 3 epochs with learning rate \(2\times10^{-4}\) and batch size 32 (max sequence length 512). Unless otherwise stated, Llama experiments use 3 epochs with learning rate \(3\times10^{-5}\), per-device batch size 4 with 8 gradient accumulation steps (effective batch size 32), and max sequence length 2048. For analyses that vary the rank budget or expert count (Figs. 4 and 5), we sweep \(r_{\text{tot}}\) and/or \(K\) while keeping all other settings fixed. For the cost analysis in Table 3, we fine-tune Llama 2-7B on the first 100k MetaMathQA samples [40] using DeepSpeed [41] ZeRO-2 on 2\(\times\) NVIDIA A40 (48GB), and report peak reserved VRAM summed over the two devices together with end-to-end wall-clock time. The Full FT accuracies follow the protocol described in the table note. Unless stated otherwise, reported results are averaged over three independent runs with different random seeds; we report mean ± standard deviation where applicable.

4.2pt

We evaluate SOS-LoRA on eight reasoning and knowledge-intensive benchmarks. As shown in Table 1, SOS-LoRA achieves the highest average among the compared PEFT methods on both backbones. On Llama 2-7B, SOS-LoRA reaches 85.5 on average, improving over PiSSA (84.5) by 1.0 point and standard LoRA (77.6) by 7.9 points. SOS-LoRA also achieves the best result on all eight benchmarks in this setting, including PIQA (87.5 vs. for PiSSA), and attains the best overall average. We also evaluate MixLoRA, a routing-based sparse MoE baseline, under the same pipeline: while it remains competitive on several benchmarks, its overall average is below both standard LoRA and SOS-LoRA (77.1 vs. and 85.5, respectively). The trend persists on Llama 3-8B: SOS-LoRA attains an average of 88.0, surpassing PiSSA (87.3) by 0.7 and DoRA (85.2) by 2.8 points, while MixLoRA achieves 83.0. Overall, these results are consistent with SOS-LoRA’s inductive bias: under a fixed rank budget, decomposing the update into static experts with pre-specified multi-scale scaling and encouraging cross-expert input-side direction diversity provides more effective adaptation than a single low-rank update.

4.5pt

4.1 Generalization to Encoder Architectures and NLU Tasks↩︎

To assess generalization to encoder architectures and NLU tasks, we evaluate SOS-LoRA on GLUE with RoBERTa-base [22]. As shown in Table 2, SOS-LoRA achieves the highest average score (88.3) among the compared PEFT methods, improving over the strongest PEFT baselines under our evaluation pipeline (Delta-LoRA/MELoRA, 87.5) by 0.8 points, and also yielding a higher average than full fine-tuning in this setting (88.3 vs.). We additionally report MSP-LoRA [17] as a recent multi-scale LoRA baseline, which attains 86.6 on average in our evaluation. SOS-LoRA achieves the best result on seven of the eight tasks, while Full FT is best on QQP, with consistent gains on paraphrase identification (MRPC), semantic similarity regression (STS-B), and natural language inference (RTE/QNLI/MNLI). Overall, these results indicate that decomposing a fixed low-rank budget into diversified static experts remains effective in encoder-based NLU settings.

4.2 Cross-Expert Input-Side Direction Diversity Analysis↩︎

To assess whether SOS-LoRA learns diverse input-side directions, we analyze the \(A\) factors after training. For each adapted linear module (224 in total), we compute a \(K{\times}K\) cross-expert similarity matrix using the same squared-cosine measure as in Eq. 5 , namely \(\frac{1}{r'^2}\|\tilde{A}_k^\top \tilde{A}_\ell\|_F^2\), where \(\tilde{A}_k\) denotes column-wise \(\ell_2\) normalization. As shown in Fig. 2 (left vs.right), full SOS-LoRA exhibits lower off-diagonal similarity, whereas removing \(\mathcal{L}_{\text{orth}}\) yields substantially higher off-diagonal values. This pattern is consistent with \(\mathcal{L}_{\text{orth}}\) discouraging different experts from converging to redundant input-side directions and promoting better cross-expert separation.

4.3 Scale Separation Analysis↩︎

We test whether the pre-specified multi-scale expert scaling promotes scale-separated update magnitudes across experts. Using Llama 2-7B, we compare Full SOS-LoRA (\(\gamma_{\max}{=}2.5\)) with a Uniform control (\(\gamma_{\max}{=}1.0\)) and measure each expert’s relative update magnitude \(p_k=\|\Delta W_k\|_F \big/ \sum_{j=1}^{K}\|\Delta W_j\|_F\). As shown in Fig. 3, the left panel indicates that Full yields a more clearly separated contribution profile across experts, whereas Uniform is closer to a balanced allocation around \(1/K\). The right panel further shows that the top-1 share \(\max_k p_k\) is consistently larger under Full. These results are consistent with multi-scale scaling separating update magnitudes across experts under a fixed rank budget, which may in turn encourage more differentiated expert roles during training.

4.5pt

4.4 Optimization Dynamics↩︎

We examine training dynamics in Fig. 4 by fine-tuning Llama 2-7B on the first 100k MetaMathQA samples [40]. At both low (\(r_{\text{tot}}{=}32\)) and high (\(r_{\text{tot}}{=}128\)) rank budgets, SOS-LoRA shows faster early loss reduction and lower final loss than standard LoRA under the same training setup. The Uniform ablation reduces this advantage, suggesting that the pre-specified multi-scale scaling scheme contributes materially to the observed acceleration.

4.5 Ablation and Sensitivity Analysis↩︎

We analyze SOS-LoRA on Llama 2-7B to assess the contribution of each design component and sensitivity to key design choices and hyperparameters (Fig. 5). Under fixed rank budgets, the ablations in Fig. 5 (a–b) show that parallel expert decomposition provides the largest gain, while multi-scale scaling and orthogonal regularization yield consistent additional improvements, indicating complementary benefits beyond decomposition alone. Fig. 5 (c) further shows that SOS-LoRA improves over standard LoRA across a wide range of ranks (\(r_{\text{tot}}\in[1,128]\)) and matches or surpasses the Full FT baseline at higher ranks in our setup. Finally, Fig. 5 (d) suggests that performance peaks around \(K{=}4\) at fixed \(r_{\text{tot}}{=}128\): larger \(K\) reduces the per-expert rank and may limit per-expert capacity, whereas \(K{=}1\) removes cross-expert diversification, consistent with a diversification–capacity trade-off.

4.6 Computational Cost Analysis↩︎

We benchmark the cost–accuracy trade-off on Llama 2-7B using 2\(\times\) NVIDIA A40 (48GB) GPUs with DeepSpeed ZeRO-2. We fine-tune on the first 100k MetaMathQA samples [40] and evaluate on GSM8K and MATH. Table 3 shows that SOS-LoRA achieves higher accuracy than standard LoRA on both benchmarks (57.03% vs.% on GSM8K; 10.06% vs.% on MATH). Relative to LoRA, SOS-LoRA improves by 14.35 points on GSM8K and 4.14 points on MATH. This gain comes with a modest increase in training cost (4:52 vs.:31 wall-clock time; 67.45 vs. GB peak reserved VRAM), likely due primarily to the orthogonal regularizer. Full fine-tuning is not feasible under the same configuration because of OOM; its reported accuracies are obtained with a modified recipe and are therefore not directly comparable in cost to LoRA and SOS-LoRA (Table 3\(^\dagger\); see Appendix A.7). Overall, SOS-LoRA offers a favorable accuracy–cost trade-off under this protocol.

5 Conclusion↩︎

We identify shared input-side coupling as a limitation of low-rank adaptation under a fixed rank budget, where a single input-side pathway can couple heterogeneous adaptation signals and induce optimization interference. We propose Static Orthogonal Subspace LoRA (SOS-LoRA), which decomposes each low-rank update into \(K\) static (always-on, non-routed) experts with a fixed multi-scale scaling scheme and explicit input-side direction diversification via orthogonal initialization and a lightweight regularizer. SOS-LoRA is mergeable into the base model, adding no inference-time parameters or latency after merging. Experiments on both encoder-only (RoBERTa) and decoder-only (Llama 2/3) backbones show consistent gains over PEFT baselines under matched training budgets, suggesting that explicit input-side direction diversification can provide a practical route to stronger parameter-efficient adaptation.

Limitations↩︎

Our evaluation covers both encoder-only (RoBERTa-base) and decoder-only (Llama 2/3) backbones, but it is still limited to a relatively narrow range of model sizes and benchmark suites. Performance in larger models and broader settings (e.g., long-context, multilingual, and tool-use scenarios) remains untested. SOS-LoRA also introduces additional hyperparameters (e.g., expert count \(K\), multi-scale range \(\gamma_{\max}\), and regularization weight \(\lambda\)). Although we use a consistent default configuration and provide ablations and sensitivity analyses for key factors, the best settings may still vary with architecture, task, and training recipe. Finally, our analysis relies on geometric proxies (e.g., cross-expert squared-cosine similarity between input-side directions and relative update magnitudes) that capture redundancy and scale separation, but do not directly measure functional specialization. Complementary behavior-level analyses, such as targeted interventions or task-conditioned attribution/probing, would help strengthen these interpretations.

Acknowledgments↩︎

This work is supported by the National Key Research and Development Program of China (No.2023YFF0905400), the National Natural Science Foundation of China (No.U2341229) and the Reform Commission Foundation of Jilin Province (No.2024C003).

6 Appendix↩︎

6.1 Method Details of SOS-LoRA↩︎

SOS-LoRA replaces a single rank-\(r_{\text{tot}}\) LoRA update with \(K\) static (always-on, non-routed) low-rank experts of rank \(r' = r_{\text{tot}}/K\). For a linear layer with frozen weight \(W_0 \in \mathbb{R}^{m \times n}\) and input activations \(X \in \mathbb{R}^{N \times m}\), the SOS-LoRA forward pass is \[Y = XW_0 + \sum_{k=1}^{K} s_k \, X A_k B_k,\] where \(A_k \in \mathbb{R}^{m \times r'}\), \(B_k \in \mathbb{R}^{r' \times n}\), and \(s_k > 0\) are fixed expert scales set by the multi-scale schedule in Eq. 3 . The parameterization is mergeable: after training, the effective update is \(\Delta W = \sum_{k=1}^{K} s_k A_k B_k\), which can be added to \(W_0\) to obtain a standard dense weight for inference, incurring no additional inference-time parameters or latency after merging. We use cross-expert orthogonal initialization with \(B_k = 0\), so that \(\Delta W = 0\) at initialization while the \(A_k\) start from diverse input-side directions. The cross-expert diversity regularizer is the squared-cosine penalty in Eq. 5 .

6.2 Algorithmic Summary of SOS-LoRA↩︎

For clarity and reproducibility, we provide a high-level outline of SOS-LoRA for a single linear layer. The same procedure is applied to every adapted linear projection while keeping the pretrained weights frozen; all steps correspond to the definitions in Section 3. The procedure is summarized in Algorithm 6.

6.3 Training Setup and Hyperparameters↩︎

All experiments are implemented in PyTorch with Hugging Face Transformers and the PEFT library. Unless otherwise noted, SOS-LoRA is applied to all linear projections. For decoder-only LLMs (the Llama family), we adapt the attention projections (q,k,v,o) and MLP projections (up,gate,down); for RoBERTa-base, we adapt the self-attention and feed-forward linear layers. Unless otherwise specified, we use \(r_{\text{tot}}{=}8\) per adapted matrix with \(K{=}4\) experts (\(r'{=}2\)), LoRA dropout rate 0.05, \(\gamma_{\max}{=}2.5\) in Eq. 3 , and \(\lambda{=}0.01\) in Eq. 5 . We optimize with AdamW with a linear learning-rate schedule and warmup ratio 0.1. For RoBERTa-base on GLUE, we train for 3 epochs with learning rate \(2\times 10^{-4}\), batch size 32, and max sequence length 512. Unless otherwise stated, Llama experiments use 3 epochs with learning rate \(3\times 10^{-5}\), per-device batch size 4 with 8 gradient accumulation steps (effective batch size 32), and max sequence length 2048. Unless stated otherwise, reported results are averaged over three independent runs with different random seeds; we report mean \(\pm\) standard deviation where applicable. For sweeps over \(r_{\text{tot}}\) and/or \(K\), we keep all other settings fixed to isolate the effect of varying rank capacity and expert granularity.

6.4 Datasets, Splits, and Metrics↩︎

We use public benchmarks with their official splits. Reasoning and knowledge-intensive evaluation includes BoolQ, PIQA, SocialIQA, HellaSwag, WinoGrande, ARC-E/ARC-C, and OpenBookQA, evaluated under a unified prompting and scoring pipeline with accuracy as the metric. NLU evaluation uses GLUE with task-specific metrics following the benchmark conventions [30]: CoLA uses Matthews correlation, STS-B uses correlation, SST-2/QNLI/RTE use accuracy, and MRPC/QQP are conventionally reported with both F1 and accuracy. In Table 2, we report accuracy for MRPC and QQP (while retaining F1 in our released logs/configs), report MNLI matched accuracy (MNLI-m), and report the simple average over the eight tasks shown (excluding WNLI). Math reasoning evaluation uses GSM8K and MATH: GSM8K is evaluated by exact-match accuracy on the final answer, and MATH is evaluated by final-answer accuracy under the same post-processing rules across methods [31], [32].

6.5 Baselines and Configuration Matching↩︎

We compare SOS-LoRA against Full FT as a reference and LoRA as the primary PEFT baseline. We additionally include representative LoRA variants reported in the main results tables: DoRA, AdaLoRA, DyLoRA, MELoRA, PiSSA, Delta-LoRA, MixLoRA, and MSP-LoRA. For fair comparison, we match (i) backbone checkpoints, (ii) training data and the number of steps/epochs, (iii) the set of adapted modules, unless a baseline by design constrains insertion locations, and (iv) the total low-rank budget per adapted matrix whenever a direct rank match is supported. When a baseline introduces additional components (e.g., routing or extra parameters), we run it under the same training pipeline and overall optimization budget, while following the authors’ recommended defaults for baseline-specific hyperparameters. All configurations are recorded in our released files.

6.6 Mechanistic Analyses: Definitions and Computation↩︎

Cross-expert input-side direction similarity. For each adapted module, we compute a \(K\times K\) matrix with entries \(\frac{1}{r'^2}\|\tilde{A}_k^\top \tilde{A}_\ell\|_F^2\), where \(\tilde{A}_k\) is obtained by \(\ell_2\)-normalizing each column of \(A_k\). This quantity equals the mean squared cosine similarity over all cross-expert column pairs and matches the quantity used in the regularizer in Eq. 5 . We report both module-wise matrices and their average across all adapted modules (224 for Llama 2-7B under our adaptation scope), and visualize them as heatmaps (Fig. 2).

Per-expert relative update magnitude. For each adapted module, we compute \(\Delta W_k = s_k A_kB_k\) and define \(p_k=\|\Delta W_k\|_F\big/\sum_{j=1}^{K}\|\Delta W_j\|_F\). We plot the distribution of \(p_k\) across modules together with the distribution of \(\max_k p_k\) to characterize scale separation (Fig. 3).

Loss dynamics. For the training curves in Fig. 4, we fine-tune Llama 2-7B on the first 100k MetaMathQA samples and log training loss at fixed intervals under identical optimization settings across methods, using the same token budget and sequence length. Curves show the mean across seeds.

6.7 Efficiency Measurement Protocol↩︎

For the cost–accuracy study (Table 3), we fine-tune Llama 2-7B on the first 100k MetaMathQA samples using 2\(\times\) NVIDIA A40 (48GB) GPUs with DeepSpeed ZeRO-2. Peak VRAM is measured as the maximum reserved memory (e.g., torch.cuda.max_memory_reserved) summed over the two devices during training. Wall-clock time is measured end-to-end under the same training budget and logging frequency across compared methods. Full FT is marked with \(^\dagger\) because it requires a modified recipe to avoid OOM; we therefore report its accuracies only as a reference, and its cost is not directly comparable under the same configuration.

6.8 ChatGPT Reference Evaluation↩︎

We include ChatGPT in Table 1 only as a reference. We evaluate it under the same prompting and scoring format used in our unified evaluation pipeline, with fixed prompt templates and deterministic decoding when supported. The exact prompts, scripts, and post-processing rules are included in our release to enable reproduction of the evaluation procedure, subject to model/version changes by the provider. We do not use ChatGPT outputs for training, validation, or hyperparameter selection.

6.9 Reproducibility Statement↩︎

We provide the public code repository, training/evaluation scripts, and SOS-LoRA adapter checkpoints under an Apache 2.0 license. Our implementation is based on PyTorch, Hugging Face Transformers, and PEFT; experiments additionally use DeepSpeed for the ZeRO-2 efficiency study. We use publicly available pretrained checkpoints (e.g., Llama 2/3 and RoBERTa-base) and standard benchmarks (e.g., GLUE, GSM8K, MATH) with official splits. Section 4 and Appendix 6.3 report key hyperparameters, including adaptation scope, ranks, \(K\), scaling, regularization, learning rates, batch sizes, sequence lengths, epochs, and optimizer schedule. Unless stated otherwise, all reported numbers are mean \(\pm\) standard deviation over three independent runs with different random seeds.

6.10 LLM Usage Statement↩︎

We used a large language model (LLM) only for language polishing and proofreading of the manuscript text. The LLM was not used to design SOS-LoRA, develop algorithms, generate experimental results, or make methodological or scientific decisions. All technical contributions, experiments, and conclusions are our own, and we take full responsibility for the content of this paper.

6.11 Responsible NLP Research Checklist↩︎

We follow the ACL community’s Responsible NLP Research guidelines and checklist and include the completed checklist in the final version, as required by the venue. Our experiments are conducted on widely used public benchmarks and standard model checkpoints; we report averaged results over multiple seeds and provide implementation details to support reproducibility. Although our work focuses on parameter-efficient adaptation rather than new data collection, downstream deployment of fine-tuned models should still consider dataset biases, evaluation limitations, and potential misuse, consistent with the checklist guidance.

References↩︎

---

1. If \(r_{\text{tot}} > m\), one can fall back to per-expert orthogonalization (orthogonalizing each \(A_k\) independently), which preserves within-expert orthogonality but cannot guarantee global cross-expert orthogonality.↩︎