1 Introduction↩︎

Large Language Models (LLMs) have achieved remarkable performance across diverse reasoning and generation tasks [1], [2]. However, adapting these models to new domains or tasks remains computationally expensive, as full fine-tuning requires updating billions of parameters. This challenge has motivated the development of Parameter-Efficient Fine-Tuning (PEFT) methods [3], which aim to adapt large pretrained models while updating only a small subset of parameters, reducing memory and training costs.

Among PEFT approaches, Low-Rank Adaptation (LoRA) [4] has emerged as one of the most widely adopted. Motivated by the evidence that task-specific updates often lie in a low-dimensional subspace [5], [6], LoRA freezes the pretrained weights and model updates using two trainable low-rank matrices, substantially reducing the number of trainable parameters while maintaining strong performance. Recent works have explored structured decompositions to further improve stability. Specifically, spectral and singular value decomposition (SVD)-based methods [7]–[9] align trainable updates with the structure of pretrained weight matrices for more efficient tuning.

Existing low-rank adaptation methods often suffer from interference between overlapping update directions, especially when models are adapted across multiple domains or sequential tasks. This is particularly problematic in continual learning, where new knowledge must be acquired without erasing prior capabilities. Since the largest singular values of the pre-trained weight matrix encode its most critical structural representations, modifications to this subspace disproportionately degrade prior knowledge.

To mitigate this, we propose a spectral regularization scheme that selectively penalizes updates to the dominant singular components while allowing greater flexibility in the lower-rank “tail”. Our specific contributions are as following:

• We introduce TailLoR, a low-rank adaptation method operating over the singular values of a weight matrix, coupled with a soft regularization that steers updates toward the spectral "tail", protecting the top singular values.

• Different from existing continual PEFT methods [10], [11] TailLoR requires no access to adapters from prior tasks, crucially, enabling sequential adaptation by different users while preserving the privacy of each user’s task-specific parameters and training data.

• We evaluate TailLoR on a suite of continual learning tasks, demonstrating that it matches state-of-the-art methods while increasing the stable rank of the weight matrix.

2 Related Work↩︎

2.0.0.1 Spectral LoRA variants

Leveraging the spectral properties of base weights \(W\) is a key strategy in PEFT. Many SVD-based approaches partition the initial weights \(W_0=U \Sigma V^\top\) into frozen and trainable components. For instance, PiSSA [7] initializes the trainable adapter with the principal singular components to accelerate convergence, while freezing the residual. Conversely, MiLoRA [8] freezes the principal components and initializes the adapter with the minor components . Other approaches retain the original singular bases as a reference frame rather than statically partitioning them. SVFT [9] adapts weights via \(W=U(\Sigma+M)V^\top\), where \(M\) is a learned matrix with fixed sparsity patterns. Our mechanism is closely related, but instead of fixing a sparsity pattern, we learn a low-rank update \(M=AB\): \(W=U(\Sigma+AB)V^\top\). We soft-regularize the update with different spectral penalties that control which couplings between \(u_i\) and \(v_j\) are favoured, by penalizing coupling of dominant directions and guiding the model to route new task adaptations into the underutilized, long-tail spectral coordinates. While this emphasis is similar in spirit to MiLoRA and PiSSA, these methods primarily use the SVD partitioning to decompose the original weights and initialize the adapter, while our work keeps the updates parameterized with respect to the original SVD directions throughout training.

2.0.0.2 Spectral Approaches for Continual Learning

Continual Learning (CL) mitigates catastrophic forgetting [12] across sequential tasks using rehearsal [13]–[15], regularization [16], or architecture-expansion [17]–[19] strategies. Recently, Parameter-Efficient Fine-Tuning (PEFT) [3], particularly Low-Rank Adaptation (LoRA) [20], has emerged as a memory-efficient paradigm for CL, solving sequential tasks via low-parameter, task-specific adapters.

To prevent new adapters from interfering with prior knowledge, recent methods impose geometric or subspace constraints. O-LoRA [11] encourages orthogonality between successive task adapters, while ELLA [10] limits interference by restricting the specific sets of parameters that can be modified when learning a new task. Other approaches leverage spectral decompositions or activation geometries: InfLoRA [21] constructs and freeezes the matrix \(B_t\) such that the update subspace for the new adapter \(\Delta W_t=A_tB_t\) is aligned with the current task activations and also orthogonal to old-task gradients, while NESS [22] restricts future updates to the null space of prior activation covariances. Closest to our work is OSFT [23], a full fine-tuning method that first discovers critical prior knowledge directions corresponding to the largest singular values. During fine-tuning on new tasks, gradients are projected to the orthogonal complement of these important directions. While these methods rely on hard gradient projections or activation tracking, TailLoR achieves similar protection purely through soft spectral regularization within the efficient LoRA framework.

3 TailLoR↩︎

Given a pre-trained weight matrix \(W \in \mathbb{R}^{d_{out} \times d_{in}}\), we extract its structural geometry via Singular Value Decomposition (SVD): \(W = U \Sigma V^T\), where \(U \in \mathbb{R}^{d_{out} \times d_{out}}\) and \(V \in \mathbb{R}^{d_{in} \times d_{in}}\) are orthogonal matrices containing the left and right singular vectors, and \(\Sigma \in \mathbb{R}^{d_{out} \times d_{in}}\) is a rectangular diagonal matrix containing the singular values \(\sigma_i\). For notational simplicity, we assume the pre-trained weight matrices are square, \(W \in \mathbb{R}^{k \times k}\).

Rather than applying updates directly in the standard weight basis, we parameterize them entirely within this spectral basis. We introduce a low-rank adapter defined by the product of two matrices \(B \in \mathbb{R}^{d_{out} \times r}\) and \(A \in \mathbb{R}^{r \times d_{in}}\), where \(r \ll \min(d_{out}, d_{in})\). The updated weight matrix during fine-tuning is constructed as: \[W' = U (\Sigma + AB) V^T\]

Similarly with [8], [24], we hypothesize that the top singular vectors (the "head") encode the foundational representations shared across tasks, while the smaller singular vectors (the "tail") represent underutilized capacity. To regularize updates \(\Delta W\) during sequential fine-tuning, we construct a spatial penalty matrix \(\Omega \in R^{d_{out} \times d_{in}}\) that applies distinct gradient resistance to different singular subspaces.

3.1 The Head Penalty (Subspace Protection)↩︎

Let \(\mathbf{\sigma} \in R^k\) be the vector of non-zero singular values extracted from \(\Sigma\), sorted in descending order. We first normalize the singular values by their maximum component: \(\tilde{\mathbf{\sigma}} = \frac{\mathbf{\sigma}}{\sigma_{\max}}\) and define the raw head penalty matrix \(\mathbf{\Omega}\) such that the penalty applied to any element \((i, j)\) in the adapter update is governed by the maximum relative importance of its interacting singular components: \(\mathbf{\Omega}_{i,j} = \max(\tilde{\sigma}_i, \tilde{\sigma}_j)^\gamma\), where \(\gamma > 0\) is a hyperparameter controlling the severity of the penalty gradient. This formulation creates a highly concentrated regularization pressure on the top principal components, which decays as the indices move toward the spectral tail.

3.1.0.1 Normalization and Baselines

To validate the importance of protecting the spectral head, we compare our approach against two baselines: a tail penalty, which reverses the normalized singular values to intentionally penalize less significant components (isolating the effect of regularization geometry), and a uniform penalty, which applies a flat, unstructured regularization: (\(\mathbf{\Omega}_{\text{uniform}} = \mathbf{1}_{k \times k}\)).

To ensure a fair comparison, we introduce a mass normalization step. We scale the raw penalty matrix \(\mathbf{P}\) so that its total mass equals \(k^2\) (yielding an average element weight of \(1.0\)). The final normalized penalty matrix \(\mathbf{\tilde{\Omega}}\) is given by: \[\mathbf{\tilde{\Omega}} = \mathbf{\Omega} \frac{k^2}{\sum_{i,j} \mathbf{\Omega}_{i,j}}\]

During fine-tuning, the adapter matrices \(A\) and \(B\) are optimized with this structured regularization term added to the primary task loss. To compute the penalty, we take the square root of the \(\mathbf{\tilde{\Omega}}\)-weighted mean of the squared adapter updates: \[\mathcal{L}_{\text{reg}} = \lambda \sqrt{\frac{1}{k^2} \sum_{i=1}^k \sum_{j=1}^k \mathbf{\tilde{\Omega}}_{i,j} (A B)_{i,j}^2 + \epsilon}\] where \((A B)_{i,j}\) represents the individual elements of the adapter product (a modification over the original \(\Sigma\) diagonal matrix), and \(\epsilon = 10^{-12}\) is a small constant ensuring numerical stability for the gradient of the square root near zero.

4 Evaluation and Results↩︎

In this section, we present the evaluation setup and results on Continual Learning tasks, as well as ablations on the spectral penalty strategy and analyse the evolution of the effective rank of the weight matrix for TailLoR and ELLA.

4.1 Continual Learning↩︎

We evaluate TailLoR on three continual learning benchmarks: Standard CL [25], Long Sequence [26], and TRACE [27]. Across all setups, we fine-tune the query and value projections of a T5-large backbone for a single epoch per task with rank \(r=8\). For hyperparameters, TailLoR uses a single, globally searched penalty weight \(\lambda\) and exponent \(\gamma\) applied statically across all tasks. In contrast, following its original protocol, ELLA’s \(\lambda\) coefficient is optimized independently for every task. Despite this static tuning constraint, Table 1 shows TailLoR (head penalty) achieves highly competitive results on the Standard CL Benchmark alongside ELLA. By matching a state-of-the-art subspace partitioning method without requiring task-specific hyperparameter tuning, TailLoR demonstrates significant robustness and efficiency. In Table 2 we show results on the TRACE benchmark with 500 samples per task on a T5-large model, with TailLoR achieving the highest overall accuracy.

3pt

4.2 Spectral Penalty Strategy Ablation↩︎

The comparison between head, tail and uniform penalties in Table 1 confirms the intuition in prior works: principal singular directions encode important information and protecting them from excessive task-specific updates results in more efficient continual learning. Penalizing couplings between these high singular value directions encourages task-specific adaptation through the underutilized minor singular components, which improves accuracy and significantly mitigates catastrophic forgetting. Furthermore, the underperformance of the uniform baseline suggests that treating all spectral directions equally is suboptimal compared to structurally guiding adaptation toward the spectral tail.

4.3 Evolution of Effective Rank↩︎

To quantify the structural changes within the weight matrices during continual learning, we track the Roy-Vetterli effective rank [28]. Given the vector of singular values \(\mathbf{\sigma}\) sorted in descending order, the effective rank \(R_{\text{eff}}\) is defined as the sum of the singular values bounded by the spectral norm (the maximum singular value, \(\sigma_{\max}\)):

\[R_{\text{eff}} = \frac{\sum_{i=1}^k \sigma_i}{\sigma_{\max}} = \sum_{i=1}^k \tilde{\sigma}_i\]

A higher \(R_{\text{eff}}\) indicates a flatter spectral distribution, reflecting how task-specific updates successfully utilize the lower-rank tail capacity rather than overriding the dominant singular vectors.

Figure 1 illustrates the evolution of \(R_{\text{eff}}\) for each regularization strategy relative to its initial value. We observe that our head penalty induces a consistent, significant increase in effective rank as the model sequentially learns new tasks. This empirically validates our core hypothesis: structurally penalizing modifications to the dominant singular vectors forces the network to route new task representations into the underutilized lower-rank subspace. This accumulation of information flattens the overall spectral distribution, driving up the effective rank. In contrast, methods lacking this structural awareness fail to exploit latent capacity. The tail penalty confines updates entirely to the already-dominant head components, resulting in a perfectly stagnant effective rank, while the uniform penalty yields only a marginal, unoptimized spread. Notably, while ELLA successfully limits interference by controlling the magnitudes of individual parameters, it ignores the underlying geometry of the weight matrix. Consequently, it leaves the spectral capacity constrained, resulting in a flat \(R_{\text{eff}}\) evolution. Ultimately, these results confirm that protecting the spectral head not only preserves pre-trained knowledge but actively forces the model to exploit latent network capacity for CL.

5 Conclusions↩︎

In this work, we introduced TailLoR, a geometrically aware low-rank adaptation method designed to mitigate representation interference in continual learning. By selectively penalizing modifications to the dominant singular vectors, our regularization scheme successfully routes new task updates into the underutilized lower-rank tail. Empirical evaluations demonstrate that TailLoR not only preserves critical pre-trained knowledge but actively increases the effective capacity of the network, matching state-of-the-art baselines without requiring explicit weight partitioning or per-task parameter tuning.

Limitations↩︎

While this work demonstrates the efficacy of TailLoR on encoder-decoder architectures (T5) for continual learning, extending the spectral routing analysis to modern causal, decoder-only LLMs remains an active area of our ongoing work. Moreover, to maintain manageable computational costs, our evaluation on the TRACE benchmark utilized a 500-sample subset per task. While sufficient to demonstrate relative performance trends, future work will scale these evaluations to the full dataset.

6 Reproducibility details↩︎

We show in Table 3 the range of hyperparameters we search over for TaiLoR and the general hyperparameters for all baselines in Table 4. We ran all experiments on an Nvidia H200 GPU.

7 Head vs Tail penalty↩︎

We show the head (Figure 2) and tail (Figure 3) penalty matrices as well as the distribution of updates across the matrix position. Shown on Long Order 4 for the T5-large model.

8 Datasets and Task Orders↩︎

In Table 5, we list the original five datasets in the Standard CL benchmark (SC). Following [29], we only used the last four datasets.

In Table 6, we list the 15 datasets in the Long Sequence Benchmark (LS). We report all task orders used for our CL experiments in Table 7, which are adopted in previous works [10]. Orders 1-3 refer to the Standard CL Benchmark, while orders 4-6 refer to the Long Sequence Benchmark.

References↩︎