1 Introduction↩︎

2 Theoretical Analysis↩︎

2.0.0.1 Connecting theory to empirical observations.

Observations 1 and 2 together imply that transformer layers differ systematically in required adaptation capacity , formalized in Theorem 1, which proves that uniform rank is suboptimal under inter-layer Fisher heterogeneity. Observation 3 further shows that this variation follows a structured depth-wise pattern, formalized in Theorem 2, which establishes that gradient norms as Fisher proxies induce a monotonic rank schedule across layers. Together, both theorems provide a principled justification for layer-adaptive rank allocation and directly motivate LAARA (Sections 2.2, 2.3, 7.5).

2.1 Setup and Notation↩︎

Consider a transformer with \(L\) layers indexed by \(\ell \in \{1,\dots,L\}\), where larger \(\ell\) denotes layers closer to the task loss. LoRA parameterized the update at layer \(\ell\) as \(\Delta W_\ell = B_\ell A_\ell\), where \(B_\ell \in \mathbb{R}^{d_{\text{out}} \times r_\ell}\), \(A_\ell \in \mathbb{R}^{r_\ell \times d_{\text{in}}}\), and \(r_\ell\) is the adapter rank. Standard LoRA fixes \(r_\ell = r\) for all \(\ell\); LAARA relaxes this constraint. Let \(\theta_\ell = \{A_\ell, B_\ell\}\) denote the trainable parameters at layer \(\ell\) and \(\mathcal{L}(\theta)\) the downstream loss. The layer-wise FIM is: \[\mathbf{F}_\ell = \mathbb{E}_{(x,y)\sim\mathcal{D}}\!\left[ \nabla_{\theta_\ell}\log p(y|x;\theta)\, \nabla_{\theta_\ell}\log p(y|x;\theta)^\top \right].\] To quantify the intrinsic dimensionality of \(\mathbf{F}_\ell\), we use the effective rank: \[\mathrm{erank}(\mathbf{F}_\ell) = \exp\!\left(-\sum_{i=1}^{p}\bar{\lambda}_i\log\bar{\lambda}_i\right), \quad \bar{\lambda}_i = \frac{\lambda_i}{\sum_j \lambda_j},\] which measures the Shannon entropy of the normalized eigenvalue distribution: a low value indicates a concentrated spectrum adaptable with a compact update; a high value demands larger adapter capacity. We denote by \(r_\ell^{*}\) the optimal rank of layer \(\ell\): the minimum rank sufficient to reduce approximation error below a prescribed tolerance \(\varepsilon > 0\) (Definition 1).

2.2 RQ1: Uniform Rank Allocation is Suboptimal↩︎

We first formalize the conditions under which uniform rank allocation provably fails. Two mild assumptions on the Fisher geometry of the pretrained model suffice.

2.2.1 Assumptions↩︎

Remark 1. Assumption 1 holds generically when the model is identifiable and the training distribution has full support over the input space — conditions satisfied in practice for large language models finetuned on sufficiently rich datasets [1], [2].

Remark 2. Assumption 2 is well-supported both theoretically and empirically. [3] showed via singular value analysis that importance scores of incremental weight matrices vary substantially across transformer layers. [4] demonstrate using Fisher information that different layers exhibit markedly different parameter sensitivities. [2] proved theoretically that the eigenvalue distribution of the FIM in deep networks is highly non-uniform across layers which is consistent with our empirical finding in Fig. [fig2], where the Fisher trace of Layer 11 exceeds that of Layer 0 by nearly two orders of magnitude.

2.2.2 Optimal Rank and Its Fisher Lower Bound↩︎

The following two lemmas connect the optimal rank to the Fisher geometry of each layer and establish that heterogeneous Fisher spectra necessarily imply heterogeneous optimal ranks.

Remark 1. Lemma 1 establishes that layers with higher effective Fisher rank require higher-rank LoRA adapters to achieve the same approximation tolerance, directly formalizing the empirical observation that later transformer layers which accumulate substantially larger Fisher traces (Figure [fig3]), cannot be adequately adapted with a uniform rank.

2.2.3 Main Theorem for RQ1↩︎

Under Assumptions 1 and 2, for any fixed rank \(r \in \mathbb{Z}_{>0}\) and any tolerance \(\varepsilon > 0\), the uniform rank assignment \(r_{\ell} = r\) for all \(\ell\) is suboptimal in the following sense: there exists at least one layer \(\ell^{+}\) for which the rank-\(r\) adapter fails to achieve tolerance \(\varepsilon\), we define this as follows: \[\min_{\boldsymbol{B}_{\ell^{+}},\boldsymbol{A}_{\ell^{+}}} \mathbb{E}\!\left[ \mathcal{L}(\boldsymbol{W}_{\ell^{+}} + \boldsymbol{B}_{\ell^{+}}\boldsymbol{A}_{\ell^{+}}) \right] - \mathcal{L}^{*} \;>\; \varepsilon,\] or at least one layer \(\ell^{-}\) for which rank \(r\) strictly exceeds the minimum rank needed to achieve tolerance \(\varepsilon\): \[r > r_{\ell^{-}}^{*}.\] Moreover, the total excess loss under uniform rank satisfies: \[\begin{align} \sum_{\ell=1}^{L} \left( \mathcal{L}(\boldsymbol{\phi}_{r}^{*,\ell}) - \mathcal{L}(\boldsymbol{\phi}^{*,\ell}) \right) \;\ge\;& \\ \frac{c(\varepsilon)}{2} \sum_{\ell=1}^{L} \max\!\left(0,\;\mathrm{erank}(\boldsymbol{F}_{\ell}) - r\right)^{2} \lambda_{\min}(\boldsymbol{F}_{\ell})\,\|\boldsymbol{\phi}^{*,\ell}\|^{2} \end{align}\] which is strictly positive whenever \(r < \max_{\ell}\, \mathrm{erank}(\boldsymbol{F}_{\ell})\).

Remark 1. Theorem 1 answers RQ1 definitively. The excess loss bound grows with the gap between the uniform rank \(r\) and the layer-wise effective Fisher rank layers that are under-ranked contribute a penalty proportional to \(\lambda_{\min}(\boldsymbol{F}_{\ell})\), the smallest eigenvalue of their Fisher matrix. Crucially, no fixed choice of \(r\) simultaneously minimizes this bound across all layers: setting \(r\) high enough to cover the most informative layers wastes capacity on less informative ones, while setting \(r\) low wastes the budget of informative layers.

2.3 RQ2: Gradient Norms Induce a Structured Rank Pattern Across Layers↩︎

We now show that the expected gradient norm at layer \(l\) is monotonically non-increasing as the layer index moves away from the loss, establishing that layer depth induces a structured pattern in the quantity that governs rank need.

2.3.1 Assumptions↩︎

2.3.2 Lemmas↩︎

2.3.3 Main Theorem for RQ2↩︎

Under Assumptions 3, 4, with \(\beta\gamma \le 1\), the expected gradient Frobenius norm satisfies \[\label{eq:gradient-chain} \mathbb{E}\!\left[ \|\nabla_{\boldsymbol{W}_{l}}\mathcal{L}\|_{F}^{2} \right] \;\le\; (\beta\gamma)^{2(L-l)}\, \mathbb{E}\!\left[ \|\nabla_{\boldsymbol{W}_{L}}\mathcal{L}\|_{F}^{2} \right] \quad \forall\, l \le L.\tag{1}\] Since the expected gradient Frobenius norm is a consistent estimator of \(\mathrm{tr}(\boldsymbol{F}_{l})\), the layer-wise Fisher trace satisfies: \[\label{eq:fisher-trace-chain} \mathrm{tr}(\boldsymbol{F}_{l}) \;\le\; (\beta\gamma)^{2(L-l)}\,\mathrm{tr}(\boldsymbol{F}_{L}) \quad \forall\, l \le L.\tag{2}\] Consequently, from lemma 1 the optimal rank \(r_{l}^{*}\) satisfies: \[\label{eq:rank-pattern} r_{l}^{*} \;\le\; r_{l+1}^{*} \quad \forall\, l \in \{1,\ldots,L-1\},\tag{3}\] i.e.the optimal rank is non-decreasing from the first layer to the last, forming a structured monotonic profile with depth. The complete proofs of the lemmas and theorems are provided in Appendix Section 7.5.

3 Proposed Method↩︎

Building on the empirical and theoretical analysis above, we present LAARA, which allocates layer-wise adapter capacity from Fisher information estimates computed during training across four components: dynamic LoRA parameterization, Fisher information estimation, adaptive rank allocation, and a vote-based dampening mechanism.

3.1 Dynamic LoRA Parameterization↩︎

LoRA [5] adapts a pretrained weight matrix \(W_0 \in \mathbb{R}^{m \times n}\) by introducing a low-rank update \(\Delta W = BA\), where \(B \in \mathbb{R}^{d_{\text{out}} \times r}\) and \(A \in \mathbb{R}^{r \times d_{\text{in}}}\) with \(r \ll \min(m,n)\). Unlike standard LoRA, which employs a fixed uniform rank across all layers, our method dynamically adjusts the rank during training. The adapted weight matrix is represented as: \[W = W_0 + \frac{\alpha}{r}\Delta W, \qquad \Delta W = BA,\] where \(\alpha\) is a scaling hyperparameter used to maintain stable output magnitudes across different rank values. For each targeted linear projection \(W \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}\), LoRA introduce trainable low-rank adapter matrices \(A \in \mathbb{R}^{r \times d_{\text{in}}}\) and \(B \in \mathbb{R}^{d_{\text{out}} \times r}\). The forward pass is defined as: \[h = Wx + \frac{\alpha}{r}\,BAx, \label{eq:lora95forward}\tag{4}\] where the pretrained weight matrix \(W\) remains frozen throughout training. Following the standard LoRA initialization strategy, \(A\) is initialized from a normal distribution while \(B\) is initialized to zero, ensuring that the adapter initially contributes no perturbation to the pretrained model.

3.1.0.1 Rank adaptation.

When layer \(\ell\) is updated from rank \(r\) to \(r'\), adapter matrices are resized in-place. For rank increases (\(r' > r\)), the additional \(r'-r\) rows of \(A\) are randomly initialised and corresponding columns of \(B\) set to zero, preserving learned directions while adding capacity. For rank decreases (\(r' < r\)), components are pruned by importance score: \[\phi_i = \|B[:,i]\|_2 \cdot \|A[i,:]\|_2, \quad i = 1,\ldots,r, \label{eq:importance}\tag{5}\] retaining only the top-\(r'\) components, consistent with prior low-rank pruning [3], [5]. To determine how capacity should be redistributed across layers, we estimate layer sensitivity via empirical Fisher information derived from adapter gradients.

3.2 Fisher Information Estimation↩︎

The diagonal Fisher information for a parameter \(\theta\) under loss \(\mathcal{L}\) is defined as: \[\mathcal{F}(\theta) = \mathbb{E}\!\left[ \left( \frac{\partial \mathcal{L}}{\partial \theta} \right)^2 \right]. \label{eq:fisher95diag}\tag{6}\]

We approximate \(\mathcal{F}\) using the empirical trace of squared gradients over a mini-batch: \[F_l^{(t)} = \sum_i \left(g_i^{(t)}\right)^2, \label{eq:fisher95trace}\tag{7}\] where \(g_i^{(t)}\) denotes the gradient of the loss with respect to the \(i^{\text{th}}\) LoRA parameter in layer \(l\) at optimization step \(t\). We maintain separate Fisher estimates for the \(A\) and \(B\) adapter matrices, denoted by \(F_{l,A}^{(t)}\) and \(F_{l,B}^{(t)}\), respectively. To reduce gradient variance and stabilize importance estimation, both quantities are tracked using an exponential moving average (EMA): \[\hat{F}_{l,\cdot}^{(t)} = \beta \hat{F}_{l,\cdot}^{(t-1)} + (1-\beta)F_{l,\cdot}^{(t)}, \label{eq:ema}\tag{8}\] where \(\beta \in (0,1)\) denotes the EMA decay factor. Bias correction is further applied to compensate for initialization effects during early optimization steps: \[\tilde{F}_{l,\cdot}^{(t)} = \frac{\hat{F}_{l,\cdot}^{(t)}}{1-\beta^t}. \label{eq:bias95correction}\tag{9}\]

The resulting Fisher estimates provide a measure of layer-wise adaptation sensitivity. However, raw Fisher values often exhibit substantial scale variations across projection types, making them unsuitable for direct rank assignment. We therefore introduce a normalized and compressed scoring mechanism to derive stable rank allocation signals.

3.3 Rank Allocation↩︎

3.3.0.1 Per-projection normalization.

Raw Fisher traces can differ significantly across projection types (e.g., query versus value projections), causing certain projections to dominate the allocation process. To mitigate this issue, we normalize Fisher scores independently within each projection type \(p \in \{\texttt{query\_proj}, \texttt{value\_proj}, \ldots\}\): \[s_{l,p} = \frac{ \tilde{F}_{l,p} - \min_{l'} \tilde{F}_{l',p} }{ \max_{l'} \tilde{F}_{l',p} - \min_{l'} \tilde{F}_{l',p} + \epsilon }, \label{eq:norm}\tag{10}\] yielding normalized per-projection scores \(s_{l,p} \in [0,1]\).

3.3.0.2 Log compression.

To reduce the influence of extreme outliers and encourage smoother rank distributions, we apply logarithmic compression: \[\hat{s}_{l,p} = \frac{ \log(1+\gamma s_{l,p}) }{ \log(1+\gamma) }, \label{eq:log95compress}\tag{11}\] where \(\gamma > 0\) controls the compression strength. This transformation preserves the relative ordering of layers while attenuating disproportionately large Fisher responses.

3.3.0.3 Blended signal.

To obtain a unified layer-wise importance estimate, we combine the compressed scores derived from the \(A\) and \(B\) adapters using weighted aggregation: \[\hat{s}_l = \alpha_b \hat{s}_{l,A} + (1-\alpha_b)\hat{s}_{l,B}, \label{eq:blend}\tag{12}\] where \(\alpha_b \in [0,1]\) controls the relative contribution of the two adapter components. This blended formulation captures complementary adaptation dynamics from both low-rank projection matrices, resulting in a more stable and representative importance signal for rank allocation. Setting \(\alpha_b = 1\) yields a signal derived solely from the \(A\) adapter, while \(\alpha_b = 0.5\) assigns equal importance to both matrices. Each projection type is normalized independently before blending to avoid scale inconsistencies.

3.3.0.4 Integer rank assignment.

The blended importance score is subsequently transformed into a discrete layer-wise LoRA rank using a bounded linear mapping: \[r_l = \operatorname{clip} \left( \operatorname{round} \left( r_{\min} + (r_{\max}-r_{\min})\hat{s}_l \right), r_{\min}, r_{\max} \right), \label{eq:rank95assign}\tag{13}\] where \(r_{\min}\) and \(r_{\max}\) denote the minimum and maximum allowable adapter ranks, respectively.

This formulation allocates larger ranks to Fisher-sensitive layers while compressing under-utilized layers, improving parameter efficiency without degrading downstream performance. Although the adaptive allocation strategy dynamically redistributes representational capacity during training, noisy gradient estimates may still induce unstable rank fluctuations. To address this issue, we introduce a vote-based dampening mechanism that stabilizes rank transitions over time.

3.4 Vote-to-Change Dampening↩︎

Frequent rank changes can destabilize optimization because each update partially disrupts optimizer momentum associated with the affected parameters. To suppress rapid oscillations, we introduce a vote-to-change mechanism in which a rank update for layer \(l\) is committed only when the same target rank is proposed consistently for \(\tau\) consecutive update steps: \[r_l^{\star} = \begin{cases} r_l, & \text{if the last } \tau \text{ proposals equal } r_l, \\ r_l^{\text{current}}, & \text{otherwise.} \end{cases} \label{eq:dampening}\tag{14}\]

Once a rank update is accepted, the corresponding vote buffer is reset, requiring the layer to accumulate another sequence of \(\tau\) consistent proposals before any subsequent modification can occur. This hysteresis-inspired mechanism suppresses cyclic rank oscillations caused by greedy updates and leads to more stable adaptive rank convergence. The algorithm of the LAARA framework is described in Algorithm 2.

4pt

6pt

4 Experiments↩︎

4.1 Experimental Setup↩︎

4.1.0.1 Model and datasets.

6pt

4.2 Results and Analysis↩︎

4.3 Parameter Efficiency↩︎

LAARA consistently reduces trainable parameters while improving performance across all settings. With \(W_q, W_k, W_v, W_o\) as target modules (Table 2), LAARA uses 987,042 parameters, a 4.6% fewer than LoRA/DyLoRA (1,034,498) and 21.4% fewer than AdaLoRA (1,256,114), while achieving the best results on RTE (88.09%) and STS-B (91.91%). At \(r{=}64\) (Table 4), the gap widens: LAARA uses 6,104,512 parameters, a 26.2% fewer than LoRA/DyLoRA (8,275,984) and 31.7% fewer than AdaLoRA (8,942,336), while outperforming all baselines on every task. On the generative reasoning setting (Table 3), LAARA uses 3,808,512 parameters, a 17.0% fewer than LoRA/DyLoRA (4,587,520) and 26.2% fewer than AdaLoRA (5,160,960), while achieving the highest accuracy (27.20%). These reductions follow directly from Fisher-guided allocation: layers with low Fisher trace are assigned \(r_{\min}\), reclaiming capacity that uniform-rank methods waste on less informative layers, with savings growing as the rank budget increases.

5 Conclusion↩︎

We presented LAARA, a parameter-efficient fine-tuning framework that allocates layer-wise ranks according to Fisher information estimates, without iterative search or auxiliary optimization. By concentrating adapter capacity in layers with high loss-landscape curvature and compressing it elsewhere, LAARA addresses the provable suboptimality of uniform rank allocation. A combination of EMA-smoothed Fisher estimation, projection-wise normalization, logarithmic compression, and vote-based dampening ensures that rank transitions are both principled and stable throughout training. Experimental results show that LAARA consistently matches or outperforms state of the art baselines at the same parameter budget, with particularly strong gains on reasoning-intensive tasks such as RTE.

6 Limitations↩︎

While LAARA consistently outperforms existing PEFT baselines, the diagonal Fisher approximation discards off-diagonal curvature and may underestimate true layer importance in some settings. Key hyperparameters including the rank update interval and Lora A/Lora B blend coefficient are fixed across tasks without extensive ablation. Future work will explore richer Fisher approximations, adaptive scheduling, and broader multi-architecture evaluations.

7 Appendix↩︎

7.1 Related Work↩︎

The increasing size of LLMs has made direct fine-tuning costly and often impractical, driving the development of a wide range of PEFT methods that enhance task-specific performance by introducing a small set of trainable parameters while keeping most pre-trained parameters fixed. Approaches like adapters [6], prefix-tuning [7], and prompt-tuning [8] exemplify this paradigm by integrating lightweight modules or augmenting input and hidden layers. Among PEFT approaches, LoRA [5] established a highly influential paradigm by decomposing weight updates into a pair of low-rank matrices while freezing the original pretrained weights. LoRA demonstrated that only a small fraction of trainable parameters is sufficient to achieve performance comparable to full fine-tuning across a variety of NLP tasks, while also avoiding additional inference latency through weight merging after training. This simplicity, efficiency, and strong empirical performance made LoRA the foundation for a rapidly growing family of PEFT methods.

Despite its effectiveness, LoRA employs a fixed rank across all transformer layers, assuming uniform adaptation requirements throughout the model. However, later studies showed that different layers exhibit heterogeneous adaptation behaviour, causing uniform rank allocation to either waste parameters or limit adaptation capacity in critical layers [9]. This limitation motivated a growing line of research on adaptive and dynamic rank allocation strategies for more efficient and effective PEFT. AdaLoRA [3] addressed this issue by reformulating LoRA updates through singular value decomposition and dynamically pruning singular values according to importance scores during training. DyLoRA [10] further relaxed the dependency on manual rank selection by jointly training nested adapters across multiple ranks, enabling inference-time rank selection without retraining. Subsequent works such as ARD-LoRA [11], DR-LoRA [12], and RA-LoRA [13] extended this direction by introducing layer-aware or training-aware rank allocation strategies, highlighting that the optimal adaptation capacity varies substantially across layers and tasks. In parallel, SoRA [14] incorporated sparsity-inducing gates into LoRA singular values to suppress redundant components, while GLoRA [15] generalized the low-rank parameterization itself to support broader adaptation structures. AutoPEFT [16] proposed an automatic configuration search framework for PEFT that jointly optimizes module type, insertion layers, and parameter budget using multi-objective Bayesian optimization. The method discovers Pareto-optimal and transferable PEFT configurations that outperform manually designed PEFT strategies while achieving performance comparable to full fine-tuning. However, AutoPEFT introduces additional search-time computational overhead and depends on a large configuration search space, which may limit efficiency and scalability for more complex large-scale tasks. L1RA [17] proposes a dynamic rank assignment strategy for LoRA fine-tuning that uses L1 regularization to prune redundant ranks and redistribute them across adapters under a fixed rank budget. The method improves parameter utilization and achieves comparable or better performance than standard LoRA while maintaining low computational overhead. However, the approach introduces additional optimization complexity for dynamic rank allocation and its effectiveness is primarily validated on limited benchmark settings, leaving large-scale generalization insufficiently explored.

Another line of research explored automated and data-driven mechanisms for determining adapter capacity. AutoLoRA [18] formulated rank selection as a meta-learning problem capable of generalizing across downstream tasks, although at the cost of expensive bi-level optimization. Dynamic Rank Assignment [17] instead employed online gradient-based heuristics to update ranks during training without relying on SVD decomposition. FlexoRA [19] introduced a continuous relaxation of rank allocation through reparameterization, while Bayesian LoRA Search [20] utilized Bayesian optimization to reduce the cost of exhaustive hyperparameter search. Collectively, these methods demonstrate that adaptive rank allocation is critical for improving the parameter-efficiency and robustness of LoRA-style fine-tuning, but many approaches still rely on iterative search procedures, auxiliary optimization objectives, or costly training-time adaptation.

Recent works have attempted to guide adaptation using activation statistics or information-aware criteria. AROMA [21] incrementally constructed adapters from rank-one components according to a relevance criterion, enabling autonomous low-rank matrix adaptation during training. Related studies have also investigated weight decomposition and compositional adaptation, such as DoRA [22], which separates magnitude and directional updates for improved adaptation stability, and LoraHub, which studies the composability of independently trained LoRA modules across tasks. These works collectively suggest that adaptation quality depends not only on parameter count, but also on how effectively the adapter subspace aligns with the underlying geometry of the pretrained model.

The Fisher Information Matrix (FIM) has long served as a principled tool for characterizing parameter importance and the geometry of neural network optimization. Natural gradient methods utilize the FIM as a Riemannian metric to perform optimization on the statistical manifold [23], while Elastic Weight Consolidation (EWC) [24] employs the diagonal Fisher to preserve important parameters during continual learning. Fisher-based pruning and compression methods have further shown that Fisher information provides reliable estimates of parameter sensitivity and contribution to model predictions. Despite its strong theoretical grounding, Fisher information has been only minimally explored in the context of PEFT and adaptive low-rank tuning. Existing LoRA variants largely rely on gradient magnitudes, activation statistics, sparsity regularization, or heuristic search strategies for rank determination. In contrast, our work directly utilizes Fisher information to estimate the adaptation sensitivity of each layer and allocate LoRA ranks accordingly, providing a statistically grounded and search-free framework for parameter-efficient fine-tuning. FLoE [25] leverages Fisher information to identify task-critical layers for sparse MoE-based adapter deployment and employs Bayesian optimization to automatically determine optimal layer-wise LoRA ranks without exhaustive hyperparameter search. Experimental results demonstrate that FLoE achieves strong efficiency–accuracy trade-offs, particularly in resource-constrained settings requiring rapid model adaptation. FiLoRA [26] introduces a Fisher information-guided PEFT framework that uses K-FAC-based curvature estimation to identify sensitive low-rank adaptation directions and optimize only a compact core matrix, thereby reducing trainable parameters while preserving performance. However, the method incurs additional computational overhead for Fisher matrix estimation and relies on static curvature information computed from limited downstream samples, which may restrict scalability and adaptability across diverse tasks.

7.2 Implementation Details↩︎

Following standard GLUE evaluation protocol, we report accuracy for MNLI (matched/mismatched), SST-2, QNLI, and RTE; Matthews Correlation Coefficient (MCC) for CoLA; F1-score for QQP and MRPC; and Pearson correlation for STS-B. All results are averaged over three runs with different random seeds. For all methods, the learning rate is selected from \(\{\)​4e-4, 5e-4, 8e-4, 1e-3, 1.2e-3, 2.2e-3\(\}\) and the best performing value per task is reported. LoRA and DyLoRA are configured with uniform rank \(r{=}8\), scaling \(\alpha{=}16\), batch size 64, and maximum sequence length 128, with the number of training epochs tuned per task; DyLoRA additionally performs rank search over \([2, 8]\) during training without retraining. AdaLoRA uses initial rank \(r_i{=}12\), target rank \(r_f{=}8\), scaling \(\alpha{=}16\), batch size 32, sequence length 128, with per-task training steps, warmup steps, rank update intervals, and regularization coefficient \(\gamma\) following [3]. LAARA uses initial rank \(r_{\text{init}}{=}4\), minimum rank \(r_{\min}{=}2\), maximum rank \(r_{\max}{=}8\), scaling \(\alpha{=}16\), EMA decay \(\beta{=}0.97\), warmup steps \(T_w{=}200\), rank update interval \(\Delta{=}200\), blend coefficient \(\alpha_b{=}0.5\), log compression \(\gamma{=}10\), and dampening patience \(\tau{=}2\), with adapters applied to \(W_q\) and \(W_v\) for the primary evaluation and additionally to \(W_k\) and \(W_o\) for the extended evaluation in Table 2. For Mathinstruct dataset, we have used max length=512, r=16,\(\alpha=32\), learning rate = 2e-4, with batch size of 4 and ran till 3 epochs. We have evaluated this on GSM8K dataset [27].

7.2.0.1 Baselines.

We compare LAARA against five baselines spanning a range of fine-tuning strategies: (i) Full fine-tuning (FT), which updates all model parameters and serves as an upper-bound reference; (ii) BitFit [28], a lightweight method that updates only bias terms; (iii) LoRA [5], the standard fixed uniform-rank baseline; (iv) AdaLoRA [3], which dynamically reallocates the parameter budget by pruning singular values during training; and (v) DyLoRA [10], which trains adapters simultaneously across a range of ranks.

7.3 Generalization to Mathematical Reasoning↩︎

Table 3 extends evaluation to MathInstruct dataset using Llama-3.2-3B, where LAARA achieves 27.20% accuracy, outperforming LoRA by +2.70% and AdaLoRA by +3.40% while using 17.0% fewer parameters than LoRA and 26.2% fewer than AdaLoRA. DyLoRA and AdaLoRA unperformed despite larger parameter budgets, consistent with our GLUE findings, confirming that Fisher-guided rank allocation generalizes beyond NLU classification to generative reasoning tasks.

7.4 Discussion↩︎

7.4.0.1 Why Fisher-guided allocation helps.

The consistent gains of LAARA over fixed-rank LoRA are explained by the three empirical observations. Fisher traces vary by nearly two orders of magnitude across layers (Observation 2), meaning that a uniform rank simultaneously over-allocates capacity to early layers and under-allocates it to late layers. By assigning ranks proportional to per-layer Fisher importance, LAARA concentrates its parameter budget where the loss landscape has the most curvature, precisely the layers that contribute most to downstream task adaptation.

7.4.0.2 Stability of rank allocation.

A practical concern with dynamic rank methods is training instability caused by frequent rank changes. LAARA addresses this through two mechanisms: the EMA-smoothed Fisher estimates (Eq. 8 ) reduce gradient noise, and the vote-to-change dampening (Eq. 14 ) prevents rank oscillation by requiring \(\tau = 2\) consecutive consistent proposals before committing a rank update. Together these ensure that rank transitions are both principled and stable, without introducing additional hyperparameters beyond those already present in standard LoRA.

7.5 Proof↩︎

Proof of Lemma  1:

Proof of Lemma 2:

Proof of theorem 1:

Proof of Lemma 3:

Proof of Lemma 5:

Proof of Theorem 2:

References↩︎