1 Introduction↩︎

The myriad uses for foundation models (FMs) have led to a proliferation of specialized models, each fine-tuned to perform a downstream task. To avoid fine-tuning foundation models with billions of parameters, parameter-efficient fine-tuning (PEFT) algorithms were proposed. An especially successful PEFT method is low-rank adaptation (LoRA) [1], which learns low-rank additive changes to neural network matrices. Because of the low-rank parameterization, these matrices (called adapter weights) contain orders of magnitude fewer parameters than the base model. Still, LoRA can achieve performance on par with full fine-tuning [1].

LoRA’s popularity has triggered a growing need to serve large collections of LoRA adapters at scale. Proprietary and open-source LLM providers offer fine-tuning services [2]–[4] with user bases likely in the thousands or even hundreds of thousands. As each user wants to use their own fine-tuned version of the LLM, serving a dedicated fine-tuned LLM per user becomes infeasible. To this end, S-LoRA [5] proposes a system where only the base LLM is placed on an inference server and individual LoRA adapters are switched as needed at inference time. S-LoRA optimizes the system’s inner workings via custom CUDA kernels and memory management to increase throughput when serving multiple LoRAs. Multi-LoRA system design has also been adopted in vLLM [6], a state-of-the-art LLM serving engine. Despite optimized system designs, serving LoRAs still has a fundamental limitation: when the number of adapters is large, they need to be constantly loaded and offloaded from GPU memory to accommodate incoming requests, degrading throughput.

The problem of accommodating multiple LoRA adapters is also apparent when placing LLMs on edge devices, where smaller LLMs are fine-tuned for various tasks, and the adapters are swapped depending on the task at hand [7]. In this setting, the number of adapters is smaller, e.g., a few dozen [7], but the memory constraints are also more stringent.

In this work, we consider the problem of compressing a collection of LoRAs. We have two key objectives: (1) preserving the performance of the original LoRAs and (2) improving the throughput of serving many LoRAs. We formulate LoRA compression as a reconstruction problem, where the goal is to approximate the original adapters via collections of matrices of a smaller size. We compress LoRAs jointly by finding a shared basis and LoRA-specific scaling matrices and propose a joint diagonalization-based algorithm (JD). To improve reconstruction error for large numbers of LoRAs while keeping the number of parameters in check, we propose a clustering approach where each cluster is compressed independently using the joint diagonalization algorithm. Our clustering algorithm is based on alternating between optimizing the cluster assignments and the per-cluster reconstruction error.

Figure 1 showcases the benefits of joint compression. When serving up to 64 unique LoRAs, we use JD without clustering and for 128 or more, we pick the number of clusters to match the performance of compressed and original LoRAs. In each case, the GPU memory footprint of the compressed and original LoRAs is matched for a fair comparison to vLLM’s multi-LoRA inference engine. When serving over 1000 LoRAs, compression increases throughput \(1.6\!\times\) and maintains 80% of the throughput of serving the base LLM (or a single LoRA merged into the LLM). §6 presents detailed results.

We summarize our main contributions below:

• We formulate the problem of compressing a collection of LoRAs and propose a joint compression scheme based on joint diagonalization.

• For large numbers of LoRAs, we scale joint compression by proposing a clustering algorithm where each cluster is jointly compressed to minimize reconstruction error.

• We establish theoretical guarantees for the reconstruction error of our compression formulation and relate reconstruction loss to performance empirically.

• We train a collection of more than 1000 high-quality LoRAs for Mistral-7B-Instruct-v0.2 [8] on 1000 natural instruction tasks [9] and demonstrate that our compression techniques preserve the performance of the original LoRAs. We will release over a 1000 LoRAs to facilitate future work as well as the code for our method.

• We incorporate LoRA compression into a state-of-the-art LLM serving system and demonstrate that it is possible to serve over 1000 LoRAs across thousands of asynchronous requests with throughput comparable to serving a single LoRA.

2 Related Work↩︎

Parameter-efficient fine-tuning (PEFT) has become prevalent for updating foundation models thanks to the need for efficiency in training and communication [10]. Many PEFT methods have been proposed, e.g.[11], [12] and LoRA [1] became the standard, partially due to the ease of switching between LoRAs in inference time.

Several works improve LoRA [13], [14], sometimes with algebraic methods like SVD [15]–[17] or by leveraging its statistical properties [18], [19]. Relatively few, however, accelerate inference times. S-LoRA [5] provides an efficient means of switching between LoRAs. [20] adapt training to reduce batch multiplications, accelerating inference. Our method achieves a similar outcome (see Appendix 11) without changing the LoRA formulation or requiring that LoRAs be trained in a dedicated way; future improvements to LoRA will also benefit from this aspect of our work (e.g., [15]).

Punica [21] introduces Segmented Gather Matrix-Vector Multiplication (SGMV) to optimize multi-LoRA serving by parallelizing feature-weight multiplications in batches and grouping requests that use the same LoRA. Our approach, by contrast, reduces parameters as a means to serve multiple LoRAs efficiently, providing an orthogonal strategy that can be seamlessly integrated with Punica’s methods to enhance performance. In our vLLM experiments, we leveraged the Punica kernel for multi-LoRA implementation, demonstrating the application of our method in conjunction with Punica’s optimizations.

Other research proposes alternative PEFT methods that can be more parameter-efficient than LoRA. For example, VeRA [22] fine-tunes LLMs by sharing global static parameters while learning local scaling variables; (IA)³ [23] also reduces adapter parameter counts. However, none of these approaches has been as extensively tested or widely-adopted as LoRA. As a result, work that builds on LoRA enjoys a practical advantage due to its broad acceptance in practice.

There are many efforts to compress models [24]–[27]. Predominantly, pruning and sparsification methods delete weights [28], and quantization methods reduce the weights’ precision [29]. Some works compress weights to reduce model size but typically require decompression and hence do not save GPU memory [30]. Similarly to our work, a few note increased performance and generalization after compression [26], [28], [30], [31].

Our work also relates to model merging [32]–[34] and mixtures of experts [35], [36]. These methods reuse models trained by others [37], [38], serving them together as one compressed model. Despite this similarity, these methods create a single general model that acts on any input, while ours yields more performant per-task solutions.

3 Rank-Based LoRA Compression↩︎

LoRA updates are parameterized by pairs of matrices \(A, B\), whose product \(BA\) updates the fixed weight matrices \(W_0\in\mathbb{R}^{d_B\times d_A}\) of a neural network foundation model. Given an input \(x\) to a layer, the output of the LoRA-updated model at this layer is \((W_0 + BA)x\).

In formulating our compression algorithms, we consider a collection of given LoRA adapters \(\{(A_i, B_i)\}_{i=1}^n\) that we would like to serve. We let \(r_i\) refer to the rank of the LoRA adapter-pair \((A_i, B_i)\), i.e., \(B_i \in \mathbb{R}^{d_B \times r_i}\), \(A_i \in \mathbb{R}^{r_i \times d_A}\).

While our compression technique has access only to a collection of \(\{(A_i, B_i)\}_{i=1}^n\) pairs, in our experiments we will assess the efficacy of compression by comparing how the compressed matrices perform relative to uncompressed LoRAs on typical data. For this reason, although in this section we optimize a Frobenius norm reconstruction error relative to the product \(B_iA_i\), this is a proxy for the nonlinear and complex way that compression errors in the adapters impact transformer performance. Our experiments will thus focus on the performance of the compressed LoRAs against the uncompressed versions on real data in §6.

Our compression methods significantly reduce the overall number of parameters. Reducing parameters theoretically accelerates storage and serving of a collection of LoRAs. This reduction, however, alters the computational dynamics during inference, so parameter reduction alone does not immediately imply faster throughput. In light of the complexities of GPU optimization, we experimentally assess throughput under realistic conditions in §6.4.

3.1 Joint Diagonalization↩︎

To scale to many LoRAs, the compressed number of parameters should not scale linearly with \(n\). Hence compressing each LoRA individually (e.g., via SVD as in our experimental baselines) is inherently limited. To address this, we suggest a Joint Diagonalization (JD) method, which optimizes a shared basis onto which we can project the set of \(n\) LoRAs. This allows structure to be shared, implicitly grouping and/or merging the collection of LoRAs.

In this model, each LoRA product \(B_iA_i\) is factorized into the form \(U \Sigma_i V\), where \(U\) and \(V\) are shared across all LoRAs and \(\Sigma_i\) is specific to each LoRA. In this formulation, every \(\Sigma_i\) shares the same rank \(r\). This allows \(U\) and \(V\) to be pre-loaded onto the GPU, with \(\Sigma_i\) loaded when necessary for each batch. The matrices \(\Sigma_i\) can be either diagonal or small square matrices, thus significantly reducing the number of LoRA-specific parameters and accelerating multi-LoRA serving.

Objective function. Motivated by the relationship of singular value decomposition to minimizing the Frobenius norm of the reconstruction error, we also propose to minimize the Frobenius norm of the adapter matrix approximation error. Specifically, we use the following objective: \[\label{eq:jointsvd} \min_{\{\Sigma_i\}_{i=1}^n, U,V} \sum_{i=1}^n \|B_iA_i - U\Sigma_i V^\top\|_\mathrm{Fro}^2.\tag{1}\] Note this problem is not solved by a single matrix SVD, since \(U\) and \(V\) are shared among all terms but the \(\Sigma_i\)’s are not. Using the Frobenius norm has the added benefit of making the objective convex in each argument separately, suggesting the possibility of efficient optimization. This objective function is underdetermined, however, so we consider two constrained regimes below.

Full \(\Sigma_i\) approximation. The first method we call JD-Full. Without loss of generality, \(U\) and \(V\) can be constrained to be orthogonal, so long as \(\Sigma_i\) remains an unconstrained full matrix. JD-Full adopts this restriction to make the optimization better posed, but note it does not restrict the expressiveness of the objective equation 1 . This setting yields the following optimization problem:

An efficient alternating algorithm to optimize this objective function can be found in Appendix 8.

Diagonal \(\Sigma_i\) approximation. As an alternative, we can leave \(U\), \(V\) unconstrained (other than to have \(r\) columns) and instead constrain the matrices \(\Sigma_i\) to be diagonal (but not necessarily positive). This formulation yields the following optimization problem:

Appendix 8 provides an efficient alternating least squares algorithm for this objective. This diagonal version has per-LoRA parameter savings when compared to JD-Full, since the diagonal \(\Sigma_i\) only needs \(r\) parameters instead of \(r^2\).

3.2 Clustering↩︎

As the number of LoRAs \(n\) grows and becomes more diverse, the rank \(r\) needed for Joint Diagonalization to achieve good performance will tend to increase. This increases the size of each \(\Sigma_i\) that needs to be stored, especially for JD-Full which will require \(O(n r^2)\) storage for these matrices. If the necessary \(r\) grows proportionally to \(n\), then this storage will eventually become the bottleneck.

To resolve this limitation with very large \(n\), we propose to group the \(n\) LoRAs into \(k\) clusters \(C_j\). Each cluster is given its own rank \(r\) for JD compression, and the clusters are chosen such that the overall reconstruction error is minimized. Specifically, the overall objective is \[\begin{align} \min_{\{\{C_j\},{U_j,V_j}\},\{\Sigma_i\}}\sum_j\sum_{i\in C_j}||B_iA_i-U_j\Sigma_iV_j||^2_{F}, \end{align}\] optimized by alternating between cluster assignments and the JD of each cluster; Appendix 8.3 provides details. Typically, the goal with large \(n\) is to have \(k\) grow with \(n\) as \(r\) becomes fixed. Comparing \(k\) rank-\(r\) JD-Full clusters to a rank-\(kr\) JD-Full single cluster compression, the clustered approach requires \(O(dkr + nr^2)\) parameters, while the single-cluster approach requires \(O(dkr + nk^2r^2)\) parameters due to the increased sizes of the \(\Sigma_i\)s. While these two approaches have the same rank, they may have different reconstruction abilities. Empirically, we find that multiple clusters significantly aid performance for \(n \geq 100\).

4 Theoretical Analysis↩︎

In this section, we seek to better understand the role of the joint diagonalization method in §3.1 and how it motivates the clustering approach. We focus on the full-\(\Sigma_i\) case with orthogonal \(U\), \(V\) matrices. Note that, for the same \(r\), the \(r\)-JD-Diag has at least as large reconstruction error as \(r\)-JD-Full since it imposes an additional constraint on the \(\Sigma_i\).

Firstly, note that perfect reconstruction can be achieved if and only if \(r\) is large enough, since there exist \(U,V\) such that all the \(B_i\), \(A_i\) are in the spans of \(U,V\) resp.if and only if \(r \geq \tilde{r}\):

Due to training noise, \(\tilde{r}\) will equal \(\sum_{i=1}^n r_i\) almost always. This implies that in most realistic settings, the joint diagonalization approach is a lossy reconstruction.

This reconstruction loss can be significant, as the following theorem shows (proved in Appendix 9):

In other words, reconstruction error is unavoidable if \(L\)’s singular values are not concentrated in the top \(r^2\) entries.

For orthogonal LoRAs, the singular values of \(L\) are the norms of the LoRAs, suggesting the following corollary:¹

This implies that for the common setting where \(r^2 \ll n\), the reconstructed LoRAs will be significantly smaller than the original LoRAs, with significant reconstruction error.

Our analysis illustrates the tradeoffs of joint diagonalization. If the LoRAs are similar or well-clustered, reconstruction error will be low. On the other hand, if the LoRAs are random and orthogonal, reconstruction error will be high.

Since the loss space of transformers is highly complex, increasing reconstruction error does not necessarily degrade LLM performance. Interestingly, Figure 3 below shows that while large reconstruction error rapidly decreases performance, moderate (but still relatively large, at around 60%) reconstruction error does not damage performance and may even slightly outperform the zero-error setting. At the same reconstruction error, clustering outperforms non-clustering. This motivates our focus on minimizing reconstruction error, while also suggesting that our approach achieves something deeper than compression. Specifically, joint diagonalization finds subspaces that are shared among many LoRAs when \(r\) is large and merges subspaces when \(r\) is small. When \(r\) is particularly small, this tendency towards averaging all or some of the LoRAs connects to merging LoRAs, whose empirical success [39], [40] could explain the procedure’s success despite the nonlinearity of transformers.

Appendix 15.11 explores this idea further, comparing reconstruction of real-world LoRAs to reconstruction of randomly sampled LoRAs. The reconstruction error is generally large, but significantly lower than the reconstruction error for random noise, indicating that a major shared component between the LoRAs is successfully retained.

That said, as the number of LoRAs grows, the shared component may not be significant enough to maintain sufficiently low reconstruction error with low rank \(r\). This motivates the introduction of clustering in §3.2, since clustering seeks to find groups of LoRAs that are similar and better compressible by joint diagonalization. In particular, if the number of clusters \(k\) grows with \(n\), the reconstruction error may no longer degrade with \(n\) even when \(r\) is fixed.

In the extreme case where \(k = n\), each LoRA is compressed independently. By the Eckart-Young Theorem, JD applied to a single LoRA reduces to an SVD, replacing each rank-\(r_i\) LoRA adapter \(B_i A_i\) with a reduced rank-\(r\) approximation, where typically \(r < \frac{1}{n}\sum_{i=1}^n r_i\): \[\begin{align} \operatorname{SVD}_r(B_i A_i) = U_i \Sigma_i V_i^\top ,\quad \forall i=1,\dots,n. \label{eq:svd} \end{align}\tag{4}\] As \(\Sigma_i V_i^\top\) can be saved as a single matrix, this approach has \(r n (d_A +d_B)\) parameters. We refer to this \(k = n\) method as r-SVD and find that it underperforms our other methods while slightly outperforming the baseline uncompressed LoRAs. This result parallels [17]’s observation that lowering LoRA ranks is beneficial for multi-task learning and model merging.

5 Training & Performance Evaluation↩︎

5.1 Training↩︎

We trained LoRA adapters on 1000 natural instruction tasks [9] using Mistral-7B-Instruct-v0.2 [8] as the base. We set all LoRA adapter ranks to 16 (i.e., \(\forall i, r_i=16\)), except for those in our ablation study (Appendix 15.1), where we vary the LoRA rank.

We selected 10 diverse tasks (Table ¿tbl:table:task95information? in Appendix 10) manually for consistent evaluation across experiments and randomly sampled an additional 990 tasks, resulting in a total of 1000 tasks (Table ¿tbl:app:500tasks?). The tasks went through a robust reviewing protocol to ensure high quality and diversity. Each task data was divided into training, validation, and test sets.

Hyperparameters, such as early stopping, were tuned using the validation sets. Table 1, Appendix 10 shows that on the test sets, LoRA consistently outperformed the base model in terms of Rouge scores and loss metrics.

5.2 Evaluation↩︎

We evaluated multiple metrics for the natural instruction tasks, including cross-entropy loss, Rouge-1, Rouge-L [41], exact match, and agreement between uncompressed and compressed LoRA. Here, agreement measures the exact match in task-generations between the uncompressed LoRA model and the compressed LoRA model, rather than comparing to ground truth data. While detailed results and discussions for all metrics are provided in Appendix 15, our primary focus in the main text is on Rouge-L. We find that all metrics correlate, but Rouge-L correlates most strongly with downstream utility. This finding aligns with prior work [9], which demonstrates that Rouge-L correlates well with classification accuracy.

While cross-entropy is used for optimization during training, identical generation outputs across models can yield different cross-entropy losses. Exact match is too rigid and does not account for the variability in task responses. Similarly, agreement does not capture the inexactness associated with most of our tasks, nor does it account for the performance gains or losses of the compressed LoRAs. Arguably, practitioners are primarily concerned with task performance in the settings for which the LoRA was designed, rather than exact generational agreement between models.

Joint diagonalization optimizes reconstruction error measured by the Frobenius norm, bounded by our theoretical analysis in §4. We empirically study the relation between the reconstruction error and downstream Rouge-L performance in Section 6.2.

Instead of listing absolute performance, we compute the performance difference between the base model and the LoRA model for each task via the ratio \[\begin{align} \text{Performance relative to LoRA} := \frac{\text{method-performance}}{\text{LoRA-performance}} \end{align}\] for the method in question, highlighting relative improvement wrt the uncompressed LoRAs.

6 Experiments↩︎

6.1 Task Performance↩︎

For each method, we vary the number \(n\) of compressed LoRAs and the compression rank \(r\). We run each experiment three times with different random seeds and report the mean and standard deviation. See Table ¿tbl:table:across-numbers-of-loras-relative? for results evaluated on the same ten manually-selected tasks (Table ¿tbl:table:task95information?) across settings. Every compressed collection of LoRAs contains these 10 tasks (i.e., in-distribution tasks), and each collection contains the smaller collections as subsets.

We normalize each LoRA adapter to have a Frobenius norm of one prior to running joint diagonalization. This normalization enhances performance and reduces the variance in reconstruction error. We restore the original norms of the LoRA adapters before reconstruction and testing.

Figure 2 illustrates the Rouge-L scores of the compressed LoRAs divided by the Rouge-L scores of the uncompressed LoRAs. JD variants often increase generalization and outperform the original LoRA. Notably, our JD methods approach the compression efficacy of a single LoRA, and with clustering, this aggressive reduction in size also maintains performance in larger collections. Appendix 15 includes tables of additional relative and absolute metrics.

For efficiency, we limited the JD methods to ten iterations instead of full convergence. While the alternating algorithm quickly reaches an approximate minimizer, squeezing out the last few digits of precision takes many more iterations with limited to no performance gain. Appendix 15.12 also evaluates an alternative iterative algorithm that converges more rapidly once \(U,V\) are close to a minimizer, with minimal performance differences.

6.2 Performance and Reconstruction Error↩︎

Figure 3 relates reconstruction error and performance. The \(y\)-axis measures the mean performance improvement of Rouge-L relative to uncompressed LoRA, and the \(x\)-axis quantifies the mean relative reconstruction error between the compressed reconstruction of the product \(BA\) and the original product \(BA\). Although performance and reconstruction error relate non-linearly, we see a decreasing, somewhat exponential trend. Notably, minimizing reconstruction error does not yield optimal performance, indicating that mild lossy reconstruction may enhance generalization. Interestingly, under the clustering approach, compared to non-clustering, even more aggressive lossy reconstruction can outperform less lossy reconstruction, suggesting that reconstruction error is even less critical for performance in the clustering scenario.

To select hyperparameters (compression rank and number of clusters) for the clustering experiments, we first assessed reconstruction error on a single LoRA module over a range of settings (see Appendix 14). These preliminary experiments enabled efficient selection of cluster counts and rank values for compressing all LoRA modules.

6.3 Benefits of Compression↩︎

Compressing LoRAs reduces their parameter counts, thus lowering their overall memory footprint. While this offers many benefits, in both training and inference scenarios parameters are often transferred between different memory hierarchies (e.g., from CPU to GPU), and these transfers usually scale linearly with the amount of data moved. At the same time, compressed LoRAs alter the forward-pass formulation (unless merged with the base model weights). As shown in Figure 5 in the Appendix 12, although compression greatly reduces memory usage and transfer time, it does not affect the forward-pass latency. Of the various ways to leverage these improvements, this work focuses on optimizing inference for multiple LoRAs using vLLM.

6.4 Throughput of Serving Compressed LoRAs↩︎

The previous sections demonstrate how to select an appropriate joint compression setting guided by the reconstruction error, such that the performance of the original LoRAs is preserved. Naturally, the rank and/or the number of clusters for the compression needs to increase as we compress larger LoRA collections to match LoRA performance.

Figure 4 studies how throughput with various compression settings compares to the vLLM multi-LoRA throughput with the matched GPU memory footprint. Specifically, for each number of unique LoRAs served and each compression setting, we compute the corresponding number of LoRAs to be placed on the GPU during serving and report the ratio of the two throughputs. For example, when serving 64 unique LoRAs and using rank 64 JD-Full compression, we report the ratio of throughputs of rank 64 JD-Full and vLLM multi-LoRA with 6 LoRAs allowed on the GPU at a time (see Appendix 13 for details). As the number of unique LoRAs increases, vLLM multi-LoRA throughput degrades as it needs to schedule the requests and load and offload the adapters. We note that vLLM multi-LoRA already employs advanced optimizations, such as efficient scheduling and non-blocking CPU-GPU communication when swapping LoRAs as well as techniques introduced in S-LoRA [5], [6], but system optimization alone is insufficient to mitigate throughput degradation when serving many LoRAs.

Figure 4 shows that across LoRA collection sizes our compression techniques improve the throughput of vLLM multi-LoRA. Additionally, we highlight regions for each compression setting where compression is sufficiently moderate to achieve 99%+ of LoRA performance, according to the results in §6.2. Compression with a larger rank or too many clusters does not improve baseline throughput when serving a smaller number of LoRAs and should not be used in such cases. For example, rank 16 JD-Full improves baseline throughput with 4 and 8 LoRAs, but will underperform with more LoRAs, while 25 clusters rank 15 JD-Full does not improve throughput with 32 or fewer LoRAs, but when serving 1000+ LoRAs it improves the throughput significantly while maintaining the performance. Overall, an appropriate joint compression setting improves vLLM multi-LoRA throughput and preserves performance for LoRA collections of any size between 4 and 1024, as in Figure 1. Appendix 13 provides compression settings for each collection size.

vLLM extensively uses custom CUDA kernels. To accommodate our compression techniques, we minimally adjusted the vLLM code to generate additional kernels needed by the compressed LoRAs and used the Punica [21] kernel to further accelerate matrix multiplication. Pseudocode is given in §13.4 to show how we use the batch multiplication kernel. There likely is room for improvement to optimize the newly added kernels.

6.4.0.1 Additional details.

In this experiment, we considered a varying number of rank-16 LoRAs, using a dataset of Shakespeare sonnets as inputs² arriving asynchronously. We measured throughput, i.e., the number of requests served per second when generating ten tokens per request. The base was Mistral 7B Instruct; we simulated random LoRAs and assigned inputs to LoRAs at random. Experiments were conducted on H100 80GB GPU capped at 40% memory consumption to reflect situations where a service provider might want to serve many LoRAs from cheaper hardware with lower memory than higher-end GPUs. This setting applies to the scenario where the LLM is large compared to the size of GPU and yet a provider may want to serve many LoRAs efficiently using one device.

6.5 Recommendations↩︎

JD-Full is generally preferred over JD-Diag, although for smaller numbers of LoRAs (less than 100), the performance difference is negligible. While JD-Full alone is effective up to 100 LoRAs, incorporating clustering at scales of 500-1000 LoRAs significantly enhances performance.

We recommend the following procedure for hyperparameter selection. For \(\leq100\) LoRAs, JD-Full can be used without substantial degradation, using a rank \(\approx(\text{number of LoRAs} / 2) + 7\). Beyond 100 LoRAs, clustering becomes increasingly critical. A robust method for any number of LoRAs up to 1000 uses JD-Full with clustering. Specifically, select a LoRA module from the middle of the network, apply a compression rank of 16, and experiment with an exponentially increasing number of clusters. Compute the reconstruction error for each setting on this module across all LoRAs—a computationally efficient process. Choose the minimal number of clusters that achieves a reconstruction loss below 0.6, and then use these settings across LoRA modules. Figure 6 in the Appendix illustrates this procedure applied to 500 LoRAs.

Tuning hyperparameters as discussed above using reconstruction loss as a validation metric is convenient since it can be done efficiently on CPU without expensive LLM evaluation. As our experiments demonstrate, compression settings that achieve below 0.6 reconstruction loss reliably preserve 99% or more of the LoRA performance, sometimes even outperforming the original LoRAs.

For inference, this procedure is executed as a preprocessing step before deploying our inference server. As new LoRAs are submitted, they are initially served uncompressed. A background CPU job can periodically re-run the compression algorithm and update the served LoRA parameters with the compressed versions.

7 Discussion↩︎

This study introduces approaches to LoRA compression, addressing significant challenges emerging as customization of foundation models such as LLMs and diffusion models becomes increasingly popular. Our contributions include theoretical formulations, empirical validation, and practical implementations that enhance the understanding and application of LLMs in scalable environments.

Our findings have several implications. Our theoretical bounds on reconstruction error not only increase confidence in the use of compressed models but also lay a groundwork for future explorations. Demonstrating that our compression techniques can preserve up to 100% of the original LoRAs’ performance highlights the effectiveness of our methods. Furthermore, integrating LoRA compression into state-of-the-art LLM serving systems demonstrates potential for resource optimization, with throughput for thousands of LoRAs nearing that of a single LoRA.

Our promising results suggest several future research directions. First, further compression may be possible via quantization, since joint-diagonalization and quantization are independent compression strategies. Second, when scaling to hundreds of thousands of LoRAs, joint compression, while effective, will be insufficient to fit all LoRAs onto the GPU, thus requiring a procedure to schedule the requests. Clustering offers opportunities for efficient scheduling that incorporates the cluster assignments of LoRAs corresponding to the incoming requests.

Privacy presents another research direction, particularly regarding the possibility of information leakage during joint compression. As a preliminary study, Appendix 15.2 investigates whether a base model with an adapter \(A\) for task \(T_A\), after being jointly compressed alongside an adapter \(B\) for task \(T_B\), inadvertently improves on \(T_B\). Such an outcome would indicate that adapter \(A\) acquired information from adapter \(B\). Our ablation study shows no performance gains on \(T_B\), suggesting that the compressed adapter \(A\) remains independent and does not leak—or gain—information from adapter \(B\). A more detailed investigation of the privacy properties of joint compression is an interesting next step.

In conclusion, our research advances LLM deployment by providing robust, scalable, and efficient compression. The ability of compressed LoRAs to maintain high performance while saving resources opens avenues for the broad application and adoption of LLMs across various industries. We encourage the community to build upon our findings and shared LoRAs to further enhance these technologies.

Impact Statement↩︎

This paper presents work whose goal is to advance machine learning. There are no societal consequences of our work that we feel must be specifically highlighted here.

Acknowledgements↩︎

The MIT Geometric Data Processing Group acknowledges the generous support of Army Research Office grants W911NF2010168 and W911NF2110293, of National Science Foundation grant IIS2335492, from the CSAIL Future of Data program, from the MIT–IBM Watson AI Laboratory, from the Wistron Corporation, and from the Toyota–CSAIL Joint Research Center.

8 Joint Diagonalization Algorithms↩︎

8.1 Alternating Methods↩︎

Our goal is to derive algorithms that optimize equation 1 . Common to both methods, we expand the objective functional: \[\begin{align} \sum_i \|B_iA_i - U\Sigma_i V^\top\|_\mathrm{Fro}^2 &=\sum_i \mathrm{tr}( (B_iA_i - U\Sigma_i V^\top)(B_iA_i - U\Sigma_i V^\top)^\top )\textrm{ by definition}\nonumber\\ &=\sum_i \left[ \mathrm{tr}( B_iA_i A_i^\top B_i^\top ) - 2\mathrm{tr}( B_iA_i V\Sigma_i^\top U^\top ) + \mathrm{tr}( U\Sigma_i V^\top V\Sigma_i^\top U^\top ) \right]\nonumber\\ &=\textrm{const.} -2\sum_i \mathrm{tr}( B_iA_i V\Sigma_i^\top U^\top ) + \sum_i \|U\Sigma_iV^\top\|_\mathrm{Fro}^2.\label{eq:expandedobjective} \end{align}\tag{5}\] Using this expansion, we now consider the two settings discussed in §3.1.

Case 1: Non-diagonal \(\Sigma_i\), orthogonal \(U,V\). Setting the derivative of equation 5 with respect to \(\Sigma_i\) to zero, we find \[\label{eq:sigmastar} \Sigma_i=\Sigma^\ast_i (U,V) = U^\top B_iA_i V.\tag{6}\]

We simplify our objective function after plugging in this expression: \[\begin{align} \sum_i \|B_iA_i &- U\Sigma_i V^\top\|_\mathrm{Fro}^2+\mathrm{const.} = \sum_i \left[\|\Sigma_i\|_\mathrm{Fro}^2 -2 \mathrm{tr}( B_iA_i V\Sigma_i^\top U^\top ) \right]\textrm{ from equation~\ref{eq:expandedobjective}}\\ &= \sum_i \left[\mathrm{tr}(U^\top B_iA_i V V^\top A_i^\top B_i^\top U) -2 \mathrm{tr}( B_iA_i VV^\top A_i^\top B_i^\top U U^\top ) \right]\textrm{ from equation~\ref{eq:sigmastar}}\\ &=- \sum_i \mathrm{tr}( B_iA_i VV^\top A_i^\top B_i^\top U U^\top ). \end{align}\]

Substituting equation 6 , we find \[U_{opt}, V_{opt} = \arg\max_{\substack{U^\top U = I\\VV^\top = I}}\;\sum_{i=1}^n \|U^\top B_i A_i V\|_\mathrm{Fro}^2 =\arg\max_{\substack{U^\top U = I\\VV^\top = I}}\;\sum_{i=1}^n \| \Sigma^\ast_i (U,V)\|_\mathrm{Fro}^2. \label{eq:optUV}\tag{7}\] Note that \[\begin{align} \sum_{i=1}^n\|U^\top B_i A_i V\|_\mathrm{Fro}^2 &= \mathrm{tr}\left(\left(\sum_{i=1}^nB_i A_i VV^\top A_i^\top B_i^\top \right)UU^\top \right)\\ &= \mathrm{tr}\left(\left(\sum_{i=1}^nB_i^\top A_i^\top UU^\top A_iB_i\right)VV^\top \right), \end{align}\] by the identity \(\|A\|_\mathrm{Fro}^2 = \mathrm{tr}(A^\top A)\). Hence, we optimize equation 7 by alternating between \(U\) and \(V\):

• \(U\) iteration: Define \(M\mathrel{\vcenter{:}}= \sum_i B_iA_i VV^\top A_i^\top B_i^\top\). Parenthesizing this expression properly requires only \(O( (m+n)r)\) storage/computation time. With this definition, we maximize \(\mathrm{tr}(MUU^\top)\) over \(U\) satisfying \(U^\top U=I\). Since \(M\) is positive semidefinite, the optimum is to take \(U\) to be the \(r\) eigenvectors of \(M\) with largest eigenvalue, equivalent to an SVD problem.

• \(V\) iteration: Define \(N\mathrel{\vcenter{:}}= \sum_i A_i^\top B_i^\top UU^\top B_iA_i\). Similarly to the previous step, we take \(V\) to contain the \(r\) eigenvectors of \(N\) with largest eigenvalue, again solvable using an SVD.

This method decreases the objective in each step. Case 2: Diagonal \(\Sigma_i\). If constrain \(\Sigma_i\) to be diagonal, we interpret our objective function equation 1 as a “triple least squares” problem. We compute gradients: \[\begin{align} \nabla_U\sum_i \|B_iA_i - U\Sigma_i V^\top\|_\mathrm{Fro}^2 &=2\sum_i (U\Sigma_i V^\top-B_iA_i)V\Sigma_i^\top \\ \nabla_V\sum_i \|B_iA_i - U\Sigma_i V^\top\|_\mathrm{Fro}^2 &=2\sum_i (V\Sigma_i^\top U^\top- A_i^\top B_i^\top) U\Sigma_i\\ \nabla_{\Sigma_i}\sum_i \|B_iA_i - U\Sigma_i V^\top\|_\mathrm{Fro}^2 &=2U^\top(U\Sigma_iV^\top-B_iA_i)V \end{align}\] These expressions suggest efficient \(r\times r\) linear systems to solve for \(U,V\): \[\begin{align} U &= \left(\sum_i B_iA_i V\Sigma_i^\top\right)\left(\sum_i \Sigma_iV^\top V\Sigma_i^\top\right)^{-1}\\ V &= \left(\sum_i A_i^\top B_i^\top U\Sigma_i\right)\left(\sum_i \Sigma_i^\top U^\top U\Sigma_i\right)^{-1}. \end{align}\] For \(\Sigma_i\), we extract the diagonal from our gradient above: \[\begin{align} \mathrm{diag}(U^\top U\Sigma_i V^\top V)_j &= (U^\top U\Sigma_i V^\top V)_{jj}\\ &= \sum_m (U^\top U)_{jm} \Sigma_{imm} (V^\top V)_{mj}\\ &= (U^\top U \circ V^\top V)\mathrm{diag}(\Sigma_i)\\ \mathrm{diag}(U^\top B_iA_i V)_j &=\sum_m (U^\top B_i)_{jm} (A_i V)_{mj}\\ &=\sum_m (U^\top B_i)_{jm} (V^\top A_i^\top)_{jm}\\ &=(U^\top B_i \circ V^\top A_i^\top) \boldsymbol{1} \\ \implies \mathrm{diag}(\Sigma_i) &= (U^\top U \circ V^\top V)^{-1}(U^\top B_i \circ V^\top A_i^\top) \boldsymbol{1} \end{align}\] Here \(\circ\) denotes the Hadamard product.

Combining these expressions, we use a simple coordinate descent algorithm cycling between the following three steps:

1. Solve for \(U\)

2. Solve for \(V\)

3. Solve for the \(\Sigma_i\)’s

4. Optionally, normalize so \(\sum_i \|\Sigma_i\|_\mathrm{Fro}^2=1\)

8.2 Additional Eigenvalue Iteration Algorithm↩︎

For the first case in §8.1, we introduce an alternative algorithm that eschews the use of SVD. This alternative is optimized for GPU execution, enabling tractable runs to convergence.

To derive this algorithm, we employ Lagrange multipliers to formulate the derived objective from equation 7 : \[U_{opt}, V_{opt} = \arg\max_{\substack{U^\top U = I\\VV^\top = I}}\;\sum_{i=1}^n \|U^\top B_i A_i V\|_\mathrm{Fro}^2,\] yielding the expression \[\Lambda = -\frac{1}{2} \|U^\top B_i A_i V\|_\mathrm{Fro}^2 - \frac{1}{2}\text{tr}(X^\top(I-U^\top U)) - \frac{1}{2} \text{tr}(Y^\top(I-V^\top V)).\]

Taking the derivatives gives \[\begin{align} \nabla_U \Lambda =& - \sum_i B_i(A_i V)(V^\top A_i^\top)(B_i^\top U) + UX \\ \nabla_V \Lambda =& - \sum_i A_i^\top (B_i^\top U)(U^\top B_i)(A_i V) + VY \end{align}\] Setting these derivatives to zero shows \[\begin{align} \sum_i B_i(A_i V)(V^\top A_i^\top)(B_i^\top U) &= UX \\ \sum_i A_i^\top (B_i^\top U)(U^\top B_i)(A_i V) &= VY. \end{align}\] Here, one can show that the Lagrange multiplier matrices \(X\) and \(Y\) are diagonal and nonnegative, since the problem reduces to an eigenvalue problem when either \(U\) or \(V\) is fixed; this is essentially the argument behind the alternating algorithm in Appendix 8. Hence, taking inspiration from classical eigenvalue iteration, we use the following updates to improve our estimates of \(U\) and \(V\): \[\begin{align} U_0^{(k+1)}&\gets\sum_i B_i(A_i V^{(k)})((V^{(k)})^\top A_i^\top)(B_i^\top U^{(k)}) \\ V_0^{(k+1)}&\gets\sum_i A_i^\top (B_i^\top U^{(k)})((U^{(k)})^\top B_i)(A_i V^{(k)})\\ U^{(k+1)}&\gets\texttt{orthogonalize}(U_0^{(k+1)})\\ V^{(k+1)}&\gets\texttt{orthogonalize}(V_0^{(k+1)}) \end{align}\] Here, the function orthogonalize orthogonalizes the columns of a matrix, e.g.by using the \(Q\) part of the reduced-size \(QR\) factorization. Although we lack a formal convergence proof, in practice we find that this method reliably reaches a local optimum of our problem.

By executing matrix operations in the specified sequence, these computations can be rapidly performed on GPUs. Note the expressions above are parenthesized to avoid constructing a large matrix product as an intermediate computation.

8.3 Clustering algorithm↩︎

Initialization: We run joint diagonalization with a single \(U,V\) then perform k-means with \(k\) clusters on the space of \(\Sigma_i\)’s. This gives us our first clusters and we can use random initialization \(U_j,V_j\) for each cluster but the \(\Sigma_i\) can be maintained as initialization.

Step 1: Using the alternating JD algorithms from earlier in this section, we optimize the problem \(\min_{U_j,V_j,\Sigma_i}\sum_{i \in C_j}||B_iA_i-U_j\Sigma_iV_j^\top||^2_{F}\) for each \(j\) independently.

Step 2: New cluster assignment for \(i\) : \(\min_j \min_{\Sigma_i} ||B_iA_i-U_j\Sigma_iV_j^\top||^2_{F}\). If any assignment changes we go to Step 1, else we have converged.

9 Proof of Theorem 1↩︎

Note that we have only used the fact that the matrix \((V \otimes U)\) has singular values equal to 1; we have not used the fact that it has Kronecker product structure. On the other hand, each vector \(\mathrm{vec}(B_iA_i)\) is a sum of \(r_i\) Kronecker products and cannot be expressed as a Kronecker product. As a result, while the upper bound in the Von Neumann trace inequality is achieved if the eigenvectors of the two matrices align, the Kronecker product structure is a severe constraint and the upper bound we have provided is generous.

10 Training LoRAs↩︎

We trained LoRA adapters on 500 natural instruction tasks [9] using Mistral-7B-Instruct-v0.2 [8] as the base model. All LoRA adapters were configured with a rank of 16, i.e., \(\forall i, r_i=16\). We selected 10 diverse tasks manually for consistent evaluation across experiments and randomly sampled an additional 490 tasks, resulting in a total of 500 tasks. These tasks were exclusively in English (both input and output), ensuring higher quality and thorough review [9]. Each task dataset was divided into training, validation, and test sets (80-10-10). Hyperparameters, such as early stopping, were tuned using the validation sets; that is, we train for five epochs and take the best-performing epoch-checkpoint per validation loss. Evaluation on the test sets demonstrated that LoRA consistently outperformed the base model in terms of both Rouge scores and loss metrics (see Table 1).

In Table 1, we compare metrics between base model and LoRA finetuning.

width=,center

In Table ¿tbl:app:500tasks? we include all 1000 tasks that were used.

center

We use Huggingface [42] in our implementation. For the base model, we use quantization with configuration:

    BitsAndBytesConfig(
        load_in_4bit=True,
        bnb_4bit_use_double_quant=True, 
        bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16,
    )

and LoRA configuration:

    LoraConfig(
        r=16,
        lora_alpha=32,
        target_modules=["q_proj", "k_proj", "v_proj"],
        lora_dropout=0.05,
        bias="none",
        task_type="CAUSAL_LM",
        init_lora_weights=init_lora_weights,
    )

11 Avoiding Batched Matrix Multiplication (BMM)↩︎

Fast LoRA [20] aims to alleviate the batched matrix multiplication (BMM) bottleneck when serving many LoRAs. They propose an adapter parameterization that replaces addition with elementwise multiplication, avoiding BMM and improving LoRA throughput at lower ranks. Our JD LoRA formulation also circumvents or heavily reduces the impact of BMM as discussed below, and both individual and joint compression methods can be applied to Fast LoRAs.

In the envisioned deployment scenario, a service provider hosts a large collection of LoRAs. Upon receiving a request, each user specifies both the input data and the desired LoRA identifier. The provider then processes the base model augmented with the specified LoRA for each user’s data. As a provider is batching a collection of requests for GPU parallelization, they can expect to frequently have more than one unique LoRA identifier per batch.

Traditionally, a specific LoRA is integrated into the base model by transforming \(W_0 \rightarrow W_0 + B_iA_i\). Serving multiple LoRAs conventionally would necessitate maintaining and executing a separate copy of the base model for each LoRA, bringing substantial computational overhead. Alternatively, the computation for \(W_0x\) and \(B_iA_ix\) can be performed independently and subsequently merged. This strategy necessitates only a single instance of \(W_0x\) computation and storage of LoRA-specific parameters rather than the entire base model.

Consider the batch processing of \(\mathbf{B}\mathbf{A}\mathbf{x}\), where boldface indicates that \(B_i, A_i\) are stacked into tensors of dimensions \((b \times m \times r)\) and \((b \times r \times n)\) respectively, with batched data \(\mathbf{x}\) shaped \((b \times l \times n)\): \[\begin{align} \mathbf{A}\mathbf{x} \leftrightarrow (b \times r \times n) \times (b \times l \times n) \rightarrow (b \times l \times r) \text{\;\;bmm} \\ \mathbf{B(Ax)} \leftrightarrow (b \times m \times r) \times (b \times l \times r) \rightarrow (b \times l \times m) \text{\;\;bmm}. \end{align}\] Here, “bmm” denotes batched matrix multiplication, a known bottleneck in both throughput and latency. Consider the corresponding operations for our joint compression scheme, \(U \mathbf{\Sigma} V^\top x\): \[\begin{align} V^\top \mathbf{x} \leftrightarrow (\tilde{r} \times n) \times (b \times l \times n) \rightarrow (b \times l \times \tilde{r}) \text{\;\;broadcasted} \\ \mathbf{\Sigma} (V^\top \mathbf{x}) \leftrightarrow (b \times \tilde{r}) \times (b \times l \times \tilde{r}) \rightarrow (b \times l \times \tilde{r}) \text{\;\;broadcasted} \\ U(\mathbf{\Sigma} V^\top \mathbf{x}) \leftrightarrow (m \times \tilde{r}) \times (b \times l \times \tilde{r}) \rightarrow (b \times l \times m) \text{\;\;broadcasted} \end{align}\] In our optimized setup, batched matrix multiplications can be completely circumvented if the \(\Sigma_i\) matrices are diagonal. If not, given that \(\tilde{r} \ll m, n\), any required batched matrix multiplication remains computationally inexpensive.

12 Simple Timing Experiments↩︎

In Figure 5, we present a set of simple experiments comparing the memory load, transfer time, and forward-pass performance of LoRA and JD-LoRA. These experiments were conducted across multiple clusters and various rank configurations. For memory usage, we measured all 96 LoRA modules in the Mistral model; however, for transfer time and forward-pass performance, we tested only a single LoRA module. We also implemented F-LoRA [20], but contrary to their reported results, we were unable to achieve faster forward-pass performance than standard LoRA.

13 GPU Memory Usage Computation for JD Compression.↩︎

The GPU memory consumption is primarily influenced by the number of parameters that need to be stored and processed during inference. In this section, we introduce the detail of how we compute the GPU consumption of our method, and how we find the number of vLLM multi-LoRA that share the same GPU utilization.

• \(D\): Hidden dimension size (e.g., \(D = 4098\)).

• \(r\): Rank of the shared basis matrices for compression (e.g., \(r = 16, 32, 64\)).

• \(N\): Maximum number of LoRA modules being served simultaneously (\(\text{max\_lora\_num}\)).

• \(c\): Number of clusters in our clustering method (e.g., \(c = 7, 10, 25\)).

In Figure 1, we use different JD-compression settings for serving different number of unique LoRAs. Specifically:

• Serving 4 unique LoRAs:
  Ours: rank 16 JD-Full.
  vLLM multiLoRA baseline: max-gpu-lora = 2.

• Serving 8 unique LoRAs:
  Ours: rank 16 JD-Full.
  vLLM multiLoRA baseline: max-gpu-lora = 2.

• Serving 16 unique LoRAs:
  Ours: rank 32 JD-Full.
  vLLM multiLoRA baseline: max-gpu-lora = 3.

• Serving 32 unique LoRAs:
  Ours: rank 64 JD-Full.
  vLLM multiLoRA baseline: max-gpu-lora = 5.

• Serving 64 unique LoRAs:
  Ours: rank 64 JD-Full.
  vLLM multiLoRA baseline: max-gpu-lora = 6.

• Serving 128 unique LoRAs:
  Ours: 7 clusters, rank 16 JD-Full.
  vLLM multiLoRA baseline: max-gpu-lora = 8.

• Serving 256 unique LoRAs:
  Ours: 10 clusters, rank 16 JD-Full.
  vLLM multiLoRA baseline: max-gpu-lora = 10.

• Serving 512 unique LoRAs:
  Ours: 25 clusters, rank 16 JD-Full.
  vLLM multiLoRA baseline: max-gpu-lora = 26.

• Serving 1024 unique LoRAs:
  Ours: 25 clusters, rank 16 JD-Full.
  vLLM multiLoRA baseline: max-gpu-lora = 28

13.1 Baseline GPU Memory Usage↩︎

The baseline for our comparison is the standard LoRA method with a rank of 16. The total parameter count for the baseline is given by:

\[\text{Params}_{\text{baseline}} = D \times 2 \times 16. \nonumber\]

This accounts for the parameters in the LoRA-adapted layers, where the factor of 2 represents the weights and biases.

13.2 GPU Memory Usage for JD Full Method↩︎

For the Joint Decomposition (JD) Full method without clustering, the total parameter count is:

\[\text{Params}_{\text{JD\_Full}} = D \times 2 \times r + N \times r^2. \nonumber\]

• \(D \times 2 \times r\): Parameters for the base model adapted with rank-\(r\) LoRA.

• \(N \times r^2\): Additional parameters introduced by each of the \(N\) LoRA modules, each of size \(r \times r\).

The GPU memory usage ratio relative to the baseline is:

\[\text{GPU Usage Ratio}_{\text{JD\_Full}} = \frac{\text{Params}_{\text{JD\_Full}}}{\text{Params}_{\text{baseline}}} = \frac{D \times 2 \times r + N \times r^2}{D \times 2 \times 16}. \nonumber\]

13.3 GPU Memory Usage for Clustering Method↩︎

When employing clustering, the parameter count changes due to the addition of cluster-specific parameters:

\[\text{Params}_{\text{Clustering}} = D \times 2 \times r \times c + N \times (r^2 + 1). \nonumber\]

• \(D \times 2 \times r \times c\): Parameters for the base model adapted with rank-\(r\) LoRA across \(c\) clusters.

• \(N \times (r^2 + 1)\): Additional parameters for each LoRA module and cluster assignments.

The GPU memory usage ratio is:

\[\text{GPU Usage Ratio}_{\text{Clustering}} = \frac{\text{Params}_{\text{Clustering}}}{\text{Params}_{\text{baseline}}} = \frac{D \times 2 \times r \times c + N \times (r^2 + 1)}{D \times 2 \times 16}. \nonumber\]

13.4 Punica↩︎

In our vLLM experiments, we specifically used the Punica kernel for implementing multi-LoRA, applying our approach in conjunction with Punica’s capabilities. Our custom function, add_lora_slice_with_sigma, implements the following key steps:

1. Initialize Buffers: Creates temporary storage for intermediate calculations if not already provided.

2. Apply Matrix A: Transforms x using matrix A, storing the result in buffer.

3. Apply Matrix Sigma: Further transforms buffer using Sigma, storing the result in buffer_sigma.

4. Apply Matrix B and Update y: Finally, transforms buffer_sigma using B, applies scaling, and updates a slice of y in place.

Below is the pseudocode for add_lora_slice_with_sigma, illustrating the integration:

Function add_lora_slice_with_sigma(y, x, wa_t_all, wb_t_all, wsigma_t_all, indices, layer_idx, scale, y_offset, y_slice_size, buffer=None):
    # Initialize buffers if not provided
    if buffer is None:
        buffer = create_tensor(shape=(x.size(0), R), dtype=float32)
        buffer_sigma = create_tensor(shape=(buffer.size(0), R), dtype=float32)
    # Step 1: Apply matrix A
    dispatch_bgmv_low_level(buffer, x, wa_t_all, indices, layer_idx, scale=1.0)
    # Step 2: Apply matrix Sigma
    dispatch_bgmv_low_level(buffer_sigma, buffer, wsigma_t_all, indices, layer_idx, scale=1.0)
    # Step 3: Apply matrix B and update y slice
    dispatch_bgmv_low_level(y, buffer_sigma, wb_t_all, indices, layer_idx, scale, y_offset, y_slice_size)
End Function

14 Selecting Number of Clusters↩︎

To identify optimal hyperparameters for the clusters compression method, we analyzed the relationship between reconstruction error and the parameter saved ratio for a single LoRA module, as shown in Figure 6. By comparing the results across different numbers of Low-Rank Adaptation (LoRA) configurations (100 and 500, depicted in subfigures 6 (a) and 6 (b)), we were able to observe the trade-off between model size reduction and reconstruction accuracy. Based on these findings, we selected the rank and number-of-clusters hyperparameters that effectively balance these two objectives. The chosen settings were then used to conduct full-scale experiments.

15 Additional Results↩︎

This section elaborates on the results that underpin the figures presented in the main text and showcases a consistent correlation across various evaluation metrics. Additionally, we assess the significance of achieving convergence and the performance of compression on new unseen LoRA models.

15.1 LoRAs of different ranks↩︎

In Table [tbl:table:different-ranks], we report the performance of our compression method on LoRAs with ranks uniformly sampled between 16 and 64 (mean rank of 43). In Table [tbl:table:different-ranks-same], we present compression results for LoRAs of rank 43, matching the average rank from Table [tbl:table:different-ranks]. Because these same-rank LoRAs have an identical parameter count, they also exhibit identical parameter-saving ratios. While performance declines slightly for LoRAs with varying ranks, our compression method still preserves over 99% of the original performance.

15.2 Privacy Ablation↩︎

We investigate whether jointly compressing certain tasks results in improved performance on tasks within that same compressed group, compared to tasks outside of the group. In other words, we examine whether information about which tasks were compressed together could be inferred from subsequent performance differences. Table 1 presents the results.

In the Compressed Together setting, tasks (task1391, task190, task280, task290, and task391) are compressed jointly with five other tasks. We then evaluate their cross-task performance within this group. In the Compressed Separately setting, the same set of tasks is each compressed alongside nine other tasks that are not part of the original group. We again evaluate the cross-task performance using the same sets of tasks, allowing us to compare and assess any differences in performance attributable to joint versus separate compression.

15.3 Relative Rouge-L Performance and Compression Rate↩︎

Table ¿tbl:table:across-numbers-of-loras-relative? presents comprehensive results from the experiments underlying Figure 2 for each evaluation task. Additionally, we incorporate results using the Ties-merging benchmark [43], which consolidates all LoRA-adapters into a single adapter of identical configuration and parameter count; this integration significantly compromises performance.

width=,center

15.4 Absolute Rouge-L Performance and Compression Rate↩︎

Table ¿tbl:table:across-numbers-of-loras-absolute-rl? provides the full results behind Table ¿tbl:table:across-numbers-of-loras-relative?, but with Rouge-L scores instead of relative performance compared to LoRA.

width=,center

15.5 Relative Rouge-1 Performance and Compression Rate↩︎

Table ¿tbl:table:relative95rouge1? provides full results for relative performance of Rouge-1, which shows the same trends as the results for relative performance of Rouge-L (Table ¿tbl:table:across-numbers-of-loras-relative?).

width=,center

15.6 Absolute Rouge-1 Performance and Compression Rate↩︎

Table ¿tbl:table:absolute95rouge1? provides full results for absolute performance of Rouge-1, which shows the same trends as the results for absolute performance of Rouge-L (Table ¿tbl:table:across-numbers-of-loras-absolute-rl?).

width=,center

15.7 Relative Exact-Match Performance and Compression Rate↩︎

Table ¿tbl:table:relative95exact95match? provides full results for relative performance of exact-match, which shows the same trends as the results for relative performance of Rouge-L (Table ¿tbl:table:across-numbers-of-loras-relative?).

width=,center

15.8 Loss and Compression Rate↩︎

Table ¿tbl:table:absolute95loss? provides full results for test loss (cross-entropy), which shows the same trends as the results for relative performance of Rouge-L (Table ¿tbl:table:across-numbers-of-loras-relative?).

width=,center

15.9 Agreement and Compression Rate↩︎

Table ¿tbl:table:absolute95agreement? provides full results for agreement, which shows the same trends as the results for relative performance of Rouge-L (Table ¿tbl:table:across-numbers-of-loras-relative?). Note that agreement measures the exact match in task generations between the uncompressed LoRA model and the compressed LoRA model, rather than comparing to the task’s ground truth data. The comparison is very strict and requires an exact match between the generations of the two models (LoRA and the compressed LoRA), comparing each sample one at a time.

width=,center

15.10 Reconstruction Error and Compression Rate↩︎

Table ¿tbl:table:across-numbers-of-loras-reconstruction-error? provides the full results of the experiments behind Figure 3 for every evaluation task.

width=,center

15.11 Reconstruction Error: Trained vs. Random↩︎

Table ¿tbl:table:across-numbers-of-loras-reconstruction-error-random? provides the reconstruction error on random (untrained) LoRA matrices. Comparing with Table ¿tbl:table:across-numbers-of-loras-reconstruction-error?, we find that reconstruction error is consistently higher on random (untrained LoRA) matrices than on trained LoRA matrices. This demonstrates that after training, LoRAs have a shared structure that JD exploits.

width=,center

15.12 Convergence↩︎

Table ¿tbl:table:across-numbers-of-loras-convergence-relative? presents outcomes where the JD-Full algorithm is executed until convergence. Our convergence criterion is defined as follows: \[\begin{align} \max \left( \|U_{t+1}-U_t U_t^\top U_{t+1} \|_\mathrm{Fro}/ \|U_{t+1}\|_\mathrm{Fro}, \|V_{t+1}-V_t V_t^\top V_{t+1} \|_\mathrm{Fro}/ \|V_{t+1}\|_\mathrm{Fro}\right) < \tau \end{align}\] where the tolerance threshold \(\tau\) is set to 0.001. Due to the slow per-iteration computation times of the primary JD-Full algorithm, which quickly reaches an approximate optimum but then has a long tail of convergence for final digits of precision, we devised an alternative eigenvalue iteration algorithm (Appendix 8.2) optimized for GPU acceleration. Our analysis indicates that adherence to this convergence criterion does not significantly alter the results.

width=,center

15.13 Out-of-distribution Performance (LoRA-hub)↩︎

For completeness, we incorporate results using the protocol of LoRA-hub [40]. That is, 100 LoRA-adapters are sampled, independent of the evaluation task, representing a measure of out-of-distribution performance. This also means that each result on a task is averaged across all 100 LoRA-adapters (as there is no a priori LoRA-to-task mapping). These results were obtained without normalizing the LoRA-adapters before applying the JD algorithms, a step we later identified as beneficial. We present performance comparison in Table ¿tbl:table:lorahub-performance?. Table ¿tbl:table:lorahub-agreement? presents the average agreement between uncompressed and compressed LoRA across 10 evaluation tasks. Results per task for JD-diagonal and JD-full are shown in Table ¿tbl:table:lorahub-diagonal? and Table ¿tbl:table:lorahub-full?, respectively.

From these tables, we find that the JD algorithms successfully maintain performance in this out-of-distribution context.

width=0.45

width=0.45

width=,center

width=,center

References↩︎