1 Introduction↩︎

Large Language Models (LLMs) are powerful general-purpose models that can be adapted to a wide range of problem types in many languages, including question answering [1], translation [2], summarization [3], text rewriting [4], and grammar correction [5]. Due to their substantial parameter count [6], [7], fine-tuning these models for task-specific applications (e.g., a problem type in a given language) is commonly approached via parameter-efficient tuning methods [8], [9]. Among these, Low-Rank Adapters (LoRAs) [10] are especially popular: they insert lightweight trainable modules into the base model and update only these modules during fine-tuning, freezing the LLM weights. This enables deploying a single base LLM and dynamically loading small LoRAs to support various tasks.

A growing use case for LLMs is their deployment on mobile and edge devices [11], driven by benefits such as improved privacy, lower latency, reduced server load, and offline access. However, the size of LLMs suitable for on-device deployment is typically limited to 1–2B parameters, much smaller than their server-scale counterparts, e.g., 70–400B models [12]. In such settings, prompting alone often results in subpar performance. Instead, task-specific LoRAs are commonly employed to recover acceptable accuracy [13]–[16], offering a practical way to extend functionality without retraining the base model.

While a single LoRA may suffice for one task, a key challenge arises in real-world usage where LoRAs arrive incrementally, as users request new functionality over time. However, on-device storage is limited, making it infeasible to retain a separate LoRA for every task. One promising solution is to construct multi-task adapters by merging existing LoRAs [17]. Several merging strategies have been proposed, from linear combinations with fixed or learned coefficients [18], [19] to more advanced approaches such as TIES [20] and DARE [21].

A remaining question then is how to integrate each new LoRA given only the existing adapters and a limited number of storage slots. Crucially, the original single-task LoRAs and any training data are unavailable when a new LoRA arrives. Moreover, future task requests are a-priori unknown. Therefore, the device must continually update its set of LoRAs in a lightweight, data-free, and storage-aware manner, preserving existing capabilities while integrating new ones.

In this work, we formalize this setting as online continual merging of LoRAs under a storage budget, and introduce a novel method tailored to this challenge. Our approach identifies which existing adapter is most similar to the incoming LoRA, decides whether to allocate a new slot (until the storage budget is hit) or to merge adapters using an efficient data-free strategy (see Fig. 1 for an overview of our pipeline).

Our contributions are summarized as follows:

1. We introduce a new and practical setting for online continual merging of adapters in resource-constrained on-device LLMs under a storage budget.

2. We propose a lightweight, data-free merging strategy that identifies suitable adapters for merging using information from the history of merges and employs dynamic weighting to balance old and new capabilities.

3. We conduct extensive evaluation on real-world tasks representative of mobile device usage: our approach achieves strong performance under realistic constraints.

2 Related Work↩︎

Parameter-Efficient Fine-Tuning (PEFT): PEFT techniques adapt pre-trained models by training only a small subset of parameters [8], [9]. Among these, LoRAs [10] have emerged as a widely adopted approach. LoRA inserts two low-rank matrices into each target layer of the model and updates only these during fine-tuning, keeping the base model weights frozen. This yields strong performance while significantly reducing memory and compute overhead [22]. Subsequent work has focused on improving LoRA’s efficiency and expressiveness, e.g., AdaLoRA [23], Delta-LoRA [24], DoRA [25], VeRA [26], and Tied-LoRA [27]. However, all these methods assume a static task setting and do not address continual merging of adapters under resource constraints.

Model Merging: Model merging aims to combine multiple models or adapters trained for different tasks into a single multi-task model. Early approaches relied on simple weight averaging [18], [19], but more sophisticated methods have since emerged [28]–[32]. Two widely used techniques are TIES [20], which resets negligible parameters and resolves sign conflicts before merging, and DARE/DARE-TIES [21], which selectively drops and rescales parameter deltas. While many model merging techniques target full-model weight merging, recent works have focused on merging LoRA adapters specifically [15], [33]–[35], often enabling more modular and efficient adaptation. Other efforts address merging in continual learning scenarios [36], [37], particularly for discriminative tasks such as image classification [38], [39] and retrieval [40]. TIME [40] sequentially trains and merges multimodal experts using tailored initialization and merging strategies for continual integration. In contrast, OPCM [38] merges models without retraining by projecting updates orthogonally to prior ones, aligning better with our setting. However, its performance deteriorates as the number of merges increases. Our work instead focuses on on-device continual merging of LoRA adapters for generative text tasks, enabling dynamic integration of new functionality over time under strict compute and memory constraints.

On-device LLMs: Standard LLMs often consist of several billion parameters and require GPU-based training and inference [41]. Since most users lack access to such infrastructure, LLM services are commonly deployed via remote servers. However, privacy concerns arise when user data must be transmitted externally–particularly in applications involving personal conversations or sensitive content. As a result, there is growing interest in on-device LLMs to perform inference locally on mobile or edge hardware [11]. Applications of interest include summarizing private chats [42] or generating personalized responses [43], while ensuring data remains local. To meet the resource demands of mobile platforms, LLMs are typically downsized to 1–2B parameters and are often further optimized via quantization or pruning [44]. Recent open-source models designed for on-device deployment include Llama-3.2-1B [12] and Qwen-2.5-1.5B [45], [46]. These models deliver strong language capabilities within the computational and memory limitations of modern smartphones.

3 Problem Formulation↩︎

Training multi-task or multi-lingual LoRAs from scratch is impractical due to its lack of flexibility, as it requires access to all datasets and necessitates retraining whenever a new capability is added. Hence we assume only single-task LoRAs are sent to the device. While storage is limited, storing a small number of (possibly multi-tasking) adapters is typically feasible. Formally, we assume the availability of a storage budget for storing a collection of \(\leq K\) LoRAs on the device, \(\mathcal{L}=\{L_i\}_{i=1}^K\).

In the conventional static setting, where all single-task LoRAs \(L^{(t)}, \forall t\) are available simultaneously, multi-task adapters can be constructed by grouping LoRAs into \(K\) clusters based on task similarity and merging each group into a shared multi-task LoRA \(L_i, i=1,\ldots, K\).

In many real-world scenarios, however, LoRAs arrive incrementally over time, giving rise to the online continual merging setup considered here. As a result, single-task LoRAs are not accessible all at once, and merging must be performed iteratively, based solely on the incoming LoRA and the current set of stored adapters, as described next.

The merging happens on-device and not on the server, because it would be impractical and inefficient to store a copy of the deployed LoRAs on both the server and the device.

3.1 Online Continual Merging↩︎

At each discrete time step \(t>0\), the current on-device collection \(\mathcal{L}^{(t-1)}\) contains \(|\mathcal{L}^{(t-1)}|\leq K\) LoRAs, where \(|\cdot|\) indicates the cardinality of the set. A new single-task LoRA \(L^{(t)}\) arrives, trained to support a new user-selected task \(\tau_t\). The system must incorporate \(L^{(t)}\) into \(\mathcal{L}^{(t-1)}\) to obtain an updated set \(\mathcal{L}^{(t)}\) whose LoRAs preserve performance across all tasks (Fig. 1). We initialize \(\mathcal{L}^{(0)}=\emptyset\) and denote the set of all tasks seen at the end of step \(t\) by \(\mathcal{T}^{(t)}=\{\tau_i\}_{i=1}^t\).

A system tackling the online continual merging setup must:

1. Update the on-device collection whenever a new adapter arrives, by performing one of the following two actions:

   1. Selecting an adapter \(L_c \in \mathcal{L}^{(t-1)}\) to be merged with \(L^{(t)}\) and replace it in \(\mathcal{L}^{(t-1)}\) to obtain: \[\label{eq:merge951} \mathcal{L}^{(t)} = \{\mathcal{L}^{(t-1)} \setminus L_c\} \cup \{\mathrm{merge}(L_c, L^{(t)})\},\tag{1}\]

      where the \(\mathrm{merge}(\cdot)\) operation returns the merged LoRA and should be data-free and computationally-efficient, suitable for on-device execution (see Sec. 4.2);

   2. Allocating a new slot, i.e., adding the incoming LoRA to the current collection by: \[\label{eq:addition} \mathcal{L}^{(t)} = \{ L^{(t)} \} \cup \mathcal{L}^{(t-1)}.\tag{2}\] Note that this operation is only available if there is still storage available, i.e., if \(|\mathcal{L}^{(t-1)}|<K\).

2. Define a map \(\Theta^{(t)}\) such that, whenever an input sample \(x\) of task \(\tau_{i}\in\mathcal{T}^{(t)}\) arrives, it identifies the corresponding adapter’s identifier, \(\hat{c}=\Theta^{(t)}(i)\), to use for inference, \[\label{eq:gamma} \Theta^{(t)}\!:\!i\!\in\!\{\!1,\ldots,\!|\mathcal{T}^{(t)}|\!\}\!\to\!\hat{c}\in\!\{\!1,\ldots,\!|\mathcal{L}^{(t-1)}|\!\}.\tag{3}\] Then, the adapter \(L_{\hat{c}}\) is loaded from storage and plugged into the model to perform inference.

3. Maintain strong performance on all previous tasks in \(\mathcal{T}^{(t)}\).

3.2 Evaluation Protocol and Aggregate Score↩︎

In the considered experimental setting, each task \(\tau_i\) is characterized by a problem type and a language. To evaluate performance, we construct a benchmark spanning \(\alpha\) problem types and \(\beta\) languages, totaling \(\gamma= \alpha \cdot \beta\) distinct tasks for which we obtain \(\gamma\) single-task LoRAs. Tasks arrive at the device in random order, one per time step. At each time step \(t\), the system updates the collection of on-device LoRAs \(\mathcal{L}^{(t)}\), and the performance is measured on all the tasks seen so far \(\mathcal{T}^{(t)}\). In our setup, \(t=1,\ldots,\gamma\), however, \(\gamma\) is unknown to the device at any time \(t\).

Each task \(\tau_i\) is associated with a dataset \(\mathcal{D}_{\tau_i}\) and an evaluation metric \(M_{\tau_i}\) that takes as input a LoRA and provides as output the performance on the associated dataset. Since different problem types may be evaluated by different metrics that operate on different scales, we normalize performance on each task by comparing it to the performance of a single-task LoRA trained directly on that task. Let \(L\) denote a generic adapter. For each task \(\tau_i \in \mathcal{T}^{(t)}\), where \(t>0\), let \(M_{\tau_i}(L; \mathcal{D}_{\tau_i})\) be the score obtained by the LLM using \(L\), and \(M_{\tau_i}(L^{(i)}; \mathcal{D}_{\tau_i})\) the single-task performance, i.e., the score obtained by the LLM using \(L^{(i)}\). We define the normalized aggregate score \(S^{(t)}\) as follows:

\[\label{eq:score} S^{(t)} = \frac{1}{|\mathcal{T}^{(t)}|}\sum_{\tau_i\in \mathcal{T}^{(t)}} \frac{M_{\tau_i}(L; \mathcal{D}_{\tau_i})}{M_{\tau_i}(L^{(i)}; \mathcal{D}_{\tau_i})}.\tag{4}\]

This score quantifies how closely the performance of a generic LoRA \(L\) aligns with that of single-task LoRAs \(L^{(i)}\). In our experiments, we evaluate LoRAs \(L\! \in \! \mathcal{L}^{(t)}\); however, the proposed score formulation is generic and allows the evaluation of any given LoRAs \(L\). For instance, \(\mathcal{L}^{(t)}\) could contain either single-task LoRAs or merged LoRAs . Additionally, \(L\) is initialized to zeroes in the zero-shot case where no LoRAs are used.

3.3 Task Identification↩︎

In the settings we consider, it is typically easy to identify which task should be performed. The user can explicitly select the task or language, e.g., via UI elements. Alternatively, the task can be specified in the user prompt (e.g., “translate this sentence”, or “summarize this passage”). In such cases, determining the appropriate adapter is straightforward as we can identify the task via matching to selected keywords. When task identity is not directly stated, lightweight task classifiers can be used. Since routing is not related to our contribution, we model this step abstractly and assume access to a perfect classifier for simplicity. This assumption isolates the evaluation of the merging method itself without conflating it with the performance of a separate routing module.

4 Proposed Method↩︎

To address the challenges of online continual merging, we propose a lightweight, data-free method that supports on-device continual merging within a given storage budget. An overview is shown in Fig. 2. At the core of our method, there are two key components: 1) Similarity-based clustering (Sec. 4.1): we identify the most suitable stored adapter to merge with for an incoming LoRA, based on an efficient similarity metric without using any training data. 2) History-aware merging (Sec. 4.2): we merge adapters using a weighted combination scheme that leverages information from the previous adapter merges.

These components are integrated into a decision-making framework to determine whether the new LoRA should be merged into an existing adapter or allocated to a new slot (subject to the storage budget). We introduce two variants:

• K-Merge, a lightweight solution that merges each new LoRA with its nearest stored adapter unless there is space available to store it separately.

• K-Merge\(++\), an improved variant that selectively merges based on a similarity threshold, preserving space for future, more diverse LoRAs.

4.1 Cluster Assignment by Similarity Scoring↩︎

At the core of both variants is the ability to match the incoming LoRA \(L^{(t)}\) to the most compatible stored adapter among \(L_i \in \mathcal{L}^{(t-1)}, \forall i\) via similarity scoring.

Each LoRA \(L\) consists of a pair of low-rank update matrices \((A^{n, p}, B^{n, p})\) for each transformer layer \(n \in \mathcal{N}\) and projection type \({p \in \mathcal{P} = \{\texttt{key}, \texttt{query}, \texttt{value}, \texttt{output}\}}\). Then, we perform the following steps:

1. Flatten each update matrix \(\Delta W^{n,p}=B^{n,p}A^{n,p}\) into a vector via the flattening operation \(f(\cdot)\).

2. Compute the cosine similarity between the flattened vector corresponding to the incoming LoRA and that of all the stored LoRAs, and average over all layers and projections: \[\label{eq:sim} \text{sim}(L^{(t)}, L_i) = \frac{\sum\limits_{n\in \mathcal{N},p\in \mathcal{P}}{\text{sim}(L^{(t)n,p}, L_i^{n,p})}}{|\mathcal{N}| \cdot|\mathcal{P}|},\tag{5}\] \[\begin{gather} {\text{sim}(L^{(t)n,p}, L_i^{n,p})}\\ = \text{cos}(f(\Delta W^{(t)n,p}), f(\Delta W_i^{n,p})). \end{gather}\]

3. Determine the index \(c\) of the most-similar LoRA by: \[\label{eq:closest} c = \text{argmax}_{i=1,\ldots, |\mathcal{L}^{(t-1)}| } \text{sim}(L^{(t)}, L_i).\tag{6}\]

4.2 Iterative History-aware Model Merging↩︎

History Design: To retrieve the correct LoRA at the inference stage, we allocate some small storage (negligible in practice) to collect the time-varying history of merges as a map \(\mathcal{H}^{(t)}\) with up to \(K\) keys, each linked to a set of task indices. The history is initialized as \(\mathcal{H}^{(0)}=\emptyset\).

Save Incoming LoRA to Storage: Whenever an incoming adapter \(L^{(t)}\) is selected to be stored on the on-device collection, a new key is added to the history and the task index is added to the value set, as: \[\label{eq:simple95addition} \mathcal{H}^{(t)}[|\mathcal{L}^{(t-1)}|+1] = \{t\}.\tag{7}\]

History-aware Merging: In case the incoming LoRA is selected to be merged with the most similar adapter \(L_c\), we implement a history-aware merging. We implement the merge operation via a running average formulation, defined by:

\[\label{eq:merge} \mathrm{merge}(L_c,L^{(t)}) = \frac{L^{(t)} + |\mathcal{H}^{(t-1)}[c]| \cdot L_c}{|\mathcal{H}^{(t-1)}[c]| + 1} ,\tag{8}\] and the history of merges is updated accordingly as follows: \[\label{eq:history} \mathcal{H}^{(t)}[c] = \mathcal{H}^{(t-1)}[c] \cup \{t\}.\tag{9}\]

This formulation enables efficient incremental merging without requiring storage of all LoRAs in each cluster. A key benefit of the formulation is also that it is order-invariant as all the LoRAs have the same contribution to the merged multi-tasking LoRA.

4.3 K-Merge and K-Merge\(++\)↩︎

We now describe the two variants of our overall system, both designed to manage a fixed adapter budget of \(K\) slots. In contrast, existing merging methods would use only one slot, while naïve storing of all single-task LoRAs would use one slot for each LoRA.

K-Merge: The basic strategy merges \(L^{(t)}\) into its closest stored adapter \(L_c\) using Eq. 8 , unless fewer than \(K\) adapters are currently stored, in which case \(L^{(t)}\) is stored directly. This approach is simple and efficient, but has a key limitation: if early incoming LoRAs are similar, it may exhaust the storage budget prematurely by storing redundant adapters, thus leading to a sub-optimal clustering of adapters at later stages.

K-Merge\(++\): To mitigate this issue, we introduce a more advanced approach relying on a similarity threshold \(s\). A new LoRA \(L^{(t)}\) is merged into \(L_c\) only when they are sufficiently similar to each other, i.e., \(\text{sim}(L^{(t)}, L_c) \geq s\). Otherwise, it is stored separately if storage is still available. Once the storage is full, the merging is performed regardless of \(s\). The threshold \(s\) is estimated empirically from an auxiliary dataset of LoRAs trained on held-out tasks. Specifically, we compute all pairwise similarities across LoRAs and set \(s\) to the median. This variant preserves storage for more diverse future LoRAs, improving adaptability across diverse deployment scenarios. Its only overhead is the need to estimate \(s\). The procedure is detailed in Algorithm 3.

4.4 Inference Stage↩︎

We implement the mapping described in Eq. 3 as an inverse function of the history \(\mathcal{H}^{(t)}\). To process a query input sample \(x\) belonging to task \(\tau_i \in \mathcal{T}^{(t)}\), we look for the key \(\hat{c}\in\{1,\ldots,|\mathcal{L}^{(t-1)}|\}\) whose value set contains \(i\), i.e., \(i \in \mathcal{H}^{(t)}[\hat{c}]\). Then, we load the adapter \(L_{\hat{c}}\) from storage, plug it into the model, and perform inference.

5 Experimental Evaluation↩︎

5.1 Experimental Details↩︎

Tasks and Metrics: We evaluate our method across a diverse set of tasks, each defined by a combination of a problem type and a target language, and each associated with a specific evaluation metric. We consider \(\alpha = 5\) problem types: Smart Reply, Summarization, Tone Adjustment, Question Answering, and Grammar Correction. Each of these problem types is instantiated in \(\beta = 8\) languages: English, Spanish, French, German, Italian, Korean, Japanese, and Chinese. As metrics, we report F-0.5 for Grammar Correction, F-1 for Question Answering, weighted ROUGE [47] for Smart Reply, and ROUGE-L for both Text Summarization and Tone Adjustment.

Datasets: We use established benchmarks for each task: 1) Smart Reply: Persona-Chat Synthetic [43]. 2) Summarization: SAMSum [42]. 3) Tone Adjustment: Sound Natural [48] modified via a publicly available model fine-tuned for tone adjustment [49]. 4) Question Answering: SQuAD [50]. Datasets for these four problem types are released in English, and we translated them into the other languages via the OPUS-MT model [51] for Spanish, German, French, and Italian, and via the M2M100 model [52] for Korean, Chinese, and Japanese. 5) Grammar Correction: original-language datasets are critical here, as machine translation systems tend to correct grammatical errors [53]. We use Write & Improve for English [54], Merlin for Italian [55], ECSpell for Chinese [56], and the GitHub Typo Corpus [57] for the other languages.

To calibrate the similarity threshold used in K-Merge\(++\), we reserve a set of held-out tasks that include unseen problem types and languages. Specifically, we use: (i) Translation to English and Title Generation as novel problem types, and (ii) Portuguese, Turkish, and Serbian as novel languages. TED Talks [58] serves as the dataset for translation, and XLSum [59] for title generation.

All datasets are split into training, validation, and test partitions; see the Appendix for details.

Models: We experiment with two lightweight, instruction-tuned models designed for on-device deployment: Llama-3.2-1B-Instruct and Qwen-2.5-1.5B-Instruct.

Baselines: We compare our solutions with several alternative approaches. We include zero-shot and single-task LoRA as lower and upper performance boundaries. We also compare to standard model merging approaches (Linear, TIES, DARE, DARE-TIES) and continual merging (OPCM). All merging methods are adapted to our online setting by applying them to the pair consisting of the incoming LoRA \(L^{(t)}\) and the most similar on-device LoRA \(L_c\). Task descriptions are always provided via prompts (see the Appendix for examples).

Hyperparameters: All LoRAs are trained using AdamW with a learning rate of 5e-5, dropout rate of 0.05, and a mini-batch size of 3. LoRAs use rank 32 and scaling factor \(\alpha = 128\). Linear merging uses equal weights (i.e., 0.5) for pairwise merging, while TIES, DARE, and DARE-TIES use unary weights and density 0.5. The similarity threshold is set using the held-out LoRA set to \(s = 0.020\) and \(s=0.028\) for Llama-3.2-1B and Qwen-2.5-1.5B models, respectively. In our experiments, the number of LoRAs that can be stored on-device, \(K\), varies from 1 to 8. In total, we simulate the sequential arrival of \(\gamma=40\) LoRAs. All results are averaged over 3 random permutations of LoRA arrival ordering (unless otherwise stated).

5.2 Experimental Results↩︎

Results with Random Task Ordering: Fig. 4 illustrates the average score of the methods for the two LLMs at varying numbers of clusters over three random task orderings. Additionally, we plot zero-shot and single-task LoRA performance as a reference. First, we note the great benefit of LoRA fine-tuning for all LLMs, as indicated by the improvement of single-task LoRAs over zero-shot performance. Second, we observe DARE, DARE-TIES, and OPCM are unable to offer competitive performance, being significantly lower than single-task LoRAs and other approaches, although the corresponding performance gap shrinks slightly as the number of clusters increases. TIES generally outperforms these methods; however, the Linear merge proves to be the most effective baseline. Remarkably, our proposed methods outperform Linear at all storage budgets. In particular, with Llama-3.2-1B our method scales better than competitors for small numbers of clusters. With Qwen-2.5-1.5B the gain is stable across different values of \(K\).

Finally, K-Merge\(++\) improves the performance of K-Merge even further by utilizing the similarity threshold mechanism, especially for larger \(K\), where there is more scope for more distinct multi-tasking LoRAs. However, the main benefit of K-Merge\(++\) is robustness against worst-case task orderings, as we analyse later. Using our methods, the overall score reaches about 80-90% of the single-task performance with just \(8\) clusters, showing the usefulness of our solution.

Robustness to Task Ordering: We investigate the robustness of our methods to different task orderings in Table ¿tbl:tab:method95vs95k?. In particular, we compare the average score obtained via random task ordering (Random) against a clustering of LoRAs done according to problem types (Problem Types) and an ordering that we empirically found to lead to low performance for our setup (Worst). In particular, in the latter, we force LoRAs of the same problem type to appear consecutively. We remark that we could only test Problem Types for \(K\!= \! \alpha \! = \! 5\) since we allocate one problem type for each on-device LoRA cluster. Additionally, evaluating K-Merge\(++\)on the Problem Types case is not meaningful, as the clustering is pre-determined and the similarity threshold is therefore not employed.

First, the scores for the Problem Types case are higher than for Random, being a favorable case where similar LoRAs are merged together (see Appendix Fig. 8). Nonetheless, this ordering makes three unrealistic assumptions: (i) the device knows a priori all future problem types; (ii) the device has exactly \(K=\alpha\) on-device LoRA storage slots; and (iii) LoRAs always have the highest similarity with other LoRAs from the same problem type rather than from different ones. In practice, these assumptions are difficult to meet, hence motivating the usefulness of our approach. Even more, K-Merge outperforms the baselines also in this setup.

Second, under the Worst case, we observe K-Merge\(++\)  in particular significantly outperforms K-Merge and shows strong robustness thanks to its threshold mechanism. Linear and TIES degrade significantly. This setting complicates the initialization of empty slots, often causing the system to fill multiple slots with similar LoRAs (e.g., targeting the same problem type across different languages). These results highlight the robustness of our K-Merge\(++\) approach in handling a-priori unknown and adversarial task orderings on the device side.

Varying Timestep: Fig. 5 shows that K-Merge initially outperforms K-Merge\(++\), but is quickly overtaken as the latter benefits from saving storage slots at early stages thus having them available when more diverse LoRAs arrive. Linear matches our methods at initial timesteps, but is then consistently surpassed as more adapters arrive.

Efficiency: K-Merge  is lightweight and fast: the integration of a new incoming LoRA takes between 0.04s and 0.18s when there are 2 to 8 stored multi-tasking LoRAs, demonstrating its practicality for on-device deployment. K-Merge\(++\)  exhibits comparable runtime. Since memory is dominated by text generation, our approach adds only negligible extra memory overhead. Full details are presented in the Appendix.

Storage Analysis: We report the number of parameters and storage used per adapter in Table ¿tbl:tab:num95params?. Storing, e.g., 5 adapters is lightweight, consuming 135-170MB of storage in total. Storing all 40 adapters would require about 1GB of storage, a rather substantial amount.

Similarity Metric: We have evaluated different similarity measures, with cosine similarity (i.e., our selected option) giving the best performance. In particular, we have compared it with the \(L^1, L^2\) and \(L^{\infty}\) measures. In our experiments, the cosine similarity yielded the best performances, as can be seen from Table 1. It shows how our approach achieves \(S^{(\gamma)}\) scores of 0.75 and 0.78 with \(K=3\) and \(K=5\), respectively. The alternative similarity measures led to significantly lower scores than cosine similarity.

History-Aware Merging: Note that the Linear approach combined with history-aware merging corresponds to our K-Merge. For this reason, we also apply similarity-based clustering to each merging approach that we report in the results. We have conducted additional experiments that integrate history-aware merging into the diverse baselines. The results in Table 2 show that they obtain worse performance than our solution (linear with history-aware merging).

Further Analyses: We include further analyses studying LoRA similarity, similarity threshold, clustering consistency and out-of-domain LoRA performance in the Appendix. The analyses show, for example, there is more similarity between LoRAs from the same problem type rather than from the same language.

6 Conclusion↩︎

We introduced the novel task of on-device continual LoRA merging, where incoming adapters must be merged over time due to storage constraints, with the goal of supporting new capabilities while preserving existing ones. We proposed two methods: K-Merge, which merges incoming adapters with their closest stored counterpart when storage is full, and K-Merge\(++\), which uses a similarity threshold to decide between merging or allocating a new slot. Both methods have been evaluated on 40 tasks across 5 domains and 8 languages, achieving strong performance and high efficiency, while requiring no data. Average performance is similar, but K-Merge\(++\)is notably more robust to the task arrival order.

Limitations↩︎

We restrict our study to LLMs, leaving extensions to multimodal tasks for future work. We only studied LoRA adapters as it is the current industry standard, but other adapter types could be considered. We evaluated LLMs that are in sizes suitable for on-device deployment as that is the desired use case. Our solution could also be evaluated for larger LLMs, which would, however, require significantly larger amount of compute.

Ethical Considerations↩︎

Our work focuses on text generation, which is an area where ethical considerations are important due to the potential for large-scale real-world impact. Our work in particular focuses on efficiency and as a result it does not introduce new capabilities that could be misused. However, merging of adapters can lead to weakening of safeguarding mechanisms. As a result, it is important to conduct experiments that explore how effective earlier safeguarding mechanisms remain before deploying the solution.

This document provides additional material that could not be included in the main paper due to space constraints. Additional details for the employed datasets, prompts, and hardware are reported in Appendix 7. Detailed results for single-task LoRAs and continual merging are reported in Appendix 8. Additional analyses describing computation time overhead of our approach, cosine similarity histograms, clustering consistency and out-of-domain performance are presented in Appendix 9.

7 Additional Details↩︎

7.1 Dataset Statistics↩︎

Table ¿tbl:tab:datasets? shows the number of training, validation, and test samples for the various problem types and languages.

7.2 Prompts↩︎

Table ¿tbl:tab:prompts95full? shows the prompts that have been used for the different problem types, which are taken from [15]. Different tones have separate prompts, which are also detailed in Table ¿tbl:tab:prompts95full?.

7.3 Hardware and Software Details↩︎

For all our experiments, we have used Python 3.9.21 with PyTorch 2.6.0+cu124. We have run our experiments on an Ubuntu 20.04.6 LTS machine with kernel version 5.4.0-205-generic, equipped with 10 NVIDIA A40 GPUs, and two Intel Xeon Gold 5218 processors (16 cores per socket). Note that the continual merging approach (i.e., our contribution) is very lightweight and fast, the hardware resources have been used mostly to train the LoRAs and perform inference.

8 Detailed Results↩︎

8.1 Single-task LoRA Performance↩︎

Table 3 shows the performance of zero-shot (i.e., without any LoRA adapter) inference and the ones of the single-task LoRA for all combinations of problem types and languages. It is possible to notice a large gap between the two options as using LoRAs typically doubles the score. Qwen-2.5-1.5B has better performances than Llama-3.2-1B, possibly due to the larger number of parameters. The scores vary across languages, with English outperforming the others. We provide a similar analysis in Table 4 for the additional languages and problem types used in the held-out set. Note that this data has been used only to select the similarity threshold \(s\) in the K-Merge\(++\) version of the approach.

8.2 Detailed Continual Merging Results↩︎

We analyzed our continual merging experiments in Fig. 4 in the main text, while in this section, we report the corresponding results in more detail, including the mean and standard deviation across three random task orderings in Table 5 and Table 6. The first table shows the results for the Llama-3.2-1B model, while the second shows them for the Qwen-2.5-1.5B model.

9 Additional Analyses↩︎

9.1 Integration Time of Incoming LoRAs↩︎

Fig. 6 shows the time that it takes to integrate a new incoming LoRA using K-Merge. The time is a function of the number of stored LoRAs (clusters), and it scales roughly linearly with the number of clusters. The proposed approach is lightweight, as the time varies between 0.04 and 0.18 seconds. The time required by K-Merge\(++\) is similar to K-Merge because all additional operations take negligible time. Memory usage is dominated by text generation.

9.2 Cosine Similarities Analysis↩︎

Fig. 7 shows the histograms of LoRA cosine similarities for text correction (TC) and question answering (QA) with respect to 1) the other problem types in the same language, 2) the same problem type in different languages. We use the Llama-3.2-1B model and English as the reference language for this analysis. The results indicate that there is less similarity between different problem types in the same language than for the same problem type in different languages. Fig. 8 reports the cross-task LoRA similarity, \(\mathrm{sim}(L^{(i)},L^{(j)}), \forall i,j=1,\ldots,\gamma, i\neq j\), across both tasks and languages. We observe that LoRAs tend to naturally cluster per problem type (red solid boxes), while similarity per language is weaker. Nonetheless, European languages display higher similarity between themselves rather than to Asian ones, as expected. We remark that the system is unaware of the overall task sequence at any time and, therefore, could not perform clustering based on this information.

9.3 Clustering Consistency (at \(K\!=\!\alpha\))↩︎

To further confirm that LoRAs tend to cluster per problem type, we take the history of merges and analyse the frequency of problem types within each cluster. Specifically, for each cluster, we identify the most common problem type and compute the fraction of items in that cluster that belong to that problem type. The overall clustering consistency is then calculated as the total number of items matching their cluster’s dominant problem type, divided by the total number of items. We restrict this analysis to the case where the number of on-device LoRA slots equals the number of problem types, i.e., \(K=\alpha=5\).

Linear has the lowest clustering consistency (82.5%). K-Mergehas a higher score (83.3%) which is further improved by K-Merge\(++\)(88.3%, which corresponds to the highest result). Interestingly, TIES has a high clustering consistency (86.7%); however, it underperforms other approaches.

9.4 Similarity Threshold↩︎

Table ¿tbl:tab:ablation95s? shows the performance for different values of the threshold \(s\). We selected \(s=0.02\) as it is the median value of pairwise similarities across LoRAs on held-out tasks, but it also performs the best compared to other options.

9.5 Larger Models↩︎

In Table 7, we report the performance of the compared methods using Llama-3.2-3B model (instruction tuned) as the base model, confirming that our approach outperforms the competing approaches also when employing larger models.

9.6 Out-of-Domain Performance↩︎

We analyse out-of-domain generalization performance of each English LoRA to other problem types in Fig. 9. The scores are normalized with respect to the ones of using the corresponding problem-type LoRA (i.e., the problem-type LoRA performance corresponds to a score of 1.0). Problem type pairs such as tone adjustment and text correction, or summarization and question answering, share more similarity and have relatively high scores, while, for example, reply and tone adjustment, or reply and text correction have low scores.

References↩︎