1 Introduction↩︎

Language Models (LMs) have been extensively utilized in a variety of Natural Language Processing (NLP) tasks, including legal advise [1], scientific research [2], etc. Despite the belief that pre-trained LMs such as ChatGPT [3] and Llama [4] have exhibited the rudiments of Artificial General Intelligence (AGI), their data-driven nature results in sub-optimal performance in specialized domains [5]. To address this issue, the general paradigm for tailoring LMs to downstream tasks consists of two steps: pre-training and fine-tuning. The pre-training process aims to learn rich language features and structures from a massive corpus in an unsupervised way, forming the pre-trained models capable of mastering general language patterns. Consequently, these pre-trained models can serve as a remarkable starting point and further fine-tuned on vertical domains in a supervised way, resulting in the specialized models that are adept at diverse downstream tasks. For example, codeLlama [5] and AstroLLaMA [6] are the fine-tuned variants of Llama-2 [4] on programming domain and astronomy domain, respectively.

The performance of the fine-tuned specialized LMs hinges on two key factors: the fine-tuning algorithms and the fine-tuning datasets (see 1). Given the current scale of parameters in LMs, the computational cost of full-parameter fine-tuning is prohibitively expensive (e.g., a 16-bit full-tuning for Llama-7B requires 60GB of GPU memory [7]). This limitation has spurred the rapid development of Parameter-Efficient Fine-Tuning (PEFT) [8]. Specifically, Low-Rank Adaptation (LoRA) [9] is the state-of-the-art (SOTA) Parameter-Efficient Fine-Tuning (PEFT) framework with the highest usage. By the end of 2023, there were over 12,000 LoRA models on Hugging Face, with some receiving over one hundred thousand downloads per month [10]. Since LoRA only trains side-loaded rank-decomposition matrices while keeping the model backbones frozen, it only needs 16GB to fine-tune Llama-7B, which can be further reduced to 6GB through its 4-bit quantization version qLoRA [11]. Moreover, a high-quality fine-tuning dataset is also paramount as it serves as the specialized knowledge source and determines the upper limit of the model’s capability. Considering the potential presence of privacy-sensitive information within fine-tuning datasets, such as those in financial and medical domains, a comprehensive assessment for privacy leakage associated with fine-tuning datasets is of vital importance.

Membership Inference Attacks (MIAs) [12], in which attackers aim to determine if a specific sample was part of the training data of a target model, pose a persistent privacy threat to machine learning models. Recently, with the emergence of an increasing number of specialized LMs in the open-source model zoo (e.g., Huggingface²), conducting MIAs against the fine-tuning datasets of the LoRA-based fine-tuned LMs has become a prominent research focus. Since LoRA only fine-tunes a small subset of the model’s parameters, recent studies suggest that their fine-tuning datasets are invulnerable to MIA  [13], [14]. However, the current research fails to recognize that the publicly accessible pre-trained model could introduce additional privacy threats.

We propose LoRA-Leak, a holistic evaluation framework for measuring the vulnerability of LoRA-based fine-tuned LMs against MIAs. To comprehensively assess the privacy risks of the LoRA fine-tuning dataset, we formulate three Research Questions (RQs):

• RQ1: Is MIA still a serious privacy threat for LoRA-based fine-tuned LMs?

• RQ2: Can incorporating pre-trained model information lead to the design of more potent MIAs?

• RQ3: What LoRA fine-tuning strategies can mitigate the threat of MIAs?

To answer RQ1, we propose LoRA-Leak, a comprehensive framework for MIA against LoRA finetuning, incorporating fifteen diverse attack methods (see 1). Subsequently, we assess the effectiveness of MIAs in LoRA-Leak against nine LoRA fine-tuned LMs, developed using three widely used LMs and three practical fine-tuning datasets under a conservative setting to prevent overfitting. Our experimental results demonstrate that LoRA-Leak can achieve high AUC scores against LoRA-based fine-tuned LMs. For instance, the AUC scores against Llama-2 model fine-tuned on AG News [15], OAsst [16], and MedQA [17] are 0.765, 0.721, and 0.775, respectively.

To demonstrate the necessity of introducing pre-trained models for answering RQ2, we compare the effectiveness of different MIAs the performance of various MIAs with and without using the pre-trained model as a reference. We observe that the calibration from the pre-trained model can consistently amplify the privacy risk (see 2). For a more in-depth analysis, we discuss the impact of different kinds of reference models [12]. As 4 shows, introducing other kinds of reference models cannot achieve the optimal attack results as introducing pre-trained models.

To address RQ3, we comprehensively discuss various fine-tuning settings. We first analyze the influence of fine-tuning hyperparameters on the attack effect, such as the fine-tuning epoch and the selection of LoRA fine-tuning modules. Furthermore, we discuss four potential defenses. We first explore three traditional defense strategies, i.e., dropout, weight decay, and differential privacy (DP), in which only dropout can mitigate the risk of MIAs while preserving utility. Furthermore, in 6.4, we demonstrate that fine-tuning excluding specific modules can also mitigate privacy risks.

In summary, our contributions are as follows:

• We introduce LoRA-Leak, a comprehensive evaluation framework on MIAs against LoRA-based fine-tuned LMs.

• We propose that introducing pre-trained models into the inference attacking pipeline can effectively amplify the privacy risks.

• We explore four defenses and find that dropout and fine-tuning excluding specific layers can mitigate the threat of LoRA-Leak.

2 Preliminaries↩︎

2.1 Causal Language Model (CLM)↩︎

A causal language model (CLM) is designed for next-token prediction tasks over a token space \(\mathcal{T}\). It first takes a sequence of tokens \(x_{1:n} \in \mathcal{T}^n\) as input and transforms each token \(x_i\) into a continuous embedding \(\boldsymbol{e}_i\). These embeddings are then fed into a decoder-only transformer \(\mathcal{M}\), which produces the probability of each token \(x_{n+1} \in \mathcal{T}\) being the next token, i.e., \(\boldsymbol{p}_n = \langle \Pr(x_{n+1}|x_{1:n})\rangle_{x_{n+1}\in \mathcal{T}}\).

The primary purpose of such a model is to generate reasonable text completions for user inputs by repeatedly performing next-token predictions. This is achieved by training the model on a collection of token sequences in an autoregressive manner: For each training sample \(x_{1:n} \in \mathcal{T}^n\), the objective is to minimize the model’s perplexity (PPL) on that sequence, defined in the form of average cross-entropy loss as \[\label{ppl} \mathcal{L}(x_{1:n};\mathcal{M})=-\frac{1}{n-1}\sum_{i=1}^{n-1}\log{p_{i,x_{i+1}}}.\tag{1}\]

2.2 Low-Rank Adaptation (LoRA)↩︎

Low-Rank Adaptation (LoRA) is one of the most widely used Parameter-Efficient Fine-Tuning (PEFT) algorithms for model fine-tuning [10]. For a pre-trained model \(\mathcal{M}_{pt}\), this method selects only a subset of layers for fine-tuning. For each selected layer, it freezes the pre-trained weight \(W_i \in \mathbb{R}^{d \times k}\) and introduces two additional decomposition matrices \(\Delta_i = (A_i, B_i) \in \mathbb{R}^{d \times r} \times \mathbb{R}^{r \times k}\) to fine-tune, where \(r \ll \min (d,k)\) is the hyperparameter of rank. In the resulting fine-tuned model \(\mathcal{M}_{ft}\), its layer is then represented as: \[\label{eq:lora} W'_{i} = W_{i} + A_i B_i.\tag{2}\]

2.3 Membership Inference Attack↩︎

Membership inference (MI) is a privacy game where an adversary \(\mathcal{A}\), given access to a machine learning model \(\mathcal{M}\), aims to determine whether a specific record \(x\) is part of the model’s training dataset \(\mathcal{D}\), i.e., \(\mathcal{A}(x;\mathcal{M})\rightarrow \{0,1\}\). The adversary wins if and only if \(\mathcal{A}(x;\mathcal{M})=\mathbb{I}[x\in\mathcal{D}]\). To achieve this, the adversary will choose a score function \(\mathcal{S}(x;\mathcal{M})\rightarrow \mathbb{R}\) and a threshold \(\tau \in \mathbb{R}\). Finally, the adversary determines membership based on the rule \(\mathcal{A}(x;\mathcal{M})=\mathbb{I}[S(x;\mathcal{M})>\tau]\).

3 Related Work↩︎

3.1 MIAs against Neural Networks↩︎

Membership inference attack (MIA) was initially proposed to detect the training sample of image classification tasks in artificial neural networks [27]. It utilizes shadow datasets to train multiple shadow models as proxies to mimic the behavior of the target model. The prediction results of these shadow models are then collected to train a binary classifier, which learns the characteristics of membership samples. Nasr et al. [28] extend this approach to the white-box setting, where the model’s internal gradients are also available. Based on this additional feature, the classifier could achieve a more accurate result. However, as the model’s complexity increases, it becomes impractical to select effective and learnable features for training the classifier.

Another approach is to identify an explicit intermediate signal from the model to determine membership. For example, using the prediction correctness [29] or prediction confidence [30] as a metric for membership. The Likelihood Ratio Attack (LiRA) [12] is one of the most effective metric-based MIAs, which calibrates the membership signal by comparing the prediction confidence between the target model and the shadow models. We extend the idea to develop MIAs against fine-tuned language models.

Despite proposing various attacks, there is also theoretical research on the mechanics of membership inference. Yeom et al. [31] highlight that the primary threat of membership inference is due to overfitting. Additionally, Bentley et al. [32] further quantitatively associate the risk of membership inference with the target model’s generalization gap. Therefore, we report the model’s generalization gap to measure its inherent vulnerability to membership inference.

3.2 MIAs against Pre-trained Language Models↩︎

With the advent of Pre-trained Language Models (PLM), the membership inference attacks against their pre-training corpora are surpassing. Earlier research primarily adapted from the MIAs against neural networks, such as the LOSS attack [18] and the LiRA attack [19]. Recently, several attacks that specifically exploit the characteristics of LMs have been proposed. For example, the difference of token probabilities inspires Min-K% [24] and Min-K%++ [25], while MoPe [23] exploits the smoothness around the training points. However, the credibility of these attacks is hindered by their faulty evaluation method [33]. Because PLMs lack transparency regarding their training corpus, these works use corpora from before and after the model’s knowledge cut-off date as a proxy for ground truth. Consequently, there is a distribution shift between members and non-members, allowing even classifiers without access to the model to achieve high accuracy [34]. As for the benchmark with a fair membership split, these attacks degrade to a nearly random performance against PLMs [35]. Therefore, we explore whether those attacks against PLMs can be effective against the fine-tuning corpus in LoRA fine-tuning under a fair membership split.

3.3 MIA against Fine-tuned Language Models↩︎

There are also MIAs against LLM’s fine-tuning samples. The neighborhood attack [21] calibrates the loss signal by the average loss of rephrased samples. MIA-SPV [22] further enhances this attack with LiRA by comparing the calibrated loss from the target model with the calibrated loss from a self-prompted shadow model. Although this attack has been reported to achieve good performance, it is important to note that their target models were trained for 10 epochs, making them inherently vulnerable to membership inference (See 5.2). Consequently, many MIAs could achieve similar performance, rendering the advantage of this method unclear. Moreover, this attack is computational-intensive for generating self-prompt samples.

Although existing MIAs against fine-tuned LLMs are delicate and promising, they often overlook the significance of the pre-trained model. The reference attack [36], a simple yet effective approach, draws inspiration from the LiRA attack by using the pre-trained model as a shadow model to calibrate the loss signal of the target model. This suggests that referencing pre-trained models could further amplify the privacy risks associated with LoRA fine-tuned LMs. Therefore, we extend this concept to fill the gaps in all existing MIAs by calibrating their signals with their pre-trained model.

LoRA has already been the most used language model fine-tuning algorithm [10], but the studies for its privacy risks are still incomplete. Wen et al. [13] reported that LoRA fine-tuning is invulnerable to MIAs. However, their work only employs the LiRA attack [19], which may not fully reveal the MIA risks of LoRA fine-tuning. In our work, we comprehensively explore the privacy risks evoked by LoRA with fifteen MIAs against different fine-tuning settings and defense strategies.

Recently, Liu et al. [14] proposed PreCurious, a framework designed to amplify membership inference risks in fine-tuned language model by poisoning its pre-trained model. However, their threat model is strong because the victim must use the corrupted model provided by the adversary. In this work, we assume the victim uses the official open-source pre-trained model that is also accessible to the adversary. This is a more practical threat model in LoRA fine-tuning scenarios that still significantly amplifies privacy risks.

4 LoRA-Leak↩︎

In this section, we introduce LoRA-Leak, a comprehensive framework for MIAs against LoRA fine-tuning. We begin by outlining the threat model of LoRA-Leak. Next, we present a systematic taxonomy that identifies the essential components of MIAs to categorize all existing MIAs within this framework. This framework proposes a neglected attacking surface that utilizing the pre-trained model as a reference could further enhance existing MIAs.

4.1 Threat Model↩︎

In our threat model, the victim is a model fine-tuner aiming to build a specialized CLM \(\mathcal{M}_{ft}\) for a downstream task using their private dataset \(\mathcal{D}_{ft}\). To achieve this, the victim first obtains a renowned PLM \(\mathcal{M}_{pt}\) that is publicly released by a benign party, such as OpenAI’s GPT-2 or Meta’s Llama-2. Subsequently, the victim fine-tunes \(\mathcal{M}_{pt}\) on \(\mathcal{D}_{ft}\) using LoRA to get the resulting model \(\mathcal{M}_{ft}\). Finally, the victim publicly releases the fine-tuned LoRA model \(\mathcal{M}_{ft}\) in a model zoo such as Hugging Face.

The adversary’s goal is to infer whether a record \(x\) belongs to the fine-tuning dataset \(\mathcal{D}_{ft}\) of the target model \(\mathcal{M}_{ft}\). Note that the adversary aims to infer the membership in the victim’s private fine-tuning dataset \(\mathcal{D}_{ft}\), rather than the pre-training dataset that is used to train \(\mathcal{M}_{pt}\).

The adversary has full knowledge of the final fine-tuned LoRA model \(\mathcal{M}_{ft}\) but does not know its fine-tuning details such as the fine-tuning hyperparameters. The attacker also does not know any information about the fine-tuning or pre-training datasets, even their domains. However, we additionally assume the adversary has full knowledge of the pre-trained model from which the target model was fine-tuned. This assumption is based on the fact that LoRA models must be used with their pre-trained model. Consequently, the name of the pre-trained model is typically specified in the target LoRA model’s metadata or model card, allowing the adversary to effortlessly obtain this PLM by referencing the name.

We assume that the adversary possesses sufficient GPU resources for model inference and backpropagation. Since the adversary fully possesses the fine-tuned model and its pre-trained model, that means they can self-host these models. As a result, the adversary has white-box access to the models, including all internal states during inference, such as sample loss, predicted token probabilities, and gradients, etc. However, the adversary cannot interfere with the pre-training and fine-tuning process, nor can they poison the victim’s pre-trained model or fine-tuning dataset, as these actions typically require sophisticated supply-chain attacks. Some literature [22] proposed a gray-box scenario, where the adversary is limited to access partial internal states, such as the loss of the sample or prediction probabilities of each token. While these attacks appear to operate in a more constrained scenario, current inference APIs, such as OpenAI’s Developer Platform³ and Hugging Face’s Inference Endpoints⁴, do not provide any internal states that aligns their assumption. Therefore, we focus on the white-box scenario to fully expose the threat of membership inference in a passive setting.

4.2 Holistic Framework for MIAs against LMs↩︎

As discussed in 2.3, the essence of MIA lies in selecting an appropriate score function \(\mathcal{S}(x;\mathcal{M})\) that effectively differentiates between members and nonmembers. Here, we identify the key components involved in the design of \(\mathcal{S}(x;\mathcal{M})\), including Intermediate States, Augmented Perturbations, and Referenced Calibrations. We will demonstrate how this holistic framework can encompass existing MIAs and lead to our proposed enhancements.

To calculate \(\mathcal{S}(x;\mathcal{M})\), the text sample \(x\) is fed into the fine-tuned model \(\mathcal{M}\) for forward or backpropagation. During this process, the adversary can collect various internal states as their knowledge base for initiating the attack. As illustrated in 2, there are four internal states that correlate with the sample’s membership status. The loss of sample is denoted as \(\mathcal{L}(x;M)\). Empirically, members tend to have smaller loss value than nonmembers, leading to the well-known LOSS attack [18], whose score function is defined as \[\label{loss} S_{loss}(x;\mathcal{M})=-\mathcal{L}(x;M).\tag{3}\] The predicted next-token probabilities for position \(i\) are denoted as \(\boldsymbol{p}_{i}=\langle \Pr(x_{i+1}|x_{1:i})\rangle_{x_{i+1}\in \mathcal{T}}\), which represents the model’s confidence that each token being the next-token given on all previous tokens. The Min-K% [24] attack utilizes the fact that the model is less likely to predict words in the membership sentence with low probabilities. Therefore, this attack selects K% of tokens with the lowest predicted probabilities and calculates the score function as the average log likelihood of these selected tokens, i.e., \[\label{mink} \mathcal{S}_\text{Min-K\%}(x;\mathcal{M})=-\frac{\sum_{x_{i+1}\in \text{Min-K\%}(x)}\log p_{i,x_{i+1}}}{|\text{Min-K\%}(x)|}.\tag{4}\] The Min-K%++ [25] attack further utilizes the probability of the nonmember tokens, resulting in the score function \[\label{minkpp} \mathcal{S}_\text{Min-K\%++}(x;\mathcal{M})=-\frac{\sum_{x_{i+1}\in \text{Min-K\%}(x)}\frac{\log p_{i,x_{i+1}}-\mu({\log\boldsymbol{p}_i})}{\sigma(\log\boldsymbol{p}_i)}}{|\text{Min-K\%}(x)|}.\tag{5}\] Another internal state is the gradients of the fine-tuned model with respect to the sample loss, i.e., \({\partial \mathcal{L}}/{\partial \Delta}(x)\). Because these gradients tend to be smaller for members, Wang et. al. [26] proposed using the norm of these gradients as the score function, i.e., \[\label{gdt} \mathcal{S}_{\text{GradNorm}_\theta}(x;\mathcal{M})=-\Vert {\partial \mathcal{L}}/{\partial \Delta}(x) \Vert.\tag{6}\] They also suggested that the gradients on the input embeddings (\({\partial \mathcal{L}}/{\partial \boldsymbol{e}}\)) could serve as an approximation of \({\partial \mathcal{L}}/{\partial \Delta}\), resulting in the score function \[\label{gdx} \mathcal{S}_{\text{GradNorm}_x}(x;\mathcal{M})=-\Vert {\partial \mathcal{L}}/{\partial \boldsymbol{e}}(x) \Vert.\tag{7}\]

In addition to performing standard forward and backward propagation, the adversary may introduce augmented perturbations into the pipeline to collect internal states of nonmembers. There are two approaches to introduce perturbations, as represented by the dashed line in 2. The neighborhood attack [21] perturbs the sample \(x\) into \(N\) paraphrased samples \(\tilde{x}_1, \dots, \tilde{x}_N\) by replacing random words with predicted ones using mask-filling models like BERT [37] or T5 [38]. The losses of these paraphrased samples are then collected to calculate the score function as \[\label{nei} \mathcal{S}_{\text{Nei}}(x;\mathcal{M})=\frac{1}{N}\sum^N_{i=1} \mathcal{L}(\tilde{x}_i;\mathcal{M})-\mathcal{L}(x;\mathcal{M}).\tag{8}\] The MoPe attack [21] adds Gaussian noise to the model parameters, resulting \(N\) perturbed models \(\tilde{\mathcal{M}}_1,\dots, \tilde{\mathcal{M}}_N\). The sample’s loss on the perturbed models are then collected to calculate the score function as \[\label{gd} \mathcal{S}_{\text{MoPe}}(x;\mathcal{M})=\frac{1}{N}\sum^N_{i=1} \mathcal{L}(x;\tilde{\mathcal{M}}_i)-\mathcal{L}(x;\mathcal{M}).\tag{9}\]

In addition to using internal states for membership inference, the adversary can employ external references to calibrate the score function. For instance, Carlini et al. [20] proposed calibrating the sample loss with the entropy of input data evaluated by the zlib compressor, i.e., \[\label{zlib} \mathcal{S}_{\text{zlib}}(x;\mathcal{M})=|\text{zlib}(x)|-S_{LOSS}(x;\mathcal{M}).\tag{10}\] Additionally, other models can serve as effective references for calibration. For example, the LiRA attack [19] utilizes the pre-trained model as a reference to calibrate the fine-tuned model’s LOSS score function, i.e., \[\label{lira} \mathcal{S}_{\text{LiRA}}(x;\mathcal{M})=S_{LOSS}(x;\mathcal{M}_{pt})-S_{LOSS}(x;\mathcal{M}).\tag{11}\] The SPV-MIA attack [22] trains a self-prompted model \(\mathcal{M}_{sp}\) to calibrate the neighborhood score function, i.e., \[\label{spv} \mathcal{S}_{\text{SPV}}(x;\mathcal{M})=S_{Nei}(x;\mathcal{M}_{sp})-S_{Nei}(x;\mathcal{M}).\tag{12}\]

4.3 Pre-trained Model Calibration↩︎

As highlighted in 1, while LoRA has emerged as the most widely adopted fine-tuning algorithm for LMs, only a limited number of existing MIAs have been explored against LoRA fine-tuning datasets. Moreover, most attacks do not calibrate their scores with external references. Given that the pre-trained model of the target LoRA model is publicly accessible, we propose pre-trained model calibration to the performance of existing MIAs without incurring additional costs. This technique leverages the pre-trained model as a reference to recalibrate the MIA scores.

Formally, let \(\mathcal{S}\) denote a score function of an MIA targeting the LoRA fine-tuned model \(\mathcal{M}\). We introduce a calibrated score function, which utilizes the pre-trained model \(\mathcal{M}{pt}\) as a reference, defined as: \[\label{enh} \mathcal{S}_{\text{pt-ref}}(x;\mathcal{M})=\mathcal{S}(x;\mathcal{M}_{pt})-\mathcal{S}(x;\mathcal{M}).\tag{13}\] Here, \(\mathcal{S}(x;\mathcal{M}_{pt})\) estimates \(\Pr(x\in \mathcal{D}_{pt};\mathcal{M}_{pt})\), which serves as the a priori probability of the membership status of \(x\). Conversely, \(\mathcal{S}(x;\mathcal{M})\) estimates \(\Pr(x\in \mathcal{D}_{ft};\mathcal{M})\). Therefore, this score function captures the variation in confidence before and after fine-tuning. Compared to the original score function derived solely from the fine-tuned model, the calibrated one can make the membership status of members and nonmembers more distinguishable.

However, among all existing MIAs, only the LOSS attack has been enhanced using this calibration technique, as demonstrated in 11 . Consequently, despite the scoring functions that already utilize reference calibration (\(S_\text{zlib}\), \(S_\text{SPV}\)) and the score function incompatible to \(\mathcal{M}_{pt}\) (\(\text{GradNorm}_\theta\)), we can enhance five MIAs by 13 , including \(S_\text{Min-k\%}, S_\text{Min-k\%++}, S_{\text{GradNorm}_x}, S_\text{Nei},\) and \(S_\text{MoPe}\).

Ultimately, as shown in 1, our proposed LoRA-Leak framework encompasses a total of fifteen MIA attacks, including five newly introduced MIAs.

5 Evaluation↩︎

In this section, we will evaluate all membership inference attacks in LoRA-Leak. First, we will outline our experimental settings and considerations for membership inference evaluation. Next, we will analyze their applicability to fine-tuning datasets and demonstrate the superiority of our pre-trained model calibration method. Finally, we will discuss the privacy risk of LoRA fine-tuning under different hyperparameters using LoRA-Leak.

5.1 Experimental Settings↩︎

We select three representative open-source LMs as our pre-trained models for fine-tuning, i.e., GPT-2 XL [39], Pythia-2.8B [40], and Llama-2 7B [4], with parameter sizes ranging from 1.5 billion, 2.8 billion, and 7 billion, respectively. The chosen pre-trained models have been extensively used for LoRA fine-tuning.

We focus on the following three downstream tasks to fine-tune LMs and evaluate the MIA risks.

• AG News [15] is a text classification task that categorizes news articles into four classes based on their titles and contents. We leverage this dataset to simulate scenarios where LLMs are fine-tuned to adherent to specific output formats.

• Open Assistant Conversations (OAsst) [16] is a textual dataset containing multi-round conversations between users and AI chatbots in real-world scenarios. In our evaluation, we utilize its subset, the OpenAssistant TOP-1 Conversation Threads dataset [41], which is particularly suitable for fine-tuning LLMs into general-purpose chatbots. The conversations of this dataset are converted to ChatML format [42].

• MedQA [17] is an English single-choice Question Answering (QA) task for medical exams. Each question has five options to choose from. We leverage MedQA to mimic the scenarios where LLMs are fine-tuned on sensitive datasets for domain adaptation.

For the AG News and OAsst tasks, we randomly select 10,000 items to construct the fine-tuning dataset \(\mathcal{D}_{ft}^{\rm AG}\) and \(\mathcal{D}_{ft}^{\rm OA}\), respectively. Similarly, for the MedQA task, we randomly select 8,000 samples to build \(\mathcal{D}_{ft}^{\rm Med}\). Additionally, for each of these three tasks, we randomly select 1,000 samples distinct from their respective \(\mathcal{D}_{ft}\) to form their validation datasets \(\mathcal{D}_{val}\) (e.g., \(\mathcal{D}_{val}^{\rm AG}\)).

To evaluate the effectiveness of MIAs, for each task, we randomly select 512 items from its \(\mathcal{D}_{ft}\) as the members to infer. Meanwhile, we randomly select 512 items that are disjoint from its \(\mathcal{D}_{ft} \cup \mathcal{D}_{val}\) as non-members.

We focus on three key metrics: PPL@val, GAP, and AUC. These metrics assess the performance of the fine-tuned models, the susceptibility of fine-tuned models to MIAs, and the effectiveness of MIAs, respectively.

• Model Utility: Perplexity (PPL) reflects the model’s uncertainty regarding a given text. We leverage the model’s PPL on the validation set (denoted as PPL@val) as an indicator of its specialized utility. PPL can be calculated through 1 . A lower PPL@val suggests that the model exhibits better utility.

• Overfitting Level: The generalization gap (GAP) is defined as the perplexity difference between the fine-tuning dataset and the validation dataset, i.e., \[\text{GAP} = \text{PPL@val} - \text{PPL@ft}.\] Since membership inference threat is primarily due to overfitting [31], we use this metric to evaluate the overfitting level of the fine-tuned models on the fine-tuning datasets, thereby reflecting their hardnesses against MIAs. A lower GAP value (around zero) implies the model is less overfitting, making membership inference more challenging and practical.

• Effectiveness of MIAs: We use the Area Under the Receiver Operating Characteristic Curve (AUC) to evaluate the effectiveness of MIAs. A higher AUC indicates that an MIA is more effective at distinguishing between members and non-members. The reason to use AUC lies in the fact that all attacks in LoRA-Leak predict a score of membership rather than making a hard decision. Therefore, varying thresholds yield the dynamic change of false-positive rates and true-positive rates, and AUC could capture this characteristic.

5.2 Effectiveness of LoRA-Leak↩︎

In this section, we fine-tune all pre-trained models for 10 epochs. For each epoch, we record the perplexity on the fine-tuning and validation sets to monitor the extent of overfitting. Additionally, we perform LoRA-Leak within each epoch and report the best AUC among the eight non-referenced MIAs and six referenced MIAs. The experimental results are shown in 3.

According to 3, we could observe that LoRA fine-tuning effectively reduces the perplexity on training samples, indicating that it helps the model memorize information about the training data. However, as the number of epochs increases, the perplexity on validation samples first decreases and then starts to increase. Consequently, the model’s performance on downstream tasks initially improves and then declines.

We notice that the perplexity gap between the fine-tuning set and the validation set increases as fine-tuning progresses, indicating that the model’s overfitting level intensifies. Consequently, the effectiveness of all MIAs also increase with this growing gap. This observation highlights the significant impact of overfitting on the risk of membership inference. Notably, the best AUC can approach 1.0 when the model is trained for 10 epochs, especially for Llama-2. However, the advantage of non-referenced MIAs versus pre-trained model-referenced MIAs varies. For instance, among all models fine-tuned on MedQA, the pre-trained model-referenced MIA consistently outperforms the non-referenced MIA. Conversely, for models fine-tuned on OAsst, the pre-trained model-referenced MIA performs well when the model has not been severely overfitted, but the non-referenced MIA gains an advantage as overfitting intensifies. This is because the scale of the membership score shrinks as fine-tuning progresses, making the calibrated score less applicable.

As illustrated above, higher epochs can lead to overfitting, enhancing the effectiveness of MIAs and achieving higher reported metrics. However, in practical scenarios, the model will be fine-tuned for optimal performance on downstream tasks rather than for the lowest perplexity on the training set. Moreover, as shown in 6 of the Supplementary Material, all attacks will achieve similarly high AUCs as overfitting intensifies, making their effectiveness indistinguishable. Therefore, we believe that the setting of 3 epochs, where the validation perplexity is relatively low, is a fair representation of the real-world risk of LoRA fine-tuning.

We first perform the non-referenced MIAs as baselines. Here we fix the epoch number to 3. The AUC for these MIAs aginst Llama-2 are reported in 2, and the AUC achieved for other models are reported in 9 of the Supplementary Material. Our findings reveal that the attacks utilize more information could infer membership better, such as logits for Min-K and Min-K%++ and gradients for \(\text{GradNorm}_{\theta}\) and \(\text{GradNorm}_{x}\). Specifically, Min-K%++ consistently acts as the most effective attack for models fine-tuned from Pythia and Llama-2. However, for GPT-2, Min-K%++ only outperforms other attacks on the MedQA dataset, while \(\text{GradNorm}_{x}\) is more effective on the AG News and OAsst datasets. Notably, some attacks perform worse than the LOSS attack, including the zlib attack, Neighborhood attack, and MoPe attack, even though they perform well on inferring membership of pre-training data [20], [21], [23]. We hypothesize that this discrepancy is due to the specific characteristics of fine-tuning data and LoRA modules: (1) Fine-tuning data often involves domain-specific knowledge rather than general corpus data. Consequently, it tends to have high entropy and hard to paraphrase, affecting the effectiveness of zlib and Neighborhood attacks. (2) The compactness of LoRA parameters makes them sensitive to perturbations, potentially impacting the effectiveness of MoPe attacks.

We further perform pt-referenced MIAs against all nine target fine-tuned LMs. As 2 and 9 shows, we could observe that using pre-trained models as references can enhance the performance of each attack compared with the corresponding baselines in most cases. For instance, introducing pre-trained Llama-2 further enhances the Min-K%++ attack from 0.689 to 0.775 on MedQA. Additionally, even though MoPe is not the best attack among non-referenced MIAs, its pt-referenced variant performs as the best attack against GPT-2 and Pythia on OAsst dataset.

5.3 Impact of Different Reference Models↩︎

Despite using pre-trained models as a reference, some MIAs utilize other models for comparison. For instance, the LiRA attack adjusts the loss signal by comparing it to the loss of a shadow model fine-tuned on a dataset with similar distributions [12]. Additionally, Fu et al. [12] propose constructing a shadow model by prompting the target model itself. In this section, we aim to assess the effectiveness of these different reference models.

We first construct shadow models following [12] by fine-tuning three pre-trained models on the TLDR News dataset [43], which shares similar domains with AG News. Additionally, we create self-prompt models following [22] by using the first 16 words of the TLDR News dataset to prompt the target model, followed by fine-tuning three pre-trained models on the resulting corpus. We fix the downstream task as AG News. The AUC results are presented in 4.

Our evaluation reveals that both shadow models and self-prompt models enhance the claimed baseline attacks. However, they are generally less effective than using the pre-trained model as a reference. For example, across all attacks, using the pre-trained model as the reference could enhance Min-K%++-\(\text{Ref}_\text{pt}\) and achieve the most effective MIA among all attacks while using another model as the reference even degrades its AUC. Moreover, we observe that some attacks, such as the Min-K% attack, Neighbourhood attack, and MoPe attack can be boosted by any reference models, resulting in AUC increases of 0.067 to 0.127 for Llama-2. Furthermore, other reference models require additional datasets and fine-tuning efforts, while the pre-trained model is naturally obtainable within the context of LoRA fine-tuning.

5.4 Impact of Fine-tuning Modules↩︎

4pt

Considering that LoRA adapters can be added to different modules of LMs, in this part, we aim to discuss the impact of different fine-tuning choices on the effectiveness of MIAs. We fix the fine-tuning dataset to AG News and keep all other hyperparameters the same as in ¿sec:sec:default-train?.

As shown in 3, we could observe that fine-tuning different modules results in varying susceptibility to MIAs. Specifically, excluding attention layers (qkv) and downscale layers (d) from the feed-forward layers has minimal impact on the AUC of the best LoRA-Leak attacks. However, excluding the upscale layer (u) during fine-tuning could significantly reduce the best MIA AUC. For instance, for all three models, the fine-tuning modules corresponding to the three lowest MIA AUC experimental results did not involve the upscale layer. Meanwhile, we also evaluate whether excluding certain modules during the fine-tuning process would affect model performance. When we only fine-tune the downscale layers, the results of PPL@val are 1.680, 1.543, and 1.110 for GPT-2, Pythia, and Llama-2, respectively, while the results will be 1.631, 1.525, and 1.098 when we include all layers. This means the impact on model performance is relatively low when excluding certain modules. Notably, regardless of whether specific modules are excluded, incorporating information from pre-trained models consistently enhances the effectiveness of MIAs.

6 Defenses↩︎

In this section, we discuss several potential defense mechanisms against LoRA-Leak. We first discuss three typical fine-tuning techniques that could potentially defend against MIAs, including dropout, weight decay, and differential privacy. Additionally, based on our findings from 5.4, we further explore a novel defense approach that excludes vulnerable layers during LoRA fine-tuning.

Given that Llama-2 exhibits the highest susceptibility to membership inference and has become the most widely deployed pre-trained model in the real world, we focus on Llama-2 throughout this section. For ease of discussion, we explore the effectiveness of each method independently, while keeping other irrelevant settings consistent with those in ¿sec:sec:default-train?.

6.1 Dropout↩︎

Dropout [44] is a traditional technique used to mitigate overfitting in deep learning models. It works by randomly deactivating partial neurons and their corresponding connections during each training step. By doing so, the dependence between neurons is reduced, which in turn decreases the risk of MIAs. We leverage the dropout rate, denoted as \(\eta \in [0,1]\), to determine the proportion of trainable neurons to be dropped. The experimental results are shown in 8. The AUC results for each specific MIA are shown in 7.

We first could observe that the fine-tuned models become less vulnerable to MIAs as the dropout rate \(\eta\) increases. Simultaneously, the overall performance of the models remains relatively stable even at \(\eta=0.95\), where the best MIA AUC could achieve a reduction of at most 0.154 (i.e., on the AG News dataset). One extreme case is that, when \(\eta=0.99\), the AUC of the best MIA on the OAsst dataset could even decrease to 0.543, however, the model utility also degrades. Considering that dropout also incurs negligible overhead in terms of computational cost, it is advisable to incorporate dropout with the LoRA fine-tuning process. Nevertheless, it’s worth noting that introducing the pre-trained model as a reference could still result in a stronger attack even when leveraging dropout.

6.2 Weight Decay↩︎

Krogh et al. [45] propose that adding a penalty term for large weights to the original training loss function \(\mathcal{L}\) can improve the generalization of machine learning models, thereby alleviating overfitting. Given that membership inference is primarily influenced by overfitting, we thus investigate the effectiveness of weight decay as a defense. To this end, the modified loss function \(\mathcal{L}_{decay}\) for the LoRA fine-tuning can be defined as \[\mathcal{L}_{decay} = \mathcal{L} + \frac{\lambda}{2} \Vert \Delta \Vert^2,\] where \(\Delta\) represents the weights of the LoRA modules, and \(\lambda\) is a hyperparameter that represents the weight decay rate.

We employ the Adam optimizer with the decoupled weight decay (AdamW) proposed by Loshchilov et al. [46] to fine-tune Llama-2 on three datasets. During fine-tuning, we vary the weight decay rate \(\lambda\) from \(10^{-4}\) to \(10^{-1}\), while also considering models without any weight decay. The results of the best MIA AUC and model utilities are summarized in 4. The AUC results of each MIA are shown in 8.

Our results reveal that applying weight decay has no significant impact on the model in terms of both MIA risks and performance. For instance, all the best AUC fluctuates less than 0.010 compared to the model without any weight decay. Additionally, the perplexity on the validation set increases by at most 0.003.

6.3 Differential Privacy (DP)↩︎

Differential Privacy (DP) [48] is a classical privacy protection technique that provides rigorous indistinguishability for a single entry. Since DP offers provable protection in terms of dataset privacy, it has become a natural defense against MIAs. Specifically, a randomized algorithm \(\mathcal{M}\) with output space \(O\) achieves \(\epsilon\)-differential privacy if \[\Pr[\mathcal{M}(\mathcal{D})\in O]\le e^\epsilon\Pr[\mathcal{M}(\mathcal{D}')\in O]+\delta\] holds for any two adjacent databases \(\mathcal{D}\) and \(\mathcal{D}'\) that differ only at one entry. The parameter \(\epsilon\) represents the privacy budget. A smaller \(\epsilon\) indicates stronger privacy guarantees.

We employ DPLoRA [49] implemented by DP-Transformers [50] to fine-tune Llama-2, varying the privacy budget \(\epsilon\) from 0.1 to 10. The results related to DP are summarized in 5. Additionally, the training and validation loss during the fine-tuning process is depicted in 11, and the AUC results of each MIA are shown in 9.

We conduct the analysis from two perspectives: the effectiveness of mitigating MIA and the impact on model performance. Regarding the effectiveness of DP, we observe that introducing DP can significantly reduce the susceptibility to MIAs: across all three datasets, the best AUCs among all MIAs drop to \(\sim\)​0.5 after applying DP, whereas the best AUCs are above 0.7 without DP. As the privacy budget decreases, the AUC of MIAs slightly decreases. However, we notice that a smaller privacy budget does not provide substantial gains of MIA protection compared to larger \(\epsilon\). For instance, on the AG News dataset, using DP with \(\epsilon=0.1\) only decreases the best MIA AUC by 0.008. Moreover, as 9 shows, non-referenced attacks occasionally outperform the pt-referenced attacks, but the differences remain marginal. This behavior suggests that DP effectively renders all attacks akin to random guessing. Furthermore, though DP is effective in relieving LoRA-Leak, the enhanced privacy comes at a significant cost to model utility. Even with a large privacy budget (e.g., \(\epsilon=10\)), models still experience noticeable performance degradation, which deteriorates further as \(\epsilon\) decreases (see 11). Besides the performance hit, we observe that the loss converges more slowly after applying DP with \(\epsilon=1.0\) and \(\epsilon=10\), and it converges at an even slower rate when \(\epsilon=0.1\). Additionally, DP introduces substantial computational overhead, i.e., fine-tuning on the AG News, OAsst, and MedQA datasets with DP takes 31\(\times\), 7\(\times\), and 11\(\times\) longer runtime, respectively, compared to fine-tuning without DP.

6.4 Excluding Vulnerable layers↩︎

In light of the findings from 5.4, we identify that some modules are the key contributors to membership inference vulnerability, thus we regard excluding these vulnerable layers as a new defense strategy for LoRA fine-tuning.

Recall that fine-tuning the up (u) and gate (g) layers of Llama-2 can amplify the risks of MIAs. To mitigate this, we fine-tune Llama-2 across all three datasets using all LoRA modules (i.e., qkv, o, u, g, d) excluding u, g, or both of them. Note that we adjust their rank \(r\) to ensure that the numbers of the tuned parameters remain roughly the same across all models. The best MIA AUC results are summarized in 6. The AUC results of each MIA when excluding certain layers are shown in 10.

First, we can observe that excluding just one of the vulnerable modules can only reduce the best MIA AUC by 0.041 to 0.080, though removing the gate layers has a more pronounced impact compared to removing the up layers. When both vulnerable modules are excluded, the best AUC values decrease significantly by 0.121 to 0.145. Compared to dropout defense, excluding one of the modules roughly corresponds to the impact of a dropout rate with \(\eta=0.85\), and excluding both modules is similar to a dropout rate with \(\eta=0.95\). However, pt-referenced MIAs still consistently perform as the most effective attack. Regarding model utility, excluding these layers has minimal impact when fine-tuning on the AG News and OAsst datasets. However, there is a moderate performance downgrade when fine-tuning on MedQA. This discrepancy may be due to the specific correlation between the ability of medical knowledge and these excluded modules.

7 Limitations↩︎

Nowadays, closed-source large models such as ChatGPT have progressively made fine-tuning capabilities available through APIs for data uploads [51]. However, the output information of closed-source LLMs is insufficient to initiate MIAs. For instance, OpenAI API [51] and HuggingFace Serverless Inference API [52] do not offer access to the loss on inputs, which is crucial for executing the so-called black-box MIAs [18]. Additionally, the fine-tuning algorithms used by closed-source models are not publicly available, and users cannot flexibly design hyperparameters. Therefore, our work focuses on attacking open-source language models. We leave the design of label-only membership inference attacks [53], [54] against language models as our future work.

Our evaluation includes a diverse set of models ranging in parameter size from 125M to 13B. While we acknowledge that experimenting with even larger language models would be an exciting opportunity to explore more advanced capabilities, such an endeavor would exceed our current technical resources. Moreover, given the scope of our study, we believe that the additional insights gained from models with significantly larger parameter sizes would be marginal, particularly in relation to the increased computational and infrastructural demands they would impose. Therefore, we chose to focus on models within this parameter range, which strikes an optimal balance between technical feasibility and the value of the insights.

8 Conclusion↩︎

In this paper, we propose LoRA-Leak, a comprehensive evaluation framework for MIAs against LMs. In LoRA-Leak, we consider eight non-referenced MIAs and six pt-referenced MIAs, which provides a systematic quantification for membership leakage. By conducting experiment on three practical datasets with three different pre-trained language models, we present the superiority of the pt-referenced MIA attacks, which achieves the best performance among existing MIAs. Meanwhile, to demonstrate the generality of our insights, we further fine-tune the LMs with LoRA variants, and launch LoRA-Leak against these fine-tuned models. We find that DoRA will slightly increase of risk of MIA, while qLoRA could mitigate MIA with the degrade of performance. Additionally, we discuss four defenses and find that excluding specific LM layers and dropout can mitigate privacy risks. We hope our work can benefit the community by presenting comprehensive insights for auditing the privacy risks of LMs, and provide valuable insights for selecting the optimal setting of LoRA fine-tuning to mitigate the risk of MIA.

Dataset Examples↩︎

8.1 AG News↩︎

8.2 OAsst↩︎

8.3 MedQA↩︎

Default Fine-tuning Settings of LoRA↩︎

We leverage the Supervised Fine-Tuning (SFT) command provided by the Transformer Reinforcement Learning (TRL) library [55] for LoRA fine-tuning. The LoRA modules are added on all linear layers of the pre-trained model, including both the self-attention modules and the feed-forward modules of the transformer. These modules are configured with a rank (\(r\)) of 4, and their scaling factor (\(\alpha\)) is set to be twice of \(r\). Each model is trained with a batch size of 16 and a dropout rate of 5%. The default fine-tuning epoch number is 3. We employ the AdamW optimizer [46], with a fixed learning rate of \(10^{-4}\) without using the weight decay technique. The text sequences are truncated to a maximum of 1024 tokens, without resorting to sequence packing or input masking techniques. All other hyper-parameters are set as the default values of the script.

Impact of LoRA Variants↩︎

Given that multiple variants of LoRA have been developed to date, their associated privacy threats also urgently require assessment. We here explore two popular LoRA variants, i.e., Weight-Decomposed Low-Rank Adaptation (DoRA) [56] and qLoRA [11]. Specifically, we consider the Int8 quantization and FP4 quantization settings for qLoRA. The pre-trained model in this part is Llama-2. We evaluate both the membership inference risks and the performance of these LoRA variants.

As 7 illustrates, using DoRA to fine-tune models slightly increases the effectiveness of LoRA-Leak by 0.004 to 0.008. However, the fine-tuned models experience a slight decrease in overall performance. For qLoRA, the effectiveness of LoRA-Leak decreases by -0.001 to 0.014 for Int8 quantization, and FP4 quantization leads to a further decrease of 0.024 to 0.030. Additionally, qLoRA fine-tuning also exhibits performance degradation. Therefore, we could conclude that lower-precision quantization for qLoRA reduces vulnerability but at the cost of greater performance degradation.

Impact of the Size of Pre-trained Model↩︎

To investigate the MIA risks associated with varying scales of pre-trained language models, we further fine-tune different sizes of pre-trained models on the AG News dataset using LoRA, including GPT-2 (124M, 335M, 774M, and 1.5B), Pythia (160M, 410M, 1B, 1.4B, 2.8B, 6.9B, and 12B), and Llama-2 (7B and 13B). We follow the same fine-tuning settings as in ¿sec:sec:default-train? while selecting different rank \(r\) so as to maintain approximately the same number of tuned parameters. As illustrated in 5, the MIA risk increases as the size of the pre-trained model scales up to around one billion (1B) parameters but decreases for models larger than this range. This is because as the number of model parameters increases, the model’s expressive power strengthens, leading to an increased degree of overfitting and thus “memorizing” more details of the fine-tuning data. This “memorization” heightens the risk of the model being susceptible to MIAs. However, once the model parameters reach a certain scale, the risk of overfitting may actually diminish, particularly when the model becomes more regularized or exhibits improved generalization capabilities. Nevertheless, under all circumstances, pt-referenced MIAs expose the MIA risk more thoroughly than non-referenced MIAs in terms of best AUC scores.

Other PEFT Methods↩︎

While our work primarily focuses on LoRA-based methods, we discuss the applicability of LoRA-Leak to other PEFT methods in this section. We fine-tune the Llama-2 model on AG News using both Prompt Tuning [57] and IA3 [58]. For Prompt Tuning, we set the number of virtual prompts as 20 tokens, initialized with “Predict the topic of this news is World, Sports, Business or Sci/Tech”. The best AUCs for non-referenced and pt-reference attacks are 0.511 and 0.546, respectively. For IA3, we add the learned vectors to all attention and feed-forward layers. The best AUC for non-referenced and pt-reference attacks are 0.517 and 0.551, respectively. Overall, the risks associated with these PEFT methods remain relatively minor. This may be due to the very small parameter sizes tuned. For instance, Prompt Tuning and IA3 only tuned 82k and 1.5M parameters, respectively, which is far less than LoRA with 10M parameters. Therefore, although LoRA-Leak does expose more MIA risks in these PEFT methods, their vulnerability to MIA remains minimal.

References↩︎