1 Introduction↩︎

Preference alignment techniques, such as RLHF [1] and DPO [2], are widely used to align LLMs with human expectations. These algorithms rely on pairwise preference data, where human annotators compare two responses and select the one that better addresses the given prompt. Such preference data is then used to fine-tune LLMs by encouraging the model to rank preferred responses over less preferred ones [3]–[5]. The effectiveness and reliability of preference alignment have motivated its adoption in applications such as chat assistants [6], mathematical reasoning tools [7], and code generators [8].

However, aligning LLMs with human preference data raises substantial privacy concerns, since these datasets frequently include real user prompts and human feedback. These prompts may disclose personal information related to health, identity, or other sensitive topics, and human feedback may reveal private beliefs, preferences, or behavioral patterns [9], [10]. Existing works [10]–[12] have explored approaches to mitigate these risks using the rigorous privacy safeguards provided by differential privacy (DP) [13] and shown encouraging results. However, many of these approaches protect only part of the data, for example, by privatizing either the user prompts [10] or the preference labels [11], [14], but not both. Such partial protection leaves potential privacy gaps, as adversaries could still infer sensitive attributes from unprotected prompts or preference annotations. Besides, such works usually focus on specific post-training algorithms, such as RLHF [12], and are not compatible with newer methods like DPO. Moreover, these methods are typically constrained by limited private preference datasets, and often fall short in achieving high-quality preference alignment, due to the high cost and scalability challenges associated with collecting human preference annotations. These gaps lead to the central question of this work:

In this work, we provide an affirmative answer to this question. Our main contributions are summarized as follows:

• We introduce a new algorithm DPPrefSyn (1) to generate DP synthetic preference data for LLM alignment. DPPrefSyn is a principled framework grounded in the Bradley–Terry preference model and the intrinsic geometric structure of pairwise human preference data. It features the following design elements: 1) Capturing diverse human preferences via shared linear structure of rewards and clustering. The Bradley–Terry preference model, together with a linear reward structure, enables us to group preference samples based on embedding differences between concatenated prompt–preferred and prompt–dispreferred pairs, and to approximate preferences within each cluster using a shared reward function. 2) Improving learning efficiency through DP-PCA. To address sample inefficiency and clustering instability in high-dimensional spaces, we apply DP-PCA to project the embeddings into a lower-dimensional space while preserving key preference signals under DP. 3) Saving privacy budgets by using public prompts. We use public prompts to generate candidate responses and apply private reward models to select preference pairs. This allows us to focus the privacy budget entirely on modeling private preferences, improving privacy efficiency.

• We conduct a rigorous privacy analysis of DPPrefSyn and confirm it follows \((\varepsilon,\delta)\)-DP. In our analysis, we first allocate a privacy budget \(\varepsilon_0\) to DP-PCA [15] for dimensionality reduction, and \(\varepsilon_1\) to DP-KMeans [16] for clustering. The remaining budget is used to train per-cluster reward models with DP-SGD. We apply the Privacy Random Variable (PRV) accountant [17] to tightly compose the DP-SGD privacy cost, ensuring that DPPrefSyn satisfies \((\varepsilon,\delta)\)-DP. Thanks to the post-processing property of DP, the resulting DP synthetic dataset can be reused across multiple preference alignment methods and different LLMs.

• We empirically evaluate DPPrefSyn on standard benchmarks, including question answering tasks from OpenAssistant [18] and Anthropic-HH [5], as well as the TL;DR summarization task [1]. Our experiments suggest that synthetic preference data generated by DPPrefSyn offers strong privacy guarantees while achieving competitive utility across diverse tasks. For example, with \(\varepsilon=2\), fine-tuning the Pythia-2.8B model using DPO on our synthetic data achieves a GPT-4o win rate of \(56.48\%\) on Anthropic-HH, significantly outperforming fine-tuning on real data (\(37.02\%\)). These results show that our DP synthetic preference data can enable effective preference alignment of LLMs with strong privacy guarantees. Moreover, DPPrefSyn can be applied to various preference optimization methods (e.g., DPO and RLHF) and remains effective under strong privacy constraints, overcoming the limitations of other privacy-preserving post-training approaches that are tailored to specific methods.

2 Related Work↩︎

Differentially Private Alignment of LLMs. [12] introduce a DP framework to align LLMs with reinforcement learning by adapting the PPO algorithm to the DP setting. However, their approach is limited to PPO, while preference alignment continues to evolve with new methods. [11] study the problem of reward estimation from preference-based feedback, using the notion of label-DP [19] to protect the privacy of human annotators. [14] propose AUP-RLHF, a user-level label-DP framework for RLHF. While both methods are effective at safeguarding preference labels, they do not address the sensitivity of prompts or responses themselves. Besides, such methods are usually constrained by the limited private user preference data, and may not achieve high-quality preference alignment, as human preference annotations are costly to obtain. [10] focus on protecting sensitive user instructions by generating DP synthetic instructions to replace real ones during data annotation and model fine-tuning.

Beyond preference alignment, several studies focus on private fine-tuning of LLMs using general labeled data, rather than preference comparisons [20]–[25], and some recent work focuses on protecting the privacy of in-context prompts for in-context learning [26]–[30].

Differentially Private Synthetic Text Generation. Our work falls within the broader scope of DP synthetic text generation. In this area, two representative lines of work have emerged. The first line fine-tunes a language model on private data under DP, and then uses the fine-tuned model to generate synthetic text [10], [31]–[38]. For example, [10] privately fine-tune the LLaMA model on private instruction data, and then use the resulting model to generate synthetic text for supervised instruction tuning.

Since DP fine-tuning can be expensive and sometimes infeasible (especially for non-public language models), a recent line of work explores DP synthetic data generation by prompting LLM APIs [39]–[42]. These approaches typically generate synthetic samples using LLM API access and then select outputs that are similar to the private data in a privacy-preserving manner. For instance, [40] propose the Aug-PE algorithm, which iteratively refines LLM-generated samples based on embedding-space similarity to private data. Our work aligns with this line of research and focuses on preference alignment for LLMs.

Diversity in Human Preferences. Recent works highlight that human preferences are inherently diverse [43]–[46]. The key factors include sociodemographic differences (e.g., race, gender, age), personal beliefs and biases, varying levels of domain expertise, and the inherent ambiguity of natural language [47], [48]. Motivated by this, recent studies have explored personalizing preference alignment to better reflect the values of different user groups [49]–[51]. For example, [51] propose a meta-learning framework where an LLM adapts to individual users using a small number of preference examples. In this work, we generate DP synthetic preference data that captures the underlying diversity of human feedback, leading to improved alignment performance compared to approaches that ignore such diversity (as shown in 5.3).

3 Problem Definition↩︎

Preference Alignment of LLMs. We consider preference alignment of an LLM based on pairwise human feedback, using a preference dataset \(\mathcal{D} = \{(x_i, a_i^+, a_i^-)\}_{i=1}^n\). Each sample consists of a prompt \(x_i\) and two responses \(a_i^+, a_i^-\), where \(a_i^+\) is preferred over \(a_i^-\) by human annotators.

Following prior works on reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO) [2], [4], [52], [53], we model human preferences using the Bradley–Terry model [54]: \[\begin{align} \mathbb{P}[a_i^+\succ a_i^- \mid x_i] &= \frac{\exp(r^\star(x_i, a_i^+))}{\exp(r^\star(x_i, a_i^+)) + \exp(r^\star(x_i, a_i^-))} \\ &= \sigma(r^\star(x_i, a_i^+)-r^\star(x_i, a_i^-)), \end{align}\] where \(a_i^+\succ a_i^-\) means \(a_i^+\) is preferred to \(a_i^-\), \(r^\star (x,a)\) is a latent reward function, and \(\sigma(z)=1/(1+e^{-z})\) is the sigmoid function.

This formulation underlies both RLHF, which explicitly learns a reward model from preference data, and DPO, which induces an implicit reward through the log-ratio between the learned policy and a reference policy.

Differentially Private Preference Data Synthesis. We aim to protect the privacy of the preference dataset \(\mathcal{D}_{\text{priv}}= \{(x_i, a_i^+, a_i^-)\}_{i=1}^n\) against an adversary who attempts to access or infer private information about individual prompts, responses, or preference labels based on outputs of the fine-tuned LLM. To achieve this, we propose to generate a DP synthetic preference dataset \(\tilde{\mathcal{D}} = \{(\tilde{x}_j, \tilde{a}_j^+, \tilde{a}_j^-)\}_{j=1}^m\) that preserves critical preference information in \(\mathcal{D}_{\text{priv}}\). Thanks to the post-processing property of DP, \(\tilde{\mathcal{D}}\) can be reused across various preference alignment methods and LLMs without incurring additional privacy cost.

We formally define \((\varepsilon,\delta)\)-differential privacy as follows.

Diverse User Preferences and Linear Reward Structure. Human preferences in the private dataset are usually not homogeneous [46], [56]–[60]. Different annotators may prioritize different aspects of a response, such as factual accuracy, politeness, creativity, or clarity, resulting in diverse and sometimes conflicting signals in the data. To synthesize \(\tilde{\mathcal{D}}\), it is thus critical to capture the diverse human preferences in \(\mathcal{D}_{\text{priv}}\).

To facilitate this, we assume a linear structure for the latent reward function [52], [61], [62]. Specifically, we adopt a linear reward model \[r_\theta (x,a)=\langle\theta, \phi(x,a)\rangle,\] where \(\phi(x,a)\in\mathbb{R}^d\) is the shared feature vector and \(\theta \in \mathbb{R}^d\) is a user-dependent parameter.

The linear structure offers a balance between expressiveness and tractability for modeling diverse user preferences. While individual users may exhibit heterogeneous preferences, their reward functions often share common semantic features in a learned embedding space (i.e., \(\phi(x,a)\)), differing primarily in how these features are weighted (i.e., \(\theta\)). Such a linear parameterization captures this shared structure while allowing variability across users or preference clusters, enabling effective modeling of preference heterogeneity with limited data and strong privacy constraints. As demonstrated by our experimental results in 9.6, the linear reward structure effectively captures preference signals and achieves competitive performance compared to a fully fine-tuned GPT-2 reward model, despite its simplicity.

4 Proposed Method↩︎

4.1 Key Components of the Algorithm Design↩︎

Before detailing our design, we highlight several key components of the algorithm that address the major challenges in synthesizing differentially private preference data. First, human preferences in the private dataset are not homogeneous [46], [56]–[60]. Different annotators may prioritize different aspects of a response, such as factual accuracy, politeness, creativity, or clarity, resulting in diverse and sometimes conflicting signals in the data. Ignoring this diversity in the underlying preferences can result in synthetic data that fails to reflect the true distribution of \(\mathcal{D}_{\text{priv}}\).

Leveraging Linear Reward Structure for Preference Clustering. The linear reward model indicates that \[\mathbb{P}[a_i^+\succ a_i^- \mid x_i] = \sigma(\langle\theta,\phi(x_i, a_i^+) - \phi(x_i, a_i^-)\rangle).\] Therefore, for a tuple \((x_i,a_i^+,a_i^-)\), \(a_i^+\succ a_i^-\) implies that \(\langle\theta,\phi(x_i, a_i^+) - \phi(x_i, a_i^-)\rangle\gg 0\), i.e., the preference vector \(\theta\) is strongly aligned with \(\phi(x_i,a_i^+)-\phi(x_i,a_i^-)\). For human annotators with similar preferences, the feature difference vectors \(\phi(x_i,a_i^+)-\phi(x_i,a_i^-)\) across their rated pairs are therefore expected to point in similar directions in the embedding space. Motivated by this intuition, we cluster the user preference samples based on the feature difference vectors \(\phi(x_i,a_i^+)-\phi(x_i,a_i^-)\), such that user preferences within the same cluster can be well approximated by a shared reward function parameterized by a common preference vector \(\theta\). In 9.9, we show that the clusters constructed using this approach are meaningful and interpretable in terms of user preference styles.

Adopting DP-PCA to Obtain Effective Features. One natural choice for the feature vector \(\phi(x,a)\) is the embedding of the corresponding (prompt, response) pair, as embeddings encode semantic similarities and differences that directly underlie human preference judgments. However, the embeddings are usually high-dimensional (e.g., 1024 dimensions) in order to capture rich semantic and stylistic features. Learning in such a high-dimensional space typically requires a large number of samples, which is inherently limited by the number of samples available within each cluster. To mitigate this issue, we apply differentially private PCA (DP-PCA) [15], [63] to reduce the dimensionality of the embeddings and obtain compact feature representations for clustering and reward model training. This step improves sample efficiency and clustering stability while preserving the essential preference signals necessary for effective alignment.

Utilizing Public Prompts to Save DP Budget. Synthesizing prompts, preferred responses and less preferred responses simultaneously can quickly exhaust the privacy budget and degrade utility. To address this, we choose to use public prompts to avoid spending privacy budget on synthesizing DP prompts, allowing us to allocate the entire budget to modeling preferences over responses. For each public prompt, we generate multiple candidate responses using an LLM and apply a distribution of reward functions, which is trained from private preference data under DP, to construct preference pairs. While public prompts may differ in distribution from private ones, our experiments show that, despite this distribution shift in prompts, the synthetic preference data preserves the statistical preference distribution in the private preference dataset and remains effective for downstream preference alignment tasks.

4.2 DPPrefSyn Algorithm↩︎

We now introduce the DPPrefSyn algorithm (2), which generates a DP synthetic preference dataset from the private dataset \(\mathcal{D}_{\text{priv}}\). DPPrefSyn consists of 3 main steps:

Step 1: Preference Representation and Clustering. We encode each prompt-response pair in \(\mathcal{D}_{\text{priv}}\) using a public sentence embedding model \(\psi\). Specifically, for each sample \((x_i, a_i^+, a_i^-)\), we first concatenate the prompt with each response to form two texts: \([x_i; a_i^+]\) and \([x_i; a_i^-]\). We then compute their embeddings via \(\psi\), and define their difference as \(d_i \gets \psi(x_i, a_i^+) - \psi(x_i, a_i^-)\) (Line [algline:embedding_diff]).

To reduce dimensionality privately, we apply DP-PCA [15] to \(\{d_i\}_{i=1}^n\), obtaining a projection matrix \(\mathbf{P} \in \mathbb{R}^{d \times p}\) under \(\varepsilon_0\)-DP (Line [algline:DP-PCA]). Roughly speaking, DP-PCA [15] approximates the eigendecomposition of the data covariance matrix by estimating the collections of eigenvalues and eigenvectors separately in a DP manner. Each \(d_i\) is then projected to a lower-dimensional space as \(z_i \gets \mathbf{P}^\top d_i\) (Line [algline:project_private_sample]). We apply DP-KMeans clustering [16] on \(\{z_i\}_{i=1}^n\) to group samples with similar preference patterns under \(\varepsilon_1\)-DP, forming \(K\) clusters \({\mathcal{C}_1, \ldots, \mathcal{C}_K}\) in Line [algline:kmeans]. Roughly, DP-KMeans [16] is a DP version of the Lloyd algorithm.

Step 2: DP Reward Model Training. For each cluster \(\mathcal{C}_k\), we learn the shared preference vector \(\theta_k \in \mathbb{R}^p\) that captures preference patterns by minimizing the negative log-likelihood (Line [algline:train_reward_model]): \[\mathcal{L}_{\text{MLE}}(\theta_k) = -\frac{1}{\lvert \mathcal{C}_k \rvert} \sum_{z_i\in \mathcal{C}_k }\log \sigma\left(\langle \theta_k, z_i\rangle\right),\label{eq:mle}\tag{1}\] where \(z_i\) is the DP-PCA projected embedding difference from Step 1. Each reward model is trained with the DP-SGD algorithm (a DP variant of SGD) [64] using noise multiplier \(\sigma_2\).

Step 3: DP Synthetic Preference Generation. We compute a histogram \(\mathbf{h} = \left[|\mathcal{C}_1|, \ldots, |\mathcal{C}_K|\right]\) representing the number of samples in each cluster (Line [algline:histogram]). We normalize it to get a probability distribution \({\mathbf{p}}\) over the clusters, i.e., \({\mathbf{p}}\gets {\mathbf{h}}/\lvert\mathcal{D}_{\text{priv}}\rvert\) (Line [algline:sample_prob]).

Next, for each public prompt \(\tilde{x}_j\), we use a relatively high temperature to prompt an LLM to generate \(L\) diverse candidate responses \({\tilde{a}_j^1, \ldots, \tilde{a}_j^L}\) (Line [algline:generate_candidate]). We then sample a cluster index \(k \sim {\mathbf{p}}\) according to the DP histogram (Line [algline:cluster_index]) and evaluate the responses \({\tilde{a}_j^1, \ldots, \tilde{a}_j^L}\) using the reward model parameterized by \(\theta_k\). To do this, we first compute the embeddings \(\psi(\tilde{x}_j, \tilde{a}_j^{l})\) for \(l=1,\ldots,L\), and project them into the DP-PCA subspace using \(\mathbf{P}\). The reward value for \(\tilde{a}_j^{l}\) is then computed as \(\langle \theta_k, \mathbf{P}^\top \psi(\tilde{x}_j, \tilde{a}_j^{l}) \rangle\). We select the responses with the highest and lowest reward values as the preferred and less preferred responses, respectively: \(c \gets \arg\max_{l\in [L]} \langle \theta_k, \mathbf{P}^\top \psi(\tilde{x}_j, \tilde{a}_j^{l}) \rangle\), \(r \gets \arg\min_{l\in [L]} \langle \theta_k, \mathbf{P}^\top \psi(\tilde{x}_j, \tilde{a}_j^{l}) \rangle\) (Lines [algline:candidate_select_c]-[algline:candidate_select_r]). If the value gap is too small (e.g., \(< 0.5\)), we discard the sample to ensure preference quality. Otherwise, we add \((\tilde{x}_j, \tilde{a}_j^c, \tilde{a}_j^r)\) to \(\tilde{\mathcal{D}}\). This process is repeated for all public prompts, producing a DP synthetic preference dataset \(\tilde{\mathcal{D}}\).

Privacy Analysis of DPPrefSyn. We allocate a privacy budget \(\varepsilon_0\) to DP-PCA (Line [algline:DP-PCA]), and \(\varepsilon_1\) to DP-KMeans (Line [algline:kmeans]), with the remaining budget used to train per-cluster reward models using DP-SGD with noise multiplier \(\sigma_2\) (Line [algline:train_reward_model]). For training DP reward models, we apply parallel composition of DP: since the clusters are disjoint, modifying one data point in the training data affects only one cluster, so the total privacy cost is bounded by the cost of training on the smallest cluster. We ensure a known lower bound on the cluster size by discarding clusters with too few samples. We use the Privacy Random Variable (PRV) accountant [17] to compose the privacy costs of each DP-SGD step, ensuring that it satisfies \((\varepsilon - \varepsilon_0-\varepsilon_1, \delta)\)-DP. All other steps involve only public data or post-processing, and incur no additional privacy cost. We provide a full proof and code for privacy accounting in 8.

Discussion on Prompt Distribution Shift. DPPrefSyn remains effective even when the prompt distribution is different in private and public datasets. We note that the diverse user preference is captured through \(\theta_k\), the parameter in the linear reward model, instead of \(\phi(x,a)\), the feature vector associated with the prompt-response pair. To make this intuitive, consider that a human user may interact with an LLM through diverse prompts, though their underlying preference does not vary. Therefore, as long as we ensure the distribution of \(\theta_k\) resembles that in the private dataset, the resulting synthetic preferences will still reflect the private preference patterns. This is achieved by randomly sampling a cluster index \(k\) according to the DP histogram (Line [algline:cluster_index]) for each public prompt, and then select the synthetic response pairs accordingly. Therefore, even if the prompt distribution may differ in private and public datasets, DPPrefSyn can still preserve the preference patterns captured in the private data.

5 Experiments↩︎

Datasets. We evaluate on question answering and summarization tasks. For question answering, we use the OpenAssistant dataset [18], which contains assistant-style conversations, and the Anthropic-HH dataset [5], which provides human preference comparisons focused on helpfulness and harmlessness. For summarization, we use the TL;DR dataset [1], which contains annotations of human preference on pairs of summaries. To simulate public data, we use prompts from Alpaca [65] for the OpenAssistant QA task, SafeRLHF [66] for the Anthropic-HH QA task, and XSum [67] for the TL;DR summarization task. Additional details are available in 9.2.

Setup for 2. By default, we use BAAI/bge-large-en-v1.5 [68] as the embedding model \(\psi\). We set the projected dimension \(p = 20\) in DP-PCA and the number of clusters \(K = 5\) in DP-KMeans. For DP-SGD, we use a learning rate of 0.1, a batch size of 4, 4 training epochs, and a gradient clipping norm of 1.0. Candidate responses are generated using the instruction-finetuned LLaMA-7B-chat model [69] with a temperature of 0.9. We set the number of candidates per prompt to \(L = 5\). We consider overall privacy budgets \(\varepsilon=0.5,1,2,4,8\) and set \(\delta=1/\lvert\mathcal{D}_\text{priv} \rvert\). We study the effect of different generator models in 3, embedding models and hyperparameters in 5.3.

Evaluation. For downstream preference alignment, we fine-tune the Pythia-2.8B model [70]. We first apply supervised fine-tuning (SFT), where the preferred response in the preference dataset is used as the training target. We then apply the DPO algorithm [2] to further fine-tune the SFT model using preference pairs. Following [2], we measure the win rate of the model-generated responses against the preferred responses in the test set using the GPT-4o model. We study more types of downstream models, and different preference alignment methods as ablation study in 5.2.

Baselines. We compare DPPrefSyn with fine-tuning the downstream model on real data using DP-SGD [64], denoted as DP-FT. Note that DPPrefSyn is more reusable and versatile than DP-FT because it generates DP synthetic data that can be used to train different models with different alignment algorithms without incurring additional privacy cost, while DP-FT consumes the privacy budget in training a single privatized model. We also include a fully private baseline \(\varepsilon = 0\), where we evaluate the base model without any preference fine-tuning.

Details about the setups, hyperparameters, metrics, and baselines are provided in 9.3.

5.1 Understanding the Performance of DPPrefSyn↩︎

In this section, we analyze the performance of DPPrefSyn by answering two research questions about its privacy-utility trade-off. Our main results are presented in 1.

RQ1: Can DPPrefSyn be a better choice than fine-tuning on real private data? DPPrefSyn consistently achieves a stronger privacy–utility trade-off than DP-FT across all tasks under \(\varepsilon=0.5,1,2,4,8\), and outperforms the utility of fine-tuning on real private data without DP constraints (DP-FT with \(\varepsilon=\infty\)), while preserving privacy. For instance, on the OpenAssistant QA task with \(\varepsilon = 2\), our approach achieves \(9.10\%\) after SFT and \(11.04\%\) after DPO, compared to DP-FT’s \(4.38\%\) after SFT and \(6.18\%\) after DPO. Remarkably, while still preserving privacy, DPPrefSyn achieves higher utility than the non-private baseline of fine-tuning on real private data (DP-FT with \(\varepsilon=\infty\)), which reaches \(4.75\%\) after SFT and \(8.20\%\) after DPO. Similarly, on the Anthropic-HH dataset with \(\varepsilon=2\), DPPrefSyn achieves a \(55.21\%\) win rate after SFT and \(56.48\%\) after DPO, compared to \(31.53\%\) and \(37.02\%\) achieved by DP-FT, respectively. It also outperforms direct fine-tuning on real data (DP-FT with \(\varepsilon=\infty\)), which achieves \(31.98\%\) after SFT and \(38.72\%\) after DPO.

These results highlight two key advantages of our DP synthetic data. First, DPPrefSyn leads to significant performance improvement after SFT alone compared to directly applying SFT on private data. For example, on the TL;DR summarization task with \(\varepsilon = 2\), DPPrefSyn achieves a win rate of \(62.49\%\) after SFT, much higher than the \(22.57\%\) achieved by SFT on private data. This is mainly because DPPrefSyn generates candidate responses using a high-performance LLM, leading to high-quality training data that provides more effective supervision during SFT. This advantage holds even with limited public data. For instance, on the OpenAssistant QA task with \(\varepsilon = 4\) and only 5K public prompts, DPPrefSyn achieves a win rate of \(6.80\%\) after SFT, compared to \(4.46\%\) for SFT directly on private data (4), which contains around 14K samples, highlighting DPPrefSyn’s ability to handle data scarcity while preserving utility. Second, applying DPO with our synthetic data on top of the SFT model leads to further improvements, showing that our synthetic preference pairs capture meaningful distinctions aligned with human preferences. This is enabled by the private reward functions that capture diverse user preferences while preserving privacy. Compared to directly using private data, the improvement from DPO is smaller in our setting, likely because the SFT model trained on our synthetic data already performs well, leaving less room for optimization. However, for the entire fine-tuning process, the performance gain using our DP synthetic dataset is still significantly higher than that using real private data.

RQ2: How does DPPrefSyn perform across different privacy budget \(\varepsilon\)? First, 1 shows that DPPrefSyn generally achieves better performance compared to DP-FT as \(\varepsilon\) increases from 0.5, 1, 2, 4, 8 to \(\infty\), indicating good scalability with privacy budget \(\varepsilon\). Second, DPPrefSyn can be more robust to DP noise than DP-FT in some cases. For example, on OpenAssistant with Llama-3-8B as the downstream model, DPO performance for DP-FT drops from 59.13% to 45.79% when \(\varepsilon\) decreases from \(\infty\) to 4, while DPPrefSyn drops only slightly from 61.52% to 60.25% (see 3 in 5.2). The reason could be that DP-FT perturbs model parameters directly through DP-SGD, whereas DPPrefSyn adds noise during data synthesis, which better preserves utility.

5.2 Understanding the Properties of DPPrefSyn↩︎

In this section, we study properties of DPPrefSyn, including its generalizability across different preference alignment methods and LLMs, its performance under varying scales of public data, and its robustness to empirical privacy attacks.

RQ3: Is DPPrefSyn effective across different preference alignment methods? DPPrefSyn supports various preference alignment methods and remains effective under strong DP constraints. In 2, we evaluate DPPrefSyn with SFT, DPO, and PPO. For PPO, we train \(K\) reward models (one for each cluster) to capture diverse preference patterns. During PPO training, we use the average output of the \(K\) reward models as the reward and optimize the policy. We find that DPPrefSyn consistently achieves better utility for SFT, DPO, and PPO, compared with fine-tuning on private data, while preserving strong privacy guarantees.

RQ4: Can more powerful downstream LLMs benefit from synthetic data generated by DPPrefSyn? DPPrefSyn can be effective across a wide range of downstream LLMs. In 3, we conduct experiments on different downstream models, including Qwen-3-4B, Gemma-2-2B, and Llama-3-8B, using Llama-3-8B-Instruct and Qwen3-4B-Instruct-2507 as generator models. We find that DPPrefSyn can outperform fine-tuning on real data, while ensuring privacy across different model families after DPO. For example, using Qwen3-4B-Instruct-2507 as the generator, DPPrefSyn achieves an 86.52% win rate on Qwen-3-4B with \(\varepsilon=4\), which significantly outperforms non-private fine-tuning on real data (59.41%). We also observe that DPPrefSyn’s performance improves with stronger generator models, with Qwen3-4B-Instruct-2507 outperforming Llama-3-8B-Instruct.

RQ5: Can DPPrefSyn be effective with varying scales of public data? We study the effect of public data size \(m\) in 4. The private dataset (OpenAssistant) contains 14K preference annotations, while the full public dataset (Alpaca) contains 52K prompts. We randomly sample subsets of the public prompts at different sizes (\(m=5\text{K},10\text{K},52\text{K}\)) to generate synthetic preference data using DPPrefSyn. We find that even with only 5K public prompts, DPPrefSyn achieves a win rate of \(6.80\%\) after SFT and \(9.44\%\) after DPO under \(\varepsilon=4\), outperforming the DP-FT baseline, which achieves \(4.46\%\) and \(6.12\%\), respectively. This shows that DPPrefSyn remains effective even when the available public data is limited, thanks to the high-quality responses generated by the LLMs and the private reward functions that accurately capture user preferences. Besides, as the public data size increases, the performance improvement over the baseline becomes more prominent, indicating that DPPrefSyn can effectively leverage the vast amounts of public data for preference alignment, further mitigating the data scarcity issue in private preference data.

RQ6: Can DPPrefSyn effectively mitigate empirical privacy attacks? In 5, we perform membership inference attacks (MIAs) [72] against the DPO fine-tuned downstream models on OpenAssistant and report the AUC score. We follow the prior work [71], where the authors instantiate a MIA on preference data for LLM alignment. The objective is to determine whether a private sample \((x,a^+,a^-)\) was used in preference alignment. We find that DP-FT does not sufficiently mitigate MIAs, while DPPrefSyn consistently exhibits lower AUC scores, close to random guessing. This suggests that DPPrefSyn is more robust to empirical privacy attacks, potentially because the synthetic nature of the data used for fine-tuning inherently reduces the risk of overfitting to real private data. We defer the details to 9.4.

5.3 Validating the Design of DPPrefSyn↩︎

We validate the design of DPPrefSyn by studying its algorithmic components in this section.

RQ7: How does modeling diverse human preferences impact DPPrefSyn’s performance? We evaluate this by removing either DP-KMeans or DP-PCA, and varying the number of clusters \(K\), using OpenAssistant with \(\varepsilon=4\). (i) Removing either DP-KMeans or DP-PCA. 3 (a) shows that removing either DP-KMeans or DP-PCA leads to a performance drop. Without DP-KMeans, all samples are assigned to a single cluster. Removing DP-PCA leads to ineffective clustering and sample-inefficient learning in a high-dimensional space. Using multiple clusters with both components improves performance over using a single cluster. (ii) Varying the number of clusters \(K\). In 3, we evaluate DPPrefSyn with \(K=1,3,5,8\) clusters. As \(K\) increases, performance first improves at \(K=3\) and then remains relatively stable at \(K=5,8\). This suggests that DPPrefSyn is robust to \(K\), and moderate values (e.g., \(K=3,5\)) provide a good trade-off between diversity and data efficiency. Overall, these results show the benefit of capturing diverse preferences using multiple clusters.

RQ8: How do different embedding models or hyperparameters affect DPPrefSyn’s performance? We perform ablation studies on the embedding model \(\psi\), DP-PCA dimension \(p\), number of candidates per prompt \(L\), filter threshold, and privacy budget allocation in 9.5. Across different sentence encoders, including bge-large-en-v1.5 [68], all-mpnet-base-v2 [73], and all-distilroberta-v1 [74], DPPrefSyn remains consistently effective. We find that \(p=20\) provides the best overall performance after DPO, \(L=5\) serves as a good default choice, and a moderate filtering threshold of \(0.5\) slightly outperforms no filtering, while a high threshold of \(1\) is too aggressive and hurts performance. Finally, DPPrefSyn is robust to reasonable changes in the budget allocation \(\varepsilon_0\) and \(\varepsilon_1\).

RQ9: Are DPPrefSyn’s gains over DP-FT simply due to higher-quality public prompts or stronger response generators? To rule out confounding effects, we conduct controlled ablations where all methods share the same prompt set or response generator. Under these settings, DPPrefSyn continues to outperform DP-FT (Appendix 9.7), showing that its performance gains do not merely arise from higher-quality public prompts or stronger response generators.

6 Conclusion↩︎

In this work, we introduced DPPrefSyn, an algorithm for synthesizing DP preference data from a private dataset. By modeling preference diversity via clustering, improving efficiency with DP-PCA, and conserving privacy budgets through public prompts, DPPrefSyn enables effective and private preference data synthesis for the alignment of LLMs. Our experiments show that DPPrefSyn achieves privacy guarantees while outperforming fine-tuning on real data.

7 Limitations↩︎

DPPrefSyn requires LLM API access to synthesize preference data, which may incur additional API costs. Our work focuses on English text-based preference data, and extending DPPrefSyn to multilingual settings and multimodal data remains an open direction. Additionally, our experiments cover models up to 8B parameters due to computational constraints.

Impact statement↩︎

This work enables preference alignment of large language models without exposing personal information, which helps balance the benefits of big data with the need to protect individual privacy, promoting ethical data usage and fostering societal trust in an increasingly data-driven world.

Acknowledgements↩︎

This work was supported in part by the U.S. National Science Foundation under the grant ECCS-2531023.

8 Privacy Analysis↩︎

In this section, we provide the privacy analysis for 2.

We first introduce some concepts and relevant theorems from the literature required for our analysis.

Many DP algorithms, such as the Gaussian mechanism, provide a family of \((\varepsilon,\delta)\)-DP guarantees. Specifically, for each fixed \(\varepsilon\), there exists a \(\delta(\varepsilon)\), such that the mechanism satisfies \((\varepsilon,\delta(\varepsilon))\)-DP.

An advantage of using privacy curves for DP mechanisms is that they allow for tighter composition guarantees than those provided by advanced composition theorems [75]. Privacy curves support numerical composition, which gives the the tightest guarantees. [17] give privacy curves for composition of several standard mechanisms such as Gaussian mechanism and subsampled Gaussian mechanism.

[17] also give the privacy loss for a subsampled mechanism given the privacy loss for the original mechanism. This can be used to bound the privacy loss of DP-SGD.

We are ready to do the privacy analysis of our algorithm.

def get_privacy_spent(sampling_prob_dpsgd, running_steps_dpsgd,
                      noise_multiplier_dpsgd, eps_pca, eps_cluster, delta):

    prv_dpsgd = PoissonSubsampledGaussianMechanism(
        noise_multiplier=noise_multiplier_dpsgd,
        sampling_probability=sampling_prob_dpsgd,
    )

    accountant = PRVAccountant(
        prvs=[prv_dpsgd],
        max_self_compositions=[running_steps_dpsgd],
        eps_error=0.01,
        delta_error=delta/10,
    )

    eps_lower, eps_estimate, eps_upper = accountant.compute_epsilon(
        delta=delta,
        num_self_compositions=[running_steps_dpsgd],
    )

    return eps_upper + eps_pca + eps_cluster

9 Experimental Supplementary↩︎

9.1 Computational Resources for Experiments↩︎

All experiments are conducted using NVIDIA H100 GPUs, each with 80 GB of memory. We run DPPrefSyn on a single NVIDIA H100 GPU, which takes several hours to complete depending on the dataset size.

9.2 Datasets↩︎

In this section, we describe the datasets used in our experiments.

• OpenAssistant. The OpenAssistant Conversations dataset [18] is a human-generated, human-annotated assistant-style conversation corpus containing 161,443 messages in 35 languages, with 461,292 quality ratings and over 10,000 fully annotated conversation trees. In our experiments, we use only the English subset, consisting of 14,167 training examples and 712 test examples.

• Anthropic-HH. The Anthropic Helpful and Harmless dataset [5] contains human preference annotations used to train reward models for RLHF. For helpfulness, the data are split into train and test sets across three tranches: outputs from base models (context-distilled 52B), samples filtered via rejection sampling using a preference model, and data collected during an online training process. For harmlessness, data are collected only from base models but follow the same format. The dataset includes 161K training and 8.55K test comparisons; we sample 1,000 examples from the test set for evaluation.

• TL;DR. The Summarize from Feedback dataset [1] contains human preference annotations over pairs of summaries, where annotators are asked to choose the best out of two summaries. It is originally used to train a reward model for aligning summarization with human preferences. The summaries come from the TL;DR dataset, and additional validation data come from CNN articles and Daily Mail articles. The dataset includes 92.9K training and 86.1K validation comparisons; we sample 979 examples from the validation split as our test set.

• Alpaca. The Alpaca dataset [65] contains 52K instruction-following examples generated using self-instruct with OpenAI’s text-davinci-003. Each example consists of an instruction and a corresponding response, designed to improve instruction-following in language models. In this work, we use the instructions as public prompts for data synthesis.

• SafeRLHF. The PKU-SafeRLHF dataset [66] is a human-labeled dataset containing both performance and safety preferences. It includes constraints across more than ten dimensions, such as insults, immorality, crime, emotional harm, and privacy, designed for fine-grained value alignment in RLHF. In this work, we use the 73.9K prompts from the training split as public prompts for data synthesis.

• XSum. The Extreme Summarization (XSum) dataset [67] is designed for evaluating abstractive single-document summarization systems. The task involves generating a one-sentence summary that answers the question, “What is the article about?” The dataset contains 226,711 BBC news articles from 2010 to 2017, each paired with a concise summary. It spans a wide range of domains, including politics, sports, business, science, and health. It includes 204,045 training, 11,332 validation, and 11,334 test examples. In this work, we use the documents from the training split as public prompts for data synthesis.

9.3 Implementation Details and Hyperparameter Settings↩︎

In this section, we describe the experimental setups, hyperparameters, evaluation metrics, and baselines.

Implementation Details of DPPrefSyn. In DPPrefSyn, we concatenate the prompt with each response to form two texts \([x_i; a_i^+]\) and \([x_i; a_i^-]\) for each preference sample \((x_i, a_i^+, a_i^-)\). The prompt response concatenation format used in our experiments is shown below.

OpenAssistant Prompt Response Concatenation Format:

Human: \(<\)question\(>\) \(\text{\textbackslash}\)n\(\text{\textbackslash}\)nAssistant: \(<\)answer\(>\)

Anthropic-HH Prompt Response Concatenation Format:

Human: \(<\)question\(>\) \(\text{\textbackslash}\)n\(\text{\textbackslash}\)nAssistant: \(<\)answer\(>\)

TL;DR Prompt Response Concatenation Format:

\(<\)document\(>\) \(\text{\textbackslash}\)n\(\text{\textbackslash}\)nSummary: \(<\)summary\(>\)

By default, we use BAAI/bge-large-en-v1.5 [68] as the embedding model \(\psi\). We use the DP-PCA implementation from [76], based on the algorithm proposed by [15]. We use the DP-KMeans implementation from [77], based on the algorithm proposed by [16]. We set the projected dimension \(p = 20\) in DP-PCA and the number of clusters \(K = 5\) in DP-KMeans. When training the reward models using DP-SGD, we discard clusters with fewer than \(\lvert \mathcal{D}_\text{priv} \rvert / (K + 4)\) samples to ensure a known lower bound on the sample size per model. For DP-SGD, we use the SGD optimizer with a learning rate of 0.1. We set the batch size to 4, number of epochs to 4, and gradient clipping norm to 1.0. The implementation for DP-SGD uses the Opacus library [78].

By default, we use the instruction-finetuned LLaMA-7B-chat model [69] to generate candidate responses for each public prompt. We set the number of candidates per prompt to \(L = 5\). The generation temperature is set to 0.9. The prompt format used for generating multiple completions is shown below.

Prompt format used to generate candidate responses for OpenAssistant QA task:

Human: \(<\)query\(>\) \(\text{\textbackslash}\)n\(\text{\textbackslash}\)nAssistant:

Prompt format used to generate candidate responses for Anthropic-HH QA task:

Human: \(<\)query\(>\) \(\text{\textbackslash}\)n\(\text{\textbackslash}\)nAssistant:

Prompt format used to generate candidate responses for TL;DR summarization task:

Summarize the following article in a paragraph of 50 words or less: \(<\)article\(>\) \(\text{\textbackslash}\)n\(\text{\textbackslash}\)nAssistant:

Privacy Parameters in DPPrefSyn. In our experiments, we set \(\delta=1/\lvert\mathcal{D}_\text{priv} \rvert\). We provide the values of \(\lvert\mathcal{D}_\text{priv} \rvert\) in 6. We present DP parameters \(\varepsilon_0\), \(\varepsilon_1\) and \(\sigma_2\) in 7.

Evaluation Details. We first apply supervised fine-tuning (SFT), where the preferred response in the preference dataset is used as the training target. We then apply the DPO algorithm¹ [2] to further fine-tune the SFT model using preference pairs. All models are trained with bfloat16 precision. Following [2], for SFT we use a learning rate of \(5\times10^{-7}\), training for 1 epoch. For DPO, we use \(\beta=0.1\), a learning rate of \(1\times 10^{-6}\), and train for 3 epochs. The batch size is 64 for Pythia-2.8B, 32 for Qwen-3-4B and Gemma-2-2B, and 16 for Llama-3-8B.

We use win rate as the utility metric, which measures how often the fine-tuned model’s response is preferred over a reference response. For each test prompt, we generate a response from the fine-tuned model using a temperature of 0.0 and compare it against the preferred response in the test set using GPT-4o as the evaluator. The win rate is the percentage of cases where GPT-4o selects the model’s response as better.

We adopt win rate because it follows the standard evaluation protocol in recent preference alignment research [2], [4], [79], [80], where a strong LLM is used as an automated judge to estimate human preference judgments. Prior work has shown that high-quality frontier models provide reliable and reproducible preference judgments that strongly correlate with human annotations, while offering better scalability and consistency than human studies [81]. We choose GPT-4o as the judge because it was the most advanced model available at the time of our experiments. We also provide the prompt formats used to generate the win rates below, following the format introduced in [2].

DP-FT Details. For DP-FT, we implement DP-SGD using Opacus [78] with FSDP2 and Ghost Clipping. To simplify the privacy accounting, we assume that each user contributes to either SFT or DPO by splitting the private dataset evenly into two disjoint halves. Privacy loss is tracked using the PRV accountant. We perform a grid search over clipping norms \(\{0.05, 0.1, 0.5, 1.0\}\) and use \(0.1\) in our final experiments. Learning rates are searched in \(\{5\times10^{-7}, 1\times10^{-6}, 5\times10^{-6}, 1\times10^{-5}, 5\times10^{-5}, 1\times10^{-4}\}\), with \(5\times10^{-5}\) selected for SFT and \(1\times10^{-5}\) for DPO on Pythia-2.8B, Qwen-3-4B, and Gemma-2-2B, while \(5\times10^{-6}\) is used for Llama-3-8B. The batch size is set to 16 for Pythia-2.8B, 8 for Qwen-3-4B and Gemma-2-2B, and 4 for Llama-3-8B. We train for 2 epochs in SFT and 3 epochs in DPO.

9.4 Empirical Privacy Evaluation by Membership Inference Attacks↩︎

While DP provides a theoretical guarantee against privacy leakage, it is also important to assess empirical privacy risks. In this section, we perform membership inference attacks (MIAs) [72] against the DPO fine-tuned downstream models on OpenAssistant and report the AUC score in 5.

We follow the prior work [71], where the authors instantiate a MIA on preference data for LLM alignment. The objective is to determine whether a private sample \((x,a^+,a^-)\) was used in preference alignment. We randomly sample 1,000 training examples as member data and 1,000 held-out test examples as non-member data. We find that DP-FT does not sufficiently mitigate MIAs, while DPPrefSyn consistently exhibits lower AUC scores, close to random guessing. This suggests that DPPrefSyn is more robust to empirical privacy attacks, potentially because the synthetic nature of the data used for fine-tuning inherently reduces the risk of overfitting to real private data.

9.5 Ablation Studies on Hyperparameters↩︎

In this section, we perform ablation studies on the embedding model \(\psi\), DP-PCA dimension \(p\), number of candidates per prompt \(L\), filter threshold, and privacy budget allocation. All experiments are conducted on OpenAssistant with \(\varepsilon=4\).

Varying the embedding model \(\psi\). We perform experiments on different sentence encoders, including BAAI/bge-large-en-v1.5 [68], sentence-transformers/all-mpnet-base-v2 [73], and sentence-transformers/all-distilroberta-v1 [74]. Our results in 8 show that although embedding quality influences performance, DPPrefSyn is effective across different embedding models.

Varying the DP-PCA dimension \(p\). We experiment with \(p=10,20,30\) as shown in 9. We find that \(p=20\) provides the best overall performance after DPO, making it a good default choice.

Varying number of candidates per prompt \(L\). We experiment with \(L=3,5,8\) as shown in 10. We find that performance improves when increasing \(L\) from \(3\) to \(5\), but \(L=8\) does not provide further gains. This indicates that \(L=5\) is a good default choice.

Varying filter threshold. We experiment with thresholds of \(0\), \(0.5\), and \(1\) as shown in 11. A moderate threshold of \(0.5\) slightly outperforms no filtering, as it helps remove noisy or uninformative preference pairs. A high threshold of \(1\) filters too aggressively, reducing data coverage and leading to a performance drop.

Varying privacy budget allocation. We perform experiments to evaluate alternative budget splits for DP-PCA (\(\varepsilon_0\)) and DP-KMeans (\(\varepsilon_1\)). In the original setting, we allocate \(\varepsilon_0=\varepsilon_1=0.5\) under a total privacy budget of \(\varepsilon=4\). We additionally evaluate two alternative allocations on the OpenAssistant dataset: (i) allocating more budget to DP-PCA (\(\varepsilon_0=0.7,\varepsilon_1=0.3\)), and (ii) allocating more budget to DP-KMeans (\(\varepsilon_0=0.3,\varepsilon_1=0.7\)). The results are shown in 12. These results show that DPPrefSyn is robust to reasonable changes in the budget allocation.

9.6 Discussion on the Linear Reward Models in DPPrefSyn↩︎

In this section, we discuss our design choice of using linear reward models in DPPrefSyn (Line [algline:train_reward_model]). This design follows the Bradley–Terry model with a linear reward function, which is a widely adopted assumption in the preference alignment literature [52], [61], [62], [82], and is formalized in 3.

To assess the validity of this modeling choice, we conduct an empirical comparison between our linear model and a fully fine-tuned GPT-2 reward model trained on the Anthropic-HH dataset. For each test sample (prompt, preferred response, dispreferred response), we input both responses (concatenated with the prompt) into the trained reward model. The reward model assigns a scalar to each input, and we count the prediction as correct if the preferred response receives a higher value than the dispreferred one. Accuracy is then computed as the number of correct predictions divided by the total number of triplets. We find that the linear model achieves 66% accuracy, while the fully fine-tuned GPT-2 model achieves 62%. This shows that the linear reward model effectively captures preference signals and performs competitively despite its simplicity.

9.7 Ablation Studies on Prompt and Response Generator Quality↩︎

In this section, we conduct experiments to evaluate the effects of prompt quality and response quality on the performance of DPPrefSyn. All experiments are conducted on OpenAssistant with \(\varepsilon=4\).

Prompt Quality. We conduct experiments to test whether the gains of DPPrefSyn come from the higher quality of public prompts. Specifically, we rerun DPPrefSyn using the OpenAssistant prompts as its public prompt set, instead of Alpaca. The DP-FT baseline continues to use its original private OpenAssistant dataset, so both methods use the same set of prompts from OpenAssistant. We report the results in 13.

Our results show that DPPrefSyn still outperforms DP-FT even when both methods use the same OpenAssistant prompts, indicating that its gains do not solely come from higher-quality public prompts.

Candidate Response Quality. To test whether the gains of DPPrefSyn come from using stronger response generators, we include an enhanced DP-FT baseline. Specifically, we first use the same LLaMA-7B-Chat generator as in DPPrefSyn to produce candidate responses for each prompt, and then rank these responses using a trained reward model to create synthetic preference pairs. Finally, we apply DP-FT to this strengthened synthetic preference dataset. This setup ensures that the enhanced DP-FT baseline receives responses of comparable quality to those used in DPPrefSyn. We provide the results in 14.

Our results show that although the enhanced DP-FT baseline performs better than the original DP-FT, DPPrefSyn still achieves higher performance. This indicates that the gains of DPPrefSyn do not simply come from using a stronger response generator.

9.8 Ablation Studies on Pairwise Win Rate↩︎

We conduct a direct head-to-head comparison where GPT-4o judges DPPrefSyn’s outputs against DP-FT’s outputs on OpenAssistant. We report the win rate of DPPrefSyn over DP-FT in 15.

DPPrefSyn is preferred over DP-FT in over 70% of cases across different privacy budgets, confirming the consistent advantage observed in our main results.

9.9 Cluster Interpretability in DPPrefSyn↩︎

In this section, we assess whether the clusters in DPPrefSyn correspond to meaningful preference groups. Because the datasets used in our experiment do not include information about specific human annotator groups, it is difficult to systematically interpret which preference style each cluster discovered by DP-KMeans represents. To address this, we simulate two groups of human annotators with different preferences using GPT-4 following [46]. Specifically, for each prompt from the Alpaca dataset [65], we use GPT-4 to generate two responses under different instructions:

(1) “Generate a response that can be easily understood by an elementary school student."

(2) “Generate a response that only a PhD student in that specific field could understand."

Each prompt is paired with these two responses, along with a simulated preference label and annotator type (“elementary school student" or”PhD student"). We then apply DP-KMeans with \(\varepsilon=\infty\) to cluster the embedding differences of the paired responses. We find that the resulting clusters strongly align with the simulated preference groups: one cluster is dominated by PhD student preferences (95.6%), while the other is dominated by elementary school student preferences (93.1%). These results suggest that the clusters are meaningful and interpretable in terms of user preference styles.

9.10 Samples from DPPrefSyn↩︎

In the section, we present generated samples using DPPrefSyn. Representative examples for each task are provided in 16. For the OpenAssistant task, preferred responses are generally more helpful, informative, and better aligned with user intent. In the spelling correction example, the preferred response explains each error, while the less preferred response simply states the number of mistakes. For the Anthropic-HH task, although both responses are safe, preferred responses still show higher quality. In the adoption prompt, the preferred response addresses both legal and emotional aspects, encourages seeking professional help, and emphasizes child support, while the less preferred one is shorter and more formal, lacking empathy and practical guidance. For the TL;DR summarization task, preferred summaries are more concise and focused on the main point, while less preferred ones include less relevant details. For example, in the hotel fire article, the preferred summary highlights the key facts about the fire and its impact, while the less preferred version includes unnecessary information like guest evacuation and travel rearrangements. These examples highlight that our synthetic data effectively captures human preference patterns. We provide more samples from DPPrefSyn below.

References↩︎