1 Introduction↩︎

The task of paraphrase generation aims at rephrasing a given text while retaining its meaning. The task has several applications including text simplification, semantic parsing, query re-writing, and data augmentation.

Recently, the use of pre-trained Transformer-based language models has become nearly ubiquitous for different natural language processing (NLP) tasks [1], [2] including paraphrase generation [3], [4]. While the standard method of utilizing a pre-trained language model for NLP tasks is to fine-tune all the parameters in the model, it is not the most parameter-efficient. For example, MegatronLM [5] has more than 11 billion parameters to fine-tune. Various methods have emerged to utilize large-scale pre-trained language models in a more parameter-efficient manner. Such methods include variants of adapter tuning [6], prompt tuning [7]–[9], and Low Rank Adaptation (LoRA) [10]. Particularly, prompt tuning and LoRA can cut down the number of trainable parameters by a factor of thousands. These approaches are particularly important today as pre-trained models continuously become larger than before. In the direction of parameter-efficient methods, we propose Retrieval Augmented Prompt Tuning (RAPT) as a method of augmenting prompt tuning with kNN-based retrieved examples to further enhance the quality of paraphrases.

Besides adaptation of pre-trained language models, we also explicitly focus on the novelty of generated paraphrases because trivial paraphrases with minimal changes may not be as helpful for applications such as data augmentation. We consider the novelty of a paraphrase to be associated with the edit distance of the paraphrase from the input. Roughly, the more edits (word deletion, word insertion, substitution, etc.) there are in a paraphrase, the more “novel" we consider it to be. Paraphrases can be generated with different levels of novelty. For example, simply changing one word would still count as paraphrasing. Usually, however, the level of lexical novelty of a generated paraphrase is left upon the model to decide. In this work, we instead propose a simple model-agnostic method to put more control on the users themselves to decide the level of novelty of the generation. To this end, we propose Novelty Conditioned RAPT (NC-RAPT) that uses specialized prompts for different levels of novelty for novelty-controlled paraphrase generation. Overall, we make the following contributions:

1. We propose Retrieval Augmented Prompt Tuning (RAPT) as a parameter-efficient method for paraphrase generation.

2. We propose Novelty Conditioned RAPT (NC-RAPT) that uses specialized prompts to generate paraphrases with different levels of novelty.

2 Task Definition↩︎

We define paraphrase generation as a sequence-to-sequence problem. Given some input sequence of tokens \(x_{1:n} = (x_1,x_2,x_3,\dots,x_n)\) as the input query, we want a model to paraphrase the input into another sequence \(y_{1:m} = (y_1,y_2,y_3,\dots,y_m)\) serving as the output.

In our experiments, we mainly explore GPT-based models which are pre-trained on autoregressive language modeling or auto-completion. Thus, we frame our problem similar to an auto-completion task such that the downstream task remains similar to the pre-training task (potentially making the transfer of pre-trained knowledge easier).

To frame paraphrasing as an auto-completion task for an autoregressive language model, we can design an input prompt such as “Input: \(x_1,x_2,x_3,\dots,x_n\) \n Paraphrase: ". The task of the model is to auto-complete the given prompt. The completed sequence will look like:”Input: \(x_1,x_2,x_3,\dots,x_n\) \n Paraphrase: \(y_1,y_2,y_3,\dots,y_m\)". We treat the generated sequence \(y_{1:m}\) as the paraphrase of the original input \(x_{1:n}\).

In this overall sequence, “input: " is the prompt prefix,”\n Paraphrase: " is the prompt infix, “\(x_1,x_2,x_3,\dots,x_n\)" is the input, and”\(y_1,y_2,y_3,\dots,y_m\)" is the paraphrased output. The prefix and infix together forms the prompt template: “Input:

\n Paraphrase:

".

We can further generalize the input prompt format by formalizing a generic sequence of tokens denoting the prompt prefix as \(p_{1:s} = (p_1,p_2,p_3,\dots,p_s)\) and a generic sequence of tokens denoting the prompt infix as \(q_{1:t} = (q_1,q_2,q_3,\dots,q_t)\). Overall the generalized input prompt will be: “\(p_1,p_2,p_3,\dots,p_s, x_1,x_2,x_3,\dots,x_n, q_1,q_2,q_3,\dots,q_t\)".

3 Baselines↩︎

In this paper, we explore the following methods to adapt large pre-trained language models. Particularly, we use GPT-based models to adapt for paraphrase generation.

3.1 Fine Tuning↩︎

Fine Tuning (FT) is the standard method of adapting a pre-trained model. In this strategy all the pre-trained parameters undergo gradient updates in some downstream tasks (in this case, paraphrase generation). In our implementation, we manually design an input prompt as “Input: \(x_1,x_2,x_3,\dots,x_n\) \n Paraphrase: " where \(x_{1:n} = x_1,x_2,x_3,\dots,x_n\) is the input as defined in the previous section. We trained the model in the auto-completion framework as discussed in the previous section.

3.2 Adapter Tuning (AT)↩︎

AT introduces some new sublayers (i.e., adapter layers) acting as low-rank bottlenecks within each Transformer layer. Generally, instead of tuning all parameters of a pre-trained model, AT focuses on tuning mainly the adapter layers. Following [6], we only tune the adapter layer parameters and the layer normalization parameters during training. We train the model with the same input prompt format and the same framework as used in fine tuning.

3.3 Low Rank Adaptation (LoRA)↩︎

Given a pre-trained matrix \(W_x \in \mathbb{R}^{d \times k}\), LoRA [10] constrains the update of \(W_x\) through a low-rank matrix \(W_{\delta} \in \mathbb{R}^{d \times k}\). More precisely, instead of directly updating \(W_x\), LoRA first reformulates \(W_x\) to \(W'_x = W_x + W_{\delta}\), and then only updates the parameters involved in the construction of \(W_{\delta}\). \(W_{\delta}\) itself is constructed through the multiplication of two matrices \(B \in \mathbb{R}^{d \times r}\) and \(A \in \mathbb{R}^{r \times k}\) as: \(W_{\delta} = BA\). We maintain \(W_{\delta}\) as a low-rank matrix by keeping \(r \ll min(d,k)\). Thus instead of tuning \(dk\) parameters we only need to tune \(r\cdot(d+k)\) parameters. Similar to [10], we only apply LoRA to the query transformation and value transformation matrices in the multi-head attention sublayers. We train the model in the same framework as used in fine tuning.

3.4 Prompt Tuning↩︎

Large-scale pre-trained language models can already contain substantial implicit knowledge for several natural language tasks before further fine tuning. For example, [2] show that language models can serve as unsupervised multi-task learners. They show that using task-specific natural language prompts enable the pre-trained language models to do specific tasks like translation or summarization without further training. However, manually designing prompt templates is not ideal because it requires human involvement and the ideal prompt may not correspond to human intuitions. Instead recent work [7]–[9] focuses on automatically tuning the prompt template itself. Moreover, some of these works also tend to tune only the prompt template parameters keeping all the pre-trained parameters frozen. As such, these methods also serve as lightweight parameter-efficient options to adapt large language models.

For our implementation of prompt tuning, we consider the generalized input prompt format formalization as discussed in the previous section: “\(p_1,p_2,p_3,\dots,p_s, x_1,x_2,x_3,\dots,x_n, q_1,q_2,q_3,\dots,q_t\)". The initial representation after passing this input prompt sequence through the embedding layer can be presented as:”\(e(p_1),\dots,e(p_s), e(x_1),\dots,e(x_n), e(q_1),\dots,e(q_t)\)". Similar to [9], we directly initialize and tune the prefix (\(e(p_1),\dots,e(p_s)\)) and infix embeddings (\(e(q_1),\dots,e(q_t)\)). When using prompt tuning alone, we only tune the prefix and infix embeddings (unique embedding vectors for each position) while keeping all other parameters frozen. We train the model in the same auto-completion framework as used in fine tuning.

3.5 LoRA + Prompt Tuning (LPT)↩︎

In this approach, we apply both prompt tuning and LoRA at the same time. We tune both the prefix-infix parameters and the newly introduced LoRA parameters. Both LoRA and prompt tuning tune only a minor fraction of the total parameters. Thus, we can easily combine them without increasing the overall parameter count significantly.

4 Proposed Approaches↩︎

In this section, we introduce our proposed approaches: Retrieval Augmented Prompt Tuning (RAPT) and Novelty Conditioned Retrieval Augmented Prompt Tuning (NC-RAPT).

4.1 Retrieval Augmented Prompt Tuning (RAPT)↩︎

In this section, we introduce our first proposed approach, RAPT, which is a synergy between an automated example retrieval strategy and an example-augmented prompt tuning strategy. We discuss both of them below. Like LPT, we also add LoRA because it does not add too many parameters (Table 2) and allows more tuning flexibility.

4.1.1 Example-Augmented Prompts:↩︎

[11] show that large-scale pre-trained auto-regressive models like GPT3 can locate or learn to perform a task from only a few input-output pairs exemplifying the task given in the input prompt context without further gradient updates. For example, let us say we have an input text sequence \(X\) \((\)where \(X = x_{1:n})\) that we want to paraphrase, and we have two exemplar paraphrase pairs \((X_1,Y_1)\) and \((X_2,Y_2)\). As such the input prompt to a GPT3-like model can be of the form: \[Z = [p_{1:s}; X_2; q_{1:t}; Y_2; p_{1:s}; X_1; q_{1:t}; Y_1; p_{1:s}; X; q_{1:t}], \label{naive_prompt_design}\tag{1}\] where \(Z\) is the full input prompt sequence, \(p_{1:s}\) is the prefix sequence for a sample input, and \(q_{1:t}\) is the infix sequence marking the end of the sample input and the start of the sample output. \([.;.]\) denotes concatenation.

We hypothesize that similarly augmenting the input prompt with task exemplars can be still beneficial for smaller pre-trained language models (e.g., GPT2). [10] show that full data fine tuning, LoRA, or prompt tuning on GPT3 can still outperform few-shot approaches. Thus, instead of applying and exploring example-augmented prompts in few shot settings, we use example-augmented prompts in our full data prompt tuning setup. That is, we can treat the embeddings of \(p_{1:s}\) and \(q_{1:t}\) (the prefix and infix embedding parameters as used in our prompt tuning setup as discussed before) as freely trainable parameters. This strategy removes the need for manually designing prefix and infix sequences to best describe the task and delimit the different example inputs and outputs.

However, simply using the prompt design format as in Eq. 1 is not without its problems. During prompt tuning the prefix length (\(k\)) and/or the infix length (\(l\)) can be quite large (e.g.,  \(250\)). Since the prefix and infix are repeated for every example in Eq. 1 , the total length of the input prompt can increase significantly. Long sequences can slow down training and cause memory issues. To solve this issue we revise the prompt design format in Eq. 1 as: \[Z = [d_{1:m}; p_{1:s}; X_2; q_{1:t}; Y_2; p_{1:s}; X_1; q_{1:t}; Y_1; p_{1:s}; X; q_{1:t}]. \label{less_naive_prompt_design}\tag{2}\] In this format, \(d_{1:m}\) serves as a global prefix serving as a sort of task description. We try to maintain \(s,t \ll m\) such that most of the tunable prompt embedding parameters are concentrated on the emebddings of \(d_{1:m}\), allowing us to keep \(s\) and \(t\) small. Thus, given the small \(s\) and \(t\), the overall input prompt sequence length does not increase as much when \(p_{1:s}\) and \(q_{1:t}\) are repeated.

4.1.2 Example Retrieval:↩︎

Above, we described how to design prompts to incorporate input-output examples of a task serving as task exemplars. However, the question remains open: where do we get the task examples? We can always try to construct the task examples manually, but that would again require human involvement.

Instead, we consider an automated method. We hypothesize that given an input sequence \(X\), input-output example pairs that are most similar to \(X\) would be the most suitable in serving as examples to guide paraphrasing of \(X\). Thus, we use a k-Nearest-Neighbor (kNN) algorithm to retrieve the top-k example pairs that have the most semantically similar (to \(X\)) inputs from the training dataset.

More precisely, we first encode each input text in the training data into a single sentence vector using sentence-transformers² [12] (we use paraphrase-mpnet-base-v2 as the pre-trained weights). Similarly, we also encode any given input \(X\) that we want to paraphrase. We select top k examples \(((X_1, Y_1), (X_2, Y_2),\dots, (X_k,Y_k))\) from the training set based on the cosine similarity scores between the embeddings of the training examples and that of the given input \(X\). We add the retrieved examples to the prompt (as in Eq. 2 ) in the ascending order of their cosine similarity scores with the input \(X\). In practice, we set \(k=2\) given the disadvantages of long sequences that we discussed.

4.2 Novelty Conditioned RAPT (NC-RAPT)↩︎

While there are prior works [13], [14] that have focused on improving the novelty of paraphrase generation, there are not many works focusing on controlling the level of novelty. In this work, we use a simple model-agnostic framework to control the novelty level of the generation. This framework allows us at least two benefits.

1. The framework puts more control over the user in deciding the trade-offs they want in increasing the novelty of paraphrasing.

2. The framework can also enforce a model to generate highly novel paraphrases and thus compete with prior novelty-enhancing methods.

Our framework requires two broad steps. In the first step, we classify each sample in the dataset according to the novelty of the ground truth paraphrase. We use Translation Edit Rate (TER) [15], [16] between the input sequence and the ground truth sequence to quantify the novelty of the ground truth. We use three classes of novelty levels: High, Medium, and Low. We classify each sample among the three novelty level classes based on the TER values between the input and the ground truth. If the TER is \(\ge 0.4\), we classify the sample as High. If the TER is \(> 0.2\) and \(< 0.4\), we classify the sample as Medium. If the TER is \(\le 0.2\), we classify the sample as Low. The thresholds were chosen intuitively based on qualitative analysis.

In the second step, during training, we use different prefix and infix embeddings for different novelty classes. Let us say \(c_{(X,Y)}\) denotes the novelty-class of an input-output paraphrase-pair \((X,Y)\), \(p^c_{1:s}\) denotes the prefix sequence for novelty class \(c\), and \(q^c_{1:t}\) denotes the infix sequence for novelty class \(c\). Given these notations, during training, we reformulate Eq. 2 for novelty-conditioned generation as: \[\begin{align} Z = [&d_{1:m};p^{c_{(X_2,Y_2)}}_{1:s};X_2;q^{c_{(X_2,Y_2)}}_{1:t};Y_2;\\ &p^{c_{(X_1,Y_1)}}_{1:s};X_1; q^{c_{(X_1,Y_1)}}_{1:t};Y_1;p^{c_{(X,Y)}}_{1:s};X; q^{c_{(X,Y)}}_{1:t}]. \label{novelty_conditioned_less_naive_prompt_design} \end{align}\tag{3}\] Here \(Y\) is the ground truth paraphrase for \(X\). We keep the same global prefix sequence \(d_{1:m}\) for all novelty classes to save parameters. As can be understood from Eq. 3 , the model learns from a ground truth with novelty class \(c\) only when the corresponding prefix (\(p^c_{1:s}\)) and infix (\(q^c_{1:t}\)) sequences for the same novelty class \(c\) are used. As such, during inference, the model learns to generate a paraphrase of novelty class \(c\) if the prefix (\(p^c_{1:s}\)) and infix (\(q^c_{1:t}\)) sequences for novelty class \(c\) is used for the input. During inference, to generate a paraphrase of a given input \(X\) with novelty class \(c\), we simply use \(p^{c}_{1:s}\) instead of \(p^{c_{(X,Y)}}_{1:s}\) and \(q^{c}_{1:t}\) instead of \(q^{c_{(X,Y)}}_{1:t}\) in Eq. 3 .

NC-RAPT is built upon RAPT. So every other details are the same as for RAPT. We show some example model generations in Table 1. In Table 2, we show the total trainable parameters for all the adaptation methods that we try over GPT2 medium and GPT2 large.

5 Experiments↩︎

5.1 Dataset↩︎

We use four datasets for our experiments as discussed below. Details of dataset split sizes are presented in Table 3.

::: {#tbl:table:dataset details} | Dataset Name | Training | Validation | Test | |:—————–|————-:|—————:|———:| | QQP 50K | 46,000 | 4,000 | 4,000 | | QQP 140K | 134,206 | 5255 | 5255 | | MSRPC | 2,203 | 550 | 1,147 | | ParaSCI ACL | 28,883 | 2753 | 2,345 |

: Dataset split sizes :::

• Quora Question Pairs 50K split (QQP 50K)³: Quora Question Paris (QQP) is a paraphrase detection dataset. We only use the true paraphrase pairs. We use the 50K dataset split as used in [13].⁴

• Quora Question Pairs 140K split (QQP 140K): We also use QQP with a different split size (QQP 140K) as used by 1. ⁵

• Microsoft Research Paraphrase Corpus (MSRPC): MSRPC [17] is another paraphrase detection corpus. We use only the true paraphrase pairs for paraphrase generation. We use the official train-test split.

• ParaSCI-ACL: ParaSCI-ACL [18] is a paraphrase generation dataset in the scientific domain. We use the official split.⁶

5.2 Evaluation Metrics↩︎

We use different evaluation metrics to account for different aspects of paraphrase quality as below:

• BLEU: Like most prior work, we use BLEU4 [19] to measure similarity between prediction and ground truths.

• Self-BLEU: BLEU by itself does not explicitly measure for the novelty of the generated paraphrase. Often, it is possible to achieve high BLEU in paraphrase generation by simply copying the original input. Thus, we also use self-BLUE4, i.e. BLEU4 between the input and the prediction to account for the novelty of the prediction. Low Self-BLEU implies high novelty.

• Self-TER: To check for the novelty of paraphrases beyond simply checking n-gram overlaps as in Self-BLEU, we also measure the Translation Edit Rate (TER) [16] between the input and the prediction. High self-TER implies high novelty.

• BERT: We also want to account for the semantic fidelity of the generated texts. A paraphrase must retain the main semantic content of the input. To measure this, we use cosine-similarity between the sentence encodings of the input and the prediction. For sentence encoding we use Sentence-BERT [12] along with paraphrase-mpnet-base-v2⁷ as the pre-trained weights.

• iBLEU: iBLEU [20] is designed to combine both BLEU and self-BLEU to measure the balance between closeness to ground truth and novelty: \[\text{iBLEU} = \alpha \cdot \text{BLEU} - (1-\alpha)\cdot\text{self-BLEU}\] We use BLEU4 and self-BLEU4 with \(\alpha\) being \(0.7\) similar to

• SARI: SARI [21] explicitly checks for the goodness of edits made on the input by the model in its predictions by comparing against the edits in the ground truth references. [4] show that it correlates well with human judgment for paraphrase generation.

5.3 Experimental Details↩︎

In the main experiments (Table [table:main results]), we explore different adaptation methods on GPT2 medium. We share some results (Table [table:large gpt results]) on GPT2 large too. Typically, during prompt tuning, we initialized the prompt prefix and infix token embeddings with random natural language token embeddings from GPT2 vocabulary. Besides the language model adaptation techniques, we also compare a copy-baseline that simply copies the input text and poses it as a paraphrase. The copy model checks how far we can go in scores like BLEU without changing a single word. We also try using the ground truths as predictions themselves to get a better sense of natural novelty and semantic fidelity of the ground truths in a particular dataset. We share other experimental details and hyperparameters in a later section.

::: {#tbl:table:prior work} | Method | Self-BLEU | BLEU | iBLEU | |:—————|————–:|———-:|———-:| | 2-4 DiPS\(^*\) | 32.45 | 18.47 | 3.19 | | SOW/REAP\(^*\) | 24.19 | 12.64 | 1.59 | | LBoW\(^*\) | 29.00 | 16.17 | 2.62 | | SEPARATOR\(^*\) | 14.84 | 14.70 | 5.84 | | RAPT | 31.78 | 34.09 | 14.33 | | NC-RAPT (High) | 19.55 | 29.88 | 15.05 |

: Comparison with prior work on QQP 140K. \(^*\) indicates that the results are taken from 1. We bold the best results. DiPS refers to the model proposed by [14]. SOW/REAP refers to the model proposed by [22]. LBoW refers to the model proposed by [23]. Separator refers to the model proposed by 1. Our models are based on GPT2 medium. :::

5.4 Experimental Results↩︎

As shown in Table [table:main results], RAPT substantially outperforms the baselines on all metrics except BERT in general. Increased novelty may slightly hurt BERT-based semantic similarity score.

NC-RAPT behaves mostly as expected. We get high novelty (high self-TER, low self-BLEU) when using prefix and infix for novelty class high; we get low novelty when conditioned on low novelty. We also observe a trade-off among novelty metrics, BLEU, and BERT in NC-RAPT. Conditioning on high novelty increases novelty but can reduce semantic fidelity (BERT) and BLEU compared to other models. However, the reduced semantic fidelity (BERT) score is still similar to or better than that achieved by ground truth paraphrases (Ground Truth Method). NC-RAPT conditioned on high generally gets better novelty scores and iBLEU compared to RAPT and the baselines. NC-RAPT conditioned on medium can sometimes gain the best SARI scores whereas NC-RAPT conditioned on low can get the best BERT and BLEU scores but at a significant cost to novelty. NC-RAPT allows the users to decide which trade-offs they are willing to take by choosing a specific novelty class.

The copy models can still achieve relatively high BLEU despite simply copying the original text. This phenomenon was also noted by [24]. This shows the limitation of relying on metrics like BLEU alone.

In Table [table:large gpt results], we show that we still get fairly consistent results even when we use GPT2 large. In Table 4, we compare with some reported results of prior work (described in the table caption). Although SEPARATOR gets the highest novelty score (lowest self-BLEU), its BLEU score is quite low. Our proposed approaches based on GPT2 medium generally achieve a much better balance between BLEU and self-BLEU and thus, a substantially higher iBLEU.

5.5 Analysis↩︎

Random Retrieval Ablation: When using RAPT we use a kNN based example retrieval as discussed before. As an ablation, we try random retrieval instead of a kNN-based approach. As shown in Table [table:rapt ablation], retrieving semantically similar examples using kNN gives us a much better performance. These results support our hypothesis that semantically similar example pairs can better guide paraphrase generation.

However, as a collolary these results may also suggest that RAPT-based methods would not perform as well on a testing dataset where the examples are semantically distant from anything in the training dataset. This can be one limitation of the method.

Prompt Tuning Variants: We also experimented with a few different variants of prompt tuning for paraphrase generation. We reported the results in Table [table:pt variants]. Here, Prompt Tuning [9] is the method that we use in our main experiments. Prompt Tuning Random is the same method but with random initialization of the prefix-infix embeddings instead of random selection from vocabulary embeddings. P-Tuning is similar to the method proposed by [25]. It uses a BiLSTM to associate the prefix and infix tokens. Prefix Tuning is similar to the method proposed by [8]. It uses the same prefix, infix (transfromed by an MLP layer) in every GPT2 layer. Prefix-layer Tuning was used by [10]. It initializes and trains prefix-infix parameters for every layer instead of just the embedding layer as in simple prompt tuning [9]. In the results, we see that prompt tuning gets the best iBLEU and SARI scores. Although Prefix and Prefix-layer Tuning increase the novelty it does so at the cost of other metrics. Thus, we chose prompt tuning for our main experiments.

5.6 Hyperparameter Details↩︎

We tune the hyperparameters on QQP 50K with GPT2 medium for all the approaches. We then use the same hyperparameters for other datasets and GPT2 large. We use random search with a maximum of \(50\) trials and \(3\) epochs per trial, and choose the hyperparameters based on validation loss. For all the approaches, we search the learning rate within \(\{0.1, 0.01, 1e-3, 1e-4, 5e-5\}\). For adapter tuning, we search the adapter bottleneck hidden state dimension within \(\{128, 256, 512\}\).

For LPT, we tune the learning rate for LoRA paramters and prompt tuning parameters separately (we use different learning rates to tune LoRA and prompt template embeddings). For LoRA, LPT, RAPT, and NC-RAPT (all approached involving LoRA), we fix \(r\) (matrix rank) as \(8\) because it worked well in [10]. We also use a weight decay of \(0.01\) for LoRA-based methods. We use the official code for LoRA. ⁸

We set the infix length for all prompt tuning methods to \(8\) because it generally provided the best performance in [10]. We search the prefix length of prompt tuning random, prefix tuning, and prefix-layer tuning within \(\{8, 64, 256\}\). We use the same prefix length as prompt tuning random for p-tuning because both similarly operates at the embedding level. For prompt tuning, we use the same hyperparameters as tuned for prompt tuning random.

For P-Tuning, we initialize the prompt template token embeddings with a dimension of \(b\). The initialized prompt tokens are passed through a BiLSTM (separately for prefix and infix but with shared parameters) with a total hidden size of \(2 \times \left \lfloor{\frac{d}{2}}\right \rfloor\) (where \(d\) is the pre-trained model emebdding dimensions). We then use two affine layers with a GELU activation [26] in-between to transform the concatenation of the forward and backward hidden states (total dimension \(2 \times \left \lfloor{\frac{d}{2}}\right \rfloor\)) into a dimension \(d\). The intermediate layer also has a dimension of \(d\). We search \(b\) within \(\{d, \left \lfloor{\frac{d}{2}}\right \rfloor , \left \lfloor{\frac{d}{4}}\right \rfloor\}\).

Also, during prefix tuning, we use a 2-layered MLP with GELU activation in-between to transform the prompt template embeddings from some dimension \(b\) to dimension \(d\). We use an intermediate layer dimension of \(d\) for the MLP. Again, for prefix tuning too, we search for \(b\) in \(\{d, \left \lfloor{\frac{d}{2}}\right \rfloor , \left \lfloor{\frac{d}{4}}\right \rfloor\}\). We use an MLP layer because it was used by [8]. We also tuned and trained a version without the MLP layer but did not observe better performance than prompt tuning on iBLEU and SARI.

For RAPT and NC-RAPT, we set the length of \(d_{1:m}\) as \(x-8\) (where the value of x is the same as the total prefix length which is searched and tuned in prompt tuning and prompt tuning random). We set the length of \(p_{1:s}\) (in RAPT and NC-RAPT) as \(8\) (same as infix length). Thus, we keep both \(p_{1:s}\) and \(q_{1:t}\) small (length \(8\)) in RAPT and NC-RAPT whereas the majority of the parameters are concentrated on the global prefix \(d_{1:m}\) (\(x-8\) length) as we discussed before.

In all cases, we use AdamW [27] as the optimizer. We also use a linear schedule with warmup for \(100\) steps (we use the get_linear_schedule_with_warmup() function from Transformers library [28]), a gradient norm clipping with a maximum of \(1\), a batch size of \(32\), and a maximum decoding length of \(n+100\) where \(n\) is the size of the input prompt (which includes the prefix tokens, infix tokens, the input to be paraphrased, and all retrieved examples if any). The selected hyperparameters for each approaches from the search are provided in Table [table:hyperparemeters].

During training, we use a maximum epoch of \(30\) with early stopping. We set the early stopping patience as \(3\). Model selection during training is done based on validation loss. The models are trained and tuned on single Tesla V100 32GB GPUs. Gradient accumulation is used to maintain the effective batch size as \(32\).

6 Related Work↩︎

Paraphrase Generation - Traditionally rule-based systems were used for paraphrase generation [29]–[31]. [32], [33] used Statistical Machine Translation (SMT) methods for paraphrasing. More recent works typically utilize Seq2Seq models for paraphrase generation [34]–[37]. [38] proposed bilingual pivoting for paraphrasing. [39] utilized deep reinforcement learning to advance paraphrase generation whereas [40] utilized imitation learning. [13] incorporated a variational autoencoding strategy to generate multiple diverse paraphrases. [14] proposed a submodular optimization-based decoding method to generate diverse paraphrases. [41] also improved the novelty of paraphrases through a GAN augmented with a diversity loss. [42] used counterfactual debiasing to generate a diverse paraphrases. [43] proposed multi-round paraphrase generation for improved diversity. [23] proposed a latent-variable model grounded by bag-of-words from target sentences for paraphrase generation. [3] use GPT2 for paraphrase generation. [4], [44] proposed novel unsupervised paraphrasing strategies.

Similar to our novelty-controlled generation, there are a few approaches [22], [45]–[50] focusing on more controllable paraphrase generation. Although these approaches can provide extra control over different aspects (eg. granularity or syntactic templates) of paraphrase generation, they do not explicitly or directly help us in controlling novelty. Moreover, most of these approaches require specialized architectures which cannot be straightforwardly utilized in the adaptation of common pre-trained language models.

Prompt Tuning - Initial work on prompt tuning focused on discrete selection of prompt template tokens [7], [51]–[53]. Some of the newer works [8]–[10], [25], instead, directly tuned the prompt template token embeddings and/or intermediate layer states in the continuous space; often achieving better performance than the former strategy [25]. Our approach follows the latter direction.

Retrieval Augmentation - [54] introduced a retrieve-and-edit framework for structured output prediction. [55] built upon [54] by using retrieved examples to augment paraphrase generation. However, their approach is not integrated with prompt tuning and uses a specialized architectural which cannot be easily utilized in adapting a generic pre-trained language model without introducing pre-training-fine-tuning discrepancies. Similar to our work, [56], [57] augmented the prompts for pre-trained models using kNN-based retrieved examples. However, unlike our work, they either use manual prompts or a separate model for discrete prompt prediction (instead of tuning the prompts directly in a continuous space) while focusing on few-shot settings. Also, they did not explore paraphrase generation. To the best of our knowledge, our work is the first to integrate kNN-based example retrieval with prompt tuning, in a continuous space [8], [9], for paraphrase generation in a standard (non-few-shot) supervised training setup while, at the same time, incorporating specialized prompts for novelty controlled generation.

7 Conclusion↩︎

In this paper, we propose RAPT as a parameter-efficient retrieval augmented prompt tuning setting for enhanced paraphrase generation. Building up on RAPT, we further introduce NC-RAPT for novelty-controlled paraphrase generation. Our experimental results from four datasets confirms the effectiveness of our methods. As future work, we are interested in applying our proposed approaches to other large pre-trained language models to investigate how performance varies with different model architectures and size. We would also like to explore RAPT for other downstream tasks like semantic parsing, natural language inference, named entity recognition etc. Another avenue for future research, would be testing the effectiveness of the paraphrases generated by the proposed approaches for data augmentation.

8 Acknowledgments↩︎

We would like to thank Edgar Meij, Srivas Prasad, Nimesh Ghelani and the anonymous reviewers for their constructive feedback and suggestions. We also thank Mounica Maddela for the valuable discussions.

References↩︎