1 Introduction↩︎

Recently, commonsense reasoning tasks [1]–[3] have been proposed to investigate the ability of machines to make acceptable and logical inferences about ordinary scenes in our daily life. Both SWAG [1] and CommonsenseQA [2] present a piece of text (an event description or a question) together with several choices (subsequent events or answers), and the system is asked to choose the correct option based on the context. Different from these two discriminative tasks, CommonGen [3] moves to a generation setting. It requires the system to construct a logical sentence based on several concepts related to a specific scenario.

The task of text generation from given concepts are challenging in two ways. First, the sentence needs to be grammatically sound with the constraints of including given concepts. Second, the sentence needs to be correct in terms of our common knowledge. Existing approaches apply pretrained encoder-decoder models [4], [5] for description construction and concepts are modeled as constraints to guide the generation process. Sentences generated by these models are fluent, however, the output might violates the commonsense. An example is shown in Table ¿tbl:comparison?. The model BART generates sentence with “guitar sits" which is incorrect. This demonstrates that the language model itself is not able to determine the rational relationship between concepts.

In order to enrich the source information and bridge the semantic gap between source and target, we argue that external knowledge related to the scene of given concepts are needed to determine the relationships between concepts. Motivated by the retrieve-and-generation framework [6], [7] for text generation, we retrieve prototypes for concepts from external corpora as scene knowledge and construct sentences by editing prototypes. The prototype would introduce scenario knowledge to make up the shortcoming of the language model in finding out reasonable concept combination. Furthermore, prototypes would provide the missing key concepts besides the provided concept set, such as “singer” of the first example in Table ¿tbl:comparison?, to help complete a natural and coherent scenario.

In order to better utilize the prototypes, we propose two additional modules on top of the pretrained encoder-decoder model with the guidance of given concepts. First, considering tokens in the prototype make various contributions in the sentence generation, a scaling module is introduced to assign weights to tokens in the prototype. Second, tokens closer to the concept words in prototypes might be more important for scene description generation, therefore, a position indicator is proposed to mark the relative position of different tokens in the prototypes. The main contributions of this work are three folds. 1) We propose a retrieve-and-edit framework, Enhanced Knowledge Injection BART, for the task of commonsense generation. 2) We combine the two modules, scaling module and prototype position indicator, to better utilize the scenario knowledge of prototype. 3) we conduct experiments on CommonGen benchmark, and results show that our method achieves significantly improvement by using both in-domain and out-of-domain plain text datasets as external knowledge source.

2 Model↩︎

In this section, we introduce our retrieve-and-generation framework based EKI-BART as \(G_{\theta}\) with parameter \(\theta\) that retrieves prototype \(\mathcal{O}=(o_{1},o_{2},\cdots,o_{n_{o}})\) from external text knowledge corpus and extracts the prototype knowledge under the guidance of concepts \(\mathcal{C}=(c_{1},\cdots,c_{n_{c}})\) to improve the commonsense generation of target \(\mathcal{T}=(t_{1},\cdots,t_{n_{t}})\). The overall framework of our proposed model is shown in Figure 1.

2.1 Pretrained Encoder-Decoder↩︎

Pretrained encoder-decoder model, BART [4], commonly follows the transformer architecture. Several encoder-layers stack as encoder and each of them is composed of a self-attention network and a feed-forward network. The input sequence would be encoded into a hidden state sequence \(\mathcal{H}^{e}=(h^{e}_{1},\cdots,h^{e}_{n_{h}})\). Decoder is also stacked by a few decoder-layers and the key difference between encoder-layer and decoder-layer is that there exists an encoder-decoder-attention in the middle of self-attention and feed-forward network. In each encoder-decoder-attention module, the decoder representation \(h^{d}_{u}\) would attend to \(\mathcal{H}^{e}\) following Equation 1 . \[\begin{align} &s_{x}(h^{d}_{u}, h^{e}_{v})=(W_{x,q}h^{d}_{u})^{T}(W_{x,k}h^{e}_{v})\big/\sqrt{d_k} \nonumber\\ &a_{x}=softmax\big(s_{x}(h^{d}_{u},h^{e}_{1}),\cdots,s_{x}(h^{d}_{u},h^{e}_{n_{h}})\big) \nonumber \\ &\hat{h}^{d}_{u}=W_{o}\big[W_{1,v}\mathcal{H}^{e}a_{1},\cdots,W_{X,v}\mathcal{H}^{e}a_{X}\big] \nonumber \\ &h^{d}_{u}=LN\big(h^{d}_{u}+\hat{h}^{d}_{u}\big) ~\label{raw-decoder-encoder-attention} \end{align}\tag{1}\] where \(x\) denotes the \(x\)th attention head, where \(\{W_{x,q},W_{x,k},W_{x,v}\}\in \mathbb{R}^{d_{k}\times d}\) are trainable parameters for query, key and value, \(x\) denotes the attention head, \(d\) is the hidden size, \(d_k\) is the attention head dimension, and \(LN\) is the layernorm function. Generally, there is a normalization operation before we get the encoder output, in other words, the correlation between \(h^{e}_{v}\) and \(h^{d}_{u}\) mainly depends on the direction of \(h^{e}_{v}\) and \(h^{d}_{u}\).

2.2 Model Input↩︎

Following the input setting of BART, we concatenate the provided concepts \(\mathcal{C}\) and the retrieved prototype \(\mathcal{O}\) as a whole input \(\mathcal{S}\) to feed into the pretrained model. \[\begin{gather} \mathcal{S}=\big[\mathcal{C},\mathcal{O}\big]=\big[c_{1},\cdots,c_{n_{c}},o_{1},\cdots,o_{n_{o}}\big] \end{gather}\] where \(\big[\cdot,\cdot\big]\) is the concatenation operation.

In our retrieve-and-generation framework, we need to modify the prototype \(\mathcal{O}\) to meet the requirement of \(\mathcal{C}\). To distinguish each token from \(\mathcal{O}\) or \(\mathcal{C}\), we add the group embedding on top of original BART embedding function as Equation 2 shows. \[\begin{align} \mathit{E}(c_{j})=\mathit{E}_{B}(c_{j})+\mathit{E}_{\mathcal{C}},\; \mathit{E}(o_{k})=\mathit{E}_{B}(o_{k})+\mathit{E}_{\mathcal{O}} \label{EMB} \end{align}\tag{2}\] where \(\mathit{E}_{B}\) stands for the original embedding function in BART including token embedding and position embedding, \(\mathit{E}_{\mathcal{C}}\) and \(\mathit{E}_{\mathcal{O}}\) are two group embedding for concepts \(\mathcal{C}\) and prototype \({\mathcal{O}}\), and \(\mathit{E}\) is the final embedding function.

2.3 Generation↩︎

The prototype \(\mathcal{O}\) not only introduces scenario bias and effective additional concepts, but also brings noises into generation. In order to inject the retrieved knowledge into generation more effectively, we argue to extract the scenario knowledge of prototype in a more fine-grained manner. From Equation 1 , we can see that each token in \(\mathcal{S}\) gets involved in encoder-decoder-attention with the encoder output \(h^{e}_{v}\), thus we propose two mechanisms, namely, scaling module and prototype position indicator, to improve the generation.

2.3.1 Encoder with Scaling Module↩︎

We observe that noises and concept tokens both appear in the retrieved prototype, and these noises would dominate the generation. The simplest solution is to utilize a hard mask, in other words, only keep those concept tokens in prototype and mask others, but the decoder would be no longer aware of the complete prototype scenario, and these effective additional concepts would be also unavailable. Instead of hard masking, we propose scaling module to assign scaling factor for input tokens which can be applied in encoder-decoder-attention, then the model is capable of receiving less noises and learn more from effective tokens.

We investigate the dot product based attention mechanism shown in Equation 1 . Function \(\mathit{F}\) with a scaling factor \(\lambda\) on top of the normalized encoder output states \(\mathcal{H}\) is defined in Equation 3 , \[\begin{align} F(\lambda)&=S(h^{d}_{u}, \lambda h^{e}_{v})=\lambda \Big(\big(W_{q}h^{d}_{u}\big)^{T}\big(W_{k}h^{e}_{v}\big)\big/\sqrt{d_k}\Big)=\lambda S(h^{d}_{u}, h^{e}_{v})=\lambda F(1) ~\label{scale1} \end{align}\tag{3}\] From Equation 3 , we can see that when \(\big(W_{q}h^{d}_{u}\big)^{T}\big(W_{k}h^{e}_{v}\big)\) is a large positive value or \(h^{e}_{v}\) takes important attention weights in \(h^{d}_{u}\), then \(F(\lambda)\) is a monotonically decreasing function. This inspires us to refine the representation of \(h^{e}_{v}\) through \(\lambda\). Viewing \(\lambda\) as an importance factor, we are able to weaken/strength \(h^{e}_{v}\) in encoder-decoder-attention through decreasing/increasing \(\lambda\).

With the awareness of the phenomenon in Equation 3 , we devise a scaling module on the basis of Equation 1 . In practice, we attach a scaling module to the encoder, which can increase the norm if \(h^{e}_{v}\) is likely to contribute to the generation and decrease when the \(h^{e}_{v}\) has a conflict with concepts. Each channel of \(h^{e}_{v}\) would be taken into account separately. This is accomplished with the following scaling module. The module is composed of \[\begin{gather} \Lambda=\mathop{Sigmoid}\Big(W_{2}ReLU\big(W_{1}h^{e}_{v}+b_{1}\big)+b_{2}\Big) \nonumber \\ h^{e}_{v}=h^{e}_{v}\odot\big(2\times\Lambda\big) \label{scale-output} \end{gather}\tag{4}\] where \(W_{1}\in\mathbb{R}^{d_{s}\times d},W_{2}\in\mathbb{R}^{d\times d_{s}},b_{1}\in\mathbb{R}^{d_{s}},b_{2}\in\mathbb{R}^{d}\) are trainable parameters in the scaling module.

Consider that the parameters of pretrained encoder-decoder model have been optimized during pretraining, simply adding the parameter \(\Lambda\) may destroy the distribution of encoder output \(\mathcal{H}\) and leads to training failure. We try to initialize these parameters in scaling module with \(N(0,var)\), where \(var\) is a small value, then the output with sigmoid activation would gather around 0.5, and \(2\times\) would make them fall near 1.0. Thus in the beginning of training, the participation of scaling module would not lead to a mess.

In our knowledge, prototype tokens that co-occur in \(\mathcal{T}\) should be more important than others for the generation of \(\mathcal{T}\). We hope these prior knowledge could help the model to better discriminate the importance of these prototype tokens, thus we introduce an encoder classification task that requires the scaling module to determine which tokens would appear in the generated sentence. \[\begin{gather} \mathcal{L}_{E}=-\sum_{s_{v}\in\mathcal{S}}\Big(\mathcal{I}_{\{s_{v}\in\mathcal{T}\}}\log\mathop{Mean}(\Lambda_{v})+\mathcal{I}_{\{s_{v}\notin\mathcal{T}\}}\log\big(1-\mathop{Mean}(\Lambda_{v})\big)\Big) \end{gather}\] where \(Mean\) is to get the mean value and \(\mathcal{I}\) is indicator function, \(\mathcal{I}_{\{s_{v}\in\mathcal{T}\}}=0\) if \(s_{v}\in\mathcal{T}\) otherwise 1, so is \(\mathcal{I}_{\{s_{v}\notin\mathcal{T}\}}\).

2.3.2 Decoder with Prototype Position Indicator↩︎

These surrounding tokens of concept tokens in prototype \(\mathcal{O}\) tend to describe how these concepts interact with the complete scenario. We argue that informing the decoder of these relative positions would help decoder better learn effective scenario bias of the prototype \(\mathcal{O}\).

Before the computation of encoder-decoder-attention, we devise a position indicator function to assign positions to those tokens in prototype. First, we assign virtual positions to tokens in prototype \(\mathcal{O}\) in sequence, from 1 to \(n_{o}\). Second, we pick up the positions of those concept tokens in prototype as multiple position centers. Third, for each token \(o_{v}\in\mathcal{O}\), we compute the smallest distance from \(o_{v}\) to those concept tokens. The process is shown in Equation 5 . \[\begin{gather} \mathit{D}(s_{v})=min\big\{|v-p|,s_{p}=c,s_{p}\in\mathcal{O},c\in \mathcal{C}\big\}\label{dist95compute} \end{gather}\tag{5}\]

Our inputs tokens are composed of prototype ones and concept ones. Considering the particularity of concept words \(\mathcal{C}\), we assign them with a default position value 0 and adjust the position indicator function of prototype tokens through adding one, the process is shown in Equation 6 . \[\begin{gather} \mathit{D}(s_{v})=\left\{\begin{array}{ll} \mathit{D}(s_{v})+1 & s_{v}\in \mathcal{O} \\ 0 & s_{v} \in \mathcal{C} \end{array} \right.\label{all95dist95compute} \end{gather}\tag{6}\]

On the basis of the prototype position indicator function in Equation 6 , we add the information of relative position from each token itself to the closest concept token in prototype into encoder-decoder-attention through Equation 7 . \[\begin{gather} \mathit{ED}(h^{e}_{v})=\mathit{E}_{\mathit{D}}\big(\mathit{D}(s_{v})\big)\nonumber \\ S(h^{d}_{u}, h^{e}_{v})=\big(W_{q}h^{d}_{u}\big)^{T}\big(W_{k}h^{e}_{v}+\mathit{ED}(h^{e}_{v})\big)\big/\sqrt{d_{k}} ~\label{new-encoder-decoder-attention} \end{gather}\tag{7}\] where \(\mathit{E}_{\mathit{D}}\) is the embedding for those distance values in \(\mathit{D}\). These prototype tokens that more close to the concept tokens are expected to receive more attention than other tokens.

2.4 Training↩︎

The objective of our model is to maximize the log-likelihood for \(\mathcal{T}\) given \(\mathcal{O}\) and \(\mathcal{C}\). \[\begin{gather} \mathcal{L}_{D}=-\mathop{log}\sum_{k}P(t_{k}|\mathcal{O},\mathcal{C},t_{<k}) \end{gather}\] where \(t_k\) in the \(k\)th token in \(\mathcal{T}\) and \(t_{<k}\) are the first \((k-1)\) tokens in \(\mathcal{T}\).

During training, we try to minimize the sum of encoder classification loss and the decoder log-likelihood loss. The \(\lambda\) is utilized to keep the balance between \(\mathcal{L}_{D}\) and \(\mathcal{L}_{E}\) such that encourages the model \(G_{\theta}\) to achieve better performance in generation. \[\begin{gather} \mathcal{L}=\mathcal{L}_{D}+\lambda\mathcal{L}_{E}\label{overall95loss} \end{gather}\tag{8}\]

During prediction, we decode with beam search, and keep the sequence with highest predicted probability among those in the last beam.

3 Experiment↩︎

In this section, we conduct experiments on CommonGen benchmark to evaluate the effectiveness of our proposed approach. To dig into our approach, we perform ablation studies to explore the different effects of scaling module and prototype position indicator.

3.1 Prototype Collection↩︎

3.1.0.1 In-Domain Corpus \(D_{in}\)

CommonGen is to describe a common scenario in our daily life. Datasets of image captioning or video captioning would contain more knowledge about spatial relations, object properties, physical rules, temporal event knowledge and social conventions that contribute to build the target scene contains these provided concepts. We utilize VaTeX [8], SNLI [9], Activity [10] and the training set of CommonGen as the external plain text knowledge datasets and retrieve prototype according to the concepts appear in the sentence.

3.1.0.2 Out-of-Domain Corpus \(D_{out}\)

In-domain corpus \(D_{in}\) may only suitable for these description sentences for daily scenario and has difficulty in generalizing to other domains, thus we also employ Wikipedia as our external knowledge dataset to retrieve prototypes to test the generalization of our model.

The number of retrieved concepts in prototypes that co-occur in ground truth sentence across different external knowledge datasets \(\mathcal{D}_{in}\) and \(\mathcal{D}_{out}\) is shown in Table ¿tbl:concept95number?. It is easy to conclude that we are able to retrieve more relevant prototypes from in-domain dataset \(\mathcal{D}_{in}\) compare to out-of-domain dataset \(\mathcal{D}_{out}\).

3.2 Experimental Setup↩︎

CommonGen [3] dataset contains 27,069, 993 and 1,497 concept-sets in training, validation and test set, the sentences are 39,069, 4,018 and 6,042 respectively. The proportion of novel concept-sets in validation and test datasets are \(95.53\%\) and \(98.49\%\), which require model to generalize well to unseen concepts. We use BLEU-3/4 [11], METEOR [12], ROUGE-2/L [13], CIDEr [14], and SPICE [15] as evaluation metrics.

We employ BART Large model [4] as the pretrained generation model. We adopt cross-entropy loss with 0.1 label-smoothing penalty. The \(\lambda\) in Equation 8 is 1.0. We use inverse-sqrt learning rate scheduler with 500 warmup steps, the learning rate, max-tokens per batch and max updates are 4e-5, 1024 and 5k. The dropout rate is 0.1. We set the standard deviation of initialization in group embedding, scaling module and prototype position indicator to 5e-3. The optimizer of model is Adam [16] with \(\beta_{1}=0.9\) and \(\beta_{2}=0.999\). During decoding, the size of beam search is 5.

3.3 Overall Performance↩︎

To compare our methods with baseline methods, we classify them into three groups.

3.3.0.1 Group 1

Models without pretraining. bRNN-CopyNet and Trans-CopyNet are based on the best popular architecture Bidirectional RNNs and Transformers [17] with attention and copy mechanism [18]. MeanPooling-CopyNet is employed to deal with the influence of the concept ordering in the sequential based methods, where the input concepts is randomly permuted multiple times and decoding is with a mean pooling based MLP network. Levenshtein Transformer [19] is an edit-based non-autoregressive generation model, where the generated sentences go through multiple refinement.

3.3.0.2 Group 2

Pretrained language generation models including GPT-2 [20], UniLM [21], UniLM-v2 [5], BERT-Gen [5], BART [4], and T5 [22]. All these models are trained with a seq2seq format.

3.3.0.3 Group 3

Methods proposed in this work which are based on external knowledge dataset \(\mathcal{D}_{in}\) and \(\mathcal{D}_{out}\). Retrieve\(_{\mathcal{D}_{*}},*\in\{in,out\}\) take the prototype retrieved from \(\mathcal{D}_{in}\) and \(\mathcal{D}_{out}\) as the hypothesises. BART\(_{\mathcal{D}_{*}},*\in\{in,out\}\) feed the concatenation of concepts and prototype retrieved from \(\mathcal{D}_{in}\) and \(\mathcal{D}_{out}\) into BART. EKI-BART\(_{\mathcal{D}_{*}},*\in\{in,out\}\) apply our proposed model in \(\mathcal{D}_{in}\) and \(\mathcal{D}_{out}\), respectively.

We list the performance of different models in Table ¿tbl:OverallPerformance?. According to the results, we have several findings.

1. Performance of pretrained models are far better than these models without pretraining, which demonstrates that training from scratch with data in CommonGen does not suffice for the concepts-based generation. Models pretrained in large scale corpus do learn more knowledge that would contribute to the generation.

2. The models with prototype retrieved from \(\mathcal{D}_{in}\) are better than those with \(\mathcal{D}_{out}\), this shows that in-domain dataset \(\mathcal{D}_{in}\) consisting of daily scenario descriptions provided more relevant and high-quality prototype than \(\mathcal{D}_{out}\).

3. BART\(_{\mathcal{D}_{*}}\) and EKI-BART\(_{\mathcal{D}_{*}}\), \(*\in\{in,out\}\) both outperform the BART baseline, which indicates that introduce external text knowledge as prototype would contribute to the concept based generation. Prototype provides effective scenario bias to find out the reasonable concept combination for the generation.

4. EKI-BART\(_{\mathcal{D}_{in}}\) and EKI-BART\(_{\mathcal{D}_{out}}\) both perform better than their count-part models BART\(_{\mathcal{D}_{in}}\) and BART\(_{\mathcal{D}_{out}}\). Our model is able to achieve improvement in both in-domain and out-of-domain datasets.

3.4 Ablation Study↩︎

In this section, we perform ablation study on the development and test dataset to dive into the effectiveness of different components in our model. We use the \(\mathcal{D}_{in}\) as knowledge dataset. The baseline is the retrieval-based model and the pretrained based model without any prototype text. We use GE, SM and PPI to denote group embedding, scaling module and prototype position indicator, respectively. Several findings stand out:

1. BART\(_{\mathcal{D}_{in}}\)+GE and BART\(_{\mathcal{D}_{in}}\)+GE+SM outperform BART\(_{\mathcal{D}_{in}}\) and BART\(_{\mathcal{D}_{in}}\)+SM, respectively. This shows that the group embedding which better distinguish concept and prototype is benefit to the generation.

2. BART\(_{\mathcal{D}_{in}}\)+SM and BART\(_{\mathcal{D}_{in}}\)+GE+SM perform better than BART\(_{\mathcal{D}_{in}}\) and BART\(_{\mathcal{D}_{in}}\)+GE, respectively. This verifies the effectiveness of scaling module that better discriminate the noises and effective concepts in retrieved prototypes.

3. BART\(_{\mathcal{D}_{in}}\)+GE+SM+PPI performs better than BART\(_{\mathcal{D}_{in}}\)+GE+SM, achieving 0.7 and 0.8 BLEU-3 improvement on development and test dataset. This demonstrates that informing decoder of the distance from each token to concepts would better identify these important factors in prototype.

3.5 Effectiveness of Scaling Module↩︎

Here, we compare our scaling module with hard mask strategy. We have two implementations of hard masking:

1. HM\(_{1}\): After encoding, we mask the output states of \(\mathcal{O}\) and only keep that of \(\mathcal{C}\).

2. HM\(_{2}\): We mask these states of tokens \(s_{v}\in\mathcal{O},\forall c\in\mathcal{C},c\neq s_{v}\).

3. SM\(_{0}\): We remove the encoder classification mechanism from scaling module.

The experiment is conducted in \(\mathcal{D}_{in}\) and we list the results in Table ¿tbl:hardmaskresult?.

From the results in Table ¿tbl:hardmaskresult?, first, we can see that the last four models perform better than the counter-part model BART\(_{\mathcal{D}_{in}}\)+GE, this verifies that it is beneficial to remove the noises in prototype. Second, performance of BART\(_{\mathcal{D}_{in}}\)+GE+SM and BART\(_{\mathcal{D}_{in}}\)+GE+SM\(_{0}\) are better than both BART\(_{\mathcal{D}_{in}}\)+GE+HM\(_{1}\) and BART\(_{\mathcal{D}_{in}}\)+GE+HM\(_{2}\), this indicates that our scaling module is better than the hard masking strategies HM\(_{1}\) and HM\(_{2}\). This phenomenon demonstrates that there exists more effective additional concepts besides those concept tokens in prototype that would contribute to build the target scene, directly masking these tokens would block the generator receiving these additional information, but our scaling module is able to keep these additional information. Third, BART\(_{\mathcal{D}_{in}}\)+GE+SM outperforms BART\(_{\mathcal{D}_{in}}\)+GE+SM\(_{0}\), this shows that injecting the prior knowledge of prototype in scaling module would boost the performance of scaling module.

3.6 Missing Concept Number in Generation↩︎

CommonGen aims to generate scenario description that contains all of these provided concepts. If the model is able to find out the most plausible scene with these concepts, these would be no concepts missing in the generated sentence. To check whether our model is able to find out better scene on the basis of retrieved prototype, thus we compare the number of missing concepts in Retrieve\(_{\mathcal{D}_{in}}\), BART\(_{\mathcal{D}_{in}}\) and EKI-BART\(_{\mathcal{D}_{in}}\) and list the results in Figure 2.

From Figure 2, there are another 300+ instances with no concepts missing in generation of BART\(_{\mathcal{D}_{in}}\) and EKI-BART\(_{\mathcal{D}_{in}}\) compared to Retrieve\(_{\mathcal{D}_{in}}\), we easily conclude that the two models are able to inject more concepts into the retrieved prototype and further utilize the prototype knowledge to generate a more plausible sentence. We also notice that the number of instances with no concept missing of EKI-BART\(_{\mathcal{D}_{in}}\) is more than that of BART\(_{\mathcal{D}_{in}}\), which shows that BART\(_{\mathcal{D}_{in}}\) is more likely to ignore the provided concepts than EKI-BART\(_{\mathcal{D}_{in}}\) and being dominated by noises in prototype. This verifies that the ability of EKI-BART\(_{\mathcal{D}_{in}}\) in dealing with prototype noises is stronger than BART\(_{\mathcal{D}_{in}}\), and removing these noises is useful to construct a more plausible scenario.

4 Related Work↩︎

4.1 Commonsense Reasoning↩︎

Recently, there are emerging works to investigate machine commonsense reasoning ability. ATOMIC [23], Event2Mind [24], MCScript 2.0 [25], SWAG [1], HellaSWAG [26], Story Cloze Test [27], CommonsenseQA [2] and CommonGen [3] are released to reason over external knowledge besides the inputs for question answering or generation. Rajani et al.  explore adding human-written explanations to solve the problem. Lin et al.  construct schema graphs from ConceptNet to reason over relevant commonsense knowledge. lv et al.  focus on automatically extracting evidence from heterogeneous external knowledge and reasoning over the extracted evidence to study this problem. Considering quite a few relationships about these concepts require a variety of background knowledge such as spatial relations, object properties, physical rules, temporal event knowledge, social conventions, etc., which may not be recorded in any existing knowledge bases, this paper focuses on retrieving knowledge from plain text in order to introduce scenario bias for concepts-set based generation.

4.2 retrieve-and-generation↩︎

The retrieve-and-generation approaches are developed for many tasks, including dialogue generation [6], [28], language modeling [29], code generation [7] and text summarization [30]–[32]. Ji et al.  and Yan et al.  focuses on prototype ranking in the retrieval-based model but they do not edit these retrieved prototypes. Re3Sum [33] is an LSTM-based model developed under the retrieve-and-generation framework that retrieves multiple headlines and pick the single best retrieved headline, then edit. Hashimoto et al. [7] Hossain et al.  presents a framework that retrieve, generation and rerank on the basis of BERT [34], but they do not deal with prototype noise in an explicit manner. Song et al.  introduces an extra encoder for the retrieved response, and the output of the encoder, together with that of the query encoder, is utilized to feed the decoder. Weston et al.  simply concatenates the original query and the retrieved response as the input to the encoder. Instead of solely using the retrieved response, Wu et al.  further introduces to encode the lexical differences between the current query and the retrieved query. Pandey et al.  proposes to weight different training instances by context similarity. Different from these works, we explore the retrieve-and-generation framework on the basis of pretrained encoder-decoder model, and identify the importance of each token in prototype in a more fine-grained manner.

5 Conclusion and Future Work↩︎

In this paper, we proposed an enhanced retrieve-and-generation model for commonsense generation. The key of CommonGen is to identify the priority of the scene based on the concept combination. We have scaling module to softly reduce the impact of prototype noises on generation and prototype position indicator to help decoder to learn the prototype scenario better. Both our retrieve-and-generation model with in-domain and out-of-domain datasets achieve better performance than retrieval and pretrained encoder-decoder methods. In future, we plan to build the relationship of these concepts in a more structure manner.

6 Acknowledgments↩︎

This work is partically supported by National Natural Science Foundation of China (No. 71991471), Science and Technology Commission of Shanghai Municipality Grant (No.20dz1200600, No.18DZ1201000, 17JC1420200).

References↩︎