Self-Supervised Knowledge Triplet Learning for Zero-shot
Question Answering
May 01, 2020
Abstract
The aim of all Question Answering (QA) systems is to be able to generalize to unseen questions. Current supervised methods are reliant on expensive data annotation. Moreover, such annotations can introduce unintended annotator bias which makes systems focus more on the bias than the actual task. In this work, we propose Knowledge Triplet Learning (KTL), a self-supervised task over knowledge graphs. We propose heuristics to create synthetic graphs for commonsense and scientific knowledge. We propose methods of how to use KTL to perform zero-shot QA and our experiments show considerable improvements over large pre-trained transformer models.
1 Introduction↩︎
The ability to understand natural language and answer questions is one of the core focuses in the field of natural language processing. To measure and study the different aspects of question answering, several datasets are developed, such as SQuaD [1], HotpotQA [2], and Natural Questions [3] which require systems to perform extractive question answering. On the other hand, datasets such as SocialIQA [4], CommonsenseQA [5], Swag [6] and Winogrande [7] require systems to choose the correct answer from a given set. These multiple-choice question answering datasets are very challenging, but recent large pre-trained language models such as BERT [8], XLNET [9] and RoBERTa [10] have shown very strong performance on them. Moreover, as shown in Winogrande [7], acquiring unbiased labels requires a “carefully designed crowdsourcing procedure”, which adds to the cost of data annotation. This is also quantified in other natural language tasks such as Natural Language Inference [11] and Argument Reasoning Comprehension [12], where such annotation artifacts lead to “Clever Hans Effect” in the models [13], [14]. One way to resolve this is to design and create datasets in a clever way, such as in Winogrande [7], another way is to ignore the data annotations and to build systems to perform unsupervised question answering [15], [16]. In this paper, we focus on building unsupervised zero-shot multiple-choice QA systems.
Recent work [16], [17] try to generate a synthetic dataset using a text corpus such as Wikipedia, to solve extractive QA. Other works [18], [19] use large pre-trained generative language models such as GPT-2 [20] to generate knowledge, questions, and answers and compare against the given answer choices.
In this work, we utilize the information present in Knowledge Graphs such as ATOMIC [21]. We define a new task of Knowledge Triplet Learning (KTL) over these knowledge graphs. For tasks which do not have appropriate knowledge graphs, we propose heuristics to create synthetic knowledge graphs. Knowledge Triplet Learning is like Knowledge Representation Learning and Knowledge Graph Completion but not limited to it. Knowledge Representation Learning [22] learns the low-dimensional projected and distributed representations of entities and relations defined in a knowledge graph. Knowledge Graph Completion [23] aims to identify new relations and entities to expand an incomplete input knowledge graph.
In KTL, as shown in Figure 1, we define a triplet \((h, r, t)\), and given any two as input we learn to generate the third. This tri-directional reasoning forces the system to learn all the possible relations between the three inputs. We map the question answering task to KTL, by mapping the context, question and answer to \((h,r,t)\) respectively. We define two different ways to perform self-supervised KTL. This task can be designed as a representation generation task or a masked language modeling task. We compare both the strategies in this work. We show how to use models trained on this task to perform zero-shot question answering without any additional supervision. We also show how models pre-trained on this task perform considerably well compared to strong pre-trained language models on few-shot learning. We evaluate our approach on the three commonsense and three science multiple-choice QA datasets.
The contributions of this paper are summarized as follows:
We define the Knowledge Triplet Learning over Knowledge Graph and show how to use it for zero-shot question answering.
We compare two strategies for the above task.
We propose heuristics to create synthetic knowledge graphs.
We perform extensive experiments of our framework on three commonsense and three science question answering datasets.
We achieve state-of-the-art results for zero-shot and propose a strong baseline for the few-shot question answering task.
2 Knowledge Triplet Learning↩︎
We define the task of Knowledge Triplet Learning (KTL) in this section. We define \(G = (V,E)\) as a Knowledge Graph, where \(V\) is the set of vertices, \(E\) is the set of edges. \(V\) consists of entities which can be phrases or named-entities depending on the given input Knowledge Graph. Let \(S\) be a set of fact triples, \(S \subseteq V \times E \times V\) with the format \((h,r,t)\), where \(h\) and \(t\) belong to set of vertices \(V\) and \(r\) belongs to set of edges. The \(h\) and \(t\) indicates the head and tail entities, whereas \(r\) indicates the relation between these entities.
For example, from the ATOMIC knowledge graph, (PersonX puts PersonX’s trust in PersonY, How is PersonX seen as?, faithful) is one such triple. Here the head is PersonX puts PersonX’s trust in PersonY, relation is How is PersonX seen as? and the tail is faithful. Do note \(V\) does not contain homogenous entities, i.e, both faithful and PersonX puts PersonX’s trust in PersonY are in \(V\).
We define the task of KTL as follows: Given input a triple \((h,r,t)\), we learn the following three functions. \[\begin{align} f_t(h,r) \Rightarrow t, \;\; f_h(r,t) \Rightarrow h, \;\; f_r(h,t) \Rightarrow r \end{align} \label{eq:gen}\tag{1}\] That is, each function learns to generate one component of the triple given the other two. The intuition behind learning these three functions is as follows. Let us take the above example: (PersonX puts PersonX’s trust in PersonY, How is PersonX seen as?, faithful). The first function \(f_t(h,r)\) learns to generate the answer \(t\) given the context and the question. The second function \(f_h(r,t)\) learns to generate one context where the question and the answer may be valid. The final function \(f_r(h,t)\) is a Jeopardy-style generating the question which connects the context and the answer.
In Multiple-choice QA, given the context, two choices may be true for two different questions. Similarly, given the question, two answer choices may be true for two different contexts. For example, given the context: PersonX puts PersonX’s trust in PersonY, the answers PersonX is considered trustworthy by others and PersonX is polite are true for two different questions How does this affect others? and How is PersonX seen as?. Learning these three functions enables us to score these relations between the context, question, and answers.
2.1 Using KTL to perform QA↩︎
After learning this function in a self-supervised way, we can use them to perform question answering. Given a triple \((h,r,t)\), we define the following scoring function: \[\fontsize{9pt}{9pt}\selectfont \begin{align} & D_t = D(t,f_t(h,r)), \;\; D_h = D(h,f_h(r,t)), \\ & D_r = D(r,f_r(h,t)) \\ & score(h,r,t) = D_t * D_h * D_r \\ \end{align} \label{eq:dist}\tag{2}\] where \(h\) is the context, \(r\) is the question and \(t\) is one of the answer options. \(D\) is a distance function which measures the distance between the generated output and the ground-truth. The distance function varies depending on the instantiation of the framework, which we will study in the following sections. The final answer is selected as: \[ans = \mathop{\mathrm{arg\,min}}_t(score(h,r,t)) \label{eq:score}\tag{3}\] As the scores are the distance from the ground-truth we select the choice that has the minimum score.
We define the different ways we can implement this framework in the following sections.
2.2 Knowledge Representation Learning↩︎
In this implementation, we use Knowledge representation learning to learn equation 1 . In contrast to triplet classification and graph completion, where systems try to learn a score function \(f_r(h,t)\), i.e, is the fact triple \((h,r,t)\) true or false; in this method we learn to generate the inputs vector representations, i.e, \(f_r(h,t) \Rightarrow r\). We can view equation 1 as generator functions, which given the two input encodings learns to generate a vector representation of the third. As our triples \((h,r,t)\) can have a many to many relations between each pair, we first project the two inputs from input encoding space to a different space similar to the work of TransD [24]. We use a Transformer encoder \(Enc\) to encode our triples to the encoding space. We learn two projection functions, \(M_{i1}\) and \(M_{i2}\) to project the two inputs, and a third projection function \(M_o\) to project the entity to be generated. We combine the two projected inputs using a function \(C\). These functions can be implemented using feedforward networks. \[\fontsize{9pt}{9pt}\selectfont \begin{align} & I_{e1} = Enc(I_1), I_{e2} = Enc(I_2), O_e = Enc(O) \\ & I_{e1} = M_{i1}(I_{e1}), I_{e2} = M_{i2}(I_{e2}), O_{p} = M_{o}(O_e)\\ & \Hat{O} = C(I_{e1},I_{e2}) \\ & loss = LossF(\Hat{O},O_{p}) \end{align}\] where \(I_i\) is the input, \(\hat{O}\) is the generated output vector and \(O_p\) is the projected vector. \(M\) and \(C\) functions are learned using fully connected networks. In our implementation, we use RoBERTa as the \(Enc\) transformer, with the output representation of the \([cls]\) token as the phrase representation.
We train this model using two types of loss functions, L2Loss where we try to minimize the L2 norm between the generated and the projected ground-truth, and Noise Contrastive Estimation [25] where along with the ground-truth we have \(k\) noise-samples. These noise samples are selected from other \((h,r,t)\) triples such that the target output is not another true fact triple, i.e, \((h,r,t_{noise})\) is false. The NCELoss is defined as: \[\fontsize{9pt}{9pt}\selectfont \begin{align} & NCELoss(\Hat{O},O_{p},[N_0...N_k]) = \\ & - \log \frac{\exp{sim(\Hat{O},O_{p})}}{\exp{sim(\Hat{O},O_{p})} + \sum_{k \in N}{\exp{(sim(\Hat{O},N_k)}}} \end{align}\] where \(N_k\) are the projected noise samples, \(sim\) is the similarity function which can be the L2 norm or Cosine similarity, \(\hat{O}\) is the generated output vector and \(O_p\) is the projected vector.
The \(D\) distance function 2 for such a model is defined by the distance function used in the loss function. For L2Loss, it is the L2 norm, and in the case of NCELoss, we use \(1-sim\) function.
2.3 Span Masked Language Modeling↩︎
In Span Masked Language Modeling (SMLM), we model the equation 1 as a masked language modeling task. We tokenize and concatenate the triple \((h,r,t)\) with a separator token between them, i.e, \([cls][h][sep][r][sep][t][sep]\). For the function \(f_r(h,t) \Rightarrow r\), we mask all the tokens present in \(r\), i.e, \([cls][h][sep][mask][sep][t][sep]\). We feed these tokens to a Transformer encoder \(Enc\) and use a feed forward network to unmask the sequence of tokens. Similarly, we mask \(h\) to learn \(f_h\) and \(t\) to learn \(f_t\)
We train the same Transformer encoder to perform all the three functions. We use the cross-entropy loss to train the model: \[\fontsize{9pt}{9pt}\selectfont \begin{align} & CELoss(h,r,mask(t),t) = \\ & - \frac{1}{n}\sum_{i=1}^{n}log_2P_{MLM}(t_i|h,r,t_{1..t_i..tn}) \end{align}\] where \(P_{MLM}\) is the masked language modeling probability of the token \(t_i\), given the unmasked tokens \(h\) and \(r\) and other masked tokens in \(t\). Do note we do not do progressive unmasking, i.e, all the masked tokens are jointly predicted.
The \(D\) distance function 2 for this model is same as the loss function defined above.
3 Synthetic Graph Construction↩︎
In this section we describe our method to create a synthetic knowledge graph from a text corpus containing sentences. Not all types of knowledge are present in a structured knowledge graph, such as ATOMIC, which might be useful to answer questions. For example, the questions in QASC dataset [26] require knowledge about scientific concepts, such as, “Clouds regulate the global engine of atmosphere and ocean.". The QASC dataset contains a textual knowledge corpus containing science facts. Similarly, the Open Mind Commonsense (OMCS) knowledge corpus contains knowledge about different commonsense facts, such as,”You are likely to find a jellyfish in a book". Another kind of knowledge about social interactions and story progression is present in several story understanding datasets, such as RoCStories and the Story Cloze Test [27]. To be able to perform question answering using these knowledge and KTL, we create the following two graphs: Common Concept Graph and the Directed Story Graph.
3.0.0.1 Common Concept Graph
In order to create the Common Concept Graph, we extract noun-chunks and verb-chunks from each of the sentences using Spacy Part-of-Speech tagger [28]. We assign all the extracted chunks as the vertices of the graph, and the sentences as the edges of the graph. To generate training samples for KTL, we assign triples \((h,R,t)\) as \((e_1,e_2,v_i)\) where \(v_i\) is the common concept present in both the sentences \(e_1\) and \(e_2\). For example, in the sentence Clouds regulate the global engine of atmosphere and ocean., the extracted concepts are clouds, global engine, atmosphere, ocean and regulate. The triplet assignment will be, [Warm moist air from the Pacific Ocean brings fog and low stratus clouds to the maritime zone., Clouds regulate the global engine of atmosphere and ocean., clouds]. We create two such synthetic graphs using the QASC science corpus and the OMCS concept corpus. Our hypothesis is this graph and the KTL framework will allow the model to understand the concepts common in two facts, which in turn allows question answering.
3.0.0.2 Directed Story Graph
This graph is created using short stories from the RoCStories and Story Cloze Test datasets. This graph is different from the above graph as this graph has a directional property and each individual story graph is disconnected. To create this graph, we take each short story with \(k\) sentences, \([s_1,s_2,s_3..,s_k]\) and create a directed graph such that, all sentences are vertices and each sentence is connected with a directed edge only to sentences that occur after it. For example, \(s_1\) is connected to \(s_2\) with a directed edge but not vice versa. We generate triples \((h,R,t)\) by sampling vertices \((s_i,s_j,s_k)\) such that there is a directed path between the sentences \(s_i\) and \(s_k\) through \(s_j\). This captures a smaller story where the head is an event that occurs before the relation and the tail. This graph is designed for story understanding and abductive reasoning using KTL framework.
3.0.0.3 Random Sampling
There are around 17M sentences in the QASC text corpus, similarly there are 640K sentences in the OMCS text corpus. Our synthetic triple generation leads to a significantly large set of triples that is in order of \(10^{12}\) and more. To restrict the train dataset size for our KTL framework, we randomly sample triples and limit the train dataset size to be at max 1M samples, we refer to this as Random Sampling.
3.0.0.4 Curriculum Filtering
Here we extract the noun and verb chunks from the context, question and answer options present in the question answering datasets. We filter triples from the generated dataset and keep only those triples where at least one of the entities is present in the extracted noun and verb chunks set. This filtering is analogous to a real-life human examination setting where a teacher provides the set of concepts upon which questions would be asked, and the students can learn the concepts. We perform the sampling and filtering only on the huge Common Concept Graphs generated from QASC and OMCS corpus.
| ARC-Easy | ARC-Chall | QASC | OpenBookQA | CommonsenseQA | aNLI | SocialIQA | ||
|---|---|---|---|---|---|---|---|---|
| Train Size | 2251 | 1119 | 8134 | 4957 | 9741 | 169654 | 33410 | |
| Val Size | 570 | 299 | 926 | 500 | 1221 | 1532 | 1954 | |
| Test Size | 2377 | 1172 | 920 | 500 | 1140 | - | - | |
| C Length | - | - | - | - | - | 9 | 15 | |
| Q Length | 19.4 | 22.3 | 13 | 12 | 14 | 9 | 6 | |
| A length | 3.7 | 4.9 | 1.5 | 3 | 1.5 | 9 | 3 | |
| # of Option | 4 | 4 | 8 | 4 | 5 | 2 | 3 | |
| KTL Graph | QASC-CCG | QASC-CCG | QASC-CCG | QASC-CCG | OMCS-CCG | DSG | ATOMIC |
| ATOMIC | QASC-CCG | OMCS-CCG | DSG | |
|---|---|---|---|---|
| Train Size | 893393 | 1662308 | 914442 | 1019030 |
| Val Size | 10000 | 10000 | 10000 | 10000 |
| H Length | 11.2 | 10.5 | 9.6 | 10.3 |
| R Length | 6.5 | 10.3 | 9.4 | 10.2 |
| T Length | 2 | 1.5 | 2 | 10.4 |
4 Datasets↩︎
We evaluate our framework on the following six datasets: SocialIQA [4], aNLI [29], CommonsenseQA [5], QASC [26], OpenBookQA [30] and ARC [31]. SocialIQA, aNLI and CommonsenseQA require commonsense reasoning and external knowledge to be able to answer the questions. Similarly, QASC, OpenBookQA, and ARC require scientific knowledge. Table 1 shows the dataset statistics and the corresponding knowledge graph used to train our KTL model. Table 2 shows the statistics for the triples extracted from the graphs. From the two tables we can observe our KTL triples have different number of words when compared to the target question answering tasks. Especially, where the context is significantly larger and human annotated as in SocialIQA, increasing the challenge for unsupervised learning.
4.1 Question to Hypothesis Conversion and Context Creation↩︎
We can observe the triples in our synthetic graphs, QASC-CCG and OMCS-CCG contain factual statements, and our target question answering datasets have questions that contain wh words or fill-in-the-blanks. We translate each question to a hypothesis using the question and each answer option. To create hypothesis statements for questions containing wh words, we use a rule-based model [32]. For fill-in-the-blank and cloze style questions, we replace the blank or concat the question and the answer option.
For questions that do not have a context, such as in QASC or CommonsenseQA, we retrieve top five sentences using the question and answer options as query and perform retrieval from respective source knowledge sentence corpus. For each retrieved context, we evaluate the answer option score using equation 2 and take the mean score.
| Models | ARC-E \(\uparrow\) | ARC-C \(\uparrow\) | OBQA \(\uparrow\) | QASC \(\uparrow\) | ComQA \(\uparrow\) | aNLI \(\uparrow\) | SocIQA \(\uparrow\) |
|---|---|---|---|---|---|---|---|
| Random | 25.0 25.0 25.0 | 25.0 25.0 25.0 | 25.0 25.0 25.0 | 12.5 12.5 | 20.0 20.0 | 50.0 51.0 | 33.3 33.3 |
| GPT-2 L | 30.5 29.1 29.4 | 23.5 25.1 25.0 | 32.0 26.6 27.8 | 12.3 13.2 | 36.4 37.2 | 50.8 51.3 | 41.2 40.8 |
| RoB-MLM | 29.8 29.6 29.0 | 24.8 25.0 25.0 | 24.8 24.4 25.0 | 12.8 17.6 | 23.6 24.8 | 51.6 52.2 | 35.6 34.5 |
| RoB-FMLM | 31.0 31.2 30.6 | 24.6 22.1 23.8 | 23.4 24.2 23.8 | 14.2 19.7 | 23.2 26.1 | 51.2 51.4 | 36.9 36.1 |
| IR | 29.4 30.4 30.2 | 18.4 20.3 21.2 | 31.4 29.4 28.8 | 18.6 19.4 | 24.6 24.4 | 53.4 54.8 | 35.8 36.0 |
| KRL-L2 | 28.8 29.6 29.8 | 26.7 26.8 25.6 | 29.6 28.8 29.2 | 20.4 20.8 | 31.4 30.6 | 57.6 57.4 | 43.2 43.8 |
| KRL-NCE-L2 | 32.4 31.8 30.6 | 27.2 27.5 26.8 | 33.2 31.6 32.8 | 22.6 23.1 | 33.4 33.8 | 59.3 60.5 | 46.4 46.2 |
| KRL-NCE-Cos | 32.8 32.0 31.8 | 27.4 27.9 27.8 | 35.6 34.8 34.4 | 23.2 24.4 | 36.8 37.1 | 60.4 60.2 | 46.6 46.4 |
| SMLM | 33.2 33.4 33.0 | 27.8 28.4 28.4 | 34.4 34.6 33.8 | 26.6 27.2 | 38.2 38.8 | 64.7 65.3 | 48.7 48.5 |
| Self-Talk | N/A | N/A | N/A | N/A | 32.4 | N/A | 46.2 |
| BIDAF Sup. | 50.1 49.8 | 20.6 21.2 | 49.2 48.8 | 31.8 | 32.0 | 67.8 | 51.2 |
| RoBerta Sup. | 85.0 | 67.2 | 72.0 | 61.8 | 72.1 | 83.2 | 76.9 |
5 Experiments↩︎
5.1 Baselines↩︎
We compare our models to the following baselines.
GPT-2 Large with language modeling cross-entropy loss as the scoring function. We concatenate the context and question and find the cross-entropy loss for each of the answer choices and choose the answer which has the minimum loss.
Pre-trained RoBerta-large used as is, without any finetuning or further pre-training, with scoring same as our defined SMLM model. We refer to it as Rob-MLM.
RoBerta-large model further fine-tuned using the original Masked Language Modeling task over our concatenated fact triples \((h,r,t)\), with scoring same as SMLM. We refer to it as Rob-FMLM.
IR Solver described in ARC [33], which sends the context, question and answer option as a query to Elasticsearch. The top retrieved sentence which has a non-stop-word overlap with both the question and the answer is used as a representative, and its corresponding IR ranking score is used as confidence for the answer. The option with the highest score is chosen as the answer.
5.2 KTL Training↩︎
We train the Knowledge Representation Learning (KRL) model using both L2Loss and NCELoss. For NCELoss we also train it with both L2 norm and Cosine similarity. Both the KRL model (365M) and SMLM model (358M) uses RoBERTa-large (355M) as the encoder. We train the model for three epochs with the following hyper-parameters: batch sizes [512,1024] for SMLM and [32,64] for KRL; learning rate in range: [1e-5,5e-5]; warm-up steps in range [0,0.1]; in 4 Nvidia V100s 16GB. We use the transformers package [34]. All triplets from the training graphs are positive samples. We learn using these triplets. For NCE, we choose \(k\) equal to ten, i.e., ten negative samples. We perform three hyper-parameter trials using ten percent of the training data for each model, and train models with three different seeds. For each of our experiments we report the mean accuracy of the three random seed runs, and report the standard deviation if space permits. Code is available here.
6 Results and Discussion↩︎
6.1 Unsupervised Question Answering↩︎
Table 3 compares our different KTL methods with our four baselines for the six question answering datasets on the zero-shot question answering task. We use Hypothesis Conversion, Curriculum Filtering and Context Creation for ARC, QASC, OBQA and CommonsenseQA for both, the baselines as well as our models. We compare the models on the Train, Dev and Test split if labels are available, to better capture the statistical significance.
We can observe our KTL trained models perform statistically significantly better than the baselines. When comparing the different KRL models, the NCELoss with Cosine similarity performs the best. This might be due to the additional supervision provided by the negative samples as the L2Loss model only tries to minimize the distance between the generated and the target projections. When comparing different KTL instantiations, we can see the SMLM model performs the best overall. SMLM and KRL differ in their core approaches. Our hypothesis is the multi-layered attention in a transformer encoder enables the SMLM model to better distinguish between a true and false statement. In KRL we are learning from both positive and negative samples, but the model still under-performs. On analysis, we observe the random negative samples may make the training task biased for KRL. Our future work would be to utilize alternative negative sampling techniques, such as selecting samples that are closer in contextual vector space.
The improvements on ARC-Challenge task is considerably less. It is observed that the fact corpus for QASC, although contains a huge number of science facts, does not contain sufficient knowledge to answer ARC questions. There is a substantial improvement in SocialIQA, aNLI, QASC and CommonsenseQA as the respective KTL knowledge corpus contains sufficient knowledge to answer the questions. It is interesting to note that for QASC, we can reduce the problem from an eight-way to a four-way classification, as our top-4 accuracy on QASC is above 92%. Our unsupervised model outperforms previous approaches such as Self-Talk [19]. It approaches prior supervised approaches like BIDAF [35], and even surpasses it on two tasks.
| Model | QASC \(\uparrow\) | OBQA \(\uparrow\) | aNLI \(\uparrow\) | ComQA \(\uparrow\) | SocIQA \(\uparrow\) |
|---|---|---|---|---|---|
| RoBerta | 44.5 \(\pm\) 1.2 | 47.8 \(\pm\) 1.4 | 68.8 \(\pm\) 1.3 | 46.4 \(\pm\) 1.5 | 44.4 \(\pm\) 1.2 |
| RoB-MLM | 43.6 \(\pm\) 0.6 | 49.4 \(\pm\) 0.8 | 67.1 \(\pm\) 0.8 | 43.2 \(\pm\) 0.8 | 46.8 \(\pm\) 0.6 |
| KRL-NCE-Cos | 48.2 \(\pm\) 0.9 | 51.2 \(\pm\) 0.6 | 73.4 \(\pm\) 0.9 | 49.5 \(\pm\) 1.1 | 58.6 \(\pm\) 0.8 |
| SMLM | 49.8 \(\pm\) 0.6 | 55.8 \(\pm\) 0.6 | 76.8 \(\pm\) 0.6 | 51.2 \(\pm\)0.7 | 69.1 \(\pm\) 0.4 |
| RoBerta-Sup | 59.40 | 71.0 | 84.3 | 71.4 | 76.6 |
6.2 Few-Shot Question Answering↩︎
Table 4 compares our KTL pre-trained transformer encoder in the few-shot question answering task. We fine-tune the encoder with a simple feedforward network for a \(n\)-way classification task, the standard question answering approach using RoBerta with \(n\) being the number of answer options, while training with only 8% of the training data. We train on three randomly sampled splits of training data and report the mean. We can observe our KTL pre-trained encoders perform significantly better than the baselines, and approaches the fully supervised model, with only 7.5% percent behind the fully supervised model on SocialIQA. We also observe our pre-trained models have a lower deviation.
| Model | QASC \(\uparrow\) | OBQA \(\uparrow\) | ComQA \(\uparrow\) | aNLI \(\uparrow\) | SocIQA \(\uparrow\) |
|---|---|---|---|---|---|
| SMLM - A | 23.4 \(\pm\) 0.6 | 28.6 \(\pm\) 0.7 | 33.6 \(\pm\) 0.5 | 64.8 \(\pm\) 0.9 | 46.2 \(\pm\) 0.7 |
| SMLM - Q | 26.7 \(\pm\) 0.8 | 33.8 \(\pm\) 0.7 | 34.4 \(\pm\) 0.8 | 65.1 \(\pm\) 0.7 | 37.8 \(\pm\) 0.5 |
| SMLM - C | 22.8 \(\pm\) 1.1 | 29.8 \(\pm\) 1.3 | 31.9 \(\pm\) 0.9 | 64.9 \(\pm\) 0.8 | 47.1 \(\pm\) 0.8 |
| SMLM - A*Q*C | 27.2 \(\pm\) 0.6 | 34.6 \(\pm\) 0.8 | 38.8 \(\pm\) 0.6 | 65.3 \(\pm\) 0.7 | 48.5 \(\pm\) 0.6 |
6.3 Ablation studies and Analysis↩︎
6.3.0.1 Effect of Context, Question, Answer Distance
In Table 5 we compare the effect of the three different distance scores. It is interesting to observe, in OpenBookQA, QASC and CommonsenseQA, the three datasets which don’t provide a context, the model is more perplexed to predict the question when given a wrong answer option, leading to higher accuracy for only Question distance score. On the other hand, in aNLI all three distance scores have nearly equal performance. In SocialIQA, the question has the least accuracy, whereas the model is more perplexed when predicting the context given a wrong answer option. This confirms our hypothesis that given a task predicting context and question can contain more information than discriminating between options alone.
6.3.0.2 Effect of Hypothesis Conversion, Curriculum Filtering and Context Retrieval
In Table 6 we observe the effect of hypothesis conversion, curriculum filtering and our context creation. Converting the question to a hypothesis provides a slight improvement, but a major improvement is observed when we filter our KTL training samples and keep only those concepts which are present in the target question answering task, compared to when the KTL model is trained with a random sample of 1M. Curriculum filtering is impactful because there are a lot of concepts which are present in our source knowledge corpus, and the random sampled training corpus only contains 50% of the target question answering task concepts on an average. Another key thing to note in Table 6 is our KTL models can strongly perform like supervised models, when the gold knowledge context is provided, which are available in QASC and OpenBookQA. This indicates a better retrieval system for context creation can further improve our models.
| Model | QASC \(\uparrow\) | OBQA \(\uparrow\) | ComQA \(\uparrow\) |
|---|---|---|---|
| SMLM - Hypo + CF | 27.2 \(\pm\) 0.6 | 34.6 \(\pm\) 0.8 | 38.8 \(\pm\) 0.6 |
| SMLM - Quesn + CF | 26.5 \(\pm\) 1.2 | 32.2 \(\pm\) 1.1 | 35.4 \(\pm\) 1.3 |
| SMLM - Hypo + Rand Sample | 22.6 \(\pm\) 1.4 | 28.4 \(\pm\) 1.5 | 32.2 \(\pm\) 1.4 |
| SMLM - Gold F+ Hypo + CF | 72.4 \(\pm\) 0.8 | 75.2 \(\pm\) 0.7 | - |
6.3.0.3 Effect of Sythetic Triple corpus size
Figure 2 compares our two modelling approaches when we train them with varying number of KTL training samples. NCE refers to our KRL model trained with NCELoss and Cosine similarity. We can observe our KRL model learns faster due to additional supervision, but the SMLM model performs the best, when trained with more samples. The performance tapers after \(10^5\) samples, indicating the models are overfitting to the synthetic data.
6.3.0.4 Error Analysis
We sampled 50 error cases from each of our question answering tasks. Our KTL framework allows learning from knowledge graphs, that includes synthetic knowledge graphs. Both our instantiation, SMLM and KRL function as a knowledge base score generator, where given the inputs and a target the generator yields a score, how improbable is the target to be present in the knowledge base. Most of our errors are when all context, question and answer-option have a large distance scores, and the model accuracy degenerates to that of a random model. This larger distance indicates the model is highly perplexed to see the input text. For aNLI and SocialIQA, we possess relevant context and our performance is significantly better in these datasets, but for other tasks we have another source of error, i.e., context creation. In several cases the context is irrelevant and acts as noise. Other errors include, when the questions require complex reasoning such as understanding negation, conjunctions and disjunctions; temporal reasoning such “6 am” being before “10 am”, and multi-hop reasoning. These complex reasoning tasks are required to answer a significant number of questions in the science and commonsense QA tasks. We also tried to utilize a text generation model, such as GPT-2, to generate and compare with ground truth text using our KTL framework, but preliminary results show the model is overfitting to the synthetic dataset, and has a significantly low performance.
6.3.0.5 Other Instantiations
Our KTL framework can be implemented using other methods, such as using a Generator/Discriminator pre-training proposed in Electra [36], and sequence-to-sequence methods. The distance functions for sequence-to-sequence models can be similar to our SMLM model, the cross-entropy loss for the expected generated sequence. Discriminator based methods can adapt the negative class probabilites as the distance function. Studying different instantiations and their implications are some of the interesting future works.
7 Related Work↩︎
7.1 Unsupervised QA↩︎
Recent work on unsupervised question answering approach the problem in two ways, a domain adaption or transfer learning problem [37], or a data augmentation problem [38]–[41]. The work of [16], [17], [42] use style transfer or template-based question, context and answer triple generation, and learn using these to perform unsupervised extractive question answering. There is also another approach of learning generative models, generating the answer given a question or clarifying explanations and/or questions, such as GPT-2 [20] to perform unsupervised question answering [18], [19], [43]. In contrast, our work focuses on learning from knowledge graphs and generate vector representations or sequences of tokens not restricted to the answer but including the context and the question using the masked language modeling objective.
7.2 Use of External Knowledge for QA↩︎
There are several approaches to add external knowledge into models to improve question answering. Broadly they can be classified into two, learning from unstructured knowledge and structured knowledge. In learning from unstructured knowledge, recent large pre-trained language models [8], [10], [20], [43]–[47] learn general-purpose text encoders from a huge text corpus. On the other hand, learning from structured knowledge includes learning from structured knowledge bases [48]–[52] by learning knowledge enriched word embeddings. Using structured knowledge to refine pre-trained contextualized representations learned from unstructured knowledge is another approach [53]–[56].
Another approach of using external knowledge includes retrieval of knowledge sentences from a text corpora [57]–[63], or knowledge triples from knowledge bases [64], [65] that are useful to answer a specific question. Another recent approach is to use language model as knowledge bases [66], where they query a language model to un-mask a token given an entity and a relation in a predefined template. In our work, we use knowledge graphs to learn a self-supervised generative task to perform zero-shot multiple-choice QA.
7.3 Knowledge Representation Learning↩︎
Over the years there are several methods discovered to perform the task of knowledge representation learning. Few of them are: TransE [67] that views relations as a translation vector between head and tail entities, TransH [68] that overcomes TransE’s inability to model complex relations, and TransD [24] that aims to reduce the parameters by proposing two different mapping matrices for head and tail. KRL has been used in various ways to generate natural answers [69], [70] and generate factoid questions [71]. The task of Knowledge Graph Completion [72] is to either predict unseen relations \(r\) between two existing entities: \((h,?,t)\) or predict the tail entity \(t\) given the head entity and the query relation: \((h,r,?)\). Whereas we are learning to predict including the head, \((?,r,t)\). In KTL, head and tail are not similar text phrases (context and answer) unlike Graph completion. We further modify TransD and adapt it to our KTL framework to perform zero-shot QA.
8 Conclusion↩︎
In this work, we propose a new framework of Knowledge Triplet Learning over knowledge graph entities and relations. We show learning all three possible functions, \(f_r\), \(f_h\), and \(f_t\) helps the model to perform zero-shot multiple-choice question answering, where we do not use question-answering annotations. We learn from both human-annotated and synthetic knowledge graphs and evaluate our framework on the six question answering datasets. Our framework achieves state-of-the-art in the zero-shot question answering task achieving performance like prior supervised work and sets a strong baseline in the few-shot question answering task.
9 Acknowledgements↩︎
The authors acknowledge support from the DARPA SAIL-ON program, and ONR award N00014-20-1-2332. The authors will also like to thank Tejas Gokhale, Arindam Mitra and Sandipan Choudhuri for their feedback on earlier drafts.