1 Introduction↩︎

End-to-end reading comprehension models have been increasingly successful at extractive question answering. For example, performance on the SQuAD 2.0 [1] benchmark has improved from 66.3 F1 to 89.5¹ in a single year. However, the Discrete Reasoning Over Passages (DROP) [2] dataset demonstrates that as long as there is quantitative reasoning involved, there are plenty of relatively straightforward questions that current extractive QA systems find difficult to answer. Other recent work has shown that even state-of-the-art neural models struggle with numerical operations and quantitative reasoning when trained in an end-to-end manner [3], [4]. In other words, even BERT [5] is not very good at doing simple calculations.

In this work, we extend an extractive QA system with numerical reasoning abilities. We do so by asking the neural network to synthesize small programs that can be executed. The model picks among simple programs of the form \(\texttt{Operation}(\text{args}, ...)\), where the possible operations include span extraction, answering yes or no, and arithmetic. For math operations, the arguments are pointers to numbers in the text and, in the case of composition, other operations. In this way, the burden of actually doing the computation is offloaded from the neural network to a calculator tool. The program additionally provides a thin layer of interpretability that mirrors some of the reasoning required for the answer. For example, in Table 1, the model predicts subtraction (\(\texttt{Diff}\)) over two numbers in the passage, and executes it to produce the final answer.

We start with a simple extractive question answering model based on BERT [5], and show the following:

1. Predicting unary and binary math operations with arguments resulted in significant improvements on the DROP dataset.

2. Our model can smoothly handle more traditional reading comprehension inputs as well as math problems with new operations. Co-training with the CoQA [6] dataset improved performance on DROP. The DROP+CoQA trained model had never seen multiplication or division examples, but can learn to predict these two ops when appropriate in a math word problem setting [7] with very few training examples.

2 Background and Related Work↩︎

2.0.0.1 Discrete Reasoning over Paragraphs (DROP)

[2] is a reading comprehension task that requires discrete reasoning. Inspired by semantic parsing tasks where models need to produce executable ‘programs’, it keeps the open-domain nature of reading comprehension tasks such as SQuAD 2.0 [1]. As shown in Table 1, the system needs to perform fuzzy matching between “from Europe” and “European nationals” in order to identify the arguments.

2.0.0.2 Numerically-aware QANet (NAQANet)

[2] is the current state-of-the-art² system for DROP. It extends the QANet model [8] with predictions for numbers (0–9) and summation operations. For the latter, it performs a 3-way classification (plus, minus, and zero) on all the numbers in the passage. While certain binary operations are expressible efficiently with flat sign prediction, it is difficult to generalize the architecture. Moreover, each number is tagged independently, which can cause global inconsistencies; for instance, in Table 1 it assigns a single minus label and no plus labels, leading to a prediction of negative people.

2.0.0.3 Mathematical Word Problems

have been addressed with a wide variety of datasets and approaches; see for an overview. One such dataset of arithmetic problems is the Illinois dataset [7]. The problems are posed in simple natural language that has a specific, narrow domain, For example: “If there are 7 bottle caps in a box and Linda puts 7 more bottle caps inside, how many bottle caps are in the box?”. Unlike DROP, the problems are typically 1–3 sentences long and do not require reading complex passages. Instead, the main challenge is mathematical reasoning. According to , the current state of the art uses syntactic parses and deterministic rules to convert the input to logical forms [9].

.5em

3 Model↩︎

We extend a BERT-based extractive reading comprehension model with a lightweight extraction and composition layer. For details of the BERT architecture see . We only rely on the representation of individual tokens that are jointly conditioned on the given question \(Q\) and passage \(P\). Our model predicts an answer by selecting the top-scoring derivation (i.e. program) and executing it.

3.0.0.1 Derivations

We define the space of possible derivations \(\mathcal{D}\) as follows:

• Literals: \(\{ \mathrm{\small Yes}, \mathrm{\small No}, \mathrm{\small Unknown}, 0,\ldots9\}\).

• Numerical operations: including various types of numerical compositions of numbers³, such as Sum or Diff.

• Text spans: composition of tokens into text spans up to a pre-specified length.

• Composition of compositions: we only consider two-step compositions, including merging text spans and nested summations.

The full set of operations are listed in Table 2. For example, Sum is a numerical operation that adds two numbers and produces a new number. While we could recursively search for compositions with deep derivations, here we are guided by what is required in the DROP data and simplify inference by heavily restricting multi-step composition. Specifically, spans can be composed into a pair of merged spans (Merge), and the sum of two numbers (Sum) can subsequently be summed with a third (Sum3). The results in Table 3 show the dev set oracle performance using these shallow derivations, by answer type.

3.0.0.2 Representation and Scoring

For each derivation \(d\in \mathcal{D}\), we compute a vector representation \(\mathbf{h}_d\) and a scalar score \(\rho(d, P, Q)\) using the BERT output vectors. The scores \(\rho\) are used for computing the probability \(P(d\mid P, Q)\) as well as for pruning. For brevity, we will drop the dependence on \(P\) and \(Q\) in this section.

Literals are scored as \(\rho(d) = \mathbf{w}_d^{\intercal} \mathrm{\small MLP}_{\text{lit}} (\mathbf{h}_{\texttt{CLS}})\), where \(\mathbf{h}_{\texttt{CLS}}\) is the output vector at the \(\texttt{[CLS]}\) token of the BERT model [5].

Numeric operations use the vector representations \(\mathbf{h}_i\) of the first token of each numeric argument. Binary operations are represented as \[\begin{align} \mathbf{h}_{d} &= \mathrm{\small MLP}_{\text{binary}} (\mathbf{h}_i, \mathbf{h}_j, \mathbf{h}_i \circ \mathbf{h}_j) \label{eq:binaryrep} \end{align}\tag{1}\] and scored as \(\rho(d) = \mathbf{w}_{op}^{\intercal} \mathbf{h}_{d}\), where \(\mathbf{h}_{d}\) represents the binary arguments and \(op\) is the operation type. \(\circ\) is the Hadamard product. Unary operations such as \(\texttt{Diff100}\) are scored as \(\mathbf{w}_{op}^{\intercal}\mathrm{\small MLP}_{\text{unary}} (\mathbf{h}_i)\).

Text spans are scored as if they were another binary operation taking as arguments the start and end indices \(i\) and \(j\) of the span [10]: \[\begin{align} \mathbf{h}_{d} &= \mathrm{\small MLP}_{\text{span}} (\mathbf{h}_i, \mathbf{h}_j) \label{eq:spanrep} \end{align}\tag{2}\] and scored as \(\rho(d) = \mathbf{w}_{\text{span}}^{\intercal} \mathbf{h}_{d} \notag\).

Compositions of compositions are scored with the vector representations of its children. For example, the ternary \(\texttt{Sum3}\), comprising a \(\texttt{Sum}\) and a number, is scored with \(\mathbf{w}_{\texttt{Sum3}}^{\intercal} \mathrm{\small MLP}_{\texttt{Sum3}} (\mathbf{h}_{d0}, \mathbf{h}_k)\), where \(\mathbf{h}_{d0}\) corresponds to the representation from the first \(\texttt{Sum}\), and \(\mathbf{h}_k\) is the representation of the third number. The composition of two spans is scored as \(\mathbf{w}_{\texttt{Merge}}^{\intercal} \mathrm{\small MLP}_{\texttt{Merge}} (\mathbf{h}_{d0}, \mathbf{h}_{d1}, \mathbf{h}_{d0} \! \circ \! \mathbf{h}_{d1})\), where \(\mathbf{h}_{d0}\) and \(\mathbf{h}_{d1}\) are span representations from 2 . The intuition for including \(\mathbf{h}_{d0}~\circ~\mathbf{h}_{d1}\) is that it encodes span similarity, and spans with similar types are more likely to be merged.

This strategy differs from the NAQANet baseline in a few ways. One straightforward difference is that we use BERT as the base encoder rather than QANet. A more meaningful difference is that we model all derivations in the unified op scoring framework described above, which allows generalizing to new operations, whereas NAQANet would require more large-scale changes to go beyond addition and subtraction. Generalizing the model to new ops is a case of extending the derivations and scoring functions. In Section 4, we will show the impact of incrementally adding Diff100, Sum3, and Merge.

3.1 Training↩︎

We used exhaustive pre-computed oracle derivations \(\mathcal{D}^*\) following . We marginalized out all derivations \(d^*\) that lead to the answer⁴ and minimized: \[\begin{align} \mathcal{J}(P, Q, \mathcal{D}^*) & = - \log \sum_{d^* \in \mathcal{D}^*} P(d^* \mid P, Q) \nonumber \\ P(d \mid P, Q) & = \frac{\exp \rho(d, P, Q)}{\sum_{d'} \exp \rho(d', P, Q)} \nonumber \end{align}\] If no derivation lead to the gold answer (\(\mathcal{D}^*\) is empty), we skipped the example.

3.1.0.1 Pruning

During inference, the \(\texttt{Merge}\) and \(\texttt{Sum3}\) operations are composed from the results of \(\texttt{Span}\) and \(\texttt{Sum}\) operations, respectively. The space of possible results of \(\texttt{Merge}\) is quadratic in the number \(|\mathcal{S}|\) of possible spans. With \(|\mathcal{S}| \sim 10^4\), the complete set of \(\texttt{Merge}\) instances becomes overwhelming. Similarly, with \(|\mathcal{N}| \sim 100\) numbers in each passage, there are millions of possible \(\texttt{Sum3}\) derivations. To do training and inference efficiently, we kept only the top 128 \(\texttt{Span}\) and \(\texttt{Sum}\) results when computing \(\texttt{Merge}\) and \(\texttt{Sum3}\).⁵

3.1.0.2 Spurious ambiguities

Of the answers for which we could find at least one oracle derivation, 36% had two or more alternatives. During training, the model became effective at resolving many of these ambiguities. We monitored the entropy of \(P(d^* \mid P, Q)\) for the ambiguous examples as training progressed. At the start, the entropy was 2.5 bits, which matches the average ambiguous oracle length of \(\sim 6\) alternatives. By the end of 4 epochs, the average entropy had dropped to \(<0.2\) bits, comparable to a typical certainty of 95–99% that one of the derivations is the correct one.

4 Experiments↩︎

Our main experiments pertain to DROP [2], using DROP and, optionally, CoQA [6] data for training. Pre-processing and hyperparameter details are given in the supplementary material. In addition to full DROP results, we performed ablation experiments for the incremental addition of the Diff100, Sum3, and Merge operations, and finally the CoQA training data. We ran on the CoQA dev set, to show that the model co-trained on CoQA can still perform traditional reading comprehension. To investigate our model’s ability to do symbolic reasoning at the other extreme, we performed few-shot learning experiments on the Illinois dataset of math problems [7].

4.1 DROP Results↩︎

As shown in Table 3, our model achieves over 50% relative improvement (over 33% absolute) over the previous state-of-the-art NAQANet system. The ablations indicate that the improvements due to the addition of extra ops (Diff100, Sum3, Merge) are roughly consistent with their proportion in the data. Specifically, the Diff100 and Sum3 derivations increase the oracle performance by 8.7% and 1.4% respectively, corresponding to model improvements of roughly 9% and 1.1%, respectively. Answers requiring two spans occur about 2.8% of the time, which is a 60.4% proportion of the Spans answer type. Merge only improves the Spans answer type by 9%, which we think is due to the significant 11:1 class imbalance between competing single and multiple spans. As a result, multiple spans are under-predicted, leaving considerable headroom there.

Pre-training on CoQA then fine-tuning on DROP lead to our best results on DROP, reported in Table 3. After fine-tuning on DROP, the model forgot how to do CoQA, with an overall F1 score of 52.2 on the CoQA dev set. If one prefers a model competent in both types of input, then the forgetting can be prevented by fine-tuning on both CoQA and DROP datasets simultaneously. This resulted in dev set F1 scores of 82.2 on CoQA and 81.1 on DROP. The CoQA performance is decent and compares well with the pre-trained model performance of 82.5. The 0.5% drop in DROP performance is likely attributable to the difference between pre-training versus fine-tuning on CoQA.

We ensembled 6 models (3 seeds \(\times\) 2 learning rates) for an additional 1% improvement.

4.2 Results on Math Word Problems↩︎

We trained our model on the Illinois math word problems dataset [7], which contains answers requiring multiplication and division—operations not present in DROP—as well as addition and subtraction, in roughly equal proportion. Given the small (\(N=562\)) dataset size, training and evaluation is done with five-fold cross-validation on a standardized set of splits. As shown in Table 4, when we added Mul and Div to our basic DROP operations, the model was able to learn to use them. Transferring from the DROP dataset further improved performance beyond that of , a model specific to math word problems that uses rules over dependency trees. Compared to other more general systems, our model outperforms the deep reinforcement learning based approach of .

5 Conclusions and Future Work↩︎

We proposed using BERT for reading comprehension combined with lightweight neural modules for computation in order to smoothly handle both traditional factoid question answering and questions requiring symbolic reasoning in a single unified model. On the DROP dataset, which includes a mix of reading comprehension and numerical reasoning, our model achieves a 33% absolute improvement over the previous best. The same model can also do standard reading comprehension on CoQA, and focused numerical reasoning on math word problems. We plan to generalize this model to more complex and compositional answers, with better searching and pruning strategies of the derivations.

Acknowledgements↩︎

We would like to thank Chris Alberti, Livio Baldini Soares, and Yoon Kim for tremendously helpful discussions, and we are grateful to all members of the Google Research Language team.

References↩︎