1 Introduction↩︎

Natural language generators (NLGs) for task-oriented dialogue take meaning representations (MRs) as inputs, i.e. a set of dialogue acts with attributes and their values, and output natural language utterances realizing the MR. Current NLGs are trained end-to-end with a corpus of MR/utterance pairs where the MRs cover a specific set of dialogue acts and domain attributes. Creation of such datasets is labor intensive and time consuming. However, when building an NLG for a new domain ontology, it should be possible to re-use data built on existing domain ontologies. If this were possible, it would speed up development of new dialogue systems significantly.

Here we experiment with one version of this task by building a new domain ontology based on combining two existing ontologies, and utilizing their training data. Each dataset is based on a different domain ontology in the restaurant domain, with novel attributes and dialogue acts not seen in the other dataset, e.g. only one has attributes representing family friendly and rating information, and only one has attributes for decor and service. Our aim is an NLG engine that can realize utterances for the extended combined ontology not seen in the training data, e.g. for MRs that specify values for family friendly, rating, decor and service. Figure [blend-example] illustrates this task. Example E1 is from a training set referred to as NYC, from previous work on controllable sentence planning in NLG [@reed2018can], while E2 is from the E2E NLG shared task [@Novikovaetal17]. As we describe in detail in Section 2, E1 and E2 are based on two distinct ontologies. Example E3 illustrates the task addressed in this paper: we create a test set of novel MRs for the combined ontology, and train a model to generate high quality outputs where individual sentences realize attributes from both ontologies.

To our knowledge, this is a completely novel task. While it is common practice in NLG to construct test sets of MRs that realize attribute combinations not seen in training, initial experiments showed that this task is surprisingly adversarial. However, methods for supporting this type of generalization and extension to new cases would be of great benefit to task-oriented dialogue systems, where it is common to start with a restricted set of attributes and then enlarge the domain ontology over time. New attributes are constantly being added to databases of restaurants, hotels and other entities to support better recommendations and better search. Our experiments test whether existing data that only covers a subset of attributes can be used to produce an NLG for the enlarged ontology.

We describe below how we create a test set — that we call com — of combined MRs to test different methods for creating such an NLG. A baseline sequence-to-sequence NLG model has a slot error rate (SER) of .45 and only produces semantically perfect outputs 3.5% of the time. To improve performance, we experiment with three different ways of conditioning the model by incorporating side constraints that encode the source of the attributes in the MR [@Sennrich_Haddow_Birch_2016; @harrison2019maximizing]. However, this only increases the proportion of semantically perfect model outputs from 3.5% to 5.5% (Section 4.1).

We then propose and motivate a novel self-training method that greatly improves performance by learning from the model mistakes. An error analysis shows that the models do produce many combined outputs, but with errorful semantics. We develop a rule-based text-to-meaning semantic extractor that automatically creates novel correct MR/text training instances from errorful model outputs, and use these in self-training experiments, thus learning from our mistakes (Section 4.2). We validate the text-to-meaning extractor with a human evaluation. We find that a model trained with this process produces SERs of only .03, and semantically perfect outputs 81% of the time (a 75.4 percent improvement). A human evaluation shows that these outputs are also natural, coherent and grammatical. Our contributions are:

• Definition of a novel generalization task for neural NLG engines, that of generating from unseen MRs that combine attributes from two datasets with different ontologies;

• Systematic experiments on methods for conditioning NLG models, with results showing the effects on model performance for both semantic errors and combining attributes;

• A novel self-training method that learns from the model’s mistakes to produce semantically correct outputs 81% of the time, an absolute 75.4% improvement.

We start in Section 2 by defining the task in more detail, describe our models and metrics in Section 3, and results in Section 4. We discuss related work throughout the paper where it is most relevant and in the conclusion in Section 5.

2 Ontology Merging and Data Curation↩︎

We start with two existing datasets, NYC and E2E, representing different ontologies for the restaurant domain. The NYC dataset consists of 38K utterances [@reed2018can; @Orabyetal18], based on a restaurant ontology used by Zagat [@Stentetal02; @StentPrasadWalker04].² The E2E dataset consists of 47K utterances distributed for the E2E Generation Challenge [@Novikovaetal17].³ Each dataset consists of pairs of reference utterances and meaning representations (MRs). Figure [blend-example] shows sample MRs for each source and corresponding training instances as E1 and E2.

We first make a new combined ontology onto-com by merging NYC and E2E. Attributes, dialogue acts, and sample values for E2E and NYC are illustrated on the left-hand side of Figure 2, and the result of merging them to create the new ontology is on the right-hand side of Figure 2. Since there are only 8 attributes in each source dataset, we developed a script by hand that maps the MRs from each source into the onto-com ontology.

As Figure 2 shows, both datasets have the inform dialogue act, and include the attributes name, cuisine, location, and price after mapping. The unique attributes for the NYC ontology are scalar ratings for service, food quality and decor. The NYC dataset also has the recommend dialogue act, seen in E1 in Figure [blend-example]. The unique attributes of the E2E ontology are customer rating, eat type (“coffee shop”), near and family friendly.

Given the combined ontology onto-com, we then map the training data for both E2E and NYC into onto-com by relabelling the MRs to have consistent names for shared attributes as illustrated in Figure 2. We create a balanced training set of \(\sim\)​77K from the two original datasets by combining all NYC references with a random same-size sample of E2E references.

We then manually create a test set, COM, consisting of 3040 MRs based on the new combined ontology onto-com. Each test MR must have at least one attribute from E2E and one attribute from NYC so that it combines attributes from both sources: these MRs provide combinations never seen in training.⁴ Example E3 in Figure [blend-example] provides an example test MR. The procedure for creating the test set ensures that the length and complexity of the test set are systematically varied, with lengths normally distributed and ranging from 3 to 10 attributes. Recommendations only occur in the NYC training data, and they increase both semantic and syntactic complexity, with longer utterances that use the discourse relation of justification [@Stentetal02], e.g. Babbo is the best because it has excellent food. We hypothesize that recommendations may be more challenging to combine across domains, so we vary MR complexity by including the recommend dialogue act in half the test references. We show in Section 4 that the length and complexity of the MRs is an important factor in the performance of the trained models.

3 Experimental Overview and Methods↩︎

Given the training and test sets for the combined ontology in Section 2, we test 4 different neural model architectures and present results in Section 4.1. We then propose a a novel self-training method, and present results in Section 4.2. These experiments rely on the model architectures presented here in Section 3.1, and the Text-to-Meaning semantic extractor and performance metrics in Section 3.2.

3.1 Model Architectures↩︎

In the recent E2E NLG Challenge shared task, models were tasked with generating surface forms from structured meaning representations (MRs) [@duvsek2020evaluating]. The top performing models were all RNN encoder-decoder systems. Here we also use a standard RNN Encoder–Decoder model [@sutskever2014sequence] that maps a source sequence (the input MR) to a target sequence (the utterance text). We first implement a baseline model and then add three variations of model supervision that aim to improve semantic accuracy. All of the models are built with OpenNMT-py, a sequence-to-sequence modeling framework [@klein2017opennmt].

Encoder. The MR is represented as a sequence of (attribute, value) pairs with separate vocabularies for attributes and values. Each attribute and each value are represented using 1-hot vectors. An (attribute, value) pair is represented by concatenating the two 1-hot vectors.

The input sequence is processed using two single layer bidirectional-LSTM [@hochreiter1997long] encoders. The first encoder operates at the pair level, producing a hidden state for each attribute-value pair of the input sequence. The second LSTM encoder is intended to produce utterance level context information in the form of a full MR encoding produced by taking the final hidden state after processing the full input sequence. The outputs of both encoders are combined via concatenation. That is, the final state of the second encoder is concatenated onto each hidden state output by the first encoder. The size of the pair level encoder is 46 units and the size of the MR encoder is 20 units. Model parameters are initialized using Glorot initialization [@glorot2010understanding] and optimized using Stochastic Gradient Descent with mini-batches of size 128.

Decoder. The decoder is a uni-directional LSTM that uses global attention with input-feeding. Attention weights are calculated via the general scoring method [@luong2015effective]. The decoder takes two inputs at each time step: the word embedding of the previous time step, and the attention weighted average of the encoder hidden states. The ground-truth previous word is used when training, and the predicted previous word when evaluating. Beam search with five beams is used during inference.

Supervision. Figure 3 shows the baseline system architecture as well as three types of supervision, based on conditioning on source (E2E, NYC) information. The additional supervision is intended to help the model attend to the source domain information. We call the three types of supervision guide, attr and bool, and the baseline architecture nosup, representing that it has no additional supervision.

The supervision methods are shown in Figure 4. The source feature has a vocabulary of three items: nyc, e2e and both. Since both is never seen in train, the source information is represented using two booleans: True\(||\)False denotes a reference from E2E while False\(||\)True denotes a reference from NYC. This encoding is intended to encourage generalization at inference time. During inference, blending of information from both sources is specified by using True\(||\)True. The attr supervision method represents the source information by concatenating the boolean source token onto each attribute as seen in Figure 4. This redundantly represents the source information locally to each attribute, which has been effective for tasks such as question generation and stylistic control [@harrison2018neural; @harrison2019maximizing]. The bool supervision method adds the boolean source token to the end of the sequence of attribute-value pairs as its own attribute, as in work on machine translation and controllable stylistic generation [@Sennrich_Haddow_Birch_2016; @yamagishi2016controlling; @Ficler_Goldberg_2017]. The guide model inputs the source information directly to the decoder LSTM. In previous work, putting information into the decoder in this way has yielded improvements in paraphrase generation and controllable generation [@Iyyeretal18; @harrison2019maximizing]

3.2 Text-to-Meaning Semantic Extractor↩︎

Much previous work in NLG relies on a test set that provides gold reference outputs, and then applies automatic metrics such as BLEU that compare the gold reference to the model output [@Papinenietal02; @duvsek2020evaluating], even though the limitations of BLEU for NLG are widely acknowledged [@BelzReiter06; @stent2005evaluating; @NovikovaetalEVAL17; @Liuetal16b]. To address these limitations, recent work has started to develop “referenceless” NLG evaluation metrics [@dusek2018referenceless; @kann2018sentence; @tian2018treat; @MehriEskenazi-FED].

Since there are no reference outputs for the COM test set, we need a referenceless evaluation metric. We develop a rule-based text-to-MR semantic extractor (TTM) that allows us to compare the input MR to an MR automatically constructed from an NLG model textual output by the TTM, in order to calculate SER, the slot error rate. The TTM system is based on information extraction methods. We conduct a human evaluation of its accuracy below. A similar approach is used to calculate semantic accuracy in other work in NLG, including comparative system evaluation in the E2E Generation Challenge [@Juraskaetal18; @duvsek2020evaluating; @WisemanSR17; @shen2019pragmatically].

The TTM relies on a rule-based automatic aligner that tags each output utterance with the attributes and values that it realizes. The aligner takes advantage of the fact that the recommend dialogue act, and the attributes and their values are typically realized from a domain-specific finite vocabulary. The output of the aligner is then used by the TTM extractor to construct an MR that matches the (potentially errorful) utterance that was generated by the NLG. We refer to this MR as the “retrofit MR". The retrofit MR is then compared to the input MR in order to automatically calculate the slot error rate SER:

where \(D\) is the number of deletions, \(R\) is the number of repetitions, \(S\) is the number of substitutions, \(H\) is the number of hallucinations and \(N\) is the number of slots in the input MR [@Nayaketal17; @reed2018can; @wenetal15]. Section 6.1 in the supplementary materials provides more detail and examples for each type of semantic error. SER is first calculated on individual utterances and then averaged over the whole test set. For additional insight, we also report the percentage of semantically perfect outputs (perfect%), outputs where the SER is 0 and there are no semantic errors. This measure is analogous to the Sentence Error Rate used in speech recognition.

We evaluated the TTM and the automatic SER calculation with a separate experiment where two NLG experts hand-labelled a random sample of 200 model outputs. Over the 200 samples, the automatic SER was .45 and the human was .46. The overall correlation of the automatic SER with the human SER over all types of errors (D,R,S,H) is .80 and the correlation with deletions, the most frequent error type, is .97.

The TTM is critical for our novel self-training method described in Section 4.2. The retrofit MRs match the (errorful) NLG output: when these MR/NLG output pairs combine attributes from both sources, they provide novel corrected examples to add back into training.

4 Results↩︎

We run two sets of experiments. We first run all of the NLG models described in Section 3.1 on the COM test set, and automatically calculate SER and perfect% as described in Section 3.2. We report these results in Section 4.1. Section 4.2 motivates and describes the self-training method and presents the results, resulting in final models that generate semantically perfect outputs 83% of the time.

4.1 Initial Model Results↩︎

Table ¿tbl:table:supervision-evaluation? summarizes the results across the four models nosup, guide, attr and bool. Overall, the results show that the task, and the COM test set, are surprisingly adversarial. All of the models have extremely high SER, and the SER for nosup, attr, and bool are very similar. Row 2 shows that the guide model has much worse performance than the other models, in contrast to other tasks [@Iyyeretal18]. We do not examine the guide model further. Row 3 shows that the attr supervision results in the largest percentage of perfect outputs (5.5%).

The results in Table ¿tbl:table:supervision-evaluation? should be compared with the baselines for testing nosup on only E2E or NYC in Table ¿tbl:table:baselines-combo-nocombo?. Both the E2E and NYC test sets consist of unseen inputs, where E2E is the standard E2E generation challenge test [@duvsek2020evaluating], and NYC consists of novel MRs with baseline attribute frequencies matching the training data.⁵ Rows 1 and 3 test models trained on only E2E or only NYC, while Rows 2 and 4 test the same trained nosup model used in Row 1 of Table ¿tbl:table:supervision-evaluation? on E2E or NYC test sets respectively. Comparing Rows 1 and 2 shows that training on the same combined data used in Table ¿tbl:table:supervision-evaluation? slightly degrades performance on E2E, however, this SER is still considerably lower than the .45 SER for the nosup model tested on the COM test set, shown in the first row of Table ¿tbl:table:supervision-evaluation?. Row 4 shows that the nosup model trained on the combined data appears to improve performance on the NYC test because the perfect% goes up from 69% in Row 3 to 71%. The SER of .06 shown in Row 4 should also be compared to the .45 SER reported for the nosup model in the first row of Table ¿tbl:table:supervision-evaluation?. These results taken together establish that the combined MRs in the COM test provide a very different challenge than the E2E and NYC unseen test inputs.

However, despite the poor performance of the initial models, we hypothesized that there may be enough good outputs to experiment with self-training. Since the original training data had no combined outputs, decoding may benefit from even small numbers of training items added back in self-training.

The automatic SER results provide insight into the semantic accuracy of the models, but no assessment of other aspects of performance. We thus conduct a human evaluation on Mechanical Turk to qualitatively assess fluency, coherency and grammaticality. We use the automatic SER to select 100 semantically perfect references from the nosup and the attr models’ test outputs, and the 86 perfect references from bool. We ask 5 Turkers to judge on a scale of 1 (worst) to 5 (best) whether the utterance is: (1) fluent and natural; (2) semantically coherent; and (3) grammatically well-formed. Table ¿tbl:table:human-evaluation? reports the average score for these qualitative metrics as well as the Turker agreement, using the average Pearson correlation across the Turkers. The results show that the agreement among Turkers is high, and that all the models perform well, but that the attr model outputs are the most natural and coherent, while the bool model outputs are the most grammatical.

4.2 Self-Training↩︎

In order to conduct self-training experiments, we need perfect outputs that combine attributes from both sources to add back into training. These outputs must also be natural, coherent and grammatical, but Table ¿tbl:table:human-evaluation? shows that this is true of all the models. A key idea for our novel self-training method is that the TTM (Section 3.2) automatically produces “retrofit” corrected MRs that match the output texts of the NLG models. Thus we expect that we can construct more perfect outputs for self-training by using retrofitting than those in Table ¿tbl:table:supervision-evaluation?. Here, we first analyse the outputs of the initial models to show that self-training is feasible, and then explain our method and present results.

An initial examination of the outputs suggests that the models simply have trouble combining attributes from both sources. We provide examples in Table ¿tbl:table:error-sentences? in Section 6.2 in the supplementary materials. To quantify this observation, we define a metric, Source Blending Rate (SB), that counts the percentage of outputs that combine attributes from both sources, whether or not the attribute values are accurate:

where \(R_{sb}\) is the count of references \(r\) that contain an attribute \(a_i\) \(\subseteq\;source_1\) and another attribute \(a_j\) \(\subseteq\;source_2\), and N is the total number of references. Only attributes that appear uniquely in each source are included in the \(a_i\), \(a_j\): the unique attributes are illustrated in Figure 2.

Figure 5 graphs SB as a function of MR length showing that indeed the models do in many cases produce combined outputs and that the type of model supervision greatly influences SB. The nosup model is the worst: a fact that is masked by the nosup model’s SER in Table ¿tbl:table:supervision-evaluation?, which appears to be on a par with both attr and bool. Interestingly, all models are more likely to produce an SB output as the MRs get longer, but Figure 5 shows clearly that the bool model especially excels.

For self-training, we also need a model that generates utterances with the recommend dialogue act. As mentioned in Section 2, recommendations increase both semantic and syntactic complexity. Half the test items contain a recommendation, so we need a model that can produce them. Table ¿tbl:rec-table? presents results for SER and SB depending on whether a recommend was in the MR, showing that the three models vary a great deal. However, the bool row for the SB column shows that when the MR includes a recommendation, the bool model produces a combined output far more frequently than nosup or attr (SB = .73).

Thus Figure 5 and Table ¿tbl:rec-table? show that the bool model produces the most combined outputs. After TTM extraction, the bool model provides the most instances (1405) of retrofit MR/output pairs to add to self-training, and we therefore use bool in the self-training experiments below.

Table ¿tbl:self-training-retrofitting-example? illustrates how the TTM works, and shows that it can effectively create a new MR that may not have been previously seen in training, allowing the model to learn from its mistakes. The caption for Table ¿tbl:self-training-retrofitting-example? explains in detail the retrofitting process and how it leads to new examples to use in self-training.

It is important to note that the retrofit MRs for some NLG outputs cannot be used for self-training. NLG model outputs whose semantic errors include repetitions can never be used in self-training, because valid MRs do not include repeated attributes and values, and the method doesn’t edit the NLG output string. However, deletion errors cause no issues: the retrofit MR simply doesn’t have that attribute. Substitutions and hallucinations can be used because the retrofit MR substitutes a value or adds a value to the MR, as long as the realized attribute value is valid, e.g. “friendly food” is not a valid value for food quality.⁶\(^{,}\)⁷

To begin the self-training experiments, we apply the source-blending metric (SB) defined above to identify candidates that combine attributes from both sources, and then apply the TTM to construct MRs that match the NLG model outputs, as illustrated in Table ¿tbl:self-training-retrofitting-example?, eliminating references that contain a repetition. We start with the same combined 76,832 training examples and the 1405 retrofit MR/NLG outputs from the bool model. We explore two bootstrapping regimes, depending on whether a model output is a repetition of one that we have already seen in training. One model keeps repetitions and adds them back into training, which we dub S-Repeat, and the other model only adds unique outputs back into training, which we dub S-Unique.

Figure 6 shows how the SER and perfect% continuously improve on the COM test set for S-Repeat over 10 rounds of self-training, and that S-Repeat has better performance, indicating that adding multiple instances of the same item to training is useful. The performance on the COM test set of the S-Unique model flattens after 8 rounds. After 10 rounds, the S-Repeat model has an SER of .03 and produces perfect outputs 82.9% of the time, a 77.4 percent absolute improvement over the best results in Table ¿tbl:table:supervision-evaluation?.

Since the the self-training procedure used the COM test set during self-training, we construct a new test with 3040 novel MRs using the procedure described in Section 2, which we call COM-2. First we test the initial models on COM-2, resulting in a best SER of 0.45 for the bool model, identical with the result for COM. For perfect% the best result was 5.3% on the attr model, which is again comparable to the original COM test set. We then tested the final self-trained model on COM-2, with the result that the SER for S-Repeat (0.03) and S-Unique (0.11) are again identical to the result for COM. The perfect% is comparable to that reported in Figure 6; it decreases by 2.2% for S-Repeat to 80.7% and increases by .2% for S-Unique to 50.7%. Overall, the performance on COM-2 improved by an absolute 75.4%.

Figure 7 shows that the results improve, not only overall, but also by MR length. It plots the SER and perfect% results, by MR length, for the bool model before and after self-training. While the perfect% decreases as the number of attributes increase, there is a large improvement over the initial model results. Also, after self-training the worst perfect% is still above 0.5, which is higher than perfect% for any MR length before self-training. The SER also improves over all MR lengths after self-training, not exceeding .06, significantly better than even the shortest MR before self-training.⁸

We also performed a human evaluation on Mechanical Turk to assess the qualitative properties of the model outputs after self-training. We selected 100 perfect references for S-Repeat and 100 for S-Unique and used the same HIT as described in Section 4.1. Table ¿tbl:table:strain-human-evaluation? reports the average score for these qualitative metrics as well as the Turker agreement, using the average Pearson correlation across the Turkers. The results show that naturalness, coherence and grammaticality are still high after self-training for both models, but that the S-Unique model produce better outputs from a qualitative perspective. We believe we could improve the self-training method used here with additional referenceless evaluation metrics that aim to measure naturalness and grammaticality [@MehriEskenazi-FED]. We leave this to future work.

.

Table ¿tbl:table:qual-example-sentences? provides outputs from the models that display different ways of combining attributes from the original sources. In Row 1 we can see that the recommend dialogue act from NYC can be combined in the same sentence as the attributes family friendly and eat type from E2E and aggregate these E2E attributes with NYC attributes decor and food quality using a “with" operator. Row 2 shows another example where the NYC and E2E attributes are joined using a”with" operator. In Row 3 there is a single sentence with four attributes where the NYC attribute is preceded and followed by E2E attributes. Row 4 concatenates the two sources in a single sentence using sentence coordination. The “east village" location from the NYC dataset, is concatenated with the attributes near from E2E and service from NYC. These examples show that the NLG models can combine attributes from both sources in many different ways. Table ¿tbl:table:st-example-sentences? in Section 6.4 provides additional detail by providing examples along with their corresponding MRs.

5 Conclusion↩︎

[@KedzieMcKeown19; @shah2018bootstrapping] [@budzianowski2018multiwoz; @eric2019multiwoz; @gavsic2015policy; @hakkani2016multi; @Cervoneetal19; @shah2018bootstrapping; @ultes2017pydial; @chen2017deep]

This paper presents the first experiments on training an NLG for an extended domain ontology by re-using existing within-domain training data. We show that we can combine two training datasets for the restaurant domain, that have different ontologies, and generate output that combines attributes from both sources, by applying a combination of neural supervision and a novel self-training method. While it is common practice to construct test sets with unseen attribute combinations, we know of no prior work based on constructing a new combined ontology. Our experiments show that the task is surprisingly adversarial, consistent with recent work suggesting that neural models often fail to generalize [@Wallaceetal19; @Fengetal18; @Ribeiroetal18; @goodfellow2014explaining]. Work on domain transfer shares similar goals to the experiments presented here [@wen2016multi; @golovanov2019large], but these methods do not produce NLG outputs that integrate attributes from two different sources into the same sentence. Our final results show that the ability of our self-training method to automatically construct new training instances results in high quality natural, coherent and grammatical outputs with high semantic accuracy.

In future, we hope to generalize our novel self-training method to build an NLG that can combine two distinct domains, e.g. hotels or movies combined with restaurants in multi-domain dialogue [@budzianowski2018multiwoz; @gavsic2015policy; @hakkani2016multi; @Cervoneetal19; @ultes2017pydial]. Ideally systems that cover multiple domains should be able to produce utterances that seamlessly integrate both domains, if data exists for each domain independently. However, there may be additional challenges in such combinations. Our results require the initial neural models to generate some combined outputs. It is not clear whether there are some aspects of our experimental setup that facilitate this, e.g. it may require some attributes to be shared across the two initial ontologies, or some shared vocabulary. Thus it is possible that initial models for two more distinct domains may not produce any combined outputs, and it may be necessary to seed the self-training experiments with a small number of combined training instances. We leave these issues to future work.

6 Supplementary Materials: Learning from Mistakes: Combining Ontologies via Self-Training for Dialogue Generation↩︎

6.1 Types of Semantic Errors↩︎

The TTM is tuned to identify 4 common neural generation errors: deletions (failing to realize a value), repetitions (repeating an attribute), substitutions (mentioning an attribute with an incorrect value), and hallucinations (introducing an attribute that was not in the original MR at all).

Table ¿tbl:table:semantics-sentences? illustrates each of these types of semantic errors. Row 1 shows deletions of cuisine, price and near which are in the MR but not in the realization. Row 2 demonstrates a repetition, where location and decor are both repeated. Decor is realized with two different lexical values, “good ambiance" and”good decor". There is a substitution in Row 3 where the MR states that the food quality is “bad”, but food quality is realized as “good”. Finally, Row 4 has a hallucination, service is not in the MR but it in the second sentence of the realization.

6.2 Example Errorful NLG Model Outputs↩︎

Table ¿tbl:table:error-sentences? provides examples of NLG model output utterances with high SERs. It illustrates how the NLG models struggle to combine attributes from the two ontologies which is required by all the input MRs (Column SB). It also illustrates cases where it is not possible to produce a valid retrofit MR that can be added back into training during self-training (Column Valid). In most cases these are due to many repetitions. Row 1 is an example where there is no source blending and since it has a repetition (price) it cannot be used for self-training (valid = no). Row 1 also illustrates an ungrammatical realization of price which we have no way to automatically detect at present it is in the high price. Row 2 has three deletions as well as two repetitions. The output repeats It is in midtown three times in a row. Row 3 has five errors, it does not realize the dialogue act recommend and has deleted three other attributes and it hallucinations food quality. While this is a significant number of errors, this realization can still be used in self-training, since none of its errors are repetitions. Row 4 has all four types of errors. It deletes cuisine, decor and service, it realizes a value for family friendly twice with different values, a substitution and finally it hallucinates food quality. Row 5 actually has more errors than slots. It deletes all but two of its attributes: name and rating. It also hallucinates food quality and repeats rating.

6.3 Performance on E2E and NYC test sets↩︎

Table ¿tbl:table:baselines-combo-nocombo? provided a baseline for nosup’s performance before self-training on the original test sets for E2E and NYC. We also verify that the self-trained model performs well after self-training. Table ¿tbl:table:self-baselines-combo-nocombo? shows that self-training improves the results for the original E2E and NYC test sets.

6.4 Example Final Model Outputs↩︎

Table ¿tbl:table:st-example-sentences? provides outputs from the final iteration of self-training that display different ways of combining different attributes from the ontologies. Row 1 shows that the model can combine attributes from the two sources in the same sentence, with attributes from each source, decor and rating, appearing in a single sentence with and. Row 2 shows a different way of combining attributes from the two sources, with family friendly and food quality, in a single sentence, this time using with. In Row 3 we can see that the model can also generate complex sentences for recommendations using the marker because. Also, the attribute used in the because clause is from E2E i.e. family friendly but such sentences never appear in the original E2E training data. The last row shows a complex sentence where decor is combined with eat type and customer rating, again a novel combination.

---

1. \(\:\:\)Work done prior to joining Amazon.↩︎

2. http://nlds.soe.ucsc.edu/sentence-planning-NLG↩︎

3. http://www.macs.hw.ac.uk/InteractionLab/E2E/↩︎

4. The train and test data are available at http://nlds.soe.ucsc.edu/source-blending-NLG↩︎

5. Previous work on the E2E dataset has also used seq2seq models, with SOA results for SER of 1% [@duvsek2020evaluating], but here we do not use the full training set. Our partition of the NYC dataset has not been used before, but experiments on comparable NYC datasets have SERs of .06 and .02 [@reed2018can; @harrison2019maximizing].↩︎

6. We applied the human evaluation in Section 3.2 to instances included in self-training: the correlation between human judgements and the automatic SER is .95, indicating that the retrofit MRs are highly accurate.↩︎

7. Table ¿tbl:table:error-sentences? in Section 6.2 provides additional examples of errorful outputs that can or cannot be used in self-training.↩︎

8. Performance results for the self-trained model on the original E2E and NYC test sets in supplement A.3 shows that performance also improves on the E2E and NYC test sets.↩︎