1 Introduction↩︎

Large pre-trained language models (LLMs)  [1]–[4] have been shown to attain reasonable zero-shot generalization on a diverse set of NLP tasks including text classification, question answering and text summarization [5]. These models when prompted with natural language description or a common template for the task along with the input, can directly generate the correct output with reasonable accuracy for many natural language processing problems.

LLMs are generally trained on publicly available text [2], [6], [7] and code [3], and they are not expected to generalize to domain-specific semantic parsing tasks in a similar way. The output of semantic parsing comprises user-defined domain-specific functions, argument names, and syntax where the inductive bias from pre-training is less helpful. Instead, we leverage the LLM’s ability to zero-shot answer reading comprehension questions. In this work, we propose \(\mathrm{\small ZeroTOP}\) that decomposes the semantic parsing task into one of answering a series of extractive and abstractive questions, corresponding to its top-level intent and a set of relevant slots. For example, as shown in Figure 1, the utterance is decomposed into multiple questions related to the intent (alarm creation) and slots (parameters of the alarm creation). The answers to these questions can be combined to infer the target meaning representation.

As illustrated in Figure 2, we cast top-level intent classification as an abstractive QA task. One intuitive way to predict intent is to constrain autoregressive LMs to search over only valid intent labels. However, LLMs are known to be biased towards predicting labels common in the pretraining data  [8]. Therefore, we propose to generate an intent description in an unconstrained manner from an LLM and infer the intent label most similar to the generated description.

We view slot value prediction as an extractive QA problem. Most utterances do not mention all the slots. For example, in the MTOP dataset, on average, only one-third of possible slots are mentioned per utterance. It is therefore essential for the model to abstain from prediction when corresponding slots are not mentioned. Through our analyses, we observe that most LLMs frequently hallucinate text for missing slots with high confidence, resulting in poor performance in slot value prediction. To address this, we fine-tune a language model on a collection of public QA datasets augmented with synthetic unanswerable samples. We call our trained model Abstainer, as it is capable of identifying unanswerable questions and abstaining from prediction. We hierarchically prompt for nested slots using the Abstainer, and infer nested intents if their corresponding slots are detected.

Our contributions are summarized as follows:

• We propose \(\mathrm{\small ZeroTOP}\) that leverages LLMs for zero-shot semantic parsing by decomposing the task into answering a series of questions.

• To address the LM’s bias towards specific intent labels, we propose intent description generation in an unconstrained manner and then selecting most similar intent label to the description.

• We show that current pre-trained LLMs fail to abstain from prediction for unanswerable questions. We address this by finetuning a language model called Abstainer on public QA datasets augmented with synthetically generated unanswerable QA pairs. Abstainer can be successively applied to generate nested meaning representations.

• Our method significantly outperforms pre-trained LLMs on the MTOP dataset.

2 Related Work↩︎

Large pre-trained language models are increasingly used to support semantic parsing in low-data scenarios.  [9] constrain GPT-3 and BART so that the LMs generate valid canonical representations that can be mapped back to meaning representations.  [10], [11] explore prompt tuning for semantic parsing with LLMs.  [12] decomposes canonical utterance generation into sub-clause generation and augments the generated sub-clauses into a canonical utterance.  [13] presents a cross-schema parser for several tasks in a given vertical by augmenting schema-specific context to the input along with the utterance. The closest work to ours is  [14] where they decompose parsing into abstractive QA tasks and aggregate answers to construct a meaning representation. These methods assume access to some data either from the same domain, in the same format from a different domain or synthetically generated from a synchronous grammar. In contrast, we focus on a strict zero-shot setting where only the schema information is available along with some natural language prompts for schema entities.

Our work is also related to approaches towards zero-shot dialog state tracking using language models [15]–[17]. Specifically,  [16] uses an Abstainer similar to ours to handle missing slots. Our method differs in that, we focus on semantic parsing where the Abstainer needs to be applied multiple times along with intent detection to create nested meaning representations. Our approach does not require enumerating choices in the prompt allowing us to handle a large number of intents and slots as found in semantic parsing datasets like MTOP.

3 \(\mathrm{\small ZeroTOP}\): Zero-Shot Task-Oriented Semantic Parsing↩︎

3.1 Problem Formulation↩︎

We focus on task-oriented parsing with hierarchical intent-slot schema. Let \(\mathcal{I} = \{\mathcal{I}_1, \mathcal{I}_2, \ldots, \mathcal{I}_n\}\) be the set of all possible top-level intents and \(\mathcal{S} = \{\mathcal{S}_1, \mathcal{S}_2, \ldots, \mathcal{S}_m\}\) be the set of all possible slots. Each intent \(\mathcal{I}_j\) has a set of slots \(\mathcal{S}^j = \{\mathcal{S}^j_1, \mathcal{S}^j_2, \ldots, \mathcal{S}^j_n\}\) that can be filled. Possible slots in an intent are represented by the intent-to-slot mapping \(\mathrm{I2S}\): \(\mathcal{I}\) \(\rightarrow\) \(\mathcal{P}(\mathcal{S})\), where \(\mathcal{P}(\cdot)\) is the powerset operator that generates the set of all subsets. Similarly, the inverse slot-to-intent mapping is represented by \(\mathrm{S2I}\): \(\mathcal{S}\) \(\rightarrow\) \(\mathcal{I}\). The input in a zero-shot setting contains no training data, however, we assume access to the intent-slot, slot-intent mappings \(\mathrm{I2S}\), \(\mathrm{S2I}\). Our method requires users to provide a question per slot, that is representative of their purpose. Let \(\mathcal{Q} = \{\mathcal{Q}_{\mathcal{S}_1}, \mathcal{Q}_{\mathcal{S}_2}, \ldots, \mathcal{Q}_{\mathcal{S}_k}\}\) represent respective questions for slots. In a real-life setting, this can be easily obtained from the domain developer who designed the schema.

3.2 Unconstrained Generation for Zero-Shot Intent Classification↩︎

We view zero-shot intent classification as an abstractive question-answering problem and use pre-trained LLMs to answer them. One intuitive way to predict the intent is to prime the LLM with a QA prompt and then constrain the generation to search over only valid intent labels [9]. However, LLMs are known to be biased towards certain text sequences [8] more common in pretraining data. For example, in the MTOP dataset, the constrained T0-3B model predicts CREATE\(\_\)CALL (make call) as intent for \(92\%\) of the data in the call domain. Therefore, we propose to first generate intent description in an unconstrained fashion and then choose the label that is most similar to the generated answer. Specifically, we prime the LM with the below prompt and let the LM generate without any constraints. Then, we choose the most similar label based on cosine similarity between intent label and generated text using RoBERTa sentence similarity  [18]. We find our proposed approach does not exhibit bias towards certain labels as noticed with constrained generation.

3.3 Leveraging QA datasets for Slot Value Prediction↩︎

Slot value prediction involves extracting phrases for a slot from the user utterance. Thus, we cast this as an extractive QA problem. All slots might not be mentioned in an input utterance. The QA model needs to abstain from prediction for the corresponding questions for the missing slots. To analyze the abstaining capability of pre-trained QA models, we consider some of the top-performing zero-shot models T0-3B [1], GPT-3 [2], and Codex [3] and experiment on a 500 sample subset of unanswerable questions from the SQuAD dataset [19]. We constrain the models to generate either from the context or from a set of manually created phrases indicating that the question cannot be answered. We use the prompts and phrases mentioned in  [1]–[3] and observe the accuracy of all models to be \(< 5\%\). We notice that the LLMs frequently hallucinate and generate answers for unanswerable questions. In our experiments section 4.5, we also consider a log-likelihood-based threshold for abstaining and we show that this threshold is difficult to tune using public QA datasets.

To address this challenge, we leverage multiple publicly available extractive and abstractive QA datasets² to train Abstainer, a QA model capable of abstaining from prediction. Specifically, we generate synthetic unanswerable training samples by modifying existing QA data, and train a QA model jointly on the existing datasets and the synthetic unanswerable questions. For every (question, answer, context) triplet, we generate synthetic unanswerable questions by either (1) removing the sentence containing the answer span from the context, or (2) randomly sampling a context that doesn’t have the same question. For example, as shown in Figure 3, we generate the first unanswerable sample by removing the sentence “Tagore was born in Calcutta” containing the answer Calcutta from the context and second unanswerable sample by switching the context of two original answerable samples. We augment these synthetic unanswerable training samples to the existing datasets and train our Abstainer. After training the Abstainer, we prompt it for each slot with its corresponding question for slot value prediction. The prompt has the following format:

3.3.0.1 Nested Intents

To identify nested intents, we assume knowledge of the candidate nested intents that can be accommodated by each slot, represented by the slot-to-candidate-nested-intent mapping \(\mathrm{S2NI}\): \(\mathcal{S}\) \(\rightarrow\) \(\mathcal{P}(\mathcal{I})\). Our method assumes that the depth of output representations is at most 4 i.e. nested intents cannot further have more nested intents.

Since we assume access to slots that accommodate nested intents, we query for possible nested intents for all such slots from their respective slot values. One intuitive way is to prompt the LLM for nested intent with the intent prediction prompt. However, our unconstrained generation-based intent model would predict many false positive nested intents. We instead use Abstainer to prompt for their respective slots. If any slot value is identified, we consider its corresponding intent via slot-to-intent mapping \(\mathrm{S2I}\) to be present as well.

3.4 \(\mathrm{\small ZeroTOP}\): Putting it all together↩︎

The pseudo-code of the proposed pipeline is mentioned in Algorithm-4. \(\mathrm{\small ZeroTOP}\) employs a top-down, greedy prompting strategy, where we first prompt for intent and then, their respective slots. First, we obtain the top-level intent using the intent model. Based on the predicted intent, we prime the Abstainer for corresponding slots using their respective questions as prompts, to identify their slot values. For each identified slot value, we prompt the Abstainer for slots of candidate nested intents. We use the same prompt format for this step with the identified slot value now considered as the input utterance. Finally, we combine predicted intent, identified slot values, and nested intents to create the meaning representation. We illustrate the pipeline with an example in Figure 2.

4 Experiments↩︎

In this section, we evaluate \(\mathrm{\small ZeroTOP}\) against multiple pre-trained LLMs under zero-shot setting.

4.1 Datasets↩︎

We experiment on the English language subset of MTOP [20] dataset. MTOP is a multilingual task-oriented semantic parsing dataset comprising data from 6 languages and 11 domains. The dataset details are mentioned in Table 1. On average, each intent has 3.6 slots and only one-third of possible slots are filled for an utterance.

4.2 Baselines↩︎

We compare \(\mathrm{\small ZeroTOP}\) with high-performing zero-shot LLMs as both intent and slot models. We consider the following models for zero-shot intent prediction:

1. T0-3B, GPT-3, Codex constrained are pre-trained T0-3B, GPT-3, Codex models that are primed with intent generation prompt and the output is constrained to search over valid intent labels.

2. T0-3B constrained calibrated is same as above T0-3B constrained that is calibrated for intent labels following  [8]. Specifically, we estimate the model’s bias towards each intent label by prompting with a content-free test input such as "N/A" and fit calibration parameters accordingly to cause the prediction for this input to be uniform across answers.

3. RoBERTa-base similarity is an ablated version of \(\mathrm{\small ZeroTOP}\) where we assign intent labels based only on their similarity with user utterance using the RoBERTa sentence transformer (stsb-roberta-base).

4. \(\mathrm{\small ZeroTOP}\)-Intent is our proposed intent prediction method that generates intents in an unconstrained fashion by priming T0-3B with intent generation prompt and assigns labels based on their similarity with intent labels using the RoBERTa sentence transformer (stsb-roberta-base).

We consider the following models for slot prediction:

1. GPT-3, Codex, T0-3B constrained are pre-trained GPT-3, Codex, T0-3B models that are primed with the question corresponding to slot and the output is constrained to be either from the user utterance or from a set of phrases indicating that the question cannot be answered. We use their corresponding phrases mentioned in  [1]–[3]. We use the OpenAI API text-davinci-001 for GPT-3 and code-davinci-002 for Codex.

2. Abstainer is our finetuned T0-3B model that abstains from prediction. It is primed with the question corresponding to the slot and the output is constrained to be either from the user utterance or from the set of phrases from [1] indicating that the question is unanswerable.

4.3 Experiment Settings↩︎

We evaluate on the zero-shot setting, therefore we have no training data. We manually create questions for slots \(\mathcal{Q}\) by looking at one example per slot. For training Abstainer, we fine-tune T0-3B on the extractive and abstractive QA datasets for \(1\) epoch with a constant learning rate of \(10^{-4}\). The Abstainer is fine-tuned on \(411732\) answerable and \(435898\) unanswerable samples. The batch size is \(32\) and each batch contains an equal number of answerable and unanswerable samples. We used 8 \(\times\) NVIDIA Tesla V100 for our experiments. We use complete meaning representation match accuracy as the performance metric.

4.4 Results and Discussion↩︎

We present intent classification results in Table 2 and complete meaning representation evaluation results in Table 3.

From Table 2, we observe that ZeroTOP-Intent performs significantly better than constrained T0-3B, GPT-3, and Codex for zero-shot intent classification. We found that constrained T0-3B is biased towards certain labels. For example, it predicts CREATE\(\_\)CALL (make call), SEND\(\_\)Message (send message), and CREATE\(\_\)REMINDER (create reminder), as intent for more than \(90\%\) of the data in call, message, reminder domains. Our proposed unconstrained formulation lets the model freely express the intent and, computing similarity later with the intent labels addresses this bias.

As shown in Table 3, the combination of \(\mathrm{\small ZeroTOP}\)-Intent model and Abstainer demonstrates superior performance than alternative combinations. We observe that T0-3B, GPT-3 and Codex all fail to abstain frequently. The T0-3B model abstains only for \(38\%\) of unanswerable slot questions whereas our Abstainer is able to abstain for \(89\%\) of the unanswerable questions. As a result, we observe a notable performance gain by plugging in Abstainer as the slot model for each intent model baseline. This abstaining capability compounded with unconstrained generation-based intent prediction results in a significant performance gain.

4.5 Confidence score-based Abstainer study↩︎

We can alternatively have LLMs abstain from prediction based on a confidence score based threshold. We consider negative log likelihood (NLL) of the predicted slot value as the confidence score and abstain from prediction if it is greater than the threshold. We experiment on slot value prediction task with T0-3B, Codex, and GPT3 as LLMs and plot macro F1 scores for multiple NLL thresholds on a randomly sampled subset of 500 samples from MTOP dataset in Figure 5 (a). Specifically, we consider the gold intent of each sample and prime LLM for extracting slot values for each slot of the gold intent. We consider F1 score as the metric due to the label imbalance across possible slot values. We present the F1-score of the Abstainer for reference. First, we observe that Abstainer is significantly better than T0-3B and GPT3 for all confidence thresholds. Second, we notice that there is no threshold that consistently results in good performance for all LLMs, which implies that this has to be individually tuned for each LLM. Finally, we observe Codex performs better than Abstainer for some thresholds. As our problem setting includes no annotated data, we investigate whether we can infer the optimal threshold for Codex using public QA datasets. Specifically, we consider 500 answerable and 500 unanswerable QA pairs from SQuAD dataset and plot F1 scores with a range of confidence thresholds in Figure 5 (b). We can observe that the performance on answerable and unanswerable subsets is mutually exclusive i.e. there is no threshold where the performance on both answerable and unanswerable subsets is high. The range of thresholds that result in the best performance on the whole set (highlighted in green) does not transfer to MTOP and is achieved at the cost of unanswerable set where the F1 score is less than 5%. Given the difficulty in tuning threshold and the API costs of Codex, we believe using Abstainer as the slot model to be a better choice.

4.6 Annotation Effort Analysis↩︎

We use 74 samples i.e. one example per slot to design questions for slots. To analyze the annotation effort, we train an end-to-end parser using these 74 samples and compare it against our method. We fine-tune T5-3B model [7] on these samples with user utterance as input and its corresponding meaning representation as output. The complete match accuracy on the MTOP dataset is shown in Table 4. We observe that \(\mathrm{\small ZeroTOP}\) performs significantly better than the parser trained on these 74 samples, justifying the annotation effort to create questions for \(\mathrm{\small ZeroTOP}\).

4.7 Greedy vs Beam search↩︎

In our proposed method, we employ a greedy strategy where we hierarchically prompt for top-level intent and for its corresponding slots. We compare it with the beam search strategy with beam size \(k\) for every prediction. Specifically, we consider \(k\) top-level intents and prompt for their corresponding slots, consider top-\(k\) slot values for every slot and finally compute the best meaning representation based on their aggregated NLL scores. The NLL score of intent \(\mathcal{I}_m\), its slots \(\mathcal{S}_j \in \mathrm{I2S}(\mathcal{I}_m)\), and their corresponding slot values \(\mathrm{slotValues}[\mathcal{S}_j]\) is aggregated as follows: \[\small \begin{align} \alpha \log p(\mathcal{I}_m)+ (1 - \alpha)\sum_{\mathcal{S}_j \in \mathrm{I2S}(\mathcal{I}_m)} \log p(\mathrm{slotValues}[\mathcal{S}_j] | \mathcal{I}_m) \end{align}\] where \(\alpha\) is tuned on a held-out validation set. Note that \(p(\mathrm{slotValues} | \mathcal{I}_m)\) is computed recursively for its nested intents. The performance comparison is reported in Table 5. We observe that beam search can further improve the performance when validation data is provided.

5 Conclusion↩︎

In this paper, we explore zero-shot semantic parsing using large language models. We propose \(\mathrm{\small ZeroTOP}\) that decomposes semantic parsing task into abstractive and extractive QA tasks and leverage LLMs to answer them. For identifying top-level intent, we view it as an abstractive QA task and propose to generate an answer in an unconstrained fashion and infer the intent label most similar to the generated description. We demonstrate that current LLMs fail to abstain from prediction for unanswerable questions, which leads to poor performance. To address this challenge, we train Abstainer using public QA datasets, that is capable of identifying unanswerable questions and abstaining from prediction. Extensive experiments on the MTOP dataset show significant improvements over pre-trained LLMs.

6 Appendix↩︎

6.1 QA Datasets for Training Abstainer↩︎

We use multiple extractive and abstractive QA datasets to generate synthetic unanswerable samples and train Abstainer. The details about datasets are mentioned in Table 6.

References↩︎