1 Introduction↩︎

Prompting large language models performs decently well in a variety of domains [1], [2]. However, for most of theses evaluation benchmarks, all the information provided in the problem description is relevant to the problem solution, as the problems in exams. This is different from real-world situations, where problems usually come with several pieces of contextually related information, which may or may not be relevant to the problems that we want to solve. We have to identify what information is actually necessary during solving those problems. Studies in psychology have shown that irrelevant information may significantly decrease some children and even adults problem-solving accuracy [3]–[5]. In this work, we study the distractibility of large language models for various prompting techniques; i.e., how is large language model prompting affected by irrelevant context, and what strategies can be used to improve performance? To measure distractibility, we construct the GSM-IC dataset, a grade-school math problem dataset derived from GSM8K [6] and introduce two different metrics. In contrast to prior work that derives benchmark variations by substituting sentences of the base problems with variations [7], [8], we keep the base problem description and add to it one irrelevant sentence, while making sure that it does not affect the solution of the problem (1).

We use Codex (code-davinci-002) and GPT-3.5 ( text-davinci-003) in the GPT3 model family to evaluate state-of-the-art prompting techniques on GSM-IC,² including chain-of-thought prompting [9], zero-shot chain-of-thought prompting [10], least-to-most-prompting [11], and prompting with programs [2]. We find that their performance on GSM-IC greatly decreases compared to the original GSM8K (without irrelevant context). We then investigate several approaches to mitigate this weakness, including self-consistency [12] and adding irrelevant information to the exemplars in the prompt. In addition to demonstrating how to handle irrelevant information via exemplars, we also investigate the usage of task-specific instructions [13]–[17], where we prepend an instruction sentence “feel free to ignore irrelevant information in the problem description” to the exemplars. We summarize our key findings below:

1. All investigated prompting techniques are sensitive to irrelevant information in the problem description. In particular, among the original problems that can be solved by baseline prompts with greedy decoding, no more than \(18\%\) of them can be consistently solved for all types of irrelevant information, showing that the large language model is easily distracted and produces inconsistent predictions when adding a small amount of irrelevant information to the problem description.

2. Self-consistency improves the performance of all prompting techniques on GSM-IC. In particular, the recall rate of the correct answer for GSM-IC is as high as 99.7% with 20 samples per problem, i.e., at least one of the 20 solutions result in the correct final answer, which means that using multiple samples allows the model to almost always retrieve the correct answer.

3. Adding irrelevant information to the exemplars shown in the prompt consistently boosts the performance, and the same holds for adding an instruction to ignore irrelevant context. This suggests that language models are—to some extent—able to learn to ignore irrelevant information by following examples or instructions.

4. We identify different factors of the irrelevant information that affect the model’s sensitivity to irrelevant context. Our breakdown analysis shows that varying the numbers in the irrelevant information does not notably change the model performance, while the degree of lexical overlap with the original problem description matters.

Filtering out irrelevant information is essential for handling real-world tasks. Our evaluation indicates that despite the strong performance on challenging reasoning problems, state-of-the-art language models still have fundamental weaknesses in context understanding and identifying the relevant information from the input. Our findings suggest that in order to gain a more holistic understanding of the reasoning capability of language models, future work should also consider the model sensitivity to irrelevant context, in addition to solving more challenging problems.

2 Related Work↩︎

Few-shot prompting. Few-shot prompting [1], [2] has been significantly boosted with various techniques, including generating intermediate steps [6], [9], [16], [18]–[20], problem decomposition [11], [21]–[24], generating programs [2], [25]–[27], marginalizing intermediate steps that share the same result [12], [28], and ensemble [21], [29]. In addition, [10] demonstrate that appropriate hint in prompts also leads to decent performance, even without any exemplar. In this work, we examine these cutting-edge prompting techniques [9]–[12] on our benchmark, and demonstrate that they are sensitive to irrelevant input context.

Natural language benchmarks with input perturbations. There has been a long line of work on adding input perturbations for natural language tasks, including model-agnostic input transformations [30], [31] and adversarial example generation against individual models [32]–[35]. In particular, prior work has constructed arithmetic reasoning benchmarks through paraphrasing or rewriting sentences in the base problems from clean datasets [7], [8]. Meanwhile, [30] evaluate various large language models under several metrics, including accuracy, robustness, fairness, etc. Specifically, the input transformations in their robustness evaluation include semantics-preserving and semantics-altering perturbations, such as injecting typos and modifying sentences to change the ground-truth classification labels. In contrast the above work where the meaning of problem descriptions may be changed with perturbations, we keep all sentences in the original problem description, and introduce an irrelevant sentence that is ensured not to affect the standard answer.

Natural language benchmarks with irrelevant input context. [32] have shown that neural question answering systems are largely affected by adversarial distracting sentences, whereas follow up work [36], [37] proposes learning strategies that mitigate the problem. Similar issues have been found for general-purpose pretrained language models, on the tasks of factual reasoning [38]–[41], code generation [42], and syntactic generalization [43]. In particular, [41] evaluated T5 [44] and PaLM [2] with few-shot prompts, and proposed knowledge-aware finetuning that finetunes the model on problems with counterfactual and irrelevant context, which strengthens the model robustness to noisy context. In our evaluation, we show that without training or finetuning, adding irrelevant context into demonstrations in the prompt also mitigates the distractibility of the underlying language model and significantly improves the model performance on our GSM-IC benchmark.

There exist some logical reasoning benchmarks that contain irrelevant content in task descriptions [45]–[49]. However, previous work largely focuses on designing models that require extra training, and prompting alone still hardly achieves the same level of performance as finetuned models for these tasks [48], [50]. In our work, we focus on arithmetic reasoning, where prompting techniques have achieved the state-of-the-art results, e.g., on GSM8K, while we show that adding a single irrelevant sentence into the problem description significantly degrades the performance.

Prompting with noisy ground truth. A line of work studies the model performance with incorrect prompting exemplars, i.e., the example problems are paired with wrong answers [51], [52]. In addition, prior work has investigated the model sensitivity to other parts of the prompt, such as instruction tuning with misleading and irrelevant instructions [53] and wrong reasoning steps in the examples [54], [55]. In particular, [54] conclude that the correctness of numbers and equations in chain-of-thought prompts does not play a key role in model performance, but using wrong entities and removing either equations or text explanation in the reasoning steps drastically hamper the performance. Different from this line of work, we always include correct answers to example problems in the prompt, and ensure that the irrelevant context added to the problem description does not change the ground truth answer. We show that the model performance significantly drops when presented with irrelevant context in problem descriptions, and different distributions of numbers and entities in the irrelevant context also lead to different levels of performance degradation.

3 The GSM-IC Dataset↩︎

In this section, we introduce the creation process of the GSM-IC dataset (3.1) and the evaluation metrics (3.2).

3.1 Dataset Creation↩︎

We randomly choose 1,000 problems from the GSM8K training set as a development set. To construct our base dataset, we then choose 100 problems from this development set that can be correctly solved by at least one of the prompting techniques mentioned in this paper;³ that is, our base dataset is an “easy” subset of GSM8K (2). Each base problem requires two to seven reasoning steps to solve.⁴ Among the 100 base problems, 60 of them can be solved with two reasoning steps. The full dataset statistics can be found in 7.

We then generate the examples of our new dataset by adding to each base problem one sentence containing irrelevant information. We use a template-based method (1) to generate these sentences, which can be characterized by the following three factors:

• Topic of the inserted sentence. We write templates for both in-topic and off-topic sentences. In-topic sentences are closely related to the topic of the original problem, whereas off-topic sentences are about a different topic.

• Role name overlap. Most sentence templates contain some role name blanks, which can be filled with names that may or may not overlap with the role names that occur in the problem. For blank fillers that have overlap with original role names, we: (1) randomly pick a role name A from the original problem description and (2) create the blank fillers with template such as A’s father and A’s sister.

• Range of numbers. Since we focus on arithmetic reasoning, most sentence templates also contain a number blank. We can choose to fill in the number blank with a number of similar or different magnitude to those in the original problem description. Concretely, for a number \(a\), if there exists a number \(b\) in the original problem description or solution such that \(\frac{1}{10} \leq \frac{a}{b} \leq 10\), we consider \(a\) as an in-range number, and otherwise an out-of-range number. Since the standard answer to GSM8K problems are all positive integers, we only consider positive integers as the number blank fillers.

We manually verify that (1) all the generated sentences are acceptable in English and that (2) adding them does not affect the standard solution of the base problem. Because the above factors are orthogonal, we generate for each base example a set of derived examples with different factor combinations. The full GSM-IC benchmark consists of 58,052 examples. More details about the dataset creation process can be found in 7.

3.2 Evaluation Metrics↩︎

For a problem \(p\), we denote its standard solution by \(s(p)\), and the solution of method \(\mathcal{M}\) by \(\mathcal{M}(p)\). To evaluate the distractibility of \(\mathcal{M}\), we consider the following two metrics:

• Micro accuracy \(\textit{Acc}_\textit{micro}(\mathcal{M}; \mathcal{P})\) is the average accuracy of method \(\mathcal{M}\) over all the test problems \(\mathcal{P}\). \[\begin{align} \textit{Acc}_\textit{micro}(\mathcal{M}; \mathcal{P}) = \frac{ \sum_{p \in \mathcal{P}} \mathbb{1}\left[ \mathcal{M}(p) = s(p) \right] }{ |\mathcal{P}| } \end{align}\] This means that the micro accuracy weighs all the individual test problems equally.

• Macro accuracy \(\textit{Acc}_\textit{macro}(\mathcal{M}; \mathcal{B})\) is the average accuracy of method \(\mathcal{M}\) over classes of test problems, where each class \(\mathcal{P}(b)\) consists of the set of test examples derived from the base example \(b \in \mathcal{B}\). We define \(\mathcal{M}\)’s prediction for a class \(\mathcal{P}(b)\) to be correct if and only if \(\mathcal{M}\)’s prediction for all problems in this class are correct. \[\begin{align} \textit{Acc}_\textit{macro}(\mathcal{M}; \mathcal{B}) = \frac{ \sum_{b\in \mathcal{B}} \mathbb{1}\left[ \bigwedge_{p\in \mathcal{P}(b)}\left[ \mathcal{M}(p) = s(p) \right] \right] }{ |\mathcal{B}| } \end{align}\] This means that the macro accuracy is the fraction of base problems that can be consistently solved no matter what irrelevant sentence is being added.

• Normalized accuracy measures how a method is affected by the distractors, considering its accuracy on base problems. For a micro or macro accuracy \(a_\mathcal{M}\) achieved by method \(\mathcal{M}\), we calculate its corresponding normalized accuracy by \[\begin{align} \textit{norm}(a_\mathcal{M}; \mathcal{M}) = \frac{a_\mathcal{M}}{n_\mathcal{M}}, \end{align}\] where \(n_\mathcal{M}\) denotes the base problem accuracy of method \(\mathcal{M}\) (2).

4 Investigated Solutions↩︎

In the following section, we review the investigated prompting techniques (4.1), present the formats of our prompts (4.2), and introduce instructed prompting (4.3).

4.1 Base Techniques↩︎

Chain-of-thought prompting [9] is a prompting technique that guides the language models to solve a problem in a step-by-step manner. By presenting exemplars that solve the corresponding problems with intermediate reasoning steps in the prompts, CoT significantly improves the reasoning performance over direct answer prediction without such intermediate reasoning steps.

Zero-shot chain-of-thought prompting [10] is a variation of CoT where the prompt does not contain any exemplar. Instead, the model is prompted directly with the problem of interest followed by the instruction “Let’s think step by step:”.

Least-to-most prompting [11] teaches language models to (1) break down a problem into subproblems, and (2) solve those subproblems sequentially using CoT. The final answer is that to the last subproblem.

Program prompts [2] represent the arithmetic reasoning process as a program. Following prior work on solving GSM8K problems with code [2], [26], [27], we include a Python program as the problem solution in the prompt, and execute the generated Python code using an external Python interpreter to obtain the final answer.

Self-consistency [12], [28] may further boost the reasoning performance by marginalizing over intermediate reasoning steps that share the same final result. In practice, SC can be implemented by (1) sampling several solutions from the large language model and (2) taking the majority vote. Note that SC is orthogonal to above techniques, and can be combined with any of them.

4.2 Prompt Design↩︎

We present some example prompts used in our experiments (2). For few-shot prompting techniques (i.e., CoT, LtM and Program), the input prompt includes exemplar problems and their solutions before the problem of interest. In order to keep simplicity and avoid over-fitting in prompt engineering, we follow [11] on exemplar creation; that is, we only use one simple exemplar for our main experiments. This exemplar is either based on the [Original Problem] or the [Problem with Irrelevant Context], which allows us to investigate the effect of irrelevant information in the prompt exemplar. For 0-CoT, we adhere to [10] and directly present the problem of interest followed by “A: Let’s think step by step:”.

4.3 Instructed Prompting↩︎

In addition to presenting irrelevant information in the exemplars, we also investigate whether natural language instructions help language models ignore irrelevant context and become less distracted. Extending the line of work [14]–[16] that includes a general task description before exemplars, we add the sentence “Solve grade school math problems. Feel free to ignore irrelevant information given in the questions.” before our exemplars in the prompt (2), which explicitly instructs the language model to ignore irrelevant information in the problem description.

5 Experiments↩︎

Being mindful of the experiment costs, we uniformly sample 4,000 examples from the GSM-IC dataset (denoted by GSM-IC-4K)⁵ for evaluation and analysis purposes throughout this paper. Unless otherwise specified, we mainly use code-davinci-002 in our experiments, and we also evaluate text-davinci-003 which is a model trained with RLHF to better follow instructions [15]. For experiments without self-consistency decoding, we use greedy decoding (i.e., temperature \(\tau=0\)); for self-consistency experiments that require multiple samples for a problem, we sample 20 responses with temperature \(\tau=0.7\) following [12].

5.1 Main Results on GSM-IC↩︎

We compare the performance of different prompting techniques on GSM-IC-4K (3), in terms of both micro and macro accuracies, as well as their corresponding normalized accuracies. Overall, we observe significant performance drop for both models with all prompting techniques. The drop on macro accuracy is especially large, showing that fewer than 30% of the base problems are consistently solved after adding distractors. Comparing the results of two models, text-davinci-003 achieves better normalized micro accuracy than code-davinci-002, though its macro accuracy is mostly worse. In 3, we present a GSM-IC-4K example where a single irrelevant sentence causes different types of errors in investigated prompting techniques. One common error type is wrongly using the number in the irrelevant sentence, as shown in the LtM prediction and other examples in 8. Even if the model does not directly use the irrelevant number for numerical calculation, the presence of the irrelevant sentence in the reasoning steps alone can still cause a wrong prediction, as shown in the CoT prediction.

LtM is generally the most robust technique to irrelevant context. In terms of micro accuracy, LtM outperforms all other prompting methods across models. Using code-davinci-002, LtM achieves about double macro accuracy of CoT. Interestingly, with text-davinci-003, despite that LtM outperforms CoT on the micro accuracy, its macro accuracy is lower. Specifically, text-davinci-003 is highly susceptible to irrelevant context with role overlap; e.g., such irrelevant sentences decrease the macro accuracy to 0 on problems with more than 2 reasoning steps. See Table 4 for the breakdown performance on different types of irrelevant context.

Selecting exemplars with distractors mitigates the distractibility. For few-shot prompts, we find that using exemplars with distractors (i.e., including problems with irrelevant context) consistently outperforms using the original exemplars without distractors across prompting techniques. While prior work has shown that training and fine-tuning with different types of problems improves model robustness [41], our results show that prompting with exemplars that demonstrate how to ignore irrelevant context also results in significant robustness improvement. In 5, we further show that using exemplars with distractors does not cause a performance drop on the original GSM8K dataset, indicating that such a prompt design can be beneficial in achieving better accuracy and robustness simultaneously.

Self-consistency significantly reduces the distractibility. Taking the majority vote from 20 samples,⁶ SC improves the overall micro accuracy by more than 11 percentage points. This means that in addition to improving model performance on clean arithmetic reasoning tasks [12], SC also substantially reduces the distractibility of large language models to irrelevant context. The gain on micro accuracy is notably large on 0-CoT(35.5 percentage points). Furthermore, the correct answer for 99.7% of the problems is in the 20 sampled answers for both CoT and LtM. Even for 0-CoT, the recall of correct solutions within 20 samples is 96.5%. Despite these improvements, the best macro accuracy among all prompting techniques is only \(45\%\), suggesting that for more than half of the base problems, SC fails to prevent the model from being distracted by different variants of irrelevant information. These results imply that a better algorithm may be developed to further reduce the distractibility based on a few sampled solutions.

5.2 Break-Down Analysis↩︎

5.2.1 Factors of the Irrelevant Context↩︎

We analyze the performance of CoT, LtM and Program with respect to the considered factors (3.1) of the irrelevant sentences (4). For both models, we find that (1) in-topic sentences with (2) role name overlap and (3) in-range numbers are generally more challenging, which is examplified by 3. For LtM, the latter two factors do not have a large effect on the micro accuracy. The difference is more significant for the macro accuracy and, as an anomaly, using distractors with in-range numbers turns out to be less challenging than out-of-range numbers when using irrelevant context in the exemplar. Again, with code-davinci-002, LtM outperforms CoT and Program on all investigated sub-categories. On the other hand, using text-davinci-003, LtM outperforms CoT in terms of the micro accuracy, but the macro accuracy is much lower on all sub-categories.

5.2.2 Break-Down Accuracies w.r.t. # Steps↩︎

We analyze the break-down accuracies for problems with respect to the reasoning steps (4). While we see a significant drop for CoT and Program on problems that require four or more steps in the reasoning process, the performance of LtM is fairly consistent across difficulty. In addition to the advantage of LtM on clean problems for complicated reasoning [11], our results show that LtM is also less sensitive to irrelevant context for complicated problems that require more steps to solve.

5.3 Instructed Prompting Improves Robustness to Irrelevant Context↩︎

We have shown that using exemplars with distractors improves robustness to irrelevant context. We also compare the performance of instructed prompting and that of the prompts without instructions in 3. Adding instructions to CoT, LtM, and Program consistently improves their performance. Surprisingly, instructed prompting with original exemplars reaches comparable or even better performance than uninstructed prompting that uses exemplars with distractors for both CoT and LtM. Note that adding the instruction “Solve grade school math problems.” alone does not significantly improve the performance, and it is the instruction “Feel free to ignore irrelevant information given in the questions.” that makes the difference. Similar to the instruction “Let’s think step by step.” employed by 0-CoT, this shows that language models are—to some extent—able to follow natural language instructions in a way that dramatically changes their problem solving behavior, suggesting that such instructions may be useful for guiding the behavior of language models on more tasks.

On the original GSM8K development set [6], [11], we do not observe a drop in accuracy when using exemplars with irrelevant information, adding natural language instructions, or both (5). The same holds for SVAMP [7], an arithmetic reasoning benchmark constructed by applying different types of variations to math problems from existing clean datasets, e.g., changing sentence structures, asking different questions with the same information, etc. This is impressive because the results on GSM-IC show that prompt exemplars with irrelevant information and instructed prompting both improve robustness. For the Program prompt, we find that using exemplars with distractors even increases performance on SVAMP.

5.4 Complicated Prompts May Hurt the Robustness to Irrelevant Context↩︎

We compare our 1-exemplar CoT prompt (2) to a 4-exemplar prompt [11], which is reported as the best-performing CoT prompt on GSM8K, on GSM-IC(6). Note that the 1-exemplar CoT prompt only includes a problem with a 2-step solution, while the 4-exemplar prompt includes problems that require more reasoning steps. While the 4-exemplar prompt leads to better performance on the original GSM8K development set, the 4-exemplar prompt is surprisingly more susceptible to the distraction provided by the irrelevant context. In particular, the 4-exemplar prompt is consistently worse than the 1-exemplar prompt on problems with more than 2 intermediate steps. Even for 2-step prompts, the accuracy improvement from adding more exemplars is almost negligible when using instructions (79.0 vs 79.2). Overall, this finding indicates that adding more exemplars can make the prompt less robust as it leads to some overfitting.

5.5 Extension to DROP↩︎

In addition to GSM-IC, we extend our evaluation to the DROP dataset [56], where the task is to answer a question according to a long passage that naturally contains irrelevant context. We show an example about football games in 8.

We use the CoT and LtM prompts in [11] as the baselines, and we evaluate the prompt variants with the instruction “Solve following questions. Feel free to ignore irrelevant information given in the questions.” added before the exemplars. Note that by adding a problem reduction step in the exemplar solution, the least-to-most prompt implicitly leads the model to come up with relevant subproblems to solve the given problem. Again, we observe that the instruction consistently improves the performance of both CoT and LtM prompting (7).

6 Conclusion and Discussion↩︎

In this work, we introduce GSM-IC, a dataset that supports comprehensive study of the distractibility of large language models when performing arithmetic reasoning in presence of irrelevant contexts. We examine a variety of prompting techniques on GSM-IC, and demonstrate that they are all sensitive to the irrelevant information in the problems. Among the studied techniques, self-consistency [12] leads to a substantial improvement in robustness to irrelevant context across the board, and presenting example problems with irrelevant context in the prompt also consistently improves the performance. Similarly, we find that simply adding an instruction to ignore irrelevant information brings notable performance gains on our benchmark.

Despite the improvement achieved by these methods, the fundamental issue remains: a single piece of irrelevant information can distract the models and substantially degrade their performance, even on problems whose clean versions they correctly solve. We encourage researchers to also prioritize improving on this fundamental limitation when developing new training and prompting techniques. We leave further investigation on the distractibility for other tasks and different language models for future work.

Acknowledgement↩︎

We would like to thank Dale Schuurmans, Olivier Bousquet and Jack Nikodem for helpful discussion and feedback.

7 GSM-IC Details↩︎

Each of the 100 base problem require two to seven steps to solve (5).

Starting from the base problems, we follow the protocols below to create GSM-IC(3.1).

1. Irrelevant sentence template.

   1. For in-topic sentences, we manually write templates within the topic that is close to the original problem description. We are particularly careful about the shareable stuff, for example, money is sometimes considered shareable between family members. In such cases, we make sure that the added do not change the amount of shareable stuff to ensure that the final standard answer is not affected.

   2. For off-topic sentences, we use general templates (9) for all problems unless some of them can be considered as in-topic sentences for some problems—for example, the sentence “The height of {role} is {number} feet.” is considered as an in-topic sentence for problems about heights of people.

      Table 9: Off-topic sentence templates for .

      The shoe size of [ROLE] is [NUMBER].

      is [NUMBER] years old.

      The height of [ROLE] is [NUMBER] feet.

      bought [NUMBER] tomatoes from the grocery store.

      has read [NUMBER] books in the past year.

   3. We make sure that all sentences derived by each template are grammatical English sentences.

   4. We write four in-topic and choose four off-topic distractor sentence templates for each problem.

2. Blank fillers: role names.

   1. We randomly choose a role name X, and use X’s father, X’s mother, X’s brother, X’s sister and X’s neighbor as the overlapped role names.

   2. We choose from the name set {Ada, David, Emma, Jack, John, Mary, Max, Tom} for non-overlapped role names.

   3. We write five names that have overlap with the original character, and five names that do not have overlap for each problem.

3. Blank fillers: numbers.

   1. For in-range numbers, we randomly sample positive integers in the range of \([\frac{\ell}{10}, 10r]\), where \(\ell\) and \(r\) denote the smallest and the largest number that appear in the problem description and standard solution, respectively.

   2. For out-of-range numbers, we choose from the range of \([2, +\infty) \backslash [\frac{\ell}{10}, 10r]\). For very few problems that \(\ell\) is relatively large (i.e., \(\ell > 10^5\)) where we choose out-of-range numbers from the range of \([2, \frac{\ell}{10}]\); for other problems we choose out-of-range numbers \(n = a\times 10^b\) from the range \([10r, \infty)\), where \(a\) and \(b\) are both non-negative integers.

   3. We write four in-range numbers and four out-of-range numbers for each problem.

4. Finally, if adding the irrelevant sentence causes ambiguity (e.g., 10), we fix the question to ensure that the standard solution to the generated problem remain the same as the base problem.

   Table 10: An example that adding irrelevant contexts causes ambiguity: after adding the sentence, it is unclear whether she refers to Kim or Kim’s mother. To ensure that the standard answer is unchanged, we modify the final question to make it clear and faithful to the original problem.

   Original Problem	Kim plants 80 cherry pits. 25% of them sprout and Kim sells 6 of the saplings. How many cherry saplings does she have left?
   Added Sentence	Kim’s mother plants 20 more potatoes.
   Ambiguous Problem	Kim plants 80 cherry pits. 25% of them sprout and Kim sells 6 of the saplings. Kim’s mother plants 20 more potatoes. How many cherry saplings does she have left?
   Fixed Problem	Kim plants 80 cherry pits. 25% of them sprout and Kim sells 6 of the saplings. Kim’s mother plants 20 more potatoes. How many cherry saplings does Kim have left?

8 Sample Predictions on GSM-IC↩︎

In addition to the example outputs shown in 3, we include more example problems and the predictions by different techniques (11 12).

9 Full prompts in experiments↩︎

We list the prompts for all experiments in 13 14.

References↩︎