1 Introduction↩︎

Although large language models have been demonstrated to be powerful in various applications [3]–[7], solving math problems that require complex reasoning skills remains a challenging task. On MATH [8], a competition-level math problem benchmark containing algebra, calculus, geometry, combinatorics and number theory problems, open-source base LLMs such as the LLaMA family [9], [10] fail to answer most of the problems correctly.

Previous work tries to enhance the mathematical reasoning abilities of base models by fine-tuning them on domain-specific data. Specifically, One line of work  [11], [12] collects math corpora from the web and fine-tunes the models on them, which is also known as the procedure of continual pre-training [13]. Another line of work focuses on constructing synthetic data through rejection sampling  [14], distilling from GPT-4/GPT-3.5 [15] or question bootstrapping [16], and then use the generated question-response pairs to perform supervised fine-tuning in the way described in [5], [17]. However, there still exists a large performance gap between these fine-tuned models and the most advanced close-source models such as GPT-4 [18] and Gemini-Ultra [19]. Given that simply adding more data does not always lead to better performance as shown in  [16], how to bridge the gap remains an open challenge.

This work tackles the challenge by combining the two lines of work. On one hand, we reuse the high-quality corpora used in the pre-training stage during fine-tuning. Specifically, MMIQC contains around 1200k question-response pairs we filtered and pre-processed from the web pages at math.stackexchange.com, which are included in the RedPajama dataset [20]. On the other hand, for the synthetic data part of MMIQC, we increase the diversity by using multiple kinds of augmentation methods listed below: 1) Prompting GPT-4 with an integrated version of the question bootstrapping prompts used in [16], and do rejection sampling with GPT-3.5-Turbo on both seed and augmented problems. 2) Using a modified prompt presented in [21] to ask GPT-4 to generate similar problems with answers given seed problems of the training set of MATH. Although the generated answers can be wrong, we perform rejection sampling on these problems as well. 3) Performing IQC (Iterative Question Composing) with 4 iterations in total. We iteratively ask GPT-4 to compose new questions from the given seed problems and do rejection sampling to filter those problems with answers aligned with GPT-3.5-turbo’s answers. 4) Filtering a 204k subset of MetaMathQA [16] and adding it to the MMIQC dataset (More details on MMIQC will be introduced in Section 4).

We fine-tune several base models on MMIQC, resulting in models consistently achieving a large margin compared to their counterparts when evaluated on MATH, as shown in Figure 2. Specifically, the models Mistral-7B-MMIQC, Llemma-34B-MMIQC, DeepSeek-67B-MMIQC and Qwen-72B-MMIQC, which are obtained by fine-tuning Mistral-7B [22], Llemma-34B [11] and DeepSeek-67B [23] on MMIQC, achieve 36.0%, 38.6%, 41.0% and 45.0% accuracy on MATH, 5.8%, 3.8%, 4.2% and 3.3% higher than the counterpart models that are fine-tuned on MetaMathQA, respectively.

We also evaluate the models on the 2023 Hungarian national high school finals in mathematics [24]. The results in Figure 1 suggest that the mathematical reasoning abilities the models acquire through being fine-tuned on MMIQC can generalize to unseen held-out problems.

We highlight our contributions as follows:

• We propose IQC (Iterative Question Composing), a data augmentation method that can iteratively generate diverse data starting from a seed dataset of math word problems.

• We release MMIQC, a mixture of processed web data and synthetic question-response pairs. In different model sizes, the models fine-tuned on MMIQC consistently outperform their counterparts by a clear margin on the MATH test set. Notably, Qwen-72B-MMIQC achieves a 45.0% accuracy, exceeding the previous open-source state-of-the-art² by 8.2% and outperforming the initial version GPT-4 released in 2023. Such improvement can generalize to unseen held-out data, e.g., Hungarian high school finals.

• Our results show that reusing the high-quality data in the pre-training corpora during the fine-tuning stage can improve the model performance, successfully combining the two lines of work of continual pre-training and supervised fine-tuning.

• Our results also show that using multiple augmentation methods to construct datasets for fine-tuning is an efficient way to boost the performance of LLMs.

2 Related Work↩︎

Base Large Language Models. Base large language models (LLMs) trained on massive corpora (e.g. 1.4T tokens of text for Llama [9]) from various sources with a simple auto-regressive next token prediction loss have achieved great success in various natural language processing tasks [4], [9], [10], [22], [26]. Although these pre-trained models are not intended to serve for solving complex mathematical problems, [27] show that few-shot prompting can help the models answer a certain fraction of problems correctly. Nevertheless, to achieve better performance, fine-tuning the base LLMs on domain-specific data is required.

Fine-tuning Base LLMs on Mathematical Datasets. Current practice of fine-tuning base LLMs on mathematical datasets can be classified into two kinds: 1) continual pretraining [11], [12]. This line of work typically collects billion-tokens level mathematical text data from the web, such as mathematical sub-sites of Stack Exchange and ArXiv, and fine-tune the model in the same way as that in the pre-training stage. 2) SFT (Supervised Fine-Tuning) [14]–[16], [28]. Works in this line collect question-response pairs via various methods and train the models on their dataset in an Alpaca style. Due to the scarcity of publicly available high-quality question-response pairs datasets and the costly nature of manually composing math word problems, how to augment new data from the existing datasets becomes the focus of these works.

Our work is located in the middle between these two: MMIQC is a mixture of filtered pre-training corpus and question-response pairs generated using various augmentation methods.

Reasoning Frameworks for Solving Mathematical Problems. Much effort has been devoted to achieving a higher accuracy on math word problem benchmarks by designing different procedures of using the given LLMs to obtain the answers, which we refer to as reasoning frameworks. Among them, Prompting-based methods [26], [27], [29] play a significant role in activating the potential reasoning abilities for base LLMs through carefully designing the prompts shown to the models. Self-consistency [30] samples multiple rationale paths for a model and then decides the answer by majority voting. In contrast of self-consistency, [31]–[33] use Outcome Reward Models (ORM) and Process Reward Models (PRM) trained on human annotations as verifiers to help select the answer with the highest score from the sampled reasoning paths of LLMs. Getting rid of the need of manual annotation, [25] score a given reasoning step by estimating the potential of that step to lead to a correct answer automatically.

Some frameworks also include the use of plug-in tools and external APIs. Program-aided prompting [15], [34] provides in-context samples containing Python codes for LLMs and uses code interpreters to execute the output to facilitate reasoning. Further, [28] interleave natural language rationales with Sympy³ code and fine-tune the model on trajectories sampled from GPT-4 to follow their framework in two steps, namely imitation learning and output space shaping.

We note that our results in Figure 2 do not include multiple times of sampling, use of verifiers or code interpreters, thus cannot be directly compared with the results reported in these works.

3 Iterative Question Composing↩︎

Traditional data augmentation methods primarily concentrate on modifying either the questions or answers while retaining their original meanings, or generating similar problems, as discussed in [16] and [21]. These methods, however, are limited in their diversity as they aim to create nearly identical problems. Our approach, termed IQC (Iterative Question Composing), deviates from this by iteratively constructing more complex problems. It augments the initial problems, adding additional reasoning steps without altering their intrinsic logical structure. This ensures that the newly formed problems are organically linked to the original problem and elaborately tries to not include extraneous elements induced by a large transition of the reasoning process.

Notations. In our description, we refer to the combination of an LLM, its tokenizer, encoding/decoding methods, and a fixed generation configuration (inclusive of generation strategy, sampling temperature, and stopping criteria) simply as ‘an LLM’. For an LLM \(\pi\), we denote the output distribution given prompt \(p \in \mathcal{A}^*\) as \(\pi(\cdot | p)\). The concatenation of two text paragraphs \(p_1\) and \(p_2\) is represented as \(p_1\oplus p_2\).

The IQC process begins with specifying an LLM \(\pi_q\) for question composing and another model \(\pi_r\) for rejection sampling. An answer extractor is needed to derive answers from responses. Two responses \(r_1\) and \(r_2\) are considered equivalent, denoted \(r_1\simeq r_2\), if the same answer can be extracted from both. The process initiates with a seed dataset \(S_0 = \{(q_i, a_i)\}_{i=1}^n\).

In iteration #1, we prompt \(\pi_q\) with \(p_1\oplus x(q, a)\) for each \((q, a)\in S_0\), where \(x(\cdot, \cdot)\) is a text template transforming a question-response pair into text, and \(p_1\) solicits a new question-answer composition. This yields a new dataset \[S_1 = \{(q'_i, a'_i)\}_{i=1}^n,\] where \((q'_i, a'_i) = x^{-1}(x'_i)\) and \(x'_i \sim \pi_q\left(\cdot | p_1\oplus x_i\right)\) is the output for the \(i\)th sample. We further enhance \(S_1\) by rejection sampling from \(\pi_r\), resulting in \[R_1 := \{(q'_i, a^{(j)}_i)| a^{(j)}_i \simeq a'_i, i\in[n], j \in [m]\},\] where \(a^{(j)}_i\) are the sampled responses from \(\pi_r(\cdot|p_r\oplus q'_i)\). The dataset \(D_1\) is then formed by uniting \(S_1\) and \(R_1\): \[D_1 := S_1 \cup R_1.\]

For each subsequent iteration #\(k\), the aforementioned procedure is repeated using \(S_{k-1}\) as the seed dataset, with varying question composing prompts \(p_k\). The complete IQC process is delineated in Algorithm 3.

4 The MMIQC Dataset↩︎

In this section, we introduce how each part of MMIQC is constructed in detail.

Subset of MetaMathQA. The original MetaMathQA dataset is constructed by sampling GPT-3.5 for \(k=20\) times under a \(T=0.7\) temperature for each problem in the training set of MATH [1] and GSM8K [31] dataset, or its bootstrapped versions. We restrict the number of samples for each completely same question to be 3 and 1 for MATH and GSM8K, respectively, to obtain a subset of MetaMathQA. This subset contains 112.2K GSM8K question-response pairs and 91.5K MATH pairs.

Answer Augmentation and Question Bootstrapping. We integrate the question bootstrapping methods used in [16] into a single prompt shown in Figure 6. Our motivation is that given GPT-4 is highly capable of natural language understanding, a few-shot prompting style used in [16] might suppress the diversity of the augmented questions. The seed dataset is constructed by the samples in the training set of MATH that do not contain Asymptote language in their question statements. We perform rejection sampling from GPT-3.5 on both the seed dataset and generated questions using the prompt shown in Figure 7, obtaining 66.5K question-response pairs. We use a temperature \(T=1.0\) for both question bootstrapping and rejection sampling.

Augmented Similar Problems. With the same seed dataset, we ask GPT-4 to generate 3 problems (with a solution, for rejection sampling) for 1 seed problem each time, using the prompt in Figure 8. This is different from the practice in [21], where they ask GPT-3.5 to generate 10 similar questions given 1 seed problem since we find that GPT tends to generate several almost the same problems regardless of the given seed problem when asked to generate up to 10 new problems. We use the stronger GPT-4 instead of GPT-3.5 considering rejection sampling needs the answer to the problem better to be correct. To control the cost, our prompt emphasizes that the solution should be as brief as possible. The total number of the augmented similar problems and the question-response pairs rejection sampled from them is 38.2K. The rejection sampling prompt is the same one in Figure 7 as well. We use a temperature \(T=1.0\) for both procedures.

Iterative Question Composing. We perform Iterative Question Composing for 4 iterations as described in Section 3. Specifically, we use GPT-4 for question composing model \(\pi_q\) with a \(T=0.7\) temperature and GPT-3.5 for rejection sampling model \(\pi_r\) with a \(T=1.0\) temperature. The question composing prompts and rejection sampling prompt are shown in Figure 5 and Figure 7, respectively. The text templater \(x(\cdot, \cdot)\) we use is a program that transforms each question-response pair into JSON text format, with fields ‘problem’ and ‘solution’. The seed dataset is also the samples in the training set of MATH that do not contain Asymptote code in their question statements. The resulting dataset has 55.1K samples in total.⁴ We provide an example of the generated questions in different iterations corresponding to the same seed problem in Figure 4. We note that although some of the questions are not rigorously a sub-problem or sub-step of the corresponding problem in the previous iteration as required in our prompt, they are still valid questions that can increase the diversity of the dataset. We have checked the correctness of 100 randomly selected QA pairs generated by IQC and find that 85% of them are correct.

Mathematics Stack Exchange. We observe that in the OpenWebMath [35] dataset, the data from Mathematics Stack Exchange shows high quality and is most related to competition-level math. Motivated by this, we extract the data collected from Mathematics Stack Exchange in RedPajama [20] and pre-process it into question-response pairs. For each Mathematics Stack Exchange page, we only retain the answer ranked first by RedPajama. Then we filter out the answer that does not contain a formula environment symbol ‘$’. This results in a dataset with 1203.6K question-response pairs.

\begin{table}[t]

-0.1in \end{table}

Table ¿tbl:tab:mmiqc? shows the make-up of MMIQC. When fine-tuning the models MMIQC contains 3 repetitions of the subsets mentioned above, except for the Mathematics Stack Exchange part. We shuffle the order of samples after combining the subsets.

5 Experiments↩︎

5.1 Fine-tuning Setup↩︎

Our fine-tuning strategy mainly follows the practice of [17], except that we use a different prompt template to transform the question-response pairs. For a sample from Mathematics Stack Exchange, the corresponding prompt fed into the model during training is a simple concatenation of the question and response with two new-line symbols. For a sample from other subsets, we additionally add a prefix ‘Please solve the following problem and put your answer at the end with “The answer is:”.’ to the question-response concatenation.

We use the HuggingFace transformers library [36] for our fine-tuning experiments.

\begin{table}[t]

-0.3in \end{table}

We fine-tune all models on MMIQC for 1 epoch, using a \(3\%\) warm-up ratio linear learning rate schedule. For the choice of maximum learning rate, we do a simple hyper-parameter selection experiment shown in Table ¿tbl:tab:lr95select? and determine it to be 1e-5. We use the BFloat16 numerical format during training. Employing the DeepSpeed Zero-3 Stage [37], we fine-tune 7B models on one node of 8xA800 GPUs with micro batch-size at 8, and gradient accumulation at 4, 34B models on 2 nodes with micro batch-size at 4 and gradient accumulation at 4 and \(\sim\)​70B models on 4 nodes with micro batch-size at 4 and gradient accumulation at 2, maintaining an effective batch size of 256. It takes around 14 hours, 61 hours and 90 hours to fine-tune 7B, 34B and \(\sim\)​70B models under the setups stated above, respectively.

\begin{table}[t]

-0.1in \end{table}

5.2 Model Evaluation↩︎

For a fair comparison, we first evaluate the fine-tuned models on MATH [1], a competition-level math word problems benchmark with 5000 test problems in a zero-shot setting. We prompt all our fine-tuned models with the test question with the prefix ‘Please solve the following problem and put your answer at the end with “The answer is:”.’, and extract the answer from the output using a modified version of the answer extractor provided in [12]. We use a series of rules to infer whether the extracted answer is the same as the ground-truth answer, including a comparison using SymPy [40]. The complete results of our evaluation on MATH and a comparison with existing models are shown in Table ¿tbl:tab:math-acc?.

For the evaluation on 2023 Hungarian national high school finals in mathematics, we use the few-shot prompt used in [24]. We manually assess the grades for every model according to the examiner instructions. The results shown in Figure 1 are the grades under a full mark of 117.

5.3 Ablation Study on Subsets of MMIQC↩︎

In order to understand the ratio of contribution to the improvement revealed in Table ¿tbl:tab:math-acc? of different subsets of MMIQC, we fine-tune Mistral-7B with a series of training sets constructed by gradually adding the subsets. When MathStackExchange is not added, we fine-tune for 3 epochs. When MathStackExchange is added to the training dataset, we mix 3 repetitions of other data with 1 repetition of the MathStackExchange, and fine-tune for only 1 epoch. It can be seen from Table ¿tbl:tab:ablation? that

• Although our filtered subset of MetaMathQA is only half the size of the original dataset (which has 395K samples, more than the total number of samples of our synthetic data), the performance drop is only 1.8%. This shows that the \(k=20\) strategy in [16] results in some redundancy.

• Our Answer Augmentation & Question Boosting data help the fine-tuned model beat Mistral-7B-MetaMathQA, verifying our hypothesis that directly asking GPT to perform question bootstrapping is more efficient than providing few-shot examples to them.

• Our IQC method leads to a significant 3.1% improvement from a high accuracy of \(31.5\%\) with only 55.1K samples, showing its efficiency. Moreover, the later iterations of IQC also account for a certain ratio of improvement, proving that IQC is a method that can continuously generate new data that can help increase the diversity when added to the data generated in previous iterations.

5.4 Contamination Test↩︎

We check the \(n\)-gram matches for MMIQC to ensure that the improvement is not a result of direct memorization. We use the script provided by [11] to check the \(n\)-gram matches between the synthetic part of the MMIQC and MATH test set. It turns out that for a 30-gram match check, there are 44 hits of match between the ‘solution’ field of MATH test set and the ‘output’ field of MMIQC, far fewer than the 168 hits between that of MATH test set and MATH training set. Moreover, we manually check these 44 hits and find that 43 among them belong to the case where intermediate steps of the solutions to similar but different questions collide, with the only exception being the question ‘A regular polygon has interior angles of 144 degrees. How many sides does the polygon have?’. This almost rules out the possibility that fine-tuned models get memorization of solutions to the problems in the test set, indicating a very low risk of data contamination for MMIQC.

6 Conclusion↩︎

In this work, we introduce a novel data augmentation method for math word problem datasets called IQC (Iterative Question Composing) and use it in the construction of our MMIQC dataset. Our evaluation results show that the models fine-tuned on MMIQC achieve new SOTAs on the MATH benchmark. The improvements of our models benefit from the diverse data sources of MMIQC and the effectiveness of IQC.

For future directions, we are interested in how to equip open-source models with the ability to compose questions, in order to perform IQC in a self-evolution style, similar to that in [41]. Besides, how to integrate the verification systems [21], [25] that are originally used to improve the accuracy during inference time into the procedure of IQC, is also an attractive topic.

Acknowledgements↩︎

We thank Yang Yuan, Kaiyue Wen, Xingyu Dang, and Jingqin Yang for their helpful discussions.

References↩︎