1 Introduction↩︎

The emergence of large language models (LLMs) has profoundly impacted numerous scientific disciplines, including mathematics. Within the mathematical community, there is a growing discourse regarding the capabilities of LLMs, with some researchers claiming that these models achieve problem-solving efficiency comparable to early-year graduate students. Beyond pure performance, there is also a popular perception that these LLMs serve as powerful tools for individual mathematical development and success.
A quick and critical examination of existing benchmarks and platforms specializing in LLM evaluations reveals an interesting characteristic. Many existing benchmarks often rely on problems from a relatively narrow range of competitions. While these problems are undeniably challenging, they represent only a subset of advanced mathematics, leaving other problem types underrepresented. Incorporating a broader spectrum would support a more comprehensive evaluation of mathematical reasoning. Our work addresses this gap by introducing a novel benchmark consisting of thirty-nine problems primarily from mathematics Olympiads of Eastern Europe and countries from the former Soviet Union and six problems from various mathematical texts specifically by Vladimir Arnold. The problems cover topics such as combinatorics, algebra, and number theory, but do not include geometry. The problems selected for this benchmark share a crucial commonality, they consistently demand an extra, nontrivial step in mathematical reasoning, often requiring novel thinking strategies beyond mere computation to pattern recognition. This main characteristic is exactly what makes them particularly suitable for unraveling and probing at the limitations of current LLMs in mathematical comprehension. The construction of this benchmark involved an extensive and meticulous search for specific problem sources. Our searching process led us to identify Eastern European and countries of the former Soviet Union mathematic regional and national Olympiads as particularly valuable. The problems, often less widely circulated than their Western counterparts, proved exceptionally effective for benchmarking due to two primary reasons, their relative obscurity minimizes the risk of LLM pre-training exposure, and their inherent difficulty requires truly novel problem solving approaches. While previous benchmarks often relied on heavily explored competition problems from the IMO and related competitions, our approach intentionally sought sources known for their equally challenging national and regional competitions from other traditions. The integrity of our evaluation is further enforced by the unambiguous nature of the problem solutions. Which is to be further discussed in the following sections.

2 Related Work↩︎

A leading resource for benchmarks based on mathematics competitions and Olympiads is MathArena, which provides a comprehensive overview of existing datasets and model performance. Our benchmark was curated with MathArena as a primary reference point, as the platform systematically presents both competition sources and large language model results. However, after analyzing the scope and problem sets emphasized in MathArena, we deliberately shifted our focus toward lesser-known mathematics Olympiads and competitions, aiming to broaden the diversity of our benchmark and reduce overlap with existing evaluations. Drawing on prior experience in mathematics Olympiads at both the national and international levels, we were able to identify clear directions for sourcing problems for our benchmark.
MathArena has already conducted evaluations of LLMs on the 2025 IMO, widely regarded as the standard for the most challenging mathematics competition. In their evaluation, GPT-5 achieved a 38.10% accuracy rate. Using our benchmark, however, GPT-5 Thinking obtained a slightly lower score of 35.89%, suggesting that our benchmark yields comparable outcomes while capturing a different problem selection than MathArena. In contrast, Gemini 2.5 Pro reached 31.55% accuracy on MathArena, but achieved 0% accuracy on our benchmark, underscoring the comparable difficulty and different emphasis of our dataset. It is worth noting that, according to MathArena’s documentation, GPT-5 was released after the 2025 IMO, raising the possibility of data contamination. By comparison, nearly all of the problems in our benchmark are at least five years old, meaning they were potentially available in model pretraining corpora. Nevertheless, GPT-5-Thinking solved only about one-third of them. This outcome suggests that the limited performance cannot be attributed to recency or contamination but rather reflects the intrinsic reasoning difficulty of the specific Olympiads we chose and the problems sourced from them. Moreover, among the various competitions reported on MathArena, which include the Harvard-MIT Math Tournament (HHMT), Stanford Math Tournament (SMT), and the Brown University Math Olympiad (BRUMO), the IMO produced some of the lowest accuracy rates across models, yet our benchmark reduced performance even further. This suggests a potential limitation of relying solely on specific competitions as the basis for benchmarking, since it may not fully capture the range of reasoning challenges posed by other mathematical Olympiads and competitions.

It is important to note, as stated on MathArena, that HHMT, BRUMO, and IMO all occurred prior to the release of GPT-5, raising the possibility of data contamination in the reported results for this model. In the case of Gemini 2.5 Pro, similar concerns arise for HHMT, since the model was released after the competition date. By contrast, our benchmark was constructed from Olympiads and competitions dating back at least five years. This makes contamination more likely, since such problems would almost certainly have been present in the model’s training data. However, if that were the case, one would expect the accuracy on our benchmark to be higher rather than lower. This contrast suggests that the range of competitions commonly used in benchmarks, though valuable, reflects only part of the mathematical landscape. These contests are unquestionably difficult, but they represent only one tradition of problem solving, leaving other competitions of comparable rigor less visible. Benchmarks such as those in MathArena provide a strong standard, yet one might anticipate that our benchmark, drawing on older, non-Western problems, would overlap more with pre-training corpora and thus yield higher accuracy. This potentially suggests that the difficulty of non-Western competitions is under recognized and the reliance on a limited benchmark set may give an incomplete picture of model capabilities. Broadening evaluation to include a wider range of mathematical reasoning challenges could therefore provide a more balanced assessment and guide further progress in LLM development.

3 Problem Curation↩︎

3.1 Olympiads↩︎

The problems used to curate this benchmark were found in the following way. A quick search of previous benchmarks showed that the United States of America Mathematical Olympiad (USAMO) and the IMO were heavily used to evaluate these LLMs. These sources were very popular which meant that their problems were already heavily utilized. This made even going back to older years of the competitions fruitless. Because of this, we decided to turn our attention to lesser-known, yet highly competitive Olympiads from countries from the former Soviet Union and Eastern European countries. These mathematical Olympiads are known in the mathematics community to be extremely competitive, with problems ranging from regional to state to national levels. This opened a new pathway to potentially more difficult problems. Yet, their main attraction was not even their difficulty, it was their unpopularity. They were not as widely circulated in the public eye, which made them highly valuable in our search.
Knowing now where to look, the issue began in the hunt for finding the specific Olympiads. Their underexposure also meant the resources would be scarce. Although most competitions were not easily tracked down, deep searching led us to many valuable sources providing problems from competitions dating back five to forty years. Typical Olympiad competitions feature a wide variety of problem types and topics, including both proof-based and numerical problems in areas such as number theory, geometry, and algebra. We focused exclusively on problems with numerical solutions, avoiding proof-based problems. Many proof-based questions involve well-known claims or standard arguments that are easily recognized by LLMs, increasing the likelihood of correct responses through pattern recognition rather than reasoning. In addition, proof-based problems require additional steps for verification and often depend on human judgment. To avoid the additional time and subjectivity involved, we focused exclusively on problems that could be evaluated unambiguously. Specifically, we prioritized problems with clear numerical answers that could be verified automatically and in seconds. These main objectives made finding the exact sources that we needed an additional challenge. Most Olympiad competitions contain more proof-based problems than computational ones, so we were typically able to extract at maximum two suitable problems per competition. However, in many cases, we were lucky to find even one viable problem. However, despite these challenges, we were able to identify a substantial number of high-quality problems, and our hypothesis to shift our focus to countries from the former Soviet Union and Eastern European mathematical Olympiads and competitions deemed to be correct.

3.2 V. I. Arnold Work↩︎

Alongside the curated problems from various mathematical Olympiads and competitions, we also included six problems drawn from works by Vladimir I. Arnold, including "Ordinary Differential Equations", "Mathematical Methods of Classical Mechanics", and "The Mathematical Trivium". The two books included solutions for selected problems, and our search followed the same approach for the mathematical competitions, we focused only on problems with numerical answers. Additionally, The Mathematical Trivium, an exam consisting of 100 problems to be solved within three hours, was another source. The Trivium was designed for advanced undergraduate and graduate students, with problems that draw on higher-level topics while retaining a largely computational format. Although not purely exercises in computation, these problems emphasize technical problem-solving in advanced areas of mathematics. While the Trivium has never had any official solutions published or verified in any text, it has long served as a recreational challenge among mathematicians. Among the first few problems in the Trivium, which is included in our benchmark, one initially appears to be a straightforward limit evaluation but turns out to be quite difficult without a clever approach. For our purposes, we solved and verified only the initial handful of problems, as the full Trivium set was not our primary focus, and remains to potentially be a topic for later work. The benchmark emphasizes Olympiad-style reasoning, and Trivium was selected primarily to test computational ability in a complementary way. We chose to include problems from Arnold’s works as part of our curation strategy, given his reputation as one of the greatest polymath mathematicians. His texts pose many challenging problems, and given their popularity, we expected LLMs to perform well on them. However, we still found several problems that the models failed to solve correctly.

Each problem selected from the Olympiads and Arnold’s texts was first solved independently, after which the proposed solutions were consulted to verify correctness. This ensured that our solutions were both self-derived and consistent with the established sources. This process not only verified the correctness of our own solutions but also served as a check on the validity of the proposed solutions themselves. After establishing the correctness of each solution, we submitted the problems to the LLMs and evaluated their responses relative to the verified solutions. Our narrowed focus on numerical problems made the initial evaluation straightforward. Often a simple ‘yes’ or ‘no’ that could be quickly confirmed using the provided solution. Additional layers of verification were also applied, which will be explained in more detail in the following sections.

3.3 Problem Types↩︎

This benchmark consists of two main sources of problems, as mentioned before, the various different mathematic competitions, exams, and Olympiads, and the various works from Vladimir I. Arnold. The majority of the problems taken from the various competitions as seen in figure a) were from combinatorics and number theory, where as in figure b) from the texts the majority was taken from Mathematical Methods for Classical Mechanics.

Problem Category and Source Breakdown↩︎

3.3.1 Combinatorics↩︎

Combinatorics has a central role in mathematical Olympiads due to its blend of creativity, logic, and minimal prerequisite knowledge. Although parts of the broader field of combinatorics intersect with algorithmic techniques, Olympiad-style combinatorics is intentionally non-algorithmic. These problems almost never collapse to standard procedures and are often designed to frustrate standard approaches. Many lack equations to manipulate, figures to analyze, or even numbers to compute. Consequently, they require flexible reasoning that entails not only clever constructions but also the invention of new structures, rather than reliance on the recognition of familiar patterns. LLMs are generally trained to identify and complete patterns based on seen data, but in Olympiad combinatorics, the structure must often be created on the spot, not retrieved from prior examples. These problems typically involve multistep chains of reasoning, where each step depends delicately on the previous one. This makes them highly sensitive to even small logical errors. However, LLMs frequently generate hallucinated intermediate steps and introduce logical inconsistencies, thereby producing solutions that are incorrect or incomplete. It is precisely these characteristics that make combinatorics account for such a significant portion of the benchmark.

3.3.2 Number Theory↩︎

Olympiad-style number theory problems are a combination of pure mathematics with formal logic, often making them some of the most deceptively difficult challenges for both humans and LLMs. Although number theory appears algorithmic, since it draws on familiar topics like modular arithmetic and Diophantine equations, the Olympiad variant is rarely computational. Instead, it emphasizes clever manipulation and substitutions with unexpected usage of elementary tools in non-standard ways, similar to that of combinatorics problems. Many number theory solutions involve a vital moment in which the initially obscure problem becomes instantly simplified through a key observation or trick. For LLMs, these problems are particularly challenging because they often cannot be solved by surface-level pattern recognition. The mistakes of LLMs in number theory problems are very similar to those of combinatorics problems as well. Typically, errors arise from hallucinated steps or the failure to recognize key simplifications. Owing to its similarity in character to combinatorics problems, number theory exhibits a disproportionately high error rate compared to other topics in our benchmark.

4 Evaluation↩︎

Evaluating the effectiveness of the LLMs in solving the curated problems was a multistep process. The first stage involved independently analyzing and solving each problem and constructing an outlined solution, which provided a skeletal outline of the proposed answer. Next, we verified the accuracy of the solutions provided by the original sources. As noted earlier, most sources included proposed solutions which allowed us to confirm the validity of both the problems and their corresponding answers. Our outlines were compared against the proposed solutions and the sources were thus verified. Once this verification step was completed, we submitted each problem to the LLMs specifically, Gemini 2.5 Pro and Chat GPT 5 Thinking using a brand-new chatbot session for every single problem to prevent any cross contamination of context. After several minutes, the model would output a numerical answer, which we then compared against the verified solution. In order to fully mark a problem as incorrectly solved, we ran each problem in a new chat session twice. This was done so because in some cases the LLM produced an incorrect answer on the first attempt, however, occasionally on the second attempt it produced the correct result. In particular, we never encountered a case in which the model produced a correct solution on the first attempt and an incorrect one on the second. Below are the results from the two leading commercial LLMs we conducted tests on.

The percentage rates for correctly solved problems from the evaluation are shown above. The emergence of GPT-5 led to a significant performance increase over Gemini 2.5 Pro, which had an overall zero percent pass rate. Overall, with a pass rate of approximately 36%, it is clear that the new advancements still require further development.
In our evaluation process, it is also worth noting that we paid very close attention to the reasoning and solution steps for the test problems. While our primary criterion was the correctness of the numerical answer, the reasoning of the LLM shed light on the potential reliance on pattern recognition. Our close attention to the LLM’s thought process was additionally in an attempt to see if the model picked up on the fact that the problems and some proposed solutions were open sourced. When the model encountered a prompt that resembled for example, a known IMO–shortlist problem, it seemed to anchor heavily on that reference. The LLM seemed to import the familiar answer structure it had been previously trained on and proceeded as if the current task were the same. In multiple instances, we believed it probable that the superficially similar source problem differs in a small but crucial way, and that deviation is enough to make the transplanted solution schema invalid. The model most likley recognizes the problem type but, biased by its training on the familiar variant, cannot adapt its predetermined reasoning to the altered constraints and therefore produces an incorrect result. In effect, it retrieves a cached solution rather than constructing a fresh argument from first principles. This is precisely why, once the technical details diverge, the LLM struggles to reconcile the mismatch. This behavior suggests that heavy exposure to IMO-style material produces a brittle competence, strong on memorized templates yet highly vulnerable to small structural changes. Below highlights an important example of this.

It is also important to note that the reasoning process employed by LLMs significantly influences their performance on our benchmark. Specifically, models such as Gemini 2.5 Pro seem to adopt one of two main strategies when addressing a given question: drawing upon their internal training data (i.e., learned knowledge) or conducting a live web search. When the model encounters a question resembling material it has already learned, it is more likely to bypass live search and rely solely on its training. While this approach is efficient in many cases, it becomes less effective when the model’s training coverage is limited. By focusing training predominantly on a limited set of challenging mathematics competitions, we risk overlooking other competitions of comparable difficulty that are equally worthy of inclusion. As noted above, minor changes in problem structure led the LLM to assume it already knew the appropriate steps to apply. However, when the model encountered the singularity, it became evident that it did not possess a clear strategy for proceeding. This leads us to another critical standpoint, is the model unable to solve the problem generally due to a lack of exposure, or is the model not noticing that the question and some proposed solution exist on the open web. Both of these pose an issue, however if the problem and a proposed solution are available with a search, the greater issue becomes why did the model not notice it when if it ran into issues using pretrained knowledge.

5 Conclusion↩︎

The distinctive aspect of this benchmark is that every problem was sourced directly from the open web. Our selection process was guided by a specific hypothesis: many existing benchmarks appear to follow a narrow pattern, drawing predominantly from the IMO and other well-known Western competitions. While these benchmarks are valuable, their concentrated focus may help explain some of the performance patterns observed in LLMs. Motivated by this observation, we sought to design a benchmark that could probe model capabilities from a different angle. Several factors contributed to the behavior we observed: the models’ limited prior exposure to the Olympiads we selected, their relative underrepresentation in public datasets, the inherent reasoning tendencies of LLMs, and the high difficulty level of the chosen problems. Although the problems in this benchmark may appear simple because of their deterministic, numerical form, they nevertheless posed significant challenges for the models and revealed important gaps in current capabilities.

6 Appendix↩︎

6.0.1 Detailed solution of Combinatorics problems↩︎

More than half of the selected Olympiad problems were combinatorical problems therefore, we give an example from our benchmark with a full solution below. This problem is one that Gemini 2.5 Pro failed to solve correctly however GPT-5 Thinking did solve correctly.

6.0.2 Detailed solution of Number Theory problems↩︎

The second main group of problems from our benchmark are number theory problems. Similarly this problem was incorrectly solved by Gemini 2.5 Pro and correctly solved by GPT-5 Thinking.

6.0.3 Detailed solution of Graph Theory problems↩︎

This problem is one that Gemini 2.5 Pro failed to solve correctly however GPT-5 Thinking did solve correctly.

6.0.4 Detailed solution of Algebra problems↩︎

This problem is one that Gemini 2.5 Pro failed to solve correctly however GPT-5 Thinking did solve correctly.