1 Introduction↩︎

llms are ml models trained on massive amounts of text to complete sentences. Aggressive scaling of these models has led to a rapid increase in their capabilities,[1], [2] with the leading models now being able to pass in some evaluations the United States Medical Licensing Examination.[3] They also have been shown to design and autonomously perform chemical reactions when augmented with external tools such as web search and synthesis planners.[4]–[6] While some see “sparks of agi” in them,[7] others consider them as “stochastic parrots”—i.e., systems that only regurgitate what they have been trained on.[8] Nevertheless, the promise of these models is that they have shown the ability to solve a wide variety of tasks they have not been explicitly trained on.[9], [10] This has led to tremendous economic interest and investment in such generative models, with an expected market of more than $1.3 trillion (almost $30 billion for drug discovery applications) by 2032.[11]

Chemists and materials scientists have quickly caught on to the mounting attention given to llms, with some voices even suggesting that “the future of chemistry is language”.[12] This statement is motivated by a growing number of reports that use llms to predict properties of molecules or materials,[2], [13]–[17] optimize reactions[18], [19] generate materials,[20]–[22] extract information,[23]–[28] or to even prototype systems that can autonomously perform reactions in the physical world based on commands provided in natural language.[5], [6], [29]

In addition, since a lot—if not most—of the information about chemistry is currently stored and communicated in text, there is a strong reason to believe that there is still a lot of untapped potential in llms for chemistry and materials science.[30] For instance, most insights in chemical research do not directly originate from data stored in databases but rather from the scientists and their ability to interpret data. Many of these insights are in the form of text in scientific publications. Thus, operating on such texts might be our best way of unlocking these insights and learning from them. This might ultimately lead to general copilot systems for chemists that can provide answers to questions or even suggest new experiments based on vastly more information than a human could ever read. Such a usage mode is, in particular, interesting in the face of recent advances in autonomous laboratories.[5], [6], [29], [31]–[35]

However, the rapid increase in capabilities of chemical ml models led (even before the recent interest in llms) to concerns about the potential for dual use of these technologies, e.g., for the design of chemical weapons.[36]–[41] To some extent, this is not surprising as any technology that, for instance, is used to design non-toxic molecules can also be used inversely to predict toxic ones (even though the synthesis would still require access to controlled physical resources and facilities). Still, it is essential to realize that such models’ user base is wider than that of chemistry and materials science experts who critically reflect on every output such models produce. For example, many students frequently consult these tools—perhaps even to prepare chemical experiments.[42] This also applies to users from the general public, who might consider using llms to answer questions about the safety of chemicals. Thus, for some users, misleading information—especially about safety-related aspects—might lead to harmful outcomes. However, even for experts, chemical understanding and reasoning capabilities are essential as they will determine the capabilities and limitations of their models in their work, e.g., in copilot systems for chemists. Unfortunately, apart from anecdotal reports, there is little evidence on how llms perform compared to expert (human) chemists.

Thus, to better understand what llms can do for the chemical sciences and where they might be improved with further developments, evaluation frameworks are needed to allow us to measure progress and mitigate potential harms systematically. For the development of llms, evaluation is currently primarily performed via standardized benchmark suites such as BigBench[43] or the LM Eval Harness.[44] Among 204 tasks (such as linguistic puzzles), the former contains only two tasks classified as “chemistry related”, whereas the latter contains no specific chemistry tasks. Due to the lack of widely accepted standard benchmarks, the developers of chemical language models[14], [45]–[48] frequently utilize language-interfaced[49] tabular datasets such as the ones reported in MoleculeNet,[50] Therapeutic Data Commons[51] or MatBench.[52] In these cases, the models are evaluated on predicting very specific properties of molecules (e.g., solubility, toxicity, melting temperature or reactivity) or on predicting the outcome of specific chemical reactions. This, however, only gives a very limited view of the general chemical capabilities of the models.

While some benchmarks based on university entrance exams[53], [54] or automatic text mining[55]–[57] have been proposed, none of them have been widely accepted. This is likely because they cannot automatically be used with black box (or tool-augmented) systems, do not cover a wide range of topics, or are not carefully validated by experts. On top of that, the existing benchmarks are not designed to be used with models that support special treatment of molecules or equations and do not provide insights on how the models compare relative to experts.

In this work, we report a novel benchmarking framework (), which we call ChemBench, and use it to reveal limitations of current frontier models for use in the chemical sciences. Our benchmark consists of question-answer pairs manually () or semi-automatically () compiled from diverse sources. Our corpus covers a large fraction of the topics taught in undergraduate and graduate chemistry curricula. It can be used to evaluate any system that can return text (i.e., including tool-augmented systems).

To contextualize the scores, we also surveyed experts in chemistry on a subset of the benchmark corpus to be able to compare the performance of current frontier models with (human) chemists of different specializations. Our results indicate that current frontier models perform “superhuman” on some aspects of chemistry but, in many cases, including safety-related ones, might be very misleading. We find that there are still numerous limitations for current models to overcome, such that they can be directly applied in autonomous systems for chemists. Moreover, we conclude that carefully created broad benchmarks represent an important stepping stone for progress in this field.

2 Results and Discussion↩︎

2.1 Benchmark corpus↩︎

To compile our benchmark corpus, we utilized a broad list of sources (see ), ranging from university exams to semi-automatically generated questions based on curated subsets of data in chemical databases. For quality assurance, all questions have been reviewed by at least one scientist in addition to the original curator and automated checks.

Importantly, our large pool of questions encompasses a wide range of topics. This can be seen, for example, in in which we compare the number of questions in different subfields of the chemical sciences (see for details on how we assigned topics). The distribution of topics is also evident from in which we visualize the questions in a two-dimensional space using a pca on the embeddings of the questions. In this representation, semantically similar questions are close to each other, and we color the points based on classification into topics. It is clear that a focus of ChemBench(by design) lies on safety-related aspects, which in appear as a large distinct clusters across the embedding space.

While many existing benchmarks are designed around mcq, this does not reflect the reality of chemistry education and research. For this reason, ChemBench samples both mcq and open-ended questions ( mcq questions and open-ended questions).

2.1.0.1 “Tiny” subset

It is important to note that a smaller subset of the corpus might be more practical for routine evaluations.[59] For instance, [60] report costs of more than $10,000 for api calls for a single evaluation on the widely used helm benchmark. To address this, we also provide a subset ( questions) of the corpus that was curated to be a diverse and representative subset of the full corpus in which topics are more balanced than in the complete corpus (see for details on the curation process). We also used this subset to seed the web application for the human baseline study.

2.2 Model evaluation↩︎

2.2.0.1 Benchmark suite design

Because the text used in scientific settings differs from typical natural language, many models have been developed that deal with such text in a particular way. For instance, the Galactica model[61] uses special tokenization or encoding procedures for molecules and equations. Current benchmarking suites, however, do not account for such special treatment of scientific information. To address this, ChemBench encodes the semantic meaning of various parts of the question or answer. For instance, molecules represented in smiles are enclosed in [START_SMILES][\END_SMILES] tags. This allows the model to treat the smiles string differently from other text. ChemBench can seamlessly handle such special treatment in an easily extensible way because the questions are stored in an annotated format.

Since many widely utilized systems only provide access to text completions (and not the raw model outputs), ChemBench is designed to operate on text completions. This is also important given the growing number of tool-augmented systems that are deemed essential for building chemical copilot systems. Such systems can augment the capabilities of llms through the use of external tools such as search apis or code executors.[62]–[64] In those cases, the llm that returns the probabilities for various tokens (that are often used for model evaluations[65]) is only a part of the whole system, and it is not clear how to interpret the probabilities in the context of the whole system. The text completions, however, are the system’s final outputs, which would also be used in a real-world application. Hence, we use them for our evaluations.

2.2.0.2 System performance

To understand the current capabilities of llms in the chemical sciences, we evaluated a wide range of leading models[66] on the ChemBench corpus, including systems augmented with external tools. An overview of the results of this evaluation is shown in . In this figure, we show the percentage of questions that the models answered correctly. Moreover, we show the worst, best, and average performance of the human experts in our study, which we obtained via a custom web application (chembench.org) that we used to survey the experts. Remarkably, the figure shows that the leading llm, Claude 3, outperforms the best human in our study in this overall metric and vastly, by more than a factor of two, exceeds the average performance of the experts in our study. Many other models also outperform the average human performance. Interestingly, the Galactica model, which was trained specifically for scientific applications, underperformed compared to many advanced commercial and some open-source models and barely exceeds the random baseline.

Given the considerable interest in tool-augmented systems, the mediocre performance of these systems (GPT-3.5 and Claude 2 augmented with tools) in our benchmark is striking. Their lack of performance, however, is partially because we limited the system to a maximum of ten llm calls. With the default tool augmented setup (using the so-called ReAct method[64]), however, this often did not allow the system to identify the correct answer (e.g., because it repeatedly tried to search for the answer on the web and did not find a solution within ten calls to the llm). This observation highlights the importance of communicating and measuring not only predictive performance but also computational cost (e.g., in terms of api calls) for tool-augmented systems.

2.2.0.3 Performance per topic

To obtain a more detailed understanding of the performance of the models, we also analyzed the performance of the models in different subfields of the chemical sciences. For this analysis, we defined a set of topics (see ). We classified all questions in the ChemBench corpus into these topics based on hand-crafted rules and classifier models operating on the question texts. We then computed the percentage of questions the models or humans answered correctly for each topic.

In this spider chart, the worst score for every dimension is zero (no question answered correctly), and the best score is one (all questions answered correctly). Thus, a larger colored area indicates a better performance. One can observe that this performance varies widely across models and topics. While macromolecular chemistry and biochemistry receive relatively high scores for many models, this is not the case for topics such as chemical safety or analytical chemistry. In the subfield of analytical chemistry the prediction of the number of signals observable in a nmr spectrum proved difficult for the models (e.g., percent correct answers for GPT-4) while this question appeared easier ( percent correct) for trained humans. Importantly, the human experts are given a drawing of the compounds, whereas models are only shown the smiles string of a compound and have to use this to reason about the symmetry of the compound (i.e., to identify the number of diasterotopically distinct protons, which requires reasoning about the topology and structure of a molecule). These findings also shine an interesting light on the value of textbook-inspired questions. A subset of the questions in the ChemBench are based on textbooks targeted at undergraduate students. On those questions, the models tend to perform better than on some of our semi-automatically constructed tasks (see ). For instance, while the overall performance in the chemical safety topic is low, the models would pass the certification exam according to the German Chemical Prohibition Ordinance based on a subset of questions we sampled from the corresponding question bank (e.g., % correct answers for GPT-4, % for Claude 3, and % for the human experts). While those findings are impacted by the subset of questions we sampled, the results still highlight that good performance on such question bank or textbook questions does not necessarily translate to good performance on other questions that require more reasoning.

We also gain insight into the models’ struggles with chemical reasoning tasks when we examine their performance as a function of molecular descriptors. If the model would answer questions after reasoning about the structures, one would expect the performance to depend on the complexity of the molecules. However, we find that the models’ performance does not correlate with complexity indicators but rather trivially with the size of the compounds (see ). This indicates that the models may not be able to reason about the structures of the molecules (in the way one might expect) but instead rely on retrieval of fragments of the training data. Those observations mirror recent findings that llms struggle to “reverse” facts they have seen in training (i.e., generalize from “A has feature B” to “B is a feature of A”).[67]–[70]

It is important to note that the model performance for some topics, however, is underestimated in the current evaluation. This is because models provided via apis typically have safety mechanisms that prevent them from providing answers that the provider deems unsafe. For instance, models might refuse to provide answers about cyanides. To overcome this, direct access to the model weights would be required, and we strive to collaborate with the developers of frontier models to overcome this limitation in the future. This is facilitated with the tooling ChemBench provides, thanks to which contributors can automatically add new models in an open science fashion.

2.2.0.4 Confidence estimates

One might wonder whether the models can estimate if they can answer a question correctly. If they could do so, incorrect answers would be less problematic as one could detect when an answer is incorrect. To investigate this, we prompted[71] some of the top-performing models to estimate, on an ordinal scale, their confidence in their ability to answer the question correctly.

In , we show that for some models, there is no significant correlation between the estimated difficulty and whether the models answered the question correctly or not. For applications in which humans might rely on the models to provide answers, this is a concerning observation highlighting the need for critical reasoning in the interpretation of the model’s outputs.[30], [72] For example, for the questions about the safety profile of compounds, GPT-4 reported an average confidence of (on a scale of 1–5) for the questions it answered correctly and for the questions it answered incorrectly. While, on average, the verbalized confidence estimates from Claude 3 seem better calibrated (), they are still misleading in some cases. For example, for the questions about ghs pictograms Claude 3 returns an average score of for correct answers and for incorrect answers.

3 Conclusions↩︎

On the one hand, our findings underline the impressive capabilities of llms in the chemical sciences: Leading models outperform domain experts in specific chemistry questions on many topics. On the other hand, there are still striking limitations. For very relevant topics the answers models provide are wrong. On top of that, many models are not able to reliably estimate their own limitations. Yet, the success of the models in our evaluations perhaps also reveals more about the limitations of the exams we use to evaluate models—and chemists—than about the models themselves. For instance, while models perform well on many textbook questions, they struggle with questions that require some more reasoning. Given that the models outperformed the average human in our study, we need to rethink how we teach and examine chemistry. Critical reasoning is increasingly essential, and rote solving of problems or memorization of facts is a domain in which llms will continue to outperform humans.

Our findings also highlight the nuanced trade-off between breadth and depth of evaluation frameworks. The analysis of model performance on different topics shows that models’ performance varies widely across the subfields they are tested on. However, even within a topic, the performance of models can vary widely depending on the type of question and the reasoning required to answer it.

The current evaluation frameworks for chemical llms are primarily designed to measure the performance of the models on specific property prediction tasks. They cannot be used to evaluate reasoning or systems built for scientific applications. Thus, we had little understanding of the capabilities of llms in the chemical sciences. Our work shows that carefully curated benchmarks can provide a more nuanced understanding of the capabilities of llms in the chemical sciences. Importantly, our findings also illustrate that more focus is required in developing better human-model interaction frameworks, given that models cannot estimate their limitations.

While our findings indicate many areas for further improvement of llm-based systems, it is also important to realize that clearly defined metrics have been the key to the progress of many fields of ml, such as computer vision. Although current systems are far from reasoning like a chemist, our ChemBench framework will be a stepping stone for developing systems that might come closer to this goal.

4 Methods↩︎

4.1 Curation workflow↩︎

For our dataset, we curated questions from existing exams or exercise sheets but also programmatically created new questions. Questions were added via Pull Requests on our GitHub repository and only merged into the corpus after passing manual review () as well as automated checks (e.g., for compliance with a standardized schema).

To ensure that the questions do not enter a training dataset, we use the same canary string as the BigBench project. This requires that llm developers filter their training dataset for this canary string.[4], [43]

4.1.0.1 Manually curated questions

Manually curated questions were sourced from various sources, including university exams, exercises, and question banks. provides an overview of the sources of the manually curated questions.

4.1.0.2 Semi-programmatically generated questions

In addition to the manually curated questions, we also generated questions programmatically. An overview of the sources of the semi-programmatically generated questions is provided in .

4.2 Model evaluation workflow↩︎

4.2.0.1 Prompting

We employ distinct prompt templates tailored for completion and instruction-tuned models to maintain consistency with the training. As explained below, we impose constraints on the models within these templates to receive responses in a specific format so that robust, fair, and consistent parsing can be performed. Certain models are trained with special annotations and LaTeX syntax for scientific notations, chemical reactions, or symbols embedded within the text. For example, all the SMILES representations are encapsulated within [START_SMILES][\END_SMILES] in Galactica[61]. Our prompting strategy consistently adheres to these details in a model-specific manner by post-processing LaTeX syntax, chemical symbols, chemical equations, and physical units (by either adding or removing wrappers). This step can be easily customized in our codebase.

4.2.0.2 Parsing

Our parsing workflow is multistep and primarily based on regular expressions. In the case of instruction-tuned models, we first identify the [ANSWER]``[\ANSWER] environment we prompt the model to report the answer in. In the case of completion models, this step is skipped. From there, we attempt to extract the relevant enumeration letters (for multiple-choice questions) or numbers. In the case of numbers, our regular expression was engineered to deal with various forms of scientific notation. As initial tests indicated that models sometimes return integers in the form of words, e.g., “one” instead of “1”, we also implemented a word-to-number conversion using regular expressions. If these hard-coded parsing steps fail, we use a llm, e.g., Claude 2, to parse the completion.

Manual validation of parsing performance

As stated, we tried to account for the variation in outputs with custom regular expressions. We selected a large, diverse subset of questions (10 per topic for all model reports) and manually investigated where the parsed output does not match the actual answer intended by the model. We found that for questions, the parsing was accurate in 99.76% of the cases, while for floating point questions, the parsing was accurate in 99.17% of the cases. The models most frequently generating errors are pplx-7b-chat and Mixtral-8x7b.

4.2.0.3 Models

4.2.0.3.1 Completion models

We used Galactica (120b)[61] with the default settings.

4.2.0.3.2 Instruction-tuned models

In addition, we used Claude 2, Claude3 (Opus),[78] GPT-4,[4] GPT-3.5-turbo,[1] Gemini Pro,[79] Mixtral-8x7b[80] LLaMA2 (7B, 13B, 70B),[81] as well as the 7B chat model from Perplexity.AI.

4.2.0.3.3 Tool augmented models

In addition to directly prompting llms, we also investigated the performance of tool-augmented systems. For this, we investigated the 7B parameter “online” model of Perplexity.AI. In addition, we utilized gpt-3.5-turbo as well as Claude 2, with ReAct-style tool augmentation.[64] The latter two models had access to WolframAlpha, the ArXiv api, a Python interpreter, and web search (using DuckDuckGo). We implemented the systems using Langchain with the default prompts and constrained the system to a maximum of ten llm calls.

4.3 Confidence estimate↩︎

To estimate the models’ confidence, we prompted them with the question (and answer options for mcq) and the task to rate their confidence to produce the correct answer on a scale from 1 to 5. We decided to use verbalized confidence estimates[71] since we found those closer to current practical use cases than other prompting strategies, which might be more suitable when implemented in systems.

4.4 Human baseline↩︎

4.4.0.1 Question selection

Since we anticipated that we would not be able to collect enough responses for every question to allow for a meaningful statistical analysis, we decided to show a relevant subset of all questions to the human scorers. For selecting the subset, we decided to address two questions:

• Are the questions for which the models scored poorly just too difficult or unanswerable?

• Are there areas in which the performance of humans is very different from the ones of the models?

To answer the first question, we selected up to 13 questions per source that llmsdid not answer correctly in an initial scoring round. In addition, we picked 100 diverse ones using greedy MaxMin sampling on the embeddings on the questions computed using BART. We added five random questions about the number of nmr signals and the point group of molecules.

4.4.0.2 Study design

For our initial study, we wanted to maximize the response rate given our available resources. For this reason, we did not opt for a highly controlled study setting. That is, while users were prompted not to use external tools other than a calculator and not consult with other humans, we do not have any way to verify that the participants complied with those rules. Note that users were also allowed to skip questions.

Another aspect of requiring unsupervised question answering is that in real life, humans have tools and can use them to answer any question of interest. In our current study, we prompted users not to use those tools.

4.4.0.3 Participants

Users were open to reporting about their experience in chemistry. Overall, did so. Out of those, reported to have been awarded a Ph.D. are beyond a first postdoc, have a master’s degree, and have a bachelor’s degree.

4.4.0.4 Comparison with models

To compare the performance of humans (who might have answered only some questions) with the performance of models (which answered all questions), we focussed on questions that at least four humans answered and limited the pool of human scorers to those who answered at least 100 questions (i.e., humans). The latter threshold was chosen to limit it to humans who seriously attempted to answer a part of the questions systematically. This analysis might lead to potential biases, most likely in favor of humans, as they were allowed to skip questions. humans answered more than 200 questions. For the analysis, we treated each human as a model. We computed the topic aggregated averages per human for analyses grouped by topic and then averaged over all humans.

4.5 Classification of questions into topics↩︎

When curating our dataset, we systematically recorded keywords and sources. To allow for analysis of the model performance as a function of the topic, we leverage this information together with the output of sequence classification models. We use this information to make the assignment for questions that can easily be assigned to a topic based on the source (e.g., number of nmr signals, chemical compatibility, toxicology exam questions). For the remaining ones, e.g., from chemistry olympiad questions, we use zero-short sequence classification[82] using the BART model[58], [83], which our preliminary analysis found to be more robust than topic modeling based on embeddings from OpenAI’s ada model or Cohere’s Cohembed-english-v3.0 model.

Data and code availability↩︎

The code and data for ChemBench is available at https://github.com/lamalab-org/chem-bench. The code for the app for our human baseline study is available at https://github.com/lamalab-org/chem-bench-app. To ensure reproducibility, this manuscript was generated using the framework.[84] The code to rebuild the paper (including code for all figures and numbers next to which there is a GitHub icon) can be found at \GitHubURL. To facilitate reproduction, some intermediate analysis results are cached at http://dx.doi.org/10.5072/zenodo.34706.

Acknowledgements↩︎

This work was supported by the Carl Zeiss Foundation, a “Talent Fund” of the “Life” profile line of the Friedrich Schiller University Jena and the FAIRmat consortium. We also thank Stability.AI for the access to its HPC cluster and donations.

M.A.expresses gratitude to the European Research Council (ERC) for evaluating the project with the reference number 101106377 titled “CLARIFIER” and accepting it for funding under the HORIZON TMA MSCA Postdoctoral Fellowships - European Fellowships. Furthermore, M.A.acknowledges the funding provided by UK Research and Innovation (UKRI) under the UK government’s Horizon Europe funding guarantee (Grant Reference: EP/Y023447/1; Organization Reference: 101106377).

M.R.and U.S.S.thanks the “Deutsche Forschungsgemeinschaft” for funding under the regime of the priority programme SPP 2363 “Utilization and Development of Machine Learning for Molecular Applications – Molecular Machine Learning” (SCHU 1229/63-1; project number 497115849).

In addition, we thank the OpenBioML.org community and their ChemNLP project team for valuable discussions. Moreover, we thank Pepe Márquez for discussions and support and Julian Kimmig for feedback on the web app. In addition, we acknowledge support from Sandeep Kumar with an initial prototype of the web app. We thank Bastian Rieck for developing the LaTeX-credit package (https://github.com/Pseudomanifold/latex-credits).

Conflicts of interest↩︎

K.M.J.is a paid consultant for OpenAI (as part of the red teaming network). M.P.is an employee of Stability.AI, and A.M.and N.A.are paid contractors of Stability.AI.

Author contributions↩︎