1 Introduction↩︎

Large language models (LLMs) show excellent performances in various tasks in general domains including Question Answering (QA) [1], [2], Machine Translation (MT) [3], Summarization [4], Machine Reading Comprehension (MRC) [5], Sentiment Analysis [6], and so on. Some researchers design proper prompts for solving biomedical tasks [7]–[9]. However, most of the existing LLMs have been pre-trained with texts in general domains, lacking domain-specific knowledge. To overcome this challenge, biomedical LLMs are proposed through pre-training on biomedical corpora [10], [11], fine-tuning with instruction data [12], or reinforcement learning with human feedback [13].

With the chain-of-thought prompting technique, [8] have achieved 60.2% accuracy on USMLE-QA [14], passing the medical licensing examination in the United States. In the most recent work, with the help of multiple agents, [9] have achieved 93.06% accuracy on the USMLE-QA dataset, similar to the performance of a human expert. With this series of techniques, biomedical LLMs are greatly promoted in English biomedical tasks. However, biomedical LLMs in other languages still have much room for improvement (e.g., Japanese, Chinese, French, etc.). Besides the relative unpopularity of existing Japanese LLMs, another important obstacle is the lack of a comprehensive benchmark for evaluation and comparison. Therefore, in this paper, we focus on constructing a benchmark for evaluating Japanese biomedical LLMs.

We selected five tasks that are widely used for evaluating LLMs and real-world applications, including multi-choice question-answering (MCQA), named entity recognition (NER), machine translation (MT), document classification (DC), and semantic text similarity (STS). Since there are only a few Japanese biomedical datasets exist and they are generally small (e.g., IgakuQA [15] only has 1,600 samples for testing), to reduce the fluctuation of evaluation results, we translate large-scale and high-quality datasets from other languages (e.g., English) to Japanese, augmenting the scale of our benchmark. Furthermore, in the field of Japanese biomedical LLM, a solid leaderboard is missing. Therefore, we select eight representative models to conduct extensive experiments, providing a standard for comparison. We hope our work can make future comparisons more convenient and fair, promoting the development in this field.

In summary, our contributions are in three folds.

• We construct a large-scale benchmark including 20 Japanese biomedical datasets across five tasks for a comprehensive evaluation.

• We evaluate eight representative models across four categories in our benchmark to provide a standard for future comparison.

• We conduct extensive analysis from aspects of the dataset, model, and prompt template, providing valuable insights for future researchers.

2 Related Works↩︎

Benchmarking is essential for the development of a specific field. ImageNet Challenge [20] is a famous benchmark in Computer Vision. Many remarkable works on image recognition have been proposed [21]–[23] throughout history and the development has increased rapidly. One reason for this success is the convenience of comparison and evaluation in this field. The GLUE [24] is another famous benchmark for evaluating and analyzing natural language understanding (NLU) systems to promote research in developing general and robust NLU systems. However, these works mainly focus on English tasks, limiting the scope of evaluating other languages like Japanese. [25] constructed the JGLUE from scratch without using any translation, including six datasets, which facilitates the research in Japanese natural language processing (NLP) [26]–[28].

Considering the wide applications of language models (LMs), researchers are trying to explore LMs’ power in biomedical tasks. [16] collected 13 biomedical NLP datasets in six tasks from different isolated work to form a benchmark called BLURB for evaluating biomedical models. MMLU [17] is a benchmark consisting of multiple topics. Specially, it contains some biomedical questions like medical questions at the college level. DrBenchmark [18] is an NLU benchmark for evaluating French biomedical models. However, they are not applicable in Japanese. JMMLU² is a translated version of the MMLU. The researchers recruited human translators to check and remove those that were difficult to translate, irrelevant, or inconsistent with the Japanese culture. Recently, [19] have proposed a multilingual benchmark with six languages for evaluating medical LMs. These benchmarks reflect some shortages of existing LLMs and provide insights into improving the Japanese biomedical LLMs, but they only focus on the MCQA tasks, which hinders the completeness of the evaluation. Considering these shortages, in this paper, we are dedicated in constructing a large-scale benchmark with diverse tasks for evaluating Japanese biomedical large language models. Table 1 shows a comparison of these benchmarks.

3 JMedBench↩︎

Our benchmark construction consists of two parts. The first part is the dataset collection, while another part is the protocol for evaluation. Firstly, we introduce the rationality of dataset selection and how we augment our benchmark with datasets from other languages. Then, we propose a protocol to obtain robust evaluation results and discuss its necessity for evaluating Japanese biomedical LLMs. Figure 1 is an overview of our benchmark.

3.1 Datasets↩︎

In the JMedBench, we include 20 datasets across five tasks containing 38K testing samples. We collect some human-manufactured Japanese datasets, like IgakuQA [15]. We also translate some high-quality large-scale English datasets into Japanese to enhance the robustness of JMedBench. Considering the convenience and performance of using OpenAI’s API, we use ChatGPT³ and GPT-4 [29] to create our evaluation datasets when translation is needed. To ensure the quality of the translated testing sets, we use the most powerful model from OpenAI, the GPT-4⁴, to perform machine translation. In-context learning is a common practice for adapting an LLM to an unseen task. Therefore, we also translate the training or validation sets to support few-shot evaluation. Due to the limitation of our budgets, we use the cheapest API⁵ from OpenAI to translate these samples. Though the translation may not be perfect, producing unfaithful content sometimes, it is good enough to provide information like some domain-specific knowledge and task format during the few-shot evaluation. Previous works [30]–[32] also have similar findings that ChatGPT has already had a comparable MT performance with specialized Neural Machine Translation systems. Here listed are the involved biomedical tasks and corresponding datasets. Detailed statistics can be found in Table 5 in the Appendix.

• MCQA is one of the most commonly used tasks for evaluating LLMs since other tasks can be easily reformulated into the MCQA task. We included IgakuQA [15], JMMLU-medical⁶, and translated MedMCQA [33], MedQA [14], USMLE-QA, PubMedQA [34], and MMLU-medical [35], [36].

• MT is an important natural language generation (NLG) task. In the biomedical domain, researchers usually need to refer to some English terminologies or communicate with other researchers. Therefore, we expect LLMs can handle cross-lingual tasks besides monolingual tasks. We included the EJMMT [37] dataset to evaluate the cross-lingual ability of LLMs.

• NER is an NLU task aiming to extract named entities like biomedical terminologies, medicines, etc. We included three Japanese medical NER datasets from JMED-LLM⁷: MRNER-disease, MRNER-medicine, and NRNER. To improve the diversity of the dataset, we also follow the BLURB benchmark and include translated BC2GM [38], BC5Chem, BC5-Disease [39], JNLPBA [40], and NCBI Disease [41].

• DC aims to classify documents into specific categories. We include three datasets from JMED-LLM: CRADE, RRTNM, and SMDIS.

• STS is a regression task to compute the semantic similarity between two biomedical sentences. We reformulate it as a classification task to output the discrete level of similarity. We include the JCSTS [42].

3.2 Evaluation Dataset Augmentation↩︎

To enlarge the size of JMedBench for obtaining robust evaluation results, we select several biomedical datasets in English, because of its popularity.

3.2.1 Multi-choice Question-Answering↩︎

Different from previous works that usually conduct machine translation at the sentence level, we perform translation at the instance level. Specifically, we translate questions and options meanwhile, so that LLM can understand the scenario better to provide more correct translations. Detailed prompt template can be found in Table 6 in the Appendix.

3.2.2 Named Entity Recognition↩︎

We also translate the NER datasets from the BLURB benchmark to improve the amount and diversity of JMedBench. There are three fields in the NER samples: entity type, text, and entities. To ensure the consistency of the translated entity types, we manually translate them into Japanese based on a dictionary (e.g., gene \(\rightarrow\)

). As for the text and entities, we also perform translation at the instance level, as described in Section 3.2.1. The prompt template for translating the biomedical NER datasets is also shown in Table 7 in the Appendix.

One of the challenges is that the translated entities may not appear in the translated text. To solve this issue, we conduct the translation in two phases: machine translation and manual modification. We first use ChatGPT and GPT-4 to translate the training and testing sets, respectively. We then collect all the invalid samples, mainly due to JSON format error and failure to include the translated entities, and re-translate them using GPT-4. We increase the temperature to 0.5 and call the GPT-4 API again at most 5 times to seek a valid sample. After the machine translation phase, 223 translated entries (0.34%) still remain invalid and then we manually modify these entries to make them valid.

During machine translation, we find that translating entities first instead of text first can reduce about 10% of invalid samples. We speculate that with the entity-first prompt, LLM can refer to the already translated entities when translating the text, thus, the translated entities are usually contained in the following translated sentence. However, since this is not the main focus of this paper, we did not conduct further analysis to verify this hypothesis. We hope this finding can inspire future researchers when performing instance-level machine translation. Despite a small number of failure cases during the machine translation phase (some bad cases can be found in the Appendix 6.3), we realized that the translation quality is very high when we conduct manual modification, which also reflects the reliability of our data augmentation method.

3.3 Evaluation Protocols↩︎

LLMs are usually sensitive to the prompt templates, especially in zero-shot evaluation [43]. To obtain a robust and fair result, we suggest reporting the maximal score of multiple runs using diverse prompt templates for benchmarking. We have also considered computing an average score using different templates, whereas this reported performance may be easily implicated by inappropriate prompts (e.g., using an English-centric prompt for a Japanese-only LLM). In the following evaluation, we use four types of prompt templates:

• Minimal: We include information as little as possible in the prompt. For example, for completing the MCQA task, we only input the question, and compute the likelihood of each possible option, namely, {question}\n.

• Standard: We use commonly used prompt templates in each task. For example, we follow [44] for evaluating MCQA tasks.

• English-centric: Some of the existing Japanese LLMs were continually pre-trained from English-centric LLMs. Therefore, we intend to explore whether an English-centric prompt template is beneficial.

• Instructed: Besides the standard input, we include a brief task instruction, evaluating the instruction-following ability of LLMs.

As for the MCQA and DC tasks, it is difficult to constrain the auto-regressive LLMs to generate one of the given options or classes. Therefore, we follow [45] to compute the likelihood perplexity of each possible answer and select the one that has the highest generation possibility as the final answer. We report accuracy on these two tasks. As for the STS task, we also calculate the likelihood perplexity of generating 0-5 and select the one that has the highest generation possibility as the final output. We use the Pearson Correlation as the evaluation metric. As for the MT and NER tasks, we generate the output and compute the BLEU [46] score and entity-level F1 score, respectively.

4 Experiments↩︎

4.1 Comparison Methods↩︎

In our experiments, we included four categories of popular and excellent LLMs to construct our benchmark, including general LLMs in other languages: Llama2 [47], Llama3 [48], Qwen-2 [49], Mistral [50]; biomedical LLM in other languages: Meditron [10]; Japanese general LLMs: llm-jp [28], SwallowLM [51]; and Japanese biomedical LLM: MMed-Llama3 [19]. The specific checkpoints are listed in Table 9 in the Appendix. Due to the computation resources, we only evaluate LLMs with around \(7\sim8\)B parameters. Llm-jp is a representative LLM that was pre-trained from scratch with Japanese and English texts. Although it does not have the 7B version of the model, we still include the llm-jp with 13B parameters in our benchmark.

4.2 Experimental Results↩︎

4.2.1 Multi-choice Question-Answering↩︎

Table 2 shows the benchmark results on Japanese biomedical MCQA tasks. Surprisingly, Qwen2 outperforms all models in MCQA, followed by MMed-Llama3. Note that Qwen2 was primarily pre-trained with Chinese and English texts. We hypothesize that one reason for its success is the considerable overlap in tokens between Chinese and Japanese. MMed-Llama3 was continually pre-trained on biomedical texts in multiple languages including Japanese, explaining its superior performance over Llama3. These observations highlight the importance of understanding the Japanese language and injecting domain knowledge. With few-shot demonstrations, all models have improved. We attribute this to the task format [52] and some domain-specific knowledge provided by the demonstrations. Comparing Llama2 and Llama3, we find that the performance gap under the zero-shot setting is larger than that under the few-shot setting. The additional improvement should be attributed to the improved in-context learning (ICL) ability of Llama3, highlighting the need to enhance the ICL ability of LLMs.

Although there is a human-translated version of MMLU-medical, namely, the JMMLU-medical dataset, we still translate the original MMLU-medical dataset using GPT-4 to enrich our benchmark. According to the performances of these two datasets (i.e., JMM & MML in Table 2), the differences between performances on these two datasets do not exceed 5% of accuracy. Furthermore, the ranking of the performances on the translated MMLU-medical dataset also reflects the ranking on the human-translated JMMLU-medical dataset. These observations confirm the quality and the applicability of our translated datasets.

Meditron was continually pre-trained with large-scale English biomedical texts from the Llama2 checkpoint. [10] showed that Meditron has been successfully shifted to the biomedical domain, outperforming the vanilla Llama2 in various biomedical MCQA tasks. However, we realize that Meditron performs worse than Llama2 in the JMedBench. Such multilingual ability degradation is probably due to the catastrophic forgetting issue during continual pre-training. How to improve an LLM safely without losing any other ability should be considered in future research. Besides, since the SwallowLM and MMed-Llama3 were continually pre-trained with additional Japanese texts from Llama2 and Llama3, respectively, they are improved by approximately \(1\%\sim5\%\) average accuracy, indicating the importance of local-language adaptation.

4.2.2 Named Entity Recognition↩︎

Table 3 shows the results on Japanese biomedical NER datasets. In the few-shot evaluation of BC2GM, BC5Chem, BC5Disease, JNLPBA, and NCBI-Disease datasets, we use three shots of examples. However, for MRNER-Disease, MRNER-Medicine and NRNER, we only use one shot of example because texts in these datasets are so long that multiple shots will exceed the input token limit of several models.

According to the results, we find that Llama3-8B outperforms other LLMs in both zero-shot and few-shot evaluations, with average F1-entity score of 40.69% and 61.69% respectively. The Japanese biomedical LLM, MMed-Llama3, has the second-best performance in both settings. Few-shot examples can significantly improve the performance of models on the NER tasks, ranging from 19.36% to 33.33% F1-entity improvement. Similar to the observations on MCQA tasks, we believe these examples help LLMs better understand the entity types’ definition and output format. Besides, we find that LLMs perform generally worse on datasets including MRNER-Disease, MRNER-Medicine, and NRNER which are derived from JMED-LLM. Note that the average text lengths of datasets from these two sources are 69.82 and 247.81 Japanese characters, while the numbers of entities are 1.33 and 2.66, respectively. Considering the longer input text, larger number of entities and sparser entity distribution, we believe these are the main reasons why the datasets derived from JMED-LLM are more challenging.

4.2.3 Machine Translation↩︎

Table 4 shows the BLEU scores for involved comparison methods on EJMMT. MMed-Llama3-8B and Llama3-8B achieve the best and second-best performance in our benchmark under the zero-shot setting. Interestingly, we find that the English-centric models (e.g., Llama2, Mistral) tend to perform better on translating Japanese texts into English, while the Japanese-centric models (e.g., SwallowLM) perform much better in translating English texts into Japanese. We believe the main reason is the text generation ability in different languages. Therefore, when applying LLMs to the MT task, we should consider more on the language generation ability instead of the language understanding ability. Although the llm-jp is also a Japanese-centric LLM, according to [28], it was pre-trained with 50-50 Japanese-English mixed data. Therefore, it has a balanced bilingual NLU and NLG ability. Furthermore, with few-shot demonstrations displaying the task format, llm-jp achieves the best performance in the MT task, which shows the prospect of developing Japanese LLMs from scratch instead of continually pre-training from checkpoints in other languages. Besides, comparing Llama2 and the continually pre-trained Meditron and SwallowLM, we find that continually pre-training with texts in biomedical domains or Japanese texts only will lead to forgetting issues. Continual Learning [53] is a potential solution,but it is still challenging to continually improve the existing LLMs while maintaining their original ability.

4.2.4 Document Classification↩︎

Performances of the DC task are also shown in Table 4. We find that Qwen2 achieves the best performance again. In the zero-shot setting, Meditron achieves the second-best performance, while MMed-Llama3 achieves the second-best performance. Most of the comparison methods achieve better performance when few-shot demonstrations are given. We believe it is because of the provided task format as we discuss in Section 4.2.1. Moreover, LLMs can also recognize the fine-grained differences between different classes given few-shot demonstrations, making better decisions in classification. Especially, we notice that Meditron performs badly under the few-shot evaluation. We attribute it to the language degradation issue since it accepts a few long documents in the context, amplifying the noise when understanding Japanese.

4.2.5 Semantic Text Similairity↩︎

The performances on the STS task are varied dramatically. Qwen2 achieves excellent performance on this task, while the prediction of other models like Llama2-based models (i.e., Llama2, Meditron, SwallowLM) is close to random guess. One possible reason is that the distribution of generating numbers is close to a uniform distribution for these models. Recent works also show the shortage of LLMs from this aspect [54], [55]. However, understanding and generating numbers accurately is essential in the biomedical domains (e.g., on blood test reports). Therefore, it is also a promising search direction in the field of biomedical NLP.

4.3 Discussions↩︎

In this section, we will conduct an integral and in-depth analysis on the experimental results.

4.3.1 Comparison of Model Performances↩︎

Figure 2 includes two radar charts that demonstrate models’ zero-shot and few-shot performance on different tasks. Besides, we also rank the model performance and visualize the rankings in Figure 6 as shown in the Appendix. A larger distance from the center represents a higher ranking and better performance. From the radar charts, we can find out that basically, MMed-Llama3, Qwen2, and Llama3 are the most outstanding LLMs on various tasks. Few-shot examples also significantly improve the model performances in all tasks.

4.3.2 Effect of Prompt Templates↩︎

We also hope to understand the performance of prompt templates across different tasks and models. In zero-shot evaluation, Figure 3 illustrates that the performance of Standard, English-centric, and Instructed prompt templates do not differ significantly, but using English-centric templates usually achieves a slightly better performance. This phenomenon is even more evident in English-centric LLMs. We believe it is because these models have a greater advantage in understanding English instructions, even when facing cross-lingual contexts. Moreover, Figure 4 shows that few-shot demonstrations reduce the differences between prompt templates to a certain extent, with a particularly noticeable enhancement for minimal prompt templates. We believe it is because the output relies less on the instructions and can instead understand the task format from the few-shot examples.

5 Conclusions↩︎

In this paper, we discuss an urgent need for the field of Japanese biomedical LLMs that requires a solid benchmark for evaluation and comparison. We collect a large collection of Japanese datasets in diverse biomedical tasks, including MCQA, MT, NER, DC, and STS. Considering the scale of the human-manufactured datasets, we translate several large-scale datasets with high quality in English to ensure robust benchmarking results.

Based on the constructed dataset collection, we conduct an evaluation of four types of models, including Japanese biomedical LLMs, Japanese General LLMs, biomedical LLMs in other languages, and general LLMs in other languages. Reported performances reveal some insights for improving existing Japanese LLMs in the biomedical domain. Furthermore, our datasets and evaluation tools are publicly available for future research.

Limitations↩︎

Considering the difficulty of evaluating natural language generation (NLG) tasks that usually require human evaluation, we only include natural language understanding (NLU) tasks or reformulate NLU tasks into NLU tasks. However, NLG tasks are also widely used in real-world applications. In the future, we consider introducing LLM-based evaluation methods to unlock an easy evaluation of NLG tasks, enriching our benchmark for a further comprehensive evaluation.

Due to the limitation of our budgets, we only translate several datasets of MCQA and NER. We only perform evaluation on models with 7B/8B model parameters. For a comprehensive evaluation, we should also perform comparison in a larger scale. We leave it as a future work to include more translated large-scale datasets in other tasks and evaluation results of larger models. Moreover, though we evaluate these models with four categories of prompt templates, each category only contains one template, which may introduce some fluctuation. To further improve the robustness of our benchmark, we consider including more diverse prompt templates in each prompt category in the future.

Evaluation results on Japanese general domains and biomedical domains in other languages are also valuable for comparison, providing some insights into developing Japanese biomedical LLMs. Such multilingual biomedical benchmark containing diverse tasks is a promising research direction in the future. However, it is out of our scope in this paper.

Ethics Statement↩︎

We follow the licenses of the involved datasets, which are mainly MIT or CC-BY-4.0⁸. However, we should note that the NRNER and JCSTS datasets are distributed under the Non-Commercial CC-BY-NC-SA-4.0 license⁹. In principle, the whole JMedBench should be distributed under a non-commercial license, whereas if it is used for the commercial scenario, these two datasets (i.e., NRNER and JCSTS) should be excluded.

Besides, considering the scale of the existing human-manufactured evaluation datasets, we adopt machine translation system (i.e., GPT-4) to translate some large-scale and high-quality English biomedical datasets into Japanese to fulfill a robust evaluation. However, machine translation system will inevitably generate unfaithful contents. Therefore, those who want to use our datasets to develop faithful biomedical LLMs or biomedical products for real-world application should be aware of this limitation.

Acknowledgments↩︎

This work was supported by JST SPRING, Grant Number JPMJSP2108 and by Cross-ministerial Strategic Innovation Promotion Program (SIP) on "Integrated Health Care System" Grant Number JPJ012425.

6 Benchmark Construction Details↩︎

6.1 Further details of datasets in the JMedBench↩︎

Table 5 shows the statistics of involved datasets in the JMedBench. IgakuQA does not have an official training set, while its genre is similar to MedQA. Therefore, we share the training set of MedQA with IgakuQA for a few-shot evaluation. JMMLU-medical only contains the translated testing set, and we also share the training set of translated MMLU-medical-JP with JMMLU-medical. Considering our limited budgets, we only translated 1,000 training samples randomly selected from the original training set of the PubMedQA. As for the datasets derived from JMED-LLM, including EJMMT, MRNER-Medicine, MRNER-Disease, NRNER, CRADE, RRTNM, and SMDIS, we randomly split a small subset from the original dataset for few-shot evaluation. The size of the training set can be found in Table 5. As for the JCSTS, we also randomly split a small subset to be the training set. For the rest of the datasets, we strictly follow the origin setting of the split and use the training set or development set for few-show evaluation.

6.2 Prompt Templates for Data Augmentation↩︎

Table 6 shows the prompt template we used when using OpenAI’s APIs for translating biomedical MCQA datasets.

Besides, Table 7 is the prompt template for translating biomedical NER datasets.

6.3 Bad Cases during NER Dataset Translation↩︎

We summarized three main failure types during machine translation: (1) ambiguity of a single word, for example, ‘depression’ can be considered as a mental illness (

) or pressing down (

); (2) multiple possible expressions of a single word, for example, ‘glucose’ can be translated into either

or

; (3) differences in grammar between English and Japanese. Table 8 shows one bad case for each typical failure type during translating NER datasets. The parts underlined indicate an inconsistency between the entity and the text translation. Although there is a small number of failure cases during the machine translation phase, we still realize that the quality of the translation for both the entities and the text is very high during the manual modification process, which can prove the reliability and the scalability of our data augmentation method.

7 Experimental Details↩︎

7.1 Development in chronological order↩︎

We sorted the various models according to their release dates. In chronological order, they are: Llama2-7B (Jul. 2023), SwallowLM-7B (Nov. 2023), Meditron-7B (Dec. 2023), Mistral-7B (May 2024), MMed-Llama3-8B (May 2024), Qwen2-7B (Jun. 2024), Llama3-8B (Jul. 2024), llm-jp-13B (Sep. 2024). Figure 5 illustrates the relationship between model performance and release date. The color of the points represents the corresponding tasks, and the shape represents their models. Colored lines reflect the trend of model performance on each task over time. The figure shows that as time progresses, the performance of models on various tasks is consistently improving, especially for the STS task. Moreover, the improvement in the in-context learning (ICL) capabilities of the models is even more pronounced.

7.2 Ranking of Models↩︎

Figure 6 shows the zero-shot and few-shot performance rankings on JMedBench tasks among all involved LLMs.

7.3 Comparison Methods↩︎

Detailed information for involved comparison methods is listed in Table 9.

7.4 Prompts for Each Task↩︎

Detailed prompt templates for each task are shown in Table 10, 11, 12, 13, and 14.

References↩︎