1 Introduction↩︎

Large Language Models (LLMs) have become the go-to solution for many tasks, showcasing promising results on standard benchmarks, potentially replacing humans across various domains [1], [2]. However, their reliability in tasks that require real attention to detail, such as those directly impacting human life, remains concerning [3]. Medical tasks, such as clinical decision-making, represent a critical domain where experts must possess comprehensive knowledge in cultural contexts, medical principles, pharmaceutical information, and numerous other specialized areas within healthcare. In other words, clinical excellence requires more than just biomedical knowledge [4].

Although recent works have demonstrated that LLMs may achieve high accuracy on English medical question-answering tasks [5], [6], their performance drops significantly in other languages [7], [8]. This gap is particularly pronounced in medicine, where high-quality corpora are centered on English, restricting the models’ applicability in global healthcare settings. Importantly, simply translating questions is inadequate, as such pipelines can strip away critical terminology, subtle cultural cues, and localized standards of care, potentially leading to life-threatening consequences in clinical practice [9].

Medical practice is inherently shaped by contextual factors, including sociocultural, socioeconomic, regional, and healthcare system variables that extend beyond language translation [10], [11]. Clinical decision-making protocols and symptom interpretation vary significantly across healthcare systems and populations due to genetic variations, dietary patterns, climate-related health risks, and socioeconomic determinants, with the same clinical presentation potentially indicating different underlying pathologies across ethnic groups [12]–[15]. Additionally, vaccination schedules, drug availability, and standard-of-care protocols differ markedly between regions, making direct translation of medical guidelines inefficient. These considerations highlight why medical AI systems cannot rely solely on linguistic translation but must incorporate an understanding of regional medical practices and population-specific health patterns.

These contextual complexities are particularly pronounced in low-resource language settings, where the intersection of linguistic barriers and distinct medical practices creates compounded challenges for AI evaluation. Limited research has investigated the specific factors that mislead LLMs in medical contexts, particularly in multilingual and low-resource language settings like Persian. A deeper investigation into the medical sub-fields in which LLMs excel or underperform is essential for identifying suitable use cases and implementing necessary safeguards.

To fill this gap, we introduce PersianMedQA, a large-scale, expert-annotated dataset covering 23 medical specialties. Given the scarcity of standardized Persian medical terminology resources, the dataset includes a comprehensive bilingual dictionary of Persian medical terms for consistent terminology usage during evaluation and model adaptation. As a benchmark, we evaluate state-of-the-art models, including general-purpose models, Persian LLMs, and medical LLMs on both original Persian questions and their English translations. Throughout this work, we use medical LLMs to refer to models that have undergone domain adaptation for medicine, either via domain-adaptive pre-training on biomedical corpora or supervised instruction fine-tuning on medical tasks. Similarly, Persian LLMs refers to models domain-adapted for Persian, via continued pre-training or Persian instruction fine-tuning. Our experiments uncover a substantial language gap: closed-weight models such as GPT-4.1 significantly outperform open-weight counterparts. Notably, Persian LLMs exhibited minimal understanding of the Persian medical field and performed the worst, while medical LLMs showed only modest improvements and failed to generalize effectively to Persian clinical data. Figure 2 illustrates the overall workflow of our study.

Section 2 reviews prior work on medical QA benchmarks and Persian language models. Section 3 describes the PersianMedQA dataset construction. Section 4 presents our experimental setup and evaluations. Section 5 concludes with key findings and future research directions.

2 Related Work↩︎

2.1 Medical Question Answering Datasets and Multilingual Challenges↩︎

Medical question answering has emerged as a critical benchmark for evaluating machine reasoning capabilities in high-stakes healthcare domains. The field has evolved from early information retrieval benchmarks [16], [17] to standardized datasets such as PubMedQA [18], MedQA [19], and MedMCQA [20], driving domain-specific model development like BioBERT [21] and PubMedBERT [22]. However, most benchmarks focus exclusively on English, creating significant evaluation gaps. While native-language datasets have emerged—including CMB [23], Huatuo-26M [24], MedQA-SWE [25], FrenchMedMCQA [26], and HeadQA [27]—many frameworks rely on problematic “translate-test” methodologies [28], [29] (the practice of machine-translating non-English inputs into English before evaluation, rather than evaluating in the source language) that distort clinical terminology and miss culturally-specific practices [28], [29]. Recent multilingual efforts like MedExpQA [8] show around 10% accuracy drops for non-English languages, with critical gaps remaining for low-resource languages requiring native evaluation approaches.

2.2 LLMs in Medical Practice↩︎

LLMs have transformed medical AI applications, with specialized models demonstrating remarkable capabilities on standardized medical examinations. Med-PaLM 2 [30] achieved groundbreaking performance on the USMLE, while general-purpose models like GPT-4 [2] showed impressive zero-shot performance across medical QA benchmarks [31]. Recent advances include open-weight models such as MEDITRON-70B [32], multilingual approaches like MMed-Llama 3 [33] covering six languages, and specialized Chinese models such as TCMChat [34] and BianCang [35]. However, systematic evaluation across diverse languages and clinical settings remains limited, particularly for morphologically rich and low-resource languages such as Persian.

2.3 Persian Language Models and Medical Applications↩︎

Persian natural language processing has witnessed significant progress with the development of robust monolingual models. ParsBERT [36] established strong baselines for various Persian NLP tasks, consistently outperforming multilingual alternatives on sentiment analysis and text classification benchmarks. Recent advances include Dorna [37], a large-scale Persian language model. In the medical domain, SINA-BERT [38] represents an early attempt at Persian medical NLP, utilizing pre-training on large-scale medical corpora including both formal and informal medical texts from diverse online resources. Furthermore, existing Persian medical NLP efforts lack the expert validation and standardized evaluation protocols necessary for reliable clinical assessment, highlighting the need for comprehensive Persian medical QA benchmarks with rigorous validation procedures.

3 PersianMedQA Construction↩︎

The PersianMedQA dataset was developed by collecting 14 years of multiple-choice questions from the official Iranian medical residency and pre-residency examinations administered by the Medical Education Assessment Center (Sanjeshp) under the Iranian Ministry of Health, the governmental body responsible for national standardized examinations in the medical domain, analogous to organizations such as the National Board of Medical Examiners (NBME) in the United States. These exams serve as the mandatory licensing gateway for medical graduates seeking specialist training in Iran, ensuring that all questions reflect real-world, high-stakes clinical evaluation standards at the national level. Each exam was created by the official Iranian medical board and reflects real-world, high-stakes evaluation standards. Each item includes the question text, four answer options, the correct answer key, and the medical field to which the question belongs. Figure 1 presents representative examples of clinical and non-clinical questions. The raw dataset underwent a rigorous preprocessing pipeline to ensure quality, consistency, and relevance for multilingual medical QA evaluation.

3.1 Data Cleaning and Filtering↩︎

In order to eliminate noise and redundancy, we ran a three-step cleaning pipeline:

• Duplicate Removal: Automatically prune exact and near-duplicate questions using string matching and sentence-embedding similarity from the Language-agnostic BERT Sentence Embedding (LaBSE) model [39] to maintain diversity.

• Image Dependent Exclusion: Discard any question that relies on medical images (e.g., radiographs, histology slides) so the benchmark remains purely text-based.

• Answer Key Verification: All reference answers were provided by the official Iranian Medical Board and underwent a rigorous three-level verification process by the National Center for Medical Education Assessment (Sanjesh): (1) Initial review by board-certified medical professionals, (2) Public comment period where students and practitioners could report concerns or discrepancies, and (3) Final review and correction incorporating community feedback. In addition to this official validation, we manually removed questions with incorrect answer formatting (e.g., missing or mismatched option numbers) and those with multiple plausible correct answers to ensure evaluation reliability.

3.2 Annotation and Categorization↩︎

To enhance interpretability and analysis, the cleaned dataset was annotated as follows:

• Subject Verification: Most questions already contained subject tags (i.e., the medical specialty labels shown in Figure 3, such as cardiology, surgery, or pediatrics) from the original examination. For questions lacking subject tags, both medical specialists and Gemini 2.5-Flash independently classified them, achieving over 90% agreement. Final subject labels were determined through expert medical review to ensure high accuracy.

• Domain Classification: Questions were labeled as clinical (patient cases and diagnosis) or non-clinical (basic sciences and theoretical concepts). This classification was performed using Gemini 2.5-Flash and validated by a medical specialist.

• Demographic Extraction: We utilized Gemini 2.5-Flash to automatically extract patient attributes (e.g., age, gender) for every question, motivated by the need to analyze data distribution to ensure comprehensive representation across patient demographics and enable future research on potential LLM performance gaps in specific demographic subgroups.

All medical annotations and validations were performed by a board-certified internal medicine specialist with 5 years of clinical practice (see Appendix 8 for full details).

3.3 Dataset Overview↩︎

The PersianMedQA dataset comprises 20,785 unique, expert-validated multiple-choice medical questions, collected over 14 years from Iranian national residency and pre-residency exams. Approximately 70% of the questions are classified as clinical, with the remaining 30% labeled as non-clinical. The items span 23 medical specialties, covering a broad range of topics relevant to medical education and practice.

We partitioned PersianMedQA into training (14,549), validation (1,000), and test (5,236) splits using stratified sampling based on year and field to facilitate future research, including potential fine-tuning studies, and to establish standardized evaluation protocols. Figure 3 summarizes the distribution of questions across medical domains.

Demographic Coverage: To ensure comprehensive representation and enable future bias analysis, we systematically extracted patient demographics using LLM-based extraction validated by medical experts. Table 1 presents the distribution across gender, age categories, and question types. The dataset demonstrates balanced representation across demographics, with substantial coverage of both male and female patients, diverse age groups from infants to adults, and a clinically-relevant mix of case-based and theoretical questions.

3.4 Data Contamination and Evaluation Integrity↩︎

To ensure the reliability of our medical evaluation, we implemented multiple safeguards against data contamination and memorization artifacts:

Secure Sourcing: The dataset is not from easily crawlable, free public websites. Questions come from official Iranian medical residency exams administered by Sanjeshp in PDF format, providing an additional layer of protection against training data leakage.

Exact Search: We conducted exact match searches on a randomly sampled subset of under 50 questions, querying both full question stems and isolated key medical terms across publicly accessible web sources. The analysis revealed minimal overlap, with the vast majority of sampled questions returning no verbatim matches on any publicly accessible platform, indicating limited or no data leakage into LLM training corpora.

Temporal Analysis: To empirically verify the absence of data leakage, we performed a year-by-year accuracy analysis across examinations from 2011-2024 (Figure 5). Model performance remained consistent even on 2023-2024 questions that post-date training cutoffs for most evaluated models, strongly suggesting our dataset was not present in training corpora. The observed performance decrease in 2020-2021 likely reflects increased exam difficulty during the COVID-19 pandemic period rather than data inconsistencies, as students had extended study time and examiners reportedly created more challenging assessments.

3.5 English Translation↩︎

To enable bilingual evaluation, we generated English translations of the dataset using three methods: Google Translate, the GPT-4.1 API, and the Gemini-2.5-Flash API. All translations were assessed by our board-certified medical expert for accuracy of medical terminology and preservation of clinical meaning. Both GPT-4.1 and Gemini-2.5-Flash produced more accurate, natural translations than Google Translate, with expert validation confirming superior preservation of medical concepts and terminology. Due to its combination of quality, accessibility, and expert-validated accuracy, we use Gemini-2.5-Flash translations as our default in all subsequent experiments.

4 Experiments↩︎

4.1 Zero-shot Scenario↩︎

We conducted zero-shot evaluations on the PersianMedQA dataset using a wide range of state-of-the-art open-weight and closed-weight LLMs in both Persian and English. All models were evaluated with temperature set to 0 and allowed to generate up to their maximum generation length. We employed identical prompts and evaluation protocols across all models to ensure fair comparison. Figure 6 shows the complete prompt template used for all zero-shot evaluations. We used English prompts across both Persian and English evaluation settings, as most models demonstrated superior instruction-following capabilities in English compared to Persian. All results reported in this study are based on the 5,236-question test set to facilitate reproducible evaluation and reduce computational costs. We validated this approach by running experiments on multiple models using the full dataset, confirming that results were consistent with the smaller test set.

Figure 4 presents the overall accuracy of the evaluated models on both Persian and English test sets. Among all models, the closed-weight GPT-4.1 achieved the highest zero-shot accuracy in both languages, scoring 83.09% in Persian and 80.71% in English. Notably, the best-performing open-weight model, DeepSeek-Chat-V3, achieved a strong 68.05% in Persian and 73.30% in English, followed closely by LLaMA-3.1-405B-Instruct with 67.02% in Persian and 73.49% in English. In terms of medical-tuned models, Meditron3-8B scored only 38.67% in Persian and 50.00% in English, revealing substantial room for improvement in domain adaptation for Persian.

Persian LLMs significantly underperformed across the board; some of them suffered greatly from poor instruction-following ability. PersianMind-1.0 achieved only 24.22% in Persian (roughly equivalent to random guessing) and 25.17% in English, suggesting limited medical knowledge and inadequate generalization capability in clinical domains. Similarly, Dorna2-LLaMA-3.1-8B-Instruct, another Persian LLM, scored just 34.87% in Persian and 51.24% in English, indicating slightly better instruction following but still poor domain alignment in the Persian medical setting.

Overall, closed-weight models consistently outperformed both open-weight and medical LLMs, particularly in Persian. While most models exhibited performance degradation when evaluated in Persian compared to English, some top-tier models — such as GPT-4.1 (83.09% Persian vs.% English) and Gemini-2.5-Flash-Preview (82.37% Persian vs.% English) — actually scored higher on the original Persian questions. This is consistent with our translation analysis: the English set is machine-translated from Persian, and even high-quality translation introduces subtle semantic drift and loss of clinical context that can slightly disadvantage models on the translated version.

We further analyzed model performance across different medical specialties. Figure 7 presents a heatmap of accuracy scores for each model across all medical fields in the PersianMedQA dataset.

Several factors shaped model performance across medical subfields. For example, pharmacology questions, which hinge on factual recall rather than complex clinical reasoning, yielded the highest accuracies for most models. Likewise, non-clinical items (theoretical or basic-science questions) tended to be answered more accurately than clinical case scenarios, reflecting their relatively straightforward nature.

In contrast, performance dropped sharply in subfields such as surgery and medical statistics, which require complex reasoning, quantitative interpretation, and a deeper understanding of language-specific clinical guidelines and protocols. These findings show that factual recall alone is insufficient: robust medical QA calls for deeper reasoning and cultural grounding across subfields.

4.2 Translation Impact↩︎

English dominates both the web-scale corpora that power modern LLMs and the medical literature on which they are trained. This bilingual evaluation is crucial for understanding a key trade-off in multilingual medical AI: while translating questions into English may align them better with a model’s core knowledge base, it risks erasing subtle clinical guidelines and cultural contexts unique to local practice. To quantify this effect, we translated the PersianMedQA dataset into English and compared model performance on the original Persian versus the translated English questions.

To better understand model behavior across languages, we analyzed performance based on whether questions were answered correctly in only Persian, only English, or both. As expected, most models performed better on the English translations—even Persian LLMs—reflecting their predominant exposure to English medical data during training. However, a substantial subset of questions (ranging from 3-10% across models) was answered correctly only in the original Persian.

Manual analysis showed these cases often involved crucial local context that is lost in translation. These included healthcare system-specific protocols where Iranian clinical guidelines differ from Western standards, population-specific clinical considerations like regional disease prevalence, and semantic drift where the precise meaning of Persian medical terms is altered. Figure 8 illustrates a representative case where Iran-specific antibiotic protocols lead to correct answers only when questions are presented in Persian, as translation obscures regional resistance patterns and drug availability that differ from Western guidelines.

Such translation errors were most damaging in fields like surgery, where imprecise terminology led to incorrect answers even when the underlying reasoning was sound (see Appendix 16 for further examples).

4.3 Impact of Model Size↩︎

Our analysis of model size versus performance revealed that scale is not a universal solution. While larger general-purpose models like GPT-4.1 (83% accuracy) clearly outperformed their smaller counterparts, increased size offered no advantage for specialized models. Both large medical-specific (e.g., MedAlpaca-13B) and Persian-tuned (e.g., Dorna2-LLaMA-3.1-8B) models struggled significantly, often scoring below smaller general-purpose models. These results underscore that model scale must be paired with high-quality, domain-relevant training data to achieve strong performance.

4.4 Prompting Strategies and Few-shot Learning↩︎

We experimented with various prompting strategies and few-shot learning approaches; the results are summarized below.

Role-based prompting, where the model was instructed to act as a specialist based on the medical field of the question (e.g., "You are a cardiologist..."), resulted in slightly improved performance, but the gains were marginal.

Few-shot learning: For every test question, we drew the in-context examples exclusively from the PersianMedQA training split (up to k = 5 per query). We experimented with several retrieval schemes for picking those training examples: LaBSE cosine similarity, TF-IDF, and random selection, but none of them produced consistent gains over the zero-shot baseline. A plausible reason is the absence of high-quality embedding models tailored to Persian medical text, which makes it difficult to retrieve truly helpful training examples.

We also experimented with augmenting each question with a medical dictionary, extracted by a larger, more capable model (Gemini-2.5-Flash), that provided both translations and concise definitions of key terms. This dictionary is released alongside the dataset to help smaller models interpret domain-specific terminology. However, we found that this augmentation had a negligible effect on overall performance, especially for weaker or instruction-tuned models.

4.5 Answer-Only Evaluation of LLM Medical Reasoning↩︎

To test whether LLMs genuinely understand medical questions or merely exploit statistical patterns in answer choices, we adopted the partial-input protocol of [40]. Each model received only the four answer options without the question stem. Gemini-2.5-Flash-Preview achieved 35.60% accuracy, substantially outperforming random guessing (25%).

Manual inspection revealed that models exploited recurrent answer-choice artifacts, particularly evident in medical ethics—the highest-performing field at 46.8% accuracy. In ethics questions, models can infer correct answers through: (i) hierarchical ethical principles, where options containing phrases like "patient autonomy," "informed consent," or "professional disclosure" signal standard bioethical frameworks; (ii) logically exclusive options, where choices violating fundamental medical ethics (e.g., "withhold information from patient") can be discarded; and (iii) linguistic cues, where options with formal ethical terminology indicate textbook-correct responses. These patterns suggest that high performance in ethics may reflect recognition of moral vocabularies rather than genuine ethical reasoning, suggesting that medical MCQ benchmarks may overstate LLM capabilities by permitting exploitation of answer-choice artifacts.

4.6 Model Ensembling↩︎

Different models exhibit varied strengths across medical subjects, suggesting that ensembling diverse models can improve accuracy. Since top-performing models like GPT and Gemini are not open-weight, developing open-weight ensembles remains highly valuable.

As shown in Table 2, majority-vote ensembles of diverse model families outperformed single models, whereas same-family ensembles offer little benefit. Notably, an ensemble of five open-weight models achieved 73.7% accuracy, a significant gain over the best individual model in that group (+0.033).

4.7 Chain-of-Thought Evaluation↩︎

We evaluated the impact of Chain-of-Thought (CoT) prompting on four models: two large general-purpose models (GPT-4.1, Gemini 2.5 Flash), one medical model (Meditron3), and one Persian-language model (Dorna).

Performance Gains: For large general-purpose models, CoT improved accuracy by approximately 2%, with the greatest gains on clinical questions, highlighting that clinical scenarios particularly benefit from explicit reasoning steps. Smaller models showed negligible improvement, likely due to weaker language and reasoning capabilities.

Expert Analysis of CoT Errors: A clinical expert reviewed GPT-4.1’s CoT responses and identified four primary error types:

• Contextual Mismatch: Applying reasoning based on non-Iranian clinical protocols.

• Ambiguity in Options: Failing to distinguish between very similar or misleading answer choices.

• Reasoning Failures: Exhibiting illogical or incomplete reasoning despite possessing the required knowledge.

• Knowledge Gaps: Lacking the necessary factual information to answer correctly.

Examples for each category are available in Appendix 10.

5 Conclusion↩︎

In this study, we present PersianMedQA, a dataset of 20,785 expert-validated Persian medical questions from 14 years of Iranian national medical exams, designed to evaluate how well current language models understand medical content across Persian and English contexts.

Our evaluation of 41 models revealed a significant performance hierarchy: closed-weight models like GPT-4.1 (83.1% Persian, 80.7% English) substantially outperform open-weight alternatives, with the best open-weight model (LLaMA-3.1-405B) achieving 67.0% in Persian. Persian LLMs performed poorly (Dorna: 34.9%), while medical LLMs showed only modest improvements over general models. Critically, our cross-linguistic analysis revealed that 3-10% of questions require Persian-specific cultural and clinical knowledge, demonstrating that simple translation-based evaluation approaches are inadequate for medical AI in non-English contexts.

Future work should focus on developing Persian medical corpora for domain-specific training, creating retrieval-augmented systems that can access culturally appropriate medical guidelines, and expanding evaluation to other low-resource languages and multimodal medical contexts. Additionally, the training split of PersianMedQA remains underutilized in this study; future work should explore fine-tuning open-weight models directly on this split and compare against the prompt-based baselines established here. This work establishes a foundation for culturally grounded medical AI evaluation beyond English-centric benchmarks.

6 Limitations↩︎

Several factors constrained this study. (i) API restrictions and computational resources: Cost and rate limits for commercial LLMs (e.g., GPT-4.1) reduced the number of evaluation runs and chain-of-thought variants we could conduct. Additionally, the lack of access to large-scale GPU infrastructure precluded fine-tuning experiments on the training split, which we leave for future work. (ii) Licensing barriers: copyright restrictions prevented us from using larger multilingual biomedical corpora, limiting the scope of our experiments. As a result, our reported scores should be considered conservative lower bounds; broader data access and greater computational resources would enable a more exhaustive evaluation. (iii) Generalization scope: The exam-based format of PersianMedQA is best suited for evaluating physician-level clinical reasoning. Results may not generalize to patient-focused question answering, such as informal health queries collected from online forums or patient communities, which represent a distinct and complementary evaluation setting.

7 Ethics Statement↩︎

This study involved the analysis and evaluation of LLMs on publicly available medical examination data. The dataset consists exclusively of multiple-choice examination questions from official Iranian medical licensing exams and contains no patient records, clinical notes, or any personally identifiable information. No anonymization was required, as the data is entirely de-identified by nature — all questions are impersonal exam items testing medical knowledge, not individual cases.

Regarding intellectual property, the questions are sourced from official Iranian national medical residency examinations administered by Sanjeshp (the Medical Education Assessment Center), a governmental body under the Iranian Ministry of Health. These examination papers are publicly released after each examination cycle and have been widely circulated by medical education institutions, test-preparation resources, and commercial study materials for over a decade. The dataset is used exclusively for non-commercial academic research purposes, consistent with established practices in the Persian NLP research community.

PersianMedQA is publicly available to support reproducible research and further development of Persian medical NLP. The dataset and bilingual medical dictionary are available at PersianMedQA.

Our findings aim to improve the responsible deployment of language models in healthcare, especially for underrepresented languages. We emphasize that the models tested are not certified for clinical use and should not be deployed in real-world healthcare settings without strict oversight. We advocate for continued expert-in-the-loop development and further inclusion of diverse linguistic and cultural considerations in medical AI research.

8 Medical Specialist Background↩︎

The medical specialist involved in this study is a board-certified internal medicine physician. She graduated from the University of Tehran with a degree in general medicine and completed her specialty training in internal medicine at Shahid Beheshti University of Medical Sciences. She has 5 years of clinical practice and has successfully passed the Iranian national medical licensing examinations. Her contributions to this work include validating subject classification, checking translation accuracy, overseeing the dataset curation process, and performing chain-of-thought (CoT) error analysis.

9 Data Verification and Quality Assurance↩︎

9.1 Answer Verification Process↩︎

As described in Section 3, all questions underwent a rigorous three-level verification process by the National Center for Medical Education Assessment (Sanjesh): (1) Initial expert committee review by board-certified medical professionals, (2) Public comment period where medical students and practitioners can report concerns or discrepancies, and (3) Final review and correction process incorporating community feedback. This established process has ensured high accuracy across 14 years of national medical examinations, though formal inter-annotator agreement scores in the traditional sense were not applicable given the institutional verification structure.

9.2 Subject Classification Validation↩︎

Subject classification was validated by the board-certified medical specialist described in Appendix 8. The specialist reviewed ambiguous cases using a custom Telegram-based annotation interface (see Appendix 13), achieving greater than 90% agreement with Gemini 2.5-Flash classifications over a randomly selected sample of 200 questions. We acknowledge that the absence of a second human annotator means formal inter-annotator agreement scores could not be computed; future work should include dual annotation on a representative subset to formally assess metadata quality.

9.3 Question Diversity Distribution↩︎

The National Center for Medical Education Assessment maintains natural diversity across medical specialties following standardized distributions mandated by the Ministry of Health. Each exam is dominated by the core clinical specialties: Pediatrics (\(\sim\)​26 questions), Surgery (\(\sim\)​24 questions), and Obstetrics/Gynecology (\(\sim\)​18 questions), followed by Infectious Disease (\(\sim\)​10 questions) and Pathology (\(\sim\)​9 questions). The remaining 18 specialties—including Cardiology, Neurology, Psychiatry, Orthopedics, Endocrinology, Dermatology, and others—each contribute between 2 and 8 questions per exam, following official Ministry of Health guidelines.

10 Examples of CoT Error Patterns↩︎

This section presents representative error patterns identified in model-generated CoT outputs, as annotated by our clinical expert. For each example, we highlight the clinical context, the correct answer, the model’s response, and a summary of the expert’s evaluation.

11 Few-shot Evaluation Prompt↩︎

In-context examples are drawn from the PersianMedQA training split using LaBSE cosine similarity, TF-IDF, and random selection (up to \(k=5\)).

            Question: [Example question 1]
            1: [Option 1]  2: [Option 2]
            3: [Option 3]  4: [Option 4]
            Answer: [Correct option number]
            ...
            Question: [Example question k]
            1: [Option 1]  2: [Option 2]
            3: [Option 3]  4: [Option 4]
            Answer: [Correct option number]

            Question: [Medical question text]
            1: [Option 1]  2: [Option 2]
            3: [Option 3]  4: [Option 4]

12 CoT Reasoning Prompt↩︎

            {
                "CoT": "your step-by-step reasoning",
                "Final_Answer": 1 | 2 | 3 | 4,
                "Reasoning": "concise justification"
            }

13 User Interfaces↩︎

To facilitate expert interaction throughout various phases of our study, we developed multiple Telegram-based interfaces to streamline collaboration with our medical specialist.

13.1 Subject Annotation Interface↩︎

We created a Telegram annotation bot to support subject-level classification. The specialist could review ambiguous or unclassified questions and select the most appropriate medical field from a predefined list of 23 specialties.

13.2 CoT Reasoning Interface↩︎

To analyze the reasoning behind model outputs, we designed an interface that presented the expert with a curated 200-question subset. For each question, the expert was asked to: (1) select whether a predefined reasoning category applied (e.g., domain knowledge, commonsense, causal inference), (2) optionally assign a new category if none fit, and (3) provide a brief explanation justifying the correct answer.

14 Persian Medical Dictionary↩︎

Table 3 summarizes the number of unique medical terms extracted per category in the bilingual Persian medical dictionary released alongside the dataset.

15 Overall Performance Comparison↩︎

Table 4 shows the zero-shot accuracy of all 41 evaluated models on the original Persian questions, the translated English questions, and their average. Models are sorted by average performance. The five lowest-performing models struggled significantly with instruction-following, resulting in accuracy scores at or below random guessing.

16 Cross-Linguistic Performance Analysis↩︎

Our cross-linguistic analysis revealed three distinct performance patterns across models. Representative examples for each category are provided below.

Category 1: Correct in Both Languages↩︎

These questions involve standardized clinical protocols and universal pathophysiological concepts that transfer seamlessly across languages.

Category 2: Correct Only After Translation↩︎

These questions benefit from the model’s stronger English medical training, particularly in specialized terminology.

Category 3: Correct Only in Persian↩︎

These questions involve Iran-specific medical practices or clinical contexts that are altered or lost during translation.

References↩︎