¹

1 Introduction↩︎

Large multimodal language models (MLLMs) have demonstrated remarkable capabilities in integrating visual and textual information to perform complex reasoning [1]–[7]. Despite their promise, current MLLMs remain fundamentally constrained by static supervision, where fixed prompts, rules, or reward models provide non-adaptive guidance throughout training [8], [9]. While such supervision can enforce output formats or improve final answer accuracy, it rarely shapes the underlying reasoning process, often leading models to learn brittle heuristics instead of robust, expert-like logic [10]–[13].

Supervised fine-tuning (SFT) on annotated datasets has been the primary method for aligning MLLMs to specific reasoning tasks [4], [6], [8]. However, the acquisition of high-quality, expert-annotated data is labor-intensive and expensive, creating a bottleneck for scaling reasoning capabilities [10], [11]. Reinforcement learning (RL) methods have been explored to address these limitations, optimizing models with scalar reward signals derived from accuracy, format compliance, or other heuristics [9], [14]–[16]. While these methods can improve generalization beyond SFT, they still act as “fixed tutors”, providing limited feedback that enforces output correctness but does not adaptively shape intermediate reasoning [17]. Consequently, models often plateau in performance, fail to generalize to complex cases, and remain prone to reward hacking [12].

Human learning provides a compelling analogy for overcoming these limitations. Experts do not rely solely on pass/fail evaluations; instead, they acquire abstract reasoning principles, apply them to diverse cases, and iteratively refine them based on experience [18], [19]. This process enables transfer to unfamiliar or complex situations and supports robust, flexible decision-making. Inspired by this cognitive process, we hypothesize that evolving principles, rather than static scalar rewards, should serve as primary supervision signals to guide MLLMs toward expert-level reasoning. Figure 1 illustrates the key insights of our idea.

To operationalize this idea, we introduce Evo-PI, a framework for Aligning Medical Reasoning via Evolving Principle-Guided Supervision. Evo-PI establishes a co-evolutionary loop in which (i) a principle bank distills initial reasoning guidelines from curated examples, (ii) the backbone MLLM receives dense feedback on its adherence to these principles via a frozen judge model, and (iii) the principles themselves are iteratively refined based on the model’s evolving failures [5], [6]. This dynamic alignment mechanism transforms static supervision into a continuous dialogue between model and principles, progressively narrowing the gap between the model’s reasoning and expert standards.

We evaluate Evo-PI in the high-stakes domain of medical Visual Question Answering (VQA) [20], which requires integrating visual evidence with structured medical knowledge. Across eight benchmarks and multiple MLLM backbones, Evo-PI consistently improves reasoning accuracy, achieving gains of up to 24.6% compared to models trained with static supervision. Qualitative analysis further demonstrates that Evo-PI produces more coherent and clinically valid reasoning traces, validating the effectiveness of language-based supervision in shaping intermediate reasoning rather than merely optimizing final answers.

In summary, our contributions are:

• We propose Evo-PI, a principle-guided supervision framework that replaces static scalar rewards with evolving, language-based reasoning principles, mimicking the iterative learning process of human experts.

• We introduce a co-evolutionary mechanism where the reasoning model and its supervising principles mutually refine each other, mitigating reward hacking and enhancing reasoning robustness.

• We empirically demonstrate that this evolving supervision paradigm scales reasoning capabilities across diverse MLLM backbones and medical VQA testbeds, providing a generalizable path toward expert-aligned multimodal reasoning.

2 Method↩︎

Existing training paradigms lack a principled way to transform abstract domain knowledge into adaptive supervision that evolves with the model. Consequently, reinforcement learning for multimodal reasoning relies on static scalar rewards that shape outputs but do not guide reasoning. We propose Evo-PI, an iterative framework that externalizes reasoning principles as explicit supervision signals and continuously refines them via model feedback.

2.1 Problem Definition↩︎

In medical VQA, an MLLM answers text questions about a medical image. Formally, given a medical MLLM \(\mathcal{M}_{\theta}\) with its parameters \(\theta\), the input is a set of medical VQA questions. Each medical VQA question \(q\) consists of a medical image \(I\) and a textual question \(T\), which can be represented as: \[\begin{align} q &= (I, T), \\ I &\in \mathbb{R}^{H \times W \times C}, \\ T &= (w_1, w_2, \dots, w_n), \quad w_i \in \mathcal{V}, \end{align}\] where the textual question \(T\) is represented as a sequence of \(n\) tokens, with each token \(w_i\) drawn from the vocabulary \(\mathcal{V}\). \(H\), \(W\), and \(C\) represent the height, width, and channel dimension of the image, respectively.

The model predicts a textual answer \(\hat{a}\) to the medical question, represented as a sequence of tokens: \(\hat{a} = (y_1, y_2, \dots, y_m)\), \(y_j \in \mathcal{V}\), where the answer \(\hat{a}\) consists of \(m\) tokens, and each token \(y_j\) is drawn from the same vocabulary \(\mathcal{V}\). A commonly adopted likelihood-based objective for medical VQA is to maximize the conditional probability of the answer given the input, defined as: \[\begin{align} &\max_{\theta} \; \log \mathcal{M}_{\theta}(\hat{a} \mid I, T) \\ =& \max_{\theta} \quad \sum_{j=1}^{m} \log \mathcal{M}_{\theta}(y_j \mid y_{<j}, I, T). \end{align}\] However, this formulation provides no explicit supervision on reasoning quality or clinical validity, motivating the need for principle-guided supervision in complex medical settings.

2.2 Overall Framework↩︎

Evo-PI implements a paradigm for aligning model reasoning with domain principles in a dynamic and iterative manner. Rather than treating medical knowledge as a static resource, Evo-PI externalizes knowledge into editable reasoning principles that serve as dense supervision signals. These principles are continuously refined in response to the model’s evolving behaviors, enabling a co-evolutionary learning loop that promotes robust and generalizable reasoning.

As illustrated in Figure 2, Evo-PI consists of three main stages:

(1) Principle Bank Initialization. A frozen, knowledgeable LLM is prompted with few-shot examples and category information to distill a set of initial reasoning principles. These principles capture clinically and visually grounded heuristics for interpreting medical images and textual questions, forming the basis for dynamic supervision during training.

(2) Principle-Guided Learning. The backbone MLLM generates answers along with reasoning chains in a rollout process. A frozen judge LLM evaluates adherence to the relevant principles, yielding a Principle Reward (for guideline adherence) and a Thinking Point Reward (for logical completeness). These dense signals are combined with a base RL environment reward to update the model [9], shaping not only the final answer but also the underlying logical process.

(3) Principle Evolution. After each iteration, the knowledgeable LLM reviews model behaviors and failure cases to refine and expand the principle set. Updated principles are then used in the next iteration, creating a feedback loop where model reasoning and guiding principles co-evolve. The process terminates once convergence criteria are satisfied.

By operationalizing supervision at the level of evolving principles rather than static scalar rewards, Evo-PI enables MLLMs to progressively improve reasoning fidelity, generalization, and alignment with expert knowledge.

2.3 Principle Bank Initialization↩︎

Unlike prior approaches that rely on static heuristics or fixed reward definitions, Evo-PI externalizes domain knowledge into an explicit and editable principle bank. Inspired by clinical training, where abstract diagnostic principles are distilled and refined through repeated case analysis, this stage converts tacit medical reasoning knowledge into structured language-based supervision. Specifically, given a medical VQA dataset with type annotations \(\mathcal{C}\), we sample a small anchor set of question-answer pairs \(\mathcal{Q}_C\) for each medical type \(\text{c}\in \mathcal{C}\). Next, Evo-PI employs a frozen Knowledgeable LLM(\(\mathcal{M}_{K}\)) first to generate a set of candidate principles \(P_c\) conditioned on a medical type name \(\text{c}\in \mathcal{C}\) (e.g., MR, CT) , for all \(\text{c}\in \mathcal{C}\). Subsequently, \(\mathcal{M}_{K}\) is then prompted with a small random set of question–answer pairs \(\mathcal{Q}_C\) drawn for type as context, and refines the principles \(P\) for coverage and specificity. The initial principle bank is denoted as \(P = \bigcup_{\text{c} \in \mathcal{C}} P_c\). The resulting principles are stored for downstream use. Prompt templates for both initial generation and iterative evolution are provided in Appendix 7.6.

We use a text LLM rather than an MLLM at this stage because the goal is medical plausibility, rather than visual grounding. This decouples principle creation from the backbone and modality, requires no shared architecture or pre-training corpus, and avoids alignment constraints.

2.4 Principle-Guided Reinforcement Learning↩︎

The objective of this stage is to train the medical MLLM to answer visual questions by converting editable principles into reward signals, overcoming fixed heuristic rewards and the lack of knowledge updates in prior work. This formulation allows Evo-PI to move beyond outcome-based supervision, providing dense feedback on how reasoning unfolds, rather than merely whether the final answer is correct. For each medical VQA instance, the backbone MLLM performs a rollout to produce a reasoning trace and an answer. A frozen judge LLM (\(\mathcal{M}_J\)) evaluates adherence to the current principle set and returns principle-based rewards. These are merged with the base RL reward, and the composite signal is used to update the parameters of this medical MLLM (\(\mathcal{M}_{\theta}\)).

Specifically, given a medical VQA input \(q=(I,T)\), Evo-PI prompts the learnable medical MLLM \(\mathcal{M}_{\theta}\) with an answer format that first elicits a step-wise reasoning trace \(rt\). Then a final answer \(\hat{a}\) based on this reasoning trace \(rt\). The medical MLLM (\(\mathcal{M}_{\theta}\)) completes one rollout \(o=(rt, \hat{a})\) or more \(\{o_i\}_{i=1}^{G}\), depending on the chosen RL algorithm. The LLM judge (\(\mathcal{M}_J\)) scores \(o\) against the principles to produce a reward for the principles and a reward for the point of thought, which is combined with the environment reward to update \(\theta\). The detailed prompt templates are available in Appendix 7.8.

2.4.0.1 Principles Judge and Principle Rewards

Evo-PI concatenates the principles \(P\) from the previous stage into a description paragraph \(D=\bigoplus_{i=1}^{n} P_i\), where \(\oplus\) denotes the concatenation operation. A frozen principles judge LLM (\(\mathcal{M}_J\)) compares the description paragraph \(D\) with the reasoning trace \(rt\), and returns a count \(c\) of satisfied principles. The principle reward can be formally defined as: \[\text{R}_P = \begin{cases} \dfrac{c}{|P|}, & 0 \leq c \leq |P|, \\ 0, & c < 0 \;\text{or}\;c > |P|, \end{cases}\] where the constraint \(c>0\) is designed to prevent MLLM from engaging in reward hacking by releasing large amounts of repetitive content in the reasoning traces \(rt\). In contrast, \(c<0\) reflects the error of the judge from \(\mathcal{M}_J\) and is neutralized by clipping. The principle rewards are normalized by \(P\) and clip out-of-range values. Simultaneously, the same frozen judge (\(\mathcal{M}_J\)) verifies, step by step, that the reasoning trace \(rt\) supports the final answer \(\hat{a}\), analogous to a clinician-style verification workflow. This design transforms abstract principles into verifiable language-based constraints, enabling principled supervision while mitigating reward hacking commonly observed in free-form reasoning rewards.

2.4.0.2 Thinking Point Reward

Beyond principle satisfaction, Evo-PI introduces a thinking point reward to encourage structured and verifiable intermediate reasoning. Rather than evaluating free-form chains of thought, this reward focuses on whether the model explicitly addresses the required reasoning components implied by the principles.

Specifically, Evo-PI extracts the reasoning trace enclosed by <think> tags and parses enumerated reasoning points using regular expressions. Let \(b\) denote the number of distinct reasoning points that are correctly addressed in the trace. The thinking point reward is defined as: \[\text{R}_T = \begin{cases} \dfrac{b}{|P|}, & 0 \leq b \leq |P|, \\ 0, & b > |P|. \end{cases}\] This reward provides a lightweight and verifiable proxy for reasoning completeness, encouraging the model to articulate clinically meaningful intermediate steps without enforcing a rigid, non-generative template.

2.4.0.3 RLVR Training Procedure

After obtaining reward signals from the Principles Reward (\(_P\)) and Thinking Point Reward (\(_T\)), Evo-PI treats the backbone MLLM as a policy and optimizes it with Reinforcement Learning with Verifiable Rewards (RLVR) [21]. Specifically, Evo-PI adopts GRPO [9] and GSPO [15] as the RLVR trainer. For example, in GRPO, the per-sample scalar return \(r_i\) is formed from the verifiable rewards and the environment signal. The normalized per-token advantage is defined as follows: \[\hat{A}_{i,t} = \frac{r_i - \mu(\{\text{R}_P, \text{R}_T, \text{R}_{RLVR}\})}{\sigma(\text{R}_P, \text{R}_T, \text{R}_{RLVR})},\] where \(\text{R}_{RLVR}\) is the original reward defined by the RLVR algorithm (e.g., format reward and accuracy reward in GPRO). Accordingly, the policy ratio is defined as follows: \[r_{i,t}(\theta) = \frac{\mathcal{M}_{\theta}\big(o_{i,t} \mid q, o_{i,<t}\big)}{\mathcal{M}_{\theta_{\text{old}}}\big(o_{i,t} \mid q, o_{i,<t}\big)},\] where \(o_i\in\) denotes the \(i\)-th rollout (i.e., reasoning trace \(rt_i\) and answer sequence \(\hat{a}_i\)) generated by the medical MLLM \(\mathcal{M}_{\theta}\) for query \(q\), with \(G\) total rollouts sampled per query. The final GRPO objective used to update the \(\theta\) combines the principle and thinking point signals with the environment reward to promote stable policy improvement, is given by: \[{!}{ \begin{align} &\mathcal{J}_{\text{Evo-PI}}(\theta) = \mathbb{E}_{\substack{(q, A) \sim \mathcal{D} \\ \{o_i\}_{i=1}^G \sim \mathcal{M}_{\theta_{\text{old}}}(\cdot \mid q)}} \Bigg[ \quad \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \\ &\quad \min\!\Big( r_{i,t}(\theta) \hat{A}_{i,t}, \text{clip}\big(r_{i,t}(\theta), 1 - \varepsilon, 1 + \varepsilon\big) \hat{A}_{i,t} \Big) \Bigg]. \end{align} }\]

2.5 Principle Evolution↩︎

As the backbone model improves, static supervision inevitably becomes misaligned with its evolving reasoning capabilities. Principle evolution enables supervision to scale in abstraction and coverage, maintaining alignment throughout training. Phase three consists of three core components to adapt the set of principles by controlling the iteration, scaling the principles, and refining them, rather than relying on fixed rewards or static knowledge during training.

Iteration and Generalization Control To prevent overfitting to cases produced during training and limit drift from medical knowledge, we cap the number of loop passes and apply early stopping on a held-out validation set.

Balance of Exploitation and Exploration We track token-level policy entropy during the process of reinforcement learning. Training terminates typically under two conditions: Entropy Collapse [22] and Abnormal Entropy Increase. Entropy collapse is a common phenomenon indicating a reduction in the MLLM’s exploration space, as the model tends to become more certain of its own inferences, also known as exploitation over exploration. High entropy indicates that the policy prioritizes exploration to discover solutions, whereas low entropy signifies a focus on consistently exploiting optimal actions. In Evo-PI, medical MLLMs serve as policy models that are updated by the RLVR trainer, where the balance of exploring diverse case patterns with reliably solving individual cases makes entropy a suitable stopping signal. Because principles are embedded in the judge and never exposed to the MLLM, reward hacking is curtailed, though mild entropy rises can occur. In practice, Evo-PI terminates training if entropy falls to a collapse threshold or exceeds the baseline recorded after the first MLLM update.

Scaling and Refinement of Principles If termination criteria are not met, a frozen Knowledgeable LLM receives the current principles from the latest round and prompts it to scale and refine these principles. The update increases coverage while keeping rules abstract, concise, and compositional by merging redundant rules, pruning low-utility rules, and adding new numbered rules only for recurring failure modes. Prompts for this step are given in Appendix 7.7.

3 Experiments↩︎

We conduct quantitative and qualitative experiments to evaluate Evo-PI.

3.1 Experimental Setting↩︎

3.1.0.1 Datasets

We conduct experiments on the OmniMedVQA dataset [23], which is used in Med-R1 and designed for the medical VQA task. OmniMedVQA spans eight imaging modalities: Computed tomography (CT), dermoscopy (DER), fundus photography (FP), microscopy images (MI), magnetic resonance imaging (MR), optical coherence tomography (OCT), ultrasound (US), and X-ray. After removing duplicate cases, we split the data into training, testing, and validation sets in an 8:1:1 ratio. Table ¿tbl:statistics? summarizes the statistics of these eight benchmark datasets.

3.1.0.2 Hyperparameter Settings

Evo-PI employs a consistent set of hyperparameters across all eight datasets. The default iteration number is three, with one training epoch in each iteration. The batch size is 256 for all backbone medical MLLMs. For group-based RL algorithms (such as GRPO), the rollout number defaults to 8. For RL algorithms using cilp-higher (e.g., GSPO), Evo-PI set the clip_ratio_low=0.0003 and clip_ratio_high=0.0004.

We recommend knowledgeable and judge LLMs with at least 7B parameters. In our setup, GPT-4o-mini² serves as the knowledgeable LLM for principle generation, scaling, and refinement, and Qwen2.5-VL-7B³ serves as the judge LLM to evaluate the responses from medical MLLM. We implemented parallelism using FSDP on the Verl training framework. All backbone models are trained on 4\(\times\)H100 80GB SXM GPUs, and the judge LLM runs on separate 2\(\times\)H100 80GB GPU. On the largest MR dataset, the total runtime is within 20 hours.

3.1.0.3 Baselines and Evaluation Metrics

We evaluate Evo-PI against two strong medical VQA backbones, HuatuoGPT-Vision  [4] and Med-R1 [8], both state-of-the-art on medical VQA. For each backbone, we report Accuracy before and after applying Evo-PI under identical decoding and data splits. We also conduct a horizontal comparison across base MLLM variants of varying sizes corresponding to the backbones’ underlying models.

3.2 Overall Performance Comparison↩︎

The main quantitative results for the medical VQA task are reported in Table 1. Several consistent trends can be observed. First, Evo-PI consistently improves all backbone MLLMs across the eight imaging modalities and different RL training algorithms. Specifically, Evo-PI achieves accuracy improvements ranging from 10.34% to 34.68%, demonstrating robust effectiveness under diverse medical imaging conditions. Second, Evo-PI remains effective when applied to strong medical VQA backbones. On HuatuoGPT-Vision (7B), Evo-PI improves the backbone performance by an average of 24.59%. Moreover, on Med-R1 models trained with the same GRPO optimizer, Evo-PI yields an average improvement of 17.18%, indicating that the proposed framework complements existing RLVR training schemes. Finally, on OCT and ultrasound datasets, Evo-PI–enhanced models surpass 99.3% accuracy for the first time, suggesting that principle-guided supervision can push medical VQA performance toward clinically reliable regimes.

3.3 Impact of Iteration Design↩︎

In Evo-PI, the guiding principles are updated at each iteration. We investigate the respective effects of each factor under an iterative mechanism.

3.3.0.1 Principles Cases in Iteration

At each iteration, the knowledgeable LLM initializes or updates the current set of principles, enabling direct observation of how they evolve.

We observe two general phenomena. (i) When the initial set is incomplete, the knowledgeable LLM refines or decomposes existing principles while introducing additional ones that better meet the task requirements. (ii) When the initial set is already reasonably comprehensive, the knowledgeable LLM performs fine-grained edits, improving individual principles one by one.

Taking the principles for X-ray as an example, we demonstrate how they are generated and how they are iteratively scaled and refined in Appendices 7.6 and  7.7, respectively.

3.3.0.2 Entropy Analysis in Iteration

Entropy quantifies the exploration–exploitation balance of a policy, offering an interpretable lens on learning dynamics [22], [24]. High entropy typically signals exploratory behavior to discover new solutions, whereas low entropy indicates confident, exploitative behavior with more consistent action selection.

In Evo-PI, medical MLLMs are optimized as policies under an RLVR trainer. They must balance exploring diverse case patterns with reliably solving individual cases. This motivates using entropy-based stopping: we terminate an iteration when entropy evidences either collapse (i.e., over exploitation) or abnormal increase (i.e., unstable exploration), marking ineffective learning progress. As shown in Figure 3, entropy decreases across all modalities, with low amplitude fluctuations near the end of training that reflect the evolving exploration–exploitation balance under Evo-PI.

3.3.0.3 Ablation on Principles Gains and Iteration Gains

The gains derived from principles and iteration demonstrate how the principles within iterative process exert their influence in Evo-PI. We conducted ablation experiments on the results for Med-R1 when using GRPO as the RLVR algorithm.

We find that applying the principles yields an average improvement of 13.48%. On top of this, introducing iteration provides an additional performance boost of around 3.7%. Detailed ablations are provided in Appendix 7.9.

3.4 Qualitative Studies↩︎

We conducted tests using Evo-PI across all modalities and observed marked improvements in reasoning quality: post-training traces become more coherent, clinically grounded, and logically ordered. We attribute the performance gains primarily to this enhancement in stepwise reasoning rather than superficial answer heuristics. A representative CT case is shown below:

From this example, the Evo-PI enhanced answer exhibits stronger logical coherence and makes fuller use of CT-specific knowledge than the original Med R1 output, leading to the correct solution. Additional case studies are provided in Appendix 7.10.

4 Related Work↩︎

4.0.0.1 General MLLMs and Medical MLLMs

Recent years have seen the emergence of large-scale MLLMs, such as GPT-4o [2] and the Gemini series [1]. These models acquire strong cross-modal reasoning by leveraging vast data collections, but often struggle to exhibit reliable, clinically grounded reasoning in specialized medical tasks. Medical MLLMs address this gap, typically through SFT on expert-annotated datasets [4], [25]. While effective, such approaches incur substantial annotation costs, creating a major bottleneck. To mitigate data scarcity, recent works have explored RL for post-training alignment [5], [6], [8]. However, these methods typically treat medical knowledge as static supervision signals (e.g., datasets or fixed reward heuristics), rather than as explicit, evolving principles that actively guide the learning process. As a result, they largely overlook structured medical knowledge during alignment, despite the demonstrated effectiveness of knowledge-aware judging in general LLM settings [26]. This gap highlights the need to rethink how medical expertise is integrated into the alignment process.

4.0.0.2 Reinforcement Learning Algorithms for Post-Training

State-of-the-art alignment commonly employs RL algorithms such as PPO [27] and GRPO [9], with subsequent variants like GSPO [15] addressing challenges including training instability and entropy collapse [22]. A common limitation across these techniques is their reliance on reward functions with fixed, pre-defined strategies. Such rigidity leaves models susceptible to reward hacking [12], particularly in knowledge-intensive domains like medicine, where they may exploit surface-level heuristics of the reward scheme rather than acquiring deeper, clinically aligned reasoning skills. These limitations motivate our central hypothesis: robust medical reasoning requires reward mechanisms that are dynamic, knowledge-driven, and grounded in explicitly articulated principles that co-evolve with the model during training.

5 Conclusion↩︎

We present Evo-PI, a principle-guided framework for medical VQA that treats reasoning principles as evolvable supervision signals. By combining principle-guided reinforcement learning with iterative refinement, Evo-PI aligns model reasoning with clinical knowledge. Experiments demonstrate consistent performance gains and robustness across modalities and backbones. Evolving supervision offers a promising direction for reliable, interpretable, and expert-aligned reasoning in MLLMs.

6 Acknowledgments↩︎

This research was supported in part by the Young Scientists Fund of the National Natural Science Foundation of China (Grant No. 32500997, S. Li), and in part by Beijing Renyixun Health Technology Co., Ltd. This research is also partly supported by the Shenzhen Science and Technology Program No. SYSPG20241211173609009, and National Science and Technology Council (NSTC), Taiwan (Grants: NSTC-114-2222-E-A49-004, and NSTC-114-2639-E-A49-001-ASP).

Limitations↩︎

Despite its effectiveness, the proposed Evo-PI framework is subject to several limitations that suggest important directions for future research.

First, Evo-PI relies on the iterative generation and refinement of guiding principles, whose quality is inherently influenced by the reasoning capacity and domain knowledge of the underlying judge model. Although we observe stable improvements across different backbones and reinforcement learning algorithms, suboptimal or overly generic principles may still limit performance gains in highly specialized or rare medical scenarios. Improving the controllability and specialization of principle induction remains an open challenge.

Second, our framework introduces additional computational overhead due to the principle evolution and reward construction process. While Evo-PI does not require modifying backbone model architectures, the iterative interaction among generation, judging, and reinforcement learning increases training cost compared to standard supervised fine-tuning. Exploring more efficient principle updating strategies or lightweight judge models would be beneficial for large-scale or resource-constrained deployments.

Third, our empirical evaluation focuses on medical VQA tasks with well-defined visual and textual supervision. Although this setting provides a rigorous testbed for principled reasoning under high-stakes conditions, the generalizability of Evo-PI to other multimodal reasoning tasks (e.g., longitudinal clinical decision-making or non-medical domains) has not yet been systematically studied. Extending the framework to broader reasoning scenarios and assessing its domain-agnostic properties constitute promising directions for future work.

Importantly, Evo-PI does not require perfectly accurate judges; principles are evaluated in a relative and iterative manner, which empirically provides robustness to moderate judging noise.

7 Appendix↩︎

7.1 The Guideline of Appendix↩︎

The appendix is organized as follows:

• In section  7.2, we introduce Ethics Statement of our paper.

• In section  7.3, we introduce Reproducibility Statement of our paper.

• In section  7.4, we introduce Potential Risks of our paper.

• In section  7.5, we introduce how we use the LLM in our work as required.

• In section  7.6, we introduce how we generate and scaling principles for each sub-modality of medical questions.

• In section  7.7, we introduce how these principles are scaling and refined.

• In section  7.8, we introduce the post-training prompt to control the rollout process.

• In section  7.9, we further introduce some ablation study about the impact of the iterations and the principles.

• In section  7.10, we introduce the case study across different datasets.

7.2 Ethics Statement↩︎

All datasets used in this research are publicly available and were sourced from previous studies that have undergone appropriate ethical review. Our work did not involve the collection of any new data from human subjects. We have adhered to all data usage agreements and licenses associated with these pre-existing datasets.

7.3 Reproducibility Statement↩︎

We are committed to making our research reproducible. All datasets used in this study are publicly available, and we provide detailed descriptions and sources in our experimental setup section.

7.4 Potential Risks↩︎

We do not identify significant risks associated with our approach, as it relies solely on open-source data and models and follows standard research practices.

7.5 Usage of LLM↩︎

This paper primarily employs large language models (LLMs) to refine the overall quality of writing, with a particular focus on eliminating incorrect expressions, minimizing grammatical errors, and enhancing clarity, coherence, and readability to ensure the text meets standards of ACL.

7.6 Prompts for Generating and Scaling Raw Principles↩︎

In this section, we demonstrate how we generate and update principles. Using the principles corresponding to X-ray cases as an example, Evo-PI utilizes five cases from the training set and then instructs the knowledgeable LLM to complete the generation of the principles.

When principles are updated, Evo-PI will provide the previously used principles and streamline and scale them.

list some principles for these key medical reasoning tasks "X-Ray"

Here are the sample questions and answers:
    {
        "image": "Images/RadImageNet/bladder_pathology/abd132420.png",
        "problem": "What is the abnormality present in the image? A)Spinal cord injury B)Ovarian cyst C)Bladder pathology D)Liver cirrhosis",
        "solution": "<answer> C </answer>"
    },
    {
        "image": "Images/RadImageNet/normal/abd-normal028416.png",
        "problem": "Is there any deviation or anomaly observed in this image? A)No B)Yes.",
        "solution": "<answer> A </answer>"
    },
    {
        "image": "Images/RadImageNet/post_op/abd000807.png",
        "problem": "What type of abnormality is present in this image? A)Foreign body reaction B)Post-operative changes C)Infection D)Fracture site healing",
        "solution": "<answer> B </answer>"
    },
    {
        "image": "Images/RadImageNet/normal/abd-normal056990.png",
        "problem": "Is anything out of the ordinary evident in this image? A)No B)Yes.",
        "solution": "<answer> A </answer>"
    },
    {
        "image": "Images/RadImageNet/interstitial_lung_disease/lung044245.png",
        "problem": "What type of abnormality is present in this image? A)Interstitial lung disease B)Pulmonary hypertension C)Asthma D)Pleural effusion",
        "solution": "<answer> A </answer>"
    },
and save it as a string list in Python.

principles_prompt = [
    {"role": "system", "content": "You are helpful AI system in medical.\n\n"},
    {"role": "user", "content": f"Here are some key general principles for medical reasoning tasks involving {question_type}:\n\n"},
    {"role": "user", "content": f"{principles}\n\n"},
    {"role": "user", "content": "The principles should keep general and abstract, and as short as possible. If necessary, extend and merge the list of principles with new numbered principles; otherwise keep current principles unchanged. The listed principles should not exceed ten points The merged principles list:"}
]

7.7 Prompt for Generating Raw Principles↩︎

In this section, we will use the principles of X-ray as an example to demonstrate how these principles are specifically updated.

7.8 Prompt for Generating Responses from Medical MLLMs↩︎

For this section, we directly adopted the prompt provided on med-r1, thereby eliminating any sensitivity associated with the prompt.

7.9 Detailed Ablation Study↩︎

In this section, we present the results of ablation experiments using Med-R1 as the backbone MLLM. We observe that the average gain achieved by applying the principles reaches 13.48%. Building upon this foundation, incorporating iteration can further enhance performance by approximately 3.7%. On datasets like DER and US, which are relatively principle-dependent, the gains from starting with good initial principles are the greatest.

7.10 Additional Qualitative Studies↩︎

For this section, we present case studies across various sub-datasets.

References↩︎

---

1. \(^*\)Corresponding author.↩︎

2. https://platform.openai.com/docs/models/gpt-4o-mini↩︎

3. https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct↩︎