Abstract

Vision Language Models (VLMs) have demonstrated remarkable capabilities in multimodal reasoning tasks, yet they still suffer from recurring failures, such as skipping key visual checks, misapplying domain rules, and hallucinating unsupported concepts. Most existing solutions rely on supervised fine-tuning (SFT) and reinforcement learning (RL), which are expensive to iterate and can be brittle under distribution shift. To this end, we propose Multimodal Mistake Notebook Learning (M$^2$Note), a training-free continual evolution framework that externalizes learning into an editable memory. M$^2$Note transforms failed trajectories into compact subject–guidance notes: the subject summarizes the underlying domain and concept, while the guidance provides actionable verification steps that can be reused in future inference. At test time, M$^2$Note retrieves relevant notes via multimodal retrieval-augmented generation (RAG) and appends them to the model context, steering reasoning away from previously observed pitfalls. To stabilize continual evolution, we adopt batch-level post-verification with rollback, which commits notebook edits only if they improve performance on the same batch, reducing noisy updates and preventing regressions. M$^2$Note supports both self-evolving, where the same VLM acts as solver and supervisor, and cross-model evolving, where a stronger supervisor guides a weaker solver, enabling capability transfer without weight updates. Experiments on six multimodal reasoning benchmarks show consistent improvements across domains and backbones, while achieving strong cost and sample efficiency and remaining complementary to Chain-of-Thought (CoT) prompting.

Figure 1: Overview of M^2Note. (Left) The system updates an external mistake notebook from incorrect responses and retrieves task-relevant guidance at inference time to refine reasoning, for instance, limiting-reagent counting for saccharin. (Right) Accuracy gains on different benchmarks (e.g., MMMU [1], MathVista [2], and AI2D [3]) via M²Note, together with cost- and sample-efficiency comparisons. — Figure 1: **Overview of M$^2$Note**. (Left) The system updates an external mistake notebook from incorrect responses and retrieves task-relevant guidance at inference time to refine reasoning, for instance, limiting-reagent counting for saccharin. (Right) Accuracy gains on different benchmarks (e.g., MMMU [1], MathVista [2], and AI2D [3]) via M²Note, together with cost- and sample-efficiency comparisons.

1 Introduction↩︎

Vision Language Models (VLMs) [4]–[8] have become a general interface for multimodal tasks, including STEM reasoning [1], [2], [9], [10], chart and diagram understanding [3], [11], document OCR [12], [13], visual question answering [14]–[16], and video understanding [17]–[19]. Despite strong progress, VLMs in realistic settings still exhibit repeatable failure modes, such as missing key visual evidence or over-relying on superficial cues. Correcting such behaviors efficiently, without sacrificing robustness or requiring costly retraining, remains a practical challenge.

The dominant adaptation paradigm is parameter-based post-training, such as supervised fine-tuning (SFT) [8], [20] and reinforcement learning (RL) [21]–[24]. Although effective, these methods are expensive, slow to iterate, and prone to regression. More importantly, once model weights are updated, test-time behavior becomes fixed, making continual improvement difficult in dynamic environments. This motivates growing interest in training-free adaptation through in-context steering.

Existing training-free approaches mainly fall into two categories: prompt optimization methods [25]–[27] refine global instructions, but often provide advice that is too coarse to target recurring errors across diverse multimodal tasks; memory-based methods [28]–[31] store instance-level experiences for retrieval, but they often lack abstraction, leading to redundant memories and limited generalization beyond superficially similar cases. These limitations are especially pronounced in multimodal settings, where visual inputs are highly diverse and sparse.

To address this, we propose Multimodal Mistake Notebook Learning (M$^2$Note), a tuning-free framework, which extends the learning-from-mistakes paradigm [29], [32] to VLMs. M$^2$Note maintains an external mistake notebook that stores structured subject-guidance pairs and retrieves relevant notes via multimodal embeddings [33], [34] to augment the model context. Given a multimodal query, the VLM first recalls relevant notes and produces an answer. If the answer is incorrect, a reflective supervisor summarizes the failure into a high-level subject and concise actionable guidance, then writes or merges the resulting note into the notebook. In this way, recurring errors are distilled into compact, reusable experience, shifting adaptation from model parameter updates to semantic updates of an external notebook.

Beyond being training-free, M$^2$Note is designed as a closed-loop evolution protocol: generate $\rightarrow$ reflect $\rightarrow$ update $\rightarrow$ verify. Whenever notebook updates are proposed, the system re-evaluates the same batch with the updated notebook and accepts the change only if performance improves; otherwise, it rolls back. This conservative accept-if-improves mechanism stabilizes continual evolution and suppresses noisy updates. The framework supports both self-evolving, where the same VLM acts as solver and supervisor, and cross-model evolving, where a stronger VLM supervises a weaker one to transfer knowledge without any parameter updates.

Experiments on six multimodal reasoning benchmarks show that M$^2$Note consistently improves VLM performance. Compared with RL-based self-evolving methods such as VisPlay [35], EvoLMM [36], and Vision-Zero [37], M$^2$Note achieves strong gains with substantially lower training cost and fewer samples, since it evolves only a compact external notebook rather than model parameters. We introduce the following technical components:

We present M$^2$Note, a multimodal mistake notebook learning framework that improves VLM reasoning by storing and retrieving subject-level guidance in an external memory.
We propose a stable closed-loop evolution protocol with batch-level verification, supporting both self-evolving and cross-model evolving settings while remaining tuning-free.
We deliver consistent gains across six multimodal reasoning benchmarks, with strong cost and sample efficiency compared with SFT- and RL-based methods.

VLMs and Multimodal Reasoning. Multimodal large language models (MLLMs) [5], [6], [38]–[41] extend LLMs with visual perception and have achieved strong performance on multimodal understanding and reasoning tasks. Benchmarks such as MMMU [1], MathVista [2], AI2D [3], ChartQA [11], MMStar [15], and RealworldQA [16] evaluate challenging abilities including STEM reasoning, diagram understanding, and real-world visual reasoning.

Self-Evolving and Training-Free Adaptation. Recent self-evolving methods improve models via closed-loop interaction, often by assigning co-evolving roles (e.g., a challenger and a solver) to the same base model. Representative examples [35], [37], [42]–[44] use self-play to generate increasingly informative training signals and typically internalize the resulting improvements through parameter updates, such as supervised fine-tuning [20], [45] and reinforcement learning [21], [23], [46], [47]. While effective, these methods are costly and can be brittle under distribution shift [48]–[50]. Training-free alternatives instead adapt models through inference-time context, primarily including prompt optimization [25]–[27], [51] and memory-augmented inference [28], [30], [31], [52]–[54]. Within the latter, learning-from-mistakes methods [28], [29], [32], [55] convert failures into reusable vexperience for future inference. Our work extends this paradigm to VLMs by distilling multimodal failures into compact subject-level guidance and updating memory with batch-level verification.

3 Methodology↩︎

3.1 Overview↩︎

We propose Multimodal Mistake Notebook Learning (M$^2$Note), a training-free framework that improves VLMs by maintaining an external mistake notebook. Unlike prior context-optimization methods mainly developed for text-only settings [29], [32], [52], [56], [57], M$^2$Note extends the learning-from-mistakes [29], [32] paradigm to multimodal tasks.

As illustrated in Figure 2, M$^2$Note involves two roles: a Tuning Model ($\pi_{\theta}$), which solves multimodal problems with retrieved notebook guidance, and a Tuner Model ($\pi_{tuner}$), which analyzes the Tuning Model’s errors and writes reusable guidance back into the notebook. The notebook is initialized as an external memory $\mathcal{M}$ that stores structured triplets $(s,g,e)$, where $s$ denotes the subject, summarizing the underlying task and domain, $g$ denotes the corresponding guidance, i.e., concise recommendations associated with that subject, and $e \in \mathbb{R}^d$ represents the embedding produced by a multimodal embedding model for retrieval. Examples of the notes are provided in Appendix 8.

The framework runs in a closed loop with three steps: the Tuning Model retrieves relevant notes from $\mathcal{M}$ and generates baseline responses; the Tuner Model summarizes failed cases into new subject-guidance notes, and decides whether to add or merge them; the system reruns the same batch with the updated memory and accepts the update only if the new memory improves batch-level performance; otherwise, it rolls back.

Figure 2: The M^2Note evolving protocol. M^2Note improves a VLM through a closed loop with two roles: a Tuning Model for solving multimodal queries with retrieved notebook guidance, and a Tuner Model for analyzing mistakes and refining the notebook. The process consists of (i) Multimodal RAG-based Guidance Retrieval, where the Tuning Model retrieves relevant subject-guidance notes from memory to generate responses, and (ii) Batch-level Memory Refinement, where the Tuner Model summarizes failures into new or merged notes, after which the update is verified and accepted only if it improves batch-level performance. — Figure 2: **The M$^2$Note evolving protocol.** M$^2$Note improves a VLM through a closed loop with two roles: a Tuning Model for solving multimodal queries with retrieved notebook guidance, and a Tuner Model for analyzing mistakes and refining the notebook. The process consists of (i) **Multimodal RAG-based Guidance Retrieval**, where the Tuning Model retrieves relevant subject-guidance notes from memory to generate responses, and (ii) **Batch-level Memory Refinement**, where the Tuner Model summarizes failures into new or merged notes, after which the update is verified and accepted only if it improves batch-level performance.

3.2 Problem Formulation↩︎

M$^2$Note can be formulated as a context optimization problem: instead of updating the parameters of $\pi_{\theta}$, we refine only the external memory to maximize the expected reward, ensuring that the retrieved guidance provides effective assistance. Formally, let $\mathcal{D}$ denote a data distribution over multimodal queries $(q,i)$, optionally paired with $y$ (when the training split is available), where $q$ denotes the text question, $i$ the associated image, and $y$ the optional ground-truth answer when available. We seek an optimal memory $\mathcal{M}^*$: \[\begin{align} &\mathcal{M}^{*} = \arg\max_{\mathcal{M}} \; \mathbb{E}_{(q,i)\sim\mathcal{D}} \Big[ R\big(\pi_{\theta}(z)\big) \Big],\\ &\text{where}~z = (q,i) \oplus \mathrm{Ret}((q,i),\mathcal{M}), \end{align}\] where $\oplus$ is context concatenation, and $\mathrm{Ret}(\cdot,\cdot)$ represents retrieving guidance from the notebook. We consider two instantiations of the reward function $R(\cdot)$: Supervised setting. If ground-truth labels $y$ are available, the model response is first parsed into a final prediction, and $R(\cdot)$ is instantiated as a binary reward by comparing the parsed prediction against the ground truth, i.e., it returns $1$ if they match and $0$ otherwise; Label-free setting. If labels are unavailable, $R(\cdot)$ adopts an LLM-as-a-Judge [58], which evaluates whether the response is correct given the input (and task instruction if available), and outputs a binary judgment as reward. In both cases, the optimization process is delegated to $\pi_{tuner}$, which identifies failures, summarizes guidance, and updates the memory.

3.3 The M$^2$Note Evolution Protocol↩︎

As illustrated in Figure 2, M$^2$Note follows a closed-loop protocol that iteratively retrieves guidance, updates memory from mistakes, and verifies whether the update should be kept.

Multimodal RAG-based Guidance Retrieval. The Tuning Model $\pi_{\theta}$ uses the current memory $\mathcal{M}_i$ in a multimodal RAG manner [59], [60]. For the $i$-th iteration, given an input batch $\mathcal{B}_i=\{(q_k,t_k)\}_{k=1}^N$, each query $(q_k,t_k)$ is fed into a multimodal embedding model $\mathcal{E}$ to obtain an embedding $e_k = \mathcal{E}(q_k,i_k) \in \mathbb{R}^d$. The system then computes the cosine similarity between $e_k$ and each note embedding in memory, ranks all notes by cosine similarity, and returns the top-$K$ results (the following shows $K=1$): \[\begin{align} (s_k,g_k) &= \arg\max_{(s,g)} \left\{ \frac{\langle e_k, e\rangle}{\lVert e_k\rVert\,\lVert e\rVert} \;\middle|\; (s,g,e)\in \mathcal{M}_i,\right. \\ &\qquad\qquad\left. \frac{\langle e_k, e\rangle}{\lVert e_k\rVert\,\lVert e\rVert} > \alpha \right\} \end{align}\] where $\langle e_k, e\rangle$ denotes the inner product; $\lVert\cdot\rVert$ denotes the $\ell_2$ normalization; and $\alpha$ controls the relevance of the retrieved content.

The retrieved notes are concatenated as advisory context to the VLM input, i.e., $z_k=(q_k, i_k) \oplus (s_k, g_k)$. In addition, the Tuning Model is instructed to incorporate the extra context critically rather than follow it blindly, which helps to reduce hallucinations (Appendix 7.2, Table 5 and Figure 6). The initial response, denoted as $\pi_{\theta}(z_k)$, is fed into the reward function $R(\cdot)$. Aggregating and partitioning the responses within the batch yields two mutually exclusive sets: correct responses are collected into $\mathcal{R}_i=\{k~|~R(\pi_{\theta}(z_k))=1\}$, while failed trajectories form $\tilde{\mathcal{R}}_i=\{k~|~R(\pi_{\theta}(z_k))=0\}$. The batch accuracy is then computed as the performance baseline for the current iteration.

Batch-Level Memory Refinement. Given the failure set $\tilde{\mathcal{R}}_i$, the Tuner Model $\pi_{\text{tuner}}$ summarizes reusable guidance and proposes notebook updates. An update is accepted only if it improves batch-level performance; otherwise, it is rolled back.

(1) Subject Extraction $\sigma$. For each failed sample index $k \in \tilde{\mathcal{R}}_i$, the system infers a subject that summarizes the underlying concept of the query. Concretely, as shown below, we apply a prompted operator $\sigma(\cdot)$ implemented by $\pi_{\text{tuner}}$: \[s'_k = \pi_{\text{tuner}}(\sigma(q_k, i_k)), \quad k \in \tilde{\mathcal{R}}_i.\] The subject is expected to be domain-aware (e.g., math, chemistry, or diagram understanding), specific enough to capture recurring error patterns, but not tied to instance-specific details (e.g., exact entities or numbers).

(2) Guidance Synthesis $\phi$. Given a failed trajectory, $\pi_{\text{tuner}}$ further generates a guidance that is directly actionable at inference time. The guidance is distilled from the failed response $\pi_{\theta}(z_k)$ and the available feedback, abstracting the cause of failure into verification-oriented steps: \[g'_k = \pi_{\text{tuner}}(\phi((q_k, i_k), \pi_\theta(z_k))), ~ k \in \tilde{\mathcal{R}}_i,\] where $\phi(\cdot)$ is a prompted “mistake-to-guidance” operator. In practice, $g'_k$ is constrained to be concise, structured, and oriented toward checks and invariants (e.g., “use the artist’s name to narrow down the search space” in Figure 2).

(3) Subject Merging and Memory Update $\mu$. To avoid fragmentation, M$^2$Note optionally merges new notes with existing ones. For each candidate note $(s'_k,g'_k)$, the tuner predicts an edit action: \[a_k = \pi_{\text{tuner}}(\mu(s'_k, s_k)) \in \{\texttt{add},\texttt{merge}\}.\] If $a_k=\texttt{Merge}$, $\pi_{\text{tuner}}$ synthesizes the merged subject $s_k^{*}$ and the merged guidance $g_k^{*}$; otherwise, they are set to $s'_k$ and $g'_k$, respectively. For each newly created or merged note, we compute its embedding as $e_k^*=\mathcal{E}(s_k^*\oplus g_k^*)$. The updated memory $\mathcal{M}'_{i}$ is obtained by incorporating all entries $\{(s_k^*, g_k^*, e_k^*)\}_{k=1}^N$ derived from this batch of data.

(4) Batch-Level Post-Verification. After proposing $\mathcal{M}_{i}'$, we rerun the same batch using the updated memory. The batch-level accuracy before and after the update is computed by the formula: \[\text{Acc}(\mathcal{B}_i|\mathcal{M})=\frac{1}{N}\sum_{k=1}^{N} R(\pi_{\theta}(z_k|\mathcal{M})).\] We accept the update if it improves performance: \[\mathcal{M}_{i+1}= \begin{cases} \mathcal{M}'_{i}, & \text{if } \text{Acc}(\mathcal{B}_i|\mathcal{M}'_i)\ge \text{Acc}(\mathcal{B}_i|\mathcal{M}_i),\\ \mathcal{M}_{i}, & \text{otherwise (rollback)}. \end{cases}\] This “accept-if-improves” rule enforces monotonic non-degradation at the batch level, making notebook evolution robust to occasional low-quality reflections. It also acts as an implicit regularizer that curbs uncontrolled notebook growth, since only useful notes survive. The pseudo-code of the overall procedure is shown in Algorithm 1, and all prompt templates are provided in Appendix 7.

Figure 3: M^2Note evolving protocol — Figure 3: M$^2$Note evolving protocol

4 Experiments↩︎

Experimental Settings. We adopt two experimental settings: Supervised manner. We hold out the test set for evaluation, and use data sampled from the training split to update and save the external mistake notebook, which is then used during evaluation. Test-time scaling (TTS). We directly evaluate on the test set while continuously updating the external memory online, allowing the model to improve progressively as testing proceeds.

Evaluation Datasets and Metrics. We evaluate on six benchmarks covering three vision language domains: STEM/Math (MMMU$_{val}$ [1], MathVista [2]), General VQA (MMStar [15], RealworldQA [16]), and Document OCR (AI2D [3], ChartQA [11]). MMMU and MathVista emphasize multi-discipline reasoning and mathematical problem solving; MMStar and RealworldQA focus on open-world visual understanding; AI2D and ChartQA evaluate diagram/chart comprehension and text-grounded reasoning. Following prior work, we report accuracy (%) as the primary evaluation metric for all datasets. Among them, MMMU and MathVista are trained in a supervised manner, while the other benchmarks are evaluated using test-time scaling (TTS); please refer to Appendix 9.2 for details.

Table 1: **Main results.** We report accuracy and average score, together with the estimated training/API cost, the average number of entries in the memory (**Mem**), and the average number of tokens in the guidance (**Len**). $\ddagger$ denotes results reproduced by us with a specific answer-parsing strategy, and the best performance is in **bold.**
Method	Cost	Mem	Len	STEM Puzzle		General VQA		Document OCR		Avg.
5-6(lr)7-8(lr)9-10				MMMU$_{val}$	MathVista	MMStar	RealworldQA	AI2D	ChartQA

Vision-Zero ([37])	Basically	–	–	58.8	72.6	65.2	68.5	84.5	86.3	72.65
EvoLMM ([36])		–	–	52.0	70.5	–	–	83.4	86.7	–
iReasoner ([61])		–	–	52.4	69.7	–	–	83.9	85.8	–
VisPlay ([35])		–	–	54.9	68.2	65.1	69.0	–	86.2	–

Vanilla$^\dagger$	–	–	–	67.1	73.2	62.1	73.2	80.8	89.6	74.3
w/ DPE ([44])	$>\$500$	–	–	69.1( $\uparrow$2.0 )	76.2( $\uparrow$3.0 )	62.1	72.1( $\downarrow$1.1 )	–	84.8( $\downarrow$4.8 )	–
w/ M$^2$Note	$\approx\$3$	50	261	68.4( $\uparrow$1.3 )	77.3( $\uparrow$4.1 )	63.9( $\uparrow$1.8 )	75.4( $\uparrow$2.2 )	82.1( $\uparrow$1.3 )	89.9( $\uparrow$0.3 )	76.2( $\uparrow$1.9 )
w/ CoT	–	–	–	68.8	82.9	62.9	78.0	81.3	91.7	77.6
w/ CoT+M$^2$Note	$\approx\$4$	32	315	69.4( $\uparrow$0.6 )	83.9( $\uparrow$1.0 )	64.9( $\uparrow$2.0 )	78.8( $\uparrow$0.8 )	81.7( $\uparrow$0.4 )	91.7	78.4( $\uparrow$0.8 )

Vanilla$^\dagger$	–	–	–	77.9	78.0	68.3	78.2	85.9	88.5	79.5
w/ M$^2$Note	$\approx\$7$	28	462	79.7( $\uparrow$1.8 )	80.1( $\uparrow$2.1 )	69.4( $\uparrow$1.1 )	79.7( $\uparrow$1.5 )	86.5( $\uparrow$0.6 )	89.5( $\uparrow$1.0 )	80.8( $\uparrow$1.3 )

Vanilla$^\dagger$	–	–	–	77.1	77.2	66.5	80.8	85.6	85.6	78.8
w/ M$^2$Note	$\approx\$7$	28	462	77.9( $\uparrow$0.8 )	78.0( $\uparrow$0.8 )	67.8( $\uparrow$1.3 )	81.8( $\uparrow$1.0 )	86.1( $\uparrow$0.5 )	85.8( $\uparrow$0.2 )	79.6( $\uparrow$0.8 )

Vanilla$^\dagger$	–	–	–	67.1	73.2	62.1	73.2	80.8	89.6	74.3
w/ M$^2$Note	$\approx\$5$	43	445	68.9( $\uparrow$1.8 )	78.3( $\uparrow$5.1 )	64.0( $\uparrow$1.9 )	77.1( $\uparrow$3.9 )	81.0( $\uparrow$0.2 )	90.1( $\uparrow$0.5 )	76.6( $\uparrow$2.3 )

2pt

Implementation Details. Our M$^2$Note does not require training model parameters; it only updates a mistake notebook stored as external memory in the JSONL format. We evaluate three VLMs: the open-source Qwen3-VL-8B-Instruct [6], the closed-source Qwen3-VL-Plus [6] and GPT-5.4 [5]. For multimodal RAG, we use Qwen3-VL-Embedding [33] as the embedding model. In addition to these models, all ablation variants are also implemented through API requests. Unless otherwise specified, we set the RAG top-$K$ to $K=1$ and the RAG threshold to $\alpha=0.4$. The batch size for both training and inference is $16$. For supervised experiments on MMMU and MathVista, we run $10$ and $20$ training steps, respectively. More details can be found in Appendix 9.1.

4.1 Main Results↩︎

Table 1 reports the main results of M$^2$Note on six multimodal reasoning benchmarks: Consistent gains across domains and backbones. M$^2$Note yields consistent improvements on different domains, and the gains hold for both an open-source backbone and a stronger proprietary backbone. Cost- and sample-efficiency. Compared to RL-based self-evolving approaches, M$^2$Note achieves competitive or even better performance with dramatically lower cost via pure API requests. Under the same backbone model (Qwen3-VL-8B-Instruct), M$^2$Note matches or slightly outperforms DPE [44] while avoiding RL-style training overhead. Multiple evolving modes. M$^2$Note supports both self-evolving (single model acts as both the tuning and tuner models) and cross-model evolving (a stronger tuner model supervises a weaker tuning model), enabling capability transfer without any weight updates. CoT compatibility. M$^2$Note can work together with Chain-of-Thought (CoT) prompting [51], [62], leading to further gains over either CoT or M$^2$Note alone. Compact notebook with interpretable scaling trends. We observe that stronger backbones make fewer mistakes and require fewer notebook entries (Mem), while a stronger tuner (Qwen3-VL-Plus) tends to produce more detailed guidance, resulting in longer length (Len).

4.2 Ablation Study↩︎

Sensitivity Analysis of Hyperparameters. Results in Figure 4 (a) and 4 (b) reveal two trends: M$^2$Note is relatively insensitive to batch size. Across a wide range (1–32), test accuracy remains stable at around 76%–77.4%, suggesting that the closed-loop update does not depend heavily on a specific batch size. Moderate batch sizes (e.g., 8 or 16) perform best, likely because they support more robust batch-level memory refinement and make the verify stage more reliable. Meanwhile, notebook size does not increase monotonically with batch size (about 41–61 entries), indicating that subject-level merging prevents memory from scaling directly with the batch. As training progresses, the notebook gradually expands (15 to 67 entries), while accuracy improves from 73.2% to 76.6%. Although there is some fluctuation around step 15, the overall upward trend suggests that continued evolution accumulates reusable guidance and improves test generalization.

Multimodal RAG Settings. Figure 4 (c) studies the effect of the retrieval threshold $\alpha$ and the retrieval top-$K$ on MathVista (Qwen3-VL-8B-Instruct). Increasing $\alpha$ makes the system more conservative, reducing the number of queries that trigger note insertion; as fewer queries retrieve any guidance, batch performance remains unchanged before and after the update, thus shrinking the final notebook size. In terms of accuracy, a moderate threshold performs best: $\alpha{=}0.4$ achieves the highest accuracy, while overly strict retrieval (e.g., $\alpha{=}0.8$) disables the notebook and degrades performance. We also observe that the system is robust to the retrieval top-$K$, with performance varying only slightly. Overall, we prefer a smaller value ($K{=}1$ by default) to reduce the risk of hallucinations.

Effectiveness of Batch-Level Post-Verification. Table 2 examines the batch-level post-verification, namely accept-if-improves. For Qwen3-VL-8B-Instruct, enabling post verification yields the best overall results, notably improving MathVista (73.2 $\rightarrow$ 77.3) and MMMU$_{val}$ (67.1 $\rightarrow$ 68.4), showing that filtering noisy updates stabilizes evolution and makes the accumulated notes more reliable. Without post verification, performance becomes less consistent (e.g., a drop on MMMU$_{val}$), indicating that blindly committing proposed notes may introduce harmful or noisy guidance. For the stronger Qwen3-VL-Plus backbone, the effect is smaller but still positive on average, suggesting that as the base model makes fewer mistakes, the notebook receives fewer high-impact updates, yet conservative verification remains useful for preventing degradation.

Table 2: **Effectiveness of batch-level post-verification.** We report results on three benchmarks across two backbones. Best results are marked in **bold**.
Method	MathVista	MMMU$_{val}$	MMStar

Vanilla	73.2	67.1	62.1
w/o Batch-Level Post-Verification	76.2	65.7	62.7
w/ Batch-Level Post-Verification	77.3	68.4	63.9

Vanilla	78.0	77.9	68.3
w/o Batch-Level Post-Verification	80.4	77.0	69.6
w/ Batch-Level Post-Verification	80.1	79.7	69.4

Ablation on retrieval embeddings. We study how the embedding model and input modality affect M$^2$Note as shown in Table 3. Using a text-only embedding model (Qwen3-Embedding) can improve MathVista, but tends to generalize worse on other benchmarks such as MMStar. In contrast, VLM-based embedding models (Qwen3/2.5-VL-Embedding) that encode both image and text consistently achieve better performance, suggesting that multimodal indexing better captures the visual cues needed to retrieve relevant notes. This is further supported by the modality ablation: using a single modality, either image or text, is inferior to using both, indicating that visual and textual signals are complementary for robust note retrieval.

Table 3: **Impact of different embedding models and retrieval modes.** We report results using *Qwen3-VL-Instruct-8B*. Best results are marked in **bold**.
Embedding Model	Image	Text	MathVista	MMMU$_{val}$	MMStar

Vanilla	-	-	73.2	67.1	62.1
Qwen3-Embedding ([63])	-		75.5	66.6	59.1
Qwen2.5-VL-Embedding ([33])			77.3	66.2	63.2
Qwen3-VL-Embedding ([33])			76.3	66.0	63.0
Qwen3-VL-Embedding ([33])			77.1	66.2	62.3
Qwen3-VL-Embedding ([33])			77.3	68.4	63.9

2pt

Figure 5: Qualitative results. For a STEM (biochemistry) question and a document understanding question, the baseline tends to rely on superficial cues and makes incorrect selections, while M^2Note retrieves subject-specific guidance from the external mistake notebook and corrects the reasoning by enforcing key structural checks (e.g., carbonyl position for aldose/ketose; option-to-cluster mapping for Belbin roles). — Figure 5: **Qualitative results.** For a STEM (biochemistry) question and a document understanding question, the baseline tends to rely on superficial cues and makes incorrect selections, while M$^2$Note retrieves subject-specific guidance from the external mistake notebook and corrects the reasoning by enforcing key structural checks (e.g., carbonyl position for aldose/ketose; option-to-cluster mapping for Belbin roles).

Notebook transfer across models. We evaluate whether an evolved mistake notebook can transfer across different VLMs (Table 4). Specifically, we first self-evolve a notebook with one backbone (e.g., Qwen3-VL-8B-Instruct or Qwen3-VL-Plus), then freeze it and use it as external guidance for a different tuning model, without further evolution or parameter updates. Overall, the transferred notebooks remain effective, indicating that M$^2$Note captures partially model-agnostic and reusable guidance. However, cross-model transfer is not always risk-free, and mismatched guidance can hurt performance in some cases. Moreover, compared with self-evolved notebooks in Table 1, transferred notebooks generally yield smaller gains, suggesting that the strongest improvements come from adapting to each model’s own failure patterns.

Table 4: **Notebook transfer across different VLMs.** We report results using three different VLMs.
Tuning Model	Mem	Len	MathVista	MMMU$_{val}$	MMStar

Qwen3-VL-Plus ([6])	–	–	78.0	77.9	68.3
w/ Specific Notebook	61	261	78.7( $\uparrow$0.7 )	76.8( $\downarrow$1.1 )	68.6( $\uparrow$0.3 )
Qwen3-VL-32B-Instruct ([6])	–	–	81.1	73.1	67.8
w/ Specific Notebook	61	261	81.4( $\uparrow$0.4 )	74.1( $\uparrow$1.0 )	68.1( $\uparrow$0.3 )

Qwen3-VL-8B-Instruct ([6])	–	–	73.2	67.1	62.1
w/ Specific Notebook	33	469	74.7( $\uparrow$1.5 )	67.4( $\uparrow$0.3 )	62.7( $\uparrow$0.6 )

2pt

4.3 Qualitative Results↩︎

Fig. 5 presents two representative cases illustrating how M$^2$Note improves multimodal reasoning. In the STEM (biochemistry) example on the left, the baseline gives a plausible explanation but misidentifies key structural cues, leading to an incorrect classification. With M$^2$Note, the model retrieves subject-level guidance from the external mistake notebook (e.g., locate the carbonyl and verify its position), which enforces essential structural checks and corrects the conclusion. In the diagram understanding example on the right, the baseline tends to pick an option that sounds reasonable without strictly aligning choices to the diagram’s defined taxonomy, making it vulnerable to introducing hallucinations. M$^2$Note retrieves guidance that explicitly requires option-to-cluster mapping and eliminates elements not present in the reference model, yielding the correct answer. More visualizations, including case analyses and mistake-notebook visualizations, are provided in Appendix 8 (Figure 6).

5 Conclusion↩︎

We introduce M$^2$Note, a training-free framework for continually improving VLMs through multimodal mistake notebook learning. Instead of updating model weights, M$^2$Note distills model failures into reusable subject-specific guidance and retrieves relevant notes at inference time via multimodal RAG, helping the model verify its reasoning and avoid repeated errors. A key mechanism is batch-level accept-if-improves verification, which filters noisy notebook updates and stabilizes continual evolution by retaining only beneficial changes. Across six benchmarks covering STEM/math reasoning, general VQA, and document understanding, M$^2$Note yields consistent gains in both self-evolving and cross-model evolving settings, while also combining effectively with CoT prompting. Overall, these results suggest that reusing verified failure cases provides a practical and effective way to enhance VLM robustness at deployment.

6 Limitation↩︎

M$^2$Note evolves in a way that resembles how humans learn from experience: instead of rewriting one’s “brain” (model weights), it repeatedly summarizes failures into reusable rules and consults them when facing similar situations. This mechanism is most effective when mistakes exhibit recurring structure, so that a distilled note can be reliably reused. As a result, M$^2$Note tends to work best in relatively narrow domains (e.g., math reasoning), where tasks share stable abstractions and retrieved guidance is more likely to transfer. In broad domains with long-tail visual diversity, relevant notes can be harder to retrieve, and mismatched guidance may introduce misleading context or amplify hallucinations, making continual evolution less stable. These limitations suggest two directions: building more generalizable notebooks through stronger abstraction and a more reliable post-update verification strategy, so that notes transfer beyond near-duplicate failures and harmful guidance is less likely to be retrieved; and enriching notes beyond text (e.g., reference images, tool-use traces, structured checklists) to provide more grounded and actionable guidance.

7 Prompts Used in the M$^2$Note Framework↩︎

Figure 6: Qualitative examples of applicability judgment for retrieved guidance. The model explicitly decides whether the injected subject-guidance pair is applicable: the first two cases are Applicable and lead to correct answers, while the last case is Irrelevant and is rejected. — Figure 6: **Qualitative examples of applicability judgment for retrieved guidance**. The model explicitly decides whether the injected subject-guidance pair is applicable: the first two cases are *Applicable* and lead to correct answers, while the last case is *Irrelevant* and is rejected.

7.1 Tuner Model Prompts↩︎

Guidance Extraction You are a Cognitive Strategy Expert. Your goal is to induce generalizable rules from specific model errors, not just correct them.

$~$

Subject: {subject}

Error Data: {error_context}

$~$

Task: Analyze the underlying logical flaws in the Error Data and formulate a reusable problem-solving heuristic.

$~$

Constraints (Total $\leq$ 350 words):

1. Abstracted Error Pattern ($\leq$ 60 words)

- Describe the type of situation where this error occurs (strip specific numbers/names).

- Do NOT simply rewrite the original questions.

2. Root Cause Analysis ($\leq$ 50 words)

- Identify the cognitive gap (e.g., confusing correlation with causation, ignoring boundary conditions).

3. Generalizable Strategy (CORE) ($\leq$ 120 words)

- Provide a step-by-step heuristic or checklist applicable to ANY similar future problem.

- Use “If [condition], Then [action]” format where possible.

- Must work beyond the provided examples.

4. Anti-Patterns & Boundary ($\leq$ 80 words)

- List 1-2 specific scenarios where this strategy should NOT be applied.

- List 1-2 common misinterpretations of this guidance.

$~$

Output Style:

- High-level, structured, knowledge-focused, and reusable for similar future questions.

- Avoid referencing specific details from the provided Error Data unless illustrating a pattern.

Subject Classification You are an expert in categorizing questions into precise, high-relevance subjects for Retrieval-Augmented Generation (RAG).

$~$

Your goal is to assign each question a subject label (about 10-50 words) that:

- Maximizes retrieval relevance by precisely describing the problem type and solution method.

- Groups only genuinely similar questions together (same domain AND same approach).

- Avoids over-broad categories that would match unrelated problems.

- Reuses the same subject name for closely related questions.

$~$

CRITICAL: The subject must be specific enough to prevent irrelevant retrieval. Include, when applicable:

1. Primary Domain (e.g., Combinatorics, Complex Analysis, Linear Algebra, Physics, Programming, Document Understanding)

2. Problem Type (e.g., counting with constraints, roots of unity products, debugging API parameters, OCR table extraction)

3. Solution Method (e.g., stars and bars, polynomial/root identities, Hensel’s lemma, reproduce-minimize-fix)

$~$

Examples of GOOD subjects (specific):

Complex Analysis: Evaluating products over roots of unity using polynomial evaluation and complex identities

Document Understanding: Extracting tables from scanned PDFs using layout detection + OCR + row/column reconstruction

$~$

Examples of BAD subjects (too broad):

number theory (too broad - could match any modulo problem)

physics (too broad - could match any physics problem)

$~$

Key principle: If a problem spans multiple domains, label it by the primary one.

$~$

Output only the finalized subject label(s).

Guidence Merge Synthesize guidance for subject: {subject}

Existing guidance from related subjects: {existing_guidance}

New guidance: {new_guidance}

$~$

Merge into a single coherent guidance (max 2048 chars) that:

- Combines insights from related subjects with new guidance

- Eliminates redundancy while preserving key information and examples of the mistakes

- Preserves and emphasizes applicability conditions

- clearly state when each method applies

- Focuses on actionable advice

- Maintains consistent style

- Includes warnings about when NOT to apply the guidance to avoid misapplication

$~$

Merged guidance:

Subject Merge The following subjects are related: {subjects}

Provide a single, concise subject name ($\leq$10 words) that best represents all of these. Only output the subject name, nothing else.

7.2 Tuning Model Prompts↩︎

User Prompt Question: {question}

Options: {options}

Please select the correct answer from the options above.

RAG-Enhanced System Prompt The following mistake notes are not necessarily tied to the current question, but you may use them to deepen your analytical approach

$~$

IMPORTANT: Before applying any guidance below, carefully evaluate:

1. Does the current problem match the applicability conditions stated in the guidance?

2. Is the problem type and context similar to the examples in the guidance?

3. If the problem is totally different (e.g., combinatorics vs modulo arithmetic, complex numbers vs number theory), do NOT force-fit the guidance.

4. Only use guidance that is clearly relevant to the current problem.

$~$

Gudience Subject: {subject} - {guidance}

Before solving, review the guidance. State whether it is: applicable, partially applicable, or irrelevant. Use only applicable parts.

Since the guidance introduced via multimodal RAG [59], [60] may not necessarily apply to the current question, in addition to the top-$K$ and threshold settings in the RAG configuration, we further emphasize in the system prompt that the model should judge whether the retrieved content is applicable to the current question. This is crucial for reducing hallucinations. As shown in Fig. 6, several cases are presented where the model’s response explicitly includes this judgment: the first two cases are applicable, while the last one illustrates an inapplicable scenario. Table 5 compares blindly following the guidance (naive) with introducing a judgment mechanism (judgement); the latter effectively improves the model’s problem-solving accuracy.

Table 5: **Effect of applicability judgment for retrieved guidance**. Results on three benchmarks and two backbones. *Naive* uses retrieved guidance directly, while *Judgment* verifies relevance first, consistently improving performance. Best results are in **bold**.
Method	MathVista	MMMU$_{val}$	MMStar

Vanilla	73.2	67.1	62.1
w/ M$^2$Note (Naive)	76.2	67.8	64.1
w/ M$^2$Note (Judgment)	77.3	68.4	63.9

Vanilla	78.0	77.9	68.3
w/ M$^2$Note (Naive)	79.5	76.9	69.0
w/ M$^2$Note (Judgment)	80.1	79.7	69.4

8 More Visualizations↩︎

Below we present several entries from the mistake notebook, obtained by training Qwen3-VL-8B-Instruct as the Tuning Model on MMMU [1]. As shown, the structured guidance is produced using the Tuner Model with the “Guidance Extraction” prompt, while the Note and the Task field are derived using the “Subject Classification” prompt.

tealNote 1: Document Understanding Task Extract structured brainstorming workflow steps from a quadrant diagram (visual category mapping + label extraction).

Abstracted Error Pattern Options that sound reasonable are mistaken as valid steps even when they contradict the diagram’s collaborative workflow (e.g., silent reading, during disagreement).

Root Cause Plausibility is prioritized over fidelity to the visual framework; behavioral/structural constraints are ignored.