1 Introduction↩︎

Vision–language models (VLMs) with reasoning capabilities [1]–[3] have achieved impressive performance on complex image–text tasks, including real-world perception [4], diagram interpretation [5], and mathematical reasoning [6]. Alongside reasoning improvements, reinforcement learning (RL) has been observed to induce emergent behaviors such as self-correction over previous reasoning steps [7], [8], often referred to as an “aha moment” [9]. These behaviors resemble how humans tackle challenging problems, suggesting that self-correction is an important capability for strong and robust reasoning.

However, current RL paradigms do not explicitly teach self-correction. Rewards are provided only at the outcome level, providing no direct signal for learning how to recover from errors. As a result, self-correction behavior arises only implicitly, is difficult to control, and cannot be reliably triggered at inference time [10]. This raises a central question: how can self-correction be learned as a controllable behavior in VLMs using RL?

Prior attempts [11] have explored encouraging self-reflection by prompting and rewarding reflective behavior during RL. While such approaches can amplify reflective tendencies, they still rely on sparse and uncontrolled emergence. Effective self-correction examples remain rare throughout training.

To address this challenge, we make a key observation: although VLMs rarely generate effective self-correction examples on their own, the necessary learning signals already exist in standard RL rollouts. For a given input, correct and incorrect self-generated reasoning trajectories often coexist, and their contrast naturally reveals how reasoning errors can be corrected. By pairing such trajectories, effective self-correction samples can be synthesized explicitly without additional computational overheads.

Based on this insight, we introduce correction-specific rollouts (Octopus), a rollout augmentation framework for learning self-correction in RL. Octopus reuses and recombines rollouts, which not only provides dense, explicit self-correction examples, but also (i) combinatorially increases training samples (creating \(n^2\) from \(n\) rollouts) and (ii) balances positive and negative examples, leading to more stable policy updates. To achieve both strong self-correction and direct reasoning, we further propose a response-masking strategy that decouples their training signals and avoids optimization conflicts. Our main contributions are summarized as follows:

• We identify a key challenge in teaching self-correction via RL: effective self-correction examples are extremely sparse. We show that this challenge can be addressed by exploiting contrastive signals already present in standard RL rollouts through pairing correct and incorrect reasoning trajectories.

• We introduce Octopus, a rollout augmentation framework that constructs dense, explicit self-correction examples in RL. Octopus simultaneously improves sample efficiency via rollout reuse and stabilizes optimization by balancing positive and negative examples.

• We propose a response-masking optimization strategy that avoids conflicting training signals between self-correction and direct reasoning, enabling the model to effectively learn both capabilities.

• Our Octopus-8B model achieves SoTA performance among models of comparable size across 7 benchmarks. It outperforms the base model Qwen3-VL-8B-Instruct by 9.5 average accuracy points, exceeds the official reasoning-enhanced RL version Qwen3-VL-8B-Thinking by 1.2 points, and surpasses the strongest RLVR baseline, GSPO, by 1.0 average accuracy point while requiring only \(0.72\times\) training time per step.

2 Preliminaries↩︎

2.1 Reinforcement Learning with Verifiable Rewards↩︎

Reinforcement Learning with Verifiable Rewards (RLVR) trains language models on tasks whose outcomes can be easily verified, such as math problems and question answering [12]. Recent studies [13], [14] have shown that RLVR effectively enhances the reasoning capabilities of language models.

Group Relative Policy Optimization (GRPO) [15] is a commonly used algorithm for RLVR. It estimates advantages over a set of responses \(\{o_1, \ldots, o_n\}\) generated by the policy \(\pi_\theta\). The GRPO objective is defined as:

\[\begin{align} \mathcal{J}_{\text{GRPO}}(\theta) = \mathbb{E}_{\{o_i\}_{i=1}^{G}}\bigg[ \frac{1}{G}\sum_{i=1}^{G}\frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \min\Big( w_{i,t}&(\theta)\hat{A}_{i,t}, \notag\\ \operatorname{clip}\!\big(w_{i,t}(\theta), 1-\epsilon, 1+\epsilon\big)\hat{A}_{i,t} \Big) &\bigg],\label{eq:grpo} \end{align}\tag{1}\] where \(\hat{A}_{i,t}=\frac{r_i - \operatorname{mean}(\{r_i\}_{i=1}^G)}{\operatorname{std}(\{r_i\}_{i=1}^G)}\) denotes the advantage estimated from the normalized rule-based reward within the rollout group, and \(w_{i,t}\) is the importance sampling ratio, computed as \(\frac{\pi_\theta(o_{i,t} \mid x, o_{i,<t})}{\pi_{\operatorname{old}}(o_{i,t} \mid x, o_{i,<t})}\). The clipping parameter \(\epsilon\) prevents overly aggressive policy updates. However, scaling the number of rollouts introduces an off-policy effect, as rollouts must be partitioned into multiple mini-batches to compute Eq. 1 . In this case, token-level importance weighting can introduce high-variance noise into the gradient estimates. Group Sequence Policy Optimization (GSPO) [16] mitigates this issue by applying a sequence-level importance weight \(s_i(\theta) = \frac{\pi_{\theta}(o_i \mid x)}{\pi_{\operatorname{old}}(o_i \mid x)}\), which prevents training collapse caused by a small number of overly off-policy tokens in long reasoning trajectories, thereby stabilizing RL training. The GSPO objective is as follows: \[\begin{align} \mathcal{J}_{\text{GSPO}}(\theta) = \mathbb{E}_{\{o_i\}_{i=1}^{G}}\bigg[ \frac{1}{G}\sum_{i=1}^{G} \min\Big( s_{i}&(\theta)\hat{A}_{i}, \notag\\ \operatorname{clip}\!\big(s_{i}(\theta), 1-\epsilon&, 1+\epsilon\big)\hat{A}_{i} \Big) \bigg].\label{eq:gspo} \end{align}\tag{2}\]

2.2 Definition of Self-Correction↩︎

Our goal is to improve the reasoning performance of VLMs by strengthening their self-correction ability. Motivated by the observation that reasoning models may spontaneously generate reflective tokens during generation, we focus on single-pass self-correction, where the model revises its reasoning within a single response: \((o_1 \oplus \texttt{<sc>} \oplus o_2) \sim \pi(\cdot \mid x)\), where \(o_1\) and \(o_2\) denote the responses before and after self-correction, and <sc> is a special token that marks the onset of corrective behavior, e.g., an “aha moment” token [7]. Unlike multi-pass self-correction or tool-based approaches that rely on additional prompting or external feedback, this formulation treats self-correction as an intrinsic behavior of a single forward generation.

3 Learning Self-Correction from Paired Rollouts↩︎

3.1 The Challenge: Self-Correction Signals Are Sparse↩︎

Teaching self-correction with RL requires learning signals that explicitly demonstrate how incorrect responses can be revised into correct ones, i.e., rollouts of the form wrong \(\rightarrow\) correct. However, standard RL relies solely on outcome-level rewards and does not explicitly encourage self-correction behavior. As a result, self-correction emerges only rarely and implicitly during training. Recent work attempts to amplify self-correction signals by prompting VLMs to generate self-correction rollouts [11]. However, prompting alone cannot substantially alter model behavior, and effective self-correction remains rare.

We empirically quantify this sparsity. Under standard RL, even with manually appended “Wait”-style aha-moment tokens, only up to 0.3% of samples exhibit effective wrong \(\rightarrow\) correct transitions (Table 1). Prompt-encouraged RL increases this fraction only marginally, to below 1% (Fig. 2). Moreover, the corresponding negative samples correct \(\rightarrow\) wrong are also rare, indicating that the model collapses to simply maintaining the initial response. This extreme sparsity limits learning self-correction via RL.

3.2 Correction-Specific Rollout Augmentation↩︎

A key observation motivating our approach is that effective self-correction signals already exist in standard RL rollouts: for a given input, incorrect and correct responses often coexist within the same rollout group. By pairing them, we can explicitly construct samples that demonstrate effective correction behavior. Based on this, we propose correction–specific rollouts (Octopus) augmentation.

A naive approach would directly pair responses from standard RL rollouts to form self-correction examples. However, these samples lie far outside the base VLM’s distribution, leading to unstable RL training. To avoid this issue, we make the model generate rollouts in an explicit self-correction format (details in § 4.1), \(\{(o_1^i \oplus \texttt{<sc>} \oplus o_2^i)\}_{i=1}^n\), where \(o_1^i\) and \(o_2^i\) denote the responses before and after the self-correction token. Importantly, neither \(o_1^i\) nor \(o_2^i\) is assumed to be correct or wrong. This step serves only to ensure that self-correction-style trajectories are in-distribution.

Given these rollouts, we construct augmented samples by recombining their components: we select \(o_1\) from \(\{o_1^i\}_{i=1}^n\) and \(o_2\) from \(\{o_2^i\}_{i=1}^n\), yielding \(n^2\) paired rollouts in total. These pairs fall into 4 categories: wrong \(\rightarrow\) correct (positive), correct \(\rightarrow\) correct (positive), correct \(\rightarrow\) wrong (negative), and wrong \(\rightarrow\) wrong (negative). Among them, wrong \(\rightarrow\) correct is the most informative, as it directly encodes effective self-correction behavior.

Assuming that \(N\) rollouts are required for each policy update, we keep the \(n\) originally generated rollouts to avoid relying entirely on offline data. We then select an additional \(N - n\) samples from the augmented pool while balancing positive and negative examples. For positive examples, we prioritize wrong \(\rightarrow\) correct, followed by correct \(\rightarrow\) correct pairs when needed. For negative examples, we randomly sample from correct \(\rightarrow\) wrong and wrong \(\rightarrow\) wrong. In practice, we set \(n = 8\) and \(N = 16\), yielding 64 augmented rollouts, from which 16 balanced samples are selected. The complete procedure is summarized in Algorithm 9.

Octopus augmentation offers three key advantages: (i) it produces dense, explicit self-correction examples; (ii) it balances positive and negative samples, stabilizing RL optimization; (iii) since rollout generation is the most costly part of RL training, it substantially improves sample efficiency by reusing existing rollouts.

4 Training Recipe↩︎

Building on the key idea of Octopus, we present our complete training recipe. We begin with a cold-start stage that establishes the self-correction output format (§ 4.1). We then analyze the learning conflict between direct reasoning and self-correction under the RL objective (§ 4.2), and introduce a response-masking strategy to decouple these learning signals (§ 4.3).

4.1 Cold-Start and Data Construction↩︎

A straightforward way to induce the self-correction format \((o_1 \oplus \texttt{<sc>} \oplus o_2)\) is to prompt the model. However, prompting alone often yields \(o_2\) responses that merely continue or partially revise \(o_1\), resulting in incomplete post-correction reasoning. Pairing such \(o_2\) with a different \(o_1\) produces incoherent examples. To avoid this issue, we introduce a cold-start format-learning stage that ensures both \(o_1\) and \(o_2\) contain complete, self-contained reasoning. We consider two sampling strategies for constructing the SFT cold-start dataset: in-distribution sampling and mixed sampling.

In-distribution Sampling. This strategy samples all responses from the policy VLM \(\pi_\theta\) and pairs them to form a self-correction format. For each input, we sample 4 responses. When all responses are correct, we select \(4\text{k}\) instances to construct \(\textcolor{mygreen}{o_1} \oplus \texttt{<sc>} \oplus \textcolor{mygreen}{o_2}\). We use best-of-\(N\) to select the best response as \(o_2\), and randomly choose one of the remaining as \(o_1\), ensuring that \(o_2\) is better than \(o_1\). When both correct and incorrect responses are present, we select \(6\text{k}\) instances to construct \(\textcolor{myred}{o_1} \oplus \texttt{<sc>} \oplus \textcolor{mygreen}{o_2}\).

Mixed Sampling. In mixed sampling, responses before <sc> are sampled from the policy model \(\pi_\theta\), while responses after <sc> are sampled from a stronger model \(\pi_s\). We reuse the same 10k inputs \(x\) and their corresponding \(o_1\) from in-distribution sampling. To obtain higher-quality corrections, we generate \(o_2\) using \(\pi_s\), conditioned on the input \(x\), the ground truth, and \(o_1\): \(o_2 \sim \pi_s(\cdot \mid x, o_1, \mathrm{gt})\).

Setup. We adopt Qwen3-VL-8B-Instruct [13] as \(\pi_\theta\) and its larger variant, Qwen3-VL-30B-A3B-Instruct, as \(\pi_s\). Since the cold-start stage mainly serves to learn the self-correction format and initialize RL training, we select the sampling strategy based on downstream RL performance conducted on ViRL-39k [7].

Results. Fig. 4 shows that in-distribution sampling induces a larger entropy drop than mixed sampling, resulting in overly low initial entropy that limits further improvement during RL. In contrast, mixed sampling maintains an entropy trajectory comparable to GSPO and achieves higher accuracy rewards than both GSPO and in-distribution sampling. These results demonstrate that self-correction format learning without entropy collapse is critical for RL training.

4.2 Conflicts Between Direct Reasoning and Self-Correction in RL Training Objective↩︎

For RL training, the objective is to maximize the final accuracy reward. When responses contain self-correction, this objective can be achieved in two ways [17]: (i) producing the correct answer before <sc>, or (ii) correcting an initially incorrect response after <sc>. We refer to the former as direct reasoning capability, and the latter as self-correction capability. An ideal VLM should possess both abilities. However, when using a conventional binary reward that assigns \(1\) to correct final outcomes and \(0\) otherwise, these two learning signals are entangled. A natural approach to decouple these signals is reward shaping [11], [17]. However, we find that under the single-pass self-correction setting, this strategy leads to reward hacking and unstable training.

Setup. We compare two reward designs: (i) a standard binary reward (\(0/1\)) based on the correctness of \(o_2\), and (ii) a shaped reward proposed in [11], defined as: \[\begin{align} \label{eq:hack95reward} r'(x,o_1, o_2) = \begin{cases} 1.0 & \text{if } r(x,o_1)=0, r(x,o_2)=1 \\ 0.75 & \text{if } r(x,o_1)=1, r(x,o_2)=1 \\ 0.0 & \text{if } r(x,o_1)=0, r(x,o_2)=0 \\ -0.25 & \text{if } r(x,o_1)=1, r(x,o_2)=0 \end{cases}. \end{align}\tag{3}\] We initialize RL training from the cold-start model in § 4.1 and train it using GSPO [16] on the ViRL-39k dataset [7]. Octopus augmentation is disabled to isolate the effect of the RL objective.

Results. Fig. 5 (a) shows that training with a binary reward fails to improve self-correction capability: the accuracy before and after <sc> remains nearly identical throughout training. The overlap of two curves during early iterations further indicates that SFT primarily serves as a format-learning stage and does not teach the model self-correction capability. Fig. 5 (b) shows that reward shaping induces reward hacking after \(\sim\)​200 training steps. As illustrated in Case [case:hack95new] in the Appendix, the model deliberately produces an incorrect first response despite correct reasoning, followed by a trivial correction after <sc>. As reflected by the second-response accuracy and the shaped self-correction reward curves, this behavior induces training instability and ultimately degrades the model’s overall reasoning capability.

4.3 Response-Masking Strategy for Decoupled Learning↩︎

To address the challenge of entangled learning objectives in single-pass RL, we decouple self-correction and direct reasoning through a two-stage training framework based on a response-masking strategy.

Stage I: Learning Self-Correction Only. To learn effective self-correction without reward hacking, we focus only on self-correction in the early stage of RL. Specifically, we treat the pre-correction response \(o_1\) as fixed context: loss is masked for all tokens in \(o_1\) and the policy is updated solely based on the post-correction response \(o_2\). Additionally, we apply a KL loss on \(o_1\), constraining its distribution to the reference model. The resulting optimization objective is: \[\begin{align} \label{eq:stage195objective} \mathcal{J}_{\text{stage I}}=\mathcal{J}_{\text{GSPO}}-\mathbb{E}_{\{o_1^i\}_{i=1}^G}\left[KL(\pi_\theta\mid\mid\pi_{\text{ref}})\right]. \end{align}\tag{4}\] The importance sampling ratio used in GSPO is only calculated based on \(o_2\) as: \[\begin{align} s_i(\theta) = \frac{ \pi_\theta\!\left(o_2 \mid x,\, o_1 \oplus \texttt{<sc>}\right) }{ \pi_{\text{old}}\!\left(o_2 \mid x,\, o_1 \oplus \texttt{<sc>}\right) } . \end{align}\] The rule-based reward is defined as: \[\begin{align} \label{eq:stage195reward} &r_\text{f}(x, o_1, o_2)=\min(r_\text{f}(x,o_1),r_\text{f}(x,o_2))\notag\\ &r_{\text{sc}}(x, o_1, o_2)=0.9\cdot r'(x, o_1, o_2)+0.1\cdot r_\text{f}(x, o_1, o_2), \end{align}\tag{5}\] where \(r_\text{f}\) is the format reward, and \(r'\) is the shaped reward defined in Eq. 3 to strengthen self-correction behavior.

We report the training curves in Fig. 6 using the above objective with Octopus augmentation. Fig. 6 (a) shows that the accuracy gap between \(o_2\) and \(o_1\) (measured on original, non-augmented rollouts) gradually widens over training, indicating that Stage I successfully improves self-correction capability. Fig. 6 (b) shows that the self-correction reward \(r_{\text{sc}}\) with Octopus augmentation remains stable, as augmentation balances positive and negative samples. In contrast, without Octopus augmentation, \(r_{\text{sc}}\) exhibits a sharp increase during training, caused by an increasing dominance of positive samples in rollout groups. Such an imbalanced advantage distribution can weaken the effective learning signal during RL training [7], [18].

Stage II: Co-evolving Reasoning and Correction. The model’s direct reasoning capability determines the starting point for self-correction. In Stage II, we jointly improve both reasoning and self-correction by unmasking \(o_1\) in the objective of Stage I and removing the KL term in Eq. 4 . However, naively unmasking \(o_1\) for all samples can introduce conflicting training signals for samples with effective self-correction signals, e.g., \((\textcolor{myred}{o_1} \oplus \texttt{<sc>} \oplus \textcolor{mygreen}{o_2})\). Because such samples receive positive rewards, backpropagating through \(o_1\) would incorrectly reinforce a wrong direct response. In contrast, samples whose correctness remains unchanged before and after <sc> (i.e., correct \(\rightarrow\) correct or wrong \(\rightarrow\) wrong) mainly provide learning signals for direct reasoning and do not suffer from this conflict. Therefore, we apply selective unmasking: \(o_1\) is unmasked only for samples with consistent correctness before and after <sc>. This design enables jointly training of both capabilities, effectively preventing gradient conflicts induced by mixed optimization.

5 Experiments↩︎

5.1 Setup↩︎

Implementation Details. For SFT cold-start, we apply the mixed sampling strategy to the LLaVA-CoT dataset [19] (§ 4.1), yielding 10k self-correction–formatted samples. For RL training, we perform the proposed RL training on ViRL-39k [7]. To handle off-policy signals introduced by augmentation, we adopt GSPO [16] without the online filter. All training is conducted on 8 NVIDIA H100 GPUs. More implementation details are provided in Appendix 8.

Baselines. We compare our method against 3 categories of baselines. (i) Closed-source VLMs: GPT-4o [20], OpenAI-o1 [1], and Claude-3.7-Sonnet [3]. (ii) Open-source reasoning VLMs around 8B scale: MiMO-VL-7B (SFT and RL) [21], InternVL3.5-8B-RL [22], and Qwen3-VL-8B-Thinking [13]. (iii) RLVR and self-correction baselines: we reproduce GRPO [15], DAPO [23], GSPO [16], and SRPO [11] using the same dataset on Qwen3-VL-8B-Instruct [13].

Benchmarks. We select 2 types of benchmarks to evaluate our model: (i) Math-related: MathVista [6], MathVerse [24], and WeMath [25]. (ii) General Task: HallusionBench [26], MMStar [27], MMMU [28], and CharXiv [29]. Evaluation is conducted using VLMEvalKit [30].

5.2 Main Results↩︎

Performance. Table 2 presents the performance of our Octopus model across 7 benchmarks. Octopus consistently and substantially improves over the base model Qwen3-VL-8B-Instruct, achieving an average accuracy gain of 9.5 points. Moreover, except on MathVerse, Octopus outperforms Qwen3-VL-8B-Thinking, the officially released reasoning-enhanced variant of the same backbone. Among RLVR baselines, we observe that GSPO, which uses sequence-level importance sampling, consistently outperforms GRPO and DAPO, both of which rely on token-level objectives. Building on GSPO, Octopus further improves reasoning performance through explicit self-correction, yielding an additional 1.0 average accuracy gain. Compared to the self-correction baseline SRPO, Octopus augmentation significantly increases effective self-correction reward signals and achieves better performance across all evaluated tasks. Overall, Octopus establishes a new state of the art among open-source VLMs of comparable size by explicitly and efficiently learning self-correction.

Efficiency. Table 2 compares training efficiency and average accuracy of different RLVR methods under varying numbers of rollout samples. We report the average per-step time cost over the first 10 training steps. Across RLVR baselines, increasing the number of rollout samples significantly improves accuracy, but at the cost of roughly doubling the training time per step. In contrast, Octopus leverages rollout augmentation to increase the number of rollouts per input from 8 to 16 without any additional cost. Since rollout generation is one of the most expensive components in RL training, this substantially reduces overall training time. As a result, our method achieves higher accuracy than GSPO with \(n=16\) rollouts while using only \(0.72\times\) the training time. Its training time is comparable to baselines with \(n=8\), with the slight overhead attributed to updating the policy with a larger rollout set. These results demonstrate that by exploiting the self-correction structure, Octopus significantly accelerates RL training while simultaneously improving reasoning performance. We also provide inference latency and output token numbers in Appendix 9.2 to demonstrate the inference efficiency of Octopus further.

5.3 Ablation Study↩︎

To understand the contributions of each component in Octopus, we conduct ablation studies along two dimensions: (i) Octopus augmentation and (ii) training strategy.

Ablation on Octopus augmentation. We first analyze the impact of the most critical component, Octopus augmentation. As shown in the second-to-last row of Table 3, removing augmentation yields performance similar to RLVR baselines trained with fewer rollout samples, leading to a substantial drop in reasoning capability. To dive into whether the gains from Octopus augmentation stem from merely increasing the number of training samples or from enriching effective self-correction signals, we report in the last row of Table 3 the results of a random augmentation, where samples are concatenated at random instead of following the rules in Section 3.2. Random augmentation provides only a slight improvement over no augmentation (67.4 to 68.6) and remains far (\(\downarrow 3.1\)) from Octopus-8B. This gap demonstrates that performance gains are driven by enriching effective self-correction signals, rather than simply increasing rollout size.

Ablation on Training Strategy. To examine the necessity of each component in the training framework, we report in Table 3 the results of the SFT cold-start model and a variant trained with only Stage II during RL. The results show that, compared to the base model, SFT alone yields only marginal improvement, increasing accuracy from 62.2 to 63.4, and remains substantially below the performance of Octopus-8B (71.7). This indicates that within the Octopus framework, SFT primarily serves to learn the self-correction format rather than to deliver generalizable capability gains. Moreover, removing Stage I during RL training leads to a 1.9-point drop in accuracy. This confirms the critical role of Stage I in decoupling self-correction learning and further demonstrates that strengthening self-correction capability is essential for improving overall reasoning performance.

5.4 More Results and Analysis↩︎

Self-correction Performance and Test-Time Scaling. To evaluate whether self-correction is truly learned as a capability that improves reasoning performance, we report the performance of Octopus-8B before and after self-correction, as well as its test-time scaling (TTS) behavior by appending additional <sc> tokens to trigger further correction. As illustrated in Fig. 7 (a), the green curve shows that responses generated after the <sc> token achieve both higher accuracy and better token efficiency than those generated before <sc>. Moreover, although the model is trained to perform only a single round of self-correction, the controllable <sc> token allows us to explicitly trigger additional correction steps at inference time. The blue curve demonstrates that forcing the model to perform further self-correction via TTS progressively improves both accuracy and inference token efficiency, validating that self-correction is a generalizable and scalable capability.

Pass@\(k\) Performance. Pass@\(k\) is a crucial metric for evaluating a model’s potential to solve a question within \(k\) attempts, and is often regarded as a proxy for the model’s reasoning boundary [31], [32]. We report the pass@\(k\) accuracy on MMStar in Fig. 7 (b). As \(k\) increases, the performance margin between Octopus-8B and the GSPO baseline becomes more pronounced, increasing from 2.5 at pass@1 to 4.6 at pass@32. Moreover, compared to both GSPO and the base model, the larger performance margin achieved by Octopus further indicates that its reasoning boundary is substantially extended by the learned self-correction capability. We attribute this improvement to the augmented signals introduced by Octopus during training, which encourage the model to explore beyond its original distribution. This effect helps maintain higher entropy throughout training, leading to a stronger and more robust reasoning boundary.

Performance on Text-only Domain. Although Octopus is trained only on data with vision-language inputs, it naturally generalizes beyond multimodal settings. As shown in Fig. 8, Octopus-8B outperforms the baseline on text-only benchmarks such as AIME2024 [33] and MMLU-Pro [34], whereas the baselines fail to improve and even degrade text-only performance. This suggests that self-correction is not confined to multimodal reasoning, but instead reflects a modality-agnostic strategy for improving model capability.

6 Related Works↩︎

6.0.0.1 VLM Reasoning.

Reasoning capability is central to solving complex tasks with large language models [1], [9]. The success of text-based reasoning has also driven the exploration of reasoning with multimodal data [19]. Two primary paradigms are commonly used to enhance VLM reasoning: supervised fine-tuning (SFT) and reinforcement learning (RL). SFT requires reasoning trajectories with chain-of-thoughts (CoT) [35], which are often distilled from more powerful VLMs. RL methods, on the other hand, actively explore diverse reasoning trajectories and only require outcome-level rewards [7], [36]–[38]. While effective, RL typically requires a large number of rollout generations, which is time-consuming, and also suffers from advantage vanishing within training groups [7]. Prior work [18] explores rollout modification by adding noise to input images to encourage exploration. However, this approach is not designed to enrich reasoning or self-correction signals and incurs additional inference cost.

6.0.0.2 Self-Correction.

Self-correction has been shown to improve reasoning performance across a range of tasks [39], [40]. To enable controllable self-correction at test time, prior work typically relies on multi-pass prompting with explicitly designed correction formats [10], [17], [41]. However, such approaches require extensive model interaction and long contextual histories, making them token-inefficient at inference. The work most closely related to ours is SRPO [11], which studies single-pass self-correction in RL. SRPO uses prompting and reward shaping to encourage reflective behavior, but effective self-correction examples must still emerge spontaneously during rollouts, resulting in sparse learning signals. Moreover, self-correction and direct reasoning are jointly optimized, which can lead to mode collapse.

7 Conclusion↩︎

In this paper, we investigate how to improve reasoning performance by learning self-correction as a controllable behavior via RL. We propose Octopus, a rollout augmentation framework that constructs dense, explicit self-correction examples by pairing responses within rollout groups, enriching learning signals without additional generation cost. Moreover, we decouple the learning of self-correction and direct reasoning by response masking, avoiding objective conflicts and enabling both capabilities to be learned. Extensive experiments across seven benchmarks demonstrate that Octopus-8B achieves the best performance in both direct generation and test-time scaling, while requiring only \(0.72\times\) training time per step compared to the best baseline.

These results highlight self-correction as a crucial capability for VLMs and show that explicitly learning it leads to more capable, efficient, and robust VLM reasoning. More broadly, our findings suggest that synthesizing structured supervision from policy samples is a promising direction for improving performance and reducing training cost.

Acknowledgements↩︎

This research is supported in part by NSF IIS-2508145 and Amazon Research Award.

Impact Statement↩︎

The goal of our paper is to improve the reasoning performance of VLMs. There is no specific social impact we need to highlight and clarify in this section.

8 Implementation Details↩︎

8.1 Training Details↩︎

In this section, we describe the training details of the different methods. We implement our SFT on the LLaMA-Factory [42] and RL on the Easy-R1 [43] framework. All RL experiments in this paper are conducted on the ViRL-39k dataset [7]. We follow the experimental setup of Easy-R1 [43], and use Geometry-3k [44] as the validation set to select the best training checkpoint.

We report the training hyperparameters used in our experiments in Table 4. For fair comparison, we fix the learning rate, max sequence length, warm-up steps, and number of training epochs across all methods. During training, we adopt vLLM [45] as the inference backend, enabling faster rollout generation. The complete <sc> self-correction tokens we used is \n\n<self-correction>\n</self-correction>\n\n.

8.2 Detailed Algorithm of Octopus Augmentation↩︎

In this section, we present the complete selection algorithm and selection rules for Octopus augmentation, as shown in Algorithm 9.

8.3 Inference Prompts↩︎

In this section, we present the reasoning prompts used during training and evaluation for different methods. For the vanilla RLVR baselines, we follow the settings of Easy-R1 [43] and prompt the models to first reason through the problem and then provide the final answer within a designated box.

For SRPO [11], we follow their RL training settings and use the following prompt:

For Octopus, we prompt the model to explicitly generate responses in the \(o_1 \oplus \texttt{<sc>} \oplus o_2\) format. The detailed prompt we use is as follows:

We also report the prompt we used during the cold-start data construction stage:

9 Evaluation Details↩︎

9.1 Evaluation Setting↩︎

We evaluate the performance of Octopus-8B across seven comprehensive benchmarks. All evaluations are conducted using the VLMEvalKit [30] framework, with vLLM [45] serving as the inference backend. Detailed benchmark information is provided in Table 5.

For sampling, we set the temperature to \(t=0.6\), top-\(p\) to \(0.95\), and top-\(k\) to \(-1\). For reasoning models, we cap the inference budget at 16,384 tokens, while instruct models are limited to 4,096 tokens. During answer evaluation, we extract responses enclosed in \boxed{} for reasoning models, and use the full generated outputs for instruct models.

9.2 Inference Efficiency↩︎

To evaluate the inference efficiency of Octopus-8B, we report the average reasoning latency and output token count on MathVista in Table 6, together with the baselines. The results show that although Octopus-8B naturally produces two responses, \(o_1\) and \(o_2\), within a single inference pass, the final output lengths of different methods converge to a similar range during RL training, leading to comparable inference latency overall. This indicates that Octopus-8B achieves stronger performance than the baselines under a similar inference token budget.

10 Failure Attempts↩︎

We also explored an alternative strategy that randomly mixes elements from \({o_1^i}\) and \({o_2^i}\) to form augmented samples. However, this approach leads to training collapse in practice, as \(o_1\) and \(o_2\) follow different distributions. We therefore restrict each component to be sampled from its corresponding set, preserving the original response structure. We visualize the collapsed training curve in Fig. 10.

11 Case Study↩︎

In this section, we provide some case studies for Octopus-8B, demonstrating how Octopus-8B recovers errors in previous reasoning trajectories.

References↩︎