1 Introduction↩︎

Recent advancements in Multimodal Large Language Models (MLLMs) have demonstrated remarkable progress across a broad range of vision-language tasks [1], [2], covering from fundamental image captioning to complex visual question answering. Moving beyond static visual perception where the visual inputs of MLLMs are fixed, immutable contexts processed in a single forward pass, explorations about interleaved visual-language reasoning have raised wide attention recently [3], [4]. These methods are often referred to as “thinking with images”. In these methods, models often actively manipulate visual inputs to capture and re-examine regions of interest, thereby acquiring finer details beyond the original input image to generate the final answer.

Among the various operations employed when thinking with images, a basic operation in this reasoning process is to zoom in on regions of interest [3], [4]. Concretely, when executing a zoom-in operation, the model would typically generate bounding boxes (BBox) that delineate the area. Subsequently, this localized area is cropped from the original image, serving as a new visual input for the model’s next step of reasoning. Despite its widespread adoption, this kind of BBox-based zoom-in operation encounters significant and often overlooked challenges in the complex practical applications.

As illustrated in 1 (a), a primary limitation arises from the inherent irregularity of target object shapes. The rigid rectangular prior of bounding boxes fails to capture the diverse geometric characteristics of natural objects, which are often irregular. Consequently, rectangular boxes inevitably encompass substantial redundant background, resulting in inefficient token consumption. Furthermore, in scenarios involving overlapping objects, simple bounding boxes often fail to disentangle the target from adjacent entities. This inability to precisely isolate the region of interest would lead to semantic interference, where the model struggles to distinguish the intended object from its surroundings. Fundamentally, these issues stem from the coarse granularity of bounding boxes, which lack the capacity to provide a precise observation of the region of interest.

To mitigate these limitations of the current BBox-based zoom-in operation, we shift from rectangular bounding boxes to pixel-level segmentation masks. As shown in 1 (b), this segmentation-based pixel grounding requires the model to generate a fine-grained and accurate segmentation mask of the region of interest, instead of coarse and rectangular bounding boxes. By leveraging these segmented visual inputs, it effectively filters out redundant background and extraneous visual signals, ensuring that only the regions of interest are preserved for subsequent reasoning. Furthermore, for the newly introduced segmented visual input, we keep the position index of each segmented image patch as it is in the original image, thereby sparse segmented image patches can effectively reflect spatial relations. This also aligns more seamlessly with the nature that MLLMs structure visual tokens via positional embedding. Then, these cleaner, unambiguous visual inputs are further fed into models to provide finer details, enhancing visual reasoning.

Based on the above pixel grounding, we propose the Segmentation before Answering (SegAnswer) method in this paper, which employs the segmentation during the visual reasoning process. Concretely, the training of SegAnswer has three stages. We first conduct pixel grounding to equip the model with semantic segmentation ability. Then, a multimodal interleaved supervised fine-tuning is conducted to instruct the model to perform pixel grounding as an intermediate conversation step, utilizing the segmented images for generating the final answer. Finally, reasoning with pixel grounding is carried out to enhance the MLLM visual reasoning by reinforcement learning.

In experiments, we first evaluate our method across a wide range of benchmarks for MLLMs, including high-resolution perception (V* [5], HR-Bench 4K, and 8K [6]), general perception (MMBench [7], VisuLogic [8], and MMVP [9]), and hallucination benchmarks (POPE [10] and Hallusionbench [11]). A broad range of empirical results across these diverse benchmarks demonstrates that our method achieves consistent and considerable improvements over existing visual reasoning baselines. Beyond reasoning outcomes, we also assess the pixel grounding ability of SegAnswer. On segmentation benchmarks (RefCOCO [12], RefCOCO+ [12], and RefCOCOg [13]), our method exhibits strong pixel-level grounding performance, surpassing prior segmentation-specific approaches. In a nutshell, our contributions are threefold:

• We propose to shift from rigid rectangular bounding boxes to pixel-level segmentation for localizing the region of interest more precisely.

• We propose the SegAnswer method, which utilizes pixel-level grounding to achieve more accurate capture of regions of interest, thereby facilitating the visual reasoning ability of MLLMs.

• Our method achieves consistent improvements over previous visual reasoning methods across multiple types of evaluation benchmarks.

2 Related Work↩︎

2.1 Multimodal Large Language Models↩︎

The integration of visual encoders with powerful Large Language Models (LLMs) endowed MLLMs with visual perception capabilities [14], [15]. Representative architectures, such as LLaVA [16] and BLIP-2 [17], established a prevailing paradigm where a visual connector projects image features into a text-aligned embedding space, enabling LLMs to process visual signals alongside textual inputs. While these advancements have significantly boosted performance across general vision-language tasks, the perception of visual inputs remains static. In other words, conventional MLLMs typically process visual inputs through a single forward pass as a fixed and immutable context. Consequently, the visual information available for reasoning is restricted to the initial abstraction, preventing the model from verifying details that were not prominent in the global view.

2.2 MLLMs Visual Reasoning↩︎

Beyond static perception, recent advancements in MLLMs have embraced visual reasoning, which is referred to as “thinking with image”. These methods empower models to actively manipulate visual content by diverse operations, including drawing auxiliary lines [18], shifting image styles liu2025Visual?, highlighting sub-regions [19], and zooming in on specific areas [3], [4]. Among these operations, the zoom-in mechanism is prevalent and effective for capturing fine-grained details for the region of interest. Nevertheless, natural objects are rarely perfect rectangles, and their shapes are often irregular or non-convex. However, typical implementations of zoom-in rely on generating bounding boxes to define the region of interest. This BBox-based zoom-in operation would capture redundant background information and fail to effectively disentangle overlapping objects, leading to semantic ambiguity. To address these limitations, we introduce SegAnswer, a method that replaces coarse bounding boxes with precise semantic segmentation. By leveraging pixel-level grounding, SegAnswer isolates the target object accurately, eliminating visual interference and providing a clean context for subsequent reasoning.

3 Method↩︎

In this paper, we introduce SegAnswer, a novel framework designed to enhance the MLLM visual reasoning by precise, pixel-level grounding. Unlike former approaches that rely on bounding boxes for visual content capture, SegAnswer empowers the MLLM to actively segment regions of interest during the reasoning process. This capability allows the model to eliminate background noise and resolve semantic ambiguities inherent in complex visual scenes. As illustrated in 2, our training has three stages: pixel grounding, multimodal interleaved supervised fine-tuning (SFT), and reasoning with pixel grounding.

3.1 Stage 1: Pixel Grounding↩︎

This stage aims to align textual semantics with pixel-level visual features, effectively enabling the model’s pixel grounding ability, catching and segmenting the region of interest based on textual instructions.

Our base model is Qwen2.5-VL-7B [2]. To enable mask prediction for segmentation, we integrate a projector and the SAM 2.1 [20] as the mask decoder. The projector is a simple MLP, and it aims to help align the hidden tokens of the MLLM with the mask decoder. To obtain the mask based on the hidden tokens of the MLLM, a special token, <|seg|>, is utilized as the interface. During the forward pass, we extract the last-layer hidden states corresponding to this token. Then, these hidden states are processed by the projector and mapped into the input space of the mask decoder. Then, the mask decoder will decode the mask result as in [20]. An example of the prompt template is as follows:

For this stage, the training objective is a linear combination of the next token prediction loss and segmentation-specific losses. For the segmentation-specific losses, we adopt the loss configuration from SAM 2.1 [20], which includes a focal loss and dice loss for mask prediction, a mean-absolute-error loss for IoU prediction, and a cross-entropy loss for object prediction. To ensure stable feature alignment, we implement a two-phase training strategy. Initially, we first freeze both the MLLM backbone and the mask decoder, updating only the newly introduced projector. Subsequently, Low-Rank Adaptation (LoRA) [21] is applied to the MLLM. The projector, mask decoder, and LoRA layers of the MLLM are jointly fine-tuned. After Stage 1, the LoRA weights are merged into the MLLM parameters, equipping the model with the ability to perform pixel-level grounding.

3.2 Stage 2: Multimodal Interleaved SFT↩︎

Building upon the pixel grounding ability established in Stage 1, this stage aims to evolve the model’s capability to handle multimodal interleaved reasoning conversation patterns, where segmented images of pixel grounding are employed as an intermediate conversation step.

As illustrated in the prompt template below, the model is instructed to first analyze the image and question to decide if a specific region requires isolation. If affirmative, it will generate a <|seg|> token to execute the segmentation. Then, the segmented image will be decoded as described in 3.1. The resulting segmented image is then fed into the model, allowing the model to observe fine-grained visual information of the region of interest.

It should be noted that we keep the position index of each segmented image patch as it is in the original image, as shown in 1 (b). In this way, sparse segmented image patches can effectively reflect spatial relations. This way also aligns more seamlessly with the nature that MLLMs structure visual tokens through positional embeddings.

In addition, the optimization of the model relies solely on the next-token prediction loss derived from the language modeling objective in this stage. We do not apply explicit supervision to the segmentation mask output. Therefore, the projector and mask decoder are frozen to preserve the segmentation ability learned in Stage 1. We unfreeze and fine-tune the full parameters of the MLLM backbone.

By the end of this stage, the model is expected to be equipped with preliminary capabilities to handle multimodal interleaved scenarios, effectively treating segmentation as an intrinsic tool and establishing a solid behavioral initialization for subsequent reinforcement learning.

3.3 Stage 3: Reasoning with Pixel Grounding↩︎

This stage focuses on optimizing the reasoning strategy through Reinforcement Learning (RL) to enhance the visual reasoning ability of the MLLM.

Unlike text-only RL, where the state consists solely of text tokens, the visual reasoning process incorporates visual tokens of segmented areas derived from pixel grounding. At each time step \(t\), the \(i\)-th rollout sequence \(o^t_i\) is defined as the interleaved sequence of text and visual history: \[o^t_i = \{ \mathbf{X}_{\leq t}, \mathbf{I}_{\leq t} \} = {(X_0, I_0), \dots, (X_t, I_t) },\] where \(\mathbf{X}_{\leq t}\) represents the accumulated text tokens, \(I_{0}\) represents the visual tokens of the original image, and \(\mathbf{I}_{\geq 1}\) represents the segmented images. Also, as stated in 3.2, the position index of segmented images is also kept the same as it is in the original image, to well preserve the spatial relations. Then, given this state, the policy model (i.e., the MLLM) generates the next token in the sequence. This process continues iteratively until the model generates a final answer or reaches a maximum step limit.

During visual reasoning, intermediate visual operations (here is pixel grounding) lack explicit ground truth supervision. Therefore, we adopt an outcome-driven reward formulation that evaluates the entire reasoning trajectory based on result quality. The total reward has two parts: an accuracy reward \(R_{\text{acc}}\), and a format reward \(R_{\text{format}}\). Given a final reasoning trajectory \(\tau\), the total reward is defined as: \[R(\tau) = w_a * R_{\text{acc}}(\tau) + w_f* R_{\text{format}}(\tau),\] where \(w_a\) and \(w_f\) is the weight of \(R_{\text{acc}}(\tau)\) and \(R_{\text{format}}(\tau)\), repsectively.

For the RL algorithm, we leverage the decoupled clip and dynamic sampling Policy Optimization (DAPO) [22]. The effectiveness and efficiency of this algorithm have been verified across different tasks. Also, for the final multimodal trajectories, we use a token-wise masking setting that restricts loss calculation solely to model-predicted tokens, omitting the observation inputs.

After three stages, it completes the training pipeline of SegAnswer, equipping the MLLM with a visual reasoning capability with pixel grounding.

4 Experiment↩︎

4.1 Training Data↩︎

Stage 1: In this stage, we utilize multiple object segmentation datasets to enable the model with pixel grounding ability. The used datasets include RefCOCO [12], RefCOCO+ [12], RefCOCOg [13], RefClef [12], ReasonSeg [23], ADE20K [24], COCOStuff [25], Mapillary Vistas [26], PACO-LVIS [27] and PASCAL-Part [28]. Stage 2: In this stage, we use VisualCOT [29] as the training data. This is a large-scale VQA dataset that contains 438k question–answer pairs, each annotated with bounding boxes that mark the critical regions needed to derive the answer. It should be noted that we do not use the bounding box supervision. We only use the ground truth answer to supervise the final result. Stage 3: For RL training, we utilize ViRL39K [30] as the training data. It is a curated collection of 39k verifiable question-answer pairs for vision-language RL training. This dataset is built on top of newly collected problems and existing datasets through cleaning, reformatting, rephrasing, and verification.

4.2 Training settings↩︎

For Stage 1, when training the projector, the learning rate is \(1e-3\). This stage uses RefCOCO, RefCOCO+, RefCOCOg, and RefClef datasets and trains the model for \(5\) epochs. Then, when training LoRA layers, the projector and the mask decoder, all datasets of Stage 1 are utilized to train the model for \(3\) epochs. The learning rate is \(2e-5\). For Stage 2, the learning rate is \(2e-6\), and the total epoch is \(1\). For Stage 3, the learning rate is \(2e-5\), and the total epoch is \(1\). The number of rollouts is \(4\). \(w_a\) is \(0.8\), and \(w_f\) is \(0.2\).

4.3 Benchmarks↩︎

High-resolution perception: We evaluate our SegAnswer method on the visual detail understanding benchmarks with high-resolution visual inputs. These benchmarks require the model have a fine-grained understanding of the high-resolution image. In this part, we adopt V* [5], HR-Bench 4K, and HR-Bench 8K [6] benchmarks. General perception: Besides the fine-grained visual understanding benchmarks, we also assess the model’s ability on widely-used general perception benchmarks, including MMBench [7], VisuLogic [8], and MMVP [9]. Hallucination: We also evaluate the model’s ability on typical hallucination benchmarks, including POPE [10] and Hallusionbench [11]. Segmentation benchmarks: To verify the pixel grounding ability of our SegAnswer, we also evaluate our model on representative segmentation datasets, RefCOCO [12], RefCOCO+ [12], and RefCOCOg [13].

4.4 Main Results↩︎

We first evaluate SegAnswer across three distinct categories of multimodal benchmarks: High-Resolution Perception, General Perception, and Hallucination Evaluation. As presented in 1, we compare our framework against leading open-source MLLMs, including LLaVA-OneVision-9B [1] and our backbone model Qwen2.5-VL-7B [2]. In addition, we also compare our method with recent MLLM visual reasoning methods, such as Pixel Reasoner [3] and DeepEyes [4], which utilize BBox-based visual perception operation.

Based on 1, not surprisingly, the advantages of our approach are most demonstrated in high-resolution tasks where fine-grained detail is vital. Notably, on the V* benchmark, our method achieves a score of \(86.4\), delivering a substantial improvement over the base Qwen2.5-VL-7B model. In addition, on benchmarks including V*, HR-Bench 4K, and HR-Bench 8K, SegAnswer achieves superior performance, surpassing the other visual reasoning baselines. These results validate that compared to rectangular bounding boxes that introduce redundant background noise, precise pixel-level segmentation effectively isolates the target, allowing the model to focus exclusively on the relevant visual features required for complex visual recognition tasks.

We also assess the model performance on the general perception and representative hallucination benchmarks. According to 1, SegAnswer still demonstrates consistent improvements on these benchmarks. By reasoning with pixel grounding, the region of interest is accurately captured and segmented, and then the model can further show general perception improvement, besides the target fine-grained visual perception scenarios.

4.5 Fine-grained Visual Perception↩︎

To further assess the model’s efficacy in handling complex visual details, more comparisons on the challenging V* benchmark are provided. V* benchmark assesses MLLMs’ ability to perform fine-grained visual detail search and relative spatial reasoning. This dataset is challenging since the visual input is high resolution, and it needs a detailed visual search in the image to answer the question correctly.

As shown in 2, we further compare with several methods that target this task, including SEAL [5], DyFo [31], and Chain-of-Focus [32]. Based on the results, for this challenging task, our SegAnswer method exhibits considerable improvement. By employing pixel-level segmentation, SegAnswer strictly isolates the target features, thereby minimizing semantic interference and enabling more accurate visual reasoning in this fine-grained visual search scenario.

4.6 Qualitative Analysis↩︎

To observe the concrete reasoning process of SegAnswer, we visualize several inference trajectories on the V* benchmark. The results are provided in 3. These examples illustrate how the model invokes pixel grounding to observe small objects in the high-resolution image, avoiding the interference of complex backgrounds.

Based on the visualization results, our SegAnswer can not only accurately catch the single object that is related to the question (e.g., woman in 3 (a)), it can also handle the question that is related to multiple objects in the image. Consider the scenario in 3 (b), where the model is asked to identify the spatial relationship between the broom and the folded chair, SegAnswer explicitly predicts the need for focus of both objects, generates the <|seg|> token, and produces a precise mask that isolates both the broom and the folded chair. By reasoning over this segmented region, the model correctly identifies that the broom is on the left side of the folded chair.

In addition, according to the visualization results, compared to rigid rectangular bounding boxes, our segmentation results effectively filter out redundant background and ambiguous visual information, providing precise regions of interest. These qualitative analyses further demonstrate the effectiveness of the pixel grounding ability of SegAnswer and the reliability of the reasoning process.

4.7 Evaluation of Pixel Grounding Capability↩︎

Since our SegAnswer method utilizes pixel grounding to enhancing visual reasoning of MLLMs, its efficacy is intrinsically dependent on the precision of the underlying segmentation. To verify the effectiveness of our model’s grounding capabilities, we evaluated SegAnswer on representative referring segmentation benchmarks: RefCOCO, RefCOCO+, and RefCOCOg. We also compare SegAnswer with other segmentation-specific methods, including LISA [23], Groundhog [33], LaSagnA [34], VideoLISA [35], VISA [36] and Vitron [37].

Based on the results in 3, despite being designed for the visual reasoning task, SegAnswer demonstrates considerable performance across different representative referring segmentation datasets. Specifically, SegAnswer achieves more substantial performance gains on the more challenging RefCOCO+ and RefCOCOg datasets compared to RefCOCO with relatively simple descriptions, demonstrating SegAnswer’s effectiveness in tackling more demanding semantic comprehension scenarios. These results confirm that our training pipeline can ensure the model executes the segmentation with high quality, providing a trustworthy pixel grounding ability for complex visual reasoning.

4.8 Learning Dynamics of RL Training↩︎

During the RL training of Stage 3, we further observe the learning dynamics of the two reward components (the accuracy reward \(R_{\text{acc}}\) and the format reward \(R_{\text{format}}\)) as well as the total reward. The results are shown in 4.

The accuracy reward curve (4 (a)) exhibits a consistent upward trend, rising from an initial value of approximately \(0.35\) to over \(0.55\). This steady ascent indicates that the model is successfully learning to leverage pixel grounding to derive correct answers. The format reward curve (4 (b)) remains consistently high from the very onset of training. This stability exhibits the effectiveness of Stage 2, where the model is instructed to handle multimodal interleaved conversation scenarios, alleviating the cold-start issue before reinforcement learning begins (e.g., how to correctly use <|seg|>, <answer>, and /<answer> tags). Overall, the total reward curve also shows a consistent upward trend, indicating a steady training process.

5 Conclusion↩︎

In this work, SegAnswer is introduced to handle the fine-grained visual reasoning of MLLMs by integrating pixel-level grounding. We identify that the prevailing BBox-based zoom-in operation for visual reasoning often suffers from inherent redundancy and semantic ambiguity, particularly when processing irregular shapes or overlapping objects. By replacing inflexible bounding boxes prediction with precise semantic segmentation, SegAnswer empowers the MLLM to actively isolate and focus on exact regions of interest, thereby eliminating visual noise and resolving conflicting semantic signals. Extensive empirical evaluations across diverse benchmarks demonstrate that SegAnswer consistently shows considerable performance. The model exhibits impressive intrinsic segmentation capabilities, validating its high-quality pixel grounding ability.

References↩︎