1 Introduction↩︎

In Large Language Models (LLMs), text serves as the universal input and output format for almost all tasks, allowing a well-trained language model to flexibly support diverse downstream applications. In vision, however, this level of unification is still far from being achieved. Different types of visual information typically require their own specific representations, such as RGB for natural images, one-hot masks for segmentation, and continuous geometric maps for depth. This diversity in representation leads to a direct challenge: for each format, we often need to design a dedicated encoder and decoder, together with additional adapter components to connect them. Such barriers in both model structure and information representation make it difficult for vision models to support free-form interactions in the same way as language models, and often limit their ability to effectively generalize beyond individual tasks, leaving most vision foundation models still closer to task-specific experts than truly general-purpose visual learners [1]–[5].

This naturally raises a question: is there a way for visual information to communicate as freely as text under a unified form? In other words, can vision develop its own universal interface? Interestingly, recent studies provide useful insights. Vision Banana [6] proposes to solve understanding problems such as segmentation, depth estimation, and surface normal estimation with a generative model; for example, instead of predicting a segmentation mask in its conventional format, it generates the mask in RGB space based on the input image. This generative understanding paradigm has also been further verified on open-source image models and shown strong robustness [7]. Inspired by these observations, we ask whether RGB can be used to unify both the input and output forms of visual information, serving a role similar to text in LLMs and forming a universal “visual language” that humans can naturally interact with.

To this end, we propose RGB In and RGB Out (RINO), a unified visual representation learning paradigm that expresses all visual input and output signals in RGB format. Unlike Vision Banana that evaluates on specific understanding tasks, we aim to investigate how broadly this paradigm can generalize: we evaluate RINO on over 20 vision tasks/benchmarks across multiple domains, including dense understanding, 3D estimation, and conditioned generation, where all visual inputs and outputs are represented in RGB format and share the same encoding/decoding system as natural images. We build RINO upon pretrained image editing models such as Qwen-Image-Edit [8] without introducing any auxiliary parameters. For each type of visual information beyond natural images, we only employ lightweight, parameter-free data conversion modules to transform signals between RGB and other formats such as segmentation masks and depth maps, while inside the model, all such visual information is interpreted or generated purely as images.

By constructing this standard RGB/text-to-RGB/text framework, we surprisingly find that RINO can broadly handle all tested understanding and generation tasks in a zero-shot manner. It stably produces visually meaningful results and achieves quantitative performance comparable to in-domain expert models across different applications. For example, on depth estimation, RINO achieves an \(\delta_1\) score of 0.938, which approaches Depth Anything [2], a specialist model trained specifically for depth estimation. We further find that for conditioned image generation, which has traditionally relied on ControlNet-like [9] methods with input-specific encoders and adapters, RINO establishes a new zero-shot state of the art under existing evaluation protocols. Visually, RINO’s understanding and generation results also receive high scores in human preference studies, suggesting that RGB-based unification has strong potential for human interaction and may serve as a transferable “language” for vision.

Overall, this work provides strong evidence for the feasibility of RGB as a unified interface for vision. We present an initial attempt toward building a large-scale, multi-task unified vision-language system, and observe highly promising signs of broad task transfer under a single RGB-based formulation. At this stage, our RINO models still have limitations: their strong transfer ability relies on generic image editing backbones like Qwen-Image-Edit [8] and FireRed [10], whose web-scale pretraining data may already contain diverse visual signals such as segmentation masks and depth maps. However, such signals likely occupy only a very small portion of the pretraining distribution, which may lead to a performance gap between RINO and in-domain expert models on some specific applications. We refer to the current fully zero-shot version as RINO-Zero. In future work, we will instruction-tune existing image editing models with standard image editing data mixed with a proportion of RGB-formatted non-natural visual data, such as segmentation masks and depth maps, so that the RINO framework is introduced during training across a wider range of tasks. We hope this work can provide useful insights for future unified vision-language systems and help extend this paradigm to broader vision domains, including 3D vision, video, and world models.

2 Method: Unified RGB Interface for Vision↩︎

We formulate general vision-language understanding and generation tasks under a unified RGB/text-to-RGB/text paradigm. Given an input image \(x_{\mathrm{img}}\) and an input text prompt \(x_{\mathrm{text}}\), a multimodal image editing model \(F\) produces an output image \(y_{\mathrm{img}}\) and, optionally, an output text response \(y_{\mathrm{text}}\): \[y_{\mathrm{img}}, y_{\mathrm{text}} = \mathcal{F}(x_{\mathrm{img}}, x_{\mathrm{text}}),\] where \(\mathcal{F}\) is a generic pretrained image editing model such as Qwen-Image-Edit [8]. In our framework, we do not introduce task-specific heads, encoders, decoders, or adapter parameters. Instead, all visual information is represented in RGB format and processed by the same image encoder and decoder inside \(\mathcal{F}\). This allows natural images, segmentation masks, depth maps, pose maps, edge maps, and other structured visual signals to share the same visual interface. We illustrate our overall framework in 1.

For visual understanding tasks, the input \(x_{\mathrm{img}}\) is a natural image, and the input \(x_{\mathrm{text}}\) specifies the task to be performed. The output image \(y_{\mathrm{img}}\) is an RGB representation of the target structured visual information. For example, in semantic segmentation, \(y_{\mathrm{img}}\) is an RGB segmentation mask; in depth estimation, \(y_{\mathrm{img}}\) is an RGB-formatted depth map. The optional text output \(y_{\mathrm{text}}\) can be used to explain the prediction or describe the generated visual result. When evaluating \(y_{\mathrm{img}}\), we use a parameter-free converter to transform the RGB output into the corresponding task-specific format. For example, an RGB segmentation map can be converted into class labels through a predefined color mapping, and an RGB depth visualization can be converted into a depth map through a predefined decoding rule. These converters are only used before or after model inference. All visual reasoning is performed inside the base model \(\mathcal{F}\) through the unified RGB interface.

For visual generation tasks, the input \(x_{\mathrm{img}}\) can be an RGB-formatted visual condition, such as a semantic layout, a pose skeleton, a depth map, an edge map, or a mask. The text prompt \(x_{\mathrm{text}}\) describes the desired generation or editing behavior. The model then generates an output image \(y_{\mathrm{img}}\), which is usually a natural image that follows the given visual condition and text instruction. This formulation treats conditioned generation as the dual problem of visual understanding. In understanding, the model maps a natural image to an RGB-formatted structured prediction. In generation, the model maps an RGB-formatted structured condition back to a natural image. For example, if semantic segmentation can be formulated as image-to-RGB-mask prediction, then segmentation-conditioned image generation can be formulated as RGB-mask-to-image generation. Similarly, pose estimation can be viewed as image-to-RGB-pose prediction, while pose-guided generation can be viewed as RGB-pose-to-image generation.

Base models. We run RINO-Zero on three open-sourced image editors: Qwen-Image-Edit [8], LongCat-Image-Edit [11] and FireRed-Image-Edit [10]. Qwen-Image-Edit builds on Qwen-Image, a 20B-parameter multimodal diffusion transformer (MMDiT) that couples a Qwen2.5-VL vision-language model with a VAE for image tokens. LongCat-Image-Edit is a compact 6B bilingual (Chinese-English) editor from Meituan, built on a hybrid MMDiT / single-stream DiT design with a Qwen2.5-VL text encoder and tuned for efficiency at a smaller size. FireRed-Image-Edit, from the FireRed team, extends an open-source text-to-image foundation model with a double-stream multimodal-diffusion design for high-fidelity, instruction-following edits. For all three we use the released weights as a black-box RGB-to-RGB editor, without adding, removing, or fine-tuning any layers.

3 Experiments↩︎

3.1 Understanding Tasks: Unifying Outputs as RGB↩︎

Depth Estimation. For depth estimation, we prompt the image editor to repaint the input image as a grayscale depth visualization, where nearby surfaces are brighter and distant surfaces are darker. We then recover a dense relative depth map from the per-pixel luminance of the generated image. In this task, we evaluate with Qwen-Image-Edit [8] and LongCat-Image-Edit [11] in a zero-shot setting with a fixed sampling configuration. Since a generative editor produces relative depth with unknown scale and offset, we follow the standard affine-invariant evaluation protocol. For each image, we fit a scale and shift to the ground-truth depth by least squares before computing \(\delta_1\) and AbsRel. 1 follows the disparity-space, or inverse-depth, convention used by MiDaS and Depth Anything [2], [12], while 2 follows the linear depth-space convention used by Marigold [13].

6pt

6pt

1 compares the two editors with three relative-depth specialists in disparity space: MiDaS V3.1 [15], Depth Anything V1 [2], and Depth Anything V2 [14], with specialist results quoted from [14]. Although neither editor is trained for depth estimation or equipped with any depth-specific parameters, Qwen-Image-Edit performs surprisingly close to dedicated models. It remains within a few \(\delta_1\) points of the specialists on DIODE-indoor [16] (\(0.938\) vs.\(0.952\)), and shows a roughly five-point gap on NYUv2 [17] and KITTI [18]. 2 reports the same comparison in linear depth space, where performance drops on wide-range scenes such as KITTI and iBims-1 [19]. We include Vision Banana [6] as a reference in this table, since it predicts raw metric depth without per-image alignment and is therefore not directly comparable to the affine-invariant editor outputs. Qualitative visualizations are shown in 2. We observe that RINO’s prediction can clearly reflect the depth layout of input images.

6pt

Surface Normal Estimation. Surface normal estimation follows the same RGB-to-RGB paradigm as depth estimation. We prompt the editors to repaint the input scene as a standard normal map, where surface orientation is encoded by color, and decode the generated RGB image back into per-pixel unit normals. We evaluate Qwen-Image-Edit and FireRed-Image-Edit in a zero-shot setting. The image editors output normals in an unknown coordinate convention, which we resolve once following [7], and report the standard mean and median angular error (degrees). DSINE [20] is scored through the same pipeline as a specialist reference. 3 reports surface normal estimation results on NYUv2, iBims-1, and DIODE-indoor. FireRed, a generic image editor without normal-specific training or parameters, is the strongest editor across all three datasets. It only slightly trails the DSINE specialist in mean angular error, although DSINE remains sharper at the pixel level, as reflected by its lower median error. These results suggest that a zero-shot image editor can approach a dedicated normal estimation model in average surface orientation prediction, while still leaving room for improvement in local precision. Qualitative visualizations are shown in 3.

Semantic Segmentation. We follow  [7], where only the categories present in the ground-truth of the current image are included in the prompt. To mitigate common generation artifacts such as color jitter and noisy pixels, we apply a lightweight parameter-free refinement stage after color decoding, consisting of local majority filtering and small connected-component removal. To handle the large label space of ADE20K, we introduce a hierarchical segmentation strategy: the model first predicts masks for 10 super-class, and fine-grained categories are subsequently generated within their corresponding regions. Unless otherwise specified, all image editing backbones use 12 denoising steps, CFG = 5.0, and the same refinement pipeline. LongCat-Image-Edit employs 50 denoising steps due to its lower generation stability. We use three standard segmentation metrics: mean Intersection over Union (mIoU), mean class accuracy (mAcc), and all-pixel accuracy (aAcc). Quantatitive results are reported on the validation sets of Cityscapes [23], Pascal VOC  [24], and ADE20K [25].

3pt

4pt

4 shows that RINO achieves semantic segmentation performance comparable to zero-shot specialist models, with mAcc and aAcc even surpassing segmentation experts on PASCAL VOC. 5 further evaluates segmentation at both coarse and fine granularities on ADE20K. We observe that under the coarse setting with 10 classes, RINO can even outperform the semantic segmentation specialist MaskCLIP. These results suggest that RINO has strong potential for dense prediction: the model already captures rich image semantics, but may require more domain-specific knowledge to better align its predictions with fine-grained category names, such as the 150 classes in ADE20K. Visualization results are shown in 4.

8pt

Object Detection and Instance Segmentation. Object detection and instance segmentation are challenging vision tasks as they require not only recognizing per-pixel semantics but also distinguishing different instances within the same category. As a result, open-vocabulary zero-shot models often struggle on datasets such as COCO, which contains 80 object categories. In RINO, we perform both tasks through per-class silhouette painting. For each present class, we prompt the editor to paint its instances as solid white blobs on a black background, while also specifying the expected number of instances. We then threshold the output and split touching blobs using morphological opening. Each connected component is treated as one instance: the blob defines its mask, and its tight bounding box defines the detection. Thus, a single prediction can be evaluated for both object detection and instance segmentation. The queried classes and instance counts are taken from the ground truth, i.e., an oracle-class and oracle-count setting, to isolate localization ability from open-set recognition. We evaluate Qwen-Image-Edit [8], FireRed-Image-Edit [10], and LongCat-Image-Edit [11] in a zero-shot manner on COCO val2017.

As shown in 6 and 5, although these tasks are highly challenging for zero-shot models, RINO still produces visually meaningful predictions, achieving 16.3% box AP\(_{50}\) and 13.9% mask AP\(_{50}\). These results remain below in-domain specialist models, such as Mask R-CNN with 63.5% box AP\(_{50}\), but they demonstrate that a zero-shot image editing model can already reach a non-trivial level of localization and instance-level prediction on COCO. 5 further suggests that the relatively low quantitative scores are often caused by task-specific failure modes such as over-detection or over-segmentation, as observed in the Qwen and FireRed examples. We believe these issues can be substantially mitigated with a small amount of task-specific fine-tuning, and expect future versions of RINO to match the performance of specialist models.

6pt

Panoptic Segmentation. We prompt the editor to paint each region with a flat color, using one named color per class and black for the background. We then decode the output by nearest-color matching: stuff classes are treated as single segments, while thing classes are split into connected components. Following the zero-shot oracle-class protocol, only the classes present in each image are specified in the prompt. We evaluate Qwen-Image-Edit [8], FireRed-Image-Edit [10], and LongCat-Image-Edit [11]. We report Panoptic Quality (PQ) and its two components, Segmentation Quality (SQ) and Recognition Quality (RQ), where \(\mathrm{PQ}=\mathrm{SQ}\times\mathrm{RQ}\). PQ is the standard metric for panoptic segmentation: SQ measures the mask quality of matched regions, while RQ measures recognition and matching quality.

7 reports results on Cityscapes [23], ADE20K [25], and COCO [38]. Among the editors, Qwen-Image-Edit performs best, followed by FireRed and LongCat. Notably, despite no panoptic-specific training, the editors already show strong zero-shot spatial segmentation ability. Their predicted masks are often well aligned with object and region boundaries, leading to high SQ scores (Qwen SQ \(65\)–\(74\)). The main limitation instead lies in recognition and instance association, as reflected by lower RQ scores (Qwen RQ \(15\)–\(17\)). In many cases, coherent regions are correctly separated but assigned to wrong semantic categories, or split into multiple segments due to task-specific decoding ambiguities. This issue is most pronounced for thing classes, where adjacent instances and small objects are often merged or missed; on Cityscapes, Qwen reaches \(22.4\) PQ on stuff but only \(1.5\) PQ on things. These results suggest that RINO can already produce structurally meaningful panoptic predictions zero-shot, while the remaining gap is largely driven by task-specific recognition and instance-matching factors rather than spatial segmentation quality. This is also supported by the visualizations in 6.

10pt

Human Pose Estimation. Following the top-down protocol, each person is cropped using its ground-truth box and letterboxed to a fixed canvas. We prompt the editor to repaint the crop as an OpenPose-style [42] colored skeleton on a black background, and then decode COCO-17 keypoints from the output. Since image editors do not reliably reproduce a fixed color palette, we use a color-agnostic decoder: joint candidates are extracted as peaks of the foreground distance transform and labeled by matching to a canonical upright skeleton template. No keypoint head or auxiliary parameters are introduced. We evaluate Qwen-Image-Edit [8], FireRed-Image-Edit [10], and LongCat-Image-Edit [11] in a zero-shot manner on COCO val2017 single-person crops. Since a generative editor may predict pose up to an unknown similarity transform, we report bbox-normalized PCK@\(\tau\) under two protocols: raw, in absolute image coordinates, following dedicated estimators, and PA, after a per-image scale-and-translation fit to the ground truth, which isolates pose structure.

8 reports results on COCO val2017. The editors already produce non-trivial zero-shot pose predictions, often recovering coherent human-body layouts and plausible limb structures. Although precise keypoint localization remains challenging, the generated skeletons are visually meaningful and can be decoded into COCO-17 keypoints without any pose-specific head or training. The improved performance under similarity alignment further suggests that these models capture the coarse structure of human pose, even when absolute localization is imperfect. These results indicate that RGB skeleton prediction is a viable interface for pose estimation under the RINO framework. With lightweight task-specific fine-tuning, we expect this ability to become substantially stronger and more reliable. Qualitative visualizations are shown in 7.

10pt

Referring Expression Comprehension and Segmentation. We perform language grounding through image editing. For each referring expression, we prompt the editor to cover the referred object in solid red while leaving all other pixels unchanged. We then compute the difference between the edited output and the input image, and keep the largest changed connected component as the prediction: its pixels define the segmentation mask, and its tight bounding box defines the comprehension result. The editor only receives the image and the expression, without access to boxes, classes, or instance counts, and no grounding head is introduced. We evaluate Qwen-Image-Edit, FireRed-Image-Edit, and LongCat-Image-Edit zero-shot on the RefCOCOg UMD validation split [46], [47]. We report the standard metrics for both subtasks: box Prec@0.5 for referring expression comprehension (REC), where a prediction is correct if its box IoU is at least \(0.5\), and cumulative IoU (cIoU) for referring expression segmentation (RES), computed as dataset-pooled intersection-over-union. Missed predictions are assigned zero score.

As shown in 9, the open-source editors achieve meaningful zero-shot grounding from free-form language to image regions. Although their segmentation accuracy is still below Vision Banana [6], whose cIoU is comparable to supervised grounders such as UNINEXT and OneRef, they can already localize and segment referred objects without any grounding-specific head, box supervision, or task-specific fine-tuning. This is a substantially harder setting than oracle-class segmentation, since the model must jointly parse the referring expression, identify the target instance, and express the grounded region as an RGB edit. The results suggest that generic image editors already possess a non-trivial ability to connect language with pixel-level visual grounding through the RGB editing interface. Qualitative examples in 8 further show that many predictions are visually interpretable and spatially well formed, indicating a promising direction for improving referring understanding under the RINO framework.

10pt

Open-Vocabulary Instance Segmentation (SA-Co). For each query, we prompt the editor once to repaint the image as a binary mask: all instances matching the phrase are painted pure black, all other regions are painted pure white, and the output should be fully white if the phrase is absent. We take black pixels as foreground and split them into instances using connected components, dropping small components and scoring each instance by area. The model is given no class list or instance count; the query is a single free-form noun phrase that may or may not appear in the image, requiring the editor to perform both concept localization and presence detection. No task-specific head is introduced. We evaluate Qwen-Image-Edit [8], FireRed-Image-Edit [10], and LongCat-Image-Edit [11] zero-shot on the attributes split of SA-Co/Gold [48]. We report the three official SA-Co metrics: pmF1, the positive-only micro mask-F1 averaged over IoU thresholds from \(0.5\) to \(0.95\); IL\(\_\)MCC, the Matthews correlation coefficient for per-query present/absent prediction; and \(\mathrm{cgF1}=100\cdot\mathrm{pmF1}\cdot\mathrm{IL_MCC}\), the headline score combining segmentation quality and presence detection. We report pmF1 and cgF1 as percentages.

10 reports results on the attributes subset. Qwen-Image-Edit performs best among the editors, followed by FireRed and LongCat. While the editors do not yet match dedicated open-vocabulary segmentation models in pmF1, they already produce coherent zero-shot masks for free-form visual concepts without any class list, detector, or segmentation head. The main weakness is open-set presence detection: since the editors tend to paint a mask even when the queried phrase is absent, IL\(\_\)MCC remains close to zero, which in turn suppresses the headline cgF1. This limitation reflects the absence of an explicit presence classifier rather than a complete failure of RGB-space segmentation; indeed, even Vision Banana relies on an external MLLM gate to obtain a meaningful IL\(\_\)MCC. These results suggest that generic image editors can localize named concepts through the RGB editing interface, but reliable absent-case rejection remains an important direction for future improvement. Qualitative visualizations are shown in 9.

3.2 Generation Tasks: Unifying Inputs as RGB↩︎

4pt

Depth-Conditioned Image Generation. We evaluate depth-to-image generation as a direct test of RGB-in/RGB-out controllable generation. Following the MultiGen-20M depth protocol, each validation sample provides an RGB depth map and a text caption. RINO passes the depth map to an image editor as an ordinary image condition and instructs it to synthesize the corresponding natural image. No depth encoder, ControlNet branch, adapter, or task-specific decoder is added; the model only sees an RGB condition image and a natural-language instruction. For the main paper-facing comparison, we evaluate Qwen-Image-Edit, FireRed-Image-Edit, and LongCat-Image-Edit on the same 5,000-image validation split with the same prompt, seed, and inference setting: steps=8, CFG=2.5, and a shared short quality prompt. We follow the ControlNet++ evaluation protocol for depth-conditioned generation [50]. Condition fidelity is measured by extracting depth from generated RGB images with DPT-Large  [56] and computing RMSE in 0–255 depth space against the input condition. Image quality is measured by FID against the real validation images, and text alignment by CLIPScore. Lower RMSE and FID are better; higher CLIPScore is better. The external rows are paper-reported anchors rather than locally re-scored outputs. We also note a resolution caveat: the editor outputs in our pipeline are generated at their native 1024-pixel resolution from 512-pixel MultiGen control maps, whereas the public depth-control baselines are reported at 512 pixels. A downsample-to-512 sanity check preserves the same qualitative conclusion, but the most directly aligned cross-method metric is the DPT-Large RMSE-255 condition score.

11 compares zero-shot RINO with trained controllable-generation systems. Among the zero-shot editors, Qwen gives the best balanced result: RMSE-255 \(33.83\), FID \(19.44\), and CLIPScore \(30.15\). FireRed has slightly higher CLIPScore (\(30.34\)) but worse depth fidelity, while LongCat keeps reasonable RMSE but falls behind in FID and CLIPScore. Qwen remains behind the strongest trained control models, such as ControlNet++ and ControlAR, which use dedicated conditioning mechanisms. However, it is competitive with several trained adapter/control baselines in depth fidelity while using only a zero-shot RGB editing interface. This supports the central RINO hypothesis: spatial controls can be consumed as images without introducing a separate encoder or adapter for each condition type. Qualitative examples in 10 show that the editors usually respect coarse depth layout, while differing in texture realism and semantic faithfulness.

Semantic Segmentation Map Conditioned Generation. Following the ControlNet++ protocol [50], each validation sample provides an RGB semantic segmentation map and a text caption. RINO treats the segmentation map as an ordinary RGB conditioning image and instructs the editor to synthesize a natural image whose semantic layout follows the per-region class colors. No segmentation encoder, ControlNet branch, adapter, or task-specific decoder is added: the model only sees an RGB condition image and a natural-language instruction. Because we observe that the richness of the prompt has a large impact on both image quality and condition fidelity, we re-caption every validation image once with an open-source VLM and use the resulting caption as the text prompt for all editors. We evaluate Qwen-Image-Edit [8], FireRed-Image-Edit-1.1 [10], and LongCat-Image-Edit [11] on the ADE20K [25] and COCOStuff [38] validation splits at the native input resolution. Each editor uses its recommended sampling configuration: Qwen-Image-Edit with steps=40 and CFG=4.0, FireRed-Image-Edit-1.1 with steps=40 and CFG=4.0, and LongCat-Image-Edit with steps=50 and CFG=4.5. We follow the ControlNet++ evaluation protocol for segmentation-conditioned generation [50]. Condition fidelity is measured by re-extracting a semantic segmentation map from each generated image with a Mask2Former [34] segmenter and comparing it against the input condition; we report mean Intersection over Union (mIoU), mean class accuracy (mAcc), and all-pixel accuracy (aAcc). Image quality is measured by FID against the real validation images, and text alignment by CLIPScore against the re-captioned prompt. Lower FID is better; higher mIoU, mAcc, aAcc, and CLIPScore are better.

12 compares the three zero-shot editors against ControlNet++ as a task-trained grounded-generation reference on ADE20K and COCOStuff. All three editors deliver strong condition fidelity: on ADE20K they cluster tightly around \(46\) mIoU (Qwen \(46.24\), FireRed \(45.82\), LongCat \(46.37\)), and on COCOStuff around \(37\) mIoU (FireRed \(37.55\), Qwen \(37.40\), LongCat \(36.38\)). Image quality follows the editor backbone rather than the conditioning interface: Qwen and FireRed are essentially tied on both datasets (\(31.70\)–\(32.05\) FID on ADE20K, \(15.54\)–\(15.94\) on COCOStuff), while LongCat trails noticeably (\(36.08\) and \(20.95\) FID). CLIPScore is also tightly clustered across editors, indicating that the re-captioned prompts are followed comparably well. Overall, a general-purpose image editor consumes a semantic segmentation map as an ordinary RGB condition and respects the per-region class layout zero-shot, with image quality limited by the editor backbone rather than by the lack of a segmentation adapter. Visualizations are shown in 11.

6pt

Human-Pose Conditioned Generation. We adopt the pose-guided text-to-image protocol of HumanSD [57] and Stable-Pose [58] on the Human-Art  [59] validation split. For each sample, the ground-truth COCO-17 keypoints are rendered into an OpenPose-style colored skeleton (\(16\) limb segments on a black canvas). The editor receives only this RGB skeleton and the associated caption, and is instructed to align each body part with the corresponding colored bone while omitting the skeleton overlay from the synthesized image. No pose encoder, ControlNet branch, or trainable adapter is introduced. We follow the HumanSD/Stable-Pose evaluation protocol. Pose fidelity is assessed by re-estimating the pose of each generated image with HigherHRNet  [60] and comparing it to the conditioning skeleton through COCOevalSimilarity, yielding a cosine-similarity-based average precision (CAP), and a people-count error (PCE). Image quality is measured by FID and KID against the real Human-Art images, and text alignment by CLIP-Score. Editor outputs are scored with the public evaluation code under which the baseline numbers were reported.

13 and 12 show that RINO achieves highly competitive zero-shot performance for human-pose conditioned image generation. Although our editors are not trained on this task, their results are already comparable to in-domain pose-guided generation models, demonstrating that generic image editors can effectively interpret RGB skeleton inputs as spatial generation conditions. Qualitatively, RINO often produces images that better follow the input pose than ControlNet-based baselines, especially in cases involving unusual body configurations or diverse artistic styles. The generated images also maintain strong text-image alignment and visual realism, suggesting that the RGB-in/RGB-out interface provides an effective and flexible alternative to task-specific pose encoders for conditioned generation.

3pt

Instance Map Conditioned Generation. Following the InstanceDiffusion protocol [61], each validation sample provides a per-instance map together with a text caption. Unlike a semantic segmentation map, this condition distinguishes individual object instances of the same class rather than just per-category regions. RINO renders the instance map as an ordinary RGB conditioning image and instructs the editor to synthesize a natural image whose object instances follow the input layout. No instance encoder, grounding module, detector head, or trainable adapter is introduced. As in the segmentation-map setting, we re-caption every validation image once with an open-source VLM and use that caption as the prompt for all editors, because we find caption richness materially affects both image quality and the recognizability of generated instances. We evaluate Qwen-Image-Edit, FireRed-Image-Edit, and LongCat-Image-Edit on the COCO2017 [61] validation split at the native input resolution. Each editor uses its recommended sampling configuration: Qwen-Image-Edit with steps=40 and CFG=4.0, FireRed-Image-Editwith steps=40 and CFG=4.0, and LongCat-Image-Edit with steps=50 and CFG=4.5. We follow the InstanceDiffusion evaluation protocol [61]. Condition fidelity is measured by running a pretrained YOLOv8m-Seg [62] detector on the generated images and comparing its detected instance masks against the input condition with COCO’s official evaluation metrics. We report mask AP (IoU \(0.5{:}0.05{:}0.95\)), AP\(_{50}\) (IoU \(\ge0.5\)), and Average Recall (AR), which together measure both the localization precision and the recall of the requested instances. Image quality is measured by FID against the real COCO2017 validation images. Higher AP, AP\(_{50}\), and AR are better; lower FID is better.

Table 14 compares the three zero-shot editors with InstanceDiffusion as a task-trained grounded-generation reference. The editors follow the per-instance layout closely: LongCat reaches the highest AP (\(28.0\)) and AR (\(44.2\)), surpassing the trained InstanceDiffusion reference on both metrics, while Qwen (\(25.4\) AP, \(44.3\) AP\(_{50}\), \(40.8\) AR) and FireRed (\(23.2\) AP, \(42.3\) AP\(_{50}\), \(37.4\) AR) remain within a few points of it. At the stricter AP\(_{50}\) threshold the trained reference still leads (\(50.0\) vs. \(42\)–\(46\) for the editors), indicating that precise instance boundaries are harder to recover than coarse instance placement. Image quality is consistently better than the trained reference across all editors (FID \(15.3\)–\(18.5\) vs. \(25.5\)), reflecting the strength of the modern editor backbones. Overall, a general-purpose image editor can consume a per-instance RGB layout zero-shot and produce images that respect individual instance placements while maintaining higher visual quality than the task-trained baseline. Visualizations are shown in 13.

Canny-Conditioned Image Generation. Following the MultiGen-20M protocol [50], each sample consists of a Canny edge map (white edges on a black background) and a text caption. RINO treats the edge map as an ordinary RGB conditioning image and instructs the editor to synthesize a natural image whose object boundaries and spatial layout conform to the edges, adding no edge encoder, ControlNet branch, adapter, or task-specific decoder. We evaluate Qwen-Image-Edit [8] on the \(5{,}000\)-image validation split at \(512{\times}512\), drawing \(4\) samples per condition with steps=30 and CFG=4.0 under a shared instruction prompt. All reported numbers are means over the \(4\) samples per image; no best-of-\(N\) selection is used. We follow the ControlNet++ protocol for the Canny condition [50]. Condition fidelity is measured by re-extracting a Canny edge map from each generated image and comparing it to the input condition; because Canny is a hard (binary) edge, the controllability metric is the per-pixel edge F1-Score, extracted with OpenCV cv2.Canny(100,200) to match the protocol exactly. We additionally report image quality as FID against the real validation images and text alignment as CLIP-Score (using CLIP ViT-B/16), both averaged over the \(4\) generated groups and computed with the same implementations as ControlNet++; the re-extracted edge maps are excluded from the FID image set.

15 reports all three metrics. On controllability, zero-shot Qwen-Image-Edit reaches a Canny F1 of \(14.90\), about \(0.40\times\) the purpose-trained ControlNet++ (\(37.04\)) and below every trained prior (\(23\)–\(37\)). The same gap holds on image quality and text alignment: its FID of \(59.4\) and CLIP-Score of \(27.3\) trail the trained controllers (FID \(15\)–\(20\), CLIP \(\approx32\)). Zero-shot canny generation therefore trails purpose-trained controllers on all three axes — edge controllability, distributional realism, and prompt alignment — as expected for a general instruction editor with no edge-conditioned training and no dedicated conditioning branch. The result nonetheless supports the central RINO hypothesis in the harder binary-edge regime: a spatial control can be consumed as an ordinary image, with above-chance controllability, without any task-specific adapter.

To verify that this controllability comes from the Canny image rather than the caption, we ablate the condition while holding the editor fixed (top block of 15). Removing the edge map and keeping only the caption (Raw Caption-only) collapses fidelity to \(6.75\), and first describing the edge map in words with a vision-language model (Qwen2.5-VL-7B) and feeding that text instead of the image (Raw Caption + Canny Caption) reaches only \(7.79\) — both far below the \(14.90\) from the edge image itself. This implies that the Canny image carries pixel-level spatial information that a language bottleneck cannot substitute. The quality metrics add nuance: supplying structure (the Canny image or its VLM description) improves FID over caption-only (\(71.6\to58\)–\(59\)), i.e.a spatial prior yields more realistic images, but routing it through a long VLM description lowers CLIP-Score (\(27.8\to24.0\)), as the appended layout text competes with the caption for prompt alignment. Visualization results are shown in 14.

4 Related Work↩︎

Visual estimation models. Estimation tasks map a natural image to a structured signal, and each task we study has a mature line of specialists that defines its performance ceiling andserves as our reference. Monocular depth estimation progresses from mixing datasets for scale- and shift-invariant relative depth [12], [15], [63] to large-scale data-driven and metric-aware systems [2], [14], [64]–[66], and pretrained diffusion generators are themselves fine-tuned into strong depth and normal estimators [13], [20]–[22]. Segmentation converges on the mask-classification view, where a transformer predicts a set of labeled masks to cover semantic, instance, and panoptic segmentation at once [31], [34]–[37], [67], [68], with an open-vocabulary branch that names novel categories via language embeddings [26]–[30]. The remaining tasks follow the same pattern of dedicated architectures: pose estimation [39]–[41], [69], detection and instance segmentation with text-grounded open-set variants [31]–[33], [49], and referring segmentation [43]–[45], many built on general-purpose backbones [1], [4], [5] but still read out through a task-specific head. Unification here stays within a family, with each keeping its own encoder, decoder, and loss.

Controllable image generation. Generation tasks move in the opposite direction, synthesizing a natural image from a structured condition—Canny edges, depth, semantic or instance maps, boxes, or poses. ControlNet [9] conditions a text-to-image backbone by attaching a trainable encoder copy that injects the control signal, and later work improves fidelity or extends it to autoregressive backbones [50], [51]; adapter methods instead learn small per-condition side networks [53], and unified frameworks route many condition types through a shared module [54], [55]. Layout- and instance-grounded generation [52], [61], [70] and pose-guided synthesis [57], [58] follow the same recipe. The recurring ingredient is a per-condition encoder or adapter trained to inject each new signal. Estimation and generation are thus near-mirror images—one reads a structured signal out of an image, the other writes one in—yet the two traditionally use separate architectures and output spaces and are never unified under a shared interface.

Toward a unified model for estimation and generation. Vision Banana [6] is the first to bridge the two directions: parameterizing the output space of vision tasks as RGB images reframes perception as image generation, and instruction-tuning a single generator makes it competitive with zero-shot specialists such as Segment Anything [48] and the Depth Anything series [2], [14], [64], with open-source editors shown to exhibit similar behavior [7]. This casts RGB as a candidate common interface for both reading and writing images, analogous to text in language models. Vision Banana, however, is built on a proprietary backbone, leaving open how far this interface carries over to open, publicly released models. RINO shares its RGB-as-interface premise and pushes it to this setting: we take open-source image editors as a frozen black box, add only parameter-free RGB conversions, and place estimation and conditioned generation under one RGB-to-RGB formulation across 25 tasks, measuring how closely this fully zero-shot system approaches the specialists above.

5 Conclusion↩︎

RINO shows that RGB can act as a common interface for both perception and generation: expressing all visual inputs and outputs as RGB lets a single frozen image editor handle a broad range of tasks without any task-specific encoders, decoders, adapters, or auxiliary parameters. Fully zero-shot and without any training, RINO-Zero approaches in-domain specialist models across 25 tasks spanning dense estimation, 3D geometry, and conditioned generation, and offers a zero-shot alternative to adapter-based controllable generation. Its accuracy remains bounded by the visual priors of the editing backbone and by the parameter-free RGB read-out, leaving a gap to specialists on some tasks; lightly instruction-tuning editors on RGB-formatted visual signals, so that the RINO interface is present during training, is a natural way to close it. These findings point to RGB as a shared interface for vision and invite its extension to 3D vision, video, and world models.

References↩︎