ChatImage: Navigating Long-Form LLM Answers through Interactive Images
July 06, 2026
1 Introduction↩︎
Large Language Models (LLMs) have become a common interface for answering complex questions [1], [2]. Their answers, however, are still delivered mainly as extended prose. This format is effective for short factual queries, but it becomes less suitable when an answer contains internal structure, such as multiple entities, comparisons, procedural stages, or spatial relations. In such cases, readers often need to locate a specific module, inspect its context, and return to it later. Plain text can encode this information, but it provides limited support for navigation.
Recent progress in text-to-image generation [3]–[5] and vision-language understanding [6], [7] suggests a complementary presentation format: rendering the answer as an image. A generated infographic, map, or scene can expose modules, hierarchies, routes, and spatial relationships in a single view. Static images, however, lose the explanatory structure associated with each region and provide no mechanism for region-level follow-up. A direct overlay of planned boxes is also unreliable, because image generators do not consistently place content at the requested coordinates [8], [9]. If the interaction layer is copied from the pre-generation layout, a user may click an empty region or an unintended visual element.
We propose ChatImage, a system that converts a long-form LLM answer into an interactive visual image: a generated image with transparent hotspots placed over visible answer regions (Figure 1). Each hotspot opens the explanation associated with that region and maintains a local follow-up thread. The central technical problem is alignment: a hotspot should cover the object, panel, landmark, or diagram region visible to the user, rather than the box specified before image generation.
ChatImage addresses this problem through a generate-then-ground interaction loop. The system converts the answer into a small set of visual modules, plans an approximate layout, synthesizes an image prompt, and generates the image. A second visual grounding pass then localizes the rendered modules and places clickable hotspots on the discovered regions. This design separates what the answer should contain from where the generator renders it. The resulting artifact supports region-level exploration: the user selects a visible region, inspects the corresponding detail and local preview, and asks follow-up questions without restating the full context. The implementation uses a provider-agnostic backend for text, image, and vision models, with deterministic local models for reproducible testing and configured external models for full-generation evaluation.
To our knowledge, prior work has not studied LLM answers as navigable, region-interactive visual images with vision-grounded hotspots. The main contributions are:
Interactive visual answers. We formulate the task of presenting a long-form LLM answer as an explorable image with clickable hotspots, detail panels, and per-region follow-up threads.
Generate-then-ground alignment. We decouple structured content from visual placement: the system first renders the image, then uses visual grounding and optional SAM-family mask refinement to place hotspots on visible regions rather than on a fixed layout grid.
Reference implementation. We provide a provider-agnostic implementation with deterministic local execution, configured external model execution, local persistence, per-hotspot follow-up threads, and calibration tools for inspecting hotspot bounds.
Benchmark and grounding evaluation. We introduce a 30-question benchmark spanning multiple answer formats and evaluate the full interaction loop with completion checks and a strict visual-alignment gate.
2 Related Work↩︎
ChatImage is related to text-to-image generation, visual grounding, interactive visualization, and structured LLM output. The specific problem addressed here, converting an LLM answer into a region-interactive visual image, has received limited direct attention. We therefore discuss the closest technical areas and identify the gap addressed by ChatImage.
2.1 Text-to-Image and Controllable Generation↩︎
Diffusion models [3]–[5], [10] can produce high-fidelity images, but they do not reliably satisfy precise spatial instructions. A prompt that requests a module at a specified canvas location does not guarantee exact placement [8], [9]. Controllable methods such as ControlNet [11], GLIGEN [12], training-free layout-to-image [13], and InfographicNLP [14] impose additional structure, but their primary objective is image synthesis rather than the construction of interactive regions aligned with a structured answer. ChatImage uses text-to-image generation as a rendering backend and estimates region locations after generation through visual grounding.
2.2 Visual Grounding and Segmentation↩︎
Referring localization methods such as MDETR [15], TransVG [16], UNINEXT [17], Grounding DINO [18], and YOLO-World [19], as well as grounding-oriented VLMs [6], [7], [20], [21] return boxes for descriptive prompts, while SAM/SAM2 [22], [23] can refine a box into a mask. Broader multimodal in-context learning work, such as UniICL [24], studies how demonstrations condition unified multimodal models across understanding and generation tasks. ChatImage instead uses multimodal models as grounding components inside an interactive artifact construction loop. The goal is not to detect a predefined object category, but to align structured answer regions with their rendered positions in a newly generated image.
2.3 Interactive Visualization↩︎
Interactive image annotation [25], [26] and VQA [27]–[30] study fixed images rather than answer-generated images. VQA also treats the image as input to be queried, whereas ChatImage treats the image as the answer interface itself. Structured-output work such as chain-of-thought [31], ReAct [32], tool use [33], and RAG [34], [35] structures reasoning, not the output modality; the answer stays text. Efficiency-oriented agent and model-compression work, such as SPIKE’s selective long-horizon controller for game agents [36] and token-adaptive knowledge distillation for LLMs [37], addresses the cost of reasoning or model serving, which is complementary to the interaction-focused problem studied here. Mind-map and chart-generation methods [38]–[41] visualize structure but as static outputs without image-level region interaction or grounding. ChatImage combines these threads in an underexplored setting: an LLM answer rendered as a navigable, region-interactive visual image with vision-grounded hotspots.
3 Method↩︎
ChatImage converts a user’s question into an interactive visual image by separating content specification from spatial realization. The system first determines what the answer should contain, and only after image generation estimates where the corresponding content appears. This section describes the end-to-end pipeline (Section 3.1), the structured answer normalization and layout planning (Section 3.2), the two-pass vision alignment that grounds hotspots to real image content (Section 3.3), and the interactive hotspot layer and per-region follow-up threads (Section 3.4).
3.1 End-to-End Interaction Loop↩︎
Given a user question \(q\), ChatImage produces an interactive visual image \(\mathcal{I}\) with a set of clickable hotspots \(\mathcal{H}\). The pipeline has two passes. The first pass specifies the answer content and generates the image; the second pass localizes the rendered content and derives the clickable hotspot layer from the resulting evidence.
3.1.0.1 Pass 1: Structure, layout, and image generation.
Raw answer. Obtain a long-form LLM answer \(a\) to \(q\) from a text model.
Normalization. Normalize \(a\) into a structured visual specification \(\mathcal{S}\), a set of modules \(\{m_i\}\) and auxiliary modules, each with a title, detail text, region prompt, and metadata (Section 3.2).
Layout planning. Plan a layout \(\mathcal{L}\) that assigns each module a region \(r_i\) with normalized bounds in \([0,1]^4\).
Image prompt synthesis. Derive an image prompt from \(\mathcal{S}\) and \(\mathcal{L}\), encoding the visual style and the intended region labels and positions.
Image generation. Generate the image via a text-to-image model, obtaining \(\mathcal{I}\) and its real pixel dimensions \((W, H)\) parsed from the image header (not the request, since generators do not guarantee exact sizes).
3.1.0.2 Pass 2: Vision alignment and hotspot derivation.
Visual grounding. Send \(\mathcal{I}\), \((W,H)\), and the modules with their planned bounds to a visual grounding model, which returns a grounded box \(b_i\) for each rendered module (Section 3.3).
Mask refinement (optional). Refine each \(b_i\), when enabled, with the SAM3 adapter to obtain a pixel mask for tighter previews.
Hotspot derivation. Copy the grounded bounds into rectangular hotspot geometry and render transparent, clickable hotspots over \(\mathcal{I}\) (Section 3.4).
ChatImage does not assume the image generator follows the planned layout. The pipeline treats layout planning as a prior and visual grounding as the source of final hotspot geometry. This design avoids requiring precise spatial control from diffusion models, which current generators do not reliably provide [8], and ties clickable regions to rendered content instead of planned coordinates.
3.2 Structured Answer Normalization and Layout↩︎
3.2.0.1 Visual specification.
The raw answer \(a\) is parsed into a visual specification \(\mathcal{S}\) with a visual mode (infographic, map, scene, or poster), a summary, and a list of modules. Each module \(m_i\) carries: title (the hotspot label), detail (the explanation shown in the detail panel), imageText (short on-image caption), regionPrompt (a locator phrase for visual grounding), and metadata (regionKind, iconHint, maskPolicy, locator queries). Because LLMs may echo prompt scaffolding into user-facing text, a sanitization pass removes visual-system vocabulary, meta-instructions, and obvious prompt fragments before display. Short labels are used as fallbacks when a detail field becomes unusable. If no valid structure is returned, the reference implementation can instantiate deterministic specifications for common infographic, map, and scene topics, which supports controlled testing of the downstream interaction pipeline.
3.2.0.2 Layout planning.
A layout planner assigns each module a region \(r_i\) with normalized bounds \(\in [0,1]^4\) relative to the image canvas. Layouts are organized by family, including grid, flow (swimlane), compare, hub, timeline, matrix, and freeform layouts, selected by the visual mode and relation type. For maps, a semantic-slot catalog assigns regions by keyword matching (e.g., "entrance" maps to bottom-center, "core area" maps to center); for infographics, index-based templates place modules in flow or grid order. These planned bounds serve as priors for the grounding step and as a fallback when grounding fails, but they are not assumed to match the generated image exactly.
3.3 Two-Pass Vision Alignment↩︎
The central technical component is the alignment of structured regions to the actual generated image. ChatImage sends the image, its real dimensions, the modules, and their planned bounds to a vision backend that runs a grounding-and-repair loop:
3.3.0.1 Grounding model.
The grounding step is model-agnostic: any vision backend that can return a normalized bounding box for a locator phrase can be used. The reference implementation supports a dedicated referring-localization model (LocateAnything-3B [20]) and a general vision-language model used in structured grounding mode (MiMo-Vision [21]). For each module the backend receives the generated image plus locator phrases built from the module title, region prompt, and semantic hints. It returns candidate boxes in normalized coordinates. The system compares the candidates with the planned layout, selects the most plausible box, and may re-ground within a crop when the initial candidate is uncertain. This is a referring localization step: the system recovers the rendered location of an answer module rather than detecting a predefined object class.
3.3.0.2 Fallback hierarchy.
When the grounding backend rejects a module or is unavailable, ChatImage uses a graded fallback hierarchy: a general VLM-based grounding pass, local OCR, and finally the planned bounds. Each module is tagged with its alignment source (e.g., locateanything, mimo-vision, sam3-refined-planned, planned-fallback, vision-low-confidence) so the system can report alignment quality per region rather than as a single aggregate score.
3.3.0.3 Robustness to partial failure.
A single missing or low-confidence grounding should not invalidate the entire result. The system rejects groundings that resemble header strips or cross-panel artifacts (falling back to planned bounds for those) while keeping far-from-planned groundings that still localize the correct region (tagged low-confidence). Modules entirely absent from the grounding response retain their planned bounds and are flagged, rather than discarding the alignments of all correctly-grounded modules. Each grounded box is repaired for clickability by enforcing a minimum area and safe margin before becoming a hotspot.
3.3.0.4 SAM3 mask refinement (optional).
For tighter visual previews, an interactive box-prompted segmentation model in the SAM family [22], [23] takes each grounded box as a prompt and predicts a pixel mask, from which a tightened box and polygon are derived. The mask is used for preview rendering, including cutouts, organic feathering, and soft fades, while click hit-testing remains rectangular in the normal interaction loop. This design preserves stable interaction behavior even when a mask is visually useful but not sufficiently reliable to define the clickable shape.
3.4 Interactive Hotspot Layer and Follow-up Threads↩︎
3.4.0.1 Hotspot rendering.
Each hotspot is a transparent, absolutely-positioned button over the image at its grounded coordinates. In normal use, hotspots are fully invisible so the generated image remains visually unobstructed; a hover state provides a soft visual cue without altering the image. A calibration mode toggles visible outlines and per-hotspot labels for inspecting and correcting bounds.
3.4.0.2 Detail panels and previews.
Clicking a hotspot opens a centered detail panel showing the module’s title, sanitized detail text, a localized preview of the image region, and a follow-up input. The preview variant is chosen by a strategy module based on the hotspot shape and mask availability: cutout (checkerboard background), organic feathering, soft radial fade, or mask-based polygon crop.
3.4.0.3 Per-region follow-up threads.
Each hotspot owns an isolated follow-up thread. A follow-up question is sent to the text model with a local context: the original question, the raw answer, the clicked hotspot’s detail, and that thread’s history. The answer is scoped to the selected region rather than to the entire image. Threads are persisted alongside the ChatImage, enabling resumption across sessions.
3.5 Implementation↩︎
ChatImage is implemented as a model-agnostic web system. The browser frontend is written in vanilla JavaScript; the local backend runs on Node.js, proxies text, image, and vision services, and stores generated ChatImages, hotspot metadata, alignment traces, and follow-up threads in SQLite. Vision models are accessed through long-lived local adapters, which decouple browser interaction from model checkpoints and service credentials. The reference implementation supports both deterministic local execution for reproducible tests and configured external services for full-generation runs. The code and example artifacts are released with a project page for inspection and reproduction.1
4 Experiments↩︎
We evaluate ChatImage along two axes. First, we measure interaction-loop completion, defined as whether a benchmark prompt yields a complete ChatImage consisting of a generated image and a hotspot layer. Second, we measure hotspot evidence, defined as whether a clickable region is supported by an explicit grounding source and usable preview geometry or mask data. Completion is computed over valid full-generation runs; attempts terminated by upstream quota limits are treated as external run-limit events rather than as failures of the proposed interaction pipeline.
4.1 Benchmark↩︎
| Mode | #Q | Avg. hotspots | Scenario coverage | Alignment stress |
|---|---|---|---|---|
| Infographic | 15 | 5.0 | technical workflows, comparisons, decision aids, checklists, timelines | compact panels, dense labels, semantically similar regions |
| Map | 5 | 6.0 | tour routes, landmark clusters, spatial itineraries, scenic areas | route-like geometry, landmark placement, irregular spatial regions |
| Scene | 10 | 6.0 | food options, coffee shops, reading spaces, record stores, plant care | object localization, overlapping items, natural scene composition |
| Total | 30 | 5.5 | three visual modes | generation plus hotspot grounding |
4pt
The benchmark contains 30 questions designed to span distinct visual modes and content types, as summarized in Table 1. Map questions stress geographic placement and route-like scenic regions; scene questions stress object localization in less regular visual compositions; and infographic questions stress compact analytical panels and decision-support layouts. Each benchmark question is processed by the full ChatImage interaction pipeline using external text, image, and vision providers. The public showcase is a curated subset produced by the same workflow and retained only after strict visual-alignment screening.
Because the grounding step is model-agnostic (Section 3.3), different regions can be aligned by different signals: a dedicated localizer, a VLM grounding fallback, OCR, a SAM-refined planned box, or a planned fallback. The evaluation reports both completion and the source of each logged hotspot alignment. Representative qualitative results are shown in Figure 2.
4.2 Interaction-Loop Results↩︎
| Metric | Unit | Count | Rate |
|---|---|---|---|
| Hotspot | |||
| Strict-gate rejects \(\downarrow\) | Hotspot | 7/24 | 29.2% |
| SAM-complete masks \(\uparrow\) | Hotspot | 13/24 | 54.2% |
| SAM-incomplete masks \(\downarrow\) | Hotspot | 11/24 | 45.8% |
8pt
Figure 3 gives a compact visual summary of hotspot quality, alignment-source mix, and the SAM mask-completeness diagnostic.
Across valid full-generation runs that reached the generation stage, all 30 benchmark questions produced a complete ChatImage with a generated image and hotspot layer, yielding an interaction-loop completion rate of \(100.0\%\). Attempts interrupted by upstream quota limits are excluded from this denominator. This result should be interpreted as a pipeline completion diagnostic under valid provider responses; the more discriminative criterion is hotspot reliability.
Table 2 uses a shared 24-hotspot checked subset to report both accepted and rejected regions under the strict visual-alignment gate. The gate accepted 17 of 24 checked hotspots (\(70.8\%\)) and rejected 7 (\(29.2\%\)). The gate is intentionally conservative: an accepted hotspot must have a primary grounding signal and usable preview geometry or mask data, so weak or missing visual evidence is rejected rather than promoted. The SAM-complete row reports a stricter mask-shape diagnostic on the same checked subset. In total, 13 hotspots pass the SAM segmentation completeness check (\(54.2\%\)), while 11 remain incomplete (\(45.8\%\)) due to obvious holes or empty cavities.
4.3 Alignment Source Analysis↩︎
| Alignment source | Hotspots | Share |
|---|---|---|
| MiMo-Vision primary grounding | 50 | 55.6% |
| LocateAnything layout-guided grounding | 9 | 10.0% |
| LocateAnything crop grounding | 2 | 2.2% |
| SAM3-refined planned region | 16 | 17.8% |
| Planned fallback | 11 | 12.2% |
| Local OCR support | 2 | 2.2% |
5pt
Table 3 reports the grounding-source distribution for 90 logged hotspot alignments from the diagnostic source sample. Most logged hotspots are supplied by primary visual grounding: MiMo-Vision accounts for 50 hotspots (\(55.6\%\)), while LocateAnything contributes another 11 hotspots when layout-guided and crop-based calls are combined (\(12.2\%\)). SAM3-refined planned regions account for 16 hotspots (\(17.8\%\)), and planned fallback is used for 11 hotspots (\(12.2\%\)). This distribution shows that ChatImage preserves the evidence used to place each hotspot: high-confidence visual groundings are retained when available, plausible boxes are refined with masks, and planned bounds are used only when stronger evidence is unavailable.
5 Conclusion↩︎
We introduced ChatImage, a framework for transforming a long-form LLM answer into an interactive visual image whose regions support localized explanations and follow-up dialogue. The central design principle is to avoid treating pre-generation layout as final interaction geometry. Since an image generator is not guaranteed to realize the planned layout faithfully, ChatImage first renders the answer and then grounds interaction regions in the visible result. The implementation supports infographic, map, and scene modes, and the evaluation reports interaction-loop completion, hotspot source attribution, and a strict visual-alignment gate.
Task positioning. ChatImage frames interactive visual answering as an interaction and systems problem rather than as a new generative-model architecture. Its contribution is a compositional pattern: structure the answer, render it, ground the rendered regions, and attach local interaction to each region.
Use cases. The proposed format is most appropriate when an answer already has visual or spatial structure. Educational explanations can expose stages of a process as selectable regions; technical documentation can become a navigable artifact with localized follow-up; spatial guidance can represent landmarks and routes directly; and infographic answers can make comparisons scannable while preserving drill-down detail. These use cases depend on the same assumption: the visual artifact should be inspected as an interface, not only as an illustration. Improvements in text-to-image generation and visual grounding are therefore expected to directly improve the fidelity of ChatImage outputs.
Limitations and future work. The current system remains limited by the vision models on which alignment depends. Dense infographics, small objects, and complex maps can still produce weak or partial groundings. Click targets are also rectangular in the normal interaction loop; polygon-level hit testing would better represent irregular regions when masks are reliable enough to support interaction, not merely preview. Finally, file context is primarily text-oriented, leaving richer document parsing for future work. Promising directions include stronger OCR-aware grounding, mask-based interaction, automated visual QA for hotspot accuracy, shareable HTML exports, and synchronized cross-device history. The open-source release and benchmark are intended to support inspection, reproduction, and extension of this interaction format.
References↩︎
mimo-v2.5 chat-completion vision endpoint“MiMo-VL: Xiaomi MiMo vision-language model.”
Model release, https://github.com/XiaomiMiMo/MiMo-VL, 2025.