1 Introduction↩︎

Repository-level issue localization takes an issue description and a repository snapshot as input, and identifies the code locations relevant to resolving the issue [1]. For automated software engineering (SE) systems, localization is a prerequisite for issue resolving: errors at this step propagate into every downstream stage of patch generation. Real-world repository issues are natively multimodal; developers routinely attach screenshots, UI states, error dialogs, rendered outputs, and log excerpts alongside textual descriptions. Recent work has begun to extend SE benchmarks beyond text, e.g., SWE-bench Multimodal [2], which studies whether SE agents generalize to visually grounded tasks.

However, no existing benchmark isolates the role of visual evidence at the localization step. Text-only benchmarks such as SWE-bench and its multilingual extensions strip images entirely [3], [4]. SWE-bench Multimodal [2] retains images but evaluates end-to-end patch generation, so localization is folded into repair, and whether an image helped identify the right file or function cannot be read off the final pass/fail signal. Neither setting provides fine-grained localization gold, per-image annotations, or paired with-image versus without-image runs. These are the ingredients needed to treat visual evidence as a first-class input to repository-level localization.

We introduce MM-IssueLoc, a controlled benchmark and evaluation protocol for multimodal repository level issue localization. Each instance is grounded in a real GitHub issue, its linked pull request, and the repository snapshot before the fix. MM-IssueLoc contains 652 issue PR instances across 23 programming languages, with 1,050 issue images, file level gold labels for all instances, and function level gold labels for 343 instances. Each image is annotated with one of seven evidence categories, such as UI screenshot, error message, rendering result, code screenshot, log output, and data visualization, as well as a relevance score indicating whether the image is harmful, neutral, helpful, or vital for localization. The benchmark also includes 55 human reviewed harmful image instances, which serve as a robustness stress test for visually plausible but localization incorrect evidence.

MM-IssueLoc is designed as an evaluation instrument rather than as a leaderboard for a single model family. It supports four paired input modes: text-only, with-image, Visual Content Evidence (VCE), and VCE+image. These modes allow us to separate the effect of visual information from the effect of raw image conditioning. We evaluate both llm-based and retrieval-based localization systems, including AgentLess, LocAgent, OpenHands, Mini-SWE-Agent, BM25, SweRank, Qwen3-VL-Embedding, and our controlled multimodal retriever, MM-IssueLoc-VL-Embedding [1], [5]–[8]. We further compare representative systems across SWE-bench Lite, SWE-bench Verified, SWE-bench Multimodal, and MM-IssueLoc (Appendix 9.4). These cross-benchmark results show that strong localization performance on text-dominant SWE benchmarks does not transfer cleanly to multimodal issue localization, which highlights the need for a benchmark that controls visual evidence explicitly.

Our findings show that MM-IssueLoc exposes a capability gap that is difficult to observe in text dominant benchmarks. On MM-IssueLoc, the best file level result reaches 38.96 Acc@5, and the best function level agent result reaches 22.45 Acc@10. Controlled ablations further show that visual evidence can provide useful localization signal: removing images reduces MM-IssueLoc-VL-Embedding by 4.44 to 4.91 File@5 points, while VCE improves several agent framework pairs by converting visual content into structured text. These results motivate MM-IssueLoc as a controlled benchmark for evaluating when and how visual evidence contributes to repository level localization.

Our contributions are three-fold: 1) we introduce MM-IssueLoc, a controlled multimodal issue localization benchmark with 652 real issue PR instances, 1,050 images, seven evidence categories, four relevance levels and file/function level gold labels; 2) we provide a controlled evaluation protocol with text only, with image, VCE, and VCE+image modes for isolating visual evidence under strict Acc@K metrics; and 3) we evaluate representative agents and retrievers, introduce MM-IssueLoc-VL-Embedding as a controlled ablation model, and show that visual evidence offers useful but model dependent localization signal.

2 Related Work↩︎

2.0.0.1 Repository-level SE benchmarks.

SWE-bench establishe repository issue resolving as a scalable benchmark for automated SE systems [3]. Its later variants and related multilingual efforts broaden the issue-resolving ecosystem, including multilingual coverage in Multi-SWE-bench [4] and larger data-construction efforts such as SWE-smith [9]. SWE-bench Multimodal is the closest prior artifact to MM-IssueLoc: it asks whether SE agents generalize to visual, user-facing JavaScript tasks [2]. However, these benchmarks evaluate issue resolving as a whole. MM-IssueLoc instead isolates localization, adds per-image relevance labels and harmful-image controls, and reports both file-level and function-level gold across 23 languages.

2.0.0.2 Repository Search and Localization agents.

Recent SE agents provide different ways for models to inspect and navigate repositories. SWE-agent studies agent-computer interfaces for software engineering [10], Agentless argues that much of end-to-end repair can be decomposed into simpler localization and patching components [5], and OpenHands exposes a production-oriented SDK for software agents [6]. Beyond these general software-agent frameworks, recent work has increasingly focused on specialized repository search. LocAgent and CoSIL formulate localization as an agentic, iterative search process over repository structure, where the model repeatedly inspects files, functions, and contextual evidence before selecting suspicious locations [1], [11]. RepoSearcher and CodeScout instead emphasize learned repository search policies, including reinforcement-learning-based training, to improve how models navigate large codebases and identify relevant code regions [12], [13]. MM-IssueLoc uses these systems as baselines but evaluates only localization outputs. This design separates localization ability from patch-generation success and from framework-specific submission failures.

2.0.0.3 Code retrieval and embedding models.

Retrieval methods offer lower-latency alternatives to full agents. SWERank and SWERank+ formulate issue localization as code ranking, including function-level and multilingual settings [7], [14]. CodeXEmbed and large-scale code representation learning study general-purpose code embeddings [15], [16]. Multimodal embedding models such as Qwen3-VL-Embedding and jina-embeddings-v4 extend dense retrieval to visual and textual inputs [8], [17]. MM-IssueLoc-VL-Embedding builds on this line but serves a different purpose: it is a controlled retriever for testing image-presence, stage-curriculum, and hard-negative design under a fixed benchmark protocol. Its contrastive objective follows the general InfoNCE/CLIP-style retrieval paradigm [18], [19].

2.0.0.4 Multimodal code understanding.

Recent work has introduced visual evidence into software-engineering evaluation. MMCode studies visually rich programming problems [20], while FailureMem and long-horizon multimodal search examine visual evidence in broader repair or search pipelines [21], [22]. These studies motivate multimodal software reasoning, but they do not provide a controlled benchmark for repository-level issue localization. MM-IssueLoc complements this line by isolating localization from repair and by evaluating how visual evidence affects file- and function-level localization.

5pt

3 Benchmark Construction↩︎

MM-IssueLoc evaluates whether visual evidence in real issue reports helps localize code responsible for repository-level failures. To make this measurable, each instance is grounded in a GitHub issue, linked pull request, and fixed repository snapshot; supports paired evaluation with and without images; includes per-image evidence type and localization relevance annotations; and provides both file-level and function-level gold labels.

3.1 Task Definition↩︎

Given an issue report and a repository snapshot, the task is to rank code locations by their likelihood of being responsible for the reported failure. Each MM-IssueLoc instance is defined as \[\label{eq:instance-definition} x_i = (t_i, b_i, I_i, R_i@c_i).\tag{1}\] where \(t_i\) is the issue title, \(b_i\) is the issue body, \(I_i=\{i_{i,1},\ldots,i_{i,n}\}\) is a set of attached images, and \(R_i@c_i\) denotes the repository snapshot at the base commit of the linked pull request. A system returns either an ordered list of candidate files \[\label{eq:file-ranking} \hat{F}_i = [f_{i,1}, \ldots, f_{i,K}],\tag{2}\] for file-level localization, or an ordered list of qualified functions \[\label{eq:function-ranking} \hat{G}_i = [g_{i,1}, \ldots, g_{i,K}],\tag{3}\] for function-level localization.

The gold file set \(F_i^\star\) consists of files edited by the human-vetted fixing pull request. For instances with function-level annotations, the gold function set \(G_i^\star\) consists of edited functions that can be matched to functions in the pre-fix repository snapshot \(R_i@c_i\). Functions newly introduced by the pull request are excluded, since they are absent from the input repository and therefore cannot be localized. Function-level gold labels are provided only when tree-sitter extraction succeeds and at least one edited pre-existing function can be identified.

3.2 Data Collection and Annotation↩︎

MM-IssueLoc is built from public GitHub repositories with at least 50 stars, closed and merged pull requests, non-bot authorship, deduplication, GitHub API enrichment, issue-PR linking, pre/post file retrieval, and complete repository snapshots at the base commit. The released benchmark contains 652 instances, 1,050 localized issue images, and 650 repository snapshots. Its file-level view covers all 652 instances; its function-level view covers 343 instances.

Human annotation covers 450 instances by six annotators over approximately 20 hours. Each instance receives one image category among ui_screenshot, behavior_demo, error_message, rendering_result, code_screenshot, data_visualization, and log_output; a relevance score in \(\{-1,0,+1,+2\}\); and a difficulty bucket derived from the number of changed files. The AI-assisted extension adds 202 instances through a four-stage VLM gate: relevance scoring, category classification with confidence threshold, patch-alignment verification, and a second-model consistency check. All AI rows retain annotation_by=ai, which enables separate sanity checks.

This split is audited in the evaluation rather than treated as bookkeeping. On file Acc@5, OpenHands Claude scores 36.00 on the human subset and 37.13 on the AI subset, while AgentLess Claude scores 27.39 and 28.00. The ranking is stable, suggesting that the AI-assisted extension broadens coverage without dominating the benchmark behavior. further benchmark construction details are provided in Appendix 10.

3.3 Function-Level Gold and Harmful-Image Controls↩︎

Function-level labels are extracted by intersecting unified-diff line ranges with tree-sitter AST function spans. Functions newly introduced by the PR are excluded from both gold and candidate pools because the task is localization in the base repository, not prediction of new code. Languages with weak grammar coverage can still be evaluated at file level.

Naturally harmful images are rare. In the human pool, images that actively mislead localization occur at less than 1%. MM-IssueLoc therefore includes controlled harmful-image synthesis for robustness testing. The synthesis uses three strategies: same-repository image-text mismatch, TF-IDF-similar issues with disjoint edit files, and promotion of misleading comment-thread screenshots into the issue-body image slot. Each synthetic instance is scored along seven dimensions, including text sufficiency, surface relevance, image direction, direction deviation, misleading score, developer impact, and suggested relevance label, followed by human review.

3.4 Training Data and Controlled Retriever↩︎

The companion MM-IssueLoc training data is disjoint from the benchmark at the instance_id level and follows the right branch of the pipeline in 2. It contains 13,196 file-level rows over 7,664 instances and 13,769 function-level rows over 3,999 instances. Each anchor is multimodal and uses at most two images. Multi-positive expansion emits one row per gold file or function, aligning training with the multi-gold evaluation protocol. For each anchor, hard negatives [23] are mined from same-repository non-gold candidates ranked by Qwen3-VL-Embedding-8B, so the model contrasts the true target with plausible repository-local distractors. MM-IssueLoc-VL-Embedding is initialized from Qwen3-VL-Embedding-2B or 8B. The query contains issue text and images; candidates are file or function strings prefixed with File: or Function:. Let \(q_i\), \(p_i\), and \(n_{ij}\) denote the normalized query, positive, and hard-negative embeddings. We optimize \[\mathcal{L}_i = -\log \frac{\exp(q_i^\top p_i / \tau)}{\exp(q_i^\top p_i / \tau) + \sum_j \exp(q_i^\top n_{ij}/\tau)},\] with \(\tau=0.05\). Training follows a file-to-function curriculum: Stage 1 fine-tunes on file-level rows, and Stage 2 continues on function-level rows.

Overall, MM-IssueLoc contains 652 issue instances, 1,050 images, and covers 24 programming languages. All instances support file-level evaluation, and 343 also support function-level evaluation. Each instance records annotation provenance, image evidence category, image relevance score, and difficulty, enabling controlled analyses by language, visual evidence type, relevance, annotation source, and search-space size. 1 summarizes these benchmark statistics, and the released JSONL files contain the same fields used in evaluation, making the statistics directly reproducible.

4 Experimental Setup↩︎

4.1 Baselines↩︎

We evaluate two representative paradigms for issue localization: retrieval-based localization and llm-based localization. These baselines cover both direct relevance ranking and interactive repository exploration, allowing us to evaluate whether visual evidence improves localization under different levels of search and reasoning complexity.

Retrieval-based Methods. This paradigm directly ranks repository files or functions based on relevance score, including BM25 [24], bge-m3 [25], SWERank-small, SWERank-large [7], Qwen3-VL-Embedding-2B/8B [8], and MM-IssueLoc-VL-Embedding-2B/8B.

Llm-based methods explore the repository with tool use before returning localized code locations, including AgentLess [5], LocAgent [1], OpenHands [6], and Mini-SWE-Agent [10] with frontier and open-weight multimodal models. All methods are scored by the same evaluation harness. Agents are constrained to output strict JSON for file/function localization and are not allowed to generate patches or modify repositories.

For multimodal benchmarks, we evaluate four input modes. The original mode passes issue text and raw images. The text-only mode removes images. The +VCE mode replaces images with a structured visual text block. The +VCE+image mode passes both the structured text and raw images.

4.2 Visual Context Extraction (VCE)↩︎

VCE is a non-trainable image-to-text adapter. For each image, a VLM extracts structured fields such as OCR text, error signals, UI elements, user actions, code hints, saliency, confidence, and notes. The fields are rendered as a compact textual block and appended to the issue body. The text-only to +VCE comparison measures the value of structured visual content, while original to +VCE+image measures whether raw pixels add further signal. See Appendix 11 for the VCE workflow and case studies.

4.3 Evaluation Granularities and Metrics↩︎

The harness evaluates both file-level (\(n=652\)) and function-level (\(n=343\)) localization. Following prior localization work [1], we use strict Acc@K as the primary metric: \[\label{eq:acc-at-k} \mathrm{Acc@K} = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}\left[ G_i^\star \subseteq \operatorname{TopK}(r_i) \right].\tag{4}\] A prediction is successful only if all gold files or qualified functions appear in the top-\(K\) ranked list. This all-gold criterion is important for multi-location issues, where finding only one correct location may still fail to recover the full edit scope. Function-level results are reported at \(K\in\{5,10\}\), since strict Acc@1 and Acc@3 are necessarily zero whenever an instance contains more than one gold function. We also report MRR, Recall@K, and Hit@K as supporting metrics.

5 Results and Analysis↩︎

5.1 RQ1: How Well Do Current Systems Localize Multimodal Repository Issues?↩︎

RQ1 establishes the capability boundary of current systems. As shown in Table 2, multimodal repository-level issue localization remains far from solved. The strongest file-level result is obtained by OpenHands with GPT-5.2, which reaches 38.96 File@5, while OpenHands with Claude-Sonnet-4.6 achieves the best File@1 and File@3. However, even the best system still misses all correct files in the top five for more than 60% of instances. This suggests that current systems can provide useful candidates, but are not yet reliable enough for downstream repair-oriented workflows.

The results further reveal a clear paradigm difference. LLM-based agents are stronger at file-level localization, likely because they can inspect the repository and follow issue-level clues during exploration. Retrieval-based methods are more competitive at function-level localization: MM-IssueLoc-VL-Emb-8B achieves the best Func@10 score of 33.86. This advantage is partly structural. Retrievers rank pre-enumerated functions that are independently encoded, so their outputs are naturally aligned with the evaluation space. In contrast, LLM-based systems must generate function identifiers as text, where incomplete names, missing qualifiers, or invalid output formats can reduce exact-match scores.

Figure 3 further explains the aggregate results. First, performance collapses with difficulty. OpenHands GPT-5.2 drops from 83.10 Acc@10 on easy instances to 2.84 on hard instances, while MM-IssueLoc-VL-Emb-8B drops from 74.18 to 3.98, showing that multi-edit issues remain the main bottleneck. Second, language slices reveal non-uniform localization behavior. Both systems perform better on JavaScript and related frontend languages, where screenshots often provide direct UI or rendering evidence, but performance is weaker on languages such as C++ and Go, where the visual signal is less directly tied to localized code. Third, agent results are also shaped by harness reliability. With the same Claude-Sonnet-4.6 backend, OpenHands achieves a 98.6% valid submission rate and 37.42 File@5, whereas Mini-SWE-Agent achieves 47.1% and 18.71. Thus, RQ1 shows that current systems are limited not only by model capability, but also by difficulty, language properties, and final-output reliability.

4pt

5.2 RQ2: Is Visual Evidence Useful, and Do Systems Use It Reliably?↩︎

RQ2 separates visual evidence from visual use. MM-IssueLoc is designed for this distinction: each image is annotated with evidence category and relevance, and the harmful subset introduces visually plausible but localization-incorrect cues. Therefore, the no-image ablation evaluates whether systems exploit image-conditioned localization evidence, rather than whether images are merely attached to issue reports.

Cross-benchmark results further motivate this question. As reported in Appendix 9.4, current systems already achieve strong localization performance on text-dominant SWE benchmarks. For example, OpenHands reaches 94.53 File@5 on SWE-bench-Lite and 90.20 File@5 on SWE-bench-Verified. However, performance drops substantially on SWE-bench-MM, where the best File@5 is 43.14. This gap suggests that repository-level localization is relatively mature in text-only settings, but remains much less developed when issue evidence involves images. MM-IssueLoc fills this diagnostic gap by providing controlled image relevance, category, and harmful-image annotations.

The main results also indicate that MM-IssueLoc contains usable visual signal. In Table 2, text-only retrievers reach 19.83–27.82 Func@10, whereas MM-IssueLoc-VL-Embedding-8B achieves 33.86 Func@10. This shows that visual evidence can be converted into fine-grained localization signal when a retriever is trained for multimodal issue localization. Detailed retriever ablations are reported in Appendix 9.2.

Table 3 reports the controlled no-image ablation. We define \[\begin{align} \Delta_{\mathrm{img}}(m) ={}& \mathrm{Acc@5}_{\mathrm{file}} (m,\text{text-only}) \\ &- \mathrm{Acc@5}_{\mathrm{file}} (m,\text{text+image}), \end{align}\] so negative values indicate that removing images hurts. The results show uneven image use across systems. AgentLess GPT-5.2 benefits from images (\(\Delta_{\mathrm{img}}=-2.66\)), while OpenHands changes by less than one point. MM-IssueLoc-VL-Embedding is more image-sensitive: removing images reduces File@5 by 4.91 points for the 2B model and 4.44 points for the 8B model. Thus, visual evidence is useful, but current agent systems do not exploit it consistently.

Fine-grained slices explain why aggregate image gains are unstable. Relevance labels do not induce monotonic accuracy changes, since harmful images are not always recognized as misleading and some instances remain solvable from text alone. Category slices further show that images are most helpful when they are directly tied to observable software behavior, such as UI screenshots, behavior demos, rendering results, code screenshots, logs, or data visualizations. These signals often help identify relevant files, but they do not always map cleanly to edited functions. Additional relevance, category, difficulty, and annotation slices are reported in Appendix 9.1.

5.3 RQ3: Does Localization Depend on Image Content or Raw Image Format?↩︎

RQ3 diagnoses whether the benefit of visual evidence comes from image content or from the raw image format. We use Visual Content Evidence (VCE)(6) to convert each image into structured textual evidence. The +VCE setting removes raw images and provides only structured visual text, while the +VCE+image setting provides both VCE and the original image. Thus, the gain from the original setting to +VCE indicates whether image content is useful after being textualized, and the difference between +VCE and +VCE+image measures the residual value of raw pixels.

Table 4 reports the VCE diagnostic results using the same three model backends under AgentLess and LocAgent. VCE improves file-level localization in several settings. For AgentLess, Claude-Sonnet-4.6 improves from 25.15 original File@5 to 28.53 with +VCE, and GPT-5.2 improves from 28.15 to 29.07. For LocAgent, Qwen3.5-122B-A10B improves substantially from 25.70 to 35.04. These results show that images contain useful localization evidence and that such evidence can be consumed in structured textual form.

However, raw images do not consistently add value once VCE is provided. We define the raw-pixel residual as \[\rho_{\mathrm{pix}} = \mathrm{Acc@5}_{f}(\text{VCE+img}) - \mathrm{Acc@5}_{f}(\text{VCE}).\] For AgentLess, the residuals are small: \(+0.22\) for Claude-Sonnet-4.6, \(+0.03\) for GPT-5.2, and \(-2.15\) for Qwen3.5-122B-A10B. In contrast, LocAgent exhibits much larger variation: Claude-Sonnet-4.6 drops by 5.62 points, GPT-5.2 gains 4.74 points, and Qwen3.5-122B-A10B drops by 11.69 points. This indicates that VCE exposes useful image content, while the effect of raw pixels is mediated by the framework’s visual-input integration.

6 Discussion, Limitations, and Future Work↩︎

6.0.0.1 Discussion.

MM-IssueLoc follows prior issue-resolving benchmarks by using merged pull request diffs as localization gold labels. Although such labels are not minimal causal explanations, they provide a scalable and auditable proxy for code locations modified by human developers. We therefore use strict all-gold Acc@K to measure whether a method recovers the full edit scope, and provide function-level labels when tree-sitter can reliably map edits to pre-existing functions. We also release per-instance predictions to support reanalysis under alternative scoring rules.

6.0.0.2 Limitations.

MM-IssueLoc is designed for controlled diagnosis, and its results should be interpreted within this scope. The harmful-image subset is primarily rule-constructed and human-reviewed, so it serves as a robustness stress test rather than an estimate of real-world frequency. VCE is a diagnostic protocol, not an upper bound on visual reasoning, since it uses fixed extraction fields and a fixed downstream localization setting. Function-level labels may be noisy for language features that are difficult for tree-sitter to resolve, such as macros, templates, or generated code. Our retriever also uses at most two images per issue, leaving many-image issues as a future stress case. Finally, MM-IssueLoc evaluates localization only, rather than end-to-end software repair or downstream patch generation.

6.0.0.3 Future work.

Future work can extend MM-IssueLoc along three directions: collecting more naturally occurring misleading images and many-image issues, improving function-level annotation with language-specific parsers or hybrid static analysis, and studying whether better multimodal localization improves broader software-engineering workflows such as repository understanding, debugging, triage, and downstream repair.

7 Conclusion↩︎

MM-IssueLoc shows that multimodal repository-level issue localization is a distinct and under-evaluated capability. Cross-benchmark results indicate that strong localization performance on text-dominant SWE benchmarks is not sufficient evidence of multimodal issue localization ability. By pairing real issue-PR instances with file-level and function-level gold labels, image-category and relevance annotations, and controlled text-only, with-image, VCE, and VCE+image modes, MM-IssueLoc makes visual evidence an explicit evaluation variable. Our results show that visual evidence can provide useful localization signal, but current agents and retrievers use it unevenly. Overall, MM-IssueLoc provides a controlled basis for evaluating how visual evidence affects repository-level localization.

8 Training and Evaluation Setup↩︎

8.1 Training Hyperparameters↩︎

We report the training details of MM-IssueLoc-VL-Embedding in the multimodal setting, where each input contains the issue text and attached images. Table 5 summarizes the two-stage protocol and the function-only ablation. Stage 1 uses file-level supervision for coarse localization, while Stage 2 continues from the Stage-1 checkpoint with function-level supervision. The function-only ablation directly trains on function-level data to assess the effect of file-level initialization.

Table 6 lists the shared optimization settings. We use full fine-tuning with the vision encoder frozen to reduce training cost while preserving pretrained visual representations.

Table 7 reports the hardware and batch configurations. All experiments use four NVIDIA H200 GPUs. The effective batch size equals the per-GPU batch size multiplied by gradient accumulation steps and the number of GPUs.

8.2 Agent Evaluation Parameters↩︎

We configure each agent framework following the setup reported in its original paper, and restrict every framework to localization-only output on a shared evaluation harness.

OpenHands uses a CodeAct-style agent equipped with a terminal, a file editor, and an atomic finish(message=...) submission action. Each run is capped at 10 iterations, with a follow-up “produce final JSON now” nudge issued when no answer is emitted. Mini-SWE-Agent uses a bash-only interface, a 10-step limit, and a heredoc-to-file final submission protocol. Both frameworks share the same repository snapshots, scoring code, retry policy for transient failures, and localization-only output schema. The observed gap between OpenHands and Mini-SWE-Agent is therefore interpreted as a framework-level submission and tool-surface effect, rather than a pure model-capability difference.

9 Extended Results↩︎

9.1 Stratified Evaluation Slices↩︎

Table 8 reports the relevance slice used in RQ2. The non-monotonic pattern is visible across systems: vital images are not always easier than helpful images, and harmful images are not automatically detected as misleading.

5pt

Table 9 reports selected category slices. Text-rich categories such as error-message and code screenshots are easier to convert into localization cues than behavior demos or visualizations, but file-level and function-level trends can diverge.

5pt

Tables 10 and 11 give the language and annotation-provenance checks referenced in the main text. The human and AI-assisted subsets preserve the same broad ordering for representative agent baselines, while language slices reveal larger variation for retrievers.

3pt

9.2 Controlled Retriever Training Ablations↩︎

The main paper uses MM-IssueLoc-VL-Embedding as a controlled probe of image-conditioned retrieval. Figure 4 and Tables 12 and 13 provide the supporting training ablations. These results are not the central contribution of the benchmark, but they verify that the controlled retriever is sensitive to curriculum and image-conditioned training choices.

4pt

9.3 External Image Ablation on SWE-bench-MM↩︎

We additionally compare with-image and no-image localization on SWE-bench-MM. This experiment serves as an external sanity check for RQ2, rather than primary evidence for visual contribution, because SWE-bench-MM does not provide controlled image relevance, category, or harmful-image annotations.

Table 14 shows that image effects on SWE-bench-MM are also mixed. Mini-SWE-Agent with Claude-Sonnet-4.6 benefits substantially from images, while OpenHands changes only slightly. Some methods even perform better without images, suggesting that current systems do not consistently exploit the visual channel. This trend is consistent with MM-IssueLoc, but MM-IssueLoc remains the primary benchmark for RQ2 because it explicitly controls visual evidence.

4pt

9.4 Cross-benchmark Localization Results↩︎

We further evaluate representative systems on SWE-bench-Lite, SWE-bench-Verified, and SWE-bench-MM. This experiment is used to contextualize MM-IssueLoc rather than as primary evidence for RQ2, since only MM-IssueLoc provides controlled image relevance and category annotations. Since none of these three benchmarks ship with localization gold labels, we follow prior work [1] and derive file- and function-level localization ground truth from the repair patches released with each benchmark. Specifically, we take the files and functions modified by the provided fixing patch as the gold set, which allows us to score the localization outputs of each system under the same strict Acc@K protocol used on MM-IssueLoc.

Table 15 shows a clear gap between text-dominant and multimodal localization. On SWE-bench-Lite and SWE-bench-Verified, current agent systems already achieve strong localization performance. For example, OpenHands reaches 94.53 File@5 on SWE-bench-Lite and 90.20 File@5 on SWE-bench-Verified. However, the same family of systems drops substantially on SWE-bench-MM, where the best File@5 is 43.14. This contrast suggests that current systems are relatively mature for text-based SWE localization, but still struggle when localization requires interpreting multimodal issue evidence. Thus, SWE-bench-MM provides additional evidence that multimodal repository-level localization remains underdeveloped, while MM-IssueLoc offers the controlled annotations needed to diagnose this gap.

3.5pt

10 Benchmark Details↩︎

10.1 Released Views and Statistics↩︎

MM-IssueLoc contains 652 canonical instances. The file-level view contains all 652 rows, and the function-level view contains 343 rows with non-empty extracted edit functions. The annotation split is 450 human and 202 AI-assisted rows. The difficulty split is 214 easy, 263 medium, and 176 hard instances. The largest language groups are TypeScript (151), Python (126), JavaScript (120), C++ (45), Java (44), Go (33), C# (33), Rust (27), C (21), and PHP (20), with 23 languages in total.

10.2 Benchmark Examples↩︎

5 presents representative examples from MM-IssueLoc, organized by the image categories defined in 16. Each example illustrates the issue context, the associated visual evidence, and the corresponding gold annotations at both file and function levels.

10.3 AI Annotation Gate↩︎

The AI-assisted extension uses four stages. Stage A estimates relevance and admits only +1 or +2 rows for the main augmented subset. Stage B assigns one of seven image categories with confidence at least 0.6. Stage C judges whether the proposed patch plausibly addresses the issue. Stage D repeats relevance and category with a second VLM and accepts only if relevance differs by at most one level and category matches. The accepted rows are marked with annotation_by=ai.

10.4 Harmful-Image Synthesis↩︎

The harmful subset uses same-repository image-text mismatches, TF-IDF-similar issues with disjoint edit files, and comment-thread image promotion. Each candidate receives a seven-axis quality assessment: text sufficiency, surface relevance, image direction, direction deviation, misleading score, developer impact, and suggested label. Human review filters implausible or insufficiently misleading candidates.

11 Visual Content Evidence Extraction↩︎

11.1 VCE Mechanism↩︎

Visual Content Evidence (VCE) converts issue images into structured textual evidence before localization. Instead of directly using the raw image as model input, VCE extracts task-relevant visual content, such as error messages, UI states, code snippets, logs, rendering artifacts, and chart patterns, into a compact textual representation. This design provides a diagnostic side channel for separating the contribution of visual content from the effect of direct image conditioning. In our evaluation, VCE is used to test whether localization improvements come from information contained in the image or from the model’s ability to process the image modality itself.

11.2 VCE Extraction Examples↩︎

Figure 7 shows representative image-to-VCE extraction examples across different image categories. Each example contains the original visual evidence, the extracted VCE text, and its corresponding interpretation for file-level and function-level localization.

11.3 VCE Prompt↩︎

The full prompt template is provided in Figure 8.

12 Prompt Templates↩︎

12.1 AI-Driven Harmful Impact Prompt↩︎

The full prompt template is provided in Figure 9.

12.2 AgentLess Prompt↩︎

The full prompt template is provided in Figure 10.

12.3 LocAgent Prompt↩︎

The full prompt template is provided in Figure 11.

12.4 OpenHands Prompt↩︎

The full prompt template is provided in Figure 12.

12.5 Mini-SWE-Agent Prompt↩︎

The full prompt template is provided in Figure 13.

References↩︎

---

1. Equal contribution.↩︎

2. Corresponding author.↩︎