1 Introduction↩︎

Retrieval-augmented generation (RAG) enhances large language models (LLMs) by retrieving information relevant to a given query from external sources and conditioning generation on this retrieved evidence [1]. While RAG has proven effective in improving factual grounding, conventional approaches predominantly retrieve from unstructured textual corpora that are noisy, fragmented, and weakly connected. These limitations often hinder reliable evidence selection and multi-step reasoning, especially for complex queries. To address these issues, knowledge graph–based RAG (KG-RAG) has been proposed [2]. By retrieving information from structured knowledge graphs, KG-RAG enables more coherent, contextualized, and comprehensive knowledge access, leading to improved answer quality and faithfulness [3]. However, existing KG-RAG methods are largely confined to textual knowledge graphs, which substantially limits their applicability in real-world scenarios where critical information is conveyed through images, charts, tables, and other non-textual modalities.

Multimodal KG-RAG. To overcome this limitation, we explore the task, multimodal KG-RAG (MKG-RAG), which integrates structured knowledge across multiple modalities to support more expressive retrieval and reasoning. Formally, given a query \(q\), a multimodal knowledge graph \(\mathcal{G}\), a retriever \(\textsf{Retriever}\), and a multimodal large language model \(\textsf{MLLM}\), the goal of MKG-RAG is to generate a response \(\mathcal{R}\), i.e., \[\mathcal{R} = \textsf{MLLM}(q, \textsf{Retriever}(q, \mathcal{G})).\] The most closely related setting is multimodal RAG  [4], which follows the traditional RAG pipeline by retrieving multimodal content from unstructured external corpora (e.g., Wikipedia) rather than from a structured multimodal knowledge graph. To the best of our knowledge, no existing benchmark or prior work systematically studies retrieval and generation in the MKG-RAG setting.

Motivation and Challenges. Multimodal KG-RAG is benchmark-worthy because it evaluates a model’s ability to retrieve, align, and reason over structured multimodal knowledge, a critical capability for real-world systems that is not captured by existing RAG, KG-RAG, or multimodal benchmarks. This work aims to introduce the first benchmark for the MKG-RAG task, consisting of curated datasets and evaluation protocols that explicitly target retrieval quality and multimodal grounding. Designing such a benchmark, however, poses several non-trivial challenges:

• Heterogeneous Retrieval. Unlike conventional knowledge graphs, where both queries and target triplets are unimodal, multimodal knowledge graphs support heterogeneous modality combinations across queries and retrieval targets. In MKG-RAG, queries may be text-only, image-only, or multimodal, while relevant graph components may combine textual, visual, and numerical information in different ways. These combinations induce distinct retrieval behaviors and demand different modeling strategies. A meaningful MKG-RAG benchmark must therefore systematically cover diverse multimodal retrieval scenarios to reflect realistic usage and to enable fair comparison among retrievers.

• Multimodal Knowledge Graph–Query Alignment. In MKG-RAG, relevant knowledge may not be explicitly aligned with the query at the surface level, particularly when the query and target nodes differ in modality or abstraction. For example, a textual query may require grounding in visual evidence, while an image-based query may necessitate retrieving semantically related textual entities and relations. Addressing such cross-modal and cross-structural alignment requires joint reasoning over graph topology, semantic relationships, and modality-specific representations. Accordingly, a MKG-RAG benchmark must include queries that demand non-trivial alignment beyond shallow modality matching.

• Benchmark Utility. The primary goal of the benchmark is to evaluate both the effectiveness of multimodal retrieval and the grounding quality of multimodal LLMs. This requires careful construction to ensure that the retrieved knowledge is informative and well-aligned with downstream queries. Multimodal knowledge graphs often contain generic or weakly informative triplets, which can lead to ambiguous or poorly grounded questions and obscure the contribution of retrieval. A useful MKG-RAG benchmark must prioritize high-utility triplets and disentangle improvements due to effective retrieval from those arising from generative priors.

Our Solution. To address these challenges, we propose a cross-domain, retrieval-oriented MKG-RAG benchmark, named MKG-RAG-Bench, comprising two multimodal knowledge graphs and corresponding multimodal question-answering datasets. The benchmark is designed to enable comprehensive evaluation across both the retrieval and generation stages of multimodal KG-RAG. The benchmark construction pipeline is shown in Figure 1. Specifically, we use an LLM to select high-utility triplets and employ heuristic strategies to ensure strong alignment between the selected triplets and the constructed queries. The same LLM, with different prompts, is further leveraged to generate multimodal, knowledge-grounded questions across diverse retrieval scenarios, thereby systematically covering heterogeneous modality combinations. We conduct extensive experiments with representative retriever architectures across different multimodal settings, evaluating both effectiveness and efficiency throughout the retrieval and generation pipeline. While this work primarily focuses on evaluating MKG-RAG using training-free approaches, the benchmark is also applicable to training-based methods, including retriever training and MLLM fine-tuning.

Contributions. In summary, our main contributions are as follows:

• We identify a critical research gap in MKG-RAG, highlighting that retrieval, despite its central role, is underexplored and insufficiently evaluated in existing multimodal RAG research.

• We introduce the first, cross-domain, retrieval-oriented MKG-RAG benchmark comprising two multimodal knowledge graphs and corresponding question-answering datasets, enabling systematic evaluation of retrieval and downstream generation.

• We propose a benchmark curation pipeline based on LLMs that directly addresses key challenges in heterogeneous retrieval, multimodal alignment, and benchmark utility.

• We develop a unified evaluation framework and conduct comprehensive experiments across diverse multimodal settings, demonstrating the importance of effective multimodal retrieval and enabling future research on more powerful retrievers for MKG-RAG.

2 Preliminaries↩︎

2.1 Benchmark Comparisons↩︎

The most closely related line of work is multimodal RAG (M-RAG), which focuses on retrieving multimodal content, such as textual passages and images, from large unstructured corpora. Retrieved multimodal evidence can be directly incorporated into the input of multimodal large language models [6], [8], or reorganized into ad-hoc multimodal structures to further enhance generation [7]. However, M-RAG follows the conventional RAG paradigm and does not assume access to a structured multimodal knowledge graph. As a result, retrieval is performed over unstructured corpora, and any induced structure is transient and query-specific rather than explicitly represented in the knowledge source.

Another related line of work is knowledge graph–based RAG (KG-RAG), which aims to retrieve relevant triplets or subgraphs from a given knowledge graph to support grounded generation. Existing KG-RAG approaches [16], [17], however, primarily focus on textual knowledge graphs and do not explicitly model or retrieve multimodal information. Consequently, they are not designed to evaluate or support retrieval and reasoning over structured multimodal knowledge.

In contrast, MKG-RAG operates over pre-defined multimodal knowledge graphs, where entities, relations, and cross-modal connections are explicitly modeled. This fundamentally changes the retrieval problem: instead of selecting isolated multimodal documents, MKG-RAG requires identifying relevant subgraphs that integrate heterogeneous modalities through structured relations. Consequently, existing M-RAG and KG-RAG benchmarks are insufficient for evaluating MKG-RAG, as they do not capture multimodal graph-based retrieval or disentangle retrieval quality from downstream generation. Table 1 compares representative benchmarks across multiple perspectives, highlighting the distinctive characteristics of MKG-RAG.

2.2 Retrievers↩︎

Although the MKG-RAG task differs from conventional RAG, multimodal RAG, and KG-RAG, many existing retrieval techniques can be directly adapted to this setting. In this work, we categorize commonly used retrievers into four groups based on their underlying retrieval strategies:

• Text-only retrievers [18]–[20]. Text-only retrievers treat both queries and candidate triplets as plain text and rank candidates using textual similarity, such as sparse lexical matching or dense semantic embeddings, without incorporating visual information.

• Fusion-based multimodal retrievers [4], [21]. Fusion-based retrievers encode textual and visual information into a shared representation space and perform retrieval by measuring cross-modal similarity between multimodal queries and multimodal targets.

• Captioning-based retrievers [22], [23]. Captioning-based retrievers first convert visual content into textual descriptions using an external image captioning model, and then reduce multimodal retrieval to text-only retrieval over the generated captions.

• Reranking-based retrievers [24], [25]. Reranking approaches typically adopt a multi-stage pipeline in which a lightweight first-stage retriever filters a candidate set, followed by a more powerful reranker that performs fine-grained relevance estimation and final selection.

To ensure fair and controlled comparison, we evaluate all retrievers in a training-free setting by directly using their pretrained weights, without fine-tuning on the proposed benchmark.

2.3 Preliminary Analysis↩︎

A seemingly straightforward approach to constructing a multimodal knowledge graph RAG benchmark is to directly combine existing multimodal knowledge graphs, such as MMKG [26] and TIVA-KG [27], with modality-relevant downstream tasks, e.g., A-OKVQA [28] and K-VQA [29]. However, as discussed in Section 1, such a naive combination fails to adequately capture the requirements of the MKG-RAG task and often leads to severe misalignment between the knowledge source, retrieval objectives, and downstream evaluation. In particular, the knowledge required to answer a given query may not be present in the knowledge graph, while irrelevant retrieved content may dominate the model’s input, introducing noise and ultimately degrading performance.

To empirically illustrate this issue, we conduct a preliminary study using MedMKG [30], a multimodal medical knowledge graph constructed from MIMIC clinical data, to support medical visual question answering on two widely used benchmarks, VQA-RAD [31] and SLAKE [32]. Specifically, we augment multimodal large language models with retrieval from MedMKG and evaluate their performance under different retrieval strategies. The results are summarized in Figure 2.

Across all experimental settings, models augmented with multimodal retrieval consistently underperform their RAG-free counterparts. Although multimodal retrievers outperform text-only retrievers, they still fail to surpass generation without retrieval augmentation. These results indicate that retrieval from MedMKG does not effectively support the downstream visual question answering tasks. We attribute this behavior to two primary factors: (1) task-critical knowledge required by the MLLM may be absent from the multimodal knowledge graph, and (2) task-irrelevant knowledge present in the graph may be retrieved and injected into the model input, introducing substantial noise that interferes with generation.

Taken together, these observations reveal a fundamental limitation of directly combining existing multimodal knowledge graphs with downstream multimodal tasks for MKG-RAG. The resulting misalignment between knowledge sources, retrieval targets, and evaluation objectives undermines the effectiveness of retrieval augmentation and obscures the true capabilities of multimodal KG-RAG systems. This motivates the need for a dedicated MKG-RAG benchmark that explicitly aligns multimodal knowledge graphs, retrieval objectives, and downstream evaluation—precisely the goal of the benchmark proposed in this work.

3 Benchmark Construction↩︎

An effective benchmark should not only cover diverse data sources but also support rigorous evaluation across multiple stages of a task. In the proposed MKG-RAG benchmark, we construct datasets from two multimodal knowledge graphs spanning distinct domains: MarKG [33] for the general domain and MedMKG [30] for the medical domain². This cross-domain design allows us to evaluate the generality of multimodal KG-RAG methods under varying knowledge structures and modality distributions. To enable faithful evaluation of both retrieval and generation in multimodal KG-RAG, we design a principled data construction pipeline that filters out low-utility knowledge, generates structurally aligned and knowledge-grounded queries with exact supervision, and supports explicit evaluation of both retrieval and generation stages.

3.1 Dataset Curation↩︎

As shown in Figure 1, our construction pipeline consists of three stages: (1) LLM-based utility filtering, (2) controlled completeness simulation via triplet masking, and (3) hybrid question synthesis. Each stage is designed to address a specific challenge in benchmarking multimodal KG-RAG.

Step 1: LLM-based Utility Filtering. A central challenge in benchmarking multimodal KG-RAG is ensuring that retrieval contributes meaningful and well-grounded evidence to downstream tasks. Multimodal knowledge graphs often contain triplets that are overly generic, definitional, or weakly informative, from which derived queries may be ambiguous or insufficiently grounded, limiting their usefulness for evaluating retrieval.

Given a triplet, we prompt an LLM to assess its retrieval utility, i.e., whether it can support a clear and informative information-seeking query. Triplets deemed low-utility are those that tend to yield questions answerable without external evidence or that fail to convey a specific retrieval intent [9]. For example, the triplet (Sun, is_a, star) may lead to the question “What is the Sun?”, which neither requires retrieval nor benefits from structured knowledge. We remove such triplets from the benchmark to ensure that generated queries are well specified and grounded in retrievable knowledge. This filtering step produces a refined triplet set that better reflects realistic retrieval scenarios in which external knowledge access is necessary for accurate reasoning. The prompt templates and decision criteria are detailed in Appendix 7.

Step 2: Controlled Completeness Simulation. To construct queries that require knowledge retrieval while preserving unambiguous supervision, we simulate controlled incompleteness over the retained triplets. Rather than presenting complete knowledge graph facts, we mask one component of each triplet and treat the remaining components as observed context, thereby mimicking information-seeking queries.

For each triplet \((h, r, t)\), we construct two masked variants: (i) relation masking, where the relation \(r\) is masked and must be inferred from the head and tail, and (ii) tail masking, where the tail \(t\) is masked and must be inferred from the head and relation. We keep the head entity unchanged as a semantic anchor, which facilitates multimodal grounding when visual information is available and ensures that each query is uniquely supported by its originating knowledge.

We avoid masking the head entity, as doing so would require entity disambiguation beyond the scope of retrieval evaluation and could introduce additional ambiguity. This design allows us to focus on assessing retrievers’ ability to identify relevant relations and entities under realistic multimodal conditions. The resulting masked triplets serve as structurally grounded templates for query generation.

Step 3: Hybrid Question Synthesis. Masked triplets provide structured but non-linguistic representations and do not directly resemble natural user queries. To generate realistic questions while preserving exact alignment with the underlying knowledge, we again employ GPT-5 [34] to synthesize natural-language questions from masked triplets under two complementary modes.

Text-only question synthesis. For masked triplets derived from text-only knowledge, GPT-5 generates textual questions that explicitly correspond to the missing component. For example, from \((\textit{penicillin}, [\mathrm{\small Mask}], \textit{bacterial infection})\), the LLM generates: “What is the relationship between penicillin and bacterial infection?” This produces text-only queries whose answers are uniquely grounded in the masked triplet.

Image-grounded question synthesis. For masked triplets whose head entity is associated with an image, we provide the image to GPT-5 and require the head entity to be referenced as [Image]. This enforces explicit visual grounding and prevents trivial lexical matching on entity names. For example, \((\textit{Eiffel Tower}, \textit{located\_in}, [\mathrm{\small Mask}])\) yields the question: ” Which city is the landmark in the image located in?” To avoid shortcut retrieval caused by identical images appearing in both queries and knowledge graph nodes, we apply image augmentations to the question images in Appendix 8.

Across both modes, question synthesis preserves the semantics of the masked component while producing natural, modality-appropriate queries, enabling retrieval evaluation under both text-only and multimodal settings.

3.2 Downstream Task Data↩︎

To enable explicit and fine-grained evaluation of retrieval performance, we construct retrieval supervision directly from the aligned query–triplet pairs. For each synthesized question, the originating triplet is treated as the primary positive target. Building on the retrieval dataset, we construct downstream generation data in a supervision-preserving manner. Because each question is derived from a masked triplet, the masked component corresponds exactly to the missing information requested by the query. We therefore treat the masked entity or relation as the ground-truth answer. This process yields question–answer (QA) pairs that are guaranteed to be supported by the aligned multimodal knowledge graph. The resulting QA data includes both text-only and visually grounded instances, reflecting the mixed-modality nature of MKG-RAG. By construction, successful generation requires both correct retrieval and faithful grounding, enabling meaningful end-to-end evaluation. An example of our benchmark construction is provided in Appendix 9.

3.3 Dataset Statistics↩︎

After curating both retrieval and generation datasets from two domain-specific multimodal knowledge graphs, we split the datasets for each domain and each stage into train/validation/test sets with an \(8{:}1{:}1\) ratio. The split is performed by questions, so the same question is never shared across train/validation/test sets, while keeping the corpus fixed, following standard IR evaluation. This split supports not only comprehensive evaluation but also retriever training when needed. To reflect realistic MKG-RAG usage, each split contains a mixture of text-only and multimodal query–triplet pairs, rather than separating modalities into disjoint partitions. As a result, we obtain  MKG-RAG-Bench, a cross-domain multimodal benchmark that supports training and evaluation under diverse settings in practical MKG-RAG scenarios. Table 2 reports the key statistics of MKG-RAG-Bench, where MKG-RAG-Bench-G represents the data obtained from the general multimodal knowledge graph MarKG, and MKG-RAG-Bench-M is the dataset constructed based on the medical multimodal knowledge graph MedMKG.

A notable difference between the two subsets lies in the query–triplet cardinality. For MKG-RAG-Bench-G, the number of queries notably exceeds the number of triplets. In contrast, MKG-RAG-Bench-M contains substantially fewer queries than triplets. This gap is primarily driven by the structural properties of the underlying medical KG, which exhibits more frequent \(1{:}n\) patterns. Popular clinical entities (e.g., lung or pneumonia) often connect to many other concepts through the same relation, so a fixed incomplete form such as \((h,r,[Mask])\) may correspond to multiple valid tails. Consequently, many triplets share the same masked form and yield identical queries, resulting in fewer unique queries despite a larger triplet set.

By contrast, the general-domain KG is relatively sparse: relations between everyday entities are less densely connected and tend to produce more unique incomplete forms, leading to a higher diversity of queries per triplet.

4 Benchmark Evaluation↩︎

4.1 Evaluation Settings↩︎

We evaluate the proposed MKG-RAG-Bench on both retrieval and generation tasks. As our primary focus is on the training-free setting, all evaluations are conducted exclusively on the test splits reported in Table 2, without using any training or validation data. As described in Section 3.1 (Step 3), the benchmark includes two types of queries—text-only and multimodal—and the underlying multimodal knowledge graphs contain both text-only and multimodal triplets³. To comprehensively assess retrieval and generation performance under different modality combinations, we evaluate the following five settings, defined by the query modality and the scope of candidate triplets: (S1) all queries with all triplets, (S2) text-only queries with text-only triplets, (S3) text-only queries with all triplets, (S4) multimodal queries with multimodal triplets, and (S5) multimodal queries with all triplets.

Setting S1 provides an overall evaluation of different retrieval techniques under the most general and realistic scenario. Comparing S2 and S3 allows us to assess whether incorporating multimodal triplets benefits retrieval and generation for text-only queries. Similarly, comparing S4 and S5 evaluates the contribution of text-only triplets when handling multimodal queries. Together, these settings enable systematic analysis of how multimodal information influences retrieval and generation performance across different query types and knowledge graph configurations.

4.2 Retrieval Evaluation↩︎

Evaluation Configuration. The objective of retrieval evaluation is to assess the effectiveness of different retrieval techniques introduced in Section 2.2, including text-only retrievers [35]–[37], fusion-based multimodal retrievers [4], [21], captioning-based retrievers [22], [23], and reranking-based retrievers [24], [25]. In addition, we include a basic random retriever as a simple lower-bound baseline for the retrieval task. For fair comparison, all retrievers are implemented using a shared CLIP encoder [38]⁴ to obtain unified representations. Additionally, the captioning-based retrievers are implemented with a BLIP model [39]⁵. Both queries and candidate triplets are embedded into the same representation space, and candidates are ranked based on cosine similarity. We report standard retrieval metrics, including NDCG\(@K\), Precision\(@K\), and Recall\(@K\) in the main experiments. Implementation details are provided in Appendix 10.

Retrieval Results and Key Findings. We evaluate retrieval performance on both MKG-RAG-Bench-G and MKG-RAG-Bench-M under different query and triplet compositions. Detailed results are reported in Table ¿tbl:tab:markg95retrieval? and Table ¿tbl:tab:medmkg95retrieval?, with efficiency analysis provided in Appendix 11. Note that Others* means the baselines mentioned in the evaluation configuration. Rather than enumerating numerical comparisons, we summarize the results through a set of key findings that highlight the challenges and opportunities of multimodal KG-RAG retrieval.

Finding 1: A modality gap persists in retrieval. For text-only queries retrieving from text-only triplets, all retrievers exhibit nearly identical performance, as they effectively reduce to the same text-based retrieval pipeline. When the candidate space is expanded to include both text-only and multimodal triplets, performance changes only marginally, and top-ranked results remain dominated by text-only triplets. This observation suggests that multimodal triplets rarely serve as effective matches for purely textual queries and primarily increase the size of the candidate pool without contributing useful evidence. More broadly, this behavior reveals a persistent modality gap in the retrieval space: cross-modal matching between textual queries and multimodal knowledge does not emerge naturally from hybrid indexing alone. This finding indicates that simply incorporating multimodal knowledge into a KG is insufficient for enabling cross-modal retrieval, and that explicit mechanisms are required to bridge modalities when such interactions are desired.

Finding 2: Multimodal embedding is critical for visually grounded retrieval. Once queries require visual grounding, unimodal retrieval strategies become unreliable. Text-only retrievers degrade sharply because they lack the ability to incorporate image evidence. Captioning-based retrievers also underperform, as the captioning process introduces an information bottleneck: omissions or inaccuracies in generated captions directly propagate into retrieval errors. In contrast, embedding-based multimodal retrievers remain effective in this regime. Fusion-based methods benefit from jointly encoding visual and textual signals into a shared representation space, enabling more robust cross-modal alignment. Reranking-based approaches further improve performance by refining candidate sets using stronger multimodal matching. These results indicate that the central capability for effective MKG-RAG retrieval is not improved text ranking, but reliable multimodal representation learning that preserves and aligns visual evidence with KG semantics.

Finding 3: Retrieval difficulty and method preference are domain-dependent. Visually grounded retrieval is consistently more challenging on MKG-RAG-Bench-M than on MKG-RAG-Bench-G. A plausible explanation is that medical images tend to be visually homogeneous and require sensitivity to subtle patterns, while the associated terminology is dense and fine-grained. As a result, generic similarity signals are weaker and more easily confounded. This domain discrepancy also explains differences in method preference. On  MKG-RAG-Bench-G, fusion-based retrieval is often sufficient, as coarse semantic alignment yields strong performance. On  MKG-RAG-Bench-M, reranking-based approaches provide greater benefits, as refinement helps disambiguate subtle visual cues and specialized language. These observations highlight that retrieval strategies effective in general domains may not transfer directly to specialized domains without adaptation.

Implications for MKG-RAG retriever design. Together, these findings suggest several directions for future retriever development. First, bridging the modality gap requires explicit cross-modal alignment objectives or supervision, rather than relying on mixed-modality indexing to induce cross-modal matching implicitly. Second, multimodal retrieval should prioritize representation learning that preserves fine-grained visual information and aligns it directly with KG semantics, as lossy caption-based conversions consistently limit performance. Third, domain-aware refinement becomes increasingly important in settings with subtle visual distinctions and specialized terminology, motivating stronger reranking or verifier-style matching modules and the use of domain-adapted visual and textual encoders. Overall, effective MKG-RAG retrieval is likely to require the integration of explicit cross-modal alignment, robust multimodal embeddings, and domain-sensitive refinement.

4.3 Generation Evaluation↩︎

Evaluation Configuration. For generation evaluation, we include a generation-only baseline (i.e., RAG-free) that queries the LLM without retrieval augmentation, and evaluate RAG variants built on the same retrievers, modality settings, and implementations described in Section 4.2. For each query, we provide the top-\(K\) retrieved triplets as additional context to the generator, using \(K{=}5\) to balance evaluation fidelity with computational cost and token budget. We use GPT-5 [34] as the generation model. Generation is evaluated as an open-ended (visual) question answering task using Exact Match (EM), token-level F1, Contains\(@1\), and BLEU-1.

Generation Results and Key Findings. Table ¿tbl:tab:rag95em95f195contains95gm? reports generation performance on  MKG-RAG-Bench-G and  MKG-RAG-Bench-M under different query and knowledge modality combinations. Rather than enumerating individual scores, we summarize the results through several key findings that reveal how retrieval quality and modality interact to shape end-to-end MKG-RAG performance.

Finding 4: Retrieval provides genuine but uneven utility for generation. Compared to the generation-only baseline, retrieval-augmented generation yields consistent improvements across most metrics and settings, confirming that retrieved multimodal knowledge can be effectively consumed by the generator. Notably, even Random retrieval occasionally improves performance, indicating that when relevant triplets appear in the context, the generator can exploit them. This serves as a sanity check for the alignment between constructed knowledge, queries, and answers, and validates the overall quality of the benchmark.

Finding 5: Generation gains are substantially weaker in multimodal regimes. Retrieval augmentation delivers the largest gains when both queries and supporting knowledge are textual. In contrast, for multimodal queries, improvements are often marginal unless a strong multimodal retriever is used, and even then absolute gains remain limited—particularly on  MKG-RAG-Bench-M. These results indicate that end-to-end generation is highly sensitive to whether retrieval can reliably supply visually grounded evidence. Simply incorporating multimodal knowledge into the pipeline does not automatically translate into proportional generation improvements, revealing a key bottleneck in current MKG-RAG systems.

Finding 6: Generation performance closely tracks retrieval quality. Generation trends closely mirror retrieval behavior. Text-only and captioning-based retrieval pipelines provide limited gains for multimodal queries because they either ignore visual information or compress it through lossy captions, constraining evidence quality before generation begins. In contrast, embedding-based multimodal retrievers yield the most reliable improvements in multimodal settings. Fusion-based methods are generally more effective on  MKG-RAG-Bench-G, while reranking-based approaches offer additional benefits on MKG-RAG-Bench-M, where subtle visual distinctions and specialized terminology demand stronger refinement. This alignment highlights retrieval quality as a primary driver of end-to-end MKG-RAG generation performance.

Implications: toward graph-aware multimodal generation. The comparatively modest gains in multimodal settings suggest that current multimodal retrievers, largely designed for unstructured corpus retrieval, do not fully exploit the structure and semantics of MKG. Improving MKG-RAG generation, therefore, likely requires retrieval methods that are explicitly graph-aware, such as relation-sensitive matching, neighborhood- or path-based evidence aggregation, and graph-constrained candidate selection to reduce distractors. Beyond retrieval, there is also headroom in how evidence is presented to the generator, including improved triplet selection, ordering, and representation formats that better convey structured graph information. Overall, these findings suggest that progress in MKG-RAG will depend less on scaling generic multimodal retrieval and more on developing graph-aware, domain-sensitive retrieval and evidence integration strategies.

4.4 Ablation on LLM-based Utility Filtering↩︎

A key design choice in MKG-RAG-Bench is to filter out low-utility triplets that tend to produce ambiguous or weakly grounded queries. To validate this choice, we conduct an ablation study comparing questions derived from filtered triplets and those derived from the remaining triplets retained in the benchmark. We randomly sample 150 multimodal questions from each group and evaluate MKG-RAG performance under Setting (S4) using our strongest Fusion-based retriever, with both RAG-free and retrieval-augmented generation. Results are reported in Table 3. Across all metrics, questions derived from filtered triplets consistently yield lower performance than those derived from remaining triplets, even without retrieval augmentation, indicating that such questions are intrinsically difficult and poorly specified. Although retrieval improves performance in both cases, gains on filtered-triplet questions are substantially smaller, suggesting that retrieval cannot compensate for unclear or weakly grounded information needs. These results demonstrate that low-utility triplets confound reliable MKG-RAG evaluation, and therefore validate the necessity of LLM-based utility filtering to ensure clear, grounded queries and faithful assessment of retrieval and generation performance.

4.5 Discussion on LLM Dependency↩︎

To evaluate whether the pipeline depends heavily on a specific LLM, we conduct a cross-LLM consistency analysis. Besides the GPT-based construction pipeline, we apply Qwen3.5 [40] and Gemini 2.5 Flash [41] to a 300-sample subset, including 150 text-only samples and 150 multimodal samples. We compare their outputs with GPT-based results using filtering alignment and question-level semantic similarity measured by BERTScore. As shown in Table 4, both alternative LLMs achieve high filtering alignment with GPT-based outputs, with Qwen3.5 reaching 99.7% and Gemini 2.5 Flash reaching 97.0%. The generated questions also show strong semantic consistency, with a BERTScore of 0.8818 for both models. These results indicate that the filtering and synthesis process is robust to the choice of LLM backbone.

We further conduct a human evaluation to assess the reliability of the filtering and question synthesis results. Human annotators reach 88% agreement with the filtering decisions, and the generated questions receive an average quality and diversity score of 3.97 out of 5. Together, the cross-LLM and human evaluation results show that the benchmark construction process maintains stable behavior across different LLMs and introduces limited model-specific bias in practice.

5 Conclusion↩︎

This work demonstrates that progress in MKG-RAG is increasingly constrained by retrieval, a component that has received limited attention largely due to the lack of suitable benchmarks. To address this gap, we introduce MKG-RAG-Bench, a cross-domain benchmark explicitly designed to evaluate retrieval in MKG-RAG, comprising two multimodal knowledge graphs and carefully aligned question-answering datasets. MKG-RAG-Bench is constructed through an LLM-based curation pipeline that filters low-utility knowledge and systematically covers diverse modality configurations, enabling controlled and realistic retrieval evaluation. Extensive experiments across representative retriever families reveal that effectively leveraging multimodal signals during retrieval remains both challenging and critical for end-to-end performance. By isolating and diagnosing these challenges, MKG-RAG-Bench provides a principled foundation for developing, evaluating, and advancing more effective retrievers for MKG-RAG. We hope this benchmark will catalyze future research on graph-aware, domain-adaptive multimodal retrieval methods and more effective integration of structured knowledge into multimodal generation.

Acknowledgements↩︎

This research was partially supported by a 2025/2026 Rising Researcher Grant from Penn State’s Institute for Computational & Data Sciences (RRID:SCR_025154) and the National Science Foundation under Grant No. 2333790 and 2238275.

None

6 Source Knowledge Graphs↩︎

We use the following two knowledge graphs:

• MarKG. MarKG [33] is a multimodal knowledge graph constructed to support multimodal analogical reasoning, where the entities involved in an analogy (e.g., head/tail/query entities) may come from different modalities. It is released together with the Multimodal Analogical Reasoning dataset (MARS) and is built by expanding and annotating seed entities and relations from E-KAR [42] and BATS [43]. In particular, MarKG links external entities to Wikidata and associates entities with images collected from LAION-5B [44]. Overall, MarKG contains 11,292 entities and 192 relation types, forming 34,420 triplets, with 76,424 linked images, providing multimodal grounding for general-domain analogy.

• MedMKG. For the multimodal knowledge graph, we use MedMKG [30]. MedMKG is a multimodal knowledge graph that fuses rich imaging data from MIMIC-CXR [45] with structured clinical knowledge from the Unified Medical Language System (UMLS) [46], using both rule-based tools and large language models for precise concept extraction and relationship modeling. To enhance graph quality and compactness, Neighbor-aware Filtering (NaF), a filtering algorithm specifically designed for multimodal knowledge graphs, is applied to MedMKG. MedMKG comprises 53,587 edges, 4,868 images, and 262 relations over 3,148 concepts.

7 Prompts for Data Construction↩︎

During MKG-RAG-Bench construction, we prompt GPT-5 twice: first to validate knowledge utility, and then to translate triplets into natural-language queries. The corresponding prompts are shown in Figure 3.

8 Image Augmentation for Visual Questions↩︎

To prevent exact image duplication from serving as a shortcut in multimodal retrieval, we apply image augmentation to the images used in visual questions. For each question image, we randomly apply one augmentation from the following set: swapping to another image in the knowledge graph that refers to the same entity, random cropping, rotation, or color jitter. These augmentations ensure that the image in the visual question differs from the image associated with the target triplet, eliminating trivial matches and better simulating realistic retrieval settings where an exact duplicate of the query image is unlikely to appear in the knowledge graph.

9 Example of MKG-RAG-Bench Construction↩︎

We use two examples to illustrate how the proposed pipeline generates benchmark data with high-utility triplets.

• Text-only Query. We take (Antigua and Barbuda, member_of, [MASK]) as example. The pipeline will generate a query first, which is “Of which global or intergovernmental bodies is Antigua and Barbuda a member?” For the retrieval task, the ground truth data include several high-utility triples, such as {(Antigua and Barbuda, member_of, Commonwealth of Nations), (Antigua and Barbuda, member_of, United Nations), (Antigua and Barbuda, member_of, World Trade Organization). For the generation task, the answers are {Commonwealth of Nations, United Nations, World Trade Organization}.

• Multimodal Query. If there is a high-utility multimodal triplet (People’s Republic of China , railway_traffic_side, left), the entity “left” is then masked, leading to the generation of the multimodal query: “On which side do trains operate in the subject’s country?” Note that the image entities can be different. For the retrieval task, the ground triplet is the original high-utility triplet, and for the generation task, the answer is the mask “left”.

10 Details about Retriever Implementation↩︎

The proposed MKG-RAG-Bench is a general benchmark, and any retrievers can be used. However, in the current version, we implement the following retrievers in the experiments.

For text-only queries and triplets, all retrievers that share the same CLIP implementation produce identical embeddings by encoding text solely with CLIP’s text encoder. As a result, they achieve the same performance on text-only query–text-only triplet retrieval due to this homogeneous encoding. For multimodal retrieval, the text-only retriever ignores visual signals and encodes only the textual entity associated with the image. The captioning-based retriever first uses BLIP to convert the image into a textual description, and then encodes the resulting text with CLIP’s text encoder.

The remaining two retrievers adopt more advanced multimodal strategies. We implement the fusion retriever using a late-fusion design, where the visual embedding from CLIP’s image encoder is averaged with the text embedding of the textual part of each multimodal triplet. The reranking retriever performs a two-stage procedure: it first retrieves candidates using image-embedding similarity, and then reranks the shortlisted triplets using text-embedding similarity.

11 Retrieval Efficiency Analysis↩︎

An additional study analyzes the retrieval efficiency of the baseline retrievers. We report Queries per Second (QPS) under an online serving setting where all embeddings are cached and the retrievers are warm-started. The results are shown in Table 5. Overall, the findings highlight a clear efficiency–effectiveness trade-off: in the multimodal setting, fusion-based and reranking-based retrievers exhibit roughly an order-of-magnitude lower throughput (about 0.1\(\times\) QPS) than the remaining baselines. By contrast, captioning-based retrieval benefits substantially from caching and achieves competitive throughput, approaching that of the text-only retriever. Taken together, this study quantifies the efficiency cost of stronger multimodal modeling and provides practical guidance for deploying MKG-RAG systems under real-time constraints.

References↩︎

---

1. The benchmark can be accessed via: https://github.com/XiaochenWang-PSU/MKG-RAG-Bench.↩︎

2. The details of these two multimodal knowledge graphs are provided in Appendix 6.↩︎

3. Note that MarKG and MedMKG do not include triplets of the form (image, relation, image).↩︎

4. https://huggingface.co/sentence-transformers/clip-ViT-B-32↩︎

5. https://huggingface.co/Salesforce/blip-image-captioning-base↩︎