1 Introduction↩︎

Visual Question Answering (VQA) is a core multi-modal task in machine learning consisting of answering text-based questions about an image. Generally speaking, VQA tasks can be classified as extractive and abstractive Question Answering (QA). Extractive QA consists of answering the question by extracting a subset of tokens within the image. On the other hand, abstractive QA aims to answer the question based on the image content, but the answer does not need to be extracted from it.

There are different VQA tasks depending on the application scenario, such as maps [1], daily photos [2], or scientific papers [3]. Among all these tasks, Document VQA (DocVQA), which aims to answer questions based on document images, is much more challenging. It requires detecting the layout objects [4]–[6] and extracting the relationships between them to extract relevant information to answer the question correctly.

This complexity is even greater when different document domains are taken into account. Different document domains differ greatly from each other. For example, scientific papers usually present a structured layout with one or two columns, same fonts for all the objects, and the relations between objects are simple and much more direct. However, other document types, like presentation slides, do not have a structured layout and the relations between objects are less structured in order to be visually appealing.

For all these reasons, Document Understanding (DU) models need to be able to adapt to different domains, not only based on the document image but also on the internal relationships between objects in different document domains. This adaptation should be present during training, but also during inference, when models may be given documents from scarce domains that have never or barely been seen during training in zero- and few-shot settings.

In recent years, Vision-Language Models (VLMs) have shown remarkable performance in many very challenging image understanding tasks [7], including DocVQA [8], where they are becoming the standard. One of their main strengths is their ability to align visual and textual representations to produce better and more reliable answers. However, VLMs tend to rely too much on visual modality, which can lead to errors, especially when dealing with complex layouts such as slides or infographics, where VLMs cannot fully understand the visual relations between objects [9].

Despite the advances made in the field of DocVQA, real-world deployment requires more robustness across different document domains, which current VLMs still lack. However, there are few comprehensive studies on how pretrained VLMs handle different VQA domains and, more precisely, how they handle different document domains in DocVQA. Furthermore, finetuning VLMs in domain-specific DocVQA datasets can affect performance on other datasets, so it is relevant to study the difference in VLMs in cross-domain evaluations, both in a zero-shot setting where finetuned VLMs do not see any target domain sample during training, and in a few-shot setting, where a small number of samples from the target domain are seen during training to improve generalization [10].

To address this gap, we present this study that assesses the performance of VLMs in DocVQA in general document domains. We selected 8 open-source VLMs and three DocVQA datasets that cover domains with several differences in the question types, the image content and granularity, and the layout structure. We selected VLMs with different numbers of parameters to also assess the effect of parameter scale on DocVQA. We evaluated the models in a zero-shot setting in all the datasets. We also finetune each model on every dataset and provide an in-domain and a cross-domain evaluation, highlighting the difference in performance between settings. Finally, we study the effect of few-shot learning in different domains and how the performance of the models changes when dealing with a small number of training samples from the target domain.

The purpose of this work is to provide researchers and practitioners with a comparative study of small-scale VLMs on different DocVQA tasks, highlighting their performance in different settings and testing their adaptability using usual domain adaptation [11] and knowledge transfer techniques. This study serves as an initial comprehensive evaluation framework for new research and methodologies on automatic domain adaptation and transferability.

The main contributions of this work are:

• We provide a comprehensive evaluation of the capabilities of 8 pretrained open-source VLMs on DocVQA in 3 different document domains, covering industrial documents of different types, infographics, and presentation slides. We study which of the models performs better in every domain and how the number of parameters affects the performance.

• We separately finetune each model on all the selected domains. We provide a comprehensive in-domain evaluation of each finetuned model, as well as a cross-domain evaluation of all the models in all domains. We show the difference in performance in all settings compared with the zero-shot evaluation of the models.

• We provide an evaluation of the effect of few-shot learning on finetuned VLMs in a cross-domain setting. We study how the number of cross-domain samples affects the VLM performance and the difference with a fully supervised finetuning on the target domain.

The rest of the work is structured as follows: Section 2 provides a study of previous works on VLMs for DU, DocVQA, and domain adaptation techniques in DU. In Section 3 we describe the models and datasets used for our experiments, and in Section 4 we report the results of the zero-shot, supervised finetuning, and few-shot experiments and discuss them. Finally, the conclusions and future research directions emerging from this work are drawn in Section 5.

2 Related Works↩︎

2.1 Vision Language Models in Document Understanding↩︎

Document understanding (DU) deals with the interpretation and comprehension of digital or scanned documents, including forms, tables, academic research papers, or slides. There are two broad categories of DU techniques. The first category uses Optical Character Recognition (OCR) annotations provided by external tools [12], [13]. The second category does not use external OCR tools [14].

In DU tasks, VLMs have shown remarkable performance using their own visual encoders enhanced with external OCR tools. LayoutLLM [15] leverages layout instruction tuning in multiple levels of granularity during pretraining and supervised finetuning stages using pretrained weights from LayoutLMv3 [16]. Other works combine layout information with a contrastive learning environment to enhance visual understanding of VLMs [17]. Although most of the works leverage the layout information for DU with VLMs, some works have also used VLMs to extract key information from historical documents without OCR tools [18].

2.2 Document Visual Question Answering↩︎

With the development of more sophisticated techniques, VQA has expanded to different and more complex domains such as general document images. DocVQA has received great attention from researchers and practitioners since its establishment. One of the main reasons is that the datasets for DocVQA are quite large and cover a wide range of document domains and question types [19], [20]. Another reason is a very active DocVQA research community, which has led to the organization of very challenging competitions such as the ICDAR 2021 competition in DocVQA [21] or the ICDAR 2023 competition in VQA in business documents [8]. In this environment, several techniques have shown impressive performance in DocVQA tasks. For example, ORCA [22] uses a specialized multi-agent framework in several stages to exploit the full capabilities of VLMs in DocVQA in a collaborative fashion. Other works perform a contrastive and generative pretraining to align visual encoders and LLMs in general NLP tasks, which have concretely shown impressive performance on DocVQA tasks [23].

2.3 Domain Adaptation Techniques for Document Understanding↩︎

In real-world applications, it is necessary for deployable models to deal with samples from domains that the model has never seen during training [24]. Domain adaptation ideally reduces this gap by increasing the adaptability of models to new samples while maintaining the same performance on previous samples. In DU tasks, several domain adaptation techniques have been proposed. One of the most prominent approaches is few-shot learning, especially in low-resource domains, where models are required to learn informative representations on target data with few samples for training. For example, FS-DAG [25] combines different few-shot learning strategies, positional embeddings, and Graph Neural Networks [6] for Key Information Extraction. Another well-established domain adaptation technique is the generation of synthetic data [26], where, given a small source dataset, new data for the same task are generated based on the distribution of the original dataset. In DU, synthetic data have been used for a wide range of sub-tasks such as Financial QA [27], generalistic QA [28] or Table QA [29].

3 Experimental Setup↩︎

3.1 VLM Selection↩︎

We selected 8 different VLMs from 3 families of models for the experiments in this study: Qwen3.5, Qwen3-VL, and Gemma 3.

Alibaba’s Qwen family ¹ is an open-weight family of Large Multimodal Language Models (LMLMs) that are showing excellent results on different DU tasks, like ICDAR’s 2021 and 2023 competitions on DocVQA [8], [21].

The Qwen3-VL sub-family [30] is a collection of VLMs designed to handle dynamic-resolution visual input [31] and process combinations of text and images within a single prompt. The main strengths of the Qwen3-VL family is its scalable performance, showing robust performance in VQA tasks ranging from its tiny 2B version up to its bigger versions, and its long-context multimodality. Due to hardware constraints, especially when finetuning the models, we selected the 2B, 4B, and 8B versions of the Qwen3-VL family for our experiments.

After developing the Qwen3-VL family, Alibaba released a better family of omnimodal models and VLMs, the Qwen3.5 family [32]. Although originally designed for agentic applications, they also show impressive performance, as usual, for various tasks. One of their main strengths is their efficiency and speed, as they use a Mixture of Experts (MoE) [33] and a highly efficient pipeline that cuts the memory overhead without degrading the models performance. Another major advantage of this family is that it is heavily biased towards analytical and structural tasks due to its agentic focus. Although this feature makes them worse in conversational tasks, it makes them especially strong in tasks like VQA, where the models are required to just output the answer of the given question. In this case, to show the different performance between large scale and smaller models, we selected the 0.8B, 2B, 4B and 9B versions of this family for our experiments.

Google DeepMind’s Gemma 3 ² is a family of lightweight open multimodal models specially designed for visual and textual understanding and optimized for developer accesibility. Their main strength is that they are specially receptive to parameter-efficient finetuning, which makes them excellent at adapting to new tasks never seen during their pretraining, as is the case of DocVQA. However, they tend to hallucinate when adapting to new tasks when the knowledge is not effectively embedded during finetuning. Due to hardware limitations, we only selected the Gemma 3 4B version of this family for our experiments.

All models were implemented and finetuned using the Unsloth library ³, which is a library for efficient inference and finetuning of LLMs and VLMs.

3.2 Datasets↩︎

To reflect the real-world diversity of documents, we selected three datasets covering well-differentiated domains: 1) Single Page DocVQA (SP-DocVQA), which covers industrial documents of various types; 2) InfographicsVQA, made up of infographics; and 3) SlideVQA, a dataset comprising presentation slides.

SP-DocVQA and InfographicsVQA are single-page DocVQA datasets, that is, the input to the models is a unique document image and question. The answer of that question must be present in the document image or be extracted from it. In contrast, SlideVQA is designed as a multi-page DocVQA dataset, that is, the models are given various document images, in this case 20 presentation slides, and a question and models are required to answer the given question by retrieving the necessary images from the images and answering the question based on the context of multiple images. To keep a fair comparison between datasets, we converted SlideVQA dataset into a single page DocVQA dataset by selecting a subset of it, as we will explain in subsequent sections.

3.2.1 SP-DocVQA:↩︎

Presented in 2021, the SP-DocVQA dataset [19] is one of the most representative DocVQA datasets in the literature. It comprises 50K Question-Answer pairs over more than 12K document images. The document images were collected from documents in the UCSF Industry Documents Library ⁴. The documents in the dataset were selected to cover \(5\) industries: Tobacco, Food, Drug, Fossil Fuel, and Chemical. The selected documents range from 1900 to 2018.

SP-DocVQA covers a wide range of document types and layouts, as can be seen in Fig. 1. Most of the documents are letters, forms, and reports, representing approximately 50% of the documents in the dataset, but there are other types of document, covering an even more general domain.

3.2.2 InfographicsVQA:↩︎

InfographicsVQA [34] was presented in 2022 to benchmark the progress of VQA research combining vision, language, and document understanding. It comprises approximately 30K question-answer pairs for more than 5K document images. These document images come from infographics collected from the Internet and later curated. The questions in InfographicsVQA are divided into 4 categories: 1) Image-span, where the answer to the question is extracted from a token or a sequence or tokens present in the image; 2) Question-span, where the answer is inside the question itself; 3) Multi-span, where the answer is present in the image in multiple spans; and 4) Non-extractive, where the answers are numerical and not present in the document image itself.

Contrary to SP-DocVQA, where all questions are extractive, InfographicsVQA presents a more complex task as there are non-extractive questions, which force VQA models to reason over the image and not only extract tokens from it. Another major difference from SP-DocVQA is the kind of layouts present in InfographicsVQA. The infographics show more complex layouts, as shown in Fig. 2, where the reading order and the relationship between objects are not easily inferred and may be subjective. This adds another layer of complexity to the VQA task in this domain.

3.2.3 SlideVQA:↩︎

In 2023, SlideVQA [20] was presented in the Thirty-Seventh AAAI Conference on Artificial Intelligence (AAAI-23). This dataset comprises more than 14K question-answer pairs from around 2.5K slide decks. The questions in this dataset are categorized into three types: 1) Single- and 2) Multi-span, which are questions whose answers are extracted from a single or various spans respectively of one or more evidence pages; and 3) Non-span questions, which are questions whose answers are composed of numerical values or visual appearances. The questions in SlideVQA are further divided in single-hop or multi-hop, depending on the number of slide pages from where the information need to be extracted to answer the question. To keep a fair comparison with the other selected datasets, which are single-page DocVQA datasets, we filtered out the multi-hop questions in SlideVQA, and kept only the single-hop questions.

Similarly to InfographicsVQA, the questions in SlideVQA are not only extractive, but also numerical and based on visual cues. This presents a complex task for models, as they not only need to reason over tokens, but also show mathematical reasoning to answer the question correctly. Another important feature of SlideVQA is that the slides do not show a regular or structured layout, as shown in Fig. 3, not even slides from the same presentation. Slides present less text than infographics or forms, so the relationships between objects are mostly based on visually-grounded information, and not so text-dependent. This presents a key factor when working with VLMs on slides, as the visual encoder of the model takes much more importance than the text encoder, which is also one of the reasons why VLMs may fail to adapt to domains where the importance between vision and language is not balanced and is variable between domains.

4 Experiments and Results↩︎

4.1 Zero-Shot Experiments↩︎

The first experiments we ran consisted of running a zero-shot evaluation of the 8 selected VLMs in the three datasets. Before passing the document images through the model, we padded them to unify the image sizes of all the datasets. The final image resolution of all images was \(1449 \times 1449\). We found that padding the images to a unified size yields better performance for all settings than passing the original images of different sizes and resolutions through the models. Apart from the image, we also passed the question at hand with the following prompt:
"Answer the following question, show only the answer of the question.
Question:
Answer:".

The results of the zero-shot evaluation are reported in Table 1. We report the mean Average Normalized Levenshtein Similarity (ANLS) over the evaluation set of each dataset. As expected, the best performing VLMs are the ones with more number of parameters, concretely Qwen3-VL 8B and Qwen 3.5 9B. It is notable that for all three datasets, the VLMs performance increases with the number of parameters, which reflects the complexity of the DocVQA task, as smaller models lack the necessary depth of understanding to fully exploit the necessary visual and text artifacts to answer the questions correctly, at least in a zero-shot setting. It is also remarkable the great difference in ANLS performance of each VLM between datasets. In SP-DocVQA, all VLM show impressive performance in a zero-shot environment that surpasses in some cases the \(90\%\) of ANLS, while in InfographicsVQA and SlideVQA the best performing VLM barely surpasses the \(60\%\) and \(50\%\), respectively. We believe that this difference is due to the visual and layout complexity of InfographicsVQA and SlideVQA, which pretrained VLMs without any finetuning are not able to fully comprehend, while the structured layout of SP-DocVQA lets VLMs to fully understand the documents, and leverage the internal previous knowledge from pretraining to understand and correctly answer the questions.

4.2 Supervised Finetuning↩︎

Previous works in VQA have addressed the possibility that VLMs cannot fully exploit their internal knowledge when answering questions [28]. To test that hypothesis in DocVQA, we decided to perform a fully supervised finetuning of each model on every dataset. This would allow us to determine if the bottlenecks when dealing with document images are the questions or the visual complexity.

For this experiment, all models were finetuned using the Unsloth library for \(3\) epochs using LoRA [10], [11], [35] with rank \(32\), LoRA alpha \(4\) and LoRA dropout of \(0.2\). For each finetuning, we report the results of inter- and intra-dataset evaluations.

4.2.1 Finetuning on SP-DocVQA.↩︎

8pt

In Table 2 we observe that almost all models outperform the zero-shot setting. As expected, the models that increase more with respect to the zero-shot setting are the smaller ones, concretely, the Qwen 3.5 0.8B and 2B versions. This may confirm our statement that smaller pretrained models lack enough depth to fully understand the DocVQA task in zero-shot evaluations and need some kind of finetuning or context learning to exploit its full potential. The only finetuned model that did not outperform the zero-shot evaluation was Gemma 3 4B, which gave a mean ANLS decrease of \(2.24\) in InfographicsVQA. This model is also the one with the worst performance in all the datasets over models with more than 4 billion parameters.

After finetuning SP-DocVQA, we observe that while the largest increase in performance occurs with Qwen 3.5 0.8B in the SP-DocVQA evaluation set, with an increase of 17.45% in ANLS, there is not much difference between the evaluation on SP-DocVQA and SlideVQA, with an increase in ANLS of approximately the same percentages. This is not the case on InfographicsVQA, where the finetuning on SP-DocVQA gave the biggest increase in ANLS with respect to the zero-shot setting among the three datasets.

4.2.2 Finetuning on InfographicsVQA.↩︎

8pt

8pt

In Table 3 we observe a similar performance in both the inter- and intra-data evaluations after finetuning in InfographicsVQA and after finetuning in SP-DocVQA. As expected, the inter-dataset evaluation gives better results than the inter-dataset evaluation with SP-DocVQA. In particular, the biggest differences of these results with respect to the results in Table 2 are the Gemma 3 results in InfographicsVQA and SlideVQA, with an increase of \(5.72 \%\) and \(6.71 \%\) respectively with respect to the zero-shot evaluation, in contrast to the results after finetuning in SP-DocVQA, where there was a decrease in ANLS when evaluating in InfographicsVQA. Another remarkable difference is that all models perform better in SlideVQA after being finetuned in InfographicsVQA than after being finetuned in SP-DocVQA, which may indicate the similarity of both datasets in their layouts and visual complexity. These results show that VLMs need to address different layout structures more efficiently to extract key visual information from different document types.

4.2.3 Finetuning on SlideVQA.↩︎

Finally, in Table 4 we observe similar results between and within the data set to those reported earlier. However, after finetuning on SlideVQA we noticed that the increase in ANLS on inter-dataset evaluations is smaller than when finetuning on SP-DocVQA and InfographicsVQA. This difference is more notable with larger models such as Qwen 3.5 9B and Qwen 3 VL 8B, where the difference in ANLS in SP-DocVQA with respect to zero-shot evaluation is less than \(1 \%\). It is also notable that Gemma 3 4B is the only model that performs worse on SP-DocVQA after being finetuned for the DocVQA task. Concretely, after finetuning Gemma 3 4B in SlideVQA it gives a decrease in the ANLS between data sets of \(0.30 \%\) in SP-DocVQA.

4.3 Few-Shot Results↩︎

In order to provide a more complete study on the effect of finetuning VLMs on different document domains, we performed a final round of few-shot experiments. For this round of experiments, we selected SlideVQA as the target dataset, as it is the dataset where most models perform worse even after specific finetuning on SlideVQA, and SP-DocVQA and InfographicsVQA as the source datasets.

We selected the finetuned models on SP-DocVQA and InfographicsVQA and finetuned them for 3 more epochs using a small number of training samples. The number of SlideVQA samples on which the model was trained was \(5, 20\) and \(50\) samples. Fig. 4 reports a graphic showing the evolution of the results of each VLM with increasing shots. We compare the performance of few-shots with the zero-shot evaluation of each finetuned model in SP-DocVQA based on the results reported in Table 2 and of each finetuned model in InfographicsVQA based on the results reported in Table 3. We also compare the few-shot results with the “best” possible results, which are expected to be the results after a fully supervised finetuning on SlideVQA.

We observe that most models perform better when more training samples are added, with some exceptions such as Qwen3.5 9B, which performed worse with \(5\) samples than with \(0\) samples. As expected, all models perform better with \(50\) training samples than with zero samples, which are the zero-shot results. However, we observe that, in general, the difference between full finetuning and finetuning using only \(50\) samples is not that much, and some VLMs even perform better when given only \(50\) training samples than with the entire training set.

These results may lead to the conclusion that pretrained and finetuned VLMs are already good at answering questions based on visual inputs, which can be inferred from the results discussed in previous sections. However, there is a huge bottleneck for VLMs, as their performance is greatly constrained by an inefficient extraction of visual information and, more concretely, a suboptimal understanding of complex layouts like the ones present on InfographicsVQA and SlideVQA.

5 Discussion and Future Work↩︎

In this work, we presented a comprehensive empirical study evaluating the capabilities, robustness and ability to adapt to different domains of \(8\) Vision-Language Models on distinct Document Visual Question Answering domains. We selected datasets \(3\) from DocVQA consisting of industrial documents of a wide range of document types, infographics, and presentation slides. We studied how the VLM family, the number of parameters and the cross-domain training influence the DocVQA task. Our experiments cover zero-shot evaluations, fully supervised finetuning with inter- and intra-dataset evaluations, and few-shot finetuning on a target domain of already finetuned models on different domains. All of these experiments provide a comprehensive understanding of the strengths and weaknesses of VLMs in different document domains for DocVQA. This may serve for future work on domain adaptation techniques using VLMs in DocVQA.

Our experiments reveal several insights regarding layout and visual complexity when working with VLMs. In zero-shot settings, pretrained models show strong performance in highly structured and easy-to-read layouts like the ones on SP-DocVQA, where an ANLS of more than \(90\%\) is achieved. However, this performance dropped sharply when confronted with complex layouts and visually dependent structures of infographics and presentation slides, which may reveal that the usual text-vision alignment of VLMs is not enough to fully exploit the models pretrained knowledge without a correct structural or domain-dependent adaptation. Furthermore, while parameter scaling consistently proved to be an important factor in performance, smaller models exhibit the highest relative gains during fully supervised finetuning. This shows that supervised optimization allows smaller architectures to demonstrate their full capabilities. We also observed that similarities in layout complexity plays an important role in generalization, as models trained on visually complex infographics transferred their knowledge to presentation slides more effectively than those trained on structured layouts present in industrial documents.

Few-shot experiments on presentation slides provide an encouraging outlook for low-resource domains in real-world scenarios. Incorporating as few as \(50\) target-domain samples into previously finetuned models on large-scale datasets led to performance improvements, with some models outperforming their fully supervised counterparts. This behavior demonstrates that the underlying bottleneck during cross-domain knowledge transfer is not a lack of domain knowledge, but rather a structural domain misalignment.

Several research directions emerge from this study. Future works should explore architectural modifications that explicitly inject domain information and modules to provide VLMs with easier-to-exploit representations of visually complex layouts to fully exploit their capacities. Additionally, mitigating negative cross-domain transfer and preventing catastrophic forgetting remain an open challenge, suggesting that parameter-efficient knowledge distillation techniques may be the way to go for researchers. Finally, this study provides a study on single page DocVQA, however, multi-page DocVQA remains a complex challenge, especially in multi-domain scenarios or open-domain settings, and a similar study on multi-page DocVQA would be of great use for the community.

In our future work, we will also study the risks in LLMs and VLMs both in the user interaction with them [36] and intrinsically based on their nature and learning elements [10], [37]). Analyzing biases [38]–[40] and synthetic manipulations in documents [41], [42] while maintaining privacy [43], [44] are also key topics in our DU research agenda. Finally, we will apply these DU technologies to specific domains such as crime investigation [45], public and legal affairs [5], [6], e-learning [46], and e-health [26], [47].

References↩︎

---

1. https://huggingface.co/qwen↩︎

2. https://huggingface.co/collections/google/gemma-3-release↩︎

3. https://github.com/unslothai/unsloth↩︎

4. https://www.industrydocuments.ucsf.edu/↩︎