A Pilot Study on Curator-Guided Multilingual Art Description for Blind and Low-Vision Audiences with Small Vision–Language Models
May 29, 2026
Abstract
Artificial intelligence (AI) can vastly improve content personalisation for diverse users. In the space of art, in particular, blind and low-vision (BLV) individuals are currently excluded from most visual content. To mitigate this, museums need secure, on-premise vision-language models (VLMs) to produce high-quality audio descriptions for BLV audiences across languages, including low-resource ones. In this work, we explore different design choices for small VLMs: under a fixed backbone and training budget, do language-specific adapters outperform a single multilingual adapter? Using Qwen2.5-VL-3B-Instruct, we build a parallel BLV-oriented caption corpus in German, Romanian, and Serbian, and fine-tune monolingual and multilingual adapters. We evaluate both regimes with automatic metrics and an LLM-as-Judge protocol spanning multiple state-of-the-art models, and calibrate judge reliability with a Romanian BLV pilot study by measuring agreement between human and LLM preferences. Under our pilot setup, language-specific adapters show more stable controllability and visually grounded description quality for Romanian and Serbian, while multilingual adaptation remains competitive in German. We interpret these findings as deployment-oriented evidence for small on-premise VLMs.
<ccs2012> <concept> <concept_id>10010147.10010257.10010293.10010294</concept_id> <concept_desc>Computing methodologies Neural networks</concept_desc> <concept_significance>500</concept_significance> </concept> <concept> <concept_id>10003120.10003121.10003129.10011756</concept_id> <concept_desc>Human-centered computing Accessibility technologies</concept_desc> <concept_significance>500</concept_significance> </concept> <concept> <concept_id>10010405.10010489.10010492</concept_id> <concept_desc>Applied computing Arts and humanities</concept_desc> <concept_significance>300</concept_significance> </concept> <concept> <concept_id>10010147.10010257.10010282</concept_id> <concept_desc>Computing methodologies Image and video captioning</concept_desc> <concept_significance>300</concept_significance> </concept> </ccs2012>
1 Introduction↩︎
Foundation models are increasingly used as general-purpose multimedia backbones in high-stakes settings [1]–[6], which raises practical questions about deploying capable models under strict privacy and infrastructure constraints. In museums and art institutions, the mandate to protect visitor privacy often necessitates on-premise inference, while the desire for controllability incentivises open-weight models [7]. These requirements raise the need for compact vision-language models (VLMs) and parameter-efficient adaptation.
Prior BLV-oriented captioning work has largely focused on assistive, in-the-wild scenarios (e.g., photos taken by blind users), where the main objective is immediate utility via object naming and short factual statements. In contrast, museum descriptions require longer, orientation-first narratives with stable structure and controllable verbosity, often closer to curated description practice than to generic image captioning [8]–[12], Snyder14-VMV?.
From a modelling perspective, lightweight adapters such as LoRA [13] make small VLMs practical without full fine-tuning, while multilingual multimodal resources (e.g., Multi30K [14] and WIT [15]) have enabled cross-lingual training beyond English. However, for multilingual BLV museum description at the \(\sim\)3B scale, the deployment choice between sharing a single multilingual adapter versus allocating the same capacity to language-specific adapters remains underexplored, especially for lower-resource languages [16]–[20].
Finally, caption evaluation is increasingly recognised as under-specified by \(n\)-gram metrics alone [21]–[24]. Rubric-driven, model-based evaluation (LLM-as-Judge) has gained traction as a scalable alternative [25]–[29], but must first be validated against human ratings to draw robust conclusions [30], [31]. Motivated by this, we complement standard automatic metrics with a BLV rubric and calibrate candidate judges on a small BLV pilot before scaling evaluation to languages where BLV annotations are unavailable.
Beyond a language-only comparison, our contribution is inherently multimedia: we study accessible description generation from the combination of artwork images, curatorial metadata, and language-specific descriptive conventions in a realistic museum deployment setting. The paper therefore addresses a multimodal problem at three levels: multimodal input (image + metadata), multimodal generation for accessibility (visual content rendered into structured BLV-oriented language), and multimodal evaluation, where lexical metrics are complemented by visually grounded rubric-based judging. Under this framing, the comparison is not an isolated NLP question, but a deployment design study for compact on-premise multimedia systems that must remain controllable, privacy preserving, and usable in low-resource languages.
1.1 Research Questions↩︎
We study small, open-weight, on-premise VLMs (approximately 3B parameters) for BLV art description in German (DE), Romanian (RO), and Serbian (SR), with a held-out test split consisting of artwork styles not seen in training. Our work investigates the following research questions:
RQ1 (Adapter strategy under a fixed budget). Given the same backbone and LoRA parameter budget, does language-specific adaptation (one adapter per language) outperform a single multilingual adapter for BLV description quality and controllability, and how do both regimes compare to the base model (zero-shot)?
RQ2 (Robustness across languages). Does multilingual parameter sharing change robustness and controllability relative to language-specific adapters across DE, RO, and SR on unseen styles, and where do the regimes differ most in error type (length, repetition, and visually grounded content)?
RQ3 (Human-grounded evaluation at scale). After calibrating an LLM-as-Judge rubric on a Romanian BLV pilot study, can LLM judges reliably reproduce human mono-multi preferences and trait-level score differences (notably for visually grounded vs affective traits), so that evaluation can be extended to DE and SR where BLV annotations are unavailable?
| Category / | # | Train | Test | Notes | Words/desc | Sent./desc | Words/sent. Corpus** | Items** | | | | | | :=================+:========:+:=========:+:========:+:=================:+:===============:+:===============:+:===============: Dataset stages and filtering (English base) | ||||||||
| 1-8 Safe dialogues (initial) | 799 | – | – | English; image, metadata | – | – | – | |
| After pruning | 765 | – | – | Drop rare styles \(<\!10\) | – | – | – | |
| Train split | 553 | 553 | – | All styles except held-out | – | – | – | |
| Test split (style-held-out) | 212 | – | 212 | Impres., Expres., Naïve/Primitivism | – | – | – | |
| Per-emotion counts after pruning and style-held-out split | s after pruning and style-held-out split* | | |||||||
| 1-8 amusement | 81 | 58 | 23 | – | – | – | – | |
| anger | 66 | 55 | 11 | – | – | – | – | |
| awe | 92 | 86 | 6 | – | – | – | – | |
| contentment | 84 | 42 | 42 | – | – | – | – | |
| disgust | 61 | 42 | 19 | – | – | – | – | |
| excitement | 55 | 39 | 16 | – | – | – | – | |
| fear | 86 | 64 | 22 | – | – | – | – | |
| neutral | 154 | 111 | 43 | – | – | – | – | |
| sadness | 86 | 56 | 30 | – | – | – | – | |
| 1-8 Totals | 765 | 553 | 212 | – | – | – | – | |
| BLV caption corpora (generated in DE, SR, RO) | ra (generated in DE, SR, RO)* | | |||||||
| 1-8 German (DE) | 765 | 553 | 212 | Caption corpus | 263.39 | 15.36 | 17.29 | |
| Serbian (SR) | 765 | 553 | 212 | Caption corpus | 215.45 | 12.43 | 17.52 | |
| Romanian (RO) | 765 | 553 | 212 | Caption corpus | 224.85 | 11.05 | 20.54 | |
| Multilingual (DE+SR+RO) | 2295 | 1659 | 636 | Merged DE+SR+RO | \(\sim\)234.6 | \(\sim\)12.95 | \(\sim\)18.45 | |
1.2 Experiment Scope and Model Choices↩︎
To isolate the mono vs multilingual effect under realistic museum constraints, we fix a single small open-weight VLM backbone: Qwen2.5-VL-3B-Instruct1. We chose this model because it is (i) small enough for on-premise deployment, (ii) trained for multilingual captioning with a strong vision encoder, and (iii) fully open, which makes replication and downstream use straightforward. All language-specific and multilingual adapters are implemented with LoRA layers added to this backbone.
For evaluation beyond automatic lexical and embedding metrics, we employ an LLM-as-Judge setup. We first calibrate several candidate judges (e.g., GPT-4o, Claude, Gemma-3-12B-IT2) against a small set of Romanian BLV human annotations and then use the best-aligned judges to compare monolingual and multilingual systems across all three languages. Details of this calibration are given in the results section.
2 Methodology↩︎
2.1 Dataset↩︎
We build on ARTEMIS [32], an image–emotion corpus linking artworks to human affect annotations, and its curated, safety-screened derivative Art-GenEvalGPT [33]. From the latter, we retain only entries flagged as safe dialogues, together with minimal metadata required for our task: artwork/style codes, author and artwork identifiers, emotion labels, and image paths. This yields 799 image–dialogue pairs (English) that serve as the source content pool from which we later derive language-specific accessible descriptions. To obtain a balanced, well-specified base for training/evaluation prior to multilingual generation, we apply two lightweight steps:
Pruning rare categories. We drop styles with fewer than 10 samples and remove the underspecified emotion label “something else”. This reduces the set to 765 items.
Style-held-out split. To test generalisation across visual domains, we construct a style-held-out evaluation: three highly entangled styles – Impressionism (often contentment-leaning), Expressionism (fear-leaning), and Naive Art/Primitivism (amusement-leaning) – are reserved for test. The remaining styles form train. This yields 553 train and 212 test samples, maintaining diverse emotion distributions in both splits as in Table 1.
Data Statistics. Each English base item yields one accessibility-oriented description per target language. We achieve this by producing one description per language using GPT-4o-mini3 with the corresponding description with the artwork’s metadata. This produces three monolingual corpora whose sizes mirror the base split (553 train / 212 test per language). A multilingual corpus is formed by concatenating the three monolingual sets and shuffling (1,659 train / 636 test). We report simple, language-agnostic text metrics (means per description): words, sentences, words/sentence, and type–token ratio (TTR). For details see Table 1.
A key design choice in our pipeline is that supervision is prototype-guided rather than prompt-only. For each target language, curator-provided BLV prototype descriptions define the desired structure, level of detail, and orientation-first style, and are combined with shared accessibility instructions, artwork metadata, and the artwork image to generate one synthetic reference description per item. These references are therefore intended to operationalise a target descriptive format grounded in professional practice, not to serve as universal ground truth for art interpretation. Accordingly, training and reference-based evaluation should be read primarily as measuring how well a model reproduces this target BLV description style; that is, controllability and distillation fidelity under a fixed multimedia generation protocol.
2.2 Training↩︎
We train the VLM in four regimes: three strictly monolingual runs (SR, DE, RO) and one multilingual run trained on the concatenation of the three languages. The multilingual run shuffles the merged pool. In all cases, the model is trained autoregressively on instruction–response samples that pair the artwork image with a BLV-oriented target description generated as described, preserving a consistent schema across languages.
4pt
| Embedding cosine \(\uparrow\) | Length error \(\downarrow\) | Repetition \(\downarrow\) | ROUGE-L \(\uparrow\) | BLEU \(\uparrow\) | chrF \(\uparrow\) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 3-5(lr)6-7(lr)8-9(lr)10-11(lr)12-13(lr)14-15 Lang. | Variant | LS | ML | \(\Delta\) | LS | ML | LS | ML | LS | ML | LS | ML | LS | ML |
| German | Base model | 0.686 | 0.686 | +0.000 | 432 | 432 | 0.378 | 0.378 | 0.162 | 0.162 | 0.041 | 0.041 | 0.380 | 0.380 |
| Fine-tuned | 0.799 | 0.688 | +0.111 | 229 | 276 | 0.376 | 0.345 | 0.160 | 0.209 | 0.090 | 0.090 | 0.405 | 0.449 | |
| Romanian | Base model | 0.651 | 0.651 | +0.000 | 386 | 386 | 0.386 | 0.386 | 0.146 | 0.146 | 0.028 | 0.028 | 0.333 | 0.333 |
| Fine-tuned | 0.796 | 0.696 | +0.100 | 209 | 541 | 0.323 | 0.261 | 0.228 | 0.225 | 0.117 | 0.075 | 0.466 | 0.353 | |
| Serbian | Base model | 0.634 | 0.634 | +0.000 | 516 | 516 | 0.509 | 0.509 | 0.001 | 0.001 | 0.014 | 0.014 | 0.262 | 0.262 |
| Fine-tuned | 0.814 | 0.746 | +0.068 | 220 | 207 | 0.311 | 0.295 | 0.564 | 0.101 | 0.097 | 0.095 | 0.425 | 0.419 | |
Optimisation follows a lightweight LoRA setup over all linear modules with \(16-bit\) floating point activations: 3 epochs, per-device batch size 2 for train and eval, learning rate \(10^{-4}\), context length \(2048\) tokens, and periodic evaluation every 25 steps with checkpointing every 100 steps. Adapters use rank 132 and \(\alpha=32\). Representative runs expose \(\approx\)247M trainable parameters (about 6% of the full backbone, as reported by our training setup). Best checkpoints are selected by held-out eval loss/accuracy (monolingual: step 100–105; multilingual: step 519).
2.3 Experiments and metrics↩︎
We compare two inference variants on the held-out split for each language: Base model (zero-shot; no LoRA) (base Qwen model with image+prompt) and Fine-tuned (LoRA-adapted, image+prompt). For every variant we evaluate both Language-Specific (LS; monolingual) and Multilingual (ML; merged/multilingual) checkpoints. Metrics are computed per image and averaged across the test set.
Embedding similarity and projections. We embed references and system outputs with the multilingual embedding model gte multilingual base4 (which supports text lengths up to 8192 tokens and over 70 languages) and report cosine similarity as a language-agnostic semantic score.
Lexical fidelity. We report BLEU-4, ROUGE-L, and chrF++ to capture surface overlap and ordering. chrF++ is included to reduce sensitivity to inflectional variation across the languages. Tokenisation and casing are held constant across LS/ML; we do not apply length penalties beyond the metric defaults.
Structure and format control. To track accessibility-oriented formatting, we measure (i) absolute character-length error between candidate and reference (lower is better) and (ii) token repetition \(1-\frac{|\text{unique tokens}|}{|\text{tokens}|}\) (lower is better), which penalises looping and redundant phrasing.
LLM-as-Judge (BLV rubric). We calibrate three candidate LLM judges, i.e., Gemma-3-12B-IT5, GPT-4o6, and Claude 3.7 Sonnet7, against the Romanian pilot study (2 BLV raters, 25 items). Trait scores are produced with DeepEval’s
GEval module8, an implementation of the rubric-driven G-Eval protocol for LLM-based evaluation[30]. For each artwork and trait, we compute the per-item LS-ML difference \(\Delta = s^{\text{LS}} - s^{\text{ML}}\) and quantify judge-human alignment using (i)
Spearman rank correlation on \(\Delta\) and (ii) sign agreement, i.e., the fraction of items where the judge and humans select the same winner (LS vs ML) (Fig. 2).
Pilot human evaluation (Romanian). We ran a small Romanian pilot to calibrate our metrics and LLM-as-Judge setup. Two native Romanian BLV participants evaluated 25 random test artworks in a custom Gradio interface, comparing two unlabelled captions (monolingual vs multilingual) with randomised left/right order. For each item, they rated both captions on five BLV traits (coherence, composition, colour fidelity, vividness, emotional tone; 1–10) and selected a preferred caption. This yields 500 trait ratings and 50 preference votes, collected independently.
Reporting. Unless otherwise stated, comparisons between LS and ML use the Fine-tuned variant as the main line.
3 Results↩︎
3.0.0.1 Embedding similarity.
Fine-tuning substantially improves over the base model across all languages (Table 2). Gains are largest for Romanian and Serbian, where language-specific adapters show the strongest improvements, while German exhibits smaller LS-ML gaps.
3.0.0.2 Surface form and control metrics.
Across all languages, LoRA fine-tuning improves both lexical overlap (ROUGE-L, BLEU, chrF) and output control (length error, repetition) over the base model (Table 2). Length error drops substantially for the LS adapters in German (432 to 229 chars) and Romanian (386 to 209 chars), with lower repetition in Romanian (0.386 to 0.323) and Serbian (0.509 to 0.311). Lexical gains are strongest in the lower-resource languages: Romanian LS improves from ROUGE-L 0.146 and BLEU 0.028 to 0.228 and 0.117 (chrF 0.466), and Serbian LS from ROUGE-L 0.001 and BLEU 0.014 to 0.564 and 0.097 (chrF 0.425). In German, LS and ML are closer on lexical metrics, with ML higher on ROUGE-L/chrF (0.209/0.449 vs 0.160/0.405) and equal BLEU (0.090), suggesting near-parity in this higher-resource setting.
3.0.0.3 LLM-as-Judge calibration.
Human inter-annotator agreement in the Romanian BLV pilot is moderate (Spearman \(\rho=0.32\)-\(0.49\) across traits), which is expected for subjective description-quality ratings. To calibrate LLM judges, we compare monolingual against multilingual captions by computing a per-item delta for each trait, \(\Delta_i = s_i^{\text{LS}} - s_i^{\text{ML}}\). We then quantify judge-human alignment in two complementary ways: (i) Spearman rank correlation between human and judge per-item deltas (\(\Delta_i\)), and (ii) sign agreement, i.e., the fraction of items where the judge and humans select the same winner (LS vs ML; Fig. 2).
Alignment is strongest for visually grounded traits. For composition and colour fidelity, Claude and GPT-4o best track the human per-item LS-ML pattern (\(\rho=0.51\)-\(0.59\)) and reach up to \(85\%\) sign agreement, indicating that they often identify the same images where LS improves (or degrades) factual visual content. Agreement is weaker and less stable for more interpretive traits (vividness and emotional tone), where both human deltas and judge deltas show higher dispersion, suggesting that these dimensions admit multiple valid rubric interpretations at the item level. Gemma shows the largest deviations from human preferences; this is plausible given its smaller capacity, and we therefore treat it as a robustness check rather than a basis for the main conclusions.
Overall, the pilot supports a trait-dependent view of judge validity: Claude is used as the primary judge and GPT-4o as a secondary judge for DE/SR analyses, with Gemma retained for additional diversity.
3.0.0.4 German and Serbian.
We next apply the two calibrated judges (Claude and GPT-4o) to German (DE) and Serbian (SR), where we do not have BLV human ratings. Table 3 reports the mean LS-ML score difference per trait, \(\Delta = s^{\text{LS}}-s^{\text{ML}}\), together with \(p_{\text{LS}}\) (the fraction of images where LS scores higher).
In German, both judges attribute a clear LS advantage to the most visually grounded dimensions. For composition, LS improves by \(+1.58\) (Claude) and \(+2.75\) (GPT-4o), with LS winning on most items (\(p_{\text{LS}}=0.75\) and \(0.96\)). For colour fidelity, the gains are similarly consistent (\(+1.88\) and \(+1.96\)), with LS preferred on at least two thirds of the images (Claude: \(0.67\); GPT-4o: \(1.00\)). In contrast, coherence, vividness, and emotional tone exhibit small deltas close to zero and mixed win rates, indicating approximate parity between LS and ML on these more interpretive traits in DE.
Serbian shows the same main pattern for composition: LS is preferred on average (\(\Delta=+0.96\) for Claude; \(+2.34\) for GPT-4o), and LS wins on a large share of items (\(p_{\text{LS}}=0.67\) and \(1.00\)). For colour fidelity, both judges again favour LS but with stronger judge dependence (Claude: \(+0.71\), \(p_{\text{LS}}=0.54\); GPT-4o: \(+2.07\), \(p_{\text{LS}}=0.92\)). By contrast, coherence, vividness, and emotional tone have negative mean deltas in both judges, suggesting a mild ML tendency on these dimensions; however, the effect sizes are small (all \(|\Delta|<0.5\)), so we interpret them as secondary relative to the robust LS gains on factual traits.
Taken together, the DE/SR results mirror the Romanian calibration: LS adaptation most reliably improves visually grounded content (composition, colour), while ML is competitive on higher-level, stylistic dimensions where preferences are weaker and more variable.
| Claude | GPT-4o | ||||
|---|---|---|---|---|---|
| 3-4 (lr)5-6 Language | Trait | \(\Delta\) | \(p_{\text{LS}}\) | \(\Delta\) | \(p_{\text{LS}}\) |
| DE | Coherence | -0.04 | 0.04 | -0.09 | 0.46 |
| Composition | 1.58 | 0.75 | 2.75 | 0.96 | |
| Colour | 1.88 | 0.67 | 1.96 | 1.00 | |
| Vividness | 0.29 | 0.33 | -0.05 | 0.54 | |
| Emotion | -0.04 | 0.08 | -0.18 | 0.38 | |
| SR | Coherence | -0.25 | 0.17 | -0.25 | 0.25 |
| Composition | 0.96 | 0.67 | 2.34 | 1.00 | |
| Colour | 0.71 | 0.54 | 2.07 | 0.92 | |
| Vividness | -0.21 | 0.08 | -0.43 | 0.29 | |
| Emotion | -0.21 | 0.12 | -0.35 | 0.25 | |
4 Discussion and Conclusion↩︎
We addressed a practical museum deployment question for small, open-weight, on-premise VLMs: given a fixed backbone and LoRA budget, should BLV-oriented art description use a single multilingual adapter or separate monolingual adapters?
4.0.0.1 Main findings.
Across German, Romanian, and Serbian, LoRA fine-tuning consistently improves over the base model in both semantic and surface-form metrics and reduces repetition. Under matched capacity in our pilot setup, language-specific adapters appear more stable in the lower-resource settings, where they yield tighter controllability (verbosity and repetition) and stronger visually grounded quality. In German, the multilingual adapter remains competitive, indicating that cross-lingual transfer can be sufficient when the language is well covered by the base model.
4.0.0.2 Evaluation implications.
Calibrating LLM-as-Judge on a Romanian BLV pilot enables scalable, trait-level comparison when BLV annotations are unavailable. In our setup, stronger proprietary judges align best with BLV preferences for factual, image-grounded traits (e.g., composition and colour fidelity), while affective/stylistic traits show lower human agreement and weaker judge alignment; these dimensions should therefore be interpreted cautiously in cross-language conclusions. The Romanian BLV pilot should be interpreted as a calibration study, not as a comprehensive user evaluation. Its role is to test whether the proposed rubric and candidate LLM judges capture the same relative LS–ML preferences as human raters, especially for visually grounded traits such as composition and colour fidelity. It does not establish broad cross-language accessibility effectiveness, nor does it replace larger BLV studies in German and Serbian. We therefore use the pilot to justify judge selection and to support cautious trait-level interpretation, while reserving stronger claims about end-user accessibility benefit for future, larger-scale human evaluation.
4.0.0.3 Limitations and next steps.
Our BLV references are synthetically generated, so reference-based metrics can inherit teacher-student biases, and automatic scores remain imperfect proxies for accessibility. The human study is a small Romanian pilot used primarily for judge calibration; broader BLV evaluations in DE and SR are needed to confirm trait-level trends and quantify practical effect sizes. We plan to release curated splits, prompts, and LS/ML adapters for reproducibility.
Acknowledgements↩︎
This work has received funding from the European Union’s Horizon Europe research and innovation programme under grant agreement No. 101060660 (SHIFT).