[ email=spiros.baxevanakis@student.uva.nl, ]

[ email=lesterpjy@gmail.com, ]

test-time scaling , vision-language models , multimodal reasoning , self-consistency , process reward models ,

1 Introduction↩︎

Multimodal reasoning remains a challenge for vision–language models (VLMs) whenever questions combine structured visual content with multi-step inference across languages. Real exam questions are a natural stress test of this combination, and the EXAMS-V benchmark [1] spans thirteen languages and twenty school subjects rendered as text-in-image panels. The 2026 ImageCLEF multimodal reasoning shared task [2], [3] fixes the practical constraints: a single A40 GPU, open-weight policies with at most 7B parameters, and a reproducibility mandate.

The natural question is whether the inference-time techniques that advanced text-only language models [4]–[6] transfer to open VLMs under this budget. The evidence is cautionary: self-refinement degrades open-source VLMs at this scale [7], long sequential “thinking” often underperforms parallel sampling on small policies [8], [9], and discriminative reward models trained on mathematics fail to transfer to humanities and non-English content [10], [11]. A viable pipeline must externalise selection, prefer parallel sampling to long single-chain reasoning, and avoid iterative self-refinement loops.

We investigate whether describe-then-reason with PRM-guided search beats flat self-consistency; how accuracy scales along two axes (chain count and per-chain token budget) under a single-A40 envelope [6]; where TTS helps and fails across subjects and languages [7]; and whether a training-free generative critic can beat majority vote on low-agreement questions, with a cross-model PRM [12] controlling for sycophantic self-scoring [13]. The latter extends generative-verifier findings [14]–[16] to a multimodal, multilingual setting under a \({\le}7\)B budget not previously studied to our knowledge.

1.0.0.1 Summary of findings.

Across all four dimensions, the policy substrate and engineering details (parser format, decoding budget) dominate any search or selector method. Our best configuration, Qwen3.5-4B at SC-\(N{=}16\) with a \(2{,}048\)-token budget, reaches \(81.6\%\) on the full validation set. ¹

2 Related Work↩︎

2.0.0.1 Test-time scaling: theory and limits.

[6] formalize TTS as a two-axis allocation (width: parallel chains; depth: per-chain revision or search), proving compute-optimal allocation can match naive Best-of-\(N\) at less cost [17], [18]. [19] show the verifier advantage grows with reasoning depth, motivating an external selector; converging findings [7]–[9], [20] argue for parallel sampling and against self-correction at our policy scale.

2.0.0.2 Multimodal verification and PRM generalization.

A process reward model (PRM) assigns each reasoning step a probability \(P(+)\) that the step is correct [5]; it is useful only when \(P(+)\) sharply separates correct from incorrect steps. Two multimodal PRMs supersede VisualPRM-8B [21]: Athena-PRM-7B [22] and Qwen-VL-PRM-7B [12], which separates perception from reasoning errors. On the text side, discriminative math-trained PRMs fail out of distribution [10], [11], [23] while generative verifiers transfer more reliably [14]–[16]. Prior EXAMS-V work [24], [25] informs our scaffold; on the search axis, PRM-BAS [26] dominates step-level Best-of-\(N\) at matched compute.

3 Method↩︎

We evaluate two pipelines: describe-then-reason with PRM-guided beam search and a three-way selector contrast, and flat self-consistency with a large token budget.

3.0.0.1 Policy and data.

We evaluate two open-weights policies: Qwen/Qwen2.5-VL-7B-Instruct (7B parameters, bfloat16) for the baseline and ablation configurations, and Qwen/Qwen3.5-4B (4B parameters, bfloat16), a smaller but newer-generation policy, for the headline configuration. We cap image resolution at max_pixels\({=}1{,}003{,}520\) (\({\approx}4\)k vision tokens after Qwen smart-resize), balancing legibility of text-in-image exam panels against context budget: at max_model_len\({=}16{,}384\), this leaves \({\sim}12\)k tokens for the prompt and generated reasoning. Inference uses vLLM [27] with batched parallel decoding (SamplingParams(n=N)) and automatic prefix caching. All experiments run on a single A100-40 GB GPU, which matches the shared task’s A40 memory constraint. We evaluate on the full EXAMS-V validation split (4,651 questions, 11 languages) for headline numbers and a 200-question \((\text{language},\text{subject})\)-stratified subset (seed 42, minimum ten per language) for ablations whose full-validation cost is prohibitive.

3.0.0.2 Describe-then-reason.

Following [24], the describe-then-reason pipeline first samples \(N\) image-grounded descriptions of the question at \(T{=}0.7\). For each description, with the image removed, the policy generates \(M\) reasoning chains in the flat variant, or reasoning step-by-step via PRM-BAS-style beam annealing [26] in the search variant: an initial beam of width \(B_0\) is expanded with \(B\) continuations per surviving beam, each step is scored by Qwen-VL-PRM-7B [12], which retains access to the original image, and the beam width is halved whenever the score range among the top beams exceeds a threshold \(\tau\). Beams terminate on the answer cue or at a maximum depth \(d\). Across all (description \(\times\) terminal beam) candidates, a final selector picks the answer. In our experiments the flat variant uses \(N{=}4\), \(M{=}4\) (\(20\) calls/q, bracketing SC-\(N{=}8\) from above to test whether structured decomposition helps at higher compute); the search variant halves descriptions to \(N{=}2\) because beam expansion is expensive, and adopts the hyperparameters reported in [26]: \(B_0{=}4\), \(B{=}2\), \(\tau{=}0.05\), \(d{=}6\) (\({\sim}13\) calls/q).

3.0.0.3 Selectors.

Three selectors are applied post hoc to the same chain pool. Majority vote: most common answer letter with log-probability tie-breaking (no extra inference). A training-free generative critic inspired by [15] scores each chain on three axes (Appendix 13). A discriminative PRM (Qwen-VL-PRM-7B; [12]), a separately trained model, scores each chain in one-shot mode (Appendix 12), controlling for sycophantic self-preference [13]. Both models co-reside on the A100-40 GB at bfloat16. Each selector uses a skip rule [28]: high-confidence majorities (\({\ge}5/8\) agreement) pass through; only the weak (\({\le}4/8\)) and all-unparseable tiers are rescored.

3.0.0.4 Self-consistency.

The simpler pipeline samples \(N \in \{8, 16\}\) chains from Qwen3.5-4B at \(T{=}0.7\) with \(\texttt{max\_new\_tokens}{=}2048\), conditioned on a CoT prompt ending with the MMMU closer [29], [30]. A regex extracts the answer letter (mapping Cyrillic labels to A–E for multilingual coverage). The final answer is the majority letter across parseable chains, with log-probability tie-breaking. We additionally sweep \(T \in \{0.3, 0.5, 0.7, 0.9\}\) at full validation scale.

3.0.0.5 Guided parse repair.

When every chain for a question fails to produce an answer letter (\(2\)–\(18\%\) of questions depending on budget), each chain’s reasoning is concatenated with the cue \n\nAnswer:  and a single token is decoded under vLLM’s guided_choice constraint over \(\{A, B, C, D, E\}\). The majority across all committed letters (original plus repaired) is the final answer. This eliminates parse failures by construction.

3.0.0.6 Stratification and evaluation.

We partition accuracy by subject and language; per-subject results are reported only for cells with \(n{\ge}50\). Pairwise method comparisons use McNemar’s paired test with Bonferroni correction for the number of reported comparisons.

4 Experiments↩︎

We evaluate seventeen configurations across two policies, three search strategies, three selectors, and two scaling axes. Table 1 summarises them; the remaining sections report findings ordered by headline result rather than by configuration. Unless noted, all headline numbers are on the full EXAMS-V validation set (\(n{=}4{,}651\)); configurations marked “dev-200” use a 200-question stratified subset.

5 Results and Discussion↩︎

We report results grouped by headline finding.

5.1 The TTS progression and the parser artifact↩︎

3pt

The three-stage TTS progression (zero-shot, chain-of-thought, self-consistency) holds for both policies at full validation scale (Table 2). Each step yields a significant gain under Qwen2.5-VL-7B, and the progression is more pronounced under Qwen3.5-4B, where the full span from zero-shot to SC-\(N{=}8\) (2k) exceeds 24 pp.

However, the relative contribution of each stage shifts with the policy. Under Qwen2.5, the CoT-to-SC margin is modest (\(+3.3\) pp), suggesting that eight samples add limited signal once CoT prompting is in place. Under Qwen3.5, the margin widens markedly: CoT accounts for roughly half the total gain, and the remaining half comes from parallel sampling and the token-budget increase (Section 5.3).

5.1.0.1 A parser artifact masked part of the CoT gain.

An early prompt format left \({\sim}9\%\) of development chains unparseable. Switching to the MMMU-standard closer [29], [30] cut parse failures below \(2\%\) and raised dev accuracy by \({\sim}6\) pp; the fix corroborates at val scale (\(+0.82\) pp on Q2.5 SC-\(N{=}8\)). The methodological lesson: what looks like reasoning failure can be an extraction failure, and TTS studies that do not control for parseability risk attributing engineering artifacts to the scaling method.

Under Q3.5, every step is significant: CoT over zero-shot gains \(+12.6\) pp (non-overlapping CIs), the 1k\(\to\)​2k budget increase recovers \(+3.7\) pp (\(171\) additional correct answers), and the \(N{=}8 \to N{=}16\) step adds only \(+0.15\) pp (\(7\) questions, within sampling noise). Q3.5 CoT itself suffers \(16.2\%\) parse-fail from the same truncation that affects SC (\(18.4\%\) at 1k), so the reported \(69.66\%\) underestimates single-chain accuracy and the CoT-to-SC(1k) gain conflates truncation absorption with diversity benefit.

5.2 Structured search underperforms flat sampling↩︎

Dev-200 CIs overlap between PRM-BAS, DTR, and flat SC-\(N{=}8\) (Table 11, Appendix 15). A val-scale run of PRM-BAS under Qwen3.5-4B (\(N{=}2\), \(B_0{=}4\), \(B{=}2\), \(\tau{=}0.05\), \(d{=}6\); \(n{=}4{,}319\), \(93\%\) of validation) resolves the ranking in favour of SC: PRM-BAS reaches \(80.74\%\) against SC majority at \(81.13\%\) on the same questions (\(-0.39\) pp). Of the \(14.6\%\) of questions where PRM-BAS produces a different answer than SC, \(233\) are corrections and \(250\) are regressions, for a net loss of \(17\) questions. Two bottlenecks interact: in the DTR pipeline the reasoning stage operates on the text description alone (no image access), so visual detail lost at the description step cannot be recovered downstream; beam search compounds this loss because the PRM scoring each step (though it retains image access) lacks sufficient discriminative signal to steer search toward the missing detail.

5.2.0.1 Diagnosis.

[26] show that PRM-guided beam-annealing search dominates flat Best-of-\(N\) on multimodal math benchmarks. Our result inverts this: on multilingual visual MCQ, PRM-BAS underperforms flat SC by \(-0.39\) pp at \(8.7\times\) the cost. Two factors explain the discrepancy. First, the PRM’s per-step P(\(+\)) saturates (\(0.962\) on descriptions, \(0.849\) on reasoning), collapsing search diversity: \(71.9\%\) of val questions have all surviving beams agreeing on the same letter (Shannon entropy \(0.259/2.322\) bits, \(89\%\) reduction from uniform); the \(8.7\times\) cost yields near-zero answer diversity. Second, slicing by SC agreement tier (Appendix 8) reveals that on high-confidence questions (\({\ge}7/8\), \(91.8\%\) SC accuracy), PRM-BAS regresses by \(-2.16\) pp (\(48\) corrections vs.\(119\) regressions), dominating the overall deficit. Only on the \(108\) all-unparseable questions does PRM-BAS produce a clear positive (\(37\) correct, \(34.3\%\)). The P(\(+\)) asymmetry identifies reasoning as the structural bottleneck (discussed in Section 7).

5.3 Token budget dominates chain count↩︎

We sweep two scaling axes (per-chain token budget and number of sampled chains), first on the 200-question development subset, then on the full validation set (Table 3).

3.5pt

5.3.0.1 Token budget dominates chain count.

Doubling the budget from 1k to 2k tokens recovers \(+3.7\) pp (Figure 1a): chains generate valid reasoning but are truncated before emitting an answer letter, producing parse failures rather than reasoning failures. Doubling chains (\(N{=}8 \to 16\)) adds only \(+0.15\) pp. The asymmetry is structural: \({\sim}60\%\) of questions already reach strong agreement at \(N{=}8\), while the weak-agreement tail (\({\le}4/8\), \(23\%\)) has below-chance accuracy that additional sampling does not correct. (Appendix 8). For compute-constrained TTS, ensuring chains run to completion matters more than sampling more of them.

5.3.0.2 Temperature sensitivity.

An inverted-U optimum at \(T{=}0.7\) replicates from dev-200 to full validation (Figure 1b), but the spread compresses to \(0.65\) pp at scale, suggesting the policy’s sampling diversity is already near optimal.

5.4 Guided repair closes the parse-fail residue↩︎

After the budget fix, val SC-\(N{=}8\) still has \(124\) all-unparseable questions (\(2.67\%\)); the test split (\(n{=}1{,}117\)) has \(82\) (\(7.34\%\)), elevated because the test mix is \(44.8\%\) English and \(30.6\%\) Chinese. Guided repair drives both rates to zero at one decoded token per affected chain. The repair targets valid reasoning that was never committed to an answer letter, a failure mode that chain scaling cannot reach, recovering more than doubling \(N\) (\(+0.15\) pp val) at a fraction of the compute.

5.5 Policy choice dominates search and selection↩︎

Switching from Qwen2.5-VL-7B to Qwen3.5-4B at matched settings yields \(+11.4\) pp (Table 2, SC-\(N{=}8\) rows); with the budget increase the gap widens to \(15.1\) pp and all eleven languages improve by \({\ge}3\) pp (Table 10). Qwen3.5-4B is smaller; the gain reflects newer-generation training, consistent with [7]. At our budget scale, upgrading the policy dominates any inference-time strategy.

5.6 Selectors yield a null result on both pools↩︎

We test whether a post-hoc selector improves on majority vote. Two selectors are compared: a training-free generative critic (three rubric axes; Appendix 13) and a discriminative PRM (Qwen-VL-PRM-7B; one-shot scoring). Both apply a skip rule preserving high-confidence majorities. We evaluate on two pools of different baseline strength (\(66\%\) Q2.5, \(81\%\) Q3.5) to control for ceiling effects (Table 4).

3.5pt

\(^{\text{n.s.}}\)Q2.5 R2: McNemar \(\chi^2{=}1.80\), \(p{\approx}0.18\). Q3.5: both negative, not tested.

5.6.0.1 Cross-pool replication.

The null replicates across both policies: on Q2.5 (\(66\%\)) the critic is an exact null and the PRM gains \(+0.45\) pp (n.s.); on Q3.5 (\(81\%\)) both turn negative (PRM: \(51\) corrected vs.\(55\) regressed; critic: \(37\) vs.\(59\)). As pool accuracy grows, the majority-wrong tail shrinks and a near-balanced selector’s net effect turns negative (Appendix 9).

5.6.0.2 Why both selectors fail.

The critic exhibits self-recognition bias [13]: \(40\%\) of chains receive the modal score \(0.75\), and the correct/incorrect gap is negligible (\(0.713\) vs.\(0.687\); Appendix 9). The PRM improves the correction rate (\(37.6\%\) vs.\(31.8\%\)), partially mitigating self-preference, but cannot distinguish right from wrong overrides: the score gap between its top-ranked chain and runner-up is \(0.049\) on corrections and \(0.048\) on degradations, yielding a net effect indistinguishable from noise.

5.7 Stratified analysis↩︎

Figure 2 and Table 10 (Appendix 14) report per-language accuracy under both policies, with the Q3.5 column split by token budget to isolate the effect of the budget increase.

5.7.0.1 The budget increase concentrates gains on English and Chinese.

English gains \(+17.3\) pp and Chinese \(+8.5\) pp when the budget doubles; under Q2.5, over a third of English questions had all eight chains unparseable. Most other languages gain only \(1\)–\(2\) pp and French (\(92\%\)) is unchanged, confirming a targeted rather than general effect.

5.7.0.2 Persistent below-average strata.

Chinese, Polish, English, and Arabic remain below \(81.5\%\) even after the budget increase. Under Q2.5, SC regresses on Chinese by \(2.8\) pp: a chain-agreement audit (Appendix 10) shows \(22.3\%\) of Chinese questions reach unanimous \(8/8\) agreement (vs.\({\le}15\%\) elsewhere). SC gains power when chains err independently; when errors are highly correlated, the vote amplifies the shared mistake. Languages benefiting most from SC (Bulgarian, Croatian, Hungarian, Serbian) sit at mid-accuracy where the independence condition holds.

5.7.0.3 Per-subject patterns.

Per-subject accuracy (Appendix 11) reveals a split: STEM subjects benefit most from TTS (Physics \(+14\) pp, Mathematics \(+20\) pp over zero-shot), while the weakest Q3.5 subjects (Social \(43\%\), Islamic Studies \(57.8\%\)) are text-heavy. Using subject as a proxy for visual complexity (EXAMS-V lacks content-type annotations), the pattern suggests two regimes: in STEM, reasoning errors vary across chains and majority vote aggregates them away; in the weakest subjects, the bottleneck is missing domain knowledge that no amount of sampling can compensate.

5.8 Held-out test performance↩︎

Our best configuration (Qwen3.5-4B at SC-\(N{=}16\), \(2{,}048\)-token budget, with guided repair) ranks first on the official Visual MCQ leaderboard at \(84.06\%\) overall, and is first on each of the six test languages (Table 12, Appendix 16).

The test split (\(n{=}1{,}117\)) is \(44.8\%\) English and \(30.6\%\) Chinese, and both exceed their validation accuracy: English rises to \(85.6\%\) (from \(73.5\%\) val, \(+12.1\) pp) and Chinese to \(78.1\%\) (from \(74.7\%\), \(+3.4\) pp). These are the two languages where the token-budget and guided-repair fixes (Sections 5.3, 5.4) had the most headroom on validation, so the test result tracks with the validation diagnosis.

6 Conclusion↩︎

We studied test-time scaling for open VLMs on multilingual visual MCQ, comparing structured search against flat self-consistency across two policies on the full EXAMS-V validation set. TTS improves accuracy: the full progression from zero-shot to self-consistency spans over 24 pp on Qwen3.5-4B, but the gains concentrate on the per-chain token budget rather than on structured search, larger chain counts, or trained selectors: neither a generative critic nor a trained PRM beats majority vote across two policy models and two pool strengths. Stratified analysis confirms that TTS gains concentrate on mid-accuracy languages where the policy is miscalibrated rather than weak, and that persistent below-average strata (Polish, Islamic Studies) resist SC because chains converge on the same wrong answer, leaving no diversity for the majority vote to correct.

The overarching lesson is that at our budget scale, the engineering substrate (parser format, decoding budget) and the policy choice dominate any search or selector method. Our best configuration, Qwen3.5-4B at SC-\(N{=}16\) with a \(2{,}048\)-token budget and guided repair (Section 5.4), reaches \(81.6\%\) on full validation and \(84.1\%\) on the held-out ImageCLEF 2026 test split, ranking first on the Visual MCQ leaderboard (Section 5.8). Future work should investigate whether decoupling perception from reasoning via a dedicated reasoning model can address the P(\(+\)) asymmetry identified in Section 5.2.0.1, and whether better-calibrated PRMs trained on multilingual non-mathematical content would break the selector null.

7 Limitations↩︎

Our findings rest on two policy models, Qwen2.5-VL-7B-Instruct and Qwen3.5-4B, both from the same family, so we cannot separate behavior typical of small open VLMs from behavior specific to Qwen models. The parseability effect in particular depends on how a model emits answer letters under a given prompt format, which may differ for other open source VLM families. The 11.4 pp gain from switching policy supports our claim that the policy substrate dominates inference-time strategy, but also shows that a third model could shift the picture; whether the pattern holds across families and across the sub-7B range is untested. Ablations whose full-validation cost was prohibitive used the dev-200 subset, whose \(\pm\)​7 pp Wilson CIs preclude ranking claims, so all headline findings rest on val-full re-runs (n=4,651). The segmented PRM null appears structural, since P(\(+\)) \({\approx}0.85\) on reasoning steps leaves little signal once aggregated, though better per-step calibration on multilingual mixed-subject content might change this. Following the 2025 ImageCLEF winners [24], we did not fine-tune, use few-shot exemplars, or apply test-time image augmentation, all of which remain open.

Declaration on Generative AI↩︎

During the preparation of this work, the authors use AI for proofreading, polishing and rephrasing sentences and paragraphs in the manuscript.

8 Chain-Agreement Analysis↩︎

Table 5 reports per-tier accuracy on the Q3.5 SC-\(N{=}8\) (2k-tok) pool. The agreement tier is the number of chains (out of 8) that commit the same answer letter.

The skip rule used by both selectors (Section 5.6) passes through every question with at least \(5/8\) agreement, i.e.the unanimous, strong, and moderate tiers (\(73.9\%\) of validation). Only the weak and all-unparseable tiers (\(26.1\%\)) are rescored. On this scored subset the base accuracy is \({\sim}37\%\), leaving limited room for any selector to improve on majority vote.

9 Generative Critic Score Distribution↩︎

The training-free generative critic scores each chain on three axes (step intent, visual alignment, logical soundness), each rated \(1\)–\(5\), mapped to \([0,1]\) via \((s{-}1)/4\), and mean-aggregated. Table 6 shows the score distribution across 5,710 scored chains from the Q2.5 SC-\(N{=}8\) pool.

The correct-vs-incorrect chain score gap is negligible: correct chains average \(0.713\) vs.\(0.687\) for incorrect (gap \(0.026\)), with an identical median of \(0.75\). In \(59\%\) of questions, multiple chains are tied at the top score, forcing arbitrary tie-breaking. The pattern is consistent with the self-recognition bias reported by [13]: the policy rates its own outputs uniformly high regardless of correctness.

10 Per-Language Chain Agreement↩︎

Table 7 breaks down chain agreement by language for the Q2.5 SC-\(N{=}8\) pool, revealing the structural differences that drive per-language selector performance.

2.5pt

English performs worst under Q2.5: only \(14\%\) unanimous, \(34\%\) all-unparseable, \(36\%\) weak agreement; \(70\%\) of English questions fall in the worst two tiers, explaining why English is the only language where SC regresses against zero-shot under Q2.5. Arabic and Hungarian have the largest weak-agreement tails (\(36\%\) and \(30\%\)), making them the languages where selectors have the most questions to rescore and the least signal to work with. Bulgarian, French, and Italian show consistent agreement distributions (\(39\)–\(45\%\) unanimous, small weak tails, \(80\)+ % SC accuracy).

11 Per-Subject Accuracy↩︎

Table 8 reports accuracy for the five weakest and five strongest subjects under the best Q3.5 and Q2.5 configurations respectively.

2.5pt

\(^*\)CoT \(>\) SC under Q2.5 (regression). \(^\downarrow\)SC regresses vs.ZS. “–” \(=\) Q3.5 per-subject data unavailable for non-bottom-5 subjects at time of writing.

12 PRM Scoring Mode (One-shot vs Per-step)↩︎

On a DTR \(N{=}4{,}M{=}4\) pool under Qwen2.5-VL-7B (dev-200), per-step (segmented) PRM scoring is an exact null (\(63.0\% \to 63.0\%\)) while one-shot scoring shows a directional \(+3.0\) pp gain (\(63.0\% \to 66.0\%\)) and is \(9.4\times\) faster (Table 9). However, the \(\pm 7\) pp Wilson CIs at \(n{=}200\) overlap fully, so the difference is not significant. One-shot evaluation gives the PRM maximal context to judge chain quality holistically, avoiding the per-step error compounding diagnosed in Section 5.2.

3.5pt

13 Generative Critic Prompt↩︎

The critic scores each chain independently on three axes, one inference call per axis. The prompt template is:

You are a rigorous evaluator reviewing a reasoning chain written by a student answering a multiple-choice exam question. The question and its answer options are shown in the image.

Rate the chain on ONE axis: {axis_name}.

Axis definition: {axis_definition}

Scale: 1 (severely deficient), 2 (weak), 3 (adequate), 4 (strong), 5 (excellent).

Reasoning chain to evaluate:
—
{chain_text}
—

Output ONLY a JSON object on a single line with exactly two fields:
{"score": <integer 1-5>, "reason": "<brief one-sentence justification>"}
Do not output anything else.

The three axes are:

Per-axis scores are mapped to \([0,1]\) via \((s{-}1)/4\) and mean-aggregated to a single chain score.

14 Per-Language Accuracy↩︎

2.5pt

15 Dev-200 Search Comparison↩︎

3pt

16 Official Leaderboard↩︎

4pt

References↩︎

---

1. Code available at: https://github.com/lesterpjy/tts-small-vlm.↩︎