Abstract

Despite Bengali being the sixth most spoken language in the world, no prior work has systematically evaluated hallucination in large language models (LLMs) for Bengali. We introduce BenHalluEval, a fine-grained hallucination evaluation framework for Bengali covering four tasks: Generative Question Answering (GQA), Bangla-English Code-Mixed QA, Summarization, and Reasoning. We construct 12,000 hallucinated candidates using GPT-5.4 across twelve task-specific hallucination types, drawn from three existing Bengali datasets. We evaluate seven LLMs spanning reasoning-oriented, multilingual, and Bengali-centric categories under a dual-track protocol that independently measures false-positive rate on ground-truth instances (Track A) and hallucination detection rate on hallucinated candidates (Track B). To jointly penalise both failure modes and prevent inflated scores from uniform response bias, we propose BenHalluScore, a dual-track calibration metric. BenHalluScore ranges from 7.72% to 55.42% across models and tasks, revealing substantial variation in hallucination calibration. Chain-of-thought prompting, applied as a mitigation strategy, shifts response distributions without consistently improving hallucination discrimination. BenHalluEval establishes the first dedicated hallucination benchmark for Bengali and highlights the inadequacy of single-track and prompting-only evaluation approaches for low-resource language settings. The dataset and code are available at https://anonymous.4open.science/r/BanglaHalluEval-EB77.

1 Introduction↩︎

Large language models (LLMs) generate outputs that contradict established facts, fabricate details, or conflict with the provided context, known as hallucination [1], [2]. A medical chatbot that invents drug dosages, a legal assistant that cites non-existent cases, or a tutoring system that presents wrong arithmetic can cause real harm. A model that scores well on standard Natural Language Understanding benchmarks can still hallucinate heavily in generation [3]. Without a comprehensive hallucination benchmark, there is no reliable way to assess how much an LLM can actually be trusted.

Figure 1: An example of hallucination detection in BenHalluEval.

Hallucination evaluation has been explored in different languages such as English [2] Arabic [4], Hindi and Turkish [5]. Bengali, being the sixth most spoken language in the world [6], remains underexplored. Dedicated pretrained models [7], multilingual benchmarks [8], and task-specific datasets [9]–[11] already exist for Bengali, yet no benchmark measures hallucination in LLMs on both native Bengali and Bangla-English code-mixed input.

To fill this gap, we introduce BenHalluEval, a hallucination evaluation framework for Bengali covering four tasks: Generative Question Answering (GQA), Bangla-English Code-Mixed QA, Summarization, and Reasoning. We use GPT-5.4² to generate hallucinated candidates, evaluate seven LLMs across three model categories, and test chain-of-thought prompting as a mitigation strategy.

Our main contributions are:

We present BenHalluEval, a multi-task hallucination evaluation framework for LLMs on Bengali and Bangla-English inputs, covering Generative Question Answering (GQA), Bangla-English code-mixed QA, summarization, and mathematical reasoning.
We construct 12,000 hallucinated candidates using GPT-5.4 across task-specific hallucination categories, including a Bangla-English code-mixed setting, and evaluate them alongside ground-truth instances using an LLM-as-judge framework.
We introduce BenHalluScore to jointly measure false positives and missed hallucinations, benchmark seven LLMs across three model categories, and evaluate chain-of-thought prompting as a mitigation strategy.

Hallucination Evaluation Benchmarks. TruthfulQA, one of the earliest works on hallucination evaluation [12], showed that model scale alone does not guarantee factual accuracy. FActScore advanced this by breaking long-form outputs into atomic claims for fine-grained precision measurement [3]. HaluEval established the sampling-then-evaluation paradigm across QA, dialogue, and summarization [2]. FaithBench [13] and HalluLens [14] subsequently refined faithfulness annotations and guarded against benchmark leakage. MedHallu extended this to the clinical domain, showing how domain specificity amplifies the impact of hallucination errors [15]. A shared limitation across these benchmarks is labelling bias where LLM judges tend to default to one verdict class regardless of content, a problem none of them directly measure [2], [16].

Low-Resource Hallucination and Bengali NLP. Hallucination rates scale inversely with pretraining data availability, with the gap widening in open-ended generation [17], [18]. AraHalluEval [4] evaluated Arabic-capable LLMs using fine-grained hallucination indicators, and Halluverse-M$^3$ [5] extended multilingual evaluation to English, Arabic, Hindi, and Turkish. Neither benchmark incorporates dual-track calibration or examines code-mixed input. More recent work covers multi-domain understanding via BnMMLU [19], reasoning via MathMist [20], and summarization consistency via BanglaSummEval [21]. Bangla-English code-mixed usage is well-documented [22]–[24], yet hallucination behaviour on Bangla-English input remains unexamined.

Mitigation Strategies. Chain-of-thought prompting [25] improves multi-step inference but can paradoxically obscure hallucination cues, making its benefit task- and model-dependent [26]. [26] show that CoT reasoning can shift a model’s prediction patterns without improving its ability to distinguish hallucinated from correct content, with gains limited to tasks that require structured, verifiable intermediate steps.

No prior work has built a hallucination evaluation framework for Bangla. BenHalluEval addresses this by covering four tasks: GQA, Bangla-English Code-Mixed, QA, Summarization, and Reasoning under a unified dual-track evaluation protocol.

Figure 2: BenHalluEval system pipeline. Four tasks (GQA, Code-Mixed QA, Summarization, Reasoning) share a common evaluation stage. Track A evaluates ground-truth correct answers; Track B evaluates hallucinated candidates.

3 Methodology↩︎

BenHalluEval is constructed through a three-stage pipeline: (i) seed dataset selection and sample filtering, (ii) hallucinated sample generation using GPT-5.4, and (iii) hallucination detection evaluation using seven LLMs. Figure 2 provides a full system overview. Each task has a task-specific selection strategy and hallucination categorization; the evaluation protocol is unified across all four tasks.

3.1 Stage 1: Seed Dataset Selection and Sample Filtering↩︎

3.1.0.1 Generative Question Answering.

We use TyDiQA-GoldP (Bengali) [9], a dataset of 2,509 question–passage–answer triples. To select the samples that are more prone to hallucination, we prompted three models zero-shot with only the question, without the passage: Qwen2.5-14B³, Gemma-2-9B⁴, and DeepSeek-R1-14B⁵. A Qwen2.5-32B-Instruct⁶ judge scores each response as correct or incorrect. Samples that all three models answer incorrectly form the hard subset, yielding 1,313 instances (52.33%), from which 1,000 are randomly selected as the GQA seed dataset. To verify judge reliability, 100 instances were manually annotated and compared against the judge verdicts, finding high agreement. Table 1 shows per-model zero-shot accuracy.

10pt

Table 1: Zero-shot accuracy of three models on TyDiQA-GoldP (Bengali, 2,509 instances).
Model	Incorrect	Err.(%)
DeepSeek-R1-14B	1,747	30.37
Gemma-2-9B	1,668	33.54
Qwen2.5-14B	1,948	22.36
All 3 incorrect	1,313	52.33

3.1.0.2 Bangla-English Code-Mixed QA.

Bangla-English code-mixed text is widely used by Bengali speakers on social media and messaging platforms [22]–[24]. To evaluate hallucination detection in this setting, we take the same 1,000 GQA samples and convert the questions from Bengali to Bangla-English code-mixed using GPT-5.4 (see Appendix 10.5). Original answers are kept in Bengali. No extra filtering is needed, since these samples are already the hard subset from the GQA selection. To verify conversion quality, we manually reviewed 100 randomly selected samples. All authors are native Bengali speakers, and the reviewed samples were found to be natural and linguistically appropriate.

3.1.0.3 Summarization.

We use BanglaCHQ-Summ [10], a dataset of 1,800 Bengali consumer health query–summary pairs. We filter the samples based on semantic similarity using BERTScore F1. The same three models generate zero-shot summaries for all 1,800 samples, which are then scored against the original summaries using BERTScore F1 [27]. The average BERTScore F1 is computed per sample across all three models. The 1,000 samples with the lowest average scores are selected, as these are the samples that all three models struggle most to summarize faithfully.

3.1.0.4 Reasoning.

We use SOMADHAN [11], a dataset of 8,792 Bengali math word problems with manually written chain-of-thought (CoT) solutions. Samples are ranked by the length of their reasoning chain.The 500 longest and 500 shortest reasoning chains are selected as the two sampling pools. For each of the five error types, 100 samples with the longest reasoning chains and 100 with the shortest are selected, yielding 1,000 samples in total (200 per type). To ensure equal coverage, samples are picked alternately from the long and short pools in a fixed round-robin order, so that each error type receives an equal number of long and short reasoning chains.

3.2 Stage 2: Hallucinated Sample Generation↩︎

All hallucinated samples are generated using GPT-5.4. Each prompt specifies the hallucination type, includes an example of hallucination, and formats the target sample with field markers.

3.2.0.1 GQA Hallucination Generation.

Four hallucination types are applied to each of the 1,000 GQA seed samples, one API call per type (full prompt is in Appendix 10.2). This generates 4,000 hallucinated candidates in total:

Factualness: Fabricates a concrete fact such as a name, date, or place that contradicts the context.
Comprehension: Misunderstands the question’s context or intention and answers a different question.
Specificity: Answers at an inappropriate level of detail, either too vague or too specific.
Inference: Draws a conclusion that cannot be logically inferred from the given information.

3.2.0.2 Bangla-English Code-Mixed QA Hallucination Generation.

The same four hallucination types used for GQA are applied to the 1,000 Bangla-English Code-mixed seed samples. The generation process mirrors the GQA pipeline exactly, with one difference: both the prompt and the generated hallucinated answers are in Bangla-English (Roman-script Bengali) rather than native Bengali script (full prompt is in Appendix 10.2).

Table 2: Hallucination generation prompt for the GQA task. Four methods correspondto factual contradiction, comprehension mismatch, inappropriate specificity, andincorrect inference errors. Each instance generates one hallucinated answer permethod under controlled constraints. Full prompt with Bengali demonstrations isin Appendix 9.
I want you act as a hallucination answer generator. The answer should be given in BANGLA. Given a question, right answer, and related knowledge, your objective is to write a hallucinated answer that sounds plausible but is factually incorrect. You SHOULD write the hallucinated answer using the following method:
1.You are trying to answer a question but there is a factual contradiction between the answer and the knowledge. You can fabricate some information that does not exist in the provided knowledge.
$\langle$Demonstrations$\rangle$
2.You are trying to answer a question but you misunderstand the question context and intention.
$\langle$Demonstrations$\rangle$
3.You are trying to answer a question but the answer is too general or too specific to answer the question at an appropriate level of specificity.
$\langle$Demonstrations$\rangle$
4.You are trying to answer a question but the answer cannot be inferred from the knowledge. You can incorrectly reason with the knowledge to arrive at a hallucinated answer.
$\langle$Demonstrations$\rangle$
You should try your best to make the answer become hallucinated. #Hallucinated Answer# can only have about 5 more words than #Right Answer#.
#Knowledge#: $\langle$insert the related knowledge/context$\rangle$
#Question#: $\langle$insert the question$\rangle$
#Right Answer#: $\langle$insert the right answer to the question$\rangle$
#Hallucinated Answer#: Generate

3.2.0.3 Summarization Hallucination Generation.

Three hallucination types are applied to each of the 1,000 summarization seed instances, generating 3,000 hallucinated summaries in total:

Fabricated Content: Introduces a detail that does not exist anywhere in the source document.
Non-Factual Addition: Inserts a plausible but unverifiable claim not supported by the source.
Direct Contradiction: Reverses information that is explicitly stated in the source.

Full prompt is in Appendix 10.3.

3.2.0.4 Reasoning Hallucination Generation.

We apply five error types, one per sample, selecting 1,000 hallucinated CoT chains. GPT-5.4 is prompted to modify exactly one reasoning step per instance and return output as a structured JSON object (full prompt in Appendix 10.4):

Arithmetic Slip: Silently alters a single numerical computation by a small amount.
Formula Misapplication: Substitutes a related but incorrect mathematical operation.
Invalid Deduction: Injects a logically unsupported transition between two reasoning steps.
Hallucinated Intermediate Fact: Introduces a fabricated assumption mid-chain as if it were given.
Semantic Drift: Reinterprets the question mid-reasoning so the final answer addresses a different goal.

Table 3: BenHalluEval benchmark statistics.”Hallu.” = hallucinated candidates generated by GPT-5.4;“Gold” = ground-truth correct instances (Track A evaluation).
Task	Types	Seed	Hallu.	Gold
GQA	4	1,000	4,000	1,000
Code-mix	4	1,000	4,000	1,000
Summarization	3	1,000	3,000	1,000
Reasoning	5	1,000	1,000	1,000
Total			12,000	4,000

3.3 Stage 3: Hallucination Detection Evaluation↩︎

3.3.0.1 Evaluation tracks.

Evaluating only on hallucinated candidates risks rewarding models that blindly predict “Yes” for every input, while evaluating only on ground-truth samples risks rewarding models that always predict “No”. To get a more accurate picture of model behaviour, we evaluate each task on two separate tracks. Track A presents 1,000 ground-truth correct answers, where the expected verdict is “No” (not hallucinated). Track B presents the hallucinated candidates, where the expected verdict is “Yes”. This separation allows hallucination detection rate and false-positive rate to be measured independently.

To measure performance across both tracks jointly, we introduce BenHalluScore (BHS). A good model must both detect hallucinated content and avoid flagging correct answers as hallucinated. Neither track alone captures this. BenHalluScore penalises both failure modes equally, so a higher score indicates worse overall calibration. Formally:

\[\text{BenHalluScore} = \frac{1}{2} \!\left(\frac{W_A}{N_A} + \frac{W_B}{N_B}\right)\! \times 100\%\] where $W_A$ and $W_B$ are the number of wrong verdicts on Track A and Track B respectively, $N_A$ and $N_B$ are the total instances per track, and the factor $\frac{1}{2}$ normalises the combined error to the $[0, 100]$ range. A score of 0% indicates perfect calibration on both tracks; 50% indicates random or uniformly biased behaviour; 100% indicates perfectly wrong predictions across both tracks.

5pt

Table 4: BenHalluScore results across all four tasks and seven models (all values in %).A-err.: Track A error rate on ground-truth instances.B-err.: Track B error rate on hallucinated candidates. green cell = best (lowest) BHS per task; red cell = worst (highest) BHS per task.Full per-task results with raw counts in Appendix 12.
Model	GQA			Code-Mixed QA			Summarization			Reasoning
2-4(lr)5-7(lr)8-10(lr)11-13	A-err.	B-err.	BHS	A-err.	B-err.	BHS	A-err.	B-err.	BHS	A-err.	B-err.	BHS
DeepSeek-R1-14B	48.00	29.20	38.60	40.10	13.78	26.94	17.60	6.73	12.17	19.90	16.30	18.10
GPT-4.1 mini	13.90	17.23	15.56	24.00	19.80	21.90	8.20	11.37	9.79	31.70	21.10	26.40
Qwen2.5-32B	27.90	65.80	46.85	17.40	30.60	24.00	4.70	10.73	7.72	24.80	28.80	26.80
Gemma-2-27B	16.40	78.50	47.45	19.80	38.15	28.98	4.20	31.40	17.80	5.50	72.60	39.05
Mistral-nemo-12B	87.70	19.98	53.84	85.90	24.93	55.42	70.50	9.87	40.19	1.60	97.00	49.30
LLaMA-3.1-8B	13.80	81.63	47.71	13.80	78.05	45.93	31.80	25.77	28.79	17.40	89.20	53.30
TigerLLM-9B	18.40	38.17	28.28	25.80	37.28	31.54	4.80	47.53	26.17	21.40	85.50	53.45

4 Experimental Setup↩︎

4.0.0.1 Models.

We evaluate seven LLMs spanning three categories (Table 5). The reasoning-oriented category comprises DeepSeek-R1-14B and GPT-4.1 mini⁷. The multilingual category includes Qwen2.5-32B-Instruct, Gemma-2-27B⁸, Mistral-nemo-12B⁹, and LLaMA-3.1-8B-Instruct¹⁰. The Bangla-centric category is represented by TigerLLM-9B [28].

Table 5: Seven LLMs evaluated in BenHalluEval, grouped bymodel category.
Category	Model	Params
Reasoning	DeepSeek-R1-14B	14B
	GPT-4.1 mini	—
Multilingual	Qwen2.5-32B-Instruct	32B
	Gemma-2-27B	27B
	Mistral-nemo-12B	12B
	LLaMA-3.1-8B-Instruct	8B
Bangla-centric	TigerLLM-9B	9B

4.0.0.2 Inference Settings.

All models produce binary verdicts (yes/no) at temperature $= 0$, with a maximum output length of 1,024 tokens. Each instance is presented with its associated context - the passage for GQA and Bangla-English code-mixed QA, the source article for summarisation, and the problem statement for reasoning. GQA, summarisation, and reasoning prompts are issued in Bengali and the code-mixed prompts are in Bangla-English. GPT-4.1 mini is accessed via the OpenAI API. All remaining models are served locally using Ollama on a PC equipped with 128 GB RAM and an NVIDIA RTX 3090 GPU (24 GB VRAM). Full evaluation prompts are provided in Appendix 11.

5 Results Analysis↩︎

5.1 Generative Question Answering↩︎

Table 4 shows GQA results under the dual-track BenHalluScore. GPT-4.1 mini achieves the best BenHalluScore (15.56%), maintaining low errors on both tracks (A-err.%, B-err.%) - the only model that performs well simultaneously on both. DeepSeek-R1-14B scores 38.60%, with a notably high Track A error of 48.00%, meaning it incorrectly flags nearly half of all correct answers as hallucinated. Mistral-nemo-12B scores the worst (53.84%), driven almost entirely by its 87.70% Track A error - it flags nearly every ground-truth answer as hallucinated. LLaMA-3.1-8B shows the opposite pattern, predicting hallucination on 81.63% of Track B instances while maintaining a low Track A error (13.80%), reflecting opposing systematic biases rather than content-sensitive judgements. Notably, TigerLLM-9B, the only Bengali-centric model evaluated, achieves the second-best BenHalluScore on GQA (28.28%), outperforming significantly larger multilingual models including Qwen2.5-32B, Gemma-2-27B, and even the reasoning-oriented DeepSeek-R1-14B. This suggests that targeted Bengali pretraining provides a meaningful calibration advantage on native-script input despite the model’s smaller parameter count.

5.2 Bangla-English Code-Mixed QA↩︎

Bangla-English code-mixed results differ from GQA for several models (Table 4), showing that script condition interacts with model behaviour. GPT-4.1 mini remains the best (BHS 21.90%), and Mistral-nemo-12B (55.42%) and LLaMA-3.1-8B (45.93%) retain their characteristic biases in the Roman-script condition. However, Qwen2.5-32B, Gemma-2-27B, and DeepSeek-R1-14B improve sharply on Bangla-English code-mixed relative to GQA (by 22.85, 18.47, and 11.66 BHS points respectively), driven primarily by lower Track A errors. A plausible explanation is that Roman-script input shifts these models toward their stronger English-anchored representations, producing more consistent verdicts than native Bengali script - the opposite of the intuitive expectation that code-mixing would degrade performance.

5.3 Summarization↩︎

Across all models, BenHalluScores on summarization are notably lower than on GQA (Table 4), suggesting that access to a full source article provides stronger contextual basis for factual consistency judgements. Qwen2.5-32B achieves the lowest (best performing) BenHalluScore (7.72%), with low errors on both tracks (A-err.%, B-err.%), while DeepSeek-R1-14B ranks second (12.17%) despite recording the lowest Track B error (6.73%). Mistral-nemo-12B again scores worst (40.19%), driven by a 70.50% Track A error. TigerLLM-9B shows the opposite pattern - a 47.53% Track B error against a 4.80% Track A error (BHS 26.17%) - indicating that labelling bias persists even when the model has access to sufficient context.

5.4 Reasoning↩︎

Among all four tasks, reasoning shows the widest spread of BenHalluScore, ranging from 18.10% to 53.45% (Table 4). DeepSeek-R1-14B achieves the best and most balanced result (A-err.%, B-err.%, BHS 18.10%). At the other end, Mistral-Nemo-12B predicts “No” for nearly every instance regardless of content (A-err.%, B-err.%, BHS 49.30%), a pattern indicating uniform label collapse. LLaMA-3.1-8B and TigerLLM-9B occupy the opposite extreme with Track B errors of 89.20% and 85.50% respectively, resulting in the worst BenHalluScores (53.30% and 53.45%). The concentration of models near or above 50% BenHalluScore indicates that reasoning hallucination in Bengali is particularly difficult to detect.

5pt

Table 6: CoT mitigation results for Mistral-nemo-12B and LLaMA-3.1-8Bacross three tasks (all values in %).A-err.: Track A error rate on ground-truth instances.B-err.: Track B error rate on hallucinated candidates.Before-CoT values are from the zero-shot baseline;After-CoT values reflect step-by-step prompting. $\downarrow$ = best (lowest) overall BHS; $\uparrow$ = worst (highest) overall BHS.
		Before CoT			After CoT
3-5(lr)6-8 Model	Task	A-err.	B-err.	BHS	A-err.	B-err.	BHS
Mistral-nemo-12B	GQA	85.90	24.93	55.42	93.30	5.90	49.60 $\downarrow$
	Summarization	70.50	9.87	40.19	45.40	14.10	29.75 $\downarrow$
	Reasoning	1.60	97.00	49.30	0.10	99.80	49.95 $\uparrow$
LLaMA-3.1-8B	GQA	13.80	78.05	45.93	83.30	12.90	48.10 $\uparrow$
	Summarization	31.80	25.77	28.79	64.90	21.80	43.35 $\uparrow$
	Reasoning	17.40	89.20	53.30	28.70	75.60	52.15 $\downarrow$

5.5 Chain-of-Thought Mitigation↩︎

Chain-of-thought (CoT) prompting has been shown to improve reasoning and reduce hallucination in LLMs by extracting step-by-step explanations before a final answer [25]. We apply CoT to the two worst-performing models in our evaluation (Table 4) - Mistral-nemo-12B and LLaMA-3.1-8B, which recorded the highest BenHalluScores across multiple tasks like GQA, summarisation, and reasoning tasks (Table 6).

Across the six model-task combinations, CoT results in a lower BenHalluScore in only three instances. However, two of these apparent gains - Mistral-nemo-12B on GQA (55.42$\rightarrow$49.60) and summarization (40.19$\rightarrow$29.75) - do not constitute genuine improvement. Track B error falls sharply but Track A error simultaneously rises above 90%, indicating that the model has simply swapped its uniform “No” posture for a uniform “Yes” posture. The one substantive improvement is LLaMA-3.1-8B on the reasoning task (53.30$\rightarrow$52.15), where Track B error decreases from 89.20% to 75.60%. However, a modest Track A deterioration can also be seen which suggests that step-by-step prompting provides limited benefit even when the task involves multi-step reasoning.

6 Discussion↩︎

6.0.0.1 Necessity of dual-track calibration.

Single-track frameworks, including AraHalluEval [4], assess models only on hallucinated candidates, leaving unexamined whether a model recognises correct outputs as non-hallucinated. BenHalluEval addresses this by introducing Track A alongside Track B, and aggregating both into BenHalluScore. Our results justify this design - models such as Mistral-nemo-12B appear competitive on Track B alone, yet collapse entirely on Track A, a failure mode that single-track evaluation would conceal.

6.0.0.2 GQA vs.Code-Mixed QA.

The Bangla-English Code-Mixed QA task shares its hallucination taxonomy and candidate instances with GQA, differing only in question script (Bengali to Bangla-English). Despite this structural equivalence, BenHalluScores diverge across both tasks (Table 4). Most multilingual models achieve better results on Bangla-English Code-Mixed QA, whereas TigerLLM-9B and GPT-4.1 mini perform more favourably on Bengali GQA. This indicates that the input script has a measurable influence on hallucination judgement even when the underlying content is identical.

6.0.0.3 Language-specific advantage.

Despite having the fewest parameters in our suite, TigerLLM-9B achieves competitive BenHalluScores on GQA and summarisation, outperforming several larger multilingual models. We attribute this to Bangla-centric pretraining rather than scale, suggesting that language-specific optimisation is a more reliable predictor of Bangla hallucination detection performance than model size.

6.0.0.4 Limits of prompting-based mitigation.

CoT prompting gives divergent outcomes across the six model-task pairs examined. Exactly three pairs show a reduction in BenHalluScore after CoT while the remaining three worsen, as visualised in Figure 4. Results differ across models and tasks, with no clear pattern of improvement. We understand from our findings that response biases persist across model categories, and that prompting strategies alone are insufficient as a hallucination mitigation approach in Bengali.

7 Conclusion↩︎

Reliable hallucination evaluation for Bengali has remained absent despite the language’s global reach. We introduced BenHalluEval, a multi-task hallucination evaluation framework covering four tasks with 12,000 hallucinated candidates, evaluated across seven LLMs under a dual-track protocol. To prevent inflated scores from uniform response bias, we proposed BenHalluScore, a calibration metric that jointly penalises false positives on ground-truth instances and missed detections on hallucinated candidates. Our evaluation reveals wide variation in hallucination calibration, with BenHalluScore ranging from 7.72% to 55.42% across models and tasks. In several cases, low error on one track is accompanied by high error on the other, a failure mode that single-track evaluation would miss. Bengali-centric pretraining proves a stronger predictor of calibration quality than parameter count, and input script measurably influences model behaviour. We further tested chain-of-thought prompting as a candidate mitigation strategy, finding that step-by-step reasoning alters prediction patterns but does not consistently improve hallucination discrimination across models and tasks. Future work should examine few-shot and instruction-tuned evaluation protocols and extend the dual-track design to other low-resource languages with comparable code-mixed usage patterns.

Limitations↩︎

While we conduct manual validation for selected components of the benchmark, the hallucinated candidates are generated automatically using GPT-5.4 rather than fully authored by human annotators. As a result, the benchmark may still reflect generation-specific patterns that differ from naturally occurring hallucinations in deployed systems. BenHalluScore assigns equal weight to errors on ground-truth and hallucinated tracks to reduce the influence of systematic response tendencies, but deployment settings with asymmetric costs may require task-specific weighting. All models are evaluated zero-shot at temperature 0, so results may differ under few-shot prompting or model-specific prompt tuning. The chain-of-thought mitigation experiment is limited to two models and three tasks, preventing broad conclusions about its effectiveness across all evaluated models and tasks. Finally, BenHalluEval is scoped to four Bengali and Bangla-English task settings; generalization to other low-resource languages, Bengali genres, domains, and more complex mathematical reasoning remains to be validated.

Acknowledgements↩︎

The authors acknowledge the use of ChatGPT for grammatical error checking only. All scientific content, analyses, and conclusions are solely the work of the authors.

8 Hallucination Type Definitions and Examples↩︎

Table 7 lists all hallucination types used across the three generation tasks, with a definition and a translated English example for each. Bangla-English Code-Mixed QA shares the same four types as GQA and is omitted here for brevity.

4pt

Table 7: Hallucination type definitions and examples across GQA,Summarization, and Reasoning. Bengali examples are translated toEnglish for readability.
Task	Type	Definition	Example
GQA	Factualness	Fabricates a concrete fact (name, date, number, place) not present in the context.	Q: When was the Padma Bridge inaugurated? Ground-Truth: 2022 Hallucinated: The Padma Bridge was inaugurated in 2019.
	Comprehension	Misunderstands the question context or intention.	Q: What is the capital of Bangladesh? Ground-Truth: Dhaka Hallucinated: Bangladesh is a South Asian country.
	Specificity	Answers at an inappropriate level of detail.	Q: How many players are in a cricket team? Ground-Truth: 11 Hallucinated: Some.
	Inference	Draws a conclusion not supported by the given information.	Q: Where is Cox’s Bazar located? Ground-Truth: Chittagong Division Hallucinated: Cox’s Bazar is the capital of Bangladesh.
Summ.	Fabricated Content	Introduces a detail not present anywhere in the source.	Context: Patient reports mild fever for two days. Hallucinated: Patient has been hospitalised for a week.
	Non-Factual Addition	Inserts a plausible but unverifiable claim into the summary.	Context: Child has a recurring cough. Hallucinated: Child likely has bronchial asthma.
	Direct Contradiction	Reverses information explicitly stated in the source.	Context: Pain improves when walking. Hallucinated: Pain does not improve with walking.
Reasoning	Arithmetic Slip	A single numerical computation is silently altered.	Correct: $25 \times 8 = 200$ hours Hallucinated: $25 \times 8 = 180$ hours
	Formula Misapplication	A related but incorrect operation is substituted.	Correct: $200 \times 5 = 1000$ Hallucinated: $200 + 5 = 205$
	Invalid Deduction	A logically unsupported transition is injected between steps.	Correct: Tax $= 20000 \times 0.1 = 2000$ Hallucinated: Tax is split per pair of employees.
	Hallucinated Intermediate Fact	A fabricated assumption is introduced mid-chain as if given.	Correct: Working days $= 25$ Hallucinated: Workers take 2 days off, so working days $= 23$.
	Semantic Drift	The reasoning drifts to answer a different question than asked.	Correct: Total cost $=$ wages $+$ tax Hallucinated: Total $=$ wages only $=$ $20,000.

9 Benchmark Statistics by Hallucination Type↩︎

Table 8 reports the number of hallucinated candidates generated per type and task, totalling 12,000 instances across the full benchmark.

Table 8: Hallucinated candidate counts by task and type.
Task	Hallucination type	Count
GQA	Factualness	1,000
	Comprehension	1,000
	Specificity	1,000
	Inference	1,000
Code-mix	Factualness	1,000
	Comprehension	1,000
	Specificity	1,000
	Inference	1,000
Summ.	Fabricated content	1,000
	Non-factual addition	1,000
	Direct contradiction	1,000
Reasoning	Arithmetic slip	200
	Formula misapplication	200
	Invalid deduction	200
	Hallucinated intermediate fact	200
	Semantic drift	200
Total		12,000

10 Hallucination Generation Prompts↩︎

The following prompts were used to generate hallucinated candidates via GPT-5.4. Each prompt specifies a hallucination method, includes a demonstration, and formats the target instance with field markers. Demonstrations are shown in Bengali for GQA and Reasoning, and in Latin-script Bangla-English for the Code-Mixed task.

10.1 GQA Generation Prompt↩︎

Table 9 shows the full GQA generation prompt with Bengali demonstrations for all four methods.

Table 9: GQA hallucination generation prompt. One API call per method per seed instance.The same structure is used for Code-Mixed QA with Bangla-English output and demonstrations.
[System instruction] I want you act as a hallucination answer generator. The answer should be given in BANGLA. Given a question, right answer, and related knowledge, your objective is to write a hallucinated answer that sounds plausible but is factually incorrect using one of the following methods:
Method 1 — Factualness: You are trying to answer a question but there is a factual contradiction between the answer and the knowledge. You can fabricate some information that does not exist in the provided knowledge.
Demonstration: #Knowledge#: “উইলিয়াম আব্রাহাম সাইমন ঔডারল্যান্ড (জন্ম: ৬ ডিসেম্বর, ১৯১৭ — মৃত্যু: ১৮ই মে, ২০০১) ছিলেন একজন ওলন্দাজ-অস্ট্রেলীয় সামরিক কমান্ডো অফিসার...” #Q#: “উইলিয়াম কবে জন্মগ্রহণ করেন?” #Gold#: “৬ ডিসেম্বর, ১৯১৭” #Hallucinated#: “৬ নভেম্বর, ১৯১৬”
Method 2 — Comprehension: You are trying to answer a question but you misunderstand the question context and intention.
Demonstration: #Q#: “স্বাধীন বাংলাদেশের প্রথম চলচ্চিত্রটির নাম কী?” #Gold#: “সুকুমারী” #Hallucinated#: “জহির রায়হান”
Method 3 — Specificity: You are trying to answer a question but the answer is too general or too specific.
Demonstration: #Q#: “কুয়েটের মোট ছাত্রছাত্রীর সংখ্যা কত?” #Gold#: “প্রায় ৬ হাজার” #Hallucinated#: “অজানা”
Method 4 — Inference: You are trying to answer a question but the answer cannot be inferred from the knowledge.
Demonstration: #Q#: “ঢাকার মোট আয়তন কত?” #Gold#: “১৩৪ বর্গমাইল” #Hallucinated#: “২০ মিলিয়ন জনসংখ্যা”
#Hallucinated Answer# can only have about 5 more words than #Right Answer#. #Knowledge#: $\langle$context$\rangle$ #Question#: $\langle$question$\rangle$ #Right Answer#: $\langle$gold$\rangle$ #Hallucinated Answer#: Generate

10.2 Code-Mixed QA Generation Prompt.↩︎

Table 10 mirrors the GQA prompt but instructs the model to generate hallucinated answers in Latin-script Bangla-English throughout.

Table 10: Code-Mixed QA hallucination generation prompt. Four methods mirror the GQA promptbut use Latin-script Bangla-English throughout.
[System instruction] I want you act as a hallucination answer generator. The answer should be given in CODE-MIXED BANGLA (Bangla-English: Latin-script Bengali). Given a question, right answer, and related knowledge, write a hallucinated answer that sounds plausible but is factually incorrect using the four methods below.
Method 1 — Factualness: Fabricate information not present in the knowledge.
Demo: #Knowledge#: “William Abraham Simon Ouderland (janma: 6 december, 1917, mrittu: 18 may, 2001) chhilen ekjon commando officer.” #Q#: “Ouderland kobe janmogrohon koren?” #Gold#: “6 december, 1917” #Hallucinated#: “6 november, 1916”
Method 2 — Comprehension: Misunderstand the question context and intention.
Demo: #Q#: “Swadhin Bangladesher prothom cholocchitrotir nam ki?” #Gold#: “Sukumari” #Hallucinated#: “Jahir Raihan”
Method 3 — Specificity: Answer at an inappropriate level of specificity.
Demo: #Q#: “bortomane kuet-er mot chatrochhatrir shongkha koto?” #Gold#: “pray 6 hajar” #Hallucinated#: “ojana”
Method 4 — Inference: Draw a conclusion not supported by the knowledge.
Demo: #Q#: “Dhakar mot ayoton koto?” #Gold#: “134 borgomail” #Hallucinated#: “20 million jonoshonkha”
#Hallucinated Answer# can only have about 5 more words than #Right Answer#. #Knowledge#: $\langle$context$\rangle$ #Question#: $\langle$question$\rangle$ #Right Answer#: $\langle$gold$\rangle$ #Hallucinated Answer#: Generate

10.3 Summarization Generation Prompt.↩︎

Table 11 shows the summarization generation prompt covering three hallucination methods.

Table 11: Summarization hallucination generation prompt. Three methods correspond tofabricated content, non-factual addition, and direct contradiction.
[System instruction] I want you act as a hallucination summary generator. The summary should be given in BANGLA. Given a document and the right summary, write a hallucinated summary that sounds plausible but is factually incorrect using one of the following methods:
Method 1 — Fabricated Content: You fabricate information that does not exist in the document.
Demo: #Doc#: “আমার মেয়ের বয়স ৫ বছর। সর্দি সমস্যা।” #Gold#: “বয়স ৫, সর্দি সমস্যা, করণীয় কী” #Hallucinated#: “বয়স ৫, জ্বর ও কাশি সহ হাসপাতালে ভর্তি প্রয়োজন”
Method 2 — Non-Factual Addition: Insert information that cannot be inferred from the document.
Demo: #Doc#: “স্ত্রীর বয়স ২২, ডেট ৭ নভেম্বর থেকে ১৪ তারিখে পরিবর্তন।” #Gold#: “স্ত্রীর বয়স ২২, ডেট পরিবর্তন” #Hallucinated#: “ডাক্তার জুলাই মাসে ডেট দিয়ে পরে ১৪ করেছে”
Method 3 — Direct Contradiction: Introduce a clear contradiction between summary and document.
Demo: #Doc#: “হাঁটলে ব্যথা ঠিক হয়ে যায়।” #Gold#: “হাঁটলে ঠিক হয়” #Hallucinated#: “হাঁটলেও ব্যথা কমে না”
#Hallucinated Summary# can only have about 5 more words than #Right Summary#. #Document#: $\langle$doc$\rangle$ #Right Summary#: $\langle$gold$\rangle$ #Hallucinated Summary#: Generate

10.4 Reasoning Generation Prompt↩︎

Table 12 shows the reasoning generation prompt. GPT-5.4 is instructed to modify exactly one step and return output as a structured JSON object.

Table 12: Hallucination generation prompt for the Reasoning task.Five methods correspond to arithmetic slip, formula misapplication,invalid deduction, hallucinated intermediate fact, and semantic drift.
[System instruction]
I want you to act as a hallucinated reasoning chain generator. The output must be in BANGLA. Given a math problem, a correct step-by-step reasoning chain, and the correct final answer, your objective is to produce a hallucinated reasoning chain that sounds plausible but is mathematically incorrect. You MUST use one of the following hallucination methods:
Method 1 — Arithmetic Slip:
Keep the logic structure completely intact but insert one subtle numerical error in a single calculation step. The wrong number should be close to the correct one (roughly 5% to 20% off). All subsequent steps must use the wrong number consistently. Preserve the <<expr=result>> annotation format.
Demonstration:
#Correct step#: “২৫ দিন * ৮ ঘন্টা/দিন = <<২৫*৮=২০০>>২০০ ঘন্টা”
#Hallucinated step#: “২৫ দিন * ৮ ঘন্টা/দিন = <<২৫*৮=১৮০>>১৮০ ঘন্টা”
Method 2 — Formula Misapplication:
Use a related but wrong operation in one step. The Bengali narrative must justify the wrong operation naturally. All subsequent steps must be consistent with the wrong result.
Demonstration:
#Correct step#: “২০০ ঘন্টা * ৳৫/ঘন্টা = ৳<<২০০*৫=১০০০>>১০০০”
#Hallucinated step#: “২০০ + ৫ = ৳<<২০০+৫=২০৫>>২০৫”
Method 3 — Invalid Deduction:
Inject one logically unsupported conclusion in a single step. All subsequent steps must follow from this invalid conclusion.
Demonstration:
#Correct step#: “৳২০,০০০ * .১ = ৳<<২০০০০*.১=২০০০>>২,০০০”
#Hallucinated step#: “যেহেতু কর্মচারীর সংখ্যা জোড় (৬ জন), তাই প্রতিটি জুটির জন্য আলাদাভাবে হিসাব করতে হবে। প্রতি জুটির মজুরি = ৳২০,০০০ $\div$ ৩ = <<২০০০০/৩=৬৬৬৭>>৳৬,৬৬৭।”
Method 4 — Hallucinated Intermediate Fact:
Introduce one fabricated assumption not present in the problem as if it naturally follows. All subsequent steps must proceed from this fabricated fact.
Demonstration:
#Correct step#: “২৫ দিন * ৮ ঘন্টা/দিন = <<২৫*৮=২০০>>২০০ ঘন্টা”
#Hallucinated step#: “কর্মীরা প্রতি মাসে ২ দিন ছুটি ভোগ করেন, তাই কার্যকর কাজের দিন = ২৫ - ২ = <<২৫-২=২৩>>২৩ দিন। মোট ঘন্টা = ২৩ * ৮ = <<২৩*৮=১৮৪>>১৮৪ ঘন্টা।”
Method 5 — Semantic Drift:
Slightly reinterpret what the problem is asking mid-reasoning so the final answer addresses a different question. The drift must feel like a plausible reading of the problem.
Demonstration:
#Correct step#: “৳২,০০০ + ৳২০,০০০ = ৳<<২০০০+২০০০০=২২০০০>>২২,০০০”
#Hallucinated step#: “যেহেতু প্রশ্নটি মূলত মজুরির মোট পরিমাণ জিজ্ঞেস করছে, তাই মোট মজুরি ৳২০,০০০ই চূড়ান্ত উত্তর।”
#Problem#: $\langle$insert Bengali math problem$\rangle$
#Correct Reasoning Chain#: $\langle$insert ground-truth CoT$\rangle$
#Correct Answer#: $\langle$insert ground-truth answer$\rangle$
#Hallucinated Chain#: Generate

10.5 Bengali-to-Bangla-English Conversion Prompt↩︎

Table 13 shows the Bengali-to-Bangla-English conversion prompt used to produce the code-mixed seed instances from the GQA questions.

Table 13: Prompt used to convert the 1,000 GQA seed instances from Bengalito Bangla-English. Applied separately to questions and knowledge passages.Ground-Truth answers are retained in their original Bengali form.
[System instruction] Convert Bengali QA text into natural Bangla-English — Bengali–English code-mix written entirely in Latin script, the way people write on South Asian social media or in casual chats. Output only Latin script - no Bengali characters anywhere. Output only the converted text. No labels, no explanations.
Rules: (1) Romanise Bengali words naturally; use English words where code-mix speakers naturally would. (2) Do NOT translate everything to English; preserve Bengali phrasing where it feels natural. (3) Preserve meaning, tone, names, numbers, dates, brands, and technical terms exactly.
Spelling variation (apply to $\sim$15% of eligible words): common Bangla-English alternates: bhai/vai, bhalo/valo, ache/ase, ki/kee, onek/anek, ektu/aktu, shob/sob, nai/nei, hoise/hoiche.
Abbreviations (apply sparingly, $\sim$10% of eligible words): phn (phone), msg (message), prob (problem), pls/plz (please), bcz/coz (because).
Do not: over-abbreviate, use excessive expressive spellings, change factual meaning, or make output hard to read.
[Text to convert]: $\langle$Bengali question and/or passage$\rangle$

11 Hallucination Evaluation Prompts↩︎

The following prompts were used to elicit binary hallucination verdicts (yes/no) from the evaluated models. Zero-shot prompts request a single-token response; chain-of-thought prompts require step-by-step reasoning before the final verdict. All prompts are in English except for the input fields, which are in Bengali or Bangla-English depending on the task.

11.1 Zero-Shot Evaluation Prompts↩︎

Tables 16, [tab:eval95summ], and [tab:eval95reason] show the zero-shot evaluation prompts for GQA/Code-Mixed QA, Summarization, and Reasoning respectively.

You are an evaluator. Decide whether the provided model answer is hallucinated relative to the context and question. Only reply with a single token: yes or no. No explanation, no punctuation, no extra text. yes means the answer contains information not supported by the context or contradicts it. Provide the answer in English only.

Context: $\langle$context$\rangle$

Question: $\langle$question$\rangle$

Model answer: $\langle$answer$\rangle$

Answer now:

You are an evaluator. Decide whether the provided summary is hallucinated relative to the document. Only reply with a single word: yes or no. yes means the summary contains information not supported by or contradicting the document. Provide the answer in English only.

Document: $\langle$Bengali medical article$\rangle$

Summary: $\langle$candidate summary$\rangle$

Answer now (yes or no):

Table 14: Zero-shot evaluation prompt for GQA and Code-Mixed QA.For code-mixed instances, all fields are in Bangla-English.
You are an expert evaluator for Bengali mathematical reasoning tasks. Decide whether the provided reasoning chain is hallucinated. Reply with a single token only: yes or no. yes = chain or answer contains incorrect steps or unsupported facts. English only.
Problem: $\langle$Bengali math problem$\rangle$
Reasoning Chain: $\langle$CoT solution$\rangle$
Final Answer: $\langle$final answer$\rangle$
Answer now:

11.2 Chain-of-Thought Evaluation Prompts↩︎

Under the CoT condition, models are asked to reason through three steps before issuing a verdict. Tables 19, [tab:cot95summ], and [tab:cot95reason] show the corresponding prompts for GQA/Code-Mixed QA, Summarization, and Reasoning respectively.

You are an evaluator checking whether a model answer is hallucinated.

Question: $\langle$question$\rangle$ Model Answer: $\langle$answer$\rangle$

Analyze step by step:

Step 1: What factual claims does the answer make?

Step 2: Are these claims supported by or inferable from the question context?

Step 3: Based on steps 1–2, is the answer hallucinated?

Final answer (write only this word on the last line): yes or no

(yes = hallucinated, no = not hallucinated)

You are an evaluator checking whether a summary is hallucinated relative to a document.

Document: $\langle$document$\rangle$ Summary: $\langle$summary$\rangle$

Analyze step by step:

Step 1: List the key claims made in the summary.

Step 2: For each claim, check whether it is directly supported by the document.

Step 3: Based on steps 1–2, decide your final answer.

Final answer (write only this word on the last line): yes or no

(yes = hallucinated, no = not hallucinated)

Table 15: Chain-of-thought evaluation prompt for GQA and Code-Mixed QA.
You are an expert evaluator for Bengali mathematical reasoning tasks.
Question: $\langle$problem$\rangle$ Reasoning Chain: $\langle$CoT$\rangle$ Answer: $\langle$final answer$\rangle$
Analyze step by step:
Step 1: Is each calculation or logical step mathematically correct?
Step 2: Does the final answer follow logically from the reasoning chain?
Step 3: Based on steps 1–2, is this reasoning chain hallucinated?
Respond ONLY with a JSON object on the last line:
{"is_hallucinated": "Yes"} or {"is_hallucinated": "No"}

12 Full Per-Task Evaluation Results↩︎

The following tables extend Table 4 in the main paper by reporting raw verdict counts alongside error rates for all seven models across each of the four tasks. A-err.: Track A error rate on ground-truth instances. B-err.: Track B error rate on hallucinated candidates. BHS: BenHalluScore $= \frac{1}{2}$(A-err.$+$ B-err.).

12.1 Generative Question Answering↩︎

Table 20 shows the full GQA evaluation results across all seven models under the dual-track BanglaHalluScore.

Table 16: GQA full results. **Bold** = best and worst BHS.
		Track B — Hallucinated (4,000)			Track A — Ground Truth (1,000)
#	Model	Correct	Wrong	B-err.	Wrong	Correct	A-err.	BHS
1	Qwen2.5-32B-Instruct	1,365	2,632	65.80%	279	771	27.90%	46.85%
2	Gemma-2-27B	860	3,140	78.50%	164	836	16.40%	47.45%
3	DeepSeek-R1-14B	2,820	1,168	29.20%	480	342	48.00%	38.60%
4	GPT-4.1 mini	3,311	689	17.23%	139	861	13.90%	15.56%
5	Mistral-nemo-12B	3,201	799	19.98%	877	123	87.70%	53.84%
6	LLaMA-3.1-8B-Instruct	735	3,265	81.63%	138	862	13.80%	47.71%
7	TigerLLM-9B	2,855	1,145	38.17%	184	816	18.40%	28.28%

12.2 Summarization↩︎

Table 21 shows the full summarization evaluation results across all seven models under the dual-track BanglaHalluScore.

Table 17: Summarization full results. **Bold** = best and worst BHS.
		Track B — Hallucinated (3,000)			Track A — Ground Truth (1,000)
#	Model	Correct	Wrong	B-err.	Wrong	Correct	A-err.	BHS
1	Qwen2.5-32B-Instruct	2,678	322	10.73%	47	953	4.70%	7.72%
2	Gemma-2-27B	2,058	942	31.40%	42	958	4.20%	17.80%
3	DeepSeek-R1-14B	2,798	202	6.73%	176	824	17.60%	12.17%
4	GPT-4.1 mini	2,659	341	11.37%	82	918	8.20%	9.79%
5	Mistral-nemo-12B	2,704	296	9.87%	705	295	70.50%	40.19%
6	LLaMA-3.1-8B-Instruct	2,227	773	25.77%	318	682	31.80%	28.79%
7	TigerLLM-9B	1,574	1,426	47.53%	48	952	4.80%	26.17%

12.3 Reasoning↩︎

Table 22 shows the full mathematical reasoning evaluation results across all seven models under the dual-track BanglaHalluScore.

Table 18: Reasoning full results. **Bold** = best and worst BHS.
		Track B — Hallucinated (1,000)			Track A — Ground Truth (1,000)
#	Model	Correct	Wrong	B-err.	Wrong	Correct	A-err.	BHS
1	Qwen2.5-32B-Instruct	703	288	28.80%	248	752	24.80%	26.80%
2	Gemma-2-27B	274	726	72.60%	55	945	5.50%	39.05%
3	DeepSeek-R1-14B	837	163	16.30%	199	801	19.90%	18.10%
4	GPT-4.1 mini	789	211	21.10%	317	683	31.70%	26.40%
5	Mistral-nemo-12B	30	970	97.00%	16	984	1.60%	49.30%
6	LLaMA-3.1-8B-Instruct	108	892	89.20%	174	826	17.40%	53.30%
7	TigerLLM-9B	145	855	85.50%	214	786	21.40%	53.45%

12.4 Bangla-English Code-Mixed QA↩︎

Table 23 shows the full Code-Mixed Banglish QA evaluation results across all seven models under the dual-track BanglaHalluScore.

Table 19: Bangla-English Code-Mixed QA full results. **Bold** = best and worst BHS.
		Track B — Hallucinated (4,000)			Track A — Ground Truth (1,000)
#	Model	Correct	Wrong	B-err.	Wrong	Correct	A-err.	BHS
1	Qwen2.5-32B-Instruct	2,776	1,224	30.60%	174	826	17.40%	24.00%
2	Gemma-2-27B	2,474	1,526	38.15%	198	802	19.80%	28.98%
3	DeepSeek-R1-14B	3,439	551	13.78%	401	596	40.10%	26.94%
4	GPT-4.1 mini	3,208	792	19.80%	240	760	24.00%	21.90%
5	Mistral-nemo-12B	3,003	997	24.93%	859	141	85.90%	55.42%
6	LLaMA-3.1-8B-Instruct	878	3,122	78.05%	138	862	13.80%	45.93%
7	TigerLLM-9B	2,496	1,491	37.28%	258	739	25.80%	31.54%

12.5 GQA Model Error Distribution↩︎

Figure 3 visualises Track A error, Track B error, and BenHalluScore (BHS) for all seven models on GQA, sorted by ascending BHS. The chart illustrates the contrasting bias profiles across model categories: GPT-4.1 mini achieves the best balance (BHS 15.56%), while Mistral-nemo-12B scores worst (BHS 53.84%), driven by its 87.70% Track A error. Models such as LLaMA-3.1-8B show the opposite pattern, with high Track B error and low Track A error, reflecting a systematic “Yes” bias rather than genuine hallucination discrimination.

Figure 3: Track A error, Track B error, and BenHalluScore (BHS)for all seven models on GQA, sorted by ascending BHS.GPT-4.1 mini achieves the best balance with the lowest BHS(15.56%). Mistral-nemo-12B scores worst (53.84%), drivenby its 87.70% Track A error.

13 Hallucination Indicators↩︎

Table 24 lists the full set of hallucination indicators used during annotation and quality review. Indicators marked with $^{*}$ are Bangla-specific and do not appear in prior English or Arabic hallucination benchmarks.

Table 20: Full hallucination indicator set.$^{*}$ = Bengali-specific indicator absent from prior benchmarks.
Indicator	Task(s)	Description
Named entity error	GQA, Summ.	Wrong person, place, or organisation
Value error	GQA, Summ.	Incorrect number, date, or quantity
Factual contradiction	GQA	Contradicts world knowledge
Response conflict	GQA	Internal inconsistency
Grammar error	GQA, Summ.	Meaning-altering mistake
Generic answer	GQA	Avoids committing to a fact
Source conflict	GQA	Merges incompatible facts
Fabrication	Summ.	Content not in source article
Inference error	Summ.	Unjustified conclusion
Context inconsistency	Summ.	Contradicts source directly
Code-switching	GQA, Summ.	Responds in English despite Bengali prompt
Transliteration err.$^{*}$	All	Incorrect Bengali$\leftrightarrow$Roman mapping
Computation err.$^{*}$	Reasoning	Wrong step or answer in CoT

Figure 4: BenHalluScore before and after CoT promptingfor Mistral-nemo-12B (M) and LLaMA-3.1-8B (L)across GQA, Summarization, and Reasoning.Lower is better.\downarrow = improvement;\uparrow = worsening. — Figure 4: BenHalluScore before and after CoT promptingfor Mistral-nemo-12B (M) and LLaMA-3.1-8B (L)across GQA, Summarization, and Reasoning.Lower is better.$\downarrow$ = improvement;$\uparrow$ = worsening.

References↩︎

[1]

L. Huang et al., “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” ACM Transactions on Information Systems, 2024, doi: 10.1145/3703155.

[2]

J. Li, X. Cheng, X. Zhao, J.-Y. Nie, and J.-R. Wen, “HaluEval: A large-scale hallucination evaluation benchmark for large language models,” in Proceedings of the 2023 conference on empirical methods in natural language processing, Dec. 2023, pp. 6449–6464, doi: 10.18653/v1/2023.emnlp-main.397.

[3]

S. Min et al., “FActScore: Fine-grained atomic evaluation of factual precision in long form text generation,” in Proceedings of the 2023 conference on empirical methods in natural language processing, Dec. 2023, pp. 12076–12100, doi: 10.18653/v1/2023.emnlp-main.741.

[4]

A. Alansari and H. Luqman, “AraHalluEval: A fine-grained hallucination evaluation framework for Arabic LLMs,” arXiv preprint arXiv:2509.04656, 2025, [Online]. Available: https://arxiv.org/abs/2509.04656.

[5]

S. Abdaljalil, P. Sharma, E. Serpedin, and H. Kurban, “Halluverse-M$^3$: A multitask multilingual benchmark for hallucination in LLMs,” arXiv preprint arXiv:2602.06920, 2026, [Online]. Available: https://arxiv.org/abs/2602.06920.

[6]

A. A. Sani, K. A. Zaoad, S. E. S. Adib, M. A. Muqtadir, and A. Abrar, “Addressing data scarcity in bangla fake news detection: An LLM-based dataset augmentation approach.” 2026, [Online]. Available: https://arxiv.org/abs/2605.01292.

[7]

A. Bhattacharjee et al., “BanglaBERT: Language model pretraining and benchmarks for low-resource language understanding evaluation in Bangla,” in Findings of the association for computational linguistics: NAACL 2022, Jul. 2022, pp. 1318–1327, doi: 10.18653/v1/2022.findings-naacl.98.

[8]

T. Hasan et al., “XL-sum: Large-scale multilingual abstractive summarization for 44 languages,” in Findings of the association for computational linguistics: ACL-IJCNLP 2021, Aug. 2021, pp. 4693–4703, doi: 10.18653/v1/2021.findings-acl.413.

[9]

J. H. Clark et al., “TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages,” Transactions of the Association for Computational Linguistics, vol. 8, pp. 454–470, 2020, doi: 10.1162/tacl_a_00317.

[10]

A. Khan, F. Kamal, M. A. Chowdhury, T. Ahmed, M. T. R. Laskar, and S. Ahmed, “BanglaCHQ-summ: An abstractive summarization dataset for medical queries in Bangla conversational speech,” in Proceedings of the first workshop on bangla language processing (BLP-2023), Dec. 2023, pp. 85–93, doi: 10.18653/v1/2023.banglalp-1.10.

[11]

S. Paul et al., “Leveraging large language models for Bengali math word problem solving with chain of thought reasoning,” arXiv preprint arXiv:2505.21354, 2025, [Online]. Available: https://arxiv.org/abs/2505.21354.

[12]

S. Lin, J. Hilton, and O. Evans, “TruthfulQA: Measuring how models mimic human falsehoods,” in Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: Long papers), May 2022, pp. 3214–3252, doi: 10.18653/v1/2022.acl-long.229.

[13]

F. Ping and G. Durrett, “FaithBench: A diverse hallucination benchmark for summarization by modern LLMs,” arXiv preprint arXiv:2410.13210, 2024, [Online]. Available: https://arxiv.org/abs/2410.13210.

[14]

Y. Bang et al., “HalluLens: LLM hallucination benchmark,” arXiv preprint arXiv:2504.17550, 2025, [Online]. Available: https://arxiv.org/abs/2504.17550.

[15]

S. Pandit et al., “Medhallu: A comprehensive benchmark for detecting medical hallucinations in large language models,” in Proceedings of the 2025 conference on empirical methods in natural language processing, 2025, pp. 2858–2873.

[16]

S. E. S. Adib, A. A. Sani, E. A. Esham, A. Abrar, and T. M. Chowdhury, “Assessing large language models for medical QA: Zero-shot and LLM-as-a-judge evaluation,” in 2025 28th international conference on computer and information technology (ICCIT), 2025, pp. 3081–3086, [Online]. Available: https://ieeexplore.ieee.org/document/11491589.

[17]

C. Chataigner, A. Taı̈k, and G. Farnadi, “Multilingual hallucination gaps in large language models,” arXiv preprint arXiv:2410.18270, 2024.

[18]

S. O. ul Islam, A. Lauscher, and G. Glavaš, “How much do LLMs hallucinate across languages? On realistic multilingual estimation of LLM hallucination,” arXiv preprint arXiv:2502.12769, 2025, [Online]. Available: https://arxiv.org/abs/2502.12769.

[19]

S. S. Joy and M. S. Islam, “BnMMLU: Measuring massive multitask language understanding in Bengali,” arXiv preprint arXiv:2505.18951, 2025, [Online]. Available: https://arxiv.org/abs/2505.18951.

[20]

M. E. Sobhani, Md. F. A. Sayeedi, T. Mohiuddin, M. M. Islam, and S. Shatabda, “MathMist: A parallel multilingual benchmark dataset for mathematical problem solving and reasoning,” arXiv preprint arXiv:2510.14305, 2025, [Online]. Available: https://arxiv.org/abs/2510.14305.

[21]

A. Rafid, R. Adib, F. Ahmed, A. Abrar, and M. S. Islam, “BanglaSummEval: Reference-free factual consistency evaluation for bangla summarization,” in Proceedings of the second workshop on language models for low-resource languages (LoResLM 2026), 2026, pp. 595–608.

[22]

S. Alam, M. F. Ishmam, N. H. Alvee, M. S. Siddique, M. A. Hossain, and A. R. M. Kamal, “Bnsentmix: A diverse bengali-english code-mixed dataset for sentiment analysis,” in Proceedings of the first workshop on language models for low-resource languages, 2025, pp. 68–77.

[23]

R. Sheth et al., “Beyond monolingual assumptions: A survey of code-switched NLP in the era of large language models across modalities,” arXiv preprint arXiv:2510.07037, 2025, [Online]. Available: https://arxiv.org/abs/2510.07037.

[24]

K. S. Y. Alam, M. T. Chowdhury, T. Ahmed, A. Abrar, and M. R. Haque, “MixSarc: A bangla-english code-mixed corpus for implicit meaning identification,” arXiv preprint arXiv:2602.21608, 2026.

[25]

J. Wei et al., “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Information Processing Systems, vol. 35, 2022.

[26]

J. Cheng et al., “Chain-of-thought prompting obscures hallucination cues in large language models: An empirical evaluation,” arXiv preprint arXiv:2506.17088, 2025.

[27]

T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi, “BERTscore: Evaluating text generation with BERT,” in International conference on learning representations, 2020, [Online]. Available: https://openreview.net/forum?id=SkeHuCVFDr.

[28]

N. Raihan and M. Zampieri, “TigerLLM-a family of bangla large language models,” in Proceedings of the 63rd annual meeting of the association for computational linguistics (volume 2: Short papers), 2025, pp. 887–896.

BenHalluEval: A Multi-Task Hallucination Evaluation Framework for Large Language Models on Bengali