1 Introduction↩︎

Question answering (QA) systems play a critical role in natural language processing (NLP), enabling users to interact with machines through natural language queries and receive concise and accurate answers. These systems are widely used in applications such as search engines, virtual assistants, and customer support platforms, and can operate over both structured and unstructured data sources [1]. With the advancement of large language models (LLMs), QA systems have become more capable of understanding complex queries, capturing contextual relationships, and generating fluent responses. This progress has significantly enhanced their applicability in real-world scenarios, particularly in domains that require efficient access to large volumes of textual information.

Despite these advancements, existing QA approaches face several limitations. One major challenge lies in the availability and quality of training data, as large-scale annotated datasets are expensive and time-consuming to construct [2]. In many cases, naturally occurring questions may be ambiguous or ill-formed, which further complicates the training process and affects model performance. Furthermore, QA systems often struggle in low-resource settings, where linguistic diversity and limited annotated data hinder generalisation, highlighting the need for more robust and adaptable models [3]. Although retrieval-augmented methods and specialised toolkits have been proposed to improve QA performance and streamline system development, they often lack flexibility in customisation and may introduce additional complexity in training and deployment pipelines [4]. Additionally, practical constraints such as computational cost, memory requirements, and latency remain significant barriers, especially when processing long contexts or handling multiple queries simultaneously [5].

Another critical set of challenges relates to the quality, reliability, and consistency of generated answers. Existing QA systems may exhibit sensitivity to noise, inconsistencies in responses, and a tendency to rely on parametric knowledge rather than the provided context, leading to incorrect or misleading outputs [6]. In extractive QA settings, models may correctly identify answer text but fail to associate it with the appropriate contextual span, which reduces answer accuracy [7]. Moreover, LLMs face difficulties in long-context reasoning, particularly when relevant information is located in the middle of the passage, commonly referred to as the “lost in the middle” problem [8]. While techniques such as similar question generation and knowledge augmentation have been introduced to improve robustness and coverage, they often require careful design and optimisation to balance efficiency and performance [9]. These limitations collectively highlight the need for approaches that improve both contextual understanding and answer precision.

To address these challenges, this work proposes a question answering system based on fine-tuning multiple LLMs on a benchmark dataset. The proposed approach focuses on improving contextual understanding and answer extraction by leveraging supervised learning with high-quality context–question–answer triplets. By training and evaluating several models under a unified framework, the study aims to systematically compare their performance and identify architectures that are more effective for QA tasks. In addition, the use of multiple evaluation metrics ensures a comprehensive assessment of both lexical accuracy and semantic similarity. This approach provides a practical and efficient solution for enhancing QA systems, particularly in scenarios that require accurate, consistent, and context-grounded responses.

2 Related Work↩︎

The rapid development of question answering (QA) systems has been largely driven by advances in large language models (LLMs), which enable systems to process natural language queries and generate concise and contextually relevant answers [1], [10]. These systems have been applied across both structured and unstructured data sources, supporting a wide range of applications such as search engines, conversational agents, and educational tools [1], [11]. Furthermore, recent frameworks and toolkits aim to streamline the development pipeline of QA systems by integrating data collection, preprocessing, fine-tuning, evaluation, and deployment into unified environments [4], [12]. Despite these improvements, building robust QA systems remains challenging due to the high cost of data annotation, ambiguity in natural queries, and the need for scalable and efficient training strategies [2], [10].

To address these challenges, several studies have explored enhancements to input representations and training methodologies. For instance, augmenting QA models with additional linguistic or answer-type features has been shown to improve performance, particularly in low-resource or linguistically complex settings [3], [13]. Similarly, transforming existing datasets into alternative query formats can reduce annotation costs while maintaining competitive performance [2], [14]. However, even with such improvements, LLM-based QA systems continue to face fundamental limitations, including hallucinations, bias, and high computational requirements [6], [10]. These issues are further exacerbated in long-context scenarios, where models struggle to retrieve relevant information effectively, often suffering from phenomena such as the “lost in the middle” problem [8], [15].

A significant body of work has focused on improving QA performance through the integration of external knowledge sources. Knowledge graphs (KGs) provide structured and verifiable information that can enhance answer accuracy, explainability, and trustworthiness [16], [17]. Hybrid approaches combining LLMs with symbolic reasoning systems further improve logical consistency and commonsense reasoning by transforming textual inputs into formal representations that support precise inference [18], [19]. In addition, unified frameworks for querying structured data sources have been proposed to support multiple data types while maintaining generalisation and reliability [20], [21]. These approaches demonstrate the importance of combining neural and symbolic methods to overcome the limitations of purely generative models.

Retrieval-based and retrieval-augmented generation (RAG) approaches represent another prominent direction in QA research. These methods enhance answer generation by incorporating relevant external documents during inference, thereby improving factual grounding and reducing hallucinations [4], [22]. However, retrieval processes are often sensitive to noise, bias, and irrelevant context, which can negatively impact performance [22]. To mitigate these issues, recent work has introduced advanced retrieval strategies, such as rationale-guided filtering and balanced multi-source retrieval, to improve the quality and diversity of retrieved information [22], [23]. Additionally, retrieval-based chatbots and similar question generation techniques have been explored to expand knowledge bases and improve system coverage while maintaining high reliability and user satisfaction [9], [24].

Another line of research investigates collaborative and multi-component QA systems, including multi-agent and ensemble-based approaches. Multi-agent frameworks decompose the QA task into subtasks such as planning, question understanding, retrieval, and answer generation, enabling more structured and effective reasoning [25], [26]. Ensemble learning techniques further improve performance by combining predictions from multiple models, leveraging their complementary strengths to achieve higher accuracy and robustness across different datasets [27], [28]. These approaches have shown particular effectiveness in specialised domains such as healthcare, where domain-specific reasoning and interpretability are essential [11], [29]. Nevertheless, they often introduce additional computational overhead and system complexity, which may limit their scalability in practical applications.

In parallel, research has also examined improvements in model architecture and training objectives to enhance contextual understanding. For example, context-aware mechanisms such as block attention and auxiliary context prediction tasks have been proposed to ensure that extracted answers are aligned with the correct context, even in passages containing multiple candidate answers [7], [30]. Additionally, efforts to improve robustness and consistency have highlighted the impact of noise and conflicting information on QA performance, revealing that many models remain sensitive to such perturbations [6], [31]. Addressing these limitations is critical for developing reliable QA systems that can operate effectively in real-world environments.

Overall, while significant progress has been made in LLM-based QA systems, existing approaches still face challenges related to reasoning, generalisation, context utilisation, and computational efficiency. These limitations motivate the need for simpler yet effective methodologies that focus on improving contextual comprehension and answer extraction through targeted fine-tuning and systematic evaluation. In this work, we build upon these directions by leveraging pre-trained LLMs and evaluating their performance under a unified framework, aiming to balance efficiency and accuracy in context-based question answering tasks.

3 Dataset↩︎

This study utilises the Stanford Question Answering Dataset (SQuAD1.1), a widely adopted benchmark for machine reading comprehension and extractive question answering tasks. SQuAD1.1 is constructed from a collection of Wikipedia articles, where questions are generated by human annotators (crowdworkers) based on the content of the passages. Each question is paired with an answer that corresponds to a specific text span within the associated context, making it suitable for evaluating a model’s ability to understand and extract relevant information from natural language text.

The dataset consists of more than 100,000 question–answer pairs derived from over 500 Wikipedia articles, covering a wide range of topics and domains. Each data instance is composed of three main components: a context paragraph, a question related to that paragraph, and one or more answers, where each answer is represented as a contiguous span of text within the context. This structured format enables supervised learning for extractive QA models, where the objective is to predict the start and end positions of the correct answer span in the given passage.

SQuAD1.1 is commonly divided into training and validation splits, The training set is used to fine-tune models, while the validation set is used for evaluation and hyperparameter tuning. In addition to the original version (SQuAD1.1 1.1), an extended version, SQuAD1.1 2.0, introduces over 50,000 unanswerable questions designed to resemble answerable ones, requiring models not only to extract correct answers but also to determine when no valid answer exists in the context.

One of the key strengths of SQuAD1.1 lies in its diversity and realism, as the questions are generated by humans rather than automatically constructed, resulting in varied linguistic patterns and reasoning types. This makes the dataset a challenging benchmark for evaluating contextual understanding, inference, and answer extraction capabilities. Consequently, SQuAD1.1 has become a standard dataset in the NLP community for developing and benchmarking QA systems, particularly those based on deep learning and large language models.

4 Methodology↩︎

The proposed question answering (QA) is based on fine-tuning multiple large language models (LLMs). The overall framework follows a supervised learning paradigm where each model learns to map a given textual context and question to a corresponding answer. The system is designed to evaluate and compare the performance of different transformer-based architectures under a unified training setup.

4.1 Problem Formulation↩︎

Let the dataset be defined as a collection of \(N\) training examples: \[\mathcal{D} = \{(c_i, q_i, a_i)\}_{i=1}^{N}\] where \(c_i\) represents the context paragraph, \(q_i\) denotes the associated question, and \(a_i\) is the ground truth answer extracted from the context.

As illustrated in Figure 1 the input to the model is constructed by concatenating the context and question: \[x_i = [c_i ; q_i]\] The objective of the QA system is to learn a mapping function: \[f_\theta : x_i \rightarrow a_i\] where \(f_\theta\) is parameterised by \(\theta\), representing the learnable weights of the model.

The predicted output is denoted as: \[\hat{a}_i = f_\theta(x_i)\]

4.2 Model Architecture↩︎

In this work, transformer-based language models are independently fine-tuned for the QA task.

Each model follows a transformer encoder-based architecture that processes the input sequence \(x_i\) and produces contextualised token representations: \[H_i = \text{Transformer}_\theta(x_i)\] where \(H_i\) denotes the hidden state representations for all input tokens.

4.3 Answer Representation↩︎

The QA task is formulated as an extractive problem, where the answer is assumed to be a continuous span within the given context. Therefore, the model predicts two values corresponding to the start and end positions of the answer span: \[\hat{a}_i = (s_i, e_i)\] where \(s_i\) and \(e_i\) denote the predicted start and end indices of the answer within the input sequence.

The probability distributions over start and end positions are computed as: \[P_{start}(t|x_i), \quad P_{end}(t|x_i)\] where \(t\) represents a token index in the input sequence.

4.4 Training Objective↩︎

The models are trained using a supervised learning objective that minimises the difference between predicted and ground truth answer spans. The total loss function is defined as: \[\mathcal{L} = \mathcal{L}_{start} + \mathcal{L}_{end}\]

where: \[\mathcal{L}_{start} = -\sum_{i=1}^{N} \log P_{start}(s_i|x_i)\] \[\mathcal{L}_{end} = -\sum_{i=1}^{N} \log P_{end}(e_i|x_i)\]

This formulation encourages the model to assign high probability to the correct start and end positions of the answer span.

4.5 Fine-Tuning Procedure↩︎

Each of the five LLMs is fine-tuned independently using the same training dataset and identical preprocessing steps. The input format remains consistent across all models, ensuring a fair comparison between architectures.

During training, parameters \(\theta\) are optimised using gradient-based optimisation to minimise the loss: \[\theta^* = \arg\min_{\theta} \mathcal{L}\]

This process allows each model to adapt its pre-trained representations to the QA task while preserving general linguistic knowledge.

4.6 Evaluation Metrics↩︎

After fine-tuning, each model generates predicted answers \(\hat{a}_i\) for the test set. These predictions are compared against the ground truth answers \(a_i\) using a set of automatic evaluation metrics that measure lexical overlap and semantic similarity.

The first metric is ROUGE-L, which evaluates the longest common subsequence between the predicted and reference answers, capturing structural similarity: \[\text{ROUGE-L}(\hat{a}_i, a_i)\]

The second metric is BLEU, which measures n-gram precision between generated and reference answers: \[\text{BLEU}(\hat{a}_i, a_i)\]

In addition, BERTScore is used to compute semantic similarity using contextual embeddings: \[\text{BERTScore}(\hat{a}_i, a_i)\]

These metrics collectively provide a comprehensive evaluation of model performance in terms of lexical accuracy, fluency, and semantic alignment.

4.7 Inference Process↩︎

During inference, the trained model receives a new input pair and generates the predicted answer.

The final answer is extracted from the context using the predicted start and end indices. This ensures that the output remains grounded in the provided passage and maintains factual consistency.

Overall, this methodology enables systematic evaluation of multiple transformer-based models under a unified QA framework, highlighting their effectiveness in contextual understanding, answer extraction, and semantic alignment with ground truth responses.

5 Results and Discussion↩︎

Experimental results are obtained by evaluating the baseline and fine-tuned models on the QA task. Performance is measured using ROUGE-L, BLEU, and BERTScore, which collectively assess lexical overlap and semantic similarity between the predicted and reference answers.

5.1 Baseline Performance↩︎

Table 2 summarises the performance of all models before fine-tuning. Overall, the results indicate generally low to moderate performance across the evaluation metrics, reflecting the limited capability of pre-trained models to perform context-specific question answering without task adaptation. Among the evaluated models, Stablelm-2 achieves the highest performance, with a ROUGE-L score of 59.75%, BLEU of 19.36%, and BERTScore of 85.24%. This suggests that larger or more recent architectures may retain stronger generalisation capabilities for QA tasks even without fine-tuning.

It is also observed that BERTScore values are consistently higher than ROUGE-L and BLEU across all models. This indicates that, although the generated answers may not closely match the exact wording of the ground truth, they still retain a certain degree of semantic similarity. However, the overall results confirm that baseline models remain insufficient for accurate QA without fine-tuning.

5.2 Fine-Tuned Performance↩︎

Table 3 presents the results after fine-tuning the models on the QA dataset. A substantial improvement is observed across all models and evaluation metrics, demonstrating the effectiveness of task-specific training.

The best overall performance is achieved by Roberta-base, which attains the highest scores in all metrics, with ROUGE-L of 86.84%, BLEU of 28.24%, and BERTScore of 95.38%. Albert-base and Bert-base also achieve very high performance, indicating strong capability in accurate answer generation and alignment with the ground truth.

Models such as Distilbert-base, Stablelm-2, and Qwen2.5 show strong improvements after fine-tuning, achieving competitive results across all metrics. Phi, Bloom, and SmolLM2 also demonstrate substantial gains, highlighting the importance of fine-tuning even for models with moderate baseline performance. Additionally, Electra-small shows a significant improvement, achieving competitive BERTScore.

Despite these improvements, smaller models such as Bert-tiny continue to lag behind larger counterparts, suggesting that model capacity plays a crucial role in capturing complex contextual relationships required for QA tasks.

5.3 Comparative Analysis↩︎

A comparison between Tables 2 and 3 clearly highlights the impact of fine-tuning. All models exhibit substantial increases in ROUGE-L, BLEU, and BERTScore, confirming that supervised training enables better alignment between predicted and reference answers.

In particular, ROUGE-L and BLEU scores show the most significant improvements, indicating enhanced lexical overlap and more precise answer generation. BERTScore also improves consistently across all models, suggesting that fine-tuned models produce answers that are not only lexically accurate but also semantically meaningful.

Furthermore, the relative ranking of models changes notably after fine-tuning. While Stablelm-2 and Qwen2.5 perform best in the baseline setting, they are surpassed by encoder-based models such as Roberta-base, Albert-base, and Bert-base after fine-tuning. This indicates that pre-training alone is not a reliable indicator of QA performance, and that adaptability to the task plays a critical role.

Overall, the results demonstrate that fine-tuning is essential for achieving high-performance QA systems. The findings also suggest that, although smaller models can benefit from fine-tuning, larger and more expressive architectures tend to deliver superior results in terms of both accuracy and semantic quality.

6 Conclusion↩︎

This work presented a question answering (QA) framework based on fine-tuning multiple large language models (LLMs) on a benchmark dataset. The study aimed to improve contextual understanding and answer extraction by adapting pre-trained models to the QA task using supervised learning. A unified training and evaluation setup was employed to ensure a fair comparison across different model architectures.

The experimental results demonstrated that fine-tuning significantly enhances model performance across all evaluation metrics, including ROUGE-L, BLEU, and BERTScore. All models showed substantial improvements compared to their baseline counterparts, confirming the importance of task-specific training. In particular, models such as Albert-base and Bert-base achieved the highest performance, indicating their strong capability in capturing contextual relationships and generating accurate answers.

The comparative analysis further revealed that model capacity and architecture play a critical role in QA performance. While lightweight models benefit from fine-tuning, they generally lag behind larger models in handling complex queries and producing precise responses. Additionally, the results highlighted that relying solely on pre-trained knowledge is insufficient for high-quality QA, and that fine-tuning is essential for achieving reliable and consistent outputs.

Overall, the findings confirm that leveraging LLMs with targeted fine-tuning provides an effective solution for context-based question answering tasks. Future work may explore advanced techniques such as retrieval-augmented generation, multi-model fusion, or domain-specific adaptation to further enhance performance and robustness across diverse QA scenarios.

References↩︎