1 Introduction↩︎

In 2014, Stuart Russell introduced the “Value Alignment Problem" in an interview [1], emphasizing the need to construct AI systems that are not just intelligent but also aligned with human values. Aligning LLM responses to user preferences has been widely studied since then. The alignment problems specifically identify the dangers of generating harmful, inaccurate, or unsafe responses.

While most bodies of work on the alignment problem mostly talk about Artificial General Intelligence tasks, we find that real-world scenarios or enterprise applications necessitate the need for contextual alignment, specifically when the users are multi-disciplinary stakeholders.

We define contextual alignment as a model’s ability to align its responses to the specific context of conversation, which could imply the domain/ discipline of the user, what role they play in their domain, what their background knowledge/ domain expertise is on, or very specifically to the exact topic of conversation.

Aligning LLM responses to the context specific to the user within the topic of conversation becomes a necessity when building a conversational model for a niche domain. We deem it necessary to augment existing datasets to provide directed context for them to be more useful in domain applications has an impact on real-world/ enterprise cases. In this paper, we illustrate the importance/ significance of contextual alignment through a very critical, real-world use case of implementing the Multi-Tier System of Supports (MTSS) framework in schools for Behavioral Health.

We evaluate the performance of 10 language models: Phi-4-mini-instruct (3.8B), Deepseek R1, Qwen3 (32B), Gemma 2 (9B), Gemini1.5, LLaMA 3.1 (8B) - Instruct, LLaMA 3.2 (1B) - Instruct, LLaMA 4 Maverick (7B), LLaMA 4 Scout (17B), and Mistral 7B Instruct. in the question-answering downstream task for a niche domain such as MTSS for Behavioral Health (MTSS(B)).
The key contributions of this paper are as follows:

• We propose a new task called Contextual Alignment for Niche Domains Question Answering (CANDI-QA).

• We present a consolidated and comprehensive dataset consisting of two types of questions:

  • T1: Information Assistance Questions

  • T2: Applied Inference Questions

• We perform systematic evaluations on state-of-the-art (SOTA) language models for question answering and establish a benchmark to evaluate performance in the CANDI-QA setting.

• We provide a baseline solution leveraging neuro-symbolic AI approaches. A simple solution that is implemented through a rule-based or logic-based methodology called the MTSS-Net framework.

2 Related Works↩︎

2.1 From General to Domain-Specific Question Answering↩︎

The evolution of Question Answering (QA) systems has progressed from open-domain architectures—where general knowledge corpora and benchmarks dominate—to increasingly specialized frameworks built for domain-specific or context-sensitive tasks. Early datasets and benchmarks such as SQuAD, Natural Questions, and TriviaQA fostered precision in retrieving and synthesizing fact-based answers from broad knowledge sources [2], [3]. However, these resources provided limited support for scenarios requiring nuanced, role- or domain-aware comprehension.

Recent efforts have targeted the limitations of standard large language models (LLMs) when applied to specialized or niche domains. Closed-book and retrieval-augmented approaches, such as the two-stage context generation method, enhance LLM responses by conditioning on personalized or contextually generated input [4], [5]. These techniques have demonstrated improvements in exploiting parameterized knowledge for QA, narrowing the performance gap between general-domain and context-rich settings.

2.2 Construction of Domain-Specific QA Datasets↩︎

Building robust QA datasets tailored for niche domains typically demands new methodologies. Recent work emphasizes semi-automatic data acquisition, leveraging generative models for question and/or answer generation, with targeted human annotation to ensure domain fidelity and cost-effective scaling. For example, Falk et al. (2024) demonstrate that selective manual labeling combined with automated augmentation yields superior adaptation and higher QA performance compared to fully manual or fully synthetic datasets [6].

Notably, M-QALM and DomainCQA introduce methodologies for flexible, scalable dataset generation in complex scientific and technical domains [7], [8]. They distinguish between fundamental, fact-based questions and advanced, inference-driven or context-dependent questions. Both utilize LLMs not only for data generation, but also for annotation and evaluation, reducing reliance on scarce domain experts and enabling research at scale.

2.3 Annotation, Validation, and Evaluation in Niche Domains↩︎

Ensuring quality and reliability in domain-specific QA datasets requires multi-stage validation. Manual annotation by domain experts, selective bootstrapping, and cross-dataset error analysis are commonly employed [9], [10]. Human-centric qualitative metrics—such as factuality, completeness, and hallucination rate—are increasingly adopted alongside traditional metrics (e.g., BLEU, ROUGE, F1) to judge alignment with expert gold standards and real-world needs.

Furthermore, rubric-based and reference-free evaluation metrics, as implemented in frameworks like RAGAS, enable nuanced judgment of contextual correctness and domain alignment [11]. These approaches are crucial in settings where literal answer matching misses contextual appropriateness or logical validity.

2.4 Neuro-Symbolic Architectures as Baselines↩︎

Hybrid QA models integrating LLMs with retrieval components and symbolic reasoning engines are gaining traction for their enhanced interpretability and modularity. For instance, neuro-symbolic approaches—such as NSQA and recent hierarchical paradigms—combine neural models for question understanding with logic-based components for rule-driven reasoning, supporting transparent, expert-understandable decision making [12], [13].

2.5 Benchmarking and Performance in Contextual Alignment↩︎

Benchmarks now encompass both standard and domain-adapted datasets, assessing LLMs - from compact open-source variants to proprietary giants [14], [15]. Comparative studies consistently show (a) retrieval-augmented and role-aware prompting strategies outperforming vanilla LLMs, and (b) significant headroom remaining for models to achieve expert-level contextual reasoning, especially on highly-specific or inferential queries [7], [10].

3 Problem Formulation↩︎

3.1 Domain - Implementing MTSS for Behavioral Health↩︎

The core MTSS(B) is a multidisciplinary team that includes leaders at both district and school levels who coordinate to align practices, programs, and personnel to implement services at three levels of impact:

• Tier 1: School wide/ Universal level

• Tier 2: Targeted, small group level

• Tier 3: Intensive individual care

The Multi-Tiered System of Supports (MTSS) framework involves both school-level and district-level teams comprising diverse stakeholders. School-level teams typically include administrators, mental health professionals (licensed and non-licensed), teachers, counselors, nurses, resource officers, community partners, caregivers, and interns or coaches at select sites. District-level teams feature executive leaders such as superintendents, directors of student services, special education, and mental health, as well as representatives from state agencies, families, and community organizations. School teams are responsible for identifying and supporting students needing academic, behavioral, or social-emotional interventions through data-driven assessment, tiered support, progress monitoring, and collaborative problem-solving. District teams oversee policy, resource allocation, fidelity monitoring, professional development, and data analysis to support and ensure consistent MTSS implementation across schools.

Although state-mandated laws exist for a majority of US states, the effective implementation of MTSS in schools is challenging and complex because the MTSS framework involves numerous components and requires a collaborative effort from a multidisciplinary team to track tier-assignment related information.

Teams often face limitations in their awareness and capacity to implement best practices effectively, and turn to experts or coaches for support. However, the availability of such coaching support is typically limited, with most coaches operating at the district level and serving large numbers of schools—resulting in a significant gap between the recommended and actual coach-to-school ratios. This disparity underscores the potential role of AI-based tools in augmenting human capacity and providing scalable, real-time support for MTSS teams.

3.2 Task Definition - CANDI-QA↩︎

For effective intervention, the MTSS Team usually has questions that can be categorized as

1. T1, Information assistance questions - where the answers are available and accessible through expert curated literature on best-practices, evidence-based practices, MTSS literacy etc.

2. T2, Applied inference questions - where the answers are not directly available in the literature, but need understanding, experience, and reasoning to mitigate or respond to the pressing concerns. These questions are typically more student-specific or scenario-specific.

Typically, these questions are addressed to MTSS Coaches for guidance and support from their knowledge and experience. Since the coaches are a limited human resource, they face too much cognitive overload to address concerns from multiple MTSS Teams working in different schools and districts. Working towards the goal of integrating AI within MTSS teams, we design our model objective as follows:
‘You are an AI assistant for the multidisciplinary team members implementing MTSS for Behavioral health in schools. Your objective is to assist them with data-driven, decision-making suggestions for the efficient implementation of MTSS(B)’.

4 Methodology- CANDI-QA↩︎

The overall methodology is designed to systematically evaluate the performance of different prompting strategies for large language models (LLMs) on a set of specialized behavioral health questions. Our pipeline, as summarized in Figure 2, begins with the curation of a proprietary dataset by human experts from the School Behavioral Health Academy (SBHA). This dataset comprises 315 questions of two distinct types: T1 and T2. We then apply three different prompting strategies (P1, P2, and P3) to a chosen language model to generate responses for all questions. The evaluation process diverges for the two question types. For T1 questions, we leverage a set of gold-standard answers provided by our expert coaches to assess the quality of the model-generated responses using a suite of reference-based metrics, including Cosine Similarity, BERTScore, MoverScore, and BLEURT. Conversely, for T2 questions, which lack gold-standard answers, we employ a retrieval-augmented generation (RAG) evaluation framework based on the RAGAs library. In this framework, we utilize responses generated by the Qwen model as our retrieved context and evaluate the LLM-generated responses against them using metrics such as context precision, response relevance, faithfulness, and semantic similarity. This dual-pronged approach allows for a comprehensive and robust evaluation of the LLM’s performance across different question types and prompting strategies.

4.1 Dataset Curation↩︎

The dataset for this study was meticulously curated with the assistance of human experts from the School Behavioral Health Academy (SBHA). The dataset consists of a total of 315 questions, categorized into two types: T1 and T2. T1 questions are designed to test foundational knowledge and are characterized by a set of 90 questions. T2 questions, comprising 225 questions, are more complex and require a deeper understanding of contextual nuances. The questions were developed to reflect the specific challenges and inquiries faced by MTSS (Multi-Tiered System of Support) teams, with the goal of providing a robust testbed for the LLMs in a real-world application setting.

4.2 Prompting Strategies↩︎

To assess the impact of different prompting techniques on model performance, we employed three distinct prompting strategies, labeled P1, P2, and P3. Each of these strategies was applied to generate responses from the chosen language model for all 315 questions in the dataset. The specific details and design of each prompting strategy are outlined below:

• P1: No-Context Prompting
  In this setup, the model receives only the user query. No additional contextual or role-based information is provided.

• P2: Role-Aware Prompting
  The system prompt includes contextual information describing the user’s role and a corresponding role description. This setup provides the model with role-specific background to guide its responses.

• P3: Task-Aware Prompting
  The system prompt includes a high-level objective of the AI assistant, along with the user’s role and role description. This strategy offers the model a comprehensive context that situates the question within the broader task it is meant to support.

4.3 Evaluation of T1 Questions↩︎

For the 90 T1 questions, we conducted a reference-based evaluation. Our human experts, who serve as coaches for the MTSS team, provided “gold-standard” answers for each question. The language model responses for each of the three prompting strategies (P1, P2, and P3) were then evaluated against these gold-standard answers using a variety of metrics designed to capture different aspects of semantic and syntactic similarity. The metrics used for this evaluation were:

• Cosine Similarity: Measures the cosine of the angle between two vectors, representing the semantic similarity between the model response and the gold answer.

• BERTScore: A metric that leverages contextual embeddings from BERT to calculate the similarity between two sentences.

• MoverScore: A distance-based metric that measures the “work” required to transform one sentence into another, providing a measure of semantic distance.

• BLEURT: A learned metric that is trained to predict human judgments of text quality, offering a more human-aligned evaluation.

4.4 Evaluation of T2 Questions↩︎

For the 225 T2 questions, which are more open-ended and do not have a single gold-standard answer, we adopted a retrieval-augmented generation (RAG) evaluation framework using the RAGAs library. This framework is designed to assess the quality of generated responses by measuring the alignment between the generated answer, the retrieved context, and the original question. The evaluation was structured as follows:

4.4.0.1 Context Generation

To create a benchmark for evaluation, we used the Qwen model to generate responses for the T2 questions, which served as the “retrieved contexts” for our analysis.

4.4.0.2 Response Evaluation

We evaluated the responses generated by our primary language model (using prompting strategies P1 and P3) against these retrieved contexts. The evaluation was performed using the following RAGAs metrics:

• Context Precision: This metric assesses whether all of the retrieved contexts are relevant to the question. A higher score indicates that the retrieved information is more focused and useful for answering the query.

• Response Relevancy: This metric measures how relevant the generated response is to the original user input. It is calculated by using an LLM to generate a set of questions from the response and then computing the average cosine similarity between these generated questions and the original question. A high score means the response is a direct and complete answer, while a low score indicates it may be incomplete or contain redundant information.

• Faithfulness: This metric evaluates the factual consistency of the generated response with the retrieved context. It works by identifying all the claims made in the response and verifying whether each claim can be directly inferred from the provided context. A score of 1.0 indicates that all claims are supported by the context, and a score of 0.0 means none are.

• Answer Semantic Similarity: This metric evaluates the semantic similarity between the generated response and a reference answer. In our case, since we do not have a gold-standard answer, we used the Qwen-generated response as the reference to compare the semantic meaning of the two outputs. This evaluation uses a cross-encoder model to compute the similarity score, which ranges from 0 to 1, with higher scores indicating better alignment.

The evaluation was performed by treating the language model’s response as response and the Qwen model’s response as retrieved_contexts. The evaluation was thus conducted on pairs of (response, retrieved_contexts), where response=<language model response on P1 or P3> and retrieved_contexts=[<qwen response on P1 or P3>].

5 Experiment Setup↩︎

5.1 Datasets↩︎

We introduce our novel dataset CANDI-QA, curated specifically for question answering tasks in the MTSS domain with the help of our MTSS expert coaches. The dataset consists of two main types of questions: T1-type questions (information assistance questions) and T2-type questions (applied inference type questions). The dataset consists of \(90\) T1-type questions and \(225\) T2-type questions, along with gold answers, provided by our experts, for each question. Each question has a corresponding role.

The MTSS framework consists of the following roles: Admin, MTSS Coach, Counsellor, Clinician, Psychologist, Nurse, and Teacher.

For T1-type questions, the dataset consists of the role-specific questions. These are the types of questions that could be asked by members from various roles to the coaches.

For T2-type questions, the dataset consists of case-based questions, where each case belongs to one of the three MTSS intervention tiers (tier 1, 2 or 3).

5.2 Input Setup↩︎

We experiment on the question-answering task using three different types of prompts. We empirically found that:

• P1-type prompt: No context, only the question is passed to the model.

• P2-type prompt: Role-aware context ({role} + {role description}) and the question are passed to the model.

• P3-type prompt: Task-aware context ({objective} + {role} + {role description}) and the question are passed to the model.

5.3 Metrics↩︎

We use two distinct sets of evaluation metrics aligned with the nature of T1 and T2 questions.

5.3.0.1 T1 Metrics: Reference-Based Evaluation

T1-type questions are direct, factual queries and include gold-standard reference answers. We apply the following reference-based metrics:

• Cosine Similarity: Given two embedding vectors \(\vec{a}\) and \(\vec{b}\) for the model response and the gold answer, respectively, \[\text{CosSim}(\vec{a}, \vec{b}) = \frac{\vec{a} \cdot \vec{b}}{\|\vec{a}\| \|\vec{b}\|}\]

• BERTScore: For reference tokens \(r_i\) and candidate tokens \(c_j\), BERTScore computes: \[\text{BERTScore} = \frac{1}{|C|} \sum_{c \in C} \max_{r \in R} \text{sim}(c, r)\] where \(\text{sim}(c, r)\) is the cosine similarity between contextual embeddings of token \(c\) and token \(r\).

• MoverScore: Measures the minimal cost of transforming one sentence into another based on token embeddings and their distances: \[\text{MoverScore}(X, Y) = \min_{T \in \mathcal{T}} \sum_{i,j} T_{i,j} \cdot d(e_i, e_j)\] where \(T\) is a transport matrix and \(d(e_i, e_j)\) is the distance between embeddings \(e_i\) and \(e_j\).

• BLEURT: Uses a pre-trained regression model \(f_\theta(x, y)\) to compute a score: \[\text{BLEURT}(x, y) = f_\theta(x, y)\] where \(x\) is the model response and \(y\) is the gold answer.

5.3.0.2 T2 Metrics: Reference-Free Evaluation (RAGAs Framework)

T2-type questions are open-ended and evaluated without gold references, using retrieval-augmented context instead. Let \(q\) be the original question, \(r\) the generated response, and \(c\) the retrieved context (from Qwen).

• Context Precision: \[\text{Precision}_{context} = \frac{|c \cap \text{Rel}(q)|}{|c|}\] where \(\text{Rel}(q)\) denotes the set of relevant context units to the query.

• Response Relevancy: \[\text{Rel}(q, r) = \cos(\phi(q), \phi(r))\] where \(\phi(\cdot)\) is a sentence embedding function.

• Faithfulness: \[\text{Faith}(r, c) = \frac{|\text{Claims}(r) \cap \text{Supported}(c)|}{|\text{Claims}(r)|}\] where \(\text{Claims}(r)\) are atomic factual assertions and \(\text{Supported}(c)\) are those verifiable by the context.

• Semantic Similarity: \[\text{Sim}_{sem}(r, c) = f_{\text{cross-enc}}(r, c)\] where \(f_{\text{cross-enc}}\) is a cross-encoder model that computes the similarity score between response and context.

5.4 Implementation Details↩︎

We evaluate a combination of open-source and proprietary language models to benchmark performance across diverse model sizes and architectures. All open-source models are accessed via HuggingFace using their official identifiers, ensuring reproducibility and public accessibility. The full list of models used in our experiments includes:

5.4.0.1 Model Identifiers

We evaluate the following models in our experiments. Open-source models are accessed from HuggingFace, while proprietary models are accessed via the Groq API.

• microsoft/Phi-4-mini-instruct-3.8B

• deepseek-ai/DeepSeek-R1-Distill-Llama-8B

• qwen/qwen3-32b

• google/gemini-pro-1.5

• google/gemma-2-9b-it

• meta-llama/Meta-Llama-3-8B-Instruct

• meta-llama/Meta-Llama-3-1B-Instruct

• meta-llama/Meta-Llama-4-Maverick-7B

• meta-llama/Meta-Llama-4-Scout-17B

• mistralai/Mistral-7B-Instruct-v0.2

5.4.0.2 Prompt Settings

Each model is evaluated using three prompting strategies: P1 (no context), P2 (role-aware context), and P3 (task-aware context). Prompts are constructed by combining system-level instructions (e.g., role descriptions and objectives) with user queries depending on the prompt type.

5.4.0.3 Generation Settings

For all open-source models, the following generation hyperparameters are used to maintain consistency across all model generations:

• max_new_tokens: 512

• temperature: 0.6

• top_p: 0.9

• do_sample: True

• batch_size: 16

5.4.0.4 Data Settings

The curated CANDI-QA dataset contains 315 total questions, divided into 90 T1-type and 225 T2-type questions. Each question is passed through all three prompt variants for every model. For T1 questions, the generated outputs are compared to gold-standard answers using reference-based metrics. For T2 questions, the outputs are evaluated using RAGAs metrics with Qwen model responses serving as the retrieval context.

The full codebase, model identifiers, and evaluation scripts will be released upon publication to support reproducibility.

6 Baseline Solution - MTSS-Net↩︎

To support future research on contextual alignment, we introduce a neuro-symbolic baseline framework, MTSS-Net, designed for the CANDI-QA benchmark. MTSS-Net models how structured, role-aware reasoning can enhance language model (LM) performance in specialized domains. It includes two variants: v1, which leverages structured JSON-based representations for context tracking, and v2, which builds on v1 by adding source attribution and role conditioning to enable more transparent and grounded reasoning.

MTSS-Net is used to address both task types in the CANDI-QA setting:

• T1: Information Assistance Questions – direct, factual queries that benefit from structured lookup.

• T2: Applied Inference Questions – scenario-specific questions requiring multi-hop reasoning or role-sensitive interpretation.

6.1 MTSS-Net v1: Structured Context Representation↩︎

MTSS-Net v1 encodes each QA instance as a structured JSON object capturing four elements:

• Task type (T1 or T2)

• Role or identity of the user (e.g., teacher, counselor)

• Intervention tier (Tier 1, 2, or 3)

• Relevant policy or procedural references

This structure acts as a semantic scaffold, supporting symbolic rule-based reasoning or enabling the construction of more interpretable prompts for small LMs. JSON is chosen for its clarity, interpretability, and ease of use with transformer-based models. This variant is especially effective for T1 tasks, where answers often rely on retrieving clearly defined information based on structured inputs.

6.2 MTSS-Net v2: Role-Aware Reasoning with Source Attribution↩︎

MTSS-Net v2 enhances the structured representation of v1 with three additional components:

• Curated QA pairs linked to expert-verified or synthesized documents,

• Role conditioning to align responses with the expected responsibilities of different users (e.g., administrator vs. paraprofessional),

• Source citation that explicitly grounds responses in the underlying documentation or procedures they rely on.

These enhancements aim to handle the complexity of T2 questions, where understanding the user’s intent, their institutional role, and the scenario-specific context is critical. Role-aware conditioning helps the system tailor responses in ways that match real-world team dynamics. The source attribution feature also promotes explainability, allowing users to trace answers back to supporting evidence.

Together, MTSS-Net v1 and v2 illustrate persistent challenges in aligning LMs with real-world domain expectations, especially around context disambiguation and inference—issues often overlooked by traditional alignment benchmarks. As part of this work, we release MTSS-Net as a reproducible baseline to support future research into interpretable and role-sensitive QA systems.

7 Results↩︎

7.1 Evaluation of T1 Questions: Information Assistance↩︎

This evaluation assesses how effectively language models answer T1 (information-assistance) questions in the behavioral health domain using three prompting strategies: P1 (zero-shot), P2 (role-aware), and P3 (context-rich). Table 1 summarizes performance across Cosine Similarity, BERTScore, MoverScore, and BLEURT. Qwen3 (32B) consistently outperforms others, with the highest semantic alignment (Cos-Sim: 0.4302–0.4344). Smaller models like Llama 3.2 1B instruct perform lowest. Role-aware prompting (P2) yields modest improvements over P1, showing the value of incorporating role context. However, gains from P2 to P3 are marginal, suggesting that task-level context provides limited benefit for straightforward factual queries.

7.2 T2 Evaluation Summary↩︎

Table 2 shows that all models benefit from task-aware prompting (P3) over P1. Mistral 7B Instruct performs best, with highest faithfulness (0.8014) and semantic similarity (0.8336). Phi-4-mini-instruct and LLaMA 3.1 (8B) also show clear gains in context precision and faithfulness. These results suggest that T2 tasks require more than retrieval—they benefit from structured, role- and task-aware prompting to support grounded, scenario-sensitive reasoning.

8 Discussions↩︎

Our experiments highlight several challenges in applying LLMs to real-world, domain-specific tasks such as MTSS QA. Strong performance on open-domain benchmarks does not guarantee alignment with expert preferences in niche domains, suggesting that preference alignment is not domain-generalizable.

Models benefit significantly from contextual disambiguation. Prompting strategies that include role and task-aware context (P2 and P3) yield improved results, especially for T2 questions requiring inferential reasoning. However, prompting alone is insufficient—scenario-specific queries demand structured guidance beyond plain text.

Structured input using formats like JSON offers clear advantages. LLMs show better semantic precision and interpretability when given predictable, serialized inputs, likely due to easier token-level parsing and alignment.

In the MTSS domain, role context and task type are critical. T1 questions depend on factual lookup, while T2 questions demand procedural reasoning and situational awareness. These findings support augmenting QA datasets with structured context and metadata to better reflect real-world usage.

This points toward hybrid approaches that combine symbolic scaffolding with neural reasoning, especially in enterprise and educational settings.

Future work may explore self-reflective or planning-based modules to improve multi-hop reasoning, along with task-type-aware benchmarking to better differentiate between factual and inferential QA tasks.

9 Conclusion↩︎

This work highlights the importance of contextual alignment for domain-specific QA, particularly within the MTSS behavioral health framework. Through systematic evaluations across prompting strategies and model types, we demonstrate that role- and task-aware prompts significantly improve performance over zero-shot baselines.

A key limitation of this study is its reliance on static prompting without adaptive reasoning or feedback mechanisms. Future work will explore integrating self-reflective or planning-based modules, as well as expanding the dataset to cover broader MTSS scenarios.

The MTSS-Net neuro-symbolic baseline solution, along with supplementary materials, will be released to support further research and reproducibility.

References↩︎

---

1. Preprint. Under review. For correspondence: megha.chakraborty@iairo.ai↩︎