1 Introduction↩︎

Generative artificial intelligence has rapidly become part of higher education. Students use LLMs to understand concepts, debug code, interpret assignments, and prepare for examinations. This development is promising because one-to-one tutoring is known to be highly effective, and intelligent tutoring systems have shown that timely feedback can improve student problem solving [1]–[3]. Cognitive tutors and knowledge tracing further show that educational support should guide the learner’s next step rather than merely provide the final answer [4], [5]. For programming education, this distinction is crucial: the same response can be useful feedback in one setting and academic-integrity leakage in another.

Despite their usefulness, unrestricted LLM tutors introduce risks. A fluent explanation may be unsupported or incorrect, a debugging suggestion may misidentify the cause of an error, and a model may reveal complete assignment solutions when only hints are allowed. Reviews of ChatGPT and GenAI in education identify personalization, accessibility, plagiarism, hallucination, and assessment disruption as central issues [6], [7]. In novice programming, these risks are amplified because students often lack the expertise to judge whether a generated code explanation is valid. Traditional intelligent tutoring systems constrain feedback through domain models and pedagogical rules, whereas many LLM tutors rely heavily on prompts and user trust.

Retrieval-augmented generation (RAG) is a natural first step toward safer educational tutoring because it grounds responses in course material rather than only in model parameters [8]–[10]. However, RAG alone does not solve educational safety. Retrieved course context may be relevant while the generated response still leaks a complete solution. A rubric prompt may describe allowed assistance but may not enforce it. A same-model self-check may fail because the generator and checker share similar failure modes. Therefore, safe GenAI tutoring requires an explicit response-control pipeline: the system must decide what type of help is pedagogically appropriate, ground claims in instructor-approved evidence, verify unsupported claims, and prevent the tutor from becoming a solution engine.

This paper proposes EduGuard, a safe RAG-based LLM tutoring framework for introductory programming education. EduGuard analyzes a student query, retrieves instructor-approved evidence, selects a pedagogical strategy, generates a response, verifies generated claims with a separate NLI model, and applies an overreliance-control policy. The system supports both English and Bangla-English code-mixed queries because many students in South Asian higher education learn technical material in English while expressing confusion using mixed language.

The proposed system introduces the following contributions:

• We present EduGuard, a safe tutoring architecture that combines hybrid retrieval, pedagogical strategy selection, rubric-aware generation, claim-level verification, and overreliance control.

• We introduce BILearn-CS, a 600-query instructor-authored and TA-validated programming-support benchmark with explicit provenance, safe-response labels, bilingual/code-mixed prompts, and direct-answer adversarial cases.

• We add two validation layers beyond the constructed benchmark: a public CS50-style course-forum set and a controlled 10-student undergraduate pilot study measuring immediate learning gain, confidence calibration, and overreliance behavior.

• We compare EduGuard with stronger named baselines, report confidence intervals and statistical tests across tables, and provide verifier threshold, noisy-retrieval, ablation, and qualitative failure analyses.

The rest of the paper is organized as follows. Section 2 reviews related works. Section 3 describes the methodology. Section 4 presents experimental results. Section 5 discusses strengths, limitations, ethics, and deployment implications. Section 6 concludes the paper.

2 Related Works↩︎

Artificial intelligence in education has evolved from intelligent tutoring systems and adaptive learning environments to automated feedback and learning analytics [11]. Classical feedback theory emphasizes that effective feedback should clarify goals, identify current performance, and guide the next action [12], [13]. Formative feedback should also support self-regulation, not only provide correctness signals [14]. These principles are directly relevant to LLM tutoring, where an answer that is factually correct may still be pedagogically harmful if it bypasses student reasoning.

Modern LLMs are built on transformer architectures and instruction-tuned pretraining [15]–[17]. In computing education, researchers have studied how code generation tools affect assignments, assessment, and feedback [18]–[21]. These studies suggest that LLMs can reduce friction in learning but also challenge academic integrity and require redesigned scaffolding. Related work on hallucination and factuality shows that neural generation can produce unsupported claims even when the output is fluent [22]–[24]. Recent work specifically investigates hallucinated feedback in learning contexts and shows that wrong feedback can affect student behavior and confidence [25].

RAG and dense retrieval have become common approaches to grounding LLM output [8], [9], [26], [27]. Educational RAG systems use course materials, rubrics, and examples to constrain generated feedback [28]–[30]. However, many RAG systems optimize answer relevance rather than pedagogical safety. AI literacy research also shows that learners need support in judging AI output rather than simply receiving it [31], [32]. Table 1 summarizes representative related works and the gap addressed by EduGuard.

3 Methodology↩︎

The methodology describes the benchmark, external validation sets, preprocessing, architecture, controller, baselines, metrics, student pilot protocol, and statistical analysis. The overall goal is to evaluate EduGuard as a safe tutoring-response framework rather than as an autonomous instructor.

3.1 Dataset Description↩︎

BILearn-CS. BILearn-CS contains 600 programming-support queries. It is not presented as a naturally collected student-log dataset. Instead, it is an instructor-authored and TA-validated benchmark designed to test correctness, grounding, pedagogy, and leakage. Two instructors and three graduate teaching assistants created the queries from common difficulties in introductory programming and data-structures courses. The benchmark contains 120 concept questions, 120 debugging cases, 120 misconception prompts, 120 assignment-support requests, 60 adversarial direct-answer prompts, and 60 Bangla-English code-mixed prompts. Each item contains the query, topic label, expected response mode, supporting evidence, misconception type when applicable, allowed assistance level, and leakage policy.

CS50-Forum validation set. To avoid relying only on a constructed benchmark, we built a 150-query external validation set from public course-oriented programming questions. The source was CS50-related public Stack Exchange and Stack Overflow questions where users explicitly referred to CS50 problem sets, course topics, or course-specific debugging issues [33]. Unlike general Stack Overflow posts, these questions are closer to a learning context because they involve course assignments, hints, and novice errors. We still treat this as course-forum validation rather than classroom deployment.

Undergraduate pilot. In addition, we conducted a small controlled pilot with 10 undergraduate students who had completed or were enrolled in an introductory programming course. The study used a counterbalanced within-subject design. Each participant solved two matched programming tasks: one with GPT-4o-mini Tutor and one with EduGuard. Tasks were matched by topic and difficulty and included debugging, loops, lists/arrays, and function reasoning. Each session included a pre-test, tutor interaction, immediate post-test, confidence rating, and short usefulness survey. No grades were affected, and participants were instructed not to submit generated content as coursework.

3.2 Data Preprocessing↩︎

Course materials were converted to text, chunked into 180–250 word segments, and tagged with source metadata. Code snippets and error messages were preserved as separate chunks because exact tokens are important for programming retrieval. Rubric entries were labeled as explanation allowed, hint only, example allowed, or full solution prohibited. Public course-forum questions were de-identified and filtered to remove personal information, answer text, and off-topic administrative content. Student pilot logs were anonymized before analysis.

Hybrid retrieval combines dense and sparse search. Dense embeddings are computed with sentence-transformers/all-mpnet-base-v2 and indexed with FAISS. BM25 retrieval is used for exact code tokens, exception names, and assignment identifiers. Dense and sparse candidates are combined with reciprocal-rank fusion, deduplicated, and truncated to five evidence chunks. This design improves both semantic retrieval and code-token recall.

3.3 System Architecture: EduGuard↩︎

Figure 1 shows the architecture of EduGuard. The system is organized into input understanding, grounded generation, and safety-learning support. It accepts a student query and returns a response that is grounded, pedagogically appropriate, and checked for unsupported claims or direct-answer leakage.

3.3.1 Query Understanding Module.↩︎

The query understanding module classifies the request into concept question, debugging help, assignment support, misconception clarification, exam preparation, or direct-answer seeking. It also detects high-risk phrases such as “write the full code,” “give me the final answer,” and “do my assignment.” The output controls both retrieval scope and response policy.

3.3.2 Pedagogical Strategy Selector.↩︎

The strategy selector maps the query to one of five actions: concept explanation, Socratic hint, debugging localization, misconception correction, or refusal-with-guidance. The selector is implemented as a policy function \[\pi(q,E,R,V)=a, \quad a \in \{explain, hint, debug, correct, refuse\},\] where \(q\) is the query, \(E\) is retrieved evidence, \(R\) is the rubric policy, \(V\) is verification status, and \(a\) is the tutoring action. This formalizes novelty as a response-control policy rather than merely combining RAG and prompting.

3.3.3 Feedback Generation Module.↩︎

The primary generator is Meta-Llama-3.1-8B-Instruct. Generation uses temperature 0.2, top-\(p=0.9\), maximum 512 new tokens, and repetition penalty 1.05. The prompt includes the query, retrieved evidence, rubric policy, selected strategy, and output requirements. For commercial-model comparison, GPT-4o-mini Tutor used model gpt-4o-mini-2024-07-18 [34] with the same maximum response length and a strong Socratic academic-integrity prompt.

3.3.4 Hallucination Verification Module.↩︎

To reduce circularity, the verifier is architecturally separate from the generator. Responses are split into atomic claims. For each claim, the system retrieves the top three evidence chunks and applies DeBERTa-v3-large-MNLI to estimate entailment. A claim is supported if at least one evidence chunk entails it with probability \(p_e \geq 0.65\). The unsupported-claim ratio is \[U(y)=\frac{N_{unsupported}(y)}{N_{claims}(y)}.\] A response is regenerated if \(U(y)>\tau\), where the default threshold is \(\tau=0.20\). If the second draft fails verification, the system returns refusal-with-guidance. We describe the verifier as architecturally separate, not as fully independent in all failure modes, because both the generator and NLI verifier may share web-corpus biases.

3.4 Inference Algorithm↩︎

Algorithm 2 summarizes the inference pipeline.

3.5 Baselines↩︎

We compare EduGuard with named and strengthened baselines. GPT-4o-mini Tutor uses OpenAI gpt-4o-mini-2024-07-18 with a strong Socratic and academic-integrity system prompt. Llama Socratic Tutor uses Meta-Llama-3.1-8B-Instruct with the same strong prompt. Basic RAG uses hybrid retrieval and Llama generation without rubric or verifier. LPITutor-style RAG implements a personalized tutoring-style RAG prompt inspired by LPITutor [30]. RAG + Rubric adds assignment-policy and rubric constraints to the retrieved context. RAG + Self-Check asks the same Llama generator to critique and revise its own answer. EduGuard uses the full pipeline.

3.6 Evaluation Metrics and Statistical Analysis↩︎

Correctness is the percentage of responses judged conceptually and programmatically valid. Grounding is the fraction of response claims supported by retrieved evidence: \[G(y)=\frac{N_{supported\;claims}(y)}{N_{total\;claims}(y)}.\] Hallucination is the fraction of unsupported claims presented as factual feedback: \[H(y)=\frac{N_{unsupported\;claims}(y)}{N_{total\;claims}(y)}.\] Direct-answer leakage is measured in two ways. Policy-based leakage uses instructor assistance labels. Rule-assisted leakage flags complete runnable code, final answers without reasoning, or step-by-step assignment completion when the policy requires hints: \[L=\frac{N_{leakage\;violations}}{N_{restricted\;prompts}}.\] Student learning impact is measured by immediate post-test accuracy and normalized learning gain: \[NLG=\frac{Posttest-Pretest}{100-Pretest}.\] We report 95% bootstrap confidence intervals using 10,000 resamples. Pairwise comparisons use paired bootstrap tests for rates, Wilcoxon signed-rank tests for student and expert ordinal scores, and Cliff’s delta for effect size. Three graduate TAs rated anonymized and shuffled outputs after a calibration round. Krippendorff’s alpha was 0.81 for correctness, 0.79 for rubric alignment, 0.76 for pedagogical usefulness, and 0.74 for hint-level appropriateness.

4 Results↩︎

This section presents results on the constructed benchmark, course-forum validation set, student pilot, ablation study, and verifier robustness analysis.

4.1 Setup for Our Experiment↩︎

Each method generated one response per query. For BILearn-CS, all 600 queries were evaluated. For CS50-Forum, 150 public course-oriented questions were evaluated. For the undergraduate pilot, each of the 10 students completed one task with GPT-4o-mini Tutor and one matched task with EduGuard in counterbalanced order. No benchmark items were used to train the models.

4.2 Implemented Results on BILearn-CS↩︎

Table 2 reports the main benchmark results. EduGuard obtains the strongest overall performance and significantly outperforms RAG + Self-Check on grounding, hallucination, rubric alignment, and leakage (\(p<0.01\)). Correctness improvement is smaller but still significant (\(p=0.03\)). The result suggests that same-model self-checking helps, but claim-level verification and explicit tutoring policy provide stronger safety control.

4.3 External Course-Forum Validation↩︎

Table 3 reports results on CS50-Forum. We do not describe this set as a classroom deployment; it is a public course-forum generalization test. The ranking remains similar, but all systems lose some grounding because public posts contain incomplete code context and references to course-specific files. EduGuard remains strongest, with 86.7% correctness and 6.8% hallucination. Confidence intervals are wider because the set is smaller.

4.4 Undergraduate Student Pilot↩︎

Table 4 reports the 10-student pilot. Because the pilot is small, the results are interpreted as preliminary educational evidence rather than a definitive classroom trial. EduGuard produced higher immediate post-test accuracy, larger normalized learning gain, lower confidence calibration error, and lower overreliance behavior than GPT-4o-mini Tutor. Students using EduGuard asked more follow-up clarification questions, while GPT-4o-mini Tutor more often produced complete answer patterns that students copied without modification.

4.5 Ablation and Verifier Robustness↩︎

Table 5 shows ablations. Removing the verifier nearly doubles hallucination. Removing the strategy selector substantially increases leakage. Replacing the NLI verifier with same-model self-check reduces performance, showing that architectural separation matters.

The verification threshold \(\tau\) controls the trade-off between hallucination reduction and over-refusal. Table 6 reports sensitivity results. A strict threshold of 0.10 minimizes hallucination but rejects too many acceptable responses. A relaxed threshold of 0.30 improves coverage but allows more unsupported claims. The default \(\tau=0.20\) provides the best balance.

5 Discussion↩︎

The results show that EduGuard’s improvement is not caused by retrieval alone or by a stronger prompt alone. GPT-4o-mini Tutor is fluent and useful, but it lacks course-grounding and assignment-specific policy. Basic RAG improves grounding but leaks answers because retrieved evidence does not determine how much help should be given. RAG + Self-Check improves over Basic RAG but remains limited because the generator checks its own output. EduGuard performs better because it separates evidence retrieval, pedagogical policy, generation, and verification.

5.1 Strengths of the Proposed System↩︎

The strongest result is the simultaneous reduction of hallucination and leakage. In education, these two goals often conflict: a system can avoid hallucination by giving very little information, or it can be helpful by giving too much. EduGuard balances this through a response-control policy. The qualitative examples in Table 7 show that EduGuard turns high-risk requests into smaller next steps while still supporting learning.

5.2 Limitations and Areas for Improvement↩︎

The 10-student pilot is useful but underpowered. It supports feasibility and short-term learning indicators, not long-term retention or course-grade improvement. Future work should include a larger randomized classroom deployment, delayed post-tests, demographic analysis, and instructor-controlled sections. The CS50-Forum set is more realistic than fully synthetic items but is still not equivalent to private classroom interaction logs. The NLI verifier is architecturally separate from the generator, but it is not immune to correlated failure modes because both models are trained on broad web data. Code-specific reasoning can still fail when a claim requires execution rather than textual entailment.

The leakage metric also has a tautology risk because policy-based leakage depends on instructor assistance labels. We reduce this risk by additionally using rule-assisted leakage checks based on observable output features such as complete runnable code, final-answer disclosure, and step-by-step assignment completion. Still, leakage policy is partly contextual; what is allowed in one course may be disallowed in another.

5.3 Failure Analysis↩︎

We examined 80 randomly sampled errors from BILearn-CS and 30 errors from CS50-Forum. Table 8 summarizes the main failure categories. The most frequent failure was incomplete retrieval, where the available course chunks did not contain enough detail to support a specific error message or assignment context. The second failure involved code-semantics limitations: the NLI verifier could detect unsupported natural-language claims but could not always determine whether a code trace or edge case was logically correct without execution. Bilingual retrieval errors occurred when a student expressed a programming concept in transliterated Bangla while the course material used English terminology. Finally, some refusals were overly conservative when a student asked for detailed debugging help that was actually allowed by the rubric.

This analysis clarifies the remaining gap between safe response control and full educational intelligence. EduGuard is effective at reducing unsupported and over-direct answers, but it does not fully understand a student’s evolving knowledge state. A future version should integrate lightweight code execution, misconception tracking across turns, and instructor review of ambiguous policy cases. In particular, code execution would complement NLI verification by checking whether generated debugging claims match actual program behavior.

5.4 Interpretation of Educational Impact↩︎

The undergraduate pilot provides initial evidence that safe tutoring can influence short-term learning behavior. The improvement in normalized learning gain is consistent with the design goal of giving smaller next steps rather than complete solutions. The reduction in overreliance is also important because a student can receive a correct AI answer while learning very little if the answer is copied without reflection. In the pilot logs, EduGuard more often asked students to trace a loop, predict an output, or explain why a condition failed. These prompts encouraged active processing, which is consistent with testing-effect and cognitive-load research [35], [36]. However, the pilot should be interpreted cautiously. With only 10 students, it cannot establish long-term retention, demographic generality, or course-grade improvement. Its purpose is to demonstrate feasibility and to provide early evidence that the response-control design changes student behavior in the intended direction.

5.5 Reproducibility Details↩︎

To support reproducibility under double-blind review, we avoid claiming a public release that is not yet available. Instead, the paper reports the main implementation details required to reproduce the study: generator model, decoding settings, embedding model, retrieval method, top-\(k\) values, verifier model, entailment threshold, leakage definitions, prompt categories, and evaluation rubrics. After review, non-sensitive benchmark items, prompt templates, and scripts can be released if institutional and copyright constraints allow it. For the student pilot, raw interaction logs should not be released because they may contain student-written code, personal writing style, or course-specific details. A de-identified derived version containing task type, system condition, outcome scores, and annotation labels is the more appropriate reproducibility artifact.

5.6 Ethics, Reproducibility, and Deployment↩︎

For the pilot, participants gave consent, no grades were affected, and logs were anonymized. Because the study involved a small educational pilot, the system should not be treated as ready for autonomous deployment. Student-facing use should include instructor oversight, opt-out options, de-identification, retention limits, and clear warnings that the system provides formative support rather than grading decisions. To avoid overstating reproducibility during double-blind review, we provide model names, decoding parameters, retrieval settings, prompt categories, metric definitions, and qualitative examples in the paper; a public release of non-sensitive benchmark items and scripts can occur after review if institutional and copyright constraints allow it.

6 Conclusions↩︎

This paper presented EduGuard, a safe RAG-based LLM tutoring framework for programming education. The system combines hybrid retrieval, pedagogical strategy selection, rubric-aware response generation, claim-level verification, and overreliance control. The evaluation includes a 600-query instructor-authored benchmark, a public course-forum validation set, a controlled 10-student undergraduate pilot, ablations, threshold sensitivity, and qualitative examples. EduGuard improves correctness, grounding, rubric alignment, and expert-rated usefulness while reducing hallucination and direct-answer leakage compared with GPT-4o-mini Tutor, Llama Socratic Tutor, Basic RAG, LPITutor-style RAG, RAG + Rubric, and RAG + Self-Check. The findings support a cautious conclusion: safe GenAI tutoring requires explicit pedagogical policy and evidence verification, not only stronger LLMs or retrieval.

References↩︎