1 Introduction and Motivation↩︎

In May of 2025, we saw the rise of AI coding agents for software development and tech company layoffs. The Computing Community Consortium (CCC) and the Computing Research Associations- Industry (CRA-I) white paper "The Future of Programming in the Age of Large Language Models" recommended educators move towards emphasizing skills needed to work collaboratively with LLMs such as decomposing problems and testing code for correctness  [1].

In response, we considered how to update our CS2 course, which already emphasized decomposing problems and unit testing, to also prepare our students to use LLMs and still be able to meet the course learning outcomes without LLM assistance. We considered updates according to DURA (Demystify-Use-Reflect-Assess) as shown in Table 1, our lightweight framework for incorporating LLMs into courses. The four components of DURA: demystify, use, reflect, and assess, deliberately support the learning process, as well as the need for reliable evidence of student learning. The foundational component of DURA is the demystification of Large Language Models (LLMs), which underpin modern AI. When students understand how the technology works, they are more likely to be effective AI users who won’t overly trust the models or attribute thoughts and feelings to them! Students can also benefit from instruction and practice to use LLMs responsibly and productively. Beyond use, students are expected to reflect on their experiences with LLMs, supporting metacognition and self-regulation. Given the rapidly changing landscape and increasing student access to LLMs, instructors must also consider authentic approaches for assessment that align with these realities. The DURA strategies we used are highlighted in Table 1 and some or all of which can be applied to most CS courses.

Prior work suggests the availability of LLMs may negatively affect course climate and reduce student use of course forums and office hours [2], [3]. We had observed such a downward trend in our CS2 course over the past few years and there was slight uptick during the semester when these changes were implemented. We found that students’ perceptions that the instructor cared about them and their learning increased during the semester when we implemented these updates.

The goal of this experience report is to describe how a course can be redesigned to incorporate student use of LLMs while preserving core learning outcomes and promoting effective learning behaviors. We aim to

1. Describe our implementation of the DURA (Demystify-Use-Reflect) framework in a large CS2 course,

2. Examine how students used LLMs, particularly in relation to help-seeking and problem-solving,

3. Examine how these course changes may be associated with student perceptions of the course.

Our intention is to provide practical guidance and reusable materials for instructors adapting their CS courses to incorporate LLMs.

2 Prior and Related Work↩︎

2.1 Demystify↩︎

In recent years, many AI-literacy frameworks for K-12 have been published. For example, Jia et al. developed a research based framework for developing a holistic AI literacy framework for children based on three dimensions: AI awareness, AI mechanics, and AI impacts [4]. Mills et al. previously published a detailed framework for K-12 to understand, evaluate, and use AI. These frameworks both emphasize an understanding of how AI works before using AI. The Digital Education Council provides a five dimension framework for AI literacy in higher education, each at three competency levels: 1) Understanding AI and Data, 2) Critical Thinking and Judgment, 3) Ethical and Responsible Use, 4) Human-Centricity, Emotional Intelligence, and Creativity, and 5) Domain Expertise[5].

Even before the emergence of AI chatbots, there was concern over the public’s limited understanding of AI technologies and opaque algorithms, especially considering their creation of "folk theories" about them [6], [7]. In 2021 Ng’s literature survey indicated that researchers advocated for students to understand underlying AI concepts for their careers and to be responsible ethical citizens  [8]. However, in Computer Science courses today, many are diving into using AI without first teaching students how it works. As stated by Miranda, "Simply knowing how to use AI is not enough."  [9]. Our approach seeks to explicitly incorporate the demystification of LLMs, both how they work and how they can influence learning, so students can establish an informed mental model.

2.2 Use↩︎

CS instructors have taken a range of approaches to respond to LLM use, including banning it, allowing unrestricted use, or adopting moderated policies in between [10], [11]. Some instructors redesign assignments or policies to account for AI capabilities, including restricting use or adapting coursework expectations [12]. Others integrate LLM chatbots with guardrails or structured activities, such as guided prompting, code explanation, or evaluation tasks, to support learning rather than replace it [13]–[15].

Research has shown that students use LLMs for a wide range of programming tasks including understanding, debugging, and generating code [2], [3], [16]. However, in [16], Arora et al. reported 37% of interactions with LLMs involved pasting entire assignment descriptions which may lead to shallow learning. In [3], students themselves reported this risk by mentioning that LLMs, while helping sometimes, can reduce human power and human thinking ability. We aimed to update our CS2 course to provide experiences for students to use LLMs but with enough boundaries so that students learn to do so effectively and ethically while still being able to meet the course learning outcomes without assistance.

2.3 Reflect↩︎

Reflection assignments are a well-known approach for improving student meta-cognition and academic self-regulation  [17]. In 2011 VanDerGrift et al incorporated a reflection assignment to help students become more meta-cognitive about their own software development processes [18] and by 2020 many CS educators had explored its use  [19]. Recently, STEM education researchers are exploring the use of AI as a meta-cognitive partner  [20], [21]. Padiyath has created a GenAI contract tool to help programming students set goals on AI use, and reflect on their alignment to those goals  [22]. In 2024, Margulieux et al. asked students in an introductory programming course to reflect on their process for completing assignments with AI repeatedly [23]. Our approach similarly aims to use reflections to mitigate the risk of students using LLMs mindlessly and encourage them to increase their awareness, critical use, and effectiveness over time. Our goal is to support both productive use of AI and development of student critical thinking. Furthermore, current industry trends indicate that communication skills are becoming even more important.

2.4 Assess↩︎

Many instructors have adjusted assessment practices by switching to in-class, and even handwritten, tests in order to verify student understanding beyond submitted code. For programming assignments, instructors are now often requiring video, written, or in person oral explanations of code [12]. For example, the 2024 AP Computer Science Principles exam was redesigned to require students describe program design, algorithm behavior, errors, and abstractions [24]. Given our emphasis on students meeting learning outcomes unassisted, we adapted our in class proctored assessments to be more aligned with programming assignments and allowed retakes.

Unlike policies that restrict or monitor LLM use, the approach we describe allows the use of LLMs but with assessment design to provide guardrails against over-reliance. While many instructors are exploring a wide variety of uses and integration approaches for LLMs, DURA fills a gap because it also presents complementary approaches. This framework, and especially our emphasis on demystifying LLMs in a programming course, are novel contributions.

3 Course Description↩︎

We updated a CS2 course at our large research-intensive institution in the southeast United States where the Department of Computer Science is part of the College of Engineering. The course was taught in Java and covered an introduction to data structures with an object-oriented approach over a 15-week semester and emphasized test-driven development. The course covered the use and implementation of the introductory data structures bags, stacks, queues, lists, and binary trees. It is a prerequisite for required core CS courses Computer Organization and Data Structures and Algorithms as well as several CS elective courses. In the fall of 2025 there were three instructors across four lecture sections teaching 442 students, of whom 53% were CS majors or first-year General Engineering majors and the others were spread across various majors with 23% from Data Science. It was a flipped course that used an interactive e-book which included instructional videos. During lecture, instructors gave demos, highlighted concepts with Peer Instruction as shown to be effective by Porter et al. [25], and provided time for learning activities. The team of 18 TAs led lab sections where students worked on weekly lab programming assignments or other coursework. Instructors held drop-in office hours and TAs offered centralized in-person office hours weekdays from 12 - 8pm and there was a course Piazza  [26] forum monitored by instructors and TAs. TAs attended a departmental workshop at the beginning of the semester, and the instructional team met weekly for one hour and communicated regularly using a chat system.

4 Course Update Approaches↩︎

In an effort to prepare students for programming with access to LLMs, we adjusted the course policy to allow students to seek external assistance such as from LLMs. With an eye towards the future, we expected these changes to be the first steps as we adapted to the changing AI landscape. They were manageable initial steps before we were ready to invest efforts to overhaul labs and projects. Instructors can easily adapt our materials and approaches to incorporate the demystify, use, and reflect components into any CS course. However, adjusting assessments to include programming assignment skills is course specific and can be a more time-intensive undertaking, especially when creating question banks to support retakes. We worked on this over the summer with UTA support, as generating question variations was not reliable without oversight. Reusable course materials are available at https://tinyurl.com/3z8kvfnm.

4.1 Demystify LLMs↩︎

In the second week of the course, we added a 30-minute introduction to how LLMs work and highlighted the CS education research on how LLM use can affect student learning. The presentation of LLMs explains how LLMs are actually a machine learning model, called a Transformer, which does next token prediction based on the training data. We emphasized that LLMs do not think or problem solve and while it’s beneficial that students learn to use them, they should be careful not to short cut their own learning and problem solving skill development.

This lesson was adapted from another introductory course where students spend more time learning details about how LLMs work [27] and includes information about CS education research which references articles covering the impact of LLMs on student ability to write, trace, and read code as well as the distinctions between higher performing students and those who may not have a solid base [23], [28]–[30]. An example of the risks for a CS2 student using an LLM for programming generated by ChatGPT, including shallow understanding and inhibited debugging skills, was also included. This LLM demystification lesson can be used in a variety of courses.

4.2 Responsible LLM use↩︎

Students were also provided with some LLM suggested use guidance both during the lesson and on a course page. During the lesson, student responsibilities to work through course assignments, document use of LLMs, and meet learning objectives unassisted were highlighted. We suggested strategies for LLM use such as trying to solve problems on your own first using tracing, diagrams, and the debugger. We encouraged students to seek guidance from course staff and that when they do use an LLM to be clear and specific, go step by step, and verify responses.

A course page had reminders to caution students about short-cutting learning as well as example prompts to encourage students to use LLMs as a learning aide. We also discussed and provided sparse and detailed prompt examples of how a students could instruct an LLM to be an AI study assistant and prioritize learning over giving answers.

Throughout the course, students are expected to provide an attribution statement when using LLMs. The expectation is that each code file should include one of the following:

• I have not used any assistance for the assignment.

• During the preparation of this assignment, [name of the student] used [name of the tool] in [name the parts of the assignment where it was used] to [specify the purpose]. After using this tool, I/we reviewed and edited the content as needed to ensure its accuracy and take full responsibility for the content in relation to grading. I understand that I am responsible for being able to complete this work without the use of assistance.

Note that the inclusion of the last sentence is important for this course design, the goal was for students to still be able to meet learning objectives without assistance.

4.3 Student Reflection Assignments↩︎

We added problem solving reflections, where students were asked to select a recent programming assignment and describe their process in their own words, in weeks 4, 8, and 13. In week 4 students were also asked to guide their former self on how to approach the assignment. In week 8, students were also asked to reflect on whether they followed their previous advice. In week 13, after the final individual project was due, students were also asked give advice to future students.

The TAs graded these reflections which stimulated valuable discussions about student perspectives, help-seeking, and LLM use, during weekly instructional team meetings. After the Week 4 Problem Solving Reflections, some TAs were surprised to read that students were intimidated by them. Although we have always explained this to TAs, reading it in students’ own words made an impression on the TAs. To better understand student perceptions of help-seeking strategies, our team analyzed the Week 13 Problem Solving Reflections which covered the most challenging individual programming projects. As shown in Table 2, the assignment included two sets of questions designed to capture students’ reflections. We created a summary of the advice for use in lecture that following semester.

4.4 Assessment Updates↩︎

We reallocated 20% of the course grade from programming assignments (45%->25%) to proctored assessments (35% -> 55%) to emphasize independent comprehension. To offset this shift to high-stakes testing and encourage student learning, we allowed an in-class proctored retake of each test during the semester  [31]. Recent versions of the course had 4 in-class proctored quizzes, so the course was rearranged to use the four class days for 2 tests each with a retake day one week later.

The questions related to programming assignments often included method javadocs or test methods closely aligned with assignments. Because the tests were multiple choice they did often require careful inspection of code by students. The intention was that students who engaged with the code when working on the assignments would be well prepared for the tests.

Tests were created from banks of questions. Each week students had sample test questions to optionally practice. After the first attempt at a test, the students could see their test score, but needed to come to office hours to look through their tests with a TA or Instructor. It was a good learning opportunity for students and also seemed to help them build relationships with the course staff. This was a strong motivator to get students to attend office hours and possibly realize it was useful to seek in-person assistance.

4.5 Instructor Impressions↩︎

Two of the three instructors during Fall 2025 are co-authors of this report. We enjoyed the changes to the course, it was refreshing to be able to openly discuss LLM use with students and there was no longer a sense that we were policing them. It also opened the door to more conversations about the hows and whys of the course design, thus creating a more transparent teaching environment  [32]. Because of this foundation, we were also well situated to discuss AI advancements and current events with the students. Some TAs expressed concern that the students were using LLMs and over-reliant on them. After the first test some students realized they needed to take their assignments more seriously and we were able to have open dialogue with them about this, especially when they came to review their tests.

In the future, we would like to have assessments earlier and more often. In general, we were encouraged that some students seemed to prefer talking with us and attending office hours, even though LLMs were allowed. As AI coding agents become more advanced, we would like to provide students access and demonstrations of coding agents in an environment that encourages student effective use of LLMs. This involves re-imagining CS2-level programming assignments and use of lab time to bolster student engagement with peers and TAs.

5 Evidence↩︎

We observed a change in student behaviors and perceptions as explained in more detail below, but we cannot attribute causality due to lack of a control group. We have IRB approval for the use of student data in this course.

5.1 Course Help-Seeking History↩︎

As in other CS courses  [2], [33], our team had observed the reduced number of students attending office hours and posting on the Piazza course forum. The decrease in student engagement in recent years in this CS2 course is shown in Table 3. Although use of LLMs was not sanctioned until 2025 Fall, notice in Figure 2 that there was steep decline in Piazza posting in 2024 Fall and office hours usage in 2025 Spring and then a rise in Fall 2025 when LLMs and retakes were allowed. To better understand student perceptions of help-seeking, our team analyzed the student responses for the Week 13 Problem Solving Reflections assignment.

5.2 Student Reflections↩︎

Of the 442 students enrolled in the course, 198 students provided informed consent and submitted valid responses for the Week 13 Problem Solving Reflection assignment, with prompts shown in Table 2.

A team of five co-authors, two of whom were instructors for the course, analyzed the anonymized responses using an iterative qualitative coding approach that combined inductive and deductive coding consistent with established practices [34], [35]. The final codebook consisted of 32 codes related to students’ help-seeking behaviors as shown in Table 4, and 35 codes related to students’ recommendations for future help-seeking as shown in Table 5.

Students reported using formal course resources such as office hours, Piazza, TAs, and course materials. Some students did not seek or use any assistance and most stated that no LLM was used for the assignment described.

S40: I did not use an LLM as it did not help fix the coverage and just took me down a rabbit hole that never even helped.

S210: I did not use assistance from LLMs, but rather went to office hours since it was much more helpful to talk to someone face to face and get a well-descriptive explanation.

Many students did seek assistance from LLMs and students described a variety of approaches for understanding, code writing assistance, and code writing that are similar to those found in prior work  [2], [3], [16], [33]. Students commonly used multiple resources and approaches. How, when, and why students did use LLMs varied, their usage can be categorized into the themes: used to understand, used to assist code writing, and used to write code.

S12: I used an LLM as a support tool, but not for writing code. My strategy was to ask it to rephrase confusing parts of the spec, help me walk through example queue states, and think through potential edge cases.

S226: I also used an LLM as a debugging partner and have it explain concepts to me. An example was asking questions like "why wasn’t this comparator working" or "has this test fully covered the code?"

Only a few reflections described using LLMs to write the code.

S232: I used to be completely against the use of AI in classwork, but in this project I saw the value of consulting an LLM for outlines and explaining methods I didn’t understand how to implement...I did change the way I would’ve implemented some methods based on Claude suggestions, which helped me think more efficiently in certain aspects, but in others I stuck with my own implementation and did not agree with the AI’s thoughts.

S10: I did use LLMs to help me figure out some more complex parts of the constructors/method as well. I would insert the prompt that I was given on canvas, along with my personal thought process. It would result in the LLM giving me a code result with an explanation that helped me understand why it was writing that code and how it worked.

Students mostly recommended future students seek help via formal help modalities and the course staff but also supplemental sources such as friends and online resources. Similar to Penney’s findings [36], students predominantly recommended that future students use office hours and they commented on how helpful the TAs were. Piazza was also recommended and students suggested using the course materials for help.

S194: I highly recommend going to office hours and talking to TAs whenever something feels confusing or even in general because they’re only there to help. Even when you think you understand everything, going to office hours can is so helpful because it teaches you so much.

S123: If a student is stuck, I would recommend them to ask their TAs or go to office hours before trying LLMs, because they really want to help you, and they might explain the problem better than LLMs.

How, when, and why students recommended getting assistance varied. There was a noticeable trend to recommend starting assignments early in order to be able to get help and not be overwhelmed. Students also explicitly mentioned that students should try to solve problems on their own first or look through course materials. Students described the main reason why a future student should seek assistance is to understand the material.

S121: Having the knowledge of how a solution works should be more important than just getting the answer correct or getting the assignment done.

S192: To understand how code works is the most important, and if you push off the work by using LLMs (or other outside assistance) you will be unprepared for the tests.

Students recommend using LLMs to assist with code writing such as for test cases or documentation. But some students cautioned the use of LLMS.

S36: Use Piazza for quick clarifications, office hours for design questions, and LLMs to rephrase specs or suggest test cases—then make sure you still understand and test every line you submit.

S43: My biggest advice to new students is to avoid using AI to do large parts of the work as it makes it harder to get a real feel for doing the code yourself

Student help-seeking was not simply AI and non-AI behaviors. There is a whole gamut of possibilities of how, when, and why students do and do not use LLMs. Increased instruction on incorporating LLMs into workflows may benefit some students.

5.3 Student Perceptions of Caring↩︎

We also compared student perceptions of the motivational climate for this CS2 course to the previous fall semester using the MUSIC Inventory as shown in Table 6. This questionnaire is based on The MUSIC Model of Motivation and measures student perceptions of course eMpowerment/autonomy, Usefulness, Success, Interest, and Caring using a Likert Scale from 1(Strongly Disagree) - 6 (Strongly Agree) [37]. Student MUSIC perceptions were quite consistent but students reported significantly higher perception of instructor Caring (p < .001) during the semester we made these DURA course updates. The 4 questions for instructor Caring on the MUSIC Inventory are:

• The instructor is willing to assist me if I need help in the course.

• The instructor cares about how well I do in this course.

• The instructor is respectful of me.

• The instructor is friendly.

Although multiple factors may have an impact, allowing assessment retakes along with the shift away from reporting students for honor code violations to guiding and encouraging responsible use of assistance may have been perceived as more caring.

Note. \(n\) = number of respondents; \(N\) = total enrollment; % = response rate. * p < .001

6 Conclusion↩︎

In this experience report we detailed course updates to promote effective and ethical use of LLMs at scale. Our materials to demystify, use, and reflect on LLMs are available and can be used in CS courses at various levels and adapted to reflect instructor policies and goals. Our use of in-person proctored tests that addressed programming assignment skills provided a guardrail against over-reliance of LLMs and requires course-specific implementation. Likewise supporting test review and retakes requires support by the instructional team. Allowing both LLM use and test retakes helped create a more supportive and open course culture.

References↩︎