1 Introduction↩︎

Problem posing is an instructional approach where learners actively generate and refine their own questions, rather than working only on pre-defined problems. Compared to problem solving, it emphasizes the construction of problems as a learning process, which can support deeper conceptual understanding and higher-order thinking [1]. In computer science education, posing a computational problem often requires reasoning about algorithmic structure, edge cases, or combinatorial relationships rather than a single symbolic answer. A key challenge is that students often lack feedback on how to improve their posed problems. Instructors also face practical constraints in providing timely, individualized feedback, especially in large classrooms. Large language models (LLMs) offer a potential solution by enabling instant feedback at scale. Prior work shows that LLMs can effectively support problem solving by explaining concepts and guiding step-by-step reasoning [2], [3], including in computing contexts where LLMs have served as tutors, code assistants, and exercise generators [4], [5]. However, problem posing introduces a fundamentally different requirement: success is not defined by reaching a correct answer, but by constructing a meaningful question. This shift in what counts as success, from correctness to construction, creates a design challenge for LLM-based scaffolding. While LLMs can provide direct, solution-like improvements, such scaffolding may reduce opportunities for students to critically evaluate and refine their own thinking. At the same time, overly open-ended guidance may leave students uncertain about how to improve their work. Prior research on LLM-assisted learning highlights this challenge between cognitive support and cognitive offloading, where strong assistance can unintentionally reduce learner engagement [6], [7]. In problem-posing contexts, this risk is particularly visible, as students may accept generated outputs without sufficient evaluation [8].

Most instructors experimenting with generative AI lack concrete design guidance on what type of AI scaffolding supports productive student thinking during open-ended tasks. In this paper, we investigate how different forms of LLM scaffolding shape students’ engagement in a problem-posing task. We design two scaffolding conditions using the same underlying rubric but different interaction strategies. The Direct condition provides worked example-based revisions, offering concrete and actionable improvements. The Indirect condition uses guiding questions to prompt reflection and self-revision, keeping the responsibility for improvement with the learner. We deploy both conditions in a within-subject pilot study with computing students across two problem scenarios. We analyze changes in problem quality, user experience, and qualitative reflections to understand how each scaffolding type supports the problem-posing process. Our goal is to understand how different scaffolding designs shape the experience of problem posing, not just whether they improve performance. We suggest that a useful design question for LLM-supported learning is how help is distributed between the system and the learner, rather than only whether help is given. Our contributions are:

1. We present the design of two LLM-powered scaffolding systems for computational problem posing, including the prompts and Bloom’s Taxonomy rubric used to generate scaffolding.

2. We share formative evidence on how problem quality changed under Direct and Indirect scaffolding across two computing scenarios, and reflect on why scenario design shaped scaffolding effectiveness.

3. We summarize participants’ perceptions of the two scaffolding approaches, including which felt more efficient and which encouraged deeper reflection, and derive practical guidance for sequencing Direct and Indirect scaffolding across a problem-posing activity.

2 Related Work↩︎

2.1 LLMs and Problem Posing in CS↩︎

Problem posing is a promising approach in computer science education, applied in introductory programming as well as advanced courses such as Data Structures and AI [9]. It is particularly effective for assessing novice learners rather than advanced learners, though more time-consuming and less familiar to students than traditional instruction [9]. With LLMs, researchers have begun exploring this task from multiple directions. Students can act as instructors, posing problems and guiding an LLM step-by-step toward a solution; since the LLM cannot solve the problem independently, students must organize and articulate their own understanding, improving performance [4]. The reverse direction has also been explored, where the LLM poses questions to help students clarify goals and reflect during problem solving, reducing cognitive load and improving computational thinking in collaborative programming [5]. LLMs have also been used to generate entirely new, personalized programming exercises, which students find engaging, though difficulty calibration and depth of personalization remain inconsistent [10]. Existing work has explored LLMs as problem solvers, tutors, and exercise generators, but how LLMs should scaffold the iterative refinement of self-generated problems remains largely unexplored.

2.2 Direct and Indirect Scaffolding↩︎

Scaffolding aims to support learners in transforming feedback into meaningful learning actions, especially in problem-solving and problem-posing tasks that require both conceptual understanding and metacognitive effort [11]. Direct scaffolding provides explicit guidance by identifying errors and suggesting next steps. It reduces cognitive load and is especially beneficial for lower-performing students by making scaffolding immediately actionable [7], [11]. However, excessive reliance on direct guidance may limit opportunities for independent reasoning over time. In contrast, indirect scaffolding uses prompts, questions, and open-ended structures to encourage learners to generate their own solutions. This approach can promote autonomy, reflection, and deeper engagement with task structure [12]. However, it may also increase cognitive and emotional demands when learners lack sufficient prior knowledge, sometimes leading to uncertainty or reduced perceived competence [13]. Overall, prior research suggests that the effectiveness of scaffolding depends on balancing support and autonomy rather than choosing one approach over the other. A common recommendation is to begin with more structured support and gradually shift toward open-ended guidance as learners develop confidence and independence [11].

2.3 LLMs as Pedagogical Scaffolding Tools↩︎

LLMs have shown promise as scaffolding tools in education, but their effectiveness depends heavily on how they are designed and deployed. When scaffolding is dynamic and personalized, students engage more deeply and experience less cognitive strain [14]. However, easy access to LLM support does not automatically translate into deeper thinking: over time, students who rely heavily on LLM assistants tend to ask simpler, lower-order questions — offloading cognition rather than building it [6], [7]. This tension between support and dependency is particularly relevant in open-ended tasks like problem posing, where the goal is not just to produce an output but to think carefully about it. One way this tension has been addressed is by distinguishing indirect and direct LLM scaffolding. Indirect scaffolding often uses Socratic prompts or guiding questions rather than direct answers, and has been shown to improve critical thinking and support deeper engagement [15], while direct scaffolding provides worked examples and explicit guidance for revision [13]. Grounding LLM support in pedagogical frameworks, such as Bloom’s Taxonomy, leads to richer learning experiences than treating the LLM as a general-purpose tool [2], [3]. For example, Bloom-based scaffolding cards provide structured prompts that reduce cognitive load and improve reasoning quality [16]. Still, the effectiveness of either modality depends on learners’ ability to respond productively to the scaffold provided. In problem-posing contexts specifically, this challenge is amplified: students using ChatGPT have been able to generate personalized problems quickly, but often accepted outputs that were mathematically inaccurate or inauthentic without noticing [8], and users often fail to identify and correct such errors even when scaffolding is present [17]. Yet little work has directly compared how indirect and direct LLM scaffolding shape the iterative refinement of self-generated problems, or examined these strategies in authentic, deployed settings that yield practical design lessons. This study addresses this gap by comparing structured direct and indirect LLM scaffolding in a problem-posing task, focusing on their effects on cognitive engagement and problem quality.

3 Method↩︎

3.1 Rationale: Indirect vs. Direct Scaffolding↩︎

We designed two LLM-powered scaffolding systems, Indirect and Direct, to examine how different forms of AI support influence computational problem posing. Both systems aimed to help students refine their problems using the same pedagogical framework, but differed in how guidance was delivered. Indirect Scaffolding responded with two guiding questions that encouraged students to reflect on their own reasoning rather than prescribing specific revisions. This approach draws on indirect tutoring methods that promote independent thinking and self-explanation. Direct Scaffolding responded with two higher-level example problems adapted from the student’s submission. This approach reflects worked-example-based instruction, which can reduce cognitive load and support efficient skill acquisition.

3.2 System Architecture and Prompt Design↩︎

The problem-posing environment was deployed as a high-fidelity web interface, where scaffolding responses were generated in real time through prompt-engineered interactions with GPT-4o via the OpenAI API. When a participant submitted a problem, the system sent it to GPT-4o together with a condition-specific prompt. The model first evaluated the submission using our Bloom’s Taxonomy–based rubric, identified the weakest quality dimension, and then generated the corresponding scaffolding response. The feedback was displayed immediately within the same interface. Both systems displayed four target qualities throughout the activity to help students understand what a good problem looks like: accuracy, clarity, cognitive challenge, and originality.

3.2.0.1 Prompt Design and Bloom’s Taxonomy Integration.

The prompt followed a two-step structure. First, the LLM evaluated the student’s problem across three dimensions: Conceptual Accuracy, Structural Clarity, and Cognitive Demand. Cognitive Demand was further mapped to four levels of Bloom’s Taxonomy: Remember/Understand, Apply, Analyze, and Evaluate/Create. Second, the LLM generated scaffolding tailored to the weakest identified dimension and the assigned condition. The Indirect condition produced exactly two reflective questions designed to encourage self-explanation without providing direct hints. The Direct condition produced exactly two revised problem examples positioned approximately one Bloom level higher than the original submission. To reduce cognitive overload while preserving actionable guidance, all responses were constrained to two items. The system was also instructed to avoid solving the problem, performing calculations, or introducing new mathematical content not present in the student’s original submission.

3.3 Computing Scenarios↩︎

Participants completed two open-ended problem-posing scenarios, each representing a common type of computational reasoning: Handshake: “There are nine friends at a party. Each friend shakes hands with everyone else when they arrive. Write three mathematical problems using this information.” This scenario involves graph/combination based reasoning and has a narrow problem space centered on a single numerical relationship. Making Change: “Sarah works at a coffee shop. A customer orders two small coffees at $1.59 each and pays with a $5 bill. Sarah only has half-dollars, dimes, and pennies. Write three mathematical problems using this information.” This scenario reflects greedy/coin-change reasoning and offers a richer set of numerical relationships to explore. A third scenario, “Food Truck,” was used only during the training phase to familiarize participants with the system before the main study began. All the scenarios were adapted from prior problem-posing research conducted by [18].

3.4 Pilot Session Protocol↩︎

3.4.0.1 Consent and Introduction Session.

Twenty Computer Science graduate students participated in this pilot study voluntarily, and no course credit or compensation was tied to performance outcomes. Seventeen of the twenty participants were also serving as Teaching Assistants (TAs). All participants provided informed consent, filled out demographic information, and were introduced to the interface.

3.4.0.2 Practice session

The participants practiced using the system with the “Food Truck” scenario to ensure that everyone was familiarized with the interface and understood how to submit a problem and request the LLM-generated scaffolding.

3.4.0.3 Pilot Study Session

After reporting comfort with the interface, participants completed both the Handshake and Making Change scenarios in a within-subject, counterbalanced design using a Latin square to control order effects. Each participant experienced both Indirect and Direct scaffolding, one per scenario, in different orders. In each scenario, participants completed three problem-posing tasks (Q1–Q3) under one scaffolding condition. For each task, they wrote an initial problem, requested an LLM scaffold, and revised it once. The scaffold was either guiding questions (Indirect) or worked examples (Direct). There was no time limit. After each scenario, participants completed the UEQ-S survey. After both scenarios, they took part in a semi-structured interview.

3.5 Formative Evaluation↩︎

We collected three types of formative evidence to understand how problem quality changed and how each system felt to use.

3.5.0.1 Manipulation Check.

After each condition, participants rated the system’s output style on two single-item checks using a 7-point Likert scale (1 = Strongly Disagree, 7 = Strongly Agree): whether the scaffolding prompted them with questions about their own problem to help them think through how to revise it, rather than telling them what to do (Indirect scaffolding item), and whether the AI showed them directly how to improve their problem (Direct scaffolding item). This confirmed whether participants actually perceived the Indirect system as prompting reflection through questions and the Direct system as giving examples, a necessary check before interpreting any outcome differences.

3.5.0.2 Problem Quality.

We collected pairs of initial and revised problems from each participant. Each problem was rated across four dimensions using a rubric informed by Bloom’s Taxonomy, shown in  1. A second rater independently scored a subset of problems, and agreement with the original ratings was high (Cohen’s \(\kappa > 0.7\)), supporting the reliability of the scoring. Improvement was calculated as the difference between each participant’s refined score and their original score on the same problem. Within each scenario, participants completed three consecutive problem-posing tasks (Q1–Q3), each followed by LLM scaffolding and a revision before moving on to the next task. Because each scenario was assigned to a single scaffolding condition under the counterbalanced design, participants completed all three tasks within a scenario under the same condition (e.g., three tasks under Indirect scaffolding in one scenario, followed by three tasks under Direct scaffolding in the other), resulting in six problem-posing tasks in total.

3.5.0.3 User Experience.

We used the Short User Experience Questionnaire (UEQ-S) to measure hedonic quality (how engaging and stimulating the system felt) and pragmatic quality (how easy and efficient it was to use). We also collected self-reported ratings of ownership, agency, and satisfaction with the refined problems. We also used the NASA Task Load Index (NASA-TLX) to capture perceived mental demand, frustration, and effort during each condition. At the end of both sessions, we conducted semi-structured post-session interviews to understand how participants experienced each system.

4 Result↩︎

4.1 Manipulation Check↩︎

Before analyzing the main outcomes, we verified that participants perceived the two LLM-based scaffolding systems as stylistically distinct. Participants rated the Indirect system higher on prompting them with questions about their own problem rather than telling them what to do (\(M=5.05\) vs.\(M=3.80\); \(W=18.0\), \(p=.016\)), and rated the Direct system higher on showing them exactly how to improve their problem (\(M=5.30\) vs.\(M=4.35\); \(W=22.5\), \(p=.031\)). These results confirm that participants distinguished the two scaffolding systems as intended.

4.2 Problem Quality Improvement↩︎

4.2.1 Overall Improvement↩︎

Both systems improved problem quality. Across all participants, refined problems scored significantly higher than original problems (Wilcoxon signed-rank test: \(Z = -3.83\), \(p < .001\), \(r = .88\)), confirming that LLM-based scaffolding supported problem posing regardless of modality ( 1). A Wilcoxon signed-rank test showed that problem quality improved significantly more under Direct scaffolding (\(Mdn = 1.83\), \(IQR = 2.00\)) than Indirect scaffolding (\(Mdn = 0.67\), \(IQR = 1.08\)), \(W = 25.5\), \(Z = -2.80\), \(p = .005\), \(r = .63\) ( 2). A mixed-effects regression confirmed this effect after accounting for participant-level variance (\(\beta = -1.23\), \(z = -3.75\), \(p < .001\)). Participants who produced higher-quality initial problems showed smaller improvements, indicating a ceiling effect (\(\beta = -0.34\), \(p = .003\)).

4.2.2 Improvement Across Scenarios↩︎

Mann-Whitney U tests showed different patterns across the two scenarios. In Making Change, Direct scaffolding produced significantly larger improvements (\(Mdn = 2.50, IQR = 1.08\)) than Indirect (\(Mdn = 0.67, IQR = 1.17\); \(U = 19\), \(p = .020\), \(r = .52\)). In Handshake, the improvement was not significantly different between Direct and Indirect scaffolding.

4.2.3 Quality Dimensions↩︎

The two scenarios showed different patterns across quality dimensions. In Handshake, none of the four dimensions showed a statistically significant difference between conditions, though Direct scaffolding showed a marginal, non-significant advantage in structural clarity (\(p = .064\), \(r = .41\)). In Making Change, Direct scaffolding produced significantly larger improvements in novelty (\(p < .001\), \(r = .71\)), cognitive demand (\(p = .005\), \(r = .61\)), and structural clarity (\(p = .027\), \(r = .49\)). Conceptual accuracy was not significantly different (\(p = .167\)).

4.2.4 Scaffolding Effectiveness Across Tasks↩︎

We analyzed how improvement changed across three sequential tasks (\(Q1 \rightarrow Q3\)) to examine temporal trends in scaffolding effectiveness. Under the Indirect condition, improvement showed a significant negative association with task order (Spearman’s \(r = -.285\), \(p = .027\)), dropping sharply from \(M = 1.15\) (\(SD = 1.23\)) at Q1 to \(M = 0.30\) (\(SD = 0.57\)) at Q2, before partially recovering at Q3 (\(M = 0.70\), \(SD = 1.75\)). In contrast, the Direct condition showed a non-significant trend over time (Spearman’s \(r = -.174\), \(p = .183\)), with relatively stable improvement across Q1 (\(M = 2.25\), \(SD = 2.15\)), Q2 (\(M = 2.15\), \(SD = 2.68\)), and Q3 (\(M = 1.45\), \(SD = 1.96\)).

4.3 User Experience↩︎

Participants reported positive experiences with both systems. Direct Scaffolding received significantly higher pragmatic quality ratings on the UEQ-S (\(M=5.51\), \(SD=0.76\) vs.\(M=4.76\), \(SD=0.94\); \(W=28.0\), \(p=.004\)) and overall UEQ scores (\(M=5.25\), \(SD=0.71\) vs.\(M=4.69\), \(SD=0.90\); \(W=26.5\), \(p =.006\)) than Indirect scaffolding. Hedonic quality, ownership, agency, satisfaction, and perceived originality did not differ significantly between conditions (all \(p >.08\)). NASA-TLX scores also showed no significant difference in overall cognitive load (\(p=.39\)), though participants in the Indirect condition reported numerically higher mental demand (\(59.10 vs.\;53.35\)).

5 Discussion↩︎

This pilot study examines how two pedagogically distinct LLM-based scaffolding strategies, Direct and Indirect, can each support computational problem posing in computing education. While prior work has explored LLMs as instructors, tutors, and exercise generators in computing education, little was known about how different scaffolding modalities shape the problems students themselves construct in computational contexts, a gap this study directly addresses. Although Direct Scaffolding produced stronger immediate results, our findings suggest that the more important design question is not which modality to adopt, but how the two can be deployed together, distributing cognitive responsibility between learner and system according to instructional goals.

5.0.0.1 Direct and Indirect Scaffolding support different learning processes.

Although Direct scaffolding produced larger immediate improvements and higher pragmatic quality ratings, the qualitative findings suggest that the two approaches serve different instructional purposes. Participants described Direct scaffolding as easier to apply because it provided concrete suggestions that simplified the revision process. In contrast, they reported that Indirect scaffolding encouraged deeper reflection, with one participant noting, “Direct was easier to act on, but Indirect made me think more” (MC04). This pattern is consistent with prior work suggesting that explicit guidance improves efficiency, whereas reflective prompts promote active knowledge construction [11], [15]. Importantly, this deeper engagement did not result in significantly higher perceived workload (NASA-TLX), although participants in the Indirect condition did report numerically higher mental demand, suggesting the cognitive cost of reflection was modest but not absent. Together, these findings suggest that the two scaffolding strategies are complementary rather than competing. Direct scaffolding appears more suitable for efficient revision and rapid improvement, while Indirect scaffolding better supports reflection and independent thinking. This distinction was also reflected in participants’ qualitative feedback. One participant (MC05) noted that Direct scaffolding was more helpful for their own problem-solving, but considered Indirect one more appropriate for supporting students. Another participant (MC16), despite preferring Indirect scaffolding for themselves, acknowledged that this preference may be shaped by their TA experience and that Direct scaffolding could be more beneficial from a learner’s perspective. Overall, participants tended to view the two modalities as serving different instructional purposes rather than one being universally superior. We also observed a ceiling effect: participants who submitted stronger initial problems showed lower improvement from scaffolding, regardless of modality, since there was less room for improvement when a problem was already well-formed. This suggests that structured feedback may be most useful for helping students refine weaker or underdeveloped problems, rather than pushing already high-quality problems further, a pattern broadly consistent with problem-posing research showing that such assessments tend to work better for novice learners than for advanced ones [9].

5.0.0.2 LLM scaffolding can support revision without reducing learner ownership.

A common concern about LLM-supported learning is that students may become passive editors of LLM-generated content instead of active authors of their own work [8]. In our study, students first created their own problem before receiving LLM scaffolding, and they understood that revising it was still their responsibility. This sequence—student-created problem, LLM scaffolding, and student-led revision—appears to have helped preserve their sense of ownership. As one participant explained, ” I felt like it was my problem that I have created, but AI helped me.” (MC05). Even in the Direct condition, participants described the worked examples as helping them move forward when they were stuck, rather than replacing their own ideas. This is consistent with our quantitative results, which showed no significant differences in ownership, agency, or perceived originality between conditions. These findings suggest that when students generate an initial problem before receiving LLM scaffolding, both Direct and Indirect scaffolding can support revision while preserving students’ sense of authorship.

5.0.0.3 Task structure and temporal dynamics shape LLM scaffolding effectiveness.

Our findings suggest that the effectiveness of LLM-based scaffolding depends not only on prompt design, but also on task structure and when support is provided. In the more open-ended Making Change scenario, which involves algorithmic reasoning (greedy-style coin change), Direct scaffolding led to significant improvements in novelty, cognitive demand, and structural clarity. In contrast, no significant differences were observed in the more constrained Handshake scenario, which involves graphs or combination, though Direct scaffolding showed a marginal advantage in structural clarity. Participants reported that Handshake offered limited room for meaningful revision, making improvement more difficult, and noted that worked examples helped them progress more effectively than reflective questions when they felt stuck. A similar pattern emerged over time: improvement under Indirect scaffolding decreased across tasks (Q1\(\to\)Q3), while Direct scaffolding remained relatively stable. This suggests that Indirect scaffolding may be more useful early in a task to encourage reflection, whereas Direct scaffolding better supports sustained performance during repeated problem-posing activities. Together, these findings highlight the need for adaptive scaffolding that adjusts to both task openness and learner progress (see 2). For example, LLM support could begin with reflective questioning and gradually shift toward more explicit guidance as learners struggle or tasks become more complex. Unlike much prior work on LLM-assisted problem posing in mathematics education [8], [17], our tasks required computational reasoning commonly found in introductory computer science, such as combinatorial reasoning and algorithmic decision-making. This suggests that scaffolding effects may generalize to CS contexts where learners construct solutions rather than simply apply procedures, such as designing test cases or specifying algorithmic requirements. Future work should examine whether behavioral or cognitive signals can be used to dynamically personalize scaffolding in such settings.

5.0.0.4 What did not work as expected.

Some participants interpreted Indirect guiding questions as prompting solution-finding rather than revision of problem quality. This suggests future prompt designs should explicitly direct attention to structure, clarity, and cognitive demand rather than solution processes [16]. The Handshake scenario’s narrow problem space likely compounded this issue by limiting opportunities for meaningful revision. Overall, effective LLM scaffolding depends on the interaction between prompt and task design; improvements in one cannot fully compensate for limitations in the other.

5.0.0.5 Limitations.

This pilot has several limitations to consider when interpreting the findings. The sample included 20 graduate students, 17 of whom were Teaching Assistants, so their prior teaching experience may limit generalizability to undergraduates. The study also did not include a no-scaffolding condition, meaning improvements reflect differences between scaffolding types rather than absolute gains. Finally, data were collected from only two scenarios in a single session, so long-term learning and retention were not examined.

6 Conclusion↩︎

This paper presented and evaluated an LLM-powered problem-posing scaffolding environment that compared Indirect (guiding questions) and Direct (worked-example revisions) scaffolding in computing education. While both approaches improved problem quality, Direct scaffolding produced larger immediate improvements. In contrast, Indirect feedback promoted deeper cognitive engagement and metacognitive reflection, though it showed a non-significant numerical decline in improvement across sequential tasks. Together, these findings highlight that the two scaffolding modalities serve complementary instructional roles, suggesting that balancing direct support with reflective guidance can better support LLM-based learning in computing education.

References↩︎