DigitalCoach: Communication and Grounding Gaps in
Human and Agentic Computer Use Coaching

Meng Chen\(^{1}\), Anya Ji\(^{1}\), Tsung-Han Wu\(^{1}\), Tobias Maringgele\(^{2}\)1,
David M. Chan\(^{1}\), Alane Suhr\(^{1}\)2, Amy Pavel\(^{1}\)
\(^{1}\)University of California, Berkeley \(^{2}\)Technical University of Munich

January 01, 1970

Abstract

Agents are increasingly capable of automating software tasks, but can they teach humans how to use software themselves? We introduce DigitalCoach, a multimodal dataset of 72 human expert-novice computer use coaching sessions consisting of 22,752 dialogue turns grounded in 28.1 hours of screen and input event recordings across 5 software applications. We then use DigitalCoach to evaluate whether state-of-the-art models can teach humans how to use computers. Our automated evaluation shows that models differ from humans in how they coach: models provide more direct instructions, but fewer explanations, error diagnoses and knowledge-check questions. When we fix the coaching method, models produce utterances that are similar to human references, but poorly grounded in visual context. Our interactive evaluation confirms that model coaches cause learners to passively follow instructions without deeper engagement and fall short in visual grounding. DigitalCoach lays a foundation for collaborative and proactive computer use coaching agents. Data and code are available at https://project-digital-coach.vercel.app.

Figure 1: An example from DigitalCoach. Human coach provides gives guidance grounded in user’s screen and with explanations while model coach gives instructions without explaining how or why.

1 Introduction↩︎

Professional software tools for creativity (e.g., Blender), engineering (e.g., OnShape), and analysis (e.g., Excel) are powerful but hard to learn. Agents now let novices use natural language instructions to easily execute creativity, engineering, and analysis tasks end-to-end without using the graphical user interfaces (GUIs) of the software [1]–[3]. However, GUIs remain important to professionals as they afford expressivity, precision, and speed that cannot be achieved with natural language alone. For example, skilled Blender users can use the hotkey “G” then drag an object to position it, rather than using natural language to describe what object to move and where to move it. For novices who want to develop such software expertise, the automation afforded by agents bypasses learning. Thus, we explore a new role for agents: teaching novices the skills they need to use these interfaces effectively.

Novices traditionally learn from expert demonstrations in webpage or video tutorials, but these materials are one-size-fits-all and require learners to manually align their own progress with the tutorial. To counter this, Human-Computer Interaction (HCI) research has prototyped systems [4], [5] that transform tutorials into task assistants. But can current models generate adaptive, grounded guidance that supports skill acquisition beyond task completion?

We introduce DigitalCoach, a dataset for (1) characterizing multimodal language-based interactions during human expert–novice computer use coaching and (2) analyzing how well models emulate human expert coaching. DigitalCoach contains 72 computer use coaching sessions of 40 participants spanning creativity, engineering, and productivity domains. DigitalCoach has 22,752 turns dialogues grounded in 28.1 hours screen recordings, 39,609 input events, and 36,724 file snapshots. To capture both the communicative and pedagogical functions of language use, we annotate utterances with dialogue acts and coaching methods.

We then evaluate 6 state-of-the-art models as computer use coaching agents using DigitalCoach. Results show that model utterances differ distributionally from human utterances and are less lexically and semantically diverse. Compared to human coaches, models predominantly use direct instruction (\(>\)​45% of turns of all models vs. 30% for humans). They provide less feedback on the screen state and ask fewer questions. The best model (Gemini-3.1-Pro) can generate plausible local guidance (CLAIR = 41.4) similar to human references when knowing the coaching method. Yet, current models all rely more on textual history than on the learner’s screen state as models perform similarly without visual context, but their scores drop substantially without dialogue context.

In real-time interaction, learners coached by models completed fewer milestones and retained fewer skills. We show that models still struggle to provide guidance grounded in evolving screen states and rarely offer reusable knowledge that helps learners accomplish similar tasks independently in the future. These findings suggest that effective computer use coaching agents require adaptive combinations of instruction and feedback based on the learner’s evolving progress and opportunities for transferable skill development.

In summary, we contribute:

  • DigitalCoach, a 28.1 hour human expert-novice GUI-grounded dialogue dataset with fine-grained annotations of dialogue acts and coaching methods.

  • Quantitative and qualitative characterizations distinguishing human and model coaching.

  • Expert rating and interactive human evaluations revealing communication and grounding gaps in real-time model coaching.

2 Related Work↩︎

Table 1: Compared to related datasets [6]–[12], captures real human–human computer use coaching with much richer multimodal data.
Dataset Domain Source Setup Duration # Turns Data
4-5 (lr)8-12 # Ppl. Relationship Text Audio Video Action File
Portal Dialogue Corpus Game Real 2 Collaborator 11.5h 24.5k
MathDial Education Semi-synthetic 1 Expert-Novice – 28.3k
VideoCAD Professional Synthetic 0 – 2.1kh –
AssistGUI Professional Real 1 – \(<\)8.3h –
RealWebAssist Everyday Real 1 – \(<\)6h –
GUIDE Professional Real 1 – 67.5h –
MixAssist Professional Real 2 Expert-Novice 7h 431
Professional Real 2 Expert-Novice h 22.7k

4.2pt

Language Use in Situated Collaboration. Prior work has studied how agents ground and coordinate through language under partial observability and differing task knowledge [13]–[19]. More recent work moves toward richer settings, such as video games, household tasks, and collaborative drawing [6], [20]–[24]. However, no existing collaborative dialogue dataset is grounded in GUI workflows where people seek and provide help via multimodal communication (e.g., spatial referencing). Our dataset fills this gap by capturing expert-novice multimodal communication during GUI tasks.

Pedagogical and Instructional Dialogue. Unlike transactional dialogue systems that focus on task completion and intent tracking, such as making reservations [25] or tool use [26], pedagogical dialogue emphasizes on how systems support learning through feedback, questioning, and scaffolding [27]. Prior datasets such as MathDial [7], MathTutorBench [28], and ConvoLearn [29] evaluate tutoring quality in domains including mathematics and geography, while MixAssist captures expert–amateur collaboration in music production workflows [12]. However, these datasets are either text-based or grounded in static content, and none capture instruction grounded in an evolving visual environment. DigitalCoach fills this gap with multimodal expert-novice interactions grounded in real-time GUI state and user actions.

Collaborative GUI Agents. GUI agent research primarily studies whether agents can interpret interface state and execute actions to complete user goals. Benchmarks such as WebArena and OSWorld evaluate agents on web and desktop tasks using screenshots, natural language goals, and interface actions [30], [31]. More recent work extends to screen recordings that capture software interactions over time [8], [32], [33]. However, existing GUI agent benchmarks primarily focus on autonomous task completion rather than collaboration with users during analytic and creative workflows where user agency matters [34]. Recent work explores more proactive assistants that pause at decision points or reason about user intent [11], [35], [36], but the language and dynamics of collaborative interaction remain underexplored. DigitalCoach addresses this gap by characterizing human and model coaching.

3 DigitalCoach Dataset↩︎

We study the collaborative task of computer use coaching, where an expert and learner collaborate to teach the learner key skills in complex software applications, so that the learner can leave the interaction with expanded creative agency in the software. In designing our study, we follow two principal design considerations: we aim to capture and characterize (a) multimodal dialogue, with language use grounded in states (file snapshots), observation (screen recordings), and action (input events), and (b) learning outcomes, using pre/post tasks measuring if learners retain/transfer learned skills.

DigitalCoach contains 22,752 utterances across 72 human expert-learner coaching sessions (Table 1). Beyond dialogue, DigitalCoach also captures rich multimodal activity traces including 28.1 hours of screen recordings, 39,609 input events, and 36,724 file snapshots to support research on language grounding in action and perception. Appendix 9 contains additional details.

Figure 2: Collection setup, illustrated in the CAD software Onshape. A novice learner operates the software while an expert coach provides real-time verbal instructions and screen annotations.

3.1 Data Collection and Construction↩︎

Tasks. We selected 5 software applications spanning 3 domains: productivity (Excel), creativity (FL Studio, Blender, Figma), and engineering (Onshape). We sourced 18 tasks (e.g., Figure 2 shows making a saddle bracket in OnShape) from official and popular online tutorials, covering a range of lengths, difficulties, and modalities (text, audio, image, and 3D model, see Appendix 11).

Participants. We recruited a total of 20 English-speaking software-specific experts and 20 English-speaking novices through professional networks and Upwork,3 comprising 20 total pairs (4 pairs \(\times\) 5 applications). Novices had on average under one year (\(\mu\) = 0.25, \(\sigma\) = 0.44) of experience with and a strong interest in the target software; experts had at least 5 years of software experience (\(\mu=9.25\), \(\sigma=5.39\)) and 6 months of coaching experience. 4

Collection Setup. Each session paired one expert with one novice for approximately 3 hours, covering 3-4 tasks in a single software application. Sessions were conducted over Zoom, with learners sharing their screens while experts coached via verbal instruction, screen annotation, or remote control (Figure 2). Before and after each tutorial task, learners also accomplished matched pre- and post-tasks targeting the same software skills as the tutorial but with variations (e.g., making a donut in Blender in tutorial v.s. making a caramel apple in pre/post task). Learners completed these independently (up to 15 min per task, Appendix 10 describes the full study protocol).

4 Dialogue Acts and Coaching Methods↩︎

Dialogue Acts. To characterize communicative functions of language use in computer use coaching, we developed a dialogue act schema based on Dialog Act Markup in Several Layers [37], [38]. We adapted DAMSL by merging similar categories (e.g., Command and Suggestion) and removing less relevant acts (e.g., Exclamation). The final schema consists of information-seeking acts (Info Request), information-providing acts (Answer, Inform, Opinion), action-oriented acts (Action Directive), and grounding acts (Backchannel) that are common in a task-oriented multimodal coaching dialogue.

Coaching Methods. We further annotate each coach utterance to capture what types of guidance are delivered, based on prior work on cognitive apprenticeship theory [27], [39], and video tutorial instruction [40]. Our schema separates procedural guidance (Direct Instruction, Plan), knowledge (Explanation, Tip), feedback (Confirmation, Diagnosis), and learner-centered elicitation (Clarification, Reflection, Articulation, Exploration).

4.1 Annotation and Classification↩︎

Three trained annotators independently labeled dialogue acts for a subset of utterances (200 total utterances; 40 randomly-sampled continuous turns per task), and coaching methods for 133 utterances from coach from the same sampled subset. The three annotators achieve substantial agreement [41]. We use an LLM classifier (GPT-5.4) to annotate dialogue acts and coaching methods across the entire dataset [42]. We evaluate classifier performance against human annotations, achieving an F1 of 0.85 for dialogue acts and 0.83 for coaching methods. See Appendix 13 for further details.

4.2 Dataset Analysis↩︎

computer use coaching dialogue acts are task-oriented and are asymmetrically distributed across participants (Coach: 65.8% vs. Learner: 34.2%). Coaches mostly give Action Directives (37%) and Inform (29%), while learners primarily provide Backchannels (50%) or ask Info Requests (13%). Human coaches rely heavily on Direct Instruction (31%), but also use confirmation (17%), explanation (20%), planning (7%), diagnosis (7%), and learner-centered elicitation (7%). Coaching method bigram sequences further show that Direct Instruction often appear after Confirmation (\(N = 1026\), 5.71%), Explanation (\(N = 1020\), 5.67%), Plan (\(N = 537\), 2.99%), and Diagnosis (\(N = 502\), 2.79%). Results highlight that experts often first orient the learner to a goal, concept, or problem before giving an actionable next step.5

5 Do Models Coach Like Humans?↩︎

Figure 3: Mean token length across dialogue progress of sampled 18 sessions for human coaches and six models under Vanilla (V) and Coach (C) prompt conditions.

We first analyze the distinctions between coaching behaviors in model and human coaches.

Task Definition. Formally, we cast the coach’s task as given the preceding context \(C_i\) consisting of the dialogue history and sampled screen frames within a time window \(\Delta\), and a task description \(T\) specifying the learning objectives, intermediate goals, and expected outcome, the model must generate a coaching utterance \(\hat{y}_i\) for each human coach turn \(y_i\) at time \(t_i\). We hypothesize that an effective coaching agent should generate \(\hat{y}_i\) that closely matches the human reference \(y_i\).

5.1 Setup↩︎

Models. We selected 4 closed and 2 open models with multimodal input: GPT-5.4 [43], Gemini-3.1-Pro and Gemini-3-Flash [44], Claude-Sonnet-4.6 [45], Qwen-3-VL-8B-Instruct [46], and Llama-4-Scout-17B [47]. All models use default settings, generating one sentence per input given conversation and visual context at 1 fps (up to 30s).

Data. We randomly sampled one complete expert–learner session per task (18 tasks \(\times\) 1 session) and run model generation over each full trajectory, yielding 18 sessions as our evaluation set.

Conditions. We compare two prompting conditions: (i) a vanilla prompt containing the task description and dialogue history, and (ii) a coach prompt that additionally provides our coaching method taxonomy and asks it to choose the most appropriate ones (see Appendix 15).

5.2 Results↩︎

Figure 3 shows that models generate longer utterances and do not shorten over the course of the dialogue as human coaches do [48]. We therefore studied whether this added verbosity reflects diverse and richer coaching behavior or not.

Figure 4: Dialogue act distributions of sampled 18 sessions for human coaches and six models under Vanilla (V) and Coach (C) prompt conditions.
Figure 5: Coaching method distributions of sampled utterances for human coaches and six models under Vanilla (V) and Coach (C) prompt conditions.

Models predominantly give direct instruction. Figure 4 shows that, compared to humans, all models predominately uses Action Directives (>53% vs. 26%) and rarely request information from learners except Gemini-3.1-Pro (Coach) (<6% vs. 8%). In terms of coaching methods, model predominantly use direct instruction (>45% vs. 30%) but rarely provides conceptual guidance such as Explanation(<12% vs. 20%) and Tip (<1% vs. 6%) and learner-centered methods (Figure 6). Llama-4-Scout and Qwen-3-VL-8b-Instruct are instruction-heavy, with over 80% of their dialogue acts as Action Directives and nearly all of their coaching methods as Direct Instruction and Plan. Gemini and Claude are more human, using relatively more Inform acts and more Explanation and Confirmation methods. GPT-5.4 is the least diverse model, using only Direct Instruction, Confirmation, and Explanation with no learner-centric elicitation.

Figure 6: Lexical diversity (Self-BLEU, \downarrow) and semantic diversity (Vendi Score, \uparrow) of coaching utterances for human coaches and six models under vanilla (blue) and coach (gold) prompt conditions.

Model utterances are less diverse than human coaching. To measure whether models repeat the same responses or adapt across turns, we report lexical diversity  [49] and semantic diversity [50]. Figure 6 shows that human coaches produce the most lexically and semantically diverse outputs, while all models fall substantially short on both metrics. This gap is consistent with the more diverse dialogue acts of human coaches, whereas models tend to generate more scripted procedural guidance, a pattern consistent with prior findings that language models often underrepresent the diversity of human communication [51], [52].

Prompting with coaching methods does not shift model behavior. Model method distributions in coach prompt remain largely unchanged relative to the vanilla condition across both dialogue acts (Figure 4) and coaching methods (Figure 5). The gap between model and human coaching is not one of awareness but of capability: models default to instructional behavior even when explicitly guided otherwise. Closing this gap likely requires training interventions rather than prompting alone.

Table 2: MAUVE scores (\([0,1]\), \(\uparrow\); mean \(\pm\) 95% CI) between model and human distributions.
Model MAUVE (95% CI)
2-3 Vanilla Coach
GPT-5.4 0.017\(_{\pm \text{0.008}}\) 0.018\(_{\pm \text{0.009}}\)
Gemini-3-Flash 0.042\(_{\pm \text{0.016}}\) 0.039\(_{\pm \text{0.019}}\)
Gemini-3.1-Pro 0.075\(_{\pm \text{0.037}}\) 0.118\(_{\pm \text{0.049}}\)
Claude-Sonnet-4.6 0.034\(_{\pm \text{0.018}}\) 0.036\(_{\pm \text{0.016}}\)
Qwen-3-VL-Instruct 0.038\(_{\pm \text{0.018}}\) 0.031\(_{\pm \text{0.018}}\)
Llama-4-Scout 0.053\(_{\pm \text{0.025}}\) 0.045\(_{\pm \text{0.027}}\)

4pt

Table 3: CLAIR scores (\([1,100]\), \(\uparrow\); mean \(\pm\) 95% CI) for next coach utterance generation. The default setting uses text-visual input, 30s context, and oracle prompt. Other columns report ablations over input modality, context window, and prompt condition. CLAIR scores are judged by GPT-4.1 over 1,427 sampled pairs per run.
Modality Context Prompt
3-4(lr)5-7(lr)8-9 Model Default Text-Only Image-Only 1s 10s 60s Vanilla Coach
GPT-5.4 32.4\(_{\pm \text{1.3}}\) 31.8\(_{\pm \text{1.3}}\) 25.3\(_{\pm \text{1.2}}\) 25.4\(_{\pm \text{1.2}}\) 31.2\(_{\pm \text{1.3}}\) 32.7\(_{\pm \text{1.3}}\) 24.0\(_{\pm \text{1.2}}\) 23.8\(_{\pm \text{1.2}}\)
Gemini-3-Flash 33.9\(_{\pm \text{1.4}}\) 32.7\(_{\pm \text{1.4}}\) 27.2\(_{\pm \text{1.3}}\) 28.4\(_{\pm \text{1.3}}\) 32.9\(_{\pm \text{1.4}}\) 34.6\(_{\pm \text{1.4}}\) 25.2\(_{\pm \text{1.3}}\) 25.0\(_{\pm \text{1.3}}\)
Gemini-3.1-Pro \(_{\pm \text{1.5}}\) \(_{\pm \text{1.5}}\) \(_{\pm \text{1.4}}\) \(_{\pm \text{1.4}}\) \(_{\pm \text{1.5}}\) \(_{\pm \text{1.5}}\) \(_{\pm \text{1.4}}\) \(_{\pm \text{1.4}}\)
Claude-Sonnet-4.6 32.3\(_{\pm \text{1.3}}\) 35.9\(_{\pm \text{1.4}}\) 23.7\(_{\pm \text{1.2}}\) 25.6\(_{\pm \text{1.2}}\) 31.4\(_{\pm \text{1.3}}\) 32.3\(_{\pm \text{1.4}}\) 22.4\(_{\pm \text{1.2}}\) 24.1\(_{\pm \text{1.2}}\)
Qwen-3-VL-Instruct 27.0\(_{\pm \text{1.2}}\) 28.3\(_{\pm \text{1.2}}\) 20.2\(_{\pm \text{1.0}}\) 23.0\(_{\pm \text{1.1}}\) 27.0\(_{\pm \text{1.2}}\) 26.7\(_{\pm \text{1.2}}\) 18.8\(_{\pm \text{1.1}}\) 19.5\(_{\pm \text{1.1}}\)
Llama-4-Scout 19.5\(_{\pm \text{1.0}}\) 24.4\(_{\pm \text{1.2}}\) 16.3\(_{\pm \text{0.8}}\) 18.6\(_{\pm \text{1.0}}\) 20.2\(_{\pm \text{1.0}}\) 19.7\(_{\pm \text{1.0}}\) 14.8\(_{\pm \text{0.8}}\) 14.2\(_{\pm \text{0.9}}\)

4pt

6 Do Models Speak Like Humans?↩︎

At the corpus level, model coaches do not speak like human coaches. Table 2 compares each model’s utterance distribution with the human coach distribution using MAUVE [53]. All model scores are low, even the best Gemini-3.1-Pro with the coach prompt (0.118 \(_{\pm\text{0.049}}\)). We further evaluate whether individual model utterances match the content and quality of human coach responses given the same context.

6.1 Automatic Evaluation↩︎

Data. We randomly sampled 1,427 coach utterances across 5 software domains (300 per domain), balancing across coaching methods within each domain (targeting 30 per method). When a method had fewer than 30 utterances, all available examples were included. We excluded utterances tagged as Not Enough Information or Non-Instructive.

Conditions. By default, models receive both dialogue history and visual context sampled at 1 fps from the previous 30 seconds, and use an oracle prompt that reveals the coaching method used in the reference human coach utterance (see Appendix 15). We ablate three factors to compare model performance: (1) Input modality: We compare text-only context, visual-only context, and combined text-visual context, (2) Context length: We vary the context window across 1s, 10s, 30s, and 60s and (3) Prompt: We compare a vanilla prompt, a coach prompt, and the oracle prompt.

Metrics. We compute content similarity between model utterances and human reference ones using CLAIR [54]. We also report standard reference-based generation metrics, including BLEU, METEOR, ROUGE-L, and BERTScore, as well as CLAIR scores split by software application and coaching method in Appendix 14.

Results. Table 3 reports CLAIR scores of all models and conditions. Gemini-3.1-Pro achieves the highest overall scores, while Llama-4-Scout lags considerably behind across all conditions.

Visual context is underutilized. Dropping visual input and using text alone causes only a modest score change for most models (e.g., Gemini-3.1-Pro: 41.4\(_{\pm\text{1.5}}\) \(\to\) 41.9\(_{\pm\text{1.5}}\)), yet providing visual input only causes a sharp drop across all models (e.g., Gemini-3.1-Pro: 41.4\(_{\pm\text{1.5}}\) \(\to\) 30.0\(_{\pm\text{1.4}}\)), suggesting models rely primarily on text.

More context improves utterance quality. Shortening the context window to 1s substantially degrades performance across all models (e.g., Gemini-3.1-Pro: 41.4\(_{\pm\text{1.5}}\) \(\to\) 33.7\(_{\pm\text{1.4}}\); Qwen-3-VL-Instruct: 27.0\(_{\pm\text{1.2}}\) \(\to\) 23.0\(_{\pm\text{1.1}}\)), while extending to 60s yields marginal further gains. Models benefit from observing a recent screen and conversations between learner and coach, but returns diminish beyond 10 seconds.

6.2 Human Expert Evaluation↩︎

Model-generated utterances may differ from the reference and still be valid and relevant. We therefore conduct a human expert evaluation to assess the quality of model-generated utterances.

Method. We randomly sampled a subset (N=100, 20 utterances per software tool) from the dataset in Section 6.1 and recruited 15 expert evaluators (3 per software tool). Each evaluator judges 20 instances (2 per coaching method), and each instance was independently annotated by all 3 evaluators assigned to that software. For each instance, evaluators saw the task description, prior dialogue, and screen context. They ranked 3 candidate coaching utterances (human coach, Gemini-3.1-Pro Vanilla, and Gemini-3.1-Pro Oracle) and then rated each utterance along 3 dimensions using a three-point scale (1 = not, 2 = partially, 3 = very), drawn from prior VQA and proactive assistant work [20], [55]: (1) Relevance: whether the utterance was grounded in visual and conversational content, (2) Correctness: how correct the utterance was and (3) Naturalness: whether the utterance sounded natural.

Figure 7: Mean normalized ranking (\uparrow) and ratings (\uparrow) for human (H), Gemini-3.1-Pro Vanilla prompt (G(V)), and Oracle prompt (G(O)) coaching utterances (* = p < 0.05, ** = p < 0.01, Friedman test followed by pairwise Wilcoxon signed-rank tests, Bonferroni correction).

Results. Figure 7 shows human evaluators ranked human and Oracle conditions significantly higher than Vanilla (\(p<.01\) and \(p<.05\) respectively), suggesting that knowing the target coaching methods improves preference. Models scored comparably to humans on Relevance and Correctness, while human utterances were rated significantly more natural than Vanilla (\(p<.05\)), implying models generate locally plausible but unnatural utterances. Expert evaluators noted that effective coaching should be concise, accurate, and grounded in the learner’s current screen state. Coaches should give direct instruction when learners are stuck, while encouraging independent thinking/using prior knowledge when possible.

7 How Do Models Interact with Humans?↩︎

We have shown that models speak and coach differently from human coaches. But what are interaction dynamics and effectiveness when models coach novices to use software in real time?

Method. We conducted an interactive evaluation with 10 participants from professional networks (P1–P10, 2 per software application) with the same setup as our data collection, yielding 36 model–human coaching sessions. To balance inference time and quality, learners completed tasks with guidance from Gemini-3-Flash with the Vanilla prompt and a 10s context window at 1 fps. Learners can interact with models via asking questions or receiving the model’s proactive instructions every 20 seconds (See Appendix 14.2 for system setup).

7.1 Learning Outcome↩︎

Both human (\(N=72\)) and model (\(N=36\)) coaching produced significant improvement in matched pre/post tasks (\(p<.001\), Wilcoxon signed-rank test), but human coaching was substantially more effective (Table 4). Human coaching yielded a mean gain of 54.75% (33.49% \(\to\) 88.24%), with 81.9% of sessions improving and none declining. Model coaching produced a smaller mean gain of 31.67% (13.33% \(\to\) 45.00%), with only 58.3% of sessions improving, 25% showing no progress, and one declining. Models were far less effective than humans in teaching learners to use software independently. Appendix 16 reports subjective ratings, pre/post task outcomes, and qualitative feedback.

Table 4: Progress outcomes in human (H; \(N=72\)) and model (M; \(N=36\)) coaching sessions.
Progress Setup Tutorial Pre-Task Post-Task
100% H 72 (100.0%) 13 (18.1%) 49 (68.1%)
M 9 (25.0%) 2 (5.6%) 9 (25.0%)
50–100% H 0 (0.0%) 7 (9.7%) 15 (20.8%)
M 7 (19.4%) 2 (5.6%) 6 (16.7%)
0–50% H 0 (0.0%) 22 (30.6%) 8 (11.1%)
M 15 (41.7%) 8 (22.2%) 12 (33.3%)
0% H 0 (0.0%) 30 (41.7%) 0 (0.0%)
M 5 (13.9%) 24 (66.7%) 9 (25.0%)
Total H 72 (100.0%) 72 (100.0%) 72 (100.0%)
M 36 (100.0%) 36 (100.0%) 36 (100.0%)

3pt

7.2 Language Use↩︎

Figure 8: (A) Token length per utterance and (B) distribution of dialogue acts of coaches (C) and learners (L) in human (H; N=72) and model (M; N=36) coaching sessions.

The 36 model-human coaching sessions comprise 3,241 turns. Compared to human coaching sessions, models use more tokens per utterance, extensively use Action Directives (63% vs. 37%), and rarely use Inform (14% vs. 29%) that build learner understanding and Info Request (2% vs. 7%) to check-in learning state. Human learners, in turn, ask more questions (29% vs. 13%) and provide more information (20% vs. 9%), primarily due to the lack of proactivity of models (Figure 8).

7.3 Failure Case Analysis↩︎

Figure 9 shows representative failure cases in model coaching sessions.

Figure 9: Representative (A) communication and (B) grounding failures in interactive evaluation.

Communication Gap. Models by default generated step-by-step instructions. While learners could replicate operations, they did not understand how or why (P4, P7, P9). Learners were still confused about the high-level plan and how to think about completing the task even after successful tutorials, leading to worse outcomes. Six participants found models too verbose and terminology-heavy, and novices found it hard to follow multiple steps and locate the relevant tools hidden in layers of menus (Figure 9 A). P3 observed that they can follow simple steps at the beginning, but guidance became increasingly confusing as the task state became more complex. Thus, they wanted slower and more concise guidance on where and how to find the target tool. P8 also suggested that, like a human coach, models could point or describe nearby icons to help learners find the right tool.

Grounding Gap. Models fell short in tracking learners’ current screen state, which required learners to explicitly describe progress (P1, P2, P5, P6) or ask what to do next (P3, P9, P10). P1 reported that they have to say “I’m done” to make the model move on to the next step. This limitation of real-time multimodal understanding prevented the model from detecting and recovering when learners were stuck or had gone off track. For example, P2 was told to click a button that did not exist, yet the model kept repeating the same instruction. Similarly, P5 could not find where to create a pivot sheet, but the model repeated the same long instruction rather than trying a different approach. Models also did not point out mistakes unless explicitly asked. Even when they did respond, models sometimes hallucinated guidance based on misleading utterances rather than the screen state (Figure 9 B).

8 Discussion and Conclusion↩︎

We introduce DigitalCoach, the first multimodal dataset of human expert-novice computer use coaching sessions. Our analysis reveals that current state-of-the-art multimodal models exhibit significant communication and grounding gaps compared to human coaches: they more frequently use direct instruction, and fail to adapt to the learner’s evolving screen state. In an interactive evaluation, these gaps directly result in worse learning outcomes, with model-coached learners completing fewer milestones and retaining fewer skills.

Agency-Preserving Language Technology. Effective coaching balances direct instruction with learner agency. Human experts combine procedural guidance with explanation to build independence rather than replacing learner agency. For surface-level friction such as locating a menu item, direct instruction reduces unnecessary cognitive load, while for conceptual tasks such as design, agents should instead encourage reflection.

Proactive and Grounded Multimodal Agents. Our findings suggest that effective coaching agents must go beyond reactive question answering. Agents should proactively monitor learner progress and select pedagogical interventions accordingly. This requires tighter multimodal grounding, including real-time screen understanding and spatially situated feedback such as screen annotations to support the coaching that humans provide naturally.

Acknowledgments↩︎

This research was supported by an Amazon Research Award, a Google gift, a Google ML and Systems Junior Faculty Award, a Technical AI Safety Research award from Coefficient Giving, and an NVIDIA Academic Grant Program award. We also thank our study participants for their time and valuable contribution to this work.

Limitations↩︎

Our dataset has several limitations. First, all sessions were collected on a Windows laptop to standardize the environment and reflect industry standards (e.g., Blender is more commonly used on Windows). Future research could include multiple operating systems to capture diverse UI layouts and interaction behaviors. Second, although DigitalCoach captures 28.1 hours of coaching sessions across diverse software domains and participants, future work could expand the dataset with more sessions, more diverse software applications, and a broader range of novice and expert participants. Third, spoken dialogue often contains pauses, repairs, partial phrases, and overlaps. As a result, utterance segmentation and annotation can be ambiguous. We mitigate this by using explicit segmentation and annotation guidelines. Future work could explore alternative segmentation and annotation methods on our raw data. Finally, evaluating computer use coaching remains an open challenge. Task completion alone does not fully capture learning. While our pre/post tasks provide an effective measure of learning, they cannot fully capture skill retention over time. Future benchmarks can therefore evaluate coaching along multiple dimensions (e.g., agency, task milestones, interaction traces, and pre/post tasks) and multiple domains (e.g., open-ended creative tasks). DigitalCoach opens directions for building gym-style benchmarks for computer use coaching agents.

Ethical Considerations↩︎

Human Data Collection and Evaluation Our human evaluation study was approved by our institution’s Institutional Review Board (IRB). In our study, we ensured that all participants were compensated fairly for their time and contributions. The payment was determined based on the average market rate for such studies, participant expertise, and the complexity and duration of the tasks ($20 / hour for learners and human evaluators; $20 – 60 / hour for coaches, depending on their experiences).

Hallucinations in Language Models Our work also uses LLMs to annotate dialogue acts and coaching methods. While we have shown that these approaches better align with humans, LLMs are not free from potential hallucinations and can lead to inaccurate results.

Automation vs. Augmentation. While LLM agents are increasingly framed as replacements for human labor, our work points to a different role for agents: not automating tasks, but supporting humans in developing new skills. However, this framing also requires caution. For example, coaching agents could be deployed as substitutes for human experts and thus reduce access to human mentorship and weaken the social and emotional support from human coaches. We therefore view coaching agents as a complement to, rather than a replacement for, human expertise.

Trust in Coaching Agents. Learners may place high trust in computer use coaching agents, especially when they lack the expertise to judge if its guidance is correct. Incorrect or overconfident instructions can mislead learners and reinforce misconceptions. We are aware that future design of computer use coaching agents should communicate uncertainty, support verification, and make it easy for learners to question or override their guidance.

Appendix↩︎

Our appendix is organized as follows:

  • 9 provides detailed statistics and linguistic examples of the dataset.

  • 10 outlines the experimental platform setup and facilitation scripts.

  • 11 catalogs the specific software tasks across the five tested domains.

  • 12 details the human-in-the-loop transcript processing workflow.

  • 13 defines the dialogue act and coaching methods codebooks.

  • 14 details model evaluation baselines and metrics.

  • 15 provides the full text prompts utilized for LLM pipelines.

  • 16 provides quantitative and qualitative human learning outcomes.

  • 17 discusses the license of produced artifacts.

  • 18 provides information on settings for used code packages.

  • 19 states our AI Use Disclosure.

9 Dataset Details↩︎

To support further research, the DigitalCoach dataset, including the screen recording, audio transcripts, mouse and keyboard events, and the file snapshots, will be made publicly available upon publication. The DigitalCoach dataset will be accessible on Hugging Face and the evaluation code on GitHub.

Figure 10: Distribution of number of utterances across 72 sessions. Each bar shows the number of recorded coaching sessions within a duration range.
Figure 11: Distribution of video durations across 72 sessions. Each bar shows the number of recorded coaching sessions within a duration range.
Table 5: Summary statistics of .
Statistic Value
# Software Applications 5
# Participants 40
# Sessions 72
Video Duration h
Avg. Duration
Min Duration
Max Duration
# Total Turns
% Coach Turns 65.81%
% Learner Turns 34.19%
Avg. Turns per Session 303.36
Avg. Turn Tokens 9.69
# Input Events
# File Snapshots

In this section, we provide detailed information about DigitalCoach and several examples of linguistic phenomena. Table 5 shows detailed statistics of DigitalCoach. Figure 10 shows the distribution of number of utterances across all sessions. Figure 11 shows the distribution of duration across all sessions. Figure 20 shows the distribution of coaching methods across the conversation progress of all 72 sessions.

9.1 Additional Language Statistics↩︎

We observed linguistic phenomena in DigitalCoach. Token frequency (Figure 12) follows a Zipfian distribution. Token length (Figure 13) and gap length((Figure 14) follow exponential distributions, with fewer tokens and smaller gaps being most common.

Figure 12: Token frequencies in DigitalCoach follow a Zipfian distribution.
Figure 13: Token length (number of words per utterance) follows an exponential distribution.
Figure 14: Distribution of gap length follows an exponential distribution. We define the gap as the duration of silence between the end of one speaker and the beginning of the next speaker [56].

9.2 Example Linguistic Phenomena↩︎

In this section we provide examples of the linguistic phenomena observed in our coaching sessions.

Convention Formation. 0104/C2L2:

  • Turn 386 (Coach): “Alright, now we want to rename this to…Card.”

  • Turn 387 (Learner): “Card.”

  • Turn 388 (Coach): “Mhm. And let’s make card have a corner radius of 8.”

Spatial Referencing. 0104/C2L2:

  • Turn 60 (Coach): “Yep, and then just rotate it, um, by dragging on this corner right here…”

  • Turn 62 (Coach): “Just hover over this part.”

  • Turn 63 (Learner): “Hover in?”

  • Turn 64 (Coach): “Hover over, like, slightly outside of it, like, where my pink is.”

  • Turn 65 (Learner): “Ah, slightly— okay.”

Repair. 0104/C2L2:

  • Turn 84 (Coach): “Yep, now we have a, um, like, component for this play button.”

  • Turn 85 (Coach): “Alright, so now we want to create, like, our album art frame, so…”

  • Turn 86 (Learner): “Wait, well, component means…uh, what again, exactly? It’s interactable with…”

  • Turns 87–90 (Coach): “So, components are, like, the parent, um, designs……That’s a variant of this parent component.”

  • Turn 91 (Learner): “Okay, okay, got it, got it.”

10 Study Protocol↩︎

Each study session involved one expert coach, one novice learner, and one researcher facilitator. Sessions lasted approximately 3 hours and consisted of consent and setup, followed by 3–4 task blocks. Each task block included a pre-task, a tutorial task, and a post-task.

10.1 Device Setup↩︎

The coach and learner communicated remotely via Zoom throughout the session. Learners conducted all software activities on a study laptop (i.e., Lenovo E14 Gen 6) and use research accounts rather than their personal account to ensure privacy.

We used Zoom as the communication platform and the source of synchronized audio and screen recordings of the session. We recorded at 25 frames per second with 1920×1128 resolution. The audio was recorded from the study laptop microphone in mono channel with a 48K sampling rate.

We used Workflow Induction Toolkit [57] to log mouse and keyboard actions on the study laptop. In addition, we implemented software-specific file loggers to save file snapshots periodically (i.e., every 10s), allowing us to reconstruct intermediate work states over time.

10.2 Study Script↩︎

One facilitator was present at each session. They were provided with the following script to facilitate the session.

10.2.1 Before Study↩︎

  • [Set up the Zoom meeting]

  • [Rename the participants as Coach, Learner, and Facilitator]

  • [Share the study laptop screen with only the software window visible]

  • [Hide the Zoom video pane and toolbar (Ctrl + Shift + Alt + H)]

Thank you both for joining today. I am [facilitator name], and I will be the session facilitator.

In this study, you will work together to complete a set of activities using [software name]. Today, [coach name] will act as the coach, and [learner name] will act as the learner. The goal of the study is to understand how experts provide novices with instructions to learn [software name].

During the session, we will record the learner’s screen, both participants’ audio, and interaction logs from the learner’s computer. I am now sending you the consent form and media release form. By signing these forms, you confirm that you understand the study procedures, that your questions have been answered, and that you agree to participate. [Send consent forms.]

We also have this survey we would like you to fill out that asks about your experiences with [software name]. Please fill it out and let us know when you’ve completed it. Let us know if you have any questions about it. [Send pre-study survey.]

[coach name], can you see the green pencil icon at the bottom left of your screen? That is the Zoom annotation tool. Set the annotation color to pink and the line thickness to thick. Please use it to briefly draw on the learner’s screen.

Next, [learner name], please grant remote control access to the coach. [coach name] may request remote control if needed during the coaching activity. [Ask the coach to test remote control by moving the mouse and typing briefly.]

If any technical issues arise during the study, please let me know through Zoom chat. Otherwise, I will remain silent during the task activities and will only intervene if necessary.

10.2.3 Task Block↩︎

Each task block lasted approximately 50 minutes and consisted of three phases:

  • Pre-task (up to 15 minutes)

  • Tutorial task (10–60 minutes)

  • Post-task (up to 15 minutes)

Pre-Task We will now begin the pre-task. In this task, [learner name] will work on the task independently based on the task description. [coach name] should remain silent. [learner name], you are encouraged to think aloud while working. You have up to 15 minutes to finish the task but you can stop at any time you want. [Share pre-task document and starter file.] [Start a 15-minute timer.]

Tutorial Task We will now begin the tutorial task. There are no restrictions on how the coach and the learner should communicate. [Share tutorial task document and starter file.]

Post-Task We will now begin the post-task. This task is the same as the pre-task. The learner will have 15 minutes to work on the task independently based on the task description. The coach should not provide instruction unless the learner appears to be stuck for 3 minutes, in which case the coach may provide at most one hint. [Share post-task document and starter file.] [Start a 15-minute timer.]

Actions Before Each Task

  • [Start file logger.]

  • [Start Workflow Induction Toolkit recorder.]

  • [Start Zoom recording.]

Actions After Each Task

  • [Stop Zoom recording.]

  • [Stop Workflow Induction Toolkit recorder.]

  • [Stop file logger.]

  • [Check that data have been saved correctly.]

11 Tasks↩︎

Table 6 lists all tasks used in DigitalCoach.

Table 6: Overview of tasks used in the study.
Software Task Description Source
Potion Bottle Combine basic geometric shapes and manipulate vector paths to build a magic potion bottle organized in frames. Official Tutorial
Loading Animation Create a looping loading animation using frames, ellipses, Smart Animate, and component variants for use in a mobile app. Official Tutorial
Responsive Card Build a responsive podcast episode card using Auto Layout, constraints, image fills, and components that adapt to different sizes. Official Tutorial
Flower Vase Illustrate a transparent glass vase with flowers by designing repeated petal shapes and placing curved stems inside the vase. Official Tutorial
Donut Modeling Create a donut mesh with icing using viewport navigation, Edit Mode, modifiers, and Sculpt Mode. Online Tutorial
Shading Assign and preview separate materials for a donut and icing using Blender’s Shading view. Online Tutorial
Dropping Animation Animate an existing donut model to fall downward along the Z axis using keyframes. Online Tutorial
Sprinkles Add procedurally generated sprinkles to an existing donut’s icing in Blender. Online Tutorial
Saddle Bracket Model a saddle bracket using sketching, extrusion, and semicircular cuts to match provided engineering dimensions. Online Tutorial
Geneva Cam Model a Geneva cam using sketch geometry, circular repetition, extrusion, and feature patterning from an engineering drawing. Online Tutorial
Hub Build a hub by revolving a base profile and adding repeated curved slots to match specified diameters and angular spacing. Online Tutorial
Beat Build a drum pattern with bass line using the Channel Rack and arrange patterns in the Playlist. Official Tutorial
Melody Create and edit a melody in the Piano Roll, shaping note timing and dynamics before placing it in the Playlist. Official Tutorial
Song Track Place, duplicate, and organize beat and melody patterns in the Playlist to build a fuller track structure. Official Tutorial
Mixer Route instruments to Mixer inserts and apply effects such as reverb and delay using FL Studio’s Mixer. Official Tutorial
SUMIFS Use SUMIFS to compute total revenue filtered by multiple conditions across a raw sales dataset. Online Tutorial
VLOOKUP Link two tables with VLOOKUP to auto-fetch prices, categories, and tax rates for revenue calculations. Online Tutorial
Pivot Table Summarize a dataset into multiple pivot views and charts to answer distinct business questions. Online Tutorial

4pt

12 Transcription↩︎

We use the automatically generated closed caption (i.e., .vtt file) from Zoom as a starting point as it includes accurate timestamps, speaker labels (coach vs.learner), and an initial segmentation of the dialogue.

12.1 Transcript Correction↩︎

We use an LLM (i.e., Gemini 3.1 Pro) to correct wrong words, homophone errors, dropped or extra words, or other mismatches with what is actually said. Prompt used for transcript correction can be found in Figure 25. One author then manually validate the transcript to correct the speaker label and utterances. We use following notation to indicate pauses and interruptions:

  • Use comma (,) to indicate a short speech pause or hesitation.

  • Use ellipsis (...) to indicate a longer pause or hesitation.

  • Use double dash (--) to indicate trailing off or interruption by another speaker.

  • Use parentheses (( )) to indicate an uncertain best-guess interpretation, e.g., (move). If no reasonable guess is available, leave the parentheses empty.

  • Use parentheses (( )) to replace any personal information, e.g., (facilitator).

In general, we will not annotate intonational features and non-spoken noises.

12.2 Utterance Segmentation↩︎

Three authors split and merged the corrected utterances according to the following guideline:

  1. Split the utterance if there is a pause of 2 seconds or longer.

  2. Use changes in intonation and what can be inferred from the semantics to decide whether to split for shorter pauses.

    1. Descending pitch can be a hint that an utterance is ending.

    2. If it is difficult to assign different functional meanings to two adjacent spans, they should likely be merged into one utterance.

  3. Context from the other speaker can also help (e.g., if one half of the utterance is responding directly to the other speaker, it may make sense to split).

  4. Use the multimodal context to decide whether to split.

    1. If a short pause coincides with a visible action in the screen state, this can be evidence for a new utterance.

    2. If the screen state remains unchanged and there is only a pause shorter than 2 seconds, the utterances of the same speaker should generally be merged.

12.3 Transcript Correction Interface↩︎

The transcript correction interface screenshot is shown in Figure 15.

Figure 15: Transcription correction interface. Editor can see the dialogue transcript alongside screen recording. They can use the interface to edit, split, merge each utterance and trim its start and end time.

13 Annotation↩︎

13.1 Human Annotation Method↩︎

To derive the taxonomy, three researchers used open coding on 200 turns to obtain potential coaching methods. Then, they met to merge together similar methods and create a codebook with a name, definition, and example for each coaching method. The researchers iteratively coded samples and revised the codebook to achieve final codes, containing 10 methods. During the coding process, we merged similar codes. Disagreements were resolved through discussion, and the finalized codebook was used for the annotations. Using the codebook with examples, three researchers annotated other 200 utterances and reached a substantial agreement for coaching methods.

13.2 Dialogue Acts Schema↩︎

The full dialogue acts definitions and examples provided to annotators are shown in Table 7.

Figure 16: Pairwise Cohen’s \kappa agreement among three human raters and three model annotators for dialogue act annotation. Rows and columns denote annotators, and each cell reports the pairwise agreement score. GPT-5.4 aligns with human rater the best.
Table 7: Dialogue act schema used to annotate coach and learner utterances.
Tag Definition Example
Info Request Seeks information, clarification, or confirmation from the addressee. “What do you want to do with them?”
Action Directive Guides the addressee toward an action. “Click on the top menu.”
Answer Provides information requested by a previous utterance. “Because the polygon is not closed.”
Inform Provides objective information or describes the current state. “The tempo is set to 120 BPM.”
Opinion Provides a subjective judgment, preference, or interpretation. “The alignment looks better now.”
Backchannel Signals attention, understanding, agreement, or receipt. “Okay.”
Not Enough Info Is too incomplete or ambiguous to classify reliably. “If you just—”

5pt

13.3 Coaching Methods Schema↩︎

The full coaching methods definitions and examples provided to annotators are shown in Table 8.

Table 8: Coaching method schema used to annotate expert instructional behavior.
Tag Definition Example
Direct Instruction Gives concrete procedural guidance about next action to take. “Click Sketch, and then select that edge.”
Plan Proposes a subgoal without specifying the exact action. “Let’s first make the basic shape, then add the details.”
Confirmation Provides affirmation on the learner’s action, result, or understanding. “Yes, that shape looks correct now.”
Diagnosis Provides feedback on the learner’s current state or identifies a problem. “The sketch isn’t closed, so the software can’t generate a solid.”
Explanation Explains a concept, tool, feature, or reason behind an action or outcome. “We use Extrude here because it turns a 2D profile into a 3D solid.”
Tip Provides a transferable shortcut, heuristic, or practical technique. “It’s often easier to model one side first and then mirror it.”
Clarification Asks about the learner’s goal, confusion, current state, or intended next step before giving guidance. “Does this make sense to you?”
Reflection Prompts the learner to evaluate an outcome, compare alternatives, or consider what happened. “Looking at the result, what do you think went wrong?”
Articulation Prompts the learner to verbalize their reasoning, plan, or decision process. “Can you walk me through your reasoning?”
Exploration Invites the learner to experiment, choose among alternatives, or practice themselves. “Try a couple of options and see which one you like better.”
Non-Instructive Talks about task-irrelevant topics or filler. “The internet is not good.”

5pt

Figure 17: Pairwise Cohen’s \kappa agreement among three human raters and three model annotators for coaching method annotation. Rows and columns denote annotators, and each cell reports the pairwise agreement score. GPT-5.4 aligns with human rater the best.

13.4 Additional Annotation Results↩︎

Figure 18 shows the distribution of dialogue acts of coaches across conversation progress. Figure 19 shows the distribution of dialogue acts of learners across conversation progress. Figure 20 shows the distribution of coaching methods across conversation progress. Figure 21 shows top-10 most frequent coaching method bigrams..

Figure 18: Distribution of dialogue acts of coaches across conversation progress. Method use remains relatively stable as the conversation progresses, except for the beginning and the end 10%.
Figure 19: Distribution of dialogue acts of learners across conversation progress. Dialogue acts remain relatively stable as the conversation progresses, except for the beginning and the end 10%.
Figure 20: Distribution of coaching methods across conversation progress. Method use remains relatively stable as the conversation progresses, except for the beginning and the end 10%.
Figure 21: Top-10 most frequent coaching method bigrams.

13.5 Annotation Interface↩︎

The annotation interface screenshot is shown in Figure 22.

Figure 22: Annotation interface for utterance labeling. Annotators can see the dialogue transcript alongside screen recording and assign tags to each utterance.

14 Additional Evaluation Results↩︎

Table 9 shows Jensen-Shannon divergence (JSD) for dialogue-act and coaching-method n-gram distributions. Table 10 shows BLEU results for next coach utterance generation. Table 12 shows METEOR results for next coach utterance generation. Table 11 shows ROUGE-L results for next coach utterance generation. Table 13 shows BERTScore results for next coach utterance generation. Table 14 shows CLAIR scores (Default) split by software application. Table 15 shows CLAIR scores (Default) split by coaching method.

Table 9: Jensen-Shannon divergence (JSD) for dialogue act and coaching method \(n\)-gram distributions under Vanilla (V) and Coach (C) conditions. JSD is bounded in \([0, 1]\); lower values indicate closer alignment with human coach turns, with \(0\) corresponding to identical distributions. Bold scores indicate the lowest value and underlined scores indicate the second-lowest value within each label type and column.
Label Type Model 1-Gram 2-Gram 3-Gram
3-4(lr)5-6(lr)7-8 V C V C V C
Dialogue Acts GPT-5.4 0.513 0.472 0.709 0.681 0.839 0.824
Gemini-3-Flash 0.257 0.251 0.454 0.443 0.644 0.641
Gemini-3.1-Pro 0.235 0.204 0.427 0.386 0.641 0.605
Claude-Sonnet-4.6 0.257 0.207 0.456 0.396 0.657 0.612
Qwen-3-VL-Instruct 0.283 0.346 0.473 0.553 0.639 0.708
Llama-4-Scout 0.336 0.371 0.541 0.589 0.693 0.738
Coaching Methods GPT-5.4 0.635 0.587 0.827 0.790 0.931 0.910
Gemini-3-Flash 0.246 0.240 0.461 0.459 0.694 0.692
Gemini-3.1-Pro 0.259 0.212 0.481 0.438 0.729 0.700
Claude-Sonnet-4.6 0.301 0.255 0.538 0.491 0.781 0.753
Qwen-3-VL-Instruct 0.225 0.279 0.439 0.503 0.656 0.707
Llama-4-Scout 0.306 0.329 0.547 0.578 0.744 0.768

3.8pt

Table 10: BLEU scores (mean \(\pm\) 95% CI) for next coach utterance generation. The default setting uses text-image input, 30s context, and oracle prompt. Other columns report ablations over modality, context window, and prompt condition. Bold values indicate the best score per column, and underlined values indicate the second-highest score.
Modality Context Prompt
3-4(lr)5-7(lr)8-9 Model Default Text-Only Image-Only 1s 10s 60s Vanilla Coach
GPT-5.4 0.0150\(_{\pm \text{0.0010}}\) 0.0150\(_{\pm \text{0.0010}}\) 0.0130\(_{\pm \text{0.0010}}\) 0.0130\(_{\pm \text{0.0010}}\) 0.0150\(_{\pm \text{0.0010}}\) 0.0150\(_{\pm \text{0.0010}}\) 0.0120\(_{\pm \text{0.0010}}\) 0.0120\(_{\pm \text{0.0010}}\)
Gemini-3-Flash 0.0180\(_{\pm \text{0.0010}}\) 0.0180\(_{\pm \text{0.0010}}\) 0.0150\(_{\pm \text{0.0010}}\) 0.0160\(_{\pm \text{0.0010}}\) 0.0170\(_{\pm \text{0.0010}}\) 0.0180\(_{\pm \text{0.0010}}\) 0.0140\(_{\pm \text{0.0010}}\) 0.0140\(_{\pm \text{0.0010}}\)
Gemini-3.1-Pro \(_{\pm \text{0.0020}}\) \(_{\pm \text{0.0030}}\) \(_{\pm \text{0.0010}}\) \(_{\pm \text{0.0020}}\) \(_{\pm \text{0.0020}}\) \(_{\pm \text{0.0030}}\) \(_{\pm \text{0.0020}}\) \(_{\pm \text{0.0020}}\)
Claude-Sonnet-4.6 0.0170\(_{\pm \text{0.0010}}\) 0.0220\(_{\pm \text{0.0020}}\) 0.0140\(_{\pm \text{0.0010}}\) 0.0150\(_{\pm \text{0.0010}}\) 0.0170\(_{\pm \text{0.0020}}\) 0.0170\(_{\pm \text{0.0010}}\) 0.0130\(_{\pm \text{0.0010}}\) 0.0140\(_{\pm \text{0.0010}}\)
Qwen-3-VL-Instruct 0.0160\(_{\pm \text{0.0010}}\) 0.0150\(_{\pm \text{0.0010}}\) 0.0130\(_{\pm \text{0.0010}}\) 0.0140\(_{\pm \text{0.0010}}\) 0.0160\(_{\pm \text{0.0010}}\) 0.0160\(_{\pm \text{0.0010}}\) 0.0130\(_{\pm \text{0.0010}}\) 0.0130\(_{\pm \text{0.0010}}\)
Llama-4-Scout 0.0160\(_{\pm \text{0.0010}}\) 0.0170\(_{\pm \text{0.0010}}\) 0.0130\(_{\pm \text{0.0010}}\) 0.0140\(_{\pm \text{0.0010}}\) 0.0150\(_{\pm \text{0.0010}}\) 0.0160\(_{\pm \text{0.0010}}\) 0.0110\(_{\pm \text{0.0010}}\) 0.0110\(_{\pm \text{0.0010}}\)

4pt

Table 11: ROUGE-L scores (mean \(\pm\) 95% CI) for next coach utterance generation. The default setting uses text-image input, 30s context, and oracle prompt. Other columns report ablations over modality, context window, and prompt condition. Bold values indicate the best score per column, and underlined values indicate the second-highest score.
Modality Context Prompt
3-4(lr)5-7(lr)8-9 Model Default Text-Only Image-Only 1s 10s 60s Vanilla Coach
GPT-5.4 0.136\(_{\pm \text{0.004}}\) 0.134\(_{\pm \text{0.004}}\) 0.120\(_{\pm \text{0.004}}\) 0.121\(_{\pm \text{0.004}}\) 0.133\(_{\pm \text{0.004}}\) 0.134\(_{\pm \text{0.004}}\) 0.105\(_{\pm \text{0.004}}\) 0.108\(_{\pm \text{0.004}}\)
Gemini-3-Flash 0.140\(_{\pm \text{0.005}}\) 0.139\(_{\pm \text{0.005}}\) 0.124\(_{\pm \text{0.004}}\) 0.128\(_{\pm \text{0.004}}\) 0.138\(_{\pm \text{0.005}}\) 0.142\(_{\pm \text{0.005}}\) 0.115\(_{\pm \text{0.004}}\) 0.111\(_{\pm \text{0.004}}\)
Gemini-3.1-Pro \(_{\pm \text{0.006}}\) \(_{\pm \text{0.006}}\) \(_{\pm \text{0.005}}\) \(_{\pm \text{0.005}}\) \(_{\pm \text{0.006}}\) \(_{\pm \text{0.006}}\) \(_{\pm \text{0.005}}\) \(_{\pm \text{0.005}}\)
Claude-Sonnet-4.6 0.149\(_{\pm \text{0.004}}\) 0.164\(_{\pm \text{0.005}}\) \(_{\pm \text{0.004}}\) 0.134\(_{\pm \text{0.004}}\) 0.147\(_{\pm \text{0.005}}\) 0.152\(_{\pm \text{0.004}}\) 0.126\(_{\pm \text{0.004}}\) 0.127\(_{\pm \text{0.004}}\)
Qwen-3-VL-Instruct 0.129\(_{\pm \text{0.004}}\) 0.126\(_{\pm \text{0.004}}\) 0.112\(_{\pm \text{0.004}}\) 0.117\(_{\pm \text{0.004}}\) 0.128\(_{\pm \text{0.004}}\) 0.129\(_{\pm \text{0.004}}\) 0.105\(_{\pm \text{0.004}}\) 0.097\(_{\pm \text{0.004}}\)
Llama-4-Scout 0.125\(_{\pm \text{0.004}}\) 0.134\(_{\pm \text{0.004}}\) 0.115\(_{\pm \text{0.004}}\) 0.120\(_{\pm \text{0.004}}\) 0.123\(_{\pm \text{0.005}}\) 0.121\(_{\pm \text{0.005}}\) 0.095\(_{\pm \text{0.004}}\) 0.089\(_{\pm \text{0.004}}\)

4pt

Table 12: METEOR scores (mean \(\pm\) 95% CI) for next coach utterance generation. The default setting uses text-image input, 30s context, and oracle prompt. Other columns report ablations over modality, context window, and prompt condition. Bold values indicate the best score per column, and underlined values indicate the second-highest score.
Modality Context Prompt
3-4(lr)5-7(lr)8-9 Model Default Text-Only Image-Only 1s 10s 60s Vanilla Coach
GPT-5.4 0.143\(_{\pm \text{0.005}}\) 0.141\(_{\pm \text{0.005}}\) 0.127\(_{\pm \text{0.005}}\) 0.127\(_{\pm \text{0.005}}\) 0.141\(_{\pm \text{0.005}}\) 0.144\(_{\pm \text{0.005}}\) 0.107\(_{\pm \text{0.004}}\) 0.106\(_{\pm \text{0.004}}\)
Gemini-3-Flash 0.144\(_{\pm \text{0.006}}\) 0.146\(_{\pm \text{0.005}}\) 0.125\(_{\pm \text{0.005}}\) 0.130\(_{\pm \text{0.005}}\) 0.142\(_{\pm \text{0.006}}\) 0.147\(_{\pm \text{0.006}}\) 0.116\(_{\pm \text{0.005}}\) 0.109\(_{\pm \text{0.005}}\)
Gemini-3.1-Pro 0.164\(_{\pm \text{0.007}}\) 0.166\(_{\pm \text{0.007}}\) 0.126\(_{\pm \text{0.005}}\) 0.140\(_{\pm \text{0.006}}\) 0.157\(_{\pm \text{0.007}}\) 0.173\(_{\pm \text{0.007}}\) 0.138\(_{\pm \text{0.006}}\) 0.136\(_{\pm \text{0.006}}\)
Claude-Sonnet-4.6 \(_{\pm \text{0.006}}\) \(_{\pm \text{0.007}}\) \(_{\pm \text{0.005}}\) \(_{\pm \text{0.005}}\) \(_{\pm \text{0.006}}\) \(_{\pm \text{0.006}}\) \(_{\pm \text{0.005}}\) \(_{\pm \text{0.005}}\)
Qwen-3-VL-Instruct 0.125\(_{\pm \text{0.005}}\) 0.123\(_{\pm \text{0.005}}\) 0.108\(_{\pm \text{0.004}}\) 0.109\(_{\pm \text{0.004}}\) 0.121\(_{\pm \text{0.005}}\) 0.123\(_{\pm \text{0.005}}\) 0.098\(_{\pm \text{0.005}}\) 0.086\(_{\pm \text{0.004}}\)
Llama-4-Scout 0.116\(_{\pm \text{0.005}}\) 0.142\(_{\pm \text{0.005}}\) 0.114\(_{\pm \text{0.005}}\) 0.127\(_{\pm \text{0.005}}\) 0.117\(_{\pm \text{0.005}}\) 0.113\(_{\pm \text{0.005}}\) 0.091\(_{\pm \text{0.004}}\) 0.079\(_{\pm \text{0.004}}\)

4pt

Table 13: BERTScore scores (mean \(\pm\) 95% CI) for next coach utterance generation. The default setting uses text-image input, 30s context, and oracle prompt. Other columns report ablations over modality, context window, and prompt condition. Bold values indicate the best score per column, and underlined values indicate the second-highest score.
Modality Context Prompt
3-4(lr)5-7(lr)8-9 Model Default Text-Only Image-Only 1s 10s 60s Vanilla Coach
GPT-5.4 0.840\(_{\pm \text{0.001}}\) 0.839\(_{\pm \text{0.001}}\) 0.836\(_{\pm \text{0.001}}\) 0.836\(_{\pm \text{0.001}}\) 0.840\(_{\pm \text{0.001}}\) 0.840\(_{\pm \text{0.001}}\) 0.834\(_{\pm \text{0.001}}\) 0.835\(_{\pm \text{0.001}}\)
Gemini-3-Flash 0.846\(_{\pm \text{0.001}}\) 0.845\(_{\pm \text{0.001}}\) 0.843\(_{\pm \text{0.001}}\) 0.844\(_{\pm \text{0.001}}\) 0.845\(_{\pm \text{0.001}}\) 0.846\(_{\pm \text{0.001}}\) 0.839\(_{\pm \text{0.001}}\) 0.839\(_{\pm \text{0.001}}\)
Gemini-3.1-Pro \(_{\pm \text{0.001}}\) \(_{\pm \text{0.001}}\) \(_{\pm \text{0.001}}\) \(_{\pm \text{0.001}}\) \(_{\pm \text{0.001}}\) \(_{\pm \text{0.001}}\) \(_{\pm \text{0.001}}\) \(_{\pm \text{0.001}}\)
Claude-Sonnet-4.6 0.841\(_{\pm \text{0.001}}\) 0.845\(_{\pm \text{0.001}}\) 0.837\(_{\pm \text{0.001}}\) 0.840\(_{\pm \text{0.001}}\) 0.842\(_{\pm \text{0.001}}\) 0.842\(_{\pm \text{0.001}}\) 0.835\(_{\pm \text{0.001}}\) 0.836\(_{\pm \text{0.001}}\)
Qwen-3-VL-Instruct 0.841\(_{\pm \text{0.001}}\) 0.840\(_{\pm \text{0.001}}\) 0.837\(_{\pm \text{0.001}}\) 0.838\(_{\pm \text{0.001}}\) 0.840\(_{\pm \text{0.001}}\) 0.840\(_{\pm \text{0.001}}\) 0.837\(_{\pm \text{0.001}}\) 0.837\(_{\pm \text{0.001}}\)
Llama-4-Scout 0.840\(_{\pm \text{0.001}}\) 0.841\(_{\pm \text{0.001}}\) 0.837\(_{\pm \text{0.001}}\) 0.838\(_{\pm \text{0.001}}\) 0.839\(_{\pm \text{0.001}}\) 0.840\(_{\pm \text{0.001}}\) 0.836\(_{\pm \text{0.001}}\) 0.835\(_{\pm \text{0.001}}\)

4pt

Table 14: CLAIR scores (mean \(\pm\) 95% CI) by domain under the text+image input, 30s, Oracle condition. Bold values indicate the best score per column, and underlined values indicate the second-highest score.
Model Overall Figma Blender OnShape FL Studio Excel
GPT-5.4 32.4\(_{\pm \text{1.3}}\) 32.4\(_{\pm \text{3.0}}\) 29.8\(_{\pm \text{2.9}}\) 32.7\(_{\pm \text{2.8}}\) 35.0\(_{\pm \text{3.2}}\) 32.5\(_{\pm \text{3.0}}\)
Gemini-3-Flash 33.9\(_{\pm \text{1.4}}\) 36.8\(_{\pm \text{3.3}}\) 30.6\(_{\pm \text{3.0}}\) 35.1\(_{\pm \text{3.1}}\) 33.7\(_{\pm \text{3.4}}\) 33.2\(_{\pm \text{3.1}}\)
Gemini-3.1-Pro \(_{\pm \text{1.5}}\) \(_{\pm \text{3.5}}\) \(_{\pm \text{3.3}}\) \(_{\pm \text{3.3}}\) \(_{\pm \text{3.6}}\) \(_{\pm \text{3.3}}\)
Claude-Sonnet-4.6 32.3\(_{\pm \text{1.3}}\) 32.4\(_{\pm \text{3.0}}\) 30.4\(_{\pm \text{3.0}}\) 34.1\(_{\pm \text{3.0}}\) 33.3\(_{\pm \text{3.1}}\) 31.5\(_{\pm \text{2.9}}\)
Qwen-3-VL-Instruct 27.0\(_{\pm \text{1.2}}\) 26.2\(_{\pm \text{2.7}}\) 24.4\(_{\pm \text{2.5}}\) 28.0\(_{\pm \text{2.5}}\) 29.1\(_{\pm \text{2.9}}\) 27.7\(_{\pm \text{2.8}}\)
Llama-4-Scout 19.5\(_{\pm \text{1.0}}\) 19.8\(_{\pm \text{2.3}}\) 17.3\(_{\pm \text{2.0}}\) 19.0\(_{\pm \text{1.9}}\) 21.9\(_{\pm \text{2.5}}\) 19.8\(_{\pm \text{2.2}}\)

4pt

Table 15: CLAIR scores (mean \(\pm\) 95% CI) by coaching method under the Text+Image, 30s, Oracle condition. Bold values indicate the best score per column, and underlined values indicate the second-highest score.
Model Overall Artic. Clarif. Confirm. Diagn. Direct Explan. Explor. Plan Reflect. Tip
GPT-5.4 32.4\(_{\pm \text{1.3}}\) 35.6\(_{\pm \text{5.1}}\) 23.4\(_{\pm \text{3.6}}\) 41.9\(_{\pm \text{4.3}}\) 23.2\(_{\pm \text{3.5}}\) 32.1\(_{\pm \text{4.1}}\) 38.6\(_{\pm \text{4.3}}\) 33.6\(_{\pm \text{4.1}}\) 35.6\(_{\pm \text{4.1}}\) 29.1\(_{\pm \text{3.9}}\) 31.7\(_{\pm \text{4.1}}\)
Gemini-3-Flash 33.9\(_{\pm \text{1.4}}\) 40.6\(_{\pm \text{5.4}}\) 27.0\(_{\pm \text{4.3}}\) 37.6\(_{\pm \text{4.4}}\) 30.0\(_{\pm \text{4.4}}\) 33.2\(_{\pm \text{4.5}}\) 37.4\(_{\pm \text{4.5}}\) 37.2\(_{\pm \text{4.2}}\) 38.4\(_{\pm \text{4.6}}\) 30.3\(_{\pm \text{4.2}}\) 29.0\(_{\pm \text{4.3}}\)
Gemini-3.1-Pro \(_{\pm \text{1.5}}\) \(_{\pm \text{5.7}}\) \(_{\pm \text{4.7}}\) \(_{\pm \text{4.8}}\) \(_{\pm \text{4.4}}\) \(_{\pm \text{4.8}}\) \(_{\pm \text{4.7}}\) \(_{\pm \text{4.5}}\) \(_{\pm \text{4.6}}\) \(_{\pm \text{4.9}}\) \(_{\pm \text{4.6}}\)
Claude-Sonnet-4.6 32.3\(_{\pm \text{1.3}}\) 39.3\(_{\pm \text{4.9}}\) 24.4\(_{\pm \text{3.8}}\) 36.1\(_{\pm \text{4.3}}\) 27.7\(_{\pm \text{4.2}}\) 33.7\(_{\pm \text{4.3}}\) 36.0\(_{\pm \text{4.2}}\) 34.5\(_{\pm \text{4.2}}\) 33.3\(_{\pm \text{4.2}}\) 28.2\(_{\pm \text{3.8}}\) 32.1\(_{\pm \text{4.1}}\)
Qwen-3-VL-Instruct 27.0\(_{\pm \text{1.2}}\) 28.1\(_{\pm \text{4.5}}\) 19.5\(_{\pm \text{2.9}}\) 33.0\(_{\pm \text{4.1}}\) 19.7\(_{\pm \text{2.8}}\) 24.4\(_{\pm \text{3.5}}\) 32.3\(_{\pm \text{4.1}}\) 28.7\(_{\pm \text{3.7}}\) 33.2\(_{\pm \text{4.0}}\) 26.0\(_{\pm \text{3.5}}\) 25.8\(_{\pm \text{3.7}}\)
Llama-4-Scout 19.5\(_{\pm \text{1.0}}\) 24.3\(_{\pm \text{4.1}}\) 15.3\(_{\pm \text{2.7}}\) 17.1\(_{\pm \text{2.8}}\) 15.0\(_{\pm \text{2.4}}\) 17.6\(_{\pm \text{2.8}}\) 20.3\(_{\pm \text{3.2}}\) 25.3\(_{\pm \text{3.6}}\) 24.9\(_{\pm \text{3.6}}\) 17.0\(_{\pm \text{2.4}}\) 19.5\(_{\pm \text{2.7}}\)

4pt

14.1 Human Expert Evaluation Setup↩︎

To ensure a fair comparison, all candidate utterances and prior chat messages are converted to audio via a AI text-to-speech voice. The human expert evaluation interface screenshot is shown in Figure 22.

Figure 23: Human expert evaluation interface. Raters can see the the previous conversation and screen context and listen to AI-narrated candidate coaching utterances.

14.2 Interactive Evaluation Setup↩︎

Interactive evaluation system uses a browser client (React, Stream Video WebRTC) paired with a Python backend (Vision agents, FastAPI). A novice shares screen and audio, and the software coach agent joins the same call server-side (Figure 24). By default, we route model inference through OpenRouter, Deepgram speech-to-text, and ElevenLabs text-to-speech. Proactive check-ins fire every 20s. Sessions log model–learner transcripts for offline analysis.

Figure 24: Interactive evaluation interface. A novice shares their screen and speaks over a live call. Model coach observes recent frames and dialogue and replies with audio.

15 Prompts↩︎

Figure 25 shows the automatic transcript correction prompt. Figure 26 shows the automatic dialogue act annotation prompt. Figure 27 shows the automatic coaching method annotation prompt. Figure 28 shows the next coach utterance generation prompt.

Figure 25: Prompt used for automatic transcript correction.
Figure 26: Prompt used for automatic dialogue act annotation.
Figure 27: Prompt used for automatic coaching method annotation.
Figure 28: Prompt structure for next coach utterance generation, including system prompt, recent dialogue, recent screen context, and final instruction. Colored text marks prompt condition additions: blue for both Coach and Oracle prompts, orange for Coach only, and red for Oracle only.

16 Additional Human Coaching Results↩︎

16.1 Problematic Outcome Tags↩︎

Table 16 shows problematic outcome tags for human and model coaching sessions.

Table 16: Problematic outcome tags for human coaching sessions (H; \(N=72\)) and model coaching sessions (M; \(N=36\)). Tags are not mutually exclusive.
Tag Setup Tutorial Task Pre-Task Post-Task
Gave up H 0 (0.0%) 19 (26.4%) 0 (0.0%)
M 15 (41.7%) 31 (86.1%) 17 (47.2%)
Wrong method H 0 (0.0%) 17 (23.6%) 2 (2.8%)
M 3 (8.3%) 7 (19.4%) 2 (5.6%)
Mistakes H 0 (0.0%) 3 (4.2%) 5 (6.9%)
M 5 (13.9%) 0 (0.0%) 7 (19.4%)
Missed steps H 0 (0.0%) 2 (2.8%) 8 (11.1%)
M 7 (19.4%) 0 (0.0%) 9 (25.0%)

4pt

16.2 Pre/Post Task Outcome Examples↩︎

Table 17 shows examples of pre-task and post-task outcomes in visual tasks (Figma, Blender, OnShape).

Table 17: Examples of pre-task and post-task outcomes in visual tasks (Figma, Blender, OnShape).
Software Task ID Session ID Target Pre-Task Post-Task
Figma 0104 C1L1 image image image
Blender 0202 C5L5 image image image
Blender 0202 C7L7 image image image
OnShape 0302 C8L8 image image image

5pt

16.3 Subjective Learning Outcome↩︎

Learners reported substantially higher confidence after completing the tasks. Averaging confidence (7-scale likert score, 7 = very confident) increased from \(\mu_{\mathrm{pre}}=2.49, \sigma_{\mathrm{pre}}=1.03\) to \(\mu_{\mathrm{post}}=5.90, \sigma_{\mathrm{post}}=0.90\), with an average improvement of \(\mu_{\mathrm{diff}}=3.41, \sigma_{\mathrm{diff}}=1.04\). A non-parametric Wilcoxon signed-rank test showed the increase is statistical significant \(W=0, p<.001\).

Coaches rated whether learners had acquired the relevant software skill for the task (7-scale likert score, 7 = the learner was fully equipped to complete similar tasks independently). Coach-rated skill increased from \(\mu_{\mathrm{pre}}=2.88, \sigma_{\mathrm{pre}}=2.06\) to \(\mu_{\mathrm{post}}=6.29, \sigma_{\mathrm{post}}=1.35\), with an average improvement of \(\mu_{\mathrm{diff}}=3.42, \sigma_{\mathrm{diff}}=1.97\). A Wilcoxon signed-rank test showed the increase was statistically significant \(W=0, p<.001\). In total, 66 of 72 task instances received higher coach-rated skill scores after coaching, 6 stayed the same, and none decreased.

16.4 Qualitative Feedback↩︎

Across both learners (L1-L20) and coaches (C1-C20), participants described effective computer use coaching an activity beyond simply giving step-by-step answers. It was a balance between procedural and conceptual guidance, guided practice, and learner agency.

Learners emphasized that practice was necessary for retention, especially after a dense coaching session. L10 noted that “more practice would be helpful to re-enforce the things I learnt,” because they were unsure “for how long I will retain the knowledge if I am not practicing it regularly.” Coaches echoed this point: C2 argued that “more practice with new tools would help with retention,” especially when learners first observe an effect and then reproduce it themselves with less guidance. This suggests that an effective digital coach should not only demonstrate procedures, but also create repeated opportunities for learners to apply newly learned operations.

Participants also emphasized that direct coaching is useful when learners lack basic orientation or do not know where to begin. L8 stated, “I want direct teaching when I lack the basic knowledge of the software,” while L20 found “detailed step-by-step” guidance helpful because each action was “illustrated in a digestible way.” Coaches similarly noted that in unfamiliar interfaces, direct pointing can reduce unnecessary friction. C13 explained that in Figma, many functions are difficult to find without prior familiarity, so “it makes more sense to have those directly pointed out.” However, participants also viewed direct instruction as only one part of effective teaching. L9 described a trade-off: direct answers are useful for “simple procedural questions,” but for design-related or conceptual tasks, they preferred “guided exploration and explanation.” Thus, digital coaches may need to dynamically shift between directive support and exploratory guidance depending on whether the learner’s immediate barrier is procedural, conceptual, or creative.

Another strong theme was that learners valued explanations of concepts, rationales, and tool functions, not only instructions about where to click. L17 said the coach first explained “the function of a component” and then showed “how to operate,” which strengthened both understanding and memory. C2 similarly argued that explanations and analogies help students understand “what specific numbers or functions do, instead of just clicking mindlessly through the steps.” Learners particularly valued explanations that made knowledge transferable. L10 appreciated learning “strategies and thought processes” in CAD that could transfer to other tasks. L8 described how understanding the difference between “extrude vs. revolve” helped them narrow down tool choices in a later post-task. L9 similarly said effective moments focused not only on how to complete the task, but also on why a certain approach works, making the knowledge transferable.

Participants also highlighted the importance of guided exploration, reflection, and learner agency. L9 described effective teaching as “guidance that helped me think rather than directly giving the answer,” using “minimal but meaningful cues” that allowed them to solve problems independently. C9 similarly described effective teaching as giving the learner “time to explore and think about the concepts” and then synthesize them individually. C20 preferred being able to ask learners questions rather than “just tell them what to do,” describing coaching as “collaborative.” Several learners valued moments where they tried, failed, and were corrected. L17 described “practicing, failing, learning what is actually correct, and then doing it again” as the best method because it forced them to think and understand what was wrong or right. L4 also found it most helpful when they tried to replicate what the coach taught, got stuck, and then received correction.

Visual grounding and screen annotation were repeatedly described as important for online computer use coaching. C12 said Zoom annotations were “incredibly helpful” because the coach could draw squares, arrows, and visual references without physically obstructing the screen. L13 said annotation made it easier to navigate the toolbar and find correct buttons, while L12 described screen annotations as “very effective in visually guiding me through the activity.” C2 described a useful pattern: first explain the concept, then annotate where to click with arrows or circles.

17 Artifact Licenses, Terms of Use, and Intended Use↩︎

Our work uses and creates several scientific artifacts: the DigitalCoach dataset, the software-task materials used to elicit coaching sessions, model outputs from the evaluated multimodal language models, and annotation/evaluation code. The DigitalCoach dataset is intended for research on situated, multimodal computer use coaching, human–AI collaboration, language grounding in graphical user interfaces, and pedagogical dialogue. It is not intended for use in systems that identify, profile, surveil, evaluate, or make consequential decisions about individual participants.

For artifacts created by this work, we will release only de-identified research data and accompanying documentation under a research-use license. The release will include derived and anonymized metadata, dialogue annotations, task metadata, model generations, and benchmark/evaluation scripts where permitted. Raw participant-identifying information, including names, contact information, and any uniquely identifying content observed in transcripts, screen recordings, or files, will not be released. All transcripts are manually reviewed for personally identifying information and redacted before release. Access to any higher-risk modalities, such as screen recordings or file snapshots, will be restricted to research use and subject to the dataset terms of use.

For existing artifacts, including the software applications, tutorials, APIs, models, and evaluation packages used in this study, our use was limited to research and evaluation. We followed the applicable licenses, API terms, and platform policies for each artifact. We cite the creators of datasets, benchmarks, models, and methods used in the paper where applicable. The use of tutorial-derived task materials was limited to creating study tasks and was not used to redistribute the original tutorials themselves. Model outputs are used only for research evaluation of coaching behavior and are not presented as authoritative instructions for end users.

18 Experimental Settings and Package Parameters↩︎

All model generations used the default settings of the corresponding model provider unless otherwise specified. We did not perform hyperparameter search over decoding parameters. Instead, we compared prompting conditions, input modalities, and context-window lengths as experimental factors. Specifically, we compared vanilla, coach, and oracle prompts; text-only, image-only, and text–visual context; and context windows of 1, 10, 30, and 60 seconds. The default automatic-evaluation setting used text–visual input, a 30-second context window, and the oracle prompt. The default interactive setting used Gemini-3-Flash, the vanilla prompt, a 10-second context window, and visual sampling at 1 frame per second.

For automatic evaluation, we used CLAIR, BLEU, METEOR, ROUGE-L, BERTScore, Self-BLEU, Vendi Score, MAUVE, Cohen’s \(\kappa\), Fleiss’ \(\kappa\), and F1 as reported in the paper. Unless otherwise stated, we used the default parameter settings from the corresponding public implementations. The exact package names, versions, and command-line arguments are provided in our released code and configuration files. We used GPT-4.1 as the judge model for CLAIR scoring in the automatic next-utterance evaluation.

19 AI Use Disclosure↩︎

The authors used AI-based tools to assist with code generation, editing, and writing during the preparation of this paper. Specifically, AI assistance was used to help draft and revise portions of the manuscript for clarity, grammar, and organization, and to support the development, debugging, and refinement of code used in the research workflow. All AI-generated or AI-assisted content, code, analyses, and interpretations were reviewed, verified, and, where necessary, modified by the authors. The authors take full responsibility for the accuracy, integrity, originality, and final content of the paper, including any code or text developed with AI assistance.

References↩︎

[1]
Z. Wang, A. Li, Z. Li, and X. Liu, “GenArtist: Multimodal LLM as an agent for unified image generation and editing,” in Advances in neural information processing systems 38: Annual conference on neural information processing systems 2024, NeurIPS 2024, vancouver, BC, canada, december 10 - 15, 2024, 2024, [Online]. Available: http://papers.nips.cc/paper\_files/paper/2024/hash/e7c786024ca718f2487712bfe9f51030-Abstract-Conference.html.
[2]
S. Yin et al., “Vision-as-Inverse-Graphics Agent via Interleaved Multimodal Reasoning,” ArXiv preprint, vol. abs/2601.11109. 2026, [Online]. Available: https://arxiv.org/abs/2601.11109.
[3]
S. Gao et al., “Empowering biomedical discovery with AI agents,” Cell, vol. 187 22, pp. 6125–6151, 2024, [Online]. Available: https://api.semanticscholar.org/CorpusId:268875818.
[4]
F. Gmeiner, K. Holstein, and N. Martelaro, “Prototyping Multimodal GenAI Real-Time Agents with Counterfactual Replays and Hybrid Wizard-of-Oz,” ArXiv preprint, vol. abs/2510.06872. 2025, [Online]. Available: https://arxiv.org/abs/2510.06872.
[5]
M. Huh, Z. Xue, U. Das, K. Ashutosh, K. Grauman, and A. Pavel, “Vid2Coach: Transforming How-To Videos into Task Assistants,” ArXiv preprint, vol. abs/2506.00717. 2025, [Online]. Available: https://arxiv.org/abs/2506.00717.
[6]
N. Tomlin et al., “Characterizing Language Use in a Collaborative Situated Game,” ArXiv preprint, vol. abs/2512.03381. 2025, [Online]. Available: https://arxiv.org/abs/2512.03381.
[7]
J. Macina et al., “MathDial: A dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems,” in Findings of the association for computational linguistics: EMNLP 2023, 2023, pp. 5602–5621, doi: 10.18653/v1/2023.findings-emnlp.372.
[8]
B. Man, G. Nehme, M. F. Alam, and F. Ahmed, “VideoCAD: A Dataset and Model for Learning Long-Horizon 3D CAD UI Interactions from Video,” ArXiv preprint, vol. abs/2505.24838. 2025, [Online]. Available: https://arxiv.org/abs/2505.24838.
[9]
D. Gao et al., “AssistGUI: Task-oriented PC graphical user interface automation,” in IEEE/CVF conference on computer vision and pattern recognition, CVPR 2024, seattle, WA, USA, june 16-22, 2024, 2024, pp. 13289–13298, doi: 10.1109/CVPR52733.2024.01262.
[10]
S. Ye, H. Shi, D. Shih, H. Yun, T. Roosta, and T. Shu, “RealWebAssist: A Benchmark for Long-Horizon Web Assistance with Real-World Users,” ArXiv preprint, vol. abs/2504.10445. 2025, [Online]. Available: https://arxiv.org/abs/2504.10445.
[11]
S. Yang et al., “GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Tasks,” ArXiv preprint, vol. abs/2603.25864. 2026, [Online]. Available: https://arxiv.org/abs/2603.25864.
[12]
M. Clemens and A. Marasović, “MixAssist: An Audio-Language Dataset for Co-Creative AI Assistance in Music Mixing,” ArXiv preprint, vol. abs/2507.06329. 2025, [Online]. Available: https://arxiv.org/abs/2507.06329.
[13]
J. F. Allen et al., _eprint: https://doi.org/10.1080/09528139508953799“The TRAINS project: A case study in building a conversational planning agent,” Journal of Experimental & Theoretical Artificial Intelligence, vol. 7, no. 1, pp. 7–48, 1995, doi: 10.1080/09528139508953799.
[14]
J. Allen and G. Ferguson, “Human-machine collaborative planning,” in Proceedings of the third international NASA workshop on planning and scheduling for space, 2002, pp. 27–29.
[15]
J. Lin, N. Tomlin, J. Andreas, and J. Eisner, “Decision-Oriented Dialogue for Human-AI Collaboration,” Transactions of the Association for Computational Linguistics, vol. 12, pp. 892–911, 2024, doi: 10.1162/tacl_a_00679.
[16]
O. Shaikh, K. Gligorić, A. Khetan, M. Gerstgrasser, D. Yang, and D. Jurafsky, “Grounding gaps in language model generations,” in Proceedings of the 2024 conference of the north american chapter of the association for computational linguistics: Human language technologies (volume 1: Long papers), 2024, pp. 6279–6296.
[17]
A. Suhr et al., “Executing instructions in situated collaborative interactions,” in Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), 2019, pp. 2119–2130, doi: 10.18653/v1/D19-1218.
[18]
C.-P. Bara, S. CH-Wang, and J. Chai, “MindCraft: Theory of mind modeling for situated dialogue in collaborative tasks,” in Proceedings of the 2021 conference on empirical methods in natural language processing, 2021, pp. 1112–1125, doi: 10.18653/v1/2021.emnlp-main.85.
[19]
T. Udagawa and A. Aizawa, “An annotated corpus of reference resolution for interpreting common grounding,” in The thirty-fourth AAAI conference on artificial intelligence, AAAI 2020, the thirty-second innovative applications of artificial intelligence conference, IAAI 2020, the tenth AAAI symposium on educational advances in artificial intelligence, EAAI 2020, new york, NY, USA, february 7-12, 2020, 2020, pp. 9081–9089, [Online]. Available: https://aaai.org/ojs/index.php/AAAI/article/view/6442.
[20]
Y. Zhang et al., “Proactive Assistant Dialogue Generation from Streaming Egocentric Videos,” ArXiv preprint, vol. abs/2506.05904, 2025, [Online]. Available: https://arxiv.org/abs/2506.05904.
[21]
A. Padmakumar et al., “TEACh: Task-driven embodied agents that chat,” in Thirty-sixth AAAI conference on artificial intelligence, AAAI 2022, thirty-fourth conference on innovative applications of artificial intelligence, IAAI 2022, the twelveth symposium on educational advances in artificial intelligence, EAAI 2022 virtual event, february 22 - march 1, 2022, 2022, pp. 2017–2025, [Online]. Available: https://ojs.aaai.org/index.php/AAAI/article/view/20097.
[22]
L. Aggarwal, V. Bahirwani, L. Li, and A. Colaco, “Generating dialogues from egocentric instructional videos for task assistance: Dataset, method and benchmark,” ArXiv preprint, vol. abs/2508.11192. 2025, [Online]. Available: https://arxiv.org/abs/2508.11192.
[23]
A. Bhattacharyya et al., “Can multi-modal LLMs provide live step-by-step task guidance?” Advances in Neural Information Processing Systems, vol. 38, pp. 22377–22410, 2025.
[24]
J.-H. Kim et al., “CoDraw: Collaborative drawing as a testbed for grounded goal-driven communication,” in Proceedings of the 57th annual meeting of the association for computational linguistics, 2019, pp. 6495–6513, doi: 10.18653/v1/P19-1651.
[25]
M. Gašić et al., “POMDP-based dialogue manager adaptation to extended domains,” in Proceedings of the SIGDIAL 2013 conference, 2013, pp. 214–222, [Online]. Available: https://aclanthology.org/W13-4035.
[26]
S. Yao, N. Shinn, P. Razavi, and K. R. Narasimhan, “{$\Tau$}-bench: A Benchmark for \underline{T}ool-\underline{A}gent-\underline{U}ser Interaction in Real-World Domains,” in Proceedings of the ICLR 2025 conference, 2024, Accessed: Apr. 14, 2026. [Online]. Available: https://openreview.net/forum?id=roNSXZpUDN.
[27]
A. Collins and M. Kapur, “Cognitive Apprenticeship,” in The Cambridge Handbook of the Learning Sciences, 3rd ed., R. K. Sawyer, Ed. Cambridge: Cambridge University Press, 2022, pp. 156–174.
[28]
J. Macina, N. Daheim, I. Hakimi, M. Kapur, I. Gurevych, and M. Sachan, “MathTutorBench: A Benchmark for Measuring Open-ended Pedagogical Capabilities of LLM Tutors,” ArXiv preprint, vol. abs/2502.18940. 2025, [Online]. Available: https://arxiv.org/abs/2502.18940.
[29]
M. Sharma, R. Pea, and H. Subramonyam, “ConvoLearn: A Dataset of Constructivist Tutor-Student Dialogue,” ArXiv preprint, vol. abs/2601.08950. 2026, [Online]. Available: https://arxiv.org/abs/2601.08950.
[30]
S. Zhou et al., “WebArena: A realistic web environment for building autonomous agents,” in The twelfth international conference on learning representations, ICLR 2024, vienna, austria, may 7-11, 2024, 2024, [Online]. Available: https://openreview.net/forum?id=oKn9c6ytLx.
[31]
T. Xie et al., “OSWorld: Benchmarking multimodal agents for open-ended tasks in real computer environments,” in Advances in neural information processing systems 38: Annual conference on neural information processing systems 2024, NeurIPS 2024, vancouver, BC, canada, december 10 - 15, 2024, 2024, [Online]. Available: http://papers.nips.cc/paper\_files/paper/2024/hash/5d413e48f84dc61244b6be550f1cd8f5-Abstract-Datasets\_and\_Benchmarks\_Track.html.
[32]
K. Q. Lin et al., “VideoGUI: A benchmark for GUI automation from instructional videos,” in Advances in neural information processing systems 38: Annual conference on neural information processing systems 2024, NeurIPS 2024, vancouver, BC, canada, december 10 - 15, 2024, 2024, [Online]. Available: http://papers.nips.cc/paper\_files/paper/2024/hash/804e757b7d7043c26701c3a313032101-Abstract-Datasets\_and\_Benchmarks\_Track.html.
[33]
L. Jang et al., “VideoWebArena: Evaluating Long Context Multimodal Agents with Video Understanding Web Tasks,” ArXiv preprint, vol. abs/2410.19100. 2024, [Online]. Available: https://arxiv.org/abs/2410.19100.
[34]
S. Z. Shen et al., arXiv:2510.25744 [cs.CL]“Completion $\neq$ Collaboration: Scaling Collaborative Effort with Agents.” arXiv, Oct. 2025, doi: 10.48550/arXiv.2510.25744.
[35]
Y.-H. Peng, D. Li, J. P. Bigham, and A. Pavel, “Morae: Proactively Pausing UI Agents for User Choices,” in Proceedings of the 38th Annual ACM Symposium on User Interface Software and Technology, 2025, pp. 1–14, doi: 10.1145/3746059.3747797.
[36]
F. Huq et al., “CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation,” ArXiv preprint, vol. abs/2501.16609. 2025, [Online]. Available: https://arxiv.org/abs/2501.16609.
[37]
M. G. Core and J. Allen, “Coding dialogs with the DAMSL annotation scheme,” in AAAI fall symposium on communicative action in humans and machines, 1997, vol. 56, pp. 28–35.
[38]
A. Stolcke et al., “Dialogue act modeling for automatic tagging and recognition of conversational speech,” Computational Linguistics, vol. 26, no. 3, pp. 339–374, 2000, [Online]. Available: https://aclanthology.org/J00-3003.
[39]
Y. Ahn, L. R. Liao, B. Bach, and N. W. Kim, “From Answer Givers to Design Mentors: Guiding LLMs with the Cognitive Apprenticeship Model,” ArXiv preprint, vol. abs/2601.19053. 2026, [Online]. Available: https://arxiv.org/abs/2601.19053.
[40]
S. Yang, S. Kwak, J. Lee, and J. Kim, “Beyond instructions: A taxonomy of information types in how-to videos,” in Proceedings of the 2023 CHI conference on human factors in computing systems, CHI 2023, hamburg, germany, april 23-28, 2023, 2023, pp. 797:1–797:21, doi: 10.1145/3544548.3581126.
[41]
J. L. Fleiss and J. Cohen, “The Equivalence of Weighted Kappa and the Intraclass Correlation Coefficient as Measures of Reliability,” Educational and Psychological Measurement, vol. 33, no. 3, pp. 613–619, 1973, doi: 10.1177/001316447303300309.
[42]
J. Cohen, “A coefficient of agreement for nominal scales,” Educational and psychological measurement, vol. 20, no. 1, pp. 37–46, 1960.
[43]
OpenAI, “Introducing GPT-5.4,” OpenAI. 2026, Accessed: May 05, 2026. [Online]. Available: https://openai.com/index/introducing-gpt-5-4/.
[44]
Google, “Gemini 3.1 Pro,” Google DeepMind. 2026, Accessed: May 08, 2026. [Online]. Available: https://deepmind.google/models/gemini/pro/.
[45]
Anthropic, Accessed: Jun 12, 2026“Claude 4.6.” 2026, [Online]. Available: https://www.anthropic.com/news/claude-sonnet-4-6.
[46]
S. Bai et al., “Qwen3-VL Technical Report,” ArXiv preprint, vol. abs/2511.21631. 2025, [Online]. Available: https://arxiv.org/abs/2511.21631.
[47]
Meta, “The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation.” 2025, Accessed: May 13, 2026. [Online]. Available: https://ai.meta.com/blog/llama-4-multimodal-intelligence/.
[48]
Y. Hua and Y. Artzi, arXiv:2408.01417 [cs.CL]“Talk Less, Interact Better: Evaluating In-context Conversational Adaptation in Multimodal LLMs.” arXiv, Aug. 2024, doi: 10.48550/arXiv.2408.01417.
[49]
D. Alihosseini, E. Montahaei, and M. Soleymani Baghshah, “Jointly measuring diversity and quality in text generation models,” in Proceedings of the workshop on methods for optimizing and evaluating neural language generation, 2019, pp. 90–98, doi: 10.18653/v1/W19-2311.
[50]
D. Friedman and A. B. Dieng, “The Vendi Score: A Diversity Evaluation Metric for Machine Learning,” ArXiv preprint, vol. abs/2210.02410. 2022, [Online]. Available: https://arxiv.org/abs/2210.02410.
[51]
Y. Guo, G. Shang, and C. Clavel, “Benchmarking Linguistic Diversity of Large Language Models,” ArXiv preprint, vol. abs/2412.10271. 2024, [Online]. Available: https://arxiv.org/abs/2412.10271.
[52]
D. M. Chan, A. Myers, S. Vijayanarasimhan, D. A. Ross, B. Seybold, and J. F. Canny, “What’s in a caption? Dataset-specific linguistic diversity and its effect on visual description models and metrics,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 4740–4749.
[53]
K. Pillutla et al., “MAUVE: Measuring the gap between neural text and human text using divergence frontiers,” in Advances in neural information processing systems 34: Annual conference on neural information processing systems 2021, NeurIPS 2021, december 6-14, 2021, virtual, 2021, pp. 4816–4828, [Online]. Available: https://proceedings.neurips.cc/paper/2021/hash/260c2432a0eecc28ce03c10dadc078a4-Abstract.html.
[54]
D. Chan, S. Petryk, J. Gonzalez, T. Darrell, and J. Canny, “CLAIR: Evaluating image captions with large language models,” in Proceedings of the 2023 conference on empirical methods in natural language processing, 2023, pp. 13638–13646, doi: 10.18653/v1/2023.emnlp-main.841.
[55]
T. Levinboim, A. V. Thapliyal, P. Sharma, and R. Soricut, “Quality estimation for image captions based on large-scale human evaluations,” in Proceedings of the 2021 conference of the north american chapter of the association for computational linguistics: Human language technologies, 2021, pp. 3157–3166, doi: 10.18653/v1/2021.naacl-main.253.
[56]
M. Heldner and J. Edlund, “Pauses, gaps and overlaps in conversations,” Journal of Phonetics, vol. 38, no. 4, pp. 555–568, 2010, doi: 10.1016/j.wocn.2010.08.002.
[57]
Z. Z. Wang, Y. Shao, O. Shaikh, D. Fried, G. Neubig, and D. Yang, “How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupations,” ArXiv preprint, vol. abs/2510.22780. 2025, [Online]. Available: https://arxiv.org/abs/2510.22780.

  1. Work done at UC Berkeley as a visiting student.↩︎

  2. Equal supervision.↩︎

  3. https://www.upwork.com/↩︎

  4. Data collection and human evaluation studies in section 6.2 and 7 are approved by our institution’s IRB.↩︎

  5. See annotation result details in Appendix 13.4.↩︎

Subjects

Updated on Academus

2026/07/01 11:28 UTC