1 Introduction↩︎

Feedback plays a vital role in student learning in writing. It helps students correct misunderstandings and refine how they apply knowledge [1], [2], and it is consistently identified as one of the strongest influences on learning and achievement [3]. But effective feedback is not one-size-fits-all. Students report that the most useful feedback is specific to their own writing [4], and effective feedback accounts for the draft’s position within the revision process [5]; poorly targeted feedback can even be detrimental [6]. In writing instruction, producing such feedback is labor-intensive, and its cost at scale discourages the sustained practice effective instruction requires [7], [8]. This opens a natural opportunity for NLP: systems that generate useful feedback on drafts could help scale writing support.

Progress on this problem is constrained by two bottlenecks. First, few public datasets preserve how instructor feedback is actually delivered in coursework (Table 1): multifaceted comments (often addressing several points at once) anchored to specific spans (a paragraph, sentence, or word) and interpretable alongside the assignment prompt, rubric, and revision history. Some resources substitute structured labels (error tags or analytic scores) for formative commentary [9]–[12], restrict coverage to a single prompt or narrow genre [13], [14], or forgo expert annotation for crowd- or model-generated feedback [15]. Without datasets that capture feedback as instructors deliver it, evaluating whether LLMs produce what an instructor would write remains out of reach.

Second, feedback evaluation is difficult for three reasons. (i) Reference-free evaluation is conceptually appealing, but ‘good’ writing feedback has no single operational definition, as its quality depends on many interacting dimensions weighted differently across raters and settings [16], [17]; LLM-as-judge proxies inherit this ambiguity while adding biases of their own [18], [19]. (ii) Reference-based evaluation mitigates this by comparing generated feedback against instructor feedback, but existing metrics fall short: n-gram overlap is unreliable for open-ended generation [20], [21], and embedding-based metrics, while less surface-bound, correlate only weakly with human judgments [22], as we also find in our setting (§4.2). (iii) A deeper, often overlooked issue is granularity: a feedback message typically bundles several distinct feedback units [23], [24] – self-contained statements about specific aspects of the writing – where even a single sentence may contain several (e.g., ‘This paragraph is well-written, but try to make it more concise’). Holistic comparison cannot tell which observations are recovered or missed, so reliable evaluation must compare feedback at the unit level and by its underlying meaning.

We address both. Sefora (Student Essays with Feedback Corpus) is a public corpus pairing authentic instructor-authored span-anchored feedback with assignment prompts, rubrics, analytic scores, and multi-draft revisions across diverse college writing genres. UniMatch is a reference-based framework that evaluates LLM feedback at the unit level rather than holistically, segmenting feedback into feedback units and aligning them via optimal matching under instructor-derived similarity criteria; the resulting similarity-weighted F\(_1\) captures both what the model omits and what it over-generates (Figure 1). Together they enable systematic study of LLM-generated writing feedback at the granularity instructors work in.

Our experiments across 74 configurations spanning multiple LLMs show that no setting exceeds \(0.4\) F\(_1\). Models struggle to produce the comments instructors would emphasize. Feedback verbosity, in the sense of producing more comments rather than longer ones, is the dominant driver of poor alignment. What UniMatch penalizes is pedagogically meaningful: overly verbose feedback overwhelms the student, leading to counterproductive results [6].

2 Related Work↩︎

6pt

2.0.0.1 Essay datasets.

Many existing writing datasets target Automatic Essay Scoring (AES), predicting numerical scores rather than formative feedback [27]–[29], or emphasize structured annotations (e.g., error tags or analytic scores) over formative text, including PERSUADE 2.0 [9], ASAP/ASAP++² [10], and NUCLE [11]. While valuable for large-scale assessment, scores rate a draft but do not tell a student how to revise it [30]; feedback generation instead requires localized, actionable, context-sensitive comments tied to the writer’s text and the assignment goals.

Efforts to collect feedback-based essay datasets have grown in recent years, yet each captures only part of the setting we target. They pair expert feedback with a narrow slice of writing: Insta-Reviewer [26] with graduate project reports, ArgRewrite V.2 [13] with a single argumentative prompt, and Exposía [14] with research exposés, while CityU [12]³ offers short tutor comments on ESL/EFL writing. Each is limited on at least one axis central to classroom feedback, lacking multiple drafts or revisions [12], [26] or covering only a single genre [13], [14], [26]. LEAF [15] scales to more essay-feedback pairs but sources them from online users and AI rather than instructors. Sefora complements these resources with authentic, instructor-authored feedback, both span-anchored and overall, at larger scale across diverse college writing genres (Table 1), paired with assignment prompts, rubrics, analytic scores, and a multi-draft revision structure, enabling analyses that prior corpora do not directly support.

2.0.0.2 Feedback evaluation.

Evaluating generated feedback is difficult, as wording and delivery can vary widely even for the same underlying issue. Reference-free evaluation raises the issues discussed previously: effective feedback has no single operational definition [16], [17], and LLM-as-judge proxies inherit this ambiguity while adding biases of their own [18], [19]. We thus focus on reference-based and human-centered evaluation, which fall into three groups [31]. Content-overlap metrics such as BLEU [32] and ROUGE [33] reward surface similarity and are brittle for open-ended feedback. Model-based metrics such as BERTScore [34], BARTScore [35], and BLEURT [36] capture general semantics better but are hard to interpret and miss whether LLM feedback identifies the same substantive points as the instructor. Human-centered evaluation is reliable but costly and hard to scale [26]. UniMatch targets both reliability and interpretability by comparing feedback as individual units and scoring their correspondence under feedback-specific similarity criteria.

3 Dataset↩︎

Sefora was collected over two semesters from a range of first-year undergraduate courses in the university’s English Department, spanning composition, ESL, narrative and creative writing, and a senior seminar. It includes assignment prompts, grading rubrics, multi-draft student essays, and two forms of instructor feedback: span-anchored (inline) comments (Figure 2) and overall assessments. The dataset is publicly available.⁴

3.0.0.1 Scope, scale, and genres.

Sefora contains 371 essays by 155 students across 34 assignments from 9 classes, comprising 564 drafts (220 single-draft, 109 two-stage, 42 three-stage) that total 8,186 paragraphs. Under the BAWE classification [25], these span four genre families: Essay, Narrative Recount, Explanation, and Empathy Writing.⁵ Instructors contributed 5,684 inline (span-anchored) annotations – highlights, sticky notes, and strikeouts, nearly all with comment – averaging 10 annotations and 147 feedback words per draft. Additionally, 437 drafts carry overall comments (121 words avg.), 295 carry analytic trait scores (6.5 per draft avg.), and 205 a holistic score, for 8,240 instructor annotations in total.

3.0.0.2 Annotation content.

Sefora preserves the original paragraph structure and aligns each annotation to the text it targets: most are paragraph-level and span-anchored – tied to the exact span the instructor marked, whether a phrase, sentence, or word – with document-level comments and scores retained when present (details in §9.2). What distinguishes Sefora from error-tag and score-only corpora is the substance of its feedback: it is overwhelmingly facilitative (inviting the writer’s own reflection) rather than directive (prescribing changes) [37]. Rather than flagging surface errors, instructors engage substantive aspects of writing – narrative development, idea elaboration, reader engagement, voice, revision guidance – and frequently pose open questions back to the writer.

3.0.0.3 Collection, Privacy, and Ethics.

Data collection was approved by the university’s IRB. The research team contacted instructors, who emailed students a recruitment notice outlining the study, its objectives, risks, and voluntary participation. Materials were gathered from instructors and consenting students across two semesters. All IRB-related materials are provided in §9.1. All essays were released with student consent and processed under strict privacy constraints. We remove direct identifiers (student and instructor names, course identifiers, submission dates) and replace names within essay bodies with consistent within-essay pseudonyms while preserving grammatical and narrative coherence; references to public figures are retained. We further screen the corpus for indirect identifying information (e.g., social media or sandbox handles), and check for abusive or harmful content using omni-moderation⁶ exclusively via the API under OpenAI’s data-handling policy.⁷ We do not mask vulgar or strong language, as doing so would alter the writer’s voice and distort the feedback context; the dataset is released with appropriate content warnings.

3.0.0.4 Parsing.

Original submissions in PDF or .docx were converted to a JSON format that preserves both paragraph segmentation and annotation anchoring. We release the deterministic parser⁸ to support future work on similar materials. Full parsing details are in §9.3.

4 Evaluation Framework↩︎

Beyond a dataset, scaling instructor-quality feedback requires a way to evaluate it; both reference-free and reference-based approaches face challenges. We introduce UniMatch, a reference-based framework that compares model and instructor feedback at the level of feedback units rather than as monolithic texts. Instructor feedback serves as a principled reference for two reasons. First, it encodes pedagogical priorities – what to flag, what to praise, and how to phrase guidance – per curriculum rubrics and teaching goals. Second, good writing feedback has no single operational definition [16], [17]; instructor judgment, while not the only valid target, is the most defensible anchor for a given draft, especially for the open-ended writing in Sefora where LLM feedback remains far from human quality [38], [39].

UniMatch compares model and instructor feedback in three stages (Figure 1): (i) segment each into feedback units, (ii) score semantic correspondence between instructor-model unit pairs, and (iii) apply maximum bipartite matching over those scores to derive interpretable unit-level alignments and aggregate precision, recall, and F\(_1\). Stages (i) and (ii) are independently verifiable against human annotation, enabling robustness checks on the pipeline.

4.1 Feedback Segmentation↩︎

Prior work has typically treated feedback as a monolithic message rather than a set of separable units [40], [41], but holistic comparison cannot tell which individual observations are recovered or missed. Other work has recognized the importance of feedback units, though with varying definitions: one or more sentences expressing a coherent thought [23], or a self-contained message targeting a specific issue [24]. Building on these [23], [24], we adopt feedback unit – a self-contained statement addressing one specific aspect of the student’s writing – as the basic evaluation granularity, and develop an annotation guideline (§10.1) for segmenting instructor and LLM-generated feedback (examples in Table 3).

This task is related to discourse parsing [42]–[44], but differs in two key ways: we omit inter-segment relations, and place emphasis on sub-sentence segmentation, as a single sentence may contain multiple feedback units, while a single unit may span multiple sentences. We therefore draw on prior work on sub-sentence segmentation in discourse [45]–[48].

4.1.0.1 Formalization and agreement metrics.

Let \(T=t_1,\dots,t_n\) be the sequence of words of a feedback message. A segmentation is represented by a boundary vector \(Q\in\{0,1\}^{n-1}\) where \(Q_i=1\) marks a boundary between \(t_i\) and \(t_{i+1}\). We measure segmentation agreement using WindowDiff [49], a standard discourse segmentation metric, with window size \(k\) set to half the average reference segment length (\(k{=}10\) in our samples): \[WD(r,h)=\frac{1}{n-k}\sum_{i=1}^{n-k} \mathbf{1}\!\left[b(r_{i:i+k}) \neq b(h_{i:i+k})\right],\] where \(r\) and \(h\) are the two segmentations being compared, one designated the reference and the other the hypothesis (the second annotator, or the model when validating automation), and \(b(x_{i:j})\) is the boundary count between \(t_i\) and \(t_j\) in segmentation \(x\). Because WindowDiff tolerates small boundary shifts, we also report boundary precision, recall, and F\(_1\) calculated over boundary indices while excluding true negatives to avoid inflated agreement.

4.1.0.2 Automatic segmentation validation.

Following the segmentation guideline (§10.1), two graduate students independently annotated more than 50 feedback instances, containing both instructor and LLM feedback. Agreement was high: WD =\(0.073\), precision =\(0.99\), recall =\(0.94\), F\(_1\)=\(0.96\). The annotators then resolved disagreements to produce a consensus segmentation, which we treat as the reference.

Using the same segmentation guideline given to human annotators as prompt, GPT-5-nano reaches comparable agreement against this consensus reference (WD =\(0.036\), precision =\(1.00\), recall =\(0.95\), F\(_1\)=\(0.97\)), indistinguishable from inter-annotator agreement.

4.2 Feedback Unit Similarity↩︎

We then compare the similarity between feedback units. This task is related to Semantic Textual Similarity (STS) [50], [51], but differs in two key ways: the texts being compared are feedback units rather than arbitrary sentences, and both address the same paragraph of the same essay.

We thus introduce a task-specific guideline (§10.2) for scoring feedback unit similarity along two dimensions of feedback: the target (the aspect, span, or idea addressed) and the comment (the evaluation, critique, or suggestion about that target). Drawing on integer-scale scoring practice in STS [52]–[54], we use a 0–4 scale chosen to suit the feedback setting: anchored at 4 (near-equivalent as feedback) and 0 (irrelevant or contradictory), with an explicit middle value (2) for pairs that share an important point but differ on another (intermediate levels and details in §10.2). The guideline was developed in collaboration with course instructors and refined iteratively by two graduate student annotators, who piloted it, discussed disagreements, and refined wording and examples until agreement stabilized. They then independently scored 120 feedback unit pairs spanning the full scale; we report inter-annotator Pearson and Spearman correlations of \(r=0.7572\) and \(\rho=0.7558\). We use the mean as the reference similarity for these 120 pairs.

4.2.0.1 Automatic similarity scorer validation.

Existing similarity methods fall short on this task: lexical-overlap metrics yield Pearson correlations below \(0.1\) and embedding-based methods reach at most \(r\approx 0.67\), both too low to be reliable (details in §11.1). We therefore experimented with several LLMs using the defined guideline as prompt, with several exceeding \(r=0.7\) and the strongest exceeding \(r=0.8\). However, as scoring every pair independently at our scale would be expensive, we introduce batching (details in §11.2), which preserves performance at a fraction of the cost (Table 2⁹). We adopt gemini-3.1-flash-lite-preview as the similarity scorer in our main pipeline (\(r=0.813\) and \(\rho=0.818\) against the human reference). Figure 4 visualizes its agreement on a sample with the pipeline’s natural score distribution.

4.3 Feedback Unit Cross Matching↩︎

Given pairwise similarity scores between instructor and model-generated feedback units, UniMatch computes a one-to-one alignment between the two sets. Intuitively, each feedback unit is matched to its most similar counterpart on the other side, subject to a globally optimal overall matching (Figure 5 illustrates). Specifically, for instructor feedback units \(G=\{G_i\}_{i=1}^{|G|}\) and predicted feedback units \(P=\{P_j\}_{j=1}^{|P|}\), let \(\phi(G_i, P_j) \ge 0\) denote the similarity scorer. We define the optimal alignment as the maximum-weight bipartite matching \[M^*=\arg\max_{M\in\mathcal{M}} \sum_{(G_i,P_j)\in M}\phi(G_i,P_j),\] where \(\mathcal{M}\) is the set of 1-to-1 matchings between \(G\) and \(P\). Computing \(M^*\) is a standard maximum-weight bipartite matching problem, and we use the Hungarian algorithm [55] for efficiency. Let \(\Phi(M^*)=\sum_{(G_i,P_j)\in M^*}\phi(G_i,P_j)\) denote the total weight of this alignment.

To turn \(\Phi(M^*)\) into interpretable precision and recall, we normalize it against per-side upper bounds, each set’s summed within-set self-similarity (\(\sum_j \phi(P_j,P_j)\) predicted, \(\sum_i \phi(G_i,G_i)\) instructor), which equals its unit count times the maximum score (4) since each unit matches itself. Following coreference resolution [56], we define soft precision and recall as \[p=\frac{\Phi(M^*)}{\sum_j \phi(P_j,P_j)}, \qquad r=\frac{\Phi(M^*)}{\sum_i \phi(G_i,G_i)},\] and the soft F\(_1\) as \[F_1=\frac{2pr}{p+r}.\] The result is an interpretable, soft precision/recall view of feedback quality: precision captures how much of the model’s feedback aligns with what an instructor wrote, and recall captures how much of the instructor’s feedback the model recovers.¹⁰

4.3.0.1 End-to-end validation.

UniMatch’s components – segmentation (§4.1) and similarity scoring (§4.2) – are validated against human judgment in isolation, but this does not guarantee that the composed metric aligns with instructor judgment. To test the pipeline end to end, we selected 20 paragraphs, each with three raw LLM feedback texts (§5) that UniMatch scores at well-separated levels (F\(_1\approx 0\), \(0.5\), \(1\)), and had a writing expert independently rank the three by quality against the reference feedback.¹¹ Across all 60 texts, UniMatch F\(_1\) correlates strongly with the instructor ranking (Pearson \(r=0.836\), Spearman \(\rho=0.837\)), showing that the full pipeline recovers expert judgments.

5 Experiments↩︎

We evaluate paragraph-level LLM-generated feedback under UniMatch, varying model choice and prompting setup to study what task framing and generation choices drive feedback quality.

5.0.0.1 Task framing and evaluation data.

We generate feedback at the paragraph level, matching the granularity of inline instructor annotations in Sefora. For each target paragraph, the model receives the paragraph plus as much surrounding context as fits within a fixed 8,192-token budget, enough to include the entire essay for most drafts. We evaluate on the 3,499 paragraphs in Sefora with at least one instructor annotation (full-corpus); of these, 2,324 carry exactly one (the single-reference subset). We report metrics on both.

5.0.0.2 Models.

To support broad ablations across settings, we primarily use small open-weight instruction-tuned models, namely, Llama-3.1-8B [57] (Llama), DeepSeek-R1-Distill-Llama-8B [58] (DeepSeek), and Mistral-7B [59] (Mistral). To probe whether the effects extend to a larger regime, we also evaluate Qwen2.5-72B-Instruct [60] (Qwen) and closed model GPT-5.1¹² (GPT) on a subset of configurations.

5.0.0.3 Prompt template.

We use two prompt templates. The default constrained prompt asks the model to identify the single most important focus in the target paragraph and produce one feedback unit under a prescribed output format; the unconstrained prompt places no limit on the number of feedback units it may return. On top of the template, we vary four further dimensions of task specification: zero-shot vs.few-shot prompting (5 exemplars) [61]; guided vs.unguided prompting, where the guided setting supplies a list of feedback categories [62], [63] in the prompt; inclusion vs.omission of the assignment prompt and rubric; and chain-of-thought prompting [64], [65]. The guided categories cover the common functional roles of writing feedback: Task Constraints, Concepts, Elaboration, Clarification, Mistakes, and Praise. Few-shot exemplars are 5 instructor-authored feedback samples covering each category. Full prompt templates and guided variants are in §14.

5.0.0.4 Generation settings.

We allocate up to 250 output tokens per target paragraph – well above the typical length of instructor feedback on a single paragraph in Sefora (\(\approx20\) tokens¹³) – to avoid truncating multi-point responses.

In total, the model and prompt combination yields 74 experimental configurations. All prompts (§14), inference pipelines, the UniMatch framework, and code are publicly available.¹⁴

6 Results↩︎

Table 3 reports UniMatch scores on the full corpus under the zero-shot setting (with rubrics and category guidance), where most models perform best; full results are in Tables 4–6 of §13. No configuration exceeds \(0.4\) F\(_1\), and both precision and recall stay low: even the strongest GPT model reaches only \(0.38\) precision and \(0.27\) recall, so models miss most of the feedback instructors prioritize while spending much of their output on points instructors do not raise. Among the small models (\(\le8\)B parameters), Mistral is consistently strongest, ahead of Llama¹⁵. The larger Qwen and GPT lead overall – GPT on the shared subset, Qwen on the single best configuration (\(0.32\) F\(_1\)) – but only narrowly, and all remain far from matching instructor feedback.

Table 3 additionally presents ablations: each varies a single variable relative to the base setting <zero-shot, rubric, guided, default prompt>. We show a subset due to space constraints, but the pattern matches the full results. On the single-reference subset (fourth column), scores are higher as expected, since our constrained default prompt emits one feedback unit per paragraph and thus naturally aligns more with single-reference paragraphs.

6.0.0.1 Verbosity.

The dominant signal in our experiments is verbosity. Across all models, the constrained prompt (our default) beats the unconstrained alternative, which yields substantially lower F\(_1\) (the unconstrained column of Table 3). Removing the single-feedback constraint lets models emit more than one feedback unit per paragraph, and the resulting over-generation sharply lowers precision. This holds even on the full corpus, where paragraphs often carry multiple instructor units and the unconstrained prompt should in principle have the edge: its extra units add low-similarity matches that depress precision more than they lift recall. Figure 6 confirms the pattern across all evaluated configurations – the number of generated feedback units is strongly negatively correlated with both precision and F\(_1\) (\(r=-0.850\) and \(r=-0.840\), respectively), while the accompanying recall gains are too modest to compensate (full results in §13).

6.0.0.2 Guiding the model with feedback categories.

Supplying explicit feedback categories (the guided setting) is the next strongest factor: it improves F\(_1\) by about \(+0.059\) without rubric and \(+0.035\) with rubric on the full-corpus evaluation in matched comparisons (the Unguided column of Table 3; per-configuration numbers in Tables 4–6, and a per-model breakdown in Figure 15, §13).

6.0.0.3 Rubric and few-shot prompting.

Two further levers contribute little (Table 3, No rubrics and Few-shot columns). Rubric inclusion is essentially neutral, with small, unstable differences: in our paragraph-level setting well-framed prompts already supply most of the structure a rubric would contribute, and rubrics largely encode whether a draft is on-task – which, in authentic coursework, it usually is, leaving little for a rubric to anchor given that Sefora’s feedback is facilitative (§3), engaging execution rather than rubric conformance. Few-shot prompting is beneficial but secondary, improving F\(_1\) by about \(+0.017\) on the full corpus, mainly through precision; its gains are small under the default prompt, as exemplars help most when task framing is weak, and once the prompt already induces the desired behavior, demonstrations add little. Larger few-shot gains under the unconstrained alternative are in Table 5 (§13); a small chain-of-thought study on the three small models follows the same pattern (§13). Per-model breakdowns of both effects appear in Figures 16 and [fig:effect95of95fewshot] (§13).

7 Conclusion↩︎

We introduce Sefora, a corpus of authentic instructor feedback on student writing, and UniMatch, a reference-based framework for evaluating generated feedback. Across 74 configurations, no setting exceeds \(0.4\) F\(_1\): the central challenge for LLMs is not producing feedback but producing the comments an instructor would prioritize. Over-generation is consistently costly, with precision dropping sharply as models generate more while recall rarely compensates, a pattern that is pedagogically meaningful, since overwhelming a student with feedback is itself counterproductive [6]. Validated end-to-end against expert judgment, UniMatch provides a reliable automatic metric for this task, and together with Sefora lays a foundation for future work on instructor-aligned feedback generation across diverse college writing genres.

8 Limitations↩︎

Although UniMatch enables fine-grained analysis, its evaluation operates on feedback units as segmented, without further decomposing compound statements. Some units express multiple claims (e.g., ‘this paragraph is detailed and smooth’) and could in principle be split further. Future work could incorporate automatic claim decomposition to isolate individual feedback propositions at the atomic level.

Our evaluation operates at the paragraph level: the model is given a target paragraph, and UniMatch assesses the feedback it produces rather than whether the model identifies which span warrants comment in the first place. Sefora’s span-anchored annotations make span identification directly studiable, but we leave this setting to future work. Our experiments also evaluate against the inline (span-anchored) feedback only, leaving the document-level overall assessments and analytic scores in Sefora for future work.

Our alignment uses maximum-weight bipartite matching over all eligible unit pairs. Alternative alignment constraints (e.g., prioritizing globally highest-scoring pairs before enforcing one-to-one matching) may produce different match structures and could be explored as alternative objectives.

Our pipeline relies on closed-source models (GPT-5-nano for segmentation and gemini-3.1-flash-lite-preview for similarity scoring), chosen because they meet the human-agreement bar required for reliable evaluation in our setting. This reliance has practical downsides: hosted models can change or be deprecated over time, and their use introduces cost and reproducibility constraints. A natural next step is to distill or fine-tune smaller open-source models for both the segmentation and similarity stages, using our human-annotated subsets as supervision. Because each stage is independently validated against human annotation (§4.1, §4.2), open-source replacements can be substituted into the pipeline whenever they reach comparable agreement, without re-validating UniMatch as a whole.

UniMatch treats instructor feedback as ground truth. Reference-free evaluation may also be possible by leveraging methods from the automated essay scoring (AES) literature. For example, generated feedback could be applied to the relevant essay span and the resulting revision evaluated with a quality scorer to estimate feedback effectiveness. Developing such metrics is beyond the scope of this work.

Finally, our study relies primarily on automatic evaluation under the proposed framework and does not include a complementary human evaluation of overall feedback quality. While UniMatch is designed to better reflect semantic, point-level alignment with instructor feedback than whole-text overlap metrics, it does not directly assess dimensions such as helpfulness, pedagogical appropriateness, or actionability as perceived by human readers. A targeted human study, or an LLM-as-judge evaluation calibrated against human judgments, would provide a useful complementary perspective. We leave such validation to future work.

9 Dataset↩︎

9.1 Data Collection Materials↩︎

The study was approved by the university’s IRB (§3.0.0.3). The recruitment materials distributed to students and instructors are reproduced below.

9.2 Annotation Representation Details↩︎

Paragraph-level annotations are stored within the corresponding paragraph, while document-level comments and scores are stored outside the paragraph structure.

Sticky notes are stored with their text and exact positions within the paragraph body, reflecting the original placement. Highlights are stored with their exact spans and associated comments; in our data, nearly all highlights contain instructor-written comment text. For all paragraph-level annotations we also retain the immediate left and right context of the marked span to facilitate locating the annotation within the paragraph, especially when the highlighted text is short or when a sticky note is tied to a specific local position in the essay. When instructors use non-generic highlight colors, we preserve these as structured signals. In this dataset, color usage follows a consistent instructor-defined legend:

• Pink: “great job, this is an awesome line”

• Green: “there is something really interesting here …expand on this point in your next draft”

• Yellow: “there is some problem with this sentence”

9.3 Parsing Details↩︎

A central preprocessing challenge is preserving both paragraph structure and annotation anchoring across annotated PDF and .docx submissions. For .docx, we extract standard highlights and comments with their anchored spans. For PDFs, however, annotations must be reconstructed from page-level geometry, including highlight quadrilaterals and note positions, rather than from a linear text stream [66].

Naive PDF text extraction can interleave headers and footers and scramble paragraph boundaries, which are often conveyed by layout rather than explicit markup. We also explored LLM-based re-paragraphization of extracted text, but observed deviations from the source text, including missing or added tokens and altered boundaries, consistent with known hallucination behavior [67]; it also incurs nontrivial computational cost [68], [69]. We therefore implemented a deterministic parser that converts annotated PDF and .docx submissions into a unified JSON format while preserving paragraph segmentation and annotation anchoring. We release this parser with the dataset. The remainder of this section describes the PDF pipeline, which is the more involved of the two; the stages are summarized in Figure 10.

9.3.0.1 Body text extraction.

The parser first extracts the essay body from the page-level text blocks. This stage is fully deterministic: every character present in the source text blocks is retained, and no content is dropped, rewritten, or reordered by a model. The extracted body text serves as the canonical reference string to which all subsequent annotations are anchored.

9.3.0.2 Highlight anchoring.

Highlights are recovered from their page-level geometry rather than from a linear text stream. For each highlight, the parser collects the highlight quadrilaterals, determines the text covered by those quads, and matches the covered text back to the corresponding span in the extracted body, anchoring the highlight to that span. The associated highlight comment, when present, is stored alongside the anchored span. Because nearly all highlights in Sefora carry instructor-written comment text (§9.2), this step preserves both the marked span and its commentary.

9.3.0.3 Highlight color recovery.

Highlight color is not reliably available as a structured attribute, so the parser recovers it visually: it performs a localized rendering of the highlighted region, reads the rendered color, and maps the resulting HEX value to the nearest entry in our set of named highlight colors. This yields the color names used as structured signals in the corpus, following the instructor-defined legend documented in §9.2 (e.g., pink, green, and yellow), as illustrated in Figure 2.

9.3.0.4 Sticky-note anchoring.

Sticky notes are positioned by page coordinates rather than attached to a text span, so the parser anchors them by location. For each note, it identifies the surrounding text within a coordinate radius of the note’s placement, locates that surrounding text within the extracted body, and injects the note at the corresponding position. This reproduces the original placement of the note within the paragraph body (§9.2).

9.3.0.5 Paragraph structure.

Text blocks do not necessarily correspond to paragraphs, so paragraph boundaries cannot be read directly from the block structure. Instead, the parser infers paragraph structure from visual layout cues, primarily the spacing between text blocks, recovering the paragraph segmentation that layout conveys without explicit markup.

9.3.0.6 Header, footer, and identifier removal.

Recurring page elements such as top-of-page names and page numbers are removed based on their position on the page. Submission metadata such as the submission date is detected using regular-expression patterns. To ensure that student and instructor names do not leak into the released corpus, positional and regex-based removal was complemented by additional scanning tools and human inspection, consistent with the privacy procedures described in §3.0.0.3.

10 Guidelines↩︎

10.1 Feedback Segmentation Annotation Guideline↩︎

The guideline used by annotators to segment instructor and LLM-generated feedback into feedback units (§4.1) is shown in Figure 11; segmented examples appear in Table 3.

10.2 Feedback Unit Similarity Annotation Guideline↩︎

The guideline used to score similarity between feedback-unit pairs on the 0–4 scale (§4.2) is shown in Figures 12–14.

11 Feedback Unit Similarity Pipeline↩︎

11.1 Automatic similarity metrics.↩︎

In our experiment and setting, lexical-overlap measures such as BLEU [32] and chrF [70] yield Pearson correlations below \(0.1\) with annotator scores. Embedding-based approaches (MiniLM [71], MPNet [72], and BERTScore [34]) perform better but still show low correlation with human scores, with Pearson correlations below \(0.3\). Cosine similarity using OpenAI’s embedding model text-embedding-3-large¹⁶ is the strongest automatic baseline, but its performance depends heavily on the context provided to the embedding model. Our best result (\(r=0.6696\), \(\rho=0.6343\)) is obtained by prefixing each feedback with “Semantic content of instructor feedback on a student-written paragraph: ” before embedding. Even so, it remains insufficiently aligned with human judgments for reliable evaluation.

11.2 Batch Similarity Scoring Details↩︎

11.2.0.1 Batched requests.

Since the similarity guideline is long, scoring every pair independently for each paragraph, draft, and experimental setting would be expensive and inefficient. We therefore score multiple pairs per request (batch_size\(=b\)), instructing the model to evaluate each pair independently (implementation details in §11.2). Batching reduces the cost in inverse proportion, but since it alters the prompting setting, we reverify the agreement with the human reference. Correlation results for multiple models under the batching setup are reported in Table 2 (§4.2), where we adopt gemini-3.1-flash-lite-preview for the main pipeline on the strength of its agreement with human scores.

11.2.0.2 Batching strategy.

The similarity guideline used for scoring feedback-unit pairs is nearly 2K tokens long (although the exact token count depends on the tokenizer), making repeated independent calls inefficient. To reduce token cost, we evaluate multiple pairs in a single request (batch_size\(=b\)), instructing the model to assign one score to each labeled pair independently.

11.2.0.3 Token cost.

Let \(A\) denote the irreducible per-request token cost (system instructions, formatting, and response overhead), and \(B\) the guideline length. When scoring \(b\) pairs per request, the total token cost per pair can be approximated as \[T(b) = A + \frac{B}{b},\] so batching reduces the amortized guideline overhead by a factor of \(O(1/b)\).

11.2.0.4 Effect on agreement.

Batching slightly changes the prompting context because the model evaluates multiple pairs within the same request. We therefore verify that this does not materially affect similarity scoring. Experiments with batch_size\(=50\) show agreement with the human reference comparable to the single-pair setting reported in §4.2, indicating that batching does not meaningfully alter model judgments.

11.2.0.5 Implementation notes.

We avoided the Batch API (i.e., offline requests used to reduce API cost) in order to preserve the ability to retry malformed outputs. In auxiliary runs with gpt-5-mini-2025-08-07, which was not the primary model used in our main evaluation, we did not observe any syntactically invalid outputs. gemini-3.1-flash-lite-preview, which was used at much larger scale, produced a small number of syntactically invalid outputs, and in those cases, maximum of two retries were sufficient to obtain a syntactically valid output.

12 Threshold-Based Cross Matching↩︎

UniMatch’s default cross-matching stage (§4.3) aggregates similarity scores into soft precision and recall by treating each unit’s full self-similarity as its maximum possible contribution. We complement this with a threshold-based variant that mirrors detection-style evaluation in computer vision [73]: given a threshold \(\tau\) on the annotator-aligned 0–4 similarity scale, a pair counts as a match only if its similarity is at least \(\tau\), and the standard hard precision, recall, and F\(_1\) follow.

12.0.0.1 Formalization.

Let \(G=\{G_i\}_{i=1}^{|G|}\) and \(P=\{P_j\}_{j=1}^{|P|}\) denote instructor and predicted feedback units for a single paragraph, with similarity scorer \(\phi(G_i,P_j)\in\{0,1,2,3,4\}\). For \(\tau\in\{1,2,3,4\}\), define the binarized adjacency \(a_{ij}(\tau)=\mathbf{1}[\phi(G_i,P_j)\ge\tau]\) and compute the maximum bipartite matching on \(a\): \[M^{*}_{\tau}=\arg\max_{M\in\mathcal{M}} \sum_{(G_i,P_j)\in M} a_{ij}(\tau),\] where \(\mathcal{M}\) is the set of 1-to-1 matchings between \(G\) and \(P\). The match count and complementary error counts are \[\begin{align} TP_{\tau} &= |M^{*}_{\tau}|, \\ FP_{\tau} &= |P| - TP_{\tau}, \\ FN_{\tau} &= |G| - TP_{\tau}, \end{align}\] yielding the hard metrics \[p_{\tau}=\frac{TP_{\tau}}{|P|},\quad r_{\tau}=\frac{TP_{\tau}}{|G|},\quad F_{1,\tau}=\frac{2\,p_{\tau}\,r_{\tau}}{p_{\tau}+r_{\tau}}.\]

12.0.0.2 Interpretation of \(\tau\).

Because the similarity scale is grounded in the annotation guideline (§10.2), \(\tau\) has a direct semantic reading: \(\tau{=}1\) requires the pair to share at least a general feedback category, \(\tau{=}2\) requires sharing an important point (same targeted aspect or same comment), \(\tau{=}3\) requires that the units differ only in a minor detail, and \(\tau{=}4\) requires near-equivalence as feedback. The threshold thus exposes how strict an “alignment” must be to count as feedback recovery.

12.0.0.3 Sensitivity to over-generation.

In the soft variant, each additional predicted unit contributes its full self-similarity \(\phi_{\max}\) to the precision denominator \(\sum_j\phi(P_j,P_j)\), so a weak-but-nonzero match still inflates the denominator without comparably increasing the numerator. The threshold-based variant attenuates this: predictions affect precision only through their count, and any pair above \(\tau\) receives full credit. Weak-but-above-threshold matches are therefore not partially discounted, and the per-extra-unit cost is smaller. The variant offers a less stringent view when over-generation is unavoidable; the soft variant remains preferred when fine-grained similarity differences carry signal.

12.0.0.4 Ranking robustness across the 74 configurations.

Despite the qualitative difference in metric form, the relative ordering of the 74 configurations from §5 is preserved under thresholding: for every \(\tau\in\{1,2,3,4\}\), the Spearman rank correlation between configuration F\(_1\) under the soft variant and \(F_{1,\tau}\) is \(\rho=1.0\). The dominant findings reported in §6 – the negative effect of verbosity, the gains from feedback-category guidance, the limited contribution of rubric inclusion, and the secondary role of few-shot prompting – therefore do not depend on the choice of matching metric. We omit per-configuration \(F_{1,\tau}\) tables, as the rank-identity result subsumes them; the implementation is released alongside the soft variant for reproduction.

12.0.0.5 Implementation.

Threshold-based matching reuses the same similarity matrices produced for the soft variant; no additional model calls are required. We binarize the matrix at \(\tau\) and run a bipartite matching solver on the resulting graph. Note that the Hungarian algorithm handles the general case of weighted edges (similarity scores), whereas in this setting edges are unweighted.

13 Full Evaluation Results↩︎

This appendix presents per-configuration UniMatch scores for all 74 settings described in §5. Tables 4, 5, and 6 report precision, recall, F\(_1\), and number of generated feedback units under zero-shot, few-shot, and chain-of-thought (CoT) prompting, respectively. Each table reports both the single reference evaluation (paragraphs with exactly one instructor feedback unit) and the full corpus evaluation (every paragraph with at least one instructor annotation); the two regimes track each other closely on every comparison reported in §6, and we therefore discuss only the full-corpus numbers throughout the main text. Figure 17 summarizes the F\(_1\) effect of the prompt template across representative configurations.

13.0.0.1 Per-condition ablation visualizations.

Figures 15, 16, and [fig:effect95of95fewshot] provide per-model, per-condition visualizations of the guidance, rubric, and few-shot effects summarized in Table 3 (§6).

13.0.0.2 Prompt template.

The default constrained prompt (\(V_2\), Figure 19) substantially outperforms the unconstrained alternative (\(V_1\), Figure 18) across models, conditions, and shot settings (Figure 17). Across 28 matched pairs (same model, guidance, rubric, and shot), \(V_2\) yields a mean F\(_1\) gain of \(+0.090\) over \(V_1\), and wins in 27 of 28 pairs (96%). The mechanism is verbosity: \(V_1\) averages roughly 12K generated feedback units per configuration vs.4.6K for \(V_2\), and the resulting precision deficit overwhelms any recall benefit. This is consistent with the broader verbosity finding in §6: the constrained prompt enforces a per-paragraph output budget that the unconstrained prompt does not.

13.0.0.3 Few-shot prompting under \(V_1\).

Few-shot exemplars contribute much more under the unconstrained prompt than under the constrained one. In matched comparisons, few-shot improves F\(_1\) by \(+0.028\) under \(V_1\) but only \(+0.005\) under \(V_2\) (Table 5). This supports the interpretation in §6 that exemplars are most useful when task framing is weakest: \(V_2\) already specifies the target output structure, leaving little additional ground for exemplars to cover. The few-shot gain is precision-driven across all 30 matched (zero, few) pairs: precision rises by \(+0.028\) on average while recall falls slightly (\(-0.004\)).

13.0.0.4 Chain-of-thought.

CoT prompting (Table 6) was evaluated on the three small models in the zero-shot setting, using the unconstrained \(V_1\) prompt. Under matched conditions, CoT improves F\(_1\) by \(+0.041\) over \(V_1\) but underperforms \(V_2\) by \(-0.056\). CoT settings produce 6.6K–8.4K generated feedback units per configuration, between \(V_1\) and \(V_2\), and the qualitative patterns from the main results hold: Mistral is the strongest small model, feedback-category guidance contributes a small positive effect, and rubric inclusion is essentially neutral. CoT does not alter the overall story; like other settings, its position in the F\(_1\) ordering is largely determined by how many feedback units it generates.

14 Prompt Templates↩︎

We use two prompt templates, both with two toggles (Figures 18, 19): Guided? controls whether the feedback categories (§5) are listed, and Rubric? whether the assignment prompt and rubric are inserted via {rubric}. The unconstrained template (\(V_1\), Figure 18) places no limit on the number of feedback units; the constrained template (\(V_2\), Figure 19) asks for a single most-important point under a fixed output format. \(V_2\) is the default in the main experiments (§5); \(V_1\) is used only in the prompt-design comparison (§13).

References↩︎

---

1. https://github.com/ShayanPey/SEFORA↩︎

2. https://www.kaggle.com/c/asap-aes↩︎

3. The original paper described the corpus as not yet publicly available at the time, and we could not verify a current public release.↩︎

4. https://github.com/ShayanPey/SEFORA↩︎

5. For each assignment we map its writing prompt to the genre family it most closely matches; BAWE’s explicit definitions and examples make this a direct mapping.↩︎

6. https://developers.openai.com/api/docs/models/omni-moderation-latest↩︎

7. https://developers.openai.com/api/docs/guides/your-data/↩︎

8. https://github.com/ShayanPey/SEFORA↩︎

9. Model performance can shift over time; results were collected in May 2026. Result using the snapshot model gpt-5-mini-2025-08-07 is also reported for stability.↩︎

10. The pipeline also admits a threshold-based instantiation with hard matches above a chosen \(\tau\), exposing strictness as an explicit knob (§12).↩︎

11. Ties were permitted but occurred for only two paragraphs.↩︎

12. https://developers.openai.com/api/docs/models/gpt-5.1↩︎

13. Exact number of tokens depends on the tokenizer.↩︎

14. https://github.com/ShayanPey/SEFORA↩︎

15. Sometimes DeepSeek as shown in §13.↩︎

16. https://developers.openai.com/api/docs/models/text-embedding-3-large↩︎