1 Introduction↩︎

Mental health conditions contribute substantially to global health challenges, partly because their symptoms are often subjective, overlapping, context-dependent, and difficult to define consistently [1]. Major depressive disorder (MDD) is especially prominent due to its prevalence, functional impact, and heterogeneous presentation [2]. Rather than reflecting a single marker, MDD involves affective, cognitive, and somatic experiences, including low mood, loss of interest, irritability, hopelessness, concentration difficulties, rumination, fatigue, sleep and appetite changes, psychomotor changes, and reduced energy [3]. These experiences are documented across varied sources such as diaries, ecological momentary assessments, questionnaires, patient narratives, electronic health records, clinician notes, and informal digital communication, creating challenges for consistent interpretation even before computational modeling is introduced [4]–[6]. Clinical practice addresses this complexity through standardized diagnostic frameworks and validated assessment tools. The Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) established a standardized diagnostic framework for classifying mental disorders using symptom-based criteria, while its Text Revision (DSM-5-TR) updates descriptive text, terminology, coding guidance, and selected clarifications [7]–[9]. Structured interviews such as the SCID-5 support criterion-level expert assessment [10], and the PHQ-9 offers a brief validated measure for depression screening and severity [11]. These tools provide strong clinical grounding and promote consistency, but they are designed primarily for clinical assessment rather than fine-grained annotation of free text, such as evidence links, criterion-level rationales, or structured decision traces for computational reuse [12].

In parallel, artificial intelligence (AI) methods have increasingly been applied to depression-related language, particularly through natural language processing (NLP) approaches for unstructured text [13]. Their primary advantage lies in scalability: they can process large volumes of narrative data and identify patterns that may be difficult to track manually across long or heterogeneous records. However, in psychiatric applications, predictive output alone is insufficient. Clinicians, annotators, and downstream model developers must also be able to inspect the textual evidence supporting a decision, understand how that evidence relates to recognized clinical constructs, and determine whether the resulting judgment is clinically defensible. For this reason, explainable AI (XAI) is especially important in mental health settings, where transparency and interpretability are closely linked to trust, accountability, and practical usability [14], [15]. These considerations suggest that the most clinically useful direction is not to replace established assessment practices with opaque automation, but to integrate DSM-5/DSM-5-TR-guided clinical structure, AI-assisted scalability, and XAI-oriented transparency within a single evidence-aware annotation framework.

In this paper, we propose an Adaptive DSM-5-TR Standardized Annotation Framework for MDD prediction. We focus specifically on MDD because of its well-defined DSM-5-TR criteria, widespread clinical assessment tools, and rich expression in narrative text. The framework is designed to support multi-label annotation through DSM-5-TR-aligned label standardization, adaptive suggestion generation, and expert-in-the-loop revision. In addition to producing structured annotations, the framework enables the analysis of system behavior through expert edits and evidence-aware review. We evaluate the framework in a pilot study with expert-reviewed samples, focusing on both system label quality and the amount of manual correction required.

Our contributions are summarized as follows:

• A Human-Centered Framework for Explainable DSM-5-TR Annotation: We propose a collaborative framework that transforms depression screening into a transparent, criterion-level process, generating structured evidence, clinical rationales, and key phrase highlighting to ensure high-fidelity explainable datasets.

• A Self-Evolving Mechanism via Dual-Memory Integration: We introduce a self-evolving architecture utilizing Example and Reflection Memories. This allows the agent to internalize expert feedback and “lessons learned” to iteratively improve diagnostic alignment without retraining the model.

• A Pilot Evaluation of Annotation Quality and Expert Effort: In a pilot study on 10 cases reviewed against gold annotations from five experts, the framework achieves high agreement with expert labels while substantially reducing manual annotation effort and time and supporting expert control through interactive conflict detection.

2 Related Work↩︎

2.1 Explainability in Mental Health Prediction↩︎

Explainability is a core requirement in mental health prediction because clinically relevant signals are often subtle, context-dependent, and difficult to interpret in isolation [14]. This is especially true for depression, where NLP models are increasingly used to infer symptoms or disorder status from language, but predictive performance alone does not show why a model reached its decision [13]. Thus, based on prior work, mental health models should do more than return accurate labels. They should support expert inspection of the evidence underlying a prediction and relate model outputs to clinically meaningful constructs [14], [16]. This requirement also extends to recent large language model (LLM) approaches, which can generate explanations alongside predictions; however, these explanations still require careful evaluation of quality, reliability, and clinical usefulness before they can support expert review [15], [17], [18].

Existing work addresses this problem in several ways. Post-hoc methods [19] such as SHapley Additive exPlanations (SHAP) [20] and Local Interpretable Model-agnostic Explanations (LIME) [21] are commonly used to show which features most influenced depression detection or severity prediction [22]. Other studies improve clinical interpretability by predicting symptoms or questionnaire items rather than only a final label, enabling outputs to be compared more directly with established assessment practice [23], [24]. Research on clinician interaction with mental-health AI further shows that explanations are most useful when they clearly state the basis of a prediction and fit naturally into expert review [25]. Recent interpretable LLM studies follow the same direction by pairing predictions with generated explanations, while also showing that explanation quality must be evaluated rather than assumed [17]. Most existing work, however, still centers on making predictions more interpretable after the fact, leaving open how explanatory content should be organized for systematic expert review.

2.2 Explainable Mental Health Annotation Frameworks↩︎

Annotation frameworks for explainable mental health prediction increasingly move beyond document-level labels toward clinically structured supervision. PsySym introduced a multi-disease symptom annotation framework built from DSM-based symptom classes and clinical questionnaires, with targeted retrieval and quality-control procedures to support symptom-assisted, interpretable detection [26]. DepreSym narrowed this focus to depression by constructing a sentence-level corpus aligned with 21 Beck Depression Inventory-II (BDI-II symptoms), using expert assessors and detailed relevance guidelines to link text to specific depressive markers [27]. A more recent depression dataset goes one step further by annotating both depressive spans and symptom categories, enabling evaluation at the level of textual evidence and symptom alignment [28]. Together, these studies point to a move toward finer annotation, from symptom categories to sentence-level relevance and span-level evidence.

A parallel line of work addresses how such annotations are produced and reviewed. In clinically enriched mental-health data, LLMs have been used to assist annotation and data collection, but reliable outputs still depend on expert-defined variables and clinician oversight [29]. More general human–LLM annotation systems make the same point from a workflow perspective by combining prompt control, annotation management, automatic labeling, and selective human verification [30]. These systems demonstrate how labels can be generated, managed, and checked using AI, but they leave room for closer integration with clinically structured symptom data in mental health settings. The remaining gap is not whether AI can assist with annotation, but how symptom-grounded evidence annotation and collaborative AI–expert review can be unified into a single pipeline for explainable mental health prediction.

3 Proposed Framework↩︎

The proposed framework, illustrated in Fig. 1, establishes a human-centered and self-evolving annotation ecosystem for DSM-5-TR depression labeling. Unlike traditional static pipelines that treat LLMs as independent classifiers, our approach formulates the annotation task as a collaborative, closed-loop process between a Self-Evolving Annotation Agent and a Human Expert.

The architecture is systematically divided into three stages: (1) Evidence-based Screening, (2) Criterion-Level DSM-5-TR Analysis, and (3) Self-Evolving Case Export. The self-evolving capability is driven by a dual-memory architecture consisting of an Example Memory (storing gold-standard few-shot examples) and a Reflection Memory (storing distilled clinical insights). This ensures that the agent progressively aligns with expert intuition through iterative feedback.

3.1 Stage 1: Evidence-based Screening↩︎

The first stage filters raw textual records (e.g., patient narratives, clinical posts) into a focused set of evidence to mitigate the expert’s cognitive burden.

3.1.1 Sentence Segmentation & Relevance Filtering↩︎

The agent begins by processing the input document. Utilizing contextual retrieval from the Example Memory, the agent identifies specific segments that potentially align with depressive symptomatology. These segments are presented as a prioritized list of Candidate Sentences.

3.1.2 Expert Verification and Rationale↩︎

To ensure a human-centered workflow, the Human Expert reviews the proposed candidates. Figure 2 illustrates the expert-facing labeling interface used in this verification stage. The UI organizes the annotation process around DSM-5-TR criteria, displays AI-selected candidate evidence with highlighted clinical cues, and allows experts to accept, revise, or reject the proposed labels. This design reduces the need for exhaustive manual screening while preserving expert control over the final clinical judgment. Expert interventions are recorded as structured feedback and later used to update the framework’s memory components. The expert can:

3.2 Stage 2: Criterion-Level DSM-5-TR Analysis↩︎

The second stage performs a granular mapping of the filtered evidence to the nine diagnostic criteria of Major Depressive Disorder (A1–A9).

3.2.1 Criteria Properties Generation↩︎

For each criterion, the agent generates a structured record known as criteria properties, which includes:

• Preliminary Conclusion: A tentative binary or categorical judgment.

• Clinical Rationale: A concise, clinically grounded explanation that links the evidence to the relevant DSM-5-TR criterion.

• Supporting Evidence: Direct quotes from the candidate sentences.

• Key Phrase Highlighting: Semantic tagging of pathognomonic terms to facilitate rapid expert verification.

3.2.2 Criteria Properties Annotating↩︎

The expert then performs the criteria properties annotation. This stage is augmented by a Conflict Warning mechanism. If the agent detects contradictory clinical signals, such as symptoms suggesting comorbid anxiety or bipolar markers, it triggers a warning icon. This directs the expert’s attention to ambiguous cases, ensuring that the final annotations remain clinically coherent and rigorous.

3.3 Stage 3: Self-Evolving Case Export↩︎

The final stage synthesizes the validated criteria into a comprehensive case profile while updating the system’s internal knowledge base.

3.3.1 Diagnosis & Severity Proposal↩︎

The agent aggregates the refined A1–A9 properties to generate a diagnosis and severity proposal. This module suggests an MDD Diagnosis (applying the five-of-nine rule) and a corresponding Severity Level. A clinical summary is then synthesized to provide a cohesive narrative of the patient’s clinical state.

3.3.2 Final Approval and Memory Integration↩︎

The expert performs the severity adjustment and final approval. Once the expert signs off on the case, the framework exports a structured labeled dataset. This dataset is uniquely comprehensive, containing:

• [Final Labels]: High-fidelity diagnostic judgments.

• [Clinical Evidence]: Grounded text spans and highlighted phrases.

• [Complete Audit Trail & Edit History]: A granular log of human-AI interactions for subsequent machine-human comparison.

3.3.3 The Self-Evolution Mechanism↩︎

The core innovation of this framework is its ability to learn from the audit trail without retraining model parameters: \[\mathcal{K}_{t+1} = \text{Distill}(\mathcal{K}_t, \Delta_{\text{expert}}),\] where \(\mathcal{K}\) represents the system’s knowledge state and \(\Delta_{\text{expert}}\) represents the delta between AI proposals and expert revisions. Expert-approved “gold cases" populate the Example Memory, while the distilled rationales are stored in the Reflection Memory. This closed-loop evolution ensures that the agent’s future proposals become increasingly accurate, making the framework a truly scalable, self-improving clinical tool.

In implementation, expert revisions are converted into two memory updates. First, cases that pass final expert approval are stored as Example Memory entries containing the source text, accepted evidence spans, criterion labels, and final diagnosis. Second, recurring correction patterns are summarized into Reflection Memory entries, such as rules for distinguishing weak affective cues from DSM-5-TR A1 evidence or for rejecting unsupported evidence-to-criterion links. During subsequent annotation, the agent retrieves relevant examples and reflections using the current case representation and incorporates them into the prompting context. No model parameters are updated; adaptation occurs only through retrieval-augmented memory updates.

4 Experiment and Results↩︎

4.1 Experimental Setup↩︎

We conduct a pilot evaluation on 10 complex clinical cases sampled from the ReDSM5 depression-related benchmark dataset [31]. Each case includes narrative evidence supporting the nine DSM-5-TR criteria for MDD. To construct expert gold labels, five experts in psychology and related fields independently annotated the cases under a shared DSM-5-TR protocol. The annotation includes clinically relevant sentence selection, criterion-level labels, evidence-to-criterion links, and final MDD diagnosis. Disagreements were cross-checked and resolved through consensus adjudication, producing gold annotations for evaluating the proposed framework.

After gold labels were established, the same cases were processed by our framework using three LLM backbones: Gemini-3.5-Flash-Lite, GPT-4o-mini, and GPT-5.4-mini. We evaluate the autonomous outputs before expert correction to assess the intrinsic quality of AI-generated annotations. The evaluation covers four levels: sentence-level evidence screening, DSM-5-TR criterion classification, evidence-pair alignment, and case-level diagnosis. In addition, we measure expert revision effort and annotation efficiency to quantify the practical utility of the human-AI workflow.

4.2 Evaluation Metrics↩︎

We evaluate the framework using both consensus-based and human-effort metrics. For autonomous annotation quality, we report precision, recall, and F1-score at the sentence level and DSM-5-TR criterion level, measuring how well the system identifies clinically relevant evidence and assigns symptom labels compared with expert gold annotations. We also compute evidence-pair F1, where each prediction is considered correct only if the model links the correct DSM-5-TR criterion to the correct supporting sentence. Case-level MDD diagnosis accuracy is used to measure agreement with expert final diagnoses.

To assess human-centered utility, we report time saved, total edits, criterion flips, and evidence edits, which quantify the amount of expert correction required when using AI-generated annotations instead of manual annotation. These effort-based metrics are computed at the case level and then averaged across the 10 evaluated samples. Time saved is calculated by comparing the average manual annotation time with the average AI-assisted review time. Total edits measure the average number of expert corrections per case, while criterion flips and evidence edits separately capture changes to DSM-5-TR criterion decisions and supporting evidence links.

4.3 Human-AI Consensus Rate↩︎

Table ¿tbl:tab:consensus? shows that the proposed framework achieves strong agreement with expert gold labels across all LLM backbones. At the sentence level, all models achieve F1 scores above 91%, indicating that the system can reliably identify clinically relevant evidence from long patient narratives. Gemini-3.5-Flash-Lite achieves the best sentence-level precision and F1 Score, with 99.1% precision and 93.8% F1, while GPT-5.4-mini achieves the highest recall at 91.4%. This suggests that the framework is effective both for precise evidence filtering and broad evidence retrieval, depending on the selected backbone.

At the DSM-5-TR criterion level, GPT-4o-mini performs best, reaching 82.9% precision and 81.0% F1. This indicates stronger capability in mapping textual evidence to the correct symptom categories. Gemini-3.5-Flash-Lite achieves the highest criterion recall of 81.4%, indicating greater sensitivity in detecting current symptoms. GPT-5.4-mini remains competitive, with 77.3% criterion F1 and 90.0% diagnosis accuracy. These results demonstrate that the system is not dependent on a single model; instead, it provides a structured DSM-5-TR reasoning layer that can operate effectively with different LLM backbones.

Evidence-pair alignment is the most challenging setting because it requires the model to identify both the correct DSM-5-TR criterion and the correct supporting sentence. Gemini-3.5-Flash-Lite achieves the highest evidence-pair F1-score at 67.0%, closely followed by GPT-4o-mini at 66.7%. The lower evidence-pair scores compared with sentence-level and criterion-level scores show that diagnosis prediction alone is insufficient for evaluating clinical annotation systems. A clinically useful system must also produce traceable and verifiable evidence links.

At the case level, Gemini-3.5-Flash-Lite and GPT-5.4-mini both achieve 90.0% MDD diagnosis accuracy, while GPT-4o-mini achieves 80.0%. This confirms that the framework can produce reliable final diagnostic decisions while preserving intermediate evidence and criterion annotations. Overall, the consensus results show that the proposed system supports multi-level clinical reasoning: it can screen evidence, classify DSM-5-TR symptoms, ground predictions in text, and produce accurate case-level diagnoses.

4.4 Efficiency and Expert Revision Effort↩︎

Table [tbl:tab:efficiency] evaluates how much expert effort is reduced when using the proposed framework. Compared with manual annotation, all AI-assisted settings substantially reduce annotation time. Gemini-3.5-Flash-Lite achieves the largest time saving at 75.0%, followed by GPT-4o-mini at 70.0% and GPT-5.4-mini at 63.0%. This indicates that the system can shift the expert’s role from fully manual annotation to efficient verification and correction of structured AI-generated drafts.

The edit-based metrics further clarify the practical annotation burden. Gemini-3.5-Flash-Lite requires the fewest total edits (10.2 per case) and the fewest evidence edits (6.5 per case). This is consistent with its highest F1-score in the strongest evidence pair in Table ¿tbl:tab:consensus?, suggesting that it provides the most usable evidence-grounded annotations. GPT-4o-mini requires a similar number of total edits, 10.4 per case, and achieves the fewest criterion flips, with only 1.6 flips per case. This aligns with its strongest criterion-level F1-score and indicates that GPT-4o-mini is particularly reliable for DSM-5-TR symptom classification.

GPT-5.4-mini achieves strong sentence recall and diagnosis accuracy, but it requires more expert correction, with 12.9 total edits and 9.3 evidence edits per case. This shows that high diagnosis accuracy does not necessarily imply low annotation cost. Fine-grained revision metrics are therefore essential for evaluating human-centered clinical annotation systems.

Overall, the efficiency results demonstrate that the proposed framework provides substantial practical benefit. It reduces annotation time by 63–75% across LLM backbones while maintaining expert control through structured revision. The system does not replace clinical judgment; rather, it accelerates the annotation process by generating clinically meaningful drafts that experts can inspect, correct, and validate.

4.5 Summary of Findings↩︎

Our pilot results demonstrate that the proposed framework achieves strong human-AI agreement, produces explainable evidence-grounded DSM-5-TR annotations, and reduces expert annotation effort. The results also reveal complementary strengths across LLM backbones. Gemini-3.5-Flash-Lite performs best in sentence precision, evidence grounding, and efficiency; GPT-4o-mini is strongest in criterion-level classification; and GPT-5.4-mini provides high recall and diagnostic accuracy. Overall, these findings suggest that the framework is model-agnostic and support a consistent expert-in-the-loop clinical annotation workflow.

5 Conclusion↩︎

This paper presents a self-evolving DSM-5-TR annotation framework for explainable MDD prediction. By combining LLM-based evidence screening, criterion-level DSM-5-TR reasoning, and expert-in-the-loop verification, the framework produces structured labels, supporting evidence, and traceable revision histories. The pilot results show strong human-AI agreement across multiple annotation levels and demonstrate that AI-assisted labeling can substantially reduce expert revision time while preserving expert control. Through Example Memory and Reflection Memory, expert corrections are further converted into reusable feedback for improving future annotations without model retraining. Future work will extend the evaluation to larger, more diverse datasets; examine long-term improvement across feedback cycles; and adapt the framework to both other mental health conditions and finer-grained assessment tasks.

Acknowledgment↩︎

This work was supported by NSERC Discovery Grant No RGPIN-2025-04478 and NSERC Discovery Supplement Award No DGECR-2025-00129.

References↩︎