1 Introduction↩︎

Autism Spectrum Disorder (ASD) is a lifelong developmental disorder that significantly affects the patient’s daily life, necessitating continuous and personalized intervention treatment [1]. However, the severe shortage of experienced therapists and the high cost of therapy prevent patients from receiving adequate intervention support [2]. Large Language Models (LLMs), with their powerful interactive capabilities and accessibility, offer a promising solution to democratize intervention resources, potentially breaking the constraints of time and location to support a wider demographic.

Despite its promising potential, the application of LLMs in clinical settings for interventions faces challenges brought by sycophancy. General LLMs (e.g., GPT-4o [3]) are trained through Supervised Fine-Tuning (SFT) and Reinforcement Learning with Human Feedback (RLHF) [4] to ensure harmlessness and helpfulness in accordance with human preferences. But they exhibit strong sycophantic tendencies, often aligning with users’ viewpoints [5]. While this may be perceived as empathetic support or polite behavior in casual conversation, it poses significant risks in ASD interventions based on Applied Behavior Analysis (ABA) [6]. As shown in the Fig. 1, when faced with the child’s error response, the therapist adheres to the ABA principle to avoid reinforcement, repeats the instruction, and provides assistance. In contrast, ASD-iLLM [7] and GPT-4o exhibit sycophancy (Quantitative results in Appendix 7): ASD-iLLM complies with the child’s response and questioning, while GPT-4o offers an empathetic reply along with a seemingly reasonable explanation. Such erroneous strategy may lead to the child forming incorrect cognitive connections, undermining intervention efficacy.

Prior work applies Direct Preference Optimization (DPO) [8], [9] to mitigate sycophancy. However, we find that sycophantic behavior can often be localized to a limited span within the model response (Appendix 8). In contrast, vanilla DPO [10] optimizes preferences at the sequence-level, which may inadvertently disrupt well-formed segments that are irrelevant to preferences, leading to a decline in language quality (e.g., repetition).

To address this, we introduce a Minimal-Edit Data Augmentation strategy (MEDA) to construct preference pairs that differ by minimal edits. Then, we propose Token-level Difference Direct Preference Optimization (TD-DPO) to selectively amplify signals on these different tokens between chosen and rejected responses while suppressing noise in the shared context. Across multiple backbones and evaluators, we find that our method achieves a better trade-off between mitigating sycophancy and retaining the ability acquired before, suggesting its promise for safety alignment in autism intervention models.

Our contributions are summarized as follows:

• Method. We propose MEDA for constructing minimal edit preference pairs and TD-DPO for token-level difference-aware preference optimization, which upweights preference-critical edited tokens and downweights shared context tokens, enabling more targeted updates with reduced drift on shared tokens.

• Diagnostic analysis. We present diagnostic analyses on why DPO degrades and why TD-DPO alleviates it: DPO exhibits shifted EOS probabilities and larger drift on shared tokens, both of which correlate with degeneration signals (e.g., long-tail outputs and repetition).

• Empirical validation. Across diverse evaluations, TD-DPO achieves a better trade-off between sycophancy mitigation and capability retention in our offline settings, suggesting its potential for reducing sycophancy.

2 Related Work↩︎

2.1 Sycophancy in LLMs↩︎

Recent research has identified sycophancy in LLMs: the models tend to alter their judgments in order to maintain conversational smoothness or cater to users’ preferences. This occurs when the models are confronted with incorrect viewpoints [11], misleading cues [12], or persistent pressure [13] from users, leading to a sacrifice of truthfulness, safety, and reliability. Furthermore, researchers have found that sycophancy exhibits a cumulative effect in multi-turn dialogues [5], [14]. This kind of sycophancy causes the model to overly empathize with and morally support users, which may temporarily increase user satisfaction but could foster dependency and result in negative societal consequences [15]–[17].

Therefore, when applying LLMs to clinical scenarios, such as autism intervention, it is crucial to be aware of the risk that models may reinforce inappropriate cognition or feedback due to sycophancy.

2.2 Autism Support via LLMs↩︎

The emergence of LLMs’ capabilities has prompted more researchers to explore the application in autism intervention [18]. On one hand, LLMs are employed as auxiliary tools to enhance existing intervention paradigms [19]–[21]. On the other hand, LLMs are used to replace therapists in intervention tasks. For instance, employing robots as conduits for social dialogue intervention [22]–[24], utilizing software to achieve social intervention [25] or emotion training [26]. [7] pointed out that the output of current advanced LLMs does not meet the requirements of clinical interventions. So they constructed a clinical dialogue intervention dataset ASD-iLLM-8K for fine-tuning, enhancing the model’s intervention capabilities.

Existing research has not considered the sycophancy risk in LLMs, which may lead to potential risks in clinical autism intervention.

2.3 Token-Level Preference Optimization↩︎

DPO [10] has emerged as a widely adopted method for preference alignment. However, vanilla DPO computes loss at the sequence level, leading to gradient dilution by irrelevant tokens and increased noise. Recent research addresses this issue by refining preference alignment to the token level.

The first category enhances signal-to-noise ratio at key positions by token weighting, such as OTPO [27], TGDPO [28], TIS-DPO [29], and SWIFT [30]. Several methods focus on token selection, optimizing a few key tokens to minimize noise and achieve sparse updates, such as ConfPO [31] and SePO [32]. Beyond weighting and selection, other methods exploit richer supervision signals. T-REG [33] infers token-level rewards from self-generated comparative rewrites and uses them as regularization in preference learning. TKPO [34] similarly leverages token-level comparisons for self-alignment in controllable generation. TDPO [35] provides a token-level decomposition with forward-KL constraints to alleviate diversity degradation, while TI-DPO [36] replaces binary pairs with a better/neutral/worse triplet to obtain smoother and more stable learning.

3 Methodology↩︎

3.1 Problem Formulation↩︎

When the autistic child exhibits incorrect or evasive responses, following the ABA principles of instruction, reinforcement and assistance, clinicians should implement the following strategies: (i) provide neutral feedback to avoid affirming incorrect responses, (ii) guide the child to the correct response through repetition or simplification of instructions, (iii) selectively offer prompts for assistance, and (iv) avoid social reinforcement like praise to prevent the child from forming incorrect cognitive associations. These strategies are strictly followed in clinical practice, whereas compliance or sycophancy could potentially disrupt the closed-loop effectiveness of intervention. For more details about ABA, please refer to Appendix 9.

Our research focuses on LLMs that can be directly used for ASD intervention dialogues following the ABA principle. However, current LLMs may exhibit sycophancy when faced with two types of responses from autistic children: (i) Factual Error: The child makes a common-sense mistake; (ii) Evasive Response: The child evades the instruction by unrelated sentences, attempting to derail the instructional loop. Therefore, sycophancy in LLMs can also be categorized into two types: (i) Error Affirmation: Affirming the child’s mistakes; (ii) Instruction Shift: Complying with the child’s unrelated sentence to continue intervention.

We define the research problem as follows: \[x = (t, h, u_c)\] where \(t\) is the intervention topic, \(h\) is the dialogue history, and \(u_c\) is the child’s latest utterance. Given the context \(x\), the LLMs generate the response \(y \sim \pi_\theta(x)\). Then, we define a binary failure indicator: \[\phi(x,y) \in \{0,1\}\] Where \(\phi(x,y)=1\) means that \(y\) exhibits sycophancy defined before and \(\phi(x,y)=0\) otherwise.

Our primary metric is the Non-Sycophancy Rate (NSR) over a benchmark set \(\{x_i\}_{i=1}^{N}\): \[NSR = \frac{1}{N}\sum_{i=1}^{N} I [\phi(x_i, \hat{y}_i)=0]\] Where \(\hat{y}_i\) is the model output for \(x_i\).

Our goal is to learn a policy \(\pi_\theta\) that increases NSR, while retaining intervention ability.

3.2 Limitations of DPO↩︎

DPO learns from input tuples \((x, y_w, y_l)\) based on Bradley-Terry model: [37]:

\[P^*(y_w \succ y_l \mid x) = \sigma(r^*(x, y_w) - r^*(x, y_l))\]

And optimizes the policy \(\pi_\theta\) without an explicit reward model by using the implicit reward \(r(x,y)=\beta\log\frac{\pi_\theta(y\mid x)}{\pi_{\mathrm{ref}}(y\mid x)}\), yielding the objective: \[\mathcal{L}_{\text{DPO}} = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma \left(\beta \Delta{r} \right) \right]\]

\[\Delta{r} = \log \frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \log \frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)}\]

A key limitation of vanilla DPO is its sequence-level optimization, which treats all tokens as equally important. This can misdirect gradients towards irrelevant tokens, disrupting well-formed shared context and leading to language degradation. This issue is particularly critical in domains like autism interventions, where the success of a response may hinge on relatively small spans.

To address this, we first introduce MEDA to construct preference pairs that differ only in a few key tokens. Then, we propose TD-DPO, which assigns distinct weights to different and shared tokens.

3.3 Minimal Edit Data Augmentation↩︎

Inspired by [38], MEDA is designed to generate high-quality preference pairs that reliably elicit sycophantic behaviors while adhering to the minimal edit principle to preserve the original distribution. It proceeds in three steps as shown in Fig. 2.

Firstly, to address the limited diversity and individual variation of children’s erroneous utterances in real clinical dialogues, we augment the dataset by inducing child responses that are likely to trigger sycophancy. Concretely, we sample a subset of real dialogue histories from the ASD-iLLM-8K [7] and use LLMs to generate two types of error responses conditioned on the history.

Secondly, we replace the original child response with the generated inducement response, prompting the doctor model to produce the doctor response under the same dialogue context.

Thirdly, we convert the doctor’s response into a preference pair. If the doctor model’s response exhibits sycophancy, we use it as the rejected response and ask LLMs (e.g., GPT-4.1) to minimally rewrite it into a chosen response. Conversely, we treat it as the chosen response and ask LLMs to generate a rejected counterpart. This approach keeps one side of the pair closer to the model’s original output, stabilizing preference optimization while preserving desirable intervention behaviors and correcting problematic patterns. More details about MEDA are provided in Appendix 10.

3.4 Token-level Difference Preference Optimization based on Minimal Edit↩︎

TD-DPO is based on the intuition that, for non-sycophantic alignment, preference supervision is typically localized: the chosen and rejected responses often share most background text and differ in a few critical edits. Therefore, TD-DPO explicitly identifies the edited regions between \((y_w,y_l)\) and re-weights tokens accordingly.

Given a preference tuple \((x, y_w, y_l)\), we compute a token-level minimal edit decomposition between \(y_w\) and \(y_l\). We first tokenize \(y_w\) and \(y_l\) via the model tokenizer, then recursively extract their Longest Common Substring (LCS) to segment each response into an alternating sequence of shared parts and edited parts. Shared parts are the maximal contiguous spans that occur in both responses, while edited parts are the remaining segments. This procedure yields a fine-grained segmentation that reflects the minimal edits transforming \(y_l\) into \(y_w\).

3pt

4pt

To focus on failure modes, masking and re-weighting are applied exclusively to \(y_l\). A binary mask \(M = (m_1, \dots, m_T)\) is constructed over its tokens, where tokens corresponding to edited parts are designated as difference tokens (\(m_t = 1\)), and the remaining tokens are marked as shared tokens (\(m_t = 0\)). Using the mask, we define a token-weighted log-probability for \(y_l\): \[\log \pi_{\text{weighted}}(y_l|x) = \sum_{t=1}^{T} w_t \cdot \log \pi(y_t | x, y_{<t})\]

The weight \(w_t\) is defined as:

\[w_t = \begin{cases} \alpha_{\text{diff}}, & \text{if } m_t = 1 \\ \alpha_{\text{shared}}, & \text{if } m_t = 0 \end{cases}\]

We typically set \(\alpha_{\text{diff}}> 1\) to amplify gradients on preference-critical edits, and \(\alpha_{\text{shared}}<1\) to suppress background noise on shared context.

TD-DPO follows the DPO objective but applies token re-weighting only to the rejected response:

\[\mathcal{L}_{\text{TD-DPO}} = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[ \log \sigma (\beta\Delta{r^*}) \right]\]

Where \(\Delta{r^*}\) is:

\[\Delta{r^*} = \left( \log \frac{\pi_\theta (y_w|x)}{\pi_{\text{ref}}(y_w|x)} - \log \frac{\pi_\theta^{\text{weighted}}(y_l|x)}{\pi_{\text{ref}}^{\text{weighted}}(y_l|x)} \right)\]

This asymmetric design sharpens the penalty on preference-critical errors in \(y_l\) while reducing gradient interference on the shared background tokens.

While the TD-DPO objective operates on token indices, in the Chinese context, we optionally identify edit spans at the character-level for more consistent segmentation, and then map the spans back to token indices for masking.

4 Experiment Setup↩︎

4.0.0.1 ASD-iLLM Dataset.

We adopted the same experimental setup as [7], conducting SFT on the training set of ASD-iLLM-8K to learn ABA principle and validating whether the model’s intervention capability degrades on its test set.

4.0.0.2 Sycophancy Benchmark.

To evaluate sycophancy mitigation, we construct a specialized benchmark derived from ASD-iLLM-8k [7] using the proposed MEDA strategy. The ASD-iLLM (Training on ASD-iLLM-8K based on Qwen2.5-7B-Instruct [39]) was selected as the doctor model to generate responses.

We extracted 500 real dialogue histories from the ASD-iLLM-8K training set and 100 dialogue histories from the test set. Using the MEDA strategy, 1,000 training preference pairs and 200 test preference pairs were generated by GPT-4.1. To improve the quality, we invited an expert (4 years of experience in autism intervention) to eliminate pairs that did not meet ABA requirements, resulting in 912 training and 182 test samples (failure rate of the MEDA strategy is approximately 9%). The expert also validated and revised the test set to ensure adherence to the minimal edit principle and ABA principle. Tab. 1 provides detailed statistics of the sycophancy benchmark, where each example consists of a dialogue context including the induced child’s response and a minimal edit preference pair. Overall, MEDA produces short preference pairs (approximately 13–17 characters). It also reports the similarity of different types of responses, as calculated by edit distance. It shows that the similarity for factual errors is generally higher than that for evasive responses. This difference is mainly because evasive responses involve topic shifts, necessitating more token edits.

4.0.0.3 Baseline.

We compare TD-DPO with DPO, SimPO [40], ConfPO [31], and OTPO [27]. Additionally, we examine other variant methods in Appendix 11, including SPPO [41], CPO [42], and ORPO [43].

4.0.0.4 Model Training.

We conduct experiments on three open-source foundation models to ensure robustness: Qwen2.5-7B-Instruct, Yi-1.5-9B-Chat [44], and Llama3-Chinese-8B-Chat [45]. For the training pipeline, we fine-tune these base models on ASD-iLLM-8k using LoRA [46] to obtain the SFT policy \(\pi_{\text{SFT}}\) to learn ABA principle. Subsequently, we perform preference optimization on \(\pi_{\text{SFT}}\) using our constructed sycophancy training set via LoRA. More training details are provided in Appendix 12.

4.0.0.5 Metrics.

We introduced GPT-4.1, Deepseek-v4 and a human expert (with three years of autism intervention experience) to judge whether the model’s outputs exhibit sycophancy and computed the NSR. Additionally, to evaluate whether the model’s language capabilities have degraded, we introduced metrics such as Distinct-2 (D-2) [47], 4-gram repetition rate (R-4), BERTScore (BS.) [48], and Negative Log-Likelihood (NLL) for comprehensive evaluation.

For human evaluation, inspired by [7], to more realistically evaluate the model’s performance in intervention scenarios, we propose a dynamic dialogue evaluation method based on role-playing. This approach employs persona modeling and few-shot learning [49] to enable the LLMs to mimic the language behavior of autistic children. GPT-4o plays the role of the autistic child, engaging in topic intervention dialogues with doctor models. The generated dialogue histories are assessed by human experts across three dimensions: Professionalism (ABA principle, Personalization), Linguistic (Relevance, Style, Fluency), and Safety (Correctness, Harmlessness, Privacy), with scores ranging from 0 to 4. Higher scores reflect more consistent adherence to intervention principles by the doctor models throughout the dialogue. For more details about the metrics and method, please refer to Appendix 13.

5 Results and Analysis↩︎

5.1 Main Result on Sycophancy Benchmark↩︎

Tab. 2 presents the NSR metric results of different methods on the sycophancy benchmark. TD-DPO achieved the best performance across all three base models, yielding an average NSR of 90.90%. Notably, the judgments of GPT-4.1, Deepseek-v4, and human evaluators exhibit highly consistent trends across all methods (Kendall’s W=0.73, ICC(2,k)=0.59), suggesting that the observed gains are stable and not tied to a specific evaluator. The relatively limited gains of ConfPO can be attributed to the partial overlap between low-confidence and sycophantic tokens in our settings (about 60%). OTPO relies on semantic matching and transport-based distribution, which makes its performance less stable across backbones. SimPO mainly introduces length-based weighting and therefore does not directly target semantic-level sycophancy mitigation. A more detailed discussion of the error types and the comparison with other existing methods (including SFT on chosen response) is provided in Appendix 11.1. We also report a strict token-level masking baseline in Appendix 11.2, which leads to consistent conclusions. The overall performance with the Qwen backbone was the best, so subsequent experiments were conducted on this.

5.2 Diagnostic Analysis↩︎

5.2.0.1 Why DPO degrades: EOS probability collapse.

We observe that DPO-aligned models are more prone to repetitive loops when confronted with inducement responses. To diagnose this behavior, we measure how likely a model is to stop at the gold end position on the sycophancy benchmark. Concretely, when conditioning on the context and the reference response prefix up to the reference end, DPO assigns a substantially lower \(P (\text{EOS} \mid x,y*)\) than SFT and our method. As shown in Fig.3, DPO and OTPO display lower EOS prediction probabilities on the reference response, indicating a higher risk of pattern collapse. In contrast, TD-DPO’s EOS prediction probability aligns closely with SFT, indicating that TD-DPO can robustly mitigate sycophancy while minimizing the risk of pattern collapse. More results refer to Appendix 11.3.

5.2.0.2 Why TD-DPO works: reduced drift on shared tokens.

We further analyze DPO and TD-DPO by measuring token-level changes in \(\Delta\)NLL for different types of tokens relative to the SFT model, aiming to explore the source of performance differences between DPO and TD-DPO. Specifically, the dialogue history containing the induced child response is concatenated with the reference response and fed into the model to calculate its logits and obtain the NLL. Then \(\Delta\)NLL of two types of tokens between the DPO or TD-DPO and the SFT model is calculated separately. As shown in Fig. 4, the distribution for different tokens is nearly identical between DPO and TD-DPO, indicating that both methods focus on different tokens for similar updates. In contrast, DPO exhibits a higher drift on shared tokens compared to TD-DPO, suggesting that DPO unnecessarily updates more task-irrelevant background tokens. Overall, these observations support our motivation that downweighting shared tokens can reduce background drift and is associated with less degeneration in our experiments.

These diagnostics provide support for our asymmetric design choice and motivate us on where to apply token re-weighting in the preference objective. Specifically, we compare rejected-only TD-DPO with chosen-only and symmetric variants. Empirically, the rejected-only variant yields a better trade-off between sycophancy mitigation and retention (Appendix 11.4).

5.3 Degradation and Retention Analysis↩︎

To further assess the extent of pattern collapse and intervention capabilities in aligned models, we calculated the output tokens length (Tokens), distribution rate (Len\(\geq\)​30), R-4, and D-2 on two benchmarks. Additionally, in the ASD-iLLM Dataset, we calculate the BertScore for semantic similarity and compute the NLL on the reference response.

The result is shown in Tab. 3. On the sycophancy benchmark, DPO exhibits clear degradation: it generates longer outputs and much stronger repetition while reducing diversity. In contrast, TD-DPO retains the lowest verbosity and repetition among all compared methods (7.75 tokens, 0.55% with length \(\geq\)​30, R-4=0.33%) while maintaining high diversity (D-2=97.80, second best), indicating that our method in robustness does not lead to pathological long-tail generation or repetitive patterns.

On the ASD-iLLM test set, TD-DPO preserves overall behavior close to SFT: it retains competitive semantic similarity and likelihood (BS=69.74, NLL=2.51) and does not induce long responses or severe repetition (10.31 tokens, 2.92% with length \(\geq\)​30; R-4=0.94%). By contrast, DPO and OTPO show worse degradation and retention, suggesting that TD-DPO can offer a better trade-off. Complete experiment results and case studies can be found in Appendix 11.5.

5.4 Ablation Study↩︎

5.4.0.1 Appropriately amplifying the \(\alpha_{\text{diff}}\) while reducing the \(\alpha_{\text{shared}}\).

Fig. 5 shows a heatmap of the NSR metric for different weight combinations. The results indicate that moderately increasing \(\alpha_{\text{diff}}\) while reducing the \(\alpha_{\text{shared}}\) contributes to enhanced performance (optimal at diff=2, shared=0.5). Excessive \(\alpha_{\text{diff}}\) can lead to overfitting and reduce generalization, while excessively low \(\alpha_{\text{shared}}\) (approaching 0) also results in performance degradation because shared tokens provide contextual anchors that stabilize alignment and mitigate drift. We further verify this claim with a strict masked DPO baseline that zeros shared token gradients on both chosen and rejected responses, which performs consistently as shown in Appendix 11.6.

3pt

5.4.0.2 Effect of the minimal-edit constraint in MEDA.

We ablate the impact of the minimal edit constraint in MEDA by constructing preference pairs without the minimal editing constraint. Specifically, we follow the MEDA process but do not explicitly prompt the model to generate with minimal changes. As shown in Tab. 4, enforcing the minimal edit constraint consistently improves NSR across all methods: DPO, SimPO, ConfPO, OTPO, and TD-DPO improved 2.74%, 1.65%, 3.30%, 1.10%, and 4.4%, respectively, indicating that the minimal edit constraint provides a more focused preference signal for mitigating sycophancy. The complete details are provided in Appendix 11.7.

5.4.0.3 Transferability of TD-DPO.

To provide evidence of transferability, we evaluate TD-DPO on the Sycophancy Answer Assessment (SAA) dataset³ [9], which tests whether a model appropriately accepts correct user suggestions, rejects incorrect ones, thereby mitigating sycophancy. As shown in Tab. 4, under a 9:1 train-test random split, TD-DPO achieves the best performance on the SAA benchmark, reaching 73.46% Acc and 74.43% F1 score. It outperforms the strongest baseline by 2.31% in Acc and 3.53% in F1 score, suggesting that TD-DPO can transfer beyond our sycophancy benchmark. More details are provided in Appendix 11.8.

5.4.0.4 Robustness to MEDA strategy.

We replicate MEDA using two extra generators (Deepseek-V3 and Qwen2.5-32B-Instruct) to claim its robustness. We conduct the experiments under the same settings described before via TD-DPO. Results show TD-DPO consistently mitigates sycophancy across generators, although absolute NSR varies with generator. More details are provided in Appendix 11.9.

5.5 Human Evaluation↩︎

We randomly selected 50 topics from the ASD-iLLM test set, using GPT-4o to simulate autistic children, and engaged in 20 turns of topic intervention dialogue with the doctor models. Ultimately, we obtained 50 dialogue samples, totaling 1,000 dialogue turns. The dialogue records were evaluated by three experienced clinicians across 8 dimensions. As shown in Fig. 6, our method achieves the best overall score among all model settings. Especially in terms of ABA, the results are closest to the real doctor, scoring 3.27, indicating that sycophancy mitigation helps the model adhere more consistently to ABA requirements. We further verify scoring consistency, obtaining Kendall’s W (0.71) and ICC(2,k)=0.48, indicating substantial agreement in relative ranking but moderate agreement in absolute scores. Since our analysis focuses on between-method comparisons rather than absolute score magnitudes, the strong rank agreement supports the reliability of our conclusions.

6 Conclusion↩︎

In this paper, we propose MEDA and TD-DPO to better target sycophancy during preference-based alignment in autism intervention dialogue by emphasizing preference-critical edits and reducing drift on shared content. Across multiple backbones and evaluators, offline experiments show improved trade-offs between sycophancy mitigation and ability retention versus strong baselines, suggesting TD-DPO as a practical safety alignment approach for autism intervention dialogue models.

Limitations↩︎

6.0.0.1 Benchmark and domain specificity.

We conducted offline evaluations to assess the effectiveness of TD-DPO on the sycophancy benchmark, the ASD-iLLM test set, and the SAA dataset. However, our study remains limited in scope and scale: we do not evaluate on large-scale multilingual or cross-cultural sycophancy benchmarks, and we do not claim clinical validation beyond autism intervention-style dialogue settings. The effectiveness of TD-DPO under substantially different languages, cultural norms, and larger-scale evaluations remains for future work.

6.0.0.2 Reliance on LLM-based services.

Our pipeline relies on LLM services for preference-pair generation and for parts of automatic evaluation, which may introduce potential bias. To reduce this risk, we incorporate human-in-the-loop steps, including expert review of the test set, human NSR assessment, and multi-criteria human scoring. Nevertheless, further work is needed to strengthen reproducibility controls and to expand human auditing where feasible.

6.0.0.3 Clinical deployment.

The results were obtained through offline experiments, and no real clinical deployment has been conducted to measure downstream clinical outcomes.

Ethics Statement↩︎

This paper studies alignment methods for high-risk clinical dialogue models in the context of autism intervention. Our experiments are offline and do not constitute clinical validation. Therefore, the trained model in this paper cannot be directly applied to formal clinical interventions without additional evidence, safeguards, and oversight.

6.0.0.4 Data privacy.

We use de-identified dialogue data and follow data minimization principles. When constructing preference data with the MEDA strategy, we avoid generating or storing personally identifiable information.

6.0.0.5 Potential risk under distribution shift.

While TD-DPO reduces the targeted sycophancy behaviors in our settings, failures may still occur in out-of-distribution scenarios (e.g., hallucinations or undesirable response shifts). Accordingly, any real-world use in clinical contexts would require conservative deployment practices, routine human review, and continuous monitoring.

6.0.0.6 Fairness, representativeness, and cultural generalization.

Due to differences in culture, language, and individual needs, the requirements for autism dialogue interventions vary. Our data may not represent all populations, and model behavior may differ across subgroups and contexts. Future work should broaden data collection with diverse expert involvement and evaluate robustness and safety across settings before deployment.

7 General LLMs Performance on the Sycophancy Benchmark↩︎

To demonstrate that sycophancy is not limited to our training settings, we also evaluate several advanced LLMs on sycophancy benchmark. We selected GPT-4o [3], Deepseek-V3.2 [50], Claude-Sonnet-4.5, and Gemini-2.5-Flash [51] for the NSR experiments. The experimental setup is consistent with the previous description, with the temperature set to 0 for reproducibility.

As shown in Tab. 5, sycophancy remains prevalent in general LLMs. We also observed that when the above-defined two types of sycophancy occur in these models, they are often accompanied by toxic positivity: demonstrating empathy and praise towards children’s incorrect responses. However, this phenomenon has been significantly mitigated in the ASD-iLLM model fine-tuned on the ASD-iLLM-8K dataset. This indicates that using general LLMs in clinical settings can present more risks, while the TD-DPO-aligned model can mitigate sycophancy while maintaining its original linguistic capabilities (e.g., adhering to ABA principles and avoiding toxic positivity).

8 Character-level sycophantic spans analysis↩︎

To quantify how concentrated sycophancy is within a response, we perform a post-hoc localization analysis on all responses judged as sycophancy on our sycophancy dataset. Given the last turn context, we use GPT-4o to mark the minimal sycophantic spans in the model response. We merge overlapping spans and compute a character-level span ratio as the number of characters covered by the merged spans divided by the total number of characters in the response. The prompt we used is shown in Fig. 7. Across 503 sycophantic responses, the average span ratio is 0.43 (median: 0.4, Q1: 0.21), indicating that sycophancy is often attributable to a localized portion of the response rather than being uniformly distributed across the entire sentence. This observation aligns with our motivation to focus optimization on preference-critical edits.

9 ABA Principle used in Clinical Intervention↩︎

Applied Behavior Analysis (ABA) is an intervention therapy method based on behavioral principles that is widely used in the treatment of autistic individuals. Its core requirement is to break down tasks, such as knowledge, skills, behaviors, and habits, into smaller and relatively independent steps following specific principles and sequences. This allows autistic children to gradually master various skills through intensive training sessions.

Discrete Trial Training (DTT) is an effective method derived from ABA. In practice, it follows five key elements: instruction, response, reinforcement, assistance, and pause.

Instruction refers to the request made by the doctor to prompt the autistic child to achieve a target behavior, typically in the form of a command or question. Instructions can generally be divided into verbal instructions and non-verbal instructions (e.g., gestures, actions, visuals). Verbal instructions are often accompanied by non-verbal cues to help the child understand the intent of the instruction more effectively and encourage their response. Instructions should be timely and easy for the child to comprehend.

Response refers to the reaction of an autistic child to a given instruction. Children’s responses are generally classified into three types: correct response, incorrect response, and no response. Doctors typically implement different intervention strategies based on the type of response exhibited by the children.

Reinforcement refers to the measures provided by the doctor based on the child’s response to encourage correct or desired behaviors. Rewards for reinforcement can be psychological (e.g., verbal praise, hugs, smiles) or material (e.g., food, toys). The content of the rewards should vary according to the child’s preferences and needs.

Assistance refers to the assistance provided by doctors when an autistic child is unable to complete a given instructional task. The goal is to help the child understand and complete the task. There are five general types of prompts:

(i) Verbal Assistance: Providing supplementary verbal instructions, such as offering key words from the answer to encourage the child to associate and complete the intended response. (ii) Visual Assistance: Using certain media to guide the child to observe and learn the target with their eyes. (iii) Positional Assistance: Placing the stimuli in a position that makes it easier for the child to give the correct response. (iv) Physical Assistance: Providing physical guidance through bodily contact to help the child achieve the correct response. (v) Demonstration: Demonstrating how to complete the task to help the child understand and perform the task.

A typical dialogue turn is illustrated in Fig. 8. Each turn begins with the instruction. If the autistic child provides a correct response, reinforcement is promptly given to encourage the continuation of the correct response. This is followed by a pause to give the child time to reflect and prepare for the next turn.

If the child does not respond (usually defined as no response within 1-3 seconds), the doctor repeats the instruction. If there is still no response after repeating the instruction twice, assistance is immediately provided to encourage the correct behavior.

If the child gives an incorrect response, this incorrect behavior should not be reinforced. Timely correction is necessary to prevent an increase in the likelihood of such behavior recurring. The next turn then begins immediately, with the instruction being repeated and supplemented with assistance to encourage the correct behavior. When the correct behavior finally occurs, reinforcement should be promptly provided, followed by a pause.

10 The Details of MEDA Strategy↩︎

The MEDA strategy consists of three steps. Firstly, two types of erroneous child responses are generated using LLMs (e.g., GPT-4.1). Then, the doctor model generates outputs to these erroneous child responses. Finally, the doctor’s responses are judged to determine whether sycophancy occurs, and based on this, the chosen and rejected pairs required for alignment are constructed.

In the first step, A portion of the real dialogue history was extracted from the ASD-iLLM training set, followed by the generation of erroneous child responses based on these dialogue histories, utilizing GPT-4.1. The specific prompt is shown in Fig. 9. It is worth noting that this prompt template enables the model to simultaneously generate two different types of incorrect responses. This approach not only enhances data utilization efficiency but also balances the types of errors.

In the second step, based on the dialogue history, replace the child’s original response with the induced responses, allowing the doctor model to generate the doctor’s outputs.

In the third step, preference data is constructed based on the evaluation of whether the doctor’s model output meets the clinical requirements. We use GPT-4.1 as a judge for evaluation, with the prompt template shown in Fig. 10.

If the model’s response demonstrates sycophancy, it is categorized as a rejected response, and GPT-4.1 generates a chosen response using the prompt template shown in Fig. 11. Conversely, it is used as the chosen response, and GPT-4.1 generates a rejected response based on the prompt template shown in Fig. 12. This approach maximizes the utilization of existing data, improving efficiency and training robustness.

Finally, we provide a case for generating preference pairs using MEDA strategy, as illustrated in the Fig. 13.

11 Additional Experiments↩︎

11.0.0.1 Baseline.

We conducted the same experiments on more baselines. Here is a brief description of the methods we compared:

• Base: The original weights of the open-source model.

• SFT: The model fine-tuned with the ASD-iLLM-8K training set using the LoRA method.

• Chosen-SFT: The model was fine-tuned based on SFT using the chosen responses from our sycophancy preference training set.

• SimPO: By introducing explicit length normalization and a target reward margin term, the reward function of SimPO is fully aligned with the optimization objective of the generation process.

• ConfPO: By selectively optimizing low-confidence tokens, ConfPO enhances the alignment performance of LLMs. It relies solely on the confidence scores of the policy model itself to identify key tokens, without requiring additional computational cost.

• OTPO: A token weighting scheme grounded in optimal transport theory. Within a pair of chosen responses and rejected responses, those parts that are semantically similar (e.g., correctly addressing the core information of the question) are considered more critical and are assigned higher weights. In contrast, portions with significant differences or irrelevant content are assigned lower weights.

• CPO: By combining preference learning terms with negative log-likelihood terms, CPO uses a contrastive learning mechanism to enable the model to distinguish and prioritize generating preferred responses while maintaining the distribution of preferred data.

• ORPO: Through single-stage odds ratio optimization, preference alignment is directly integrated into supervised fine-tuning, enabling efficient preference learning without the need for a reference model.

• SPPO: its core idea is modeling the alignment problem of LLMs as a two-player constant-sum game, approximating a Nash equilibrium policy through iterative self-play updates. SPPO directly optimizes preference probabilities without relying on traditional reward models. (Here, we only utilize the designed loss function for training without performing the self-play update process.)

11.1 NSR Experiment Results↩︎

The detailed NSR experimental results judged by GPT-4.1 are shown in the Tab. 6. For fair comparison, we also fine-tuned SFT using the chosen responses from the preference training set and calculated its NSR on the test set. The training parameter settings are consistent with those used for fine-tuning ASD-iLLM. It can be observed that the TD-DPO method achieves the best NSR performance and lowest (or tied-lowest) error rates across all compared methods, demonstrating its effectiveness and robustness.

The performance differences primarily stem from differences in the method. Chosen-SFT brings moderate improvements over the SFT model, showing that direct supervision on chosen responses can already alleviate sycophancy to some extent. However, its performance remains clearly below TD-DPO and other preference-optimized methods, indicating that simple supervised learning on positive responses is insufficient to fully capture sycophantic behavior. ConfPO’s overall NSR improvement is relatively limited, indicating that in sycophancy scenarios, low-confidence tokens are not equivalent to sycophantic tokens. OTPO performs impressively in certain settings (e.g., in Qwen, NSR=90.10), which demonstrates the effectiveness of using semantic matching for critical weight allocation. However, its reliance on semantic matching and transport-based distribution makes its performance inconsistent across different backbones. SimPO, which applies weighting from the length dimension, does not focus on semantic-level sycophancy mitigation. On the other hand, sequence-level methods such as DPO, CPO, ORPO, and SPPO dilute gradients over preference-irrelevant tokens and introduce background drift, thereby failing to effectively mitigate sycophancy.

Insight on ConfPO. ConfPO assumes that low-confidence tokens are a reliable proxy for the tokens responsible for preference differences. In our sycophancy benchmark, we directly test this assumption by comparing the “sycophantic token set” (difference tokens between rejected and chosen responses) with the “low-confidence token set” defined by ConfPO (tokens below the sequence-average probability threshold). We find that sycophantic tokens are indeed lower-confidence on average than non-sycophantic (shared) tokens (mean confidence: 0.39 vs. 0.61), suggesting confidence provides some signal. However, the proxy is neither sufficient nor selective: the alignment between sycophantic and low-confidence tokens is only moderate (overlap F1 = 0.63) and only slightly above a size-matched random baseline (F1 = 0.59), indicating that a substantial portion of low-confidence tokens are preference-irrelevant noise under this criterion. Moreover, sycophantic-token confidence exhibits a broad distribution (e.g., p90 = 0.93), meaning a non-trivial fraction of sycophantic cues are high-confidence that ConfPO is unlikely to prioritize. These findings suggest a proxy mismatch: ConfPO may focus on many low-confidence but preference-irrelevant tokens and miss some highly confident sycophantic cues, which helps explain its limited improvement compared to methods that target preference-differentiating or semantically critical tokens more directly.

Style matching is not a sufficient explanation. A naive expectation is that backbones closer to the generator might benefit from style matching, however, this is not consistently observed in our settings. As shown in Tab. 6, methods such as SimPO, ConfPO, and CPO exhibit better NSR results on other backbones in some cases. This suggests that our benchmark primarily measures sycophancy mitigation rather than stylistic similarity. One plausible explanation is that overly strong style matching may encourage preserving superficial phrasing patterns that co-occur with the generated pairs, instead of focusing on the few critical edits that determine whether a response is sycophantic. This issue may be more pronounced in our setting, where chosen and rejected responses are short and often differ by only a few tokens.

11.2 Comparison between Token-level and Character-level Implementation↩︎

11.2.0.1 Motivation for character-level implementation.

Although TD-DPO is defined over token indices (the masking and re-weighting are applied at the token level), our default Chinese implementation first localizes minimal edit spans at the character level and then maps the resulting spans back to token indices using the model tokenizer for masking. On one hand, in the Chinese context, tokenizer boundaries do not consistently align with human-perceived minimal edits, so character-level span segmentation yields more stable and interpretable edit regions. On the other hand, character-level spans also align better with expert auditing and MEDA’s minimal edit constraint checking, since experts operate on surface text rather than tokenizer-specific subwords.

11.2.0.2 Implementation Details.

To verify that our conclusions do not depend on character granularity, we implement and evaluate two implementation variants:

(i)Character-level(default): Applying recursive LCS on character sequences to obtain shared and edited char parts, then map spans to token indices via the model tokenizer to construct masks/weights.

(ii)Token-level: Applying recursive LCS directly on token sequences to obtain shared and edited token parts. Masking and token re-weighting are then constructed directly on token indices, without any character-level intermediate step.

11.2.0.3 Experiments Results.

We conducted experiments on both the sycophancy benchmark and the ASD-iLLM test set using the two implementation variants, with the experimental settings consistent with those described in Sec. 4. Tab. 7 presents the performance of the two implementations on two test sets. Overall, the token-level variant yields comparable performance to the default character-level implementation. The NSR score difference is less than one point, while factual error, evasive response, and retention metrics on ASD-iLLM remain at similar levels.

This suggests that the improvements of the proposed method do not hinge on Chinese character granularity, indicating its potential for tokenization structure robustness.

11.3 EOS Termination Signal under Greedy Continuation↩︎

To further evaluate the model’s performance in achieving stable termination, we track the EOS probability trajectory after the reference completion. Given a test sample \((x, y^*)\), we compute \(p_0 = p(\mathrm{EOS}\mid x, y^*)\) under teacher forcing at the reference end (offset=\(0\)). Then we continue generation greedily for \(r{=}16\) steps to obtain \(\hat{y}_{1:r}\) to calculate \(p_t\): \[p_t = p(\mathrm{EOS}\mid x, y^*, \hat{y}_{1:t}),\quad t=1,\dots,r\]

From this, we can obtain the model’s predicted probability of EOS \(\{p_t\}_{t=0}^{r}\) at each decoding step after the reference end.

Fig. 14 shows the median \(p(\mathrm{EOS})\) trajectory with interquartile range (IQR) across the sycophancy benchmark. Compared with SFT, DPO exhibits consistently lower \(p(\mathrm{EOS})\) after the reference end, indicating a weakened termination signal. In contrast, TD-DPO substantially restores the EOS trajectory toward the SFT baseline, suggesting better preservation of stopping and formatting behaviors.

To quantify how many cases fail to develop a strong termination signal, we calculate:

\[S_{\tau}(t) = \Pr \left(\max_{0 \leq j \leq t} p_j \leq \tau\right), t=0,\dots,r\]

Where \(\Pr(\cdot)\) denotes the empirical probability over the test set and \(\tau\) is a fixed threshold. \(S_{\tau}(t)\) measures the fraction of cases whose EOS probability never exceeds \(\tau\) up to step \(t\). Therefore, a higher curve indicates more cases that remain unable to stop, which is associated with degradation and pattern collapse.

We focus on the trends under two thresholds on sycophancy benchmark as shown in Fig. 15: (i) Under the more lenient threshold \(\tau{=}0.3\), the SFT and TD-DPO curves nearly overlap throughout the continuation window, indicating that both models develop a moderate EOS signal in a similar proportion of cases, while DPO remains consistently above them, suggesting a larger tail of cases that stay unable to stop. (ii) Under the stricter threshold \(\tau{=}0.5\), TD-DPO is slightly higher than SFT but still significantly lower than DPO across steps, indicating that TD-DPO largely restores strong termination behavior relative to DPO. These results provide additional evidence that DPO tends to weaken the EOS-based stopping mechanism after the reference completion, whereas TD-DPO substantially mitigates this effect and better preserves the termination behavior learned before alignment.

11.4 Asymmetric Design Variants↩︎

11.4.0.1 Design Comparison.

TD-DPO decomposes each preference pair \((x, y_w, y_l)\) into shared and edited parts under the minimal edit principle. During loss computation, token-level re-weighting is applied based on the token types, which serves to amplify differences in the edited parts while suppressing contextual noise in the shared parts.

We can also analyze the design concept of TD-DPO from a semantic perspective: the degree of semantic distance or significance is implicitly controlled by \(\alpha_\text{shared}\) and \(\alpha_\text{diff}\): error affirmation and instruction shift are included in the different parts, while stylistic preferences are contained in the shared parts. Therefore, TD-DPO amplifies the weights of the different parts to mitigate sycophancy, while suppressing updates to the shared parts to reduce noise and preserve the clinical dialogue style learned during SFT. Theoretically, these two dimensions are orthogonal and can be controlled separately.

A key design choice of TD-DPO is where this token-level re-weighting is applied in the preference objective. In our default implementation, re-weighting is applied only to the rejected response. Intuitively, this keeps updates on the chosen response relatively mild, helping preserve the capability learned during SFT while contrastively pushing down the likelihood of undesired tokens in the edited parts of the rejected response, thereby mitigating sycophancy without sacrificing the model’s intervention capability.

To validate that this asymmetry design is beneficial, we additionally consider two variants: chosen-only and symmetric.

• Rejected-only (default). Apply token re-weighting only to the rejected response \(y_l\), the chosen response \(y_w\) uses the standard token log-probabilities. This asymmetric design aims to avoid over-updating well-formed tokens in \(y_w\) and to penalize undesirable behaviors in \(y_l\).

• Chosen-only. Apply token re-weighting only to the chosen response \(y_w\), leaving \(y_l\) unweighted. Intuitively, this variant amplifies updates toward the chosen response, but may also induce larger drift since \(y_w\) typically contains many shared tokens and stable language patterns.

• Symmetric. Apply the same token re-weighting to both \(y_w\) and \(y_l\). While seemingly balanced, this variant can increase update magnitude on shared tokens on both sides, potentially aggravating drift or degeneration in minimal edit settings.

11.4.0.2 Implementation Note.

Our code supports setting separate \(\alpha_{\text{shared}}\) and \(\alpha_{\text{diff}}\) for the chosen and rejected responses. In this ablation study, we use the same values (\(\alpha_{\text{shared}}=0.5\), \(\alpha_{\text{diff}}=2\)) for all weighted settings, and keep all other training settings identical to the main experiments.

11.4.0.3 Experiments Results.

Tab. 8 reports results on both the Sycophancy Benchmark and ASD-iLLM Test Set. The rejected-only design yields the best overall trade-off: it achieves the highest NSR and lowest factual and evasive errors, while exhibiting lower degeneration signals (shorter outputs and lower repetition) than chosen-only and symmetric variants. In contrast, chosen-only shows substantial degradation with much longer outputs and higher repetition, suggesting that aggressively upweighting tokens on the chosen branch can amplify drift and harm sycophancy mitigation.

11.4.0.4 Discussion: Why rejected-only design helps.

A key characteristic of our dataset is its minimal-edit structure, where chosen and rejected responses share extensive spans but differ in a few localized edits. In such regimes, token-level re-weighting is critical: chosen-only updates are better suited when the goal is primarily to imitate reliably good responses and negative examples are noisy or uninformative, while symmetric updates can be beneficial when both chosen and rejected provide comparably informative supervision and differences are not localized. In our setting, however, strong updates on the chosen branch (chosen-only) or on both branches (symmetric) tend to amplify gradients on broadly shared, already well-formed tokens, increasing drift and language degeneration (e.g., longer or repetitive outputs), and chosen-only may also weaken contrastive signals of sycophancy.

This hypothesis is further supported by the token-drift analysis in Fig. 16, which reports the magnitude of \(\Delta{\text{NLL}}\) relative to the SFT policy on a fixed reference response, computed separately on different and shared tokens. Across the three variants, the drift on different tokens remains relatively small and comparable, indicating that all designs make only mild updates to the edited regions. However, the shared token drift exhibits a clearer separation: chosen-only shows noticeably larger deviation from SFT, symmetric falls in between, and rejected-only stays closest to SFT. This aligns with our prior mechanistic analysis and helps explain the empirical results: larger drift on shared tokens correlates with worse overall performance and stronger degeneration signals, whereas keeping shared token predictions anchored (as in rejected-only) yields a better trade-off between sycophancy mitigation and capability retention.

11.5 Degradation and Retention Result↩︎

11.5.0.1 Complete Experiment Results.

Tab. 9 reports quantitative indicators for response degradation on the sycophancy benchmark and language retention on the ASD-iLLM test set. On the sycophancy benchmark, we measure degradation via response length statistics (Tokens, Len\(\ge\)​30) and repetition (R-4). On ASD-iLLM test set, we additionally report BERTScore (BS.) and NLL to assess retention of pre-alignment response quality and distribution. Overall, TD-DPO achieves the best trade-off: it produces short and less over-generated responses on the sycophancy benchmark (lowest Tokens and Len\(\ge\)​30), while keeping repetition low, and it preserves in-distribution behavior on ASD-iLLM test set without the large length inflation and repetition observed in some baselines (e.g., DPO). These results support our claim that token-difference weighting mitigates sycophancy while minimizing pattern collapse and preserving well-formed dialogue behaviors acquired before.

Additionally, we report the results of different methods on the two benchmarks under the w/o Minimal Edit setup. On the sycophancy benchmark, applying minimal edit constraint typically decreases response length and repetition for several methods (e.g., DPO: Tokens 19.20 to 17.76 and R-4 4.57% to 4.13%; SPPO: Tokens 10.61 to 9.80 and R-4 1.82% to 0.87%; ORPO: R-4 0.74% to 0.50%). On ASD-iLLM test set, it often improves retention-related indicators by keeping generations closer to the in-distribution style (e.g., SimPO reduces length and repetition, Tokens 10.41 to 9.14 and R-4 0.44% to 0.31% R). These trends are consistent with the intuition that constraining \((y_w,y_l)\) to differ by a minimal edit suppresses background discrepancies and thus reduces gradient interference on shared tokens.

We also observe that the effect of the minimal edit constraint is not uniformly monotonic across all objectives. For instance, OTPO exhibits a clear length and repetition increase when it is enabled (Tokens 11.16 to 15.95 and R-4 0.59% to 1.93% on the sycophancy benchmark; similarly on ASD-iLLM), suggesting that the interaction between the preference objective and localized supervision can be method-dependent. TD-DPO shows consistent improvements on both benchmarks with minimal edit constraint: on the sycophancy benchmark, it achieves the best length control (Tokens 8.95 to 7.75; Len\(\ge\)​30 1.65% to 0.55%) with low repetition (0.45% to 0.33%), and on the ASD-iLLM test set, it also reduces over-generation and repetition (Tokens 11.57 to 10.31; R-4 1.40% to 0.94%) while maintaining comparable retention scores. This supports the intended mechanism that localized preference construction complements token-difference weighting by concentrating updates on preference-critical errors and avoiding unnecessary shifts in response patterns.

11.5.0.2 More Case Study.

Fig. 17 presents a test example from the sycophancy benchmark. The child’s response contains a factual error ("Excavators have wings"), and the misaligned SFT model mistakenly confirms the child’s error and continues to ask follow-up questions. This could potentially lead to the child forming incorrect cognitive understandings. In contrast, the DPO method corrects the error but exhibits repetitive behavior (repeat "What type of machine is it?" many times). The TD-DPO, however, not only corrects the error but also avoids pattern collapse, demonstrating its superiority to mitigate sycophancy by focusing on key differences while suppressing background noise, thereby better preserving the linguistic patterns.

11.6 Hyperparameters Ablation: Mask DPO↩︎

To rigorously validate that shared tokens indeed provide effective contextual anchors, we implement a strict masked DPO variant.

11.6.0.1 Implementation Details.

We modify the implementation to allow the token-level weights \(\alpha_{\text{shared}}\) and \(\alpha_{\text{diff}}\) to be applied to both the chosen and rejected responses. Concretely, we compute the token masks between \((y_w, y_l)\) as in TD-DPO, and then apply the same weighting mechanism to the token log-probabilities of both sequences. We then construct a strict hard masking baseline by setting \(\alpha_{\text{shared}}^{w} = 0\), \(\alpha_{\text{shared}}^{l} = 0\), so that shared tokens on both the chosen and rejected sides receive zero weight in the preference objective, resulting in strictly zeroed gradient contributions from shared tokens within this term. All other settings are kept identical to the main experiments.

11.6.0.2 Experiments Results.

Tab. 10 compares TD-DPO with the strict masked DPO baseline on both the Sycophancy Benchmark and ASD-iLLM Test Set. Strictly halting shared-token updates leads to the performance drop (NSR decreases from 92.31 to 87.91) and increases degeneration risks, as reflected by longer outputs and higher repetition. This further supports the claim that shared tokens are necessary contextual anchors.

11.7 MEDA Strategy without Minimal Edit Constraint↩︎

11.7.0.1 Sycophancy Training Set without Minimal Edit Constraint.

To validate the effectiveness of the MEDA strategy, we constructed a sycophancy training set based on non-minimal edits and conducted the same experiments as described in Sec. 5.1. Specifically, to investigate the importance of minimal edits in the MEDA process, we used the training set generated through the MEDA strategy as the baseline. During the final step of constructing preference pairs, we removed all fields in the prompts containing "minimal edit" descriptions, allowing GPT-4.1 to freely generate chosen/rejected responses. This resulted in a sycophancy training set without the minimal edit constraint.

Tab. 11 summarizes the resulting data distribution. Removing the minimal edit constraint substantially changes the shape of the preference pairs, while leaving the contexts unchanged. Firstly, responses become notably longer: on average, the chosen length increases from 16.26 to 22.75 characters (+6.49), and the rejected length increases from 12.78 to 15.16 tokens (+2.38). Secondly, the similarity drops sharply from 37.11% to 22.38% (with similar trends for both factual error and evasive response). These statistics indicate that without minimal edit prompting, GPT-4.1 tends to rewrite responses more freely, producing preference pairs that differ in many background tokens rather than in a small set of critical edits.

11.7.0.2 Complete ablation results for the impact of the minimal edit constraint in the MEDA strategy.

Tab. 12 provides a complete ablation result on the MEDA strategy by removing the minimal edit constraint when constructing preference training pairs (w/o Minimal Edit). Overall, enforcing minimal edits consistently improves non-sycophantic alignment across all baselines: DPO (+2.74), SimPO (+1.65), ConfPO (+3.30), OTPO (+1.10), CPO (+4.95), ORPO (+2.20), and SPPO (+2.20) all achieve higher NSR under MEDA. These gains support our motivation that sycophancy-related supervision is typically localized: when preference pairs are constrained to differ only in a small number of critical tokens, models become more focused on the causal edits responsible for sycophancy.

Notably, TD-DPO benefits the most from the minimal edit constraint (+4.40). By concentrating learning on preference-critical edits and suppressing gradients on shared background tokens, TD-DPO can more effectively mitigate sycophancy without inducing unnecessary pattern updates. In summary, this ablation indicates that minimal edit constraint is an important ingredient of MEDA strategy for producing cleaner, more targeted preference supervision, and it complements TD-DPO particularly well.

11.8 Sycophancy Mitigation on SAA Dataset↩︎

Due to the lack of publicly available sycophancy benchmarks in the autism domain, we evaluate our method on the SAA dataset, which contains 3,000 samples. Each sample includes an instruction with a potentially correct or incorrect “user suggestion” together with a chosen response and a rejected response. The dataset is designed to teach models to exhibit two complementary behaviors: accepting and adopting the user suggestion when it is correct, and rejecting incorrect ones while providing the correct answer, thereby mitigating sycophancy.

11.8.0.1 Implementation Details.

Initially, we processed the dataset as follows: we excluded samples from the instruction that did not contain options, followed by verification of both chosen and rejected responses to ensure they included the correct options. After preprocessing, we obtained 2,539 samples. We then divided the SAA dataset into training and testing sets in a 9:1 ratio, while maintaining all other training hyperparameters consistent with those utilized in the original study.

Then, we use Qwen2.5-7B-Instruct as the base model and fine-tune it with different alignment methods on the training set, evaluating on the test set. Due to instances where the model outputs give the correct options, successfully avoiding sycophancy, yet not adhering to the exact format of the chosen responses, we utilized GPT-4o to extract the options that correspond to the model outputs and the chosen responses, with the corresponding prompt shown in Fig. 18. We subsequently calculated the accuracy and F1 score to assess the model’s performance.

11.8.0.2 Experiments Results.

As shown in Tab. 13, TD-DPO achieves the best overall performance on the SAA dataset, obtaining the highest ACC of 73.46% and F1 score of 74.43%, outperforming all other baselines. In particular, TD-DPO improves over the strongest competing method, SPPO, by 1.92% in ACC and 2.59% in F1 score, demonstrating that our method generalizes well to this sycophancy mitigation setting. Moreover, TD-DPO maintains a reasonably high Parse Rate of 95.77%, indicating that its performance gain is not achieved at the cost of significantly degraded output format compliance.

However, the improvement is relatively modest, which can be attributed to two factors. First, TD-DPO’s minimal-edit advantage becomes less pronounced in this option selection scenario, as there are fewer fine-grained token overlaps from which to derive additional discriminative gradients. Second, the SAA preference pairs are mainly distinguished by different options rather than subtle semantic edits. Taken together, the task is closer to coarse-grained answer selection than to fine-grained preference disambiguation, leaving less room for TD-DPO to exploit token-level advantages. Overall, these results highlight TD-DPO’s ability to generalize its advantages to other preference benchmarks, further supporting its effectiveness in mitigating sycophantic behavior.

11.9 MEDA Generator Ablation Study↩︎

11.9.0.1 Motivation.

To verify that the MEDA strategy and TD-DPO algorithm do not depend on a specific model (e.g., GPT-4.1), we replicate MEDA using two additional generators: DeepSeek-V3 and the open-source Qwen2.5-32B-Instruct.

11.9.0.2 Implementation Details.

The MEDA pipeline was maintained without modification, except for the final step in which the model is required to generate chosen or rejected responses in accordance with the minimal edit principle. In this step, DeepSeek-V3 and Qwen2.5-32B-Instruct were substituted for the original model GPT-4.1.

Tab. 14 summarizes the statistics of training datasets produced by different generators. Overall, these datasets exhibit comparable surface-level minimal edit characteristics (length and overlap), suggesting that MEDA strategy can be instantiated with different LLMs, including open-source models, under the same prompting and constraints.

To verify that preference pairs generated by different models can also effectively mitigate sycophancy, we fine-tune the same SFT initialization (ASD-iLLM based on Qwen2.5-7B-Instruct) with TD-DPO on each generated preference dataset, using the same training setup as described in the main paper (Sec. 4). We then evaluate on both the Sycophancy Benchmark and ASD-iLLM Test Set.

11.9.0.3 Experiments Results.

Tab. 15 reports the downstream performance after TD-DPO training on each generated dataset. While absolute performance varies across generators, TD-DPO consistently mitigates sycophancy while maintaining the intervention capability at a similar level, indicating that the proposed method is not tied to a specific proprietary generator or a single rewriting style.

11.9.0.4 Conclusion.

These results provide two takeaways. Firstly, MEDA can be reproduced with non-proprietary generators, supporting the feasibility of open-source replication. Secondly, although generator choice affects the absolute performance, the overall conclusion remains consistent: TD-DPO effectively mitigates sycophancy while maintaining intervention capability learned during SFT.

12 Implementation Details↩︎

12.1 ASD-iLLM-8K Dataset↩︎

The ASD-iLLM-8K dataset [7] is a clinical dialogue intervention dataset for autistic children, consisting of a mix of real and synthetic conversations. The training set includes 7,935 multi-turn dialogue samples, while the test set contains 100 real multi-turn dialogue samples, for a total of 1,989 dialogue turns.

12.2 Training Setting↩︎

We conducted experiments using 8 GTX 4090 GPUs with the ms-swift [52] framework. To ensure a fair comparison, all experiments were conducted using the same hyperparameters listed in the Tab. 16.

12.3 Training Stability of TD-DPO↩︎

12.3.0.1 Loss and accuracy.

Fig. 19 reports the training dynamics of TD-DPO on three backbone models. As shown in the left part, the training loss decreases smoothly during optimization without exhibiting abrupt oscillations or divergence, indicating that TD-DPO provides a stable optimization trajectory. Meanwhile, the right part shows that the training reward accuracy increases steadily over time, suggesting that the aligned policy consistently improves its capability for sycophancy mitigation throughout training.

Notably, the overall trends remain highly consistent across different backbones, despite differences in model scale and architecture. This implies that TD-DPO is not sensitive to the choice of backbone and can be reliably applied to heterogeneous LLM families. The observed stability further demonstrates that the proposed alignment method does not introduce training collapse or severe variance, supporting its robustness as a general preference optimization approach.

12.3.0.2 Gradient allocation.

To further analyze the allocation of gradient updates across different types of tokens in TD-DPO, we calculated the proportion of gradient quality associated with shared and different tokens within chosen and rejected responses during training. Each bar in the Fig. 20 is normalized to 100%. It can be observed that the rejected responses consistently allocate a greater proportion of gradient quality to different tokens (nearly 80%) at all training stages, while the allocation for chosen responses is relatively more balanced (about 60% for different tokens). This phenomenon indicates that TD-DPO employs a differentiated gradient weighting mechanism, focusing the optimization process on the disparity between chosen and rejected responses and reducing ineffective updates to shared content.

13 Metrics Details↩︎

13.1 Automatic Metrics↩︎

We use the Distinct-2 (D-2) metric to measure the diversity of the model’s output \(t\). The formula for calculating Distinct-n is as follows:

\[Distinct_n(t)=\frac{Unique_n(t)}{Total_n(t)}\]

Where \(Unique_n(t)\) refers to the number of unique n-grams in \(t\) and \(Total_n(t)\) denotes the total number of n-grams in \(t\).

We use the 4-gram repetition rate (R-4) to measure the proportion of repeated n-grams for model’s output \(t\):

\[Rep_n(t) = 1-\frac{Unique_n(t)}{Total_n(t)}\]

13.2 Dynamic Dialogue Evaluation Method based on Role-Playing↩︎

ASD-iLLM [7] simulates autistic children by randomly selecting from four predefined response intents: no response, correct response, erroneous response, and repetitive response. The randomness of these intents serves as a stress test for the model’s intervention capabilities, effectively mimicking the social impairment and logical inconsistencies that may arise in real scenarios.

However, this distortion-based simulation approach cannot effectively replicate the linguistic characteristics of autistic children, making it inadequate for testing the doctor model’s generalization ability in different scenarios. Therefore, inspired by [53], we propose a dynamic dialogue evaluation method based on role-playing, aiming to more accurately simulate the linguistic behavior of autistic children and better assess the doctor’s intervention capabilities.

13.2.0.1 Methodology.

The workflow of this method is divided into three steps, as shown in Fig. 21. Firstly, we model the linguistic behavior of autistic children by analyzing their dialogue history in the database. Based on this, we use GPT-4o to generate role cards through the prompt template shown in Fig. 22. The output will be incorporated as part of the system prompt in subsequent intervention dialogue interactions to simulate the behavior of autistic children.

Next, during topic-based dialogue execution, we dynamically extract the three most relevant dialogues from the child’s dialogue history based on the current topic as few-shot examples. These examples are used as part of the prompt, enabling the model to better emulate the linguistic behavior of autistic children within specific topic contexts through in-contextual learning. Specifically, we use the Qwen3-Embedding-0.6B [54] to extract the features of one child’s multiple dialogue histories. Then, cosine similarity is calculated to determine the relevance of these dialogues to the current topic, and the three most relevant dialogues are recalled as few-shot examples.

Secondly, using the prompt template shown in Fig. 23, the GPT-4o is guided to imitate the linguistic behavior of autistic children in alternating N-turn topic dialogues with the doctor model, resulting in dynamic intervention dialogue records. Specifically, before each dialogue, we randomly select an autistic child from the database and use GPT-4o to simulate the child’s linguistic behavior to interact with the doctor model.

Thirdly, the collected simulated multi-turn dialogues are evaluated by human experts to comprehensively assess the doctor model’s intervention capabilities in dynamic simulated intervention scenarios. Inspired by [7], [55]–[58], we designed a scoring rule comprising three dimensions: Professionalism, Linguistic, Safety, across 8 categories, as shown in Tab. 17.

Specifically, human experts evaluate each dialogue turn based on the scoring rule, breaking multi-turn dialogues into individual turns in a step-by-step manner to assess the doctor’s professionalism, linguistic, and safety in each interaction. They assigned scores between 0 and 4 as follows:

0: None of the doctor’s responses in the dialogue meet the requirements.

1: A small portion of the doctor’s responses meet the requirements (<30%).

2: Some of the doctor’s responses meet the requirements (<60%).

3: Most of the doctor’s responses meet the requirements.

4: All of the doctor’s responses meet the requirements.

13.2.0.2 Ablation Experiment Result.

To verify that the simulation method combining role cards with few-shot examples achieves the best results, we conducted the following experiments on the ASD-iLLM test set: We included autistic children with more than 10 dialogue records in the database, as too few records cannot adequately reflect their linguistic characteristics. The dataset was split 8:2 into a training set and a test set, with the training set used to construct role cards and few-shot examples, and the test set used to evaluate the degree of simulation. The testing method is next sentence prediction (NSP): given a context \(x=(t,s,h_{1:t})\) where t refers to the current topc, s denotes the system prompt as shown in Fig. 23, and \(h_{1:t}\) represents the dialogue history from rounds 1 to t.

GPT-4o generates the the child’s following response and then compared with the reference child’s response using n-gram metrics (BLEU [59], GLEU [60], ROUGE [61], METEOR [62]) and semantic similarity measures (BertScore [48], BGE Simalirty [63]) to evaluate its imitation performance. BGE similarity is computed by inner product to evaluate the semantic similarity between two sentence embeddings generated by the BGE-m3 model.

Furthermore, to evaluate the GPT-4o Agent’s simulation, we invited a clinician with over three years of experience to score the authenticity of the child responses (i.e., whether the child’s output aligns with the characteristics of autism). Other settings remain consistent with the human evaluation mentioned before, scoring from 0 to 4.

Specifically, we conducted experiments on 29 multi-turn dialogue records from 8 autistic children, comprising a total of 667 dialogue turns. To compare performance, we performed the following ablation experiments:

1. Direct: Removed both role cards and few-shot examples from the prompt template, as shown in Fig. 23.

2. Only Card: Retained only the role card section within the prompt template.

3. Few-shot(Random): Randomly selected dialogue history samples without performing topic similarity calculation.

4. Few-shot(Sim): Used only a few-shot examples that are similar to the current topic in the prompt template.

The experimental results are shown in Tab. 18. It can be observed that the combination of role cards and topic-similarity-based few-shot examples achieves the best performance across most metrics. The BLEU, GLEU, ROUGE-1 (R-1), ROUGE-L (R-L), METEOR (MET.), BertScore (BS.), and BGE Similarity (BGE) metrics are 22%, 32.48%, 39.51%, 38.75%, 31.12%, 72.92%, and 72.10%, respectively. The average score is 44.13%, which is 0.18% higher than the suboptimal Few-shot(Sim) and 4.81% higher than Direct.

Furthermore, expert evaluations show that the authenticity score under the Card+Few-shot(Sim) setting reaches 2.96, outperforming all configurations except Few-shot(Sim). This suggests that in most simulated dialogues, its outputs align with autistic characteristics, demonstrating that the simulator can realistically emulate autistic children.

Overall, the experimental results indicate that combining role cards with few-shot examples can better simulate the linguistic behavior of autistic children, thus enabling a more realistic evaluation of the doctor model’s dialogue intervention capabilities during interactions.

13.2.0.3 Details for Human Experts.

Tab. 19 provides detailed information on three invited experts participating in the human evaluation, each possessing over four years of experience in autism therapy. Their extensive expertise in intervention ensures their qualification for the professional evaluation task. Each expert is compensated with a labor fee of 100 yuan per hour based on their working hours, which exceeds the standard salary rate.

References↩︎

---

1. https://github.com/Shuzhong-Lai/TD-DPO↩︎

2. Corresponding author.↩︎

3. https://github.com/ntunlplab/saa-dataset↩︎