1 Introduction↩︎

Conversational Speech Synthesis (CSS) aims to leverage multimodal dialogue history to synthesize target speech with contextually appropriate affective prosody in User-Agent Interactions (UAI) [1]–[3]. In UAI systems, the ability of the agent to generate speech that is appropriate for the dialogue context and conveys human-like emotional expression is vital for enhancing user experience. As UAI becomes increasingly prevalent, CSS has become a crucial component of intelligent interactive systems [4]–[6] and plays an important role in embodied intelligence applications such as virtual assistants, voice agents, intelligent robots, smart home devices, and in-vehicle infotainment systems.

Previous CSS studies [1]–[3], [7]–[16] mainly rely on a set of predefined emotion labels (including anger, disgust, fear, happiness, sadness, surprise, and neutral) derived from Ekman’s basic emotion theory [17], and directly model the multi-turn dialogue history for context modeling. For instance, ECSS [2] encodes multimodal dialogue history into a heterogeneous graph to improve the emotional expressiveness of the synthesized target speech. GPT-Talker [15] converts multimodal information [18]–[20] in the dialogue history into discrete token sequences containing semantic and style information, and utilizes an autoregressive model to predict target acoustic tokens for synthesizing emotional speech. Recently, Chain-Talker [3] further generates empathetic captions based on these predefined emotion labels, adopting a three-stage framework that includes emotion understanding, semantic understanding, and empathy rendering to enhance the empathetic expression of the target speech.

Despite the progress of these advanced CSS methods in speech quality and expressiveness, they still face the following limitations, as illustrated in Fig. 1 (top): 1) Limited Emotion Label Space: In real-world dialogues, emotional expressions are richer and broader [21], [22]. For example, emotions related to “happiness” include excitement, pleasure, and tearful joy. Such a broad emotional space is more capable of reflecting humans’ authentic emotions (AuEmo). However, conventional predefined emotion labels cannot distinguish these richer and more diverse emotional states, limiting a model’s ability to learn authentic emotional expressions. 2) Redundant Multimodal Context Tokens: As dialogue turns increase, multimodal token sequences become increasingly long, inevitably introducing a large amount of redundant information [3], [15]. These redundant tokens introduce significant noise into contextual modeling, severely interfering with emotion understanding and high-quality speech generation for the target utterance [23].

To address these issues, we propose AuEmoChat, a CSS framework for authentic emotion understanding and rendering. Specifically, we first develop AuEmoCodec, which learns a discrete authentic emotion token space from large-scale emotional speech. AuEmoCodec quantizes emotional speech into an AuEmo token in an emotion codebook via Finite Scalar Quantization (FSQ) [24]. The AuEmo token is then used to reconstruct the perceived emotion scores of the source speech for seven basic emotion axes, enabling the model to capture more authentic and fine-grained emotional states. Second, we design an AuEmo-guided token merging (AuEmoToMe) algorithm that merges redundant text and speech tokens in multimodal dialogue history. This design reduces the interference caused by redundant tokens during emotion modeling while preserving emotion-relevant contextual information. As a result, the model can better infer the target AuEmo token and speech tokens for the target utterance. Finally, we propose Authentic Emotion Flow Matching, which generates emotionally expressive speech conditioned on the merged dialogue context and the predicted AuEmo token. In addition, this mechanism incorporates AuEmo classifier guidance during the flow matching process, enabling the synthesized speech to better align with the target authentic emotion. In summary, the main contributions of this work are as follows:

• We propose AuEmoChat, the first CSS framework dedicated to authentic emotion understanding and rendering.

• We develop AuEmoCodec, which leverages finite scalar quantization to learn an authentic emotion token space from large-scale emotional speech, thereby providing more authentic emotional representations.

• We design AuEmoToMe, an authentic-emotion-guided token merging algorithm that merges redundant multimodal context tokens while preserving emotion-relevant context. Furthermore, we propose Authentic Emotion Flow Matching to improve context-consistent emotional speech rendering.

• Extensive experiments on the NCSSD-EmCap dataset demonstrate that AuEmoChat significantly outperforms advanced baselines in both emotional expressiveness and speech quality.

2 Related Works↩︎

2.1 Speech Neural Codec↩︎

Speech neural codecs convert continuous speech waveforms into discrete tokens, providing the foundation for token-based speech modeling and large language model (LLM)-based audio processing [25], [26]. SpeechTokenizer [27] uses an encoder-decoder architecture with residual vector quantization (RVQ) to model semantic and acoustic information in speech jointly. SemantiCodec [26] adopts a dual-encoder architecture to extract ultra-low-bitrate and semantically rich audio tokens for speech, general sounds, and music. FACodec [28] decomposes speech into content, prosody, timbre, and acoustic-detail subspaces through factorized vector quantization (FVQ), enabling separate generation of different speech attributes. TF-Codec [29] integrates latent-domain predictive coding and learnable time-frequency compression into VQ for low-latency speech coding. CosyVoice [30] extracts semantically rich speech tokens with an Automatic Speech Recognition (ASR)-based self-supervised method, while CosyVoice2 [31] further improves token representation by introducing FSQ. Our AuEmoCodec has the following key distinctions: 1) It quantizes emotional speech into fixed AuEmo tokens rather than semantic or acoustic-related tokens. 2) Unlike conventional codecs that rely on ASR supervision to learn semantic information or reconstruct the original speech waveform, AuEmoCodec is trained to reconstruct quantized AuEmo tokens into a richer emotion perception space by modeling perceived scores across seven basic emotion axes, allowing the tokens to capture authentic human-like emotion.

2.2 Emotion Label Space↩︎

The emotion label space is commonly used to quantify emotional representations in speech and serves as a control condition for emotional speech synthesis [2], [16]. Traditional CSS methods learn emotions based on Ekman’s basic emotion theory [17], which categorizes emotions into six basic types: anger, disgust, fear, happiness, sadness, and surprise. Emotions that cannot be clearly categorized are usually labeled as neutral. This type of model aligns better with people’s intuitive understanding of emotions in daily life. However, its label space is limited and cannot fully cover the rich and subtle emotional states in real dialogues. Plutchik [32] points out that humans can express about 34,000 distinct emotions, which further indicates that traditional discrete models with limited categories cannot fully represent the authentic emotion space. To address the limitation of label space, Lian et al. [33] introduce open-vocabulary emotion recognition, which expands emotion categories through open-vocabulary emotion labels. However, although it increases the number of categories, it also introduces many semantically similar or synonymous labels, failing to effectively disentangle the emotion representation space and thus increasing modeling difficulty. To construct a more authentic emotion token space, we use AuEmoCodec to learn AuEmo tokens from large-scale emotional speech. Specifically: 1) We build a discrete emotion codebook, where each token represents a distinct authentic emotional state, providing a finer-grained representation than traditional emotion categories. 2) Instead of relying on predefined emotion labels, AuEmoCodec learns the AuEmo token representation directly from large-scale emotional speech.

2.3 Token Merging↩︎

Token merging improves model efficiency and performance by combining redundant tokens in multimodal sequences [34]. For instance, ToMe [34] gradually merges similar tokens within the Transformer, enhancing inference speed while largely preserving performance. Token Merger [35] identifies meta tokens representing meaningful image cues and merges similar tokens adaptively, better preserving semantic context and generalizing across vision tasks. FastAST [36] applies token merging to audio spectrograms, improving inference speed while maintaining recognition accuracy. In multimodal dialogue history, speech tokens typically have high bitrate, and long conversations further increase sequence length, making emotional information highly redundant. Redundant tokens dilute model attention to key emotional cues, hindering accurate emotion understanding. Inspired by these works, we propose AuEmoToMe in AuEmoChat: 1) It is the first attempt to introduce token merging into AR-based CSS for modeling multimodal dialogue context. 2) It computes intra-modal similarity among text tokens and speech tokens in the multimodal dialogue history and merges redundant tokens. 3) To preserve emotional information, AuEmoToMe adopts an AuEmo-guided strategy that prioritizes merging redundant tokens while minimally affecting emotion-related key tokens.

3 Task Definition↩︎

A dialogue can be defined as a sequence of alternating interactions between the user and the agent, denoted as \(\{U_1, U_2, \ldots, U_{\mathcal{J}}, U_{\mathcal{N}}\}\), where \(\{U_1, U_2, \ldots, U_{\mathcal{J}}\}\) denotes the dialogue history, and \(U_{\mathcal{N}}\) denotes the target utterance to be processed by the agent. For each historical utterance \(U_i\), it contains multiple modalities \(\{\text{P}_i, \text{T}_i, \text{S}_i\}\), which represent the speaker, text, and speech of the \(i\)-th turn, respectively. For the target utterance \(U_{\mathcal{N}}\), the speaker \(\text{P}_{\mathcal{N}}\) and text \(\text{T}_{\mathcal{N}}\) are given, and the goal is to generate the corresponding speech \(\text{S}_{\mathcal{N}}\). CSS based on authentic emotion understanding and rendering needs to address the following points: 1) How to more accurately represent the authentic emotion of each utterance in the dialogue. 2) How to effectively infer the authentic emotion token and speech token of the target utterance from a long multimodal dialogue history. 3) How to synthesize speech with authentic emotion expression that is consistent with the current dialogue context.

4 Methodology: AuEmoChat↩︎

Fig. 2 (left) illustrates the overall architecture of AuEmoChat. The Multimodal Dialogue Tokenization module extracts text, speech, speaker, and AuEmo tokens from the input dialogue. The AuEmoToMe-based Authentic Emotion Understanding module incorporates the proposed AuEmoToMe into the LLM to merge redundant text and speech tokens in the dialogue history, based on which the LLM predicts the AuEmo token and speech tokens of the target utterance. The Merged Context-Aware Authentic Emotion Rendering module conditions on the merged history tokens, the predicted AuEmo token, and the speech tokens to synthesize authentic emotional speech that is consistent with the context through an authentic emotion flow matching mechanism. In addition, Fig. 2 (right) illustrates the training process of AuEmoCodec, which will be described in detail at the end of this section.

4.1 Multimodal Dialogue Tokenization↩︎

In this section, we introduce how multimodal dialogue data are processed to extract the corresponding multimodal tokens. The implementation details are as follows.

Text Tokenizer. We adopt a BPE-based Text Tokenizer [37] to tokenize the dialogue text. Specifically, as shown in Eq. 1 , the input dialogue text sequence \({\text{T}_{1 \rightarrow \mathcal{N}}}\) is processed by the Text Tokenizer to obtain the corresponding text token embeddings: \[f^{T}_{1 \rightarrow \mathcal{N}, 1 \rightarrow max_{t}} = \text{Text Tokenizer}(\text{T}_{1 \rightarrow \mathcal{N}}) \label{eq:1}\tag{1}\] where “\(1 \rightarrow \mathcal{N}\)” denotes the dialogue turn indices, and “\(1 \rightarrow max_{t}\)” denotes the text token indices within each utterance.

Speaker Embedding. We use Speaker Embedding to encode the discrete identities of different speakers. Specifically, the speaker sequence \({\text{P}_{1 \rightarrow \mathcal{N}}}\) of the current dialogue is processed by the Speaker Embedding to obtain the corresponding speaker token embeddings: \[f^{P}_{1 \rightarrow \mathcal{N}} = \text{Speaker Embedding}(\text{P}_{1 \rightarrow \mathcal{N}}) \label{eq:2}\tag{2}\]

Speech Tokenizer. We adopt the Speech Tokenizer proposed in CosyVoice2 [31] to discretize the speech signals in the dialogue history. Specifically, the dialogue history speech sequence \({\text{S}_{1 \rightarrow \mathcal{J}}}\) is processed by the Speech Tokenizer to obtain the corresponding speech token embeddings: \[f^{S}_{1 \rightarrow \mathcal{J}, 1 \rightarrow max_{s}} = \text{Speech Tokenizer}(\text{S}_{1 \rightarrow \mathcal{J}}) \label{eq:3}\tag{3}\] where “\(1 \rightarrow max_{s}\)” denotes the speech token indices within each utterance.

AuEmo Tokenizer. To represent authentic emotional expressions in dialogue, we obtain an AuEmo Tokenizer by training AuEmoCodec, as shown in Fig. 2 (right). AuEmoCodec is trained on large-scale emotional speech data to learn a discrete authentic emotion token space via FSQ. We construct an emotion codebook of size 1000, from which 750 tokens are activated after training and used as AuEmo tokens. The detailed training procedure of AuEmoCodec is described in Sec. 4.4. The dialogue history speech sequence \({\text{S}_{1 \rightarrow \mathcal{J}}}\) is then processed by the AuEmo Tokenizer to obtain the corresponding AuEmo token embeddings: \[f^{E}_{1 \rightarrow \mathcal{J}} = \text{AuEmo Tokenizer}(\text{S}_{1 \rightarrow \mathcal{J}}) \label{eq:4}\tag{4}\]

4.2 AuEmoToMe-based Authentic Emotion Understanding↩︎

In AR-based CSS, multimodal dialogue history is typically organized as a unified token sequence for contextual modeling. As dialogue turns increase, text and speech tokens accumulate rapidly, resulting in a long multimodal input sequence with many redundant tokens. To address this issue, we design the AuEmoToMe-based Authentic Emotion Understanding module, which merges redundant emotion-related text and speech tokens in the dialogue history and predicts the target AuEmo and speech tokens from the merged history representation. The module consists of three execution steps:

Multimodal Dialogue Token Sequence Construction. Following the order of dialogue turns, the dialogue history is constructed as the following multimodal dialogue history token sequence: \[\begin{align} U_{1 \rightarrow \mathcal{J}} = \{ & f^P_1, f^T_{1,1 \rightarrow max_t}, f^E_1, f^S_{1,1 \rightarrow max_s}, \\ & \dots, \\ & f^P_{\mathcal{J}}, f^T_{\mathcal{J},1 \rightarrow max_t}, f^E_{\mathcal{J}}, f^S_{\mathcal{J},1 \rightarrow max_s} \} \end{align} \label{eq:history95seq}\tag{5}\] where \(f^P_i\), \(f^T_{i,1 \rightarrow max_t}\), \(f^E_i\), and \(f^S_{i,1 \rightarrow max_s}\) denote the speaker token, text tokens, AuEmo token, and speech tokens of the \(i\)-th utterance.

For the target utterance, we only use the speaker token and text tokens as conditional inputs to construct the target utterance token sequence: \[U_{\mathcal{N}} = \{\tikz[baseline=(char.base)]{ \node[draw,circle,inner sep=1pt] (char) {S}; }, f^P_{\mathcal{N}}, f^T_{\mathcal{N},1 \rightarrow max_t}\} \label{eq:target95seq}\tag{6}\] where \(\tikz[baseline=(char.base)]{ \node[draw,circle,inner sep=1pt] (char) {S}; }\) denotes the start token of the target utterance.

Authentic Emotion-driven Token Merging (AuEmoToMe). To reduce the interference of redundant information in multi-turn multimodal dialogue history on target emotion modeling, we design AuEmoToMe based on ToMe [34]. It computes cosine similarity between tokens to identify redundant text and speech tokens, and merges them into a more compact merged history tokens sequence. To preserve authentic emotional information during compression, AuEmoToMe employs an AuEmo-guided merging strategy, where the AuEmo token of each historical utterance acts as an emotion anchor, guiding the weighted aggregation of matched text and speech redundant tokens separately. Thus, AuEmoToMe preserves more emotion-relevant information while merging redundant tokens.

Specifically, as shown in Algorithm 3, AuEmoToMe takes the dialogue history token sequence \(U_{1 \rightarrow \mathcal{J}}\) as input and performs token merging for each historical utterance. For each utterance \(U_i\), its text tokens \(f^T_{i,1 \rightarrow max_t}\) and speech tokens \(f^S_{i,1 \rightarrow max_s}\) are separately used to construct token sequences \(X^M_i\), where \(M \in \{T, S\}\) denotes text or speech. The AuEmo token \(f^E_i\) serves as an emotion anchor. Given the merging ratio \(\rho\), \(X^M_i\) is partitioned into a source set \(A^M\) and a destination set \(B^M\). Cosine similarities between tokens in \(A^M\) and \(B^M\) are computed to select the top-\(k\) matched pairs for merging. The selected pairs are then weighted by their similarities to the AuEmo anchor and aggregated accordingly. Finally, the merged tokens are restored to their original temporal order, and the merged tokens of all utterances are concatenated to form the merged dialogue history tokens \(f^{his}_{merged}\).

Target AuEmo Token and Speech Token Inference. The merged history tokens are concatenated with the target utterance token sequence: \[\{ f^{his}_{merged}, \tikz[baseline=(char.base)]{\node[draw,circle,inner sep=1pt] (char) {S};}, f^P_{\mathcal{N}}, f^T_{\mathcal{N},1 \rightarrow max_t} \} \label{eq:llm95input}\tag{7}\]

The model first predicts the AuEmo token of the target utterance \(f^E_{\mathcal{N}}\), and then generates the speech token sequence \(f^S_{\mathcal{N},1 \rightarrow max_s}\), which is terminated by an end token \(\tikz[baseline=(char.base)]{ \node[draw,circle,inner sep=1pt] (char) {E}}\). Therefore, the target output sequence can be defined as: \[\mathcal{Y}_{\mathcal{N}} = \{f^E_{\mathcal{N}}, f^S_{\mathcal{N},1 \rightarrow max_s}, \tikz[baseline=(char.base)]{ \node[draw,circle,inner sep=1pt] (char) {E}}\} \label{eq:target95output}\tag{8}\]

The generation process follows a conditional autoregressive formulation: \[\begin{align} p(\mathcal{Y}_{\mathcal{N}} \mid f^{his}_{merged}, \tikz[baseline=(char.base)]{ \node[draw,circle,inner sep=1pt] (char) {S}}, f^P_{\mathcal{N}}, f^T_{\mathcal{N},1 \rightarrow max_t}; \Theta) \; \end{align} \label{eq:ar95factorization}\tag{9}\] where \(\Theta\) denotes the model parameters.

To explicitly model the authentic emotion understanding process, the prediction of the target AuEmo token is formulated as: \[p(f^E_{\mathcal{N}} \mid f^{his}_{merged}, \tikz[baseline=(char.base)]{ \node[draw,circle,inner sep=1pt] (char) {S}}, f^P_{\mathcal{N}}, f^T_{\mathcal{N},1 \rightarrow max_t}; \Theta) \label{eq:emo95pred}\tag{10}\] and the speech token sequence is then generated conditioned on the predicted emotion token: \[p(f^S_{\mathcal{N},1 \rightarrow max_s} \mid f^{his}_{merged}, \tikz[baseline=(char.base)]{ \node[draw,circle,inner sep=1pt] (char) {S}}, f^P_{\mathcal{N}}, f^T_{\mathcal{N},1 \rightarrow max_t}, f^E_{\mathcal{N}}; \Theta) \label{eq:speech95pred}\tag{11}\]

4.3 Merged Context-Aware Authentic Emotion Rendering↩︎

This module aims to generate conversational speech with authentic emotional expression by jointly conditioning on the merged history tokens \(f^{his}_{merged}\), the target AuEmo token \(f^{E}_{\mathcal{N}}\), and the target speech token sequence \(f^{S}_{\mathcal{N},1 \rightarrow max_s}\).

Merged Context Encoder. To obtain a context condition for Flow Matching, we feed \(f^{his}_{merged}\) into the Merged Context Encoder. This encoder uses a bidirectional GRU [38] to model the merged history tokens and capture the global dialogue context. The resulting context condition can be formulated as: \[\mathcal{F}_{ctx} = \text{Merged Context Encoder}(f^{his}_{merged}) \label{eq:ctx95encoder}\tag{12}\]

AuEmo Encoder. To obtain an authentic emotion condition for Flow Matching, we feed the predicted AuEmo token \(f^{E}_{\mathcal{N}}\) into the AuEmo Encoder. This encoder projects the AuEmo token into the conditioning space of the Flow Matching model. The resulting authentic emotion condition can be formulated as: \[\mathcal{F}_{e} = \text{AuEmo Encoder}(f^{E}_{\mathcal{N}}) \label{eq:auemo95encoder}\tag{13}\]

Authentic Emotion Flow Matching. Given the predicted target speech token sequence \(f^{S}_{\mathcal{N},1 \rightarrow max_s}\), we first extract its acoustic prior representation, denoted as \(\mu\). The context condition \(\mathcal{F}_{ctx}\) and the target emotion condition \(\mathcal{F}_e\) are then injected into the Flow Matching network together with \(\mu\), such that the generation process is jointly constrained by the target speech prior, the merged dialogue context, and the target authentic emotion.

Following conditional Flow Matching, let \(x_1\) denote the ground-truth mel-spectrogram of the target utterance. Let \(x_0 \sim \mathcal{N}(0, I)\) denote the Gaussian noise. We define the interpolation path between \(x_0\) and \(x_1\) as: \[\phi_t(x) = \big(1 - (1-\sigma_{min})t\big)x_0 + t x_1, \quad t \in [0,1] \label{eq:fm95path}\tag{14}\] where \(\sigma_{\min}=10^{-6}\), following CosyVoice2 [31]. The corresponding target vector field is: \[u_t(\phi_t(x)\mid x_1) = x_1 - (1-\sigma_{min})x_0 \label{eq:target95vector95field}\tag{15}\]

Conditioned on \(\mu\), \(\mathcal{F}_{ctx}\), and \(\mathcal{F}_e\), the Flow Matching network predicts the vector field: \[v_t(\phi_t(x) \mid \mu, \mathcal{F}_{ctx}, \mathcal{F}_e; \theta) \label{eq:base95vector95field}\tag{16}\] where \(\theta\) denotes the model parameters. The main objective is to make the predicted vector field match the target transport direction from noise to the real mel-spectrogram. Therefore, the Flow Matching loss (\(\mathcal{L}_{fm}\)) is defined as: \[\mathcal{L}_{fm} = \mathbb{E}_{x_0,x_1,t} \left[ \left\| v_t(\phi_t(x) \mid \mu, \mathcal{F}_{ctx}, \mathcal{F}_e; \theta) - u_t(\phi_t(x)\mid x_1) \right\|_2^2 \right] \label{eq:fm95loss}\tag{17}\]

Although the \(\mathcal{L}_{fm}\) learns the acoustic transport path, it does not explicitly constrain the intermediate mel states to preserve the target authentic emotion. To address this issue, we further introduce an AuEmo classifier guidance mechanism. Specifically, an auxiliary AuEmo classifier \(\psi\) is applied to the intermediate state \(\phi_t(x)\) to estimate the probability that the current mel state matches the target AuEmo token \(f^{E}_{\mathcal{N}}\). Based on this classifier, the guided vector field can be written as: \[\tilde{v}_t = v_t(\phi_t(x) \mid \mu, \mathcal{F}_{ctx}, \mathcal{F}_e; \theta) + \mathbb{I}(t > r)\cdot \nabla_{\phi_t(x)} \log p_{\psi}\big(f^{E}_{\mathcal{N}} \mid \phi_t(x)\big) \label{eq:guided95vector95field}\tag{18}\] where \(\mathbb{I}(\cdot)\) is the indicator function, \(r\) is a activation threshold set to \(0.7\), and \(\psi\) denotes the pre-trained AuEmoCodec.

The purpose is to guide the intermediate mel state toward a direction that is more consistent with the target authentic emotion. In this way, the optimization trajectory is guided not only by the acoustic transport objective, but also by the target authentic emotion. Notably, we activate this guidance only when \(t > r\), because the intermediate state is still close to Gaussian noise when \(t\) is small and does not yet contain stable emotion-related spectro-temporal patterns. Applying emotion guidance too early may introduce noisy gradients and weaken the learning of the basic transport path. When \(t > r\), the intermediate mel state becomes more structured, and the emotion classifier can provide more reliable guidance.

During inference, the model starts from Gaussian noise \(x_0 \sim \mathcal{N}(0, I)\) and solves the following ordinary differential equation: \[\frac{d\phi_t(x)}{dt} = v_t(\phi_t(x) \mid \mu, \mathcal{F}_{ctx}, \mathcal{F}_e; \theta), \quad t \in [0,1] \label{eq:ode95sampling}\tag{19}\] The final output \(x_1\) is the generated mel-spectrogram, which is then converted into a waveform by the HiFi-GAN vocoder [39].

4.4 Training Process of AuEmoCodec↩︎

AuEmoCodec aims to learn discrete AuEmo tokens from large-scale emotional speech data that are more discriminative and more consistent with real human emotional expression. Specifically, given an emotional speech segment \(S_i\), we first use a Speech Emotion Encoder to extract its emotion representation. We then map this representation into a low-rank quantization space through FSQ-Down and discretize it with a bounded rounding operation. In our implementation, the quantization levels are set to \(\text{levels}=[8,5,5,5]\). The resulting discrete code is defined as the AuEmo token: \[f^{E}_{i} = \mathrm{ROUND}(\mathrm{FSQ-Down}(\text{Speech Emotion Encoder}(S_i))) \label{eq:auemo95token}\tag{20}\]

To enable AuEmoCodec to focus exclusively on emotion representation learning, we do not train it to reconstruct the original waveform. Instead, motivated by the fact that authentic human emotion is often expressed as a complex mixture of multiple affective states [21], [22], we use the perceived scores of speech sample \(S_i\) over seven basic emotion axes as the reconstruction target. Specifically, the quantized AuEmo token \(f^E_i\) is first mapped back to a high-dimensional representation through FSQ-Up, from which the model reconstructs the multi-axis perceived emotion scores, denoted as \(P^{ae}_i\). To obtain supervision targets, we use Gemini-2.5-Flash [40] to annotate the perceived scores of input speech along each emotion axis, denoted as \(GT_i^{ae}\). The prompt used for Gemini-2.5-Flash, together with representative examples of the resulting perceived scores over the seven emotion axes, is provided in Appendix A. AuEmoCodec is then trained by minimizing the mean squared error (MSE) between \(P_i^{ae}\) and \(GT_i^{ae}\), which encourages the learned AuEmo tokens to capture emotional nuances and thereby establish a more effective authentic emotion token space for CSS.

5 Experiments↩︎

This section presents the experimental setup, baseline and ablation models, and the evaluation metrics used to assess speech quality and emotional expressiveness. Additional experimental details are provided in Appendix B.

5.1 Experimental Setup↩︎

We use the open-source NCSSD-EmCap [3] dataset to evaluate the effectiveness of AuEmoChat. This dataset integrates three high-quality dialogue speech datasets: DailyTalk [7], NCSSD [15], and MultiDialog [41]. Specifically, NCSSD-EmCap contains approximately 384 hours of speech data. It comprises 18,580 dialogues with a total of 245,984 utterances, averaging 13.2 turns per dialogue, and covers 25 speakers. Among them, 124,599 utterances are from male speakers and 121,385 utterances are from female speakers. The dataset is divided into training, validation, and test sets with a ratio of 8:1:1, and the same split is used for training both AuEmoCodec and AuEmoChat. For training, AuEmoChat uses 4 NVIDIA A100 GPUs with a batch size of 4 and 8 gradient accumulation steps.

5.2 Baseline and Ablation Models↩︎

Baseline Models. To evaluate the effectiveness of AuEmoChat, we compare it with state-of-the-art (SOTA) emotional CSS systems, including 1) BaseCSS [1], [7], 2) ECSS [2], 3) GPT-Talker [15], and 4) Chain-Talker [3]. For fair comparison, all baseline models are configured to use the authentic emotion token space.

Ablation Models. To assess the contribution of each component in AuEmoChat, we conduct three groups of ablation studies. 1) Ablations on AuEmo Tokenizer: We replace AuEmo Tokenizer with three alternative emotion representations to examine the role of the authentic emotion token space in supporting emotion understanding and rendering: Abl.1 w/ LimEmo, which uses a traditional limited emotion label space. Abl.2 w/ OVEmo, which uses an open-vocabulary (OV) emotion label space. Abl.3 w/ EmoCap, which uses emotion captions. 2) Ablations on AuEmoToMe: We remove AuEmoToMe or its AuEmo-guided merging strategy to investigate its importance for target-utterance emotion understanding and speech token prediction: Abl.4 w/o AuEmoToMe removes the entire AuEmoToMe module. Abl.5 w/o AuEmo-guided Strategy removes the AuEmo-guided token merging strategy. 3) Ablations on Authentic Emotion Flow Matching: We remove different conditioning components to analyze the roles of merged dialogue context and authentic emotion constraints in context-consistent emotional speech rendering: Abl.6 w/o FM-AuEmo removes the AuEmo condition. Abl.7 w/o FM-ACG removes AuEmo classifier guidance (ACG). Abl.8 w/o FM-Context removes merged context condition.

5.3 Evaluation Metrics↩︎

We use the following metrics to evaluate each model’s performance:

Subjective Metrics: Naturalness-DMOS (N-DMOS) [12], [43], [44] assesses the naturalness and overall quality of synthesized speech. Emotion-DMOS (E-DMOS) [2], [3] measures the emotional expressiveness and consistency with ground truth. A total of 30 trained, English-proficient evaluators participated in the subjective evaluation. Each evaluator reviewed the dialogue context and listened to each synthesized sample at least three times before rating its naturalness and emotional consistency.

Objective Metrics: We employ Word Error Rate (WER) [45], Mel Cepstral Distortion (MCD) [46], [47], and Speaker Similarity (SpkSIM) [48], [49] to evaluate the quality of the synthesized speech. For emotion expressiveness, Emotion Accuracy (EmoACC) evaluates performance under a seven-category basic emotion space using emotion2vec [50], while Authentic Emotion Accuracy (AuEmoACC) measures consistency in the authentic emotion token space using the trained AuEmoCodec.

AuEmoCodec Analysis Metrics: To validate the training strategy and hyperparameter design of AuEmoCodec, we employ the following metrics: 1) Tolerance at \(m\) axes (Tol@\(m\)) evaluates the reconstruction accuracy of multi-axis perceived emotion scores while allowing at most \(m\) mismatched axes. 2) Strict Emotion Axis Alignment (S-EAA) measures the reconstruction accuracy of perceived emotion scores across all emotion axes. 3) Dominant Emotion Axis Alignment (D-EAA) evaluates the accuracy of the reconstructed dominant emotion axis. 4) Usage measures codebook utilization.

6 Results and Discussion↩︎

In this section, we comprehensively evaluate AuEmoChat through baseline comparisons, component ablations, analysis of the AuEmoCodec architecture, and analysis of the context token merging rate. Additional experimental results are provided in Appendix C.

6.1 AuEmoChat vs. Baseline Models↩︎

Table 1 presents the subjective and objective evaluation results comparing AuEmoChat with SOTA CSS baselines. Overall, AuEmoChat achieves the best performance across all metrics, demonstrating its superiority in both speech quality and authentic emotion modeling. For subjective evaluations, AuEmoChat achieves the highest scores in both N-DMOS (4.171) and E-DMOS (3.979), surpassing the strongest baseline Chain-Talker by 0.216 and 0.223, respectively. This indicates that AuEmoChat generates more natural speech with more accurate and context-consistent emotional expression. For objective evaluations, AuEmoChat consistently outperforms all baselines in both speech quality and emotion expressiveness. Compared with the strongest baseline Chain-Talker, AuEmoChat reduces WER from 15.07 to 9.14 and MCD from 7.686 to 6.847, indicating improved pronunciation accuracy and acoustic fidelity. It also achieves the highest speaker similarity (78.03), showing better preservation of speaker identity. In terms of emotional expression, AuEmoChat improves EmoACC from 57.16 to 61.04 and AuEmoACC from 24.12 to 28.71. The improvement in AuEmoACC further demonstrates the effectiveness of the proposed AuEmoChat in capturing authentic emotional states beyond basic emotion labels. Overall, these results confirm that AuEmoChat effectively addresses the limitations of existing CSS systems by improving authentic emotion understanding, reducing redundant context interference, and generating more expressive and context-consistent speech.

6.2 Ablation Results of Key Components↩︎

Table 2 presents the ablation results obtained by removing or replacing key components in AuEmoChat. Overall, all ablation variants perform worse than the full AuEmoChat model across both speech quality and emotion expressiveness metrics, confirming the necessity of each proposed component. In Abl.1-Abl.3, replacing AuEmo Tokenizer with alternative emotion representations leads to clear degradation in emotional expressiveness. This indicates that the learned authentic emotion token space is more effective than limited emotion labels, open-vocabulary emotion labels, and emotion captions, enabling the model to better capture emotional states in conversational speech. In Abl.4-Abl.5, removing AuEmoToMe or its AuEmo-guided merging strategy degrades both speech quality and emotional expressiveness. This suggests that redundant dialogue tokens hinder effective context modeling, whereas AuEmoToMe improves target emotion understanding and speech token prediction by reducing irrelevant information. In Abl.6-Abl.8, removing the AuEmo condition, AuEmo classifier guidance, or merged dialogue context degrades both speech quality and emotional expressiveness. This demonstrates that jointly modeling authentic emotion and merged dialogue context is essential for generating context-consistent and emotionally expressive speech. Notably, Abl.1, which uses traditional limited emotion labels, achieves the worst performance on both E-DMOS and AuEmoACC, highlighting the importance of modeling an authentic emotion. In Abl.8, removing merged dialogue context in flow matching significantly degrades speech quality, with WER, MCD, and SpkSIM reaching the worst values, indicating that contextual information plays a critical role in speech generation.

3.5pt

6.3 Analysis of AuEmoCodec Architecture↩︎

In this section, we analyze the key design choices of AuEmoCodec from three aspects: quantization paradigms, codebook capacity, and the judge model, as shown in Table 3. For quantization paradigms, we evaluate Vector Quantization (VQ) [25], Residual VQ (RVQ), Finite Scalar Quantization (FSQ) [24], Residual FSQ (R-FSQ), and Grouped FSQ (G-FSQ). FSQ achieves the best performance on Tol@1, S-EAA, and D-EAA, and shows the most stable overall performance. Although its codebook usage is slightly lower than R-FSQ, it achieves stronger authentic emotion modeling, indicating that FSQ provides a better balance between representation capacity and efficiency. For codebook capacity, as the codebook size increases from 200 to 1000, the performance on Tol@1, S-EAA, and D-EAA consistently improves, indicating that a larger codebook enhances the discrimination ability and representation precision of authentic emotions. However, as the codebook size increases further, codebook usage decreases and performance degrades, suggesting that an excessively large codebook reduces effective utilization and impacts authentic emotion modeling performance. Overall, a codebook size of 1000 achieves the best balance between performance and usage. For the judge model selection, we evaluate Qwen2-Audio [51], Kimi-Audio [52], Qwen3-Omni-Captioner [53], and Gemini-2.5-Flash [40]. Gemini-2.5-Flash achieves the best performance on Tol@1, S-EAA, and D-EAA, demonstrating stronger authentic emotion perception ability. In contrast, Qwen3-Omni-Captioner performs better on Tol@2 and usage. Overall, Gemini-2.5-Flash provides more accurate emotion supervision, which helps learn high-quality emotion representations. Based on the above analysis, we adopt FSQ as the quantization method, set the codebook size to 1000, and use Gemini-2.5-Flash as the judge model.

6.4 Analysis of Context Token Merging Rate↩︎

To analyze the impact of the token merging rate in AuEmoToMe, we conduct experiments with merging ratios of 0%, 15%, 30%, 45%, 60%, and 75%. As shown in Fig. 4, we report the performance on AuEmoACC, EmoACC, WER, MCD, and SpkSIM, where the five sectors correspond to different evaluation metrics. Overall, a token merging rate of 30% achieves the best overall performance. Under this setting, the model attains the best results on AuEmoACC, EmoACC, WER, and MCD, while maintaining near-best performance on SpkSIM. For emotion metrics, as the merging ratio increases from 0% to 30%, both AuEmoACC and EmoACC consistently improve. This indicates that moderate token merging effectively reduces the interference of redundant tokens and enhances the model’s ability to capture the authentic emotion of the target utterance. When the merging ratio is further increased, the performance degrades, suggesting that excessive merging impairs emotional expressiveness. For speech quality metrics, WER and MCD achieve the best performance at 30%, indicating that appropriate token merging alleviates the interference of redundant context in speech modeling, thereby improving pronunciation accuracy. At higher merging ratios, the performance degrades, further confirming the negative impact of information loss on speech generation. Finally, we adopt a token merging rate of 30% during inference to achieve the best balance between emotional expressiveness and speech quality.

7 Conclusion and Future Work↩︎

In this work, we propose AuEmoChat, a novel CSS framework for authentic emotion understanding and rendering. AuEmoChat introduces AuEmoCodec to learn an authentic emotion token space, AuEmoToMe to merge redundant multimodal context tokens while preserving emotion context information, and Authentic Emotion Flow Matching to generate context-consistent authentic emotional speech. To the best of our knowledge, AuEmoChat is the first CSS system explicitly devoted to authentic emotion modeling. We hope this work can inspire further research on more human-like emotional speech synthesis for conversational agents. In future work, we will extend AuEmoChat to multilingual settings, scale up AuEmoCodec training, and further improve the interpretability of authentic emotion for more comprehensive emotional CSS. More detailed limitations and future works are provided in Appendix D.

References↩︎