1 Introduction↩︎

Existing style-controllable TTS systems mainly rely on either text prompts or speech prompts. Text-prompted methods [1], [2] enable intuitive control through natural language descriptions such as “happy”, “old”, or “male”, but such prompts are inherently partial and cannot capture fine-grained acoustic variations such as timbre, rhythm, breathiness, speaking texture, and micro-prosody. Speech-prompted methods and voice cloning systems [3]–[6] preserve richer acoustic information from reference utterances, but typically encode speaking style as a holistic representation, entangling factors such as speaker timbre, age, gender, language, and emotion, which makes category-level editing difficult. Recent multimodal and disentangled speech generation methods [7]–[10] attempt to combine text-based controllability with speech-based expressiveness, yet robust category-level control remains challenging. Classification- or cross-modal-alignment-based methods, such as CLAP-style contrastive learning [11], [12], encourage semantic separation of labeled categories but may discard residual stylistic information, while supervised multi-category disentanglement can suffer from category interference and negative transfer [13], [14]. These limitations indicate that controllable speech generation should not only modify user-specified categories, but also preserve unspecified and residual style information from reference speech.

In this paper, we formulate controllable speech generation as an Arbitrary Style Infilling (ASI) task. Given a text prompt that specifies an arbitrary subset of style categories and a reference speech utterance that contains rich acoustic style information, the goal is to synthesize speech that follows the text-specified categories while inheriting all unspecified and residual styles from the reference speech. In this formulation, text provides explicit semantic control, whereas speech serves as a reservoir of implicit stylistic details. ASI therefore requires selective style editing: replacing only the categories described by text while infilling the remaining style information from speech. Unlike sequential pipelines that perform text-to-speech followed by style transfer [5], [6], or existing text-prompted [2], [15] and speech-prompted [4], [7], [16] SC-TTS methods, ASI addresses a fundamental technical challenge: the trade-off between explicit control and implicit preservation. A naive combination often leads to style interference, where global style transfer methods inadvertently overwrite the fine-grained categories already specified by the text, or fail to preserve the subtle, non-categorical nuances (e.g., unique timbre or breathing) of the reference speaker due to style entanglement. ASI, therefore, requires a dedicated, disentangled framework capable of selective editing, performing surgical intervention on text-specified semantic categories while keeping residual styles intact and unperturbed.

We propose AutoSIFT, a controllable speech generation framework for ASI. AutoSIFT separates speaking style into two complementary components: known style categories, which correspond to text-describable attributes such as gender, age, emotion, and language, and residual style factors, which capture fine-grained acoustic characteristics that are difficult to describe using discrete labels. The framework consists of two core modules. First, a Style Disentangler decomposes a holistic speech style embedding into category-specific prototypes and a residual representation. Instead of relying solely on discriminative classification, the disentangler is trained with a reconstruction-guided objective together with prototypical alignment, encouraging the decomposed components to preserve generative style information while remaining semantically grounded. Second, an Arbitrary Style Infiller combines text-derived category representations with speech-derived unspecified categories and residual style factors. This asymmetric design enables the model to follow the text prompt where explicit control is provided, while automatically preserving missing stylistic information from reference speech.

We evaluate AutoSIFT on bilingual, multi-style speech generation benchmarks covering gender, age, emotion, and language. To directly measure the objective of ASI, we introduce two evaluation metrics: Described Style Accuracy (DSA), which measures whether generated speech follows the style categories specified by text, and Residual Style Accuracy (RSA), which measures whether unspecified categories are preserved from reference speech. Our contributions are summarized as follows:

• We formulate Arbitrary Style Infilling (ASI), a controllable speech generation task that requires modifying text-specified style categories while preserving unspecified and residual styles from reference speech.

• We propose AutoSIFT, a framework that decomposes speaking style into category-level prototypes and residual acoustic factors, enabling selective style editing under partially specified control.

• We design an Arbitrary Style Infiller that fuses text-derived style categories with speech-derived unspecified styles, allowing flexible control over arbitrary subsets of style attributes.

• We introduce ASI-specific evaluation metrics, including Described Style Accuracy (DSA) and Residual Style Accuracy (RSA), and demonstrate the effectiveness of AutoSIFT for fine-grained style-controllable speech generation.

2 Related Work↩︎

Prompt-based Controllable Text-to-Speech. Controllable text-to-speech (TTS) aims to synthesize speech that follows both linguistic content and user-specified speaking styles. Existing methods typically use either natural-language prompts or speech prompts as the control signal. Text-prompted methods, such as PromptTTS [17], PromptTTS 2 [1], PromptTTS++ [15], and ParaStyleTTS [2], allow users to describe desired attributes such as speaker timbre, emotion, age, gender, pitch, or speaking rate. While intuitive, text prompts are inherently underspecified: fine-grained acoustic details such as timbre, breathiness, rhythm, and micro-prosody are difficult to fully express in short natural-language descriptions. Speech-prompted TTS and voice cloning methods, including YourTTS [3], F5-TTS [16], CosyVoice [7], IndexTTS [4], AutoVC [5], and SpeechSplit 2.0 [6], instead condition generation on reference speech and can preserve richer speaker and prosodic information. However, they usually treat reference style as a holistic conditioning signal, making them effective for style replication but less suitable for category-level editing. In contrast, AutoSIFT targets arbitrary style infilling, where text-specified categories should be modified while unspecified and residual styles are preserved from reference speech.

Disentangled and Factorized Speech Representations. Disentangled and factorized speech representation learning aims to separate speech into interpretable components such as content, speaker timbre, prosody, timbre, and emotion. Prior voice conversion methods such as AutoVC [5] and SpeechSplit 2.0 [6] disentangle speaker- and content-related information for voice conversion, while recent TTS systems such as NaturalSpeech 3 [18] and StyleTTS 2 [19] introduce structured latent factors or style diffusion to improve generation quality, diversity, and adaptation. Another related direction learns semantically grounded style representations through audio-text alignment or multi-task supervision: CLAP [11] and ParaCLAP [12] align audio with natural-language descriptions, and ParaMETA [10] learns disentangled paralinguistic representations for categories such as emotion, age, gender, and language.

These methods are closely related to category-aware style control, but they mainly emphasize semantic separability or generation quality. AutoSIFT instead focuses on the joint requirement of controllability and preservation: it replaces only the text-specified style categories while infilling unspecified categories and residual acoustic nuances from reference speech.

3 Preliminary↩︎

3.1 Terminology Definition↩︎

We first define the key terminology used throughout this paper. A raw speech waveform is typically represented in \(\mathbb{R}^T\), but we operate on its latent speech representation rather than the waveform directly. Specifically, speech is first transformed into a spectrogram using the Short-Time Fourier Transform (STFT) [20], and then mapped by an encoder or neural vocoder (e.g., HiFi-GAN [21]) into a latent space. Thus, a speech sample is denoted as \(Y \in \mathbb{R}^{D \times T}\), where \(D\) is the hidden dimension, and \(T\) is the number of frames. In our implementation, each frame corresponds to approximately 16 ms of raw speech.

We decompose speaking style \(S\) into a set of text-describable style categories \(C=\{c_1,c_2,\dots,c_n\}\), such as gender, age, emotion, and language. For each category \(c_i\), we define a discrete label set \(L_i=\{l_{i,1},l_{i,2},\dots,l_{i,K_i}\}\), where \(K_i\) is the number of style classes in that category. For example, male and female are classes under gender, while happy, angry, and sad are classes under emotion. This formulation supports category-level style modeling, where each category can be independently identified, preserved, or modified. Beyond text-describable categories, speech also contains residual styles \(c_r\), including subtle timbre, micro-prosody, speaking texture, breathiness, and other fine-grained acoustic nuances that are difficult to express using natural language or discrete labels.

3.2 Problem Formulation↩︎

Given a reference speech embedding \(Y \in \mathbb{R}^{D \times T}\) with speaking style \(S\), style-controllable speech generation aims to synthesize speech from content text \(X\) while controlling its speaking style. Speech-prompted methods, such as voice cloning, can preserve the overall style of the reference speech but provide limited explicit control over individual style categories. Text-prompted methods allow users to specify high-level styles, such as “happy” or “old male”, but such descriptions are inherently incomplete and often fail to capture fine-grained acoustic details such as timbre, rhythm, breathiness, and micro-prosody.

To bridge this gap, we formalize the set of \(N\) text-describable style categories as \(C=\{c_1, c_2, \dots, c_n\}\) (e.g., gender, age, emotion, and language). Given a specific text style prompt \(t\), let \(C_t \subseteq C\) be the subset of categories explicitly specified by the user, and \(C_{\bar{t}} = C \setminus C_t\) be the remaining unspecified categories. Crucially, we define \(c_r\) as the Residual Style Category. This category encapsulates the exhaustive set of paralinguistic and acoustic styles that lie outside the descriptive scope of \(C\), such as fine-grained vocal textures and idiosyncratic prosodic habits. The complete style taxonomy is thus defined as \(C \cup \{c_r\}\), while the corresponding style representation is denoted as \(S=\{s_{c_1},\ldots,s_{c_n},s_r\}\).

We propose Arbitrary Style Infilling (ASI) to address the limitations of prompt-based control. The goal of ASI is to generate speech that adheres to the text prompt for categories in \(C_t\), while seamlessly "infilling" the unspecified categories \(C_{\bar{t}}\) and the residual style \(c_r\) from the reference speech.

Formally, AutoSIFT derives a target style embedding \(\hat{S}\) by applying a style sifting and fusion function \(\Phi\) to the text-specified style \(S_t\) and the reference style \(S_s\): \[\hat{S} = \Phi(S_t, S_s),\] where \(\Phi\) denotes the style sifting and fusion function. For each text-describable category \(c_i\), the generated style is expected to satisfy: \[\hat{S}^{c_i} = \begin{cases} S_t^{c_i}, & \text{if } c_i \in C_t, \\ S_s^{c_i}, & \text{if } c_i \in C_{\bar{t}}, \end{cases},\quad \hat{S}^{c_r} = S_s^{c_r},\] where \(S_t^{c_i}\) denotes the text-specified style for category \(c_i\), and \(S_s^{c_i}\) denotes the corresponding style extracted from the reference speech. Residual styles from the reference speech, denoted as \(S_s^{c_r}\), are preserved: Thus, AutoSIFT modifies only the text-described style categories \(C_t\) while infilling unspecified categories \(C_{\bar{t}}\) and residual styles \(c_r\) from reference speech.

4 Methodology↩︎

The training of AutoSIFT follows a hierarchical three-stage pipeline designed for modular stability and reproducibility. In Stage one, we establish the foundational style-controllable speech generation framework by jointly optimizing the TTS backbone and the Style Extractor. The primary objective of this stage is to ensure that the Style Extractor produces a comprehensive style embedding \(S\) from speech \(Y\), which contains sufficient style information for high-fidelity speech reconstruction. In Stage two, the Style Extractor is frozen, and the Style Disentangler is trained to decouple the style embedding \(S\) into \(N\) text-describable category embeddings \(C=\{c_1,c_2,\cdots,c_n\}\) and a residual style embedding \(c_r\). In Stage three, we introduce the Arbitrary Style Infiller (ASI) as the central style manipulation module. All preceding modules are kept frozen, and ASI is trained to dynamically sift and fuse styles to produce a manipulated style embedding \(\hat{S}\) that follows the text-described categories in \(C_t\) while infilling the text-undescribed categories \(\bar{C}_t\) and the residual style \(c_r\) from the reference speech. Figure 1 presents an overview of AutoSIFT.

4.1 Style Extractor↩︎

The Style Extractor is designed to extract a global speech style embedding \(S \in \mathbb{R}^{D}\) from the speech embedding \(Y \in \mathbb{R}^{D \times T}\) in a completely unsupervised manner. It comprises a Transformer-based encoder coupled with an attentive pooling layer, integrated into a Flow-Matching [22] Diffusion Transformer [23] (FM-DiT) to achieve effective disentanglement between content and style.

Specifically, the Style Extractor contains a multi-layer Transformer encoder [24] to capture long-range temporal dependencies from speech embeddings. The encoder processes \(Y\) through successive self-attention and feed-forward layers, outputting a sequence of hidden embeddings that represent frame-level speaking styles. An attentive pooling layer is employed to condense variable-length hidden embeddings into a single embedding that represents the global speaking style. The attentive layer adaptively assigns weights to hidden embeddings, allowing the extractor to focus on the most salient stylistic frames. The resulting embedding is then normalized through \(L_2\) normalization for numerical stability. The normalized embedding \(S \in \mathbb{R}^D\) is further transformed by a linear projector into a block-wise style embedding \(S' \in \mathbb{R}^{L \times 2H}\), where \(L\) denotes the number of DiT blocks and \(H\) is the hidden size of the block-wise style embedding \(S'\).

To facilitate disentanglement, we leverage a frozen text encoder from a pre-trained TTS model [2] to provide a fixed content prior. A trainable Feed-Forward Transformer (FFT) [25] is then deployed to adapt these content embeddings \(X \in \mathbb{R}^{H \times T}\) to the style-controllable TTS generation task. During training, the DiT model, conditioned on both the adapted content and the extracted style via AdaLN-Zero, learns to map Gaussian noise \(\epsilon \sim \mathcal{N}(0, I)\) to the speech embedding \(Y\).

The generative process is formulated as \(Y = \mathcal{G}(\epsilon \mid X, S)\), where \(\mathcal{G}\) represents the FM-DiT generator conditioned on both the adapted content and the extracted style. We optimize the model using the standard flow-matching objective, minimizing the \(L_2\) regression loss between the predicted velocity \(v_\tau\) and the target velocity \(u_\tau\): \[\begin{align} \mathcal{L}_{FM} &= \mathbb{E}_{\tau, \epsilon, Y} [ \| v_\tau(\mathbf{x}_\tau \mid X, S) - (Y - \epsilon) \|^2 ], \quad \mathbf{x}_\tau = \tau Y + (1-\tau)\epsilon, \quad \tau \sim (0,1) \end{align}\]

By freezing the content prior, the framework compels the Style Extractor to capture all essential styles such as prosody and rhythm necessary to accurately reconstruct the target speech along the probability path, ensuring that the correct content is rendered with the right speaking style.

4.2 Style Disentangler↩︎

While the Style Extractor produces a holistic style embedding \(S \in \mathbb{R}^{D}\) from speech \(Y\), different categories (e.g., gender, age, emotion) are inherently entangled within this \(D\)-dimensional space. This entanglement hinders granular control over specific style categories. To achieve category-level disentanglement, we propose a Style Disentangler designed to partition the holistic embedding into independent, semantically meaningful, and category-specific subspaces.

Style Decomposition: The process begins with a Style Decomposer, which employs a set of \(N+1\) category-specific linear projectors \(\{\psi_{c_1}, \psi_{c_2}, \dots, \psi_{c_n}, \psi_{r}\}\). Given a style embedding \(S\), each projector maps it into a \(d\)-dimensional subspace to yield corresponding embeddings: \[s_{c_i} = \psi_{c_i}(S), \quad s_{r} = \psi_{r}(S),\] where \(s_{c_i} \in \mathbb{R}^d\) represents the \(i\)-th text-describable category \(c_i \in C\), and \(s_r \in \mathbb{R}^d\) captures the residual style information that falls outside the defined categorical taxonomy.

Prototypical Alignment and Retrieval: To ground these decomposed embeddings in concrete semantics, we introduce a set of learnable Style Prototypes. For each category \(c_i\), we maintain a prototype \(\mathbf{P}_{c_i} = \{ \mathbf{p}_{c_i}^1, \dots, \mathbf{p}_{c_i}^{K_i} \}\), where each \(\mathbf{p}_{c_i}^j \in \mathbb{R}^d\) serves as a representative centroid for a specific class (e.g., a particular emotion or language). To ensure the Style Disentangler maintains a robust style manifold even when explicit ground-truth labels \(y_{c_i}\) are missing, we employ a retrieval-based filling strategy during training. For unlabeled samples, the model identifies the most semantically similar prototype via cosine similarity to generate a pseudo-label \(\hat{y}_{c_i}\): \[\hat{y}_{c_i} = \arg\max_{j \in {1, \dots, K_i}} \mathrm{cos}(s_{c_i}, \mathbf{p}_{c_i}^j).\] The disentanglement is enforced by the Prototypical Alignment Loss (PAL) [10]. For a sample with a label (or pseudo-label) \(y_{c_i}\), PAL encourages \(s_{c_i}\) to align with its corresponding semantic anchor: \[\mathcal{L}_{PAL} = \sum_{c_i \in C} \left| s_{c_i} - \text{sg}(\mathbf{p}_{c_i}^{y{c_i}}) \right|^2.\] By pulling \(s_{c_i}\) toward fixed semantic centroids and employing the stop-gradient \(\text{sg}(\cdot)\) operator, the Decomposer is forced to discard information irrelevant to category \(c_i\), thereby achieving effective disentanglement and enhancing the interpretability of the latent space.

Residual Quantization and Style Reconstruction: While text-describable categories are constrained by prototypes, the residual embedding \(s_r\) captures paralinguistic nuances that are difficult to categorize. To regularize this space, we pass \(s_r\) through a Vector Quantization (VQ) layer [26] with a trainable codebook \(\mathcal{B} = \{ \mathbf{b}_1, \dots, \mathbf{b}_K \}\). The continuous embedding \(s_r\) is mapped to its nearest neighbor \(\hat{s}_r = \text{vq}(s_r) = \mathbf{b}_k\). The VQ objective is defined as: \[\mathcal{L}_{VQ} = \| \text{sg}(s_r) - \hat{s}_r \|^2 + \beta \| s_r - \text{sg}(\hat{s}_r) \|^2.\] Finally, to ensure that the decomposed components collectively preserve the information of the original embedding, we perform a reconstruction-based fusion. We select the target prototypes \(\hat{s}_{c_i}\) (using ground-truth or retrieved labels) and concatenate them with the quantized residual \(\hat{s}_r\) to form \(s_{\text{joint}} = \mathrm{concat}(\hat{s}_{c_1}, \dots, \hat{s}_{c_n}, \hat{s}_r )\). A Style Fuser \(\psi_{\text{fuse}}\) (a residual network with SiLU activations) then projects \(s_{\text{joint}}\) back to the original dimension \(D\) to produce the reconstructed style embedding \(\hat{S} = \psi_{\text{fuse}}(s_{\text{joint}})\). The final loss for the Style Disentangler is: \[\mathcal{L}_{disen} = \mathcal{L}_{recon} + \mathcal{L}_{PAL} + \mathcal{L}_{VQ}, \quad \mathcal{L}_{\text{recon}} = | \hat{S} - \text{sg}(S) |^2.\] The reconstruction loss \(\mathcal{L}_{recon}\) ensures that the discretized prototypes and residual codebook effectively span the original style embedding, providing a solid foundation for subsequent style infilling.

4.3 Arbitrary Style Infiller↩︎

To fully bridge the gap between text prompts and speech prompts, we propose Arbitrary Style Infiller (ASI), an arbitrary style sifting and fusion module. The core idea of ASI is to use the text-described style as the primary control signal while treating the reference speech as a reservoir of unspecified and residual style information. This allows the system to prioritize user-specified style categories from the text prompt and adaptively fill in unspecified style categories and subtle textures or breathing patterns from speech.

Following the disentangled space established in Section 4.2, let \(S \in \mathbb{R}^D\) denote the holistic style embedding extracted from reference speech \(Y\), and \(T \in \mathbb{R}^D\) denote the text-prompted style embedding derived from a text description.

Using the Style Decomposer, both embeddings are projected into category-specific subspaces, yielding speech-derived style category prototypes \(\{\hat{s}_{c_1}, \ldots, \hat{s}_{c_n}, \hat{s}_r\}\) and text-derived style category embeddings \(\{t_{c_1}, \ldots, t_{c_n}\}\). ASI implements a gated sifting process that selectively aggregates embeddings based on the presence of textual descriptions. For each text-describable category \(c_i \in C\), the fused latent \(f_{c_i}\) is determined by a conditional gate: \[f_{c_i} = \begin{cases} t_{c_i}, & \text{if category } c_i \text{ is specified in text prompt} \\ \text{sg}(\hat{s}_{c_i}), & \text{otherwise} \end{cases}\]

Crucially, the residual style embedding \(\hat{s}_r\) is always inherited directly from the speech embedding. This asymmetric inheritance ensures that the generated style embedding maintains the rich, non-categorical textures of the original speaker. The concatenated fused embedding is then passed through a linear style fuser \(\psi_f(\cdot)\). The style fuser is optimized using a Style Reconstruction Loss, by minimizing the distance between the reconstructed style embedding \(\hat{S}_f\) and the original style embedding \(S\): \[\mathcal{L}_{asi} = \|\hat{S}_f - \mathrm{sg}(S)\|^2, \quad \hat{S}_f = \psi_f(\mathrm{concat}(f_{c_1}, \ldots, f_{c_n}, \mathrm{sg}(\hat{s}_r))).\] This objective encourages the fused style embedding to follow text-described categories while preserving text-undescribed and residual styles from the speech style embedding \(S\).

To equip the model with robust style infilling capabilities, we introduce a Censored Learning strategy. During training, we simulate scenarios with missing categorical text prompts by randomly dropping categorical labels at a rate of 25%. This mechanism compels the Style Fuser to dynamically pivot to the speech-derived branch whenever a textual anchor is unavailable. By optimizing this sift-and-fill operation, ASI learns to synthesize a hybrid embedding that balances explicit instruction with implicit imitation. For instance, if a user specifies "angry" via text, ASI retains the corresponding "angry" prototype while automatically infilling the speaker’s unique timbre and rhythmic micro-variations from the reference speech.

5 Experiments↩︎

In this section, we aim to answer the following research questions (RQs):

• Does the proposed Style Disentangler learn category-aware style representations?

• Can AutoSIFT synthesize intelligible and natural speech while performing style control?

• Can AutoSIFT preserve speaking style and reference-derived styles from reference speech?

• Can AutoSIFT correctly realize the style categories explicitly specified by text prompts?

• Can AutoSIFT preserve unspecified reference styles while modifying only the text-specified style categories?

5.1 Experiment Setting↩︎

Dataset. We constructed a comprehensive bilingual, multi-style speech dataset by combining multiple open-source datasets, including Baker [27], LJSpeech [28], ESD [29], CREMA-D [30], CommonPhone [31], and the Genshin Voice dataset [32]. The resulting dataset contains approximately 110k speech samples, totaling 138 hours of speech from 6,361 speakers. The rationale for combining multiple datasets is to overcome the narrow style coverage of individual datasets; for instance, ESD lacks age diversity, while CommonPhone lacks emotional intensity. This combination enables the Style Disentangler to span a comprehensive taxonomy of age, gender, and emotion. More details are provided in Appendix 11.

Evaluation Metrics. We evaluate AutoSIFT from three perspectives: category-aware style representation, speech generation quality, and arbitrary style infilling. For category-aware style representation, we follow the speaker-independent evaluation protocol of ParaMETA [10] and report balanced accuracy (B.Acc), Macro-F1, and Weighted-F1 for each style category, including gender, age, emotion, and language. For speech generation quality, we report phoneme error rate (PER) to measure content intelligibility, mean opinion score (MOS) to assess perceptual naturalness, and style similarity (S-SIM) to evaluate whether the generated speech preserves the reference style. For arbitrary style infilling, we report Described Style Accuracy (DSA), Residual Style Accuracy (RSA). DSA measures whether the generated speech follows the style categories explicitly specified by the text prompt, while RSA measures whether the unspecified style categories are preserved from the reference speech. Detailed metric definitions are provided in Appendix 8.

Implementation Details. We implement AutoSIFT in PyTorch. The style-controllable TTS backbone adopts a Diffusion Transformer (DiT) [23] with 12 Transformer blocks, a hidden size of 384, and 6 attention heads. The Style Extractor outputs a 192-dimensional style embedding, which is further decomposed by the Style Disentangler into four 48-dimensional category-specific subspaces corresponding to gender, age, emotion, and language. For text-style conditioning, we use a pre-trained MPNet encoder [33] and project its 768-dimensional semantic embedding into the shared 192-dimensional style space. All baselines are tuned following their recommended settings or official implementations when available. Full training configurations, optimization details, and compute requirements are provided in Appendix 12.

5.2 Category-Aware Style Disentanglement (RQ1)↩︎

We evaluate AutoSIFT’s category-aware style disentanglement across four dimensions: emotion, gender, age, and language. Table 1 reports the classification performance of the Style Disentangler. AutoSIFT (PAL) consistently outperforms all baselines, including large-scale pre-trained audio-text models such as CLAP [11] and ParaCLAP [12]. For example, on Emotion, AutoSIFT (PAL) achieves 67.2% Balanced Accuracy, substantially higher than 22.1% from CLAP (Speech & Music), indicating that our reconstruction-guided Style Disentangler learns more effective category-level style representations from speech than text-speech aligned embeddings.

Under the same backbone and dataset, AutoSIFT (PAL) also surpasses classification-based CrossEntropy, alignment-based InfoNCE [11], [34], and disentanglement-based ParaMETA [10]. On Language, AutoSIFT reaches 98.8% B. Acc, compared with 89.5% for CrossEntropy and 91.1% for ParaMETA. This advantage comes from prototype-driven reconstruction: the learnable Style Prototypes \(\mathbf{P}\) are not only discriminative anchors, but also structural components for reconstructing the original style embedding \(S\). Unlike classification-based methods that only need to learn decision boundaries, AutoSIFT forces each category-specific embedding \(s_{c_i}\) to align with prototypes that span the corresponding style manifold. The ablation further confirms the importance of Prototypical Alignment Loss (PAL). Removing PAL causes classification performance to collapse, e.g., Age B. Acc drops to 0.9%. Although the reconstruction loss \(\mathcal{L}_{recon}\) can still decrease without PAL, the decomposed subspaces are no longer organized into human-interpretable semantic categories. This shows that PAL is essential for grounding the “sift-and-fill” operation in meaningful style semantics, turning reconstruction-based decomposition into an interpretable style manipulation framework.

5.3 Speech Generation Quality and Reference Style Preservation↩︎

Intelligibility and Naturalness (RQ2). As shown in Table 2, AutoSIFT achieves competitive performance in both intelligibility and naturalness across prompt modalities. In terms of intelligibility, AutoSIFT achieves a low Phoneme Error Rate (PER) of 3.59 in English and 14.32 in Chinese for text-prompted generation, consistently outperforming other controllable models like CosyVoice and ParaMETA. Furthermore, AutoSIFT reaches high Mean Opinion Scores (MOS) of up to 4.11, surpassing top-tier zero-shot models such as F5-TTS and IndexTTS. These results confirm that by leveraging a pre-trained backbone and a disentangled style manifold, AutoSIFT produces high-fidelity, human-like speech while maintaining precise control over specified stylistic attributes.

Speaking Style Preservation (RQ3). AutoSIFT exhibits an exceptional ability to preserve speaking style and reference-derived characteristics, significantly outperforming all baseline models in Style Similarity (S-SIM). Specifically, when using speech prompts, AutoSIFT achieves an S-SIM of 0.8512 in English and 0.8100 in Chinese, providing a massive gain (approximately 20% to 30%) over competitive zero-shot models like IndexTTS (0.6501) and F5-TTS (0.6422). Even under text-only prompts, it maintains an S-SIM of over 0.57, which is remarkably high for cross-modal generation. This performance validates that our style extractor can effectively learn the representative speaking style and achieve precise style-controllable speech generation.

5.4 Text-Specified Style Control and Arbitrary Style Infilling (RQ4-RQ5)↩︎

To evaluate AutoSIFT’s fine-grained control ability, Figure 2 visualizes its performance under different control modalities and infilling ratios. As shown in Figure 2 (a), AutoSIFT consistently outperforms existing SC-TTS baselines in following both text and speech prompts. With the ASI module, it achieves nearly 100% accuracy on Language and Gender, and around 90% on Emotion and Age. The lower accuracy on Emotion and Age is expected, as these categories involve more subjective and overlapping paralinguistic cues, whereas Language and Gender are acoustically more distinguishable.

Figure 2 (b) further analyzes ASI under partial text specification, where some categories are controlled by text while the remaining ones are infilled from reference speech. AutoSIFT performs best when only a few categories are edited; for single-category intervention, it achieves nearly 100% control accuracy while preserving other speech-derived styles. As the text-prompting rate increases, performance gradually decreases, suggesting that harmonizing multiple text-specified prototypes introduces stronger inter-category interference. These results indicate that AutoSIFT is particularly effective for surgical style editing, where a small number of target categories are modified while most reference-derived style information is preserved.

6 Conclusion↩︎

We presented AutoSIFT, a controllable speech generation framework for Arbitrary Style Infilling (ASI). AutoSIFT modifies only text-specified style categories while preserving unspecified and residual acoustic styles from reference speech. It achieves this by decomposing speaking style into category-specific prototypes and residual factors, then selectively fusing text-derived and speech-derived style components. Experiments demonstrate that AutoSIFT improves category-level controllability while maintaining reference-derived style information, validating ASI as an effective formulation for fine-grained speech style control.

7 Limitations↩︎

AutoSIFT currently focuses on a predefined set of text-describable style categories, such as gender, age, emotion, and language. While these categories cover common controllable speech generation scenarios, real-world speaking styles may involve more fine-grained or continuous attributes, such as accent, speaking rate, pitch range, and conversational intent. In addition, the current framework relies on category-level annotations to learn semantically grounded style prototypes, and its disentanglement quality may be affected by the coverage and consistency of these annotations. Future work could extend AutoSIFT to broader style dimensions and explore weaker or automatically discovered forms of style supervision.

8 Evaluation Metrics↩︎

We evaluate AutoSIFT from three complementary perspectives: category-aware style representation, speech generation quality, and arbitrary style infilling.

8.0.0.1 Category-aware style representation.

To evaluate whether the Style Disentangler learns category-aware representations, we follow the speaker-independent evaluation protocol of ParaMETA [10]. For each style category \(c_i \in C\), we train or evaluate a category-specific classifier and report balanced accuracy (B.Acc), Macro-F1, and Weighted-F1.

Let \(y_{c_i}\) denote the set of classes for category \(c_i\), and let \(|y_{c_i}^k|\) be the number of test samples belonging to class \(k\). For each class \(k\), let \(\mathrm{TP}_k\), \(\mathrm{FP}_k\), and \(\mathrm{FN}_k\) denote the number of true positives, false positives, and false negatives, respectively. The recall and F1 score for class \(k\) are defined as: \[\mathrm{Recall}_k = \frac{\mathrm{TP}_k}{\mathrm{TP}_k+\mathrm{FN}_k},\] \[\mathrm{F1}_k = \frac{2\mathrm{TP}_k}{2\mathrm{TP}_k+\mathrm{FP}_k+\mathrm{FN}_k}.\] The balanced accuracy, Macro-F1, and Weighted-F1 for category \(c\) are then computed as: \[\mathrm{B.Acc}_c = \frac{1}{K_c}\sum_{k=1}^{K_c}\mathrm{Recall}_k,\] \[\mathrm{Macro\text{-}F1}_c = \frac{1}{K_c}\sum_{k=1}^{K_c}\mathrm{F1}_k,\] \[\mathrm{W.F1}_c = \sum_{k=1}^{K_c}\frac{n_k}{\sum_{j=1}^{K_c}n_j}\mathrm{F1}_k.\] These metrics measure whether each disentangled subspace captures the intended category-level information.

8.0.0.2 Speech generation quality.

To evaluate whether AutoSIFT generates intelligible and natural speech while performing style control, we report phoneme error rate (PER), mean opinion score (MOS), and style similarity (S-SIM).

PER measures the phoneme-level mismatch between the generated speech and the target content. Given the reference phoneme sequence and the recognized phoneme sequence of generated speech, PER is computed as: \[\mathrm{PER} = \frac{N_{\mathrm{sub}} + N_{\mathrm{del}} + N_{\mathrm{ins}}}{N_{\mathrm{ref}}},\] where \(N_{\mathrm{sub}}\), \(N_{\mathrm{del}}\), and \(N_{\mathrm{ins}}\) denote the numbers of substitutions, deletions, and insertions, respectively, and \(N_{\mathrm{ref}}\) is the number of phonemes in the reference sequence. A lower PER indicates better content intelligibility.

MOS is used to measure the perceptual naturalness of generated speech. Given \(M\) generated utterances and \(R\) human ratings per utterance, MOS is computed as: \[\mathrm{MOS} = \frac{1}{MR}\sum_{i=1}^{M}\sum_{r=1}^{R} m_{i,r},\] where \(m_{i,r}\) denotes the rating assigned by rater \(r\) to utterance \(i\). A higher MOS indicates better perceptual quality.

S-SIM measures style similarity between the generated speech \(\hat{Y}\) and the reference speech \(Y_s\). Let \(E_{\mathrm{sty}}(\cdot)\) denote the style extractor. We compute S-SIM using cosine similarity in the style embedding space: \[\mathrm{S\text{-}SIM}(\hat{Y},Y_s) = \frac{ E_{\mathrm{sty}}(\hat{Y})^\top E_{\mathrm{sty}}(Y_s) }{ \|E_{\mathrm{sty}}(\hat{Y})\|_2 \|E_{\mathrm{sty}}(Y_s)\|_2 }.\] The final S-SIM is averaged over all evaluation samples: \[\mathrm{S\text{-}SIM} = \frac{1}{M}\sum_{i=1}^{M} \mathrm{S\text{-}SIM}(\hat{Y}_i,Y_{s,i}).\] A higher S-SIM indicates stronger preservation of reference-derived style information.

8.0.0.3 Arbitrary style infilling.

To directly evaluate the proposed Arbitrary Style Infilling (ASI) task, we report Described Style Accuracy (DSA), Residual Style Accuracy (RSA).

For each evaluation sample \(i\), let \(C_t^{(i)} \subseteq C\) denote the set of style categories explicitly specified by the text prompt, and let \(C_{\bar{t}}^{(i)} = C \setminus C_t^{(i)}\) denote the unspecified categories. Let \(y_{t,i}^{c}\) be the target label specified by the text prompt for category \(c\), and let \(y_{s,i}^{c}\) be the reference-speech label for category \(c\). We use a category-specific evaluator \(h_c(\cdot)\) to predict the style label of generated speech \(\hat{Y}_i\) for category \(c\).

DSA measures whether generated speech follows the text-specified categories: \[\mathrm{DSA} = \frac{ \sum_{i=1}^{M}\sum_{c \in C_t^{(i)}} \mathbb{1}\left[h_c(\hat{Y}_i)=y_{t,i}^{c}\right] }{ \sum_{i=1}^{M}|C_t^{(i)}| }.\] A higher DSA indicates better controllability over the categories described by the text prompt.

RSA measures whether generated speech preserves the unspecified categories from the reference speech: \[\mathrm{RSA} = \frac{ \sum_{i=1}^{M}\sum_{c \in C_{\bar{t}}^{(i)}} \mathbb{1}\left[h_c(\hat{Y}_i)=y_{s,i}^{c}\right] }{ \sum_{i=1}^{M}|C_{\bar{t}}^{(i)}| }.\] A higher RSA indicates better preservation of undescribed reference styles.

9 Research with Human Subjects↩︎

We recruited 10 native English speakers and 10 native Chinese speakers for subjective perceptual evaluation using mean opinion score (MOS) tests. All participants were recruited on a voluntary basis, provided informed consent prior to the study, and were not financially compensated for this voluntary evaluation. We provide the full instructions given to participants, along with a screenshot of the evaluation interface.

Instructions given to human participants: Please listen to the following audio and rate it based on its naturalness and sound quality.

• 5-Excellent-Very natural, sounds like a real person, no robotic artifacts.

• 4-Good-Fairly natural, slight artifacts present but doesn’t affect listening.

• 3-Fair-Average, noticeable artifacts, but speech is clear.

• 2-Poor-Poor quality, obvious robotic sounds, noise, or stuttering.

• 1-Bad-Very poor, difficult to understand or severely distorted.

10 Case Study↩︎

Beyond quantitative assessment, AutoSIFT is designed to address practical needs in controllable speech generation. Its arbitrary style infilling capability is particularly useful in scenarios where users need to modify certain speaking-style categories while preserving styles from reference speech. In this section, we discuss several representative application scenarios.

10.1 Immersive Narrative and Gaming↩︎

In interactive storytelling and game production, a character’s voice needs to remain recognizable across different narrative situations. For example, the same character may need to sound calm, frightened, exhausted, or aged depending on the plot. However, conventional style-controllable TTS systems may unintentionally alter the speaker identity when applying a strong emotional or demographic style.

AutoSIFT provides a more flexible solution by using reference speech to preserve the speaker’s residual paralinguistic characteristics, while allowing specific style categories, such as emotion or age, to be controlled through text prompts. In practice, a developer could provide a neutral voice sample of a character and generate variants such as “terrified”, “sad”, or “elderly” without recording a large number of additional lines. This reduces production cost while helping the generated speech remain consistent with the character’s original voice.

10.2 Cross-Lingual Dubbing and Localization↩︎

Dubbing and localization require more than accurate translation. A localized voice should ideally preserve the performance characteristics of the original speaker, including emotional intensity, rhythm, and subtle prosodic habits. Existing cross-lingual TTS or voice conversion methods often focus on speaker similarity, but may lose these fine-grained expressive details.

With AutoSIFT, the original speech can be used as a reference source for speaker- and performance-related characteristics, while the target language is specified through the text prompt. This allows the model to replace the language-related style attribute while preserving other aspects of the speaker’s delivery. As a result, the generated speech can better retain the original actor’s expressive characteristics in the target language, which is valuable for film, animation, and other multilingual media production.

10.3 Audio Post-Production↩︎

In audio post-production, a line may need to be adjusted because its emotion, tone, or delivery does not fully match the scene. Re-recording can be expensive, especially when actors, studios, and production schedules are involved. A practical speech editing tool should therefore support targeted modification rather than regenerate the entire utterance in a way that changes unrelated acoustic details.

AutoSIFT can be used for this type of fine-grained editing. For instance, if a recorded line needs to sound more angry, the original take can be used as the reference speech, while an “angry” prompt specifies the desired emotional category. The model then modifies the target style attribute while preserving other unspecified characteristics from the original recording, such as speaker identity, pitch tendency, speaking rhythm, and recording condition. This makes the edited line easier to integrate with surrounding dialogue and reduces the risk of perceptual inconsistency.

10.4 Personalized Digital Voice and Accessibility↩︎

Personalized speech generation is another important use case. Users may want a digital voice that sounds like themselves but can express a wider range of speaking styles than what is available in a short reference sample. AutoSIFT enables this by using a brief recording to capture the user’s voice characteristics, while text prompts specify the desired expressive style. For example, the same voice can be generated in a calm, energetic, or whispering style depending on the application context.

This capability is also relevant to accessibility. For individuals with speech impairments, historical recordings of their own voice can serve as references for preserving personal identity. AutoSIFT can then generate clearer or more controlled speech based on text-specified attributes, while retaining the voice characteristics that make the output personally recognizable. In this way, the model has potential not only for entertainment and content creation, but also for communication tools that support more personal and expressive speech restoration.

11 Statistics of Datasets↩︎

We constructed a comprehensive bilingual, multi-style speech dataset by combining multiple open-source datasets, including Baker [27], LJSpeech [28], ESD [29], CREMA-D [30], CommonPhone [31], and the Genshin Voice dataset [32]. The resulting dataset contains approximately 110k speech samples, totaling 138 hours of speech from 6,361 speakers.

The dataset covers four speaking-style categories: gender, age, emotion, and language. The gender category includes male and female speakers; the emotion category includes happy, angry, sad, neutral, surprise, disgust, and fear; the age category includes child, teenager, young adult, adult, and senior groups; and the language category covers both Chinese and English. For the classification performance evaluation, we adopt a speaker-independent train-test split following the ParaMETA experimental setting. Specifically, the test set contains 28 speakers, 18k samples, and 21.6 hours of speech, while the training set contains 6,333 speakers, 92k samples, and 116.6 hours of speech.

12 Implementation Details↩︎

We implement all models in PyTorch. For all baselines, we carefully tune their hyperparameters following the recommended settings or official implementations when available, and select the best-performing configuration under the same evaluation protocol. All objective metrics are reported as mean \(\pm\) standard deviation over three independent runs with different random seeds. Statistical significance is assessed using a two-sided paired t-test against the strongest baseline for each metric, with \(p<0.05\). For MOS, all values are reported as mean \(\pm\) standard deviation. For fair evaluation, all generated samples from different methods are evaluated by the same frozen category-specific evaluators, which are trained independently from AutoSIFT on held-out real speech and are not updated during generation.

12.0.0.1 Style-controllable TTS backbone.

The style-controllable TTS model adopts Diffusion Transformer (DiT) [23] as its generative backbone. The DiT backbone consists of 12 Transformer blocks with a hidden size of 384 and 6 attention heads, following the scale of DiT-S. The Style Extractor produces a 192-dimensional global style embedding. We optimize the Style Extractor and the TTS backbone for 800k steps using AdamW [37], with a learning rate of \(2 \times 10^{-4}\) and a batch size of 32. Training is conducted on a single NVIDIA RTX 4090 GPU and takes approximately 48 hours.

12.0.0.2 Style Disentangler.

The Style Disentangler takes the 192-dimensional style embedding as input and decomposes it into category-specific subspaces. In our experiments, we consider four text-describable style categories: gender, age, emotion, and language. Accordingly, the disentangler contains four category-specific projectors, each mapping the input style embedding into a 48-dimensional subspace. The Style Disentangler is trained for 200k steps using AdamW [37], with a learning rate of \(1 \times 10^{-3}\) and a batch size of 256. This component is lightweight and can be trained on a laptop NVIDIA RTX 4060 GPU in approximately 2 hours.

12.0.0.3 Arbitrary Style Infiller.

For textual style conditioning, AutoSIFT uses a pre-trained MPNet encoder [33], which outputs a 768-dimensional semantic embedding. This embedding is first projected into a 192-dimensional style space and then decomposed into \(N\) category-specific sub-embeddings, where \(N=4\) in our experiments. Each sub-embedding has 48 dimensions, matching the latent subspaces of the Style Disentangler. The Arbitrary Style Infiller is trained for 200k steps using AdamW [37], with a learning rate of \(1 \times 10^{-3}\) and a batch size of 256. Under the same laptop NVIDIA RTX 4060 setup, training the infiller takes approximately 4 hours.

12.0.0.4 Compute resources.

The main TTS backbone and Style Extractor are trained on a single NVIDIA RTX 4090 GPU. The Style Disentangler and Arbitrary Style Infiller are lightweight auxiliary modules and are trained separately on a laptop NVIDIA RTX 4060 GPU. The total training time is approximately 48 hours for the TTS backbone, 2 hours for the Style Disentangler, and 4 hours for the Arbitrary Style Infiller.

13 Embedding Analysis↩︎

To qualitatively evaluate the effectiveness of the Style Disentangler, we visualize the style embeddings using t-SNE, as shown in Figures 4 to 7.

Comparing the (a) "Raw" and (b) "Disentangled" subfigures across all categories reveals a consistent trend: while the original holistic style embeddings are notably intertwined and lack clear categorical boundaries, the embeddings processed by our Style Decomposer exhibit significantly more discernible clustering behavior. This demonstrates that the model successfully partitions the intertwined style manifold into semantically organized subspaces.

Specifically, for Gender (Fig. 4) and Language (Fig. 5), the disentangled spaces show highly distinct and well-separated clusters with minimal overlap, indicating that the model has captured the robust acoustic signatures of these attributes. For more nuanced categories like Emotion (Fig. 6) and Age (Fig. 7), although the disentanglement is remarkably improved, some degree of proximity remains between semantically related classes. For instance, in the emotion space, happiness and surprise show a partial overlap, which is consistent with the high-arousal paralinguistic features shared by both emotional states. These visualizations provide qualitative evidence that our Prototypical Alignment strategy organizes the latent space into interpretable speaking-style subspaces.

14 Supplementary Audio Demo↩︎

Figure 8 shows a screenshot of the supplementary audio demo page. To support qualitative assessment, we include an anonymous supplementary audio demo page. The demo presents four types of audio samples: reference speech utterances, text-prompted style samples, speech-prompted TTS outputs, and Arbitrary Style Infilling (ASI) results. In the ASI setting, each sample is generated from a text style prompt and a reference speech utterance. The text prompt specifies the target style categories, such as gender, age, or emotion, while the reference speech provides the unspecified categories and residual acoustic information to be preserved. The demo page displays the corresponding style attributes and playable audio clips for each sample. In this case study, we exclude the language category to focus on fine-grained control over gender, age, and emotion. The audio examples provide qualitative evidence that AutoSIFT can perform selective style modification while preserving reference-derived speaking characteristics.

References↩︎