1 Introduction↩︎

Supervised fine-tuning (SFT) is a dominant paradigm for adapting self-supervised pretrained speech models [1]–[3] to downstream classification tasks [4]–[6]. In common practice, researchers compare SFT variants by downstream performance and often treat small gains under a single pretrained checkpoint as evidence that a recipe is “better,” i.e., achieving a higher attainable performance ceiling [7], [8].

This practice implicitly assumes that the relative effectiveness of SFT methods is stable across pretrained model instances. In other words, a recipe that is “better” under one pretrained checkpoint is expected to remain competitive under other checkpoints of comparable capability [9], [10]. However, this assumption overlooks an important interaction in the adaptation process. Downstream performance is jointly shaped by the backbone architecture, the pretraining data, and the SFT recipe [11]. Yet existing evaluations often consider these factors in isolation. In particular, the interaction between a recipe and a specific pretrained instance is rarely examined [12], [13]. As a result, even when the architecture and the model size are held fixed, the apparent superiority of an SFT method may depend on which pretrained checkpoint it is applied to.

We propose to view SFT as a process of capacity elicitation: fine-tuning attempts to activate and reorganize latent knowledge already present in pretrained representations. Under this view, apparent SFT superiority can reflect elicitation match, meaning that a recipe more reliably activates the checkpoint’s usable capacity rather than increasing the capacity itself. Different SFT configurations impose different elicitation biases, which may match some pretrained instances better than others, leading to differences in activation reliability. Consequently, two SFT methods can exhibit ranking discrepancies across checkpoints: Method A may outperform Method B on one pretrained instance, while Method B surpasses Method A on another. Such discrepancies challenge the external validity of conclusions drawn from single-checkpoint comparisons.

To study this effect, we conduct a large-scale empirical investigation of SFT behavior in downstream speech classification. We evaluate eight SFT configurations along two axes: which intermediate representations are used for supervision, and how much of the pretrained network is frozen during adaptation. These configurations are applied to nine widely used self-supervised checkpoints from wav2vec 2.0, HuBERT, and WavLM. Experiments are conducted on three SUPERB classification tasks [14]: intent classification (IC) [15], [16], emotion recognition (ER) [17], [18], and speaker identification (SID) [19], [20]. To characterize run-to-run variability, we further repeat experiments with multiple random seeds on representative base-scale checkpoints.

Our results show that SFT comparisons can depend on the specific pretrained instance. Across tasks, switching checkpoints can change the relative ordering of SFT methods, including cases where architecture and model size are identical and only the pretraining data differs. Multi-seed repetitions further reveal bidirectional transitions between fully activated and under-activated outcomes for the same checkpoint and configuration, which can both expose and mask row-wise top-group differences. Overall, these findings suggest that reported improvements from SFT variants should be interpreted with caution: observed gains may reflect instance- and seed-dependent activation behavior, rather than a universally higher attainable performance ceiling.

This workmakes two contributions.

• We provide systematic empirical evidence that the identity of the best-performing (or top-group) SFT configuration can vary across pretrained instances, even when the backbone architecture and the model size are fixed.

• We introduce the perspective of elicitation match to interpret apparent SFT superiority as differences in activation reliability across pretrained instances, explaining why top-group results on a single checkpoint may not transfer consistently.

2 Problem Setup↩︎

2.1 Problem Statement↩︎

We study pretraining-dependent instability in choosing the top SFT configuration for downstream classification: swapping the pretrained SSL checkpoint (within the same backbone family and scale) can change which SFT configuration is best, or statistically indistinguishable from the best, with all other settings fixed.

2.2 Formal Setup↩︎

Let \(M\) be a pretrained SSL checkpoint, \(T\) a downstream task, and \(s\) a random seed. Each SFT configuration \(a\in\mathcal{A}\) yields \[y=f(a,M,T,s),\] where \(y\) is the task metric reported by the evaluator.

2.3 SFT Configuration Space↩︎

We parameterize configurations by FEATURE_MODE \(m_f\in\{1,2,3\}\) and FREEZE_MODE \(m_z\in\{0,1,2\}\), denoted \(a=(m_f,m_z)\). We evaluate \[\mathcal{A}=\{(1,0),(1,1),(1,2),(2,0),(2,1),(2,2),(3,0),(3,1)\}.\]

2.4 Top-Group Instability Across Checkpoints↩︎

For fixed \((M,T,s)\), let \[\begin{align} y^\star(M,T,s) &= \max_{a\in\mathcal{A}} f(a,M,T,s),\\ a^\star &= \arg\max_{a\in\mathcal{A}} f(a,M,T,s). \end{align}\] Define the top group as configurations not significantly worse than \(a^\star\) under the paired exact McNemar test at level \(\alpha\): \[\mathcal{G}(M,T,s;\alpha)=\{a\in\mathcal{A}:\; a \nsucc a^\star \text{ at level }\alpha\}.\] We can suspect top-group instability occurs between checkpoints \(M\) and \(M'\) if \[\mathcal{G}(M,T,s;\alpha)\neq \mathcal{G}(M',T,s;\alpha).\]

3 Experiments↩︎

3.1 Datasets↩︎

We evaluate downstream speech classification on three tasks from the SUPERB benchmark: intent classification (IC), emotion recognition (ER), and speaker identification (SID). We follow the official SUPERB data splits and evaluation protocols for each task. Unless otherwise stated, we use the default evaluation metrics specified by SUPERB for the corresponding task.

3.2 Backbones↩︎

We consider nine widely used self-supervised speech checkpoints spanning three model families: wav2vec 2.0, HuBERT, and WavLM. Table 1 summarizes each checkpoint, including parameter count and pretraining data. The table includes same-architecture checkpoints pretrained on different data to enable controlled analysis of pretraining-instance dependence.

3.3 Configurations↩︎

Eight SFT configurations are defined by FEATURE_MODE\(\in\{1,2,3\}\) and FREEZE_MODE\(\in\{0,1,2\}\). Each configuration is denoted as \(F m_f\)-\(Z m_z\) (e.g., \(F1\)-\(Z2\) denotes FEATURE_MODE=1 and FREEZE_MODE=2).

Feature modes. FEATURE_MODE=1 uses the final-layer last_hidden_state as the supervised representation. FEATURE_MODE=2 uses a single selected intermediate layer; we fix the selected layer to the fourth layer from the end, i.e., \(\ell^\star=L-3\). FEATURE_MODE=3 uses a weighted-sum fusion over transformer layers as the supervised representation.

Freeze modes. FREEZE_MODE=0 fine-tunes the entire pretrained network together with the task head. FREEZE_MODE=1 freezes only the convolutional front-end and fine-tunes all transformer layers plus the task head. FREEZE_MODE=2 freezes the convolutional front-end and the first \(N\) transformer layers, fine-tuning the rest and the task head (\(N=4\) in all runs).

3.4 Training Setup↩︎

Our training recipe follows the official SUPERB downstream configurations for IC, ER, and SID as closely as possible. All downstream fine-tuning and evaluation use the official SUPERB training pipeline and built-in evaluator, with identical scripts across runs; only the pretrained checkpoint and random seed are varied. For each task, we keep the entire pipeline fixed across all SSL checkpoints and SFT configurations, including data preprocessing, batching, and validation-based checkpoint selection. Task-specific hyperparameters (optimizer, learning-rate schedule, batch size, and training duration) are taken directly from the corresponding SUPERB configuration files and held constant within each task. Each experiment is run on a single NVIDIA H20 GPU, totaling approximately 10,000 GPU-hours across all runs.

3.5 Seed Protocol↩︎

We use a two-level seed design. For the full experimental matrix covering all nine SSL checkpoints, we run each (checkpoint, task, SFT configuration) setting with a fixed default seed of 1337. To characterize run-level stochasticity, we additionally perform multi-seed repetitions on three representative base-scale checkpoints: facebook/wav2vec2-base-960h, facebook/hubert-base-ls960, and microsoft/wavlm-base. For these three checkpoints, we repeat every (task, SFT configuration) setting with two extra seeds, 2048 and 7395, resulting in three seeds in total (1337, 2048, 7395) for each of the corresponding runs.

3.6 Evaluation Outputs and Analysis Targets↩︎

For each run, we report the task-defined metric using the official SUPERB evaluation scripts. The experimental matrix produces scores indexed by (checkpoint, task, SFT configuration, seed). All result aggregation, top-group identification via paired exact McNemar tests, and checkpoint- and seed-sensitivity analyses (including under-activation and pairwise sign flips) are presented in Section 4.

4 Results↩︎

Each row in Tables 2 and 3 corresponds to a fixed downstream task and a fixed pretrained checkpoint, under which we compare the eight SFT configurations in Section 3. We mark systems that are not significantly worse than the row-wise best as the top group, i.e., statistically indistinguishable from the best under the paired exact McNemar test [21] at \(\alpha=0.05\).

4.1 Why the differences are not ceiling improvements↩︎

At a glance, bold entries are not uniformly distributed across columns, and some SFT configurations enter the top group more frequently across pretrained checkpoints. A naive interpretation is that these configurations have an intrinsically higher attainable performance ceiling. However, several observations are inconsistent with a ceiling-improvement explanation.

First, configurations that frequently enter the top group can still exhibit under-activation on certain checkpoints, producing abnormally low performance on the same task. If the advantage primarily came from a systematically higher ceiling, one would expect the relative ordering to be more consistent and the “best” configurations to rarely collapse. Instead, extreme failures among apparent winners indicate that the observed gaps are not simply explained by different upper bounds.

Second, Table 3 shows that the effect is, to some extent, seed-dependent and bidirectional. For the same task, pretrained checkpoint, and SFT configuration, changing only the seed can occasionally turn an under-activated run into a fully activated one that reaches the row-wise top group statistically indistinguishable from the best. Conversely, a configuration that is in the top group under one seed can become under-activated (or markedly degraded) under another seed, without changing data, hyperparameters, or training mode. Such bidirectional transitions are difficult to reconcile with a stable ceiling-improvement story, but are consistent with sensitivity to the optimization trajectory.

Finally, for the hubert-large-ll60k checkpoint on the ER task, all eight configurations achieve statistically indistinguishable performance, placing the entire row in the top group. This limiting case indicates that, for some pretrained instances under our protocol, the choice among these SFT configurations has limited marginal effect on the final metric, further weakening the claim that certain configurations provide a generally higher ceiling.

4.2 Why the differences reflect activation matching↩︎

Taken together, the results are better explained by an activation matching perspective: different SFT configurations differ primarily in how reliably they can activate the latent capability of a given pretrained checkpoint under a particular training stochasticity, rather than in how high they can ultimately go.

The non-uniform bold-entry distribution then reflects differences in activation success rate. A configuration that appears in the top group more often is one that more frequently reaches a high-performing solution region for many checkpoints under the default recipe. Under-activation corresponds to a failure to enter that region for a specific checkpoint–seed combination, even for configurations that otherwise succeed often.

The seed results provide direct support for this view. The fact that activation can be flipped from failure to success (and vice versa) by changing only the seed suggests that the main variability lies in whether optimization finds a suitable basin that aligns the SFT configuration with the pretrained representation. When activation succeeds, multiple configurations can reach comparable top-group performance for the same checkpoint, implying that the primary distinction is how easy it is to get there rather than a systematically higher attainable ceiling.

Notably, the abnormally low scores marked as convergence anomalies (*) do not stem from accidental training interruptions: all runs complete the full training pipeline under the same configuration. This further confirms that poor performance primarily reflects a failure of the SFT procedure to successfully activate the pretrained model’s usable capacity.

5 Conclusion↩︎

Across nine SSL checkpoints and eight SFT configurations, we show that selecting the statistically indistinguishable top-group SFT recipe is pretrained-instance dependent. Even with fixed architecture and scale, the best-performing configuration can change across checkpoints, and pairwise orderings may flip as a secondary effect. Multi-seed results further reveal this fragile activation, where the same recipe can alternate between fully activated and under-activated outcomes due to optimization stochasticity. Overall, apparent SFT gains often reflect instance and seed chdependent activation reliability rather than universally better recipes, motivating evaluation across multiple checkpoints and seeds.

6 Acknowledgments↩︎

This work was supported by JST BOOST, Grant Number JPMJBS2407.

7 Generative AI Use Disclosure↩︎

Generative AI tools (Gemini and ChatGPT) were used for language editing and improving the phrasing of this manuscript.

References↩︎