1 Introduction↩︎

Large language models (LLMs) have recently been extended beyond text to incorporate audio perception, enabling progress in tasks such as classification, captioning, and question answering [1], [2]. Among these, AudioQA [3] has attracted increasing attention as a unified data format for training Audio LLMs (ALLMs), consolidating diverse audio tasks into a single framework and supporting broad application scenarios from assistive technology to human–computer interaction. While recent efforts have integrated pretrained audio encoders into LLMs to build general-purpose AudioQA models [4], [5], scaling across datasets with disparate domains, annotation styles, and difficulty levels introduces significant yet often overlooked training inefficiencies.

Multi-task learning [6] provides a natural theoretical lens for generalization across diverse datasets, where in practice each dataset is often treated as an instantiation of a task. Extensive explorations have been conducted in traditional machine learning [7], [8], encompassing investigations of curriculum learning [9], task sampling [10], and various training paradigms [11]. However, these approaches typically rely on task-specific assumptions, architectural modifications, or auxiliary training mechanisms, limiting their applicability to general ALLMs.² Consequently, most existing ALLM training pipelines resort to naive strategies such as uniform dataset mixing, and fail to account for substantial data heterogeneity. Although such mixing ensures task coverage, it often induces suboptimal training dynamics characterized by redundant updates and slow convergence. These inefficiencies are further exacerbated in ALLMs by the inherent complexities of long audio sequences, cross-modal alignment, and the sheer scale of the models.

Recent advances in taskonomy and transferability estimation [12], [13] have established that dataset-level relationships can be successfully quantified. Yet despite this progress, a systematic understanding of how dataset-level heterogeneity governs the training of ALLMs remains elusive. Current analyses are largely restricted to post-hoc evaluations, leaving a gap in understanding how these heterogeneous traits directly impact real-time training dynamics, and more importantly, how they can inform the design of training strategies. Building on this intuition, this work addresses a central question: How can these dataset properties be explicitly leveraged to schedule efficient multi-dataset training for ALLMs, thereby accelerating convergence without architectural overheads?

To address this, we investigate multi-dataset training for ALLMs through the lens of convergence analysis. We begin by theoretically analyzing the convergence behaviors of different training regimes, focusing on how gradient variance and dataset heterogeneity affect training dynamics. Our analysis reveals the relative advantages of parallel versus sequential training under differing heterogeneity degrees, which then motivates our proposal of Grouped Sequential Training (GST). As a paradigm that interpolates between parallel and sequential optimization, GST organizes datasets into groups with reduced intra-group heterogeneity. By executing sequential updates at the group level, this strategy yields provably tighter convergence bounds in realistic multi-dataset settings. For practical implementation, we cluster datasets using metrics that capture optimization-level disparities, such as gradient-based distance and empirical transferability. To further scale the framework to large-scale ALLMs, we introduce an efficient progressive training procedure that approximates grouped sequential optimization while minimizing computational overhead and mitigating catastrophic forgetting. Importantly, our approach is model-agnostic and requires no architectural modifications. Extensive evaluations across 14 AudioQA datasets demonstrate that our method significantly accelerates convergence compared to naive multi-task baselines, while simultaneously maintaining or enhancing performance across diverse domains. Our contributions can be summarized as follows:

• Theoretical Framework: We establish a convergence analysis for ALLMs that rigorously characterizes the impact of gradient variance and dataset heterogeneity.

• Paradigm Shift: We propose Grouped Sequential Training (GST), which interpolates between parallel and sequential optimization and moves dataset affinity estimation from post-hoc analysis into training loop design.

• Scalable Implementation: We introduce a progressive training procedure that approximates the optimal grouping strategy, enabling efficient large-scale training without architectural changes.

• Benchmark & Insights: We consolidate 14 datasets into a unified AudioQA format, providing systematic insights and actionable strategies for heterogeneous multi-dataset training.

2 Related Works↩︎

2.1 Audio Large Language Model↩︎

ALLMs have transformed audio-related tasks by demonstrating remarkable zero-shot performance and classification, understanding, and reasoning capabilities. Existing approaches typically adopt either an encoder-only or an encoder-decoder architecture [14]. The encoder-only paradigm employs a dual-tower framework where the text and audio encoders process paired text-audio data. Following the training objective of Contrastive Language-Image Pre-training (CLIP) [15], such models are trained to learn a joint embedding space by drawing the embeddings of paired data closer while pushing apart those of unpaired data.

In parallel, driven by the success of vision-language models, various encoder-decoder ALLMs have been proposed to leverage the large language model for audio comprehension. Prior works such as SALMONN [5], Pengi [16], Qwen-Audio [4], [17], and the Audio Flamingo series [1], [18], [19] have achieved state-of-the-art performances across a wide range of audio tasks. Those methods integrate an audio encoder with a decoder-only language model in a cascade manner. Unlike encoder-only methods that solely rely on contrastive training objectives, encoder–decoder ALLMs are trained with generative objectives, such as masked reconstruction or next-token prediction. This paradigm enables the model to capture richer semantic audio information, facilitating more flexible reasoning and instruction-following abilities.

2.2 Training Strategies for Multi-Dataset Learning↩︎

Training models on multiple datasets is a fundamental challenge across multi-task [6], continual [20], and federated learning [21]. In the context of LLMs, research has increasingly shifted from purely architectural innovations [22], [23] to data-centric optimization strategies. While substantial progress has been made in data selection and sampling weighting [24]–[26] to curate optimal training mixtures, the role of learning scheduling remains underexplored.

Existing paradigms generally fall into two categories: parallel and sequential training (Fig. 1). Parallel training jointly optimizes all datasets via uniform mixing or weighted sampling [27]. While stable, it often suffers from gradient interference [28], where conflicting optimization directions from diverse domains lead to suboptimal convergence. In contrast, sequential training introduces datasets chronologically [29]. While this can prioritize high-quality data to bootstrap representations, it risks catastrophic forgetting of earlier tasks and often lacks a principled basis for determining the optimal sequence. Most theoretical analyses [30], [31], often established within the context of federated optimization, are confined to idealized formulations, focusing on the two extremes: fully parallel or strictly sequential regimes. Strategies that interpolate between these extremes, such as data-driven grouping [7] or stage-wise scheduling [12], often lack systematic, convergence-oriented analyses tailored for dataset-level ALLM training. Furthermore, their post-hoc design and task dependency hinder their integration into the dynamic training loops of large-scale models.

3 Problem Setup↩︎

3.1 Formulation↩︎

Modern ALLMs unify diverse audio understanding tasks through a shared language interface, enabling natural language reasoning over acoustic signals [32]. Consider the multi-dataset training problem for AudioQA. Let \(\mathcal{D}=\{D_m\}_{m=1}^{M}\) denote a collection of \(M\) datasets, where each dataset \(D_m = \{(a_i, x_i, y_i)\}_{i=1}^{N_m}\) consists of audio inputs \(a_i\) and their corresponding textual question-answer pairs \((x_i, y_i)\).

Given a model \(f\) parameterized by \(\Theta\), our objective is to minimize the aggregate loss across all tasks: \[\begin{align} \min_{\Theta}\left\{F(\Theta):=\frac{1}{M}\sum_{m=1}^M F_m(\Theta)\right\}, \end{align}\] where \(F_m(\theta)\) denotes the task-specific loss on \(D_m\), measured by cross-entropy in our ALLM training. Following standard ALLM architectures [5], we initialize \(f\) from a pretrained LLM backbone (e.g., a LLaMA-style decoder) and extend it to the audio modality with audio feature extractors and learnable adapters (e.g., LoRA layers), while the backbone weights maintain frozen.

3.2 Characterizing Dataset Heterogeneity↩︎

Despite sharing a common input-output format, AudioQA datasets exhibit pronounced heterogeneity due to acoustic characteristics, annotation styles, and task complexities. To rigorously quantify this, we adopt the gradient decomposition framework from federated optimization [31], characterizing heterogeneity through gradient inconsistency across datasets: \[\begin{align} \nonumber &\frac{1}{M}\sum_{m=1}^M\; \|\nabla F_m(\Theta) - \nabla F(\Theta)\|^2\\ &\le \beta^2 \|\nabla F(\Theta)\|^2 + \zeta^2, \label{eqn:gvariance} \end{align}\tag{1}\] where the constants \(\beta\) and \(\zeta\) capture the degree of dataset-level divergence. Intuitively, larger values of \(\beta^2\) and \(\zeta^2\) indicate stronger disagreement among gradients derived from different datasets, a phenomenon known to impede convergence in both sequential and parallel optimization regimes. In the context of ALLMs, such gradient inconsistency is not merely an artifact of distribution shifts but reflects intrinsic semantic disparities. For example, datasets emphasizing temporal localization may induce gradients that prioritize fine-grained acoustic cues, whereas captioning datasets favor global semantic representations. These differences persist even when datasets are projected onto a unified language output space, necessitating a more sophisticated training strategy than naive mixing.

4 Methodology↩︎

4.1 Sequential and Parallel Training under Dataset Heterogeneity↩︎

To analyze the impact of dataset heterogeneity, we first establish a theoretical foundation, measuring convergence by the minimum expected gradient norm across \(R\) training rounds: \[\begin{align} \mathcal{C}(R):=\min_{0 \le r \le R} \mathbb{E}\big[\|\nabla F(\Theta^{(r)})\|^2\big], \end{align}\] where the convergence rate is characterized by the decay of \(\mathcal{C}(R)\) with respect to \(R\). We then adapt the theoretical findings from federated learning [31] to the multi-dataset setting by treating each dataset as a distinct “client", simplifying the complex distributed dynamics into two primary factors - stochastic gradient variance and dataset heterogeneity. Specifically, we have the following assumptions and results:

AS1 (Smoothness): Each dataset-specific objective \(F_m\) is \(L\)-smooth.

AS2 (Bounded Variance): The gradient computed on each dataset has a variance bounded by \(\sigma^2\).

AS3 (Bounded Heterogeneity): Dataset-level heterogeneity is bounded by constants \(\beta\) and \(\zeta\) as defined in Eqn. 1 .

Comparing the bounds in Lemma 1 reveals a fundamental trade-off. Parallel training benefits from joint sampling, which stabilizes optimization and reduces the effective \(\sigma^2\). However, it is fully exposed to the global heterogeneity terms \(\beta^2\) and \(\zeta^{2/3}\). In contrast, sequential training suppresses these heterogeneity effects by factors of \(1/M\) and \(1/M^{1/3}\), effectively “decoupling" the interference between divergent datasets at the cost of losing joint variance reduction. Consequently, while parallel mixing is robust for mild heterogeneity, the sequential regime becomes theoretically superior as dataset divergence increases - as is common in complex ALLM tasks. This observation suggests that neither extreme is globally optimal, motivating a strategy that balances variance reduction with heterogeneity suppression.

4.2 Grouped Sequential Training (GST)↩︎

AudioQA datasets often exhibit structured heterogeneity. For instance, datasets involving music perception share closer optimization trajectories than those requiring speech comprehension. To capture this structure and achieve the “best of both worlds" discussed in Sec. 4.1, we propose to partition the \(M\) datasets into \(K\) affinity-based groups \(\{\mathcal{G}_k\}_{k=1}^K\) and conduct Grouped Sequential Training (GST), a hybrid paradigm that leverages local variance reduction while maintaining global heterogeneity suppression (Fig. 1). For each group, the group-level objective is defined as: \[\begin{align} F^{(k)}(\Theta) :=\frac{1}{M_k} \sum_{m \in \mathcal{G}_k} F_m(\Theta), \end{align}\] where \(M_k\) is the number of datasets in \(\mathcal{G}_k\) and \(F^{(k)}\) inherits \(L\)-smoothness from its constituent datasets. The global gradient variance can thus be decomposed as follows:

Following this decomposition, we characterize heterogeneity bound at two levels: intra-group \((\beta_k, \zeta_k)\) and inter-group \((\beta_g, \zeta_g)\). Analogous to Eqn. 1 , they satisfy: \[\begin{align} \frac{1}{M_k} \sum_{m \in \mathcal{G}_k} &\|\nabla F_m(\Theta) - \nabla F^{(k)}(\Theta)\|^2 \nonumber\\ \le \;\beta_k^2& \|\nabla F^{(k)}(\Theta)\|^2 + \zeta_k^2, \tag{2} \\ \frac{1}{K}\sum_{k=1}^K&\|\nabla F^{(k)}(\Theta)-\nabla F(\Theta)\|^2 \nonumber\\ \le \;\beta_g^2 &\|\nabla F(\Theta)\|^2 + \zeta_g^2. \tag{3} \end{align}\]

We establish the relationship between group-wise and global heterogeneity from Prop. 1:

The proof is given in Appendix 8.1.2.

Based on Lemma 2, GST satisfies the three fundamental optimization assumptions: (i) \(F^{(k)}\) inherits \(L\)-smoothness; (ii) the stochastic gradient variance within groups is bounded by \(\sigma_k^2<\sigma^2\) guaranteed by the bound of intra-group heterogeneity \((\beta_k, \zeta_k)\);³ and (iii) inter-group heterogeneity is bounded by \((\beta_g, \zeta_g)\). We then apply the convergence results at the group level and have:

Theorem 1 identifies GST as a superior middle ground. By optimizing over \(K\) groups rather than \(M\) datasets, GST suppresses the impact of inter-group heterogeneity by factors of \(1/K^{1/3}\) and \(1/K\). Crucially, as shown in Lemma 2, proper grouping ensures that the inter-group divergence \((\beta_g, \zeta_g)\) remains a controlled fraction of the global heterogeneity. Meanwhile, the joint training within each group retains the variance reduction benefit (\(\sigma_k < \sigma\)) typical of parallel mixing. Consequently, the GST bound is strictly tighter than that of fully parallel training in Eqn. ?? , which is hindered by large global heterogeneity, and fully sequential training in Eqn. ?? , which suffers from unmitigated stochastic noise.

4.3 Affinity-Aware Grouped Training↩︎

To implement the GST strategy, we require an affinity-aware mechanism to partition datasets. Following our convergence analysis, our primary objective is to cluster datasets with high optimization affinities to minimize the intra-group divergence \((\beta_k, \zeta_k)\). This ensures that the parallel mixing within each group remains stable, while the residual inter-group heterogeneity is effectively compressed through the sequential stages of GST.

4.3.0.1 Gradient-Based Affinity.

We quantify affinity via the pairwise distance between dataset-level gradients. Based directly on the notion of heterogeneity in Eqn. 1 , we define the dataset distance as: \[\begin{align} Dist(T_m,T_n)=\|\nabla F_m(\Theta) - \nabla F_n(\Theta)\|^2, \label{eq:gradient} \end{align}\tag{4}\] where \(\sum_{m,n=1}^M \|\nabla F_m(\Theta) - \nabla F_n(\Theta)\|^2=M\cdot\sum_{m=1}^M\; \|\nabla F_m(\Theta) - \nabla F(\Theta)\|^2\).

A smaller distance signifies a higher affinity, indicating that the datasets share consistent optimization trajectories. In practice, we estimate \(Dist(D_m, D_n)\) through a controlled probing procedure: starting from a common initialization, we finetune the model on each dataset for a fixed number of steps and accumulate the gradients. We then apply spectral clustering on the resulting distance matrix to form the task groups \(\{\mathcal{G}_k\}\).

4.3.0.2 Alternative: Empirical Transferability.

We also consider an empirical transferability based criterion [12] as a complementary baseline.⁴ This measure evaluates affinity by observing cross-dataset performance gains: \[\begin{align} &Trf(D_m\to D_n) = \frac{F_n(\Theta_m)-F_n(\Theta_n)}{F_n(\Theta_n)}. \label{eq:taskonomy} \end{align}\tag{5}\] Here \(\Theta_m\), \(\Theta_n\) denote the model parameters after training on \(D_m\) and \(D_n\) respectively. While intuitive, this metric is computationally expensive (requiring \(M^2\) testing pairs) and less directly coupled with the optimization dynamics in Theorem 1. We thus primarily use it for comparative analysis.

4.3.0.3 Training Implementation: From Sequential to Progressive.

While Theorem 1 assumes strict sequential transitions, repeatedly cycling through groups after short update intervals can present significant practical hurdles to large-scale ALLM training, entailing catastrophic forgetting and prohibitive data loading overhead. To bridge this gap, we adopt in practice a Progressive Grouped Training schedule: we begin with group \(\mathcal{G}_1\) and incrementally expand the training pool to \(\bigcup_{j=1}^{k+1} \mathcal{G}_j\) at each subsequent stage. This progressive expansion serves as an efficient approximation of GST; it mitigates initial gradient conflicts through sequential introduction while maintaining the stability and I/O throughput of joint mixing. Consequently, this hybrid regime balances the variance reduction of parallel training with the heterogeneity suppression of the sequential regime.

5 Experiments↩︎

3pt

5.1 Experimental Setup↩︎

We empirically evaluate our proposed data-level training strategy to verify if the theoretical advantages of GST translate into superior performance and convergence in ALLM training.

5.1.0.1 Model Architecture.

We select SALMONN-13B [5] as our experimental platform. While newer models have emerged, SALMONN remains a highly representative and open-source foundation for ALLMs, featuring a standard architecture that combines a Whisper encoder with a Vicuna-13B backbone. Its well-documented training pipeline makes it an ideal testbed for isolating the effects of dataset scheduling from architectural innovations. We strictly follow the official configuration to ensure all gains stem solely from GST strategy. The performance is then evaluated by token-level prediction accuracy.

5.1.0.2 AudioQA Dataset Formulation.

We evaluate our method on 14 diverse datasets covering speech, music, and environmental sounds. Following [3], all tasks are unified into an AudioQA format. This collection serves as a benchmark for structured heterogeneity in general audio perception, with significant variations in annotation style and domain-specific knowledge. Dataset descriptions are detailed in Appendix 8.2.

5.1.0.3 Training Strategies.

We evaluate several canonical dataset scheduling strategies. To ensure a rigorous controlled experiment, all methods share identical model architectures, optimization settings, and training budgets, differing only in their dataset organization:⁵ Parallel Training: Jointly optimizes by uniformly sampling data from all 14 datasets at each iteration; Sequential Training: Optimizes datasets one at a time in a fixed order, representing a zero-mixing extreme; Independent Training: Finetunes a separate model for each dataset, serving as an empirical upper bound for single-task performance; GST (Proposed): Our affinity-aware strategy. To explore the granularity and criteria of grouping, we evaluate three variants, i.e. GST-T2, Taskonomy-based grouping (\(K=2\)) using empirical transferability (Eqn. 5 ), and GST-G2 / G3, Gradient-based grouping (\(K=2\) / \(K=3\)) using gradient distance (Eqn. 4 ). Unless otherwise noted, we employ the Progressive implementation for stability.

3pt

5.2 Main Results↩︎

We evaluate training strategies under two distinct settings to comprehensively assess their impact on optimization:

Full-Data Training: Utilizing all available data and training from scratch, this setting emulates the large-scale joint training phase typical in building foundation models.

Low-Resource Finetuning: Capping each dataset at \(N=250\) samples and initializing from SALMONN checkpoints, this setting emulates supervised finetuning of models with scarce data.

Tab. 1 summarizes the performance across 14 AudioQA datasets. We report the raw testing accuracy for each task, along with the cumulative time elapsed (\(\mathcal{T.E.}\)) and two aggregate metrics: the unweighted average accuracy (Avg), and the sample-size weighted average accuracy (W.Avg).

5.2.1 Performance on Full-Data Training↩︎

As illustrated in Tab. 1, the performance of GST strategies in full-data training is characterized by:

Superior Performance. Our proposed GST variants consistently outperform Parallel Training (Mix-all). Specifically, GST-T2 achieves the highest unweighted average, while GST-G3 leads in the weighted average. Notably, all variants significantly exceed the Original SALMONN baseline and closely approach the Independent Training reference (the empirical upper bound), underscoring the validity of our experimental comparison.

Robustness Against Forgetting. Compared to Sequential Training, which suffers from a drastic drop in accuracy due to catastrophic forgetting, our GST implementation maintains a high performance ceiling, confirming that the progressive introduction of groups serves as an effective buffer.

Enhanced Optimization Efficiency. A key highlight of our results is the significant reduction in training time. While the parallel baseline requires approximately 4 days to reach convergence, GST variants complete the process within 2–3 days. This 30–40% speedup in large-scale training empirically validates the tighter convergence bounds derived in Theorem 1.

5.2.2 Stability in Low-Resource Scenarios↩︎

In the low-resource regime, the primary challenge shifts from managing massive heterogeneity to ensuring sample efficiency. We omit the Sequential baseline here as it leads to severe forgetting with sparse data, yielding non-informative results. Findings for the remaining strategies are as follows:

Performance Stability. In the low-resource regime, GST variants achieve comparable performance to the Parallel baseline. This result notably demonstrates that the group-wise sequential nature of GST does not compromise the model’s adaptation capability even under extreme data sparsity.

Diminishing Acceleration Returns. Unlike the full-data regime, the training acceleration effect is less pronounced here, with marginal reductions in wall-clock time. This phenomenon can be attributed to a trade-off between heterogeneity suppression and early-stage overfitting: when training on a small initial group \(\mathcal{G}_1\), the model may rapidly overfit to these sparse samples before the data pool expands. Consequently, the potential time gains from reduced gradient variance are partially offset by the optimization overhead required to stabilize learning on such limited data.

5.3 Discussion↩︎

5.3.1 Convergence Behaviors↩︎

To further investigate how GST features in optimization dynamics, we plot in Fig. 2 the training of three representative strategies: Mix-all, GST-G3 (Strict Sequential), and GST-G3 (Progressive). As depicted in the curves, Strict Sequential training exhibits a “volatile" pattern; while performance peaks at the end of each stage, it suffers drastic drops upon returning to previous data groups (e.g., from Stage 3 back to Stage 1), characterizing classic catastrophic forgetting. This instability forces the model to undergo significantly more training cycles to reach a stable state. In contrast, Progressive GST maintains a steady and monotonic improvement in test accuracy, proving the effectiveness of our progressive implementation. Compared with Mix-All, Progressive GST also reaches a stable performance plateau slightly earlier (approx. epoch 25 vs. epoch 27). More importantly, the per-epoch training time for Mix-all is substantially longer than that of GST. We attribute this to the high gradient heterogeneity in Mix-all, which forces the optimizer to reconcile conflicting update directions within every batch, leading to higher computational overhead and slower stable descent.

5.3.2 Impact of Group Ordering: The Stability-First Curriculum↩︎

To examine how the temporal sequence of dataset groups influences the optimization trajectory, we compare three scheduling strategies: Strict Cycle Sequential, Progressive, and Reverse Progressive (see Tab. 2). Our analysis reveals that the effectiveness of GST is highly sensitive to the initial training phase and the data retention mechanism. The superiority of the positive Progressive order over the Reverse variant underscores a "stability-first" curriculum effect. When high-variance or outlier tasks are introduced prematurely (as in the Reverse order), the model encounters a "noisy" initialization phase that is both computationally expensive and optimizationally destructive. A complex initial group not only requires excessive epochs to reach marginal convergence but also forces the model into suboptimal regions of the loss landscape. Furthermore, the poor performance of Strict Cycle Sequential highlights the danger of abrupt distribution shifts introduced by the original sequential strategy. These results confirm that an affinity-aware initialization and optimization stability are essential for navigating the complex heterogeneity of multi-dataset ALLM training.

6 Conclusion↩︎

In this paper, we introduced Grouped Sequential Training (GST), a principled optimization framework designed to mitigate the challenges of gradient heterogeneity in multi-dataset ALLM training. By strategically grouping datasets based on affinity measures and introducing them via a progressive scheduling protocol, GST effectively balances inter-dataset interference and optimization efficiency. Theoretically, we derived tighter convergence bounds for GST, demonstrating that reducing intra-group gradient variance directly accelerates stable descent. Empirically, evaluations across 14 diverse AudioQA datasets show that GST variants not only outperform parallel and sequential training baselines but also achieve a 30–40% reduction in training time. Our findings underscore the importance of affinity-aware scheduling in large-scale joint training and provide a scalable, interpretable pathway for developing robust ALLMs.

7 Limitations↩︎

While the proposed GST framework demonstrates significant acceleration and stability in ALLM training, we acknowledge several limitations that open avenues for future research.

Diversity of Backbones and Frameworks. Our current evaluation is primarily conducted using the SALMONN framework with a Vicuna-13B backbone. Although these are representative of state-of-the-art audio-language models, the scalability of GST to even larger models (e.g., 70B parameters) or different architectural paradigms (e.g., MoE-based audio LLMs) remains to be extensively verified. Future work will investigate whether the gradient affinity patterns observed here persist across diverse model scales and initialization states.

Bottleneck Analyses in Mix-all Training Efficiency. We observed that the Mix-all baseline exhibits slower per-epoch training speeds compared to partitioned strategies. Our profiling indicates this is not merely due to data loading overhead, but stems from gradient variance explosion in the early stages of training. When heterogeneous datasets are shuffled into a single stream, the frequent directional shifts in gradients (\(\nabla F_m\)) lead to unstable optimizer states and more rejected updates in the loss landscape. GST mitigates this by grouping consistent signals, but a more rigorous theoretical quantification of this "mixing penalty" is still needed.

Refinement of the "Stability-First" Ordering. Our Stability-First ordering strategy (beginning with high-affinity, low-variance groups) is based on the heuristic that establishing a robust representation early on prevents computational waste and optimizational destruction. However, a purely optimal ordering might require a more dynamic approach. The current static ordering, while effective, does not account for the temporal evolution of dataset relationships—as the model learns, two datasets that were initially conflicting might become synergistic. Exploring a dynamic, feedback-driven ordering mechanism is a promising direction.

8 Appendix↩︎

8.1 Theoretical Proof↩︎

8.1.1 Proof of Prop. 1 (Heterogeneity Decomposition)↩︎

For any dataset \(m\) belonging to group \(\mathcal{G}_k\), we decompose the deviation from the global mean by inserting the group mean \(\nabla F^{(k)}\): \[\begin{align} &\|\nabla F_m - \nabla F\|^2 \\&= \|(\nabla F_m - \nabla F^{(k)}) + (\nabla F^{(k)} - \nabla F)\|^2 \end{align}\] Expanding the squared norm: \[\begin{align} &\|\nabla F_m - \nabla F\|^2\\ &= \|\nabla F_m - \nabla F^{(k)}\|^2 + \|\nabla F^{(k)} - \nabla F\|^2 \\&+ 2\langle \nabla F_m - \nabla F^{(k)}, \nabla F^{(k)} - \nabla F \rangle \end{align}\] Summing over all datasets \(m \in \{1, \dots, M\}\): \[\begin{align} &\sum_{M=1}^m \|\nabla F_m - \nabla F\|^2 \\ &=\sum_{k=1}^K \sum_{m \in \mathcal{G}k} \|\nabla F_m - \nabla F\|^2 \\&= \sum_{k=1}^K \sum_{m \in \mathcal{G}k} \|\nabla F_m - \nabla F^{(k)}\|^2 \\&+ \sum_{k=1}^K \sum_{m \in \mathcal{G}k} \|\nabla F^{(k)} - \nabla F\|^2 + \text{Cross Terms} \\ &=\sum_{k=1}^K \sum_{m \in \mathcal{G}k} \|\nabla F_m - \nabla F^{(k)}\|^2 \\&+ M_k\sum_{k=1}^K \|\nabla F^{(k)} - \nabla F\|^2+ \text{Cross Terms}, \end{align}\] where \(\text{Cross Terms}=0\), due to: \[\begin{align} &2 \sum_{m \in \mathcal{G}k} \langle \nabla F_m - \nabla F^{(k)}, \nabla F^{(k)} - \nabla F \rangle \\&= 2 \langle \underbrace{\sum_{m \in \mathcal{G}k} (\nabla F_m - \nabla F^{(k)})}_{\mathbf{0}}, \nabla F^{(k)} - \nabla F \rangle = 0. \end{align}\] Therefore, we have \[\begin{align} \small &\frac{1}{M} \sum_{m=1}^M \|\nabla F_m - \nabla F\|^2 \\ = &\frac{1}{M} \sum_{k=1}^K \sum_{m \in \mathcal{G}_k} \|\nabla F_m - \nabla F^{(k)}\|^2 \\+& \frac{1}{M} \sum_{k=1}^K M_k \|\nabla F^{(k)} - \nabla F\|^2. \end{align}\]

8.1.2 Proof of Lemma 2 (Heterogeneity Bound)↩︎

For intra-group heterogeneity, we have: \[\begin{align} &\frac{1}{K}\sum_{k=1}^K\|\nabla F^{(k)}(\Theta)-\nabla F(\Theta)\|^2 \nonumber\\ &\le\frac{M}{KM_{min}}\cdot\frac{1}{M}\sum_{k=1}^KM_k\|\nabla F^{(k)}(\Theta)-\nabla F(\Theta)\|^2 \nonumber \\ &\le \frac{M\beta^2}{KM_{min}} \|\nabla F(\Theta)\|^2 + \frac{M\zeta^2}{KM_{min}} \nonumber\\ &- \frac{1}{KM_{min}}\sum_{k=1}^K\sum_{m \in \mathcal{G}_k}\|\nabla F_m(\Theta) - \nabla F^{(k)}(\Theta)\|^2. \label{eqn:variance95decompose} \end{align}\tag{6}\] Comparing Eqn. 6 with the form of bound in Eqn. 3 , we have \(\zeta_g^2<\frac{M\zeta^2}{KM_{min}}- \Gamma_{\mathcal{G}}\) and \(\beta_g^2<\frac{M\beta^2}{KM_{min}}-\Delta_{\mathcal{G}}\), with \(M_{min}\) denoting the smallest group size and \(\Gamma_{\mathcal{G}}, \Delta_{\mathcal{G}} \ge 0\) are the residual terms representing the heterogeneity absorbed within groups via affinity-based alignment.

8.1.3 Proof of Stochastic Gradient Variance↩︎

Noting the stochastic gradient on dataset m as \(g_m(\Theta)\) and stochastic gradient in \(\mathcal{G}_k\) as \(g^{(k)}(\Theta)\), for the intra-group stochastic gradient variance, we have: \[\begin{align} &\|g^{(k)}-\nabla F^{(k)}\|^2\\ &=\|\frac{1}{M_k}\sum_{m\in\mathcal{G}_k}(g_m-\nabla F_m) \\&+ \frac{1}{M_k}\sum_{m\in\mathcal{G}_k} (\nabla F_m-\nabla F^{(k)})\|^2\\ &\le \|\frac{1}{M_k}\sum_{m\in\mathcal{G}_k}(g_m-\nabla F_m)\|^2\\& + \|\frac{1}{M_k}\sum_{m\in\mathcal{G}_k} (\nabla F_m-\nabla F^{(k)})\|^2, \end{align}\] with \[\begin{align} &\mathbb{E}\|\frac{1}{M_k}\sum_{m\in\mathcal{G}_k}(g_m-\nabla F_m)\|^2\\ &\le\frac{1}{M_k^2} \sum_{m\in\mathcal{G}_k}\mathbb{E}\|g_m-\nabla F_m\|^2 \le \frac{\sigma^2}{M_k},\\ &\|\frac{1}{M_k}\sum_{m\in\mathcal{G}_k} (\nabla F_m-\nabla F^{(k)})\|^2\\ &\le \frac{1}{M_k}\sum_{m\in\mathcal{G}_k}\|\nabla F_m-\nabla F^{(k)}\|^2\\ & \le \beta_k^2\|\nabla F^{(k)}\|^2+\zeta_k^2. \end{align}\] Hence, we have \[\begin{align} &\mathbb{E}\left[\|g^{(k)}-\nabla F^{(k)}\|^2\right] \\ &\le \frac{\sigma^2}{M_k}+\beta_k^2\|\nabla F^{(k)}\|^2+\zeta_k^2=\sigma_k^2. \end{align}\] Here \(\sigma_k\) is given by \(\frac{\sigma^2}{M_k}\) and intra-group heterogeneity \((\beta_k,\zeta_k)\), and \(\sigma_k<\sigma\) can be ensured with bounded intra-group heterogeneity.

8.2 Dataset Description↩︎

To evaluate the proposed GST framework across diverse acoustic scenarios, we curate a comprehensive benchmark consisting of 14 datasets covering ambient sounds, speech, and music. These datasets are categorized into three primary types of tasks: Audio Captioning (CAP), Audio Question Answering (AQA), and Audio Classification (CLS). For classification tasks, we follow a standard “Classification to QA" prompting strategy to ensure compatibility with LLM instruction-following. For classification tasks, we follow a standard”Classification to QA" prompting strategy to ensure compatibility with LLM instruction-following. Specifically, we list the datasets below:

• Ambient & General Audio: We include AudioCaps [33], Clotho_v2 [34], WavCaps [35], and MACS [36] for general audio captioning. SoundDescs [37] provides fine-grained descriptions, while OpenAQA [3] (derived from AudioSet [38]) offers large-scale temporal audio reasoning.

• Acoustic Scenes: ChimeHome [39] and CochlScene [40] are used for scene recognition. To adapt these CLS datasets for ALLM training, we hard-code labels into natural language QA pairs (e.g., "Where is the audio recorded?" \(\rightarrow\) "The audio is recorded in a car.").

• Music Analysis: MusicQA [41], MusicNet [42], and Jamendo [43] cover music captioning, ensemble recognition, and mood analysis.

• Speech & Emotion: IEMOCAP [44] is utilized for emotion recognition using SALMONN annotations. PromptSpeech [45] and TextrolSpeech [46] provide speech-to-captioning tasks focusing on speaker traits and style.

8.3 Details of Training Settings↩︎

Model Architecture & Configuration. We build our model based on the SALMONN framework. The architecture consists of a dual-encoder system: a Whisper-Large-v2 for speech feature extraction and a BEATs (iter3+) for audio semantic encoding. Both encoders are kept frozen during training to preserve their pre-trained acoustic representations. A window-level Q-Former is employed to aggregate audio features with a window size and stride of 0.33 seconds, compressing the sequence into a single query token per window.To efficiently adapt the large language model, we utilize Vicuna-13B-v1.1 as the backbone and integrate Parameter-Efficient Fine-Tuning (PEFT) via LoRA. The LoRA adapters are applied with a rank \(r=8\) and \(\alpha=32\), significantly reducing the number of trainable parameters while maintaining competitive performance.

Optimization Details. The model is optimized using the AdamW optimizer with a weight decay of 0.05 and \(\beta_2=0.999\). We employ a learning rate schedule starting with a warmup of 3,000 steps, increasing from 1e-6 to a peak of 3e-5, followed by a cosine decay to a minimum of 1e-5. The training is conducted over a maximum of 30 epochs, with an early stopping patience of 5 epochs based on validation loss to prevent overfitting.

Hardware & Distributed Training. Our training infrastructure and hyperparameter settings vary based on the data scale:

• Full-resource Training: Conducted on a cluster of 4 \(\times\) NVIDIA A100 (80GB) GPUs. We use a batch size of 4 per GPU with an accumulation step of 1, resulting in an effective global batch size of
• Low-resource Training: For specific ablation studies, we use 2 \(\times\) NVIDIA A100 GPUs, with the batch size per GPU and other configurations kept consistent, effectively halving the global batch size to 8.

All experiments are implemented using the Singularity container platform for environment consistency. Automatic Mixed Precision (AMP) is enabled to accelerate training and reduce memory footprint.

To evaluate the model’s performance and convergence efficiency across the multi-dataset benchmark, we employ Token-level Accuracy as our primary metric. This metric is defined as the ratio of correctly predicted tokens to the total number of target tokens in the sequence: \[\text{Acc}_{token} = \frac{\sum \text{Correct Tokens}}{\sum \text{Total Tokens}}\] Unlike sentence-level metrics which can be overly sensitive to minor variations in decoding, Token-level Accuracy provides a high-resolution view of the model’s optimization progress. It directly reflects the model’s ability to minimize the cross-entropy loss across heterogeneous datasets, making it an ideal proxy for analyzing the convergence behaviors of different training protocols such as Mix-all and GST.

8.4 Interpreting Dataset Relationships: Gradient vs. Taskonomy-Based Metrics↩︎

Comparison of Metric Characteristics. As shown in Fig. 3, both metrics capture certain aspects of dataset interdependencies. While showing some coarse consistency, it is notable that Taskonomy is sensitive to dataset scale. For instance, large-scale datasets like WavCaps (321k) and OpenAQA (213k) tend to negatively affect the learning of other datasets, creating "gravity wells" in the affinity matrix. They also differ significantly in their mathematical and computational properties, as shown in Tab. 4.

The Acoustic-Optimization Gap. To further investigate the "Audio" specificity, we performed a t-SNE visualization on the latent acoustic features extracted from the frozen encoders (Fig. 3).

• Acoustic Clustering: The t-SNE plot reveals clear, distinct clusters aligned with human-defined domains: Music (MusicQA, Jamendo), Speech (IEMOCAP, PromptSpeech), and Environmental sounds (ChimeHome, CochlScene).

• The Discrepancy: Interestingly, neither our Taskonomy-based nor Gradient-based metrics strictly favor these acoustic domains. A music dataset may show higher gradient affinity with a speech-captioning dataset than with another music-tagging task.

This discrepancy indicates that the AudioQA relationship is influenced by factors far more complex than raw acoustic similarity. In a generative ALLM framework, the optimization direction \(\nabla F_m\) is a joint product of Acoustic spectral features (the "what" is being heard), Instruction complexity and label distribution (the "how" it is being answered), and Backbone alignment state (the current mapping of audio to Vicuna tokens). While t-SNE captures the "acoustic appearance", our Gradient-based metric captures the "optimization essence".

References↩︎

---

1. Corresponding Author↩︎

2. See Sec. 2.2 for further discussion.↩︎

3. Refer to Appendix 8.1.3 for the proof.↩︎

4. A discussion of the affinity metrics and dataset relationships is included in Appendix 8.4.↩︎

5. Training details are provided in Appendix 8.3.↩︎