1 Introduction↩︎

Video Diffusion Models (VDMs) [1]–[4] have revolutionized video creation, enabling breakthroughs in simulation, advertising, and film production. However, their practical deployment is severely constrained by high inference latency and substantial computational resource requirements.

Recent advancements in post-training acceleration have been significant, focusing on step reduction [5]–[7], attention optimization [8], [9], quantization [10], token compression [11], and streamlined generation [12], [13]. Despite notable progress, these methods share a common limitation: they treat the denoising network as a static architecture, applying the same heavy computation uniformly across all timesteps. However, the diffusion process is inherently heterogeneous. Existing works [14], [15] observe that generation proceeds in a coarse-to-fine manner, where the model first constructs global structures and later refines high-frequency details. This suggests that distinct denoising stages require varying computational capacities, raising a pivotal question: Can we exploit this heterogeneity to dynamically allocate computation across timesteps, removing step-specific redundancy to accelerate VDMs?

While prior works [15], [16] have explored dynamic computation, they are typically trained under standard denoising objectives and often become unstable or ineffective when adapted to few-step distillation [15]. Given that few-step distillation has become the de facto paradigm for efficient VDMs, this incompatibility severely restricts their utility. Moreover, straightforward decoupled pipelines—either “prune-then-distill” or “distill-then-prune”—are insufficient to resolve this mismatch. Specifically, pruning before distillation limits the model capacity required to learn the complex few-step mapping, while pruning after distillation disrupts the carefully aligned generation trajectory, leading to severe error accumulation, see [sec:jointvssep].

To translate dynamic computation into practical efficiency, we formulate dynamic pruning and few-step distillation as a unified optimization problem. Leveraging distribution matching—a robust objective validated in 3D generation [17] and attention optimization [13], [18]—we distill a dynamic structure specifically tailored to the compressed generation trajectory. This results in a step-aware Mixture-of-Models (MoM) framework capable of high-quality 4-step inference, where the network architecture is adaptively pruned at each step, inherently bypassing the incompatibility worries associated with post-hoc distillation.

Achieving this joint objective is non-trivial, as simultaneously optimizing for parameter sparsity and step reduction leads to gradient conflicts and training instability. To overcome this, we introduce a Progressive Training Strategy augmented with an Output Rollout mechanism. Our core insight is that diffusion is a sequential process where the ultimate objective is the perceptual quality of the final output. Guided by this, we adopt a reverse-order curriculum: this strategy first stabilizes the generation quality of the later denoising stages before progressively sparsifying the earlier, noisier stages. Furthermore, we leverage the output rollout strategy in Distribution Matching Distillation (DMD) [19], [20], enabling supervision based on the final output distribution. These designs ensure the stable training of highly efficient models. Finally, to translate theoretical sparsity into practical acceleration, we implement a specialized inference engine that effectively leverages this dynamic mixture of models.

Our contributions are summarized as follows:

1. We propose an end-to-end framework that jointly optimizes few-step distillation and dynamic structural sparsification, yielding a step-aware Mixture-of-Models that removes both temporal and parametric redundancy.

2. We introduce a progressive training plan combined with an output rollout strategy. This design effectively coordinates the co-optimization and ensures convergence to a high-quality solution.

3. Equipped with a specialized inference engine for the MoM architecture, our method removes 24% of per-step FLOPs on Wan-14B and adds a 1.2\(\times\) speedup over 4-step distillation, giving a 30\(\times\) speedup over the 50-step teacher with competitive VBench scores.

2 Related Work↩︎

2.1 Efficient Video Diffusion Generation↩︎

Acceleration techniques for pretrained Video Diffusion Models can be broadly categorized into training-free optimization [21]–[24] and post-training frameworks. We focus on the latter category in this work. Existing post-training approaches explore multiple dimensions of optimization. Step distillation methods [5], [6], [14], [25] compress the long sampling trajectory into a few-step regime (e.g., 4–8 steps), substantially reducing inference latency. Attention optimization techniques [9], [18], [26] alleviate quadratic complexity through linearization or sparsification. Token reduction strategies [11] employ fine-tuning to adapt high-compression VAEs, effectively shortening spatial-temporal tokens. Causal distillation methods [12], [13] convert slow bidirectional generation into efficient streaming generation. Quantization-aware training techniques [10] utilize low-bit parameterization to reduce inference overhead.

These approaches typically treat the denoising network as a static architecture, applying uniform computation across all timesteps. Our approach offers an orthogonal optimization perspective by targeting the structural redundancy inherent in the multi-step diffusion process, introducing step-dependent dynamic optimization.

2.2 Structural Pruning and Dynamic Networks↩︎

Structural pruning removes redundant higher-level units (e.g., channels, filters, and blocks) to reduce parameters and FLOPs while preserving accuracy. Classical structural pruning identifies unimportant units based on magnitude [27], [28], importance scoring [29], or gradient-sensitive criteria [30], and derives a compact static architecture for deployment. Dynamic networks extend this paradigm by introducing conditional computation, representative methods like [31], [32] dynamically adjust network depth or width based on input tokens. A prominent example is Mixture-of-Experts (MoE) [33], [34], which activates subsets of parameters—typically in feed-forward layers—through learned routing mechanisms. While effective for scaling model capacity, these token-dependent dynamic structures impose strict requirements on routing strategies and system-level implementation to achieve practical wall-clock acceleration [33], [35].

In diffusion models, the varying noise levels across timesteps introduce a unique dimension of heterogeneity. DyDiT series [15], [16] explores this by proposing time-dependent pruning, learning a specific network structure for each timestep. As the routing depends only on the diffusion step, efficient batched inference is friendly supported without complex routing implementations. However, DyDiT is trained under standard multi-step denoising objectives, lacking the mechanism to co-optimize with step reduction. As demonstrated in their original paper [15], combining it with step distillation leads to model collapse or ineffective training. This incompatibility with modern few-step distillation frameworks severely restricts its practical utility. We address this issue by designing a joint optimization method.

2.3 Step-Aware Mixture-of-Models↩︎

Our method transforms a pretrained VDM into a step-aware Mixture-of-Models (MoM), where networks are pruned distinctly for different denoising stages. Several large-scale generative models, such as ERNIE-ViLG 2.0 [36] and Wan2.2 [37], also adopt MoM designs by training separate denoisers for different noise intervals. Their objective is to expand model capacity during pre-training to improve generation quality. Similarly, PhasedDMD [14] introduces a MoM structure within step distillation primarily for increased model expressiveness.

Conversely, we employ MoM as a post-training acceleration mechanism. We exploit the structural heterogeneity across noise levels to compress a pre-trained model, aiming for efficiency rather than capacity expansion.

3 Preliminaries↩︎

We perform step distillation based on the Distribution Matching Distillation method with specific modifications. Additionally, we apply the tri-level pruning scheme from [38] for structural sparsification. We briefly introduce these methods in this section.

3.1 Modified Distribution Matching Distillation↩︎

We employ Decoupled Distribution Matching Distillation (D-DMD) [39] to distill a multi-step diffusion model into a \(T\)-step student generator \(G_{\theta}\) (we set \(T=4\)). The student operates on a discrete set of timesteps \(\mathcal{T} = \{t_1, t_2, \dots, t_T\}\), where \(t_1\) denotes the initial highest noise level and \(t_T\) denotes the final lowest noise level prior to the clean data. The objective is to minimize the Kullback-Leibler (KL) divergence \(\mathcal{D}_{\text{KL}}(p_{\theta} \| p_{\text{real}})\) between the student distribution \(p_{\theta}\) and the real data distribution \(p_{\text{real}}\). Since direct optimization of this divergence is intractable, DMD-based methods approximate the gradient \(\nabla_{\theta} \mathcal{D}_{\text{KL}}\) with respect to the student parameters \(\theta\) using score estimation (refer to [19] for details). Specifically, D-DMD re-formulates the gradient by explicitly expanding the teacher’s Classifier-Free Guidance (CFG) term and rearranging the components, resulting the following decoupled form:

\[\label{eq:dmd95gradient} {!}{\displaystyle \nabla_{\theta} \mathcal{D}_{\scalebox{0.6}{\text{KL}}} \approx \mathbb{E}_{t \in \mathcal{T}, \tau_{\scalebox{0.6}{\text{DM}}}, \tau_{\scalebox{0.6}{\text{CA}}}} \left[ - \left( \underbrace{\left(\mathbf{s}_{\text{real}}(\mathbf{x}_{\tau_{\scalebox{0.6}{\text{DM}}}}, c) - \mathbf{s}_{\text{fake}}(\mathbf{x}_{\tau_{\scalebox{0.6}{\text{DM}}}}, c) \right)}_{\scalebox{0.9}{\text{Distribution Matching:}}\Delta_{\text{DM}}} + \underbrace{(\alpha-1) \left(\mathbf{s}_{\text{real}}(\mathbf{x}_{\tau_{\scalebox{0.6}{\text{CA}}}}, c) - \mathbf{s}_{\text{real}}(\mathbf{x}_{\tau_{\scalebox{0.6}{\text{CA}}}}, \emptyset)\right)}_{\scalebox{0.9}{\text{CFG Augmentation:}}\Delta_{\text{CA}}} \right) \frac{\partial G_{\theta}(\mathbf{z}_t)}{\partial \theta} \right], }\tag{1}\]

where \(\mathbf{z}_t\) denotes the latent input at few-step noise level \(t \in \mathcal{T}\), and \(\mathbf{x}_{\tau}\) represents the generator output re-noised to a continuous noise level \(\tau\) (\(\tau=0\) for pure noise, \(\tau=1\) for clean data). The terms \(c\) and \(\emptyset\) denote the text condition and the null condition, respectively. \(\mathbf{s}_{\text{real}}\) estimates the score of the real data distribution using the frozen pre-trained teacher network, and \(\mathbf{s}_{\text{fake}}\) approximates the student’s score via a fake network trained concurrently with a denoising objective on \(G_{\theta}\)’s generated samples.

The gradient comprises two components: \(\Delta_{\text{CA}}\) encapsulates the CFG pattern (\(\alpha\) is the CFG guidance scale), acting as the distillation engine that injects knowledge of teacher model to the student. \(\Delta_{\text{DM}}\) serves as a regularizer, prevents the training process from diverging and ensures the quality of the final output. Following D-DMD, we apply decoupled re-noising schedules: \(\tau_{\text{CA}} \sim \mathcal{U}[t, 1]\) focuses on refining fine-grained details in later denoising stages, whereas \(\tau_{\text{DM}} \sim \mathcal{U}[0, 1]\) provides global regularization across all noise levels.

Moreover, we remove the ODE trajectory initialization in standard DMD as we found it causes generation instability. We incorporate the GAN loss suggested in DMD2 [20] to enhance video quality. Consequently, our modified distillation objective is formulated as \(\mathcal{L}_{\text{DMD}} = \mathcal{D}_{\text{KL}} + \mathcal{L}_{\text{GAN}}\).

3.2 Step-Aware Tri-level Structural Pruning↩︎

The Video Diffusion Model employed in our work is built upon the Diffusion Transformer (DiT) architecture. Following previous practices [38], we adopt a tri-level pruning mechanism operating at the block, attention-head, and channel levels. We formulate the pruning scheme using discrete binary gates \(\mathbf{g} \in \{0, 1\}\), where \(0\) indicates a pruned component and \(1\) indicates a retained one.

To accommodate the sequential nature of few-step inference, we extend this static pruning into a step-aware dynamic structure. Specifically, for each discrete inference step \(t_i \in \mathcal{T}\), we maintain a specific pruning scheme \(\mathbf{g}_{t_i}\). Consequently, the gate tensors for the block, head, and feed-forward network (FFN) levels are defined with shapes \([T, N_{\text{block}}]\), \([T, N_{\text{head}}]\), and \([T, N_{\text{channel}}]\), where \(N_{\text{block}}\), \(N_{\text{head}}\), and \(N_{\text{channel}}\) denote the total number of transformer blocks, multi-head attention heads, and FFN hidden channels, respectively.

During training, these discrete gates are applied to the intermediate feature representations via element-wise multiplication to mask out redundant computations (see 1 (a)). Formally, for a specific architectural component \(\mathcal{F}\) at inference step \(t_i\), the gated output is computed as: \[\label{eq:gating} \mathbf{x}_{\text{out}} = \mathbf{g}_{t_i} \odot \mathcal{F}(\mathbf{x}_{\text{in}}),\tag{2}\] where \(\odot\) denotes broadcasted element-wise multiplication. Once the gates are fully optimized, the resulting sparse architecture can be exported into a dense format. As each inference step possesses a distinct structural configuration, the final exported model effectively constitutes a compact Mixture-of-Models (MoM), translating the theoretical sparsity of gates into actual computational speedup during inference.

4 Method↩︎

In this section, we introduce our unified framework. Our goal is to learn a step-adaptive architecture that maximizes computational efficiency while preserving the high-fidelity generation capability of few-step distillation. As illustrated in 1, our framework consists of three key components: (1) a joint optimization objective that unifies few-step distillation with dynamic structural pruning; (2) a progressive training strategy augmented with output rollout to stabilize the co-optimization process; and (3) a specialized inference engine designed to fully leverage the acceleration potential of the dynamic-in-few-step MoM architecture.

4.1 Unifying Step Reduction and Dynamic Sparsification↩︎

To exploit the temporal redundancy inherent in diffusion generation, we propose learning a dynamic pruning policy that adapts to the specific structural requirements of each denoising step. Our strategy is to update the pruning gates directly via the few-step distillation objective. However, the discrete binary gates \(\mathbf{g}_{t_i}\) defined in 3.2 are non-differentiable, preventing standard gradient-based updates. To resolve this, we introduce continuous learnable structural masks \(\mathbf{m}_{t_i}\) for each inference step \(t_i \in \mathcal{T}\) to enable optimization within the distillation framework.

The mapping from the continuous soft masks \(\mathbf{m}_{t_i}\) to the discrete hard gates \(\mathbf{g}_{t_i}\) is tailored to each pruning granularity:

• Block-level: the gate is directly determined by an indicator function \(\mathbf{g}_{t_i} = \mathbb{I}(\mathbf{m}_{t_i} > 0)\).

• Attention-head level: for each head with a feature dimension of \(D_{\text{head}}\), the entire head is retained (\(\mathbf{g}_{t_i} = 1\)) if the mean of its corresponding \(D_{\text{head}}\) mask parameters is positive.

• FFN channel level: We group the hidden channels into sequential chunks of size \(C\); a single scalar mask controls an entire chunk, yielding a block-wise gate vector \(\mathbf{g}_{t_i} = \mathbb{I}(\mathbf{m}_{t_i} > 0) \cdot \mathbf{1}^C\).

We employ the Straight-Through Estimator (STE) to integrate the hard gates into the computational graph of the distillation task. Specifically, we construct a differentiable surrogate \(\tilde{\mathbf{g}}_{t_i}\) for the forward pass: \[\label{eq:ste} \tilde{\mathbf{g}}_{t_i} = \mathbf{g}_{t_i} + \mathbf{m}_{t_i} - \text{stop\_grad}(\mathbf{m}_{t_i}).\tag{3}\] During training, we substitute the discrete \(\mathbf{g}_t\) with \(\tilde{\mathbf{g}}_t\) in 2 . This formulation ensures that the forward pass utilizes the exact discrete gates to mask the corresponding components, while the backward pass routes the distillation task gradients directly to the continuous masks \(\mathbf{m}_{t_i}\).

Furthermore, we introduce a sparsity penalty \(\mathcal{L}_{\text{sparse}}\) to encourage structural sparsity and control the overall model capacity. When optimizing for step \(t_i\), the global retention rate for component \(k\) combines the differentiable surrogate \(\tilde{\mathbf{g}}_{k, t_i}\) and the frozen hard gates \(\mathbf{g}_{k, t_j}\) of other steps \(t_j \neq t_i\): \[\label{eq:sparse95loss} \mathcal{L}_{\text{sparse}}(\tilde{\mathbf{g}}_{t_i}) = \sum_{k \in \{b, h, f\}} \text{ReLU}\left( \frac{1}{T} \left( \mu(\tilde{\mathbf{g}}_{k, t_i}) + \sum_{t_j \neq t_i} \mu(\mathbf{g}_{k, t_j}) \right) - \eta_k \right),\tag{4}\] where \(\mu(\cdot)\) calculates the spatial mean of the gate tensor for component \(k\) (block, head, or FFN), and \(\eta_{k}\) is the corresponding target sparsity ratio.

The overall training objective combines the distribution matching loss with the sparsity penalty: \[\mathcal{L}_{\text{total}} = \mathbb{E}_{t \sim \mathcal{T}} \left[ \mathcal{L}_{\text{DMD}}(\theta, \tilde{\mathbf{g}}_t) + \lambda \mathcal{L}_{\text{sparse}}(\tilde{\mathbf{g}}_t) \right].\] By jointly optimizing the model parameters \(\theta\) and the structural masks \(\mathbf{m}\), our framework simultaneously uncovers both step and parameter redundancies. This co-optimization enables the model to learn specific structural demands across different timestep levels, leading to intriguing architectural findings discussed in 6.

4.2 Progressive Training with Output Rollout↩︎

Directly optimizing the joint objective \(\mathcal{L}_{\text{total}}\) results in unsatisfactory outcomes. We attribute this to the imbalanced distillation gradient norms across different noise levels. As the sparsity penalty is applied uniformly, stages with smaller distillation gradients are disproportionately dominated by the sparsity loss, leading to arbitrary over-pruning rather than learning genuinely step-adaptive structures.

To resolve this, we propose a progressive training strategy augmented with an output rollout mechanism. Our core insight is that diffusion is a sequential process where intermediate errors accumulate, yet the ultimate objective is solely the perceptual quality of the final output. Therefore, we anchor our co-optimization to the final generation distribution.

First, we adopt a reverse-order curriculum to stabilize the structural search. We prioritize establishing a robust generation foundation at the later denoising stages (i.e., lower noise levels) before progressively sparsifying the earlier, noisier stages. We initially optimize the distill loss and sparsity penalty exclusively for \(t_4\) (the lowest noise level). Subsequently, we progressively incorporate earlier steps (\(t_3 \rightarrow t_2 \rightarrow t_1\)) into the training pipeline, and apply the sparsity penalty only to the newly introduced step. Formally, for the \(T\)-step model, the training proceeds backwards across \(T\) progressive stages. In Stage \(k\) (\(k \in \{1, \dots, T\}\)), the active training subset is \(\mathcal{T}_k = \{t_T, t_{T-1}, \dots, t_{T-k+1}\}\). The newly introduced step is \(t_{new} = t_{T-k+1}\). The objective for Stage \(k\) is formulated as: \[\mathcal{L}_{\text{Stage-}k} = \mathbb{E}_{t \sim \mathcal{T}_k} \left[ \mathcal{L}_{\text{DMD}}(\theta, \tilde{\mathbf{g}}_t) \right] + \lambda \mathcal{L}_{\text{sparse}}(\tilde{\mathbf{g}}_{t_{new}}).\] After the \(T\) progressive training stages, a final stage then jointly trains all \(T\) steps using the full objective \(\mathcal{L}_{\text{total}}\). This curriculum ensures the model secures its output quality before aggressively pruning the highly-noised input stages.

Furthermore, we align the training of the fake score network \(\mathbf{s}_{\text{fake}}\) with this final-output-centric philosophy via an output rollout strategy. In standard distillation practices, \(\mathbf{s}_{\text{fake}}\) is typically trained on the single-step predicted clean data \(\hat{\mathbf{x}}_1\) (computed from the model prediction \({\mathbf{v}}_t\)) derived directly from an intermediate noisy state. While effective for static architectures, this intermediate supervision exhibits reduced stability when dynamic structural pruning is introduced. Therefore, we choose to model the true multi-step generation outcome for a more reliable constraint. Specifically, we employ an output rollout mechanism: when the student denoiser operates at an intermediate high-noise step (e.g., \(t_1\)), we iteratively apply the student model—with gradients detached—to sequentially denoise the sample all the way to the final step \(t_T\) (see 1 (b)). This final output \({\mathbf{x}}_1^{rollout}\) is then utilized as the training sample to update \(\mathbf{s}_{\text{fake}}\). By training the fake model to capture the final generation distribution, we inject a more robust regularization signal, ensuring the stable co-optimization of our dynamic MoM architecture.

4.3 Specialized Inference Engine for MoM↩︎

To translate the learned theoretical sparsity into tangible wall-clock acceleration, we design a specialized inference engine. After training, we construct a single unified super-model containing the union of all active parameters across all steps: \(\theta_{\text{union}} = \bigcup_{t \in \mathcal{T}} \{ \theta_i \mid m_{t, i} > 0 \}\).

Alongside this super-model, we maintain a lightweight index table \(\mathcal{I}_t\) for each step \(t \in \mathcal{T}\), explicitly pointing to the specific subset of \(\theta_{\text{union}}\) required for that stage. During inference, the engine efficiently gathers the active dense parameters via \(\mathcal{I}_t\). This dynamic routing eliminates the latency overhead of reloading different weights and significantly reduces memory consumption compared to storing \(T\) separate models (see 1 (c)).

5 Experiments↩︎

5.1 Implementation Details↩︎

Model Architecture and Initialization. We evaluate our method on Wan2.1-14B as the main model. We also use the smaller Wan2.1-1.3B for baseline comparison, ablation, and architectural analysis, where extensive training sweeps are tractable. Both models are trained for text-to-video generation at \(480 \times 832\), generating 5-second videos at 16 frames per second (fps). The teacher and fake-score network are initialized from the full pre-trained checkpoint. The student generator adopts the same initialization but is augmented with our learnable structural masks (initialized to \(1.0\)) to enable dynamic pruning.

Training Configurations. Our training subset comprises 50,000 high-quality videos curated from the Koala-36M dataset [40]. We implement the discriminator architecture and GAN loss formulation following [41]. The Classifier-Free Guidance scale is randomly sampled from \([3, 5]\) during training. We optimize the models using AdamW across 16 Kunlun P800 XPUs with a total batch size of 16. The learning rates are set to \(1 \times 10^{-6}\) for the student model and fake score network, and \(5 \times 10^{-7}\) for the GAN discriminator. The learning rate for the structural masks warms up from \(1 \times 10^{-3}\) to a peak of \(1 \times 10^{-2}\). The total training process spans 2,000 steps. For sparsity loss, the coefficient \(\lambda\) is adaptively determined at each stage by monotonically mapping the corresponding gradient norm to the interval \((0,1)\). Additional implementation details are provided in the Appendix 8.

Progressive Pruning Schedule. We implement the reverse-order curriculum described in 4.2 using a FLOPs-driven stage transition criterion. Specifically, we advance the training to incorporate the next noisier step (e.g., transitioning from \(t_4\) to \(t_3\)) whenever the newly introduced step’s FLOPs are reduced by \(5\%\). We stop training once the average FLOPs across all steps \(t \in \mathcal{T}\) drop to \(75\%\) of the original dense model.

5.2 Evaluation Metrics↩︎

We evaluate our method along two dimensions: generation quality and computational efficiency. For generation quality, we adopt VBench [42] and report its eight primary dimensions, covering frame-wise quality (Imaging Quality, Aesthetic Quality), temporal quality (Motion Smoothness, Dynamic Degree, Background Consistency, Subject Consistency, Scene Consistency), and text alignment (Overall Consistency), following [26]. The complete 16 metrics and additional qualitative results are provided in the Appendix 9. We employ augmented prompts provided in VBench and generate five videos per prompt. For efficiency, we report inference latency (seconds per video) measured on a single Kunlun P800 XPU. The reported runtime corresponds exclusively to the DiT backbone inference, excluding VAE decoding.

5.3 Baselines↩︎

Our method exposes a new acceleration axis—per-step computation redundancy. Accordingly, our comparisons serve two purposes: first, to test whether this step-wise pruning can be obtained by a two-stage pipeline; second, to place our method in the context of existing acceleration frameworks.

Two-Stage Optimization Baselines. These baselines separate step distillation and structural pruning. They use the same 4-step distillation recipe as our method, but apply pruning either before or after distillation. For each setting, we sweep the pruning ratio and report the highest value that does not lead to visual collapse:

Category 1: Prune-then-Distill. The model is first pruned and then distilled into a 4-step generator.

(1) Static \(\rightarrow\) Distillation: We apply classical magnitude-based static pruning [27] to the FFN layers (attention head pruning is omitted as we observe it may trigger generation collapse). The sparsity ratio is capped at \(10\%\).

(2) Dynamic \(\rightarrow\) Distillation: Following DyDiT [15], we train step-conditioned masks under the standard denoising objective with a sparsity loss. The noise schedule is divided into four intervals to match the later 4-step distillation.

Category 2: Distill-then-Prune. The model is first distilled into 4 steps and then pruned.

(3) Distillation \(\rightarrow\) Static: The reverse pipeline of (1), where magnitude-based pruning is applied to a frozen distilled model.

(4) Distillation \(\rightarrow\) Dynamic: We freeze the pre-distilled model weights and optimize only the dynamic masks using the same training pipeline described in 4.

Distillation Baseline and Teacher. Ours (Distill) is the 4-step DMD-distilled model without pruning, trained with the same data and recipe as Ours (Joint). It is the direct baseline for measuring the extra benefit of our dynamic pruning. We also report the 50-step Wan model [1] as the teacher and quality reference.

SOTA Acceleration Frameworks. We report two representative open-source acceleration frameworks: TurboDiff [43] (we evaluate the TurboWan2.1-T2V-1.3B checkpoint) and LightX2V [22] (the Wan2.1-T2V-14B-StepDistill-CfgDistill checkpoint). These methods combine step distillation with other optimizations, such as sparse attention, quantization, and inference-system acceleration. Their reported speedups (marked with \(^{*}\) in 1) serve as a holistic reference for the overall acceleration landscape, but they are not direct comparison of our method: we study hidden-channel pruning, while these frameworks optimize other complementary dimensions. Because we cannot reproduce their full hardware and deployment environment, we cite their official latency gains; all VBench scores are measured under our own unified evaluation protocol.

5.4 Main Results↩︎

1 reports the results. We decompose each method’s speedup into the step-reduction factor (relative to the 50-step teacher) and the additional pruning factor on top of 4-step distillation (“\(\Delta\) vs.Distill”). Step distillation alone already gives a \(\sim\)​25\(\times\) saving; any further gain must come from making each remaining step cheaper, which the \(\Delta\) column directly measures.

Speedup on Wan-14B. On Wan-14B, Ours (Joint) reduces per-step FLOPs by 24% and cuts latency from 1222.16 s (50-step teacher) to 40.72 s. This represents a 30.01\(\times\) end-to-end speedup over the teacher. Crucially, while step distillation contributes a 25.00\(\times\) reduction, our dynamic pruning provides an additional 1.20\(\times\) wall-clock acceleration. Despite this substantial capacity compression, our method maintains highly competitive VBench scores, confirming that the pruned capacity is genuinely redundant for the few-step trajectory.

Decoupled pipelines fail to exploit per-step redundancy. As shown in 1, decoupling pruning and distillation fails to deliver a favorable speed-quality trade-off. First, Static\(\rightarrow\)Distill and Distill\(\rightarrow\)Static are limited by the static pruning pattern, yielding a negligible 1.01\(\times\) marginal speedup. Second, two-stage dynamic pipelines similarly falter due to sequential optimization conflicts. Applying pruning after distillation (Distill\(\rightarrow\)Dynamic) severely disrupts the delicately aligned few-step generation trajectory, resulting in a precipitous degradation in temporal dynamics, most notably Dynamic Degree. Conversely, applying pruning prior to distillation (Dynamic\(\rightarrow\)Distill) restricts the representational capacity essential for effectively learning the highly compressed 4-step mapping. Consequently, it suffers from noticeable performance drops in metrics like Imaging Quality and Dynamic Degree. Beyond quantitative metrics, our supplementary videos reveal that this variant generates visually inferior texture details and exhibits diminished motion amplitude. In contrast, our unified framework resolves these capacity bottlenecks by jointly optimizing step reduction and dynamic pruning. This co-design achieves superior acceleration while preserving high-fidelity spatial details and rich, coherent temporal dynamics (further analysis in Appendix11 confirms the high dynamic degree is free from artifact-driven flickering).

Comparison with SOTA frameworks. We position our method alongside integrated open-source acceleration frameworks, TurboDiffusion and LightX2V, to contextualize its performance. As demonstrated in 1, our method preserves competitive generation quality across VBench dimensions. It is important to emphasize that our approach is not designed to directly compete with these holistic, system-level pipelines, which bundle distillation with sparse attention, quantization, and infrastructure-level engineering. Instead, our contribution uncovers an orthogonal algorithmic acceleration axis: the per-step parametric redundancy along the hidden-channel dimension, driven by the coarse-to-fine dynamics of diffusion. Because our MoM exports standard dense architectures for each step, it is structurally compatible with these broader optimizations. We therefore position our method as complementary to, rather than competing with, such pipelines, with the potential to be combined for further gains.

5.5 Ablation Study↩︎

To validate the efficacy of our proposed training techniques, we conduct a comprehensive ablation study. Qualitative results are illustrated in 3, with the corresponding quantitative bounds detailed in Appendix 10. Our experiments include:

Step-Specific vs. Shared Mask. We ablate the step-aware dynamic structure by enforcing a shared mask across all inference steps.

Effect of Progressive Training. We remove the reverse-order curriculum, directly optimizing all steps jointly from the very beginning (i.e., exclusively using the Final Stage setup).

Effect of Output Rollout. We replace our output rollout mechanism and use the standard intermediate prediction \(\hat{\mathbf{x}}_1\) for the fake score network training.

It can be observed that the step-specific configuration demonstrates superiority over the shared mask, as it generally achieves higher sparsity ratios and preserves basic structural fidelity. Furthermore, a consistent observation across all ablated settings is the existence of a strict sparsity wall. Without our tailored techniques, the models fail to reach high sparsity levels; forcing the optimization further leads to a cliff-like drop in structural integrity and a rapid collapse in generation quality. While the ablated models max out at a stable sparsity of only around 2.3%, our full framework stably achieves a 20.1% sparsity without visual degradation, highlighting the absolute indispensability of our training strategy for stabilizing the dynamic MoM co-optimization.

6 Exploration and Findings↩︎

By jointly optimizing the structural masks and the distillation objective, our framework automatically discovers how to allocate computation across denoising steps. Beyond the efficiency gains, the learned policy reveals interpretable structure in how redundancy is distributed both temporally (across steps) and spatially (across components and depth), as visualized in 4 for the Wan-1.3B model. The Wan-14B model follows a similar pattern.

U-Shaped Computational Allocation. Temporally, the learned architecture exhibits a distinct “U-shaped” capacity demand. The model retains the highest computational footprint at the two ends of the generation trajectory (88.4% and 89.1% FLOPs retention at \(\mathbf{t}_{1}\) and \(\mathbf{t}_{4}\), respectively) while aggressively pruning the intermediate steps (78.5% and 77.1% at \(\mathbf{t}_{2}\) and \(\mathbf{t}_{3}\)). We further verified (8 (e)–(h)) that the U-shape emerges progressively during training and persists across a broad range of sparsity levels. The pattern mirrors the physics of diffusion generation: the initial phase needs ample capacity to construct the global layout from pure noise, the final phase to render high-frequency visual details, while intermediate phases mostly transition the established semantics and can be heavily compressed.

Semantic Freezing in Deep Layers. Spatially, we observe that deeper layers are pruned much more aggressively than shallower layers, especially within the Self/Cross-Attention modules during \(\mathbf{t}_{2}\) and \(\mathbf{t}_{3}\). In Transformer-based diffusion models, deeper layers typically process high-level semantic representations, while shallower layers handle low-level textures. This pruning pattern suggests a semantic freezing phenomenon: high-level semantics are rapidly decided and frozen in the very early stages of inference. Consequently, the network automatically learns to shut down the deep-layer semantic updates in subsequent steps, allocating its limited computational budget to the shallow layers for fine-grained texture refinement.

7 Conclusion↩︎

We presented an end-to-end framework that unifies few-step distillation and dynamic structural pruning for accelerating Video Diffusion Models. By treating step reduction and parameter sparsification as a single joint optimization, we derive a step-aware Mixture-of-Models that allocates computation across denoising stages, and we keep this optimization stable with a reverse-order progressive curriculum and an output-rollout fake-score objective. On Wan-14B, our method achieves an 30\(\times\) speedup while maintaining satisfied visual quality and temporal dynamics. We believe our unified framework and architectural findings will pave the way for designing fundamentally more efficient generative models in the future.

This supplementary material provides further details and comprehensive results to support the main paper. It is organised as follows:

• Section 8 details the extended implementation hyperparameters and training configurations.

• Section 9 presents the complete set of VBench evaluation metrics alongside additional qualitative visualisations.

• Section 10 provides the quantitative results for our ablation studies.

• Section 11 illustrates the training dynamics, the continuous quality-vs.-sparsity trade-off, and an in-depth analysis of our final model architecture.

8 Extended Implementation Details↩︎

Due to space constraints in the main text, we provide the complete hyperparameter settings and training configurations here.

Dynamic Sparsity Penalty Weighting. As briefly discussed in the main text, we dynamically adjust the sparsity penalty weight \(\lambda\) for each denoising step to balance the distillation and pruning objectives. Specifically, we maintain an Exponential Moving Average (EMA) of the task gradient norm, denoted as \(n_t\), for each step \(t\). To prevent the sparsity penalty from being overshadowed by large distillation gradients, we map this EMA value to a monotonically increasing scalar in the range \((0, 1)\) using the following smooth normalization function: \[\lambda_t = \frac{n_t + \epsilon}{n_t + \epsilon + 1},\] where \(\epsilon = 10^{-6}\) is a small constant added for numerical stability.

FFN Reordering and Chunking. For the FFN channel-level pruning, we set the chunk size to \(C=8\). We perform a pre-processing step before chunking: we reorder the FFN hidden channels of the pre-trained base model according to the \(L_2\) norm of their corresponding weights. This norm-based sorting maximizes the intra-chunk homogeneity, allowing the single scalar mask to effectively control the entire chunk without sacrificing representational capacity.

Memory Optimization and Hardware. The model is trained using Fully Sharded Data Parallel (FSDP) combined with mixed-precision training (e.g., bfloat16). Additionally, we utilize gradient checkpointing to trade computation for memory, ensuring the joint optimization fits within the GPU memory limits. The training is conducted on 16 Kunlun P800 (96GB) XPUs.

9 Full Evaluation Metrics and Additional Visualizations↩︎

The VBench suite comprehensively evaluates video generation models across multiple dimensions, including temporal quality (Subject Consistency, Background Consistency, Temporal Flickering, Motion Smoothness, Dynamic Degree), frame-wise quality (Imaging Quality, Aesthetic Quality), semantic consistency (Object Class, Multiple Objects, Human Action, Color, Spatial Relationship, Scene), style consistency (Appearance Style, Temporal Style), and text alignment (Overall Consistency).

In the main paper, we reported 8 core dimensions from the VBench suite. Here, we present the remaining 8 VBench metrics in Table 2, along with the Total Score that reflects the comprehensive generation quality aggregated across all 16 dimensions.

As shown in Table 2, consistent with the findings in the main text, our joint optimization method (Ours (Joint)) achieves a highly competitive Total Score of 83.62, closely matching our dense distillation baseline (Ours (Distill), 83.64) while reaching a 27.66\(\times\) end-to-end speedup over the 50-step teacher on Wan-1.3B. Additional qualitative results of Ours (Joint) are provided in Figure 6 and Figure  7.

10 Quantitative Ablation↩︎

Ablated variants of our dynamic MoM framework experience rapid generation collapse before reaching meaningful sparsity ratios. 3 provides the quantitative metrics of these failures on Wan-1.3B.

The results clearly demonstrate that without our proposed progressive training and output rollout mechanisms, the optimization hits a rigid “sparsity wall” at roughly 2.0%–2.5%. Pushing beyond this threshold results in collapsed models that suffer severe degradation across core metrics, particularly in Imaging Quality, Aesthetic Quality, and Dynamic Degree. While our main experiments demonstrate that joint training is the key to unlocking dynamic acceleration, the ablation study reveals that such co-optimization is highly non-trivial. It highlights the absolute indispensability of our progressive curriculum and rollout strategies for stabilizing the dynamic MoM co-optimization.

11 Extended Findings↩︎

We track the evolution of the model’s structural sparsity and its corresponding generation quality throughout the training process, as visualized in Figure 8 (on Wan 1.3B).

Training Dynamics and the Sparsity-Quality Trade-off. We record the training dynamics starting from the Final State training. As the training progresses, the structural sparsity continuously increases across all denoising steps (Figure 8 (e)-(h)), leading to a steady decline in the global computation cost (Figure 8 (d)). However, thanks to our joint distillation strategy, the generation metrics do not degrade monotonically alongside the parameter reduction. Instead, the model maintains a highly competitive generation capability, with key metrics such as Imaging Quality and Dynamic Degree exhibiting stable fluctuations and even reaching peak performance at specific intermediate stages (Figure 8 (a)-(c)).

Identifying a Favorable Sparsity-Quality Trade-off. Through this trajectory, we observe a practical capacity threshold that yields a highly favorable balance between computational efficiency and generation quality. The model demonstrates excellent performance when the average FLOPs ratio is reduced to approximately 83%. At this specific operating point, the model attains a high Dynamic Degree and robust Imaging Quality. Pushing the sparsity beyond this threshold (e.g., dropping FLOPs below 81%) results in a noticeable degradation, as excessively aggressive pruning begins to impair the representational capacity required for complex few-step generation. Consequently, we select the checkpoint at the \(\sim\)​83% FLOPs ratio as our final model, as it represents a well-balanced empirical sweet spot.

Dynamic Degree Analysis on Wan 1.3B. Regarding the Dynamic Degree score of 80.56, which exceeds the teacher’s 65.19, it is important to note two contextual factors. First, Dynamic Degree is an inherently high-variance metric. As shown in panels (a)–(c), it naturally fluctuates over \([51.4, 80.6]\) across viable training checkpoints. Second, While standard DMD degrades dynamics, our pipeline mitigates this via GAN training and extra real-video data: Ours (Distill) alone already reaches 68.06(\(>\) teacher’s 65.19). TurboDiffusion reaches 86.11, showing that a student exceeding its teacher on this metric is not anomalous.

Crucially, our method maintains an exceptionally high Motion Smoothness (98.17), which directly contradicts the possibility of temporal flickering or artefact-driven motion. Instead, we hypothesize that the structured stochastic perturbations introduced by our step-conditioned mask updates counteract distillation’s mode-seeking tendency, yielding genuinely richer motion dynamics. This is corroborated by our supplementary videos, which exhibit larger, coherent motion rather than jitter.

In-depth Structural Analysis. To further understand the resulting architecture of our joint optimization, we conduct a detailed component-level analysis, as shown in 8 (i)-(l). The retention rates of Self-Attention, Cross-Attention, and Feed-Forward Network (FFN) modules exhibit distinct patterns across the denoising steps. As observed in 8 (l), the sparsification of Self/Cross-Attention components constitutes a substantial portion of the total parameter reduction. The network learns to aggressively compress these Attention components particularly during the intermediate steps (\(t_2\) and \(t_3\)), while maintaining a relatively higher retention rate for FFN modules. This fine-grained breakdown empirically illustrates how the joint optimization automatically distributes structural redundancy across different architectural components and inference stages.

References↩︎