1 Introduction↩︎

Latent diffusion models [3] have become the dominant framework for image [4]–[6] and video generation [2], [7]–[10]. An encoder [11], [12] compresses pixels into a low-dimensional latent space where diffusion models [13]–[15] generate content, and a decoder then reconstructs the final output pixels. By operating in latent space rather than pixel space, this design reduces computational costs by orders of magnitude, enabling scaling to billions of parameters with Diffusion Transformers (DiT) [5], [16]. However, most progress has focused on accelerating the generation stage, while the decoder that maps latents back to pixels has remained largely convolutional and comparatively underexplored.

Originally, the main inference bottleneck was iterative denoising. Since then, more efficient architectures [8], [17]–[19], higher-compression VAEs [2], [20]–[23], and few-step distillation [24]–[32] have largely removed this bottleneck. But real-time streaming also requires causal frame-by-frame generation. Recent causal video generators [1], [33]–[37] are closing in on interactive frame rates. With latent generation nearing real-time speed, the bottleneck has shifted to the decoder: VAE decoding consumes over 64.6% of total inference time with the Wan2.2 decoder [2] at 720p (Figure 1).

Existing video decoders are predominantly 3D causal convolutional networks [2], [8], [38]–[41]. They reconstruct well but are slow and memory-intensive, making real-time streaming impractical. High-resolution decoding further requires spatial-temporal tiling, which multiplies decoder evaluations and latency. Transformer-based [42] decoders have been explored as alternatives, but face a trade-off between streaming and quality. Causal variants [43], [44] require explicit causal masks that prevent efficient FlashAttention [45] usage at high resolutions, limiting reconstruction fidelity. Bidirectional variants [46], [47] achieve better quality by attending across all frames, but cannot stream because each frame requires access to future frames.

The limitations of both convolutional and Transformer decoders suggest four desired properties for a real-time video decoder: (1) frame-by-frame decoding without padding or blending, (2) reconstruction quality competitive with convolutional decoders, (3) fast inference with consistent per-frame latency and bounded memory, and (4) high-resolution and long-duration decoding without spatial-temporal tiling. Transformers can in principle provide all four: sequential processing enables frame-by-frame decoding, self-attention captures spatial-temporal dependencies for high-quality reconstruction, and windowed attention bounds memory and computation per frame, ensuring consistent latency without tiling. However, no existing Transformer-based decoder satisfies all four properties.

We introduce FlashDecoder, a pure-Transformer latent-to-pixel decoder that satisfies all four properties. FlashDecoder processes one latent frame at a time with a fixed-size rolling KV cache. The fixed temporal window keeps per-frame computation and memory bounded regardless of video length, enabling constant-latency streaming without spatial-temporal tiling. Prior causal Transformer decoders train with explicit causal masks but switch to KV caching at inference. FlashDecoder instead uses the same temporal-order processing for both training and inference, removing the need for attention masks entirely. This makes high-resolution training feasible, matching the reconstruction quality of convolutional decoders. Evaluated on the Wan2.1 and Wan2.2 [2] latent spaces, FlashDecoder matches each convolutional decoder in reconstruction quality (e.g., 41.55 vs. PSNR at 1080p on Wan2.2) while providing \(3.6{\times}\)–\(4.7{\times}\) higher throughput and up to \(11{\times}\) lower memory on a single H100 GPU. With architecture-aware inference optimizations, the throughput gap widens to up to \(12{\times}\).

2 Related Work↩︎

2.1 Autoencoders for Visual Generation↩︎

Latent diffusion models [3] rely on learned autoencoders to compress inputs into a low-dimensional latent space. For images, autoencoders have evolved from Stable Diffusion VAE [3] to more efficient designs such as FLUX VAE [6] and DC-AE [20], improving compression ratios and reconstruction quality, with most designs primarily based on convolutions. For video, these spatial autoencoders have been extended to spatiotemporal variants that compress both spatial and temporal dimensions [2], [7], [8], [10], [22], [38]–[41], [46], [49], [50]. Dedicated video VAE works such as MagViT-v2 [51], CV-VAE [52], WF-VAE [53], VidTok [54], and VideoVAE+ [55] have further advanced reconstruction fidelity through improved architectures, loss formulations, and temporal modeling. Despite this progress, spatiotemporal decoding remains computationally expensive, particularly at high resolutions. Lightweight alternatives such as TurboVAED [56] and TAEHV [48] trade fidelity for speed, and all of the above video decoders remain convolutional.

A separate line of work has begun moving away from convolutional decoders entirely. Representation Autoencoders such as RAE [57], [58] and LV-RAE [59] pair frozen pretrained encoders (e.g., DINOv2 [60] and SigLIP 2 [61]) with Transformer decoders, achieving faster convergence and stronger generation quality than convolutional VAEs for image generation. This trend has yet to reach video, where a streaming-capable Transformer decoder would be needed.

2.2 Transformer-based Video Decoders↩︎

For video, Transformer-based decoders have been explored across discrete tokenization models [43], [62], hybrid discrete-continuous VAEs [44], [47], and fully continuous latent VAEs [46], [63]. These designs face a fundamental trade-off in how they handle temporal attention. Causal attention, as in OmniTokenizer [44], enables streaming through KV caching but requires explicit causal masks during training, making memory-efficient high-resolution training difficult and limiting reconstruction quality. Per-frame latency also grows as temporal context accumulates. Bidirectional attention, as in AToken [47] and the MAGI-1 VAE decoder [46], improves quality by attending across all frames but cannot support streaming decoding due to global temporal dependencies. FlashDecoder takes the causal approach but removes its key limitations: causality is enforced by processing order rather than masks, and a fixed-size window bounds memory regardless of video length, unifying training and inference under the same streaming mechanism.

3 Method↩︎

Figure 3 illustrates the overall pipeline. We first describe the latent diffusion framework (Sec. 3.1), then detail the Transformer backbone (Sec. 3.2), the streaming mechanism with rolling KV cache (Sec. 3.3), the temporal-first upsampling strategy (Sec. 3.4), and training objectives (Sec. 3.5).

3.1 Preliminaries: Latent Diffusion Model↩︎

FlashDecoder operates within the Latent Diffusion Model (LDM) framework [3]. A pretrained encoder \(\mathcal{E}\) maps a video \(\mathbf{x}\in\mathbb{R}^{B\times C\times T\times H\times W}\) to a latent tensor \(\mathbf{z}=\mathcal{E}(\mathbf{x})\in\mathbb{R}^{B\times C'\times T'\times H'\times W'}\), where \((C', T', H', W')\) are the compressed channel, temporal, and spatial dimensions. The diffusion process operates on \(\mathbf{z}\), and a decoder \(\mathcal{D}\) reconstructs pixels via \(\hat{\mathbf{x}}=\mathcal{D}(\mathbf{z})\). FlashDecoder is agnostic to the choice of encoder; we train decoders on both the Wan2.1 and Wan2.2 [2] latent spaces and use Wan2.2 as the primary example throughout.

3.2 Transformer Backbone Design↩︎

3.2.0.1 Base Architecture.

The decoder is a Transformer [42] with stacked self-attention and feedforward layers. We use Grouped-Query Attention (GQA) [64], which shares key-value heads across query groups to reduce KV cache memory during streaming. For stable training, we apply RMSNorm [65] before each sublayer and normalize keys and values within attention (KV-norm) [66]. Spatiotemporal position is encoded with 3D Rotary Positional Embeddings (3D-RoPE) [67], applied separately to the temporal (\(T'\)) and spatial (\(H' \times W'\)) dimensions.

3.2.0.2 Latent Projection.

Each latent frame \(\mathbf{z}_t\) is flattened into \(L_{\text{frm}} = H'W'\) spatial tokens in raster order, and a linear layer maps the latent channels \(C'\) to model dimension \(D\): \[\mathbf{P}=\mathrm{Linear}_{C'\rightarrow D}(\mathbf{z})\in\mathbb{R}^{B\times L\times D}, \label{eq:projection}\tag{1}\] where \(L = T' \cdot L_{\text{frm}}\) is the total sequence length.

3.3 Streaming with Rolling KV Cache↩︎

FlashDecoder processes video latents one frame at a time while maintaining a sliding-window KV cache of fixed size \(W_{\text{frm}}\) frames. We set \(W_{\text{frm}}{=}2\) throughout, so each latent frame attends only to itself and the immediately preceding latent frame; the effect of this choice is ablated in Sec. 4.4. Because future frames have not yet been fed to the model, temporal causality is enforced by construction without explicit attention masks.

3.3.0.1 Frame-by-Frame Processing.

Given \(T'\) latent frames \(\{\mathbf{z}_0, \ldots, \mathbf{z}_{T'-1}\}\), we process them sequentially. At each timestep \(t\): (1) the latent frame \(\mathbf{z}_t\) is projected into \(L_{\text{frm}}\) tokens via Eq. 1 ; (2) new key-value pairs \((\mathbf{K}_{t}^{\text{new}}, \mathbf{V}_{t}^{\text{new}})\) are computed with 3D-RoPE at temporal offset \(t \cdot L_{\text{frm}}\), appended to the cache, and the oldest frame is evicted if the cache exceeds \(W_{\text{frm}}\) frames; (3) current queries \(\mathbf{Q}_{t}^{\text{new}}\) attend to the full cache. The resulting cache shape is: \[\mathbf{K}_{t},\mathbf{V}_{t} \in \mathbb{R}^{B\times G\times (W_{\text{frm}}L_{\text{frm}})\times D_h}, \label{eq:kv95cache}\tag{2}\] where \(G\) is the number of KV groups in GQA and \(D_h\) is the head dimension.

3.3.0.2 Attention Pattern.

The resulting pattern is a sliding window: within each frame, tokens attend to all \(L_{\text{frm}}\) spatial positions (bidirectional); along the temporal axis, attention is restricted to the most recent \(W_{\text{frm}}\) frames (causal).

3.3.0.3 Training–Inference Consistency.

A distinctive property of FlashDecoder is that training and inference follow an identical streaming protocol: the model never sees more than \(W_{\text{frm}}\) frames at once during either phase. Conventional approaches load all \(T'\) frames into a single forward pass, which requires a full-sequence causal mask that FlexAttention [68] must materialize, causing out-of-memory failures at 480p, 720p, and 1080p on an H100 80 GB GPU. FlashDecoder instead performs \(T'\) sequential forward passes, each attending to at most \(W_{\text{frm}} \cdot L_{\text{frm}}\) tokens with standard FlashAttention [45] at per-step memory cost \(O(W_{\text{frm}} \cdot L_{\text{frm}})\).

3.3.0.4 Complexity Analysis.

The attention cost per latent frame is \(\mathcal{O}(N W_{\text{frm}} L_{\text{frm}}^2 D_h)\), linear in the temporal window \(W_{\text{frm}}\) and head count \(N\), but quadratic in spatial tokens \(L_{\text{frm}}\). The KV cache memory per layer is \(\mathcal{O}(B\,G\,W_{\text{frm}}\,L_{\text{frm}}\,D_h)\), benefiting from GQA’s reduced group count (\(G \ll N\)). After temporal upsampling (Sec. 3.4), refinement layers operate on \(r_{\text{t}} \cdot L_{\text{frm}}\) tokens per frame with cache capacity \(r_{\text{t}} W_{\text{frm}} L_{\text{frm}}\), preserving the same scaling properties.

3.4 Temporal-First Upsampling Strategy↩︎

Progressive spatio-temporal upsampling is computationally prohibitive for Transformers. Spatial upsampling by factor \(r_{\text{s}}\) increases tokens per frame by \(r_{\text{s}}^2\), resulting in \(O(r_{\text{s}}^4)\) attention cost (\(65{,}536{\times}\) for \(r_{\text{s}}{=}16\)). Temporal upsampling by factor \(r_{\text{t}}\) incurs only \(O(r_{\text{t}}^2)\) cost (\(16{\times}\) for \(r_{\text{t}}{=}4\)), a \(4{,}096{\times}\) difference. We therefore adopt a temporal-first strategy: temporal upsampling via Transformer layers, followed by spatial upsampling via MLP and PixelShuffle [69].

3.4.0.1 Step 1: Temporal Upsampling.

Starting from the backbone output \(\mathbf{Y}\!\in\!\mathbb{R}^{B\times L\times D}\) (where \(L = T' H' W'\)), a linear layer expands channels by temporal factor \(r_{\text{t}}\): \[\mathbf{P}^{\text{temp}} = \mathrm{Linear}_{D \rightarrow D \cdot r_{\text{t}}}(\mathbf{Y}) \in \mathbb{R}^{B\times L\times (D \cdot r_{\text{t}})}. \label{eq:temporal95upsample}\tag{3}\] Reinterpreting expanded channels as new temporal indices yields \(\mathbf{P}^{\text{full}}\in\mathbb{R}^{B\times (T' r_{\text{t}} H' W')\times D}\).

3.4.0.2 Step 2: Temporal Refinement.

Two Transformer blocks process \(\mathbf{P}^{\text{full}}\) using the same streaming mechanism from Sec. 3.3, with expanded window size \(W_{\text{frm}}^{\text{full}} = r_{\text{t}}\cdot W_{\text{frm}}\) to preserve effective temporal context.

3.4.0.3 Step 3: Spatial Upsampling.

A 2-layer MLP projects features from \(D\) to \(C\cdot r_{\text{s}}^{2}\) channels, followed by PixelShuffle [69] to produce the final output \(\hat{\mathbf{x}}\in\!\mathbb{R}^{B\times C\times (T' r_{\text{t}})\times (H'r_{\text{s}})\times (W'r_{\text{s}})}\).

3.5 Training Objectives↩︎

The decoder \(\mathcal{D}\) is trained with a combination of pixel-wise, perceptual, and adversarial losses: \[\mathcal{L}_{\text{total}} = \lambda_{\text{L1}} \mathcal{L}_{\text{L1}} + \lambda_{\text{LPIPS}} \mathcal{L}_{\text{LPIPS}} + \lambda_{\text{adv}} \mathcal{L}_{\text{adv}}, \label{eq:loss}\tag{4}\] where \(\mathcal{L}_{\text{L1}}\) is the pixel-wise L1 loss between the reconstructed video \(\hat{\mathbf{x}}\) and ground truth \(\mathbf{x}\), \(\mathcal{L}_{\text{LPIPS}}\) [70] measures perceptual similarity in a pretrained feature space, and \(\mathcal{L}_{\text{adv}}\) [71] is computed using a 3D patch-based discriminator [72], [73]. L1 ensures pixel-level fidelity, LPIPS encourages perceptually realistic outputs, and the adversarial term produces sharp high-frequency details. Hyperparameter values are provided in the supplementary material. Algorithm 4 summarizes the full streaming pipeline.

4 Experiments↩︎

Unless stated otherwise, all experiments use FlashDecoder trained on the Wan2.2 latent space (4\(\times\)​16\(\times\)​16). All training is conducted on a single node with 8 NVIDIA H100 GPUs.

3pt

4.1 Datasets and Evaluation Protocol↩︎

4.1.0.1 Training Data.

FlashDecoder is trained on joint image–video data in three stages. We use DataComp-small [74] (12.8M image–text pairs) and video data from Kinetics-600 [75] and an internal collection, with a 2:8 image-to-video sampling ratio. Stage 1 trains at 224\(\times\)​224 for fast convergence. Stage 2 transitions to 480p, 720p, and 1080p. Stage 3 adds adversarial training. Full details are provided in the supplementary material.

4.1.0.2 Evaluation Data.

We evaluate on the UltraVideo [76] dataset (clips_short_1.zip split), which provides 1,145 high-quality videos at 4K+ resolution covering diverse content from static scenes to fast motion. Each video is first resized to 720\(\times\)​1280 using PIL bicubic interpolation [77], [78], then center-cropped to the target resolution (480p, 720p, or 1080p), and 25-frame clips are extracted. Unless stated otherwise, all evaluations use 25-frame clips. We re-evaluate all baselines using their official implementations and checkpoints on a single H100 GPU.

4.1.0.3 Evaluation Metrics.

We report PSNR for pixel-level fidelity, LPIPS [70] for perceptual quality, and Content-Debiased FVD (rCD-FVD) [79] for temporal realism. Standard FVD [80] is known to be biased toward per-frame spatial quality rather than temporal consistency; rCD-FVD removes this content bias, providing a more faithful measure of temporal coherence. We refer to rCD-FVD as rFVD for simplicity. Throughput (FPS) is measured as decoded frames per second, and Mem denotes peak GPU memory in GB during decoding. FlashDecoder uses streaming mode while other methods use their native inference modes.

4.2 Effectiveness of Each Component↩︎

Table 1 ablates each component using FlashDecoder-S (Table 4). We evaluate on 480p with 17-frame clips for fast iteration. The baseline is a vanilla Transformer with full block causal attention (no windowing). Components are added incrementally: rows (a)–(d) train for 100K iterations at 224\(\times\)​224; rows (e)–(g) add scale-up, streaming training at 480p/720p, and adversarial training, respectively.

4.2.0.1 Baseline and Streaming Efficiency.

The baseline’s KV cache grows linearly with video length, collapsing throughput from 331.4 to 16.6 FPS. We additionally compare Sliding-Window Causal Attention (SW-CA), a separate reference that restricts attention to a fixed window during training. SW-CA restores stable throughput (333.8 FPS) but still requires mask materialization, causing out-of-memory failures at 480p and 720p. Both SW-CA and GQA increase rFVD over the baseline, indicating that limiting temporal context without dedicated temporal modeling hurts temporal coherence.

4.2.0.2 Architectural Components.

Among the added components, Temporal Refinement (TR) yields the largest single gain (rFVD: 121.87\(\rightarrow\)​86.94), confirming that raw channel expansion alone produces temporally inconsistent frames that benefit from dedicated refinement. Spatial Upsampling (SU) improves pixel fidelity (PSNR, LPIPS) but slightly increases rFVD, suggesting a trade-off between spatial detail and temporal smoothness. Model scaling provides consistent improvements across all metrics.

4.2.0.3 Streaming Training and Adversarial Loss.

The most significant jump occurs when streaming training is introduced (row e\(\rightarrow\)f). Streaming training enables high-resolution fine-tuning at 480p and 720p, directly improving reconstruction quality by closing the domain gap between 224\(\times\)​224 pretraining and the evaluation resolution. Adversarial training (row f\(\rightarrow\)g) trades a small PSNR decrease (37.52\(\rightarrow\)​37.08) for sharper outputs (rFVD: 12.29\(\rightarrow\)​10.77), a well-known characteristic of GAN-based losses [71].

4.3 Video Reconstruction Results↩︎

Table ¿tbl:tab:video95benchmark? compares FlashDecoder against state-of-the-art video decoders on UltraVideo at 480p, 720p, and 1080p, grouped by compression ratio (T\(\times\)H\(\times\)W). We train separate FlashDecoder-XL models for each compression group: one on the Wan2.1 encoder latent space (4\(\times\)​8\(\times\)​8) and one on the Wan2.2 encoder latent space (4\(\times\)​16\(\times\)​16).

4.3.0.1 Convolution-based Models.

In the 4\(\times\)​16\(\times\)​16 group, FlashDecoder-XL closely matches Wan2.2 [2] in PSNR and LPIPS across all three resolutions while streaming \(3.6{\times}\)–\(4.7{\times}\) faster with up to \(11{\times}\) lower peak memory. The memory gap is most pronounced at 1080p: 3.7 GB versus 41.0 GB. In the 4\(\times\)​8\(\times\)​8 group, FlashDecoder-XL achieves reconstruction quality comparable to Wan2.1, though it falls behind HunyuanVideo [40]. While rFVD is moderately higher than convolutional baselines in both groups, we note that these are production-level decoders likely trained with significantly more compute and data than our single-node setup. Visualizations are shown in Figure 2 and the supplementary material.

4.3.0.2 Transformer-based Decoders.

Existing Transformer decoders face a quality–streaming trade-off: OmniTokenizer [44] supports streaming but at low quality, while MAGI-1 [46] and AToken [47] achieve better quality through bidirectional attention but cannot stream and degrade in throughput with video length. FlashDecoder-XL resolves this by achieving higher reconstruction quality than both MAGI-1 and AToken while streaming at \(2.5{\times}\)–\(13{\times}\) higher throughput, without padding or blending.

3pt

3pt

4.3.0.3 Generalization Across Latent Spaces.

FlashDecoder is not tied to a specific encoder. Table 2 shows results when training on the Wan2.1 latent space, which uses \(8{\times}\) spatial compression instead of \(16{\times}\). Because \(8{\times}\) compression produces \(2{\times}\) more spatial tokens per dimension (\(4{\times}\) total), we apply PixelUnshuffle with factor 2 to the latent, folding the extra spatial dimensions into channels. This yields the same spatial token count \(L_{\text{frm}}\) as the \(16{\times}\) setting, so the Transformer backbone operates with minimal architecture changes. FlashDecoder-XL achieves comparable quality to Wan2.1 (37.46 vs. PSNR at 720p) with \(4.8{\times}\) higher throughput and \(6.8{\times}\) lower memory.

4.3.0.4 Long Video Decoding.

Because the KV cache window is fixed, FlashDecoder maintains constant memory regardless of video length. We assign RoPE positions relative to the current window rather than the absolute frame index, so positional encodings always stay within the range seen during training, enabling theoretically infinite-length decoding. Figure 5 shows per-frame PSNR on 720p videos exceeding 400 frames. FlashDecoder-XL maintains stable reconstruction quality throughout. Wan2.2 also supports streaming but consumes significantly more memory per frame.

3pt

4.4 Window Size Ablation↩︎

Table 3 ablates the window size \(W_{\text{frm}}\). Performance is stable across \(W_{\text{frm}} \in \{2, 3, 4\}\), with \(W_{\text{frm}}{=}2\) providing the best trade-off: similar quality to \(W_{\text{frm}}{=}4\) with lower memory (2.4 vs. GB) and higher throughput (76.3 vs. FPS). This suggests that for latent decoding, attending to just one previous latent frame provides sufficient temporal context.

4.5 Scaling Analysis↩︎

Table 4 shows four FlashDecoder variants trained at 224\(\times\)​224 for 150K iterations and evaluated at 480p. All variants share the same architecture (GQA Transformer blocks with temporal refinement and PixelShuffle spatial upsampling) and differ in width (\(D\)), depth, number of heads (\(N\)), and KV groups (\(G\)). FlashDecoder-XL, the largest variant, uses \(D{=}1536\) with 20 backbone blocks, 2 temporal refinement blocks, 24 attention heads, and 3 KV groups (full configuration in supplementary material). Reconstruction quality improves steadily with model size: PSNR rises from 30.90 to 33.81 and rFVD drops from 89.23 to 31.00. Throughput decreases with scale (254.5 \(\rightarrow\) 166.0 FPS), yet even FlashDecoder-XL comfortably maintains real-time streaming performance at all resolutions.

4.6 Inference Optimization↩︎

FlashDecoder’s streaming architecture is particularly amenable to inference optimization because each frame-level forward pass has a fixed, data-independent compute graph. We apply four progressive optimizations, each building on the previous: (1) torch.compile fuses elementwise operations (RMSNorm, SiLU, residual additions) into single GPU kernels, reducing memory bandwidth pressure; (2) CUDA graph capture eliminates per-step Python dispatch and kernel launch overhead by replaying the entire forward pass as a single graph; (3) precomputed RoPE tables and a FlashAttention-3 custom operator remove dynamic allocations and graph breaks that would otherwise prevent end-to-end graph capture; (4) static-calibrated FP8 quantization of all MLP layers exploits H100 FP8 Tensor Cores for higher matmul throughput. The first three optimizations are lossless. FP8 quantization incurs a quality trade-off: PSNR drops by 0.06–0.71 dB and rFVD increases by up to 0.94 depending on resolution (Table ¿tbl:tab:video95benchmark?). These optimizations are complementary and stack multiplicatively. The resulting FlashDecoder-XL-Opt (Table ¿tbl:tab:video95benchmark?) achieves up to \(12{\times}\) higher throughput than Wan2.2 at 480p while using under 2 GB peak memory on the 4\(\times\)​16\(\times\)​16 latent space.

5 Conclusion↩︎

We introduced FlashDecoder, a pure-Transformer latent-to-pixel decoder that achieves real-time streaming by processing one frame at a time with a fixed-size rolling KV cache. Two findings stand out from our experiments. First, enforcing causality through temporal processing order rather than explicit masks removes the memory barrier to high-resolution training, enabling stable training up to 1080p. Second, this high-resolution training closes the reconstruction quality gap with convolutional decoders, a gap that has limited prior Transformer decoders. On both the Wan2.1 and Wan2.2 latent spaces, FlashDecoder-XL matches convolutional decoder reconstruction quality while delivering \(3.6{\times}\)–\(4.7{\times}\) faster streaming throughput and up to \(11{\times}\) lower GPU memory. With architecture-aware inference optimizations, the throughput gap widens to \(12{\times}\), enabling real-time high-resolution decoding on a single GPU. A natural next step is to pair FlashDecoder with a streaming Transformer encoder and train the full VAE from scratch, potentially unlocking latent spaces better suited to Transformer-based generation and decoding.

Acknowledgments↩︎

Minguk Kang is a participating researcher at POSTECH and a full-time employee at Pika Labs. We thank Joonghyuk Shin for helpful discussions and assistance with MotionStream experiments, and Zhicheng Sun and Cade Li for valuable discussions. This work was supported by Samsung Electronics Co., Ltd.(Samsung AI Center) and the IITP grants (RS-2022-II220290, RS-2022-II220926, RS-2019-II191906) funded by the Korea government (MSIT).

This supplementary material provides training specifications (Section 6), inference protocols (Section 7), additional visual results (Section 8), and limitations and future directions (Section 9).

6 Training Specifications↩︎

6.1 Dataset Details↩︎

Image Data. We utilize DataComp-small [74], comprising 12.8M image-text pairs. During preprocessing, we apply probabilistic augmentation that randomly selects among random cropping (40%), center cropping (30%), or resizing (30%) to the target resolution. Images smaller than the target resolution are filtered to prevent upsampling artifacts.

Video Data. Our video corpus combines Kinetics-600 [75] and an internal high-resolution collection of approximately 200K clips. From each video, we sample 17 consecutive frames at native frame rate. Preprocessing employs a two-stage spatial transformation: frames are first resized so that the shorter side matches the target resolution (480p, 720p, or 1080p depending on the training stage) while preserving the original aspect ratio, then cropped to the target resolution using either center crop (60%) or random crop (40%). Videos below the target resolution are filtered out. All resizing uses anti-aliased PIL bicubic interpolation [77], [78].

6.2 Multi-Stage Training Protocol↩︎

FlashDecoder follows a sequential three-stage training protocol, with each stage building upon the previous one. Training hyperparameters are summarized in Table 5.

Stage 1: Low-Resolution Pre-training. This stage establishes fundamental reconstruction capabilities at reduced computational cost. We train on 224\(\times\)​224\(\times\)​17 video clips and 256\(\times\)​256 images with a 2:8 image-to-video sampling ratio to balance temporal coherence with spatial fidelity. Training proceeds for 200K iterations with batch size 16. The reconstruction objective combines L1 loss and perceptual loss [70] with weights of 1.0 and 0.1, respectively.

Stage 2: High-Resolution Training. We transition to higher resolutions to minimize the domain gap between training and inference. The model is trained on 480p clips (480\(\times\)​832\(\times\)​17), 720p clips (720\(\times\)​1280\(\times\)​17), 1080p clips (1080\(\times\)​1920\(\times\)​17), and 512\(\times\)​512 images. This diverse resolution mixture enables the model to handle varying spatial resolutions during inference. We reduce the learning rate by 10\(\times\) and train for 100K iterations with batch size 8. Loss weights are adjusted to 1.0 for L1 and 0.25 for perceptual loss. The perceptual loss is computed on random 224\(\times\)​224 crops to keep memory manageable at high resolutions.

Stage 3: Adversarial Post-training. To enhance fine-grained details, we introduce adversarial training using the same data configuration as Stage 2 but excluding 1080p clips due to the additional memory overhead of the discriminator. This stage enables the decoder to synthesize sharper high-frequency textures that reconstruction losses alone cannot capture. We extend VQGAN’s 2D PatchGAN discriminator [72], [81] to 3D for spatiotemporal processing, and train it with non-saturating logistic loss [71] and R1 regularization [82]. Both the perceptual and adversarial losses are computed on random 224\(\times\)​224 crops of the decoded output.

7 Inference Protocols↩︎

For fair comparison, we re-evaluate all baseline models using their official repository implementations and released checkpoints under identical settings on a single NVIDIA H100 GPU (80GB).

7.1 Throughput Measurement↩︎

To ensure fair comparison, we evaluate each model in its officially supported inference mode. Wan2.2-TAEHV [48], Wan2.2 [2], and FlashDecoder natively support streaming and are evaluated accordingly. Other VAEs (HunyuanVideo, AToken, MAGI-1) process entire clips in batch mode using their official implementations, as forcing them into a streaming setup would require chunking and blending that degrades their reconstruction quality. Throughput is measured in frames per second (FPS), calculated as total decoded frames divided by total decoding time.

8 More Visual Results↩︎

Figures 6–8 show additional 720p reconstruction results from Wan2.2-TAEHV [48], AToken [47], Wan2.2 [2], and FlashDecoder-XL-Opt. Wan2.2-TAEHV struggles with fine details; AToken produces smoother but blurrier outputs. Both Wan2.2 and FlashDecoder-XL-Opt produce sharp results, with Wan2.2 showing slightly finer details in some cases. FlashDecoder-XL-Opt achieves over \(9{\times}\) higher throughput (151.0 vs. FPS).

9 Limitations and Future Directions↩︎

Decoder-only design. FlashDecoder replaces only the decoder while keeping the pretrained convolutional encoder fixed. The latent space therefore inherits the encoder’s characteristics, which favor spatial locality by design; what properties a Transformer-based encoder–decoder pair would learn remains unexplored. Designing a streaming Transformer encoder to pair with FlashDecoder and training the full VAE from scratch is a clear next step. Such an encoder–decoder pair would also eliminate the resolution gap between low-resolution VAE training (\(256{\times}256\)) and high-resolution diffusion training (720p, 1080p), since our streaming architecture scales to high resolutions. This is particularly relevant for recent end-to-end frameworks such as Unified Latents [84], which jointly train the VAE and diffusion model. Existing convolutional video VAEs are difficult to jointly train with a diffusion model at 480p or 720p due to their high memory consumption. FlashDecoder’s low memory footprint makes such joint training feasible. Training the VAE and diffusion model at the same resolution used during inference would ensure that the latent space has good diffusibility at that resolution, avoiding potential mismatches caused by training at a lower resolution.

rFVD gap. FlashDecoder-XL falls short of Wan2.2 [2] and HunyuanVideo [40] in rFVD [79], [80], despite comparable PSNR and LPIPS. We trained on a single 8-GPU node, whereas these production decoders likely used significantly more compute and data. Scaling up model capacity and adversarial training duration is expected to close this gap.

Integration with Representation Autoencoders. Recent work on Representation Autoencoders (RAE [57]) pairs frozen pretrained encoders (e.g., DINOv2 [60], SigLIP [61]) with Transformer decoders for image generation. Extending this paradigm to video with a streaming-capable decoder like FlashDecoder is a promising direction.

References↩︎