1 Introduction↩︎

Video super-resolution (VSR) aims to restore high-resolution (HR) videos from low-resolution (LR) inputs by leveraging temporal information across adjacent frames. While conventional deep learning-based VSR methods [1]–[7] have achieved impressive reconstruction accuracy, they often suffer from over-smoothed textures and fail to synthesize realistic high-frequency details. To overcome this perception-distortion trade-off, recent diffusion model (DM)-based approaches [8]–[13] have brought powerful generative priors to the VSR task, significantly improving perceptual quality. Despite their impressive generative capabilities, most existing DM-based VSR models are designed and trained for a fixed integer scaling factor (typically \(4\times\)). Given that practical scenarios frequently demand arbitrary and continuous resolution adjustments, independently training computationally expensive diffusion models for every specific scale is highly inefficient.

To support various upsampling scales within a single network, arbitrary-scale VSR (AVSR) methods [14]–[19] have been proposed by adopting coordinate-based implicit neural representations (INRs) [20]. However, since these methods predominantly rely on deterministic regression losses (, MSE, \(L_1\)), they struggle to capture high-frequency details, often leading to severe over-smoothing at large upscaling factors (, \(6\times\) or \(8\times\)) [17], [18], [21]. Motivated by this limitation, recent approaches in the single-image domain [21]–[23] have combined latent diffusion models (LDMs) with INR decoders, demonstrating that generative priors can synthesize high-fidelity details while retaining continuous-scale rendering. Although integrating generative priors with continuous coordinate representations is promising, extending this paradigm to the video domain introduces a distinct set of non-trivial challenges.

A central hurdle is temporal consistency, as continuous coordinate decoding is highly sensitive to the stability of the underlying features. In frame-wise diffusion models, the inherent stochasticity of per-frame sampling induces minor frame-to-frame variations, which the subsequent decoding process can easily amplify into severe temporal flickering [10]. While recent DM-based VSR methods introduce various temporal cues, such as warped frame-level guidance [8] and feature-aware modules within the diffusion pipeline [9]–[12], these mechanisms are primarily tailored for fixed-scale VSR. Consequently, they do not explicitly provide the scale-agnostic, strictly aligned feature priors required for seamless continuous decoding. Alternatively, relying on full video diffusion models with heavy 3D U-Nets [24] to explicitly upsample the input sequence prior to denoising, as conceptually illustrated in Fig. 2 (a), entangles the heavy denoising process with the target resolution. As empirically demonstrated in Fig. 1 (b), this approach incurs massive memory growth as the scaling factor increases, rendering practical long-video inference fundamentally challenging.

To break this entanglement between the heavy denoising process and target resolution, we propose AVSR-Diff (Arbitrary-scale VSR with Diffusion), a novel decoupled framework designed to ensure both temporal stability and scale-agnostic computational efficiency. As conceptually illustrated in Fig. 2 (b), our approach strictly decouples the generative prior extraction from the resolution-specific rendering. We perform diffusion sampling exclusively within a fixed LR latent space, ensuring the generative sampling cost remains completely independent of the target scales (, scale-agnostic). However, directly translating these sampled latents via continuous coordinate decoding easily magnifies latent stochasticity into severe temporal flickering. To solve this, we introduce a Temporally-Gated Feature Recurrence (TGFR) module operating within the residual branch of a ControlNet [25], which guides the frozen denoising U-Net [26]. By employing deformable convolutions (DCNs) [27] for precise sub-pixel feature alignment and an adaptive gate-based fusion strategy, TGFR effectively suppresses drift and flickering artifacts, thereby providing the temporally aligned latent priors required for seamless continuous decoding.

Building upon these temporally aligned priors, we extend the pre-trained fixed-scale image decoder into a continuous video decoder to flexibly translate the latent features into arbitrary resolution video frames. To facilitate this transition, we repurpose the TGFR mechanism within the decoding process, ensuring that continuous spatial querying maintains temporal consistency. Crucially, recognizing that the requirement for high-frequency details varies significantly depending on the target scale, we propose a Scale-Aware Fourier Refinement (SAFR) module. Operating within the deep feature space of the frozen VAE decoder, SAFR dynamically modulates the frequency components of the features based on the target scale. These scale-aware features are then continuously queried and decoded by a coordinate-based INR [20]. As a result, this decoupled architecture enables high-fidelity, temporally consistent pixel rendering at any continuous resolution, effectively obviating the need for the computationally expensive HR denoising typically required by video diffusion models.

In summary, the main contributions of our work are three-fold:

• We propose AVSR-Diff, a novel decoupled framework for DM-based arbitrary-scale VSR. By separating the diffusion process from the target resolution, our approach performs scale-agnostic latent denoising for computational efficiency, while the proposed Temporally-Gated Feature Recurrence (TGFR) module extracts the temporally aligned priors necessary to support sensitive continuous decoding.

• We design an arbitrary-scale continuous video decoder to flexibly translate latent priors into high-fidelity pixels. By integrating a Scale-Aware Fourier Refinement (SAFR) module, our decoder dynamically adapts frequency components to the target scale, effectively synthesizing high-frequency details while rigorously preserving temporal coherence.

• Extensive experiments demonstrate the superiority of AVSR-Diff. Remarkably, our arbitrary-scale framework outperforms recent fixed-scale generative models even on their native \(4\times\) resolution. Furthermore, it consistently preserves high-frequency details and strong temporal stability across arbitrary scales, surpassing state-of-the-art arbitrary-scale baselines.

2 Related Work↩︎

2.1 Video Super-Resolution↩︎

Conventional VSR methods [1]–[7] rely on regression objectives, which inherently produce over-smoothed results. To synthesize high-frequency textures, GAN-based VSR models [28]–[30] have been proposed, but they often struggle with a severe consistency-quality dilemma [30], where their generative capacity amplifies temporal flickering. Recently, driven by the superior perceptual quality of diffusion models (DMs) [26], [31], DM-based VSR approaches have been actively explored. To mitigate the temporal flickering caused by DMs’ inherent stochasticity, recent methods incorporate various temporal conditioning mechanisms, such as temporal attention modules, warping-based guidance, and motion-guided sampling, into pre-trained diffusion priors [8]–[12]. Despite their impressive generative capabilities, most mainstream frameworks are restricted to a fixed integer scaling factor (typically \(4\times\)). While VEnhancer [24] supports arbitrary-scale upsampling, its reliance on heavy 3D U-Nets for spatio-temporal denoising at the target resolution becomes computationally impractical at large scaling factors. Thus, efficiently bridging generative priors with continuous-scale rendering remains a critical challenge.

2.2 Arbitrary-Scale Super-Resolution↩︎

To overcome the limitations of fixed-scale upsampling, arbitrary-scale super-resolution methods have been proposed by adopting coordinate-based INRs. LIIF [20] pioneered this paradigm in the single-image domain by formulating SR as a continuous image representation. Extending this concept to the video domain, subsequent works [14]–[19] have actively advanced continuous spatio-temporal representations by incorporating sophisticated motion compensation and temporal feature alignment mechanisms. Despite their architectural flexibility in querying continuous spatial coordinates, these models predominantly rely on deterministic regression objectives during training. Consequently, while they achieve impressive reconstruction metrics, they fundamentally suffer from the perception-distortion trade-off. Particularly at large scaling factors (, \(6\times\) or \(8\times\)), they exhibit severe over-smoothing artifacts and fail to recover the missing high-frequency details.

2.3 Arbitrary-Scale Generative Super-Resolution↩︎

To leverage the powerful generative priors of LDMs for arbitrary-scale rendering, recent studies [21]–[23] have successfully integrated LDMs with coordinate-based INRs. By establishing this continuous generative paradigm in the single-image domain, they can synthesize realistic high-frequency details at any target scale. However, extending this framework to the video domain remains largely unexplored due to a critical bottleneck arising from the incompatibility between the stochastic feature variations of diffusion sampling and the high sensitivity of continuous coordinate decoders, which inevitably amplify temporal flickering. To address this gap, we propose AVSR-Diff. By synergistically integrating the TGFR mechanism for strict temporal feature alignment and the SAFR module for scale-adaptive frequency modulation, our decoupled architecture achieves scale-agnostic computational efficiency. Furthermore, it successfully adapts diffusion priors to arbitrary video scales while rigorously preserving temporal coherence.

3 Method↩︎

3.1 Overview of AVSR-Diff↩︎

Fig. 3 shows the overall architectural design of our proposed AVSR-Diff. Given an LR input video sequence \(\mathbf{x} = \{x^i\}_{i=1}^N \in \mathbb{R}^{N \times 3 \times H \times W}\) and a target continuous scaling factor \(s > 1\), our goal is to reconstruct an HR video sequence \(\hat{\mathbf{y}} = \{\hat{y}^i\}_{i=1}^N \in \mathbb{R}^{N \times 3 \times sH \times sW}\) that is both perceptually faithful and temporally consistent. Directly applying diffusion sampling in the HR pixel space is computationally prohibitive and memory-intensive, particularly for video tasks. To overcome this, our AVSR-Diff is designed as a completely decoupled framework built upon a pre-trained single-image super-resolution LDM (SD\(\times 4\) Upscaler) [26], consisting of a VAE encoder-decoder pair \((\mathcal{E}, \mathcal{D})\) and a denoising U-Net \(\epsilon_\theta\). Our architecture is explicitly decomposed into two stages: scale-agnostic latent denoising and arbitrary-scale continuous decoding.

Scale-Agnostic Latent Denoising. To avoid expensive resolution-specific diffusion sampling, as in VEnhancer [24], we perform the generative denoising process entirely within a fixed-resolution LR latent space. Since the spatial dimensions of the LR frame \(x^i\) naturally match those of the latent space, at each diffusion timestep \(t\), we directly concatenate it with the noisy latent \(z_t^i \in \mathbb{R}^{C \times H \times W}\) to serve as a structural condition for both the U-Net and the ControlNet [25]. Crucially, to prevent the severe temporal flickering caused by stochastic sampling, we introduce the Temporally-Gated Feature Recurrence (TGFR) module into the ControlNet branch (detailed in Sec. 3.2). The TGFR module explicitly aligns and propagates structural priors across adjacent frames, providing the frozen U-Net \(\epsilon_\theta\) with strictly aligned conditions to produce temporally coherent denoising predictions, resulting in a temporally consistent latent sequence \(\mathbf{z}_0 = \{z_0^i\}_{i=1}^N \in \mathbb{R}^{N \times C \times H \times W}\).

Continuous Arbitrary-Scale Decoding. Given the temporally aligned latent sequence \(\mathbf{z}_0\), we directly decode these priors into the target resolution. Specifically, we extend the pre-trained fixed-scale image VAE decoder \(\mathcal{D}\) into a continuous video decoder \(\mathcal{D}_{s}\) by introducing three key components: (i) repurposed TGFR blocks for bidirectional feature propagation to maintain temporal coherence within the deep feature space, (ii) a Scale-Aware Fourier Refinement (SAFR) module that dynamically modulates high-frequency components according to the target scale \(s\), and (iii) a coordinate-based INR [20] for continuous spatial querying. This decoupled design effectively renders the final HR frames \(\hat{\mathbf{y}}\) from the refined features, enabling arbitrary-scale VSR with stable temporal dynamics without the need for resolution-specific diffusion sampling.

3.2 Temporally-Gated Feature Recurrence (TGFR)↩︎

During the scale-agnostic latent denoising process, our goal is to produce temporally coherent latent priors. At each denoising step \(t\), the LR frame \(x^i\) is directly concatenated with the noisy latent \(z_t^i\) to serve as a base structural condition for both the frozen U-Net \(\epsilon_\theta\) and the trainable ControlNet \(\mathcal{C}_\phi\). While this provides robust spatial guidance for individual frames, stochastic sampling across frames inevitably leads to severe temporal flickering.

To ensure robust temporal consistency, we adopt the temporal conditioning strategy of StableVSR [8], which encompasses both a frame-wise bidirectional sampling strategy and warped RGB guidance. Since the sampling direction alternates at each diffusion timestep, we formulate the feature propagation from the previous frame \(i-1\) to the current frame \(i\) within a single forward step for notational simplicity. Under this setting, we first approximate the clean latent \(\hat{z}_0^{i-1}\) from \(z_t^{i-1}\) using the noise predicted by \(\epsilon_\theta\): \[\hat{z}_0^{i-1} = \frac{1}{\sqrt{\bar{\alpha}_t}} \left( z_t^{i-1} - \sqrt{1-\bar{\alpha}_t} \epsilon_\theta([z_t^{i-1}, x^{i-1}], t; \mathcal{C}_{\phi}) \right),\] where \(\bar{\alpha}_t\) follows the standard noise schedule of the diffusion process. We then decode this latent into the RGB space, \(\hat{y}_{\text{base}}^{i-1} = \mathcal{D}(\hat{z}_0^{i-1})\), and backward warp [32] it to the current frame using an optical flow \(f^{i \rightarrow i-1}\) estimated from the LR inputs by pretrained RAFT [33]. This warped image \(\hat{y}_{\text{base}}^{i-1 \rightarrow i} = \mathrm{Warp}(\hat{y}_{\text{base}}^{i-1}, f^{i \rightarrow i-1})\) serves as an explicit pixel-level condition for ControlNet \(\mathcal{C}_\phi\). Although \(\hat{y}_{\text{base}}^{i-1}\) is decoded at the fixed-scale VAE’s native resolution (, \(4\times\)), it is strictly utilized as a conditioning anchor to stabilize the latent denoising process, and the final arbitrary-scale rendering is handled entirely by our continuous decoder \(\mathcal{D}_{s}\).

However, relying solely on warped RGB guidance stabilizes content only at a macroscopic level. Our arbitrary-scale continuous video decoder \(\mathcal{D}_{s}\) is highly sensitive to feature-level temporal drift, which easily magnifies into flickering artifacts. Therefore, we introduce the TGFR module to recurrently align the deep residual features inside the ControlNet.

Let \(C^i\) denote the intermediate residual feature extracted by the ControlNet at the current frame, and \(H^{i-1}\) be the recurrent feature propagated from the previous frame. To align \(H^{i-1}\) with \(C^i\), we first apply a coarse flow-based warping, followed by a refinement block: \(\tilde{H}^{i-1 \rightarrow i} = \mathrm{ResBlock}(\mathrm{Warp}(H^{i-1}, f^{i \rightarrow i-1}))\). To rectify the inherent inaccuracies of optical flow at the sub-pixel feature level, we employ a DCN [27], inspired by the flow-guided deformable alignment in BasicVSR++ [2], but repurposed here to recurrently align ControlNet residual features rather than backbone features: \[H_{\text{aligned}}^{i-1 \rightarrow i} = \mathrm{DCN}(\tilde{H}^{i-1 \rightarrow i}, \Delta p^i, \Delta m^i), \label{eq:dcn}\tag{1}\] where the dynamic offsets \(\Delta p^i\) and modulation masks \(\Delta m^i\) are learned from the concatenated features \([C^i, \tilde{H}^{i-1 \rightarrow i}]\). Finally, while the DCN mask resolves local spatial misalignments, we employ a spatial gated fusion mechanism to evaluate the temporal reliability of the aligned features: \[\begin{align} G^i &= \sigma\Big(\mathrm{Conv}\big([C^i, H_{\text{aligned}}^{i-1 \rightarrow i}, |C^i - H_{\text{aligned}}^{i-1 \rightarrow i}|]\big)\Big), \\ H^i &= C^i + G^i \odot H_{\text{aligned}}^{i-1 \rightarrow i}, \end{align} \label{eq:fusion}\tag{2}\] where \(G^i \in [0,1]\) is a learned gate map and \(\sigma\) denotes the sigmoid function. By jointly processing the features and their absolute difference \(|C^i - H_{\text{aligned}}^{i-1 \rightarrow i}|\), the gate actively suppresses unreliable past information, preventing progressive error accumulation and subsequent temporal artifacts. Note that while the above formulation is defined for a single feature level for brevity, we apply this TGFR mechanism across all multi-scale ControlNet residual feature maps in a recurrent manner. The optical flow \(f\) is appropriately resized for each feature scale prior to warping. These fused multi-scale residuals \(H^i\) are both propagated to the next frame and injected into the frozen U-Net \(\epsilon_\theta\), strictly constraining the denoising trajectory to produce a temporally aligned latent sequence.

3.3 Arbitrary-Scale Continuous Video Decoding↩︎

Given the temporally aligned latent sequence \(\mathbf{z}_0 = \{z_0^i\}_{i=1}^N\) obtained from the scale-agnostic latent denoising process, our next objective is to render the HR frames \(\hat{\mathbf{y}}\) at an arbitrary target scale \(s > 1\). Instead of executing computationally heavy diffusion sampling directly at the target resolution, we extend the pre-trained fixed-scale image VAE decoder \(\mathcal{D}\) into a continuous video decoder \(\mathcal{D}_{s}\) by integrating an implicit coordinate-based renderer and frequency modulation.

Scale-Aware Feature Extraction. We bypass the pixel-level reconstruction of the pretrained \(\mathcal{D}\), which is strictly bound to a fixed \(4\times\) resolution. Instead, given the intermediate deep feature \(F^i \in \mathbb{R}^{C' \times H \times W}\) extracted from \(\mathcal{D}\) for the latent \(z_0^i\), we apply a scale-aware adaptation. Since the optimal high-frequency details required for reconstruction vary depending on the target scale \(s\), we modulate the normalized deep features \(F^i\) using a sinusoidal scale embedding: \[u^i = \mathrm{GN}(F^i) \odot (1 + \gamma(s)) + \beta(s),\] where \(\mathrm{GN}\) denotes Group Normalization [34], and \((\gamma(s), \beta(s))\) are channel-wise scale-aware affine parameters predicted by an MLP from the positional encoding [35] of \(s\).

To ensure that this spatial modulation does not disrupt temporal coherence, and to explicitly extend the single-image representation into a temporal video model, we apply a bidirectional feature propagation mechanism [36]. By repurposing our TGFR module, we recurrently align and fuse features from adjacent frames in a cascaded manner. Specifically, we sequentially perform forward and backward propagation: \[\begin{align} u_{\text{fwd}}^i &= \mathrm{TGFR}(u^i, u_{\text{fwd}}^{i-1}, f^{i \rightarrow i-1}), \\ \tilde{u}^i &= \mathrm{TGFR}(u_{\text{fwd}}^i, \tilde{u}^{i+1}, f^{i \rightarrow i+1}), \end{align}\] where \(\mathrm{TGFR}(v_{\text{curr}}, v_{\text{prior}}, f)\) denotes the DCN-based alignment and gated fusion (as defined in Eqs. 1 and 2 ) of a prior feature \(v_{\text{prior}}\) into the current feature \(v_{\text{curr}}\) guided by optical flow \(f\). This sequential bidirectional propagation effectively transforms the frame-wise modulated features into the temporally enriched features \(\tilde{u}^i\).

Scale-Aware Fourier Refinement (SAFR). While the spatial scale conditioning adapts the overall feature amplitude, reconstructing arbitrary-scale textures requires explicit modulation of their frequency-domain representations. As the target scale \(s\) increases, high-frequency components need to be dynamically synthesized or preserved to prevent over-smoothing. To achieve this, our SAFR module operates directly in the Fourier domain. Given the temporally enriched feature \(\tilde{u}^i\), we compute its frequency representation \(U^i = \mathcal{F}(\tilde{u}^i)\). We then apply complex-domain channel mixing followed by a scale-conditioned spectral gate: \[u_{\text{ref}}^i = \tilde{u}^i + \mathrm{Conv}\Big(\mathcal{F}^{-1}\big( \mathrm{Mix}_{\mathbb{C}}(U^i) \odot \psi(s) \big)\Big),\] where \(\mathcal{F}\) and \(\mathcal{F}^{-1}\) denote the 2D Fast Fourier Transform (FFT) and its inverse, respectively, and \(\psi(s)\) is a channel-wise spectral gate vector predicted by an MLP. Here, \(\mathrm{Mix}_{\mathbb{C}}\) performs \(1 \times 1\) convolution in the complex domain to blend channel-wise spectral information, and \(\mathrm{Conv}\) is a spatial convolution layer. This spectral gating mechanism explicitly modulates the spectral features conditioned on the target scale \(s\).

Continuous Coordinate-Based Rendering. Finally, to restore the HR video \(\hat{\mathbf{y}}\) without resolution constraints, we employ the LIIF [20] formulation. For any target HR pixel coordinate \(p \in [-1, 1]^2\), we query the surrounding local features from the scale-refined map \(u_{\text{ref}}^i\) using a continuous MLP to predict the RGB residual \(\Delta y^i(p)\). To stabilize the continuous rendering and accelerate convergence, we add a bilinearly sampled color from the original LR input \(x^i\): \[\hat{y}^i(p) = \Delta y^i(p) + \mathrm{Sample}_{\text{bilinear}}(x^i, p).\] This decoupled architecture effectively renders arbitrary-scale HR frames with strictly preserved temporal dynamics.

3.4 Training Objectives↩︎

To fully leverage the pre-trained fixed-scale single-image LDM, we freeze the original VAE and the denoising U-Net \(\epsilon_\theta\). The ControlNet \(\mathcal{C}_\phi\) and the continuous video decoder \(\mathcal{D}_s\) are trained independently, as both modules operate within the fixed latent space of the pre-trained LDM.

ControlNet Training. We optimize \(\mathcal{C}_\phi\) using a combination of a denoising loss and a gate regularization term: \[\mathcal{L}_{\text{CNet}} = \| \epsilon - \hat{\epsilon} \|_2^2 + \lambda_{\text{gate}} \frac{1}{|\mathcal{G}|} \sum_{G \in \mathcal{G}} \|G\|_1, \label{eq:gate}\tag{3}\] where \(\epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})\) is added Gaussian noise and \(\hat{\epsilon} = \epsilon_\theta([z_t, x], t; \mathcal{C}_\phi)\) is predicted noise. The first term supervises this noise prediction, while the second term applies a sparsity penalty to all spatial gate maps \(G\) collected in the set \(\mathcal{G}\) from the multi-scale TGFR blocks. This explicitly regularizes the ControlNet \(\mathcal{C}_\phi\) to open the gate and propagate features only when the resulting temporal denoising benefit strictly outweighs the activation penalty.

Continuous Decoder Training. The arbitrary-scale decoder \(\mathcal{D}_s\) is trained in two phases using cropped HR patches \(y\) and our predictions \(\hat{y}\) at randomly sampled continuous scales \(s\): \[\mathcal{L}_{\text{Decoder}} = \| \hat{y} - y \|_1 + \lambda_{\text{percep}} \mathcal{L}_{\text{percep}} + \lambda_{\text{adv}} \mathcal{L}_{\text{adv}}. \label{eq:decoder}\tag{4}\] In the first phase, \(\mathcal{D}_s\) is trained with the \(\mathcal{L}_1\) loss and the perceptual loss [37] \(\mathcal{L}_{\text{percep}}\) (\(\lambda_{\text{adv}}=0\)). In the second phase, we introduce a PatchGAN [38]-based adversarial loss \(\mathcal{L}_{\text{adv}}\) to further enhance fine-grained high-frequency realism.

4 Experiments↩︎

4.1 Implementation details↩︎

Our AVSR-Diff is built upon the pre-trained Stable Diffusion \(\times 4\) Upscaler (SD\(\times 4\) Upscaler) [26]. We employ the pre-trained RAFT [33] for optical flow estimation. As described in Sec. 3.4, the ControlNet \(\mathcal{C}_\phi\) and the continuous video decoder \(\mathcal{D}_s\) are trained independently using the Adam optimizer [39]. Across all training processes, the batch size is set to 32, and each input sequence consists of 8 LR frames with a spatial size of \(64\times64\). For the latent denoising, \(\mathcal{C}_\phi\) integrating our TGFR module is trained for 30K steps with a fixed learning rate of 1e-5. During inference, we employ DDPM [40] sampling with 50 steps for the denoising process. For the continuous decoding, \(\mathcal{D}_s\) is trained in two phases. The first phase optimizes the decoder solely for the \(\mathcal{L}_1\) and perceptual losses for 100K steps, using an initial learning rate of 1e-4 that gradually decreases via cosine annealing [41]. In the second phase, we introduce the adversarial loss and fine-tune \(\mathcal{D}_s\) for an additional 40K steps with a fixed learning rate of 1e-5. During the training of \(\mathcal{D}_s\), the target continuous scale \(s\) is uniformly sampled from \([1.1, 4.0]\).

4.2 Datasets and Evaluation Metrics↩︎

Datasets. We train our proposed AVSR-Diff on the standard REDS training set [42]. For testing, we evaluate on the REDS4 [42] and Vid4 [43] benchmarks.

Evaluation Metrics. We comprehensively evaluate the upscaled videos across three distinct aspects. To assess the perceptual quality, we adopt LPIPS [37] and DISTS [44]. For pixel-wise reconstruction fidelity, we report the standard PSNR and SSIM [45]. Finally, to rigorously measure temporal consistency and the suppression of flickering artifacts, we employ tLPIPS [46] and tOF [46]. For better readability across all quantitative evaluations, LPIPS, DISTS, and tOF are scaled by \(10^2\), and tLPIPS is scaled by \(10^3\).

4.3 Comparison with State-of-the-Art Methods↩︎

We extensively compare our proposed AVSR-Diff against state-of-the-art methods. The baselines comprise fixed-scale regression-based methods (BasicVSR++ [2], RVRT [3]), arbitrary-scale regression-based methods (VideoINR [14], MoTIF [15], ST-AVSR [17], SAVSR [16], BF-STVSR [18], and V\(^3\)VSR [47]), fixed-scale generative methods (RealBasicVSR [29], Upscale-A-Video (UAV) [9], MGLD-VSR [10], StableVSR [8], and STAR [11]), and the arbitrary-scale generative method VEnhancer [24]. Table 1 presents the comparison on the standard \(4\times\) VSR, while Table 2 extends the evaluation to arbitrary continuous scales (\(2\times\), \(3.25\times\), and \(8\times\)). Since existing fixed-scale generative methods only support integer scales (, \(4\times\)), we adapt them for arbitrary scales by first generating the \(4\times\) HR videos and subsequently resizing them to the target scale using bicubic interpolation.

Evaluation on Fixed \(4\times\) Scale. Table 1 presents the quantitative comparison on REDS4 and Vid4 for \(4\times\) VSR. As expected by the perception-distortion trade-off, regression-based methods achieve higher PSNR but suffer from low perceptual quality. Remarkably, despite being an arbitrary-scale continuous model, our AVSR-Diff outperforms existing fixed-scale generative VSR models on their native \(4\times\) resolution. It consistently achieves the best perceptual quality while maintaining the highest fidelity among all generative methods, demonstrating the optimal perception-distortion trade-off. Furthermore, our model exhibits exceptional temporal consistency. It achieves the best tLPIPS across both datasets and the best tOF on REDS4, confirming that our TGFR module successfully suppresses the severe flickering artifacts inherent to diffusion models.

Evaluation on Arbitrary Scales. Table 2 demonstrates our model’s superiority across continuous (\(3.25\times\)) and large (\(8\times\)) scales. While fixed-scale generative models suffer from severe aliasing when relying on bicubic interpolation, AVSR-Diff directly renders HR frames, consistently outperforming them and the recent arbitrary-scale VEnhancer [24] in both perceptual quality and temporal consistency. Notably, at the \(2\times\) scale, AVSR-Diff achieves a PSNR (35.47 dB) comparable to strong regression-based baselines despite being a generative model, while delivering far superior perceptual quality and temporal stability, effectively bridging the perception-distortion gap. This confirms our continuous video decoder’s robustness and fine-detail preservation for arbitrary and large-scale VSR.

Qualitative Results. Fig. 4 and Fig. 1 (a) provide visual comparisons across various scales. As observed, regression-based methods inherently produce overly smoothed outputs, failing to recover complex textures. Conversely, while recent generative models actively synthesize high-frequency details, they frequently fail to preserve the original structural fidelity, often generating incorrect patterns or structurally distorted artifacts. In contrast, our AVSR-Diff consistently maintains the structural identity of the LR inputs. By effectively balancing robust reconstruction fidelity with generative priors, our method synthesizes sharp, fine-grained textures that remain faithful to the original content, without introducing unnatural structural distortions, even at large scaling factors (, \(8\times\)). Further quantitative and qualitative comparisons across diverse scales are provided in the Suppl.

4.4 Ablation Study↩︎

Effectiveness of Key Components. Table 3 details the progressive improvements achieved by our proposed modules. Starting from the baseline (a), which represents StableVSR [8], directly substituting the pretrained decoder \(\mathcal{D}\) with our continuous video decoder \(\mathcal{D}_s\) in (b) fails to stabilize the decoding process and even degrades the temporal metrics. This directly validates that continuous coordinate rendering is sensitive to latent stochasticity, making explicit feature-level alignment essential. To resolve this, we introduce the TGFR module. Integrating flow-based coarse alignment (c) and DCN-based sub-pixel refinement (d) strictly aligns the temporal priors and steadily improves both temporal consistency and reconstruction fidelity. With these aligned priors, re-integrating our continuous video decoder in (e) not only preserves but further enhances temporal stability. However, this transition inherently compromises fine-grained details, as evidenced by the simultaneous degradation in perceptual metrics. Remarkably, the integration of our SAFR module (Ours) effectively recovers these high-frequency components, yielding the best overall perceptual quality and temporal consistency.

Long-term Stability and Gate Design. Recurrently propagating ControlNet residual features can accumulate errors over long sequences. Table 4 isolates the effects of the learned gate and its sparsity regularization by comparing clip-wise and single-pass inference on the same \(100\)-frame sequences. When processed as five consecutive \(20\)-frame clips, both ablated variants complete inference without collapse, yet underperform the full TGFR. In a single \(100\)-frame pass, however, replacing gated fusion with concatenation after the same flow-guided DCN alignment (w/o Gate (Concat.)) leads to structural collapse, indicating that motion alignment alone cannot prevent unreliable recurrent information from accumulating over time.

Retaining the learned gate but removing its sparsity penalty (Gate w/o Sparsity, \(\lambda_{\text{gate}}=0\)) is likewise unstable, degenerating after roughly \(50\) frames as visualized in Fig. 5. By discouraging unnecessary gate activations and promoting selective temporal propagation, the sparsity term enables the full TGFR to remain stable and achieve the best performance across all metrics throughout the single \(100\)-frame pass.

Visualization of Scale-Aware Refinement. To deeply understand how our continuous video decoder \(\mathcal{D}_s\) adapts to arbitrary resolutions, we visualize the scale-aware feature representations \(u_{\text{ref}}^i\) before the coordinate-based rendering. Fig. 6 compares the feature activation heatmaps alongside the target high-frequency residuals (GT - Bilinear) for both \(2\times\) and \(4\times\) upscaling factors. Without our scale-aware modules, the network produces a static, scale-agnostic feature map that fails to accommodate the varying high-frequency demands of different target resolutions. This rigidity leads to over-smoothing at large scales and aliasing at small scales. In contrast, our SAFR module dynamically modulates the spectral energy based on the target scale \(s\). At the \(2\times\) scale, the feature activation is tightly concentrated on structural edges to prevent over-enhancement. Conversely, at the \(4\times\) scale, SAFR explicitly amplifies the high-frequency activations to synthesize the missing complex textures. This scale-adaptive behavior correlates with the target residual distributions, confirming that our scale-aware refinement successfully bridges generative priors with arbitrary-scale continuous decoding.

5 Conclusion↩︎

We present AVSR-Diff, a novel decoupled framework for DM-based arbitrary-scale VSR. By separating scale-agnostic latent denoising from continuous arbitrary-scale decoding, AVSR-Diff successfully bridges the gap between reconstruction fidelity and perceptual quality without incurring heavy resolution-specific sampling costs, making it highly memory-efficient. Within this framework, the Temporally-Gated Feature Recurrence (TGFR) module robustly suppresses diffusion-inherent flickering, while the Scale-Aware Fourier Refinement (SAFR) integrated into the continuous video decoder \(\mathcal{D}_s\) synthesizes fine-grained high-frequency details across arbitrary resolutions. Furthermore, the introduction of an explicit gate sparsity penalty effectively prevents progressive error accumulation, ensuring exceptional stability in long-term sequence processing. Extensive evaluations on standard benchmarks demonstrate that AVSR-Diff not only outperforms existing continuous models but also surpasses recent fixed-scale generative methods on their native \(4\times\) resolution, establishing a new optimal perception-distortion trade-off in the VSR domain.

Acknowledgements.↩︎

This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korean Government [Ministry of Science and ICT (Information and Communications Technology)] (Project Number: RS-2022-00144444, Project Title: Deep Learning Based Visual Representational Learning and Rendering of Static and Dynamic Scenes).

In this Supplementary Material, we provide additional details and results to support the main paper. First, in Sec. 6, we provide the additional implementation details, including the loss weights and hardware environment. Second, in Sec. 7, we elaborate on our bounding strategy for out-of-distribution scales. Third, in Sec. 8, we present the complete algorithmic representation of our proposed decoupled inference pipeline. Furthermore, we report additional quantitative results in Sec. 9 and additional qualitative comparisons in Sec. 10. Finally, we analyze the scale-wise computational efficiency of AVSR-Diff in Sec. 11, and discuss its limitations and future directions in Sec. 12.

6 Additional Implementation Details↩︎

The training of our AVSR-Diff is conducted on two NVIDIA RTX A6000 GPUs. During the training of the ControlNet \(\mathcal{C}_\phi\), the weight for the explicit gate sparsity penalty (Eq. 3 of the main paper) is set to \(\lambda_{\text{gate}} = 0.01\). For the training of the continuous video decoder \(\mathcal{D}_s\), we empirically set the perceptual loss weight (Eq. 4 of the main paper) to \(\lambda_{\text{percep}} = 1.0\) across all phases, and introduce the adversarial loss (Eq. 4 of the main paper) with a weight of \(\lambda_{\text{adv}} = 0.05\) specifically during the second fine-tuning phase. Furthermore, during the continuous coordinate-based rendering phase, to maintain a constant memory footprint regardless of the target scale \(s\), we evaluate the queried spatial coordinates in chunks (, 50,000 points per batch). This memory-efficient implementation explicitly bounds the peak GPU memory, ensuring it remains invariant to the output resolution, as demonstrated in Fig. 1 (b) of the main paper.

7 Inference Strategy for Out-of-Distribution Scales↩︎

While AVSR-Diff is trained on continuous scale factors up to \(4\times\), we evaluate its robustness on large out-of-distribution (OOD) scaling factors (, \(8\times\)). Although our continuous implicit neural representation (INR) natively supports arbitrary resolutions, directly feeding an OOD scale factor into the scale-aware modules (Scale-Aware Modulation and SAFR) causes spatial artifacts due to the extrapolation limits of absolute positional encodings (see Fig. 7).

To mitigate this, we adopt a simple bounding strategy during large-scale inference. The scale condition injected into the scale-aware modules is explicitly clipped to \(s=4\) (the maximum training scale), while the subsequent INR still queries the exact continuous coordinates for the target resolution. As shown in Table 5 and Fig. 7, this clipping strategy successfully suppresses OOD artifacts while preserving the fine-grained generative priors learned at the \(4\times\) boundary, outperforming both the unclipped setting and the baseline without scale-aware conditioning.

8 Complete Inference Pipeline↩︎

To provide a comprehensive and transparent overview of our proposed method, we detail the complete inference pipeline of AVSR-Diff. To explicitly demonstrate how the framework is decoupled into two distinct stages, we present the pseudo-code in two separate algorithms. Specifically, Algorithm 9 outlines the frame-wise bidirectional sampling process for scale-agnostic latent denoising. Subsequently, Algorithm 10 details the sequence-level continuous arbitrary-scale decoding.

9 Additional Quantitative Results↩︎

Due to space constraints, Table 2 of the main paper reports only a subset of metrics (LPIPS, DISTS, PSNR, and tOF) at \(2\times\), \(3.25\times\), and \(8\times\). Here, Table ¿tbl:tab:supp95master95table? extends the evaluation to additional scales (\(2.5\times\), \(3.5\times\), and \(6\times\)) and reports all six metrics, including SSIM and tLPIPS, at each reported scale.

As shown in Table ¿tbl:tab:supp95master95table? and Fig. 8, AVSR-Diff performs consistently well across diverse upscaling scenarios. At \(2\times\), it achieves the best perceptual quality (LPIPS and DISTS) and temporal stability (tLPIPS and tOF) among the evaluated methods, while remaining competitive with the strongest regression-based baselines in pixel-level fidelity (PSNR and SSIM). At the remaining arbitrary and out-of-distribution scales, AVSR-Diff consistently leads in perceptual quality and temporal stability. These results support the effectiveness of our decoupled framework in maintaining strong perceptual quality and temporal consistency across a wide range of target scales, including scales beyond the training range.

10 Additional Qualitative Results↩︎

To further validate our proposed AVSR-Diff, we provide extensive visual comparisons at both arbitrary (\(3.25\times\)) and large (\(8\times\)) scaling factors, as shown in Fig. 11. Consistent with the main paper, regression-based methods inherently produce overly smoothed outputs and struggle to recover complex textures. Furthermore, while recent DM-based models actively synthesize fine-grained details, they frequently fail to preserve the structural fidelity of the original LR inputs, leading to incorrect patterns or severe structural distortions, especially at large scaling factors. In contrast, our AVSR-Diff successfully achieves an optimal perceptual-distortion trade-off. By effectively balancing robust reconstruction fidelity with generative priors, our method synthesizes sharp, photo-realistic textures that remain faithful to the original content without introducing unnatural artifacts, even at the large \(8\times\) scale.

11 Scale-wise Efficiency Analysis↩︎

Table 6 compares AVSR-Diff with VEnhancer [24], a directly comparable arbitrary-scale video diffusion baseline, on the same 10-frame \(180\times320\) LR sequence. AVSR-Diff performs denoising in a fixed-resolution LR latent space, maintaining a peak memory of \(8.7\) GB and a runtime of approximately \(42\) s/frame across \(2\times\) to \(8\times\). In contrast, VEnhancer performs denoising at the target resolution, causing its peak memory and runtime to increase substantially with the target scale, from \(40.3\) GB to \(72.4\) GB and from \(24.88\) to \(106.52\) s/frame. For AVSR-Diff, the chunked INR querying only batches independent output-coordinate queries in the final rendering stage and does not partition the diffusion denoising process. Although VEnhancer requires fewer absolute FLOPs and is faster at \(2\times\) and \(4\times\), its cost grows rapidly with the target scale: from \(2\times\) to \(8\times\), the total FLOPs of AVSR-Diff increase by only \(1.35\%\), whereas those of VEnhancer increase by \(13.5\times\). Consequently, AVSR-Diff becomes approximately \(2.5\times\) faster than VEnhancer at \(8\times\), while requiring substantially less peak memory. Overall, these results demonstrate that the efficiency advantage of AVSR-Diff is scale-wise rather than absolute: its memory footprint and computational cost remain nearly invariant as the requested output resolution increases.

12 Limitations and Future Work↩︎

Absolute inference cost. Despite the favorable scale-wise efficiency demonstrated in Sec. 11, AVSR-Diff remains computationally expensive in absolute terms due to its iterative diffusion sampling. Unlike INR-based models that synthesize outputs in a single forward pass, our method relies on a diffusion-based generative prior involving 50 sampling steps, as detailed in Sec. 4.1 of the main paper. For instance, upsampling a \(180\times320\) input frame takes approximately 42 seconds. Accelerating this sampling process is a valuable direction for future research; we believe that integrating advanced sampling strategies [48], [49] or recent diffusion distillation techniques [50]–[52] could substantially reduce the inference time without compromising the structural fidelity and temporal coherence achieved by our model.

Fixed-scale generative prior. A further limitation stems from our reliance on the pretrained SD\(\times\)​4 Upscaler [26]: since the generative prior is bounded at a fixed \(4\times\) scale, at larger target scales (, \(8\times\)) the continuous decoder extrapolates in feature space rather than drawing on additional generative capacity. We mitigate this via the scale-clipping strategy (Sec. 7), but exploring generative backbones trained across a range of native resolutions remains a promising direction for future work.

References↩︎