1 Introduction↩︎

Video has become the dominant form of Internet traffic, continuously driving the demand for more efficient video compression. Traditional video codecs, represented by H.264/AVC, H.265/HEVC, and H.266/VVC, achieve remarkable compression performance through a highly adaptive hybrid coding framework. Instead of relying on a fixed coding strategy, the encoder dynamically selects among multiple coding modes according to the content characteristics and rate–distortion optimization. For example, it adaptively chooses motion prediction modes to accommodate diverse motion patterns, adjusts quantization parameters to allocate bits across spatial regions according to both content characteristics and user requirements, and switches between fidelity-oriented and perception-oriented coding configurations under different application scenarios. Such content- and user-adaptive decision making forms the foundation of modern video compression, enabling traditional codecs to flexibly balance coding efficiency, visual quality, and deployment requirements across diverse scenarios.

In recent years, the pursuit of higher compression performance has driven the emergence of neural video coding (NVC) [1]–[22]. NVC replaces key components of the hybrid video coding framework—motion estimation, motion compression, motion compensation, residual coding, and entropy modeling—with neural networks, and trains the entire model end-to-end under a rate-distortion loss. Through architectural innovations such as scale-space flow [4], feature-space motion estimation and compensation [3], temporal context mining [17], and quadtree-based entropy models [20], neural video codecs have surpassed the compression performance of traditional standards. However, this performance gain has come at the cost of extreme computational complexity—many NVC models exceed one million multiply-accumulate operations (MACs) per pixel, making them impractical for real-world deployment. To further bridge the gap between compression efficiency and practical deployment, the recently proposed real-time neural video codec DCVC-RT [13] removes explicit motion estimation, motion compression, and motion compensation modules, replacing them with lightweight implicit temporal modeling. This simplified architecture significantly reduces computational complexity while preserving competitive compression performance, making real-time neural video coding practically feasible.

Despite the remarkable progress in real-time neural video coding, pursuing efficiency comes at the cost of adaptivity. Unlike traditional codecs, whose coding decisions are dynamically optimized for every frame according to content characteristics and rate–distortion optimization, a real-time neural codec performs almost all coding decisions implicitly through a fixed neural network learned offline. Once training is completed, its coding strategies are fixed and cannot adapt to the characteristics of the current video and user requirements. As a consequence, several adaptive capabilities that have long been fundamental to video compression are inherently lost.

First, temporal prediction can no longer adapt to diverse motion patterns and bitrate constraints. Traditional codecs provide rich prediction diversity by evaluating multiple motion hypotheses and selecting the one that minimizes the rate–distortion cost. Earlier neural video codecs retain a similar capability through explicit motion estimation, motion compression, and motion compensation, where the prediction quality can be adjusted by allocating different numbers of bits to motion information. In contrast, recent real-time neural codecs replace explicit motion modeling with an implicit prediction network. Such a fixed mapping generates temporal context in exactly the same manner regardless of motion complexity or bitrate, making it difficult to accommodate diverse motion patterns or dynamically balance prediction accuracy under different rate constraints.

Second, spatial bit allocation cannot adapt to user requirements. Although existing neural video codecs learn spatially adaptive bit allocation through end-to-end optimization, the learned allocation strategy becomes fixed once training is completed. Unlike traditional codecs, where users can freely adjust quantization parameters for different coding units to satisfy application-specific requirements such as region-of-interest (ROI) coding [23], neural codecs provide almost no flexibility for user-controlled spatial bit allocation at deployment. Existing ROI-oriented neural codecs [24]–[26] attempt to recover this capability by introducing ROI masks or ROI-aware supervision during training. However, these methods generally require ROI annotations or ROI-aware supervision, complicate the training procedure, and may compromise reconstruction quality outside the selected regions. Consequently, current neural codecs still lack a general mechanism for user-driven spatial rate control without retraining.

Third, the codec cannot adapt its quality preference to varying application requirements. Existing neural video codecs are typically optimized for a single quality preference throughout training, causing each model to specialize in either signal fidelity or perceptual quality. Fidelity-oriented codecs achieve excellent objective reconstruction quality but tend to produce overly smooth textures at low bitrates, whereas perceptual codecs generate visually realistic details at the expense of signal fidelity, with fidelity remaining substantially low even at high bitrates. As application requirements vary continuously in practice, different quality preferences often coexist within the same deployment environment. However, satisfying these different objectives usually requires maintaining multiple independently trained codecs, which substantially increases storage, maintenance, and deployment costs. A practical neural video codec should therefore support flexible quality adaptation within a single model.

Motivated by the above observations, we propose URVC, a unified real-time neural video codec that achieves temporal, spatial, and perceptual adaptivity within a single lightweight framework. First, we propose a rate-aware adaptive temporal prediction method that generates diverse prediction candidates through a multi-candidate architecture and couples candidate selection directly to rate‑distortion optimization. This enables temporal prediction to adapt jointly to motion complexity and bitrate constraints without explicit motion estimation. Second, we propose a decomposition-based spatial rate control method that achieves finer-grained spatial bit allocation through feature decomposition and separate quantization, and allows users to perform direct spatial rate control at test time according to their requirements. This enables zero-shot ROI coding without retraining or additional supervision. Third, we propose a perceptual switching strategy that allows a neural video codec to switch between signal-fidelity and perceptual-quality modes according to different quality preferences, eliminating the need for multiple separately trained models.

The main contributions are summarized as follows:

• We revisit neural video coding from the perspective of adaptive coding and identify a fundamental limitation of existing neural codecs: although they achieve remarkable compression performance, their coding strategies remain largely fixed at deployment, limiting adaptivity in temporal prediction, spatial rate allocation, and quality preference.

• We propose URVC, a unified real-time neural video codec that simultaneously achieves temporal, spatial, and perceptual adaptivity, transforming a fixed-strategy codec into a deployment-adaptive coding framework.

• We develop a rate-aware adaptive temporal prediction method, a decomposition-based spatial rate control method, and a perceptual switching method, which collectively enable adaptive temporal prediction, user-controllable spatial rate allocation, and allow a neural codec to serve both signal fidelity and perceptual quality.

• Extensive experimental results show that our codec surpasses the state-of-the-art real-time codec DCVC‑RT in terms of YUV420 PSNR under intra period \(-1\), delivers substantially better perceptual quality, and achieves superior compression performance on video motion control, an emerging ROI coding task, without compromising the video generation quality.

The remainder of this paper is organized as follows. Section 2 reviews related work. Section 3 describes our proposed methods. Section 4 presents experimental results. Section 5 concludes the paper.

2 Related Work↩︎

2.1 Temporal Prediction for Neural Video Coding↩︎

Temporal prediction, which removes inter‑frame redundancy by leveraging the information of previously decoded frames, is a cornerstone of video compression. In traditional codecs, this is achieved through explicit motion estimation and compensation that can generate multiple prediction candidates and select the optimal one under rate‑distortion constraints. Early neural video codecs inherit this philosophy but replace hand‑crafted modules with learnable networks. The pioneering DVC [1] introduces an end‑to‑end framework where an optical flow network estimates pixel‑wise motion vectors, which are then compressed via an auto‑encoder and used for bilinear warping‑based motion compensation. This motion‑estimation‑compression‑compensation pipeline has become the dominant paradigm for temporal prediction in NVCs.

Since DVC, many works have advanced this paradigm by improving motion representation and alignment. Scale‑space flow [4] augments the conventional two‑channel optical flow with a per‑pixel scale field, allowing the warping operation to fall back to a blurred prediction when motion is unreliable. FVC [3] moves all operations—motion estimation, compression, and compensation—into a feature space and uses deformable convolutions for more flexible alignment. DCVC‑TCM [17] propagates decoded features instead of decoded frames and learns multi‑scale temporal contexts. Despite their success in rate‑distortion performance, these codecs incur enormous complexity—often exceeding thousands of MACs per pixel—due to the many layers required for motion estimation, compression, and compensation. The high computational cost makes real‑time deployment impractical.

To break this complexity barrier, DCVC‑RT [13] discards the entire explicit motion branch, including motion estimation, motion compression, and warping-based motion compensation. Instead, temporal contexts are generated by a lightweight implicit feature extractor composed solely of depth‑wise convolutions, which operates on a cached reference feature. This design reduces the number of modules drastically and achieves real‑time coding while maintaining competitive rate‑distortion performance. However, the implicit temporal modeling of DCVC‑RT relies on a fixed prediction rule that lacks the adaptivity in motion complexity and the bitrate constraints.

2.2 Spatial Rate Control for Neural Video Coding↩︎

In traditional block‑based video codecs, spatial rate control can be achieved by varying the quantization parameter (QP) across coding units, enabling fine‑grained bit allocation that respects region‑of‑interest priorities or content complexity. In existing neural video codecs, such as DCVC‑DC [20], DCVC‑FM [22], and DCVC‑RT [13], their entropy models learn a spatially varying quantization step, which modulates the latent representation before rounding. This enables some degree of content‑adaptive bit allocation—the quantization step can vary across spatial locations based on local content complexity. However, the quantization step operates at the resolution of the latent representation, which is far below the granularity needed for precise spatial control. Moreover, once trained, this allocation strategy is fixed. At test time, users cannot freely adjust the quantization strength across spatial regions, making it impossible to control the bit allocation according to user requirements.

Some works have explored ROI‑based video compression in this context. Fathima et al. [24] introduced two ROI‑based neural codecs based on scale‑space flow, where an ROI mask is concatenated to the input and the distortion loss is weighted per region. Wu et al. [25] applied a ROI loss to DVC, using a detection network to generate masks and a resolution‑adaptive inference strategy. Liu et al. [26] proposed ROI‑aware dynamic quantization for NVC, where a bit‑allocator assigns different bit‑widths to ROI and non‑ROI based on motion and texture complexity. However, these methods require the ROI mask to be provided during training. Most of them achieve high ROI fidelity at the cost of severely degraded non‑ROI reconstruction. When ROI coding is not needed, the performance of these codecs drops significantly below that of a model trained without ROI loss. None of existing codecs support zero‑shot ROI coding at test time.

2.3 Perceptual Optimization for Neural Video Coding↩︎

Existing perceptual neural video codecs can be broadly classified into four categories: generative adversarial network (GAN)-based, vector quantization (VQ)-based, diffusion-based, and visual auto‑regressive (VAR)-based.

GAN‑based codecs pioneered perceptual neural video coding. PLVC [27] conditions a recurrent discriminator on motion and latent representations to improve spatio‑temporal realism. HVFVC [28] introduces confidence‑based feature reconstruction and a periodic compensation loss to mitigate checkerboard artifacts. VQ‑based codecs leverage vector‑quantized auto‑encoders for a semantically rich latent space. GLC‑Video [29] performs transform coding in the latent space of a generative VQ‑VAE, achieving ultra‑low bitrate compression with high perceptual fidelity. Diffusion‑based codecs exploit pre‑trained diffusion models as generative priors. DiffVC [30] integrates a multi‑step diffusion process into conditional coding. DiffVC‑RT [31] reduces inference to a single diffusion step and prunes the backbone for real‑time decoding. GNVC‑VD [32] uses a video diffusion transformer for sequence‑level latent refinement, significantly improving temporal coherence. VAR‑based codecs adopt a next‑scale prediction paradigm. ProGVC [33] unifies progressive scalability and detail synthesis within a single autoregressive pipeline, enabling bitrate‑adaptive perceptual compression.

Despite their impressive perceptual gains, these methods share two fundamental limitations. First, they heavily sacrifice signal fidelity even at high bitrates. Second, their computational complexity remains prohibitively high for practical deployment. For example, the latest diffusion‑based DiffVC‑RT consumes 2462K MACs per pixel. This complexity, combined with the need for two separate models to cover both fidelity and perceptual scenarios, severely limits their practicality in resource‑constrained environments.

3 Method↩︎

3.1 Overview↩︎

We first present an overview of our real‑time neural video codec—URVC, which integrates our proposed adaptive temporal prediction method, spatial rate control method, and perceptual switching method, as illustrated in Fig. 1.

3.1.1 Rate-Aware Adaptive Temporal Prediction↩︎

To overcome the deterministic nature of implicit temporal prediction, we design a rate-aware adaptive temporal prediction module. It takes the reference feature \(\hat{F}_{t-1}\) and produces a temporal context \(C_t\) and an auxiliary temporal prior \(C^p_t\). The module structurally introduces prediction diversity through a multi-branch architecture that generates multiple prediction candidates. The final context \(C_t\) is formed by fusing these candidates, with the learnable quantization vectors serving as the fusion weights. This couples candidate selection to both the rate constraint and motion complexity, restoring the temporal adaptivity lost in deterministic implicit prediction. Details are provided in Section 3.2.

3.1.2 Decomposition-based Contextual Encoder and Decoder↩︎

The contextual encoder transforms the input frame \(x_t\) into a compact latent representation \(y_t\). A pixel‑unshuffle layer first reduces the spatial resolution by a factor of \(8\), and the resulting feature is concatenated with the temporal context \(C_t\) and processed by depth‑wise convolutional layers. Before the final down‑sampling, the intermediate feature undergoes our proposed spatial decomposition quantization, which enables finer‑grained bit allocation according to content complexity. The contextual decoder mirrors the encoder, up‑sampling the quantized latent \(\hat{y}_t\) and applying the same spatial decomposition quantization to its intermediate feature. At test time, the down‑up sampling-based decomposition in the encoder can be replaced by an arbitrary spatial mask, giving users direct spatial rate control. The decoded feature \(\hat{F}_t\) is stored in the decoded buffer as the reference for the next frame and forwarded to the frame generator. Details are given in Section 3.3.

3.1.3 Frame Generator with Perceptual Switching↩︎

The frame generator refines the decoded feature \(\hat{F}_t\) by a series of depth‑wise convolutional layers and performs a pixel‑shuffle operation with an up‑sampling factor of \(8\) to produce the reconstructed frame \(\hat{x}_t\). Two separate instances of the frame generator are maintained: \(\theta_r^{\text{Fid}}\) for the fidelity mode and \(\theta_r^{\text{Per}}\) for the perceptual mode. At inference time, users can switch between the two modes by selecting the corresponding instance. Further details are provided in Section 3.4.

3.1.4 Entropy Model↩︎

We use a hyperprior entropy model to provide hyper prior for \(y_t\). In parallel, the temporal prior \(C^p_t\) is fused with the hyper prior to predict the Gaussian distribution parameters for \(y_t\). The quantized side information \(\hat{z}_t\) is compressed using a separate factorized entropy model [13], which is part of the module bank and indexed by the current QP to match the distribution of \(\hat{z}_t\) at different bitrates.

3.1.5 Module Bank with Perceptual Switching↩︎

The learnable quantization vectors that modulate the encoder, decoder, temporal prediction module, and frame generator, together with the factorized entropy model for compressing the side information \(\hat{z}_t\), form a module bank in which all components are indexed by QP. To enable perceptual switching, we maintain two distinct module banks: \(\mathcal{B}^{\text{Fid}}\) optimized for signal fidelity, and \(\mathcal{B}^{\text{Per}}\) trained for perceptual quality. At inference time, users can switch between a fidelity mode and a perceptual mode by simply loading the corresponding module bank. More details can be found in Section 3.4.

3.2 Rate-Aware Adaptive Temporal Prediction↩︎

In existing real‑time neural video codecs, temporal context prediction is formulated as a deterministic mapping: \[C_t = f_\theta(\hat{F}_{t-1}),\] where \(f_\theta\) is a neural network with fixed parameters \(\theta\) and \(\hat{F}_{t-1}\) denotes the reference feature from the previously decoded frame. Let \(F_t\) be the feature representation of the current frame \(x_t\) (e.g., obtained after the pixel‑unshuffle and initial convolutional layers of the encoder). From a probabilistic perspective, this implicit temporal modeling assumes that the conditional distribution of \(F_t\) given \(\hat{F}_{t-1}\) can be approximated by a unimodal Gaussian with constant variance: \[p(F_t \mid \hat{F}_{t-1}) \sim \mathcal{N}\bigl(f_\theta(\hat{F}_{t-1}),\; \sigma^2).\]

Real‑world videos, however, exhibit highly non‑stationary temporal statistics due to fast translation, non‑rigid deformation, and occlusion. When the actual motion deviates from the training distribution, a statistical mismatch occurs, making the empirical prediction variance far exceed the model’s assumption: \[\mathbb{E}\bigl[\|F_t - f_\theta(\hat{F}_{t-1})\|^2\bigr] \gg \sigma^2.\] Because the fixed mapping can adapt neither to motion complexity nor to bitrate constraints, it may provide adequate prediction under simple motion, but can lead to substantially increased residuals under complex motion or at low bitrates, rendering temporal prediction potentially fragile.

To overcome the statistical mismatch caused by the deterministic unimodal assumption, we draw on the bias‑diversity trade‑off from ensemble learning. According to the Ambiguity Decomposition theorem [34], the mean squared error (MSE) of an ensemble \(\bar{f} = \sum_{k=0}^{K-1} w_k \phi_k\) (with \(\sum_k w_k = 1\)) can be rigorously decomposed as \[\text{MSE}(\bar{f}) = \sum_{k=0}^{K-1} w_k \text{MSE}(\phi_k) - \sum_{k=0}^{K-1} w_k \|\phi_k - \bar{f}\|^2, \label{eq:4}\tag{1}\] where the first term represents the average individual error and the second term quantifies the ensemble diversity (or ambiguity). It indicates that to reduce the total prediction error, it is beneficial to construct a set of heterogeneous base estimators that possess high diversity, thereby increasing the second term. Homogeneous branches would substantially reduce the diversity term, limiting the ensemble’s ability to capture multimodal dependencies.

Guided by this principle, as shown in Fig. 2, we design a multi‑branch context prediction architecture that produces a candidate set \(\mathcal{C} = \{C_t^0, C_t^1, C_t^2\}\) through structurally diverse processing:

• Full‑resolution branch \(C_t^0\): \[C_t^0 = \phi_0(\hat{F}_{t-1}; \theta_0).\] This branch operates at the original feature resolution, preserving the fine‑grained spatial details present in the reference feature \(\hat{F}_{t-1}\).

• Low‑resolution parallel branches \(C_t^1, C_t^2\): \[\begin{align} &C_t^1 = \text{Up}(\phi_1(\text{Down}(\hat{F}_{t-1}); \theta_1)), \\ &C_t^2 = \text{Up}(\phi_2(\text{Down}(\hat{F}_{t-1}); \theta_2)), \end{align}\] where \(\text{Down}(\cdot)\) denotes a bilinear down‑sampling operation and \(\text{Up}(\cdot)\) denotes a bilinear up‑sampling operation. The down‑sampling operation reduces the spatial resolution, which enlarges the effective receptive field of subsequent convolutions and discards high‑frequency information. The two branches receive the same down‑sampled input but are initialized and trained with independent weights \(\theta_1\) and \(\theta_2\), allowing them to learn complementary features and produce distinct prediction candidates.

This heterogeneous design promotes a larger ensemble diversity term, providing a joint estimate against complex temporal misalignment. The structural diversity across resolutions and parallel streams enables the ensemble to capture a range of possible motion patterns.

To perform rate‑distortion optimized mode selection without discrete search, we let the learnable quantization vectors \(q_c^0, q_c^1, q_c^2 \in \mathbb{R}^{C \times 1 \times 1}\), which are part of the module bank and indexed by QP, serve as fusion weights. The final temporal context \(C_t\) is obtained as: \[C_t = f_{\text{fuse}}\Big( \big(q_c^0 \odot C_t^0\big) \;\|\; \big(q_c^1 \odot C_t^1\big) \;\|\; \big(q_c^2 \odot C_t^2\big) \Big),\] where \(\odot\) is channel‑wise multiplication, \(\|\) denotes concatenation along the channel dimension, and \(f_{\text{fuse}}\) is a subsequent depth‑wise convolutional fusion layer. These \(q_c^k\) function as differentiable proxy variables [13] that directly modulate the contribution of each branch in the feature domain, driven by the rate‑distortion optimization. The role of \(q_c^k\) can be understood by examining the gradient of the rate-distortion objective with respect to these quantization vectors. Since \(q_c^k\) multiplies the branch output \(C_t^k\) before the fusion network, the chain rule gives \[\frac{\partial \mathcal{L}}{\partial q_c^k} = \left( \lambda \frac{\partial d}{\partial (q_c^k \odot C_t^k)} + \frac{\partial R}{\partial (q_c^k \odot C_t^k)} \right) \odot C_t^k, \label{eq:grad95q}\tag{2}\] where \(\odot\) denotes element-wise multiplication. This gradient jointly combines a distortion-driven component and a rate-driven component, both of which are balanced by the Lagrange multiplier \(\lambda\). When \(\lambda\) is small and the rate penalty dominates, the optimization tends to suppress high‑frequency channels in the full‑resolution branch \(C_t^0\), allowing the coarser branches \(C_t^1\) and \(C_t^2\) to contribute more to the fused context \(C_t\). When \(\lambda\) is large and the distortion penalty dominates, the suppression is relaxed, and the full‑resolution branch can contribute its fine‑grained details more fully. This rate‑dependent adaptive selection emerges from end‑to‑end training, thereby restoring the adaptivity of temporal prediction to motion complexity and bitrate constraints that was lost in deterministic implicit prediction.

3.3 Decomposition-based Spatial Rate Control↩︎

Existing neural video codecs employ a two‑level quantization mechanism [13], [20], [22]. The first level operates in the encoder and decoder: learnable one‑dimensional quantization vectors \(q^E, q^D \in \mathbb{R}^{C \times 1 \times 1}\) uniformly modulate the encoder intermediate feature \(F_t^E\) and the decoder intermediate feature \(F_t^D\) across all spatial locations: \[\tilde{F}_t^E = F_t^E \odot q^E, \quad \tilde{F}_t^D = F_t^D \odot q^D. \label{eq:uniform95quant}\tag{3}\] The second level is realized by the entropy model: in addition to predicting the mean \(\mu_t\) and scale \(\sigma_t\), the entropy model learns a quantization step \(Q_t\) of the same size as the latent representation \(y_t\). Before rounding (\(\left\lfloor \right\rceil\)) at the encoder, \(y_t\) is divided by \(Q_t\) element‑wise; at the decoder, \(\hat{y}_t\) is multiplied back by \(Q_t\): \[y_t^{\text{quant}} = \left\lfloor \frac{y_t}{Q_t} \right\rceil, \quad \hat{y}_t = y_t^{\text{quant}} \odot Q_t. \label{eq:entropy95quant}\tag{4}\]

Since the first‑level quantization is spatially uniform, and although the second‑level quantization enables some degree of content‑adaptive spatial bit allocation, it operates at the low spatial resolution of \(y_t\) (16\(\times\) smaller than the original frame), which limits its allocation precision. More importantly, once trained, users cannot freely control the quantization strength across spatial regions at test time.

To achieve direct spatial rate control at test time, we propose a spatial decomposition‑based quantization mechanism. For the encoder intermediate feature \(F_t^E\) and the decoder intermediate feature \(F_t^D\), we obtain their smooth structural component by bilinearly down‑sampling the features to half resolution and then up‑sampling it back to the original resolution: \[F_t^{E/D,s} = \text{Up}\bigl(\text{Down}(F_t^{E/D})\bigr). \label{eq:spatial95dec}\tag{5}\] The detail component captures the texture and edge information filtered out by down‑sampling, and is computed as the residual: \[F_t^{E/D,d} = F_t^{E/D} - F_t^{E/D,s}. \label{eq:spatial95res}\tag{6}\] Each component is then modulated by its own learnable quantization vector \(q_{s}^{E/D}, q_{d}^{E/D} \in \mathbb{R}^{C \times 1 \times 1}\), and the two are summed back together: \[\tilde{F}_t^{E/D} = F_t^{E/D,s} \odot q_{s}^{E/D} + F_t^{E/D,d} \odot q_{d}^{E/D}. \label{eq:spatial95quant}\tag{7}\] The learnable quantization vectors \(q_{\mathrm{s}}^{E/D}\) and \(q_{\mathrm{d}}^{E/D}\) are part of the module bank and are indexed by the current QP. Trained end‑to‑end under the rate‑distortion loss, they allow the network to allocate bits unevenly across spatial structures. The structural component can be heavily quantized to save rate, while the detail component can retain finer precision.

A crucial property of this design is that the down‑up sampling-based decomposition in the encoder can be replaced at test time by an arbitrary spatial mask-based decomposition. Concretely, let \(M_t \in \{0,1\}^{1 \times H \times W}\) be a user‑defined binary ROI mask, down‑sampled to the resolution of \(F_t^E\). The mask‑based decomposition can be performed as: \[F_t^{bg} = F_t^E \odot M_t, \qquad F_t^{fg} = F_t^E \odot (1 - M_t). \label{eq:roi95split}\tag{8}\] Then we apply the structural quantization vector \(q_{s}^E\) to the background and the detail quantization vector \(q_{d}^E\) to the foreground: \[\tilde{F}_t^E = F_t^{bg} \odot q_{s}^E + F_t^{fg} \odot q_{d}^E. \label{eq:roi95quant}\tag{9}\] Since \(q_{d}^E\) has learned to preserve fine‑grained information and \(q_{s}^E\) has learned to encode smooth content at lower fidelity during training, this substitution naturally grants higher reconstruction quality to the user‑specified foreground regions while compressing the background regions more efficiently. The decoder continues to use the original down‑up decomposition, so no modification is needed at the receiver side. The mask \(M_t\) itself is not transmitted, incurring zero bitrate overhead.

3.4 Perceptual Switching↩︎

The perception‑distortion trade‑off [35] shows that the fidelity‑optimal and perception‑optimal reconstructions occupy distinct points on the rate‑distortion‑perception surface, and therefore require different sets of model parameters. Following this, existing neural video codecs train two separate models. A model optimized for signal fidelity outputs the conditional expectation \(\mathbb{E}[X|Y]\), where \(X\) denotes the original signal and \(Y\) denotes the available observations at the decoder, minimizing pixel‑level error but producing smooth reconstructions. A model optimized for perceptual quality sacrifices pixel fidelity to produce visually plausible textures and achieve better perceptual quality. However, this two‑model solution substantially increases storage and maintenance costs, which is particularly burdensome for resource‑constrained applications.

A closer examination of the theory, however, reveals an unexploited degree of freedom. The perception‑distortion trade‑off constrains only the conditional distribution \(p_{\hat{X}|Y}\) that the codec realizes, not the network architecture that produces it. In other words, while the two modes must correspond to different conditional distributions, nothing in the theory requires them to be realized by two entirely separate networks. This observation motivates a design in which the model backbone is shared between the two modes, and only a small set of mode‑specific parameters is switched. Building on this insight, we propose a perceptual switching method.

We first identify the quantization vectors as the primary mode‑specific parameters. After end‑to‑end training for signal fidelity, the backbone convolutional layers have learned to extract generic feature representations from the input frames. The learnable quantization vectors apply channel‑wise modulation to these features. Different quantization vectors do not alter how the backbone convolution kernels extract features, but only change the retained amplitude of each channel. Hence, the backbone convolution kernels can be shared, while the quantization vectors can be switched. This design can be further understood from the perspective of covariance. Consider a feature representation \(F \in \mathbb{R}^{C \times H \times W}\) before modulation, and let \(\tilde{F} = F \odot q\) be the feature after modulation by a quantization vector \(q \in \mathbb{R}^{C \times 1 \times 1}\). The channel covariance matrix \(\Sigma_F = \mathbb{E}[(F - \mu_F)(F - \mu_F)^\top]\) becomes: \[\Sigma_{\tilde{F}} = \operatorname{diag}(q) \; \Sigma_F \; \operatorname{diag}(q)^{\top}, \label{eq:cov95rescale95percep}\tag{10}\] where \(\operatorname{diag}(q)\) is a diagonal matrix with the entries of \(q\) on the diagonal, and \(\mu_F\) is the channel‑wise mean of \(F\). This identity shows that replacing the quantization vectors is equivalent to independently rescaling the diagonal entries of the feature covariance matrix. Two different sets of quantization vectors therefore produce different feature energy distributions, each corresponding to one of the two modes.

Replacing the quantization vectors leads to two further requirements. First, the factorized entropy model must be updated synchronously. When the distribution of the hyper‑prior latent \(\hat{z}_t\) shifts due to the changed quantization vectors, the original entropy model would produce inaccurate probability estimates. Hence, the two modes require separate factorized entropy models \(\phi_z^{\text{Fid}}\) and \(\phi_z^{\text{Per}}\). Second, the frame generator \(\theta_r\), which maps the decoded feature \(\hat{F}_t\) to the reconstructed frame \(\hat{x}_t\), also needs to be replaced. When the statistics of the decoded feature change under perceptual quantization vectors, the original frame generator may no longer produce outputs aligned with perceptual objectives. The two modes therefore also require separate frame generators \(\theta_r^{\text{Fid}}\) and \(\theta_r^{\text{Per}}\).

Building on the above analysis, as presented in Fig. 4, we maintain two module banks (\(\mathcal{B}^{\text{Fid}}\), \(\mathcal{B}^{\text{Per}}\)) and two frame generators (\(\theta_r^{\text{Fid}}, \theta_r^{\text{Per}}\)) for the two modes, both having the same structure. Each module bank is indexed by QP and contains a set of channel‑wise quantization vectors—the structural/detail quantization vectors \(q_s^{E/D}, q_d^{E/D}\), the temporal prior quantization vector \(q_p\), the context fusion quantization vectors \(q_c^0, q_c^1, q_c^2\), the reconstruction quantization vector \(q_r\), and a factorized entropy model \(\phi_z\).

Training proceeds in two stages. In the first stage, all convolutional layers in the contextual encoder, contextual decoder, and temporal prediction module, together with \(\mathcal{B}^{\text{Fid}}\) and the frame generator \(\theta_r^{\text{Fid}}\), are jointly trained under a rate‑fidelity loss. In the second stage, all convolutional weights are frozen. \(\mathcal{B}^{\text{Per}}\) and the frame generator \(\theta_r^{\text{Per}}\) are initialized from \(\mathcal{B}^{\text{Fid}}\) and \(\theta_r^{\text{Fid}}\), respectively, and fine‑tuned using a perceptual loss. Because the backbone is trained once for signal fidelity and then frozen, the fidelity mode remains completely unaffected when the perceptual mode is added. At inference time, users can switch between two modes by changing the loaded module bank and frame generator parameters, incurring negligible additional storage and identical computational cost.

4 Experiments↩︎

4.1 Datasets↩︎

We train our model on the training split of the Vimeo-90k dataset [36]. Following [2], [13], we further process the original Vimeo video clips into longer sequences for cascaded training. For evaluation, we use three widely adopted benchmark datasets. The HEVC Common Test Sequences [37] (Classes B, C, D, and E) cover resolutions from 416\(\times\)​240 to 1920\(\times\)​1080, with content ranging from static conference scenes to fast‑paced sports. The UVG dataset [38] contains seven 1080p high‑frame‑rate sequences. The MCL‑JCV dataset [39] consists of thirty 1080p sequences with diverse content. All test sequences are in YUV420 format.

4.2 Test Details↩︎

For the traditional codec, we compare with the official reference software VTM‑17.0 of the H.266/VVC [40] standard under the encoder_lowdelay_vtm.cfg configuration. For neural codecs, we compare with DCVC‑DC [20], DCVC‑FM [22], PRAVC [2], and the state‑of‑the‑art real‑time codec DCVC‑RT [13]. We also include our reproduced version of DCVC‑RT, denoted as DCVC‑RT\(^*\), which serves as the direct baseline of our model. Due to limited GPU memory, we are unable to perform the cascaded training [17] with 64 frames at a resolution of 512\(\times\)​512 as done in the original DCVC‑RT. Consequently, DCVC‑RT\(^*\) achieves lower compression performance than the released model.

We conduct three sets of experiments to evaluate our codec from different perspectives. For signal fidelity, we assess rate‑distortion performance using YUV420 PSNR with VTM‑17.0 as the anchor. For perceptual quality, we evaluate DISTS [41] and LPIPS [42] with DCVC‑FM as the anchor. For spatial rate control, we test on two representative videos—a dancing sequence and a martial‑arts sequence—using a CLIP [43] semantic similarity and a motion similarity as evaluation metrics. All frames are encoded with intra period set to \(-1\) and reported in the YUV420 color space.

4.3 Training Details↩︎

To support variable‑rate coding within a single model, we randomly sample a QP from the range \([0,63]\) in each training iteration. To implement a hierarchical quality structure, we adopt the QP offset pattern \([0,8,0,4,0,4,0,4]\) for a group of 8 pictures and assign distortion weights of [\(1.2\), \(0.5\), \(0.9\), \(0.5\)] to the four corresponding hierarchical levels. Each QP serves as an index to select a corresponding set of components in the module bank. The Lagrangian multiplier \(\lambda\) is interpolated between \(1\) and \(768\) according to the sampled QP. The loss function differs between the fidelity mode and perceptual mode. For signal fidelity training, we use a scaled YUV mean squared error (MSE) loss with the Y, U, and V components weighted at a ratio of \(6{:}1{:}1\). For perceptual quality training, the distortion term is a weighted combination of YUV MSE, LPIPS, and a GAN loss [27]. The model is optimized with the AdamW optimizer [44] using a batch size of \(8\). All training is performed on \(8\) NVIDIA RTX 3090 GPUs.

4.4 Experimental Results for Signal Fidelity↩︎

Table ¿tbl:table:ip195yuv42095psnr? reports the BD-rate savings in YUV420 PSNR with VTM‑17.0 as the anchor. Our codec achieves an average BD-rate of \(-22.4\%\). When compared with codecs whose computational complexity exceeds millions of MACs per pixel, our codec substantially outperforms DCVC‑DC (\(14.5\%\)) and PRAVC (\(-12.3\%\)), and remains slightly below DCVC‑FM (\(-25.2\%\)) in overall compression efficiency. When compared with lightweight real‑time codecs, our codec delivers over \(10\%\) bitrate saving over our reproduced baseline DCVC‑RT\(^*\) (\(-11.5\%\)) under the same constrained training protocol. This substantial improvement directly validates the effectiveness of our proposed rate-aware adaptive temporal prediction and finer‑grained spatial bit allocation. More importantly, our codec even surpasses the officially released DCVC‑RT (\(-22.4\%\) vs. \(-18.8\%\)) on average. This improvement is consistent across most individual datasets, with notable gains on HEVC Class C (\(-25.4\%\) vs. \(-17.4\%\)) and HEVC Class D (\(-35.8\%\) vs. \(-25.7\%\)). Nevertheless, on high-resolution sequences such as 1080p videos, the rate-distortion curves in Fig. 5 show that our codec performs slightly below the released DCVC‑RT at high bitrates. This is mainly due to our limited GPU memory, which prevents us from using the same training protocol of the original DCVC‑RT. We can only train with smaller-resolution crops of \(256\times384\) (instead of \(512\times512\)), and although we also use \(64\)-frame sequences, we are forced to adopt a grouped cascaded training scheme [45] with detached computation graphs rather than the unrolled continuous cascading of the official implementation. These constraints slightly reduce the model’s capacity at very high bitrates. We believe that with access to GPUs with larger memory, our model would achieve further performance gains.

4.5 Experimental Results for Spatial Rate Control↩︎

We evaluate our spatial rate control method on a video motion control task, a representative ROI coding scenario. In this task, a user uploads a reference video and a character image to a cloud server, and a generative model animates the character to mimic the motion of the reference video. As exemplified by recent project such as Seedance [46], the generative model only needs to extract the character’s facial expressions and body movements from the reference video, while the background is largely ignored, as illustrated in Fig. 6. Consequently, the foreground (the person) requires high reconstruction quality, while the background can be heavily compressed without affecting the generated video. This makes the problem a natural fit for ROI coding.

We test on two representative reference videos, including a dancing video (Dancer) and a martial‑arts video (KungFu). Fig. 7 compares the reconstructed reference videos with and without our spatial rate control. As expected, the background regions—e.g., the fence in Dancer and the grass in KungFu—become noticeably blurred when spatial rate control is applied, because they are quantized more heavily. In contrast, the foreground remains sharp. We feed the compressed reference videos into the same video motion control model [46]. Visual inspection shows that the generated videos are virtually indistinguishable from each other.

For objective evaluation, we compute two similarity metrics between the video frames (denoted as \(\{G_i^o\}_{i=1}^{N}\)) generated from the original uncompressed reference video and those (denoted as \(\{G_i^c\}_{i=1}^{N}\)) generated from the compressed reference video. The first metric is CLIP semantic similarity. Let \(f_{\text{CLIP}}(\cdot)\) denote the CLIP image encoder (ViT-base-patch32) that maps an frame to a normalized feature vector. The semantic similarity for frame \(i\) is the cosine similarity between \(f_{\text{CLIP}}(G_i^o)\) and \(f_{\text{CLIP}}(G_i^c)\). The average over all frames is: \[\text{CLIP-Sim} = \frac{1}{N} \sum_{i=1}^{N} \frac{f_{\text{CLIP}}(G_i^o) \cdot f_{\text{CLIP}}(G_i^c)}{\|f_{\text{CLIP}}(G_i^o)\| \|f_{\text{CLIP}}(G_i^c)\|}.\] The second metric is motion similarity. Let \(\text{flow}(\cdot,\cdot)\) denote the dense optical flow field computed between two frames. For adjacent frames \(i\) and \(i+1\), define \(\mathbf{u}_i^{G^c} = \text{flow}(G_i^c, G_{i+1}^c)\) and \(\mathbf{u}_i^{G^o} = \text{flow}(G_i^o, G_{i+1}^o)\). The motion similarity is the average cosine similarity over all adjacent frame pairs: \[\text{Motion-Sim} = \frac{1}{N-1} \sum_{i=1}^{N-1} \frac{\mathbf{u}_i^{G^c} \cdot \mathbf{u}_i^{G^o}}{\|\mathbf{u}_i^{G^c}\| \|\mathbf{u}_i^{G^o}\|}.\] As shown in Fig. 7, both metrics remain nearly identical for the videos generated with and without spatial rate control, while our method achieves 20–30% bitrate savings on the reference video. For the KungFu sequence, the CLIP similarity is 0.987 vs., and the flow similarity is 0.777 vs., while the bitrate is reduced by 31.7%. Similar results are observed on the Dancer sequence. These results confirm that our spatial rate control can effectively allocate bits to semantically relevant regions without harming the downstream generative task.

4.6 Experimental Results for Perceptual Quality↩︎

We present the rate-perception curves in terms of DISTS and LPIPS in Fig. 8. Detailed BD‑rate saving values are reported in Tables ¿tbl:table:ip195combined95dists95lpips?, with DCVC‑FM as the anchor. The results show that our codec with the perceptual mode achieves an average BD‑rate of \(-57.3\%\) in DISTS and \(-48.7\%\) in LPIPS, substantially outperforming both DCVC‑FM and the released DCVC‑RT (\(-7.0\%\) DISTS, \(-4.9\%\) LPIPS). Fig. 9 provides visual comparisons on two sequences. On BasketballDrive_1920x1080_50, our perceptual mode operates at a lower bitrate of 0.019 bpp, yet achieves markedly better DISTS of 0.098 and LPIPS of 0.190, compared with 0.144 and 0.257 for DCVC‑FM at 0.023 bpp. The visual gain is evident on the player’s shorts, where the stripe patterns are noticeably sharper and more faithfully rendered. On BlowingBubbles_416x240_50, our codec delivers a DISTS of 0.139 and an LPIPS of 0.111 at 0.026 bpp, while DCVC‑FM obtains 0.148 and 0.120 at 0.031 bpp. The improvement is visible on the girl’s eyebrows, which appear clearer and more naturally textured. These results confirm that our proposed perceptual switching method yields substantial perceptual gains.

4.7 Complexity Analysis↩︎

Table 1 compares the encoding and decoding speed, MACs per pixel, and model parameters of our codec with those of several representative neural video codecs. All measurements are taken at 1080p resolution on a single NVIDIA RTX 3090 GPU. The results show that our codec encodes one 1080p frame in \(0.027\) seconds and decodes in \(0.028\) seconds, slightly higher than the real‑time baseline DCVC‑RT (\(0.020\) s and \(0.021\) s). The computational cost increases from \(195.23\)K to \(223.00\)K MACs per pixel, and the model size grows from \(19.93\)M to \(24.89\)M parameters. Despite these increases, our codec remains firmly in the real‑time regime. Compared with the non‑real‑time codecs DCVC‑DC, DCVC‑FM, and PRAVC, it is more than \(20\times\) faster while delivering competitive or superior compression performance. The modest additional complexity over DCVC‑RT is the cost paid for the temporal, spatial, and perceptual adaptivity that no existing real‑time neural codec provides.

4.8 Ablation Study↩︎

4.8.1 Effectiveness of Rate-Aware Adaptive Temporal Prediction↩︎

To verify the effectiveness of the rate-aware adaptive temporal prediction (RAATP) method, we conduct an ablation study on the HEVC dataset. As shown in Table 2, starting from the baseline model \(M_1\) that employs neither RAATP nor the decomposition-based spatial rate control (DSRC) method, equipping RAATP alone in model \(M_2\) yields a BD‑rate saving of \(6.9\%\) in YUV420 PSNR. Fig. 10 offers a qualitative explanation for this gain by visualizing the predicted temporal contexts and the corresponding prediction residuals with and without RAATP. Without RAATP, the codec produces relatively smooth predictions that fail to capture fine motion details, leaving larger residuals in complex or occluded areas. When RAATP is enabled, the predicted context aligns more accurately with the current frame and the residuals are reduced.

4.8.2 Effectiveness of Decomposition-based Spatial Rate Control↩︎

Table 2 also quantifies the contribution of the decomposition-based spatial rate control (DSRC) method. Adding DSRC on top of RAATP (from \(M_2\) to \(M_3\)) yields an additional BD‑rate saving of \(3.8\%\), confirming that spatial decomposition-based quantization further improves compression efficiency. Fig. 11 provides a visual interpretation of how DSRC achieves this gain. The first row shows the result of the down‑up sampling-based decomposition. The structural component appears smoother than the original frame, resembling a low‑pass filtered version that captures the main structures of the video, while the detail component retains the fine textures and edges. The second row compares the bit allocation maps with and without DSRC. Without DSRC, the codec distributes bits more uniformly across the textured and flat regions. When DSRC is enabled, the codec can adaptively allocate fewer bits to flat regions and more bits to texture-rich regions, leading to an efficient use of rate budgets.

A further benefit of this decomposition is that, at test time, the down‑up sampling operation in the encoder can be replaced by a user‑specified mask, enabling zero‑shot ROI coding. Since the detail quantization vector \(q_d^E\) has learned to preserve fine‑grained information and the structural quantization vector \(q_s^E\) has learned to encode smooth content at lower fidelity, substituting the mask naturally assigns higher reconstruction quality to the foreground regions while allowing the background to be more heavily quantized. As shown in Fig. 11, taking the Dancer sequence as an example, the user can specify the foreground to use the quantization vector corresponding to the high bitrate and the background to use that of the low bitrate. With DSRC, the user can effectively reduce the background bitrate without affecting the video generation quality.

4.8.3 Effectiveness of Perceptual Switching↩︎

To verify the effectiveness of the proposed perceptual switching method, we conduct an ablation study that progressively enables its two key components: the module bank-based perceptual switching (MBPS), which replaces all quantization vectors and the factorized entropy model with their perceptual counterparts, and the frame generator-based perceptual switching (FGPS), which additionally replaces the frame generator with one trained for perceptual quality. We take our codec with signal‑fidelity mode as the baseline (\(M_4\)) and evaluate DISTS and LPIPS BD‑rate savings against it. As shown in Table 3, enabling MBPS alone in model \(M_5\) achieves BD‑rate savings of \(27.5\%\) in DISTS and \(16.9\%\) in LPIPS. This demonstrates that switching the module bank already yields a substantial perceptual improvement over the fidelity‑only baseline. When FGPS is further enabled in model \(M_6\), the BD‑rate savings increase to \(51.8\%\) in DISTS and \(44.1\%\) in LPIPS. The large additional gains indicate that a dedicated perceptual frame generator can further benefit the overall perceptual quality.

5 Conclusion↩︎

In this paper, we propose a unified real-time neural video codec with temporal, spatial, and perceptual adaptivity. We first propose a rate‑aware adaptive temporal prediction method that generates diverse prediction candidates and couples candidate selection directly to bitrate constraints. Second, we propose a decomposition‑based spatial rate control method that achieves finer‑grained spatial bit allocation and enables direct spatial rate control at test time. Third, we propose a perceptual switching method that allows a codec to switch between a signal fidelity mode and a perceptual quality mode. Extensive experiments validate that our codec delivers excellent objective quality, perceptual quality, and zero-shot ROI coding. These results confirm that our methods significantly enhance the practicality of neural video coding, advancing it toward broader real-world deployment.

References↩︎

---

1. Date of current version 2026-07-18.↩︎

2. X. Sheng and C. Chen are with the Department of Computing, The Hong Kong Polytechnic University, Hong Kong SAR, China (e-mail: xihua.sheng@polyu.edu.hk, changwen.chen@polyu.edu.hk).↩︎

3. Corresponding Author: Chang Wen Chen.↩︎