2

1 Introduction↩︎

With the continuous development of video capture, storage, compression, and display technologies, numerous video applications continue to emerge, such as video communications, online conferences, cloud gaming, immersive video experiences, and so on. These advancements bring forth new challenges for video coding technologies. To meet the increasing demand for video compression, various video coding tools and technologies have been proposed, leading to continuous evolution in video coding standards. A significant milestone in this progression was the finalization of the High Efficiency Video Coding (HEVC) [1] standard in 2013, which achieved approximately 50% bitrate saving compared to its predecessor, the Advanced Video Coding (AVC) standard [2]. The latest video coding standard, Versatile Video Coding (VVC) [3], has further improved upon HEVC by achieving roughly 50% bitrate reduction. While H.266/VVC demonstrates excellent video compression capabilities, there remains significant potential for enhancing video coding efficiency further. In the pursuit of exploring advanced video encoding tools, a software model named the Enhanced Compression Model (ECM) has been introduced to further explore the potential of video compression [4].

As a result of the prevalent utilization of block-based operations and coarse quantization within contemporary video coding standards, artifacts such as blocking and ringing have become inherent in compressed frames, thereby markedly diminishing both objective and subjective quality. To mitigate these compression artifacts, extensive exploration has been conducted on in-loop filter algorithms during the evolution of video coding standards. These filters serve to enhance the quality of reconstructed frames while also furnishing high-fidelity reference frames for subsequent images, thereby facilitating more accurate motion compensation.

There are four kinds of in-loop filters in VVC [5], i.e., deblocking filter (DBF) [6], sample adaptive offset (SAO) [7], adaptive loop filter (ALF) [8], and luma mapping with chroma scaling(LMCS) [9]. The bilateral filter (BIF)[10], was newly adopted in ECM. These filters are depicted in Fig. 1.

DBF aims to remove the blocking artifact by applying a set of low-pass filters to the boundaries of the coding unit, the prediction unit, and the transform unit. SAO is conducted by conditionally adding an offset to the reconstructed samples after DBF, which shows promising performance in reducing the mean sample distortion and the ringing artifacts. ALF is a Wiener-based spatial filter. It enhances reconstructed video fidelity by taking the weighted average of reference samples as the filtered samples. The weighting coefficients are derived by minimizing the mean square error between the original and decoded samples in the encoder and transmitted to the decoder. LMCS does not particularly focus on artifact reduction but aims to enhance coding efficiency by better utilizing the dynamic range. BIF is a nonlinear, edge-preserving, and noise-reducing filter that is newly introduced in ECM. Similar to the ALF, it also replaces the intensity of each pixel with a weighted average of intensity values from nearby pixels. While, the difference lies in that the weights of BIF depend on the Euclidean distance of pixels and the radiometric differences. This preserves sharp edges. These weights can be calculated both in the encoder and decoder.

In addition to the above-mentioned local filters adopted in the ECM, some other in-loop filters based on the image non-local similarity have been studied, such as structure-driven adaptive non-local filter(SANF) [11], non-local structure-based loop filter (NLSF) [12]–[14], novel adaptive loop filter utilizing image non-local prior knowledge [15], parametric non-local loop filter (PNLF) [16], deformable wiener filter (DWF) [17]. Some of these methods were also discussed in the JVET meeting [18]–[21].

Though the aforementioned in-loop filters are effective in reducing compression artifacts, these conventional methodologies characterized by hand-craft designs exhibit constraints in addressing more intricate artifacts. In response to this constraint, in-loop filters leveraging convolutional neural networks (CNNs) have been developed, demonstrating superior performance compared to conventional filtering methods [22]–[25]. Various neural network-based loop filtering tools have been proposed and adopted by ECM, achieving significant performance improvement [26]–[29].

While the coding techniques mentioned above only focus on single-component in-loop filtering, ignoring the correlation between different components. Extensive research has demonstrated that there is a high correlation between luma and chroma components in the YUV format[30]–[33]. Based on this correlation, some prediction techniques were proposed, such as Cross Component Prediction (CCP)[34] supported in the HEVC Range Extensions and Cross Component Linear Model (CCLM)[35]. Besides, cross-component techniques are also applied in end-to-end image compression[36], which effectively improve compression performance. Recently, the correlation between different components was also considered in the in-loop filters.

Several cross-component in-loop techniques were proposed and adopted in H.266/VVC and Audio Video Coding Standard (AVS3), an independently developed Chinese audio-video coding standard. There is also continuous study on these methods during the development of ECM. In the ECM-12.0, there are two cross-component filters, namely Cross Component Adaptive Loop Filter (CCALF), and Cross Component Sample Adaptive Offset (CCSAO). CCALF was initially proposed and adopted during the development of H.266/VVC and was optimized and improved in ECM. Similar to ALF, CCALF is also a Wiener Filter. The difference is that it only applies to chroma samples, and it utilizes luma samples as the reference samples and corrects the target chroma pixel by applying a linear filter to these selected luma samples. The filter parameters are trained following the principle of minimizing the mean square error (MSE) in the encoder and transmitted to the decoder. CCSAO was adopted by AVS3 and ECM. Specifically, it uses the correlation between luma and chroma components to classify the reconstructed samples into different categories and assigns each category an offset value for sample adjustment.

Compared to ECM-12.0 without CCALF [4], ECM-12.0 with CCALF achieves 2.49% and 2.90% coding gains for the Cb and Cr components under All intra (AI) configuration, and 1.48% and 2.12% coding gains for Cb and Cr components under Random Access (RA) configuration. While in the VTM-10.0, CCALF can achieve 13.88% and 13.73% coding gains under AI configuration, and 9.69% and 8.55% coding gains under RA configuration for Cb and Cr components respectively [37]. The decrease in the coding gain may caused by the new cross-component techniques introduced in the prediction process of ECM. For CCSAO, 1.28% and 1.08% coding gains can be achieved for Cb and Cr components under AI configuration, and 3.02% and 2.79% coding gains for Cb and Cr components can be achieved under RA configuration, respectively.

The remainder of this paper is organized as follows, Section 2 introduces the theory of CCALF and summarizes the development of CCALF. Section 3 introduces the fundamental principles of CCSAO and the proposals about CCSAO. Experimental results and discussions are shown in Section 4. Section 5 concludes this paper.

2 CCALF↩︎

CCALF is fundamentally a Wiener Filter [38]. Specifically, CCALF derives a correction signal for chroma samples based on the weighted average of luma reference samples. These reference samples are the neighboring samples of the to-be-filtered chroma sample’s collaborated luma sample. The collaborated luma sample is derived based on the chroma format of the video.

Both the ALF and the CCALF use the reconstructed sample of SAO as input, while the CCALF only calculates the offsets for chroma components as shown in Fig. 2. The filtering operation can be represented using the equation below, Let us assume the following for 2-D images.

) Sample location \(r=(x,y)\) belongs to the to-be-filtered region \(R\), \(r'=(x',y')\) means the collocated position to the to-be-filtered sample.
2) Original sample: \(s[r]\)
3) To-be-filtered sample: \(I[r]\)
4) Collocated luma samples of \(I[r]\): \(L(r')\)
5) \(N\)-tap filter coefficients: \(\mathbf{c} = [c_{0}, c_{1}, c_{2},...,c_{N-1}]\)
6) Filter tap position offset: \(\left\{\mathbf{d}_{0}, \mathbf{d}_{1}, \mathbf{d}_{2},...,\mathbf{d}_{N-1}\right\}\), where \(d_{i}\) denotes the sample location offset to \(L(r')\) of the \(i\)th filter tap.
7) The difference values between neighboring reference luma samples and the to-be-filtered chroma sample: \(\mathbf{p} = [p_{0}, p_{1}, p_{2},...,p_{N-1}]\)
8) Filtered chroma sample:\(\tilde{I}(r)\)

\[{\tilde{I}(r) = I(r)+\sum_{i = 0}^{N-1} c_{i} p_{i}}, \label{eq1}\tag{1}\]

\[{p_{i}=L(r'+\mathbf{d}_{i})-I(r),}\]

The coefficients of CCALF are derived by minimizing the mean square error between the reconstructed chroma component after SAO and the original chroma sample, similar to the parameter derivation process of Chroma-ALF. Specifically, a correlation matrix is derived, and the coefficients are calculated using the Cholesky decomposition solver to minimize the mean square error.

The coefficient values at different positions are obtained from the bitstream. The filter coefficients are derived by solving the following optimization problem as shown in Eqn.( 2 ), \[{\min_c \sum_{r\in R}^{} {\left ( \mathbf{c}\odot \mathbf{p} - s\left [ r \right ] \right ) } ^{2} ,} \label{eq3}\tag{2}\] \[{\mathbf{c} = R_{r,r}^{-1} R_{r,s}^{},} \label{eq4}\tag{3}\] where \(\odot\) is the inner product. By solving the Wiener-Hopf equations as shown in Eqn.( 3 ), the filter coefficients can be calculated. \(R_{r,r}^{-1}\) denotes the auto-correlation matrix of the to-be-filtered samples, \(R_{r,s}^{}\) is the cross-correlation vector of the to-be-filtered and the original samples.

false

2.1 Evolution of CCALF↩︎

CCALF is employed within the in-loop filtering process to enhance chroma fidelity by leveraging the correlation between luma and chroma. In the initial study, ALF was conducted on both luma and chroma components, enabling parallel processing of different sample components [8]. However, with the continuous advancement of video coding standards, the robust correlation between the luma and chroma components has been utilized. CCALF was proposed at the 15th JVET meeting in June 2019 and has since been integrated into the H.266/VVC.

CCALF was introduced in H.266/VVC, and the ALF filtering framework containing CCALF is illustrated in Fig. 2. In H.266/VVC, CCALF utilizes the output luma sample of SAO as input, and its output is employed to refine the chroma reconstructed sample of ALF. Unlike some tools that use luma samples to predict chroma samples, such as the CCLM intra prediction, CCALF doesn’t introduce latency as the luma sample used does not require extensive processing before reconstructing the chroma sample. The data flow depicted in Fig. 2 demonstrates that CCALF can be parallelized with ALF.

In H.266/VVC, the \(3\times4\) diamond filter was selected to compute the filter coefficients of CCALF, with the center of the diamond filter aligned with the chroma sample. Fig. 3 illustrates the relative location of the chroma sample being filtered and its support region in the luma sample when CCALF is adopted in H.266/VVC. Consequently, each CCALF filter only needs to store 7 filter coefficients. In H.266/VVC, the CCALF filter coefficients are computed by minimizing the mean square error of each chroma component relative to the original chroma sample. The H.266/VVC algorithm employs a similar parameter derivation process to Chroma-ALF. Specifically, a correlation matrix is derived, and the coefficients are calculated using the Cholesky decomposition solver to minimize the mean square error.

In ALF, the number of coefficients is reduced by utilizing symmetry. However, CCALF does not adopt this method due to its potential impact on compression efficiency. Instead, CCALF employs a more restricted symmetry scheme at the tips of the diamond.

Due to the abundant texture features of the luma component, CCALF may introduce artifacts with overly abundant chroma texture, thereby reducing the subjective quality of the image, especially at high QP. Therefore, the H.266/VVC reference encoder can achieve subjective tuning by configuring the config file. Specifically, it can attenuate the application of CCALF in high QP encoding and areas with high-frequency luminance. Algorithmically, CCALF is deactivated on CTUs that meet the established conditions.

Numerous proposals related to CCALF have been put forward, considering the trade-off between performance and running time, complexity, or other factors. However, some of these proposals were not adopted.

In [39], an extension to CCALF has been proposed. This contribution suggests extensions to CCALF in both the number and size of filters. While this extension can enhance chroma components, it may lead to some loss in the luma component. Additionally, CCALF could introduce artifacts in chroma components, which is why certain constraints are set in high QP regions. Therefore, proposals regarding CCALF must avoid reintroducing these artifacts.

Considering that the correlation between neighboring pixels may depend on the characteristics of the video content, a single filter shape may not be optimal for all types of video content. To further enhance the performance of CCALF, a Coding Tree Block (CTB) level filter shape selection process has been proposed [40]. This contribution introduces two filter shapes, as shown in Fig. 4. Within each Adaptation Parameter Set (APS), multiple filters and their corresponding shapes with coefficients are signaled. For each CTB, the decoder specifies which filter shape and coefficients are used based on the signaled index.

On the other hand, this contribution has also proposed to remove the power of 2 constraints on filter coefficient values, which was proposed again in the JVET meeting in [41], [42].

This contribution demonstrates significant gains in chroma components. However, the necessity of adaptively selecting CCALF shapes was questioned. In a subsequent exploration experience [43], in addition to the adaptive selection of the two filter shapes proposed in [40], another scheme involving larger size filters was proposed. Specifically, a 25-tap long-tap CCALF was introduced. This long-tap filter was considered a simpler scheme to achieve the gain. After joint tests of the modified CCALF and other in-loop filters [44], the long-tap CCALF scheme was eventually adopted. The new size of CCALF is illustrated in Fig. 5.

Because residual values have been stored and used in luma ALF, the concept of residual-based taps in chroma ALF and CCALF has also been proposed [45]. Before this contribution, CCALF only had one online-trained CCALF filter with a cross-liked filter shape mentioned above, as depicted in Fig. 5. Since the residual values are utilized in the unfixed luma filter of ALF, there is no need to store luma residual values additionally. In this contribution, only one luma-residual-based tap is added. Furthermore, chroma residual values are incorporated into the chroma online-trained filter of ALF, while luma residual values are employed in CCALF. However, considering that chroma residual values were not stored previously and the additional memory required, the resulting gain is comparatively low. Therefore, this proposal was recommended for further study.

At the next round of the Meeting, the introduction of luma residual taps in chroma ALF and CCALF was proposed [46]. Five luma residual taps in a cross 3x3 shape were added. These extended taps take the collocated and neighboring luma residual values as input. The inclusion of the luma residual taps in CCALF was adopted due to its relatively higher standalone gain [47]. The filter shape of CCALF in ECM-12.0 is illustrated in Fig. 6.

The coefficients that need to be calculated are divided into two parts: spatial luma sample-based taps and luma residual-based taps. The linear filtering operation can be represented using Eqn.(1 ).

At the 32nd JVET meeting, coefficient precision adjustment for ALF was proposed, demonstrating promising coding performance with negligible increases in encoding and decoding time [48]. Similarly, at the 33rd JVET meeting, adaptive coefficient precision for CCALF was introduced [41], [42]. Since CCALF involves different coefficient derivation compared to ALF, removing the power of 2 constraints was also proposed in this context. This adjustment can enhance the accuracy of coefficients, though a two-bit syntax element needs to be signaled for each set of luma filter sets to indicate the number of bits. These two contributions have been further investigated.

2.2 Filter Shape↩︎

The filter shape of CCALF was a \(5\times6\) diamond-shaped filter with 14 filter coefficients and 18 taps when it was initially proposed [49]. Considering the trade-off among performance, line buffer, and computational complexity, several reduced filter shapes were proposed [50]–[53]. Finally, the \(3\times4\) diamond filter shape was adopted in H.266/VVC. Fig. 3 illustrates the relative location of the chroma sample being filtered and its support region in the luma sample when CCALF is adopted in H.266/VVC. Consequently, each CCALF filter has only 7 filter coefficients, the filtering operation is shown in Eqn.(1 ) and \(N=7\).

To further improve the performance of CCALF, numerous proposals related to CCALF have been put forward, considering the trade-off between performance and running time, complexity, or other factors. However, some of these proposals were not adopted.

In [39], an extension to CCALF has been proposed. This contribution suggests extensions to CCALF in both the number and size of filters. While this extension can enhance chroma components, it may lead to some loss in the luma component. Additionally, CCALF could introduce artifacts in chroma components, which is why certain constraints are set in high QP regions. Therefore, proposals regarding CCALF must avoid reintroducing these artifacts.

Considering that the correlation between neighboring pixels may depend on the characteristics of the video content, a single filter shape may not be optimal for different video content. To further enhance the performance of CCALF, a Coding Tree Block (CTB) level filter shape selection scheme was proposed [40]. This contribution introduces two filter shapes, as shown in Fig. 4. Within each Adaptation Parameter Set (APS), multiple filters and their corresponding shapes with coefficients are signaled. For each CTB, the decoder specifies which filter shape and coefficients are used based on the signaled index.

This contribution demonstrates significant gains in chroma components. However, the necessity of adaptively selecting CCALF shapes was questioned. In a subsequent exploration experience [43], in addition to the adaptive selection of the two filter shapes proposed in [40], another scheme involving larger-size filters was proposed. Specifically, a 25-tap long-tap CCALF was introduced. This long-tap filter was considered a simpler scheme to achieve better gain. After joint tests of the modified CCALF and other in-loop filters [44], the long-tap CCALF scheme was eventually adopted. The new shape of CCALF in ECM is illustrated in Fig. 5, the filtering operation is shown in Eqn.(1 ) and \(N=25\).

Because residual values have been stored and used in luma ALF, the concept of residual-based taps in chroma ALF and CCALF has also been proposed [45]. Before this contribution, CCALF only had one online-trained CCALF filter with a cross-liked filter shape mentioned above, as depicted in Fig. 5. Since the residual values are utilized in the unfixed luma filter of ALF, there is no need to store luma residual values additionally. In this contribution, only one luma-residual-based tap is added. Furthermore, chroma residual values are incorporated into the chroma online-trained filter of ALF, while luma residual values are employed in CCALF. However, considering that chroma residual values were not stored previously and the additional memory required, the resulting gain is comparatively low. Therefore, this proposal was recommended for further study.

At the 31st JVET meeting, the introduction of luma residual taps in chroma ALF and CCALF was proposed [46]. Five luma residual taps in a cross 3x3 shape were added. These extended taps take the collocated and neighboring luma residual values as input. The inclusion of the luma residual taps in CCALF was adopted due to its relatively higher standalone gain [47]. The filter shape of CCALF in ECM-12.0 is illustrated in Fig. 6.

The coefficients that need to be calculated are divided into two parts: spatial luma sample-based taps and luma residual-based taps. The linear filtering operation can be represented using Eqn.(4 ).

\[\begin{align} {\tilde{I}(x, y) = I(x, y)+\sum_{i = 0, 12} c_{i}\left(f_{i, 0}+f_{i, 1}\right)}\\ {+\sum_{i = 1}^{21} c_{i} f_{i}+\sum_{i = 22}^{26} c_{i} g_{i},} \end{align} \label{eq2}\tag{4}\] \[\begin{align} {f_{i, j}=L(x'+x_{i,j}, y'+y_{i,j})-I(x, y),} \end{align}\] \[\begin{align} {g_{i}=Clip(R(x'+x_{i}, y'+y_{i})),} \end{align}\]

where \((x,y)\) is the coordinate of the center sample, \((x',y')\) is the coordinate of collocated luma sample. \((x'+x_{i}, y'+y_{i})\) and \((x'+x_{i,j}, y'+y_{i,j})\)are the coordinates of the reconstructed samples corresponding to coefficient \(c_{i}\), \(f_{i,j}\) and \(f_{i}\) is the difference between neighboring luma samples \(L(x',y')\) and current sample \(I(x, y)\), \(g_{i}\) is the clipped value of luma residual samples \(R(x',y')\) which is the residual between prediction samples and reconstructed samples. \(Clip\) is the function that limits the values within a certain range to reduce the impact of significant differences in sample values, the value of the clipping operation depends on the clipIdx of APS and bitdepth.

2.3 Filter Coefficient Calculation and Representation↩︎

Except for the filter shape of the CCALF, the optimization of coefficient calculation and signaling [49], [54]–[57] is also an important topic to improve the performance of CCALF.

When CCALF was proposed, each filter had 14 filter coefficients and 18 taps, and every coefficient has an 8-bit dynamic range and is signaled with a third-order exponential-Golomb code [49]. However, it would increase complexity with additional multiplications per chroma pixels. To simplify the computation overhead, a bit shifting scheme was proposed to replace the multiplications [54]. The results show that this scheme can reduce the complexity of the CCALF filter with accepted loss, so it was adopted. Besides, a contribution was proposed to reduce memory access, encoding latency, and power consumption [57]. It proposes a method to estimate CCALF filtering distortion without conducting real filter operations. With this proposal, the number of encoding passes can be reduced from 152 to 1 while did not affect the coding performance. As a desirable simplification, this proposal was adopted.

At the 32nd JVET meeting, coefficient precision adjustment for ALF was proposed, demonstrating promising coding performance with negligible increases in encoding and decoding time [48]. Similarly, at the 33rd JVET meeting, adaptive coefficient precision for CCALF was introduced [41], [42]. Since CCALF involves different coefficient derivation compared to ALF, removing the power of 2 constraints was also proposed in this context. This adjustment can enhance the accuracy of coefficients, though a two-bit syntax element needs to be signaled for each set of luma filter sets to indicate the number of bits. These two contributions have been further investigated.

2.4 Syntax Design↩︎

Compared to H.266/VVC, ECM-12.0 utilizes luma residual samples additionally, as shown in Fig. 6. The residual correction is generated for chroma samples according to Eqn.(4 ). For each picture, two types of information need to be coded for CCALF, i.e., filter coefficient parameters and filter control on/off flags. The filter coefficient parameters include the number of cross-component filters and the coefficients of the corresponding filter. CCALF can transmit up to 8 CCALF filters, with the resulting filters then indicated for each of the two chroma channels on a CTU basis. Each slice only has one APS, and the Cb component and Cr component can have different APSs, which are signaled separately at the slice header. Similar to luma ALF, to reduce bit overhead, filter coefficients of different classifications can be merged. The filter control on/off flags are used to provide better local adaptation, there are sequence-level, picture-level, slice-level and CTU-level filter on/off control. When the value of sequence-level and picture-level control flags is not present, it is inferred to be equal to 0. When slice-level on/off control flag is not present, it is inferred to be equal to picture-level on/off control flags. If the slice-level on/off control flag indicates ALF-on, CTU-level filter on/off control flags are interleaved in slice data and coded with CTUs; otherwise, no additional CTU-level filter on/off control flags are coded and all CTUs of the slice are inferred as ALF-off.

Due to the abundant texture features of the luma component, CCALF may introduce artifacts with overly abundant chroma texture, thereby reducing the subjective quality of the image, especially at high QP. Therefore, the H.266/VVC reference encoder can achieve subjective tuning by configuring the config file. Specifically, it can attenuate the application of CCALF in high QP encoding and areas with high-frequency luminance. Algorithmically, CCALF is deactivated on CTUs when any of the following conditions are true,

(1) The slice QP value minus 1 is less than or equal to the base QP value.

(2) The number of chroma samples for which the local contrast is greater than \(( 1 << ( bitDepth-2 ) )-1\) exceeds the CTU height, where the local contrast is the difference between the maximum and minimum luma sample values within the filter support region.

(3) More than a quarter of chroma samples are in the range between \(( 1 << ( bitDepth -1 ) )-16\) and \(( 1 << ( bitDepth -1 ) ) + 16\).

3 CCSAO↩︎

CCSAO is conceptually similar to SAO, as it initially classifies the samples to be filtered into different categories, then derives an offset value for each category, and finally corrects the pixels in that category with the corresponding offset value. It uses the reconstructed sample of DBF, which is the same as SAO, the offsets are derived for three channels respectively. The reconstruction operation of CCSAO can be represented by the equation below, \[{\tilde{C_{rec}} = Clip(C_{rec}+offset_{i}),} \label{eq8}\tag{5}\] \(C_{rec}\) and \(\tilde{C_{rec}}\) are the reconstructed sample after DBF and CCSAO, respectively. \(i\) represents the class index of the corresponding sample, \(offset_{i}\) is the corresponding offset value.

The difference between SAO and CCSAO lies in CCSAO’s utilization of the strong correlation between the luma and chroma components in the classification process. It optimizes the reconstruction of one component of the sample by leveraging the information contained in the other component of the sample [58].

3.1 Classifier Extension↩︎

The original CCSAO includes only a classification based on band information to avoid a significant increase in complexity. Corresponding band offsets are obtained by minimizing the sum of squared error (SSE) between the original sample and the corrected reconstruction sample. This approach keeps computational complexity low while enabling CCSAO to handle certain encoded artifacts. It should be noticed that the offsets need to be signaled in the bitstream.

CCSAO is applied to the output of DBF reconstructed samples, and the offset calculated for each category is added to the output sample from the SAO process. Therefore, CCSAO can be parallelized with SAO, as shown in Fig. 11.

The band information-based classification of CCSAO utilizes the reconstructed sample of three components to process the classification for each component. Specifically, the collocated samples for each component are first selected. Then, an index representing a category is calculated based on the band number of the three components and their collocated samples. The offset value of a sample depends on its category. Regarding the collocated samples for each component, the collocated luma sample can be chosen from 9 candidates, while the collocated chroma samples have fixed positions, as shown in Fig. 12.

CCSAO was first proposed and adopted[59] in the AVS3 video coding standard, in which collocated luma component samples are classified by equally dividing the range of the sample values. For each category, an offset value is derived and used for the chroma samples whose collocated luma sample belongs to the category.

Although cross-component tools in in-loop filters always act on chroma components, regarding cross-component proposals, attention should not only be given to the gain of chroma components but also to the effects on the luma component. Furthermore, subjective quality improvement needs to be considered as well. Considering these reasons, CCSAO was introduced to ECM. This proposal showed great performance improvement in chroma components while introducing minimal loss in the luma component. Initially, CCSAO only used the band classifier when it was adopted in ECM [60], and the category index will be calculated using the equation below,

\[{classIndex=band_{Y} * (N_{Cb}*N_{Cr}) + band_{Cb} * N_{Cr} + band_{Cr}},\] \[{band_{L}=P(x_{Y}, y_{Y}) * N_{Y} >> BitDepth},\] \[{band_{Cb}=P(x_{Cb}, y_{Cb}) * N_{Cb}>> BitDepth},\] \[{band_{Cr}=P(x_{Cr}, y_{Cr}) * N_{Cr}>> BitDepth},\]

where \(P(i, j)\) is the sample value of different component at position \((i, j)\), \(N_{i}\) is the number of band for each component, \((x_{C}, y_{C})\) is the current chroma sample position, \(x_{L}, y_{L}\) is the collocated luma sample position. As a new in-loop filter tool, several schemes have been proposed to optimize the original CCSAO. An extension of CCSAO was proposed at the 24th of JVET meetings, where the proponents extended the design of CCSAO by adding the edge-based classifier [61], [62]. Similar to the edge-based classification method in SAO, the edge-based classification of CCSAO also uses four 1-D directional patterns, including horizontal, vertical, 45°, and 135°, as shown in Fig. 13. The best direction mode is determined at the encoder through rate-distortion optimization (RDO). Edge information used for classification is derived by calculating the difference between the center pixel and its two adjacent pixels, and then comparing the difference with a predefined threshold value to derive the final class index. The best threshold values are also selected from an array of predefined threshold values based on RDO. If the edge-based classifier is selected, the category index will be calculated as follows, given the chroma sample and the collocated luma samples,

\[\begin{align} {classIndex = BandNum \ast 16 + q_{a} \ast 4 + q_{b}}, \end{align} \label{cuskzpfn}\tag{6}\] \[{q_{i} =\left\{ \begin{array}{lc} 0 & d_{i}<-Th, \\ 1 & -Th<d_{i}<0,\\ 2 & 0<d_{i}<Th,\\ 3 & Th<d_{i}.\\ \end{array} \right.}\]

\[{BandNum = cur_{i} * N_{i} >> BitDepth}, \label{eq9}\tag{7}\] where \(i\) can be chosen from the two co-located samples based on RDO, \(d_{i}\) is the delta value between the center sample \(c\) and the neighboring sample \(a\) or \(b\). \(q_{i}\) is the quantized value of \(d_{i}\). The position of neighboring sample \(a\) or \(b\) depends on the best 1-D directional pattern selected from the four 1-D directional patterns. Besides, the range of the offset value is also constrained to \([-15, 15]\), and these offsets are needed to transmit to the decoder.

Unlike SAO, the edge-based classifier in CCSAO combines luma edge and the band index of the sample at the corresponding collocated position to determine the final classification of a given sample. Additionally, CCSAO uses collocated luma samples to derive edge information for chroma samples, while SAO uses neighboring samples of the same component to derive edge information.

A similar contribution has been proposed at AVS [63], the Enhanced Cross-Component Sample Adaptive Offset (ECCSAO) method further improves encoding performance, which includes an extension of the edge-based classification method, using the edge information of collocated luma samples to classify chroma samples. Moreover, a four-layer quad-tree structure is proposed. The former method has been adopted by AVS.

In the ECM, the edge classifier was further optimized with more edge/band combinations, and the component used for edge classification can be selected from any of the three components [64], [65]. The new edge-based classification scheme, which is a subset of the original one with fewer edge range divisions was added. This allows for more flexible edge/band combinations to adapt to the local characteristics of video sequences. This contribution was adopted at the 31st JVET meeting. The second edge-based classifier is formulated as follows, \[\begin{align} {classIndex = BandNum \ast 4 + q_{a} \ast 2 + q_{b}}, \end{align}\] \[{q_{i} =\left\{ \begin{array}{lc} 0 & d_{i}<-Th,\\ 1 & -Th<d_{i}<0.\\ \end{array} \right.}\]

3.2 Signaling Overhead Reduction↩︎

Similar to the APS design in H.266/VVC, the inheritance scheme of CCSAO was also proposed in [64], [65]. It was noted that there is a strong correlation between the CCSAO offsets and classifier parameters of different pictures. To reduce signaling overhead, the offsets/parameters of some coded pictures can be stored at both the encoder and decoder, allowing them to be used by future pictures. This contribution has also been adopted.

false

3.3 CCSAO in ECM↩︎

During the development of ECM, CCSAO has been adopted due to its excellent coding performance. Two modes can be selected in ECM: band mode, which is based on texture information, and edge mode, which is based on edge information.

If the edge mode is selected, the category index will be calculated as follows, given the chroma sample and the collocated luma samples,

\[\begin{align} {classIndex = BandNum \ast 16 + q_{a} \ast 4 + q_{b}}, \end{align} \label{sfdmpoxy}\tag{8}\] \[{q_{i} =\left\{ \begin{array}{lc} 0 & d_{i}<-Th, \\ 1 & -Th<d_{i}<0,\\ 2 & 0<d_{i}<Th,\\ 3 & Th<d_{i}.\\ \end{array} \right.}\]

where \(d_{i}\) is the delta value between the center sample \(c\) and the neighboring sample \(a\) or \(b\). \(q_{i}\) is the quantized value of \(d_{i}\). Variable \(BandNum\) in Eqn.(5 ) is derived as follow, \[{BandNum = cur_{i} * N_{i} >> BitDepth}, \label{uclidhgx}\tag{9}\] where \(i\) can be chosen from the current sample being processed and the two co-located samples based on RDO.

4 Performance Evaluation↩︎

To improve the coding performance, both CCALF and CCSAO are integrated into ECM-12.0 seamlessly. A comparative analysis is conducted to evaluate the efficiency and effectiveness of the cross-component in-loop filter tools. With continuous development, both CCSAO and CCALF have achieved remarkable performance gains. To evaluate the coding performance of CCALF and CCSAO, ECM-12.0 without CCALF and CCSAO are regarded as anchor respectively [66].

As shown in Table 1, CCALF can achieve 2.49% and 2.90% coding gains for Cb and Cr components under AI configuration. For RA configuration, 1.48% and 2.12% coding gains for Cb and Cr components can be achieved. In VTM-10.0, CCALF can achieve 13.88% and 13.73% coding gains for Cb and Cr components under AI configuration, and 9.69% and 8.55% coding gains for Cb and Cr components under RA configuration. The decrease in gain may caused by the newly proposed and optimized cross-component techniques in the prediction process. For CCSAO, 1.28% and 1.08% coding gains can be achieved for Cb and Cr components under AI configuration. For RA configuration, 3.02% and 2.79% coding gains for Cb and Cr components can be achieved.

It can be noted that the Y component coding performance of CCSAO on screen content sequences is significantly greater than that of natural sequences. This may be caused by the more obvious relationship between the texture and directional features of luma and chroma components in screen content videos. Furthermore, we also compared the subjective performance under different configurations. The subjective testing materials consist of the sequences mentioned in CTC and each sequence has been encoded with four QPs(QP=22, 27, 32 and 37) under AI and RA configurations. Partial visual quality comparison results of reconstructed sequences are shown in Fig. 5. In Fig. 5, the decoded images with CCALF and CCSAO are provided in the first column, the decoded images without CCALF, the last column shows the reconstructed image without CCSAO. The red boxes are highlighted to indicate the region with significant subjective improvement.

4.1 the trend of loop filters↩︎

The loop filters are designed to correct artifacts that are introduced before loop filtering. Different kinds of loop filters can deal with different artifacts like blocking, ringing, blurring, mosquito, etc. There are three kinds of crucial loop filters in VVC in total, namely DBF, SAO, and ALF. In VVC, the cross-component loop filter called CCALF is introduced to ALF to utilize the relationship between luma and chroma components fully. To further exploit the relationship, another cross-component loop filter is proposed for ECM named CCSAO, which is paralleled with SAO. With the development of ECM, the classifiers of CCSAO become more refined and diverse [61], [62], [64], [65], [67], the structure of CCALF become more complex and comprehensive, more kinds of samples were added to the filters of CCALF [45], [46], the shape and calculation method of filters are also constantly being optimized [41], [50]–[54]. Besides, some other in-loop filters based on the image non-local similarity have been studied [11], [12], [15], [16] because the loop filter in existing video coding standards focus only on the local correlation.

While these non-local loop filters can yield certain performance gains, their high computational demands and hardware limitations render their application in video coding standards challenging. Therefore, relevant methods to optimize non-local filters are exploring [13], [14]

false

5 Conclusion↩︎

Cross-component filters play a crucial role in the future of video coding standards. By leveraging the correlation between luma and chroma components, cross-component filters can achieve substantial coding performance improvement, leading to the adoption of various video coding standards such as VVC and AVS3. Compression distortion can be effectively mitigated, thereby improving the accuracy of the reconstructed pixel. Nevertheless, the philosophy of current cross-component filters primarily emphasizes utilizing luma information to refine chroma pixels, which neglects the potential impact of chroma information on luma pixels and the correlation between two chroma components. In some scenarios, the chroma texture information and edge details can also contribute to correcting luma inaccuracies. Therefore, cross-component filters still have the potential to achieve substantial performance improvement by delving into the filtering manner and relationship between different channels.