Division-of-focal-plane image polarimeter, color-polarization filter array, denoising, demosaicking

2.0ex plus 0.2ex minus 0.1ex 1.0ex plus 0.2ex Introduction

Color-polarization imaging captures both color texture and polarization information of the scene and has various computer vision applications, such as material segmentation [1] and 3D reconstruction [2]. A division-of-focal-plane polarimeter equips a color–polarization filter array (CPFA, see Fig. 1) on an image sensor, enabling the acquisition of four polarization-orientation data (\(0^\circ\), \(45^\circ\), \(90^\circ\), and \(135^\circ\)) in one shot [3]. However, since CPFA consists of a mosaic of 12-channel data (four polarization orientations and three color channels), an interpolation process of missing information called demosaicking (DM) is required to obtain full 12-channel color-polarization data. In addition, CPFA data suffer from noise in some scenarios, such as fast (short-exposure) and low-light imaging, necessitating denoising (DN) at the same time.

Most existing studies focus on either DM or DN in isolation, while little attention has been paid to designing a unified pipeline for CPFA sensors. As a result, current solutions rely on a simple sequential combination of DM and DN, arranged sequentially as DM-to-DN or DN-to-DM. However, these separated approaches overlook the strong coupling between DM and DN, which not only prevents feature sharing but also propagates errors. In particular, DM-to-DN tends to propagate noise during DM,whereas DN-to-DM often oversmooths signal-dependent structures during DN. Although CPFA-specific DM or DN methods have been developed (e.g., [4]–[7]), they still lack a holistic design for CPFA imaging with DM and DN as core components and thus perform poorly in high-noise conditions.

In this paper, we propose a color-polarization denoising and demosaicking network, namely CPDDNet. Figure 1 illustrates the differences between the straightforward pipelines (DM-to-DN and DN-to-DM) and the pipeline of our CPDDNet, where we consider the two-step approach of [8] for DM. Although our pipeline is based on the DN-to-DM pipeline, we newly introduce a feature fusion encoder to DM that fuses the features from the noisy raw CPFA image with those from the denoised CPFA image. This retains the information from the raw CPFA data at both the DN and the DM stages, thereby reducing information loss in the whole pipeline. We evaluate our proposed CPDDNet on a recently published real dataset of [9] and demonstrate its superior performance. The main contributions of this study are summarized as follows.

• We propose CPDDNet, which is the first network architecture combining DN and DM for CPFA sensors, to the best of our knowledge.

• We design a unified DN-to-DM pipeline equipped with a feature fusion encoder, enabling information bridging between the DN and the DM stages.

• We experimentally validate that our CPDDNet achieves state-of-the-art performance and outperforms existing approaches both numerically and visually. The source code is publicly available at our project page¹.

2.0ex plus 0.2ex minus 0.1ex 1.0ex plus 0.2ex Related Work

While CFPA-specific DN [6], [7] and DM [4], [5], [8], [10], [11] methods have been developed in both signal-processing-based [4], [6], [10] and deep-learning-based [5], [7], [8], [11] manners, only a few studies focus on jointly addressing DN and DM for CPFA sensors. A recent study of [9] addressed the lack of an evaluation dataset for this joint task and constructed a new polarization image dataset containing 40 scenes with three noise-level conditions. This study also constructed a baseline signal-processing-based method, which adopts the DN-to-DM pipeline and demonstrates its effectiveness compared with the DM-to-DN pipeline. While this baseline method effectively combines the pseudo four-channel color denoising (PFCD) of [12] for DN and the intensity-guided residual interpolation (IGRI) of [4] for DM, a method utilizing powerful deep learning techniques is expected to outperform this signal-processing-based baseline method.

To our knowledge, only Li et al. [13] explicitly perform joint DN and DM, targeting monochrome long-wave infrared division-of-focal-plane polarization images with mixed-noise estimation. However, the task is not designed for CPFA sensors, and the approach is essentially a simple stacking of DN and DM modules without specific designs for the joint task.

When joint DN and DM methods are absent, the sequential pipelines of DN-to-DM and DM-to-DN are common options. However, these straightforward pipelines are likely to produce suboptimal results, since noise is typically propagated during DM, and image structures are often oversmoothed during DN.

With the above backgrounds, in this paper, we propose CPDDNet, which is the first network architecture combining DN and DM for CPFA sensors, to our knowledge. Following the observations and the results in [9], we adopt the DN-to-DM pipeline as the base and newly introduce a feature fusion module to bridge the information between DN and DM, reducing the information loss in the whole pipeline.

2.0ex plus 0.2ex minus 0.1ex 1.0ex plus 0.2ex Proposed Method

1.0ex plus 0.2ex minus 0.1ex 0.5ex plus 0.1ex Problem Formulation

In this study, we consider the \(4\times4\) pattern of Sony’s CPFA sensors [3], as shown in Fig. 1, and aim to reconstruct the full noise-free 12-channel color-polarization data described as \[\label{eq:cube95def} \mathbf{I} = \{ I_{\theta}^c \} \in \mathbb{R}^{H\times W \times (4\times3)},\tag{1}\] where \(H \times W\) represents image height and width, \(\theta \in \{0^\circ,45^\circ,90^\circ,135^\circ\}\) is a polarization orientation, and \(c \in \{R,G,B\}\) is a color channel. The input noisy CPFA raw data \(X \in \mathbb{R}^{H\times W}\) is expressed as \[\begin{align} \label{eq:sampling} X = \sum_{k=1}^{12} M_k\odot I_k + N, \end{align}\tag{2}\] where \(I_k \in \mathbb{R}^{H\times W}\) is the \(k\)-th channel image of \(\mathbf{I}\), \(M_k\in\{0,1\}^{H\times W}\) is a binary mask for each channel satisfying \(\sum_{k} M_k(i,j)=1\), representing that only one channel among 12 channels is measured at each pixel \((i,j)\) in the CPFA data, \(\odot\) denotes the Hadamard product, and \(N \in \mathbb{R}^{H\times W}\) represents measurement noise. The joint task of polarization DN and DM is to reconstruct \(\mathbf{I}\) from \(X\).

1.0ex plus 0.2ex minus 0.1ex 0.5ex plus 0.1ex Overall Pipeline

Figure 2 shows the overall pipeline and data structure of our proposed method, which consists of three networks, as explained below.

1) CPFA-domain DN: We first rearrange the \(4 \times 4\) mosaic CPFA pattern into a 16-channel tensor as \(\mathbf{X} \in \mathbb{R}^{\frac{H}{4} \times \frac{W}{4} \times 16}\). Then, we apply a U-Net architecture [14] to that tensor to remove noise while preserving signal-dependent structures. This stage follows the polarization-aware DN paradigm of [7] and produces a denoised tensor \(\mathbf{X}^{\text{DN}}\). This is returned to the mosaic pattern, which is expressed as \(X^{\text{DN}} \in \mathbb{R}^{H\times W}\).

2) Bayer-domain color DM: In the DM stage, we follow the two-step approach of [8], which first applies color DM. From the denoised CPFA data \(X^{\text{DN}}\), Bayer-patterned images [15] for each polarization orientation are constructed by \(2 \times 2\) sub-sampling as \(Y^{\text{DN}}_{\theta} \in \mathbb{R}^{\frac{H}{2} \times \frac{W}{2}}\). While the straightforward DN-to-DM pipeline only adopts the denoised input, we add the noisy Bayer-patterned images \(Y_{\theta}\) constructed from the original CFPA raw data \(X\) as the input for color DM. The multi-scale features extracted from the two inputs are fused by a gated fusion module (GFM), inspired by the decouple-and-feedback fusion [16], adaptively combining high-frequency details from the raw input and robust low-frequency structures from the denoised input. The details of GFM are explained in the next subsection. The outputs of color DM are half-sized three-channel color images for each polarization orientation, which are expressed as \(\mathbf{C}_{\theta} \in \mathbb{R}^{\frac{H}{2} \times \frac{W}{2} \times 3}\).

3) Polar DM: The final step is to apply polar DM. From the set of half-sized three-channel color images for all polarization orientations \(\{\mathbf{C}_{0}, \mathbf{C}_{45}, \mathbf{C}_{90}, \mathbf{C}_{135}\}\), the polarization mosaic data for each color channel is constructed by pixel rearrangement, which is expressed as \(Z_c \in \mathbb{R}^{H\times W}\). This is fed into the polar DM, which produces the full polarization image for each color channel, denoted by \(\mathbf{P}_c \in \mathbb{R}^{H\times W \times 4}\). We follow the network architecture of [8] in this part. Finally, the reconstructed full 12-channel color-polarization data \(\tilde{\mathbf{I}}\) is formed by stacking all the outputs as \(\tilde{\mathbf{I}} = Concat[\mathbf{P}_R, \mathbf{P}_G, \mathbf{P}_B]\), where \(Concat\) represents the concatenation along the channel dimension.

1.0ex plus 0.2ex minus 0.1ex 0.5ex plus 0.1ex Gated Fusion Module (GFM)

The right side of Fig. 3 shows the architecture of our color DM network with GFMs. We adopt a U-Net architecture as the backbone [14] and introduce dual-encoder branches. Our GFM adaptively fuses the features from the dual-encoder branches so that high-frequency features from the raw input are retained while noise-suppressed features from the denoised input represent the base image structures. This design follows the spirit of the decouple-and-feedback fusion [16], and employs efficient point-wise and depth-wise convolutions widely used in modern restoration networks [17]. At stage \(l\), the features from the raw and the denoised branches are first mixed and then split channel-wise into a gating tensor \(F_{gate}\) and a content tensor \(F_{con}\). The gate, after a Gaussian error linear unit (GELU) activation, modulates the content via element-wise multiplication, and the result is projected and residually combined with the features from the denoised branch. The operations at the \(l\)-th (\(l \in \{1,2,...,L\}\)) stage are described as \[\begin{align} F^{l}_{gate}, F^{l}_{con} &= \text{DConv3}(\text{PConv}(Concat[F^{l}_{raw}, F^{l}_{dn}])), \tag{3} \\ F^{l}_{fuse} &= \text{PConv}(F^{l}_{con} \odot \text{GELU}(F^{l}_{gate})) + F^{l}_{dn}, \tag{4} \end{align}\] where \(F^{l}_{raw}\) and \(F^{l}_{dn}\) are the features from the raw and the denoised encoder branches, respectively. PConv and DConv3 represent a point-wise \(1\times1\) convolution and a depth-wise convolution with a \(3 \times 3\) kernel, respectively. \(\odot\) denotes the Hadamard product and \(F^{l}_{\text{fuse}}\) is the fused output forwarded to the decoder.

1.0ex plus 0.2ex minus 0.1ex 0.5ex plus 0.1ex Loss Functions

For the DN network, we take an L1 loss between the denoised output \(\mathbf{X}^{\text{DN}}\) and the corresponding ground truth \(\mathbf{X}^{\text{GT}}\) as \[\mathcal{L}_{DN} = \frac{1}{HW} \left\|\mathbf{X}^{\text{DN}} - \mathbf{X}^{\text{GT}}\right\|_{1}. \label{eq:loss95d}\tag{5}\]

For the color DM and the polar DM networks, we employ the same loss function as used in TCPDNet [8]. In short, TCPDNet uses a sub-sampled RGB reconstruction term \(\mathcal{L}_{C}\) and a full-resolution color-polarization reconstruction term (in YCbCr) \(\mathcal{L}_{CP}^{YCbCr}\). The overall DM loss is described as \[\mathcal{L_{DM}} = \mathcal{L}_{C} + \alpha \mathcal{L}_{CP}^{YCbCr}, \label{eq:loss95total}\tag{6}\] where \(\alpha\) is fixed to \(4\), following [8]. Each term is described as \[\begin{align} \mathcal{L}_{C} &= \frac{1}{3HW} \sum_{\theta}\left\|\mathbf{C}_{\theta} - \mathbf{C}_{\theta}^{\text{GT}}\right\|_{1},\\ \mathcal{L}_{CP}^{YCbCr} &= \frac{1}{12HW} \left\|YCbYr(\tilde{\mathbf{I}}) - YCbCr(\mathbf{I}^{\text{GT}})\right\|_{1}. \end{align}\] where \(\mathbf{C}_{\theta}^{\text{GT}}\) and \(\mathbf{I}^{\text{GT}}\) are the corresponding ground truths, and \(YCbCr\) represents the transformation from the RGB space to the YCbCr space. We refer to the paper [8] for more detailed information.

1.0ex plus 0.2ex minus 0.1ex 0.5ex plus 0.1ex Training Strategy

Stage I: Separated DN and DM training. The first stage trains the DN and the DM network backbones separately. For DN, we train the DN network using the loss function of Eq. (5 ). For DM, we train the color DM and the polar DM networks using the loss function of Eq. (6 ), where we remove the dual-encoder branches and GFMs from the color DM network and use clean (noise-free) CPFA raw data as the input for DM to learn only the DM task, as shown in Fig. 3.

Stage-II: GFM training. We freeze all network parameters obtained in Stage I for the DN and the DM backbones. Then, we add our proposed dual-encoder branches and train GFMs for the DM stage using the same loss function, where we use the noisy CPFA raw data and its denoised version as the network inputs, as shown in Fig. 3.

4pt

2.0ex plus 0.2ex minus 0.1ex 1.0ex plus 0.2ex Experimental Results

1.0ex plus 0.2ex minus 0.1ex 0.5ex plus 0.1ex Dataset

We used the polarization denoising and demosaicking dataset published in [9] containing 40 scenes. Among the provided three noise-level conditions, we used the highest noise-level condition (High) to evaluate our proposed method under a severe noise condition. The dataset was split into 30 scenes for training, 5 scenes for validation, and the rest 5 scenes for testing.

1.0ex plus 0.2ex minus 0.1ex 0.5ex plus 0.1ex Compared Methods

For the joint task of polarization DM and DN, existing studies mainly focus on a single processing module instead of the complete pipeline. Thus, we compared our proposed method with the methods in three categories: (1) DM Only: This category performs only DM. We adopted one signal-processing-based DM method (IGRI-2) [4] and one deep-learning-based method (TCPDNet) [8], which is the backbone for our color DM. (2) DM-to-DN: As a straightforward pipeline, we combined IGRI-2 and BM3D [18] as a signal-processing-based method, and combined TCPDNet and the same U-Net DN architecture as ours as a deep-learning-based method, as shown in Fig. 1 (a). (3) DN-to-DM: As a signal-processing-based method, we compared the baseline method proposed in [9], which combines PFCD [12] for DN and IGRI-2 for DM. As a deep-learning-based method, we applied the pipeline shown in Fig. 1 (b), where we adopted the same U-Net architecture as ours for DN and TCPDNet for DM. To ensure a fair comparison, all deep-learning-based methods are trained on the same dataset.

1.0ex plus 0.2ex minus 0.1ex 0.5ex plus 0.1ex Results

Table 1 shows the quantitative evaluation results using peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM) [19], and mean angular error (MAE). MAE (lower is better) is exclusively used for the evaluation of angle-of-polarization (AoP), while PSNR and SSIM are applied to all remaining components, including a Stokes parameter (\(S_0\)) and degree-of-polarization (DoP). As demonstrated in Table 1, our proposed method consistently outperforms existing methods across all metrics in the joint task of DM and DN. In particular, our method achieves notable improvements in PSNR and SSIM due to its effective fusion architecture rather than relying on sequential applications of the two tasks. A visual comparison on one of the test scenes is provided in Fig. 4, where we can confirm that our proposed method produces the closest result to the ground truth (GT), especially for the AoP-DoP visualization. The signal-processing-based methods (the methods in the top row) generally produce inferior results, compared with the deep-learning-based methods (the methods in the bottom row). Importantly, our method significantly outperforms the baseline method proposed in [9] (PFCD\(\rightarrow\)IGRI-2), which is a sole framework for the joint task of DN and DM for CPFA sensors in the past literature. More visual comparisons can be seen in the supplementary material.

1.0ex plus 0.2ex minus 0.1ex 0.5ex plus 0.1ex Ablation Study

To validate the effectiveness of our proposed fusion module, we conducted an ablation study. We compared the performance of the complete model (Ours) and the version without the fusion module (Ours w/o Fusion). The results are shown in Table 2, where the same evaluation metrics as Table 1 are used. From the results, it can be seen that the fusion module plays a meaningful role in improving model performance. After removing the fusion module, there are decreases in the numerical performance in all the evaluated metrics, which demonstrates the effectiveness of our fusion module.

6pt

2.0ex plus 0.2ex minus 0.1ex 1.0ex plus 0.2ex Conclusion

In this paper, we have presented CPDDNet, a unified framework of joint DN and DM for CPFA sensors. By integrating a DN-to-DM design with GFMs, our CPDDNet preserves high-frequency structure while effectively reducing noise. Experimental results show that CPDDNet consistently improves visual quality and polarization parameter accuracy compared to existing approaches. As one of the limitations, the training of CPDDNet is currently performed under a fixed noise level. Thus, our future work includes generalizing our CPDDNet to a more diverse range of noise levels using a single model.

References↩︎