1 Introduction↩︎

Chromoendoscopy (CE) is a well-adopted clinical practice to improve the visibility of gastric lesions, such as early cancer, by spraying indigo carmine blue dye from the endoscope tip onto the gastric surface. As shown in Fig. 1, the CE image shows better lesion visibility (around the center of the image) than the standard endoscopy (SE) image, because the blue dye enhances surface structure and color contrast. While CE is effective in assisting the detection of early lesions [1], preparing and spraying the blue dye during endoscopy requires additional cost and time, which is undesirable not only for patients but also for medical practitioners. This prevents practitioners from performing CE proactively and frequently.

To overcome the above-mentioned issue, our previous studies [2]–[5] and the other study [6] proposed a virtual chromoendoscopy (V-CE) technique. As shown in Figs. 2 (a) (a) and 2 (a) (b), V-CE eliminates the necessity of physically spraying the blue dye by applying a learned image translation model, typically a cycle-consistent generative adversarial network (CycleGAN [7]), to virtually generate a CE image from an SE image. In our clinical studies, we reported that V-CE demonstrates better visibility for gastric neoplasms and early gastric cancers compared with SE [2], [4], [5]. We also confirmed the feasibility of V-CE by developing a real-time V-CE system working with a real endoscope setup [4], [5].

In this paper, we aim to further improve the lesion visibility by combining V-CE with image enhancement techniques. We first present our image enhancement method consisting of gradient enhancement and color enhancement. Figure 1 shows an example of the comparison between a CE image and an enhanced chromoendoscopy (ECE) image by our method. We can confirm that the ECE image demonstrates better visibility for the lesion existing around the center of the image. Motivated by this, we introduce virtual enhanced chromoendoscopy (V-ECE), which learns a CycleGAN model that translates an SE image to the ECE image, as illustrated in Fig. 2 (a) (c)⁵. In real clinical practice, a desired image enhancement level depends on the inspected lesion and the medical practitioner’s preference. Thus, we propose a novel image translation model named tunable CycleGAN that can generate a V-ECE image using an enhancement level tunable by a user. By this translation model, the practitioner can adjust the image enhancement level according to real endoscopy inspection. We experimentally validate that our tunable CycleGAN model can generate plausible V-ECE images with provided enhancement levels by a single unified model and provide consistent results compared with the dedicated models learned for each enhancement level.

2 Virtual Enhanced Chromoendoscopy (V-ECE)↩︎

In this section, we introduce V-ECE, which learns a translation model to convert an SE image to a V-ECE image. V-ECE consists of two steps. (i) Visibility enhancement: This step applies image enhancement techniques to CE images to generate the corresponding ECE images to improve the visibility. (ii) CycleGAN training: This step learns a CycleGAN model using the unpaired data of SE images (domain \(X\)) and the generated ECE images (domain \(Y\)). Each step is detailed in Sec. 2.1 and Sec. 2.2. Then, we further propose a tunable CycleGAN model that makes the enhancement level of V-ECE tunable by a user in the application phase, which is detailed in Sec. 2.3.

2.1 Visibility Enhancement↩︎

2.1.1 Overview↩︎

We apply two types of image enhancement: gradient and color enhancements. This is based on the following observations: (i) The human visual perception is more sensitive to pixel value differences, i.e., image gradients, than pixel values themselves. Thus, the visibility of subtle surface structures visualized by the blue dye is expected to be improved by applying the gradient enhancement. (ii) Gastric lesions typically become reddish in color, and their borders are made easier to detect by the blue dye. Thus, the visibility of the lesion is expected to be improved by applying the color contrast enhancement between reddish and bluish colors.

Figure 3 (a) shows the overview of our visibility enhancement method. It first converts a CE image from the RGB color space to the YCbCr color space [8] as

\[\begin{align} \begin{bmatrix} Y\\Cb\\Cr \end{bmatrix} &=\begin{bmatrix} 0.257&0.504&0.098\\ -0.148&-0.291&0.439\\ 0.439&-0.368&-0.071 \end{bmatrix}\begin{bmatrix} R\\G\\B \end{bmatrix}+\begin{bmatrix} 16\\128\\128 \end{bmatrix}, \label{eq:rgb2ycbcr} \end{align}\tag{1}\] where Y represents luminance, and Cb and Cr represent chrominance. We apply the gradient enhancement to the Y component and the color enhancement to the CbCr component, because image gradients and colors are encoded in Y and CbCr, respectively. After the enhancements, the image is returned to the RGB space by the reverse process of Eq. (1 ).

2.1.2 Gradient enhancement↩︎

We apply the image reconstruction method of [9]. For simplicity, we consider uniformly enhancing the gradients for every pixel and provide the target gradient to the method [9] as \[q_d(i,j)=N_g \cdot \partial_d u(i,j), \label{eq:gradient95operation}\tag{2}\] where \(u(i,j)\) is the pixel value of the pixel position \((i,j)\) and \(\partial_d\) represents a partial derivative along the direction \(d \in \{h,v\}\) to derive the image gradients in the horizontal (\(h\)) and the vertical (\(v\)) directions. \(q_d(i,j)\) is the target gradient, which is the multiplication of the original gradient by the gain factor \(N_g\). With this target gradient, the method [9] reconstructs the gradient-enhanced image through an optimization process.

2.1.3 Color enhancement↩︎

Color enhancement is designed to enhance red-blue color contrast. By rearranging Eq. (1 ), we obtain \[\begin{align} \begin{bmatrix} Y-16\\Cb-128\\Cr-128 \end{bmatrix} &=\begin{bmatrix} 0.257&0.504&0.098\\ -0.148&-0.291&0.439\\ 0.439&-0.368&-0.071 \end{bmatrix}\begin{bmatrix} R\\G\\B \end{bmatrix}. \label{eq:rgb2ycbcrw47offset} \end{align}\tag{3}\] According to this equation, Cb and Cr with the offset, i.e., \(Cb-128\) and \(Cr-128\), contain red-blue differences, i.e., \(-0.148R+0.439B\) and \(0.439R-0.071B\), respectively. Thus, increasing these differences enhances the red-blue color contrast. However, independently performing it requires two gain parameters, which prevents intuitive parameter control. Thus, we introduce a single gain model.

Firstly, the Cr component with the offset (the left side of Eq. (3 )) is simply multiplied by the color gain \(N_c\) to increase the red-blue difference as \[\begin{align} Cr'-128&= N_c \cdot (Cr-128) \label{eq:cr39w47offset}. \end{align}\tag{4}\] From this equation, we calculate the enhanced Cr as \[\begin{align} Cr'= N_c \cdot (Cr-128)+128. \label{eq:cr39} \end{align}\tag{5}\] Then, we design the enhanced Cb so that the G value among the RGB values remains constant as an anchor before and after the color enhancement as \[\begin{align} Cb'= \alpha(N_c-1)(Cr-128) + Cb, \label{eq:Cb95modify} \end{align}\tag{6}\] where \(\alpha \approx 2.08\). By doing this, the color enhancement can be performed using the single gain parameter in an interpretable manner. The derivation of the fixed constant \(\alpha\) is described in the Appendix.

Figure 3 (b) shows examples of our enhancement results. We can confirm that our enhancement method combining the gradient and the color enhancements effectively improves the visibility of the gastric surface.

2.2 CycleGAN Training↩︎

Because the endoscope or the gastric surface inevitably move during spraying the blue dye, the pixel-aligned paired data cannot be obtained for SE and CE/ECE images. Thus, we adopt CycleGAN [7], which is a representative image translation model learnable with the unpaired sets of images in two style domains. Two approaches can be considered for V-ECE image generation using CycleGAN: (i) Sequential: This approach first applies a CycleGAN model learned to translate an input SE image to a V-CE image and then applies the visibility enhancement method to generate the enhanced result. (ii) Direct: This approach learns a direct mapping from the SE image to the ECE image by a CycleGAN model. In this study, we adopt the direct approach because of two reasons: (i) It requires only a one-step process and is simpler and faster than the two-step process of the sequential approach. (ii) It can be implemented in a real clinical environment by changing the CycleGAN model parameters of our developed real-time V-CE system [4], [5]

Figure 4 shows the applied model architecture of CycleGAN. We define the domain \(X\) as SE images and the domain \(Y\) as ECE images, which are generated as explained in Sec. 2.1. The generator \(G_{X \rightarrow Y}\) translates the domain from \(X\) to \(Y\) and vice versa for the generator \(G_{Y \rightarrow X}\). The discriminators \(D_X\) and \(D_Y\) discriminate real samples and generated samples in each domain.

The loss functions for the generators and the discriminators are described as \[\begin{align} \min_{G_{X \rightarrow Y}, G_{Y \rightarrow X}} &\mathcal{L}^\text{gen}_{\text{GAN}}+\lambda_\text{cyc}\mathcal{L}_{\text{cyc}}+\lambda_\text{identity}\mathcal{L}_{\text{identity}}, \tag{7} \\ \min_{D_X, D_Y} \; &\mathcal{L}^\text{disc}_{\text{GAN}}, \tag{8} \end{align}\] where we follow the original CycleGAN implementation by the authors [7] with least squares generative adversarial networks [10].

The adversarial loss \(\mathcal{L_\text{GAN}}\) is designed so that the generator tries to generate images that fool the discriminator, while the discriminator tries to discriminate them from real images. The loss functions for the pair of the generator \(G_{X \rightarrow Y}\) and the discriminator \(D_Y\) are as follows. \[\begin{align} \mathcal{L}^\text{gen}_{\text{GAN}} &= \mathbb{E}_{x\sim p_{\text{data}}(x)}[(D_Y(G_{X \rightarrow Y}(x))-1)^2],\\ \mathcal{L}^\text{disc}_{\text{GAN}} &= \mathbb{E}_{x\sim p_{\text{data}}(x)}[D_Y(G_{X \rightarrow Y}(x))^2] \\ &\quad + \mathbb{E}_{y\sim p_{\text{data}}(y)}[(D_Y(y)-1)^2], \end{align} \label{eq:GANloss}\tag{9}\] where \(x\in X\) and \(y\in Y\) represent a sample from each domain, and \(p_{data}(x)\) and \(p_{data}(y)\) are probability distributions of them, respectively. By jointly learning the generator and the discriminator using these adversarial losses, the generator is encouraged to generate a virtual image close to a real image. The loss functions for the pair of \(G_{Y \rightarrow X}\) and the discriminator \(D_X\) are defined in the same manner.

The cycle-consistency loss \(\mathcal{L}_{\text{cyc}}\) is designed so that the translation from \(X\) to \(Y\) and the reverse translation from \(Y\) to \(X\) should be cyclic. The loss function is as follows. \[\begin{align} \mathcal{L}_{\text{cyc}} &= \mathbb{E}_{x\sim p_{\text{data}}(x)}[\|G_{Y \rightarrow X}(G_{X \rightarrow Y}(x))-x\|_1] \\ &\quad + \mathbb{E}_{y\sim p_{\text{data}}(y)}[\|G_{X \rightarrow Y}(G_{Y \rightarrow X}(y))-y\|_1]. \end{align} \label{eq:cycloss}\tag{10}\] Through this loss, CycleGAN achieves unsupervised learning and eliminates the need for paired data between the domains \(X\) and \(Y\).

The identity loss \(\mathcal{L_\text{identity}}\) is introduced to prevent excessive or unnecessary transformations as \[\begin{align} \mathcal{L}_{\text{identity}} &= \mathbb{E}_{x\sim p_{\text{data}}(x)}[\|G_{Y \rightarrow X}(x)-x\|_1] \\ &\quad +\mathbb{E}_{y\sim p_{\text{data}}(y)}[\|G_{X \rightarrow Y}(y)-y\|_1], \end{align}\] which regularizes the network training.

2.3 Tunable CycleGAN Model↩︎

We here propose a tunable CycleGAN model that makes the enhancement level, i.e., the gradient and color gains, tunable by a user in the application phase. Figure 5 shows the model architecture, where we apply simple yet effective modifications to the original CycleGAN. To prepare the ECE image in the domain \(Y\), we randomly select the gradient and the color gains \((N_g, N_c)\) from practical ranges for each gain. The selected gain pair is used to enhance a real CE image, which is sampled from the training dataset, by the visibility enhancement method in Sec. 2.1. Inspired by the study of [11], which provides a noise level map as the network input to achieve flexible denoising capability, we construct two-channel gain maps (constant-value images of the same size as the RGB image) and concatenate them to the original three-channel RGB image to input to the generator \(G_{X \rightarrow Y}\) and the discriminator \(D_Y\). With this architecture, the generator tries to generate V-ECE images with the corresponding gains, while the discriminator tries to discriminate them from real ECE images enhanced using the same gains. Here, the role of the discriminator is to identify not only whether the ECE image is real or not, but also whether the enhancement level is correct or not, directing the generator to generate the V-ECE images with the correct enhancement level. For the reverse translation, the gain maps are fed as the inputs only for the generator \(G_{Y \rightarrow X}\), because the role of the discriminator \(D_{X}\) is to identify whether the SE image is real or not, which is irrelevant to the enhancement level. The loss functions for the generators and the discriminators are the same as Eqs. (7 ) and (8 ). In the application phase, we simply input the gains (gain maps) provided by the user to the generator \(G_{X \rightarrow Y}\) to generate the V-ECE images with the corresponding enhancement level.

3 Experimental Results↩︎

3.1 Ethics↩︎

This study was conducted in accordance with the Declaration of Helsinki. The Institutional Review Board at International University of Health and Welfare approved the study protocol and the data collection on June 25, 2024 (Approval No. 24-Ic-002). Informed consent was obtained from patients before collecting endoscopic images. This study was also approved by the Research Ethics Committee of Institute of Science Tokyo, on July 25, 2024 (Approval No. 2024150) to process the collected data.

3.2 Datasets↩︎

We constructed two datasets, one for Fujifilm endoscopes and the other for Olympus endoscopes. We collected endoscope images capturing or inspecting gastric lesions, including early cancers, using the endoscopes of each manufacturer. The Fujifilm dataset consists of 1,063 SE images as the domain \(X\) and 1,216 CE images, from which ECE images are generated as the domain \(Y\). For the evaluation of the V-ECE image generation, we randomly selected 15 SE images as the input for testing. Similarly, the Olympus dataset consists of 1,064 SE images and 818 CE images. For testing, we randomly selected 14 SE images.

3.3 Implementation Details↩︎

We conducted two types of experiments to evaluate our tunable CycleGAN model: (i) V-ECE image generation with only gradient enhancement (one-parameter model), and (ii) V-ECE image generation with both gradient and color enhancements (two-parameter model). For the one-parameter model, the color gain \(N_c\) was fixed to 1.2 and the gradient gain \(N_g\) was randomly sampled from the discrete gain set of \(\left[1.0, 2.0, \cdots, 9.0\right]\). For the two-parameter model, the color gain \(N_c\) was also randomly sampled from the discrete gain set of \(\left[1.0, 1.1, \cdots, 1.8\right]\). To construct the gain maps, we normalized each gain range from \([1.0,9.0]\) or \([1.0,1.8]\) to \([0,255]\) by applying a linear transformation, so that the generator and the discriminator do not neglect the small differences of the original values. For each experiment, we separately learned our tunable CycleGAN models for the Fujifilm and the Olympus datasets, considering that the internal image processing pipeline of the endoscope system depends on each manufacturer.

Regarding the training settings, we followed the default settings of the original CycleGAN implementation. The batch size was set to 1, and the number of epochs was set to 200. The learning rate was 0.0002 for the first 100 epochs and linearly decayed from 0.0002 to 0 during the remaining 100 epochs. The loss weights were set as \(\lambda_\text{cyc} = 10\) and \(\lambda_\text{identity} = 5\). The training was conducted using a single NVIDIA H100 GPU and took approximately two days.

3.4 Results of One-Parameter Tunable Model↩︎

We first evaluate the results of the one-parameter model. In addition to a visual evaluation, we perform a numerical evaluation using this model, because it is simpler than the two-parameter model, making the numerical results more interpretable.

Figure 6 shows the translation results of our tunable CycleGAN model for test SE images in the Fujifilm (top) and the Olympus (bottom) datasets. We can confirm that the enhancement level is gradually increased according to the input gains at inference time, despite the ECE images being generated from a single model.

We then perform a numerical evaluation. To assess the validity of our tunable CycleGAN model, we evaluate the consistency between our single tunable CycleGAN model and independent CycleGAN models trained for each gain (fixed for all training images). Figure 7 shows the results using the 15 test SE images of the Fujifilm dataset, where the vertical axis represents the boxplot of the gradient magnitude, which is calculated as \[q_m(i,j) = \sqrt{(\partial_hu(i,j))^2+(\partial_vu(i,j))^2}. \label{eq:gradient95calc}\tag{11}\] Here, \(u(i,j)\) is the pixel value of the generated V-ECE image at pixel \((i,j)\), and (\(\partial_h\),\(\partial_v\)) are partial derivative operators in the horizontal and the vertical directions, respectively. In Fig. 7, V-ECE represents the results of the independent CycleGAN models for each gain, presenting the reference results for the gradient magnitudes. V-ECE-T represents the results of our tunable CycleGAN model. We can confirm that our tunable CycleGAN can generate consistent results with the independent CycleGAN models, which validates the reasonableness of our results. As an ablation study, we also evaluate a tunable model, where the gain maps are inputted only for the generator, but not for the discriminator (V-ECE-T w/o gain input for \(D\)). This model generates a very diverse range of gradient magnitudes and significantly inconsistent results with the references (V-ECE), highlighting the importance of the gain map inputs to the discriminator.

3.5 Results of Two-Parameter Tunable Model↩︎

Figure 8 shows the visual results of our tunable CycleGAN with the two-parameter model for one of the test SE images in the Fujifilm dataset. We can confirm that the strengths of the image gradients are gradually increased along the horizontal axis, while the levels of the color enhancements are gradually increased along the vertical axis. These results validate the effectiveness of our tunable CycleGAN, allowing us to jointly tune the gradient and color gains.

More results can be seen on our project page⁶.

4 Conclusion↩︎

In this paper, we have proposed V-ECE with a tunable CycleGAN model. Compared with the existing V-CE, our V-ECE adds the following values: (i) It generates enhanced images with improved visibility through gradient and color enhancements. (ii) Its enhancement level is tunable by a medical practitioner in the application phase. These factors improve the practicability of the virtual system for CE. Our tunable CycleGAN can be implemented in real time by replacing the model parameters of our current V-CE system [4], [5] with those of the tunable CycleGAN. In future work, we plan to assess our V-ECE system from clinical aspects in real endoscopy environments.

Appendix↩︎

This appendix explains the derivation of the fixed value \(\alpha\) in Eq. (6 ). The conversion from YCbCr to RGB is described as \[\begin{align} \begin{bmatrix} R\\G\\B \end{bmatrix} =T\begin{bmatrix} Y-16\\Cb-128\\Cr-128 \end{bmatrix}, \label{eq:ycbcr2rgb} \end{align}\tag{12}\] where \(T\) is the conversion matrix [8] as below.

\[T = \begin{bmatrix} 1.164&0&1.596\\ 1.164&-0.391&-0.813\\ 1.164&2.018&0 \end{bmatrix}.\] We let \([G, Cb, Cr]\) be the original pixel values before the color enhancement and \([G', Cb', Cr']\) be the pixel values after the enhancement. Then, the change of the G value, \(\Delta G = G' -G\), is described as \[\begin{align} \Delta G &= T_{22}\Delta Cb + T_{23}\Delta Cr, \end{align}\] where \(T_{nm}\) is the element of the matrix \(T\) at \(n\)-th row and \(m\)-th column, and \(\Delta Cb = Cb' -Cb\) and \(\Delta Cr = Cr' -Cr\), respectively. When we impose the condition that the G value is constant before and after the color enhancement, i.e., \(\Delta G = 0\), we obtain the following equations. \[\begin{align} 0 &= T_{22}\Delta Cb + T_{23}\Delta Cr,\\ \Delta Cb &= -T_{23}/T_{22}\Delta Cr. \label{eq:delta95cb} \end{align}\tag{13}\] Because we design \(Cr' = N_c \cdot (Cr - 128) +128\) as in Eq. (5 ), \(\Delta Cr\) is described as \[\begin{align} \Delta Cr&=Cr'-Cr\\ &=N_c(Cr-128)+128-Cr\\ &=(N_c-1)(Cr-128). \end{align} \label{eq:delta95cr}\tag{14}\] According to Eqs. (13 ) and (14 ), we finally obtain \(Cb'\) as \[\begin{align} \Delta Cb &= -T_{23}/T_{22}(N_c-1)(Cr-128), \\ Cb'&= \alpha(N_c-1)(Cr-128)+Cb, \end{align}\] where \(\alpha = -T_{23}/T_{22} \approx 2.08\) is a fixed constant.

References↩︎