3.1. Network architecture

The proposed 3D CycleGAN method, an extension of the 2D CycleGAN architecture⁽Reference Zhu, Park, Isola and Efros³³⁾, employs 3D convolutions to simultaneously extract the spatial and depth information inherent in image stacks. Given an OCT domain $ X $ and a Confocal domain $ Y $ , the aim of our model is to extract statistic information from both $ X $ and $ Y $ and then learn a mapping $ G:X\to Y $ such that the output $ \hat{y}\hskip2pt =\hskip2pt G(x) $ , where $ x\hskip2pt \in \hskip2pt X $ and $ y\hskip2pt \in \hskip2pt Y $ . An additional mapping $ F $ transfers the estimated Confocal $ \hat{y} $ back to the OCT domain $ X $ . The framework comprises two generators and two discriminators to map $ X $ to $ Y $ and vice versa. The input images are processed as 3D stacks, and all learnable kernels within the network are three-dimensional, as depicted in Figure 1.

Figure 1. The proposed OCT-to-Confocal image translation method is based on 3D CycleGAN.

3.2. Generators and discriminators of 3D CycleGAN

3.3. The loss function

The objective consists of four terms: (1) adversarial losses⁽Reference Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville and Bengio³⁰⁾ for matching the distribution of generated images to the data distribution in the target domain, (2) a cycle consistency loss to prevent the learned mappings $ G $ and $ F $ from contradicting each other, (3) an identity loss to ensure that if an image from a given domain is transformed to the same domain, it remains unchanged, and (4) the gradient loss to enhance the textural and edge consistency in the translated images

1) Adversarial loss: In our model, the adversarial loss is based on the binary cross-entropy (BCE) loss, as used in traditional GANs⁽Reference Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville and Bengio³⁰⁾. It adapts the style of the source domain to match the target by encouraging the generators to produce outputs that are indistinguishable from the target domain images and is defined as follows:

(1) $$ {\displaystyle \begin{array}{r}{\mathcal{L}}_{\mathrm{Adv}}(G,{D}_Y)={\unicode{x1D53C}}_{y\sim {p}_{\mathrm{data}}(y)}[-\mathrm{log}{D}_Y(y)]+{\unicode{x1D53C}}_{x\sim {p}_{\mathrm{data}}(x)}[-\mathrm{log}(1 - {D}_Y(G(x)))],\end{array}} $$

2) Cycle consistency loss: Cycle consistency loss⁽Reference Zhou, Krahenbuhl, Aubry, Huang and Efros⁵²⁾, defined in Equation (2), ensures the network learns to accurately translate an image $ x $ from domain $ X $ to domain $ Y $ and back to $ X $ via mappings $ G $ and $ F $ (forward cycle) and vice versa for an image $ y $ (backward cycle), preserving the original image’s integrity.

(2) $$ {\displaystyle \begin{array}{r}{\mathcal{L}}_{\mathrm{cyc}}(G,F)={\unicode{x1D53C}}_{x\sim {p}_{\mathrm{data}}(x)}[\Vert F(G(x))-x{\Vert}_1]+{\unicode{x1D53C}}_{y\sim {p}_{\mathrm{data}}(y)}[\Vert G(F(y))-y{\Vert}_1].\end{array}} $$

The $ {L}_1 $ loss between the original and translated backed image minimizes information loss, ensuring that the transformed image retains essential details and the core content of the input image.

3) Identity loss: It was shown in⁽Reference Taigman, Polyak and Wolf⁶¹⁾ that adding identity losses can enhance the performance of the CycleGAN by preserving color consistency and other low-level information between the input and output, defined as follows:

(3) $$ {\displaystyle \begin{array}{cc}{\mathcal{L}}_{\mathrm{id}}(G,\,F)=& \hskip-3pt {\unicode{x1D53C}}_{x\sim {p}_{\mathrm{data}}(x)}[\Vert F(x)-x{\Vert}_1]+{\unicode{x1D53C}}_{y\sim p\mathrm{data}(y)}[\Vert G(y)-y{\Vert}_1].\end{array}} $$

The identity loss is calculated by taking the $ {L}_1 $ norm of the difference between a source domain image and its output after being passed through the generator designed for the opposite domain. For instance, if an OCT image is fed into a generator trained to translate confocal images to OCT images (opposite domain), the generator should ideally return the original OCT images unchanged. This process helps maintain consistent color and texture and indirectly stabilizes style fidelity.

4) Gradient loss: The gradient loss promotes textural fidelity and edge sharpness by minimizing the $ {L}_1 $ norm difference between the gradients of real and synthesized images⁽Reference Sun, Fan and Li⁵⁷⁾, thereby preserving detail clarity and supporting both style rendering and information preservation through the enhancement of smooth transitions and the maintenance of edge details. The gradient loss is defined as follows:

(4) $$ {\displaystyle \begin{array}{cc}{\mathcal{L}}_{\mathrm{GL}}(G,\ F)=& \hskip-3pt {\unicode{x1D53C}}_{x\sim {p}_{\mathrm{data}}(x)}[\Vert \mathrm{\nabla}G(x)-\mathrm{\nabla}y{\Vert}_1]+{\unicode{x1D53C}}_{y\sim {p}_{\mathrm{data}}(y)}[\Vert \mathrm{\nabla}F(y)-\mathrm{\nabla}x{\Vert}_1],\end{array}} $$

The total objective loss to minimize is the weighted summation of the four losses: the adversarial, the cyclic, the identity, and the gradient, given as follows:

(5) $$ {\displaystyle \begin{array}{cc}{\mathcal{L}}_{\mathrm{total}}=& \hskip-3pt {\mathcal{L}}_{\mathrm{Adv}}(G,\ {D}_Y)+{\mathcal{L}}_{\mathrm{Adv}}(F,{D}_X)+{\lambda}_1{\mathcal{L}}_{\mathrm{cyc}}+{\lambda}_2{\mathcal{L}}_{\mathrm{id}}+{\lambda}_3{\mathcal{L}}_{\mathrm{GL}}\end{array}} $$

where $ {\lambda}_1 $ , $ {\lambda}_2 $ and $ {\lambda}_3 $ are hyperparameters.

5.1. Dataset augmentation

Our dataset expansion employed horizontal flipping, random zooming (0.9–1.1 scale), and random cropping, which aligns with common augmentation practices in retinal imaging⁽Reference Goceri⁶⁸⁾. Horizontal flipping is justified by the inherent bilateral symmetry of the ocular anatomy, allowing for clinically relevant image transformations. Random zoom introduces a controlled variability in feature size, reflecting physiologic patient diversity encountered in clinical practice. Random cropping introduces translational variance and acts as a regularization technique, mitigating the risk of the model overfitting to the borders of training images. These augmentation strategies were specifically chosen to avoid the introduction of non-physiological distortions that could potentially affect clinical diagnosis.

5.2. Implementation details

The implementation was conducted in Python with the PyTorch library. Training and evaluation took place on the BluePebble supercomputer⁽⁶⁹⁾ at the University of Bristol, featuring Nvidia V100 GPUs with 32 GB RAM, and a local workstation with RTX 3090 GPUs.

For our experiments, the OCT image cubes processed by the generator $ G $ were sized $ 512\times 512\times 9\times {C}_1 $ , with $ {C}_1=1 $ indicating grayscale images. Similarly, the confocal images handled by the generator $ F $ had dimensions $ 512\times 512\times 9\times {C}_2 $ , where $ {C}_2=3 $ represents the RGB color channels.

Optimization utilized the Adam optimizer⁽Reference Kingma and Ba⁷⁰⁾ with a batch size of 1, and a momentum term of 0.5. The initial learning rate was set at $ 2\times {10}^{-5} $ , with an input depth of 9 slices. Loss functions were configured with $ {\lambda}_1=8 $ , $ {\lambda}_2=0.1 $ , and $ {\lambda}_3=0.1 $ . The 400-epoch training protocol maintained the initial learning rate for the first 200 epochs, then transitioned to a linear decay to zero over the next 200 epochs. Weights were initialized from a Gaussian distribution $ \mathcal{N}\left(\mathrm{0,0.02}\right) $ , and model parameters were finalized at epoch 300 based on FID and KID performance.

5.3. Evaluation methods

5.3.1. Quantitative evaluation

The quantitative evaluation of image translation quality is conducted employing Distribution-Based (DB) objective metrics⁽Reference Rodrigues, Lévêque and Gutiérrez⁷¹⁾ due to their ability to gauge image quality without necessitating a reference image. Specifically, the Fréchet inception distance (FID)⁽Reference Heusel, Ramsauer, Unterthiner, Nessler and Hochreiter⁷²⁾ and KID scores⁽Reference Bińkowski, Sutherland, Arbel and Gretton⁷³⁾ were utilized.

These metrics are distribution-based, comparing the statistical distribution of generated images to that of real images in the target domain. Their widespread adoption in GAN evaluations underscores their effectiveness in reflecting perceptual image quality. FID focuses on matching the exact distribution of real images using the mean and covariance of features, which can be important for capturing the precise details in medical images and the correct anatomical structures with the appropriate textures and patterns. KID, on the other hand, emphasizes the diversity and general quality of the generated images without being overly sensitive to outliers ensuring that the generated images are diverse and cover the range of variations seen in real medical images. Lower FID and KID scores correlate with higher image fidelity.

1. a) FID ⁽ Reference Heusel, Ramsauer, Unterthiner, Nessler and Hochreiter^{72 )} is calculated as follows:

(6) $$ {\displaystyle \begin{array}{r}FID(r,g)=\Vert {\mu}_r-{\mu}_g{\Vert}_2^2+\mathrm{T}\mathrm{r}({\Sigma}_r+{\Sigma}_g-2({\Sigma}_r{\Sigma}_g{)}^{\frac{1}{2}}),\end{array}} $$

where $ {\mu}_r $ and $ {\mu}_g $ are the feature-wise mean of the real and generated images, respectively, derived from the feature vector set of the real image collection as obtained from the output of the Inception Net-V3⁽Reference Szegedy, Vanhoucke, Ioffe, Shlens and Wojna⁷⁴⁾. Correspondingly, $ {\Sigma}_r $ and $ {\Sigma}_g $ are the covariance matrices for the real and generated images from the same feature vector set. $ \mathrm{Tr} $ denotes the trace of a matrix, and $ \parallel \cdot {\parallel}_2 $ denotes the $ {L}_2 $ norm. A lower FID value implies a closer match between the generated distribution and the real image distribution. Specifically in this study, the higher-dimensional feature vector sets characterized by 768-dimensional (FID768) and 2048-dimensional (FID2048) vectors are utilized as they capture higher-level perceptual and semantic information, which is more abstract and complex compared to the direct pixel comparison done by lower-dimensional feature spaces. These higher-dimensional features are likely to include important biomarkers and tissue characteristics critical for accurate image translation.

1. b) KID ⁽ Reference Bińkowski, Sutherland, Arbel and Gretton^{73 )} is calculated using the maximum mean discrepancy (MMD) with a polynomial kernel, as follows:

(7) $$ {\displaystyle \begin{array}{l} KID\left(r,g\right)=\frac{1}{m\left(m-1\right)}\sum \limits_{i\ne j}k\left({x}_i^r,{x}_j^r\right)+\frac{1}{n\left(n-1\right)}\sum \limits_{i\ne j}k\left({x}_i^g,{x}_j^g\right)-\frac{2}{mn}\sum \limits_{i,j}k\left({x}_i^r,{x}_j^g\right)\end{array}} $$

where $ m $ and $ n $ are the numbers of real and generated images, respectively, $ {x}_i^r $ and $ {x}_j^g $ are the feature vectors of the real and generated images, respectively, and $ k\left(x,y\right) $ is the polynomial kernel function.

5.3.2. Qualitative evaluation

The current objective metrics have been designed for natural images, limiting their performance when applied to medical imaging. Therefore, a subjective test leveraging a remote, crowd-based assessment was conducted for qualitative evaluation. This approach, contrasted with lab-based assessments, involved distributing the images to participants rather than hosting them in a controlled laboratory environment. The evaluation compared image-to-image translation results from five different methods: the UNSB diffusion model⁽Reference Kim, Kwon, Kim and Ye⁴³⁾, 2D CycleGAN, and three variations of the proposed 3D CycleGAN approach. This evaluation involved a panel of experts comprising five ophthalmologists and five individuals specializing in medical image processing. Participants were tasked with evaluating and ranking five images in their relative quality score for 13 sets of images. The five images in each set are resulted from the translation process of retinal OCT to confocal image translation. To mitigate sequence bias, the order of images within each set was randomized. Scores collected from the subjective testing were quantified and expressed as a mean opinion score (MOS), which ranges from 1 to 100. Higher MOS values denote translations of greater authenticity and perceived quality. The evaluation was structured as follows:

1. 1) Initial familiarization: The first three image sets included an original OCT image alongside its corresponding authentic confocal image and five translated confocal images from different methods and models, referred to as the with reference (W Ref) group. These were provided to acquaint the participants with the defining features of confocal-style imagery.

2. 2) Blind evaluation: The subsequent ten sets presented only the original OCT and five translated confocal images, omitting any genuine confocal references to ensure an unbiased assessment of the translation quality, referred to as the without reference (W/O Ref) group.

The participants were instructed to rank the images based on the following criteria:

• • Authenticity: The degree to which the translated image replicates the appearance of a real confocal image.

• • Color code preservation: Participants were advised to focus on the accuracy of color representation indicative of high-fidelity translation which are: (i) The presence of green and red color to represent different cell types, with blue indicating cell nuclei, (ii) The delineation of vessels by red, with green typically enclosed within these regions, (iii) The alternation of green in vessels, where a green vessel is usually adjacent to non-green vessels, and (iv) The co-occurrence of red and green regions with blue elements.

• • Overall aesthetic: The visual appeal of the image as a whole was also considered.

• • Artifact exclusion: Any artifacts that do not impact the justification of overall image content should be overlooked.

Additionally, to substantiate the reliability of selected metrics(FID768, FID2048, and KID) for evaluating OCT to confocal image translations against MOS, Spearman’s rank-order correlation coefficient (SROCC) and linear correlation coefficient (LCC) were applied to both selected DB metrics and a range of no-reference (NR) metrics, including FID64, FID192, FID768, FID2048, KID, NIQE⁽Reference Mittal, Soundararajan and Bovik⁷⁵⁾, NIQE_M (a modified NIQE version trained specifically with parameters from the original confocal image dataset), and BRISQUE⁽Reference Mittal, Moorthy and Bovik⁷⁶⁾. Both SROCC and LCC range from $ -1 $ to $ +1 $ , where $ +1 $ indicates a perfect positive correlation, 0 denotes no correlation, and $ -1 $ signifies a perfect negative correlation. These analyses correlate the metrics with the MOS to assess the consistency and predictive accuracy of FID and KID in reflecting subjective image quality assessments.

From Table 1, the negative correlation of FID and KID metrics with MOS, as indicated by their SROCC values, aligns with the expectation for lower-the-better metrics. Notably, KID demonstrates the strongest negative correlation (−0.8271), closely followed by FID2048 (−0.7823) and FID768 (−0.7666), suggesting their effectiveness in reflecting perceived image quality. Conversely, NIQE’s positive correlation (0.6346) contradicts this principle, questioning its suitability, while the modified NIQE_M shows some improvement with a negative correlation (−0.5416). BRISQUE’s low positive correlation (0.032) indicates a nearly negligible relationship with MOS. LCC results reinforce these findings, particularly highlighting KID’s superior correlation (−0.8099). These analyses collectively suggest that KID, FID768, and FID2048 are relatively the most reliable metrics for evaluating the quality of translated Confocal images in this context, while the results for NIQE and BRISQUE imply limited applicability.

Table 1. Correlation of selected DB and NR image quality metrics with MOS

Note: FID and KID metrics assess image quality, with lower values indicating better quality. NIQE and BRISQUE are no-reference image quality evaluators; lower NIQE scores suggest better perceptual quality, whereas BRISQUE evaluates image naturalness. SROCC and LCC measure the correlation between objective metrics and subjective MOS ratings. SROCC and LCC values closer to −1 or 1 indicate a strong correlation, with positive values suggesting a direct relationship and negative values an inverse relationship.

6.1. Ablation study

6.1.1. Generator architecture

In our experiments, the structure of the generator is found to have the most significant impact on the generated results, overshadowing other factors such as the hyperparameters of gradient loss and identity loss. From Table 2, the ResNet 9 configuration emerges as the most effective structure, outperforming both U-Net and WGAN-GP models. The ResNet 9’s lower FID scores suggest a superior ability to produce high-quality images that more closely resemble the target confocal domain. While WGAN-GP attains the lowest KID score, visual assessment in Figure 4 shows that it still produces significant artifacts, underscoring the limitation of WGAN-GP and the limitation of FID and KID metrics assessing image quality in medical imaging contexts. On the other hand, the U-Net architecture, although commonly used for medical image segmentation, falls short in this generative task, particularly in preserving the definition and complex anatomical structures such as blood vessels and positions of the optic disc (where blood vessels converge), as shown in the second column of Figure 4. Meanwhile, the ResNet 9 maintains spatial consistency and detail fidelity, ensuring that synthesized images better preserve critical anatomical features, which is paramount in medical diagnostics.

Table 2. Comparative results of a different generator architecture in 3D CycleGAN-3. The table presents FID768, FID2048, and KID scores for U-Net, WGAN-GP, and ResNet 9 generators. Lower scores indicate better performance, with the best result colored in red

Note: FID768 and FID2048 refer to the Fréchet Inception Distance computed with 768 and 2048 features, respectively. KID refers to the Kernel Inception Distance. Both FID and KID indicate better performance with lower scores.

Figure 4. Visual comparison of translated images using different generator architectures. This figure displays the translated confocal images using U-Net, WGAN-GP, and ResNet 9 architectures.

6.1.2. Impact of gradient and Identity loss hyperparameters

In our evaluation of the impact of identity loss ( $ {\lambda}_2 $ ) and gradient loss ( $ {\lambda}_3 $ ), we explore a range of values: $ {\lambda}_2 $ at 0, 0.1, 0.5, and 1.5, and $ {\lambda}_3 $ at 0, 0.1, 0.3, and 1.0. The line graphs in Figure 5 illustrate how these values affect the FID and KID scores, with the optimal balance achieved at 0.1 for both parameters, where the fidelity, the textural, and edge details from the original OCT domain the target confocal domain are balanced.

Figure 5. Impact of Gradient and Identity Loss Hyperparameters $ {\lambda}_2 $ and $ {\lambda}_3 $ on FID and KID. The lowest (optimal) score is highlighted in red.

We observe that the absence of identity loss ( $ {\lambda}_2=0 $ ), as visualized in Figure 6, sometimes results in color misrepresentation in the translated images, such as pervasive blue or absent green hues, underscoring its role in maintaining accurate color distribution. In contrast, overemphasizing identity loss ( $ {\lambda}_2=1.0 $ ) could lead to the over-representation of specific colors, raising the likelihood of artifacts.

Figure 6. Visual comparison of translated confocal images with different $ {\lambda}_2 $ and $ {\lambda}_3 $ values against the optimized setting.

Similarly, without the gradient loss ( $ {\lambda}_3=0 $ ), as shown in Figure 6, some images exhibit a loss of detail, particularly blurring the delineation of cellular and vascular boundaries. Conversely, an excessive gradient loss ( $ {\lambda}_3=1.0 $ ) overemphasizes minor vessels in the background and over-sharpens structures, occasionally distorting primary vessels.

In conclusion, the identity loss and the gradient loss are 2 losses with small but significant weights that help the model to focus on important essential features without causing an overemphasis that could detract from the overall image quality for OCT-to-confocal image translation.

6.1.3. Impact of the input number of slices of OCT and confocal images

In our assessment of the 3D CycleGAN model’s performance with different numbers of input slices (depth) for OCT and confocal images, we experimented with 5, 7, 9, and 11 slices. Due to the limited correlation between FID and KID metrics with visual quality across different slice counts, we primarily relied on visual assessments, as detailed in Figure 7.

Figure 7. Visual comparison of translated images with varying input slice depths (5, 7, 9, 11 slices). This figure demonstrates the impact of different slice depths on the quality of image translation by the 3D CycleGAN-3 model.

Our findings indicate that at a depth of 5 slices, the model frequently exhibited repetitive artifacts and blocky textures, struggling to accurately map the color distribution from confocal to OCT images, which resulted in spatial inconsistencies and shadowing effects on blood vessels. Increasing the slice count to 7 improved color code preservation, yet issues with background shadowing remained, likely due to persisting spatial discrepancies. The optimal outcome is achieved with 9 slices, which effectively represented cell color distribution and maintained edge details, with minimal artifacts confined to less critical areas such as image borders. Although 11 slices theoretically should provide further improvements, it did not significantly outperform the 9 slice input and sometimes introduced central image artifacts. Considering computational efficiency and image quality, an input depth of 9 slices is selected as the standard for our model.

6.2. Quantitative evaluation

In Table 3, we present both the DB image quality assessment results and subjective scores of the 13 selected OCT images set used in the subjective test. Across all DB metrics, the 3D CycleGAN-3 model outperformed other methods, achieving the lowest FID and KID scores in all scenarios (with reference, without reference, and total dataset). These results suggest that this model is most effective in aligning the statistical distribution of generated images with those of real images, indicating higher image fidelity and better perceptual quality. The 3D CycleGAN-2 model follows as the second best, performing notably well in the with-reference scenario. This suggests that the additional complexity of a third downsampling layer in 3D CycleGAN-3 does confer an advantage. An inference is that an extra downsampling layer in a 3D convolutional network improves feature extraction by broadening the receptive field, enabling the model to better discern and synthesize the key structural elements within volumetric medical images. Overall, the 3D CycleGAN models outperform the UNSB diffusion model and the 2D CycleGAN, demonstrating the inadequacy of the diffusion model-based UNSB for translating OCT to confocal images and illustrating the limitations of 2D models when dealing with volumetric data.

Table 3. The performance of models was evaluated by DB metrics FID scores and KID scores, alongside the subjective MOS rating. The results are referred to categories with reference (’W Ref’), without reference (’W/O Ref’), and total (‘Total’) image sets. For each column, the best result is colored in red and the second best is colored in blue

Note: FID768 and FID2048 refer to the Fréchet Inception Distance computed with 768 and 2048 features, respectively. KID refers to the Kernel Inception Distance. Both FID and KID indicate better performance with lower scores. Mean Opinion Score (MOS) rates the subjective quality of images with higher scores reflecting better quality.

6.3. Qualitative evaluation

The 3D CycleGAN-3 model, as shown in Table 3, scored the highest in the MOS across all three scenarios as determined by the expert panel’s rankings. This reflects the model’s superior performance in terms of authenticity, detail preservation, and overall aesthetic quality. Notably, it also minimizes the presence of non-impactful artifacts, which is critical for the utility of translated images in clinical settings.

Subjective test. Analysis based on MOS and visual observations from Figure 8 and Figure 9 indicates that all 3D CycleGAN models effectively preserve blood vessel clarity, shape, and color code. The 3D CycleGAN-3 model, which received the highest MOS ratings in all scenarios, is reported to reflect the capacity for retaining more background detail and overall authenticity. Particularly in translating lower-quality in vivo OCT images (e.g., Set 6 in Figure 9), the 3D CycleGAN-3 model demonstrates superior performance, highlighting its effectiveness in capturing the complex relationships between OCT and confocal domains.

Figure 8. Visual comparative translation results with reference.

Figure 9. Visual comparative translation results without reference.

In contrast, the 2D models (2D CycleGAN and UNSB) sometimes introduce random colors, disregard edges, and inaccurately replicate retinal vessel color patterns. The UNSB, as a diffusion model, theoretically can generate diverse outputs. However, as indicated by its lower MOS scores and observed in Figures 8 and 9, it struggles significantly with preserving accurate color codes and structural details, leading to reduced visual quality and clinical usability in OCT to confocal translation. Conversely, CycleGAN-based models employ adversarial training to directly learn the transformation of input images into the target domain. This method is better at maintaining continuity in image quality and structure, providing visual advantages over the UNSB model. However, when compared to 3D models, these advantages diminish.

Specifically, when compared to the 3D CycleGAN-3, reconstructions from the 2D CycleGAN exhibit significant issues: assembling 2D-processed images back into 3D often results in discontinuities in blood vessels and features across slices (z-direction). It manifests as repeated artifacts and features at various locations across different slices (xy-plane) and duplicated structures in 2D projections. As observed in Figure 8 (Set 2) and Figure 9 (Sets 6 and 11), the 2D CycleGAN results in more visible hallucinations than 3D CycleGAN-3 in the 2D projection images, where green artifacts occur at the optic disc (the convergence point for blood vessels). Moreover, numerous fine, hallucinated vascular structures appear in the background beyond the main vascular structures, which are absent in both the original OCT and confocal images, underscoring the limitations of 2D CycleGAN in handling the complexity of 3D data structures and maintaining spatial consistency.

Figure 10 presents a boxplot of MOS for the five evaluated methods, where 3D CycleGAN models outperform 2D models in translating OCT to confocal images. Specifically, the 3D CycleGAN-3 exhibits a more concentrated distribution of scores in MOS, indicating a consensus among experts on the quality of the generated confocal images by this model, underlining its proficiency in producing consistent and reliable translations. The statistical analysis conducted via Kruskal–Wallis tests across each scenario confirms significant differences among the methods ( $ p<0.001 $ ). Subsequent pairwise Mann–Whitney U tests with Bonferroni adjustments clearly demonstrate that the 3D CycleGAN-3 model significantly outperforms both the 2D CycleGAN and UNSB models in all scenarios evaluated. For more detailed qualitative and quantitative results, please refer to Appendix A, where Table 5 presents FID, KID, and MOS scores for each set evaluated in the subjective test.

Figure 10. Boxplot of subjective evaluation scores for comparison across scenarios with reference (‘W Ref’), without reference (‘W/O Ref’), and the combined total (‘Total’). The circles indicate outliers in the data.

Ophthalmologist feedback. In the subjective evaluations, ophthalmologists primarily assessed the clarity and shape of blood vessels, with the majority acknowledging that the 3D CycleGAN-3 model preserved blood vessel clarity and shape effectively, as well as the edges. The next aspect they considered was color code preservation, particularly the representation of the green channel, which is crucial for biological interpretation. Attention was also given to background detail, overall quality, aesthetics, and the correct distribution of colors, a critical factor for the biological accuracy of the images. For example, in scoring Set 2 of Figure 8, some experts preferred the 3D CycleGAN-3 for its accuracy in the green channel, compared to the 3D CycleGAN-GL, which displayed slightly more background vessels but less accuracy.

The UNSB model, however, received criticism for incorrect color code preservation. Set 6 of Figure 9 was noted for instances where the 2D CycleGAN and UNSB models ignored edges, and in Set 7 of the same figure, the 2D CycleGAN was criticized for exhibiting too much random color, missing the green staining seen in the reference, and unclear imaging.

The feedback from ophthalmologists suggests that the 3D CycleGAN-3 model not only effectively achieved a stylistic modal transfer but also more importantly preserved the biological content of the medical images, which is vital for clinical interpretation and diagnosis.

Analysis of hallucinations. Following feedback from ophthalmological evaluations, we now focus on analyzing how accurately our models avoid introducing hallucinations – false features not actually present in the true anatomy.

For subjectively evaluating the model hallucinations, we focus on two key biological features relevant to retinal imaging: vessel structures in the red channel and immune cell markers, CD4+ T cells, in the green channel. These features are crucial for assessing vascular structure and immune responses within the retina for uveitis, respectively.

As shown in Figure 11, our analysis in Set 2 reveals notable differences in the clarity and fidelity of vascular structures among the models. The 3D CycleGAN-3 model generally outperforms both the 2D models and other 3D variations in preserving the integrity of major blood vessels with minimal distortions. Specifically, areas highlighted in the red channel exhibit fewer hallucinated vessels, which are incorrectly generated features not aligned with the underlying anatomical structure of the retina.

Figure 11. Example of model hallucination analysis. Focusing on the red channel for vascular structures and the green channel for CD4+ T cells. Areas highlighted (yellow boxes) show where each model introduces inaccuracies in the representation of vascular and immune cell distributions.

Similarly, in the green channel, which focuses on the distribution of CD4+ T cells indicative of immune activity, the 3D CycleGAN-3 model shows a stronger correlation with the original confocal images in terms of brightness which indicates the immune response areas. However, artifacts around the ONH in the center are present across all models, with our proposed 3D CycleGAN-3 model demonstrating the least severity in artifact generation.

6.4. Computational efficiency

To assess the computational demands of each model, we analyzed the number of parameters (#Params), number of Floating Point Operations (#FLOPs), and RunTime (RT) for each translation process. The #Params and #FLOPs represent the total number of trainable parameters and floating-point operations required to generate an image, respectively. #Params measures the model complexity and memory usage. #FLOPs provides an estimate of computational intensity, crucial for understanding the processing power required and potential latency in real-time applications. The RT is measured during inference, indicating the practical deployment efficiency of each model, reflecting the time taken to process an image. For 2D models, the RT is calculated by summing the times required to process nine of 512 × 512 images, simulating the workflow for generating a complete 3D volume from 3D models.

Table 4 illustrates the tradeoff between computational efficiency and image quality. While the 2D models (UNSB and 2D CycleGAN) demonstrate quicker processing times, their lower MOS scores suggest a compromise in image quality. In contrast, the extended runtimes associated with 3D models, although potentially limiting for real-time applications, result in higher-quality images that are more clinically valuable, as reflected by their higher MOS scores and positive feedback from ophthalmologists.

Table 4. Comparative computational of different models. For each column, the red indicates the most computationally efficient values for each metric

Note: ‘#Params’ denotes the total number of trainable parameters in millions (M), ‘#FLOPs’ represents the computational complexity in billions (G) of floating-point operations, and ‘RT’ indicates the average execution time in seconds (s) per image. Lower values in each metric indicate more efficient computational performance. MOS (Mean Opinion Score) rates the subjective quality of images with higher scores reflecting better quality.

The 2D CycleGAN, with the lowest #Params (11.378M) and moderate #FLOPs (631.029G) among all models, offers rapid inference times at 0.945s for processing 512 × 512 × 9 3D data. This indicates that 2D CycleGAN is more suitable for applications requiring quick image processing such as real-time. However, as revealed by its MOS and the feedback from quality evaluations, the lower complexity cannot adequately capture the spatial relationships and structural complexity inherent in 3D data.

Despite having a higher parameter count than a 2D model, the UNSB model exhibits relatively fewer #FLOPs (253.829G), which may be attributed to its diffusion-based generative process. Although this process involves numerous iterations, each iteration consists of simpler operations, thus accumulating a lower total computational load (#FLOPs). However, the need for multiple iterations to refine image quality leads to significantly longer runtimes—up to ten times longer than the 2D CycleGAN—illustrating its inefficiency in time-sensitive scenarios.

The 3D CycleGAN models involve substantially higher #Params and #FLOPs. Particularly the 3D CycleGAN-3, with 191.126M #Params and 1585.332G #FLOPs and the longest RT of 94.451s. However, this investment in computational resources facilitates a more accurate rendering of complex 3D structures, as evidenced by its highest MOS of 56.469, suggesting superior image quality and detail retention. However, the increased computational demands of 3D models present a challenge for real-time applications, where quick processing is essential. Therefore, future efforts will focus on optimizing the computational efficiency of 3D models without compromising their ability to deliver high-quality 3D image translations to enable time-sensitive applications.