2.1. The advance baseline imager (ABI)

The ABI, developed by NASA in collaboration with NOAA, delivers continuous Earth observations from the geostationary orbit of the GOES family of satellites (Schmit et al., Reference Schmit, Griffith, Gunshor, Daniels, Goodman and Lebair2017). It captures imagery across 16 spectral wavelengths, as detailed in Table A1, providing a high spatiotemporal resolution that surpasses earlier meteorological Earth observation satellites (Menzel and Purdom, Reference Menzel and Purdom1994). Vandal et al. (Reference Vandal, McDuff, Wang, Duffy, Michaelis and Nemani2022) emphasize that the transferability between different satellite sensors remains a limitation for current deep learning approaches in this domain. The ABI’s broad spectral range and its role as a blueprint for subsequent imagers (Park et al., Reference Park, Kim and Hong2021) may help address this challenge. NOAA’s public release of the extensive GOES dataset can facilitate the training of more generalizable deep learning models using imagery captured by the ABI and driving advancements in the field.

Expanding on the ABI’s capabilities, the imager can operate in multiple scan modes to address various observational requirements. In scan mode 6, the current default, the ABI captures a full-disk image of Earth every 10 minutes and images of the contiguous United States (CONUS) every 5 minutes. Additionally, it acquires two mesoscale images every minute, each covering approximately $ \mathrm{1,000}\;\mathrm{k}{\mathrm{m}}^2 $ (Schmit and Gunshor, Reference Schmit and Gunshor2024). These mesoscale images target regions of meteorological significance, with locations determined by NOAA staff and adjusted hourly or daily based on evolving conditions. This ability to focus on specific areas ensures continuous monitoring of critical weather phenomena such as developing cyclones and thunderstorms, enabling more precise and timely forecasting. Owing to their localized detail and meteorological interest, these images are particularly well-suited for this work and form the basis of our data selection.

2.2. Convolutional approaches to synthetic VIS generation

Kim et al. (Reference Kim, Kim, Moon, Park, Shin, Kim, Kim and Hong2019) conducted the first experiments using deep convolutional networks to generate synthetic nighttime reflectance by employing the Pix2Pix architecture (Isola et al., Reference Isola, Zhu, Zhou and Efros2018) to transform longwave infrared (LWIR) imagery into red-band visible (VIS) data. Their experiments revealed that longwave infrared (LWIR) bands in the $ 10.3\;\unicode{x03BC} \mathrm{m}-11.3\;\unicode{x03BC} \mathrm{m} $ range exhibit the highest correlation with the visible red band ( $ 0.55\;\unicode{x03BC} \mathrm{m}-0.80\;\unicode{x03BC} \mathrm{m} $ ), establishing a foundational baseline for this task. While their model demonstrates the potential of generative modeling for this task, its performance was hindered by seasonal variability, particularly due to changes in the spatial distribution of clouds and low-level fog. A further shortcoming of the Pix2Pix model was its difficulty in distinguishing cloud cover from bare land at night, leading to frequent misclassification of cold desert or sparsely vegetated regions as cloud-filled (Han et al., Reference Han, Jang, Ryu, Sohn and Hong2022).

By incorporating four additional LWIR bands (Kim et al., Reference Kim, Ryu, Jeong, So, Ban and Hong2020, advanced the initial approach and helped formalize the use of multi-band inputs as a standard strategy for improving generation fidelity. Expanding the multi-band approach (Cheng et al., Reference Cheng, Li, Wang, Zhang and Huang2022) integrate ERA5-derived scale and depth priors to enhance RGB VIS generation. While this improves image quality, it introduces computational overhead by increasing input size to 83 channels and requiring a Squeeze-and-Excitation preprocessing step (Hu et al., Reference Hu, Shen, Albanie, Sun and Wu2019). More recently, Yao et al. (Reference Yao, Du, Zhao and Wang2024) introduced solar positioning parameters, azimuth and zenith angles, into the generation process, resulting in synthetic red VIS imagery with highly realistic lighting that reflects variations across different times of day. Diverging from prior efforts that focus on data enrichment, our work centers on architectural design, specifically the development of a context-rich latent space. To this end, we employ a multi-band approach with three input bands and assess the performance of a Vector Quantized Variational Autoencoder (VQVAE) (van den Oord et al., Reference van den Oord, Vinyals and Kavukcuoglu2018; Esser et al., Reference Esser, Rombach and Ommer2021), trained adversarially.

2.3. Multi-layer perceptron approaches to synthetic VIS generation

While (Harder et al., Reference Harder, Jones, Lguensat, Bouabid, Fulton, Quesada-Chacón, Marcolongo, Stefanović, Rao, Manshausen and Watson-Parris2020) explore convolutional generative models, including UNet and UNet+, and identify Pix2Pix CGAN as the most effective for realistic generations, alternative approaches take a fundamentally different route by modeling the task as a regression problem using MLPs. NightDNN (Yan et al., Reference Yan, Qu, An and Zhang2023) employs six LWIR wavelength bands that capture key atmospheric features such as water vapor, cloud cover, and surface temperatures. These bands are flattened into 1D arrays and subsequently passed through a multi-layer perceptron (MLP). Since pixel-by-pixel training tends to lose surface texture and spatial structure, the authors mitigate this by training a second, shallow MLP to encode geographic features from NASA’s Blue Marble images. This additional input helps the model retain the local and global details lost by focusing solely on minimizing pixel-wise mean square error (MSE). The method is applied to full-disk imagery, which benefits from the added spatial encoding to maintain high-quality image generation. The model outputs a 1D array of values, which is reshaped into an RGB image. This technique effectively removes solar lighting artifacts, including glint, and shows how modeling pixel interactions can also lead to high-quality results.

In building on this prior work, Pasillas et al. (Reference Pasillas, Kummerow, Bell and Miller2024) introduces a deep MLP framework, called machine learning nighttime visible imagery (ML-NVI). This model is trained on Day Night Band (DNB) data from the Visible Infrared Imaging Radiometer Suite (VIIRS), producing consistent DNB-derived imagery with accurate cloud representation throughout the lunar cycle, thus enabling robust nighttime cloud detection. The study demonstrates that MLPs are effective in generating consistent imagery by modeling pixel-level interactions, as opposed to relying solely on the local interactions facilitated by convolutions. However, the “curse of dimensionality” makes MLP-based approaches impractical for satellite imagery, given the large number of pixels typically involved. In response, our approach integrates pixel-level interactions using a self-attention layer (Vaswani et al., Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017) within the VQVAE framework at the encoder’s lowest resolution.

3.1. Data processing

The study uses the ABI-L2-MCMIPM product from the GOES West satellite (NOAA, n.d.). This dataset comprises co-registered satellite images with a spatial resolution of $ 2\hskip0.22em {\mathrm{km}}^2 $ , resampled to $ 500\times 500 $ pixels. The data, originally provided in NetCDF format, is converted to NumPy (.npy) arrays for compatibility with deep learning frameworks. Min-max normalization is applied using valid minimum and maximum values for the ABI sensor at each wavelength band, ensuring consistent scaling. Since the ABI sensor lacks a dedicated green band, a synthetic green band is generated using the Cooperative Institute for Meteorological Satellite Studies formula:

(1) $$ \mathrm{Green}\_\mathrm{Band}=0.45\left(\mathrm{Red}\_\mathrm{Band}\right)+0.1\left(\mathrm{Veggie}\_\mathrm{Band}\right)+0.45\left(\mathrm{Blue}\_\mathrm{Band}\right). $$

To reduce dataset size and computational complexity, a subset of spectral bands is chosen based on prior research (Park et al., Reference Park, Kim and Hong2021). The selected bands are 1, 2, 3, 8, 9, 10, 11, 13, 14 (see Table A1), along with the constructed green band, forming a 3D array of shape $ 10\times 500\times 500 $ . The ground truth (target domain $ \mathbf{Y} $ ) comprises RGB images from the Red, Green, and Blue bands, and the input domain $ \mathbf{X} $ consists of three bands selected from $ \left\{\mathrm{8,9,10,11,13,14}\right\} $ . The study evaluates various band combinations to optimize the translation task.

After preprocessing, each sample includes:

• • Input domain: Three selected LWIR bands.

• • Target domain: Corresponding RGB images from VIS bands.

The dataset is partitioned as follows:

• • Training: 6,000 examples (daytime imagery between 10 am and 4 pm PST, from 2019 to 2021).

• • Validation: 600 examples (daytime imagery between 10 am and 4 pm PST, from 2019 to 2021).

• • Testing: 360 nighttime images (1 am–4 am PST), 356 daytime images (10 am–2 pm PST), 20 daytime land-only images, and 20 daytime ocean-only images (all testing samples collected from 2022 imagery).

3.2. Model architecture

Image-to-image translation involves transforming an image from one domain to another while preserving its essential content (Isola et al., Reference Isola, Zhu, Zhou and Efros2018). In our case, the task involves converting satellite imagery across different spectral representations, ensuring spatial consistency and accurate domain-specific characteristics. Our approach is framed as an image-to-image translation problem where the goal is to learn a function $ F:X\to Y $ that maps an input image $ x\in X $ to its corresponding output $ y\in Y $ . To accomplish this, we leverage a combination of discrete variational autoencoders (VQVAEs) and generative adversarial networks (GANs). The following components make up our model architecture.

3.2.1. Discrete variational autoencoders (VQVAEs)

VQVAEs compress high-dimensional data into a discrete latent space, facilitating efficient image reconstruction and translation. Their architecture consists of three main parts:

Encoder. The encoder $ E(x) $ maps the input $ x $ to a lower-dimensional latent representation $ \hat{z} $ . Unlike standard VAEs that learn continuous latent representation, the VQVAE encoder outputs are served to a quantization layer which discretizes the latent representation $ \hat{z}=E(x) $ .

Quantizer. The quantizer discretizes the encoder’s output by mapping each index vector to the closest one among $ K $ predefined embedding vectors in the space $ Z $ . This is formalized as

(2) $$ {z}_q=q\left(\hat{z}\right):= \left(\arg \underset{z_k\in Z}{\min}\parallel {\hat{z}}_{ij}-{z}_k\parallel \right). $$

Since the quantization step is non-differentiable, we use the straight-through estimator to approximate gradients and facilitate end-to-end training.

Decoder. The decoder reconstructs the input image from the quantized latent variable $ {z}_q $ . The overall training objective for the VQVAE integrates three loss components: reconstruction loss, vector quantization loss (aligning the embedding vectors), and commitment loss (ensuring the encoder commits to specific embeddings). The combined loss is given by

(3) $$ {L}_{VQ}\left(E,G,Z\right)=\parallel x-\hat{x}{\parallel}_2^2+\parallel \mathrm{sg}\left[E(x)\right]-{z}_q{\parallel}_2^2+\parallel \mathrm{sg}\left[{z}_q\right]-E(x){\parallel}_2^2, $$

where $ \mathrm{sg}\left[\cdot \right] $ denotes the stop-gradient operator.

3.2.2. Generative adversarial networks (GANs)

GANs are employed to enhance the perceptual realism of the generated images. The discriminator $ D(x) $ distinguishes between real and generated examples, while the generator $ G\left(\hat{z}\right) $ produces realistic outputs. The GAN loss is defined as

(4) $$ {L}_{GAN}\left(\left\{E,G,Z\right\},D\right)={\unicode{x1D53C}}_x\left[\log D(x)+\log \left(1-D\left(\hat{x}\right)\right)\right]. $$

Following the framework of Esser et al. (Reference Esser, Rombach and Ommer2021), adversarial training is combined with the VQVAE losses to yield the VQGAN model, with the overall learning objective formulated as

(5) $$ {Q}^{\ast }=\arg \underset{E,G,Z}{\min}\underset{D}{\max }{\unicode{x1D53C}}_{x\sim p(x)}\left[{L}_{VQ}\left(E,G,Z\right)+\lambda {L}_{GAN}\left(\right\{E,G,Z\Big\},D\Big)\right], $$

where $ \lambda $ balances the perceptual reconstruction and adversarial losses. A schematic of the complete VQGAN model used in this pipeline is presented in Figure 1.

Figure 1.

Diagram of the VQGAN model for image-to-image translation in the training pipeline.

3.3. Training protocol and hyperparameter tuning

Our training strategy follows the framework introduced by Esser et al. (Reference Esser, Rombach and Ommer2021), focusing on constructing a robust latent space for effective image-to-image translation. Key aspects include the following:

3.3.5. Varying input bands

Building on the stabilized training regime and insights from prior ablation studies, we further evaluate the effect of varying LWIR wavelength bands and embedding dimensions $ {Z}^D\in \left\{4,6\right\} $ . These experiments retain two residual blocks in the encoder-decoder architecture and introduce water vapor imagery as an additional input channel. The discriminator activation step is extended to 70,000 steps to allow the generator more time to learn stable representations.

Although two band combinations were considered, Bands 11, 13, and 14 versus Bands 10, 11, and 14, all results reported here are from the latter configuration. This combination was chosen based on its more consistent output quality during exploratory testing. The 4- and 6-dimensional embeddings both yield high-quality outputs (Figure A2), with no significant advantage observed for the higher-dimensional case aside from a longer convergence time. Notably, the inclusion of water vapor imagery substantially improves image realism (Table 1).

Table 1.

Final configurations of evaluated models

Note. All models were trained for 100 epochs with consistent architecture and loss settings.

4.1. Evaluation of land, ocean, and nighttime image generation

The top two models were evaluated across land and ocean imagery to assess their performance over different surface types. Interestingly, land imagery consistently outperformed ocean imagery across all evaluation metrics (Table 2), contrary to initial expectations. This discrepancy may be due to a bias in the training data as most mesoscale scenes were centered over the western United States, comprising predominantly land. Consequently, the model may have become more adept at reconstructing daytime land scenes due to their relative abundance during training. Visual comparisons underscore this difference in performance (Figure A1).

Table 2.

Performance metrics comparison

Note. Best-performing metrics (highlighted in yellow) for our model evaluated on combined land/ocean, land-only, and ocean-only scenes. Best overall performance per category is highlighted in green.

^a RMSE values are not directly comparable across methods due to differences in data domains, pre-processing pipelines, and evaluation setups.

Despite the lower numerical scores for ocean scenes, these examples revealed promising qualitative results. A key success lies in the model’s ability to reconstruct subtle atmospheric features from longwave infrared (LWIR) inputs. Features like low-level fog—which typically blend into the background in LWIR—were clearly generated in the visible output. Similarly, the model successfully highlighted low-altitude cumulus clouds associated with the early stages of tropical cyclones. While the model did not fully capture the structured vortex of a developing storm, it did reconstruct the general circular cloud pattern and a perceptible center of rotation, showing promise in its ability to infer complex weather dynamics from thermal data (Figure A1).

Nighttime evaluations further confirmed the model’s strengths and limitations. Over ocean regions, the model retained high levels of detail and contrast (Figure A2). Land imagery, however, often defaulted to overwhelmingly white land cover (Figure A4). This behavior is consistent with Han et al. (Reference Han, Jang, Ryu, Sohn and Hong2022) and likely arises because land surfaces cool more rapidly than oceans at night. The model appears to infer cloud presence over cooled land surfaces based on their temperature similarity, suggesting it is learning surface temperature rather than implicitly modeling cloud or atmospheric structure.

These findings point to a deeper limitation in current image-to-image translation models when applied to geophysical data: they lack an understanding of the physical world. The persistent nighttime artifact over land, for instance, signals the model’s reliance on statistical correlations rather than physical reasoning. This highlights the need for incorporating physically-informed representations. Embedding techniques such as Sphere2Vec (Mai et al., Reference Mai, Xuan, Zuo, He, Song, Ermon, Janowicz and Lao2023), or broader frameworks that acknowledge the spatiotemporal and physical structure of atmospheric data, may offer a way forward. Treating satellite imagery as generic natural images overlooks domain-specific constraints—and this evaluation underscores the need for dedicated methods tailored to geoscientific image modeling.

4.2. Input bands and the learned latent space

The performance improvements observed with the inclusion of a water vapor imagery channel may stem from the additional vertical distribution information it provides. Specifically, the low-level water vapor band introduces information from a distinct atmospheric layer, effectively acting as a weak depth prior for the model (Figure A5). While the other LWIR channels offer integrated signals across the full atmospheric column, water vapor imagery tends to be more level-specific, allowing the model to better localize features in its latent representation.

Despite still exhibiting some of the same challenges as the baseline, such as color inversion in reconstructed VIS output, the qualitative detail in the generated images was noticeably improved. Fine-grained structures and spatial coherence were better preserved, suggesting that the multi-level atmospheric context helped the model disentangle relevant features more effectively. A comparison between the input bands, baseline model, ground truth, and final generation is shown in Figure A6.

These results indicate the model learning a robust, generalizable representation of the earth and atmosphere that is, to some extent, independent of the specific input bands presented. This suggests the potential for a refined approach, where the latent space could be designed to be invariant to the wavelength inputs. If we view the different bands as multimodal inputs, we can then imagine that the learned latent space would be conditioned on all these multimodal inputs. For tasks such as latent diffusion, it could be possible to train a model on a single satellite with a sufficiently broad variety of wavelengths and transfer the learned latent space to other tasks. In such cases, the only requirement would be to train a new encoder model on the specific inputs for a given task. The model would replace the inputs with their nearest neighbor vector embeddings from the learned latent space, allowing the model to operate fully in the latent space for objectives like diffusion or super-resolution. This approach would make computation more efficient, as it would leverage a common embedding space, enabling future research to focus on improving interpretability across multiple models that share this latent space.