2.1. Related works

In recent years, there has been meaningful progress in applying deep learning techniques to problems related to weather and climate. The authors in Bihlo (Reference Bihlo2021) use an ensemble of GANs to predict the future (one year) weather using the European Centre for Medium-Range Weather Forecasts (ECMWF) ERA5 ( $ {0.25}^{\circ}\times {0.25}^{\circ } $ spatial and three-hour time resolution) reanalysis data from the prior four years, and evaluate their models using root-mean-squared error (RMSE), anomaly correlation coefficients (ACC), and the continuous ranked probability scores (Zamo and Naveau, Reference Zamo and Naveau2018), which are commonly used to evaluate forecasts. Meng et al. (Reference Meng, Rigall, Chen, Gao, Dong and Chen2021) develop a physics-informed GAN for sea subsurface temperature prediction at $ {0.25}^{\circ}\times {0.25}^{\circ } $ spatial and daily mean resolutions. In Keisler (Reference Keisler2022), the authors utilize Graph Neural Networks (GNN) for global weather forecasts. The model uses local information to step the current 3D atmospheric state forward by six hours, and show results that rivals state of the art forecasting models such as the ECMWF model, while reducing computational cost; some months later, Lam et al. (Reference Lam, Sanchez-Gonzalez, Willson, Wirnsberger, Fortunato, Pritzel, Ravuri, Ewalds, Alet and Eaton-Rosen2022) (also GNN) and Bi et al. (Reference Bi, Xie, Zhang, Chen, Gu and Tian2022) introduced global weather forecasting models that surpassed the state of the art physics-based numerical weather prediction model on most metrics. In Bhatia et al. (Reference Bhatia, Jain and Hooi2021), the authors adopt a gradual distribution shifting and resampling approach to model extreme precipitation, but they evaluate the outputs using Fréchet Inception Distance (FIT) proposed by Heusel et al. (Reference Heusel, Ramsauer, Unterthiner, Nessler, Klambauer and Hochreiter2017) which does not explain the temporal veracity of the generated rainfall (distribution). Additionally, we suggest that FIT is not appropriate for understanding the model’s performance on generating physical variables, because the intermediate layer from which scores are obtained does not produce values with units that have an easily discernible physical meaning. In Puchko et al. (Reference Puchko, Link, Hutchinson, Kravitz and Snyder2020), the authors use a GAN to model average daily temperatures from the Coupled Model Intercomparison Project Phase (CMIP) data, taking an autoregressive approach, and evaluate their model by visually comparing the histogram plot of true and generated data, which can be useful, but is not rigorous and can be biased by the human eye. Additionally, they do not produce samples at an hourly resolution, but rather a daily average temperature, which may not be sufficient for some downstream applications (i.e. reliability assessments). They also acknowledge that more work needs to be done to thoroughly evaluate their GAN. In Besombes et al. (Reference Besombes, Pannekoucke, Lapeyre, Sanderson and Thual2021), the authors use a GAN to model the daily mean climate variables at a 2.8° resolution and propose some approaches to evaluation, which include principal component Analysis (PCA), Wasserstein distance, and visual inspection, recognizing that traditional methods for evaluating GANs for natural image synthesis community may not suffice. More recently, Izumi et al. (Reference Izumi, Amagasaki, Ishida and Kiyama2022) leverage existing GANs for super-resolution of sea surface temperature, and evaluate the models by comparing the outputs with high resolution optimum interpolation sea surface temperature (OISST), using the learned perceptual image patch similarity (LPIPS) and RMSE as metrics.

Although recent work has shown that GANs have the potential for modeling climate variables, utilizing GANs in this regime is still in its infancy. Consequently, there is a dearth of intuitive and consistent evaluation metrics/benchmarks specific to this application. Evaluation metrics that can be adopted by machine learning (ML), power, and climate communities, are critical for comparing generative models and their outputs, especially for researchers working at the confluence of these fields.

3.1. Data

Direct station observations (e.g., the automated surface observation system in the US) are reliable, accurate, and can have decades of historical records, but are spatially sparse. Satellites can provide spatially gridded observations of some atmospheric variables, but data may not exist for long time periods. Reanalysis datasets blend these sources together along with forecasting/simulation models to produce spatially and temporally consistent maps of climate variables. The dataset used for this work is the mosaic land surface model forcing temperatures data from the North American Land Data Assimilation System (NLDAS) (Xia et al., Reference Xia, Mitchell, Ek, Sheffield, Cosgrove, Wood, Luo, Alonge, Wei and Meng2012). The dataset is well-suited for this task as it provides fine spatial and temporal resolution relevant to power systems studies. NLDAS (a collaborative effort between NOAA, NASA, and others) integrates satellite and surface-level observational data with reanalysis datasets and includes multiple surface state and flux variables in North America from 1979 to near-present day. This data has a $ {0.125}^{\circ}\times {0.125}^{\circ } $ ( $ 13.8\times 11 $ km) spatial resolution (taken from an equidistant cylindrical projection) at an hourly timescale and has a size of about 700GB for the contiguous US. Training using data at a $ {0.125}^{\circ}\times {0.125}^{\circ } $ resolution, aids the possibility for a model to capture fine spatiotemporal dependencies that a physics-driven model cannot without a costly downscaling effort. We limit the scope of this work to the United States (US) West Coast region due to computational resources, and regions without data are zero-padded. We aggregate the raw data both spatially and temporally, as single grid point measurements are not sufficient for generating distributions with a high degree of confidence.

3.1.2. Temporal data aggregation

Temporally, we posit that temperature distributions are non-stationary over sufficiently long periods Donat and Alexander (Reference Donat and Alexander2012). The data is aggregated such that each example is a 24-hour (daily) temperature map; one can imagine each example as a video with 24 frames. We introduce the idea of periods—a period is a stipulated number of years for which one can assume that the overall climate does not change significantly. To make this concrete, we aggregate the data into 24-hour daily time-series, then group all the 24-hour time series by their respective months, and finally, the same month within the elected period is also thrown into the same bucket. In this work, we elect 4-year periods. This makes the implicit assumption that for a 4-year period $ {k}_i $ , at a given region $ R $ , and month, $ M $ , the diurnal cycles come from the same distribution, or are independently and identically distributed (IID). For example, if the entire historically observed record spans 1979 to present day, then the first period, $ {k}_0 $ , will encompass observations from 1979–1982, inclusive, the second period $ {k}_2 $ , spans 1983–1986 inclusive, the third period $ {k}_3 $ spans 1987–2000, etc. Selecting quadrennial (4-year) periods has not been rigorously justified, as one might elect 1-, 2-, or 5-year periods instead. The choice of number of years within a period constitutes a design trade-off. If there are too few years (e.g. one year period) then the modeling assumption departs too far from known climatology and will be difficult to evaluate empirically with confidence. If there are too many years, one might unintentionally average out temporal effects or years/periods of significant temperature distribution shifts or dilation—we find 4 years to be a reasonable choice and defer more thorough investigation to future work. Figure 2 describes the temporal data aggregation scheme.

Figure 2.

Image depiction of data aggregation scheme for a specific region $ R $ and for the chosen month $ M $ of January. For a 4-year period $ {k}_i $ , all the 24-hour time-series (31 for each year because January has 31 days) within all four months of January are grouped into the same bucket (making it a total of 124 examples) and have the same labels.

Adopting $ {k}_i $ , where $ i\in \left\{\mathrm{0,1,2},\dots, n\right\} $ , offers the ability to smooth over inter-year macro scale climate events (e.g. El Niño) in the ground-truth observations and capture medium/longer term trends. Another important motivation for this modeling decision is that it lends itself well to evaluation, because by aggregating over regions and years we have more data per sample, making parametric and non-parametric goodness-of-fit tests for evaluating the generated samples from the model more tractable.

3.2. Model architecture and training

Video generation (Saito et al., Reference Saito, Matsumoto and Saito2017; Clark et al., Reference Clark, Donahue and Simonyan2019; Chu et al., Reference Chu, Xie, Mayer, Leal-Taixé and Thuerey2020; Gur et al., Reference Gur, Benaim and Wolf2020; Wang et al., Reference Wang, Bilinski, Bremond and Dantcheva2020; Gupta et al., Reference Gupta, Keshari and Das2022) is one of the most challenging GAN applications. In Clark et al. (Reference Clark, Donahue and Simonyan2019), the authors propose a dual-video-discriminator (DVD) to handle the memory bottleneck for video datasets (terming the model DVD-GAN), using two discriminators—a spatial and temporal discriminator. Spatial discriminator $ {D}_{\mathrm{s}} $ inspects an individual video frame (a static image) for texture quality and temporal discriminator $ {D}_{\mathrm{t}} $ for penalizes the generated frame-by-frame transitions. For TemperatureGAN, we also leverage two discriminator/critic networks, but take a different approach from DVD-GAN. We build a convolutional neural network-based temporal discriminator $ {D}_{\mathrm{t}} $ , and rather than training on videos (in our case a video is a spatio-temporal time series of temperature values) our model is trained on temporal gradients, distinguishing it from DVD-GAN. Training on the temporal gradients separately guides the model to focus on the learning the daily (hourly) diurnal cycles and to produce hourly (or temporal) temperature transitions that are consistent with the ground-truth’s diurnal cycles. Thus, for a given 24-hour sample, we have 23 gradients $ \frac{\partial T}{\partial t} $ . We use the Wasserstein loss with a gradient penalty (Gulrajani et al., Reference Gulrajani, Ahmed, Arjovsky, Dumoulin and Courville2017) as a soft constraint to satisfy the 1-Lipschitz continuity (Gouk et al., Reference Gouk, Frank, Pfahringer and Cree2021). We also experimented with directly constraining the layer weights via spectral normalization introduced in Miyato et al. (Reference Miyato, Kataoka, Koyama and Yoshida2018) to satisfy the 1-Lipschitz continuity conditions and found it to perform poorly, thus was not further pursued. The loss functions are:

(1) $$ {L}_{\mathrm{D},\mathrm{temporal}}=\underset{\overset{\sim }{\boldsymbol{T}}\sim {\unicode{x2119}}_g}{\unicode{x1D53C}}\left[{D}_{\mathrm{t}}\left(\frac{\partial \overset{\sim }{\boldsymbol{T}}}{\partial \boldsymbol{t}}\right)\right]-\underset{\boldsymbol{T}\sim {\unicode{x2119}}_r}{\unicode{x1D53C}}\left[{D}_{\mathrm{t}}\left(\frac{\partial \boldsymbol{T}}{\partial \boldsymbol{t}}\right)\right]+{\lambda}_{GP}\underset{\hat{\boldsymbol{T}}\sim {\unicode{x2119}}_{\hat{\boldsymbol{T}}}}{\unicode{x1D53C}}\left[{\left({\nabla}_{\hat{\boldsymbol{T}}}{D}_{\mathrm{t}}{\left(\frac{\partial \hat{\boldsymbol{T}}}{\partial \boldsymbol{t}}\right)}_2-1\right)}^2\right] $$

(2) $$ {L}_{\mathrm{D},\mathrm{spatial}}=\underset{\overset{\sim }{\boldsymbol{T}}\sim {\unicode{x2119}}_g}{\unicode{x1D53C}}\left[{D}_s\left(\overset{\sim }{\boldsymbol{T}}\right)\right]-\underset{\boldsymbol{T}\sim {\unicode{x2119}}_r}{\unicode{x1D53C}}\left[{D}_{\mathrm{s}}\left(\boldsymbol{T}\right)\right]+{\lambda}_{GP}\underset{\hat{\boldsymbol{T}}\sim {\unicode{x2119}}_{\hat{\boldsymbol{T}}}}{\unicode{x1D53C}}\left[{\left({\nabla}_{\hat{\boldsymbol{T}}}{D}_{\mathrm{s}}{\left(\hat{\boldsymbol{T}}\right)}_2-1\right)}^2\right] $$

(3) $$ {L}_{\mathrm{G}}=\underset{\overset{\sim }{\boldsymbol{T}}\sim {\unicode{x2119}}_g}{-\unicode{x1D53C}\left[{D}_{\mathrm{s}}\left(\overset{\sim }{\boldsymbol{T}}\right)\right]-\unicode{x1D53C}\left[{D}_{\mathrm{t}}\left(\frac{\partial \overset{\sim }{\boldsymbol{T}}}{\partial \boldsymbol{t}}\right)\right]} $$

$ \overset{\sim }{\boldsymbol{T}}\sim {\unicode{x2119}}_{\mathrm{g}} $ represents examples sampled from the GAN (the generator) and $ \boldsymbol{T}\sim {\unicode{x2119}}_{\mathrm{r}} $ represents examples sampled from the real (observed) data. The discriminators/critics and generator seek to minimize their respective losses. Observe that the first two terms in Equations (1) and (2) instruct the discriminator D to maximize the gap between the expected values of the true and fake samples for the temperature gradients and the temperature values—this is done by minimizing the loss functions in (1) and (2). The objective of the discriminators $ {D}_{\mathrm{t}} $ and $ {D}_{\mathrm{s}} $ is to minimize Equations (1) and (2), because by minimizing these losses, it is encouraged to assign higher scores to true (observed) data and lower scores to generated (“fake”) examples. However, the objective of the generator $ G $ is to minimize Equation (3), which means it is encouraged to produce examples that will yield high scores from the discriminator, suggesting that it attempts to produce samples that resemble those from the observed (ground-truth) data, thereby implicitly estimating the true probability distribution $ {\unicode{x2119}}_{\mathrm{r}} $ . The final terms in Equations (1) and (2) highlight an important concept regarding the training stability of Wasserstein GANs (WGANs), which is the notion of Lipschitz continuity. For training stability, the loss functions of the discriminator should be 1-Lipschitz continuous for the Wasserstein distance approximation to be valid; this constrains how quickly the models’ parameters can change during training. In other words, enforcing Lipschitz continuity ensures that the Discriminator’s loss does not grow too quickly such that the Generator cannot learn. $ {\nabla}_{\hat{\mathbf{T}}}D\left(\hat{\mathbf{T}}\right) $ is the gradient of the discriminator’s outputs with respect to its inputs, which are temperature maps. We follow a similar convention as in Gulrajani et al. (Reference Gulrajani, Ahmed, Arjovsky, Dumoulin and Courville2017), where the authors define $ {\mathrm{\mathbb{P}}}_{\hat{\mathbf{T}}} $ as sampling uniformly along straight lines between pairs of points sampled from the data distribution $ {\mathrm{\mathbb{P}}}_r $ and the generator distribution $ {\unicode{x2119}}_{\mathrm{g}} $ . $ {\lambda}_{\mathrm{GP}} $ is a hyperparameter for weighting how much importance the model places on the final terms representing Lipschitz continuity in Equations (1) and (2); we elect $ {\lambda}_{\mathrm{GP}}=1 $ for training.

Figures 3–5 show the architectures of the generator and discriminator neural networks. They are convolution-based networks.

Figure 3.

Generator $ G $ architecture. The sampled noise is concatenated with the learned label embedding and passed through a dense, fully-connected (FC) linear block with Rectified Linear Unit (ReLU) activation functions. The output of the linear block is sent through series of convolution layers with batch normalization to obtain the desired ( $ 8\times 8\times 24 $ ) output shape.

Figure 4.

Spatial discriminator $ {D}_{\mathrm{s}} $ architecture. The inputs to the spatial discriminator $ {D}_s $ , are the $ 8\times 8\times 24 $ temperature maps. The input is then passed through a series 2D convolution layers with the output of the prior layer serving as input into the following layer. The final convolution layer outputs a 2-dimensional $ 24\times 1 $ vector, which is flattened into a one-dimensional vector before it is passed through a dense FC linear block to produce a score.

Figure 5.

Temporal discriminator $ {D}_{\mathrm{t}} $ architecture. The inputs to the temporal discriminator $ {D}_t $ , are the $ 8\times 8\times 24 $ temperature maps and then a gradient computation is followed, to compute $ \frac{\partial T}{\partial t} $ . The output from the convolution layers is flattened and passed through a series of fully-connected (FC) linear layers to produce a score.

The generator is an arguably modest 562,206 parameter model. We normalize and standardize the data and train the GAN for 2000 epochs using a batch size of 4096 and ADAM optimizer (Kingma and Ba, Reference Kingma and Ba2014) ( $ {\beta}_1=0.5,{\beta}_2=0.99 $ ) for gradient descent, with an exponential learning rate (LR) decay every 100 epochs. We train on the first 8 periods (1979–2010), about 2.7 million examples, where each example is 3-dimensional (lon $ \times $ lat $ \times $ time). The model takes 4 days to train for 2000 epochs on a single 80GB NVIDIA A100 GPU. Because we do not explicitly constrain the network outputs (constraining the output would imply we know the upper bound on temperature values), it is important to use activation functions that are bounded, especially at later layers of the generator $ G $ . We observed that using only ReLU/LeakyReLU layers in the generator could yield implausibly extreme samples which was rectified by the choosing more bounded activation functions (i.e., $ \tanh $ ).

4.1. Model evaluation

Because generative models attempt to learn a probability distribution $ \hat{p}\left(x,y\right) $ (or $ \hat{p}(x) $ if labels are not observed) that estimates the true probability distribution $ p\left(x,y\right) $ (where $ x $ represents the data sample or its features and $ y $ represents the labels/class) of data, it is important to develop methods/metrics for evaluation that quantify how well a model performs this task. This can be achieved by estimating the likelihood that an example $ \hat{x}\sim \hat{p}(x) $ comes from the true distribution $ p(x) $ .

Using generative ML for climate variables differs from traditional ML tasks such as image, speech, or video synthesis, because models generate physical values for which a unit discrepancy has physical consequence. It is important to establish a simple yet efficient method for quantifying how well or poorly a model performs on any given climate variable, offering standard baselines that future models can be compared to. Metrics should ideally (1) be efficiently computed, (2) consistently track quality, and (3) be relevant and easily adopted by the ML, climate, and energy/power systems communities. We discuss and propose ideas for evaluation below.

If the true distribution $ p(x) $ of $ x $ is known or $ p(x) $ approximately admits a certain functional form (e.g. Gaussian), one can compute the distance between the estimates of the sufficient statistics of the generated $ \hat{p}(x) $ and true distribution $ p(x) $ . A metric that uses this approach is the Fréchet Inception Distance (FID) (Heusel et al., Reference Heusel, Ramsauer, Unterthiner, Nessler, Klambauer and Hochreiter2017), which is commonly used on GANs. FID implicitly assumes that the intermediate feature vectors extracted from images using an Inception V3 model trained on the ImageNet data set, come from a multivariate normal distribution. That is, $ {X}_r\sim \mathcal{N}\left({\mu}_{\mathbf{r}},{\mathtt{\varSigma}}_{\mathbf{r}}\right) $ and $ {X}_g\sim \mathcal{N}({\mu}_{\mathbf{g}},{\mathtt{\varSigma}}_{\mathbf{g}}) $ with $ \left({\mu}_{\mathbf{r}},{\mathtt{\varSigma}}_{\mathbf{r}}\right) $ and $ ({\mu}_{\mathbf{g}},{\mathtt{\varSigma}}_{\mathbf{g}}) $ as the mean-covariance pairs for the ground-truth and generated images, respectively. In some other cases, the distribution $ p(x) $ may not be continuous, which can make evaluation more challenging. In discontinuous cases, one may take the approach of piece-wise evaluation, if there exists a continuous form of the distribution $ {p}_q(x) $ for a certain interval $ {\mathbf{q}}_{\mathbf{0}}\le \mathbf{q}\le {\mathbf{q}}_{\mathbf{1}} $ . And, if the distribution within these intervals admits a known functional form, then a weighted average of the distances between the sufficient statistics for all intervals can be adopted; this approach is limited to real-valued distributions.

In some other cases, a functional form of $ p $ is unknown, thus a non-parametric goodness of fit (e.g. Kolmogorov–Smirnov statistic) test can be adopted, or entropy-based (Kullback–Leibler (KL) and Jensen–Shannon (JS)) (Kullback and Leibler, Reference Kullback and Leibler1951; Lin, Reference Lin1991) methods can be leveraged. The JS divergence is generally accepted as the symmetric distance measure borne from the KL divergence, thus can be formally considered a metric. For evaluating a generative model when the functional form of $ p $ is unknown, one can empirically estimate the JS-divergence by taking the following steps:

1. 1. Choose the number of bins $ n $ , which decides how many quantile intervals the datapoints will be placed into,

2. 2. Sort the data and place every datapoint/sample into a bin corresponding to its quantile range within the dataset,

3. 3. Empirically calculate $ {D}_{\mathrm{JS}}\left(\boldsymbol{P}\Big\Vert \boldsymbol{Q}\right) $ for each bin and then compute the average JS-divergence.

We discuss and formalize the evaluation of TemperatureGAN for the rest of this section.

4.1.3. Spatial pixel-wise average correlation distance (SPAC’D)

We introduce SPAC’D (pronounced “spaced”). To evaluate the spatial representation, we leverage the idea of covariance. Specifically, we calculate the L1-norm distance of the Pearson product–moment correlation coefficient matrices (Benesty et al., Reference Benesty, Chen, Huang and Cohen2009) between the ground-truth and generated data samples. It is worth noting that the choice of the L1-norm is not arbitrary. We elect the L1-norm because it ensures the metric has a fixed range, making it suitable for frames with varying length and width; an intuitive description of SPAC’D is included in Appendix B.1. SPAC’D estimates how well the pixel-wise correlations (or relationships), in the ground-truth are replicated by the generator. It has a range $ \left[0,2\right] $ , with the quality of spatial representation increasing with decreasing value. Additionally, it is resolution-invariant, meaning the spatial resolution (or size) of the samples being evaluated does not alter its range. We define SPAC’D below with $ \overset{\sim }{T}\sim {\unicode{x2119}}_{\mathrm{g}} $ and $ T\sim {\unicode{x2119}}_{\mathrm{r}} $ , where the subscripts $ r $ and $ g $ represent the generated and ground-truth data, respectively.

(4) $$ \mathrm{SPA}{\mathrm{C}}^{\prime}\mathrm{D}=\frac{1}{N}{\left\Vert {\rho}_{\overset{\sim }{T},\overset{\sim }{T}}-{\rho}_{T,T}\right\Vert}_1 $$

where $ N=W\times D $ is the total number of pixels per frame with W and D representing the number of pixels in the x and y directions. $ {\left\Vert \cdot \right\Vert}_1 $ , a matrix L1-norm is the maximum sum of absolute values of the column vectors of a matrix. For any matrix $ A $ , the L1-norm is given by:

(5) $$ {\left\Vert A\right\Vert}_1=\underset{x\ne 0}{\sup}\frac{{\left\Vert Ax\right\Vert}_1}{{\left\Vert x\right\Vert}_1}=\underset{j}{\max}\sum \limits_{i=1}^n{a}_{ij} $$

$ \rho $ is the correlation coefficient matrix with each entry given by Equation (6) below.

(6) $$ {\rho}_{x,y}=\frac{\unicode{x1D53C}\left[\left(x-{\mu}_x\right)\left(y-{\mu}_y\right)\right]}{\sigma_x{\sigma}_y};{\sigma}_x{\sigma}_y>0, $$

where $ x $ and $ y $ represent the two features for which $ {\rho}_{x,y} $ is calculated.

We call TemperatureGAN ‘ $ G $ ’ and the baseline ‘ $ B $ ’. We sample temperature maps $ {T}_G\sim G\left(z,M,R,k\right) $ and $ {T}_{\mathrm{B}}\sim B\left({\hat{\mu}}_{\mathrm{S}},{\hat{\sigma}}_{\mathrm{S}}\right) $ (where each sample $ {T}_{\mathrm{G},\mathrm{B}}\in {\mathcal{R}}^{24\times 8\times 8} $ ), from the GAN and baseline, respectively and report this metric for different regions. Because we are introducing this metric for the first time in this regime, we cannot compare it to other existing models, but it offers a baseline for future comparison. The plots in Figure 10 show the SPAC’D values over training steps. The initial examples produced by the GAN, display inferior SPAC’D values compared to the baseline. Because, during the earlier stages of training, the parameters of the neural network are in proximity to their randomly initialized values. However, later into training (see plots on the right and notice training steps), there is a significant decline (improvement) in the SPAC’D values. This shows that not only are the generated temperature ranges (distribution) accurate, the spatial temperature fields generated have structure.

4.1.4. Fréchet daily-mean temperature distance (FDTD)

In addition to evaluating spatial correlations, it is important to evaluate the temperature values being generated. By visual inspection of Figure 6 and 7, observe that TemperatureGAN generates realistic temperatures for the given regions and months. However, we propose a metric to measure its performance. Daily mean temperatures are typically assumed to follow Gaussian distributions (Meehl et al., Reference Meehl, Karl, Easterling, Changnon, Pielke, Changnon, Evans, Groisman, Knutson and Kunkel2000). Thus, we compute the distance between the sufficient statistics of the ground-truth and generated data as parameterized by a Gaussian. For a certain region $ R $ , in a given month $ M $ , and period $ k $ , the FDTD is calculated by taking the daily average temperatures across every observation. Then, the central bulk of the data is estimated by a normal distribution. The bulk is chosen as the 10th to 90th percentile daily mean temperatures from the ground-truth data, excluding the tails, as they usually admit a different functional form, usually characterized by generalized extreme value (GEV) or generalized pareto (GPD) distributions. Thus, for a set of daily mean temperatures $ \overline{\mathbf{T}}=\left\{{\overline{T}}_1,{\overline{T}}_2,{\overline{T}}_3,\dots, \right\} $ , we select a subset $ {\overline{\mathbf{T}}}_{\mathbf{bulk}}\subset \overline{\mathbf{T}} $ , fit a normal distribution to $ {\overline{\mathbf{T}}}_{\mathbf{bulk}} $ , and compute the distance of the sufficient statistics of the generated examples from the ground-truth data as expressed in Equation (7).

(7) $$ FDTD=\sqrt{{\left\Vert {\mu}_{\mathrm{r}}-{\mu}_{\mathrm{g}}\right\Vert}_2^2+{\left\Vert {\sigma}_{\mathrm{r}}-{\sigma}_{\mathrm{g}}\right\Vert}_2^2} $$

Figure 7.

Ground-truth (top) and generated (bottom) hourly snapshots of samples of a winter day in Nevada. (2011–2014).

Figure 8.

Histograms with kde plots comparing ground-truth (blue) and generated (red) empirical distribution plots for a $ {1}^{\circ}\times {1}^{\circ } $ region around the Los Angeles California (2011–2014).

Figure 9.

Q–Q Plot envelopes for winter (9a) and summer (9b) in Nevada region.

Figure 10.

SPAC’D plots for Nevada, San Francisco Bay, Washington, and Oregon regions (top to bottom). The left plots represent initial training steps and the right plots display latter training steps. The red line is the baseline for which the model is compared with at various regions (1979–1982). It is evident that the model learns some of the structure of the temperature fields later into training.

Figure 11.

Temporal gradients distribution plots, Los Angeles (1979–1982).

Figure 12.

Histograms with kernel density estimate plots for maximum daily temperatures. Nevada (1979–1982).

Figure 13.

Four-year timeseries plots of daily maximum temperatures for the ground truth, GAN, and WeaGETS. Nevada (1979–1982).

The subscripts $ r $ and $ g $ represent ground-truth and generated examples, respectively. $ {\mu}_{\mathrm{r}} $ and $ {\mu}_{\mathrm{g}} $ are ground-truth and generated data sample means respectively, and $ {\sigma}_{\mathrm{r}} $ and $ {\sigma}_{\mathrm{g}} $ are the ground-truth and generated data sample standard deviations, respectively. The results are reported in Tables 1–4, and B2–B4.

Table 1.

FDTD (K) for period 0 (1979–1982), San Francisco Bay area avg. distance 0.4 K/°C

Table 2.

FDTD (K) for period 8 (2011–2014), San Francisco Bay area avg. distance 0.6459 K/° C (Note: model was not trained on data from this period)

Table 3.

FDTD (K) for period 0 (1979–1982), Portland, Oregon area avg. distance 0.4750 K/°C

Table 4.

FDTD (K) for period 8 (2011–2014), Portland, Oregon area avg. distance 0.3655 K/°C (Note: model was not trained on data from this period)

4.1.5. Temporal gradient distribution distance (TGDD)

For some applications, generating realistic hourly temperature maps can be useful; for example, power system reliability or capacity sufficiency studies may benefit from this (Panteli and Mancarella, Reference Panteli and Mancarella2015; Perera et al., Reference Perera, Nik, Chen, Scartezzini and Hong2020). Because we generate hourly spatial maps for any given day, we aim to estimate the integrity of the generated diurnal cycle. Visual inspection may be used to validate general cyclical patterns. However, we propose an approach to numerically estimating the model’s performance. We obtain the distribution of temperature gradients by empirically estimating the cumulative density functions (CDFs) of the temperature hourly gradients $ \frac{\partial \mathbf{T}}{\partial \mathbf{t}} $ and $ \frac{\partial \tilde{\mathbf{T}}}{\partial \mathbf{t}} $ , where $ \boldsymbol{T}\sim {\unicode{x2119}}_{\mathrm{r}} $ and $ \overset{\sim }{\boldsymbol{T}}\sim {\unicode{x2119}}_{\mathrm{g}} $ and $ {\unicode{x2119}}_{\mathrm{r}} $ and $ {\unicode{x2119}}_{\mathrm{g}} $ represent the ground-truth and generated distributions (see Figure 11), respectively. To do this, we split the data samples into $ n=10 $ bins, estimate $ {D}_{\mathrm{JS}}\left(p\Big\Vert q\right) $ for each bin,

(8) $$ {D}_{\mathrm{JS}}\left(p\Big\Vert q\right)=\frac{1}{2}{D}_{\mathrm{KL}}\left(p\Big\Vert \frac{p+q}{2}\right)+\frac{1}{2}{D}_{\mathrm{KL}}\left(q\Big\Vert \frac{p+q}{2}\right) $$

after which we compute the average across all bins to obtain the metric we call TGDD,

(9) $$ \mathrm{TGDD}=\frac{1}{N}{\sum}_{i=1}^N{D}_{\mathrm{JS}}\left({p}_i\Big\Vert {q}_i\right), $$

with a range $ \left(0,\ln 2\right) $ , and improving with decreasing value. We report the monthly TGDD in Table 5. Diurnal cycle patterns are displayed in Figure B1 in Appendix B for visual inspection.

Table 5.

TGDD values. $ {k}_0 $ (1979–1982)