Impact Statement
As global deployment of wind turbines continues to rise toward tackling climate, there are growing concerns on operations and maintenance in cold climates. To prevent significant downtimes or damage to the turbines, preventive intervention is critical by facilitating early ice detection. Existing research has used artificial intelligence (AI) for ice detection in standalone wind parks, but fails to generalize to new, unseen wind parks. We perform domain-invariant icing detection independent of the wind parks that the AI models have been trained on, and predict icing on the face of lack of ice images in new wind parks. This can help make ice detection with AI scalable in the wind industry, facilitating wider adoption across different wind parks around the globe.
1. Introduction
With a growing awareness of the pressing need to transition to renewable energy sources globally toward combating climate change, the total global installed wind power capacity has touched over 837 GW in 2022 (Global Wind Energy Council, 2022). This fact shows that the energy transition is possible and wind energy contributes to around one-fifth of the electricity generated by renewable sources (Global Wind Energy Council, 2022). Winters are generally considered to be highly promising for wind power generation, owing to higher wind speeds and increased air density accompanied with low prevailing temperatures (Gao and Hu, Reference Gao and Hu2021). In particular, cold regions are usually sparsely populated, have a good wind availability and are suitable for deployment of wind parks (Clifton et al., Reference Clifton, Barber, Stökl, Frank and Karlsson2022). However, such regions, particularly in Northern Europe and North America, are highly prone to icing conditions on the wind turbine rotor blades—leading to high stress on the overall structure of the turbines that prohibits their safe operation (Kreutz et al., Reference Kreutz, Ait-Alla, Varasteh, Oelker, Greulich, Freitag and Thoben2019; Alvela Nieto et al., Reference Alvela Nieto, Gelbhardt, Ohlendorf, Thoben, Valle, Lehmhus, Gianoglio, Ragusa, Seminara, Bosse, Ibrahim and Thoben2023). The ice throw scenario, which is considerably worse, occurs when the accumulated ice breaks free from the blades and can harm adjacent living beings. Such icing events not only give rise to unexpected downtimes, but also reduce the potential energy yield and shorten the mechanical lifetime of the turbines (Wallenius and Lehtomäki, Reference Wallenius and Lehtomäki2016).
Ice formation on wind turbine rotor blades is influenced by a number of factors, including external temperature, wind speed, humidity, and so forth. But also less common factors, such as the liquid water content and median volume diameter of water droplets can have an influence. The original Messinger model (Messinger, Reference Messinger1953) and the Makkonen model (Makkonen, Reference Makkonen2000) are only a few examples of physical models that describe the production of ice on rotor blades. These models typically call for additional parameters unique to wind turbines—mostly related to wind turbine SCADA data (Chatterjee and Dethlefs, Reference Chatterjee and Dethlefs2020; Turnbull et al., Reference Turnbull, Carroll and McDonald2021) in addition to the ones already discussed. Existing methods and models are complex and require certain parameters that are not typically available for wind turbine systems. As a potential solution, there has been a rising interest in leveraging color images (RGB) of turbine rotor blades acquired through cameras installed on the nacelles and the application of computer vision techniques for detecting ice accretion on the surface of the blades (Denhof et al., Reference Denhof, Staar, Lütjen and Freitag2019; Kabardin et al., Reference Kabardin, Dvoynishnikov, Gordienko, Kakaulin, Ledovsky, Gusev, Zuev and Okulov2021). A camera generally captures images of the complete rotor blade even under harsh weather conditions (e.g., foggy environments), making this technique more robust than some sensor-based approaches (Chatterjee and Dethlefs, Reference Chatterjee and Dethlefs2020) and suitable for remote autonomous inspection (Gu et al., Reference Gu, Hu, Cheng, Cao, Rizzo and Valavanis2020). As a potential benefit of leveraging preventive maintenance strategies for condition monitoring (Stetco et al., Reference Stetco, Dinmohammadi, Zhao, Robu, Flynn, Barnes, Keane and Nenadic2019) through computer vision-based icing prediction systems (i.e., before ice accretion can lead to unexpected downtimes or total shutdowns)—rotor blades can be monitored and heating systems can be triggered in advance to prevent ice formation (Merizalde et al., Reference Merizalde, Hernández-Callejo, Duque-Perez and Alonso-Gómez2019).
There has been very limited research on applying intelligent algorithms, for example, artificial intelligence (AI), for ice recognition on rotor blades. Only a few studies have applied deep learning (DL) methods for detecting icing based on image data (Kreutz et al., Reference Kreutz, Alla, Eisenstadt, Freitag and Thoben2020; Alvela Nieto et al., Reference Alvela Nieto, Gelbhardt, Ohlendorf, Thoben, Valle, Lehmhus, Gianoglio, Ragusa, Seminara, Bosse, Ibrahim and Thoben2023). For instance, Alvela Nieto et al. (Reference Alvela Nieto, Gelbhardt, Ohlendorf, Thoben, Valle, Lehmhus, Gianoglio, Ragusa, Seminara, Bosse, Ibrahim and Thoben2023) used CNN models with unfrozen backbones and learning schedulers for detecting ice events on images, with no data augmentation. Basically, the models were created using data of individual wind turbines, as the merge of data from two or more wind turbines led to lower performance. In real-world settings, wind turbine designs vary from stakeholder to stakeholder, as do the rotor blade forms, colors, and lengths, but also the turbine’s surroundings (i.e., landscape, light pollution), which can entail a drop of accuracy in model predictions. Further, the installation of new wind parks, consisting of individual design characteristics and new locations, initially requires a phase of data collection for model creation. While this study has achieved near-perfect accuracy in individual wind turbine data sets (Alvela Nieto et al., Reference Alvela Nieto, Gelbhardt, Ohlendorf, Thoben, Valle, Lehmhus, Gianoglio, Ragusa, Seminara, Bosse, Ibrahim and Thoben2023), our experiments showed that: (a) Such models generally perform poorly in generalizing to new turbines and/or wind parks. (b) Occurrence of ice events within the first few days after turbine installation is extremely unusual, prompting the application of methods for generating rotor blade images of the newly installed turbine containing ice for better model generalization. We aim to tackle all these challenges by facilitating domain adaptation using generative AI. The goal of this article is to ensure that a model trained on data from a specific wind park (source domain) is able to make effective predictions in new wind park locations (target domain), where no ice images or only a few are available. We propose the utilization of synthetic data augmentation performed with generative AI models. Our proposed technique for the problem task in this paper is a neural-style transfer algorithm with CNNs. We leveraged a state-of-the-art method in generative AI modeling—CycleGAN for comparison against the proposed method. To the best of our knowledge, this is the first study to propose synthetic data augmentation with two different generative AI strategies toward improving the predictive performance of existing intelligent models for detecting domain-invariant icing events in the wind industry.
Our study showcases that the neural style transfer algorithm helps to improve the generation of synthetic images that can capture transferable fine-grained icing representations that are not bound to a specific wind park. By training the domain-specific models with the curated synthetic data, the generalizability of the models is improved toward detecting icing across different wind parks. Notably, the main contribution of this paper is its specific focus on application to new wind parks which do not have any historical data available for icing. In our study, a new wind park signifies that there is no data (RGB images) of the rotor blade (neither plain nor with icing) available. In the initial few days of operation of a new wind park, we can collect background and plain rotor blade images, which can then be augmented with icing characteristics from another wind park to facilitate ice detection in the new park. This is based on the foundation that icing characteristics across different wind parks would be similar. The training of a classifier without synthetic data, however, assumes that images containing ice are available. That is not always the case, specially after turbine installation in summer periods. Note that this is in contrast to the idea of training a classification model combining all the images from both wind parks, as it would still require images from the new wind park. Our approach ensures that there is no data augmentation period for the new wind park since real-time images can be instantaneously analyzed without any requirement for additional data acquisition of icing images beforehand. This can help facilitate the adoption of such domain-agnostic models across multiple wind parks in the near future for preventive maintenance of turbines, providing instrumental decision support for making wind turbines more reliable toward tackling climate change.
2. Related Work
Planning the location of wind turbines can sometimes be challenging—for instance, due to legal safe distance requirements prohibiting close proximity to residential properties. Therefore, less populated locations are often more suited for the installation of new wind turbines. In particular, cold regions at high altitudes are typically unpopulated and have strong wind availability, making them attractive locations for the placement of wind turbines (Ibrahim et al., Reference Ibrahim, Ghandour, Dimitrova, Ilinca and Perron2011). In such regions, there is a substantial risk of ice formation on the rotor blades during the winter (Fakorede et al., Reference Fakorede, Feger, Ibrahim, Ilinca, Perron and Masson2016). Moreover, the available wind power is nearly 10% higher in such areas than in other regions (Fakorede et al., Reference Fakorede, Feger, Ibrahim, Ilinca, Perron and Masson2016). Severe icing events can occur for up to 30 days a year. Such unexpected events lead to forced shutdowns of wind turbines during ice accretion, leading to significant downtimes and financial loss for wind park operators.
Many techniques exist that can combat the problem of icing on rotor blades and similar structures. Deicing removes the ice once it has been formed on the structure. A heater at the blade in combination with an ice sensor is one valid solution. In contrast, anti-icing prevents any ice from forming at all. Preheating the blade is an active anti-icing method so that icing events do not occur. Most of the ice detection systems mounted on rotor blades at present vary greatly in terms of their quality and are not sufficiently accurate (Parent and Ilinca, Reference Parent and Ilinca2011; Kreutz et al., Reference Kreutz, Alla, Eisenstadt, Freitag and Thoben2020). Additionally, these approaches suffer from several drawbacks (Parent and Ilinca, Reference Parent and Ilinca2011)—such as the inability to provide direct ice measurements owing to dependence on secondary operational parameters including temperature or oscillation frequencies.
Despite the rapid and continuous decline in levelized cost of electricity in recent years (Wiser et al., Reference Wiser, Rand and Seel2021), the wind industry still faces multiple operational challenges. Some of the current anti-icing or deicing techniques still rely on forced shutdown of the wind turbine. When there is a severe risk of ice projection, the wind turbine pitch motors cannot supply the turbine and the system simultaneously (Fakorede et al., Reference Fakorede, Feger, Ibrahim, Ilinca, Perron and Masson2016). To overcome icing effects, precise ice detection techniques are a necessity as a first step (Parent and Ilinca, Reference Parent and Ilinca2011). Existing ice detection techniques are based on various concepts, which can be classified into two main categories: (a) indirect and (b) direct approaches. These are described below in further detail.
2.1. Indirect approaches
Fikke et al. (Reference Fikke, Ronsten, Heimo, Kunz, Ostrozlik, Persson, Sabata, Wareing, Wichura, Chum, Laakso, Säntti and Makkonen2007) listed the available prototypes of ice detectors in the market. External weather parameters, such as temperature and humidity, combined with deterministic models, can be utilized to predict icing events indirectly (Molinder et al., Reference Molinder, Körnich, Olsson, Bergström and Sjöblom2018). Deterministic models require several parameters which must be provided as inputs in order to characterize the beginning or end of all types of icing events (wind, humidity, precipitation, etc.) (Wei et al., Reference Wei, Yang, Zuo and Zhong2020). Unfortunately, such parameters are not always measured in a standard way for icing risk assessment, especially in older-generation wind turbines (which may or may not have sensors for detecting such external parameters). Nevertheless, it may be possible to use weather forecasting models to derive an approximation of these values (Fikke et al., Reference Fikke, Ronsten, Heimo, Kunz, Ostrozlik, Persson, Sabata, Wareing, Wichura, Chum, Laakso, Säntti and Makkonen2007; Kreutz et al., Reference Kreutz, Alla, Lütjen, Ohlendorf, Freitag, Thoben, Zimnol and Greulich2023). However, dependence on approximations derived from such weather forecasting models (often based on numerical simulations) makes the operations and maintenance (O&M) process more complex, expensive, and not entirely reliable.
Despite the prevalence of existing sensor-based systems, there are still no industry standards in place which clearly define the values of the parameters for the tools—such as the threshold ratio for icing. Others have a coarse accuracy, which makes such sensors unstable for measuring light icing events. Pivotal measured parameters for detecting ice on the rotor blades are commonly used, like the droplet size distribution (Cattin et al., Reference Cattin, Heikkilä, Bourgeois, Raupach and Storck2016). In summary, most existing studies on indirect approaches have outlined that all systems have their shortcomings under specific conditions. In summary, and to our understanding, none of the existing ice detectors mentioned here have leveraged intelligent algorithms for detecting icing events.
2.2. Direct approaches
Direct ice detection techniques analyse changes in physical parameters caused by icing and can depend on (amongst others) inductance or optic systems (Battisti, Reference Battisti2015; Fakorede et al., Reference Fakorede, Feger, Ibrahim, Ilinca, Perron and Masson2016; Wei et al., Reference Wei, Yang, Zuo and Zhong2020). The latter are able to accomplish this on a thermal (heated wires) or optical (laser scanning) basis (Cattin, Reference Cattin2012). The main drawback of thermal solutions is that they are either very expensive or not able to derive the measurements automatically. Additionally, there can be large uncertainties depending on the utilized approach (Cattin, Reference Cattin2012). Another recent direct approach is the use of optical cameras to detect icing on the blades of a wind turbine. A camera is mounted on the spinner of the rotor, so that the camera is rotating and always pointing to the rotor blades. However, a key disadvantage of this approach is that the camera is strongly exposed to the external weather and the view angle to the blade is very flat. This makes it highly challenging to monitor the icing status on the tip of the blades (Wallenius and Lehtomäki, Reference Wallenius and Lehtomäki2016).
Previous studies (Zhang et al., Reference Zhang, Liu, Wang and Omariba2018; Liu et al., Reference Liu, Cheng, Kong, Wang and Cui2019; Yuan et al., Reference Yuan, Wang, Luo, Jiang, Long, Yu and Liu2019; Jiang and Jin, Reference Jiang and Jin2021; Kreutz et al., Reference Kreutz, Alla, Varasteh, Ohlendorf, Lütjen, Freitag and Thoben2021; Kemal et al., Reference Kemal, Başağa, Yavuz and Karimi2022) have utilized image data from automated optical cameras to train machine learning models (e.g., decision trees) as well as leveraged DL techniques for ice detection. They have all demonstrated near-perfect accuracy with regards to the degree of detecting ice, including near and far views of a blade, ice density, and light (Alvela Nieto et al., Reference Alvela Nieto, Gelbhardt, Ohlendorf, Thoben, Valle, Lehmhus, Gianoglio, Ragusa, Seminara, Bosse, Ibrahim and Thoben2023). However, it is integral to note that the characteristics of a wind park are unique. In the case where a neural model has been developed from a single scenario, the model compatibility with all scenarios (e.g., all rotor blade types) is not always implied.
Generative AI models such as CycleGAN have shown immense success toward synthetic data generation in multiple domains (including safety-critical areas) like healthcare (Sandfort et al., Reference Sandfort, Yan, Pickhardt and Summers2019; Motamed et al., Reference Motamed, Rogalla and Khalvati2021), remote sensing for wildfire detection (Park et al., Reference Park, Tran, Jung and Park2020), flood event detection (Pouyanfar et al., Reference Pouyanfar, Tao, Sadiq, Tian, Tu, Wang, Chen and Shyu2019), road surface detection for autonomous vehicles (Choi et al., Reference Choi, Heo and Ahn2021) and so forth. Particularly, the utilization of CycleGAN was helpful to make models from a single satellite compatible with all the satellites of the constellation. GANs have also been used for condition monitoring of various types of wind turbine gearboxes (Zhang et al., Reference Zhang, Wu, He, Lou and Gao2020).
In our study, the automated camera was mounted on the nacelle catching the blade by motion detection when it moves through the image. This approach has the advantage that it is not necessary to put a hole into the spinner, the camera is less exposed and the view angle toward the blade is more steep, that is, it is possible to see the whole blade in perspective. Further, this paper focuses on a different strand of research compared to existing studies. The use of algorithms based on generative AI detects icing on different blades from the RGB-camera images automatically—while most previous studies rely on manual analysis. The generation of synthetic data facilitates better compatibility of DL models originally developed for one specific wind turbine (i.e., rotor blades of a single turbine) to any other wind turbines.
3. Dataset Description and Preprocessing
We utilized RGB images for our study that were recorded by cameras in two real-world wind parks—wind park A located in North America and wind park B located in Northern Europe. Note that the images acquired from wind park A are of significantly higher quality than wind park B owing to a better camera quality. The images were manually labeled into three classes by two humans (with cross-validation also performed between the labels)—no rotor blade on foreground, rotor blade without icing and rotor blade with icing. We experiment with both scenarios—wind park A as source domain and wind park B as target domain and vice versa. The training data of the base sets contains 150 background images, 20 rotor blade images plus 50 rotor blade images from the target domain and 70 icing images from the source domain. The rotor blade and ice images are augmented with up to 10% random rotation, reaching 400 images. The test data includes around 200 images of each class for wind park A and around 800 for wind park B. Examples of images belonging to the datasets used in this paper are available for the interested reader in Alvela Nieto et al. (Reference Alvela Nieto, Gelbhardt, Ohlendorf, Thoben, Valle, Lehmhus, Gianoglio, Ragusa, Seminara, Bosse, Ibrahim and Thoben2023).
4. Proposed Methodology and Learning Models
We intend to utilize existing DL models which have already achieved success in domain-specific ice detection in past literature as baselines, including MobileNetV2, VGG-19, and Xception. We utilized the same dataset types from wind parks used in the past study in our experiments for fair comparison (Alvela Nieto et al., Reference Alvela Nieto, Gelbhardt, Ohlendorf, Thoben, Valle, Lehmhus, Gianoglio, Ragusa, Seminara, Bosse, Ibrahim and Thoben2023). However, in this study, we aim to develop models that are fine-tuned to the domain-specific target using generative AI for better generalization to other wind parks. As successful training of CNN models for domain-specific applications requires substantial amounts of data, the standalone networks generalize poorly on the face of small, limited datasets with significant class imbalance (Motamed et al., Reference Motamed, Rogalla and Khalvati2021). As the existing models are domain-specific and bound to distinct wind parks, we apply transfer learning to accomplish generalized ice detection that is independent of characteristics of the wind parks that the models have previously been trained on. Consider the RGB images from wind park A as the source domain, and the images from wind park B as the target domain (or vice versa). The target domain is significantly different from the source because of the varying rotor blade shape, background of the geographical area, quality of the recorded images and so forth. However, both domains show some similarities regarding the presence or absence of ice. The goal is to train models which can make more effective predictions for the target domain, when trained only with ice images from the source domain. For accomplishing domain adaptation, we propose to perform synthetic data augmentation which is eventually utilized to train the standalone ice-prediction models. We experiment with two different approaches for generating synthetic data—one being a state-of-the-art method in unsupervised image translation (CycleGAN) and another being a more traditional method (neural style transfer algorithm). Both approaches are described below and will be compared later in Section 6.
4.1. CycleGAN for unpaired image-to-image translation
Image-to-image translation (Liu et al., Reference Liu, Breuel and Kautz2017) is a popular computer vision task that aims to automatically map images in one domain onto corresponding images in another domain. Conventional image-to-image translation relies on supervised learning models requiring labeled pairs of images between the source and target domains, which is rarely available in many real-world practical problems. This includes the wind industry in which there is often a lack of historically paired images across different scene representations (e.g., image of the same wind turbine rotor blade in summer and winter, day and night). Owing to this challenge, unsupervised image-to-image translation models are recently gaining prominence—as they do not require a paired dataset of images or additional domain expertise and can simply leverage existing unlabelled datasets (Hoyez et al., Reference Hoyez, Schockaert, Rambach, Mirbach and Stricker2022).
CycleGANs (Zhu et al., Reference Zhu, Park, Isola and Efros2017) are a type of generative adversarial network (GAN) model that can be used for unpaired image-to-image translation. They rely on unsupervised learning to automatically map patterns between images from two different domains—and thus learn to translate the images from one domain to the style of the other domain. The CycleGAN is based on the typical GAN architecture consisting of:
-
1. Generator: The generator is a deep neural network (DNN) that aims to generate new (fake) images which would match the style of the target domain images, while preserving the underlying content of the source domain images. It achieves this by receiving continuous feedback from the discriminator as described next.
-
2. Discriminator: The discriminator is another DNN that checks whether the new image generated by the generator is actually a real image (i.e., a copy of the original target domain images), or a truly novel (fake) image that modifies the source domain image to the style of the target domain. It acts as a basic classification model that aims to distinguish between real and fake images generated by the generator.
While the traditional GAN only has a single generator and a discriminator, the CycleGAN consists of two generators and two discriminators. It uses them to compute the cyclic-consistency loss—a loss metric that is used when translating images from domain A to B (or vice versa) by leveraging a mapping function G and mapping the translated image back to A (or vice versa) by utilizing a different mapping function F to quantify how close the generated (fake) image is to the original (real) image. The CycleGAN aims to learn a mapping
$ G:\mathrm{Domain}\hskip0.35em A\to \mathrm{Domain}\hskip0.35em B $
during its training process to ensure that the probabilistic distribution of the images obtained with the function
$ G(X) $
is identical to the probabilistic distribution of domain B images based on an adversarial loss (Zhu et al., Reference Zhu, Park, Isola and Efros2017). The model also learns an inverse mapping
$ F:\mathrm{Domain}\;B\to \mathrm{Domain}\;A $
and computes the cyclic-consistency loss
$ {\mathrm{\mathcal{L}}}_{\mathrm{cyc}}\left(G,F\right) $
:
$$ {\displaystyle \begin{array}{l}{\mathrm{\mathcal{L}}}_{\mathrm{cyc}}\left(G,F\right)={E}_{x\sim {p}_{\mathrm{data}}(x)}\left[\parallel F\left(G(x)\right)-x{\parallel}_1\right]\\ {}\hskip8.7em +{E}_{y\sim {p}_{\mathrm{data}}(y)}\left[\parallel G\left(F(y)\right)-y{\parallel}_1\right],\end{array}} $$
wherein,
$ x $
represents the images in training data for domain A (
$ x\hskip0.35em \in \hskip0.35em A $
),
$ y $
represents the images for domain B (
$ y\hskip0.35em \in \hskip0.35em B $
) and
$ E $
represents the expected values to ensure that
$ F\left(G(X)\right)\approx X $
(or vice versa) by minimizing the value of the cyclic-consistency loss. We propose to utilize the CycleGAN model for learning to translate plain rotor blade images in domain A (without ice, as obtained during the summers) to rotor blade images with icing (as evident in the winter) in domain B. Figure 1 describes the proposed framework, wherein, CycleGAN is used as a generative AI model. Note that for simplicity, only one part of the CycleGAN is shown in this figure with one generator and one discriminator—there would be another similar generator and discriminator that would learn to translate from domain B to A. For more details on the generic model architecture of CycleGAN, the interested reader is referred to Liu et al. (Reference Liu, Breuel and Kautz2017) and Sinha (Reference Sinha2021).
Figure 1. Framework for leveraging CycleGAN for translating plain rotor blade images (without ice) into rotor blade images (with icing)—CycleGAN generic model architecture.
4.2. Neural style transfer algorithm
Neural style transfer with CNNs (Gatys et al., Reference Gatys, Ecker and Bethge2015; Jing et al., Reference Jing, Yang, Feng, Ye, Yu and Song2020) is a more traditional method of facilitating domain adaptation in computer vision tasks. The algorithm employs an optimization technique in a pretrained CNN (e.g., based on weights from VGG-16, VGG-19) to transform a content image to the style of a reference style image—while ensuring that the weighted sum of the content loss and style loss functions computed across the content, style and generated stylized images are minimized (Ganegedara, Reference Ganegedara2020; Jing et al., Reference Jing, Yang, Feng, Ye, Yu and Song2020). The weighted sum is represented as:
$$ L=\alpha {L}_{\mathrm{content}}+\beta {L}_{\mathrm{style}}, $$
where
$ {L}_{\mathrm{content}} $
represents the content loss,
$ {L}_{\mathrm{style}} $
represents the style loss and
$ \alpha $
,
$ \beta $
are user-specified hyper-parameters during the CNN training process.
Note that while the CycleGAN does not require a corresponding style image to generate stylized images as it directly learns transferable representations from the content image, we observed that the CycleGAN generated suboptimal results for our problem compared to the neural style transfer algorithm. For our study, we used a combination of two different approaches for synthetic image generation with the neural style transfer algorithm:
-
1. A VGG-19 model architecture (Simonyan and Zisserman, Reference Simonyan and Zisserman2014; Kavitha et al., Reference Kavitha, Dhanapriya, Vignesh and Baskaran2021), originally pretrained with weights from ImageNet for image classification.
-
2. A pretrained fast style transfer model leveraging arbitrary image stylization (Ghiasi et al., Reference Ghiasi, Lee, Kudlur, Dumoulin, Shlens, Kim, Zafeiriou, Brostow and Mikolajczyk2017) (no fine-tuning required).
We leveraged images from the source domain that represent the ice texture on the rotor blades as style images. Note that the icing characteristics on wind turbine rotor blades are domain-invariant irrespective of rotor blade designs in the different wind parks—we aimed to reproduce this characteristic in our study. The goal is to modify the plain rotor blade images of the target domain (content images) with the icing characteristics of the source domain, to improve the generalizability of the model.
We realized that directly modifying the content images led to distorted background as well. To specifically modify the parts of the source image which contain the rotor blade, we followed an overlaying process—wherein, a mask is applied to the content image and a reverse mask is applied to the generated stylized image. The overlayed images are preprocessed (as discussed in Section 5) and finally utilized for training the CNN models for generalized ice detection. Figure 2 depicts the complete process of the proposed approach. It is integral to highlight that with the above process, the source domain images are only utilized for acquisition of the ice textures to modify the target domain plain rotor blade images. In simple terms, we have cropped-out the ice textures from source domain images before using them as style images, which is based on the foundation that ice textures will be same in both the source as well as target domains and are independent of wind park locations. Additionally, the above method ensures that we are not modifying the whole target domain image, but only the segment of the plain rotor blade (without ice) since the background is removed beforehand with the masking/overlaying process.
Figure 2. Framework for generalized ice detection with neural style transfer. Plain rotor blades in the target domain (B) are styled with ice from the source domain (A).
5. Experiments
Three CNN models—MobileNetV2, VGG-19, and Xception—were trained as these have achieved the best results in a past study in this domain (Alvela Nieto et al., Reference Alvela Nieto, Gelbhardt, Ohlendorf, Thoben, Valle, Lehmhus, Gianoglio, Ragusa, Seminara, Bosse, Ibrahim and Thoben2023). Before feeding the images to the CNN models, the default preprocessing steps (e.g., reshaping the images) were followed. Two distinct strategies were used to train the CNN models:
-
1. Unfrozen backbone: An output layer (dense layer, three classes) was appended to the model and all model layers were trainable.
-
2. Frozen backbone: The generic model backbone was frozen and only the output layer (dense layer, three classes) appended to the model was trainable.
For both unfrozen and frozen CNN backbone strategies, we used Stochastic Gradient Descent (SGD) for optimization with an initial learning rate of 0.0015 and momentum of 0.9, and a learning rate scheduler that decreases the learning rate every 3 epochs by a factor of 0.94. The models were trained for 30 epochs with a batch size of 16. We used a random seed value of 42 throughout all our experiments in this study. Below, we describe the specific experimental details for the CycleGAN and the neural style transfer algorithm respectively—which are the two different strategies for synthetic data generation in this study.
5.1. CycleGAN
For generating synthetic data with the CycleGAN, we experimented by considering three different scenarios:
-
1. Training CycleGAN with wind park A plain rotor blade images as source domain, wind park A rotor blade with ice images as target domain, wind park B plain rotor blade images as test data during inference. The goal here is to transform the plain rotor blade images of wind park B to rotor blade with ice images.
-
2. Training CycleGAN with wind park B plain rotor blade images as source domain, wind park B rotor blade with ice images as target domain, wind park A plain rotor blade images as test data during inference. The goal here is to transform the plain rotor blade images of wind park A to rotor blade with ice images.
-
3. Using a pretrained Summer2Winter Yosemite model,Footnote 1 with (a) wind park A and (b) wind park B plain rotor blade images as test data during inference.
For CycleGAN modeling strategies (1) and (2), we used 200 images for the background class and 400 images for every other class to train the models. An initial learning rate of 0.0002 was used for the CycleGAN. The number of training samples was the same for both wind parks. While for Wind park A the remaining 200 images per class were used as a test set, for wind park B the remaining 800 images per class were incorporated as a test set. The class containing plain rotor blades that were modified to show ice characteristics in the training set consisted of 50 unique modified plain rotor blade images that were augmented using random rotation to an amount of 400 images.
We used CycleGAN’s resize and crop method for preprocessing the images for all three strategies—with a load size of 256 and crop size of 256 to prevent losing out on the original image segments (particularly the rotor blade) during training. For all strategies, the models were trained for a total of 200 epochs (100 epochs with the initial learning rate
$ + $
100 epochs with a step-based decay in learning rate). A batch size of 16 was used during training. For all other hyper-parameters, default values were used as in Zhu et al. (Reference Zhu, Park, Isola and Efros2017). Note that the same train-test splits of the dataset are used in both the CycleGAN and the neural style transfer algorithm (described below) for fair comparison.
5.2. Neural style transfer algorithm
For neural style transfer, we leveraged the intermediate layers (without the classification head) of the VGG-19 model and applied the Adam optimizer with the following hyper-parameters: a learning rate of 0.02, beta_1 = 0.99 and epsilon = 0.1 to train the model for 40 epochs with 100 steps per epoch and modify the content with the style from our preprocessed images. As the generation of the images at this stage leads to high-frequency artifacts and significant variation loss, a denoising process for further optimization of the images over 40 epochs with 100 steps per epoch and a total variation weight of 30 was used. Default hyperparameter values for the fast style transfer model (Ghiasi et al., Reference Ghiasi, Lee, Kudlur, Dumoulin, Shlens, Kim, Zafeiriou, Brostow and Mikolajczyk2017) were applied. Fifty rotor blade images of the target domain are style-transferred to generate 200 additional synthetic images for the ice class with the previously described techniques.
6. Results
In this section, we present the experimental results obtained from leveraging the CycleGAN and neural style transfer algorithm as generative AI models for synthetic data augmentation. Note that our goal is to evaluate the feasibility and potential of the two techniques in facilitating domain-invariant learning—that is, to make effective predictions for the target domain when the models are only trained with images from the source domain.
6.1. CycleGAN
Table 1 outlines the experimental results obtained by training three different CNN models for generalized ice detection, both with and without leveraging the synthetic data generated by the CycleGAN. Two different modeling strategies (unfrozen and frozen backbones) were used as previously described in Section 5. Note that we also used two distinct target datasets for evaluating the models—wind park A and wind park B. The baseline models were trained without the synthetic images, while using the synthetic data ensures that the model makes predictions for the target dataset when only trained with images from the source domain. We observed that using the pretrained Summer2Winter Yosemite model for our problem in generating synthetic images led to poor results.
Table 1. Experimental results for ice detection before and after synthetic data augmentation with the frozen and unfrozen backbones of the CNNs for both wind parks images used as target datasets—performance with synthetic data generated with both CycleGAN and neural style transfer algorithm are compared.
Note. With CycleGAN, there is negligible improvement in the model performance compared to the baseline while there is substantial performance gain with the neural style transfer algorithm. The best F1 scores for each method (CycleGAN and neural style transfer) are highlighted in bold.
Based on the strategies of training the CycleGAN from scratch—as evident from the F1 scores, the Xception model obtains the best performance (accuracy of up to 67.5%, F1 score of 0.604) when wind park A is used as the target dataset—showcasing an accuracy gain of 3.4% compared to the baseline model (without synthetic data). When wind park B is used as the target dataset, the VGG-19 model arises as the best performer (accuracy of up to 43.1%, F1 score of 0.355)—achieving an accuracy gain of 3.5%. Figure 4 shows the confusion matrices for the results obtained before (best baseline model is Xception) and after synthetic data augmentation (best model is also Xception) with wind park A as the target dataset. There is a negligible reduction in the number of misclassifications for the ice class.
Figure 3. Comparative plot between the achieved F1-scores by models using the CycleGAN or neural style transfer methods to create synthetic data. Each point is representing one experimental setup and the corresponding number refers to the “Experiment” column in Table 1. The crosses represent the best baseline for each windpark without synthetic data. If both methods would perform equally in all experiments, all points would lay on the separating line. Clearly, synthetic data generated with style transfer outperforms the CycleGAN over all experiments and the baseline in all but two experiments (nos. 3 and 11).
Figure 4. Confusion matrices for the results based on test set of wind park A as target dataset: (a) Baseline models, no synthetic data augmentation); (b) With synthetic data augmentation through CycleGAN; and (c) With synthetic data augmentation through neural style transfer algorithm. The Xception model performed the best with both CycleGAN as well as neural style transfer. Also, neural style transfer clearly has significantly lower number of icing missclassifications compared to CycleGAN.
Clearly, the accuracy gain for both target datasets is marginal and negligible. While the results are more plausible for wind park A, the predictions are extremely poor when wind park B is used as the target dataset (which may primarily be due to the lower quality of images in wind park B compared to wind park A). These results clearly demonstrate that synthetic data augmentation with CycleGAN as the generative AI model fails to show any promise for our problem task in facilitating domain-invariant icing prediction.
6.2. Neural style transfer algorithm
Table 1 also describes the experimental results obtained when leveraging the neural style transfer for synthetic data augmentation and provides a comparison with the CycleGAN results discussed previously. The unfrozen and frozen backbone strategies and target datasets are the same—as previously described in Section 6.1. Based on the F1 score, clearly, the VGG-19 model achieves the best performance in generalized ice detection with synthetic images (see Table 1). Additionally, this model achieves an accuracy of up to 83.6% and an F1 score of 0.831 for wind park A as the target dataset, representing an accuracy gain of 19.5% compared to the best baseline model for wind park A (Xception with 64.1% accuracy and F1 score of 0.516).
While the prediction results are not as promising when wind park B is used as the target dataset (which may primarily be due to the lower quality of images in wind park B compared to wind park A), the proposed approach still showcases a noticeable performance gain compared to the baseline models. These results highlight that synthetic data augmentation with neural style transfer as the generative AI model yields improved predictions for the target domain, representing a significant boost in domain-invariant predictive capability of the standalone CNN models.
To further clearly enunciate the differences in the performances of the two discussed methods, we provide a comparative plot in Figure 3 that visually outlines the differences in performances for the CNN models to perform ice detection based on synthetic data generated by CycleGAN against synthetic data generated by the neural style transfer algorithm. This is basically based on the number of experiments highlighted in Table 1 and visually depicts the representative F1 score across all 12 experiments that are conducted (for each model—two experiments per wind park summarized across the three different models: MobileNetV2, VGG-19, and Xception as previously discussed in the paper). Clearly, the best results for ice detection in wind park A are obtained in experiment 6 (VGG-19 with frozen backbone) in case of neural style transfer, with the CNN model reaching a F1 score of 0.831. Whereas, for the very same experiment 6, with the CycleGAN, the F1 score is only 0.573. Likewise, for wind park B, experiment 8 (VGG-19 with frozen backbone) achieves the best F1 score with the neural style transfer technique, reaching 0.402. For the same experiment, the CycleGAN showcases a marginally lower F1 score of 0.355. Notably, the CNN models trained with neural style transfer generated synthetic data outperform their CycleGAN counterpart over all experiments and the baseline in all but two experiments (nos. 3 and 11).
7. Discussion
Our experiments show that the neural style transfer algorithm performs superiorly in comparison to the CycleGAN for generating realistic synthetic images to train existing CNN models to facilitate domain-invariant detection of icing. While the CycleGAN is a recent state-of-the-art model that is able to automatically identify low-level features in the data for unpaired image-to-image translation, it performs poorly in our problem task. This can be attributed to the small size and the high-class imbalance prevalent in the data—an inadvertent real-world challenge in the wind industry. We also noticed significant noise in the images generated by the CycleGAN owing to checkerboard artifacts due to deconvolution (Odena et al., Reference Odena, Dumoulin and Olah2016)—a major limitation of synthetic images generated by most GANs. As the neural style transfer algorithm performs very effectively in the experiments for our problem task, we did not consider other methods for, for example, eliminating the checkerboard artifacts prevalent in past literature. As a limitation of our proposed approach, it is important to emphasize that the neural style transfer algorithm relies on significant manual efforts for data annotation in the source and target domains and masking, while the CycleGAN can perform automated image-to-image translation in an unsupervised manner. This may form an interesting path for future research on the optimization of synthetic images generated by the CycleGAN.
8. Conclusion
The study shows that synthetic data augmentation through neural style transfer improves the generalization of AI models used for ice detection. We have compared our proposed approach to the state-of-the-art CycleGAN model as a baseline and outlined the superiority of the proposed method in facilitating improved predictions in new domains. This is evidenced by the substantial improvement in F1 score and accuracy metrics for icing prediction across different wind parks compared to standalone ice detection techniques. Even though our results are mainly significant for ice detection in one target wind park (wind park A) in North America, the method shows the ability to generalize to another wind park (wind park B) in Northern Europe as well. To the best of our knowledge, this is the first study to propose generalized detection of ice accretion on rotor blades and can be useful in making more effective icing predictions, for example, in new wind parks that the DL models have not been previously trained on. This can help prevent unexpected downtimes and failures in wind turbines due to icing, making wind energy a more promising source of renewable energy. Despite its promise, our study has the limitation of only being able to demonstrate high accuracy in generalized ice detection when the target dataset has high-quality images (in wind park A), while with lower-quality images the performance gain is marginal (in wind park B). Another limitation may be the hand-labeling of data in our study. Although two different humans annotated the datasets, models trained with such data may be affected by the inherent bias of the annotators. Future work aims to automatically create segmentation masks using U-Net, to feed them to paired image-to-image translation models like Pix2Pix toward improving the characteristics of synthetic images. In addition, future research may use regression models for quantifying the ice accumulation and the fusion of data from sensor-based methods (e.g., SCADA data) may further improve the performance of ice detection models.
Acknowledgments
We acknowledge the German Federal Ministry of Education & Research (BMBF) for supporting this research through the Green Talents Award and wpd windmanager GmbH Co. KG for the provision of data.
Author contribution
Conceptualization: J.C., M.T.A.N., H.G.; Data curation: M.T.A.N., H.G.; Formal analysis: J.C., M.T.A.N.; Investigation: J.C., M.T.A.N., H.G.; Methodology: J.C., M.T.A.N., H.G., N.D.; Resources: A.G.; Software: J.C., H.G.; Supervision: N.D., J.-H.O., K.-D.T.; Validation: J.C., H.G.; Writing—original draft: J.C., M.T.A.N.; Writing—review and editing: J.C., M.T.A.N., H.G., N.D., J.-H.O., A.G., K.-D.T.
Competing interest
A.G. is employed at wpd windmanager GmbH Co. KG. The other authors declare no competing interests exist.
Data availability statement
The code used for experiments in this study is available at: https://github.com/malvela/WindTurbine-IceDetection_GenerativeAI. The data used in this study are not publicly available due to confidentiality reasons.
Ethics statement
The research meets all ethical guidelines, including adherence to the legal requirements of the study country.
Funding statement
This work received no specific grant from any funding agency, commercial or not-for-profit sectors.
Provenance statement
A preliminary version of this paper (Chatterjee et al., Reference Chatterjee, Alvela Nieto, Gelbhardt, Dethlefs, Ohlendorf and Thoben2022) appeared in the NeurIPS Climate Change AI workshop.
Open access
