Impact Statement
AI and drone technologies have immense potential to enable data-driven decision-making in support of resilient housing and infrastructure initiatives across climate-vulnerable regions like the Caribbean. In this work, we have developed an end-to-end pipeline for automatically generating building footprints and roof classification maps using computer vision and drone images in support of government-led climate resilience programs. By leveraging innovative technologies and building local capacity, government agencies are better equipped to not only respond to disasters but also anticipate risks and mitigate impacts, thereby increasing climate resilience in the region.
1. Introduction
The Caribbean is among the world’s most climate-vulnerable regions due to the prevalence and intensity of extreme climatic hazards such as tropical cyclones, floods, and landslides. In recent years, Category 5 hurricanes like Dorian, Irma, and Maria have devastated many small island developing states (SIDSs), leaving widespread trails of loss and destruction across the region. SIDS often bear the brunt of the economic costs from climate-extreme events, with the highest degree of losses typically sustained in the housing sector (Government of the Commonwealth of Dominica, 2023). Hurricane Maria, for example, damaged over 28,000 homes—roughly 90% of Dominica’s housing stock—and accumulated costs over 200% of the nation’s GDP (Government of the Commonwealth of Dominica, 2023). Moreover, extensive damage to structural assets from natural hazards can bring about numerous challenges, such as precarious housing conditions, building collapse, and loss of life. Postdisaster housing recovery, which involves retrofitting damaged buildings, reconstructing public infrastructure, and relocating displaced populations, can result in severe budget cuts and increased debt, further straining already fragile economies (Government of the Commonwealth of Dominica, 2023). As global temperatures continue to rise, extreme weather events will only grow in severity, putting many more vulnerable populations at risk.
The loss and devastation brought about by super typhoons have spurred ambitious climate resilience programs by national government agencies aiming to reduce the adverse effects of extreme weather events in the housing sector (Government of the Commonwealth of Dominica, 2023; World Bank, 2022). For instance, the Disaster Vulnerability Reduction Project (DVRP) by the Government of Saint Lucia (GoSL) seeks to measure and reduce vulnerability to climate change impacts through institutional strengthening and retrofitting of structural assets (Government of Saint Lucia, 2023). Another example is the Resilient Housing Scheme by the Government of the Commonwealth of Dominica (GoCD), which strives to make 90% of Dominica’s housing stock resilient by 2030 through the construction of 5000 new resilient homes for vulnerable citizens (Government of the Commonwealth of Dominica, 2023). For these climate adaptation initiatives to be successful, local governments require accurate, complete, and up-to-date housing stock information to inform pre- and postdisaster retrofitting, reconstruction, and relocation strategies. However, the traditional house-to-house approach to identifying high-risk structures is often cost-prohibitive for many developing countries, prompting the need for more timely and cost-efficient alternatives.
In response to these challenges, researchers have turned to artificial intelligence (AI) and Earth observation (EO) to quickly and efficiently distil meaningful information from large unstructured geospatial data (World Bank, 2022; Triveno et al., Reference Triveno, Antos, Koers and Endo2019). Previous works demonstrate how deep learning (DL) techniques can be used to extract building footprints and their corresponding characteristics from high-resolution aerial images (Partovi et al., Reference Partovi, Fraundorfer, Azimi, Marmanis and Reinartz2017; Castagno & Atkins, Reference Castagno and Atkins2018; Solovyev, Reference Solovyev2020; Buyukdemircioglu et al., Reference Buyukdemircioglu, Can and Kocaman2021; Kim et al., Reference Kim, Bae, Kang and Lee2021; Huang et al., Reference Huang, Ren, Liu, Wang, Yu, Schmitt, Hänsch, Sun, Huang and Mayer2022; Tingzon et al., Reference Tingzon, Cowan and Chrzanowski2023). More recent years have seen a growing interest in applying DL to drone images and leveraging their very high spatial resolutions to generate more accurate and granular building stock information (Triveno et al., Reference Triveno, Antos, Koers and Endo2019; Akbari et al., Reference Akbari, Almaadeed, Al-Maadeed and Elharrouss2021; Calantropio et al., Reference Calantropio, Chiabrando, Codastefano and Bourke2021; Takhtkeshha et al., Reference Takhtkeshha, Mohammadzadeh and Salehi2022; Fujita & Hatayama, Reference Fujita and Hatayama2023). Yet despite the evident advantages of using AI and drone technologies to support climate resilience strategies, the widespread adoption of these solutions is hindered by gaps in the local capacity to develop and maintain systems for generating critical baseline exposure datasets.
This work aims to bridge these gaps by providing government agencies with an end-to-end workflow for rapidly generating critical baseline housing information using very high resolution (VHR) drone images. Specifically, we leverage computer vision (CV) models such as segment anything model (SAM) and CNNs for building footprint delineation and rooftop classification, respectively. We also evaluate the cross-country generalizability of roof classification models across SIDS to determine the extent to which models trained in one country can be adapted to another. Finally, we underscore the importance of strengthening local capacity through ongoing efforts to promote the adoption of AI and EO-based solutions among national and regional government agencies. This work is done under the Digital Earth Project for Resilient Housing and Infrastructure, a World Bank project funded by the Global Facility for Disaster Reduction and Recovery (GFDRR), in partnership with GoCD and GoSL, in support of government-led initiatives to enhance the climate resilience of the housing and infrastructure sector in Caribbean SIDS.
2. Data
In this section, we detail our process for generating our ground truth housing stock datasets using VHR aerial images in the form of aircraft- and drone-derived optical imagery and building footprints for Dominica and Saint Lucia, our primary regions of interest.
2.1. VHR aerial images
We acquired the following VHR aerial images from various data sources, including partner government agencies GoCD and GoSL, the World Bank Global Program for Resilient Housing (GPRH) (World Bank Global Program for Resilient Housing (GPRH), 2023), and the open data platform OpenAerialMap (OAM) (OpenAerialMap, 2023):
-
• Aircraft-derived nationwide orthophotos of Dominica (2018–2019) and Saint Lucia (2022)
-
• Pre- and postdisaster drone images of 10 villages in Dominica, namely Colihaut, Coulibistrie, Delices, Dublanc, Kalinago, Laplaine, Marigot, Pichelin, Roseau, and Salisbury (2017–2018)
-
• Drone images of 3 districts in Saint Lucia, namely Dennery, Castries, and Gros Islet (2019)
The spatial resolutions of aircraft-derived images range from 10 to 20 cm/px, whereas the resolutions of drone-derived images range from 2 to 5 cm/px. Table 1 further details the spatial resolution, coverage, year of acquisition, and number of annotated buildings per aerial image used in this study.
2.2. Building footprints data
For the aircraft-derived orthophotos, we obtained nationwide building footprints in Dominica and Saint Lucia in the form of vector polygons delineated from the corresponding aerial image, as provided by the World Bank. For the drone images, we initially looked to alternative data sources such as OpenStreetMap (OSM) (OpenStreetMap Buildings, 2023), Microsoft Building Footprints (Microsoft Building Footprints, 2023), and Google Open Buildings (Google Open Buildings, 2023); however, we observed significant misalignment between the publicly available building footprint polygons and the underlying drone images, as illustrated in Figure 1. In response to this issue, we turned to SAM to delineate building instances directly from drone images (Kirillov et al., Reference Kirillov, Mintun, Ravi, Mao, Rolland, Gustafson, Xiao, Whitehead, Berg and Lo2023). Additional information on the SAM configuration is detailed in Section 3.
Building footprint polygons from (a) Microsoft Building Footprints, (b) Google Open Buildings, (c) OpenStreetMap Buildings, and (d) SAM superimposed on an OAM drone image taken in Salisbury, Dominica (OpenAerialMap, 2023).
2.3. Rooftop image tiles
We generated our dataset for roof classification by selecting 80 randomly sampled 500 m x 500 m tiles across Dominica and Saint Lucia and removing tiles with no building footprints using the datasets described in the previous section. We further augmented these tiles with the villages, cities, and districts where drone images were available (Tingzon et al., Reference Tingzon, Cowan and Chrzanowski2023). For each building footprint within the selected tiles and regions, we extracted the minimal bounding rectangle of the building polygon from the corresponding aerial image, scaled by a factor of 1.5 and zero-padded to a square. We then proceeded to annotate 9150 buildings in Dominica and 5175 buildings in Saint Lucia via visual interpretation of the aerial images for a total of 14,325 labeled building footprints across the two countries. We classified the buildings based on two attributes: (1) roof type, which includes hipped, gabled, flat, and no roof, and (2) roof material, which includes healthy metal, irregular metal, blue tarpaulin, concrete or cement, and incomplete rooftops. The class distributions for each roof attribute are presented in Table 2. Examples of drone image tiles for each roof type and roof material category are illustrated in Figure 2.
Class distributions for roof type and roof material labels across Dominica and Saint Lucia
Examples of VHR drone-derived roof image tiles for each of the roof material categories (top row) and roof type categories (bottom row).
3. Methods
This section outlines our workflow for generating critical housing stock information in the Caribbean using drone images, as summarized in Figure 3.
Proposed workflow for the automatic generation of housing stock information from drone images using DL models.
3.1. Building footprint delineation
To extract building footprints from drone images, we leveraged the Segment-Geospatial Python package for segmenting raster images based on SAM (Wu & Osco, Reference Wu and Osco2023; Osco, Reference Osco2023; Osco, Reference Osco2023; Kirillov et al., Reference Kirillov, Mintun, Ravi, Mao, Rolland, Gustafson, Xiao, Whitehead, Berg and Lo2023). We also used Language SAM (LangSAM) (Medeiros, Reference Medeiros2023) Python package to combine instance segmentation with text prompts to generate masks of specific objects in the drone images. In our study, we set our text prompt to “house,” the box threshold, that is the threshold value used for object detection in the image, to 0.30, and the text threshold, that is the threshold value used to associate the detected objects with the provided text prompt, to 0.30. In practice, these parameters can be tweaked as necessary to achieve the best results. As a postprocessing step, we applied the Douglas–Peucker algorithm to simplify the generated building polygons, as implemented in GeoPandas (Douglas & Peucker, Reference Douglas and Peucker1973).
3.2. Roof classification
Given the VHR aerial images and corresponding building footprint polygons, we proceeded with developing our roof classification models. We began by fine-tuning CNN models pretrained on the ImageNet dataset (Deng et al., Reference Deng, Dong, Socher, Li, Li and Fei-Fei2009) with architectures ResNet50 (He et al., Reference He, Zhang, Ren and Sun2016), VGG16 (Simonyan & Zisserman, Reference Simonyan and Zisserman2015), Inceptionv3 (Szegedy et al., Reference Szegedy, Vanhoucke, Ioffe, Shlens and Wojna2016), and EfficientNet-B0 (Tan & Le, Reference Tan and Le2019) using cross-entropy loss for both roof type and roof material classification tasks. The input rooftop image tiles are zero-padded to a square and resized to 224 x 224 px for ResNet50, EfficientNet-B0, and VGG16 and 299 x 299 px for InceptionV3. We set the batch size to 32 and the maximum number of epochs to 60, and we use an Adam optimizer with an initial learning rate of
$ 1{e}^{-5} $
, which decays by a factor of 0.1 after every 7 epochs with no improvement. For data augmentation, we implemented horizontal and vertical image flips with a probability of 0.50 and random rotations ranging from
$ -90{}^{\circ} $
to
$ 90{}^{\circ} $
. Given that our data is imbalanced (see Table 2), we also implemented random oversampling for the minority classes. To prevent overconfident predictions, we applied label smoothing as a regularization technique, with smoothing set to 0.1 (Müller et al., Reference Müller, Kornblith and Hinton2019).
3.3. Model evaluation
We split each country-level dataset into training (80%) and test (20%) sets using stratified random sampling to preserve the percentage of samples per class, as shown in Table 2. Due to the high costs associated with acquiring nationwide aircraft-derived orthophotos, we recommend government partners leverage drone technologies to collect VHR aerial images; thus, for our experiments, the test sets for Dominica and Saint Lucia comprise entirely of drone images. Additionally, to test whether geographically diverse training data improves the prediction, we combine the training sets across Dominica and Saint Lucia (henceforth referred to as the “combined” dataset).
We report the F1 score, precision, recall, and accuracy using standard definitions as follows:
$$ \mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}} $$
$$ \mathrm{Recall}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}} $$
$$ \mathrm{F}1\;\mathrm{score}=\frac{2\cdot \mathrm{Precision}\cdot \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}} $$
$$ \mathrm{Accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{TP}+\mathrm{TN}+\mathrm{FP}+\mathrm{FN}} $$
where TP is the number of true positives, TN is the number of true negatives, FP is the number of false positives, and FN is the number of false negatives. The precision, recall, and F1 score metrics are computed as the unweighted mean of the metrics (that is macro-averaged) calculated per class.
4. Results and Discussion
For roof type classification, we achieve the highest F1-score of 87.1% for Dominica using an EfficientNet-B0 model and 89.5% for Saint Lucia using a ResNet50 model. Likewise, for roof material classification, we achieve our best score of 89.5% for Dominica using a ResNet50 model and 91.7% for Saint Lucia using an EfficientNet-B0 model. Among the models trained on the combined datasets from Dominica and Saint Lucia, our highest scores of 90.0% and 90.4% were achieved using an EfficientNet-B0 model for roof type classification and an Inceptionv3 model for roof material classification, respectively. Note that the latter models were evaluated using the combined test sets of Dominica and Saint Lucia. Table 3 details the complete F1-scores, precision, recall, and accuracy scores, and Figure 4 illustrates sample model outputs in the form of roof material classification maps using drone images taken before and after Hurricane Maria in Coulibistrie, Dominica.
Test set model performance scores (%) in terms of F1-score (F1), precision (P), recall (R), and accuracy (Acc) of CNN architectures for roof type and roof material classification trained using (a) only Dominica data, (b) only Saint Lucia data, and (c) using the combined datasets of Dominica and Saint Lucia
Drone images (left) and roof material classification maps (right) of Coulibistrie, Dominica taken before (top) and after (bottom) Hurricane Maria in 2017. Roof categories include healthy metal (green), irregular metal (red), concrete/cement (yellow), blue tarpaulin (blue), and incomplete (purple).
Lastly, we investigate the cross-country generalizability of the best models by evaluating their performance on the designated test sets of each country. As shown in Table 4, our results indicate that for roof type classification, the best model trained on the combined data from Dominica and Saint Lucia (that is the “combined model”) performs marginally better than models trained using only local data (that is data from the same country); however, for roof material classification, we find that locally trained models consistently outperform the combined model. These results also indicate that local models trained on the Dominica subset are more robust to geographical shifts than local models trained on the Saint Lucia subset, as local models for Saint Lucia appear to generalize poorly in the Dominica context. We posit that this is likely due to differences in class distributions between Dominica and Saint Lucia. Specifically, the training data for Dominica is derived primarily from postdisaster aerial images taken in the aftermath of Category 5 Hurricane Maria, meaning approximately 15% of the training data are comprised of blue tarpaulins typically used to cover severely damaged rooftops. Meanwhile, the aerial images for Saint Lucia were taken in nondisaster contexts, and thus, the Saint Lucia dataset is completely devoid of blue tarpaulins (see Table 2). More generally, the high levels of variability in local model performance for out-of-distribution countries indicate the importance of collecting highly localized and contextualized training data. Further research on domain adaptation is needed to reduce performance degradation in the face of geographic distribution shifts.
Cross-country generalizability in terms of F1-score (F1), precision (P), recall (R), and accuracy (Acc) of the best roof type and roof material classification models as shown in Table 3
4.1. Building local capacity in Caribbean SIDS
The overarching goal of the Digital Earth for a Resilient Caribbean Project is to strengthen local capacity in Caribbean SIDS to harness AI and EO technologies in support of resilient housing operations. The project has three main components:
-
1. Capacity building. This project supports the training and upskilling of government staff, academics, and other key stakeholders in the use of EO data, tools, and services.
-
2. Operational support. Our team aims to support the generation, integration, and use of critical baseline exposure datasets using AI and EO-based technologies.
-
3. Knowledge sharing. Through knowledge exchange and dissemination, we strive to promote our methods and tools to encourage wider adoption across the Caribbean region.
To this end, our team is assisting government agencies in establishing geographic information systems (GISs) units capable of generating, managing, and maintaining large-scale disaster risk datasets. In partnership with technical experts and local stakeholders, we are designing training programs on geospatial data analytics, community mapping, drone operations, pilot coordination, image processing, machine learning, and large-scale geospatial data management (Humanitarian OpenStreetMap Team (HOT), 2023). Additionally, we have developed educational resources to enable the rapid generation of baseline exposure datasets aimed at local government staff, community mappers, and disaster responders. This includes executable Google Colaboratory notebook tutorialsFootnote 1 , Footnote 2 demonstrating how to run the SAM and CNN models to quickly extract building footprints and create roof classification maps from locally collected drone images.
5. Conclusion
This work proposes an end-to-end workflow for filling critical baseline exposure data gaps using VHR aerial images and DL techniques in support of government-led climate resilience initiatives in the Caribbean. Specifically, we demonstrate how CV and drone images can be used to rapidly generate housing stock information, including building footprints and roof type and roof material classification maps, for disaster risk reduction and recovery. Based on our evaluation of the cross-country generalizability of DL models, we urge caution in applying locally trained models off the shelf to new geographic regions and emphasize the significance of collecting local, highly contextualized training data. We also highlight the importance of local capacity building, skills development, and cocreation of geospatial datasets in deploying sustainable AI-for-climate solutions, especially in Global South contexts. Through this work, we hope to empower government agencies to strengthen the local capabilities needed to sustainably generate AI and EO-derived housing stock data to better inform climate resilience programs in the Caribbean.
Open peer review
To view the open peer review materials for this article, please visit http://doi.org/10.1017/eds.2024.46.
Acknowledgments
We are grateful to the Government of Saint Lucia (GoSL) and the Department of Physical Development and Urban Renewal (DPDUR) for providing the EO datasets for Saint Lucia. We also appreciate the support from the Government of the Commonwealth of Dominica (GoCD) in providing the EO datasets for Dominica and implementing component 2 of the Disaster Vulnerability Reduction Project (DVRP), funded by the World Bank Group and Climate Investments Fund (CIF) under the Pilot Program for Climate Resilience (PPCR). Additionally, this endeavor would not have been possible without the initial work of the World Bank Global Program for Resilient Housing (GPRH), which laid the foundations for this study. We thank Sarah Antos, Michael Fedak, and Christopher Williams for their support in providing access to the datasets and for sharing their unique insights on the data landscape in the Caribbean.
Author contribution
Isabelle Tingzon: Methodology, Software, Data Curation, Writing—Original Draft; Nuala Margaret Cowan: Conceptualization, Data Curation, Resources, Writing—Review & Editing; Pierre Chrzanowski: Conceptualization, Project administration, Funding acquisition. All authors approved the final submitted draft.
Competing interest
None.
Data availability statement
The raw data that support the findings of this study are available from OAM, GPRH, GoCD, and GoSL. Restrictions apply to the availability of these data, which were used under license for this study. The open-source code can be found in the GFDRR Github repository: https://github.com/GFDRR/caribbean-rooftop-classification/.
Funding statement
This research was supported by the Digital Earth Project for Resilient Infrastructure and Housing in the Caribbean, a World Bank Group project funded by the Global Facility for Disaster Reduction and Recovery (GFDRR).
Ethical standard
The research meets all ethical guidelines, including adherence to the legal requirements of the study country.
Provenance
This article was accepted into the Climate Informatics 2024 (CI2024) Conference. It has been published in Environmental Data Science on the strength of the CI2024 review process.
Open access
Comments
Dear Editor,
We wish to submit an original research article entitled, Mapping Housing Stock Characteristics from Drone Images for Climate Resilience in the Caribbean, for consideration by the Environmental Data Science Journal.
In this paper, we propose an end-to-end workflow for filling critical baseline exposure data gaps using very high-resolution aerial images and computer vision techniques in support of government-led climate resilience initiatives in the Caribbean. Specifically, our work leverages the Segment Anything Model (SAM) and convolutional neural networks (CNN) to automate the generation of building footprints and roof classification maps. Furthermore, we highlight the importance of local capacity building, skills development, and co-creation of geospatial datasets in deploying sustainable AI-for-climate solutions, especially in Global South contexts. Through this work, we hope to empower government agencies to strengthen the local capabilities needed to sustainably generate AI and EO-derived housing stock data to better inform climate resilience programs in the Caribbean.
We have no conflicts of interest to disclose.
Please address all correspondence concerning this paper to me at itingzon@worldbank.org or issatingzon@gmail.com.
Thank you very much for your consideration of this manuscript.
Sincerely,
Isabelle Tingzon
Disaster and Climate Risk Data Fellow
The World Bank Group, GFDRR