2.1. From shallow to deep neural networks

Deep learning is the subset of AI that relies on computational models to make accurate data-driven decisions by representing and learning complex concepts modeled as hierarchical representations of simpler concepts (Goodfellow, Yoshua, & Aaron Reference Goodfellow, Yoshua and Aaron2016). Typically, these models are feedforward neural networks composed of a sequence of layers with differentiable mathematical parameters, operations, and functions. These layers act as a hierarchical representation for identifying and extracting patterns from large datasets that accurately map from sets of complex inputs to good decision outcomes (Kelleher Reference Kelleher2019).

Deep learning emerged after the second AI winter (1987–1993), the period when the previous dominant paradigm, expert systems, had reached its limits and research funding radically shrank. AI researchers re-invented the back-propagation algorithm in the 1980s, which revived the interest in the connectionist models of AI (Russell & Norvig Reference Russell and Norvig2021). Around the same time, there was an interest in establishing more scientific approaches to AI research. Many subfields, such as computer vision, robotics, speech recognition, multiagent systems, and natural language processing were gradually reunified with the “newfound appreciation for data, statistical model, optimization, and machine learning” (Russell & Norvig Reference Russell and Norvig2021).

In the 2000s, advances in computing power and the creation of the worldwide web supported the exponential growth and availability of training data and the shift toward ML. This has lightened the key burden of statistical estimation, leading to the dramatic progress of statistical generalization and applicability of deep learning (Goodfellow et al. Reference Goodfellow, Yoshua and Aaron2016).

Developments in computer infrastructure, such as larger memory capacity and denser computing units, enabled faster and heavier computation and have also contributed to deep learning. While neural networks have been developed since the 1940s, the recent spread of computing environments enabled stacking more layers in a neural network to perform ‘deep’ computations with high accuracy to process larger datasets (Goodfellow et al. Reference Goodfellow, Yoshua and Aaron2016). Furthermore, neural network models have not only increased in their number of processing units or layers but also in the configuration types used to address different forms of representations. From the general use neural networks, such as multilayer perceptron and artificial neural networks, to more specialized models, such as convolutional neural networks and graph convolutional networks. These terms are well known; their definitions can be found in Goodfellow et al. (Reference Goodfellow, Yoshua and Aaron2016).

Overall, in the last decade, a wave of deep learning has emerged with the successful development and application of neural network-based algorithms and models to problems from different domains. Two examples of such breakthroughs are as follows: AlexNet (Krizhevsky, Sutskever, & Hinton Reference Krizhevsky, Sutskever and Hinton2012), a learning system for image classification proposed by Geoffrey Hinton and his team; and AlphaGo (Russell & Norvig Reference Russell and Norvig2021; Silver et al. Reference Silver, Huang, Maddison, Guez, Sifre, van den Driessche, Schrittwieser, Antonoglou, Panneershelvam, Lanctot, Dieleman, Grewe, Nham, Kalchbrenner, Sutskever, Lillicrap, Leach, Kavukcuoglu, Graepel and Hassabis2016), a reinforcement learning model able to defeat human champions in the game of Go.

2.2. Mapping AI technologies in CAAD

The deep learning wave has radically impacted and changed many fields and industries, and CAAD is no exception. In the face of growing deep learning research in the field, it is important to both understand the trends and to map predecessors.

In this regard, Mateusz Zwierzycki, a Polish architect and researcher in digital design published a quantitative review of AI in CAAD. He considers that “AI research efforts kept intensifying, with a growth peak around 1995–2005” (Zwierzycki Reference Zwierzycki, Werner and Koering2020) and “in the next decade (2010–2020) there seems to have been another growth period in the field” (Zwierzycki Reference Zwierzycki, Werner and Koering2020). He suggests three factors to explain the growth of AI-related design research: the maturity of Internet applications and data-oriented services, the popularity of parametric modeling, and a general increase in architectural research. These factors offer an interesting hypothesis to explain the rapid growth of AI research in CAAD.

Zwierzycki’s analysis relies on large domains of AI and application, which limits the capacity to infer specific trends and details about machine learning. For example, his analysis of AI tools used in design research over time comprises a wide range of computational concepts, such as genetic algorithms and CNN.

Abraham Noah Wu and his colleagues reviewed articles that used a specific deep learning model, generative adversarial networks (GAN), for solving challenging tasks in the built environment (Wu, Stouffs, & Biljecki Reference Wu, Stouffs and Biljecki2022). They found that GAN is a cutting-edge technology with a wide range of applications, from improving performance in existing problems to opening new frontiers in previously overlooked areas. They also claimed that GAN can be applied at various scales in the built environment and is being used in several unique application domains including data augmentation, privacy protection, and building design generation. They also pointed out that a common challenge with GAN currently is the lack of high-quality datasets curated specifically for problems in the built environment.

The study conducted by Wu, Stouffs, and Biljecki diverged from the investigation carried out by Zwierzycki in its focus on the examination of the utilization and trends of a particular deep learning model. This research attempts to bridge the gap between these two different scales by exploring the general trend of deep learning in CAAD research. We narrow the scope of our review to machine learning with neural networks in CAAD research over the past 30 years. This refined scope enables a more accurate understanding and analysis of trends and precedents related to deep learning in CAAD.

2.3. Taxonomy and terms

To systematically collect and analyze CAAD research papers that employ machine learning with neural networks, we established a clear terminology related to deep learning based on Liu & Lang (Reference Liu and Lang2019) and Sarker (Reference Sarker2021). For this study, deep learning technology is classified into two categories based on the role of the model, namely, whether it is discriminative or generative. A discriminative model uses conditional probability to make predictions on unseen data and can be applied to classification or regression problems. In contrast, a generative model centers on the distribution of a dataset, returning a probability for a specific instance (Figure 1).

Figure 1. Taxonomy of deep learning models based on neural networks.

Table 1. Six main- and 15 sub-categories of research problem types

Various discriminative and generative models were considered, and the following were selected based on the presence of neural networks: multi-layer perceptron (MLP), simple neural network (NN), convolutional neural network (CNN), and recurrent neural network (RNN) for discriminative models; and GAN, autoencoder (AE), self-organizing map (SOM), restricted Boltzmann machine (RBM), and deep belief network (DBN) for generative models.

It is important to state that some of the categories are not completely exclusive. For example, generative models, such as GAN and AE, can incorporate convolutional layers or entire CNNs as part of their architecture. In this study, cases of non-generative use of CNNs are classified under the CNN category, while cases of generative use of CNNs are classified based on the type of model that consists of multiple CNN layers. Despite the widespread use of the terms and concepts used in the proposed taxonomy, it is important to acknowledge that slight variations in usage exist.

3.1. Scraping the literature

The first step in analyzing trends and characteristics of research in CAAD that employ machine learning with neural networks is collecting data on research publications. The source for our data is the Cumulative Index about publications in Computer Aided Architectural Design, CumInCAD (1998), the earliest open access database initiated by Martens and Turk in 1998 (Martens, B., & Turk, Z. Reference Martens and Turk2003). CumInCAD is the dedicated digital archiving platform supported by several conferences and journals in the CAAD research community. A site-specific crawler was developed to collect information on publications. The crawler and statistical data used in this research can be accessed through here.Footnote ¹

At this juncture, it is worth noting that there are limitations to our study which should be acknowledged. The data source, being limited to CumInCAD, likely excludes relevant research found in publications relating to CAAD in less open/public access repositories. Moreover, as the study aims to capture the domain of machine learning and deep learning, specific models from related fields such as deep reinforcement learning, and natural language processing may also be omitted.

Prior to crawling, a list of keywords belonging to the field of deep learning is set: [“deep learning”, “deep neural network”, “artificial neural network”, “neural network”, “multi-layer perceptron”, “convolutional neural network”, “recurrent neural network”, “generative adversarial network”, “autoencoder”, “self-organizing map”, “Boltzmann machine”]. The keywords are established by using a minimal matching algorithm where, for example, terms like “deep learning,” “deep-learning” and “deep-learning-based” would all be equivalently matched. Note that terms relating to either “deep learning” or to “neural network” are included in the list, since the term neural network also pertains to research specific to deep learning with deep neural networks. Data collection is restricted to publications between 1990 and 2021.

The crawler accessed all 16,182 publications in CumInCAD and extracted research information such as research id, title, year, authors, source, abstract, references, and so forth. It then checked whether the information has at least one of the keywords. If the publication information includes a keyword, the full-text pdf file, and its respective repository information as a csv file are saved. A total of 362 research papers and journal articles were collected (Figure 2).

Figure 2. Pipeline for establishing a neural network-related research database from CumInCAD.

3.2. Refinement: Filtering out papers that exclude learning

We manually filtered out papers that are not closely related to deep learning. For this, each of the collected research papers was reviewed and tagged for identifying their statistical features. The tags are as follows: validity, data, model, and content.

Validity is binary valued: 0, if the publication does not include any learning process or only includes keywords in its references; and 1, otherwise. By this refinement, 175 of the original research papers and journal articles were excluded, and the remaining 187 publications were set as the final pool for statistical analysis.

There are tags that indicate the main neural network model and data type identified in the research samples, which are further analyzed in the following sections. Besides, there are three distinct data tags and five distinct model tags related to reproducibility. The tags represent reader access author’s data and models: data description, data sample, entire data set, model description or structure, model parameters, model accuracy after training, training process, and model file. These tags are used to estimate the level of reproducibility of the papers in the next sections.

Tags also comprise two characteristics of the content of the research: the research problem and the research subject. There are 15 different types of research problems that are categorized into six larger clusters: analysis, generation, fabrication, theory, recognition, and evaluation. These are used in the evaluation of trends in the Analysis section.

We likewise tagged research subjects with as much detail as possible, referencing the keywords list in CumInCAD. The resulting number of subjects is 60 and is used in analysis and described in Section. Research subject.

In the sequel, the term “neural network-related research” is used to denote the selected publications in the final pool that either apply or review deep learning techniques in CAAD. This terminology is employed to clearly distinguish the nature of the studies included.

4.1. Trend and research problems

General trend analysis

Figure 3 has three axes: year, problem categories, and the number of publications. The axis of year starts in 1990 and ends in 2021. Six different types of problems constitute the axis of problem categories. The bars depicts the number of publications by problem type. The evaluation category encompasses issues related to assessing the efficacy and efficiency of models developed through simulations or computational calculations using neural networks. The recognition category pertains to the challenge of recognizing drawings or objects using neural networks. The theory category encompasses the development of design theories, education, and teaching methodologies utilizing neural networks. The fabrication category encompasses digital fabrication techniques such as robotics or three-dimensional (3D) printing, which are enabled by neural networks. The generation category encompasses the creation of design vocabularies, the development of design tools, and the synthesis of design outcomes through generative models enabled by neural networks. The analysis category encompasses the challenge of transforming elements associated not only with cities and environments but also with the design process, into data for examination using neural networks.

Figure 3. Number of neural network-related research in CAAD by time and research problem.

One notable inference that can be drawn from Figure 3 is that integrating neural networks into CAAD research is not an entirely new phenomenon. From the early 1990s to 2015, the number of papers on neural network-related research has been relatively steady with small fluctuations year by year. Because of the archiving limitations, there is no clear indication of machine learning in design prior to 1990 in CumInCAD. However, Figure 3 also suggests that efforts to employ neural networks in design research is an extension of earlier research initiatives within the CAAD realm. Specifically, as described in Section. Model trend, neural networks have been consistently used.

Another notable indication from Figure 3 is that there are two distinct periods when neural network-related research was active: 1995 to 2005, and 2015 to 2021. The first period follows the second AI winter. This period had a small number of, but continuing, attempts to implement neural networks in CAAD. However, by 2005, research using neural networks declined until 2015. During this period, shallow neural networks were still predominantly used.

Post-2015, the second period illustrates a steep rise in the number of papers on neural network-related research. Prior to 2016, the number of publications related to neural network-related research within the CAAD research community was fewer than 5 or more less per year. Afterward, it has been mostly exponentially increasing with 3 publications in 2016, 5 in 2017, 15 in 2018, 35 in 2019, 43 in 2020, and 29 in 2021.

4.2. Research subject

The multifaceted nature of CAAD research often requires a multidisciplinary approach that encompasses multiple themes. In order to effectively categorize and analyze these complex subject areas, we have chosen to focus on the specific domain to which the neural network is being applied. By doing so, we aim to provide a clearer examination of the various applications of neural networks in CAAD research.

Figure 4 presents through a pie chart the composition of the subjects in neural network-related research in CAAD. The area of the pies is proportional to the number of publications. The darker shades also indicate more publications. The chart shows that there are various subjects in neural network-related research and a distinctive trend of dominance in subjects.

Figure 4. The composition of the subjects of neural network-related research in CAAD.

Drawing and form are the two slightly dominant research subjects in the dataset—both at 11.76% of the total—followed by other subjects with considerably smaller slices. On the other extreme, there are 34 subjects with only one publication, which comprises approximately 18% of the total.

4.3. Model trend

Figure 5 shows trends in the usage of different machine learning models in CAAD by year. Eight publications were excluded from the model analysis as they did not involve the use of model. The rows contain models that were used in the publications and the columns represent years. There are three main findings.

• • NNs have been used over the whole period reviewed.

• • CNNs are emerging models in CAAD research.

• • A transition in the employment of models from MLP or SOM to more diversified neural network-based models since 2015.

Figure 5. Changes in different models of neural network-related research in CAAD.

The chart in Figure 5 shows that simple NNs, including ANN, are models that have been in use over the past 30 years in neural network-related research. The darker-colored cells in the row neural network in the period of 2016–2021 show that even the standard neural network models have been widely used in the framework of deep learning.

Another finding from Figure 5 is that CNN are the emerging models in neural network-related research. Considering that GAN models usually rely on CNN, the latter can be considered the most popular model over the past 5 years. We postulate that the dominance of CNN over the last 5 years is related to the strong visual culture in architecture, in comparison to other models such as MLP or SOM. In the period between 2010 and 2021 there were fewer occurrences of models other than CNN. Specifically, Figure 5 shows that the occurrence of MLP was interrupted in 2015. After 2015, SOM was sporadically adopted and newer neural network-based models emerged in research. This transition suggests that the introduction of CNN-based models has increased the possibilities of machine learning research in CAAD.

4.4. Data type

Figure 6 illustrates the occurrence of the different data types used in the publications, arranged by year, showing the dominance and trends in CAAD. Eight publications were excluded from the model analysis as they did not involve the use of data to be processed in such research subject of theory and pedagogy. We classify the data types into five categories: graph, pattern, matrix, point cloud, and voxel. A graph is an abstract structure that represents entities as vertices and their potential relations as connections. An image is a raster graphic that represents drawings and photographs as matrices of pixels. A pattern is a raster graphic that represents simplified visual entities as an arrangement of binary values. A matrix is an arrangement of numbers into rows and columns. A point cloud is a collection of points representing 3D forms. A voxel is a regular and volumetric representation of forms in a 3D grid based on discrete entities called voxels. Based on these categories, there are four main findings:

• • Matrices have been used steadily over the last 30 years; and its use recently is rapidly increasing.

• • Image data have been extensively used recently, which may be correlated to the large adoption of CNN indicated in Section Model trend.

• • Newer 3D data types, such as point clouds and voxels, have emerged.

• • The utilization of pattern data, which is a technique for representing shapes through occupancy in a 2D grid, has experienced a marked decrease.

Overall, image and matrix have replaced pattern data, and newer data types that directly represent 3D space and forms are starting to be adopted in machine learning for CAAD.

Figure 6. Changes in different data format of neural network-related CAAD research.

4.5. Reproducibility

An important criterion in evaluating neural network-related research is reproducibility, an important principle of the scientific method and a key aspect of computer science research. For the findings of any study to be reproducible it should contain sufficient information for an experienced practitioner to repeat the experiment with (nearly) identical results (Kajiya Reference Kajiya1993). In this respect, machine learning and deep learning conferences such as ICML (International Conference of Machine Learning) and neurIPS (Neural Information Processing Systems) now require submission of code, model, and data for the purpose of reproducibility (Pineau Reference Pineau2019; NeurIPS 2020). Although in CAAD publications neural network-related research may neither be fully technical or scientific, nor presented in a similar structure or format, deep learning implementations should be properly presented within a scientific frame. In our analysis, we did not take into consideration the availability of code repositories such as Github, which was released in 2008, and results from earlier publications would not have been so readily reproducible.

Nine publications were excluded from the reproducibility analysis as they did not involve the use of data and were focused on design theory and physical tool-making instead. Three data tags (red-schemed in the rows in Figure 7) and five model tags (blue-schemed in the rows in Figure 7) were used to investigate how neural network-related research publications address reproducibility. Figure 7 shows the frequency of combinations of the eight tags; the more on the left, the more frequently appeared.

Figure 7. Reproducibility of neural network-related research in CAAD.

Most publications include data description, data sample, and model structure, but do not include data files, learning process, and model (hyper)parameters. So, while these publications provide important information about the data, the information on the model is relatively insufficient. Considering both data and model information is critical for reproducing deep learning-implemented research, for most publications, the experiments were impossible to reproduce in terms of data and model.