Replacement of subjective expert coding with objective computer algorithms

Creativity could be modeled computationally (Sosa and van Dijck, Reference Sosa and van Dijck2021). The theoretical foundations of automated problem solving are based on automated reasoning systems based on heuristic search techniques such as general problem solver (Newell et al., Reference Newell, Shaw and Simon1959). Early theories of problem-solving use computer simulations to predict human performance, explain the underlying processes and mechanisms, account for incidental phenomena, show how performance changes under different conditions, and explain how problem-solving skills are learned (Simon and Newell, Reference Simon and Newell1971). The challenges of computational problem-solving in design revolve around encoding, representation, constraint analysis, and reduction of the effective solution space (Perkowski, Reference Perkowski2022).

Protocol analysis of concurrent verbalization from professional design teams allows for the identification of reasoning patterns in idea generation (Cramer-Petersen et al., Reference Cramer-Petersen, Christensen and Ahmed-Kristensen2019), dynamic process patterns in design communication (Cash et al., Reference Cash, Dekoninck and Ahmed-Kristensen2020), success of design ideation (Maccioni and Borgianni, Reference Maccioni, Borgianni, Boujut, Cascini, Ahmed-Kristensen, Georgiev and Iivari2020) or evaluation of external sources of inspiration (Borgianni et al., Reference Borgianni, Maccioni, Fiorineschi and Rotini2020). Typically, the protocol analysis requires protocol coding performed by an expert, which invariably introduces a level of subjectivity that limits reproducibility by independent research teams. Furthermore, significant attention has been focused on the subjective nature of evaluating creativity of design ideas using metrics (Fiorineschi and Rotini, Reference Fiorineschi and Rotini2023). The subjective judgments required in metrics can significantly impact the evaluation of design creativity, owing to differences in how evaluators perceive and prioritize different aspects of the design ideas (Fiorineschi et al., Reference Fiorineschi, Frillici and Rotini2022). Hence, further research is needed to address these challenges and develop more robust methods for evaluating design ideas (Borgianni et al., Reference Borgianni, Maccioni, Fiorineschi and Rotini2020). To dispense with the reliance on experts and ensure maximal reproducibility, here we describe an objective method for textual processing and construction of semantic networks that is fully automated by applied computer algorithms.

Semantic networks as graphs

Semantic networks represent knowledge in the form of graphs that consist of vertices and edges (Diestel, Reference Diestel2017). Vertices indicate individual concepts, whereas edges between pairs of vertices indicate specific semantic connections (Boden, Reference Boden2004). Graphs could be divided into directed or undirected depending on the type of edges contained in the graph. The lack of loops, which are edges that connect vertices to themselves, and multiple edges with the same source and target vertices generates a simple directed graph. The lack of directed cycles generates a directed acyclic graph (DAG). The underlying undirected graph of a DAG, however, could contain cycles (Figure 2). Further introduction of a root vertex such that all edges of the directed graph are directed either away from or towards the root generates a rooted directed acyclic graph. Assigning different weights to the edges of directed or undirected graphs generates weighted graphs.

Figure 2.

The subgraph of meanings M in WordNet3.1 does not have directed cycles when the edges remain directed, however, when all edges are converted into undirected edges using the graph operator U, the graph U (M) becomes cyclic. One consequence of the structure of M is that even for two monosemous words such as “workspace” and “yellow” the shortest path in the undirected graph U(M) may not pass through the lowest common subsumer, which in this case is the root meaning vertex M00001740. In this example, the length of the shortest path between “workspace” and “yellow” is 12, whereas the distance between “workspace” and “yellow” through their lowest common subsumer M00001740 is 14. To avoid clutter in the image, we have added only a single word for each meaning vertex, however, many of the meaning vertices are subsumed by synsets of words. For example, both words “yellow” and “yellowness” subsume the meaning vertex M04972838.

WordNet is a lexical database, which represents human knowledge in a graph form (Miller et al., Reference Miller, Beckwith, Fellbaum, Gross and Miller1990; Miller, Reference Miller1995, Reference Miller and Fellbaum1998; Fellbaum, Reference Fellbaum1998a, Reference Fellbaum1998b). This is particularly suitable for imposing a distance function onto constructed semantic networks, which in turn enables quantitative exploration of dynamic cognitive processes such as creative cognition. Throughout this work, we employ WordNet 3.1, which is represented by a rooted directed acyclic graph of meanings (Figure 2). The constructed semantic networks are represented by undirected cyclic weighted graphs (Figure 3).

Figure 3.

Modeling the creative design process of an “Electric car” with a dynamic semantic network. The dynamics of the semantic network from the stage of idea generation to the stage of fully developed solution provides a useful, reproducible, and fast computational tool for exploration of creative thinking.

Semantic networks as models of the creative mind

Semantic networks constructed from conversation transcripts represent computational models of conceptual associations and structures (Georgiev et al., Reference Georgiev, Nagai and Taura2008, Reference Georgiev, Nagai and Taura2010; Han et al., Reference Han, Hua, Park, Wang and Childs2020, Reference Han, Sarica, Shi and Luo2021, Reference Han, Sarica, Shi and Luo2022). Individual concepts in the semantic network need not be only single words but could also be phrases combining different parts of speech (Esparza et al., Reference Esparza, Sosa and Connor2019), and can employ specific technical concepts (Sarica et al., Reference Sarica, Song, Luo and Wood2021). However, for ensuring reproducibility and objectiveness, the extraction of concepts for the semantic networks should be based on a set of rules that can be automated by a computer program without the need of human intervention. This problem is addressed satisfactorily by identifying individual concepts with single words that could be further narrowed down to a single lexical category (nouns) with the use of part-of-speech tagging performed by natural language processing software (Georgiev and Georgiev, Reference Georgiev and Georgiev2018).

Polysemy necessitates simultaneous use of words and meanings

Polysemy is essential for creative conceptual blending and associative thinking (Ravin and Leacock, Reference Ravin and Leacock2000; Fauconnier and Turner, Reference Fauconnier, Turner, Nerlich, Zazie, Vimala and Clarke2003; Nerlich et al., Reference Nerlich, Zazie, Vimala and Clarke2003; Georgiev and Taura, Reference Georgiev and Taura2014). In particular, designer’s mind could work simultaneously with several meanings of a word, similarly to how writers employ polysemy in humorous works (Boxman-Shabtai and Shifman, Reference Boxman-Shabtai and Shifman2014). Meaning is deemed an essential component of creativity (Sääksjärvi and Gonçalves, Reference Sääksjärvi and Gonçalves2018). To capture the possible impact of polysemous words in design thinking, we employ a computational method that does not compromise, reduce, or disambiguate between the multiple senses. The linguistic distinction between words and meanings could be approached in two different ways. Inclusion of a pre-processing step called disambiguation of senses could convert all words into senses. This would modify the verbal data through injection of interpretation even before the data analysis is started and would delete potentially useful information about underlying difficult-to-observe cognitive processes. For example, the nouns entering into the description of creative ideas may acquire different senses that disagree with those listed in a dictionary (Georgiev and Taura, Reference Georgiev and Taura2014;Taura and Nagai, Reference Taura and Nagai2013). Furthermore, the polysemy of nouns was found to be instrumental in the association of ideas, which were previously not considered to be related by the creative problem solver (Georgiev and Taura, Reference Georgiev and Taura2014). To avoid the latter shortcomings, we construct semantic networks from verbal data without any disambiguation of senses of extracted words (nouns) (Georgiev and Georgiev, Reference Georgiev and Georgiev2018), but preserving two types of vertices in the semantic network, word vertices, and meaning vertices. The benefit is that discovered functional relationships in the semantic network could be correlated to the neural activity in specialized brain cortical areas such as Broca’s area, which translates meanings into words, and Wernicke’s area, which translates words into meanings (Georgiev et al., Reference Georgiev, Georgieva, Gong, Nanjappan and Georgiev2021). The utility of semantic networks of nouns for studying creativity and reconstruction of difficult-to-observe cognitive processes in conceptual design was demonstrated previously by different research teams with the use of several experimental datasets (Georgiev et al., Reference Georgiev, Nagai and Taura2008, Reference Georgiev, Nagai and Taura2010; Yamamoto et al., Reference Yamamoto, Goka, Yusof, Taura, Nagai, Bergendahl, Grimheden, Leifer, Skogstad and Lindemann2009; Taura et al., Reference Taura, Yamamoto, Fasiha, Goka, Mukai, Nagai and Nakashima2012).

Creative cognition modeled as a dynamic semantic network

To assess the dynamic aspects of cognitive processes, the constructed semantic network should be allowed to evolve in time (Figure 3). One inefficient approach is to consider the cumulative growth of the semantic network as new concepts appearing in the transcribed conversations are added to those concepts that have already appeared. This cumulative approach creates large networks fast even for relatively short conversations, which produces a large sample size for statistical analyses. The disadvantage is that it retains in the semantic network concepts that may have been briefly considered but then discarded by the cognitive processes underlying the given task. An alternative, much more efficient approach is to coarse–grain the time into time intervals, each of which will encompass a corresponding part of the conversation (Georgiev and Georgiev, Reference Georgiev and Georgiev2018). The advantage of the noncumulative approach is that concepts are retained in the semantic network only if they repeatedly appear in the course of the problem solving conversation. The minimal time interval should be coarse–grained to contain a sufficient number of concepts to form a meaningful network. For real-time monitoring of verbal output, the semantic network dynamics could be tracked with a moving time window allowing for dynamic update of the semantic measures with each new word added in the conversation.

Lexical categories in WordNet

WordNet 3.1 is publicly available (http://wordnet.princeton.edu), lexical database for English created under the direction of G. A. Miller and hosted at Princeton University (Miller et al., Reference Miller, Beckwith, Fellbaum, Gross and Miller1990; Miller, Reference Miller1995; Fellbaum, Reference Fellbaum1998a, Reference Fellbaum1998b). WordNet contains four subnets that correspond to four basic lexical categories: nouns, verbs, adjectives, and adverbs (Miller, Reference Miller and Fellbaum1998; Fellbaum, Reference Fellbaum1998b). Because there are only a few cross-subnet pointers (Fellbaum, Reference Fellbaum1998b), calculation of graph-theoretic distances between words could be achieved by confining the constructed semantic networks to a single subnet. The subnet of nouns provides the largest and deepest hierarchical taxonomy in WordNet, which can be efficiently used for the construction of semantic networks of nouns. In an experimental study using design review conversations, it was found that over 99.8% of all nouns used in the conversations are also listed in WordNet 3.1 (Georgiev and Georgiev, Reference Georgiev and Georgiev2018). Further motivation for working with nouns in research of creative problem solving is provided by developmental linguistics findings that the category corresponding to nouns is, at its core, conceptually simpler or more basic than those corresponding to verbs and other parts of speech, which is exemplified by infants early advantage for learning nouns over verbs (Gentner, Reference Gentner and Kuczaj1982; Waxman et al., Reference Waxman, Fu, Arunachalam, Leddon, Geraghty and Song2013). In addition, experimental creativity research has shown that networks of nouns stimulate the generation of creative ideas (Georgiev and Georgiev, Reference Georgiev and Georgiev2019), different combinations of nouns and relations between nouns are associated with a display of creative thought (Dong, Reference Dong2009), and dissimilarity of noun pairs produces a higher number of emergent features of creative ideas (Wilkenfeld and Ward, Reference Wilkenfeld and Ward2001).

Hypernym-hyponym hierarchy of nouns

The two basic semantic relations in WordNet are synonymy, where sets of word synonyms (synsets) form the basic building blocks of the lexical hierarchy and hyponymy (subordination of synsets) where if a hyponym (subordinate) X is subsumed by a hypernym (superordinate) Y then it follows that “An X is a kind of Y” (Miller et al., Reference Miller, Beckwith, Fellbaum, Gross and Miller1990; Miller, Reference Miller and Fellbaum1998). The hypernym-hyponym (is-a) relationship provides a taxonomy of nouns in WordNet as follows: the root synset {entity} subsumes directly three different synsets, which can be viewed as classifying entities into {abstract_entity, abstraction}, {thing} or {physical_entity}. Each of these synsets then subsumes directly other synsets, further classifying classes of entities into subclasses, and so on. The hypernym-hyponym (is-a) hierarchy of nouns in WordNet is conceptually clear if it is represented in the form of a graph with two types of vertices: meaning vertices and word vertices.

The distinction between words and meanings would not have been required if there were one-to-one relationship between words and meanings. In natural language, however, one word can have several meanings (polysemy) and one meaning can be expressed by several words. For example, the word “horse” has five meanings thereby entering into five synsets as follows: M02377103 with synset {Equus_caballus, horse}, M03543217 with synset {gymnastic_horse, horse}, M03629976 with synset {horse, knight}, M04147696 with synset {buck, horse, sawbuck, sawhorse}, or M08414813 with synset {cavalry, horse, horse_cavalry}. Here, ‘M’ stands for meaning, the subsequent digits indicate the number by which the particular synset is referred to in WordNet. Explicitly labeling the meaning vertices with a numeric code further clarifies the significance of the synset and avoids possible misunderstanding of the synset as a list of words—in fact, the synset stands only for the meaning that is in common for all words in the list. For example, the meaning of M02377103 with synset {Equus_caballus, horse} is “the particular animal species in the genus Equus,” whereas the meaning of M03543217 with synset {gymnastic_horse, horse} is “an artistic gymnastics apparatus.” The fact that the meaning is the essential ingredient of a synset can be illustrated with a somewhat rare example of a meaning, which is difficult to guess from the words in the synset alone: the meaning of M03629976 with synset {horse, knight} is actually “a chess piece shaped to resemble the head of a horse” – this has to be understood from a sample sentence used in WordNet to clarify the meaning.

Composition of word and meaning subgraphs

The lexical hierarchy of nouns in WordNet 3.1 is comprised of 158,441 word vertices and 82,192 meaning vertices. Word vertices and meaning vertices are organized in two subgraphs, subgraph M, composed of 84,505 meaning → meaning edges between hypernyms and hyponyms, and subgraph W, composed of 189,555 word → meaning edges. The subgraph M is a rooted directed acyclic graph, which has as a root the meaning vertex M00001740 corresponding to the single-word synset {entity}. To compute graph theoretic measures for words, however, the subgraph M should be expanded with edges extracted from the subgraph W. For example, if we are interested in any semantic measure characterizing the word “horse,” we will need to extract the five edges from W that connect “horse” to each of its five meaning vertices. Only in the composite graph containing both meanings and words, we are able to comprehend the content of transcribed conversation.

The reason for appending extracted word edges to the graph M, instead of working in the full graph $ M\cup W $ , is to achieve computational efficiency, namely, the subgraph W, which is effectively discarded, has twice as many edges as the subgraph M and finding shortest paths is much faster in smaller graphs. Additional flexibility is achieved by the possible application of directed graph operators such as R(G), whose action on the graph G is to reverse the direction of all edges, or U(G), whose action on the graph G is to remove the directionality of all edges (Georgiev and Georgiev, Reference Georgiev and Georgiev2018). As an example, W contains word → meaning edges, R(W) contains meaning → word edges, and U(W) contains meaning – word edges. In this way, it is possible to add words as subordinate vertices (subsumed by meanings) or as superordinate vertices (subsumers of meanings) depending on the semantic measures that need to be computed (Figure 4).

Figure 4.

Graph composition of meaning vertices and word vertices for different WordNet 3.1 searches. (A) Adding words as subordinate vertices subsumed by meanings allows for computing the depth in the taxonomy and listing the meaning subsumers of words. (B) Adding words as subsumers of meanings allows for listing the meaning subvertices and leaves. (C) Adding two distinct words {w _1; w ₂} as subordinate vertices subsumed by meanings allows for computing their lowest common subsumer $ \mathcal{K}\left({w}_1,{w}_2\right) $ . (D) Adding two distinct words {w _1; w ₂} into an undirected graph allows for computing the shortest path distance between the two words, which may not pass through their lowest common subsumer. Different graph compositions are needed for path-based or information content-based quantification of word similarity.

Graph-theoretic functions

Defining semantic measures for words from the graph structure of WordNet requires the introduction of several basic functions that take as arguments a graph, denoted with capital letter, and one or more vertices, denoted with small letters (Georgiev and Georgiev, Reference Georgiev and Georgiev2018, Reference Georgiev and Georgiev2023). Hereafter, general type of vertices, either meanings or words, will be denoted as v ₁, v ₂,…, v_n, whereas word vertices will be specified as w ₁, w ₂,…, w_n.

Semantic measures for single words

The level of abstraction, polysemy and information content are semantic measures applicable to single words. For a semantic network composed of n word vertices, the average for each of these semantic measures could be determined with n searches in WordNet.

Level of abstraction

The level of abstraction of a word is the complement to unity of the word concreteness. For nouns, both measures are related to the noun depth in the WordNet taxonomy such that nouns located higher in the hierarchy are more abstract and less concrete, whereas nouns located lower in the hierarchy are less abstract and more concrete (Meng et al., Reference Meng, Huang and Gu2013; Georgiev and Georgiev, Reference Georgiev and Georgiev2018). In graph theoretic notation, the level of abstraction of the word w is

(1) $$ \mathrm{Abstraction}(w)=1-\frac{\mathcal{T}(w)-1}{{\mathcal{T}}_{\mathrm{max}}-1} $$

where $ {\mathcal{T}}_{\mathrm{max}}=19 $ is the maximal depth of WordNet 3.1 taxonomy and $ \mathcal{T}(w) $ is the depth of the word w computed as the shortest path distance between the root meaning vertex M00001740 with synset {entity} and the word vertex w in the graph $ M\cup \mathcal{J}\left[R(W),w\right] $ .

Polysemy

The polysemy measures the number of different meanings possessed by a given word. About 44% of English words are polysemous, which means that they have more than one meaning (Britton, Reference Britton1978). The log transformed value of polysemy quantifies the missing bits of information required for correct understanding of the intended meaning of a given word. That missing information is usually extracted from the context of the conversation. For monosemous words, which have only one meaning, there is no ambiguity and no information is missing. In graph theoretic notation, the polysemy is

(2) $$ \mathrm{Polysemy}(w)=\mid \mathcal{A}\left(W,w\right)\mid $$

which counts the number of all meaning vertices that are adjacent to the word vertex w.

Information content

The intrinsic information content of words or meanings is measured in bits solely from the graph-theoretic structure of WordNet 3.1. For comparison of different formulas that measure the information content of words in WordNet 3.1, however, it is helpful to work with normalized values in the unit interval [0,1]. To write compactly the formulas for information content, we will need the following word functions.

For a given word w: $ \mathcal{S}(w) $ lists the meaning subsumers in the graph $ M\cup \mathcal{J}\left[R(W),w\right] $ , $ \mathcal{V}(w) $ lists the meaning subvertices in the graph $ M\cup \mathcal{J}\left(W,w\right) $ , $ \mathcal{H}(w) $ lists the meaning hyponyms in the set difference $ \mathcal{V}(w)\backslash \left[\mathcal{A}\left(W,w\right)\cup w\right],\mathcal{L}(w) $ lists the leaves in the graph $ M\cup \mathcal{J}\left(W,w\right) $ , and $ \mathcal{C}(w) $ computes the commonness $ {\sum}_{i\in \mathcal{L}\left(G,w\right)}\frac{1}{S\left(M,i\right)} $ in the graph $ M\cup \mathcal{J}\left(W,w\right) $ .

Several constants specific for WordNet 3.1 are also useful: $ {\mathcal{V}}_{\mathrm{max}}=82192 $ is the maximal number of meaning subvertices, $ {\mathcal{L}}_{\mathrm{max}}=65031 $ is the maximal number of leaves, $ {\mathcal{C}}_{\mathrm{min}}=1/35 $ is the minimal commonness, and $ {\mathcal{C}}_{\mathrm{max}}=6863.6 $ is the maximal commonness.

Below, we summarize seven information content formulas whose performance for the analysis of creativity in design review conversations has been tested previously (Georgiev and Georgiev, Reference Georgiev and Georgiev2018).

Information content by Blanchard et al. (Reference Blanchard, Harzallah, Kuntz, Ghallab, Spyropoulos, Fakotakis and Avouris2008):

(3) $$ IC(w)=1-\frac{\log \mid \mathcal{L}(w)\mid }{\log \left({\mathcal{L}}_{\mathrm{max}}\right)} $$

Information content by Meng et al. (Reference Meng, Gu and Zhou2012):

(4) $$ IC(w)=\frac{\log \left[\mathcal{T}(w)\right]}{\log \left({\mathcal{T}}_{\mathrm{max}}\right)}\left[1-\frac{\log \left[1-{\sum}_{i\in \mathcal{H}(w)}\frac{1}{\mathcal{T}(i)}\right]}{\log \left({\mathcal{V}}_{\mathrm{max}}\right)}\right] $$

Information content by Sánchez et al. (Reference Sánchez, Batet and Isern2011):

(5) $$ IC(w)=\frac{\log \left({\mathcal{L}}_{\mathrm{max}}\right)+\log \mid \mathcal{S}(w)\mid -\log \mid \mathcal{L}(w)\mid }{\log \left({\mathcal{L}}_{\mathrm{max}}\right)-\log \left({\mathcal{C}}_{\mathrm{min}}\right)} $$

Information content by Sánchez and Batet (Reference Sánchez and Batet2012):

(6) $$ IC(w)=\frac{\log \left({\mathcal{C}}_{\mathrm{max}}\right)-\log \left[\mathcal{C}(w)\right]}{\log \left({\mathcal{C}}_{\mathrm{max}}\right)-\log \left({\mathcal{C}}_{\mathrm{min}}\right)} $$

Information content by Seco et al. (Reference Seco, Veale and Hayes2004):

(7) $$ IC(w)=1-\frac{\log \mid \mathcal{V}(w)\mid }{\log \mid {\mathcal{V}}_{\mathrm{max}}\mid } $$

Information content by Yuan et al. (Reference Yuan, Yu and Wang2013):

$$ IC(w)=\frac{\log \left[\mathcal{T}(w)\right]}{\log \left({\mathcal{T}}_{\mathrm{max}}\right)}\left(1-\frac{\log \mid \mathcal{L}(w)\mid }{\log \left({\mathcal{L}}_{\mathrm{max}}\right)}\right)+\frac{\log \mid \mathcal{S}(w)\mid }{\log \left({\mathcal{V}}_{\mathrm{max}}\right)} $$

Information content by Zhou et al. (Reference Zhou, Wang and Gu2008a):

(9) $$ IC(w)=\frac{1}{2}\left[1-\frac{\log \mid \mathcal{V}(w)\mid }{\log \left({\mathcal{V}}_{\mathrm{max}}\right)}+\frac{\log \left[\mathcal{T}(w)\right]}{\log \left({T}_{\mathrm{max}}\right)}\right] $$

Semantic similarity for word pairs

For a semantic network composed of n word vertices, the average semantic similarity for all pairs of vertices could be determined with (n ² − n)/2 searches in WordNet. Different definitions of semantic similarity between a pair of distinct words w ₁ and w ₂ rely on the lowest common subsumer of w ₁ and w ₂, the fraction of common subsumers, or the shortest path distance between w ₁ and w ₂ (Georgiev and Georgiev, Reference Georgiev and Georgiev2018).

The lowest common subsumer $ \mathcal{K}\left({w}_1,{w}_2\right) $ of a pair of distinct words $ {w}_1 $ and $ {w}_2 $ in the graph $ G=M\cup \mathcal{J}\left[R(W),\left\{{w}_1,{w}_2\right\}\right] $ is the deepest meaning vertex in the taxonomy among all vertices i whose sum $ \mathcal{D}\left[G,i,{w}_1\right]+\mathcal{D}\left[G,i,{w}_2\right] $ is minimal. If several common subsumers of w ₁ and w ₂ are at the same depth in the WordNet 3.1 taxonomy, the meaning vertex with lowest entry number is considered to be the unique $ \mathcal{K}\left({w}_1,{w}_2\right) $ . The depth $ \mathcal{T}\left[\mathcal{K}\left({w}_1,{w}_2\right)\right] $ of the lowest common subsumer $ \mathcal{K}\left({w}_1,{w}_2\right) $ is determined solely within the subgraph M.

The shortest path distance $ \mathcal{D}\left({w}_1,{w}_2\right) $ between a pair of distinct words w ₁ and w ₂ in the graph $ U(M)\cup \mathcal{J}\left[U(W),\left\{{w}_1,{w}_2\right\}\right] $ is the number of edges on the shortest path with subtracted two-edge contribution outside the subgraph U(M).

Path-based similarity measures

Semantic similarity by Al-Mubaid and Nguyen (Reference Al-Mubaid and Nguyen2006):

(10) $$ \mathrm{sim}\left({w}_1,{w}_2\right)=1-\frac{\log \left[1+\mathcal{D}\left({w}_1,{w}_2\right)\left\{{\mathcal{T}}_{\mathrm{max}}-\mathcal{T}\left[\mathcal{K}\left({w}_1,{w}_2\right)\right]\right\}\right]}{\log \left[1+2{\left({\mathcal{T}}_{\mathrm{max}}-1\right)}^2\right]} $$

Semantic similarity by Leacock and Chodorow (Reference Leacock, Chodorow and Fellbaum1998):

(11) $$ \mathrm{sim}\left({w}_1,{w}_2\right)=1-\frac{\log \left[\mathcal{D}\left({w}_1,{w}_2\right)+1\right]}{\log \left[2{\mathcal{T}}_{\mathrm{max}}-1\right]} $$

Semantic similarity by Li et al. (Reference Li, Bandar and McLean2003):

(12) $$ \mathrm{sim}\;\left({w}_1,{w}_2\right)={e}^{-0.2D\left({w}_1,{w}_2\right)}\frac{e^{1.2T\left[K\left({w}_1,{w}_2\right)\right]}-1}{e^{1.2T\left[K\left({w}_1,{w}_2\right)\right]}+1} $$

Semantic similarity by Rada et al. (Reference Rada, Mili, Bicknell and Blettner1989):

(13) $$ \mathrm{sim}\left({w}_1,{w}_2\right)=1-\frac{\mathcal{D}\left({w}_1,{w}_2\right)}{2\left({\mathcal{T}}_{\mathrm{max}}-1\right)} $$

Semantic similarity by Wu and Palmer (Reference Wu and Palmer1994):

(14) $$ \mathrm{sim}\left({w}_1,{w}_2\right)=\frac{2\left\{\mathcal{T}\left[\mathcal{K}\left({w}_1,{w}_2\right)\right]-1\right\}}{2\left\{\mathcal{T}\left[\mathcal{K}\left({w}_1,{w}_2\right)\right]-1\right\}+\mathcal{D}\left({w}_1,{w}_2\right)} $$

Subsumer-based similarity measures

Subsumer-based similarity measures reduce to path-based ones only for monosemous words, whereas for polysemous words they produce distinct results.

Semantic similarity by Jaccard (Reference Jaccard1912):

(15) $$ \mathrm{sim}\left({w}_1,{w}_2\right)=\frac{\mid \mathcal{S}\left({w}_1\right)\cap \mathcal{S}\left({w}_2\right)\mid }{\mid \mathcal{S}\left({w}_1\right)\cup \mathcal{S}\left({w}_2\right)\mid } $$

Semantic similarity by Braun-Blanquet (Reference Braun-Blanquet1932):

(16) $$ \mathrm{sim}\left({w}_1,{w}_2\right)=\frac{\mid \mathcal{S}\left({w}_1\right)\cap \mathcal{S}\left({w}_2\right)\mid }{\max \left[|\mathcal{S}\left({w}_1\right)|,|\mathcal{S}\left({w}_2\right)|\right]} $$

Semantic similarity by Dice (Reference Dice1945):

(17) $$ \mathrm{sim}\left({w}_1,{w}_2\right)=\frac{2\mid \mathcal{S}\left({w}_1\right)\cap \mathcal{S}\left({w}_2\right)\mid }{\mid \mathcal{S}\left({w}_1\right)\mid +\mid \mathcal{S}\left({w}_2\right)\mid } $$

Semantic similarity by Otsuka (Reference Otsuka1936) and Ochiai (Reference Ochiai1957):

(18) $$ \mathrm{sim}\left({w}_1,{w}_2\right)=\frac{\mid \mathcal{S}\left({w}_1\right)\cap \mathcal{S}\left({w}_2\right)\mid }{\sqrt{\mid \mathcal{S}\left({w}_1\right)\Big\Vert \mathcal{S}\left({w}_2\right)\mid }} $$

Semantic similarity by Kulczyński (Reference Kulczyński1927):

(19) $$ \mathrm{sim}\left({w}_1,{w}_2\right)=\frac{\mid \mathcal{S}\left({w}_1\right)\cap \mathcal{S}\left({w}_2\right)\mid }{2}\left(\frac{1}{\mid \mathcal{S}\left({w}_1\right)\mid }+\frac{1}{\mid \mathcal{S}\left({w}_2\right)\mid}\right) $$

Semantic similarity by Simpson (Reference Simpson1960):

(20) $$ \mathrm{sim}\left({w}_1,{w}_2\right)=\frac{\mid \mathcal{S}\left({w}_1\right)\cap \mathcal{S}\left({w}_2\right)\mid }{\min \left[|\mathcal{S}\left({w}_1\right)|,|\mathcal{S}\left({w}_2\right)|\right]} $$

Information content-based similarity measures

The following five information content-based similarity formulas could take as an input any of the seven information content formulas to give a total of 35 different information content-based similarity measures whose performance for the analysis of creativity in design review conversations has been tested previously (Georgiev and Georgiev, Reference Georgiev and Georgiev2018).

Semantic similarity by Jiang and Conrath (Reference Jiang and Conrath1997):

(21) $$ \mathrm{sim}\left({w}_1,{w}_2\right)=1-\frac{1}{2}\left\{ IC\left({w}_1\right)+ IC\left({w}_2\right)-2 IC\left[\mathcal{K}\left({w}_1,{w}_2\right)\right]\right\} $$

Semantic similarity by Lin (Reference Lin1998):

(22) $$ \mathrm{sim}\left({w}_1,{w}_2\right)=\frac{2 IC\left[\mathcal{K}\left({w}_1,{w}_2\right)\right]}{IC\left({w}_1\right)+ IC\left({w}_2\right)} $$

Semantic similarity by Meng et al. (Reference Meng, Huang and Gu2014):

(23) $$ \mathrm{sim}\left({w}_1,{w}_2\right)={\left\{\frac{2 IC\left[\mathcal{K}\left({w}_1,{w}_2\right)\right]}{IC\left({w}_1\right)+ IC\left({w}_2\right)}\right\}}^{\frac{1-\exp \left[-0.08\mathcal{D}\left({w}_1,{w}_2\right)\right]}{\exp \left[-0.08\mathcal{D}\left({w}_1,{w}_2\right)\right]}} $$

Semantic similarity by Resnik (Reference Resnik1995):

(24) $$ \mathrm{sim}\left({w}_1,{w}_2\right)= IC\left[\mathcal{K}\left({w}_1,{w}_2\right)\right] $$

Semantic similarity by Zhou et al. (Reference Zhou, Wang and Gu2008b) is a weighted average of k× path-based Leacock–Chodorow similarity and (1 − k)× information content-based Jiang–Conrath similarity. Usually, the two weights are set to be equal k = 1 – k = $ \frac{1}{2} $ .

Cluster analysis based on RG-65 dataset

The development of many semantic similarity measures based on the hypernym–hyponym hierarchy in WordNet was done by their authors using as a testbed the RG-65 dataset containing 65 noun–noun pairs, whose semantic similarity is evaluated from subjective reports collected from human subjects (Rubenstein and Goodenough, Reference Rubenstein and Goodenough1965). Consequently, we have also used the RG-65 dataset to assess the degree of correlation between subjective human evaluation of word similarity and all semantic similarity measures reported in this work. Pearson correlation analysis shows that subsumer-based similarity measures are least correlated with subjective human evaluation (average r = 0.67, P < 0.001), followed by path-based similarity measures (average r = 0.82, P < 0.001), and information content-based similarity measures (average r = 0.84, P < 0.001). Consequent hierarchical clustering segregates subsumer-based similarity measures into a single small cluster that is least correlated to human evaluation, but mixes path-based and information content-based similarity measures into another large cluster (Figure 5). Although any of the available path-based or information content-based similarity measures could be used for exploration of divergent or convergent thinking in conversational transcripts, previous experimental study of creative cognition by design students in real-world educational setting has found that information content-based similarity measures exhibit highest statistical power to differentiate between successful and unsuccessful ideas (Georgiev and Georgiev, Reference Georgiev and Georgiev2018). Here, we have identified the information content formula by Sánchez–Batet (6), as the one that exhibits highest correlation with human evaluation of word similarity, with an average r = 0.85 across all five information content-based semantic similarity formulas. Furthermore, the semantic similarity formula by Lin (22) has the highest r = 0.85 when used with Sánchez–Batet formula (6) among all purely information content-based semantic similarity formulas, which are computationally fast to execute in real-time application. Thus, the combination of formulas (6) and (22) ensures best correlation with human evaluation of similarity and optimizes computational speed for engineering applications.

Figure 5

Pearson correlation matrix map with hierarchical clustering (dendrogram) based on the Pearson correlation distance between subjective human evaluation (HE) of word similarity for noun–noun pairs in the RG-65 dataset and 46 objective semantic similarity measures computed with the use of WordNet 3.1. The similarity measures segregate into two clusters, a larger cluster composed from information content-based or path-based similarity measures, and a smaller cluster composed from subsumer-based similarity measures. Formulas for the similarity measures are provided in the main text. Abbreviations: AN: Al-Mubaid–Nguyen, B: Braun-Blanquet, BHK: Blanchard–Harzallah–Kuntz, D: Dice, J: Jaccard, JC: Jiang–Conrath, K: Kulczyński, L: Lin, LBM: Li–Bandar– McLean, LC: Leacock–Chodorow, MGZ: Meng–Gu–Zhou, MHG: Meng–Huang–Gu, OO: Otsuka–Ochiai, R: Resnik, RMBB: Rada–Mili–Bicknell–Blettner, S: Simpson, SB: Sánchez–Batet, SBI: Sánchez–Batet–Isern, SVH: Seco–Veale–Hayes, WP: Wu–Palmer, YYW: Yuan–Yu–Wang, ZWG: Zhou–Wang–Gu.

Construction of semantic networks from design conversations

The concurrent verbalization may not access all cognitive processes involved in design thinking. Nevertheless, the language provides an output channel of information that could be used to monitor the ongoing design process and an input channel of information that could be used by the designer to incorporate external feedback on the designed product. Thus, it is desirable to assess whether verbalization and language could be useful in aiding the design process, even though they do not exhaust everything that goes on in the designer’s mind. Next, we illustrate with a concrete empirical example how design review conversations could be analyzed using moving time window and demonstrate the utility of dynamic semantic measures to differentiate between successful and unsuccessful ideas.

Distinguishing successful ideas from unsuccessful ideas

For numerical analysis, we have employed the experimental dataset of complete transcripts with design review conversations provided as a part of the 10th design thinking research symposium (DTRS 10) including two subsets with students majoring in industrial design: a subset with seven junior students and a subset with five graduate students (Adams and Siddiqui, Reference Adams and Siddiqui2013, Reference Adams and Siddiqui2015). Each design project consisted of five stages: (1) task review, (2) concept review, (3) client review, (4) concept reduction review, and (5) final presentation. For each project, the students developed several possible design solutions (Figure 6), from which only the best one was selected to appear in the final presentation.

Figure 6.

Part of design conversation (concept review) from DTRS 10 dataset.

This final design solution, which was selected after consultation with the client, is considered to be the successful idea because it has won the competition with other design solutions (ideas) that were not successful in regard to appearing in the final presentation. Here, our main motivation is to distinguish the best design solution from all the rest. Segmentation of the transcripts with regard to successful and unsuccessful ideas was performed based on the videos and the presentation slides. Overall, there were 12 successful ideas, and 41 unsuccessful ideas in the DTRS 10 dataset (Georgiev and Georgiev, Reference Georgiev and Georgiev2018).

Construction of moving time window

To test for possible relationship between attributes of divergent/convergent thinking and the success of design ideas, we have split the design conversation transcripts for each idea and have constructed dynamic semantic networks with a moving time window that contains six distinct nouns. This approach is different from bag-of-words, because it does not keep multiplicity of nouns. The first time window is constructed by removing repeated nouns in the conversation until there are collected six distinct nouns. Average (mean) information content of the six nouns in each time window was computed using the Sánchez–Batet formula (6), whereas average (mean) semantic similarity of the 15 noun pairs was computed with the Lin formula (22). These particular formulas have been also found to differentiate well between successful and unsuccessful ideas when the design conversations are split into three equal parts (Georgiev and Georgiev, Reference Georgiev and Georgiev2018).

The temporal duration for the development of each idea was normalized within the unit interval, so that time = 0 indicates the start and time = 1 indicates the end of the design review conversation that pertains to the idea under consideration. The shortest time step corresponds to the appearance of the next noun in the conversation. If the next noun is already repeated in the preceding time window, the time step is added but there is no dynamic change of the semantic network. Alternatively, if the next noun is not repeated in the preceding time window, then both the time step is added and there is a dynamic change of the semantic network.

Because different students had generated different numbers of unsuccessful ideas, to ensure equal weight of each project we have first averaged the dynamic trajectories per student and only then we have averaged the trajectories of unsuccessful ideas across students. Despite that the resulting average trajectories of semantic measures appear to be noisy, it is possible to extract smooth trendlines using linear best fit that minimizes the sum of squared residuals (Figure 7).

Figure 7.

Dynamics of semantic measures for successful ideas (selected for final presentation) or unsuccessful ideas (not selected for final presentation) computed from design review conversations. (A) The information content increases in time for successful ideas, whereas it decreases for unsuccessful ideas. (B) The semantic similarity decreases in time for successful ideas, whereas it increases for unsuccessful ideas. Legend: s, average trajectory of successful ideas; u, average trajectory of unsuccessful ideas; L(s), linear best fit of successful ideas; L(u), linear best fit of unsuccessful ideas. The averages are based on n = 12 student projects in industrial design from DTRS10 dataset.

Statistical differences in the rates of change of semantic measures

To test whether the linear trendlines constructed with moving time window are able to extract faithfully the rates of change (slopes of the trendlines) for information content and semantic similarity reported in previous study where the design review conversations were divided into three equal parts (Georgiev and Georgiev, Reference Georgiev and Georgiev2018), we have employed one tailed paired t-tests. The statistical analysis indeed confirmed that the information content exhibited positive rate of change (k _s = 0.024) for successful ideas, whereas it exhibited negative rate of change (k _u = –0.012) for unsuccessful ideas, (t = 2.24, P = 0.024). Opposite dynamics was observed for semantic similarity, which exhibited negative rate of change (k _s = –0.08) for successful ideas and positive rate of change (k _u = 0.01) for unsuccessful ideas, (t = –2.05, P = 0.032). Computationally, divergent thinking is identified by negative rate of change of semantic similarity in time, whereas convergent thinking is identified by positive rate of change of semantic similarity in time. Thus, consistent with psychological theories linking creativity with divergent thinking (Guilford, Reference Guilford1957; Hudson, Reference Hudson1974; Runco, Reference Runco2004, Reference Runco2007; Runco and Pritzker, Reference Runco and Pritzker2020), we have found that successful ideas exhibit computational attributes of divergent thinking such as increasing information content (Figure 7a) and decreasing semantic similarity (Figure 7b) during the development of ideas in time, whereas unsuccessful ideas exhibit computational attributes of convergent thinking such as decreasing information content (Figure 7a) and increasing semantic similarity (Figure 7b). These results were obtained in retrospective fashion through analysis of human curated transcripts, which eliminated machine errors in speech-to-text conversion and also performed post-processing of nouns such as conversion of plural to singular and omission of nouns that are absent from WordNet 3.1. However, the numerical plots establish as a proof of principle that conversation analysis could be employed in real time and the trendlines for semantic measures could be provided to the designer as a future forecast of whether the design product is going to be successful.