1. Introduction
1.1. Creative state and resonance
Design, by its nature, requires collaboration and interaction. There have been many discussions in design fields related to collaboration that have focused on the results (Reference Välk and MougenotVälk & Mougenot, 2019), on qualitative analyses of the ways in which creativity is demonstrated in groups (Reference Akaki and MaenoAkaki & Maeno, 2023), and on the impact of group characteristics such as group diversity (Reference Hoever, van Knippenberg, van Ginkel and BarkemaHoever et al., 2012) and culture (Reference Gong, Gonçalves, Nanjappan and GeorgievGong et al., 2024) on creativity. Most of these studies have focused on the creativity that results from collaboration, and the relationship between team cohesion, synchronization, and team performance is often discussed (Persaud et al., Reference Persaud, Prakash and Flipsen2021; Sitarama & Agogino, Reference Sitarama and Agogino2024). In recent years, there has been research on quantitative modeling, including research that simulates this using a computational model (Reference Perišić, Štorga and GeroPerišić et al., 2023) and research that uses brain waves to capture the relationship between synchronization and creativity (Reference Liang, Li, Zhou, Chen, Li, Chen, Zhao, Huang, Lu, Yu and ZhouLiang et al., 2022). Meanwhile, design is expanding in definition and scope, and it is said that there are three kinds of design: design for expression, design to solve a given problem, and design that does not aim to solve a problem but pursues an ideal, such as a design for society (Reference Taura and NagaiTaura & Nagai, 2010). Problem-solving design focuses mainly on the creativity of the product, whereas design in pursuit of an ideal focuses on the creative moment (Reference Gonçalves, Cardoso and Badke-SchaubGonçalves et al., 2013). For the latter type of design, subjects should ideally feel the joy and excitement of creation (Reference Taura and NagaiToshiharu Taura & Yukari Nagai, 2010).
Co-creation describes a collaboration in which individuals share the socialization stage and work together to create something and, in the process, find a shared goal (Reference Matsumae and NagaiMatsumae & Nagai, 2019). In this situation, “resonance” is the ideal state. This is where each participant’s creative moment resonates with the others to reach the ideal creative state. This phenomenon sees each person in the design process experiencing not only independent creative moments but also sharing mental images with others and stimulating and resonating with each other’s creative states. The importance of discussions focusing on the subject experiencing “resonating” in co-creation has been pointed out. For example, it has been suggested that resonating co-creation forms a co-creative relationship called “intersubjectivity” among co-creation subjects. This is more dynamic than the creative acts of individuals or cooperative collaborations based on defined static relationships, and it leads to the sustained development of co-creation phenomena (Reference Matsumae, Matsumae and NagaiMatsumae et al., 2020). The importance of co-creation has also been pointed out, as the intimate interaction in co-creation forms intersubjectivity and influences creativity (Reference RouseRouse, 2020). However, the dynamics of this resonance have not been clarified, and it remains to be seen what kind of experience resonance gives the co-creation subjects and how it differs from the creative moment in individual creative acts.
1.2. Quantitative evaluation of resonance and Hidden Markov Models
It is only in hindsight that a designer or observer can identify the point at which important concepts in the design process began to emerge (Reference Dorst and CrossDorst & Cross, 2001). In order to adequately describe the design process, many protocol studies have been conducted (Reference Perisic, Štorga and GeroPerisic et al., 2021; Reference Vieira, Kannengiesser and BenedekVieira et al., 2022). When attempting to clarify the cognition of the design subject, such as resonance, many of these studies have used self-reported measurements. However, these are sometimes insufficient to quantify cognition, so it is important to combine both subjective and objective indicators (Reference Cass and PrabhuCass & Prabhu, 2023). From this, we can see the importance of evaluation methodologies that can quantitatively estimate resonance.
Previous studies of quantitative evaluations and modeling of creative states that arise during co-creation, including resonance, pursued subjective quantitative evaluations of intersubjectivity formation and co-creativity as key human factors of co-creation mechanisms (Reference Matsumae and NagaiMatsumae & Nagai, 2019), and there was an attempt to grasp the degree of intersubjectivity formation using the electromyography of facial muscles as a cue (Reference Ehkirch, Kakiuchi, Motomura and MatsumaeEhkirch et al., 2021). However, few studies have discussed co-creation dynamics and group creativity using objective indices, or discussed the relationship between resonance and individual creativity. Here, the authors have attempted to quantitatively evaluate this resonance using the Levenshtein distance (Reference Shoji, Motomura and MatsumaeShoji et al., 2022), and were able to capture the difference between resonance and when both parties were simply in a creative state. However, they were only able to capture the differences and could not reach a quantitative evaluation method that describes the dynamics of resonance.
In this study, by using biosignal indicators to estimate unobserved creative states, the authors attempted to apply Hidden Markov Model (hereafter, HMM) that have been successfully used mainly in the speech field (Reference Mustafa, Allen and AppiahMustafa et al., 2019; Reference RabinerRabiner, 1989) and which are a method of analysis (Reference Uchiumi and MochihashiUchiumi & Mochihashi, 2018) applied to a wide variety of time series data with both unobserved state variables and observed variables. It is clear that the volume of biosignal indicator activity differs for each creative state (Reference Shoji, Motomura and MatsumaeShoji et al., 2022) and the authors expect this provides the potential to estimate creative states from observed speech waveforms. If the HMM can estimate such creative states, this research could contribute to our understanding of the characteristics that promote resonance.
2. Research method
A concept generation experiment in pair work was conducted with the assumption of a discrete transition of the creative states during co-creation, which cannot be directly observed. The authors attempted to estimate the creative state in pair work by using the HMM, which is itself used to estimate the transition of unobserved hidden states from the observed series. The results of the estimation of the creative state arrived at by using the HMM with the results of the subjective evaluation drawn from each examinee’s reflective description were examined to determine the validity and potential of the methodology. This model was also applied to estimate the creative state during individual work. The authors attempted to understand the dynamics and the nature of resonance during co-creation by comparing co-creative pair work and individual creative work.
2.1. Experimental method
2.1.1. Examinees
The examinees in this experiment were already familiar with collaborative concept generation, since it would be difficult to create resonance if they were not familiar with the experimental task itself. In all, 14 pairs of 28 undergraduate students in their third and fourth years in the School of Design at Kyushu University participated. Each of them confirmed in advance that they had experienced resonance during concept generation in collaborative design on a daily basis.
2.1.2. Experimental environment
The experiment was conducted in a laboratory on Ohashi Campus, Kyushu University, in November 2020. A clock was placed on the desk to show the time. Pens, colored pencils, and blank sheets of paper were provided for free and unlimited use. Four video cameras recorded the experiment for review. One camera was positioned to record an overall view of the experiment, while another was used to record the drawings on the pair’s working table. Each of the other two cameras was focused on each examinee’s facial expressions.
2.1.3. Experimental procedure
After entering the room, examinees were told about the experimental procedure and fitted with electrodes for measurement. They practiced an icebreaking exercise for 20 minutes and individual concept generation for 15 minutes so that they could get accustomed to the experiment in advance. After practice, there was a 10-minute break before the experiment started with concept generation work in pairs. The pair concept generation tasks were conducted for about 20 minutes. Immediately following pair and individual concept generation, the examinees each reviewed their work with recorded materials, video, and worksheets made during each concept generation work, and the examiner recorded each of their reviews on a common template (Figure 1).
Experimental procedure
2.1.4. Experimental task
The examinees were asked to work in pairs on a concept generation task based on combinations of two different nouns. The creativity of conceptual combinations in this case was manifested in the diversity of interpretations and polysemous phrases that lead to more interpretations. Noun-noun compounds generate more meanings on average than adjective-noun compounds, and those containing artifacts and superordinate concepts lead to yet significantly more interpretations. (Reference Costello and KeaneCostello & Keane, 2000). With this in mind, the authors chose a set of two polysemous nouns combinations which contain artifacts and superordinate concepts: “sound” and “vehicle” for individual work and “weather” and “drawing tool” for pair work. As all examinees in this study were students of design, the examiners focused on choosing nouns with similar conditions among them, considering the diverse design fields and knowledge levels of the examinees. Prior to starting the experiment, all examinees were shown the same sets of concept generation examples created from “tomato” and “snow” from previous studies (Reference Nagai, Taura and MukaiNagai et al., 2009) to confirm their understanding of what they needed to do for their experimental task. To lessen any inhibition they might have been feeling, examinees were told that there would be no evaluations of their concepts.
2.1.5. Multimodal biosignal indicators
Zygomaticus major and orbicularis oculi muscles are known to have increased activity when pleasant emotions arise, with the orbicularis oculi in particular being sensitive to the intensity of pleasant emotions (Reference Cacioppo, Martzke, Petty and TassinaryCacioppo et al., 1988). Cacioppo et al. also found that the depth of cognitive processing of language was related to the activity of the mentalis (Reference Cacioppo and PettyCacioppo & Petty, 1981). Thus, by measuring multi-modal biosignal indicators during concept generation in pairs and individual, the authors examined the relationships between each of the bio-indicators and the examinees’ cognitive or emotional status. A multi-modal biosignal amplifier system (Polyam4/Japan Suntech Co., Ltd.) and measurement electrodes (fEMG x 6, ocular EMG x 2, body ground x 1) were used to simultaneously measure multi-modal signals.
-
fEMG: Electromyogram of facial muscles (corrugator supercilii, orbicularis oculi, mental-is)
-
vEOG: Electrooculogram of vertical eye movement to measure blinking
2.1.6. Subjective creative states and resonance
Immediately after completing the experimental task, each of the examinees was asked to rank transitions in their creative states during the experimental task on a 5-level scale from -1 to 3. They were instructed to try to keep it between 0 and 2 (0: non-creative state, 1: moderately creative state, 2: strongly creative state) and to use -1 or 3 only for outliers. The examiner then interviewed each examinee to add the examinee’s thinking processes to the record sheet corresponding to the transitions of creative states, reviewing the video recorded during the experiment at the same time to tag each of the states in an experimental timeline. The examinees were also asked to specify the timing of when they felt resonance.
2.2. Evaluation methodology
2.2.1. Subjective evaluation
To verify whether it was possible to estimate the creative states, these states were mainly identified based on the examinee’s answers. For the subjective evaluations, the authors binarized the creative state into creative or non-creative states for each examinee based on the transition of the creative state, the thinking process, and the recorded video. The threshold was set for each examinee to binarize the creative state based on their descriptions of their thinking process, with a state recorded above the threshold determined to be creative and that below as non-creative. These binarized creative states were carefully reviewed along with each examinee’s description of his/her thinking process and the observed video. For instance, if the examinee explained the state as “I stopped thinking and had an absent state of mind,” the state was judged as non-creative even if the binarized data showed it to be creative.
Subjective evaluation of creative states
2.2.2. Objective evaluation
The fEMGs (corrugator supercilii, orbicularis oculi, mentalis) and vEOG were measured at a sampling frequency of 1.0 kHz, and waveforms were obtained at 100 Hz by extracting one of every 10 data. Based on the subjective evaluation, representative R, H, and L datasets were extracted. The fEMG datasets were rectified and transformed into rectified waves by ARV (Average Rectified Value) every 0.2 seconds, and the vEOG datasets were averaged every 0.05 seconds.
2.3. Analysis method
This study estimated the creative state transitions occurring during the work as recorded by the examinees by modeling them with an HMM, which is used to estimate the transitions of hidden states that are not observed in an observation series. Therefore, the first-order HMM was used to estimate the discrete transitions of the creative state of the three states (L, H, R) shown in Figure 3 below, with the biosignal indicator group as the observed variable, and assuming a multivariate normal distribution between the state variable and the observed variable. In addition, although state variables are generally assumed to be unobservable in HMMs, in this study, the transitions of creative states obtained from examinees’ responses are treated as correct data. Based on this information, initial state probabilities, transition probabilities, and occurrence probabilities were calculated and set as model parameters. The observed series were inserted into the model and decoded to estimate the transition of the creative state. For pair-work estimation, a 5-fold cross-validation was used to separate the training and validation data sets. For individual work estimation, the model built with the pair-work data set was used to estimate creative states during individual work.
Transition model
2.3.1. Analysis process
To calculate the HMM parameters, the following processes were performed for subjective evaluation and biosignal indices. To assume a self-loop of states in the HMM of this study, it was necessary to separate creative states and biosignal indices at timed intervals and to specify the discrete states of each. The time interval was set to 15 seconds because the smallest unit of resonance is about 15 seconds (based on the examinee’s reflections).
-
Subjective creative states: The 15-second mode of L, H, and R as the creative state in that 15-second period.
-
fEMG: 15-second average of rectified waves by ARV (average rectified value) every 0.2 seconds.
-
vEOG: 15-second blink time (frequency) calculated by detecting blinks with a -50μV threshold from the smoothed waveform averaged every 0.05 seconds.
2.3.2. Parameters of HMM
The parameters of the HMM were set as described below, and the creative state was estimated by inserting an observation series consisting of four biosignal indices into the model. In addition, the data for two examinees were excluded from the training and estimation datasets due to the unreliability of the subjective evaluation records of the transition of creative states, and the observation series were missing.
Initial state probability and transition probability:
The initial state probability was calculated by confirming the state of each person at the start of the work based on the transition of their creative state every 15 seconds, and the transition probability was similarly calculated based on the transition of their creative state every 15 seconds.
Emission probability:
Assuming a multivariate normal distribution between state variables and observed variables, the mean value and variance-covariance matrix were calculated for each state (L, H, R) for each group of biosignal indicators (facial muscles and electrooculogram) that had undergone preparatory processing for analysis, and these were used as parameters for the multivariate normal distribution.
3. Result and discussions
3.1. Parameters of HMM
The parameters shown in this chapter are calculated according to the data from all examinees during the pair work, except for the examinees mentioned above. Table 1 shows the probability of each state (L, H, R) being the initial state. The probability of the initial state being R was 0. Table 2 shows the probability of transitioning from one previous state (PREVIOUS) to the current state (PRESENT). Specifically, “0.947” in the upper left corner indicates the probability that the current state would be “L” when the previous state was “L”.
Initial probability
Transition probability
Table 3 and Table 4 show the mean value for each biosignal index by state (Table 3) and the value of the variance-covariance matrix by state, respectively, of the probability of occurrence (multivariate normal distribution) for each state. Specifically, “0.006” in the upper left corner of Table 3 indicates the mean value of the corrugator supercilii muscle in multivariate normal distribution at state L. Table 4 shows the values of the variance-covariance matrix by state. When both rows and columns have the same biosignal index in a state, it shows the variance of the multivariate normal distribution assumed by the relevant biosignal index in that state. When the biosignal index differs between rows and columns, it shows the covariance between the multivariate normal distributions assumed by each of the two biosignal indexes in question. So, for example, if the row is for the corrugator supercilii muscle and the column is for the orbicularis oculus, and if the row is for the orbicularis oculus and the column is for the corrugator supercilii muscle, the covariance is equal, so the values are the same in the table. Specifically, the “3.23 × 10−6” in the upper left corner indicates the variance of the corrugator supercilii muscle in multivariate normal distribution at state L, and the “5.62 × 10−6” to the right of it indicates the value of the covariance between the corrugator supercilii muscle in multivariate normal distribution and the orbicularis oculus in multivariate normal distribution.
Mean value of each objective indicator for each state
Variance-covariance matrix for each state
3.2. Estimation accuracy
The resulting estimations of the HMM were evaluated as “well-matched” if the agreement between the examinee’s response and the estimated results was significantly greater than the probability of the event occurring by chance. The estimations were evaluated as “not well-matched” otherwise. The ratio of the well-matched groups to the total estimation was then determined. As shown in Figure 4, there were 18 well-matched groups and 8 not-well-matched groups, indicating that the HMM can be used to estimate the creative state during co-creation. Specific estimation results for each examinee are shown in Figure 5.
Match between examinee’s responses and estimations of creative state during pair work
Results of estimating examinee responses and creative states for each examinee
It was suggested that it would be possible to estimate the creative state during co-creation with a HMM. In addition, there was also an example in which the difference between the subjective creative state and resonance could be estimated, quantitatively suggesting that the state of feeling resonance is different from that of simply being in a creative state. This suggested the application of feedback to the design process based on the estimation results of the HMM. The authors hope to pursue this point by analyzing whether estimation results differ depending on the subject’s level of skill in the collaborative design act and individual characteristics.
3.3. Application of HMM for pair work to individual work
3.3.1. Applicability to individual work
The results of the estimations of creative states during individual work using the HMM with modeling using data from the paired work are shown in Figure 6 below. As there are no data missing from the data on individual work, the data of 28 people are estimated.
Match between examinee’s and estimations of creative state during individual work
As shown in Figure 6, there were 16 well-matched groups and 12 not-well-matched groups. The results of the individual work using the HMM that was modeled using the data from the paired work were also good, and it was shown that it is possible to estimate the creative states during individual work using the model from the paired work.
3.3.2. Understanding resonance during co-creation by comparing individual and pair work
The authors estimated the creative states during individual work using a model that includes the R transition unique to co-creation. When estimating the creative state during individual work using an HMM, R (resonance) was estimated to be about 3.4% of the total. The authors focused on that time estimated as R. As a result, it was confirmed that there were sections where specific images were formed and where ideas were developed, marked by factors such as the further elaboration of the ideas, and sections where participants were excited by the ideas that had emerged and where their creative state was particularly high. There were also sections where no particular trends were seen, but these were only temporary, lasting around 30 seconds.
These findings suggest that resonance in pair work and self-resonance in individual work represent qualitatively similar cognitive phenomena. Therefore, resonance can be understood to be perceived at the individual level during co-creation and interpreted as the sharing of self-resonance, wherein concrete concept is generated within an individual, strongly stimulating their creative states and enhancing interaction between the individual and their concept. Based on these findings, resonance during co-creation is suggested to be a cognitive state equivalent to self-resonance. It is a phenomenon in which self-resonance—directly stimulated by concept generation—becomes shared among individuals. Furthermore, the results support the validity of defining sharing the socialization phase of SECI, the process of interpersonal tacit knowledge sharing through interaction (Reference Nonaka and TakeuchiNonaka & Takeuchi, 1995), as a key marker of co-creation (Reference Matsumae and NagaiMatsumae & Nagai, 2019).
4. Conclusion
The results indicated potential of HMM as the method to estimate creative states during co-creation. The findings suggest that resonance during pair concept generation and self-resonance during individual concept generation are qualitatively similar cognitive states. During co-creation, resonance is perceived individually as self-resonance, where concept generation stimulates creativity and enhances interaction. Thus, resonance in co-creation can be understood as a shared form of self-resonance driven by concept generation.
The samples may not represent a diverse range of design teams, and the results may not be applicable to other situations. In the future, the authors would like to consider a wider range of participant attributes and skill levels, and would also like to improve the model by optimizing the measurement indicators and considering other non-invasive approaches.
Ethical statement
This study was approved by the Institutional Review Board of Kyushu University.
Acknowledgement
This work was supported by JSPS Grant-in-Aid for Scientific Research JP20K20119.
Open access
