2.1. Intersubjectivity & interactive level

Interaction forms the basis of our understanding of reality, and it is through the experience of interaction that we can create our subjectivities. First conceptualised by Husserl in his Fifth Cartesian Meditation as the mechanism behind empathy, see Bower (Reference Bower2015), the main idea of intersubjectivity is that subjects do not constitute a world alone but jointly with other subjects. It can be understood as a field that allows social interactions created jointly between subjects – a socially interactive field (Ehkirch & Matsumae Reference Ehkirch and Matsumae2024). Serving as the main idea to fuel the shift to dialogism in fields studying human factors in social situations, such as cognitive science, psychology and behavioural science, this new view has led to important steps forward in our understanding of social interactions. Coming from the study of the interpersonal aspects of language and the social co-construction of meaning in it, this shift from the previous personal perspective of cognition was then applied to study the interactional aspects of social cognition in Linell (Reference Linell2007), pushing for a new way to analyse intersubjectivity as an aspect of discourse subjects (Gillespie & Cornish Reference Gillespie and Cornish2009). From a dialogical perspective, intersubjectivity can be approximated through the flow of interaction. This flow is understood through the measurement of interpersonal factors to integrate interactive moments together through their intensity level, dependent on the social situation studied (Ehkirch & Matsumae Reference Ehkirch and Matsumae2024). Applied in a design context, it can overcome the previous individual aspects of creativity to focus on the collaboration that sustains it at an interpersonal level. Pushing for a better understanding of creativity in design using knowledge and tools from socio-cognitive research (Gero & Milovanovic Reference Gero and Milovanovic2020).

In the case of co-design, to quantify interactive level dynamics, previous research has used factors such as shared understanding in Cash et al. (Reference Cash, Dekoninck and Ahmed-Kristensen2017), or boundary objects (Star Reference Star2010). These factors focus on the results of the dynamic context created by the integration of personal static moments through the flow of interactions that is formed during the co-design process. They can then help quantify not only the level of the interaction (static) but also its direction (dynamic), and are key to understanding the steps that will follow. The interactive level can then be quantified using interpersonal factors that are reflective of the principles that make empathy through the sharing of cognitive states such a powerful tool in the shared design context between designers (Chang-Arana et al. Reference Chang-Arana, Piispanen, Himberg, Surma-Aho, Alho, Sams and Hölttä-Otto2020; Svanæs & Barkhuus Reference Svanæs and Barkhuus2020). These feelings of connection and sharedness, in this study, are referred to as the pairness factor, which allows the creation of a new social space, where the sharing of norms and values becomes easier and more complete (Keating & Jarvenpaa Reference Keating and Jarvenpaa2011). It has also been related to intersubjective engagement, correctly measuring the interactive level (García-Pérez et al. Reference García-Pérez, Lee and Hobson2007).

In addition, collaboration is reached by creating engagement from its participants to maintain it (Cheung & To Reference Cheung and To2011; Kleinsmann et al. Reference Kleinsmann, Deken, Dong and Lauche2012). In that sense, co-design is also heavily reliant on the motivation of the individuals participating, especially so in the case of co-creative work. Indeed, a positive correlation was found between the rise of intersubjectivity (understood as the interactive level) and co-creativity (defined as a shared motivation among individuals) in realising a concept (Matsumae & Nagai Reference Matsumae and Nagai2018). Motivation is then closely related to the goal and individual design context of each participant, as explained by Amabile (Reference Amabile1983), and co-creative collaboration (co-creation) cannot be sustained without a shared motivation and agreement on common directions for the creative process. Using the knowledge from intersubjectivity research opens the possibility to go beyond the personal study of such factors and to develop the interpersonal aspects of creativity, shining light on the social mechanisms at play in co-design.

2.2. Co-creation

The need to bring the concept of creativity from the personal to the interpersonal level was recognised during the development of design methodology in the 1970s as the key to the successful transition from participatory design to collaborative design. Co-creative collaboration needs more complex and intricate social interactions between its participants to foster not only interesting and creative results but also a better experience of design as a social activity (Busciantella-Ricci & Scataglini Reference Busciantella-Ricci and Scataglini2024). From a design cognition perspective, it is characterised by both divergent and convergent processes implicated in the co-evolution of the iterative exploration between problem space and solution space (Cascini et al. Reference Cascini, Nagai, Georgiev, Zelaya, Becattini, Boujut, Casakin, Crilly, Dekoninck, Gero, Goel, Goldschmidt, Gonçalves, Grace, Hay, Le Masson, Maher, Marjanović, Motte and Wodehouse2022). This means that the social interactions experienced during co-creation directly influence the creative output of it by modifying the dynamics of said interactions (Cheung & To Reference Cheung and To2011; Ozer & Zhang Reference Ozer and Zhang2022). There is then a need to focus on human factors in co-design to better understand the social interaction mechanisms that influence interpersonal creativity, as explained by Beghetto & Karwowski (Reference Beghetto, Karwowski, Beghetto and Corazza2019), leading to a better experience of design that could help bridge the boundary between designers and users in a social design approach.

Furthermore, from a process perspective, there are two different collaborations taking place during co-design, one being co-operative collaboration (i.e. cooperation), where only participation is needed to achieve a defined common goal based on each member’s subjectivity, and the other being co-creative collaboration (i.e. co-creation), where each member’s creative participation is required. The latter is based on intersubjectivity during the socialisation phase, making possible the sharing of tacit knowledge and allowing a richer experience (Nonaka et al. Reference Nonaka, Toyama and Konno2000; Matsumae & Nagai Reference Matsumae and Nagai2018). Co-creation is then a process that occurs during co-design where the co-creativity of participants synergistically reaches heights that exceed the sum of their creativity (Trischler et al. Reference Trischler, Pervan, Kelly and Scott2018). It is a similar idea to group flow creativity, presenting creativity as interpersonal and collaborative by nature (Sawyer Reference Sawyer2017). The idea of a flow state in a group collaborative effort (as a shared experience of an interaction) can be associated with the idea of intersubjectivity being approximated by the flow of interaction, understood through the measurement of interpersonal factors to integrate interactive moments together through their intensity level. It defends the idea that this creative shared flow state is dependent on a shared understanding of the goal, a diverse communicational style and tacit knowledge created by the team. The main difference between the two concepts is the distinction between subject and process, which is interpersonal subjectivity for intersubjectivity (i.e. emerged in interactions) while being mostly situational for group flow (i.e. emerged from social contexts). This paper then defends an interpersonal approach to creativity as a cognitive state of the co-design experience. This state can be understood through the interactive level dynamics approximating the intersubjective field in co-creation (Ehkirch & Matsumae Reference Ehkirch and Matsumae2024). It thus seems that intersubjectivity can be understood as a shared basis that maintains and aggregates the vectors of the different participants, leading them to co-creative collaboration. This transition in the shared creative state of a team in co-design is made possible by the intersubjectivity formed between participants’ subjectivities, arising from their experience during the design process. It is an unstable state that requires a high level of interactivity before it can blossom from cooperation to co-creation (Roschelle & Teasley Reference Roschelle and Teasley1995), and is sustained by the sharing of motivation, goals and design context between participants, as explained by Amabile (Reference Amabile1983), which fuels the push forward. The creative resonance that results can then grow into a richer experience of participatory creation, pushing even everyday creativity to new heights, in particular in the simple creative task realised in pairs, hereinafter referred to as pairwork (Ho & Lee Reference Ho and Lee2012; Won et al. Reference Won, Bailenson, Stathatos and Dai2014; Matsumae et al. Reference Matsumae, Shoji and Motomura2022). Pairwork showed improvement in measurable requirements of design compared to solo work (Al-Kilidar et al. Reference Al-Kilidar, Parkin, Aurum and Jeffery2005). However, most research done in this direction still focuses on a single measurement approach to these constructs, relying on qualitative analysis to study the outcomes afterwards of the interaction, losing the essence of their dialogical aspects (Sosa & Gero Reference Sosa and Gero2016; Hay et al. Reference Hay, Cash and McKilligan2020). This means that there is a need to study them dynamically from a dialogical perspective using measurement of behaviour and cognition simultaneously (Granados Reference Granados2000; Balters & Steinert Reference Balters and Steinert2017). Such cognitive states are heavily dependent on smooth social interactions that can be objectively observed through biosignal indicators of social behaviour.

2.3. Biosignal indicators of social behaviour

Design cognition has seen increasing attempts to connect with psychological theory to integrate and build upon the findings of earlier exploratory protocol analyses (Hay et al. Reference Hay, Cash and McKilligan2020). This led to advocating for multi-modal measurements that include measuring behaviour, cognition, physiology and neurophysiology concurrently (Cascini et al. Reference Cascini, Nagai, Georgiev, Zelaya, Becattini, Boujut, Casakin, Crilly, Dekoninck, Gero, Goel, Goldschmidt, Gonçalves, Grace, Hay, Le Masson, Maher, Marjanović, Motte and Wodehouse2022). While also proving to be useful to study design creativity and innovation in the interaction between personal and interpersonal levels (Somech & Drach-Zahavy Reference Somech and Drach-Zahavy2013).

Social cognitive studies have long examined the role of specific social and communicative behaviours in interactions, such as the chameleon effect in Chartrand & Bargh (Reference Chartrand and Bargh1999) or gaze (mutual eye contact) effects (Akechi et al. Reference Akechi, Senju, Uibo, Kikuchi, Hasegawa and Hietanen2013). These interpersonal factors have so far been mainly studied using subjective reports (Silvia et al., Reference Silvia, Beaty, Nusbaum, Eddington, Levin-Aspenson and Kwapil2014) and/or exterior observations (Mundy et al., Reference Mundy, Kasari and Sigman1992; McClintock & Hunt, Reference McClintock and Hunt1975), but the shift towards dialogism has opened the possibility of linking both objective and subjective results at an interpersonal level to provide a dynamic view of the interactions by introducing new interdisciplinary methodologies. For example, a recent study Barraza et al. (Reference Barraza, Pérez and Rodríguez2020) of the concept of shared intentionality used the interbrain method to deepen our knowledge of such socio-cognitive processes. From an objective standpoint, non-verbal communicative behaviour can help approximate the interactive level by providing objective markers (Mundy et al. Reference Mundy, Kasari and Sigman1992; García-Pérez et al. Reference García-Pérez, Lee and Hobson2007; Won et al. Reference Won, Bailenson, Stathatos and Dai2014). Furthermore, there is evidence that by focusing on biosignals related to different socio-cognitive processes at different levels of cognition, a better understanding of the roles these behaviours play in specific contexts (in our case, during co-creation) can be found (Ehkirch & Matsumae Reference Ehkirch and Matsumae2024). These biosignals need to be reactive to the social environment and constituted of empathy through a shared perception of social interactions at an interpersonal level to become key interpersonal factors in a specific interaction (Balters & Steinert Reference Balters and Steinert2017; Gero & Milovanovic Reference Gero and Milovanovic2020). In any team creation, a deep cognitive process is needed that shares not only a current understanding of the situation but also how one feels about it. In other words, the cognitive state must be shared (Mundy et al. Reference Mundy, Kasari and Sigman1992; Masclet et al. Reference Masclet, Boujut, Poulin and Baldaccino2021).

Previous studies have shown how the interactive level of interactive situations can be studied by using objective markers of shared behaviours, such as the study of sympathy using PET scan data together with skin conductance (SCR), blood volume pulse (BVP) and respiratory rates in Decety & Chaminade (Reference Decety and Chaminade2003), or shared subjectivities from physiological signals using data from: electroencephalography (EEG) recording, electrooculography (EOG), electrocardiogram (ECG), respiration, electrodermal activity (EDA) and finger electromyography (EMG) in (Maÿe et al. Reference Maÿe, Wang and Engel2021). In design research, such a physiological state has shown the importance of including the cognitive status related to empathy, strongly influencing interactions (Salmi et al. Reference Salmi, Li and Holtta-Otto2023; Chang-Arana et al. Reference Chang-Arana, Surma-Aho, Hölttä-Otto and Sams2022; García-Pérez et al. Reference García-Pérez, Lee and Hobson2007). From an interactive perspective, such social cues and answers can be measured through facial electromyography analysis (fEMG) (Huang et al. Reference Huang, Chen and Chung2004). This method is used to measure muscle activity by detecting and amplifying the tiny electrical impulses that are generated by muscle fibres when they contract. Previous research has shown that fEMG can identify a social cognitive state by focusing on the zygomaticus major and corrugator supercilii muscles, see Larsen et al. (Reference Larsen, Norris and Cacioppo2003), linking them to strong emotional reactions to social situations (Wolf et al. Reference Wolf, Mass, Ingenbleek, Kiefer, Naber and Wiedemann2005). This method provided an indication of underlying affective valence rather than literal proxies of corresponding facial expressions. A link between the results of a comparison of fEMG data and the intersubjective state corresponding to their interactive level was also found (Ehkirch et al. Reference Ehkirch, Kakiuchi, Motomura, Matsumae and Matsumae2021). In design research, empathy has been studied using EMG before, see Chang-Arana et al. (Reference Chang-Arana, Piispanen, Himberg, Surma-Aho, Alho, Sams and Hölttä-Otto2020), or similar facial mimicry from automated facial expression recognition, focusing on the same associated shared behaviours: smiling and frowning (Ikäheimonen et al. Reference Ikäheimonen, Li, Yao, Zuo, Aledavood and Hölttä-Otto2024). Proving to be a good compromise between the precision of measurement methods and complexity of use (Geoghegan et al. Reference Geoghegan, Kwasnicki, Kanabar, Pethers and Nduka2018). In fact, fEMG is able to detect even unconscious patterns, allowing separation between real and feigned associated behaviours as shown in Korb et al. (Reference Korb, Didier and Scherer2008).

In a similar vein, another shared behaviour related to a deeper cognitive state is the blinking pattern (Irwin Reference Irwin2011). Electrooculography (EOG) can be used to measure blinking patterns that are related to the deeper cognitive functions needed during creation, such as mind wandering and cognitive flexibility (Kruis et al. Reference Kruis, Slagter, Bachhuber, Davidson and Lutz2016). It can be used to approximate the amount of sharing in a cognitive state. EOG is measured by observing the corneo-retinal standing potential that exists between the front and the back of the human eye and is often used to study eye movements and recognise blinks (Bulling et al. Reference Bulling, Ward, Gellersen and Tröster2011).

This leads us to the use of fEMG and EOG biosignal data to objectively grasp the current social cognitive state of someone engaged in a co-creative task. However, following the dialogical approach to intersubjectivity, to move from this personal status to the interpersonal, these data need to be compared with other participants to study the synchrony of social behaviours. Here, synchrony should be understood as a measure of the similarities (timewise) of specific socio-cognitive behaviours between persons who are interacting. The study of the synchrony of physiological statuses in design cognition already has produced promising results from a dialogical point of view (Chang-Arana et al. Reference Chang-Arana, Surma-Aho, Hölttä-Otto and Sams2022; Papadopoulou Reference Papadopoulou2024). To this end, this research will propose some new methods of quantifying the synchrony rate of biosignals that can then become objective indicators of the interactive level in co-creation.

4.1. Experiment

4.1.2. Procedure

The experiment was carried out over 20 sessions, as the equipment used to record biosignals was only able to record two persons (corresponding to one pair) at a time. To be able to collaborate without hindrance, the participants were seated in front of each other, across a table on which a canvas (created using a panel and plastic film to make sure that the clay would not stick) with 12 coloured lumps of clay was set. You can see the setting of the experiment in the following picture (Figure 1).

Figure 1.

Setting of the experiment.

To facilitate the appearance of co-creation, the pairwork was divided between two creative tasks, the first one serving as an icebreaker that allowed examinees to get used to the situation, while the second was the actual co-creative task. The first task was defined as the shared creation of “a character that can fly” to be completed within a 5-minute time limit. The goal of this task was to get the examinees used to their partner (as some were meeting for the first time), the setting of the experiment and the equipment used to record the biosignals. It also served to foster the basis needed for interactions that would bring about co-creation during the second task. The instruction of task one to create “a character who can fly” was also given as a precursor to support the next creative task. Five minutes was considered to be sufficient time to serve as an icebreaker, thanks to the playful aspect of the creative task introduced by the use of clay.

The second task was defined as the “creation of a scene (explained as a scene such as one can see in a movie) around the first character,” with the freedom to add any elements needed to make it a coherent story (storytelling aspect). This task was given a 20-minute time limit, but examinees were allowed to continue beyond 20 minutes if needed to avoid disturbing the flow of interactions. The goal of this second task was to spur the appearance of co-creation between the examinees by removing any specific predefined goals or constraints and letting them decide on their own how to proceed.

After these two tasks were completed, the examinees were each asked to subjectively report their interactive level during task 2 (see section evaluation of the interactive level) along with an indication of their familiarity level (i.e. whether the examinees in the pair knew each other beforehand). This experimental procedure is illustrated below (Figure 2).

Figure 2.

Experimental procedure.

These results were related to the biosignal data recorded during task 2 and quantified through a synchrony rate computed from the data of both examinees (see section synchrony rate of biosignals). The main goal of this study was to find at the pair level a relation between the synchrony rate of the three biosignals (fEMG × 2, EOG × 1) computed by two methods (Ld & DTW) and the interactive level quantified through three different approaches (average, difference and dynamics). Each construct is explained in the following section.

4.2. Evaluation of the interactive level

Referring to previous studies such as Matsumae et al. (Reference Matsumae, Shoji and Motomura2022), the interactive level was subjectively evaluated after the completion of task 2. The requirements for the measurement methods were that they needed to concern key interpersonal factors in co-creation that could be studied from a subjective perspective. These factors should be integrated as results of the interaction while having an unstable intensity, meaning enough influence to shift the dynamics of the interactions and change the intensity of the interactive level (see Ehkirch & Matsumae (Reference Ehkirch and Matsumae2024) for the selection of suitable methods for subjective evaluation). They also needed to be shared between the participants to make it pertinent at the pair level. Each examinee was asked to fulfil a dynamic/continuous rating between 0 and 100 of two interpersonal factors: motivation and pairness. Arguments can be made on the choice of these two factors in particular, to quantify the interactive level in the specific case of co-creation as explained in the Background section. There are not only necessary but also directly vital as determinant interpersonal factors for the dynamic fluctuations of the co-creative interaction (García-Pérez et al. Reference García-Pérez, Lee and Hobson2007; Matsumae & Nagai Reference Matsumae and Nagai2018; Ehkirch & Matsumae Reference Ehkirch and Matsumae2024). Hence, their choice as the first two “candidates” for proxies of the interactive level. To help with this process, a research assistant put these ratings in a waveform to reveal their dynamics. Asking the examinee to report their motivation and pairness level while looking at a video recording of task 2 immediately after the session to reduce recall bias (this reporting took around twice as long as task 2, see Figure 2 for details). Finally, it could be argued that assigning numerical values to feelings is inherently problematic due to the subjective nature of the scale, also wondering why our constructs were assigned values between 0 and 100, contrary to the often-used predefined questionnaires, using a Likert scale. This continuous approach to these factors was justified by the dialogical perspective taken from intersubjectivity studies that argues for focusing on the dynamics of constructs, which cannot be done with standardised techniques. This approach for continuous physiological measures was inspired by previous work on affect reporting (Gottman & Levenson Reference Gottman and Levenson1985).

• • Motivation: Motivation was defined beforehand as a key factor needed for the appearance of co-creation (see background). Any type of collaboration needs engagement that is measured through their degree of personal motivation to complete the task given. It was explained to the examinee as: “your feeling of engagement to realise/finish the task given, with 0 corresponding to none and 100 corresponding to full interest.”

• • Pairness: To help understand the interactive level in co-creation, the pairness factor was designed as a measure of the depth of sharing between participants at a given moment (see background). It corresponds to a subjective feedback measurement method (as explained in Ehkirch & Matsumae (Reference Ehkirch and Matsumae2024)) of the degree of connection with their partner and was previously used in research as a main factor to study the interactive level as felt by the examinees themselves (García-Pérez et al. Reference García-Pérez, Lee and Hobson2007). It was explained as: “the feeling of shared behaviours, emotions, motivation and goal with your partner, with 0 corresponding to none and 100 corresponding to a full resonance/connection with your partner (feeling as one).”

As the recording continued, many of the examinees chose to exceed 100 to define their subjective perspective on the interactions at that point. Motivation values ended up ranging from 20 to 150, while pairness values were between 0 and 150. However, this overrating did not present a problem as the goal was to record the dynamics of these factors, and, as such, the measurements were rescaled afterwards to make them compatible. This was justified to standardise the measurements between all examinees, following recommendations explained in McKeown & Sneddon (Reference McKeown and Sneddon2014), especially as it would be elevated at the pair level afterwards. This is explained in more detail in the following pair interactive level method explanation. Here is a pattern of a report for the pairness factor of the interactive level (Figure 3).

Figure 3.

Subjective report of interactive level pairness factor.

4.2.2. Pair level

To quantify the interactive level, individual-level data at the pair level was examined by comparing the results of both examinees in a pair. To simplify the analysis, one factor needed to be selected to indicate the pair’s interactive level. A stronger correlation coefficient was found between the pairness level of both examinees in a pair, $ r(3514)=.35,p<.001 $ , than with their motivation level, $ r(3514)=.20,p<.001 $ . To verify that there was a significant difference between the two correlation coefficients, a Z-test was carried out (Viertl Reference Viertl2009). The calculated Z value is 9.5706, and the acceptance region for the null hypothesis at $ \alpha =0.05 $ is $ -1.96<Z<1.96 $ . So, we can reject the null hypothesis and conclude that the pairness level of two examinees in a pair has a significantly higher correlation than that of the motivation level. This is why three approaches were designed to quantify the pair interactive level from the pairness level value:

1. 1. Average

   The average approach was used to obtain the intensity value of the pair’s interactive level, from low to high. It was computed by using the mean of the relative measures of pairness levels of both examinees in a pair (see Eq 1). It was then divided into three categories: low (between min and Q1), middle (between Q1 and Q3) and high (from Q3 to max). Here, min, Q1, Q3 and max are to be understood as quartiles used in order statistic.

(1) $$ {\displaystyle \begin{array}{c} Average=\left(\left({PL}_{EXa}/\max {PL}_{EXa}\right)+\left({PL}_{EXb}/\max {PL}_{EXb}\right)\right)/2\\ {}{PL}_{EXx}:\mathrm{Pairness}\ \mathrm{level}\ \mathrm{of}\ \mathrm{examinee}\;\left(\mathrm{x}\right)\\ {}\max {PL}_{EXx}:\mathrm{Maximum}\ \mathrm{reported}\ \mathrm{values}\ \mathrm{of}\ \mathrm{the}\ \mathrm{pairness}\ \mathrm{level}\ \mathrm{of}\ \mathrm{examinee}\;\left(\mathrm{x}\right)\end{array}} $$

1. 2. Difference

   The difference approach was used to understand the disparity of interactive levels between the examinees in a pair, from low to high. It was computed by using the absolute differences of the relative measures of pairness levels of both examinees in a pair (see Eq 2). They were then divided into three categories: low (between min and Q1), middle (between Q1 and Q3) and high (from Q3 to max).

(2) $$ Difference=\mid {PL}_{EXa}/\max {PL}_{EXa}-{PL}_{EXb}/\max {PL}_{EXb}\mid $$

1. 3. Dynamics

   The dynamics approach was used to understand the direction of the next interactive level by looking at the derivatives of pairness levels of both examinees in a pair. For each participant pairness level, there are three possible directions, creating a combination of five categories at the pair level (see Table 1 for details of the calculation using the derivate of each examinee). From there, they were categorised incrementally into categories relating to the variety of possible next interactive levels: increase (i), alone (a), same (s), opposite (o) and decrease (d). Details are in Table 1.

Table 1.

Direction categories from the dynamic approach

These three approaches not only helped to quantify the interactive level intensity (average) at a given moment (corresponding to a 15-second datapoint) but also how it is shared between the two examinees in a pair through its disparity (difference) and how it can predict the direction (dynamics) of the next level.

4.3. Familiarity level

Though the experiment was designed to keep the influence of situational factors (e.g. creative environment used) as low as possible, co-creation, like any social activity, is dependent on social factors that can influence the interactive level or, more specifically, its initial state. However, to focus on the link between the biosignal synchrony and the interactive level, there was a need to reduce as much as possible these external (in the sense of not mainly responsible for the co-creation interaction dynamics) factors. As such, only one factor was kept for further analysis, as it was not possible to reduce its influence, the familiarity level between examinees. Previous research showed a link between the level of familiarity and the interactive level attainable (Deppermann Reference Deppermann2019). In a control environment, with a limited amount of time and a controlled interactive channel, pairs that already know each other were able to get more reliably to a higher interactive level during a coordination task. To balance this out, the measure of familiarity examinees in a pair had in working with each other was added as an independent factor in the study. This measurement was done at the same time as the subjective report on the interactive level following task 2. Each examinee was asked how used they were to working together on a 3-point Likert scale. This was done to obtain a simple, unbalanced view of the familiarity the examinees had with each other and to limit possible sway between the pair’s members, going from 0 (meaning the first time) to 2 (pretty used to working together). This measure of familiarity at the individual level was then raised to the pair level by taking the average value of both examinees in a pair, meaning that the scale was divided into five possible values (0/0.5/1/1.5/2). This value was then tested with the other data coming from biosignals synchrony and the interactive level to see if it was influencing our results.

4.4. Biosignal measurement

4.4.1. Biosignal indicators in design settings

During the experiment, the measurement of biosignals needed to be as unintrusive as possible to minimise its impact on the interactions. In recent years, interesting research has been conducted using neurocognitive tools, such as electroencephalography (EEG). The main drawback of these protocol-heavy physiological measurements is that they limit the possible situations studied to simplified tasks far from real-world design practices (Gero & Milovanovic Reference Gero and Milovanovic2020). These limitations from cognitive tools developed to understand human behaviour in more socially realistic and complex settings were determinant to the feasibility of this research. Indeed, fEMG, EOG and other biosignals were still difficult to measure accurately without heavy equipment just a decade ago, making it unrealistic to attempt to use them in tasks where communications between examinees would be the subject of study. However, thanks to today’s lighter equipment, researchers can easily dimmish and/or control the influence of the measurement on the situation itself, making it more relaxing for the participants and allowing them to focus on the task at hand. These improvements justify the choice of this measurement method to get a detailed reading of socially related behaviours without being too invasive for the examinee (see Figure 4 for the level of invasiveness) (Franz et al. Reference Franz, de Filippis, Daloiso, Biancoli, Iannacone, Cazzador, Tealdo, Marioni, Nicolai and Zanoletti2024). Computer vision–based expression/emotion recognition and eye tracking represent important non-invasive alternatives that have shown significant progress. Still, fEMG and EOG emphasise physiological accuracy (some may not be visually observable or detectable by computer vision methods), robustness to environmental factors (e.g. no lighting issues) and fine-grained temporal resolution. Previous studies, such as Larsen et al. (Reference Larsen, Norris and Cacioppo2003); Wolf et al. (Reference Wolf, Mass, Ingenbleek, Kiefer, Naber and Wiedemann2005), presented the possibilities of using fEMG analysis of the zygomaticus major (EMGZ) and corrugator supercilii muscles (EMGC) to understand the social cognitive emotional state. In addition, fEMG has been used to study empathy as a socio-cognitive status in the design process (Chang-Arana et al. Reference Chang-Arana, Surma-Aho, Hölttä-Otto and Sams2022). This was rounded off with an EOG (electrooculogram of vertical eye movement used to measure blinking) to better understand the relation of this social cognitive indicator on the interactive level. Both fEMG and EOG use surface electrodes to measure accurately the related behaviour, having a better precision than comparable methods such as computer recognition, while staying relatively easy to use compared to multimodal affective computing systems (Egger et al. Reference Egger, Ley and Hanke2019).

Figure 4.

Placement of electrodes for biosignal indicator measurements.

4.5. Synchrony rate of biosignals

To understand the relationships between social cognitive statuses in pairs using biosignals, a comprehensive amount of time-wise similarities between biosignals coming from each participant needs to be computed. However, as biosignals are physiologically unique for each person, direct comparison is difficult. To address this issue, a synchrony rate must be computed between the two datasets to see if there are similarities in their waveforms, considering the time delay that is inherent in any conversation between people. Following previous research done by Ehkirch et al. (Reference Ehkirch, Kakiuchi, Motomura, Matsumae and Matsumae2021), two methods were used to compute the synchrony rate of biosignals: the Damerau–Levenshtein Distance (Ld) and the dynamic time warping method (DTW). The Ld method was developed to answer the problem of direct comparison and the delay inherent to any study of synchrony of biosignals. From there, the DTW method was proposed to elaborate on the synchrony rate computed from the Ld method to integrate more subtle changes in the behaviours studied.

1. 1. Ld: As fEMG and EOG are by nature continuous voltage fluctuations related to the movement of specific muscles, or of the eye, there was a need to focus on the comparison time-wise of the activation of the behaviours as detected by it, corresponding to a peak in the data wave. However, to efficiently compare that data, it had to be simplified through binarisation to then correspond to the study of similarity between binarised character data. This was done using the Damerau–Levenshtein distance. The computed distance between two words corresponds to the minimum number of operations (consisting of insertions, deletions, substitutions of a single character, or transposition of two adjacent characters) required to change one word into another. The formal definition of a generic recursion found in Boytsov (Reference Boytsov2011) describes a method that gives a reliable approximation of the similarity between two waves of binarised character data while considering chronological misalignment. It was implemented using the Damerau–Levenshtein distance function of the jellyfish library (Turk Reference Turk2022). To be used, raw biosignal data must be converted into a binarised series, with “0” corresponding to no activation and “1” corresponding to activation of the studied behaviour, that is, activation of the corresponding muscle in the case of fEMG and blinking in the case of EOG. To be able to binarise the dataset, the neurokit2 library was used to detect activation of the muscle using the BioSPPy methodology for EMG, while the neurokit method was used to detect blinking from the EOG (Makowski et al. Reference Makowski, Pham, Lau, Brammer, Lespinasse, Pham, Schölzel and Chen2021). As Ld is a distance, it is affected by the total length of the dataset used to compute it. Thus, it was divided by this same length, which in this case was 15 seconds worth of data (15,000 data divisions for one biosignal for one examinee), to facilitate analysis by corresponding to the subdivision of the interactive level of the same moment.

2. 2. DTW: The Ld method requires processing of the data before it can be used, as some of the finer details may be lost if used as is. To study these more complex and subtle matches, the dynamic time-warping method (DTW) was implemented. DTW measures a distance-like quantity between two given sequences that vary by time to determine their similarities. As such, the DTW method is used to understand the synchrony of patterns inside the biosignals. Contrary to the Ld method, it does not concentrate only on the “peaks” but also gives an indication of the resting state of behaviours, considering downtime and tempo differences to compute the overall synchrony rate. The main algorithm of DTW can be found in (Müller Reference Müller2007). It is often used in partial shape-matching applications and presents an interesting aid to understanding the synchrony rate of two waveforms. It was implemented using the FastDTW library (Salvador & Chan Reference Salvador and Chan2007). As with Ld, DTW results were also divided into 15 seconds worth of data corresponding to the total length of the dataset used.

By using either the Ld or DTW method, it is possible to compute a numerical value representing the synchrony rate of given biosignals between two examinees in a pair. As both methods represent a distance, the longer they get, the more differences they indicate between the datasets, meaning more asynchrony. Conversely, smaller values correspond to a higher synchrony rate between examinee biosignals. Hence, either Ld or DTW results are inversely proportional to the amount of synchrony between the two participants for each given biosignal and the associated shared behaviour. See the following figure (Figure 5) for a simpler representation of the distance results.

Figure 5.

Relation between biosignals synchrony and distance results.

It is worth mentioning that, by design, these methods will produce non-parametric data, requiring further analysis with a Kruskal–Wallis test and completion with Pairwise Mann–Whitney tests. Following the recommendation of Ikäheimonen et al. (Reference Ikäheimonen, Li, Yao, Zuo, Aledavood and Hölttä-Otto2024) for a simplified reporting of the data-intensive research, we created a study workflow with steps and outcomes (see Figure 6).

Figure 6.

Study workflow steps and outcomes.