Effort

Effort in its reserved, everyday use reflects our understanding of how hard a person must try in order to achieve an intended goal. In other words, it refers to the forcefulness or power of an action and it stresses the importance of intentionality and compulsion. Still, it seems to be a loosely defined term in different branches of science, such as physiology, kinesiology, biology, neuroscience, and psychology (Massin, Reference Massin2017; Richter & Right, Reference Richter and Wright2014). The reason for this lies in its perceptual and subjective character (Steele, Reference Steele2020) and in the compound nature of the goal (Dewey, Reference Dewey1897), consisting of both overt (physical) and covert (cognitive) aspects, which make effort difficult to grasp and quantify.Footnote ²

In psychology and neuroscience, mental (or cognitive) effort is a concept that represents the subjective sense of cognitive strain (Westbrook & Braver, Reference Westbrook and Braver2015) required for a given task and has been associated with mental imagery (Papadelis et al., Reference Papadelis, Kourtidou-Papadeli, Bamidis and Albani2007) and the level of attention (Bruya & Tang, Reference Bruya and Tang2018; Mulder, Reference Mulder, Hockey, Gaillard and Coles1986; Kahneman, Reference Kahneman1973). Mental effort is shaped in the form of peaks and decays or impulses and rebounds, which are also supposed to have emerged from our general capacity to move through approach and withdrawal (Stern, Reference Stern2010).

In the realms of dance, choreography, and kinesiology, effort is considered a subjective measure of expression used to describe qualities of movement performed with respect to inner intention. It is associated with the forces that cause and constrain movement—the active or passive attitude of a person in resisting or yielding to the physical conditions that influence a movement, that is, its dynamics—and at the same time it reflects the inner impulse or source of the movement—the inner attitude of power in terms of motivation and intentionality—that is harnessed by a person to accomplish an intended goal (Maletic, Reference Maletic1987; Bartenieff & Lewis, Reference Bartenieff and Lewis1980; Laban & Ullmann, Reference Laban and Ullmann1971). It “exposes the mover’s manner, tone and level of energy” (Hackney, Reference Hackney2002) and features recognizable “motion bell” patterns (Camurri et al., Reference Camurri, Lagerlöf and Volpe2003) of two consecutive distinct phases—tension/exertion and release/relaxation—that shape and phrase expressive physical action. Most notable in movement studies has been Laban’s movement analysis, a system that Laban developed for categorizing movement in the context of Western modern dance choreography (Laban & Lawrence, Reference Laban and Lawrence1974).

In the acoustic world, musical effort has been described as “the element of energy and desire, of attraction and repulsion in the movement of music” (Ryan, Reference Ryan1991: p. 6). It has been reported that it reflects the musical tension of a piece (Cox, Reference Cox2016; Krefeld & Waisvisz, Reference Krefeld and Waisvisz1990), which is manifested as patterns of musical intensification and abatement (Kurth, Reference Kurth1922) giving rise to emotional responses (Lerdahl & Krumhansl, Reference Lerdahl and Krumhansl2007). The notion of effort has been appreciated as a basic feature of expressive power in EMIs (Wanderley & Battier, Reference Wanderley and Battier2000; Ryan, Reference Ryan1991) and is considered as essential to both performers and audiences alike (Olsen and Dean, Reference Olsen and Dean2016). Performers need to suffer and even sweat a bit in order to bring this tension to the fore and audiences need to perceive effort in order to recognize particularly intense musical passages played by the musician (Vertegaal et al. Reference Vertegaal, Kieslinger and Ungvary1996); in fact, much of the excitement in audiences is often driven by levels of virtuosity and effort observed in performers. Thus, effort in music can be conveyed and induced by a performer, as well as assessed and experienced by an observer. It can be visible in human movement, audible in the sound, and sensed through proprioception.

While in movement action is directed toward a designated pragmatic goal (Rosenbaum et al., Reference Rosenbaum2009), in music the goal is primarily musical and refers to salient moments of musical expression that are regulated or accompanied by music-related gestures. Hence, effort in music can combine bodily and melodic movement (overt), mental impulse, and musical intentionality (covert). Nevertheless, none of the descriptions or definitions has captured the term in its entirety and none has acknowledged sound as part of both its overt (auditory) and covert (imagery) aspects. Therefore, a working definition of effort in music—one designed specifically for the purpose of this study—might combine and extend previous definitions to not only include movement but also the auditory modality:

In MIIOs, effort can be imagined and experienced by the performer and perceived (visually and auditorily) by an observer. Dhrupad vocalists seem to manipulate notes by grasping, holding, extending, restricting, and releasing an object, which signifies successive intensification and relaxation phases. Thus, effort is expressed through the subjective quality of balance between, on the one hand, the imagined resistive force imposed by the object that is employed by the performer and, on the other hand, the force that the performer exerts to defy it. It is a measure for the power of this interaction and can be potentially regarded as a cross-modal (or amodal) descriptor that combines all implicated modalities (sound, vision, movement, mental imagery). This highlights the originality of the multimodal feature fusion method in the regression analysis—described in detail in the Methodology—to efficiently combine movement and acoustic features into a common effort representation. It also justifies the sequential mixed methodology followed, which combines interviews, audio-visual material, and movement data.

The assessment of effort relies commonly on the subjective experience of how difficult the compound task seems to be for the person actively involved or observing, and therefore effort (in music too) is a perceptual measure that does not offer straightforward ways to quantify it. This might explain why analytic works showing a clear focus on the concept of effort with this specific understanding are limited and why there is a shortage of successful quantitative methods and measures, especially those looking at music performance.

For instance, tension has been studied extensively in music (melodic: Granot & Eitan, Reference Granot and Eitan2011; Bigand & Parncutt, Reference Bigand and Parncutt1999, vocal: McKenna & Stepp, Reference McKenna and Stepp2018), but it is a complex phenomenon to quantify and its validity as an indirect measure of effort is not well-justified in literature. Laban’s effort ideas have proven valuable for developing computational methods and have laid the foundations for the majority of work on the analysis (Glowinski et al., Reference Glowinski, Coll, Baron, Sanchez, Schaerlaeken and Grandjean2017; Fdili Alaoui et al., Reference Fdili Alaoui, Françoise, Schiphorst, Studd and Bevilacqua2017; Broughton & Davidson, Reference Broughton and Davidson2014; Maes et al., Reference Maes, Dyck, Lesaffre, Leman and Kroonenberg2014; Camurri et al., Reference Camurri, Canepa, Ghisio and Volpe2009; Petersen, Reference Petersen2008) and synthesis (Volpe, Reference Volpe2003; Zhao, Reference Zhao2001) of movement in music, as well as the development of interactive systems for music and dance (Souza & Freire, Reference Souza and Freire2017; Françoise et al., Reference Françoise, Fdili Alaoui, Schiphorst and Bevilacqua2014; Maes et al., Reference Maes, Leman, Lesaffre, Demey and Moelants2010; Bennett et al., Reference Bennett, Ward, O’Modhrain and Rebelo2007). However, his ideas have limited potential for developing explicit quantitative measures, and in the abovementioned computational methods Laban’s effort qualities are only indirectly estimated by extracting features that are usually evaluated as relevant to the concept of effort by observers.

It can be tempting to approach effort through force-related measures borrowed from physics and mechanics, such as “weight” (not necessarily in the vertical direction), “power,” or “pressure” (Moore & Yamamoto, Reference Moore and Yamamoto2012), as they reflect the action’s power or the ease and struggle in performing an action. “Weight” has been indirectly computed by extracting measures for acceleration (Niewiadomski et al., Reference Niewiadomski, Mancini, Piana, Rojc and Campbell2013; Kapadia et al., Reference Kapadia, Chiang, Thomas, Badler and Kider2013; Samadani et al., Reference Samadani, Burton, Gorbet and Kulie2013; Castellano & Mancini, Reference Castellano and Maneini2009; Caridakis et al., Reference Caridakis, Raouzaiou, Bevaequa, Maneini, Karpouzis, Malatesta and Pelachaud2007; Bernhardt & Robinson, Reference Bernhardt and Robinson(s. d.); Hartmann et al., Reference Hartmann, Mancini, Buisine and Pelachaud2005; Volpe, Reference Volpe2003; Camurri et al., Reference Camurri, Lagerlöf and Volpe2003) and kinetic energy (Piana et al., Reference Piana, Maneini, Camurri, Varni and Volpe2013; Samadani et al., Reference Samadani, Burton, Gorbet and Kulie2013; Hachimura et al., Reference Hachimura, Takashina and Yoshimura2005; Nakata, Reference Nakata, Mori and Sato2002) or “Quantity of Motion” (Mazzarino et al., Reference Mazzarino, Peinado, Boulie, Volpe and Wanderley2009) of the whole body or different body parts (other indirect measures have been also probed, for example, Mentis & Johansson, Reference Mentis and Johansson2013; Zhao & Badler, Reference Zhao and Badler2005; Morita et al., Reference Morita, Nagai and Moritsu2013; Alaoui et al., Reference Alaoui, Caramiaux, Serrano and Bevilacqua2012).

However, there is a lack of general agreement on how to compute “weight” (Niewiadomski et al., Reference Niewiadomski, Mancini, Piana, Rojc and Campbell2013), which has even led to avoiding it entirely (Santos, Reference dos Santos2013; Camurri and Trocca, Reference Camurri, Trocca, Wanderley and Battier2000). First, it is worth emphasizing that “weight” should not be assessed through measures that include kinetic properties (kinetic energy, acceleration, and velocity)—in other words measures that reflect the impact of one’s movement (Chi et al., Reference Chi, Costa, Zhao and Badler2000). These would fail, for instance, to represent effort exerted in trying to move an object fixed in space (such as a wall) or in holding a heavy load for a long period of time without moving. More importantly, studies approaching effort through force-related measures do not acknowledge its compound character (comprising both physical and mental aspects) and additionally presuppose that the sense of an action’s power can be somehow objectively assessed and quantified. Since this is not necessarily the case, this assumption needs to be further examined. Considering that different people have a different capacity (the intrinsic factor related to a person’s level of fitness or ability) in achieving the same physically and/or mentally demanding goal (the extrinsic factor related to the task’s difficulty level), effort constitutes a perceptual concept, related to the subjective sense of how intense the specified task is for the particular subject.

A few approaches exist that rely on the physiologically measurable quantities that are supposedly the prevailing indicators of effort in music, such as muscular tension, breathing rate, biochemical energy expended in form of heat (calories), sweating, pupillary dilation, and brain activity (Jagiello et al., Reference Jagiello, Pomper, Yoneya, Zhao and Chait2019; O’Shea & Moran, Reference O’Shea and Moran2018; Ward et al., Reference Ward, Ortiz, Bernardo and Tanaka2016; Tanaka, Reference Tanaka2015; Gibet, Reference Gibet, Leman and Godøy2010; Marcora, Reference Marcora2009; Enoka & Stuart, Reference Enoka and Stuart1992; Kahneman, Reference Kahneman and Beatty1967). However, these are absolute measures that do not necessarily account for the capacity of a person in executing the task and may not reflect the subjective sensation of effort, therefore they do not hold wide consensus. More systematic work needs to be done in this direction. For instance, the absolute measure of force may be low, but due to low muscle power or fatigue, burned calories may exceed the resting burn rate (measured through metabolic equivalents). Additionally, the compound character of effort (comprising both physical and mental aspects) would require capturing a combination of physiological measures from different modalities, which is usually not the case. The shortage of successful quantitative approaches underlines the original contribution of this study in combining qualitative with quantitative methods.

Effort in EMIs

Archetypical interactions with objects of the real world that we are familiar with may offer an easily accessible interaction metaphor for both novice and expert music performers of novel empty-handed EMIs (Bevilacqua et al., Reference Bevilacqua, Schnell, Françoise, Boyer, Schwarz, Caramiaux, Lesaffre, Maes and Leman2017; Ward, Reference Ward2013; Essl & O’Modhrain, Reference Essl and O’Modhrain2006; Fels et al., Reference Fels, Gadd and Mulder2002). However, most digital-music interfaces do not offer “ergotic” gesture-sound couplings (Castagné et al., Reference Castagné, Cadoz, Florens and Luciani2004); although the automatic tendency for effort minimization has been evidenced in multiple fields (Cheval & Boisgontier, Reference Cheval and Boisgontier2021) and while effortless movements may be perfectly suited for word processor design, they may have no comparable utility in the design of musical instruments (Ryan, Reference Ryan1991: p. 3). There is extensive evidence that effort can add value and therefore be opted against actions that are effortless (Inzlicht et al., Reference Inzlicht, Shenhav and Olivola2018). Together effort and sound reinforce one another, and their strong link increases expressive power and the likelihood of the EMI to last over time (Fels et al., Reference Fels, Gadd and Mulder2002). For instance, Fels et al. (ibid.) have reported that while claying was not considered a compelling metaphor for shape manipulation due to the absence of tactile feedback, the stretching metaphor proved to compensate for this absence by forcing the player’s frame of reference to remain attached to the object.

Therefore, there is a number of indicative approaches (Battey et al., Reference Battey, Giannoukakis and Picinali2015) that consider how haptic- (especially force-) feedback interfaces could be used as a means for improving the control accuracy of synthetic sound, for example, pitch-selection accuracy on a continuous pitch controller (Berdahl et al., Reference Berdahl, Niemeyer and Smith2009), which is especially crucial when many degrees of freedom need to be managed (Wisneski & Hammond, Reference Wisneski and Hammond1998). In empty-handed interactions, as in the case of MIIOs, it may be possible to compensate for the absence of tactile feedback by encouraging users to imitate the exertion of effort against forces that are elicited by physically inspired images of interactions with real objects and restricting patterns of sonic qualities to only the ones that the implicated materials can afford. By using gesture-sound mappings that conform to the amount of required effort elicited through images of interactions, such as those of MIIOs in Dhrupad singing, the intention is to render the control of artificial sounds more natural, intuitive, and “physically plausible” (Castagné & Cadoz, Reference Castagné and Cadoz2005) and contribute to enhancing EMIs in terms of controllability, expressivity, and virtuosity (O’Modhrain & Gillespie, Reference O’Modhrain, Gillespie, Papetti and Saitis2018; Yu & Bowman, Reference Yu and Bowman2018; Tanaka, Reference Tanaka2015; Ward, Reference Ward2013; Essl & O’Modhrain, Reference Essl and O’Modhrain2006).

Sensations of elasticity, weight, and other physical metaphors in musical thought are not exclusive to Hindustani music, and listeners and performers in other musical genres also report that they experience virtual worlds of forces and motion in relation to music and sound (Fatone et al., Reference Fatone, Clayton, Leante, Rahaim, Gritten and King2011; Eitan & Granot, Reference Eitan and Granot2006). From an engineering point of view, Mion et al. (Reference Mion, de Poli and Rapanà2010) have in fact modeled gesture-sound links in the context of instrumental gestures based on the concept of resistance or impedance. However, Dhrupad vocal music constitutes a distinctive case in this sense for reasons which are explained next. A short introduction to Dhrupad and MIIOs is first given.

Introduction to Dhrupad

Dhrupad (Widdess & Sanyal, Reference Widdess and Sanyal2004) is one of the two predominant styles of Hindustani music. It is a monophonic (primarily) vocal music tradition, and it relies heavily on improvisation, called ālāp, which is not free but strictly rule based and conforms to the so-called rāga system (a form of melodic mode lying between scale and tune; Powers & Widdess, Reference Powers and Widdess2001). The tonic is defined by the main performer’s most comfortable pitch and all other pitches are tuned relative to this. The improvisation starts at a strikingly low pace in the vocalist’s lowest pitch range; it has a very slow melodic development, which builds up only gradually in pace, pitch, and melodic tension as the melody ascends over about 2.5 octaves toward the climax (typically the 3^rd degree of the highest octave) and it is sung without apparent rhythm (non-metered) to a repertoire of non-lexical syllables (e.g., “ra,” “na,” and “num”). Melodic tension is periodically released by longer stops on the tonic and shorter stops on the 5^th and is finally resolved when the climax is reached.

MIIOs in Dhrupad

Hindustani singers seem to engage with the melodic content in two main ways (Rahaim, Reference Rahaim2009), which are supposed to reflect two different modes of body–voice relationship: the open handed and the closed handed. In the closed-handed mode, which starts with forming a grip with the hands as in grasping an object, singers look like they are stretching or compressing an elastic material, pulling or pushing away a heavy object, throwing or bouncing something like a ball on the floor, and executing other powerful movements in changing the objects’ shape, size, and position, all of which comprise gripping, intensification, and releasing phases.

In contrast to the open-handed mode (ibid.), in which the hands seem to trace curves and trajectories effortlessly in space, the powerful movements of the closed-handed mode do not seem to offer a simple melographic representation of the sound (spatial representation of pitch height) and do not carry any symbolic meaning, but rather they appear to indicate proprioceptive sensations of resistance that singers employ in manipulating notes as smooth pitch glides; Figure 1 illustrates such an example. There is an observable match between the voice and the manipulative gestures in terms of synchronization and temporal congruency of various features, mostly melody and melodic tension, dynamics, and timbre and—despite the absence of a real object—the link to the sound seems to be mediated by the imagined material.

Figure 1. Zia Fariduddin Dagar in concert, 16.06.2007 (Mana, Reference Mana2007).

From the point of view of enactive theories and ecological psychology, this match attests to the importance of sensorimotor skills or patterns of robust movement-sound contingencies (“know-how”) that a person develops over time by manipulating real objects of the environment (Freed, Reference Freed1990; Warren & Verbrugge, Reference Warren and Verbrugge1984; Gibson, Reference Gibson, Shaw and Bransford1977). These patterns underpin much of our conception of any sound (Maes et al., Reference Maes, Dyck, Lesaffre, Leman and Kroonenberg2014, Küssner, Reference Küssner2014; Cox, Reference Cox2011; Zbikowski, Reference Zbikowski2002), thus we attend any sonic event—be it real or virtual (Clarke, Reference Clarke2005)—with certain expectations (Huron, Reference Huron2006) through an incessant process in the imagistic domain of mentally simulating features of sound or actions that may have produced it (Cox, Reference Cox2011; Lahav et al., Reference Lahav, Saltzman and Schlaug2007). As it has been suggested that we are able to grasp objects figuratively because we are able to grasp them literally (Küssner, Reference Küssner2014), it would not be a crude assumption to also think of an imagined object in MIIOs as a carrier of certain universal patterns, general rules, and opportunities for behaving (Camurri et al., Reference Camurri, De Poli, Leman and Volpe2001) that are defined by the imagined resistive force according to size, shape, material, and so forth and that also afford specific types of sonic results (Krueger, Reference Krueger2014; Tanaka et al., Reference Tanaka, Altavilla and Spowage2012; Menin & Schiavio, Reference Menin and Schiavio2012; Reybrouck, Reference Reybrouck2012; Godøy, Reference Godøy, Sales Dias, Gibet, Wanderley and Bastos2009; Nussbaum, Reference Nussbaum2007; Jensenius, Reference Jensenius2007; Clarke, Reference Clarke2005). However, the link between hand gestures and voice in MIIOs has not been studied before in a systematic way in respect to the concept of effort.

Dhrupad as Case Study

Singing offers a unique opportunity to study profound gesture-sound links that are driven by the musicians’ mental constructions and musical intentions, as—in contrast to instrumental playing—the hands are not bound by the mechanical constraints posed by a particular instrument. In fact, it is quite surprising that singing gestures have drawn less attention than instrumental ones (Pearson Reference Pearson2016; Clayton & Leante, Reference Clayton, Leante, Clayton, Dueck and Leante2013; Fatone et al. Reference Fatone, Clayton, Leante, Rahaim, Gritten and King2011; Rahaim, Reference Rahaim2009; Moran, Reference Moran2007). Even more, the spontaneous and easily identifiable gestural demonstration of physically inspired concepts by MIIOs is not common in the performance practice of music genres other than Dhrupad, making it unique in allowing an ecologically valid methodological approach in recording performances without any explicit instructions like those required in designed experiments. It also imposes constraints upon musicians’ movements that reduce the otherwise high complexity of full-body motion and the necessity for the use of artificially induced exogenous constraints or dimensionality reduction methods (Nymoen et al., Reference Nymoen, Godøy, Jensenius and Torresen2013).

The conception of melody as a continuous pitch “space”Footnote ³ (Fatone et al., Reference Fatone, Clayton, Leante, Rahaim, Gritten and King2011), in which Dhrupad vocalists often approach discrete notes through smooth trajectories rather than simply scale steps (Battey, Reference Battey2004), makes Dhrupad particularly suitable for studying links to the non-discrete nature of movement and for enhancing mapping strategies of EMIs in which performers’ gestures serve the continuous control—monitoring and adjustment—of patterns generated by computer algorithms rather than the triggering of individual events. The precise intonation of Dhrupad musicians and the slow rendering of melodic phrases in the opening section of the ālāp improvisation make it ideal for studying MIIOs, as for slow musical stimuli there is a tendency to ascribe motion to the impact of an outside force on an imagined character (Eitan & Granot, Reference Eitan and Granot2006). The lack of meter and meaningful lyrics is suited to focusing on melodic factors and the expressive rather than semantic content imprinted on gestures. Additionally, Dhrupad offers a rigid framework for the study of melodic tension, in terms of both specified ālāp improvisation-relevant macro-structure (gradual intensification from lower pitches toward the climax) and rāga mode-specific micro-structure (locally raised tension for specific melodic movements according to the rāga “topography” (Rahaim, Reference Rahaim2009)).

As Dhrupad is an oral music tradition, knowledge is not transmitted through music notation but through direct demonstration and imitation, which includes not just sound but also movement. The resulting gestural resemblance that is evident between teacher and students (Rahaim, Reference Rahaim2009) justifies the deliberate choice of collecting material from a single music lineage and permits examining similarities (inherited bodily disposition) versus differences (idiosyncrasy) in the gesturing habits of musicians who share the same teacher. Finally, interviewees’ frequent recourse to motor-based metaphors allows a mixed—thus richer—methodological approach to be followed, one combining measurements with interviews (Leante, Reference Leante2009).

Computational approaches of cross-modal relationships for singing movements have been few in Western types of music (Brunkan, Reference Brunkan2015; Pfordresher et al., Reference Pfordresher, Halpern and Greenspon2015; Luck & Toiviainen, Reference Luck and Toiviainen2008) and even fewer in Indian music (Pearson, Reference Pearson2016; Moran, Reference Moran2007). Even more, none of the work has focused solely on MIIOs, and specifically in Dhrupad, none has applied quantitative methods through such an unusually strong ecological validity in the capture of data in the field, including 3D-movement, and, most importantly, none has acknowledged the role of effort as central to the subject of music performance and approached it in a systematic way. Although regression has been previously used in gesture-sound studies (e.g., Caramiaux et al., Reference Caramiaux, Bevilacqua, Schnell, Kopp and Wachsmuth2009), to the best of our knowledge this is the first time that the compound—and therefore complex—character of effort has been considered by combining both movement and acoustics features in each single model for inferring and predicting effort (with preliminary results first presented in Paschalidou et al., Reference Paschalidou, Eerola and Clayton2016). Despite the specificity of the genre that is being used here as a case study, this study aims to address concerns that are of interest to the wider research community and extend outcomes to other music-making practices and new EMIs in specific.

Methodology

Due to the explorative character of the work, an empirical sequential mixed methodology was followed, which relies on first- and third-person perspectives for gesture analysis in music (Leman, Reference Leman, Leman and Godøy2010) and combines qualitative ethnographic methods (thematic analysis of interviews and video observation analysis of vocal improvisation performances) with quantitative methods (regression analysis of the same performances) on original recordings of interviews, audio-visual material, and 3D-movement data. The qualitative part of the work preceded, informed, and guided the quantitative methods that followed. The entire methodology is briefly outlined here and illustrated in Figure 3; however, the current paper reports exclusively on results drawn from the quantitative methods used. The reader can refer to Paschalidou & Clayton (Reference Paschalidou and Clayton2015) for a brief outline of the qualitative analysis, which clearly revealed a significant visual element in the conceptualization of music and a frequent use of sensorial descriptors and metaphors of interactions with objects.

Figure 3. The sequential mixed methodology that was followed. Gesture images taken from Mulder (Reference Mulder1998).

For the interviews, thematic analysis was applied to motor-based metaphors and physically inspired linguistic descriptors of 8 performers, which were coded and organized in meaningful ways in classes of overarching themes into a final table intended to inform the coding scheme of the video analysis that followed. For the non-participant observation analysis, the audio-visual material of the vocal improvisation by four performers was manually annotated by a single observer, who visually identified, segmented, and labeled ΜΙΙΟ events and coded them for a number of melodic and movement aspects, most importantly:

1. (a) effort levels (numerical) that each gesture was perceived by the observer to require, in an integer scale of 0–10 (10 being the highest) and

2. (b) MIIO classes, such as stretching, compressing, pulling, pushing-away, collecting, and throwing (categorical descriptors).

These manual annotations were cross-validated by two choreographers by conducting an inter-coder agreement test and were then used as the “true” response values in fitting simple, interpretable linear models (LMs) with a reasonable accuracy to a small number of combined movement, and audio features extracted from the raw data of two vocal improvisation performances (which were selected from the entire dataset based on the high number of identified MIIO events). Specifically, regression analysis aimed to validate, augment, and formalize findings from the qualitative methods by avoiding manual labor and inferring annotations computationally with the following intention:

1. (1) to test the null hypothesis that physical effort and MIIO types are unrelated to the melodic context against the alternative hypothesis that such a relation does exist and

2. (2) to devise formalized descriptions by probing various sets of statistically significant movement and acoustic features for two different tasks:

3. (a) to infer effort levels

4. (b) to classify MIIOs.

   Different models were devised for each task, (i) those that best fit each individual performer (thus reflect mostly idiosyncratic elements of each individual performer) and (ii) those that—despite their lower goodness of fit—overlapped to a greater extent across performers in terms of number and type of features (thus describe more generic gesture-sound links).

There are several challenges in applying traditional statistics by fitting linear regression models to time-varying data as the ones examined in this paper: data not following a normal distribution, limited dataset not allowing the separation between training and testing, issues of data temporal dependency (Schubert, Reference Schubert2001), and non-linear elements that are not captured by the fitted LMs (Stergiou, Reference Stergiou and Decker2011). However, linear regression offers the unique advantage of illustrating a rather complex phenomenon of the real world in a compact and easily interpretable way through a model that allows one to verbally describe a relationship in simple terms. The first part of this work seeks to gain a deeper understanding of the Dhrupad performance practice on the occasions of MIIOs based on the relatively limited size of the originally recorded dataset; therefore, regression analysis was considered as a suited first step and was opted for this study in proposing enhancements to EMI gesture-sound mappings.

Response Values

For inferring effort, the full range of integer values assigned by the annotators was used for the models. However, for the classification task the five fine MIIO classes by which the material was originally annotated were reduced through simple observation of the video footage to only two coarse categories, specifically discriminating between interactions with (a) elastic versus (b) rigid objects. This reduction was a requirement for running multinomial (logistic) regression models (response classes > 2) with a small sample size (64 for vocalist Hussain and 102 for Sahu) and working with skewed samples of real performances rather than designed experiments. Interaction with elastic objects refers to the deformation of a material with elastic properties by extending or compressing it from its equilibrium (as in stretching/compressing), while interaction with rigid objects refers to the spatial translation of a non-malleable body of some considerable mass (as in pulling/pushing away/collecting/throwing). Visual discrimination by the annotator between these two classes was led by the dynamic profile of the gesture events and the existence or not of a subsequent recoil phase toward the equilibrium. Ambiguous classes were excluded.

Table 1 summarizes all relevant information about the data that were used for each of the two performances and illustrates the correspondence between fine and coarse gesture classes.

Table 1. Data overview for vocalists Afzal Hussain and Lakhan Lal Sahu

Feature extraction

Raw mocap data were exported into c3d format and were processed by:

1. 1. Filling gaps that are caused by occasional optical occlusion of reflective markers, through linear interpolation between adjacent points in time and

2. 2. Smoothing to eliminate noise introduced by arbitrary reflections in space and occasional confusion in identifying the rigid bodies, through a low-pass Savitzky–Golay FIR filter of 9^th order with a window of 90 msec for position and velocity data, and 130 msec for acceleration data.

Representative statistical global measures (such as mean, standard deviation (SD), minimum (min), maximum (max)) were computed from time-varying movement and audio features (Matlab 2015: Mocap Toolbox v. 1.5/MIR Toolbox v. 1.6.1, Praat v. 6.0Footnote ⁴) and were normalized prior to their use in the models. Due to the endless possibilities in the quest for the best combination of features, this work started from a basic set of features that has been previously proposed in a different experimental context (Nymoen et al., Reference Nymoen, Godøy, Jensenius and Torresen2013), as basic descriptors have proven robust in sound tracing experiments (ibid.) and violin bow strokes (Rasamimanana et al., Reference Rasamimanana, Flety and Bevilacqua2006). This was then progressively enriched with alternative or novel features considered more relevant to the specific study and performance context and expected to raise the explained variance in the estimated responses.

For instance, to account for the different ways by which Dhrupad singers seem to display similar qualities of movement in a real performance setting without any instructions, hand distance was computed according to handedness (uni- or bi-manual), that is, the active hand(s) participating in the movement (rather than simply hand distance as in the basic feature set), and the point of reference to which gestures appear to be performed (according to gesture type this can be the position of the other hand, the torso, or the starting point of the movement and features are designated by the suffix “_accord”), which were both drawn from the manual video annotations. Additionally, it made more sense to compute position features in relation to the centroid of the torso (point 3 in Figure 3) rather than the ground, which was used in the basic feature set, because not only gravity but other forces are imagined acting too and Dhrupad singers often conceptualize pitch as a movement in relation to their own body.

Pitch-related features for the three most typical melodic movements in Dhrupad (ascending, descending, double-sloped) were computed in three alternatives: (a) the values of critical pitches (minPre-max-minPost) as shown in Figure 4, (b) the coefficients of polynomial expressions (of 2^nd–5^th degree) that were fit to pitch data and forced to pass through the critical melodic points, and (c) the asymmetry in strength and rate of pitch ascent versus descent (calculated in a similar way to the sum of signed vertical kinetic energy of a unit mass in the basic set; Nymoen et al., Reference Nymoen, Godøy, Jensenius and Torresen2013). Critical pitches were extracted in three different scales: linear, absolute logarithmic, and relative logarithmic (designated by the suffix “_lin,” “_abslog,” “_rellog”). Absolute logarithmic values refer to the absolute pitch height and thus they better reflect the mechanics of voice production, while relative logarithmic values refer to degrees of the scale in relation to the tonic of each octave and thus they are assumed to better reflect the melodic organization that is specific to the rāga mode:

• - linear: $ f $ in Hz,

• - absolute logarithmic: $ 100\bullet \log \left(\frac{f}{f_{ref- abs}}\right) $ ,

• - relative logarithmic: $ 100\bullet \log \left(\frac{f}{f_{ref- rel}}\right), $

where f_ref-abs = 440 Hz and f_ref-rel = the tonic of the specific octave in which the melodic movement takes place (drawn from manual annotations).

Figure 4. Critical pitches defined by three values (minPre-max-minPost) for double-sloped melodic glides.

Determining Best Models

The best models were determined (R v. 3.2.3/R-studio v. 0.99.491) by trial-and-error in finding the best trade-off between accuracy of model fit, compactness (small number of independent variables), and simplicity in interpretation and feature extraction. Apart from the best-fitting models to a single performer, slightly less successful but largely overlapping cross-performer models were also devised, representing more generic cross-modal relationships. Whether these models can be potentially generalized to a wider population is currently unknown; however, they are useful in indicating trends that can be further explored in larger datasets. As methods were mostly intended for inference (and less for prediction) and the dataset that followed from the use of uninstructed real performances rather than designed experiments was limited, the entire dataset was used for fitting models (no separate training data were used for prediction).

The goodness of fit, according to which the best models were selected, was assessed based on the adjusted R-squared (R²adj) coefficient of determination (James et al., Reference James, Witten, Hastie and Tibshirani2013: 212) for the LMs in inferring effort and on the area under the curve (AUC) of the receiver operating characteristic curve (ibid.: 147) for the general logistic models (GLM) in classifying MIIOs. The strength and the direction of the correlation of each individual feature to the response were rated based on the absolute value and the sign of each individual coefficient and the probability (p) value was used as an indication of likelihood in observing an association between feature and response due to pure chance in the absence of any real association. A significance level α of 0.05 (Moore & McCabe, Reference Moore and McCabe2006) was used in comparison to the p-value for examining the null hypothesis.Footnote ⁵