4.1. How the interaction is performed

During a canonical live coding performance, the user interacts in a similar way to how end-users type in a word processor software. One persona of Nick Collins, known as Click Nilson, used a cloud-basedFootnote ³ word processor to collaboratively re-write his presentation script with the audience during the live code festival symposium in Karlsruhe in 2013 (Nilson Reference Nilson2016). The main difference when a text editor is used alongside a programming environment for music-making is that the code evaluations are performed when a code chunk can be executed, which usually requires the conscious allocation of resources. Short edit-execution cycles, such as numerical parameter adjustments, are unlikely to enter conscious awareness, as this may take up to a few milliseconds. An average typing speed is 60–80 words per minute (wpm), or 3–4 characters per second (Marklin and Simoneau Reference Marklin and Simoneau2004). Short edits may be performed even faster than that, potentially approaching the upper limits of delayed auditory feedback, as the fastest typists can exceed several hundred wpm.

This strategy, which is predominantly found in full slateFootnote ⁴ live coding – an approach where pre-written code is used during a performance – is widely used by coders as a risk management technique or as a means of maintaining pace and flow (Roberts and Wakefield Reference Roberts, Wakefield, McLean and Dean2018). This full slate approach, a weak approach to live coding (Magnusson Reference Magnusson2014b), exploits pre-written code. Thus, it can be common that the performer is only editing the code instead of writing the performance script on-the-fly, which Magnusson refers to as the strong criterion. Identifying the limit between a weak approach to live coding and simply a generative reproduction of the material can be difficult, similar to how hard it is to draw the line between interpretation and improvisation (McPherson and Limb Reference McPherson, Limb, Rentfrow and Levitin2019).

It is generally accepted that typing on a keyboard is an activity based on serial actions. This has similarities with traditional music performance, as serial skilled actions are carried out during a musical performance (Palmer Reference Palmer1997). (In fact, sequence production spans a variety of daily activities such as speech and walking.) The main difference between live coding and traditional music performances is that the live coder does not make sound-producing bodily motions. This makes live coding a performance practice that can hinder engagement with audiences and co-performers related to embodied experience and the aesthetic enjoyment of the music (Brattico, Brattico and Jacobsen Reference Brattico, Brattico and Jacobsen2009). The main reason is that when humans observe sound-producing movements, they can mentally simulate these actions in their sensorimotor network (Keller and Appel, Reference Keller and Appel2010; Keller Reference Keller2012). Sometimes this can also produce overt bodily movement, such as dancing (Keller Reference Keller2008). The same does not apply to live coding, as the performed typing actions are mostly rendered in an unordered manner, or non-linearly, in terms of both temporal order and motoric sequential patterns. Commenting on the temporal aspects of live coding, Emmerson (Reference Emmerson2017: 115) states that ‘the code writing of deferred time computer programming may be assembled out of time order’. In terms of how motoric sequences unfold, this means that the same typing sequence can be produced in different ways. Thus, audience engagement via mental simulations of observed bodily movements cannot easily happen. Instead, the audiences and co-performers tend to bodily entrain through music listening and bodily movement. Maybe the only aspect we can mentally simulate is an expectation about a new code chunk evaluation by simply seeing the visual highlighting of the code. Hernani Villaseñor (Reference Villaseñor2019) has been performing live while filming his typing. This can be seen as a ‘virtual action-sound mapping’ (Jensenius Reference Jensenius2022: 155), denoting the multiple layers between action and sound, coupled with the discrete action of running the code. Live coders exploit this relation between code executions and resultant sounds. I usually do not look at the interpreter’s output following every code evaluation; instead, I expect to hear the sonic changes.

During a live coding performance, novel code evaluations can have entirely predictable or not-at-all-predictable musical outcomes. Typically, a novel musical outcome may induce surprise to both performer and audience, expressed as a violation of expectations and may arise from creative explorations of the generative space (Dahlstedt Reference Dahlstedt, McCormack and d’Inverno2012). Maybe the most common example of when our expectations are violated in a live coding performance is when a programming error occurs, and the generated sound patterns stop. The same can happen when we explore a generative music space, and the musical parameters suddenly tune into silence. This can make a live coding performance alien to an unfamiliar audience. The unfamiliar spectators may be ignorant of how the sound is produced, as reported in the documentary Show Us Your Screens (McCallum Reference McCallum2011). Maybe the audiences’ perception will change as live coding becomes more popular. Magnusson (Reference Magnusson2019) reports that audiences unfamiliar with programming can follow the sound-generation process by attending to the live writing process. Visual aspects such as code highlighting are particularly important to this end, and the progress of special-purpose text editors, such as Gibber (Roberts and Kuchera-Morin Reference Roberts and Kuchera-Morin2012) and Strudel (Ross and McLean Reference Roos and McLean2023) interfaces, has been tremendous in the last decade.

To summarise this subsection, canonical live coding involves typing on a keyboard. I have presented the activity of typing as serial skilled actions, which also applies to traditional music performance. However, whereas in traditional performance, the serial actions are tightly coupled in time and can give emergence to sensorimotor synchronisation, in live coding, the serial actions are out of time order. This suggests that using typing to interface with the programming language is not likely to result in the development of entrained bodily movements such as coordinated and rhythmic actions. Furthermore, it is unlikely that typing can induce mental simulations due to the multiplicities of individualistic typing styles, keyboard layouts and the error rate of open-loop motor programs. I can only speculate that exploiting typing gestures may induce mental simulations to co-performers and audiences on larger temporal chunks, where each micro-movement loses its significance and the gestalt of the typed chunks matters. But would that apply only to those exposed to live coding practice? Is familiarity important here? These are some open questions.

4.2. Pre-reflective processes in machine musicianship

The effect of action and perception on music cognition has been studied extensively. There is growing evidence that these two are inseparable from one another. We make our understanding by enacting our world (Varela et al. Reference Varela, Thompson and Rosch2017), and the role of the human body is of fundamental importance. Here, musical gestures are seen as non-verbal communication, which has pro-social functions and can enable ‘pre-reflective experiences’ (Leman Reference Leman2016: 92). For example, in electronic dance parties, a sweep from low to high frequencies may result in excitement, manifested in people raising their hands. How pre-reflective (or subconscious) processes are manifested in live coding has been barely discussed (Diapoulis and Zannos Reference Diapoulis and Zannos2014; Dahlstedt Reference Dahlstedt, McLean and Dean2018). This can be important as it will enable a discussion on how to better conduct live coding sessions and, consequently, how to better coordinate live coding ensembles.

I have already discussed that traditional music performance is tightly connected to bodily movement, but the movement is also related to expressive performance. Several mechanisms facilitate musicians’ engagement with expressive performance (e.g., planning and anticipating future actions), entrained processes (e.g., loosely coupled bodily movement) and reward mechanisms linked to emotional resonance and aesthetic enjoyment (Leman Reference Leman2016). As the embodied paradigm of music performance has gained more attention, entrainment is considered one of the fundamental mechanisms during music performance (Repp and Su Reference Repp and Su2013; Clayton et al. Reference Clayton, Jakubowski, Eerola, Keller, Camurri, Volpe and Alborno2020). Entrainment is a phenomenon that appears in several mechanical systems and may bring about stability in oscillatory systems, regardless of any small perturbations within a system. Recent studies suggest that sensorimotor synchronisation (SMS) and coordination are responsible for the emergence of entrained processes during a duet performance (Clayton et al. Reference Clayton, Jakubowski, Eerola, Keller, Camurri, Volpe and Alborno2020). SMS is a pre-reflective process roughly within 200–2000ms (ibid.), whereas coordination emerges at larger timescales of more than two seconds but may indicate coordination over several seconds to minutes. Godøy (Reference Godøy2021: 2) also identifies roughly the same time intervals (0.3–5s) as being of utmost importance during a performance, describing it as the ‘enigmatic relationships between notions of continuity and discontinuity in both philosophy and psychology’. Indeed, timescales below 200–300ms are mostly concerned with sound quality, including its early reflections within the acoustic surrounding. For longer durations, we mediate sound through our bodies and make music realisations. Over larger timescales, we coordinate, and social phenomena emerge as the outcome of complex interactions, such as a music performance in public.

These indicative timeframes would suggest that SMS is essentially impossible during canonical live coding. Sayer (Reference Sayer2015) supports this argument while Goldman (Reference Goldman2019) suggests that live coding can be seen as a propositional improvisation practice, which operates on long-term memory mechanisms. Goldman (Reference Goldman2019) and Sayer (Reference Sayer2015) suggest that live coding incorporates slow, conscious processes, making the emergence of fast, pre-reflective experiences impossible. Several reasons are offered for this, such as the content of feedback, the temporal nature of the feedback and the discrete nature of decision-making (Goldman Reference Goldman2019).

It is indeed true that the so-called canonical live coding incorporates mostly slow processes. Several systems, such as ixi-lang and TidalCycles (McLean and Wiggins Reference McLean and Wiggins2010b), have been developed, taking such perspectives into account. Indicatively, ixi-lang was developed with the constraint of writing and executing a code chunk in less than five seconds (Magnusson Reference Magnusson2011). Even this may not help in enabling fast, pre-reflective processes in a live coding session. So, the question is immanent: Can pre-reflective processes be enabled during live coding performance?

4.3. Summary on interaction, perception and cognition

I would argue that from a theoretical perspective, fast and entrained processes are unlikely to occur when using typing-based live coding systems. Entrained processes can afford small perturbations and eventually reach stability, but the multiplicities of typing a single line of code can be devastating to SMS. We can see bodily entrainment in live coders primarily as dancing to the beat, such as in the case of Algorave parties. But these are merely secondary aspects of musical gestures. It is unclear how these can influence the production of micro-movements. Maybe systems that require short typed sequences to make a sound (e.g., ixi-lang, TidalCycles), albeit being out of order and non-synchronised, can provide the best chance for the case of canonical live coding to employ pre-reflective processes, along with practices such as short edits. Orca,Footnote ⁵ a two-dimensional control interface inspired by trackers that affords single letter commands (for a description, see Blackwell, Cocker, Cox, McLean and Magnusson Reference Blackwell, Cocker, Cox, McLean and Magnusson2022: 147), operates on stream-based evaluations, which helps to reduce the delayed auditory feedback.

5.1. The case of canonical live coding

Canonical live coding is the most common approach. This includes the ‘standard’ interface of a computer screen and a keyboard. Whether gestural actions can be performed on a keyboard is a matter of debate; however, the current consensus is that typing on a keyboard is ‘neither observed nor significant’ (Jensenius et al. Reference Jensenius, Wanderley, Godøy, Leman, Godøy and Leman2010: 16). While this is undoubtedly the case in everyday interactions with computers, the same argument cannot hold in a live coding performance context. Typing is typically observed in this scenario as the performers share their screens. Furthermore, this becomes significant at certain times, specifically when code evaluations are performed and rendered as perceivable sound patterns. The sound vibrates molecular structures in the affine medium, that is, gas, liquid or solid. This transforms the meaning of typing into significant actions, as Baalman (Reference Baalman2009; Baalman Reference Baalmann.d.) demonstrate in the Code Live Code Live video (Table 1). Baalman switches on the onboard microphone of the laptop, and the typing sounds are used as raw material for the musical outcome. This performance setup demonstrates how typing on a keyboard can be both observed and significant.

Baalman (Reference Baalman2015) has reported that our programming language of preference can influence our motoric execution patterns. She elaborates more on embodiment during a performance and provides anecdotal evidence of typing automaticity. The argument is that certain typing tasks have been automatised to such an extent that they require minimum effort. These are known as body schemas, cognitive organisations of one’s body that can also monitor sequences of bodily motions and appear as learned motor patterns (Leman and Godøy Reference Leman, Godøy, Godøy and Leman2010). Whether gestural unfoldings, a term to indicate a weak temporalityFootnote ⁷ of bodily gestures, may influence our mental model of the running program is an open question (Diapoulis and Dahlstedt Reference Dahlstedt and Miranda2021a). To elaborate, can this weak temporality of typing influence how we internally represent code-sound relations and reason about the running program? Godøy (Reference Godøy2004: 58) has argued that we can mentally ‘compress’ bodily gestures in time using imagery, what he refers to as gestural imagery. The same cannot apply to sounds: we cannot mentally compress a sound and experience the same characteristics. Maybe gestural imagery can compensate on-the-fly due to our frustration with attending to every single moment of sound and actually influence our mental model (by encodings on our sensorimotor network). Baalman reports that our programming language of preference can shape our motoric actions so that certain sequences are encoded in our sensorimotor network as typing gestures that unfold in time when necessary. If the language of our preference can shape our motoric patterns, then can motoric patterns shape the language we use? Simply put, can the performer’s gestures influence the development of the programming language? I will elaborate on these two questions in the next section.

5.2. The case of bottom-up live coding

I argue that bottom-up systems exhibit interactivity variations where the gestures can influence the programming language. I will mainly focus on two cases: CodeKlavier CKalcuλator by Noriega and Veinberg (Reference Noriega and Veinberg2019) and the stateLogic machine (Diapoulis and Zannos Reference Diapoulis and Zannos2012). These were chosen because they share distinct characteristics of gestural interactions. These two cases, along with Al-jazzari and Betablocker by Griffiths (Reference Griffiths2007) and iMac Music by Reus (Reference Reus2012), belong to the category of bottom-up live coding systems.

CodeKlavier CKalcuλator (Table 1) is a performance system that recognises the musical patterns of the pianist-coder. The extracted MIDI patterns correspond to gestural sequences that the pianist performs to write lambda functions, a common construct in functional programming, on-the-fly. The recognition algorithm identifies repetitive melodic patterns and then instructs the lambda functions to perform the corresponding code evaluations. The system affords simple operations, such as addition and multiplication, and anecdotal evidence from the pianist-coder suggests that it is a highly challenging approach to live code, as the mind is split into two tasks simultaneously. The first task corresponds to serial skilled actions resulting in the musical outcome, and the second task is to perform conscious operations to write and modify a running program. Playing the piano indeed involves pre-reflective processes. Then, on the notational level of the generated code, the piano-coder has to allocate cognitive resources that may hinder expressivity during the performance. This approach showcases how musical gestures can be employed to write computer programs. In practice, the pianist plays repetitive melodic patterns and incorporates them into musical improvisation.

When developing the approach to live hardware coding (Diapoulis and Zannos Reference Diapoulis and Zannos2012, Reference Diapoulis and Zannos2014), the initial motivation was to make live coding somewhat more humane. At the time, as a student of material science, I naively imagined material crystal structures that could be live-coded based on a notation analogous to Miller indices, a notation system used in crystallography. This notation system offers a three-dimensional spatial representation of crystal structures in a binary-like notation. In this manner, elementary building blocks of material structures can be abstracted so that the engineer can imagine larger-scale crystal structures. As a result, what we literally see as a grain of salt can be reduced to a minimal expression, and from the minimal expression, we can deduce some of the material properties such as stiffness and conductance. The stateLogic machine (Table 1) is an experimental interface still in development (Diapoulis et al. Reference Diapoulis, Zannos, Tatar and Dahlstedt2022) and demonstrates the generation of a formal language from the bottom up. Initially, three input buttons were used, which depended on a clock-based paradigm. In this way, the user could apply stream-based updates within less than 0.5 seconds. These short-duration updates can enable pre-reflective processes. It is possible to automate certain pattern predictions without expending mental effort, such as looping into even and odd numbers. However, the clock-based interaction distorts the perceived immediacy of the interface, especially when the updates are larger than 100 milliseconds (Nash Reference Nash2012). In principle, Al-jazzari, Betablocker and iMac Music are also clock-based systems. Nevertheless, there are different embodied percepts when the clock rate is fast enough, transforming the interaction into phenomenally continuous percepts.

Technically speaking, these systems may not influence the language development process but they do affect the algorithmic structure of the written programs. We gesturally intervene to modify the algorithm’s very workings and there is clearly a minimal distance between the two. For instance, in Al-jazzari and Betablocker, the coder may experiment with multiple instruction sets or update them on-the-fly. Such dynamic updates on the level of an instruction set can account for developing a language on-the-fly. This is because all studied systems construct the running program by on-the-fly assemblages of low-level languages. In iMac Music, Reus (Reference Reus2012) is taking a radical stance and rewires the internal hardware components of a personal computer (Table 1). This approach goes one step further and addresses live coding by intervening in the wirings of the hardware. How would live coding look when we can hot-swap modular hardware components? What would previewing algorithms look like? Would the system re-arrange its constituent structure to make a new component? I find a large range of possibilities for human–material interactions (Ishii et al. Reference Ishii, Lakatos, Bonanni and Labrune2012) in live coding, and it is fascinating that we have not yet scratched the surface of this topic.

5.3. Mixed systems

All studied performances in the mixed category utilise direct manipulation with the mouse to some extent. The most diverse performance is that by The Duchess of Turing, where various systems are used, some of which are non-conventional, and one that can be regarded as a bottom-up system. Specifically, the TOPLAP app (Collins Reference Collins2015) uses machine listening algorithms to program an instruction set. I present this performance, called Using various, because of the broad variety of systems used. All other performances use a single programming system and direct manipulation. The performance by uiae and redFrik share many similarities, but the difference between textual and visual language is a major difference. Threnoscope by Magnusson presents an important contribution to notation systems while Approximate Programming by Kiefer presents a sophisticated algorithm for on-the-fly adjustments. I selected The Duchess of Turing performance to question whether we can actually form a mental model of the running program using listening and all other systems because of the diverse direct manipulation interactions, ranging from simple parameter adjustments to complicated tree algorithms.

What happens when the coder is using a mixture of different systems? How can the performer form a mental model of the running program(s) by simply listening to the musical outcome? These questions are well illustrated in a live coding performance by The Duchess of Turing (Table 1) for the fifteenth anniversary of TOPLAP. During the performance, the composer-programmer uses a variety of systems, such as PureData (PD), MAX/MSP, CSound and Scheme, and various systems developed by Nick Collins, such as BBCut, Autom8 and TOPLAP app. The live coder switches between systems to generate sound patterns, creating a mashup of live coding systems that resembles DJ-ing or bricolage of live coding systems (McLean and Wiggins Reference McLean and Wiggins2010a). The software systems are not interconnected; it is rather the musical outcome that connects everything together.

Another interpretation of this performance is that the language design and programming structures no longer matter when rendered to sound, as auditory perception can compensate and perceptually ‘bind’ everything together. Would that be somehow opposed to the view about the idiomaticity of low-level tools, as presented in McPherson and Tahıroğlu (Reference McPherson and Tahıroğlu2020)? Idiomaticity denotes that the design of a tool, regardless of how open-ended it is, carries design constraints that can distinctly shape the musical outcome. This discussion opens questions about agency. The notion of idiomaticity of programming languages looks aligned with a theoretical view of how influential agency progresses from all parties involved, to the spectator of the artwork by creating feedback networks (Dahlstedt Reference Dahlstedt and Miranda2021). The performance in question appears to suggest that when information is materialised, it loses its affiliation.

One engaging live coding performance by uiae, Using PD (Table 1), demonstrates how gestural interaction with the mouse modifies the system’s algorithmic structure. Interestingly, as the different PD objects move around the programming panel, they modify their connectivity, likely from the relative distance between nodes. Fredrik Olofsson (Table 1, superCollider) also demonstrated a similar approach, where direct manipulation using the mouse modified the routing of the audio busses. Another system that uses direct manipulation is the Threnoscope by Thor Magnusson (Reference Magnusson2014b), which uses a generative system combining graphic scores and text-based programming. Using Threnoscope (Table 1), the coder can continuously control the parameters of the generated drone sounds, using interactive visuals that display a descriptive notation shown as coloured rings.

Chris Kiefer (Reference Kiefer2015) presented an approach based on a gestural controller called Approximate Programming (Table 1). This is based on genetic algorithms and the exploration of parameter spaces for machine musicianship. Approximate programming is an interactive programming paradigm where the user controls the generation of different unit generators, using a multiparameter gestural controller. The major contribution of Kiefer’s work is the dynamic modification of algorithmic tree structures of interconnected unit generators. Such an approach to live coding demonstrates how gestural interactions can be applied to modify the algorithmic structure of a binary tree.

While it can be difficult to distinguish between algorithmic and parameter modifications, I would like to discuss this from a perspective that might clarify how I use the two terms. For a programmer, a variable is a powerful abstraction. In programming environments such as SuperCollider (McCartney Reference McCartney2002), a variable can be a generic box that holds different things. When a system is using variables as numerical parameters, such as a filter’s cutoff frequency, then most likely, the generated sound patterns are predictable. If we use a variable to apply simple mappings between a control parameter and a range of numerical values, then we do not exploit the affordances of a variable. Of course, in some cases, simple 1-to-1 mappings are necessary and desirable; for instance, we may want a single knob to control the tempo and nothing more than that. Still, I draw attention to this here to clarify that gestural interactions using the mouse or other input devices may correspond to either simple modifications of a numerical parameter or complicated algorithmic modifications such as structural rearrangements of binary trees (e.g., Kiefer Reference Kiefer2015).

This discussion on intervening with gestures on the algorithm itself is not related to gesture-sound mappings in the sense of running an algorithm to identify some gestural characteristics and render plausible sounds. I am interested in how we gesturally modify or re-program the algorithms on-the-fly. So, to avoid confusion with gesture-sound mappings, I here focus on gesture-algorithm mappings, which share some similarities with dynamic mapping strategies (Dahlstedt Reference Dahlstedt2009; Kiefer Reference Kiefer2015).

5.4. Summary of the observations

In this section, I have discussed how meaningful musical gestures can be produced using typing during canonical live coding. Inspired by Baalman’s typing automaticity, I question whether gestures can influence the programming language. The answer becomes apparent when I examine the bottom-up systems, where bodily gestures can operate on pre-reflective processes and influence the development of the programming language. An online performance that creates mashups from various programming languages may indicate the importance of sound, not the tool. It is similar to when we have a phase transition in a material (e.g., water crystallises to ice), but the process is not reversible anymore, meaning that when the code is rendered to sound, the process is no longer reversible. Maybe this performance suggests that when the tool is ignorant of its consequences to the environment, it loses its agency.

Several performance systems incorporating direct manipulation have been examined, and I underline the broad spectrum of interactions, from simple manipulations of numbers to sophisticated algorithms operating on binary trees. These systems differ from canonical systems as they offer the user an intuitive manner for interactions. In cases such as Approximate Programming, the live coder can modify the algorithmic structure on-the-fly, using continuous gestural interactions on multiparameter controllers. This also exhibits pre-reflective activations during a performance. On the other hand, such systems may not allow code changes to be monitored. When this happens, we lose the possibility of involuntary imageries due to notational audiation. Thus, there is a fine line between intuitive control and how to monitor informative notations. For instance, in Threnoscope, the live coder can monitor the descriptive notation as interactive visualisations of coloured rings and can continuously control the parametric changes. Thus, if the system’s temporal updates on the prescriptive notation are too fast, causing them to become uninformative, we can facilitate the use of descriptive notations instead (Magnusson Reference Magnusson2019).