2.1. Live coding as improvisation

Live coding performances tend to be characterised by an improvisational approach, in which performers embrace the ‘from-scratch’ challenge, structuring and developing their pieces entirely in real-time. To advance a definition of this approach, Magnusson has used the lemma ‘strong coding’, as opposed to ‘weak coding’ (where the code is written in advance and simply executed or slightly modified during the performance) (Magnusson Reference Magnusson2014a). In a strong coding performance, coders start with a blank page; additionally, the code is usually not saved at the end of the performance (Magnusson Reference Magnusson2015). As such, live coding performances are focused on momentum and tend to have a scarce interest in reproducing previous performances. Indeed, Magnusson suggested that the live coding practice shares similarities with oral tradition as it tends to be primarily extemporaneous (Magnusson Reference Magnusson2016). Parkinson and Bell (Reference Parkinson and Bell2015) discussed the lack of pre-packaged structures in live coding improvisation and proposed an analogy with free improvisation as discussed by Derek Bailey (Reference Bailey1975) in the context of his musical practice. With this term, Bailey indicated the free exploration of the gamut of instrumental sonorities during an open improvisation. Such a form of exploration also tends to occur in live coding, when the performer explores the possibilities of a specific live coding system.

Overall, the practice of live coding is characterised by specific actions that define a high level of difficulty in this type of performance. First, musical ideas cannot be expressed immediately, as the performer needs to formalise them in the abstract form of code: idea-to-code latency (McLean and Wiggins Reference McLean and Wiggins2009). Indeed, live coders make improvisational choices based on a need for the ‘now’, but also need to elaborate on them in the (near) future (ibid.). Second, live coding improvisations tend to be error-prone due to typos while writing code, and such risk increases with the complexity of the code (Blackwell and Collins Reference Blackwell and Collins2005). Finally, two main interaction feedback loops (manipulation feedback and performance feedback) coexist during a live coding set, and therefore there is a probability of overloading the cognitive processes of the performer (Nash and Blackwell Reference Nash and Blackwell2011). Some live coders have stated that they are rarely able to develop new ideas while performing, and they tend to recycle similar patterns and structures well assimilated into memory (McLean and Wiggins Reference McLean, Griffiths, Collins and Wiggins2010). As a consequence, the structural organisation of the sonic material and the musical form has often been overlooked in live coding-related literature. We argue that this element is under-scrutinised due to live coding intrinsic lean towards improvisation.

Moreover, Sarath proposed that the main difference between composition and improvisation is the possibility to move back and forth in time (Sarath Reference Sarath1996). Composing with the support of traditional scores allows a musician to modify and refine musical material regardless of the order in which it appears during a piece. On the contrary, if we improvise, we cannot refine the introduction of a piece based on an idea that we developed in the coda. With this perspective, we can observe how the two approaches imply different cognitive and musicking strategies in the relationship between musicians and music. The traditional Western dichotomy of composing–performing indeed tends to fail to represent contemporary electronic and digital music performances (see, for instance, Lansky Reference Lansky1990). However, we argue that this difference in the way composing and improvising deal with time is relevant in relation to musicking strategies.

2.2. Visual support in live coding practice

The relationship between computer music and live visuals has a long tradition. Far from aiming to provide our reader with an exhaustive account of such a relationship, we present here some main references that acted as a background against which we framed our two systems.

The use of visuals in digital music has been primarily used to compensate for the lack of feedback for the audience when a laptop is involved in the performance (Correia, Castro and Tanaka Reference Correia, Castro and Tanaka2017). Starting from this original purpose, different visual solutions have been proposed to also support the understanding from the perspective of the performer (Joaquim-Fernandes and Barbosa, Reference Joaquim-Fernandes and Barbosa2013) or of the musical piece (Hunt, Mitchell and Nash Reference Hunt, Mitchell and Nash2017). Live coding lies in an advantageous position compared with other laptop music practices, as the code is always exposed to the audience. Overall, in our approach we do not manage visuals from this traditional audience-centric perspective; rather, our visuals are a form of descriptive scores that provide the performer itself with different visual feedback. We do not exclude the possibility of using such systems as live visuals for the audience, simply, in this article, we just focus on the relationship between the performers and the systems.

According to Magnusson, the code can be considered as an archetype of visualisation of formal processes (2011), therefore as a form of score in itself (2014b). In the digital music domain, the term ‘score’ has been used for a variety of different purposes. An overview of the scores in digital musical instruments has been recently proposed in a systematic scrutiny of the NIME performance (Masu, Correia and Romao Reference Masu, Correia and Romao2021), and five main uses of scores were identified: 1) scores as instructions or information (suggesting how to play an instrument; as in Hamano, Rutkowski, Terasawa, Okanoya and Furukawa Reference Hamano, Rutkowski, Terasawa, Okanoya and Furukawa2013); 2) scores as an interface to play a DMI (score as a controller that can be tangible; e.g., Tomás and Kaltenbrunner Reference Tomás and Kaltenbrunner2014), virtual (e.g., Masu et al. Reference Masu, Bala, Ahmad, Correia, Nisi, Nunes and Romão2020), or in form of code with a graphic visualisation (e.g., Magnusson Reference Magnusson2014b); 3) score as synchronisation (the system uses a score to synchronise various events; e.g., Orio, Lemouton and Schwarz Reference Orio, Lemouton and Schwarz2003); 4) score creation (tools that support the creation of instrumental scores; e.g., Garcia, Tsandilas, Agon and Mackay Reference Garcia, Tsandilas, Agon and Mackay2011); and 5) score recording (score as a recording of performative actions; e.g., Liang, Fazekas, McPherson and Sandler Reference Liang, Fazekas, McPherson and Sandler2017).

In this taxonomy, live coding would fit in the category of ‘scores as an interface’, where the code (a form of notation) is the input of the digital system that is created or manipulated during a performance. However, as highlighted by Collins, since the code is generally overwritten or erased during the performance, it may not represent the piece as a whole, but only a cross-section of it at a given moment (Collins Reference Collins2003). To cope with this issue and support the understanding of specific musical elements, various visualisation systems have been implemented. While some of them actually constitute new programming environments, mostly replacing the coding part (e.g., McLean, Griffiths, Collins and Wiggins Reference McLean, Griffiths, Collins and Wiggins2010; McLean and Wiggins Reference McLean and Wiggins2011), others rely on existing systems, complementing and supporting the coding part. A relevant example is Magnusson’s Threnoscope, which consists of a series of concentric circles each representing a drone and its parameters (Magnusson Reference Magnusson2014b). Abreu proposed another system, the Didactic Pattern Visualizer, which allows the visualisation of sound events sequenced from the TidalCycles library, arranged on a temporal grid (Abreu Reference Liang, Fazekas, McPherson and Sandlern.d.). Although these two systems propose different temporal representations, in both cases the entire form/score of the piece is not visible. These systems are hierarchically subordinate to the sound component, and their purpose is primarily to visualise some aspects of the musical creation that are strongly correlated to specific musical needs. To underline the support function of these two systems, both authors use terms such as ‘helpful’ and ‘didactic’. Purcell reflected on these differences and proposes two macro-categories of visual techniques in live coding: aesthetic and didactic (Purcell, Gardner and Swift Reference Purcell, Gardner and Swift2014).

The two visualisation systems that we proposed in this article as probes belong to the category of ‘scores as an interface’. However, they play a complementary function to the code, indeed the score acts as feedback providing a general view of the musical events. Additionally, while the code usually displays what is playing now and what will play in the near future, our visualisation systems focus on already played events. As such we try to 1) bring part of the improviser’s focus into the past temporal dimension, and 2) support the musicking process of developing a musical piece.

5.1. New technology suggests new ideas

A musical instrument – and per extension, our visualisation tools – embeds a musical vision. Magnusson suggested that a musical tool ‘has such a high degree of symbolic pertinence that it becomes a system of knowledge and thinking in its own terms’ (Magnusson Reference Magnusson2009: 168). The author further reflects that the technologies used as part of practices of making and thinking music incorporate the musical ideas of its author(s): ‘Writing digital musical interfaces therefore necessarily entails the encapsulation of a specific musical outlook’ (ibid.: 173). Using terminology from Latour (Reference Latour1987), the musical ideas are black-boxed in the interface itself.

Musicians offload part of their musicking thinking process into the tools and instruments they used. Magnusson proposed the idea of an epistemic tool as ‘a system of knowledge and thinking in its own terms’ (Magnusson Reference Magnusson2009: 168). The idea of epistemic tools incorporates elements from the extended mind theory by Clark and Chalmers (Reference Clark and Chalmers1998), as well as elements from enactment by Varela, Thompson and Rosch (Reference Varela, Thompson and Rosch2017). Enactment refers to the idea that cognition is the enactment of a mind and world on the basis of the variety of actions performed (ibid.). Learning to play is, therefore, an enactive activity. Clark’s concept of an extended mind proposes the idea that humans tend to extend the cognitive process outside the head, offloading part of it on tools and instruments. In a live coding performance, the creative process is partially offloaded in the code. When we added the visual components, we inserted a layer that provides different feedback on what the code does, and therefore, it can change the musicking process. A new layer requires specific focus and attention. This redundancy of stimuli, in which each layer actually represents the same thing in different ways, was overall appreciated, but also required a different way to manage focus and attention while performing:

Due to the need to switch attention, introducing graphic visuals could increase the idea-to-code latency (McLean and Wiggins Reference McLean and Wiggins2009), since the code is not only translated into sound (code-to-sound) but also into visuals, creating a parallel code-to-visual process. Such a process could in some cases initiate a visual-to-idea-to-code loop, increasing the overall idea-to-sound latency during the performance. However, loops can also reduce this latency, as they provide feedback that helps understand the musical patterns in a specific moment:

Additionally, by observing the visuals some specific new ideas emerged. As discussed by Magnusson (Reference Magnusson2009), playing music is an enactive activity. In this context, enactment refers to the idea that cognition is the enactment of a mind and world based on the variety of actions performed (Varela et al. Reference Varela, Thompson and Rosch2017). Further developing the relationship between cognition and the external world, Rowlands developed this idea by asserting ‘things going on in the environment partially constitute a cognitive process’ (Rowlands Reference Rowlands2010: 21).

As musicking is a cognitive process, it comes naturally that the visualisations – an add-on to the live coding environment – have an impact on the performative strategies and contribute to the germination of novel musical ideas. Live coding is also an enactive activity, in our case mediated by the visual. This extension of the cognitive music process in the visuals is exemplified by the fact that visuals suggested new musical patterns. For instance, Time_Z suggested to Etol to fill the graphic space until the patterns became ‘something else, completely different’ (Etol). This idea emerged by watching the visual that reminded him of ‘some kind of cloud’ (Etol). In some cases, MrReason and Etol developed strategies by purposefully changing the graphic behaviour of the visuals (not just passively observing them).

Overall, the introduction of visuals modified the musician–code–music relationship, providing new interactive patterns that opened up different possibilities. Historian of technology Melvin Kranzberg wrote: ‘Technology is neither good nor bad; nor is it neutral’ (Kranzberg Reference Kranzberg1986: 545). Additionally, music ‘programming languages may be as culturally loaded as the communities of practice that produce and use them’ (McPherson and Lepri Reference McPherson and Lepri2020). It is therefore quite obvious that adding visualisations affects the performance. These modifications in the performance produced new ideas, as suggested by Etol:

In this sense, the observation by Etol echoes what was stated by Evan Parker in Phil Hopkin’s film Amplified Gesture:

Furthermore, through our interactions with the three live coders, we observed how the visualisations have agency. As any artefact has agency, as pointed out by Latour in his Actor Network Theory: ‘Any thing that does modify a state of affairs by making a difference is an actor’ (Latour Reference Latour2005: 71). In the musical domain, in particular, technology always expresses a specific musical vision, practice or theory:

5.2. Different visualisations for different musical approaches

In the previous section, we have seen that instruments have agency, and as such, they introduce a new level of complexity in the musician–code–music relationship that promotes new musicking strategies. We will now discuss how the agency of the visualisation does not have an absolute inscribed and immutable value that suggests the same musical ideas to any live coder. To this end, we rely now on the concept of affordance as conceptualised by Gibson (Reference Gibson2014). Indeed, on the one hand, the fact that artefacts have agency is a truism, on the other hand, the affordances are determined by the coupling relation between an object – in our case the visualisation – and a person – in our case the live coder (Gibson Reference Gibson2014).

The musical ideas that a certain type of visualisation affords are not intrinsically determined in the system but determined in the ecological relationship between the live coder and the system. Etol and u-mano u-dito reflected upon the fact that each of the two systems better fit different approaches and that different live coders would naturally prefer one or the other. The two of them expressed a preference towards Time_Z, as it gives more immediacy and is more congruent with their usual way of performing live coding. In alignment with Etol and u-mano u-dito, MrReason argued that different systems better accommodate different composition approaches. However, he preferred Time_X, and added that this system helped him to reflect upon a problem musically related to the formal development he was already considering, because ‘I can directly see the development of the performance. I think that’s positive’ (MrReason).

MrReason also reflects on the scarce variety that his live coding performances have in terms of structure. We can speculate that the lack of a form of notation helping to visualise the piece in its entirety caused to a certain extent a scarce variety of different musical structures. Sarath supported that the main distinction between composing and improvising is the possibility of going back in time (Sarath Reference Sarath1996). Despite Time_X not allowing the performer to go back in time (as it is still generated in real time), it allows visualising the events at the beginning of the performance, as such, it allows for the possibility to keep considering events occurring in a longer period. As a result, MrReason de-intensified parts of the piece by introducing rests and silence, supporting variety in the overall form. However, we also observed that this feature was not particularly valued by two out of three of our participants.

Indeed, Etol perceived the lack of zoom details in Time_X as a limitation, and was forced to explore the ‘speed’ function more than he usually does to create a more sparse performance ‘in order to create a more easily searchable graphic result’. He discovered that this strategy integrates well with the rest of his live coding ones. This observation on constrained creativity is in line with what psychologist Margaret Boden stated about human creativity. The author proposed that limitations and constraints are far from being the antithesis of creativity. On the contrary, they map out a territory of structural possibilities that can then be creativity explored (Boden Reference Boden2004). Therefore, a limitation in a musical system can be a resource to develop new music strategies (Gurevich, Stapleton and Marquez-Borbon Reference Gurevich, Stapleton and Marquez-Borbon2010). It is interesting to notice how both Etol and MrReason have created a less dense musical structure despite having reached it from different paths: purposely exploring the form (MrReason) and overcoming a limitation (Etol).

As we have seen, the systems afforded different music ideas to the various musicians we were working with. While discussing affordances in digital interactive systems, Gaver advanced the proposal of hidden affordances that emerge in the use of a specific piece of technology of which the designer was not aware (Gaver Reference Gaver1991). Such an interactive system has a level of ambiguity that emerged by coupling systems with people. ‘Things themselves are not inherently ambiguous’, rather ambiguity is determined through an ‘interpretative relationship between people and artefacts’ (Gaver, Beaver and Benford Reference Gaver, Beaver and Benford2003: 235). Reflecting on ambiguity in music systems and how it begets agency, Stapleton and Davis proposed that:

The musicians we collaborated with engaged with our visualisation system from different experiences in contexts co-determined by their usual connections, samples, functions and synths. The specificities of their new strategies are idiosyncratic and cannot be generalised as characteristic of our visualisation systems, nor should they be. That being said, it is interesting to observe how in general different cognitive processes led to ideas related to musical structures and manipulation of density.

5.3. Visualisations as scores

While developing their personal relationships with our systems, our participants related themselves to the visuals as different forms of musical scores. When communicating with our participants, to avoid biases, we purposefully avoided using the term ‘score’. However, the concept of score was so clearly embedded in our study that the term emerged both in the diaries and in the interviews. We trace here how it assumed different meanings in relation to the different considerations that emerged.

Time_X has been perceived as a score in the sense of a recording or a ‘trace of the sound’ – using an expression proposed by Etol in his diary – of the performance. This would place Time_X as a form of ‘score as a recording’ according to the recently proposed taxonomy of score based on the analysis of the NIME proceedings (Masu et al. Reference Masu, Correia and Romao2021). Indeed, Etol stated that, by looking at the score, he was able ‘to identify macro-spots in the structure.’ He also supported that these scores could be useful to revise performances afterwards, and claimed that he could probably retrace what he did.

Based on this comment, we could position Time_X as belonging to a trend that has a long history in visuals score for electronic music, which is visualising existing pieces (Adhitya and Kuuskankare Reference Adhitya and Kuuskankare2012). Already in 1954, German composer Karlheinz Stockhausen created a visual score for his electronic piece Studie II. In that case, the score would grant the possibility to actually recreate the piece. Another historical example is Rainer Wehinger’s visual listening score to accompany Gyorgy Ligeti’s Artikulation. Magnusson has recently called for a musicology of code, arguing that the ability to read code should be a natural extension of the musicologist’s skill set in the modern age (Magnusson Reference Magnusson2019). We agree with this statement, however, a graphic score that visualises the overall structure of performance could complement the information provided by the code, for instance by providing an overview of the structure. This last point was also highlighted by MrReason:

In the Western tradition, scores progressively assumed the role of the incarnation of the ideal artwork. The idea of Werktreue (a German word that could be translated into ‘true work’) represents such an idealisation (Goehr Reference Goehr2007). The cases that we are discussing here, especially in light of MrReason’s comment, propose a notion of score that at least partially does not align with the notion of score as Werktreue. Indeed, despite that the visuals created with Time_X can be used a posteriori to recall, study, and even partially recreate a performance, it does not represent the piece in its entirety, nor provide all the information necessary to recreate it (it would be necessary to know the mapping, and have the actual samples and synths controlled via a code-score).

Despite being less similar to a traditional score, Etol reported in the diary that Time_Z, which does not allow for later recovery of the piece at all, represents a better score for live coding. Elaborating more on this during the interview, however, he reflects on the fact that Time_Z is not actually a score, as ‘it gives you a visual mapping to the things you are doing at the moment. I liked very much seeing the segmentation of my material’ (Etol). Similarly, u-mano u-dito also preferred Time_Z, as it ‘paints’ a visual counterpart to what he was doing musically, ‘slowly vanishing without leaving a trace’. He added that he prefers ‘the classic interactive aspect of the performance, as what I did before may not interest me.’ (u-mano u-dito).

Overall, we can reflect on how shaded the concept of score can be. The idea of a score as a representation of an entire piece is still quite strong – ‘It shows what I am doing in real time, it’s not a score’ (u-mano u-dito), ‘Time_Z … is not really a score’ (Etol). However, in its initial statement, Etol argued that Time_Z is a better score for live coding. By looking at these statements, our interpretation is that, on the one hand, the term score is still impregnated with the idea of Werktreue, on the other hand, the new forms of digital performance are indeed pushing towards a new conception of score. Alvin Lucier, while commenting on Gordon Mumma’s Hornpipe (1967) proposed that ‘the scores were inherent to the circuits’ (Lucier Reference Lucier1998). Therefore, we can argue that new conceptions of scores have been proposed at least for 50 years. So why are we still so bound by a traditional conception of score? On the one hand, we have to remember that our participants come from European countries, and can therefore have a natural bias towards the latter conception of score. On the other hand, if that is the case, we could assume this bias can be encountered in many other European live coders. Is that because in the ‘common language’ a score is a piece of paper with staves and notes whose graphemes were formalised more than 500 years ago? Replying to these questions is beyond the scope of this work. However, we wish to highlight that to work towards a visualisation of live coding as a form of musical reflection, the intrinsic bias in the term ‘score’ needs to be considered. Overall, a ‘traditional’ form of score, where the entirety of the musical symbols is arranged along the x-axis, might not be preferred by some live coders, but it can give new ideas to others.

5.4. Reflections on graphic visualisations

We finally collected a number of practical suggestions and feedback on issues or improvements related to the visual components. We report them here, hoping that they can be of use to design other systems in the future. MrReason and Etol state that in some moments of their performances the visuals (Time_X in particular) distracted them, this can happen because ‘they are very present in the foreground’ (MrReason).

With Time_Z, we have gathered mixed opinions on whether the object clusters are too many or too few to deal with during a performance. For instance, MrReason reported that

On the other hand, Etol ‘would have added other shapes and colours, further subdividing the macro-categories we had previously clustered in order to have more references.

However, this remains a completely personal factor and depends on ‘the type of use that each performer makes of his/her sound material and on what and how much he/she wishes to visualise’ (u-mano u-dito). Finally, in the Time_X approach it emerged that there is a risk for ‘too much rhythmic density, therefore [with Time_X] the visualisation became a kind of a big stain’ (Etol).