Set-up

Working with complex technologies in live situations presents a number of challenges. They concern the players as well as the audience, and they are practical as well as theoretical. In fact, these different dimensions are inextricably intertwined. The primary issues centre around questions of agency and sense – two categories that are themselves closely connected. In the sense that agency and sense are called into question for all involved, playing with complex (algorithmic) instruments can be considered a particularly graphic example of what Alessandro Bertinetto calls the ‘grammar of contingency’ that is at the centre of the artistic practice of improvisation as such: ‘Articulating an artistic grammar of contingency means constructively facing accidents, disorder, and chaos – relative to a system of expectations – by inducing (in) an unforeseen (way an) aesthetic sense through this felicitous encounter’. (Bertinetto Reference Bertinetto2022, 2:28–29) This account calls for a detailed analysis of specific situations.

In our artistic research project ‘Spirits in Complexity – Making Kin with Experimental Music Systems’Footnote ¹ , we investigate the functional and affective relationships between humans and technological objects within the realm of experimental music-making, practising making kin (Haraway Reference Haraway2016) as the main artistic research method. Our focus lies on complex technological devices, especially those with programmable components.

One experimental set-up we developed is called ‘Black Box Music’. This approach to musical improvisation involves electro-acoustic black box systems: each system is created or selected by one team member but performed by another, who is unaware of its inner workings. However, as Borgo (Reference Borgo, Siddall and Waterman2016, 7) argues, black boxes are not static or abstract unknowns, but instead have a gradual nature and are ubiquitous in daily life. They afford ‘a fundamentally performative engagement: it is something that does something, that one does something to, and that does something back’ (ibid.).

‘Black Box Music’ refers to this unstable situation, forming an ensemble of human players and non-human ‘black box’ participants. These non-human partners include both analogue and digital technologies. The sonic character of all four systems draws loosely on the sketchy, noisy textures of AM/FM radio reception, inspired by a speculative extrapolation of Thomas A. Edison’s purported ‘Spirit Phone’ (see Lautour Reference Lautour2015).

Prior to performances, the instruments were briefly introduced to the players in terms of fundamental dos and don’ts. However, there were no rehearsals in order to retain the opaque character of the situation. Given the complexity and unpredictable nature of the devices – due to potentially multiple layers of technology and arbitrary mapping (cf. Magnusson Reference Magnusson2009) – unknown behaviours are likely to be spontaneously exhibited. The fact that there was always one of the participants who knew more about the instrument than its actual player gave rise to interesting dynamics of musical anticipation: The creator of an instrument might anticipate what its player might do next and try to react to it. The distribution of instruments followed a certain symmetry: two instruments were AI-focused, and the other two followed a more physical idea. We did not aim at making interaction purposefully difficult, e.g., by sabotaging players (cf. Dannemann et al. Reference Dannemann, Bryan-Kinns and McPherson2023); in fact, guidelines for instrument design were not rigorously defined.

The central idea was to suspend all relevant making kin processes – specific for each instrument/player combination – to the realm of public performances.

A ‘Black Box Music’ performance can be seen as a musical improvisation that extends practices of free improvisation. While all improvisation involves navigating unpredictability and responding to the unexpected (Bertinetto Reference Bertinetto, Bertram, Feige and Deines2021), ‘Black Box Music’ radicalises this uncertainty by embedding it directly into the instruments themselves.

Unlike typical improvisation, where unpredictability stems from interpersonal dynamics, ‘Black Box Music’ performances are based on the unfamiliarity and non-triviality (see Von Foerster Reference Von Foerster2003) of instruments (cf. Grill et al. Reference Grill, Döttlinger, Lechner, Castelló and Flexer2025). The usage of the term Non-trivial instruments should not imply that other musical instruments are of little value or are relatively easy to play or learn. Non-trivial instruments resist full control or prediction, making it impossible for performers to rely on habitual techniques or personal creative repertoires. Instead, each musician must explore how to interact with their specific instrument ad hoc – an exploration that often delays meaningful group responsiveness.

This setting might discourage clichéd gestures and demands acute listening – not only to the other players but also to the instrument’s own behaviour. The instruments invariably take on an active, quasi-autonomous role in the performance, making it unclear whether a sound originates from human intention, technical failure, or the machine itself. Over time, performers typically develop a better feel for their instruments and the ensemble, allowing for a shift from negative (reactive) to positive (proactive) improvisation, using Dehlin (Reference Dehlin2008)’s terminology.

Theory

Working with highly complex (algorithmic) instruments provokes and produces very distinct types of ‘accidents, disorder, and chaos’ that have to be described and understood. Trying to make sense of what is happening is something all participants are constantly engaged in, albeit in different ways. Since the interaction takes place in the context of a musical performance, the question of a specifically aesthetic sense is one of the dimensions of this ongoing process of sense-making.

Actor-Network Theory (ANT) can provide the framework within which the constellations we are dealing with can be described: In the situation of performing, the players and the instruments alike have to be considered contributors to a situational network, actors or ‘actants’ in their own right. ANT’s famous creed, ‘Follow the actors!’ calls for a close description of all entities involved in a given situation and the actual roles they play, without any theoretical preconceptions that would limit the researcher’s ability to acknowledge the actual contributions of non-human actors. In this sense, an adequate description can claim ontological status: it lays out an ontology by tracing roles, relations, and interactions of everything and everyone involved (cf. Latour Reference Latour2007).

Latour and others have always stressed that what we are dealing with are complex relations in formation rather than fully formed, static networks. This entails that ontology is at stake rather than fixed. However, the practice of ‘Black Box Music’ shows that things may be more complicated still: There are conflicting interpretations within the situation itself, interpretations that are influenced by the actors’ background convictions and theoretical commitments. In the case of interaction with algorithms, the question of agency is constantly at issue, and it can be dealt with rather differently. Furthermore, the relation between theoretical commitments and pragmatic modes of interacting is not always clear.

In their critique of ANT, Georgina Born and Andrew Barry point out ‘the tendency to elide or confuse the two: analytical ontology and the ontology of the people/culture/music that are the focus of research’ (Born and Barry Reference Born and Barry2018: 478). Their point is that the ‘flat’ ontology of ANT, in which no person, entity, or structure can claim special status, is not as neutral as it seems: It can be at odds with the ontologies of the participants. Following the actors may not be all that easy when these actors are themselves unsure what to make of a situation. What if there is no unambiguous ontology, not even a situational and pragmatic one? How things relate to each other, how they are pragmatically treated, and how they are understood may or may not coincide and may or may not be shared and stable. The task of research would then be to trace these ambiguities and instabilities rather than assuming a coherent ontology that can be uncovered or simply reproduced.

In principle, ANT allows that. The flat ontology that Born and Barry criticise is problematic as a presupposition that determines the outcome of any investigation. It does make sense, however, as a starting point of an investigation that suspends all preconceptions concerning the roles that actants can play. The unstable, fractured ontologies that we are dealing with can be grasped if we insist on the processual, open character of the network that includes the equally unstable attitudes and beliefs of its human participants.

In this context, George Lewis’s position is illuminative: ‘In the context of interactive computer music, I am not proposing an exploration of the metaphysics of machine consciousness but of a phenomenology of freedom as dialogic interaction, in which a creative machine can be perceived by us as acting in a free way’. (Lewis Reference Lewis2019b: 445) This can be read as an agnostic stance: Without actually imputing freedom or consciousness to the machine in question, it is dealt with as a free agent simply because this is the prerequisite for productively interacting with it at all. Building on the ‘intentional stance’ by Daniel Dennett (Dennett Reference Dennett1989), Eric Lewis points out that it is ‘often advantageous to us to treat systems that are very different from us as if they manifest intentionality’, especially ‘when doing so has us gain predictive power regarding the future states of the system’ (Lewis Reference Lewis2019a: 96). Even if the player held the conviction that there can be no such thing as machine consciousness, this would simply not be relevant. Of course, this agnosticism is still only one possibility, and it might turn out to be impossible to uphold: the practice of treating a machine as a free agent might unconsciously slip into the belief that it is one. Hence, the ‘distinction between belief and make-believe’ that Eric Lewis applies here, appropriating it from the theory of fiction, might not be as stable as one thinks. (Lewis Reference Lewis2019b: 97) In practice, a player may be sure that she is dealing with nothing but a complex, non-trivial machine or be convinced that there are actual spirits that inhabit this complexity, which might or might not make a difference in their actual practice, and it might or might not make a difference for the audience as well. What is more, these convictions and their relevance may not be stable but shift in the course of the performance.

This agnostic stance and its tendency to collapse into a clear ontological commitment can be related back to the ‘imitation game’ by Alan Turing (Turing Reference Turing1950), which later came to be known as the Turing test. In this test, an evaluator has to tell a machine and a human apart in a natural language conversation. If the evaluator is not able to distinguish them reliably, the machine has passed the test irrespective of its inner workings. This behaviouristic position was the foundation of the scientific field of Artificial Intelligence (AI), where, at its eponymous founding conference in 1956, it was stated that any ‘feature of intelligence can in principle be so precisely described that a machine can be made to simulate it’. (McCarthy et al. Reference McCarthy, Minsky, Rochester and Shannon2006). The point was to simulate intelligent human-like behaviour without caring about implementation details and how a machine actually achieves it. However, this has invited an anthropomorphisation of machines and computer programs in the interactions themselves, a tendency that is doubtlessly reinforced by the very term artificial ‘intelligence’. A famous early example is the ELIZA system (Weizenbaum Reference Weizenbaum1966) that convincingly simulated a psychotherapist via a very simple textual interface. The tendency to project human traits even into rather simple computer programs has come to be known as the ‘ELIZA effect’ (Hofstadter Reference Hofstadter1995) and is prevalent in today’s interaction with advanced AI systems like ChatGPT. (Akbulut et al. Reference Akbulut, Weidinger, Manzini, Gabriel and Rieser2024)

To make this a little more specific with regard to complex musical instruments, we can draw on the categories Don Ihde introduced. Instead of raising the question of ontology, Ihde focuses on the way humans relate to technology. These relations encompass practices and convictions, which, as we have seen, don’t necessarily align. Ihde distinguishes ‘embodiment relations’, ‘alterity relations’, ‘hermeneutic relations’, and ‘background relations’. (Ihde Reference Ihde1990) It’s the first two that are particularly relevant for us: An embodiment relation occurs when an instrument is incorporated into an actor’s relation to the world to such an extent that it becomes all but invisible to her. This type of relation to objects was first prominently theorised in the phenomenology of the body and later introduced into cognitive science as ‘extended cognition’: ‘For anything external to the body’s boundary to count as a part of the cognitive system, it must function transparently in the body’s sense-making interactions with the environment’. (Thompson and Stapleton Reference Thompson and Stapleton2009, 29) The classic example is the blind person’s cane that Maurice Merleau-Ponty refers to: using the cane means incorporating it into one’s sensory apparatus (Merleau-Ponty Reference Merleau-Ponty2002, 165–66), almost like a kind of prosthesis (Flusser Reference Flusser2010). To the extent that the blind person’s awareness is on the tip of her cane, it turns from an object into a means of perceiving and does indeed become transparent. Something similar may be true for a competently mastered instrument, which turns into a medium of articulation, a ‘phantom limb’ (Szendy Reference Szendy2015). Still, it never loses its materiality and with it the possibility to be viewed as an object, and it may even be treated as a familiar counterpart in an interaction, which brings us to the next type of relation.

An alterity relation emerges when the technological object is experienced as a partner in an interaction – an other. To be sure, this is a ‘quasi-otherness’, but there is the ‘tendency to fantasize its quasi-otherness into an authentic otherness’ (Ihde Reference Ihde1990, 106). As in Lewis’s observation, it is important to note that the question of real or authentic vs. imagined otherness simply may not arise, even in Ihde’s examples of early video games. In competing against the machine, one is effectively treating it as an other without making any ontological commitments. Rather than exclusive alternatives, these types of relations should be considered extremes on a continuum. All along this continuum, from the cane to AI, the technological objects are actants whose actual contributions to the situation vary greatly.

The fundamental question is thus how players, audience, and observers make sense of what they are doing and observing. This starts at a very basic level: In philosophy and cognitive science, the concept of sense-making has been employed to describe a range of relations from the fundamental interaction of an organism with its surroundings through communication to higher cognitive processes. In the most general terms, it can be described as ‘transform[ing] the world into a place of salience, meaning, and value’ (Thompson and Stapleton Reference Thompson and Stapleton2009, 25) – an activity that is always already going on and does not imply explicit interpretations. Rather than being imposed, sense emerges out of the interaction itself. Implicitly or explicitly, this process constantly asks and negotiates: What is this and what relation does it have to me/to us? And who am I/who are we in relation to this and to each other? Sense-making is thus where questions of ontology are negotiated.

Since humans are social beings through and through, social interaction is the general condition of human sense-making as such and not a special field to which the concept can be applied. In this sense, all human sense-making has a social aspect and can be seen as participatory in a loose sense (De Jaegher and Di Paolo Reference De Jaegher and Di Paolo2007), although the definition of ‘participatory sense making’ is typically reserved for mutual understanding between individuals (Di Paolo et al. Reference Di Paolo, Cuffari and De Jaegher2018). In music, sense-making may be described as ‘epistemic interactions with the sounds’ (Reybrouck Reference Reybrouck2012: 392), but this process is always embedded in a social context and has a strong aesthetic dimension that acts as a constraint. In order for the interaction to ‘work’, i.e., to be convincing, it has to result in a compelling, perceptually coherent structure. What the players consider successful and what the audience finds convincing do not necessarily coincide. All this is part of what Christopher Small calls ‘musicking’ (Small Reference Small1997). As a type of collective sense-making, it encompasses different roles – composing, building instruments, playing, listening… It is participatory but not uniform.

Performing

So far, ‘Black Box Music’ has been publicly performed three times, with ever-shifting combinations of performers and instruments. The duration of each performance was set to be 25 minutes, including the performance at the Speculative Sound Synthesis Symposium (see Grill et al. Reference Grill, Castelló, Lechner and Döttlinger2024) in September 2024. This public event was part of the Symposium’s artistic programme and presented at the György-Ligeti-Saal of KUG Graz. At the Spirits in Complexity workshops with our cooperation partners, the Royal Institute of Technology (KTH Stockholm), in a public event at the Klangtheater of the University of Music and Performing Arts Vienna in November 2024. At the Intelligent Instruments Lab (IIL) at the University of Reykjavik, in August 2025, at a public event at the lovely 12 Tónar record store in Reykjavik. See Figure 1 for a depiction of the four `black box’ instruments and Figure 2 for a photo of the music performance in Reykjavik, Iceland.

Figure 1.

Black Box instruments, made by Marco Döttlinger (top left), Thomas Grill (top right), Angélica Castelló (bottom left) and Patrik Lechner (bottom right).

Figure 1. Long description

The collage consists of four distinct images related to electronic music instruments and equipment. In the top left image, a laptop screen displays a software interface with diagrams and text, and a black keyboard is positioned in front of it. The top right image showcases a control panel with sixteen knobs and switches, labeled ‘DOSAFEER P T CONTROL’ at the top. The bottom left image features a vintage amplifier with a red device placed on top and a black cassette player in front of it. The bottom right image shows another laptop with a software interface displaying a graph, and a sound mixer with various controls is placed below the laptop.

Figure 2.

Black Box Music performance at the 12 Tónar record store in Reykjavik, Iceland, as part of the ‘Spirits in Complexity’ symposium at the Intelligent Instruments Lab.

Figure 2. Long description

Four individuals are gathered around a table, engaged with various electronic music equipment. The person on the left, wearing a hat and a scarf, is handling a record. The second person from the left, wearing glasses and a dark sweater, is observing the equipment. The third person, with curly hair and glasses, is also interacting with the devices. The person on the right, wearing a striped shirt and glasses, is adjusting a piece of equipment. The table is cluttered with various electronic devices, cables, and records, indicating a collaborative music performance setup.

In the following, accounts are given from the playing experiences by the performers of the ‘black box’ instruments designed by other team members. Descriptions of the instruments’ technical functionality can be found in Grill et al. (Reference Grill, Döttlinger, Lechner, Castelló and Flexer2025), section ‘Black box instruments and performance experiences’.

In the accounts, we compare opposite perspectives on the interaction with the instruments. One (by Döttlinger and Grill in ‘Playing with expectations’), which uses a quite analytical perspective, informed by instrument-design know-how, and one (by Castelló and Lechner in ‘Playing as if’), which takes the personhood of the objects quite literally. As noted, these attitudes should not be considered as stable convictions but rather as tendencies, extremes on a continuum in which any actual interaction happens.

These interactions with our instruments do not depend on full comprehension or control, but rather emerge through making-kin with the system’s otherness. Kinship is a performative act; it is therefore mediated by pre-rational knowledge. Yet, ‘making kin while performing’ is always an attempt to ‘feel’ the instrument (cf. De Souza Reference De Souza2017).

Reflection

‘Black Box Music’ is an exercise in producing ambiguity by increasing complexity. It is helpful to distinguish complexity from complication, as Shirley Strum and Bruno Latour did in their 1987 paper on baboon and human societies: whereas situations where a ‘profusion of other variables impinge simultaneously’ (Strum and Latour Reference Strum and Latour1987: 790) are complex, complicated arrangements can be broken down to ‘a succession of simple operations’ (ibid., p. 791). According to Strum and Latour, the latter is the main achievement of human sociality: establishing formal and material arrangements that allow us to reduce complexity and focus on a limited number of variables at a time. Musical conventions like scores or traditions of performance practises can be seen as an example of such arrangements.

Regardless of the instruments used, free improvisation in itself entails an increase in complexity in musical interaction. Because it is a practise with a huge variety of sonic outcomes, reacting and interacting requires a high degree of constant attention, severely limiting predictability. Still, the fact that all this takes part in the situation of playing and listening to music creates an unambiguous frame that allows the participants to tune out everything that is not related to this highly specific situation. We are dealing with a localised, bounded complexity.

Introducing non-trivial instruments further increases this complexity. Not only does the number of variables increase, but it becomes unclear what they are in the first place. If anything you do produces unforeseeable results, there are no simple operations, at least not for the players. In the case of acoustic and most straightforward electric and electronic instruments, agency is limited to enabling certain operations and precluding others. In a sense, playing an acoustic instrument means navigating its specific resistance: Resistance is at the core of what a musical instrument can do and what it can’t do. With ‘Black Box Music’ and algorithmic instruments in general, agency becomes an open question that cannot be reduced to material resistance.

For the players, the most salient interaction is the one with the instrument they are playing, and this is where the negotiation of agency in all its ambiguity mainly takes place. But each participant’s perspective on the others is far from uniform, either. Rather, what we have is a multiplying of perspectives: Each player has a different view of their own instrument, their colleague playing this instrument, the foreign instrument they themselves have to deal with, and their colleagues playing one of the other foreign instruments. All these are bound to inform the actual interaction, which is fragmented into a multitude of different observations and expectations that influence each participant’s actions and ways of making sense of the process.

The anthropomorphising ELIZA effect (Hofstadter Reference Hofstadter1995) is bound to be at play in some of these relations. In the case of the ‘Black Box Music’ interaction, although the exact nature of the instruments remained opaque, all four musicians were familiar with each other’s previous artistic work and hence had certain expectations regarding the instruments and their ‘intelligent’ algorithmic nature. These expectations might have helped in evoking an alterity relation and caused players to anthropomorphise the instruments as ‘Others’.

In this situation, the question ‘who is doing what?’ becomes riddled with ambiguity. In the attempt to control them, the instruments literally do things that one has to react to, just like one has to respond to what the others and their instruments are doing. We have seen two different ways of interpreting this. If we take the ambiguity seriously, we have to accept that these interpretations may not be entirely a matter of decision – they may be something that happens and that changes over time. A musician may find herself acting contrary to her beliefs or even believing something that she finds hard to accept, only to find herself in a different position still during the next concert.

What the audience sees and hears, on the other hand, are not primarily musicians interacting with an electronic entity but musicians playing their instruments and interacting with each other. This is clearly different in the case of systems like George Lewis’s Voyager, which he has been developing and working with for decades and which he refers to in the statement quoted above. Voyager actually responds and plays, so that even for the audience, it is obviously a partner in the interaction. This does not mean that the ambiguities that are clearly present to the musicians simply disappear for the audience, but they do play out differently.

In listening to any kind of improvised music, the distinction between action and event becomes less clear-cut. What happens is usually something that somebody does, but for the listeners, it may still be perceived as an event whose intentionality is not altogether clear. This becomes particularly obvious in the case of listening to non-trivial instruments, since the sonic events do not necessarily stem from a performative action. And this is especially true for sonic constellations between different musicians that are never reducible to anybody’s intentions, even for the players themselves. In this sense, the events produced by uncontrollable technology do not introduce a whole new category into the audience’s experience but enhance a dimension that is always already there. Additionally, it is almost impossible to tell whether what we hear was intentionally produced or whether it surprised the player as much as the listeners. The lack of correspondence between observed actions and resultant sounds that characterises electronic music in general prevents clear attribution, see also findings of Gurevich and Fyans (Reference Gurevich and Fyans2011).

In this regard, there are bound to be differences between an unprepared lay audience, an audience that is generally aware of the set-up and its challenges, and listeners who are familiar with electronic and/or algorithmic instruments. For a lay audience, there is always an element of inexplicability; whether they associate this with magic or simply with unobservable technological functioning will depend on personal inclinations. In a time where relatively sophisticated tools for digital sound production and manipulation are freely available for anyone who owns a smartphone, the propensity to assume magic may be less widespread than it used to be.

Paradoxically, lay listeners who know little about the actual functioning of the instruments but who are familiar with GarageBand might be less inclined to hear spirits at play because they presuppose that the musicians exert complete control. An expert audience, on the other hand, will constantly be on the lookout for signs of a lack of control. Being familiar with the challenges, they might be more willing to concede an independent agency, maybe even the presence of a spirit.

Another question is how this plays out aesthetically. Asking who is doing what, what entities are involved and how the interaction works are all aspects of making sense of what is happening. Since the aesthetic is not just a by-product of this interaction but the reason for its existence, the question of making aesthetic sense is also always present, even when the overwhelming task of operating the instruments pushes it into the background. Part of the struggle for control may be a game, a competition between the musicians and their obstreperous instruments, but at the core, it is an attempt to create something that makes sense sonically. Temporarily letting go and seeing what happens may be another vital strategy in this process.

The fact that there are stretches during which aesthetic considerations tend to be suspended by the players because they are struggling to get a grip on their instrument does not mean that these periods are perceived by the audience as non-aesthetic. In the context of a musical performance, nothing is perceived as non-aesthetic or rather aesthetically irrelevant. However, it may be that what they are producing is perceived to make no aesthetic sense – to be disjointed, inarticulate, incoherent, uninteresting. This might even happen in situations with a high degree of control, while a complete loss of control might produce aesthetically convincing results. All these remain embedded in the overall situation of ambiguous agency.

It would be mistaken to limit aesthetic sense to the sonic dimension. In collective improvisation, interaction and communication become especially salient – though not necessarily as continuous mutual responsiveness (cf. Keith Rowe in Cobussen and Nielsen Reference Cobussen and Nielsen2012: 60). From this perspective, even deliberately ignoring one another constitutes a distinct mode of communication. From this perspective, even deliberately ignoring each other counts as a distinctive type of communication. Watching and listening to musicians interacting cannot be considered a distraction from the aesthetic appreciation of the music; rather, it is an integral part of it (cf. Bailey Reference Bailey1993: 44). The lack of correspondence between observed actions and sonic results, the shifting between action and event, the observed struggle, the concentration of the musicians, the gestural quality of their playing or a lack thereof all become part of the aesthetic sense of the performance. Improvisational strategies are multiple and emerge from attempts to engage with and partially grasp the behaviour of the machine systems. In doing so, performers become attuned both to the operational logics of the machines and to the responsive strategies of the other human participants within the quartet.

Our approach does not seek to achieve mastery of, or progressive ‘understanding’ of, the instrument in the conventional sense. Instead of treating the system as something to be learned through extended practice, we are interested in the conditions of a first encounter. In this context, the limited timeframe is not a constraint to be overcome, but rather a constitutive element of the experiment.

The associated uncertainty, risk and public exposure are central to forming what we describe as a situated relationship or ‘kinship’ with the system. This relationship does not depend on full comprehension or control, but rather emerges through engagement with the system’s otherness.

If this is successful, the irresolvable ambiguities concerning intentionality and agency cease to be a source of frustration and become interesting in themselves. They make sense precisely by making no definite sense, i.e., by offering open concepts.