1. Experience

There is a direct connection with the material. My right hand is in a stretchy black-knit glove with crumpled-up black rice paper glued to the fabric, forming a ruffled texture that covers the entire surface. In my left hand is a white feather, roughed up so that its downy barbs shoot out into the air. As I adjust the side lighting and camera aperture, the white feather blades slice through the black of the paper glove. Projected on the screen, the materials fuse together to give the impression of a living creature. Cropped in the macro-lens perspective, the paper forms the body of some kind of unfamiliar bird, or perhaps an insect, white feather spikes exploring the air-like antennae. As I turn the metal wire that holds the feather in my left hand, the antennae probe and turn in the microscope-like environment, peeking out from under the breathing body of the black paper. A synthesised sound moves with the antennae – granular clicks and flutters, micro-rhythms like tiny metallic feet or mechanical joints turning and clasping. The speed and sequences of intertwined rhythms adapt and modulate depending on the speed and distance of the movements. The experience is unified and physical. And yet something is missing. What does the material want? Does the character feel incomplete? The creature sounds thin, perhaps I should add another layer. Maybe a new mapping to create a sound that connects with the texture and mass of the paper? And a background layer to give more depth to the sonic image?

To keep track of the progress, I make a short recording of the movements in the scene and put it on loop. Placing the black paper glove and white feather interface down on the table, I turn back to the screen and watch the movement as an audience member. What should this sound like? What is the context of this situation? How do the materials interact? I open my signal processing software and begin coding an implementation for some sounds that could work to connect with the paper. White noise, down-sampling, resonant filter, panning based on x-axis position, which all should be polyphonic – so that means it needs a slightly more developed approach, and so on. After some time, I have a working synth that I can test with the objects. Putting the glove back on I try out the new sound material in context and realise the down-sampling needs to be more dynamic, and the filters should change resonance based on the size of the movements. I take the glove off again, and go back to programming, keeping the memory of the feeling of performance in my mind – remembering the way the paper appears to capture the feather like a pet, or prey, or at other times seems to be the body belonging to the antennae, which explore with a sense of intelligence.

The working method of the ‘video-puppet-instrument’ system I have been developing, like all digital instrument systems, requires a constant shifting between modes of engagement. The experience of the performance is one of holistic gestalt, a fusing of media – but the creation process divides the experience between the whole and the technical development of the parts: the video image, sound, computer vision mappings, puppet/object materials and the mechanics for movement. In hybrid media systems like this, there is a pronounced latency between the idea and the concrete realisation. As a result, there is a danger of getting lost in the minutia of technical issues, forgetting the overarching life of the piece and losing the sense of context and purpose that drives the process.

After spending hours adjusting the computer vision and signal processing code, I look again at the looping sequence. The material on the video moves and suddenly the piece comes to life again, the material coalescing into a multifaceted creature. I can predict its next movements, sounds and actions.

2. Sound-based material theatre

Material theatre is a working term for a stream of performance practice based in puppetry and object theatre that decentres the human and figural representation in favour of an open field of possible forms of character, scenography, dramaturgy and meaning (Grazioli Reference Grazioli2009). This tradition is closely connected with ideas developed in E. Gordon Craig’s Übermarionette (Reference Craig1908), Futurist Theatre, Dada, Antonin Artaud’s Theatre of Cruelty (Reference Artaud1958), Fluxus, and Heiner Goebbels’ Aesthetics of Absence (Reference Goebbels2015), where the audience, and performers, are confronted with new forms of life and contexts which they must form a relationship with in the shared space of performance. In contemporary music, these streams flow into the vibrant scene of intermedial composer-artists working in ‘expanded field’ (Ciciliani Reference Ciciliani2017) and ‘post-instrumental’ practices (Stene Reference Stene2016; Devenish Reference Devenish2021; Gottfried Reference Gottfried, Toro Pérez, Bennett and Hiekel2024).

The synthesis of post-puppetry performance with sound practices works towards an ideal of creating, or discovering life in the movements and combinations of objects, materials, sound and spatial experience. This merging into life echoes the vision of Dick Higgins’ Fluxian concept of ‘intermedia’, where different practices and modalities are so fused together that they cannot be clearly separated (Higgins Reference Higgins1967; Higgins and Higgins Reference Higgins and Higgins2001). Similarly, the expanded practices of instrument design, and more broadly human–computer interaction studies, focus on the creation of relationships between media and diverse modes of gesture.

In my work with video-puppet-instrument systems (e.g., Apophänie (2016), Scenes from the Plastisphere (2017/8), Animism (2021)), the puppeteer/object-performer/instrumentalists learn to engage with the objects in an embodied way. Interacting with the material directly with their hands, they experience their actions through the video camera’s macro lens, projected onto a large semi-transparent screen between them and the audience. At the same time they hear the sonic results of their movements via a computer vision system that maps the analysis data to real-time sound synthesis.

The physical materials are often organic forms (crumpled up pieces of paper, tangles of wire, feathers, plants), and in the highly mediated context of the video-puppet system, the performers must learn to feel the connection between their physical gestures and the resulting image and sound. Due to the instability of the material form, the behaviour of the perceived character changes quickly, often unpredictably, and so the performance practice requires an ongoing interpretation of the material actions on the screen, to identify what creature is there, how it moves and what possible continuations the material suggests. I think of this sympathetic relation with the material as a kind of animism, where the performers, and by extension the audience, discover the living nature of things. Through an observational practice focusing on the unique character and presence of material form, the animist approach in performance fosters an increased sensitivity to the ways materials are inclined to move and develop – building empathetic relations with nonhuman entities (Bennett Reference Bennett2010; Stengers Reference Stengers2012; Haraway Reference Haraway2016; Morton Reference Morton2017).

To assist in conceiving, discussing and forming collaborations to support this evolving practice of sound-based material performance, the emerging theoretical framework presented here focuses on the ways composers, performers and materials relate to each other as they develop towards collective forms of production. Alfred North Whitehead’s ‘philosophy of organism’ serves as a foundation, providing a vocabulary to describe the way processes perceive and respond to one another as dynamic entities, much like those discovered in material theatre performance practice.

3. Characters, interactions, agencies, entities

For Whitehead, each individual thing in existence is not a solid substance but a composite of processes – basic units of existence that he calls ‘actual entities’ or ‘actual occasions’ (used interchangeably). The identity expressed by the actual entity/occasion is formed in a process of ‘becoming’, in which the entity defines itself through the ‘aims’ that drive its behaviour and interaction with other entities. Whitehead refers to this interaction as ‘prehension’, a basic level of ‘feeling’ by which actual entities digest and respond to data from other entities, in an active ‘grasping’ of one another (Stengers Reference Stengers2023: 76). Prehension is described as having a ‘vector character’ operating in dimensions of emotion, purpose, valuation and causation (Whitehead Reference Whitehead1978: 19; O’Shea and Segall Reference O’Shea and Segall2022, Reference O’Shea and Segall2023).

Prehension has positive and negative forms. In ‘positive prehension’, relational data expressed by another entity is experienced as a feeling which harmonises with the aims of the entity, and so is ingested, becoming incorporated into the composition of the entity. In ‘negative prehension’, the data are felt as an aversion – a dissonance, clashing with its aims, and is rejected, not incorporated.

There is a distinct sense of subjective agency implied in prehension. Isabelle Stengers notes:

This aim that animates the prehension is a form of desire, a magnetic pull – in the way that certain molecules want, or are inclined to bind together. The actual entity/occasion composes itself through a synthesis of prehensions – a process of becoming which Whitehead calls ‘concrescence’.

As Stengers writes, the characteristic manner of the actual entity is embodied through composition, a process of concrescence, which gives the actual occasion its unique formal consistence, driven by its vector of aims, and aesthetic values which function as a ‘lure for feeling’ (Whitehead Reference Whitehead1978: 25).

Thus, by drawing on Whitehead, and the many theorists who have followed his trajectory, we gain access to a highly developed framework for understanding the intermedial objects of material theatre, not as lifeless matter, but as self-composing temporal creatures with their own agencies and tendencies – interacting through prehension, into a concrescence of becoming in a relational ecology of entities.

4. Object–entity relations, enactive instruments, ecological composition

In practice, the objects used in puppetry and material performance parallel the instruments used in musical performance – artistic tools and technologies which extend our modes of sense, medial articulation and range of expression. The more we use and rely on tools, the more they become incorporated Footnote ¹ into our sense of embodied being and action (McLuhan Reference McLuhan2002; Leman Reference Leman2007). In the process of learning how to use a tool, we gain understanding of the particular agency (Latour Reference Latour1987), expressed via its affordances and constraints (Gibson Reference Gibson2015) – the way an object suggests how it might be interacted with, or collaborated with. In dialog with Whitehead, we can think of Gibson’s affordances and constraints as the possibilities and limitations of an interaction, mirroring the entity/occasion’s expression of its character through positive and negative prehension.

In his work on instrument design, David Wessel similarly discusses the process of learning to perform with an instrument as an interactive exploration of the affordances and constraints of a system’s behavioural character. Referencing embodied cognition theory (Varela et al. Reference Varela, Rosch and Thomson1991), Wessel outlines an ‘enactive’ process of learning, emphasising the role of collaborative sensory-motor engagement with an instrument system towards the formation of a human-instrument symbiosis (Wessel Reference Wessel and Orlarey2006). An important element he identifies is the act of improvisatory ‘babbling’, named after the exploratory process of vocal experimentation in infant speech development, through which we learn to connect sounds with distinct physical processes. Then, as we learn to identify reoccurring actions and reactions, causes and effects, we develop the ability to shift into an ‘intending’ mode of interaction with our vocal system.

Joel Chadabe often expressed a similar contextual approach to performing with interactive systems, which he likened to ‘sailing on a stormy sea’, where the sailors have to adapt, learning to achieve balance in order to steer the ship as the wind and waves toss and turn around them.Footnote ² Chadabe referred to the epistemic nature of interactive systems as ‘intelligent instruments’ (Chadabe Reference Chadabe1984), or ‘composed instruments’ (Battier and Schnell Reference Battier and Schnell2002), where the system is designed in order to produce specific types of events, processes, algorithms – affordances and constraints that influence the ways performers interact with the system.

This collaborative perspective is an important principle in the practice of improvisation (Lewis Reference Lewis1999). As Derek Bailey writes, ‘to speak of “mastering” the instrument in improvisation is misleading. The instrument is not just a tool but an ally’. (Bailey Reference Bailey1993: 99) Mirroring Whitehead’s concept of prehension, a performer grasps the affordances of their instrument, learning to harmonise their actions with the inclinations of the tool/ally through enactive interaction, concrescing to form a human–nonhuman symbiosis.

We are constantly interacting with the things around us: living, organic, inorganic, soft, rigid, dry, wet – intellectually, emotionally, physically and through a potentially infinite number of other comingled materials and modes of perception. As dynamic entities, we are in a perpetual process of exchange with the other things and beings that form our environment. An ecology of actors inherently situates all of our actions (Suchman Reference Suchman1985) in a relational network of agencies (Latour Reference Latour1987). As actors in an ecology of entities and tendencies, we are continually ‘sailing the stormy seas’ as Chadabe would say – adjusting and shifting balance to maintain stability, or conversely stirring dynamic change among the multiplicity of interwoven processes – at multiple scales, modalities and contexts, from molecular to complex technological, social, biological and cybernetic systems (Di Scipio Reference Di Scipio2003, Reference Di Scipio, Toro Pérez, Bennett and Hiekel2024). Each system functions as an organism, an entity/occasion characterised by its vector of affordances, its ‘desire’ to move in one way or another.

An object of any form can be understood as expressive of its unique character and encoded with information (e.g., an instrument, puppet, scenography, software or even a simple piece of thread). Technical objects in particular are designed with an embedding of knowledge and contextual assumptions. Thor Magnusson writes about this epistemic nature of tools:

Thus, composing with intelligent systems involves a curation of entity processes interconnected to form an ecology exhibiting a particular behaviour. The non-neutrality of technical objects highlights the pronounced agency of our tools – entities that interact and collaborate with us, mediating our actions.

5. Mediation, collaboration, latency, fragmentation

We can think of mediation as an intermediary layer that situates an action: the context the action navigates (e.g., instrument, method, process, media) to produce a desired (or accidental) result. Mediation theory is a way of describing the agencies and relationships between entities in an ecosystem, forming into networks, or ‘societies’ in Whitehead’s terminology, which produce an outcome through their collective interaction.

In the context of intermedia, there are three types of mediation which are particularly useful to consider: nodal (agential), temporal and representational.

Nodal, or agential mediation, is the most general form – the mediation of action through an intermediary process. The node is a way of describing the situatedness of each entity as an actor within a network of interconnected agent nodes. A mediating node is one that lies between two other nodes, translating an input action through its own internal process before outputting some other signal. The functional process within the node can be visible, but often is a ‘black box’ that hides its internal logic (Latour Reference Latour1994). Agential mediation captures the agency of the node, determined by the epistemic and aesthetic directions embedded into its nature, its characteristic vector of positive and negative prehension. The influence of nodal agency reveals itself in various ways. For example, the affordances and constraints of a tool, the way the shape of a brush changes what types of textures and ways of moving the brush occur to the artist who holds it in their hand. Or the unique way the artist interprets the situation, based on the historical, social and aesthetic contexts that mediate their engagement with the work.

Agential mediation is particularly pronounced in mechanical and digital systems. For example, when we press the key of a piano or organ, the sound we hear is the product of a mechanically decoupled action. In digital systems, this decoupling is even more extreme, where millions of switches perform abstracted operations – symbolic software logics controlling processes of processes of processes (Kittler Reference Kittler1995).

Temporal mediation describes several forms of latency that are encoded in intermedial experience. There is the physical latency of media traveling through space to reach us, followed by a perceptual latency of the eye moving across the image to take in the local and global forms–while in time-based media morphologies are perceived accumulatively over time. Temporal mediation also describes the latency of production. What we see when we encounter a completed painting is the result of a temporal process of actions, marking and layering over time, and the temporal gap between planning and realisation.

This mediating latency of production is a key challenge in all forms of art. In particular, the process of working with new tools creates a complication in the throughput of our interactions (e.g., in the use of software, digital instrument systems, computer vision algorithms and interfaces for puppet articulation). Until we have gained fluidity with the system through free experimentation – babbling, as Wessel would say – we are not able to fully realise (communicate) our intentions to a satisfying level. The creation process similarly requires coming to know the ecological system, to form a productive relationship with the tools and material – a symbiosis from which emerges a new synthesised entity.

Similarly, in works that require the design of new tools, there is often an oscillation throughout the duration of the creation process between modes of engagement – the most important poles being: a top-down, dramaturgical, macro experience of the piece/system as a whole, versus a bottom-up, micro, detail-oriented process of solving technical problems and low-level operations. When one mode outweighs the other, there is a risk of imbalance in the ecosystem, which may trigger an unintended formation.

For example, in artistic projects that require designing a new tool, the development of the tool may subtly begin to take priority over the original artistic intension that required it in the first place, resulting in a ‘demo aesthetic’. For example, spending weeks programming elegant solutions for new software features, to achieve ‘maximum flexibility and possibilities’, while forgetting (neglecting) that it also takes time to learn the affordances of the instrument, and to compose for it intelligently. Like many, I learned this the hard way. Abstraction from the top-down angle results in a different type of imbalance, for example, cases where artists have very clear conceptual ideas for the meaning of their work but have not spent the time necessary to find effective expressive means for their ideas, which results in a kind of overloading of intention that fails to communicate. Both are examples of what Whitehead refers to as a ‘misplaced concreteness’ that situates thought in a ‘simple location’ without connecting it to the multiplicity of the ecological whole (Whitehead Reference Whitehead1929: 72; Stengers Reference Stengers2014: 83).

Representational mediation is a translation of form through a methodology that mediates presence through abstraction. In the visual arts, the development of linear perspective established a mathematical basis for representing form and projection of light in a homogeneous perspectiva artificialis (Panofsky Reference Panofsky2012), where illusions of three-dimensionality could be systematically represented on a two-dimensional surface. Photographic techniques of capturing spatial scenes through a fixed-point perspective onto a photosensitive substrate is a mechanical example of representational mediation. The later development of moving image techniques created the possibility for the representation of spatial action over time.

Up until the twentieth century, the medium of sound in the Western art context was predominantly focused on tonal and rhythmic structures. While this perhaps could be said to represent movement, dance or other abstract processes, tonal music is generally considered a non-representational medium, experienced through various social contexts, sensory perception, analytical and emotive feeling (Clarke Reference Clarke2005). With the advent of recording technologies, the ability to work with concrete, real-world sounds transformed the relationship between sound and representation.Footnote ³ Like photography, sound recording is an electromechanical technology that mediates the representation of a spatial event. The microphone measures sound pressure waves in the original physical context which is recorded on a substrate (wax, tape, hard drive, etc.) and can then be played back as sound pressure waves via a loudspeaker. Paralleling the development of linear perspective in visual art, sound synthesis technologies created the possibility to render virtual sound scenes utilising techniques of psychoacoustics-informed signal processing, e.g. spatialisation, room impulse responses, artificial reverberation (Barrett Reference Barrett2002, Reference Barrett, Paterson and Lee2021).

In both auditory and visual methods of representation, the physical world is quantified into isolated sensory streams of information, a useful method for analysis and synthetic creation. This analytical isolation is also traceable in the way the specialisation of discourse and techniques in various media has led to a branching of isolated disciplinary kingdoms. From the perspective of intermedial practice, we can observe that the process of analytical isolation, as in temporal mediation, creates the potential for a ‘misplaced concreteness’ between the senses – resulting in unbalanced, fragmentary combinations, where one modality or technique obscures the others. What methods and approaches might help us begin to re-entangle these streams into a living transdisciplinary intermedia? What is intermedial experience?

6. Multimodality, sensory integration, intermedial creatures

Walking down the sidewalk thinking about the nature of sensory streams and intermedia, I pick up a leaf and look down at it, wondering, what does this sound like? I lean in to listen. All I hear is the sound of cars, bicycles, wind in the branches and far away voices. Of course, I think to myself, it has to be activated. I rub my finger along the surface of the leaf and hear the particles of my winter gloves gently scrape against the structure of the leaf’s surface. I press harder, and suddenly the leaf releases a crunching sound as it begins to break apart, partly dried from the cold weather.

I look up from the leaf and around at the sounding scene around me: everything I see has the possibility of making sound, everything physical has the affordance of movement, of being picked up and bowed against another surface, hit with a stick, blown with air. And at the same time, I see these objects, comprised of microscopic intermedial entities, prehensing vectors of sensory information, concrescing into assemblages of experience.

In this way, all physical objects are inherently multimodal, sonic (when activated), visual (when illumined), haptic, olfactory and affective as Whitehead emphasises. This is perhaps the simplest natural expression of intermedia. The ultimate anti-illusionary ‘non-art reality’ described in the Fluxus Manifesto (Maciunas Reference Maciunas1963) is the physical world, where modes of experience are naturally ‘fused’, synthesised as one indivisible expressive entity.

The illusion of medial separation is created by conceptual abstraction and the specialisation of our technical methods – powerful means for working in a specific modality, yet, from an intermedial perspective quickly slips into a fallacy of ‘misplaced concreteness’, pulling the ecosystem out of balance. But when we focus on one node of the sensory network, we are not breaking its intermedial nature, not slicing – we are weighting that node, emphasising it. The network’s centre of gravity shifts to that node, the point of focus, but we cannot truly isolate a single sense.

When we close our eyes, we still receive visual sensory data, a residue of lights fading slowly into darkness with some kind of granular pattern. When we plug our ears with our fingers, we hear the sound of our fingers in our ears. The full system is still engaged, but these alterations create a negative weighting, blocking outside signals, which changes the way we balance our sensory parsing, emphasising senses with the most salient information.

At a museum of visual art, the customary focus is the content, the artworks on the walls, but we are still there, present in the space physically, and still engaging with our full modal spectrum. The architecture, lighting, acoustics, temperature, our personal interests, curation, spatial and social configurations all have an impact on our experience (Krauss Reference Krauss1979; Bishop Reference Bishop2012). Similarly, at a concert we focus on listening; however, we still see. What do we see at an acousmatic concert? Usually a concert hall, an array of speakers, an illuminated exit sign and a person sitting at a computer, a mixing desk, potentially moving sliders or pressing a spacebar. Often they are behind you, so you have to turn your head to see them. As in the art museum, the socio-spatial context situates the experience.

Memory also plays an important role in multisensory integration, for example, watching and listening to traffic outside the window, we can close our eyes and continue to track the sound of cars we saw a moment ago. Similarly, if we plug our ears we can imagine the sound of the cars continuing in our imaginations.

While modern video recorders typically capture audio and video as synchronised sensor data, cinema and acousmatic practices still work heavily with divided sensory layers. In the musique concrète tradition, reduced listening focuses on the morphology of sound as a disembodied object, abstracted from its causal origin (Smalley Reference Smalley1997; Kane Reference Kane2007; Thoresen Reference Thoresen2007). Film production relies on a full isolation of visual and auditory streams, which are merged in the final experience of synchronised presentation (Chion Reference Chion1994). Both practices work deeply with questions of semantic connection and abstracted form.

Early twentieth-century filmmakers like Eisenstein and Pudovkin critically examined the synchronisation of sound and image. Drawing on perceptual theories of Vygotsky and Luria, Eisenstein identified the problem of synchronised dialog as obscuring the perception of the ‘inner speech’ of a character on screen, silently intuited in the audience’s mind, as compared to the direct ‘outer speech’ of synchronised sound, which creates realistic soundscapes and vocal embodiment but produces a less active engagement of the audience’s imagination (Burke Reference Burke2007; Olenina Reference Olenina2021).

Heiner Goebbels similarly discusses a ‘theatre of absence’ which leaves space for the imagination of the audience, by intentionally removing aspects of the scene (sound, images, characters, settings) in order to produce an active creation in the viewer.

The shifting of emphasis in a network to a specific node – or as in ‘absence’, an extreme reduction of nodal weight – creates an active counterbalancing where we instinctively work to fill in the gaps by imagining the missing elements, or subtly repurposing other existing elements in substitution.

In the early stages of development, the latency of artistic practice also produces an absence of the work itself. I have found that one of the most clarifying methods in my practice is to sit in (or study images of) the space where the work will be performed and imagine what will happen, visualising being in the audience and experiencing the work. By connecting to the future ecosystem of the piece, visualisation activates the prehensions of context, the affordances and constraints of the situation, the details spontaneously concresce into life – as if the work is there, drawing itself into existence.

7. Resemblance, prehensions

Material-theatre-based composition is a practice of forming linkages between materials and actions to create coherent intermedial creatures. As in human experience, attributes about the creature are absorbed as sensory and contextual data, which fuse together, synthesising into a perception of the entity/occasion as a unified whole. Towards the development (or discovery) of intermedial creatures, we can think of the process of concrescence through positive prehension as a syntax which binds material together, reinforcing the character of the entity through a resonant semblance between the materials.

In The Order of Things (1970), Michel Foucault details how until the end of the sixteenth century, knowledge was based on a similar principle of ‘resemblance’ which aimed to read the world through locating correspondences.

Foucault goes on to describe how at the beginning of the seventeenth century, Descartes led a critique against resemblance as a ‘confused mixture’, emphasising instead that knowledge should be based on quantitative analysis in terms of ‘identity, difference, measurement and order’ (Foucault Reference Foucault2005: 58). However, the experiential epistemology of resemblance did not fully disappear, but became obscured, for example in poetry, subverting the Cartesian revolution as an ‘allusive […] living being of language’ (Foucault Reference Foucault2005: 48). Thus, with our focus on living beings of intermedia, the methodology of resemblance provides a useful vocabulary for studying qualitative correspondences between materials.

Foucault identifies four main modes of resemblance: (1) convenientia (convenience), a direct connection in space, proximity, conjunction; (2) aemulatio (emulation), like ‘convenience’ but at a distance, a mirroring, the reflection of the moon in water, birds calling to each other from afar, imitation without connection; (3) analogy, a resemblance of character, or essence, a metaphor, with maximum flexibility and room for subtlety in its definition; (4) sympathy, a kind of attraction or affordance that draws things together into a symbiosis, ‘assimilating’ the other (Foucault Reference Foucault2005: 26) as in positive prehension. Like Whitehead, Foucault adds an additional negative prehension of antipathy, which ‘maintains the isolation of things’ (Foucault Reference Foucault2005: 27). Foucault uses the term signature, to name the underlying connective thread deciphered through resemblances.

Whitehead reminds us that this ‘signature’ is not a static object but a ‘nexus’ or ‘society’ of entities/occasions. He defines ‘nexus’ as a set of entities that create a sense of a unity through a ‘relatedness’ emerging through ‘their prehensions of each other’ (Reference Whitehead1978: 24). A ‘society’ is defined as a ‘self-sustaining’ nexus of enties characterised by a principle of ‘order’ (Reference Whitehead1978: 89). As in the epistemology of resemblance, Whitehead emphasises that, a society of entities is more than a simple grouping by type or mathematical order, rather the entities in a society are alike due to their ‘common character’ (ibid.), which we might read as signature of resemblance. Like the vector of relational modes in prehension (emotion, purpose, valuation and causation), resemblances can also be thought of as a vector of different types of relation used to describe a layered multiplicity of correspondances.

From a practitioner perspective, the epistemology of resemblance functions as a qualitative method of producing knowledge through experiential, embodied engagement and interpretation – closely parallelling methodologies of artistic research (Mersch Reference Mersch2015). In addition to psychoacoustics-based gestalt grouping principles, for example, auditory stream segregation (Bregman Reference Bregman1984) and spectromorphology (Smalley Reference Smalley1997), we can draw on the methodology of resemblance to identify modes of linkage between materials that might synthesise into coherent multimodal entities – Whiteheadian societies of entities/occasions acting collectively as creatures of intermedial orchestration.

8. Nexus orchestration, palettes of societies

‘To orchestrate is to create, and this is something which cannot be taught’. Rimsky-Korsakov’s famous phrase (Reference Rimsky-Korsakov1912) gestures towards the inherently experiential, embodied nature of artistic knowledge. This is equally true in the practice of ecological, ‘scenographic composition’. What produces an experience of synthesis or creature-like unity in intermedial orchestration? There is no one-size-fits-all equation.

Classical orchestration manuals illustrate methods for blending, contrasting, highlighting, reinforcing macro articulations of gesture, but as Rimsky-Korsakov points out, these techniques do not teach ‘the art of poetic orchestration’. I would suggest that this poetic sensitivity is rather a product of entering into a collaborative relationship with the materials – to ask, where does the material lead?

The vocabulary of resemblance can help to interpret and articulate the relationships between entities – ideas, media, processes – by describing the modes of counterpoint, imitation, resonances, entrainments between materials integrated in ‘transmodal perception’ (Smalley Reference Smalley2007).

The principles of connection (convenientia) and emulation (aemulatio) are the most directly related to traditional music theory and gestalt principles of onset synchrony and semblant motion (Huron Reference Huron1989) – a description of correspondences between voices in musical time and pitch space.

In composing for human performers, objects and scenographic scenarios, I often focus on the resemblance between sound and gesture as a unified spatial form, through what I have referred to as ‘tactile listening’, a kind of haptic-synaesthesia or feeling of sound texture (Gottfried Reference Gottfried, Toro Pérez, Bennett and Hiekel2024). This mode of listening emphasises the resemblances between physical spatial articulations and the inherent spatial-contrapuntal properties of the sound. While I think I have always listened in this way, I began formally developing the idea in the context of 3D audio research, where I found there was an increased sense of embodiment of the spatial experience when I linked the textural aspects of sound with similar textures of spatial articulation (Carpentier et al. Reference Carpentier, Barrett, Gottfried and Noisternig2016; Gottfried Reference Gottfried2012). This grouping of sound and spatial texture is a kind of resemblance by imitative emulation, which seems closely related to the ‘kiki-bouba effect’ (Alper and Averbuch-Elor Reference Alper and Averbuch-Elor2023), a phenomenon of ‘sound symbolism’ in which qualitative associations have been shown to be widely consistent in cross-modal groupings of objects into groups of kiki (sharp, pointy, rough, short, acidic, dissonant things) and bouba (rounded, smooth, legato, consonant things).

Strategies of grouping in this way can help form a multidimensional nexus of material. Just as a flute tone will form a connective resemblance with cello harmonics due to their similar spectromorphology, a rapidly changing spatialisation approach might correspond well with fast enveloped granular synthesis due to an emulation between the speed or spatial movement and duration of the grains. Through the integration of different ‘societies’ of events, we can create a palette, or qualitative map of entities who share corresponding characteristics within a multidimensional feature space of resemblance (Wessel Reference Wessel1979; Huron Reference Huron1989).

In an ecological system, each creature ‘harmonises’ in its own way with the environment (Clarke Reference Clarke2005). This harmonising can be thought of as a kind of sympathetic vibration, an entrainment of actions that bend and follow other actions in a causal chain – as in the physical processes of instrumental performance. Recalling our discussion of multimodal perception, we might consider the experience of a leaf being crushed by our fingers as a physical vector of sympathies and antipathies, actions and reactions joined in a society of events: the haptic pressing, the visual image, the pressure of sound waves moving to our ears. In the highly mediated audio-visual context of cinema, we can similarly view the representational practice of Foley to be a process of sympathetic resemblance, linking a visual action with an artificially created sound which could have been physically produced by the object on screen.

And finally, the resemblance of analogy is perhaps the most poetic, which like a metaphor, can be evocative and multidirectional – activating connections and meanings uncovered through interpretation. This allegorical, poetic process is the why that guides compositional decisions.