Whilst the engine runs, the people must work:
men, women and children yoked together with iron and steam.
The animal machine… is chained fast to the iron machine.
(J.P. Kay 1832, in Rule 1991).
__
But the miracle of concrete music… is that, in the course of experimentation, things begin to speak by themselves, as if they were bringing a message from a world unknown to us and outside us
(Schaeffer Reference Schaeffer2012: 91–2).
1. Introduction
NEURAL MATERIALS (2024) is a live AV show created by SONAMB, an alias of composer Vicky Clarke.Footnote 1 It utilises a hybrid performance set-up in combination with a sound sculpture and the output of a machine learning (ML) model trained on a ‘post-industrial’ sonic dataset. The dataset renders in sound Manchester’s industrial past and present through field recordings of cotton mills, the canal network and the electromagnetic resonances of a newly gentrified city centre. NEURAL MATERIALS is an ‘ode to [Clarke’s] hometown of Manchester’ (Clarke Reference Clarke2022a). In making field recordings the raw materials for an audio-generative AI model, Clarke’s work calls into question how sound’s indexical properties are altered through use as part of a dataset for ML. She writes: ‘if a machine can generate new sonic materialities, what happens to the authentic matter when processed by a model?’(Clarke Reference Clarke2022b).
As hinted at above, the sonic materials utilised in the work do not only index sources but also histories and geographies too. For Clarke, the recordings are ‘imbued with a sense of place, emotion, and industry’ (Clarke Reference Clarke2021a). The question for NEURAL MATERIALS, then, is: ‘[can] a machine generate a sense of space and heritage?’ (Ibid.). If a sense of heritage is inherent in the sonic materials of the post-industrial dataset, what happens to it when it is processed by an audio-generative AI? And how might this transformation impact our understanding of industrial processes more broadly? More speculatively, Clarke’s work asks how concrete materials – processed via audio-generative neural networks – might ‘project future sonic realities’ (Clarke Reference Clarke2022b).
2. Methodology and materials
This article explores the aforementioned questions through an analysis of NEURAL MATERIALS: as musical composition, live AV show and a demonstration of creative audio-generative AI. Alongside this analysis, the article untangles some of the contextual threads running through the work, as well as demonstrating the Sonic Materialist scholarship echoed in Clarke’s compositional approach and the influence of the work of Pierre Schaeffer. This analysis interrogates both contextual and technical materials and is underpinned by a close listening of the premiere performance to ground claims in specific sonic moments.
For Clarke, NEURAL MATERIALS is tripartite in its musical agenda. The work is at once:
-
1. A love letter made from the materials of a post-industrial city;
-
2. A study of the rhythm and noise of neural synthesis via the sonic language of cotton mill machinery, urban noise and canal network waters; and
-
3. A system for sound sculpture, modular electronics and ML (Clarke Reference Clarke2024).
These three areas – post-industrial urban materials, neural synthesis and performance system – form the structure through which this article is organised.
NEURAL MATERIALS is used as a means of understanding Sonic Materialism as it applies in the age of artificial intelligence. To do so, it draws on three principal sources of evidence: documentation of the première performance, publicly available writings by Clarke, the author’s own interviews with the artist and supplementary materials such as dataset descriptions and system schematics. The analysis makes regular reference to the première performance in 2024, providing evidence in sound of the broader theoretical concepts under discussion. My position as author is that of an external analyst, facilitating a level of critical distance that is nonetheless underpinned by practitioner knowledge gleaned during interviews with the artist. Whilst this article’s emphasis on a single case study is potentially limiting, focusing on an individual work enables detailed examination with a level of specificity that comparative studies might obscure and guards against flattening the work-specific sociocultural contexts that are central to both Clarke’s compositional agenda and the analytical argument evinced here. This article extends existing Sonic Materialist and Schaefferian frameworks by demonstrating how their integration makes audible the tension between context and abstraction, providing a coherent methodological orientation for evaluating NEURAL MATERIALS.
The originality of this article lies in three related contributions. First, it demonstrates how Sonic Materialist and Schaefferian theories of sound might be understood in relation to neural synthesis. Such a demonstration is supported by close listening, grounding theoretical claims in the sonic fabric of the performance. Second, it documents and interprets a hybrid performance system, demonstrating how notions of site, source and abstraction come to bear in the physical performance infrastructure of NEURAL MATERIALS. Third, it advances discussion around AI-facilitated composition by demonstrating how site-specific sonic datasets, when subjected to generative processes, continue to carry sociohistorical resonances. This directly challenges the view of neural synthesis as merely decontextualising material. Together, these contributions further debates in sound studies by offering both a methodology for analysing AI-facilitated composition and a critical framework for assessing how heritage and context are negotiated in and through sound.
3. Sonic materialism
This article takes the work’s titular focus on ‘materials’ as instructive. The turn to materiality in scholarship is well documented (see e.g., Cobussen Reference Cobussen2022; Tang and Cooper Reference Tang and Cooper2024) and has predominantly been characterised by a resistance to anthropocentrism as well as the ‘deconstruction of longstanding binaries between humans and non-humans, culture and nature, and meaning and matter’ (Tang and Cooper Reference Tang and Cooper2024: 3). Central to this body of work is Karen Barad’s notion of ‘intra-action’, a term that frames agency as ‘the iterative reconfiguring of the materiality of human, non-human, cyborgian, and other such forms’ (Barad Reference Barad2007: 178). Within the broad discipline of ‘sound studies’, the legacies of New Materialism are manifold (see e.g., Born and Snape Reference Born, Snape and Born2022; Browning Reference Browning2021). In his 2011 article ‘Beyond Representation and Signification: Toward a Sonic Materialism’, Christoph Cox articulates his frustration with musicological analyses of sonic artworks that ‘fail to capture the nature of the sonic’, proposing by way of an alternative a ‘materialist account able to grasp the nature of sound’ (Cox Reference Cox2011: 145–6).Footnote 2
Like Cox, Salomé Voegelin’s chapter ‘Sonic Materialism: Hearing the Arche-Sonic’ echoes seminal New Materialist scholarship. Her focus on a ‘world of fleeting things… [in which] everything remains fluid and uncertain, not necessarily as precarity, as a state of anxious fragility, but as a serendipitous collaboration between the multiplicities of the “what is”’ speaks to the influence of the concept of ‘intra-action’ (Voegelin Reference Voegelin, Grimshaw, Walther-Hansen and Knakkergaard2019: 561). For Voegelin, ‘a sonic conception and sensibility of the world is the point of access to pure possibility as actuality’, echoing Barad’s contention that ‘the future is radically open at every turn’ (Barad Reference Barad2007: 178; Voegelin Reference Voegelin, Grimshaw, Walther-Hansen and Knakkergaard2019: 561). So too is the influence of Baradian entanglement running through her assertion that Sonic Materialism should start from ‘within the texture of the world, which includes me simultaneously as a thing in the weave of things’ (Voegelin Reference Voegelin, Grimshaw, Walther-Hansen and Knakkergaard2019: 574). Marcel Cobussen also stresses the entanglement of sound, listener and environment, where sound exists within a complex assemblage of historical, social and material factors, evoking for the listener ‘experiences and sensations connected to memories, psycho-acoustic or semantic meaning as well as geographical, biological, or sociocultural contexts’ (Cobussen Reference Cobussen2022: 21).
This article adopts sonic materialist analysis as a way of listening attuned to the historical, geographical and more-than-human actants entangled in Clarke’s work. As an interpretative practice, it listens for indexical cues in sound. These cues are not treated as transparent signifiers of their source, but as signals of the persistence of context, even when neural synthesis threatens to abstract, blur or erase aspects of sonic heritage. Crucially, this approach does not assume that ML models have any capacity to ‘hear’ heritage in the human sense. Rather, it asks how listeners – adopting a sonic materialist stance – interpret the continuities and discontinuities between original recordings and AI-generated outputs and their relationship with contextual resonances. This is achieved through close listening to NEURAL MATERIALS in performance, signalling moments where sonic materials audibly index heritage. Through Voegelin, sonic materialism also offers a way of talking about the generative nature of artificial intelligence models: thinking through the recurrent neural network (RNN) as a ‘point of access to pure possibility as actuality’ (Ibid.: 561).
4. A love letter made from the materials of a post-industrial city
Salomé Voegelin’s concept of the ‘weave’ is important, for NEURAL MATERIALS starts with cotton. The scope of this article necessitates only a brief recounting of the history of the cotton textile industry in Britain.Footnote 3 Principally, this history underpins the field recordings made by Clarke to form a ‘post-industrial dataset’ organised into three categories: cotton, water and noise. While the dataset is not currently publicly available, selected excerpts are documented in Clarke’s project notes and in supplementary materials made available to the author.
4.1. Cotton
Until the eighteenth century, cotton weavers in England typically worked using their own looms.Footnote 4 As early as the 1630s, raw fibre began to be imported from Barbados, with Jamaica and other Caribbean islands supplying cotton after 1660 (Styles Reference Styles2020: 6). This process was part of the so-called ‘triangular trade’ model that trafficked enslaved people from Africa, primarily to work on plantations producing ‘cash crops’. During this period, Lancashire, in North West England, built on its existing heritage of spinning and weaving to become a centre for the burgeoning cotton industry. From the early eighteenth century, this process was increasingly industrialised, owing to a number of inventions including the flying shuttle in 1733, James Hargreaves’ spinning jenny in the mid-1760s and Lancastrian Richard Arkwright’s spinning frame and water frame in 1769. These innovations are not only technological milestones but also sonic ones: the new mechanical periodicities introduced by these devices come to bear in the rhythmic character of NEURAL MATERIALS.
Clarke’s dataset captures this heritage through field recordings of surviving looms at Quarry Bank Mill in Styal, Cheshire. The ‘cotton’ recordings in Clarke’s dataset were made at Quarry Bank Mill in August 2022 using a mixture of directional and stereo microphones to capture the timbres, drones, power-up and power-down sounds and looped rhythms of historical machinery including a Draw Frame, a Slub Roving Machine, a Carding Machine, a Ring Spinning Frame and a Lancashire Loom (Clarke Reference Clarke2024; see Figure 1).
Figure 1. Vicky Clarke recording at Quarry Bank Mill (Photo: Mark Dyer).
The sounds of machinery are particularly noticeable during the opening minutes of the première performance (e.g., 00:41–00:54; 03:52–04:10).Footnote 5 Here, the indexical traces of the loom are clear, with regular clusters of machine sound recalling the rhythms of textile production. Later in the performance, however, the sounds of the loom return in an altered form, more audibly resynthesised via the neural network model – detailed later – into a ghostly recall (07:55–08:09). What was regular has been made unstable, as if heritage is both present and estranged. This double function illustrates how Clarke’s dataset operates as both archive and generative force and is evocative of the fact that whilst the machines are of the industrial era, they are being recorded in the present day, already a resonance of something long past. This is particularly interesting to consider in the context of the neural network, examined in greater detail in Section 6: A Study of the Rhythm and Noise of Neural Synthesis.
4.2. Water
Water was also extremely important to the development of the factory system. First and foremost, it provided a means of power. Whilst the first cotton mill powered by water, Marvel’s Mill, was opened in 1742, it is the second, developed by Richard Arkwright in 1771 in Cromford, Derbyshire, that would be the blueprint for a number of Arkwright-type cotton mills opened in the subsequent decade-and-a-half.Footnote 6 Many of these were built by merchants from or with significant interests in Manchester. Even after industrial decline, the canals remain as traces of Manchester’s post-industrial identity.
Clarke’s dataset documents the persistence of water in this industrial history by capturing sounds from the Bridgewater and Rochdale canals, using hydrophones at different points along the route. The recordings carry geographic and material traces of this heritage, mapping out the route from the countryside towards urban centres. In the première performance, this is most audible in the section beginning at 05:11, where the turbulent sounds of water passing a recording device fill the sonic field. The texture conjures the aquatic without a strong sense of site-specificity, but, taken in combination with the machinery sounds, points towards a broader sonic palette that evokes Manchester’s industrial past. The canals therefore serve not only as historical infrastructure but also carry sonic traces of the past: hydrophone recordings translate the waterways’ industrial role into audible textures.
4.3. Noise
Steam power wouldn’t arrive at Quarry Bank until 1810, nearly 30 years after Richard Arkwright opened Manchester’s first steam-driven cotton mill utilising this technology in Shudehill in 1781. This mill is of historical significance because it was built away from a river, signalling – as Miller and Glithero put it – ‘the birth of the steam-powered textile mill, a ground-breaking development that was fundamental to Manchester’s rapid accession as the most highly concentrated centre of cotton spinning in the world’ (Miller and Glithero Reference Miller and Glithero2017: 98). Many new mills were built in the suburb of Ancoats, producing what has been described as ‘the first industrial landscape based on steam-power’ (Rose et al. Reference Rose, Falconer and Holder2011: 1; in Caignet Reference Caignet2022: 1).Footnote 7
Another significant Manchester redevelopment project in the last 30 years has been Deansgate Square, a cluster of skyscrapers in the city centre. Architecture firm SimpsonHaugh attests that the skyscrapers are ‘located on the site of a former public car park area’, yet an archaeological survey by Salford Archaeology revealed the presence of ‘features from the Roman period’ as well as the remains of eighteenth-and nineteenth-century industrial developments (Miller and Cook Reference Miller and Cook2019: 1; SimpsonHaugh 2024). So whilst Ancoats more visibly accommodates period features, Manchester’s industrial past is always present.
Clarke documents the noise of the electrical age using a custom-built circuit designed to pick up electromagnetic waves. Several electromagnetic field (EMF) recordings were made at Deansgate Square and in Ancoats, treating electrical noise not as unwanted artefact but as integral to Manchester’s post-industrial soundscape. These sounds are present in the première recording as part of the soundscape of the opening section (e.g., 02:09–02:25), as a buzzing, irregular counterpoint to the more obviously metrical machinery.
4.4. Journeying
NEURAL MATERIALS therefore articulates two journeys: the first is through time, following early weavers and spinners out of their homes and into the cacophonous locomotion of the industrial-era factory. The second is cartographic, taking us from Quarry Bank Mill in the outskirts of Manchester, along the arterial canals of Bridgewater and Rochdale, and into the urban centres of Deansgate Square and Ancoats. It is through these two journeys that Clarke ‘tells the story of Manchester’s industrial past and present’ (Clarke Reference Clarke2024). And whilst these times and these spaces prove our primary threads, also inseparably woven into the fabric are the histories of colonial exploitation, industrial revolution, and contemporary gentrification.
5. Concrete music
Interestingly, this was not Clarke’s first attempt at dataset creation for the purposes of ML. Clarke’s AURA MACHINE (2021) is an earlier experiment with neural synthesis. It employs PRiSM Sample RNN, an audio-generative RNN (Melen Reference Melen2020). Much has been written elsewhere about the history of SampleRNN, PRiSM SampleRNN and its implementation in other musical contexts (see e.g., PRiSM 2020; Laidlow, Reference Laidlow2022; Packham Reference Packham2024). AURA MACHINE (2021) presents training data in combination with the AI’s output. The dataset’s recordings were categorised into three ‘distinct classes’: ‘Echoes of Industry’, ‘Age of Electricity’ and ‘Materiality’ (Clarke Reference Clarke2021b). Like NEURAL MATERIALS, the work uses field recordings pertaining broadly to industrial processes and materials, predominantly in and around the city of Manchester, including sounds from Salford’s Islington Mill and Wellington House Mill in Ancoats. When producing the ‘concrete dataset’ for AURA MACHINE, Clarke worked to ‘a set of self-directed rules’ (Ibid.). Three of these are of particular interest:
-
1. Original Recordings: All recordings are of known material origin and self-recorded.
-
2. No Melodic or Forced Rhythm: Concentrate on the texture of the sonic fragments.
-
3. Raw Material: No effects added to the material or overproduction (Clarke Reference Clarke2022a).
The same rules apply to the creation of the dataset for NEURAL MATERIALS. The emphasis on collecting sonic ‘raw material[s]’ is interesting, as the wording calls to mind both industrial production processes – the starting point of most supply chains – as well as the writing of one of the work’s primary influences: Pierre Schaeffer. These rules reflect Schaeffer’s own outlining of the agenda for concrete music in ‘Vers une musique expérimentale’ (see Table 1).
Table 1. Comparison of Clarke’s ‘Building a Concrete Dataset’ with Schaeffer’s ‘Vers une musique expérimentale’
Although Clarke did not frame these rules in explicit dialogue with Pierre Schaeffer, the overlap is striking. Clarke does, however, cite the second journal of Schaeffer’s À la recherche d’une musique concrète (published in English as In Search of a Concrete Music) in a 2022 blog post:
But the miracle of concrete music… is that, in the course of experimentation, things begin to speak by themselves, as if they were bringing a message from a world unknown to us and outside us (Schaeffer Reference Schaeffer2012: 91–2; see also Clarke Reference Clarke2022c).
Schaeffer’s description of things ‘[speaking] by themselves’ is reflective of the development of his notion of the ‘sound fragment’ in the late 1940s (Kane Reference Kane2014: 16).Footnote 8 Concrete music affords recorded fragments ‘the plasticity of compositional material’, delinking the sound from its ‘physical-causal source’ (Ibid.: 16). Clarke’s materials are raw in two senses, then: the absence of post-production effects, but also – following Schaeffer – free of the baggage of their sources. Yet is this really true? For Schaeffer – via Kane – ‘repetition musicalizes the sound fragment by removing the dramatic and anecdotal traces of its original causal context’ (Ibid.: 16; see also Schaeffer Reference Schaeffer2012: 13). Schaeffer’s journal entries from the period he was developing ‘Étude aux chemins de fer’ reflect on the fact that repeating recorded sounds ‘made [him] forget it’s a train’ (Schaeffer Reference Schaeffer2012: 13). But for Clarke, repetition has a less musicalising effect. As their appearance in the première proves, the recordings from Quarry Bank Mill of industrial-era machinery are already very repetitious (see e.g.,13:56–14:15, as well as those examples detailed in Section 4). In fact, whilst the sound is admittedly a byproduct of its primary function, these machines’ capacity to ‘repeat themselves’ was integral to their role as part of the factory system of manufacture. So does the repetition of the Lancashire Loom musicalise or contextualise the sound of its oscillations?
Whilst NEURAL MATERIALS is clearly inspired by concrete music and Schaeffer’s understanding of the sound fragment, to my mind an examination of the particular historical, social, cultural and geographical resonances of Clarke’s ‘raw materials’ – made possible via a Sonic Materialist analysis – demonstrates that the ‘raw’ material isn’t that raw at all. Just as ‘raw’ cotton transported to England carried with it the weight of colonial histories of enslavement, Clarke’s post-industrial dataset arrives laden with sociohistorical baggage. Sonic Materialism’s assertion that ‘sounds will evoke experiences and sensations connected to memories, psycho-acoustic or semantic meaning as well as geographical, biological, or sociocultural contexts’ is at odds with the abstracted musicality Schaeffer perceives in sound fragments (Cobussen Reference Cobussen2022: 21).
6. A study of the rhythm and noise of neural synthesis
Clarke uses the ML model in part to test whether source-bonding can be disrupted by algorithmic processes. According to Clarke, ‘a machine learning model has by default no understanding of context’: so does training an audio-generative neural network on the dataset help sidestep the issue of source-bonding, short-circuiting the associative listening positionality we adopt when we hear the repetitious oscillations of the industrial machinery (Clarke Reference Clarke2022b)?
The post-industrial dataset was first used as training data for PRiSM SampleRNN. Clarke’s interest in ‘project[ing] future sonic realities’ was initially explored through rhythm (Clarke Reference Clarke2022b). The sounds of a Draw Frame, a Slub Roving Machine, a Carding Machine, a Ring Spinning Frame and a Lancashire Loom at Quarry Bank Mill were used to generate samples at various epochs – referring to the number of times the model has trained on the dataset – and temperatures, a hyperparameter determining how likely the model is to generate more ‘random’ outputs (Clarke Reference Clarke2024). More expressly timbral or noisy samples produced by SampleRNN were set aside at this stage; Clarke collected those featuring rhythmic material.
The rhythmic material described above was also converted to MIDI and then compared with the field recordings from Quarry Bank Mill (see Figure 2).
Figure 2. DAW window showing Quarry Bank Mill field recordings, MIDI drum conversion and comparison with post-training rhythms (Clarke Reference Clarke2024).
This neurally synthesised MIDI material is most obviously heard in performance at 02:23–03:35, when a fragmented rhythmic gesture marked by irregular phrase lengths at approximately 88 BPM is introduced. The drum sounds – dry, somewhat metallic, with a short decay – recall the timbres of electronic dance music, whilst the off-kilter groove is closer to the rhythm of a loom with its consistent syncopation and lack of metrical regularity. A similar – albeit more regular – effect is present at 10:06–10:41, where a 134-BPM drum pattern redolent of early dubstep emerges, a reconstitution of a factory rhythm.
The process of comparison – and even composition – began with rhythm. Clarke’s rejection of timbre is reflective of the influence of Schaeffer. Her focus on a conventionally ‘musical’ characteristic helps to delink the sound from its ‘physical-causal source’, shedding some of the factory context of the recording (Kane Reference Kane2014: 16). This is particularly pertinent due to the industrial associations a repetitious sound might bring with it. The ML model has no sense of the context for these sounds – that they are a byproduct of another process – so can only render industrially imperfect repetitions. This dislocation of original context in fact echoes Schaeffer’s thinking around concrete music. However, instead of Schaeffer’s notion of ‘repetition’, here it is neural synthesis that ‘musicalise[s] the sound fragment’, removing some ‘anecdotal traces of its… context’ (Ibid.: 16). Significantly, the process of neural synthesis via PRiSM SampleRNN is an inherently repetitive process – it is a recurrent neural network, after all.
Whilst Schaefferian repetition informs one element of Clarke’s process, she also isolates some of the rhythmic audio generated by the model and employs it as one-shot drum samples in the opening moments of the performance. Clicking sounds at 00:16 and 00:37, a scratchy beep at 00:21 and the machinic rattle at 00:41 are all divorced from their repetitious, rhythmic provenance, serving to undermine the notion that the material’s indexical properties lie only in their metrical character.
Clarke’s original field recordings are first abstracted from their source through their use in a dataset for an ML model. However, it is also abstracted even further through the use of audio-to-MIDI conversion. This bifurcated approach – using the audio generated by the model for its sonic/timbral affordances (such as in the case of the one-shot samples) as well as its particular rhythmic character (such as in the MIDI implementation described above) – speaks to Clarke’s multifaceted compositional agenda. On the one hand, the effort she undertakes to abstract the recordings from their context speaks to the influence of concrete music on her practice. Yet her interest in retaining some of the original context for the recordings means that Clarke is not content to abstract these sounds from their industrial origin. By allowing traces of the original context to persist, Clarke demonstrates that neural synthesis cannot fully detach sound from its material origins. Instead, it makes those entanglements audibly unstable. This instability reorients Schaeffer’s notion of the sound fragment for the age of artificial intelligence: where context reasserts itself through algorithmic repetition.
7. A system for sound sculpture, modular electronics and ML
One of the most visually striking elements of Clarke’s hybrid performance set-up is the sound sculpture (see Figure 3). The design is adapted from a ‘visual symbolic language’ that ‘represent[s] the sound object in latent space’, the output of a workflow first developed as part of the AURA MACHINE project (Clarke Reference Clarke2024).
Figure 3. Aluminium sound sculpture for NEURAL MATERIALS (Clarke Reference Clarke2024).
To create the sculpture, Clarke used the style-based generative adversarial network StyleGAN2, training a model on a dataset of ancient alchemical electrical circuits, technical media drawings found in the National Science & Media Museum’s collections and ‘techno-mystical symbols’ (Clarke pers. comm. 12 December 2023; Clarke Reference Clarke2022d). These images were digitised and ‘graphically developed as symbols’ in Adobe Illustrator before being rendered in Blender as 3D objects (Ibid.). Photographs of these 3D digital objects were then used as the dataset for the generative adversarial network, after which individual icons of interest were considered and developed using a combination of StyleGAN2 and Blender to generate a 3D object that could be fabricated, first using a Prusa Mini 3D printer and latterly via manufacturer Flux Metal. This signalled a shift from virtual to concrete materialities.Footnote 9 Having manufactured the sound sculpture in aluminium, Clarke began the process of developing the hybrid modular system. The sound sculpture functions as part of the performance system in two ways: as a means of generating pitch material and as a resonant body and interface for the playback of AI-generated samples.
7.1. Pitch
The sound sculpture is used as a resonant body through which pitches used in NEURAL MATERIALS are derived. Clarke first mapped the frequency response of the sculpture when striking different points on the sculpture with a beater. She used contact mics connected to two Eurorack modular units: the Ears module by Mutable Instruments – which amplifies the input – and the DATA module by Mordax Systems, featuring a single-FFT Spectral Analyser and multi-FFT Spectograph. The fundamental frequencies of each plane were then mapped onto a 3D render of the icon; Clarke noticed harmonics recurring at multiple points across the sculpture (Figure 4).
Figure 4. Resonant frequencies mapped onto 3D render for AURA MACHINE sound sculpture (Clarke Reference Clarke2024).
This ‘process of extraction’ results in a series of pitches in hertz, which Clarke then quantised to the nearest 12-TET note, creating a set of MIDI pitches that could be used to control virtual instruments within Ableton, as well as converted to CV (control voltage) to use as part of the modular system (Clarke pers. comm. 12 December 2023). This process is most clearly heard at 12:57–17:27, where synthesised strings realise the set of chords generated by the process above. The overwhelming character is unusually consonant: where neural processes have in audio-generative contexts generated more noise or rhythmic instability, the resonant frequency mapping process – owing in part to Clarke’s quantisation to 12-TET – results in a relatively stable, coherent harmonic grammar.
Clarke’s harmonic workflow is in some respect similar to her approach to rhythmic material. Rather than extracting musical parameters directly from the original samples through a more straightforward methodology (e.g., spectral analysis), Clarke stays true to her ‘No Melodic or Forced Rhythm’ rule and develops both pitch and rhythm material via a workflow incorporating both the materiality of industry and ML (Clarke Reference Clarke2022a). She sees the musical building blocks of NEURAL MATERIALS as a consequence of the materiality of the object – in the case of pitch – or as a byproduct of an industrial process, in the case of rhythm. In both instances, ML helps obfuscate the contextual associations of the source. In this respect, the rhythmic and harmonic workflows echo Schaeffer’s thinking around concrete music, where conventional musical parameters come about via material provenance. He writes:
In all this wooden and tin junk and in my bicycle horns I rediscover my violin, my voice. I am seeking direct contact with sound material, without any electrons getting in the way (Schaeffer Reference Schaeffer2012: 7).
7.2. Sample playback
The second way the sound sculpture functions in Clarke’s performance system is as a tactile means of playing back samples generated by PRiSM SampleRNN. This was facilitated via Clarke’s collaboration with Bela, a commercial startup developed out of the Augmented Instruments Laboratory, directed by Professor Andrew McPherson and affiliated with both the Dyson School of Design Engineering at Imperial College London and the Centre for Digital Music at Queen Mary, University of London.
Clarke collaborated with Chris Ball and Luke Dobbin on a PureData patch hosted by Bela’s PEPPER, a passive breakout interface for Bela. The PureData patch is not currently publicly available, but the flow of signal is visible from Figure 5. The PEPPER module formed part of Clarke’s Eurorack system and was combined with two TRILL capacitive touch sensors – one square and one ring – attached to the sculpture that allow for tactile playback and manipulation of both raw and AI-generated samples (Clarke Reference Clarke2024). The square sensor maps the volume and speed of the sample to the X and Y axes, respectively, whilst the circular sensor controls a granulated loop (Ibid.). Clarke can be heard manipulating the playback of a sample at 01:25–01:40, where a loop is subjected to a ‘tape stop’ effect through tactile engagement with the sculpture. The playback system is perhaps most evident in performance at 13:48–15:00, where she uses both the square and circular sensors to manipulate the speed and volume of two AI-generated samples of machinery, connecting the neural and concrete materiality at play in the performance infrastructure. These examples emphasise the system’s design focus on live improvisation, as the TRILL sensors function more like an expressive interface for performance rather than a precise means of manipulating samples. Seeing Clarke manipulate samples in real time through close tactile engagement with the sound sculpture, one is reminded of industrialist J.P. Kay’s description of a nineteenth-century factory worker, ‘yoked’ to the machine.
Figure 5. PureData patch by Ball, Dobbin and Clarke (Clarke Reference Clarke2024).
Clarke’s integrated musical performance system emphasises performative flexibility, whilst the tactile method of playing back samples emphasises the materiality of the sound sculpture, making sense of the sensors’ placement. The system also uses a Dayton Audio transducer attached to the sound sculpture, to create a feedback system whereby the sculpture is both the interface and an additional resonating chamber for playback of the samples.
Finally, the performance system utilises an EMF circuit first created for AURA MACHINE (see Figure 6a) in combination with a neon sign – based on Clarke’s set of ‘techno-mystical symbols’ – fabricated by Neon Creations (Figure 6b) (Clarke Reference Clarke2022d). Like the transducers in the sound sculpture, the EMF circuit ‘uses stereo inductors to amplify the electromagnetic fields’, so Clarke ‘can amplify and listen to the sound of the modular instrument itself, creating a live looped feedback system’ (Clarke pers. comm. 10 May 2024). The EMF circuit takes centre stage in the première performance at two moments, at 05:54–07:20 and at 11:30–12:24, where Clarke amplifies the sound of the modular system and the neon sign to create distorted gestural sweeps that reverberate against the noise field.
Figures 6. (a,b) Custom EMF circuit (left) and performer set-up featuring neon sign (right) used in NEURAL MATERIALS.
The performance infrastructure as a whole is illustrated in Figure 7, tracing the flow of data through the system.
Figure 7. NEURAL MATERIALS routing diagram (Clarke pers. comm. 12 May 2024).
7.3. A performance system
Clarke’s bipartite workflow is an attempt at maximising the ‘acoustic potential’ of the StyleGAN2-produced icons, which become ‘transmutational objects’ as sculptures that can feed back into the performance system both as resonant bodies and as tactile objects (Clarke pers. comm. 12 December 2023). The rendering of neurally generated materials in the real world begins to point towards Voegelin’s ‘pure possibility as actuality’: fabricated masses of steel and aluminium that are the provenance of so much musical material in AURA MACHINE and NEURAL MATERIALS (Voegelin Reference Voegelin, Grimshaw, Walther-Hansen and Knakkergaard2019: 561).
More broadly, the icons connect the notion of the ‘sound fragment’ from theories of concrete music with the ‘equally [decontextualised] datapoint in machine learning’ (Clarke pers. comm. 12 December 2023). The icons intend to evoke the real technical media drawings that in part constitute the dataset, but at an abstracted distance conjured via ML.
Structurally speaking, the work is organised into four principal sections, each with its own sonic character and particular use of technology (see Table 2). The composition is imagined as a system for performance rather than a series of sections and/or events, so other iterations of the work will differ.
Table 2. Structure of NEURAL MATERIALS (Clarke pers. comm. 12 May 2024)
Clarke’s form – and especially through its evocative titling – speaks to an industrial operative structure, in which a machine is brought slowly into action, finds a repetitive working rhythm through the playback of both original and AI-generated samples, and then ‘powers down’. Interestingly, the samples – perhaps the sounds in the piece with the greatest referential weight – only appear in the more ‘ambient’ opening and closing sections. In the central two sections, the ‘machine itself’ takes over, using the abstracted data as CV input for the modular rather than relying on the actual or AI-generated industrial samples to signal a sense of industry. Through a structure which incorporates AI-and non-AI-generated materials in a fluid and improvisational manner, Clarke articulates an aural and affective link between the real recordings of industrial-era machinery and both the AI-generated samples and their abstracted rhythms. The formal connections between concrete and neural materials articulated in Table 2 become an associative link through performance. In this sense, Clarke is in touch with both Schaeffer’s ‘world unknown to us or outside us’ of abstracted musicality and also the richly context-and reference-laden materials collected as part of the post-industrial dataset (Schaeffer Reference Schaeffer1977: 100–1, transl. in Palombini Reference Palombini1993: 551–2).
8. Discussion
The preceding analysis demonstrates that NEURAL MATERIALS emerges through the triangulation of three distinct sets of materials: the historical (post-industrial dataset), the machinic (SampleRNN) and the performative (hybrid system). Listening across layers reveals repeated negotiation between preservation and abstraction – but how does this impact our understanding of the broader questions established at the outset of this article? Clarke asked: ‘[can] a machine generate a sense of space and heritage’ (Clarke Reference Clarke2021a)? The compositional approach evinced in NEURAL MATERIALS demonstrates that the referential affordances of the field recordings that constitute the post-industrial dataset are just one element of their utility as musical materials. So perhaps it is not the case that audio-generative AI can generate a sense of industrial space and/or heritage but rather serve to index wider contexts that through a Sonic Materialist listening positionality intensify the samples’ affective and associative function. Ultimately, Clarke’s post-industrial materials are reconfigured rather than erased. This is in part achieved through the use of a small, site-and heritage-specific dataset, employing ML techniques as a critical means of sonically re-indexing historical phenomena rather than generating radically new materials. Where Schaeffer used repetition to erase physical-causal context, Clarke’s repetitious network reactivates context: the loom’s persistent rhythm, the canal’s turbulence and the city’s electromagnetic noise are exemplary of entanglement, of a ‘weave’. The influence of Schaeffer is also relevant more broadly. In some respects, the history of cotton production – the shift from an individually labour-intensive domestic system of spinners and weavers towards industrialised mills – finds an analogue in the move from time-consuming tape-based methods of Schaeffer and the Groupe de recherches musicales (GRM) towards the AI-assisted sound fragment generation practised by Clarke.
The dual analytic lens developed here – combining Sonic Materialism with Schaeffer’s theory of the sonic fragment – proves effective in examining AI-based sound art because it bridges the theoretical and the sonic. The combination of close listening analysis with critical contextualisation offers a transferable model for analysing sonic artworks grounded in material heritage. Where Schaeffer emphasises the morphology of sound, Sonic Materialism insists on sound’s inseparability from context. Clarke’s practice forces these positions into dialogue, rendering this tension analytically audible. This article therefore contributes to debates on sound’s indexical properties, showing through close listening that neural abstraction need not erase contextual traces but might instead reconfigure them. While grounded in a single case study, this analysis suggests broader implications for AI-based creative practice. In particular, Clarke’s use of a small, site-specific dataset models an alternative to large-scale training datasets, acting as a form of resistance to dominant discourses and in the spirit of ‘data feminism’ (D’Ignazio and Klein Reference D’Ignazio and Klein2020).
NEURAL MATERIALS therefore operates both as an artwork in and of itself (a ‘thing’) and as a model for thinking about how place, heritage, sound and ML intersect (‘in the weave of things’) (Voegelin Reference Voegelin, Grimshaw, Walther-Hansen and Knakkergaard2019: 574). This combined methodological and listening approach could be replicated across other AI-facilitated works to evaluate how context persists after neural abstraction.
9. Conclusion
In Ted Chiang’s historical science fiction story ‘Seventy-Two Letters’, Victorian-era scientist Robert Stratton reflects on the role newly developed automata – golem-like clay figures animated through Kabbalistic nomenclature with the capacity to craft engines – might have on the textile industry. It is slowly revealed that Stratton’s interlocutor is uninterested in his proposal for a more equitable future for the cotton industry, using technology in service of a eugenicist plot that takes further power from the working poor instead of allowing them to create and own the means of production. Chiang’s ‘automatous engine’ makes for an interesting comparison with the ML systems in play here. Each possesses the capacity to both address and intensify inequity in their respective contexts (Chiang Reference Chiang2020: 186–7).
One form of inequity that Clarke is particularly sensitive to is the historical and contemporary male-dominatedness of the related fields of electronic music and artificial intelligence. I want to end by noting the more explicitly activist elements of her broader practice. As part of her work on AURA MACHINE in 2022, Clarke led a workshop with young women and people of marginalised genders in collaboration as part of a residency with Brighter Sound, an organisation committed to equity and diversity in music development which works often with those facing barriers to entry. By emphasising the need for accessibility in ML, Clarke demonstrates her belief in ML as a tool for the many. She asks: ‘what is the potential of creative AI in exploring alternative worlds and narratives (Clarke Reference Clarke2022e)?’ Where Chiang’s antagonists saw the capacity for a less fair tomorrow, Clarke’s ‘radically open’ future sees more equitable access to AI and its affordances within creative music practice (Barad Reference Barad2007: 178). Perhaps this is the ‘future sonic [reality]’ Clarke is ultimately most interested in projecting (Clarke Reference Clarke2022b).
Data availability statement
The data used in this research are not publicly available as participants have not consented to public sharing of these materials. Selected documentation materials are available on request to researchers.
Funding statement
This research was undertaken as part of Cyborg Soloists, supported by a UKRI Future Leaders Fellowship and Royal Holloway, University of London (grant number MR/T043059/1).
Open access
