1. Setting an aesthetic scene: data-based and data-evoking art

This practice-based research work focuses on the creation of audiovisual compositions whose materials are processed or transformed based on the training processing data of certain artificial neural network algorithms (Pocquet Reference Pocquet2023, Reference Pocquet2024, Reference Pocquet2025). These pieces are examples of ‘information art’ (Wilson Reference Wilson2002) (or ‘data art’), framing numerical information produced by NNs (particularly Generative Adversarial Networks (GANs)) during their training stage. One question that has seemed pressing throughout work on this project is ‘What makes something information art?’. Is it the real use of data to produce it, or the mere evocation for its audience of some sort of ‘data-related’ mental image, from the use of sounds, visuals, symbols, etc. that have over time become associated with such an idea? One might argue that more than being data-based, it is whether a work is data-evoking that will have more impact on how it is perceived.

Really, all art we experience is information, apprehended by our senses, encoded as neuronal signals and interpreted by our brains. But such a complex level of information gathering and interpretation is hardly automatically described as ‘information art’. For this title to apply, one needs to look at works that frame artificial, human-made systems encoding information, chief among which is digital data. These various forms of information storage or transmission are then usually interpreted, but for a work to be perceived as information art, it often needs to skip this interpretation step and present untreated information. Hence, more than humanly intelligible forms of information (words, symbols, images, sounds, etc.), data-evoking art might focus more on the abstracted forms it is stored in (binary information, hexadecimal memory addresses, bar codes, punched cards, etc.).

In data-evoking music, the sounds used to represent this ‘data’ world are rarely literal, instead, as in the rest of electronic music, ‘the relationship between signifier and signified is not based on simple resemblance but rather on conventions that over time have paired a sound with an exterior concept’. (Demers Reference Demers2010: 45). This association of a sound with an imagined source, and the advent of music evoking imagined but specific media, might be born of electronic music’s need to rematerialise sound (Eno in Weium and Boon Reference Weium and Boon2013). From there, the sounds used to represent immaterial media (like data) become a sort of transfictionality; a ‘phenomenon by which at least two works refer jointly to the same fiction’ (Saint-Gelais, Reference Saint-Gelais2011: 7).

The post-digital aesthetic (Cascone Reference Cascone2000), perhaps the current that the audiovisual works of this project are most indebted to, is one that has developed from such heavily codified, transfictional set of associations between sound and idea. This tradition, that ‘developed in part as a result of the immersive experience of working in environments suffused with digital technology’ (Cascone Reference Cascone2000: 12), uses sounds that go beyond the literal, and that have become tightly associated with the inherently immaterial idea of the digital world.

The notion of the immaterial is worth commenting on briefly, as some argue that even with digital information, ‘algorithms are themselves local, for they are created and applied within particular historical and material conditions. For example, data based on the WSJ provide a basis for NLTK’ (Loukissas Reference Loukissas2019: 115). In the case of this research, however, the data used are arbitrarily defined mathematical information or ubiquitous training sets whose individual elements hold too little information to be attached to any material locality. Furthermore, the training information is abstracted and transformed from random numbers, making the claim for locality and materiality even more precarious.

The fact that the post-digital tradition ‘takes digital to be the pinnacle of audio fidelity (technology driven by the ultimate desire for perfect clarity and the elimination of noise) and situates itself after: after the revolution, amidst the rediscovery of noise through digital malfunction’ (Haworth in Goddard et al. Reference Goddard, Halligan and Spelman2013: 187) is crucial to understanding the sounds it has adopted; the constant juxtaposition of clinical sound edits and so-called ‘glitch’ textures: white noise, digital clicks, sine beeps, etc. Casconne describes that ‘There are many types of digital audio “failure.” Sometimes, it results in horrible noise, while other times it can produce wondrous tapestries of sound’ (Cascone Reference Cascone2000: 13). However, in many cases, real digital errors result in a corrupted file, and silence, rather than any sound at all.

In fact, paradoxically, a lot of the sounds used by these traditions are more to do with analogue issues (white noise from interference in an analogue signal, raw cuts and bursts from sudden signal changes in bad connections, induced clicks from neighbouring periodic-circuit appliances etc.), and even when mixed with digital artefacts (such as digital clicks due to sudden gain changes), these are only very specific, and ironically highly curated, digital ‘errors’. Similarly, the structure of these works is often extremely precisely worked out, which goes against the supposed ‘malfunctions’ it represents. This further supports the idea of transfictionality in this aesthetic, of using sounds arbitrarily associated with, but not literally linked to, digital errors or a ‘post-digital’ paradigm, demonstrating an interest in staged errors more than real ones.

Nonetheless, while the claim that these sounds are the sounds of errors is fragile at best if looked at empirically, their association with the concept of errors after the fact is just as, if not more, interesting artistically. Although I insist on being aware of its fictional nature, the contradiction of highly unpredictable information and its highly curated representations form a kind of paradoxical semantic starting point which remains highly engaging to me as an artist. Therefore, I have decided to make use of the same sounds, codes, and transfiction built by this aesthetic.

Moreover, in many data-driven pieces, there seems to be a need for aural representations of parsing, scanning, filtering, grouping, ordering, etc., all concepts we associate with data-handling. This can take the form of sounds associated with barcode scanners (Ikeda Reference Ikeda2008, Reference Ikeda2005); hard drives’ read-and-write head (Panacea 2000); electromagnetic glitch/clicks/interference (Ikeda Reference Ikeda1996; SND 2010); a human voice reading out literal data/numerical information (alva noto 2008); visual/semantic representations of data like barcodes (Ikeda Reference Ikeda2005); binary numbering (Ikeda Reference Ikeda2008); references to coding (SND 2000); mathematical concepts (Ikeda Reference Ikeda1996); or a self-referential focus (the music itself as the data) (SND 2000; Cyclo 2001; Cascone et al. Reference Cascone and Kahn2004; Goem 2013). This need for visual and aural representations of immaterial processes is something the pieces made in this research also attempt to address, through their iterative movements (representing consecutive runs and the constant rewriting of weight values); their discrete structures (caused by individual epochs being distinct points, not a continuous transition) and minimal colour palettes (representing raw numerical information with fidelity and lack of embellishment).

Beyond these audio and visual codes, these works also build upon wider aesthetic movements that have accepted that ‘Database and narrative are natural enemies’ (Manovich Reference Manovich2002: 225) and decided to embrace the abstraction of the very data they present in all its unintelligibility, rather than offer a clear interpretation of it, seeing this mass of information as something to be experienced, rather than understood: ‘these data present themselves as a canvas for the contemporary artist’. (Lynch and Paradiso Reference Lynch and Paradiso2016: 142). Generally, the fact that ‘many new media objects do not tell stories; they don’t have a beginning or end; in fact, they don’t have any development, thematically, formally or otherwise’ (Manovich in Vesna Reference Vesna2007: 39) offers a starting point for art that often questions the medium more than the message. In the compositions from this research, the presentation of vast volumes of simultaneous visual data streams that are inherently unintelligible offers an effective representation of the sheer scale of these models and their rate of data-parsing, away from a traditional narrative structure.

Bringing the idea of unparsable, unintelligible information to an extreme brings us to another key idea that these works explore: that ‘the point of art is detritus’ (Weiner Reference Weiner2012). This links back to ‘post-digital’ and ‘glitch’ aesthetics, and the question of waste as material, but this time in a very literal sense. Rather than gradually create pseudo-ecological links between an idea of digital errors and specific sound textures, these works frame data that is normally discarded: training results, suboptimal runs and intentional push towards inefficiency, as close as one can get to literal digital detritus. As Alonso Trillo and Poliks noted, there is ‘tremendous discursive fecundity found in digital goo’ (Alonso et al. Reference Alonso Trillo and Poliks2023: 394). Their research, which detailed the creation of a machine learning tool exploring waste as ‘endless sound generation (waste as overflow), the introduction of waste into that process as part of the involved down- and up-scaling of the audio files (e.g., as processing artefacts – glitch as waste), waste as discarded material, and so on’ (Alonso et al. Reference Alonso Trillo and Poliks2023: 393), shared this intention of making use of whatever digital information is usually discarded or ignored. Rethinking how waste can become music, and how any art engages with detritus.

Perhaps this ties in with an argument that sonification pieces often focus on the grandiose aspects, and often the breakdown of the data they look at, that ‘the threatening or destructive aspects of what is being sonified are often emphasised rather than downplayed; the goal is not to convey a sense of beauty, but one of awe, enthralment and terror’ (Supper Reference Supper2014: 46). Although I find this analysis particularly extreme, perhaps it holds some truth as to why focusing on the failures of the data I look at might be interesting.

2. What data? and what do you do with it?

Alonso Trillo and Poliks’ work also serves as a good introduction to how I decided to approach the making of musical works from this mass of discarded digital information, as a ‘human-in-the-loop reworking of discarded materials ‘incidentally’ produced in the synthesis of something else’ (Alonso et al. Reference Alonso Trillo and Poliks2023: 393); it is not about the product, but the process.

Moreover, when I started working on this project, I also had the naïve hope they seem to have had of making sense of neural network’s inner processes; the ‘assumption that there are nascent musicalities implicit in machine learning architectures’. (Alonso et al. Reference Alonso Trillo and Poliks2022: 504). I believe this excitement in the face of algorithms so notoriously unintelligible is natural, but I have concluded that their scale of operation (in time and in volume of information) is so different from music or any humanly parsable structure that once you open the black box, its contents remain incomprehensible. I instead focused on presenting this incomprehensibility as itself worthy of interest.

2.1. Data

With the ideas of waste, process and information abundance at the fore to drive the works’ aesthetics, I had to decide on which specific data to use and frame to best reflect these priorities. In the works I am presenting, I worked with various kinds of data from NNs, of various abstraction levels, experimenting between literal and still highly interpreted.

The first level is neurone activations and weight values; the most abstracted type of data, looking at the actual changes in neurones inside the network architecture, the most literal ‘opening of the black box’ possible and therefore the least intelligible. (And the last I worked with because of this.) I used this in Reiterate something so many times that it changes (Pocquet Reference Pocquet2025), which showed various runs of a single feed-forward network, trained on non-linearly separatable data (a foundational problem of machine learning), to display how completely different the weight structures can look on the same network, trained on the same data, arriving at the same result, from one training run to the other (see Figure 1).

Figure 1.

Two weight structures after two different training runs of the same feed-forward network trained on the same set of non-linearly separatable data (from Reiterate something so many times that it changes).

Figure 1. Long description

The image displays a network structure with clusters of data points connected by lines. The clusters are scattered across the image, with some points highlighted in orange, indicating a specific path or connection. The lines between the points form a circular pattern, suggesting a cyclical or iterative process. The overall layout suggests a visual representation of data analysis or machine learning processes, where data points are grouped and connected based on their relationships.

Second was in-training outputs, samples produced by networks during training, before they reached any desired level of accuracy. This was done in Noise through to twos and sevens (Pocquet Reference Pocquet2024), looking at a GAN generating MNIST images (see Figure 2), and Spin, split spread, splatter (in eighths) (Pocquet Reference Pocquet2023), looking at dot distributions onto eight Gaussian distributions, mimicking the placement of the eight speakers the sound was mapped onto (see Figure 3).

Figure 2.

Two ensembles of 3600 output samples from a GAN trained on the MNIST dataset, in untrained and uniform (top) and trained and generalised (bottom) forms. (From Noise through to twos and sevens).

Figure 2. Long description

The image displays two ensembles of 3600 output samples generated by a Generative Adversarial Network (GAN) trained on the MNIST dataset. The top section shows untrained and uniform samples, while the bottom section presents trained and generalized samples. The samples transition from random noise to recognizable digits, primarily twos and sevens.

Figure 3.

Two in-training output samples of a GAN trained on an eight-zone point distribution, in mode collapse (left) and successfully trained (right) states. (from Spin, split, spread, splatter (in eighths)).

Figure 3. Long description

The image contains two diagrams representing the output samples of a Generative Adversarial Network (GAN) trained on an eight-zone point distribution. The left diagram shows the GAN in a mode collapse state, where the network fails to generate diverse outputs and collapses into producing limited variations. The right diagram illustrates a successfully trained GAN, demonstrating a more diverse and accurate distribution of points across the eight zones. Each diagram consists of interconnected nodes and lines, visualizing the network’s structure and the relationships between different points. The diagrams highlight the differences in performance and output quality between the two training states.

The third were standard performance metrics (loss, accuracy, etc.) in Noise through to twos and sevens and Spin, split spread, splatter (in eighths): standard metrics used to assess how well a network is performing, which were mapped onto various music parameters, leading to more noise and jittery loops when the outputs were noisier, and more pitch and stable lines when the images became clearer.

In Noise through to twos and sevens, I also used synthetic metrics (entropy values, signal to noise ratio, edge intensity, etc.) worked out manually from the produced samples to drive more musical parameters, again reflecting the gradual move from chaos to order.

In all the choices of data I made, my aim was to leave the processes in the fore, whether or not they are intelligible. This resonates with the call to ‘recognise the cultural status and agency of algorithms (and thus neural networks) themselves’ (Dyer Reference Dyer2022: 224); more than things I comment on, they become an inherent part of the work through their specific idiosyncrasies. This links to a wider shift in electronic music generally, ‘from the creation of novel sounds to the manipulation of sound materials inherent in a culture of electric and electronic devices of sound production’ (Paiuk Reference Paiuk2013: 306).

Although none of the NNs I used output audio, their impact is just as identifiable as when using any compositional tool. If Charrieras & Mouillot have argued that the ‘strata of machines identified as computers need to be thought of as instruments within music environments’ (Charrieras and Mouillot Reference Charrieras and Mouillot2015: 191), I would extend this call to non-musical algorithms used as musical agents. Just as Tudor’s early repurposing of neural networks for non-generative music-making (Tudor Reference Tudor1995), this work is about finding new creative potential in tools not at all aimed at creative purposes. This ties back to motivations of the post-digital aesthetic described above, where ‘the tools themselves have become the instruments, and the resulting sound is born of their use in ways unintended by their designer. […] The medium is no longer the message: the tool has become the message’. (Cascone Reference Cascone2000: 16–17).

Faithful to this exploration of the tool rather than its aim, in Reiterate something so many times that it changes, I do not even show what the network is producing. The algorithm, the abstract movement between epochs, and the evolution of something so removed from any human logic are the focus. In the other two works, the focus is on every individual epoch of training before the output could ever be described as satisfactory, on iterative procedural change, on imprecisions and once again on detritius.

Regardless of ‘how’ a neural network works, presenting its inner workings will lead the audience to interpret it in some way, no matter how senseless or unparsable the data. Even in the mapping of highly complex data, ‘the resulting music doesn’t sound particularly random. This is due in part to the large amount of compositional effort directed toward polishing the musical surface materials. Another factor is apophenia – the human tendency to see patterns in random or meaningless data’. (Lyon in Goddard et al. Reference Goddard, Halligan and Spelman2013: 231). Any interpretation is probably as artistically interesting as the ‘right’ answer, which, in this case, is senseless to humans anyway.

2.2. Sonification mapping

Once the idea of walking the rope between a coherent summary and the presentation of a more chaotic assemblage was established, I worked on the mapping of this data onto sound. The three works exemplify quite different mapping strategies. The first work uses a complex parameter-mapping sonification approach, one which ‘represents changes in some data dimension with changes in an acoustic dimension to produce a sonification’ (Nees & Walker in Neuhoff, Reference Nees, Walker, Hermann, Hunt and Neuhoff2011). In this case, the data represented are secondary metrics derived from output samples, defined and calculated after their generation, and the sound parameters impact multiple audio effects used in parallel. These included two granular synthesis engines using organ sounds, organ loops with varying levels of distortion, digital crackle of varying volume and samples of click sounds organised in corpus form. For a detailed rendition of the mapping, see Figure 4.

Figure 4.

Mapping of GAN parameters and synthetic metrics to audio parameters in Noise through to twos and sevens.

Figure 4. Long description

The diagram illustrates the relationship between GAN parameters and synthetic audio metrics. It includes components such as real images, a generator, a discriminator, and a decision point. The generator creates images that are evaluated by the discriminator, which then makes a decision. The diagram also shows various functions like calculate_snr, calculate_ssim, calculate_entropy, calculate_edge_intensity, and calculate_normalized_contrast, each contributing to different audio parameters such as volume, length of loop playback window, bitcrush sampling ratio and bit resolution, and digital crackle.

If mapping is the ‘designed connection between signals and parameters that creates the interdependencies between otherwise independent parts, making them into a whole’ (West et al. Reference West, Caramiaux and Wanderley2020), then in this case, the whole contains so many strands that they might not be identifiable indivisually, but instead compound to create a general sound movement to represent the data’s overall evolution. Rather than sensing a change of entropy in the outputs cause an immediate change in bitcrush intensity on organ loops, the hope is that all parallel metrics used create a general movement from noise to order and that the number of independent but interconnected data streams cause a richness and complexity in the sound output that would be harder to realise if mapping a single changing value.

If ‘gesture-sound relationships which comprise a mapping strategy are arbitrarily assigned by the designer based on their aesthetic values or practical needs’ (de las Pozas Reference de las Pozas2020), and therefore inherently subjective, there still ‘seems to be some agreement among listeners about what sound attributes are good (or poor) at representing particular data dimensions’ (Nees & Walker in Hermann, Hunt & Neuhoff, Reference Nees, Walker, Hermann, Hunt and Neuhoff2011). For instance, pitch is generally good for representing temperature, whereas tempo is not as effective (Walker Reference Walker2002). However, this would hold true for data dimensions that are already understood by the listener, to have been able to develop such an expectation. In the case of the project, a lot of the data mapped is too abstract for this to be the case.

Yet changes in the various metrics calculated for this piece generally correlate with an overall evolution of the output data from noisier to clearer. The sound elements used similarly display changes between a clean signal and a distorted/processed one. The aim is for this noise to order parallel to be communicated to the listener, again rather than individual data strands’ evolution. Of course, scaling and polarity decisions still had to be made to ensure datapoints that primarily ranged between 0 and 1 could be applied to sound parameters (time lengths, distortion levels, volumes) of differing scales, as it has been shown that ‘important data characteristics (function slope, shape, and level) were perceptually salient’ (Flowers and Hauer Reference Flowers and Hauer1995: 553) in sonifications. Overall, I have worked to ‘maintain a balance in the relationship between [data] and resultant sound that is easy to perceive for the audience’ (Bencina et al. Reference Bencina, Wilde and Langley2008), even if in this case a lot of individual data changes are not directly perceivable. If ‘direct mappings impose a one-to-one relationship between data items and sonic events (possibly involving some scaling and quantisation) whilst metaphoric or analogic mappings impose interpretive filters or mapping functions to the data before it is rendered’ (Vickers and Hoggs Reference Vickers and Hogg2006: 210), I have superimposed enough direct mappings based on pre-processed data to lead to an overall more metaphorical approach.

Moreover, this piece also included introductory and concluding sections that were not mapped to any data and instead acted as poetic commentaries on the type of data (large number sets, in the introduction), and the process looked at (the slow adjustment of weight values in a NN, in the conclusion). In this and in the overall complexity and lack of strict empiricism of the individual representations of data strands, I go against the argument that ‘In the pursuit of æsthetic excellence we must be careful not to tip the balance too far in favour of artistic form’, which seems a valid argument mostly if one has strict scientific auralisation aims and not really in art research.

The second work offers a comparatively much simpler mapping strategy. Still parameter-based, it focuses on just one parameter instead of multiple. It also leaves the secondary metrics, sonifying various performance indices, for a sonification of just the output samples. Looking again at Vickers and Hoggs’s (Reference Vickers and Hogg2006: 210) definitions cited above, this is an example of a much more direct sonification approach, mapping the movement of points across an eight-zone distribution to eight corresponding speakers. Each speaker plays a chord part of an overall chord that becomes fully audible when the data are successfully distributed and all speakers are firing (see Figure 5).

Figure 5.

Chords used in each of the 8-channel speakers in Spin, split, spread, splatter (in eighths).

Figure 5. Long description

The image displays a detailed musical notation divided into eight sections, each labeled with a number from 1 to 8. Each section contains specific chord arrangements designed for eight-channel speakers. The notation includes various musical symbols such as treble and bass clefs, eighth notes, quarter notes, and sharps. The chords are arranged in a structured manner, indicating the precise musical output for each channel in the piece titled ‘Spin, split, spread, splatter’. The notation reflects the complexity and precision required in data-driven musical compositions, addressing the need for aural representations of data-handling processes.

The third work contrasts with the other two by having no mapping of data in the audio dimension. This is because the aim of this work is to show how abstract the information flowing through a NN’s neurones is; there is no intelligible structure to uncover from individual weight value changes. The arbitrary value changes and resulting weight structures are presented in visual form, but would have been too arbitrary to map onto sound given my aesthetic priorities for the works.

2.3. Sounds

The sounds used were a mix of the ‘post-digital’ timbres that were outlined above and of organ recordings, making up harmonically intricate drone textures. This contrast between the purely synthetic and highly organic (here sounds from wooden pipes often made to emulate human-blown wind instruments, and characterised by unpredictable variations in airflow and timbre) questions the traditional use of codified sounds (described earlier) to semantically represent data that is by nature immaterial and therefore without sound. On the one hand, I take advantage of the ‘fake’ ecological approach, if referencing Keller and Stevens’ (Reference Keller and Stevens2004: 4) ‘analogic-symbolic representation continuum,’ where an ecological relation is when ‘target and surrogate referents are identifiable with one another because they tend to coexist in the world’ (ibid). Here, as outlined above, sounds have become semantically tied to an ecological perception of data through transfictionality. On the other hand, I question this association, by using sounds that might be perceived as far removed from the idea of data as one can be.

The use of these sounds acknowledges the impossibility for universally (temporally, geographically and culturally) understood symbols, as ‘all acoustic symbolism, even that associated with archetypes, is slowly but steadily undergoing modification’ (Schaffer Reference Schaffer1977). And through it I argue that if musical landscape, as ‘the perceived source or sources of a musical sound’ (Wishart Reference Wishart1996: 139) can be created from real ecological ties (bird sound for a bird), it can be just as strong for ‘fake’ ecological/semantic connections (synthetic ‘glitch’ textures for data) developed through transfictionality and reiterative associations, and that challenging these associations (for instance through the use of a highly contrasting sound world) can be an interesting artistic investigation.

I believe Bown’s framework for artists’ possible approaches to a work using technology is useful to reflect on my approach to this research, which has changed over the course of the project. I started off with the mindset of ‘I have a creative goal, find me a system that can realise it’. (Bown et al. Reference Bown, Fergusson, Dias Pereira Dos Santos and Mikolajczyk2021: 309), moved quickly to ‘Show me the conceivable limits of what this system can do and I will find my creative space’. (ibid) when realising some of the limits of the system in the face of my goal, and experimented at times with ‘Show me some things that this system can do, and I will create something within that space’. (ibid), when faced with the vastness of possibilities offered by that same system. I started out with an aesthetic aim, inspired by the database and post-digital aesthetics described earlier, then hit some limits of how I could use the algorithms I was working with, and after experimentation with various parameters, applications, datastreams obtainable from them, decided on more specific pieces. This last step resulted in a reworking of the technical aspects of the work (how I coded and interacted with the networks) influenced by the artistic/aesthetic reflections I made while working on the compositions.

2.4. Visuals in relation to audio

Unlike the varying degrees of abstraction present in the sonification mappings, visuals for the pieces were always direct mappings of the data, that is, with no loss of dimensionality: the visuals always represent either output samples or the internal structure of the network in their complete form, not as synthesised performance metrics or heavily scaled abstract representations. If one argues that ‘data and their experience must be carefully composed, if they are to be comprehensible by a broad audience’ (Loukissas Reference Loukissas2019: xv), in this case the visualisations are very close to a comprehensive representation of the data, requiring little composition.

If some Schaeffer spoke of a ‘visual bias of modern Western culture’ (Schaffer Reference Schaffer1977), I choose to see it as a product of how our brains tend to work; the ears are mostly there to tell the eyes where to look. Moreover, there is a case to be made for the higher level of recognisability of multi-media experience over a mono-sensory experience: ‘the recreation of the effect ‘fire’ by purely auditory means, can simply fail to evoke the power of the multi-media image of fire’ (Wishart Reference Wishart1996), even in the case where the source is immaterial and abstract and produces neither sounds nor image.

While it has been argued that ‘A visual representation is no more objective than an acoustic one’ (Caiola et al. Reference Caiola, Ricco and Lenzi2022: 2), which may be true, our perception of visuals can often lead to better empirical estimations (most can name a colour they see but not a pitch they hear, or estimate a distance better than a tempo). The main benefit of a direct visual mapping is that image data is represented as images in the first piece, 2D plot data is represented on a 2D plot in the second, and a network is displayed as a network in the third, which would be impossible with just audio. Therefore, visuals are useful tools to contextualise the specific data strands that drive the audio. In this, I have ‘recognized the fundamental perceptual differences between sight and hearing and rely on the features already known by the user’ (ibid., 2022:9), by allowing visuals to act as direct representations, as ‘a stepping stone to gaining a clearer [analytical] understanding’ (Munzner Reference Munzner2015: 3), and on the other hand using sounds to trigger a more abstract and emotional response.

Some things ‘exist on a scale inaccessible to human perception. In these cases, sonification serves to make phenomena accessible to the senses that do not allow for direct sensory experience’ (Supper Reference Supper2014: 48). This is what I have aimed to do to some degree with the transformation of abstract numerical information and the slow ordering of information. In this, sound serves as a more metaphorical tool alongside the more direct/literal representation of the visuals. Overall, this approach is a blend between the redundant, where ‘a design mapping all displayed information to both senses redundantly’ (Enge et al. Reference Enge, Elmquist, Caiola, Rönnberg, Rind and Iber2024: 9) and complementary approaches, where ‘a design mapping part of the information exclusively to the visualisation and another part of the information exclusively to the sonification’ (ibid).

Regardless of the reasons for using them, the visual representations of either the output (Noise through […] and Spin split […]) or of the actual network structure (Reiterate something […]), mapped in a similarly minimal aesthetic to the sounds, also drawing on post-digital influences, and generally allow for a reiteration or clarification of the sound information.

3. Let the product influence the process

If some have argued that ‘the poetic of computers lies in the genius of individual programmers to express the beauty of their thought using such an inexorable medium’. (Hamilton and Bonk Reference Hamilton and Bonk1997: 309), looking at the programmer as solely responsible for a computer programme’s value takes attention away from the plurality of this medium and its authorship. Instead looking back at Dyer’s interpretation of the layered nature of pieces using NNs, ‘assemblages of the artistic agencies of [the] composers, the algorithmic agency (the successive layers of refining and derived probability distribution sets) of their neural networks, and the creative-computational agencies of the networks’ engineers’ (Dyer Reference Dyer2022: 224) pushes me to consider the networks as another creative ‘actor’ in the construction of these pieces (given that in this case, I am the composer and engineer). The generated materials from NNs can only be moulded into artworks, rather than defined completely by the artist, which implies a loss of some agency, or at least a creative restriction. I have argued the aesthetic interest of putting suboptimal computational runs in perspective, of framing malfunction and inaccuracies, which is another set of material that constrains my creative space. After deciding on this aesthetic focus, I also chose to build inefficiencies into the technical makeup of the algorithms to outline specific narrative structures.

While I have outlined that the mass of data I present at any one time is inherently non-narrative, NNs themselves theoretically have equally inherent narrative potential. That is because of the constant rewriting of their output, the incremental move from chaos to order, their teleological nature. In many ways, their operation fits Bailey’s description of palimpsests, objects where:

This constant rewriting, at the micro and macro scales (epochs and runs), and the focus on the evolution of its architecture, ‘the components that constitute an assemblage at a given moment and polarise it towards such and such a behaviour’ (Guattari Reference Guattari2011: 195), are what I have attempted to put into focus by purposefully hindering networks’ performance. In this case, both the relation of my creative agency and network behaviour, and the network structure itself (made up of number of disconnected runs brought together) constitute assemblages, Deleuze and Guattari’s (Reference Deleuze and Guattari1970) concept dealing with the play of contingency and structure, organisation and change, all of which are at the core of this research work.

In a complete human-in-the-loop approach to ML (Mosqueira-Rey et al. Reference Mosqueira-Rey, Hernández-Pereira, Alonso-Ríos, Bobes-Bascarán and Fernández-Leal2023), I have played with network settings, to force them to specific patterns of malfunction. For instance:

• – Mode collapse (by mildly reducing the discriminator’s learning rate in a GAN): a common error whereby a GAN fails to generalise, and produces one type of sample endlessly (Goodfellow et al. Reference Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley and Ozair2014)

• – Noisy output (with too inefficient an architecture): where the network fails to improve past a certain limit, and continuously outputs data resembling the training data, but noticeably ‘off’

• – Slow training (by significantly reducing the general learning rate): where the network runs at an excessively slow pace.

• – Chaotic outputs (by significantly increasing the learning rate past any realistic value, whereby the GAN cannot ever approach a solution).

By constraining the networks to these patterns, and ensuring odd development, I emphasise their palimpsest nature outlined above. Sometimes this rewriting incrementally leads from chaos to order (as would be expected in an efficient model), sometimes it is static (failure to learn), sometimes unexpectedly random. The use of these types of runs comes from a need to ‘accept the procedural significance of failure, to embrace glitch, […] to understand glitch as more than mechanical error or unforeseen failure to function but also as a ‘form of refusal’, an ‘erratum’, and a ‘non-performance’. (Alonso et al. Reference Alonso Trillo and Poliks2022: 505).

This is a ‘dirty-electronics’ approach to neural networks, running them on a small-scale, with a focus on creativity rather than efficiency; encompassing ‘a notion of the post-digital, the self-made and do-it-yourself in contrast to the mass-produced, and the reinvigoration of the role of the human’ (Richards Reference Richards2008: 25). Its main aim is for both its process and results to offer a parallel to the aesthetic nature of the works. Therefore, the choice of sounds outlined earlier, made as a reference to other data-evoking traditions, now influences what output I aim to get out of those networks, in a feedback loop. Noisy data (as well as the original noise fed to the GANs) offers parallels to my use of white noise in sound, ‘dirty’ data fits with clicky and distorted textures, ‘human’ looking outputs (for instance handwritten digits) are paired with sounds associated with long-standing cultural traditions (for instance organ sounds), etc. This crossbreeding between process and output, and the treatment of the algorithm as having an artistic value in its own right are central to this approach.

This framing of the technical as artistic extends to the datasets I use, which are generally extremely common/mathematically simple, as far removed from any human-made art as possible: the MNIST dataset, abstract point distributions, small-scale black and white spectrograms, etc. I find this makes transformations clearer, but also confers a definite ‘data’ aesthetic to the pieces, again influenced by the aesthetic choices.

If one sees the resurgence of ‘dirty electronics’ as ‘a reaction to the vestiges of the digital world: the virtual, wireless, pseudo-modernist design, utilitarianism and seemingly endless possibilities’ (Richards Reference Richards2008: 26), maybe this approach to neural networks as similarly unstable and imperfect tools is a reaction to and a reflection on the waste inherent to ML-adjacent technologies:

Perhaps wanting to explore these imperfections rather than fight them, including in the algorithms themselves, is a logical next step. Hence, this approach comes full circle back to the inception of the post-digital as an idea, emerging ‘from the “failure” of digital technology’ (Cascone Reference Cascone2000:12).

4. “But can we actually gain insights from this mess?”

It has become difficult to discuss machine learning without mentioning the black box nature of its systems, the fact that ‘because of their nested non-linear structure, these highly successful models are usually applied in a black box manner, that is, no information is provided about what exactly makes them arrive at their predictions’ (Samek et al. Reference Samek, Wiegand and Müller2017: 1). Naturally, extensive work has been done on explainable AI (XAI) to remedy this fact, with countless approaches (Adadi and Berrada, Reference Adadi and Berrada2018; Ali et al. Reference Ali, Abuhmed, El-Sappagh, Muhammad, Alonso-Moral and Confalonieri2023). Most focus on trained networks, narrowing down which parts of the input data lead to a particular decision (Baehrens et al. Reference Baehrens, Schroeter, Harmeling, Kawanabe, Hansen and Müller2010; Shrikumar et al. Reference Shrikumar, Greenside and Kundaje2017; Chang et al. Reference Chang, Creager, Goldenberg and Duvenaud2019), as the ethical need to be able to explain networks’ specific human-impacting decisions is much greater than any theoretical understanding of its processes.

Yet ‘interpretable AI’, defined by having a ‘level of understanding how the underlying (AI) technology works’ (ISO 2020: 3.1.42) (in contrast to XAI’s ‘level of understanding how the AI-based system […] came up with a given result’ (ISO 2020: 3.1.31)) is a field of its own, and one (unlike XAI) to which I originally aimed to contribute in a very modest way. My aim was to represent how networks’ structures would change over the course of a training run in audiovisual form, in parallel with changes in output, to provide insights on how the overall system worked. Yet the abstracted and highly complex nature of these systems means that a different initialisation leads to a completely different weight structure when it gets to a solution, and finding patterns in these structures between different runs is essentially impossible.

I quickly pivoted to look at the output over the course of training, where the evolution from noise to order is clear, and patterns can be deduced between runs (as detailed above). The ‘waste’ nature of in-training intermediary outputs, also acts as a ‘trace’ of the network state at that point, without having to be confronted by abstract weight or activation values: ‘this waste itself functioned as historical documentation tracing the technical progress of the platform’ (Alonso Trillo and Poliks Reference Alonso Trillo and Poliks2023: 393).

Moreover, my focus on rewriting networks to encourage errors/inefficiency in their operation allowed for these works to clearly represent common NN errors (mode collapse, slow learning, vanishing gradient, etc.). I believe that seeing/hearing these inaccuracies can help understand how they are expected to operate: ‘whenever the GANs delivered something different from what we had foreseen and what we had coded them to do, the sensibilities of their architectures became more apparent’ (Alonso Trillo and Poliks Reference Alonso Trillo and Poliks2022: 505).

I returned to mapping the abstract activation and weight data of a network later in the research, in Reiterate something so many times that it changes. Training the same network tens of times on the same problem, with the same architecture, and ending up with completely different weight structures was a valuable representation of how complex and beyond human logic these systems work. I tried to alleviate as much as possible the fact that ‘Communication always seems to happen in the shadow of a lost immediacy with the totality’. (Galloway et al. Reference Galloway, Thacker and Wark2013: 160) by reiterating on the same problem tens of times.

In the end, the inherent difficulty of finding any patterns or explanation of the networks’ behaviour from their operation was in itself an explanation. Again, there is something to learn from being faced with errors, waste, abstraction, unintelligible masses of numbers, even if that is that these networks should not be thought of as simply existing in a ‘problem + architecture = solution’ paradigm.

Going back to the question of XAI, few studies actually visualise training at all, partly because of the inability to gain any clear insights from these structure changes. Even when looking at a trained network, ‘visualising features to gain intuition about the network is common practice, but mostly limited to the 1st layer where projections to pixel space are possible’. (Zeiler and Fergus Reference Zeiler and Fergus2013: 2). Quickly, any information becomes undecipherable. Yet that is not to say there is no need for a narrative representation in the ML field. Indeed, Guattari had already theorised and predicted that: