2.1. Generative synthesis and process-based music

Generative music and generative art have a long history, with many examples of systems and machines within musical and artistic works or appearing within the processes of making those artworks. A recent definition by Brian Eno describes generative music as ‘the idea that it’s possible to think of a system or a set of rules which once set in motion will create music for you’ (D’Errico Reference D’Errico2022). The assumption behind the creation of music as a result of a process is that music can be generated by applying a combination of algorithmic techniques based on music theory. This topic has been approached by serialist and minimalist composers such as Michael Nyman and Steve Reich, who have explored rule-based transformation processes with notated musical works (Schwarz Reference Schwarz1980). Musicians interested in cybernetics have employed feedback networks as a way to construct semi-autonomous systems (Sanfilippo and Valle Reference Sanfilippo and Valle2013), and several other techniques have been used in custom algorithmic systems over the years in contexts of composed and improvised music in many styles, encompassing ambient electronic works such as 65daysofstatic’s Wreckage Systems (65daysofstatic 2022) and interactive improvisation systems like George Lewis’ Voyager (Steinbeck Reference Steinbeck2018).

In this article, we are specifically interested in generative synthesis. We define generative synthesis models as sound-generating systems that can organise sound patterns and their spectromorphology from the micro up to the meso time scale of musical perception (Smalley Reference Smalley1986). The meso time scale is described by Roads (Reference Roads2003) as a local time scale with respect to a global music composition, whose duration is measured in seconds. At the meso time scale, sound objects are organised into melodic, harmonic, rhythmical and contrapuntal relations. In electronic music, this is the time scale at which the evolution of sound masses is perceived, with variations in amplitude, tempo, density, harmonicity and spectrum (Roads Reference Roads2001). Events at this time scale are described by Wishart (Reference Wishart1994) as sequences, while Smalley identifies this scale as the one in which spectromorphological evolution occurs.

Several algorithmic techniques have been used to create such generative synthesis models. Some examples are modular synthesis patches (Teboul and Kitzmann Reference Teboul and Kitzmann2024), stochastic sequencers (Luke and Carnovalini Reference Luke and Carnovalini2024), rule-based systems (Spangler Reference Spangler1999), statistical sequence modelling (Van Der Merwe and Schulze Reference Van Der Merwe and Schulze2010), genetic algorithms (Alfonseca et al. Reference Alfonseca, Cebrian and Ortega2007), cellular automata (Agostini et al. Reference Agostini, Daubresse and Ghisi2014), feedback systems (Sanfilippo and Valle Reference Sanfilippo and Valle2013) and chaotic algorithms, the latter of which is explored in this article. The recent development of deep learning has introduced a large amount of novel deep generative models (Tomczak Reference Tomczak2022) such as autoregressive models, autoencoders and generative adversarial networks. These algorithms can be used to generate control data for sound synthesis (Barkan et al. Reference Barkan, Tsiris, Katz and Koenigstein2019), symbolic notation (Herremans et al. Reference Herremans, Chuan and Chew2017; Huang et al. Reference Huang, Vaswani, Uszkoreit, Simon, Hawthorne, Shazeer, Dai, Hoffman, Dinculescu and Eck2018; Mittal et al. Reference Mittal, Engel, Hawthorne and Simon2021) or to generate audio (Caillon and Esling Reference Caillon and Esling2021; Mehri et al. Reference Mehri, Kumar, Gulrajani, Kumar, Jain, Sotelo, Courville and Bengio2022; Tatar et al. Reference Tatar, Cotton and Bisig2023).

When performing with generative synthesis, musicians have control over a set of sound-generating parameters, whose effect is typically non-linearly correlated with changes in behaviour of the system. A change in control parameters affects the system’s state, which determines a change in its time behaviour up to the meso time scale. This control approach reconfigures the relationship between the musician and the instrument, transforming it into a dialogue in which agency is shared, a collaboration between the system and the composer (Di Scipio and Sanfilippo Reference Di Scipio and Sanfilippo2019; Erdem et al. Reference Erdem, Wallace, Glette and Jensenius2022; Magnusson et al. Reference Magnusson, Kiefer and Ulfarsson2022).

In this article, we specifically investigate control and notation strategies for the Benjolin, a generative synthesiser whose sound qualities depend on a finite set of eight continuous synthesis parameters, with no external sound input. The Benjolin is a chaotic synthesis model (Slater Reference Slater1998) which, as we argue in this article, represents a specific instance of generative synthesis technique.

2.2. Timbre-based control of synthesis

The possibilities given by electronic synthesis and elaboration of samples have shifted the focus of electroacoustic composers from categories like pitch, rhythm and harmony to the evolution of spectral properties. In this context, timbre has been used to refer to sonic properties that cannot be described by traditional categories. However, timbre is a debated concept whose definition varies widely across disciplines and musical practices, so the representation (and therefore notation) of timbre is an open problem in contemporary research. A wide range of disciplines have discussed the concept of timbre, including but not limited to music psychology, music informatics and (ethno)musicology. Timbre, as argued by ethnomusicologist Cornelia Fales (Reference Fales2005), exists only in the mind of the listener, and musicologist Nina Sun Eidsheim (Reference Eidsheim2019, p.10) even disputes that timbre is a knowable entity. In contrast, in music psychology, timbre spaces have been proposed as a means to map the similarity of sounds as perceived by experiment participants to a Euclidean space (Grey Reference Grey1975).

A timbre space is a representation of the relations between a set of sounds, such that neighbouring points are similar, while distant points sound very different from each other (Wessel Reference Wessel1979). Movements in the timbre space should map linearly to a perceived change in the quality of the sound. Initially, timbre spaces have been constructed using multidimensional scaling on a dataset of pairwise similarity scores obtained from surveys (McAdams Reference McAdams2019). Recent approaches are based on computing perceptually relevant audio features like spectral shape, loudness and mel-frequency cepstrum coefficients (MFCCs) on a dataset, reducing the dimensionality of that dataset using dimensionality reduction techniques like principal component analysis (PCA) or uniform manifold approximation projection (UMAP) (Garber et al. Reference Garber, Ciccola and Amusategui2020; Moore and Brazeau Reference Moore and Brazeau2023), or deep learning approaches like variational autoencoders (VAEs) (Tatar et al. Reference Tatar, Bisig and Pasquier2021; Esling et al. Reference Esling, Masuda and Chemla-Romeu-Santos2021; Caillon and Esling Reference Caillon and Esling2021).

Timbre spaces have been applied to explore the sonic output of complex synthesis models to develop timbre-based control strategies (Fasciani and Wyse Reference Fasciani and Wyse2012). The timbre space of a synthesis model can be constructed by evenly sampling its control parameters and encoding the corresponding synthesis sounds based on their timbral properties. Controlling parameters based on their timbre representation can be challenging, especially for synthesis models whose relationship between changes in parameters and changes in timbre is non-linear (Hayes et al. Reference Hayes, Saitis and Fazekas2025).

Several software to compute and navigate timbre spaces of sound libraries and synthesis models have been released in recent years. Examples are Audiostellar (Garber et al. Reference Garber, Ciccola and Amusategui2020), Timbre Space Analyzer & Mapper (TSAM) (Fasciani Reference Fasciani2016), FlowSynth (Esling et al. Reference Esling, Masuda and Chemla-Romeu-Santos2021), Flucoma (Tremblay et al. Reference Tremblay, Green, Roma, Bradbury, Moore, Hart and Harker2022) and Latent Timbre Synthesis (Tatar et al. Reference Tatar, Bisig and Pasquier2021). Audiostellar visualises a user-supplied audio corpus as an interactive scatter plot in two dimensions by extracting timbral features and then reducing the dimensionality using PCA, t-SNE or UMAP (Garber et al. Reference Garber, Ciccola and Amusategui2020); users interact with the timbre space with the mouse, and hovering over a point triggers the sample to play (Moore and Brazeau Reference Moore and Brazeau2023). An alternative option to sample playback is mapping synthesis controls to timbre space positions. An example of this is the TSAM by Fasciani (Reference Fasciani2016), a tool which analyses the possible timbres of a given synthesiser, finds a two- or three-dimensional representation using Isomap and then maps the synthesiser parameters to the timbre-based control space. Another approach for synthesiser control through a latent space is FlowSynth, which combines VAE and normalising flows (NF), using the VAE to learn a compact representation of the audio and NF to map this to the synthesis parameters (Esling et al. Reference Esling, Masuda and Chemla-Romeu-Santos2021).

These methods mainly employ timbre similarities computed over mathematical descriptors based on psychoacoustic perceptual studies. The limitations of this approach have been dissected by Morrison (Reference Morrison2024) as going from the ‘thick event’ (Eidsheim Reference Eidsheim2019: 5) of perceiving sound in socioculturally situated interactions between listeners and sound producers to a single point in a Euclidean timbre space. To further complicate things, research in music informatics has developed with a ‘curious divide’ (Siedenburg et al. Reference Siedenburg, Fujinaga and McAdams2016) to music psychology, for instance, by using a larger number of signal descriptors to estimate the similarity between sounds.

In this article, we employ timbre descriptors as a mathematical way of representing audio where traditional categories of pitch, harmony and rhythm are not suited for representation of complex spectral evolutions. Our study entails the computation of timbre descriptors to arrive at a timbre space that provides meaningful interpretations in the ‘thick event’ of notating generative music, bridging the described extreme positions. We acknowledge that we ‘flatten’ (Morrison Reference Morrison2024) a thick event in this process, but we will return to a culturally situated depth through our interviews with composers using the system.

2.3. Notation of gesture, sound or process

The design of an interface (physical or screen-based) for a digital sound model is strictly related to how music for that interface can be notated. Usually, notation is based on some abstraction that is necessary for being able to translate musical instruction to gesture, process or sound form. Historically, notation embeds instructions relative to musical actions, characteristics of sound and/or ways in which musicians (and instruments) should behave. These aspects are embedded in different ways in various kinds of notation. For example, a live coding script describes a sequence of operations that may be followed to create similar musical profiles using different scripting environments. A graphical score such as Cage’s Aria (Reference Cage1958) asks the performer to interpret melodic contour lines using thoughtful choices, steering the monodic instrument to represent changes in sound. Digital insctruments such as the T-stick may also be notated at the gesture level, instructing the performer how exactly to move the instrument (Moriceau et al. Reference Moriceau, Yan, Thibault and Wanderley2024).

The way instruments are notated is inseparable from the way they are accessed and controlled gesturally, retaining links to ideas that are central to the music theory(ies) informing the performance practice. For our case with the Benjolin, Figure 1 represents an example of graphically notated pieces for Benjolin created by Pete Gomes, where the notation directly instructs the player how to interact gesturally with the synthesis parameters describing the piece.

Figure 1.

Notation for Benjolin pieces. Images by Pete Gomes, used with permission. This notation is based on the hardware Benjolin instrument, representing the eight knobs and the modular patch. The graphic signs around each knob portray different behaviours with which each knob governing synthesis parameters should be approached while performing.

Several approaches to notation of generative processes have been proposed throughout the years. Pre-algorithmic generativity is discussed by Rochais (Reference Rochais, Soddu and Colabella2024) with several examples of generative and score-like formulations. A common strategy for multidimensional notational representations is to use graphical notation. Schaeffer’s propositions for acousmatic music describe an early abstraction based on a multimodal understanding of the interaction between shape and sound (Couprie Reference Couprie2018). Drawing as a strategy for representing electroacoustic music (Thiebaut et al. Reference Thiebaut, Healey and Bryan-Kinns2008) is found in works not only dealing with notation but also to study perception and musical representation of the acousmatic (Godøy Reference Godøy2006; Jensenius Reference Jensenius2013; Nymoen et al. Reference Nymoen, Godøy, Jensenius and Torresen2013; Kelkar et al. Reference Kelkar, Roy and Jensenius2018). Xenakis’ ideas (Nelson Reference Nelson1997) for UPIC graphical scores represent a graphical notation directly linked to the spectromorphology of sound, while contemporary cluster-based methods revolve around corpus-based, information-heavy styles of notating music (Scordato Reference Scordato2017; Garber et al. Reference Garber, Ciccola and Amusategui2020; Bell, Reference Bell2023; Roma et al. Reference Roma, Green and Tremblay2019; Esling et al. Reference Esling, Masuda and Chemla-Romeu-Santos2021; Coduys and Ferry Reference Coduys and Ferry2004). Vickery (Reference Vickery2014) proposes screen-based score paradigms for representing segmented scores in their articles, with an overview of interactions between temporality and screen display for score mappings. Bell, Reference Bell2023) raises the question of the extent to which timbre space representations (Couprie Reference Couprie2018) present a typology of graphical scores for composing and transcribing electroacoustic music. However, even the multidimensional potentials of graphical scores can be challenged when generative and chaotic instruments need to be represented due to the unpredictability and wide variations of their sonic outputs.

Magnusson (Reference Magnusson2014) presents several case studies for interpreting algorithmic scores, establishing the idea of computer codes being literally interpretable as musical scores. In their work on agential instruments, Armitage and Magnusson (Reference Armitage and Magnusson2023) explore dynamic scores further with these ideas of agential notation. Animated notation and evolving graphical notation also represent paradigms lending themselves to multidimensional, time-varying representations. Newer representations of graphic notations integrated with machine learning have led to explorations of novel temporal, sonic and spatial interactions.

The dimensions of interface, control and music theory have many intersecting concepts when it comes to notation (Hope Reference Hope2017). Many interactive notation systems can be considered at the same time as control interfaces and musical representations, because in these cases the notation system actively serves as a control structure for a sound model as well. UPICS, cluster-based methods, live coding languages and agential scores are clear examples of this duality.

3.1. The Benjolin

The Benjolin is a hardware synthesiser designed by Rob Hordijk in 2009, making features of his commercial Blippoo Box (Hordijk Reference Hordijk2009) more accessible as a do-it-yourself project for the analogue synthesist community. The Benjolin, shown in Figure 2, is made up of two voltage-controlled oscillators with frequency control parameters, Oscillator 1 (O1 FRQ) and Oscillator 2 (O2 FRQ), and one voltage-controlled low-pass filter with frequency (FIL FRQ) and resonance (FIL RES) controls. The Benjolin’s audio output is the result of a comparator operation of the triangle waves from O1 and O2 fed into the low-pass filter (see Figure 3).

Figure 2.

A hardware Benjolin as a standalone synthesiser. Macumbista Instruments, 2025.

Figure 3.

Diagram of signal flow and operations in the Benjolin synthesiser.

The defining characteristic of both the Benjolin and the Blippoo is a chaotic control structure called the Rungler. The Rungler utilises a shift-register-based pseudo-random number generator to generate a 3-bit, stepped analogue control voltage signal. The square wave of O1 provides a digital data source for the pseudo-random generator, while the square wave of O2 acts as a digital clock controlling the generator’s rate of change. The resulting Rungler control voltage can then be applied as the variable modulation source to the frequency parameters of the oscillators and the filter. In total, there are eight user-controllable parameters in the instrument:

O1 FRQ: variable frequency of Oscillator 1;

O1 RUN: variable amount of Rungler stepped wave control over Oscillator 1 frequency;

O2 FRQ: variable frequency of Oscillator 2;

O2 RUN: variable amount of Rungler stepped wave control over Oscillator 2 frequency;

FIL FRQ: variable cutoff frequency of the filter;

FIL RES: variable resonance of the low-pass filter;

FIL RUN: variable amount of Rungler stepped wave control over filter cutoff frequency;

FIL SWP: variable amount of Oscillator 2 triangle wave control over filter cutoff frequency;

Because the output of the Rungler can modify the frequency of both the clock and data oscillators, the overall arrangement fulfils the basic criteria of a chaotic synthesis system: it displays a high sensitivity to initial conditions, feedback is present within the system and the system contains an element of non-linearity (Slater Reference Slater1998). Hordijk notes that the interaction of the Rungler and its corresponding oscillators represents a chaotic attractor, which seeks a new balanced state from the turbulence created whenever its control parameters are disturbed (Hordijk Reference Hordijk2009).

For the purposes of this project, we created a digital emulation of the Benjolin sound model using the Pure Data (PD) environment. The emulation facilitates network connections to these parameters from other programming environments, such as Python, via the Open Sound Control (OSC) protocol. Our PD BenjolinFootnote ³ emphasises the functional aspects of the original instrument, with less priority given to a completely accurate reproduction of its output sound. In this article, we refer to a state of the Benjolin as the sonic output of the Benjolin produced by a unique combination of its eight sound-generating parameters.

3.2. Latent space

The Meta-Benjolin is based on a latent representation of the timbre of the Benjolin. Each state in the Benjolin corresponds to a sequence pattern of changes in spectromorphology and rhythm whose period is determined by the clock. The rhythmic and spectromorphological aspects of the sound in each state are strongly interrelated and can be difficult to control separately. Therefore, a notation system for this instrument needs to consider this unique aspect. For this reason, we decided to use a latent timbre representation in which the complex multidimensional spectral characteristics of each state are reduced to three dimensions. This representation has been constructed using a VAE (Kingma and Welling Reference Kingma and Welling2013), an unsupervised machine learning model that learns a compressed representation of a dataset by projecting the distribution of training datasets to a lower-dimensional space.

3.2.2. VAE training

A VAE neural network (Pinheiro Cinelli et al. Reference Pinheiro Cinelli, Araujo Marins, Barros da Silva and Lima Netto2021) was used to obtain a latent representation of the timbre space. The VAE model consists of an encoder network, a sampling layer and a decoder network. We employed a VAE where encoder and decoder networks each consist of three hidden fully connected layers. The encoder receives a 36-dimensional input vector, x, reducing it to 16 latent dimensions.

The loss function consists of the reconstruction error (mean-squared error) and the fl-weighted Kullback–Leibler divergence with fl = 0.01. Training ran for 200 epochs using an Adam optimiser with a learning rate of 0.001 and a learning rate scheduler (patience = 5, factor = 0.5). Mixed precision training was used to speed up training and reduce computational resources. After training was completed, the entire dataset was fed through the encoder network to compute the mean vector of each data point, which was used as the latent space projection for each data point. PCA was used to further reduce the latent space to three principal components, retaining 96.6% of the variance of feature vectors. Lastly, only the three-dimensional coordinates of each data point in the final latent space are used to map from coordinates to the corresponding synthesis parameters in a lookup table, as shown in Figure 4.

Figure 4.

Data flow diagram of the Meta-Benjolin structure. The latent representation based on timbre learnt by the VAE is used in the screen-based interface to map three-dimensional coordinates to the corresponding synthesis parameters.

3.3. Screen-based notation interface

The screen-based interface of the Meta-Benjolin allows the user to navigate the virtual three-dimensional timbre space as a point cloud. The design of the interface aims to enable visualisation and access to the full range of possibilities of the instrument, interacting with the sound process created by the Benjolin based on its timbral qualities, without any knowledge of the parameters of the original synthesis model. A diagram of the interface is shown in Figure 5.

Figure 5.

The Meta-Benjolin interface is composed of a three-dimensional navigable point cloud, a vertical composition timeline (left) and a menu (top-left).

The graphic user interface of the Meta-Benjolin has been developed in JavaScript, and it is accessed from a browser. The sounds are generated using the PD Benjolin running in the background. Whenever synthesis parameters change in response to a user action, the interface accesses a table mapping the three-dimensional coordinates of each point in the space to the corresponding eight-dimensional parameters of the synthesiser, whose values are sent to the PD Benjolin through OSC, causing a change in sound. The user interface is composed of three main components: a three-dimensional point cloud, a vertical composition timeline and a menu.

Point cloud. The three-dimensional point cloud occupies the entire screen. A three-dimensional representation has been chosen over a two-dimensional space to encourage nuance and variations. The user can navigate the space where the cloud is located using the mouse, zooming in and out to specific regions. Each point in the cloud corresponds to one state of the Benjolin from the training corpus of the VAE. When the user hovers the mouse pointer over a point in the cloud, the sound of the corresponding state of the Benjolin is played; if the mouse pointer is not hovering on any point, the instrument is silent. This interaction was designed as a ‘scrubbing’ of the corpus (Wessel and Wright Reference Wessel and Wright2002). When a user clicks on a point, it changes in colour, and a circle of the same colour is created in the composition timeline, indicating that the state has been chosen by the user to be part of the piece. The colours are chosen randomly.

Composition timeline. The composition timeline is located on the left side of the interface. The composition flows vertically from the top to the bottom as a sequence of states and transitions between them. States are represented as circles and transitions as arrows. The states selected by the user are represented as circles of different colours, stacked vertically. The vertical order of the states in the composition timeline represents the time evolution of the piece. The size of each circle in the composition timeline represents the time duration of the corresponding state, which can be increased or decreased by dragging the circle’s border. The user can change the order of the states in time by dragging elements in the composition bar in between each other. When two states are subsequent in the timeline, a line is drawn between them in the point cloud, indicating that there is a transition between those two states. In this way, the whole composition is represented as a path in the three-dimensional space, superimposed on the point cloud.

Menu. The menu is located at the top left of the screen. It is composed of seven buttons in three groups. The buttons in the menu allow to introduce crossfade and meander transitions between states, delete states from the composition timeline, play back, stop and record the whole composition, and download the coordinates of the states as a text file.