1. INTRODUCTION
This article presents an idiosyncratic approach to the development of an augmented violin I improvise with in live performance. It offers a philosophical perspective informed by materialist thinkers – especially Gilles Deleuze, Felix Guattari, Gilbert Simondon and Francisco Varela – on how such digital extensions evolve and what sorts of approaches support the development of highly refined digital musical instruments (DMIs). I interpret the violin’s technical evolution from its early inception to digital-physical approaches in contemporary performance and improvisation, highlighting development of my computational system for violin performance, Windowless.
As a composer–performer and music technologist, I approach improvisation in a way that eschews embedding a dialogical or agent-like element in the computational media. Instead, I favour an ‘instrument paradigm’ (Rowe Reference Rowe1993: 8) affording more primordial vectors of response: densely layered, spectrally distinctive operations occurring across multiple time scales, generating a thick performative medium of extraordinary physics – a digital ‘aural architecture as an extension of the musical instrument’ (Blesser and Salter Reference Blesser and Salter2007: 213). My approach to leveraging digital tools in improvised performance therefore tends to be less dramatically stylised by ‘stuff coming at you from left field to make you react’ (Jon Rose, pers. com., 25 March 2019), and more about ‘playing the room’ (Trueman Reference Trueman1999) as violinists have done for hundreds of years, and humans for thousands, in responding to the unique physics of different spaces. I have played and presented my system at a variety of diverse venues and conferences, from improvisation summits to computing conferences on cognition and tangible media (Figure 1).
Demonstration at ACM Creativity & Cognition, San Diego, June 2019.
2. MODEL-FREE DIGITAL MUSICAL INSTRUMENTS
A distinctive affordance of real-time signal processing is the ability to symmetrise action and perception through improvisation, insofar as DMIs can be continuously adapted and experimentally modulated by their designers, who construct gesture and audio descriptors on the fly (Thorn and Sha Reference Thorn and Sha2019). The continuity of the historical development of the acoustic violin and its contemporary uptake in digital–physical systems is evidenced by a progressively and immanently enacted ‘cognitive architecture’ (Varela, Thompson and Rosch Reference Varela, Thompson and Rosch2016: 106) displayed in these instruments: to compose a DMI through model-free tinkering is isomorphic with the enactive approach to cognition as ‘perceptually guided action’ (Varela Reference Varela1999: 12). Trial-and-error revision, in other words, symmetrises sensory feedforward and feedback paths – particularly whenever sound production is multimodally reinforced – with the designer pursuing a sui generis intensive development trajectory that steers away from embedding reified models of traditional instrumental techniques or agency in order to be responsive to any sonic or physical input, thus accommodating unexpected behaviours, novel gestures or extreme input magnitudes (Sha Reference Sha2013: 211–14). My approach is different from machine learning approaches using data sets that classify canonical bowing styles such as détaché, martelé and spicatto to select different electronic processes in live performance (Bevilacqua, Rasamimanana, Fléty and Lemouton Reference Bevilacqua, Rasamimanana, Fléty and Lemouton2006), and to approaches modelled for general users affording selection among traditional algorithms such as phase vocoding, feedback delay or granulation (Overholt and Gelineck Reference Overholt and Gelineck2014). By contrast, the signal processing logic of systems developed through tinkering will resemble the ‘patchwork of subnetworks’ evident in biological systems, which are not the result of ‘clean, unified design’ but of immanent, processual development (Varela etal. Reference Varela, Thompson and Rosch2016: 106).
The significance of this design approach is articulated by Deleuze and Guattari in their depiction of ‘minor science’ (Deleuze and Guattari Reference Deleuze and Guattari1987), a concept that richly articulates the immanent logic implied by the improvised development of model-free DMIs, the technical evolution of acoustic instruments, and the act of musical improvisation itself. For in each case it is a question of following a flow of inhomogeneous matter. What do Deleuze and Guattari mean with this image?
Deleuze and Guattari distinguish between ‘two types of science’, one that ‘reproduces’ and another that ‘follows’ (Deleuze and Guattari Reference Deleuze and Guattari1987: 372). Sciences that follow are described as ‘itinerant’, ‘nomadic’ or ‘minor’. ‘Royal science’, on the other hand, idealises ‘reproduction, deduction, or induction’. ‘Reproducing implies the permanence of a fixed point of view that is external to what is reproduced: watching the flow from the bank’ (ibid.). In minor science ‘one engages in a continuous variation of variables, instead of extracting constants from them’ (ibid.).
Artisanal practice, an ancient mode of production, is illustrative of minor science. Crafting a violin, for instance, is a process shaped by mechanistic and final causes but reduces to neither: the luthier crafts a violin, but the contingencies of the material singularities of the wood imply that the violin’s final form is not entirely dictated in advance. The French philosopher Henri Bergson elucidated this path of ‘creative evolution’ with the example of a canal zigzagging its way to a destination, revealing a course shaped by the avoidance of obstacles (Bergson Reference Bergson1944: 104). Likewise, DMIs take on unforeseen trajectories when they are composed without models. This is why Deleuze and Guattari emphasise that royal science takes a ‘fixed point of view’ on matter, implying an ocularcentric distancing or reflective, theoretical detachment – theoria: ‘to behold’ (Jay Reference Jay1993: 23) – that misses the pre-reflective, sensorimotor skill of scrupulously following singular material qualities. Such creative evolution is descriptive of the artful skill of luthiers, digital tinkerers and musical improvisers alike. These processes are ‘molecular’, a term appropriated by Deleuze and Guattari in reference to non-covalent bonding, cogently summarised by Brian Massumi thus: ‘fragmentary processes operating particle by particle through strictly local connections’ (Massumi Reference Massumi1992: 48).
The implications of ‘following’ for DMI development and musical improvisation are clear: to practise minor science is to improvise, to respond to a singular condition. Etymologically, to im-pro-vise is to not-fore-see (Evens 2004: 147). Just as for minor science, what counts as success in improvisation is incommensurate with what counts as the successful execution of a predefined goal, and this may be what is most salient about improvisation in the context of DMI development: that, despite – or because of – the lack of research methodologies or criteria promising results on the basis of established procedures, DMIs built up from processual, model-free tinkering turn out to be highly refined. Their success – the finesse, nuance and responsivity of matured morphology – entails no generalisable categories, no claim to universalisability/validity. This is characteristic of minor scientific practices as such. Philosopher of science Paul Feyerabend articulates the implications bluntly: ‘the success of “science” cannot be used as an argument for treating as yet unsolved problems in a standardized way. That can be done only if there are procedures that can be detached from particular research situations and whose presence guarantees success. The thesis says that there are no such procedures’ (Feyerabend Reference Feyerabend2010: xix).
As an educator and researcher, I view this methodological situation as the key contribution of DMIs developed without models in the context of computational creativity, namely that creators of such instruments demonstrate the capacity to reason with abductive agility, that is, to pursue the work without defining the salient features or categories a priori, or laying down a strong theory or method in advance of the actual work (Filmowicz and Tzankova Reference Filmowicz and Tzankova2017: 5). Through trial and error, salient features are designated and constructed, certain types of attacks or bands of energy deemed relevant, and the resulting media constructed by the mathematical decisions in the analysis of the feature vectors. This is minor science, ‘intuition in action’ (Deleuze and Guattari Reference Deleuze and Guattari1987: 409).
It is for the same reason that one can have an experience of yielding to the immanent demands of a ‘composed instrument’ (Schnell and Battier Reference Schnell and Battier2002), the ‘tendency of the material’ (Adorno Reference Adorno2006: 31), or laud those artists who are always starting over from scratch (Guattari Reference Guattari2000). What is affirmed here is a suspension of the human faculty in favour of an ecological sensitivity, a listening to – so highly praised by adept improvisers – and following inhomogeneous flows, that better reflects this general phenomenon of material being articulated by Varela, Bergson, Deleuze, Guattari and other materialist thinkers. For even music composers (in the traditional sense) report recourse to their theoretical training only at impasses, when calculative reflection is provisionally engaged to move things forward. Like digging a canal, the rhythm between the two – reflection and following – is germane to improvisation across the arts.
Before relaying the development of my augmented violin as an example of model-free design symmetrising action and perception through real-time signal processing, I will gloss the evolution of the acoustic violin and playing technique by leveraging Gilbert Simondon’s description of the evolution of technical objects as processes of spontaneous, stepwise ‘concretization’ (Simondon Reference Simondon2017). This shores up the understanding of technical development as a process of ongoing, unforeseen surprises experienced in acts of following, ‘tinkering with potentialities present in the material’ (Dumouchel Reference Dumouchel, Feenberg and Hannay1995: 259). This style of thinking can be carried into digital practice.
3. THE VIRTUAL VIOLIN
Dan Trueman – who along with Dan Overholt, Mari Kimura and Jon Rose counts among the most prolific practitioners of digitally augmented violin performance – describes the violin thus: ‘I regard the “violin” as an instrumental archetype – a superclass, or meta-instrument – that specifies little about the instrument and nothing about the music it is used to play (or its sound!)’ (Trueman Reference Trueman1999). The archetype is ‘fuzzy’; its criteria, ‘loose’ and ‘incomplete’. Trueman’s examples are the electric violin, ‘Schubert’s violin’, ‘Bartok’s violin’ and the Hardanger fiddle. These subclasses inherit the properties of the superclass, but each is idiosyncratic, a single ‘realized potential’ (ibid.).
My intention in this section is to supplement Trueman’s theoretical approach, which I draw on extensively in this article, by introducing concepts from the philosophy of Gilbert Simondon, a philosopher of technical evolution. I share Trueman’s motivation to rattle the orthodoxies surrounding this instrument for the sake of reimagining it, but my strategy is to locate the ‘incompleteness’ of the violin in the native abstractness of technical objects. Following this exposition, I will translate the analysis into the context of trial-and-error development of signal processing in model-free (improvisatory) DMI development.
3.1. Technical evolution: from abstract to concrete
In Aden Evens’s compelling phenomenology of musical instruments, the musical instrument presents ‘a leading edge of indeterminacy, which is only defined progressively’, so that at every moment, the instrument is situated between a virtual potential and concrete actualisation (Evens Reference Evens2005: 160). A productive struggle between the musician and the instrument’s ‘resistance’ occurs here; the musician is provoked to continue at this abstract edge, which leaves behind a residual indeterminacy that must be picked up again.
A parallel abstractness is evident in the history of the violin’s technical evolution. Emerging around 1520 in northern Italy, the violin synthesised musical features of the rebec, Renaissance fiddle and lira da braccio, but its remarkable musical and virtuosic potential remained shrouded during a century in which idiomatic instrumental music was left unexplored (Boyden Reference Boyden1990: 63). I anticipate the vocabulary of Simondon here in describing the violin as a ‘leap’, a sudden, spontaneous reconfiguration of technical elements snapping into place that produces an immense – and in this case, historically still latent – virtual field of musical play.
According to Simondon, technical progress is gauged by two criteria: ‘concretization’ and ‘hypertely’. Technical objects entrain a reciprocal causality in self-actualising leaps, functionally synergising or ‘concretising’ technical systems that begin as abstractly ideated, independent elements (Simondon Reference Simondon2017: 43). For instance, the size and position of the violin’s bass bar, affixed to the underside of the top table, is of great consequence to the violin’s sound, yet the bass bar originates as a thickening of the table’s spine, a means of providing internal stability to a carved top (Dilworth Reference Dilworth and Stowell1992: 8). Violin tables are no longer carved, but the bass bar remains, and indeed the appellation ‘bass bar’ can only arise when the emergent relationship with the bass foot of the bridge, and the consequences for the sound, are understood. In unexpectedly and spontaneously adopting two critical functions that become inseparable, the violin’s bass bar fulfils Simondon’s first criterion of technical evolution.
But technical evolution is not yet technical progress. Another criterion now comes into play: objects must be ‘free in their evolution and not pushed by necessity in the direction of a fatal hypertely’ (Simondon Reference Simondon2017: 58). ‘Hypertely’ refers to a situation of maladaptation caused by narrow specialisation (Simondon Reference Simondon2017: 53). The modern violin is hypertelic: the trade-off for more powerful sound projection driven by the development of overspun strings is a compromised pianissimo, the mellow sound of the early violin with its relaxed, responsive clarity (Boyden Reference Boyden1990: 69). With tighter strings, pressure on the table is increased via the bridge. Luthiers accommodate this increased tension by arching the violin’s neck and bolstering the bass bar to prevent collapse of the table (Dilworth Reference Dilworth and Stowell1992: 11). Bows become longer, mutating from convexity to concavity to reflect a legato ideal different from the détaché style predominant during the Baroque period (Boyden Reference Boyden1990: 71).
If the addition of a chin-rest frees cultivation of a fluid vibrato, while use of a shoulder rest resolves the difficulty of caterpillar-like down-shifting, relieving the left arm of the need to support the instrument, both evolutions must nevertheless be compensated with new pressure at the neck, locking the violin in a more rigid position. Such changes reflect technical evolution, but not optimisation. They are simply variants or ‘subclasses’ instantiating variations of the violin hardware and/or the technique and musical tradition of the player. Thus, as Trueman notes, a ‘fiddle’ is defined by a posture, style and musical tradition in which it evolved (Trueman Reference Trueman1999).
Subclasses of violins and violin technique are not optimisations. This situation can be compared to biological evolution, ‘drifting’ according to conditions satisfying forms of life that have enough integrity to continue existing (Varela etal. Reference Varela, Thompson and Rosch2016: 191). ‘Much of what an organism looks like … is completely underdetermined by the constraints of survival’ (ibid., 196). According to this conception, both violin and violin technique, too, are ‘bricolage’, that is, organised one way rather than another on the basis of possibility. The fact that ‘good sounding’ violins have strong resonances near the D and A strings is not an a priori rule based on optimum design but an expectation produced by experience (Richardson Reference Richardson and Stowell1992: 37). To speculate, perhaps what makes the violin such a fertile instrument, and violinistic technique so rich, is this capacity to drift, the ‘potential for reinvention’, to use Trueman’s locution.
Finally, just as a musical instrument’s ‘edge’ leaves behind a trace of indeterminacy, technical objects never attain perfect concreteness. ‘The last product of technical evolution … is still artificial’ (Simondon Reference Simondon2017: 51). If the crystallisation of the violin in the sixteenth century is an extraordinary leap, an event generating an expansive new field of possibility, it nevertheless remains an abstract, unsaturated potential that must be concretised yet again. And if the ‘violin’ is abstract – if it is, according to Trueman, a superclass – one can productively entertain the suspicion that, with the violin, there is no such optimisation. Appreciating the logic of the ‘edge’ Evens describes, it follows that arresting its free movement – those experiments and variations by individuals that also propel the violin into the contemporary electroacoustic and digital contexts – collapses the indeterminacy that generates music.
4. ELECTROACOUSTICS AND DIGITAL SIGNAL PROCESSING
Simondon’s description of technical evolution has been highly productive for my augmented violin practice. Trueman shows that the term ‘violin’ is underdetermined. Simondon shows that it is ‘abstract’. This is encouraging to abductive experimental practice, as one learns to appreciate and anticipate ‘the realization of one process … [in] the virtualities and potentialities of another’ (Dumouchel Reference Dumouchel, Feenberg and Hannay1995: 259). A materialist perspective looks for the virtual and vital synergies across the variegated strata of the human posture, violin, electricity, sensors and signal processing.
Pre-digital, electroacoustic techniques explore the violin’s virtual potential. A microphone and electricity are the initial elements. With amplification, the modern violin can bend towards a Baroque predilection for transparency and clarity. At the limit, there are the ecological violins of Bennett Hogg, whose murmurings he records as they are floated in rivers or dragged through undergrowth like pitted Aeolian harps (Hogg Reference Hogg2013). Spatial patterns may be explored, as Annea Lockwood does in ‘Deep Dream Dive’, calling for at least seven microphones on or near the violin, amplifying and remixing the different spatial resonances of the instrument as well as the quiet sounds of touching and brushing the violin’s body (Strange and Strange Reference Strange and Strange2001: 206–7). Perry Cook and Dan Trueman have analysed these spatial patterns in depth (Cook and Trueman Reference Cook and Trueman1999).
Signal processing offers still more potential for exploration of the violin’s virtual synergies. A frequently explored technique is the use of pitch detection. Violinist and composer Mari Kimura has produced a large repertoire of pieces based on this procedure. In ‘ECO II’, for instance, Kimura uses pitch tracking to drive dynamic changes to the violin’s morphology as she plays the piece. In his organology of ‘virtual violins’, Trueman notes that signal processing alone, which remains ‘external’ to the violin, does not change the performer’s physical relationship to the violin in the way that physical changes to the violin do (Trueman Reference Trueman1999). Physical controllers can be designed to ‘invite the performer to play the instrument differently’ or simply keep ‘close watch’ over the violinist without changing the physical relationship to the instrument, potentiating ‘measurement’ rather than ‘transformation’ (ibid.).
An advanced practitioner exploring the violin’s virtualities today is Dan Overholt, who has created augmented violins that preserve conventional technique while affording opportunities for fresh musical expression. His ‘overtone violin’, for instance, invites the nuanced gestures of skilled violinists while leveraging augmentation with sensors better suited to parametric control of audio effects (Overholt Reference Overholt2005). Overholt subsequently extended this design by creating a new model, the ‘overtone fiddle’, that embeds sonic actuators in the body of the violin, making the unique morphology of the instrument achieved with the use of signal processing not an ‘external’ sensation, but one that the performer can actively feel (Overholt Reference Overholt2011). This instrument is an ‘actuated musical instrument’, that is, an instrument that ‘produce[s] sound via vibrating element(s) that are co-manipulated by humans and electromechanical systems’, a concept that Overholt helped to pioneer (Overholt, Berdahl, and Hamilton Reference Overholt, Berdahl and Hamilton2011: 155). The haptic and acoustic feedback of actuated instruments empowers the performer to interact with them differently due to the more intuitive and inviting means of interaction.
To limit the restraints on feedback signals generated by signal processing, Overholt’s violins use optical or magnetic pickup systems to detect string vibration. His instruments are therefore ‘composed instruments’ in the strong sense that the sound-producing and gestural input parts of the instrument are radically decoupled (Schnell and Battier Reference Schnell and Battier2002). Actuated instruments, which hyperbolise the dynamic potential for modulating sensorimotor learning, are potentially strong instantiations of what I am referring to as the enacted symmetry of action and perception in multimodal DMIs.
5. IMPROVISING AN AUGMENTED VIOLIN
Similar to Overholt’s approach, I developed and patented a shoulder rest with embedded actuators that can be coupled to the output of my sound processing software. I now present this augmented instrument as an example of DMI development following a minor scientific approach. I describe the shoulder rest; a sensor glove I built; the first generation of my signal processing software, Windowless; and a later generation of the software built in Max for Live (M4L), Transference. The transition from the former to the latter system reflects another important aspect of minor science: minor scientific practices turn out to be vital sources for royal science’s production of categories, constants and models (Deleuze and Guattari Reference Deleuze and Guattari1987). Through improvised performance with my violin and ongoing tinkering with the design and mechanics of the hardware and software systems, the abstract musical intentions with which I began this project have coalesced into a more concrete set of novel techniques and approaches.
5.1. Hardware: shoulder rest
A shoulder rest is an ergonomic device that raises the violin above the collarbone, promoting easier shifting and reduced neck strain. In the scope of the violin’s 500 years of development, this accessory is nascent and sometimes controversial, to the extent that it would inhibit, as violin virtuoso Aaron Rosand states, ‘the violin becom[ing] part of your body’ (Niles Reference Niles2014). With my colleague Byron Lahey, we added two amplified voice coils to the body of a shoulder rest that I couple to the output of my Windowless and Transference systems (Thorn and Lahey Reference Thorn and Lahey2019) (Figure 2). Adding voice coils to the shoulder rest in the context of digital hybridity can only shore up Rosand’s desideratum: as an emergent concretisation synergistically adopting multiple functions, it extends the violin’s cognitive architecture by increasing the sophistication of the ‘sensory apparatus’ (Capra and Luisi Reference Capra and Luisi2014: 254).
A modified violin shoulder rest with embedded voice coils developed by the author.
Since my shoulder rest is intended as an addition to the acoustic violin, I encounter the feedback problems Overholt avoided with his overtone fiddle. While the musical potential of feedback self-oscillations can be explored in acoustic instruments augmented with actuators, this is not something I have actively explored. Rather, I have sought to minimise feedback by carefully composing the vibro-tactile dynamics of the shoulder rest through use of a dedicated return track to selectively route audio signals to it. Notch filtering near 440 Hz and 293 Hz (A and D strings) is also useful for this purpose. With thoughtful tuning, the shoulder rest increases the sensorimotor cohesiveness and multimodal integration of the augmented violin in performance.
5.2. Hardware: glove
I use a sensor glove I designed, the ‘alto.glove’, which uses an inertial measurement unit (IMU), vibro-tactile feedback, and strategically placed force-sensitive resistors (FSRs) and flex sensors to track motion of the right hand (Thorn Reference Thorn2018) (Figure 3). The FSRs are positioned to allow voluntary actuation within the otherwise limited spare ‘bandwidth’ of the violinist’s bowing hand during performance (Cook Reference Cook2001: 5). A flex sensor on the wrist approximates the distance of the right hand from the violin, while a flex sensor on the small finger tracks the continuous horizontal pivoting of the bow. The circuit board rests atop the forearm below the wrist, secured in place by a metal bracket and Velcro-elastic strip latching over a row of momentary contact buttons (which thereby demonstrate a moment of concretisation: a synergistic relationship between the buttons and latching mechanism).
Alto.glove, a wireless sensor glove for violin developed by the author.
5.3. Signal processing: Windowless
While I explored other approaches to feature extraction, including supervised learning using the ‘Wekinator’ software (Schedel and Fiebrink Reference Schedel and Fiebrink2011), I chose to avoid an approach that removes the need to continuously hammer out in code, as it were, the relationship between the sensor data and violin playing. The purpose of machine learning is to codify precisely those relationships that resist procedural codification in order to generate a model. By contrast, my approach is to intervene in that code at a low level by creatively evolving ambient nets of operators, using Max MSP, that texture, combine and transform audio signals and sensor data into rich, dynamic sound.
The organisation of my Windowless system into three standalone applications reflects the ad hoc development of this computational machinery and its layered evolution according to the affordances of parallel processing. One of these applications receives data from the glove, performs feature extraction and uses open sound control (OSC) to send the data to two other applications performing sound processing. Raw sensor data is sent as well, so that novel features can be constructed extemporaneously in tandem with the hard-coded features. The system evolved into a total of 12 sound ‘modules’ of varying sonic density and complexity, offering a large number of parameters that can be varied and finely tuned to condition the overall sonic response of the system. As the system grew, I pared down its parametric complexity by constraining each module to a set of ten possible unique parameter configurations (‘presets’) to be recalled.
Rather than wrapping the complexity of the successive abstractions in the software from raw sensor and audio data, to numerically reprocessed and normalised data, feature vectors, mapping and audio synthesis, any of these inputs and outputs can be ‘linked up’ – another locution Deleuze and Guattari use to describe processual following (Deleuze and Guattari Reference Deleuze and Guattari1987: 373) – in unforeseen ways, increasing the combinatorial possibilities that make systems fertile for the emergence of novel structures. The concrete structural results can be of such complexity as to evade conceptual explication, just as nonlinear and complex mappings in DMIs create co-contingencies that eliminate independent variables, but thereby better reflect the properties of acoustic instruments and compel exploratory, experimental musicianship and improvisation (Evens Reference Evens2005: 165–6). Wherever the hard edges of formalisation exist in DMIs – such as classification features or parameter configurations – the materialist perspective suggests that the richest combinatorial possibilities emerge from putting these structures back in touch, in ad hoc ways, with the soft, continuous substrata out of which they develop, just as mineralised bone combines with soft tissue to create radical new forms of motility (De Landa Reference De Landa2000: 26). Such is the ‘geological’ situation of the more or less ‘hard’ acoustic violin and the ‘fleshy’ human violinist, the substrata out of which the digital augmentations emerge, but with which they continue to interact and combine.
I describe the 12 modules of Windowless in the following. While many of these effects are ‘linked up’ in highly specific ways with particular bowing techniques, a few are less overtly adapted to violinistic gesture. The point in enumerating all of them is to give a thorough account of the performance seen in Video example 1 that accompanies this article. I use most or all of these modules simultaneously, carefully mixed, in performance.
5.3.1. ‘Polyphonic bow freeze’
This module responds to the detection of sustained up-bows, down-bows or both. When a new stroke begins, a timer begins counting. When it crosses a threshold, the microphone signal from the violin is captured and sustained with a spectral ‘freeze’. This signal is controlled with an adjustable attack, delay, sustain and release (ADSR) amplitude envelope, with the release occurring after the stroke stops or reverses direction. The freeze effect is polyphonic, so that several ‘freezes’ with independent amplitude envelopes, as well as individual tremolo effects, can occur simultaneously. I limit the polyphony in the software to two voices and utilise voice stealing. Harmonization, amplitude modulation, emulated tape warble and reverberation effects are available at the end of the signal chain.
Unique parameter configurations for this module are labelled thus: ‘up-bow/500 ms/minor’, which captures the sound after an up-bow stroke is sustained for 500 milliseconds. The signal is pitch-shifted to create a minor harmonisation. The configuration ‘bidirectional/25 ms/fifth’ gives more sporadic results, since the frozen sound rapidly follows changes to bowing direction and may capture some of the acoustic noise that occurs at bowing onsets. ‘Fifth’ is shorthand for pitch-shifting decisions using octaves and fifths. Shorter trigger times allow less deliberate control, while longer times allow methodical capture and deliberate use of harmonic layering.
Whenever I create an effect that uses ‘triggering’ in some fundamental way according to a discrete threshold, I try to imagine ways to recuperate gestural nuance and continuous control of the triggered sound. One way to do this is to couple the microphone amplitude to the gain of the effect, the speed and the depth of a global tremolo effect, or other means of textural nuancing. I am not ideologically opposed to ‘triggered’ sound or always in favour of continuous ‘gestural’ sound – novelty emerges in combining these strata. Aesthetically, both afford distinctive sonic experiences as incremental or continuous spectral motion (Smalley Reference Smalley2007: 45).
5.3.2. ‘Arvo’
This module uses a noise gate to detect onsets of sound, whether from the violin or the environment – hence the signal processing is ‘ecosystemic’ (Di Scipio Reference Di Scipio2003) – to advance a counter controlling sequential recording into one of 16 buffers. Buffer playback is looped for a period of time determined by global attack and release parameters. Buffer looping can be sustained by pause and resume messages. Patterning is created by varying the sample rate of the playback for individual buffers sequentially (e.g., by continuously repeating a sequence such as half-speed, double-speed, normal speed). A signal routing matrix applies audio effects to buffers according to their playback speed. Actuating the FSR near the index finger on the alto.glove briefly passes the audio signal from the module into a reverberator just before clearing the buffers.
Adjusting the ‘hold’ time of the noise gate dramatically changes the texture of the sound generation. If my original intention with this module was to methodically capture violin ‘notes’ in a deliberate way, the exploration of short ‘hold’ times generating sporadic responses affirms an ‘anti-proceduralist’ understanding of play, namely that it is not the prior intention of a designer but rather the actual play of a user that defines the meaning of a game or an instrument (Sicart Reference Sicart2011). This point reaffirms my motivation to design model-free, responsive sonic media.
5.3.3. ‘FM + granular string crossings’
String crossings are detected by tracking a windowed average of the absolute difference between gyroscopic x-axis peak and trough values. When crossings are detected, a glissando effect is produced by using a phasor sweeping a wide frequency range to drive and modulate the frequency of a square wave. The phasor frequency is controlled by an unusual ‘many-to-one’ mapping (Hunt and Wanderley Reference Hunt and Wanderley2002: 99) that combines the amplitude envelope of the microphone signal with the time interval between zero-crossings of the gyroscopic x-axis, with the latter roughly indicating how rapidly the string crossings are being performed and/or the breadth of the string crossings (i.e., if two, three or four strings are being played). The frequency of the string crossings is multiplied by the amplitude of the signal from the microphone, with the result coarsely rounded off to create arbitrary stepwise changes in intensity. If the resulting value is greater than a predefined value, that result is subtracted from it, otherwise no further calculation is performed. This creates a fickle threshold that unexpectedly changes the sound generation according to a combined metric of bowing vigour and microphone amplitude. As I will explain in the next section, such unexpected – yet non-random – behaviours are fertile ground for improvisation.
5.3.4. ‘Stockhausen’
This module is based on a 50-voice polyphonic pulsar synthesis engine I coded, following Curtis Roads’s description, that probabilistically generates extended pulsar trains coupled to a resonant bandpass filter (Roads Reference Roads2001: 137–57). Upper and lower magnitudes can be selected for frequency, duty cycle, pulse probability, ‘burst’ patterns, duration, amplitude and panning trajectory. Audio output from the pulsar engine is passed through a delay line with feedback. The delay mixing is controlled by the duration of a sustained bow stroke. Stroke duration is coupled to multiple parameters, including amplitude modulation of pitch-shifted sound that varies according to changes in bowing direction, and delay modulation of the pulsar synthesis. The FSR on the index finger of the sensor glove triggers a pulsar train but also affects reverberation added to the signal. There are many more correlations; ‘Stockhausen’ is a complex module with highly interdependent, ‘many-to-many’ instrument mappings.
5.3.5. ‘Holst’
Triggering this module performs a spectral ‘freeze’ that moves through a spatial choreography. Amplitude and panning trajectories follow a sequential threefold path: an initial attack with a fixed panning position, followed by a sustain portion of the envelope during which the sound pans either fully left or fully right, with the release occurring as soon as the sound reaches an extreme in the stereo field. The sequence is triggered by the detection of bowing tremolo, which is measured using a windowed average of time intervals between zero-crossings of the gyroscopic z-axis. There is no low-pass filtering or minimum interval set, with the result that very fine jitters of the right hand can also trigger the effect. A signal routing matrix applies differential audio effects to the three stages of the envelope.
5.3.6. ‘Pulsar bow’
A pulsar synthesiser with dynamic pulse train sequences sent to left and right stereo outputs passes through two resonant filters and a distortion effect. Extension of the violinist’s right wrist typically corresponds to bow contact with strings at the tip of the bow. This is mapped to the frequency of the pulse trains. Pulse frequency is inversely coupled to duty cycle, the centre frequency of the resonant filter and the intensity of the distortion.
5.3.7. ‘Cello bow’
Input from the microphone is dropped one octave with a pitch-shifter. The amplitude of the resulting signal is controlled by the extension of the wrist. The signal passes through a reverberator. The reverberation mix is also controlled by the extension of the wrist, but the mapping is inverted such that the sound becomes dryer with greater extension. Moving away from the tip of the bow, the output gain of the reverberator increases.
5.3.8. ‘Pizzicato reverb’
A bow grip indicative of a pizzicato technique causes input from the microphone to pass through two pitch-shifters, one dropping the input by a fifth, and the other by two octaves. The signal passes through a reverberator with a very long decay. The input gain attenuates slowly after the violinist restores an arco grip, skewing the mapping correlation in a way I find rich in improvised performance.
5.3.9. ‘Bow reverb’
Violin sound is pitch-shifted down one octave and passed through a reverberator with a long decay. A sustained bow stroke begins a timer that slowly increases the output gain from the reverberator. Differential scaling causes additional pitch-shifting to be mixed into the signal at different rates as the stroke is sustained further.
5.3.10. ‘Stuttered octaves + delay’
A signal capture buffer produces granular playback of recorded samples. Sound grains are pitched up three octaves. The signal is passed into a routing matrix, the outputs of which are algorithmically delayed and panned.
5.3.11. ‘Pitched sinusoid bass’
A pitch-detection algorithm tracks audio from the microphone. Estimated frequencies above 110 Hz are divided in half until the result is equal to or less than 110. This sets the pitch of an overdriven waveform, with the gain set by the amplitude envelope from the microphone.
5.3.12. ‘Matrix delay’
Live violin sound is mixed with pitch-shifted sound and passed into a delay line with eight tap points. An attenuated signal from the last tap is low-pass filtered and recirculated into the delay line. The eight tap points pass into a signal routing matrix that reroutes the delayed signals through eight channels. The routing is algorithmically controlled with adjustable speed, decay and ramp values, essentially making the device a crude multichannel reverberator. Decay and ramp values are inversely coupled to bowing intensity.
5.4 Signal processing: Transference and generalisation
Building Windowless was a meticulous process of designing signal processing from the ground up by intercalating multiple orders of sensor input, feature extraction and audio processing in the 12 composed sound modules. But there are practical obstacles to being both coder and violinist. Symmetrising action and perception, in this case, denotes picking up the violin, playing, listening, ideating, then setting the violin down again to concretely reprogram and adjust the instrument using, typically, a keyboard and mouse. A different outcome could result from this process of refining the instrument if that process were less cumbersome.
To address this, I began building a new system, Transference, that I constructed using Ableton Live and bespoke M4L devices. Transference inherits many of the modules I created for Windowless as individual M4L devices. It also inherits signal processing techniques I created and employed often when mapping bowing data to audio effects, such as ‘temporalisation’ of the bow (so that a sustained bow stroke sequentially activates different signal processing), measures of periodicity generated by IMU zero-crossings, and ‘capture’ processes performing spectral ‘freezes’ or activating buffer recording and looping playback. I embedded these techniques in a set of M4L devices for mapping that generalise the tactics for control signal shaping that I followed in Windowless. It is still possible to do fine coding and elaborate ‘linking up’ in bespoke M4L devices, but it is also possible to quickly drag-and-drop or rearrange different modules and effects in the signal chain, as well as rapidly reconfigure parameters shaping the signal. Effective ‘one-to-many’ mapping (Hunt and Wanderley Reference Hunt and Wanderley2002: 99) as well as mitigation of some of the ‘linear design ontology’ (Di Scipio Reference Di Scipio2003: 270) present in Live are facilitated with a module that takes a single control signal and allows reshaping by multiple independent attack and decay envelopes, rescaling, and mapping to any parameter in the Live Session. In addition, I found it useful to add a parametric control to these modules that emulates the incoming control signal from the alto.glove using an onscreen control.
As I embedded these features into my M4L devices, I also imagined ways that this sensitivity to violinistic bowing gesture might be generalised to broader contexts of movement and sound computing. Can the system inform development of other such systems, retaining its refined mapping and signal processing while becoming less hypertelic? The quotidian gestures of non-violinists moving in a responsive media space can be legato, détaché, staccato or tremolo, too.
Speculating about decoupling violinistic gesture from violin playing touches on Trueman’s useful notion of the ‘virtual violin’, a concept he uses to elucidate more radical interventions by experimental violinists, such as Laurie Anderson’s ‘Tape-bow violin’. According to Trueman, ‘the virtual violin is a subclass of the violin superclass … [that] generates a set of data that can be mapped to essentially anything; it is therefore without acoustical constraints and could just as easily produce lighting effects as it could sound’ (Trueman Reference Trueman1999).Footnote 1
Thinking of this ‘virtual violin’ as decoupled violinistic gesture more broadly, I created an additional mapping device decoupled from the more idiosyncratic features of my alto.glove (such as the flex sensors and FSRs) that preserves some of the signal shaping techniques I used in Windowless while anticipating further ways to work with and refine IMU data, such as aggregations of angular velocity, modulation of a signal by intercalation with angle data (‘many-to-one’ mapping), and other transformations. Following the implications of conceptualising my system as an extended ‘aural architecture’, I built a framework allowing input from multiple IMUs across one or more bodies – violinistic movement from arms, legs, shoulders or an orchestral assemblage of such virtually violinistic movers – along with statistical measurements traversing multiple sensors modulating global audio processes (Thorn, Willcox and Sha Reference Thorn, Willcox and Sha2020). This follows from thinking of this digital aural architecture, or software acoustics, as not only reflecting sound but also reflecting and responding to gesture. Broadly considering this notion of the ‘virtual’ thus carries my practice forward.
6. CONCLUSION
The concepts from materialist thinking that I have put to work in this article – minor science, following, abstraction, concretisation, hypertely – enable a unique mode of thinking about the production of DMIs, and may be useful signposts, as they have been for my practice, for digital tinkerers in search of novel and highly refined results. The practice of improvisation in music is imbued with that spirit: we look for ways in which music might be prodded, all of a sudden, to erupt into something new and unfamiliar.
Such is the value, as Derek Bailey writes, of improvisation qua ‘method of working’ (Bailey Reference Bailey1992: 142). Improvisation is abductive, molecular and singular. In the context of designing interactive systems for improvisation, this would be the fundamental justification for refraining from embedding assumptions about intention, telos, or agency into the system by pre-schematising sound or gesture. Bailey differentiates ‘improvisation’ from ‘composition’ according to the constraint of time, with the improviser having to formulate a response without the buffer of calculated reflection. Improvisation is constrained. But Bailey’s position also entails that these constraints are the means of its realisation. Such is the path of technical evolution, which Simondon cogently summarises thus: ‘What was once an obstacle must become a means of realization’ (Simondon Reference Simondon2017: 32–3).
With his conception of the musical instrument’s ‘resistance’, Evens draws out the consequences of this for the improvising musician. Constrained time is not the only fertile obstacle; the instrument itself presents a material and conceptual resistance with which the musician productively engages in a struggle of ‘accord and discord, push and pull’ that generates the music (Evens Reference Evens2005: 161). In the context of DMI development, the upshot is that resistance must be cleverly devised rather than stamped out – ‘to symmetrize action and perception’ does not mean to smooth out the instrument by making everything exceedingly transparent.
The need to craft resistance is reflected in the decisions I make about the parameterisation and mapping of my augmented violin’s dense sonic physics. Taking stock of the incremental decisions I have made over time through abductive experiments, I observe a trend whereby I ‘loosen up’ the otherwise tight coupling between input and output, gesture and sonic response.
For instance, as mentioned previously, the ‘pizzicato reverberation’ response is offset by a short delay that allows some initial plucks to bypass the reverberator while continuing to affect the return to arco playing. In the context of electroacoustic composition, Roads describes the appeal of ‘ambiguous beginning and ending times’, the ‘fuzzy timing’ of the sonic material as it mutates spectromorphologically (Roads Reference Roads2015: 287). A similar appeal is evident with ‘fuzzy mapping’, as it were, which slackens the coupling between the electronic sound and familiar technical categories of violin playing. This hysteresis fruitfully prolongs gesture: a style of playing affecting the response resonates for a bit longer than it otherwise would, and this creates an opportunity for playful engagement with something unfamiliar. Nothing random is injected here; the effect works because it potentiates unique expressive results without rendering the kinesics of the instrument unlearnable.
Likewise, the ‘hold’ parameter of the noise gate in ‘Arvo’ generates surprising sonic outcomes at thresholds that less reliably parse distinctive note onsets. I like to run two of these modules in parallel, with one entrained to more deliberate note onsets while the other triggers more sporadically. By actuating the FSR that activates reverberation in both units and clears their buffers, I am able to use ‘reverberant space as a cadence’, a convincing means of phrasing in electroacoustic music (Roads Reference Roads2015: 189–90). This tool is tremendously valuable in improvisation; it navigates the risk of stagnation in live improvisation by articulating clear phrasing. The dialectic it potentiates between resistance and recovery – the instrument escaping you and you reining it in – allows for playful emergence of spontaneous musical form.
In digital improvisation, abandoning simple, linear models such as ‘pizzicato activates reverberation’ and ‘note onsets activate loop recording’ in the design engenders a conceptual resistance encouraging more pre-reflective engagement. The instrument demands a more assiduously probing touch and careful listening.
Alongside tools aiding the articulation of clear phrasing, another important element in live improvisation is surprise, insofar as it challenges the performer’s sense that things are more or less predetermined. Surprise tends to prompt moments of musical intensification. For instance, the polyphonic pulsar synthesiser I coded synthesises percolating, ebullient textures that bubble up and decay according to subtle changes in bowing motion or audio input. This mapping arouses a sensation of volatility, an instrument that can never quite find its ‘balance’, as it were, but is all the more technically engaging for that very reason. The emergence of these different textures – which rise in amplitude and provoke a response by the performer that cuts them short – occasions abrupt intensifications, a precipitous ‘forward impetus’ emerging from textural stasis (Smalley Reference Smalley1997: 113–14).
In the context of digital improvisation, error is also useful to this end – the breaking down of the predictable behaviour of an instrument or system (Evens Reference Evens2005: 167). For instance, the string crossing feature I constructed is based on the analysis of rhythmic changes of the x-axis of the gyroscope; sequential activations toggle between a noisy live granulator and ‘glisson synthesis’ (Roads Reference Roads2001: 121–5). When improvising with the system, this feature would sometimes be activated unexpectedly, yet I found this to be advantageous: ‘loose’ system dynamics such as these allow for unanticipated responses without introducing mere randomness.
The word ‘error’ itself is suggestive of the ethos of following a flow: ‘to err’ is ‘to stray’, to follow an unanticipated course that negotiates – like Bergson’s zigzagging canal – more abstract intentions with concrete discoveries made along the way. In ‘circuit bending’, a technique I practice and teach, error becomes method: techniques such as ‘voltage starving’ tease out subtleties and surprises from the electrical substrates of these circuits by thwarting their symbolic operations (Collins Reference Collins2009). Similarly, I employ pitch estimation techniques on signals I have deliberately attenuated, yielding splendidly erratic results. Like a potentiometer pulling current and voltage from a CMOS chip, an abductive experiment with input gain to a pitch detector uncovers a highly expressive, sensitive musical parameter.
The numerous dynamic affordances for creating spontaneous form and sudden intensifications acquire an even greater charge when they become densely layered together as a single instrument. This is why I describe my augmented violin, with its thick enveloping diffusion, as a rich ‘aural architecture’. As the individual potentials of the various modules and intricate mappings accumulate, the instrument passes a critical threshold. There is a sense that something else has emerged; the augmented violin seems to acquire properties that are not present in its individual parts.
Tracing these variegated pathways of accumulated digital logic that make up my augmented violin yields an ecosystemic diagram, a ‘patchwork of subnetworks’ – dense and variegated strata developed through processual trial-and-error tinkering. But the project began more abstractly, and I ideated in less subtle terms: I would analyse discrete note onsets, pitches, bowing styles and the like. But the actual material practice of symmetrising action and perception is a much finer, less semantic process. The instrument is ‘interesting’ – in the strict etymological sense – insofar as I followed a path, in constructing it, ‘between’ those more familiar categories.
Part of the thrust of materialist thinking is that the thinking itself will be appropriated in ways it could never anticipate, enriching minor practices that evolve with its concepts. In the context of this article, it means articulating and advocating the work of those individual musicians and collective musical entities who now have the means to abductively build up their own experimental instruments and improvisation practices in a ceaseless digital lutherie following flows of inhomogeneous matter. Minor science recalls ancient artisanal practices from which modern computing – achieving ‘quasi-continuous operations’ approximating the density of physical matter (McCullough Reference McCullough1996: 214) – is not excluded. Thus the ‘reappearance’, as Jacques Attali wrote presciently 35 years ago, ‘of very ancient forms of production’ (Attali Reference Attali1985: 140) by means of the most advanced and refined real-time signal processing symmetrising action and perception.
SUPPLEMENTARY MATERIAL
To view supplementary material for this article, please visit https://doi.org/10.1017/S1355771821000066
Open access
