Skip to main content
×
×
Home

A Manual Actions Expressive System (MAES)

  • Rajmil Fischman (a1)
Abstract

This article describes a manual actions expressive system (MAES) which aims to enable music creation and performance using natural hand actions (e.g. hitting virtual objects, or shaking them). Gestures are fully programmable and result from tracking and analysing hand motion and finger bend, potentially allowing performers to concentrate on natural actions from our daily use of the hands (e.g. the physical movement associated with hitting and shaking). Work carried out focused on the development of an approach for the creation of gestures based on intuitive metaphors, their implementation as software for composition and performance, and their realisation within a musical composition through the choice of suitable mappings, sonic materials and processes.

Copyright
References
Hide All
Bencina R. 2006. P5 Glove Developments. http://www.simulus.org/p5glove (accessed on 24 December 2012).
Berger M. 2010. The GRIP MAESTRO: Idiomatic Mappings of Emotive Gestures for Control of Live Electroacoustic Music. Proceedings of the Conference on New Interfaces for Musical Expression. Sydney. http://www.nime.org/proceedings/2010/nime2010_419.pdf (accessed on 14 January 2013).
Berthaut F., Katayose H., Wakama H., Totani N., Sato Y. 2011. First Person Shooters as Collaborative Multiprocess Instruments. Proceedings of the International Conference on New Interfaces for Musical Expression, 2011. Oslo, 44–7. http://www.nime.org/proceedings/2011/nime2011_044.pdf (accessed on 8 March 13).
Beyer G., Meier M. 2011. Music Interfaces for Novice Users: Composing Music on a Public Display with Hand Gestures. Proceedings of the International Conference on New Interfaces for Musical Expression, 2011. Oslo, 507–10. http://www.nime2011.org/proceedings/papers/M23-Beyer.pdf (accessed on 20 January 2013).
Bongers B. 2007. Electronic Musical Instruments: Experiences of a New Luthier. Leonardo Music Journal 17: 916.
Cadoz C. 1988. Instrumental Gesture and Musical Composition. Proceedings of the 1988 International Computer Music Conference. Cologne/San Francisco: ICMA, 1–12.
Cadoz C., Luciani A., Florens J. 1984. Responsive Input Devices and Sound Synthesis by Simulation of Instrumental Mechanisms: The Cordis System. Computer Music Journal 8(3): 6073.
Cadoz C., Ramstein C. 1990. Capture, Representation and ‘Composition’ of the Instrumental Gesture. Proceedings of the 1990 International Computer Music Conference. Glasgow/San Francisco: ICMA, 53–56.
Charles J.F. 2008. A Tutorial on Spectral Sound Processing Using Max/MSP and Jitter. Computer Music Journal 32(3): 87102.
Choi I., Bargar R., Goudeseune C. 1995. A Manifold Interface for a High Dimensional Control Space. Proceedings of the 1995 International Computer Music Conference. Banff/San Francisco: ICMA, 385–92.
Comajuncosas J.M., Barrachina A., O'Connell J., Guaus E. 2011. Nuvolet: 3D Gesture-Driven Collaborative Audio Mosaicing. Proceedings of the International Conference on New Interfaces for Musical Expression, 2011. Oslo, 252–4. http://www.nime2011.org/proceedings/papers/G11-Comajuncosas.pdf (accessed on 23 January 2013).
Cook P., Leider C. 2000. Making the Computer Sing: The SqueezeVox. Proceedings of the XIII Colloquium on Musical Informatics. L'Aquila.
De Poli G., Piccialli A., Roads C. 1991. Representation of Musical Signals. Cambridge, MA: The MIT Press.
de Quay Y., Skogstad S. 2010a. Dance Jockey. http://www.youtube.com/watch?v=m1OffxIArrA (accessed on 19 December 2012).
de Quay Y., Skogstad S. 2010b. Dance Jockey v0.3. http://www.youtube.com/watch?v=HaD9MJzW59s (accessed on 19 December 2012).
de Quay Y., Skogstad S. 2011. Dance Jockey @ NIME. https://www.youtube.com/watch?v=PgE6YxhHgvM (accessed on 19 December 2012).
DuBois R.L. 2012. Interviews with Matthew Ostrowski: The Glove. http://cycling74.com/2012/10/16/interviews-with-matthew-ostrowski (accessed on 25 February 2013).
Essl G., O'Modhrain S. 2006. An Enactive Approach to the Design of New Tangible Musical Instruments. Organised Sound 11(3): 285296.
Fels S., Hinton G. 1998. Glove-TalkII: A Neural-Network Interface which Maps Gestures to Parallel Formant Speech Synthesiser Controls. IEEE Transactions on Neural Networks 9(1): 205212. http://www.cs.toronto.edu/~hinton/absps/glovetalkii.pdf (accessed on 20 December 2012).
Fels S., Gadd A., Mulder A. 2002. Mapping Transparency Through Metaphor: Towards More Expressive Musical Instruments. Organised Sound 7(2): 109126.
Fischman R. 1997. The Phase Vocoder: Theory and Practice. Organised Sound 2(2): 127145.
Fischman R. 2011. Back to the Parlour. Sonic Ideas – Ideas Sónicas 3(2): 5366.
Fischman R. 2012. Ruraq Maki (2012). https://vimeo.com/55093629 (accessed on 17 December 2012).
Fischman R. 2013 (forthcoming). MAES, MAX patch and P5GloveRF.mxe external object.
Gadd A., Fels S. 2002. MetaMuse: Metaphors for Expressive Instruments. Proceedings of the International Conference on New Interfaces for Musical Expression, 2002. Dublin. http://hct.ece.ubc.ca/publications/pdf/gadd-fels-NIME02.pdf (accessed on 20 December 2012).
Gillian N., Paradiso J. 2012. Digito: A Fine-Grain Gesturally Controlled Virtual Musical Instrument. Proceedings of the International Conference on New Interfaces for Musical Expression, 2012. Ann Arbor. http://www.eecs.umich.edu/nime2012/Proceedings/papers/248_Final_Manuscript.pdf (accessed on 17 January 2013).
Goto S. 1999. The Aesthetics and Technological Aspects of Virtual Musical Instruments: The Case of the Super-Polm MIDI Violin. Leonardo Music Journal 9: 115120.
Goto S. 2005. Virtual Musical Instruments: Technological Aspects and Interactive Performance Issues. HZ Journal 6. http://hz-journal.org/n6/goto.html (accessed on 24 December 2012).
Goudeseune C., Garnett G., Johnson T. 2001. Resonant Processing of Instrumental Sound Controlled by Spatial Position. New Instruments and Musical Expression workshop, SIGCHI ‘01. Seattle. http://zx81.isl.uiuc.edu/camilleg/nime01.pdf (accessed on 14 December 2012).
Hunt A.D., Paradis M., Wanderley M. 2003. The Importance of Parameter Mapping in Electronic Instrument Design. Journal of New Music Research 32(4): 429440. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.20.3098&rep=rep1&type=pdf (accessed on 14 December 2012).
Jensenius A.R, Godoy R, Wanderley M.M 2005. Developing Tools for Studying Musical Gestures within the Max/MSP/Jitter Environment. Proceedings of the 2005 International Computer Music Conference. Barcelona/San Francisco: ICMA, 282–5.
Jordá S. 2005. Digital Luthery: Crafting Musical Computers for New Musics, Performance, and Improvisation. PhD dissertation, Universidad Pompeu Fabra (Barcelona).
Kenner C. 2010. GlovePIE. http://www.glovepie.org (accessed on 25 February 2013).
Kiefer C., Collins N., Fitzpatrick G. 2009. Phalanger: Controlling Music Software With Hand Movement Using A Computer Vision and Machine Learning Approach. Proceedings of the International Conference on New Interfaces for MusicalExpression, 2009. Pittsburgh: 246–9. http://sro.sussex.ac.uk/2202/1/ChrisKieferNIME2009Paper.pdf (accessed on 8 March 13).
Krefeld V. 1990. The Hand in the Web: An Interview with Michel Waisvisz. Computer Music Journal 14(2): 2833.
Levitin D.J., McAdams S., Adams R.L. 2002. Control Parameters for Musical Instruments: A Foundation for New Mappings of Gesture to Sound. Organised Sound 7(2): 171189.
McMullan J. 2003. libp5glove. P5 Glove library for low level access of raw data. http://noisybox.net/computers/p5glove/libp5glove_svn_20050206.tar.gz(accessed on 12 December 2012).
McMullan J. 2008. P5glove. http://noisybox.net/computers/p5glove/ (accessed on 24 December 2012).
Miranda E.R., Wanderley M.M. 2006. New Digital Musical Instruments: Control and Interaction Beyond the Keyboard. Middleton, WI: A-R Editions.
Mitchell T., Heap I. 2011, SoundGrasp: A Gestural Interface for the Performance of Live Music. Proceedings of the International Conference on New Interfaces for Musical Expression, 2011. Oslo. http://www.nime2011.org/proceedings/papers/M13-Mitchell.pdf (accessed on January 2013).
Mitchell T., Madgwick S., Heap I. 2012. Musical Interaction with Hand Posture and Orientation: A Toolbox of Gestural Control Mechanisms. Proceedings of the International Conference on New Interfaces for Musical Expression, 2012. Ann Arbor. http://www.eecs.umich.edu/nime2012/Proceedings/papers/272_Final_Manuscript.pdf (accessed on 11 January 2013).
Momeni A., Henry C. 2006. Dynamic Independent Mapping Layers for Concurrent Control of Audio and Video Synthesis. Computer Music Journal 30(1): 4966.
Momeni A., Wessel D. 2003. Characterizing and Controlling Musical Material Intuitively with Geometric Models. Proceedings of the International Conference on New Interfaces for Musical Expression, 2003. Montreal. http://www.nime.org/proceedings/2003/nime2003_054.pdf (accessed on 14 December 2012).
Mulder A. 1994. Virtual Musical Instruments: Accessing the Sound Synthesis Universe as a Performer. Proceedings of the First Brazilian Symposium on Computer Music: Caxambu, Minas Gerais: 243–50. Online: http://www.xspasm.com/x/sfu/vmi/BSCM1.pdf (accessed on 24 December 2012).
Mulder A.G.E. 1996. Getting a GRIP on Alternate Controllers: Addressing the Variability of Gestural Expression in Musical Instrument Design. Leonardo Music Journal 6: 3340.
Mulder A., Fels S., Mase K. 1997. Empty-Handed Gesture Analysis in Max/FTS. Proceedings of Kansei: The Technology of Emotion, AIMI International Workshop. Genova. http://hct.ece.ubc.ca/publications/pdf/mulder-fels-mase–1997.pdf (accessed on 14 December 2012).
Mulder A., Fels S., Mase K. 1999. Design of Virtual 3d Instruments for Musical Interaction. Graphics Interface ‘99: 7683.
Oliver J. 2010. The MANO Controller: A Video Based Hand Tracking System. http://www.jaimeoliver.pe/pdf/MANO-joliverl.pdf (accessed on 20 December 2012).
Ortiz M., Coghlan N., Jaimovitch J., Knapp B. 2011. Biosignal-Driven Art: Beyond Biofeedback. Sonic Ideas – Ideas Sónicas 3(2): 4352.
Ostrowski M. 2010. Congeries 2. https://soundcloud.com/#harvestworks/matthew-ostrowski-congeries-2 (accessed on 25 February 2013).
Paine G. 2009. Towards Unified Design Guidelines for New Interfaces for Musical Expression. Organised Sound 14(2): 142155.
Roads C. 1996. The Computer Music Tutorial. Cambridge, MA: The MIT Press.
Rovan J.B. 2009. COLLIDE: For Custom Glove Controller, Interactive Computer Music & Interactive Video (2002). http://vimeo.com/2729606 (accessed on 27 February 2013).
Rovan J.B. 2010. Butch Rovan. http://www.soundidea.org/rovan/research_glove.html (accessed on 27 February 2013).
Rovan J.B., Wanderley M.M., Dubnov S., Depalle P. 1997. Instrumental Gestural Mapping Strategies as Expressivity Determinants in Computer Music Performance. Proceedings of Kansei: The Technology of Emotion, AIMI International Workshop. Genova. http://www.ircam.fr/equipes/analyse-synthese/wanderle/Gestes/Externe/kansei_final.pdf. (accessed on 20 December 2012).
Sapir S. 2000. Interactive Digital Audio Environments: Gesture as a Musical Parameter. Proceedings COST-G6 Conference on Digital Audio Effects (DAFx'00): 25–30. http://profs.sci.univr.it/~dafx/Final-Papers/pdf/Sapir.pdf (accessed on 13 December 2012).
Singer E. 2010. Cyclops. http://cycling74.com/products/cyclops/ (accessed on 18 December 12).
Skogstad S, de Quay Y, Jensenius AR 2011. OSC Implementation and Evaluation of the Xsens MVN suit. Proceedings of the International Conference on New Interfaces for Musical Expression, 2011. Oslo, 300–3.http://www.nime2011.org/proceedings/papers/G23-Skogstad.pdf (accessed on 24 December 2012). See also: http://www.youtube.com/watch?v=KI6BF_RRxQs (accessed on 19 December 2012).
Smalley D. 1986. Spectro-Morphology and Structuring Processes. In S. Emmerson (ed.) The Language of Electroacoustic Music. Basingstoke: Macmillan, 6193.
Smalley D. 1997. Spectromorphology: Explaining Sound-Shapes. Organised Sound 2(2): 107120.
Sonami L. 2010. Why __ dreams like a loose engine (autoportrait) (2000–8). http://vimeo.com/11316136 (accessed on 27 February 2013).
Sonami L. 2013. Home Page. http://www.sonami.net (accessed on 27 February 2013).
Spasov M. 2009–10. ENACTIV. http://www.keele.ac.uk/music/people/miroslavspasov/ (accessed on 17 December 12).
Spasov M. 2011. Music Composition as an Act of Cognition: ENACTIV – interactive multi-modal composing system. Organised Sound 16(1): 6986.
Steiner M. 2006. Data Glove Concert. http://www.youtube.com/watch?v=uleNZXNACJM (accessed on 25 February 2013).
Torre G. 2012a. Molitva (2009). http://vimeo.com/44650248 (accessed on 27 February 2013).
Torre G. 2012b. Mani (2010). http://vimeo.com/44762821 (accessed on 27 February 2013).
Torre G. 2012c. Agorá (2012). http://vimeo.com/44650248 (accessed on 27 February 2013).
Torre G. 2013. The Design of a New Musical Glove: A Live Performance Approach. Unpublished PhD thesis. University of Limerick.
Torre G., Torres J., Fernstrom M. 2008.The Development of Motion Tracking Algorithms for Low Cost Inertial Measurement Units. Proceedings of the Conference on New Interfaces for Musical Expression, 2008. Genova. http://nime2008.casapaganini.org/documents/Proceedings/Papers/192.pdf (accessed on 14 January 2013).
Vertegaal R., Ungvary T., Kieslinger M. 1996. Towards a Musician's Cockpit: Transducer, Feedback and Musical Function. Proceedings of the 1996 International Computer Music Conference. Honk Kong/San Francisco: ICMA, 308–11.
Vigliensoni G., Wanderley M.M. 2009. Soundcatcher: Explorations in Audio-Looping and Time-Freezing Using an Open-Air Gestural Controller http://www.music.mcgill.ca/~gabriel/courses/mumt620/Final_Project/SoundCatcher_Final_Report.pdf (accessed on 2 June 2013). See also http://vigliensoni.com/blog/soundcatcher (accessed on 6 February 2013).
Vigliensoni Martin G. 2010. The Enlightened Hands: Navigating through a Bi-Dimensional Feature Space Using Wide and Open-Air Hand Gestures. Proceedings of the International Conference on New Interfaces for Musical Expression, 2011. Sydney. http://www.music.mcgill.ca/~gabriel/courses/mumt619/The_Enlightened_Hands/The_Enlightened_Hands_FINAL_REPORT.pdf (accessed on 6 February 2013). See also http://vimeo.com/10721519 (accessed on 6 February 2013).
Waisvisz M. 2003. Michael Waisvisz: The Hands. http://www.youtube.com/watch?v=SIfumZa2TKY&feature=related (accessed on 27 February 2013).
Wanderley M.M. 2001. Gestural Control of Music. Proceedings of the International Workshop Human Supervision and Control in Engineering and Music. Kassel. http://www.engineeringandmusic.de/individu/Wanderley/Wanderley.pdf (accessed on 13 December 2012).
Waters S. 2000. The Musical Process in the Age of Digital Intervention. Ariada 1. http://www.ariada.uea.ac.uk/ariadatexts/ariada1/content/Musical_Process.pdf (accessed on 24 December 2012).
Wessel D., Wright D., Schott D. 2002. Intimate Musical Control of Computers with a Variety of Controllers and Gesture Mapping Metaphors. Proceedings of the International Conference on New Interfaces for Musical Expression, 2002. Dublin. http://opensoundcontrol.org/files/NIME02WesselWrightSchottDmo.pdf (accessed on 14 December 2012).
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Organised Sound
  • ISSN: 1355-7718
  • EISSN: 1469-8153
  • URL: /core/journals/organised-sound
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×
Type Description Title
VIDEO
Supplementary materials

Fischman Supplementary Material
Movie 2

 Video (4.2 MB)
4.2 MB
VIDEO
Supplementary materials

Fischman Supplementary Material
Movie 3

 Video (3.2 MB)
3.2 MB
VIDEO
Supplementary materials

Fischman Supplementary Material
Movie 4

 Video (75.0 MB)
75.0 MB
VIDEO
Supplementary materials

Fischman Supplementary Material
Movie 1

 Video (3.0 MB)
3.0 MB

Metrics

Full text views

Total number of HTML views: 4
Total number of PDF views: 45 *
Loading metrics...

Abstract views

Total abstract views: 212 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 22nd January 2018. This data will be updated every 24 hours.