Skip to main content

The Limitations of Representing Sound and Notation on Screen

  • Lindsay Vickery (a1)

Animated screen-based notation and visual representation of sound provide an important solution to visualising a range of musical phenomena and techniques including continuous parametrical changes, synchronisation with prerecorded audio or live processing, and nonlinear formal organisation. The limitations of human visual capabilities, however, place some constraints upon the efficacy of screen-based representation, particularly in regard to notation reading. Findings from sightreading studies exploring the manner in which notation is encoded, processed and executed are examined with the aim of identifying the perceptual and practical boundaries of presenting animated notation on screen. The development of efficient visual representation is proposed as an important requirement for alleviating the issues created by the time constraints of reading on screen. Studies in semantics and cross-modal activation are discussed as a foundation for the expansion of approaches to the visualisation of sound.

Corresponding author
Hide All
Adkins, M. 2008. The Application of Memetic Analysis to Electroacoustic Music.
Blackburn, M. 2009. Composing from Spectromorphological Vocabulary: Proposed Application, Pedagogy and Metadata.
Blackburn, M. 2011. The Visual Sound-Shapes of Spectromorphology: An Illustrative Guide to Composition. Organised Sound 16(1): 513.
Cardew, C. 1981. Wiggly Lines and Wobbly Music. Breaking the Sound Barrier: A Critical Anthology of the New Music. New York: Dutton. 235253.
Dannenberg, R. B. 1996. Extending Music Notation Through Programming. Computer Music in Context 13(2): 6376.
Deroy, O. and Spence, C. 2013. Weakening the Case for ‘Weak Synaesthesia’: Why Crossmodal Correspondences are not Synaesthetic. Psychonomic Bulletin & Review 20: 643664.
Dewar, J. 1998. The Information Age and the Printing Press: Looking Backward to See Ahead. Santa Monica, CA: RAND.
Eitan, Z. and Timmers, R. 2010. Beethoven’s Last Piano Sonata and Those Who Follow Crocodiles: Cross-Domain Mappings of Auditory Pitch in a Musical Context. Cognition 114: 405422.
Freeman, J. 2008. Extreme Sight-Reading, Mediated Expression, and Audience Participation: Real-Time Music Notation in Live Performance. Computer Music Journal 32(3): 2541.
George, S. E. 2004. Visual Perception of Music Notation: On-line and Off-line Recognition. Hersey, PA: IRM Press.
Giannakis, K. 2006. A Comparative Evaluation of Auditory-Visual Mappings for Sound Visualization. Organised Sound 11(3): 297307.
Gilman, E. and Underwood, G. 2003. Restricting the Field of View to Investigate the Perceptual Spans of Pianists. Visual Cognition 10(2): 201232.
Goolsby, T. 1994a. Eye-Movement in Music Reading: Effects of Reading Ability, Notational Complexity, and Encounters. Music Perception 12: 7796.
Goolsby, T. 1994b. Profiles of Processing: Eye-Movements during Sight-Reading. Music Perception 12: 97123.
Greco, M., Stucchi, N., Zavagno, D. and Marino, B. 2008. On the Portability of Computer-Generated Presentations: The Effect of Text-Background Color Combinations on Text Legibility. Human Factors. The Journal of the Human Factors and Ergonomics Society 50: 821.
Green-Armytage, P. 2010. A Colour Alphabet and the Limits of Colour Coding. Colour: Design & Creativity 510: 123.
Grill, T. and Flexer, A. 2012. Visualization of Perceptual Qualities in Textural Sounds. International Computer Music Conference 2012, Ljubljana, Slovenia.
Griscom, W. S. and Palmer, S. E. 2012. The Color of Musical Sounds in Non-Synesthetes. 12th Annual Meeting of the Vision Science Society, Naples, FL.
Griscom, W. S. and Palmer, S. E. 2013. Cross-Modal Sound-to-Sight Associations with Musical Timbre in Non-Synesthetes. 13th Annual Meeting of the Vision Science Society, 10–15 May, Naples, FL.
Gunter, T. C., Schmidt, B.-H. et al. 2003. Let’s Face the Music: A Behavioral and Electrophysiological Exploration of Score Reading. Psychophysiology 40: 742751.
Hachman, M. 2010. Analyst Challenges Apple’s iPhone 4 ‘Retina Display’ Claims. PCMAG ,,2817,2364871,00.asp.
Hajdu, G. and Didkovsky, N. 2009. On the Evolution of Music Notation in Network Music Environments. Contemporary Music Review 28(4): 395407.
Honing, H. 2002. Structure and Interpretation of Rhythm and Timing. Dutch Journal of Music Theory (Tijdschrift voor Muziektheorie) 7(3): 227232.
Horn, R. E. 1998. Visual Language: Global Communication for the 21st Century. Bainbridge Island, WA: MacroVU.
Jones, B. 2012. Apple Retina Display (25 August).
Karkoschka, E. 1972. Notation in New Music: A Critical Guide to Interpretation and Realisation. New York: Praeger.
Kim-Boyle, D. 2010. Real-Time Score Generation for Extensible Open Forms. Contemporary Music Review 29(1): 315.
Kinsler, V. and Carpenter, R. H. S. 1995. Saccadic Eye Movements while Reading Music. Vision Research 35(10): 14471458.
Köhler, W. 1929. Gestalt Psychology. New York: Liveright.
Kojs, J. 2011. Notating Action-Based Music. Leonardo Music Journal 21: 6572.
Lee, J. I. 2003. The Role of Working Memory and Short-Term Memory in Sight Reading. The 5th Triennial ESCOM Conference, Hanover University of Music And Drama, Germany.
Locher, E. J. and Nodine, C. G. 1974. The Role of Scanpaths in the Recognition of Random Shapes. Perception and Psychophysics 15: 308314.
Marks, L. E. 1996. On Perceptual Metaphors. Metaphor & Symbolic Activity 11: 3966.
Marks, L. E. and Odgar, E. C. 2005. Developmental Constraints on Theories of Synesthesia. Synesthesia: Perspectives from Cognitive Neuroscience. New York: Oxford University Press, 214–36.
Martino, G. and Marks, L. E. 2001. Synesthesia: Strong and Weak. Current Directions in Psychological Science 10: 6165.
McClelland, C. and Alcorn, M. 2008. Exploring New Composer/Performer Interactions Using Real-Time Notation. International Computer Music Conference ʼ08. Belfast, Northern Ireland.
Medley, S. and Haddad, H. 2011. The Realism Continuum: Representation and Perception. The International Journal of the Image 1(2): 145156.
Mulder, A. 1994. Human Movement Tracking Technology. Hand Centered Studies of Human Movement Project, Technical Report 94-1. Burnaby, BC, Simon Fraser University.
Pasoulas, A. 2011. Temporal Associations, Semantic Content and Source Bonding. Organised Sound 16(1): 6368.
Patel, R., Schooley, K. and Wilner, J. 2007. Visual Features that Convey Meaning in Graphic Symbols: A Comparison of PCS and Artists’ Depictions. Assistive Technology Outcomes and Benefits 41: 6280.
Patton, K. 2007. Morphological Notation for Interactive Electroacoustic Music. Organised Sound 12: 123128.
Picking, R 1997. Reading Music from Screens vs Paper. Behaviour and Information technology 162: 7278.
Pluth, E. 2012. Signifiers and Acts: Freedom in Lacan’s Theory of the Subject. New York: SUNY Press.
Prado-Leon, L. R., Schloss, K. B. and Palmer, S. E. 2011. Color, Music, and Emotion in Mexican and US Populations. New Directions in Colour Studies. Amsterdam: John Benjamins.
Ramachandran, V. S. and Hubbard, E. M. 2001. Synaesthesia: A Window into Perception, Thought and Language. Journal of Consciousness Studies 8(12): 334.
Rayner, K., Well, A.D., Pollatsek, A. and Bertera, J. H. 1982. The Availability of Useful Information to the Right of Fixation in Reading. Perception and Psychophysics 31: 537550.
Sternberg, Robert J. 1996. Cognitive Psychology. Fort Worth, TX: Harcourt Brace College Publisher.
Sudarshan, B. and Wyse, L. 2012. Computer Mediated Visual Communication in Live Musical Performance: What’s the Score? In Arts and Technology. Berlin and Heidelberg: Springer. 5462.
Tanzi, D. 2011. Extra-Musical Meanings and Spectromorphology. Organised Sound 16(1): 3641.
Thoreson, L. 2007. Spectromorphological Analysis of Sound Objects: An Adaptation of Pierre Schaeffer’s Typomorphology. Organised Sound 12(2): 129141.
Thoreson, L. 2010. Form-Building Patterns and Metaphorical Meaning. Organised Sound 15(2): 8295.
Truitt, F. E., Clifton, C., Pollatsek, A. and abd Rayner, K. 1997. The Perceptual Span and Eye-Hand Span in Sight Reading Music. Visual Cognition 4: 143161.
Vickery, L. 2010. Mobile Scores and Click-Tracks: Teaching Old Dogs. Proceedings of the Australasian Computer Music Conference 2010. Canberra, 63–70.
Vickery, L. 2011. The Possibilities of Novel Formal Structures through Computer Controlled Live Performance. Australasian Computer Music Association. Auckland, New Zealand, 112–17.
Vickery, L. 2012. The Evolution of Notational Innovations from the Mobile Score to the Screen Score. Organised Sound 17(4): 128136.
Vinot, J.-L. and Athènes, S. 2012. Legible, Are You Sure? An Experimentation-Based Typographical Design in Safety-Critical Context. CHI 2012. Austin, TX.
Walker, P. 2012. Cross-Sensory Correspondences and Cross Talk Between Dimensions of Connotative Meaning: Visual Angularity is Hard, High-Pitched, and Bright. Attention Perception and Psychophysics 74: 17921809.
Waters, A. J., Underwood, G. and Findlay, J. M. 1997. Studying Expertise in Music Reading: Use of a Pattern-Matching Paradigm. Perception and Psychophysics 594: 477488.
Waters, A. J., Townsend, E. and Underwood, G. 1998. Expertise in Musical Sight-Reading: A Study of Pianists. British Journal of Psychology 89: 123149.
Wierzbicka, A. 1996. Semantics: Primes and Universals. New York: Oxford University Press.
Winkler, Gerhard E. 2004. The Real Time-Score. The Missing-Link in Computer-Music Performance. Sound and Music Computing ʼ04. IRCAM.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Organised Sound
  • ISSN: 1355-7718
  • EISSN: 1469-8153
  • URL: /core/journals/organised-sound
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed