Skip to main content

Audiovisual Harmony: The realtime audiovisualisation 1 of a single data source in Construction in Zhuangzi

  • Ryo Ikeshiro (a1)

This paper explores the context and technical and aesthetic considerations behind the author's generative and improvisational audiovisual work, Construction in Zhuangzi (2011), and in particular the approach of ‘audiovisualising’ the same source of data and its validation, and its possibilities as an artistic practice. First, the origins of integrated audiovisual art in the output of John Whitney are explored. Then, metaphors based on musical textures are used to describe different approaches to the audiovisual medium. Research into perception and auditory displays are next used to justify the simultaneous representation of the same data in both the audio and the video. The aesthetic potential of this practice is then corroborated using Michel Chion's theory of sound in cinema. In concluding, its possibilities for providing an appropriate form and aesthetic approach to the audiovisual material are discussed.

Corresponding author
Hide All

The term is used throughout the paper to indicate simultaneous sonification and visualisation, sonification consisting of both direct audification i.e. the translation of data into the audible range, and parameter mappings related to the production of audio.

Hide All
Abbado, A. 1988. Perceptual Correspondences of Abstract Animation and Synthetic Sound. Leonardo Supplemental Issue 1: 35.
Alves, B. 2005. Digital Harmony of Sound and Light. Computer Music Journal 29(4): 4554.
Bertelson, P., Radeau, M. 1981. Cross-modal Bias and Perceptual Fusion with Auditory-visual Spatial Discordance. Perception & Psychophyics 29(6): 578584.
Bouchara, T., Katz, B.F.G., Jacquemin, C., Guastavino, C. 2010. Audio-visual Renderings for Multimedia Navigation. Proceeding of the 16th International Conference on Auditory Display. Washington, DC: ICAD.
Cádiz, R.F. 2004. A Fuzzy Logic Model for Compositional Approaches to Audio-Visual Media. Proceedings of the International Computer Music Conference. Miami: ICMA.
Chion, M. 1994. Audio-Vision: Sound On Screen. Foreword by W. Murch. New York: Columbia University Press.
Coulter, J. 2010. Electroacoustic Music with Moving Images: The Art of Media Pairing. Organised Sound 15(1): 2634.
CVM (Center for Visual Music). 2012. Research: Errata. Accessed on 19 April 2012.
DeLanda, M. 2002. Intensive Science and Virtual Philosophy. London and New York: Continuum.
Deleuze, G., Guattari, F. 1994. What is Philosophy?. London and New York: Verso.
Di Scipio, A. 2001. Iterated Nonlinear Functions as a Sound-Generating Engine. Leonardo 34(3): 249254.
Dombois, F. 2002. Auditory Seismology: On Free Oscillations, Focal Mechanisms, Explosions, and Synthetic Seismograms. Proceedings of the 8th International Conference on Auditory Display. Kyoto: ICAD, pp. 27–30.
Edmonds, E. 2003. Logics for Constructing Generative Art Systems. Digital Creativity 14(1): 2328.
Faulkner, M./D-Fuse (ed.) 2006. VJ: Audio-visual Art and VJ Culture. London: Laurence King.
Ghirardelli, G.T., Scharine, A.A. 2009. Auditory-visual Interactions. In C.E. Rash, M.B. Russo, T.R. Letowski and E.T. Schmeisser (eds) Helmet-Mounted Displays: Sensation, Perception and Cognition Issues. Fort Rucker, AL: US Army Aeromedical Research Laboratory.
Grierson, M. 2005. Audiovisual Composition. PhD thesis, University of Kent.
Gröhn, M., Lokki, T., Takala, T. 2003. Comparison of Auditor, Visual, and Audio-Visual Navigation in a 3D Space. Proceedings of the 2003 International Conference on Auditory Display. Boston: ICAD.
Kramer, G., Walker, B., Bonebright, T., Cook, P., Flowers, J.H., Miner, N., Neuhoff, J. 1997. Sonification Report: Status of the Field and Research Agenda. Prepared for the NSF by members of ICAD.
Levin, G. 2000. Painterly Interfaces for Audiovisual Performance. MSc thesis, Massachusetts Institute of Technology.
Lorenz, E.N. 1963. Deterministic Nonperiodic Flow. Journal of the Atmospheric Sciences 20: 130141.
Manovich, L. 2001. The Language of New Media. Cambridge, MA: The MIT Press.
McGurk, H., MacDonald, J. 1976. Hearing Lips and Seeing Voices. Nature 264: 746748.
Peacock, K. 1988. Instruments to Perform Color-music: Two Centuries of Technological Experimentation. Leonardo 21(4): 397406.
Petržela, J., Hanus, S. 2005. Experimental Observations of Strange Attractors in the Modified Lorenz System. Proceedings of the 15th International Conference Radioelektronika. Brno: Radioelektronika.
Radeau, M., Bertelson, P. 1977. Adaptation to Auditory–visual Discordance and Ventriloquism in Semirealistic Situations. Perception and Psychophysics 22: 137146.
Ramachandran, V.S., Hubbard, E.M. 2003. Hearing Colors, Tasting Shapes. Scientific American 288(5): 5359.
Ritter, D. 1993. Interactive Video as a Way of Life. Music Works 56: 4854.
Roads, C. 1996. The Computer Music Tutorial. Cambridge, MA: The MIT Press.
Rudi, J. 1998. Computer Music Animations. Organised Sound 3(3): 193198.
Saue, S., Fjeld, O. 1997. A Platform for Audiovisual Seismic Interpretation. Proceedings of the 4th International Conference on Auditory Display. Palo Alto, CA: ICAD.
Smith, D.W. 2003. Mathematics and the Theory of Multiplicities: Badiou and Deleuze Revisited. Southern Journal of Philosophy 41(3): 411449.
Talsma, D., Woldorff, M.G. 2005. Selective Attention and Multisensory Integration: Multiple Phases of Effects on the Evoked Brain Activity. Journal of Cognitive Neuroscience 17(7): 10981114.
Vroomen, J., de Gelder, B. 2004. Perceptual Effects of Cross-modal Stimulation: Ventriloquism and the Freezing Phenomenon. In G. Calvert, C. Spence and B.E. Stein (eds) The Handbook of Multisensory Processes. Cambridge, MA: The MIT Press.
Walker, B.N., Kramer, G. 2004. Ecological Psychoacoustics and Auditory Displays: Hearing, Grouping, and Meaning Making. In J.G. Neuhoff (ed.) Ecological Psychoacoustics. San Diego, CA: Elsevier Academic Press, 150175.
Whitney, J. 1980. Digital Harmony: On the Complementarity of Music and Visual Art. Peterborough, NH: Byte Books.
Whitney, J. 1991. Fifty Years of Composing Computer Music and Graphics: How Time's New Solid-state Tractability Has Changed Audio-Visual Perspectives. Leonardo 24(5): 597599.
Whitney, J. 1994. To Paint on Water: The Audiovisual Duet of Complementarity. Computer Music Journal 18(3): 4552.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Organised Sound
  • ISSN: 1355-7718
  • EISSN: 1469-8153
  • URL: /core/journals/organised-sound
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
Type Description Title
Supplementary materials

Ikeshiro Supplementary Movie

 Video (36.6 MB)
36.6 MB
Supplementary materials

Ikeshiro Supplementary Movie

 Video (2.0 MB)
2.0 MB
Supplementary materials

Ikeshiro Supplementary Movie

 Video (6.0 MB)
6.0 MB
Supplementary materials

Ikeshiro Supplementary Movie

 Video (70.8 MB)
70.8 MB
Supplementary materials

Ikeshiro Supplementary Movie

 Video (8.8 MB)
8.8 MB
Supplementary materials

Ikeshiro Supplementary Movie

 Video (8.3 MB)
8.3 MB
Supplementary materials

Ikeshiro Supplementary Movie

 Video (4.4 MB)
4.4 MB


Full text views

Total number of HTML views: 7
Total number of PDF views: 69 *
Loading metrics...

Abstract views

Total abstract views: 299 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 19th July 2018. This data will be updated every 24 hours.