Hostname: page-component-6bb9c88b65-fsdjw Total loading time: 0 Render date: 2025-07-17T13:47:20.272Z Has data issue: false hasContentIssue false
  • English
  • Français

Design Considerations for Real-Time Collaboration with Creative Artificial Intelligence

Published online by Cambridge University Press:  04 March 2020

Jon McCormack ,
Patrick Hutchings ,
Toby Gifford ,
Matthew Yee-King ,
Maria Teresa Llano  and
Mark D’inverno
Jon McCormack*
Affiliation:
Monash University, Victoria, Australia. Emails: Jon. McCormack@monash.edu; Patrick. Hutchings@monash.edu; Toby. Gifford@monash.edu
Patrick Hutchings
Affiliation:
Monash University, Victoria, Australia. Emails: Jon. McCormack@monash.edu; Patrick. Hutchings@monash.edu; Toby. Gifford@monash.edu
Toby Gifford
Affiliation:
Monash University, Victoria, Australia. Emails: Jon. McCormack@monash.edu; Patrick. Hutchings@monash.edu; Toby. Gifford@monash.edu
Matthew Yee-King
Affiliation:
Goldsmiths, University of London, UK. Emails: m.yee-king@gold.ac.uk; m.llano@gold.ac.uk; dinverno@gold.ac.uk
Maria Teresa Llano
Affiliation:
Goldsmiths, University of London, UK. Emails: m.yee-king@gold.ac.uk; m.llano@gold.ac.uk; dinverno@gold.ac.uk
Mark D’inverno
Affiliation:
Goldsmiths, University of London, UK. Emails: m.yee-king@gold.ac.uk; m.llano@gold.ac.uk; dinverno@gold.ac.uk
*
Email: Jon. McCormack@monash.edu
Get access Rights & Permissions [Opens in a new window]

Abstract

Machines incorporating techniques from artificial intelligence and machine learning can work with human users on a moment-to-moment, real-time basis to generate creative outcomes, performances and artefacts. We define such systems collaborative, creative AI systems, and in this article, consider the theoretical and practical considerations needed for their design so as to support improvisation, performance and co-creation through real-time, sustained, moment-to-moment interaction. We begin by providing an overview of creative AI systems, examining strengths, opportunities and criticisms in order to draw out the key considerations when designing AI for human creative collaboration. We argue that the artistic goals and creative process should be first and foremost in any design. We then draw from a range of research that looks at human collaboration and teamwork, to examine features that support trust, cooperation, shared awareness and a shared information space. We highlight the importance of understanding the scope and perception of two-way communication between human and machine agents in order to support reflection on conflict, error, evaluation and flow. We conclude with a summary of the range of design challenges for building such systems in provoking, challenging and enhancing human creative activity through their creative agency.

Information

Type
Articles
Information
Organised Sound , Volume 25 , Issue 1: Computation in the Sonic Arts , April 2020 , pp. 41 - 52
Check for updates
Copyright
© Cambridge University Press, 2020

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Article purchase

Temporarily unavailable

References

REFERENCES

Agres, K., Forth, J. and Wiggins, G. A. 2016. Evaluation of musical creativity and musical metacreation systems. Computers in Entertainment 14(3): 1–33.CrossRefGoogle Scholar
Amabile, T. M. 1996. Creativity in Context. Boulder, CO: Westview Press.Google Scholar
Assayag, G. and Chemillier, M. 2006. Omax brothers: A dynamic topology of agents for improvization learning. Proceedings of the 1st ACM workshop on Audio and music computing multimedia. New York: ACM, 125–32.Google Scholar
Avila, L. and Bailey, M. 2016. Art in the digital age. IEEE Computer Graphics and Applications 36(4): 6–7.Google Scholar
Berger, T. and Luckmann, P. L. 1967. The Social Construction of Reality: A Treatise in the Sociology of Knowledge. New York: Double and Co.Google Scholar
Best, J. 2013. IBM Watson the inside story of how the jeopardy winning supercomputer was born and what it wants to do next. TechRepublic, 9 September. www.techrepublic.com/article/ibm-watson-the-inside-story-of-how-the-jeopardy-winning-supercomputer-was-born-and-what-it-wants-to-do-next/Google Scholar
Beyls, P. 1988. Introducing Oscar. Proceedings of the International Computer Music Conference. ICMA.Google Scholar
Biles, J. A. 1994. Genjam: A genetic algorithm for generating jazz solos. ICMC 94: 131–7.Google Scholar
Boden, M. A. 1991. The Creative Mind: Myths & Mechanisms. New York: Basic Books.Google Scholar
Bown, O. and McCormack, J. 2009. Creative agency: A clearer goal for artificial life in the arts. In Kampis, G., Karsai, I. and Szathmáry, E. (eds.) ECAL (European Conference on Artificial Life), vol. 5778. Heidelberg: Springer, 254–61.Google Scholar
Bown, O. and McCormack, J. 2010. Taming nature: Tapping the creative potential of ecosystem models in the arts. Digital Creativity 21(4): 215–31.CrossRefGoogle Scholar
Brown, A., Gifford, T. and Voltz, B. 2013. Controlling interactive music performance (CIM). Proceedings of the Fourth International Conference on Computational Creativity. Sydney, Australia.Google Scholar
Brown, A. R., Gifford, T. and Voltz, B. 2017. Stimulating creative partnerships in human-agent musical interaction. Computers in Entertainment 14(2): 1–17.CrossRefGoogle Scholar
Brown, A. R. 2018. Creative improvisation with a reflexive musical bot. Digital Creativity 29(1): 5–18.CrossRefGoogle Scholar
Brown, D., Nash, C. and Mitchell, T. 2017b. A user experience review of music interaction evaluations. NIME 17: 370–5.Google Scholar
Cassion, C., Ackerman, M., Loker, D. and Palkki, J. 2017. With bolts of melody! Songwriting with ALYSIA & Emily Dickinson. In P. Pasquier, O. Bown and A Eigenfeldt (eds.), Proceedings of MUME 2017, Musical Meta-Creation Workshop. http://musicalmetacreation.org/mume2017/proceedings/Cassion.pdfGoogle Scholar
Chen, J. X. 2016. The evolution of computing: Alphago. Computing in Science Engineering 18(4): 4–7.CrossRefGoogle Scholar
Cohen, P. R. and Levesque, H. J. 1990. Intention is choice with commitment. Artificial Intelligence 42(2–3): 213–61.CrossRefGoogle Scholar
Colton, S. 2012. The Painting Fool: Stories from Building an Automated Painter. In McCormack, J and d’Inverno, M (eds.) Computers and Creativity, Berlin and Heidelberg: Springer, 3–38.CrossRefGoogle Scholar
Cook, M. and Colton, S. 2018. Redesigning computationally creative systems for continuous creation. Proceedings of the Ninth International Conference on Computational Creativity. ICCC.Google Scholar
Cook, M., Colton, S. and Gow, J. 2017. The ANGELINA videogame design system – part I. IEEE Transactions on Computational Intelligence and AI in Games 9(2): 192–203.CrossRefGoogle Scholar
Cope, D. 1989. Experiments in musical intelligence (EMI): Non-linear linguistic-based composition. Interface 18(1–2): 117–39.CrossRefGoogle Scholar
Cropley, A. 2006. In praise of convergent thinking. Creativity Research Journal 18(3): 391–404.CrossRefGoogle Scholar
DeChurch, L. A. and Mesmer-Magnus, J. R. 2010. The cognitive underpinnings of effective teamwork: A meta-analysis. Journal of Applied Psychology 95(1): 32–53.CrossRefGoogle ScholarPubMed
Deutsch, M. 1960. The effect of motivational orientation upon trust and suspicion. Human Relations 13: 123–39.CrossRefGoogle Scholar
Dewey, J. 1930. Construction and Criticism, Volume 1. New York: Columbia University Press.CrossRefGoogle Scholar
d’Inverno, M. and McCormack, J. 2015. Heroic versus collaborative AI for the arts. Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence. AAAI Press, 2438–44.Google Scholar
Eigenfeldt, A. 2007. Real-time composition or computer improvisation? A composer’s search for intelligent tools in interactive computer music. Proceedings of the Electronic Music Studies. EMS.Google Scholar
Eisenberg, J. and Thompson, W. F. 2003. A matter of taste: Evaluating improvised music. Creativity Research Journal 15(2–3): 287–96.CrossRefGoogle Scholar
Elgammal, A. M., Liu, B., Elhoseiny, M. and Mazzone, M. 2017. CAN: Creative adversarial networks, generating ‘art’ by learning about styles and deviating from style norms. CoRR, abs/1706.07068.Google Scholar
Ericsson, K., Krampe, R. and Tesch-Römer, C. 1993. The role of deliberate practice in the acquisition of expert performance. Psychological Review 100: 363–406.CrossRefGoogle Scholar
Freeland, C. A. 2001. But is it Art?: An Introduction to Art Theory. Oxford: Oxford University Press.Google Scholar
Gatys, L. A., Ecker, A. S. and Bethge, M. 2016. Image style transfer using convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2414–23.Google Scholar
Ghedini, F., Pachet, F. and Roy, P. 2016. In: Corazza, G and Agnoli, S (eds) Multidisciplinary Contributions to the Science of Creative Thinking. Creativity in the Twenty First Century. Singapore: Springer.Google Scholar
Gibson, C., Folley, B.S. and Park, S. 2009. Enhanced divergent thinking and creativity in musicians: A behavioral and near-infrared spectroscopy study. Brain and Cognition 69(1): 162–9.CrossRefGoogle ScholarPubMed
Gifford, T. and Brown, A. R. 2011. Beyond reflexivity: Mediating between imitative and intelligent action in an interactive music system. Proceeding of 25th BCS Conference on Human-Computer Interaction. Swinton: BCS.Google Scholar
Gifford, T., Knotts, S., McCormack, J., Kalonaris, S., Yee-King, M. and d’Inverno, M. 2018. Computational systems for music improvisation. Digital Creativity 29(1): 19–36.CrossRefGoogle Scholar
Gladwell, M. 2008. Outliers, the Story of Success. London: Allen Lane.Google Scholar
Glăveanu, V. P. 2015. From individual agency to co-agency. In Gruber, C. W., Clark, M. G., Klempe, S. H. and Valsiner, J. (eds.) Constraints of Agency. Berlin: Springer, 245–65.Google Scholar
Gombrich, E. H. 1995. The Story of Art, 16th, rev., expanded and redesigned edn. Englewood Cliffs, NJ: Prentice-Hall.Google Scholar
Guilford, J. P. 1967. The Nature of Human Intelligence. New York: McGraw-Hill.Google Scholar
Hagberg, G. L. 2017. Jazz improvisation and peak performance: Playing in the zone. In Jordan, T., McClure, B. and Woodward, K. (eds.) Culture, Identity and Intense Performativity. Abingdon: Taylor & Francis.Google Scholar
Hantula, O. and Linkol, S. 2018. Towards Goal-aware Collaboration in Artistic Agent Societies. Proceedings of the Ninth International Conference on Computational Creativity. ICCC.Google Scholar
Hutchings, P. 2018. Adaptive Music Scores for Interactive Media. PhD thesis, Monash University.Google Scholar
Iba, T. 2010. An autopoietic systems theory for creativity. Procedia Social and Behavioral Sciences 2: 6610–25.CrossRefGoogle Scholar
Jordà, S. and Mealla, S. 2014. A methodological framework for teaching, evaluating and informing NIME design with a focus on expressiveness and mapping. NIME 14: 233–8.Google Scholar
Keller, R. and Morrison, D. R. 2007. A grammatical approach to automatic improvisation. In Spyridis, C, Georgaki, A, Kouroupetroglou, G, & Anagnostopoulou, C (eds.), Proceedings of the 4th Sound and Music Computing Conference, Lefkada, Greece: National and Kapodistrian University of Athens, 330–7.Google Scholar
Kitani, K. and Koike, H. 2010. Improvgenerator: Online grammatical induction for on-the-fly improvisation accompaniment. Proceedings of NIME2010, June 15–18, Sydney, Australia, 469–472.Google Scholar
Lee, J. D. and Moray, N. 1994. Trust, self-confidence, and operators’ adaptation to automation. International Journal of Human-Computer Studies 40: 153–84.CrossRefGoogle Scholar
Lee, J. D. and See, K. A. 2004. Trust in automation: Designing for appropriate reliance. Human Factors 46(1): 50–80.CrossRefGoogle ScholarPubMed
Lencioni, P. 2006. The Five Dysfunctions of a Team. Chichester: John Wiley & Sons.Google Scholar
Lewis, G. E. 1999. Interacting with latter-day musical automata. Contemporary Music Review 18(3): 99–112.CrossRefGoogle Scholar
Lewis, G. E. 2000. Too many notes: Computers, complexity and culture in voyager. Leonardo Music Journal 10: 33–9.CrossRefGoogle Scholar
Liapis, A., Yannakakis, G. N. and Lopes, P. 2016. Can computers foster human users’ creativity? theory and praxis of mixed-initiative co-creativity. Digital Culture & Education (DCE) 8(2): 136–52.Google Scholar
Linson, A., Dobbyn, C. and Laney, R. 2012. Critical issues in evaluating freely improvising interactive music systems. Proceedings of the 3rd International Conference on Computational Creativity. ICCC, 145–49.Google Scholar
Luck, M. and d’Inverno, M. 1995. A formal framework for agency and autonomy. Proceedings of the First International Conference on Multi-Agent Systems. AAAI Press/MIT Press, 254–60.Google Scholar
McCormack, J. 1996. Grammar-Based Music Composition. In Stocker, R., Jelinek, H., Durnota, B. and Bossomaier, T. (eds.) Complex Systems 96: From Local Interactions to Global Phenomena. Amsterdam: ISO Press, 321–36.Google Scholar
McCormack, J. 2009. The evolution of sonic ecosystems. In Komosinski, M. and Adamatzky, A. (eds.) Artificial Life Models in Software. London: Springer, 393–414.CrossRefGoogle Scholar
McCormack, J., Gifford, T., Hutchings, P., Rodriguez, M. T. L., Yee-King, M. and d’Inverno, M. 2019. In a silent way: Communication between AI and improvising musicians beyond sound. Proceedings of ACM SIGCHI 2019. New York. ACM SIGCHI.CrossRefGoogle Scholar
McCrae, R. R. 1987. Creativity, divergent thinking, and openness to experience. Journal of Personality and Social Psychology 52(6): 1258–65.CrossRefGoogle Scholar
Mohammed, S., Ferzandi, L. and Hamilton, K. 2010. Metaphor no more: A 15-year review of the team mental model construct. Journal of Management 36(4): 876–910.CrossRefGoogle Scholar
Mordvintsev, A., Olah, C. and Tyka, M. 2015. Deepdream-a code example for visualizing neural networks. Google Research 2: 5.Google Scholar
Morgan, J., Ackerman, M. and Cassion, C. 2018. Co-Creative Conceptual Art. Proceedings of the Ninth International Conference on Computational Creativity. ACC.Google Scholar
Mori, M., MacDorman, K. F. and Kageki, N. 2012. The uncanny valley. IEEE Robotics Automation Magazine 19(2): 98–100.CrossRefGoogle Scholar
Muir, B. M. 1987. Trust between humans and machines, and the design of decision aides. International Journal of Machine Studies 27: 527–39.CrossRefGoogle Scholar
Mumford, M. D., Baughman, W. A. and Sager, C. E. 2003. Picking the right material: Cognitive processing skills and their role in creative thought. In Runco, M. (ed) Critical Creative Processes. Cresskill, NJ: Hampton Press, 19–68.Google Scholar
Ng, H. H. 2019. Collective Free Music Improvisation as a Sociocommunicative Endeavor: A Literature Review. Update: Applications of Research in Music Education, 37(2), 15–23. https://doi.org/10.1177/8755123318784109Google Scholar
Nort, D. V. 2018. Conducting the in-between: improvisation and intersubjective engagement in soundpainted electro-acoustic ensemble performance. Digital Creativity 29(1): 68–81.CrossRefGoogle Scholar
Nunn, T. E. 1998. Wisdom of the Impulse: On the Nature of Musical Free Improvisation. Thomas E. Nunn, International Improvised Music Archive, http://intuitivemusic.dk/iima/tn_wisdom_part2.pdf.Google Scholar
O’Hear, A. 1995. Art and technology: An old tension. Royal Institute of Philosophy Supplement, 38: 143–58.CrossRefGoogle Scholar
O’Modhrain, S. 2011. A framework for the evaluation of digital musical instruments. Computer Music Journal 35(1): 28–42.CrossRefGoogle Scholar
Pachet, F. 2012. Musical virtuosity and creativity. In McCormack, J and d’Inverno, M (eds.) Computers and Creativity. Berlin and Heidelberg: Springer, 115–46.CrossRefGoogle Scholar
Pachet, F., Roy, P., Moreira, J. and d’Inverno, M. 2013. Reflexive loopers for solo musical improvisation. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York: ACM, 2205–8.CrossRefGoogle Scholar
Pereira, F. C. 2007. Creativity and Artificial Intelligence: A Conceptual Blending Approach, Volume 4. Berlin: Walter de Gruyter.Google Scholar
Pink, D. H. 2006. A Whole New Mind: Why Right-Brainers will Rule the Future. London: Penguin.Google Scholar
Rowe, R. 1992. Interactive Music Systems: Machine Listening and Composing. Cambridge, MA: MIT Press.Google Scholar
Runco, M. A. 2014. Creativity: Theories and Themes: Research, Development, and Practice. Amsterdam: Elsevier.Google Scholar
Sawyer, R. K. 2011. Explaining Creativity: The Science of Human Innovation. Oxford: Oxford University Press.Google Scholar
Schellenberg, E. G. 2004. Music lessons enhance IQ. Psychological Science, 15(8): 511–14.CrossRefGoogle ScholarPubMed
Schellenberg, E. G. and Weiss, M. W. 2013. Music and cognitive abilities. In Deutsch, D. (ed.) The Psychology of Music. Amsterdam: Elsevier, 499–550.CrossRefGoogle Scholar
Still, A. and d’Inverno, M. 2016. A history of creativity for future AI research. Proceedings of the 7th Computational Creativity Conference. Universite Pierre et Marie Curie.Google Scholar
Stocker, G. 2019. AIxMusic festival ’19. Festival Program, Ars Electronica 2019, Festival for Art, Technology & Society, Linz, Austria.Google Scholar
Stowell, D., Robertson, A., Bryan-Kinns, N. and Plumbley, M. D. 2009. Evaluation of live human–computer music-making: Quantitative and qualitative approaches. International Journal of Human-Computer Studies 67(11): 960–75.CrossRefGoogle Scholar
Thom, B. 1999. Learning models for interactive melodic improvisation. In Proceedings of the International Conference on Computer Music, China, October, 190–3.Google Scholar
Thornton, S. 2009. Seven Days in the Art World. London: Granta.Google Scholar
Waterman, E. 2015. Improvised trust: Opening statements. In Caines, R. and Heble, A. (eds.) The Improvisation Studies Reader: Spontaneous Acts. Abingdon: Routledge, 59–62.Google Scholar
Wegner, D. M. 1987. Transactive Memory: A Contemporary Analysis of the Group Mind. In Mullen, B. and Goethals, G. R. (eds.) Theories of Group Behavior. New York: Springer-Verlag, 185–208.CrossRefGoogle Scholar
Weinberg, G., Raman, A. and Mallikarjuna, T. 2009. Interactive jamming with Shimon: A social robotic musician. Proceeding of the 4th ACM/IEEE International Conference on Human-Robot Interaction (HRI). ACM/IEEE, 233–4.Google Scholar
Yee-King, M. and d’Inverno, M. 2016. Experience driven design of creative systems. Proceedings of the 7th Computational Creativity Conference (ICCC 2016). Universite Pierre et Marie Curie.Google Scholar