As discussed in the previous chapter, multimedia is closely related to how humans experience the world. In this chapter, we first introduce the role of different sensory signals in human perception for understanding and functioning in various environments and for communicating and sharing experiences. A very important lesson for multimedia technologists is that each sense provides only partial information about the world. One sense alone, even the very powerful sense of vision, is not enough to understand the world. Data and information from different sensors must be combined with other senses and prior knowledge to understand the world – and even then we only obtain a partial model of the world. Therefore, different sensory modalities should be combined with other knowledge sources to interpret the situation. Multimedia computing and communication is fundamentally about combining information from multiple sources in the context of the problem being solved. This is what distinguishes multimedia from several other disciplines, including computer vision and audio processing, where the focus is on analyzing one medium to extract as much information as possible from it.

In multimedia systems, different types of data streams simultaneously exist, and the system must process them not as separate streams, but as one correlated set of streams that represent information and knowledge of interest for solving a problem. The challenge for a multimedia system is to discover correlations that exist in this set of multimedia data and combine partial information from disparate sources to build the holistic information in a given context.