In the previous chapters, we described many signal processing and content analysis techniques. However, the content of an image or audio file does not alone determine its meaning and impression on the user. In this chapter, we will therefore describe other factors that are very important to consider in multimedia computing: the set of surrounding circumstances in which the content is presented, otherwise known as context. Context is often neglected in academic work because it can be leveraged in many ways in multimedia systems and is often so effective that the content analysis approach becomes secondary. So let us first find out what context really is.

Almost two centuries ago, George Berkeley asked: If a tree falls in a forest and no one is around to hear it, does it make a sound? Sound is often defined as the sensation excited in the ear when the air or other medium is set in motion. Thus, if there is no receiving ear, then there is no sound. In other words, perception is not only data – it is a close interaction between the data, transmission medium, and the interpreter. This is shown in Figure 19.1.

Characteristics of each of these must be considered in designing and developing a multimedia system. It has been very well realized, and rigorously articulated and represented, that we understand the world based on the sensory data that we receive using our sensors and the knowledge about the world that we have accumulated since our birth. Both the data and the knowledge are integral components of understanding.