Introduction

The rich, multi-dimensional, multi-modal data channel offered by multimedia brings with it unique challenges for information retrieval. The old saying equates a picture to a thousand words. While this is true for the amount of information that a picture can convey to a user, from a data management and information retrieval perspective the equation is not so straightforward. More than a thousand words and more complex representations are often needed to identify relevant media and assist users in their data finding missions successfully.

Traditionally digital and analogue multimedia documents have been indexed and queried using text descriptions in the form of metadata. Metadata for multimedia can be modelled or classified in a number of ways (Boll, Klas and Sheth, 1998; Troncy et al., 2007). It is useful to group multimedia metadata into four rough categories, as shown in Figure 13.1 overleaf, arranged by the level of human effort potentially required to create the descriptors, to illustrate the concept of the ‘semantic gap’. The semantic gap in multimedia information retrieval describes the difficulty in automatically generating usable high-level semantic descriptions (e.g. dog, party, ocean and so on) from automatically extracted low-level media features such as colour, texture, tempo, pitch and so on. The concept originates in psychology (Finke, 1989) but its use within multimedia information retrieval is defined by Smeulders et al. (2000) as ‘the discrepancy between the information that one can extract from the visual data, and the interpretation that the same data has for a user’.

When managing high volumes of multimedia data, it is very attractive to try and generate quality semantic annotations from features automatically extracted from the pixels or bits. Not only is this cheaper and more efficient than manual human annotations but it also improves the consistency of the annotations by reducing the influence of user opinion. Bridging the multimedia semantic gap is the target of a large field of research. We will discuss some aspects of it in the section ‘Piggy-back text and automatic annotation’ in the context of automatically annotating media; good general starting points can be found in Smeulders et al. (2000), Zhao and Grosky (2001) and Hare et al. (2006).