As we have seen in the previous chapters, a variety of sensors can measure and record data in various modalities. However, how does a stream of sensory data become multimedia? We have discussed the multi part before. In this chapter, we will take a first look at a very important concept that combines sensory data into a palatable and presentable format, making it a medium. We are talking about the concept of a document.

The concept of a document has been used over centuries as a device or mechanism to communicate information. Because the technology to store and distribute information has been evolving and changing, the nature and concept of a document has been evolving to take advantage of the new media technology. At one time, one considered a document in the form of physical embodiment, such as a book, that mostly contained text as the source of information. This popular notion of a book as a document has gone through changes over the past few decades and has now transformed the notion of document into a (virtual) container of information and experiences using sensory data in multiple forms. In this modern reincarnation of the document, it is not limited to one medium, but can use different media as needed to communicate the information most effectively to a recipient. This means, however, that textual and sensory output needs to be combined in various ways to communicate different aspects and perspectives of information related to an event or an object. This chapter deals with the properties of applications and systems that do exactly that.