Multimedia communication design is a form of complex problem-solving. It requires heuristics which take into account addressees’ cognitive abilities and prior knowledge, the complexity of the subject matter, processing conditions, time limits, and other factors. Multimedia designers have to be aware of the asymmetry between texts and pictures in terms of representational principles and communication functions, including the fact that texts and pictures compensate for their inherent ambiguities by reciprocal disambiguation. Designers have to be further aware that multimedia comprehension starts with initial mental model construction primarily guided by the text, which is then followed by adaptive mental model specification primarily guided by the picture for specific task requirements. Text design should enable smooth continuous coherence formation within the right text modality. Picture design should enable scaffolding for mental model construction and visualize the essential structure of a subject matter with regard to future tasks. Above all, multimedia design needs to adequately synchronize the different comprehension processes.

In future, an increasing number of people will need to learn continuously in order to orient themselves in the quickly changing world around them. Multimedia communication and multimedia comprehension will be key elements of their learning. Accordingly, it is very important to have a sufficiently deep understanding of the psychological processes behind multimedia comprehension. This understanding should be rooted in theory-driven empirical research about the cognitive processing of multiple representations, particularly of texts and pictures. It should also allow practice-oriented basic recommendations to be derived for the design and usage of multimedia. These recommendations should go beyond everyday knowledge, practical experience, intuition, and the use of seemingly professional surface features. Design of multimedia communication has to be based on sufficiently deep knowledge about the psychological processes involved in comprehension and knowledge construction. Practitioners need to receive scientific support for them to better understand the laws of perception and cognitive processing underlying comprehension and knowledge acquisition.

Pictures are two-dimensional depictive representations. They include static pictures and animations. The latter are defined as pictorial displays that change their structure or other features over time and trigger perception of a continuous change. Static and animated pictures can display static as well as dynamic content. Both can have an envisioning, explanatory, orientation, organizing, and argumentative function. Picture comprehension entails sub-semantic perceptual processing, semantic perceptual processing, and conceptual processing. Sub-semantic perceptual processing is primarily pre-attentive and data-driven. It results in viewer-cantered and object-cantered visual representations. Semantic perceptual processing is attentive and data- as well as knowledge-driven. It results in object or event recognition. Conceptual processing is attentive and primarily knowledge-driven. It creates complex propositional structures and mental models in working memory. Picture comprehension is based on analog structure mapping under the guidance of perceptual and conceptual representations.

Text comprehension and picture comprehension can be synthesized into a common conceptual framework which differentiates between external and internal descriptive and depictive representations. Combining this framework with the human cognitive architecture including sensory registers, working memory, and long-term memory leads to an integrated model of text and picture comprehension. The model consists of a descriptive branch and a depictive branch of processing. It includes multiple sensory modalities. Due to a flexible combination of sensory modalities and representational formats, the model covers listening comprehension, reading comprehension, visual picture comprehension, and sound comprehension. The model considers text comprehension and picture comprehension to be different routes for constructing mental models and propositional representations with the help of prior knowledge. It allows us to explain the effects of coherence, text modality, split attention, text–picture contiguity, redundancy, sequencing, and the effects of different types of visualization.

Human history has created a large variety of sign systems for communication. These systems were developed at different times for different purposes. While oral language has developed as part of human biological evolution, written texts, realistic pictures, maps, and graphs are cultural inventions. Human oral language might have originated from gestures supplemented by sound patterns. It is a biological anchored feature of the human species, as manifested in somatic, perceptual, and neurological pre-adaptations. Early writing systems used iconic ideograms which were gradually transformed into symbols. This made production and discrimination easier but increased the required amount of learning. Further development led to writing systems using phonograms plus orthographic ideograms. Realistic pictures are older than writing systems. They represent content by similarity but also show allegories of social relationships. Maps are realistic pictures of a geographic area facing the problem of how to present a curved earth surface on a two-dimensional surface. Graphs are visuo-spatial objects representing a subject matter based on analogy due to inherent common structural properties.

Spoken or written texts are coherent sequences of sentences. Text comprehension is equivalent to the construction of multiple mental representations in working memory. It is based on an interaction between external text information and internal prior knowledge information stored in long-term memory. Mental representations include a text surface representation, a propositional representation, and a mental model. They are characterized by different forgetting rates. As speakers and authors omit information which can be easily completed by listeners and readers, text comprehension always includes inferences. Listening and reading comprehension use the same lexicon and the same syntax but qualitatively different text surface structures. Due to local and global coherence of texts, comprehension is also a process of mental coherence formation. Limitations of working memory require focused attention on the construction of topic-specific mental models which are carried along from sentence to sentence by a flow of consciousness. Speakers and authors can direct this process through topic information within the text surface.

Text and pictures serve different purposes in multimedia comprehension. Conceptual processing of texts and pictures results in propositions, whereby text-based propositions and picture-based propositions specialize in different kinds of information. These propositions are merged into an overarching conceptual semantic network guiding mental model construction. The construction process receives descriptive guidance by text-based and picture-based propositional representations as well as depictive guidance by perceptual representation of pictures through structure mapping. Because texts are more constrained in terms of processing order, they can provide more conceptual guidance through a subject matter than pictures. A distinction can be made between initial model construction and adaptive model specification. Initial model construction aims at general coherence formation; adaptive model specification aims at selective processing of task-relevant information. Initial mental model construction is more likely to be text-driven than adaptive mental model specification, while adaptive mental model specification is more likely to be picture-driven than initial mental model construction.

Sign systems help to create descriptive and depictive representations. Descriptive representations operate on symbols. They are based on conceptual analyses identifying objects or events as well as attributes and interrelations. Attributes and relations are ascribed by predications to entities according to syntactic rules resulting in so-called propositions (“idea units”). These propositions can be integrated into coherent semantic networks. Propositional representations are considered as mental structures which can be externalized in the form of spoken or written texts. Despite their informational incompleteness, descriptions have high representational power. Depictive representations are based on inherent commonalities between a representing object and the represented subject matter. The inherent commonalities can be based on similarity or analogy. These representations are complete with regard to a certain class of information. Due to their completeness and consistency and because information can be read off directly, depictive representations have high computational efficiency.

Comprehension can be the starting point for further cognitive activities such as thinking and problem-solving. Productive thinking requires a specific interaction between descriptive and depictive representations combining representational power and inferential power. The interaction takes place through processes of mental model construction and model inspection. Descriptive representations are combined and coordinated with the corresponding depictive representations, whereby each representation constrains the construction and usage of the other representation. Model inspection requires systematic and exhaustive interrogation of depictive representations in order to read off relevant information. Depictive representations have to grasp task-relevant structures and facilitate performance of the required procedures. Required operations should not be difficult to perform and the sequences of operations should be relatively short. Inappropriate perceptual structures of visualizations can obscure relevant structural attributes, preventing the application of correct procedures and stimulating the application of incorrect procedures.

This chapter aims at clarifying basic concepts related to multimedia: communication, comprehension, and learning. Multimedia communication is considered as the intentional creation, display, and reception of multiple kinds of signs in order to convey messages about some content. It entails two subprocesses: meaning and comprehension. Multimedia meaning is a process in which the producer of a message creates multiple external signs based on his or her prior knowledge in order to direct the recipient’s mind so that the recipient understands what the producer means. Multimedia comprehension is the complementary process of reconstructing the previously externalized knowledge in the mind of the recipient. It can be seen as the bottleneck of multimedia communication. Multimedia comprehension and multimedia learning are related but are nevertheless different: While multimedia comprehension results in transient changes in working memory, multimedia learning results in permanent changes in long-term memory. Multimedia learning is a byproduct of multimedia comprehension. Further, an overview of the book is presented.

An integrated model of text and picture comprehension is presented in this chapter which takes into account that learners can use multiple sensory modalities combined with different forms of representation. The model encompasses listening comprehension, reading comprehension, visual picture comprehension, and auditory picture comprehension (i.e., sound comprehension). The model’s cognitive architecture consists of modality-specific sensory registers, working memory, and long-term memory. Within this architecture, a distinction is made between perception-bound processing of text surface or picture surface structures, on the one hand, and cognitive processing of semantic deep structures, on the other hand.

Research has shown that animated graphics are not the educational magic bullet that many expected them to be. They are neither necessarily superior to static graphics nor intrinsically effective in their own right. The Animation Composition Principle characterizes learning from animation as a hierarchical relation-building process by which mental models of the depicted subject matter are progressively and cumulatively constructed from discrete information primitives. It helps explain the limited success of previous attempts to improve animation’s effectiveness that took no account of their fundamental design. By giving due consideration to both perceptual and cognitive aspects of animation processing, the Animation Processing Model that embodies this Principle opens the door to novel, more effective compositional design options. These compositional animations significantly improve learning outcomes.