The language connection: phonetic coding and the rebus principle

Archaeologists believe that the use of true cuneiform writing (as opposed to proto-cuneiform) began during the Early Dynastic I period (c. 2950–2750 bce) in Mesopotamia. Two key changes are considered especially important in this development. The first has to do with the material techniques used in writing, which led to increasingly abstract forms. The second, perhaps enabled by this increased abstraction, is the way written signs came to represent speech sounds.

Because wet clay did not preserve curved lines well, and because it dried quickly in the hot, dry climate, lines drawn with a sharp pointed stylus in proto-cuneiform were replaced by short, straight impressions made with a reed stylus with a triangular tip, explaining the characteristic wedge-shaped marks that give cuneiform its name (see Figure 5.7). With this new wedge-shaped stylus, curved lines became straight strokes (Figure 5.8), circles became squares, and fine details were eliminated, all of which increased writing speed (Gaur, Reference Gaur1984, p. 48). This is the first instance of many that we will encounter of changes in written forms brought about by material factors.

Figure 5.7 Stylus shapes and their respective impressions

(Nissen et al., Reference Nissen, Damerow and Englund1993, p. 118)

Figure 5.8 Breaking up of curved lines with change of stylus

(Nissen et al., Reference Nissen, Damerow and Englund1993, p. 118)

Because early clay tablets were bookkeeping devices, they did not have any discernable syntax and were mostly organized in spatial hierarchies (Damerow, Reference Damerow2006).Footnote ⁹ For example, names of donors or recipients of a transaction were consistently found below signs indicating the goods transacted, allowing information such as ‘ten sheep (received from) Kurlil’ to be recorded even in the absence of signs to indicate verbs and prepositions (Schmandt-Besserat, Reference Schmandt-Besserat1996, p. 98). While such extreme economy might seem overly ambiguous to the modern reader, it is not entirely unlike today's pared down text messages, tweets, and emoticons. Nissen (Reference Nissen1986) points out that for a long time writing was used “only as a means of producing catchwords for someone who was more or less familiar with the context but needed to be reminded of particular details” (p. 329). Nissen illustrates this point with the example shown in Figure 5.9. While the inscriptions can be interpreted as ‘Two sheep delivered to the temple (or house) of the goddess Inanna,’ they can also be read as ‘Two sheep received from the temple/house of the goddess Inanna.’ Furthermore, because the starlike sign (AN – ‘heaven’) can refer to An, the god of the heavens, or can function as a semantic determinative (marking divinity), it is not clear whether the house or the temple belongs to ‘the goddess Inanna’ or ‘the gods An and Inanna’ (1986, p. 329). Contingency of meaning was thus a feature of written texts right from the start.

Figure 5.9 The problem of relating signs in a proto-cuneiform text fragment

(Nissen, Reference Nissen1986, p. 330)

Over a period of centuries, ‘naturalistic’ signs became increasingly abstract, and they continued to change over subsequent millennia as they came to be used not just for Sumerian but also other languages such as Akkadian, Hittite, Elamite, Hurrian, and Urartian. Consider, for example, the transformation of the signs for Sumerian words mušen (‘bird’), še (‘barley’), and gu (‘ox’) from pictograms to cuneiform signs (Figure 5.10). The 90-degree left rotation (shown in the third column of Figure 5.10) occurred sometime during the early third millennium, corresponding to the transition from writing in vertical columns (read from right to left) to writing in rows (read left to right).Footnote ¹⁰

Figure 5.10 Evolution of the signs for ‘bird,’ ‘barley,’ and ‘ox’ from c. 3100 bce to c. 700 bce

(based on Gaur, Reference Gaur1984, p. 49)

The second and crucial change, occurring during the Early Dynastic period (c. 2800 bce), was the linkage of Sumerian writing and speech through phonetic coding. This adaptation changed both the structure and the range of application of proto-cuneiform writing. Although proto-cuneiform was almost entirely logographic, meaning that one graphic sign represented a whole word, the system gradually became more logosyllabic, meaning that signs took on phonetic value through the use of the rebus principle. In a rebus, a picture or symbol is used purely for its sound to represent a spoken word or syllable (see Figure 5.11).

Figure 5.11 Example of a rebus in English. May I see you home, my dear? Escort Card, c. 1865, US

In cuneiform writing, the sign for one word was used to represent another word with the same or similar sound.Footnote ¹¹ For example, the sign for the Sumerian word ti (‘arrow’), written , was also used to designate the near homophone til (‘life’). Rebuses and homophony not only made it possible to write abstract nouns but also afforded the possibility of writing multisyllabic words and names by representing the sounds of each individual syllable. An analogy in English would be using an icon of the sun not only to represent the words sun or son, or the name Sunn, but also the syllable in sundry, lesson, Anderson, and so on.

But homophony has its limitations as well. Whereas context might clarify meaning in face-to-face speech, in the context-reduced situation of writing, a systematic one-to-one sound–symbol correspondence system could lead to frequent ambiguity. Because some Sumerian words had a large number of homophones, they needed to be distinguished in writing. For example, fourteen homophones of gu were represented by distinct visual signs. The first four are presented below (from Walker, Reference Walker1987, p. 12):

This is similar to how contemporary languages like English and French attribute multiple meanings to the same sound pattern, but differentiate each meaning with a distinct visual sign (e.g., there, their, they're; ver, vers, vert, vair, verre). And this is what allows tongue-in-cheek use of homophony as in the email forwarding tag ‘Sent from my eyePhone.’

To further complicate things, Sumerian was also characterized by homophony's complement, polyphony, meaning that one graphic sign could represent semantically related words that nevertheless had vastly different sounds. For example, the sign for ka‘mouth,’ originally a pictogram of a head with the mouth drawn on it, is also used for words associated with the mouth; for example, gu₃ ‘voice’, zu ‘tooth’, du₁₁ ‘speak,’ and inim ‘word’ (Walker, Reference Walker1987). Because of homophony and polyphony, the intended meaning of cuneiform signs could often only be determined by context.

A partial solution to the one-to-many correspondence problem was the development of determinatives – signs placed before or after other signs to specify the relevant semantic domain (e.g., geography, divinity, metal, wood, and so on). Perhaps the most common determinative in Sumerian was an () as seen in the tablet in Figure 5.9 above to designate a deity. Other markers were phonetic, to indicate how a sign should be pronounced (akin to our use of st, nd, rd in 1st, 2nd, 3rd in English). Because they ‘determine’ how a stretch of writing should be read, these particles are vaguely analogous to emoticons in electronic texts today, which are often intended to clarify the tone of an utterance.

An important structural change resulting from cuneiform's new function of representing speech was the standardization of sign order and linearity. Whereas early accounts and administrative records used formats in which written signs were not necessarily placed in the order in which they would be spoken, by the middle of the third millennium bce, the order of signs usually reflected the syntax of spoken discourse (Veldhuis, Reference Veldhuis and Houston2012, p. 6). This happened at the same time that cuneiform was being used for the first time on a large scale to record lengthy prose and poetic compositions.

This story of the origins of writing still poses intriguing puzzles for scholars. We still do not know the extent to which particular threads of the story interact. As we have seen, for example, two important changes occurred between proto-cuneiform and cuneiform writing: material changes in the technique of writing, and functional changes with the beginnings of phonetic representation. Was there a relationship between these two developments? Here are the facts as we currently know them. First, we have seen that the earliest written signs depicted things in the world. When the wedge-shaped reed stylus was introduced, curved lines became straight impressions, fine details were lost, and signs became more stylized and abstract – a trend that continued over centuries and millennia. Signs lost any resemblance they might have originally had to things in the world. Might it have been this increasing abstraction of signs, grounded in the very materiality of writing practices, that facilitated the development of phonetic coding by distancing signs from their original real-world referents and thus making them more suitable for ‘arbitrary’ and multifunctional use? We will never know. But we do know that as cuneiform writing interfaced more fully with the Sumerian language, the degree of multifunctional use of cuneiform signs increased remarkably. Logograms served additionally as phonograms, and homophony and polyphony led to a complex web of relationships between signs, sounds, and meaning. Certain signs were used as semantic determinatives. Sumerian logograms were loosened from the language and came to be used to write Akkadian and other languages as well. Thus a limited inventory of signs was used to accomplish a maximum of communicative work. We can see how this principle still operates today as the Roman alphabet is used to write multiple languages, and when we use letters and numbers to represent words or parts of words, as when we write CU l8r for ‘see you later’ in a text message.

To round out the story, cuneiform became one of the most successful writing systems in human history. Over a period of three millennia, cuneiform writing served multiple needs, purposes, cultures, and languages. Despite its purely administrative origins, cuneiform writing came to preserve and disseminate literature (e.g., the Sumerian/Akkadian Epic of Gilgamesh), law (e.g., the Code of Hammurabi), religion (prayers, hymns, omens, divinations), and scholarship in fields such as astronomy, mathematics, and medicine.

Cuneiform and design

The above description has been somewhat detailed because the specific processes involved in establishing early Mesopotamian writing 5,000 years ago – the use of tokens, seals, tablets, pictograms, and abstract signs that were re-worked in relation to one another – are connected to general processes that have been reiterated throughout the history of literacy, and have direct relevance to our understanding of the relationship of language, literacy, and technology today.

We have seen that writing in Mesopotamia did not begin as a representation of speech but rather was born out of administrative needs in an increasingly complex society, to allow records of transactions to be made and preserved. What thus began as an interface to administrative bureaucracy was subsequently adapted to become a visual interface to language. This sequence is important, as it reminds us that writing is first and foremost a social phenomenon, and that its formal linguistic features follow from its social function as well as material constraints. In previous chapters we have seen this idea play out in diverse examples from handwriting to Greeklish, texting, online chat, and Facebook, and we will see it again in the next chapter when we consider new conventions that developed with the printing press. It will also help us to think about educational issues related to literacy in Part III. In writing classrooms today, for example, it all too often seems that writing is made to be about rules of standard grammar, spelling, and punctuation but without any real exploration of the social and pragmatic underpinnings of those conventions. The case study of cuneiform affirms the importance of conventions, but also highlights their social basis, which is what gives them meaning.

Material dimensions of literacy are also clearly illustrated in the case of cuneiform writing. As we saw in Chapter 2, the physical medium used has important implications for the form of the script, the size of texts, and their relative permanence. The medium of clay and reed stylus lent itself to short, straight impressions rather than round drawn lines. Because most tablets were small enough to fit in a scribe's palm, texts were generally short and required economical use of space. To date, clay has proved to be the most durable of any material used for writing – and its durability was only increased by fires!

Another important point that we can take from the story of cuneiform is that writing co-evolved with Mesopotamia's increasing social, economic, and cultural complexity, but it was not its cause. Technology is often considered an autonomous force that brings about advances in society, but the cases of innovations such as cuneiform writing, the telephone, the printing press, and the computer show us that it is not technology per se, but the interaction of technologies with social life, in the context of particular social conditions, that gives rise to broad social change. As functional needs change, so do material forms of communication, and these changes in turn open up possibilities for new functions, and so on.

The story of cuneiform also reminds us that the repurposing of existing resources depends on the ability to view familiar signs in a new way, drawing on repertoires of symbol manipulation. That ability is facilitated by the abstraction of signs. As users of an alphabet whose origins go back to the Phoenicians, writers of English no more think of an ox head when they write the letter A than they think of a little snake when they write the letter N. And no doubt the Phoenicians didn't either. The letters A and N make meaning when positioned in relation to other letters, but have no intrinsic meaning as letter forms.Footnote ¹² The leap made by the Sumerians was to make signs abstract, which was at least partly due to material factors in using clay and stylus. Abstraction allowed signs to designate other signs (e.g., impressions on a bulla of tokens contained within it, which were in turn signs of categories of objects in the world; signs inscribed in clay being used to designate spoken words as well as functions that had no spoken equivalent, such as determinatives). The fact that a sign might refer to a notion of an object in the world, or it might refer to a language sound, or it might be used as a semantic determinative also suggests that extralinguistic context must have played an important role in the Sumerians’ interpretations of texts.

Another significant point illustrated by the case of cuneiform is that technologies of communication don't simply replace one another in a neat succession. Rather, they overlap and inform one another, creating a synergistic panoply of resources for record-keeping, for authenticating, for making and communicating meaning – all operating within a sociocultural ecology (as described in the Introduction and Chapter 3). When writing appeared, it supplanted neither token use nor seals, and it certainly did not supplant the use of speech. Nor did writing likely have just one precursor. As Lieberman, Michalowski and others have argued, writing emerged from a matrix of existing forms (i.e., available designs) coming from different, but interacting, semiotic systems.Footnote ¹³ Furthermore, the technology of cuneiform was itself an ‘available design’ in the sense that it went on to influence the development of other scripts, such as Ugaritic and Old Persian. Gaur (Reference Gaur2000) likens this (re)invention process to the turning of a kaleidoscope: “at every turn new patterns emerge but the basic components from which these patterns are made remain the same” (p. 6).

Finally, the case of cuneiform furthermore illustrates that writing systems are not static but respond dynamically to the ecologies in which they are used: they are variable in extension and adapt to the needs of the reading culture to which they belong. Thus whereas Old Assyrian merchants needed only about one hundred cuneiform signs, Old Babylonian scholars recording the Sumerian literary tradition used hundreds more, and many of which had several distinct values (Veldhuis, Reference Veldhuis and Houston2012, p. 8). Yet they were working with a system that had a recognizable quality of ‘sameness’ to it.Footnote ¹⁴ By virtue of interfacing with fourteen languages other than Sumerian, cuneiform writing also disabused us of the notion that writing systems ‘represent’ languages in any essential way. Rather, they have the potential to interface with multiple languages, and are often adopted for political or cultural reasons rather than for any linguistic reasons.

Print and society

If the technology existed across the Asian continent long before it did in Europe, why didn't printing become widespread there before it did in Europe? Essentially because of different social conditions. In the case of China, printing was under the strict control of its rulers, who limited the number of print copies and restricted their dissemination to an elite. Furthermore, unlike Europe, where printing was immediately realized as a profit-producing industry, China did not have a capitalist economy. In the Muslim world, printing encountered powerful social resistance.Footnote ⁹ The reasons were partly religious, partly economic, and partly technical. With regard to religion, only the handwritten word was considered sacred. As Ogier Ghiselin de Busbecq, the European ambassador to the Ottoman Empire, wrote from Istanbul in 1560: “the scriptures, their holy letters, once printed would cease to be scriptures” (Káldy-Nagy, Reference Káldy-Nagy and Káldy-Nagy1974, p. 203). Furthermore, the practice of cleaning typesettings with hog-bristle brushes made it all the more inconceivable to print the name of Allah in type (ibid., p. 203).Footnote ¹⁰ Another reason was economic: printing would have supposedly put thousands of manuscript copyists out of work.Footnote ¹¹ A third reason for the late adoption of printing had to do with the complexity of typesetting Arabic script. Because Arabic is a cursive script, letters are connected to one another and have different forms when they are in initial, medial, final, and free-standing positions. This means that a complete Arabic font, including vowel marks, can run to over 600 glyphs (Bloom, Reference Bloom2001, p. 218). Typographically, then, Arabic presented unique difficulties – and required greater typesetting skill – than languages that could be printed in uniform, discrete characters like English or Chinese. Ironically, the movable type printing press became widely used in the Islamic world only after the French occupation of Egypt in 1798.Footnote ¹²

Clearly, the mere existence of the technology of movable type was not sufficient for its extensive use; the right social ecology was needed as well. The comparatively rapid success of printing in Europe depended in particular on three key factors. One was a good supply of readers to make the mass production of books be economically feasible. Related to this were the entrepreneurial ambitions of printers, who were motivated by the commercial potential of publishing texts. The third necessary ingredient was an abundant supply of a durable yet inexpensive material on which to print.

Concerning readers, a “vigorous literate culture” (Clanchy, Reference Clanchy1979, p. 8) had been developing in Europe since the late twelfth century, when the growth of universities created an unprecedented demand for written texts, commentaries, and reference works. In cities such as Paris, Bologna, and Oxford, lay scribes worked in guilds to hand copy academic texts long before the arrival of print (Steinberg, Reference Steinberg1974). By the late fifteenth century, a growing population (and growing numbers of readers) combined with an economy on the ascent translated into a viable market for booksellers. The printing press may have amplified the demand for books, but it did not create it in the first place. Like cuneiform writing in Mesopotamia, the print ‘revolution’ co-evolved with, but did not cause, increased societal complexity.

Within the first fifty years following Gutenberg's invention, hundreds of printing presses were established in cities throughout Europe, and Febvre and Martin (Reference Febvre, Martin and Gerard1976) estimate that some twenty million books were printed (p. 248). Compared to the limited capacity of scribal guilds, this was clearly a huge expansion in the scale of production. Printers sought the most lucrative markets – university towns and commercial centers – and developed standardized procedures to maximize speed and productivity.

The third requirement, the plentiful availability of inexpensive material on which to print, was satisfied by paper. Introduced into southern Europe from the Muslim world in the eleventh century, paper was strong, smooth, flexible, and could be produced in uniform formats. Because it was made from recycled linen rags, paper was much cheaper than parchment, and could be produced in virtually unlimited quantities. If early printers had only had parchment to print on, their books would have been almost as expensive as handwritten manuscripts, and printing would not likely have taken hold to the extent that it did.

Although printing presses and books proliferated rapidly in Europe in the first fifty years after Gutenberg printed his first Bible, they did not immediately change the existing power dynamics of literacy. Three-quarters of the books published were in Latin and most readers were clerics or academics, so print began as a product of – and for – a very circumscribed segment of society. Although the printing press was initially welcomed by the Church because it increased dissemination of Bibles and other religious materials, it soon became apparent that the new print era also heralded unwelcome changes. The Church's tight control over the production of books was threatened, as was the Church's very language (since the Bible could now be published in vernacular languages as well as Latin). Mass dissemination of print also threatened communal recitation of prayers by making it possible for individuals to read and contemplate scripture privately. Circulation of the printed writings of Martin Luther, Jean Calvin, and other Protestant reformers so enraged France's King François I that he issued a series of restrictive laws, culminating in a 1535 decree outlawing all printing (by punishment of hanging) and ordering the closure of all book shops.Footnote ¹³

Nor did the printing press immediately usher in more widespread literacy. Non-literate aristocrats could hire readers and scribes for their literacy needs. Peasants, on the other hand, had little need of or access to writing. It was among the emerging middle classes that literacy held the greatest promise, for in a culture that was increasingly in demand of written records, the ability to write assured a livelihood. But this demand for written documents preceded the printing press, and for many lay scribes, print was more of a threat than a boon. During the sixteenth and seventeenth centuries, literacy rates in Europe remained low, especially in rural areas, and it wasn't until the late eighteenth century – more than three hundred years after Gutenberg's invention – that literacy rates came to approach even 50 percent among men in France and 60 percent among men in England.Footnote ¹⁴

Scholarly practices remained largely unchanged following the arrival of the printing press. Given the still relatively high cost of books, scholars continued to copy texts by hand, including texts that had been produced on a printing press.Footnote ¹⁵ Even in the early eighteenth century, access to university libraries was often limited to professors, and teaching practices typically followed the medieval tradition of reading aloud from authoritative texts and commenting upon them (Melton, Reference Melton2001, p. 90).

If print did not suddenly transform an oral society into a literate one, it was nevertheless appropriated in interesting ways into oral culture. Historian Natalie Zemon Davis writes that printing in sixteenth-century France (like the Internet today) established “new networks of communication…new options for the people…new means of controlling the people” (1975, p. 190). She explains how in village veillées in which books were read aloud, ‘reading aloud’ meant translating texts into the local dialect so the villagers could understand. In the case of long works, it also meant editing while reading. Reading was therefore far from a simple vocalization of print – it was a wholesale recreation and performance of texts. Non-literate peasants on the listening end were not passive recipients of information, Davis tells us, but participated in their own way in the new world of printing – not by reading and writing, but by actively interpreting and using the information that was read to them. Orality and literacy therefore suffused and transformed one another.

Similarly, Harold Love points out that even in late sixteenth and early seventeenth century England many key cultural texts were oral, and texts that were written were often intended for oral performance (2002, pp. 100–101). Because people's actual experience of texts often involved a mixture of speech, print, and handwriting, there was no clear-cut transition from ‘oral’ to ‘literate’ cultures, and practices in early modern Britain were better characterized as “intricate negotiations between the media” (ibid., p. 102). In the next chapter we will see that the same holds true today, and that “negotiations between the media” aptly characterizes twenty-first-century literacies as well.

Finally, a common assertion is that the printing press was the key factor in disseminating Luther's and other Protestant reformers’ ideas during the Reformation in Germany. It is undeniable that print played a significant role, but given the low literacy rates in sixteenth-century Germany, Scribner and Dixon (Reference Scribner and Dixon2003) estimate that only 2.3 percent of the German population could have encountered Luther's ideas via print. They argue that most information was passed by word of mouth through personal contacts, and that the real role of print was to inform ‘opinion leaders’ – literate and influential people – who disseminated the new ideas through oral communication, with sermons being the most powerful communication platform (p. 20). What print did, then, was to accelerate and broaden dissemination of information to such opinion leaders, creating more networks and spheres of influence through which new ideas could be spread.Footnote ¹⁶

Print and language

Probably the most significant way in which the printing press influenced language was to introduce an ethos of uniformity and standardization. This happened at two levels: at the visual surface of the printed text and more broadly in the standardization of language use and in the determination of what counted as a language. Although manuscripts had been written in standard hands, variability from one scribe to the next was inevitable, and of course any individual scribe's handwriting could vary slightly from one writing session to the next, depending on posture, fatigue, ambient temperature, and so on. As manuscripts were transformed into printed texts, such individual variability was lost, as the same letter forms were used consistently within and across texts produced by a given printer. As Amalia Gnanadesikan puts it, printing shifted the process from creating letters to selecting letters (Reference Gnanadesikan2009, p. 252). This shift was later extended by the typewriter and the computer, which allowed texts to be composed from the outset by means of selecting letters.

On a broader level, the printing press provided the impetus for European vernaculars (which had been mostly spoken) to be written, codified, and thereby legitimized as languages. Prior to the arrival of print in Europe, a diglossic linguistic configuration existed whereby the language of the Church, authority, and literacy was Latin, and the language of daily life in society was the regional vernacular. Although in the early years of printing most texts were published in Latin, by the sixteenth century printers produced increasing numbers of vernacular texts, motivated by the prospect of broadening their market base. The wide circulation of such texts helped consolidate and establish the literary languages of Europe and ultimately contributed to the decline of Latin.

Of course, there were many regional vernaculars. Those chosen for print (i.e., those most closely associated with power) became standard languages, whereas those that remained unprinted took on the status of provincial dialects. Early grammars for the most prestigious vernaculars were developed during the sixteenth century (Nebrija's 1493 Grammatica Castellana being a notably early contribution). These grammars provided a basis for filtering out many of the regionalisms that had marked earlier vernacular publications.

Print also gradually introduced new formatting, spelling, and punctuation conventions. Although many of the earliest printed books followed manuscript tradition in terms of layout and punctuation, new practices came to be established. For example, whereas rubrication (literally ‘red-inking’) had been used to show divisions in a text, printers found this too costly and made use of different fonts for headings, initial words, and main text. Quotations, which in manuscripts had been underlined in red ink, came to be indicated by quotation marks. On the other hand, paragraphs, which had been explicitly marked with symbols (such as the pilcrow, ¶), came to be marked by white space (indentation of the first line, or most recently, a preceding blank line in ‘block’ paragraphing) (Baron, Reference Baron2000, p. 169).

Before print, spelling was highly variable, and abbreviations were commonly used. Producing a manuscript book was expensive, not only in terms of labor, but also in terms of materials (a Bible written on parchment required the skins of hundreds of sheep). Because of this expense, scribes saved space wherever possible and developed elaborate abbreviation schemes (Cappelli's Lexicon Abbreviaturarum lists some 13,000 Latin abbreviations).Footnote ¹⁷ Printed books, which at first attempted to reproduce manuscripts as closely as possible, preserved abbreviations along with handwriting style, ligatures, illuminations, and layout conventions. Similarly, in both manuscript and early print books, word spellings were adjusted by adding or subtracting letters to fine-tune the length of each line of text so that a flush right margin could be maintained. Once printed, spellings took on a certain authority, and some of the ‘anomalous’ modified spellings became standard when dictionaries were made. Other techniques scribes and printers used to make text fit were adding or deleting whole words, increasing or decreasing space between words or between letters, substituting words for phrases, or vice versa (N. S. Baron, Reference Baron2000).

Abbreviations gradually came to be less used in printed books because printers did not want to risk losing readers who might not understand what they meant. Moreover, as they shifted from expensive parchment to comparatively cheap paper, printers were less compelled to maximize the efficiency of the texts they printed by using abbreviations.

The history of these early printing practices reminds us that spelling is not ‘natural’ and is often shaped by the technological medium in which it is used. Moreover, the control and standardization of spelling (as well as grammar) is also a matter of social forces whereby the power elite set standards, and ordinary people's access and adherence to those standards largely determines their social mobility.

It is clear that the printing press both reflected and shaped the cultural evolution of Europe. However, as mentioned earlier, print was not a catalyst for social change in Central and East Asia. We will now return to this story to see how another technological innovation – paper – was.

Paper

The invention of paper is commonly attributed to Tsai Lun, a eunuch at the court of the Han emperor Wu Di in China in 105 ce, although archeological findings from the Xuanquanzhi ruins of Tunhuang in China's northwest Gansu province place the invention of paper some two hundred years earlier (Yi & Lu, Reference Yi, Lu, Allen, Zuzao, Xiaolan and Bos2010). Made from rags, hemp, or tree bark dissolved in water and then sieved through woven cloth stretched over a bamboo frame, paper was from the beginning made of waste products and therefore relatively cheap to produce.Footnote ¹⁸ It was also very light in weight. These were substantial advantages over competing writing materials such as bamboo, which was heavy, and silk, which was expensive. Over time, production techniques were perfected and paper was made in various qualities for different purposes. One key use of paper was for printing, which evolved from the ancient use of carved stone or bronze seals to make impressions on clay or silk, and from the practice of making rubbings from stone and bronze reliefs (Bloom, Reference Bloom2001, p. 36).

Papermaking spread to Korea, where the product was refined to new levels of quality, and subsequently to Japan in the early seventh century. Paper was introduced to Central Asia in the eighth century, allegedly by Chinese prisoners taken captive by Arabs during the Battle of Talas, although archeological evidence from Samarkand suggests that paper may have existed there centuries earlier (ibid., pp. 43–45). In any event, with the unification of West Asia under Islam, the practice of papermaking migrated westward to Baghdad, Mecca, Egypt, Morocco, and finally to Spain. According to Bloom (ibid.), Spanish Christians began to use paper well before the year 1000, and their use of paper grew as they came to dominate greater areas of the Iberian peninsula. However, it was only after the arrival of the printing press in the fifteenth century that paper gradually came to replace parchment as the preferred medium for recording European thought.

In Islamic civilization, it was paper, not printing, that made possible huge advances in learning and new ways of thinking between the eighth and the sixteenth centuries. Paper was originally produced as an adjunct to papyrus and parchment to serve the administrative needs of the huge bureaucracy that developed during the Caliphates and the Ottoman Empire.Footnote ¹⁹ But its light weight, availability, and relatively low cost made it a catalyst for social, intellectual, and artistic innovation. The Hindu numeral system had spread through the Islamic world by the ninth century, and by the tenth century, the mathematician Abu al-Hasan Ahmad ibn Ibrahim al-Uqlidisi adapted the Hindu system to the affordances of paper and ink, developing the notion of positional decimal fractions and showing how to perform calculations without deletions (ibid.). Scholars collected and codified the oral traditions of Muhammad on paper, Greek bookrolls were translated into Arabic and written on paper, and new forms of literature, such as cookbooks and The Thousand and One Nights were copied on paper and sold (ibid., p. 12).

Paper provided a convenient means for textualizing not only language, but also artistic designs, architectural plans, genealogy charts, and battle plans, which could be composed in one place and put to use in another (ibid., pp. 215–16). This ability to separate things from their original context of conception and to recontextualize them in new settings, in new mediums, for new purposes, allowed ideas to be not only disseminated, but also transformed:

The idea of divorcing design from its original context (like writing, which divorces language from its original context of utterance) is crucial to understanding relations among language, technology, and literacy. In Chapter 2 we saw how this idea is extended in today's digital texts, whose content is divorced from form (at the level of machine representation) in order to allow those texts to be appropriately scaled across a broad variety of devices.

The typewriter: a personal printing press

The origins of the typewriter go back to at least 1714, when an English engineer named Henry Mill applied for a patent for his machine for transcribing letters that would print letters on paper “so neat and exact as not to be distinguished from print” and whose impression would be “deeper and more lasting than any other writing, and not to be erased or counterfeited without manifest discovery” (Bliven, Reference Bliven1954, p. 24). However, it took over 150 years and dozens of other similar inventions before a marketable machine actually named ‘Type-Writer’ arrived on the scene. Attributed to Christopher Latham Sholes, a newspaper editor and politician from Wisconsin, the prototype Type-Writer was inspired by the piano, with alternating black and white keys that triggered typeface hammers that brought an inked cloth ribbon into contact with paper to leave a printed impression. The ribbon had to be hand-inked, the hammers produced only upper case letter forms, and typists couldn't see what they had typed until they had completed an entire line of text. Nevertheless, the arrival of the ‘literary piano,’ as the editors of Scientific American had called an even earlier prototype, signaled the obsolescence of “the laborious and unsatisfactory performance of the pen” and heralded “a revolution as remarkable as that effected in books by the invention of printing.”Footnote ²¹

After failing to interest Western Union in his machine, Sholes turned to Philo Remington, whose company manufactured firearms and sewing machines. Demand for firearms was down since the end of the Civil War, and the company had extra production capacity. Remington bought Sholes's patent and marketed the first Remington typewriter in 1874. Sales were slow, given that the Type-Writer cost $125, but one of the early adopters was Mark Twain, who, despite his distaste for the machine, became the first writer to submit a typescript to a publisher.Footnote ²²

One particular technical problem that Sholes and his collaborator James Densmore had faced was that the type keys often jammed if they were struck in too quick succession, requiring the typist to stop and pry the type bars apart. The original keyboard displayed the alphabet in sequential order but split across two rows of keys, as follows:

Densmore suggested reordering letter keys in an unfamiliar pattern to slow down typing. Sholes consulted with his mathematician brother-in-law, asking him to reorganize the key configuration so that the type bars that most often got stuck would be separated. Many calculations and experiments led to the four-row ‘QWERTY’ configuration that is still used on English language keyboards to this day. Sholes claimed that this scientific arrangement of the keys would boost typists’ speed and efficiency. It may have done so to the extent that typists spent less time prying apart stuck type bars, but, keeping in mind that typing was done with only two fingers at the time, the QWERTY layout actually made typists’ movements less efficient because it maximized the distance that their fingers needed to travel in typing most words (Beeching, Reference Beeching1974). Besides reducing the incidence of jams, the QWERTY layout also positioned all of the letters in the word TYPEWRITER in the top row of letter keys, a feature that salesmen used to full advantage when they demonstrated the machine.

Although a number of other keyboard layouts have been invented since Sholes's time, and these have been claimed to be more scientific and efficient (most notably the one developed by Dr. August Dvorak in 1932), none has been able to replace the QWERTY standard.Footnote ²³ Once again, we see that technological factors may sometimes be an impetus for change, but in many cases they are not, and that what makes the difference is the social context of the time and place. The technical problem that QWERTY targeted no longer exists – it has been irrelevant ever since the development of the typeball and was permanently put to rest with the personal computer. And yet QWERTY has prevailed – not because of any inherent technical superiority, but rather because of social inertia born of comfortable human habit and the consequent economic risk that any manufacturer proposing a new standard would face.

Unlike the technologies of handwriting and print, which do not favor certain scripts over others, the typewriter and the computer keyboard that evolved from it have a clear structural bias toward alphabetic writing. This is not surprising: a limited character set was a precondition for the typewriter; without an alphabet, the typewriter would not even have been conceivable. Attempts were made at adapting the typewriter to other scripts. For example, Underwood developed a Japanese typewriter based on the Katakana syllabary in 1923. However, because it could not include Kanji (Chinese) characters, which were typically used in conjunction with Katakana, the typewriter was not even marketed as a writing tool, and its use was largely limited to the preparation of billing statements in large companies (Gottlieb, Reference Gottlieb2000). Another limitation was that the typewriter imposed horizontal writing, whereas Japanese writing was most commonly formatted vertically. Finally, the typewriter's need to space characters uniformly created an unusual appearance on the page, and typists had to remember to insert word spaces to avoid ambiguities (Gnanadesikan, Reference Gnanadesikan2009, p. 129). A later typewriter designed in the 1960s for the Hiragana syllabary suffered a similar fate. It was not until the late 1970s, when word-processing technology made it possible to input Kana characters and convert them to Kanji, that the potential of the personal printing press was finally realized – but at the staggering cost of $37,000 per machine (ibid., p. 130). Today, Japanese computer keyboards retain the QWERTY keyboard, but they also print kana characters on the keys, facilitating the inputting of multiple scripts.

If the typewriter's design posed problems for languages written in non-alphabetic scripts, it contributed to the global spread of alphabetic writing by influencing decisions about what script to use for newly written languages. As Gnanadesikan points out:

In sub-Saharan Africa, for example, the Roman alphabet (sometimes extended with additional letters) has often been deemed the most practical means to transcribe languages that have not traditionally been written “because it looks international, because individuals already educated in the colonial languages (English, French, and Portuguese) already know it…and because of the lasting, arguably pernicious influence of the typewriter” (ibid., p. 261).

With electronics, type keys were freed from singular mechanical connections to typeface characters. Any key could theoretically be programmed to produce any character – in any language or script. This is the principle that allows the QWERTY keyboard to be used in Chinese, for example. Perhaps the most common way to write Chinese characters on computers is to use the standard Roman alphabet keys on a QWERTY keyboard to write in pinyin (Roman phonetic transliteration), which activates a menu of Chinese characters that correspond to that phonetic representation. The user then chooses the appropriate Chinese character from the menu, and this is inserted into the text. An alternative procedure (called Wubi) maps components of Chinese characters (radicals and strokes) onto the QWERTY keyboard. One enters these character components in the same sequence as one would handwrite them on paper. Wubi-configured keyboards are divided into five regions, each designating a different type of stroke: horizontal, vertical, left-falling, right-falling, and hook. For experienced typists, this way of entering characters is far faster than using pinyin. Yet another way of entering Chinese characters is with finger movements on a touchscreen – pattern recognition software converts the hand-traced forms into printed characters.

Today, the accessibility of self-publishing via websites, blogs, social networking sites, and online forums has led to the production of a staggering amount of writing published by non-professional writers. The publishing industry's traditional role in legitimizing text and gatekeeping what can become ‘public writing’ has become to some extent democratized. This significant change has also made possible the sharing of what was once specialized knowledge of professional printers with ordinary individuals. One important such area of knowledge has to do with the semiotics of typeface.

The semiotics of typefaces

Typographers often seek to make type as ‘transparent’ as possible, not distracting the reader's attention from the writer's work (Warde, Reference Warde1956). At the same time, however, there is a very long tradition, from Egyptian hieroglyphs to illuminated manuscripts to advertising today, of texts that purposely call attention to their surface visual form, beckoning the reader to look at them as well as through them (Lanham, Reference Lanham1993). Calligraphy and typography are semiotic modes in their own right, contributing their own nuances to the meanings prompted by words (van Leeuwen, Reference van Leeuwen2005a). Fonts put new faces on words, and in so doing they affect our reading of those words. Consider how font design characteristics add a dimension of coolness, warmth, or cutesiness to a greeting.

Conversely, we can be led to make associations between different words by means of similar typographic design, as in a 2010 Greenpeace campaign aimed against Nestlé, the company that makes KitKat candy bars, for buying palm oil from companies that destroy Indonesian rainforests and pushing orangutans toward extinction. The typography and red color was designed to lead readers to make an association between a KitKat bar and the word ‘killer,’ as shown in Figure 6.2.

Figure 6.2 Greenpeace's appropriation of the visual design of the Nestlé KitKat logo

Like language, typeface has meaning potential, but not predetermined meaning as in a code. Typefaces, like other aspects of style, get culturally appropriated and therefore can ‘mean’ different things in different cultural and historical contexts. For example, the Nazis appropriated Gothic script as a symbol of German nationalism; stickers from the early years of the Third Reich displayed slogans like “Feel German, think German, speak German, be German, even in your script” (C. Burke, Reference Crystal1998, p. 148). However, in 1941, when Hitler realized the importance of communicating his case to the wider world, he rejected Gothic script, calling it a Jewish invention (“Schwabacher-Judenlettern”), and decreed that the Antiqua typeface was to be the “normal script” of the German people (Steinberg, Reference Steinberg1974, p. 293). In early modern England, on the other hand, Gothic script had been appropriated by the common people because it was thought to be the easiest script to read.

Today, Helvetica is an example of a typeface that is both revered and reviled, and is even the subject of a feature film.Footnote ²⁴ Developed in 1957 by Swiss typographer Max Miedinger, Helvetica was designed to be modern, legible, comforting, and ‘neutral’ in the sense that the typeface itself should not convey any meaning and thus be ‘open’ to interpretation. Many cities have adopted Helvetica for their public signs displaying the dos and don'ts of street life. Critics of Helvetica condemn its lack of rhythm and contrast, its boring predictability, and its overuse in society.

Typefaces might be considered expressions of identity – just like clothes, haircuts, eyeglasses, and other visual design accouterments – and in this light it is notable that social networking sites such as Facebook and MySpace allow users to modify typefaces in their personal spaces (the default fonts are Lucida Grande and Verdana respectively). However, in comparison with the many ways in which social networking users are unwittingly being homogenized by algorithms that make them behave in consistent, predictable ways, the typeface choices may be little more than a superficial gimmick to provide an illusion of agency.

Because visual design features such as typeface and layout contribute in significant ways to the meanings people make and take from texts, there is a substantial responsibility to understand the visual pragmatics that underlies their use. Kress and van Leeuwen are two scholars who have contributed much valuable work in this area (Kress & van Leeuwen, Reference Kress and van Leeuwen1996, Reference Kress and van Leeuwen2001; van Leeuwen, Reference van Leeuwen2005a, Reference van Leeuwen2005b, Reference van Leeuwen2006) and we will consider the pedagogical implications of this topic in Part III.

Conclusion

We have seen in the last two chapters how technologies of literacy have affected humans’ relationship to language. The invention of writing allowed language to be separated from speakers’ bodies and distanced from the original context of utterance. Paper, by providing a cheap, light, foldable medium, offered wider access to writing and drawing, and thus broader use of textualization and recontextualization. Print introduced the mechanization of writing, imposing another degree of spatial and temporal distance between the author and the final form of the work (a distance that had been introduced during the manuscript era by scribes). The typewriter made a simplified version of print accessible to the masses; it made printing personal and portable, but still maintained mechanical intermediaries between writer and written product.

The story of paper and print illustrates how material, social, and individual dimensions interact in producing new designs of knowledge, learning, and social life. The fact that printing did not become widespread for centuries in either China or the Muslim world leads us to the conclusion that it was not the technological innovation of movable type alone that made printing viable. Rather, a favorable social–economic–material ecology had to be in place to allow the technological innovation to take hold. Paper and print influenced the social–economic–material ecology by making it possible to circulate ideas to an unprecedented degree, and, for the first time, to archive texts at a potentially unlimited number of sites. In so doing, paper and print changed both the individual's and society's relationship to written language. These developments were crucial for the development of nation-states, the notion of the public sphere, and the consolidation of academic disciplines, especially the sciences.

Today, digital devices introduce a further stage of transformative separation between humans and language (albeit hidden from the eye), as HTML and other computer codes intervene between the embodied biomechanical movements of writing and the display of signs on screens of various types and sizes. The connections between these devices also make it possible for individuals to disseminate their writing with unprecedented speed and breadth of dispersal. It is to electronically mediated writing that we turn in the next chapter.

Phonological and graphic strategies

Affective elements

Because writing is very often unaccompanied by voice, facial expression, gesture, or body language, it has to develop its own devices for conveying emotion. Written texts have always managed to express emotion when their writers have wished to express it, and they often do so in extraordinarily potent ways. Although some early pundits predicted that computer-mediated writing would be adequate for informational memos but not for meaningful personal interaction, research has shown on aggregate that online communication is no less emotional or engaging than face-to-face communication, and if anything, it involves more explicit and frequent expression of emotion (Derks, Fischer, & Bos, Reference Derks, Fischer and Bos2008).

Walther (Reference Walther1996) suggests that online writing encourages ‘hyperpersonal’ communication, referring to the stronger affect that people sometimes experience in technology-mediated communication as compared to face-to-face communication. This effect arises partly because people can often craft a more interesting image of themselves online (and can also better monitor and control that projected image). This curation of the self can lead others to idealize and over-estimate one's qualities, which, in turn, leads to enhanced feedback that seems much better than what one gets in face-to-face interaction. This generates a loop: the more interest one's interlocutor seems to show, the more one will be encouraged to be even more witty and engaging to keep the good feedback coming. All this may lead people to interact quite differently from how they might in a face-to-face context.

What is important to recognize is that this hyperpersonal argument is not limited to twenty-first-century online communication, but it has applied to the technology of writing for centuries. Consider the bashful boy who wants to tell a girl he loves her, but he's struck dumb by her beauty every time he sees her. He may be speechless in her presence, but there's a good chance the letter or sonnet he writes will win her over. The same goes for expressing grief. People can write a heartfelt, articulate sympathy note, but when face to face with the grieving person, they often can't get past clichés or awkward silence.

What is new in electronic discourse are some specific (and comparatively trivial) conventions that have developed to express affect, some language-based and others non-linguistic. These include initialisms such as LOL (laughing out loud), emoticons ;-), capitalization (I AM SHOUTING), punctuation marks (!!!), modified spelling (miss u soooo much!), expressions typical of spoken conversation (Geez! Mm hmm. Yum!), and ‘emotes’ that verbally describe an action or response (e.g., ^*groan, ^*shrug, ^*grimace).Footnote ¹²

‘Shouting,’ the use of capital letters for emphasis or to express strong emotion (e.g., I AM SO MAD!), is so widely used a convention that even in face-to-face contexts young people sometimes refer to loud, emotion-laden speech as using an ‘upper case voice.’ A similar technique is to flank words with asterisks or underlines to intensify feeling (e.g., I'm ^*crazy about^* that band; I _really_ hate this idea). These devices borrow from the well-established typographical conventions of underlined, boldface, or italic print, and were developed as ‘work-arounds’ before those techniques could be implemented in online writing. Even though formatting options have expanded in many online interactive environments, these work-arounds are nevertheless still commonly used.

Two affective devices that have developed specifically in the context of text-based digitally mediated communication are emoticons and LOL.

Emoticons

As discussed in Chapter 1, emoticons are pictographs that add a non-verbal affective dimension to online communication. Assembled from punctuation marks, they are a good example of how one set of available resources (symbols that mark syntactic boundaries) can be put to use in new configurations to serve a wholly new purpose (to inject friendliness into an interaction or to dissuade readers from taking what is written too seriously). Unlike the expression of emotion in face-to-face communication, which often happens uncontrollably or even unconsciously, a writer's use of emoticons is always conscious and sometimes contrived. That is, when someone uses an emoticon, it does not mean that the person is actually experiencing the represented emotion, but that he or she wants to evoke an aura of that feeling for some communicative effect. This remove is what allows emoticons to be used non-literally for effects such as irony and humor, as in this instant messaging example:

A: tell me why im taking math again? >.<

B: because you're an idiot

A: T_T thanks a lot oh lovely friend of mine

A: lol

Here A is not literally wincing in pain (>.<), but humorously expressing her feelings of frustration with her math course. Her feigned flowing tears (T_T) in response to being called an idiot are tempered by both the sarcasm of “thanks a lot oh lovely friend of mine” and by the lol she sends as a coda in line 4.

Since the 1990s in Japan, and since about 2009 in the US and Europe, smartphones allow users to insert emoji, or full color emoticons, into their texts. Some emoji are specific to Japanese culture (a cup of sake, a bowl of ramen noodles, a white flower for brilliant homework), but many others are more broadly applicable.

Emoji go well beyond just expressing affect; they can be used as a kind of rudimentary sign language. The “Narratives in Emoji” blog offers scenarios and movie plots recounted in emoji. For example, Les Miserables is illustrated in ninety-five emoji, and Titanic is illustrated in fourteen emoji starting with a ship's anchor and ending with a broken heart.Footnote ¹³ In certain online environments, ASCII emoticons are automatically converted into emoji. For example, when I type :-) in Microsoft Word, my keystrokes are converted into ☺. Many people find these transformations annoying precisely because they are automatic (though the autocorrect feature can be overridden) and they produce an overly ‘cutesy’ look that may be ill suited to the impression the writer wants to convey.

Emoji have become a standard built-in feature in certain chatrooms and blogging environments, and often they are animated as well. Applications such as CLIPish provide full-motion animations that can be inserted into texts to express affect (Figure 7.1).

Figure 7.1 Still from a full-motion animation expressing jubilation

Given that writing has always had its ways of expressing affect and attitude, why have emoticons and emoji developed now? No doubt, the rapid pace and truncated language of chat, instant messaging, and texting has something to do with it, for one has little time or space to elaborate on one's mental states in those environments. Emoticons are quick, easy, and synoptic. Their less frequent use in email (Randall, Reference Randall2002) suggests that when people do have the time and space to write more fully, they also have less need of emoticons.Footnote ¹⁴ However, there is also an organic, cultural dimension to the arrival of emoticons. On computers and smartphones, where icons are the dominant organizational and navigational devices, where images increasingly compete with and complement words, and where one person's creative moment can quickly go viral, the emoticon is a logically consistent outgrowth that is superficially emblematic of the digital ethos. ‘Superficially’ is precisely why the use of emoticons varies so much from individual to individual: some computer novices wanting to fit in will use emoticons as digital membership badges. Some more experienced digiterati eschew emoticons because they see them as silly and devoid of any real meaning. Most people are probably in between, willing to use an occasional emoticon for a touch of humor, but essentially indifferent to them. In any event, emoticons are for now a fixture in the online landscape and, as we will see in the next section, people have developed nuanced conventions by which to interpret them.

Discursive dimensions of online communication

Electronically mediated writing can be synchronous (such as chat and MOOs) or asynchronous (such as blogs, websites, wikis, forums). A number of platforms can be used either synchronously or asynchronously (for instance, email, texting, instant messaging, Twitter). In this section we will contrast two examples of online collaboration, one synchronous (MOO) and the other asynchronous (Wiki).

The kind of writing produced in synchronous environments is first and foremost social interaction, and its form reflects that purpose. Like speech, written chat exchanges are characterized by direct interpersonal address, rapid topic shifts, and frequent digressions. Similarly, the functions expressed in chat frequently overlap with purposes normally associated with speech. Writing becomes a channel for lively, spontaneous exchange of thoughts, feelings, ideas, and wit, and tends to be oriented toward developing social relationships and entertaining others more than for informing, explaining, or persuading.

Conclusion

We have seen that electronically mediated discourse is not just shaped by technology but also by broad social forces as well as the widely varying needs of individuals in particular situations. Each act of online communication brings into play a particular set of language forms and communicative practices that are dynamically adapted to the setting and task at hand. Although these adaptations get socially shared (i.e., taken up as new Available Designs), there is no uniform language of electronically mediated communication as might be suggested from David Crystal's (Reference Crystal2006) term Netspeak. Even the various media within the broad category of electronically mediated writing (i.e., email, chat, texting, etc.) cannot be unambiguously associated with particular genres – in fact, each of them can support multiple genres.

The language inventions found in chat, IM, and SMS environments (which sometimes bleed over into blogs, email, discussion lists, etc.) are not really a simplification of writing systems, but an adaptation of those writing systems to allow inclusion of features needed by the online culture of communication. That is, they are features that the forms of writing associated with print culture don't adequately provide. This adaptation is nothing new; this is what has happened with writing again and again throughout its 5,000-year history.

Moreover, graphic reductions are not specific to online environments. Just to give one example, terms of address are frequently abbreviated (e.g., Mr., Mrs., Dr. in English; M., Mme, Mlle in French; Sr., Sna., Da., Ud., Uds. in Spanish).Footnote ¹⁹ Such forms are not signs of the degradation of language, and neither are those found in online environments. Indeed, we have seen that the strategies involved in using reduced or recoded forms often involve shifting among multiple representational systems, thus demonstrating the symbolic sophistication and cognitive flexibility of those who use them.

Clearly, writing is not a purely linguistic activity; it involves designing meaning using forms and space. It is partly linguistic, but never exclusively linguistic. The forms of writing may come from different classes of signs, grounded in different representational systems, and an important part of literacy is developing the ability to deal with the multiple systems that underlie written texts.

In the next chapter we will extend our exploration of electronically mediated discourse beyond writing, considering multimedia environments that involve image, sound, or video.

The old and the new

Multimodal communication is not new. Indeed, it has historically been the norm for most communication, whether spoken, written, or signed.Footnote ⁴ That is to say, although most acts of communication involve some form of language, language rarely stands on its own. We have seen how images and language have complemented one another in texts since the origins of writing, and images were no doubt combined with speech in communication and artistic expression for millennia before that. Medieval illuminated manuscripts, rebus books, theater performance, science textbooks with diagrams and illustrations, concrete poetry, comic books, newspapers and magazines, advertising, instructional manuals, television captioning are all common examples.Footnote ⁵

What is new is that in digital environments, verbal, visual, and sonic modes are all represented numerically. This provides a common architecture for integrating music, sound, film, and graphic arts in ‘written’ texts.Footnote ⁶ By the same token, tools for ‘writing’ may be used to compose in other modes. For example, the voice tracks for video animations can be produced by typing on a keyboard with interfaces such as Xtranormal or GoAnimate. In these kinds of situations, ‘authoring,’ ‘composing,’ or ‘designing’ are more apt terms than ‘writing,’ even though the biomechanical processes look identical to those involved in writing text. Numerical representation also means that anyone who has access to the requisite computer software has the wherewithal to create, transform, or re-mediate content and to disseminate it widely on the Internet, potentially attracting massive audiences.Footnote ⁷ For example, visual mash-ups, such as the one shown in Figure 8.1, are a popular internet practice of combining images and text from a variety of sources to produce something similar to a comic strip, in which the semantic load is distributed across linguistic and visual modes. Here, Kobe Bryant of the Los Angeles Lakers is put into dialogue with Gandalf from the movie version of Tolkien's The Lord of the Rings: The Fellowship of the Ring, with the humor centered on the ambiguity of the word pass and Bryant's penchant for hogging the ball and shooting. Mash-ups rely on readers’ familiarity with pop culture and are frequently found on Facebook and other social networks.

Figure 8.1 Visual mash-up: Kobe Bryant and Gandalf in conversation

Remixes like this circulate widely as internet memes and give rise to many variations on the basic theme. Originality does not derive from the borrowed images or utterances themselves, but rather from the particular juxtapositions that bring the images and utterances into novel relations with one another. These kinds of recontextualizations of others’ work remind us that creativity is not just an individual trait, but also a socially distributed and collaborative practice.

Recontextualizations also introduce questions about intellectual property and ethics – questions that I myself face as I cite this example in this book. For although I can identify the middle image of Figure 8.1 as belonging to the film The Lord of the Rings: The Fellowship of the Ring, I cannot identify the sources of the other images nor the author who put the words and images together.Footnote ⁸ In sum, there is no readily identifiable original source to cite. But authentication and citation is not part of mash-up culture – it only becomes an issue when I remove the mash-up from its native material and social ecology and try to incorporate it into a non-native medium (a published book) that conventionally requires explicit acknowledgment of sources. This transplantation of a text (which is itself composed of other transplanted texts) illustrates how material and social conventions interact and shows us that what is distinctive about this act of recontextualization is not the borrowing itself (for, as Bahktin (1986) reminds us, our texts are always filled with others’ words) but rather how we acknowledge that borrowing in different contexts. The world of internet remixing and repurposing is like one that Foucault invited us to imagine, namely: “a culture where discourse would circulate without any need for an author,” a world in which discourses “would unfold in a pervasive anonymity” (Reference Foucault, Bouchard, Simons and Bouchard1977b, p. 138). Such visions are at the heart of current debates about plagiarism and its cultural relativity (Pennycook, Reference Pennycook1996). But multimodal texts take the debate to a new level in that they involve the borrowing not only of language, but also of design more broadly, since the look, sound, and feel of a digital object can now be copied and pasted just as easily as language can.Footnote ⁹

Sharing the semiotic load: graphic overtitles

Let's now take a closer look at how meaning making involves interactions among semiotic modes in multimodal texts. We will take as an example a popular form of captioning used in Korean television shows. Unlike closed captioning, in which ongoing speech is simultaneously shown onscreen in written form, or subtitling, in which speech is translated into another language, this particular form of captioning does not reiterate what is being said onscreen. Rather, it expresses unspoken voices – the internal thoughts of a character, a comment by the producer, a projected reaction from the audience – which may sometimes contrast with the spoken language, images, and actions that are unfolding on the screen. Linguist Junghee Park (Reference Park2009) calls these captions “graphic overtitles” and points out that they add a dialogic dimension to the viewing experience, making it like watching a television show in the company of friends who laugh, make snide remarks, admire stunts, imagine what a character is thinking, and make predictions about what will happen next. Similar in many ways to the overtitles used in silent films to read the thoughts of characters, Korean graphic overtitles are distinctive in that they draw heavily on graphics and symbols from Korean comic books and electronic mediated communication, with typographical variation and color used to express different voices. However, there is no explicit specification of whose ‘voice’ we are reading, since overtitles float anonymously on the screen.

For example, in Figure 8.2, the three co-hosts are laughing about a man's story about his grades in elementary school, and the graphic overtitles can be translated as ‘big laughter’ written diagonally in purple, followed by ‘got bowled over’ written horizontally in yellow below. What is interesting here from a linguistic standpoint is the absence of a grammatical subject, which Park argues is a characteristic syntactic feature of many graphic overtitles. So, instead of saying ‘the co-hosts were bowled over,’ only the words ‘got bowled over’ appear on the screen. The grammar is adapted to the multimodal context, where the grammatical subject (here, the co-hosts) is represented visually on the screen and only the predicate needs to be expressed in language. Park points out that this principle of omitting subjects that are represented visually onscreen also holds in the case of relative clauses. For instance, ‘a girl who is pretty’ reads simply as ‘who is pretty’ in the graphic overtitle because the girl appears on the screen. In other words, expression of the total semantic structure is distributed across linguistic and non-linguistic resources. Viewers thus read the television screen somewhat like a rebus text.

Figure 8.2 Graphic overtitles that read “Big laughter. Got bowled over”

To a non-Korean sensibility, overtitles might seem to overdetermine meaning. Are they too controlling of audience response? Do they reduce the interpretive work of viewers and consequently limit the range of possible responses and interpretations? Whatever the answer, one thing is clear: Korean viewers consider them an integral part of TV game and reality shows. Park reports that when one episode of the show Infinite Challenge aired without graphic overtitles, viewers responded immediately by creating their own overtitles and uploading the modified video onto the Internet (2009, p. 22). Such acts show how active a role individual viewers can play in modifying and re-disseminating media broadcasts. At the same time, such acts remind us that even in a globalized world, the particular forms, norms, and practices that people apply are deeply embedded in culture-specific expectations and literacy traditions.

If semiotic modes interact, they also affect how the signs they mediate are interpreted. When information is represented and re-represented across modes, the meanings it generates can be altered. We will explore this idea in the following section.

Modes and mediation

Multimedia can filter and transform realities in subtle ways that are not immediately recognized. In other words, what seems ‘natural’ may in fact be highly mediated.

Consider, for example, the 2008 ‘text message scandal’ of Detroit mayor Kwame Kilpatrick. As described by Squires (Reference Squires, Thurlow and Mroczek2011), private text messages between Kilpatrick and his chief of staff (and love interest) Christine Beatty were remediated on television news broadcasts. In the process, the text messages became laden with ideological overtones as they moved from ‘private’ to ‘public’ and from ‘source’ to ‘representation.’ Squires outlines the multiple layers of remediation and recontextualiztion of Kilpatrick's and Beatty's text messages as they appeared in the Sky Tel digital log file, in court document spreadsheets, in on-screen television displays of the texts, and as they were read aloud on news broadcasts. In the process, the particular ways the text messages were reproduced, contextualized, and sequenced in the news broadcasts had the effect of toning down the playful, inventive qualities of the text messages (see Chapter 7) and radically reframing them as ‘adult’ discourse, thereby negatively shaping viewer perceptions of Kilpatrick and Beatty.

It is important to recognize, however, that it is not the technologies themselves that produce these rhetorical effects. It is rather the journalists who use the retextualizing technologies to present the ‘data’ in seemingly objective fashion, as they conceal their true rhetorical intent. Technology may contribute to the shaping of discourse, but it is also guided by the imagination, rhetorical skill, and ethical responsibility of the individuals who design the discourse to pursue a communicative agenda. We turn now to a second, more involved example.

Videoconferencing between California and France

People separated by distance have traditionally been obliged to communicate by writing or by telephone. The Internet has made synchronous audio-visual communication possible through videoconferencing programs such as Skype. Although at first blush videoconferencing gives the impression of immediacy (Bolter & Grusin, Reference Bolter and Grusin2000), what gets included in the audio-visual signal transmitted from one conversational partner to another is filtered and transformed by both hardware and software. This means that people sometimes have to manage semiotic resources differently than they do in face-to-face conversation.

One research project I've been involved with for a number of years examines videoconferencing interactions between learners of French in the United States and graduate students in France who are preparing to become teachers.Footnote ¹⁴ Both American and French cohorts consider their online interactions to be authentic, highly engaging, and a welcome addition to their regular academic courses. Nevertheless, a number of significant mediational issues underlie their videoconferencing exchanges. Below we will consider a number of ways in which visual and audio cues can be transformed by technological mediation and how the students we have studied adapt to those transformations.

The webcam is the eye through which all visual communication occurs in videoconferencing. Because webcams are often built into computers, they cannot be easily repositioned, and users have to remain relatively still if they are to remain in the webcam's field of view. This means that people may unnaturally restrict their movement when videoconferencing. The webcam also exaggerates the effects of physical movement (see Figure 8.3). Whereas a short distance from the webcam creates a sense of immediacy and intimacy, a distance of even three feet makes one look distant. Parkinson and Lea (Reference Parkinson, Lea, Kappas and Krämer2011) found that when people videoconference with people they don't know well, they tend to compensate for relatively intimate visual contact by talking about less personal topics in order to increase social distance. “Paradoxically,” they write, “one consequence may be that [videoconferencing] produces less intimacy than text-based or audio-only communication, because, in the latter cases, interactants may seek to increase rather than decrease the emotional relevance of the conversation itself when fewer alternative cues are available” (p. 103).

Figure 8.3 Shifts in position are exaggerated by the webcam

As anyone who has used a videoconferencing application knows, eye gaze is managed very differently online. Real eye contact does not exist. When interlocutors look at each other, they appear to be looking downward. If they want to create the illusion of looking into their interlocutor's eyes, they have to look directly at the webcam, but then paradoxically they cannot see their interlocutor at all. Figure 8.4 shows a woman asking a question, looking directly into the webcam to create an illusion of eye contact, and then looking at the screen as she listens to her interlocutors’ response. Users adjust quickly to the new dynamics of gaze management in videoconferencing – sometimes even claiming during interviews that they maintained frequent direct eye contact with their interlocutors (even though mutual eye contact is currently impossible).

Figure 8.4 Looking at the webcam (left) and looking at her interlocutors (right)

Webcams mediate gestures as well. Although people often gesture extensively during videoconferencing exchanges, when their gestures occur outside the webcam's field of view they remain invisible to their onscreen partners. Ironically, the closer a speaker is to the webcam (e.g., leaning in toward the computer, signaling a high level of involvement), the less likely it is that his or her gestures will be captured by the webcam. On the other hand, the greater the speaker's distance from the webcam (suggestive of social distance in face-to-face interaction), the greater the likelihood that gestures will be picked up by the webcam. Perhaps as a way of compensating for the webcam's limited visual field, people often heighten their facial expressiveness as they speak (Figure 8.5).

Figure 8.5 Animated facial expressivity online

Perhaps the most significant limitation of webcams is that they can create the illusion that people are looking at each other when in fact they may not be. In one instance, two Californian students launched a video that had been mentioned by their French online partners, while they were interacting with them. As they watched the video (which covered their Skype window), their French partners tried in vain to engage them. The Californian students could not see their French interlocutors (and could not hear them over the sound of the video), yet to the French it appeared that the Californian students were looking at them normally, so they erroneously concluded there was a technical problem with the connection.Footnote ¹⁵

Audio mediation is significant as well. We are accustomed to the transistorized sound of speech in the telephone, but bandwidth and connection issues can make speech in videoconferencing downright garbled at moments. Moreover, there is no way to know if one's own voice is distorted unless one's interlocutor says it is. Lag and desynchronization of the audio and video signals can also affect communication. While videoconferencing, people generally have the impression they are seeing their interlocutors directly, as if they were right in front of them. But in fact tiny delays are produced as the signal is compressed, delivered over hundreds or thousands of miles, and decompressed. What is perceived, both audially and visually, has been produced in the past, however slightly. The effect this has is to introduce a slightly awkward rhythm to the interaction, making one feel like one is always just a tad behind the beat. At moments when the delays become more pronounced, users sometimes wonder whether the desynchronized smiles, gestures, or facial expressions they see onscreen are responses to what they are saying right now or whether they correspond to what they said a moment earlier. And people sometimes mistake a transmission lag for hesitation on the part of their interlocutor. These ambiguities, combined with dropped frames, which result from bandwidth limitations and produce a jerky appearance to body motions, can potentially present real challenges to understanding.

Mediation and learning

From the above discussion, it might seem that technological mediation in videoconferencing has a largely negative effect. That would be to overestimate the impact of the medium, since, as mentioned earlier, participants have overwhelmingly enjoyed their online interactions. The point is that even though participants are satisfied with their overall experience, they nevertheless face small challenges introduced by the medium itself – in addition to the larger challenges they face in understanding and making themselves understood in a non-native language. What is important is that all participants be aware of the small effects of the medium so that ‘medium behavior’ does not become confused with participant behavior.Footnote ¹⁶

From an educational standpoint, videoconferencing offers not only an opportunity to learn from others at a distance, but also an opportunity to learn from retrospective analysis of the interaction itself. Recording technology textualizes communicative interactions, making it possible for students to archive, analyze, discuss, and learn from whatever misunderstandings might have occurred during their online exchanges. One student described the experience of reviewing a recording of a videoconferencing session as follows:

It's kind of like rereading a book that you've already read in terms of familiarity. You kind of know what's coming up next, so the next time you catch more stuff. So I felt like I had a better clarity of what exactly happened in the conversation. I have more self-awareness in terms of what I said, I'll be like, “oh, ok” I know what I was trying to say at that point in time, I can hear now what I did say, but I think now that I could have said it better this way. So it's sort of similar to going back and editing an essay or anything like that, you're re-appraising your own work. And plus it's nice to kind of pick up maybe on a few things you missed out on in that conversation because there's so much to be learned in the course of those conversations, and you hate the fact that there's only so much you can learn.

This student's comments raise a number of key points. First, because he is no longer in the heat of the communicative moment, he can attend to details of the interaction that he hadn't noticed the first time around. Second, having directly experienced the event, he can anticipate moments of uncertainty or misunderstanding and pay special attention to these points. Third, when he compares his memory of the interaction with the objective data of the video recording, he comes away with a greater sense of self-awareness. Finally, likening the process to revising an essay, the student has an opportunity to self-assess and to think about alternative words or actions he could have used, potentially enlarging his repertoire for future interactions.

We will return to the use of technology to heighten users’ awareness of mediation in Chapter 10. We now turn to the question of how multimodal discourse might affect how we think about literacy.

Literacy and multimodal discourse: is the image replacing the word?

It is frequently observed that literacy is undergoing an important shift from a dominance of writing to a dominance of image. Pundits now take it as a given that images speak louder than words. For example, commenting on the image of President Obama speaking at the ceremony commemorating the fiftieth anniversary of Martin Luther King's “I Have a Dream” speech, critic Alessandra Stanley wrote: “Rhetoric soars, but everyone knows a visual image can convey more than even the most dramatic or inspiring language. Dr. King's words still captivate, but nowadays, the camera trumps sound” (2013, p. A16). One may not agree with Stanley's assessment, but it certainly resonates with much of the discourse in the popular press, and with some market trends. For example, in US mobile networks, images are trending up and text is trending down. According to a wireless trade association report, US mobile networks carried 41 percent more multimedia images but 4.9 percent fewer text messages in 2012 than they did in 2011 (Lawson, Reference Lawson2013).Footnote ¹⁷ Images and especially video are also becoming increasingly common ways of accessing information: YouTube has become a first-choice search site and internet entry point for many young computer users – hence Google's purchase of YouTube. We increasingly develop visual models of information and concepts that used to be presented only in prose. PowerPoint presentations are now the default rather than the exception at professional conferences. Online, we engage in new forms of participation that may involve avatars whose appearance we design.Footnote ¹⁸

What are the implications for literacy? Because writing and images operate with different logics, social semiotician Gunther Kress predicts that

Kress (Reference Kress2005, Reference Kress2010) elaborates on this point by arguing that telling and showing involve different epistemological commitments. Kress compares writing or saying “A plant cell has a nucleus” with drawing a picture of a cell, pointing out that whereas the verb have expresses a relation of possession or ownership, the drawing does not.Footnote ¹⁹ However, the act of drawing forces a decision about where the nucleus should be positioned within the cell walls, which the linguistic utterance does not. Thus writing or speaking confronts us with a different kind of epistemological commitment from that of drawing. Kress argues that when we use language we deal in categories of relations (such as causality, possession, and so on) and when we use images we are concerned with the positioning of elements in a framed space.

This line of argumentation is perhaps overly reductive in the sense that the language/image distinction is not always so stark. Writing always involves spatial arrangement and speech is most frequently accompanied by visible gestures, body language, and facial expressions that are often crucial to interpretation. And drawing can show logical relations through conventional signs like arrows. Nevertheless, Kress's distinction between telling and showing does seem useful when thinking broadly about how we are positioned to think in certain ways by different modes and mediums of expression. Jones and Hafner (Reference Jones and Hafner2012) offer the example of describing a traffic accident. English speakers will tend to formulate their narrative in a subject–verb–object pattern, attributing agency (as in ‘The truck hit the car’), whereas if they produce a drawing of the accident, they will attend to the particular spatial relationship between the truck, the car, and the road, and perhaps other vehicles as well, but might not include any information about agency (pp. 52–53).

If language and image offer different resources for meaning making and involve different epistemological commitments, then there are also implications for learning. Chun and Plass (Reference Chun and Plass1997), for example, point out that learning from text and learning from images involve different cognitive processes and that educators can take advantage of these differences to improve learners’ comprehension through strategic uses of different modes. However, when it comes to graphics, the technology that allows a teacher to conveniently prepare illustrations of complete models, data charts, and so on in advance of a presentation may not always produce a good learning experience for students. It is often far more comprehensible to watch a presenter draw a diagram or a model on a blackboard, seeing the step-by-step process that lies behind what is being drawn, than to see only the finished product, however aesthetically superior it might be, flashed on a screen.

Communications scholar Mitchell Stephens speculates that the predominance of images will eventually affect both our writing and our speech: “the language our descendants write and speak will increasingly be a less precise, less subtle language – one designed for use with images” (Stephens, Reference Stephens1998, p. 207). Whether or not language becomes less precise and subtle, one thing is clear: if language was king in the mediums of print and radio, which made language look and sound autonomous, today's television and computer mediums remind us how fundamentally language interacts with and complements other semiotic modes. Word and image are both important technologies for the design of meaning, and literacy requires competence with the one just as much as the other. As we have seen, technologies do not replace one another, but remediate and complement one another. Written text is at no risk of disappearing, but it will increasingly share the semiotic stage with other players. This mutual dependency can be seen on the audio-visual side as well: photos, videos, and music have become ‘linguistic’ in the sense that they have verbal tags that allow them to be searched, grouped, and shared.Footnote ²⁰ These tags add value to the original visual object by allowing its features and qualities to be specified by multiple users. The identity (and usefulness) of visual objects is thus increasingly textualized over time.

Conclusion

Literacy has always been multimodal. But today we are faced with more choices than ever about how to communicate. What to express through still or moving images? What to express in speech? What in writing? And what will it ‘say’ to express it in that particular medium? Furthermore, each choice entails subsequent decisions about the particular mediational tools and techniques to be used, and the most appropriate styles and forms of language to employ. These decisions matter, because modes, technologies, and language are not neutral conduits.

Designing meaning is relational work, and it is rhetorical work. Whereas the rhetorical decisions involved in technology-mediated communication used to be reserved for specialists, they are now often in the hands of ordinary people. This change has important implications for education. If different modes and media have different affordances, then it is important for educators, first, to understand what those affordances are in specific acts of reading and writing and, second, to help young people develop their understanding of the ‘invisible’ processes that go into their practices. This will be taken up in Part III.