We discussed in Chapter 2 the importance of cohesion and coherence to comprehension and how these findings were the main impetus for developing Coh-Metrix. Our primary goal in the Coh-Metrix project has been to develop, explore, and validate measures of text cohesion. Throughout the Coh-Metrix Project we have developed and implemented many approaches to assessing cohesion as well as other levels of language and discourse. The magnifying glass has primarily been on cohesion, so we have developed literally hundreds of cohesion indices that vary in generality (see Chapter 4 for the distinction between measure, index, bank, and variable). Some indices have targeted one general construct, such as referential cohesion, whereas others have drilled to a more specific level, such as temporal and verb cohesion. A significant portion of our efforts has gone toward rooting among the indices to choose the best ones and validating new ones. When there are many indices to measure a similar construct, it has been necessary to identify which ones rise to the top across the various studies and within studies. The indices need to be validated so that we have some assurance that they assess what we think they are assessing and that they are theoretically compatible with patterns of data corresponding to types of texts or human performance. For example, some studies show how particular indices account for differences between texts that fit predictions based on theory or well-accepted empirical findings. Alternatively, some indices are validated by patterns of data in psychological experiments using behavioral tasks. We have conducted many such validation studies. This chapter describes some of the studies we have conducted, particularly as they relate to referential, semantic, and situation model cohesion. The chapter begins by examining measures of cohesion in the context of empirical text comprehension studies and differences between types of text. We subsequently describe our work examining the role of cohesion in writing.

As we mentioned at the beginning of Chapter 9, the Method section of most Coh-Metrix research papers comprises a section on the corpus and a section on the tool. In the last chapter, we discussed the corpus, and in this chapter we complete the traditional Method section by discussing the tool (which in this case is Coh-Metrix). Of course, by the time you have reached this point in the book, we assume that you already know a thing or two about Coh-Metrix. That is, you will likely know what it is for, which buttons to press to make it function, what the output looks like, and what some of the indices are. But knowing what Coh-Metrix is and knowing how to describe it for your readership are two quite different things. It is this description of the tool and its reason for inclusion in your project that forms the focus of this chapter.

The Four Major Moves of the Tool Section

In a research paper, the section describing the tool is relatively short because your study is not likely to be about Coh-Metrix; it is merely using Coh-Metrix. Therefore, all you need in order to describe the tool is to employ the following four moves: explain (1) what the tool is, (2) what it does, (3) why it can be trusted, and (4) why it is appropriate for the current study. To better understand these moves, let’s look at some examples that we’ve adapted from a study by Hall et al. (2007).

Some texts are easy to read. Others are difficult. That is perfectly obvious. The challenge lies in devising an objective means to measure texts on how difficult they are to read. That is one of the puzzles that motivated our development of Coh-Metrix and ultimately the writing of this book. How do we scale texts on comprehension difficulty? Or on the flip side: easability?

It is often quite clear when texts are difficult or easy. Consider the two texts below and cast your vote on which is difficult and which is easy.

Lady Chatterley’s Lover

He spread the blankets, putting one at the side for a coverlet. She took off her hat, and shook her hair. He sat down, taking off his shoes and gaiters, and undoing his cord breeches. “Lie down then!” he said, when he stood in his shirt. She obeyed in silence, and he lay beside her, and pulled the blanket over them both.

A Mortgage

The assignment, sale, or transfer of the servicing of the mortgage loan does not affect any term or condition of the mortgage instrument, other than terms directly related to the servicing of your loan. Except in limited circumstances, the law requires your present servicer send you this notice within 15 days before this effective date or at closing.

We do not need to conduct a survey to discover how most English speakers will vote. The Chatterley text by D. H. Lawrence is clearly easier than the mortgage text. The question is why?

This chapter describes the scientific and technological advances that were the precursors to the development of Coh-Metrix. The Coh-Metrix team has developed numerous computational algorithms and procedures for measuring ease (versus difficulty) at the various levels of language and discourse. We are satisfied with our progress and achievements, but we cannot emphasize too much that Coh-Metrix was hardly built in a vacuum. Coh-Metrix can be viewed as a sandbox of automated language and discourse facilities that were developed not only by our research team but also by others in computational linguistics, corpus linguistics, discourse processes, cognitive science, psychology, and other affiliated fields. We were able to build Coh-Metrix because we had the advantage of standing on the shoulders of giants.

The contributions of our predecessors come in many varieties. Some noteworthy examples of these contributions are highlighted below.

1. One type of contribution is lexicons or dictionaries of words that list qualitative features or quantitative values for each word. For example, WordNet (Fellbaum, 1998; Miller, Beckwith, Fellbaum, Gross, & Miller, 1990) stores semantic and syntactic features of nouns, verbs, adjectives, and other content words in the English language. The MRC Psycholinguistic Database (Coltheart, 1981) has human ratings of thousands of words on familiarity, imagery, concreteness, and meaningfulness. The CELEX Lexical Database (Baayen, Piepenbrock, & Gulikers, 1995) has estimates of how frequently English words are used in a very large corpus of documents.

2. A second type of contribution is from applications. An application is a fully functioning program that takes text as input and computes some language or discourse code as output. We use the output when we create a Coh-Metrix measure. A good example of this is when we used a syntactic parser (the one Coh-Metrix uses was based on Charniak, 2000). The parser takes each sentence as input and computes a syntactic tree structure. These tree structures are used when we scale texts on syntactic ease or difficulty.

3. …

By now you should have a fair idea of what the Coh-Metrix tool is, what it is for, where it all came from, and how to use it. However, knowing how to operate a text analysis tool like Coh-Metrix and knowing how to write up a research paper using a tool like Coh-Metrix are two very different things. In this part of the book, our goal is to show you how to write such a paper. What we have in mind is a short project paper, the kind of paper that would serve well as a term paper, a conference proceedings manuscript, or even the basis of a journal article, thesis, or dissertation.

A term paper, a conference proceedings manuscript, a journal article, a thesis, and a dissertation may all sound like very different composition types. However, there is a remarkably similar thread that runs through each of them. After all, whatever the Coh-Metrix project is, there is still the need to inform the project’s audience of such questions as What is the project about?, Why was it done?, How was it done?, What are the results?, and What does it all mean? In many ways then, whether writing something as short as an abstract or as long as a dissertation, the key aspects of a research paper are almost always present. It is those key aspects, questions, or communication moves (Swales, 1981, 1990) that we will be highlighting and discussing in this part of the book. By showing you where in the composition these moves occur, what they function as, what they look like, and how to write them, we hope to provide you with a thorough guide to writing an excellent Coh-Metrix research paper.

The Results section in a research paper is generally the last section to get started but it is often the first section to get finished. That is, once you have collected your data (the corpus), you’ll need to analyze it, and once you’ve analyzed it, it is a relatively simple task to write it up. It is a relatively simple task to write it up because the results section is (or least can be) highly formulaic. Indeed, some software (e.g., the Gramulator; McCarthy, Watanabi, & Lamkin, 2012) actually conducts statistical analyses and automatically outputs an acceptable (if highly formulaic) results section.

As in Chapters 8, 9, and 10, this chapter looks at the writing process for a short Coh-Metrix paper in terms of moves and frozen expressions. We will also look briefly at the meaning of those strange letters and numbers that are the main feature of the results section (the t’s, p’s, d’s, etc.). We have made every effort to make this chapter as accessible as possible, assuming that the reader is relatively new to reporting statistical results; however, as mentioned at the beginning of Part II, we have also assumed that the reader has some statistical knowledge. Therefore, the reader should be aware that it is beyond the scope of this book to explain in any great depth what statistics are, which kinds of statistics are appropriate for which kinds of analyses, how statistics work, how they are calculated, how they should be interpreted, and how they are often misinterpreted. To address questions such as these in more detail, there are excellent resources available, such as the textbooks SPSS Made Simple (Kinnear & Gray, 2008) and SPSS for Intermediate Statistics: Use and Interpretation (Leech, Barrett, & Morgan, 2008). There are also excellent Web resources such as www.talkstats.com and http://vassarstats.net.

This book describes Coh-Metrix, a computational tool that provides a wide range of language and discourse measures. It is a linguistic workbench that researchers, teachers, and students of many different disciplines can use to obtain information about their texts on numerous levels of language. This book consists of two parts. The first section focuses on the theoretical motivations and perspectives that led to the development of Coh-Metrix. Part I describes its technological foundations, the measures it provides, and empirical work that has been conducted using Coh-Metrix. We see Part I as being invaluable to researchers who wish to situate their Coh-Metrix work within the theoretical and empirical fields of discourse processing, psycholinguistics, text design, and related fields.

Part II shifts to the practical and pedagogical arena, describing how to use Coh-Metrix and how to analyze, interpret, and describe Coh-Metrix results. This section is written for computational novices and students who wish to not only use Coh-Metrix (or similar computational tools), but also describe the resulting studies and their outcomes.

This appendix provides norms for the indices described in Chapters 4 and 5. To create these norms, we analyzed a subset of a large corpus of texts created by the Touchstone Applied Science Associates (TASA), Inc. The total TASA corpus includes 9 genres consisting of 119,627 paragraphs taken from 37,651 samples. The norms are provided for the three largest domains represented in TASA: language arts, social studies, and science texts. To do so, we randomly chose 100 passages from each of the 3 genres and each of 13 grade levels, for a total of 3,900 passages.

Grade level in the TASA corpus is indexed by the Degrees of Reading Power (DRP; Koslin et al., 1987), which is a readability measure that includes word- and sentence-level characteristics. As can be observed in the table, DRP is highly correlated with the Flesch Reading Ease and Flesch–Kincaid Grade Level measures of readability.

To simplify the data analysis and presentation, DRP levels were translated to their corresponding grade-level estimates and then collapsed according to the grade bands used within the Common Core State Standards: grades K to 1, 2 to 3, 4 to 5, 6 to 8, 9 to 10, and 11 and higher. Each grade level within each genre was represented by 100 passages. Because the Common Core grade bands include different numbers of grade levels per band (e.g., 2–3 includes two grades, 6–8 includes three grades), there are different numbers of passages represented for each grade band. The average DRP values as well as the range of DRP values for each grade band are provided in the Table B.1.