Introduction

The language tests reviewed in the previous chapter involved the grammar sections from large-scale tests designed to measure global language proficiency, typically for academic purposes. Like other large-scale and often high-stakes tests, they were designed to make institutional decisions related to placement into or exit from a language program, screening for language proficiency or reclassification of school status based on whether a student had achieved the language skills necessary to benefit from instruction in the target language. These tests provide assessments for several components of language ability including, among others, aspects of grammatical knowledge. In terms of the grammar sections of the tests reviewed, a wide range of grammar points were assessed and, except perhaps for the CEP Placement Test, the selection of test content was relatively removed from the local constraints of instruction in specific contexts. These large-scale tests were designed as one-shot, timed assessments for examinees who bring to the testing situation a variety of experiences and proficiency levels. The tests were different in the ways in which the qualities of usefulness were prioritized, and the compromises that ensued from these decisions.

Although large-scale, standardized tests have an important role to play in some school decisions and can have a positive impact on learning and instruction, the primary mandate of large-scale exams is different from that of classroom assessment. In the first place, large-scale language assessments are not necessarily designed to promote learning and influence teaching in local contexts.

Introduction

In the previous chapter, I discussed the role of grammar in models of communicative competence and showed how a more detailed depiction of grammar was needed in order to assess how learners use grammatical forms as a resource for conveying a variety of meanings. I argued, as many others have, that language consists of grammar and pragmatics, and these two components should be clearly differentiated. Finally, I presented a theoretical model of language in which grammar is defined in terms of both form and meaning, and where pragmatics refers to a variety of implied meanings superimposed upon the grammatical forms and meanings of an utterance.

In the current chapter, I will discuss how the proposed model of grammar can be used to define what it means to have second language grammatical ability. In so doing, I will describe a theoretical model of grammatical knowledge that can serve as a basis for grammar-test construction and validation. Before describing this model in much greater detail and applying it more specifically to assessment, I will clarify the many terms that are used in this discussion.

What is meant by grammatical ability?

Having described how grammar has been conceptualized, we are now faced with the challenge of defining what it means to ‘know’ the grammar of a language so that it can be used to achieve some communicative goal. In other words, what does it mean to have ‘grammatical ability’?

Introduction

In the previous chapters I reviewed several theories of grammar and we saw how language teachers and SLA researchers drew on these theories to define ‘grammar’ in their work. I also discussed how different instructional practices influenced how learners acquire knowledge of a second or foreign language grammar. Implicit in this discussion was the notion that knowledge of the L2 grammatical system can be demonstrated by a learner on some outcome measure, whatever form that might take, and that teaching can potentially influence the results obtained on this measure. From the results of these assessments, we can then make inferences about the students' grammatical ability, which would subsequently provide an empirical basis for decision-making. For example, language teachers use test results to make decisions about student placement in a language program or about the degree to which their students have mastered the material in a course, and SLA researchers use test results to make decisions about whether young learners acquire grammatical features better than older learners. Implicit was also the notion that if more than one assessment of grammatical ability was obtained over time, inferences related to grammatical learning or even the effectiveness of instruction could be determined based on the observed changes in what learners demonstrate on these measures. This information is of particular concern to language teachers, testers and SLA researchers for making instructional recommendations and for theory-building.

Grammar, the structural glue, the “code” of language, is arguably at the heart of language use, whether this involves speaking, listening, reading or writing. Grammar has also been central to language teaching and assessment historically, from the Middle Ages, when “rhetoric” was a key component of a university education, to the “skills-and-components” models of the 1960s that informed both language pedagogy and language testing.

However, although the way grammar is currently viewed, both in theoretical and applied linguistics, and in language learning and language teaching, is vastly different from the perspective that informed grammar tests in the 1960s, very little has changed since then in the way language testers conceive of grammar and in the way it is assessed in practice. Thus, many of the grammar tests that are currently in use, both in largescale and in classroom assessment, reflect the perspectives of structural linguistics and discrete-point measurement. This book takes a completely new look at the assessment of grammar, placing it in the context of current views of linguistic pragmatics and functional grammar. It thus brings the assessment of grammar into sync with current thinking and practice in applied linguistics and language pedagogy.

The author of this book, Jim Purpura, has extensive experience not only in teaching and assessing grammar, but in training language teachers in grammar and assessment. In this book, he presents a new theoretical approach to defining grammatical ability that provides a basis for designing, developing and using assessments of grammar for a wide range of uses.

Introduction

In previous chapters, the contexts of grammar assessment were considered and the nature of grammatical ability defined for assessment purposes. Then, in Chapter 5, I discussed the notion of task and described how grammatical constructs could be operationalized to measure grammatical ability for different purposes. Building on the procedures for designing grammar-test tasks, this chapter addresses the process of grammar-test construction, that is the principles underlying the design, development and scoring of grammatical assessments. To do this, I will first describe the characteristics of ‘useful’ tests so that these qualities may be used to drive the test development process. I will discuss these qualities both generally and in relation to grammatical assessments. After that, I will detail the process used to construct tests of grammatical ability from the initial design phase through operational use. I will address the process of test construction principally, but not exclusively, from the perspective of large-scale assessment. Large-scale grammar assessment refers to tests that reach beyond the confines of the individual classroom. These include the grammar sections in placement or proficiency exams, the grammar sections of final exams given to several class sections in a school, or the grammar sections of measurement instruments used in research projects. The procedures for constructing large-scale assessments are somewhat different from those typically encountered in smallscale or classroom assessments, since they each have different goals and priorities.

Introduction

Research and theory related to the teaching and learning of grammar have made significant advances over the years. In applied linguistics, our understanding of language has been vastly broadened with the work of corpus-based and communication-based approaches to language study, and this research has made pathways into recent pedagogical grammars. Also, our conceptualization of language proficiency has shifted from an emphasis on linguistic form to one on communicative language ability and communicative language use, which has, in turn, led to a deemphasis on grammatical accuracy and a greater concern for communicative effectiveness. In language teaching, we moved from a predominant focus on structures and metalinguistic terminology to an emphasis on comprehensible input, interaction and no explicit grammar instruction. From there, we adopted a more balanced approach to language instruction, where meaning and communication are still emphasized, but where form and meaning-focused instruction have a clear role. Current research in grammar instruction involves investigations into the effect of teaching grammar explicitly or implicitly, reactively or proactively, and integrated in the curriculum at one point in time or sequentially (Doughty, 2002). Findings from SLA have also shown that the processing constraints underlying certain developmental orders (e.g., negation) cannot be contravened (Doughty, 2002) and that the optimal conditions for processing meaning, function and form in language learning are still to be discovered. All these developments have implications for how grammatical ability needs to be assessed and how assessments of grammatical ability might conceivably be used. However, theoretical discussions on the nature of grammatical ability, such as those initiated by Rea-Dickins (1991) and Larsen-Freeman (1991, 1997), have been few and far between, as have discussions of the construction of reliable and valid assessments.

Introduction

In this chapter I will examine several examples of professionally developed language tests that measure grammatical ability. Some of these tests contain separate sections that are exclusively devoted to the assessment of grammatical ability, while others measure grammatical knowledge along with other components of language ability in the context of language use – that is while test-takers are listening, speaking, reading or writing. The purpose of examining these tests is to illustrate how a few large-scale grammar tests have been designed and operationalized in light of their purpose(s), intended use(s) and the construct(s) they are trying to measure. The framework of grammatical knowledge presented in Chapter 4 provides a backdrop for considerations of construct definition. A second goal of this chapter is to examine these grammar tests in view of the qualities of test usefulness discussed in the previous chapter. Given limitations of space, I will comment briefly on all the qualities except practicality. Since the tests I have chosen to review are all operational, practicality might be considered a bottom-line function of how the other qualities were prioritized and resources allocated (Bachman, personal communication, 2002); therefore, I will focus this discussion on how the priorities of test usefulness appear to be prioritized. Finally, this chapter makes no attempt at providing a comprehensive review of all grammar tests available; nor does it attempt to provide model examples of how grammatical ability should be conceived and operationalized in language tests.

Introduction

As we saw in the last chapter, second and foreign language educators have looked to different schools of linguistics for insights about language. This has considerably broadened our notion of grammar and has led to a deeper understanding of the role that grammar plays in conveying meaning in communication. However, although linguistic analysis can tell us what the language system is and how it works, it still cannot tell us how second or foreign languages are best learned or what teaching practices most effectively promote L2 learning. With respect to learning to use grammar communicatively, several questions arise. First, do learners best learn an L2 naturalistically surrounded by other speakers of the target language, or does L2 learning require instruction? If it does, what type of instruction makes a difference? Does the explicit presentation of grammar rules lead to improved L2 communicative ability, or do rules just confuse students and raise their level of anxiety? And if instruction makes a difference, when should it be implemented? Does timing make a difference? Also, if instruction works for some and not for others, under what conditions is instruction most effective? What, in fact, are the cognitive underpinnings of grammar learning and how does this influence students' rate of learning, their ultimate level of achievement and the ease with which they acquire the grammar? Finally, the assessment question – how do we know that grammar learning has occurred?

Introduction

In the previous chapters, we saw that performance on grammar tests can be influenced by many other factors besides grammatical ability. In fact, test scores can vary as a result of the personal attributes of testtakers such as their age (Farhady, 1983; Zeidner, 1987), gender (Kunnan, 1990; Sunderland, 1995) and language background (Zeidner, 1986, 1987). They can also fluctuate due to their strategy use (Cohen, 1994; Purpura, 1999), motivation (Gardner, 1985) and level of anxiety (Gardner, Lalonde, Moorcroft and Evans, 1987). However, some of the most important factors that affect grammar-test scores, aside from grammatical ability, are the characteristics of the test itself. In fact, anyone who has ever taken a grammar test, or any test for that matter, knows that the types of questions on the test can severely impact performance. For example, some test-takers perform better on multiple-choice tasks than on oral interview tasks; others do better on essays than on cloze tasks; and still others score better if asked to write a letter than if asked to interpret a graph. Each of these tasks has a set of unique characteristics, called test-task characteristics. These characteristics can potentially interact with the characteristics of the examinee (e.g., his or her grammatical knowledge, personal attributes, topical knowledge, affective schemata) to influence test performance. Given the potential impact of test-task characteristics on performance, it is important for test developers to understand the individual characteristics of the tasks they use and to follow systematic procedures for designing and developing tasks that will elicit the best possible manifestations of grammatical ability.

Introduction

The study of grammar has had a long and important role in the history of second language and foreign language teaching. For centuries, to learn another language, or what I will refer to generically as an L2, meant to know the grammatical structures of that language and to cite prescriptions for its use. Grammar was used to mean the analysis of a language system, and the study of grammar was not just considered an essential feature of language learning, but was thought to be sufficient for learners to actually acquire another language (Rutherford, 1988). Grammar in and of itself was deemed to be worthy of study – to the extent that in the Middle Ages in Europe, it was thought to be the foundation of all knowledge and the gateway to sacred and secular understanding (Hillocks and Smith, 1991). Thus, the central role of grammar in language teaching remained relatively uncontested until the late twentieth century. Even a few decades ago, it would have been hard to imagine language instruction without immediately thinking of grammar.

While the central role of grammar in the language curriculum has remained unquestioned until recent times, grammar pedagogy has unsurprisingly been the source of much debate. For example, some language educators have argued that foreign languages are best learned deductively, where students are asked to memorize and recite definitions, rules, examples and exceptions.