Introduction

Up to this point, we have been considering writing assessment primarily in terms of collecting and evaluating individual writing samples as single performances, and making inferences from these performances about the writing ability of the test takers. However, as has been noted earlier, this approach to writing assessment is limited in several ways. Two of the most serious limitations are: (1) the fact that writing done under timed conditions on an unfamiliar topic does not accurately reflect the conditions under which most writing is done in non-testing situations or writing as it is taught and practiced in the classroom, and (2) the fact that it is difficult to generalize from a single writing sample to a much broader universe of writing in different genres and for different purposes and audiences. Chapter 8 deals to some extent with the first limitation, in that classroom evaluation of writing can be accomplished through untimed as well as timed writing, but a different approach is needed to deal with the second limitation. Portfolio assessment is seen by many as an alternative approach to writing assessment that can allow broader inferences about writing ability than are possible with single-shot approaches to evaluating writing, both in the individual classroom and on a larger scale.

Portfolios have long been a standard form of assessment in fields related to the visual arts such as architecture, design, and photography. In first-language writing as well, portfolios have a fairly long history, although they have only recently become used in large-scale assessment.

Introduction

In Chapters 5 and 6, issues in developing writing tasks and scoring procedures for large-scale assessment were discussed. This chapter presents examples from several tests of second-language writing as a way of illustrating how test developers have dealt with some of the issues raised in the previous two chapters for specific groups of test takers. These issues include:

• defining the writing construct for the purpose of the test;

• designing writing tasks from specific TLU domains so that they are likely to be authentic for prospective examinees;

• developing scoring criteria that reflect the most important aspects of writing for those test takers; and

• balancing the various aspects of test usefulness for a particular situation.

Five tests will be discussed in this chapter. Three of them are well-known language tests that include writing as a distinct component of the test: these are the TOEFL, the International English Language Testing System (IELTS), and the UCLES First Certificate in English (FCE). Scores on these tests are internationally recognized for such purposes as university admissions or certification of language proficiency for employment. The other two tests that will be discussed are tests of quite a different nature. The Basic English Skills Test (BEST), developed by the Center for Applied Linguistics in Washington, D.C., is a test intended for adult immigrant learners of English who are not necessarily literate in their first language and who need basic functional English skills for everyday life.

Introduction

This section of the book provides guidance for developing large-scale writing tests; that is, tests of writing beyond the level of the classroom. Chapter 4 presents an overview of research in what is often referred to as ‘direct’ writing assessment, while Chapters 5 and 6 build on the theoretical underpinnings of Chapter 4 to provide practical guidance in developing tasks for writing assessment (Chapter 5) and scoring procedures (Chapter 6).

An assessment task in which test takers actually produce a sample of writing, often referred to as a ‘direct’ test of writing, is probably the most common method for testing writing in both first-and second-language contexts. Hamp-Lyons (1991a: 5) gives five characteristics of a so-called ‘direct’ test of writing. Candidates must write at least one piece of continuous text (here Hamp-Lyons suggests a minimum number of 100 words, but this minimum presupposes a fairly high level of language proficiency and may not be appropriate for low-proficiency learners); test takers are given a set of instructions (or ‘prompt’) but have considerable leeway in responding to the prompt; each text is read by at least one, and normally two or more, trained raters; judgements are tied to a common yardstick, such as a set of sample responses or one or more rating scales; and judgements are expressed as numbers, rather than or in addition to verbal descriptions. Two additional characteristics of most such writing tests are the fact that texts are written in a limited time frame, generally between thirty minutes and two hours, and that the topic is unknown to test takers in advance.

The ability to write effectively is becoming increasingly important in our global community, and instruction in writing is thus assuming an increasing role in both second-and foreign-language education. As advances in transportation and technology allow people from nations and cultures throughout the world to interact with each other, communication across languages becomes ever more essential. As a result, the ability to speak and write a second language is becoming widely recognized as an important skill for educational, business, and personal reasons. Writing has also become more important as tenets of communicative language teaching – that is, teaching language as a system of communication rather than as an object of study – have taken hold in both second-and foreign-language settings. The traditional view in language classes that writing functions primarily to support and reinforce patterns of oral language use, grammar, and vocabulary, is being supplanted by the notion that writing in a second language is a worthwhile enterprise in and of itself.

Wherever the acquisition of a specific language skill is seen as important, it becomes equally important to test that skill, and writing is no exception. Thus, as the role of writing in second-language education increases, there is an ever greater demand for valid and reliable ways to test writing ability, both for classroom use and as a predictor of future professional or academic success.

What does it mean to test writing ability? A common-sense answer to this question is that “the best way to test people's writing ability is to get them to write” (Hughes, 1989: 75).

Introduction

The format for writing tests that has been emphasized up to this point – specifically, the timed impromptu test – has a solid research base and has become widely accepted as a means for assessing writing in many situations. Cohen (1994) notes that assessments can be classified under three main uses: administrative, instructional, and research. Most of the situations in which such tests have been researched have been administrative tests – i.e. placement, general proficiency assessments, and the like. Timed impromptu writing tests are also frequently used in research, for example, to collect samples of productive language use for studies of second-language development under a variety of conditions. However, for most instructional test uses (i.e. diagnosis, evidence of progress, feedback to students, and evaluation of teaching or curriculum) the timed impromptu test has several limitations as a test of writing ability. The next two chapters of the book deal with classroom writing assessment and show how, while the model of writing assessment that has been developed and researched for large-scale testing does have benefits for classroom assessment, it is not the most appropriate model for most classroom test uses. This chapter discusses the role of writing in the L2 classroom and goes on to discuss how assessment models for large-scale testing can be applied and adapted for classroom testing. Chapter 9 discusses an assessment model that goes beyond consideration of single texts to consideration of a collection of written texts that show a student's depth of knowledge about writing, flexibility in using strategies for different audiences and writing purposes, and progress over a period of time – that is, portfolio assessment.

This chapter presents principles of test design for large-scale writing assessment – that is, testing beyond the level of the individual classroom. The chapter is divided into two parts; the first part presents general considerations in test development, specifically the use and importance of test specifications and the importance of pre-testing; the second half of the chapter presents considerations for designing writing tasks, which take into account the research on assessment discussed in Chapter 4. Throughout the chapter, sample writing tasks are presented that illustrate the main points under discussion.

The process of test development

General procedures for language test development can be found in numerous sources (e.g. Bachman and Palmer, 1996; Alderson et al., 1995; Norris et al., 1998; Davidson and Lynch, 2002). These sources emphasize that test development involves several stages, which do not proceed in a strictly sequential or linear fashion, but inform each other in an iterative fashion; that is, the feedback received at one stage of the process may make it necessary to return to a previous step to rectify a problem. One way to conceptualize the development process for a test of writing is to consider it as consisting of three stages: design, operationalization, and administration (Bachman and Palmer, 1996). The design stage involves gathering information about such things as the test purpose, characteristics of the target population and their real-world writing needs, and available resources.

Introduction

When we look back at writing assessment in the 20th century, we see a number of phenomena. These include the rise of large testing firms and the concomitant rise in the popularity of so-called ‘objective tests’ that purported to measure writing ability through multiple-choice tests; the subsequent movement – led primarily by writing teachers – to measure writing through actual writing, and the resulting refinement and general acceptance of the impromptu essay test; and another teacher-led movement, this time towards portfolios, as a reaction to the ‘one-shot’ approach to essay testing. At the same time, rapid advances in technology and increased global communication have been leading to radical changes in the way writing is used and tested, changes we are only just now beginning to appreciate. Furthermore, there is a growing awareness among educators, theorists, and assessment specialists of the sociopolitical aspects of assessment. Thus, rather than viewing writing assessment as neutral, value-free activities, we now clearly understand that decisions about whom to test, what to test, how, when, and where to test it, and how test results are to be used take place in arenas where different groups of stakeholders have different agendas and value systems. As we progress through the 21st century, therefore, there are a number of areas in which we can expect to see changes and challenges in writing assessment. In this chapter I will focus on two of these areas.

Writing, which was once considered the domain of the elite and well-educated, has become an essential tool for people of all walks of life in today's global community. Whether used in reporting analyses of current events for newspapers or web pages, composing academic essays, business reports, letters, or e-mail messages, the ability to write effectively allows individuals from different cultures and backgrounds to communicate. Furthermore, it is now widely recognized that writing plays a vital role not only in conveying information, but also in transforming knowledge to create new knowledge. It is thus of central importance to students in academic and second language programs throughout the world. In many of these settings, the assessment of writing ability is of critical importance. Employers, academic instructors and writing teachers need to make decisions about potential employees and students, based on how well they can communicate in writing. But while the history of writing assessment goes back for centuries, it continues to be one of the most problematic areas of language use to assess. This is partly because of the vast diversity of writing purposes, styles, and genres, but primarily because of the subjectivity of the judgements involved in assessing samples of writing.

The author of this book, Dr. Sara Cushing Weigle, has extensive experience in teaching and assessing writing, and has conducted seminal research in this area. Her doctoral dissertation on writing assessment was awarded the TOEFL Award for Outstanding Doctoral Dissertation in Second/Foreign Language Testing in 1996, and she has since published numerous research studies in this area.

In Chapters 1 and 2, the nature of writing ability and the use of writing by different groups of language learners were considered. In this chapter, I will focus more specifically on issues related to the assessment of language in general, and writing in particular. As noted earlier, the models of the writing process presented in Chapter 2 were developed primarily with first-language writers in mind; thus, they are not concerned specifically with the development of language ability as expressed through writing, but assume a more or less stable language system and focus on the development of cognitive and meta-cognitive strategies that are involved in generating coherent texts for specific audiences and purposes. In this chapter I will discuss writing assessment from the point of view of language testing, which has traditionally concerned itself with defining what is meant by language ability in general, as an underlying cognitive ability and as manifested through the traditional skills of speaking, listening, reading, and writing. For the purposes of this chapter I will therefore be looking at writing tests as a specific type of language test, keeping in mind, however, that the degree to which a writing test is specifically measuring language as opposed to measuring other cognitive skills is not always clear-cut.

The framework for much of the discussion in this chapter comes from Bachman and Palmer's (1996) volume on language testing. The chapter is organized as follows. First, various test purposes are described.

Introduction

In Chapter 1, the role of writing in second-language learning was explored. In this chapter, we turn to a consideration of the nature of writing ability. Denning the skill that we want to test is a critical starting point in designing a test, and, as we shall see, the definition of writing ability for a particular context will depend in large measure on the considerations discussed in Chapter 1: that is, the specific group of second-language writers and the type of writing that these writers are likely to engage in.

This chapter looks at the nature of writing ability from several perspectives: first, in comparison with the other so-called productive skill of speaking, next as a social and cultural phenomenon, then as a cognitive activity. Finally, the relationship between writing and second-language proficiency is discussed.

The relationship between writing and speaking

It is traditional in language teaching and testing to categorize instances of language use into four skills: reading, writing, listening, and speaking, using channel (aural versus visual) and mode (productive versus receptive). The extent to which these different skills actually involve different cognitive mechanisms or are simply various socio-culturally mediated manifestations of a more general language ability is a matter of some controversy. While a full treatment of this issue is beyond the scope of this volume, in coming up with a definition of writing that can be useful for assessment it may be worthwhile spending some time considering the relationship between writing and the two skills most closely related to it: speaking (the other productive skill) and reading (the other visual skill).