Sample report: Conclusions

Criterion-referenced tests are difficult and time consuming to develop, administer, score, analyze, and revise. Such a project involves a immense amount of work, but a dedicated and resolute group of teachers with good leadership, can create such tests, even in multiple forms like we did. However, the benefits can also be enormous. For instance, the information that we gained from administering our CRTs as diagnostic pretests and achievement posttest was directly and immediately applicable to what we are doing on a daily basis in our ESL classes and helped us improve all of the following components of our curriculum: students’ needs, goals and objectives, the tests themselves, materials development, teacher support, and program evaluation (for more on these curriculum components, see Brown, 1995).

First, the Criterion-referenced tests helped us to examine our views of the students’ needs. Using our CRT results, we were able to examine how the students performed at the beginning and end of our reading courses on each and every objective, as well as the degree to which those objectives were defined in clear, observable, and defensible terms. Some of our initial perceptions turned out to be completely wrong, and, as mentioned above, the tests had to be completely overhauled. However, that outcome was better than blithely persisting in teaching the students material that they did not need to learn. Hence, our knowledge of the students’ needs was considerably improved by the implementation and analysis of our CRTs.

Second, learning which goals and objectives were and were not functioning well made it possible for us to focus on those objectives that best reflected the students’ needs while adding new objectives designed to meet more advanced needs. The new objectives helped us avoid wasting time and energy teaching material the students already knew. We could instead concentrate on material they truly needed with a great deal more efficiency. Perhaps we were falling prey to what Tumposky (1984) cynically called "the cult of efficiency." Frankly, we do not see anything wrong with trying to deliver ESL instruction in a relatively efficient way. After all, the students pay good money for our services. In short, our objectives, newly honed by our CRT results, will simply help us to foster as much language learning as possible during the shortest period of time.

Third, and most obvious, implementing, analyzing, and revising the Criterion-referenced helped us to improve the tests themselves. But there is much more to it than that. The process of doing Criterion-referenced testing afforded us the luxury of working together as groups of teachers to create, administer, score, analyze, and revise classroom tests that were probably much more effective than any single teacher could have done in isolation. In addition, the information learned from developing Criterion-referenced tests can be used to improve other types of tests. For example, information learned from CRT achievement test development might be useful for revising our placement procedures in order to overcome any mismatch that develops between placement test decisions and the curricula of the courses into which the students are being placed (as demonstrated in Brown, 1981). An example of how Criterion-referenced testing practices can be used to modify placement procedures so that they are more closely aligned with courses is explained in Brown (1989a).

Fourth, developing CRTs and changing our objectives based our diagnostic and achievement test results naturally lead us to rethinking our materials development. Naturally, our goal in rethinking our materials was to better match our new perceptions of what the students needed. In some cases, the materials have changed completely, but more often, the materials, teaching techniques, and practice exercises have been incrementally modified. In general, the CRT development process helped us better gauge the correct levels and objectives for the textbooks that we ended up adopting, or for the materials modules that we developed to fill gaps in the textbooks.

Fifth, the primary purpose of all of our curriculum development activities was teacher support. Basically, we just wanted to free teachers of onerous tasks so they could focus on what they do best—teaching. To that end, our Criterion-referenced tests help teachers do rational and well-designed achievement testing (which is required in all of UHM’s undergraduate courses). Teachers are welcome to add their own sections to the final examination, but the core test is essentially provided for them, by having them work together on it. Because these CRTs are so important to the teachers and their students, the teachers most directly involved review and revise them in groups. As such, the CRTs serve as an important focal point for discussion, while the teachers are working constructively together toward a common teaching/learning goal.

The sixth and final benefit gained from our CRT development involves program evaluation. Program evaluation can be viewed as being formative or summative. Taking the formative view, our CRTs have clearly helped us to revise and fine tune our curriculum as it has continued to evolve. However, if called upon to perform a summative program evaluation by some outside agency, the CRTs would also put us in a very powerful position to defend our program because we would have a staggering amount of information ready to furnish: (a) norm-referenced information about the overall proficiency of our students in terms of their TOEFL scores for admissions, (b) norm-referenced information about their placement based on the six subtests of the ELI Placement Test, (c) Criterion-referenced information on the students’ knowledge at the beginning (diagnostic) and end (achievement) of each course, and (d) Criterion-referenced information about what and how much the students have learned in our courses. In short, our CRTs will clearly put us in a strong position to write a summary report that describes our program in terms of students’ needs, goals and objectives, materials development, teacher support, and of course, program evaluation. More importantly, our CRTs put us in a strong position for suggesting positive program changes as part of our on-going curriculum development processes.

We sincerely hope that most of our students are served well by the procedures discussed in this report. We especially hope that those students are correctly admitted, placed, diagnosed, and promoted during their time in the ELI at UHM. We recognize that these decisions are made by human beings and that, even using apparently objective test scores, human beings do not always make correct judgments. We further recognize that incorrect decisions may cost students a great deal in terms of money (e.g., extra tuition paid) or time (e.g., extra, unnecessary time spent in ESL classes). As a result, we take our decisions about our students’ lives very seriously and base them on the best available information. One important source of that information comes from our Criterion-referenced tests.

References

[The paper references are integrated with the book references]