Hostname: page-component-7bb8b95d7b-fmk2r Total loading time: 0 Render date: 2024-10-07T01:17:55.575Z Has data issue: false hasContentIssue false

Identifying off-topic student essays without topic-specific training data

Published online by Cambridge University Press:  22 May 2006

D. HIGGINS
Affiliation:
Educational Testing Service, Rosedale Road, Princeton, NJ 08541, USA e-mail: dhiggins@ets.org
J. BURSTEIN
Affiliation:
Educational Testing Service, Rosedale Road, Princeton, NJ 08541, USA e-mail: dhiggins@ets.org
Y. ATTALI
Affiliation:
Educational Testing Service, Rosedale Road, Princeton, NJ 08541, USA e-mail: dhiggins@ets.org

Abstract

Educational assessment applications, as well as other natural-language interfaces, need some mechanism for validating user responses. If the input provided to the system is infelicitous or uncooperative, the proper response may be to simply reject it, to route it to a bin for special processing, or to ask the user to modify the input. If problematic user input is instead handled as if it were the system's normal input, this may degrade users' confidence in the software, or suggest ways in which they might try to “game” the system. Our specific task in this domain is the identification of student essays which are “off-topic”, or not written to the test question topic. Identification of off-topic essays is of great importance for the commercial essay evaluation system CriterionSM. The previous methods used for this task required 200–300 human scored essays for training purposes. However, there are situations in which no essays are available for training, such as when users (teachers) wish to spontaneously write a new topic for their students. For these kinds of cases, we need a system that works reliably without training data. This paper describes an algorithm that detects when a student's essay is off-topic without requiring a set of topic-specific essays for training. This new system is comparable in performance to previous models which require topic-specific essays for training, and provides more detailed information about the way in which an essay diverges from the requested essay topic.

Type
Papers
Copyright
2006 Cambridge University Press

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

Footnotes

The authors would like to thank Chi Lu and Slava Andreyev for their help in carrying out the experiments described in this paper.