2 results
Developing, evaluating, and refining an automatic generator of diagnostic multiple choice cloze questions to assess children's comprehension while reading*
- JACK MOSTOW, YI-TING HUANG, HYEJU JANG, ANDERS WEINSTEIN, JOE VALERI, DONNA GATES
-
- Journal:
- Natural Language Engineering / Volume 23 / Issue 2 / March 2017
- Published online by Cambridge University Press:
- 14 April 2016, pp. 245-294
-
- Article
- Export citation
-
We describe the development, pilot-testing, refinement, and four evaluations of Diagnostic Question Generator (DQGen), which automatically generates multiple choice cloze (fill-in-the-blank) questions to test children's comprehension while reading a given text. Unlike previous methods, DQGen tests comprehension not only of an individual sentence but of the context preceding it. To test different aspects of comprehension, DQGen generates three types of distractors: ungrammatical distractors test syntax; nonsensical distractors test semantics; and locally plausible distractors test inter-sentential processing.
(1) A pilot study of DQGen 2012 evaluated its overall questions and individual distractors, guiding its refinement into DQGen 2014.
(2) Twenty-four elementary students generated 200 responses to multiple choice cloze questions that DQGen 2014 generated from forty-eight stories. In 130 of the responses, the child chose the correct answer. We define the distractiveness of a distractor as the frequency with which students choose it over the correct answer. The incorrect responses were consistent with expected distractiveness: twenty-seven were plausible, twenty-two were nonsensical, fourteen were ungrammatical, and seven were null.
(3) To compare DQGen 2014 against DQGen 2012, five human judges categorized candidate choices without knowing their intended type or whether they were the correct answer or a distractor generated by DQGen 2012 or DQGen 2014. The percentage of distractors categorized as their intended type was significantly higher for DQGen 2014.
(4) We evaluated DQGen 2014 against human performance based on 1,486 similarly blind categorizations by twenty-seven judges of sixteen correct answers, forty-eight distractors generated by DQGen 2014, and 504 distractors authored by twenty-one humans. Surprisingly, DQGen 2014 did significantly better than humans at generating ungrammatical distractors and marginally better than humans at generating nonsensical distractors, albeit slightly worse at generating plausible distractors. Moreover, vetting DQGen 2014's output and writing distractors only when necessary would halve the time to write them all, and produce higher quality distractors.
Contributors
-
- By Nicholas B. Allen, Stephanie Assuras, Robert M. Bilder, Joan C. Borod, John L. Bradshaw, Warrick J. Brewer, Ariel Brown, Nik Brown, Tyrone Cannon, Audrey Carstensen, Cameron S. Carter, Luke Clark, Phyllis Chua, Thilo Deckersbach, Richard A. Depue, Tali Ditman, Aleksey Dumer, David E. Fleck, Lara Foland-Ross, Judith M. Ford, Nelson Freimer, Paolo Fusar-Poli, Nathan A. Gates, Terry E. Goldberg, George Graham, Igor Grant, Melissa J. Green, Michelle M. Halfacre, Wendy Heller, John D. Herrington, Garry D. Honey, Jennifer E. Iudicello, Henry J. Jackson, J. David Jentsch, Donald Kalar, Paul Keedwell, Ester Klimkeit, Nancy S. Koven, Donna A. Kreher, Gina R. Kuperberg, Edythe London, Dan I. Lubman, Daniel H. Mathalon, Patrick D. McGorry, Philip McGuire, George R. Mangun, Gregory A. Miller, Albert Newen, Jack B. Nitschke, Jaak Panksepp, Christos Pantelis, Mary Philips, Russell A. Poldrack, Scott L. Rauch, Susan M. Ravizza, Steven Paul Reise, Nicole Rinehart, Angela Rizk-Jackson, Trevor W. Robbins, Tamara A. Russell, Fred W. Sabb, Cary R. Savage, Kimberley R. Savage, J. Cobb Scott, Marc L. Seal, Larry J. Seidman, Paula K. Shear, Marisa M. Silveri, Nadia Solowij, Laura Southgate, G. Lynn Stephens, D. Stott Parker, Stephen M. Strakowski, Simon A. Surguladze, Kate Tchanturia, René Testa, Janet Treasure, Eve M. Valera, Kai Vogeley, Anthony P. Weiss, Sarah Whittle, Stephen J. Wood, Steven Paul Woods, Murat Yücel, Deborah A. Yurgelun-Todd
- Edited by Stephen J. Wood, University of Melbourne, Nicholas B. Allen, University of Melbourne, Christos Pantelis, University of Melbourne
-
- Book:
- The Neuropsychology of Mental Illness
- Published online:
- 10 May 2010
- Print publication:
- 01 October 2009, pp xv-xx
-
- Chapter
- Export citation