Skip to main content
    • Aa
    • Aa

A scaffolding approach to coreference resolution integrating statistical and rule-based models


We describe a scaffolding approach to the task of coreference resolution that incrementally combines statistical classifiers, each designed for a particular mention type, with rule-based models (for sub-tasks well-matched to determinism). We motivate our design by an oracle-based analysis of errors in a rule-based coreference resolution system, showing that rule-based approaches are poorly suited to tasks that require a large lexical feature space, such as resolving pronominal and common-noun mentions. Our approach combines many advantages: it incrementally builds clusters integrating joint information about entities, uses rules for deterministic phenomena, and integrates rich lexical, syntactic, and semantic features with random forest classifiers well-suited to modeling the complex feature interactions that are known to characterize the coreference task. We demonstrate that all these decisions are important. The resulting system achieves 63.2 F1 on the CoNLL-2012 shared task dataset, outperforming the rule-based starting point by over seven F1 points. Similarly, our system outperforms an equivalent sieve-based approach that relies on logistic regression classifiers instead of random forests by over four F1 points. Lastly, we show that by changing the coreference resolution system from relying on constituent-based syntax to using dependency syntax, which can be generated in linear time, we achieve a runtime speedup of 550 per cent without considerable loss of accuracy.

Hide All
Bagga A., and Baldwin B., 1998. Algorithms for scoring coreference chains. In Proceedings of the LREC 1998 Workshop on Linguistic Coreference, Granada, Spain, pp. 563–6.
BBN Technologies 2006. Coreference Guidelines for English OntoNotes – Version 6.0.
Bengtson E., and Roth D., 2008. Understanding the value of features for coreference resolution. In Proceedings of EMNLP 2008, Honolulu, Hawaii, pp. 294303.
Björkelund A., and Kuhn J., 2014. Learning structured perceptrons for coreference resolution with latent antecedents and non-local features. In Proceedings of ACL 2014, Baltimore, Maryland, pp. 4757.
Boyd A., Gegg-Harrison W., and Byron D., 2005. Identifying non-referential it: a machine learning approach incorporating linguistically motivated features. In Proceedings of the ACL Workshop on Feature Engineering for Machine Learning in NLP, Ann Arbor, Michigan, pp. 40–7.
Breiman L., 2001. Random forests. Machine Learning 45 (1): 532.
Burges C. J., 1998. A tutorial on support vector machines for pattern recognition. Data Mining and Knowledge Discovery 2: 121–67.
Cer D. M., Marneffe M.-C. D., Jurafsky D., and Manning C. D., 2010. Parsing to Stanford dependencies: trade-offs between speed and accuracy. In Proceedings of LREC 2010, Valletta, Malta, pp. 1628–32.
Chen C., and Ng V., 2012. Combining the best of two worlds: a hybrid approach to multilingual coreference resolution. In Proceedings of EMNLP-CoNLL 2012, Jeju Island, Korea, pp. 5663.
Chen D., and Manning C. D., 2014. A fast and accurate dependency parser using neural networks. In Proceedings of EMNLP 2014, Doha, Qatar, pp. 740–50.
Clark K., and Manning C. D., 2015. Entity-centric coreference resolution with model stacking. In Proceedings of ACL, Beijing, China, pp. 1405–15.
Clark K., and Manning C. D., 2016. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, pp. 2256–62.
Collins M. 1999. Head-Driven Statistical Models for Natural Language Parsing. PhD Thesis, University of Pennsylvania, Philadelphia, PA.
Connolly D., Burger J. D., and Day D. S., 1994. A machine learning approach to anaphoric reference. In Proceedings of the International Conference on New Methods in Language Processing (NeMLaP-1), UMIST, Manchester, pp. 255–61.
Daumé III H., and Marcu D., 2005. A large-scale exploration of effective global features for a joint entity detection and tracking model. In HLT-EMNLP 2005, Vancouver, B.C. Canada, pp. 97104.
de Marneffe M.-C., and Manning C. D., 2008. The Stanford typed dependencies representation. In Proceedings of COLING Workshop on Cross-framework and Cross-domain Parser Evaluation, Manchester, UK, pp. 18.
Denis P., and Baldridge J., 2007. Joint determination of anaphoricity and coreference resolution using integer programming. In Proceedings of NAACL-HLT 2007, Rochester, NY, pp. 236–43.
Denis P., and Baldridge J., 2008. Specialized models and ranking for coreference resolution. In Proceedings of EMNLP 2008, Honolulu, HI, pp. 660–9.
Durrett G., and Klein D. 2013. Easy victories and uphill battles in coreference resolution. In Proceedings of EMNLP-2013, Seattle, Washington.
Durrett G., and Klein D., 2014. A joint model for entity analysis: coreference, typing, and linking. TACL 2: 477–90.
Fernandes E. R., dos Santos C. N., and Milidiú R. L., 2012. Latent structure perceptron with feature induction for unrestricted coreference resolution. In EMNLP-CoNLL, Jeju, Republic of Korea, pp. 41–8.
Gabbard R., Freedman M., and Weischedel R., 2011. Coreference for learning to extract relations: yes Virginia, coreference matters. In ACL 2011, Portland, Oregon, pp. 288–93.
Gilbert N., and Riloff E., 2013. Domain-specific coreference resolution with lexicalized features. In Proceedings of ACL 2013, Sofia, Bulgaria, pp. 81–6.
Haghighi A., and Klein D., 2009. Simple coreference resolution with rich syntactic and semantic features. In Proceedings of EMNLP 2009, Suntec, Singapore, pp. 1152–61.
Haghighi A., and Klein D., 2010. Coreference resolution in a modular, entity-centered model. In Proceedings of HLT-NAACL 2010, Los Angeles, CA, pp. 385–93.
Hajishirzi H., Zilles L., Weld D. S., and Zettlemoyer L. 2013. Joint coreference resolution and named-entity linking with multi-pass sieves. In Proceedings of EMNLP 2013, Seattle, Washington.
Hobbs J. R., 1978. Resolving pronoun references. Lingua 44 (4): 311–38.
Jindal P., and Roth D., 2013. Using domain knowledge and domain-inspired discourse model for coreference resolution for clinical narratives. Journal of the American Medical Informatics Association (JAMIA) 20 (2): 356–62.
Kehler A., 1997. Probabilistic coreference in information extraction. In Proceedings of EMNLP 1997, Providence, Rhode Island, pp. 163–73.
Kilicoglu H., Fiszman M., and Demner-Fushman D., 2013. Interpreting consumer health questions: the role of anaphora and ellipsis. In Proceedings of the 2013 Workshop on Biomedical Natural Language Processing, Sofia, Bulgaria, pp. 5462.
Klein D., and Manning C. D., 2003. Accurate unlexicalized parsing. In Proceedings of ACL 2003, Sapporo, Japan, pp. 423–30.
Kummerfeld J. K., and Klein D., 2013. Error-driven analysis of challenges in coreference resolution. In Proceedings of EMNLP 2013, Seattle, Washington, pp. 265–77.
Lappin S., and Leass H. J., 1994. An algorithm for pronominal anaphora resolution. Computational Linguistics 20 (4): 535–61.
Lee H., Chang A., Peirsman Y., Chambers N., Surdeanu M., and Jurafsky D., 2013. Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics 39 (4): 885916.
Lee H., Peirsman Y., Chang A., Chambers N., Surdeanu M., and Jurafsky D., 2011. Stanford’s multi-pass sieve coreference resolution system at the CoNLL-2011 shared task. In Proceedings of CoNLL 2011: Shared Task, Portland, Oregon, pp. 2834.
Levy R., and Andrew G., 2006. Tregex and tsurgeon: tools for querying and manipulating tree data structures. In Proceedings of LREC 2006, Genoa, Italy, pp. 2231–4.
Luo X., 2005. On coreference resolution performance metrics. In Proceedings of HLT-EMNLP 2005, Vancouver, B.C., Canada, pp. 2532.
Luo X., Ittycheriah A., Jing H., Kambhatla N., and Roukos S. 2004. A mention-synchronous coreference resolution algorithm based on the Bell tree. In Proceedings of ACL 2004, Barcelona, pp. 21–6.
Manning C. D., Surdeanu M., Bauer J., Finkel J., Bethard S. J., and McClosky D., 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of ACL 2014, Baltimore, Maryland, pp. 5560.
Mccallum A., and Wellner B. 2004. Conditional models of identity uncertainty with application to noun coreference. In Proceedings of NIPS 2004, Vancouver, British Columbia, Canada.
McCarthy J. F., and Lehnert W. G. 1995. Using decision trees for coreference resolution. In Proceedings of IJCAI 1995, Montréal, pp. 1050–5.
Mikolov T., Sutskever I., Chen K., Corrado G., and Dean J., 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems (NIPS’13), Lake Tahoe, Nevada, USA, pp. 3111–9.
Mitkov R., 2002. Anaphora Resolution. London: Longman.
Mitkov R., Evans R., Orăsan C., Dornescu I., and Rios M. 2012. Coreference resolution: To what extent does it help nlp applications? In International Conference on Text, Speech and Dialogue, Berlin, Heidelberg: Springer, pp. 1627.
Mitkov R., Evans R., Orasan C., Ha L. A., and Pekar V. 2007. Anaphora resolution: to what extent does it help NLP applications? In Branco A. (ed.), Proceedings of DAARC 2007, LNAI, vol. 4410, pp. 179–90. Berlin/Heidelberg: Springer-Verlag.
Müller C., 2006. Automatic detection of nonreferential it in spoken multi-party dialog. In Proceedings of EACL 2006, Trento, Italy, pp. 4956.
Ng V., 2010. Supervised noun phrase coreference research: the first fifteen years. In Proceedings of ACL, Uppsala, Sweden, pp. 1396–411.
Ng V., and Cardie C. 2002. Improving machine learning approaches to coreference resolution. In Proceedings of ACL 2002, Philadelphia, pp. 104–11.
Petrov S., and Klein D., 2007. Improved inference for unlexicalized parsing. In Proceedings of HLT-NAACL 2007, Rochester, New York, pp. 404–11.
Pradhan S., Moschitti A., Xue N., Uryupina O., and Zhang Y. 2012. CoNLL-2012 shared task: modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of the 16th Conference on Computational Natural Language Learning (CoNLL 2012), Jeju, Korea.
Pradhan S. S., Hovy E., Marcus M., Palmer M., Ramshaw L., and Weischedel R., 2007. OntoNotes: A unified relational semantic representation. In Proceedings of ICSC, Irvine, CA, USA, pp. 517–26.
Raghunathan K., Lee H., Rangarajan S., Chambers N., Surdeanu M., Jurafsky D., and Manning C., 2010. A multi-pass sieve for coreference resolution. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (EMNLP), Cambridge, Massachusetts, pp. 492501.
Rahman A., and Ng V., 2009. Supervised models for coreference resolution. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing (EMNLP), Suntec, Singapore, pp. 968–77.
Ratinov L., and Roth D., 2012. Learning-based multi-sieve co-reference resolution with knowledge. In Proceedings of EMNLP-CoNLL 2012, Jeju Island, Korea, pp. 1234–44.
Recasens M., Can M., and Jurafsky D., 2013. Same referent, different words: unsupervised mining of opaque coreferent mentions. In Proceedings of NAACL 2013, Atlanta, Georgia, pp. 897906.
Roark B., and Hollingshead K. 2008. Classifying chart cells for quadratic complexity context-free inference. In Proceedings of the 22nd International Conference on Computational Linguistics (COLING), Manchester, United Kingdom.
Sculley D., Holt G., Golovin D., Davydov E., Phillips T., Ebner D., Chaudhary V., and Young M. 2014. Machine learning: The high interest credit card of technical debt. In SE4ML: Software Engineering for Machine Learning (NIPS 2014 Workshop), Montreal, Canada.
Soon W. M., Ng H. T., and Lim D. C. Y., 2001. A machine learning approach to coreference resolution of noun phrases. Computational Linguistics 27 (4): 521–44.
Steinberger J., Poesio M., Kabadjov M. A., and Jezek K., 2007. Two uses of anaphora resolution in summarization. Information Processing and Management 43 (6): 1663–80.
Stoyanov V., Gilbert N., Cardie C., and Riloff E., 2009. Conundrums in noun phrase coreference resolution: Making sense of the state-of-the-art. In Proceedings of ACL-IJCNLP 2009, Suntec, Singapore, pp. 656–64.
Stuckardt R. 2002. Machine-learning-based vs. manually designed approaches to anaphor resolution: the best of two worlds. In Proceedings of the 4th Discourse Anaphora and Anaphor Resolution Colloquium (DAARC2002), University of Lisbon, pp. 211–6.
Stuckardt R. 2005. A machine learning approach to preference strategies for anaphor resolution. In Branco A., McEnery A., and Mitkov R. (eds.), Anaphora Processing: Linguistic, Cognitive and Computational Modeling, pp. 4772. John Benjamins, Amsterdam/Philadelphia.
Surdeanu M., Hicks T., and Valenzuela-Escárcega M. A. 2015. Two practical rhetorical structure theory parsers. In Proceedings of NAACL-HLT 2015, Denver, Colorado, USA.
Vilain M., Burger J., Aberdeen J., Connolly D., and Hirschman L., 1995. A model-theoretic coreference scoring scheme. In Proceedings of MUC-6, Columbia, Maryland, pp. 4552.
Wiseman S., Rush A. M., and Shieber S. M., 2016. Learning global features for coreference resolution. In Proceedings of NAACL 2016, San Diego, CA, pp. 9941004.
Wiseman S., Rush A. M., Shieber S. M., and Weston J., 2015. Learning anaphoricity and antecedent ranking features for coreference resolution. In Proceedings of ACL-IJCNLP 2015, Beijing, China, pp. 1416–26.
Yang X., Su J., Lang J., Tan C. L., Liu T., and Li S., 2008. An entity-mention model for coreference resolution with inductive logic programming. In Proceedings of ACL-HLT 2008, Columbus, Ohio, pp. 843–51.
Yi Y., Lai C.-Y., Petrov S., and Keutzer K., 2011. Efficient parallel CKY parsing on GPUs. In Proceedings of the 2011 Conference on Parsing Technologies, Dublin, Ireland, pp. 175–85.
Yuan B., Chen Q., Xiang Y., Wang X., Ge L., Liu Z., Liao M., and Si X., 2012. A mixed deterministic model for coreference resolution. In Proceedings of EMNLP-CoNLL 2012, Jeju, Republic of Korea, pp. 7682.
Zhang X., Wu C., and Zhao H., 2012. Chinese coreference resolution via ordered filtering. In Proceedings of EMNLP-CoNLL 2012, CoNLL’12, Jeju, Republic of Korea, pp. 95–9.
Zhou G., and Su J., 2004. A high-performance coreference resolution system using a constraint-based multi-agent strategy. In Proceedings of COLING 2004, Geneva, Switzerland, pp. 522–9.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Natural Language Engineering
  • ISSN: 1351-3249
  • EISSN: 1469-8110
  • URL: /core/journals/natural-language-engineering
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Full text views

Total number of HTML views: 13
Total number of PDF views: 80 *
Loading metrics...

Abstract views

Total abstract views: 621 *
Loading metrics...

* Views captured on Cambridge Core between 21st March 2017 - 24th October 2017. This data will be updated every 24 hours.