Skip to main content
×
Home

Construction and evaluation of event graphs

  • GORAN GLAVAŠ (a1) and JAN ŠNAJDER (a1)
Abstract
Abstract

Events play an important role in natural language processing and information retrieval due to numerous event-oriented texts and information needs. Many natural language processing and information retrieval applications could benefit from a structured event-oriented document representation. In this paper, we propose event graphs as a novel way of structuring event-based information from text. Nodes in event graphs represent the individual mentions of events, whereas edges represent the temporal and coreference relations between mentions. Contrary to previous natural language processing research, which has mainly focused on individual event extraction tasks, we describe a complete end-to-end system for event graph extraction from text. Our system is a three-stage pipeline that performs anchor extraction, argument extraction, and relation extraction (temporal relation extraction and event coreference resolution), each at a performance level comparable with the state of the art. We present EvExtra, a large newspaper corpus annotated with event mentions and event graphs, on which we train and evaluate our models. To measure the overall quality of the constructed event graphs, we propose two metrics based on the tensor product between automatically and manually constructed graphs. Finally, we evaluate the overall quality of event graphs with the proposed evaluation metrics and perform a headroom analysis of the system.

Copyright
References
Hide All
ACE. 2005. Evaluation of the Detection and Recognition of ACE: Entities, Values, Temporal Expressions, Relations, and Events. Gaithersburg, MD: NIST.
Agirre E., and Soroa A. 2009. Personalizing PageRank for word sense disambiguation. In Proceedings of the 12th Conference of the European Chapter of the Association for Computational Linguistics (EACL ’09). Athens, Greece. Stroudsburg, PA: ACL, pp. 3341.
Ahn D. 2006. The stages of event extraction. In Proceedings of COLING/ACL 2006 Workshop on Annotating and Reasoning about Time and Events, Sydney, Australia. Stroudsburg, PA: ACL, pp. 18.
Ahn D., Schockaert S., De Cock M., and Kerre E. 2006. Supporting temporal question answering: strategies for offline data collection. In Proceedings of the 5th International Workshop on Inference in Computational Semantics. Buxton, UK: ACL, pp. 127–32.
Allan J. 2002. Topic Detection and Tracking: Event-Based Information Organization, vol. 12. Dordrecht, Netherlands: Kluwer.
Allen J. R., 1983. Maintaining knowledge about temporal intervals. Communications of the ACM 26 (11): 832–43.
Aone C., and Ramos-Santacruz M. 2000. REES: a large-scale relation and event extraction system. In Proceedings of the Sixth Conference on Applied Natural Language Processing. Seattle, WA. Stroudsburg, PA: ACL, pp. 7683.
Bagga A., and Baldwin B. 1999. Cross-document event coreference: annotations, experiments, and observations. In Proceedings of the Workshop on Coreference and its Applications. Stroudsburg, Pennsylvania. Stroudsburg, PA: ACL, pp. 18.
Baker C. F., Fillmore C. J., and Lowe J. B., 1998. The Berkeley framenet project. In Proceedings of the 17th International Conference on Computational linguistics (COLING ’98), Montreal, Canada. Stroudsburg, PA: ACL, pp. 8690.
Bejan C., and Harabagiu S. 2008. A linguistic resource for discovering event structures and resolving event coreference. In Proceedings of the 6th International Conference on Language Resources and Evaluation (LREC 2008), Marrakech, Morocco. Paris, FranceELRA.
Bejan C., and Harabagiu S., 2010. Unsupervised event coreference resolution with rich linguistic features. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics (ACL 2010), Uppsala, Sweden. Stroudsburg, PA: ACL, pp. 1412–22.
Bejan C. A., and Hathaway C., 2007. UTD-SRL: a pipeline architecture for extracting frame semantic structures. In Proceedings of the Fourth International Workshop on Semantic Evaluations (SemEval 2007), Prague, Czech Republic. Stroudsburg, PA: ACL, pp. 460–63.
Bethard S. 2008. Finding Event, Temporal and Causal Structure in Text: A Machine Learning Approach. PhD thesis. University of Colorado at Boulder, USA.
Bethard S. 2013. ClearTK-TimeML: a minimalist approach to TempEval 2013. Second Joint Conference on Lexical and Computational Semantics (*SEM), vol. 2, Atlanta, GA. Stroudsburg, PA: ACL, pp. 1014.
Boguraev B., and Ando R. K. 2005. TimeBank-driven TimeML analysis. In Annotating, Extracting and Reasoning about Time and Events, Dagstuhl Seminar Proceedings, Dagstuhl, Germany.
Borgwardt K. M. 2007. Graph Kernels. PhD thesis, Ludwig-Maximilians-Universität München, Munich, Germany.
Bramsen P., Deshpande P., Lee Y. K., and Barzilay R., 2006. Inducing temporal graphs. In Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing (EMNLP ’06), Sydney, Australia. Stroudsburg, PA: ACL, pp. 189–98.
Chambers N., and Jurafsky D. 2008. Unsupervised learning of narrative event chains. In Proceedings of 46th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT ’08), pp. 789–97.
Chambers N., and Jurafsky D., 2009. Unsupervised learning of narrative schemas and their participants. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP vol. 2, Singapore. Stroudsburg, PA: ACL, pp. 602–10.
Chang C. C., and Lin C. J. 2011. LibSVM: a library for support vector machines. ACM Transactions on Intelligent Systems and Technology 2 (27): 127. Software available at http://www.csie.ntu.edu.tw/cjlin/libsvm.
Chang A. X., and Manning C. D., 2012. SUTIME: A Library for Recognizing and Normalizing Time Expressions. Istanbul, Turkey: ELRA.
Chen B., Su J., Pan S., and Tan C. 2011. A unified event coreference resolution by integrating multiple resolvers. In Proceedings of 5th International Joint Conference on Natural Language Processing (IJCNLP 2011), Chiang Mai, Thailand. New York, NY: Curran Associates.
Cordella L. P., Foggia P., Sansone C., and Vento M., 2004. A subgraph isomorphism algorithm for matching large graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence 26 (10): 1367–72.
Davidson D., 1967. The logical form of action sentences. Essays on Actions and Events 5: 105–48.
De Marneffe M. C., MacCartney B., and Manning C. D., 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of 5th International Conference on Language Resources and Evaluation (LREC 2006), Genoa, Italy, vol. 6. Paris, France: ELRA, pp. 449–54.
Derczynski L., and Gaizauskas R. 2013. Temporal signals help label temporal relations. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, Sofia, Bulgaria, vol. 78. Stroudsburg, PA: ACL.
Fan R. E., Chang K., Hsieh C. J., Wang X. R., and Lin C. J., 2008. LibLinear: a library for large linear classification. Journal of Machine Learning Research 9: 1871–4.
Fillmore C. J., 1976. Frame semantics and the nature of language*. Annals of the New York Academy of Sciences 280 (1): 2032.
Finkel J. R., Grenager T., and Manning C. D., 2005. Incorporating non-local information into information extraction systems by Gibbs sampling. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics (ACL ’05), Ann Arbor, MI. Stroudsburg, PA: ACL, pp. 363–70.
Gärtner T., Flach P., and Wrobel S. 2003. On graph kernels: hardness results and efficient alternatives. In Schölkopf B. and Warmuth M. K. (eds.), Learning Theory and Kernel Machines. Berlin, Germany: Springer-Verlag, pp. 129–43.
Gildea D., and Jurafsky D., 2002. Automatic labeling of semantic roles. Computational Linguistics 28 (3): 245–88.
Glavaš G., and Šnajder J. 2013. Recognizing identical events with graph kernels. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL 2013), Sofia, Bulgaria (accepted). Stroudsburg, PA: ACL.
Grishman R., and Sundheim B., 1996. Message understanding conference-6: a brief history. In Proceedings of International Conference on Computational Linguistics (COLING 1996), Copenhagen, Denmark, vol. 96. Dresden, Germany: ICCL, pp. 466–71.
Grover C., Tobin R., Alex B., and Byrne K., 2010. Edinburgh-LTG: TempEval-2 system description. In Proceedings of the 5th International Workshop on Semantic Evaluation (SemEval 2010), Uppsala, Sweden. Stroudsburg, PA: ACL, pp. 333–6.
Haghighi A., Toutanova K., and Manning C. D., 2005. A joint model for semantic role labeling. In Proceedings of the Ninth Conference on Computational Natural Language Learning, Ann Arbor, MI. Stroudsburg, PA: ACL, pp. 173–6.
Hammack R., Imrich W., and Klavžar S., 2011. Handbook of Product Graphs. Discrete Mathematics and Its Applications. Boca Raton, FL: CRC Press.
Hatzivassiloglou V., Gravano L., and Maganti A. 2000. An investigation of linguistic features and clustering algorithms for topical document clustering. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, NY: ACM, pp. 224–31.
Humphreys K., Gaizauskas R., and Azzam S., 1997. Event coreference for information extraction. In Proceedings of a Workshop on Operational Factors in Practical, Robust Anaphora Resolution for Unrestricted Texts, Madrid, Spain. Stroudsburg, PA: ACL, pp. 7581.
Humphreys K., Gaizauskas R., Azzam S., Huyck C., Mitchell B., Cunningham H., and Wilks Y. 1998. University of Sheffield: description of the LaSIE-II system as used for MUC-7. In Proceedings of the Seventh Message Understanding Conferences (MUC-7), San Diego, CA. Gaithersburg, MD: NIST.
Jans B., Bethard S., Vulić I., and Moens M. F., 2012. Skip n-grams and ranking functions for predicting script events. In Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics, Avignon, France. Stroudsburg, PA: ACL, pp. 336–44.
Jun P., and Min Y., 2012. Improved temporal relation classication using dependency parses and selective crowdsourced annotations. In Proceedings of the International Conference on Computational Linguistics (COLING 2012), Mumbai, India, pp. 2109–24.
Karttunen L., and Zaenen A. 2005. Veridicity. Annotating, Extracting and Reasoning about Time and Events, Dagstuhl Seminar Proceedings. Dagstuhl, Germany: IBFI.
Kingsbury P., and Palmer M., 2002. From Treebank to PropBank. In Proceedings of the 3rd International Conference on Language Resources and Evaluation (LREC ’02), Las Palmas, Spain. Paris, France: ELRA, pp. 1989–93.
Kolomiyets O., Bethard S., and Moens M., 2012. Extracting narrative timelines as temporal dependency structures. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (ACL 2012), Jeju Island, Korea. Stroudsburg, PA: ACL, pp. 8897.
Kumaran G., and Allan J. 2004. Text classification and named entities for new event detection. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. New York, NY: ACM, pp. 297304.
Lavie A., and Denkowski M. J., 2009. The METEOR metric for automatic evaluation of machine translation. Machine Translation 23 (2–3): 105–15.
Lee H., Peirsman Y., Chang A., Chambers N., Surdeanu M., and Jurafsky D., 2011. Stanford’s multi-pass sieve coreference resolution system at the CoNLL-2011 shared task. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task (CoNLL 2011), Portland, OR. Stroudsburg, PA: ACL, pp. 2834.
Lee H., Recasens M., Chang A., Surdeanu M., and Jurafsky D., 2012. Joint entity and event coreference resolution across documents. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP 2012), Jeju Island, Korea, Stroudsburg, PA: ACL, pp. 489500.
Llorens H., Saquete E., and Navarro B., 2010. TIPSem (English and Spanish): evaluating CRFs and semantic roles in TempEval-2. In Proceedings of the 5th International Workshop on Semantic Evaluation, Uppsala, Sweden. Stroudsburg, PA: ACL, pp. 284–91.
Llorens H., Saquete E., and Navarro-Colorado B., 2013. Applying semantic knowledge to the automatic processing of temporal expressions and events in natural language. Information Processing & Management 49 (1): 179–97.
Makkonen J., 2003. Investigations on event evolution in TDT. In Proceedings of the Student Research Workshop at Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology (NAACL-HLT ’03), Edmonton, Canada. Stroudsburg, PA: ACL, pp. 4348.
Makkonen J., Ahonen-Myka H., and Salmenkivi M., 2004. Simple semantics in topic detection and tracking. Information Retrieval 7 (3): 347–68.
McCallum A., and Li W., 2003. Early results for named entity recognition with conditional random fields, feature induction and web-enhanced lexicons. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, Sapporo, Japan, Stroudsburg, PA: ACL, pp. 188–91.
Melli G., Wang Y., Liu Y., Kashani M. M., Shi Z., Gu B., Sarkar A., and Popowich F. 2006. Description of SQUASH, the SFU question answering summary handler for the DUC-2005 summarization task. In Proceedings of the Document Understanding Conference, New York NY. Gaithersburg, MD: NIST.
Menchetti S., Costa F., and Frasconi P., 2005. Weighted decomposition kernels. In Proceedings of the 22nd International Conference on Machine Learning (ICML ’05), Bonn, Germany. New York, NY: ACM, pp. 585–92.
Moreda P., Llorens H., Saquete E., and Palomar M., 2011. Combining semantic information in question answering systems. Information Processing & Management 47 (6): 870–85.
Nadeau D., and Sekine S., 2007. A survey of named entity recognition and classification. Lingvisticae Investigationes 30 (1): 326.
Nallapati R., Feng A., Peng F., and Allan J. 2004. Event threading within news topics. In Proceedings of the 13th ACM International Conference on Information and Knowledge Management. New York, NY: ACM, pp. 446–53.
Navigli R., and Lapata M., 2007. Graph connectivity measures for unsupervised word sense disambiguation. In Proceedings of the 20th International Joint Conference on Artifical Intelligence, Hyderabad, India. Burlington, MA: Morgan Kaufmann, pp. 1683–8.
Palmer M., Gildea D., and Kingsbury P., 2005. The proposition bank: an annotated corpus of semantic roles. Computational Linguistics 31 (1): 71106.
Polanyi L., and Zaenen A., 2006. Contextual valence shifters. Computing Attitude and Affect in Text: Theory and Applications 20: 110.
Pradhan S., Ramshaw L., Marcus M., Palmer M., Weischedel R., and Xue N., 2011. CoNLL-2011 shared task: modeling unrestricted coreference in OntoNotes. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task (CoNLL 2011), Portland, OR. Stroudsburg, PA: ACL, pp. 127.
Pustejovsky J., 1991. The syntax of event structure. Cognition 41 (1): 4781.
Pustejovsky J., Castano J., Ingria R., Sauri R., Gaizauskas R., Setzer A., Katz G., and Radev D., 2003b. TimeML: robust specification of event and temporal expressions in text. New Directions in Question Answering 2003: 2834.
Pustejovsky J., Hanks P., Sauri R., See A., Gaizauskas R., Setzer A., Radev D., Sundheim B., Day D., and Ferro L., 2003a. The TimeBank corpus. In Proceedings of Corpus Linguistics, vol. 2003. Lancaster, UK: UCREL, pp. 40.
Quine W. 1985. Events and reification. In LePore E. and McLaughlin B. P. (eds.), Actions and Events: Perspectives on the Philosophy of Donald Davidson, Oxford UK: Blackwell, pp. 162–71.
Resnik P., 1999. Semantic similarity in a taxonomy: an information-based measure and its application to problems of ambiguity in natural language. Journal of Artificial Intelligence Research 11: 95130.
Rosen S., 1999. The syntactic representation of linguistic events. Glot International 4 (2): 311.
Salton G., Wong A., and Yang C., 1975. A vector space model for automatic indexing. Communications of the ACM 18 (11): 613–20.
Saquete E., González J. L. V., Martínez-Barco P., Munoz R., and Llorens H., 2009. Enhancing QA systems with complex temporal question processing capabilities. Journal of Artificial Intelligence Research 35 (2): 775.
Šarić F., Glavaš G., Karan M., Šnajder J., and Bašić B. D. 2012. TakeLab: systems for measuring semantic text similarity. In Proceedings of the 6th International Workshop on Semantic Evaluation (SemEval 2012) in Conjunction with the First Joint Conference on Lexical and Computational Semantics (*SEM 2012), Montreal, Canada. Stroudsburg, PA: ACL.
Saurí R., and Pustejovsky J., 2012. Are you sure that this happened? Assessing the factuality degree of events in text. Computational Linguistics 38 (2): 261–99.
Setzer A., Gaizauskas R., and Hepple M. 2003. Using semantic inferences for temporal annotation comparison. In Proceedings of the Fourth International Workshop on Inference in Computational Semantics (ICOS-4). Nancy, France: INRIA, pp. 25–6.
Smith C. S. 1999. Activities: states or events? Linguistics and Philosophy 22 (5): 479508.
Soon W., Ng H., and Lim D., 2001. A machine learning approach to coreference resolution of noun phrases. Computational Linguistics 27 (4): 521–44.
Sun W., Rumshisky A., and Uzuner O., 2013. Evaluating temporal relations in clinical text: 2012 i2b2 challenge. Journal of the American Medical Informatics Association 20 (5): 806–13.
Surdeanu M., Harabagiu S., Williams J., and Aarseth P., 2003. Using predicate-argument structures for information extraction. In Proceedings of the 41st Annual Meeting on Association for Computational Linguistics, Sapporo, Japan, vol. 1. Stroudsburg, PA: ACL, pp. 815.
Surdeanu M., and Turmo J., 2005. Semantic role labeling using complete syntactic analysis. In Proceedings of the Ninth Conference on Computational Natural Language Learning, Ann Arbor, MI. Stroudsburg, PA: ACL, pp. 221–4.
Tannier X., and Muller P., 2008. Evaluation metrics for automatic temporal annotation of texts. In Proceedings of the Conference on Language Resources and Evaluation, Marrakech, Morocco. Paris, France: ELRA, pp. 150–5.
Tenny C., and Pustejovsky J., 2000. A history of events in linguistic theory. Events as Grammatical Objects 32: 337.
Tong H., Faloutsos C., Gallagher B., and Eliassi-Rad T. 2007. Fast best-effort pattern matching in large attributed graphs. In Proceedings of the 13th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, NY: ACM, pp. 737–46.
UzZaman N., and Allen J., 2010. TRIPS and TRIOS system for TempEval-2: extracting temporal information from text. In Proceedings of the 5th International Workshop on Semantic Evaluation, Uppsala, Sweden. Stroudsburg, PA: ACL, pp. 276–83.
UzZaman N., and Allen J. F., 2011. Temporal evaluation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies (ACL-HLT 2011), Portland, OR. Stroudsburg, PA: ACL, pp. 351–6.
UzZaman N., Llorens H., Derczynski L., Verhagen M., Allen J., and Pustejovsky J. 2013. SemEval-2013 Task 1: TempEval-3: evaluating time expressions, events, and temporal relations. In Proceedings of the 7th International Workshop on Semantic Evaluation (SemEval 2013), in Conjunction With the Second Joint Conference on Lexical and Computational Semantcis (* SEM 2013), Association for Computational Linguistics, June, Atlanta. Stroudsburg, PA: ACL.
Vendler Z., 1957. Verbs and times. The Philosophical Review 66 (2): 143–60.
Verhagen M. 2007. Drawing TimeML relations with TBox. In Schilder F., Katz G. and Pustejovsky J. (eds.), Annotating, Extracting and Reasoning about Time and Events. Berlin, Germany: Springer-Verlag, pp. 728.
Verhagen M., Gaizauskas R., Schilder F., Hepple M., Katz G., and Pustejovsky J., 2007. SemEval-2007 Task 15: TempEval temporal relation identification. In Proceedings of the 4th International Workshop on Semantic Evaluations (SemEval 2007), Prague, Czech Republic. Stroudsburg, PA: ACL, pp. 7580.
Verhagen M., Sauri R., Caselli T., and Pustejovsky J., 2010. SemEval-2010 Task 13: TempEval-2. In Proceedings of the 5th International Workshop on Semantic Evaluation (SemEval 2010), Uppsala, Sweden. Stroudsburg, PA: ACL, pp. 5762.
Wei C. P., and Chang Y. H., 2007. Discovering event evolution patterns from document sequences. IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans 37 (2): 273–83.
Wolf F., and Gibson E., 2005. Representing discourse coherence: a corpus-based study. Computational Linguistics 31 (2): 249–87.
Wu Z., and Palmer M., 1994. Verbs semantics and lexical selection. In Proceedings of the 32nd Annual Meeting of the Association for Computational Linguistics (ACL 1994), Las Cruces, NM. Stroudsburg, PA: ACL, pp. 133–8.
Yang Y., Carbonell J. G., Brown R. D., Pierce T., Archibald B. T., and Liu X., 1999. Learning approaches for detecting and tracking news events. Intelligent Systems and their Applications 14 (4): 3243.
Yang C. C., Shi X., and Wei C. P., 2009. Discovering event evolution graphs from news corpora. IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans 39 (4): 850–63.
Yangarber R., Grishman R., Tapanainen P., and Huttunen S., 2000. Automatic acquisition of domain knowledge for information extraction. In Proceedings of the 18th Conference on Computational Linguistics, Hong Kong, vol. 2. Stroudsburg, PA: ACL, pp. 940–6.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Natural Language Engineering
  • ISSN: 1351-3249
  • EISSN: 1469-8110
  • URL: /core/journals/natural-language-engineering
Please enter your name
Please enter a valid email address
Who would you like to send this to? *
×

Metrics

Full text views

Total number of HTML views: 3
Total number of PDF views: 51 *
Loading metrics...

Abstract views

Total abstract views: 483 *
Loading metrics...

* Views captured on Cambridge Core between September 2016 - 17th November 2017. This data will be updated every 24 hours.