Skip to main content Accessibility help

TempCourt: evaluation of temporal taggers on a new corpus of court decisions

  • María Navas-Loro (a1), Erwin Filtz (a2), Víctor Rodríguez-Doncel (a1), Axel Polleres (a2) and Sabrina Kirrane (a2)...


The extraction and processing of temporal expressions (TEs) in textual documents have been extensively studied in several domains; however, for the legal domain it remains an open challenge. This is possibly due to the scarcity of corpora in the domain and the particularities found in legal documents that are highlighted in this paper. Considering the pivotal role played by temporal information when it comes to analyzing legal cases, this paper presents TempCourt, a corpus of 30 legal documents from the European Court of Human Rights, the European Court of Justice, and the United States Supreme Court with manually annotated TEs. The corpus contains two different temporal annotation sets that adhere to the TimeML standard, the first one capturing all TEs and the second dedicated to TEs that are relevant for the case under judgment (thus excluding dates of previous court decisions). The proposed gold standards are subsequently used to compare ten state-of-the-art cross-domain temporal taggers, and to identify not only the limitations of cross-domain temporal taggers but also limitations of the TimeML standard when applied to legal documents. Finally, the paper identifies the need for dedicated resources and the adaptation of existing tools, and specific annotation guidelines that can be adapted to different types of legal documents.



Hide All

The two first authors equally contributed to this work.



Hide All
Bethard, S. 2013. ClearTK-TimeML: a minimalist approach to TempEval 2013. In Proceedings of the Workshop SemEval 2013, 1014. ACL.
Chambers, N. 2013. NavyTime: event and time ordering from raw text. In Proceedings of the Workshop SemEval 2013, 2, 7377.
Chambers, N., Cassidy, T., McDowell, B. & Bethard, S. 2014. Dense event ordering with a multi-pass architecture. Transactions of the ACL, 2, 273284.
Chang, A. X. & Manning, C. D. 2012. SUTime: a library for recognizing and normalizing time expressions. In Proceedings of LREC 2012.
Chang, A. X., et al. 2014. TokensRegex: Defining Cascaded Regular Expressions Over Tokens. Technical report CSTR 2014-02, Department of Computer Science, Stanford University.
Cohen, J. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement 20(1), 3746.
Cunningham, H., Tablan, V., Roberts, A. & Bontcheva, K. 2013. Getting more out of biomedical documents with GATE’s full lifecycle open source text analytics. PLOS Computational Biology 9(2), 116.
Dell’Orletta, F., Marchi, S., Montemagni, S., Plank, B. & Venturi, G. 2012. The splet–2012 shared task on dependency parsing of legal texts. In Semantic Processing of Legal Texts (SPLeT-2012) Workshop Programme, 42.
Derczynski, L. & Gaizauskas, R. 2010. USFD2: annotating temporal expressions and TLINKs for TempEval-2. In Proceedings of the Workshop SemEval, 337340. ACL.
Ferro, L., Gerber, L., Mani, I., Sundheim, B. & Wilson, G. 2005. Tides 2005 Standard for the Annotation of Temporal Expressions. Technical report, MITRE.
Filannino, M. & Nenadic, G. 2015. Temporal expression extraction with extensive feature type selection and a posteriori label adjustment. Data & Knowledge Engineering 100, 1933. ISSN 0169-023X. 10.1016/j.datak.2015.09.002.
Freund, Y., Schapire, R. & Abe, N. 1999. A short introduction to boosting. Journal-Japanese Society For Artificial Intelligence 14(771–780), 1612.
Gildea, D. & Jurafsky, D. 2002. Automatic labeling of semantic roles. Computational Linguistics 28(3), 245288. 10.1162/089120102760275983.
Han, B., Gates, D. & Levin, L. 2006. From language to time: a temporal expression anchorer. In Thirteenth International Symposium on Temporal Representation and Reasoning, 2006, TIME 2006, 196203. IEEE.
Hellmann, S.Lehmann, J. & Auer, S. 2012. NIF: an ontology-based and linked-data-aware NLP Interchange Format.
Isemann, D., Ahmad, K., Fernando, T. & Vogel, C. 2013. Temporal Dependence in Legal Documents, 497504. Springer. ISBN 978-3-642-41278-3.
Ji, H., Cassidy, T., Li, Q. & Tamang, S. 2014. Tackling representation, annotation and classification challenges for temporal knowledge base population. Knowledge and Information Systems 41(3), 611646. ISSN 0219-3116. 10.1007/s10115-013-0675-1.
Kajiwara, T. & Komachi, M. 2016. Building a monolingual parallel corpus for text simplification using sentence similarity based on alignment between word embeddings. In Proceedings of COLING 2016: Technical Papers, 11471158.
Kolya, A. K., Kundu, A., Gupta, R., Ekbal, A. & Bandyopadhyay, S. 2013. JU_CSE: a CRF based approach to annotation of temporal expression, event and temporal relations. In Second Joint Conference on Lexical and Computational Semantics (* SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), 2, 6472.
Lafferty, J. D., McCallum, A. & Pereira, F. C. N. 2001. Conditional random fields: probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning (ICML 2001),Williams College, Williamstown, MA, USA, June 28–July 1, 2001, Brodley, C. E. & Danyluk, A. P. (eds). Morgan Kaufmann, 282289. ISBN 1-55860-778-1.
Lee, K., Artzi, Y., Dodge, J., & Zettlemoyer, L. 2014. Context-dependent semantic parsing for time expressions. In Proceedings of the 52nd Annual Meeting of the ACL,1, 14371447.
Llorens, H., Saquete, E. & Navarro, B. 2010. TIPSem (english and spanish): evaluating CRFs and semantic roles inTempEval-2. In Proceedings of the Workshop SemEval, 284291. ACL.
Llorens, H., Derczynski, L., Gaizauskas, R. J. & Saquete, E. 2012. TIMEN: an open temporal expression normalisation resource. In Proceedings of LREC 2012, 30443051.
Loper, E. & Bird, S. 2002. NLTK: The Natural Language Toolkit. CoRR, cs.CL/0205028.
Mani, I. & Wilson, G. 2000. Robust temporal processing of news. In Proceedings of the 38th Annual Meeting on ACL, 6976. ACL.
Manning, C., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S. & McClosky, D. 2014. The Stanford CoreNLP Natural Language Processing Toolkit. In Proceedings of the 52nd Annual Meeting of the ACL 2014, System Demonstrations, 5560.
Mazur, P. & Dale, R. 2009. The dante temporal expression tagger. In Human Language Technology. Challenges of the Information Society, Vetulani, Z. & Uszkoreit, H. (eds). 245257. Springer. ISBN 978-3-642-04235-5.
Mazur, P. & Dale, R. 2010. Wikiwars: a new corpus for research on temporal expressions. In Proceedings of EMNLP 2010, 913922. ACL.
Minard, A.-L., Speranza, M., Urizar, R., Altuna, B., Van Erp, M., Schoen, A. & Van Son, C. 2016. Meantime, the newsreader multilingual event and time corpus. In Proceedings of LREC 2016, European Language Resources Association.
Naik, V., Vanitha, G. & Inturi, S. 2011. Reasoning in legal text documents with extracted event information. International Journal of Computer Applications 28, 8–-13.
Nivre, J., et al. 2007. The CoNLL 2007 shared task on dependency parsing. In Proceedings of EMNLP-CoNLL 2007.
Northwood, C. 2010. TERNIP: Temporal Expression Recognition and Normalisation in Python. Master’s thesis, University of Sheffield.
Pellow, D. & Eskenazi, M. 2014. An open corpus of everyday documents for simplification tasks. In Proceedings of the 3rd Workshop on Predicting and Improving Text Readability for Target Reader Populations (PITR), 8493.
Pustejovsky, J., Hanks, P., Sauri, R., See, A., Gaizauskas, R., Setzer, A., Radev, D., Sundheim, B., Day, D., Ferro, L. & Lazo, M. 2003a. The TimeBank corpus. In Corpus Linguistics, Lancaster University (UK), 2003, 2831.
Pustejovsky, J., Castano, J. M., Ingria, R., Sauri, R., Gaizauskas, R. J., Setzer, A.Katz, G. & Radev, D. R. 2003bb. TimeML: robust specification of event and temporal expressions intext. In New Directions in Question Answering, Papers from 2003 AAAI Spring Symposium, Maybury, M. T. (ed). AAAI Press, 2834. ISBN 1-57735-184-3.
Quinlan, J. R. 1993. C4.5: Programs for Machine Learning. Morgan Kaufmann. ISBN 1-55860-238-0.
Rijsbergen, C. J. V. 1979. Information Retrieval, 2nd edition. Butterworth-Heinemann. ISBN 0408709294.
, R., Littman, J., Knippen, B., Gaizauskas, R., Setzer, A. &Pustejovsky, J. TimeML Annotation Guidelines. Version 1.2.1, 2006.
Schilder, F. 2005. Event extraction and temporal reasoning in legal documents. In Annotating, Extracting and Reasoning about Time and Events,International Seminar, Dagstuhl Castle. Revised Papers, 5971.
Scott, W. A. 1955. Reliability of content analysis: the case of nominal scale coding. Public Opinion Quarterly, 19(3), 321325.
Steedman, M.Jason, B. 2011. Combinatory categorial grammar, In Non-Transformational Syntax: Formal and Explicit Models of Grammar, Borsley, R. & Borjars, K. (eds). Wiley-Blackwell, 181224.
Strötgen, J. & Gertz, M. 2016. Domain-sensitive temporal tagging. Synthesis Lectures on Human Language Technologies 9(3), 1151.
Strötgen, J. & Gertz, M. 2012. Temporal tagging on different domains: challenges, strategies, and gold standards. In Proceedings of LREC 2012, 12, 37463753.
Styler IV, W. F., Bethard, S., Finan, S., Palmer, M., Pradhan, S., de Groen, P. C., Erickson, B., Miller, T., Lin, C., Savova, G., et al. 2014. Temporal annotation in the clinical domain. Transactions of the Association of Computational Linguistics 2(1), 143154.
Tabassum, J., Ritter, A. & Xu, W. TweeTime: a minimally supervised method for recognizing and normalizing time expressions in Twitter. arXiv preprint arXiv:1608.02904, 2016.
UzZaman, N. & Allen, J. F. 2010. Event and temporal expression extraction from raw text: first step towards a temporally aware system. International Journal of Semantic Computing 4(04), 487508.
UzZaman, N., Llorens, H., Derczynski, L., Allen, J., Verhagen, M. & Pustejovsky, J. 2013. SemEval-2013 Task 1: TempEval-3: evaluating time expressions, events, and temporal relations. In Proceedings of the Workshop SemEval 2013, 19.
Verhagen, M., Mani, I., Sauri, R., Littman, J., Knippen, R., Jang, S.B, Rumshisky, A., Phillips, J., & Pustejovsky, J. 2005. Automating temporal annotation with TARSQI. In Proceedings of the ACL 2005 on Interactive Poster and Demonstration Sessions, ACLdemo ’05, 8184. ACL.
Vlek, C. S., Prakken, H., Renooij, S. & Verheij, B. 2013. Representing and evaluating legal narratives with subscenarios in a Bayesian network. In OASIcs-OpenAccess Series in Informatics, 32. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.
Zhong, X., Sun, A. & Cambria, E. 2017 Time expression analysis and recognition using syntactic token types and general heuristic rules. In Proceedings of the 55th Annual Meeting of the ACL, 1, 420429.

TempCourt: evaluation of temporal taggers on a new corpus of court decisions

  • María Navas-Loro (a1), Erwin Filtz (a2), Víctor Rodríguez-Doncel (a1), Axel Polleres (a2) and Sabrina Kirrane (a2)...


Altmetric attention score

Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed