Skip to main content

Wisdom of crowds versus wisdom of linguists – measuring the semantic relatedness of words


In this article, we present a comprehensive study aimed at computing semantic relatedness of word pairs. We analyze the performance of a large number of semantic relatedness measures proposed in the literature with respect to different experimental conditions, such as (i) the datasets employed, (ii) the language (English or German), (iii) the underlying knowledge source, and (iv) the evaluation task (computing scores of semantic relatedness, ranking word pairs, solving word choice problems). To our knowledge, this study is the first to systematically analyze semantic relatedness on a large number of datasets with different properties, while emphasizing the role of the knowledge source compiled either by the ‘wisdom of linguists’ (i.e., classical wordnets) or by the ‘wisdom of crowds’ (i.e., collaboratively constructed knowledge sources like Wikipedia).

The article discusses benefits and drawbacks of different approaches to evaluating semantic relatedness. We show that results should be interpreted carefully to evaluate particular aspects of semantic relatedness. For the first time, we employ a vector based measure of semantic relatedness, relying on a concept space built from documents, to the first paragraph of Wikipedia articles, to English WordNet glosses, and to GermaNet based pseudo glosses. Contrary to previous research (Strube and Ponzetto 2006; Gabrilovich and Markovitch 2007; Zesch et al. 2007), we find that ‘wisdom of crowds’ based resources are not superior to ‘wisdom of linguists’ based resources. We also find that using the first paragraph of a Wikipedia article as opposed to the whole article leads to better precision, but decreases recall. Finally, we present two systems that were developed to aid the experiments presented herein and are freely available1 for research purposes: (i) DEXTRACT, a software to semi-automatically construct corpus-driven semantic relatedness datasets, and (ii) JWPL, a Java-based high-performance Wikipedia Application Programming Interface (API) for building natural language processing (NLP) applications.

Hide All
Anscombe, F. J. 1973. Graphs in statistical analysis. American Statistician 27: 1721.
Banerjee, S., and Pedersen, T. 2002. An adapted lesk algorithm for word sense disambiguation using WordNet. In CICLing '02: Proceedings of the Third International Conference on Computational Linguistics and Intelligent Text Processing, pp. 136145, London: Springer Verlag.
Bernard, J. 1986. The Macquarie Thesaurus. Sidney, Australia: Macquarie Library.
Boyd-Graber, J., Fellbaum, C., Osherson, D., and Shapire, R. 2006. Adding dense, weighted, connections to WordNet. In Proceedings of the Third Global WordNet Meeting, Jeju Island, Korea.
Budanitsky, A., and Hirst, G. 2006. Evaluating WordNet-based measures of semantic distance. Computational Linguistics 32 (1): 1347.
Fellbaum, C. 1998. WordNet an Electronic Lexical Database. Cambridge, MA: MIT Press.
Finkelstein, L., Gabrilovich, E., Matias, Y., Rivlin, E., Solan, Z., and Wolfman, G. 2002. Placing search in context: the concept revisited. ACM Transactions on Information Systems 20 (1): 116–31.
Gabrilovich, E., and Markovitch, S. 2007. Computing semantic relatedness using Wikipedia-based explicit semantic analysis. In Proceedings of The 20th International Joint Conference on Artificial Intelligence (IJCAI), pp. 1606–11, Hyderabad, India.
Galley, M., and McKeown, K. 2003. Improving word sense disambiguation in lexical chaining. In Proceedings of 18th International Joint Conference on Artificial Intelligence (IJCAI'03), pp. 1486–8, Acapulco, Mexico.
Gurevych, I. 2005. Using the structure of a conceptual network in computing semantic relatedness. In Proceedings of the 2nd International Joint Conference on Natural Language Processing, pp. 767–78. Jeju Island, Korea.
Gurevych, I., Müller, C., and Zesch, T. 2007. What to be? – electronic career guidance based on semantic relatedness. In Proceedings of ACL, pp. 1032–9, Prague, Czech Republic. Association for Computational Linguistics.
Gurevych, I., and Strube, M. 2004. Semantic similarity applied to spoken dialogue summarization. In The 22nd International Conference on Computational Linguistics (COLING), pp. 764–70, Geneva, Switzerland.
Halliday, M. A. K., and Hasan, R. 1976. Cohesion in English. London: Longman.
Hirst, G., and St-Onge, D. 1998. Lexical chains as representation of context for the detection and correction malapropisms. In Fellbaum, Christiane (ed.), WordNet: An Electronic Lexical Database and Some of Its Applications, pp. 305332. Cambridge, MA: The MIT Press.
Jarmasz, M., and Szpakowicz, S. 2003. Roget's thesaurus and semantic similarity. In Proceedings of Recent Advances in Natural Language Processing, pp. 111–20.
Jiang, J. J., and Conrath, D. W. 1997. Semantic similarity based on corpus statistics and lexical taxonomy. In Proceedings of the 10th International Conference on Research in Computational Linguistics, Taipei, Taiwan.
Kozima, H., and Furugori, T. 1993. Similarity between words computed by spreading activation on an English dictionary. In Proceedings of the sixth conference of the European chapter of the Association for Computational Linguistics, pp. 232–9, Morristown, NJ.
Kunze, C. 2004. Computerlinguistik und Sprachtechnologie. In Carstensen, K. U., Ebert, C., Endriss, C., Jekat, S., Klabunde, R., and Langer, H. (eds.), Lexikalisch-semantische Wortnetze, pp. 423–31. Berlin: Spektrum Akademischer Verlag.
Leacock, C., and Chodorow, M. 1998. Combining local context and WordNet similarity for word sense identification. In WordNet: An Electronic Lexical Database, pp. 265–83. Cambridge, MA: MIT Press.
Lesk, M. 1986. Automatic sense disambiguation using machine readable dictionaries: how to tell a pine cone from an ice cream cone. In Proceedings of the 5th Annual International Conference on Systems Documentation, pp. 24–6. Toronto, Canada.
Li, Y., Bandar, Z. A., and McLean, D. 2003. An approach for measuring semantic similarity between words using multiple information sources. IEEE Transactions on Knowledge and Data Engineering 15: 871–82.
Lin, D. 1998. An information-theoretic definition of similarity. In Proceedings of International Conference on Machine Learning, pp. 296304. Madison, WI.
McHale, M. 1998. A comparison of wordnet and roget's taxonomy for measuring semantic similarity. CoRR, cmp-lg/9809003.
Mihalcea, R., and Moldovan, D. I. 1999. A method for word sense disambiguation of unrestricted text. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pp. 152–8, Maryland, MD: Association for Computational Linguistics.
Miller, G. A., and Charles, W. G. 1991. Contextual correlates of semantic similarity. Language and Cognitive Processes 6 (1): 128.
Mohammad, S., Gurevych, I., Hirst, G., and Zesch, T. 2007. Cross-lingual distributional profiles of concepts for measuring semantic distance. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pp. 571–80, Prague, Czech Republic: Association for Computational Linguistics.
Morris, J., and Hirst, G. 1991. Lexical cohesion computed by thesaural relations as an indicator of the structure of text. Computational Linguistics 17 (1): 2148.
Morris, J., and Hirst, G. 2004. Non-classical lexical semantic relations. In Workshop on Computational Lexical Semantics, Human Language Technology Conference of the North American Chapter of the ACL, pp. 4651. Boston, MA.
Patwardhan, S., Banerjee, S., and Pedersen, T. 2003. Using measures of semantic relatedness for word sense disambiguation. In Proceedings of the Fourth International Conference on Intelligent Text Processing and Computational Linguistics, pp. 241–57, Mexico City, Mexico.
Patwardhan, S., and Pedersen, T. 2006. Using WordNet based context vectors to estimate the semantic relatedness of concepts. In Proceedings of the EACL 2006 Workshop Making Sense of Sense - Bringing Computational Linguistics and Psycholinguistics Together, pp. 18, Trento, Italy: Association for Computational Linguistics.
Pirro, G., and Seco, N. 2008). Design, implementation and evaluation of a new semantic similarity metric combining features and intrinsic information content. In OTM '08: Proceedings of the OTM 2008 Confederated International Conferences, CoopIS, DOA, GADA, IS, and ODBASE, pp. 1271–88, Monterrey, Mexico.
Procter, P. 1978. Longman Dictionary of Contemporary English. Longman, London.
Qiu, Y., and Frei, H. P. 1993. Concept based query expansion. In Proceedings of the 16th ACM International Conference on Research and Development in Information Retrieval, ACM
Rada, R., Mili, H., Bicknell, E., and Blettner, M. 1989. Development and application of a metric on semantic nets. IEEE Transactions on Systems, Man, and Cybernetics 19 (1): 1730.
Resnik, P. 1995 Using information content to evaluate semantic similarity. In Proceedings of the 14th International Joint Conference on Artificial Intelligence, pp. 448–53, Montreal, Canada.
Roget, P. 1962. Roget's International Thesaurus, 3rd ed. Berrey, L. V., and Carruth, G. (eds.), New York: Thomas Y. Crowell Co.
Rubenstein, H., and Goodenough, J. B. 1965. Contextual correlates of synonymy. Communications of the ACM 8 (10): 627–33.
Salton, G., and McGill, M. J. 1983. Introduction to Modern Information Retrieval. New York: McGraw-Hill.
Seco, N., and Hayes, T. V. J. 2004. An intrinsic information content metric for semantic similarity in WordNet. In Proceedings of ECAI'2004, the 16th European Conference on Artificial Intelligence, Valencia, Spain.
Silber, H. G., and McCoy, K. F. 2002. Efficiently computed lexical chains as an intermediate representation for automatic text summarization. Comput. Linguist. 28 (4): 487–96.
Stevenson, M., and Greenwood, M. A. 2005. A semantic approach to ie pattern induction. In ACL '05: Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pp. 379–86. Morristown, NJ: Association for Computational Linguistics.
Strube, M., and Ponzetto, S. P. 2006. WikiRelate! Computing semantic relatedness using Wikipedia. In Proceedings of the 21st National Conference on ArtificialIntelligence (AAAI-06), pp. 1419–24, Boston, MA.
Turney, P. 2006. Expressing implicit semantic relations without supervision. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th annual meeting of the ACL, pp. 313–20, Sydney, Australia: Association for Computational Linguistics.
Voss, J. 2006. Collaborative thesaurus tagging the Wikipedia way. CoRR, abs/cs/0604036.
Wallace, D., and Wallace, L. A. 2001–2005. Reader's Digest, das Beste für Deutschland. January 2001–December 2005. Stuttgart: Verlag Das Beste.
Weeds, J. E. 2003. Measures and Applications of Lexical Distributional Similarity. PhD thesis, East Sussex, UK: University of Sussex.
Wu, Z., and Palmer, M. 1994. Verb semantics and lexical selection. In 32nd Annual Meeting of the ACL, pp. 133–8, Las Cruces, Mexico: Association for Computational Linguistics.
Yang, D., and Powers, D. M. W. 2006. Verb similarity on the taxonomy of WordNet. In Proceedings of the Third International WordNet Conference (GWC-06), pp. 121–8, Jeju Island, Korea.
Zesch, T., and Gurevych, I. 2006. Automatically creating datasets for measures of semantic relatedness. In Proceedings of the ACL-Workshop on Linguistic Distances, pp. 1624, Sydney, Australia: Association for Computational Linguistics.
Zesch, T., and Gurevych, I. 2007. Analysis of the Wikipedia category graph for NLP applications. In Proceedings of the TextGraphs-2 Workshop (NAACL-HLT 2007), pp. 18, Rochester, NY. Association for Computational Linguistics.
Zesch, T., Gurevych, I., and Mühlhäuser, M. 2007a. Analyzing and accessing Wikipedia as a lexical semantic resource. In Rehm, G., Witt, A., and Lemnitzer, L. (eds.), Data Structures for Linguistic Resources and Applications, pp. 197205. Tuebingen, Germany: Gunter Narr.
Zesch, T., Gurevych, I., and Mühlhäuser, M. 2007b. Comparing Wikipedia and german wordnet by evaluating semantic relatedness on multiple datasets. In Proceedings of Human Language Technologies: The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT), pp. 205–8. Rochester, NY: Association for Computational Linguistics.
Zesch, T., Müller, C., and Gurevych, I. 2008a. Extracting lexical semantic knowledge from Wikipedia and Wiktionary. In Proceedings of the Conference on Language Resources and Evaluation (LREC), Marrakech, Morocco.
Zesch, T., Müller, C., and Gurevych, I. 2008b Using Wiktionary for computing semantic relatedness. In Proceedings of AAAI, pp. 861–7. Chicago, IL.
Recommend this journal

Email your librarian or administrator to recommend adding this journal to your organisation's collection.

Natural Language Engineering
  • ISSN: 1351-3249
  • EISSN: 1469-8110
  • URL: /core/journals/natural-language-engineering
Please enter your name
Please enter a valid email address
Who would you like to send this to? *


Full text views

Total number of HTML views: 0
Total number of PDF views: 0 *
Loading metrics...

Abstract views

Total abstract views: 0 *
Loading metrics...

* Views captured on Cambridge Core between <date>. This data will be updated every 24 hours.

Usage data cannot currently be displayed