2.1 Tasks and datasets

Entailment between pairs of words has been studied extensively both as one of the fundamental relationships in the lexicon and as an essential building block of models of natural language inference. Several recent task formulations equate lexical entailment with hypernymy/hyponymy or the IS_A relationship (Vulić et al. Reference Vulić, Gerz, Kiela, Hill and Korhonen2017; Vulić, Ponzetto, and Glavaš Reference Vulić, Ponzetto and Glavaš2019), treating it as a relationship between two entries in a lexicon and creating datasets of labeled pairs of words such as Hyperlex (Vulić et al. Reference Vulić, Gerz, Kiela, Hill and Korhonen2017). Other works are concerned with the entailment relationship between two words in their respective contexts. Pointing out that eliminate entails treat in Aspirin eliminates headaches but not in Aspirin eliminates patients, Levy and Dagan (Reference Levy and Dagan2016) introduce a dataset of annotated relation pairs. This dataset uses question–answer pairs as context for lexical entailment, other approaches involve providing context as pairs of arguments (Zeichner, Berant, and Dagan Reference Zeichner, Berant and Dagan2012) or pairs of argument types (Berant, Dagan, and Goldberger Reference Berant, Dagan and Goldberger2011; Schmitt and Schütze Reference Schmitt and Schütze2019). Lexical entailment can also be viewed as a special case of natural language inference (NLI), modern systems for this task are commonly trained and evaluated on the Stanford Natural Language Inference (SNLI) (Bowman et al. Reference Bowman, Angeli, Potts and Manning2015) dataset and the Multi-Genre NLI Corpus (MultiNLI) (Williams, Nangia, and Bowman Reference Williams, Nangia and Bowman2018) dataset.

The approach taken in this article, outlined in Section 3.2, is based on the 4lang formalism for representing (lexical) semantics and is capable of inspecting the relationship between the meaning of any two utterance fragments. We will evaluate our system on two recent benchmarks. The datasets used in the 2020 Semeval task “Predicting Multilingual and Cross-lingual (graded) Lexical Entailment’’ (Glavaš et al. Reference Glavaš, Vulić, Korhonen and Ponzetto2020) are derived from HyperLex (Vulić et al. Reference Vulić, Gerz, Kiela, Hill and Korhonen2017), a dataset of monolingual and cross-lingual–graded lexical entailment. Candidate word pairs for human annotation were gathered from the USF (Nelson, McEvoy, and Schreiber Reference Nelson, McEvoy and Schreiber2004) and WordNet (Miller Reference Miller1995) databases. For our experiments, we will use the binary labels of the monolingual subsets for English, German, and Italian. Some examples for each language are shown in Table 1. On this dataset, we will compare our method to the GLEN system for measuring multilingual and cross-lingual lexical entailment using specialized word embeddings (Vulić et al. Reference Vulić, Ponzetto and Glavaš2019), which outperforms previous baselines in Upadhyay et al. (Reference Upadhyay, Vyas, Carpuat and Roth2018), and also to the other systems participating in the shared task, including our own earlier system presented in Kovács et al. (Reference Kovács, Gémes, Kornai and Recski2020).

Table 1.

Example entries of the monolingual binary portion of the Semeval lexical entailment dataset (Glavaš et al. Reference Glavaš, Vulić, Korhonen and Ponzetto2020)

A more challenging task formulation is provided by the SherLlic dataset of lexical inference in context (Schmitt and Schütze Reference Schmitt and Schütze2019), which was built by extracting inference candidates from an entity-linked portion of the ClueWeb corpus (Gabrilovich, Ringgaard, and Subramanya Reference Gabrilovich, Ringgaard and Subramanya2013) and using them as input to human annotation. Because annotation candidates were chosen based on distributional evidence, many entailment pairs in the final dataset are completely novel, missing from existing knowledge bases such as WordNet. Argument types for event pairs are necessary to disambiguate between word senses. For example, run entails lead if its arguments are of type PERSON and COMPANY (e.g., Bezos runs Amazon) but not if they are COMPUTER and SOFTWARE, as in my mac runs macOS. Table 2 shows further examples of entries in the SherLlic dataset. An extensive evaluation of various LE systems on this dataset presented in Schmitt and Schütze (Reference Schmitt and Schütze2019) will serve as the starting point for our evaluations in Section 4.

Table 2.

Example entries of the SherLlic dataset (Schmitt and Schütze Reference Schmitt and Schütze2019). Argument labels indicate entity types: PER – person, LOC – location, ORGF – organization_founder, EMPL – employer, AUTH – book_author

The need for such novel datasets has been made clear by several recent experiments that point out the biases of deep learning based models of NLI. Glockner, Shwartz, and Goldberg (Reference Glockner, Shwartz and Goldberg2018) constructed a new NLI test set from SNLI by replacing a single word in sentences from the training set, and used this new benchmark to expose top NLI systems’ inability to perform true lexical inference. The only model in their evaluation not showing a major drop in performance was the one incorporating lexical knowledge (Chen et al. Reference Chen, Cui, Ma, Wang, Liu and Hu2018). The inability of deep learning based NLI models to generalize across datasets was also shown in a more recent study across six systems and three datasets (Talman and Chatzikyriakidis Reference Talman and Chatzikyriakidis2019). These findings call into question whether black box models trained for high performance on any single NLI benchmark can be regarded as true models of inference. We believe that rule-based models such as the one proposed in this paper can facilitate further qualitative analysis of deep learning models by providing strong explainable baselines on multiple tasks and datasets.

2.2 Approaches to entailment and inference

When seen as a task of detecting hypernymy, lexical entailment is most often addressed using distributional methods. Hypernymy candidates are encoded using word embeddings and classified by either neural networks (Nguyen et al. Reference Nguyen, Köper, Schulte im Walde and Vu2017; Shwartz, Goldberg, and Dagan Reference Shwartz, Goldberg and Dagan2016; Yu et al. Reference Yu, Wang, Lin and Wang2015) or non-neural classifiers such as Support Vector Machines (SVMs) and logistic regression (Baroni et al. Reference Baroni, Bernardi, Do and Shan2012; Levy et al. Reference Levy, Remus, Biemann and Dagan2015; Roller, Erk, and Boleda Reference Roller, Erk and Boleda2014). Yu et al. (Reference Yu, Wang, Lin and Wang2015) proposes a neural model for supervised learning of hypernymy-specific embeddings. Nguyen et al. (Reference Nguyen, Köper, Schulte im Walde and Vu2017) argues that standard distributional models cannot account for the asymmetric property of hypernymy, and introduces HyperVec, a hierarchical approach to learning hypernymy embeddings that allowed for significant improvement over the state-of-the-art on the HyperLex dataset. Glavaš and Ponzetto (Reference Glavaš and Ponzetto2017) proposes Dual Tensor, an approach based on neural models to explicitly model the asymmetric nature of the hypernymy relation. Dual Tensor transforms generic embeddings into specialized vectors for scoring concept pairs based on whether the asymmetric relation holds. A different approach is taken by HypeNET (Shwartz et al. Reference Shwartz, Goldberg and Dagan2016), a method based on extracting paths between premise and hypothesis from dependency trees and using them as inputs to Long Short-term Memory Networks (LSTMs). Fine-tuning generic word vectors using external knowledge such as WordNet (Miller Reference Miller1995) has improved performance on a range of language understanding tasks (Glavaš and Vulić Reference Glavaš and Vulić2018). To extend this method to unseen words, Kamath et al. (Reference Kamath, Pfeiffer, Ponti, Glavaš and Vulić2019) introduced POSTLE (post-specialization for LE), a model that learns an explicit global specialization function captured with feed forward neural networks.

Inference systems trained on the SNLI and MultiNLI datasets mostly use neural language models based on the Transformer architecture (Vaswani et al. Reference Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser and Polosukhin2017), in particular BERT (Devlin et al. Reference Devlin, Chang, Lee and Toutanova2019). SemBERT (Zhang et al. Reference Zhang, Wu, Zhao, Li, Zhang, Zhou and Zhou2020) uses a BERT backbone enhanced with a Semantic Role Labeler (SRL), MT-DNN (Liu et al. Reference Liu, He, Chen and Gao2019) enhances the system presented in Liu et al. (Reference Liu, Gao, He, Deng, Duh and Wang2015) with BERT. Top results on the MultiNLI benchmark were achieved by optimized, pretrained, and finetuned versions of BERT, RoBERTa (Zhuang et al. Reference Zhuang, Wayne, Ya and Jun2021), and ALBERT (Lan et al. Reference Lan, Chen, Goodman, Gimpel, Sharma and Soricut2020). Using rule-based models in NLI and combining them with deep learning based language models (BERT, ALBERT, RoBERTa) has recently also led to competitive results (Haruta, Mineshima, and Bekki Reference Haruta, Mineshima and Bekki2020; Kalouli, Crouch, and de Paiva Reference Kalouli, Crouch and de Paiva2020). In this article, we will also present an improvement over such a model using our fully rule-based method for detecting entailment between pairs of words.

2.3 Semantic parsing

Semantic parsing is the task of mapping natural language text to some model of its meaning, and as a language processing step it can only be defined with respect to a particular system of semantic representation. As interest in symbolic representations has increased, so has the variety of such systems. Most practical frameworks link to lexical databases, including the graph of synonym sets in WordNet (Miller Reference Miller1995), the semantic role inventory in VerbNet (Kipper et al. Reference Kipper, Korhonen, Ryant and Palmer2008), or the ontology of semantic frames in FrameNet (Ruppenhofer et al. Reference Ruppenhofer, Ellsworth, Petruck, Johnson and Scheffczyk2006). Abstract meaning representations (AMR) (Banarescu et al. Reference Banarescu, Bonial, Cai, Georgescu, Griffitt, Hermjakob, Knight, Koehn, Palmer and Schneider2013), which became one of the most widely used representations in semantic parsing, is based on the PropBank database (Palmer, Gildea, and Kingsbury Reference Palmer, Gildea and Kingsbury2005). AMR handles a range of phenomena in the semantics of English but does not provide any treatment of word meaning and is also not intended as a universal representation of natural language semantics, despite some early efforts in Xue et al. (Reference Xue, Bojar, Hajič, Palmer, Urešová and Zhang2014) and the release of a Chinese AMR bank (Li et al. Reference Li, Wen, Qu, Bu and Xue2016). Automatic construction of AMR graphs from text is usually performed using deep neural models (Konstas et al. Reference Konstas, Iyer, Yatskar, Choi and Zettlemoyer2017; Lyu and Titov Reference Lyu and Titov2018; Zhang et al. Reference Zhang, Ma, Duh and Van Durme2019) trained on AMR banks. A language-agnostic approach to meaning representation is taken by Universal Conceptual Cognitive Annotation (UCCA) (Abend and Rappoport Reference Abend and Rappoport2013), which abstracts away from syntax by representing words of a sentence as leaf nodes of directed acyclic graphs (DAGs) representing scenes evoked by predicates. As in the case of AMRs, top-performing parsers for UCCA all employ neural networks trained on manually annotated sentences (Hershcovich, Abend, and Rappoport Reference Hershcovich, Abend and Rappoport2017, Reference Hershcovich, Abend and Rappoport2018; Ozaki et al. Reference Ozaki, Morio, Koreeda, Morishita and Miyoshi2020; Samuel and Straka Reference Samuel and Straka2020). Not only are these formalisms often dependent on large manually built databases, existing parsing systems also rely on large datasets of hand-crafted representations (sembanks) for training. Transferring such systems across domains and languages therefore requires a considerable amount of expert human labor. In this article, we will use a meaning representation formalism designed to enable robust rule-based parsing and relying only on language-agnostic resources. Here we provide an overview only, our extensions and modifications shall be presented in Section 3.

4lang is a theory and formalism for representing the semantics of natural language, developed in Kornai (Reference Kornai2012, Reference Kornai2019), Kornai et al. (Reference Kornai, Ács, Makrai, Nemeskey, Pajkossy and Recski2015), and partially implemented in Recski (Reference Recski2016, Reference Recski2018). 4lang concept graphs represent meaning as directed graphs of language-independent concepts. Edges connecting concepts have one of three labels. Predicates are connected to their arguments via edges labeled 1 and 2, for example, cat $\xleftarrow1$ catch $\xrightarrow2$ mouse, while 0-edges represent all relationships involving inheritance, including not only hypernymy (dog $\xrightarrow0$ mammal) but also attribution (dog $\xrightarrow0$ four-legged), and unary predication (dog $\xrightarrow0$ bark). This implies a broad definition of lexical entailment: unless explicitly overridden, dog entails not only mammal, but also bark and four-legged. In its current implementations, 4lang concepts have no grammatical attributes and no event structure, for example, the phrases water freezes and frozen water would both be represented as water $\xrightarrow0$ freeze. Figure 2 shows the 4lang definition of the concept jewel obtained by processing the dependency parse in Figure 1 of the Wiktionary definition A precious or semi-precious stone; gem, gemstone.

Figure 1.

Dependency parse of the definition of jewel.

Figure 2.

4lang definition graph of jewel.

Optionally, the 4lang system allows us to expand graphs, a process which unifies one graph with the definition graphs of the concepts within that graph. For example, a graph containing the node jewel will be expanded to include as a subgraph the entire definition graph in Figure 2. This step will be essential to our method presented in Section 3.2. 4lang graphs can be built automatically from Universal Dependencies (Nivre et al. Reference Nivre, Abrams and Agić2018) using the rule-based dep_to_4lang module, which we extend to improve performance across languages. Section 3 will describe these changes, as well as our reimplementation of the parsing algorithm using Interpreted Regular Tree Grammars (IRTGs) (Koller Reference Koller2015). While in the present work this method is used to map natural language definitions to concept graphs representing the meaning of individual words, the system is capable of processing any UD graph and can be used to construct the 4lang semantic representation of any text.

2.4 Entailment in semantic graphs

Lexical entailment is explicitly encoded by several semantic formalisms. The hyponymy/ hypernymy relation, the narrow interpretation of lexical entailment used for example in the Semeval 2020 shared task, is represented directly by WordNet. The 0-edge in 4lang graphs is a more generic relation that subsumes the hypernymy relation along with all other types of predication, and accessibility of one concept from another in a 4lang graph can be seen as a broad definition of lexical entailment. Recently, we have shown (Kovács et al. Reference Kovács, Gémes, Kornai and Recski2020) the direct applicability of such semantic graphs to hypernymy detection task by using them in a competitive system at the Semeval entailment task (Glavaš et al. Reference Glavaš, Vulić, Korhonen and Ponzetto2020). When using 4lang definition graphs, we defined entailment to hold between a pair of premise and hypothesis words if and only if in the twice-expanded definition graph of the premise there is a directed path of 0-edges leading from the premise word to the hypothesis word (the maximum number of expansions was chosen arbitrarily). Because entailment does not flow through locative and negative modifier clauses, inference had to be blocked explicitly where the path would go through prepositions (e.g. English in, of, on, German in, auf, Italian di, su, il) or words conveying negation (English not, German keine, etc.). For example, where nose is defined as “a protuberance on the face’’, 4lang graphs would contain a path of 0-edges from nose to face, falsely representing entailment. One of the modifications of the parsing algorithm to be presented in this article will ensure that subgraphs representing such relations do not contain 0-paths.

On the Semeval dataset of word-level entailment, the above method detected only about one-third of all true entailments in the dev dataset but achieved nearly perfect precision. On well-resourced languages such as English, WordNet was shown to be a very strong baseline both in terms of precision and recall, and the main contribution of 4lang was its ability to increase recall without hurting precision, increasing the performance of strong WordNet-based baselines on three languages, ranking first in English and Italian and second-best on German, see Table 3. For English and Italian official WordNet releases were accessed via the nltk Footnote d package. In Kovács et al. (Reference Kovács, Gémes, Kornai and Recski2020) for German, we did not have access to a high-coverage WordNet release, word pairs were therefore translated from German to English using the wikt2dict system (ács, Pajkossy, and Kornai Reference Ács, Pajkossy and Kornai2013) to enable the use of English WordNet on the German task. In Section 4, when evaluating the methods presented in this article and comparing them to previous methods, we include a variant of this system using a recent German WordNet release, GermaNet (Hamp and Feldweg Reference Hamp and Feldweg1997; Henrich and Hinrichs Reference Henrich and Hinrichs2010).

Table 3.

Official monolingual LE results on the ANY track (F-scores)

Bold values represents best scores.

3.1 Semantic parsing with Interpreted Regular Tree Grammars

Many NLP tasks involve constructing or transforming graphs that represent syntax and/or semantics. Interpreted Regular Tree Grammars (IRTGs) (Koller Reference Koller2015) can be used to encode the correspondence between sets of such structures and have in recent years been used to perform syntactic parsing (Koller and Kuhlmann Reference Koller and Kuhlmann2012), generation (Koller and Engonopoulos Reference Koller and Engonopoulos2017), semantic parsing (Groschwitz, Koller, and Teichmann Reference Groschwitz, Koller and Teichmann2015; Groschwitz et al. Reference Groschwitz, Lindemann, Fowlie, Johnson and Koller2018), and surface realization (Kovács et al. Reference Kovács, Ács, Ács, Kornai and Recski2019; Recski et al. Reference Recski, Kovács, Gémes, Ács and Kornai2020). The system presented in this article uses an IRTG for transforming UD trees into 4lang graphs.

An IRTG rule is a (possibly weighted) rewrite rule of a Regular Tree Grammar that is mapped to an arbitrary number of named interpretations, each of which are operations of an algebra with the same arity as the RTG rule. Thereby any particular derivation of an IRTG grammar deterministically maps to a sequence of operations in each of the interpretation algebras. Parsing an object of one algebra involves finding the IRTG derivation(s) of the highest likelihood that would generate this object, while decoding is the subsequent construction of an object in another algebra using this sequence. The grammars used by our system establish a mapping between operations of two algebras of directed graphs, one for constructing UD representations and another for constructing 4lang graphs. The overall structure of an IRTG rule with two interpretations is shown in Figure 3. The graph operations used in this example will now be introduced.

Figure 3.

Example of an IRTG rule.

Following the practice of Koller and Kuhlmann (Reference Koller and Kuhlmann2011), we use s-graph algebras. We give an informal overview of its operations, see Courcelle and Engelfriet (Reference Courcelle and Engelfriet2012) for a more formal explanation. S-graphs are graphs whose vertices may be labeled by one of a countable set of sources, which are essentially special node labels accessible by operations of the algebra. The binary merge operation creates an s-graph by taking the union of its argument graphs and merging nodes with identical sources. In other words, when two s-graphs $G_1$ and $G_2$ are merged, the resulting s-graph G’ will contain all nodes of $G_1$ and $G_2$ , and when a pair of nodes $(v_1,v_2) \in V(G_1) \times V(G_2)$ have the same source name, they will be mapped to a single node v’ in G’ that has all adjacent edges of $v_1$ and $v_2$ . Sources can be manipulated by the rename and forget operations for changing or deleting a given source label from all nodes of an s-graph.

We illustrate the algebra operations and the structure of an IRTG using a simple example. Figure 3 shows an IRTG rule encoding the correspondence between two operations: one that adds a directed amod edge between the root nodes of two UD graphs, and another that adds a directed 0 edge between the root nodes of two 4lang graphs. The first line in the example is the IRTG rule, specifying that an operation called NOUN_AMOD_ADJ takes as its arguments two objects of type NOUN and ADJ. The second and third lines are the interpretations, each specifying the same sequence of operations: the first argument is merged with a graph consisting of a single directed edge, the second argument is merged with the resulting graph after its root source has been changed to dep1 using the rename operation. Finally, the dep1 source is deleted, using the forget operation. This sequence of operations is equivalent to adding a single directed edge between the root nodes of the two argument graphs, ensuring that the root of the first becomes the root of the new graph. The syntax used in this example is specified by the Algebraic Language Toolkit, or alto Footnote e (Gontrum et al. Reference Gontrum, Groschwitz, Koller and Teichmann2017), an open-source parser for IRTGs that implements a variety of algebras, including the s-graph algebras used in this article. Graph literals in each interpretation are given using the PENMAN notation,Footnote f in this simple case the only difference between the two strings is the edge label. The correspondence expressed by the rule in Figure 3 can also be represented by the pair of graph templates in Figure 4, we will use this simplified format in future examples.

Figure 4.

Graph representation of sample rule in Figure 3.

Parsing UD graphs and transforming them into 4lang graphs on a large scale would be possible using a single grammar with terminal rules corresponding to each word of the input graph. But since building and continuously extending such a large set of rules would be inefficient, we instead chose to dynamically generate individual grammars for each input UD graph, a process that makes it possible to use sets of rule templates for generating similar rules, and to organize them into configurable, application-specific rule lexica. We then construct IRTG grammars for individual UD trees by looking up their edges in such lexica, for example, the UD edge NOUN $\xrightarrow{amod}$ ADJ will always warrant an IRTG rule that maps this edge to the 4lang edge NOUN $\xrightarrow{0}$ ADJ. Other patterns require the lexicon to reference additional nodes in the input graph, we discuss some examples below. Terminal rules that map POS-tags to words in both interpretations are added to each grammar in a trivial step. The grammar generation framework described here is available as open-source software as part of the tuw-nlp Footnote g Python package, and is a core dependency of the system presented in this article, which is also available as part of an open-source library.Footnote h

The starting point for creating rule lexica was the mapping of the original dep_to_4lang paper (Recski Reference Recski2018), which establishes all trivial 1-to-1 mappings between pairs of UD and 4lang relations. For example, all UD relations representing modification (amod, advmod, nummod) are mapped to 0-edges, while relations between predicates and their objects (obj, nsubj:pass) become 2-edges. Subjects relations (nsubj, csubj) are mapped to a pair of 0- and 1-edges; in the sentence John is supporting Bill, the UD relation support $\xrightarrow{nsubj}$ John becomes support $\overset{1}{\underset{0}{\rightleftharpoons}}$ John. Clausal modifiers (acl, advcl) are generally also mapped to 0-edges, some newly introduced exceptions will be discussed below. Additionally, we introduce a new mechanism for non-core (oblique) arguments marked by the obl relation.

Consider the Wiktionary definition of teacher: someone who teaches, especially in a school. The UD edge teach $\xrightarrow{obl}$ school does not in itself reveal the semantic relationship between the two concepts, thus we introduce a pattern that will also take into account the case relation marking the argument: the full UD analysis of this sentence contains the subgraph teach $\xrightarrow{obl}$ school $\xrightarrow{case}$ in, allowing us to build the 4lang graph teach $\xleftarrow1$ in $\xrightarrow2$ school using an IRTG rule represented in Figure 5. We also implemented an English-specific exception to this rule: the preposition by will trigger the configuration for the predicate–subject relation, so that the UD analyses of the sentences John is supported by Bill and Bill is supporting John shall both be mapped to the same 4lang graph (see Figure 6).

Figure 5.

Obliques in UD and 4lang.

Figure 6.

4lang definition graph of John is supported by Bill and Bill is supporting John.

Another shortcoming of the original algorithm for mapping dependency graphs to 4lang representations is its treatment of coordination. The strategy introduced in Section 3.4.1 of Recski (Reference Recski2018) simply copied all semantic relations between all elements of coordinating constructions, which has proved practical for downstream applications despite introducing some erroneous edges. Our system replicates this behavior in its mapping from UD to 4lang. Some simple patterns over specific conjunctions (and, or, etc.) could be used to differentiate between occurrences of the conj dependency, similar to the approach of Enhanced Universal Dependencies (Schuster and Manning Reference Schuster and Manning2016), but modeling the semantics of coordinating conjunctions would nevertheless require considerable language-specific effort (see Gerdes and Kahane Reference Gerdes and Kahane2015; Kanayama et al. Reference Kanayama, Han, Asahara, Hwang, Miyao, Choi and Matsumoto2018).

Perhaps the most significant limitation of the original system was its lack of treatment for relationships between clauses. This is partly due to the fact that the type of the UD relation connecting the heads of two clauses often reveals very little about the semantic function of the dependent clause. The UD relation acl is definedFootnote i as clausal modifier of noun (adjectival clause), and indeed the general case can be handled by 0-edges, but this rule will currently create erroneous edges for sentences such as I have a parakeet named Cookie, since there is no mechanism to detect that in this case parakeet is the object of named. Another issue is posed by the UD relation acl:relcl (relative clause modifier of a noun), which is used both in an animal that moves and the man you love, constructions that would warrant the edges animal $\xrightarrow{0}$ move and man $\xleftarrow{2}$ love, respectively. The first of these two examples can receive the correct treatment based on the presence of the nsubj edge between move and that, thus we introduce a rule that implements the correspondence in Figure 7. Other occurrences of acl:relcl are currently not processed. For the full mapping between UD and 4lang structures see Table 4.

Figure 7.

Relative clause modifier of a noun in UD and 4lang.

Table 4.

Mapping from UD relations to 4lang subgraphs

While some of these newly introduced mechanisms are still rudimentary, improving the general mechanisms of building task-independent semantic representations is a central goal of our work—we present our error analysis in Section 5. The pipeline for building 4lang definition graphs from Wiktionary is currently implemented for three languages: English, German, and Italian. Extending it to additional languages requires the existence of an accurate UD parser and a machine-readable monolingual dictionary of sufficient coverage. Once the UD parses of definitions are available, the remainder of the pipeline is language-independent, although for some languages it might be necessary to extend the dep_to_4lang mapping to include reference to morphological features, as done in Recski, Borbély, and Bolevácz (Reference Recski, Borbély and Bolevácz2016). We apply this pipeline to dictionary definitions extracted from data dumps of Wiktionary, a large crowd-sourced dictionary containing more than 100,000 entries for 40+ languages (and more than 10,000 entries for about twice as many). Section 3.2 will describe our method for detecting entailment between pairs of words or predicates using their corresponding 4lang graphs. Entailment relations extracted using this method may be of lower quality than those encoded by manually built databases such as WordNet, but cover larger vocabularies and thus improve performance even for languages with large WordNets (see Section 4 for details). While in our current experiments we chose to build pipelines for three relatively well-resourced languages (English, German, Italian), both a large Wiktionary and an UD parser model are available for many more.

3.2 Modeling lexical entailment

A Wiktionary page for a given word form typically contains several definitions corresponding to multiple word senses and/or parts-of-speech, but, given the crowd-sourced nature of the dataset, without adhering to any particular lexicographic principles. After identifying individual entries on each Wiktionary page using simple language-specific templates, our approach is to pick the first definition of each word unless it is explicitly marked by editors as obsolete, archaic, historical, or rare. Based on manual analysis of a small sample, we estimate that for over 98% of words in the Semeval dataset this method chooses the sense that is apparently intended in the dataset. For example, for the word pair letter–mail in the Semeval dataset, we assume that the intended sense is that defined as a written or printed communication and not a symbol in an alphabet.

The entailment candidates in the SherLliC dataset pose a greater challenge. Predicates are implicitly disambiguated by their type signatures, yet we do not attempt to select some subset of the available definitions based on this information, instead we establish entailment between a pair of predicates iff there is any pair of definitions for which the conditions of entailment, to be defined in this section, are fulfilled. In Section 4, we shall see that this does not lead to a drastic decrease of precision as it would on the Semeval dataset, likely because of the relatively higher bar of matching an argument-predicate structure as opposed to a single word. Meanwhile this enables detecting entailment based on any listed definition of predicates; for example, we correctly detect that the premise “A is releasing update for B’’ entails the hypothesis “A is releasing version of B’’ based on the third Wiktionary definition of update: “A modification of something to a more recent, up-to-date version; (in software) a minor upgrade”. We shall discuss the issue of polysemy in more detail in Section 5.

Our core method for modeling entailment is based on the intuition that all semantic relationships marked by 0-edges in 4lang graphs constitute entailment, that is dog entails not only mammal but also four-legged and bark, and that this relation is transitive, that is if dog entails bark and bark entails sound then dog must also entail sound. In other words, we equate entailment with inheritance and consequently with accessibility via a directed path of 0-edges in a 4lang concept graph. Here, we shall not discuss whether the relationship between for example the concepts dog and bark are an example of entailment, causation, correlation, or something else; this comprehensive definition is rooted in our view that modeling lexical entailment should be an enabler of the modeling of natural language inference (NLI). We note that this view on entailment may be incompatible with task formulations involving statements about specific events or scenes, such as the SNLI dataset that is largely based on image captions, since in such descriptions A dog is there should not entail A dog is barking. It can be argued, however, that it is precisely the omission of such type-theoretic details that gives the 4lang system its flexibility. Where there is a dog, there can be a bark(ing). Kornai (Reference Kornai2010) further argues that the episodic readings that have occupied Montague Grammarians ever since the inception of MG are practically nonexistent in natural language.

Figure 8.

4lang representations of A is nation in B and A  is country in B.

Implementing the above definition as a function over pairs of semantic graphs involves the recursive expansion of the premise graph based on the definitions of its defining words. This, after only a small number of iterations, leads to the proliferation of errors and ambiguities, caused primarily by imperfections of the dep_to_4lang mapping and the inherent ambiguity of definitions (see Section 5 for more details). We found it practical in our experiments to limit the depth of expansion to 2 (although in certain cases 3 would lead to a higher F-score, see Section 4 for details). On the Semeval dataset of word pairs, we then establish entailment iff the hypothesis word is present in the expanded premise graph. The depth of recursive expansion is the major tunable parameter of our method, set to 2 for both tasks based on early experimental results, but recall can be increased considerably by allowing our system to establish entailment without full coverage, requiring only some percentage of edges in the hypothesis graph to be covered by the premise graph. Optimal values for this threshold were obtained by optimizing on the development portion of the dataset. For the high-precision configurations, we set the threshold to 0.8 while for the highest F-score required a value of 0.2. Such tweaking admittedly weakens full explainability, since one has to justify a partial overlap of premise and hypothesis graphs. Still, the interpretability of our representation remains, enabling deep error analysis (see Section 5) and further development of our semantic parsing methods.

On the SherLlic task, we use 4lang graphs built from the example sentences associated with each predicate in the dataset. For example, the pair of premise and hypothesis predicates “A is nation in B” and “A is country in B” will be represented by the graphs in Figure 8. We define entailment to hold iff all edges of the hypothesis graph are found in the expanded premise graph, which requires us to refine the expansion process to also allows nodes to inherit relations along paths of 0-edges. After a graph has been extended with the definition graphs of its nodes, we allow relations to be inherited along directed paths of 0-edges using the algorithm in Figure 9 and exemplified in Figure 10, representing the modus ponens reasoning that if A is a nation in B and nation is a type of country, then A is a country in B.

Figure 9.

Appending zero edges to premise.

Figure 10.

Expanded 4lang representations of A is nation in B.

Let us now consider the premise “A is center of B” and the hypothesis “A is heart of B”. The concept heart is not accessible from the premise, but one of its definitions is The centre. In this case, we can detect entailment via the reduction mechanism, which substitutes a concept with its definition graph (while still allowing relations to be inherited by defining concepts, as in the case of expansion). In other words, reduction is equivalent to expansion followed by removal of the original concept, as shown in Figure 11. This mechanism represents the intuition that if all defining properties of some concept can be inferred then the concept itself is also entailed.

Figure 11.

4lang definition graphs of expanding and reducing A is heart of B.

On the simpler word-level task, we increase our overall performance by incorporating additional lexical resources: we extract for all premise words lists of synonyms from their Wiktionary pages as well as a list of synonyms and all their hypernyms from WordNet (Miller Reference Miller1995). We use such additional resources in two ways. First, we can choose to establish entailment if the hypothesis word is present in the set of hypernyms for any WordNet synset containing the premise word. Second, we may use synonyms of the premise word from both WordNet synsets and Wiktionary to extend the 4lang definition graph of the premise with additional concepts before performing expansion. In Section 4, we shall quantify the contribution of these additions. On the SherLlic dataset, we decided not to use these extensions as they appeared to introduce too much noise.

Our method for detecting entailment described in this section produces decisions that are directly explainable due to the presence of at least one path of 0-edges leading from the premise word to the hypothesis word, and these paths can be further explained by the dictionary definitions on which they were based. For example, the decision in the above example that “A is center of B” entails “A is heart of B” could be explained by citing the original Wiktionary definition heart: 12. (figuratively) The centre, essence, or core. Although we have not observed it during our manual error analysis (see Section 5), it is possible for our method to make a correct decision based on edges that were introduced in error, which would ultimately result in an incorrect explanation.

Our approach treats lexical entailment detection as a binary classification task and in this work we shall not attempt to give an account of entailment as a graded phenomenon, even though the Semeval dataset (see Section 2.1) provides evaluation data also for graded lexical entailment. While this dataset is validated by high agreement among annotators answering the question To what degree is X a type of Y and represents common intuitions such as that chess is a sport to a lesser degree than basketball (Vulić et al. Reference Vulić, Gerz, Kiela, Hill and Korhonen2017), we consider such distinctions more an issue of prototypicality than of implicature as such. Basketball may be uniformly viewed as more of a sport than chess, but the implications we want to employ, for example that it is a contest, a fight between participants with a winner and a loser, and that the winner is proven stronger by this outcome in the sport at hand, are the same, and they operate in a 0-1, rather than a gradient, fashion.