2.1 Term extraction

Obtaining the terminology of a domain is needed for tasks such as building/enriching lexical repositories, lexicon updating, summarisation, named entity recognition or information retrieval among others. At the same time, it is a problematic task. Two problems arise: first, a well-organised corpus of texts representative of the domain is needed and, second, how the terms of this corpus could be obtained.

Compiling a corpus is an expensive task in time and resources. Obtaining the terms included in such a corpus is also problematic. Manual processing is unrealistic, and thus automatic methods are used, although the results are not perfect.

In the case of lacking of terminological resources in the domain of interest, the use of a term extractor system could help. Usually, terms are defined as lexical units that designate concepts in a thematically restricted domain. As mentioned before, terms are useful for a number of NLP tasks. For many of them, term recognition constitutes a serious bottleneck. Since the nineties, this task has been object of research but, in spite of these efforts, it cannot be considered solved. Term extraction can be seen as a semantic tagging task as it adds meaning information to the text.

The way to tackle this task depends on the available resources, mainly ontologies and lists of terms. If these resources are not available, it is necessary to resort to indirect information of a linguistic and/or statistical nature. The results obtained in these ways are limited and therefore these tools tend to favour coverage over accuracy. The consequence is that many extractors get long lists of candidates to be verified manually. One of the reasons for this behaviour is the lack of semantic information.

Due to the lack of semantically tagged resources and as shown in Cabré et al. (Reference Cabré, Estop, Vivaldi, Bourigault, Jacquemin and L’Homme2001) and Pazienza et al. (Reference Pazienza, Pennacchiotti and Zanzotto2005), many indirect methods have been proposed to obtain the terms included in texts. Some of them are based on linguistic knowledge, as in Heid et al. (Reference Heid, Jauss, Krueger and Hohmann1996), others use statistical measures, such as ANA (Enguehard and Pantera Reference Enguehard and Pantera1995). Some approaches such as TermoStat (Drouin Reference Drouin2003) or (Frantzi et al. Reference Frantzi, Ananiadou and Tsujii1998) combine both. Linguistic methods are based on the analysis of morphosyntactic patterns, but this leads to a noisy result; furthermore, the resulting candidates cannot be scored, leaving the measure of relevance to the evaluation of experts in the domain. Statistical methods, instead, focus on detecting terms using statistical measures, often the simple frequency. Terms candidates may be ranked but again an expert is necessary to measure its actual relevance.

Recently, Lossio-Ventura et al. (Reference Lossio-Ventura, Jonquet, Roche and Teisseire2016) proposed a huge set of measures based on linguistic, statistical, graphical and web information to evaluate the termhood of set of term candidates. Some of them are new while other are modification of already known measures.

Machine learning methods have also been applied to terms extraction including, by design, both term extraction and term classification tasks. Usually, these methods require huge amount of training data. The lack of reliable tagged resources for training constitutes one of the main issues. Another issue concerns the detection of term boundaries which are difficult to learn. Some examples of these techniques are shown at Conrado et al. (Reference Conrado, Pardo and Rezende2013) and Newman et al. (Reference Newman, Koilada, Lau and Baldwin2012). Also, recent techniques like those using deep learning have been applied to the task, as shown in Wang and Liu (Reference Wang and Liu2016) and Bay et al. (Reference Bay, Bruneÿ, Herold, Schulze, Guckert and Minor2021).

A common limitation of most extractors is the lack of semantic knowledge. Notable exceptions for the medical domain are MetaMap (Aronson and Lang Reference Aronson and Lang2010) and YATE (Cabré et al. Reference Cabré, Estop, Vivaldi, Bourigault, Jacquemin and L’Homme2001). Most approaches focus on technical domains in which specific resources are availableFootnote l and term extraction is easier. As documents use to be terminologically dense, term detection is easier. A drawback of many documents in this domain (health records, clinical trial descriptions, events reports, etc.) is that often they include spelling errors, domain/institution specific abbreviations and can be syntactically ill formed.

Recently, deep learning models have achieved great success in fields such as computer vision and pattern recognition among others. NLP research has also followed this trend as shown in Young et al. (Reference Young, Hazarika, Poria and Cambria2017). This success is based on deep hierarchical features construction and capturing long-range dependencies in data. These techniques have also been applied in EHR for clinical informatics tasks (see Shickel et al. (Reference Shickel, Tighe, Bihorac and Rashidi2018) for a good review).

When several resources are available for extracting terminologies for a domain, some approaches take profit of redundancy for improving the accuracy of the individual extractors. Dinh and Tamine (Reference Dinh, Tamine, Peleg, LavraČ and Combi2011) are an example of such approaches in the biomedical domain using approximate matching for increasing its coverage.

2.2 Combining terminologies in medicine

Medical terminology is especially challenging because of its richness, high level of polysemy and lexical variants. Therefore, applications based on medical terminologies could improve their performance by taking advantage of combining different resources (Smith and Scheuermann Reference Smith and Scheuermann2011). Existing approaches to the combination of terminological resources in the medical domain include both repositories of resources, as UMLS or BioPortal, allowing the interconnection of independent resources and software facilities for a uniform access, and tool suites, as PyMedTermino, that provide software support, as wrappers, for the integration of external resources. Some examples are as follows.

UMLS Footnote m (Bodenreider Reference Bodenreider2004). Unified medical language system comprises datasets and software that brings together many health and biomedical vocabularies and standards to enable interoperability between computer systems. It includes three main sources of knowledge, although the only relevant one in this context is the Metathesaurus. UMLS is probably the richest repository of medical terminology, although it has a serious limitation: it basically is a monolingual English resource. It includes vocabularies, hierarchies, definitions, relationships and attributes but only a fraction of them involve languages other than English. More specifically, UMLS includes 216 resources but only 35% involves languages other than English. Some efforts have been carried out recently for increasing its multilinguality (Hellrich et al. Reference Hellrich, Schulz, Buechel and Hahn2015), with little success for the moment.

BioPortal Footnote n (Whetzel et al. Reference Whetzel, Noy, Shah, Alexander, Nyulas, Tudorache and Musen2011; Salvadores et al. Reference Salvadores, Horridge, Alex, Fergerson, Musen and Noy2012) is a web-based service to give access to commonly used biomedical ontologies, and 873 are currently availableFootnote o. These resources are true ontologies, richer than vocabularies or terminologies. BioPortal offers several ways of uniformly accessing its content through a REST APIFootnote p. The format for accessing the content is simple and powerful. A popular service is the NCBO annotatorFootnote q that allows selection of some ontologies for performing semantic tagging of a document. Unfortunately, the resource is basically English monolingual although a few of the included ontologies allow the so-called multilingual views. Some effort has been made to improve multilingualism in the BioPortal, for example, the roadmap proposed by Jonquet et al. (Reference Jonquet, Emonet and Musen2015), but currently these proposals are still only a project.

PyMedTermino (Medical Terminologies for Python)Footnote r (Lamy et al. Reference Lamy, Venot and Duclos2006) is a Python module for easy access to an open collection of medical terminologies. Although built for the French language and including only wrappers for French resources, it seems not difficult to extend its coverage to other languages and resources.

3.1 Defining the tagset

Selecting a tagset is a task that tries to balance the usefulness of the categories and the difficulty of the task of classification. The granularity of the tagset is the most important choice to make. For instance, should we use a unique category for clinical findings or do we have to distinguish among diseases, disorders, symptoms, etc.?

Several choices arise for defining the tagset: defining it from scratch or using categories of already existing medical ontologies, such as UMLS, ICD10 and MeSH. We decided to use an already defined tagset instead of creating a new one from scratch.

The choice of which tagset to use, like any choice, is never without controversy. In this case, the choice was Snomed-CT due to its good coverage throughout the medical field, its adequate granularity, its ease of mapping to other resources and its wide dissemination in hospital settings. For this last point, it is worth mentioning that it is used in a number of relevant medical institutions around the world (Mount Sinai and Mayo Clinic in the USA, Hospital Clínico in Europe and HIBA in South America among others).

Table 2 shows the full tagset used in this task together with a short description of each category, taken from Snomed-CT documentationFootnote s.

Table 2. Snomed-CT semantic categories

3.2 Lexical datasets

The lexical resources we have used for our task can be classified along several axes:

• Regarding the organisation of the resource, we include ontologies, thesauri, lexical databases, lexicons, vocabularies, terminologies and annotated corpora.

• Regarding the domain scope, we can consider general, medical and sub-medical specific resourcesFootnote t. WN, WP and DBP are examples of the general category, Snomed-CT, UMLS, MeSH and Galen belong to the medical category and FMA, DrugBank, ICD10, CIE and CIM belong to the latter.

• Regarding the languages covered, we distinguish between monolingual and cross-lingual resourcesFootnote u. Examples of monolingual resources are MeSH, Galen, FMA and DrugBankFootnote v for English. WN, DBP, Snomed-CTFootnote w and ICD10Footnote x are cross-lingual.

3.3 Datasets used in developing MHeTRep

The resources we have are listed in Table 3. Most of them are monolingual; therefore, the column ‘Language’ contains a single language. In the case of multilingual resources (WordNet, Orphanet, DBP) the different languages and sizes are presented in separated rows. In some cases (DrugBank, RadLex) most of the terms correspond to a main language, but there are also small amounts of terms in other languages. In these cases, we include in the ‘language’ column all the languages, with the main one in bold, and we present in the ‘size’ column the summation of terms in all the languages. We give in what follows some details on the resources.

Table 3. Resources used in the work

SNOMED Clinical Terms. Snomed-CT (Donnelly Reference Donnelly2006) is a systematically organised collection of medical terms providing codes, terms, synonyms and definitions used in clinical documentation and reporting. It is considered to be the most comprehensive, multilingual clinical health care terminology worldwide. Snomed-CT provides the core general terminology for electronic health records. It includes terms for all the classes in our tagsetFootnote y. Snomed-CT, with more than 350,000 concepts, 950,000 English descriptions (concept names) and 1,300,000 relationships, is the largest single vocabulary ever integrated into UMLS. As mentioned in Section 3.1, we have used the top classes of Snomed-CT as the tagset for classifying our terms.

DrugBank. This resource (Wishart et al. Reference Wishart, Knox, Guo, Shrivastava, Hassanali, Stothard, Chang and Woolsey2006) is a database containing information on drugs and drug targets. DrugBank combines detailed drug (i.e., chemical, pharmacological and pharmaceutical) data with comprehensive drug target (i.e., sequence, structure and pathway) information. We have used version 5.0.11 (Wishart et al. Reference Wishart, Feunang, Guo, Lo, Marcu, Grant, Sajed, Johnson, Li and Sayeeda2017) containing 11,002 drug entries. Additionally, 4910 non-redundant proteins sequences are linked to these drug entries. From the database fields, we have used those containing lexical information, namely the DrugBank ID, the Name, that is, standard name of a drug as provided by its manufacturer, Brand names, commercial names used by drug manufacturers, Synonyms and ATC Code.

CIMA. Clasificación Internacional de MedicamentosFootnote z is a resource on drug information in Spanish. CIMA supports coding of several tags describing its entries. The tagsets include ATC codes, forms of presentation, forms of administration, laboratories, containers, excipients, doses, quantities, units, etc.

The denomination of CIMA entries is extremely low level and, so, some higher level denominations of the products have to be derived from the original entry. For instance, a CIMA entry is ‘SERACTIL 400 mg COMPRIMIDOS RECUBIERTOS CON PELICULA’Footnote aa.

From this entry, we obtain as well ‘SERACTIL 400 mg COMPRIMIDOS’, ‘SERACTIL 400 mg’, ‘SERACTIL 400’ and ‘SERACTIL’. In this denotation, ‘400’ has been recognised as a <QUANTITY>, “mg” as a <UNIT>, “COMPRIMIDOS” as a <PRESENTATION> and “RECUBIERTOS CON PELICULA” as a <FORM>. Using these semantic tags as terminals, we have manually built a regular grammar for analysing the decomposition of the denomination. The whole grammar consisted of 67 rules. Obviously, the new terms generated by the grammar are not error free, that is, there could be false positives. The design of the grammar was, however, very conservative and the ratio of false positives is below 2%.

FMA. Foundational Model of Anatomy (Rosse and Mejino Reference Rosse and Mejino2003; Mejino et al. 2003) is ‘an ontology of biomedical informatics concerned with the representation of classes or types and relationships necessary for the symbolic representation of the phenotypic structure of the human body’.

Although created in Protégé, there are ontological representations available in OWL or RDF/XML (Noy et al. Reference Noy, Musen, Mejino and Rosse2004). These representations are complete but contains information useless for our purpose. BioPortal provides a simplified CSV table of FMA’s ontological terms. This flattened representation is very suitable for our purposes. To further improve the utility of this tabular FMA representation, Michael HalleFootnote ab has converted the CSV format into an SQLite database. This is the version we have used in our project.

MesH. Medical Subject Headings (Lipscomb Reference Lipscomb2006) is a controlled vocabulary for indexing and searching biomedical literature. Its main purpose is to provide a hierarchically organised terminology for the indexing and cataloguing of biomedical information. MeSH terms and subheadings are used to indicate the topics of an article. Currently, there are 28,489 descriptors in MeSH 2017, the version we have used.

Galen. Generalised Architecture for Languages, Encyclopedias and Nomenclatures in medicine common reference model (Galen CRM) is a part of the GALEN project (Rector et al. Reference Rector, Qamar and Marley2009). It is the pioneer of the use of formal logic in biomedical terminologies (de Freitas et al. Reference de Freitas, Schulz and Moraes2009). It is organised as a set of is-a hierarchies containing overall about 25,000 nodes (concepts) using 26 link types (relations) (Corsar et al. Reference Corsar, Moss, Sleeman and Sim2009). OpenGALEN uses GRAIL, a description logic based language (Galen Representation And Integration Language). Implementations in OWL and RDF are also provided.

ICD-10. The International Classification of Diseases, revision 10, is the most widely used version of ICD. It is translated into 42 languages. Its nodes are structured into 22 chapters (Infections, Neoplasms, Blood Diseases, Endocrine Diseases, etc.) and denote about 14,000 classes of diseases and related problems. It is splitted into ICD-10-CM for diseases and ICD-10-PCS for procedures. ICD-10-CM codes are organised hierarchically, where top-level entries are general groupings and bottom-level codes indicate specific symptoms or diseases and their location. ICD-10-PCS contains procedure codes organised as a taxonomy of up to 7 levels with 17 codes at first level.

WordNet. Due to its success, although it has been initially developed for English, wordnets for many other languages have been built. Such borders are defined as those synsets whose (direct/indirect) hyponyms belong (with some confidence degree) to the medical domain while their hypernyms are outside this domain. Therefore, such synsets constitute a border between in domain and out of domain areas in this resource. For example, there is a synset in MCR that localises as ‘body substance’, this implies that all its hyponyms are all different types of substance located in the human body.

In this way, we obtain all the existing nouns in the medical domain. But it is also possible to obtain related words using other relations in MCR. We obtain adjectives following the relation ‘pertains_to’ and verbs using the relation ‘related_to’. The former relation allows to relate the adjective ‘bronchial’ with ‘bronchus’ while the latter connects ‘injection’ with the verb ‘to inject’.

RadLex. Radiology reports are probably the most studied type of clinical narrative. They contain terminology not only from the medical domain but also from the domain of imaging and graphical software. The coding system of RadLex is the Index for Radiological Diagnoses (ACR Index) that offers both anatomic and pathological identifiers.

Orphanet. Orphanet (rare diseases and orphan drugs) is a reference European resource initiated by the INSERM (French National Institute for Health and Medical Research) and devoted to gather and improve knowledge on rare diseases and orphan drugs. It aims to improve the diagnosis, care, treatment and information for patients with rare diseases.

Although many of the entries of the list of rare diseases are already mapped to Snomed-CT, our interest in this resource is to provide completeness and mapping to different languages and sources.

DO. The Disease Ontology (DO) has been developed as a standardised ontology for human diseases. DO integrates disease and medical vocabularies through extensive cross mapping of DO terms to MeSH, ICD, Snomed-CT and other resources. In fact, the reason for including this ontology into MHeRep is not the coverage of the diseases, satisfied enough by other resources but the very rich coverage of mappings into other resources. For instance, in Table 4 the entry angiosarcoma has one main DO identifier, two alternate DO identifiers and seven mappings to other ontologies. DO contains currently 8043 human diseases.

Table 4. Fragment of the entry angiosarcoma within DO ontology

Figure 1. Compilation methodology.

4.1 Process of acquisition of original sources

As mentioned above, the first phase of our process consists of the acquisition and classification of vocabularies coming from our original resources.

We briefly described in Section 3.3 the datasets we have used for this step. Table 2 shows a summary of each resource. Then, in Section 4.1, we describe the acquisition and classification processes from these datasets.

The initial datasets, that is, those that can be obtained and classified without any additional support, can be carried out with simple procedures. Next, we face two challenging tasks: the acquisition and classification of MeSH and Galen. These two datasets pose a number of difficulties that are addressed below in this section.

Snomed-CT:. We performed the task for English, Spanish and Catalan. The acquisition task was straightforward because we had the ontologies of the resource and from them locating the lexical nodes was easy. For classifying the lexical items, we navigated the ontology from the lexical nodes through the hypernym edges until reaching the top classes of the ontology. Ideally, the organisation should be a taxonomy but this was not always the case, sometimes a node had more than one parent and, so, the structure was a bit tangled. In these cases, we incorporated the ambiguous lexical items into all the accessible top classes.

DrugBank:. Next, we incorporated DrugBank. The process was also simple. We obtained the lexical items corresponding to names, synonyms, brand names, etc. Although the database is basically for English, some synonyms are tagged with other languages (French, German or Spanish). All the collected items were directly classified as a ‘Pharmaceutical/biologic product’.

CIMA:. As mentioned in Section 3.3, the acquisition of the terms from CIMA needs the generation of partial denotations using a regular grammar. The rest of the tasks (extraction and classification) are simple.

FMA:. We accessed the FMA-lite representations using the sqlite3 python moduleFootnote ac.

ICD:. The process of incorporation of English ICD-10 and its French and Spanish counterparts is straightforward. ICD-10-CM for diseasesFootnote ad and ICD-10-PCSFootnote ae for procedures are processed in the same way.

DO:. As mentioned above, we selected DO not for enriching our terminologies but for the richness of mappings from each term into other resources. The data are encoded into lines consisting on a tag and a value. As all the terms are diseases, the task is easy.

Orphanet:. The data are directly available on the web for each language as a simple list of terms although not all the information is directly downloadable. To include these data in our resource was simple because we know in advance that all terms should be classified as ‘Clinical finding’.

RadLex:. We have used the RDF version of the resource. From the 15 top categories occurring in it, we reduced our obtention process to the classes shown in Table 5 that are both relevant and easy to map to our tagset. We got an ontology of 850,631 triples. We selected all the nodes of type LEXICAL: 390,556 nodes from which 252,075 are literals. There are 98 relations defined in the ontology but we only use those supporting taxonomic links such as ‘subclass’ and equivalence links ‘equiv’ and those related to naming, that is, ‘name’, ‘preferredName’, ‘preferredNameGerman’, ‘synonym’, ‘synonymGerman’ and ‘umlsTerm’. There are also some useful properties attached to nodes, such as the ‘language’ property.

Table 5. Top categories of RadLex mapped into our tagset

Although RadLex is basically an English resource, there is a small set of links to gain access to the translation to other languages. The coverage is (1) English: 109,763, (2) German: 46,560, (3) Latin: 4123, by default assigned to all languages having the Latin script, and (4) none: 91,629, by default assigned to English.

MeSH:. MeSH is a knowledge structure organised from the skeleton of a taxonomy associated with the MeSH Headings. Table 6 presents the top level of the MeSH Headings. Table 7 shows a fragment of this taxonomy. Beyond the MeSH Headings, entries contain a huge number of properties that can be used for classification. The set of allowable qualifiers can be consulted at the MeSH siteFootnote af. Selecting the MeSH entries can be done straightforwardly but mapping the MeSH entries into our tagset is far from easy because of two issues:

Table 6. MesH descriptor categories

Table 7. MesH Heading examples: fragment of the taxonomy headed by nervous system

If we look at Table 6, it seems intuitive that some top (one character) MeSH headings can be directly mapped into Snomed-CT top categories.

Mapping ‘A’ (‘anatomy’) into ‘Body structure’, ‘B’ (‘organisms’) into ‘Organism’, ‘C’ (‘diseases’) into ‘Clinical Findings’ or ‘D’ (‘drugs and chemicals’) into ‘Pharmaceutical/biologic product’ seem to be quite safe but, unfortunately, this is not always the case. We have computed the distribution of MeSH terms already classified in our vocabularies from the resources applied so far. There are 112 terms having MeSH Heading starting with ‘A’. Of these, 50 have been classified as ‘Substance’, 43 as ‘Body structure’ and the rest fall into up to 6 other categories. So, classifying all ‘A’ as ‘Body structure’ will fail 57% of the cases. Better results are produced in cases ‘B’ (from 1130 terms, 1071, that is, 95%, have been classified correctly as ‘Organism’) and ‘C’ (from 1483 terms, 1414, also 95%, were correctly classified as ‘Clinical finding’) while in case ‘D’ from 5584 terms, the majority, 3224, have been erroneously classified as ‘Substance’ and only 2354 correctly as ‘Pharmaceutical/biologic product’. For the other cases, but ‘F’, the results are also bad. So another approach should be followed.

We conducted experiments using the second and third levels of the MeSH Headings, and some cases were solved satisfactorily but not all. So, we decided to learn a set of 19 binary classifiers (one for each class). After running the binary classifiers, a voting mechanism was activated for getting the final class if possible. The candidates that could not been assigned were simply rejected. So the number of terms in our datasets coming from MeSH is smaller than the full coverage of this resource. We admit this drop on coverage for assuring an accurate assignment to the tagset. A similar limitation occurs in the case of Galen.

We used as classifier a simple multilayer perceptron followed by a SoftMax layer. The classifiers were implemented using the Keras environmentFootnote ag. The learning features consist of the extremely rich information contained in MeSH entries, while the supervision is provided by terms present in MeSH and already classified by previous likely accurate sources. The positive examples for each class are those classified in the corresponding class while the negative examples are randomly chosen from those classified in the other classes. The proportion of positive and negative examples is maintained.

Galen:. This source is organised into 83 files (chapters). We have used the RDF implementation of the ontologyFootnote ah. Each file contains a separate ontology but we have integrated all of them into a single one. We have used the ‘Lexical’ nodes for obtaining the terms and the others for navigating the ontology. From the various relations included, we have used the ‘subClassOf’ for navigating the taxonomic links, and ‘EquivalentClass’ for synonymy links. An issue for selection is that the lexical items have a peculiar notation. For instance, ‘presencewhichisExistenceOfHyperthyroidism’ should be normalised into ‘Hyperthyroidism’ and mapped into ‘Clinical finding’. For performing the normalisation, we manually built a set of about 100 regular transformation rules. The performance of this grammar is not error-free but, by favouring precision over recall (terms occurring only once are rejected), we have limited the ratio of false positives to be under 2%.

For learning the classifier, we built a system similar to the one described for MeSH using all the already classified terms for learning, including MeSH terms in this case. As Galen entries do not have the rich property framework of MeSH, we used as features the original and normalised forms of the entry and its local environment (edges and nodes adjacent to the one to be classified, the hypernymy path from the node to the top of the hierarchy and the file from which the node has been selected).

4.2 Multilingual extensions

We performed monolingual and multilingual extensions using DBP, WN and WP. Below, we briefly describe these processes.

5.1 MHeTRep implementation

All the system has been built using Python. The whole MHeTRep repository is represented as an instance of the class VOCABULARY_COLLECTION that can be loaded from and saved to a Json fileFootnote aj.

From the VOCABULARY_COLLECTION, a specific vocabulary for one language and one category can be extracted. Each of these vocabularies is an instance of the class VOCABULARY allowing querying for the existence of a term. Each term existing in the VOCABULARY has associated an instance of the class LEXICAL_ENTRY, containing information of the sources, coding, mappings to other languages, etc.

Managing of these information includes querying, updating and deleting term entries. To allow updating of the data, in case of new versions of the resources, a simple mechanism is also provided that allows to incorporate to MHeTRep the tagged terms included in a text file.

6.1 Approximate matching

A basic component of systems for processing biomedical documents is the recognition and tagging of biomedical entities. For this task, many lexical resources are available and our claim is that the one presented here could be useful. A main problem is that some entities may have a complex word form. For instance, the following entries have been randomly extracted from our terminology coming from the MeSH ontology.

• 2-(hydroxymethyl)-N,N-dimethylpiperidinium benzilate (classified as ‘Pharmaceutical/ biologic product’)

• Interstitial cell of Cajal like cells (a ‘Body structure’)

• Deficiency disease, betalipoprotein (a ‘Clinical finding’).

It is extremely unlikely that these forms occur verbatim in biomedical texts. Additionally, (partial) abbreviations and spelling errors are frequent in free text like EHRs, that are the main source of documents for using our terminologies. Also inflectional (as plural forms) and derivational (as adjectival) variants frequently occur. So, some kind of approximate matching is needed.

We have examined the histogram of the length of the terms in the vocabulary of ‘Body structure’ for English. Figure 5 shows the distribution of length of the terms in characters with a maximum around 40 characters. The distribution of length of the terms in tokens shows a similar shape with a maximum around 5 tokens, so most of the terms are multi-word.

Figure 5. Histogram of atomic terms length in tokens.

The MHEtrep includes 12,439 atomic tokensFootnote al. The type/token distribution of the atomic terms is shown in Figure 6, note that the x-axis is logarithmic. The distribution is extremely skewed, close to a Zipfian. 3690 (30%) of the terms occur only once, 9215 (74%) occur less than 10 times and 11,429 (92%) occur less than 100 times. In the other extreme, the terms ‘right’, ‘structure’, ‘left’, ‘nerve’, ‘entire’ and ‘of’ occur more than 15,000 times.

Figure 6. Type/token distribution of the atomic terms.

With the character and token histograms in mind, we have built an approximate matching detailed in this section.

For implementing an approximate matching system, a simple exhaustive access to the whole content of the vocabularies is too costly in time. A reasonable alternative could be representing the whole vocabulary (or a fragment) as ST, as shown in Figure 7. This compact representation allows an efficient access to both exact and approximate matching.

Figure 7. Vocabulary representation in our approximate matching system.

For this purpose, we used the package ‘ST’Footnote am. Our implementation of approximate matching using ST is very simple because we are interested in this task only for the purpose of evaluation. It is worth noting that replicating the algorithm is easy for a specific resource. Our advantage is that we can apply the algorithm over a ST representation of the whole collection and the alternative could be using different representations for each resource.

For a given language and a semantic category, we built three vocabularies: the vocabulary of simple terms (mono-word), the vocabulary of complex terms (multi-word) and the vocabulary of atomic terms (atomic components of complex terms). For all these vocabularies, we built a corresponding ST. Over this infrastructure, we built a set of approximate extraction rules that we summarise below.

• Look for approximate extension: We obtain all the character n-grams (larger than 2) in the term candidate. We look, then, in the simple and complex STs, for all the occurrences of each n-gram. For each occurrence found, we obtain the containing word and we compute the string distance between these words and the candidate. For this purpose, we used the python-string-similarity libraryFootnote an. We selected as approximate matched the cases where the distance is below a thresholdFootnote ao. An example of application of this rule is: beta-adrenergic blocking agent $\leftarrow$ beta adrenergic blocking agents

• Look for approximate shrink: We look for the term candidate in the simple and complex STs. For all the fragments found, including a full word, we compute the string edit distance (Levenshtein distance) between these words and the candidate using the same thresholding mechanism. An example is fluorouracil $\leftarrow$ 5-fluorouracil.

• Compatibility checking: As an alternative to string similarity, we performed a compatibility checking between the term candidate and the terms in the ST. For this checking, a pre-process of lemmatisation, morphological analysis and semantic tagging has been performed using SpaCyFootnote ap. This checking consists of the sequential application of decreasing confidence compatibility rules, some of them requiring additional processes for their application, such as lemmatisation, WordNet linking and others. For instance, when the word forms are identical the compatibility is 1, when the two lemmas are identical the compatibility is 0.9 and when the difference is due just to the capitalisation there is an additional penalty of 0.05. Other rules check the WN overlap, the semantic distances in WN, the local hypernymy, hyponymy, and sibling relations in WN, etc. Some of the rules are language dependent. An example involving two variants of the same synset is blocker $\Leftrightarrow$ blocking agent.

• Removing diacritics: In some cases, candidate terms lack diacritics and as the vocabularies contain diacritics the matching is more difficult or impossible. We have implemented a rule for facing this issue. An example in Spanish is corazón $\Leftarrow$ corazon (heart)

• Number variations: When one of the terms to compare is singular and the other plural. An example is antimuscarinic $\leftarrow$ antimuscarinics

• Stop words removing in MW: sulfate atropine $\leftarrow$ sulfate of atropine.

• Change of order in atomic components of a MW: h1 antagonist $\leftarrow$ antagonist h1.

• Atomic components of a MW remotion.: h1 type antagonist $\leftarrow$ h1 antagonist.

The use of approximate matching substantially increases the coverage of the terminology for term look up but, obviously, the process is not error free as we discuss in Section 6.4.

6.2 Intrinsic evaluation

Our terminology is built from a set of resources, as shown in Table 3, that support some degree of confidence on the two tasks involved, extracting the terminology and classifying each term into the appropriate category. As an individual term could be incorporated from more than one source, our intuition is that the number of sources (its support) is a good indicator of our confidence of this term for the corresponding category.

Consider, for instance, the case of the English language and ‘Body structure’ category. There are 256,179 terms in this vocabulary. The histogram of the number of sources per term is shown in Figure 8.

Figure 8. Histogram of sources per terms for English Body structure.

Although the shapes of the histograms for different languages and categories have some variations, the general distribution is similar to the one shown in Figure 8. In Table 11, we present the integrated (summing up all categories) results for all the languages. There is a good correlation between the distribution for English/Body structure and the global results in Table 11.

Table 11. Distribution of sources per term for all the terms languages and categories in MHeTRep

Although the distribution of sources is highly skewed towards terms coming from a unique source, the number of terms supported by more than one resource (almost 10%) assures a good confidence.

6.3 Extrinsic evaluation

To evaluate the content of a resource such as MHeTRep, we need one or more reference repositories or external material already validated. As the first possibility is not feasible, this evaluation was done against external resources (vocabularies, medical documents, contests material, etc.) for some languages and semantic categories.

Obviously, this evaluation covers only some of our cases. We have selected different sources to try to cover as many languages and semantic categories as possible. In most of cases, the sources correspond to learning and test material for NLP contests. In the late case, it is worth noting that we are not interested on the results of the task, but only on the list of medical terms validated by the organisation without discussing their validity or trying to do another term extraction task.

So, for each test we compute (1) the number of terms correctly extracted and classified using EXACT matching on MHeTRep, (2) the same using APPROX matching and (3) the coverage using only the best individual resource. Our claim is that (3) should be overtaken by (1) and with great difference by (2). We have obtained the list of terms for each of the available resources and we have looked for these terms in MHeTRep. The search is done for EXACT matching and, in case of failure, for APPROX. The following sections briefly describe each of the sources used and assessments made.

6.3.2 MIMIC

MIMIC-IIIFootnote ar is a publicly available EHRs collection (Lee et al. Reference Lee, Scott, Villarroel, Clifford, Saeed and Mar2011; Johnson et al. Reference Johnson, Pollard, Shen, Lehman, Feng, Ghassemi, Moody, Szolovits, Celi and Mark2016), implemented as a database comprising deidentified health data associated with 40,000 critical care patients. For our test, we have selected only the tables containing descriptive fields:

1. (1) D_ITEMS: Dictionary of local codes appearing in the MIMIC database.

2. (2) D_LABITEMS: Dictionary of local codes related to laboratory tests.

3. (3) DIAGNOSES_ICD: Diagnoses, coded using ICDFootnote as.

4. (4) DRGCODES: Diagnosis Related Groups (DRG).

5. (5) PRESCRIPTIONS: Medications.

6. (6) PROCEDURES_ICD: Patient procedures, coded using ICD-PRM

Some extra comments are needed for this test. For each of the MIMIC tables, a category is expected. In some cases as ‘D_ICD_DIAGNOSES’ and ‘D_ICD_PROCEDURES’ what is expected is what is found (a ‘Clinical finding’ and a ‘Procedure’, respectively). In other cases, however, due to the different criteria used by MIMIC annotators and Snomed-CT ones the obtaining process is more problematic. To face this issue, we decided to detect the terms by EXACT matching in three alternative steps: (1) looking for the expected category, (2) looking for the pair of more likely categories and (3) looking for all the categories.

The results of our EXACT matching process are presented in Table 12. Column 1 shows the index of the MIMIC database table (shown above), column 2 its size (normalised unique terms), column 3 the expected category, column 5 the alternate categoryFootnote at and columns 4, 6, and 7 present the number of terms found for the three steps (expected category, first and second, all). We include the percent separated by a slash. Row 7 accounts for the overall results. Globally, MHeTRep contains 9699 terms from 46,757 existing in MIMIC (i.e., our coverage is 21%). Columns 8 and 9 present the best individual resource and its coverage. The most useful resource proved to be ‘Snomed-CT’ with an overall contribution of 5372 terms (12%) clearly above our results.

Table 12. Result of MIMIC experiments using EXACT matching

Then, we performed an APPROX matching search whose results are shown in Table 13. The overall results are that 62% of the pending terms are recognised, an excellent figure.

Table 13. Result of MIMIC experiments using APPROX matching

6.3.3 BARR acronym recognition contest

Acronyms have a high reference value, as they act as reference anchors of textual context most of the time. Because of this and the common problem of recognition of abbreviations, acronyms and symbols, and their disambiguationFootnote au, the Biomedical Abbreviation Recognition and Resolution (BARR) track was developed in 2017 (Intxaurrondo et al. Reference Intxaurrondo, Pérez-Pérez, Pérez-Rodríguez, López-Martín, Santamaria, de la Peña, Villegas, Akhondi, Valencia, Lourenço and Krallinger2017) within the framework of the Ibereval CampaignFootnote av. The task covered two languages, English and Spanish, and consisted on detecting short forms and long forms of acronyms or abbreviations occurring in EHRs. We have selected the learning data of this competition for testing our system. The recognition task is challenging, especially for recognising short forms.

The training data consisted of 525 documents containing 2959 annotations. Although the types of annotation consisted of 10 tags, we have reduced the cases to ‘short’ and ‘long’ annotations corresponding roughly to acronyms or abbreviations and their expansions. Results are presented in Table 14. Note that the 2959 annotations correspond to mentions (tokens) in the documents while the figures in column 2 of the table correspond to unique terms (types). Column 3 shows the figures of EXACT matching. The figures and the total percentage (19%) seem a bit low compared with the other tests, mainly due to the difficulty of the task for short forms. The distance between our full terminological resource (column 3) and the one for the best source (column 5) is maintained consistently.

Table 14. Results obtained on BARR test

The best source is MCR for the two English types and MCR and Orphanet for the Spanish case. The figures for APPROX matching are presented in column 6. The results are good, in fact the loss of EXACT match in this test is recovered by APPROX match. We have added a column combining the two matching for making clear this fact. Note that the results for short forms are consistently better than those for long forms for both languages (about 20%) while results for English are consistently better than those for Spanish for both short and long forms (about 15%).

6.3.4 Clef 2015 and 2016 contests

The CLEF Initiative (Conference and Labs of the Evaluation Forum, formerly known as Cross-Language Evaluation Forum)Footnote aw focuses on multilingual and multi-modal information with various levels of structure.

CLEF eHealth offered challenges addressing several aspects of clinical information extraction including NER, normalisation and attribute extraction. Starting in 2015, the lab’s IE challenge evolved to address lesser studied biomedical texts in languages other than English; in the 2015 and 2016 editions the language studied was French.

The dataset used in these competitions is the QUAERO French Medical Corpus. It was developed as a resource for NER and normalisation and fully described in Névéol et al. (Reference Névéol, Grouin, Leixa, Rosset and Zweigenbaum2014). It includes a selection of MEDLINE titles and EMEA documents. The entities that were manually annotated are Anatomy, Chemical and Drugs, Devices, Disorders, Geographic Areas, Living Beings, Objects, Phenomena, Physiology, Procedures. All these categories were easily mapped into our tagset.

In Table 15, we present the results. Figures separated by a ‘/’ represent the absolute number and the percentage in relation to the total. There is a high degree of correlation between all the figures for both years. Performing EXACT matching, we extracted a 25.4% (and 27.3%) of the terms. 364 (390) new terms, that is, 15% (15.8%) were located by APPROX matching. There is a high degree of correlation between the results obtained in both years. Performing EXACT matching, we extracted 25.4% (and 27.3%) of the terms. 364 (390) new terms, that is, 15% (15.8%) were located by APPROX matching. These figures clearly outperform the coverage obtained by the best source alone (cross-lingual links from English, Spanish and German).

Table 15. Results using the medical terms defined in the Clef eHealth competition

6.3.6 Structured discharge data from HIBA

This set of terms has been compiled from a set of 2819 HIBAFootnote ay discharge notes. HIBA is a reference hospital in Buenos Aires closely involved in Snomed-CT implementation.

The results are presented in Table 16. From 4827 unique terms, 1113 have been recognised by EXACT matching (23%). This figure is in line with the corresponding figures in other datasets. From these 1113 terms, 868 (18%) come from one source: Snomed-CT. If we do a stratified selection, we can see that only 35 terms constitute 25% of all terms present in the collection. From this quantity, more than 60 are found in MHeTRep. These figures are relatively high. In fact, the terms come from 11 different contributors, most of them coming from ‘Snomed-CT’ (the second contributor, ‘DBPedia_es’, provides only 439 terms). The APPROX matching procedure allows the detection of 936 terms. The combination of these facts provides a reasonable explanation of the figures. APPROX and EXACT match contribute similarly, coming both mainly from the ‘Snomed-CT’. So, the conclusion is that the annotators of the documents seem to accurately follow the ‘Snomed-CT’ terminology.

Table 16. Results using the medical terms collected from medical discharge reports

6.3.7 SemEval 2017 task 3 subtask D (Arabic community question answering)

This contest refers to Community Question Answering on medical fora for Arabic. Although Arabic has a relatively poor coverage in our terminologies, we think that it can be useful to perform a test on this language also. Additionally, the data for this task come from a challenging genre, medical fora. The SemEval Task 3 subtask D, (Nakov et al. Reference Nakov, Hoogeveen, Màrquez, Moschitti, Mubarak, Baldwin and Verspoor2017), asks, given a query, consisting of a question in Arabic, and a set of question–answer pairs (for each query, 30 for learning, 9 for testing), (1) to classify each pair as relevant or not regarding the query and (2) to re-rank the question–answer pairs according to their relevance with respect to the original question. An important issue we have to face is that this material lacks annotations of medical entities.

Thus, for selecting the candidate terms, we extracted all the word n-grams (up to 5-grams) satisfying termhood conditions, that is, combinationsFootnote az as N, NN, JN, …, for English or N, NN, NJ, …, for Arabic. Note that the usual approaches to this task are based on computing the semantic similarity between the query and the query and answer of the thread pairs. Locating the medical tokens in these strings is the core task of these approaches.

As we have of the Arabic texts for all the questions, and questions and answers for all the pairs as well as the English translations using Google Translate APIFootnote ba we will test the coverage for both languages in order to compare the results. Results are presented for both languages and semantic classes considered in Table 17.

Table 17. Results on SemEval 2017 test (BLE: Bilingual extension, BC: Best count)

The third column of Table 17 reflects the sizes of the word n-grams vocabularies. These vocabularies, however, contain many out-of-domain n-grams. In order to measure the degree of coverage of our terminologies, we have estimated the proportion of true medical terms within the sets of candidates. We have carried out this estimation by manually examining a set of 700 sentences (for each language) from our corpus. The sizes of these estimations are presented in column 4 of the table.

Next columns present the number of exact matches, approximate matches and overall matches for each semantic class and language. The best source and its results are shown in columns 8 and 9.

It is worth noting that the best sources for Arabic correspond to DBPs for English and French. The results for Arabic, 441 exact matches (14.1%), 1025 for approximate matches (32.8%), totalling 1466 (46.9%), are similar to the obtained in other tests. However, the results for English, are very low. The reason for this is probably due to the bad quality of the translations of the Arabic original texts, especially for some semantic classes.

In order to evaluate the use of MHeTRep in SemEval Task 3 subdues D, we have slightly modified our algorithm by including the Arabic vocabularies of MHeTRep. The original algorithm is described in Adlouni et al. (Reference El Adlouni, Lahbari, Rodriguez, Meknassi, El Alaoui and Noureddine2017)Footnote bb. In Table 18, we present the official result of the primary submissions (three first rows).

For improving the results using MHeTRep, we have looked for the occurrences of terms in our vocabularies in each of the 1400 queries in the test dataset. From the 1400 queries, 1239 include at least one term of MHeTRep. For these queries, we looked for all the recognised terms into the question/answer pairs in the thread. 6154 matching were obtained (on average 5 terms per query). We have added a new feature to our algorithm for learning a classifier consisting on the size of the intersections of the sets of MHeTRep terms in the original query and the candidate pair. After running the updated algorithm, we obtained an improvement in two metrics (MAP, 2.43%, accuracy, 2.88%).

Table 18. Official results of the task and improved version

6.4 Discussion on extrinsic evaluation

From the evaluation presented above, some general considerations can be done:

1. (1) MHeTRep has been tested using mainly sources where the terminological strings have been already identified (all but the SemEval 2017 cases). Such sources have been compiled from different origins including medical articles, discharge notes, EHRs and contest materials. Some issues can be identified:

   1. − In some cases, the reliability is not guaranteed as they has been tagged by the contest organiser. Data obtained from Tass 2018, for example, has been obtained from text with a low specialisation level (e.g. enfermedad -disease-, resultado -result-, …). Some of the validated terms may not even be considered medical terms (e.g. proveedor -provider-, insecticidas -insecticides-, pintura -paint-, …).

   2. − Sometimes the terms contain spelling errors due to the process of building. See for example: sospecha de sepsis or úlceras por presion en el talon; such strings contains extra words or spelling errors making its identification impossible or they can even contain more than one term.

2. (2) Most of the resources used to compile our collection include just specific items, not their super-classes. For example, in evaluating DDI it is possible to observe that most of the unrecognised terms correspond to generic drugs (e.g. adrenergic bronchodilators, alpha glucosidase inhibitors and ergot alkaloid class among many others). Obviously, they are chemical compounds widely used in medicine but the resources analysed to build our term collection only include specific compounds not their super-classes. A similar issue appears with the disease names, if such a name correspond to a group of diseases it can happen that the generic name is not included in such a resource. See, for example, cancers digestifs or maladies du foie in French and infecciones urinarias and problemas de salud in Spanish.

3. (3) We also used the WP as a resource to build our collection but it causes a problem: it uses a redirection mechanism to direct the user to some kind of standard terms. But it happens that this mechanism is ambiguous because it is also used to correct some user spelling mistakes (ex rinon redirects to riñon -kidney-). It is also used to resolve ambiguities (remember the typical “bank” example). For this reason, we do not use this information to collect terms; therefore a string like fat-soluble vitamin that is redirected to vitamin is not included in our collection.

Regarding our approximate matching algorithm, it should be noted that although this technique is useful for recognising terms that have not been correctly spelled, but some nonrecognition can occur. The algorithm works correctly when it meets terms where an accent is missing, the term is in the plural or there are simple errors in the order of the characters. But we cannot understand that all errors are corrected; it may happen that misrecognition occurs; that is, the recognised term may have a dubious relationship with the original term. Table 19 shows some examples of these situations.

Table 19. Examples of approximate matching results. (all are English terms come from DDI, examples 11 and 12 is a Spanish example and example 13 in a French example

2 and 8 are examples of misspelled terms although our algorithm successfully solves them. The first needs only one insertion and reaches a score 0.91. The second has a score 0.9 and requires one insertion and one modification but the length is higher.

4 and 5 correspond to misdetection: The term under evaluation is a direct hypernym of the term saved in our collection. It requires the removal of d and a and two deletions.

1 and 7 are errors probably because the right terms are not included in the vocabularies. The term vitamin d 3 exists in the vocabulary, vitamin b3 does not exist but it should be included. The same observation can be applied to h1 and h2 as modifiers of antagonist.

9 and 10 are examples of negated terms. In 9 the negated term exists in the vocabulary but not the affirmed one. In 10, the situation is the contrary. When the negated term occurs in the vocabulary the affirmed term should occur as well and has to be included. When only the affirmed term occurs the inclusion of the negated term is dubious, the negation of a term probably is not a valid term, this situation should be carefully analysed by a terminologist. 3 and 6 correspond to cases of terms corresponding to words that are similar to other terms included in our vocabulary. As a matter of fact, dermax correspond to just a trademark of a therapeutic shampoo that has been wrongly tagged as a term.

In example 3, articaine hydrochloride corresponds to an anaesthetics not included in our vocabulary but whose spelling is similar to a chemical compound included in the vocabulary with a quite different purpose.

11 corresponds to a term not found (nacimiento bebé) in spite of the fact that very similar terms are included (nacimiento de un niño) in Snomed-CT and may be considered as a variant of the same concept. In this case, it seems clear that the terminology used in this institution do not exactly match the one proposed by Snomed-CT.

Finally, 12 and 13 reaches a score of 0.83 and 0.71 but even though they require 1 and 2 single character replacements they are clear mistakes. This is a particularly complicate case since the word that the algorithm manipulates leads to a word that corresponds to a term that has no relation to the origin. These are not real examples, they has been included just to show the difficulty of the task.

Some of the issues mentioned above (like negation) could be solved relatively easily by improving our algorithm but probably its solution would be language dependant. Some issues requires a treatment specific for those terms that includes a number (i.e., vitamin d 3 vs vitamin b 3 or hepatitis type I vs hepatitis type II). Other cases requires an enlargement of the vocabulary.

Table 21 shows a full view of all the evaluations. In spite of the issues regarding the evaluation material mentioned above, the results of the evaluation are acceptable. A remarkable exception is the evaluation of terms in the SemEval competition due to requirement to use automatic translation of Arabic texts.

Table 20. Coverage of Snomed-CT classes of English terms from different sources

Table 21. Summary of the results obtained by extrinsic evaluation

It is important to check to what extent using MHeTRep vocabularies is better than using directly one or more of the original sources. We consider two cases depending of the language: languages other than English and English. In the first case the analysis is clear, the coverage is heavily dependent of the terms obtained from other languages, mainly English, due to the limited amount of its own resources. Such languages range from Spanish, with a relatively high number of resources to Arabic with a 100% of their terms coming from other languages. In any case, the number of resources and their completeness is much lower than for English.

For English, we need a more accurate analysis. In Table 20, rows 1 to 12 show the contribution of the different sources to the vocabularies of the different classes in columns. As expected, the contribution of each source alone is lower than the global coverage of MHeTRep but let us look more in depth. In rows 15 and 13, we present the first contributor for each class and the number of terms contributed. For most of the classes, the first source was Snomed-CT but for some important ones the first contributor was another source (FMA, DBPedia, MeSH). We devise, thus, two scenarios: (i) using only one resource, that is, Snomed-CT, and (ii) using the best resource for each class. We computed, for both cases and for each class the fraction of the MHeTRep vocabulary covered by the corresponding source. $\Delta\_SN$ (row 16) covers the former case and $\Delta\_m$ (row 13) the latter. For instance, for Body Structure (column 2), the best contributor is FMA with 176,491 terms. As the size of MHeTRep for this class is 312,312 terms, 43% of the terms (row 14) are not covered by the best resource. In the second case, using Snomed-CT, this resource covers only 55,755 and, so, 82% of the terms are not provided by this resource. The rest of classes confirm this figures and, so, we can conclude that MHeTRep performs the best in both scenarios.