2.1. Korean linguistics redux

Korean is considered to be an agglutinative language in which functional morphemes are attached to the stem of the word as suffixes. The language possesses a subject-object-verb (SOV) word order, which means the object precedes the verb in a sentence. This is different from English grammar where the object succeeds the verb in a sentence. Examples from Park and Kim (Reference Park and Kim2023) are presented as follows.

1. (1)

   

Two properties of the Korean language are observed. While Korean generally follows the SOV word order (1a), scrambling may occur from time to time which shifts the order of the subject and the object. Moreover, the postpositions in Korean, such as the nominative marker -i in (1a) and (1b), follow the stems to construct words, but there are also words in Korean that take no postposition, such as the subject jon (“John”) as presented in (l11).

Another notable property of Korean is its natural segmentation, namely eojeol. An eojeol is a segmentation unit separated by space in Korean. Given that Korean is an agglutinative language, joining content and functional morphemes of words (eojeols) is very productive, and the number of their combinations is exponential. We can treat a given noun or verb as a stem (also content) followed by several functional morphemes in Korean. Some of these morphemes can, sometimes, be assigned its syntactic category. Let us consider the sentence in (2). Unggaro (“Ungaro”) is a content morpheme (a proper noun) and a postposition -ga (nominative) is a functional morpheme. They form together a single eojeol (or word) unggaro-ga (“Ungaro+nom”). The nominative case markers -ga or -i may vary depending on the previous letter – vowel or consonant. A predicate naseo-eoss-da also consists of the content morpheme naseo (“become”) and its functional morphemes, -eoss (“past”) and -da (“decl”), respectively.

1. (2)

   

2.2. NEs in Korean

Generally, the types of NE in Korean do not differ significantly from those in other languages. NE in Korean consists of person names (PER), locations (LOC), organizations (ORG), numbers (NUM), dates (DAT), and other miscellaneous entities. Some examples of NE in Korean are as follows:

• • PER: ichangdong (“Lee Chang-dong,” a South Korean film director)

• • LOC: jeju (“Jeju Island”)

• • ORG: seouldaehaggyo (“Seoul National University”)

While NEs can appear in any part of the sentence, they typically occur with certain distributions with regard to the postpositions that follow the NEs. Figure 1 presents the overall distribution of the postpositions after NEs, whereas Table 2 lists the percentage of the postpositional markers with respect to three types of typical NEs, namely PER, ORG, and LOC, based on NAVER’s Korean NER dataset.Footnote ^b For instance, a PER or an ORG is often followed by a topic marker eun/neun (JX), a subject marker -i/-ga (JKS), or an object marker -eul/-leul (JKO). As detailed in Table 2, the percentages of JX, JKS, and JKO after ORG and PER are sufficiently high for us to make this conclusion, considering that in several cases, NEs are followed by other grammatical categories (e.g., noun phrases), which takes up high percentages (58.92% for LOC, 60.13% for ORG, and 66.72% for PER). Moreover, a PER or an ORG tends to occur at the front of a sentence, which serves as its subject. Higher percentages of occurrences of JX and JKS than JKO after ORG and PER prove this characteristic. Meanwhile, a LOC is typically followed by an adverbial marker (JKB) such as -e (locative “to”), -eseo (“from”), or -(eu)ro (“to”), because the percentage of JKB after LOC is considerably higher.

Table 1. XPOS (Sejong tag set) introduced in the morpheme-based CoNLL-U data converted from NAVER’s NER corpus and its type of named entities.

Figure 1. Distribution of each type of postposition/particle after NEs (NER data from NAVER). The terms and notations in the figure are described in Table 1.

As mentioned in Section 1, NE in the Korean language is considered to be difficult to handle compared to Latin alphabet-based languages such as English and French. This is mainly because of the linguistic features of NE in Korean. The Korean language has a unique phonetic-based writing system known as hangul, which does not differentiate lowercase and uppercase. Nor does it possess any other special forms or emphasis regarding NEs. Consequently, an NE in Korean, through its surface form, makes no difference from a non-NE word. Because the Korean writing system is only based on its phonological form owing to the lack of marking on NE, it also creates type ambiguities when it comes to NER.

Another feature of NE in Korean is that the natural segmentation of words or morphemes does not always represent NE pieces. The segmentation of Korean is based on eojeol, and each eojeol contains not only a specific phrase as a content morpheme but also postpositions or particles that are attached to the phrase as functional morphemes, which is typical in agglutinative languages. Normally, Korean NEs only consist of noun phrases and numerical digits. But under special circumstances, a particle or a postposition could be part of the NE, such as the genitive case maker -ui. Moreover, an NE in Korean does not always consist of only one eojeol. In Korean, location and organization can be decomposed into several eojeols (Yun Reference Yun2007), which implies that there is no one-to-one correspondence with respect to a Korean NE and its eojeol segment.

Table 2. Percentage (%) of occurrences of JKB/JKO/JKS/JX among all types of parts-of-speech after LOC/ORG/PER (NER data from NAVER).

Figure 2. Various approaches of annotation for named entities (NEs): the eojeol-based approach annotates the entire word, the morpheme-based annotates only the morpheme and excludes the functional morphemes, and the syllable-based annotates syllable by syllable to exclude the functional morphemes.

Therefore, the main difficulties in Korean NEs are as follows: (1) NEs in Korean are not marked or emphasized, which requires other features from adjacent words or the entire sentence to be used; (2) in a particular eojeol segment, one needs to specify the parts of this segmentation that belong to the NE, and within several segments, one needs to determine the segments that correspond to the NE.

5.1. Baseline CRF-based model

The objective of the NER system is to predict the label sequence $Y=$ { $y_1, y_2,\ldots, y_n$ }, given the sequence of words $X=$ { $x_1, x_2,\ldots, x_n$ }, where $n$ denotes the number of words. As we presented in Section 2, $y$ is the gold label of the standard NEs (PER, LOC, and ORG). An advantage of CRFs compared to previous sequence labeling algorithms, such as HMMs, is that their features can be defined as per our definition. We used binary tests for feature functions by distinguishing between unigram ( $f_{y,x}$ ) and bigram ( $f_{y^{\prime},y,x}$ ) features as follows:

\begin{align*} f_{y,x} (y_i, x_i) & = \mathbf{1}(y_i = y, x_i = x) \\ f_{y^{\prime},y,x} (y_{i-1}, y_i, x_i) & = \mathbf{1}(y_{i-1} = y^{\prime}, y_i = y, x_i = x) \end{align*}

where $\mathbf{1}({condition})$ = 1 if the condition is satisfied and 0 otherwise. Here, $({condition})$ represents the input sequence $x$ at the current position $i$ with CRF label $y$ . We used word and other linguistic information such as POS labels up to the $\pm$ 2-word context for each surrounding input sequence $x$ and their unigram and bigram features.

5.2. RNN-based model

There have been several proposed neural systems that adapt the LSTM neural networks (Lample et al. Reference Lample, Ballesteros, Subramanian, Kawakami and Dyer2016; Strubell et al. Reference Strubell, Verga, Belanger and McCallum2017; Jie and Lu Reference Jie and Lu2019). These systems consist of three different parts. First, the word representation layer transforms the input sequence $X$ as a sequence of word representations. Second, the encoder layer encodes the word representation into a contextualized representation using RNN-based architectures. Third, the classification layer classifies NEs from the contextualized representation of each word.

We construct our baseline neural model following Lample et al. (Reference Lample, Ballesteros, Subramanian, Kawakami and Dyer2016) by using pre-trained fastText word embedding (Bojanowski et al. Reference Bojanowski, Grave, Joulin and Mikolov2017; Grave et al. Reference Grave, Bojanowski, Gupta, Joulin and Mikolov2018) as the word representation layer and LSTM as the encoder layer. The overall structure of our baseline neural model is displayed in Figure 4.

Figure 4. Overall structure of our RNN-based model.

Word representation layer. To train an NER system, the first step is to transform the words of each input sequence to the word vector for the system to learn the meaning of the words. Word2vec (Mikolov et al. Reference Mikolov, Sutskever, Chen, Corrado, Dean, Burges, Bottou, Welling, Ghahramani and Weinberger2013) and GloVe (Pennington, Socher, and Manning Reference Pennington, Socher and Manning2014) are examples of word representations. Word2Vec is based on the distributional hypothesis and generates word vectors by utilizing co-occurrence frequencies within a corpus. fastText, on the other hand, is an extension of the Word2Vec that incorporates subword information. Rather than treating each word as a discrete entity, fastText breaks words down into smaller subword units, such as character n-grams, and learns embeddings. This approach allows fastText to capture information about word morphology and handle out-of-vocabulary words more effectively than other methods.

Given an input word $x_i$ , we initialize word embedding $v^{(w)}_i$ using pre-trained fastText embeddings (Grave et al. Reference Grave, Bojanowski, Gupta, Joulin and Mikolov2018) provided by the 2018 CoNLL shared task organizer. Randomly initialized POS features, UPOS $v^{(u)}_i$ and XPOS $v^{(x)}_i$ are concatenated to $v^{(w)}_i$ if available.

Encoder layer. The word embedding proposed in this study utilizes context-aware word vectors in a static environment. Therefore, in the NER domain we are using, it can be considered context-independent. For example, each word representation $v^{(w)}_i$ does not consider contextual words such as $v^{(w)}_{i+1}$ and $v^{(w)}_{i-1}$ . Furthermore, since the POS features are also randomly initialized, there is an issue of not being able to determine the context. LSTMs are typically used to address the context-independent problem. The sequence of word embeddings is fed to a two-layered LSTM to transform the representation to be contextualized as follows:

(1) \begin{eqnarray} h_i^{\text{(ner)}} = \text{BiLSTM}\left(r^{\text{(ner)}}_{0}, \left(v^{(w)}_{1},\ldots, v^{(w)}_{n}\right)\right)_{i} \end{eqnarray}

where $r^{\text{(ner)}}_0$ represents the randomly initialized context vector, and $n$ denotes the number of word representations.

Classification layer. The system consumes the contextualized vector $h^{\text{(ner)}}_i$ for predicting NEs using a multi-layer perceptron (MLP) classifier as follows:

(2) \begin{eqnarray} z_{i} = Q^{\text{(ner)}}\text{MLP}\left(h^{\text{(ner)}}_{i}\right) + b^{\text{(ner)}} \end{eqnarray}

where MLP consists of a linear transformation with the ReLU function. The output $z_i$ is of size $1 \times k$ , where $k$ denotes the number of distinct tags. We apply Softmax to convert the output, $z_i$ , to be a distribution of probability for each NE.

During training, the system learns parameters $\theta$ that maximize the probability $P(y_{i}|x_{i},\theta )$ from the training set $T$ based on the conditional negative log-likelihood loss.

(3) \begin{equation} \text{Loss}(\theta ) = \sum _{(x_{i},y_{i})\in T}-\log P(y_{i}|x_{i},\theta )\end{equation}

(4) \begin{equation}\hat{y}= \arg \max _{y}P(y|x_{i},\theta ) \end{equation}

where $\theta$ represents all the aforementioned parameters, including $Q^{\text{(ner)}}$ , $\text{MLP}^{\text{(ner)}}$ , $b^{\text{(ner)}}$ , and $\text{BiLSTM}$ . $(x_{i},y_{i})\in T$ denotes input data from training set $T$ , and $y_{i}$ represents the gold label for input $x_{i}$ . $\hat{y}=\arg \max _{y}P(y|x_{i},\theta )$ is a set of predicted labels.

It is common to replace the MLP-based classifiers with the CRF-based classifiers because the possible combinations of NEs are not considered in MLP. Following the baseline CRF model, we also implemented CRF models on top of LSTM output $h$ as the input of CRF. The performance of the CRFs and MLP classifiers is investigated in Section 7.

5.3. BERT-based model

As mentioned in the previous section, the word vectors we used were trained in a static environment, which led to a lack of context information. To address this issue, a system should model the entire sentence as a source of contextual information. Consequently, the word representation method has been rapidly adjusted to adapt new embeddings trained by a masked language model (MLM). The MLM randomly masks some of the input tokens and predicts the masked word by restoring them to the original token. During the prediction stage, the MLM captures contextual information to make better predictions.

Recent studies show that BERT trained by bidirectional transformers with the MLM strategy achieved outperforming results in many NLP tasks proposed in the General Language Understanding Evaluation (GLUE) benchmark (Wang et al. Reference Wang, Singh, Michael, Hill, Levy and Bowman2018). Since then, several transformer-based models have been proposed for NLP, and they show outstanding performances over almost all NLP tasks. In this section, we propose an NER system that is based on the new annotation scheme and apply the multilingual BERT and XLM-RoBERTa models to the Korean NER task.

BERT representation layer. BERT’s tokenizer uses a WordPiece-based approach, which breaks words into subword units. For example, the word “snowing” is divided into “snow” and “##ing”. In contrast, the XLM-RoBERTa tokenizer uses a BPE-based approach (Sennrich, Haddow, and Birch Reference Sennrich, Haddow and Birch2016), which can produce more subwords than the WordPiece approach. XLM-RoBERTa uses bidirectional modeling to check context information and uses BPE when training MLM. The XLM-RoBERTa decomposes a token $x_i$ into a set of subwords. For example, a token silnae (“interior”) is decomposed to three subwords, namely “ si”, “ l”, and “ nae” by the XLM-RoBERTa tokenizer. Since Korean has only a finite set of such subwords, the problems of unknown words that do not appear in the training data $T$ are addressed effectively. Once the BERT representation is trained by MLM using massive plain texts, the word representation and all parameters from training are stored as the pre-trained BERT. Recently, there have been several pre-trained transformer-based models in public use. Users are generally able to fine-tune the pre-trained BERT representation for their downstream tasks. We implemented our BERT-based NER systems using the pre-trained BERT models provided by Hugging Face.Footnote ^g In the BERT-based system, a word is tokenized and transformed into a sequence of BERT representations as $V^{(b)}_{i} =$ $\left\{v^{(b)}_{i,1}, v^{(b)}_{i,2},\ldots, v^{(b)}_{i,j}\right\}$ , where $i$ and $j$ j denote the $i$ -th word from the number of $n$ words and the number of BPE subwords of the $i$ -th word, respectively. We only take the first BPE subword for each word as the result of the BERT representations proposed in Devlin et al. (Reference Devlin, Chang, Lee and Toutanova2019). As an illustration, our system obtains embeddings for the subword “snow” in the word “snowing,” which is segmented into “snow” and “##ing” by its tokenizer. This is because BERT is a bidirectional model, and even the first BPE subword for each token contains contextual information over the sentence. Finally, the resulting word representation becomes $V^{(b)} =$ $\left\{v^{(b)}_{1,1}, v^{(b)}_{2,1},\ldots, v^{(b)}_{n,1}\right\}$ . Following what has been done to the RNN-based model, we concatenate the POS embedding to $v^{(b)}_{i,1}$ if available.

Classification layer. We apply the same classification layer of the RNN-based model with an identical negative log-likelihood loss for the BERT-based model.

6.1. Data

The Korean NER data we introduce in this study are from NAVER, which was originally prepared for a Korean NER competition in 2018. NAVER’s data include 90,000 sentences, and each sentence consists of indices, words/phrases, and NER annotation in the BIO-like format from the first column to the following columns, respectively. However, sentences in NAVER’s data were segmented based on eojeol, which, as mentioned, is not the best way to represent Korean NEs.

We resegment all the sentences into the morpheme-based representation as described in Park and Tyers (Reference Park and Tyers2019), such that the newly segmented sentences follow the CoNLL-U format. An eoj2morph script is implemented to map the NER annotation from NAVER’s eojeol-based data to the morpheme-based data in the CoNLL-U format by pairing each character in NAVER’s data with the corresponding character in the CoNLL-U data, and removing particles and postpositions from the NEs mapped to the CoNLL-U data when necessary. The following presents an example of such conversions:

In particular, the script includes several heuristics that determine whether the morphemes in an eojeol belong to the NE the eojeol refers to. When both NAVER’s data that have eojeol-based segmentation and the corresponding NE annotation, and the data in the CoNLL-U format that only contain morphemes, UPOS (universal POS labels, Petrov et al. Reference Petrov, Das and McDonald2012), and XPOS (Sejong POS labels as language-specific POS labels) features are provided, the script first aligns each eojeol from NAVER’s data to its morphemes in the CoNLL-U data and then determines the morphemes in this eojeol that should carry the NE tag(s) of this eojeol, if any. The criteria for deciding whether the morphemes are supposed to carry the NE tags are that these morphemes should not be adpositions (prepositions and postpositions), punctuation marks, particles, determiners, or verbs. However, cases exist in which some eojeols that carry NEs only contain morphemes of the aforementioned types. In these cases, when the script does not find any morpheme that can carry the NE tag, the size of the excluding POS set above will be reduced for the given eojeol. The script will first attempt to find a verb in the eojeol to bear the NE annotation and, subsequently, will attempt to find a particle or a determiner. The excluding set keeps shrinking until a morpheme is found to carry the NE annotation of the eojeol. Finally, the script marks the morphemes that are in between two NE tags representing the same NE as part of that NE (e.g., a morpheme that is between B-LOC and I-LOC is marked as I-LOC) and assigns all other morphemes an “O” notation, where B, I, and O denote beginning, inside and outside, respectively. Because the official evaluation set is not publicly available, the converted data in the CoNLL-U format from NAVER’s training set are then divided into three subsets, namely the training, holdout, and evaluation sets, with portions of 80%, 10%, and 10%, respectively The corpus is randomly split with seed number 42 for the baseline. In addition, during evaluation of neural models, we use the seed values 41–45 and report the average and their standard deviation. Algorithm 1 describes the pseudo-code for converting data from NAVER’s eojeol-based format into the morpheme-based CoNLL-U format. Here, $w_i:t_i$ in $S_{\text{NAVER}}$ represents a word $w_i$ and its NE tag $t_i$ (either an entity label or O). $w^{\prime}_{i-k} m_i \ldots m_k$ in $S_{\text{CoNLL-U}}$ displays a word $w^{\prime}_{i-k}$ with its separated morphemes $m_i \ldots m_k$ .

Algorithm 1. Pseudo-code for converting data from NAVER’s eojeol-based format into the morpheme-based CoNLL-U format

We also implement a syl2morph script that maps the NER annotation from syllable-based data (e.g., KLUE and MODU) to the data in the morpheme-based CoNLL-U format to further test our proposed scheme. While the eoj2morph script described above utilizes UPOS and XPOS tags to decide which morphemes in the eojeol should carry the NE tags, they are not used in syl2morph anymore, as the NE tags annotated on the syllable level already exclude the syllables that belong to the functional morphemes. Additionally, NEs are tokenized as separate eojeols at the first stage before being fed to the POS tagging model proposed by Park and Tyers (Reference Park and Tyers2019). This is because the canonical forms of Korean morphemes do not always have the same surface representations as the syllables do. Because the syl2morph script basically follows the similar principles of eoj2morph, except for the two properties (or the lack thereof) mentioned above, we simply present an example of the conversion described above:

We further provide morph2eoj and morph2syl methods which allow back-conversion from the proposed morpheme-based format to either NAVER’s eojeol-based format or the syllable-based format, respectively. The alignment algorithm for back-conversion is simpler and more straightforward given that our proposed format preserves the original eojeol segment at the top of the morphemes for each decomposed eojeol. As a result, it is not necessary to align morphemes with eojeols or syllables. Instead, only eojeol-to-eojeol or eojeol-to-syllable matching is required. The morph2eoj method assigns NE tags to the whole eojeol that contains the morphemes these NE tags belong to, given that in the original eojeol-based format, eojeols are the minimal units to bear NE tags. The morph2syl method first locates the eojeol that carries the NE tags in the same way as described above for morph2eoj. Based on the fact that NEs are tokenized as separate eojeols by syl2morph, the script assigns the NE tags to each of the syllables in the eojeol.

Back-conversions from the converted datasets in the proposed format to their original formats are performed. Both eojeol-based and syllable-based datasets turned out to be identical to the original ones after going through conversions and back-conversions using the aforementioned script, which shows the effectiveness of the proposed algorithms. Manual inspection is conducted on parts of the converted data, and no error is found. While the converted dataset may contain some errors that manual inspection fails to discover, back-conversion as a stable inspection method recovers the datasets with no discrepancy. Therefore, we consider our conversion and back-conversion algorithms to be reliable. On the other hand, no further evaluation of the conversion script is conducted, mainly because the algorithms are tailored in a way that unlike machine learning approaches, linguistic features and rules of the Korean language, which are regular and stable, are employed during the conversion process.

6.2. Experimental setup

Our feature set for baseline CRFs is described in Figure 5. We use crf++ Footnote ^h as the implementation of CRFs, where crf++ automatically generates a set of feature functions using the template.

The hyperparameter settings for the neural models are listed in Table 9, Appendix A. We apply the same hyperparameter settings as in Lim et al. (Reference Lim, Park, Lee and Poibeau2018) for BiLSTM dimensions, MLP, optimizer including $\beta$ , and learning rate to compare our results with those in a similar environment. We set 300 dimensions for the parameters including $LSTM$ in (1), $Q$ , and $MLP$ in (2). In the training phase, we train the models over the entire training dataset as an epoch with a batch size of 16. For each epoch, we evaluate the performance on the development set and save the best-performing model within 100 epochs, with early stopping applied.

The standard $F_{1}$ metric ( $= 2 \cdot \frac{P \cdot R}{P + R}$ ) is used to evaluate NER systems, where precision ( $P$ ) and recall ( $R$ ) are as follows:

\begin{align*} P = \frac{\textrm{retrieved named entities} \cap \textrm{relevant named entities}}{\textrm{retrieved named entities}} \\ R = \frac{\textrm{retrieved named entities} \cap \textrm{relevant named entities}}{\textrm{relevant named entities}} \end{align*}

We evaluate our NER outputs on the official evaluation metric script provided by the organizer.Footnote ⁱ

Table 4. CRF/Neural results using different models using NAVER’s data converted into the proposed format.

fastText (Joulin et al. Reference Joulin, Grave, Bojanowski, Douze, Jégou and Mikolov2016) for LSTM+CRF word embeddings.

Figure 5. CRF feature template example for word and pos.