3.1 Tokenization

First, we used the preprocessing tools from Moses (Koehn et al. Reference Koehn, Hoang, Birch, Callison-Burch, Federico, Bertoldi, Cowan, Shen, Moran, Zens, Dyer, Bojar, Constantin and Herbst2007) for normalizing punctuation marks and tokenization. However, there are several issues and errors in French tokenization. We corrected the tokenization results, including identifying pre-defined entities, such as telephone numbers (02 $ \sqcup$ 31 $\sqcup$ 77 $\sqcup$ 01 $\sqcup$ 16 $\rightarrow$ 02-31-77-01-16), times (12h $\sqcup$ 45 $\rightarrow$ 12h45), and web addresses. A binary operator $\sqcup$ represents a whitespace delimiter in the written text. We manually identified these patterns and constructed regular expressions for post-processing.

3.2 POS tagging and rough sentence boundaries

Second, POS labels and “rough” sentence boundaries were handled by TreeTagger (Schmid Reference Schmid1994) based on added punctuation marks, which is required for NER in the next step. Because TreeTagger only classifies punctuation marks for the ends of sentences, we refer to TreeTagger’s sentence boundaries as “rough” sentence boundaries.

3.3 NER

Third, because the corpus contains a significant number of proper nouns, named entity labels were assigned. We used NER data provided by Europeana newspapers for French (Neudecker Reference Neudecker2016) and trained them with NeuroNER (Dernoncourt, Lee, and Szolovits Reference Dernoncourt, Lee and Szolovits2017), which implements a bidirectional long short-term memory recurrent NN. We evaluated NER models for French using various sequence labeling algorithms and improved the NER results using semi-supervised learning (Park Reference Park2018). To refine the current NER task, we added geographical entities using a list of communes in France,Footnote a which improved the NER results. The NER-processed corpus on the left side of Figure 3 shows geographical entities (third column, using beginning–inside–outside (bio) annotation using a geographical entity dictionary) and RNN-annotated entities (fourth column, with using the inside–outside (io) format instead of using the bio, owing to the original annotation of the NER corpus, which introduces the io format). In the corpus, we removed the surnames to maintain anonymity. We compared two entities via geographical entity assignment and RNN labeling and selected the more pertinent entity. When assigned entities were different, the geographical entity was selected if its length (the number of tokens for the entity) was $>$ 1; otherwise, the RNN-annotated entity was assigned. For example, Chaume (a village attached to the city of Sables-d’Olonne) was annotated as I-LOC by geographical entity assignment and I-PER by RNN labeling. We selected “I-LOC” because the length of the geographical entity assignment was greater than one. We removed B annotations of the geographical entity assignments to retain only io annotations of named entities for the consistency of NER labels (see the right side of Figure 3).

Figure 3. Preprocessed SBD data for training (sent marked): Les Sables-d’Olonne La Chaume and Philippe et Véronique, son … are annotated as two sentences where the latter represents the sentence middle that no punctuation marks precede.

3.4 Marking SENT labels and a manual correction

Finally, we created heuristic rules to determine whether a fragment (e.g., a noun phrase) can be an independent sentence (e.g., geographical entities at the beginning of the text). However, heuristic rules to mark SENT labels (for the beginning of a sentence) were weak, and we manually verified all sentences to correctly mark SENT labels. During manual verification, we corrected SENT labels in addition to incorrectly assigned POS and NER labels, which were initially automatically assigned using heuristic rules. Errors in POS labels are often caused by words that start with a capital letter in the middle of a sentence. For example, Très (‘very’) is automatically labeled as a proper noun in Remungol, Guénin Plounévézel Très touchée par … (‘loc, loc loc very touched by ... ’). If we can detect the correct beginning of the sentence in which a fragment, Remungol (‘loc’), Guénin Plounévézel (‘loc loc’), is considered to be an independent sentence, and the following sentence starts from Très (‘very’), the word Très would be correctly labeled as an adverb. There were person- and location-label errors in NER, and we manually corrected them as much as possible. We corrected more than 29-K tokens for their POS and NER labels from automatic annotation. From 36,392 TreeTagger-separated sentence boundaries, we arrived at 42,023 sentences, including those starting without punctuation marks from the previous sentence in the corpus. Because we merged the corpus into a single file to process, the order of sentences was randomized based on TreeTagger-assigned sentence boundaries using punctuation marks. We split the corpus according to an 80:10:10 ratio for training, development, and testing datasets.

Thereafter, we removed all duplicated sentences. Finally, after splitting, the datasets contained approximately 763-K, 86-K, and 85-K tokens for 33-K, 3-K, and 3-K sentences, respectively. After automatically assigning POS and NER labels using the existing POS and NER models and SBD labels using the heuristic rules, it took longer than 2 weeks (approximately 80 hours) for a language expert to manually verify the labels and correct them. The verification process was straightforward, wherein the expert verified and corrected manually assigned POS, NER, and SBD labels using a simple text editor.

The detailed statistics of the corpus are provided in Table 2. Although the average number of tokens in all sentences was 23.42, the average number of tokens in the middle of sentences was 52.90, which is much larger. This is mainly because, at the beginning of the TreeTagger-separated sentence, relatively short noun fragments (e.g., Les Sables-d’Olonne La Chaume), such as a place name or a title of the document and paragraph, which are not part of the main sentence, can appear, as seen in Figure 2. Figure 3 shows a preprocessed SBD dataset for training, wherein sent labels were marked and manually verified.

Table 2. Detailed statistics of the corpus.

Sentences (middle) represent the number of sentences in which a new sentence starts in the middle of the line. No punctuation marks precede these sentences. Avg. B-sent indicates the average number of B-sent between punctuation mark-based separated sentences where they may contain more than one sentence and a fragment. Tokens (middle) show the number of tokens in only middle sentences where punctuation marks are not preceded. They exclude tokens from (1) punctuation mark-based separated sentences that contain only one sentence and (2) the beginning of the sentence preceded by middle sentences.

4.1 CRF baseline SBD model

An advantage of CRFs, compared with previous sequence algorithms (e.g., Hidden Markov Models (HMMs)), is that we can assign our own defined features. Thus, CRFs can usually outperform HMMs, owing to the relaxation of independence assumptions. We used binary tests for feature functions by distinguishing between unigram ( $f_{y,x}$ ) and bigram ( $f_{y',y,x}$ ) features:

(1) \begin{align}f_{y,x} (y_i, x_i) & = \textbf{1}(y_i = y, x_i = x) \nonumber \\[3pt] f_{y',y,x} (y_{i-1}, y_i, x_i) & = \textbf{1}(y_{i-1} = y', y_i = y, x_i = x) \end{align}

where $\textbf{1}({condition})=1$ if the condition is satisfied and 0 otherwise. (condition) represents the input sequence, x, at the current position, i, with CRF label y. We used word, POS, and NER for the input sequence, x, and for the unigram ( $w_{t-2}, w_{t-1}, w_{t}, w_{t+1}, w_{t+2},$ ) and bigram ( $w_{t-2}/w_{t-1}, w_{t-1}/w_{0}, w_{0}/w_{t+1}, w_{t+1}/w_{t+2}$ for the word, e.g.,) features. In addition to the features, we also used a Capitalized feature, where $w_t$ matches [A-Z][a-z]+. Previous work on NER utilized text chunking information, which divides a text into phrases in such a way that it syntactically relates words (Tjong Kim Sang and Buchholz Reference Tjong Kim Sang and Buchholz2000). However, we do not do so because detecting phrase boundaries is especially challenging in French, owing to the flat structure of the French treebank (Abeillé, Clément, and Toussenel Reference Abeillé, Clément and Toussenel2003), wherein we would otherwise obtain text chunking information. However, it gives ambiguous phrase boundaries.

Figure 4. Overall structure of our roberta-sbd model.

4.2 Contextualized NN SBD model

We investigated the performance of our system based on the CRF model and checked the effect on the proposed features, including named entities. However, we were still able to explore the performance of SBD using state-of-the-art techniques. In this section, we introduce our own neural network (NN)-based SBD system. Following the previously proposed NN-based systems, we implemented an SBD system using XLM-RoBERTa (Conneau et al. Reference Conneau, Khandelwal, Goyal, Chaudhary, Wenzek, Guzmán, Grave, Ott, Zettlemoyer and Stoyanov2020). RoBERTa (Liu et al. Reference Liu, Ott, Goyal, Du, Joshi, Chen, Levy, Lewis, Zettlemoyer and Stoyanov2019) is a BERT-like model that applies the masked language model and shows outstanding performance on several NLP tasks (Devlin et al. Reference Devlin, Chang, Lee and Toutanova2018). XLM-RoBERTa is a multilingual RoBERTa trained in several languages. Figure 4 shows the overall structure of our system. The system consumes a list of words $X = (x_1, x_2, ..., x_m)$ , where $x_i$ ( $1\leq i \leq m$ ) consists of a word form, a POS label, and an NER label. We convert the word form of X to a list of word representations, $E^b = (e^b_1, e^b_2, ..., e^b_m)$ based on the pretrained XLM-RoBERTa model.Footnote b First, we convert POS and NER to their distributional vector representation by assigning each POS and NER label to randomly initialized embedding vectors, $E^p = (e^p_1, e^p_2, ..., e^p_m)$ and $E^n = (e^n_1, e^n_2, ..., e^n_m)$ , respectively. The same value is assigned for the same POS and NER labels. To consider the word form, the POS, and the NER feature as a unified embedding, we concatenate them ( $[e^b_i;\ e^p_i;\ e^n_i]$ ) and transform the unified embedding using LSTM as follows:

(2) \begin{align}e_i &= \left[e^b_i;\ e^p_i;\ e^n_i\right] \nonumber\\[3pt] f_i, b_{i} &= BiLSTM\!\left(r_{0}, (e_{1}, .., e_{m})\right)_{i}\nonumber \\[3pt] h_{i} &= \left[\,f_{i};\ b_{i}\right] \end{align}

where $r_0$ denotes a randomly initialized initial vector for the LSTM hidden layer, $f_i$ is the forward-pass hidden layer of the BiLSTM for word i, $b_i$ is the backward-pass, and $h_i$ is the concatenation of the two. Previous studies have shown that applying LSTM after the concatenation of different embeddings showed better performance because the output of the LSTM keeps track of the contextual information (Lim et al. Reference Lim, Park, Lee and Poibeau2018). Finally, we apply a multilayered perceptron (MLP) classifier with a weight parameter, $Q^{(sbd)}$ , including a bias term, $b^{(sbd)}$ , to classify sentence boundaries for the output hidden state, $h_i$ , as follows:

(3) \begin{align} p^{(sbd)}_{i} &= Q^{(sbd)}MLP(h_{i}) + b^{(sbd)} \nonumber\\[3pt] y^{(sbd)}_{i} &= \textrm{arg}\!\max_{j} p^{(sbd)}_{i,j} \end{align}

where the value of j in $p^{(sbd)}_{i,j}$ is two because we have two labels (B-SENT and O) for the sentence boundary. During training, the system adjusts the parameters of the network, $\theta$ , that maximizes the probability, $P(Y|X,\theta)$ , from the training set, T, based on the conditional negative log-likelihood, $\text{sbd-loss} (\theta)$ . Thus,

(4) \begin{eqnarray}\text{sbd-loss} (\theta) =\sum_{(X,Y)\in T}-\log P(Y|X,\theta) \end{eqnarray}

where $(X,Y)\in T$ denotes an element from the training set, T, a set of sentence boundary labels, Y, and a predicted label of a token, $y^{(sbd)}_{i}$ . We trained our system using a single Adam optimization algorithm (Kingma and Ba Reference Kingma and Ba2015) with a cross-entropy loss. During training, we set the number of input batch sizes to 16 and run our system over 100 epochs. For each epoch, we trained our system using only training data and evaluated the validation data. Finally, we selected the best performing model among 50 different ones and run the test data to evaluate the scores.

4.3 Contextualized multitask SBD model

In reality, the input of an SBD task is plain text without any linguistic information. A potential approach may involve a cascade pipeline system, wherein the first tagger assigns POS labels to each word, the second tagger assigns NER labels based on words and POS labels, and the third tagger assigns SBD labels based on previous information. This system uses predicted POS and NER labels incrementally to detect sentence boundaries. However, these tasks can be achieved simultaneously. In this section, we propose a multitask learning scenario that handles POS and NER labeling and SBD labeling simultaneously.

Following the previously investigated multitask learning problem with a shared lexical representation (Hashimoto et al. Reference Hashimoto, Xiong, Tsuruoka and Socher2016; Lim et al. Reference Lim, Park, Lee and Poibeau2018, Reference Lim, Lee, Carbonell and Poibeau2020), we propose a more realistic SBD model that can be deployed as a real-world application. Our method trains POS, NER, and SBD labels simultaneously, rather than applying POS and NER labeling as a separate task. To obtain the predicted POS and NER features, we introduce two different classifiers in the middle of our neural model as follows:

(5) \begin{align}p^{(pos)}_{i} &= Q^{(pos)}MLP\!\left(e^b_{i}\right) + b^{(pos)} \nonumber\\[3pt] y^{(pos)}_{i} &= \textrm{arg}\!\max_{k} p^{(pos)}_{i,k}\nonumber \\[3pt] e^p_i &= Embedding^{(pos)}\left(y^{(pos)}_{i}\right) \nonumber \\[3pt] p^{(ner)}_{i} &= Q^{(ner)}MLP\!\left(e^b_{i}\right) + b^{(ner)}\nonumber \\[3pt] y^{(ner)}_{i} &= \textrm{arg}\!\max_{l} p^{(ner)}_{i,l}\nonumber \\[3pt] e^n_i &= Embedding^{(ner)}\left(y^{(ner)}_{i}\right) \end{align}

where the value of k in $p^{(pos)}_{i,k}$ and l in $p^{(ner)}_{i,l}$ is the number of POS and NER labels, respectively. The Embedding(pos) and Embedding(ner) denote randomly initialized vectors to represent each POS and NER label. For example, our system computes a logistic value using the MLP classifier; thereafter, we can predict a POS label using the argmax computation with a logistic value. Finally, the system converts the predicted POS label as a vector representation by the Embedding function. During training, we added two additional losses, pos-loss and ner-loss by changing a set of SBD labels, Y, in (4) to the POS and NER labels, respectively. The multitask loss is defined as follows:

(6) \begin{align}\text{multi-loss} (\theta) & = \alpha\ \text{sbd-loss}\nonumber\\[3pt] &\quad + \beta\ \text{pos-loss} + \gamma\ \text{ner-loss}\end{align}

where each $\alpha$ , $\beta$ , and $\gamma$ indicate the ratio of how much the system learns from each task. We empirically set { $\alpha=1.5$ , $\beta=0.5$ , $\gamma=1$ }, and the effect of different values for $\alpha$ , $\beta$ , $\gamma$ is further discussed in Section 5.2. From a practical perspective, our multitask model has the advantage of producing POS and NER labels alongside SBD labels.

Figure 5 shows the overall structure of our multitask system. The hyperparameter values that we applied for our neural system (roberta-sbd in Section 4.2) and multitask-sbd in Section 4.3) are listed in Table 4.

Table 4. Hyperparameters in neural models.

5.1 Results

Table 5 shows results of precision, recall, and the F1 score for how we used different linguistic information to improve SBD results using CRFs, roberta-sbd, and multitask-sbd. We also present the SBD results of the middle, where sentence boundaries occur in the middle of the sentence without punctuation marks. Overall, each linguistic feature improves the SBD labeling results for the baseline CRFs and neural models. Although these linguistic features are predicted labels, they can improve SBD results for all experiment settings compared with results that use only word features. Our neural models, roberta-sbd and multitask-sbd, outperformed CRFs for all experimental settings, including word-only (word) and predicted linguistic features (+pos and +ner). We speculate the following plausible explanations. First, the neural model adapts well to the given SBD corpus. Second, the BERT-like model yields more accurate sentence boundaries because it has been trained on a huge number of unlabeled data. Third, the proposed multitask approach efficiently transfers linguistic knowledge by leveraging a shared BERT representation among the following three tasks: POS tagging, NER, and SBD.

Table 5. SBD results using different linguistic information.

POS (+p) and NER (+n) labels or together (+p+n) are used as features alongside word features. We use predicted linguistic labels (+pos and +ner) from the system (p). We show the entire SBD results (all) and those only for sentence boundaries without precedent punctuation marks (middle).

Figure 5. Overall structure of our multitask model.

5.2 Discussion

5.2.3 Cost-effectiveness of the proposed system

The multitask model normally requires more computing resources and training time. Therefore, it is important to investigate the cost-effectiveness of the proposed model from a practical point of view. Table 6 shows the comparison between the single and multitasks in terms of cost-effectiveness. The proposed single and multitask models consume 13.4- and 14.1-GB GPU memory, respectively, when training with a batch size of 16. 330-M parameters are required for the single task, which includes in the XLM-RoBERTa model. The single-task model runs for approximately 16 minutes over the training data, and it handles 929 tokens per second. The multitask model runs for 17 minutes and handles 875 tokens. However, it should be noted that the system can yield higher accuracy classifiers at the expense of 6.17% more training time, including predicting POS and NER labels. During the inference phase of the test data, the multitask model can predict 2801 tokens/s for SBD, POS, and NER labels. As our model needs to consider the number of n words as the input, the time complexity is O(n).

Table 6. Comparison between the single-task and multitask learning in terms of required computing resources and training time.

6.1 Effect of the multitask learning procedure

As mentioned in Section 4.3, we empirically set the learning weight for each task as { $\alpha=1.5$ , $\beta=0.5$ , $\gamma=1$ } for SBD, POS, and NER, respectively. However, performance varies depending on the training procedure of the multitask model. Hence, we empirically determined which task would be more significant to SBD performance by using different training procedures and considering the learning weight. We investigated two different learning methods: sequential and simultaneous. The sequential method trains a task only for a certain number of epochs and moves on to train another. In Table 7, sequential denotes the performance of the sequential model. Conversely, the simultaneous learning method trains three different tasks for every epoch with a particular task’s learning weight. Table 7 shows the SBD performance of each learning method. The parameters $\alpha$ , $\beta$ , and $\gamma$ indicate the ratio of how much our system learns from each task. For example, the second low, where $\alpha=1$ , $\beta=0$ , and $\gamma=0$ , represents single-task learning for SBD with embedding, $e_i = e^b_i$ in (2). Meanwhile, the seventh row with the parameter values { $\alpha=1.5$ , $\beta=0.5$ , and $\gamma=1$ } represents the application of multitask learning from three different tasks. The sequential method trains each task sequentially. We trained a POS tagger for the first 20 epochs and then trained a NER tagger from 20 to 40 epochs. We finally trained SBD from 40 to 80 epochs. Although the three tasks were trained separately, the shared BERT embedding $e^b_i$ was affected by all the tasks by updating the BERT embedding $e^b_i$ . The sequential method has the advantage of fine-tuning parameters for the particularities of a single task to bootstrap its final SBD performance. Overall, the simultaneous method slightly outperforms the sequential method by up to 0.72 in SBD.

Table 7. SBD (middle) results based on different multitask models.

The parameter values $\alpha$ , $\beta$ , and $\gamma$ are described in (6). In sequential POS tagging (accuracy), NER (F1 score), and SBD (F1 score) are sequentially trained.

However, the single-task learning showed the same or slightly better results than our multitask approach when observing two experimental results that set { $\alpha=0$ , $\beta=1$ , and $\gamma=0$ } for a POS task and { $\alpha=0$ , $\beta=0$ , and $\gamma=1$ } for a NER task, respectively. This is because our multitask model focuses more on the SBD task by learning the POS and NER tasks. By following McCann et al. (Reference McCann, Keskar, Xiong and Socher2018), we also investigated an experiment on the fine-tuning method, which applied both simultaneous and sequential methods simultaneously. We first trained our model with the simultaneous method using parameter values of { $\alpha=0$ , $\beta=1$ , and $\gamma=0$ } until 100 epochs; we then fine-tuned only for SBD for 20 epochs. The result is shown as +fine-tune. We found that the fine-tuning method with simultaneous does not have positive effects on our model; rather, it shows performance degradation in NER and POS tasks. The main reason for performance degradation might be that the BERT embedding was fully adjusted only for the SBD task during fine-tuning. Thus, POS and NER performances decrease, and the lower-performing POS and NER results affect SBD directly because the system considers the predicted results of the tasks.

6.2 Effect of POS and NER information

Although the proposed multitask learning method has been adapted to several NLP tasks, it has not been explained in detail how multitask learning leverages overall performance. McCann et al. (Reference McCann, Keskar, Xiong and Socher2018) showed an approach to investigating correlations among tasks using multitask learning based on performance changes. The key idea is to test whether a pretrained model trained by a task can leverage a new task’s performance. Inspired by the previously proposed method, we investigated the following two questions: “does a pretrained model learn from POS and NER tasks to improve SBD performance?” and “does the pretrained model achieve a better performance than a randomly initialized model in early training?” We assume that if the learned knowledge trained by POS and NER tasks is transferable, it positively affects training SBD during the early epochs of training. In Figure 6, SBD-single-random shows that the model trained only for SBD as a single-task learning, SBD-multi-random is the model trained on multitask learning with { $\alpha=1.5$ , $\beta=0.5$ , and $\gamma=1$ }, and SBD-multi-fine-tuned represents the pretrained model trained by POS and NER tasks for 20 epochs, respectively. We observe that the SBD-multi-fine-tuned model outperforms the other models over the first six epochs. The average performance gap between SBD-multi-fine-tuned and SBD-single-random has a 1.78 F1 score during the first six epochs. We conjecture that the BERT embedding in the single model could not obtain any syntactic and named entity information from the POS and NER tasks, whereas the fine-tuned model’s BERT acquired general syntactic information from them, and the informed linguistic knowledge was transferred to a new task for SBD. In contrast, as shown in Table 7, the fine-tuned model performed worse than the SBD-multi-random model when training for more than 20 epochs (i.e., +fine-tune and simultaneous) as described previously, where the BERT embedding was adjusted only for the SBD task.

Figure 6. Evaluation results are based on the number of training epochs. The Y-axis represents F1 scores.

6.3 Effect of the size of training data

In low-resource NLP, which frequently occurs in real-world settings, the size of training data matters. Figure 7 shows the evaluation results based on sbd-single and sbd-multi as well as POS and NER. This shows that their results can converge after using 5000 sentences, and sbd-multi always outperforms sbd-single. In particular, the multitasking setting still performs better than sbd-single when sbd-multi only utilizes 70% of the training dataset (approximately 20,000 sentences).

6.4 Limitation of the proposed model

The currently proposed system is highly reliant on named entity information in the dataset where results using +ner (+n) and +pos+ner (+p+n) features in multitask-sbd in Table 5 are similar. However, improving NER results using extrinsic factors, such as adding additional pseudo datasets by semi-supervised learning, is very difficult, as demonstrated by Park (Reference Park2018) for French. Although different learning algorithms show various F1 score results ranging from 45.76 (HMM) to 76.26 (bi-LSTM) using the French NER data provided by Europeana Newspapers, semi-supervised learning, in which we automatically annotate a large monolingual corpus, could not improve NER results significantly for F1 scores of 49.69 (HMM) to 76.65 (bi-LSTM). In the proposed multitask learning model, we can still improve the NER results in areas in which further improvement would otherwise be difficult, apart from introducing a completely different and new learning mechanism.

Table 8. Multitask SBD (middle) results based on different BERT models.

The parameter values $\alpha$ , $\beta$ , and $\gamma$ are described in (6). POS tagging (accuracy), NER (F1 score), and SBD (F1 score) are presented.

Figure 7. Evaluation results based on the number of training data. The Y-axis represents F1 scores for SBD and NER and accuracy for POS tagging.

6.5 Comparison between multilingual and French monolingual BERTs

We have shown that the BERT models outperform the CRF model. However, the BERT and the XLM-RoBERTa models that we used are multilingual. There is also a French monolingual BERT proposed by Martin et al. (Reference Martin, Muller, Ortiz Suárez, Dupont, Romary, de la Clergerie, Seddah and Sagot2020) (CamemBERT). The monolingual French BERT model was trained only with French plain text from Wikipedia and corpora taken by the Common Crawler. Table 8 shows the ablation study on multilingual and French monolingual BERT models. However, the CamemBERT model performs better for SBD. It shows relatively lower-performing results for POS and NER tasks. We leave the detailed discussion on performance between multilingual and French monolingual BERTs as future work.

6.6 Experiments on the heterogeneous domain

The Europarl corpus (Koehn Reference Koehn2005) provides French translations of Europarl proceedings, and it is a good candidate for evaluating our SBD models for general heterogeneous purposes. We constructed an evaluation dataset from the data of Q4/2000 (October–December 2000) in which the same portion was used for machine translation evaluation. It contained approximately 50-K sentences and 150-K words. To prepare evaluation data of the Europarl corpus, we first detected the beginning sequences of the sentence by using XML tags in which each was also the beginning of a sentence as shown in Figure 8. We assigned POS labels and punctuation mark-based sentence boundaries as described in Section 3.2. We also assigned NER labels as described in Section 3.3. There was an empty line for each punctuation mark-based boundary-detected sentence for rough sentence boundaries. There was no empty line for sentence boundaries only XML tags for evaluation purposes to allow the proposed models to automatically detect them. Although we did not verify automatically assigned POS and NER labels, we manually checked B-SENT labels. The dataset contains over 50-K sentences with 1.4 M tokens, with a small ratio (0.0084) of the middle. Table 9 shows the results (F1 scores) using CRFs, roberta-sbd and multitask-sbd as experiments in one of the heterogeneous domains. Although overall results were promising, results on middle were much lower. As one of the characteristics of the Europarl corpus, as seen in Figure 8, there are noun phrase fragments lacking named entities. The distinction between a noun phrase fragment and the following sentence is immensely challenging in the SBD system, but the semantic property of the noun phrase fragment should be identified. The proposed method using multitask learning relies on sequence-level shallow linguistic information, such as POS and named entities. Even deep linguistic processing, such as syntactic analysis, may not distinguish between the noun phrase fragment and the following sentence because such fragments can be considered as an adverbial phrase in the sentence. Although such a sentence can be considered as being grammatically correct, it is not semantically acceptable. The proposed model attempts to resolve this linguistically difficult problem in SBD by using currently exploitable linguistic properties. Notably, existing SBD methods in previous work cannot detect such boundaries at all, as we showed in Section 1. Additionally, sentence boundaries without punctuation marks, as shown in Figure 8, can be easily remedied by simple heuristics (e.g., using title or subtitle tags in XML). However, we did not use explicit information to conduct experiments with more realistic conditions, which are not always available in heuristics.

Table 9. SBD results of heterogeneous domains using the Europarl corpus.

Figure 8. Example of the Europarl corpus for French: (translation) “Opening of the session I declare resumed the 2000-2001 session of the European Parliament. Agenda Mr President, the second item on this morning’s agenda is the recommendation for second reading on cocoa and chocolate products, for which I am the rapporteur. Quite by accident I learnt yesterday, at 8.30 p.m., that the vote was to take place at noon today.’’ We note that Je déclare ouverte … (‘I declare resumed ...’) and Monsieur le Président, le deuxième ... (‘Mr President, the second ...’) are considered as middle sentences because punctuation marks are not preceded.

6.7 Experiment on domain adaptation

As was shown in the previous section, the overall performance of our model is promising, although it is still poor in detecting middle sentence boundaries in a heterogeneous domain. Generally, a domain adaptation approach would be a good solution to solve such a problem. Table 10 reports on the performance of our model with a domain adaptation method. We split the Europarl corpus in an 80:10:10 ratio for training, development, and testing datasets, respectively. We performed three different experiments: (1) out-of-domain, which determines the performance of our previous model (using the proposed dataset) onto the new Europarl test dataset; (2) in-domain, which uses the Europarl dataset both for training and evaluation; (3) domain adaptation, which fine-tunes our previous model using the Europarl development dataset to evaluate the Europarl test dataset. Based on the domain adaptation approach, we can observe performance improvement in detecting middle sentence boundaries by a 0.52 F1 score as well as the performance of all where it remains slightly high compared to the in-domain model.

Table 10. SBD results of domain adaptation using the Europarl corpus based on multitask-sbd with +p+n (p).

6.8 Extrinsic evaluation for SBD

We use a semantic relation recognition task for extrinsic evaluation of SBD. A semantic relation recognition task consists of the automatic recognition of semantic relationships between pairs of entities in a text. Since Roth and Yih (Reference Roth and Yih2004) proposed semantic relation recognition data for English, several works on semantic relation recognition have been explored and proposed. This section describes our semantic relation recognition system for French using the proposed SBD system with a dataset for extrinsic evaluation. We specified (1) a segmentation problem to help decide whether a sentence in the obituary contains the information of kinship relations for the deceased person, and (2) a classification problem to decide kinship relations of the deceased person for semantic relation recognition of genealogical relation. The problem of being survived by whom identifies which kinship relations in obituaries should be considered for the deceased person after determining whether the information of kinship relations is to appear. Figure 9 provides an example of an obituary, which contains the information of kinship relations and its possible genealogy tree diagram. Lucienne (a deceased person) is survived by Gaston et Marie-Claude (her children) and other grandchildren and great-grandchildren. An end-to-end semantic relation recognition system uses the heuristic symbolic rules to fill tabular cells based on kinship-related words after detecting sentences in obituaries.

Figure 9. Sentence example for obituaries and possible genealogy tree diagram: (translation) “GLANGES (Cramarigeas, Le Ch^ataignier) Gaston and Marie-Claude, his children; Laurent and Christelle, his grandchildren; Evelyne, Guillaume, her great-grandchildren, As well as all the family and her friends, are sad to inform you of the death of Madame Lucienne at the age of 84 years. Her funeral will take place on Monday, October 19, 2009, at 2:30 p.m., in the church of Glanges. Condolences on register at the church. The family thanks in advance all the people who will take part in their grief. PF Graffeuil-Feisthammel, St-Germain-les-Belles.’’

We consider only direct familial relationships of the deceased person in a genealogy tree, including parents, spouse, children, grandchildren, and great-grandchildren. We obtained 3000 additional random obituary documents crawled from the internet to evaluate the end-to-end system and analyze the results. Because the deceased person is given by the document, we identified kinship relations of the deceased person. Table 11 shows the evaluation and statistics of results from the end-to-end system. We noted misc relationships for beyond direct relationships, such as (1) siblings (i.e., brother and sister) or other relatives (i.e., second-degree relatives for aunt/uncle and niece/nephew), (2) person names without kinship relationships, and (3) other kinship, friend or colleague relationships without person names. 2760 est_décédé (‘is deceased’) relations were given from 3000 documents. 240 missing est_décédé were from the original document errors in which they do not explicitly annotate the deceased person. There may be sentences wherein kinship relationships appear without the given est_décédé relation. Hence, 1902 sentences where kinship relationships appear were identified using sentence classification from 1878 documents (four false-positive examples). We analyzed 1122 documents where kinship relationships did not appear according to sentence classification results. There were 273 false-negative examples. Although some errors came from results of SBD in which the system cannot provide correct sentence boundaries by merging several sentences because of missing punctuation marks, most presented real classification errors. The number of a_^* (‘has *’) relations presented their occurrences according to the system specification. They also included 545 relations without person names in which only relationship words occurred in the sentence without specific names of kinships. The misc relationships contained 415 person names without kinship relationships, and 941 other relationships lacked person names. For evaluation, we manually verified the quality of extracted semantic relations with two native French speakers. By using the proposed SBD system to feed the semantic relation recognition system, the average accuracy for extracted semantic relations was 92.36% by human judgment. When we used the conventional sentence boundary disambiguation system (i.e., TreeTagger) for French, the system did not detect sentence boundaries without full-stop punctuation marks. Therefore, the number of extracted relationships was much smaller because the system could not detect correct sentences that contained information of kinship relations for the deceased person. Additionally, extracted relationships may have been unacceptable because the named entities could not be correctly recognized based on the wrongly segmented sentences.

Table 11. End-to-end system result.

The system extracts 6553 relationships for 2760 deceased persons.