2.1 Related work on available Turkish datasets

Turkish is an agglutinative language, where words are formed by productive affixation and multiple suffixes can be attached to a root to form a new word form:

Example 2.1.

gidebilirsek (in English, “if we can go”)

It is possible to generate an infinite number of words in Turkish, as shown by Sak et al. (Reference Sak, Güngör and Saraçlar2011). This poses a challenge for many NLP tasks, such as language modelling, spell checking, and machine translation, because of the out-of-vocabulary problem. Turkish grammar presents also other challenges, such as free word order in a sentence and the use of clitics (Oflazer, Reference Oflazer2014; Azin and Eryiğit, Reference Azin and Eryiğit2019), which may have various meanings depending on the context.

The datasets for many NLP tasks have been released mostly in English. The datasets released in Turkish are very limited. Current Turkish NLP studies generally focus on building datasets for syntactic parsing, such as the METU-Sabanci Treebank (Atalay et al., Reference Atalay, Oflazer and Say2003; Oflazer et al., Reference Oflazer, Say, Hakkani-Tür and Tür2003) and the IMST Dependency Treebank (Sulubacak et al., Reference Sulubacak, Eryiğit and Pamay2016). Although semantic annotations are crucial for NLP tasks, there are few studies on Turkish semantic annotation (Şahin and Adalı, Reference Şahin and Adalı2018; Azin and Eryiğit, Reference Azin and Eryiğit2019; Oral et al., Reference Oral, Acar and Eryiğit2024). One of the semantic datasets in Turkish is the Turkish Proposition Bank (PropBank) (Şahin and Adalı, Reference Şahin and Adalı2018), the first semantically annotated corpus built particularly for semantic role labelling (SRL). The other annotated dataset in Turkish is the AMR corpus presented by Azin and Eryiğit (Reference Azin and Eryiğit2019) and Oral et al., (Reference Oral, Acar and Eryiğit2024). Azin and Eryiğit (Reference Azin and Eryiğit2019) presented the preliminary investigation on Turkish AMR with $100$ annotated sentences obtained from the Turkish translation of the novel The Little Prince based on AMR specifications to demonstrate the differences between Turkish and English annotations. Oral et al. (Reference Oral, Acar and Eryiğit2024) extended the annotation procedure by converting the annotation process into a semi-automatic annotation using a rule-based parser in which PropBank (Şahin and Adalı, Reference Şahin and Adalı2018) sentences are converted into AMR graphs. Human annotators used the output of the rule-based parser to build the dataset rather than annotating it from the very beginning. The presented AMR corpus contains $700$ sentences ( $100$ sentences from the Turkish translation of the novel The Little Prince and $600$ sentences from the IMST Dependency Treebank (Sulubacak et al., Reference Sulubacak, Eryiğit and Pamay2016; Sulubacak and Eryiğit, Reference Sulubacak and Eryiğit2018; Şahin and Adalı, Reference Şahin and Adalı2018)).

The first Turkish UCCA dataset in Turkish was introduced by Bölücü and Can (Reference Bölücü and Can2022b), who also reported a limited set of initial rules for the UCCA annotation framework with $50$ annotated sentences. This study extends the Turkish UCCA dataset by providing a preliminary investigation of Turkish grammar and the effectiveness of using zero-shot learning with an external semantic parser for the UCCA annotation by increasing the dataset size and providing a more comprehensive guideline that covers all possible syntactic rules which will be sufficient to annotate a Turkish dataset using the UCCA framework. The annotated dataset will serve as a valuable semantic annotation resource, akin to the Turkish AMR dataset. The general rules, which are not language-specific, have the potential to be used in the annotation of other languages in the UCCA framework. Moreover, the annotated dataset can be used in other NLP tasks (Issa et al., Reference Issa, Damonte, Cohen, Yan and Chang2018; Naseem et al., Reference Naseem, Ravishankar, Mihindukulasooriya, Abdelaziz, Lee, Kapanipathi, Roukos, Gliozzo and Gray2021; Kapanipathi et al., Reference Kapanipathi, Abdelaziz, Ravishankar, Roukos, Gray, Astudillo, Chang, Cornelio, Dana and Fokoue-Nkoutche2021).

2.2 Related work on UCCA semantic parsing

UCCA parsing has been applied to various languages including English, German, and French. The proposed UCCA parsing approaches can be divided into four classes: (1) transition-based, (2) graph-based, (3) compositional-based, and (4) encoder–decoder-based. Transition-based approaches use new transition actions that handle features of the UCCA framework such as discontinuity and reentrancy. The first UCCA parser is a transition-based model called TUPA parser (Hershcovich, Abend, and Rappoport, Reference Hershcovich, Abend and Rappoport2017), which uses new transition rules (e.g., NODE, LEFT-EDGE, RIGHT-EDGE, LEFT-REMOTE, RIGHT-REMOTE, and SWAP) with new features (e.g., separating punctuation and gap type). Hershcovich et al. (Reference Hershcovich, Abend and Rappoport2018) extend the TUPA parser with multi-task learning using a set of graph-based frameworks, namely AMR (Banarescu et al., Reference Banarescu, Bonial, Cai, Georgescu, Griffitt, Hermjakob, Knight, Koehn, Palmer and Schneider2013), UD (Universal Dependencies) (Nivre et al. Reference Nivre, De Marneffe, Ginter, Goldberg, Hajic, Manning, McDonald, Petrov, Pyysalo and Silveira2016, Reference Nivre, de Marneffe, Ginter, Hajic, Manning, Pyysalo, Schuster, Tyers and Zeman2020), and PSD (Oepen et al., Reference Oepen, Kuhlmann, Miyao, Zeman, Cinková, Flickinger, Hajic, Ivanova and Uresova2016). Pütz and Glocker (Reference Pütz and Glocker2019) implement the TUPA parser with a set of features based on the top three items on the stack and the buffer, as well as deep contextualised word embeddings of the rightmost and leftmost parents and children of the respective items by using additional training data. With the popularity of graph-based semantic frameworks, shared tasks have been conducted in several workshops such as SemEval (Hershcovich et al. Reference Hershcovich, Aizenbud, Choshen, Sulem, Rappoport and Abend2019b) and MRP (Oepen et al. Reference Oepen, Abend, Hajic, Hershcovich, Kuhlmann, O’Gorman, Xue, Chun, Straka and Urešová2019, Reference Oepen, Abend, Abzianidze, Bos, Hajic, Hershcovich, Li, O’Gorman, Xue and Zeman2020), which also introduced new transition-based methods, many of which are extensions of the TUPA parser (Lai et al. Reference Lai, Lo, Leung and Leung2019; Arviv, Cui, and Hershcovich, Reference Arviv, Cui and Hershcovich2020). Bai and Zhao (Reference Bai and Zhao2019) extend the TUPA parser with hand-crafted features. Dou et al. (Reference Dou, Feng, Ji, Che and Liu2020) use Stack-LSTM instead of BiLSTM used in TUPA parser, and Lyu et al. (Reference Lyu, Huang, Khan, Zhang, Sun and Xu2019) use TUPA $_{BiLSTM}$ and TUPA $_{MLP}$ as a cascaded parser with a multi-stage training procedure, first training TUPA $_{BiLSTM}$ and retraining the model with TUPA $_{MLP}$ .

Graph-based approaches (Cao et al. Reference Cao, Zhang, Youssef and Srikumar2019; Koreeda et al. Reference Koreeda, Morio, Morishita, Ozaki and Yanai2019; Droganova et al. Reference Droganova, Kutuzov, Mediankin and Zeman2019) aim to generate a graph with the highest score among all possible graphs in the graph space, which can be seen as a graph-searching problem. The most common approach tackles the task as a constituency parsing problem (Jiang et al. Reference Jiang, Li, Zhang and Zhang2019; Li et al. Reference Li, Zhao, Zhang, Wang, Utiyama and Sumita2019; Zhang et al. Reference Zhang, Jiang, Xia, Cao, Wang, Li and Zhang2019; Cao et al., Reference Cao, Zhang, Youssef and Srikumar2019; Bölücü et al., Reference Bölücü, Can and Artuner2023), where directed acyclic graphs (DAGs) are converted into constituency trees. For example, Bölücü et al. (Reference Bölücü, Can and Artuner2023) adopts self-attention mechanism proposed by Kitaev and Klein (Reference Kitaev and Klein2018b) to learn UCCA semantic representations, with zero- and few-shot experiments on different languages.

Composition-based approaches follow the compositionality principle and perform semantic parsing as a result of a derivation process that incorporates both lexical and syntactic-semantic rules to develop a semantic graph parser (Che et al. Reference Che, Dou, Xu, Wang, Liu and Liu2019; Donatelli et al. Reference Donatelli, Fowlie, Groschwitz, Koller, Lindemann, Mina and Weißenhorn2019; Oepen and Flickinger, Reference Oepen and Flickinger2019). Donatelli et al. (Reference Donatelli, Fowlie, Groschwitz, Koller, Lindemann, Mina and Weißenhorn2019) apply the AM (Abstract Meaning) dependency parser of Lindemann et al. (Reference Lindemann, Groschwitz and Koller2019) after converting UCCA annotations into AM dependency graphs.

Encoder–decoder approaches use an encoder–decoder architecture to convert an input sentence into a semantic graph as performed in neural machine translation (NMT) (Na et al. (Reference Na, Min, Park, Shin and Kim2019); Yu and Sagae (Reference Yu and Sagae2019); Kitaev and Klein (Reference Kitaev and Klein2018a); Ozaki et al. (Reference Ozaki, Morio, Koreeda, Morishita and Miyoshi2020)).

For the annotation, we utilised the semantic parser that adopts a graph-based approach proposed by Bölücü et al., (Reference Bölücü, Can and Artuner2023) in which the effectiveness of zero- and few-shot experiments on various languages has been already demonstrated. This parser has been also applied to the annotation of the initial Turkish UCCA dataset (Bölücü and Can, Reference Bölücü and Can2022b).

3.1 UCCA categories

The foundational layer views the text as a collection of Scenes. A Scene describes a movement, an action, or a temporally persistent state. It usually contains information regarding when the Scene happens, the location, and the ground that explains how it happens.

3.2 Remote and implicit units

While some relations are clearly described in the text, there are instances where a sub-unit in a given unit is not explicitly mentioned:

1. 1. If the missing entity is referenced from another position in the text, we add a reference to the missing unit, which is labelled as a REMOTE unit (the minimal unit is used as REMOTE).

2. 2. When the missing entity does not appear in any place in the text, we add an IMPLICIT entity that stands for the missing sub-unit.

The reader may refer to the actual English guideline for an elaborate presentation and example annotations for various scenarios (Abend et al. Reference Abend, Schneider, Dvir, Prange and Rappoport2020).Footnote g We provide further examples in Turkish as part of the Turkish guideline in Appendix A.

4.1 Morphology

Turkish is an agglutinative language where suffixes are attached to word roots or stems to form words, known as suffixation. Therefore, the vast majority of Turkish words contain more than one syllable (Lewis Reference Lewis1967).

Morphemes are defined as the smallest meaning-bearing units in a language. There are two types of morphemes, depending on whether they are attached to a word or stand alone in a sentence: free morphemes, also called unbounded morphemes can stand alone as if they are words, bound morphemes can only be seen attached to a word. Morphemes can be further analysed in two categories, depending on whether they modify the grammatical role (i.e., part-of-speech (PoS)) and the fundamental meaning of a word.

Derivational morphemes. Derivational morphemes are bound morphemes that have the ability to modify the meaning and the PoS of a word (e.g., “boya” (the paint) - “boya-cı” (the painter)).

Inflectional morphemes. Inflectional morphemes are bound morphemes attached to nouns, pronouns, nominal phrases, verbs, and verb frames that modify functional relations such as case, tense, voice, mood, person, and number.

They are divided into two categories:

1. 1. Nominal inflectional morphemes indicate number, possession, and case (e.g. “çocuk” (kid) -lar (number) -ın (possession)-a (case) - “çocuklarına” (to your children)).

2. 2. Verbal inflectional morphemes attach to verbs in two ways: finite verb forms and non-finite verb forms. A finite verb acts as a verb in a sentence and a non-finite verb acts as a nominal in a sentence. The ones that are attached to finite verbs such as voice suffixes, negative markers, tense, and modality markers are not separately labelled but only as a whole verb, which usually refers to an action in the annotation.

   Example 4.3. Her şey bitmişti , anlamıştım . (in English, “It had ended , I had realised .”, i.e. “bit-” (to end), “bitmiş” (ended), “bitmişti” (had ended), “anla-” (to realise), “anlamış” (realised), “anlamıştım” (had realised), where the final verbs refer to an action in the sentence.)

   Copula markers are one of the peculiarities in Turkish grammar (Lewis Reference Lewis1967). A copula is a connecting word or phrase in a particular form of a verb that links a subject and a complement.

   Subordinating suffixes appear only in non-finite verb forms and are the main means of forming subordinate clauses in Turkish. They are combined with verb stems to form nominals (e.g., “Okula git-me-yeceğ-i belli.” (It is clear [that s/he won’t go to school] .)).

Reduplication. Reduplication is the repetition of a word or part of a word. There are three types of reduplication in Turkish: emphatic reduplication, m-reduplication, and doubling.

1. 1. Emphatic Reduplication: It is used to emphasise the quality of an adjective (“uzun” (long) $\rightarrow$ “upuzun” (very long)).

2. 2. m-Reduplication: In this form, a word is duplicated by adding “m” letter to the beginning of the second word either by replacing the first consonant with “m” or adding “m” if the word begins with a vowel (“etek” (skirt) $\rightarrow$ “Etek metek” (skirt and the like)).

3. 3. Doubling: In this form, a word is doubled with the same form of the word (“koş-” (to run) $\rightarrow$ “koşa koşa” (“in a hurry” or “willingly”, depending on the context)).

4.2 Syntax

Turkish has a free-order sentence structure with features such as head-final, pro-drop, and subject-verb agreement. Here, we briefly describe the grammatical structure of Turkish, which will help to understand the UCCA annotation guideline proposed for Turkish in this study.

Constituents of a sentence. In Turkish, sentences can be simple or complex in terms of the subordinate clauses that are semantically dependent on the main clause.Footnote j Two main components of a simple sentence are subject and predicate. In a sentence, the predicate expresses an event, process, or state of affairs with an agreement of the subject, and the subject of the sentence may be a person, place, or thing that possesses or is affected by the predicate:

Although Turkish is a syntactically free-order language, the unmarked order (neutral word order) is subject-(object)-predicate (SOV) (e.g., “Ahmet okula gitti.” (in English, “Ahmet went to the school.”)).

The sentence types are divided into two classes based on the predicate:

1. 1. Verbal sentences have predicates that are finite verbs and they indicate a process in a given sentence:

   Example 4.12. Kerem bana baktı . (in English, “Kerem looked at me.”)

2. 2. Nominal sentences have predicates that do not contain a verb or use a verb in the form of a copula (e.g., ol- (be)). They indicate a state in a given sentence:

   Example 4.13. Bir tutsağ ım ben. (in English, “I am a prisoner.”)

   Example 4.14. Ahmet doktor ol acaktı. (in English, “Ahmet was going to be a doctor.”)

Phrases. A phrase is a syntactic unit that is a collection of syntactically coherent words that function as a unit in a sentence and can be nested. For example, VPs can contain NPs. There are five types of phrases: verb phrases, noun phrases, adjectival, adverbial, and postpositional phrases.

1. 1. A verb phrase consists of a verb and additionally may contain a complement of a verb or an adverbial. A verb usually refers to a process in a sentence.

   Example 4.15. Ahmet suyu soğuk soğuk içti . (in English, “Ahmet drank the cold water too fast .”)

2. 2. A noun phrase primarily contains a noun or a group of words containing a noun. The role of a noun phrase is that of a subject and some kind of complement such as an object, subject complement, and complement of a postposition.

   Example 4.16. Ahmet geldi. (in English, “ Ahmet came.”)

   Example 4.17. [Ahmet’in sevdiği] ekşi elma bitmişti. (in English, “ [Ahmet’s favorite] sour apple was finished.”)

   Noun phrases usually correspond to participants (e.g., “Ahmet’in sevdiği ekşi elma”) in a sentence where they may have their own participants (e.g., “Ahmet”) and processes (e.g., “sevdiği”) in the phrase and may constitute a sub-scene in the entire sentence.

3. 3. An adjective phrase contains a group of words that act as modifiers of nouns, providing additional information about the quality, characteristics, or attributes of the nouns they describe.

   Example 4.18. Bu restoran, şehirdeki en iyi yemekleri sunuyor. (in English, This restaurant offers the best meals in the city )

4. 4. An adverb phrase is composed of a group of words that act as modifiers of adjectives, verbs, or other adverbs to provide additional information about the cause, effect, space, time, and condition.

   Example 4.19. Oldukça yavaş koştu. (in English, S/he ran quite slowly )

   Example 4.20. Sabah evde çalışacağım. (in English, I will work at home in the morning )

5. 5. A postpositional phrase is usually composed of a noun phrase and a postposition that follows the noun phrase. While the postposition is the head of a phrase, the noun phrase is the complement of the phrase:

   Example 4.21. Akşama doğru yağmur yağabilir. (in English, “It may start raining towards evening.” )

Depending on its meaning, a postpositional phrase may correspond to an adverbial clause in a sentence.

Complex sentences and subordination. Complex sentences contain at least one subordinate clause in addition to the main clause, where a subordinate clause is a group of words that functions as a unit in a sentence. In Turkish, there are three types of clauses: noun, relative, and adverbial clause.

1. 1. Noun Clause occurs as a noun phrase in a complex sentence as the subject or object in the sentence:

   Example 4.22. Ahmet [senin okula gittiğini] bilmiyordu. (in English, “Ahmet didn’t know that [you were going to school].” )

2. 2. Relative Clause occurs as an adjectival phrase that is used to modify noun phrases:

   Example 4.23. [Öğretmen olan] oğlu, Ankara’da yaşıyor. (in English, “Her/His son, [who is a teacher] , lives in Ankara.”)

3. 3. Adverbial Clause occurs as an adverbial phrase that expresses the verb in terms of time, place, manner, and degree:

   Example 4.24. [Kedimiz FC;ş FC;mesin diye] ısıtıcıyı açtık. (in English, “We turned on the heater [so that our cat would not get cold] .”)

Conditional sentences. Conditional sentences are based on a condition that is met when a certain action takes place.

4.3 Closed-class words

Closed-class words in Turkish, also known as function words, are crucial for constructing sentences in Turkish, providing grammatical structure, and conveying relationships between different elements in a sentence. They are considered closed-class because they are relatively stable and less prone to change compared to open-class words such as nouns and verbs. Although they fall into pre-existing UCCA categories, we describe them separately because they are language-specific. The closed-class word categories in Turkish are as follows:

Pronouns. Pronouns refer to the persons, objects, or situations mentioned above, their references being clear from the context or only partially specified. They have different forms based on inflectional suffixes added to them, such as number and possession.

Postpositions. Postpositions are closed-class words that follow a noun or a pronoun to indicate its relationship to another element in a sentence. They also indicate relationships such as location, direction, and possession.

Clitics. Clitic is a type of bound morpheme that functions similarly to an affix but is usually in the form of a free morpheme (e.g., mI, dA, ya, ki and bile, ile). They don’t have stress of their own and depend on the host word for their pronunciation.

Conjunctions and discourse connectives. In Turkish, conjunctions and discourse connectives are used to link two or more items that have the same syntactic function. While conjunctions are used to link phrases, subordinate clauses, or sentences, discourse connectives are used to link sentences.

5.1 Dataset

METU-Sabanci Turkish Treebank (Atalay et al., Reference Atalay, Oflazer and Say2003; Oflazer et al., Reference Oflazer, Say, Hakkani-Tür and Tür2003) is a morphologically and syntactically annotated treebank. It consists of $7262$ sentences taken from $10$ different genres, that is, narratives, memories, research papers, travel writings, diaries, newspaper columns, articles, short stories, novels, and interviews. Since the corpus is annotated at both lexical and syntactic levels, it is one of the most valuable syntactically annotated datasets in Turkish. By adding graph-based semantic representations of the sentences in the same dataset, we aim to make the dataset more accessible for other NLP tasks that require semantic information as well as syntactic and morphological information.

A sample annotation from the METU-Sabanci Turkish Treebank is given in Table 1 for Example 5.1. As seen, each sentence involves PoS tags, dependencies, dependency labels, as well as morphological tags of each word. The UCCA annotation of the sentence is illustrated in Figure 3.

Table 1. A sentence “Ama hiçbir şey söylemedim ki ben sizlere” (in English, “But I didn’t say anything to you”) in the METU-Sabanci Turkish Treebank (Atalay et al., Reference Atalay, Oflazer and Say2003; Oflazer et al., Reference Oflazer, Say, Hakkani-Tür and Tür2003). The columns correspond to the positions of the words within the sentence, surface forms, lemmas, parts-of-speech (PoS) tags, morphological features separated by $|$ , head-word indices (index of a syntactic parent, 0 for ROOT), and syntactic relationships between HEAD and the word, respectively.

Example 5.1.

5.2 Self-attentive semantic parser

We utilised an external semantic parser to perform the annotation with a semi-automatic approach that significantly reduced the annotation effort. For this purpose, we used the semantic parser proposed by Bölücü et al., (Reference Bölücü, Can and Artuner2023). The model is based on an encoder–decoder architecture that tackles parsing as a chart-based constituency parsing problem. In the encoder, self-attention layers are used along with two multi-layer perception (MLP) classifiers, two fully connected layers, and a non-linear activation function ReLU in the output layer. The output layer of the encoder produces the per-span scores where spans correspond to the constituents in the constituency tree. The CYK (Cocke-Younger-Kasami) algorithm (Chappelier and Rajman, Reference Chappelier and Rajman1998) is used in the decoder to generate the tree with the maximum score using the scores that are produced by the encoder. The architecture of the model is given in Figure 4.

Figure 3. UCCA Annotation of “Ama hiçbir şey söylemedim ki ben sizlere” (in English, “But I didn’t say anything to you”)

Figure 4. An overview of the external semantic parser

The input given to the encoder is a sentence $s= \{w_1, \cdots, w_n\}$ , where each word $w_t$ is mapped into a dense vector that is the concatenation of multilingual contextualised embeddings $con_t$ and syntactic embeddings (i.e., PoS tags, dependency tags, entity types, and entity iob types)—extracted from an off-the-shelf open-source NLP library— which are word embeddings $e_{w_t}$ , PoS tag embeddings $e_{p_t}$ , dependency label embeddings $e_{d_t}$ , entity type embeddings $e_{e_t}$ , and entity iob (inside-outside-beginning) category embeddings $e_{e_ob_t}$ :

(1) \begin{eqnarray} x_t &=& e_{w_t} \oplus e_{p_t} \oplus e_{d_t} \oplus e_{e_t} \oplus e_{e_ob_t} \end{eqnarray}

(2) \begin{eqnarray} w_t &=& x_t \oplus con_t \end{eqnarray}

The output of the semantic parser model is a UCCA (Abend and Rappoport, Reference Abend and Rappoport2013a) representation of the sentence in the form of a directed graph.

5.3 Dataset annotation

As stated in the study, dataset annotation consists of two steps: (1) obtaining a partially annotated dataset using an external semantic parser (see Section 5.2), and (2) refining the partially annotated dataset by human annotators. Here, we explain the steps of annotating the Turkish UCCA dataset.

5.4 Inter-annotator agreement

We employed two annotators who are native Turkish speakers and fluent in English and have expertise and knowledge in computational linguistics. The annotators were initially trained for UCCA annotation based on the official UCCA guideline (Abend and Rappoport, Reference Abend and Rappoport2013a). The annotation step was performed individually, which was followed by a comparison phase in which the two annotations were compared to each other to identify annotation agreements. To show how clear our annotation guideline is and how consistently it was understood by the annotators, we used accuracy and Cohen’s kappa ( $\kappa$ ) (Cohen, Reference Cohen1960). There was disagreement on $616$ edges out of $3,981$ edges in a total of $400$ sentences. Thus, the disagreement between the two annotators is $15.47\%$ and Accuracy is $84.53\%$ based on the annotated tokens. Accuracy does not consider the expected chance of agreements that are likely to occur. Therefore, we also calculated Cohen’s kappa ( $\kappa$ ), a statistical measure of the reliability of annotations between different annotators. Cohen’s kappa ( $\kappa$ )(*100) score for the annotation is $82.29$ . The scores between $80$ and $90$ indicate strong agreement between the annotators. The general disagreement that recurs in the training procedure mainly concerns the annotation of the clitics (e.g., mI, dA).

Figure 5. Confusion matrix for the outputs in partial annotation (predicted) and refined annotation (gold). Category abbreviations: A: Participant, C: Center, D: Adverbial, E: Elaborator, F: Function, G: Ground, H: Parallel Scene, L: Linker, N: Connector, P: Process, R: Relator, S: State, U: Punctuation

5.5 Comparison of the outputs in partial and refined annotation

Figure 5 illustrates the confusion matrix acquired from the outputs of the partial annotation (predicted) and refined annotation (gold). We analyse the results of the semantic parser model, which correspond to the partial annotations, to identify the most commonly corrected errors in the refined annotation. We added all $IMPLICIT$ edges manually because the model cannot predict the $IMPLICIT$ edges. Typically, the external parser model does not confuse the Punctuation (U) label, as it is a straightforward task to recognise punctuation. However, the model often has difficulty distinguishing between labels such as Participant (A), Center (C), Adverbial (D), Elaborator (E), and State (S). This makes sense because Turkish allows the creation of new words by adding derivational morphemes, and words with these labels may share common roots. Other commonly confused labels include Relator (R), Ground (G), and Linker (L), which are known to be challenging even for human annotators. In addition, Relator (R) presents complexity due to the presence of clitics (please see Section 4 for the relevant rules), which leads the semantic parser to occasionally misclassify it as Ground (G), Parallel Scene (H), or Linker (L). Finally, despite the absence of any words labelled as Connector (N) in the refined annotation dataset, the model annotated some words as Connector (N) possibly due to their similarity to words in other languages (English, German, and French).

Two example sentences parsed with the external semantic parser are given in Figure 6 and 7 along with their gold (refined) annotations. For the first sentence in Figure 6, we added only an IMPLICIT edge based on the annotation rule for pronoun-dropping. The model cannot learn the implicit edges, which are all added manually during annotation. Morphological and more contextual information (possibly in paragraph level) should be provided to learn such implicit relations. For the second sentence given in Figure 7, most of the labels were incorrect and therefore had to be corrected manually.

Figure 6. The semantic parse tree obtained from the semantic parsing model and the gold annotation obtained from the manual annotation of the sentence “(O) Yerinden kalkmıştı.” (in English, “S/he had stood up.”). Category abbreviations: H: Parallel Scene, A: Participant, P: Process, U: Punctuation

Figure 7. The semantic parse tree obtained from the semantic parsing model and the gold annotation obtained from the manual annotation of the sentence “(Sen) Kurtulmak istiyor musun oğlum? diye sordu Şakir.” (in English, “Do you want to be saved son? asked Şakir.”). Category abbreviations: H: Parallel Scene, D: Adverbial, C: Center, U: Punctuation, R: Relator, P: Process, A: Participant, F: Function, G: Ground

When correcting the annotations obtained from the semantic parser model, we made almost no additional corrections for the short sentences (with less than five words). The labels of the majority of the terminal nodes were also mostly correct. We corrected most of the annotations with a Parallel Scene and, for that matter, the entire annotation of the sentence.

5.6 Annotation Statistics

We provide the statistical distributions in the annotated dataset with a comparison to other annotated datasets in English, German, and French (see Table 2). In particular, we provide the proportions of the edges and the labels in the final annotated dataset.

Table 2. Proportions of the edges and labels as well as the number of sentences and tokens in the UCCA datasets in Turkish, English, French, and German. The statistical details of English, French, and German datasets are taken from Hershcovich et al. (Reference Hershcovich, Aizenbud, Choshen, Sulem, Rappoport and Abend2019b).

The average sentence length in the Turkish dataset is comparably shorter than that of the other languages. This is due to the morphological complexity of the language although the semantic complexity and relations are included but encoded differently. Turkish sentences in the dataset are generally syntactically simple and involve one predicate and one subject. The average length of the sentences is $6.19$ (# of tokens/ # of sentences), the average number of edges is $9.95$ (# of edges/ # of sentences), and finally, the average number of IMPLICIT edges in the dataset is $0.89$ (# of IMPLICIT edges/ # of sentences).

6.1 Experimental details

6.1.1 Datasets for external parser

For training the external semantic parser (Bölücü et al., Reference Bölücü, Can and Artuner2023), we used the combination of English, German, and French UCCA datasets released in SemEval-2019 (Hershcovich et al. Reference Hershcovich, Aizenbud, Choshen, Sulem, Rappoport and Abend2019b). The details of the datasets are given in Table 3.

Table 3. The number of sentences in each UCCA-annotated dataset provided by SemEval 2019 (Hershcovich et al. Reference Hershcovich, Aizenbud, Choshen, Sulem, Rappoport and Abend2019b)

6.2 Results

The experimental results of zero-shot and few-shot learning are given in Table 4. As we can see, the model struggles to predict the remote edges, including unlabelled edges, which is considered an easier task than predicting labelled edges in zero-shot learning. However, when a limited Turkish dataset is used to train the parser model in the few-shot learning setting, the model shows effectiveness in predicting remote edges. Although the amount of Turkish training data is small compared to the datasets in the other languages (English, German, and French) used for training in few-shot learning, the model shows a remarkable improvement compared to the results obtained in the zero-shot setting. In addition to better results, few-shot learning proves to make the parser model more stable (with lower variance) across folds compared to zero-shot learning. This shows that adding Turkish data improves the generalisability of the parser model. Finally, although the performance of the unlabelled remote edge prediction is higher than the labelled one, the improvement is not significant, showing the challenge of Remote edge prediction in Turkish.

Table 4. F-1 results obtained from zero-shot and few-shot learning on the Turkish UCCA dataset. Avg is the macro average of F1 metric. $\uparrow$ means a statistically significant improvement over the zero-shot learning.

Figure 8. Results obtained from few-shot learning according to their sentence length

6.3 Error analysis

6.3.1 Sentence length

In Figure 8, we present the labelled and unlabelled results obtained by the parser trained in the few-shot learning setting for different sentence lengths. It can be seen that the model performs better in predicting shorter sentences which make up the majority of the UCCA dataset compared to the other datasets in English, German, and French. In addition, shorter Turkish sentences are mostly simple sentences consisting of a Participant (A), a Process (P) or State (S), and a Punctuation (U) such as “ $\langle $ (Sen) $\rangle$ Anlattın.” (in English, “You told.”) and “ $\langle $ (Ben) $\rangle$ Biliyorum.” (in English, “I know.”), making it easier for the parser to predict UCCA categories for shorter sentences. These UCCA labels are also the most frequent categories in the datasets (see Table 2). However, the performance of the parser for longer sentences is superior in unlabelled prediction, showing that predicting UCCA categories is a challenge in semantic parsing. In other words, even if we parse a sentence correctly, we may not be able to predict the correct UCCA category of the tokens. Since the labelled results of shorter sentences are remarkably high, we could not see a big gap between labelled and unlabelled results in short sentences. However, it becomes noticeable in longer sentences due to the lower performance of the UCCA parser on these sentences.

The results for different sentence lengths presented in Figure 8 show the reason for the improvement in the unlabelled results. The prediction of unlabelled UCCA representation of longer sentences is more accurate than for the labelled ones and has more impact on the improvement of the unlabelled results given in Table 4.

Figure 9. The semantic parse tree obtained from the semantic parsing model and the gold annotation obtained from the manual annotation of the sentence “(O) Evet, dedi çaresizlikle.” (in English, “S/he said yes with desperation.”). Category abbreviations: H: Parallel Scene, D: Adverbial, G: Ground, U: Punctuation, P: Process, A: Participant

6.4 Discussion

In the literature, semantic representations are widely employed in NLP and NLU applications that require comprehension of text (Liu et al., Reference Liu, Flanigan, Thomson, Sadeh and Smith2015; Issa et al., Reference Issa, Damonte, Cohen, Yan and Chang2018; Kapanipathi et al., Reference Kapanipathi, Abdelaziz, Ravishankar, Roukos, Gray, Astudillo, Chang, Cornelio, Dana and Fokoue-Nkoutche2021), making annotated datasets crucial resources. However, the annotation of datasets is a labour-intensive and expensive process, demanding expertise and strict adherence to annotation guidelines.

Figure 10. The semantic parse tree obtained from the semantic parsing model and the gold annotation obtained from the manual annotation of the sentence, “(Ben) Kurtulup buraya gelmeyi başardım.” (in English, “I managed to escape and come here.”). Category abbreviations: H: Parallel Scene, D: Adverbial, U: Punctuation, P: Process, A: Participant

Figure 11. The semantic parse tree obtained from the semantic parsing model and the gold annotation obtained from the manual annotation of the sentence “(Sen) Kaçıp kurtulmak istedin.” (in English, “You wanted to escape and get away.”). Category abbreviations: H: Parallel Scene, U: Punctuation, P: Process, A: Participant

Figure 12. The semantic parse tree obtained from the semantic parsing model and the gold annotation obtained from the manual annotation of the sentence, “(O) Onu elinden kaçırmış, bir başka erkeğe kaptırmıştı.” (in English, “S/he missed her/him, s/he had lost her/him to another man.”). Category abbreviations: H: Parallel Scene, D: Adverbial, F: Function, U: Punctuation, P: Process, E: Elaborator, C: Center, A: Participant

Figure 13. The semantic parse tree obtained from the semantic parsing model and the gold annotation obtained from the manual annotation of the sentence “Geldik! diye bağırdı Kerem.” (in English, “Kerem shouted that we had arrived.”). Category abbreviations: H: Parallel Scene, R: Relator, U: Punctuation, P: Process, A: Participant

UCCA facilitates knowledge transfer across languages. Bölücü et al. (Reference Bölücü, Can and Artuner2023) show the efficacy of zero-few and few-shot learning, demonstrating that a semantic parser trained in one language can perform reasonably well in another, even when linguistic structures differ. Motivated by this, we propose a pipeline annotation procedure that employs an external semantic parser trained on UCCA datasets from various languages to generate a partially annotated dataset in Turkish. Despite the linguistic differences, we successfully transfer cross-lingual features, exemplified by achieving an F-1 unlabelled score of $85.9\%$ for Turkish in the zero-shot setting. The substantial reduction in total annotation time, approximately $1/3$ rd of the expected duration, justifies the inclusion of an external semantic parser in the annotation process.

In the context of Turkish UCCA annotation, previous work (Bölücü and Can, Reference Bölücü and Can2022b) explored a similar approach on a smaller corpus, while our study introduces a larger dataset of $400$ sentences. The guideline provided in the prior study covered only a subset of rules, insufficient for fully annotating a Turkish dataset within the UCCA framework. In contrast, our more comprehensive rules (Section 4) encompass all possible syntactic rules necessary for UCCA annotation in Turkish that involve both language-specific closed-class words and more generic rules applicable to other syntactically similar languages. Therefore, the present study significantly expands upon previous research, offering a detailed analysis of results and insights applicable to researchers working on semantic annotation in other languages.

A recognised limitation is the involvement of an external semantic parser and the need to analyse the outputs of the parser (to obtain a partially annotated dataset), which complicates a fully automatic annotation process. For researchers venturing into new languages, it may be required to define rules that may deviate from the English guideline. However, extending the training to new languages can potentially streamline the annotation time for building diverse language models, and our guideline will be a potential guideline for use in such other language annotations.