2.1 WSD datasets for foreign languages

English being the most commonly used language worldwide has been the hotspot for NLP research. Different WSD datasets have been developed till now for English with different sizes and characteristics. SemCor (Miller et al. Reference Miller, Chodorow, Landes, Leacock and Thomas1994) is the largest available manually curated English sense-tagged dataset aligned with WordNet senses. It has about 234,000 tokens attached with syntactic category and appropriate sense. It has been extensively used by researchers for different NLP tasks. Some important research works in which SemCor is used include Song et al. (Reference Song, Ong, Ng and Lin2021); Luan et al. (Reference Luan, Hauer, Mou and Kondrak2020); Nodehi and Charkari (Reference Nodehi and Charkari2022). DSO Corus (Ng and Lee Reference Ng and Lee1996) is another WSD corpus with 192800 instances for a set of 191 target words among which 121 are nouns and 72 are verbs. This corpus has also been employed by different researchers for evaluating WSD systems. Patankar and Devane (Reference Patankar and Devane2017) utilized the DSO corpus to analyze the WSD problem in depth and its role to resolve the multilingual translation problem. Itankar and Raza (Reference Itankar and Raza2020) also utilized the DSO corpus for analyzing the role of WSD in machine translation by carrying out experiments on fifteen ambiguous words. MASC (Ide et al. Reference Ide, Baker, Fellbaum and Passonneau2010), a multilayered annotated dataset, stands for Manually Annotated Sub-Corpus. The corpus has 500000 lexical items with named entity annotations, part-of-speech tags, syntactic structure, etc. making it applicable in multiple NLP tasks like WSD, syntactic parsing, named entity recognition, etc. Wijeratne et al. (Reference Wijeratne, Balasuriya, Sheth and Doran2017) utilized MASC for WSD task for emoji sense discovery. de Lacalle and Agirre (Reference de Lacalle and Agirre2015) also used data from MASC to train the It Makes Sense (IMS) (Zhong and Ng Reference Zhong and Ng2010) WSD model and produced good results. In İlgen et al. (Reference İlgen, Adali and Tantuğ2012), a Lexical Sample Dataset for Turkish is developed. The dataset comprises of noun and verb sets with each 15 highly ambiguous words. For each ambiguous word, the dataset contains at least 100 instances collected from various online sources. In Rakho et al. (Reference Rakho, Laporte and Constant2012), WSD corpus is created for 20 polysemous French verbs. The corpus created has been annotated using four different sense inventories taking into account various types of evidences. OMSTI dataset (Taghipour and Ng Reference Taghipour and Ng2015) acronym for One Million Sense Tagged Instances is a huge sense tagged dataset and is a valuable asset for computational linguistics research. The dataset consists of one million English words, each of which has been sense-tagged according to its meaning in context. Its diversity makes it fruitful for resolving the English WSD problem. Le et al. (Reference Le, Postma, Urbani and Vossen2018) utilized this dataset to carry out WSD task using LSTM (Hochreiter and Schmidhuber Reference Hochreiter and Schmidhuber1997) algorithm. Kokane and Babar (Reference Kokane and Babar2009) used the OMSTI dataset to carry out the supervised WSD task based on neural networks. Kokane et al. (Reference Kokane, Babar and Mahalle2023) utilized the OMSTI dataset along with other two datasets to carry out the WSD experiments based on the adaptive WSD model. The results showed that the proposed WSD model produced better results when utilizing the OMSTI dataset. Authors in Scarlini et al. (Reference Scarlini, Pasini and Navigli2020) also created large-scale multilingual WSD datasets for five different languages to mitigate the knowledge-acquisition bottleneck faced in multilingual WSD. The dataset released contains about 15 million instances with sense annotations for five languages (English, German, Spanish, French, and Italian). It also contains five English datasets belonging to distinct semantic domains. The dataset is annotated automatically instead of using the manual annotation method and when tested on supervised methods showed better performance compared to other automatically created corpora. In Pasini et al. (Reference Pasini, Raganato and Navigli2021), cross-lingual evaluation framework for WSD, XL-WSD, covering sense-tagged datasets for 18 languages. The datasets so developed are then utilized for carrying out the WSD task implementing knowledge-based and neural network approaches. The results produced by these approaches are quite encouraging. In Zheng et al. (Reference Zheng, Li, Dai, Chen, Liu, Sun and Liu2021), Lexical-Sample WSD dataset is developed for Chinese. The developed corpus is then utilized to enhance the performance of the Chinese WSD task by utilizing word-formation knowledge. Kirillovich et al. (Reference Kirillovich, Loukachevitch, Kulaev, Bolshina and Ilvovsky2022) created a sense-tagged Russian corpus. The corpus is created manually with raw data obtained from OpenCorpora^{Footnote b} and senses from RuWordNet.^{Footnote c} It contains a total of 6751 sentences with 109,893 lemmas. The corpus is then utilized for testing the unsupervised WSD system. In Laba et al.(Reference Laba, Mudryi, Chaplynskyi, Romanyshyn and Dobosevych2023), a fine-tuned pretrained language model is presented for Ukrainian WSD. A WSD evaluation dataset is created to achieve this objective. The dataset has 2882 samples each having lemmas, possible meanings with examples for each meaning.

2.2 WSD datasets for Indian languages

In comparison to foreign languages, Indian languages have not been explored much in the NLP domain, and there are no such standard linguistic resources available for these languages. Although researchers have made several attempts to tackle the WSD problem in Indian languages, but the datasets used are usually small in size and private. Here we will discuss some important WSD datasets available in Indian languages and important attempts to handle the WSD problem in these languages. WSD in Hindi, the national language of India, has been explored by different researchers. A Lexical Sample Hindi WSD dataset is discussed in Singh and Siddiqui (Reference Singh and Siddiqui2016). The dataset has only 7506 for 60 ambiguous words. For this corpus, instances are reaped from Hindi Corpus^{Footnote d} and Internet and senses from Hindi WordNet.^{Footnote e} Based on this corpus, Hindi WSD task has been analyzed in research efforts presented in Singh and Siddiqui (Reference Singh and Siddiqui2015a), Singh and Siddiqui (Reference Singh and Siddiqui2015b); Singh et al. (Reference Singh, Siddiqui and Sharma2014). Singh and Siddiqui (Reference Singh and Siddiqui2015a) used two supervised algorithms, one based on the conditional probability of co-occurring words and another one is the Naïve Bayes algorithm to analyze the impact of karaka relations on Hindi WSD. Singh and Siddiqui (Reference Singh and Siddiqui2015b) compared the performance of three WSD algorithms trained on this dataset. The first algorithm used is corpus-based Lesk (Reference Lesk1986), the second algorithm is based on conditional probability and co-occurring words and the third algorithm is based on a classification information model. Singh et al. (Reference Singh, Siddiqui and Sharma2014) used this dataset to develop Hindi WSD based on the Naïve Bayes algorithm. In this different feature combinations have been explored to check their impact on the performance of the WSD system. In Sarmah and Sarma (Reference Sarmah and Sarma2016), WSD for Assamese is carried out based on supervised methodology. Naïve Bayes WSD model is trained on a manually created WSD corpus having 2.7K sentences, and it yielded 71% accuracy. Next, the size of the WSD corpus is increased to 3.5K sentences which enhanced the results by 7%. The WSD corpus is created utilizing Assamese corpus (Sarma et al. Reference Sarma, Bharali, Gogoi, Deka and Barman2012) and Assamese WordNet (Sarma et al. Reference Sarma, Medhi, Gogoi and Saikia2010). In Walia et al. (Reference Walia, Rana and Kansal2018), supervised methodology to WSD is adopted with reference to Punjabi language. The researchers created a WSD corpus for 100 ambiguous Punjabi words utilizing Punjabi Corpus obtained from ELRA^{Footnote f} and Punjabi WordNet (Kaur et al. Reference Kaur, Sharma, Preet and Bhatia2010). The same dataset has been utilized by other researchers for Punjabi WSD (Walia et al. Reference Walia, Rana and Kansal2020). Singh and Kumar (Reference Singh and Kumar2020) also created a sense-tagged corpus and employed deep learning techniques for WSD. The corpus created has 9251 instances for 66 ambiguous Punjabi words with 25.7 average number of instances per sense. In Saeed et al. (Reference Saeed, Nawab, Stevenson and Rayson2019), a benchmark Lexical Sample WSD corpus, ULS-WSD-18 Corpus, is developed for Urdu. UrMono Corpus Jawaid et al. (Reference Jawaid, Kamran and Bojar2014), the largest available raw Urdu corpus is utilized to obtain instances, and Urdu Lughat Board (Reference Board2008) is used as sense inventory. For each target word, there are 75 + 15n instances (n gives the number of senses for the target word) in the developed WSD corpus. The applicability of the developed dataset for the development and evaluation of the Lexical Sample WSD system for Urdu is then checked by using different supervised WSD algorithms, viz. Naïve Bayes, SVM, ID3 Quinlan (Reference Quinlan1986), k-NN. Different types of features like bag-of-words (BoW), most-frequent sense, part-of-speech, and word embeddings were used to train these models so as to analyze their role in WSD. In Sruthi et al. (Reference Sruthi, Kannan and Paul2022), unsupervised LDA-based WSD disambiguation methodology is employed to Malayalam. A WSD corpus with 1147 contexts is created for target words to carry out experiments. The experiments have shown better performance utilizing co-occurrence information and synonyms as features. In Das Dawn et al. (Reference Das Dawn, Khan, Shaikh and Pal2023), Bengali sense-annotated corpus for the Lexical Sample WSD task is created. The dataset covers a total of hundred ambiguous Bengali words. There are three to four senses used in the annotation process for each ambiguous word. For each sense of a particular word used in the annotation, there are ten paragraphs in the developed corpus. The authors also proposed a generic architecture for utilizing this dataset to analyze the WSD task in Bengali. In Patil et al. (Reference Patil, Bhavsar and Pawar2023), WSD for Marathi is investigated using BERT Tenney et al. (Reference Tenney, Das and Pavlick2019). The model is evaluated on the WSD dataset having 5285 instances for 282 dubious terms.

3.1 Raw corpus

Due to the deficiency of data in digital form for the Kashmiri language, we faced a huge challenge in managing the dataset to carry out this research work. We explored different resources available both online and offline to prepare the dataset for smooth conduction of research work. PoS tagged dataset produced by Lawaye and Purkayastha (Reference Lawaye. and Purkayastha2014) was obtained which contains about 120K tokens. Other options available for obtaining data include Kashmiri WordNet (Kak et al. Reference Kak, Ahmad, Mehdi, Farooq and Hakim2017), trilingual (English-Hindi-Kashmiri), and E-dictionary (Banday et al. Reference Banday, Panzoo, Lone, Nazir, Malik, Rasoool and Maqsood2009). In addition, we synthesized primary data wherever we found the deficiency. The corpus compiled from different resources is generic and has text from various domains like history, sports, politics, religion, and news.

3.2 Data preprocessing

The raw corpus written in the Perso-Arabic script, we collected in the first step which is not PoS tagged was passed through a few preprocessing steps so as to make it handy for further processing. The first step in the preprocessing we did was to tokenize it and divide it into sentences as it was too messy and the sentences were not properly separated. Using NLPTK library the dataset was tokenized. We stored the dataset in a separate text file (8 MB) with one sentence per line. As the data were collected from multiple resources, and there is no standard writing system in Kashmiri, the same word has different spelling in different resources. For example, the words and seem different but are the same with different spellings. After analyzing the raw corpus carefully, we resolved all such cases by taking spellings from Kashmiri WordNet as standard. PoS tagger developed by Lawaye and Purkayastha (2014) is employed on the non-PoS tagged data for PoS tagging with an accuracy of 94%. The file containing the PoS-tagged dataset has one token per line along with its PoS tag. A sample of PoS data is shown in Figure 2.

Figure 2.

PoS tagged instance extracted from PoS tagged dataset.

3.3 Target word selection

The ultimate product of the dataset preparation process is the sense-tagged dataset for the Kashmiri Lexical Sample WSD task. In order to achieve this objective target words are selected. We extracted ambiguous-word list present in the Kashmiri WordNet. The PoS-tagged dataset is then analyzed for the presence of these ambiguous words. We calculated the frequency of ambiguous words present in the dataset. Out of the ambiguous words present in the dataset, we selected 124 words that have the maximum frequency in the dataset for further processing. Not only frequency we also checked whether the word takes different senses in the sufficient number of instances in the dataset so as to make it good for training the WSD model for predicting different senses that the word can take in different situations. For example, the word (aamut) although is ambiguous and has a good frequency in the dataset still it was not chosen as the target word as it takes single sense in most of the sentences out of the three senses available in the Kashmiri WordNet for it. Based on the same reason, we dropped many other words from the target word list, even though they have good frequency in the dataset. Some of these words include (zyad), (aagur), (dil), (aham), (shamil), (istimall), and (amal). Table 1 gives important information about the ambiguous words selected as target words.

Table 1.

Target-words with total senses in Kashmiri WordNet and instances in annotated Dataset.

3.4 Sense inventory

After finalizing the target word list, we need possible senses that these words can take in different contexts. To get the different senses resources like machine readable dictionaries, Wikipedia, and WordNet have been used by researchers. In this research work, we had two resources available to us; Kashmiri WordNet and trilingual (English-Hindi-Kashmiri) E-Dictionary. We carefully analyzed the senses obtained from these resources to select the best option for sense inventory creation. It was observed that Kashmiri WordNet has a greater number of senses for many target words than are available in trilingual E-Dictionary. From Kashmiri WordNet, the range of senses for target words spans from 2 to 33 which is higher than the range of senses for these words available in trilingual E-Dictionary. For example, for the word (thud) having NOUN as a lexical category, Kashmiri WordNet gives 5 senses as shown in Table 2 below.

Table 2.

Senses for word (thud) in Kashmiri WordNet

In trilingual (English-Hindi-Kashmiri) trilingual E-Dictionary the word (Door) has 11 meanings. For the word (travun), Kashmiri WordNet has 33 meanings, whereas trilingual E-Dictionary returns 20 meanings for the word. There are 9 senses for the word (manun) in Kashmiri WordNet, whereas trilingual E-Dictionary contains only 7 senses for this word. Kashmiri WordNet gives 13 senses for the word (seud) for which trilingual E-Dictionary has only 7 senses. Likewise, there are 12 senses for the word (pakun) in Kashmiri WordNet, whereas trilingual E-Dictionary has only 2 senses for the same. There are many other cases where Kashmiri WordNet gives more senses than trilingual E-Dictionary; hence, we choose it for sense inventory creation. After choosing Kashmiri WordNet for sense inventory creation, we extracted senses for the target words from Kashmiri WordNet. Kashmiri WordNet actually divides the words in terms of concepts. The fundamental unit in Kashmiri WordNet is synset in the same manner WordNet for English and other languages. Each word is associated with a synset and has a synset ID, lexical category, concept, and example sentence. Kashmiri WordNet can be accessed from the Indo WordNet^{Footnote g} interface. Two important characteristics that we found in the senses available in the Kashmiri WordNet for the target words are a) the senses are very much fine-grained and b) it gives senses that are not commonly used. As far as our task is concerned, it is very much difficult to get instances for fine-grained senses. So, to overcome this problem, we adopted the same procedure as is followed by other researchers, i.e., convert these fine-grained senses into coarse-grained senses. This transformation involves grouping the fine-grained senses that are similar in meaning to a more general sense. This objective is achieved by following the sense compression technique proposed in Vial et al. (Reference Vial, Lecouteux and Schwab2019). The compression technique utilizes the different semantic relationships (hypernymy/hyponymy/synonymy/antonymy/meronymy/holonymy $\ldots$ ) available in Kashmiri WordNet. Synsets are divided into clusters C( $c_1$ , $c_2$ , $\ldots$ , $c_n$ ) based on these relationships. These clusters are formed iteratively. Initially, each synset from the synset list S( $s_1$ , $s_2$ , $\ldots$ , $s_n$ ) is placed in a separate cluster, i.e., C= (( $s_1$ ), ( $c_2$ ), $\ldots$ , ( $c_n$ )). Then at each iteration, we arrange C according to the sizes of the clusters, and then we analyze whether the smallest cluster $c_i$ is related to another cluster to $c_j$ . If the two clusters are related by having some common semantic relation, then we merge the two clusters $c_i$ , $c_j$ . If there is no common semantic link between the two clusters, then the merge operation is canceled and then we take the next smallest cluster and repeat the same process. The algorithm stops when there is no further link possible. After converting fine-grained senses into coarse-grained senses, we created a sense inventory. The sense inventory generated contains the target word and the set of senses finalized to be used in the annotation process for the target words. Figure 3 shows a snapshot of the sense inventory.

Figure 3.

Sense inventory snapshot.

3.5 Target word instances

The instances for the target words are extracted from the PoS-tagged dataset that we created after the preprocessing and PoS tagging discussed in Subsection 3.2. We obtained at least 75 + 15n (n is the number of senses for target word) number of sentences for each target word as per the guidelines set in the SenseEval-2 task for Lexical-Sample WSD task (Edmonds and Cotton Reference Edmonds and Cotton2001; Saeed et al. Reference Saeed, Nawab, Stevenson and Rayson2019). For example, the target word (taiz) has 8 senses used in annotation and has ( $75+15\times 8$ ) 195 instances in the final sense-tagged corpus.

3.6 Sense annotation

The next phase in the dataset preparation process was to carry out the annotation. We employed the manual sense annotation technique for this purpose. Three annotators all naïve speakers of Kashmiri were given this task after making them familiar with the annotation task. To achieve a high-quality annotated corpus, we followed the steps used in Saeed et al. (Reference Saeed, Nawab, Stevenson and Rayson2019). We divided the overall annotation process into two phases. Two annotators were given the task to annotate the target word with the appropriate sense as per the context. They were provided with files containing sentences separately for each target word and the senses to be used for annotation. Once the annotators finished the annotation process, they discussed the annotation task and resolved the cases where they found difficulty or confusion in the annotation. We then calculated the inter-annotator agreement and found it 92% and 0.83 weighted Kappa score (McHugh Reference McHugh2012) that indicated a high level of agreement. In the second phase of the annotation, the linguist expert was involved in reviewing the annotated dataset so as to make corrections wherever necessary.

3.7 XML encoding

The output of the preceding task is sense tagged corpus where the target words are tagged with the appropriate sense. This corpus contains simple text files one for each target word. In the next step, we transformed the sense-tagged corpus into XML format which is the commonly used format for storing sense-tagged datasets. This task is carried out using xml.etree. ElementTree module available in Python. The XML-encoded corpus has 124 total files one for each target word. Each file has <contextfile fileno=’filenumber’ filename=’kmr_word’>as the root element in which the fileno attribute specifies the file number (1,2, $\ldots$ 124) and the filename attribute which specifies the target word for which the file contains instances. Each sentence starts with <sentence s_id=”sentence_number”>element which indicates the start of a new sentence and the s_id attribute gives the sentence number assigned to the sentence in the file. Each word in the sentence is enclosed within the <wf>element pair which has pos as an attribute specifying the PoS category of the word. <wf>has sense_id as an additional attribute for target words which specifies the unique number assigned to the sense that fits the target word in the given sentence. Figure 4 below shows an instance of the target word (thud) in the sense annotated corpus.

Figure 4.

Example sentence from sense annotated corpus.

The sense-tagged dataset so created is finally divided into two partitions, training set and test sets for each target word. Splitting the dataset into training and test suites is governed by the following:

• Total instances for a particular word are divided into 80:20 ratio for training and test purposes.

• In order to make sure there is a proper mix of instances for each sense that a target word has in both training and test suites, 20% of instances for each sense are placed in the test suite and 80% instances in the training suite.

• Training and test splits for each target word are stored in separate XML file.

4.1 Most frequent sense

In natural languages, it is common that a polysemous word takes one sense more frequently than other senses it can have. This most frequently occurring sense of polysemous word is termed as MFS. In WSD systems, MFS-based approach is commonly used as a baseline to assess the performance of other machine learning techniques on a given dataset (Saeed et al. Reference Saeed, Nawab, Stevenson and Rayson2019; Kumar Reference Kumar2020). In this research work, we compared the results produced by other algorithms with MFS-based method.

4.2 Bag of words (Count vectorizing) model

BoW, also known as count vectorizing, is the basic model used to transform the raw data into a vector representation. In this approach, we calculate the prevalence of content words in the given data. This model converts the given input into numeric form in which different dimensions represent the words existing in the given input without taking into account the syntax or semantics. The dimensions may simply contain 0s or 1s representing the presence or absence of words in the given corpus or it may also represent frequency for that word in the given corpus. By default, this approach carries out some preprocessing steps like conversion of input to lowercase, utf-8 encoding, ignoring single characters, and punctuations. It also provides the user with the facility to customize tokenization, preprocessing, etc. so that it can be made helpful for languages other than English. In this study, we have implemented this approach for the Kashmiri language.

4.3 Word2Vec model

It is a deep-learning approach used to obtain dense vector representation for a given input. The vector representations computed using Word2Vec have proven to be fruitful in different downstream NLP tasks. Word2Vec requires a large corpus as input and generates the vocabulary of all the words in the corpus. Finally, dense embeddings are obtained for each word in the vocabulary. As vector representations produced by Word2Vec capture semantic and contextual information from the given input very well, many researchers have used this approach to solve the WSD problem (Kumari and Lobiyal Reference Kumari and Lobiyal2022; Saeed et al. Reference Saeed, Nawab, Stevenson and Rayson2019). There are two variants present in Word2Vec to produce vector representations; a) continuous bag-of-words (CBOW) and b) Skip-Gram. The CBOW variant uses the context words in a specific window size to forecast the middle word (target word). It works on the assumption that the order of words in the context widow is irrelevant to the prediction hence is named so. The objective of the Skip-Gram variant of Word2Vec is to forecast the context words within a specific window size for a target word. In this study, we employed the Skip-Gram architecture to obtain word embeddings (WE) for all the target words. The model is trained on the whole corpus with the objective of maximizing the probability of forecasting the context in which the target word exists. Considering the word sequence ( $w_1$ , $w_2$ , $\ldots$ , $w_n$ ), the objective function can be written as (Mikolov et al. Reference Mikolov, Sutskever, Chen, Corrado and Dean2013):

(1) \begin{equation} \frac{1}{T} \Sigma _{t=1}^{T} \Sigma _{-n \leq i\leq n,i \neq 0} \log p(x_t + i|x_t) \end{equation}

where n is training context size. The basic formulation of Skip-Gram defines $p(x_t+i|x_t)$ using softmax function :

(2) \begin{equation} p(w_O|x_I ) \equiv \frac{exp(v'w_O \intercal v_wI)}{\Sigma _{w=1}^{W}exp(v'w \intercal v_wI)} \end{equation}

vw and v’w depict input and output vector representations of w. w gives word count in vocabulary. We used the Gensim library to create the Word2Vec model. The model so created is used to obtain the nearest word embeddings for all target words keeping dimensions set to different values (100, 200, 300). Using these embeddings as features, we trained different machine learning algorithms and then tested these algorithms.

4.4 Classification algorithms used

Once the feature extraction process is over in the next step, we use these features to train machine learning algorithms. In this research work, we employed five classification algorithms which are discussed briefly here:

4.5 Evaluation

Experiments are carried out on all target words separately using classification algorithms discussed in the above section. To get a better overview of the performance reported by system and avoid complexity, we calculated the average of all the results produced by the system. To evaluate the performance of the WSD classifiers developed based on the created sense annotated corpus we used standard evaluation measures commonly used in machine learning, i.e., accuracy; precision, recall, and F1-measure.

Accuracy (A): Accuracy(A) is the accurate sense assignments made by the system over the total number of instances.

(3) \begin{equation} Accuracy(A) \equiv \frac{accurate assignments made}{total instances} \end{equation}

Precision (P): Precision (P) is obtained by dividing the correct sense assignments total assignments made by the system.

(4) \begin{equation} Precision(P) \equiv \frac{correct assignments}{total testset instances} \end{equation}

Recall (R): Recall (R) is obtained by dividing the correct assignments by the total expected assignments by the system.

(5) \begin{equation} Recall(R) \equiv \frac{accurate assignments made}{total assignment} \end{equation}

F1-measure: Obtained by using equation (6) and is the harmonic mean of P and R.

(6) \begin{equation} F1-measure \equiv \frac{2*P*R}{P+R} \end{equation}