2.1 Corpus-based decompounding methods

Koehn and Knight (Reference Koehn and Knight2003) proposed a corpus-based decompounding model in German-English statistical machine translation. They used the geometric mean of word frequency to detect the splitting point location. They pointed out that decompounding algorithms improve the efficiency of machine translation. Leveling et al. (Reference Leveling, Magdy and Jones2011) investigated a corpus-based decompounding model in German-English cross-language patent retrieval. They noticed that the decompounding model reduces the OOV and improves the effectiveness of the IR system. Hollink et al. (Reference Hollink, Kamps, Monz and De Rijke2004) evaluated the effect of stemming, lemmatisation, and decompounding algorithms in eight European languages. They observed that the corpus-based decompounding model improves retrieval effectiveness in Dutch, Finnish, German, and Swedish languages. The decompounding algorithm improves an mean average precision (MAP) score of 4–18.7% in different morphologically rich European languages. Airio (Reference Airio2006) examined the impact of stemming, lemmatisation, and decompounding methods on monolingual and bilingual retrieval. They noticed that the decompounding algorithm does not improve effectiveness in English, Finnish, German, and Swedish monolingual retrieval. However, the decompounding method enhances effectiveness in English to Dutch, Finnish, German, and Swedish bilingual retrieval. Rigouts Terryn et al. (Reference Rigouts Terryn, Macken and Lefever2016) investigated the impact of a decompounding module on the Dutch hypernym detection. They observed that the decompounding module improves precision and recall.

Braschler and Ripplinger (Reference Braschler and Ripplinger2004) evaluated the effects of stemming and decompounding techniques in German IR. They found that stemming and careful decompounding enhance the effectiveness of IR systems. Alfonseca et al. (Reference Alfonseca, Bilac and Pharies2008) presented a language-independent decompounding method in German. They observed that the decompounding method improves precision, recall, and accuracy in morphologically complex European languages. Laureys et al. (Reference Laureys, Vandeghinste and Duchateau2002) evaluated the impact of the decompounding module in large vocabulary continuous speech recognition (LVCSR) in Dutch. The decompound module combines a rule-based approach with statistical pruning. They noticed that the decompounding module decreased the 30% lexicon size and improved the 11% word error rate (WER) in the Dutch broadcast news recognition. Erbs et al. (Reference Erbs, Santos, Zesch and Gurevych2015) explored the effect of a decompounding model in keyphrase extraction. They found that the decompounding enhances the word frequency count and identifies more keyphrase candidates in German.

In the past decade, a few studies were also conducted on the corpus-based decompounding models in the Indian languages from NLP and IR perspectives. Deepa et al. (Reference Deepa, Bali, Ramakrishnan and Talukdar2004) used a trie-based dictionary for the splitting of a compound word presented in Hindi speech synthesis. They showed that the algorithm has a split rate of 92–96% of the input compound words, and the percentage of splitting accuracy is 83–87%. Devadath et al. (Reference Devadath, Kurisinkel, Sharma and Varma2014) proposed a hybrid sandhi splitting method in Malayalam. They applied statistical methods and predefined character-level sandhi rules. They observed that the hybrid sandhi splitting method outperformed the rule-based approach. Sahu and Mamgain (Reference Sahu and Mamgain2019) evaluated a corpus-based decompounding model in Sanskrit and improved the splitting accuracy by different ranking methods. They found that the decompounding model enhances precision, recall, and accuracy. Ganguly et al. (Reference Ganguly, Leveling and Jones2013) observed that relaxed and co-occurrence-based constituent selection techniques outperform the standard frequency-based decompounding approach in Bengali IR.

2.2 Machine learning-based decompounding methods

Pellegrini and Lamel (Reference Pellegrini and Lamel2009) investigated an unsupervised data-driven decompounding method in automatic speech recognition for Amharic text. They observed that the decompounding method reduces the out-of-vocabulary (OOV) words from 35% to 50% and slightly reduces the WER. Daiber et al. (Reference Daiber, Quiroz, Wechsler and Frank2015) proposed an analogy-based greedy unsupervised decompounding algorithm for German-to-English machine translation. They noticed that the decompounding algorithm is highly effective for ambiguous compound words. They found that the semantic analogy-based algorithm outperformed the frequency-based approach in German-to-English machine translation. Riedl and Biemann (Reference Riedl and Biemann2016) presented an unsupervised decompounding method in Dutch and German. They observed that the unsupervised method outperformed existing decompounding methods in German; however, the rule-based approach provided better effectiveness in Dutch.

Haruechaiyasak et al. (Reference Haruechaiyasak, Kongyoung and Dailey2008) evaluated the effect of word segmentation in Thai. They looked at two different word segmentation methods. One is a dictionary-based, and the other is a machine-learning approach. In the dictionary approach, they investigated longest matching and maximal matching techniques. In the machine learning approach, they investigated naive bayes, decision tree, support vector machine, and conditional random field techniques. They observed that the conditional random field (CRF) outperformed all other word segmentation methods. Ganguly et al. (Reference Ganguly, Bhat and Biswas2022) proposed a stochastic word transformation skip-gram algorithm, which improves the representation of the compositional compound words by leveraging information from the contexts of their constituents. They noticed that addressing the compounding effect as a part of the word embedding objective outperforms existing compounding-specific post-transformation-based approaches on word semantics and word polarity predictions. Minixhofer et al. (Reference Minixhofer, Pfeiffer and Vulić2023) built a dataset of 255k compound and non compound words across 56 diverse languages. They evaluated the effect of the decompounding method on the large language models (LLMs). They found that the LLMs perform poorly in different languages. Hence, they built self-supervised models that outperformed the unsupervised decompounding models and improved accuracy by 13.9%.

Ajees and Graham (Reference Ajees and Graham2018) presented a hybrid decompounding method in Malayalam. The decompounding method integrates a rule-based and machine-learning approach. The decompounding module can be used as a preprocessing step in machine translation, question-answering, and extractive summarisation applications. Das et al. (Reference Das, Radhika, Rajeev and Raj2012) evaluated the hybrid compound splitting method in the Malayalam text. They used a machine learning approach to classify the word as being split or not. Subsequently, a rule-based sandhi splitter is used to split the word. Shree et al. (Reference Shree, Lakshmi and Shambhavi2016) presented a CRF tool to split a compound word in the Kannada language. They evaluated the model’s effectiveness using five different combinations of training data. They achieved a compound splitting accuracy of 91.19%. Xue (Reference Xue2003) proposed a maximum entropy model for Chinese word segmentation. They found that the model can handle unknown words more efficiently than the maximum matching algorithm. They observed that the maximum entropy model provides precision and recall of 95.01% and 94.94%, respectively.

2.3 Deep learning-based decompounding methods

Chen et al. (Reference Chen, Qiu, Zhu, Liu and Huang2015) proposed an LSTM architecture for Chinese word segmentation. They experimented with different LSTM layers and dropout rates. They observed that the LSTM model provides a precision, recall, and f-score of 97.5%, 97.3%, and 97.4%, respectively. Kitagawa and Komachi (Reference Kitagawa and Komachi2018) also presented an LSTM architecture for Japanese word segmentation. They used a character-based n-gram embedding and a gold dictionary created from the test corpus. They noticed that the model achieves an F1-score of 98.67%. Premjith et al. (Reference Premjith, Soman and Kumar2018) investigated different deep learning-based decompounding methods in Malayalam. They found that the recurrent neural network (RNN), long short-term memory (LSTM), and gated recurrent units (GRU) provide an accuracy of 98.08%, 97.88%, and 98.16%, respectively. Hellwig (Reference Hellwig2015) presented a neural network-based decompounding model in Sanskrit. The neural network comprises an input layer and a hidden forward and backward layer. They used LSTM as a hidden layer to avoid the vanishing gradient problem. They observed that the decompounding model improves precision, recall, and accuracy. Reddy et al. (Reference Reddy, Krishna, Sharma, Gupta and Goyal2018b) introduced a deep sequence to sequence (seq2seq) model that takes the sandhied string as the input and predicts the unsandhied string as output. The model comprises multiple layers of LSTM cells with attention. They achieved an improvement of the F-score of 16.79%.

Hellwig and Nehrdich (Reference Hellwig and Nehrdich2018) developed a Sanskrit word segmentation model using character-level convolutional and RRNs. The model achieves a sentence splitting and string splitting accuracy of 85.2% and 96.7%, respectively. Aralikatte et al. (Reference Aralikatte, Gantayat, Panwar, Sankaran and Mani2018a) presented a double decoder (DD-RNN) with an attention model for the word decompounding in Sanskrit. They found that the model provides the splitting location and prediction accuracy of 95% and 79.5%, respectively, which outperforms the state of art by 20%. They also demonstrate the model’s generalisation capability by applying the word segmentation method in Chinese. Dave et al. (Reference Dave, Singh, D. P. and Lall2021) proposed a two-stage deep learning-based decompounding model in Sanskrit. In the first stage, they predict the sandhi window using the RNN model. In the second stage, the sandhi window is split into two words using the seq2seq model. They observed a location prediction and split prediction accuracy of 92.3% and 86.8%, respectively.

Few researchers developed software tools for decompounding methods in morphologically rich European and Asian languages. Biemann et al. (Reference Biemann, Quasthoff, Heyer and Holz2008) developed a modular ASV toolbox for scientific and educational purposes. The tool supports different sub-modules such as linguistic annotation, classification, clustering, language detection, POS–tagging, named entity recognition, German noun compound splitting, and base form reduction for German, English, and Norwegian. The tool split the compound word recursively using a trie-based data structure. In addition to the above, a prominent Sanskrit sandhi splitter is available on the web. Sachin (2007) proposed a JNU sandhi splitter.Footnote c The sandhi splitting tool is also called a Sanskrit sandhi recogniser and analyser. The tool provides an accuracy of 8.1% in Sanskrit split prediction. Similarly, UoH sandhi splitterFootnote d and INRIA sandhi splitterFootnote e provide a split prediction accuracy of 47.2% and 59.9%, respectively.

Based on the above findings, decompounding models play an important role in text analysis tasks. The effect of the decompounding method is thoroughly investigated in European and a few Asian languages. However, no previous studies exist on decompounding methods in Indian languages like Marathi, Hindi, and Sanskrit from an IR perspective. Hence, we evaluated different decompounding techniques in Indian languages in the IR domain. Our work is motivated by the earlier work of Koehn and Knight (Reference Koehn and Knight2003), Ganguly et al. (Reference Ganguly, Leveling and Jones2013), Ajees and Graham (Reference Ajees and Graham2018), Aralikatte et al. (Reference Aralikatte, Gantayat, Panwar, Sankaran and Mani2018a).

4.1 Corpus-based decompounding approaches

We evaluate here some of the existing decompounding techniques that have provenly yielded high effectiveness in different morphologically rich European and Asian languages Koehn and Knight (Reference Koehn and Knight2003), Deepa et al. (Reference Deepa, Bali, Ramakrishnan and Talukdar2004), Ganguly et al. (Reference Ganguly, Leveling and Jones2013). The impact of decompounding has been investigated from an NLP perspective. However, we study it from the IR viewpoint. In the corpus-based decompounding approach, the dictionary comprises the simple words of the document collection.

4.1.1 Frequency-based approach

In this approach, we split the compound words based on the frequency of probable constituents in the collection. The more frequently a probable constituent independently occurs in the collection, the more likely it is to be a part of a given compound word. This insight defines that a splitting point of a word depends upon the frequency of the constituent words in the collection. The splitting point of a compound $S$ is detected by the highest geometric mean obtained among considering word frequencies of all possible splits ( $e_i$ ) given by equation (1) ( $n$ being the number of splits, we consider here $n=2$ only). This approach is in line with the earlier work of the compound splitting method applied in the German-English machine translation Koehn and Knight (Reference Koehn and Knight2003).

(1) \begin{equation} \mbox{argmax}_{S}{\left(\prod _{e_i \in S} \mbox{frequency}(e_i)\right)}^{\frac{1}{n}} \end{equation}

4.1.4 Co-occurrence-based decompounding approach

In this approach, we first split the compound words using the frequency-based approach described above. We also apply relaxation and co-occurrence-based techniques to deal with the morphological features of Indian language compound words. The relaxed decompounding technique does not split the prefix-based compound words such as ‘upanagar’ into ‘upa’ and ‘nagar’ the co-occurrence-based decompounding technique tries to avoid some of the wrong splitting of the frequency-based approach. For example, in the frequency-based approach, the word ‘Loksabha’ is split into ‘lok’ and ‘sabha’ due to the high frequency of compound constituents in the collection. However, in the co-occurrence-based technique, the compound word ‘Loksabha’ is not split into ‘Lok’ and ‘sabha’ because the terms ‘Lok’ and ‘sabha’ will not frequently co-occur in the document collection. The co-occurrence-based decompounding technique splits the compound word into constituents only if the compound constituents co-occur with the compound word higher than a particular threshold. Here, we consider an optimal threshold value $\tau = 0.2$ for different Indian languages as we find $\tau = 0.2$ provides the best retrieval effectiveness in Bengali IR Ganguly et al. (Reference Ganguly, Leveling and Jones2013). The co-occurrence measure is the overlap coefficient between the set of documents $D(c)$ containing the constituent term $c$ and $D(w)$ containing the compound word. We measure the overlap coefficient by equation (2). This approach aligns with the earlier work of the compound splitting method evaluated in Bengali IR Ganguly et al. (Reference Ganguly, Leveling and Jones2013).

(2) \begin{align} Overlap (w,c)=\frac{\left | D(w) \bigcap D(c) \right |}{min{\left \{ \left | D(w) \right |,\left | D(c) \right | \right \}}} \\[-24pt] \nonumber \end{align}

4.2 Machine learning-based decompounding methods

Basic machine learning models such as support vector machine (SVM), decision trees, CRF, and random forests are widely used in different computational areas like text classification, part-of-speech (POS) tagging, and other NLP tasks. However, these models are yet to be explored in the word decompounding task for Indian languages. Here, we evaluate the impact of different hybrid (dictionary and machine-learning-based) decompounding models in different Indian languages. We used machine learning models such as decision tree, random forest, $k$ -nearest neighbours, SVM, and CRF. At first, we split the simple concatenation of compound words using a dictionary in each language. To split sandhi-ed compound words, we use different machine learning models. We train the machine learning models using a set of training examples. The training examples comprise three types of sandhi as described in Section 3. Post-training, the models are checked using compound words from the test dataset and evaluation is done by looking at the change in the retrieval effectiveness of different IR techniques. Here, we briefly describe different ML techniques used in this context, followed by the general steps adopted for these machine learning-based decompounding models.

Algorithm for machine learning-based decompounding approaches

The basic machine learning-based decompounding model is implemented in the following steps. We first split the simple concatenation of compound words followed by the sandhi-ed compound word.

1. (1) Read a set of input tokens from the user.

2. (2) Remove the stopwords from the input tokens.

3. (3) For each prefix in the prefix list (Table 12) ( $L_{pre}$ ), check whether the prefix is there in the given token or not. If yes, consider the entire input word as a compound word only.

4. (4) Check the length of the word in each token. If the token length is less than seven ( $7$ ), then EXIT.

5. (5) If the word length exceeds six ( $6$ ), traverse the dictionary and look for any compound constituent present; if yes, break that compound word into its constituent morphemes.

6. (6) If any of the above conditions are not satisfied, check for the sandhi-ed compound word splitting. Sandhied compound word splitting

7. (7) Assign a numeric number (Unicode) to each letter in the Devanagari script.

8. (8) Train different machine learning models by training datasets. The training dataset comprises the compound words and their constituent morphemes. Store the number corresponding to the first morpheme’s last and second morpheme’s first character in the training dataset.

9. (9) During training, the machine learning model learns from different sandhi examples.

10. (10) During testing, the machine learning model classifies the compound words into one of the trained sandhi examples and generates constituent morphemes.

11. (11) After decompounding, different retrieval models are applied to evaluate the change in retrieval effectiveness using the MAP scores.

The primary difficulty in developing a machine learning-based decompounding model is that it requires enormous training data. We also observe that the machine-learning-based model requires a complex feature representation. Hence, we also evaluate the deep learning-based decompounding models.

4.3 Deep learning-based decompounding approaches

In recent years, neural network models such as RNN, LSTM, and GRU have shown to provide good effectiveness in the text segmentation task of Chinese Chen et al. (Reference Chen, Qiu, Zhu, Liu and Huang2015), Japanese Kitagawa and Komachi (Reference Kitagawa and Komachi2018), and Sanskrit Hellwig (Reference Hellwig2015), Hellwig and Nehrdich (Reference Hellwig and Nehrdich2018), Reddy et al. (Reference Reddy, Krishna, Sharma, Gupta, M R and Goyal2018a). However, these neural network models have not yet been evaluated in the word decompounding task of Indian languages. This study explores different word decompounding models based on deep learning, such as RNN, LSTM, and GRU, its bidirectional versions, such as Bi-RNN, Bi-LSTM, Bi-GRU, and attention versions, such as Bi-RNN with attention (referred to as Bi-RNN-A), Bi-LSTM with attention (referred to as Bi-LSTM-A) and Bi-GRU with attention (referred to as Bi-GRU-A) model and evaluates their effectiveness in the Indian language IR. The decompounding task is seen here, in a sense, similar to the language translation problem – as the sequence of characters or compound words is taken as input and produces another sequence of characters or decompounded words as output. The sequence-to-sequence model Sutskever et al. (Reference Sutskever, Vinyals and Le2014) is widely used in the NLP domain. Here, we use the same model for word decompounding in Indian languages. The proposed model uses an RNN-based two-stage deep neural network for compound word splitting. The step for the decompounding algorithm is given below. The code is implemented in Python 3.5 with Keras API running on the Tensor Flow back-end. The code for different decompounding models is publicly available on GitHub.Footnote i

4.3.1 Encoder-decoder model

The proposed encoder–decoder model is inspired by the character level sequence to sequence model Sutskever et al. (Reference Sutskever, Vinyals and Le2014). The encoder comprises one recurrent layer, i.e. RNN, LSTM, or GRU, and converts the input character sequence into a one-hot encoding vector and processes it to the subsequent layer. Similarly, the decoder comprises one recurrent layer (RNN, LSTM, or GRU) followed by a dense soft-max layer. For training, we use the Marathi dataset developed by Singh et al. (Reference Singh, Bhingardive and Bhattacharyya2016) and SanskritFootnote j dataset developed at the University of Hyderabad. We train the encoder–decoder model by providing a compound word at a time and corresponding decompounded words concatenated with a ‘+’ in between. The input and output character sequences are replaced by their one-hot encoded vector sequences. Characters ‘&’ and ‘$’ are used as start and end markers in the output sequence. The output of the encoder–decoder model is the same one-hot encoded vector of decompounded words but with a left shift of characters by length one, i.e. offset by a one-time step in the future. We use a two-stage deep neural network for compound splitting. The reason behind using this two-stage deep neural network is that it detects the splitting point location efficiently Dave et al. (Reference Dave, Singh, D. P. and Lall2021), Aralikatte et al. (Reference Aralikatte, Gantayat, Panwar, Sankaran and Mani2018b). In the first stage, the model predicts a sandhi-window (a maximum of 5-letter window where at least two letters come from the first constituent word and at least another two letters from the second constituent word). In the second stage, the most probable splits that are likely to constitute the compound word are identified. Both the stages use the seq2seq model.

In the first stage, we provide a compound word as input to the sequential model that the model converts into a one-hot encoding vector and stores it in an array. All the array elements are assigned value zero except the sandhi window element, which is assigned value one, as shown in Figure 1 (the highlighted letters constitute sandhi-window). In the second stage, using the sandhi window, we split the compound word into two words. The input sequence is a sandhi window, and the output sequence is two words with a space delimiter between them. The model architecture of sandhi splitting is shown in Figure 2. The model is trained by the Adam optimiser Kingma and Ba (Reference Kingma and Ba2015) and the categorical cross-entropy loss function. The training vectors are divided into a batch size of 64. In the encoder–decoder model, we experiment with a different version of sequential models and parameters and the best MAP score achieved at a given parameter setting is shown in Table 1. At first, we trained the basic RNN model, but the model was not effective in different Indian languages. Hence, we experiment with other sequential models such as Bi-RNN, Bi-RNN-A, LSTM, Bi-LSTM, Bi-LSTM-A, GRU, Bi-GRU, and Bi-GRU-A. A detailed explanation of different sequential models is given below.

Figure 1. Sandhi-window (AnaNg) as the prediction target in a compound word.

Figure 2. Model architecture for Sandhi Split – Stage 1.

4.3.2 Recurrent neural network (RNN)

Recurrent neural network (RNN) Medsker and Jain (Reference Medsker and Jain2001) is a class of artificial neural networks which deals with the variable length sequential data $X$ = ( $x_1, x_2, \ldots, x_t, \ldots$ ). RNN uses memory at each internal node to process variable-length sequential data. In an RNN, the output state $h(t)$ depends on the previous state information $h(t-1)$ and present input $x_t$ . At any time-step $t$ , the value of the current state is updated by the equation (3). A basic RNN architecture is shown in Figure 3.

(3) \begin{equation} h(t)=f(h(t-1), x_t) \end{equation}

Figure 3. A basic RNN architecture.

where $f$ represents the nonlinear activation function. In a simple RNN, we use the soft-max activation function. We train the RNN by gradient-descent methods and back-propagation. In each training iteration, the gradient is evaluated and updated with the current weight. During the training of sequential data, the gradient will not be updated, which will prevent changes in the RNN model’s weight. Hence, the model does not learn from the previous inputs, which causes the vanishing gradient problem. We use other sequential models like LSTM and GRU to overcome the vanishing gradient problem.

4.3.3 Long short-term memory network (LSTM)

The traditional RNN suffers from the problem of vanishing gradient. Hence, the RNN could not handle the long-term sequential data. To overcome the vanishing gradient problem, Hochreiter and Schmidhuber (Reference Hochreiter and Schmidhuber1997) proposed an LSTM network. The LSTM network can resolve the issue by maintaining different memory blocks and gates. The memory block comprises memory cells with self-connections, and different gates are used to regulate the flow of information. Primarily, the LSTM network comprises four types of gates, i.e. input gate, output gate, forget gate and memory-cell activation gate, as described below.

Table 1. Show the Parameter of different Encoder–Decoder models

1. (1) Input gate: The input gate ( $i_t$ ) determines which input should be kept at the cell state and which input is transmitted to the next long-term state, i.e. $c_t$ . The gate determines that some part of the information is transmitted and reflected in long-term memory.

2. (2) Forget gate: The forget gate ( $f_t$ ) determines which part of the long-term data should be kept and passed to the next state ( $c_t$ ) and which part of the long-term data is to be thrown away from the cell state. The decision is made by the sigmoid layer called the ‘forget gate layer’.

3. (3) Memory-cell activation gate: The memory-cell activation gate ( $c_t$ ) updates the information in the memory state cell by taking the information from the input gate and forget gate.

4. (4) Output gate: The output gate ( $o_t$ ) determines which part of the long-term state should appear in the output network.

5. (5) Mathematical expressions of different gates are as follows.

   (4) \begin{equation} i_t= \sigma (W_{xi}\cdot x_t + W_{hi}\cdot h_{t-1}+W_{ci}\cdot c_{t-1}+b_i) \end{equation}
   (5) \begin{equation} f_t= \sigma (W_{xf}\cdot x_t + W_{hf}\cdot h_{t-1}+W_{cf}\cdot c_{t-1}+b_f) \end{equation}
   (6) \begin{equation} o_t= \sigma (W_{xo}\cdot x_t+W_{ho}\cdot h_{t-1}+W_{co}\cdot c_{t-1}+b_o) \end{equation}
   (7) \begin{equation} c_t = f_t \cdot c_{t-1} + i_t \cdot \tanh (W_{xc}\cdot x_t + W_{hc}\cdot h_{t-1}+b_c) \end{equation}

A hidden state is represented as follows:

(8) \begin{equation} h_t= o_t \cdot \tanh (c_t) \end{equation}

$W_{xi}, W_{xf}, W_{xo}, W_{xc}$ are the weight matrices of different layers connected to the input vector $X_t. W_{hi}, W_{hf}, W_{ho}, W_{hc}$ are the weight matrices connected to the state $h_{t-1}$ and $ b_i, b_f, b_o, b_c$ are the biases at different layers. A single-cell LSTM architecture is shown in Figure 4.

Figure 4. A single cell LSTM architecture.

4.3.4 Gated recurrent unit (GRU)

A gated recurrent unit is a variant of RNNs like RNN and LSTM introduced by Cho et al. (Reference Cho, van Merriënboer, Gulcehre, Bahdanau, Bougares, Schwenk and Bengio2014). GRU also solves the problem of vanishing gradient. GRU is like an LSTM network but is comprised of fewer parameters and gates.

1. (1) Update gate: The update gate ( $z_t$ ) determines the amount of information transferred from one hidden state to the next hidden state. If the output gate is close to 1, then all current hidden state information is transferred to the next hidden state.

2. (2) Reset gate: The reset gate ( $r_t$ ) determines the significance of previously hidden state information in updating current hidden state information. If the output of the reset gate is close to 0, the hidden state is forced to ignore the previously hidden state information and reset with the current input only. It signifies that the hidden state can drop any information that is found to be irrelevant, allowing a more compact representation.

3. (3) Hidden state: The hidden state ( $h_t$ ) is updated using the current input and the information obtained from the previous hidden state ( $h_{t-1}$ ). Mathematical expressions of different gates are as follows.

   (9) \begin{equation} z_t = \sigma (W_{z}\cdot x_t + U_{z}\cdot h_{t-1}+b_z) \end{equation}
   (10) \begin{equation} r_t = \sigma (W_{r}\cdot x_t + U_{r} \cdot h_{t-1}+b_r) \end{equation}
   (11) \begin{equation} h_t = (1-z_t) \cdot \tanh (r_t \cdot U \cdot h_{t-1} + W \cdot x_t) + z_t \cdot h_{{t-1}} \end{equation}

$W_{z}, W_{r}$ , and $W$ are the weight matrices of different layers connected to the input vector $X_t$ , while $U_r$ , $U_z$ , and $U$ for the $U$ for $h_{t-1}$ . $b_z$ , and $b_r$ are the biases at different layers. A single-cell GRU architecture is shown in Figure 5.

Figure 5. A single cell GRU architecture.

6.1 BM25 model

Okapi BM25 Robertson et al. (Reference Robertson, Walker and Beaulieu2000) is a popular model where BM stands for ‘best matching’. The term weight of a query term in a document is given by Equation 12.

(12) \begin{equation} w(t,d)= tf_{\textrm{d}} \cdot (\frac{\log (\frac{N-n+0.5}{n+0.5})}{k_{\textrm{1}} \cdot ((1-b)+b \cdot \frac{dl}{avdl}+tf_{\textrm{d}}) }) \end{equation}

6.2 TF-IDF model

In this model, the importance of a query term in a document is determined by the combination of term frequency (tf) and inverse document frequency (idf). Here, the tf represents the frequency of a term present in a given document, and idf is the inverse of document frequency (df) where df represents the number of documents that contain the given term. We call this quantity as term-weight ( $w(t,d)$ ) of term $t$ in document $d$ . The scalar combination of all such $w(t,d)$ s in the query terms is used in cosine-similarity. The TF-IDF model within the Terrier retrieval system uses Robertson’s tf and Sparck Jones idf Sparck Jones (Reference Sparck Jones1972).

(13) \begin{equation} w(t,d)= Robertson\_tf \cdot idf \end{equation}

where,

$$Robertson\_tf = \frac{{t{f_{\text{d}}}}}{{{k_{\text{1}}}\cdot((1 - b) + b\cdot\frac{{dl}}{{avdl}}) + t{f_{\text{d}}}}}idf = \log (N/{d_{\text{f}}} + 1)tfn = tf\cdot{\log _2}(1 + c\cdot\frac{{avdl}}{l})$$

\begin{eqnarray*} dl &:& \text{document length in number of terms}\\[5pt] avdl &:& \text{average document length}\\[5pt] d_{\textrm{f}} &:& {\text{document frequency of term} {\;} \{t\}}\\[5pt] N &:& \text{total number of documents in the collection}\\[5pt] n &:& \text{number of documents containing at least one term}{\;\{t\}}\\[5pt] k_{\textrm{1}} &:& \text{term-frequency parameter, a constant}\\[5pt] b &:& \text{document length normalization parameter} \\[5pt] n_{\text{t}} &:& \text{document frequency of term}{\;\{t\}} \\[5pt] tfn &:& \text{normalised term frequency} \\[5pt] c &:& \text{free parameter} \\[5pt] F &:& {\text{frequency of term}{\;}\{t\} \text { in the collection}} \end{eqnarray*}

We also consider other probabilistic models such as In_expC2, BB2, and InL2. These models come from the divergence from randomness (DFR) family Amati and Van Rijsbergen (Reference Amati and Van Rijsbergen2002). The DFR models are based on the idea that the more the divergence of the within-document term frequency from its frequency within the collection, the more information is carried by the term ( $t$ ) in the document ( $d$ ) Robertson et al. (Reference Robertson, van Rijsbergen and Porter1980).

6.3 In_expC2 model

In this model, the relevance score of a document is calculated by Equation (14).

(14) \begin{equation} w(t,d)= \frac{F+1}{n_t(tfn_e+1)}\big (tfn_e\cdot \log _2\frac{N+1}{n_e+0.5}\big ) \end{equation}

$tfn_e$ denotes the normalised term frequency. A modified version of normalisation two is given by

(15) \begin{equation} tfn_e=tf\cdot \log _e(1+c\frac{avdl}{l}) \end{equation}

6.4 BB2 model

In the Bose–Einstein model for randomness, the weight of a query term in a given document is given by the ratio of two Bernoulli’s processes (first normalisation and second normalisation) for term frequency normalisation as shown in Equation (16).

(16) \begin{equation} \begin{split} w(t,d)= \frac{F+1}{n_t\cdot (tfn+1)} & \big [(-\log _2(N-1)-\log _2(e) \\ & \quad +f(N+F-1,N+F-tfn-2) -f(F,F-tfn)\big )] \end{split} \end{equation}

(17) \begin{equation} tfn=tf\cdot \log _2(1+c\cdot \frac{avdl}{l}) \end{equation}

where,

\begin{eqnarray*} F &:& \text{frequency of term}\; \textit{t}\; \text{ in the collection}\\ n_t &:& \text{document frequency of term}\; \textit{t}\\ tfn &:& \text{normalised term frequency} \\ c &:& \text{ free parameter} \end{eqnarray*}

6.5 InL2 model

In this model, the weight of a query term in a document is given by the ratio of two Laplace processes (first normalisation and normalisation 2) for term frequency normalisation as shown in equation (18).

(18) \begin{equation} w(t,d)= \frac{1}{tfn+1}\big (tfn\cdot \log _2\frac{N+1}{n_t+0.5}\big ) \end{equation}

6.6 Hiemstra language model

We also evaluate using a non-parametric probabilistic or language model proposed by Hiemstra (Reference Hiemstra2001). The probability of a query term $t$ in a document $d$ depends upon the term frequency of $t$ in both $d$ and the entire corpus. This model uses a smoothing parameter $\lambda$ to tune their contribution (a popular default value of $\lambda$ is $0.15$ ). The similarity between a query term $t$ and document $d$ is given by Equation 19.

(19) \begin{equation} P(D, T_{1}, T_{2}, T_{3}, \cdot \cdot \cdot \cdot T_{n}) = P(D)\prod _{i=1}^{n}((1-\lambda _{i}) P(T_{i})+ \lambda _{i} P(T_i |D)) \end{equation}

where D is a document and $T_{i}$ are query terms.

7.1 Precision

It is the fraction of relevant documents retrieved among the retrieved documents.

(20) \begin{equation} Precision = \frac{\left |\text{Relevant documents} \bigcap \text{Retrieved documents}\right |}{\text{Total number of retrieved documents}} \end{equation}

7.2 Recall

It is the fraction of relevant documents retrieved from a set of relevant documents.

(21) \begin{align} Recall = \frac{\left |\text{Relevant documents} \bigcap \text{Retrieved documents} \right |}{\text{Number of relevant documents in the collection}} \end{align}

In both, the numerator is the number of relevant and retrieved documents. To improve the performance of any system, we improve precision and recall everywhere – which essentially requires an increase in the number of relevant retrieves (Rel.Ret.).

7.3 AP

It is the average of the precision values whenever a relevant document is retrieved. R $_{\textrm{d}}$ stands for the d-th relevant document among a ranked list of retrieved documents. The total number of relevant documents for a given query is ‘R’.

(22) \begin{equation}{average precision} (AP) =\frac{1}{R}\sum _{d=1}^{R}{precision(R _{\textrm{d}})} \end{equation}

7.4 MAP

AP corresponds to a single query. However, the comparison between two IR models should not be based on a single query but on a set of queries so that variation in retrieval effectiveness over individual queries is averaged out. Hence, MAP is defined for a set of queries and is computed as the simple arithmetic mean of AP scores of all such queries.

(23) \begin{equation}{mean average precision} (MAP)= \frac{1}{|Q|}\sum _{t=1}^{|Q|}{AP(t)} \end{equation}

$|Q|$ : Number of queries

10.1 Effect of corpus-based decompounding on retrieval

In the first set of experiments, we evaluate the impact of different corpus-based decompounding models in Indian language IR. The effect of different corpus-based decompounding models for Marathi, Hindi, and Sanskrit languages is shown in Tables 3, 4, and 5, respectively. In these tables, the best MAP scores among the retrieval models are shown in boldface. It is observed that the different corpus-based decompounding models improve retrieval effectiveness in different Indian languages IR. However, the effect of decompounding models varies from one language to another. The frequency-based decompounding model improves an MAP score of 12.94% in Marathi, 4.56% in Hindi, and 0.37% in Sanskrit (In_expC2 model). Similarly, the maximum branching factor, tri-based, and co-occurrence-based decompounding models improve the MAP scores by 13.5%, 13.87%, and 14.98% in Marathi, 4.34%, 4.22%, and 4.86% in Hindi, and 2.8%, 4.75%, and 4.58% in Sanskrit languages, respectively (In_expC2 model). Other models also show improvements in general, although at different levels, including some drops. These observations are in line with earlier findings as different corpus-based decompounding models provide comparable effectiveness in the European languages IR Monz and Rijke (Reference Monz and Rijke2001).

Table 3. MAP scores of different corpus-based decompounding evaluation in Marathi (39 T queries)

Table 4. MAP scores of different corpus-based decompounding evaluation in Hindi (50 T queries)

Table 5. MAP scores of different corpus-based decompounding evaluation in Sanskrit (50 T queries)

10.2 Effect of machine learning-based decompounding on retrieval

In the second set of experiments, we propose hybrid machine learning-based decompounding models followed by their evaluation. The decompounding models comprise the dictionary and machine learning-based approach to split the compound words. Impact of different hybrid machine learning-based decompounding models on Marathi, Hindi, and Sanskrit languages is shown in Tables 6, 7 and 8, respectively. It is observed that the different decompounding models enhance the effectiveness of document retrieval. On closer observation, we find that the effect of decompounding models varies from one language to another. The SVM model improves MAP score by 22.08% in Marathi, 10.59% in Hindi and 2.95% in Sanskrit languages. Similarly, the CRF, KNN, Decision Tree, and Random Forest-based models improve MAP scores by 22.27%, 15.73%, 16.84%, and 16.7% in Marathi; 10.83%, 16.41%, 16.65% and 17.14% in Hindi; and 2.82%, 4.75%, 4.58% and 4.43% in Sanskrit languages, respectively (all in In_expC2 model). In other models, the scores also mostly improve with a few exceptions. Different hybrid machine learning-based models outperform the pure corpus-based models in Indian and European languages IR Monz and Rijke (Reference Monz and Rijke2001), Ganguly et al. (Reference Ganguly, Leveling and Jones2013). The fundamental reason behind this is that the machine learning-based decompounding models can efficiently split the simple concatenation and sandhi-ied compound words.

Table 6. MAP scores of different hybrid machine learning-based decompounding evaluation in the Marathi (39 T queries)

Table 7. MAP scores of different hybrid machine learning-based decompounding evaluation in the Hindi (50 T queries)

Table 8. MAP scores of different hybrid machine learning-based decompounding evaluation in the Sanskrit (50 T queries)

10.3 Effect of deep learning-based decompounding on retrieval

In the third set of experiments, we propose different deep learning-based decompounding models and evaluate their effectiveness from an IR perspective. The impact of different deep learning-based decompounding models for Marathi, Hindi, and Sanskrit languages is shown in Tables 9, 10, and 11. Deep learning-based decompounding models are seen to considerably improve the effectiveness of an IR system. However, the impact of deep learning-based decompounding models varies from one language to another. The RNN-based decompounding model improves MAP score of 23.66% in Marathi, 16.28% in Hindi,and 2.38% in Sanskrit compared to the baseline approach (In_expC2 model). Similarly, the LSTM and GRU models improve MAP score by 23.94% and 23.66% in Marathi, 17.02% and 16.84% in Hindi, 4.94% and 1.15% in Sanskrit. We observe no significant difference in MAP score between the vanilla RNN, LSTM, or GRU and their bidirectional counterparts. However, the Bi-RNN-A-based decompounding model improves MAP score by 27.61% in Marathi, 18.18% in Hindi, and 6.1% in Sanskrit compared to the baseline approach (In_expC2 model). Similarly, the Bi-LSTM-A and Bi-GRU-A decompounding models improve MAP score by 28.02% and 27.98% in Marathi, 17.11% and 17.45% in Hindi, 5.24% and 2.83% in Sanskrit. Among the different deep learning-based decompounding models, the attention-based models outperform the other models in the Indian language IR. We also observe that the deep learning-based models for word-decompounding outperform the hybrid machine learning-based and corpus-based models in Indian language IR.

Table 9. MAP scores of different deep learning-based decompounding evaluation in the Marathi (39 T queries)

Table 10. MAP scores of different deep learning-based decompounding evaluation in the Hindi (50 T queries)

Table 11. MAP scores of different deep learning-based decompounding evaluation in the Sanskrit (50 T queries)

10.4 Retrieval effectiveness analysis: some insights

In this work, the statistically significant differences are detected by a two-sided t-test (significance level $\alpha$ = 5%) with Bonferroni correction Simes (Reference Simes1986). We use without decompounding as a baseline (Tables 9, 10 and 11). We also observe that the decompounding methods typically improve MAP scores in different retrieval models across the Indian languages, but the changes in effectiveness are not statistically significant. To get more insights into the effect of decompounding, we perform a query-by-query analysis. Here, we consider the In_expC2 model for Marathi and Hindi languages and the BB2 model for Sanskrit as the best-performing models for their respective languages. On closer observation, we find that the Bi-LSTM-A model improves the average precision scores in 28 topics and reduces in 5 topics in Marathi. The percentage change in the average precision score for each query is shown in Figure 6. Similarly, the Bi-RNN-A model improves the average precision scores in 42 and 29 topics in Hindi and Sanskrit languages and reduces in 8 and 15 topics, respectively. The percentage change in the average precision score for each query is shown in Figures 7 and 8. In Sanskrit, some queries provide phenomenal retrieval effectiveness. For example, in Topic 47, BJP party’s national executive meeting held in New Delhi, the decompounding model improves the average precision score by 102.93%. A similar observation is found in Topic 34, corruption in P.N.B. bank (improvement of 86.59%).

Figure 6. A query by query evaluation in the Marathi by In_expC2 model.

Figure 7. A query by query evaluation in the Hindi by In_expC2 model.

Figure 8. A query by query evaluation in the Sanskrit by BB2 model.