2.1 Search queries, sessions, and missions

The seminal high-level categorization of search behavior by Broder (Reference Broder2002), classifying the queries in navigational, informational, or transactional, has been subsequently extended as more online search activities are identified (Rose and Levinson, Reference Rose and Levinson2004; Marchionini, Reference Marchionini2006; Jansen, Booth, and Spink, Reference Jansen, Booth and Spink2008). Of particular interest for our work is the multi-step search task (Hassan Awadallah et al., Reference Hassan Awadallah, White, Dumais and Wang2014a), which encompasses the behavior where a complex task, involving multiple particularities, aims to be fulfilled. A large body of work targets the detection of search sessions in query logs, with the ultimate aim of identifying search patterns in user behavior and of providing additional support on them. In the field of query log analysis, a search session is usually defined as a sequence of queries where the elapsed time between two consecutive queries is no larger than a delta value, which ranges from 30 min (He, Göker, and Harper, Reference He, Göker and Harper2002; Radlinski and Joachims, Reference Radlinski and Joachims2005) to 90 min (Hagen et al., Reference Hagen, Gomoll, Beyer and Stein2013). Beyond session detection, focus was also placed on the challenge of understanding the task-based user interaction with search engines. Here, “task” refers to the overall goal (“work task”) of the user that motivated her to search in the first place. A notion of search mission is established by Jones and Klinkner (Reference Jones and Klinkner2008). They introduce the concept of hierarchical search, instituting the idea of a search mission as a set of topically related queries, not necessarily consecutive, composed by simpler search goals. Their work also discusses a method for detecting whether a given query pair belongs to the same goal. We adhere to their concept of a search mission, to emphasize the coherent nature of search queries motivated by a single underlying goal, as opposed to other lines of work that focus on multitasking behavior across sessions. Moreover, and for the sake of simplicity, we relax the definition of mission by discarding the notion of sequence and only considering a search mission as an unordered set of queries all with the same underlying task.

Several works have extended the problem of detecting search missions, and proposed approaches accordingly. In the work by Mei et al. (Reference Mei, Klinkner, Kumar and Tomkins2009), a method is proposed for finding user search sequences at multiple levels of granularity within the hierarchical search approach. Furthermore, Donato et al. (Reference Donato, Bonchi, Chi and Maarek2010) address on-the-fly identification of search tasks, that is as the user interacts with an engine. Intent-aware query similarity is introduced by Guo et al. (Reference Guo, Cheng, Xu and Zhu2011), aiming to capture the search intent behind a query and using these underlying tasks for recommending related queries. Kelly et al. (Reference Kelly, Arguello and Capra2013) propose a supervised model to analyze user search behavior spanning multiple sessions. They address two problems: identifying related queries for a query in previous sessions by that user, and predicting whether the user will return to a given multiquery task. The problem of mission detection framed as search task discovery within a user session is introduced in the work by Lucchese et al. (Reference Lucchese, Orlando, Perego, Silvestri and Tolomei2011), where the authors also propose a method based on graph query representation. Hagen et al. (Reference Hagen, Gomoll, Beyer and Stein2013) extend task detection to the cross-session modality, using a cascading approach, to account for the user behavior of carrying on a search task across multiple sessions. Further approaches in cross-session search task detection include modeling intrinsic and extrinsic diversity (Raman, Bennett, and Collins-Thompson, Reference Raman, Bennett and Collins-Thompson2013), temporal closeness (Li et al., Reference Li, Deng, Dong, Chang and Zha2014), topic membership (Li et al., Reference Li, Deng, He, Dong, Chang and Zha2016), and embedded trees in query similarity graph (Wang et al., Reference Wang, Song, Chang, He, White and Chu2013). Lucchese et al. (Reference Lucchese, Nardini, Orlando and Tolomei2016) propose a framework for finding search tasks combining a Learning-to-Rank-based query similarity and graph-based query clustering. Lugo et al. (Reference Lugo, Moreno and Hubert2020b) address task-based query log segmentation with a method that does not depend on information such as clicked URLs. The authors also approach search task detection in a user-agnostic, unsupervised fashion (Lugo, Moreno, and Hubert, Reference Lugo, Moreno and Hubert2020a). We assume that we have access to a mission detection oracle that correctly identifies a set of queries all belonging to the same (unknown) task. In practical applications, this assumption would correspond to having a component in place that detects search missions, on top of which our task recommendation approach can be performed. The assumption of such a mission oracle in place avoids any potential issues with the errors of the mission detection method when observing task recommendation performances.

Recommendations based on search queries have been studied for specific domains, such as movies or celebrities (Yu et al., Reference Yu, Ma, Hsu and Han2014; Bi et al., Reference Bi, Ma, Hsu, Chu, Wang and Cho2015). Such model-based methods typically rely on manually designed domain-dependent features, related to specific properties of entities, and usually deal with queries that consist of names of entities. Instead, we address search queries that express the need for solving an underlying task. We recommend elements from a repository of task descriptions using different attributes in the document representation of a task, such as title or explanation. We argue that our problem of task recommendation based on search queries is a significant conceptual transition from an elementary approach of ad hoc document retrieval. An analogous and very successful transition can be seen in entity retrieval, this is, the problem of obtaining a ranked list of entities relevant to a search query. The INEX 2007–2009 Entity Retrieval track (de Vries et al., Reference de Vries, Vercoustre, Thom, Craswell and Lalmas2008; Demartini et al., Reference Demartini, de Vries, Iofciu and Zhu2009, Reference Demartini, Iofciu and de Vries2010) was the first benchmark campaign on studying entity retrieval, using Wikipedia as an entity repository of reference. In other words, the operational representation of entities was based on their corresponding repository of Wikipedia articles. Hence, our operationalization of the abstract notion of a task via a repository of task descriptions to be recommended for search queries resembles the one utilized in the conceptualization of entity retrieval.

2.2 Procedural knowledge

Task recommendations do not necessarily benefit from descriptive or declarative knowledge bases (KBs). Indeed, such KBs do not contain sufficient procedural knowledge. Procedural knowledge (Georgeff and Lansky, Reference Georgeff and Lansky1986) is the knowledge involved in accomplishing a task. A few procedural KBs are specialized in semi-structured knowledge (Pareti, Klein, and Barker, Reference Pareti, Klein and Barker2014; Chu, Tandon, and Weikum, Reference Chu, Tandon and Weikum2017), that has been extracted from corpora of how-to guides collaboratively created in projects like WikiHow or eHow.Footnote ^b Jung et al. (Reference Jung, Ryu, Kim and Myaeng2010) automatically create a situation ontology from how-to instructions through a model of daily situational knowledge with goals, action sequences, and contextual items. They also envisage a schema mapping actions to actual service providers. As mentioned above, for pragmatic reasons, we choose to work with a collection of procedural articles from WikiHow as our repository to recommend tasks from. Pareti et al. (Reference Pareti, Klein and Barker2014) identify properties that define community-centric tasks. They then design a framework to represent the semantic structure of procedural knowledge in web communities. The obtained representation operates distributedly across different KBs and comprises a process ontology to express tasks, subtasks, requirements, and execution status.

Chu et al. (Reference Chu, Tandon and Weikum2017) propose to build a procedural KB by normalizing the how-to tasks from WikiHow into a hierarchical taxonomy that also represents the temporal order of subtasks, as well as the attributes of involved items. Their method relies on hierarchical agglomerative clustering of task frames obtained by open-domain semantic role labeling. They show the usefulness of this KB in expanding individual, task-oriented queries for retrieving relevant YouTube videos. Although we address a different problem, namely explicit task recommendation rather than leveraging tasks for recommending other kind of items, we use their dataset of WikiHow articles as our task repository.

Other more recent works approach the procedural nature of WikiHow in a data-driven fashion using representation learning. Problems addressed in these works include the correspondence between a subtask and its main task, approached as a single-class prediction (Zhang, Lyu, and Callison-Burch, Reference Zhang, Lyu and Callison-Burch2020a) or as a binary classification (Zhou, Shah, and Schockaert, Reference Zhou, Shah and Schockaert2019), and the reasoning about the possible temporal ordering between two subtasks (Zhou et al., Reference Zhou, Shah and Schockaert2019; Zhang, Lyu, and Callison-Burch, Reference Zhang, Lyu and Callison-Burch2020b). Zhang et al. (Reference Zhang, Lyu and Callison-Burch2020b) have studied the ability of WikiHow to be a resource for knowledge transfer to other tasks in natural language understanding. To the best of our knowledge, we are the first in using WikiHow as the main resource of task descriptions and building a corresponding dataset suited for the novel problem of task recommendation based on search queries and missions, that we also introduce in this work.

3.1 Query-based task recommendation

The first problem is concerned with the recommendation of tasks in response to a web search query. A key step in addressing this problem has to do with the operationalization of the abstract notion of a task. We need to establish the universe of possible tasks as well as a computational representation of tasks. For that, we introduce the concept of a task repository.

A task description can be, for example, a how-to article illustrating how to change a tire (see Figure 2). It typically contains parts such as a title and a brief explanation of the task, then enumerates the steps needed to accomplish the task. We say that the task description is semi-structured to emphasize that some of these document elements may be missing. Each element may be viewed as a task attribute, similar to attributes of entities in a KB. Throughout the article, when we say task, we simply refer to this structured representation of a task description.

Figure 2.

WikiHow article corresponding to the task “How to change a tire.” The task has a title and a brief explanation. Each step has a main act corresponding to a subtask (here, the first sentence of each step), and is possibly complemented with a detailed act.

Our envisioned system would need to handle any input query and accordingly address task recommendation in the case where the query indeed corresponds to an instance of task-based search. In this work, we assume that there is an intent classifier component in place that determines whether the query has indeed a task intent. When the type of the query has been determined as having a task-based intent, a subsequent component performs the ranking of relevant tasks for the query. The problems that we address in this work correspond to this particular component, while assuming that the other parts of the system are in place. These relevant task descriptions are then expected to be recommended to the user accordingly.

Now, we can formally define the problem of query-based task recommendation.

An illustration of this problem is given by Figure 1, where an example query “birthday party” leads to the recommendation of five tasks.

3.2 Mission-based task recommendation

We now define the mission-based task recommendation problem. It generalizes the query-based task recommendation to recommend tasks for a set of queries, all of which are motivated by the same (unknown) underlying task. We refer to such query sets as search missions. Specifically, a search mission, or simply mission, is a set of queries $m=\{q_1,\dots,q_n\}$ , possibly spanning multiple search sessions, that all share the same underlying task.

We do not address the problem of search mission detection, and instead assume that there is a process in place that identifies and groups queries that share the same underlying task (Jones and Klinkner, Reference Jones and Klinkner2008; Hagen et al., Reference Hagen, Gomoll, Beyer and Stein2013; Lucchese et al., Reference Lucchese, Orlando, Perego, Silvestri and Tolomei2013).

Similar to the previous example, given a single query “cake recipe,” a highly relevant task to be recommended is $t_1 =$ “Make a cake.” Now, assuming that the search mission also contains a second query “cake decorating ideas,” the task $t_2 =$ “Make a birthday cake” would also be relevant. Further, if the mission had a third query “birthday invitation cards,” then the recommended tasks could also include $t_3 =$ “Organize a birthday party.”

4.1 Query-based task recommendation

Our approach for query-based task recommendation is based on document-ranking techniques. By considering individual task attributes as documents, we can apply traditional term frequency-based techniques to rank tasks. Further, we propose to take semantic matching signals into account as well, to extend the capabilities of these already strong traditional techniques.

4.1.1 Term-based matching

We consider every task $t \in \mathcal{T}$ as a document and build an inverted index for each task attribute $t_a$ (Title, Explanation, MainAct, and DetailedAct) for efficient computation. We then score each task attribute $t_a$ against the input query $q$ using the well-established BM25 retrieval method (Robertson and Walker, Reference Robertson and Walker1994; Robertson et al., Reference Robertson, Walker, Beaulieu, Gatford and Payne1996; Robertson and Zaragoza, Reference Robertson and Zaragoza2009), obtaining $\mathrm{score}_{\text{BM25}}(t_a, q)$ :

(1) \begin{equation} \mathrm{score}_{\text{BM25}}(t_a, q) = \sum _{w \in q}{\frac{f_{w, t_a} \cdot (1 + k_1)}{f_{w, t_a} + k_1 \left(1 - b + b{\frac{|t_a|}{\text{avgl}}}\right)}} idf_w, \end{equation}

where $f_{w, t_a}$ is the number of occurrences of word $w$ in the (document corresponding to) task attribute $t_a$ , $idf_w$ is the inverse document frequency for $w$ , $\text{avgl}$ is the average document length, $k_1$ is the calibrating term frequency scaling, and $b$ is the document length normalization factor.

4.1.2 Semantic matching

Consider the query “how to loan shark” (taken from our test corpus, cf. Section 5.1). When matching potential task descriptions, a traditional retrieval method based on term frequencies would fail to capture that the term “shark” does not refer to the fish but to the financial practice of predatory lending with extremely high-interest rates. Another example is the query “how do i put photos in my ipod.” A standard document-based scoring method would not be able to capture that “pictures” is a synonym of “photos” in the task title “How to add pictures to my ipod.” This kind of vocabulary mismatch is the motivation behind our choice for additionally leveraging semantic representations of terms provided by continuous word embeddings.

We use the cosine similarity between a query $q$ and a given task attribute $t_a$ as the semantic score for $q$ and $t_a$ :

(2) \begin{equation} \mathrm{score}_{\text{sem}}(t_a, q) = \text{cos}(\boldsymbol{t}_a, \boldsymbol{q}). \end{equation}

The query and task attribute are each represented by a vector, $\boldsymbol{q}$ and $\boldsymbol{t}_a$ , respectively, which are taken to be the centroids of the word vectors corresponding to the individual terms making up $q$ and $t_a$ . The set of words may be further restricted to a word function, that is filtered according to the part-of-speech (POS) tag of a term. Formally

(3) \begin{equation} \boldsymbol{q} = \frac{1}{|\{w \in q \wedge F(w)\}|} \sum _{w \in q \wedge F(w)} \boldsymbol{w}, \end{equation}

where a term vector $\boldsymbol{w}$ is given by some word embedding, and a query term $w$ is only considered if it satisfies a given word function $F$ . The task attribute vector $\boldsymbol{t}_a$ is calculated analogously. As discussed next in Section 4.1.3, regarding word functions, we propose to use combinations of POS tags that correspond to retaining only content words (nouns, verbs, adjectives). This selection by POS is shown to be beneficial as the best-performing feature subsets are based on these selected combinations (cf. Section 6.2).

We propose to combine these semantic matching signals together with the term-based matching score in a learning-to-recommend (LTR) approach. LTR has been widely used for various recommendation problems in the past (Shi et al., Reference Shi, Karatzoglou, Baltrunas, Larson, Hanjalic and Oliver2012a, Reference Shi, Karatzoglou, Baltrunas, Larson, Oliver and Hanjalic2012b, Reference Shi, Karatzoglou, Baltrunas, Larson and Hanjalic2013a, Reference Shi, Karatzoglou, Baltrunas, Larson and Hanjalic2013b; Rafailidis and Crestani, Reference Rafailidis and Crestani2017). The novelty of our work lies in developing features that are appropriate for the task at hand. Specifically, we instantiate the semantic matching score in Equation (2) for each task attribute using multiple pre-trained word embeddings and combinations of word functions.

Table 1.

Features used for learning to recommend tasks

4.2 Mission-based task recommendation

We turn now to the problem of mission-based task recommendation. Given a search mission $m = \{q_1, \ldots, q_n\}$ , let $R_i$ denote the ranked list of recommended tasks for $q_i$ , obtained by a query-based task recommender method (cf. Section 4.1). Further, we let $T$ be the set of all tasks in all rankings, $T = \bigcup _{i = 1}^n \{t \,:\, t \in R_i\}$ .

We introduce a general approach for mission-aware task recommendation that aggregates the individual query-based rankings for each query $q_i \in m$ into a mission-level task ranking. Specifically, let $s$ be a scoring function, such that $s(q_i, t)$ is the score corresponding to task $t$ in the ranking $R_i$ . Then, for each task $t \in T$ , the mission-based recommendation score of $t$ is obtained by aggregating the query-based recommendation scores with a given aggregator function $\text{aggr}$ :

(4) \begin{equation} \mathrm{score}_{\text{aggr}}(m, t) = \text{aggr}_{q_i \in m} \{s(q_i,t)\}. \end{equation}

We propose two ways to estimate $s(q_i, t)$ , each leading to a particular family of mission-based task recommendation methods.

• • Score-based method: we take $s(q_i, t)$ to be the raw score of the query-based task recommender, $\mathrm{score}(q_i,t)$ (with $s(q_i, t)$ equal to $0$ if $t \notin R_i$ ).

• • Position-based method: we define $s(q_i,t)$ to be the reciprocal rank position of $t$ in the ranking for $q_i$ . That is, $s(q_i,t) = 1/ r_i(t)$ , where $r_i(t)$ is the rank position of recommended task $t$ in the ranking $R_i$ (with indexing starting from 1 for the top task, and setting $r_i(t) = |R_i|+1$ if $t \notin R_i$ , where $|R_i|$ is the length of the ranked list $R_i$ ).

The aggregator function is one of $\{\text{sum}, \text{max}, \text{avg}\}$ . Accordingly

• • if $\text{aggr} = \text{sum}$ , we have $\mathrm{score}_{\text{sum}}(m, t) = \sum _{q_i \in m}{s(q_i, t)}$ ;

• • $\text{aggr} = \text{max}$ leads to $\mathrm{score}_{\text{max}}(m, t) = \max _{q_i \in m} \{s(q_i, t)\}$ ;

• • and when $\text{aggr} = \text{avg}$ , $\mathrm{score}_{\text{avg}}(m, t) ={\frac{1}{n}} \sum _{q_i \in m}{s(q_i, t)}$ .

5.1 Corpus of search queries and missions

We base our experiments on the Webis Search Mission Corpus 2012 (Webis-SMC-12)Footnote ^f developed by Hagen et al. (Reference Hagen, Gomoll, Beyer and Stein2013). It contains 8840 queries issued by 127 users, corresponding to a 3-month sample from the 2006 AOL query log. These queries have been manually assigned to a total of 1378 search missions by two human assessors in full inter-annotator consensus (Hagen et al., Reference Hagen, Gomoll, Beyer and Stein2013). By using this dataset, we eliminate potential errors associated with mission detection and thus ensure that our evaluation is not affected by noisy input from an upstream processing step.

In the original dataset, consecutive queries that belong to different missions are delimited by a special “logical break” marker. Since we are not interested in the multi-tasking aspect of search behavior (i.e., switching between missions in a same session), it is only the set of all queries within a given mission that matters to us. Hence, we post-process the original corpus to merge all queries that belong to the same mission into a single set, by ignoring the logical breaks. We further remove duplicate queries when they correspond to consecutive queries with the same timestamp (effectively, we ignore multiple consecutive clicks for the same query). After this removal, the relative order among the queries in each mission is still preserved. This results in a corpus of 1378 missions, as originally, but now with a total of 3873 queries.

5.2 Corpus of procedural search missions

In the next step, three human experts assessed each of the search missions to decide whether a mission are procedural or not. The Fleiss’ Kappa inter-annotator agreement for this assessment was 0.55, which is considered moderate (Fleiss, Reference Fleiss1971). For each mission, we take the majority vote to be the final label. In the case of a mission being assessed as procedural, each assessor was also requested to choose a single query from the mission that is the most representative for the task. Further, we take the union of all queries that were chosen as most representative by at least one assessor and refer to this set as the best queries of the mission. As a result, 54 out of the 1378 missions were labeled as procedural. This amounts to approximately 4% of all missions in our sample, which is a considerable proportion. Among the 54 procedural missions, the vast majority has a single best query each (six missions have two best queries and two missions have three best queries).

Figure 3 shows the distribution of missions in our corpus according to the number of queries they contain. 724 out of the 1378 missions (i.e., around 52%) consist of a single query, and 280 missions (20%) contain each exactly two queries (Figure 3(a)). As for the procedural missions, 19 out of the 54 missions (35%) consist of a single query (Figure 3(b)). This leaves us with 35 procedural missions with 2+ queries, which is a suitable sample to be used for evaluating our second problem (mission-based task recommendation). Two example procedural missions are shown in Table 2.

Table 2.

Examples of queries for two procedural missions in our corpus

Best queries in boldface.

Figure 3.

Distribution of (a) missions and (b) procedural missions according to the number of queries they contain.

5.3 Tasks repository

As our tasks repository $\mathcal{T}$ (cf. Section 3), we use the HowToKB datasetFootnote ^g developed by Chu et al. (Reference Chu, Tandon and Weikum2017). This dataset contains 168,697 articles crawled from WikiHow. Figure 2 shows an excerpt from a WikiHow article. Each article describes a procedural task and contains the following elements: title, explanation, and sequence of steps (each made of a main act, and possibly also of a detailed act) corresponding to its subtasks. Thus, each task $t \in \mathcal{T}$ is uniquely identified by its WikiHow ID, and represented by a set of four task attributes: Title, Explanation, MainAct, and DetailedAct.

5.4 Relevance assessments

In order to build a test collection for task recommendation, we collect annotations for query-task pairs via crowdsourcing. For each annotation instance (query-task pair), crowd workers were presented with a search query, and a candidate WikiHow article consisting of a title and an explanation. We annotate every best query for all the procedural missions. We asked crowd workers to decide whether a task is highly relevant, relevant, or nonrelevant for the query. A document is highly relevant if it is the exact task that the user wants to accomplish when issuing the query. A document is relevant if it is somehow related to the task behind the query, but not exactly the searched task. Otherwise, it is nonrelevant. Table 3 lists a number of specific examples from our test collection, illustrating the relevance of tasks for queries. Our annotation process uses the well-established top-N pooling technique (Voorhees, Reference Voorhees2014); further details are provided below.

Table 3.

Examples of queries, recommended tasks, and corresponding relevance assessments in our test collection

6.1 Experimental setup

6.2 Query-based task recommendation

We begin our experimental investigation by assessing the extent to which content-based recommendation is able to recommend tasks for search queries, and how much we are able to improve by incorporating distributional semantic features. Hence, we start by identifying a ranking baseline. Our first research question is (RQ1) Which approach is a solid text-based baseline for task recommendation?

The top block of Table 4 presents the document ranking methods we experiment with. We apply the three ranking models on each of the indexed task attributes. The main observation we make is that the performances of the three models are very close to each other, for all metrics and task attributes. Indeed, using a two-tailed paired t-test with Bonferroni correction (Miller, Reference Miller1981), we find that no statistical significance exists either between SDM and BM25 or between RM3 and BM25 (for any metric or task attribute). While SDM and RM3 are known to be effective techniques across a broad range of ranking problems (Dalton, Dietz, and Allan, Reference Dalton, Dietz and Allan2014; Guo et al., Reference Guo, Fan, Ai and Croft2016), for task recommendation they fail to bring any improvements over BM25. If we focus on contrasting by attributes, we observe that Title is consistently the best performing attribute by a clear margin in all metrics and all models. This is reasonable for some of our procedural missions, since their best queries are rather well-formed, and WikiHow article titles are syntactically correct phrases, mostly adhering to the pattern “How to $\langle$ verb $\rangle$ $\langle$ noun phrase $\rangle$ $\langle$ complement $\rangle$ ,” which often matches the query very well. Examples of these queries in our test collection are “how to develop a web site,” “how to quit smoking,” and “opening a small business.” The Title attribute is followed by Explanation, in terms of importance, while the other two attributes do not have a clear order. Based on the above observations, we answer RQ1 by identifying BM25-Title as a solid term-based matching method, and will consider that henceforth as our baseline for query-based task recommendation.

Table 4.

Performance of query-based task recommendation, measured in terms of NDCG@10, P@10, and MAP

Best scores are in boldface. The symbol $^\ddagger$ denotes statistical significance of the LTR method against the baseline (BM25-Title), tested using a two-tailed paired t-test with Bonferroni correction (Miller, Reference Miller1981) at $p\lt 0.000\widehat{3}$ .

Next, we wish to assess the effectiveness of our semantic matching features that are combined in a LTR approach. This gives rise to the following research question: (RQ2) How effective are our semantic matching features and our LTR approach for query-based task recommendation?

Evaluation results for our LTR approach are shown in the bottom line of Table 4. Clearly, the improvements w.r.t. the baseline, BM25-Title, are substantial (42% in terms of NDCG@10, 38% in terms of P@10, and 65% in terms of MAP). Moreover, we use a two-tailed paired t-test with Bonferroni correction at $p\lt 0.000\widehat{3}$ and verify that these improvements are statistically highly significant for every metric. Hence, the answer to RQ2 is that leveraging semantic matching signals in a LTR approach highly significantly constitutes a more effective method than any of the existing term-based baselines. These results attest that we are able to help substantially and significantly reduce the vocabulary mismatch problem between queries and tasks.

The semantic features proposed in our approach (cf. Section 4.1.3) encompass several different signals, taken from all task attributes, word embeddings, and different combinations of possible word functions. To understand how each of these features contribute to the overall performance, we formulate the following question to guide our analysis: (RQ3) What are the most important features when learning to recommend tasks?

We start answering this research question by performing an ablation study on the semantic feature groups. Table 5 presents the results of this additional experiment. The first three rows of the ending block correspond to studying the removal of one or both embedding-based feature groups from LTR. The experimental results show that even after removing either W2V or DESM groups entirely, the respective method still outperforms the baseline highly significantly across all metrics, and in some cases achieves performances comparative to or even slightly better than LTR. The removal of both W2V and DESM feature groups affects the performance with higher impact, yet retains significance in its improvements w.r.t. the baseline. We also observe the performance of the LTR configuration when we remove one or both transformer-based feature groups from it. This ablation results are reported in the last three rows of the ending block of Table 5. Across all metrics, it can be seen the large contribution of these features for LTR to improve w.r.t. the baseline with high significance. It also shows that these two underlying transformer-based models complement each other, with MPNet features contributing more than RoBERTa ones to the overall performance of the full set. The configuration where any of the two transformer-based feature groups is removed is significantly better than the baseline, in terms of NDCG@10 and P@10, but these improvements are not highly significant. The removal of both transformer-based feature groups produces a loss of all significance in the improvements measured with any of these two metrics. However, in terms of MAP, the improvement of any of these LTR configurations with less features is always highly significant. Hence, in a scenario where the interest is in the whole ranking, a smaller, simpler set of features may be enough.

Table 5.

Performance of query-based task recommendation, in terms of NDCG@10, P@10, and MAP

The best performing configuration (LTR) is compared to its corresponding configurations without the features from one or more feature groups. The symbols $^\dagger$ and $^\ddagger$ denote statistical significance of a method against the baseline (BM25-Title), tested using a two-tailed paired t-test with Bonferroni correction (Miller, Reference Miller1981) at $p\lt 0.01\widehat{6}$ and $p\lt 0.000\widehat{3}$ , respectively.

Furthermore, we analyze the discriminative power of our features by sorting them according to their individual information gain, measured in terms of Gini importance; these are shown as the vertical bars in Figure 4. Then, we incrementally add features, one by one, according to their importance, to obtain corresponding feature sets made of the $I$ most important features ( $I \in \{1, \ldots, 100\}$ ), and report on performance, in terms of NDCG@10; this is shown as the line plot in Figure 4. Since the number of features changes in each iteration, we set the $m$ parameter of the Random Forests algorithm to (the ceiling of) the 10% of the size of the feature set.

Figure 4.

Performance of our LTR approach, in terms of NDCG@10, when incrementally adding features according to their individual information gain, measured by Gini score. The feature grouping is described in Table 1.

Our findings are as follows. The top three features are, in order, MPNet-Title, RoBERTa-Title, and BM25-Title. Moreover, 11 out of the 15 most important features are all obtained from the Title attribute. This confirms the importance of Title as the most informative task attribute. Still observing the 15 most important features, after the top three, the following 12 include RoBERTa and MPNet each obtained from Explanation and DetailedAct attributes, and another eight semantic features which all use the W2V embedding. The expected importance of the features based on transformer-based models is confirmed again in these results, as well as the complementary contributions of these two models. Also, the prominence of W2V features shows that, contrarily to what we expected as explained in Section 4.1.2, the typical similarities that dominate the way this embedding is usually employed contributes more to the semantic matching than the topical similarity in the IN-OUT strategy with DESM. These semantic features use mainly the word functions $V+N+A$ and $V+N$ , which shows, as expected, the importance of selected words as verbs and nouns, yet achieving practically same contributions since very few adjectives occur in the queries or task titles. These are followed by COMPLEMENT word functions.

Looking at how performance changes when adding features one-by-one, we can already achieve a result (NDCG@10 of 0.3571) comparable to the baseline with only the most important feature. Furthermore, we can actually outperform the baseline by adding just a single feature (NDCG@10 of 0.4411). The addition of the third most important feature, the baseline itself BM25-Title, slightly keeps improving the performance (NDCG@10 of 0.4552). From this point on, we observe a low fluctuation in performance, where each new feature sometimes leads to an improvement and other times produces a small deterioration. We find that performance peaks at an NDCG@10 of 0.5277, which is achieved with 35 features. Adding more features slightly hurts performance, but not significantly so. In answer to RQ3, we remark the effectiveness of (i) textual similarity, based on distributional semantic representations, in particular, from transformer-based models, (ii) selected word functions, and with (iii) highest contributions from the Title task attribute.

Since the task attributes are well-formed expressions in natural language, we conduct an additional experiment aiming to shed light on how task recommendation performs for queries that might not fall under this kind of expressions. We approach this observation by partitioning the query set into disjoint subsets of queries, according to the number of words in each query. In our test collection, the largest number of words for a query is 10. This is, given the set $Q$ of 59 queries, we measure the LTR performance in each element of the partition $\mathcal{Q} = \{Q_i \subseteq Q | \text{each query in } Q_i \text{ has } i \text{ words}\}_{i=1}^{10}$ . Figure 5 shows the the results obtained in this experiment. We can observe that, with the exception of the single query made of one word, for very short queries, the performance is lower across all the metrics. LTR performs better as the word counter increases; this should correspond with queries made of four or more words, such as “how to mount tires” or “learn to remodel a house,” which are the ones that can present a verb and a direct object, as well as some opening segment (e.g., “how to,” “learn to,” “how do I” ) related to the user intent for a procedural task. Some queries of these lengths present slightly different structure (e.g., no opening segment before the verb in “block incoming email addresses”) but in general these are queries long enough to express their underlying intents in a more specific level of detail. For queries with seven or more words, task recommendation increasingly deteriorates, which makes sense as longer queries seem to become too specific to get appropriate recommendations from the task repository.

6.3 Mission-based task recommendation

In this second part of the analysis, our experiments aim to answer the research question (RQ4) Can task recommendation performance benefit from having access to all search queries in a mission, in addition to the best query?

We report on all combinations of scoring methods (score-based vs. position-based) and aggregator functions ( $\text{sum}$ , $\text{max}$ , or $\text{avg}$ ) in Table 6. Building on our feature analysis, we conduct this part of the experiments learning to recommend with only the 35 most important features. In particular, the first row of Table 6 reports the performance of query-based task recommendation using only such an optimized features set. For mission-based task recommendation, in order to make a fair comparison, we randomly select a single best query per mission and compare against the query-based LTR restricted to only these selected best queries. Overall, the results are mixed. In terms of NDCG@10 and MAP, none of the mission-based methods improve over the query-based approach. On the other hand, when observing P@10, most cases in the mission-based approach exhibit an improvement. However, none of these differences is statistically significant for any metric.

Table 6.

Performance of mission-based task recommendation, compared against a query-based method, in terms of NDCG@10, P@10, and MAP

Best scores for each metric are in boldface; improvements over the query-based method are underlined. None of these performance differences are found to be statistically significant.

Figure 5.

Performance of our LTR approach for query-based task recommendation (in terms of NDCG@10, P@10, and MAP), measured on each of the subsets given by partitioning the query set according to the number of words in each query.

To shed some light on the reasons behind this, we perform the following analysis. In Figure 6, we plot the differences in NDCG@10, for each individual mission, between a given mission-based configuration and the query-based method. Each positive bar corresponds to a mission where the mission-based method improved over the query-based one, and each negative bar represents a mission where the opposite is true. A key observation we make here is that there are indeed substantial differences on some missions, and that in some cases there are about as many wins as losses, with other cases having a tendency to the latter. There are several possible reasons for this. Since the query-based method uses the best query in a mission, which is identified by an oracle, it may be that, on average, it simply does not help to consider other queries in the mission as those often only bring in noise. Weighing queries in a mission may help to rectify it to some extent; we leave this exploration to future work. Another reason may lie in how the test collection was created. Recall that relevance labeling was done based only on the best queries, and not on other queries in the mission. Those other queries could possibly yield more relevant tasks, which have not been labeled as such.

Figure 6.

Differences in NDCG@10 per query between a given mission-based method and the query-based method.

We now report some of the particular examples when analyzing results of individual missions. Table 7 presents best and worst performing missions per configuration, as well as examples of relevant tasks in the test collection, and their rank position in each ranking. A mission with an NDCG@10 difference of 0.46 on each of the score-based methods with $\text{sum}$ or $\text{avg}$ aggregator (NDCG@10 difference of 0.47 in the $\text{max}$ -aggregator method) has “how to create a petition not online” as one of its two best queries, the other best query being “how to create a petition.” For this mission, all score-based methods rank one of the only two highly relevant tasks, “Write a Petition,” as the top result while LTR ranks it as its third best result. This mission comprises three queries. Only the two best queries have a verb, “create,” but one of them does not include “not online.” When these two queries in the mission are taken into account, alongside its third query, “petition,” it becomes possible for the highly relevant task to improve its ranking with the mission-based method. The score-based method also exhibits a mission with a negative NDCG@10 difference of −1.0 for $\text{sum}$ (Figure 6a) and $\text{avg}$ (Figure 6c) aggregators (NDCG@10 difference of −0.67 for $\text{max}$ ; Figure 6b) as its worst performing mission. Represented by “michigan teacher certification,” the mission has a single relevant task in the test collection, “Become Certified to Teach Hunter Safety in Michigan,” ranked as top result by query-based LTR, but only as low ranked as the eleventh or seventh result by a score-based method. This mission has five queries, most of them mentioning terms such as “teaching license exam” and “gid promotions,” so these additional queries in the mission should help to express the actual user intent searching for general (e.g., primary and secondary school) teaching. And in practice, they contribute to lower the rank for the task of getting certified for hunter safety, which is realistically only tangentially relevant for teacher certification. However, since this is the only relevant task in the test collection, in our evaluation, the mission-based method performs as if it is introducing noise for this query.

Table 7.

Queries with the largest $\Delta$ NDCG@10 differences, using a mission-based configuration, in comparison with the query-based method

Examples of relevant tasks in the test collection, as well as their rank position in the respective mission-based (MB) and query-based LTR ranking, are also presented.

For position-based methods (Figure 6d–6f), the phenomena regarding best and worst differences are very similar to score-based methods. The positive differences for position-based methods are slightly larger than for score-based methods, while the negative differences for position-based methods are slightly smaller than their score-based counterpart. The largest NDCG@10 difference (+0.63) for a position-based method with either $\text{sum}$ or $\text{avg}$ aggregator corresponds to a mission with “install spyware” as its best query. For this mission, both position-based methods, as well as query-based LTR, rank a non-relevant task, “Find Spyware,” as their top result. However, the only relevant task, “Remove Spyware Manually (Windows),” is ranked by query-based LTR beyond the rank 10 cutoff, whereas each position-based method ranks it as its second best result. This mission comprises 10 queries, most of them with an occurrence of the term “spyware.” Only the best query has a verb, “install,” antonym of the verb “remove” in the relevant task. This vocabulary drift hinders the query-based ranking for this query, whereas the mission-based method is able to address it.

A mission with one of the worst differences in all position-based methods is represented by the best query “using the normal curve to find probability” and has a total of 17 queries drifting through several topics related to statistics, such as “linear regression” and “sample size.” The single relevant task for this mission, “Calculate Probability,” is the top recommended task by query-based LTR but only ranked as the third result at highest in all position-based methods. The best-performing mission for position-based method with $\text{max}$ has a best query “how to create a petition not online,” and corresponds to the same mission that performed best for the score-based methods previously analyzed. The position-based method with $\text{max}$ aggregator ranks one of the only two highly relevant tasks, “Write a Petition,” as its second best result, whereas LTR ranks it at rank 3.

In answer to RQ4, we find that our extensions of the query-based approach are promising, but more work is needed to be able to benefit from additional queries in a mission and to reach significant improvements.