2.1 Tree-ensemble learning algorithms

Tree-Ensemble (TE) learning algorithms are machine learning methods widely used in practice, typically, when learning from tabular datasets. A trained TE model consists of multiple base decision trees, each trained on an independent subset of the input data. For example, Random Forests (Breiman Reference Breiman2001) and Gradient Boosted Decision Tree (GBDT) (Friedman Reference Friedman2001) are tree-ensemble learning algorithms. Recent surge of efficient and effective GBDT algorithms, for example, LightGBM (Ke et al. Reference Ke, Meng, Finley, Wang, Chen, Ma, Ye and Liu2017), has led to wide adoption of TE learning algorithms in practice. Although individual decision trees are considered to be interpretable (Huysmans et al. Reference Huysmans, Dejaeger, Mues, Vanthienen and Baesens2011), ensembles of decision trees are seen as less interpretable.

The purpose of using TE learning algorithms is to train models that predict the unknown value of an attribute $y$ in the dataset, referred to as labels, using the known values of other attributes $\mathbf{x}=(x_1,x_2,\ldots, x_m)$ , referred to as $\textit{features}$ . For brevity, we restrict our discussion to classification problems. During the training or learning phase, each input instance to the TE learning algorithm is a pair of features and labels, that is $(\mathbf{x}_i, y_i)$ , where $i$ denotes the instance index, and during the prediction phase, each input instance only includes features, $(\mathbf{x}_i)$ , and the model is tasked to produce predictions $\hat{y}_i$ . A collection of input instances, complete with features and labels, is referred to as a dataset. Given a dataset $\mathscr{D}=\{(\mathbf{x}_i, y_i)\}$ with $n\in \mathbb{N}$ examples and $m\in \mathbb{N}$ features, a decision tree classifier $t$ will predict the class label $\hat{y}_i$ based on the feature vector $\mathbf{x}_i$ of the $i$ -th sample: $\hat{y}_i = t(\mathbf{x}_i)$ . A tree-ensemble $\mathscr{T}$ uses $K\in \mathbb{N}$ trees and additionally an aggregation function $f$ over the $K$ trees which combines the output from the trees: $\hat{y}_i = f(t_{k\in K}(\mathbf{x}_i))$ . As for Random Forest, for example, $f$ is a majority voting scheme (i.e., argmax of sum), and in GBDT $f$ may be a summation followed by softmax to obtain $\hat{y}_i$ in terms of probabilities.

In this paper, a decision tree is assumed to be a binary tree where the internal nodes hold split conditions (e.g., $x_1 \leq 0.5$ ) and leaf nodes hold information related to class labels, such as the number of supporting data points per class label that have been assigned to the leaf nodes. Richer collections of decision trees provide higher performance and less uncertainty in prediction compared to a single decision tree. Typically, each TE model has specific algorithms for learning base decision trees, adding more trees and combining outputs from the base trees to produce the final prediction. In GBDT, the base trees are trained sequentially by fitting the residual errors from the previous step. Interested readers are referred to Friedman (Reference Friedman2001), and its more recent implementations, LightGBM (Ke et al. Reference Ke, Meng, Finley, Wang, Chen, Ma, Ye and Liu2017) and XGBoost (Chen and Guestrin Reference Chen and Guestrin2016).

2.2 Answer set programming

Answer Set Programming (Lifschitz Reference Lifschitz2008) has its roots in logic programming and non-monotonic reasoning. A normal logic program is a set of rules of the form

\begin{equation*}\mathrm {a_1} \ \text {:-} \ \ \mathrm {a_2},\ \dots, \ \mathrm {a_m}, \ \mathrm {not} \ \mathrm {a_{m+1}}, \ \dots, \ \mathrm {not} \ \mathrm {a_n}.\end{equation*}

where each $\mathrm{a_i}$ is a first-order atom with $1 \leq \mathrm{i} \leq \mathrm{n}$ and not is default negation. If only $\mathrm{a_1}$ is included ( $\mathrm{n} = 1$ ), the above rule is called a fact, whereas if $\mathrm{a_1}$ is omitted, it represents an integrity constraint. A normal logic program induces a collection of intended interpretations, which are called answer sets, defined by the stable model semantics (Gelfond and Lifschitz Reference Gelfond and Lifschitz1988). Additionally, in modern ASP systems, constructs such as conditional literals and cardinality constraints are supported. The former in clingo (Gebser et al. Reference Gebser, Kaminski, Kaufmann and Schaub2014) is written in the form $\{ \mathrm{a}(\texttt{X}) \ \text{:} \ \mathrm{b}(\texttt{X}) \}$ Footnote ² and expanded into the conjunction of all instances of $\mathrm{a}(\texttt{X})$ where corresponding $\mathrm{b}(\texttt{X})$ holds. The latter are written in the form $s_1 \ \{ \mathrm{a}(\texttt{X}) \ \text{:} \ \mathrm{b}(\texttt{X}) \} \ s_2$ , which is interpreted as $s_1 \leq \texttt{#count} \{ \mathrm{a}(\texttt{X}) \ \text{:} \ \mathrm{b}(\texttt{X}) \} \leq s_2$ where $s_1$ and $s_2$ are treated as lower and upper bounds, respectively; thus the statement holds when the count of instances $\mathrm{a}(\texttt{X})$ where $\mathrm{b}(\texttt{X})$ holds, is between $s_1$ and $s_2$ . The minimization (or maximization) of an objective function can be expressed with $\texttt{#minimize}$ (or $\texttt{#maximize}$ ). Similarly to the #count aggregate, the #sum aggregate sums the first element (weight) of the terms, while also following the set property. clingo supports multiple optimization statements in a single program, and one can implement multi-objective optimization with priorities by defining two or more optimization statements. For more details on the language of clingo, we refer the reader to the clingo manual.Footnote ³

2.3 Pattern mining

In a general setting, the goal of pattern mining is to find interesting patterns from data, where patterns can be, for example, itemsets, sequences, and graphs. For example, in frequent itemset mining (Agrawal and Srikant Reference Agrawal and Srikant1994), the task is to find all subsets of items that occur together more than the threshold count in databases. In this work, a pattern is a set of predictive rules. A predictive rule has the form $ c \Leftarrow s_1 \wedge s_2 \wedge, \ldots, s_n$ , where $c$ is a class label, and $\{s_i\}$ ( $1 \leq i \leq n$ ) represents conditions.

For pattern mining with constraints, the notion of dominance is important, which intuitively reflects the pairwise preference relation $(\lt ^*)$ between patterns (Negrevergne et al. Reference Negrevergne, Dries, Guns and Nijssen2013). Let $C$ be a constraint function that maps a pattern to $\{\top, \bot \}$ , and let $p$ be a pattern, then the pattern $p$ is valid iff $C(p)=\top$ , otherwise it is invalid. An example of $C$ is a function which checks that the support of a pattern is above the threshold. The pattern $p$ is said to be dominated iff there exists a pattern $q$ such that $p \lt ^* q$ and $q$ is valid under $C$ . Dominance relations have been used in ASP encoding for pattern mining (Paramonov et al. Reference Paramonov, Stepanova and Miettinen2019).

3.1 Problem statement

The rule set generation problem is represented as a tuple $P=\{R,M,C,O\}$ , where $R$ is the set of all rules extracted from the tree-ensemble, $M$ is the set of meta-data and properties associated with each rule in $R$ , $C$ is the set of user-defined constraints including preferences, and $O$ is the set of optimization objectives. The goal is to generate a set of rules from $R$ by selection under constraints $C$ and optimization objectives $O$ , where constraints and optimization may refer to the meta-data $M$ . In the following sections, we describe how we construct each $R$ , $M$ , $C$ , and $O$ , and finally, how we solve this problem with ASP.

3.2 Rule extraction from decision trees

This subsection describes how $R$ , the set of all rules, is constructed. The first two steps in “tree-ensemble processing” in Figure 1 are also described in this subsection. Recall that a tree-ensemble $\mathscr{T}$ is a collection of $K$ decision trees, and we refer to individual trees $t_k$ with subscript $k$ . An example of a decision tree-ensemble is shown in Figure 2. A decision tree $t_k$ has $N_{t_k}$ nodes and $L_{t_k}$ leaves. Each node represents a split condition, and there are $L_{t_k}$ paths from the root node to the leaves. For simplicity, we assume only features that have orderable values (continuous features) are present in the dataset in the examples below.Footnote ⁴ The tree on the left in Figure 2 has four internal nodes including the root node with condition $[x_1 \leq 0.2]$ and five leaf nodes; therefore, there are five paths from the root note to the leaf nodes 1 to 5.

Fig 2. A simple decision tree-ensemble consisting of two decision trees. The rule associated with each node is given by the conjunction of all conditions associated with nodes on the paths from the root node to that node.

From the left-most path of the decision tree on the left in Figure 2, the following prediction rule is created. We assume that node 1 predicts class label 1 in this instance.Footnote ⁵

(1) \begin{equation} class(1) \Leftarrow (x_1 \leq 0.2) \wedge (x_2 \leq 4.5) \wedge (x_4 \leq 2) \end{equation}

Assuming that node 2 predicts class label 0, we also construct the following rule (note the reversal of the condition on $x_4$ ):

(2) \begin{equation} class(0) \Leftarrow (x_1 \leq 0.2) \wedge (x_2 \leq 4.5) \wedge (x_4 \gt 2) \end{equation}

Algorithm 1 Construct candidate rule set R

To obtain the candidate rule set, we essentially decompose a tree-ensemble into a rule set. The steps are outlined in Algorithm1. By constructing the candidate rule set $R$ in this way, the bodies (antecedents) of rules included in rule sets are guaranteed to exist in at least one of the trees in the tree-ensemble. Rule sets generated in this manner are therefore faithful to the representation of the original model in this sense. If we were to construct rules from the unique set of split conditions, the resulting rule may have combinations of conditions that do not exist in any of the trees.

We now analyze the computational complexities associated with constructing the set of all rules $R$ . Let us assume that (1) all $K$ trees in the ensemble are perfect binary decision trees and have the same height $h$ , (2) there are $n$ examples and $m$ features in the dataset, and (3) there are no duplicate rules and conditions across trees.

This follows immediately from the number of leaf nodes in a perfect binary decision tree with height $h$ , that is $2^h$ . In practice, there are duplicate split conditions across trees in a tree-ensemble, so the unique count of rules is often smaller than the maximum value.

3.3 Computing metrics and meta-data for selection

After the candidate rule set $R$ is constructed, we gather information about the performance and properties of each rule and collect them into a set $M$ . This is the last step in the tree-ensemble processing process depicted in Figure 1 (“Assign Metrics”). The meta-data, or properties, of a rule are information such as the size of the rule, as defined by the number of conditions in the rule, and the ratio of instances which are covered by the rule. Computing classification metrics, for example, accuracy and precision, requires access to a subset of the dataset with ground truth labels, which could be either a training or a validation set. On the other hand, when access to the labeled subset is not available at runtime, these metrics and their corresponding predicates cannot be used in the ASP encoding. In our experiments, we used the training sets to compute these classification metrics during rule set generation, and later used the validation sets to evaluate their performance.

Performance metrics measure how well a rule can predict class labels. Here we calculate the following performance metrics: accuracy, precision, recall, and F1-score, as shown below.

(3) \begin{align} \textit{accuracy} &= \frac{TP + TN}{TP + TN + FP + FN} \qquad & \textit{precision} &= \frac{TP}{TP+FP} \nonumber \\ \textit{recall} &= \frac{TP}{TP+FN} \qquad & \textit{F1-score} &= 2 \times \frac{precision \times recall}{precision + recall} \end{align}

For classification tasks, a true positive (TP) and a true negative (TN) refer to a correctly predicted positive class and negative class, respectively, with respect to the labels in the dataset. Conversely, a false positive (FP) and a false negative (FN) refer to an incorrectly predicted positive class and negative class, respectively. These metrics are not specific to the rules and can be computed for trained tree-ensemble models, as well as explanations of trained machine learning models, as we shall show later in Section 5.2.4. We compute multiple metrics for a single rule, to meet a range of user requirements for explanation. One user may only be interested in simply the most accurate rules (maximize accuracy), whereas another user could be interested in more precise rules (maximize precision), or rules with more balanced performance (maximize F1-score).

The candidate rule set $R$ and meta-data set $M$ are represented as facts in ASP, as shown in Table 1. For example, Rule 1 (the first rule in Section 3.2) may be represented as follows:Footnote ⁶

Unique conditions are indexed and denoted by the condition predicate. For instance, in the example above (representing Rule 1), “condition(1,1)” represents $(x_1 \leq 0.2)$ , “condition(1,2)” corresponds to $(x_2 \leq 4.5)$ , and so forth.

Table 1. List of predicates representing a rule in ASP

$^{a}$ Properties and metrics marked with asterisks(*) are multiplied by 100 and rounded to the nearest integer.

3.4 Encoding inclusion criteria and constraints

As with previous works in pattern mining in ASP, we follow the “generate-and-test” approach, where a set of potential solutions are generated by a choice rule and subsequently constraints are used to filter out unacceptable candidates. In the context of rule set generation, we use a choice rule to generate candidate rule sets that may constitute a solution (“Generate Candidate Rule Sets” in Figure 1). In this section, we introduce the following selection criteria and constraints: (1) individual rule selection criteria that are applied on a per-rule basis, (2) pairwise constraints that are applied to pairs of rules, and (3) collective constraints that are applied to a set of rules.

The “generator” choice rule has the following form:

Example 1.

The choice rule above generates candidate subsets of size between 1 and 5 from $R$ , where we use the selected/1 predicate to indicate that a rule (rule(X)) is included in the subset.

Individual rule selection criteria are integrated into the generator choice rule by the valid/1 predicate, where a rule rule(X) is valid whenever invalid(X) cannot be inferred.

Example 2. The following criterion excludes rules with low support from the candidate set:

Pairwise constraints can be used to encode dominance relations between rules. For a rule X to be dominated by Y, Y must be strictly better in one criterion than X and at least as good as X or better in other criteria. In the following case, we encode the dominance relation between rules using the accuracy metric and support, where we prefer rules that are accurate and cover more data.

Collective constraints are applied to collections of rules, as opposed to individual or pairs of rules. The following restricts the maximum number of conditions in rule sets, using the #sum aggregate:Footnote ⁷

We envision two main use cases for the criteria and constraints introduced in this section: (1) to generate rule sets with certain properties, and (2) to reduce the computation time. For (1), the user can use the individual selection criteria to ensure that the rules included into the candidate rule sets have certain properties, or the collective constraints to put restrictions on the aggregate properties of the rule sets. The latter use-case has more practical relevance because in our case, as in pattern mining, the complexity of a naive “generate-and-test” approach is exponential with respect to the number of candidates.

To reduce the search space, one can place an upper bound on the size of generated candidate sets and use the invalid/1 predicate to prevent unacceptable rules being included into the candidates, as shown above. Because setting unreasonable conditions leads to zero rule sets generated, care should be taken when using the selection criteria and constraints for this purpose. In particular, if any of the metric predicates listed in Table 1 are used in defining invalid/1, for example, invalid(X) :- rule(X), metric(X, N), N < B., to avoid all rule(X) being invalid(X), one should respect the conditions listed in Table 2. In the following example, we will show how the invalid/1 predicate can be used to reduce the search space.

Table 2. List of minimum and maximum values for the bounds used in defining $\texttt{invalid/1}$

Example 3. Let the logic program be:

Then, there is at least one valid rule if $\texttt{B} \leq max(\texttt{N}_1,\ldots, \texttt{N}_{|R|})$ . Let $\texttt{B} = 1+max(\texttt{N}_1,\ldots, \texttt{N}_{|R|})$ , then by line 3 ( $\texttt{N} \lt \texttt{B}$ ), all rules will be invalid, and valid(X) cannot be inferred. Then, the choice rule (line 1) is not satisfied. Alternatively, let $\texttt{B} = max(\texttt{N}_1,\ldots, \texttt{N}_{|R|})$ , then there is at least one rule such that $\texttt{N} = \texttt{B}$ . Since $\texttt{invalid(X)}$ cannot be inferred for such a rule, it is $\texttt{valid}$ and the choice rule is satisfied.

The upper bound parameter (5 in Example1 and 1 in Example3) controls the potential maximum number of rules that can be included in a rule set. The actual number of rules that emerge in the final rule sets is highly dependent on the selection criteria, user preferences, and the characteristics of the tree-ensemble model. Practically, we recommend initially setting this parameter to a lower value (e.g., 3) while focusing on refining other aspects of the encoding, since this allows for a more manageable starting point. Given the “generate-and-test” approach, high values may lead to excessively slow run time. If it becomes evident that a larger rule set could be beneficial, the parameter can be incrementally increased. This ensures more efficient use of computational resources while also catering to the evolving needs of the encoding process.

3.5 Optimizing rule sets

Finally, we pose the rule set generation problem as a multi-objective optimization problem, given the aforementioned facts and constraints encoded in ASP. The desiderata for generated rule sets may contain multiple competing objectives. For instance, we consider a case where the user wishes to collect accurate rules that cover many instances, while minimizing the number of conditions in the set. This is encoded as a group of optimization statements:

Instead of maximizing/minimizing the sums of metrics, we may wish to optimize more nuanced metrics, such as average accuracy and coverage of selected rules:

This metric can be maximized by selecting the smallest number of short and accurate rules. Similar metrics can be defined for precision-coverage,

and for precision-recall.

For optimization, we introduce a measure of overlap between the rules to be minimized. Intuitively, minimizing this objective should result in rule sets where rules share only a few conditions, which should further improve the explainability of the resulting rule sets. Specifically, we introduce a predicate rule_overlap(X,Y,Cn) to measure the degree of overlap between rules X and Y.

5.1 Experimental setup

5.1.1 Datasets

We used in total 14 publicly available datasets, where except for the adult Footnote ⁸ dataset, all datasets were taken from the UCI Machine Learning RepositoryFootnote ⁹ (Dua and Graff Reference Dua and Graff2017). Datasets were chosen from public repositories to ensure a diverse range in terms of the number of instances, the number of categorical variables, and class balance. This was done intentionally, to observe the variance in, for example, explanation generation times. Additionally, the variation in categorical variables ratio and class balance was designed to produce a wide array of tree-ensemble configurations (e.g., more or fewer trees, varying widths and depths). We expected these configurations, in turn, to influence the nature of the explanations generated. We included 3 datasets (adult, credit german, compas) for comparison because they were widely used in local explainability literature. The adult dataset is actually a subset of the census dataset, but we included the former for consistency with existing literature, and the latter for demonstrating the applicability of our approach to larger datasets. The summary of these datasets is shown in Table 4.

Table 4. Dataset description, selected hyperparameters and candidate rule counts

$^{a}$ The maximum depth parameter. The minimum and maximum values set during the hyperparameter search are shown in parentheses. The hyperparameters shown in this table are averaged over 5 folds. $^{b}$ The number of candidate rules, averaged over 5 folds. $^{c}$ The number of trees (estimators) parameter. $^{d}$ The number of features (columns). The number of categorical features is shown in parentheses.

5.1.2 Experimental settings

We used clingo 5.4.0Footnote ¹⁰ (Gebser et al. Reference Gebser, Kaminski, Kaufmann and Schaub2014) for ASP and set the time-out to 1,200 s. We used RIPPER implemented in Weka Witten et al. (Reference Witten, Frank and Hall2016) and an open-source implementation of RuleFitFootnote ¹¹ where Random Forest was selected as the rule generator, and scikit-learnFootnote ¹² (Pedregosa et al. Reference Pedregosa, Varoquaux, Gramfort, Michel, Thirion, Grisel, Blondel, Prettenhofer, Weiss, Duobourg, Vanderplas, Passos, Cournapeau, Brucher, Perrot and Duchesnay2011) for general machine learning functionalities. Our experimental environment is a desktop machine with Ubuntu 18.04, Intel Core i9-9900K 3.6 GHz (8 cores/16 threads) and 64 GB RAM. For reproducibility, all source codes for the implementation, experiments, and preprocessed datasets are available from our GitHub repository.Footnote ¹³

Unless noted otherwise, all experimental results reported here were obtained with 5-fold cross validation, with hyperparameter optimization in each fold. To evaluate the performance of the extracted rule sets, we implemented a naive rule-based classifier, which is constructed from the rule sets extracted with our method. In this classifier, we apply the rules sequentially to the validation dataset and if all conditions within a rule are true for an instance in the dataset, the consequent of the rule is returned as the predicted class. More formally, given a set of rules $R_s \subset R$ with cardinality $|R_s|$ that shares the same consequent $class(Q)$ , we represent this rule-based classifier as the disjunction of antecedents of the rules:

\begin{equation*} class(Q) \Leftarrow body(R_1) \vee body(R_2) \vee \ldots \vee body(R_r) \textrm { where } 1 \leq r \leq |R_s| \end{equation*}

For a given data point, it is possible that there are no rules applicable, and in such cases the most common class label in the training dataset is returned.

5.2 Evaluating global explanations

Let us recall that the purpose of generating global explanations is to provide the user with a simpler model of the original complex model. Thus, we introduce proxy measures to evaluate (1) the degree to which the model is simplified, by the number of extracted rules and conditions, (2) the relevance of the extracted rules, by comparing classification performance metrics against the original model, and (3) the degree to which the explanation accurately describes the behavior of the original model, by fidelity metrics.

We conducted the experiment in the following order. First, we trained Decision Tree, Random Forest, and LightGBM on the datasets in Table 4. Selected optimized hyperparameters of the tree-ensemble models are also reported in Table 4. Further details on hyperparameter optimization are available in Appendix B. We then applied our rule set generation method to the trained tree-ensemble models. Finally, we constructed a naive rule-based classifier using the set of rules extracted in the previous step and calculated performance metrics on the validation set. This process was repeated in a 5-fold stratified cross validation setting to estimate the performance. We compare the characteristics of our approach against the known methods RIPPER and RuleFit.

We used the following selection criteria to filter out rules that were considered to be undesirable; for example, those rules with low accuracy or low coverage. We used the same set of selection criteria for all datasets, irrespective of underlying label distribution or learning algorithms. When the candidate rules violate any one of those criteria, they are excluded from the candidate rule set, which means that in the worst case where all the candidate rules violate at least one criterion, this encoding will result in an empty rule set (see Section 3.4).

Another scenario in which our method will produce an empty rule set is when the tree-ensemble contains only “leaf-only” or “stump” trees, that have one leaf node and no splits. In this case, we have no split information to create candidate rules; thus, an empty rule set is returned to the user. This is often caused by inadequate setting of hyperparameters that control the growth of the trees, especially when using imbalanced datasets. It is however outside the scope of this paper, and we will simply note such cases (empty rule set returned) in our results without further consideration.

5.2.1 Number of rules

The average sizes of generated rule sets are shown in Table 5. The sizes of candidate rule sets, from which the rule sets are generated, are listed in the $|R|$ columns in Table 4. Rule set size of 1 means that the rule set contains a single rule only. As one might expect, the Decision Tree consistently has the smallest candidate rule set, but in some cases the Random Forest produced considerably more candidate rules than the LightGBM, for example, cars, compas. Our method can produce rule sets which are significantly smaller than the original model, based on the comparison between the sizes of the candidate rule set $|R|$ and resulting rule sets.

Table 5. Size of rule sets, total and average number of conditions in rules

$^{a}$ Decision Tree + ASP $^{b}$ Random Forest + ASP $^c$ LightGBM + ASP

We will now compare our method to the two benchmark methods, RuleFit and RIPPER. The average size of generated rule sets is shown in Table 5. RuleFit includes original features (called linear terms) as well as conditions extracted from the tree-ensembles in the construction of a sparse linear model, that is to say, the counts in Table 5 may be inflated by the linear terms. On the other hand, the output from RIPPER only contains rules, and RIPPER has rule pruning and rule set optimization to further reduce the rule set size. Moreover, RIPPER has direct control over which conditions to include into rules, whereas our method and RuleFit rely on the structure of the underlying decision trees to construct candidate rules. Our method consistently produced smaller rule sets compared to RuleFit and RIPPER, although the difference between our method and RIPPER was not as pronounced when compared to the difference between our method and RuleFit. RuleFit produced the largest number of rules compared with other methods, although they were much smaller than the original Random Forest models (Table 5).

5.2.3 Relevance of rules

To quantify the relevance of the extracted rules, we measured the ratio of performance metrics using the naive rule-based classifier by 5-fold cross validation (Table 6). A performance ratio of less than 1.0 means that the rule-based classifier performed worse than the original classifier (LightGBM and Random Forest), whereas a performance ratio greater than 1.0 means the rule set’s performance is better than the original classifier. We used a version of the ASP encoding shown in Section 3.5 where the accuracy and coverage are maximized. RIPPER was excluded from this comparison because it has a built-in rule generation and refinement process, and it does not have a base model, whereas our method and RuleFit use variants of tree-ensemble models as base models.

Table 6. Average ratio of rule-based classifiers’ performance vs. original tree-ensembles, averaged over 5 folds. (Global explanations)

$^a$ Decision Tree + ASP $^b$ Random Forest + ASP $^c$ LightGBM + ASP

From Table 6 we observe that in terms of accuracy, RuleFit generally performs as well as, or marginally better than, the original Random Forest. On the other hand, although our method can produce rule sets that are comparable in performance against the original model, they do not produce rules that perform significantly better. With Decision Tree and Random Forest, the generated rule sets perform much worse than the original model, for example, in kidney, voting. The LightGBM+ASP combination resulted in the second-best performance overall, where the resulting rules’ performances were arguably comparable (0.8-0.9 range) to the original model with a few exceptions (e.g., census F1-score) where the performance ratio was about half of the original. While RuleFit’s performance was superior, our method could still produce rule sets with reasonable performance with much smaller rule sets that are an order of magnitude smaller than RuleFit. A rather unexpected result was that using our method (Random Forest) or RuleFit significantly improved the F1-score in the census dataset. In Table 6 we can see that recall was the major contributor to this improvement.

5.2.4 Fidelity metrics of global explanations

In Section 5.2.3, we compared the ratio of performance metrics of different methods when measured against original labels. In the context of evaluating explanation methods, it is also important to investigate the fidelity, that is to which extent the explanation is able to accurately imitate the original model (Guidotti et al. Reference Guidotti, Monreale, Ruggieri, Turini, Glannotti and Pedreschi2018). A fidelity metric is calculated as an agreement metric between the prediction of the original model and the explanation, the latter in this case is the rule set. More concretely, when the predicted class by the model is positive and that by the explanation is positive, it is a true positive (TP), and when the latter is negative, it is a false negative (FN). Thus, the fidelity metrics can be calculated in the same manner as performance metrics using the equations shown in Section 3.3. RIPPER was excluded from this comparison for the same reasons as outlined in Section 5.2.3.

The average fidelity metrics (accuracy, F1-score, precision, and recall) are shown in Table 7. The overall trend is similar to the previous section on rule relevance, where RuleFit performs the best overall in terms of fidelity. The accuracy metrics for our method shows that the global explanations, in general, behaved similarly to the original model, although RuleFit was better in most of the datasets. The precision metrics show that, even when excluding the results for Decision Tree (which is not a tree-ensemble learning algorithm), our method could produce explanations that had high fidelity in terms of precision compared to RuleFit. The fidelity metrics may be improved further by including them in the ASP encodings, since they were not part of the selection criteria or optimization goals.

Table 7. Fidelity metrics, averaged over 5 folds. (Global explanations)

$^a$ Decision Tree + ASP $^b$ Random Forest + ASP $^c$ LightGBM + ASP

5.2.5 Changing optimization criteria

The definition of optimization objectives has a direct influence over the performance of the resulting rule sets, and the objectives need to be set in accordance with user requirements. The answer sets found by clingo with multiple optimization statements are optimal regarding the set of goals defined by the user. Instead of using accuracy, one may use other rule metrics as defined in Table 1 such as precision and/or recall. If there are priorities between optimization criteria, then one could use the priority notation (weight@priority) in clingo to define them. Optimal answer sets can be computed in this way, however, if enumeration of such optimal sets is important, then one could use the pareto or lexico preference definitions provided by asprin (Brewka et al. Reference Brewka, Delgrande, Romero and Schaub2015) to enumerate Pareto optimal answer sets. Instead of presenting a single optimal rule set to the user, this will allow the user to explore other optimal rule sets.

To investigate the effect of changing optimization objectives, we changed the ASP encoding from max. accuracy-coverage to max. precision-coverage (shown in Section 3.4) while keeping other parameters constant. The results are shown in Table 8. Note that it is the ratio of precision score shown in the table, as opposed to accuracy or F1-score in the earlier tables. Here, since we are optimizing for better precision, we expect the precision-coverage encoding to produce rule sets with better precision scores than the accuracy-coverage encoding. For the Decision Tree and Random Forest + ASP, the effect was not as pronounced as we expected, but we observed noticeable differences in datasets compas and credit german. For the LightGBM+ASP combination, we observed more consistent difference, except for the credit german dataset, the encoding produced intended results in most of the datasets in this experiment.

Table 8. Average ratio of rule-based classifier’s precision vs. original tree-ensembles, averaged over 5 folds. (Global explanations)

$^a$ Performance ratio of 1 means the rule set’s precision is identical to the original classifier. Numbers are shown in bold where the performance ratio was better than more than 0.01 compared to the other encoding. $^b$ acc.cov=accuracy and coverage encoding, see Section 4. $^c$ prec.cov=precision and coverage encoding, see Section 4.

5.2.6 Global explanation running time

The average running time of generating global explanations is reported in Table 9. The running time measures the rule extraction and rule set generation steps for our method, and measures the running time for RuleFit, but excludes the time taken for the model training (e.g., Random Forest) and hyperparameter optimization. Comparing the methods that share the same base model (RF + ASP and RuleFit, both based on Random Forest), we observe that our method is slower than RuleFit except when the datasets are relatively large (e.g., adult, census, compas, and credit taiwan), and in the latter cases our method can be much faster than RuleFit. Similar trend is observed for LightGBM, but here in some cases our method was faster than RuleFit (e.g., autism, heart, and voting).

Table 9. Average running time of generating global explanations, averaged over 5 folds. (Global explanations)

$^a$ DT = Decision Tree $^b$ RF = Random Forest. $^c$ LGBM = LightGBM.

5.3 Evaluating local explanations

The purpose of generating local explanations is to provide the user with an explanation for the model’s prediction for each predicted instance. Here, we use commonly used metrics local-precision and coverage as proxy measures for the quality of the explanation.Footnote ¹⁴ The local-precision compares the (black-box) model predictions of instances covered by the local explanation and the model prediction of the original instance used to induce the local explanation. The coverage is the ratio of instances in the validation set that are covered by the local explanation. These two metrics are in a trade-off relationship, where pursuing high coverage is likely to result in low precision explanation and vice versa. Furthermore, we also study the number of conditions in the explanation to measure the conciseness of the extracted rules. Additionally, we will also compare the running time to generate the local explanation.

The experiments were carried out similarly to the global explanation evaluation, except that: (1) we replaced RIPPER and RuleFit with Anchors, (2) instead of using the full validation set, we resampled the validation dataset to generate 100 instances in each cross-validation fold for each dataset to estimate the metrics, to complete the experiments in a reasonable amount of time, and (3) in the ASP encoding, we removed the rule selection criteria to avoid excluding rules that are relevant to the predicted instance. We were unable to complete Anchors experiment with the census dataset due to limited memory (64 GB) on our machine. For comparison, we computed direct and sufficient explanations with the PyXAIFootnote ¹⁵ library, which internally uses SAT and MaxSAT solvers to compute local explanations (Audemard et al. Reference Audemard, Bellart, Bounia, Koriche, Lagniez and Marquis2022b,a). Our method currently does not include the rule simplification feature. Therefore, to maintain a consistent comparison, we also deactivated the rule simplification feature in PyXAI. Furthermore, as of writing, PyXAI does not yet support LightGBM classifier, so only results for decision tree and random forests are included. For the running time comparison, we exclude all data preprocessing, training and tree processing and focus solely on the time taken to generate local explanations.

5.3.1 Number of conditions in rules

Similarly to the evaluation of global explanations (Section 5.2.2), we evaluate the number of conditions in local explanations in this section. The average number of conditions in rules are listed in Tables 10, 11, 12. For the Decision Tree (Table 10) and Random Forest (Table 11), the Anchors produced rules with smaller number of conditions on average overall compared to our method. As for the LightGBM, the Anchors produced rules with significantly larger number of conditions than our method, and often there was an order of magnitude of difference in the number of rules. It is possible that the precision guarantee of the Anchors required the algorithm to produce more specific rules, as also indicated by the long run time especially on the datasets that produced the longest rules (e.g., census, credit taiwan, and adult). This result shows that depending on the underlying learning algorithm, our method can produce shorter and more concise explanations compared to the Anchors.

Table 10. Decision tree local explanation metrics

$^a$ Anchors. $^b$ PyXAI, direct explanations. $^c$ PyXAI, sufficient explanations. $^d$ Excluded from the run time comparison since the SAT solver is not involved in computing direct explanations.

Table 11. Random forest local explanation metrics

$^a$ Anchors. $^b$ PyXAI, direct explanations. $^c$ PyXAI, sufficient explanations. $^d$ Excluded from the run time comparison since the SAT solver is not involved in computing direct explanations.

Table 12. LightGBM local explanation metrics

$^a$ Anchors.

It is interesting to note that, while PyXAI’s direct explanations performed almost identically to ours for Decision Tree, it produced much larger rules for Random Forest. Similarly, PyXAI’s sufficient explanations produced more conditions in the explanations compared to our method or Anchors. It is possible that rule simplification could help reduce the number of conditions in such cases.