2.1 Motivating example

Consider a Datalog points-to analysis in Figure 2. Here, we show an input program to analyze (Figure 2a), which is encoded as a set of tuples (Figure 2b) by an extractor, which maintains a mapping between tuples and source code Jordan et al. (Reference Jordan, Scholz and Subotić2016); Schäfer et al. (Reference Schäfer, Avgustinov and De moor2017); Vallée-Rai et al. (Reference Vallée-rai, Co, Gagnon, Hendren, Lam and Sundaresan2010). We have relations new, store, load, and assign capturing the semantics of the input program to analyze. These relations are also known as the Extensional Database (EDB), representing the analyzer’s input. The analyzer written in Datalog computes relations vpt (Variable Points To) and alias as the output, which is also known as the Intensional Database (IDB). For the points-to analysis, Figure 2c has four rules. A rule is of the form:

\begin{equation*}R_h(X_h) \ \texttt {:-}\ R_1(X_1), \ldots, R_k(X_k).\end{equation*}

Each $R(X)$ is an atom, with $R$ being a relation name and $X$ being a vector of variables and constants of appropriate arity. The predicate to the left of :- is the head and the sequence of predicates to the right is the body. A Datalog rule can be read from right to left: “for all rule instantiations, if every tuple in the body is derivable, then the corresponding tuple for the head is also derivable”.

For example, $r_2$ is $\mathtt {vpt}(\mathtt {Var}, \mathtt {Obj}) \ \texttt {:-}\ \mathtt {assign}(\mathtt {Var}, \mathtt {Var2}),\ \mathtt {vpt}(\mathtt {Var2}, \mathtt {Obj})$ , which can be interpreted as “if there is an assignment from Var to Var2, and if Var2 may point to Obj, then also Var may point to Obj”. In combination, the four rules represent a flow-insensitive but field-sensitive points-to analysis. The IDB relations vpt and alias represent the analysis result: variables may point to objects and pairs of variables that may be an alias with each other.

Suppose the input program in Figure 2a changes by adding a method to upgrade a user session to an admin session with the code:

The result of the points-to analysis can be incrementally updated by inserting the tuples assign(upgradedSession, userSession) and load(userSession, admin, session). After computing the incremental update, we would observe that alias(userSession, sec) is now contained in the output. However, we may wish to maintain that userSession should not alias with the secure session sec. Consequently, the incremental update has introduced a fault, which we wish to localize and initiate a rollback.

A fault localization provides a subset of the incremental update that is sufficient to reproduce the fault, while a rollback suggestion is a subset of the update which fixes the faults. In this particular situation, the fault localization and rollback suggestion are identical, containing only the insertion of the second tuple, load(userSession, admin, session). Notably, the other tuple in the update, assign(upgradedSession, userSession), is irrelevant for reproducing or fixing the fault and thus is not included in the fault localization/rollback.

In general, an incremental update may contain thousands of inserted and deleted tuples, and a set of faults may contain multiple tuples that are changed in the incremental update. Moreover, the fault tuples may have multiple alternative derivations, meaning that the localization and rollback results are different. In these situations, automatically localizing and rolling back the faults to find a small relevant subset of the incremental update is essential to provide a concise explanation of the faults to the user.

The scenario presented above is common during software development, where making changes to a program causes faults to appear. While our example concerns a points-to analysis computed for a source program, our fault localization and repair techniques are, in principle, applicable to any Datalog program.

2.2 Approach overview

Fig. 3.

Fault localization and repair system.

An overview of our approach is shown in Figure 3. The first portion of the system is the incremental Datalog evaluation. Here, the incremental evaluation takes an EDB and an incremental update containing tuples inserted or deleted from the EDB, denoted $\Delta$ EDB. The result of the incremental evaluation is the output IDB, along with the set of IDB insertions and deletions from the incremental update, denoted $\Delta$ IDB. The evaluation also enables incremental provenance, producing a proof tree for a given query tuple.

The second portion of the system is the fault localization/rollback repair. This process takes a set of faults provided by the user, which is a subset of $\Delta$ IDB where each tuple is either unwanted and inserted in $\Delta$ IDB or is desirable but deleted in $\Delta$ IDB. Then, the fault localization and rollback repair algorithms use the full $\Delta$ IDB and $\Delta$ EDB, along with incremental provenance, to produce a localization or rollback suggestion.

The main fault localization and rollback algorithms work in tandem to provide localizations or rollback suggestions to the user. The key idea of these algorithms is to compute proof trees for fault tuples using the provenance utility provided by the incremental Datalog engine. These proof trees directly provide localization for the faults. For fault rollback, the algorithms create an Integer Linear Programming (ILP) instance that encodes the proof trees, with the goal of disabling all proof trees to prevent the fault tuples from appearing.

The result is a localization or rollback suggestion, which is a subset of $\Delta$ EDB. For localization, the subset $S \subseteq \Delta \mathrm {EDB}$ is such that if we were to apply $S$ to $\mathrm {EDB}$ as the diff, the set of faults would be reproduced. For a rollback suggestion, the subset $S \subseteq \Delta \mathrm {EDB}$ is such that if we were to remove $S$ from $\Delta$ EDB, then the resulting diff would not produce the faults.

3.1 Background

3.1.1 Provenance

Provenance Caballero et al. (Reference Caballero, Riesco and Silva2017); Zhao et al. (Reference Zhao, Subotić and Scholz2020); Raghothaman et al. (Reference Raghothaman, Mendelson, Zhao, Naik and Scholz2019) provides machinery to explain the existence of a tuple. For example, the tuple vpt(userSession, L3) could be explained in our running example by the following proof tree:

This proof tree explains the derivation of vpt(userSession, L3) through input and intermediate tuples and which Datalog rules were involved in the derivation. Therefore, these proof trees provide a link between faulty tuples and the structure of the program itself. This forms the basis for our incremental Datalog debugging approach.

A two-phase approach for computing provenance was introduced in Zhao et al. (Reference Zhao, Subotić and Scholz2020). The key idea is to instrument the Datalog program to compute some provenance information alongside the actual tuples, and then to use this information to produce proof trees.

The two-phase process consists of:

1. 1. Instrumented Datalog evaluation: annotating each tuple with provenance annotations while computing the IDB. In particular, for each tuple $t$ , the system stores the height of the minimal height proof tree for $t$ . For an EDB tuple, the height is $0$ , meanwhile the height of an IDB tuple is computed by $h(t) = \max \{ h ({t_1}), \ldots, h ({t_k}) \} + 1$ if $t$ is derived by a rule instantiation $t \ \texttt {:-}\ t_1, \ldots, t_k$ .

2. 2. Provenance query answering: using the annotated IDB to answer provenance queries of the form of “explain vpt(userSession, L3)”. Internally, the system generates and solves constraints using the provenance annotations to construct the proof trees one level at a time.

For the running example, the tuple vpt(userSession, L3) gets a height annotation of $2$ during the instrumented evaluation phase. Then, during the query answering, the system generates a constraint saying “find the tuples matching the body of a rule for vpt with height less than $2$ ”.

3.1.2 Incremental evaluation

Incremental Datalog evaluation provides an efficient method of updating the results of a computation given some small changes to the input (where some tuples are inserted and/or deleted). Previous incremental evaluation approaches, such as Delete-Rederive (DRed) Gupta et al. (Reference Gupta, Mumick and Subrahmanian1993), had shortcomings concerning over-deletion and the re-derivation step required to resolve this. However, modern incremental evaluation approaches typically use a counting-based approach. For these strategies, each IDB tuple is associated with a count representing the number of different derivations for that tuple. When an EDB tuple $t$ is inserted or deleted, tuples depending on $t$ have their counts incremented or decremented respectively. If a tuple’s count reaches 0, then that tuple is removed from the IDB.

To handle recursion, techniques such as Differential Dataflow McSherry et al. (Reference Mcsherry, Murray, Isaacs and Isard2013) propose storing counters for each tuple per iteration of the recursion. Note that if rules are alternately increasing and decreasing, then they would be operating in different iterations. Therefore, each tuple is associated with an iteration number and a count. This increases the space overhead for keeping the auxiliary information but allows for precise recording of the incremental updates in a recursive setting. Variations of this idea, such as Elastic Incremental Evaluation Zhao et al. (Reference Zhao, Subotic, Raghothaman and Scholz2021), propose a trade-off of lower space overhead at the cost of requiring some recomputation to maintain precision of the derivation counts.

In our running example, the insertion of two lines in the source program result in the insertion of two EDB tuples for the pointer analysis: assign(upgradedSession, userSession) and load(userSession, admin, session).

Using a counting approach Zhao et al. (Reference Zhao, Subotic, Raghothaman and Scholz2021); McSherry et al. (Reference Mcsherry, Murray, Isaacs and Isard2013), we can apply incremental counting versions of the Datalog rules to compute an updated IDB. In this instance, we can apply rule $r_2$ as

The superscripts denote the changes in count as a result of the newly inserted EDB tuple assign(upgradedSession,userSession). These updated counts are then propagated in subsequent rules, until the IDB has reached fixpoint. Further details are described in Zhao et al. (Reference Zhao, Subotic, Raghothaman and Scholz2021).

3.2 Combining provenance and incremental evaluation

For fault localization and rollback, a novel provenance strategy is required that builds on incremental evaluation. Incremental provenance restricts the computations of the proof tree to the portions affected by the incremental update only. For example, Figure 4 shows an incremental proof tree for the inserted tuple alias(userSession,sec). The tuples labeled with (+) are inserted by an incremental update. Incremental provenance would only compute provenance information for these newly inserted tuples and would not explore the tuples in red already established in a previous epoch.

Fig. 4.

The proof tree for alias(userSession,sec). (+) denotes tuples that are inserted as a result of the incremental update, red denotes tuples that were not affected by the incremental update.

To formalize incremental provenance, we define $inc-prov$ as follows. Given an incremental update $\Delta E$ , $inc-prov(P, E, t, \Delta E)$ should consist of tuples that were updated due to the incremental update.

To compute provenance information efficiently in an incremental evaluation setting, we introduce a novel method combining the provenance annotations of Zhao et al. (Reference Zhao, Subotić and Scholz2020) with the incremental annotations of Zhao et al. (Reference Zhao, Subotic, Raghothaman and Scholz2021). Recall, from Section 3.1, that provenance annotations include the height of the minimal height proof tree for a tuple, computed by taking the maximum height of all tuples in its derivation. Also recall that incremental annotations include the iteration number in which a tuple is derived.

To combine these two, we can observe a correspondence between the iteration number and provenance annotations. A tuple is produced in some iteration if at least one of the body tuples was produced in the previous iteration. Therefore, the iteration number $I$ for a tuple produced in a fixpoint is equivalent to

\begin{align*} I(t) = \max \{ I(t_1), \ldots, I(t_k) \} + 1 \end{align*}

if $t$ is computed by rule instantiation $t \ \texttt {:-}\ t_1, \ldots, t_k$ . This definition of iteration number corresponds closely to the height annotation in provenance. Therefore, the iteration number is suitable for constructing proof trees similar to provenance annotations.

For fault localization and rollback, it is also important that the Datalog engine produces only provenance information that is relevant for faults that appear after an incremental update. Therefore, the provenance information produced by the Datalog engine should be restricted to tuples inserted or deleted by the incremental update. Thus, we adapt the user-driven proof tree exploration process in Zhao et al. (Reference Zhao, Subotić and Scholz2020) to use an automated procedure that enumerates exactly the portions of the proof tree that have been affected by the incremental update.

As a result, our approach for incremental provenance produces proof trees containing only tuples inserted or deleted due to an update. For fault localization and rollback, this property is crucial for minimizing the search space when computing localizations and rollback suggestions.

4.1 Preliminaries

We first define a fault to formalize the fault localization and rollback problems. For a Datalog program, a fault may manifest as either (1) an undesirable tuple that appears or (2) a desirable tuple that disappears. In other words, a fault is a tuple that does not match the intended output of a program.

Given an intended output for a program, a fault can be defined as follows:

We can formalize the situation where an incremental update for a Datalog program introduces a fault. Let $P$ be a Datalog program with intended output $I_{\checkmark } = (I_+, I_-)$ and let $E_1$ be an input EDB. Then, let $\Delta E_{1 \rightarrow 2}$ be an incremental update (or diff), such that the application operator $E_1 \uplus \Delta E_{1 \rightarrow 2}$ results in another input EDB, $E_2$ . Then, assume that $E_1$ is correct w.r.t $I_{\checkmark }$ , but $E_2$ is incorrect.

4.2 Fault localization

In the context of incremental Datalog, the fault localization problem provides a small subset of the incremental changes that allow the fault to be reproduced. On its own, fault localization forms an important part of the reproduction step of any fault investigation. Moreover, it is also fundamental for rollback repair of missing tuples or negations (see Section 4.4).

Consider the example in Figure 5. This diagram illustrates that a fault localization is a subset of the input changes $L \subseteq \Delta E$ such that when $L$ is used as the input changes in the incremental evaluation, the resulting update still produces the faults.

Fig. 5.

A fault localization is a subset of input changes such that the faults are still reproduced.

We begin by considering a basic version of the fault localization problem. In this basic version, we have a positive Datalog program (i.e., with no negation), and we localize a set of faults that are undesirable but appear (i.e., $P(E) \cap I_-$ ). The main idea of the fault localization algorithm is to compute a proof tree for each fault tuple. The tuples forming these proof trees are sufficient to localize the faults since these tuples allow the proof trees to be valid and, thus, the fault tuples to be reproduced.

Algorithm 1

Localize-Faults(P, E ₂, ΔE _{1 → 2}, F): Given a diff ΔE _{1 → 2} and a set of fault tuples F, returns $\delta E \subseteq \Delta E_{1\in2}$ such that $E_1 \uplus \delta E$ produces all t $\in$ F

The basic fault localization is presented in Alg. 1. For each fault tuple $t \in F$ , the algorithm computes one incremental proof tree $\texttt{inc-prov}(t, \Delta E_{1 \rightarrow 2})$ . These proof trees contain the set of tuples that were inserted due to the incremental update $\Delta E_{1 \rightarrow 2}$ and cause the existence of each fault tuple $t$ . Therefore, by returning the union $\cup _{t \in F} (\texttt{inc-prov}(t, \Delta E_{1 \rightarrow 2}) \cap \Delta E_{1 \rightarrow 2})$ , the algorithm produces a subset of $\Delta E_{1 \rightarrow 2}$ that reproduces the faults.

4.3 Rollback repair

A rollback repair is a subset of the input changes such that the remaining changes ‘fix’ the faults. Consider Figure 6, which shows that a rollback repair is a small subset of the input changes, such that the remainder of the changes no longer produce the faults when used as an incremental update.

Fig. 6.

An input debugging suggestion is a subset of input changes such that the remainder of the input changes no longer produce the faults.

The rollback repair algorithm produces a rollback suggestion. As with fault localization, our presentation begins with a basic version of the rollback problem, where we have only a positive Datalog program and wish to roll back a set of unwanted fault tuples. The basic rollback repair algorithm involves computing all non-cyclic proof trees for each fault tuple and ‘disabling’ each of those proof trees, as shown in Alg. 2. If all proof trees are invalid, the fault tuple will no longer be computed by the resulting EDB.

Algorithm 2

Rollback-Repair(P, E ₂, ΔE _{1 → 2}, F): Given a diff ΔE _{1 → 2} and a set of fault tuples F, return a subset $\delta$E ⊆ ΔE _{1 → 2} such that $E_1 \uplus(\Delta E_{1\in 2} \backslash \delta E)$ does not produce t _r

Alg. 2 computes a minimum subset of the diff $\Delta E_{1 \rightarrow 2}$ , which would prevent the production of each $t \in F$ when excluded from the diff. The key idea is to use integer linear programming (ILP) Schrijver (Reference Schrijver1998) as a vehicle to disable $\Delta$ EDB tuples so that the fault tuples vanish in the resulting IDB. We phrase the proof trees as a pseudo-Boolean formula Hooker (Reference Hooker1992) whose propositions represent the $\Delta$ EDB and IDB tuples in the proof trees. In the ILP, the faulty tuples are constrained to be false, and the $\Delta$ EDB tuples assuming the true value are to be maximized, that is we wish to eliminate the least number of tuples in $\Delta$ EDB for repair. The ILP created in Alg. 2 introduces a variable for each tuple (either IDB or $\Delta$ EDB) that appears in all incremental proof trees for the fault tuples. For the ILP model, we have three types of constraints:

1. 1. to encode each one-step proof tree,

2. 2. to enforce that fault tuples are false, and

3. 3. to ensure that variables are in the $0$ - $1$ domain.

The constraints encoding proof trees correspond to each one-step derivation which can be expressed as a Boolean constraint, where $t_1, \ldots, t_k$ and $t_h$ are Boolean variables:

\begin{align*} \frac {t_1\ \ \ldots \ \ t_k}{t_h}\ \equiv \,{t_1} \land \ldots \land {t_k} \implies {t_h} \end{align*}

Using propositional logic rules, this is equivalent to $\overline {t_1} \lor \ldots \lor \overline {t_k} \lor {t_h}$ . This formula is then transformed into a pseudo (linear) Boolean formula where $\varphi$ maps a Boolean function to the $0-1$ domain, and $x_t$ corresponds to the $0$ - $1$ variable of proposition $t$ in the ILP.

\begin{align*} \varphi \left ( \overline {t_1} \lor \ldots \lor \overline {t_k} \lor {t_h} \right ) &\equiv (1-x_{t_1}) + \ldots + (1-x_{t_k}) + t_h \gt 0 \\ &\equiv x_{t_1} + \ldots + x_{t_k} - x_{t_h} \leq k - 1 \end{align*}

The constraints assuming false values for fault tuples $t_f \in F$ are simple equalities, that is $x_{t_f} = 0$ . The objective function for the ILP is to maximize the number of inserted tuples that are kept, which is equivalent to minimizing the number of tuples in $\Delta E_{1 \rightarrow 2}$ that are disabled by the repair. In ILP form, this is expressed as maximizing $\sum _{t \in \Delta E_{1 \rightarrow 2}} t$ .

\begin{equation*} \begin{array}{lll} \mbox {max.} & \sum _{t \in \Delta E_{1 \rightarrow 2}} x_{t} \\ \mbox {s.t.} & x_{t_1} + \ldots x_{t_k} - x_{t_h} \leq k -1 & \left (\forall \ \frac {t_1\ \ \ldots \ \ t_k}{t_h} \in T_i \right ) \\ & x_{t_f} = 0 & \left ( \forall t_f \in F \right )\\ & x_t \in \{0, 1\} & \left ( \forall \text {tuples } t \right ) \end{array} \end{equation*}

The solution of the ILP permits us to determine the EDB tuples for repair, that is $\{t \in \Delta E_{1 \rightarrow 2} \mid x_t = 0\}$ This is a minimal set of inserted tuples that must be removed from $\Delta E_{1 \rightarrow 2}$ so that the fault tuples disappear.

This ILP formulation encodes the problem of disabling all proof trees for all fault tuples while maximizing the number of inserted tuples kept in the result. If there are multiple fault tuples, the algorithm computes proof trees for each fault tuple and combines all proof trees in the ILP encoding. The result is a set of tuples that is minimal but sufficient to prevent the fault tuples from being produced.

4.4 Extensions

4.4.2 Stratified negation

Stratified negation is a common extension for Datalog. With stratified negation, atoms in the body of a Datalog rule may appear negated, for example

\begin{equation*} R_h(X_h) \ \texttt {:-}\ R_1(X_1), \ldots, !R_k(X_k), \ldots, R_n(X_n). \end{equation*}

The negated atoms are denoted with $!$ , and any variables contained in negated atoms must also exist in some positive atom in the body of the rule (a property called groundedness or safety). Semantically, negations are satisfied if the instantiated tuple does not exist in the corresponding relation. The ‘stratified’ in ‘stratified negation’ refers to the property that no cyclic negations can exist. For example, the rule $A(x) \ \texttt {:-}\ B(x, y), !A(y)$ causes a dependency cycle where $A$ depends on the negation of $A$ and thus is not allowed under stratified negation.

Consider the problem of localizing the appearance of an unwanted tuple $t$ . If the Datalog program contains stratified negation, then the appearance of $t$ can be caused by two possible situations. Either (1) there is a positive tuple in the proof tree of $t$ that appears, or (2) there is a negated tuple in the proof tree of $t$ that disappears. The first case is the standard case, but in the second case, if a negated tuple disappears, then its disappearance can be localized or rolled back by computing the dual problem as in the missing tuple strategy presented above. We may encounter further negated tuples in executing the dual version of the problem. For example, consider the set of Datalog rules $A(x) \ \texttt {:-}\ B(x), !C(x)$ and $C(x) \ \texttt {:-}\ D(x), !E(x)$ . If we wish to localize an appearing (unwanted) tuple A(x), we may encounter a disappearing tuple C(x). Then, executing the dual problem, we may encounter an appearing tuple E(x). We can generally continue flipping between the dual problems to solve the localization or repair problem. This process is guaranteed to terminate due to the stratification of negations. Each time the algorithm encounters a negated tuple, it must appear in an earlier stratum than the previous negation. Therefore, eventually, the negations will reach the input EDB, and the process terminates.

4.4.3 Changes in datalog rules

The algorithms are presented above in the context of localizing or debugging a change to the input tuples. However, with a simple transformation, the same algorithms can also be applied to changes in Datalog rules. For each Datalog rule, introduce a predicate Rule(i), where i is a unique number per rule. Then, the unary relation Rule can be considered as EDB, and thus the set of rules can be changed by providing a diff containing insertions or deletions into the Rule relation. For example, a transformed set of rules may be:

Then, by including or excluding 1 or 2 in the EDB relation Rule, the underlying Datalog rules can be ‘switched on or off,’ and a change to the Datalog program can be expressed as a diff in the Rule relation.

4.5 Full algorithm

Algorithm 3

Full-Rollback-Repair(P, E ₁, ΔE _{1 → 2}, (I ₊,I ₋)): Given a diff ΔE _{1 → 2} and an intended output (I ₊,I _,), compute a subset $\delta$E ⊆ ΔE _{1 → 2} such that ΔE _{1 → 2} \ $\delta$E satisfies the intended output

The full rollback repair algorithm presented in Alg. 3 incorporates the basic version of the problem and all of the extensions presented above. The result of the algorithm is a rollback suggestion, which fixes all faults. Alg. 3 begins by initializing the EDB after applying the diff (line 1) and separate sets of unwanted faults (lines 2) and missing faults (3). The set of candidate tuples forming the repair is initialized to be empty (line 4).

The main part of the algorithm is a worklist loop (lines 5 to 15). In this loop, the algorithm first processes all unwanted but appearing faults ( $F^+$ , line 6) by computing the repair of $F^+$ . The result is a subset of tuples in the diff such that the faults $F^+$ no longer appear when the subset is excluded from the diff. In the provenance system, negations are treated as EDB tuples, and thus the resulting repair may contain negated tuples. These negated tuples are added to $F^-$ (line 7) since a tuple appearing in $F^+$ may be caused by a negated tuple disappearing. The algorithm then repairs the tuples in $F^-$ by computing the dual problem, that is localizing $F^-$ with respect to $\Delta E_{2 \rightarrow 1}$ . Again, this process may result in negated tuples, which are added to $F^+$ , and the loop begins again. This worklist loop must terminate, due to the semantics of stratified negation, as discussed above. At the end of the worklist loop, $L$ contains a candidate repair.

While Alg. 3 presents a full algorithm for rollback, the fault localization problem can be solved similarly. Since rollback and localization are dual problems, the full fault localization algorithm swaps Rollback-Repair in line 6 and Localize-Faults in line 11.

4.5.1 Example

We demonstrate how our algorithms work by using our running example. Recall that we introduce an incremental update consisting of inserting two tuples:

assign(upgradedSession, userSession) and load(userSession, admin, session). As a result, the system computes the unwanted fault tuple alias(userSession, sec). To rollback the appearance of the fault tuple, the algorithms start by computing its provenance, as shown in Figure 4. The algorithm then creates a set of ILP constraints, where each tuple (with shortened variables) represents an ILP variable:

\begin{align*} &\text {maximize} \sum \mathtt {load(u,a,s)} \text { such that} \\ &\mathtt {load(u,a,s)} - \mathtt {vpt(u,L2)} \leq 0,\\ &\mathtt {vpt(u,L2)} - \mathtt {alias(u,s)} \leq 0,\\ &\mathtt {alias(u,s)} = 0 \end{align*}

For this simple ILP, the result indicates that the insertion of load(userSession, admin, session) should be rolled back to fix the fault.

4.6 Correctness and Optimality

In this section, we discuss the correctness and optimality of our algorithms. Consider the problem set up with a Datalog program $P$ , an EDB $E_1$ , an incremental update diff $\Delta E_{1 \rightarrow 2}$ , and a set of fault tuples $F$ . For the fault localization algorithm, correctness implies that the result reproduces the faults, that is that $F \subseteq (E_1 \uplus \delta E)$ . Meanwhile, the correctness of a rollback repair implies that the result prevents faults from appearing, that is that $(E_1 \uplus (\Delta E_{1 \rightarrow 2} \setminus \delta _{\times } E)) \cap F = \emptyset$

Optimality is measured by how minimal the solution is. For both fault localization and rollback repair, a solution is optimal if there is no smaller subset of the solution which correctly solves the problem.

6.1 Experimental setupFootnote ²

Our main point of comparison in our experimental evaluation is the delta debugging approach, such as that used in the ProSynth Datalog synthesis framework Raghothaman et al. (Reference Raghothaman, Mendelson, Zhao, Naik and Scholz2019). We adapted the implementation of delta debugging used in ProSynth to support input tuple updates. Like our fault repair implementation, the delta debugging algorithm was implemented in Python; however, it calls out to the standard Soufflé engine since that provides a lower overhead than the incremental or provenance versions.

For our benchmarks, we use the Doop program analysis framework Bravenboer and Smaragdakis (Reference Bravenboer and Smaragdakis2009) with the DaCapo set of Java benchmarks Blackburn et al. (Reference Blackburn, Garner, Hoffman, Khan, Mckinley, Bentzur, Diwan, Feinberg, Frampton, Guyer, Hirzel, Hosking, Jump, Lee, Moss, Phansalkar, Stefanović, Vandrunen, Von dincklage and Wiedermann2006). The analysis contains approx. 300 relations, 850 rules, and generates approx. 25 million tuples from an input size of 4–9 million tuples per DaCapo benchmark. For each of the DaCapo benchmarks, we selected an incremental update containing 50 tuples to insert and 50 tuples to delete, which is representative of the size of a typical commit in the underlying source code. From the resulting IDB changes, we selected four different arbitrary fault sets for each benchmark, which may represent an analysis error.

6.1.1 Performance

Table 1.

Repair size and runtime of our technique compared to delta debugging

The results of our experiments are shown in Table 1. Our fault repair technique performs much better overall compared to the delta debugging technique. We observe a geometric mean speedup of over 26.9 $\times$ Footnote ³ compared to delta debugging. For delta debugging, the main cause of performance slowdown is that it is a black-box search technique, and it requires multiple iterations of invoking Soufflé (between 6 and 19 invocations for the presented benchmarks) to direct the search. This also means that any intermediate results generated in a previous Soufflé run are discarded since no state is kept to allow the reuse of results. Each invocation of Soufflé takes between 30–50 s, depending on the benchmark and EDB. Thus, the overall runtime for delta debugging is in the hundreds of seconds at a minimum. Indeed, we observe that delta debugging takes between 370 and 6135 s on our benchmarks, with one instance timing out after two hours (7200 s).

On the other hand, our rollback repair technique calls for provenance information from an already initialized instance of incremental Soufflé. This incrementality allows our technique to reuse the already computed IDB for each provenance query. For eight of the benchmarks, the faults only contained missing tuples. Therefore, only the Localize-Faults method was called, which only computes one proof tree for each fault tuple and does not require any ILP solving. The remainder of the benchmarks called the Rollback-Repair method, where the main bottleneck is for constructing and solving the ILP instance. For three of the benchmarks, bloat-1, bloat-4, and lusearch-2, the runtime was slower than delta debugging. This result is due to the fault tuples in these benchmarks having many different proof trees, which took longer to compute. In addition, this multitude of proof trees causes a larger ILP instance to be constructed, which took longer to solve.

7.1 Logic programming input repair

A plethora of logic programming paradigms exist that can express diagnosis and repair by EDB regeneration Kakas et al. (Reference Kakas, Kowalski and Toni1993); Fan et al. (Reference Fan, Geerts and Jia2008a); Gelfond and Lifschitz (Reference Gelfond and Lifschitz1988); El-Hassany et al. (Reference El-hassany, Tsankov, Vanbever, Vechev, Majumdar and Kunčak2017); Liu et al. (Reference Liu, Mechtaev, Subotic, Roychoudhury, Chandra, Blincoe and Tonella2023). Unlike these logic programming paradigms, our technique is designed to be embedded in high-performance modern Datalog engines. Moreover, our approach can previous computations (proof trees and incremental updates) to localize and repair only needed tuples. This bounds the set of repair candidates and results in apparent speedups. Other approaches, such as the ABC Repair System Li et al. (Reference Li, Bundy and Smaill2018), use a combination of provenance-like structures and user-guided search to localize and repair faults. However, that approach is targeted at the level of the Datalog specification and does not always produce effective repairs. Techniques such as delta debugging have recently been used to perform state-of-the-art synthesis of Datalog programs efficiently Raghothaman et al. (Reference Raghothaman, Mendelson, Zhao, Naik and Scholz2019). Our delta debugging implementation adapts this method, given it produces very competitive synthesis performance and can be easily re-targeted to diagnose and repair inputs.

7.2 Database repair

Repairing inconsistent databases with respect to integrity constraints has been extensively investigated in the database community Fan (Reference Fan2009); Bravo and Bertossi (Reference Bravo, Bertossi, Lutfiyya, Singer and Stewart2004); Arenas et al. (Reference Arenas, Bertossi and Chomicki2003); Fan et al. (Reference Fan, Geerts and Jia2008). Unlike our approach, integrity constraints are much less expressive than Datalog; in particular, they do not allow fixpoints in their logic. The technique in Fan et al. (Reference Fan, Geerts and Jia2008) shares another similarity in that it also presents repair for incremental SQL evaluation. However, this is limited to relational algebra, that is SQL and Constrained Functional Dependencies (CFDs) that are less expressive than Datalog. A more related variant of database repair is consistent query answering (CQA) Bravo and Bertossi (Reference Bravo, Bertossi, Lutfiyya, Singer and Stewart2004); Arenas et al. (Reference Arenas, Bertossi and Chomicki2003). These techniques repair query answers given a database, integrity constraints and an SQL query. Similarly, these approaches do not support recursive queries, as can be expressed by Datalog rules.

7.3 Program slicing

Program slicing Weiser (Reference Weiser1984); Binkley and Gallagher (Reference Binkley and Gallagher1996); Ezekiel et al. (Reference Ezekiel, Lukas, Marcel and Zeller2021); Harman and Hierons (Reference Harman and Hierons2001) encompasses several techniques that aim to compute portions (or slices) of a program that contribute to a particular output result. For fault localization and debugging, program slicing can be used to localize slices of programs that lead to a fault or error. The two main approaches are static program slicing, which operates on a static control flow graph, and dynamic program slicing, which considers the values of variables or execution flow of a particular execution. As highlighted by Cheney (Reference Cheney2007), data provenance is closely related to slicing. Therefore, our technique can be seen as a form of static slicing of the Datalog EDB with an additional rollback repair stage.

7.4 Database rollback

Database transaction rollback and partial rollback are well established Mohan et al. (Reference Mohan, Haderle, Lindsay, Pirahesh and Schwarz1992); Coburn et al. (Reference Coburn, Bunker, Schwarz, Gupta and Swanson2013) and supported in many DBMS’s Oracle rollback (2023); Accelerated database recovery (2023). These techniques often perform rollback for a transaction in the context of data recovery, by logging the effects of each action in the transaction. Techniques such as Antonopoulos et al. (Reference Antonopoulos, Byrne, Chen, Diaconu, Kodandaramaih, Kodavalla, Purnananda, Radu, Ravella and Venkataramanappa2019) improve rollback time by using versioning information. These techniques are limited to SQL transactions while our technique targets recursive datalog queries in an incremental update setting. For the static analysis use case, to the best of our knowledge, we are the first to provide an automated partial commit rollback mechanism based on the analysis output. Nevertheless, it is interesting future work to investigate if our technique can assist in making more efficient partial rollbacks in a DBMS setting.

7.5 Automated commit rollback

There is not a lot of work in the literature on automatically detecting and partially rolling back buggy commits, despite several studies Shimagaki et al. (Reference ShimagakI, Kamei, Mcintosh, Pursehouse and Ubayashi2016); Yan et al. (Reference Yan, Xia, Lo, Hassan and Li2019) highlighting the benefits of identifying such commits and rolling them back as soon as possible. The closest works to ours are techniques Yan et al. (Reference Yan, Xia, Lo, Hassan and Li2019); Rosen et al. (Reference Rosen, Grawi and Shihab2015); Mockus and Weiss (Reference Mockus and Weiss2000); Kim et al. (Reference Kim, Whitehead and Zhang2008) that seek to identify through statistical models commits that are most likely to be reverted. In contrast, our technique works with a static analyzer that detects bugs in code, and provides users with the option to partially revert the commit so the bug is eliminated.