3.1 Spectrum-based fault localization

Fig. 2.

On the left of this figure we show a Prolog program, while on the right we show an execution trace for the query .

SBFL is a widely known approach to finding faults in imperative programming languages (Wong et al. Reference Wong, Gao, Li, Abreu and Wotawa2016). Spectrum-based tools collect program execution traces (called program spectra) from passing and failing test cases. By comparing which program elements (e.g., statements) are executed during successful runs versus failed runs, SBFL assigns a suspiciousness score to each program element.

While spectrum-based methods are not directly applicable to declarative programming languages, due to the lack of an explicit control flow, we can exploit the mixed execution model of Prolog. Although Prolog programs have a declarative logical meaning, they are executed via a procedural mechanism based on unification, search, and backtracking, which can produce an observable trace.

Some Prolog interpreters, such as SWI-Prolog, have native support for extracting execution traces. These traces usually contain four types of information: call when a goal is started, exit when a goal is proved, redo when backtracking happens, and fail when a goal fails. In Figure 2, we show an example of a Prolog program, the instrumented version of that program, and an execution trace for the query . In ProDebug, a program element corresponds to a Prolog clause (i.e., a rule or fact). Suspiciousness scores are thus assigned at the clause level, and spectrum information is extracted directly from the call and redo events of the trace.

Spectrum-based methods collect information from the execution of several tests. For each program element, $s$ , four pieces of information are collected: number of tests that passed and $s$ was executed ( $e_p$ ), number of tests that passed and $s$ was not executed ( $n_p$ ), number of tests that failed and $s$ was executed ( $e_f$ ), and number of tests that failed and $s$ was not executed ( $n_f$ ). The intuition behind the SBFL method is that program elements that were disproportionately executed more often in failed test cases are more likely to be buggy. Crucially, the suspiciousness formulas compare relative execution frequencies: a clause executed equally often across passing and failing tests receives a low score regardless of the absolute counts. In our setting, this is further supported by the use of test suites that include both positive tests (queries that should succeed) and negative tests (queries that should fail), ensuring that correct clauses are exercised in passing tests and providing a meaningful baseline for comparison. Several formulas exist for turning these metrics into a suspiciousness score for each element. ProDebug implements some of the most commonly used ones (Chatterjee et al. Reference Chatterjee, Campos, Abreu and Roy2023): Tarantula, Jaccard, Ochiai, Barinel, Kulczynski, Op2 and DStar.

3.2 Mutation-based fault localization

Mutation-based Fault Localization (MBFL) is a fault localization approach that leverages the principles of mutation testing (Moon et al. Reference Moon, Kim, Kim and Yoo2014). Mutation-based tools generate a set of mutants by applying small syntactic changes to the program under test. These mutants are then executed against both passing and failing test cases to observe how the introduced changes affect program behavior. By analyzing which mutants are “killed” (detected by a test, meaning their behavior differs from the original program) and which survive, MBFL computes a suspiciousness score for each clause. Clauses whose mutants are disproportionately killed by failing test cases are considered more likely to contain the fault (Moon et al. Reference Moon, Kim, Kim and Yoo2014).

Fig. 3.

Example of a program composed of two clauses and possible mutants generated by ProDebug for each of those clauses.

A central point of MBFL is generating the program mutants. ProDebug accomplishes this through a logic-based mutation enumeration engine using Satisfiability Modulo Theories (SMT) (Barrett et al. Reference Barrett, Sebastiani, Seshia and Tinelli2021). This mutation enumeration engine encodes Prolog Prolog Abstract Syntax Trees (ASTs) as logic formulas and then relaxes parts of those formulas in order to enumerate mutations (a detailed technical description is provided in Appendix B). While our approach is based on previous techniques for Answer Set Programming (ASP) bug repair (Brancas et al. Reference Brancas, Manquinho and Martins2025), adapting this method to Prolog requires addressing several key differences. Unlike ASP, where the order of rules and literals is irrelevant, Prolog’s execution is sensitive to the order of clauses and body terms. Furthermore, the two languages differ significantly in operator and negation semantics (Negation as Failure in Prolog vs. Stable Model semantics in ASP). ProDebug also accounts for Prolog-specific features such as support for higher-order predicates, which are not present in ASP. Figure 3 shows some mutants generated by ProDebug for the two clauses of the introduction example program. Like for SBFL, several suspiciousness formulas exist for MBFL. ProDebug implements the two most commonly used ones: MUSE (Moon et al. Reference Moon, Kim, Kim and Yoo2014) and Metallaxis (Papadakis and Traon Reference Papadakis and Traon2015).

3.3 Large language model-based fault localization

The third fault localization method included in ProDebug is LLMFL. This approach takes advantage of LLMs trained on large amounts of data and especially code. Due to the specific deployment requirements of ProDebug (real-time usage in classroom with possibly few resources), we focus on fine-tuned open-access small LLMs ( $\sim \! 4$ billion parameters) in opposition to using closed-access models or very large instruction-tuned models. Our prompt format, shown in Appendix A, includes a description of the program’s desired functionality, a reference implementation provided by the faculty, and the faulty student submission. The output format consists of a sequence of buggy clauses identified in the student’s code.

To fine-tune our fault localization models, we used Low-Rank Adaptation (LoRA) (Hu et al. Reference Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang and Chen2022) and Group Relative Policy Optimization (GRPO) (Shao et al. Reference Shao, Wang, Zhu, Xu, Song, Zhang, Li, Wu and Guo2024). LoRA is a technique that uses rank decomposition matrices to greatly decrease the number of trainable parameters of a model. This technique has two main advantages: it reduces GPU VRAM requirements during training and it serves as a model regularizer, reducing the risk of overfitting. GRPO is an online reinforcement learning method for training neural networks. In GRPO, the model generates multiple candidate completions for each training example. A reward is then assigned to each completion using a user-defined reward function. Instead of optimizing each completion in isolation, GRPO compares them against one another: the relative advantages between completions are used to guide updates to the model’s policy, encouraging it to prefer higher-reward outputs. For more details on GRPO, we refer to the original paper by Shao et al. (Reference Shao, Wang, Zhu, Xu, Song, Zhang, Li, Wu and Guo2024).

4.1 Mutation-based repair

Fig. 4.

A buggy Prolog rule, its AST representation, and the SMT encoding for that tree.

ProDebug’s mutation-based repair is based on the MBFL mutation engine introduced in Subsection 3.2. Figure 4 shows a concrete example of how ProDebug enumerates program mutations. First, the Prolog program (or program section) is parsed and transformed into an AST (shown on the left side of Figure 4). Next, the AST is used to create a logic formula that represents the program (shown on the right side of Figure 4). By relaxing some of the constraints in the formula and using an SMT solver, we can generate new versions of the program with small differences (i.e., mutations). For instance, relaxing the constraint would allow the solver to enumerate the terms or . ProDebug enumerates candidate repairs in increasing order of complexity: first one mutation, then two mutations, and so on.

Using an SMT-based approach to generate mutations allows us to capture part of Prolog’s semantics within the logical encoding. This enables ProDebug to constrain the search space by filtering out equivalent or semantically invalid mutations during enumeration. However, this expressiveness comes at the cost of increased computational overhead, as SMT-based enumeration typically incurs higher runtime compared to purely syntactic mutation generation techniques.

Before creating the SMT formula, the AST can be artificially expanded in order to support more diverse mutations. For example, by adding two empty nodes as children of $n_9$ , we can enumerate the term , which would be impossible otherwise. ProDebug automatically completes the AST by adding empty nodes so that all branches have the same depth and branching factor. Additionally, when the mutation engine is used for program repair, ProDebug enables some pruning constraints that help reduce the amount of equivalent and/or incorrect mutations. Appendix B provides a complete description of the encoding, including the relaxation mechanism, AST completion, and pruning constraints.

4.2 Large language model-based repair

Our LLM-based repair approach uses a prompt that contains a short description of the student’s assignment, a reference implementation created by the faculty, the student’s incorrect submission and the list of clauses reported by the fault localization module (see Appendix A). As in the LLM-based fault localization approach, we use small open-access LLMs to support classroom deployment.

For the repair fine-tuning, we used a combination of Supervised Fine-tuning (SFT) and GRPO. As introduced in Subsection 3.3, GRPO creates several completions for each prompt and learns based on the relative rewards attributed to each of those completions. However, if the initial model performs the desired task very poorly, it can happen for all the completions to receive very similar rewards (and even all be 0 in extreme cases). In such cases, the model may learn very slowly or fail to learn altogether. To address this problem, we performed a short run of classical SFT and then continue training with GRPO from there. Like for the LLM fault localization approach, we used LoRA to decrease VRAM requirements during training and to serve as a regularizer.

5.1 Methodology

5.2 Implementation

ProDebug is implemented in Python and uses the SWI-Prolog dialect and interpreter through the Machine Query Interface. Parsing of Prolog programs is accomplished through a custom-built grammar. ProDebug uses a modified version of the Trinity framework (Martins et al. Reference Martins, Chen, Chen, Feng and Dillig2019) for the enumeration of mutations for fault localization, repair and creation of synthetic instances for fine-tuning. Results were obtained within a 10-minute time limit (wall clock time) and with 60 GB of RAM per instance, which was strictly enforced through runsolver (Roussel Reference Roussel2011) and runhelper.Footnote ² We use a time limit of 10 minutes in this evaluation because we consider it the upper limit of time a student would accept to wait for automated feedback. Results were obtained on an Intel Xeon Silver 4210R processor. We employed the Qwen 3 family of models (Team 2025) for fault localization, and the Qwen 2.5 Coder family (Hui et al. Reference Hui, Yang, Cui, Yang, Liu, Zhang, Liu, Zhang, Yu, Dang, Yang, Men, Huang, Ren, Ren, Zhou and Lin2024) for program repair. These model families were selected empirically based on their performance in preliminary experiments for each respective task. LLMs were fine-tuned on four Nvidia RTX A4000 GPUs, and inference was performed on a single RTX A4000. ProDebug’s source code, data, and logs are publicly available.Footnote ³

Table 1.

Different evaluation metrics for each fault localization method split among the practice exercises and the project submissions. Acc@ $k$ : percentage of faults ranked within the top $k$ positions (higher is better); Expense: MinRank divided by the total number of clauses, i.e., the fraction of the program inspected before the first fault is found (lower is better)

Table 1 Long description

The table compares different fault localization methods across practice exercises and project submissions. It has two main sections: Practice exercises (n = 229) and Project (n = 1270). Each section lists methods: SBFL, MBFL, LLMFL, ProFL SBFL, and ProFL MBFL. The columns include MinRank, Acc@1, Acc@3, Acc@5, Acc@10, Expense, and Timeouts. Row 1: Practice exercises (n = 229), SBFL, 1.23, 83.57, 97.73, 48.29, 4.4%. Row 2: Practice exercises (n = 229), MBFL, 1.42, 70.95, 96.19, 53.71, 5.7%. Row 3: Practice exercises (n = 229), LLMFL, 1.23, 84.75, 97.01, 47.53, 0.0%. Row 4: Practice exercises (n = 229), ProFL SBFL, 1.68, 58.54, 93.01, 62.67, 7.9%. Row 5: Practice exercises (n = 229), ProFL MBFL, 1.76, 50.24, 91.63, 67.20, 7.9%. Row 6: Project (n = 1270), SBFL, 3.33, 56.82, 63.22, 71.35, 82.74, 10.86, 9.1%. Row 7: Project (n = 1270), MBFL, 3.76, 45.08, 57.58, 70.05, 85.49, 16.05, 59.9%. Row 8: Project (n = 1270), LLMFL, 6.32, 24.11, 41.06, 55.77, 74.85, 19.11, 0.0%. Row 9: Project (n = 1270), ProFL SBFL, 3.28, 48.67, 60.32, 70.91, 84.21, 28.42, 90.6%. Row 10: Project (n = 1270), ProFL MBFL, 4.80, 34.65, 42.74, 50.54, 78.50, 43.96, 91.5%.

5.3 How effective are the different fault localization methods?

We evaluated the three fault localization methods proposed in Section Section 3 and present the results in Table 1 under the labels SBFL, MBFL and LLMFL. Of these methods, the one with the best overall performance when considering both types of instances is SBFL, achieving 83.6% Accuracy@1 in the Practice Exercises and 56.8% Accuracy@1 in the Project submissions. This technique also has a fairly low percentage of timeoutsFootnote ⁴ ( $8.4\%$ ), meaning it can provide accurate answers in most instances. The second best performing technique is LLMFL, with a very similar performance in the practice exercises. However, this approach struggles with the longer context of Project submissions, only achieving an Accuracy@1 of 24.1%. The LLMFL approach also has the big advantage of not having any timeouts. Finally, the worst performing approach is MBFL, which is less accurate across the board. Furthermore, this approach has the drawback of having to generate and test large amounts of mutants, which scale linearly with the number of clauses in the program. This is reflected in the much larger percentage of timeouts in the MBFL approach for the project (59.9%) versus for the practice exercises (5.7%).

Table 2 shows summarized results for the SBFL and MBFL approaches using the different spectrum formulas implemented in ProDebug. The best performing formula for SBFL is Ochiai, with Jaccard as a close second. For MBFL, the Metallaxis formula is slightly better than MUSE. Based on these results, ProDebug utilizes Ochiai and Metallaxis as its default formulas.

Table 2.

Fault localization performance across SBFL and MBFL formulas

Table 2 Long description

The table compares fault localization performance across different SBFL and MBFL formulas. It has two main sections: SBFL formula and MBFL formula, each with their respective formulas and accuracy values. The SBFL section includes Tarantula, Ochiai, Op2, Barinel, Jaccard, DStar, and Kulczynski with accuracy values of 55.75 percent, 63.70 percent, 60.96 percent, 55.75 percent, 63.57 percent, 50.31 percent, and 50.57 percent respectively. The MBFL section includes Metallaxis and MUSE with accuracy values of 52.87 percent and 50.29 percent respectively.

5.4 How do proDebug’s SBFL and MBFL compare with prior tools?

Table 1 also includes the results of the SBFL and MBFL approaches implemented by a previous Prolog fault localization tool, ProFL (Thompson and Sullivan Reference Thompson and Sullivan2020). Although both ProFL and ProDebug implement SBFL and MBFL methods, their implementations differ substantially. We highlight three key distinctions: (1) ProDebug communicates with Prolog via the Prolog Machine Query Interface, rather than launching a new Prolog process for each query, (2) ProDebug supports a broader range of program mutations through its SMT-based mutation engine, and (3) ProDebug measures program coverage directly using Prolog’s tracing capabilities, whereas ProFL estimates clause coverage indirectly through program mutations. These differences address one of ProFL’s major limitations: its need to enumerate mutations even for the SBFL approach, combined with launching a fresh process for each mutant, resulting in substantial execution overhead. Consequently, ProFL exhibits a high rate of timeouts across both approaches – particularly in the project submissions, where more than 90% of instances exceed the 10-minute timeout used in our evaluation.

For the instances that finish within the time limit, ProDebug’s SBFL approach outperforms ProFL’s SBFL on the practice exercises (83.6% vs. 58.5% Acc@1), while both tools show comparable performance on the project submissions (56.8% vs. 48.7% Acc@1, with ProFL achieving a slightly better MinRank of 3.28 vs. 3.33). However, ProFL’s MBFL approach performs consistently worse than ProDebug’s MBFL across all instance types and metrics, likely due to its more limited set of generated mutants. Notably, ProFL’s MBFL achieves only 50.2% Acc@1 on practice exercises compared to ProDebug’s 70.9%, and 34.7% on project submissions compared to ProDebug’s 45.1%.

Table 3.

Evaluation results for ProDebug’s program repair approaches

Table 3 Long description

The table compares evaluation results for ProDebug’s program repair approaches across practice exercises (n = 229) and projects (n = 1270). It has two main sections: Simulated perfect fault localization and SBFL fault localization. Each section includes three rows: Mutation, LLM w/ 100 completions (default), and LLM w/ 30 completions. The columns represent the percentage results for practice exercises and projects. Simulated perfect fault localization: Mutation: 21.0 percent, 1.7 percent; LLM w/ 100 completions (default): 81.7 percent, 34.3 percent; LLM w/ 30 completions: 79.0 percent, 26.1 percent. SBFL fault localization: Mutation: 20.2 percent, 0.5 percent; LLM w/ 100 completions (default): 73.8 percent, 30.9 percent; LLM w/ 30 completions: 69.0 percent, 15.0 percent.

5.5 How effective is the program repair?

To evaluate the effectiveness of our program repair methods, we tested them under two conditions: using a simulated perfect fault localization and using the best-performing fault localization technique identified in Subsection 5.3. This design allows us to assess both the idealized repair performance, as well as to quantify the extent to which repair success is constrained by inaccuracies in fault localization. Table 3 summarizes the outcomes for the two repair strategies, mutation-based and LLM-based, across these contexts.

Under simulated perfect fault localization, the mutation-based approach has a repair rate of 21.0% on practice exercises and 1.7% on project submissions. Although relatively low, these results are consistent with prior work on mutation-based repair for short declarative programs which achieved a 19% repair rate (Brancas et al. Reference Brancas, Manquinho and Martins2025). In contrast, the LLM-based repair method performs substantially better, reaching 81.7% for practice exercises and 34.3% for project submissions. As described in Subsubsection 5.1.4, the LLM repair method samples multiple completions and selects the best candidate among them. Depending on deployment constraints, the number of completions can be reduced to improve runtime at a modest cost to repair accuracy. For example, reducing from 100 to 30 completions decreases average repair time from 106 s to 77 s, while lowering repair success by only 2.7 percentage points on exercises and 8.2 percentage points on projects.

When using the SBFL, repair performance decreases slightly. Mutation-based repair drops from 21.0% to 20.2% on exercises and from 1.7% to 0.5% on projects. The LLM-based method also declines, from 81.7% to 73.8% on exercises and from 34.3% to 30.9% on projects. The reduction is more pronounced for the configuration using only 30 completions, especially on the project dataset, suggesting that diversity in completions helps compensate for fault localization errors.

5.6 What is the impact of fine-tuning?

In this work, we focused our LLM-based approaches on relatively small models with around 4 billion parameters. To evaluate the limitations of these smaller models and assess the impact of fine-tuning, we compared our fine-tuned 4B/3B models with larger non–fine-tuned models from the same families. Table 4 summarizes these results. As shown in Table 4, fine-tuned models consistently outperform their non–fine-tuned counterparts of the same size in both fault localization and repair. Moreover, the fine-tuned 4B/3B models achieve results comparable to or even surpassing those of much larger (8B–14B) pre-trained models.

Table 4.

Summarized results for fine-tuned and non fine-tuned Large Language Models. The models used for the fault localization were Qwen 3 with sizes 4B, 8B and 14B, while the models used for repair were Qwen 2.5 Coder with sizes 3B, 7B and 14B

Table 4 Long description

The table presents a comparison of fault localization accuracy and repair rates for various large language models. It has four rows and three columns. The columns are labeled ‘Fault localization Acc@1’ and ‘Repair rate’. The rows are labeled with different model types and sizes: Fine-tuned LLM 4B/3B, Pre-trained LLM 4B/3B, Pre-trained LLM 8B/7B, and Pre-trained LLM 14B. Row 1: Fine-tuned LLM 4B/3B, Fault localization Acc@1, 33.4 percent; Repair rate, 34.2 percent. Row 2: Pre-trained LLM 4B/3B, Fault localization Acc@1, 14.2 percent; Repair rate, 14.1 percent. Row 3: Pre-trained LLM 8B/7B, Fault localization Acc@1, 6.5 percent; Repair rate, 21.5 percent. Row 4: Pre-trained LLM 14B, Fault localization Acc@1, 6.6 percent; Repair rate, 37.9 percent.

Interestingly, the non–fine-tuned Qwen 3 4B model performs substantially better than its 8B and 14B counterparts. According to the Qwen developers, the 4B version underwent a longer and more extensive training process than the other two variants, which likely explains its stronger baseline performance.