3.1 Language

We consider an idealized functional language similar to the core of ML. Our target language features algebraic data types and recursive functions with the syntax definition depicted in Figure 2. Programs P are recursive functions whose bodies are expressions e. Application is written $e_1 ~ e_2$ , $\kappa$ ranges over data type constructors, $a(\kappa)$ denotes the arity of $\kappa$ , $\kappa^{-1}$ denotes a destructor which extracts all the subcomponents of a constructor application of $\kappa$ as a tuple. An expression $e.n$ projects the n-th component of a tuple. We use ML-style pattern $\textsf{match}$ expressions. We use $\overline{\kappa_j \; \_ \to e_j}^{k}$ to denote $\kappa_1 \; \_ \to e_1 \mid \dots \mid \kappa_k \; \_ \to e_k$ . A hole is written $\Box_u$ , where u is the hole name, which we tacitly assume is unique. Each hole is associated with input–output examples, a finite function from input values to output values. Values v are made up of constructor values for data types, tuples, and recursive functions. Recursive functions can be used as input examples when synthesizing higher-order functions but cannot be used as output examples. Environments $\sigma$ map variables to values. For conciseness, we assume that all functions take a single argument, which does not harm the expressivity of the language since we can represent multiple inputs as a single tuple.

Fig. 2.

Our ML-like language.

Though we use two parameters ${\texttt x}$ and ${\texttt y}$ in the overview example for better readability, we use a single tuple parameter ${\texttt x}$ in the actual program since our language assumes a single argument for functions.

3.2 Notations

We will use some notations throughout the remaining sections. An open hypothesis (resp. open expression) is a program (resp. expression) that contains one or more holes. A closed hypothesis is a program that does not contain any holes. We will use the fixed variables $\texttt{f}$ and $\texttt{x}$ to denote the target function and its formal parameter, respectively. We use $\sigma \vdash P \Rightarrow v$ to denote the standard multistep call-by-value operational semantics of a program P without holes under environment $\sigma$ . Lastly, we denote the set of all subexpressions of an expression e as $\textsf{SubExprs}(e)$ .

3.3 Problem definition

Given an environment $\sigma$ that provides definitions of external functions and an initial open hypothesis $P_{in} = \textsf{rec} \; \texttt{f}(\texttt{x}) = \Box_{in}$ where $\Box_{in} = \bigcup_{1 \leq j \leq n} \{i_j \mapsto o_j\}$ represents input–output examples that should be satisfied by the function body, our goal is to find a closed hypothesis of form $P = \textsf{rec} \; \texttt{f}(\texttt{x}) = e$ that satisfies the input–output examples of $\Box_{in}$ . Formally, $\forall 1 \leq j \leq n.~ \sigma[\texttt{f} \mapsto \textsf{rec} \; \texttt{f}(\texttt{x}) = e, \texttt{x} \mapsto i_j] \vdash P~\texttt{x} \Rightarrow o_j$ (denoted $P \models_{\sigma} \Box_{in}$ ). We just use the user-provided external functions without inventing new ones.

4.1 Overall algorithm

Figure 1 shows the high-level structure of our algorithm. The algorithm takes as input an environment $\sigma$ that provides definitions of external functions, which we tacitly assume to be globally accessible throughout the algorithm, an initial open hypothesis with input–output examples, and initial component size n. Finally, it returns a program P that satisfies the input-output examples. With initially empty component set C, the main loop of our synthesis procedure (lines 2–28) is repeated until a solution is found. The loop starts by invoking the ComponentGeneration procedure (line 3) which takes a current component pool C, the input-output examples $\Box_{in}$ , and the target component size n. The procedure generates new components by composing existing components in C. It applies the standard pruning technique based on observational equivalence with respect to the input examples. Expressions with recursive calls to the target function being synthesized can be included in the resulting component pool. Because we cannot evaluate such recursive components as the function is unknown yet, we cannot apply the observational equivalence reduction based on their outputs. Instead, we exploit functional congruence, i.e., the same input to the function always results in the same output. For example, we do not maintain both of $\texttt{f}~2$ and $\texttt{f}~(1+1)$ in the component pool. Next, the LibrarySampling procedure (Section 4.2) is invoked to derive an inverse map for each function expression in the component pool (line 4). Next, the BlockGen procedure is invoked to obtain satisfying blocks for each input-output example (line 6). Each block must be recursion- and conditional-free because the target function is unknown yet and conditionals are not necessary when it comes to a single input-output example. Therefore, any components containing recursive calls and conditionals must not be used in blocks. We exclude such components from C (line 5) and provide the reduced component set to the BlockGen procedure (Section 4.4). With inverse maps $\mathcal{I}$ and blocks B, the inner loop (lines 8–26) iteratively processes elements in the priority queue Q. The priority queue Q contains hypotheses (initially only $P_{in}$ ) and is sorted according to the cost (Section 5.3) of each hypothesis. In each iteration, we pick a minimum-cost hypothesis P from the queue (line 9). We first check if P is structurally recursive and guaranteed to terminate (line 10) (Section 4.6). If not, we continue to the next hypothesis. If P is closed (line 13) and correct with respect to the top-level input–output examples, P is returned as a solution (line 14). Otherwise, we continue investigating other hypotheses in the queue. If a chosen hypothesis P is open, we pick a hole $\Box_u$ in P (line 18). Then, the Deduce procedure (Section 4.3) returns possible replacements for the hole $\Box_u$ (line 19). A replacement e for the hole may be a closed expression satisfying the example of $\Box_u$ , or an open expression with new holes. For each replacement e, we obtain a new hypothesis P’ by replacing the hole with e (line 20). If P’ is closed, there are no unknowns left to be synthesized (line 21). Hence, we add P’ into the queue, so that its correctness can be checked in the next iterations. If P’ is open, we check its consistency with the blocks B before adding it to the queue (line 22) by invoking the BlockConsistent procedure (Section 4.5). If the queue becomes empty before a solution is found, we increase the component size n by 1 (line 27) and restart the main loop.

Our algorithm is sound and complete in that it finds a program correct with respect to the given input–output examples if it exists in the search space.

Algorithm 1

The TRIO Algorithm

4.2 Getting inverse maps of external functions by library sampling

This section describes the LibrarySampling procedure that derives a set $\mathcal{I}$ whose each element is a triple $(g, v_o, v_i)$ where g is a function value and $v_i$ and $v_o$ are non-function values, meaning that $\sigma \vdash g~v_i \Rightarrow v_o$ . We will write $(g, v_o, v_i)$ as $g^{-1}(v_o) = v_i$ .

Given the component pool C and the top-level input–output examples $\Box_{in} = \bigcup_{1 \leq j \leq n}\{i_j \mapsto o_j\}$ as input, we first compute a finite domain $D = \{ v \in {Val} \mid \exists 1 \leq j \leq n. ~ v \sqsubseteq i_j \}$ where $\sqsubseteq$ denotes a well-founded ordering on values. In our implementation, we represent values as abstract syntax trees and use the subtree relation. Using the values in D as inputs, we compute a set of inverse maps as follows:

\[\mathcal{I} = \{ g^{-1}(v_o) = v_i \mid g,v_o \in {Val},v_i \in D, \exists e \in {\bf C}, 1 \leq j \leq n.~ \sigma[\texttt{x} \mapsto i_j] \vdash e \Rightarrow g, \sigma \vdash g~v_i \Rightarrow v_o\}\]

The use of the component pool C is for computing inverse maps of functions provided as input examples, which is useful for synthesizing higher-order functions.

where $\Box_{in} = \{ (\textsf{rec}\ \texttt{one}\ (\texttt{n})= \texttt{S}(\texttt{Z}), 1) \mapsto 1, (\textsf{rec}\ \texttt{inc} \ (\texttt{n})= \texttt{S}(\texttt{n}), 0) \mapsto 1\}$ . The solution is

Suppose the component pool ${\bf C} = \{\texttt{x}.1, \texttt{x}.2\}$ . The domain D is $\{0, 1\}$ . We derive $(\textsf{rec}\ \texttt{one}(\texttt{n})= \texttt{S}(\texttt{Z}))^{-1}(1) = 0 \in \mathcal{I}$ because $\sigma[\texttt{x} \mapsto (\textsf{rec}\ \texttt{one}(\texttt{n})= \texttt{S}(\texttt{Z}), 1)] \vdash \texttt{x}.1 \Rightarrow \textsf{rec}\ \texttt{one}(\texttt{n})= \texttt{S}(\texttt{Z})$ and $\sigma \vdash (\textsf{rec}\ \texttt{one}(\texttt{n})= \texttt{S}(\texttt{Z}))~0 \Rightarrow 1$ . In a similar manner, we conclude $(\textsf{rec}\ \texttt{inc}(\texttt{n})= \texttt{S}(\texttt{n}))^{-1}(1) = 0$ , $(\textsf{rec}\ \texttt{inc}(\texttt{n})= \texttt{S}(\texttt{n}))^{-1}(2) = 1 \in \mathcal{I}$ . In conclusion,

\[ \begin{array}{rcl} \mathcal{I} &=& \{(\textsf{rec}\ \texttt{one}(\texttt{n})= \texttt{S}(\texttt{Z}))^{-1}(1) = 0, (\textsf{rec}\ \texttt{one}(\texttt{n})= \texttt{S}(\texttt{Z}))^{-1}(1) = 1, \\ & & \ \ (\textsf{rec}\ \texttt{inc}(\texttt{n})= \texttt{S}(\texttt{n}))^{-1}(1) = 0, (\textsf{rec}\ \texttt{inc}(\texttt{n})= \texttt{S}(\texttt{n}))^{-1}(2) = 1\}. \end{array}\]

We consider D to be the domain for sampling for the following reason. We permit recursive calls on values that are strictly smaller than the input to ensure that our synthesized programs terminate, and inputs to a function often flow to other functions called inside of it. Therefore, it is likely that $\mathcal{I}$ captures input–output behaviors of library functions that can be observed during the evaluation of the desired program with the user-provided input examples.

4.3 The Deduce procedure

We now describe the $\textsf{Deduce}$ procedure that returns a set of expressions that are either closed or open as possible replacements for a given hole $\Box_u$ .

$\textsf{Deduce}({\bf C}, \mathcal{I}, \Box_u)$ is the smallest set of expressions satisfying the constraints depicted in Figure 3. The rule says if we have components in C that immediately satisfy $\Box_u$ , every such component can fill the hole ( $e \models_{\sigma} \Box_u$ denotes $\forall i \mapsto o \in \Box_u. ~ \sigma[\texttt{x} \mapsto i] \vdash e \Rightarrow o$ ). indicates that all recursive components in C are considered potential replacements for the hole. This rule is under an optimistic assumption that any recursive expressions whose semantics is unknown yet may satisfy the hole, although in reality not all recursive expressions can do. Later, any hypothesis containing a recursive call that is determined to be infeasible will be discarded (will be detailed in Section 4.5). generates new examples for arguments of a constructor application. If the example values in the hole $\Box_u$ consist of constructor values with a shared constructor $\kappa$ of arity k, then it creates k new examples constraints over the k arguments of the constructor value. creates a new example for the argument of a destructor value. The new example consists of constructor values with a shared constructor $\kappa$ where $\kappa$ can be any constructor. creates closed expressions as replacements for the hole. is for creating new examples corresponding to arguments that must be synthesized for a tuple expression. The deductive reasoning process is similar to that of . first identifies components that can be used as scrutinees. Then, for each $\textsf{match}$ expression whose scrutinee is such a component, it distributes the given examples in the hole to each branch. Lastly, uses the inverse maps of external functions. For example, for an input-output example $i \mapsto o$ , if we can find a triple $g^{-1}(o) = v$ in $\mathcal{I}$ , then we can deduce an example $i \mapsto g$ for the function part and another example $i \mapsto v$ for the argument part.

Fig. 3.

Inference rules for Deduce.

Comparison with prior work

Compared to the IRefine rules in Myth (Osera and Zdancewic, Reference Osera and Zdancewic2015) and the deductive reasoning rules in $\lambda^2$ (Feser et al., Reference Feser, Chaudhuri and Dillig2015) that also propagate examples to holes in a top-down manner, the novelty of $\textsf{Deduce}$ lies in the d_extcall rule. In Myth, new function applications are generated by enumerating all possible combinations of functions and arguments. In contrast, by using the inverse maps in the d_extcall rule, we can expedite the search in a goal-directed manner. The deductive reasoning of $\lambda^2$ is only applicable to a fixed set of predefined functions such as filter and map. In contrast, $\textsf{Deduce}$ is applicable to any external function.

Example 4. Consider the overview example in Section 2. Let us denote the target function as $\texttt{f}$ . Suppose we have a hole $\Box_u = \{(1,2) \mapsto 2\}$ and a component pool ${\bf C}=\{$ $\texttt{x}$ , $\texttt{f}~(\texttt{S}^{-1}(\texttt{x}.1), \texttt{x}.2)$ , 2, $\texttt{add}$ $\}$ . We can deduce the following constraints by applying the rules in Figure 3.

Note that we cannot apply the rule because $\Box_u$ does not contain any tuple. Also, when applying the rule, we cannot use the components $\texttt{x}$ and $\texttt{f}~(\texttt{S}^{-1}(\texttt{x}.1), \texttt{x}.2)$ as a scrutinee because neither of them evaluates to a constructor application (in particular, $\texttt{f}~(\texttt{S}^{-1}(\texttt{x}.1), \texttt{x}.2)$ cannot evaluate to a concrete value as $\texttt{f}$ is not defined yet). The following is the smallest solution satisfying the constraints over $\textsf{Deduce}({\bf C}, \Box_u)$ .

\[\small \begin{array}{l} \textsf{Deduce}({\bf C}, \mathcal{I},\Box_u) = \{ 2, \texttt{f}~(\texttt{S}^{-1}(\texttt{x}.1), \texttt{x}.2), \texttt{S}(\Box_{u_1}), \texttt{S}^{-1}(\Box_{u_2}), \texttt{x}.2, \textsf{match} \; 2 \; \textsf{with} \; \texttt{Z} \to \Box_{u_3} \mid \texttt{S} \_ \to \Box_{u} , \\ \qquad \qquad \qquad \qquad \Box_{u_4}~\Box_{u_5}, \Box_{u_4}~\Box_{u_6}, \Box_{u_4}~\Box_{u_7} \} \end{array}\]

The Deduce procedure is sound in the following sense.

Definition 5 (Soundness of Deduction). Let $\Box_u$ be a set of input–output examples, and let C and $\mathcal{I}$ be a set of components and a set of inverse maps, respectively. If there exists an expression satisfying $\Box_u$ , for every open expression $e \in \textsf{Deduce}({\bf C}, \mathcal{I}, \Box_u)$ , for every hole $\Box_{u'}$ in e, there exists an expression satisfying the hole $\Box_{u'}$ .

Intuitively, the deduction procedure is sound if there exists a solution to a synthesis task, then there also exists a solution to every synthesis subtask derived from the original synthesis task.

Theorem 6. Without using the rule, the Deduce procedure is sound.

Proof. Available in the appendix.

The following example shows that the Deduce procedure can be unsound when using the rule.

Example 7. Recall Example 3. Let us denote the first and second input examples in $\Box_{in}$ as $i_1$ and $i_2$ respectively (i.e., $i_1 = (\textsf{rec}\ \texttt{one}\ (\texttt{n})= \texttt{S}(\texttt{Z}), 1)$ , $i_2 = (\textsf{rec}\ \texttt{inc} \ (\texttt{n})= \texttt{S}(\texttt{n}), 0) $ ). We can deduce the following fact by applying the rule because $(\textsf{rec}\ \texttt{one}\cdots)^{-1}(1) = 0 \in \mathcal{I}$ .

\[\small\begin{array}{l} \Box_{u_1}~\Box_{u_2} \in \textsf{Deduce}({\bf C}, \Box_{in}) \quad (\Box_{u_1} = \{i_1 \mapsto \textsf{rec}\ \texttt{one} \cdots, i_2 \mapsto \textsf{rec}\ \texttt{one} \cdots \}, \Box_{u_2} = \{i_1 \mapsto 0, i_2 \mapsto 0 \})\end{array}\]

We cannot synthesize an expression satisfying the hole $\Box_{u_1}$ . That is because we cannot synthesize an expression that evalutes to the one function under the environment where $\texttt{x}$ is bound to $i_2$ (the only available function is $\texttt{inc}$ ). Recall that we do not synthesize any new auxiliary functions.

4.4 Constructing blocks from each input–output example

This section describes the BlockGen procedure for computing satisfying blocks for each input–output example in $\Box_{in}$ . The set of blocks is stored in a version space (Gulwani, Reference Gulwani2011) which is a compact representation of expressions.

We begin with the definition of version spaces.

• • A union: ${\bigcup} \textbf{V}$ where $\textbf{V}$ is a set of version spaces

• • An expression

• • An application: written $(\widetilde{e_1}~\widetilde{e_2})$ where $\widetilde{e_i}$ are version spaces

• • A tuple: written $(\widetilde{e_1}, \cdots, \widetilde{e_k})$ where $\widetilde{e_i}$ are version spaces

• • A constructor: written $\kappa(\widetilde{e_1}, \cdots, \widetilde{e_k})$ where $\widetilde{e_i}$ are version spaces and $\kappa$ is a constructor

• • A destructor: written $\kappa^{-1}(\widetilde{e})$ where $\widetilde{e}$ is a version space and $\kappa$ is a constructor

• The empty set, $\emptyset$

A version space can be understood as an E-graph where each node represents a set of expressions. Each leaf node represents a single expression, and they are composed into larger sets. The union operator ${\bigcup}$ symbolizes a nondeterministic choice between multiple expressions, allowing version spaces to compactly represent huge sets of expressions.

The set of expressions encoded by a version space is defined as follows:

Definition 10. The set represented by a version space is written and is defined recursively as

With this in mind, we are ready to describe how to obtain a version space of blocks. Given a set C of components and the top-level input–output examples $\Box_{in}$ , the BlockGen procedure computes the smallest version spaces satisfying the constraints in Figure 4. The result maps each input–output example to a version space of satisfying blocks. In the rule, $\textsf{Blocks}({\bf C}, i \mapsto o)$ denotes the version space of blocks satisfying a single input-output example $i \mapsto o$ . The other rules depict how to compute a version space for a single example. $\textsf{SimpleBlocks}$ denotes a set of component expressions satisfying that example. $\textsf{CompoundBlocks}$ denotes a set of version spaces each of which is not a single expression. We reuse the Deduce procedure to obtain $\textsf{CompoundBlocks}$ , which can be derived by the other remaining rules . Note that the given set of components C only includes recursion- and conditional-free expressions (by line 4.1 in Algorithm 1), and there are no rules for deriving version spaces containing $\textsf{match}$ expressions and recursive calls, thereby ensuring that the resulting version space represents a set of blocks.

Fig. 4.

Inference rules for BlockGen.

Example 11. Recall $\textsf{Deduce}({\bf C}, \mathcal{I}, \Box_u)$ in Example 4. We describe how to obtain a version space of blocks for the input–output example $\Box_u = \{(1,2) \mapsto 2\}$ using the rules in Figure 4. $\textsf{Blocks}({\bf C}, \mathcal{I}, \Box_u)$ is computed as follows: first, by the B_GEN_PER_EX rule,

$$ \textsf{Blocks}({\bf C}, \mathcal{I}, \Box_u) = {\bigcup}\ \textsf{CompoundBlocks}({\bf C}, \mathcal{I}, \Box_u) \cup \textsf{SimpleBlocks} $$

where $\textsf{SimpleBlocks} = \{ 2 \}$ because 2 is the only component satisfying the example. We can deduce the following constraints over $\textsf{CompoundBlocks}({\bf C}, \mathcal{I}, \Box_u)$ as follows:

where $\Box_{u_1}, \cdots, \Box_{u_7}$ are the ones defined in Example 4. We keep applying the rules to generate constraints over $\textsf{Blocks}({\bf C}, \mathcal{I}, \Box_{u_1}), \cdots, \textsf{Blocks}({\bf C}, \mathcal{I}, \Box_{u_7})$ . For example, by the B_GEN_PER_EX rule, $\textsf{Blocks}({\bf C}, \mathcal{I}, \Box_{u_1}) = $ ${\bigcup}\ \textsf{CompoundBlocks}({\bf C}, \mathcal{I}, \Box_{u_1}) \cup \textsf{SimpleBlocks}$ where $\textsf{SimpleBlocks} = \emptyset$ because no component in C satisfies the example $\Box_{u_1} = \{(1,2) \mapsto 1\}$ . Constraints over $\textsf{CompoundBlocks}({\bf C}, \mathcal{I}, \Box_{u_1})$ are generated by applying the rules in a similar manner. For instance, by the B_PROJ rule, a constraint $\texttt{x}.1 \in \textsf{CompoundBlocks}({\bf C}, \mathcal{I}, \Box_{u_1})$ will be generated since $\texttt{x}.1 \in \textsf{Deduce}({\bf C}, \mathcal{I}, \Box_{u_1})$ . $\textsf{Blocks}({\bf C}, \mathcal{I}, \Box_{u_4})$ will include a version space of a single expression $\texttt{add}$ since $\texttt{add}$ is a component satisfying the example. $\textsf{Blocks}({\bf C}, \mathcal{I}, \Box_{u_6})$ (where $\Box_{u_6} = \{(1,2) \mapsto (1,1)\}$ ) will include a version space of a tuple ( $\textsf{Blocks}({\bf C}, \mathcal{I}, \Box_{u_1})$ , $\textsf{Blocks}({\bf C}, \mathcal{I}, \Box_{u_1})$ ) by the B_TUPLE rule.

When generating constraints, a cycle that leads to blocks of infinite length may occur. For example, we may generate the following two constraints: $\texttt{S}(\textsf{Blocks}({\bf C}, \mathcal{I}, \Box_{u_1})) \subseteq \textsf{CompoundBlocks}({\bf C}, \mathcal{I}, \Box_u)$ and $\texttt{S}^{-1}(\textsf{Blocks}({\bf C}, \mathcal{I}, \Box_{u})) \subseteq \textsf{CompoundBlocks}({\bf C}, \mathcal{I}, \Box_{u_1})$ that can be used to generate blocks of form $\texttt{S}(\texttt{S}^{-1}(\texttt{S}(\texttt{S}^{-1}( \cdots)))$ . In our implementation, we bound the maximum height of version spaces to avoid generating blocks of infinite length.

We can derive the final version space of blocks for $\Box_u$ by finding the following smallest version space satisfying the above constraints.

\[\begin{array}{l} \textsf{Blocks}({\bf C},\mathcal{I}, \Box_u) = \\ \qquad {\bigcup} \{2, \texttt{x}.2, \texttt{S}(\texttt{x}.1), (\texttt{add}~({\bigcup} \{\texttt{x}.1, \texttt{S}^{-1}(\texttt{x}.2), \cdots\}, {\bigcup} \{\texttt{x}.1, \texttt{S}^{-1}(\texttt{x}.2), \cdots\} )), \cdots \}.\end{array}\]

4.5 Pruning infeasible hypotheses using blocks

Finally, in this section, we describe the $\textsf{BlockConsistent}$ procedure that prunes infeasible hypotheses using the blocks generated by the $\textsf{BlockGen}$ procedure.

Deriving blocks from a hypothesis by unfolding

We first describe how to obtain blocks from an open hypothesis which we want to determine the feasibility of by a technique we call unfolding. Suppose a currently considered hypothesis is $P = \textsf{rec}\; \texttt{f}(\texttt{x}) = e_{\textrm{body}}$ . For each input example i in the top-level input–output example $\Box_{in}$ , we perform symbolic evaluation (interleaved with concrete evaluation) over $e_{\textrm{body}}$ and obtain a block, which possibly contains holes. We call such a block possibly with holes (resp. without holes) an open block (resp. closed block). We formalize our symbolic evaluation via the transition relation $e \ \to_{\!P,i}\ e'$ induced by the target hypothesis P and the input i. The relation says that the expression e takes a single step to the expression e’. The transition relation is formally defined by the rules in Figure 5. The rules , , , and perform symbolic evaluation on the arguments of constructor, destructor, tuple, and projection expressions, respectively. and perform symbolic evaluation on the left and right hand sides of applications. The most notable part is the remaining two rules. for $\textsf{match}$ expressions concretely evaluates the scrutinee e of a given $\textsf{match}$ expression with input i. Then, a branch is chosen by the concrete value of the scrutinee. To obtain concrete values of scrutinees, we require scrutinees not to contain recursive calls to the target function, which is unknown yet. Therefore, any hypothesis containing a match expression that pattern matches on a recursive call to the target function (called inside-out recursion (Osera, Reference Osera2015)) will get stuck and thus will be determined to be infeasible. This means Candidate generator will never generate programs with inside-out recursion. However, such programs can still be synthesized by our algorithm as Bottom-up enumerator will eventually enumerate all programs. is a special rule for recursive calls. Any recursive call to the target function $\texttt{f}$ is replaced by the body of the function where every occurrence of the parameter $\texttt{x}$ is replaced by the argument expression. Note that there are no transition rules for variables and holes. That is, every variable and hole in the hypothesis remains unchanged.

Fig. 5.

Rules for unfolding (symbolic evaluation interleaved with concrete evaluation) for deriving open blocks from $P=\textsf{rec}\ \texttt{f}(\texttt{x}) = e_{\textrm{body}}$ with input i.

Given the top-level input-output examples $\Box_{in} = \bigcup_{1 \leq j \leq n} \{ i_j \mapsto o_j \}$ , the set of open blocks derivable from hypothesis $P = \textsf{rec} \ \texttt{f}(\texttt{x}) = e_\textrm{body}$ (denoted $B_P$ ) is defined as follows:

\[B_P = \{(i_j,o_j) \mapsto e \mid e_\textrm{body} \ \to_{P,i_j}^*\ e, 1 \leq j \leq n\}.\]

where $\to_{P,i_j}^*$ is the transitive closure of $\to_{P,i_j}$ . That is, with each input example, we apply the transition rules till the end to obtain an open block.

Example 12. Recall the hypothesis $P_4$ in the overview example in Section 2.

Using the rules in Figure 5, we can derive an open block from $P_4$ with input example $i = (1,2)$ as follows:

Checking feasibility of a hypothesis

We check the feasibility of a hypothesis P by checking if it is block consistent with respect to the set $\textbf{B}$ of closed blocks from the BlockGen procedure, which is formally defined as follows:

Definition 13. Given the top-level input-output examples $\Box_{in}$ , a hypothesis $P = \textsf{rec}\ \texttt{f}(\texttt{x}) = e$ is block consistent with respect to a set B of blocks if and only if

\[ \forall i_j \mapsto o_j \in \Box_{in}. ~ B_P(i_j, o_j) \sim \textbf{B}(i_j, o_j) \]

where $\sim $ is a binary relation over $\it Exp$ and version spaces, which is defined in Figure 6.

Fig. 6.

Matching rules for checking block consistency.

The relation $\sim $ relates an open block to a set of closed blocks. Specifically, for an open block e and a version space of closed blocks $\widetilde{e}$ , $e \sim \widetilde{e}$ holds if we can obtain an expression in $\widetilde{e}$ by properly filling each occurrence of the holes in e. Checking if $e \sim \widetilde{e}$ resembles conventional syntactic matching between different expressions but with the following differences. Syntactic matching has as a goal to determine whether two expressions can be made equal by searching for a proper substitution from variables into expressions. For example, $\texttt{add}~(\texttt{x}, \texttt{Z})$ can be matched with $\texttt{add}~(\texttt{Z}, \texttt{Z})$ since we can substitute x with Z. On the other hand, in our method, not variables but only holes are targets for substitution. In addition, in contrast to syntactic matching that traverses two expressions, our method simultaneously traverses one expression and a version space to figure out if an open block can be matched with a closed block in the version space.

The first rule in Figure 6 says that any hole can be matched with any expression in $\widetilde{e}$ as long as $\widetilde{e}$ is not empty. The other rules recursively traverse the version space $\widetilde{e}$ of blocks and check if e can be matched with any expression in $\widetilde{e}$ .

Finally, the BlockConsistent procedure is defined as follows:

\[ \textsf{BlockConsistent}(P, \textbf{B}, \Box_{in}) = \left\{ \begin{array}{ll} \texttt{true} & (\text{if } \forall i_j \mapsto o_j \in \Box_{in}. ~ B_P(i_j, o_j) \sim \textbf{B}(i_j, o_j)) \\ \texttt{false} & (\text{otherwise}) \end{array} \right.\]

Example 14. The following derivation tree shows how the open block in Example 12 can be matched with the version space $(\texttt{add}~({\bigcup}\ \{\texttt{Z}, \texttt{S}^{-1}(\texttt{x}.1)\}, {\bigcup}\ \{\texttt{x}.2, \texttt{S}(\texttt{x}.1)\})$ using the rules in Figure 6.

As already mentioned in Section 2, the block-based pruning presented may be unsound; a valid open hypothesis that can be a solution in the future may be pruned by the block-based pruning. Such a situation may occur if Block generator is not able to generate closed blocks for the valid hypothesis due to a lack of components. However, as the component pool grows, such unsoundness may be mitigated.

Please recall that even though the block-based pruning is unsound, our algorithm finds a solution if exists by resorting to Bottom-up enumerator that will eventually generate a solution of finite size.

4.6 Ensuring termination of synthesized programs

In this section, we describe how to ensure termination of synthesized programs. Through the Terminate procedure on line 10 of Algorithm 1, we check if a chosen candidate program $P = \textsf{rec}~\texttt{f}(\texttt{x}) = e_{\text{body}}$ is guaranteed to terminate. If P is an open hypothesis containing holes, the Terminate judgment answers if P can be completed to a terminating program. If P is a closed program, the Terminate judgment checks if P is terminating.

The pseudo-code and the inference rules in Figure 7 define our termination checking procedure. Figure 7(a) shows the Terminate procedure and its helper functions. The Terminate procedure takes a target function of the form $\textsf{rec}~\texttt{f}(\texttt{x}) = e_{\text{body}}$ and returns true if the program is guaranteed to terminate. If there is no recursive call, the program is guaranteed to terminate (line 3).Otherwise, the program is guaranteed to terminate if all recursive calls are structurally decreasing. This is checked by the Struct function (line 5). For every recursive call $\texttt{f} ~ e$ in the body of the target function (denoted $\textsf{RecursiveCalls}(e_{\text{body}})$ ), we check if the recursive call is valid. The judgment $\textsf{Struct}(\texttt{f} ~ e, K)$ states that the argument expression e of the recursive call is deemed structurally decreasing where K is a set of indices of the arguments that may have to be structurally decreasing. Let us call such arguments key arguments. To see the role of K in ensuring termination of recursive calls, consider the following example.

Fig. 7.

Termination checking procedure.

The first component of the input tuple is a list to be reversed, and the second component is an accumulator. The function pattern matches on the first component of the input tuple. Therefore, the first component should be structurally decreasing in each recursive call to ensure termination. On the other hand, the second component does not affect the termination of the function. To see if the function is terminating, we keep track of the indices of the key arguments and check if the arguments are structurally decreasing. In this case, the key argument is the first component of the input tuple and the set K is $\{1\}$ .

The function $\textsf{KeyArgs}$ takes an expression e as input and returns a set of indices of the key arguments. If e is a $\textsf{match}$ expression (line 17), the key arguments are the indices of the arguments that appear in the scrutinee of the $\textsf{match}$ expression (line 18). The reason is as follows: recursive calls are typically made in the branches of $\textsf{match}$ expressions (otherwise, the program would never terminate because of unconditional recursion), and the arguments that appear in the scrutinee of the $\textsf{match}$ expression determine which branch to take, deciding whether recursive calls are made further or not. Therefore, it is likely that the arguments that appear in the scrutinee of the $\textsf{match}$ expression are key arguments. Because $\textsf{match}$ may be nested, key arguments in the branches are collected and merged (line 19). If e is not a $\textsf{match}$ expression, the $\textsf{KeyArgs}$ function recursively calls itself on the sub-expressions of e and collects the key arguments by unioning the results (line 21).

Given a set of key arguments K and an expression e that may contain a tuple of arguments or a single argument, the function $\textsf{Struct}(e, K)$ checks if the arguments are structurally decreasing. If e is a tuple expression (line 8), it first checks if K is empty (line 9). If K is empty, the function returns false because there is no key argument to check (line 9). Otherwise, the function extracts the components of e at the indices in K and the corresponding components of $\texttt{x}$ (line 10 and 11). Here, $(e_k)_{k \in K}$ denotes a tuple of $(e_{k_1}, e_{k_2}, \cdots)$ where $k_1, k_2, \cdots$ are the indices in K and $(\texttt{x}.i)_{i \in K}$ denotes a tuple of $(\texttt{x}.{k_1}, \texttt{x}.{k_2}, \cdots)$ where $k_1, k_2, \cdots$ are the indices in K. The function then checks if the extracted components are structurally decreasing (line 12) using the partial order relation $\sqsubset$ defined in Figure 7(b). The partial order relation $\sqsubset$ is defined by the rules Ord_dtor, Ord_proj, and Ord_tuple. The rule Ord_dtor states that a destructor expression is structurally smaller than the expression it destructs. The rule Ord_proj states that a projection expression is structurally smaller than the expression it projects if the expression itself is structurally smaller than the projected expression or the two expressions are equal. The rule Ord_tuple states that a tuple expression $e = (e_1, \cdots, e_m)$ is structurally smaller than another tuple expression $e' = (e_1', \cdots, e_m')$ if any components of e are structurally smaller than the corresponding components of e’ and the rest of the components are equal. Lastly, if e is not a tuple expression (line 13), the function checks if e is structurally smaller than $\texttt{x}$ using the partial order relation $\sqsubset$ (line 14).

Theorem 17 guarantees that if the Terminate procedure accepts a closed hypothesis, then it is guaranteed to terminate on any input.

4.7 Optimizations

We describe several optimizations that we use to improve the efficiency of our algorithm.

Using another version of the d_extcall rule

As described in Example 7 in Section 4.3, the d_extcall rule in Figure 3 may be unsound (i.e., some of generated holes cannot be filled with any expression). This deduction unsoundness may lead to a scalability issue by generating too many unsatisfiable holes if the number of examples is greater than a certain threshold. In such a case, we use the following two rules instead of the d_extcall rule.

Both rules use components to generate closed expressions without any holes. The difference between the two rules is in whether a component for the argument part contains recursive calls. The d_extcall2 rule considers every component cotaining recursive calls to be the potential argument of a library function call. This is based on an optimistic assumption that any recursive expression may be a proper argument (like the d_rec rule in Figure 3). These rules do not generate any unsatisfiable holes, but the search space explored in a single iteration of the main loop (lines 2–28 in Algorithm 1) is smaller than the case where the d_extcall rule is used.

5.1 Preventing unsafe destructor applications

We do not permit potentially unsafe destructors to be used in any candidate program. For example, suppose the following candidate is explored during the search.

This program is considered invalid and not generated during the search. In the branch of S, the expression S $^{-1}$ (S $^{-1}$ (x)) is not allowed because the pattern match does not guarantee that the input x is of the form S(S(…)). The only safe destructor usable in the branch of S is S $^{-1}$ (x). To prevent such unsafe destructor applications, $\textsf{Deduce}({\bf C}, \mathcal{I}, \Box_u)$ on line 19 in Algorithm 1 is changed to $\textsf{Deduce}(\textsf{RemoveUnsafeComp}(P, \Box_u, {\bf C}), \mathcal{I}, \Box_u)$ where $\textsf{RemoveUnsafeComp}(P, \Box_u, {\bf C})$ returns a subset of C whose components do not contain any unsafe destructor applications at the position of $\Box_u$ in P. This filtering prevents components of unsafe destructor applications from being used to fill the hole $\Box_u$ in the candidate program P.

5.2 Ensuring termination of block and candidate generation

In our implementation, to guarantee the termination of the BlockGen procedure and the Deduce procedure, we limit the maximum number of subsequent steps of deduction to a certain number.Footnote ⁵

In other words, to avoid generating infinitely many open hypotheses from a given initial hypothesis, we permit the Deduce procedure to be terminated after a certain number of steps of applications of rules in Figure 3. Because the BlockGen procedure relies on the Deduce procedure, this also guarantees termination of the BlockGen procedure. Note that despite this finitization, the search space is still infinite because there is no limit on the maximum component size (i.e., the component pool will keep growing until a solution is found).

5.3 Program selection

In order to synthesize likely programs, we utilize a cost function: the cost of each candidate program $P = \textsf{rec} \; \texttt{f}(\texttt{x}) = e $ is determined by the cost of its body e (denoted $\mathcal{C}(e)$ ), which is a nonnegative number. Costs of expressions satisfy the following constraints (some cases are omitted):

Intuitively, we penalize the use of constructors (thereby constants) and prioritize the use of variables, destructors, and projections. The reason for this is that they are used to extract subcomponents of constructors, which are essentially the same as variables bound by patterns in $\textsf{match}$ expressions. For example, consider the solution $P_{sol}$ of the overview example problem in Section 2.

This program can be written as the following program by introducing a new variable x’ bound by the pattern of S.

Note that $\texttt{S}^{-1}(\texttt{x})$ and x’ play the same role. Therefore, destructors and projections can be understood as variables in many cases, and prioritizing variables has been proved to be a good heuristic to avoid overfitting (Feser et al., Reference Feser, Chaudhuri and Dillig2015; Gulwani, Reference Gulwani2011). Lastly, in case of a tie, we pick a smaller program in terms of AST size. This heuristic has also been popularly used in the majority of previous approaches (Albarghouthi et al., Reference Albarghouthi, Gulwani and Kincaid2013; Feser et al., Reference Feser, Chaudhuri and Dillig2015; Wang et al., Reference Wang, Dillig and Singh2017; Miltner et al., Reference Miltner, Nuñez, Brendel, Chaudhuri and Dillig2022).

5.4 Checking block consistency

We describe implementation details for improving the pruning power of the BlockConsistent procedure. In Algorithm 1, when choosing a hole in a hypothesis of a $\textsf{match}$ expression, we prefer holes for base cases to ones for inductive cases. By filling holes for base cases first, we can effectively prune infeasible recursive hypotheses. For example, suppose we encounter the following hypothesis while synthesizing mul in Section 2.

Suppose we first fill the hole $\Box_2$ with y, obtaining the following hypothesis.

Note that this hypothesis cannot become a solution no matter what expression we put in the remaining hole. To check block consistency, for every input, we will obtain the open block $\Box_1$ as a result of our symbolic evaluation. Although the hypothesis $P_2$ is infeasible, because a hole can be matched with any expression according to the rules in Figure 6, the hypothesis will be determined to be block consistent with respect to any set of blocks, and will not be pruned.

Now, suppose we first fill the hole $\Box_1$ in $P_1$ with x, obtaining the following hypothesis.

Note that this hypothesis also cannot become a solution no matter what expression we put in the remaining hole. For every input, we will obtain a closed block x as a result of our symbolic evaluation. Because x is not included in the blocks for the example $(1,2) \mapsto 2$ , the hypothesis $P_3$ is determined to be block inconsistent and will be pruned.

5.5 Finding a solution vs. all solutions

We allow the user to choose between finding all possible solutions synthesizable using some set of components and picking the best one, and stopping the search as soon as a solution is found. This allows the user to find a good balance between speed of synthesis and accuracy (i.e., the possibility of generating intended programs). Finding all solutions and picking the best one only requires a slight modification in Algorithm 1 as follows: even if a solution is found on line 14, the main loop continues to explore the search space until the queue becomes empty. Then, we pick the best one among the multiple solutions found so far. In addition, we can further expedite the process of finding a single solution with a slight modification in Figure 3. Instead of including all components consistent with a given set of input–output examples in the rule, we just include a single component whose score is the best among the satisfying components.

6.1 Experimental setup

Benchmarks

We use 65 recursive functional programs. 45 out of 65 have been used to evaluate prior work (Osera and Zdancewic, Reference Osera and Zdancewic2015; Lubin et al., Reference Lubin, Collins, Omar and Chugh2020; Miltner et al., Reference Miltner, Nuñez, Brendel, Chaudhuri and Dillig2022). The remaining 20 programs are from the exercises in the official OCaml online tutorial and their slight variants. The details can be found in Table 1.

Table 1.

List of new 20 benchmarks collected from the exercises in the official OCaml online tutorial (https://ocaml.org/exercises) and their variants

For these benchmarks, we consider the following two classes of specifications to evaluate Trio over different specifications.

• • IO: We use input–output examples written by developers of SMyth (Lubin et al., Reference Lubin, Collins, Omar and Chugh2020) and Burst (Miltner et al., Reference Miltner, Nuñez, Brendel, Chaudhuri and Dillig2022).

• • Ref: For 45 benchmarks, we use reference implementations from prior work written by developers of Burst (Miltner et al., Reference Miltner, Nuñez, Brendel, Chaudhuri and Dillig2022). For the other 20 benchmarks, we use reference implementations from the official OCaml online and the ones written by us.

6.2 Effectiveness of Trio for input-output examples

We evaluate Trio on synthesis problems with IO specifications. The initial component size n for Trio is set to be 6. For each instance, we measure the running time of Trio and the size of the synthesized program.

The results are summarized in Table.2. The column “Correct” shows if the synthesized program is the one intended by the user. We manually checked if the synthesized program is semantically equivalent to the known solution for each problem.

Trio outperforms the other baselines in terms of both the number of solved problems and synthesis time. Trio can synthesize 65 out of 65 problems, with an average time of 1.5 seconds. On the other hand, Burst and SMyth can synthesize 54 and 53 problems, with average times of 2.6 and 2.7 seconds, respectively. In addition, Trio is the fastest tool in 42 problems, whereas Burst and SMyth are the fastest tools in 14 and 27 problems, respectively.Footnote ⁶

For every problem, the time taken for synthesizing blocks and constructing inverse maps is negligible (usually less than a second).

We observe Burst and SMyth occasionally take a large amount of memory, whereas Trio only requires a small amount of memory. While solving all of the tasks, the peak memory usage of Burst is 5.3GB and that of SMyth is 1.2GB. On the other hand, Trio only requires 88 MB even in the worst case. On average, Burst and SMyth use 647 and 40 MB of memory respectively, whereas Trio uses 24 MB. Thus, we can conclude Trio is more memory efficient than the other baselines.

Table 2.

Results for the IO benchmark suite (with 15 easy problems omitted), where “Time” gives synthesis time in seconds, and “Size” shows the size of the synthesized program (measured by number of AST nodes). Synthesis time of the fastest tool for each problem is highlighted in bold.

Thanks to the performance gain of Trio, we can synthesize programs that are hard for the other baselines. The problem expr_div is hard in that it requires complex pattern matching involving many external operators. It concerns synthesizing a simple calculator with addition, subtraction, multiplication, and division. The specification is given as follows:

where $\Box_{in}$ is the set of input–output examples $\{\texttt{NAT}\ 1 \mapsto 1, \texttt{ADD}(\texttt{NAT}\ 1, \texttt{NAT}\ 2) \mapsto 3, \cdots\}$ , and add, sub, mul, and div are the external operators. Finding the following solution is non-trivial because there are extremely many possible combinations of recursive calls, external operators, and case matching.

However, Trio can find the solution in 27 seconds.Footnote ⁷ On the other hand, the other baselines fail to solve all the problems that concern synthesizing calculators (i.e., expr, expr_sub, expr_div).

Tail-recursive functions

Trio can synthesize all of the 6 tail-recursive benchmarks (the benchmarks with the suffix _tailcall) thanks to the termination checking mechanism that permits tail-recursive calls. On the other hand, neither of the other baselines can synthesize all of the tail-recursive benchmarks.

Burst cannot synthesize tail-recursive calls because its termination checker is based on the default value order which does not permit tail-recursive calls. For example, list_rev_tailcall requires a recursive call on ([2],[1]) for input ([1;2],[]), but its value ordering does not consider ([2],[1]) to be strictly smaller than ([1;2],[]). However, it produces the correct solution for some of the tail-recursive benchmarks (list_sum_tailcall and list_length_tailcall) by finding a non-tail-recursive solution that is semantically equivalent to the tail-recursive one. For example, Burst synthesizes the following solution for list_sum_tailcall:

which is not tail-recursive but semantically equivalent to the tail-recursive solution.

SMyth employs a check during synthesis to ensure that the argument to a recursive function recursive call is a strict subterm of the parameter to the recursive call. However, all functions in SMyth are single-parameter functions, and multi-parameter functions are curried. As a result, only recursive calls that are structurally decreasing on the first parameter of multi-parameter (curried) functions are allowed (Lubin, Reference Lubin2020). This restriction limits the scope of programs that can be synthesized by SMyth. As an evidence, we have tried to put the tail-recursive argument (i.e., the accumulator) as the first parameter for list_sum_tailcall. As expected, Smyth fails to synthesize the solution because the first parameter is not strictly decreasing within the timeout limit. Burst also times out for the same reason. However, Trio can synthesizing the solution. This observation suggests that the termination checking mechanism in Trio is more flexible than the one in SMyth.

The overhead of the termination checking mechanism in Trio is negligible. On average, the termination checking mechanism takes 0.05 seconds. With the exceptions of tree_notexist that require 1.5 seconds respectively because of the large number of candidates explored during the search, the termination checking mechanism takes less than 0.1 seconds for all the other benchmarks.

6.3 Effectiveness of Trio for reference implementations

In this section, we evaluate Trio on synthesis problems with Ref specifications. We follow the same evaluation procedure as the evaluation of Burst (Miltner et al., Reference Miltner, Nuñez, Brendel, Chaudhuri and Dillig2022) for Ref specifications. The authors of Burst integrated Burst and SMyth into a CEGIS loop and, for each candidate program proposed by each tool, they use the verifier to determine whether the candidate is semantically equivalent to the reference implementation. If not, a new input–output example comprising a counterexample input generated by the verifier and its corresponding output is added.Footnote ⁸ This process is repeated until the desired program is found.

The goal of this experiment is to confirm how the tools deal with the random examples generated by the verifier, rather than hand-crafted examples.

The results are summarized in Table 3. The column “# Iters” shows the number of CEGIS iterations required until a solution is found. Trio also outperforms the other baselines in terms of both the number of solved instances and synthesis time. Trio can synthesize 60 instances, with an average time of 4.5 seconds and an average number of CEGIS iterations of 5.2. Burst can synthesize 56 instances, with an average time of 5.0 seconds and an average number of CEGIS iterations of 5. SMyth can synthesize 42 instances, with an average time of 3.0 seconds and an average number of CEGIS iterations of 4.7.

Table 3.

Results for the Ref benchmark suite where “# Iter” shows the number of CEGIS iterations.

Overall these results suggest that Trio can deal better with random examples generated by the verifier compared to the other baselines.

6.4 Ablation study for block-based pruning and library sampling

We now evaluate the effectiveness of the block-based pruning and library sampling techniques used by Trio. For this purpose, we compare the performance of four variants of Trio, each using a different combination: Trio with block-based pruning and library sampling, only with block-based pruning, only with library sampling, and with both techniques disabled.

Table 4 summarizes the results of this ablation study (more detailed results can be found in cactus plots in Figure 8). For each variant of Trio, we report the number of solved benchmarks with the IO and Ref specifications, respectively. In this experiment, we only consider the 20 newly added benchmarks because we realize the other 45 benchmarks from prior work are easy, so that they can be quickly solved by all the variants of Trio. We conjecture that the reason why even can solve all of the 45 benchmarks is that it enjoys the benefit of the synergistic combination of top-down and bottom-up search strategies. As can be seen in the table, Trio with the two techniques solves more benchmarks than the other three variants. Trio solved 100% of the new benchmarks with IO specifications, whereas could solve 70% of the benchmarks. Such a trend can also be observed in the reference implementation experiment. We notice the efficacy of block-based pruning is higher than that of library sampling because the difference between and is more significant than the difference between and .

Table 4.

Number of instances that can be solved by four variants of Trio among 20 newly added benchmarks

Fig. 8.

Comparison of different variants of Trio.

6.5 Benefits of our method compared to prior work

In this section, we analyze why our method outperforms the previous methods. As a representative example, we investigate how the tools work for a simpler version of the expr benchmark where Trio can quickly find the solution in contrast to the other baselines.

where $\Box_{in} = \{i_1 \mapsto 1, i_2 \mapsto 4, i_3 \mapsto 7\}$ , $i_1 = \texttt{NAT}\ 1$ , $i_2 = \texttt{ADD}(\texttt{NAT}\ 3, \texttt{NAT}\ 1)$ , and $i_3 = \texttt{ADD}(\texttt{NAT}\ 4, \texttt{NAT}\ 3)$ . The solution in our language (depicted in Figure 2) is as follows:

Comparison to SMyth.

Similarly to our method, SMyth explores the search space by performing top-down propagation. There are two major differences between SMyth and our method. First, whenever a hole is filled with some expression, SMyth “updates” the other remaining holes according to the hole filling, so that the holes are more likely to be filled with the correct expressions. Second, SMyth solely relies on a top-down search strategy without any bottom-up search. We observe these two differences are the main reasons why SMyth fails to solve the benchmark. Consider the following hypothesis generated by SMyth during the search.

where $\Box_1 = \{i_1 \mapsto 1\}$ , $\Box_2 = \{i_2 \mapsto 4, i_3 \mapsto 7\}$ . SMyth further refines the hypothesis $P_1$ by filling the hole $\Box_2$ with a recursive call to eval. Like Trio, SMyth enumerates structurally-decreasing recursive calls to guarantee the termination of synthesized programs. Suppose the following hypothesis is generated.

Obviously, the hypothesis $P_2$ is not desired because it cannot be the solution. However, SMyth cannot detect this problem for the following reason. As hole $\Box_2$ is filled, SMyth updates the other remaining hole. SMyth generates the following hypothesis.

where $\Box_3 = \Box_1 \cup \{\texttt{NAT}\ 3 \mapsto 4, \texttt{NAT}\ 4 \mapsto 7\}$ . The additional examples $\{\texttt{NAT}\ 3 \mapsto 4, \texttt{NAT}\ 4 \mapsto 7\}$ are originated from the original examples $\{i_2 \mapsto 4, i_3 \mapsto 7\}$ and partial evaluation of $P_2$ with the input examples. SMyth refines the hole $\Box_3$ by generating the following hypothesis.

where $\Box_4 = \{i_1 \mapsto 1\}$ and $\Box_5 = \{\texttt{NAT}\ 3 \mapsto 4, \texttt{NAT}\ 4 \mapsto 7\}$ . SMyth keeps refining this hypothesis, which is fruitless. In summary, SMyth’s updating holes by partial evaluation sometimes makes the search more difficult.

On the other hand, Trio can quickly identify the infeasibility of $P_2$ as follows: Trio does not update the other remaining hole $\Box_1$ after filling $\Box_2$ . Then, the hole $\Box_1$ can be easily filled with $\texttt{NAT}^{-1}(\texttt{x})$ , generating the following program, which can be easily proved to be infeasible by concrete evaluation.

In addition, we note that another source of inefficiency of SMyth is that it redundantly generates many semantically equivalent hypotheses. For instance, the followings are some of hypotheses generated by SMyth by filling $\Box_1$ in $P_1$ with different expressions. The more library functions are usable, the more redundant hypotheses are generated.

Note that the first two hypotheses and the last two hypotheses are semantically equivalent respectively. However, Trio avoids generating such redundant hypotheses because Bottom-up enumerator exploits observational equivalence to avoid generating multiple components of the same behaviors.

Comparison to Burst.

Burst performs bottom-up synthesis with angelic execution.Footnote ⁹ It first synthesizes a program assuming any recursive calls to the function being synthesized angelically behave to make the program correct. Then, it checks if the assumptions made in the previous step are correct. If they are, the solution is found. Otherwise, those assumptions are refuted and never made again by being added to the list of anti-specifications. This process is repeated, progressively strengthening the specification of the target function and eventually leading to a solution.

Burst fails to find the solution because it runs into extensive backtracking (i.e., too many steps of specification strengthening). Specifically, given the specification of the target function $ \texttt{eval}\ i_1 = 1 \land \texttt{eval}\ i_2 = 4 \land \texttt{eval}\ i_3 = 7 $ (which is from the three input-output examples), Burst first enumerates the following candidate program.

Obviously, the above program does not satisfy the second and third input-output examples. However, at this stage, Burst generates a program assuming any recursive call to eval can return anything to satisfy the constraints. The above program is generated by assuming $\texttt{eval}\ (\texttt{NAT}\ 3) = 4 \land \texttt{eval}\ (\texttt{NAT}\ 4) = 7$ . Burst then checks if this assumption is correct. It clearly does not because $\texttt{eval}\ (\texttt{NAT}\ 3) = 3$ and $\texttt{eval}\ (\texttt{NAT}\ 4) = 4$ . Then, it re-attempts synthesis with the following strengthened specification.

$$ \texttt{eval}\ i_1 = 1 \land \texttt{eval}\ i_2 = 4 \land \texttt{eval}\ i_3 = 7 \land \texttt{eval}\ (\texttt{NAT}\ 3) = 4 \land \texttt{eval}\ (\texttt{NAT}\ 4) = 7. $$

After the search within a bounded space, Burst fails to find a program that satisfies the strengthened specification. It concludes that the assumption made in the previous step is incorrect. Then, it adds the negation of the assumption (i.e., $ \neg (\texttt{eval}\ (\texttt{NAT}\ 3) = 4 \land \texttt{eval}\ (\texttt{NAT}\ 4) = 7 )$ ) into the list of anti-specifications, searches for a program that does not violate the anti-specifications and generates the following program.

Obviously, this program also does not satisfy the original specification. However, it does not violate the anti-specification because the above program is correct assuming $\texttt{eval}\ (\texttt{NAT}\ 3) = 2 \land \texttt{eval}\ (\texttt{NAT}\ 4) = 5$ . Then, it re-attempts synthesis with the following strengthened specification.

\[\begin{array}{c}\texttt{eval}\ i_1 = 1 \land \texttt{eval}\ i_2 = 4 \land \texttt{eval}\ i_3 = 7 \land \neg (\texttt{eval}\ (\texttt{NAT}\ 3) = 4 \land \texttt{eval}\ (\texttt{NAT}\ 4) = 7) \\\land \texttt{eval}\ (\texttt{NAT}\ 3) = 2 \land \texttt{eval}\ (\texttt{NAT}\ 4) = 5\end{array}\]

Again, after a bounded search, Burst fails to find a program that satisfies the strengthened specification. It concludes that the assumption made in the previous step is incorrect. Then, Burst refutes the assumption, increasing the list of anti-specifications. In this manner, Burst initially overapproximates the specification of the target function and then refines it by repeatedly adding anti-specifications. However, because the space of possible anti-specifications is too large, this method is not effective.

6.6 Sensitivity to the quantity and quality of examples

In this section, we compare Trio with SyRup (Yuan et al., Reference Yuan, Radhakrishna and Samanta2023) in terms of the number of input-output examples (chosen randomly) required for synthesizing the desired programs. The goal of this experiment is to confirm how the quantity and quality of examples affects the performance of Trio by comparing it with SyRup which aims to synthesize generalizable programs from a few examples by leveraging a version space algebra.

Setup.

We use the same benchmark set as the one used for evaluating SyRup, which consists of 43 benchmarks. This set is a subset of the benchmarks used in our previous experiments with the 20 newly added benchmarks and two trivial benchmarks (bool_always_true and bool_always_false) excluded.Footnote ¹⁰ For each benchmark, we generate 10 sets of random input–output examples with sizes ranging from 1 to 8. Every set includes the base case for the recursive programming task (i.e., the example(s) with the smallest input) because SyRup is known to perform better with the base case according to the paper of SyRup.

Table 5 summarizes the results. For each size of example set, we report the success rate, average synthesis time, and the number of timeouts for each tool. The success rate is defined to be the ratio of the number of successful synthesis trials (i.e., the number of desired programs found) to the number of trials (i.e., the number of example sets multiplied by the number of benchmarks, which is 430 in this case). SyRup performs better than Trio in terms of success rate when the number of examples is small (1–4). In terms of efficiency, Trio is faster than SyRup as shown in the average synthesis time and the number of timeouts. However, when the number of examples increases (5–8), Trio consistently outperforms SyRup in both success rate and efficiency. We observe that SyRup suffers from the scalability issue as the number of examples grows. SyRup’s lower success rate is due to the fact that it often times out before finding the desired program when the number of examples is large. This is because SyRup performs computationally expensive version space intersections, which become more expensive as the number of examples increases. This performance degradation can be observed in terms of memory usage as well. The average memory usage of SyRup is 29MB when handling a single example and scales up to 131.6MB when handling 8 examples. In contrast, Trio’s memory usage is much lower, starting at 16.4MB for 1 example and growing gradually to 22.1MB with 8 examples.

Table 5.

Comparison of Trio and SyRup on the 43 benchmarks with random input-output examples. Each row represents the results for a different number of examples. “Succ. Rate” gives the success rate of each tool. “Avg Time” shows the average synthesis time for successful trials. “# T/O” denotes the number of time-outs

Figure 9 shows more detailed results for 12 chosen benchmarks. We present the success rate of each tool for each benchmark. The x-axis label in each plot indicates the number of examples, and the y-axis label indicates the success rate. There are three cases: (1) the two tools show similar success rates (bool_band, bool_neg, bool_xor, and list_sort_sorted_insert), (2) Trio consistently outperforms SyRup irrespective of the number of examples (nat_max, list_rev_tailcall, tree_binsert, tree_count_nodes, and tree_preorder), and (3) SyRup outperforms Trio when the number of examples is small but SyRup is comparable to or worse than Trio when the number of examples is large (nat_add, nat_iseven, and list_append). The lower success rates of SyRup is due to the version space intersection, which becomes more expensive as the number of examples increases. In contrast, Trio’s performance remains stable or even improves as the number of examples grows because more examples can resolve the ambiguity in the search space. In conclusion, SyRup cannot enjoy the benefits of more examples that can resolve the ambiguity in the search space because of the computational cost of version space intersection.

Fig. 9.

Success rates of Trio and SyRup for 12 chosen benchmarks for different numbers of examples (1–8). The x-axis label indicates the number of examples, and the y-axis label indicates the success rate. The plots for the other 31 benchmarks are available in the appendix.