Sketched Answer Set Programming

Answer Set Programming (ASP) is a powerful modeling formalism for combinatorial problems. However, writing ASP models is not trivial. We propose a novel method, called Sketched Answer Set Programming (SkASP), aiming at supporting the user in resolving this issue. The user writes an ASP program while marking uncertain parts open with question marks. In addition, the user provides a number of positive and negative examples of the desired program behaviour. The sketched model is rewritten into another ASP program, which is solved by traditional methods. As a result, the user obtains a functional and reusable ASP program modelling her problem. We evaluate our approach on 21 well known puzzles and combinatorial problems inspired by Karp's 21 NP-complete problems and demonstrate a use-case for a database application based on ASP.


Introduction
Many AI problems can be formulated as constraint satisfaction problems that can be solved by state-of-the-art constraint programming (CP) (Rossi et al., 2006) or answer set programming (ASP) techniques (Lifschitz, 2008). Although these frameworks provide declarative representations that are in principle easy to understand, writing models in such languages is not always easy.
On the other hand, for traditional programming languages, there has been significant attention for techniques that are able to complete (Lezama, 2013) or learn a program from examples (Gulwani et al., 2015). The idea of program sketching is to start from a sketched program and some examples to complete the program. A sketched program is essentially a program where some of the tests and constructs are left open because the programmer might not know what exact instruction to use. For instance, when comparing two variables X and Y , the programmer might not know whether to use X < Y or X ≤ Y or even X + 1 < Y . By providing a few examples of desired program behaviour, the sketched program can then be completed.
In this paper, we show that sketches can be used in ASP. We develop a method (based on ASP itself) to complete such sketched answer set programs. We also argue that this can be useful when writing ASPs, especially when having already identified the potentially problematic parts of a model. In this regard, SkASP differs from traditional debugger, which can be used to locate where the model is wrong. But typical debuggers provide only little support for repairing such bugs once they have been located. SkASP, in contrast, provides a constructive solution for repairing such bugs or completing partial programs by using examples together with sketched variables.
Intuitively, the program specifies constraints on the relationship between two queens on the rows (first rule), columns (second rule) and diagonals (third rule). Now assume that the user realizes that the program is wrong and he might have been to quick. What sketching allows to do in such cases is to mark the potentially erroneous predicates with question marks, to specify possible alternatives and to provide a few examples of the intended behaviour. Figure 1d describes a possible sketch for this problem. It basically states that it is unknown whether the = is correct in the first two rules, and whether the arithmetic operation is correct. By using a few positive and negative examples, as well as the built-in alternatives for ?= (from its domain =, =, <, >, ≤, ≥) and for ?+ (from arithmetic operations), the SkASP solver will produce the correct answer set program.
Below, in Section 2, we introduce the problem of sketched Answer Set Programming. In Section 3, we show how to transform a sketched ASP into a form that can be solved by a regular ASP solver and analyze its theoretical properties. More specifically, we demonstrate that there is no complexity jump with respect to the complexity of existence of answer sets of a regular ASP program. In Section 4, we evaluate our approach on a set of classical NP-complete problems (inspired by Karp's 21 NP-complete problems). In Section 5, we demonstrate an application of our method to a database system based on ASP. Finally, we discuss related work in Section 6, and conclude in Section 7.

ASP and Sketching
Answer Set Programming (ASP) is a form of declarative programming based on the stable model semantics (Gelfond and Lifschitz, 1988) of logic programming (Lifschitz, 2008). We follow the standard syntax and semantics of ASP as described in the Potassco project (Gebser et al., 2012). A program is a set of rules of the form a ← a 1 , . . . a k , not a k+1 , . . . , not a n . where a is a propositional atom, called a head, and for i between 1 and k, a i is a positive propositional atom and for i between k + 1 and n, a i is a negative propositional atom. All these atoms a i together are referred to as the body. A positive or negative atom is called a literal. A rule of the form a ← . is called a fact and abbreviated as a. and a rule without a head is called an integrity constraint. Conditional literals, written as a : l 1 , . . . , l n , and cardinality constraints, written as c min {l 1 , . . . , l n }c max , are used (l 1 , . . . , l n are literals here, and c min , c max are non negative integers). A conditional atom holds if its condition is satisfied and a cardinality constraint is satisfied if between c min and c max literals hold in it. Furthermore, as ASP is based on logic programming and also allows for variables, denoted in upper-case, the semantics of a rule or expression with a variable is the same as that of its set of ground instances. We restrict the ASP language to the            (c) Sudoku sketch (ASP due to Hölldobler and Schweizer (2014);c stands for cell) [SKETCH] :-q(w,Rw,Cw), q(b,Rb,Cb), Rw ?= Rb. :-q(w,Rw,Cw), q(b,Rb,Cb), Cw ?= Cb. :-q(w,Rw,Cw), q(b,Rb,Cb), Rw ?+ Rb ?= Cw ?+ Cb.
[EXAMPLES] positive: q(w,1,1). q(w,2,2). ... negative: q(b,2,2). q(w,3,1). ...  We extend the syntax of ASP with sketched language constructions. Instead of allowing atoms of the form p(t 1 , ...,t n ), where p/n is a predicate and the t i are terms (variables or constants), we now allow to use sketched atoms of the form ?q(t 1 , ...,t n ) where ?q is a sketched predicate variable with an associated domain d q containing actual predicates of arity n. The meaning of the sketched atom ?q(t 1 , ...,t n ) is that it can be replaced by any real atom p(t 1 , ...,t n ) provided that p/n ∈ d q . It reflects the fact that the programmer does not know which p/n from d q should be used. Sketched atoms can be used in the same places as any other atom.
We also provide some syntactic sugar for some special cases and variants, in particular, we use a sketched inequality X ?= Y , a sketched arithmetic operator X ?+ Y , 1 and sketched negation ?not p(X). The domain of X ?= Y is the set {=, =, <, >, ≥, ≤, }, where is the atom that is always satisfied by its arguments, the domain of X ?+ Y is the set {+, −, ×, ÷, dist} where dist(a, b) is defined as |a − b|, and the domain of ?not is { / 0, not}. An example of sketched inequality can be seen in Line 3 of Figure 1c, examples of sketched predicates and negation in Line 4 of Figure 1a and sketched arithmetic in Line 5 of Figure 1d.
A sketched variable is a sketched predicate, a sketched negation, a sketched inequality or a sketched arithmetic operator. We refer to the set of all sketched variables as S. Predicate p directly depends on predicate q iff q occurs in the body of a rule with p in the head or p is a sketched predicate and q is in its domain. We say that p depends on q iff q is in the transitive closure of all predicates on which p directly depends. An example is a set of ground atoms.
A preference is a function from S to Z. A substitution θ is preferred over θ given preferences f iff for all s i → d i ∈ θ and s i → d i ∈ θ it holds that f (d i ) ≥ f (d i ) and at least one inequality is strict.
Definition 1 (Sketched Answer Set Problem (SkASP)) Given a sketched answer set program P with sketched variables S of domain D and preferences f , and positive and negative sets of examples E + and E − , the Sketched Answer Set Problem is to find all substitutions θ : S → D preferred by f such that Pθ ∪ e has an answer for all e in E + and for no e in E − . The decision version of SkASP asks whether there exists such a substitution θ . E.g., Figure 1a represents a sketch of Hamiltonian Cycle with P being the rules in Lines 2, 4, 5 and S={?p, ?q, ?not}, It will be useful to distinguish different versions w.r.t. the preferences. First, when f (θ ) is constant, all substitutions are equal and there are no preferences. While this suffices in many cases, it is often useful to express preferences over the substitutions. Because this might impose an extra burden on the user, we also provide default preferences for the built-in sketched variables (like inequality, negation and arithmetic), cf. the experiments in Section 4.

Rewritting Schema
The key idea behind our approach is to rewrite a SkASP problem (P, S, D, f , E + , E − ) into an ASP program such that the original sketching program has a solution iff the ASP program has an answer set. This is achieved by 1) inserting decision variables into the sketched predicates, and 2) introducing example identifiers in the predicates.
The original SkASP problem is then turned into an ASP problem on these decision variables and solutions to the ASP problem allow to reconstruct the SkASP substitution.
The rewriting procedure has four major steps: example expansion, substitution generation, predicate reification and constraint splitting.
Example Identifiers To allow the use of multiple examples in the program, every relevant predicate is extended with an extra argument that represents the example identifier. The following steps are used to accommodate this in the program.
1. Let SP be the set of all predicates that depend on a predicate occurring in one of the examples. 2. Replace each literal p(t 1 , ...,t n ) for a predicate p ∈ SP in the program P by the literal p(E,t 1 , ...,t n ), where E is a variable not occurring in the program. 3. Add the literal examples(E) as a guard to the body of each rule in P. 4. For each atom p(t 1 , ...,t n ) in the i-th example, add the fact p(i,t 1 , ...,t n ) to P. 5. For each positive example i, add the fact positive(i) to P, and for each negative one, the fact negative(i).
As an example the rule in Line 2 of Figure 1a becomes Line 9 of Figure 1b, and the example in Line 16 corresponds to rewritten Line 2.
Substitution Generation We now introduce the decision variables: 1. For each sketched variable s i with domain d i , add the following constraint to P: This constraint ensures that each answer set has exactly one value from the domain assigned to each sketched variable. This results in a one-to-one mapping between decision atoms and sketching substitution θ . An example can be see in Lines 4 and 5 of Figure 1b.

Predicate Reification
We now introduce the reified predicates, 1. Replace each occurrence of a sketched atom s(t 1 , ...,t n ) in a rule of P with the atom reified s(D,t 1 , . . . ,t n ), and add decision s(D) to the body of the rule. 2. For each sketched variable s and value d i in its domain, add the following rule to P: where the first argument represents the decision variable for s.
Thus, semantically reified s(d i , X 1 , . . . , X n ) is equivalent to d i (X 1 , . . . , X n ) and decision s(d i ) indicates that the predicate d i has been selected for the sketched variable s. Notice that we focused here on the general case of a sketched predicate ?p(. . . ). It is straightforward to adapt this for the sketched inequality, negation and arithmetic operator. Examples of reification can be seen in Lines 7 of Figure 1b for the sketched ?q of the sketch in Figure 1a and in Lines 11, 12 for an example of reified negation.
Integrity Constraint Splitting 1. Replace each integrity constraint ← body by: 2. And add the rule to the program: This will ensure that all positives and none of the negatives have a solution. For example, the constraint in Line 5 of Figure 1a is rewritten into a positive constraint in Lines 14, 15 and into a negative in Lines 17, 18.
Another important result is that the preferences do not affect the decision complexity. Proofs can be found in the supplementary materials.
Let us state the key property of this rewriting: Theorem 1 (Sound and Complete Sketched Rewriting) A sketched ASP program (P, S, D, f , E + , E − ) has a satisfying substitution θ iff the rewritten ASP program P has an answer set.
Interestingly, the sketched ASP problem has the same complexity as the original ASP program.
Theorem 2 (Complexity of Sketched Answer Set Problem) The decision version of SkASP is NP-complete.

Proof
Membership. Follows from the encoding of SkASP into a fragment of ASP which is in NP.
Completeness. Let P be an arbitrary answer set program, then the question of whether it has an AS is reduced to asking if an empty θ satisfies the following SkASP: 0}, f ≡ 1 and P = P . If θ satisfies the SkASP, then P and consequently P have an answer set.
Dealing with preferences Preferences are, as we shall show in our experiments, useful to restrict the number of solutions. Currently, we have implemented preferences using a post-processing approach (which will also allow to apply the schema to other formalism such as CP or IDP (Bruynooghe et al., 2013)), in which we first generate the set of all solutions O (without taking into account the preferences), and then post-process O. Basically, we filter out from O any solution that is not preferred (using tests on pairs (o, o ) from O × O). Note that the preferences introduce a partial order on the solutions. In future work, we plan to make the search more effective by using the preferences in the search process (using ASP extensions (Brewka et al., 2015)) or by formulating the SkASP task as an optimisation problem (using built-in ASP constructs).

Experimental Evaluation
For the experimental evaluation we have created a dataset consisting of 21 classical combinatorial problems among which most are NP-complete. Our dataset contains Graph Clique; 3D Matching; Vertex and Edge Coloring; Vertex and Edge Domination; Exact, Vertex and Clique Covers; Sudoku; Latin Square; B&W and n-Queens; Hitting Set; Hamiltonian Cycle; Feedback Vertex and Arc Sets; Radio Frequency Assignment (FA); Set Packing and Splitting; Subgraph Isomorphism. For more detailed dataset description we refer to Appendix A.
All problems, their code, and implementation details, can be found in the accompanying Github repository: https://github.com/SergeyParamonov/sketching Implementation details and limitations. The SkASP engine is written completely in Python 3.4 and requires the library pyasp to work. All examples have been run in a 64-bit Ubuntu 14.04 environment. The current implementation does not support language constructs such as choice rules and aggregates in the rules of sketched ASP.
Sketches. For many problems there are already existing ASP encodings that we sketched and tried to reconstruct, such as Sudoku in Figure 1c from Hölldobler and Schweizer (2014) or Hamiltonian Cycle in Figure 1a from Gebser et al. (2012). For the ones without an ASP encoding, we followed a canonical ASP encoding strategy of generate-and-test to create ASP encodings that should be reconstructed by our method. We use the default preferences in the experimental section for the built-in inequality sketch X ?= Y : namely = and = have equal maximal preference, this preference still implies a partial order. A user can redefine the preferences by giving them weights. Our experiments indicate that for other sketched types no default preferences are needed.
Combinatorial explosion. One might consider a baseline approach that would enumerate all instances of the ASP sketch, and in this way produce one ASP program for each assignment that could then be tested on the examples. This naive grounding and testing approach is, however, infeasible: the number of possible combinations grows exponentially with the number of sketched variables. For example, for the Radio Frequency Problem there are around 100,000 possible assignments to the sketched variables. If we multiply them by the number of examples, around a million ASP programs would have to be generated and tested. This is infeasible in practice.
Experimental questions. We investigate the following experimental questions here: On some other problems, however, after 7 examples there still remain many solutions (on average 18 for problems that do not converge). Figure 2b reports the same information as Figure  2a for all 21 problems: the average on the 21 problems; the average on the 9 that converge within 7 examples, referred to as the easy problems; and the average on the 12 that still have several solutions after 7 examples, referred to as the hard problems. When SkASP does not converge to a unique solution, this leaves the user with choices, often amongst equivalent ASP programs, which is undesirable.
For problems that do not converge after a few examples, we propose to use preferences, as provided by our SkASP framework. We use the default preference setting described earlier: for the built-in inequality sketch X ?= Y = and = have equal maximal preference, all other relations have the minimal one; for other sketched types no preferences are used.
We investigate Q 2 by measuring again the impact of the number of examples on the number of solutions. In Figure 3a, we observe that all problems have converged in less than 7 examples (under default preferences). The impact of preferences on the speed of convergence is even more visible on the whole set of problems, as reported in Figure 2b. The number of solutions with preferences is smaller, and often much smaller than without preferences, whatever the number of examples provided. With preferences, all our 21 problems are learned with 7 examples.
To answer Q 3 , we look into the convergence of FAP by varying the number of sketched variables. The original sketched program of FAP contains 5 sketched variables. We vary it from 2 to 5 by turning 3, 2, 1, or 0 sketched variables into their actual value. As expected, in Figure 3b, we observe that the more there are sketched variables in the SkASP, the slower the number of solutions decreases. Furthermore, the number of sketched variables has a greater impact on the convergence without preferences, as we see in Figure 3c. After 3-5 examples under preferences we have fewer than 10 solutions, while without preferences there are still dozens or hundreds of solutions.
To address Q 4 , we set up the following experiment: we have taken the standard 9 × 9 Sudoku encoding (Hölldobler and Schweizer, 2014) and varied the number of sketched inequalities in it. We then run it against randomly generated positive and negative examples (in equal quantities). The outcome is depicted in Figure 4a. We observe that the runtime depends on the number of sketched variables, i.e. the runtime exhibits a linear behavior with the slope determined by the number of sketched variables. To measure the effect of the examples size, i.e. the number of ground facts in a single example, we set up the following experiment: for the n × n Latin Square In Figure 5b we see that while the encodings show certain variability in the runtime, they demonstrate a similar growing runtime pattern and there is no significant (e.g., of an order of magnitude) difference in the runtime exhibited by the models. All models needed under 3 seconds (on average for 100 examples) to learn the constraints. In Figure 5a, we observe a similar situation: while there is a certain variation in the number of solutions, they demonstrate similar behavior. Of course, since the examples are random and are not designed to be diverse, we do not have fast convergence as in Figures 3a and 2b.
Overall, we observe that only few examples are needed to converge to a unique or a small group of equivalent solutions (provided that examples are diverse). An example where such equivalent 2 http://www.cs.uni-potsdam.de/~torsten/Lehre/ASP/Folien/asp-handout.pdf http://www.hakank.org/answer_set_programming/nqueens.lp https://www.cs.nmsu.edu/~tson/tutorials/asp-tutorial.pdf 3 While conducting this experiment, we stumbled upon a peculiar bug. One encoding worked mostly by pure luck.
The following constraint :-queen(X1,Y1), queen(X2,Y2), X1 < X2, abs(Y1 -X1) == abs(Y2 -X2). works because abs is not actually absolute value but is an uninterpreted function, it essentially checks that iff X == Y , and that is indeed the solution found by our system. We have contacted the author and the model has been updated. solutions are found is the edge coloring problem, where two equivalent (for undirected graphs) constraints are found: These two syntactically different constraints are both equivalent for this problem and cannot be differentiated by any valid example. In the experiments (with manually created examples) we have only two such cases: the one described above and n-queens with three symmetric solutions under the default preferences. An interesting observation we made on these experiments is that the hardness (e.g., in terms of runtime) of searching for a solution of a problem is not directly connected to the hardness of learning the constraints of this problem. This can be explained by the fact that the search spaces are incomparable. The SkASP searches through the search space of sketched variables, which is usually much smaller than the search space of the set of decision variables of the problem to learn.

Use-case: database query completeness
To illustrate SkASP on a real application of ASP, we consider the ASP solution for reasoning about query completeness in databases, which is related to data quality issues (Nutt et al., 2015;Razniewski and Nutt, 2011). According to Nutt et al. (2013), the problem of query completeness is to determine whether a conjunctive query (i.e., a positive Datalog rule) over a partially complete database (a set of ground facts) can be answered completely. Table completeness rules (i.e., positive Datalog rules) specify which parts of a database are complete. The problem then is to decide whether these rules allow to retrieve exactly the same answers as if the database had complete information.
We now consider an ASP implementation, called MAGIK (Savkovic et al., 2012), for reasoning about query completeness, which also uses various schema constraints (Nutt et al., 2015), 4 and which generates ASP code that allows to decide whether the query will be complete. More formally, we consider the following notion of query completeness: a query completeness statement T is a rule of the form Q T (X) ← R(X), G, where G is a conjunction of atoms. We denote as T (D) the set of ground facts {R(x) | T ∪ D |= Q T (x)}, i.e. the rule filters the database and outputs the facts that are complete. A database D satisfies a foreign key f k if and only if for any fact d it holds that D |= d ⇐⇒ D ∪ f k |= d. Let Q be a conjunctive query, T be a set of table completeness rules and S be the set of schema constraints. Q is complete wrt to S and T if and only if for any database D satisfying S it holds that Q(D) = Q( T ∈T T (D)).
Essentially, the MAGIK system generates three sets of ASP constraints: schema constraints, table completeness (TC) rules and the query completeness check. The fine grained details are not important here but the general idea is the following: a lot of TC rules might be accumulated over time and entered into the system by a database administrator (see e.g., Lines 6-15 in Figure 6), the query itself may be complicated and error-prone (see Line 17), whereas schema constraints typically do not change over time (see Lines 3 and 4).
We use the following scenario to illustrate SkASP: assume that a user obtained the ASP code via the MAGIK tool, including many TC rules, and that she is not aware of which ones are still valid. Furthermore, assume that the user does not fully understand a certain detail in the generated query check. What the user knows, however, is that the query is not complete without the presence of the foreign key constraints. Let us illustrate this scenario on a school schema and an accompanying conjunctive query: We encode uncertainty over which TC rule applies by creating a sketched constant ?true that can be mapped to { , ⊥} for each rule and appending it to each TC statement as a propositional atom (Lines 6-15 in Figure 6). The way we specify uncertainty in the query is similar to the examples before: we state that a certain atom, for instance an equality atom, is sketched (Line 17). The schema constraints are encoded as normal rules with the activation guards that depend on the examples (Lines 3 and 4). Then, we introduce a preference of ⊥ over for the sketched ?true. Running SkASP on the encoding gives us both a minimal set of TCs covering the examples (five combinators are produced), and the assignment for the query (only one satisfies the examples).
This example shows how SkASP can be used in a fairly general setting as a useful autocompletion tool that allows to synthesize a correct combination of rules. If a user had to do this manually, he would have to go through a painful search process whereby numerous combinations would potentially have to be tested out.

Related Work
The problem of sketched ASP that we introduced is related to a number of topics. First, the idea of sketching originates from the area of programming languages, where it relates to so called selfcompleting programs (Lezama, 2013), typically in C (Lezama, 2009)

19
[EXAMPLES] 20 negative: p(i,n,c). s(i,cs,mi).  , where an imperative program has a question mark instead of a constant and a programmer provides a number of examples to find the right substitution for it. While sketching has been used in imperative programming languages, it has -to the best of the authors' knowledge -never been applied to ASP and constraint programming. What is also new is that the sketched ASP is solved using a standard ASP solver, i.e., ASP itself. Second, there is a connection to the field of inductive (logic) programming (De Raedt, 2008;Muggleton and De Raedt, 1994;Gulwani et al., 2015) as one induces programs from examples of their intended behaviour. Particularly interesting in this regard is also the work on metainterpretive learning (Muggleton et al., 2014 where a Prolog or Datalog program is learned from a set of higher-order rules and examples. The higher-order rule act as a kind of template that can be used to complete the program. However, meta-interpretive learning differs from SkASP in that it induces full programs and pursues as other inductive logic programming approaches in that it pursues a search-and a trace-based approach guided by generality, whereas SkASP using a constraint solver (i.e., ASP itself) directly. Furthermore, the target is different in that ASPs are learned (which include constraints) rather than Prolog or Datalog programs. On the other hand, meta-interpretive learning also supports predicate invention.
There also exists interesting work in the intersection of ILP, program synthesis and ASP, e.g. (Law et al., 2014), where an ASP program is learned from positive and negative examples, and a set of modes, while minimizing a metric through a search process. The program, learned completely from scratch, is not necessarily the best program from the user's point of view and may limit the possibility to localize the uncertainty from the user's knowledge of the problem.
Third, there is also work on constraint learning, where a number of systems such as CONACQ (Bessiere et al., 2005(Bessiere et al., , 2017 and QUACQ (Bessiere et al., 2013) learn a set of propositional constraints, and ModelSeeker (Beldiceanu and Simonis, 2012) learns global constraints governing a particular set of examples. The subject has also been investigated in an inductive logic programming setting by (Lallouet et al., 2010). However, the idea in all these approaches is to learn the complete specification of the constraint satisfaction problem from scratch, and hence, these approaches do not take a debugging perspective. The present setting is probably more realistic from a user perspective as it allows to use the knowledge that the user no doubt possesses about the underlying CSP problem, and also requires much less examples. On the other hand, SkASP also makes, as other sketching approaches, the strong assumption that the intended target program is an instance of the sketched one. This may not always be true, for instance, when rules are missing in the program. This is also an interesting issue for further research.
Fourth, our approach is related to the line of research in debugging of ASP (Gebser, Pührer, Schaub, and Tompits, Gebser et al.;Oetsch et al., 2011). Unlike SkASP such debuggers can be used to locate the bug, but typically do not provide help for fixing the bugs. On the other hand, once the bug has been identified SkASP could be used to repair the bug by introducing a sketch and a number of examples. The approach of Li et al. (2015), contrary to our approach, is based on classical ILP techniques of generalization and specification and does not provide the freedom to indicate uncertain parts of the program.
Finally, from the technical point of view our work bears certain technical similarities with meta-interpretation in ASP (Eiter and Polleres, 2006), such as rule and decision materialization. The purpose of their work is, however, different: they aim at synthesizing a new program of higher complexity (Σ P 2 ) given two programs of lower complexity (NP and Co-NP).

Discussion and Conclusions
Our contribution is four-fold: we have introduced the problem of sketched ASP; we have provided a sound and complete rewriting schema for sketched ASPs; we showed that the complexity of the rewriting is the same as that of the existence of an AS; we have create a dataset of sketched problems and evaluated our approach empirically demonstrating its efficiency and effectiveness. User interaction is an interesting direction for future work, namely to suggest constraints and examples. For the former, if we are not able to reject a negative example, we can construct a constraint that would reject the negative examples and none of the positive examples. As for the examples, if we have two solutions to a problem, we can generate an example discriminating between them and ask user to clarify it, while this might not always be possible, since symmetric assignments might lead to semantically identical programs. In practice, however, this might be an important addition to simplify sketching for end users. Another interesting direction is to incorporate preference handling into the ASP model using the extensions of ASP for preference handling, such as asprin (Brewka et al., 2015).