CS703: PROJECT GUIDELINES 1

Logistics: Project Most important part of the course Teams of 1 or 2 people Expectations commensurate with size of team Deliverables Oct 2: me a list of 3 project ideas Oct 16: Project proposal Nov 4: Description of progress 1 Nov 19: Description of progress 2 Dec 7-14:Presentation to the class Dec 15:Final write-up 2

How do I choose my project? I really want you to work on something you like! Ideally something that intersects with your research interests Talk to your advisor and explain what the class is about Ask for suggestions: “Is there a verification or synthesis problem related to our research?” If you need suggestions on papers to read based on a topic ask me … if you really cannot find anything I can help you and guide you towards some projects 3

Verification projects You’ll have a portfolio of program verification techniques plenty of open-source and freely available verification tools (NuSMV, Z3, Boogie, Uppaal) If you also took CS704, you can combine all you have learned Possible domains Networks, GPU code, hardware binary, XML queries, CSS, GUIs, … Think outside the box… Example1: automatically detect whether an Android app can ever render 2 images on top of each-other. Example2: is CSS file 1 equivalent to CSS file 2. Which one is faster? It doesn’t have to use what we used in class verbatim 4

Theory projects Least recommended (might be hard to get deliverables) We are seeing many models, logics, and techniques in this course Many have open-problems, some of which are (might be) approachable Example: an algorithm for learning symbolic automata over the theory of BDDs Example: prove complexity lower-bound for some problem related to automata/verification (if you want to work on this type of problem, come and talk to me) Example: analyze complexity of synthesis questions Example: combine models together and analyze algorithms (e.g. symbolic MSO) 5

AN OVERVIEW OF PROGRAM SYNTHESIS Enough info so that you can think about a project 6

What is program synthesis? 7 User intent Domain knowledge Program can only use: Length(x), if(x)then y else z, x[i],… Synthesizer Input Output Program 1 -> > > 500 Function f(x){ If(length(x)<3) return 0 Else return x[0]+`00’ }

For every synthesis problem… How do you tell the system what you want? What is the specification formalism What is the interaction model What happens when there is ambiguity How do you represent domain knowledge? How do you guide the system towards relevant programs How can you take advantage of the structure of the space of programs you care about How does the system find the program you actually want? And how does it know it is the program you want

Different types of synthesis Inductive Synthesis Functional Synthesis Reactive Synthesis Quantitative Synthesis

Traditional Machine Learning  Learn a function from a set of examples  Scalability is very important, algorithms must scale to millions of data points  Data is assumed to be noisy;  need to avoid overfitting  Space of possible functions is highly stylized  Background knowledge incorporated as preprocessing and feature selection Inductive Synthesis  Learn a function from a set of examples  Scalability is not so important, usually we are dealing with small numbers of examples  Data is assumed to be clean  It’s annoying when user says f(x)=y and the system assumes the user is wrong and decides that f(x)=z  Space of possible functions can be arbitrary  Background knowledge encoded in the description of the space and in the search itself Inductive Synthesis Synthesize a program whose behavior satisfies a set of examples

Programming by Example: Motivation Two major criticisms of synthesis: It’s too hard to make it work Even if it works, it ends up being too hard to use Students and Teachers Algorithm Designers Software Developers End-Users Most Useful Target (logics, automata, etc.) (Examples!)

FlashFill: a feature of Excel 2013 (Sumit Gulwani et al.)

Different types of synthesis Inductive Synthesis Functional Synthesis Reactive Synthesis

Functional Synthesis Goal: Synthesize a function that satisfies a specification How do we know the specification has been satisfied? Isn’t verification itself already quite hard? What is the relevant space of functions? How do we explore this space efficiently?

Example:Least Significant Zero Bit  Trick: Adding 1 to a string of ones turns the next zero to a 1 i.e = int W = 32; bit[W] isolate0 (bit[W] x) { // W: word size bit[W] ret = 0; for (int i = 0; i < W; i++) if (!x[i]) { ret[i] = 1; return ret; } }

Space of possible implementations /** * Generate the set of all bit-vector expressions * involving +, &, xor and bitwise negation (~). * the bnd param limits the size of the generated expression. */ generator bit[W] gen(bit[W] x, int bnd){ assert bnd > 0; if(??) return x; if(??) return ??; if(??) return ~gen(x, bnd-1); if(??){ return {| gen(x, bnd-1) (+ | & | ^) gen(x, bnd-1) |}; }

Example: Least Significant Zero Bit generator bit[W] gen(bit[W] x, int bnd){ assert bnd > 0; if(??) return x; if(??) return ??; if(??) return ~gen(x, bnd-1); if(??){ return {| gen(x, bnd-1) (+ | & | ^) gen(x, bnd-1) |}; } bit[W] isolate0sk (bit[W] x) implements isolate0 { return gen(x, 3); }

Different types of synthesis Inductive Synthesis Functional Synthesis Reactive Synthesis

while(true){ read inputs; make decisions; update state; write outputs; } Reactive Synthesis EnvironmentSystem Inputs Outputs

Reactive Synthesis Challenge How do we specify correct behavior? We now have to relate infinite sequences of inputs to infinite sequences of outputs. What if there is feedback between the system and environment Often the goal of the system is to control the environment Very nice theory around finite state systems

Game based formalisms for Reactive Synthesis For every move of the adversary (ever action of the environment) the synthesized program must make a counter-move that keeps the system working correctly. The game can be modeled with an automaton

Different types of synthesis Inductive Synthesis Functional Synthesis Reactive Synthesis Quantitative Synthesis

Boolean specifications are often insufficient They can be too hard to write Fail to capture important properties of the problem What do we want to optimize? A quantity that is inherent to the program e.g. lines of code A quantity that depends on the input e.g. running time of the program In the worst case? The best case? On average? On expected value? For what distribution?

WHAT DO THEY ALL HAVE IN COMMON 24

For every synthesis problem… How do you tell the system what you want? What is the specification formalism What is the interaction model What happens when there is ambiguity How do you represent domain knowledge? How do you guide the system towards relevant programs How can you take advantage of the structure of the space of programs you care about How does the system find the program you actually want? And how does it know it is the program you want

How do you tell the system what you want? (Input, output) examples Sketched program LTL formula for reactive system One of the above + optimization criteria 26

How do you represent domain knowledge? A grammar of possible operations (you can only use substring, …) Generator: A program that can generate programs (see Sketch) Every reactive controller will be a game automaton 27

How does the system find the program you actually want? Enumeration-based search Requires quick way to discard wrong solutions Requires pruning techniques (don’t enumerate redundant functions) Constraint-based search Encode the search problem into a constraint in a logic Solve the constraint For example Sketch encodes everything in a constraint over the integers Stochastic search Randomly pick next program to search based on heuristic Requires good heuristics Nice algorithms for the automata case in reactive synthesis 28

EXAMPLE APPLICATIONS 29

Inductive synthesis Spreadsheet transformations in excel They are typically small programs but tricky to write Parsers Very natural way to provide examples (highlight part of text you want to parse with some nonterminal) A lot of domain knowledge to exploit Small recursive functional programs Examples guide the search (each suffix is an example itself) 30

Functional synthesis Superoptimizers (via Sketch) They are small programs Operate over (large) finite domains Query extractions (via Sketch) Extract SQL query from Java code Very specific theory for relations Program repair and automatic grading Small edit should turn the program into a correct one Wrong program provides a hopefully close to correct solution 31

Reactive synthesis A theory by itself… Can only be used by domain experts But result is guaranteed to be correct BUS controllers in hardware 32