Download presentation

Presentation is loading. Please wait.

Published byKiya Madson Modified about 1 year ago

1
Dimensions in Synthesis Part 2: Applications (Intelligent Tutoring Systems) Sumit Gulwani sumitg@microsoft.com Microsoft Research, Redmond May 2012

2
Domain Insight Bit-vector Algorithms Geometry Constructions Testing & Verification Symbolic verification runs in reasonable time Testing is probabilistically sound Synthesis Strategy Counter-example guided inductive synthesis Brute-force search: Generate and Test 1 Recap

3
Students and Teachers End-Users Algorithm Designers Software Developers Most Transformational Target Potential Users of Synthesis Technology 2 Most Useful Target Vision for End-users: Enable people to have (automated) personal assistants. Vision for Education: Enable every student to have access to free & high-quality education.

4
Motivation –Online learning sites: Khan academy, Edx, Udacity, Coursera Increasing class sizes with even less personal attention –New technologies: Tablets/Smartphones, NUI, Cloud Various Aspects –Solution Generation –Problem Generation –Automated Grading/Feedback –Content Entry Various Domains –K-12: Mathematics, Physics, Chemistry –Undergraduate: Introductory Programming, Automata Theory –Language Learning 3 Intelligent Tutoring Systems

5
4 Joint work with: Cerny, Henzinger, Radhakrishna, Zufferey

6
5 Classic Problem in Automata Theory Course Solution Generation Engine Let L be the language containing all strings over {a,b} that have the same number of occurrences of “ab” as occurrences of “ba”. Construct an automata that accepts L, or prove that L is non-regular. “Regular”,

7
6 Classic Problem in Automata Theory Course Solution Generation Engine Let L be the language containing all strings over {a,b} that have the same number of occurrences of “a” as occurrences of “b”. Construct an automata that accepts L, or prove that L is non-regular. “Non-regular”,

8
Formal Description of Input Regular Languages –Algorithm for automata synthesis. Non-regular Languages –Formal description of non-regularity proof. –Algorithm for proof synthesis. 7 Outline

9
Problem Description Languages Examples User-friendly logic Context-free grammar Interfaces Membership Test –Required for inductive synthesis of automata or non-regularity proof. Symbolic Membership Test –Required for verification of non-regularity proof. 8 Formal Description of Input

10
9 User-friendly Logic

11
Formal Description of Input Regular Languages Algorithm for automata synthesis. Non-regular Languages –Formal description of non-regularity proof. –Algorithm for proof synthesis. 10 Outline

12
11 Automata Generation for Regular Languages

13
12 Distribution of Automata sizes The automata size of educational problems is small!

14
13 Distribution of counterexample lengths (relative to automata size) The counterexample lengths are quite smaller than worst-case possibility of twice the automata size.

15
Formal Description of Input Regular Languages –Algorithm for automata synthesis. Non-regular Languages Formal description of non-regularity proof. –Algorithm for proof synthesis. 14 Outline

16
15 Myhill-Nerode Theorem: Non-regularity Condition

17
16 Examples of Congruence/Witness Functions

18
17 Language for Congruence/Witness Functions

19
Formal Description of Input Regular Languages –Algorithm for automata synthesis. Non-regular Languages –Formal description of non-regularity proof. Algorithm for proof synthesis. 18 Outline

20
19 Algorithm for constructing Congruence/Witness Fns.

21
20 Experimental Results IdPDLGeneration Congruence Fn.Witness Fn. TestVerify 1UFPDL50.1 2UFPDL50.20.1 3UFPDL50.90.1 3CFG50.40.1 4UFPDL51.60.10.2 5CFG100.20.1 6CFG50.1 9CFG100.1 11UFPDL812.10.40.1 12UFPDL510.10.40.2 13UFPDL56.40.20.1 15UFPDL77.81.10.1 15CFG70.1 16CFG50.20.1 17UFPDL51.60.40.1 18CFG50.30.1 19UFPDL1015.84.51.7 21CFG50.50.1

22
21 Intelligent Tutoring Systems AAAI 2012: Singh, Gulwani, Rajamani.

23
22 Trigonometry Problem

24
23 Algebra Problem Generation Example Problem Query Generation Query Query Execution New Problems Query Refinement Results OK? Refined Query No Yes Similar Problems

25
24 Limits/Series Problem

26
25 Integration Problem

27
26 Determinant Problem

28
27 Intelligent Tutoring Systems Arxiv TR 2012: Rishabh Singh, Gulwani, Armando Solar-Lezama.

29
28 Background: PexForFun

30
using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 29 Buggy Program for Array Reverse 6:28::50 AM

31
using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length-1; i < a.Length-1; i--) { b[count] = a[i]; count++; } return b; } 30 Buggy Program for Array Reverse 6:32::01 AM

32
using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length-1; i < a.Length-1; i--) { b[count] = a[i]; count++; } return b; } 31 Buggy Program for Array Reverse 6:32::32 AM No change! Sign of Frustation?

33
using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i <= a.Length; i--) { b[count] = a[i]; count++; } return b; } 32 Buggy Program for Array Reverse 6:33::19 AM

34
using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) {Console.Writeline(i); b[count] = a[i]; count++; } return b; } 33 Buggy Program for Array Reverse 6:33::55 AM Same as initial attempt except Console.Writeline!

35
using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) {Console.Writeline(i); b[count] = a[i]; count++; } return b; } 34 Buggy Program for Array Reverse 6:34::06 AM No change! Sign of Frustation?

36
using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i <= a.Length; i--) {Console.Writeline(i); b[count] = a[i]; count++; } return b; } 35 Buggy Program for Array Reverse 6:34::56 AM The student has tried this before!

37
using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 36 Buggy Program for Array Reverse 6:36::24 AM Same as initial attempt!

38
using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length-1; i < a.Length-1; i--) { b[count] = a[i]; count++; } return b; } 37 Buggy Program for Array Reverse 6:37::39 AM The student has tried this before!

39
using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i > 0; i--) { b[count] = a[i]; count++; } return b; } 38 Buggy Program for Array Reverse 6:38::11 AM Almost correct! (a[i-1] instead of a[i] in loop body)

40
using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i >= 0; i--) { b[count] = a[i]; count++; } return b; } 39 Buggy Program for Array Reverse 6:38::44 AM Student going in wrong direction!

41
using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 40 Buggy Program for Array Reverse 6:39::33 AM Back to bigger error!

42
using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 41 Buggy Program for Array Reverse 6:39::45 AM No change! Frustation!

43
using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 42 Buggy Program for Array Reverse 6:40::27 AM No change! More Frustation!!

44
using System; public class Program { public static int[] Puzzle(int[] a) { int[] b = new int[a.Length]; int count = 0; for(int i=a.Length; i < a.Length; i--) { b[count] = a[i]; count++; } return b; } 43 Buggy Program for Array Reverse 6:40::57 AM No change! Too Frustated now!!! Gives up.

45
Provides additional value over counterexample feedback. More friendly feedback. –Helpful for students who give up after several tries (with only counterexample feedback). Grading –Counterexample feedback does not distinguish between a slightly incorrect solution and one that is very far off from being correct. 44 Proposal: Semantic Grading

46
45 Demo

47
Simplifying Assumptions Correct solution is known. Errors are predictable. Programs are small. Challenging Aspects No logical specification. –Instead program equivalence. Higher density of errors than production code. Generate multiple fixes –To remain faithful to the student’s thought process. 46 Relation with Automated Bug Fixing

48
Teacher provides: –A reference implementation –Model of errors that students make Sketch encoding: –Fuzz the program using error model. –Use a counter to keep track of number of changes. Sketch solving: –To generate minimal fixes, assert (counter = i) for increasing values of i. –Use off-the-shelf SAT solver to explore the state space of possible corrections. 47 Technique

49
Array Index Fuzzing: v[a] -> v[{a+1, a-1, v.Length-a-1}] Initialization Fuzzing: v=n -> v={n+1, n-1, 0} Increment Fuzzing: v++ -> { ++v, v--, --v } Return Value Fuzzing: return v -> return ?v Conditional Fuzzing: a op b -> a’ ops { a+1, a-1, 0 } where ops = {, =, ==, != } 48 Error Model for Array Reverse Problem

50
49 Effectiveness of Error Models

51
50 Efficiency of Error Models

52
51 Generality of Error Models

53
BenchmarkTotalFixedChangesTime(s) Array Reverse3052541.732.69 String Palindrome86641.283.52 Array Maximum99681.257.47 Is Increasing Order51381.723.56 Array Sort74351.1732.46 Factorial70301.254.99 Friday Rush174112.42 52 Experimental Results

54
53 Potential Workflow Teacher grades an ungraded answer-script. System generalizes corrections into error models. System performs automated grading by considering all possible combinations and instantiations of all error models. Any ungraded answer scripts? Yes No

55
54 Intelligent Tutoring Systems Joint work with: Alex Polozov and Sriram Rajamani

56
State-of-the-art Mathematical Editors Text editors like Latex –Unreadable text in prefix notation WYSIWIG editors like Microsoft Word –Change of cursor positions multiple times. –Back and forth switching between mouse & keyboard Our proposal: An intelligent predictive editor. Mathematical text has low entropy and hence amenable to prediction! 55 Mathematical Intellisense

57
Terms connected by the same AC operator can be thought of as terms belonging to a sequence. There are 2 opportunities for predicting such terms. Sequence Creation: T 1, T 2, T 3, … Learn a function F such that F(T i ) = T i+1 Sequence Transformation: T 1, T 2, T 3, T 4 -> S 1, S 2, … Learn a function F such that F(T i ) = S i 56 Reducing (Term) Prediction to Learning-By-Examples

58
57 Mathematical (Syntactic) Intellisense

59
58 Mathematical (Semantic) Intellisense

60
Long-term Goals Ultra-intelligent computer Model of human mind Inter-stellar travel 59

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google