Tutorial Pex4Fun: Teaching and Learning Computer Science via Social Gaming Nikolai Tillmann, Jonathan de Halleux, Judith Bishop, Michal Moskal Microsoft Research Tao Xie NCSU

Pex4Fun: Teaching and Learning Computer Science via Social Gaming Agenda Background: Code Analysis by Dynamic Symbolic Execution Writing Code in a Browser Coding Duels Social Experience Teaching Platform Conclusion

DYNAMIC SYMBOLIC EXECUTION IN THE CLOUD Pex – The Pex4Fun Engine – History

Before the cloud…

void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == ) throw new Exception("bug"); } a.Length>0 a[0]==123… T F T F F a==null T Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]== Input null {} {0} {123…} Execute&Monitor Solve Choose next path Observed constraints a==null a!=null && !(a.Length>0) a==null && a.Length>0 && a[0]!= a==null && a.Length>0 && a[0]==  Generates test data systematically Done: There is no path left. Dynamic Symbolic Execution in Pex

Dynamic Symbolic Execution Exercises ArrayIndexLength Pex knows about all implicit, exception- throwing control-flow branches ArrayHeap Pex models the heap Assert, Assert123 Assertions connect code coverage and correctness

Limitations 16K characters of code (single file / editor window) * Single-threaded only No environment interactions * No non-determinism * Try to avoid floating-point computations * Limitation only applies to pex4fun.com, but not Pex tool

Dynamic Symbolic Execution Summary “Ask Pex” sends code to cloud Code is compiled and analyzed in cloud Dynamic Symbolic Execution automatically finds relevant interesting test inputs that achieve high code coverage Results are shown in browser

WRITING CODE IN A BROWSER Code Auto-Completion

Auto-Completion

Writing Code in Browser Exercise 1.Go to 2.Click “New” 3.Write some code with public static “Puzzle” method using System; class Vector { int x, y; public Vector(int x, int y) { this.x = x; this.y = y; } public int Length() { return (int)Math.Sqrt(x*x+y*y); } public Vector Normalize() { return new Vector(x/Length(), y/Length()); } public bool Equals(Vector other) { return this.x==other.x && this.y == other.y; } } public class Program { public static void Puzzle(Vector v) { var n1 = v.Normalize(); var n2 = n1.Normalize(); if (!n1.Equals(n2)) throw new Exception(“bug in normalize”); } } 4.“Ask Pex!”

Writing Code in Browser Summary Code is compiled in cloud Code is executed in cloud Auto-completion via cloud/Javascript in browser

CODING DUELS Fun and Engaging Serious Game – Win by Writing Code

Coding Duels

Pex computes “semantic diff” in cloud  code written in browser vs.  secret reference implementation You win when there are no differences secret

Coding Duels class VisibleProgram { public static int Puzzle1(int x) { return x; } class SecretProgram { public static int Puzzle2(int x) { return x * x; } public class MetaProgram { public static void Puzzle(int x) { if (VisibleProgram.Puzzle1(x) != SecretProgram.Puzzle2(x)) throw new System.Exception("mismatch!"); } class VisibleProgram { public static int Puzzle1(int x) { return x; } class SecretProgram { public static int Puzzle2(int x) { return x * x; } public class MetaProgram { public static void Puzzle(int x) { if (VisibleProgram.Puzzle1(x) != SecretProgram.Puzzle2(x)) throw new System.Exception("mismatch!"); }

Coding Duels Teaching Debugging/Problem Solving Skills From Hypothesis to Theorem – Step 1: Inference of hypothesized (delta) requirement When delta: Inference of failure-inducing conditions If difficult: forced generation of new data points – By adding data-point-fitting conditionals – Step 2: Writing of requirement-satisfying solution When delta: – Derivation of hypothesized faulty line(s) of code – Derivation of patches on the faculty line(s) of code – Step 3: “Ask Pex” to validate the hypothesis Initial almost-complete (delta) working implementation helps teach debugging skills

Coding Duels Teaching Abstraction Skills Step 1: Inference of hypothesized (delta) requirement from observed data points – Pattern recognition – Generalization from data points to formula – Inversion of constraint solving

Teaching Programming, Algorithm, and Program Understanding Skills Step 2: Writing of requirement-satisfying solution Many choices, e.g. recursion vs. loop implementation Teacher can give guidance: Providing hints in comments Providing initial almost-complete working implementation Teacher can give guidance: Providing hints in comments Providing initial almost-complete working implementation

Coding Duels Exercise 1.Go to optionally: sign in with Windows Live IDhttp://pex4fun.com 2.Click “New” 3.Write secret implementation with public static “Puzzle” method using System; public class Program { public static int Puzzle(int x, int y) { return x + y – 42; } } 4.“Ask Pex!” 5.Enter Coding Duel Name 6.Click “Turn This Puzzle Into A Coding Duel” 7.Open Permalink 8.Edit Description 9.Click “Permalink” 10.Use it, or create ulta-short version, e.g. with

Coding Duels Fun and Engaging Iterative gameplay Adaptive Personalized No cheating Clear winning criterion

Coding Duels Summary Coding Duel: write function that matches specification given by another implementation Semantic equivalence checked in cloud Student’s tasks: – Inference of hypothesized (delta) requirement – Writing of requirement-satisfying solution

SOCIAL EXPERIENCE Connecting Students – Competitive yet Self-Paced Environment

Social Experience

Social Experience Summary Community High score lists, leaderboard Live feed

PEX4FUN, A TEACHING PLATFORM Pages – Courses – Automated Assessments

Teaching

Teaching Summary How to become a teacher: – Sign in, choose nickname – Send us with nickname Teachers can – Create courses – Reuse existing or author existing pages – Get automated assessments of students – See students code

Outlook I Analyzing code submissions – Detecting duplicate submissions – Ranking submissions (how correct, how beautiful) – If faulty, give user guidance on “the next step”.

Outlook II Better code development experience – Debugging – Dynamic input/output table – More languages Beyond the web – Phone

More Reading Nikolai Tillmann, Jonathan de Halleux, and Tao Xie Pex for Fun: Engineering an Automated Testing Tool for Serious Games in Computer Science Microsoft Research, Technical report MSR-TR , March

Conclusion Taking programming into the browser, cloud –.NET: C# (with Intellisense), Visual Basic, F# – Execution, Code analysis with Pex Fun learning experience with Coding Duels Social experience: Live feed, sharing duels Teaching with automatic grading Create your own courses!