1. Code Hunt: Experience with Coding Contests at Scale Judith Bishop, R Nigel Horspool, Tao Xie, Nikolai Tillmann, Jonathan de Halleux Microsoft Research, University of Victoria and UIUC

2. Working and learning for fun Enjoyment adds to long term retention on a task Discovery is a powerful driver, contrasting with direct instructions Gaming joins these two, and is hugely popular Can we add these elements to coding? Code Hunt can! www.codehunt.com

3. Websites Game Project Community Data Release Blogs www.codehunt.com research.microsoft.com/codehunt research.microsoft.com/codehuntcommunity github.com/microsoft/code-hunt Linked on the Project page

4. The Code Hunt Game

5. Code Hunt Is a serious programming game Works in C# and Java (Python coming) Appeals to coders wishing to hone their programming skills And also to students learning to code Code Hunt has had over 300,000 users since launching in March 2014 with around 1,000 users a day Stickiness (loyalty) is very high

6. The Story

11. Gameplay 1.User writes code in browser 2.Cloud analyzes code – test cases show differences As long as there are differences: User must adapt code, repeat When they are no more differences: User wins level! secret code test cases

12. It’s a game! 1.iterative gameplay 2.adaptive 3.personalized 4.no cheating 5.clear winning criterion Score is based on how many puzzles solved, how well solved, and when solved secret code test cases

13. void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == 1234567890) throw new Exception("bug"); } a.Length>0 a[0]==123… T F T F F a==null T Constraints to solve a!=null a!=null && a.Length>0 a!=null && a.Length>0 && a[0]==123456890 Input null {} {0} {123…} Execute&Monitor Solve Choose next path Observed constraints a==null a!=null && !(a.Length>0) a==null && a.Length>0 && a[0]!=1234567890 a==null && a.Length>0 && a[0]==1234567890 Done: There is no path left. Dynamic Symbolic Execution

14. Available in Visual Studio since 2010 (as Pex and Smart Unit Tests) Powered by a IntelliTest in Visual Studio 2015 Nikolai Tillmann, Jonathan de Halleux, Tao Xie: Transferring an automated test generation tool to practice: from pex to fakes and code digger. ASE 2014: 385-396Jonathan de HalleuxTao XieASE 2014

15. Code Hunt Contests

16. Code Hunt PropertiesCode Hunt Contests Browser-basedHave world-wide reach on all platforms Automatic gradingAre cost effective Cloud based hostingScale to 100,000s players Clear scoring criteriaDetermine winners precisely Results availableCan identify top coders The game is fun, fresh and different.

17. The ah-ha moment Code Hunt can identify top coders

18. 30 Contests in the past year

20. What the player sees

21. Dashboard and Leaderboard Publically visible when the contest ends Visible only to the organizer

22. Puzzles and Analytics

23. Puzzles About 400 puzzles Classified and rated by difficulty Puzzle Bank is highly protected PuzzleGroupDescriptionSource Difficult y Calculat edPlayers P249-1numbersCompute 23*X-42NH21335 P277stringConvert N to base 4 stringNH23332 P115arrayConvert array of scores to letter gradesAPCS32329 P002arrayCount occurrences of a string in an arrayAPCS12325 P260-1boolsCompute x && (y || !z)NH12325 Number: P067 Group: Arrays Description: Remove duplicates from an array Source: NH Difficulty: 2

24. Puzzle Bank Statistics Modify the Calculated rating based on user experience. D = a + tries * b + tries * c * distance where a, b and c are constants with values a = 1 b = 0.02c = 0.05 tries is the average attempts for all players who solved the puzzle distance is the number of levels solved so far. Tries does depend on Mix of students who enter The internet speed Whether students use an IDE in addition

25. Laser, Sept 2014 Original Difficulty Average Calculated Difficulty s Total Try Count Average Try Count Max Try Count Total Solved Users 242.214.60 1.12 1.79 1068.832712 1.22 1.42 564.671312 1.33 2.42 15815.804510 1.42 2.43 13013.003110 1.52 4.05 21123.44739 1.61 3.29 12215.25528 2.13 4.29 11619.33436 2.23 1.87 234.6075 2.32 1.84 204.0085 2.42 9.05 7035.00682 2.51 3.10 428.40275 2.62 18.10 19063.331383

26. How players perceive difficulty Beauty of ProgrammingSubjective difficulty Perceived difficulty Players who started Qualification1.592.721511 Preliminary A2.171.84251 Preliminary B2.501.84247 Semi-Final2.602.22140 CSTA and TEALSSubjective difficulty Calculated difficulty Players who scored Students1.965.2261 Teachers1.964.3814

27. Code Hunt - the APCS (default) Zone Opened in March 2014 129 problems covering the Advanced Placement Computer Science course By August 2014, over 45,000 users started.

28. Effect of difficulty on drop off in sectors 1- 3 Yellow – Division Blue – Operators Green - Sectors

29. Aug 2014 and Feb 2015 PuzzleLevelAugFeb-A Compute -X1.11722 Compute 4 / X1.61821 Compute X-Y1.71822 Compute X/Y1.113238 Compute X%3+11.131518 Compute 10%X1.141216 Construct a list of numbers 0..N-12.13748 Construct a list of multiples of N2.21923 Compute x^y3.11118 Compute X! the factorial of X3.21619 Compute sum of i*(i+1)/23.51722

30. One week 3-10 October, 2014 One year May 2014 – April 2015

31. Survey results – before and after contest season How much did the puzzle aspect of Code Hunt keep you interested in reaching a solution? Till Sept 2014 Sept 2014 – May 2015

32. Survey Results cont. In your opinion, were your final solutions well-structured code? Till Sept 2014 Sept 2014 – May 2015

33. Gender and Code Hunt FemaleMale Respondents12%80% Played as practice for a course in C# 11%18% Played as practice for a course in Java 40%23% Played for own enjoyment 47%57% Open survey with 850 respondents

34. Beyond contests

35. Towards a Course Experience

36. Total Try Count Average Try Count Max Try Count Total Solved Users 1337436313061581 Public Data release in open source For ImCupSept 257 users x 24 puzzles x approx. 10 tries = about 13,000 programs For experimentation on how people program and reach solutions Github.com/microsoft/code-hunt

37. Upcoming events PLOOC 2015 PLOOC 2015 at PLDI 2015, June 14 2015, Portland, OR, USA PLDI 2015 CHESE 2015CHESE 2015 at ISSTA 2015, July 14, 2015, Baltimore, MD, USAISSTA 2015 Worldwide intern and summer school contests Public Code Hunt Contests are over for the summer Special ICSE attendees Contest. Register at aka.ms/ICSE2015 Code Hunt Workshop February 2015

38. Summary: Code Hunt: A Game for Coding 1.Powerful and versatile platform for coding as a game 2.Unique in working from unit tests not specifications 3.Contest experience fun and robust 4.Large contest numbers with public data sets from cloud data Enables testing of hypotheses and making conclusions about how players are mastering coding, and what holds them up 5.Has potential to be a teaching platform collaborators needed

39. Websites Game Project Community Data Release Blogs Office Mix ICSE Contest www.codehunt.com research.microsoft.com/codehunt research.microsoft.com/codehuntcommunity github.com/microsoft/code-hunt Linked on the Project page mix.office.com aka.ms/icse2015 to register Aka.ms/icse2015codehunt to play Grazie!

40. Appendix

41. Results and rewards Detailed results are presented Ranking is always based on score, not on time or attempts Score is based on how many puzzles solved, how well solved, and when solved Special rewards for contests

42. Comparing Code Hunt Code Hunt A serious online coding game Built on symbolic execution Addressing various audiences Data available in the cloud IPCC Annual programming contest Scored by unit testing Addressing university students Questions available on the web

43. BoP China 2014