1. Contributions of SCG to SDG Karl Lieberherr Northeastern University College of Computer and Information Science Boston, MA joint work with Ahmed Abdelmeged and Bryan Chadwick Karl Lieberherr Northeastern University College of Computer and Information Science Boston, MA joint work with Ahmed Abdelmeged and Bryan Chadwick Supported by Novartis SCG = Scientific Community Game = Specker Challenge Game

2. Contents Gamification of Software Development Scientific Community Game(X) – How it works – Advantages – Disadvantages Potential benefits for IBM Collaboration we seek Conclusions 7/7/2016Games for SD2 Paper available with more details.

3. Gamification of Software Development etc. at Northeastern Want reliable software to solve a computational problem? Design a game where the winning team will create the software you want. (Want to teach a STEM domain? Design a game where the winning students demonstrate superior domain knowledge.) 7/7/2016Games for SD3 Doesn’t TopCoder already do this?

4. SCG = Scientific Community Game Make software development more scientific. Software developers build reputation – propose and defend claims about their software – oppose claims made by others refute claims strengthen claims claim defined by refutation protocol 7/7/2016Games for SD4

5. Claims and Refutation Protocol Alice claims: I have a program that solves inputs in domain X with quality Q and resources R. – AliceClaim(X,Q,R) Bob is critical. He prepares an input in X and gives it to Alice who applies her program. Bob refutes AliceClaim(X,Q,R) iff Alice achieves R. – Refutation protocol 7/7/2016Games for SD5

6. Who are Alice and Bob? They are avatars developed by real Alice and real Bob. Alice and Bob compete with 10 other avatars in a full-round robin tournament. Who is the winner: The avatar with the highest reputation, i.e., the avatar who has the strongest, not successfully opposed claims (like in a real scientific community). 7/7/2016Games for SD6 Why a web application with avatars? Fair Evaluation.

7. What we want Engage software developers – let them produce software that models an organism that fends for itself in a real virtual world while producing the software we want. Have fun. Focus them. – let them propose claims about the software they produce. Reward them when they defend their claims successfully or oppose the claims of others successfully. 7/7/2016Games for SD7 Clear FeedbackSense of Progress Possibility of Success Authenticity (Facebook)

8. Life with SCG(X) with SCG structured collaboration between software developers, frequent feedback propose and oppose non- trivial claims to gain reputation. Drive to win knowledge accumulation in claims that have not been opposed successfully management effort goes into X without SCG collaboration is unstructured, less effective reputation gain is delayed knowledge is scattered in emails, programs and minds more management effort required 7/7/20168Games for SD

9. Software Engineering Properties fostered by SCG Reliable (otherwise the avatar is removed from the game) Flexible, modular (otherwise the avatar cannot be easily updated between competitions) Efficient (otherwise you cannot defend your claims and oppose the claims of others) 7/7/2016Games for SD9

10. How to define X Claim definition – Language to define claims – Refutation protocol information exchanged – language to define inputs, outputs – functions to check inputs, outputs refute function 7/7/2016Games for SD10

11. Benefits for IBM of using SCG(X) Teams perform know-how retrieval and integration and maybe some research. – Participating teams try to find the best knowledge in the area. – Claims language gives control! The non-opposed claims give hints about new X- specific knowledge. A well-tested solver for X-problems that integrates the current algorithmic knowledge in field X. 7/7/201611Games for SD

12. Benefits for IBM of using SCG(X) Also great for evaluating potential employees. 7/7/2016Games for SD12

13. Avatars propose and oppose 7/7/2016Games for SD13 CA1 CA2 CA3 CA4 egoistic Alice egoistic Bob reputation 1000 reputation 10 CB1 CB2 opposes (1) provides problem (2) solves problem not as well as she expected based on CA2 (3) WINS! LOSES proposed claims transfer 200 social welfare Life of an avatar: (propose+ oppose+ provide* solve*)*

14. What is SCG(X)? Teams Design Problem Solver Develop Software Deliver Avatar Agent AliceAgent Bob Administrator SCG police I am the best No!! Let’s play constructively 7/7/201614Games for SD Team Alice Team Bob

15. competitive / collaborative 7/7/2016Games for SD15 Avatar Alice: claim H Avatar Bob: opposes H, refutes: provides evidence for !H loses reputation rwins knowledge k wins reputation rmakes public knowledge k

16. Disadvantages of SCG The game is addictive. After Bob having spent 4 hours to fix his avatar and still losing against Alice, Bob really wants to know why! Overhead to learn to define and participate in competitions. The administrator for SCG(X) must perfectly supervise the game. Includes checking the legality of X-problems. – if admin does not, cheap play is possible – watching over the admin 7/7/201616Games for SD

17. How to compensate for those disadvantages Warn the scholars. Use a gentleman’s security policy: report administrator problems, don’t exploit them to win. Occasionally have a non-counting “attack the administrator” competitions to find vulnerabilities in administrator. – both generic as well as X-specific vulnerabilities. 7/7/201617Games for SD

18. Opening the development approach Problem to be solved: Develop the best practical algorithms for solving computational problems in domain X. Issue: There are probably hundreds of papers on the topic with isolated implementations. What are the best practical algorithms? Our solution: Use the scientific community game SCG(X) with a suitably designed claims language to compare the software. The winning avatar has the best practical algorithms/software. 7/7/201618Games for SD

19. Collaboration with IBM Offer SCG(X) as a web application for software development and education in STEM areas. We have some of this generic software in place but a lot more work is needed. Use SCG(X) as internal tool at IBM. 7/7/2016Games for SD19

20. Related Work TopCoder Renaissance mathematicians Various benchmark based competitions What is new? – Software that has an ego – Scientific Community Game 7/7/2016Games for SD20

21. Example: Independent Set An independent set in a graph is a set of mutually nonadjacent vertices. The problem of finding a maximum independent set in a graph, is one of most fundamental combinatorial NP-hard problems. 7/7/2016Games for SD21

22. Example: Independent Set Alice claims IndSet(m, 0.9, t(m)): – Alice can construct graphs G with at most m edges and she can construct a secret independent set I1 for G so that Bob, given G, size(I1) and t(n) minutes only, cannot find an independent set I2 with size(I2) >= size(I1)*0.9. 7/7/2016Games for SD22

23. Example: Refutation Protocol Alice constructs graph G and deposits her secret independent set I1. Alice gives G as well as the size of I1 to Bob. Bob has 10 minutes to construct his independent set I2 which he gives to Alice. Alice reveals her secret set I1. Bob refutes iff size(I2) >= size(I1)*0.9 7/7/2016Games for SD23

24. Conclusions To address a problem domain X: – “map it to second life”: define a scientific community game for X on the web: SCG(X) – let the game SCG(X) run a few times and choose the winner Benefits – Evaluates fairly, frequently, constructively and dynamically. Encourages retrieval of state-of-the-art know-how, integration and discovery. – Challenges humans, drives innovation, both competitive and collaborative. – Agents point humans to what needs attention in problem solution / software. 7/7/2016Games for SD24

25. Conclusions SCG(X) provides a structured process for developing software for computational problems. Benefits – Social Engineering: makes it fun through game. – Fair: Only hard work makes you win. – Engage a large community on one domain X. Tools 7/7/2016Games for SD25

26. Conclusions SCG has many applications of potential value to IBM – Training employees in constructive domains – Software development process – Hiring – Driving innovation in constructive domains 7/7/2016Games for SD26

27. Thank you 7/7/2016Games for SD27

28. Axioms for reputation Confidence factors are real numbers in [0,1] that indicate how confident the proposer is in the claim. The confidence factor should reflect the amount of effort that has been put into the claim. For example, one could use confidence 1 for a claim for which the scholar has a proof. Confidence 1 means that we believe that the claim cannot be opposed. The scholars are encouraged to give their "true" confidence in a claim. If not, they will be penalized. 7/7/2016Games for SD28

29. Axioms Scholars gain reputation either by opposing other scholars' claims or by having their claims recognized by other scholars (when the other scholars fail to oppose). Scholar's gain from their claim is proportional to both the credibility of their claim and the recognition of the claim after the refuting protocol. Note: the payoff for strengthening comes after the strengthened claim is recognized (successfully defended or never opposed). 7/7/2016Games for SD29

30. Axioms One scholar's reputation gain is another scholar's reputation loss. The sum of all scholars's reputation is preserved. 7/7/2016Games for SD30

31. Axioms The refutation protocol recognizes claims by a recognition factor in [-1,1]. A recognition factor of 1 means that the other scholar has completely failed to refute that claim. A recognition factor of -1 means that the other scholar has completely succeeded to refute that claim. 7/7/2016Games for SD31

32. Axioms Claims have a credibility in [0..]. The Credibility of a claim made by a \scholar is proportional to either the scholar's ``confidence'' in the claim or the scholar's reputation or both. The claim's confidence reflects the amount of effort made by the scholar behind the claim to prove the claim (i.e., turning it into a theorem or finding other supporting evidence). Scholar's reputation is the sum of the scholar's initial reputation plus the reputation gains and losses; thus reflecting the past performance of the scholar. 7/7/2016Games for SD32

33. Software Development Governance Software Development Governance (SDG) is defined as: – Establishing chains of responsibility, authority and communication to empower people within a software development organization – Establishing measurement and control mechanisms to enable software developers, project managers and others within a software development organization to carry out their roles and responsibilities 7/7/2016Games for SD33

34. Applications Develop algorithms/software for new computational domain X – Scientific Community Game Software Development: Describe a problem domain X so that SCG(X) provides the best algorithms and their implementations for problems in X. (best within the participating scientific community) 7/7/2016Games for SD34 SCG = Scientific Community Game = Specker Challenge Game

35. 7/7/2016Games for SD35

36. Plan Why is it relevant, useful? – Larger context: Open Innovation, Wikinomics – Applications: Netflix in the small, teaching What is it? What is new? – Map problem domain to “second life”, find best solution there and map it back to real life. What do we improve: benefits of SCG How to use SCG Disadvantages Experience with current implementation Related work Detailed example Conclusions 7/7/2016Games for SD36

37. Introduction (2) Scientific Community Game(X) [SCG(X)] – Goal: Foster innovation and reliable software for solving optimization problems in some domain X A virtual scientific community consists of virtual scholars that propose and oppose claims maximizing their reputations 7/7/2016Games for SD37

38. Claim Subdomain N – subset of problems Confidence [0,1] Valuation [0,1] 7/7/2016Games for SD38 confidence 0 1 valuation (how well problems in N can be solved)

39. Claim 7/7/2016Games for SD39 0 1 valuation strengthening correct valuation over strengthening

40. Hypothesis hypothesis by Alice: for all problems F in niche N there exists a solution J: p(F,J) Bob opposes: F’ to Alice, Alice cannot find J’:p(F’,J’) therefore she loses reputation. 7/7/2016Games for SD40

41. Full Round Robin Tournaments or Swiss-Style Agents to play the SCG(X). Repeat a few times with feedback used to update agents. Within the group of participating agent, the winning agent has the – best solver for X-problems – best supported knowledge about X 7/7/201641Games for SD

42. What is the purpose of SCG? The purpose of playing an SCG(X) competition is to assess the "skills" of the agents in: – "approximating" optimization problems in domain X, – "figuring-out" the wall-clock-time-bounded approximability of niches in domain X, – "figuring-out" hardest problems in a specific niche, and – "being-aware" of the niches in which their own solution algorithm works best. This multi-faceted evaluation makes SCG(X) more superior to competitions based on benchmarks that only test the player's skills in approximating optimization problems. During SCG, players cross-test each others' skills. 7/7/2016Games for SD42

43. How to use SCG Company A provides a problem domain description X and submits it to the SCG server. The game SCG(X) runs on the web (with human algorithm/software developers involved) and company A receives good, tested software and knowledge about problem domain X 7/7/2016Games for SD43

44. Plan Why is it relevant, useful? – Larger context: Open Innovation, Wikinomics – Applications: Netflix in the small, teaching What is it? What is new? – Map problem domain to “second life”, find best solution there and map it back to real life. What do we improve: benefits of SCG How to use SCG Disadvantages Experience with current implementation Related work Detailed example Conclusions 7/7/2016Games for SD44

45. Benefit: Improving Benchmark-Driven Algorithm Development A number of autonomous teams Each team develops an agent that embodies their own heuristics Agents participate in a competition (various benchmarks are used) Teams develop their agents for the next competition Examples: SAT, CSP, SMT, ASP (Answer Set Programming) competitions, etc. 7/7/201645Games for SD

46. From Benchmark-Driven to SCG-Driven Algorithm Development Hard to measure and detect what is fraud. Instead: Design a system that needs a much weaker “gentleman’s agreement” or none at all The Static Benchmark Problem is ONE problem that SCG solves. Dynamic Benchmarks Others: crowd sourcing management, new software development process that engages software developers and that fosters ease of evolution (e.g., good separation of concerns, …) 7/7/2016Games for SD46

47. Problems with Static Benchmarks http://www.cs.kuleuven.be/~dtai/events/ASP- competition/index.shtml Policy against special purpose solutions The purpose of the competition is to be as informative as possible about strengths and weaknesses of … Submission of special purpose programs for solving certain benchmark problems falsifies the information that we get from the rankings and goes against the spirit of the competition. … the use of special purpose programs for certain benchmarks can rightfully be considered as scientific fraud. We appeal to participants … 7/7/2016Games for SD47

48. SCG-Driven Algorithm Development Differences to Benchmark-Driven – You don’t rank chess players by giving them a benchmark; you let them play – We turn the algorithms into egoistic virtual scientists that fend for themselves – social welfare: constructive knowledge based on good algorithms 7/7/2016Games for SD48

49. What is SCG(X) 7/7/2016Games for SD49 no automation human plays full automation agent plays degree of automation used by scholar our focus some automation human plays 0 1 more applications: test constructive knowledge transfer to reliable, efficient software agent Bob Alice

50. Scholars and Agents: Same rules Are encouraged to 1.offer results that are not easily improved. 2.offer results that they can successfully support. 3.strengthen results, if possible. 4.stay active and publish new results or oppose current results. 5.become famous! 7/7/201650Games for SD

51. More Applications Special issue editors for problem domain X. publish top 15 submissions Professor teaching a software development class: students develop fighting agents for full-round robin tournament Teaching constructive topics etc. 7/7/2016Games for SD51

52. Soundness Theorem SCG is sound: The agent with the best algorithms / knowledge wins (there is no way to cheat) – best: within the group of participating agents – issues: Does an agent win because she is good at solving? Or good at proposing, opposing and providing? Answer: proposing, opposing and providing all reduce to solving. 7/7/2016Games for SD52

53. Justifying benefits (1) Benefit: competitive – collaborative Game component: hypotheses propose-oppose : problems provide-solve How this game component brings the benefit – hypothesis by Alice: for all problems F in niche N there exists a solution J: p(F,J) – Bob opposes: F’ to Alice, Alice cannot find J’:p(F’,J’) therefore she loses reputation. – Alice lost but she now knows F’ where she cannot achieve what she claimed. F’ was harder than what Alice expected. 7/7/2016Games for SD53

54. Justifying benefits (2) Benefit: competitive – collaborative Game component: hypotheses propose-oppose : problems provide-solve How this game component brings the benefit – hypothesis HA by Alice: for all problems F in niche N there exists a solution J: p(F,J) – Bob opposes by non-trivially strengthening HA to HB: HB => HA. Alice cannot discount HB. Therefore she loses reputation. – Alice lost but she now knows that her hypothesis HA might not be the strongest. 7/7/2016Games for SD54

55. Benefits of SCG-driven Focus on understanding problem domain. – What are the niches where specialized algorithms perform well? – What are the hard problems in a niche? Knowledge maintenance system Control of niches to be explored 7/7/2016Games for SD55

56. Reputation Gain Challenging (C) Gain for A (A supporting), Loss for A (B discounting) 7/7/201656Games for SD

57. Plan Why is it relevant, useful? – Larger context: Open Innovation, Wikinomics – Applications: Netflix in the small, teaching What is it? What is new? – Map problem domain to “second life”, find best solution there and map it back to real life. What do we improve: benefits of SCG How to use SCG Disadvantages Experience with current implementation Related work Detailed example Conclusions 7/7/2016Games for SD57

58. How to use SCG(X) ABB needs new ideas about how to solve optimization problems in domain X. Define hypothesis language for X – X-problems – hypotheses, includes protocol Submit hypothesis language definition to SCG server. 7/7/201658Games for SD

59. How to use SCG(X) Offer prize money for winner with conditions, e.g., performance must be at least 10% higher as performance of agent XY that ABB provides. 10 teams from 6 countries sign up, committing to 6 competitions. Player executables become known to other players after each competition. One team from ABB. The SCG server sends them the basic agent and the administrator for testing. 7/7/201659Games for SD

60. How to use SCG(X) Game histories known to all. Data mining! First competition is at 23.59 on day 1. Registration starts at 18.00 on same day. The competition lasts 2.5 hours. Repeat on days 7, 14, … 42. The final winner is: Team Mumbai, winning 10000 Euro. Delivers source code and design document describing winning algorithm to ABB. 7/7/201660Games for SD

61. Benefits for ABB of using SCG(X) Teams perform know-how retrieval and integration and maybe some research. – Participating teams try to find the best knowledge in the area. – Hypothesis language gives control! The non-discounted hypotheses give hints about new X-specific knowledge. A well-tested solver for X-problems that integrates the current algorithmic knowledge in field X. 7/7/201661Games for SD

62. Plan Why is it relevant, useful? – Larger context: Open Innovation, Wikinomics – Applications: Netflix in the small, teaching What is it? What is new? – Map problem domain to “second life”, find best solution there and map it back to real life. What do we improve: benefits of SCG How to use SCG Disadvantages Experience with current implementation Related work Detailed example Conclusions 7/7/2016Games for SD62

63. GIGO: Garbage in / Garbage out If all agents are weak, no useful solver created. WEAK against STRONG: – STRONG refutes a claim that is true but WEAK cannot support it. Correct knowledge might be discounted. – STRONG strengthens a hypothesis too much that it becomes discountable, but WEAK cannot discount it. Incorrect knowledge might be supported – STRONG is discouraged to exploit WEAK by game rules 7/7/2016Games for SD63

64. Plan Why is it relevant, useful? – Larger context: Open Innovation, Wikinomics – Applications: Netflix in the small, teaching What is it? What is new? – Map problem domain to “second life”, find best solution there and map it back to real life. What do we improve: benefits of SCG How to use SCG Disadvantages Experience with current implementation Related work Detailed example Conclusions 7/7/2016Games for SD64

65. Experience Used for 3 years in undergraduate Software Development course. Prerequisites: 2 semesters of Introductory Programming, Object-Oriented Design, Discrete Structures, Theory of Computation. – Collect and integrate knowledge from prerequisite courses, lectures, and literature. – Teach it to the agent. 30% of grade is allocated for agent performance in weekly competitions. 7/7/201665Games for SD

66. Mechanics of using current implementation We define X = MAX-CSP. We produce administrator and baby agent for X at beginning of course. Game flow: – Agents register with administrator – After deadline, administrator tells agents when it is their turn (1 minute) sending them all currently proposed hypotheses – After 1 minute, agent sends back transactions. 7/7/2016Games for SD66

67. Mechanics of using current implementation 3 competitions per week. Last about 12 hours each. 75% of competitions count towards grade. 1 competition: attack the administrator. 7/7/2016Games for SD67

68. Experience MAX-CSP MAX-CSP Problem Decompositions T-Ball (one relation), Softball (several relations, one implication tree), Baseball (several relations). ALL, SECRET 7/7/201668Games for SD

69. Stages for SECRET T-Ball MAXCUT – R(x,y)= x!=y – fair coin ½ – maximally biased coin ½ – semi-definite programming / eigenvalue minimization 0.878 7/7/201669Games for SD

70. Stages for SECRET T-Ball One-in-three – R(x,y,z) = (x+y+z=1) – fair coin: 0.375 – optimally biased coin: 0.444 7/7/201670Games for SD

71. Stages for ALL Baseball Propose/Oppose/Provide/Solve – based on fair coin – optimally biased coin correctly optimize polynomials – correctly eliminate noise relations – correctly implement weights – … 7/7/201671Games for SD

72. Life with SCG with SCG structured collaboration between scholars, frequent feedback motivation: propose and oppose non-trivial hypotheses to gain reputation. Drive to win knowledge accumulation in undiscounted hypotheses target scholars on a topic without SCG collaboration is unstructured, less effective motivation: reputation gain is delayed knowledge is scattered in emails, programs and minds more management effort required 7/7/201672Games for SD

73. How to model a hypothesis A problem space. A discounting predicate on the problem space. A protocol to set the predicate through alternating “moves” (decisions) by Alice and Bob. If the predicate becomes true, Alice wins. 7/7/201673Games for SD

74. How to model a hypothesis Proposing and challenging a hypotheses is risky: your opponent has much freedom to choose its decisions within the game rules. Alternating quantifiers. Replace “exists” by agent algorithm kept by administrator. 7/7/201674Games for SD

75. Hypothesis [Example] 1in3 example. 7/7/201675Games for SD

76. X = Boolean MAXCSP Given a sequence of Boolean constraints formulated using a set R of Boolean relations, find an assignment that maximizes the fraction of satisfied constraints. Niche defined by R. 7/7/201676Games for SD

77. 1in3 niche Only relation 1in3 is used. 1in3 problem F: v1 v2 v3 v4 v5 1in3( v1 v2 v3) 1in3( v2 v4 v5) 1in3( v1 v3 v4) 1in3( v3 v4 v5) secret 1 0 0 1 0 Truth Table 1in3 000 0 001 1 010 1 011 0 100 1 101 0 110 0 111 0 Secret quality SQ = 3/4 7/7/201677Games for SD

78. 1in3 Hypothesis 1in3 hypothesis H proposed by Alice: exists F in 1in3 niche so that for all S Bob that opponent Bob searches in time t (small constant) seconds: Quality(F,S Bob ) < 0.4 * Quality(F,S Alice ). H = (niche = (1in3), AR =0.4, confidence = 0.8) Bob has clever knowledge that Alice does not have. He opposes the hypothesis H by challenging it using his randomized algorithm. 7/7/201678Games for SD

79. Bob’s clever knowledge 4/9 for 1in3 4/9 for 1in3: For all F in 1in3 niche, exists S so that Quality(F,S) >= 0.444 * SQ. Proof: la(p)=3*p*(1-p) 2 has the maximum 4/9. argmax p in [0,1] la(p) = 1/3. Without search, in PTIME. Derandomize Bob successfully discounts Alice gets a hint – Was Bob just lucky? Truth Table 1in3 000 0 001 1 010 1 011 0 100 1 101 0 110 0 111 0 7/7/201679Games for SD

80. 1in3 Hypothesis Bob does not know whether 4/9 is best possible. Should check Semidefinite Programming. Bob only knows that the set of 1in3 problems having a solution satisfying 4/9 + eps, eps > 0, is NP-complete. 7/7/201680Games for SD

81. Related Work Renaissance mathematicians Various benchmark based competitions What is new? – Software that has an ego – Holistic software with introspection – Evaluating software through a game – Scientific Community Game Software Development 7/7/2016Games for SD81

82. Conclusions To address a problem domain X: – “map it to second life”: define a scientific community game for X on the web: SCG(X) – let the game SCG(X) run a few times and choose the winner Benefits – Evaluates fairly, frequently, constructively and dynamically. Encourages retrieval of state-of-the-art know-how, integration and discovery. – Challenges humans, drives innovation, both competitive and collaborative. – Agents point humans to what needs attention in problem solution / software. 7/7/2016Games for SD82

83. Conclusions SCG(X) provides a structured process for developing software for optimization problems. Benefits – Social Engineering: makes it fun through game. – Fair: Only hard work makes you win. – Engage a large community on one domain X. Tools 7/7/2016Games for SD83