1. The Scientific Community Game as A Crowdsourcing Platform to Distinguish Good from Bad Presentation to Clients by Software Development Organization 4/24/20111

2. Crowdsourcing Platform Crowdsourcing – is the act of taking a job traditionally performed by a designated agent (usually an employee) and outsourcing it to an undefined, generally large group of people in the form of an open call. – enlists a crowd of humans to help solve a problem defined by the system owners. A crowdsourcing platform is a generic tool that makes it easy to develop a crowdsourcing system. 4/24/20112

3. Crowdsourcing Platform The job, target problem is – to solve instances of a problem and make claims about the solution process. – to build knowledge base of claims and techniques to defend the claims 4/24/20113

4. Requirements for Crowdsourcing Platform Find a good way to combine user contributions to solve the target problem. Find a good way to evaluate users and their contributions. Find a good way to recruit and retain users. 4/24/20114

5. Combine user contributions We found an excellent way. Users build on each others work: strengthening. Users check each others claims for correct judgment. – Claims are defended and refuted. Users trade reputation and information. 4/24/20115

6. Voting with Justification I vote – for this claim (agree) because I can defend it and refute its negation. – against this claim because I can oppose it (refute or strengthen). 4/24/20116

7. Evaluate users and their contributions Calculate reputation – confidence by the proposer that a claim is good (gc) – confidence by the opposer (refute or strengthen) that the claim is bad (bc) The scholars are encouraged to set their confidences truthfully. Otherwise they don't gain enough reputation or they lose too much reputation. 4/24/20117

8. Reputation Update Claimgoodbad proposeupdown opposedownup up: if you are good, there is a chance that you win down: if the other is good, there is a chance that you lose up: reputation goes up, but has to provide knowledge that might reveal secret technique. down: reputation goes down, but might gain knowledge that reveals secret technique. 4/24/20118

9. Reputation Update Claimgoodbad proposeupdown opposedownup up: if you are good, there is a chance that you win down: if the other is good, there is a chance that you lose confidence: proposer: claim is good: gc opposer: claim is bad: bc r = result of reputation protocol. Reputation update: r*gc*bc (various refinements are possible) 4/24/20119

10. Perfect Being perfect means to make perfect decisions. up: if you are perfect, you will not lose. down: if the other is perfect, you will not win. Claimgoodbad proposeupdown opposedownup up: if you are good, there is a chance that you win down: if the other is good, there is a chance that you lose 4/24/201110

11. Result: We have successfully developed the SCG Crowdsourcing platform: SCG Court Requirements Document User’s Guide Continuous Testing Approach Well documented software with Design Document for Maintainers Acceptance Test – CSP tournament – HSR tournament – NetworkFlow tournament 4/24/201111

12. Software Development Process Customized Spiral Process using continuous testing – http://groups.csail.mit.edu/pag/continuoustesting/ http://groups.csail.mit.edu/pag/continuoustesting/ Software Management Approach – Motivating software developers with interesting project that provides numerous learning opportunities. – Overcoming initial resistance by persistence and significant involvement of teaching staff. Adding Ahmed. 4/24/201112

13. Software Management Approach Hire good software developers with significant initiative and involvement in the project. Flexible project organization: very open to numerous requirements changes. Trusting software developers with significant responsibilities. Greek Gods: found subproject that was optimal for their time table. Dan Marcucci: Let’s use DemeterF the right way. 4/24/201113

14. Software Management Approach I amplified the good ideas that I spotted and did not force them on you! I continued to refine the game with Ahmed and Yue and we carefully justified all requirements changes. Reto: continuous testing approach. Setting up svn on assembla. 4/24/201114

15. Domain of Requests and Responses Three kinds of requests – Instance Requests Response: Instance Boolean valid(Instance) – Solution Requests Response: Solution Boolean valid(Instance,Solution) – Claim Analysis Requests Response: good/bad decision determine which claims are – good: agree – bad: refute or strengthen Make prediction about how protocol will run – Claims are about the relationship between instances and their solutions 4/24/201115

16. What is good/bad? A claim is – good if both scholars agree if it is predominantly defended – bad if scholars don’t agree if it is predominantly refuted Refutation is the complement of defense and is based on the requests and responses exchanged. 4/24/201116

17. Good/ Bad Meaning for Mathematical Claims Good – true claim: if there is no strengthening relation between claims (strengthen(c1,c2) is false if c1!=c2.) – optimal claim: if there is a strengthening relation between claims. Bad – false claim – non-optimal claim 4/24/201117