1. Dynamic Benchmarking Software development though competition Alex Dubreuil Northeastern University dubreuil.a@husky.neu.edu acdubre@gmail.com

2. This slide intentionally left blank

3. Contents Dynamic Benchmarking Introduction Uses of the Benchmarking Game model Software Development (CS 4500) A Lesson I’ve learned Caution: Slide layout may cause drowsiness.

4. Benchmarking Assesses relative performance Typically by running standardized tests –Produces scores which are then compared –SATs Other options exist –Allowing software to compete directly –Chess game

5. The Traditional Approach Software A Static Benchmark Software B Software C Score A Score C Score B Developer A Developer B Developer C Parameterized by the domain.

6. The Dynamic Approach Team A Software A Benchmark A Team C Software C Benchmark C Team B Software B Benchmark B Artificial World (Game) Agent Ranking Parameterized by the domain. Agent

7. An Artificial World Agent’s View Administrator Agent Opponents’ communication, Feedback Beliefs, Challenges, Problems, Solutions Results Problems: Benchmark output Solutions: Software output Beliefs/Challenges: statements about algorithms

8. Problems & Solutions Problem communication: –Define an instance of a problem in the domain Solution communication: –Respond to an opponent’s problem –Administrator has a metric for determining how good a solution is –This metric is well defined and known by all

9. Beliefs & Challenges General statements about algorithms –Belief: Defines a subset of the problems in the domain Makes a statement about the problems in that subset –Challenge: A response to a belief of an opponent

10. Administrator Opponents’ communication –Filter all communication through the Administrator for security –Filter information when necessary Feedback: –Inform agents of rule violations –Inform agents of status changes

11. Administrator Results –Track state changes through the game –Produce the agent ranking from the end game state

12. What’s next Dynamic Benchmarking Introduction Uses of the Benchmarking Game model Software Development (CS 4500) A Lesson I’ve learned If you can read this, you don’t need glasses.

13. Overhead Requires mature Administrator, communication system for accurate results –Reuse between domains is possible Requires new translation for each problem domain

14. Software Development Ranks software without a mature benchmark –Dynamic approach excels when a well- defined benchmark does not exist Creates data to build better benchmarks –Because Agents, not Software, are ranked Forces developers to consider both their solutions and the problem domain

15. Education Motivates students Mature Administrator/Agent not required Creates interesting student interaction Creates a realistic software development environment

16. What’s next Dynamic Benchmarking Introduction Uses of the Benchmarking Game model Software Development (CS 4500) A Lesson I’ve learned Yeah, I got nothing.

17. Specker Challenge Game The SCG is the basis for Professor Karl Lieberherr’s Software Development class Uses an arity 3 boolean constraint satisfaction problem (CSP) as our domain Teams of 2~3 produce the components of an Agent

18. (Some of the) Skills Involved Using outsourced tools –DemeterF (developed by Bryan Chadwick) –Component Market Dealing with users –Underspecified requirements Source control Constraint Satisfaction algorithms Data mining

19. Added bonus Programmers RequirementsLimitations Domain Knowledge Experts Customers Users How-to So what? Salespeople Code Gibberish Non-technical Requirements

20. It’s a busy class Traditional grading would not work The competition keeps students motivated

21. What’s next Dynamic Benchmarking Introduction Uses of the Benchmarking Game model Software Development (CS 4500) A Lesson I’ve learned

22. Administrator Security Never accept extra input –Transaction: Challenge: ID, Type, Price –vs. –Transaction: Challenge: ID Check all necessary input –Transaction: Deliver Problem: ID, Problem –Check: Does the Problem match the Type?

23. General Lesson Never trust user input –Sanitize data –Protect against buffer overflows

24. More General Lesson It’s good to see things before they can do you or others harm –Users you can yell at –Security flaws that don’t cost money –Underspecified requirements

25. Alex Dubreuil Northeastern University dubreuil.a@husky.neu.edu acdubre@gmail.com Thank you!