1. Identifying Programmer Ability Using Peer Evaluation: An Exploratory Study Jeffrey C. Carver, University of Alabama Lorin Hochstein, USC Information Sciences Institute Jason Oslin, University of Alabama

2. The Importance of Categorizing Programmer Ability In Industry In Research In Teaching 2

3. How We Measure Ability Professional vs. Novice Years of Experience Elaborate Evaluations 3

4. Is There A Better Way? Peer evaluation “We know a good programmer when we see one” Subjective judgment as a predictor 4

5. Outline Study Design Data Analysis Conclusions Future Work 5

6. Our Study Goal – Characterize peer assessment of programmer ability from the point of view of a researcher in the context of an undergraduate computer science course Hypotheses – H1: Developers are able to accurately predict the programming ability of their peers – H2: Low-performers tend to over-estimate their ability 6

7. Test Environment Senior Level Programming Languages Course – Five individual assignments – Graded by the unbiased instructor – Student participation had no effect on grade 7

8. Data Collected Subjective Data – Participant evaluation of self – Participant evaluation of peers – Post-study questionnaire Objective Data – Grades on projects – “True” measure for this study 8

9. Collecting Subjective Data 9 Sample data collection form

10. Coding the Grades 10 A = 5 Top 20% B = 4 C = 3 D = 2 E = 1 Bottom 20%

11. Coding the Grades 11

12. A Look at the Data Set 12 Raters Ratees

13. A Look at the Data Set 13

14. Evaluating Hypothesis 1 Developers are able to accurately predict the programming ability of their peers 14 Actual Grade Mode Of Peer Ratings

15. Additional Observation 15 Actual Grade Difference Between Mode and Actual

16. Evaluating Hypothesis 2 Low-performers tend to over-estimate their ability 16 Actual Grade Self Rating

17. A Qualitative Analysis Rating oneself Compare my self perception to my peers’ perception of my ability Confidence in rating peers 17

18. Threats to Validity Construct – Are grades a valid indicator of ‘true’ ability? – Ceiling effect displayed by skew of grades? – Are we evaluating enough skill sets involved in software development? Low number of ratings What happens when we move beyond from classrooms to industry? 18

19. Future and Ongoing Work Running the study again Goals of future work – Characterize high-performers – Understand programming ability – Improve competent programmers 19