1. Nathan Jones Andrew Cann Hina Popal Saud Almashhadi ASSESSMENT OF SOCCER REFEREE PROFICIENCY IN TIME-SENSITIVE DECISION-MAKING

2. Agenda 1.Context 2.Problem & Need Statement 3.Design Alternatives 4.Simulation 5.Simulation Output 6.Utility Analysis 7.Conclusions 8.Management 2

3. Introduction to Soccer Soccer is the world’s most popular sport. Generates the most revenue: In 2009-2010 season the English Premier League generated roughly 3.2 billion dollars European soccer generated 21.6 billion dollars Highest average attendance for international club competitions : FIFA World Cup UEFA Champions League 3 Information taken from: http://www.economist.com/blogs/gametheory/ 2011/09/ranking-sports%E2%80%99-popularity

4. Introduction to Soccer The game is played by two teams (11 vs. 11). Field dimensions: 115 by 74 yards 2 – 45 minute periods 3 Referees – 1 main referee and 2 assistant referees Responsible for upholding the integrity of the game 4 MR ARAR ARAR

5. Referee Responsibilities Upholding the integrity of the game: Make accurate calls Make calls that don’t interrupt the flow of the game Be in proper position, to assess, process, and identify correct call 5 Referees are categorized as either junior referees (entry level) or senior referees (advanced level). Current MLS referees make 86.1 % correct calls. (USSF)

6. Acknowledgement of Sponsor Metro DC Virginia State Referee Program (MDCVSRP) oversees all soccer referees in the Commonwealth of Virginia (over 5400 referees) Responsibilities: 1)Train and evaluate junior and senior referees 2)Assign Referees to officiate games 3)Promote high quality referees to senior ranks 6 Responsibilities 2 and 3 depend heavily on ability to assess referee call accuracy

7. Referee Call Making Process 7

8. Referee Assessment 8 On-Field Assessments Written Exam on Knowledge of the Game Fitness Test

9. Referee Assessment is Broken Junior referees do not undergo fitness tests or on field assessments (Preventing evaluation of Fitness or GFU attributes) The evaluation process for referees is broken: 96% of total MDCVSRP Referees (junior level) do not receive assessment in two of three attributes. 9 Referee Attributes Assessment Method FitnessFitness Test (senior referees) Call Decision Making (CDM)Written exam on rules (All referees) Game Flow Understanding (GFU)Indirectly using on field assessment (senior referees)

10. Agenda 1.Context 2.Problem & Need Statement 3.Design Alternatives 4.Simulation 5.Simulation Output 6.Utility Analysis 7.Conclusions 8.Management 10

11. Problem Statement 96 % of MDCVSRP referees (Junior level) do not receive assessment for Game Flow Understanding and fitness attributes as predictors of call accuracy. 11

12. Need Statement 12 An assessment method is needed to evaluate referee accuracy in a cost effective manner utilizing fitness and/or Game Flow Understanding (GFU). Scope:Our analysis will focus on determining the best system concept for assessing MDCVSRP junior referees. Specifics of design and implementation are considered future work.

13. Agenda 1.Context 2.Problem & Need Statement 3.Design Alternatives 4.Simulation 5.Simulation Output 6.Utility Analysis 7.Conclusions 8.Management 13

14. Design Alternatives 14 #AlternativeDescriptionTests Total Cost (5,139 Referees) 1Fitness Test A baseline fitness test equivalent to those administered at senior grades Fitness $26,990 2Game Flow Evaluation Video performance assessments conducted by official assessors GFU $337,995 3 Combined Evaluation Combination of first two evaluations Fitness GFU $341,870 4No Assessment Not conducting any referee evaluations (status quo) None$0.00 Costs defined as physical + implementation cost for one time evaluation of all junior referees.

15. Evaluation Of Alternatives Utility of each alternative defined as: Expected call accuracy of the top 100 referees identified using each alternative within junior referee pool (5000 referees). To determine utilities, a two part analysis was conducted: 1)Function for call accuracy based on fitness and GFU levels developed using discrete soccer game simulator. 2)Using part 1 function, expected call accuracy of top 100 referees selected by each alternative computed through Monte Carlo analysis. 15

16. Agenda 1.Context 2.Problem & Need Statement 3.Design Alternatives 4.Simulation 5.Simulation Output 6.Utility Analysis 7.Conclusions 8.Management 16

17. Simulation: Input / Outputs 17

18. Expansion on Prior Work 18 Simulation was re-designed and re-coded from scratch. Simulation ElementSolomon, et al. (2011)This Project Probability Maps1 map for all teams, all time, and all score19 maps dependent on team, time, an score Ball Position Function1 event4 state cycle scaled to time Referee Position Function1-D, chase ball on left diagonal2-D, based on GFU, scaled to time Fitness3 levels5 levels GFUNone5 levels based on probability maps Call GridsNoneSurvey 16 senior state referees Call Event TriggerSimple probabilityCalls grids and position in cycle Distance vs. Call Accuracy FunctionEstimated Figure of Merit Surveyed 16 senior state referees and generated regression Number of Teams in GameHome vs. HomeHome vs. Away (4 Options) Number of Teams Simulated14 Team Strategy ChangesNeverTime / Score Referee/Ball Movement Scaled to TimeNoYes

19. Simulation – Ball and Referee Position In the discrete event simulation, a soccer field is divided into a fine grid of cells. 19 8510 cells Each Cell 1x1 yd Cell Groupings: 60 Movement Polygons 24 Call Grids 0.5 s refresh rate (game time)

20. Two Teams - Possession Shifts The ball shifts possession between two different teams, each executing its own unique strategy. Changes in possession occur due to failed passes or shot events. 20

21. Simulation: Ball Movement 21

22. Cycle of Events 22

23. Ball & Referee Movement Algorithm At any time in simulation, ball moving to set destination in straight line. Destination changes during dribbling / passing. Time taken for ball to move incrementally to destination (# Refreshes) is reflective of ball speed and distance: 23

24. Shot Events Whenever ball finishes dribbling, probability determines if ball is shot at goal. Shot either results in goal or turnover. 24

25. Pass Events If no shot, Ball passed between polygons controlled by movement probability maps indicating destination and chance of success. Polygon (n+1) = Polygon (n) * Prob. Map When new polygon selected, destination is set to random cell within polygon Map sets are formulated for : 25 Manchester United Arsenal Wigan Athletic Stoke City

26. Probability Maps Ball movement and shot data were gathered from the Guardian Chalkboard Website. 80 total games (over 35,000 pass & shot events) recorded for Stoke City, Manchester United, Arsenal, Wigan. 26 Data was analyzed for strategy and used to produce shot and movement probability maps

27. Probability Maps – Team Strategy Two way ANOVA Analysis: Time + Score + Time*Score Time has an effect on pass accuracy: Arsenal(p = 0.777); United(p=0.142); Stoke (p=0.001); Wigan (p=0.001) Score has an effect on pass accuracy: Arsenal(p = 0.231);United(p=0.001);Stoke(p=0.000); Wigan (p=0.000) Score*Time has an effect on pass accuracy: Arsenal(p = 0.338);United(p=0.000);Stoke(p=0.000);Wigan(P= 0.116) 27 Strategy Analysis conducted to determine when strategy maps should be changed (Metric = % completed passes)

28. Simulation: Referee Movement 28

29. Referee Profile Definition To determine the effect of Fitness and Game Flow Understanding on call performance: 25 referee “profiles” defined as combinations of fitness and game flow understanding. 29 Referee Game Flow Understanding Referee Fitness 0255075100 /////////0.250.410.580.740.9 0 2.023 yds / sRef 1,1Ref 1,2Ref 1,3Ref 1,4Ref 1,5 25 2.495 yds / sRef 2,1Ref 2,2Ref 2,3Ref 2,4Ref 2,5 50 2.967 yds / sRef 3,1Ref 3,2Ref 3,3Ref 3,4Ref 3,5 75 3.439 yds / sRef 4,1Ref 4,2Ref 4,3Ref 4,4Ref 4,5 100 3.911 yds / sRef 5,1Ref 5,2Ref 5,3Ref 5,4Ref 5,5

30. Simulation – Ref movement 30 One main referee running within left hand diagonal route area. Referee movement speed depends on fitness level of profile tested.

31. Simulation - Ref Movement At each refresh rate (0.5 s), referee will compute desired location relative to ball using one of 2 movement scripts: 1)No Prediction – Referee will set destination to closest cell within 11 – 13 yds of ball’s current location. 2) Prediction – If dribbling: Referee will set destination to closest cell within 11 – 13 yds of next most probable pass destination. 31 Once destination is set, referee will begin moving to destination (rate = speed). Process repeats at each refresh

32. Simulation – Ref Movement Proportion of time referee utilizes script 2 depends on GFU level. Referee with (GFU = 0.75) with remain in script 2 75% of time. 32 GFU also includes an ability of referee to recognize a build up to a call: Probability that predicting referee anticipates the call and switches to script 1 until the call occurs.

33. Simulation: Call Events 33

34. Call Events Call grid probabilities used to generate events based on ball location whenever new cycle begins. Further probabilities determine where in cycle event occurs. Source: Senior MDCVSRP referee surveys (n = 16) 34 Roughly 90 events per game Passing: 0.21 Dribbling: 0.44 En-route: 0.15 Receiving: 0.21

35. Simulation - Call Accuracy Whenever a call event occurs, referee must make a decision regarding the nature of the event (infraction, no infraction). The probability that he makes the correct call depends on the distance from the ball. 35

36. Referee Call Accuracy Function 36 Source: Senior MDCVSRP referee surveys (n = 16) Distance > 20 ydsDistance <= 20 yds Accuracy Peaks at 11 – 13 yds

37. Simulation: Output 37

38. Simulation - Output Simulation output : Each profiles simulated through 2,000 games (200 per team comb.) Referee call accuracy was calculated for each game. 38

39. Simulation Demo 39

40. Validation of Simulator STATISTICSimulationProfessional Soccer Average Goals per game0.8266̴ 1.553 (EPL 4 team Average) [1] Average Team Passes per game 449̴ 424 (EPL 4 team Average) [2] Average Referee Distance Run per game (yds) 11, 68611, 289 (NZFC) [3] 40 [3] - D.R.D. Mascarenhas et al. (2009) "Physical Performance and Decision Making in Association Football Referees: A Naturalistic Study" [online]. Available: http://www.benthamscience.com/open/tossj/articles/V002/1TOSSJ.pdf [1] - http://soccernet.espn.go.com/stats/_/league/eng.1/year/2010/barclays-premier-league?cc=5901 [2] - http://www.whoscored.com

41. Agenda 1.Context 2.Problem & Need Statement 3.Design Alternatives 4.Simulation 5.Simulation Output 6.Utility Analysis 7.Conclusions 8.Management 41

42. Simulation - Call Accuracy Results 42 Average 2000 games per profile

43. Simulation Results - Regression Accuracy (Fitness, GFU): 0.713491 + 0.000923486 *Fitness + 1.28791e-005*GFU 6.4846e-005*Fitness^2 + 1.12504e-006*GFU^2 + 1.26193e- 006*Fitness^3- 6.75305e- 009*Fitness^4 43 R-Sq = 99.51% Fitness, GFU nonlinear No interaction (p = 0.813)

44. Agenda 1.Context 2.Problem & Need Statement 3.Design Alternatives 4.Simulation 5.Simulation Output 6.Utility Analysis 7.Conclusions 8.Management 44

45. Defining Referees for Utility Analysis Referees are defined as a combination of two independent traits (Fitness, GFU) Each trait is scaled from worst (0) to best (100) possible The distribution of referees for each trait is Normal at mean 50 and st. dev 15 45 Call accuracy for each referee defined using Call Accuracy Regression

46. Utility Analysis Method – Monte Carlo 5000 Referees (Junior level) were generated. For each alternative, a cutoff was defined on each attribute assessed where if a referee preformed above the cutoff on all attributes, he would be selected by program. Cutoff developed using Normal CDF to ensure top 100 referees selected 46 Alternatives assumed to have perfect ability to identify if referees make the cutoff AlternativeAttributesCutoff Avg. # Referees Chosen Fitness TestFitnessFitness > 81 97 Game Flow EvaluationGFUGFU > 8197 Combined EvaluationFitness, GFU Fitness >66 & GFU > 66 102 No AssessmentN/A 100

47. Analysis Method – Monte Carlo  For each alternative, referees are identified that meet the selection cutoff.  The average call accuracy of referees selected (% correct calls) is used to determine alternative utility. 47

48. Utility Analysis Results 48 AlternativeCutoffAvg. Call Accuracy 95 % Half-Width Call Accuracy Fitness Test Fitness > 81 0.749260.00012 Game Flow Evaluation GFU > 81 0.726930.00028 Combined Evaluation Fitness >66 & GFU > 66 0.741740.00021 No Assessment N/A 0.720990.00004 Based on n = 30 trials

49. Agenda 1.Context 2.Problem & Need Statement 3.Design Alternatives 4.Simulation 5.Simulation Output 6.Utility Analysis 7.Conclusions 8.Management 49

50. Alternative Cost vs. Benefit 50 “Fitness Test” dominates all other assessment based alternatives.

51. Recommendations for MDCVSRP Recommendation: It is not cost effective to implement assessments on junior referees within MDCVSRP. 51 Fitness Test vs. No Assessment (status quo) Marginal Cost Fitness Test:$26,990 Marginal Utility Fitness Test: Accuracy improvement of 2.8% for top 100 referees identified

52. Further Findings – Impact of Teams 52 Impact of different team strategies on game flow has noteworthy effect on referee performance

53. Impact of Teams – Call Distance 53 Same Referee Profile (GFU = 50, Fitness = 50) 500 Simulated games (30,000 calls) per team combination Team combination has substantial effect on distribution of call distances.

54. Additional Findings – Recommendation for USSF When comparing the quality of multiple referees based on in-game performance, match difficulty in terms of game flow and team combination must be taken into consideration. 54

55. Agenda 1.Context 2.Problem & Need Statement 3.Design Alternatives 4.Simulation 5.Simulation Output 6.Utility Analysis 7.Conclusions 8.Management 55

56. Work Breakdown Structure 56

57. Work Breakdown: Systems 490 57

58. Work Breakdown: Systems 495 58

59. Budget 59

60. Earn Value Management 60 Cost Performance Index =.9289 Schedule Performance Index =.954

61. Sponsor Testimony “ The analysis done by the students has been incredibly eye-opening. They have changed the way our management at MDCVSRP think about referee development and where to use our budget.” -Pat Delaney MDCVSRP Chairman 61