1. Rating Systems Vs Machine Learning on the context of sports George Kyriakides, Kyriacos Talattinis, George Stefanides Department of Applied Informatics, University Of Macedonia

2. Aim of the paper Study the performance of linear algebra rating systems and machine learning methods. Evaluate the accuracy of each method. Evaluate the quality of the predictions.

3. Structure of the presentation Clarify what is rating and ranking. Explain linear algebra rating systems used in this paper. Explain machine learning methods used in this paper. Compare the rating systems and machine learning methods by predicting soccer games (English Premier League). Conclusions. Possible future work.

4. Related research Nivard van Wijk uses a Pseudo Least-Squares Estimator to predict soccer matches. Paul Kvam and Joel S. Sokol use logistic regression and markov chains to predict basketball matches. Keeneth Massey uses linear algebra to rank basketball and football teams. Search engines use machine learning to rank search results. No comparison between machine learning and linear algebra has been made. Studies on accuracy have been conducted for machine learning and linear algebra independently, but never for profitability of the methods.

5. Rating The evaluation of an object, based on some desirable criteria. For example, a car may have a 1 to 5 stars NCAP safety rating.

6. Ranking A relationship between a set of objects, such that for any two items, one is ranked higher than the other and is consequently better, assuming that the object ranked highest is the best. Rating can be used to rank a set of objects. Sorting a vector containing ratings we are effectively ranking the elements.

7. Ranking

8. Rating Systems Many systems have been proposed: Massey Colley Markov Keener Google Page Rank mHITS Elo

9. Methods used in this paper Massey Colley mHITS

10. Massey Method Proposed by Kenneth Massey in 1997 to rank NCAA (National Collegiate Athletic Association) teams. It uses a linear least squares regression to solve a system of linear equations. A rating vector is calculated, where each entry corresponds to the rating of the team.

11. Massey Method (2) First, the matrix M is generated, where M ij is the number of games teams i and j played, multiplied by -1 and M ii is the total number of games team i has played. The second step is to calculate the vector p, where p i is the total number of points scored by team i, minus the points that were scored against the team. The final step is to solve the system of equations: Mr= p where r is the vector of the ratings, so if r i > r j team i is better than team j.

12. Colley Method Proposed by astrophysicist Dr.Wesley Colley in 2001. Variation of a simple method used to rank teams, which calculated the win ratio of each team (wins divided by total games). A system of linear equations is solved in order to find a rating vector.

13. Colley Method (2)

14. mHITS(Offence-Defense model)

15. mHITS(Offence-Defense model) (2)

16. Machine Learning

17. Methods used in this paper Decision Trees Artificial Neural Networks Random Forests

18. Artificial Neural Networks Networks of nodes which accept inputs and produce an output based on an activation function. Nodes’ connections are weighted. Usually organized in layers. Each layer is a group of nodes not connected to any node of the same group. Input data is presented to the input layer, “hidden” layers process the data and the presentation layer outputs the results.

20. Artificial Neural Networks Multilayer Perceptron, which uses back propagation was used in the paper. Learning rate:0.3 Momentum:0.2 Hidden layers:3 Epochs:500 10-Fold Cross-Validation

21. Decision Tree Learning Uses a decision tree to classify/predict. Also known as classification/regression trees. Leaves represent class labels. Branches split the data into appropriate sets. Different algorithms use different criteria to split the data.

22. C4.5 Uses Information Gain to measure the quality of the split. Confidence factor: 0.7 Unpruned 10-Fold Cross-Validation

23. Random Forest A multitude of decision trees is generated Each tree is trained independently, using a different subset of the data Input data is presented to all trees The class that the highest percentage of trees produce is the output

24. Soccer

25. Ternary Result

26. Predictions Hindsight Foresight Betting

27. Implementation Weka Open-source GNU GPL Data mining software Implemented in Java Association rules, Classification, Clustering Custom Java code mHits Colley Massey Jamma Numerical linear algebra library

28. Hindsight Accuracy Method Season NeuralTreesForestMHITSColleyMassey 2008 /200952.63%71.32%97.11%51.37%53.62%38.90% 2009 / 201056.32%72.11%94.74%55.20%56.54%48.67% 2010 /201151.32%60.79%94.74%45.07%47.05%42.18% 2011 /201250.53%58.42%96.32%54.95%55.46%47.68% 2012 /201345.79%55.00%95.79%50.88%51.88%42.95%

29. Hindsight (2)

30. Confusion Matrices Confusion matrices allow the visualization of the performance of an algorithm It indicates the quality of the predictions 95% Accuracy: Class Actual AB A955 B 0 0

31. Confusion Matrices (Hindsight) Class Actual LossWinDraw Loss8722 Win51862 Draw1887 Random Forest Decision TreeANN 94.74% 72.11% 56.32% Class Actual LossWinDraw Loss65215 Win111784 Draw135231 Class Actual LossWinDraw Loss184627 Win816718 Draw115629

32. Foresight Accuracy Method Season NeuralTreesForestMHITSColleyMassey 2008 /200956.84%49.47%50.00%56.97%48.78%36.25% 2009 / 201050.00%52.63%38.42%53.05%48.17%42.10% 2010 /201146.32% 41.58%46.63%42.78%41.06% 2011 /201246.84% 37.89%53.35%46.90%45.83% 2012 /201350.53%48.74%48.42%52.40%47.70%40.54%

33. Foresight (2)

34. Confusion Matrices (Foresight) Random Forest Decision Tree ANN 37.89% 46.84% 46.84% Class Actual LossDrawWin Loss211517 Draw181213 Win312439 Class Actual LossDrawWin Loss171125 Draw13624 Win151366 Class Actual LossDrawWin Loss16433 Draw19024 Win21073

35. Making a profit Quantity VS Quality

36. Making a profit Quantity. 90% Accuracy – 1.05 average booking odds In 100 games, betting 1 unit each time: Win 90*0.05 = 4.5 Lose 10 Net profit = 10-4.5 = -5.5

37. Making a profit Quality 60% Accuracy – 2.5 average booking odds In 100 games, betting 1 unit each time: Win 60*1.5 = 90 Lose 40 Net profit = 90 - 40 = 50

38. Betting The average odds of the 5 biggest online booking companies. 1000 money units starting capital. 50 units betted each time. Each method chose the outcome it thought would occur.

39. Foresight Betting Money Sum Method Season NeuralTreesForestMHITSColleyMassey 2008 /20091368142211571802510-690 2009 / 2010816-91854480157-722 2010 /201113873661899132-707-351 2011 /2012541-559916298392252 2012 /20132010181516351133993551

40. Foresight Betting Net Profits sum

41. Conclusions Machine learning proved to be superior in hindsight predictions and prediction quality. Draws are the most difficult to predict. mHITS is the best in foresight prediction accuracy, but not in quality. Neural Networks are the most profitable of all. Random Forests constructed the best hindsight models.

42. Future Work Other Machine Learning Methods Portfolio management for betting section Other sports, where a draw is much less likely to be the outcome

43. Thank you very much for your attention!