1. Reliable Probability Forecasting – a Machine Learning Perspective David Lindsay Supervisors: Zhiyuan Luo, Alex Gammerman, Volodya Vovk

2. Overview What is probability forecasting? Reliability and resolution criteria Experimental design Problems with traditional assessment methods: square loss, log loss and ROC curves Probability Calibration Graph (PCG) Traditional learners are unreliable yet accurate! Extension of Venn Probability Machine (VPM) Which learners are reliable? Psychological and theoretical viewpoint

3. Probability Forecasting Qualified predictions important in many applications (especially medicine). Most machine learning algorithms make bare predictions. Those that do make qualified predictions make no claims of how effective the measures are!

4. Probability Forecasting: Generalisation of Pattern Recognition Goal of pattern recognition = find the best label for each new test object. Example Abdominal Pain Dataset: Training Set to learn from Label Diagnosis Object Patient Details Name: David Sex: M Height: 62 Appendicitis Name: Daniil Sex: M Height: 64 Dyspepsia Name: Mark Sex: M Height: 61 Non-specific,..., Name: Sian Sex: F Height: 58 Dyspepsia,, Name: Wilma Sex: F Height: 56 ? Test Object, what is the true label? True label unknown or withheld from learner

5. Probability Forecasting: Generalisation of Pattern Recognition Probability forecast – estimate the conditional probability of a label given an observed object learner Training set Name: Helen Sex: F Height: 56 Name: Helen Sex: F Height: 56 Name: Helen Sex: F Height: 56 Name: Helen Sex: F Height: 56 Test object ? Name: Helen Sex: F Height: 56 = 0.1 Name: Helen Sex: F Height: 56 = 0.7 Name: Helen Sex: F Height: 56 = 0.2 Name: Helen Sex: F Height: 56 etc… We want learner to estimate probabilities for all possible class labels:

6. Probability forecasting more formally… X object space, Y label space, Z = X Y example space Our learner makes probability forecasts for all possible labels Use probability forecasts to predict label most likely label:

7. Back to the plan… What is probability forecasting? Reliability and resolution criteria Experimental design Problems with traditional assessment methods: square loss, log loss and ROC curves Probability Calibration Graph (PCG) Traditional learners are unreliable yet accurate! Extension of Venn Probability Machine (VPM) Which learners are reliable? Psychological and theoretical viewpoint

8. Studies of Probability Forecasting Probability forecasting is well studied area since 1970s: Psychology Statistics Meteorology These studies assessed two criteria of probability forecasts: Reliability = the probability forecasts should not lie Resolution = the probability forecasts are practically useful

9. When an event is predicted with probability should have approx chance of being incorrect Reliability a.k.a. well calibrated, Considered an asymptotic property. Dawid (1985) proved no deterministic learner can be reliable for all data – still interesting to investigate This property is often overlooked in practical studies!

10. Definition of Reliability

11. Resolution Probability forecasts are practically useful, e.g. they can be used to rank the labels in order of likelihood! Closely related to classification accuracy - common focus of machine learning. Separate from reliability, i.e. do not gohand in hand (Lindsay, 2004)

12. Back to the plan… What is probability forecasting? Reliability and resolution criteria Experimental design Problems with traditional assessment methods: square loss, log loss and ROC curves Probability Calibration Graph (PCG) Traditional learners are unreliable yet accurate! Extension of Venn Probability Machine (VPM) Which learners are reliable? Psychological and theoretical viewpoint

13. Experimental design Tested several learners on many datasets in the online setting: ZeroR = Control K-Nearest Neighbour Neural Network C4.5 Decision Tree Naïve Bayes Venn Probability Machine Meta Learner (see later…)

14. The Online Learning Setting 2 7617?? 2 76172? Before After Update training data for learning machine for next trial Learning machine makes prediction for new example. (label withheld) Repeat process for all examples

15. Lots of benchmark data Tested on data available from the UCI Machine Learning repository: Abdominal Pain: 6387 examples, 135 features, 9 classes, Noisy Diabetes: 768 examples, 8 features, 2 classes Heart-Statlog: 270 examples, 13 features, 2 classes Wisconsin Breast Cancer: 685 examples, 10 features, 2 classes American Votes: 435 examples, 16 features, 2 classes Lymphography: 148 examples, 18 features, 4 classes Credit Card Applications: 690 examples, 15 features, 2 classes Iris Flower: 150 examples, 4 features, 3 classes And many more…

16. Programs Extended the WEKA data mining system implemented in Java: Added VPM meta learner to existing library of algorithms Allow learners to be tested in online setting. Created Matlab scripts to easily create plots (see later)

17. Results, papers and website All results that I discuss today can be found in my 3 tech reports: The Probability Calibration Graph - a useful visualisation of the reliability of probability forecasts, Lindsay (2004), CLRC-TR-04-01 Multi-class probability forecasting using the Venn Probability Machine - a comparison with traditional machine learning methods, Lindsay (2004), CLRC-TR- 04-02 Rapid implementation of Venn Probability Machines, Lindsay (2004), CLRC-TR-04-03 And on my web site: http://www.david-lindsay.co.uk/research.html

18. Back to the plan… What is probability forecasting? Reliability and resolution criteria Experimental design Problems with traditional assessment methods: square loss, log loss and ROC curves Probability Calibration Graph (PCG) Traditional learners are unreliable yet accurate! Extension of Venn Probability Machine (VPM) Which learners are reliable? Psychological and theoretical viewpoint

19. Loss Functions Square loss Log loss There are many other possible loss functions… Degroot and Feinberg (1982) showed that all loss functions measure a mixture of reliability and resolution Log loss punishes more harshly: forced to spread its bets

20. ROC Curves Naïve Bayes on the Abdominal pain data set 1.Graph shows trade off between false and true positive predictions 2.Want curve to be as close to the upper left corner as possible (away from diagonal) 3.My results show that this graph tests resolution. 4.Area under curve provides measure of quality of probability forecasts.

21. Table comparing traditional scores VPM C4.5 Naïve Bayes VPM Naïve Bayes 10-NN 20-NN C4.5 Neural Net 30-NN VPM 1-NN 1-NN 0.76 (1) 0.72 (5) 0.75 (2) 0.54 (10) 0.55 (9) 0.57 (8) 0.75 (3) 0.74 (4) 0.61 (6) 0.59 (7) 0.49 (11) 0.8 (4) 1.3 (7) 0.6 (1) 2.6 (10) 2.2 (9) 3.3 (11) 0.72 (2) 0.73 (3) 0.9 (5) 2.1 (8) 1.1 (6) 0.54 (5) 0.50 (4) 0.44 (1) 1.0 (11) 0.96 (10) 0.67 (7) 0.45 (2) 0.47 (3) 0.58 (6) 0.73 (8) 0.74 (9) 40.7 (8) 29.2 (2) 28.9 (1) 33.4 (4) 39.6 (7) 30.5 (3) 34.3 (5) 41.6 (9) 34.6 (6) 55.6 (10)ZeroR PCGROC Area Log Loss Sqr Loss ErrorAlgorithm

22. Problems with Traditional Assessment Loss functions and ROC give more information than error rate about the quality of probability forecasts. But… loss functions = mixture of resolution and reliability ROC curve = measures resolution Dont have any method of solely assessing reliability Dont have method of telling if probability forecasts are over- or under- estimated

23. Back to the plan… What is probability forecasting? Reliability and resolution criteria Experimental design Problems with traditional assessment methods: square loss, log loss and ROC curves Probability Calibration Graph (PCG) Traditional learners are unreliable yet accurate! Extension of Venn Probability Machine (VPM) Which learners are reliable? Psychological and theoretical viewpoint

24. Inspiration for PCG (Meteorology) Murphy & Winkler (1977) Calibration data for precipitation forecasts Reliable points lie close to diagonal

25. A PCG plot of ZeroR on Abdominal Pain Predicted Probability Empirical frequency of being correct Line of calibration PCG coordinates Reliability PCG coordinates lie close to line of calibration i.e. ZeroR may is not accurate but it is reliable! Plot may not span whole axis – ZeroR makes no predictions with high probability

26. PCG a visualisation tool and measure of reliability Total2764.5 Mean0.0483 Standard Deviation0.0757 Max0.4203 Min4.9e-17 Naïve BayesVPM Naïve Bayes VPM is reliable as PCG follows the diagonal! Total496.7 Mean0.0087 Standard Deviation0.0112 Max0.1017 Min9.2e-8 Over and under estimates its probabilities – much like real doctors! Unreliable, forecast of 0.9 only has 0.55 chance being right! (over estimate) Unreliable, forecast of 0.1 only has 0.3 chance being right! (under estimate)

27. Learners predicting like people! Naïve BayesPeople Lots of psychological research people make unreliable probability forecasts

28. Back to the plan… What is probability forecasting? Reliability and resolution criteria Experimental design Problems with traditional assessment methods: square loss, log loss and ROC curves Probability Calibration Graph (PCG) Traditional learners are unreliable yet accurate! Extension of Venn Probability Machine (VPM) Which learners are reliable? Psychological and theoretical viewpoint

29. Table comparing scores with PCG 838.1 (4)0.76 (1)0.8 (4)0.54 (5)40.7 (8)VPM C4.5 2764.5 (7)0.72 (5)1.3 (7)0.50 (4)29.2 (2)Naïve Bayes 496.7 (1)0.75 (2)0.6 (1)0.44 (1)28.9 (1)VPM Naïve Bayes 5062.9 (11)0.54 (10)2.6 (10)1.0 (11)33.4 (4)10-NN 4492.7 (10)0.55 (9)2.2 (9)0.96 (10)33.4 (4)20-NN 3481.2 (8)0.57 (8)3.3 (11)0.67 (7)39.6 (7)C4.5 1320.5 (6)0.75 (3)0.72 (2)0.45 (2)30.5 (3)Neural Net 921.2 (5)0.74 (4)0.73 (3)0.47 (3)34.3 (5)30-NN 554.6 (2)0.61 (6)0.9 (5)0.58 (6)41.6 (9)VPM 1-NN 4307.5 (9)0.59 (7)2.1 (8)0.73 (8)34.6 (6)1-NN 678.6 (3)0.49 (11)1.1 (6)0.74 (9)55.6 (10)ZeroR PCGROC Area Log Loss Sqr Loss ErrorAlgorithm

30. Correlations of scores Inverse No-0.1ROC vs. Sqr Reliability Direct Weak0.26PCG vs. Error Direct No0.04PCG vs. Sqr Resolution Direct Strong0.76PCG vs. Sqr Reliability InterpretationCorr. Coeff.Scores Inverse Moderate-0.52ROC vs. Error Direct Strong0.67ROC vs. Sqr Resolution

31. Back to the plan… What is probability forecasting? Reliability and resolution criteria Experimental design Problems with traditional assessment methods: square loss, log loss and ROC curves Probability Calibration Graph (PCG) Traditional learners are unreliable yet accurate! Extension of Venn Probability Machine (VPM) Which learners are reliable? Psychological and theoretical viewpoint

32. What is the VPM meta-learner? Volodyas VPM 1.Predicts a label 2.Produces upper u and lower l bounds for predicted label only My VPM extension 1.Extracts more information 2.Produces probability forecast for all possible labels 3.Predicts a label using these probability forecasts. 4.Produces Volodyas bounds as well! Learner Γ VPM meta learning framework VPM sits on top of existing learner to complement predictions with probability estimates

33. Volodyas original use of VPM Online Trial Number Error rate and bounds 22.1%1414.1Low Error 28.9%1835Error 34.7%2216.5Up Error Upper (red) and lower (green) bounds lie above and below the actual number of errors (black) made on the data.

34. Output from VPM compared with that of original underlying learner Key: Predicted = underlined, Actual = NA 7.6e- 9 6.3e- 10 4.0e-112.2e-91.3e- 9 0.071.7e- 13 2.9e-90.935831 NA 2.2e- 4 2.2e- 7 0.20.460.162.3e- 5 0.170.019.4e-52490 NA 1.3e- 4 4.1e- 10 3.4e-34.2e-30.994.4e- 5 3.3e- 6 4.5e-63.08e- 9 1653 Naïve Bayes LowUp Dysp.Ren al. PancrIntest obstr CholiNon. Spec Perf. Pept. Div.Appx BoundsProbability forecast for each class label Trial # 0.410.680.01 0.00.01 0.420.00.01 0.53 5831 0.070.710.40.090.080.150.050.070.100.030.022490 0.080.820.090.010.040.00.730.080.030.00.031653 VPM Naïve Bayes

35. Back to the plan… What is probability forecasting? Reliability and resolution criteria Experimental design Problems with traditional assessment methods: square loss, log loss and ROC curves Probability Calibration Graph (PCG) Traditional learners are unreliable yet accurate! Extension of Venn Probability Machine (VPM) Which learners are reliable? Psychological and theoretical viewpoint

36. ZeroR Heart DiseaseLymphography Diabetes ZeroR outputs probability forecasts which are mere label frequencies ZeroR predicts the majority class label at each trial. Uses no information about the objects in its learning – the simplest of all learners. Accuracy is poor, but reliability is good.

37. K-NN 10-NN 20-NN 30-NN K-NN finds subset of K closest (nearest neighbouring) examples in training data using a distance metric. Then counts the label frequencies amongst this subset. Acts like a more sophisticated version of ZeroR that uses information held in the object. Appropriate choice of K must be made to obtain reliable probability forecasts (depends on data).

38. Traditional Learners and VPM Traditional learners can be very unreliable (yet accurate) - depends on data. My research shows empirically that VPM is reliable. And it can recalibrate a learners original probability forecasts to make them more reliable! Improvement in reliability often without detriment to classification accuracy. Naïve Bayes VPM Naïve Bayes C4.5 VPM C4.5 Neural Net VPM Neural Net 1-NN VPM 1-NN

39. Back to the plan… What is probability forecasting? Reliability and resolution criteria Experimental design Problems with traditional assessment methods: square loss, log loss and ROC curves Probability Calibration Graph (PCG) Traditional learners are unreliable yet accurate! Extension of Venn Probability Machine (VPM) Which learners are reliable? Psychological and theoretical viewpoint

40. Psychological Heuristics When faced with the difficult task of judging probability, people employ a limited number of heuristics which reduce the judgements to simpler ones: Availability - An event is predicted more likely to occur if it has occurred frequently in the past Representativeness - One compares the essential features of the event to those of the structure of previous events Simulation - The ease in which the simulation of a system of events reaches a particular state can be used to judge the propensity of the (real) system to produce that state.

41. Interpretation of reliable learners using heuristics ZeroR, K-NN and VPM learners are reliable probability forecasters. Can identify heuristics in these learning algorithms Remember psychological research states: More heuristics More reliable forecasts

42. Psychological Interpretation of ZeroR The simplest of all reliable probability forecasters uses 1 heuristic: The learner merely counts labels it has observed so far, and uses the frequencies of labels as its forecasts (Availability)

43. Psychological Interpretation of K-NN More sophisticated than the ZeroR learner, the K-NN learner uses 2 heuristics: Uses the distance metric to find subset of K closest examples in training set. (Representativeness) Then counts the label frequencies in the subset of K-nearest neighbours to makes its forecasts (Availability)

44. Psychological Interpretation of VPM Even more sophisticated the VPM meta- learner uses all 3 heuristics: The VPM tries each new test example with all possible classifications (Simulation) Then under each tentative simulation clusters training examples which are similar into groups (Representativeness) Finally the VPM calculates the frequency of labels in each of these groups to make its forecasts (Availability)

45. Theoretical justifications ZeroR can be proven to be asymptotically reliable (but experiments show well in finite data) K-NN has lots of theory Stone (1977) to support its convergence to true probability distribution VPM has a lots of theoretical justification for finite data using martingales

46. Take home points Probability forecasting is useful for real life applications especially medicine. Want learners to be reliable and accurate. PCG can be used to check reliability. ZeroR, K-NN and VPM provide consistently reliable probability forecasts. Traditional learners Naïve Bayes, Neural Net and Decision Tree can provide unreliable forecasts. VPM can be used to improve reliability of probability forecasts without detriment to classification accuracy.

47. Supervision Alex Gammerman Volodya Vovk Zhiyuan Luo Mathematical Advice Daniil Riabko Volodya Vovk Teo Sharia Proofreading Zhiyuan Luo Siân Cox Graphics & Design Siân Cox Catering Siân Cox Fin Acknowledgments

48. What next? Look at applications in bioinformatics and medicine – noisy data really needs reliable probability forecasts so user can know whether to trust predictions! Results with time series data. Investigate further relationships with psychology. Recursive application of VPM to improve reliability and accuracy.