1. Department of Surgery and Cancer Imperial College London 20 May 2014 Norman Fenton Queen Mary University of London and Agena Ltd Improved Medical Risk Assessment and Decision-making with Bayesian Networks

2. Overview Why Bayes? Why Bayesian networks? Why NOT learn the models from data only? Case study Challenges and conclusions

3. 1. WHY BAYES?

4. The Harvard Problem One in a thousand people has a prevalence for a particular heart disease. A test to detect this disease has: 100% sensitivity 95% specificity If a randomly selected person tests positive what is the probability that the person actually has the disease?

5. Bayes Theorem E (evidence) We now get some evidence E. H (hypothesis) We have a hypothesis H with prior probability P(H) We know P(E|H) but we want the posterior P(H|E) P(H|E) = P(E|H)*P(H) P(E) P(E|H)*P(H) P(E|H)*P(H) + P(E|not H)*P(not H) = 1*1/1000 1*1/1000+ 5/100*999/1000 P(H|E)  = = 0.001 0.001 + 0.04995 0.0196 Waste of time showing this to most people!!!

6. Slide 6 Imagine 100,000 people

7. Slide 7 Out of whom 100 has the disease

8. Slide 8 But about 5% of the remaining 99900 people without the disease test positive. That is 4995 people

9. Slide 9 So 100 out of 5095 who test positive actually have the disease That’s just under 2% That’s very different from the 95% assumed by most medics

10. Total people 100,000 1/1000 999/1000 Have the disease 100 Don’t have the disease 99,900 So 100 out of 5,095 who test positive actually have the disease, i.e. under 2% Test positive 100 Test negative 0 Test positive 4,995 Test negative 94,905 100% 0% 5% 95%

11. 2. WHY BAYESIAN NETWORKS?

12. A Simple Bayesian Network

13. ..but here is a typical causal model Calculations from first principles are infeasible and incomprehensible

14. Actual model in medical negligence case This model already reaches limit of comprehensibility for manual calculations and event trees MRA CA Ischaemic Small aneurysm Large aneurysm CSP

15. Detected by Test 9,900 Undetected by Test 100 Detected by Test 90 Undetected by Test 10 Detected by Test 0 Undetected by Test 10,000 Die from burst/bleeding Die from CSP 99% 1% 90% 10% 50% 0% 100% 2% CA Test Pathway Cause of PalsyTest ResultOutcomeDeaths 2 0 5000 5002 TOTAL = 1.495% 1 14,952 out of 1,000,000 give risk Stroke Strokes Don’t die 99 Stroke Die from burst/bleeding Don’t dieStroke Don’t dieStroke 1% 50% 98% 99 2 98 1 1 1 0 100 0 5000 50 1% Stroke 50 9799 9950 Total people 1,000,000 Large 9,900 Small 100 CSP 10,000 Others (ischaemic) 980,000 1% 98% Aneurysm 10,000 99%

16. Total people 1,000,000 Large 9,900 Small 100 CSP 10,000 Others (ischaemic) 980,000 Detected by Test 9,405 Undetected by Test 495 Detected by Test 50 Undetected by Test 50 Detected by Test 9,000 Undetected by Test 1000 Die from burst/bleeding Die from CSP 1% 98% 95% 5% 50% 90% 10% 2% 20% MRA Test Pathway Cause of Palsy Test ResultOutcomeDeaths 10 1 1800 500 2311 TOTAL = 0.2311% 0 0 0 2311 out of 1,000,000 give risk Aneurysm 10,000 99%

17. Much better solution …use a Bayesian Network tool

18. Computation for Catheter Angiogram Mean: 9950 Mean: 5002

19. Computation for MRA Scan Mean: 0 Mean: 2311

20. The Calculator Analogy

21. No need for p-tests or classical confidence intervals Drug “Precision” weight loss: Everyone in trial lost between 4.5 and 5.5 pounds Drug “Oomph” weight loss: Everyone in trial lost between 10 and 30 pounds Which drug can we ‘accept’, i.e. reject null hypothesis of ‘no weight loss’? Classical stats provides nonsensical answers

22. No need for p-tests or classical confidence intervals

23. 3. WHY NOT LEARN THE MODELS FROM DATA ONLY?

24. A typical data-driven study AgeDelay in arrival Injury type Brain scan result Arterial pressure Pupil dilation Outcome (death y/n) 1725ANLYN 3920BNMYN 2365ANLNY 2180CYHYN 6820BYMYN 2230ANMNY ………..……

25. Delay in arrival Injury type Brain scan result Arterial pressure Pupil dilation Age Outcome A typical data-driven study Purely data driven machine learning algorithms will be inaccurate and produce counterintuitive results e.g. outcome more likely to be OK in the worst scenarios

26. Delay in arrival Injury type Brain scan result Arterial pressure Pupil dilation Age Causal model with intervention Danger level Outcome TREATMENT..crucial variables missing from the data

27. Determining drug effectiveness

28. Basic results for drug effectiveness Drug A The mean financial benefit is $4156 Drug B The mean financial benefit is $2777 Ban drug B?

29. Model with latent variable (same data) Note that most patients in the sample had minor case of the condition …and most patients were given drug A

30. Results with 'Patient condition' major Drug B 30% positive outcome. The mean financial benefit is $1000 Drug A Only 10% positive outcome. The mean financial benefit is $400

31. OK, so we might need expert judgment when we have missing data, but with good experimental design and lots of good quality data we can surely remove dependency on experts ……

32. A machine learning fable A and B are two medical conditions very well known to doctors Bill and Ludmila. These conditions are pretty rare (both have an incidence of about one in 1,000 people). There is a third medical condition C (whose name is “FiroziliRalitNoNeOba”) that Bill has heard the name of, but knows nothing about. But Bill has heard that patients with either A or B usually also have C. Bill has a massive database of 600,000 people with the details of which conditions they have.

33. Bill’s data

34. Bill’s machine learning mate Fred Can use this database to ‘discover’ the underlying causal model (Bayesian Network) relating A, B, and C. But Ludmila says she knows the correct model without data: Fred warns against this She also “knows” the probability tables

35. Fred’s learnt model Ludmilla disagrees with the last column of table C Fred: “Not enough data for that” Bill: “…why can’t we simply conclude that C must be true when both A and B are?” 600 out of 600,000 have condition A 600 out of 600,000 have condition B Every single person with condition A also has C and every single person with B also has C.

36. Ludmilla’s knowledge The name of Condition C - FiroziliRalitNoNeOba - is actually a Russian word. Its literal translation is: – ‘A person suffering from either Firoz or Ralit but not both’. – ‘Firoz’ is the Russian word for condition A and ‘Ralit’ is the Russian word for condition B.”

37. Moral of the story Sometimes you have to trust experts to provide more informed quantitative judgement than you will get from data alone. Even really big datasets will be insufficient for some very small problems. Trusting the expert can save you a whole load of unnecessary data-collection and machine learning effort.

38. 4. CASE STUDY

39. Trauma Care Case Study QM RIM Group – William Marsh – Barbaros Yet The Royal London Hospital – Mr Zane Perkins – Mr Nigel Tai – ACIT Data US Army Institute of Surgical Research – Lower Extremity Injury Data Yet, B., Perkins Z., Fenton, N.E., Tai, N., Marsh, W., "Not Just Data: A Method for Improving Prediction with Knowledge", Journal of Biomedical Informatics, 2014 Apr;48:28-37

40. BN v MESS Score Prediction: coagulopathy, death (c.f. GCS, TRISS) Flexible inputs Patient’s physiological state – Causal modelling: informed by knowledge How the BN Model Differs

41. Life Saving: Prediction of Physiological Disorders

42. Limb Saving: Prediction of Limb Viability

43. www.traumamodels.com

46. 5. CHALLENGES AND CONCLUSIONS

47. Challenges Apparent paradox on using experts Expert systems have a bad reputation Resistance to subjective priors Building new large-scale BN models, especially with minimal data Interacting with large-scale BN models Explaining the results of BN models BAYES-KNOWLEDGE (Effective Bayesian Modelling with Knowledge Before Data) www.eecs.qmul.ac.uk/~norman/projects/B_Knowledge.html

48. Conclusions (1) Purely data driven approaches using Machine learning and statistics DO NOT WORK At best captures what did happen Vs what would have happened Need to move to data + knowledge approach BNs provide the key

49. Conclusions (2): BN Benefits Data + knowledge Models uncertainty and causality Predictions and diagnosis Avoid medical statistics fixation on p-values and confidence intervals Incorporate qualitative and quantitative variables Identify causal effects without RCTs New generation expert systems

50. Blatant Plug for Book CRC Press, ISBN: 9781439809105, ISBN 10: 1439809100