1 Using Bayesian networks for Water Quality Prediction in Sydney Harbour Ann Nicholson Shannon Watson, Honours 2003 Charles Twardy, Research Fellow School of Computer Science and Software Engineering Monash University

2 Overview l Representing uncertainty l Introduction to Bayesian Networks »Syntax, semantics, examples l The knowledge engineering process l Sydney Harbour Water Quality Project 2003 l Summary of other BN research

3 Sources of Uncertainty l Ignorance l Inexact observations l Non-determinism l AI representations »Probability theory »Dempster-Shafer »Fuzzy logic

4 Probability theory for representing uncertainty l Assigns a numerical degree of belief between 0 and 1 to facts »e.g. “it will rain today” is T/F. »P(“it will rain today”) = 0.2 prior probability (unconditional) l Posterior probability (conditional) »P(“it will rain today” | “rain is forecast”) = 0.8 l Bayes’ Rule: P(H|E) = P(E|H) x P(H) P(E)

5 Bayesian networks l A Bayesian Network (BN) represents a probability distribution graphically (directed acyclic graphs) l Nodes: random variables, »R: “it is raining”, discrete values T/F »T: temperature, cts or discrete variable »C: colour, discrete values {red,blue,green} l Arcs indicate conditional dependencies between variables P(A,S,T) can be decomposed to P(A)P(S|A)P(T|A)

6 Bayesian networks l Conditional Probability Distribution (CPD) –Associated with each variable –probability of each state given parent states “Jane has the flu” “Jane has a high temp” “Thermometer temp reading” Flu Te Th Models causal relationship Models possible sensor error P(Flu=T) = 0.05 P(Te=High|Flu=T) = 0.4 P(Te=High|Flu=F) = 0.01 P(Th=High|Te=H) = 0.95 P(Th=High|Te=L) = 0.1

7 BN inference l Evidence: observation of specific state l Task: compute the posterior probabilities for query node(s) given evidence. Th Y Flu Te Diagnostic inference Th Flu Te Predictive inference Intercausal inference Te FluTB Flu Mixed inference Th Flu Te

8 BN software l Commerical packages: Netica, Hugin, Analytica (all with demo versions) l Free software: Smile, Genie, JavaBayes, l See appendix B, Korb & Nicholson, 2004 l Example running Netica software

9 Decision networks l Extension to basic BN for decision making »Decision nodes »Utility nodes l EU(Action) =  p(o|Action,E) U(o) o »choose action with highest expect utility l Example

10 Elicitation from experts l Variables »important variables? values/states? l Structure »causal relationships? »dependencies/independencies? l Parameters (probabilities) »quantify relationships and interactions? l Preferences (utilities)

11 Expert Elicitation Process l These stages are done iteratively l Stops when further expert input is no longer cost effective l Process is difficult and time consuming. l Current BN tools »inference engine »GUI l Next generation of BN tools? BN EXPERT BN TOOLS Domain EXPERT

12 Knowledge discovery l There is much interest in automated methods for learning BNS from data »parameters, structure (causal discovery) l Computationally complex problem, so current methods have practical limitations »e.g. limit number of states, require variable ordering constraints, do not specify all arc directions l Evaluation methods

13 Knowledge Engineering for Bayesian Networks (KEBN) 1. Building the BN »variables, structure, parameters, preferences »combination of expert elicitation and knowledge discovery 2. Validation/Evaluation »case-based, sensitivity analysis, accuracy testing 3. Field Testing »alpha/beta testing, acceptance testing 4. Industrial Use »collection of statistics 5. Refinement »Updating procedures, regression testing

14 The KEBN process

15 Quantitative KE process

16 Water Quality for Sydney Harbour l Water Quality for recreational use l Beachwatch / Harbourwatch Programs l Bacteria samples used as pollution indicators l Many variables influencing Bacterial levels – rainfall, tide, wind, sunlight temperature, ph etc

17 Past studies l Hose et al. used multi dimension scaling model of Sydney harbour »low predictive accuracy, unable to handle the noisy bacteria samples, explained 63% of bacteria variablity (Port Jackson) l Ashbolt and Bruno: »agree with Hose et al, + wind effects, sunlight hours, tide l Crowther et al (UK): »rainfall, tide, sampling times, sunshine, wind »Explained 53% of bacteria variablility l Other models developed by the USEPA to model estuaries are: »QUAL2E – Steady-state receiving water model »WASP – Time Varying dispersion model »EFDC – 3D hydrodynamic model l EPA in Sydney interested in a model applying the causal knowledge of the domain

18 EPA Guidelines TodayYesterdayDay Before Yesterday Pollution IF T>4THENLikely ELSE IF T  4AND Y  4AND DBY  4 THENUnlikely ELSE IF T  4AND Y  4AND DBY  4 THENUnlikely for 24h flushing But Likely for 48h flushing ELSELikely for all other results

19 Stages of Project l Preparation of EPA Data rainfall only l Hand-craft simple networks for rainfall data l Comparison of hand-crafted networks with range of learners (using Weka software) l Using CaMML to learn BN on extended data set 2003 Hons proj 2003/04 Summer Vac proj

20 EPA Data l Database 1: »E.coli, Enterococci (cfu/100mL), thresholds 150 & 35. »60 water samples each year since 1994 at 27 sites in Sydney Harbour. »Enterococci E.coli, Raining, Sunny, Drain running, temperature, time of sample, direction of sampling run, date, site name, beach code l Database 2: »Rainfall readings (mm) at 40 locations around Sydney

21 Data Preparation New file format: Date BeachCode Entc Ecoli D1 D2 D3 D4 D5 D6 D1 = rainfall on day of collection D6 = rainfall 5 days previously l Rainfall data had many missing entries

22 Rainfall BNs l Hand-crafted BNs to predict bacteria using rainfall only l Started with deterministic BN that implemented EPA guidelines l Looked at varying number of previous days rainfall for predicting bacteria l Investigated various discretisations of variables

23 EPA Guidelines as BN

24 Davidson BN: 1 day rainfall

25 Davidson BN: 6 days rainfall

26 Evaluation l Split data training/testing l 10 fold cross validation l Measures: Predictive Accuracy & Information Reward l Also looked at ROC curves (correct classification vs false positives) l Using Weka: Java environment for machine learning tools and techniques l Small data: 4 beaches: Chinamans, Edwards, Balmoral (all middle harbour), Clifton (Port Jackson) l Using 6 days rainfall averaged from all rain gauges

27 Predictive accuracy l Examining each joint observation in the sample l Adding any available evidence for the other nodes l Updating the network l Use value with highest probability as predicted value l Compare predicted value with the actual value

28 Information Reward l Rewards calibration of probabilities l Zero reward for just reporting priors l Unbounded below for a bad prediction l Bounded above by a maximum that depends on priors Reward = 0 Repeat If I == correct state IR += log ( 1 / p[i] ) else IR += log ( 1 / 1 - p[i] )

29 Evaluation: Weka learners l Naïve Bayes l J48 (version of C4.5) l CaMML –Causal BN learner, using MML metric l AODE l TAN l Logistic l “Davidson” BN – 6 days previous rainfall »With and without adaptation of parameters (case learning) l “Guidelines” BN – 3 days previous rainfall »Deterministic rule »With adaptation of parameters (case learning) Pr=1/3

30 Results LearnerPred AccuracyInfo Reward Prior Naïve Bayes J CaMML AODE TAN Logistic Davidson Davidson CL Guidelines (det) Guidelines CL

31 Results: ROC Curves

32 Results: area under ROC Curves Perfect0.999 AODE0.733 Logistic0.729 CaMML0.718 J Naïve0.679 Davidson CL0.645 Guidelines CL0.643 Guidelines0.637 Davidson0.620 TAN0.561 Prior0.496

33 Results: ROC Curves l For ~20% false-positive, can get ~60% of events l For ~45% false-positive, can get ~75% of events l For ~60% false-positive, can get ~80% of events l Implications? »Using current guidelines, if accept 45% false-positive, getting 60% hit rate »Can either keep that false-positive rate, get extra 15% »Or, keep same hit rate at half the false positive rate

34 Example of CaMML BN

35 Future Directions?

36

37 Early BN-related projects l DBNS for discrete monitoring (PhD, 1992) l Approximate BN inference algorithms based on a mutual information measure for relevance ( with Nathalie Jitnah, ) l Plan recognition: DBNs for predicting users actions and goals in an adventure game ( with David Albrecht, Ingrid Zukerman, ) l DBNs for ambulation monitoring and fall diagnosis ( with biomedical engineering, ) l Bayesian Poker ( with Kevin Korb, )

38 Knowledge Engineering with BNs l Seabreeze prediction: joint project with Bureau of Meteorology »Comparison of existing simple rule, expert elicited BN, and BNs from Tetrad-II and CaMML l ITS for decimal misconceptions l Methodology and tools to support knowledge engineering process »Matilda: visualisation of d-separation »Support for sensitivity analysis l Written a textbook: »Bayesian Artificial Intelligence, Kevin B. Korb and Ann E. Nicholson, Chapman & Hall / CRC,

39 Current BN-related projects l BNs for Epidemiology (with Kevin Korb, Charles Twardy) »ARC Discovery Grant, 2004 »Looking at Coronary Heart Disease data sets »Learning hybrid networks: cts and discrete variables. l BNs for supporting meteorological forecasting process (DSS’2004) (with Ph. D student Tal Boneh, K. Korb, BoM) »Building domain ontology (in Protege) from expert elicitation »Automatically generating BN fragments »Case studies: Fog, hailstorms, rainfall. l Ecological risk assessment »Goulburn Water, native fish abundance »Sydney Harbour Water Quality

40 Open Research Questions l Methodology for combining expert elicitation and automated methods »expert knowledge used to guide search »automated methods provide alternatives to be presented to experts l Evaluation measures and methods »may be domain dependent l Improved tools to support elicitation »Reduce reliance on BN expert »e.g. visualisation of d-separation l Industry adoption of BN technology