1. Applied Analytics in Business Plans Lessons learnt during the SRC15 business planning process Robert Murray – Scottish Water Analytics Team Leader – 27th February 2014

2. Our key observations 1. Ensure that presentation is clear and objective 2. Start early and respond to the changing needs of your stakeholders 3. Invest significant time in validation 5. Prepare the right level of detail 6. Address the data challenge 4. Think about how multiple sources of evidence will be used

3. 1. Ensure that presentation is clear and objective presenting statistical data - stakeholder landscape

4. Presenting Statistical Data Present data in context

5. Presenting Statistical Data Ensure that visuals are fair

6. Presenting Statistical Data Choice of colour can make a difference

7. Presenting Statistical Data Avoid chart junk

8. Presentation should enable consistent and correct interpretation across all interested parties regardless of experience and capability. - Business plans are key public documents. - Greater emphasis on customer engagement and involvement: - Customer research. - Customer Forum. - Customer education. Stakeholder landscape

9. 2. Start early and respond to the changing needs of your stakeholders demand – development model – other considerations

10. Demand Profile 2008 2007 2010 20112012 2013 2008 Period of maximum engagement Expect frequent scenario requests Diminishing opportunity to deliver big changes and degree of stability expected Expect questions about assumptions | confidence | overlaps | cause and effect

11. Demand Profile 2008 2007 2010 20112012 2013 2008 How can you maximise acceptance and use of the analysis during this time? How can you encourage early engagement and development during this time?

12. Analysis Spiral

13. 2008 2007 2010 20112012 2013 2008 Maturity Business Use Present Feedback Development Engage stakeholders every 6 months. Deliberately start with the simplest approach Focus on big details. Increase frequency of cycles to 2-3 months. Focus on small details, scenarios, testing

14. - Phase 1: - Limited need to document - Stakeholder requirements easy to track - Informal deployment of results. - Focus on developing the best solution. - Phase 2: - Produce frequent and short information papers. - Need to track emerging requirements – open / in progress / resolved - Formalise transfer of results to establish audit trail. - Track changes and keep records of why changes have been made. - Focus on developing the most appropriate solution. Other considerations

15. 3. Invest significant time in validation examples of approach taken in Scottish Water - benefits

16. - Encourage extensive internal challenge and collaboration - Undertake statistical tests. - Undertake ‘in-the-field’ studies or obtain physical observations. - Collaborate with external experts: - In the process. - Independent of process. - Review against external sources: - Literature - Available industry data Our approach to validation

17. - Testing assumptions of technique applied. - Cross-validation (leave-one-out, 10-fold, time-based), e.g. Example Statistical Tests data for testingdata for analysis All Data Output, e.g. a model FITNESS evaluation

18. Example Statistical Tests

19. Field Observations

20. University Partnerships Industry Collaboration

21. - Internal collaborationsense checks and ensures buy in and use - Statistical testsessential practice, quick and robust - Field studieseasy to understand and harder to discredit - External experts introduces independence and new capabilities - External sourcesexplores biases Benefits

22. 4. Think about how multiple sources of evidence will be used sources of information – methods

23. “How much do we need to invest in water mains renewal?” Could be investigated in a number of different ways: - Historic run-rates - End of life model - Replace mains with known problems - Comparative analysis - Economic optimisation model - Gut feel … finance constrained … and many others Water Mains Capital Maintenance

24. Jury vs. Judge Expanded evidence pool that is potentially ambiguous and lacking consensus Narrower evidence pool that is potentially more vulnerable to error or biases

25. Potential basic Jury Methods - Use the average - Use the median - Use the minimum or maximum - Use the majority vote - Use a weighted combination Or … compare the approaches and use the best estimator

26. Selecting the best estimator data for testingdata for analysis All Data Estimator 1 FITNESS 1 Estimator 2 FITNESS 2 Estimator 3 FITNESS 3

27. Selecting the best estimator

28. 5. Prepare the right level of detail effort vs. use

29. Prepare the right level of detail amount of effort to prepare extent of use Detailed Technical Reports Information / SG Papers Presentations Data Tables & Charts Walkthroughs

30. 6. Address the data challenge Our Approach

31. The Data Challenge That’s all really great, BUT how good is your data?

32. Sensitivity Testing Average Deterioration Rate : 1.009 or ~1% per year. Random Error in GIS AgeChange in Deterioration Rate 0 – 2.5%< 1% 2.5 – 10%1 – 2.7% 10 – 20%2.7 – 4.6% 20 – 30%4.6 – 7.6% 30 – 40%7.6 – 9.5% 40 – 50%9.5 – 11.6%

33. Expert Knowledge - Generally viewed as a less objective than empirical data - Situation can be greatly improved by using a structured elicitation process - Expect questions about corporate biases. - Two ways to address these concerns: - Obtain physical ‘in-field’ observations and compare. - Compare against available industry data.

34. Our key observations 1. Ensure that presentation is clear and objective 2. Start early and respond to the changing needs of your stakeholders 3. Invest significant time in validation 5. Prepare the right level of detail 6. Address the data challenge 4. Think about how multiple sources of evidence will be used