Build Your Own Forecaster 03/23/11 Build Your Own Forecaster Immunization Project Nathan Bunker 03/29/2011

Build Your Own Forecaster 03/23/11 Build Your Own Forecaster Design model Write software Populate meta data Integrate software Identify test cases Test forecaster Make changes Repeat steps 5 – 7

03/23/11

Step 1: Design Model Break vaccines down into antigens 03/23/11 Step 1: Design Model Break vaccines down into antigens Break schedule down to dose states that indicate which vaccination dose is due next Vaccinations given on schedule transition a patient through the states Measles MMR Mumps Rubella Protected Dose 3 Dose 2 Dose 1 Born

Initial Design for MMR in 2009 03/23/11 Initial Design for MMR in 2009 MMR No doses given yet!

Initial Design for PCV-7 in 2009 03/23/11 Initial Design for PCV-7 in 2009 PCV-7

03/23/11 MMR

Step 2: Write Software Create core forecasting software 03/23/11 Step 2: Write Software Create core forecasting software Should support these activities: forecaster recommendation report reminder/recall activities vaccination rate assessment Forecast should: Evaluate vaccines given as invalid/valid Set dates for next vaccine dose due Identify missed vaccination opportunities

03/23/11 Raw Forecast Output

Step 3: Populate Meta Data 03/23/11 Step 3: Populate Meta Data Establish vaccination expert team Train expert team on the model Technical team must be involved Expert team should consult: Pink Book ACIP CDC (MMWR, NIP Info) Immunization Action Coalition Expect a Long Process!

03/23/11 Now Put It All Together

Step 4: Integrate Software 03/23/11 Step 4: Integrate Software Immunization Information System DB Post Processing Core Forecaster Meta Data Test System Core Forecaster Meta Data

03/23/11 Not Done Yet

Step 5: Identify Test Cases 03/23/11 Step 5: Identify Test Cases Assemble test cases Patient DOB Vaccination history Contraindications Expected results Considering rotavirus and PCV Forecaster changes with patient age Possible date types: relative and specific Specific dates must "aged" periodically

Step 6: Test Forecaster Create testing system that can: 03/23/11 Step 6: Test Forecaster Create testing system that can: Create forecast recommendation Provide forecast result explanation Compare expected and actual results Track pass-fail status Have space for notes and comments Support regression testing

03/23/11 Forecast Test System

03/23/11 Forecast Test System

Step 7: Make Changes Step 8: Lather, Rinse, Repeat 03/23/11 Step 7: Make Changes Create new test cases as needed Fix problems found in testing Most problems fixed in meta data Some problems fixed in forecaster Deploy changes in testing system Verify problems were fixed Perform regression testing Deploy changes to IIS Step 8: Lather, Rinse, Repeat

Integrating Forecaster with EHR 03/23/11 Integrating Forecaster with EHR There is no standard way to integrate so we explored several options: Write integrated EHR application module Create integrated EHR application module that supports a web services interface and create forecasting web service engine Create an HL7 query interface to allow EHR to query for forecast Create simple web link in EHR, display forecast results in web browser

03/23/11 Lessons Learned

03/23/11 Lessons Learned Creating a forecasting algorithm itself is relatively simple Creation and maintenance of meta data is a much bigger effort The technical guidance from ACIP covers expected scenarios; the forecaster must cover all potential scenarios

03/23/11 Lessons Learned Consistent process should be developed and applied to deal with gaps in recommendations Some recommendations were difficult to operationalize

Lessons Learned A robust testing system is critical: 03/23/11 Lessons Learned A robust testing system is critical: Ensures accuracy of forecaster Retains shared expert knowledge Facilitates collaboration Helps answer questions from users Helps you sleep at night!

03/23/11 What We Need

03/23/11 What We Need National expert forecast guidance to translate clinical recommendations into implementable computer algorithms National forecaster test case repository More collaboration between other forecaster development projects Common web service interface for getting forecasts from a forecaster

Thank You Expert Team Technical Team Leila Sahni, MPH Laura King, BSN 03/23/11 Thank You Expert Team Leila Sahni, MPH Laura King, BSN Rachel Cunningham, MPH Julie Boom, MD Technical Team Nathan Bunker, BS

Contact Nathan Bunker Immunization Project Texas Children's Hospital 03/23/11 Contact Nathan Bunker Immunization Project Texas Children's Hospital Nathan.Bunker@gmail.com