1. Innovative Use of Multiple Methods to Assess Health Informatics System Standards APHA 2013 -- Boston, MA November 6, 2013 Linda Piccinino, Andrew Westdorp, Jennifer Kwan, Lydia Rogers, Jean Wilson, & Sue Griffey

2. Presentation Outline Background Purpose Framework Evaluation Data Collection Results Conclusion 2

3. BACKGROUND 3

4. NCS Vanguard Study The National Children’s Study (NCS) – Large-scale, dynamic health study – 20-year time horizon NCS Vanguard Study – Pilot study for the NCS main study – Uses multiple information management systems (IMS) to support one research data collection effort 4

5. NCS Vanguard Study IMS Evaluation Assessment of IMS to see how well meeting the study’s needs – Current needs – Future projected needs 5

6. PURPOSE 6

7. IMS Evaluation Develop and implement an evidence-based evaluation approach Answer fundamental questions: – Does each IMS support the study’s basic functional requirements? – Does each IMS perform as expected? Leverage evaluation framework and methodology 7

8. Evaluation Phases 8 Phase 1 – Short-term: 2-3 months – High level assessment of basic functionality of IMS solutions – Completed in Fall 2012

9. Evaluation Phases (continued) 9 Phase 2 – Longer-term: 9-12 months – Identify/refine preferred practices, processes, and functionality of an IMS solution used to support NCS Vanguard protocol – Identify gaps in IMS solutions with respect to evaluation criteria

10. FRAMEWORK 10

11. Phase 1 Evaluation Framework 11

12. EVALUATION DATA COLLECTION 12

13. Phase 1 IMS Evaluation 13 Benchmarking System Characteristics Real-Case Scenario-Testing Reviewed system and user documentation Performed system stand up for available solutions Developed standardized scenarios Conducted testing using host-provided environment

14. Example: Role-Based Scenario Functions Tested AdministratorData Collector and Supervisor Supervisor/Manager  Perform Study Center setup  Assign user roles  Preload pregnancy screeners  Consent mother and child  Conduct Pregnancy Visit 1 event  Complete participant record of visit  Enter Operational Data Elements  Enter linkages  Assign cases  Manage field work  Provide time and expense information  View staff certifications  View and edit instrument data  Obtain basic cooperation rates  Obtain basic refusal rates  Obtain rates by disposition code 14

15. Phase 1 Overview: IMS Functional Requirements Tested 15 IMS Functional Requirements Doc. Review Scenario Testing Architecture 1. Is the IMS system open-source?√√ 2. Does the system utilize open architecture and use of modern non-proprietary programming languages? √√ 3. Does the system have an open API for easy integration with other products? √ 4. Does the system have any special hardware requirements? √√ 5. Does the system have the support 2-factor authentication? √ Case Management 1. Cases can be flagged for supervisor review √ 2. System allows for creation of new person records to be associated with each case √

16. RESULTS 16

17. Benchmarking Benchmarking results showed where systems were strongest and where they needed additional functionality 17

18. Review of Documentation on IMS Characteristics 18 CategoryIMS 1IMS 2IMS 3 Documentation AttributesAll General System Characteristics≥ ½All< ½ Architecture < ½≥ ½ Case Management ≥ ½All≥ ½ IMS Performance < ½≥ ½< ½ IMS Data Collection < ½All≥ ½ Data Management < ½≥ ½ CATI < ½≥ ½ Sample Tracking* < ½ AllAll characteristics documented ≥ ½Half or more characteristics documented < ½Less than half of characteristics documented *Few sites collecting biospecimens at the time.

19. Real-Case Scenario Functional Testing Results provided: – Objective information on functionality – Qualitative input from IMS testers 19

20. Data Collector User Role Scenario Testing 20 FunctionalityIMS 1IMS 2IMS 3 1. Can understand how to enter data and use system based on documentation available? YYY 2. Can view prior events? YYY 3. Can view contact records? YYY 4. Next event needed is indicated in system? YYN 5. Can schedule events? YYY 6. Requires Pregnancy Screener records to allow consent entry? NNn/a 7. Can enter Study consent information for mother? YYY 8. Can view Study consent information for mother? YYY

21. Functionality of IMS Solution Characteristics > ½More than half of characteristics tested were functional ≤ ½Half or fewer of characteristics tested were functional NCharacteristics were not observed 21 IMS 1IMS 2IMS 3 Case Management> ½ IMS Performance> ½ IMS Data Collection> ½ Data Management> ½N≤ ½

22. Translating Results into Use Results used to: – Identify high-level areas already meeting requirements – Identify specific gaps or areas needing improvement – Inform Phase 2 assessment 22

23. CONCLUSION 23

24. NCS Vanguard Study IMS Evaluation Framework – Grounded in theory Collaboration – Involving and communicating with multiple stakeholders – Critical when testing systems Innovation – Moved beyond traditional IT evaluation to include aspects of program evaluation – Implemented ways to test systems, not users, using protocol-driven real-life standardized scenarios 24

25. Thank You! Authors Linda Piccinino, Senior Evaluator, SSS, lpiccinino@s-3.com Andrew Westdorp, Center Director, SSS, awestdorp@s-3.com Jennifer Kwan, Evaluation Team Co-Lead, Program Office, National Children’s Study, Eunice Kennedy Shriver National Institute of Child Health and Human Development, jennifer.kwan@nih.gov Lydia Rogers, Project Director, SSS, lrogers@s-3.com Jean Wilson, Senior Study Director, SSS, jwilson@s-3.com Sue Griffey, Vice President and Director, Evaluation Center, SSS, sgriffey@s-3.com 25