1. Amanda Recker Jamie Pina, MSPH, PhD Barbara L. Massoudi, MPH, PhD RTI International 2013 International Symposium on Human Factors and Ergonomics in Health Care: Advancing the Cause March 11, 2013 Long-term Usability Testing for Public Health Information Technology: BioSense 2.0

2. BioSense 1.0: Web-based syndromic surveillance Mandated in the Public Health Security and Bioterrorism (BT) Preparedness and Response Act of 2002 Nationwide integrated system for early detection and assessment of potential BT-related illness 2003 Funding provided by Congress to CDC Development of BioSense infrastructure started, initial focus on: – VA and DoD – Direct reporting to CDC of detailed clinical data by civilian hospitals Began soliciting more limited data from health departments (HDs) that had already established automated systems for ED- based syndromic surveillance – By 2007, 8 state/local HDs connected

3. Recommendations from Prior Evaluations Strengthen state and local public health engagement – Enhance state/local HD syndromic surveillance capacity – Increase participation of state/local HD syndromic surveillance systems (improve coverage) – Share data with HDs from hospitals reporting directly to CDC – Share governance with public health community Leverage investments in electronic health records (EHRs) GAO, 2008: Adopt an “open, distributed computing model” Improve utility of the data and data sources Preparedness role: Greater “all hazards” emphasis Expand uses for broader spectrum of public health concerns

4. BioSense 2.0: Timeline June 2010: Redesign begins November 2011: Opened for business November 2011 – June 2013: Onboarding new jurisdictions – 35 jurisdictions signed the Data Use Agreement (DUA) – 17 fully onboarded April 2012: Retired BioSense 1.0

5. BioSense 2.0: Approach Shift from a need-to-know to a need-to-share and co-create approach User-centered design – Stakeholders engaged in every step of the redesign – HDs fully control “their data” at the level of granularity they choose – More options for data sharing with other jurisdictions and CDC Alignment with ONC and Meaningful Use – Agreed-upon core syndromic surveillance data elements – Collaborations with public health professional associations – Funding to states: Meaningful Use syndromic surveillance adoption, build capacity, join BioSense 2.0 Cloud technology: distributed, easy to adopt, cost effective, secure

6. Application Home Page

7. Encephalitis, Meningitis, WNV = CNS Inflammatory Disease

8. Why Long-term Usability Testing? Long user-centered design lifecycle Expectation management Stakeholder ownership BioSense 2.0 continuously changing and growing Longitudinal usability testing – User satisfaction – Efficiency Functions not changing: building a query, viewing results, analyzing the data, sending and saving information Two approaches to testing

9. 1. User-Centered Design Qualitative data collection methods The user knows best Test the right participants Broad range of public health professionals – Public health generalist – less sophisticated users – Syndromic surveillance epidemiologist – more sophisticated users

10. 2. Activity-Centered Design Quantitative data collection methods Behavior vs. opinions Based on empirical data – Time-on-task analysis – Mouse-click analysis – Pathway analysis

11. 1.Establish requirements 2.Design alternatives 3.Develop prototype 4.Conduct evaluation Basic Activities in Interaction Design (Rogers, Sharp, & Preece, 2011)

12. How to Choose Users Interact directly with the system – Epidemiologists, state and local public health professionals Manage direct users – decision-makers – Public health directors Use similar syndromic applications

13. User-centered design – Open question response – Focus groups – Expectation testing – SUS Expert evaluation – Usability heuristics (Nielsen) Activity-centered design – Morae software – Scenarios and tasks – Closed/open question response – Time on tasks – Mouse-click analysis – Pathway analysis – Critical incidents Generating Design Alternatives

14. Scenarios and Tasks Scenario: “Over dinner at [a public health conference] an argument has erupted, but luckily as a BioSense 2.0 user you can settle this dispute. Health authorities in Virginia suspect that the flu season was more severe than it was in Michigan.” Task: “Please determine which state, Virginia or Michigan, had more cases of influenza-like illness (ILI) starting in October 1, 2010 through March 1, 2011.”

15. Pathway Efficiency Analysis Time on task Mouse-click analysis Pathway analysis

16. Design Alternatives

17. How to Choose Among These Alternatives? If one person says something is a problem, do you change the design? Resolved conflicting alternatives Conducted feasibility analysis In the end…relied on face validity

18. What Happens After Design Changes? User training through webinars and videos Expert user testing Focus group sessions Continually comparing SUS scores

19. Future Activities Incorporate eye tracking into testing protocols Conduct on-site testing and evaluation Task analyses of routine versus event surveillance Information models for routine and event surveillance