1. MHV Usability Testing Neale R. Chumbler, PhD, Jason Saleem, PhD, David Haggstrom, MD, MAS VA HSR&D CIEBP and Stroke QUERI

2. Acknowledgements Alissa Russ, PhD, Josette Jones, PhD, Scott Russell, BS, Wanda Hines The research reported/outlined here was supported by the Department of Veterans Affairs, Veterans Health Administration, Health Services Research and Development Service (SHP 08-192). Dr. Chumbler is the Co-PI of the HSR&D CIEBP at Roudebush VAMC.

3. Personal Health Records Holds great promise in promoting patient- centered care Very few studies have tested the efficacy or implementation of PHR systems.Very few studies have tested the efficacy or implementation of PHR systems. WSJ/Harris---75% of Americans would communicate with their physicians if given means to do so 2 nd survey --- 60% of patients would look up test results & track medication use if records were available. Kaelber et al., 2008

4. Personal Health Records  VA’s PHR, MHV, has reported satisfaction survey results. Many positive findingsMany positive findings Room for improvement in:Room for improvement in: Ease of navigationEase of navigation Need for more usability testingNeed for more usability testing

5. AIMS  Identify factors impacting patient use of MHV  Explore how to engage patients in the best use and navigation of MHV  Usability testing is congruent with the MHV Evaluation framework foci

6. MHV Evaluation Framework  MHV framework has adopted the RE-AIM Performance Framework as a tool to organize and synthesize work on MHV.  Our work is congruent:  Functionality  Navigation  Search

7. What is Usability  Usability measures the quality of a user’s experience when interacting with a product or system  Its not a single one-dimensional property. It has been associated with five attributes:  LearnabilityLow error rate  EfficiencySatisfaction  Memorability

8. What is Usability Testing  Usability testing is carrying out experiments to find out specific information about a design.  Evaluating a single design  Comparing two or more designs  Determining if a design works in real life (ideally done prior to full implementation)  Meant for rapid feedback to design team so changes can be made iteratively throughout the design cycle.

9. Factors Influencing Implementation of MHV  Phase I --- semi-structured interviews with 8 patients who previously used MHV  How often do you use MHV?  Are their specific things that prevent you form using MHV more frequently?  What type of things do you use MHV for?  Have you experienced frustration in finding information in MHV?  Are there specific functions in MHV that you find difficult to use?

10. Usability Test of MHV  HCI/IT lab…n = 24…simulate a typical setting that a patient would access MHV.  Qualitative interviews helped inform the design of the usability scenarios  For example…  Prescription refill (10 minutes)

11. Usability Metrics  Quantitative ---  1) Efficiency (time to complete scenarios) through time stamped Morae digitial video recordings 2) Error rates (observed deviations from intended use of MHV) 3) Satisfaction (7-point Likert Scale)

12. Initial Usability Results (In Progress)  Rx refill time on task  No significant difference in actual time task performance (mean = 209 sec) for 1 Rx refill and target task performance (240 sec; 4 min) for first Rx refill task  Actual task performance (mean = 102 sec) for same Rx refill task after second run was significantly lower than target task performance (240 sec; 4 min); p=.007 

13. Initial Usability Results (In Progress)  Participants rated two usability statements related to error messages and recovery from mistakes significantly lower than target usability levels (target = 5 on a 7 point Likert-type scale where 1 = strongly disagree and 7 = strongly agree)  The system gives error messages that clearly tell me how to fix problems. (mean rating = 3.2), p-value =.001  Whenever I make a mistake using the system, I recover easily and quickly. (mean rating = 3.5), p-value =.001

14. Lessons Learned  Usability testing can provide important and prompt design feedback to increase user satisfaction through participation of actual end users.  Collaboration with central office partners (Kim Nazi) increases value of work to VA.  A priori usability targets, like a priori hypotheses, are of greatest value from study design standpoint.  Basic steps in application design should not be overlooked  e.g., log-in functions

15. Plans for Moving Ahead  Consider usability scenarios for disease- specific conditions  e.g., cancer, stroke, diabetes  Further integrate MHV usability research agenda with work of MHV design team