1. Embedding Evaluation into mHealth Activities: Opportunities and Challenges Kelly Ladin L’Engle, PhD, MPH Behavioral and Social Sciences FHI 360 mHealth Working Group Meeting: Innovative Monitoring and Evaluation of mHealth, October 20, 2011

2. We need more evaluation! While mHealth has great potential, current research…does not provide much evidence ror actual and wide-scale health impacts, nor answers to critical research questions…. (Mechael et al., 2010)

3. mHealth M&E Opportunities Real-time data, and lots of it Ability to automate systems Increased and additional access to program users Two examples: 1.m4RH (Mobile for Reproductive Health): opt-in, menu-based, pingpong, SMS system providing information about 9 family planning methods and clinic locations in Kenya and Tanzania 1.ECINFO: Randomized controlled trial testing timed delivery of family planning role model stories to emergency contraception users, compared to m4RH access, and control group in Ghana

4. KeywordContent # of queries KenyaTanzania m4RHMain Menu67874333 00, 01, 02, 03, 99Secondary Menus79953135 11Implants1520685 21IUDs1319650 31Permanent Methods1084620 41Injectables1467671 51OCPs1492598 61Emergency Contraception14081020 71Condoms2199748 81 Natural Methods (incl. LAM in Kenya) 31701362 91Clinic Locations1281813 Total Hits 29,72214,635 Example: Monitoring User Data from m4RH

5. Example: Automated Systems for ECINFO

6. Example: Increased Access to m4RH Users PurposeText Introduction Thanks for using m4RH! Please help us with 4 questions and stand a chance to win FREE air time. M4RH is free, voluntary, and confidential…You can use m4RH even if you don’t answer these questions. You will receive the questions shortly... Gender Please tell us your gender. Reply F if you are female, M if you are male, and send to 4128. Age Which is true for you? If you are 19 or younger reply A; If 20-29 reply B; If 30-39 reply C; If 40 or older reply D. Promotion Point How did you learn about m4RH? From a clinic, reply R; partner, friend or family member- S; community health or peer educator- T; community event- U; radio- V. Associated KAB Change How has m4RH changed your use of family planning? Reply CHANGE then your change, like ‘CHANGE got an IUD’. ReminderHello from m4RH, this is a friendly reminder to please help by answering the 4 questions sent to you earlier this week. You still stand a chance to win.

7. Open-ended Texts m4RH Started using condom with my partner Protected me from early pregnancy Got to know clinics in my region hence access of services easy and from a reliable source Got IUD that has helped me manage my only two kids with my little monthly salary from casual jobs Use 3-month injections enjoy sex like never before Got implant for wife

8. Example: Additional Access to m4RH Users 1.Continuous electronic data capture and monitoring 2.Fielding of text questions to m4RH users to assess age, gender, where they learned about m4RH 3.Potential for pre-post data collection to assess change in use of family planning methods 4.Telephone interviews to obtain feedback on system, evidence for clinic referrals and FP uptake, and m4RH “intervention map” 5.Clinical-level data collection to assess if clients visit clinic as a result of m4RH 6.Future: Conduct rigorous experiment or quasi-experiment to assess m4RH impact

9. M&E: New Methodological Challenges Analysis issues Response rates Limits on SMS data collection: – Question format and order – Data validity and reliability – Amount of data that can be collected Participant recruitment and study consent Etc.

10. CONTACT 10 Visit our m4RH website: www.fhi.org/m4RH Kelly L’Engle klengle@fhi360.org 919-544-7040