1. mobile data collection & monitoring:
the good, the bad and the hope for the future
Open source mobile phone-based data collection contributes to cost savings, improved data quality and efficiency of monitoring (Nepal, 2012 shown here)
Evidence-based programming is the core of most international development projects. Over the past 3 years, World Vision Canada (WV) has moved beyond paper based systems to a digital approach for monitoring and evaluation in an attempt to improve efficiency and data quality. Use includes small surveys of health facilities to large scale baseline surveys with twenty thousand participants. WV has made great strides in improving data quality and plausibility, but still face challenges with data storage, security and ownership rights, as well as meta-analysis and interface across platforms. WV has compiled lessons learned from numerous large-scale surveys conducted, leveraging different functionalities of the Open Data Kit platform. WV has tried many different methods to specifically improve data quality of anthropometry measurement including the use of digital scales, programming Z-Score calculation directly into mobile device and building in plausible data ranges, as well as increased training on mobile data collection. Increasing adoption on the use of mobile data collection by field staff has been a challenge. Strategies employed to overcome these challenges have included capacity building around survey design, the use of tutorial videos for enumerators, and development of recommended data collection policy and procedures. Overall, mobile data collection improved quality and access to real time data, and was shown to be cost-saving for WV. Given the improvement on efficiency and data quality, WV is now developing an enhanced web-based data discovery and visualization portal for the various WV grant data stores to provide transparency into activities. The aim is to improve the transparency, completeness, historical accuracy, and timeliness of data on grants via a highly usable tool that makes such data available to the international development community, meeting the need to provide high-quality evidence on maternal and child health and nutrition programming globally.

2. Child Focused Community Based Measurable programming
WV Canada works in 49 countries
Kids are at the center of all we do. If a child is thriving, everyone is thriving.
The best way to help a child is to sustainably grow their community.
We constantly track our progress and share the results with everyone involved.
World Vision is an internationally recognised non-governmental development agency with over six decades of experience, including competencies for the delivery of innovative, effective, and community-driven maternal and child health and nutrition programs. These are implemented in long-term development and humanitarian / emergency contexts. Our policy work is intended to support sustainability.

3. mobile data collection with Open Data Kit (ODK)
our challenge:
our solution:
mobile data collection with Open Data Kit (ODK)
& mHealth apps
Our challenge
 - Wasteful paper based surveys and registers with errors detected only at data analysis or reporting
- Delayed data analysis
- Inconsistent survey questionnaires limits comparability across projects
Our solution
Mobile data collection using Open Data Kit (ODK) and mHealth applications

4. mobile data process 1
train, supervise

5. mobile data process 1 Server / ODK Aggregate app / monitor form
ODK collect
monitor form
(.xml)
data (.csv)
Forms for periodic or regular monitoring is created in Excel and converted to .xml file then sent to cloud (server or ODK aggregate). Then downloaded to mobile device. Data recorded on mobile device then completed forms uploaded to cloud. Excel (.csv) file available for data analysis.
This slide illustrates points where issues occur. Even creating the questionnaire can take time, as untrained staff may alter the syntax. All forms must be field tested before being uploaded to mobile devices.
Some of our users are unfamiliar with phones., let alone smart phones. Simple and quick support is still needed.
The data flow used in a joint study between World Vision Canada and our academic partner included several levels of data security and protocol for dataflow and sharing.
Tablets for data collection use ODK collect.
Encrypted data storage
Password protected
Remote “data wipe’ functionality
2. ODK Aggregate with secure storage in cloud
Encrypted data upload
Secure data connection
Restricted access
3. Google spreadsheet/ dashboard
Encrypted data download
4. Synch
Data sharing API
5. Researcher’s computer
Data can be de-identified

6. mobile data process 3
train, supervise

7. mobile data advantages 1
saves time, $$, trees
customizable
allows complex scripts
allows mapping
encrypted & password protected data
reduces data entry error
immediate referral prompt
this innovation improves quality, saves time and money (open source) and trees
Electronic surveys offer benefits over paper-based surveys:
substantial cost savings compared to data collection on paper and manual data entry (33%), even after the initial purchase of mobile phones. This savings is multiplied as the phones are used repeatedly for other surveys. In India, we found a consistent time savings of at least 16 days per survey, or 67% total time is saved when using mobile technology for conducting a household survey.
Can be customized, allowing for self-made changes.
Complex scripts can be used for questions and responses.
GIS allows for mapping of trends Our first experience resulted in only 7% of households with recorded GPS coordinates. This was due to the 30 minute time frame for the phone to synchronise coordinates. Repeated surveys with android version 4+ resulted in 100% of sites recording GPS coordinates quickly and accurately
Security is enhanced with encrypted data storage and password protected questionnaire/ forms and data.
pre-programmed data range limits, prompts and skip patterns at point of data collection contributed to improved data quality and faster data cleaning, with fewer questionnaires/data points being excluded due to error.
Prompts are immediately sent for cases requiring referral

8. mobile data advantages 2
offline data entry
daily data quality control
flag implausible data
timely decision-making
smartphone use allows photos, AV clips
database compatible with analysis programs
Electronic surveys offer benefits over paper-based surveys:
Offline data entry can be done, with upload of data when in are of wifi.
Daily checking of data online allows for timely correction of errors.
Implausible data is flagged, providing immediate prompt for correction—with some constraints only allowing feasible responses (no pregnant men)
Real-time access to datasets for multiple stakeholders allows managers to make timely programming decisions and share data.
Smart phones allow us of photographs, audio & video clips
The database can be downloaded as .csv or through google spreadsheet as .xlsx

9. mHealth monitoring
Growth monitoring & promotion (AWW/ anganwadi worker)
Timed & targeted counseling (ASHA/ accredited social health activist)
We partner with local health providers, here in India, this has been with both the MOHFW (ministry of health and family welfare) and ICDS (integrated child development services) in one block in Madhya Pradesh, covering 200,000 people. Standard registers were copied into mHealth syntax. Over 2 years, we reached 72 ASHA with TTC for 2734 mothers and 2226 U2s. With GMP, we have reached 105 AWW with for 5831 U5s. Frontline workers prefer the mobile devices over the multiple paper registers they have to carry. Incorporated messages are used for counselling, according to recorded observations.
mHealth allows for data collection and prompted counselling messages. With anthropometry embedded into syntax, specific messages are delivered at appropriate times. For example, a child with growth faltering (weight not increasing properly), the CHW has pre-coded messages to share with caregiver of child or is prompted to refer the child.
Monthly GROWTH MONITORING AND PROMOTION (GMP) among children under 5.
POSITIVE DEVIANCE (PD) HEARTH rehabilitates underweight children
COMMUNITY MANAGEMENT OF ACUTE MALNUTRITION (CMAM) is used with high levels of acute malnutrition.
INTEGRATED COMMUNITY CASE MANAGEMENT (iCCM) delivers basic medical care outside of the health facility.
TIMED AND TARGETED COUNSELLING (ttC) deploys trained CHWs for care of pregnant and lactating women and children under 2.
5 programs have been developed:
GMP—growth monitoring and promotion
PD Hearth– positive deviance hearth to rehabilitate underweight children
CMAM– community management of acute malnutrition
TTC—timed and targeted counselling
iCCM—integrated community case management

10. Nutrition applications: Anthropometry challenges
Example of Anthro
- Using WHO plausibility we coded cutoffs / error messages directly into ODK and mHealth syntax.
- Previously were many instance where we had plausibility well above the acceptable range of quality
With ODK we are able to reduce outliers (ex. > 5SD for weight or height)
Having access to daily data records allowed for quality of data to be analysed daily. Feedback to enumerator provide quickly, improving data quality. One example was rounding numbers of measurements—with daily data quality checks, this was fixed immediately. With paper-based records, this information was typically available 1-2 months after the survey was completed. ENA Plausabilty scores were improved (<10%= good)

11. Widely used in 24 countries
Our reach with ODK
Widely used in 24 countries
First use of mobile data collection was with Epi-Surveyor (now magpi.com) in 2012 in Bangladesh and Nepal.
In 2013, ODK was used in 8 countries (Bangladesh, Cambodia, India, Indonesia, Nepal, Philippines, Sri Lanka, Thailand)
design survey questionnaires, collect and upload data to a cloud-based server and download the database for analysis.
In Canada, a technical specialist retrieved the dataset online each day to verify data quality.
44 surveys were completed in the first 6 months, following pilots in the 8 countries.
Orientations and trainings of trainers were conducted. A few staff have capacity to create survey forms, eliminating the need for external consultants.
In 2017, ODK is used widely with WV in over 24 countries.
Africa (13): Burundi, Ethiopia, Ghana, Kenya, Malawi, Mali, Rwanda, Senegal, Sierra Leone, South Sudan, Tanzania, Zambia, Zimbabwe
Asia (11): Bangladesh, Cambodia, India, Indonesia, Mongolia, Myanmar, Nepal, Pakistan, Philippines, Sri Lanka, Thailand
Kenya success = 1 survey for 53 sites = standardization of questions + data aggregation + faster analysis = timely programming decisions
Grant program used 1 survey across 5 countries

12. building ODK capacity (coding, setup, server)
+ reusable template =
standard datasets
+ reduced survey cost
- limited IT capacity to create complex surveys - staff turnover
We have created video/interactive training materials to expand the scope of ODK use for periodic surveys.
Benefit:  
Open source / free
Allowed us to expand across multiple countries / regions (1 URL per country or project)
In addition to quickly obtaining the desired dataset, we found the cost of conducting survey with mobile devices to e cheaper than paper-based surveys. This saving is multiplied with repeated surveys
challenge:
costly capacity building
costly especially when there is a significant amount of staff turn over (generic surveys based on common indicators allowed countries to simply copy/paste/delete syntax specific to the modules they were implementing)

13. mobile data lessons learned 1
data loss
low/ no wifi
complex script
save data frequently
work offline
delay data upload
use smartphone
Benefit:  skip logic and custom program reduced human error. 
The improved data quality we found is due to the programmed data range limits, prompts and skip patterns at point of data collection. Thus resulted in excluding fewer questionnaires and fewer data points.
save records frequently to reduce data loss
- delay data upload in area with poor internet
- use quality smartphone for complex scripts and better performance Training on phone use critical

14. mobile data lessons learned 2
no power—reduced coverage per day
data/ phone loss
phone/ app errors
extra batteries,
solar charger
establish security protocol
consistent IT + SIM
distributing extra batteries to maintain continual use
- establish security protocol to mitigate data, phone loss
- supply consistent IT support, SIM + credit to resolve errors

15. mobile data lessons learned 3
daily quality control
faster data analysis
need AV/ GPS
correct errors immediately
analysis when data at last HH uploaded
use smartphone
Using a smart phone allows audio visual content and GPs for mapping.
The daily up-loading of data allows for real-time access to dataset for quality control for anyone with permission. This allows for timely correction of errors.
Faster compilation of data allows for faster data analysis and faster decision-making and timely programming response.

16. taking mobile data solutions to scale
benefits
simple to use
quick setup
standardize template (reusability)
allows complex scripts
challenges
low smartphone penetration
high smartphone cost
low IT capacity
Mobile data is easy and fast
Benefit: standardized templates allow for re-using survey forms that allows for multi-site or multi-country analysis
Challenge:  Startup cost (smartphone), low smartphone penetration, and limited IT staff time 

17. summary of mobile data collection
open source platform
quality data collection
design requires IT + sector specialists
funds needed for training for ODK survey design
what’s next
continued use of standardized tools
multi-country data analysis / comparison
dashboards for routine monitoring
Summary
ODK has provided WV with a platform for quality data collection, analysis and management
While the software is Open Source / Free and User Friendly, the complexity of our work necessitates high capacity users for survey design
The hidden costs – training capacity building for survey design / writing syntax and IT specialists for trouble shooting / back end technical support
What’s next with ODK
Continued use of standardized tools/questionnaires
Multi-country data analysis / comparison
Where do we go from here?
Digital Databases for routine monitoring
Report dashboards
Benefit:  simple to use, quick setup, standardize template (reusability) which opens opportunity for multi-country analysis
Challenge:  Startup cost (smartphone), low smartphone penetration.  Our rough estimate puts it at needing to do 3 baseline survey to recuperate the initial startup cost for mobile (not bad)
Example slides:
TP baseline (large 27,000 household survey)
In conclusion, we found mobile phones are promising devices to improve project data collection in resource-poor settings like Butwal, Nepal:
data consistency and quality is enhanced
program managers and decision makers have timely access to survey data
costs (financial and time) associated with data transcription, coding and entry are reduced

18. Public Health Specialist
Barbara Main
Public Health Specialist
World Vision Canada