1. Data Quality Assessment
Good morning and welcome to this Data Quality Session. It is great to be with you here at USAID/Jordan.
As you are aware, the Mission is in the final phases of updating its PMP. You have been working on this for sometime now and you were busy in selecting/developing indicators. Some of these indicators will be reported externally and thus they will need to undergo a DQA.
So it’s the DQA season and we think it’s a great chance to have this session at this time.
But before we go further, lets have a chance to introduce ourselves.
Ask participants to introduce themselves by name, title, and office, have you conducted a DQA before, and what do you hope to learn today.
its
Wednesday, June 10, 2015
USAID/Jordan

2. By the end of this session, participants should:
SESSION GOALS
By the end of this session, participants should:
Understand the reasons for conducting Data Quality Assessment (DQA)
Be familiar with the processes and best practice of conducting a DQA
I hope we will be able to address all your learning priorities, but what we expect you to do at the end of this session is the following:

3. DATA QUALITY ASSURANCE
Data informs decisions across the Program Cycle
High quality data is the cornerstone for evidence based decision-making
USAID’s credibility when communicating and reporting requires realistic understanding of the limitations of data
So why do we care about data quality?
Ask: any body knows?
We live in a world where data is every where. And in the development world in general and at USAID in particular this couldn’t be more true.
Data is important across all the elements and phases of the program cycle. Show the program cycle.
In addition, good decisions require good data.
More over, our credibility when we communicate to hour partners, host government, internally or externally within the agency requires that we understand the strengths, weaknesses and limitations of the data we report.
But given all this importance for data quality, what do you think about the next question?

4. Data Quality – Key Trade-Offs
TRUE OR FALSE?
The goal of data quality assessment is to eliminate all data limitations
FALSE
Cost
Quality
Data Quality – Key Trade-Offs
Quality vs. Cost
Between dimensions of quality
Even under favorable circumstances, data will never be perfect.
Data Quality is a chess game of cost versus quality. That is to say, you are always balancing between achieving the highest levels of quality while also working within the reasonable limits of cost.
The goal of data quality management is not to eliminate all data quality issues. This is not possible. The goal is to identify possible threats to the quality of data and manage them as best possible. Sometimes, the most that one can do is to note that there is a data quality issue.
Performance data should be as complete and consistent as management needs and resources permit

5. DATA QUALITY ASESSMENT (DQA) (ADS 203.3.11)
What is it? What is it’s purpose?
A process to ensure that the Mission, DO team, and IPs are aware of the:
Strengths and weaknesses of data (vs five data quality standards), and
Extent to which data integrity can be trusted to influence management decisions
Note that the process ends in a document.

6. STANDARDS FOR DATA QUALITY (VIPRT) (ADS 203.11.1)
Validity: data (and the indicator) clearly and adequately represent the intended result
Integrity: data have safeguards to minimize risk of transcription error or data manipulation
Precision: data have sufficient level of detail to permit management decision-making
Reliability: data reflect stable and consistent data collection processes and analysis methods over time and across sites/partners
Timeliness: data available at a useful frequency, are current, and timely enough to influence management decision-making
To be of use in performance monitoring and credible for reporting, data should reasonably meet these five standards of data quality:
When we refer to Data Quality, validity means that whatever is being measured, is actually what we have intended to capture or measure.

7. DATA QUALITY ASESSMENT (DQA)
What is it not?
An audit
although reported data are auditable and are routinely audited
An exercise that seeks to find fault and/or place blame
Something to fear
DQA is an opportunity for USAID and IPs to
understand results better and make improvements

8. Those reported externally to Washington.
WHO IS RESPONSIBLE?
COR/AOR/AM is responsible for conducting DQAs with the participation of the PRO M&E Specialists, for indicators in their AMEPs.
Ask the audience:
Who is responsible: Typically the person responsible for the indicator.
Which: examples (PPR indicators AND narrative, data calls, frequently quoted data, etc)
WHICH INDICATORS?
Those reported externally to Washington.
However, “while managers are not required to conduct DQAs on all performance data, they should be aware of the strengths and weaknesses of all indicators they collect to monitor performance.”
ADS

9. WHEN?
A DQA must occur for indicators that are externally reported within six months before reporting initial results, and at least every 3 years thereafter
In addition, a number of circumstances might prompt a manager to conduct DQA, if:
Critically or strategically important data
Potential indicator data issues
Previous DQA follow up
Ask the audience:
When are they required A: At least once within the three years prior to reporting to Washington.

10. COMMON DATA QUALITY ISSUES ENCOUNTERED
Lack of documentation (particularly PIRS)
IP collected data do not match USAID definition
Double-counting
Untimely data
Inconsistent/Missing data
Biased data collection procedures
Data procedures are not clear to all staff
Data procedures/tools are not standardized/consistent across Activities, sites and/or over years
Data transcription/calculation errors
Unclear/inconsistent inclusion/exclusion criteria
The Mission have conducted a few DQAs in the last year. By reviewing those and through working with IPs we have identified several common issues that might affect data quality that we wanted to share with you today.
I will try to tell you a little about each of those issues but I would like also to hear from you, particularly those who have been on a DQA team before or have reviewed DQA reports. Telll us if you have experienced the issue and how was it, but try to be as concise as possible.

11. STEPS FOR CONDUCTING DQAS
Plan
Communicate with IPs
Request/review Mission and IP documents
Identify specific areas of focus to discuss with IP
This is the story of DQAs
To conduct DQAs there are three distinctive steps: Plan, Conduct, and Analyze, Document, and Follow up.
You have seen those steps before in many activities you do. DQA is not different.
I want to emphasize here on the importance of planning in the process of conducting DQAs. Particularly communicating with IPs. As you might expect, IPs get very stressed when one of their indicators is DQAed. From our discussion with them, perhaps this is one of the most important issues that they keep mentioning. It is very important for AORs/CORs to inform their partners of which indicators are Mission required indicators, which of them will be DQAed, and when if possible.
Planning ad communicating with partners early allows them to prepare and provide you with the documentation you need to conduct a better DQA. It will allow you as well to identify specific areas of focus that you want to discuss with your partner.
So lets go through the three steps in more details one by one.
Conduct
Visit IP/field offices
Examine data collection process
Verify data compliance against VIP-RT
Analyze, Document, and Follow Up
Complete DQA checklist
Share
Implement and monitor recommendations

12. Communicate with partners/stakeholders
PLAN
Communicate with partners/stakeholders
What are the indicator(s)?
When will the DQA visit be?
Who needs to be there?
What background documents are needed?
How can/should IP prepare?
Review DQA checklist and prepare questions for how you will get the answers
Remember: (DQA Checklist is not an interview guide!)
One checklist per indicator per Activity

13. Review relevant Mission and IP documents ahead of time
PLAN
Review relevant Mission and IP documents ahead of time
PMP and AMEP, including PIRS (Mission and IP)
For new indicators have IP complete PIRS before DQA
All reports to USAID in which performance data was reported (quarterly, annual, others)
IP M&E guidelines/SOPs related to different stages of data handling (collecting, monitoring, assessing, sampling, etc.)
Previously conducted DQA reports (Mission and IP)
Identify particular areas of focus
All reports within the DQA period

14. EXERCISE 20 minutes
Review PIRS to plan for DQA and identify areas of specific focus during the DQA
Definitions
Calculation challenges
Availability of data
Potential bias
Other issues …

15. CONDUCT Conduct IP office visits (and to field office when possible)
Include IP staff involved in data handling at all stages
Ask questions! Lots of them (system vs indicator specific)
Review lots of documents! Data verification materials (original sign-in sheets, score cards, photos, inventory record, activity reports, etc.)
Assess compliance of data reported with the documented definition
Assess the data collection processes and obtain a clear picture of how data is managed from source to report
Track a reported result/value back through the system to replicate it
Compare data: in different reports, field vs reported, and hard copy vs electronic copy
Assess the data storage system
Obtain copies of documents/data you review or spot-check
Assess difference between documentation and real practice.

16. EXERCISE 20 minutes Remember: DQA checklist is not an interview guide.
Develop questions to address DQA checklist or other items. Using the same PIRS from the previous exercise and the DQA checklist, identify and practice specific questions that you might want to ask your IP during the DQA to get more information

17. If you want to know… Ask the partner…
Does the indicator actually measure what you want it to measure?
Is this indicator useful for monitoring your activity’s intended results? Why or why not?
What procedures are used for data collection and analysis?
Describe to me how data are managed from point of collection to final reporting? Show me the different tools along the way.
Are there systems to ensure consistent collection, analysis and reporting of data?
Who are all of the people that are involved in data collection and reporting? Are there different field offices or sub-contractors that are involved? How do you ensure consistency among different staff/offices?
What is the quality of data entry system to record and maintain data?
How are data checked for errors in entry, transcription or math errors?
Do data reflect the current situation?
How long does it take to go from data collection to reporting?
The DQA checklist is not an interview guide. You need to ask many more questions to complete the checklist.

18. ANALYZE, DOCUMENT, AND FOLLOW UP
Complete DQA checklist for each indicator and IP
Not just to check the boxes. Write lots of comments!
Assess the findings and determine if improvement is needed
Write up recommendations and action items for improving the quality of data where applicable
Solicit and incorporate IP responses into the completed checklist
Attach all necessary reviewed supporting documents
Debrief IP and agree on action plan, schedule, and responsibilities
Share completed checklist and report with PRO
Implement and Monitor recommendations
There need to be sufficient information in the checklist in order for people can understand how you reached your conclusion

19. COMMON DQA QUESTIONS
How do I document DQA for indicators reported by multiple activities?
How many pieces of data or documents should be reviewed?
Should reported data be corrected if issues are found and how?
What if I don’t know what DQ standard an issue identified falls under?
What if I don’t know as much about M&E methods as the IP?
Should IPs do their own DQAs and how should they be used in USAID DQAs?

20. DQA TOOLS AND REFERENCES
ADS 203
PMP Toolkit (Part 2, Module 7)
DQA checklist
A Step by Step DQA Planning and Implementation Guide

21. Thank You