Presentation on theme: "Implementing the Data Quality Assessment Tool"— Presentation transcript:
1 Implementing the Data Quality Assessment Tool Data Systems QualityImplementing the Data Quality Assessment Tool
2 Session Overview Why data quality matters Dimensions of data quality Thoughts about improving data qualityData Quality Assurance ToolActivity: Implementing the Tool
3 Data Management System Data QualityThe REAL worldIn the real world, project activities are implemented in the field. These activities are designed to produce results that are quantifiable.Data Management SystemAn information system represents these activities by collecting the results that were produced and mapping them to a recording system.Data Quality: How well the DMS represents the real world?DataManagementSystemThe real world can be thought of as the services that are being delivered by your program/project/interventionWe want our data management system to reflect the real world.Think of a mirror – a perfect, high quality mirror v. a convex/concave/rolling mirrorHow well do our data collection tools (mirrors) reflect what is really happening in our programs?RealWorld
4 Testing and Counseling: How are these data collected? A person walks into the facilityFacility registrationHow aggregated at facility level?How forwarded to next level?How forwarded to national level?How forwarded to international level?
5 ARV Treatment: How are these data collected? A person tests positive for HIVWhen begin receiving ARV?How recorded in facility records?How aggregated at facility level?How aggregated at next level?How aggregated at national level?How aggregated at international level?
6 OVC Care: How are these data collected? A child is identified as being an orphan or vulnerable—how?Receives care from an organization—which ones? How many?How recorded at organizational level?How aggregated at next level?How aggregated at national level?How aggregated at international level?How do we know that child did not receive care from more than one organization?
7 Why is data quality important? Governments and donors collaborating on “Three Ones”Accountability for funding and results reported increasingly importantQuality data needed at program level for management decisions
8 Data quality and PEPFAR/GFATM TargetSettingImprovedProgram &ResourceManagementData QualityData drives this continuous process that is shown in the slide…good quality data is the foundation upon which we make good decisions: improving program and resource mmgt so that we may set realistic targets!ResultsReporting
9 Data Quality REAL WORLD INFORMATION SYSTEM In the real world, project activities are implemented in the field. These activities are designed to produce results that are quantifiable.INFORMATION SYSTEMAn information system represents these activities by collecting the results that were produced and mapping them to a recording system.Data Quality: How well the information system represents the real worldData QualityReal WorldInformation System1. Accuracy2. Reliability3. Completeness4. Precision5. Timeliness6. Integrity
10 Dimensions of Data Quality ValidityValid data are considered accurate: They measure what they are intended to measure.ReliabilityThe data are measured and collected consistently.CompletenessCompletely inclusive: an information system represents the complete list of eligible names and not a fraction of the list.PrecisionThe data have sufficient detail.TimelinessData are up-to-date (current), and information is available on time.IntegrityThe data are protected from deliberate bias or manipulation for political or personal reasons.
11 Validity/Accuracy: Questions to ask… What is the relationship between the activity/program & what you are measuring?What is the data transcription process?Is there potential for error?Are steps being taken to limit transcription errordouble keying of data for large surveys, built in validation checks, random checks
12 Reliability: Questions to ask… Is the same instrument used from year to year, site to site?Is the same data collection process used from year to year, site to site?Are procedures in place to ensure that data are free of significant error and that bias is not introduced (e.g., instructions, indicator reference sheets, training, etc.)?
13 Reliability: Questions to ask… If there are data errors, what do you do with that information?If raw data need to be manipulated, are the correct formulae being applied—across site and consistently?How to handle missing/incomplete data?Are final numbers reported accurately—does the total add up?
14 Completeness: Questions to ask Are the data from all sites that are to report included in aggregate data?If not, which sites are missing?Is there a pattern to the sites that were not included in the aggregation of data?What steps are taken to ensure completeness of data?
15 Precision: Questions to ask… How is margin of error being addressed?Are the margins of error acceptable for program decision making?Have issues around precision been reported?Would an increase in the degree of accuracy be more costly than the increased value of the information?
16 Timeliness: Questions to ask… Are data available on a frequent enough basis to inform program management decisions?Is a regularized schedule of data collection in place to meet program management needs?Are data from within the policy period of interest (i.e. are the data from a point in time after the intervention has begun)?Are the data reported as soon as possible after collection?
17 Integrity: Questions to ask… Are there risks that data are manipulated for personal or political reasons?What systems are in place to minimize such risks?Has there been an independent review?
18 During this workshop, think about… How well does your information system function?Are the definitions of indicators clear and understood at all levels?Do individuals and groups understand their roles and responsibilities?Does everyone understand the specific reporting timelines—and why they need to be followed?
19 …Keep thinking about…Are data collection instruments and reporting forms standardized and compatible? Do they have clear instructions?Do you have documented data review procedures for all levels…and use them?Are you aware of potential data quality challenges, such as missing data, double counting, lost to follow up? How do you address them?What are your policies and procedures for storing and filing data collection instruments?
20 Data Quality Assessment Tool For Assessment & Capacity BuildingAnother way to assess data quality is through a data quality assessment.
21 Purpose of the DQAThe Data-Quality Assessment (DQA) Protocol is designed:to verify that appropriate data management systems are in place in countries;to verify the quality of reported data for key indicators at selected sites; andto contribute to M&E systems strengthening and capacity building.
22 DQA Components Determine scope of the data quality assessment Suggested criteria for selecting Program/project(s) & indicatorsEngage Program/project(s), obtain authorization for DQATemplates for notifying the Program/project of the assessmentGuidelines for obtaining authorization to conduct the assessmentAssess the design & implementation of the Program/project’s data collection and reporting systems.Steps & protocols to ID potential threats to data quality created by Program/project’s data management & reporting system
23 DQA Components Trace & verify (recount) selected indicator results Protocol with special instructions based on indicator & type of Service Delivery Site (e.g. health facility or community-based)Develop and present the assessment Team’s findings and recommendations.instructions on how and when to present the DQA findingsrecommendations to Program/project officials for how to plan for follow-up activities to ensure strengthening measures are implemented
24 Example: Indicator Selection DISEASEINDICATORSREPORTING PERIODHIV/AIDSNumber of patients on ARV3-month period [1-Nov-05 / 31-Jan-06]National NumbersTBNumber of smear positive TB cases registered under DOTS who are successfully treated3-month period [1-Oct Dec-04]MalariaNumber of insecticide-treated bed nets (ITNs) distributed (i.e., number of vouchers redeemed)6-month period[1-Nov-2005 / 30-Apr-2006]Reported numbers to Global Fund
25 Chronology and Steps of the DQA PHASE 1PHASE 2PHASE 3PHASE 4PHASE 5PHASE 6Preparation and Initiation(multiple locations)M&E Management UnitService Delivery Sites / OrganizationsIntermediate Aggregation levels(eg. District, Region)M&E Management UnitCompletion(multiple locations)Assess Data Management and Reporting SystemsDraft initial findings and conduct close-out meetingSelect Indicators and Reporting PeriodDraft and discuss assessment ReportObtain National Authorizations and notify ProgramSelect/Confirm Service Delivery Points to be visitedTrace and Verify Reported ResultsInitiate follow-up of recommended actionsThe DQA is implemented chronologically in 6 Phases.Assessments and verifications will take place at every stage of the reporting system:M&E Management UnitIntermediate Aggregation Level (Districts, Regions)Service Delivery Sites.
26 DQA Outputs Completed protocols and templates Part DQA Tool.Write-ups of observations, interviews, and conversationsKey data quality officials at the M&E UnitIntermediary reporting locations & Service Delivery SitesPreliminary findings, draft recommendations notesBased on evidence collected in protocolsFinal assessment ReportSummarizes evidence collectedIDs specific assessment findings & gaps related to evidenceIncludes recommendations to improve data quality directly linked to assessment findingsSummary statistics calculated from systems & data verification protocols
27 DQA Outputs Strength of the M&E System Verification Factors Evaluation based on review of data management & reporting system including summary responses on system design & implementationVerification FactorsGenerated from trace & verify recounting exercise performed on primary records/aggregated reports% comparison of reported numbers to the verified numbersAvailable, timely & complete reports percentagesCalculated at Intermediate aggregation level and the M&E unitSummary stats developed from systems & data verification protocolsAll follow-up communication with program/project related to results and recommendations of DQA
28 PROTOCOL 1:Assessment of Data Management and Reporting SystemsM&E Management UnitPHASE 2Service Delivery Sites / OrganizationsPHASE 3Intermediate Aggregation levels(eg. District, Region)PHASE 4Assess Data Management and Reporting SystemsPurposeID potential risks to data quality created by data management & reporting systems at:M&E Management Unit;Service Delivery Points;Intermediary Aggregation Levels (District or Region)The DQA assesses both design and implementation of data-management & reporting systems.Assessment covers 8 functional areas (HR, Training, Data Management Processes , etc.)
29 Functional Areas of M&E System that affect Data Quality SYSTEMS ASSESSMENT QUESTIONS BY FUNCTIONAL AREAFunctional AreasSummary QuestionsIM&E Capabilities, Roles and Responsibilities1Are key M&E and data-management staff identified with clearly assigned responsibilities?IITraining2Have the majority of key M&E and data-management staff received the required training?IIIData Reporting Requirements3Has the Program/Project clearly documented (in writing) what is reported to who, and how and when reporting is required?IVIndicator Definitions4Are there operational indicator definitions meeting relevant standards and are they systematically followed by all service points?VData-collection and Reporting Forms and Tools5Are there standard data-collection and reporting forms that are systematically used?6Are source documents kept and made available in accordance with a written policy?
30 Functional Areas of an M&E System that Affect Data Quality VIData Management Processes7Does clear documentation of collection, aggregation and manipulation steps exist?VIIData Quality Mechanisms and Controls8Are data quality challenges identified and are mechanisms in place for addressing them?9Are there clearly defined and followed procedures to identify and reconcile discrepancies in reports?10Are there clearly defined and followed procedures to periodically verify source data?VIIILinks with National Reporting System11Does the data collection and reporting system of the Program/Project link to the National Reporting System?
31 Trace and verification exercise - two stages: PROTOCOL 2:Trace and verify Indicator DataM&E Management UnitPHASE 2Service Delivery Sites / OrganizationsPHASE 3Intermediate Aggregation levels(eg. District, Region)PHASE 4Trace and Verify Reported ResultsPURPOSE: Assess on limited scale if Service Delivery Points and Intermediate Aggregation Sites are collecting & reporting data accurately and on time.Trace and verification exercise - two stages:In-depth verifications at the Service Delivery Points; andFollow-up verifications at the Intermediate Aggregation Levels (Districts, Regions) and at the M&E Unit.
32 DQA Protocol 2: Trace and Verification M&E Unit/NationalMonthly ReportDistrict 165District 245District 375District 4250TOTAL435District 1Monthly ReportSDS 145SDS 220TOTAL65District 2Monthly ReportSDS 345TOTALDistrict 3Monthly ReportSDS 475TOTALDistrict 4Monthly ReportSDP 550SDP 6200TOTAL250Service Delivery Site 1Service Delivery Site 2Monthly ReportARV Nb.20Service Delivery Site 3Monthly ReportARV Nb.45Source Document 1Service Delivery Site 4Monthly ReportARV Nb.75Source Document 1Service Delivery Site 5Monthly ReportARV Nb.50Service Delivery Site 6Monthly ReportMonthly ReportARV Nb.This chart shows visually that the trace and verify protocol starts with data at the level of service delivery (either in a health facility or a community based program) and how the data are “traced” to the “intermediate aggregation level” (in this case a district) and then to the M&E Unit Level (in this case the national level).45ARV Nb.200Source Document 1Source Document 1Source Document 1Source Document 1
33 Service Delivery Points – Data Verification SERVICE DELIVERY POINT - 5 TYPES OF DATA VERIFICATIONSVerificationsDescription-Verification no. 1:Describe the connection between the delivery of services/commodities and the completion of the source document that records that service delivery.In all casesVerification no. 2:Documentation ReviewReview availability and completeness of all indicator source documents for the selected reporting period.Verification no. 3:Trace and VerificationTrace and verify reported numbers: (1) Recount the reported numbers from available source documents; (2) Compare the verified numbers to the site reported number; (3) Identify reasons for any differences.Verification no. 4:Cross-checksPerform “cross-checks” of the verified report totals with other data-sources (eg. inventory records, laboratory reports, etc.).If feasibleVerification no. 5:Spot checksPerform “spot checks” to verify the actual delivery of services or commodities to the target populations.At the service delivery level, these five verifications are done as part of the trace and verify protocol.
34 DQA Summary Statistics These charts show the summary statistics that are automatically generated from the trace and verify protocol of the data quality assessment tool. They show for the reporting period and indicator assessmented, the:Verification factor (how the recounted data compare to the reported data)Availability of ReportsTimeliness of ReportsCompleteness of Reports
35 Illustration 1 - Trace and Verification at the M&E Unit (HIV/AIDS) Number of patients on ARV - 31st of August 200644,7%(21,449 Recounted)55,3%(26,654 Unaccounted)20%40%60%80%100%1- VERIFICATION FACTOR(% difference in the reported / re-aggregated numbers)41% Incomplete69% CompleteNo tracking of timelinessAvailability67% Missing33% Available20%40%60%80%100%Completeness *Timeliness2- AVAILABILITY, COMPLETENESS AND TIMELINESS OF REPORTS* Report has to include (1) Name of site; (2) Reporting Period; (3) Name of submitting person; (4) Cumulative data.Findings from country where pilot-tested
36 Illustration 3 – Systems’ Finding at the M&E Unit (HIV/AIDS) REPORTING LEVELFINDINGSRECOMMENDATIONSNational M&E UnitNo specific documentation specifying data-management roles and responsibilities, reporting timelines, standard forms, storage policy, …Develop a data management manual to be distributed to all reporting levelsInability to verify reported numbers by the M&E Unit because too many reports (from Service Points) are missing (67%)Systematically file all reports from Service PointsDevelop guidelines on how to address missing or incomplete reportsMost reports received by the M&E Unit are not signed-off by any staff or manager from the Service PointReinforce the need for documented review of submitted data – for example, by not accepting un-reviewed reports
37 Findings from DQAs Data not collected routinely - ‘reporting flurry’ Documentation of what was reported (can’t locate source documents/lack of filing system for easy retrieval)Issues around double-countingIntegrity – incentives for over-reportingEffect of staff turnoverInvolving staff in M&E – definitions of indicators, value of data, data useHere are some common themes from data quality assessments and assessments.
38 MEASURE Evaluation is a MEASURE project funded by the U.S. Agency for International Development and implemented bythe Carolina Population Center at the University of North Carolinaat Chapel Hill in partnership with Futures Group International,ICF Macro, John Snow, Inc., Management Sciences for Health,and Tulane University. Views expressed in this presentation do notnecessarily reflect the views of USAID or the U.S. Government.MEASURE Evaluation is the USAID Global Health Bureau'sprimary vehicle for supporting improvements in monitoring andevaluation in population, health and nutrition worldwide.