Presentation on theme: "The Data Quality Assessment Framework"— Presentation transcript:
1The Data Quality Assessment Framework OECD Meeting of National Accounts ExpertsOctober 2001
2Purpose of this Presentation To describe:The IMF’s Data Quality Assessment Framework (DQAF), andExperience to date with the DQAF for Reports on Observance of Standard and Codes (ROSCs) and beyond.
3Plan for Presentation Origins of DQAF DQAF Approach Links to SDDS/GDDS Framework: what is it?Process: how was it developed?Draft framework: an overviewThe DQAF suite of assessment toolsThe work aheadLinks to SDDS/GDDSWorking with the DQAFIn describing the work on data quality, I will touch upon:To date, this is staff work undertaken within the Statistics Department.
4Origins of Recent WorkSDDS and GDDS: broadening the scope of data standards to strengthen the link with data qualityProvision of data by members to the IMF: a need to be clearer about what is called forROSC’s: a need for an even-handed approach to assessing data quality
5Increased Interest in Data Quality More widely, interest in quality follows from explicit use of statistics in policy formulation and goal setting:Inflation targeting (spotlight on CPI)Stability Pact in the context of EMU (spotlight on debt/deficit ratios to GDP)UN Conferences on Least Developed Countries (inclusion and graduation from list is based on specified economic indicators)
7The IMF’s ApproachData Quality Reference Site at the IMF’s Dissemination Standards Bulletin BoardThe Site provides an introduction to the topic of data quality and includes a selection of reference materials and articles on data quality issues.
8The DQAF: What is its Purpose? Its potential usesTo guide data users—to complement the SDDS and GDDSTo guide IMF staffin assessing data for IMF surveillance and operations,in preparing ROSCs, andin designing Technical AssistanceTo guide country efforts (self-assessment)
9The DQAF: Requirements Given these differing potential uses, the framework should be:ComprehensiveBalanced between experts’ rigor and generalists’ bird’s-eye viewApplicable across various stages of statistical developmentApplicable to the major macroeconomic datasetsDesigned to give transparent resultsArrived at by drawing on national statisticians’ best practices
10The DQAF : What Is It? Generic Dataset- Specific etc. etc. etc. GFS BOPNA
11How the DQAF Was Developed We engaged a national statistical office to help develop the generic frameworkIn parallel, IMF staff worked on frameworks for several datasetsNational accounts was reviewed in June 2000National accounts (revised) and four other specific frameworks were circulated informally in the international statistical community for comment in August-September 2000
12How the DQAF Was Developed Drafts were discussed in topical or regional meetings, e.g.East Asian Heads of NSOsECB Working Group on Money and Banking StatisticsIMF BOP Statistics CommitteeGFS Expert Group meeting
13How the DQAF Was Developed IMF staff tested the frameworks in the fieldA paper for the Statistical Quality Seminar in December 2000 presented:Revised generic frameworkRevised BOP dataset-specific frameworkAlternatives for a preview (“lite”) toolSample summary presentations of resultsTo access the paper:
14DQAF: an Overview Uses a cascading structure - and for each element, Five dimensions of quality- and for each dimension,Elements that can be used in assessing quality- and for each element,Indicators that are more concrete and detailed- and for each indicator,Focal issues that are tailored to the dataset- and for each focal issueKey points
15DQAF: an OverviewThe five dimensions of the IMF’s Data Quality Assessment Framework1. Integrity2. Methodological soundness3. Accuracy and reliability4. Serviceability5. Accessibility
16DQAF: an OverviewAlso, some elements/indicators are grouped as “prerequisites of quality”Pointers that are relevant to more than one of the five dimensionsGenerally refer to the umbrella agencyExample: quality awareness
17Prerequisites for Quality Legal and institutional frameworkRoles and responsibilities of statistical agenciesData sharing and coordination between data producing agenciesAccess to administrative and other data for statistical purposesNature of reportingResourcesQuality awareness
18Elements of IntegrityProfessionalismTransparencyEthical standards
19Elements of Methodological Soundness Concepts and definitionsScopeClassificationsBasis for recording: accounting rules and valuation principles
20Elements of Accuracy Source data Statistical techniques: compilation procedures and statistical methods and adjustmentAssessment and validation
21Elements of Serviceability Relevance of the national accounts programTimeliness and periodicityConsistencyRevision policy and practice
22Elements of Accessibility Data accessibilityMetadata accessibility: documentationAssistance to users: service and support
23Indicators of Consistency Temporal consistencyInternal consistencyIntersectoral consistency
24Focal Issues for Internal Consistency Internal consistency of the annual accountsInternal consistency between quarterly and annual estimates
25Key Points Internal Consistency of the National Accounts Discrepancies between approaches shown?Size of discrepancies?Differences between growth rates?Supply and use framework applied?Do total supply and use match?Does net lending/borrowing match between sectors?
26General Reactions “Welcome initiative” “Fills important gap” “Is careful and thoughtful”“Provides basis for coherent and practical way forward in a complex field”
27General Reactions Some other points Is the framework really operational for small countries?Can it be used without giving a “black mark” for points that are irrelevant to a country?Is the framework able to identify “poor” statistics prepared within a developed statistical system?
28General Reactions Some other points (cont’d) Expand the range of datasets coveredCoordination with other organizations working on data quality is importantContinue working in a consultative manner
29The DQAF Suite of Tools DQAF “Lite” Background: interest in a version that might serve as a diagnostic preview or for a non-statistician’s assessmentIMF is field testing a “Lite” made up of 13 indicators.
30The DQAF Suite of Tools Summary presentation of results Background: Interest in a presentation of results for, e.g., policy advisorsIMF is testing a summary presentationFor each dataset, a one-page tableAt the two-digit level (21 elements)On a 4-point scale, from “practice observed” to “practice not observed”With an “n.a.” columnWith a “comments” column
31Data Quality Assessment Framework Summary for [dataset] Note: O = Practice Observed; LO = Practice Largely Observed: MNO = Practice Materially Nonobserved;NO = Practice Nonobserved; NA= Not Applicable Comment: only if different from O.
32The DQAF Suite of Tools Dataset (6-digit) Generic (3-digit) “Lite” Summary of Resultsetc.Dataset(6-digit)etc.Dataset Specific(5-digit)etc.GFSBOPNA
33Work ahead Test the suite Refine and revise the suite in a wider range of country situationsespecially with non-statisticiansRefine and revise the suiteComplete supporting materialsA GlossarySupporting Notes for specific datasetsA Methodology (a how-to-do-it guide)Develop frameworks for other datasets
34Links to the SDDS/GDDS Summary: The DQAF complements the SDDS/GDDS All of the elements of the SDDS/GDDS are also found within the DQAF
35Links to SDDS/GDDSThe purpose and scope of the SDDS/GDDS and DQAF differ:In SDDS/GDDS, as dissemination standards, quality is a dimension.That dimension takes an indirect approach to dealing with, e.g., accuracy--it calls for dissemination of relevant information.In DQAF, as an assessment tool, quality is the umbrella concept.That concept covers collection, processing, and dissemination of data.
36Links to SDDS/GDDSThe DQAF definition of “quality” has been brought into line with the emerging consensus that quality is a multidimensional concept.Some aspects relate to the productSome aspects relate to the institution
37Links to SDDS/GDDSDQAF is “more active” in dealing with, e.g., conformity with international guidelines, accuracy, and reliability.SDDS and to a lesser degree GDDS left users on their own to make judgmentsDQAF guides users in making such judgments by providing two structured dimensions:Methodological soundnessAccuracy and reliability
38Working with the DQAFThe earlier list of potential uses of the DQAF included “To guide IMF staff “Largely this refers to staff of the IMF Statistics DepartmentInterrelated uses:in assessing data for IMF’s use in surveillance and operations,in preparing ROSCs, andin designing technical assistance
39Working with the DQAF We are now using the DQAF in the field In capacity building advisory missionsIn ROSCs
40Working with the DQAF What do we see from the experiences? Advantages Provides more structure to technical assistancePromotes consistency across staff/expertsPotentially provides input for useful databasePlaces data standards in the center of work on the international financial architectureChallengesPuts premium on consistencyCalls for explicit judgments