ICP-Africa Regional Workshop Pretoria, South Africa June 2011

2 Outline Validation at regional level Inter-country data validation Timeline and deliverables

3 Inter-country validation is a stage carried out by the NCAs and RCAs to establish that Countries within the region have correct i.e. error free data Countries within the region have priced comparable products 1 2 Introduction to Regional level validation Validation stage for items on regional item list National and sub- national Quarterly Data Annual Data NOTE inter-country validation is an iterative validation stage!

4 The Quaranta Tables The main data analysis within inter-country validation is carried out by using two validation tables that are Screen the national average prices for possible errors by comparing the average prices for the same product in different countries The purpose of the both tables is to The Dikhanov Tables Editing of price data for an item This analysis can lead to Editing of metadata for an item Possible errors are highlighted by special indices Introduction to inter-country validation

5 Example of Quaranta tables QUARANTA TABLE DIAGNOSTICS - Rice Data Selection Criteria Basic Heading Code Time PeriodYearly Run Date4/13/2011 Averaging MethodArithmetic MeanImputationCPD Summary Information No of Items included in the Analysis6 out of 6 Average weight of Basic Heading in Total Expenditure0.0 No of Countries included in the Analysis18 out of 18Average Coefficient Variation27.4 Base CountryUSA Country Level Details # Shares are multiplied by CountryXRPPPPLI(%)Weight #ItemsVar.Co. Country %0.02;*07.2 Counrty %0.05;*08.2 Counrty %0.02;*033.8 Item Level Details Long grain rice, prepacked Var.Co. :25.91-Kilograms CountryNC-Price Quotati onsVar.Co.XR-prXR-ratio CUP- price CUP- ratioPref. UoM Country NA Country NA Country NA Table details BH table Item table

6 Example of Dikhanov tables Table details BH table Item table Dikhanov Temporal AnalysisCountry1Country2Country3 Yearly PPP STD No.of Priced Items ER (LCU/US$) Rebased_XR PLI Item Level DetailsCountry1Country2Country3 Item CodeItem Name Yearly Rice PPP STD PLI No.of Priced Items Long grain rice, prepacked Average Price No.of Observations Coefficient of Variation XR Ratio

7 Validation phases within inter-country validation Quick validation phase before in-depth validation to find out extreme problems with the data 1 st Phase: Initial Data Validation for Quarterly data Phase to check metadata for large deviations and other comparability issues 2 nd Phase: Validation of Metadata for Quarterly and Annual Data 3 rd Phase: Validation at BH level for Quarterly and Annual Data In-depth BH level analysis

8 Validation phases within inter-country validation 4 th Phase: Validation at aggregated levels for Quarterly and Annual Data Validation at levels over the BH 5 th Phase: Temporal Analysis for Quarterly and Annual Data Time wise plausibility analysis 6 th Phase: Finalization of data for Quarterly and Annual Data Phase to finalize the data and to send it to the GO

9 Add price and metadata to the validation software Step1 Calculate initial validation tables Step2 Check initial analytical tables for extreme XR-RatiosStep3 Note: Since focus is on clear problems with the data, one initial validation round should be enough Quaranta Tables are strongly recommended as initial validation tables Make sure that the data is in right format When the data is compared for the first time it does normally include number of severe errors Focus only on extreme rations meaning ratios below 40 or over 300 Most common errors are quantity related or simple typos Initial data validation is a quick phase before more analytical validation phases This phase is only for Quarterly Data – Annual Data is compilation of already “cleaned” quarter data 1 st Phase inter-country Initial Data Validation

10 2 nd Phase inter-country Validation of Metadata Check metadata for large deviations between requested and priced products and their parameters Step1 If priced products do not systematically fit to the product specifications this may not be visible on the indices as they are systematically biased Small deviations e.g. with the quantities should be allowed Common rules needs to applied for all countries Check metadata to assure that survey frames (shop and area sample) are comparable enough between the countries Step2 NCAs should follow on their frames typical purchasing patterns for the country In some cases this can result in incomparable price data (e.g. cloths priced only in department stores vs. only in supermarkets) RCAs need to assess comparability of national survey frames Note Edits under Step1 can normally be conducted relatively easily. Edits under Step2 are more difficult and need careful considerations by all sides. It is only possible to introduce limited edits to survey frame after the price collection. Between the collection periods this may be possible.

11 Calculate validation tablesStep1 Check plausibility of BH PLIs i.e. the order of countriesStep2 QTs or DTs Is the order according to the economic expectations? If it is not, is there systematic comparison problems (derived e.g. from a shop or product sample) or result of a few outlier items? 3 rd Phase inter-country Validation at BH Level Check high country variation coefficientsStep3 Summarizes efficiently potentially problematic country datasets Check high BH variation coefficientsStep4 Summarizes efficiently potentially problematic BHs Checks for BH tables Check number of important itemsStep5 At least 1 GCL and 1 regional item or absolute minimum 1 GCL item Very low and high numbers of important items are potentially problematic

12 Check high item variation coefficients or deviationStep7 High values mean that one or several countries have for this item average prices that differ from their typical price level for items within this BH 3 rd Phase inter-country Validation at BH Level Check high price observation variation coefficients or deviationStep8 Check high/low XR- and CUP-ratios or residualsStep9 As the validation proceeds focus moves to CUP-ratios or residuals Limits not definite and depend on the product type, but low or high ratios or residuals may point out to incorrect price data or incomparable products being priced Checks for ITEM tables or rows High values may mean that intra-country validation was not carried out properly High variation maybe reality, but as arithmetic average is used, the situation should be checked for erroneous outliers

13 Repeat validation steps carried out at BH level validationStep 4 th Phase inter-country Validation at Aggregated Levels Calculate validation tables for aggregated levelsStep1 Only DTs can be complied at different levels of aggregation Note Validation at aggregated levels is important Because it puts the editing and verification of average prices into a broader context By doing so it enables to check are the average prices consistent within a larger set of products, not just within a BH Analysis may reveal complete country datasets or BHs to be outliers If this is the case, possible edits are to be carefully considered by all parties Move to this validation phase only after data is checked to be plausible at BH level!

14 Check tables for large temporal fluctuations in PLIs or PPPsStep2 5 th Phase inter-country Temporal Analysis Compile temporal validation tables for BH and aggregated levelsStep1 ICPkit does not produce these comparison tables automatically and thus they need to be “handcrafted” NoteAbout temporal analysis Possible differences can be result of problems with the previous data or CPI indices – not necessarily with the current data The scale of temporal changes is good to keep in mind when the current data is being validated National average prices can also be compared temporally e.g. if potential outlier average prices are checked – complete temporal average price analysis is difficult due to different product definitions Previous ICP round (CPI corrected) for validation of Q1 data and annual results Previous quarter data for subsequent quarters

15 Potential outliers inter-country validation What to do in practice when a deviant index or residual is discovered on the validation tables? 1) Check underlying prices 2) Check underlying metadata 3) Check cases for systematic variation Outlier prices that affect the average price? Clusters of different prices within the same item? Right quantities (within requested range, correct reference quantities)? Comparable models, brands, types priced? Justified sub-national price differences? Are certain item types (SB, WKB, BL, services) causing deviant indices? Brands and services comparable across the countries? Can survey frame have an impact and it is justified? Delete or keep? Delete, keep or split the item? Delete or readjust survey frame? Delete, keep or split the item? Edit?

16 6th Phase inter-country Finalization of data Confirm prices and metadata to be inter-country validatedStep1 Ensure that all steps have been carried successfully Ensure that data is in right (pre-agreed) format Submit prices and metadata to the GOStep2

17 Quarterly Price Data and Metadata & Validation Table Annual Price Data and Metadata & Validation Table Q1Q Q4Q4 Q4Q4 Q3Q3 Q3Q3 Q4Q4 Q4Q4 Q3Q3 Q3Q3 Q4Q4 Q4Q4 Q3Q3 Q3Q3 Q2Q2 Q2Q2 Q1Q1 Q1Q1 Q2Q2 Q2Q2 Q1Q1 Q1Q1 Q2Q2 Q2Q2 Q1Q1 Q1Q1 PPPs by ICP Classification & Metadata (Preliminary and Final) Q2Q2 Q3Q3 Q4Q4 Q1Q1 Q2Q2 Q3Q3 Q4Q4 P A A NCs to RCs RCs to GO F Recommended Schedule of Data Submission

18 THANK YOU