Rome, 8-11 July 2008 QUALITY 2008 An application to the Italian IIP Revision analysis to detect possible weakness in the estimation procedures A. Ciammola, T. Gambuti and A. Mancini ISTAT European Conference on Quality in Official Statistics

Rome, 8-11 July 2008 QUALITY /21  Introduction  Why revision analysis?  A case study  Next steps Outline

Rome, 8-11 July 2008 QUALITY /21 Quality and some of its dimensions Accuracy Closeness of the estimate to the true (but unknown) value of the variable to be measured Timeliness Span between the reference period and the publication period Revision Reliability measure Reliability Closeness between preliminary estimate and subsequent estimates U

Rome, 8-11 July 2008 QUALITY /21 Revision analysis  Real-time databases  collection of vintages  computation of revisions  Revisions  Rt = Lt – Pt  Rt = (Lt – Pt) / Lt  Revision measures  Size (MAR, RMAR, …)  Bias (MR, T-test)  Efficiency (News or Noise?, MSR, …)

Rome, 8-11 July 2008 QUALITY /21 Useful references OECD / Eurostat Guidelines on Revisions Policy and Analysis Themes related to revision policy and analysis  Recommendations for revisions policy and analysis  Guidelines for establishing a real-time database  Recommended statistical measures  Pre-programmed software for performing revisions analysis  A framework for revisions policy of key economic indicators  Comprehensive framework of reasons for revisions and their timing  Guidelines on how to decompose total revision into different reasons for revisions  Guidelines on how to use the results from revision analyses to improve compilation methods  Case studies on the relationship between timeliness of release and size of revisions

Rome, 8-11 July 2008 QUALITY /21 For users Objective  Availability of all the relevant information for using appropriately the estimates of short-term indicators at different stages of the revision process provision of information about past revisions schedule future revisions (statistical and definitional) real-time databases gathering all the vintages analysis of size, bias and efficiency of revisions Why do we measure revisions?

Rome, 8-11 July 2008 QUALITY /21 For producers Underlying issues Targets Why do we measure revisions? Bias in the revision process Inefficiency in compilation of preliminary estimates Reduction of (the size of) “avoidable” revisions Detection of the source for bias / inefficiency

Rome, 8-11 July 2008 QUALITY /21 A case study Italian Index of Industrial Production (IIP) 1.Sources and timing of revisions 2.Revision analysis 3.Identification of specific sources for bias 4.Some evidences

Rome, 8-11 July 2008 QUALITY /21 1. Sources and timing of revisions Y(t-3)Y(t-2)Y(t-1) Current Year Y(t) – Reference month JFMAMJJASOND M LR CE A LRCEPC M LR CE J J A S O LRCELR CE N D J F First estimateSecond estimateSix-month revisionAnnual revision LRLate respondentsCECorrection of errorsPCProductivity coefficients

Rome, 8-11 July 2008 QUALITY /21 IIP - Revisions on year-on-year growth rates (raw indices) Legend*  h=1 – after one monthh=12 – after 12 months MAR – Mean Absolute RevisionRMAR – Relative MAR MR – Mean Revision SD – Standard Deviation Period: Jan-03 / Dec-07 h=1h=12 # of revisions6048 MAR RMAR MR SD of MR(HAC) T-value Significance of MRYes *No * 2. Revision analysis

Rome, 8-11 July 2008 QUALITY /21 2. Revision analysis IIP - Revisions after one month on year-on-year growth rates (raw indices)

Rome, 8-11 July 2008 QUALITY /21 2. Revision analysis  Why this systematic component?  Late respondents?  Correction of errors?  Productivity coefficients? (In revisions after 1 month, only July 2004, January 2005, January 2006 and January 2007 are affected)  Which sectors?  All sectors?  Some specific sector?  How to proceed?  Simulation exercise  Top-down approach  Quality indicators

Rome, 8-11 July 2008 QUALITY /21 3. Identification of specific sources for bias Simulation exercise Removal of the effect of the productivity coefficients  Isolate sources of revisions external to the survey  Fulfil the condition necessary to compute the average contribution of each components to the IIP revisions PS: revisions computed on y-o-y growth rates

Rome, 8-11 July 2008 QUALITY /21 Diagram describing the top-down approach 3. Identification of specific sources for bias

Rome, 8-11 July 2008 QUALITY /21 Quality indicators  Revision measures  Contribution of each component to the mean revision of the higher component  Response rates 3. Identification of specific sources for bias

Rome, 8-11 July 2008 QUALITY /21 MIGS - Revisions after one month on raw Y-o-Y growth rates Period: Jan-03 / Dec-07 CNDCDUCAPINTENE Weights % MAR RMAR MR Contribution to MR ° SD of MR(HAC) T-value Significance of MRNo * Yes *No * Legend CND – Consumer non durables CDU – Consumer durables CAP – Capital goods INT – Intermediate Goods ENE – Energy ° Period Jan-04 / Dec-07 *  4. Some evidences

Rome, 8-11 July 2008 QUALITY /21 Revisions after one month on raw Y-o-Y growth rates 4. Some evidences

Rome, 8-11 July 2008 QUALITY /21 Average weighted response rates YearEstimateIIPCNDCDUCAPINTENE 2004 First Second First Second First Second First Second Some evidences

Rome, 8-11 July 2008 QUALITY /21 Revisions after one month on raw Y-o-Y growth rates Period: Jan-04 / Dec-07 SS C,INT S C,IIP Weights %32.3 / MAR RMAR MR Contribution to MR0.088 / T-value P-value Legend S – Selected subset of INT (19 classes) S C,INT – Complement of S in INT (S U S C,INT = INT) S C,IIP – Complement of S in IIP (S U S C,IIP = IIP) 4. Some evidences

Rome, 8-11 July 2008 QUALITY /21 Next steps  Checking the stability of results over time  Experimenting possible countermeasures to biased revisions  Treatment of late respondents with different estimators  Assessment of their effects on revisions  ISTAT web page on revisions  Real-time database for several short-term indicators  Revision analysis

Rome, 8-11 July 2008 QUALITY /21 THANK YOU!