# Theory and Application of Benchmarking in Business Surveys

## Presentation on theme: "Theory and Application of Benchmarking in Business Surveys"— Presentation transcript:

Theory and Application of Benchmarking in Business Surveys
Susie Fortier and Benoit Quenneville Statistics Canada -TSRAC ICES – June 2007

Content Introduction Notation Benchmarking methods Timeliness issue
Implied forecasts and annual growth rates Other uses: Seasonally adjusted data Linking problem Conclusions

Introduction Main references
Dagum, E.B. and Cholette, P. (2006) Benchmarking, Temporal Distribution, and Reconciliation Methods for Time Series, New York: Springer-Verlag, Lecture Notes in Statistics 186. Bloem, A. M., R. J. Dippelsman, and N. Ø. Mæhel (2001): Quarterly National Accounts Manual, Concepts, Data Sources and Compilation. International Monetary Fund, Washington DC.

Introduction Benchmarking :
Combining a series of high-frequency data with a series of less frequent data into a consistent time series. Monthly/Quarterly Annual Explicit information about the short-term movement Reliable information on the overall level and long-term movement The “indicator” series The “benchmarks”

Introduction Issues in Benchmarking :
Preserve period to period movement of the indicator (monthly/quarterly) series while simultaneously attaining the level of the benchmarks (annual). Consider the timeliness of the benchmarks.

Example of a quarterly series
Introduction Example of a quarterly series

A quarterly series with its auxiliary source
Introduction A quarterly series with its auxiliary source

Introduction Timeliness issue

Introduction Benchmarked series

Indicator (monthly/quarterly)
Notation Methodological details : Indicator (monthly/quarterly) Benchmarks (annual) DATA mySeries; year 4. @06 period 1. @08 value; CARDS; ; RUN; DATA myBenchmarks; startYear 4. @06 startPeriod 1. @08 endYear 4. @13 endPeriod 1. @15 value; To introduce notation and “input requirement”, i.e. coverage periods for each benchmarks.

Notation Methodological details :
With binding benchmarking, the benchmarked series is such that With the coverage periods for each benchmark, we can easily identify which periods t correspond to a given m. We will use a shortcut with the “t in m” notation We will only (mostly??) consider binding cases for this IOL, that is that respecting the annual benchmarks is imperative

Notation A bias parameter can be estimated and used to pre-adjust the indicator series: A bias corrected series is obtained as: Bias can be computed on all m (years) or only the most recent

Notation Alternatively, the bias can be expressed in terms of a ratio:
The bias corrected series is then: Bias can be computed on all m (years) or only the most recent

Notation Bias correction is a preliminary adjustment to reduce, on average, the discrepancies between the two sources of data. Useful for periods not covered by benchmarks. Bias can be computed on all m (years) or only the most recent

Effect of the Bias Correction (ratio)
Notation Effect of the Bias Correction (ratio) Re-scaling always the same direction even if the final discrepancies to correct may vary.

Methods : Pro-rating A simple way to respect the constraints is to use
This is the well-known formula for pro-rating. To explain the choice of the minimisation formulae s* ?

Benchmarked series with pro-rating
Methods : Pro-rating Benchmarked series with pro-rating s* ?

BI ratio with pro-rating
Methods : Pro-rating BI ratio with pro-rating

Growth rates with pro-rating
Methods : Pro-rating Growth rates with pro-rating

Growth rates with pro-rating
Methods : Pro-rating Growth rates with pro-rating DATE Indicator Series Benchmarked Series Growth Rate in Indicator Series (%) Growth Rate in Benchmarked Series (%) 1851 . 2436 31.60 3115 27.87 2205 -29.21 1987 -9.89 -17.90 2635 32.61 3435 30.36 2361 -31.27 2183 -7.54 -10.93

Methods : Proportional Denton
Benchmarked series with Prop. Denton

Methods : Proportional Denton
BI ratio with Prop. Denton

Methods : Proportional Denton
Growth rates with Prop. Denton

Methods : Proportional Denton
Growth rates with Prop. Denton DATE Indicator Series Benchmarked Series Growth Rate in Indicator Series (%) Growth Rate in Benchmarked Series (%) 1851 . 2436 31.60 30.86 3115 27.87 26.18 2205 -29.21 -30.86 1987 -9.89 -12.66 2635 32.61 29.04 3435 30.36 27.62 2361 -31.27 -32.14 2183 -7.54 -8.16 . 31.60 27.87 -29.21 -17.90 32.61 30.36 -31.27 -10.93 Pro-rating s* ?

Main method Based on Dagum and Cholette (2006).
Generalization of many well-known methods: Pro-rating Denton (and proportional Denton) Implemented at Statistics Canada with a user-defined SAS procedure: PROC BENCHMARKING Project Forillon Software Demo

Main method : Formula The benchmarked series can be obtained as the solution of a minimization problem. For given parameters and find the values that minimize the following function of : subject to s* to be explained… Explain choice of minimization function (later)

Main method : Formula Solution when : Solution: “Regression-based”
model from Dagum & Cholette For rho < 1

Main method : Formula Solution when : Solution:
where W is the T x M upper-right corner matrix from : For rho < 1

Main method : Formula We can obtain pro-rating with the general formula with and : minimise under gives How to obtain the pro-rating with the minimisation formulae.

Main method : Effect of Consider the case where and .
The function to be minimized under the constraints which aims at preserving the period-to-period change in the original series. Modified Denton method To motivate the choice of the minimisation formulae

Main method : Effect of Consider the case where and .
The function to be minimized under the constraints which seeks to minimize the change in the ratios (not to preserve the growth rates but a fairly close approx). Variant of Proportional Denton method with positive data! To motivate the choice of the minimisation formula (We suppose data is positive

Main method : Effect of 3 parameters at play: subject to
: model adjustment parameter : “smoothing” parameter bias (implied with ) subject to s* to be explained… Explain choice of minimization function (later)

Main method : Effect of Also to introduce bias…

Main method : Effect of bias
Benchmarking without bias ( )

Main method : Effect of bias
Benchmarking with bias ( )

Main method : Effect of bias
Benchmarking without bias ( ) Coverge to 1

Main method : Effect of bias
Benchmarking with bias ( ) Coverge to the bias ( Overlay the two graphs?)

Proportional Denton (ρ=1, λ=1)
Timeliness issues Adjustments for periods without benchmarks: Benchmarked series give an implicit forecast for the unknown annual values. The better the forecast, the lesser the revision! Proportional Denton (ρ=1, λ=1) Benchmarking with bias (ρ=0.93, λ=1)

Timeliness issues 2 implicit forecasts for 2006: 2004 2005 2006
Enhanced benchmarking method with explicit forecasts Year Benchmark Indicator Benchmarked (bias) (prop Denton) 2004 11,582 4.37% 11,891 1.98% 2005 11,092 -4.23% 12,399 4.27% 2006 n/a 12,196 -1.64% 11,352 2.35% 10,689 -3.64%

Timeliness issues One possibility for explicit forecast: 2004 2005
Use the annual growth rate from the indicator series on the last known benchmark. Year Benchmark Indicator Benchmarked (bias) (prop Denton) 2004 11,582 4.37% 11,891 1.98% 2005 11,092 -4.23% 12,399 4.27% 2006 10,910 -1.64% 12,196 11,352 2.35% 10,689 -3.64%

With explicit forecast ( )
Timeliness issues With explicit forecast ( )

With explicit forecast ( )
Timeliness issues With explicit forecast ( )

With ″recent″ bias( , bias=0.94)
Timeliness issues With ″recent″ bias( , bias=0.94) Bias computed with last 3 years

With ″recent″ bias( , bias=0.94)
Timeliness issues With ″recent″ bias( , bias=0.94) Bias computed with last 3 years

Timeliness issues Minimize revision? Bias Explicit forecast
(based on indicator) Will change annual growth rate of indicator series Preserve annual growth rate of indicator when nothing else is available Could be ″infected″ with non-representative historical data Annual discrepancies based only on one year Minimize revision? See Bloem

Methods : Summary so far!
Summary of methods presented: Pro-rating Denton (and proportional Denton) Regression-based (Dagum and Cholette) with or without bias correction Denton with explicit forecast Results from all of the above can be obtained by PROC BENCHMARKING.

Methods Other methods Future version of PROC benchmarking?
Other numerical methods revolve around different minimisation functions. Statistical model-based approaches See annex 6.1 in Bloem, Dippelsman, and Mæhel (2001) for variants and references See also Chen and Wu (2006) for link between numerical, regression based and signal extraction methods. Future version of PROC benchmarking?

Syntax : PROC Benchmarking
PLEASE SEE SOFTWARE DEMO !! PROC BENCHMARKING BENCHMARKS=myBenchmarks SERIES=mySeries OUTBENCHMARKS=outBenchmarks OUTSERIES=outSeries OUTGRAPHTABLE=outGraph RHO= LAMBDA=1 BIASOPTION=3; RUN;

In SAS Enterprise Guide®(Demo)
Provide input information to the procedure Mainly for training and demonstration Easy access to results in graphical format Available graphs for Benchmarking Original scale Adjustment scale ( Benchmarked to Indicator ratio) Growth rates

Seasonally adjusted series can be required to ″match″ given annual totals : System of National Accounts (typical cases) X-12-ARIMA version 0.3+ FORCE spec (table D11 A) With argument Type=regress : same methodology as PROC BENCHMARKING

X-12-ARIMA V0.3 Bias parameter option is replaced with argument target, which specifies which series is used as the target for forcing the totals of the seasonally adjusted series. The choices are: Original Caladjust (Calendar adjusted series) Permprioradj (Original series adjusted for permanent prior adjustment factors) Both (Original series adjusted for calendar and permanent prior adjustment factors)

X-12-ARIMA V0.3 By default, the FORCE spec implies that the calendar year totals in the SA = calendar year totals of the target series. Alternative starting period for the annual total can be specified with start argument. Annual total starting at any other period other than start may not be equal.

X-12-ARIMA V0.3 : example spec series{… save = A18} transform{function=log} regression{ variables=(TD easter[8])} outlier{ …} arima{…} forecast{…} x11{… save = D11} force{ type=regress lambda=1 rho=0.9 target=calendaradj save=SAA }

Canadian Department Stores Sales SA (D11) and SA with forced annual totals (D11 A)

Annual total starting at any other period other than start may not be equal.

Linking segments of time series with different levels or ranges. Used to minimize breaks caused by survey redesign, reclassification, change in concept… Challenges: Estimation of the potential break (parallel run, forecasting, backcasting, …) Preserve data coherence.

Can usually be achieved with PROC BENCHMARKING: If the two segments overlap (if not, use a model to extend one of the two segments) With proper identification of “anchor” points as benchmarks The smoothing parameter can gradually “bridge” the gap between the two levels

Two segments of a series

Adjusted as a level shift (λ=1, ρ=0.9, bias)

Adjusted as a level shift (λ=1, ρ=0.9, bias)

BI ratio for a gradual level shift (λ=1, ρ=0.9, no bias)

Conclusions Summary : Future developments in PROC BENCHMARKING
Many numerical methods can be achieved through PROC BENCHMARKING Different uses of benchmarking Future developments in PROC BENCHMARKING Simplify the use of explicit forecasts Improve bias estimation Enhance batch processing (VAR and BY statements) Include more options provided in Dagum and Cholette (2006): more generalised autocorrelation structure of the residuals, measurement errors in the input series, variance estimation of the results.

Susie.Fortier@Statcan.ca Benoit.Quenneville@Statcan.ca