Methods, Diagnostics, and Practices for Seasonal Adjustment Catherine C. H. Hood Introductory Overview Lecture: Seasonal Adjustment

Acknowledgements Many thanks to David Findley, Brian Monsell, Kathy McDonald-Johnson, Roxanne Feldpausch Agustín Maravall Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Outline Basic concepts Software packages for seasonal adjustment production Mechanics of X-12 and SEATS Overview of current practices Recent developments in research areas Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Time Series A time series is a set of observations ordered in time Usually most helpful if collected at regular intervals In other words, a sequence of repeated measurements of the same concept over regular, consecutive time intervals Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Time Series Data Occurs in many areas: economics, finance, environment, medicine Methods for time series are older than those for general stochastic processes and Markov Chains The aims of time series analysis are to describe and summarize time series data, fit models, and make forecasts Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Why are time series data different from other data? Data are not independent Much of the statistical theory relies on the data being independent and identically distributed Large samples sizes are good, but long time series are not always the best Series often change with time, so bigger isn’t always better Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

What Are Our Users Looking for in an Economic Time Series? Important features of economic indicator series include Direction Turning points In addition, we want to see if the series is increasing/decreasing more slowly than it was before Consistency between indicators Econometric models require seasonally adjusted data. Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Why Do Users Want Seasonally Adjusted Data? Seasonal movements can make features difficult or impossible to see Econometric models require seasonally adjusted data. Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Classical Decomposition One method of describing a time series Decompose the series into various components Trend – long term movements in the level of the series Seasonal effects – cyclical fluctuations reasonably stable in terms of annual timing (including moving holidays and working day effects) Cycles – cyclical fluctuations longer than a year Irregular – other random or short-term unpredictable fluctuations Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Causes of Seasonal Effects Possible causes are Natural factors Administrative or legal measures Social/cultural/religious traditions (e.g., fixed holidays, timing of vacations) Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Causes of Irregular Effects Possible causes Unseasonable weather/natural disasters Strikes Sampling error Nonsampling error Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Other Effects Trading Day: The number of working or trading days in a period Moving Holidays: Events which occur at regular intervals but not at exactly the same time each year Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

May 2007 S M T W T F S 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

June 2007 S M T W T F S 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Moving Holiday Effects Holidays not at exactly the same time each year Easter Labor Day Thanksgiving Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

“Combined” Effects Trading day and moving holiday effects are both persistent, predictable, calendar-related effects, so trading day and holiday effects often included with the seasonal effects Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

The Simple Case The time series would have No growth or decline from year to year, only rather repetitive within-year movements about an unchanging level No trading day or moving holidays Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Change in Variations What if the magnitude of seasonal fluctuations is proportional to level of series? take logarithms Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Log Transformations Appropriate when the variability in a series increases as its level increases, and when all values of the series are positive Change multiplicative relationships into additive relationships Increases/decreases can be thought of in terms of percentages Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Problem: Extreme Values Solution: These effects can be estimated also, but they can be difficult to estimate when seasonality and trend are present How do you estimate extreme values (outliers) in the presence of seasonality and trend? Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Which of these values are outliers (extreme values)?

Trading Day and Other Effects What if trading day and/or other effects (holiday, outliers) are present? X-11: TD, holiday regression on the irregular component, extreme value modifications SEATS: RegARIMA models for a regression on the original series X-12: Use X-11 methods or RegARIMA models Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Models Multiplicative model: Yt = St´ × Tt × It = St´ × Nt where St´ = St × TDt × Ht Additive model: Yt = St´ + Tt + It = St´ + Nt where St´ = St + TDt + Ht Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Objectives Estimate Nt (remove effects of St ) for seasonal adjustment Estimate Tt (remove effects of St and It) for trend estimation Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

How Do We Estimate the Components? Seasonal adjustment is normally done with off-the-shelf programs such as: X-11 or X-12-ARIMA (Census Bureau), X-11-ARIMA (Statistics Canada), Decomp, SABL, STAMP, TRAMO/SEATS (Bank of Spain) Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

(Forecasts, Backcasts, and Preadjustments) RegARIMA Models (Forecasts, Backcasts, and Preadjustments) Modeling and Model Comparison Diagnostics and Graphs Seasonal Adjustment Seasonal Adjustment Diagnostics and Graphs

( ) RegARIMA Model log = ´ Xt + Zt Yt Dt transformations ARIMA process Xt = Regressor for trading day and holiday or calendar effects, additive outliers, temporary changes, level shifts, ramps, and user-defined effects Dt = Leap-year adjustment, or “subjective” prior adjustment ( Yt ) Dt Catherine Hood Consulting

ARIMA Models and Forecasting If we can describe the way the points in the series are related to each other (the autocorrelations), then we can describe the series using the relationships that we’ve found AutoRegressive Integrated Moving Average Models (ARIMA) are mathematical models of the autocorrelation in a time series One way to describe time series Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Autocorrelation The major statistical tool for ARIMA models is the sample autocorrelation coefficient n __ __  ( Yt – Y ) ( Yt-k – Y ) The autocorrelation is the correlation of the time series with itself, lagged by 1,2, or more periods. which describes the relationship between various values of the time series that are lagged k periods apart. rk = t=k+1 n  __ ( Yt – Y )2 t=1 Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Autocorrelations r1 indicates how successive values of Y relate to each other, r2 indicates how Y values two periods apart relate to each other, and so on. Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

ACF Together, the autocorrelations at lags 1, 2, 3, etc. make up the autocorrelation function or ACF and then we plot the autocorrelations by the lags The ACF values reflect how strongly the series is related to its past values over time Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Autoregressive Processes The autoregressive process of order p is denoted AR(p), and defined by Zt =  r Zt-r + wt where 1 , . . . , p are fixed constants and {wt} white noise, a sequence of independent (or uncorrelated) random variables with mean 0 and variance  2 p r=1 Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Moving Average Processes The moving average process of order q, denoted MA(q), includes lagged error terms t–1 to t–q, written as Zt = wt –  r wt-r where 1 , 2 , … , q are the MA parameters and wt is white noise q r=1 Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Random Walk Constrained AR Model Zt = Zt-1 + wt with 1 = 1 First differenced model Zt = Zt-1 + wt Zt – Zt-1 = wt (1 – B) Zt = wt Seasonal difference model Zt – Zt-12 = wt Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

ARMA processes The autoregressive moving average process, ARMA(p,q) is defined by Zt –  r Zt–r =  r wt–r where again wt is white noise p q r=1 r=0 Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Seasonal Processes A seasonal AR process Zt =  r Zt-Sr + wt A seasonal MA process Zt = wt –  Θr wt-r where 1 , . . . , P and Θ1 , … , ΘQ are fixed constants, {wt} is white noise, and S is the frequency of the series (12 for monthly or 4 for quarterly) p Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

( ) RegARIMA Model log = ´ Xt + Zt Yt Dt transformations ARIMA process Xt = Regressor for trading day and holiday or calendar effects, additive outliers, temporary changes, level shifts, ramps, and user-defined effects Dt = Leap-year adjustment, or “subjective” prior adjustment ( Yt ) Dt Catherine Hood Consulting

RegARIMA Model Uses Extend the series with forecasts (or possibly backcasts) Detect and adjust for outliers to improve the forecasts and seasonal adjustments Estimate missing data Detect and directly estimate trading day effects and other effects (e.g. moving holiday effects, user-defined effects) Forecasting isn’t just for seasonal adjustment. FTD is using forecasts from X-12 for quality control. Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Automatic Procedures Both X-12-ARIMA and SEATS have procedures for the automatic identification of ARIMA model Outliers Trading Day effects Easter effects Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

(Forecasts, Backcasts, and Preadjustments) RegARIMA Models (Forecasts, Backcasts, and Preadjustments) Modeling and Model Comparison Diagnostics and Graphs Seasonal Adjustment Seasonal Adjustment Diagnostics and Graphs

How are component estimates formed? X-11, X-12: limited set of fixed filters ARIMA Model-based (AMB): Fit ARIMA model to series This model, plus assumptions, determine component models Signal extraction to produce component estimates and mean squared errors (MSE) Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Example Trend Filter from X-12-ARIMA A centered 12-term moving average Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Example: 3x3 Filters 3 x 3 filter for Qtr 1, 1990 (or Jan 1990) 1988.1 + 1989.1 + 1990.1 + 1989.1 + 1990.1 + 1991.1 + 1990.1 + 1991.1 + 1992.1 9 Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Example Seasonal Filter from X-12-ARIMA: 3x3 Filter Recall that Y = TSI, so SI = Y/T, i.e., the detrended series Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

AMB Approach Fit RegARIMA model yt = x´t  + Zt Given an ARIMA model for series Zt,  (B)  (B) Zt = Θ (B)  (B) wt and the model Yt = St + Nt , determine models for components St and Nt Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Where . . . St independent of Tt independent of It (  St independent of Nt ) St , Tt , It follow ARIMA models consistent with the model for Zt (hence so does Nt) It is white noise (or low order MA) Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Canonical Decomposition Problem: There is more than one admissible decomposition Solution: Use the canonical decomposition, the decomposition that corresponds to minimizing the white noise in the seasonal component Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Properties of the Canonical Decomposition Unique (and usually exists) Minimizes innovation variances of seasonal and trend; maximizes irregular variance Forecasts of St follow a fixed seasonal pattern Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Advantages of AMB Seasonal Adjustment Flexible approach with a wide range of models and parameter values Model selection can be guided by accepted statistical principals Filters are tailored to individual series through parameter estimation, and are “optimal” given Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Advantages of AMB Seasonal Adjustment (2) Signal extraction calculations provide error variances of component estimates with MSE based on the model Approach easily extends (in principle) to accommodate a sampling error component Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

At the End of the Series X-11: asymmetric filters (from ad-hoc modifications to symmetric filters) X-11-ARIMA, X-12: one year (optionally longer) forecast extension AMB: full forecast extension Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Issues Relating to Current Practices X-12 versus SEATS Use of RegARIMA models, for forecasting, trading day, holidays, etc. Diagnostics Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Agreement in Current Practices Compute the concurrent factors (running the seasonal adjustment software every month with the most recent data) instead of projected factors Use regARIMA models whenever possible (ARIMA models required for SEATS) Continue to publish the original series along with the seasonal adjustment Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

X-12 vs SEATS Eurostat recommends use of either program US Census Bureau recommends use of X-12-ARIMA According to research, X-12 is more accurate than SEATS for most series X-12 works better for short series (4 to 7 years) and for longer series (over 15 years) X-12 has better diagnostics In practice, I’ve noticed with SEATS smoother adjustments with smaller revisions for some of our irregular series Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Setting Options To reduce revisions, best to set certain options for production Most agencies let the software choose the options and then fix the settings for production Problems come with SEATS because model used is not always the model specified, and model coefficients also are not always the ones specified Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Trading Day and Moving Holiday Settings In Europe, there has been a lot of work on “user-defined” variables that include trading days and moving holidays to incorporate country-specific holidays Most agencies in the U.S. use built-in trading day and built-in moving holidays from X-12-ARIMA Unfortunately, not all the built-in variables are useful for every situation Some agencies avoid trading day altogether Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Outlier Settings At the Australia Bureau of Statistics, they have a very rigorous procedure of outlier identification, including meta data on certain unusual events Most other agencies use the automatic outlier selection procedure At the U.S. Census Bureau Choose new outliers with every run At annual review time, set outliers for current data and set a high critical value for the new data coming in Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Direct/Indirect Definitions If a time series is a sum (or other composite) of component series Direct adjustment – a seasonal adjustment of the aggregate series obtained by seasonally adjusting the sum of the component series Indirect adjustment – a seasonal adjustment of the aggregate series obtained from the sum of the seasonally adjusted component series Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Example – Direct and Indirect Adjustment US = NE + MW + SO + WE Indirect seasonal adjustment of US: SA(NE) + SA(MW) + SA(SO) + SA(WE) Direct seasonal adjustment of US: SA( NE + MW + SO + WE ) Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Comment on Yearly Totals When do yearly totals of the original series and the seasonally adjusted series coincide? When the series has An additive decomposition A seasonal pattern that is fixed from one year to the next No trading adjustments Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Areas for Improvement in Current Practices Concurrent adjustment Use of regARIMA models Moving holidays and other user-defined effects Setting options (to reduce revisions) and checking the options regularly Software to make it easier to check diagnostics regularly Training in ARIMA modeling and diagnostics Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Recent Developments and Research Areas X-13 (X-13-SEATS) Improved and new diagnostics (for both X-12 and SEATS) New filters for X-12 and new, more flexible models for SEATS Supplemental and utility software Documentation and training Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Newest X-12 Version 0.3 includes a new automatic ARIMA-modeling procedure based on the program TRAMO from the Bank of Spain The next release (X-13) will include ARIMA-model-based seasonal adjustment options Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Model-based Adjustment SEATS, developed by Agustín Maravall at the Bank of Spain REGCMPT, developed by Bill Bell at the Census Bureau Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

SEATS Disadvantages No diagnostics for the adjustment No methods for series with different variability in different months No user-defined regressors Not very flexible ARIMA models Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

REGCMPT Advantages Still being tested Methods for different variability in different months Can build very flexible regARIMA models Still being tested Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

X-13-SEATS Advantages Disadvantage Would combine the model-based adjustments from SEATS with diagnostics from X-12, and keep the ability to use X-11-type adjustments also Disadvantage ???? Possibly, within the next few years, besides all the other choices we have to make, we’ll also have to decide between X11 adjustments and SEATS adjustments for production work at the Bureau Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Running in Windows TRAMO/SEATS for Windows Windows Interface to X-12-ARIMA Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Supplemental Software X-12-Graph in SAS and in R X-12-Data and X-12-Rvw Programs to help write user-defined variables for custom trading day and moving holidays Excel interfaces to run SEATS and X-12 from Excel Interfaces to other software are available Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Documentation and Training “Getting Started” papers to use with the Windows version, written for novice users Documentation on commonly used options for both X-12 and SEATS Training Advanced Diagnostics RegARIMA Modeling Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Resources X-12-ARIMA website www.census.gov/srd/www/x12a Seasonal adjustment papers pages TRAMO/SEATS website www.bde.es/english/ Papers and course information www.catherinechhood.net Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007

Contact Information Catherine Hood Catherine Hood Consulting 1090 Kennedy Creek Road Auburntown, TN 37016-9614 Telephone: (615) 408-5021 Email: cath@catherinechhood.net Web: www.catherinechhood.net Catherine Hood Consulting Methods, Diagnostics, and Practices for Seasonal Adjustment---June 2007