2 Outline Introduction: Stationary / Non-stationary TS Data Time Series DataStationary / Non-stationary TS DataExisting TSA ModelsAR (Auto-Regression)MA (Moving Average)ARMA (Auto-Regression Moving Average)ARIMA (Auto-Regression Integrated Moving Average)SARIMA (Seasonal ARIMA)ExamplesExample 1: International Airline PassengerExample 2&3: Energy Load PredictionTime Series Data MiningTime Series Classification (SVM)ExampleExample 4: Stock Market Analysis
3 Time Series DataIn many fields of study, data is collected from a system over time.This sequence of observations generated a time series:Examples:Closing prices of the stock marketA country’s unemployment rateTemperature readings of an industrial furnaceSea level changes in coastal regionsNumber of flu cases in a regionInventory levels at a production site
4 Temporal BehaviourMost physical processes do not change quickly, often makes consecutive observation correlated.Correlation between consecutive observation is called autocorrelation.Most of the standard modeling methods based on the assumption of independent observations can be misleading.We need to consider alternative methods that take into account the serial dependence in the data.
5 Stationary Time Series Data Stationary time series are characterized by having a distribution that is independent of time shifts.Mean and variance of these time series are constantsIf arbitrary snapshots of the time series we study exhibit similar behaviour in central tendency and spread, we can assume that the time series is indeed stationary.
6 Stationary or Non-Stationary? In practice, there is no clear demarcation line between a stationary and a non-stationary process.Some methods to identify:Visual inspectionUsing intuition and knowledge about the processAutocorrelation Function (ACF)Variogram
7 Visual InspectionA properly constructed graph of a time series can dramatically improve the statistical analysis and accelerate the discovery of the hidden information in the data.“You can observe a lot by watching.” This is particularly true with time series data analysis! [Yogi Berra, 1963]
8 Intuition and knowledge Inspection Does it make sense...for a tightly controlled chemical process to exhibit similar behaviour in mean and variance in time?to expect the stock market out it “to remain in equilibrium about a constant mean level”The selection of a stationary or non-stationary model must often be made on the basis of not only the data but also a physical understanding of the process.
9 Autocorrelation Function (ACF) Autocorrelation is the cross- correlation of a time series data with itself based on lag kACF summarizes as a function of k, how correlated the observations that are k lags apart are.If the ACF does not dampen out then the process is likely not stationary(If a time series is non-stationary, the ACF will not die out quickly)Confidence interval = indicate the reliability of an estimate
10 VariogramThe Variogram Gk measures the variance of differences k time units apart relative to the variance of the differences one time unit apartFor stationary process, Gk when plotted as a function of k will reach an asymptote line. However, if the process is non-stationary, Gk will increase monotonically.
11 Modeling and Prediction “If we wish to make predictions, then clearly we must assume that something does not vary with time.” [Brockwell and Davis, 2002]Let’s try to predict and build a model for our time series process based on:Serial DependencyLeading IndicatorsDisturbanceTrue disturbances caused by unknown and/or uncontrollable factors that have direct impact on the process.It is impossible to come up with a comprehensive deterministic model to account for all these possible disturbances, since by definition they are unknown.In these cases, a probabilistic or stochastic model will be more appropriate to describe the behaviour of the process.
13 Auto-Regressive Models AR(P)Where at is an error term (called white error) assumed to be uncorrelated with zero mean and constant variance.The random error at cannot be observed. Instead we estimate it by using the one-step-ahead forecast errorThe regression coefficients , i = 1, ... , p, are parameters to be estimated from the data
14 Moving Average Current and previous disturbances affect the value. We have a sequence of random shocks bombarding the system and not just a single shock.MA(q)Uncorrelated random shocks with zero mean and constant varianceThe coefficients , i = 1, ... , q are parameters to be determined from the data
15 Auto-Regressive Moving Average ARMA(p,q)Typical stationary time series models come in three general classes, auto-regressive (AR) models, moving average (MA) models, or a combination of the two (ARMA).
16 Identifying appropriate Model The ACF plays an extremely crucial role in the identification of time series modelsThe identification of the particular model within ARMA class of models is determined by looking at the ACF and PACF.
17 Partial Autocorrelation Function (PACF) Partial Autocorrelation is the partial cross-correlation of a time series data with itself based on lag kPartial correlation is a conditional correlation:It is the correlation between two variables under the assumption that we know and take into account the values of some other set of variablesHow Zt and Zt-k are correlated taking into account how both Zt and Zt-k are related to Zt-1 , Zt-2 , ... , Zt-k+1The kth order PACF measure correlation between Zt and Zt+k after adjustments have been made for the intermediate observations Zt-1 , Zt-2 , ... , Zt-k+1where denotes the projection of x onto the space spanned by Zt-1 , Zt-2 , ... , Zt-k+1Partial Correlation: For instance, consider a regression context in which y = response variable and x1, x2, and x3 are predictor variables. The partial correlation between y and x3 is the correlation between the variables determined taking into account how both y and x3 are related to x1 and x2.In regression, this partial correlation could be found by correlating the residuals from two different regressions: (1) Regression in which we predict y from x1 and x2, (2) regression in which we predict x3 from x1 and x2. Basically, we correlate the “parts” of y and x3 that are not predicted by x1 and x2.
18 ARMA Model identification from ACF and PACF AR(p)MA(q)ARMA(p, q)Infinite dampedexponentials and/ordamped sine waves;Tails offFinite; cuts offafter q lagsInfinite dampedexponentials and/ordamped sine waves;Tails offACFInfinite dampedexponentials and/ordamped sine waves;Tails offFinite; cuts offafter p lagsInfinite dampedexponentials and/ordamped sine waves;Tails offPACFSource: Adapted from BJR
20 Models for Non-Stationary Data Standard autoregressive moving average (ARMA) time series models apply only to stationary time series.The assumption that a time series is stationary is quite unrealistic. (Stationary is not natural!)For a system to exhibit a stationary behaviour, it has to be tightly controlled and maintained in time.Otherwise, systems will tend to drift away from stationary
21 Converting Non-Stationary Data to Stationary More realistic is to claim that the changes to a process, or the first difference, form a stationary process.And if that is not realistic, we mat try to see if the changes of the changes, the second difference, form a stationary process.If that is the case, we can then model the changes, make forecasts about the future values of these changes, and from the model of the changes build models and create forecasts of the original non-stationary time series.In practice, we seldom need to go beyond second order differencing.
22 Auto Regressive Integrated Moving Average (ARIMA) In the case of non-stationary data, differencing before we use the (stationary) ARMA model to fit the (differenced) data is appropriate.Because the inverse operation of differencing is summing or integrating, an ARMA model applied to d differenced data is called an autoregressive integrated moving average process, ARIMA (p, d, q).In practice, the orders p, d, and q are seldom higher than 2.
23 Stages of the time series model building process using ARIMA Consider a generalARIMA ModelIdentify the appropriatedegree of differencing if neededUsing ACF and PACF, find a tentative modelEstimate the parameters of the model using appropriate softwarePerform the residual analysis.Is the model adequate?Start forecasting
24 Model EvaluationOnce a model has been fitted to the data, we process to conduct a number of diagnostic checks.If the model fits well, the residuals should essentially behave like white noise.In other words, the residuals should be uncorrelated with constant variance.Standard checks areto compute the ACFand PACF of the residuals.If they appear in theconfidence interval there isno alarm indications thatthe model does not fit well.
25 Exponentially Weighted Moving Average Special case of ARIMA model: EWMAUnlike a regular average that assigns equal weight to all observation, an EWMA has a relatively short memory that assigns decreasing weights to past observations.EWMA made practical sense that a forecast should be a weighted average that assigns most weight to the most immediate past observation, somewhat less weight to the second to the last observation, and so on.It just made good practical sense.
26 Seasonal ModelsFor ARIMA models, the serial dependence of the current observation to the previous observations was often strongest for the immediate past and followed a decaying pattern as we move further back in time.For some systems, this dependence shows a repeating, cyclic behaviour.This cyclic pattern or as more commonly called seasonal pattern can be effectively used to further improve the forecasting performance.The ARIMA models are flexible enough to allow for modeling both seasonal and non-seasonal dependence.
28 Trend and Seasonal Relationship Two relationship going on simultaneously:Between observations for successive months within the same yearBetween observation for the same month in successive years.Therefore, we essentially need to build two time series models, and then combine the two.If the season is s period long, in this example s = 12 months, then observation that are s time intervals apart are alike.
29 Pre-Processing Log Transformation The variability is not constant, but increases over time and gets larger.The goal of the transformation is to identify a scale where the residuals, after fitting a model, will have homogeneous variability.
30 Apply Differencing on Seasonal Data For seasonal data, we may need to use not only regular difference but also a seasonal difference .Sometimes, we may even need both (e.g., ) to obtain an ACF that dies out sufficiently quickly.
31 Investigate ACFsOnly the last one (combination of regular difference and seasonal difference) is stationary:
32 Model IdentificationIdentifying stationary seasonal models is a modification of the one used for regular ARMA time series models where the patterns of the sample ACF and PACF provide guidance.First, look for similarities that are 12 lags apart.ACF seems to cut off after the first one (in k=12).This is a sign of a Moving Average Model applied to the 12- month seasonal pattern.Second, look for patterns betweensuccessive monthsACF seems to cut off after the first oneFirst order MA term in the regular modelIn non-seasonal data, it is usually sufficient to consider the ACF and PACF for up to lags. Bur for seasonal data, we recommend increasing the number of lags to at least 3 or 4 multiples of the seasonality.ACFPACF
33 Model EvaluationACF of the residuals after fitting a first order SMA model to :We see that the ACF shows a significant negative spike at lag 1, indicating that we need an additional regular moving average term
35 Example 2: Energy Peak Load Prediction The hourly peak load followsa daily periodic patternS=24 hoursCovert peak loadvalues into and thenapply ARMAACFPACF
36 Example 3: Energy Load Prediction Daily, weekly, andmonthly periodic patternsExogenous Variables(Temperature)They proposed to apply Periodic Auto-Regression (PAR)* An auto-regression is periodic when the parameters are allowed to vary across seasons.
37 Example 3 (cont’d) Proposed model template: Seasonality varying intercept termProposed model template:Dummy variablefor weekly seasonalDummy variablefor monthly seasonalExogenous variablefor temperaturesensitivity
38 Time Series Data Mining Using Serial Dependency of forecasting variable to build the training set.Leading indicators might exhibit similar behaviour to forecasting variableThe important task is to find out whether there exists a lagged relationship between indicators and predicted variableIf such a relationship exists, then from the current and past behaviour of the leading indicators, it may be possible to determine how the sales will behave in the near future.
39 Time Series SVM Optimization problem in SVM: Error in SVM: Error in Modified SVM:
40 Example 4: Stock Market Analysis Portfolio optimization is the decision process of asset selection and weighting, such that the collection of assets satisfies an investor’s objectivesSerial dependency or LaggedRelationship betweenstock performance andfinancial indicators fromthe companies.
41 Stock RankingLearn relationship between stocks’ current features and their future rank score. (Lagged Relationship)By Applying modified version of SVM Rank Algorithm for time series based on exponential weighted error.
42 References Søren Bisgaard and M. Kulahci, TIME SERIES ANALYSIS AND FORECASTING BY EXAMPLE: A JOHN WILEY & SONS, INC.,  Rayman Preet Singh, Peter Xiang Gao, and Daniel J. Lizotte, "On Hourly Home Peak Load Prediction," in IEEE SmartGridComm,  Marcelo Espinoza, Caroline Joye, Ronnie Belmans, and Bart De Moor, "Short-Term Load Forecasting, Proﬁle Identiﬁcation, and Customer Segmentation: A Methodology Based on Periodic Time Series," Power Systems, vol. 20, pp ,  F. E. H. Tay and L. Cao, "Modified support vector machines in financial time series forecasting," Neurocomputing, vol. 48, pp , 2002