1. Stationary Time Series
AMS 586

2. The Moving Average Time series of order q, MA(q)
Let {Xt|t  T} be defined by the equation.
where {Zt|t  T} denote a white noise time series with variance s2.
Then {Xt|t  T} is called a Moving Average time series of order q. (denoted by MA(q))

3. The mean value for an MA(q) time series
The autocovariance function for an MA(q) time series
The autocorrelation function for an MA(q) time series

4. Comment
The autocorrelation function for an MA(q) time series
“cuts off” to zero after lag q. q

5. The Autoregressive Time series of order p, AR(p)
Let {Xt|t  T} be defined by the equation.
where {Zt|t  T} is a white noise time series with variance s2.
Then {Zt|t  T} is called a Autoregressive time series of order p. (denoted by AR(p))

6. The mean of a stationary AR(p)
Assuming {Xt|t  T} is stationary, and take expectations of the equation, we obtain the mean μ:
Now we can center (remove the mean of) the time series as follows:

7. Computing the autocovariance of a stationary AR(p)
Now assuming {Xt|t  T} is stationary with mean zero:
Multiplying by Xt-h, h ≥ 0, and take expectations of the equation, we obtain the Yule-Walker equations for the autocovariance. Note, for a zero mean sequence:

8. Note: For h > 0, we have:
The Autocovariance function s(h) of a stationary AR(p) series
Satisfies the equations:

9. Satisfies the equations:
The Autocorrelation function r(h) of a stationary AR(p) series
Satisfies the equations:
with
for h > p
and

10. or:
where r1, r2, … , rp are the roots of the polynomial
and c1, c2, … , cp are determined by using the starting values of the sequence r(h).

11. Conditions for stationarity
Autoregressive Time series of order p, AR(p)

12. If b1 = 1 and d = 0.
The value of Xt increases in magnitude and Zt eventually becomes negligible.
The time series {Xt|t  T} satisfies the equation:
The time series {Xt|t  T} exhibits deterministic behavior.

13. For a AR(p) time series, consider the polynomial
with roots r1, r2 , … , rp
then {Xt|t  T} is stationary if |ri| > 1 for all i.
If |ri| < 1 for at least one i then {Xt|t  T} exhibits deterministic behavior.
If |ri| ≥ 1 and |ri| = 1 for at least one i then {Xt|t  T} exhibits non-stationary random behavior.

14. since:
and |r1 |>1, |r2 |>1, … , | rp | > 1 for a stationary AR(p) series then
i.e. the autocorrelation function, r(h), of a stationary AR(p) series “tails off” to zero.

15. Special Cases: The AR(1) time
Let {Xt|t  T} be defined by the equation.

16. Consider the polynomial
with root r1= 1/b1
{xt|t  T} is stationary if |r1| > 1 or |b1| < 1 .
If |ri| < 1 or |b1| > 1 then {Xt|t  T} exhibits deterministic behavior.
If |ri| = 1 or |b1| = 1 then {Xt|t  T} exhibits non-stationary random behavior.

17. Special Cases: The AR(2) time
Let {Xt|t  T} be defined by the equation.

18. Consider the polynomial
where r1 and r2 are the roots of b(x)
{Xt|t  T} is stationary if |r1| > 1 and |r2| > 1 .
This is true if b1+b2 < 1 , b2 –b1 < 1 and b2 > -1.
These inequalities define a triangular region for b1 and b2.
If |ri| < 1 or |b1| > 1 then {Xt|t  T} exhibits deterministic behavior.
If |ri| ≥ 1 for i = 1,2 and |ri| = 1 for at least on i then {Xt|t  T} exhibits non-stationary random behavior.

19. Patterns of the ACF and PACF of AR(2) Time Series
In the shaded region the roots of the AR operator are complex b2

20. The Mixed Autoregressive Moving Average
Time Series of order p,q
The ARMA(p,q) series

21. The Mixed Autoregressive Moving Average Time Series of order p and q, ARMA(p,q)
Let b1, b2, … bp , a1, a2, … ap , d denote p + q +1 numbers (parameters).
Let {Zt|t  T} denote a white noise time series with variance s2.
uncorrelated
mean 0, variance s2.
Let {Xt|t  T} be defined by the equation.
Then {Xt|t  T} is called a Mixed Autoregressive- Moving Average time series - ARMA(p,q) series.

22. Mean value, variance, autocovariance function, autocorrelation function of an ARMA(p,q) series

23. Similar to an AR(p) time series, for certain values of the parameters b1, …, bp an ARMA(p,q) time series may not be stationary.
An ARMA(p,q) time series is stationary if the roots (r1, r2, … , rp ) of the polynomial
b(x) = 1 – b1x – b2x2 - … - bp xp
satisfy | ri| > 1 for all i.

24. Assume that the ARMA(p,q) time series {Xt|t  T} is stationary:
Let m = E(Xt). Then or

25. The Autocovariance function, s(h), of a stationary mixed autoregressive-moving average time series {Xt|t  T} be determined by the equation:
Thus

26. Hence

28. We need to calculate:
etc

29. The autocovariance function s(h) satisfies:
For h = 0, 1. … , q:
for h > q:

30. We then use the first (p + 1) equations to determine:
s(0), s(1), s(2), … , s(p)
We use the subsequent equations to determine:
s(h) for h > p.

31. Example:The autocovariance function, s(h), for an ARMA(1,1) time series:
For h = 0, 1: or
for h > 1:

32. Substituting s(0) into the second equation we get:or
Substituting s(1) into the first equation we get:

33. for h > 1:

34. The Backshift Operator B

35. Consider the time series {Xt : t  T} and Let M denote the linear space spanned by the set of random variables {Xt : t  T}
(i.e. all linear combinations of elements of {Xt : t  T} and their limits in mean square).
M is a vector space
Let B be an operator on M defined by:
BXt = Xt-1.
B is called the backshift operator.

36. Note: We can also define the operator Bk with
BkXt = B(B(...BXt)) = Xt-k.
The polynomial operator
p(B) = c0I + c1B + c2B ckBk
can also be defined by the equation.
p(B)Xt = (c0I + c1B + c2B ckBk)Xt .
= c0IXt + c1BXt + c2B2Xt ckBkXt
= c0Xt + c1Xt-1 + c2Xt ckXt-k

37. The power series operator p(B) = c0I + c1B + c2B2 + ...
can also be defined by the equation.
p(B)Xt = (c0I + c1B + c2B )Xt
= c0IXt + c1BXt + c2B2Xt + ...
= c0Xt + c1Xt-1 + c2Xt
If p(B) = c0I + c1B + c2B and q(B) = b0I + b1B + b2B are such that
p(B)q(B) = I
i.e. p(B)q(B)Xt = IXt = Xt
than q(B) is denoted by [p(B)]-1.

38. Other operators closely related to B:
F = B-1 ,the forward shift operator, defined by FXt = B-1Xt = Xt+1 and
∇ = I - B ,the first difference operator, defined by ∇Xt = (I - B)Xt = Xt - Xt-1 .

39. The Equation for a MA(q) time series
Xt= a0Zt + a1Zt-1 +a2Zt aqZt-q + m
can be written
Xt= a(B) Zt + m
where
a(B) = a0I + a1B +a2B aqBq

40. The Equation for a AR(p) time series
Xt= b1Xt-1 +b2Xt bpXt-p + d + Zt
can be written
b(B) Xt= d + Zt
where
b(B) = I - b1B - b2B bpBp

41. The Equation for a ARMA(p,q) time series
Xt= b1Xt-1 +b2Xt bpXt-p + d + Zt + a1Zt-1 +a2Zt aqZt-q
can be written
b(B) Xt= a(B) Zt + d
where
a(B) = a0I + a1B +a2B aqBq
and
b(B) = I - b1B - b2B bpBp

42. Some comments about the Backshift operator B
It is a useful notational device, allowing us to write the equations for MA(q), AR(p) and ARMA(p, q) in a very compact form;
It is also useful for making certain computations related to the time series described above;

43. The partial autocorrelation function
A useful tool in time series analysis

44. The partial autocorrelation function
Recall that the autocorrelation function of an AR(p) process satisfies the equation:
rx(h) = b1rx(h-1) + b2rx(h-2) bprx(h-p)
For 1 ≤ h ≤ p these equations (Yule-Walker) become: rx(1) = b1 + b2rx(1) bprx(p-1)
rx(2) = b1rx(1) + b bprx(p-2)
...
rx(p) = b1rx(p-1)+ b2rx(p-2) bp.

45. In matrix notation:
These equations can be used to find b1, b2, … , bp, if the time series is known to be AR(p) and the autocorrelation rx(h) function is known.

46. If the time series is not autoregressive the equations can still be used to solve for b1, b2, … , bp, for any value of p >1.
In this case
are the values that minimizes the mean square error:

47. Definition: The partial auto correlation function at lag k is defined to be:

48. Comment:
The partial auto correlation function, Fkk is determined from the auto correlation function, r(h)

49. Some more comments:
The partial autocorrelation function at lag k, Fkk, can be interpreted as a corrected autocorrelation between Xt and Xt-k conditioning on the intervening variables Xt-1, Xt-2, ... , Xt-k+1 .
If the time series is an AR(p) time series than
Fkk = 0 for k > p
If the time series is an MA(q) time series than
rx(h) = 0 for h > q

50. A General Recursive Formula for Autoregressive Parameters and the Partial Autocorrelation function (PACF)

51. Let
denote the autoregressive parameters of order k satisfying the Yule Walker equations:

52. Then it can be shown that:
and

53. Proof:
The Yule Walker equations:

54. In matrix form:

55. The equations for

56. and
The matrix A reverses order

57. The equations may be written
Multiplying the first equations by or

58. Substituting this into the second equationor
and

59. Hence
and or

60. Some Examples

61. Example 1: MA(1) time series
Suppose that {Xt|t  T} satisfies the following equation:
Xt = Zt Zt – 1
where {Zt|t  T} is white noise with s = 1.1.
Find:
The mean of the series,
The variance of the series,
The autocorrelation function.
The partial autocorrelation function.

62. Solution Now {Xt|t  T} satisfies the following equation:
Xt = Zt Zt – 1
Thus:
The mean of the series,
m = 12.0
The autocovariance function for an MA(1) is

63. Thus:
The variance of the series,
s(0) =
and
The autocorrelation function is:

64. The partial auto correlation function at lag k is defined to be:
Thus

66. Graph: Partial Autocorrelation function Fkk

67. Exercise: Use the recursive method to calculate Fkk
and

68. Exercise: Use the recursive method to calculate Fkk
and

69. Example 2: AR(2) time series
Suppose that {Xt|t  T} satisfies the following equation:
Xt = 0.4 Xt – Xt – Zt
where {Zt|t  T} is white noise with s = 2.1.
Is the time series stationary?
Find:
The mean of the series,
The variance of the series,
The autocorrelation function.
The partial autocorrelation function.

70. The mean of the series
The autocorrelation function.
Satisfies the Yule Walker equations

71. hence

72. the variance of the series
The partial autocorrelation function.

73. The partial autocorrelation function of an AR(p) time series “cuts off” after p.

74. Example 3: ARMA(1, 2) time series
Suppose that {Xt|t  T} satisfies the following equation:
Xt = 0.4 Xt – Zt Zt – Zt – 1
where {Zt|t  T} is white noise with s = 1.6.
Is the time series stationary?
Find:
The mean of the series,
The variance of the series,
The autocorrelation function.
The partial autocorrelation function.

75. Theoretical Patterns of ACF and PACF
Type of Model
Typical Pattern of ACF
Typical Pattern of PACF
AR (p)
Decays exponentially or with damped sine wave pattern or both
Cut-off after
lags p
MA (q)
lags q
Declines exponentially
ARMA (p,q)
Exponential decay

76. Reference
GEP Box, GM Jenkins, GC Reinsel (1994) Time series analysis: Forecasting and control, Prentice-Hall.
Brockwell, Peter J. and Davis, Richard A. (1991). Time Series: Theory and Methods. Springer-Verlag.
We also thank colleagues who posted their notes as on-line open resources for time series analysis.