1. TESTING AMI PERFORMANCE OF J-DEMETRA+ Turaç YAVUZ TURKSTAT

2. Contents Why to test ? Log / Level Test
Errors in detection of seasonality
Average number of outliers
Correct model identification
Average number of stationary parameters
Conclusion

3. Why to test ? Importance of AMI
An average user of a seasonal adjustment software might not be a modeling expert.
It can get chaotic when one would have to deal with large number of series.

4. Testing Environment; Data used; CPU: Intel Core i5 3.20 Ghz RAM: 4 GB
Platform: Windows 7 64 bit
Software: Jdemetra & 1.2.0
Data used;
Simulated series by Agustin Maravall using Matlab

5. Data Structure
120 obs
Log
Level
240 obs

6. Data Structure
For each category ≈500 series generated by 50 different type of ARIMA model;
16 Airline-type (0,1,q) (0,1,Q)
16 Non-seasonal
18 Other seasonal

7. Errors in Log/Level Test (in % of series) TRAMO-SEATS
Series in Logs
Series in Levels
120
240
Airline-type
2.6
0.0
Non-seasonal
5.3
0.5
Other seasonal
3.3
0.2
0.1
TOTAL
3.7
TOTAL (TSW+)
1.0
0.4
0.2
0.0
Difference between version and is insignificant

8. Errors in Log/Level Test (in % of series) X13
Series in Logs
Series in Levels
120
240
Airline-type
4.1
0.0
Non-seasonal
10.0
1.0
Other seasonal
3.7
0.2
TOTAL
5.8
0.4
0.1
TOTAL (TSW+)
1.0
0.4
0.2
0.0
Difference between version and is insignificant

9. Errors in Detection of Seasonality Test (%) TRAMO-SEATS
Number of Obs.
120
240
Airline-type
0.0
Non-seasonal
3.3
2.6
Other seasonal
0.1
TOTAL
1.1
0.8
TOTAL (TSW+)
0.7
0.8
Difference between version and is insignificant

10. Detection of Seasonality (frequencies for “Seasonality” variable in output) X13
120 obs.
240 obs. . 1
Airline-type 7
15593 3
15997
Non-seasonal
153
14846 80
14920
Other seasonal
16449
16500
For version, all values of seasonality variable in output file are “1”.

11. Detection of Seasonality Visual Spectral Test (%) TRAMO-SEATS
Bad
Good
Severe
Airline-type
120
4.3
95.1
0.6
240
4.8
94.4
0.8
Non-seasonal
21.8
68.1
10.1
20.9
68.3
10.8
Other seasonal
5.4
93.6
1.0
5.6
93.3
1.1
TOTAL
10.2
85.9
3.9
Difference between version and is insignificant

12. Average Number of Outliers per Series TRAMO-SEATS
Series in Logs
Series in Levels
120
240
Airline-type
0.18
0.11
Non-seasonal
0.15
0.10
0.09
Other seasonal
0.17
0.16
TOTAL
0.12
TOTAL (TSW+)
0.17
0.10
Difference between version and is insignificant

13. Average Number of Outliers per Series X13
Series in Logs
Series in Levels
120
240
Airline-type
0.07
0.06
0.08
Non-seasonal
Other seasonal
0.05
TOTAL
TOTAL (TSW+)
0.17
0.10
Difference between version and is insignificant

14. Correct Model Identification (%) TRAMO-SEATS
Complete Model Orders
Differencing
120
240
Airline-type
71.3
75.2
96.9
98.1
Non-seasonal
58.5
68.0
90.0
92.0
Other seasonal
45.0
70.6
90.6
96.1
TOTAL
58.1
71.4
92.5
95.5
TOTAL (TSW+)
64.5
78.9
94.0
97.5
Difference between version and is insignificant

15. Correct Model Identification (%) X13
Complete Model Orders
Differencing
120
240
Airline-type
68.1
73.8
94.4
96.6
Non-seasonal
33.6
29.1
72.2
74.8
Other seasonal
34.9
47.2
86.7
87.6
TOTAL
45.7
50.4
84.7
86.6
TOTAL (TSW+)
64.5
78.9
94.0
97.5
Difference between version and is insignificant

16. Correct Model Identification By Model Order (Top 5) TRAMO-SEATS
Complete Model Orders
Differencing
(3,0,0) (0,0,0)
93.2
99.5
(2,0,0) (0,0,0)
99.3
(1,1,0) (0,0,0)
92.4
99.2
(1,0,0) (0,0,0)
91.7
96.6
(0,0,2) (0,0,0)
90.3
97.0
Difference between version and is insignificant

17. Correct Model Identification By Model Order (Bottom 5) TRAMO-SEATS
Complete Model Orders
Differencing
(0,0,0) (0,0,0)
0.0
90.5
(0,1,0) (0,0,0)
98.2
(2,0,1) (0,0,0)
2.9
47.1
(1,0,1) (1,0,0)
14.0
74.8
(1,0,1) (0,0,0)
23.2
72.4
Difference between version and is insignificant

18. Average Number of Stationary Parameters TRAMO-SEATS
120
240
In Simulation Model
Airline-type
1.86
1.85
1.69
Non-seasonal
1.59
1.63
1.50
Other seasonal
2.36
2.58
2.64
TOTAL
1.95
2.03
1.96
TOTAL (TSW+)
1.93
1.97
1.94
Difference between version and is insignificant

19. Average Number of Stationary Parameters X13
120
240
In Simulation Model
Airline-type
1.86
1.84
1.69
Non-seasonal
2.03
2.44
1.50
Other seasonal
2.42
2.75
2.64
TOTAL
2.11
2.35
1.96
TOTAL (TSW+)
1.93
1.97
1.94
Difference between version and is insignificant

20. Conclusions Run-time facts;
Jdemetra+ is remarkably faster than other SA softwares (TSW, TSW+)
Average process time of 8000 series with 120 observation is ≈ 29 seconds for TRAMO-SEATS method (45 seconds for X13)
10 of series could’nt be processed with TRAMO-SEATS method (and 1 of for X13)

21. Conclusions I/O facts;
Results of some diagnostic tests are not included in the csv output file (Friedman, Kruskall-Wallis, Evolutive Seasonality etc.).
Even if the combined seasonality test result is “Identifiable seasonality not present”, the value of “Seasonality” variable in csv output is “1”.

22. Conclusions ARIMA Model Identification;
Reliability level of X13 is not adequate,
For TRAMO-SEATS method, +Jdemetra is more reliable than TSW but not TSW+.