On the Trend, Detrend and the Variability of Nonlinear and Nonstationary Time Series Norden E. Huang Research Center for Adaptive Data Analysis National.

Slides:

Advertisements

Similar presentations

Time series modelling and statistical trends

Advertisements

Filtering the data. Detrending Economic time series are a superposition of various phenomena If there exists a « business cycle », we need to insulate.

Decomposition Method.

On the Trend, Detrend and the Variability of Nonlinear and Nonstationary Time Series A new application of HHT.

An Introduction to HHT: Instantaneous Frequency, Trend, Degree of Nonlinearity and Non-stationarity Norden E. Huang Research Center for Adaptive Data Analysis.

Norden E. Huang Research Center for Adaptive Data Analysis

Quantification of Nonlinearity and Nonstionarity Norden E. Huang With collaboration of Zhaohua Wu; Men-Tzung Lo; Wan-Hsin Hsieh; Chung-Kang Peng; Xianyao.

What is Frequency? Norden E. Huang Research Center for Adaptive Data Analysis Center for Dynamical Biomarkers and translational Medicine National Central.

Properties of EMD Basis The Adaptive Basis based on and derived from the data by the empirical method satisfy nearly all the traditional requirements for.

STAT 497 APPLIED TIME SERIES ANALYSIS

Nonstationary Signal Processing Hilbert-Huang Transform Joseph DePasquale 22 Mar 07.

A Plea for Adaptive Data Analysis: An Introduction to HHT for Nonlinear and Nonstationary Data Norden E. Huang Research Center for Adaptive Data Analysis.

The Hilbert Transform and Empirical Mode Decomposition: Suz Tolwinski University of Arizona Program in Applied Mathematics Spring 2007 RTG Powerful Tools.

Stationarity and Degree of Stationarity

End Effects of EMD An unsolved, and perhaps, unsolvable problem.

Chapter 5 Time Series Analysis

A Plea for Adaptive Data Analysis: Instantaneous Frequencies and Trends For Nonstationary Nonlinear Data Norden E. Huang Research Center for Adaptive Data.

The Empirical Mode Decomposition Method Sifting. Goal of Data Analysis To define time scale or frequency. To define energy density. To define joint frequency-energy.

A Confidence Limit for Hilbert Spectrum Through stoppage criteria.

Regionalized Variables take on values according to spatial location. Given: Where: A “structural” coarse scale forcing or trend A random” Local spatial.

On the Marginal Hilbert Spectrum. Outline Definitions of the Hilbert Spectra (HS) and the Marginal Hilbert Spectra (MHS). Computation of MHS The relation.

Zhaohua Wu and N. E. Huang:

Part II – TIME SERIES ANALYSIS C2 Simple Time Series Methods & Moving Averages © Angel A. Juan & Carles Serrat - UPC 2007/2008.

Business Cycles Empirical Properties. What do we mean by “The Business Cycle”?

X-12 ARIMA Eurostat, Luxembourg Seasonal Adjustment.

MATH 3290 Mathematical Modeling

Global analysis of recent frequency component changes in interannual climate variability Murray Peel 1 & Tom McMahon 1 1 Civil & Environmental Engineering,

On the Marginal Hilbert Spectrum. Outline Definitions of the Hilbert Spectrum (HS) and the Marginal Hilbert Spectrum (MHS). Computation of MHS The relation.

The Analytic Function from the Hilbert Transform and End Effects Theory and Implementation.

On the Trend, Detrend and the Variability of Nonlinear and Nonstationary Time Series.

ENSEMBLE EMPIRICAL MODE DECOMPOSITION Noise Assisted Signal Analysis (nasa) Part II EEMD Zhaohua Wu and N. E. Huang: Ensemble Empirical Mode Decomposition:

Statistical Methods for long-range forecast By Syunji Takahashi Climate Prediction Division JMA.

Time Series and Forecasting

CHAPTER 18 Models for Time Series and Forecasting

Estimating potential output using business survey data in a SVAR framework 3° annual WORKSHOP on Macroeconomic Forecasting Montreal 5-6 october 2007 Tatiana.

CHAPTER 3 FORECASTING.

Non-continuous Relationships If the relationship between the dependent variable and an independent variable is non-continuous a slope dummy variable can.

TIME SERIES by H.V.S. DE SILVA DEPARTMENT OF MATHEMATICS

Oceanography 569 Oceanographic Data Analysis Laboratory Kathie Kelly Applied Physics Laboratory 515 Ben Hall IR Bldg class web site: faculty.washington.edu/kellyapl/classes/ocean569_.

On the relationship between C n 2 and humidity Carlos O. Font, Mark P. J. L. Chang, Erick A. Roura¹, Eun Oh and Charmaine Gilbreath² ¹Physics Department,

E MPIRICAL M ODE D ECOMPOSITION BASED T ECHNIQUE APPLIED IN EXPERIMENTAL BIOSIGNALS Alexandros Karagiannis Mobile Radio Communications Laboratory School.

An introduction to Empirical Mode Decomposition. The simplest model for a signal is given by circular functions of the type Such “Fourier modes” are of.

Data Mining: Neural Network Applications by Louise Francis CAS Annual Meeting, Nov 11, 2002 Francis Analytics and Actuarial Data Mining, Inc.

Time series Decomposition Farideh Dehkordi-Vakil.

Week 11 Introduction A time series is an ordered sequence of observations. The ordering of the observations is usually through time, but may also be taken.

Correlation of temperature with solar activity (SSN) Alexey Poyda and Mikhail Zhizhin Geophysical Center & Space Research Institute, Russian Academy of.

Testing seasonal adjustment with Demetra+ Dovnar Olga Alexandrovna The National Statistical Committee, Republic of Belarus.

ECNE610 Managerial Economics Week 4 MARCH Dr. Mazharul Islam Chapter-5.

Economics 173 Business Statistics Lecture 26 © Fall 2001, Professor J. Petry

Dr. C. Ertuna1 Required Textbook “Econometrics: By Example " Gujarati Palgrave-Macmillian 2011.

COMPLETE BUSINESS STATISTICS

Time Series and Forecasting

Ensemble Empirical Mode Decomposition Zhaohua Wu Center for Ocean-Land-Atmosphere Studies And Norden E Huang National Central University.

Detection of Intermittent Turbulence In Stable Boundary Layer Using Empirical Mode Decomposition Xiaoning Gilliam, Christina Ho, and Sukanta Basu Texas.

Time Series and Forecasting Chapter 16 McGraw-Hill/Irwin Copyright © 2012 by The McGraw-Hill Companies, Inc. All rights reserved.

Detecting Signal from Data with Noise Xianyao Chen Meng Wang, Yuanling Zhang, Ying Feng Zhaohua Wu, Norden E. Huang Laboratory of Data Analysis and Applications,

Chapter 20 Time Series Analysis and Forecasting. Introduction Any variable that is measured over time in sequential order is called a time series. We.

The Empirical Mode Decomposition Method Sifting. Goal of Data Analysis To define time scale or frequency. To define energy density. To define joint frequency-energy.

Inhomogeneities in temperature records deceive long-range dependence estimators Victor Venema Olivier Mestre Henning W. Rust Presentation is based on:

Chapter 20 Time Series Analysis and Forecasting. Introduction Any variable that is measured over time in sequential order is called a time series. We.

ENSEMBLE EMPIRICAL MODE DECOMPOSITION Noise Assisted Signal Analysis (nasa) Part II EEMD Zhaohua Wu and N. E. Huang: Ensemble Empirical Mode Decomposition:

Forecasting. Model with indicator variables The choice of a forecasting technique depends on the components identified in the time series. The techniques.

Chapter 11 – With Woodruff Modications Demand Management and Forecasting Copyright © 2010 by The McGraw-Hill Companies, Inc. All rights reserved.McGraw-Hill/Irwin.

Yandell – Econ 216 Chap 16-1 Chapter 16 Time-Series Forecasting.

Lecture 9 Forecasting. Introduction to Forecasting * * * * * * * * o o o o o o o o Model 1Model 2 Which model performs better? There are many forecasting.

Lecture 16: Hilbert-Huang Transform Background:

Wu, Z. , N. E. Huang, S. R. Long and C. K

HISTORICAL AND CURRENT PROJECTIONS

Part 5 - Chapter 17.

Seasonal adjustment with Demetra+

Presentation transcript:

On the Trend, Detrend and the Variability of Nonlinear and Nonstationary Time Series Norden E. Huang Research Center for Adaptive Data Analysis National Central University, Taiwan

Satellite Altimeter Data : Greenland

Two Sets of Data

IPCC Global Mean Temperature Trend

The State-of-the-Arts “ One economist’s trend is another economist’s cycle” Engle, R. F. and Granger, C. W. J Long-run Economic Relationships. Cambridge University Press. Simple trend – straight line Stochastic trend – straight line for each quarter

Philosophical Problem 名不正則言不順言不順則事不成 —— 孔夫子

On Definition Without a proper definition, logic discourse would be impossible. Without logic discourse, nothing can be accomplished. Confucius

Definition of the Trend Within the given data span, the trend is an intrinsically determined monotonic function, or a function in which there can be at most one extremum. The trend should be determined by the same mechanisms that generate the data; it should be an intrinsic and local property. Being intrinsic, the method for defining the trend has to be adaptive. The results should be intrinsic (objective); all traditional trend determination methods give extrinsic (subjective) results. Being local, it has to associate with a local length scale, and be valid only within that length span as a part of a full wave cycle.

Definition of Detrend and Variability Within the given data span, detrend is an operation to remove the trend. Within the given data span, the Variability is the residue of the data after the removal of the trend. As the trend should be intrinsic and local properties of the data; Detrend and Variability are also local properties. All traditional trend determination methods are extrinsic and/or subjective.

The Need for HHT HHT is an adaptive (local, intrinsic, and objective) method to find the intrinsic local properties of the given data set, therefore, it is ideal for defining the trend and variability.

Two Sets of Data

Global Temperature Anomaly Annual Data from 1856 to 2003

Global Temperature Anomaly 1856 to 2003

IMF Mean of 10 Sifts : CC(1000, I)

Mean IMF

STD IMF

Statistical Significance Test

Data and Trend C6

Data and Overall Trends : EMD and Linear

Rate of Change Overall Trends : EMD and Linear

Variability with Respect to Overall trend

Data and Trend C5:6

Data and Trends: C5:6

Rate of Change Trend C5:6

Trend Period C5

Variability with Respect to 65-Year trend

How are GSTA data derived? Noise Reduction Using Global Surface Temperature Anomaly data 1856 to 2003

Jones (2003) Monthly GSTA Data

Jones (2003) 12 Monthly GSTA Data

Jones (2003) GSTA Data Seasonal Variation

Jones (2003) GSTA Data Seasonal Variance

Jones Monthly GSTA Data : Fourier Spectrum

Observations Annual data is actually the mean of 12:1 down sample set of the original monthly data. In spite of the removal of climatologic mean, there still is a seasonal peak (1 cycle / year). Seasonal Variation and Variance are somewhat irregular. Data contain no information beyond yearly frequency, for higher frequency part of the Fourier spectrum is essentially flat. Decide to filtered the Data with HHT before down sample.

Need a Filter to Remove Alias Traditional Fourier filter is inadequate: –Removal of Harmonics will distort the fundaments –Noise spikes are local in time; signals local in time have broad spectral band HHT is an adaptive filter working in time space rather than frequency space.

EMD as filters

Jones Monthly GSTA Data : IMF

Jones Monthly GSTA Data : IMF Smoothed

Jones Monthly GSTA Data & HHT Smoothed

Jones Monthly GSTA Data : Fourier Spectrum Data & Smoothed

12 Monthly GSTA Data HHT Smoothed

Jones (2003) 12 Monthly GSTA Data

GSTA : Annual Data Jones and HHT Smoothed For the Difference : Mean = ; STD =

GSTA : Annual Variance Jones and HHT Smoothed Mean HHT = ; Jones =

GSTA : HHT Smoothed Seasonal Variation

GSTA : HHT Smoothed Seasonal Variance

Summary Global Surface Temperature Anomaly should not be derived from simple annual average, because there are noises in the data. Noise with period shorter than one year could have caused alias in down sampling. Smoothing the data by removing any data with a period shorter than 8 months should improved the annual mean.

Financial Data : NasDaqSC October 11, 1984 – December 29, 2000 October 12, 2004

NasDaq Data

NasDaq IMF

NasDaq IMF Reconstruction : A

NasDaq IMF Reconstruction : B

NasDaq Various Overall Trends

NasDaq various Overall Detrends Mean : L = 0 Exp = EMD = STD : L = Exp = EMD =

NasDaq Trend IMF (C8-C9)

NasDaq Local Period for Trend IMF (C8-C9) mean = 796.6

NasDaq Trend IMF (C7-C9)

NasDaq Local Period for Trend IMF (C7-C9) Mean = 425.7

NasDaq Trend IMF (C6-C9)

NasDaq Local Period for Trend IMF (C6-C9) Mean = 196.5

NasDaq Traditional Moving Mean Trends: Details

NasDaq Trends: Moving Mean and EMD : Details

NasDaq Period of EMD Trend (C4) Mean = 35.56

NasDaq Distribution of Period for EMD Trend (C4)

NasDaq Detrended Data (C4-C9)

NasDaq Detrended Data (C4-C9) : Details

NasDaq Histogram Detrended Data (C1-C3)

Various Definitions of Variability Variability defined by percentage Gain is the absolute value of the Gain. Variability defined by daily high-low is the percentage of absolute value of High-Low. Variability defined by Empirical Mode Decomposition is the percentage of the absolute value of the sum from selected IMFs. Financial data do not look like ARIMA.

NasDaq Variability defined by EMD : C1

NasDaq Variability defined by Gain

NasDaq Variability defined by Daily High-Low

NasDaq Period of Variability defined by EMD : C1 Mean = 8.38

NasDaq Histogram Period of EMD Variability : C1

NASDAQ Price gradient vs. Gain Variability

NASDAQ Price gradient vs. High-Low Variability

NASDAQ Price gradient vs. EMD Variability

Relationship between Variability: Gain vs. EMD

Relationship between Variability: Gain vs. High- Low

Relationship between Variability: EMD vs. High- Low

Statistical Significance Test Only the statistical Significant IMF components are signal above noise; therefore, they might be predictable.

Statistical Significance Test : Gain

Statistical Significance Test : High-Low

Statistical Significance Test : EMD

Statistical Significance Test : All Variability Definitions

The Sum of all the Statistical Significance IMFs

Relationship among Trends: Gain vs. EMD

Relationship among Trends: Gain vs. High-Low

Relationship among Trends: EMD vs. High-Low

Summary A working definition for the trend is established; it is a function of the local time scale. Need adaptive method to analysis nonstationary and nonlinear data for trend and variability. Various definitions for variability should be compared in details to determine their significance. Predictions should be made based on processes driven models, not on data.

Conclusion Trend is a local property of the data; it should associate with a length scale. Trend should be determined adaptively; therefore, we should not pre-select the functional form of the trend. Variability should have a reference; the trend is a good reference.