Presentation on theme: "Environmental Data Analysis with MatLab Lecture 2: Looking at Data."— Presentation transcript:
Environmental Data Analysis with MatLab Lecture 2: Looking at Data
Lecture 01Using MatLab Lecture 02Looking At Data Lecture 03 Probability and Measurement Error Lecture 04Multivariate Distributions Lecture 05Linear Models Lecture 06The Principle of Least Squares Lecture 07Prior Information Lecture 08Solving Generalized Least Squares Problems Lecture 09Fourier Series Lecture 10Complex Fourier Series Lecture 11Lessons Learned from the Fourier Transform Lecture 12Power Spectra Lecture 13Filter Theory Lecture 14Applications of Filters Lecture 15Factor Analysis Lecture 16Orthogonal functions Lecture 17Covariance and Autocorrelation Lecture 18Cross-correlation Lecture 19Smoothing, Correlation and Spectra Lecture 20Coherence; Tapering and Spectral Analysis Lecture 21Interpolation Lecture 22 Hypothesis testing Lecture 23 Hypothesis Testing continued; F-Tests Lecture 24 Confidence Limits of Spectra, Bootstraps SYLLABUS
purpose of the lecture get you started looking critically at data
Objectives when taking a first look at data Understand the general character of the dataset. Understand the general behavior of individual parameters. Detect obvious problems with the data.
Tools for Looking at Data covered in this lecture reality checks time plots histograms rate information scatter plots
Black Rock Forest Temperature I downloaded the weather station data from the International Research Institute (IRI) for Climate and Society at Lamont-Doherty Earth Observatory, which is the data center used by the Black Rock Forest Consortium for its environmental data. About 20 parameters were available, but I downloaded only hourly averages of temperature. My original file, brf_raw.txt has time in a format that I thought would be hard to work with, so I wrote a MatLab script, brf_convert.m, that converted it into time in days, and wrote the results into the file that I gave you.
format conversion calendar date/time days from start of first year of data sequential time variable need for data analysis but format conversions provide opportunity for error to creep into dataset 0100-0159 2 Jan 1997 1.042
Reality Checks properties that your experience tells you that the data must have check you expectations against the data
Reality Checks What do you expect the data to look like? hourly measurements thirteen years of data location in New York (moderate climate)
take a moment... to sketch a plot of what you expect the data to look like
Reality Checks What do you expect the data to look like? hourly measurements thirteen years of data location in New York (moderate climate) time increments by 1/24 day per sample about 24*365*13 = 113880 lines of data temperatures in the -20 to +35 deg C range diurnal and seasonal cycles
Does time increment by 1/24 days per sample? 0 17.2700 0.0417 17.8500 0.0833 18.4200 0.1250 18.9400 0.1667 19.2900 1/24 = 0.0417 Yes D(1:5,:)
Are there about 24*365*20 = 113880 lines of data ? length(D) 110430 Yes
temperatures in the -20 to +35 deg C range? diurnal and seasonal cycles?
annual cycle cold spikes hot spike data drop-outs -20 to +35 range Temperatures in the -20 to +35 deg C range? Mostly Diurnal and seasonal cycles? Certainly seasonal.
Data Drop-outs common in datasets the instrument wasn’t working for a while … take two forms: missing rows of table data set to some default value 0 n/a -999 all common
cold spike diurnal cycle data drop-out 50 days of data from winter50 days of data from summer
Histograms determine range of the majority of data values quantifies the frequency of occurrence of data at different data values easy to spot over-represented and under- represented values
temperature, ºC counts Histogram of Black Rock Forest temperatures
B)A) temperature, ºC counts Alternate ways of displaying a histogram
Series of histograms, each on a relatively short time interval of data Advantage: Shows the way that the frequency of occurrence of data varies with time Disadvantage: Each histogram is computed using less data, and so is less accurate Moving-Window Histograms
60 0 40 temperature, C 05000 time, days Moving-Window Histogram of Black Rock Forest temperatures
good use of FOR loop offset=1000; Lw=floor(N/offset)-1; Dhist = zeros(Lh, Lw); for i = [1:Lw]; j=1+(i-1)*offset; k=j+offset-1; Dhist(:,i) = hist(d(j:k), bins)'; end
Rate Information how fast a parameter is changing with time or with distance
Hypothesis rate of change in discharge correlates with amount of discharge logic a river is bigger when it has high discharge a big river flows faster than a small river a river that flows faster drains away water faster (might only be true after the rain has stopped)
MatLab Script purpose: make two separate plots, one for times of increasing discharge, one for times of decreasing discharge pos = find(dddt>0); neg = find(dddt<0); - - - plot(d(pos),dddt(pos),'k.'); - - - plot(d(neg),dddt(neg),'k.');
Atlantic Rock Dataset I downloaded rock chemistry data from PetDB’s website at www.petdb.org. Their database contains chemical information about ocean floor igneous and metamorphic rocks. I extracted all samples from the Atlantic Ocean that had the following chemical species: SiO2, TiO2, Al2O3, FeOtotal, MgO, CaO, Na2O and K2O My original file, rocks_raw.txt included a description of the rock samples, their geographic location and other textual information. However, I deleted everything except the chemical data from the file, rocks.txt, so it would be easy to read into MatLab. The order of the columns is as is given above and the units are weight percent.
Using scatter plots to look for correlations among pairs of the eight chemical species 8! / [2! (8-2!)] = 28 plots
Al 2 0 3 Ti0 2 Al 2 0 3 Si0 2 K20K20 Fe0 Mg0 Al 2 0 3 A)B) C)D) four interesting scatter plot