1. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam WFM 5201: Data Management and Statistical Analysis Akm Saiful Islam Lecture-5: Data Exploration March, 2011 Institute of Water and Flood Management (IWFM) Bangladesh University of Engineering and Technology (BUET)

2. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Data Exploration through Graphical Means Bar diagram Pie chart Histogram Stem and Leaf Plot Frequency Plot Scatter diagram Line graph

3. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Bar Diagram Vertical Bar Diagram Horizontal Bar Diagram Component Bar Diagram Multiple Bar Diagram

4. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Vertical Bar Diagram ResponseFrequencyRelative frequency Frequently490.327 Occasionally710.473 Rarely240.160 Never60.040

5. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Horizontal bar diagram

6. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Example: Global Emission of CO 2

7. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Component Bar Diagram Region Population in '000Percent of population MaleFemaleMaleFemale A112281063751.348.7 B176341630652.048.0

8. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Multiple bar diagram Normalized deviation of monthly rainfall over GBM basins for (a)May, (b) June, (c) July, (d) August, and (e) September derived from (b)TRMM Satellite data during 1998 to 2007.

9. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Example: Monthly temperature over Bangladesh (1948-2007)

10. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Pie Chat 2D Pie Chart3D Pie Chart

11. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Example of 2D Pie chart

12. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Histogram

13. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Frequency Polygon Expen.No. 4.5-9.5 8 9.5-14.5 29 14.5-19.5 27 19.5-24.5 12 24.5-29.5 4 Total 80

14. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Scattered Diagram Husband's ageWife's age 19 26 27 29 36 40 35 15 17 19 21 28 36 32 39 26 33 39 25 40 32 30 19 27 34 21 38 21 26 28 29 37 31 - 14 19 21 29 27 -

15. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Scattered Diagram Measured versus predicted water level of Brahmaputra river Prediction was conducted by ANN model with 1 – day lead time

16. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Line Diagram

17. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Line Diagram and Trend of global mean temperature

18. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Example: Trends of precipitation

19. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Exploration of data using Geo- Statistical Analyst Tool of ArcGIS

20. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Explore the Data Distribution of the data, looking for data trends, looking for global and local outliers, examining spatial autocorrelation, understanding the co- variation among multiple data sets.

21. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Explore data Histogram Q-Q plot Trend Analysis Voronoi map Semivariogram Cross covariance

22. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Histogram Show frequency distribution as a bar graph that displays how often observed values fall within certain intervals or classes

23. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Normal distribution Skewness is zero for normal distribution Normally distributed Positively skewed

24. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Histogram Distribution is not normal (+skewness 0.486)

25. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Transformation Log- transformation change distribution pattern (skewness 0.2888)

26. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Q-Q Plot Normal QQ Plot is created by plotting data values with the value of a standard normal where their cumulative distributions are equal

27. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Normal Q-Q Plot Normal Q-Q plot is straight line which represents normal distribution

28. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Trend Analysis The Trend Analysis tool provides a three-dimensional perspective of the data. The locations of sample points are plotted on the x,y plane. Above each sample point, the value is given by the height of a stick in the z dimension. The unique feature of the Trend Analysis tool is that the values are then projected onto the x,z plane and the y,z plane as scatter plots. This can be thought of as sideways views through the three-dimensional data. Polynomials are then fit through the scatter plots on the projected planes.

29. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Trend  An example of a global trend can be seen in the effects of the prevailing winds on a smoke stack at a factory (below).  In the image, the higher concentrations of pollution are depicted in the warm colors (reds and yellows) and the lower concentrations in the cool colors (greens and blues).  Notice that the values of the pollutant change more slowly in the east–west direction than in the north–south direction.  This is because east–west is aligned with the wind while north–south is perpendicular to the wind.

30. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Trend Analysis Shows trend in both X and Y direction since the projection lines (blue and green) are not straight.

31. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Detrending tool

32. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Voronoi map Voronoi maps are constructed from a series of polygons formed around the location of a sample point. Voronoi polygons are created so that every location within a polygon is closer to the sample point in that polygon than any other sample point. After the polygons are created, neighbors of a sample point are defined as any other sample point whose polygon shares a border with the chosen sample point. For example, in the following figure, the bright green sample point is enclosed by a polygon, given as red. Every location within the red polygon is closer to the bright green sample point than any other sample point (given as small dark blue dots). The blue polygons all share a border with the red polygon, so the sample points within the blue polygons are neighbors of the bright green sample point.

33. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Voronoi map Shows the zone of influence of known data points

34. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Semi-variogram

35. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Co-variance

36. WFM 5201: Data Management and Statistical Analysis © Dr. Akm Saiful IslamDr. Akm Saiful Islam Cross variance The Crosscovariance cloud shows the empirical crosscovariance for all pairs of locations between two datasets and plots them as a function of the distance between the two locations.