1. TerraStream: From Elevation Data to Watershed Hierarchies Thursday, 08 November 2007 Andrew Danner (Swarthmore), T. Moelhave (Aarhus), K. Yi (HKUST), P. K. Agarwal (Duke), L. Arge (Aarhus), H. Mitasova (NCSU)

2. Andrew Danner2 Current Problem: Large Point Data Sets LIDAR –NC Coastline: 200 million points – over 7 GB –Neuse River basin (NC): 500 million points – over 17 GB Grid elevation models –Neuse River basin: *20ft – 2.5 GB *10ft – 10GB *5ft – 40 GB Data too big for RAM –Must reside on disk –Disk is slow

3. Andrew Danner3 Traditional algorithms optimize CPU computation –Not aware of performance penalty of disk access –Virtual memory, swap space can’t predict disk access I/O model –Memory is finite –Data is transferred in blocks –Complexity measured in disk blocks transferred – I/O-efficient Algorithms [AV88] Disk RAM CPU M B

4. Andrew Danner4 TerraStream: Terrain processing pipeline

5. Andrew Danner5 TerraStream Goals Scalable – All stages must work for 100+ million points/cells General – Stages should work with either TIN or grid data Automated – No need for manual intervention/preprocessing Modular – Users only need to run the stages they want Adaptable – Allow each stage to support multiple models

6. Andrew Danner6 TerraStream: Terrain processing pipeline

7. Andrew Danner7 Points to DEM Grid DEM –Interpolation –Use quad tree to automatically tile terrain –Use quad tree neighbors for smooth boundary transitions TIN –I/O efficient Delaunay triangulation –Constrained Delaunay also possible if constraints (breaklines) fit in memory Height graph −View both grids and TINs as a height graph. −Nodes, neighbors, and edges between neighboring nodes −Definition of node, neighbor different in TIN/Grid −Design algorithms to work on height graphs

8. Andrew Danner8 Flow Modeling

9. Andrew Danner9 Identifying minima due to noise Removing noise from terrains Modeling flow directions, extracting river networks Flow Modeling

10. Andrew Danner10 Identifying minima likely due to noise –Topological persistence – Computed in Sort(N) I/Os –Assign a significance score to each minima (low score  likely noise) –Provide mechanism for removing low scoring sinks –User can select score threshold Coping with Noisy Data

11. Andrew Danner PhD Defense 11 Noise Removal Flooding in Sort(N) I/Os Other Mechanisms? Carving? Noisy terrain After noise removal

12. Andrew Danner12 From Elevation to River Networks Where does water go? From higher elevation to lower elevation Single flow directions form a tree Support for multiple flow directions Sea 80 90 100 95 85 110

13. Andrew Danner13 Drainage Area How much area is upstream of each node? Each node has initial drainage area (1) Drainage area of internal nodes depends on drainage area of children 1 1 1 1 1 1 1 1 1 1 3 3 2 2 2 3 2 5 7 17 25 14 11 9 7

14. Andrew Danner14 Computing Flow Directions/Drainage Terraflow –Sort(N) I/Os on grids Modified to work on height graphs –Same I/O bound –Now works on TINs New implementation –More robust, portable –Incorporate new sink removal –Better handling of flat areas…

15. Andrew Danner15 Flow Modeling Improvements Detection of flat areas –Improved method on grids if O(1) rows fit in memory Routing on flat areas –Soille extension of Garbrecht & Martz Flat areas usually result of hydrological conditioning with flooding

16. Andrew Danner16 Hierarchical Watershed Decomposition

17. Andrew Danner17 Watershed Hierarchies Decompose a river network into a hierarchy of hydrological units All water in HU flows to a common outlet Hierarchy provides tunable level of detail Method used: Pfafstetter Want a solution scalable to large modern hi-res terrains

18. Andrew Danner18 Pfafstetter Find main river Find four largest tributaries Label basins/interbasins Recurse until single path 1 1 1 1 1 1 1 1 1 1 2 2 2 3 2 7 17 25 14 11 9 7 3 3 5 8 6 4 2 7 5 3 1 9

19. Andrew Danner19 Example Watershed Boundaries 1 2 3 4 5 6 7 8 9 91 92 93 94 95 96 97 98 99 991 992 993 994 995 996 997 998 999 All levels computed in one run. User selects level of detail with map algebra

20. Andrew Danner20 A Complete Pipeline

21. Andrew Danner21 Implementation TPIE: C++ primitives for I/O-efficient algorithms Standalone command line apps with GDAL GRASS: Open Source GIS Plugins ArcGIS Plugins (soon) Test Data: –North Carolina LIDAR *Neuse river basin: 400 million points (NC Floodmaps) *Outer banks coastal data : 128 million points (NOAA CSC) –USGS 30m NED

22. Andrew Danner22 Our Results Experimental Results –Scales to over 400 million points –Other software tools crash at 25 million points –Keeps memory usage low using I/O efficient methods Format20ft grid10ft gridTIN HG vertices (millions)3971590469 Pipeline stage Dem Construction19h 56m27h 12m4h 20m Building height graph0h 07m0h 30m11h 42m Hydro. Conditioning1h 17m7h 25m10h 03m Flow Modeling Routing1h 26m6h 34m15h 08m Accumulation1h 40m7h 35m2h 05m Watershed Delineation2h 28m14h 39m6h 26m Total25h 54m63h 34m49h 44m

23. Andrew Danner23 Future Directions – Grid Construction Interpolate leaves in parallel Test other interpolation methods Test with more data sources Finding the “ideal” resolution

24. Andrew Danner24 Future Directions – Noise Removal Bridge detection/removal Hydrological conditioning with carving Scoring of sinks based on volume Other flow routing methods Further flat routing improvements

25. Andrew Danner25 Flow Routing and Bridges Use flooded terrain for connectivity but Use original terrain for routing

26. Andrew Danner26 Questions?