1. pRasterBlaster: High-Performance Small-Scale Raster Map Projection Transformation Using the Extreme Science and Engineering Discovery Environment U.S. Department of the Interior U.S. Geological Survey Michael P. Finn

2. Collaborators/ Co-Authors Yan Liu – University of Illinois at Urbana-Champaign (UIUC), CyberInfrastructure and Geospatial Information (CIGI) Laboratory David M. Mattli – USGS, Center of Excellence for Geospatial Information Science (CEGIS) Qingfeng (Gene) Guan – University of Nebraska -- Lincoln, School of Natural Resources Anand Padmanabhan – UIUC, CIGI Laboratory Kristina H. Yamamoto – USGS, CEGIS Babak Behzad – UIUC, Department of Computer Science Eric Shook – UIUC, CIGI Laboratory

3. Overview Parallel computing for raster (map projection, other) – Raster was born to be parallelized Use a large amount of computing power for a relatively short period of time CEGIS parallel computing as part of the US National Science Foundation CyberGIS Grant Data Input/ Output (I/O) Challenges

4. Raster Processing & HPC-Research (map projection case) Raster processing too slow or even impossible on desktop machines for large datasets Map projection/ reprojection for raster datasets mapIMG Example: Re-projecting a 1GB raster dataset can take 45-60 minutes pRasterBlaster: mapIMG in HPC environment Rigorous geometry handling and novel resampling – Resampling options for categorical data and population counts (also standard continuous data resampling methods)

5. Motivation for HPC-Research Solve problems using multiple processors In this case, XSEDE, a virtual system that scientists can use to interactively share computing resources, data and expertise Create user-centric Web 2.0 interface that hides non-important-to-user, complex HPC implementation details

6. Objectives Develop a rapid raster projection transformation and Web service using HPC technology -- pRasterBlaster Provide a software package that better handles known problems for wide use in the modeling community

7. pRasterBlaster (the Early Days) Very similar to mapIMG (V. 03) Row-wise decomposition I/O occurred directly in program inner loop Disadvantages – Inefficient I/O caused by too many I/O requests – Changing I/O implementation difficult – Difficulty with multiple processors writing to a shared file system

8. Fast, Accurate Raster Reprojection in Three (primary) Steps Step 1: Calculate and Partition Output Space Step 2: Read Input and Reproject Step 3: Combine Temporary Files

9. Step 1: Calculate and Partition Output Space (Each map projection represents a distinct coordinate system) The area of the output raster dataset must be calculated by finding a minbox. The edge of the input raster dataset is iterated over translating input coordinates to output The smallest box that contains all of the calculated output coordinates is the minbox The calculated output minbox is then partitioned into areas to be assigned to processors Each output partition is matched with a partition in the input raster dataset This partition pair, input and output, is a single task for a processor

10. Step 2: Read Input and Reproject (Each processor is assigned a quantity of input/output partition pairs) Memory is allocated to hold the output and input partitions The input partition is read from the file system For each pixel in the output, the equivalent pixels in the input are used to find the resampled value Once the resampling is complete, write the output raster to a per- processor temporary file

11. Steps 1& 2 Output Space Partitioning Details (partitions can be arbitrary rectangular areas) - An example output coordinate space overlayed with a grid representing the partitions (left) - The magenta rectangle from the output raster coordinate space and the rhombus from the in put raster dataset represent the same area. (Each area would be loaded into memory.) Step 1: Calculate and Partition Output Space (below) Step 2: Read Input and Reproject (right)

12. Step 3: Combine Temporary Files (Each processor writes its output to an exclusive temporary file) After all of the processors finish their partitions, they each take turns copying their temporary file contents to the final output file Output Raster Dataset (Mollweide projection) Note the red areas - they are outside of the projected space -- these areas have important performance consequences, especially for “Load Balancing.”

13. Why Output to a Set of Temp Files? (Instead of a Single Output) Because, file locking and I/O traffic congestion on shared file system – This is caused by too many processors making I/O requests – The issue is not obvious on small-size clusters; But on supercomputers, it's a big issue The temp file strategy is particularly efficient on supercomputers – Because of fast local I/O but limited network-based I/O For example, Trestles@SDSC.; each trestles node has a 100G solid- state drive as temp storage at runtime; But the shared file system (Lustre) has limited bandwidth

14. Program Evaluation & Integration Process Wall time vs. speedup – Amdahl’s Law: calculate maximum speedup Scalability profiling Resolving computational bottlenecks Build, test, and deployment pRasterBlaster in CyberGIS Toolkit – http://cybergis.cigi.uiuc.edu/toolkit pRasterBlaster as service

15. Computational Bottleneck I: Workload Distribution 15

16. Profiling 16

17. Profiling cont’d 17

18. Workload Distribution Issue 18 N rows on P processor cores When P is smallWhen P is big

19. Load Balancing 19 19 N rows on P processor cores When P is smallWhen P is big

20. Computational Bottleneck II: Spatial Data- dependent Performance Anomaly

21. Profiling 21

22. Analysis The anomaly is data dependent Four corners of the raster were processed by processors whose indexes are close to the two ends Discovery: Exception handling in C++ is costly – Coordinate transformation on nodata area was handled as an exception Solution – Use a more efficient method to handle the nodata areas instead of using C++ exception handling 22

23. Performance after Improvement 23

24. Performance Observations Dynamic Partitioning Load Balancing Spatial Data-dependent Performance File I/O on Shared Clusters/ Parallel File I/O

25. Current Implementation (1.0) I/O and reprojection separated into stages I/O can be batched and reordered Partitioning more flexible – Partition can be less than a row

26. Version 1.0 Components Prologue – Open input files, calculate size of output, calculate partitions, create output and temp files Reprojection – for each partition in parallel – Read input to memory, iterate through output pixels and reproject, write partition to temporary files Epilogue – Write final output by aggregating temporary files from all computing nodes

27. Why Aggregate Temporary Files from All Computing Nodes? (to Write Final Output) Because the temp file strategy in current implementation (V 1.0) cannot leverage full potential of underlying parallel file system on supercomputers – Even though the strategy employed in V 1.0 allows for a certain degree of performance improvement Parallel I/O is such a solution

28. I/O Scalability I/O subsystems are slow compared to other parts of cyberinfrastructure resources – I/O bandwidth is easily saturated – Once saturated I/O performance stops scaling Adding more compute nodes increases aggregated memory bandwidth and flops/s, but not I/O I/O is hence a very expensive operation 28

29. Motivations Using low-level parallel I/O libraries requires in-depth understanding of MPI and parallel I/O – Could be challenging for GIS developers Generic parallel I/O libraries ignore spatial characteristics of data – Understanding the spatial nature of data leads to better optimized I/O operations – e.g. Row-based I/O, Column-based I/O, Block- based I/O 29

30. Solution to the I/O Problem Provide a usable spatially-aware parallel I/O library Included as part of CyberGIS Toolkit – Approach that focuses on developing advanced computational capabilities – Establishes an unique service to resolve I/O issues significant to scientific problem solving 30

31. Serial I/O: Spokesperson Approach One process performs I/O – Data aggregation or duplication – Limited by a single I/O process Simple, but does not scale! 31 P0P0 P1P1 P2P2 PnPn Storage Device … One File

32. Parallel I/O: One-File-per-Process All processes perform I/O from/to individual files – Limited by the file-system Does not scale to large process count – A large number of files may lead to a bottleneck due to metadata operations 32 P0P0 P1P1 P2P2 PnPn Storage Device … File n File 2 File 1 File 0

33. Parallel I/O: Shared File Shared File – Each process performs I/O to a single shared file Performance – Data organization within the shared file is very important – For the cases of large process counts, contention may build for file system resources 33 P0P0 P1P1 P2P2 PnPn … Shared File Storage Device

34. Parallel I/O Methods and Tools Different categories of I/O software and libraries have been developed to address I/O challenges – Parallel file systems: maintains logical space, provides efficient access to data (e.g., Lustre, GPFS) – MPI-IO: deals with organizing access by many processes – High-level libraries: maps application abstractions to structured and portable file formats (e.g. HDF5, NetCDF-4) 34 Storage Hardware HDF5, NetCDF-4 MPI I/O Parallel File System Application

35. Parallel Spatial Data I/O Library Provides a simple and easy to use API to read/write in parallel from/to a NetCDF-4 file – Uses NetCDF API – Written in C – Uses MPI High level library – Exploits spatial characteristics of data – Data could be read/written in row, column, or chunks 35 P0P0 P1P1 P2P2 PnPn … Storage Device...

36. Parallel Spatial Data I/O Library 36 P0P0 P1P1 P2P2 PnPn … Storage Device... Provides a simple and easy to use API to read/write in parallel from/to a NetCDF-4 file – Uses NetCDF API – Written in C – Uses MPI High level library – Exploits spatial characteristics of data – Data could be read/written in row, column, or chunks

37. Parallel Spatial Data I/O Library 37 P0P0 P1P1 P2P2 PnPn … Storage Device... Provides a simple and easy to use API to read/write in parallel from/to a NetCDF-4 file – Uses NetCDF API – Written in C – Uses MPI High level library – Exploits spatial characteristics of data – Data could be read/written in row, column, or chunks

38. A NetCDF Example 38 netcdf example_file { dimensions: x = 512 ; y = 256 ; variables: float temp(x, y) ; temp:units = "celsius" ; data: temp = 10.3, 15.7, …, 20.1, 14.2, 12.5, …, 18.9, … 11.3, 13.8, …,10.7; } example_file temp x y units 10.315.7…20.1 14.212.5…18.9 ………… 11.313.8…10.7

39. A Sample Application in pIOLibrary Main() { – Initialize – Read from the input file read_dbl_variable_by_name(“test”, read_ncid, read_vinfo, read_dinfo); – Do the computation and store your results in memory(i.e. arrays) – Write the results write_dbl_variable_by_name(“test”, write_ncid, write_vinfo, write_dinfo, &data[0]); – Finalize } 39

40. pIOLibrary API Documentation: – http://sandbox.cigi.illinois.edu/pio http://sandbox.cigi.illinois.edu/pio 40

41. Preliminary Results Performance of our library executing parallel read and write, to and from memory tested using the 5.4 GB NetCDF-4 dataset from NCAR 41

42. Current Status pRasterBlaster development continuing Deployment: Trestles@SDSC, Ranger@TACC pRasterBlaster Version 1.0 released 01 April 2012 pRasterBlaster in CyberGIS Toolkit – Co-development between USGS and UIUC using the CyberGIS SVN repository Identified two substantive bottlenecks pRasterBlaster as service – Open service API interface available in the prototype – Demand for high-performance reprojection in CyberGIS analytics pRasterBlaster Version 1.5 soon (fix identified bottlenecks; parallel I/O Library) Iterative testing mapIMG 4.0 (dRasterBlaster -- continuing to be released this fall) Library of core functions shared by both dRasterBlaster & pRasterBlaster (libRasterBlaster -- almost complete) pIOLibrary API (& Documentation): http://sandbox.cigi.illinois.edu/piohttp://sandbox.cigi.illinois.edu/pio

43. Plans for Near-Term Release pRasterBlaster 1.5 Finish libRasterBlaster Release dRasterBlaster Final Decisions on User Processes/ Interface(s)

44. References Atkins, D. E., K. K. Droegemeier, et al. (2003). Revolutionizing Science and Engineering Through Cyberinfrastructure: Report of the National Science Foundation Blue-Ribbon Advisory Panel on Cyberinfrastructure. Arlington, VA, National Science Foundation. Canters, F. (2002). Small-Scale Map Projection Design. London: Taylor & Francis. Finn, Michael P., and David M. Mattli (2012). User’s Guide for the mapIMG 3: Map Image Reprojection Software Package. U. S. Geological Survey Open-File Report 2011-1306, 12 p.. Finn, Michael P., Daniel R. Steinwand, Jason R. Trent, E. Lynn Usery, Robert A. Buehler and David Mattli (2011). An Implementation of MapImage, a Program for Creating Map Projections of Small Scale Geospatial Raster Data. Paper submitted to the journal Cartographic Perspectives. Jenny, B., T. Patterson, and L. Hurni (2010) Graphical Design of World Map Projections, International Journal of Geographical Information Science, Vol. 24, No. 11, 1687-1702. Slocum, T. A., R. B. McMaster, F. C. Kessler, and H. H. Howard (2009). Thematic Cartography and Geovisualization. 3 rd Edition. Upper Saddle River, NJ: Pearson Prentice Hall. Wang, Shaowen and Yan Liu (2009) TeraGrid GIScience Gateway: Bridging cyberinfrastructure and GIScience. International Journal of Geographical Information Science, Volume 23, Number 5, May, pages 631-656. Wang, Shaowen, Yan Liu, Nancy Wilkins-Diehr, and Stuart Martin (2009) SimpleGrid toolkit: Enabling geosciences gateways to cyberinfrastructure. Computers and Geosciences, Volume 35, Number 12, December, pages 2283-2294. Zaslavsky, Ilya, Richard Marciano, Amarnath Gupta, and Chaitanya Baru (2000) XML-Based Spatial Data Mediation Infrastructure for Global Interoperability. Proceedings 4th Global Spatial Data Infrastructure Conference, Cape Town.

45. DISCLAIMER & ACKNOWLEDGEMENT DISCLAIMER: Any use of trade, product, or firm names in this paper is for descriptive purposes only and does not imply endorsement by the U.S. Government. ACKNOWLEDGEMENT: This work used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation (NSF) grant number OCI-1053575. Additional support: NSF grants, BCS- 0846655, OCI-1047916, and XSEDE SES070004N

46. pRasterBlaster: High-Performance Small-Scale Raster Map Projection Transformation Using the Extreme Science and Engineering Discovery Environment U.S. Department of the Interior U.S. Geological Survey QUESTIONS?