1. pRasterBlaster: High-Performance Small-Scale Raster Map Projection Transformation Using the Extreme Science and Engineering Discovery Environment 
Michael P. Finn
U.S. Department of the Interior
U.S. Geological Survey

2. Collaborators/ Co-Authors
Yan Liu
University of Illinois at Urbana-Champaign (UIUC), CyberInfrastructure and Geospatial Information (CIGI) Laboratory
David M. Mattli
USGS, Center of Excellence for Geospatial Information Science (CEGIS)
Qingfeng (Gene) Guan
University of Nebraska -- Lincoln, School of Natural Resources
Anand Padmanabhan
UIUC, CIGI Laboratory
Kristina H. Yamamoto
USGS, CEGIS 
Babak Behzad
UIUC, Department of Computer Science  
Eric Shook

3. Parallel computing for raster (map projection, other)
Overview
Parallel computing for raster (map projection, other)
Raster was born to be parallelized
Use a large amount of computing power for a relatively short period of time
CEGIS parallel computing as part of the US National Science Foundation CyberGIS Grant
Data Input/ Output (I/O) Challenges

4. Raster Processing & HPC-Research (map projection case)
Raster processing too slow or even impossible on desktop machines for large datasets
Map projection/ reprojection for raster datasets
mapIMG Example: Re-projecting a 1GB raster dataset can take minutes
pRasterBlaster: mapIMG in HPC environment
Rigorous geometry handling and novel resampling
Resampling options for categorical data and population counts (also standard continuous data resampling methods)

5. Motivation for HPC-Research
Solve problems using multiple processors
In this case, XSEDE, a virtual system that scientists can use to interactively share computing resources, data and expertise
Create user-centric Web 2.0 interface that hides non-important-to-user, complex HPC implementation details

6. Objectives
Develop a rapid raster projection transformation and Web service using HPC technology -- pRasterBlaster
Provide a software package that better handles known problems for wide use in the modeling community

7. pRasterBlaster (the Early Days)
Very similar to mapIMG (V. 03)
Row-wise decomposition
I/O occurred directly in program inner loop
Disadvantages
Inefficient I/O caused by too many I/O requests
Changing I/O implementation difficult
Difficulty with multiple processors writing to a shared file system

8. Fast, Accurate Raster Reprojection in Three (primary) Steps
Step 1: Calculate and Partition Output Space
Step 2: Read Input and Reproject
Step 3: Combine Temporary Files

9. Step 1: Calculate and Partition Output Space (Each map projection represents a distinct coordinate system)
The area of the output raster dataset must be calculated by finding a minbox.
The edge of the input raster dataset is iterated over translating input coordinates to output
The smallest box that contains all of the calculated output coordinates is the minbox
The calculated output minbox is then partitioned into areas to be assigned to processors
Each output partition is matched with a partition in the input raster dataset
This partition pair, input and output, is a single task for a processor

10. Step 2: Read Input and Reproject (Each processor is assigned a quantity of input/output partition pairs)
Memory is allocated to hold the output and input partitions
The input partition is read from the file system
For each pixel in the output, the equivalent pixels in the input are used to find the resampled value
Once the resampling is complete, write the output raster to a per-processor temporary file

11. Steps 1& 2 Step 2: Read Input and Reproject (right)
Step 1: Calculate and Partition Output Space (below)
Output Space Partitioning Details
(partitions can be arbitrary rectangular areas)
- An example output coordinate space overlayed with a grid representing the partitions (left)
- The magenta rectangle from the output raster coordinate space and the rhombus from the in put raster dataset represent the same area. (Each area would be loaded into memory.)

12. Step 3: Combine Temporary Files (Each processor writes its output to an exclusive temporary file)
After all of the processors finish their partitions, they each take turns copying their temporary file contents to the final output file
Output Raster Dataset (Mollweide projection)
Note the red areas - they are outside of the projected space -- these areas have important performance consequences, especially for “Load Balancing.”

13. Why Output to a Set of Temp Files? (Instead of a Single Output)
Because, file locking and I/O traffic congestion on shared file system
This is caused by too many processors making I/O requests
The issue is not obvious on small-size clusters; But on supercomputers, it's a big issue
The temp file strategy is particularly efficient on supercomputers
Because of fast local I/O but limited network-based I/O
For example, each trestles node has a 100G solid-state drive as temp storage at runtime; But the shared file system (Lustre) has limited bandwidth

14. Program Evaluation & Integration Process
Wall time vs. speedup
Amdahl’s Law: calculate maximum speedup
Scalability profiling
Resolving computational bottlenecks
Build, test, and deployment
pRasterBlaster in CyberGIS Toolkit
pRasterBlaster as service

15. Computational Bottleneck I: Workload Distribution

16. Profiling

17. Profiling cont’d

18. Workload Distribution Issue
N rows on P processor cores
When P is small
When P is big

19. Load Balancing N rows on P processor cores When P is small
When P is big 19

20. Computational Bottleneck II: Spatial Data-dependent Performance Anomaly

21. Profiling

22. Analysis The anomaly is data dependent
Four corners of the raster were processed by processors whose indexes are close to the two ends
Discovery: Exception handling in C++ is costly
Coordinate transformation on nodata area was handled as an exception
Solution
Use a more efficient method to handle the nodata areas instead of using C++ exception handling

23. Performance after Improvement

24. Performance Observations
Dynamic Partitioning
Load Balancing
Spatial Data-dependent Performance
File I/O on Shared Clusters/ Parallel File I/O

25. Current Implementation (1.0)
I/O and reprojection separated into stages
I/O can be batched and reordered
Partitioning more flexible
Partition can be less than a row

26. Version 1.0 Components Prologue
Open input files, calculate size of output, calculate partitions, create output and temp files
Reprojection – for each partition in parallel
Read input to memory, iterate through output pixels and reproject, write partition to temporary files
Epilogue
Write final output by aggregating temporary files from all computing nodes

27. Parallel I/O is such a solution
Why Aggregate Temporary Files from All Computing Nodes? (to Write Final Output)
Because the temp file strategy in current implementation (V 1.0) cannot leverage full potential of underlying parallel file system on supercomputers
Even though the strategy employed in V 1.0 allows for a certain degree of performance improvement
Parallel I/O is such a solution

28. I/O Scalability
I/O subsystems are slow compared to other parts of cyberinfrastructure resources
I/O bandwidth is easily saturated
Once saturated I/O performance stops scaling
Adding more compute nodes increases aggregated memory bandwidth and flops/s, but not I/O
I/O is hence a very expensive operation

29. Motivations
Using low-level parallel I/O libraries requires in-depth understanding of MPI and parallel I/O
Could be challenging for GIS developers
Generic parallel I/O libraries ignore spatial characteristics of data
Understanding the spatial nature of data leads to better optimized I/O operations
e.g. Row-based I/O, Column-based I/O, Block-based I/O

30. Solution to the I/O Problem
Provide a usable spatially-aware parallel I/O library
Included as part of CyberGIS Toolkit
Approach that focuses on developing advanced computational capabilities
Establishes an unique service to resolve I/O issues significant to scientific problem solving

31. Serial I/O: Spokesperson Approach
One process performs I/O
Data aggregation or duplication
Limited by a single I/O process
Simple, but does not scale! P0 P1 P2 Pn
Storage Device …
One File

32. Parallel I/O: One-File-per-Process
All processes perform I/O from/to individual files
Limited by the file-system
Does not scale to large process count
A large number of files may lead to a bottleneck due to metadata operations P0 P1 P2 Pn
Storage Device …
File n
File 2
File 1
File 0

33. Parallel I/O: Shared File
Each process performs I/O to a single shared file
Performance
Data organization within the shared file is very important
For the cases of large process counts, contention may build for file system resources P0 P1 P2 … Pn
Shared File
Storage Device

34. Parallel I/O Methods and Tools
Different categories of I/O software and libraries have been developed to address I/O challenges
Parallel file systems: maintains logical space, provides efficient access to data (e.g., Lustre, GPFS)
MPI-IO: deals with organizing access by many processes
High-level libraries: maps application abstractions to structured and portable file formats (e.g. HDF5, NetCDF-4)
Storage Hardware
HDF5, NetCDF-4
MPI I/O
Parallel File System
Application

35. Parallel Spatial Data I/O Library
Provides a simple and easy to use API to read/write in parallel from/to a NetCDF-4 file
Uses NetCDF API
Written in C
Uses MPI
High level library
Exploits spatial
characteristics of data
Data could be read/written
in row, column, or chunks P0 P1 P2 … Pn .
Storage Device

36. Parallel Spatial Data I/O Library
Provides a simple and easy to use API to read/write in parallel from/to a NetCDF-4 file
Uses NetCDF API
Written in C
Uses MPI
High level library
Exploits spatial
characteristics of data
Data could be read/written
in row, column, or chunks P0 P1 P2 … Pn .
Storage Device

37. Parallel Spatial Data I/O Library
Provides a simple and easy to use API to read/write in parallel from/to a NetCDF-4 file
Uses NetCDF API
Written in C
Uses MPI
High level library
Exploits spatial
characteristics of data
Data could be read/written
in row, column, or chunks P0 P1 P2 … Pn .
Storage Device

38. A NetCDF Example example_file x temp y units netcdf example_file {
dimensions:
x = 512 ;
y = 256 ;
variables:
float temp(x, y) ;
temp:units = "celsius" ;
data:
temp =
10.3, 15.7, …, 20.1,
14.2, 12.5, …, 18.9, …
11.3, 13.8, …,10.7; }
10.3
15.7 …
20.1
14.2
12.5
18.9
11.3
13.8
10.7

39. A Sample Application in pIOLibrary
Main() {
Initialize
Read from the input file
read_dbl_variable_by_name(“test”, read_ncid, read_vinfo, read_dinfo);
Do the computation and store your results in memory(i.e. arrays)
Write the results
write_dbl_variable_by_name(“test”, write_ncid, write_vinfo, write_dinfo, &data[0]);
Finalize }

40. pIOLibrary API
Documentation:

41. Preliminary Results
Performance of our library executing parallel read and write, to and from memory tested using the 5.4 GB NetCDF-4 dataset from NCAR

42. Current Status pRasterBlaster development continuing
Deployment:
pRasterBlaster Version 1.0 released 01 April 2012
pRasterBlaster in CyberGIS Toolkit
Co-development between USGS and UIUC using the CyberGIS SVN repository
Identified two substantive bottlenecks
pRasterBlaster as service
Open service API interface available in the prototype
Demand for high-performance reprojection in CyberGIS analytics
pRasterBlaster Version 1.5 soon (fix identified bottlenecks; parallel I/O Library)
Iterative testing
mapIMG 4.0 (dRasterBlaster -- continuing to be released this fall)
Library of core functions shared by both dRasterBlaster & pRasterBlaster (libRasterBlaster -- almost complete)
pIOLibrary API (& Documentation):

43. Plans for Near-Term Release pRasterBlaster 1.5 Finish libRasterBlaster
Release dRasterBlaster
Final Decisions on User Processes/ Interface(s)

44. References
Atkins, D. E., K. K. Droegemeier, et al. (2003). Revolutionizing Science and Engineering Through Cyberinfrastructure: Report of the National Science Foundation Blue-Ribbon Advisory Panel on Cyberinfrastructure. Arlington, VA, National Science Foundation.
Canters, F. (2002). Small-Scale Map Projection Design. London: Taylor & Francis.
Finn, Michael P., and David M. Mattli (2012). User’s Guide for the mapIMG 3: Map Image Reprojection Software Package. U. S. Geological Survey Open-File Report , 12 p..
Finn, Michael P., Daniel R. Steinwand, Jason R. Trent, E. Lynn Usery, Robert A. Buehler and David Mattli (2011). An Implementation of MapImage, a Program for Creating Map Projections of Small Scale Geospatial Raster Data. Paper submitted to the journal Cartographic Perspectives.
Jenny, B., T. Patterson, and L. Hurni (2010) Graphical Design of World Map Projections, International Journal of Geographical Information Science, Vol. 24, No. 11,
Slocum, T. A., R. B. McMaster, F. C. Kessler, and H. H. Howard (2009). Thematic Cartography and Geovisualization. 3rd Edition. Upper Saddle River, NJ: Pearson Prentice Hall.
Wang, Shaowen and Yan Liu (2009) TeraGrid GIScience Gateway: Bridging cyberinfrastructure and GIScience. International Journal of Geographical Information Science, Volume 23, Number 5, May, pages
 Wang, Shaowen, Yan Liu, Nancy Wilkins-Diehr, and Stuart Martin (2009) SimpleGrid toolkit: Enabling geosciences gateways to cyberinfrastructure. Computers and Geosciences, Volume 35, Number 12, December, pages
Zaslavsky, Ilya, Richard Marciano, Amarnath Gupta, and Chaitanya Baru (2000) XML-Based Spatial Data Mediation Infrastructure for Global Interoperability. Proceedings 4th Global Spatial Data Infrastructure Conference, Cape Town.

45. DISCLAIMER & ACKNOWLEDGEMENT
DISCLAIMER: Any use of trade, product, or firm names in this paper is for descriptive purposes only and does not imply endorsement by the U.S. Government.
ACKNOWLEDGEMENT: This work used the Extreme Science and Engineering Discovery Environment (XSEDE), which is supported by National Science Foundation (NSF) grant number OCI Additional support: NSF grants, BCS , OCI , and XSEDE SES070004N

46. pRasterBlaster: High-Performance Small-Scale Raster Map Projection Transformation Using the Extreme Science and Engineering Discovery Environment 
QUESTIONS?
U.S. Department of the Interior
U.S. Geological Survey