High Performance GridFTP Transport of Earth System Grid (ESG) Data 1 Center for Enabling Distributed Petascale Science

Description Transfer 10TBs of climate data into the SC09 show floor from three sites – the Argonne Leadership Computing Facility (ACLF), the National Energy Research Scientific Computing center (NERSC) and LLNL. As the data arrives at its destination in the University of Utah’s SC09 booth, it will be stored on disks provided by the Data Direct Networks. Data will be processed using climate data analysis and visualization tool and then publicly displayed along with graphs depicting the characteristics of the transfer.

End-to-End Flow

Scientific Purpose Climate data is moved in this challenge  Climate is a discipline that is highly collaborative, and its datasets are distributed across the globe.  An interesting feature of climate data is that the actual file size is not very large compared to that of other sciences.  Climate researchers, however, need to move hundreds or thousands of files in a single transfer.  Volume of data to be moved across the network is massive. Multiple TB of data from Climate Research Program Coupled Model Intercomparison Project, Phase 3 (CMIP3) is moved  This data was used in the Intergovernmental Panel on Climate Change (IPCC) Fourth Assessment Report (AR4)  This data is used in anticipation of the approaching IPCC Fifth Assessment Report (AR5)

How Computing and Network map into Climate Modeling Efforts Each Climate Modeling task maps onto these strategic objectives from:

Network Challenges in ESG Independent gateways federating metadata and users Individual data nodes responsible for publishing services Designed for model output data sets

Technical Approach and Methods Transfers initiated by the climate community can be between a client and a server or between two remote servers initiated by the user from a third machine. GridFTP and other data movement tools developed by Center for Enabling Distributed Petascale Science (CEDPS) are ideal for these types of transfers GridFTP is optimized for high-bandwidth, wide area networks. Globus implementation of GridFTP provides a software suite optimized for a broad range of data access applications  Including bulk file transfer and data extraction from complex storage systems.

GridFTP Advantages Performance - Orders of magnitude performance improvements over standard FTP  Uses parallel TCP streams and non-TCP protocols such as UDT  coordinated transfer using multiple computers at source and destination. Secure - GridFTP supports the PKI/X.509 based Grid Security Infrastructure (GSI) – simple options to encrypt/integrity check data GridFTP also supports SSH security Robust - Restart markers allow interrupted transfers to restart with minimal delay overhead. Extensible – Clear abstractions to interface with various transport protocols and with different storage systems  Completely shields user from the complexities of underlying storage systems including tape archves such as HPSS

Key GridFTP Features used in the Challenge Concurrency and Pipelining  Allows the client to simultaneously maintain multiple outstanding, unacknowledged transfer commands  Greatly improves performance lots of small files transfers File Request 1 File Request 2 File Request 3 DATA 1 DATA 2 DATA 3 ACK 1 ACK 2 ACK 3 File Request 1 File Request 2 File Request 3 DATA 1 DATA 2 DATA 3 ACK 1 ACK 2 ACK 3 Traditional Pipelining

GridFTP Clients and Netlogger Three different GridFTP clients are used to move the 10 TB data set for the challenge  Globus.org – hosted data movement service  BDM – Bulk Data Mover  Globus-url-copy Netlogger – used to monitor transfers and troubleshoot problems  Distributed performance analysis and troubleshooting  Standard log format and best practices  Log collection tools  Log parser  Data analysis tools

What is the Globus.org Data Movement Service (a.k.a. DataKoa) ? A new Globus data movement service  The same vision, but an updated implementation  Hosted  Domain-independent, multi-use Enables scientists to focus on domain-specific work  Manages technology failures  Sends notifications of interesting events Enables non-experts to easily and efficiently move data  No operations overhead  Minimal user-side software installation  User interfaces require no special expertise  Built-in data transport configuration expertise

GridFTP Server A GridFTP Server B Globus.org Laptop Globus.org Data Movement Service The client connects to Globus.org and submits requests. It can then disappear from the network Globus.org orchestrates the transfer between GridFTP servers.

What is BDM? BDM: Bulk Data Mover  Scalable data movement management tool Calls GridFTP file transfers  Designed for climate community (Earth System Grid) needs Efficient and reliable transfer management from user’s point of view Simple to install and maintain as a novice user Scalable to large in volume Scalable to large in number of files Efficient handling on extreme variance in file sizes Scalable to future performance expectations  Network performance improvements – 100Gbps and beyond  Storage performance improvements – distributed, parallel, SSD, etc.  Multiple transfer protocol support  Able to work with other applications with similar needs Information   Contact: Dean Williams

Globus-url-copy Commonly used command line scriptable GridFTP client Supports various transfer optimizations including parallel TCP streams, concurrent file transfers New features  Fault tolerant Store state in a file Restarting globus-url-copy transfers only the remaining data  Associate multiple physical endpoints with single logical endpoint  Load balance across all the physical endpoints 9/15/09Argonne National Laboratory

NetLogger BWC Deployment ALCF LLNL NERSC LBNL GridFTP servers SC09 Show Floor Data Logs NetLogger DB Plots on the web

Data Direct Networks Silicon Storage Architecture (S2A)

ESnet Science Data Network Good network is as important having the right tools and applications.  needed a good network that would move these datasets at high speeds to the convention center  ESnet was the perfect fit to pull data from national labs Science Data Networks (SDN) and On-Demand Secure Circuit and Advance Reservation System (OSCARS)  guarantees that we will have a dedicated circuit on the network for the duration of the challenge  don’t have to compete with anyone else for bandwidth 9/15/09Argonne National Laboratory

Data Analysis and Visualization The data were analyzed using the Climate Data Analysis Tools (CDAT) developed by Program for Climate Model Diagnosis and Intercomparison (PCMDI) CDAT is a suite of interrelated diagnostic software tools  Flexible, portable, adaptable, efficient, easy-to-use, shareable and free  Capable of operating in a distributed environment 3D Interface provided by the ViSUS plugin developed at the SCI Institute at University of Utah and LLNL  Streaming and progressive data flow  Integrated analysis and illustration tools 9/15/09Argonne National Laboratory

Data Analysis and Visualization Full Video is available at /

Overarching Research Agenda Climate community is expecting to generate petabytes of simulated data for analysis and future climate predictions. In the next few years, climate researchers will be moving terabytes of data to collaborators across the globe for IPCC Fifth Assessment Report (AR5), which will be published in Moving large amounts of data seamlessly, reliably and quickly is required to make sense of the enormous AR5 climate data set  Help scientists understand climatic imbalances and the potential impacts of future climate change scenarios. 9/15/09Argonne National Laboratory

Overarching Research Agenda This demonstration highlights the tools and services that will help them transport their data quickly and reliably Hope that the lessons learned in this experiment will help us to do this better Improve the transport and monitoring tools further and help not only the climate researchers but also other researchers in getting their science done faster than before 9/15/09Argonne National Laboratory