1. ADIOS – http://www. adiosapi.org1 Jay Lofstead Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS) Jay Lofstead (GT), Scott Klasky (ORNL), Karsten Schwan(GT), Norbert Podhorszki (ORNL), Chen Jin (ORNL) June 23, 2008

2. ADIOS – http://www. adiosapi.org 2Jay Lofstead Outline Motivation Design Goals Architecture Related Work Supported Features Examples Codes & Performance Future Work Collaborators

3. ADIOS – http://www. adiosapi.org 3Jay Lofstead Motivation Multiple HPC architectures Cray, IB-based clusters, BlueGene Many different APIs MPI-IO, POSIX, HDF5, netCDF GTC (fusion) has changed IO routines 8 times so far based on moving platforms Different IO patterns Restarts, analysis, diagnostics Different combinations provide different levels of I/O performance

4. ADIOS – http://www. adiosapi.org 4Jay Lofstead Design Goals ADIOS Fortran and C based API almost as simple as standard POSIX IO External configuration to describe metadata and control IO settings Take advantage of existing IO techniques (no new native IO methods) Fast, simple-to-write, efficient IO for multiple platforms without changing the source code

5. ADIOS – http://www. adiosapi.org 5Jay Lofstead Design Goals ADIOS Contributions: 1.Data Grouping: Multiple, independently controlled IO settings diagnostics, restarts, analysis, viz 2.Optional data items: Different data from different processes Single group write has header from a single proc and data from all 3.Data sizes change dynamically: Datasets vary in size Run size changes data sizes; also AMR codes 4.Constructed output is special: Reused buffers must be handled properly stack temporaries... 5.IO memory is second to science: Buffer space for IO is strictly limited respect the memory needs of the scientific codes

6. ADIOS – http://www. adiosapi.org 6Jay Lofstead Design Goals Provide support for these future features: Provide integration opportunities for other systems Workflow In situ visualization Other simulations Provenance collection Other data management tools Science aware data compression

7. ADIOS – http://www. adiosapi.org 7Jay Lofstead Architecture Thin API XML file data groupings with annotation IO method selection buffer sizes Common tools Buffering Scheduling Pluggable IO routines External Metadata (XML file) Scientific Codes ADIOS API DART LIVE/DataTapMPI-IO POSIX IO HDF-5 pnetCDF Viz Engines Others (plug-in) bufferingschedulefeedback

8. ADIOS – http://www. adiosapi.org 8Jay Lofstead Architecture Data groupings logical groups of related items written at the same time. Not necessarily one group per writing event IO Methods Choose what works best for each grouping Vetted and/or written by experts for each POSIX (Wei-keng Lao Northwestern) MPI-IO (Steve Hodson, ORNL) MPI-IO Collective (Wei-keng Lao Northwestern) NULL (Jay Lofstead GT) Ga Tech DataTap Asynchronous (Hasan Abbasi GT) others..

9. ADIOS – http://www. adiosapi.org 9Jay Lofstead Related Work Specialty APIs HDF-5 – complex API Parallel netCDF – no structure File system aware middleware MPI ADIO layer – File system connection, complex API Parallel File systems Lustre – Metadata server issues PVFS2 – client complexity LWFS – client complexity GPFS, pNFS, Panasas

10. ADIOS – http://www. adiosapi.org 10Jay Lofstead Supported Features Platforms tested Cray CNL (ORNL Jaguar) Cray Catamount (ORNL old-Jaguar and SNL Redstorm) Linux Infiniband (ORNL Ewok) IO Methods MPI-IO independent, MPI-IO collective, POSIX, NULL, Ga Tech DataTap asynchronous, Rutgers DART asynchronous

11. ADIOS – http://www. adiosapi.org 11Jay Lofstead Examples Setup/Cleanup code call adios_init (‘config.xml’)... call adios_finalize (my_rank) adios_init – reads the XML file (once for every process) adios_finalize – provide opportunity for cleanup

12. ADIOS – http://www. adiosapi.org 12Jay Lofstead Examples IO code call adios_open (handle, ‘filename’, ‘groupname’) call adios_write (handle, ‘varname’, var) ADIOS_WRITE(handle,var_name)... call adios_close (handle)

13. ADIOS – http://www. adiosapi.org 13Jay Lofstead Examples IO code call adios_open (handle, ‘filename’, ‘groupname’) ADIOS_GWRITE (handle) call adios_close (handle) - ADIOS_GREAD coming soon

14. ADIOS – http://www. adiosapi.org 14Jay Lofstead Examples IO hints (for asynchronous IO) call adios_end_iteration () pacing hints for asynchronous transport methods use in conjunction with ‘iterations’ attribute of method element in XML call adios_begin_calculation () hint for asynchronous IO that a low-IO phase is starting call adios_end_calculation () hint for asynchronous IO that a low-IO phase is ending

15. ADIOS – http://www. adiosapi.org 15Jay Lofstead Examples XML file contents (data elements) <var name=“data” type=“double” path=“/” dimensions=“elements” copy-on-write=“yes”/>

16. ADIOS – http://www. adiosapi.org 16Jay Lofstead Examples XML file contents (other) params

17. ADIOS – http://www. adiosapi.org 17Jay Lofstead Codes & Performance Codes GTC (fusion), Chimera (supernova), GTS (fusion), XGC0 (fusion), XGC1 (fusion), Flash (astrophysics), S3D (combustion) Performance Measurements GTC: nearly 25 GB/sec GTS: 12 GB/sec Chimera: 6.5% reduction in wall clock time

18. ADIOS – http://www. adiosapi.org 18Jay Lofstead Codes & Performance

19. ADIOS – http://www. adiosapi.org 19Jay Lofstead Codes & Performance Old GTC IO New GT ADIOS/DataTap IO 12 GB/restart GTC On Rizzo at ORNL

20. ADIOS – http://www. adiosapi.org 20Jay Lofstead Codes & Performance June 7, 2008: 24 hour GTC run on Jaguar at ORNL 93% of machine (28,672 cores) MPI-OpenMP mixed model on quad-core nodes (7168 MPI procs) three interruptions total (simple node failure) with 2 10+ hour runs Wrote 56 TB of data at 20 GB/sec (21 TB for post analysis) IO overhead 3% of wall clock time Mixed IO methods of synchronous MPI-IO and POSIX IO configured in the XML file

21. ADIOS – http://www. adiosapi.org 21Jay Lofstead Support Tools bpdump – similar to h5dump bp2h5 – convert output file to HDF5 bp2nc – convert output to netCDF

22. ADIOS – http://www. adiosapi.org 22Jay Lofstead Future Work Work on read functionality Index files Additional native transports direct parallel HDF5 and parallel netCDF asynchronous MPI Visit others as demanded

23. ADIOS – http://www. adiosapi.org 23Jay Lofstead Future Work Offload to staging area for other processing

24. ADIOS – http://www. adiosapi.org 24Jay Lofstead Collaborators Chen Jin, Scott Klasky, Stephen Hodson, James B. White III, Weikuan Yu (Oak Ridge National Laboratory) Hasan Abbasi, Greg Eisenhauer, Karsten Schwan, Matthew Wolf, Fang Zheng (Georgia Tech) Wei-keng Liao, Alok Choudhary, Northwestern University Manish Parashar, Ciprian Docan, Rutgers University. Ron Oldfield, Sandia Labs

25. ADIOS – http://www. adiosapi.org 25Jay Lofstead References Jay Lofstead, Scott Klasky, Karsten Schwan, Norbert Podhorszki, Chen Jin. “Flexible IO and Integration for Scientific Codes Through The Adaptable IO System (ADIOS).” In CLADE 2008 at HPDC, Boston, MA, June 2008. ACM. Hasan Abbasi, Matthew Wolf, Karsten Schwan. “Live data workspace: A flexible, dynamic and extensible platform for petascale applications.” In Cluster Computing, Austin, TX, September 2007. IEEE International. Chen Jin, Scott Klasky, Stephen Hodson, Weikuan Yu (Oak Ridge National Laboratory) Jay Lofstead, Hasan Abbasi, Karsten Schwan, Matthew Wolf (Georgia Tech) Wei-keng Liao, Alok Choudhary(Northwestern University) Manish Parashar, Ciprian Docan, Rutgers University. Ron Oldfield (Sandia National Laboratories). “Adaptive IO System (ADIOS).” In CUG 2008. Helsinki, Findland, May 2008. Sarp Oral. NCCS Jaguar GTC 128 Nodes Test Results. http://users.nccs.gov/~oral/jagregtests/gtc128.html.