1. An Advanced Simulation & Computing (ASC) Academic Strategic Alliances Program (ASAP) Center at The University of Chicago The Center for Astrophysical Thermonuclear Flashes Real World IO Experiences With The FLASH Code May 19, 2008 Anshu Dubey

2. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Outline  I/O options for parallel applications  Parallel I/O : What we want  Flash I/O  Libraries  Checkpointing and restarts  Plotfiles  Performance  Experiences  Non-adaptive turbulence simulation on BG/L  Fully adaptive GCD simulations on Power 5/ XT4  Fully adaptive RTFlame simulations on BG/P

3. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago I/O Options: Serial I/O 01234 File processors  Each processor sends its data to the master who then writes the data to a single file  Advantages:  Don’t need a parallel file system  Simple  Disadvantages:  Not scalable  Not efficient 5

4. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago I/O Options: Parallel Multi-file 01234 File processors  Each processor writes its own data to a separate file  Advantages: Fast!  Disadvantages:  can quickly accumulate many files  hard to manage  requires post processing 5 File

5. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago I/O Options: Parallel I/O to a Single-file File processors  Each processor writes its own data to a section of the data array in the same file  Each must know the offset and number of elements to write  Advantages:  single file, scalable  Disadvantages  requires MPI-IO mapping or other higher level libraries 01 2 34 5 array of data

6. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago What Applications Want from I/O q Write data from multiple processors into a single file q File can be read in the same manner regardless of the number of CPUs that read from or write to the file. q Want to see the logical data layout… not the physical layout q Do so with few overheads q Same performance as writing one-file-per-processor q Make all of the above, including files, portable across platforms q self describing formats

7. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago I/O Formats In FLASH qDistribution comes with support for HDF5 and PnetCDF libraries and basic support for direct binary I/O q Direct binary format is for “all else failed” situation only qBoth libraries are q Portable q Use MPI-IO mappings q Are self describing and translate data between systems qThe libraries can be used interchangeably in FLASH qPnetCDF has better performance qHDF5 is more robust, and supports larger file sizes

8. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago I/O in FLASH qLarge Files : q Checkpoint files q Save full state of the simulation q Plot files q Data for analysis qSmaller Files : q Dat files q Integrated quantities, output in serial files q Log files q Status report of the run and logging of important run specific information q Input files q Some simulations need to read files for initialization or table look-up purposes

9. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Checkpoint Files qSimulations can stop at intermediate time for many reasons q Unintentionally for reasons such as machine failure, queue window closing, failure in execution etc q Intentionally for analysis of intermediate results q Determine if the simulation is proceeding as expected q Decide if any parameters need to change q Tweak the direction of the simulation qCrucial to be able to resume the simulation from most of the above mentioned stoppages qCheckpoint files save the full state of the simulation Simulations can be restarted transparently from checkpoint files

10. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago More on Checkpoint Files qCheckpoint files are saved in full precision qThey have no knowledge of hardware that wrote them q Restarts can be done on q Different number of processors q Different platform qTypical Flash Center production run checkpoints are a few GB in size q Some larger runs have reached a few hundred GB for a single checkpoint qBecause they are large and take a long time to write, their frequency needs to be chosen judiciously qWhen disk space is an issue we use rolling checkpoints

11. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH Plotfiles qFLASH can generate two types of plot files q Mesh data (Eulerian) q Particles data (Lagrangian) qPlotfiles are smaller than checkpoints q Checkpoints save all variables, plotfiles save only those that are needed for analysis q Checkpoints are full precision, plotfiles are half precision q Sometimes coarsening of data can be employed to reduce disk use qPlotfiles are more frequent and cannot be rolled Plotfiles store data needed for analysis

12. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH I/O Performance : HDF5 vs. PnetCDF

13. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH IO Performance on BG/P

14. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Improving I/O Performance qSplit I/O q Instead of a single file, write to a few files q Easy in parallel libraries with grouping of processors qUtility to stitch them together, most often concatenation enough

15. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH I/O Experiences qHighlight three different simulations q Run on five different platforms q With two different file systems qTurbulence run in 2005-2006 q On BG/L at Lawrence Livermore National Laboratory q Lustre file system qGravitationally Confined Detonation Simulations q On UP at LLNL, on Seaborg and FRANKLIN at NERSC q GPFS file system with IBM platforms, Lustre with FRANKLIN qRayleigh Taylor Flame Simulations q Currently being run on BG/P at Argonne National Laboratory q GPFS file system

16. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Turbulence Run: The I/O Challenge qLustre file system at LLNL was unknown q FLASH had never run in that combination qNo Parallel I/O library scaled beyond 1024 nodes qQuickly put together Direct I/O implementation q Each node wrote to its own file q Each snapshot meant 32,768 files qLagrangian particles move randomly q For analysis, sorting algorithms needed q Huge processing issue We generated 74 million files in all

17. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Turbulence Run: Overview qLargest homogeneous, isotropic compressible turbulence run q 1856 ^3 base grid size q 256 ^3 Lagrangian tracer particles q 3D turbulent RMS Mach number = 0.3 (1D =.17)  Re ~ 500 -- 1000 q Full eddy-turnover time in steady-state q Roughly one week wall clock on 32,768 nodes qCode Components q Hydrodynamics q Tracer particles q Uniform grid q Direct I/O

18. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Tracer Particles at Time = 0.10

19. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Tracer Particles at Time = 1.00

20. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Tracer Particles at Time = 2.50

21. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Turbulence Run: I/O Statistics q200 checkpoints, each giving full double precision snapshot of the simulation state q 6,553,600 files for a total of about 140 TB q700 plotfiles at reduced precision for Eulerian grid data q Coarsened to half the resolution, making them 1/8th the size q 22,937,600 files for a total of 14 TB q1400 plotfiles for Lagrangian particles q 45,875,200 files for a total of 0.658 TB qTotal disk use about 154 TB qTook more than a month to transfer plotfiles using GridFTP qAvailable to anyone interested q Required months of stitching together and post-processing

22. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Gravitationally Confined Detonation (GCD) Runs Simulation Description qStart the simulation with an off center bubble qThe bubble rises to the surface, developing Rayleigh-Taylor instabilities qThe material cannot escape because of the gravity, so it races around the star qAt the opposite end, the fronts collide to initiate detonation Code Components qHydro with shock capturing scheme (PPM) qNewtonian self gravity (Multi-pole solver) qNuclear flame model qLagrangian tracer particles qAMR qParallel I/O (HDF5)

23. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago GCD Runs: I/O Statistics qSuccessfully used fully parallel HDF5 I/O on all platforms qTypical run generates q 60 checkpoint files q About 1,000 each of grid and particle plot files q Spends about 17% computation time in I/O qSize of files q checkpoint files ~20 GB q plot files ~5GB q Overall storage used: 5 TB per simulation

24. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago RTFlame Simulations qObjective: Study fundamental properties of Rayleigh-Taylor driven turbulent nuclear burning to verify assumptions in GCD model q Whether sub-grid model is needed q Not if nuclear burning occurs primarily at large scales q Physical conditions under which flame transitions to distributed burning qCode Components q Hydro with shock capturing scheme (PPM) q Constant gravity q Nuclear flame model q Lagrangian tracer particles q AMR Adaptive Mesh q Parallel I/O

25. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago RTFlame: Specifications qSimulate flame in an elongated box with a square base qSequence of resolutions q currently running 256 ^2 x 1024 and 512 ^2 x 2048 on BG/P qNode use varies from 512 to 4096 qRunning in dual or quad modes

26. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago RTFlame: I/O qCheckpoint sizes are 9 -- 70 GB in range qPlotfiles are 2 -- 16 GB in range qScaling of I/O is not very clear q Increasing the number of processors can sometimes cause the write time to increase substantially Observations : Read of checkpoints is extremely fast (2 min) Write was unstable without MPI barriers between different datasets Though there have been failures during output, files that get completed are fine There is a balance between scaling and I/O performance I/O takes away some memory footprint from the simulation

27. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago RTFlame Simulation : One Checkpoint 512^2 on 4/8K procs 4k8k 5121k qTime to write a single checkpoint qFirst plot ~72GB, second ~9GB qAccounts for little less than 10% of execution time 256^2 on 512/1K procs

28. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Questions?