1. An Advanced Simulation & Computing (ASC) Academic Strategic Alliances Program (ASAP) Center at The University of Chicago The Center for Astrophysical Thermonuclear Flashes Transitioning from FLASH2 to FLASH3 Lynn Reid May 23, 2007

2. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH3 Goals qConfigurable q Flexibility to easily configure the code for new applications q A traditional strong point of FLASH qExtensible q Users can add functionality quickly q API allows easy use of any unit qVerifiable q Users should be able to test their applications easily q Code units should be tested individually q The FLASH test suite was release as a stand alone application in FLASH3 qWell Documented q A necessity for any community code that hopes to gain broad acceptance q Will ensure ‘survivability’ of FLASH3 regardless of Center funding

3. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH3 Enhancements: Interchangeable Grids Paramesh Uniform Grid  One block per processor  No AMR related overhead Patch Based Mesh (planned)  Adaptive mesh with variable block size  Block Structured  Fixed sized blocks  Specified at compile time  Not more than one level jump at fine course boundaries Make sure to allow different grids when programming! #ifndef FIXEDBLOCKSIZE allocate(variable,MAXBLOCKS) #endif

4. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH3 Enhancements: API Interface qEach unit has a well defined API q In Computer Science terms, each unit has a set of public routines which other units can call  Use FORTRAN interfaces to help force correct use of API use Grid_interface, ONLY: Grid_getBlkBoundBox qExamples for Grid unit: q Grid_get/putBlkData q Grid_getBlkBoundBox q Grid_updateRefinement q Grid_moveParticles q Grid_getNumProcs qEach routine should be fully documented with argument and algorithm descriptions!!

5. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Robodocs API Grid_getBlkBoundBox

6. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH3 Enhancements: Decentralized Data qIn FLASH2 a centralized database was the key way to access all data q Positives: easy for developers to get data they needed quickly and easily q Negatives: not clear who owned the data, when it was last modified or if it was safe to modify data qIn FLASH3 each unit contains a FORTRAN module to hold data specific to that unit q Still have accessor functions so developers can get data easily q Now more clear who owns the data q Much of the data stored in the FLASH2 database is located in the Grid_data module in FLASH3 or in the Flash.h file

7. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Example Flash.h File #define DENS_VAR 1 #define EINT_VAR 2 #define ENER_VAR 3 #define GAMC_VAR 4 #define GAME_VAR 5 #define PRES_VAR 6 #define TEMP_VAR 7 #define VELX_VAR 8 #define VELY_VAR 9 #define VELZ_VAR 10 #define MFRAC_SPEC 11 #define NPROP_VARS 10 #define NSPECIES 1 #define NMASS_SCALARS 0 #define NUNK_VARS (NPROP_VARS + NSPECIES + NMASS_SCALARS) #define E_FLUX 1 #define EINT_FLUX 2 #define P_FLUX 3 #define RHO_FLUX 4 #define U_FLUX 5 #define UT_FLUX 6 #define UTT_FLUX 7 #define NPROP_FLUX 7 #define NSPECIES_FLUX 1 #define NMASS_SCALARS_FLUX 0 #define NFLUXES (NPROP_FLUX + NSPECIES_FLUX + NMASS_SCALARS_FLUX) #define NDIM 2 #define MAXBLOCKS 1000 #define TEMP_SCRATCH_GRID_VAR 1 #define NSCRATCH_GRID_VARS 1 qThe Flash.h file is written by the setup script for your specific application qDeclares variable and flux indices in multidimensional datastructures qDefines number of dimensions, maximum blocks on a proc and much more... variable indices flux indices

8. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Example Data on Grid FLASH2 FLASH3 Get index into database idens = dbaseKeyNumber(‘dens’) call dbasePutData(idens,....) use index directly call Grid_putBlkData(DENS_VAR,....)

9. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Moving a FLASH2 module to FLASH3 qCreate a ‘UnitName’_data FORTRAN module to store data needed from other units in the code. q Create storage in ‘UnitName’_data for all necessary variables storing runtime parameters or physical constants. q Use ‘save’ attribute on runtime parameters qInitialize data in UnitName_init. q This routine eliminates Flash2’s “first call” constructs q Do not call RuntimeParameters_get from ANYWHERE else qFinalize data in UnitName_finalize. qMake ‘UnitName_interface’ to describe the public routines API. qMost kernels are the same as in FLASH2 and may be arbitrarily messy qWrite good documentation in Robodoc format  Document runtime parameters in the Config file qWhen moving code, see the name changes chart and descriptions on the web. qSee “example of a FLASH3 Unit” on web

10. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago FLASH3 Enhancements: Unit Testing and the Test Suite qUsers can and should easily add in new Unit Tests qAttempt to test units independently – verification! qEssential for immediately identifying bugs unintentionally introduced into the code –runs daily qNew test suite was released with FLASH3 so users can monitor their own research problems and performance q test parameters easily modified through GUI q handles unit tests, compilation tests, comparison tests Green light indicates all runs were successful Floating statistics box gives immediate overview of results Platform Date of run Red light indicates 1 or more tests failed

11. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago Common Mistakes in FLASH3 qRetain the order of calls in Driver_evolveFlash and Driver_initFlash qUse the FLASH APIs in general! q Don’t use Paramesh datastructures directly q Don’t call private functions of other units q Don’t use data variables from other modules qUse the Grid and Geometry APIs! q Don’t use NXB directly; use Grid_getBlkIndexLimits instead q Don’t calculate your own grid coordinates; use Grid_getCellsCoords or Grid_getBlkCenterCoords instead qUse the FLASH define files! q Use Flash_mpi.h and FLASH_REAL datatype rather than mpif.h and MPI_DOUBLE_PRECISION or MPI_REAL qDon’t duplicate runtime parameters indiscriminately q OK to redefine in Simulation Unit q Pay attention to setup script warnings! qUse Fortran90 features to help q Use interfaces q Use coding violations script (run every night)

12. The ASC/Alliances Center for Astrophysical Thermonuclear Flashes The University of Chicago … which brings us to Discussion and Questions