1. Current Progress on Developing a Subsurface Simulation Framework Using CCA Bruce Palmer, Yilin Fang, Glenn Hammond, Vidhya Gurumoorthi, Jim Fort Pacific Northwest National Laboratory Richland, WA

2. Smooth Particle Hydrodynamics Use a discrete sampling of points to approximate continuum hydrodynamic fields where W is a smooth, normalized weighting function with support h and ρ is

3. Smooth Particle Hydrodynamics The equations of motion for the particles are where P is the pressure and Π is a stress tensor that depends only on the properties of particles at locations i and j.

4. Smooth Particle Hydrodynamics Algorithmically similar to molecular dynamics Explicit time integration of particle trajectories Finite range, pairwise interactions between particles For large simulations, particles are distributed on processors based on spatial location All large data objects consist of 1-dimensional arrays of integers and double precision numbers

5. Spatial Distribution P0P0 P1P1 P2P2 P3P3 P4P4 P5P5 P6P6 P7P7 Spatial Decomposition Algorithm: Partition particles among processors Update coordinates at every step Update partitioning after fixed number of steps

6. Componentization Defining components: What functionality and data belong together in a component Granularity: At what level is componentization compatible with performance? Abstraction of Interfaces: Can interfaces be defined that support multiple implementations representing different models and/or algorithms? Ownership: Which components allocate memory for particular data elements and how is this communicated to other components? Who is responsible for keeping data current?

7. Old Framework for SPH Communication Forces Time Integration

8. Defects in Old SPH Framework Data is allocated in driver but data requirements are determined by force component Input is read in by driver and communicated downward to different components Peripheral components (e.g. output for visualization) are tied to specific data elements Overall architecture is hardwired to the presence of specific data vectors. Addition of new data elements requires modification to many or all components.

11. Increasing the Modularity of SPH Framework Create a “dictionary” of data elements. All data elements that could be used outside a single component are given a name and attributes that uniquely identify it Create a data manager that can be queried by any component to get a pointer to a data element using its name and attributes All components receive their “keyword” input directly. We use a read_input component to manage this, but it could be done using individual files.

12. Dictionary All data elements (integer and double vectors) are given a name and class Name is moderately unique handle that refers to role of data Class assigns it to different functional groups, e.g. coordinates or concentration and to whether data represents locally or remotely held particles

13. Elements in Dictionary Names: xcoord ycoord zcoord zvel yvel zvel. Classes: coord bcoord conc bconc concf bconcf

14. Data Manager package dtmgr version 0.0 { interface DataPort extends gov.cca.Port { void dtmgr_init(in int t_max_len); void num_dbl(out int t_num_vec); void get_dbl(in int t_index, out array t_vec, out string t_name, out string t_class); void num_int(out int t_num_vec); void get_int(in int t_index, out array t_vec, out string t_name, out string t_class); void dtmgr_terminate(); }

15. Force Component package force version 0.0 { interface ForcePort extends gov.cca.Port { void force_allct(in int t_max_len); void force_init(); void num_dbl(out int t_num_vec); void get_dbl(in int t_index, out array t_vec, out string t_name, out string t_class); void num_int(out int t_num_vec); void get_int(in int t_index, out array t_vec, out string t_name, out string t_class); void set_rparam(in double t_param, in string t_name); void set_iparam(in int t_param, in string t_name); void eval_forces(); void force_terminate(); }

16. package rdcomp version 0.0{ interface ReadInputPort extends gov.cca.Port { void rdfile(in string t_filename, out bool value); void ignore_caps(in bool t_flag); void set_token_pattern(in string t_pattern); void open_input(in string t_field); void close_input(); void rewind(); void next_line(out string t_line, out bool value); void tokenize(in string t_line, out array tokens, out int size, out bool value); void get_line(in int t_idx, out string t_line, out bool value); void find_line(in string t_token, out string t_line, out bool value); void is_int(in string t_token, out bool value); void is_real(in string t_token, out bool value); void to_int(in string t_token, out int value); void to_real(in string t_token, out float value); } Read Input Component

17. Actual Input input driver 1000000 maximum atoms per processor end input force cutoff 1.0 skin_depth 0.3 gravity 0.001 cspd 0.02 viscosity 10.0 end input stepper 100000, 0.0050 #steps, timestep 20,1 frequency to update configuration, print results end input remap grid 70 4 4 smooth_rad 0.5 print scalar type density print vector xvel yvel zvel end These names are in the dictionary!

18. Remap Component

19. STOMP STOMP is a subsurface simulation code designed to simulate multiphase, reactive flows in the subsurface environment Currently designed to use a regular orthogonal grid Parallelization achieved through use of a preprocessor and directives in the source code

20. STOMP Componentization Replace loops over IJK for regular lattice with loops over node (cell centers), connections (edges), and boundary connections Create grid component to set up grid and manage access to node, connection, and boundary connection fields Currently implementing grid component using ITAPS libraries for unstructured grids and GA for regular, orthogonal grids

21. Grid Component package grid version 0.0 { interface GridPort extends gov.cca.Port { void grid_init(); void grid_terminate(); void get_node_num(out int t_num); void get_local_node_num(out int t_num); void get_local_node_list(out array t_list); void get_node_mask(out array t_mask); void get_cnx_num(out int t_num); void get_bcnx_num(out int t_num); void create_node_dfield(in string t_string, out array t_field); void create_node_ifield(in string t_string, out array t_field); void get_node_dfield(in string t_string, out array t_field, out bool t_ok); void get_node_ifield(in string t_string, out array t_field, out bool t_ok); void list_node_fields(out array t_field_name, out array t_type, out int t_num);

22. Grid Component void create_cnx_dfield(in string t_string, out array t_field); void create_cnx_ifield(in string t_string, out array t_field); void get_cnx_dfield(in string t_string, out array t_field, out bool t_ok); void get_cnx_ifield(in string t_string, out array t_field, out bool t_ok); void list_cnx_fields(out array t_field_name, out array t_type, out int t_num); void create_bcnx_dfield(in string t_string, out array t_field); void create_bcnx_ifield(in string t_string, out array t_field); void get_bcnx_dfield(in string t_string, out array t_field, out bool t_ok); void get_bcnx_ifield(in string t_string, out array t_field, out bool t_ok); void list_bcnx_fields(out array t_field_name, out array t_type, out int t_num); void update_nodes(); }

23. Bocca Can’t figure out what the philosophy on packages is. Seems different for ports and components Might be nice if there was more directory structure for components e.g. directory for package, subdirectories for individual components

24. Opaque Data Types These seem impossible to use. I’d go with the flow on this and not use them but the ITAPS interface uses them extensively... The only way we can use some of this data is to make a copy of it which is not something HPC programmers like to do We seem to be being protected from ourselves to an extent that we can’t actually do anything

25. Bad Ideas I Still Like Declare ports universal in rc file. If port implementation exists, any component that needs that port is automatically hooked up to it. If more than 1 implementation exists, throw an error sidl_integer: data type that follows fortran integer size or at least is configurable. Maybe sidl_long already does this? sidl_void: For C and C++ code pass a void pointer through argument list. For Fortran codes, put variable in argument list and let developer declare type in user portion of code. Maybe if we could at least cast sidl_opaque data objects to something useful….