Wael Elwasif ORNL/IU Fusion Frameworks Workshop 1 Integrating Fusion Codes Using CCA – an ORNL LDRD Project Wael R. Elwasif & Lee Berry Computer Science & Fusion Energy Division Mathematics Division Oak Ridge National Laboratory

Wael ElwasifORNL/IU Fusion Frameworks Workshop2 Project Goals Gain experience in using CCA components for integrated plasma simulation. Explore code integration issues: Interface Development. Data Coupling. Have model sufficiently complex to understand weaknesses/strengths of CCA. But have it fit within a two year, ~1.5 FTE/year effort.

Wael ElwasifORNL/IU Fusion Frameworks Workshop3 Project Members - Fusion Lee Berry Overall coordination, driver development, integration, design. Wayne Houlberg NTCC-like routines for calculations metrics from eqdsk files, advancing ion densities, and advancing electron and ion energy. Fred Jaeger Implementations of AORSA and routines for brining in equilibrium, plasma profiles into the form needed for AORSA

Wael ElwasifORNL/IU Fusion Frameworks Workshop4 Project Members – Computer Science & Mathematics Wael Elwasif: Build system, CVS, implementation strategies assistance, component infrastructure. James Kohl: State component, Visualization development. David Bernhardt: CS coordination. Ed D’Azevedo: Development of iterative AORSA algorithm.

Wael ElwasifORNL/IU Fusion Frameworks Workshop5 Prototype Components Physics Equilibrium – based on EFIT (F95). RF Heating – based on AORSA (F95). Transport (F95). Driver Overall time stepping and coordination (Python). State Repository (C++). Data couplers and transforms (F95).

Wael ElwasifORNL/IU Fusion Frameworks Workshop6 The Physics Solving a model problem (nothing new vis-à-vis fusion). Couple together codes for: Equilibrium – using EQDSK files. Transport – code from Wayne Houlberg. RF - AORSA2D (All Orders Spectral Algorithm) – code from Fred Jaeger.

Wael ElwasifORNL/IU Fusion Frameworks Workshop7 Code Status Pre-integration Stand alone codes. File-based control and input/output. Code-specific pre-processing (e.g. Coordinate translation and re-griding). Implemented in F95, with some language-specific features (e.g. optional arguments).

Wael ElwasifORNL/IU Fusion Frameworks Workshop8 Implementation Strategy Getting everybody to know what CCA is, and how to use available tools (especially Babel/SIDL). Initial interface specification (keeping it intentionally high level). Getting a working component application (that doesn’t do much). Repeat: Refine interfaces and update implementation. Add new capabilities (e.g. remove dependency on input files).

Wael ElwasifORNL/IU Fusion Frameworks Workshop9 Prototype Components: Physics Driver Physics Data Transform State

Wael ElwasifORNL/IU Fusion Frameworks Workshop10 The Physics: Transport interface Xport extends gov.cca.Port { int init( in string DATA_PATH, out string init_xp_message, out int i_flag, … … … ); /// Advance density equation int pushN( in double time, in double dt, in array k_nbc_i, … … … ); /// Advance temperature equation int pushT( in double time, in double dt, in double tebc, … … … ); } DATA_PATH set by driver. Interface evolved over project duration. Implementation wrapped existing F95 module (with one-to-one correspondence). Mapping between SIDL arrays and native F95 arrays was the most “challenging” aspect.

Wael ElwasifORNL/IU Fusion Frameworks Workshop11 The Physics: RF & Equilibrium interface RF extends gov.cca.Port { /// Get RF heating parameters int getRF( in string DATA_PATH, in double time, in double power, out string rf_message, in int k_grid ); } DATA_PATH set by driver. RF: Implementation wrapped re-factored AORSA2d core). Equilibrium: Uses EQDSK file. Updates the equilibrium part of the state interface Eql extends gov.cca.Port { /// Get equilibrium parameters int getEql( in string DATA_PATH, in string file_name, in double eq_time, out string eq_message ); }

Wael ElwasifORNL/IU Fusion Frameworks Workshop12 Prototype Components: State Driver Physics Data Transform State

Wael ElwasifORNL/IU Fusion Frameworks Workshop13 The PlasmaState Component interface EQState extends gov.cca.Port{ int initEQ( in int nx_xy, in int ny_xy, … … … ); } // End interface EQState interface RFState extends gov.cca.Port { int initRF( in int n_rgrid, in int n_zgrid, … … … ); } interface XPState extends gov.cca.Port { int initXport( in int n_r, in int n_I ); } One component implements state repository for Transport, Equilibrium, and RF. In-memory data repository. Allow centralization of data archiving, checkpointing, and restart. No data transform within the state component. Data exchange between physics components happens through the state, or the driver (no direct coupling).

Wael ElwasifORNL/IU Fusion Frameworks Workshop14 Prototype Components: Data Couplers Driver Physics Data Transform State

Wael ElwasifORNL/IU Fusion Frameworks Workshop15 The Need for Data Couplers Data consumers adapt “raw” data to their specific requirements (coordinate transforms, and re-gridding). Separate this preprocessing stage into stand alone components. NOT VERY SCALABLE. Future goal: Evolve towards generalized (centralized) data transforms with controllable mathematical properties.

Wael ElwasifORNL/IU Fusion Frameworks Workshop16 Prototype Components: Driver Driver Physics Data Transform State

Wael ElwasifORNL/IU Fusion Frameworks Workshop17 The Driver Component The “brains” of the overall simulation. RAD using Python enables speedy experimentation, parameter tuning, and quick on-line visualization. Can be quite complex. Using Python allows easy access to high level matrix operations via the numeric python module – whenever it makes sense.

Wael ElwasifORNL/IU Fusion Frameworks Workshop18 Coupled Simulation: It actually Works!!!

Wael ElwasifORNL/IU Fusion Frameworks Workshop19 Implementation Issues: THE BUILD Adopted from CCA template used in CCA tutorial. Was adequate for the scale and scope of this project Scaling to accommodate more components with varying “native” build systems is still an issue. Needed tweaking when deployed to different Linux platforms (compilers, location of “legacy” libraries). Legacy libraries build system changed as well.

Wael ElwasifORNL/IU Fusion Frameworks Workshop20 Implementation Issues: CVS Initially used only for the components. Extended to include a snapshot of the legacy codes. Legacy codes re-factored (change into libraries, streamline file access). Question: How to manage continuous parallel development of legacy codes and integration?

Wael ElwasifORNL/IU Fusion Frameworks Workshop21 Lessons Learned “Throwing over the fence” does NOT work. Interface development was “easy”: We control everything. The build system is VERY important. Language interoperability using Babel makes things possible, but not necessarily easy. Incorporating independently developed F95 codes into a single application has its own pitfalls (e.g. compiler flags for code correctness). Integration exercises codes in unforeseen ways (bugs found in established codes).

Wael ElwasifORNL/IU Fusion Frameworks Workshop22 Future Plans Next four months. Implement iterative RF module with time varying densities. Use HDF5 as back-end for the state component. Run on the Altix at ORNL.