ARCS Data Analysis Software An overview of the ARCS software management plan Michael Aivazis California Institute of Technology ARCS Baseline Review March 14, 2002

2 Overview Software engineering practices –Version control –Configuration management –Bug tracking –Regression testing Architectural overview –A success story –The key design elements –Design goals Compatibility –Programming languages –File formats –Data analysis and visualization environments

3 Software engineering practices Version control –Provides a record of the evolution of the software –CVS: well supported, open source Configuration management –Uniform, portable build procedure –Automatic, regular builds of the entire software base –config: a system based on make Regression testing –Test cases that Exercise expected behavior Exercise fixes for known bugs Bug tracking –Organize the “to do” list, the feature requests … and the known defects –Gnats: well supported, open source

4 Overview of the Caltech ASCI center Constructing a Virtual Testing Facility (VTF) Large, multi-disciplinary project –Five Caltech research groups and a dozen sub-contractors Application highlights: –Currently: 250 kloc in C, F77, F90 and C++ –Massively parallel: ~10K processors –Large data storage requirements: ~10Tb per run –Relies heavily on existing code, both solvers and infrastructure Python provides the glue –Extensibility is an essential feature Pyre: the application framework –Python scripts: 700 classes, 35,000 lines of code –C++: lines of bindings and infrastructure

5 The scope of Pyre Problem specification Solid modeling Boundary and initial conditions Materials and constitutive models Solvers Simulation driver Simulation instrumentation Integration of parallel/distributed programming infrastructure Real-time visualization Full simulation archiving

6 Flexibility through the use of scripting Scripting enables us to –Organize the large number of parameters –Allow the environment to discover new capabilities without the need for recompilation or relinking The python interpreter –The interpreter modern object oriented language robust, portable, mature, well supported, well documented easily extensible rapid application development –Support for parallel programming (if necessary…) trivial embedding of the interpreter in an MPI compliant manner a python interpreter on each compute node MPI is fully integrated: bindings + OO layer –Experience: no measurable impact on either performance or scalability

7 Integrated facilities The existence of the framework enables the introduction of new facilities without requiring any work on the existing ones Scripted facilities: –Infrastructure: MPI, ACIS, GrACE –Solvers: adlib, tetra; rm3d, rm3dge, arm3d –HDF5 –Sensors and probes In progress: –Existing material models –Material properties Near future –Grid support –Next generation of solvers

8 MATLAB MATLAB is fully accessible from within Python –The reverse is not true (…yet)

9 IRIS Explorer

10 Design goals Integrate analysis modules using scripting –Python Data flow paradigm –Well understood –Easy to implement and document XML used to construct a fully reproducible description of the data analysis pipeline –Tag and archive data –Record the version number of each module File formats: –Reuse, reuse, reuse –Augment, contribute –HDF5 NeXus + XML meta-data (?)

11 Conclusions The software engineering issues are well understood The architectural vision for the analysis software –Is based on an intuitive paradigm –Leverages existing work ARCS is in a good position to have a positive influence on the field –Enough lead time –Well funded –Multiple leveraging –Low risk Near term activities –Define policies –Implement and deploy software management tools –Take inventory of existing analysis and visualization tools –Construct a prototype of analysis