Page 1 Trilinos Usage of CMake/CTest/CDash Roscoe A. Bartlett Department of Optimization & Uncertainty Estimation Sandia National Laboratories Kitware/SNL CMake/CTest/CDash Workshop May 7, 2009 Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy under contract DE-AC04-94AL P

Page 2 Primary Motivations for Switching Trilinos Native support for MS Windows (MS Visual Studio projects and self- extracting installers) and Mac (XCode project files and binary installers) Improve robustness of our development and testing environments –Improve the stability of important Trilinos code –Support stronger collaborations with important customer codes (i.e. Almost Continuous Integration) –Better support R&D of deep stacks of vertically integrated algorithms Decrease the maintenance costs of the framework and testing infrastructure –Reduce maintenance cost by 1/2 to 1/3! –Allow Jim Willenbring and Brent Perschbacher to do more important things Total Effort Time Autotools CMake CMake Transition

Page 3 Gains & (Initial) Looses Switching to CMake for Builds What we gain: –Full dependency tracking of every kind possible on all platforms (i.e. header to object, object to library, library to executable, and build system files to all built files) –Support for shared libraries on a variety of platforms –Support for MS Windows (i.e. Visual Studio projects, Windows installers, etc.) –Simplified build system and easier maintenance (extremely easy to add new packages and maintain existing packages) –Improved mechanism for extending capabilities (as compared to M4 in autotools) –Ability to affect the development of the build tools with good existing collaborations –Significant ``in house'' knowledge-base –One hundred percent automated intra-package dependency tracking and handling (built into the prototype Trilinos/CMake build system) What we lose: –CMake requires that all uses have 'cmake' installed on their machine when building from source and users will need to have at a very recent version of cmake. (However, cmake is very easy to build from source)

Page 4 Gains (no losses) Switching to CTest/CDash for Testing What we gain: –Test time-outs (this is a major maintenance issue for the current Perl-based test harness) –Memory testing with Valgrind and purify that is backed up by Kitware and a larger development community –Line coverage testing that is backed up by Kitware and a large development community –Support for selecting and excluding subsets of tests based on regular expressions or named keywords –Better integration with the build system (e.g. easier to support more advanced features like PBS batch systems and flexible testing control) –Better tracking of specific tests (i.e. each and every test can have a unique name that is easy to find) What we lose: –Nothing!

Page 5 Design Principles for Trilinos CMake Build System: #1 Make it exceedingly easy to define CMake files for new packages and to define libraries, tests, and examples in those packages. Create a design for building individual package CMake files that automatically results in uniformity of how things are done. –Use standard macros to define every package's main features –Simplify maintenance. Allow changes to logic and functionality that apply to all Trilinos packages without having to touch each individual Trilinos package's CMake files. Provide 100% automatic intra-package dependencies handling –Avoid mistakes –Avoid duplication –Robustifies a number of important features. Provide built-in automated support for as many critical software engineering practices a possible –Proper pre-checkin testing when continuous integration is being performed

Page 6 Design Principles for Trilinos CMake Build System: #2 Avoid duplication of all kinds as much as possible. –Fundamental software maintenance issue. The build system should be able to reproduce 100% update-to-date output by simply typing ‘make’. –E.g. Coping test input files to binary directory Aggregate as much common functionality as possible to the top-level CMake files but allow individual CMake packages to refine the logic if they really need to. Accept extra complexity at the global framework level to reduce it at the package level –Apply solid software engineering design principles to manage the complexity –Remove duplication

Page 7 Outline of CMake PackageArch Build System Trilinos/ CMakeLists.txt # Top-level build file cmake/ # Cmake support TrilinosPackages.cmake # List of Trilinos packages TrilinosTPLs.cmake # Lists of Trilinos TPLs... package_arch # Generic package architecture macros PackageArchGlobalMacros.cmake PackageMacros.cmake PackageLibraryMacros.cmake PackageAddExecutable.cmake... packages/... epetraext/ CMakeLists.txt # Top-level package build file cmake/ Dependencies.cmake # Defines intra-package dependency and TPL lists EpetraExt_config.h.in # Copied from../src/ and hand modified src/ CMakeLists.txt # Defines library sources and library(s) test/ CMakeLists.txt MatrixMatrix/ CMakeLists.txt # Define actual test executables and test runs...

Page 8 Adding Packages in TrilinosPackages.cmake Adding a new Trilinos Package is a 1-line addition at the Framework Level! NOTE: The packages must be listed in a order of strictly increasing dependences! NOTE: If you get the ordering wrong, the automated dependency handling CMake scripts will automatically detect this and issue a very good error messages before the build is performed! PS = Primary Stable Code, SS = Secondary Stable Code, EX = Experimental Code SET( Trilinos_PACKAGES_AND_DIRS_AND_CATEGORIES Teuchos teuchos PS RTOp rtop PS Epetra epetra PS Triutils triutils PS EpetraExt epetraext PS Thyra thyra PS RBGen rbgen SS ForTrilinos ForTrilinos EX PyTrilinos PyTrilinos SS... )

Page 9 Defining a Package’s CMake Build Files INCLUDE(PackageMacros) INCLUDE(AddOptionAndDefine) # # A) Define the package # PACKAGE(EpetraExt) # # B) Set up package-specific options # ADD_OPTION_AND_DEFINE(${PACKAGE_NAME}_BUILD_TRANSFORM HAVE_TRANSFORM "Enable transform functionality." ON )... # # C) Add the libraries, tests, and examples # ADD_SUBDIRECTORY(src) PACKAGE_ADD_TEST_DIRECTORIES(test) # # D) Do standard postprocessing # PACKAGE_POSTPROCESS() Trilinos/packges/ epetraext/CMakeLists.txt SET(LIB_REQUIRED_DEP_PACKAGES Epetra Teuchos) SET(LIB_OPTIONAL_DEP_PACKAGES Triutils) SET(TEST_REQUIRED_DEP_PACKAGES) SET(TEST_OPTIONAL_DEP_PACKAGES) SET(LIB_REQUIRED_DEP_TPLS) SET(LIB_OPTIONAL_DEP_TPLS) SET(TEST_REQUIRED_DEP_TPLS) SET(TEST_OPTIONAL_DEP_TPLS) epetraext/cmake/Depencencies.cmake Utility macros provide framework hooks into package functionality –Defines common behavior across all packages –Avoids duplication –Facilitates maintenance Intra-package dependencies ! –Used in all intra-package dependency handing –All header-file paths and link libraries and directories are handled automatically –These dependencies can not be wrong! (i.e. the libs and execs would not build and link otherwise) –Optional dependencies create variables _ENABLE_ and HAVE_ _ !

Page 10 Automatic Intra-Package Dependency Handling $./do-configure -D Trilinos_DUMP_PACKAGE_DEPENDENCIES:BOOL=ON Configuring Trilinos build directory Printing package dependencies Teuchos_FORWARD_LIB_REQUIRED_DEP_PACKAGES='RTOp;EpetraExt;Isorropia;Thyra;Galeri;Amesos;Ifpack;Belos;RBGen' -- Teuchos_FORWARD_LIB_OPTIONAL_DEP_PACKAGES='AztecOO;ML' -- Epetra_FORWARD_LIB_REQUIRED_DEP_PACKAGES='Triutils;EpetraExt;Isorropia;AztecOO;Galeri;Amesos;Ifpack;Belos' -- Epetra_FORWARD_LIB_OPTIONAL_DEP_PACKAGES='Thyra;ML;RBGen' -- Zoltan_FORWARD_LIB_REQUIRED_DEP_PACKAGES='Isorropia' EpetraExt_LIB_REQUIRED_DEP_PACKAGES='Epetra;Teuchos' -- EpetraExt_LIB_OPTIONAL_DEP_PACKAGES='Triutils' -- EpetraExt_FORWARD_LIB_OPTIONAL_DEP_PACKAGES='Isorropia;Thyra;Galeri;Amesos;ML' -- EpetraExt_FORWARD_TEST_OPTIONAL_DEP_PACKAGES='Belos' -- Isorropia_LIB_REQUIRED_DEP_PACKAGES='Teuchos;Epetra;Zoltan' -- Isorropia_LIB_OPTIONAL_DEP_PACKAGES='EpetraExt' -- Isorropia_FORWARD_LIB_OPTIONAL_DEP_PACKAGES='ML' -- Thyra_LIB_REQUIRED_DEP_PACKAGES='RTOp;Teuchos' -- Thyra_LIB_OPTIONAL_DEP_PACKAGES='EpetraExt;Epetra' -- Thyra_FORWARD_LIB_REQUIRED_DEP_PACKAGES='Stratimikos' Stratimikos_LIB_REQUIRED_DEP_PACKAGES='Thyra' -- Stratimikos_LIB_OPTIONAL_DEP_PACKAGES='Amesos;AztecOO;Belos;Ifpack;ML' -- Stratimikos_TEST_OPTIONAL_DEP_PACKAGES='Triutils'

Page 11 Adding a Library in PACKAGE/src/CMakeLists.txt INCLUDE(PackageLibraryMacros) # A) Package-specific configuration options PACKAGE_CONFIGURE_FILE(${PROJECT_NAME}_config.h) # B) Define the header and source files (and directories) INCLUDE_DIRECTORIES(${CMAKE_CURRENT_SOURCE_DIR}) SET(HEADERS EpetraExt_ConfigDefs.h... ) SET(SOURCES EpetraExt_ProductOperator.cpp... )... # C) Define the targets for package's library(s) PACKAGE_ADD_LIBRARY( epetraext HEADERS ${HEADERS} SOURCES ${SOURCES} ) Dependent package header directories, libraries, and library link directories handled automatically! Macros provide uniform behavior across all libraries across all packages! Test-only libraries are supported with ‘TESTONLY’ flag (see NOX for an example)

Page 12 Adding a Test in PACKAGE/test/CMakeLists.txt INCLUDE(PackageAddExecutableAndTest) INCLUDE(PackageCopyFilesToBinaryDir) PACKAGE_ADD_EXECUTABLE_AND_TEST( MatrixMatrix_test SOURCES cxx_main.cpp COMM serial mpi ) PACKAGE_COPY_FILES_TO_BINARY_DIR( EpetraExtMatrixMatrixCopyFiles DEST_FILES infileAB infileATBT infileAB2 infiles infileABT infileAB3 infileATB infileATB2 C.mtx C4x4.mtx C4x12x12x4.mtx C4x6.mtx C6x4.mtx C6x6.mtx mat6x4.mtx mat6x6.mtx mat12x4.mtx mat4x12.mtx mat4x4.mtx mat4x6.mtx Y.mtx YTC.mtx Y_transp.mtx roman roman2 romancase romancase2 cdt cdt_case cdt_d.mtx cdt_m.mtx cdt_tce.mtx SOURCE_DIR ${${PACKAGE_NAME}_SOURCE_DIR}/test/MatrixMatrix SOURCE_PREFIX "src_“ EXEDEPS MatrixMatrix_test ) All header paths, link libraries etc are handled automatically! Define executable and test in one shot! 100% correct dependency tracking! epetraext/test/MatrixMatrix/CMakeLists.txt # Compile against epetra_test_err.h in all tests? INCLUDE_DIRECTORIES(${CMAKE_CURRENT_SOURCE_DIR})... ADD_SUBDIRECTORY(MatrixMatrix) epetraext/test/CMakeLists.txt Add common header include dirs Just include the subdirectories

Page 13 Defining slightly more complex tests INCLUDE(PackageAddExecutableAndTest) PACKAGE_ADD_EXECUTABLE( test_linear_op_with_solve SOURCES test_linear_op_with_solve.cpp COMM serial mpi ) PACKAGE_ADD_TEST( test_linear_op_with_solve NAME test_linear_op_with_solve_n1_n2 ARGS "--n=1" "--n=2" NUM_MPI_PROCS 1 COMM serial mpi ) PACKAGE_ADD_TEST( test_linear_op_with_solve NAME test_linear_op_with_solve_n4 ARGS "--n=4" NUM_MPI_PROCS 1 COMM serial mpi XHOST s s ) Define test cases separately from executable if needed! thyra/test/operator_solve/CMakeLists.txt

Page 14 Defining More Sophisticated Tests Current Limits of built-in CTest tests: Runs a single executable (or script) with a set of arguments Pass/fail determined by return value (0), or regex fail or regex any pass on STDOUT Runs test in current directory What we would like: Run multiple commands Define pass/fail for each command in various ways (examine other output files, examine output from multiple commands, etc.) Run tests in subdirectories created for the test Solution: Write user-defined CMake function PACKAGE_ADD_ADVANCED_TEST(...) Write ‘cmake –P’ scripts

Page 15 Defining More Sophisticated Tests Define advanced test: PACKAGE_ADD_ADVANCED_TEST( TEST_0 (EXEC | CMND )... [TEST_1 [EXEC | CMND ]...]... [TEST_N [EXEC | CMND ]...] [OVERALL_WORKING_DIRECTORY ] [FAIL_FAST] [KEYWORDS...] [COMM [serial] [mpi]] [OVERALL_NUM_MPI_PROCS ] [HOST...] [XHOST...] [FINAL_PASS_REGULAR_EXPRESSION | FINAL_FAIL_REGULAR_EXPRESSION ] ) Where: TEST_ EXEC [NOEXEPREFIX] [NOEXESUFFIX] [ADD_DIR_TO_NAME] | CMND [ARGS... ] [MESSAGE " "] [WORKING_DIRECTORY ] [NUM_MPI_PROCS ] [OUTPUT_FILE ] [NO_ECHO_OUTPUT]] [PASS_ANY | PASS_REGULAR_EXPRESSION " " | PASS_REGULAR_EXPRESSION_ALL " " " "... " " | FAIL_REGULAR_EXPRESSION " " | STANDARD_PASS_OUTPUT ]

Page 16 Defining More Sophisticated Tests PACKAGE_ADD_ADVANCED_TEST( ValidateParameters_compareTestOutput COMM serial mpi OVERALL_NUM_MPI_PROCS 1 # Should be 1,4,9 TEST_0 EXEC ml_Validate ARGS input1.xml OUTPUT_FILE ValidateParameters_compareTestOutput.out TEST_1 CMND ${PYTHON_EXECUTABLE} ARGS ${PROJECT_SOURCE_DIR}/commonTools/test/utilities/compareTestOutput evaluateCriteria baseline1.txt ValidateParameters_compareTestOutput.out PASS_REGULAR_EXPRESSION "Test passed." FAIL_FAST )

Page 17 Defining More Sophisticated Tests XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Advanced Test: ML_ValidateParameters_compareTestOutput Running test commands: TEST_0, TEST_1 ================================================================================ TEST_0 Running: "/Users/bmpersc/nightly/Trilinos.base/SERIAL_DEBUG/BUILD/packages/ml/test/ValidateParams/ML_ml_Validate.ex e" "input1.xml" Writing output to file "/Users/bmpersc/nightly/Trilinos.base/SERIAL_DEBUG/BUILD/packages/ml/test/ValidateParams/ValidateParameter s_compareTestOutput.out" TEST_0: Return code = 0 TEST_0: Pass criteria = Return code TEST_0: Result = PASSED ================================================================================ TEST_1...

Page 18 Defining More Sophisticated Tests XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX Advanced Test: ML_ValidateParameters_compareTestOutput Running test commands: TEST_0, TEST_1 ================================================================================... ================================================================================ TEST_1 Running: "/usr/bin/python2.5" "/Users/bmpersc/nightly/Trilinos.base/SERIAL_DEBUG/Trilinos/commonTools/test/utilities/compareTestOutput" "evaluateCriteria" "baseline1.txt" "ValidateParameters_compareTestOutput.out" Test passed TEST_1: Return code = 0 TEST_1: Pass criteria = Match REGEX {Test passed.} TEST_1: Result = PASSED ================================================================================ OVERALL FINAL RESULT: TEST PASSED XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Page 19 Trilinos CMake Workshop Outline Automatic Intra-Package Dependency Handling

Page 20 Quickstart: Special Configuration Modes for Trilinos Configuring Trilinos to build all packages with all tests and examples: -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=ON -D Trilinos_ENABLE_TESTS:BOOL=ON Configuring a package(s) along with all of the packages it can use -D Trilinos_ENABLE_Stratimikos:BOOL=ON -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=ON -D Trilinos_ENABLE_TESTS:BOOL=ON Configuring Trilinos to disable a package(s) and all packages it depends on: -D Trilinos_ENABLE_Stratimikos:BOOL=ON -D Trilinos_ENABLE_Amesos:BOOL=OFF

Page 21 Automatic Intra-Package Dependency Handling Trilinos/cmake/python/data/TrilinosPackageDependenciesTable.html

Page 22 Package Dependency Structure for Thyra RTOpTeuchosEpetraTriutilsThyraEpetraExt Required Dependence Optional Dependence

Page 23 Example: Enabling a Package and All Optional Packages Configuring Trilinos build directory... Enabling all optional packages for current set of enabled packages Setting Trilinos_ENABLE_EpetraExt=ON because Trilinos_ENABLE_Thyra=ON -- Setting Trilinos_ENABLE_Epetra=ON because Trilinos_ENABLE_Thyra=ON -- Setting Trilinos_ENABLE_Triutils=ON because Trilinos_ENABLE_EpetraExt=ON Enabling all remaining required packages for the current set of enabled packages Setting Trilinos_ENABLE_RTOp=ON because Trilinos_ENABLE_Thyra=ON -- Setting Trilinos_ENABLE_Teuchos=ON because Trilinos_ENABLE_Thyra=ON Enabling all optional intra-package enables that can be if both sets of packages are enabled Setting EpetraExt_ENABLE_Triutils=ON since Trilinos_ENABLE_EpetraExt=ON AND Trilinos_ENABLE_Triutils=ON -- Setting Thyra_ENABLE_EpetraExt=ON since Trilinos_ENABLE_Thyra=ON AND Trilinos_ENABLE_EpetraExt=ON -- Setting Thyra_ENABLE_Epetra=ON since Trilinos_ENABLE_Thyra=ON AND Trilinos_ENABLE_Epetra=ON Final set of enabled packages: Teuchos RTOp Epetra Triutils EpetraExt Thyra 6 $./do-configure –D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \ -D Trilinos_ENABLE_Thyra:BOOL=ON \ -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES:BOOL=ON

Page 24 Dependency Handling for Pre-Checkin Testing $./do-configure \ -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \ -D Trilinos_ENABLE_Epetra:BOOL=ON \ -D Trilinos_ENABLE_ALL_FORWARD_DEP_PACAKGES:BOOL=ON \ -D Trilinos_ENABLE_TESTS:BOOL=ON RTOpTeuchosEpetraTriutilsThyraEpetraExtLib OnlyLibs & Tests

Page 25 Dependency Handling for Pre-Checkin Testing $./do-configure \ -D Trilinos_ENABLE_ALL_PACKAGES:BOOL=OFF \ -D Trilinos_ENABLE_RTOp:BOOL=ON \ -D Trilinos_ENABLE_ALL_FORWARD_DEP_PACAKGES:BOOL=ON \ -D Trilinos_ENABLE_TESTS:BOOL=ON RTOpTeuchosEpetraTriutilsThyraEpetraExtLib OnlyLibs & Tests

Page 26 Summary There is a lot of functionality in the current tools CMake/CTest/CDash CMake/CTest scripting gives great flexibility to extend functionality Various Sandia projects duplicate effort in build, testing, and reporting tools However, complex computational code projects require more functionality! LET’S TALK ABOUT WHAT ELSE IS NEEDED!