1. Thyra For Developers Roscoe A. Bartlett Department of Optimization & Uncertainty Estimation Sandia National Laboratories Trilinos Users Group Meeting (Developers Day), November 9 th, 2006 Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy under contract DE-AC04-94AL85000.

2. Outline What’s New with Thyra Cross-Package issues ???

3. Outline What’s New with Thyra Cross-Package issues ???

4. What’s New with Thyra (for Developers) Complete implicit linear operators Handle classes Operator/Vector adapters for Tpetra Linear solver adapter for Belos Stratimikos wrapper for linear solvers and preconditioners

5. Outline What’s New with Thyra Cross-Package issues ???

6. Cross Package Issues related to Thyra Structured output Teuchos::Describable Teuchos::VerboseObject Teuchos::FancyOStream Reporting timings Teuchos::Time[Monitor] Accepting parameters (hieratically) Teuchos::ParameterList  Thyra interfaces and implementation depend on all of these!  These require inner package support for uniformity!

7. Structured Outputting : Teuchos::VerboseObject Layered solvers will generate lots of output Need to indent output for each solver layer (Teuchos::FancyOStream and OSTab) Line prefixes with Teuchos::FancyOStream may help Mostly handled as the Thyra Level! Issues for non-Thyra code: Always print to an std::ostream (or a Teuchos::FancyOStream), never std::cout or std::cerr directly! Using Teuchos::VerboseObjectBase::getDefaultOStream() is aways better than using std::cerr or std::cout! Don’t use if(procID ==0) to determine when to create output, let the std::ostream decide (i.e. use Teuchos::blackholestream or Teuchos::FancyOSteam to control process output) Want the option to print from specific processes for debugging etc … Don’t print your main ouput to a file!

8. Reporting of Timings : Teuchos::Time[Monitor] Teuchos::TimeMonitor Can be used to time almost any type of operation Reports timing summary across processes Naming of timers should be namespace! (with the package name) e.g. “Belos::Solve …” instead of just “Solve …”” Enhancements: See Bug 2107

9. The Significance of Parameter Lists and Outputting In the future, most users will only interact with Trilinos solvers through setting parameters and looking at output! … … … … … Time Stepper Nonlinear Solver Lin. Solver Precond. Linear Solver Example: Time Stepper => Nonlinear Solver => Linear Solver => Preconditioner

10. User Requirements for use of Parameter Lists Allow specification of as many parameters as possible from an input file i.e. XML format or some other format Get back default values of parameters after a run for parameters not set by user e.g. NOX returns the default values of parameters Misspelling a parameter or sublist should result in a helpful error message and will not be silently ignored Make it easy change choices and sub-parameters back and forth easily Make it easy to find documentation and for the accepted parameter values

11. Developer Requirements for use of Parameter Lists Make it easy to extract parameters and their values Make it easy to validate parameters, sublists, and parameter values Allow flexibility in how validation is done Make it easy to avoid mistakes

12. Outline Developer Interaction with the Parameter Lists

13. Standardized Approach for Object Handling of Parameter Lists Teuchos::ParameterListAcceptor Parameter list objects (not just parameter values) are “set” and remembered by the object! Parameters may only be read later during a solve Parameter list objects are non-const and can be augmented with default option values, output lists … The “valid parameters” parameter list can be used for automatic documentation and for external validation is some cases … Many Thyra interfaces inherit from Teuchos::ParameteListAcceptor: e.g. Teuchos::LinearOpWithSolveFactoryBase, Teuchos::PreconditionerFactoryBase Namespace Teuchos { // Base class mix-in interface for objects that accept parameter lists class ParameterListAcceptor { public: // Set parameters from a parameter list and return with default values virtual void setParameterList( RCP paramList ) = 0; // Get the parameter list that was set using setParameterList() virtual RCP getParameterList() = 0; // Unset the parameter list that was set using setParameterList() virtual RCP unsetParameterList ()=0 // Return a const parameter list of all of the valid parameters virtual RCP getValidParameters() const;... }; }

14. Parameter List Validation A valid parameter list is compared against *this parameter list Parameters and sublists can be validated to any level desired => Gives control over how validation is done namespace Teuchos { class ParameterList { public:... void validateParameters( ParameterList const& validParamList,int const depth = 1000,EValidateUsed const validateUsed = VALIDATE_USED_ENABLED,EValidateDefaults const validateDefaults = VALIDATE_DEFAULTS_ENABLED ) const;... }; }

15. Recent Additions to the Teuchos::ParameterList class Parameter Entry documentation: Optional (multi-line) documentation string Parameter Entry Validators: Used in PL::validateParameters(…) the type and/or value of a parameter! namespace Teuchos { class ParameterList { public:... template void set( string const& name,T const& value,std::string const& docString = "",RCP const& validator = null );... }; } namespace Teuchos { class ParameterEntryValidator { public: // Print documentation for this parameter. virtual void printDoc( std::string const& docString,std::ostream & out ) const = 0; // Return an array of strings of valid values if applicable. virtual Teuchos::RefCountPtr > validStringValues() const = 0; // Validate a parameter entry value and throw exception if validation fails. virtual void validate( ParameterEntry const& entry,std::string const& paramName,std::string const& sublistName ) const = 0; }; }

16. Constructing a “Valid” Parameter List // Setup static objects that define parameters const std::string AztecSolver_name = "Aztec Solver"; const Teuchos::RefCountPtr > aztecSolverValidator = Teuchos::rcp( new Teuchos::StringToIntegralParameterEntryValidator ( Teuchos::tuple ("CG","GMRES","CGS","TFQMR","BiCGStab","LU"),Teuchos::tuple (AZ_cg,AZ_gmres,AZ_cgs,AZ_tfqmr,AZ_bicgstab,AZ_lu),AztecSolver_name ) ); const std::string AztecSolver_default = "GMRES";... Teuchos::RefCountPtr validAztecOOParams;... // Create valid parameter list Teuchos::RefCountPtr getValidAztecOOParameters() {... RefCountPtr pl = validAztecOOParams; if(pl.get()) return pl; pl = validAztecOOParams = rcp(new ParameterList()); // pl->set( AztecSolver_name,AztecSolver_default,"Type of linear solver algorithm to use.",aztecSolverValidator );.. // return pl; } See: Trilinos/packages/aztecoo/thyra/src/AztecOOParameterList.cpp

17. Reading from a User’s Parameter List // Setup static objects that define parameters const std::string AztecSolver_name = "Aztec Solver"; const Teuchos::RefCountPtr > aztecSolverValidator = Teuchos::rcp( new Teuchos::StringToIntegralParameterEntryValidator ( Teuchos::tuple ("CG","GMRES","CGS","TFQMR","BiCGStab","LU"),Teuchos::tuple (AZ_cg,AZ_gmres,AZ_cgs,AZ_tfqmr,AZ_bicgstab,AZ_lu),AztecSolver_name ) ); const std::string AztecSolver_default = “GMRES";... Teuchos::RefCountPtr validAztecOOParams;... // Read the parameters (initial validation is performed by client!) void setAztecOOParameters( Teuchos::ParameterList *pl,AztecOO *solver ) {... // Aztec Solver solver->SetAztecOption( AZ_solver,aztecSolverValidator->getIntegralValue(*pl,AztecSolver_name,AztecSolver_default) );... #ifdef TEUCHOS_DEBUG // Check to make sure that I did not use the PL incorrectly! pl->validateParameters(*getValidAztecOOParameters()); #endif // TEUCHOS_DEBUG } See: Trilinos/packages/aztecoo/thyra/src/AztecOOParameterList.cpp

18. Outline User Interaction with the Parameter Lists

19. Parameter List and Sublist Structure : Stratimikos.................. Linear Solvers Preconditioners See Doxygen documentation for Thyra::DefaultRealLinearSolverBuilder! Sublists passed on to package code! Top level parameters Every parameter and sublist not in red is handled by Thyra code and is fully validated!

20. Automatically Generated Parameter List Documentation “Human readable” automatically generated documentation for Stratimikos Linear Solver Type : string = Amesos # Determines the type of linear solver that will be used. # The parameters for each solver type are specified in the sublist "Linear Solver Types" # Valid values: "Belos", "Amesos", "AztecOO" Preconditioner Type : string = ML # Determines the type of preconditioner that will be used. # This option is only meaningful for linear solvers that accept preconditioner factory objects! # The parameters for each preconditioner are specified in the sublist "Preconditioner Types" # Valid values: "None", "Ifpack", "ML" Linear Solver Types -> AztecOO -> Output Every RHS : bool = 0 # Determines if output is created for each individual RHS (true or 1) or if output # is just created for an entire set of RHSs (false or 0). Forward Solve -> Max Iterations : int = 400 # The maximum number of iterations the AztecOO solver is allowed to perform. Tolerance : double = 1e-06 # The tolerence used in the convergence check (see the convergence test # in the sublist "AztecOO Settings") AztecOO Settings -> Aztec Solver : string = GMRES # Type of linear solver algorithm to use. # Valid values: "CG", "GMRES", "CGS", "TFQMR", "BiCGStab", "LU" Convergence Test : string = r0 # The convergence test to use for terminating the iterative solver. # Valid values: "r0", "rhs", "Anorm", "no scaling", "sol"... Generated (hiearchially) by Thyra::DefaultRealLinearSolverBuilder::getValidParameters()->print(…) See simple_stratimikos_example.exe --only-print-options --print-readable-format

21. Output of Default Parameter Values Example input parameter (sub)list to Stratimikos (use simple_stratimikos_example.exe)... Output (augmented) parameter (sub)list => Ifpack is not showing its default parameters!

22. Why Parameter Validation is Important Silently ignoring misspelled parameter names is unacceptable Many users will have to wade through thousands of lines of out to find warnings Makes it difficult to write testing software that ensures that parameters are indeed recognized Almost every other kind of software will step and print error message e.g. compilers, scripting languages, … … A thousand lines of output … Unused parameters: Statimikos -> Linear Solver Types -> AztecOO -> AztecOO Settings -> Aztec Solver : string = GMRES [default] … tens of parameters … … ztec Solver : string = GMRES [unused] … A hundred lines of output … Example: User misspells “Aztec Solver” as “ztec Solver” Poor old user has to ask: Was this option spelled correctly but just not read? Was this option spelled incorrectly and therefore not used?

23. Error Messages for Improper Parameters/Sublists Example: User misspells “Aztec Solver” as “ztec Solver” Error message generated from PL::validateParameters(…) with exception: p=0: *** Caught standard exception of type 'N7Teuchos10Exceptions20InvalidParameterNameE' : /home/rabartl/PROJECTS/Trilinos.base/Trilinos/packages/teuchos/src/Teuchos_ParameterList.cpp:352: Throw test that evaluated to true: !validEntry Error, the parameter {name="ztec Solver",type="string",value="GMRES"} in the parameter (sub)list "DefaultRealLinearSolverBuilder->Linear Solver Types->AztecOO- >Forward Solve->AztecOO Settings" was not found in the list of valid parameters! The valid parameters and types are: { "Aztec Preconditioner" : string = ilu "Aztec Solver" : string = GMRES … }

24. Error Messages for Improper Parameters/Sublists Example: User specifies the wrong type for “Aztec Solver” Error message generated from PL::validateParameters(…) with exception: p=0: *** Caught standard exception of type 'N7Teuchos10Exceptions20InvalidParameterTypeE' : /home/rabartl/PROJECTS/Trilinos.base/Trilinos/packages/teuchos/src/Teuchos_StandardParameterEntry Validators.hpp:279: Throw test that evaluated to true: !validType Teuchos::StringToIntegralParameterEntryValidator::getIntegralValue(...): Error, the parameter {paramName="Aztec Solver",type="int"} in the sublist "DefaultRealLinearSolverBuilder->Linear Solver Types->AztecOO->Forward Solve- >AztecOO Settings" has the wrong type. The correct type is "string"!

25. Error Messages for Improper Parameters/Sublists Example: User specifies the wrong value for “Aztec Solver” Error message generated from PL::validateParameters(…) with exception: p=0: *** Caught standard exception of type 'N7Teuchos10Exceptions21InvalidParameterValueE' : /home/rabartl/PROJECTS/Trilinos.base/Trilinos/packages/teuchos/src/Teuchos_StandardParameterEntry Validators.hpp:260: Throw test that evaluated to true: itr == map_.end() Teuchos::StringToIntegralParameterEntryValidator::getIntegralValue("GMRESS",...): Error, the value "GMRESS" is not recognized for the parameter "Aztec Solver" in the sublist "". Valid selections include: "CG", "GMRES", "CGS", "TFQMR", "BiCGStab", "LU".

26. A few Comments on Parameter List Validation PL::validateParameters(…) is only a tool for validation, not a “catch-all’ solution How validation is performed is not important as long as it gets done! Validation is ultimately the responsibility of the objects that accept and read from a parameter lists Full validation may not occur until the underlying solver finishes We should not have to (dramatically) change current parameter list structures e.g. Ifpack, ML, Amesos,... 100% validation may not be possible in all cases

27. Parameter List Issues? Handling multiple value types for a parameter and validation? Accepting an string value or an enum value (e.g. Epetra_CombineMode) In code as: Add, Zero, Insert, InsertAdd, Average, or AbsMax In XML as: “Add”, “Zero”, “Insert”, “InsertAdd”, “Average”, or “AbsMax” Allowing for implicit type conversions (e.g. “level-of-fill”) Input as int, double, float …, Output as int, double, float Solution => Teuchos::ParameterEntryValidator subclasases! Handling and validation of non-“parameter” parameters? Status tests in NOX or null-space vectors in ML Adding an RCP-wrapped derived class object and getting back base class object? Solution => Add specialized handling of RCP-wrapped objects for NOX PL class …

28. Parameter List Issues? Preserving ordering of parameter and sublists? Issue for Teuchos::ParameterList Issue for Teuchos::XMLObject Adding documentation and valid parameter value lists to the XML representation Reading in sublists from another file (in the XML parsesr) Broadcasting a parameter list? Non-parameter parameters? … GUI for setting XML parameters? Issues Ken mentioned to me: Case sensitivity? Accept “symmetric” for “Symmetric”? Partial matching? Accept “Sym” for “Symmetric”? Error messages? Suggesting correct spellings of parameters? Suggest “Symmetric” when user inputs “symmetric”? Any other issues?

29. Parameter List Summary Consistency in the use and the look-and-feel of parameter lists is needed User (should) expect parameter lists to be validated User parameter lists and developer use of parameter lists must be validated! Many tools for validation PL::validateParameters( validParamList,... ) ParameterListValidator (StringToIntegralParameterListValiator, …)