Maria Grazia Pia, INFN Genova Test & Analysis Project aka “statistical testing” Maria Grazia Pia, INFN Genova on behalf of the T&A team Geant4 Workshop, CERN, 2 October 2002

Maria Grazia Pia, INFN Genova What is the Test & Analysis project?  Test & Analysis  Test & Analysis is a project to develop a statistical analysis system for usage in Geant4 testing Provide tools to compare Geant4 simulation results with reference data – equivalent reference distributions (for instance, regression testing) – experimental measurements – data libraries from reference distribution sources – functions deriving from theoretical calculations or from fits physics validation regression testing system testing Main application areas: Interest in other areas, not only Geant4? LCG?

Maria Grazia Pia, INFN Genova History “Statistical testing” agreed as a collaboration-wide goal Initial ideas for this project presented at a TSB meeting ~end 2001 LowE + Anaphe expressed interest to work at the project, Nov 2001 Informal discussions with STT Coord. + Gabriele Informal discussions with e.m. physics coord. & developers Open brainstorming workshop 31 May 2002 Vision + User Requirements Document Open discussion and agreement on vision and requirements at this workshop

Maria Grazia Pia, INFN Genova Vision: the basics Rigorous software process –no cow-boy process please… Have a vision for the project –A general tool for physics & STT? –Only “statistical testing? in STT? –Other parties than Geant4 interested? Who are the stakeholders? Who are the users? Who are the developers? Build on a solid architecture –OOAD, not just hacking some code… Clearly define scope, objectives Flexible, extensible, maintainable system Software quality Clearly define roles

Maria Grazia Pia, INFN Genova The problem we want to solve (1) The problem of comparing Geant4 physics results with reference data or analytical functions affects the Geant4 (electromagnetic) physics Working Groups the impact of which is large amount of time spent in series of tedious, repetitive comparisons performed manually, or limited test performed, because of limited availability of resources A successful solution would be general tools for statistical comparisons being able to perform such testing by means of general tools for statistical comparisons

Maria Grazia Pia, INFN Genova The problem we want to solve (2) The problem of regression testing affects the STT and the Geant4 (electromagnetic) physics coordinators the impact of which is a significant workload to perform it manually, or a limitation of the test to very simple comparisons, such as the ASCII dump of the information of each event A successful solution would be the availability of tools for automated statistical comparisons

Maria Grazia Pia, INFN Genova The scope of the project tools for statistical testing The Test & Analysis project will provide tools for statistical testing of Geant4 physics and related quantities unit and system level –both at unit and system level physics validation regression testing It will address primarily the areas of physics validation and regression testing –by means of comparison between simulation results and reference distributions Generality Generality (for application also in other areas) should be pursued

Maria Grazia Pia, INFN Genova Guidelines architectural software process The Test&Analysis tool adopts a solid architectural approach and a rigorous software process: functionalityquality –to offer the functionality and the quality needed by the users maintainable –to be maintainable over a large time scale extensible –to be extensible, to accommodate future evolutions of the requirements Alternatives: hacking code for ad-hoc solutions –Pro: some specific problems solved quickly –Contra: difficult to cover all requirements, not easily maintainable or extensible a project limited to statistical regression testing for the STT –Pro: some useful results in a critical area –Contra: –not encompassing the testing requirements specific to physics Working Groups –by neglecting the commonality of some problems, it would not optimise the exploitation of the resources

Maria Grazia Pia, INFN Genova Who are the stakeholders? NameDescriptionResponsibilities Geant4 STT Coordinator Coordinates system testing Ensures that the system meets the needs of Geant4 System Testing Geant4 physics coordinators Coordinate Geant4 WGs Ensure that the system meets the needs of Geant4 Physics WGs Geant4 TSB Is responsible for Geant4 technical matters Reviews the project, monitors progress INFN Computing Committee National Committee whom part of the project developers responds to; has 3 referees assigned to Geant4 Recommends funding; reviews the project and monitors progress Student Technical student at CERN and student at the Univ. of Genova Is the main developer, should do work of value for her university degrees

Maria Grazia Pia, INFN Genova Who are the users? Other potential users: users of the Geant4 Toolkit, wishing to verify the results of their applications with respect to reference data or their own experimental results other projects with similar requirements, or using the Geant4 Toolkit (e.g. the LHC Computing Grid project) NameResponsibilities Physics developers Provide and document requirements, provide feedback on prototypes, perform beta testing on preliminary releases of the product, provide use cases for acceptance testing STT Provide and document requirements, perform formal acceptance testing for adoption in system testing

Maria Grazia Pia, INFN Genova What do they want? Need Prio rity ConcernsCurrent Solution Proposed Solutions Perform regression testing ?Degraded behaviour may slip into a Geant4 release Done manually on limited set of observables (physics), or automatically on reference ASCII dump Automated tool for regression testing Do comparison with reference data ? Either the individual physics packages, or Geant4 as a whole, may not be subject to verification against reference data as part of the release testing. Done manually, not all the test comparisons available are run regularly General tool for physics validation See the URD for specific requirements

Maria Grazia Pia, INFN Genova Are there any constraints? To operate in the same environment as the development of Geant4 Possibility to integrate it in system testing Based on AIDA No concrete dependencies on specific AIDA implementations should appear in the code of the system tests User-friendly interface, to make the system easy to use and customise even by non-expert users

Maria Grazia Pia, INFN Genova Prototype(s) Useful to address some risks –the viability of the product being developed –some technological concerns, like the usage of Python and the adequacy of AIDA implementation –the understanding of requirements Useful to developers to get familiar with tools, environments etc. Simple prototype for comparison between NIST reference data of electron range and Geant4 LowE/standard simulation Risk list at

Maria Grazia Pia, INFN Genova Prototype (S. Guatelli, A. Pfeiffer) Al The reference data are automatically downloaded from the NIST website The user can select the physical quantity to be tested (range, stopping power etc.), the type of material and the energies of the electron The user can visualise the reference data The user can load the simulation results from a file and visualise them The user can overlay simulated and reference data, with their respective errors, on the same plot The user obtains the difference (%) between simulated and reference data, for all the energies selected, and plots it together with the errors of the reference data

Maria Grazia Pia, INFN Genova Related sessions Tuesday –Benchmark & statistical tests: automation and infrastructure issues Tuesday –Statistical tests: review of vision and requirements Thursday –Benchmark and statistical tests: summary, work plan Goals: –Agree on vision for the project, agree on initial set of requirements –Agree on objectives, milestones, activities, responsibilities, schedule