1. The Atlas Software Distribution Christian ArnaultAlessandro De SalvoSimon GeorgeGrigori Rybkine arnault@lal.in2p3.frAlessandro.DeSalvo@roma1.infn.itarnault@lal.in2p3.frAlessandro.DeSalvo@roma1.infn.it S.George@rhul.ac.ukgrigori.rybkine@rhul.ac.ukS.George@rhul.ac.ukgrigori.rybkine@rhul.ac.uk CVS CMT Work area Nightly build Pacman cache Integration tests Grid CE Laptop cmt co cvs ci cvs tag Kit construction pacman -get Tag collector Prod release Kit validation DC job Work area cmt co Packages are first created in a work area, registered to the TagCollector and imported to the CVS repository Packages are checked out to work area(s) Tags are declared to the TagCollector, registered for the next release Nightly builds take all registered tags for the next release Quality checks and Integration tests results are reported to developers (through Web pages) Production releases take validated tags Distribution kit is automatically constructed from production releases Release kits are installed on Grid CE and laptops KitValidation jobs are run after installation Users may develop packages against installed kits Register package register tag The software flow Individual packages are tagged randomly Tags can be submitted to the Tag Collector for the next release until it is locked Every day the last registered tags are selected for the nightly build Every three weeks the tag collection is locked during the release build (between the last nightly and the end of the release validation) Tags for core packages are locked one week before the release build Twice a year a major release is produced (global tag M.0.0) Patch releases may be created after any minor or major release (global tag M.m.p) 8.2.0 days weeks Any scale Development:tagging Tag collector Tag registering Nightly build Tag selection Core tags lock All tags lock Minor release cycle months Major release cycle A-00-01-07 A-01-00-03 8.1.0 8.3.0 8.0.0 9.0.0 10.0.0 The software life cycle Atlas packages Gaudi packages LCGCMT interfaces CMT packages Software Interfaces Policy Containers Atlas External software Generators Local installations of other software Geant4 Java suites LCG software SEAL POOL PI … HEP or Free software Root MySQL … Atlas generic data files Managed by CMT Three sub-projects Atlas, Gaudi and LCGCMT Each sub-project has its own global version (the release tag) Atlas selects the release of Gaudi Gaudi selects the release of LCGCMT Each release of each sub-project selects the set of versions of individual packages Both Atlas and Gaudi define their specific centralized installation area Binaries Include files jobOptions Scripts (shell, python) Java packages The software structure Container packages: Purely managerial packages Not involved when using the software Only provide a CMT requirements file with a set of explicit use statements (no wild card) Purpose: Specifies the set of exact version for a release of the software group Drive the cmt broadcast commands over the complete software group The top container is AtlasRelease and describes the complete Atlas software release Every new release of the Atlas software yields a new release of the AtlasRelease container. Policy packages: Provide policy statements for a software domain Include dir strategy CMT strategies CMT patterns Generic symbols (macros, sets, …) AtlasPolicy is the main policy package, mandatory for all Atlas packages. Examples: ExternalPolicy, AtlasCxxPolicy, AtlasFortranPolicy Interface packages: Provide an interface between a software project managed by CMT and a so-called external product (not managed by CMT) Specify the native version tag defined by the product (as opposed with the CMT version tag) Macro _native_version Adapt the policies and conventions to the project’s ones Eg Include directory Construct the compiler and linker options for this product Set the runtime options (PATH, LD_LIBRARY_PATH, etc…) Specify the material to export when constructing a distribution kit Examples MySQL ROOT SEAL Geant4 Java kits The LCGCMT project is entirely made of such interface packages Connect to LCG internal (SEAL, POOL, PI, …) or external products Package semantics http://atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/The atlas software pagehttp://atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/ http://isnpx0168.in2p3.fr/athena/The tag collectorhttp://isnpx0168.in2p3.fr/athena/ http://www.cmtsite.orgCMThttp://www.cmtsite.org Links Run quality checks Run unit and integration tests Generate documentation Operate on fast local disks Then transferred to AFS Alex Undrus NICOS: the build driver Jérôme Fulachier – Johann Collot The Tag Collector Control the consistency of package versions assigned to any release Can be locked to prepare the release Support branch release management Bug fixes Late patches Directly connected to CVS Perform cross checks with CMT settings Handle the container hierarchy

2. The Atlas Software Distribution Christian ArnaultAlessandro De SalvoSimon GeorgeGrigori Rybkine arnault@lal.in2p3.frAlessandro.DeSalvo@roma1.infn.itarnault@lal.in2p3.frAlessandro.DeSalvo@roma1.infn.it S.George@rhul.ac.ukgrigori.rybkine@rhul.ac.ukS.George@rhul.ac.ukgrigori.rybkine@rhul.ac.uk CMT statementPacman statementLocation in the tar ball Usedepends applicationInstallArea/$(bin)/lib LibraryInstallArea/$(bin)/bin JarInstallArea/share/lib Special macros _runtimeInstallArea/share _joboptionsInstallArea/jobOptions/ _xmlsInstallArea/XML/ _scriptsInstallArea/share/bin include_dirsInstallArea/include/ Whole cmt directory / / /cmt/... All specifications to create the kit for one package are uniquely extracted from the requirements file of this package CMT Use statements produce Pacman depends statements The entire dependency graph is preserved Dependency cycles are filtered out Constituent definitions populate the tar ball from the binary installation area Conventional macros describe special files to bring to the tar ball (relocated to the installation area) Runtime files Gaudi jobOptions Scripts XML files Header files The cmt directory and its supporting directory tree is copied into the tar ball to permit configuration activity against the installed kit The generated tar ball is relocatable description='Package IdDictParser (v1)' source = '../kits/8.7.0' download = { '*':'IdDictParser-opt.tar.gz' } depends = [ 'dist-8.7.0-opt', '8.7.0/AtlasPolicy-opt', '8.7.0/XMLCoreParser-opt', '8.7.0/IdDict-opt', '8.7.0/Identifier-opt' ] kits/ / / -.tar.gz cache/ / / -.pacman cache/dist/8.7.0/IdDictParser-opt.pacman kits/dist/8.7.0/IdDictParser-opt.tar.gz description='Package IdDictParser (v1)' source = '../kits/8.7.0' download = { '*':'IdDictParser-opt.tar.gz' } depends = [ 'dist-8.7.0-opt', '8.7.0/AtlasPolicy-opt', '8.7.0/XMLCoreParser-opt', '8.7.0/IdDict-opt', '8.7.0/Identifier-opt' ] cache Manifest files describe packages Version Link to the contents tar ball Describe dependencies Specific installation procedure Contribution to the global post-install setup procedure Specifies and checks the site configuration A cache is a web-visible disk space Packages may be found in different caches Caches may be virtual or mirrored (snapshots) Installation transitively and recursively fetches packages from caches Apply the installation procedures Generate setup scripts Saul Youssef youssef@bu.edu http://physics.bu.edu/pacman/ cache Pacman –get ATLAS:8.7.0/IdDictParser-opt Saul Youssef Pacman From the requirements to the pacman kit package Expat use LCG_Settings v* macro Expat_home "$(LCG_external)/expat/$(Expat_native_version)/$(LCG_system)" macro Expat_native_version "1.95.5" macro Expat_export_paths "$(Expat_home)/lib/libexpat.so $(Expat_home)/include/expat.h“ include_path none include_dirs $(Expat_home)/include macro Expat_linkopts "-L$(Expat_home)/Libs -lexpat" \ WIN32 "/LIBPATH:$(Expat_home)/Libs libexpat.lib" path_remove LD_LIBRARY_PATH "/expat/" WIN32 "" path_prepend LD_LIBRARY_PATH "$(Expat_home)/Libs" WIN32 "" path_remove PATH "" WIN32 "\expat\" path_prepend PATH "" WIN32 "$(Expat_home)/Libs" An external package is described in a CMT Interface package is characterized by two special macros _native_version _export_paths description='Package Expat v1 opt in project=LCGCMT/LCGCMT_26_2c (v1)' url = 'http://atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/Development/' source = '../kits/LCGCMT_26_2c' download = { '*':'Expat-opt.tar.gz' } depends = [ 'LCGCMT-LCGCMT_26_2c-opt', 'LCGCMT_26_2c/LCG_Settings-opt', 'Expat- 1.95.5-opt' ] install = { '*': ['. $PACMAN_INSTALLATION/CMT/*/mgr/setup.sh; cd $PACMAN_INSTALLATION/LCGCMT/LCGCMT_26_2c/LCG_Interfaces/Expat/cmt; cmt -quiet -no_cleanup config; a=1'] } The tar ball only contains the CMT directory kits/ / -.tar.gz kits/LCGCMT_26_2c/Expat-opt.tar.gz Cache/ / -.pacman Cache/LCGCMT_26_2c/Expat-opt.pacman The tar ball contains the real package contents kits/ - -.tar.gz kits/Expat-1.95.5-opt.tar.gz description='External Package Expat v1 opt (v2)' url = 'http://atlas.web.cern.ch/Atlas/GROUPS/SOFTWARE/OO/Development/' source = '../kits' download = { '*':'Expat-1.95.5-opt.tar.gz' } Cache/ - -.pacman Cache/Expat-1.95.5-opt.pacman The pacman kit for the physical contents Independent of CMT Fully standalone kit The pacman kit for the interface package Related with CMT, and to the top project (e.g. LCGCMT) The special case of external packages Creating the kit: Define CMTPATH for the subproject Select a cache directory Select a platform tag Launch a cmt broadcast loop over the release package Will reach all packages of the sub-project Run the cmtpacman/scripts/create_kit.sh shell script Installing the kit: Select an installation location Select a release Run pacman Setup the runtime environment: Source the setup script generated by pacman Source the setup script for the specific application Run athena straight from the installed kit Develop against an installed kit: Setup the runtime environment Select a work area Checkout a package using client CVS Build it and run it > cmt broadcast $CMTROOT/../cmtpacman/scripts/create_kit.sh -platform opt /afs/cern.ch/…/pacman > cd /myinstall > pacman –get ATLAS:8.7.0/AtlasRelease-opt > source setup.sh > source dist/8.7.0/Control/AthenaRunTime/*/cmt/setup.sh > cd /myworkarea > export CMTPATH=`pwd`:$CMTPATH > cmt co DetectorDescription/IdDictParser > cd DetectorDescription/IdDictParser/*/cmt > gmake > … Currently we only produce development kits Source kits can also be produced Development against the installed kit implies getting the source through CVS Pacman 3 permits to install several releases in a common location External packages are shared between releases Operations One of the Atlas packages Available from pacman too Provide scripts to automate the post-install kit validation Structured as a set of transformation scripts Transform data from a state to another state (generation, simulation, etc…) Launch athena.py over existing job options (picked up from the installed release) Run various configuration tests Checking of DLLs Test the development scenario (building against the installed kit) Can generate dedicated data files or job options Construct job parameters IO filenames Random seeds Number of events Provide a more general framework of batch job submission Data Challenges productions Grid job submissions Format and capture the job output Can describe the PASS/FAIL conditions in terms of patterns (regular expressions) Filter PASS/FAILURE conditions according to the patterns Generate a consolidated report Can be formatted as a web page (or even Wiki page) ALL:SimpleChecks.kvt:Release Simple Checks ALL:AthExHelloWorld.kvt:Athena Hello World ALL:AthenaPoolTest.kvt:Athena POOL Test ALL:InDetDetDescrExample.kvt:InDetDetDescr Example ALL:MooEventCompile.kvt:MooEvent compilation ALL:EvgenMuon.kvt:Athena Muon Generation ALL:DC2EvgenZee.kvt:DC2 Z -> e e Event Generation ALL:DC2EvgenZmm.kvt:DC2 Z -> mu mu Event Generation ALL:DC2EvgenZtt.kvt:DC2 Z -> tau tau Event Generation ALL:DC2EvgenJJ.kvt:DC2 DiJet Event Generation ALL:DC2EvgenMinBias.kvt:DC2 Minimum Bias Event Generation ALL:DC2EvgenSusy.kvt:DC2 Susy Event Generation ALL:DC2EvgenTop.kvt:DC2 Top Event Generation ALL:DC2EvgenJetCalib.kvt:DC2 Jet Calibration Event Generation ALL:DC2EvgenJetFilter.kvt:DC2 Jet (pt>17) Event Generation ALL:DC2AtlfastZee.kvt:DC2 atlfast Z -> e e Ntuple/Histo production ALL:DC2G4simZee.kvt:DC2 Z -> e e G4 Simulation ALL:DC2G4digitZee.kvt:DC2 Z -> e e Digitization ALL:DC2reconZee.kvt:DC2 Z -> e e Reconstruction Kit validation