1. Organization and Management of ATLAS Nightly Builds F. Luehring a, E. Obreshkov b, D.Quarrie c, G. Rybkine d, A. Undrus e University of Indiana, USA a, DESY, Germany b, LBNL, USA c,LAL, France d,BNL, USA e Organization and Management of ATLAS Nightly Builds F. Luehring a, E. Obreshkov b, D.Quarrie c, G. Rybkine d, A. Undrus e University of Indiana, USA a, DESY, Germany b, LBNL, USA c,LAL, France d,BNL, USA e Abstract 25 branches of automated multi-platform software nightly builds is a major component in ATLAS collaborative software organization and code approval scheme. The successful nightly releases are transformed into stable releases used for data processing worldwide. ATLAS nightly builds are managed by the fully automated NICOS framework [1]. The modular structure of NICOS allows for an easy integration of third-party build and validation tools. The ATN [2] test tool is embedded within the nightly system and provides the first results even before the full compilations completion. Several ATLAS test frameworks are synchronized with NICOS jobs and run larger production jobs with the nightly releases. NICOS web pages dynamically provide information about the progress and results of the builds. For faster feedback the e-mail notifications about nightly build problems are automatically distributed to responsible developers. Abstract 25 branches of automated multi-platform software nightly builds is a major component in ATLAS collaborative software organization and code approval scheme. The successful nightly releases are transformed into stable releases used for data processing worldwide. ATLAS nightly builds are managed by the fully automated NICOS framework [1]. The modular structure of NICOS allows for an easy integration of third-party build and validation tools. The ATN [2] test tool is embedded within the nightly system and provides the first results even before the full compilations completion. Several ATLAS test frameworks are synchronized with NICOS jobs and run larger production jobs with the nightly releases. NICOS web pages dynamically provide information about the progress and results of the builds. For faster feedback the e-mail notifications about nightly build problems are automatically distributed to responsible developers. Nightly Framework References [1] A. Undrus, CHEP 03, La Jolla, USA, 2003, eConf C0303241, TUJT006 [hep-ex/0305087]; http://www.usatlas.bnl.gov/computing/software/nicos/index.html http://www.usatlas.bnl.gov/computing/software/nicos/index.html [2] A. Undrus, CHEP 04, Interlaken, 2004, Conference proceedings, p. 521 [3] http://www.cmtsite.orghttp://www.cmtsite.org [4] Tag Collector reference [5] AMI reference [6] PackDist reference [7] RTT reference [8] FCT reference [9] TCT reference [10] KV rererence References [1] A. Undrus, CHEP 03, La Jolla, USA, 2003, eConf C0303241, TUJT006 [hep-ex/0305087]; http://www.usatlas.bnl.gov/computing/software/nicos/index.html http://www.usatlas.bnl.gov/computing/software/nicos/index.html [2] A. Undrus, CHEP 04, Interlaken, 2004, Conference proceedings, p. 521 [3] http://www.cmtsite.orghttp://www.cmtsite.org [4] Tag Collector reference [5] AMI reference [6] PackDist reference [7] RTT reference [8] FCT reference [9] TCT reference [10] KV rererence NICOS FRAMEWORK NIGHTLY CONFIGURATION DISTRIBUTION CODE MANAGEMENT TESTING nightly release CMT build ATN testing AMI DB Tag Collector CVS Repository RTT FCT TCT KIT VALIDATION KITS CERN AFS ATLAS Tier Centers NICOS WEB PAGES NICOS is connected with ATLAS collaborative tools:  CMT [3] code management and build tool  Tag Collector [4] web based tool for managing the tags of packages in release  ATLAS CVS code repository  ATLAS metadata DB (AMI) [5] for storage of NICOS configurations for different nightlies branches  Nightly releases are installed for worldwide access at CERN AFS and also can be downloaded with ATLAS distribution kits tools [6]  Integrated ATN [2] and external RTT [7], FCT [8], TCT [9], Kit Validation [10] testing frameworks NICOS Web Pages All nightlies summary with message of day, distribution kit and tests status Nightlies computer farm status with dynamic information on nodes Log files with error diagnostics with links to problem messages Nightly branch summary with results for inside projects and platforms Summary for project/platform with latest releases comparisons Nightly release summary with diagnostics for individual packages Software Release Structure AtlasOffline ATLAS PROJECTS ATLAS PATCH PROJECTS PACKAGE S a) Containers: directories for related packages b) Leaf: source code and/or scripts c) Glue: interfaces to externals ………. Number of leaf + container packages: EXTERNALS AtlasAnalysis AtlasTrigger AtlasReco AtlasSimulation AtlasEvent AtlasConditions AtlasCore GAUDI Framework DetCommon Data Quality Monitoring Data Acquisition Software Interfaces to externals (“LCGCMT”) Externals (ROOT, Geant, …) 26+20 234+84 187+30 452+117 190+35 293+90 206+55 178+20 15+5 Total: 1781 leaf, 456 container packages as of 02/15/09 Projects are groups of packages with similar dependencies managed by CMT [3]. They are built as units and can evolve at different rates. Patch projects sit at the top of project hierarchy. They contain override versions of packages from downstream projects. Software Validation Release Coordinator ensure design goals Chief Architect Validation Data Manager provides data samples Production Manager oversees jobs definitions Project Coordinators: evaluate new software submissions Package Managers: request upgrades through Tag Collector New software versions are verified in validation or migration nightlies. Upon approval from coordinators they are moved to the primary branch. Stable software releases are created from successful primaries. All patches to stable releases are sent to primary branch. Package versions associated with releases and project dependencies are handled by the Tag Collector. Primary development nightly branch Validation nightly branch Migration nightlies: Verification of disruptive changes LCG nightlies: Verification of externals upgrades Stable release Stable release Stable release Patch release Patch release ATLAS Nightlies in Numbers  Number of branches: 25  Total number of platforms in all branches: 40  All nightlies are rebuilt and tested daily  Old nightly releases are kept for 2 - 7 days  Nightlies computing farm: 40 nodes, 4x3.0 or 8x2.33 GHz 8 - 16 GB RAM  Time to rebuild nighty release: 10 hours  Number of ATN tests: 300  Additional time to complete ATN tests: 5 hours ATLAS Nightlies in Numbers  Number of branches: 25  Total number of platforms in all branches: 40  All nightlies are rebuilt and tested daily  Old nightly releases are kept for 2 - 7 days  Nightlies computing farm: 40 nodes, 4x3.0 or 8x2.33 GHz 8 - 16 GB RAM  Time to rebuild nighty release: 10 hours  Number of ATN tests: 300  Additional time to complete ATN tests: 5 hours NICOS Design NICOS nightly jobs consist of tightly synchronized streams. The results for different platforms are combined in a single release. NICOS Design NICOS nightly jobs consist of tightly synchronized streams. The results for different platforms are combined in a single release. Master platform Slave platf. Tag Collector accessCode checkout proj.1Code checkout proj.2Code checkout proj.3 Build project 1Build project 2Build project 3 Install 2Install 1 Tests project 1Tests project 2 Build project 1Build project 2Build project 3 Install 1Install 2wait