CernVM-FS – Best Practice to Consolidate Global Software Distribution Catalin CONDURACHE, Ian COLLIER STFC RAL Tier-1 ISGC15, Taipei, March 2015

Outline Introduction Brief history CernVM-FS deployment at RAL EGI CernVM-FS Task Force EGI CernVM-FS infrastructure CernVM-FS impact on user communities biomed and CernVM-FS ISGC15, Taipei, March 2015

Introduction – CernVM-FS ? Read-only, globally distributed filesystem, optimized to access experiment software and conditions data Built using standard technologies (fuse, sqlite, http, squid and caches) Files and directories are hosted on standard web servers and mounted in the universal /cvmfs namespace ISGC15, Taipei, March 2015

Introduction – CernVM-FS ? Software needs one single installation, then it is available at any site with CernVM-FS client installed and configured Repository meta-data and files are downloaded on demand and locally cached ISGC15, Taipei, March 2015

Brief History – CernVM-FS Last 5 years distribution of experiment software and conditions data to WLCG sites has been changed massively –CernVM-FS became the primary method –No need of local installation jobs local SW tags (!) (heavy loaded) file servers –It relies on a robust decentralised network of repositories replicas ISGC15, Taipei, March 2015

Brief History – CernVM-FS In parallel the use of CernVM-FS in communities outside WLCG has been increasing steadily –Growing number of repositories and CernVM-FS servers around the world Over 2 years ago RAL Tier-1 started a non-LHC Stratum-0 service ISGC15, Taipei, March 2015

Non-LHC CernVM-FS Deployment at RAL Supported by GridPP UK project Initially (Dec’12) for UK VOs, then extended to international small VOs (2+ NGIs) Local installation jobs used to automatically publish the Stratum-0 – now obsolete Replaced by web interface used by VO Software Grid Managers to upload, unpack tarballs and publish – now less used ISGC15, Taipei, March 2015

Non-LHC CernVM-FS Deployment at RAL Recently Grid Security Interface (GSI) added to transfer and process tarballs and publish –Faster and easier –Allows a programmatic way to transfer and process the tarballs –Initially based only on DN access, then VOMS Roles access also enabled – January 2015 –Software available at WN level after max 3 hours ISGC15, Taipei, March 2015

Non-LHC CernVM-FS Deployment at RAL At the moment 21 repos (~500 GB) are hosted and published at RAL Separate Stratum-1 service for non-LHC VOs –Initially shared the WLCG Stratum-1 –Recently deployed HA 2-node cluster – plans for a consolidated Stratum-1 at RAL (WLCG and non-WLCG) ISGC15, Taipei, March 2015

Software Installation Mechanism at RAL ISGC15, Taipei, March 2015 /home/augersgm /home/biomedsgm. /home/t2ksgm GSI Interface CVMFS Uploader Stratum-0 GSIssh/scp DN credentials VOMS Role credentials VO Software Grid Managers /cvmfs/auger.egi.eu /cvmfs/biomed.egi.eu. /cvmfs/t2k.egi.eu

EGI CernVM-FS Task Force EGI initiative to establish a CernVM-FS infrastructure that allows EGI Virtual Organizations to use it as a standard method of distribution of their software at grid sites Kick-off meeting – August 2013 Members: representatives from NGIs and other collaborations, VO SGMs, system admins at sites ISGC15, Taipei, March 2015

EGI CernVM-FS Task Force To promote the use of CernVM-FS technology amongst user communities To create a network of sites providing CernVM-FS services (Stratum-0, Stratum-1, Squid proxies) To encourage cooperation with other organizations (OSG, WLCG, regional grid infrastructures) by cross-replicating repositories for VOs supported by multiple collaborations ISGC15, Taipei, March 2015

EGI CernVM-FS Infrastructure The model follows the CernVM-FS topology for WLCG ISGC15, Taipei, March 2015

EGI CernVM-FS Infrastructure ‘relaxed’ topology Stratum-0 are disjoint and represent the source repositories where software is installed by VOs Stratum-0, Stratum-1 can be geographically co- located or not Stratum-1 can replicate an entire Stratum-0 or can only partially replicate – the ’relaxed’ model It partially makes use of the existent hierarchy of proxy servers used for LHC software distribution ISGC15, Taipei, March 2015

EGI CernVM-FS Infrastructure ISGC15, Taipei, March 2015 Stratum-0 NIKHEF nikhef.nl Stratum-1 NIKHEF Proxy Hierarchy Stratum-1 CERN Proxy Hierarchy Stratum-0 DESY desy.de Stratum-1 RAL Stratum-0 RAL egi.eu Proxy Hierarchy Stratum-1 DESY Proxy Hierarchy

EGI CernVM-FS Extended Infrastructure ISGC15, Taipei, March 2015 Stratum-0 NIKHEF nikhef.nl Stratum-1 NIKHEF Proxy Hierarchy Stratum-1 CERN Proxy Hierarchy Stratum-0 DESY desy.de Stratum-1 RAL Stratum-0 RAL egi.eu Proxy Hierarchy Stratum-1 DESY Proxy Hierarchy Stratum-1 ASGC Stratum-1 TRIUMF

EGI CernVM-FS Infrastructure 31 software repositories currently hosted –HEP and non-HEP –Stratum-0s at RAL, NIKHEF, DESY –5 repos at the time of kick-off –BIG change in two years! ISGC15, Taipei, March 2015

EGI CernVM-FS Infrastructure ‘egi.eu’ domain –Its importance recognized at the CernVM-FS WG meeting (March 2014) –Taken into account by latest CernVM-FS packages releases public key and CernVM-FS domain configuration now part of standard installation (as for ‘cern.ch’) – cvmfs-keys v1.5 no longer CERN-centric configuration – cvmfs v new puppet module available ISGC15, Taipei, March 2015

CernVM-FS Impact on User Communities Encouraged by the expansion of CernVM-FS infrastructure, more VOs have identified CernVM- FS as a facility that suited better their needs –HEP (auger, t2k.org, hyperk.org, glast.org) –Space, Natural and Life Sciences (biomed, km3net.org, enmr.eu, pheno) –EGI fedcloud community – exploratory discussions But software distribution to grid and cloud sites still an obstacle for some non-LHC VOs ISGC15, Taipei, March 2015

CernVM-FS Impact on User Communities Deployment issues –VOs are lacking manpower to progress more –Technology and its advantages are understood, just they do not have time –Subgroups within same VO are using subsets of software at specific sites coordination needed to consolidate software under single CernVM-FS space –Sometimes VOs wait too long for sites to help them with initial tests before rolling out more widely ISGC15, Taipei, March 2015

biomed and CernVM-FS biomed - large scale international and multi- disciplinary VO supporting communities from the Life Sciences sector Divided into three sectors: medical imaging, bioinformatics and drug discovery Member of the Life Science Grid Community –Other members: vlemed, lsgrid, vo.eu-decide.eu, enmr.eu Openly accessible to academics and to private companies for non-commercial purposes ISGC15, Taipei, March 2015

biomed and CernVM-FS biomed - Status at September 2013 –~80 supporting sites –~50 sites running significant production –3M jobs during Sep’13 Many biomed sub-groups installed various software packages at supporting sites –‘biomedsgm’ installation jobs –Need for NFS (or AFS) space at sites –Distribution of new SW releases was time consuming ISGC15, Taipei, March 2015

biomed and CernVM-FS Timeline –Sep’13 – initial discussions (EGI TF Madrid) –Oct’13 – Mar’14 Stratum-0 repository at RAL local install jobs used initially for upload web upload interface and GSI interface tested work to prove the concept within biomed communities first test jobs run at RAL using CernVM-FS space – Jan’14 –Apr’14 supporting sites asked (GGUS, personal contact) to enable biomed ISGC15, Taipei, March 2015

biomed and CernVM-FS Timeline –Jul’14 GSI as standard method to upload biomed shifters invited to use CernVM-FS –Sep’14 GGUS campaign for more sites to enable CernVM-FS support –Nov – Dec’14 44 out of 90 sites provide CernVM-FS VO-biomed-CVMFS tag used to identify supporting sites biomed community informed about CernVM-FS availability ISGC15, Taipei, March 2015

biomed and CernVM-FS ISGC15, Taipei, March 2015 Current status –75 sites running biomed jobs –90 out of 169 CEs advertise VO-biomed-CVMFS tag

Summary – CernVM-FS ISGC15, Taipei, March 2015 Success story Critical service for WLCG –soon to become critical for others Non-LHC Virtual Organizations are successfully following the model –biomed, enmr.eu, auger –increasing number of EGI VO repos –75 sites running biomed jobs –90 out of 169 CEs advertise VO-biomed-CVMFS tag

ISGC15, Taipei, March 2015 Thank you! (more) Questions?