WLCG Collaboration Workshop; Clouds and Virtualisation Ian Bird, CERN WLCG Collaboration Workshop; DESY; 12th July 2011 Accelerating Science and Innovation 1

Activities GDB – requested experiments to briefly explain their positions regarding virtualisation and clouds HEPiX – working group (>1 year) to propose ways to enable trusted image sharing in particular to use cernvm images Here: What are sites doing? Planning? Can we clarify the likely use cases from both experiment and site viewpoints?

ATLAS R&D: Use cases: Virtualisation: use of VMs at sites evaluate cloud technologies design a model for clouds interacting with ATLAS software implement in DDM, Panda, etc activity on cvmfs + multicore Use cases: MC on cloud with stage out to traditional grid storage or long term cloud storage data reprocessing in the cloud (cost?) distributed analysis with data at remote grid sites, or moved to cloud resource capacity bursting for urgent tasks prototype by end 2011 for simulation, reproc, analysis Virtualisation: use of VMs at sites No real position yet, but several unknowns; e.g. who builds and manages VMs? who instantiates at site? requirements on VMs (cores, memory etc)

CMS Whole node approach Virtualisation clouds execute process in user space on many core host - manage workload - wait for task force output Virtualisation  not interested per se:  mainly a site business. happy with real nodes (or virtual nodes), nothing against sites using VM's as long as performance is OK. and monitoring is not impacted clouds commercial clouds may be used for simulation (bursting) but currently thought to be too expensive use of cloud interface for LCG resources is possible but depends on implications prefer efficient local access to site storage and not root access on machine in DMZ current grid interfaces may be used to access whole nodes could use commercial clouds if cost was OK (for some tasks).

LHCb Aim to replace Dirac pilot with a customized cernvm VM, using cernvm certified image using contextualisation for pilot credentials, starting dirac job agent requirements: outbound connectivity to central VOboxes, and storage, LFC etc starting a machine with EC2 would be OK CVMDirac could run on Amazon LXCloud or similar institutional cloud - as alternative to batch system opportunistic usage - e.g. BOINC can also be used with multicore - some development needed

ALICE Currently no interest

HEPiX WLCG expects that sites will not be prepared to instantiate random images in the way that Amazon does: Site infrastructure may not be adequately protected against images that allow root access to unknown people, Sites are required to maintain traces of activity at sites (syslog/process accounting) to enable investigations in case of security incidents. The HEPiX process documents a way to create "certified images" that address these concerns and should, therefore, be freely transmissible between sites. Pilot job frameworks: no reason for end users to be concerned with image creation interface is with the central experiment task queue. Which images are instantiated where is the concern of the central experiment team

WLCG Site use cases? Site Summary: Future: (Many) Sites will run virtualised infrastructures (some may be private clouds) Many different implementations and prototypes CERN interested in “standard” implementations (e.g. Openstack) Interest in ability to burst out to commercial clouds Essentially should be hidden from applications Future: How far can “standard” cloud interfaces supplement/replace grid job management?

Some discussion questions What are sites doing? Planning? Where does CERNVM fit? Where does whole-node scheduling fit? Can we clarify the likely use cases from both experiment and site viewpoints