1. WLCG Collaboration Workshop;
Clouds and Virtualisation
Ian Bird, CERN
WLCG Collaboration Workshop;
DESY; 12th July 2011
Accelerating Science and Innovation 1

2. Activities
GDB – requested experiments to briefly explain their positions regarding virtualisation and clouds
HEPiX – working group (>1 year) to propose ways to enable trusted image sharing
in particular to use cernvm images
Here:
What are sites doing? Planning?
Can we clarify the likely use cases from both experiment and site viewpoints?

3. ATLAS R&D: Use cases: Virtualisation: use of VMs at sites
evaluate cloud technologies
design a model for clouds interacting with ATLAS software
implement in DDM, Panda, etc
activity on cvmfs + multicore
Use cases:
MC on cloud with stage out to traditional grid storage or long term cloud storage
data reprocessing in the cloud (cost?)
distributed analysis with data at remote grid sites, or moved to cloud
resource capacity bursting for urgent tasks
prototype by end 2011 for simulation, reproc, analysis
Virtualisation: use of VMs at sites
No real position yet, but several unknowns; e.g.
who builds and manages VMs?
who instantiates at site?
requirements on VMs (cores, memory etc)

4. CMS Whole node approach Virtualisation clouds
execute process in user space on many core host - manage workload - wait for task force output
Virtualisation 
not interested per se:  mainly a site business.
happy with real nodes (or virtual nodes), nothing against sites using VM's as long as performance is OK. and monitoring is not impacted
clouds
commercial clouds may be used for simulation (bursting) but currently thought to be too expensive
use of cloud interface for LCG resources is possible but depends on implications
prefer efficient local access to site storage and not root access on machine in DMZ
current grid interfaces may be used to access whole nodes
could use commercial clouds if cost was OK (for some tasks).

5. LHCb
Aim to replace Dirac pilot with a customized cernvm VM, using cernvm certified image
using contextualisation for pilot credentials, starting dirac job agent
requirements: outbound connectivity to central VOboxes, and storage, LFC etc
starting a machine with EC2 would be OK
CVMDirac
could run on Amazon
LXCloud or similar institutional cloud - as alternative to batch system
opportunistic usage - e.g. BOINC
can also be used with multicore - some development needed

6. ALICE
Currently no interest

7. HEPiX
WLCG expects that sites will not be prepared to instantiate random images in the way that Amazon does:
Site infrastructure may not be adequately protected against images that allow root access to unknown people,
Sites are required to maintain traces of activity at sites (syslog/process accounting) to enable investigations in case of security incidents.
The HEPiX process documents a way to create "certified images" that address these concerns and should, therefore, be freely transmissible between sites.
Pilot job frameworks:
no reason for end users to be concerned with image creation
interface is with the central experiment task queue.
Which images are instantiated where is the concern of the central experiment team

8. WLCG Site use cases? Site Summary: Future:
(Many) Sites will run virtualised infrastructures (some may be private clouds)
Many different implementations and prototypes
CERN interested in “standard” implementations (e.g. Openstack)
Interest in ability to burst out to commercial clouds
Essentially should be hidden from applications
Future:
How far can “standard” cloud interfaces supplement/replace grid job management?

9. Some discussion questions
What are sites doing? Planning?
Where does CERNVM fit?
Where does whole-node scheduling fit?
Can we clarify the likely use cases from both experiment and site viewpoints