CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium An Edge Services Framework (ESF) for EGEE, LCG, OSG The XVth International Conference on Computing in High Energy and Nuclear Physics (CHEP’06) February 15, 2006 TIFR, Mumbai Abhishek Singh Rana UC San Diego Frank Würthwein UC San Diego

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 2 MALON, David (Argonne National Laboratory, IL, USA) MAY, Ed (Argonne National Laboratory, IL, USA) POPESCU, Razvan (Brookhaven National Laboratory, Upton, NY, USA) SOTOMAYOR, Borja (University of Chicago, IL, USA) SHANK, Jim (Boston University, MA, USA) LAURE, Erwin (CERN (European Organization for Nuclear Research), Geneva, Switzerland) BIRD, Ian (CERN (European Organization for Nuclear Research), Geneva, Switzerland) SCHULZ, Markus (CERN (European Organization for Nuclear Research), Geneva, Switzerland) FIELD, Laurence (CERN (European Organization for Nuclear Research), Geneva, Switzerland) PORDES, Ruth (Fermi National Accelerator Laboratory, IL, USA) SKOW, Dane (Fermi National Accelerator Laboratory, IL, USA) LITMAATH, Maarten (CERN (European Organization for Nuclear Research), Geneva, Switzerland) CAMPANA, Simone (CERN (European Organization for Nuclear Research), Geneva, Switzerland) WENAUS, Torre (Brookhaven National Laboratory, Upton, NY, USA) SMITH, David (CERN (European Organization for Nuclear Research), Geneva, Switzerland) BLUMENFELD, Barry (Johns Hopkins University, Baltimore, MD, USA) MARTIN, Stuart (Argonne National Laboratory, IL, USA) DE, Kaushik (The University of Texas, Arlington, TX, USA) VRANICAR, Matthew (PIOCON, IL, USA) WEICHER, John (PIOCON, IL, USA) SMITH, Preston (Purdue University, IN, USA) WANG, Shaowen (University of Iowa) RANA, Abhishek Singh (University of California, San Diego, CA, USA) WUERTHWEIN, Frank (University of California, San Diego, CA, USA) GARDNER, Robert (University of Chicago, IL, USA) KEAHEY, Kate (Argonne National Laboratory, IL, USA) FREEMAN, Timothy (Argonne National Laboratory, IL, USA) VANIACHINE, Alexandre (Argonne National Laboratory, IL, USA) HOLZMAN, Burt (Fermi National Accelerator Laboratory, IL, USA) Authors (ESF mailing list)

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 3 Outline ESF Activity ESF Phase 1 –Concepts and Design ESF future direction Xen overview Phase 1 Status Next Steps

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 4 Vision

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 5 Can there be a shared Services Framework that makes site admins happy? No login access to strangers. Isolation of services. –VOs can’t affect each other. –VOs receive a strictly controlled environment. Encapsulation of services. –Service instances can receive security review by site before they get installed. Explore solution based on virtual machines.

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 6 OSG-ESF Activity Started in September Physicists, Computer Scientists & Engineers, Software Architects. Chairs: Kate Keahey and Abhishek Singh Rana. Workspace Services Architecture and Design: –Globus Alliance and UC San Diego. Edge Services Implementations: –USATLAS: Teams at U Chicago and ANL. –USCMS: Teams at UC San Diego and FNAL. Mailing List and Discussion Forum – Web collaborative area –

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 7 ESF - Phase 1

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 8 No ESF - Phase 0 SECE Site

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 9 No ESF - Phase 0 Site SECE CMSATLASCDF Static Deployment of VO Services on a Site

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 10 ESF? SECE Site

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 11 ESF - Phase 1 ESF SE Site CE CDF CMS ATLAS Guest VO Snapshot of ES Wafers implemented as Virtual Workspaces

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 12 An attempt at ESF Terminology Edge Services Wafer (ES Wafer) –A specific instance of a dynamically-created VM (workspace) is called an Edge Services Wafer. –An ES Wafer can have several Edge Services running. –A VO can have multiple ES Wafers up at a Site. Edge Services Slot (ES Slot) –An ES Slot has hardware characteristics specified by the Site Admin. –An ES Slot can be leased by a VO to host an ES Wafer. Edge Service (ES) –A VO-specific service instantiated by a VO in a Wafer. Workspace Service (WS) –Service at a Site that allows VOs to instantiate ES Wafers in ES Slots.

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 13 ESF - Phase 1 CDF CMS ATLAS Guest VO ESF SECE Site GT4 Workspace Service & VMM Dynamically deployed ES Wafers for each VO Wafer images stored in SE Compute nodes and Storage nodes Snapshot of ES Wafers implemented as Virtual Workspaces

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 14 User jobs at Compute nodes using ES Wafers for VO Edge Services ESF SECE Site CDF CMS ATLAS Guest VO

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 15 VO Admin transporting/storing ES image to a remote Site....Deploying ES using image stored in Site’s local repository

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 16 ESF - Phase 1 ESF SECE Site CMS Role=VO Admin

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 17 ESF - Phase 1 ESF SECE Site CMS Role=VO Admin PEP

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 18 ESF - Phase 1 ESF SECE Site CMS Role=VO Admin

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 19 ESF - Phase 1 ESF SECE Site Role=VO Admin

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 20 ESF - Phase 1 ESF SECE Site Role=VO Admin PEP

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 21 ESF - Phase 1 ESF SECE Site Role=VO Admin

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 22 ESF - Phase 1 ESF SECE Site Role=VO Admin PEP

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 23 ESF - Phase 1 ESF SECE Site CMS Role=VO Admin

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 24 ESF - Phase 1 ESF SECE Site CMS Role=VO Admin

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 25 ESF - Phase 1 ESF SECE Site CMS Role=VO Admin

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 26 ESF - Phase 1 ESF SECE Site CMS Role=VO Admin ES Wafer (Multiple VO Services at a Site’s Edge)

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 27 A VO User using ES..

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 28 ESF - Phase 1 ESF SECE Site CMS Role=VO User

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 29 ESF - Phase 1 ESF SECE Site CMS Role=VO User PEP

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 30 ESF - Phase 1 ESF SE Site CMS Role=VO User CE

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 31 ESF - Phase 1 ESF SE Site CMS Role=VO User CE PEP

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 32 ESF - Phase 1 ESF SE Site CMS Role=VO User CE Resource Slice (User execution environment at a WN)

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 33 ESF - Phase 1 ESF SECE Site CMS Role=VO User

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 34 ESF - Phase 1 ESF SECE Site CMS Role=VO User PEP

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 35 ESF - Phase 1 ESF SECE Site CMS Role=VO User

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 36 ESF - future direction

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 37 ESF - future direction Same concept. Deeploy a cluster of ES slots that are fully schedulable by any VO allowed at the grid site.

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 38 ESF - future direction ESF Site Brokering and Scheduling by Edge Services Framework Dynamically deployed ES Wafers for many VOs Cluster of ES Slots with different properties ATLAS1 CMS ATLAS2 CDF

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 39 Xen overview Hardware Virtual Machine Monitor (VMM) Public Network Private Network

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 40 Phase 1 on OSG

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 41 Phase 1 on OSG ATLAS & CMS procure one piece of hardware at their Sites on OSG that runs ESF (called ESF node). –Dual CPU recommended. –4GB RAM (Xen2 has no PAE support, Xen3 has.). Site administrators install: –Xen (Xen 2.0.7, Xen 3.0.0). –GT4 Workspace Service. VO administrators use ESF to fire-up Xen VMs that instantiate VO Services – Edge Services in an ES Wafer. A single ESF node hosts ES Wafers for both ATLAS & CMS.

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 42 Site Administrator Responsibilities Deploy: –Xen. –Custom kernel for domain 0 (Grub bootloader required). –Custom kernel for domain U. –Prepare RAMdisk image if needed. –GT4. –GT4 Workspace Service. Provision: –One public IP, One private IP per VM. –Host certificates per VM. –Disk space per VM. Declare available ES Slots and their properties to ESF.

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 43 VO Administrator Responsibilities Fetch a standard OS filesystem image from a central ESF repository. Deploy the desired service on OS filesystem image. Thus, prepare (freeze) ES Wafer instance. Develop portable methods to dynamically configure all networking properties at a remote Site. Package these. Prepare this image into file for transport. SRMCP the image to remote Site’s SE. Use ESF to fire-up Xen VM with VO-Services (ES Wafer) at remote Site, from image file in remote SE, using role based authorization. Advertise the running Edge Services as needed.

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 44 Status New features added to GT4 Workspace Service. First prototype of ESF with Integration-testbed (Xen2.0.7) consisting of sites at ANL, FNAL, UCSD, U Chicago; and a Production-testbed (Xen3.0.0) with a site at UCSD. Pure OS Filesystem Images: SL3.0.3, SL4, LTS 3, LTS 4, FC4, CentOS4. USCMS Edge Service: FroNTier (Squid db). USATLAS Edge Service: DASH (MySQL db) General Edge Service: A subset of OSG 0.4 CE. Stress/throughput testing performed at ANL and UCSD. Based on parts of above results, a publication submitted for peer-review to IEEE HPDC-15.

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 45 Partial list of features added to GT4 WSS (WSS Release: VM Technology Preview 1.1) Support for a new, allocate networking method that allows the workspace service administrator to specify pools of IP addresses (and DNS information) which are then assigned to virtual machines on deployment. The resource properties have been extended to publish deployment information about a workspace, such as its IP address. Workspace metadata validation has been extended to support requirement checking for specific architecture, Xen version, and CPU. The workspace factory advertises the supported qualities as a resource property; the requirement section of workspace metadata is checked against the supported set. The workspace service can now accept and process VOMS credentials and GridShib SAML attributes. Support for Xen3 has been added. The workspace client interface has been extended to enable subscribing for notifications and specifying the resource allocation information at command-line. Installation has been improved. The client now requires only a minimal installation (as opposed to the full service installation).

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 46 Next Steps Verify performance, functionality, robustness. Gain production use experience. –CDF is capable of failover operations between multiple squids, thus allowing production use experience without negative impact on users. –Example squid use cases: DB cache (FroNTier) Application tarball serving (see glideCAF & OSG-CAF presentations) Parrot based CDF software mounts. Further evolve GT4 Workspace Service design. Widen deployment to more USCMS and USATLAS sites, using CMS & ATLAS services as use cases.

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 47

CHEP 2006 Mumbai INDIA February Frank Würthwein and Abhishek Singh Rana Edge Services Framework for EGEE, LCG and OSGwww.opensciencegrid.org The Open Science Grid Consortium 48 Thank You.