EGEE-II INFSO-RI Enabling Grids for E-sciencE An Introduction to the EGEE Project Presented by Min Tsai ISGC 2007, Taipei With thanks to EGEE colleagues for many of these slides

Enabling Grids for E-sciencE EGEE-II INFSO-RI Contents What is EGEE? –Goals –Status –Activities Grid services: gLite 3.0 Sources of further information

Enabling Grids for E-sciencE EGEE-II INFSO-RI EGEE – international e-infrastructure Network infrastructure & Resource centres Operations, Support and training Collaboration Pan-European Grid A four year programme: Build, deploy and operate a consistent, robust a large scale production grid service that –Links with and build on national, regional and international initiatives Improve and maintain the middleware in order to deliver a reliable service to users Attract new users from research and industry and ensure training and support for them

Enabling Grids for E-sciencE EGEE-II INFSO-RI In the first 2 years (to March 2006) Established production quality sustained Grid services –3000 users from at least 5 disciplines –Goal was to integrate 50 sites into a common infrastructure  currently 180 –offer 5 Petabytes (10 15 ) storage Demonstrated a viable general process to bring other application communities on board Secured a second phase from April 2006 Pilot New

Enabling Grids for E-sciencE EGEE-II INFSO-RI Natural continuation of EGEE –Expanded consortium –Emphasis on providing an infrastructure  increased support for applications  interoperate with other infrastructures  more involvement from Industry SA: service activities - establishing operations NA: network activities - supporting VOs JRA: “joint research activities” - e.g. hardening middleware EGEE-II

Enabling Grids for E-sciencE EGEE-II INFSO-RI The EGEE Virtuous Cycle NA2, NA3, NA4 NA3, NA4 SA1 JRA1 Building effective user communities

Enabling Grids for E-sciencE EGEE-II INFSO-RI The EGEE Infrastructure Certification testbeds (SA3) Pre-production service Production service Test-beds & Services Operations Coordination Centre Regional Operations Centres Global Grid User Support EGEE Network Operations Centre (SA2) Operational Security Coordination Team Support Structures Operations Advisory Group (+NA4) Joint Security Policy GroupEuGridPMA (& IGTF) Grid Security Vulnerability Group Security & Policy Groups Infrastructure: Physical test-beds & services Support organisations & procedures Policy groups

Enabling Grids for E-sciencE EGEE-II INFSO-RI Grid management: structure Operations Coordination Centre (OCC) –management, oversight of all operational and support activities Regional Operations Centres (ROC) –providing the core of the support infrastructure, each supporting a number of resource centres within its region –Grid Operator on Duty Resource centres –providing resources (computing, storage, network, etc.); Grid User Support (GGUS) –At FZK, coordination and management of user support, single point of contact for users

Enabling Grids for E-sciencE EGEE-II INFSO-RI Production service sites CPU Size of the infrastructure today: 192 sites in 40 countries ~ CPU ~ 3 PB disk, + tape MSS

Enabling Grids for E-sciencE EGEE-II INFSO-RI Usage of the infrastructure >50k jobs/day ~7000 CPU-months/month

Enabling Grids for E-sciencE EGEE-II INFSO-RI Related projects: infrastructure, engineering, education NameDescription BalticGridEGEE extension to Estonia, Latvia, Lithuania EELAEGEE extension to Brazil, Chile, Cuba, Mexico, Argentina EUChinaGRIDEGEE extension to China EUMedGRIDEGEE extension to Malta, Algeria, Morocco, Egypt, Syria, Tunisia, Turkey ISSeGSite security eIRGSPPolicies ETICSRepository, Testing OMII-Europeto provide key software components for building e-infrastructures; BELIEFDigital Library of Grid documentation, organisation of workshops, conferences BIOINFOGRIDBiomedical Health-e-ChildBiomedical – Integration of heterogeneous biomedical information for improved healthcare ICEAGEInternational Collaboration to Extend and Advance Grid Education

Enabling Grids for E-sciencE EGEE-II INFSO-RI Gate Keeping: Identify users and give permissions Authentication, Authorisation (AA) Users in many locations and organisations Data Storage Grid Services Access services (User Interface) : logon, upload credentials, run m/w commands Built on Grid Security Infrastructure Encryption and Data Integrity, Authentication and Authorization Computation High Level Service Software Resources in many locations and organisations

Enabling Grids for E-sciencE EGEE-II INFSO-RI Work Load Management Users Compute elements Network resources Data storage Resources How do I run a job on a Compute Element (CE) ? (CE = batch queue) Tools for: Submit jobs to a CE Monitor jobs Get outputs Transfer files to CE Transfer files between CE and SE

Enabling Grids for E-sciencE EGEE-II INFSO-RI Information System (IS) Users Compute elements Network resources Data storage Resources Which resources can I use? Resource publishes: Configuration Latest status Access information

Enabling Grids for E-sciencE EGEE-II INFSO-RI Data Management Users Compute elements Network resources Data storage Resources My data are in files, and I’ve terabytes Storage Space Data Transfer Replication Our data are in files, and I’ve terabytes How can I manage and share my data?

Enabling Grids for E-sciencE EGEE-II INFSO-RI User Interface (UI) User Interface (UI): The place where users logon to the Grid Computing Element (CE) Computing Element (CE): A batch queue on a site’s computers where the user’s job is executed Storage Element (SE) Storage Element (SE): Provides (large-scale) storage for files Resource Broker (RB) Resource Broker (RB): Matches the user requirements with the available resources on the Grid Main components Information System Information System: Characteristics and status of CE and SE (Uses “GLUE schema”)

Enabling Grids for E-sciencE EGEE-II INFSO-RI Current production middleware FileCatalogue Logging & Book-keeping ResourceBrokerStorageElementComputingElement InformationService Job Status DataSets info Author. &Authen. Job Submit Event Job Query Job Status Input “sandbox” Input “sandbox” + Broker Info Output “sandbox” Publish SE & CE info “User interface”

Enabling Grids for E-sciencE EGEE-II INFSO-RI Example JDL file Executable = “gridTest”; StdError = “stderr.log”; StdOutput = “stdout.log”; InputSandbox = {“/home/joda/test/gridTest”}; OutputSandbox = {“stderr.log”, “stdout.log”}; InputData = “lfn:/grid/gilda/training/testbed ”; DataAccessProtocol = “gridftp”; Requirements = other.Architecture==“INTEL” && \ other.OpSys==“LINUX”; Rank = “other.GlueHostBenchmarkSF00”; Building on basic tools and Information Service Submit job to grid via the “resource broker (RB)”, glite_job_submit my.jdl Returns a “job-id” used to monitor job, retrieve output

Enabling Grids for E-sciencE EGEE-II INFSO-RI Who provides the resources?! ServiceProviderNote Access service User / institute / VOComputer with client software Resource Broker (RB) VO Information System Information System:Grid operations Computing Element (CE) VOs - EGEE does not fund CEs Scalability requires that VOs provide resources to match average need Storage Element (SE) VOs “VO”: virtual organisation “Grid operations”: funded effort

Enabling Grids for E-sciencE EGEE-II INFSO-RI :00 on 8 March

Enabling Grids for E-sciencE EGEE-II INFSO-RI What is happening now?

Enabling Grids for E-sciencE EGEE-II INFSO-RI Further information EGEE Conference: September September-2006-geneva/ September-2006-geneva/ EGEE digital library: EGEE EGEE: 1 st user Forum gLite Open Grid Forum Globus Alliance VDT

Enabling Grids for E-sciencE EGEE-II INFSO-RI Summary EGEE is running the largest multi-VO grid in the world! Creating the “grid layer” in e-Infrastructure for research, public service and industry Key concepts for EGEE –Sustainability – planning for the long-term –Production quality –And… Grids are fundamentally about people … how people in different organisations commit to cooperate … and how that cooperation can be enabled by operations, training, support, and (most transient of all?) middleware