Presentation is loading. Please wait.

Presentation is loading. Please wait.

Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 0 Lezione 7b - 9 Dicembre 2009 Il materiale didattico usato in questo corso è stato mutuato.

Similar presentations


Presentation on theme: "Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 0 Lezione 7b - 9 Dicembre 2009 Il materiale didattico usato in questo corso è stato mutuato."— Presentation transcript:

1 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 0 Lezione 7b - 9 Dicembre 2009 Il materiale didattico usato in questo corso è stato mutuato da quello utilizzato da Paolo Veronesi per il corso di Griglie Computazionali per la Laurea Specialistica in Informatica tenuto nellanno accademico 2008/09 presso lUniversità degli Studi di Ferrara. Paolo Veronesi Università degli Studi di Bari – Corso di Laurea Specialistica in Informatica Tecnologia dei Servizi Grid e cloud computing A.A. 2009/2010 Giorgio Pietro Maggi

2 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 1 Defining the Grid A Grid is the combination of networked resources and the corresponding middleware, which provides services for the user.

3 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 2 The EGEE Project Aim of EGEE: to establish a seamless European Grid infrastructure for the support of the European Research Area (ERA) EGEE 1 April 2004 – 31 March partners in 27 countries, federated in regional Grids EGEE-II 1 April 2007 – 30 April 2008 Expanded consortium EGEE-III 1 May 2008 – 30 April 2010 Transition to sustainable mod el

4 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 3 Enabling Grid for E-sciencE project Archeology Astronomy Astrophysics Civil Protection Comp. Chemistry Earth Sciences Finance Fusion Geophysics High Energy Physics Life Sciences Multimedia Material Sciences … >250 sites 48 countries >50,000 CPUs >20 PetaBytes >10,000 users >150 VOs >150,000 jobs/day Flagship Grid infrastructure project co- funded by the European Commission starting from April 2004 Entering now in the 3° phase

5 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 4 Defining the Grid A Grid is the combination of networked resources and the corresponding middleware, which provides services for the user.

6 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 5 EGEE Infrastructure

7 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 6 EGEE Infrastructures Production service Scaling up the infrastructure with resource centres around the globe Stable, well-supported infrastructure, running only well-tested and reliable middleware Pre-production service Run in parallel with the production service (restricted nr of sites) First deployment of new versions of the gLite middleware Test-bed for applications and other external functionality T-Infrastructure (Training&Education) Complete suite of Grid elements and application (Testbed, CA, VO, monitoring, support, …) Everyone can register and use GILDA for training and testing 20 sites on 3 continents

8 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 7 EGEE Operations Process Geographically distributed responsibility for operations: There is no central operation Regional Operation Centers Responsible or resource centers in their region Tools are developed/hosted at different sites: GOC DB (RAL), SAM (CERN), GStat (Taipei), CIC Portal (Lyon) Grid operator on duty 10 teams working in weekly rotation Crucial in improving site stability and management Operations coordination Weekly operations meetings Regular ROC managers meetings Series of EGEE Operations Workshops Procedures described in Operations Manual Introducing new sites Site downtime scheduling Suspending a site Escalation procedures; etc. Highlights: Distributed operation Evolving and maturing procedures Procedures being in introduced into and shared with the related infrastructure projects

9 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 8 Central probes (SAM) Local probes Network monitoring Doubled size and usage without impact on operations Improved reliability through multi-level monitoring

10 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 9 EGEE operations Operations Coord. Centre (OCC) - management, oversight of all operational and support activities Regional Operations Centres (ROC) - providing the core of the support infrastructure, each supporting a number of resource centres within its region Resource Centres (RC) - providing resources (computing, storage, network…) - At FZK, coordination and management of user support, single point of contact for users

11 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 10 Monitoring Visualization 10

12 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 11 The EGEE support infrastructure RC A RC B RC C RC A RC B RC C ROC C ROC B ROC N RC A RC B RC C TPM VO TPM C VO TPM B VO TPM A GGUS Central System Middleware support Middleware support Middleware support Deployment support Middleware support Deployment support VO Support C VO Support B VO Support A Middleware support Middleware support Middleware support ROC C ROC B ROC N Network Support Other Grids COD CIC Portal

13 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 12 Defining the Grid A Grid is the combination of networked resources and the corresponding middleware, which provides services for the user.

14 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 13 gLite Middleware Distribution Combines components from different providers Condor and Globus (via VDT) LCG EGEE Others Focus on providing a deployable MW distribution for EGEE production service Middleware services + configuration tools Follows a service oriented approach Usage of webservices where useful and possible performance-wise Complemented by application-level servcies

15 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 14 Production Grid Middleware Key factors in EGEE Grid Middleware Development: 1. Strict software process Use industry standard software engineering methods Software configuration management, version control, defect tracking, automatic build system, … 2. Conservative approach in what software to use Avoid cutting-edge software Deployment on over 200 sites cannot assume a homogenous environment – middleware needs to work with many underlying software flavors Avoid evolving standards Evolving standards change quickly (and sometime significantly cf. OGSI vs. WSRF) – impossible to keep pace on > 200 sites Long (and tedious) path from prototypes to production

16 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 15 Certification Functional Tests Testbed Deployment gLite Process Development Software Error Fixing Integration Deployment Packages Integration Tests Installation Guide, Release Notes, etc Pre-Production Scalability Tests Pre-Production Deployment Fail Pass Production Infrastructure Problem Directives External Software Directives

17 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 16 gLite Software Process Technical Coordination Group (TCG) gathers & prioritizes user requirements from HEP, biomed, (industry), sites gLite development is client-driven! Software from EGEE-JRA1 and other projects JRA1 preview testbed (currently being set up) early exposure to users of uncertified components SA3 Integration Team Ensures components are deployable and work Deployment Modules implemented high-level gLite node types (WMS, CE, R-GMA Server, VOMS Server, FTS, etc) Build system now spun off into the ETICS project (Jan 2006) SA3 Certification Team Merge of the JRA1 testing and SA1 certification teams Dedicated testbed; test release candidates and patches Develop test suites SA1 Pre-Production System Scale tests by users

18 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 17 Middleware Globus GT4Condor APST Platform Infrastructure UnixWindowsJVMTCP/IPMPI.Net Runtime Environmental Sciences Life & Pharmaceutical Sciences Applications Geo Sciences Building Software for the Grid VPNSSH Courtesy IBM

19 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 18 Middleware Globus GT4Condor APST Platform Infrastructure UnixWindowsJVMTCP/IPMPI.Net Runtime Environmental Sciences Life & Pharmaceutical Sciences Applications Geo Sciences Building Software for the Grid VPNSSH Courtesy IBM, Upper Middleware & Tools Lower Middleware Bonds

20 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 19 Defining the Grid A Grid is the combination of networked resources and the corresponding middleware, which provides services for the user.

21 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 20 EGEE Applications >270 VOs from several scientific domains Astronomy & Astrophysics Civil Protection Computational Chemistry Comp. Fluid Dynamics Computer Science/Tools Condensed Matter Physics Earth Sciences Fusion High Energy Physics Life Sciences Further applications under evaluation Applications have moved from testing to routine and daily usage ~80-95% efficiency

22 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 21 The Future of Grids Increasing the number of infrastructure users by increasing awareness Dissemination and outreach Training and education Increasing the number of applications by improving application support and middleware functionality Improved usability through high level grid middleware extensions Increasing the grid infrastructure Incubating related projects Ensuring interoperability between projects Protecting user investments Towards a sustainable grid infrastructure

23 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 22 Grid Interoperability Incubator for new Grid efforts world-wide Infrastructure and application efforts Leading role in building world-wide Grids through interoperation efforts Bilateral: EGEE/OSG, EGEE/NDGF, EGEE/NAREGI, EGEE/Unicore/DEISA Multilateral: Grid Interoperability Now (GIN) Experiences and requirements fed back into standardization process (OGF) Many EGEE members are area directors, WG chairs, WG members Contacts with industry strengthened Industry Days, Industry Task Force, Business Associates Programme GINGIN

24 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 23 EGEE working with related infrastructure projects GINGIN

25 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 24 Evolution European e-Infrastructure Testbeds Utility Service Routine Usage National Global

26 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 25 Need to prepare permanent, common Grid infrastructure Ensure the long-term sustainability of the European e-Infrastructure independent of short project funding cycles Coordinate the integration and interaction between National Grid Infrastructures (NGIs) Operate the production Grid infrastructure on a European level for a wide range of scientific disciplines Must be no gap in the support of the production grid

27 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 26 Summary Grids represent a powerful new tool for science Today we have a window of opportunity to move grids from research prototypes to permanent production systems (as networks did a few years ago) EGEE offers … … a mechanism for linking together people, resources and data of many scientific community … a basic set of middleware for gridfying applications with documentation, training and support … regular forums for linking with grid experts, other communities and industry

28 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 27 gLite Middleware overview

29 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 28 Grid Middleware When using a PC or workstation you Login with a username and password (Authentication) Use rights given to you (Authorisation) Run jobs Manage files: create them, read/write, list directories Components are linked by a bus Operating system One admin domain When using a Grid you – Login with digital credentials (Authentication) – Use rights given you (Authorisation) – Run jobs – Manage files: create them, read/write, list directories Services are linked by the Internet Middleware Many admin domains

30 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 29 EGEE Project and gLite Enabling Grids for E-sciencE (EGEE) is the largest multi-disciplinary grid infrastructure in the world – Brings together more than 120 European organisations – Consists of 250 sites in 48 countries and more than 68,000 CPUs – Is available to some 8,000 users 24 hours a day, 7 days a week – Processes more than 150,000 jobs per day from different scientific domains gLite is the middleware powering the EGEE infrastructure and many other related projects – Is an integrated set of components designed to enable resource sharing among different institutions – Pulls together contributions from many other projects, including LCG and VDT – Enable users with a large set of services

31 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 30 The global grid e-Infrastructures adopting gLite e-Infrastructures interoperable or in pro- gress to be made interoperable with gLite

32 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 31 Additional Infrastructures: GILDA EGEE provides a training infrastructure: GILDA (Grid INFN Laboratory for Dissemination Activities) – Runs the entire gLite stack protocols – Used to demonstrate EGEE grid technology project – Supports beginner and expert training courses on gLite Adopted by several Grid projects worldwide Own Certification Authority Available 365 days for everyone ! Used in the ISSGC schools series Since 2007 other middleware than gLite are tested on GILDA

33 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b sites in 3 continents > certificates issued, >20% renewed at least once > 250 courses, training events, official university curricula > 2,000,000 hits on the web site from >100 different countries > 4.5 TB of training material downloaded from the web site The GILDA t-Infrastructure (https://gilda.ct.infn.it)

34 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 33 gLite in the Grid ecosystem... LCG EGEE Used in USAEU NextGridDEISAGridCC Future grids EDG GlobusMyProxyCondor... VDT DataTAGCrossGrid... OSG, … SRM …interactive

35 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 34 The Middleware structure Applications have access both to Higher-level Grid Services and to Foundation Grid Middleware Higher-Level Grid Services are supposed to help the users building their computing infrastructure but should not be mandatory Foundation Grid Middleware are actually developed in EGEE – Must be complete and robust – Should allow interoperation with other major grid infrastructures – Should not assume the use of Higher-Level Grid Services

36 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 35 gLite infrastructure Workload Management System (WMS) Data Management

37 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 36 Typical Job workflow JDL Logging & Book-keeping Resource Broker Job Submission Service Storage Element ComputingElement Information Service Job Status Replica Catalog Job Submit Event Input Sandbox JDL Job Input Sandbox Output Sandbox User Interface Author. Service voms-proxy-init GSI data acc/transf glite-job-submit myjob.jdl Myjob.jdl Executable = gridTest; StdError = stderr.log; StdOutput = stdout.log; InputSandbox = {/home/joda/test/gridTest}; OutputSandbox = {stderr.log, stdout.log}; InputData = lfn:testbed ; DataAccessProtocol = gridftp; Requirements = other.Architecture==INTEL && \ other.OpSys==LINUX; Rank = other.GlueHostBenchmarkSF00;

38 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 37 Security System

39 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 38 gLite Security Authentication based on X.509 PKI infrastructure – Certificate Authorities (CA) issue (long lived) certificates identifying individuals (much like a passport) – Trust between CAs and sites is established (offline) – In order to reduce vulnerability, Grid user identification is done by (short lived) proxies of their certificates Proxies can – Be delegated to a service such that it can act on the users behalf – Include additional attributes (like VO information via the VO Membership Service VOMS) – Be stored in an external proxy store (MyProxy) – Be renewed (in case they are about to expire)

40 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 39 Which CA are trusted in LCG/EGEE? The EUGridPMA is the international organization to coordinate the trust fabric for e-Science grid authentication in Europe. It collaborates with the regional peers APGridPMA for the Asia-Pacific and The Americas Grid PMA in the International Grid Trust Federation. The charter document defines the group's objective, scope and operation. It is the basis for the guidelines documents on the accreditation procedure, the Authentication profile for X.509 secured "classic" certification authorities and other IGTF recognised Profiles. In LCG/EGEE CA are installed on machine trough rpms.

41 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 40 Grid resources (A) Grid resources (B) Conventional grid security Certification Authority (CA) Bob Cert request User Interface (UI) Bob´s Grid certificate Sysadmin A : - Create user grid1 - Map Bob´s certificate to grid01 Sysadmin B : - Create user user001 - Map Bob´s certificate to user001 - Single sign-on - Delegation through proxy certificate - Manual user mapping - No info about VOs grid-proxy-init

42 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 41 gLite: VOMS Virtual Organization Membership Service (VOMS) EGEE/gLite enhancement for VO management Provides information on user's relationship with Virtual Organization (VO) Membership Group membership Roles of user Multiple VO User can register to multiple VOs and create an aggregate proxy Access ressources in every registered VO Backward compatibility Extra VO related information in users proxy certificate Users proxy can still be used with non VOMS-aware services 7 Maggio 2009 – Paolo VeronesiGriglie Computazionali - Lezione

43 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 42 gLite: VOMS - Web interface Requires a valid certificate from a recognized CA imported on the browser VO user can Query membership details Register himself in the VO Needs a valid certificate Track his requests VO manager can Handle requests from users Administer the VO Everybody can Get information about the VO

44 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 43 Grid resources (A)Grid resources (B) gLite – Enhanced security in gLite Certification Authority (CA) Bob Cert request User Interface (UI) Bob´s Grid certificate VO Database VO Service VO Manager VO membership request VO Account Pool VO Account Pool Automatic mapping for Bob Automatic mapping for Bob voms-proxy-init

45 Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 44 LCAS & LCMAPS At resources level, authorization info is extracted from the proxy and processed by LCAS and LCMAPS Local Centre Authorization Service (LCAS) – Checks if the user is authorized – Checks if the user is banned at the site Local Credential Mapping Service (LCMAPS) – Map remote credentials to local credentials (eg. different UNIX uid/gid) – Map also VOMS group and roles (full support of FQAN) enables privileges separations


Download ppt "Tecnologia dei Servizi Grid e cloud computing - Lezione 7b 0 Lezione 7b - 9 Dicembre 2009 Il materiale didattico usato in questo corso è stato mutuato."

Similar presentations


Ads by Google