Presentation is loading. Please wait.

Presentation is loading. Please wait.

Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Introduction to Linux Clusters and Grids Design and Basic Services of LCG Grid Middleware SEE-GRID Infrastructure.

Similar presentations


Presentation on theme: "Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Introduction to Linux Clusters and Grids Design and Basic Services of LCG Grid Middleware SEE-GRID Infrastructure."— Presentation transcript:

1 Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Introduction to Linux Clusters and Grids Design and Basic Services of LCG Grid Middleware SEE-GRID Infrastructure Overview Antun Balaž SCL, Institute of Physics

2 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Linux Clusters Commodity hardware become available in the last 10 years Local network Mbps easily deployed Linux mature and widely available Software available and even standardized - MPI

3 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Science and technology are team sports

4 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Unifying concept: Grid Resource sharing and coordinated problem solving in dynamic, multi- institutional virtual organizations.

5 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session What types of problems is the Grid intended to address? Too hard to keep track of authentication data (ID/password) across institutions Too hard to monitor system and application status across institutions Too many ways to submit jobs Too many ways to store & access files/data Too many ways to keep track of data Too easy to leave dangling resources lying around (robustness)

6 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Requirements Security Monitoring/Discovery Computing/Processing Power Moving and Managing Data Managing Systems System Packaging/Distribution What end users need? What end users need? Secure, reliable, on-demand access to data, software, people, and other resources (ideally all via a Web Browser!)

7 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Set of basic Grid services Job submission/management File transfer (individual, queued) Database access Data management (replication, metadata) Monitoring/Indexing system information

8 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Multi-institution issues Trust Mismatch Mechanism Mismatch Certification Authority Certification Authority Domain A Server X Server Y Policy Authority Policy Authority Task Domain B Sub-Domain A1Sub-Domain B1 No Cross- Domain Trust

9 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Why Grid security is hard Resources being used may be valuable & the problems being solved sensitive - Both users and resources need to be careful Dynamic formation and management of virtual organizations - Large, dynamic, unpredictable… VO Resources and users are often located in distinct administrative domains - Cant assume cross-organizational trust agreements - Different mechanisms & credentials

10 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Why Grid security is hard 2 Interactions are not just client/server, but service-to-service on behalf of the user - Requires delegation of rights by user to service - Services may be dynamically instantiated Standardization of interfaces to allow for discovery, negotiation and use Implementation must be broadly available & applicable - Standard, well-tested, well-understood protocols; integrated with wide variety of tools Policy from sites, VO, users need to be combined - Varying formats Want to hide as much as possible from applications!

11 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Grid solution: use of VOs Certification Domain A GSI Certification Authority Sub-Domain B1 Authority Federation Service Virtual Organization Domain No Cross- Domain Trust

12 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Effective policy governing access within a collaboration

13 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Use delegation to establish dynamic distributed system Computing Center VO Rights Computing Center Service

14 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session GSI implementation Compute Center SSL/WS-Security with Proxy Certificates VO Rights VO Users Services (running on users behalf) Rights Access Local Policy on VO identity or attribute authority CAS or VOMS issuing SAML or X.509 ACs Authz Callout KCA MyProxy

15 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Logging on to the Grid To run programs, authenticate to Grid: voms-proxy-init –voms VONAME Enter PEM pass phrase: *************** Creates a temporary, local, short-lived proxy credential for use by our computations Delegation = remote creation of a (second level) proxy credential, which allows remote process to authenticate on behalf of the user

16 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training SessionMiddleware LCG: Large Hadron Collider Computing Grid LCG infrastructure running LCG-2 is EGEE-0 In parallel producing new web-service-oriented middleware (gLite), which will replace LCG-2 as production facility this year Globus 2 basedWeb services based EGEE-2EGEE-1LCG-2LCG-1

17 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session User view of the Grid User Interface Grid services User Interface

18 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session What really happens ReplicaCatalogue Logging & Book-keeping ResourceBrokerStorageElementComputingElement InformationService Job Status DataSets info Auth. &Auth. Job Submit Event Job Query Job Status Input sandbox Input sandbox + Broker Info Output sandbox Output sandbox Publish SE & CE info User interface

19 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Workload Management System (WMS) Distributed scheduling multiple UIs where you can submit your job multiple RBs from where the job can be sent to a CE multiple CEs where the job can be put in a queuing system Distributed resource management multiple information systems that monitor the state of the grid Information from SE, CE, sites

20 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Authentication and Authorization Authentication User obtains certificate from CA Connects to UI by ssh Downloads certificate Invokes Proxy server Single logon – to UI - then Secure Socket Layer with proxy identifies user to other nodes Authorization - currently User joins Virtual Organisation VO negotiates access to Grid nodes and resources (CE, SE) Authorization tested by CE, SE: gridmapfile maps user to local account

21 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session User Interface (UI) UI is the users interface to the Grid - Command-line interface to Proxy server Job operations To submit a job Monitor its status Retrieve output Data operations Upload file to SE Create replica Discover replicas Other grid services To run a job user creates a JDL (Job Description Language) file

22 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Computing Element (CE) Homogeneous set of worker nodes Grid gate node Local resource management system: Condor / PBS / LSF master Gatekeeper Job request Info system Logging gridmapfile I.S. Logging A CE is a grid batch queue with a grid gate front-end:

23 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Storage Element (SE) Storage elements hold files: write once, read many Replica files can be held on different SE: close to CE; share load on SE Replica Catalogue - what replicas exist for a file? Replica Location Service - where are they? Local Info Event Logging gridmapfile GridFTP Disk arrays or tapes Info system Logging Gatekeeper File transferRequests

24 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Resource Broker Run the Workload Management System To accept job submissions Dispatch jobs to appropriate Compute Element (CE) Allow users To get information about their status To retrieve their output A configuration file on each UI node determines which RB node(s) will be used When a user submits a job, JDL options are to: Specify CE Allow RB to choose CE (using optional tags to define requirements) Specify SE (then RB finds nearest appropriate CE, after interrogating Replica Location Service)

25 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Logging and Bookkeeping Who did what and when? Whats happening to my job? Usually runs on RB node Information System Receives periodic (~5 min) updates from CE, SE Used by RB node to determine resources to be used by a job Currently BDII is used

26 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session What have we learn so far? Grid structure is complicated but hidden from end-users, enabling all the comfort they need Users just need to join the VO and obtain certificates: we already have the SEE-GRID VO! Use of Grid is then just as easy as the use of a computer cluster

27 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session SEE-GRID Overview SEE-GRID is EU FP6 project, involving 11 partners from 11 European countries: Greece, Switzerland, Bulgaria, Romania, Turkey, Hungary, Albania, Bosnia and Herzegovina, FYR of Macedonia, Serbia and Montenegro, Croatia Each partner collaborates with one or more 3 rd parties Project started in May 2004, lasts 2 years, SEE-GRID-2 on its way

28 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session SEE-GRID Objectives (1) Human network in the area of grid computing eScience and eInfrastructures Integrate incubating and existing National Grid infrastructures in all SEE-GRID countries Ease the digital divide and bring SEE Grid communities closer to the rest of the continent

29 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session SEE-GRID Objectives (2) Establish a dialogue at the level of policy developments for research and education networking and provide input to the agenda of national governments and funding bodies Promote awareness in the region regarding Grid developments through dissemination conferences, training material and demonstrations for hands-on experience Migrate and test Grid middleware components and APIs developed by pan-European and national Grid efforts in the regional infrastructure

30 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session SEE-GRID Objectives (3) Deploy (adapt if necessary) and test Grid applications developed by EGEE Demonstrate an additional Grid application of regional interest Integrate available pilot Resource Centres of Albania, Bosnia-Herzegovina, Croatia, FYR of Macedonia, Serbia-Montenegro and Turkey into the EGEE- compatible infrastructure Expand the operations and support centre of the EGEE SE Europe Federation to cater for the operations in the above countries

31 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session SEE-GRID Infrastructure Overview (1) At least one SEE-GRID site per country, (currently 15+1!), each deploying CE, SE, MON, UI, and a number of WNs SEE-GRID regional services: SEE-GRID CA (Greece) RB and BDII (Turkey + Serbia and Montenegro) VOMS (Croatia) R-GMA (Bulgaria) SFTs and GridICE (FYR of Macedonia) P-GRADE portal (Hungary) MYProxy (Greece + Serbia and Montegro) LFC (Serbia and Montenegro)

32 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session SEE-GRID Infrastructure Overview (2) SEE-GRID applications: SE4SEE (Turkey) VIVE (Serbia and Montenegro) Technical Forum (Hungary) SEE-GRID Web site and WIKI (Greece) Infrastructure mailing list: Strong human network

33 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Hands-on Plan Hands-on I: UI Installation and Configuration Hands-on II: Certificates, Proxies, Test Jobs TOMORROW: Hands-on III: Composing the site-info.def file Hands-on IV: UI/CE Installation and Configuration Hands-on V: SE/MON Installation and Configuration Hands-on VI: WNs Installation and Configuration Hands-on VII: Testing and SEE-GRID Tuning

34 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Hands-on II: Certificates, Proxies, Test Jobs

35 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Grid Certificates Each user must have a valid X.509 certificate issued by a recognized Certification Authority (CA) Before doing any Grid operation, user must log in to User Interface (UI) machine and create a proxy certificate. A proxy certificate is a delegated user credential that authenticates the user in every secure interaction, and has a limited lifetime: in fact, it prevents having to use one's own certificate, which could compromise its safety voms-proxy-init –voms VONAME Voms-proxy-info; voms-proxy-destroy

36 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Job Submission (1) User have to create a file describing the submitted job in Job Description Language (JDL) User submits jobs to Resource Broker (RB) JDL for simple test job: antun]$ cat test.jdl Executable = "/bin/hostname"; StdOutput = "std.out"; StdError = "std.err"; OutputSandbox = {"std.out","std.err"};

37 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Job Submission (2) edg-job-list-match test.jdl edg-job-submit test.jdl edg-job-status JobID edg-job-cancel JobID edg-job-get-output JobID edg-job-get-logging-info JobID Bypassing RB: globus-job-run CE command

38 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Using myproxy server Myproxy server is used for Very long jobs (that normal proxy may be expired) Getting proxy on other machines than UI (typical for portals) myproxy-init –s MYPROXYSERVER myproxy-get-delegation myproxy-info myproxy-destroy

39 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session In a nutshell voms-proxy-init –voms VONAME edg-job-submit job.jdl edg-job-status JobID edg-job-get-output JobID

40 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Monitoring, SEE-GRID SFTs and GridICE (1) Qstat, showq, pbsnodes on CE Ldapsearch of GIISes: ldapsearch -x -h -p b mds-vo- name=local,o=grid ldapsearch -x -h -p b mds-vo-name=,o=grid ldapsearch -x -h -p b o=grid Useful entries: GlueCEUniqueID, GlueSEUniqueID, GlueSEName, GlueCESEBindSEUniqueID

41 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Monitoring, SEE-GRID SFTs and GridICE (2) For some grid components there are custom checking tools, e.g. rgma-client- check ps on all nodes – do not forget about excellent ps! Submitting test jobs SEE-GRID GStat

42 A E G I S Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Monitoring, SEE-GRID SFTs and GridICE (3) SEE-GRID SFTs SEE-GRID GridICE Real Time Monitor


Download ppt "Nov 5-6, 2005 SEE-GRID Banjaluka Training Session Introduction to Linux Clusters and Grids Design and Basic Services of LCG Grid Middleware SEE-GRID Infrastructure."

Similar presentations


Ads by Google