EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks GINGIN Grid Interoperation on Data Movement between NAREGI and EGEE gLite Hideo MATSUDA 1,2, Yoshiyuki KIDO 3,2, Kentaro WAKATSUKI 4 1 NAREGI, 2 Osaka University, 3 Mitsui Knowledge Industry Co.,Ltd. 4 Hitachi Software Engineering Co.,Ltd.

Enabling Grids for E-sciencE EGEE-II INFSO-RI GINGIN To change: View -> Header and Footer 2 GIN (Grid Interoperation Now) An activity of OGF for interoperation among production grids Major grid projects are participating –EGEE, NAREGI, UK National Grid Service, NorduGrid, OSG, PRAGMA, TeraGrid,... Trying to identify islands of interoperation between production grids and grow those islands Areas –GIN-auth: Authorization and Identity Management –GIN-data: Data Management and Movement –GIN-jobs: Job Description and Submission –GIN-info: Information Services and Schema –GIN-ops: Operations Experience of Pilot Test Applications

Enabling Grids for E-sciencE EGEE-II INFSO-RI GINGIN To change: View -> Header and Footer 3 NAREGI GIN Activities Developing an interoperation island with EGEE –GIN-jobs, GIN-auth, GIN-info, and GIN-data

Enabling Grids for E-sciencE EGEE-II INFSO-RI GINGIN To change: View -> Header and Footer 4 Architecture SC06 Demo NAREGI  EGEE: using NAREGI Workflow EGEE  NAREGI: using gLite WMS commands EGEE userNAREGI user gLite-WMSgLite-BDIINAREGI-ISGIN-BDII lcgCE PreWS-GRAM gLite-UI NAREGI Portal Computing Resource NAREGI GridVM WS GRAM gliteCE NAREGI-GAHP NAREGI Client Lib NAREGI-SS NAREGI-SC Interop-SC GIN-jobs: NAREGI-EGEE Architecture & SC06 Demo

Enabling Grids for E-sciencE EGEE-II INFSO-RI GINGIN To change: View -> Header and Footer 5 GIN-auth: Authentication IGTF is framework of International Grid Trust Federation. IGTF consists of APGridPMA, EUGridPMA and TAGPMA. NAREGI CA joined the APGrid PMA. NAREGI CA has been approved as a production-level CA by APGridPMA. EUGridPMATAGPMA APGridPMA NAREGI PMA IGTF (International Grid Trust Federation) GSI compliant with x.509 proxy certificates for authentication. It has become available to use grid computing easily on the worldwide Internet by IGTF.

Enabling Grids for E-sciencE EGEE-II INFSO-RI GINGIN To change: View -> Header and Footer 6 All of grid information can be retrieved by each of grid in its fashion WRT resource description schema, data format, query language, client API, … Each information service in grid acts as an information provider for the other and translator embedded in the provider performs conversion between different schemas. Generic Information Provider GIN-BDII EGEE OSG NDGF NAREGITeraGrid Pragma Cell Domain connecting with BDII LRPS OS Processor Storage CIM Providers with Glue=>NRG translator ● ●● ● JobQueue Service OGSA -DAI Aggregator RDB CIM v2.12 /w ext. LDIF xmlCIM ARC -BDII Glue v1.2 NAREGI TeraGrid/ MDS4 Glue v1.1 ARC LDIF providers with X  Glue translators : “Site on a map” GIN-info: Architecture

Enabling Grids for E-sciencE EGEE-II INFSO-RI GINGIN To change: View -> Header and Footer 7 GIN-data Data Management and MovementData Management and Movement Agreements: –Grid FTP is the lowest common denominator for file transfer –SRM and SRB islands for data management are being established

Enabling Grids for E-sciencE EGEE-II INFSO-RI GINGIN To change: View -> Header and Footer 8 NAREGI Software and Data Grid RISM Job Local Scheduler GridVM Local Scheduler IMPI Server GridVM FMO Job Local Scheduler GridVM Super Scheduler WFT Input files Work- flow Data Grid CA Site ρ Site αSite μ 3: Negotiation Agreement 6: MPI job starts 9: Accounting 2: Monitoring 4: Reservation 5: IMPI starts c: Edit b2: Data import 2: Resource discovery Information Service Co-Allocation 8: Visualization 1: Submission Application requirement definition a: Sign-on Portal 7: MPI init. GridMPI Data Grid Gfarm File System Network monitor

Enabling Grids for E-sciencE EGEE-II INFSO-RI GINGIN To change: View -> Header and Footer 9 Gfarm File System (1) Developed by AIST, Japan. Commodity-based distributed file system that federates local disks of compute nodes It can be shared among all cluster nodes and clients –Just mount it as if it were high-performance NFS It provides scalable I/O performance wrt the number of parallel processes and users It supports fault tolerance and avoids access concentration by automatic replica selection

Enabling Grids for E-sciencE EGEE-II INFSO-RI GINGIN To change: View -> Header and Footer 10 Gfarm File System (2) Files can be shared among all nodes and clients Physically, it may be replicated and stored on any file system node Applications can access it regardless of its location File system nodes can be distributed GridFTP, samba, NFS server Compute & fs node GridFTP, samba, NFS server Gfarm metadata server Compute & fs node Client PC Note PC /gfarm metadata Gfarm file system … File A File B File C File A File B File C File B EU Japan

Enabling Grids for E-sciencE EGEE-II INFSO-RI GINGIN To change: View -> Header and Footer 11 GIN-data: Architecture GridFTP Server EGEE gLite Client SRM Client NAREGI NAREGI Client SRM Client Gfarm API NAREGI Metadata Server LFC (Metadata Server) Gfarm Server DPM (SRM Server) Storage NAREGI and EGEE gLite clients can access to both data resources (e.g., bi-directional file copy) using SRM interface. GridFTP is used as its underlying file transfer protocol. File catalog (metadata) exchange is planned.

Enabling Grids for E-sciencE EGEE-II INFSO-RI GINGIN To change: View -> Header and Footer 12 GIN-data: File Transfer with GridFTP-DSI It is not easy to bridge between different file access protocols SRM (gLite) and Gfarm. As the 1st step, bi-directional file transfer between gLite and Gfarm using GridFTP. Problem: GridFTP authentication (delegation) cannot be directly passed to Gfarm file server. GridFTP-DSI (Data Storage Interface) has been used for integrating Gfarm API into GridFTP. Gfarm metadata sever Gfarm file system GridFTP client Gfarm file server GridFTP server DSI for Gfarm Gfarm API Gfarm client lib Proxy cert export Proxy cert GridFTP server for Gfarm access

Enabling Grids for E-sciencE EGEE-II INFSO-RI GINGIN To change: View -> Header and Footer 13 NAREGI (pbg1052.naregi.org) GridFTP server for Gfarm access EGEE (lxdpm01.cern.ch) srmcp gsiftp://pbg1052 srm://lxdpm01 GIN-data: SC06 Demo SRM copy (srmcp) command was ported in NAREGI. Bi-directional file transfer can be performed using GridFTP with DSI for Gfarm. SRM client SRM (DPM) Server Gfarm Server srmcp srm://lxdpm01 gsiftp://pbg1052 Gfarm DSI

Enabling Grids for E-sciencE EGEE-II INFSO-RI GINGIN To change: View -> Header and Footer 14 Summary NAREGI developed EGEE-NAREGI island as an activity of GIN –Bilateral information exchange –Bilateral job submission –Bilateral file exchange –Interoperable security properties Next steps –Improve interoperation interfaces and functions WS-GRAM, BES, JSDL, … –Grow the island with other EGEE partners –KEK will use NAREGI-EGEE interoperation environment for their high energy physics calculations