NAREGI WP4 (Data Grid Environment) Hideo Matsuda Osaka University.

Slides:

Advertisements

Similar presentations

X-SIGMA (An XML based Simple data Integration system for Gathering, Managing and Accessing scientific experimental data in grid environments) Karpjoo

Advertisements

Motivation 1.Application resources setup – make it easy 2.Transform PRAGMA grid – add on demand –Continue using Grid resources –Add cloud resources Current.

GPE4UNICORE Grid Programming Environment for UNICORE

© 2008 Open Grid Forum Data Grid Federation by RNS GFS-WG, OGF23 Balcelona Hideo Matsuda Osaka University / NAREGI.

© 2007 Open Grid Forum Data Management Challenge - The View from OGF OGF22 – February 28, 2008 Cambridge, MA, USA Erwin Laure David E. Martin Data Area.

Data Management Expert Panel. RLS Globus-EDG Replica Location Service u Joint Design in the form of the Giggle architecture u Reference Implementation.

FP7-INFRA Enabling Grids for E-sciencE EGEE Induction Grid training for users, Institute of Physics Belgrade, Serbia Sep. 19, 2008.

C. Grimme, A. Papaspyrou Scheduling in C3-Grid AstroGrid-D Workshop Project: C3-Grid Collaborative Climate Community Data and Processing Grid Scheduling.

Data Grid: Storage Resource Broker Mike Smorul. SRB Overview Developed at San Diego Supercomputing Center. Provides the abstraction mechanisms needed.

Adopting and Setting Standards The NAREGI Strategy Satoshi Matsuoka Professor, Global Scientific Information and Computing Center, Deputy Director, NAREGI.

Holding slide prior to starting show. Supporting Collaborative Working of Construction Industry Consortia via the Grid - P. Burnap, L. Joita, J.S. Pahwa,

MTA SZTAKI Hungarian Academy of Sciences Grid Computing Course Porto, January Introduction to Grid portals Gergely Sipos

Seminar Grid Computing ‘05 Hui Li Sep 19, Overview Brief Introduction Presentations Projects Remarks.

USING THE GLOBUS TOOLKIT This summary by: Asad Samar / CALTECH/CMS Ben Segal / CERN-IT FULL INFO AT:

The Globus Toolkit Gary Jackson. Introduction The Globus Toolkit is a product of the Globus Alliance ( It is middleware for developing.

GGF Toronto Spitfire A Relational DB Service for the Grid Peter Z. Kunszt European DataGrid Data Management CERN Database Group.

Data Grids: Globus vs SRB. Maturity SRB  Older code base  Widely accepted across multiple communities  Core components are tightly integrated Globus.

4b.1 Grid Computing Software Components of Globus 4.0 ITCS 4010 Grid Computing, 2005, UNC-Charlotte, B. Wilkinson, slides 4b.

UMIACS PAWN, LPE, and GRASP data grids Mike Smorul.

Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.

Data Grid Web Services Chip Watson Jie Chen, Ying Chen, Bryan Hess, Walt Akers.

Makrand Siddhabhatti Tata Institute of Fundamental Research Mumbai 17 Aug

EGEE-II INFSO-RI Enabling Grids for E-sciencE EGEE and gLite are registered trademarks GINGIN Grid Interoperation on Data Movement.

SUN HPC Consortium, Heidelberg 2004 Grid(Lab) Resource Management System (GRMS) and GridLab Services Krzysztof Kurowski Poznan Supercomputing and Networking.

- 1 - Grid Programming Environment (GPE) Ralf Ratering Intel Parallel and Distributed Solutions Division (PDSD)

OSG End User Tools Overview OSG Grid school – March 19, 2009 Marco Mambelli - University of Chicago A brief summary about the system.

Grid security in NAREGI project NAREGI the Japanese national science grid project is doing research and development of grid middleware to create e- Science.

Grid ASP Portals and the Grid PSE Builder Satoshi Itoh GTRC, AIST 3rd Oct UK & Japan N+N Meeting Takeshi Nishikawa Naotaka Yamamoto Hiroshi Takemiya.

Grid security in NAREGI project July 19, 2006 National Institute of Informatics, Japan Shinichi Mineo APAN Grid-Middleware Workshop 2006.

Holding slide prior to starting show. A Grid-based Problem Solving Environment for GECEM Maria Lin and David Walker Cardiff University Yu Chen and Jason.

The Japanese Virtual Observatory (JVO) Yuji Shirasaki National Astronomical Observatory of Japan.

Data Management Kelly Clynes Caitlin Minteer. Agenda Globus Toolkit Basic Data Management Systems Overview of Data Management Data Movement Grid FTP Reliable.

WP9 Resource Management Current status and plans for future Juliusz Pukacki Krzysztof Kurowski Poznan Supercomputing.

GT Components. Globus Toolkit A “toolkit” of services and packages for creating the basic grid computing infrastructure Higher level tools added to this.

Grids and Portals for VLAB Marlon Pierce Community Grids Lab Indiana University.

ESP workshop, Sept 2003 the Earth System Grid data portal presented by Luca Cinquini (NCAR/SCD/VETS) Acknowledgments: ESG.

Grid Resource Allocation and Management (GRAM) Execution management Execution management –Deployment, scheduling and monitoring Community Scheduler Framework.

GT4 as an Implementation Kernel for NAREGI Grid MW beta 1 Satoshi Matsuoka Professor, Global Scientific Information and Computing Center, Deputy Director,

ILDG Middleware Status Chip Watson ILDG-6 Workshop May 12, 2005.

CYBERINFRASTRUCTURE FOR THE GEOSCIENCES Data Replication Service Sandeep Chandra GEON Systems Group San Diego Supercomputer Center.

OGSA-DAI in OMII-Europe Neil Chue Hong EPCC, University of Edinburgh.

D C a c h e Michael Ernst Patrick Fuhrmann Tigran Mkrtchyan d C a c h e M. Ernst, P. Fuhrmann, T. Mkrtchyan Chep 2003 Chep2003 UCSD, California.

Resource Brokering in the PROGRESS Project Juliusz Pukacki Grid Resource Management Workshop, October 2003.

1 1 EPCC 2 Curtin Business School & Edinburgh University Management School Michael J. Jackson 1 Ashley D. Lloyd 2 Terence M. Sloan 1 Enabling Access to.

Author - Title- Date - n° 1 Partner Logo WP5 Summary Paris John Gordon WP5 6th March 2002.

Globus Replica Management Bill Allcock, ANL PPDG Meeting at SLAC 20 Sep 2000.

 CASTORFS web page - CASTOR web site - FUSE web site -

The Replica Location Service The Globus Project™ And The DataGrid Project Copyright (c) 2002 University of Chicago and The University of Southern California.

NA-MIC National Alliance for Medical Image Computing UCSD: Engineering Core 2 Portal and Grid Infrastructure.

GO-ESSP Workshop, LLNL, Livermore, CA, Jun 19-21, 2006, Center for ATmosphere sciences and Earthquake Researches Construction of e-science Environment.

Web Portal Design Workshop, Boulder (CO), Jan 2003 Luca Cinquini (NCAR, ESG) The ESG and NCAR Web Portals Luca Cinquini NCAR, ESG Outline: 1.ESG Data Services.

Japanese Virtual Observatory Project Abstract : The National Astronomical Observatory of Japan (NAOJ) started the Japanese Virtual Observatory (JVO) project.

MTA SZTAKI Hungarian Academy of Sciences Introduction to Grid portals Gergely Sipos

GRID Overview Internet2 Member Meeting Spring 2003 Sandra Redman Information Technology and Systems Center and Information Technology Research Center National.

CGW 04, Stripped replication for the grid environment as a web service1 Stripped replication for the Grid environment as a web service Marek Ciglan, Ondrej.

The Global Land Cover Facility is sponsored by NASA and the University of Maryland.The GLCF is a founding member of the Federation of Earth Science Information.

© 2008 Open Grid Forum File Catalog Development in Japan e-Science Project GFS-WG, OGF24 Singapore Hideo Matsuda Osaka University.

Replica Management Kelly Clynes. Agenda Grid Computing Globus Toolkit What is Replica Management Replica Management in Globus Replica Management Catalog.

1 e-Science AHM st Aug – 3 rd Sept 2004 Nottingham Distributed Storage management using SRB on UK National Grid Service Manandhar A, Haines K,

Globus: A Report. Introduction What is Globus? Need for Globus. Goal of Globus Approach used by Globus: –Develop High level tools and basic technologies.

Rights Management for Shared Collections Storage Resource Broker Reagan W. Moore

Data Breakout. OGSA Architecture – databases Eldas, OGSA-DAI and GridMiner implement a slightly old version of OGSA / DAIS –Architecture doc describes.

The Roadmap of NAREGI Security Services Masataka Kanamori NAREGI WP

ACGT Architecture and Grid Infrastructure Juliusz Pukacki ‏ EGEE Conference Budapest, 4 October 2007.

2 nd EGEE/OSG Workshop Data Management in Production Grids 2 nd of series of EGEE/OSG workshops – 1 st on security at HPDC 2006 (Paris) Goal: open discussion.

NAREGI PSE with ACS S.Kawata 1, H.Usami 2, M.Yamada 3, Y.Miyahara 3, Y.Hayase 4 1 Utsunomiya University 2 National Institute of Informatics 3 FUJITSU Limited.

Data services on the NGS

Data Bridge Solving diverse data access in scientific applications

Thank you for your kind introduction.

Data services in gLite “s” gLite and LCG.

Presentation transcript:

NAREGI WP4 (Data Grid Environment) Hideo Matsuda Osaka University

2 NAREGI Software Stack (Beta Ver. 2006) Computing Resources and Virtual Organizations NII IMS Research Organizations Major University Computing Centers （ WSRF (GT4+Fujitsu WP1) + GT4 and other services) SuperSINET Grid-Enabled Nano-Applications (WP6) Grid PSE Grid Programming (WP2) -Grid RPC -Grid MPI Grid Visualization Grid VM (WP1) Packaging Distributed Information Service (CIM) Grid Workflow (WFML (Unicore+ WF)) Super Scheduler Grid Security and High-Performance Grid Networking (WP5) Data (WP4) WP1 WP3 WP4

3 NAREGI Data Grid Overview Composed of three parts Data Access Management System Metadata Management System Data Resource Management System Functionalities Grid File System using AIST Gfarm –Logical to physical fIle name mapping, File replication, etc File Import to WFT (Workflow Tool) File Staging Support to SS (Super Scheduler) File Retrieval by Metadata –For storing file attributes (owner, created date & condition (what application creates the file, etc.). SQL access to Metadata DB (OGSA-DA)

4 Grid File System (AIST Gfarm) Metadata Management WFT User Fileserver#1 Fileserver#2 Fileserver#3 File Mapping SS Metadata DB Data Access Management Data Resource Management File1 File2 File3 Computation node Job File Computation node Job File File Staging or File Access NAREGI Data Grid Architecture (beta 1) File Import

5 Data 1 Data 2 Data n Grid-wide File System Metadata Management Data Access Management Data Resource Management Job 1 Meta- data Meta- data Data 1 Grid Workflow Data 2Data n NAREGI Data Grid Functionalities Job 2Job n Meta- data Job 1 Grid-wide Data Sharing Service Job 2 Job n Data Grid Components Import data into workflow Place & register data on the Grid Assign metadata to data Store data into distributed file nodes

6 Data 1 Data 2Data n Gfarm 1.2 PL4 (Grid FS) Data Access Management NAREGI WP4: Standards Employed in the Architecture Job 1 Data Specific Metadata DB Data 1Data n Job 1Job n Import data into workflow Job 2 Computational Nodes Filesystem Nodes Job n Data Resource Information DB OGSA-DAI WSRF2.0 OGSA-DAI WSRF2.0 Globus Toolkit Tomcat PostgreSQL 8.0 Workflow (NAREGI WFML =>BPEL+JSDL) Super Scheduler (SS) (OGSA-RSS) Data Staging Place data on the Grid Data Resource Management Metadata Construction OGSA-RSS FTS SC GridFTP

7 Grid File System Adoption of AIST Gfarm UNIX file I/O API (open, close, read, write, etc.), and file/directory operations (mkdir, rmdir, ls, etc.). Single Virtual Filesystem (Logical to Physical File Name Mapping) Client library, File servers, Metadata server (mapping & space balancing) Mapping “/gfarm/file1” to “HOST:/GFARM_DIR/file1” Extensible file space by adding many file servers File replication to user-specified hosts. File Metadata (management with PostgreSQL) File location (host, pathname), Username, Access/ Modify/ Creation dates, Size Access Security with GT4 GSI API Certificate-based file access, Access using gridmapfile WSRF interface to File Metadata with OGSA-DAI Access Control not yet implemented (planned in Gfarm 2.0)

8 File Import & Staging Import files in GFS (Grid File System) to user workflows. Imported files can be used from jobs in their local disks (file staging). SS transfers GFS files to Job nodes.

9 File Metadata Management Many applications (or many instances of the same application with different. parameters) can be coupled in a workflow. Need many input and output files of the applications. This system provides a metadata DB storing application and parameter information as file attributes. User can retrieve files on GFS by their metadata.

10 Computation Node Local Disk GridFTP Server GridVM Job Gfarm Client Super Scheduler (SS) Node SS File Staging Tool Portal Server Node Gfarm Staging Node Workflow Tool (WFT) Node WFT Metadata Node PostgreSQL Metadata DB User Client OGSA- DAI Client Metadata Management Frontend Data Access Management Frontend File-Import Tool Data Resource Management Frontend OGSA- DAI Client Workflow Tool Client Data Access Management Client OGSA-DAI Gfarm Node Local Disk Gfarm Server Gfarm Node Local Disk Gfarm Server Gfarm Client GRAM4 FIle Staging / import Interface Local Disk GridFTP Server OGSA- DAI PostgreSQL Gfarm DB

11 EGEE-NAREGI Interoperation Issues in Data Management Data Transfer –both groups use GridFTP ? Storage Element –NAREGI does not have this. Adoption of SRMv2.1 ? or SRB? Replication Catalog –NAREGI only provides simple file replication (flat table). Adoption of LFC? or RLS? Metadata Catalog –NAREGI Metadata Schema & Structure are not fixed yet. Again it is just a flat table. Adoption ?