The Global Land Cover Facility is sponsored by NASA and the University of Maryland.The GLCF is a founding member of the Federation of Earth Science Information.

Slides:



Advertisements
Similar presentations
National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center Data Grids for Collection Federation Reagan W. Moore University.
Advertisements

The Storage Resource Broker and.
The Storage Resource Broker and.
Peter Berrisford RAL – Data Management Group SRB Services.
Chapter 10: Designing Databases
Data Grid: Storage Resource Broker Mike Smorul. SRB Overview Developed at San Diego Supercomputing Center. Provides the abstraction mechanisms needed.
NATIONAL PARTNERSHIP FOR ADVANCED COMPUTATIONAL INFRASTRUCTURE SAN DIEGO SUPERCOMPUTER CENTER Particle Physics Data Grid PPDG Data Handling System Reagan.
Background Chronopolis Goals Data Grid supporting a Long-term Preservation Service Data Migration Data Migration to next generation technologies Trust.
A Very Brief Introduction to iRODS
Chronopolis: Preserving Our Digital Heritage David Minor UC San Diego San Diego Supercomputer Center.
Chapter 2 Database Environment.
Office of Science U.S. Department of Energy Grids and Portals at NERSC Presented by Steve Chan.
Supporting Customized Archival Practices Using the Producer-Archive Workflow Network (PAWN) Mike Smorul, Mike McGann, Joseph JaJa.
On Developing Data Grid Workflows using Storage Resource Broker (SRB) and Kepler Tim H. Wong - UC Davis Efrat Frank - SDSC Bertram Ludäscher - UC Davis.
Robust Tools for Archiving and Preserving Digital Data Joseph JaJa, Mike Smorul, and Mike McGann Institute for Advanced Computer Studies Department of.
Mike Smorul Saurabh Channan Digital Preservation and Archiving at the Institute for Advanced Computer Studies University of Maryland, College Park.
UMIACS PAWN, LPE, and GRASP data grids Mike Smorul.
Data Grid: GRASP Mike Smorul. Grid Retrieval and Search Platform Based on concepts developed in the Earth Science Data Interface (ESDI) developed at the.
On Developing Data Grid Workflows using Storage Resource Broker (SRB) and Kepler Tim H. Wong - UC Davis Efrat Frank - SDSC Dr. Bertram Ludäscher - UC Davis.
Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.
Database Environment 1.  Purpose of three-level database architecture.  Contents of external, conceptual, and internal levels.  Purpose of external/conceptual.
Databases Illuminated
San Diego Supercomputer CenterUniversity of California, San Diego Preservation Research Roadmap Reagan W. Moore San Diego Supercomputer Center
CI Days: Planning Your Campus Cyberinfrastructure Strategy Russ Hobby, Internet2 Internet2 Member Meeting 9 October 2007.
Data Management Kelly Clynes Caitlin Minteer. Agenda Globus Toolkit Basic Data Management Systems Overview of Data Management Data Movement Grid FTP Reliable.
Jan Storage Resource Broker Managing Distributed Data in a Grid A discussion of a paper published by a group of researchers at the San Diego Supercomputer.
 DATABASE DATABASE  DATABASE ENVIRONMENT DATABASE ENVIRONMENT  WHY STUDY DATABASE WHY STUDY DATABASE  DBMS & ITS FUNCTIONS DBMS & ITS FUNCTIONS 
CORE 2: Information systems and Databases CENTRALISED AND DISTRIBUTED DATABASES.
1 School of Computer, National University of Defense Technology A Profile on the Grid Data Engine (GridDaEn) Xiao Nong
Global Land Cover Facility The Global Land Cover Facility (GLCF) is a member of the Earth Science Information Partnership (ESIP) Federation providing data,
The Data Grid: Towards an Architecture for the Distributed Management and Analysis of Large Scientific Dataset Caitlin Minteer & Kelly Clynes.
ESP workshop, Sept 2003 the Earth System Grid data portal presented by Luca Cinquini (NCAR/SCD/VETS) Acknowledgments: ESG.
San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center National Partnership for Advanced.
Production Data Grids SRB - iRODS Storage Resource Broker Reagan W. Moore
IPlant Collaborative Tools and Services Workshop iPlant Collaborative Tools and Services Workshop Collaborating with iPlant.
The PROGRESS Grid Service Provider Maciej Bogdański Portals & Portlets 2003 Edinburgh, July 14th-17th.
CYBERINFRASTRUCTURE FOR THE GEOSCIENCES Data Replication Service Sandeep Chandra GEON Systems Group San Diego Supercomputer Center.
Introduction to dCache Zhenping (Jane) Liu ATLAS Computing Facility, Physics Department Brookhaven National Lab 09/12 – 09/13, 2005 USATLAS Tier-1 & Tier-2.
Middleware for Grid Computing and the relationship to Middleware at large ECE 1770 : Middleware Systems By: Sepehr (Sep) Seyedi Date: Thurs. January 23,
National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center Persistent Management of Distributed Data Reagan W. Moore.
National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center Persistent Archive for the NSDL Reagan W. Moore Charlie Cowart.
GEON2 and OpenEarth Framework (OEF) Bradley Wallet School of Geology and Geophysics, University of Oklahoma
GRID Overview Internet2 Member Meeting Spring 2003 Sandra Redman Information Technology and Systems Center and Information Technology Research Center National.
May 6, 2002Earth System Grid - Williams The Earth System Grid Presented by Dean N. Williams PI’s: Ian Foster (ANL); Don Middleton (NCAR); and Dean Williams.
Michael Doherty RAL UK e-Science AHM 2-4 September 2003 SRB in Action.
Topic Distributed DBMS Database Management Systems Fall 2012 Presented by: Osama Ben Omran.
Introduction to The Storage Resource.
7. Grid Computing Systems and Resource Management
Distributed Data for Science Workflows Data Architecture Progress Report December 2008.
Cyberinfrastructure Overview Russ Hobby, Internet2 ECSU CI Days 4 January 2008.
Object storage and object interoperability
Biomedical Informatics Research Network The Storage Resource Broker & Integration with NMI Middleware Arcot Rajasekar, BIRN-CC SDSC October 9th 2002 BIRN.
The Storage Resource Broker and.
Presentation on Database management Submitted To: Prof: Rutvi Sarang Submitted By: Dharmishtha A. Baria Roll:No:1(sem-3)
ETICS An Environment for Distributed Software Development in Aerospace Applications SpaceTransfer09 Hannover Messe, April 2009.
Building Preservation Environments with Data Grid Technology Reagan W. Moore Presenter: Praveen Namburi.
The Virtual Observatory and Ecological Informatics System (VOEIS): Using RESTful architecture and an extensible data model to provide a unique data management.
Collection-Based Persistent Archives Arcot Rajasekar, Richard Marciano, Reagan Moore San Diego Supercomputer Center Presented by: Preetham A Gowda.
IT 5433 LM1. Learning Objectives Understand key terms in database Explain file processing systems List parts of a database environment Explain types of.
Data Infrastructure in the TeraGrid Chris Jordan Campus Champions Presentation May 6, 2009.
1 Copyright © 2008, Oracle. All rights reserved. Repository Basics.
Data Grids, Digital Libraries and Persistent Archives: An Integrated Approach to Publishing, Sharing and Archiving Data. Written By: R. Moore, A. Rajasekar,
What are they? The Package Repository Client is a set of Tcl scripts that are capable of locating, downloading, and installing packages for both Tcl and.
The Data Grid: Towards an architecture for Distributed Management
Problem: Ecological data needed to address critical questions are dispersed, heterogeneous, and complex Solution: An internet-based mechanism to discover,
OGSA Data Architecture Scenarios
Arcot Rajasekar Michael Wan Reagan Moore (sekar, mwan,
Database Environment Transparencies
Presentation transcript:

The Global Land Cover Facility is sponsored by NASA and the University of Maryland.The GLCF is a founding member of the Federation of Earth Science Information Partners. The Global Land Cover Facility Integrating Earth Science Data With Grid Technologies ProblemSolution Storage Resource Broker (SRB) SRB Architecture The Earth Science community generates large quantities of scientific data and being able to bring and use these data together presents a difficult challenge. Data are typically stored at many small, distributed repositories which have different access methods and query mechanisms. A typical workflow involves downloading all of the required data before processing or to work with data in iteration using small and manageable batches. By using data grid technologies, data can be viewed as belonging to one large virtual repository. Data access and retrieval is performed by referencing the data itself in a uniform namespace, independent of the data’s physical location and underlying storage format and medium. Data can be obtained from the grid when needed and services required to access the data grid can be provided through one access method. This simplifies working with data in a distributed environment. Through funding from a SEEDS grant, the University of Maryland, University of New Hampshire, and George Mason University are working together to create a data grid test-bed using the Storage Resource Broker (SRB) developed by the San Diego Supercomputer Center. Data is registered and organized in the SRB as objects and collections. Files are considered objects, the fundamental storage item. These objects then can be grouped together using collections and collections can be grouped together in another collection. In addition to these constructs, containers can be used to group together small files that should be read and written as one unit. For example, thumbnails and metadata files for a remote-sensing imagery can be placed in a container since these are typically retrieved together when requested. The figure on the left shows how Landsat satellite imagery could be organized inside the SRB. Data is registered into the SRB data grid with the MCAT server. Clients requesting data contact the MCAT to find where the needed data resides and directs the client to the appropriate SRB master. Each site with data runs an SRB master which serves data to the clients. The MCAT stores all of the necessary catalog information in database. Many popular databases are supported including Oracle, DB2, PostgreSQL, Informix, and Sybase. Data registered in the MCAT can also be loaded with relevant metadata. This allows clients to not only access data by name but also by querying and filtering against attributes. SRB masters support various native access methods to retrieve the data at each site. Data can be served transparently to the client without exposing the underlying storage--from simple filesystems to complex storage hierarchical systems like HPSS. Additional Features Parallel Transfers: Retrieval of data can be faster through the SRB as compared to traditional methods by using parallel transfers from one or many sites. Replication: Data registered can be replicated to multiple sites for reliability through redundancy or copied and moved to other sites transparent to the clients. Security: The Grid Security Infrastructure provides a common authentication mechanism and allows for granular access control on data stored in the grid. Utilities: Many utilities, such as inQ, mySRB, and the S-command suite, already exist for the input and output of data to the SRB and APIs for many of the popular languages provide solutions for custom applications. Planned Work At this time, the partners in this project have set up a basic test- bed with some minimal data registered. We plan to expand our holdings and make more data available in the current grid. The SRB has support for basic metadata information but is lacking in the storage and querying of spatial information. We plan on creating extensions to the SRB to integrate the MCAT with standard spatial database extenders such as the Informix Spatial Datablade. The GLCF also plans on modifying its current version of the Earth Science Data Interface (ESDI) to create a web-based query and download tool for all data registered in this test-bed.