SAN DIEGO SUPERCOMPUTER CENTER HDF5/SRB Integration June 14, 2006 Mike Wan SRB, SDSC Peter Cao

Slides:

Advertisements

Similar presentations

National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center Data Grids for Collection Federation Reagan W. Moore University.

Advertisements

OGF-23 iRODS Metadata Grid File System Reagan Moore San Diego Supercomputer Center.

Inktomi Confidential and Proprietary The Inktomi Climate Lab: An Integrated Environment for Analyzing and Simulating Customer Network Traffic Stephane.

CUBIE ETD Independent Study Stephanie. A GAME An iOS running game 2D Unity Simple interaction – Tab to Jump 3-peoples team Lucien – Designer Peter – Artist.

UNIVERSITY OF CALIFORNIA, SAN DIEGO SAN DIEGO SUPERCOMPUTER CENTER UC Grid Summit -- April 1, 2009 UC San Diego Campus Grid Update Shava Smallen San Diego.

Jialin Liu, Bradly Crysler, Yin Lu, Yong Chen Oct. 15. Seminar Data-Intensive Scalable Computing Laboratory (DISCL) Locality-driven High-level.

Data Grid: Storage Resource Broker Mike Smorul. SRB Overview Developed at San Diego Supercomputing Center. Provides the abstraction mechanisms needed.

NATIONAL PARTNERSHIP FOR ADVANCED COMPUTATIONAL INFRASTRUCTURE SAN DIEGO SUPERCOMPUTER CENTER Early Experiences with Datastar: A 10TF Power4 + Federation.

SAN DIEGO SUPERCOMPUTER CENTER HDF5/SRB Integration July 10, 2006 Mike Wan SRB, SDSC Peter Cao

October 15-17, 2008HDF and HDF-EOS Workshop XII, Denver, CO1 HDF5-iRODS Peter Cao The HDF Group Mike Wan San Diego Supercomputer Center HDF and HDF-EOS.

Visualization of ENZO - Tiger Simulation Amit Chourasia Sr. Visualization Scientist San Diego Supercomputer Center, UCSD AUS Telecon Jan 14, 2010 Collaborative.

SDM center Questions – Dave Nelson What kind of processing / queries / searches biologists do over microarray data? –Range query on a spot? –Range query.

Chronopolis: Preserving Our Digital Heritage David Minor UC San Diego San Diego Supercomputer Center.

Massive High-Performance Global File Systems for Grid Computing -By Phil Andrews, Patricia Kovatch, Christopher Jordan -Presented by Han S Kim.

HDF 1 NCSA HDF XML Activities Robert E. McGrath Mike Folk National Center for Supercomputing Applications.

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center National Partnership for Advanced.

Descriptive Data Analysis of File Transfer Data Sudarshan Srinivasan Victor Hazlewood Gregory D. Peterson.

Event Metadata Records as a Testbed for Scalable Data Mining David Malon, Peter van Gemmeren (Argonne National Laboratory) At a data rate of 200 hertz,

Amit Chourasia Visualization Scientist Visualization Services Presented at : Florida State University, Nov 20 th 2006 Scientific Visualization of Large.

Amit Chourasia Visualization Scientist San Diego Supercomputer Center Presented at : Cyberinfrastructure Internship Experiences for Graduate Students Spring.

SAN DIEGO SUPERCOMPUTER CENTER HDF5/SRB Integration August 28, 2006 Mike Wan SRB, SDSC Peter Cao

HDF5 A new file format & software for high performance scientific data management.

CYBERINFRASTRUCTURE FOR THE GEOSCIENCES High Performance Computing applications in GEON: From Design to Production Dogan Seber.

Rule-Based Data Management Systems Reagan W. Moore Wayne Schroeder Mike Wan Arcot Rajasekar {moore, schroede, mwan, {moore, schroede, mwan,

1 Overview of HDF5 HDF Summit Boeing Seattle The HDF Group (THG) September 19, 2006.

February 2-3, 2006SRB Workshop, San Diego P eter Cao, NCSA Mike Wan, SDSC Sponsored by NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration Object-level.

2005 Materials Computation Center External Board Meeting The Materials Computation Center Duane D. Johnson and Richard M. Martin (PIs) Funded by NSF DMR.

December 1, 2005HDF & HDF-EOS Workshop IX P eter Cao, NCSA December 1, 2005 Sponsored by NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration.

VISUALIZING EARTHQUAKE SIMULATION DATA Amit Chourasia 1, Steve Cutchin 1, Alex DeCastro 1, Geoffrey Ely 2 1 San Diego Supercomputer Center 2 Scripps Institute.

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center National Partnership for Advanced.

AMIT CHOURASIA VISUALIZATION SCIENTIST VISUALIZATION SERVICES SAN DIEGO SUPERCOMPUTER CENTER PRESENTED AT : GEOINFORMATICS 2007, MAY 18 TH VISUAL REPRESENTATION.

March 17, 2006CIP Status Meeting March 17, 2006 Sponsored by NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration Project Report at CIP AG Meeting.

San Diego Supercomputer Center National Partnership for Advanced Computational Infrastructure SRB + Web Services = Datagrid Management System (DGMS) Arcot.

Kurt Mueller San Diego Supercomputer Center NPACI HotPage Updates.

June 29 San FranciscoSciDAC 2005 Terascale Supernova Initiative Discovering New Dynamics of Core-Collapse Supernova Shock Waves John M. Blondin NC State.

National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center Persistent Management of Distributed Data Reagan W. Moore.

National Partnership for Advanced Computational Infrastructure San Diego Supercomputer Center Persistent Archive for the NSDL Reagan W. Moore Charlie Cowart.

NEES Cyberinfrastructure Center at the San Diego Supercomputer Center, UCSD George E. Brown, Jr. Network for Earthquake Engineering Simulation Analyzing.

Scalable Systems Software for Terascale Computer Centers Coordinator: Al Geist Participating Organizations ORNL ANL LBNL.

SIG: Synthetic Seismogram Exchange Standards (formats & metadata) Is it time to establish exchange standards for synthetic seismograms? IRIS Annual Workshop.

NEES Cyberinfrastructure Center at the San Diego Supercomputer Center, UCSD George E. Brown, Jr. Network for Earthquake Engineering Simulation Enabling.

The Global Land Cover Facility is sponsored by NASA and the University of Maryland.The GLCF is a founding member of the Federation of Earth Science Information.

SAN DIEGO SUPERCOMPUTER CENTER By: Roman Olschanowsky An Introduction to the.

Center for Computational Visualization University of Texas, Austin Visualization and Graphics Research Group University of California, Davis Molecular.

19 May 2006 NCSA CIP Status Meeting 1 HDF5/SRB Integration Peter Cao & Mike Folk, NCSA Mike Wan & Reagan Moore, SDSC.

HDF and HDF-EOS Workshop VIII, October 26-28, /12 Peter Cao, National Center for Supercomputing Applications Ray Milnurn, Dave Buto, L-3 Communications.

Introduction to The Storage Resource.

Amit Chourasia Visualization Scientist San Diego Supercomputer Center Presented at : Scientific Computing Institute, Univ. of Utah. Jun 20, 2007 Scientific.

Parallel I/O Performance Study and Optimizations with HDF5, A Scientific Data Package MuQun Yang, Christian Chilan, Albert Cheng, Quincey Koziol, Mike.

Visualizing TERASHAKE Amit Chourasia Visualization Scientist Visualization Services San Diego Supercomputer center Geon Visualization Workshop March 1-2,

Microsoft Word Screen Parts

HDF-EOS Workshop IV September 19-21, 2000 Richard E. Ullman ESDIS Information Architect NASA/ GSFC, Code 423.

Improving long-term preservation EOS data by independently mapping HDF4 data objects Mike Folk, Ruth Aydt, Peter Cao, Kent Yang Ruth Duerr Christopher.

Visualization Efforts at San Diego Supercomputer center Amit Chourasia Visualization Scientist Visualization Services Presented to: SCEC-CME All Hands.

SDM Center High-Performance Parallel I/O Libraries (PI) Alok Choudhary, (Co-I) Wei-Keng Liao Northwestern University In Collaboration with the SEA Group.

SAN DIEGO SUPERCOMPUTER CENTER Advanced User Support Project Overview Thomas E. Cheatham III University of Utah Jan 14th 2010 By Ross C. Walker.

National Archives and Records Administration1 Integrated Rules Ordered Data System (“IRODS”) Technology Research: Digital Preservation Technology in a.

The Storage Resource Broker and.

Visualizing large scale earthquake simulations Amit Chourasia Visualization Scientist San Diego Supercomputer Center Presented to: Advanced User Support,

SAN DIEGO SUPERCOMPUTER CENTER Replication Policies for Federated Digital Repositories Robert H. McDonald Chronopolis Project Manager

Data Grids, Digital Libraries and Persistent Archives: An Integrated Approach to Publishing, Sharing and Archiving Data. Written By: R. Moore, A. Rajasekar,

SCEC Capability Simulations on TeraGrid

HDF Product Designer: Using Templates to Achieve Interoperability

HDF5 for Real-Time and/or Embedded Test Data

Large Data Visualization of Seismic Data (TeraShake)

Horizontal Centering Using the menu bar

Growing importance of metadata for synthetics: Calculating and Sharing Synthetic Seismic Data Dogan Seber University of California, San Diego San Diego.

OGCE Portal Applications for Grid Computing

TeraScale Supernova Initiative

Presentation transcript:

SAN DIEGO SUPERCOMPUTER CENTER HDF5/SRB Integration June 14, 2006 Mike Wan SRB, SDSC Peter Cao HDF, NCSA Sponsored by CIP/NLADR, NFS PACI Project in Support of NCSA-SDSC Collaboration

SAN DIEGO SUPERCOMPUTER CENTER Current Status Publish HDF5 AIP documents White paper: HDF5 METS template: Finish h5ingest command line tool Create HDF5 METS template file Validate HDF5 METS document Work on test suite and bug fix (Peter Cao) Work on performance improvement (Mike Wan) Setup a demo server to support SCEC files

SAN DIEGO SUPERCOMPUTER CENTER Potential SAC Projects SDSC ENZO project Enzo, 3D cosmological hydrodynamics code, simulating the process of massive star formation and destruction HDF5 is used as file format and parallel file I/O access FLASH Program The UC/DOE collaboration on creating three-dimensional, virtual reality projections of the cosmic explosions HDF5 is used for storing the data and high I/O access SCEC Terascale Earthquake Simulations Over 100 TB data/year Collections at SRB – 2.6 million files, 114 Terabytes

SAN DIEGO SUPERCOMPUTER CENTER TeraShake Surface Seismograms 4D Array (1.2 TB) Time (22,728) Horizontal (3,000) Vertical (1,500) Vector Component (3) Each file: 22,728 x 3,000 x 5 x 1 1,363,680,000 Bytes TeraShake scenario 900 files

SAN DIEGO SUPERCOMPUTER CENTER Example HDF5 File xhist00001hpss-scec xhist00002hpss-scec xhist00003hpss-scec xhist00004hpss-scec xhist00005hpss-scec HDF5 File 32-bit float 22,728 3,000 25

SAN DIEGO SUPERCOMPUTER CENTER File on SRB server

SAN DIEGO SUPERCOMPUTER CENTER Select a Subset

SAN DIEGO SUPERCOMPUTER CENTER HDFView