Biomedical Informatics Research Network The Storage Resource Broker & Integration with NMI Middleware Arcot Rajasekar, BIRN-CC SDSC October 9th 2002 BIRN All Hands Meeting, Durham, NC

SRB in BIRN BIRN Toolkit Mediator Viewing/Visualization Queries/ResultsApplications Data Management File System MCAT HPSS Data Model Data Access Data Grid Computational Grid Collaboration NMI Grid Management Globus GridPort Scheduler Distributed Resources Database SRB Database

Data Architecture to Enable Collaboration  Objectives Foster collaboration Provide controlled data sharing Facilitate data-intensive computing Provide Data Persistence Support massive data scales  Secure storage and access to peta bytes of data  Containing several million data sets  Housing thousands of data collections Query metadata to locate files Access files regardless of location, operating system, media or hierarchy

Creating the BIRN Data Grid Provide uniform access to Lots of data Lots of metadata Lots of Locations Heterogeneous Data Organizations Lots of storage gadgetry Need consistent and straightforward interface Rich Access Control Logical Collection Creation Transparent Data Replication

Data Grid Using a Data Grid – in Abstract Ask for data User asks for data from the data grid Data delivered The data is found and returned Where & how details are hidden

User asks for data Using a Data Grid - Details Storage Resource Broker Data request goes to SRB Server Storage Resource Broker Metadata Catalog DB Server looks up data in catalogCatalog tells which SRB server has data1 st server asks 2 nd for dataThe data is found and returned

Using a Data Grid - Details SRB MCAT DB SRB Data Grid has arbitrary number of servers Complexity is hidden from users

SRB Features  Resource Transparency - uniform access interface  Location Transparency - logical naming in collections  Cross-Domain Authentication - single account management  Rich Access Control - Users, Groups, Resources  User Transparency Uniform User Name Space  Replicated Data fault tolerant  Data Discovery User-defined Metadata  Multiple Access Methods GUI, API, Command-line  Supported on Heterogeneous Platforms Unix, Windows, Linux DR DL MC Client SRB

SRB Projects  Digital Libraries UCB, UMich, UCSB, Stanford,CDL NSF NSDL - UCAR / DLESE  NASA Information Power Grid  Astronomy National Virtual Observatory 2MASS Project (2 Micron All Sky Survey)  Particle Physics Particle Physics Data Grid (DOE) SLAC Synchrotron Data Repository  Medicine Digital Embryo (NLM)  Earth Systems Sciences ESIPS LTER  Persistent Archives NARA LOC  Neuro Science & Molecular Science BIRN SLAC, AfCS, … Over 40 Tera Bytes in 6.4 million files

BIRN SRB Technology  SRB brings state-of-the-art data technology to BIRN It is an existing middle-ware Field-tested in multiple projects Proven scalability Input from other research communities Leverages NPACI resources to deliver real solutions

BIRN SRB - Current Status  SRB deployed at all BIRN sites 10+ Racks  SRB comes installed with the BIRN rack  MCAT running at BIRN-CC  File storage & databases at each site  Provides access to HPSS at SDSC for archival storage  Uses GSI authentication

BIRN SRB - Current Status (Contd.)  SRB & BIRN Applications Integration with SDSC/NPACI 3-D toolkit DICOM Metadata dump & extraction proxies Other application integration under development  Graphical User Interfaces to SRB BIRN portal – combining distributed data management and distributed computation MySRB – web-based access for data & metadata inQ – Windows-specific access

BIRN SRB Team  Accomplishments Learning the new vocabulary Learning about HIPAA, IRB, … Learning the complex schemas and formats used Learning about DICOM and its intricacies Mapping these into the SRB-MCAT world Learning about the tools used and their complexities Problems with data transfer -  File Systems, Networks & Firewalls, {first,last} mile Teaching about the SRB and learning in the process Built the BIRN virtual data grid

BIRN SRB - Charting Ahead  Integration with Other BIRN Components Work Flow  LONI Pipeline Visualization Systems  3D Slicer  FreeSurfer  NCMIR Tools  NPACI Visualization Tools Mediation Systems  Database Mediators NMI components  Globus, Condor, NWS

BIRN & NMI  NMI - NSF Middleware Initiative - Sept 2001 To aid the development of "middleware" to help scientists and researchers use the Internet to effectively share instruments, laboratories and data, and to collaborate with their colleagues.  BIRN stands at a focal point in contributing to NMI Provide a real-life testbed for NMI software Become a major player in NMI Applying distributed capabilities of software, resources and instruments to set directions for NMI