1. By: Roman Olschanowsky roman2u@sdsc.edu An Introduction to the

2. Outline SDSC and History of SRB –Example Project Introduction to SRB –Discussion on SRB basics –SRB Clients Overview of a Data Grid –Infrastructure –Topology Teragrid Demo –How to use your TG SRB account –How to access Digital Data Collections

3. Archival Systems 18 PB 15.6 TF DataStar IBM Power4 4.4 TF TeraGrid Linux Cluster (IA64) 1.4 PB Storage Area Network Disk Sun F15K Disk Server Networking Visualization Storage and Compute Resources Human infrastructure: Experienced multi- disciplinary staff support a broad spectrum of national science, engineering and technology projects Blue Gene/L (Due 12/04) 2.8/5.7 TF www.sdsc.edu

4. Sites Using the SRB

5. SDSC SRB Projects (60 million,.5 PB ) Digital Libraries –UCB, Umich, UCSB, Stanford,CDL –NSF NSDL - UCAR / DLESE NASA Information Power Grid Astronomy –National Virtual Observatory –2MASS Project (2 Micron All Sky Survey) Particle Physics –Particle Physics Data Grid (DOE) –GriPhyN –SLAC Synchrotron Data Repository Medicine –Digital Embryo (NLM) Earth Systems Sciences –ESIPS –LTER Persistent Archives –NARA –LOC Neuro Science & Molecular Science –TeleScience/NCMIR, BIRN –SLAC, AfCS, …

6. Storage Resource Broker (SRB) A distributed file system (Data Grid) –Client-Server, Server-Server architecture. –Abstracts physical SRB provides the ability to transparently share data across remote sites. –Heterogeneous Resources –Single sign on –Single logical file hierarchy

7. What we are familiar with

8. What we are not familiar with, yet

9. How do the file systems differ? Logical Abstraction –Folders are NOT physical –Files do NOT inherit physical location –Everything is potentially distributed Access Control –Permissions are NOT rwxrwxrwx –Permissions ARE on a object by object basis –Groups and permissions ARE more similar to NTFS Domains –Geographical / logical grouping of users –Namespace scalability: john@harvard john@mit –Also doubles as groups

10. Interfaces to the Storage Resource Broker inQ– Windows Client Scommands– UNIX, DOS Command line Client Jargon– Java API and GUI components mySRB– Web Client Matrix– WSDL, Data Grid Workflows C, C++– C and C++ API Python– Python API Perl– Perl API

11. Common Scommands (75 total) Sinit Senv Spwd Sls Scd Sget Sput Ssh Scp Smv (logical) Sphymove (physical) Srm Smkdir Srmdir Serror Schmod Sexit

12. mySRB

13. BIRN Portal (perl based)

14. NEEScentral Portal (php based)

15. Biomedical Informatics Research Network (BIRN) Major collaboration with SDSC, several of the projects’ Co- Investigators and Co-PIs are at SDSC. BIRN’s purpose is to provide it’s consortium of neuroscience laboratories the ability to share, compute, and collaborate. The Storage Resource Broker provides the ability to transparently share data across remote sites.

16. The BIRN SRB Data Grid

17. Doing this “Manually”

18. The BIRN Data Grid

19. The grid is in the details

20. File Replication Sls /home/Demo/SRB-Tutorial/files-2: Doc.txt Sls -l /home/Demo/SRB-Tutorial/files-2: romanoly 0 z-ucsd-ncmir-nas1 15 2003-07-09-05.15 Doc.txt romanoly 1 z-jhu-cis-nas0 15 2003-07-09-05.16 Doc.txt romanoly 2 z-stanford-lucas-nas 15 2003-07-09-05.16 Doc.txt romanoly 3 z-umn-cmrr-nas0 15 2003-07-09-05.16 Doc.txt romanoly 4 z-uci-bic-nas0 15 2003-07-09-05.17 Doc.txt

21. Teragrid SRB –All Teragrid accounts are given a SDSC SRB Teragrid account The ‘username’ is the same as your SDSC UNIX account name Your SRB ‘domain’ is ‘teragrid’ You must register your DN string with SDSC’s grid-mapfile or request a SRB password to activate your SRB account. Instructions to do so are here: http://www.sdsc.edu/user_services/srb/account_form.html http://www.sdsc.edu/user_services/srb/account_form.html –Your ~/.srb/.MdasEnv file OR env variables mdasCollectionHome '/home/.teragrid' mdasDomainName 'teragrid' srbUser ' ' #AUTH_SCHEME 'ENCRYPT1' AUTH_SCHEME 'GSI_AUTH' srbHost 'srb.sdsc.edu' srbPort '7321' defaultResource 'sfs-tape-tgd' SERVER_DN '/C=US/O=NPACI/OU=SDSC/CN=Storage Resource Broker/USERID=srb'

22. Scommand Features Command line interface -> SCRIPTING Available for all of the most popular UNIX flavors and DOS S-commands are the most flexible and powerful of the clients They are the fastest, and most reliable They are multithreaded for big gains in data flow They are great for scripts, perl wrappers, batch jobs, etc… Installed man pages via “man [Scommand]” –man Sput

23. Scommand Notes Shelp –Gives list of commands with brief summary –“[Scommand] ” gives usage info (usually) or try –h flag Sinit – establishes a session Senv – displays connection information Spwd – display current working directory Sexit – ends session

24. Some Public SRB Collections Southern California Earthquake Center /home/public.teragrid/SCEC Two Micron All Sky Survey /home/public.teragrid/2MASS The Palomar Digital Sky Survey /home/public.teragrid/DPOSS

25. Watch me do a SRB demo

26. Thanks! SRB handles large data and provides the ability to share and collaborate on distributed heterogeneous resources. Questions? Teragrid SRB userguide: http://www.sdsc.edu/user_services/srb http://www.sdsc.edu/user_services/srb SRB website: www.sdsc.edu/srb SRB email: srb@sdsc.edu