1. A. Sim, CRD, L B N L 1 Production Data Management Workshop, Mar. 3, 2009 BeStMan and Xrootd Alex Sim Scientific Data Management Research Group Computational Research Division Lawrence Berkeley National Laboratory

2. A. Sim, CRD, L B N L 2 Production Data Management Workshop, Mar. 3, 2009 What is SRM? SRM : Storage Resource ManagerSRM : Storage Resource Manager Well-defined storage management interface specification based on standard Different implementations for underlying storage systems are based on the same SRM specification Provides dynamic space allocation and file management on shared storage components on the Grid Over 300 deployments of different SRMs in the world Managing more than 10 PB

3. A. Sim, CRD, L B N L 3 Production Data Management Workshop, Mar. 3, 2009BeStMan Light-weight implementation of SRM v2.2Light-weight implementation of SRM v2.2 Works on existing disk storages with posix-like file systems E.g. NFS, GPFS, GFS, NGFS, PNFS, HFS+, PVFS, Lustre, Xrootd, Hadoop, Ibrix Supports multiple partitions Adaptable to other file systems and storages Supports customized plug-in for MSS to stage/archive such as HPSS Easy adaptability and integration to special project environments Supports two modeSupports two mode Full mode : management on request queues and spaces Gateway mode : faster performance without queue/space management Supports multiple transfer protocolsSupports multiple transfer protocols Supports load balancing for multiple transfer servers Scales well with some file systems and storagesScales well with some file systems and storages Xrootd, Hadoop, GPFS, PVFS, etc Works with grid-mapfile or GUMS serverWorks with grid-mapfile or GUMS server Simple installation and easy maintenanceSimple installation and easy maintenance Packaged in VDT using Pacman Who would benefit from BeStMan?Who would benefit from BeStMan? Sites with limited resources and/or limited admin effort

4. A. Sim, CRD, L B N L 4 Production Data Management Workshop, Mar. 3, 2009 Xrootd and XrootdFS Developed at SLAC and INFN PadovaDeveloped at SLAC and INFN Padova Suite of tools for serving dataSuite of tools for serving data server daemons, clients (ROOT TXNetFile, POSIX), use xrootd protocol Provides high performance file-based accessProvides high performance file-based access Scales on server and client sideScales on server and client side Separate data flow and control flowSeparate data flow and control flow Fault tolerantFault tolerant SecureSecure rootd compatiblerootd compatible Xrootd provides a Posix I/O interfaceXrootd provides a Posix I/O interface in addition to the native XTNetFile class XrootdFS provides a Posix File System layerXrootdFS provides a Posix File System layer Mounts the Xrootd storage cluster to a desktop machine Provides a complete view of storage tree Base on FUSE, implemented using Xrootd Posix I/O library

5. A. Sim, CRD, L B N L 5 Production Data Management Workshop, Mar. 3, 2009 How it works all together File System Gridftp server...... BeStMan Gateway Client srmPrepareToGet/Put TURL GridFTP file transfers srmReleaseFiles/srmPutDone in PUT/GET File system Gridftp server...... BeStMan Gateway Client srmLs/srmRm/srmMkdir/srmRmdir in Ls/Rm/Mkdir/Rmdir XrootdFSXrootdXrootdFSXrootd

6. A. Sim, CRD, L B N L 6 Production Data Management Workshop, Mar. 3, 2009 Some Use Cases CMSCMS BeStMan Gateway as an SRM frontend for Hadoop at UNL Passed all the automated CMS tests through EGEE SAM at UNL ATLASATLAS BeStMan on NFS BeStMan Gateway on Xrootd/FS, GPFS, Ibrix STARSTAR Data replication between BNL and LBNL HPSS access at BNL and NERSC SRMs in production for over 4 years Part of analysis scenario to move job-generated data files from PDSF/NERSC to remote BNL storage Earth System GridEarth System Grid Serving about 6000 users Over a million files and 170TB of climate data from 5 storage sites (LANL, LLNL, NCAR, NERSC, ORNL) Uses an adapted BeStMan for NCAR’s own MSS

7. A. Sim, CRD, L B N L 7 Production Data Management Workshop, Mar. 3, 2009Summary BeStMan is an implementation of SRM v2.2.BeStMan is an implementation of SRM v2.2. Great for smaller disk-based storage and file systems BeStMan Gateway mode on some file systems and storage gives scalable performanceBeStMan Gateway mode on some file systems and storage gives scalable performance Install/maintain through VDTInstall/maintain through VDT Works with other SRM v2.2 implementationsWorks with other SRM v2.2 implementations Servers: CASTOR, dCache, DPM, StoRM, SRM/SRB, … Clients: FTS, PhEDEx, glite-url-copy, lcg-cp, srm-copy, srmcp, … In OSG, WLCG/EGEE, ESG, …

8. A. Sim, CRD, L B N L 8 Production Data Management Workshop, Mar. 3, 2009 Documents and Support BeStManBeStMan http://datagrid.lbl.gov/bestman http://hep-t3.physics.umd.edu/HowToForAdmins.html#osgBestman http://wt2.slac.stanford.edu/xrootdfs/bestman-gateway.html https://www.usatlas.bnl.gov/twiki/bin/view/Admins/BestMan https://twiki.grid.iu.edu/bin/view/Documentation/BestmanGateway https://twiki.grid.iu.edu/bin/view/Documentation/BestmanGateway-Xrootd XrootdXrootd http://xrootd.slac.stanford.edu http://wt2.slac.stanford.edu/xrootdfs/xrootdfs.html SRM Collaboration and SRM SpecificationsSRM Collaboration and SRM Specifications http://sdm.lbl.gov/srm-wg Contact and support : srm@lbl.govContact and support : srm@lbl.gov