1. a brief summary for users
Xrootd SLAC
a brief summary for users
Wei Yang
SLAC ATLAS forum

2. How Xrootd works Glue file servers together by a redirector
User only need to know XROOT path: root://redirector:port//path/file
Simple, low overhead
No complex features such as locking
Good for reading dominated environment, e.g. HEP data analysis
/xrd/file1
/xrd/file2
/xrd/file3
redirector
I want /xrd/file2
Time out…
Who has /xrd/file2
Waiting for a reply
from data nodes
I have /xrd/file2
/xrd/fileN
Please talk to data node 2
data node 1,2,3,…,N
Nobody has /xrd/file2

3. User interface to Xrootd
TXNetFile class (C++)
Fault tolerance
High performance thought intelligent logics in TXNetFile and server
Command line tools
xrdcp
simple, native, light weight, high performance
Posix preload library
export LD_PRELOAD=/…/libXrdPosixPreload.so
ls/cat/cp root://redirector:port//path/file
A subset of UNIX I/O command will work with Posix preload library
on files, not on directories
Some overhead, I/O performance isn’t as good as xrdcp

4. User interface to Xrootd, cont’d
XrootdFS
File system interface for Xrootd
Mount the Xrootd cluster on client host’s local file system tree
Provide standard Posix I/O interface to the Xrootd cluster
open(), close(), read(), write(), lseek(), unlink(), rename()
opendir(), closedir(), readdir(), mkdir()
Work with most UNIX commands/tools
cd, ls, cp, rm, mkdir, cat, grep, find
ssh/sftp server, gridftp server, SRM, xrootd server
scp/sftp, gridftp clients, SRM clients, dq2 clients
Be aware: no file locking, no ownership/protection
file creation delay
some UNIX command are not scalable, e.g. find, ls
cp is slow (due to small I/O block size)
More overhead on I/O performance

5. GRID interface for Xrootd
Grid Storage Element (SE) is not clearly defined by Globus
Currently LCG and OSG define SE interface
GridFTP
FTP client/server based on GSI authentication
A GridFTP plug-in module for Xrootd was developed at SLAC
Storage Resource Manager (SRM)
Many functions. Mainly used as a load balancer for GridFTPs
Berkeley Storage Manager (BeStMan)
A trim down, high performance SRM, BeStMan-xrootd, is in
production at 3 US ATLAS Tier 2 sites.
Backend storage used by ATLAS: Castor, dCache, DPM, Xrootd, others
dq2 clients use GridFTP and SRM client tools to access remote data

6. Bestman-xrootd implementation
Implemented or partially implemented
srmLs() srmRm() srmMkdir() srmRmdir()
srmPrepareToPut() srmPrepareToGet()
Support WLCG static space token
srmGetSpaceTokens()
Bestman-xrootd itself does NOT transfer data
Rely on PtP() and PtG() and GridFTP servers to transfer data
No request queuing
No srmStatusOfPut() and srmStatusOfGet()

7. How srmPreparToPut and srmPrepareToGet work
SURL: srm://srm.a.com/srm/v2/server?SFN=/path/file
O O ~
Client
SRM
TURL: gsiftp://gftp03.a.com/path/file
TURL
SURL : Storage URL
TURL : Transfer URL
SURL
SRM
GridFTP
GridFTP
GridFTP
GridFTP
GridFTP

8. WT2 Architecture ~ 600 cores, 270 TB usable Cisco Cisco SLAC Firewall
10 Gbit
Cisco
Cisco
SLAC Firewall
Grid Interface/Proxy
DQ2/LRC
Cond DB
dq2 client
GridFTP
SRM
Internet Free Zone
LSF Master
1Gbit
Xrootd redirector
NFS servers
grid user home WN
Cisco switch
Xrootd server WN
1Gbit WN WN
10 Gbit
SUN X4500
4 Gbit WN
Cisco switch WN
Xrootd server
RHEL 4/64
~ 600 cores, 270 TB usable WN

10. ATLAS Distributed Data Management system
US cloud
New dataset
Lyon T1
NE T2
MW T2
RAL T1
BNL storage
File Transfer Service
CERN storage
DQ2/LRC
dataset A
Central catalog DB
DQ2/Catalog
WT2 disk
Dataset replica info
Dataset info
subscript WT2 to dataset A
Other DS
File
File
DQ2 tools
dq2_* (old, out-of-date)
dq2–* (new, not yet deployed at OSG)
File
Site subscription info
File
Don Quijote II

13. Panda Panda is a US ATLAS production system
result
job
DDM central catalog
Panda
BNL
Panda Mover
input
task 1
task 2
Catalog
output
task 3
Storage
Grid
task 4
DQ2 …
task n
Panda is a US ATLAS production system
Uses DDM for output, Panda Mover for input
Supports distributed analysis via the GRID