EMI is partially funded by the European Commission under Grant Agreement RI DPM in EMI-II HTTP and NFS interfaces Oliver Keeble On behalf of DPM team CERN EGI CF 29/03/2012

EMI INFSO-RI Over 250 DPM endpoints HTTP and NFS interfaces for DPM/LFC

EMI INFSO-RI Reliable grid disk storage solution Simplicity Stability VOMS security Access protocol support SRM Key Features HTTP and NFS interfaces for DPM/LFC

EMI INFSO-RI Architecture HTTP and NFS interfaces for DPM/LFC HEAD NODE DISK NODES

EMI INFSO-RI DPM daemons HTTP and NFS interfaces for DPM/LFC CLIENT DPNSDPMSRMHTTP/DAVNFS GRIDFTP RFIOHTTP/DAVNFSXROOT HEAD NODE DISK NODE(s) FILE METADATA OPS FILE ACCESS OPS RFIO HTTP NFS XROOT

EMI INFSO-RI EPEL compliant packaging (with source rpms) NFS HTTP / DAV Nagios probes GridPP Contrib Admin Tools Initial SL6 support Synchronous GET requests (dpm_get) Thread safe client authz choice (CSEC_MECH) Improved DNS load balancing What comes with EMI-II HTTP and NFS interfaces for DPM/LFC

EMI INFSO-RI Installation is from the EMI-II repositories. We recommend following EMI instructions for repos – Gives EMI repos priority We make simultaneous releases to EPEL. – packages will arrive in EPEL earlier. Installation for EMI-II HTTP and NFS interfaces for DPM/LFC

EMI INFSO-RI Yaim is supported: /opt/glite/yaim/bin/yaim -c -s /etc/yaim/site-info.def -n emi_dpm_disk /opt/glite/yaim/bin/yaim -c -s /etc/yaim/site-info.def -n emi_dpm_mysql YAIM configuration HTTP and NFS interfaces for DPM/LFC

EMI INFSO-RI In EMI-II, packaging is EPEL compliant, so you should find config files where you expect. – Configs in /etc/sysconfig – Logs in /var/log – Etc… Underlying config HTTP and NFS interfaces for DPM/LFC

EMI INFSO-RI ● Standard HTTP and DAV access ● Use with standard clients ● Plus, with some extra on the clients ● Support for multi-stream transfers – Big throughput gain for WAN ● 3rd party copies using HTTP – FTS will support this – Destination can be any HTTP that support PUT ● Source can be DPM and, in the future, dCache DPM-HTTP/DAV: Motivation HTTP and NFS interfaces for DPM/LFC

EMI INFSO-RI ● For each node (LFC, DPM Head or Disk) ● Will arrive automatically with metapackage ● Otherwise, yum install lcgdm-dav-server ● Configuration files are under the expected location ● /etc/httpd/conf.d ● Specific file: zlcgdm-dav.conf ● All the rest is standard Apache stuff DPM-HTTP/DAV: Installation HTTP and NFS interfaces for DPM/LFC

EMI INFSO-RI ● Any web browser can be used to explore the namespace and get a file ● Some metadata given (mode, size, owner...) ● Any DAV client can be used to retrieve more metadata (replicas, comments, …) ● Almost none support redirections (Some do on GET) ● i.e. Cadaver, MS Windows Explorer ● A specific client aware of some extensions is provided ● htcopy – Download and upload with multiple streams – 3rd party copies (including delegation) DPM-HTTP/DAV: Usage HTTP and NFS interfaces for DPM/LFC

EMI INFSO-RI And if you install it on LFC… HTTP and NFS interfaces for DPM/LFC CLIENT LFC DPM HEAD DPM DISK GET / PUT REDIRECT DATA This has been demonstrated using the HTTP interface to LFC & DPM Can even incorporate dCache

EMI INFSO-RI One daemon, deployed on all nodes Current implementation relies on DPNS and DPM daemon Future implementations will talk directly to the database, using our new DMLITE library DPM-NFS: Server Overview HTTP and NFS interfaces for DPM/LFC

EMI INFSO-RI EMI2 to have the first production version – Read-only at the start – Writable with a subsequent update Userspace daemon (based on Ganesha) Fully functional Already deployed at a few sites as a beta – ASGC, UK GridPP, … – And we keep releasing a new beta every month Unclear how to provide X509 authorization – Kerberos or machine trust possible though DPM-NFS: Server Availability HTTP and NFS interfaces for DPM/LFC

EMI INFSO-RI In EMI2, the metapackages (emi-dpm*) will include the DPM NFS daemon – Otherwise: yum install dpm-nfs-server dpm-devel glite-yaim-dpm vim – DPM_NFS="yes“ yaim -r -s site-info/site-info.def -n emi- dpm_mysql -f config_DPM_nfs – Starting dpm-nfs: [ OK ] But make sure you check the latest instructions DPM-NFS: Server Installation HTTP and NFS interfaces for DPM/LFC

EMI INFSO-RI DPM-NFS: Clients HTTP and NFS interfaces for DPM/LFC pNFS support in kernel >= – Your Fedora / Debian has it… seriously nfs-utils >= But the ones we currently support do not – RHEL5: – Debian 6/Squeeze: RHEL 6.2 has a pNFS aware kernel We provide packages for RHEL5/RHEL6 – Enabled pNFS in the elrepo mainline kernel – nfs-utils we package ourselves This slide should be blank…

EMI INFSO-RI SLC5 (requires elrepo) – rpm --import – rpm -Uvh – Add nfs41 repo (see details in wiki link below) – yum install --enablerepo=elrepo-kernel --enablerepo=nfs41 kernel-ml nfs-utils – echo nfs_layout_nfsv41_files >> /etc/rc.modules – chmod +x /etc/rc.modules – (without these two last steps, pNFS file layout will not be enabled) SLC6 – kernel with pNFS is already there – but make sure the nfs_layout_nfsv41_files module is loaded DPM-NFS: Client Installation HTTP and NFS interfaces for DPM/LFC

EMI INFSO-RI In EMI-II we will release – HTTP support for DPM and LFC – NFS support for DPM Part of a strategic orientation toward support of standards Please try them and let us know – – GGUS Summary HTTP and NFS interfaces for DPM/LFC

EMI INFSO-RI Extras HTTP and NFS interfaces for DPM/LFC

EMI INFSO-RI ● On DPM ● Head – Defaults are good enough – change DPMHost and NSHost if not localhost ● Disk – Must change DPMHost, NSHost and NodeType ● On LFC ● Need to change the NodeType and Location ● Must remove DPMHost ● Read-only by default DPM-DAV: Configuration HTTP and NFS interfaces for DPM/LFC

EMI INFSO-RI ● mod_ssl gives support for user certificates ● If proxies wanted, mod_gridsite must be installed and enabled ● Gridsite is used for delegation too (3rd party copies) ● Remember to configure properly the certificate and revocation list location ● More detailed information: ● V/Setup V/Setup DPM-DAV: Configuration HTTP and NFS interfaces for DPM/LFC

EMI INFSO-RI ● Global Access Service ● What is it – Browse an LFC through HTTP – Click on the file you want – You get it ● Will be enabled by default – Both on DPM and LFC (and dCache will join) – The number of replicas you get in one go can be configured (3 by default) DPM-DAV: Coming soon HTTP and NFS interfaces for DPM/LFC

EMI INFSO-RI DPM is widely deployed and used – 32PB advertised in the InfoSys – Over 200 sites 56 countries Usage outside HEP – 7 sites over 1PB – Over 300 VOs with access to DPMs DPM numbers HTTP and NFS interfaces for DPM/LFC