1. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 1 Installazione, configurazione, testing e troubleshooting di Storage Element SE [D-CACHE] Giacinto Donvito giacinto.donvito@ba.infn.it INFN-Bari I Corso di formazione INFN per amministratori di siti Grid Martina Franca (TA), 07 November 2007

2. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 2 Outline Introduction on SRM Introduction on dCache How it works –What is happening under the hood Installation of dCache, in theory: –Layout of a standard installation –Layout of a complex installation Installation of dCache: practical: –Installing using YAIM –Installing … by hand dCache: news, future, issues, etc. Conclusions

3. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 3 SRM Overview “Storage Resource Manager” SRM is a Control protocol What it does: –Ask to make file ready for upload/download –Basic metadata (size, checksum,…) –Many components optional Web service (over GSI HTTP) What it doesn’t: –Data transfer  However it can do third party transfer –Access control & permissions  However some implementation have already been tried

4. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 4 SRM functionalities Features from SRM v1.1 –Get –Put –copy –getFileMetaData –getRequestStatus –getProtocols –AdvisoryDelete Features from SRM v2.2 – File types -> (“Storage Classes”) – Space reservation – Permission functions – Directory functions – Data transfer control functions – Relative paths – Query supported protocols

5. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 5 Tape: 1 Disk: 0 DATA TYPESTORAGE TYPE Tape: 1 Disk: 1 Tape: 0 Disk: 1

6. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 6 dCache overview It is developed in a large collaboration between Desy and FNAL (plus some other minor contributions) GOALS: –To make a distributed storage system that can use cheap disk- server to gain high performance and high-availability –To provide an abstraction of whole disk space under a unique NFS like file-system (just for metadata operations) –To possibly add the support for its own MSS system  They are needed only 2 or 3 scripts (put/get/remove) –To provide a system that scale at  hundreds of TB of disk cache  hundreds of pool nodes  hundreds of TB per day to clients File access: –provides local and remote access (posix like) with many protocols (dcap, ftp) both with and without authentication (gsi or kerberos)

7. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 7 dCache overview (2) Access management: access priority and load balancing obtained trough the use of different queue Allows multiple copy of files spread over different pools to improve performance and HA –pool-2-pool automatic (or manually) transfers Allows dynamic “match-making” between pools –According to the parameters chosen by the administrator (they can be based on disk space, load, network, type of access etc.) It is possible to split different type of “access point” (doors) on different nodes It is possible to move all the files in a pool to put it in a “scheduled downtime” –Or just to choose which file you want to move and where. Also the “central services” can be split on different nodes to improve the scalability

8. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 8 dCache overview (3) Pool management: –gives the possibility to create groups of pools named “storage class” (read, write, cache, or per VO and user bases or use bases) –Can be useful for quota management Web monitoring, statistical module (also with rate-plot), and SRM Monitoring The SRM layer can be used as stand-alone software (on standard Unix file-system) It is possible to choose the space used by dCache pool in a partition (you can host many “services” on the same partition) JAVA GUI for administration Also Xrootd protocol is supported Accounting system flat-files or DB based (not user friendly but there are many information) and space used per VO It is possible to use WN (or other “not reliable” space) disks to improve performance for local access

9. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 9 dCache overview (4)

10. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 10 dCache: A file read (What is happening under the hood)

11. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 11 dCache: A file read (What is happening under the hood)

12. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 12 dCache: A file read (What is happening under the hood)

13. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 13 dCache: A file read (What is happening under the hood)

14. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 14 dCache: A file read (What is happening under the hood)

15. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 15 dCache: A file read (What is happening under the hood)

16. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 16 dCache: A file read (What is happening under the hood)

17. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 17 dCache: A file read (What is happening under the hood)

18. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 18 dCache: A file read (What is happening under the hood)

19. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 19 dCache: Advanced Installation Layout dCache CORE PNFS Server Postgres DB Admin nodePNFS Server DB Server Pool node SRM door Gsi-dcap door Pool service (read) gridftp door Pool node SRM door Gsi-dcap door Pool service (write) gridftp door Pool node SRM door Gsi-dcap door Pool service (xrootd) gridftp door DNS Aliased dCap door xRootd door

20. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 20 YAIL: Yet Another Installation Layout

21. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 21 Best Practices Admin Node must be “resilient” PNFS-DB ( “admin”, “data1”, … ) are crucial: loosing this DB means loosing all files into that dir –It is better to create a PNFS-DB per each VO or each type of usage: better performances and scalability All other DB are not crucial Use the latest Postgres version –Is more stable and better performances It is better to have many small “pool” Doors should be always replicated Automatic installation with YAIM should be avoided –It is better to install and configure the system manually and then run “configure_node” with YAIM If the service is high loaded PNFS server can be splitted in a separate machine –Usually for a Tier2 site this is not needed

22. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 22 Reference Site: –www.dcache.org Installation instruction: –http://www.dcache.org/manuals/Book/in-install.shtml YAIM Installation instruction: –http://www.dcache.org/manuals/yaim-dcache-howto.shtml Main documentation: –http://www.dcache.org/manuals/Book/index.shtml

23. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 23 Prerequisite SLC3 for Admin node (SLC4 already available, maybe not so stable) Not so many problem with ANY OTHER OS on pool node JAVA >=1.4 for 1.7.x version –JAVA >=1.5 for 1.8.x version Host certificates for all pool nodes APT Repository (not yet for 1.8.x): –echo 'rpm http://cvs.dcache.org/repository/apt / ' > /etc/apt/sources.list.d/desy_dcache.list A lot of patience –A bit of perseverance

24. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 24 YAIM Installation YAIM Installation instruction: –http://www.dcache.org/manuals/yaim-dcache-howto.shtml Needed parameters: –MY_DOMAIN=gs.ba.infn.it –JAVA_LOCATION="/usr/java/j2sdk1.5.x_x” –DCACHE_ADMIN=”my-admin.gs.ba.infn.it” –DCACHE_POOLS="dcache.desy.de:7:/dCachePools/pool1 dcache.desy.de:7:/dCachePools/pool2”  # the pools : hostname:size:path –DCACHE_DOOR_SRM="my-admin.gs.ba.infn.it" –DCACHE_DOOR_GSIFTP="my-admin.gs.ba.infn.it" –DCACHE_DOOR_GSIDCAP="my-admin.gs.ba.infn.it" –DCACHE_DOOR_DCAP="my-admin.gs.ba.infn.it” –RESET_DCACHE_CONFIGURATION=yes –RESET_DCACHE_PNFS=yes –RESET_DCACHE_RDBMS=yes –VOS="ops dteam" Starting from dCache 1.8 - > only JAVA 1.5.x is supported

25. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 25 YAIM Installation For admin nodes : –/opt/glite/yaim/scripts/install_node ~/site-info.def glite- SE_dcache_admin_postgres –/opt/glite/yaim/scripts/configure_node ~/site-info.def glite- SE_dcache_admin_postgres For pool nodes : –/opt/glite/yaim/scripts/install_node ~/site-info.def glite- SE_dcache_pool –/opt/glite/yaim/scripts/configure_node ~/site-info.def glite- SE_dcache_pool

26. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 26 Manual Installation Installation instruction: –http://www.dcache.org/manuals/Book/in-install.shtml –“wget” all the rpm in: http://repository.gs.ba.infn.it/dcache/

27. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 27 And now… check you installation Look at: –http://your-host.gs.ba.infn.it:2288/cellInfo –dccp -d 3 /tmp/test_file dcap://your- host.gs.ba.infn.it/pnfs/gs.ba.infn.it/data/test1 –srmcp -debug=true file:////tmp/test_file srm://your- host.gs.ba.infn.it:8443/pnfs/gs.ba.infn.it/data/test1 –ls -ltr /var/log/*Domain*.log –tail -n40 -f /opt/d-cache/libexec/apache-tomcat-5.5.20/logs/catalina.out  Only to debug SRM –tail -n 30 -f /opt/d-cache/billing/YYYY/MM/billing-YYYY.MM.DD –tail -n 30 -f /opt/d-cache/billing/YYYY/MM/billing-error-YYYY.MM.DD

28. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 28 Go on debugging… Look at: –http://www.gridpp.ac.uk/wiki/DCache_Log_Message_Archive –http://www.sysadmin.hep.ac.uk/wiki/DCache_Troubleshooting –Look at web monitoring pages

29. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 29 dCache new Release (1) Full SRM v2.2 support gPlazma authorization added. (For VOMS support) –GsiFtp and SRM understand extended proxies. Pools prepared to run on Windows XP dCap (client and server) now supports passive connections. [firewall issue is solved] Error type Fatal added. This allows for advanced actions (e-mail, sms, firealarm) dCap Door : improved permission handling FTP Door : Commands chmod and rmdir added Cost calcuation for multi I/O queues –fast cost prediction was added for multi I/O queues. Files can be automatically replicated on arrival in the dCache Pool to pool transfers pool destinations are treated seperately from 'read' pool selection. A set of important parameters can be now be defined 'per dCache partition'. SRM monitoring system xRoot protocol (as in 1.7.0) integrated as any other protocol

30. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 30 dCache new Release (2) bug fixed Support Multiple PNFS server on different machines dCap : large file problem fixed. dCap lib always opens local files with O_LARGEFILE. SRMCP Client: –Ensure reliable srmcp return codes. The return code is 0 only if all individual file transfers are successful. In case of any failure the return code is 1. –Added several new command line options: gss_expected_name, globus_tcp_port_range, streams_num, and server_mode.

31. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 31 dCache issues It is written in JAVA –CPU and memory issues The configuration of the advanced features is not so easy The documentation has been improved, but still the system is complex than not so easy! The support is on best-effort bases –User Forum really helpful The license is free but not completely Open- Source

32. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 32 dCache future plans New software instead of PNFS (Chimera) -> Will improve performances –Chimera : you may run 1.7.0 or 1.8 with Chimera Acl's available September for testing –Acl's in production end of the year (might be sooner) The StorageInfoQuotaObserver cell: advanced Quota support –Quotas will come with chimera nfs4.1 already in very good shape

33. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 33 Conclusions dCache is a complex system –GOOD:  Powerful System  Many Advanced functionality  Complete set of functionalities  Proved Scalability (at a tier1 level)  Easy portable to many software architecture (it works also on S.O. different from SLC3: i.e. Solaris) –BAD:  JAVA  Single Point of failure  Can be e little bit more difficult to manage (compared with DPM or Classic SE)

34. www.ccr.infn.it http://grid.infn.it/ Martina Franca (TA), 07 November 2007 - 34