1. iRODS hands-on tutorial Christine Staiger SURFsara Amsterdam, 8 th May 2014 8 th May 2014

2. The Basics ①Installation ②Tools for administration ③Adding data and metadata Complex storage systems ①Federations between data grids ①Resources and Compound resources Contents

3. The Basics

4. Demo Data Grids 10 Virtual machines: ssh user@IP 4 IP adressUserPassword 145.100.58.65irods01irods0104 145.100.57.78irods2irods204 145.100.57.82irods3irods304 145.100.57.83irods4irods404 145.100.58.67irods5irods504 145.100.57.86irods6irods604 145.100.57.87irods7irods704 145.100.57.91irods8irods804 145.100.57.107irods9irods904 145.100.57.110irods10irods1004

5. Installation 5 Execute./irodssetup in the folder iRODS Export the icommands export PATH=$PATH:/home/user/iRODS/clients/icommands/bin ~/iRODS/config/irods.config # Database configuration $DATABASE_TYPE = 'postgres'; … $DATABASE_HOST = 'localhost'; $DATABASE_PORT = '5432'; $DATABASE_ADMIN_PASSWORD = 'irods2'; $DATABASE_ADMIN_NAME = ’alice'; # iRODS configuration IRODS_HOME = '/home/alice/iRODS'; $IRODS_PORT = '1247’; … $IRODS_ADMIN_PASSWORD = ’alice'; $IRODS_ICAT_HOST = '';

6. iRODS i-commands Documentation: https://wiki.irods.org/index.php/icommands (we don’t provide a exhaustive list of i-commands) 6

7. iRODS Control Execute./irodsctl arg in the folder iRODS Restarting, starting and stopping the iRODS and iCAT server with arg = restart, start, stop Restarting, starting and stopping only the iRODS server with arg = irestart, istart, istop Initialise an account and log in with iinit

8. .irodsEnv file # iRODS personal configuration file. # # This file was automatically created during iRODS installation. # Created Mon Jun 18 13:13:27 2012 # # iRODS server host name: irodsHost ’surfsara-cloud' # iRODS server port number: irodsPort 1247 # Default storage resource name: irodsDefResource 'demoResc' # Home directory in iRODS: irodsHome '/alicesZone/home/alice' # Current directory in iRODS: irodsCwd '/alicesZone/home/alice' # Account name: irodsUserName ’alice' # Zone: irodsZone ‘alicesZone' 8 The.irodsEnv determines the zone and the user for the icommands-client. You can only have one such file at a time.

9. The icommands client icommands are automatically installed when you install the iRODS server Users will have to install them seperately to connect to an iRODS server Categories: Informational UNIX and FTP like Metadata Functional 9

10. ihelp 10 ihelp Lists all commands ihelp -h Usage : ihelp [-ah] [icommand] Display i-commands synopsis or a particular i-command help text Options are: -h this help -a print the help text for all the i-commands Run with no options to display a synopsis of the i-commands iRODS Version 3.3.1 February 2014 ihelp

11. ienv Lists the iRODS environment variables. Equivalent to “iinit -l” Usage : ienv [-h] 11 NOTICE: Release Version = rods3.3.1, API Version = d NOTICE: irodsHost=surfsara-cloud NOTICE: irodsPort=1247 NOTICE: irodsDefResource=demoResc NOTICE: irodsHome=/alicesZone/home/alice NOTICE: irodsCwd=/alicesZone/home/alice NOTICE: irodsUserName=alice NOTICE: irodsZone=alicesZone

12. iuserinfo Show information about your iRODS user account or the entered user Usage: iuserinfo [-vVh] [user] 12 name: alice id: 10007 type: rodsadmin zone: alicesZone info: comment: create time: 01396887567: 2014-04-07.18:19:27 modify time: 01396887567: 2014-04-07.18:19:27 member of group: alice member of group: public

13. imiscsvrinfo Connect to the server and retrieve some basic server information Can be used as a simple test for connecting to the server. Usage: imiscsrvinfo [-hvV] 13 RCAT_ENABLED relVersion=rods3.3.1 apiVersion=d rodsZone=alicesZone up 0 days, 5:54

14. ilsresc ilsresc lists iRODS resources and resource-groups Usage: ilsresc [-lvVhA] [Name] 14 alice@sara-cloud:~$ ilsresc demoResc alice@sara-cloud:~$ ilsresc –l demoResc resource name: demoResc resc id: 10010 zone: alicesZone type: unix file system class: archive location: surfsara-cloud vault: /home/alice/iRODS/Vault …

15. Some commands Unix-like: ils, ipwd, icd, imkdir, irm, icp ichmod, ipasswd, irsync, ichksum, imv FTP-like: iinit, iput, iget, iexit Test: Create folder in the logical name space ipwd imkdir NewFolder ils NewFolder irm –r NewFolder 15

16. icommands - ACLs 16 List ACLs (Access Control Lists) in a collection: alice@sara-cloud:~$ ils -A /alice/home/alice: ACL - alice#alicesZone:own Inheritance - Disabled

17. Administration 17 Change to admin mode: iadmin Quit admin mode: q Make/delete a user: mkuser, rmuser, moduser (iadmin) mkuser user1 rodsuser (iadmin) moduser user1 password ***** Parameters and their values: iadmin lt, iadmin lt user_type iadmin mkresc, iadmin rmresc Create new resource: mkdir Data (in your home, not in the iRODS environment) iadmin mkresc NewDataResc “unix file system” cache 145.100.57.139 “/home/alice/Data”

18. Put and replicate data 18 Add the file /home/login/put1.txt to resource iput –v put1.txt iput -h ichksum put1.txt (check sum of copy in Vault) md5sum put1.txt (check sum of file in /home/login) iput -R NewDataResc put2.txt Replicate data: irepl -R NewDataResc put1.txt Trim number of copies: itrim -S demoResc -N 1 put1.txt

19. Access Control Lists 19 Change permission for a user: ichmod read user1 put1.txt List access: ils -A Change permissions for group “public”: Make a directory in the iRODS environment: imkdir temp Make directory readable for “public”: ichmod -r read public temp ils –A /bobsZone/home/bob/temp: ACL - bob#bobsZone:own g:public#bobsZone:read object Inheritance - Disabled

20. iRODS Metadata Commands: imeta, iquest, idbo Add metadata: imeta add -d put1.txt “Date” “08.10.2013” imeta add -d put1.txt “AnotherDate” “04.04.2014” imeta ls -d put1.txt Remove metadata: imeta rm -d put1.txt “Date” “08.10.2013” Querying the iCAT: iquest "SELECT DATA_NAME,DATA_CHECKSUM WHERE DATA_RESC_NAME like ’%Resc%’” DATA_NAME = put1.txt DATA_CHECKSUM = c16d7d3488677a3348b12eb82795d28c 20

21. Complex storage systems

22. Federation between data grids List all existing zones: ils / Zone A acknowledges Zone B and vice-cersa iadmin mkzone B remote Host:Port Zone A adds remote user from Zone B: iadmin mkuser user#B rodsuser Example: Alice and Bob: Alice acknowledges Bob’s zone: iadmin mkzone bobsZone remote 145.100.57.74:1247 Bob acknowledges Alice’s zone: iadmin mkzone alicesZone remote 145.100.57.66:1247 Alice creates remote user Bob: iadmin mkuser bob#bobsZone rodsuser Now Bob can list and alter files in Alice’s zone: ils /alicesZone 22

23. Trouble shooting ERROR: putUtil: put error for /alicesZone/home/bob#bobsZone/put1.txt, status = -9000 status = -9000 SYS EXCEED CONNECT CNT Host configuration problem on the server. Solution: Edit iRODS/server/config/irodsHost localhost you.rIP.XX.XX surfsara-cloud localhost 12.0.0.1./irodsctl restart 23

24. Data Replication between grids Bob copies a file from his zone to his account in Alice’s zone irsync –R demoResc i:/bobsZone/home/bob/put1.txt i:/alicesZone/home/bob#bobsZone/put1.txt -R specifies the target resource Bob copies a frile from his home directory to Alice’s zone irsync –R demoResc put3.txt i:/alicesZone/home/bob#bobsZone/put3.txt Bob copies a file from Alice’s zone to his home directory mkdir test irsync i:/alicesZone/home/bob#bobsZone/put3.txt test/put3.txt Bob deletes a file in Alice’s zone irm /alicesZone/home/bob#bobsZone/put3.txt 24

25. Remote Resources Install iRODS, iCAT disabled Build an iRODS server [yes]? Make this Server ICAT-Enabled [yes]? no Host running iCAT-enabled iRODS server? 145.100.58.11 … Resource name [demoResc2]? myResc2 Resource storage area directory [/home/irodsadmin/iRODS/Vault]? /home/irodsadmin/iRODS/alicesVault … The irodsServer will authenticate to the other servers via an existing iRODS administrator account. Existing iRODS admin login name [rods]? alice Password [rods]? ******** Enter the zone as defined in your ICAT iRODS zone name [tempZone]? alicesZone

26. Resource Groups Exercise Resource Groups Create two data resources of type cache Group them: iadmin atrg groupName resourceName Deposit data in resource group: iput file groupName Add some data to the resource group Check with ils where the file is located virtually and physically How to automatically replicate data to resources in the same resource group?

27. iRODS Microservices Micro-services are small, well-defined procedures/functions that perform a simple task (https://www.irods.org/index.php/iRODS_Micro- services)https://www.irods.org/index.php/iRODS_Micro- services Examples: “msiCollCreate”, “msiDataObjRepl”, “msiDataObjChksum”, … To convert any C procedure (createCollection) into a microservice create an interface routine (msiCreateCollection) 27

28. iRODS Rules Example: replicate.r replicate{ msiWriteRodsLog("starting replication", *status); msiWriteRodsLog("source = *source", *status); msiDataObjRsync(*source, "IRODS_TO_IRODS", "null", *destination, *rsyncStatus); } INPUT *source="/bobsZone/home/bob/put3.txt", *destination="/alicesZone/home/bob#bobsZone/put3.txt" OUTPUT ruleExecOut  Execute irule -vF ~/rules/replicate.r  Check with ils /alicesZone/home/bob#bobsZone 28

29. Time delayed and periodic jobs Example test.r myTestRule{ delay(" 60s "){ msiWriteRodsLog("Hello World.", *status); } INPUT *A="1" OUTPUT ruleExecOut Writes the output to iRODS/server/log/reLog*, also check rodsLog* Use iqstat, iqdel and iqmod to list, delete and modify delayed rules. delay(” 5m EF>1h ”) Start job in 5 minutes and restart it every 1 hour Predefined rules: iRODS/server/config/reConfigs/core.re 29

30. Compound resources List resource types: iadmin lt resc_type A compound resource consists of cache and a storage resource (s3, MSS universal driver) Storage resource can be located on several different severs Example: Universal Mass Storage User writes data to cache, system replicates data to storage and deletes it from cache admin mkresc Cache "unix file system" cache 145.100.57.197 /home/alice/Cache iadmin mkresc mssStore 'MSS universal driver' compound 145.100.57.197 /home/alice/MSS iadmin atrg mssGroup Cache iadmin atrg mssGroup mssStore

31. Compound resources The mssStore can be of any hardware type which might require certain protocols to put and get data, e.g. gridFTP The resource type 'MSS universal driver’ triggers iRODS to look for a file iRODS/server/bin/cmd/univMSSInterface.sh to overwrite the put and get functions Template in iRODS/server/bin/cmd/examples Example file in TODO

32. The Universal Mass Storage Interface Logging: _log() LOGFILE=/home/irodsadmin/univMSSInterface.log  Change the path Copy between cache and storage: syncToArch(), stageToCache() _log 2 syncToArch "entering syncToArch()=$*" _log 2 syncToArch "executing: /bin/cp -f \"$1\" \"$2\"" /bin/cp -f "$1" "$2” error=$? if [ $error != 0 ] # copy failure then STATUS="FAILURE" fi _log 2 syncToArch "The status is $error ($STATUS)" return $error

33. Replicate data replicateDiskCache { delay(" 30s 15m ") { … *Condition = "DATA_RESC_NAME = '*CacheRescName' AND COLL_NAME like '*Collection%'"; msiMakeGenQuery("DATA_NAME, COLL_NAME, DATA_SIZE, order(DATA_CREATE_TIME)",*Condition,*Query2); msiExecGenQuery(*Query2,*List); while ( *ContInxOld > 0 ) { foreach ( *List ) { msiGetValByKey(*List,"DATA_NAME",*D); msiGetValByKey(*List,"COLL_NAME",*C); msiDataObjRepl("*C/*D","backupRescName=*PnfsRescName++++ rescName=*CacheRescName++++irodsAdmin=",*status)} } }}

34. Replicate data Save as YourFilename.re in iRODS/server/config/reConfigs/ Edit server/config/server.config reRuleSet core,YourFilename irule replicateDiskCache "*Collection=/yourZone%*PnfsRescName= yourMssStore%*CacheRescName=yourCache" ruleExecOut alice@surfsara-cloud:~/iRODS$ iput -R mssGroup../put1.txt alice@surfsara-cloud:~/iRODS$ ils -l /alicesZone/home/alice: alice 0 Cache 7 2014-04-24.09:26 & put1.txt Check with iqstat -l when the rule will be executed Some time later: alice@surfsara-cloud:~/iRODS$ ils -l /alicesZone/home/alice: alice 0 Cache 7 2014-04-24.09:26 & put1.txt alice 1 mssStore 7 2014-04-24.09:36 & put1.txt No space!

35. Purge Cache Write a rule to purge the cache Purge cache when data is replicated and if not enough space left in cache Call by: irule purgeDiskCache "*Collection=/testZone%*PnfsRescName=mssStore %*CacheRescName=Cache%*MaxSpAlwdTBstring=2" ruleExecOut MaxSpAlwdTBstring: the size when to purge, string

36. Thank you !