iRODS hands-on tutorial Christine Staiger SURFsara Amsterdam, 8 th May th May 2014

The Basics ①Installation ②Tools for administration ③Adding data and metadata Complex storage systems ①Federations between data grids ①Resources and Compound resources Contents

The Basics

Demo Data Grids 10 Virtual machines: ssh 4 IP adressUserPassword irods01irods irods2irods irods3irods irods4irods irods5irods irods6irods irods7irods irods8irods irods9irods irods10irods1004

Installation 5 Execute./irodssetup in the folder iRODS Export the icommands export PATH=$PATH:/home/user/iRODS/clients/icommands/bin ~/iRODS/config/irods.config # Database configuration $DATABASE_TYPE = 'postgres'; … $DATABASE_HOST = 'localhost'; $DATABASE_PORT = '5432'; $DATABASE_ADMIN_PASSWORD = 'irods2'; $DATABASE_ADMIN_NAME = ’alice'; # iRODS configuration IRODS_HOME = '/home/alice/iRODS'; $IRODS_PORT = '1247’; … $IRODS_ADMIN_PASSWORD = ’alice'; $IRODS_ICAT_HOST = '';

iRODS i-commands Documentation: (we don’t provide a exhaustive list of i-commands) 6

iRODS Control Execute./irodsctl arg in the folder iRODS Restarting, starting and stopping the iRODS and iCAT server with arg = restart, start, stop Restarting, starting and stopping only the iRODS server with arg = irestart, istart, istop Initialise an account and log in with iinit

.irodsEnv file # iRODS personal configuration file. # # This file was automatically created during iRODS installation. # Created Mon Jun 18 13:13: # # iRODS server host name: irodsHost ’surfsara-cloud' # iRODS server port number: irodsPort 1247 # Default storage resource name: irodsDefResource 'demoResc' # Home directory in iRODS: irodsHome '/alicesZone/home/alice' # Current directory in iRODS: irodsCwd '/alicesZone/home/alice' # Account name: irodsUserName ’alice' # Zone: irodsZone ‘alicesZone' 8 The.irodsEnv determines the zone and the user for the icommands-client. You can only have one such file at a time.

The icommands client icommands are automatically installed when you install the iRODS server Users will have to install them seperately to connect to an iRODS server Categories: Informational UNIX and FTP like Metadata Functional 9

ihelp 10 ihelp Lists all commands ihelp -h Usage : ihelp [-ah] [icommand] Display i-commands synopsis or a particular i-command help text Options are: -h this help -a print the help text for all the i-commands Run with no options to display a synopsis of the i-commands iRODS Version February 2014 ihelp

ienv Lists the iRODS environment variables. Equivalent to “iinit -l” Usage : ienv [-h] 11 NOTICE: Release Version = rods3.3.1, API Version = d NOTICE: irodsHost=surfsara-cloud NOTICE: irodsPort=1247 NOTICE: irodsDefResource=demoResc NOTICE: irodsHome=/alicesZone/home/alice NOTICE: irodsCwd=/alicesZone/home/alice NOTICE: irodsUserName=alice NOTICE: irodsZone=alicesZone

iuserinfo Show information about your iRODS user account or the entered user Usage: iuserinfo [-vVh] [user] 12 name: alice id: type: rodsadmin zone: alicesZone info: comment: create time: : :19:27 modify time: : :19:27 member of group: alice member of group: public

imiscsvrinfo Connect to the server and retrieve some basic server information Can be used as a simple test for connecting to the server. Usage: imiscsrvinfo [-hvV] 13 RCAT_ENABLED relVersion=rods3.3.1 apiVersion=d rodsZone=alicesZone up 0 days, 5:54

ilsresc ilsresc lists iRODS resources and resource-groups Usage: ilsresc [-lvVhA] [Name] 14 ilsresc demoResc ilsresc –l demoResc resource name: demoResc resc id: zone: alicesZone type: unix file system class: archive location: surfsara-cloud vault: /home/alice/iRODS/Vault …

Some commands Unix-like: ils, ipwd, icd, imkdir, irm, icp ichmod, ipasswd, irsync, ichksum, imv FTP-like: iinit, iput, iget, iexit Test: Create folder in the logical name space ipwd imkdir NewFolder ils NewFolder irm –r NewFolder 15

icommands - ACLs 16 List ACLs (Access Control Lists) in a collection: ils -A /alice/home/alice: ACL - alice#alicesZone:own Inheritance - Disabled

Administration 17 Change to admin mode: iadmin Quit admin mode: q Make/delete a user: mkuser, rmuser, moduser (iadmin) mkuser user1 rodsuser (iadmin) moduser user1 password ***** Parameters and their values: iadmin lt, iadmin lt user_type iadmin mkresc, iadmin rmresc Create new resource: mkdir Data (in your home, not in the iRODS environment) iadmin mkresc NewDataResc “unix file system” cache “/home/alice/Data”

Put and replicate data 18 Add the file /home/login/put1.txt to resource iput –v put1.txt iput -h ichksum put1.txt (check sum of copy in Vault) md5sum put1.txt (check sum of file in /home/login) iput -R NewDataResc put2.txt Replicate data: irepl -R NewDataResc put1.txt Trim number of copies: itrim -S demoResc -N 1 put1.txt

Access Control Lists 19 Change permission for a user: ichmod read user1 put1.txt List access: ils -A Change permissions for group “public”: Make a directory in the iRODS environment: imkdir temp Make directory readable for “public”: ichmod -r read public temp ils –A /bobsZone/home/bob/temp: ACL - bob#bobsZone:own g:public#bobsZone:read object Inheritance - Disabled

iRODS Metadata Commands: imeta, iquest, idbo Add metadata: imeta add -d put1.txt “Date” “ ” imeta add -d put1.txt “AnotherDate” “ ” imeta ls -d put1.txt Remove metadata: imeta rm -d put1.txt “Date” “ ” Querying the iCAT: iquest "SELECT DATA_NAME,DATA_CHECKSUM WHERE DATA_RESC_NAME like ’%Resc%’” DATA_NAME = put1.txt DATA_CHECKSUM = c16d7d a3348b12eb82795d28c 20

Complex storage systems

Federation between data grids List all existing zones: ils / Zone A acknowledges Zone B and vice-cersa iadmin mkzone B remote Host:Port Zone A adds remote user from Zone B: iadmin mkuser user#B rodsuser Example: Alice and Bob: Alice acknowledges Bob’s zone: iadmin mkzone bobsZone remote :1247 Bob acknowledges Alice’s zone: iadmin mkzone alicesZone remote :1247 Alice creates remote user Bob: iadmin mkuser bob#bobsZone rodsuser Now Bob can list and alter files in Alice’s zone: ils /alicesZone 22

Trouble shooting ERROR: putUtil: put error for /alicesZone/home/bob#bobsZone/put1.txt, status = status = SYS EXCEED CONNECT CNT Host configuration problem on the server. Solution: Edit iRODS/server/config/irodsHost localhost you.rIP.XX.XX surfsara-cloud localhost /irodsctl restart 23

Data Replication between grids Bob copies a file from his zone to his account in Alice’s zone irsync –R demoResc i:/bobsZone/home/bob/put1.txt i:/alicesZone/home/bob#bobsZone/put1.txt -R specifies the target resource Bob copies a frile from his home directory to Alice’s zone irsync –R demoResc put3.txt i:/alicesZone/home/bob#bobsZone/put3.txt Bob copies a file from Alice’s zone to his home directory mkdir test irsync i:/alicesZone/home/bob#bobsZone/put3.txt test/put3.txt Bob deletes a file in Alice’s zone irm /alicesZone/home/bob#bobsZone/put3.txt 24

Remote Resources Install iRODS, iCAT disabled Build an iRODS server [yes]? Make this Server ICAT-Enabled [yes]? no Host running iCAT-enabled iRODS server? … Resource name [demoResc2]? myResc2 Resource storage area directory [/home/irodsadmin/iRODS/Vault]? /home/irodsadmin/iRODS/alicesVault … The irodsServer will authenticate to the other servers via an existing iRODS administrator account. Existing iRODS admin login name [rods]? alice Password [rods]? ******** Enter the zone as defined in your ICAT iRODS zone name [tempZone]? alicesZone

Resource Groups Exercise Resource Groups Create two data resources of type cache Group them: iadmin atrg groupName resourceName Deposit data in resource group: iput file groupName Add some data to the resource group Check with ils where the file is located virtually and physically How to automatically replicate data to resources in the same resource group?

iRODS Microservices Micro-services are small, well-defined procedures/functions that perform a simple task ( services) services Examples: “msiCollCreate”, “msiDataObjRepl”, “msiDataObjChksum”, … To convert any C procedure (createCollection) into a microservice create an interface routine (msiCreateCollection) 27

iRODS Rules Example: replicate.r replicate{ msiWriteRodsLog("starting replication", *status); msiWriteRodsLog("source = *source", *status); msiDataObjRsync(*source, "IRODS_TO_IRODS", "null", *destination, *rsyncStatus); } INPUT *source="/bobsZone/home/bob/put3.txt", *destination="/alicesZone/home/bob#bobsZone/put3.txt" OUTPUT ruleExecOut  Execute irule -vF ~/rules/replicate.r  Check with ils /alicesZone/home/bob#bobsZone 28

Time delayed and periodic jobs Example test.r myTestRule{ delay(" 60s "){ msiWriteRodsLog("Hello World.", *status); } INPUT *A="1" OUTPUT ruleExecOut Writes the output to iRODS/server/log/reLog*, also check rodsLog* Use iqstat, iqdel and iqmod to list, delete and modify delayed rules. delay(” 5m EF>1h ”) Start job in 5 minutes and restart it every 1 hour Predefined rules: iRODS/server/config/reConfigs/core.re 29

Compound resources List resource types: iadmin lt resc_type A compound resource consists of cache and a storage resource (s3, MSS universal driver) Storage resource can be located on several different severs Example: Universal Mass Storage User writes data to cache, system replicates data to storage and deletes it from cache admin mkresc Cache "unix file system" cache /home/alice/Cache iadmin mkresc mssStore 'MSS universal driver' compound /home/alice/MSS iadmin atrg mssGroup Cache iadmin atrg mssGroup mssStore

Compound resources The mssStore can be of any hardware type which might require certain protocols to put and get data, e.g. gridFTP The resource type 'MSS universal driver’ triggers iRODS to look for a file iRODS/server/bin/cmd/univMSSInterface.sh to overwrite the put and get functions Template in iRODS/server/bin/cmd/examples Example file in TODO

The Universal Mass Storage Interface Logging: _log() LOGFILE=/home/irodsadmin/univMSSInterface.log  Change the path Copy between cache and storage: syncToArch(), stageToCache() _log 2 syncToArch "entering syncToArch()=$*" _log 2 syncToArch "executing: /bin/cp -f \"$1\" \"$2\"" /bin/cp -f "$1" "$2” error=$? if [ $error != 0 ] # copy failure then STATUS="FAILURE" fi _log 2 syncToArch "The status is $error ($STATUS)" return $error

Replicate data replicateDiskCache { delay(" 30s 15m ") { … *Condition = "DATA_RESC_NAME = '*CacheRescName' AND COLL_NAME like '*Collection%'"; msiMakeGenQuery("DATA_NAME, COLL_NAME, DATA_SIZE, order(DATA_CREATE_TIME)",*Condition,*Query2); msiExecGenQuery(*Query2,*List); while ( *ContInxOld > 0 ) { foreach ( *List ) { msiGetValByKey(*List,"DATA_NAME",*D); msiGetValByKey(*List,"COLL_NAME",*C); msiDataObjRepl("*C/*D","backupRescName=*PnfsRescName++++ rescName=*CacheRescName++++irodsAdmin=",*status)} } }}

Replicate data Save as YourFilename.re in iRODS/server/config/reConfigs/ Edit server/config/server.config reRuleSet core,YourFilename irule replicateDiskCache "*Collection=/yourZone%*PnfsRescName= yourMssStore%*CacheRescName=yourCache" ruleExecOut iput -R mssGroup../put1.txt ils -l /alicesZone/home/alice: alice 0 Cache :26 & put1.txt Check with iqstat -l when the rule will be executed Some time later: ils -l /alicesZone/home/alice: alice 0 Cache :26 & put1.txt alice 1 mssStore :36 & put1.txt No space!

Purge Cache Write a rule to purge the cache Purge cache when data is replicated and if not enough space left in cache Call by: irule purgeDiskCache "*Collection=/testZone%*PnfsRescName=mssStore %*CacheRescName=Cache%*MaxSpAlwdTBstring=2" ruleExecOut MaxSpAlwdTBstring: the size when to purge, string

Thank you !