Introduction to iRODS Christine Staiger SURFsara 10 th Oct 2014

iRODS Concepts The Basics ① Installation ② Tools for administration ③ Adding data and metadata ④ iRODS rules Contents

iRODS Concepts 3

What is iRODS? 4

What does iRODS provide? Storage virtualization of different disk and tape storage systems A logical namespace across storage locations A rule engine to automate data management according to defined policies Various client interfaces 5

iRODS Components 6

In the background: iRODS Resources (Storage) Resource is a Software or Hardware system that stores data 3 Resource classes: 7 High Latency High Latency Low Latency Low Latency Archive Resource Archive Resource Cache Resource Cache Resource Compound Resource Compound Resource POSIX like IO calls

Resource Groups: Distributing files 8 Low Latency Low Latency Cache Resource Cache Resource POSIX like IO calls Resource Group Resource Group Low Latency Low Latency Cache Resource Cache Resource Low Latency Low Latency Cache Resource Cache C

Resource Groups: Data replication 9 Low Latency Low Latency Cache Resource Cache Resource POSIX like IO calls Resource Group Resource Group Low Latency Low Latency Cache Resource Cache Resource Low Latency Low Latency Cache Resource Cache C Replicate

The users view: iRODS Data Collections 10 Collection 0 Data Object Data Object Data Object Data Object A Collection 1 Data Object Data Object Data Object X …/Collection0/ …/Collection0/DataObject A-B …/Collection0/Collection1/ …/Collection0/Collection1/DataObject X-Z Logical Name Space (iCAT) independent from physical resources Similar to Unix Directory and Files ichmod to set user/group ACLs Core.re: msiSetDefaultResc(Resource).irodsEnv: irodsDefResource=Resource Similar to Unix Directory and Files ichmod to set user/group ACLs Core.re: msiSetDefaultResc(Resource).irodsEnv: irodsDefResource=Resource

Hands on The Basics 11

Demo Data Grids Virtual machines: ssh IP adressUserPassword irods1irods irods2irods irods3irods irods4irods irods6irods irods7irods irods8irods irods9irods irods10irods1004

Installation and configuration iRODS3: Install script./irodssetup iRODS4: (rpm, dmg) package and source code ~/iRODS/config/irods.config # Database configuration $DATABASE_TYPE = 'postgres'; … $DATABASE_HOST = 'localhost'; $DATABASE_PORT = '5432'; $DATABASE_ADMIN_PASSWORD = 'irods2'; $DATABASE_ADMIN_NAME = ’alice'; # iRODS configuration IRODS_HOME = '/home/alice/iRODS'; $IRODS_PORT = '1247’; … $IRODS_ADMIN_PASSWORD = ’alice'; $IRODS_ICAT_HOST = '';

.irodsEnv file # iRODS personal configuration file. # # This file was automatically created during iRODS installation. # Created Mon Jun 18 13:13: # # iRODS server host name: irodsHost ’surfsara-cloud' # iRODS server port number: irodsPort 1247 # Default storage resource name: irodsDefResource 'demoResc' # Home directory in iRODS: irodsHome '/alicesZone/home/alice' # Current directory in iRODS: irodsCwd '/alicesZone/home/alice' # Account name: irodsUserName ’alice' # Zone: irodsZone ‘alicesZone' The.irodsEnv determines the zone and the user for the icommands-client. You can only use one such file at a time.

iRODS i-commands Documentation: (we don’t provide a exhaustive list of i-commands)

iRODS control Execute./irodsctl in the folder iRODS Restarting, starting and stopping the iRODS and iCAT server with = restart, start, stop Restarting, starting and stopping only the iRODS server with = irestart, istart, istop Initialise an account and log in with iinit 16

ilsresc ilsresc lists iRODS resources and resource-groups Usage: ilsresc [-lvVhA] [Name] ilsresc demoResc ilsresc –l demoResc resource name: demoResc resc id: zone: alicesZone type: unix file system class: archive location: surfsara-cloud vault: /home/alice/iRODS/Vault …

User and Data administration Change to admin mode: iadmin Quit admin mode: q iadmin mkresc, iadmin rmresc Create new resource: mkdir Data (in your home, not in the iRODS environment) iadmin mkresc “unix file system” archive you.rIP.add.ress “/home/ /Data” Make/delete a user: mkuser, rmuser, moduser (iadmin) mkuser user1 rodsuser (iadmin) moduser user1 password ***** Parameters and their values: iadmin lt, iadmin lt user_type

Put and replicate data Add the file /home/login/put1.txt to resource iput -v put1.txt ils -l or ils -L (gives also physical location) iput -h Create a collection and add data imkdir / /archive iput put2.txt / /archive/put2.txt Check physical location of the data ils -L / /archive Replicate data across resources: irepl -R put1.txt Check with ils -L Trim number of copies: itrim -S -N 1 put1.txt

iRODS Metadata Commands: imeta, iquest, idbo Add metadata: imeta add -d put1.txt “Date” “ ” imeta add -d put1.txt “AnotherDate” “ ” imeta ls -d put1.txt Remove metadata: imeta rm -d put1.txt “Date” “ ” Querying the iCAT: iquest "SELECT DATA_NAME,DATA_CHECKSUM WHERE DATA_RESC_NAME like ’%Resc%’” DATA_NAME = put1.txt DATA_CHECKSUM = c16d7d a3348b12eb82795d28c

iRODS Microservices and rules Micro-services are small, well-defined procedures/functions that perform a simple task Examples: “msiCollCreate”, “msiDataObjRepl”, “msiDataObjChksum”, … Convert any C procedure (createCollection) to a microservice by creating an interface routine (msiCreateCollection) Micro-services can be combined in rules to enforce policies Rules can be executed on action, with time-delay or periodically

iRODS Rule Engine 22

Replicate to another resource replicate{ msiWriteRodsLog("starting replication", *status); msiWriteRodsLog("source = *source", *status); msiDataObjRepl(*source, *destRescName, *CC); } INPUT *source=”/ /home/ /put1.txt”, *destRescName=“testResc” OUTPUT ruleExecOut  Save as replicate.r  Execute irule -vF replicate.r  Check with ils -L

Replicate to another collection Useful for backup of a data collection Changing the storage medium iput put2.txt replicate{ msiWriteRodsLog("starting replication", *status); msiWriteRodsLog("source = *source", *status); msiDataObjRsync(*source, "IRODS_TO_COLLECTION", *destRescName, *destination, *rsyncStatus); } INPUT *source="/ /home/ /put2.txt", *destination=”/ /coll1", *destRescName=“ " OUTPUT ruleExecOut ils -L /irodsXZone/coll1/home/irodsX

Thank you !

Appendix

Trouble shooting ERROR: putUtil: put error for /alicesZone/home/bob#bobsZone/put1.txt, status = status = SYS EXCEED CONNECT CNT Host configuration problem on the server. Solution: Edit iRODS/server/config/irodsHost localhost you.rIP.XX.XX surfsara-cloud localhost /irodsctl restart