Workflows over Grid-based Web services General framework and a practical case in structural biology gLite 3.0 Data Management Hands-on David García Aristegui Grid Applications Developer CNB/CSIC

Set up your environment Check the following environment variables, used to specify the catalog type and its location, are set up properly: export LCG_CATALOG_TYPE=lfc export LCG_GFAL_VO=biomed export LFC_HOST=cclcglfcli02.in2p3.fr Ensure you have created a proxy certificate and it is still valid. If not create it by: voms-proxy-init --voms biomed voms-proxy-info -all

lcg-infosites lcg-infosites: no manual entry for this command –lcg-infosites --help Is a very useful command to interact with the EGEE Information System Execution: lcg-infosites –vo biomed (se | ce | rb | all ) We detect the resources for a certain VO with lcg-infosites Biomed SEs? –lcg-infosites –vo biomed se BDII value: david]$ env | grep BDII LCG_BDII_INFOSYS=lcg-bdii.cern.ch:2170

Listing the entries of a LFC directory lfc-ls [-cdiLlRTu] [--comment] path… where path specifies the LFC pathname (mandatory) –Remember that LFC has a directory tree structure – /grid/biomed/ –All members of a given VO have read-write permissions under their directory –-l (it is a lowercase “L”) outputs long listing –-R lists the contents of directories recursively (don’t use it so often!) LFC Catalog commands Defined by the userLFC Namespace

lfc-* commands exercices FIRST TESTS –lfc-ls /grid/ –lfc-ls /grid/biomed –lfc-ls /grid/biomed/david MORE OPTIONS??? –type “man lfc-ls” ;-) OTHER OPTIONAL ENV. VARIABLE –export LFC_HOME=/grid/biomed –then execute lfc-ls GENERATING A DIRECTORY –lfc-mkdir /grid/biomed/YOURUSERNAME

lcg-utils: lcg-cr Upload a file to a SE and register it into the catalog lcg-cr -d dest_file | dest_host -l lfn [-g guid] [-l lfn] [-v | --verbose] --vo vo src_file where –dest_host is the fully qualified hostname of the destination SE –dest_file is a valid SURL (both sfn:// or srm:// format are valid) –guid specifies the Grid Unique IDentifier. If this option is not present, a GUID is generated internally –lfn specifies the Logical File Name associated with the file –vo specifies the Virtual Organization the user belongs to –src_file specifies the source file name: the protocol can be file:/// or gsiftp:///

lfc-* and lcg-* commands exercices HELLO WORLD, ONCE AGAIN –generate a hellowordl.txt in your UI home (echo “hello” > helloword.txt) –man lcg-cr –lcg-cr --vo biomed -d baudelaire.cnb.uam.es file:///home/david/helloworld.txt guid:76fd3066-c5cf a8-7ce08a2e9f1c –lfc-ls /grid/biomed/ ??? –man lcg-cp –lcg-cp --vo biomed guid:76fd3066-c5cf a8-7ce08a2e9f1c file:///tmp/toto DEFAULT SE david]$ env | grep BIOMED VO_BIOMED_DEFAULT_SE=baudelaire.cnb.uam. –lcg-cr --vo biomed file:///home/david/helloworld.txt OTHER SEs? –remember lcg-infosites!!!

Scared of GUIDs? More lfc/lcg usages WORKING WITH LFNs –generate a ihateguids.txt in your UI home (echo “hate” > ihateguids.txt) –lcg-cr --vo biomed -d baudelaire.cnb.uam.es file:///home/david/ihateguids.txt -l lfn:///grid/biomed/david/ihateguids.txt (all in the same line!!!)file:///home/david/ihateguids.txt –lfc-ls /grid/biomed/david ??? –lcg-cp --vo biomed lfn:/grid/biomed/david/ihateguids.txt file:///tmp/hate DELETE A FILE –man lcg-del (pay attention to replicas, later) –lcg-del --vo biomed -a lfn:/grid/biomed/directory/file DELETE A DIRECTORY –lfc-rm -r /grid/biomed/BECAREFUL

Replicas and replication Replication –man lcg-rep –lcg-rep --vo biomed -d cluster.pnpi.nw.ru lfn:/grid/biomed/david/text_file.txt –lcg-* commands are very shy. There is no output from some commands on success (lcg-rep). You can check that the replica was created by listing all the replicas of your file, that is done by using the LCG list replicas command david]$ lcg-lr --vo biomed lfn:/grid/biomed/david/text_file.txt sfn://baudelaire.cnb.uam.es/storage/biomed/generated/ /fileda45e711-f69d e28-b6e85335e48e srm://cluster.pnpi.nw.ru/dpm/pnpi.nw.ru/home/biomed/generated/ /file521a02b3-12b4-4af7-82bc c9 Symbolic links –man lfc-ln –lfc-ln -s /grid/biomed/david/text_file.txt /grid/biomed/david/text_file_symlink.txt –lfc-ls -l /grid/biomed/david ???

DIY: do-it-yourself Exercice –Create a new directory –Put two files in it –Replicate the first one in two SEs –Create a symbolic link –Add a comment in one file. Clues: lfc-setcomment /grid/biomed/etc/etc lfc-ls –comment /grid/biomed/etc/etc Any questions? –Theory? –Hands-on?