1. FP6−2004−Infrastructures−6-SSA-026409 www.eu-eela.org E-infrastructure shared between Europe and Latin America Data Management Hands-on Juan Eduardo Murrieta León DGSCA - UNAM Thirteenth EELA Tutorial La Antigua, 19.10.2007

2. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 2 La Antigua, 13th EELA Tutorial, 19.10.2007 Outline Introduction to LCG File Catalogue (LFC) LFC and lcg_utils hands-on

3. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 3 La Antigua, 13th EELA Tutorial, 19.10.2007 File Catalog Motivations Users and programs produce and require data Data may be stored in Grid datasets (files) –Located in Storage Elements (SEs) –Several replicas of one file in different sites –Accessible by Grid users and applications from “anywhere” –Locatable by the WMS (data requirements in JDL) –Data may be copied from/to local filesystems (WNs, UIs) to the Grid Why File Catalogs are used? –They keep track of the location of copies (replicas) of Grid files –They provide DM tools and APIs through which users and grid services can deal with files

4. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 4 La Antigua, 13th EELA Tutorial, 19.10.2007 Logical File Name (LFN) –An alias created by a user to refer to some item of data, e.g. “lfn:cms/20030203/run2/track1” Globally Unique Identifier (GUID) –A non-human-readable unique identifier for an item of data, e.g. “guid:f81d4fae-7dec-11d0-a765-00a0c91e6bf6” Site URL (SURL) (or Physical File Name (PFN) or Site FN) –The location of an actual piece of data on a storage system, e.g. “srm://pcrd24.cern.ch/flatfiles/cms/output10_1” (SRM) “sfn://lxshare0209.cern.ch/data/alice/ntuples.dat” (Classic SE) Transport URL (TURL) –Temporary locator of a replica + access protocol: understood by a SE, e.g. “rfio://lxshare0209.cern.ch//data/alice/ntuples.dat” Naming conventions

5. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 5 La Antigua, 13th EELA Tutorial, 19.10.2007 LFC features –Hierarchical namespace –Integrated GSI Authentication + Authorization –Access Control Lists (Unix Permissions and POSIX ACLs) LFN acts as main key in the database. It has: –Symbolic links to it –Unique Identifier (GUID) –System metadata –Information on replicas –One field of user metadata LFC (LCG File Catalog)

6. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 6 La Antigua, 13th EELA Tutorial, 19.10.2007 Practice on LFC and lcg-utils

7. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 7 La Antigua, 13th EELA Tutorial, 19.10.2007 Set up your environment Check that the following environment variables, used to specify the catalog type and its location, are set up properly: – export LCG_CATALOG_TYPE=lfc – export LFC_HOST=lfc-gilda.ct.infn.it Ensure you have created a proxy certificate and it is still valid. If not create it by: – voms-proxy-init --voms gilda Remember: The Passphrase is ANTIGUA

8. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 8 La Antigua, 13th EELA Tutorial, 19.10.2007 Listing the entries of a LFC directory lfc-ls [-cdiLlRTu] [--comment] path… where path specifies the LFC pathname (mandatory) –Remember that LFC has a directory tree structure –/grid/ / –All members of a given VO have read-write permissions under their directory –-l (it is a lowercase “L”) outputs long listing –-R lists the contents of directories recursively (don’t use it so often!) –You can set LFC_HOME to use relative paths LFC_HOME=/grid/gilda/myDir  /grid/gilda/myDir/myFile becomes myFile LFC Catalog commands Defined by the user LFC Namespace

9. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 9 La Antigua, 13th EELA Tutorial, 19.10.2007 lfc-ls examples $ lfc-ls –l /grid/gilda/tutorial [antigua01]$ lfc-ls -l /grid/gilda/tutorial drwxrwxr-x 0 107 101 0 Sep 06 18:37 aduarte -rw-rw-r-- 1 107 101 18 Sep 06 18:35 anotherfile drwxrwxr-x 0 107 101 0 Sep 06 18:38 grunfeld drwxrwxr-x 0 107 101 0 Sep 06 18:37 jcmartinez drwxrwxr-x 0 107 101 0 Sep 06 18:37 matias -rw-rw-r-- 1 107 101 18 Sep 06 18:33 note.txt -rw-rw-r-- 1 107 101 18 Sep 06 18:35 onefile drwxrwxr-x 0 107 101 0 Sep 06 18:37 rausch drwxrwxr-x 0 107 101 0 Sep 06 18:37 vicinanza -rw-rw-r-- 1 107 101 18 Sep 06 18:36 yetanotherfile $ export LFC_HOME=/grid/gilda/tutorial/antigua $ lfc-ls –l note.txt [rausch@santiago05 rausch]$ export LFC_HOME=/grid/gilda/tutorial/antigua [rausch@santiago05 rausch]$ lfc-ls -l note.txt -rw-rw-r-- 1 107 101 18 Sep 06 18:33 /grid/gilda/tutorial/antigua/note.txt

10. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 10 La Antigua, 13th EELA Tutorial, 19.10.2007 LFC Catalog commands Creating a symbolic link lfc-ln -s file linkname lfc-ln -s directory linkname Create a link to the specified file or directory with linkname –Example: $ lfc-ln -s /grid/gilda/tutorial/antigua/note.txt /grid/gilda/tutorial/antigua/linkToNote.txt Let’s check the link using lfc-ls with long listing (-l) $ lfc-ls -l /grid/gilda/tutorial/antigua lrwxrwxrwx 1 101 102 Oct 19 01:32 linkToNote.txt -> /grid/gilda/tutorial/antigua/note.txt

11. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 11 La Antigua, 13th EELA Tutorial, 19.10.2007 LFC Catalog commands Creating directories in the LFC lfc-mkdir [-m mode] [-p] path... Where path specifies the LFC pathname Remember that while registering a new file (using lcg-cr, for example) the corresponding destination directory must be already created in the catalog Examples: $ lfc-mkdir /grid/gilda/tutorial/antigua/antiguaXX You can just check the directory with: $ lfc-ls -l /grid/gilda/tutorial/antigua

12. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 12 La Antigua, 13th EELA Tutorial, 19.10.2007 LFC Catalog commands Adding/deleting metadata information lfc-setcomment path comment lfc-delcomment path lfc-setcomment adds/replaces a comment associated with a file/directory in the LFC Catalog lfc-delcomment deletes a comment previously added Example: lfc-setcomment /grid/gilda/tutorial/note.txt ‘Hello World!’ Check your job with lfc-ls --comment /grid/gilda/tutorial/note.txt /grid/gilda/tutorial/note.txt Hello World!

13. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 13 La Antigua, 13th EELA Tutorial, 19.10.2007 Example: lfc-delcomment /grid/gilda/tutorial/antigua/note.txt Check if it was successful with lfc-ls –l --comment /grid/gilda/tutorial/antigua/note.txt -rw-rw-r-- 1 101 102 13 Apr 03 11:23 /grid/gilda/tutorial/antigua/note.txt LFC Catalog commands

14. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 14 La Antigua, 13th EELA Tutorial, 19.10.2007 Hands-on Session Exercise No.1: –Log into the UI and initialize your proxy credentials if not already done –check that your environment variables are properly set up to use the EELA catalog –have a look inside the catalog –create a sub-directory, under /grid/gilda/tutorial/antigua, with your surname –put inside the just created dir a link to an existing file (from files in other directories) –add a comment to that file and verify it

15. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 15 La Antigua, 13th EELA Tutorial, 19.10.2007 LFC Catalog commands lfc-chmodChange access mode of the LFC file/directory lfc-chownChange owner and group of the LFC file-directory lfc-delcommentDelete the comment associated with the file/directory lfc-getaclGet file/directory access control lists lfc-lnMake a symbolic link to a file/directory lfc-lsList file/directory entries in a directory lfc-mkdirCreate a directory lfc-renameRename a file/directory lfc-rmRemove a file/directory lfc-setaclSet file/directory access control lists lfc-setcommentAdd/replace a comment Summary of the LFC Catalog commands

16. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 16 La Antigua, 13th EELA Tutorial, 19.10.2007 lcg-utils The LCG Data Management tools (usually called lcg-utils) allow users to copy files between UI, CE, WN and a SE, to register entries in the File Catalogs and replicate files between SEs. Check if LCG_GFAL_INFOSYS environment variable is correctly set to the local GILDA Information Index (BDII) – echo $LCG_GFAL_INFOSYS grid004.ct.infn.it:2170

17. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 17 La Antigua, 13th EELA Tutorial, 19.10.2007 lcg-utils: lcg-cr Upload a file to a SE and register it into the catalog lcg-cr -d dest_file | dest_host -l lfn [-g guid] [-l lfn] [-v | --verbose] --vo vo_name src_file where –dest_host is the fully qualified hostname of the destination SE –dest_file is a valid SURL (both sfn:// or srm:// format are valid) –guid specifies the Grid Unique IDentifier. If this option is not present, a GUID is generated internally –lfn specifies the Logical File Name associated with the file –vo specifies the Virtual Organization the user belongs to –src_file specifies the source file name: the protocol can be file:/// or gsiftp:/// The command help is misleading: -d –l and --vo are MANDATORY!

18. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 18 La Antigua, 13th EELA Tutorial, 19.10.2007 lcg-utils: lcg-cr To discover which SEs the user is allowed to use, you can use the lcg-infosites command. lcg-infosites --vo gilda se The output is a list of SEs and related information on available/used space lcg-cr usage example: [antigua01]$ lcg-cr –d grid005.iucc.ac.il -l lfn:/grid/gilda/tutorial/message.txt -v --vo gilda file:///home/antigua01/message.txt [antigua39@glite-tutor antigua39]$ lcg-cr -d grid005.iucc.ac.il -l lfn:/grid/gilda/tutorial/message.txt -v --vo gilda file://$(pwd)/message.txt Using grid catalog type: lfc Using grid catalog : lfc-gilda.ct.infn.it Using LFN : /grid/gilda/tutorial/message.txt Using SURL : sfn://grid005.iucc.ac.il/storage/gilda/generated/2007-10-19/file910cc4ec-a0e9-4e8a-8e6b- b66d20d22862 Source URL: file:///home/antigua39/message.txt File size: 12 VO name: gilda Destination specified: grid005.iucc.ac.il Destination URL for copy: gsiftp://grid005.iucc.ac.il/storage/gilda/generated/2007-10-19/file910cc4ec-a0e9- 4e8a-8e6b-b66d20d22862 Alias registered in Catalog: lfn:/grid/gilda/tutorial/message.txt 12 bytes 0.01 KB/sec avg 0.01 KB/sec inst Transfer took 2040 ms Destination URL registered in Catalog: sfn://grid005.iucc.ac.il/storage/gilda/generated/2007-10- 19/file910cc4ec-a0e9-4e8a-8e6b-b66d20d22862 guid:a6d7aa04-1d56-437b-9e7b-401f2f21f635

19. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 19 La Antigua, 13th EELA Tutorial, 19.10.2007 lcg-utils: lcg-aa and lcg-la Adding an alias for a given GUID lcg-aa --vo vo guid lfn where –vo specifies the Virtual Organization the user belongs to –guid specifies the Grid Unique Identifier of the file you want to add the alias to –lfn specifies the new alias Example: $ lcg-aa --vo gilda guid:402fc31a-b549-43ac-8272-5540fc24137d lfn:/grid/gilda/tutorial/aliasToNote.txt To check if the previous command was successful, you can use lcg- la command to list the aliases for a given LFN, GUID or SURL $ lcg-la --vo gilda lfn:/grid/gilda/tutorial/note.txt lfn:/grid/gilda/tutorial/note.txt lfn:/grid/gilda/tutorial/aliasToNote.txt

20. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 20 La Antigua, 13th EELA Tutorial, 19.10.2007 Hands-on session Exercise No.2: –verify that your LCG_GFAL_INFOSYS is correctly set up –create a dummy file –check the available storage elements –copy and register the previous created file into your previously created dir –add an alias to the just uploaded file –check if the alias was assigned correctly

21. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 21 La Antigua, 13th EELA Tutorial, 19.10.2007 lcg-utils commands for replicas (I) Copying a file from one SE to another one and register it in the Catalog lcg-rep -d dest_file | dest_host [-v | --verbose] --vo vo src_file where –dest_host is the fully qualified hostname of the destination SE –dest_file is a valid SURL (both sfn:// or srm:// are valid) –vo specifies the Virtual Organization the user belongs to –src_file specifies the source file name: the protocol can be LFN, GUID or SURL. An SURL scheme can be sfn: for a classical SE or srm : [antigua01]$ lcg-rep -v -d gilda02.lcca.usp.br --vo gilda lfn:/grid/gilda/tutorial/antigua/note.txt Using grid catalog type: lfc Using grid catalog : santiago03.reuna.cl Source URL: lfn:/grid/gilda/tutorial/note.txt File size: 18 VO name: gilda Destination specified: gilda02.lcca.usp.br Source URL for copy: gsiftp://grid038.ct.infn.it/grid038.ct.infn.it:/storage/gilda/2006-09-07/fileb68d0ff0-d703- 4f49-9ca5-346b30f05e98.448.0 Destination URL for copy: gsiftp://gilda02.lcca.usp.br/gilda02.lcca.usp.br:/storage/gilda/2006-09- 07/fileee141414-b682-495c-a2dc-99ca8a5bc939.31.0 # streams: 1 # set timeout to 0 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst Transfer took 13150 ms Destination URL registered in LRC: srm://gilda02.lcca.usp.br/dpm/lcca.usp.br/home/gilda/generated/2006-09- 07/fileee141414-b682-495c-a2dc-99ca8a5bc939

22. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 22 La Antigua, 13th EELA Tutorial, 19.10.2007 lcg-utils commands for replicas (II) Listing of replicas for a given LFN, GUID or SURL lcg-lr --vo vo_name file where –vo_name specifies the Virtual Organization the user belongs to –file specifies the Logical File Name, the Grid Unique IDentifier or the Site URL. An SURL scheme can be sfn: for a classical SE or srm: Example: $ lcg-lr --vo gilda lfn:/grid/gilda/tutorial/antigua/note.txt srm://gilda02.lcca.usp.br/dpm/lcca.usp.br/home/gilda/generated/2006-09-07/fileee141414-b682-495c-a2dc-99ca8a5bc939 srm://grid038.ct.infn.it/dpm/ct.infn.it/home/gilda/generated/2006-09-07/fileb68d0ff0-d703-4f49-9ca5-346b30f05e98 we can have the same output using the GUID: $ lcg-lr --vo gilda guid:402fc31a-b549-43ac-8272-5540fc24137d

23. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 23 La Antigua, 13th EELA Tutorial, 19.10.2007 lcg-utils commands for replicas (III) Deleting replicas lcg-del [ -a ] | [ -s se ] [ -v | --verbose ] --vo vo file where –a is used to delete all replicas of the given file –se specifies the SE from which you want to remove the replica –vo specifies the Virtual Organization the user belongs to –file specifies the Logical File Name, the Grid Unique IDentifier or the Site URL. An SURL scheme can be sfn: for a classical SE or srm:. Example: delete one replica $ lcg-del --vo gilda -s gilda02.lcca.usp.br lfn:/grid/gilda/tutorial/antigua/note.txt delete all the replicas $ lcg-del -a --vo gilda lfn:/grid/gilda/tutorial/antigua/note.txt let’s check if the previous command was successful $ lcg-lr --vo gilda lfn:/grid/gilda/tutorial/antigua/note.txt lcg_lr: No such file or directory or by lfs-ls /grid/gilda/tutorial/antigua (you will not see anymore note.txt and its alias)

24. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 24 La Antigua, 13th EELA Tutorial, 19.10.2007 lcg-utils: lcg-cp Downloading a Grid file in a SE to a local destination lcg-cp [ -v | --verbose ] --vo vo src_file dest_file where –vo specifies the Virtual Organization the user belongs to –src_file specifies the source file name: the protocol can be LFN, GUID, SURL or local file. An SURL scheme can be sfn: for a classical SE or srm: –dest_file specifies the destination. The protocol can be file:/// or gsiftp:/// Example: [antigua01]$ lcg-cp -v --vo gilda lfn:/grid/gilda/tutorial/antigua/note.txt file:///home/antigua01/note2.txt Using grid catalog type: lfc Using grid catalog : lfc-gilda.ct.infn.it Source URL: lfn:/grid/gilda/tutorial/antigua/note.txt File size: 18 VO name: gilda Source URL for copy: gsiftp://gilda02.lcca.usp.br/gilda02.lcca.usp.br:/storage/gilda/2006-09- 07/fileee141414-b682-495c-a2dc-99ca8a5bc939.31.0 Destination URL: file:///home/antigua01/note2.txt # streams: 1 # set timeout to 0 (seconds) 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst Transfer took 4050 ms

25. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America 25 La Antigua, 13th EELA Tutorial, 19.10.2007 Hands-on session Exercise No.3: –Create two replicas of the file you previously uploaded (you could also use the alias to point it out) –Check if the operation was successful –Download the file back in your UI –Delete just one replica and verify that –Delete all the replicas and verify that –Verify if the entry is still into the catalog