1. FP6−2004−Infrastructures−6-SSA-026409 www.eu-eela.org E-infrastructure shared between Europe and Latin America Data Management Hands-on Tony Calanducci INFN Catania First Latin American EELA Tutorial Mèrida - Venezuela, 27-29th April 2006

2. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 2 Outline Introduction to LCG File Catalogue (LFC) LFC and lcg_utils hands-on LFC References

3. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 3 User and programs produce and require data Data may be stored in Grid datasets (files) –Located in Storage Elements (SEs) –Several replicas of one file in different sites –Accessible by Grid users and applications from “anywhere” –Locatable by the WMS (data requirements in JDL) –Data may be copied from/to local filesystems (WNs, UIs) to the Grid Why File Catalogs are used? –They keep track of the location of copies (replicas) of Grid files –They provide DM tools and APIs through which users and grid services can deal with files File Catalog Motivations

4. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 4 Logical File Name (LFN) –An alias created by a user to refer to some item of data, e.g. “lfn:cms/20030203/run2/track1” Globally Unique Identifier (GUID) –A non-human-readable unique identifier for an item of data, e.g. “guid:f81d4fae-7dec-11d0-a765-00a0c91e6bf6” Site URL (SURL) (or Physical File Name (PFN) or Site FN) –The location of an actual piece of data on a storage system, e.g. “srm://pcrd24.cern.ch/flatfiles/cms/output10_1” (SRM) “sfn://lxshare0209.cern.ch/data/alice/ntuples.dat” (Classic SE) Transport URL (TURL) –Temporary locator of a replica + access protocol: understood by a SE, e.g. “rfio://lxshare0209.cern.ch//data/alice/ntuples.dat” Name conventions

5. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 5 LFC features –Hierarchical namespace –Integrated GSI Authentication + Authorization –Access Control Lists (Unix Permissions and POSIX ACLs) LFN acts as main key in the database. It has: –Symbolic links to it –Unique Identifier (GUID) –System metadata –Information on replicas –One field of user metadata The LFC

6. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 6 Practicals on LFC and lcg-utils

7. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 7 Set up your environment Check the following environment variables, used to specify the catalog type and its location, are set up properly:  export LCG_CATALOG_TYPE=lfc  export LFC_HOST=lfc.cecalc.ula.ve Ensure you have created a proxy certificate and it is still valid. If not create it by:  voms-proxy-init --voms gilda –Remember: The Passphrase is MERIDA

8. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 8 Listing the entries of a LFC directory lfc-ls [-cdiLlRTu] [--comment] path… where path specifies the LFC pathname (mandatory) –Remember that LFC has a directory tree structure –/grid/ / –All members of a given VO have read-write permissions under their directory –-l (it is a lowercase “L”) outputs long listing –-R lists the contents of directories recursively (don’t use it so often!) –You can set LFC_HOME to use relative paths LFC_HOME=/grid/gilda/myDir  /grid/gilda/myDir/myFile becomes myFile LFC Catalog commands Defined by the user LFC Namespace

9. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 9 lfc-ls examples $ lfc-ls –l /grid/gilda/tutorial... drwxrwxr-x 0 117 102 0 Mar 15 16:53 lookee drwxrwxr-x 0 122 102 0 Mar 15 16:54 miacartella dr--rwxrwx 0 114 102 0 Mar 15 16:50 myfolder drwxrwxr-x 0 114 102 0 Mar 15 16:51 myfolder2 -rwxrwxr-x 1 129 102 193 Mar 15 17:06 nome.txt lrwxrwxrwx 1 129 102 0 Mar 15 17:11 nome2.txt -> /grid/gilda/tutorial/nome.txt drwxrwxr-x 0 129 102 0 Mar 15 16:53 p19 drwxrwxr-x 3 124 102 0 Mar 15 17:13 pippo -rwxrwxr-x 1 124 102 318 Mar 15 17:18 pippopro.txt drwxrwxr-x 0 128 102 0 Mar 15 17:00 pollo... $ export LFC_HOME=/grid/gilda/tutorial $ lfc-ls –l nome.txt -rwxrwxr-x 1 129 102 193 Mar 15 17:06 /grid/gilda/tutorial/nome.txt

10. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 10 LFC Catalog commands Creating a symbolic link lfc-ln -s file linkname lfc-ln -s directory linkname Create a link to the specified file or directory with linkname –Example: $ lfc-ln -s /grid/gilda/tutorial/prova.txt /grid/gilda/tutorial/linkToProva.txt Let’s check the link using lfc-ls with long listing (-l) $ lfc-ls -l /grid/gilda/tutorial lrwxrwxrwx 1 101 102 0 Apr 03 11:42 linkToProva.txt -> /grid/gilda/tutorial/prova.txt Original File Symbolic link

11. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 11 LFC Catalog commands Creating directories in the LFC lfc-mkdir [-m mode] [-p] path... Where path specifies the LFC pathname Remember that while registering a new file (using lcg-cr, for example) the corresponding destination directory must be created in the catalog before Examples: $ lfc-mkdir /grid/gilda/tutorial/Examples You can just check the directory with: $ lfc-ls -l /grid/gilda/tutorial

12. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 12 LFC Catalog commands Adding/deleting metadata information lfc-setcomment path comment lfc-delcomment path lfc-setcomment adds/replaces a comment associated with a file/directory in the LFC Catalog lfc-delcomment deletes a comment previously added Example: lfc-setcomment /grid/gilda/tutorial/hello.txt ‘Hello World!’ Check your job with.. lfc-ls --comment /grid/gilda/tutorial/hello.txt /grid/gilda/catania hello.txt Hello World!

13. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 13 Example: lfc-delcomment /grid/gilda/tutorial/hello.txt Check if it was successfull with.. lfc-ls –l --comment /grid/gilda/tutorial/hello.txt -rw-rw-r-- 1 101 102 13 Apr 03 11:23 /grid/gilda/catania/hello.txt LFC Catalog commands

14. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 14 Hands-on Session Exercise No.1: –Log into glite-tutor.ct.infn.it and initialize your proxy credentials if not already done –check that your environment variables are set up properly to use lfc.cecalc.ula.ve catalog –have a look inside the catalog –create a sub-directory, under /grid/gilda/merida, with your surname –put inside the just created dir a link to an existing file (from files in other directories) –add a comment to that file and verify it

15. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 15 LFC Catalog commands lfc-chmodChange access mode of the LFC file/directory lfc-chownChange owner and group of the LFC file-directory lfc-delcommentDelete the comment associated with the file/directory lfc-getaclGet file/directory access control lists lfc-lnMake a symbolic link to a file/directory lfc-lsList file/directory entries in a directory lfc-mkdirCreate a directory lfc-renameRename a file/directory lfc-rmRemove a file/directory lfc-setaclSet file/directory access control lists lfc-setcommentAdd/replace a comment Summary of the LFC Catalog commands

16. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 16 lcg-utils The LCG Data Management tools (usually called lcg- utils) allow users to copy files between UI, CE, WN and a SE, to register entries in the File Catalogs and replicate files between SEs. Check if LCG_GFAL_INFOSYS environment variable is correctly set to the local GILDA Information Index (BDII) –export LCG_GFAL_INFOSYS=bdii.cecalc.ula.ve:2170

17. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 17 lcg-utils: lcg-cr Upload a file to a SE and register it into the catalog lcg-cr -d dest_file | dest_host -l lfn [-g guid] [-l lfn] [-v | --verbose] --vo vo src_file where –dest_host is the fully qualified hostname of the destination SE –dest_file is a valid SURL (both sfn:// or srm:// format are valid) –guid specifies the Grid Unique IDentifier. If this option is not present, a GUID is generated internally –lfn specifies the Logical File Name associated with the file –vo specifies the Virtual Organization the user belongs to –src_file specifies the source file name: the protocol can be file:/// or gsiftp:///

18. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 18 lcg-utils: lcg-cr To discover which SEs the user is allowed to use, remember you can use lcg-infosites command. lcg-infosites --vo gilda se The output is a list of SEs and related information on available/used space lcg-cr usage example: $ lcg-cr -v -d life006.dft.unipa.it -l lfn:/grid/gilda/tutorial/note.txt --vo gilda file:///home/tcaland/note.txt Using grid catalog type: lfc Using grid catalog : lfc.trigrid.it Source URL: file:///home/tcaland/note.txt File size: 279 VO name: gilda Destination specified: life006.dft.unipa.it Destination URL for copy: gsiftp://life006.dft.unipa.it/flatfiles/SE00/gilda/generated/2006-04- 03/file2cc930b1-c61f-4747-b55a-bc521c93622b # streams: 1 # set timeout to 0 seconds Alias registered in Catalog: lfn:/grid/gilda/tutorial/note.txt 279 bytes 0.12 KB/sec avg 0.12 KB/sec inst Transfer took 29520 ms Destination URL registered in Catalog: sfn://life006.dft.unipa.it/flatfiles/SE00/gilda/generated/2006-04- 03/file2cc930b1-c61f-4747-b55a-bc521c93622b guid:aa5952f7-27b8-4acc-b6ee-0031af0a0f19

19. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 19 lcg-utils: lcg-aa and lcg-la Adding an alias for a given GUID lcg-aa --vo vo guid lfn where –vo specifies the Virtual Organization the user belongs to –guid specifies the Grid Unique Identifier of the file you want to add the alias to –lfn specifies the new alias Example: $ lcg-aa --vo gilda guid:aa5952f7-27b8-4acc-b6ee-0031af0a0f19 lfn:/grid/gilda/tutorial/aliasToNote.txt To check if the previous command was successful, you can use lcg- la command to list the aliases for a given LFN, GUID or SURL $ lcg-la --vo gilda lfn:/grid/gilda/tutorial/aliasToNote.txt lfn:/grid/gilda/tutorial/note.txt lfn:/grid/gilda/tutorial/aliasToNote.txt

20. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 20 Hands-on session Exercise No.2: –verify that your LCG_GFAL_INFOSYS is correctly set up –create a dummy file –check the available storage elements –copy and register the previous created file into your previously created dir –add an alias to the just uploaded file –check if the alias was assigned correctly

21. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 21 lcg-utils commands for replicas (I) Copying a file from one SE to another one and register it in the Catalog lcg-rep -d dest_file | dest_host [-v | --verbose] --vo vo src_file where –dest_host is the fully qualified hostname of the destination SE –dest_file is a valid SURL (both sfn:// or srm:// are valid) –vo specifies the Virtual Organization the user belongs to –src_file specifies the source file name: the protocol can be LFN, GUID or SURL. An SURL scheme can be sfn: for a classical SE or srm : $ lcg-rep -v -d grid009.ct.infn.it --vo gilda lfn:/grid/gilda/tutorial/note.txt Using grid catalog type: lfc Using grid catalog : lfc.trigrid.it Source URL: lfn:/grid/gilda/tutorial/note.txt File size: 279 VO name: gilda Destination specified: grid009.ct.infn.it Source URL for copy: gsiftp://gildase.oact.inaf.it/flatfiles/SE00/gilda/generated/2006-04-03/file02c844ad-1299- 44db-a347-75ed3003c11e Destination URL for copy: gsiftp://grid009.ct.infn.it/flatfiles/SE00/gilda/generated/2006-04-03/file1e0415e0- 3486-42bc-947a-824451f69dc4 # streams: 1 # set timeout to 0 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst Transfer took 2020 ms Destination URL registered in LRC: sfn://grid009.ct.infn.it/flatfiles/SE00/gilda/generated/2006-04- 03/file1e0415e0-3486-42bc-947a-824451f69dc4

22. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 22 lcg-utils commands for replicas (II) Listing of replicas for a given LFN, GUID or SURL lcg-lr --vo vo file where –vo specifies the Virtual Organization the user belongs to –file specifies the Logical File Name, the Grid Unique IDentifier or the Site URL. An SURL scheme can be sfn: for a classical SE or srm: Example: $ lcg-lr --vo gilda lfn:/grid/gilda/tutorial/note.txt sfn://gildase.oact.inaf.it/flatfiles/SE00/gilda/generated/2006-04-03/file02c844ad-1299- 44db-a347-75ed3003c11e sfn://grid009.ct.infn.it/flatfiles/SE00/gilda/generated/2006-04-03/file1e0415e0-3486-42bc- 947a-824451f69dc4 sfn://life006.dft.unipa.it/flatfiles/SE00/gilda/generated/2006-04-03/file2cc930b1-c61f- 4747-b55a-bc521c93622b or we got the same output using its GUID $ lcg-lr --vo gilda guid:aa5952f7-27b8-4acc-b6ee-0031af0a0f19

23. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 23 lcg-utils commands for replicas (III) Deleting replicas lcg-del [ -a ] | [ -s se ] [ -v | --verbose ] --vo vo file where –a is used to delete all replicas of the given file –se specifies the SE from which you want to remove the replica –vo specifies the Virtual Organization the user belongs to –file specifies the Logical File Name, the Grid Unique IDentifier or the Site URL. An SURL scheme can be sfn: for a classical SE or srm:. Example: delete one replica $ lcg-del --vo gilda -s grid009.ct.infn.it lfn:/grid/gilda/tutorial/note.txt delete all the replicas $ lcg-del -a --vo gilda lfn:/grid/gilda/tutorial/note.txt let’s check if the previous command was successful $ lcg-lr --vo gilda lfn:/grid/gilda/tutorial/note.txt lcg_lr: No such file or directory or by lfs-ls /grid/gilda/tutorial (you will not see anymore note.txt and its alias)

24. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 24 lcg-utils: lcg-cp Downloading a Grid file in a SE to a local destination lcg-cp [ -v | --verbose ] --vo vo src_file dest_file where –vo specifies the Virtual Organization the user belongs to –src_file specifies the source file name: the protocol can be LFN, GUID, SURL or local file. An SURL scheme can be sfn: for a classical SE or srm: –dest_file specifies the destination. The protocol can be file:/// or gsiftp:/// Example: $ lcg-cp --vo gilda lfn:/grid/gilda/tutorial/note.txt file:/home/tcaland/note2.txt Using grid catalog type: lfc Using grid catalog : lfc.trigrid.it Source URL: lfn:/grid/gilda/tutorial/note.txt File size: 279 VO name: gilda Source URL for copy: gsiftp://gildase.oact.inaf.it/flatfiles/SE00/gilda/generated/2006-04- 03/file02c844ad-1299-44db-a347-75ed3003c11e Destination URL: file:/home/tcaland/note2.txt # streams: 1 # set timeout to 0 (seconds) 0 bytes 0.00 KB/sec avg 0.00 KB/sec inst Transfer took 1030 ms

25. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 25 Hands-on session Exercise No.3: –Create two replicas of the file you previously uploaded (you could also use the alias to point it out) –Check if the operation was successful –Download the file back in your UI –Delete just one replica and verify that –Delete all the replicas and verify that –Verify if the entry is still into the catalog

26. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 26 Handling entries permissions (I) Get LFC entries (file/directories) permissions and ACLs –lfc-getacl path... where path is a LFC file or directory Example: $ lcg-getacl /grid/gilda/tutorial # file: /grid/gilda/tutorial # owner: /C=IT/O=GILDA/OU=Personal Certificate/L=CATANIA/CN=CATANIA01/Email=roberto.barbera@ct.infn.it # group: gilda user::rwx group::rwx #effective:rwx other::r-x default:user::rwx default:group::rwx default:other::r-x

27. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 27 Handling entries permissions (II) Change the permission of a file/directory lfc-chmod absolute_mode path... where –absolute_path is the permission pattern expressed in octal mode –path is a LFC file or directory Example: $ lfc-chmod 750 /grid/gilda/tutorial/note.txt $ lfc-getacl /grid/gilda/tutorial/note.txt # file: /grid/gilda/tutorial/note.txt # owner: /C=IT/O=GILDA/OU=Personal Certificate/L=INFN Catania/CN=Tony Calanducci/Email=tony.calanducci@ct.infn.it # group: gilda user::rwx group::r-x #effective:r-x other::---

28. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 28 Setting ACLs (I) Set LFC directory/file access control lists with lfc-setacl [-d] [-m] [-s] acl_entries path... where –acl_entries is a comma separated list of entries. Each entry has colon separated fields: ACL type, id (uid or gid), permission. The entries look like: user::perm user:uid:perm group::perm group:gid:perm mask:perm other:perm default:user::perm default:user:uid:perm default:group::perm default:group:gid:perm default:mask:perm default:other:perm The ACL type can be abbreviated with the first letter user and group -> owner permission user:gid and group:gid -> specific users and groups mask -> maximum permissions to specific users/groups default permissions (only for a directory) are inherited by the files or sub-directories of that dir

29. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 29 Setting ACLs (II) –path specifies the LFC pathname –uid can be given as the DN or the corresponding numeric id. –gid can be given as the VO name or the corresponding numeric id. –perm can be expressed as a combination of characters rwx- or as a value between 0 and 7. OPTIONS -d remove ACL entries. The "perm" field is ignored. -m modify existing ACL entries or add new entries. -s set the ACL entries. The complete set of ACL entries is replaced.

30. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 30 ACLs example Example: $ lfc-setacl -m u:'/C=IT/O=GILDA/OU=Personal Certificate/L=INFN/CN=Diego Scardaci/Email=diego.scardaci@ct.infn.it':rwx,g:trigrid:r-x,m:rwx /grid/gilda/tutorial/note.txt $ lfc-getacl /grid/gilda/tutorial/note.txt # file: /grid/gilda/tutorial/note.txt # owner: /C=IT/O=GILDA/OU=Personal Certificate/L=INFN Catania/CN=Tony Calanducci/Email=tony.calanducci@ct.infn.it # group: gilda user::rwx user:/C=IT/O=GILDA/OU=Personal Certificate/L=INFN/CN=Diego Scardaci/Email=diego.scardaci@ct.infn.it:rwx #effective:rwx group::r-x #effective:r-x group:trigrid:r-x #effective:r-x mask::rwx other::---

31. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 31 Exercise No. 4 Create an entry in your working directory. Give a look to the default permission of it. Change the permission in a way that only you (the owner) can delete the entry Ask your neighbor to delete that (he should not be able) Ask again your neighbor his DN and add it as an ACL entry granting full permission. Ask now to delete it (it should be able to do that now) – delete with lcg-del -s STORAGEHOST

32. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 32 Practicals on gLite File Transfer Service (FTS)

33. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 33 glite-transfer-channel-list CHANNEL CHANNEL... List the available channels for data transfer. –If one or more CHANNELs are given on the command line (and you have CHANNEL ADMINISTRATOR rights), detailed information about the specified channels are listed. Otherwise, without any arguments, you will get the names of all defined channels. > glite-transfer-channel-list channelCtCnaf Channel: channelCtCnaf Between: ct.infn.it and cnaf.infn.it State: Active Contact: emidio.giorgio@ct.infn.it Bandwidth: 1024 Nominal throughput: 1024 Number of files: 1, streams: 10 Number of VO shares: 1 VO 'gilda' share is: 100 gLite FTS: channel informations

34. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 34 glite-transfer-submit SOURCE-SURL DEST-SURL | -f FILE It returns the ID of the job by which it can be monitored. The command can be invoked in two different ways: either by specifying the source and the destination SURL or by giving the name of a file using the -f option which contains source and destination SURL pairs, one per line, separated by whitespace. Whitespace in the names of the SURLs needs to be protected with a backslash \. Data transfer job submission

35. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 35 Additional available options are: -bBlocking mode. Don't return an ID but simply block until the transfer has been successfully carried out. -i TIME Interval between status calls in blocking call. Works only together with -b. The TIME parameter has to be given in seconds. -p PASSWORD The MyProxy password to be sent with the job. -m SERVER Use the given MyProxy server. -f FILE The file to be read (a pair of SURLs per line, separated by whitespace). If the FILE is given as - (a dash) then stdio is read. glite-transfer-submit options

36. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 36 >glite-transfer-status JOBID It prints the status of the given job id request. Usage: glite-transfer-status [options] JOBID [JOBID...] Available options: -h Print this help text and exit. -q Quiet operation. -s URL Use the transfer service at the specified URL. -v Print details about the service. --verbose Be more verbose. --source SSITE Specify source site name --dest DSITE Specify destination site name -V Print the version number and exit. -l List status for all files Monitoring the job transfer’s status

37. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 37 >glite-url-copy [options] sourceSURL destSURL This command can be used to test SRM - SRM transfer directly. OPTIONS -help | -usage Print help. -version Print the version of this program. -vb | -verbose during the transfer, display the number of bytes transferred and the transfer rate per second. -nodcau | -no-data-channel-authentication turn off data channel authentication for ftp transfers. SRM - SRM transfer

38. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 38 Make sure you have a valid proxy, otherwise initialize it : > voms-proxy-info -all Couldn't find a valid proxy. > voms-proxy-init --voms gilda Store your credential on a myproxy server. The credential must be created with the -d option. > myproxy-init -s grid001.ct.infn.it -d Exercise: preliminar

39. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 39 Register the file on a Storage Element of your choice, but you must be sure that this SE implements SRM interface (aliserv6.ct.infn.it or egee016.cnaf.infn.it). [glite-ui] /home/merida18 > lcg-cr --vo gilda -v -l lfn:/grid/gilda/tony/start_hostname.sh -d aliserv6.ct.infn.it file:/home/merida18/start_hostname.sh Using grid catalog type: lfc Using grid catalog : lfc.cecalc.ula.ve Source URL: file:/home/merida18/start_hostname.sh File size: 31 VO name: gilda Destination specified: aliserv6.ct.infn.it Destination URL for copy: gsiftp://aliserv6.ct.infn.it/aliserv6:/data01/gilda/2006-04-24/file742f27e0- 3900-4917-8277-af6b2d0a60de.3534.0 # streams: 1 # set timeout to 0 seconds Alias registered in Catalog: lfn:/grid/gilda/tony/start_hostname.sh 31 bytes 0.00 KB/sec avg 0.00 KB/sec inst Transfer took 10110 ms Destination URL registered in Catalog: srm://aliserv6.ct.infn.it/dpm/ct.infn.it/home/gilda/generated/2006-04-24/file742f27e0-3900-4917- 8277-af6b2d0a60de guid:540c506b-b448-4ff5-8078-28d56b6f5b06 Register a file on a SRM Storage Element

40. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 40 In order to replicate the file, you must know the file SURL, because you have to specify the source SURL and the destination one. You can obtain the source URL from lcg-lr command: [glite-ui] /home/merida18 > lcg-lr --vo gilda lfn:/grid/gilda/tony/start_hostname.sh srm://aliserv6.ct.infn.it/dpm/ct.infn.it/hom e/gilda/generated/2006-04-24/file742f27e0- 3900-4917-8277-af6b2d0a60de Get the SURL of the saved file

41. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 41 Before submitting, you may want to list the channel available for file transfer. This is important because if the couple of SE that you are using for file transfer it's not covered, the transfer will fail. You can use the glite- transfer-channel-list command: >export CHAN=https://fts.ct.infn.it:8443/glite-data-transfer-fts/services/ChannelManagement > glite-transfer-channel-list -s $CHAN channelCnafCt channelCnafPd channelCtCnaf channelCtPd channelPdCnaf channelPdCt List the available transfer channels

42. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 42 The commands for file replication between SEs is glite-transfer-submit where you have to specify the source URL. Then you must insert the destination URL, i.e. the physical file name that the file will have on the remote SE when replicated. It's up to the user the correct insertion of the destination URL! [glite-ui] /home/merida18 > glite-transfer-submit -v srm://aliserv6.ct.infn.it/dpm/ct.infn.it/home/gilda/generated/2006-04-24/file742f27e0-3900-4917- 8277-af6b2d0a60de srm://egee016.cnaf.infn.it/dpm/cnaf.infn.it/home/gilda/tony/start_hostname.sh # Using endpoint https://fts.ct.infn.it:8443/glite-data-transfer-fts/services/FileTransfer # Service version: 3.1.0 # Interface version: 3.1.0 # Schema version: 2.1.0 # Service features: glite-data-transfer-fts-3.1.0-17 # Client version: 3.1.2 # Client interface version: 3.1.2 Enter MyProxy password: Enter MyProxy password again: 349c5461-d3b6-11da-be75-e65715ba5813 Submit the transfer job

43. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 43 You can check out all the other options of glite-transfer-submit looking at the man page. If the submission is successful, a JobID is returned, that you can use to monitor your job execution. > [glite-ui] /home/merida18 > glite-transfer-status -l 349c5461-d3b6-11da-be75- e65715ba5813 Done Source: srm://aliserv6.ct.infn.it/dpm/ct.infn.it/home/gilda/generated/2006-04- 24/file742f27e0-3900-4917-8277-af6b2d0a60de Destination: srm://egee016.cnaf.infn.it/dpm/cnaf.infn.it/home/gilda/tony/start_hostname.sh State: Done Retries: 0 Reason: (null) Duration: 15 Monitor the transfer job

44. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 44 [glite-ui] /home/merida18 > lcg-rf -v --vo gilda -l lfn:/grid/gilda/tony/start_hostname.sh srm://egee016.cnaf.infn.it/dpm/cnaf.infn.it/home/gilda/tony/s tart_hostname.sh Using grid catalog type: lfc Using grid catalog : lfc.cecalc.ula.ve set timeout to 0 seconds Site URL to be registered: srm://egee016.cnaf.infn.it/dpm/cnaf.infn.it/home/gilda/tony/start_hostname.sh File size: 31 guid:540c506b-b448-4ff5-8078-28d56b6f5b06 [glite-ui] /home/merida18 > lcg-lr --vo gilda lfn:/grid/gilda/tony/start_hostname.sh srm://aliserv6.ct.infn.it/dpm/ct.infn.it/home/gilda/generated/2006-04-24/file742f27e0-3900-4917- 8277-af6b2d0a60de srm://egee016.cnaf.infn.it/dpm/cnaf.infn.it/home/gilda/tony/start_hostname.sh Register the replica in the File Catalog

45. FP6−2004−Infrastructures−6-SSA-026409 E-infrastructure shared between Europe and Latin America First Latin American EELA Tutorial, Mèrida, 27-29th April 2006 45 References LFC API / CLI https://uimon.cern.ch/twiki/bin/view/LCG/LfcApi http://grid-deployment.web.cern.ch/grid- deployment/documentation/LFC_DPM/lfc/html/ gLite FTS homepage http://egee-jra1-dm.web.cern.ch/egee-jra1-dm/FTS