1. EMI INFSO-RI-261611 The ARC data client in EMI-1 Jon Kerr Nilsen Dept. of Physics, Univ. of Oslo On behalf of the ARC Data PT

2. EMI INFSO-RI-261611 Why ARC data libs? ARC data in production ARC data Architecture ARC client tools ARC data in EMI Plans / way forward Outline Jon Kerr Nilsen 2

3. EMI INFSO-RI-261611 Can be used on different platforms (Linux, Windows, Mac, Solaris) Can be used from different languages (C++, Python, java) Pluggable infrastructure which allows for a very limited set of dependencies from other libraries Plugins exist for several widely used data transfer and management protocols (GridFTP, SRM, LFC, https, stdio, …) Light-weight and user-friendly, with easy client- side configuration (the default config would work in most cases) Why use ARC data libs? Jon Kerr Nilsen 3

4. EMI INFSO-RI-261611 ARC DDM in production ARC is not only for the Nordics Highly distributed – e.g. NDGF T-1 is spread over four countries (both CPU and storage) Most resources shared with other users/VO’s Jon Kerr Nilsen 4

5. EMI INFSO-RI-261611 ARC DDM in production 4-6k cores being used 20-40k jobs per day Transfering 500MB/s on average for downloads and uploads –> ~100TB/day Approx. half of it on NDGF T-1 – Not very big but very efficient Jon Kerr Nilsen 5

6. EMI INFSO-RI-261611 ARC Internals Clients can submit jobs using WS interface or to ARC classic and other legacy services via specialized adaptors HED handles communication between services and with outside world A-REX handles data staging (through libarcdata2) and batch job submission (through LRMS) No middleware on compute- nodes Jon Kerr Nilsen 6

7. EMI INFSO-RI-261611 libarcdata2 Jon Kerr Nilsen 7 Generic library for basic data access and movement Protocols are supported through plugins (DMCs) No external dependencies in libarcdata2, only in plugins Same library used on client side and service side – smaller codebase, easier to maintain

8. EMI INFSO-RI-261611 Data client functionality Three command line tools –arcls –arccp –arcrm Extra functionality in CLI options and URL options Focused on what functionality is needed in grid DDM – not on providing a fully POSIX- compliant interface Jon Kerr Nilsen 8

9. EMI INFSO-RI-261611 Data client functionality Protocols are supported through DMC plugins – GridFTP, SRM, LFC, HTTP, file, stdio Clients support whatever protocol with corresponding plugin installed All plugins installs via nordugrid-arc-client-tools metapackage Jon Kerr Nilsen 9

10. EMI INFSO-RI-261611 Documentation man arccp arccp --help Usage: arccp [OPTION...] source destination The arccp command copies files to, from and between grid storage elements. Help Options: -h, -?, --help Show help options Application Options: -p, --passive use passive transfer (does not work if secure is on, default if secure is not requested) -n, --nopassive do not try to force passive transfer -y, --cache=path path to local cache (use to put file into cache)... Jon Kerr Nilsen 10 http://www.nordugrid.org/documents/ arc-ui.pdf http://www.nordugrid.org/documents/ar c-client-install.html

11. EMI INFSO-RI-261611 Download from LFC –arccp lfc://lfc1.ndgf.org/grid/test/file.01./file.01 Upload to LFC specifying SRM location and spacetoken –arccp./file.01 lfc://lfc.ndgf.org/grid/myvo/file.01\ --location=\ srm://srm.ndgf.org;spacetoken=MY_TOKEN/test/file.01.a List LFC replicas –arcls -L lfc://lfc.ndgf.org/grid/test/file.01 srm://srm.ndgf.org/test/file.01.a:guid=d11c1c69-3400-466c-995e- 7d5681a33692 Remove LFC entry –arcrm lfc://lfc.ndgf.org/grid/test/file.01 Example: LFC Jon Kerr Nilsen 11

12. EMI INFSO-RI-261611 Copying to/from stdio –arccp gsiftp://grid.uio.no/jobdir/145434643346354/out.txt stdio:///stdout hello –arccp gsiftp://grid.uio.no/jobdir/145434643346354/out.txt – hello –arccp - - hello hello Example: stdio Jon Kerr Nilsen 12

13. EMI INFSO-RI-261611 libarcdata2 API available in Python import arc from arcom import datapoint_from_url src = datapoint_from_url(turl, ssl_config) dst = datapoint_from_url(fobj.name) mover = arc.DataMover() mover.verbose(verbose) mover.retry(False) mover.passive(True) status = mover.Transfer(src, dst, arc.FileCache(), arc.URLMap()) if status != status.Success: raise Exception, ”””Failed to download file. Try again!””” return str(status) Python API Jon Kerr Nilsen 13

14. EMI INFSO-RI-261611 ARC participates in the EMI Data group with two data experts – Zsombor Nagy, Jon Kerr Nilsen ARC has no storage element in EMI, limited amount of efforts needed for ARC data libraries so far (need GLUE2 and SRM changes on SEs first) Most important task so far: Consolidation ARC Data in EMI Jon Kerr Nilsen 14

15. EMI INFSO-RI-261611 Two sets of data clients/libs in EMI – arcdatalib2 and arc data tools – GFAL and lcg_util These have overlapping functionality Important goal for EMI to reduce code base (and thus reduce maintenance costs) Need to identify overlap and potential consolidation possibilities Data lib consolidation Jon Kerr Nilsen 15

16. EMI INFSO-RI-261611 High-level plan – Evaluate current libs/clients – Investigate what features in the libs/clients are used/needed – Identify scenarios and analyze Efforts, costs, benefits, risks, dependencies… – Suggest a decision for the best solution – Implement this solution Data lib consolidation Jon Kerr Nilsen 16

17. EMI INFSO-RI-261611 Had F2F meeting with the gLite and ARC data client developers in March A couple of possible consolidation possibilities were found – SRM and LFC libraries Also found surprisingly small feature overlap – GFAL focused on POSIX API – ARC data libs focused on grid user needs – functionality not covered by POSIX API – Both are useful More details on EMI all-hands in June :) Data lib consolidation Jon Kerr Nilsen 17

18. EMI INFSO-RI-261611 Consolidation, consolidation and consolidation Also – Phasing out ng* clients, replaced by arc* clients – Move from SRM through httpg to SRM through https – GLUE2 as soon as SEs have implemented – Bugfixing and improvements Way forward Jon Kerr Nilsen 18

19. EMI INFSO-RI-261611 Jon Kerr Nilsen 19 Thank you! Manual: http://www.nordugrid.org/documents/arc-ui.pdfhttp://www.nordugrid.org/documents/arc-ui.pdf Installation: http://www.nordugrid.org/documents/arc-client-install.htmlhttp://www.nordugrid.org/documents/arc-client-install.html Bugs and feature requests: http://bugzilla.nordugrid.orghttp://bugzilla.nordugrid.org Contact: nordugrid-discuss@nordugrid.org