1. GFAL 2.0
Devresse Adrien
CERN lcgutil team

2. What is GFAL 2.0 ? → One and only one Toolkit for all
the Grid and Cloud data operations :
→ Client data access :
→ support for all common protocols
→ support for all common storage systems
→ Data management
→ One API for all data management
→ Data transfers
→ 3rd party transfer support
CERN, IT-GT-DMS

3. Goal of GFAL 2.0 ? → Hide the Grid complexity from the client side.
→ Simplify the data access world
→ One API.
→ Reduce the number of dependencies.
→ Plugin system
→ Makes application independent of the technology / protocols.
CERN, IT-GT-DMS

4. The complexity of a random read file operation
Client
LFC
get file replicas list
SRM
Server
DPM
Head Node
Disk node
SRM GET
rfio open
rfio lseek
rfio read
lfc_stat
GFAL 2.0
gfal2_open
gfal2_pread
SRM surl → turl
CERN, IT-GT-DMS

5. An old-style Grid application :
Grid Application X
Core logic
File access layer
dCache,
DPM,
Castor lib
Globus
framework
Xrootd
framework
SRM
client
Http lib
CERN, IT-GT-DMS

6. A GFAL 2.0 Grid application :
Grid Application X
Core logic
GFAL 2.0
API
CERN, IT-GT-DMS

7. GFAL 2.0 architecture : Extensible... API POSIX File API Transfer
Plugin layer
Plugin
LFC
Plugin
SRM
Plugin
RFIO
Plugin
Dcap
Plugin
Http/dav
Plugin
xrootd
Plugin
S3 ( ? )
experience
catalogue plugin (???) 
Any Cloud
( Http/Dav )
DPM
Castor
Dropbox
Extensible...
EOS
dCache

8. What GFAL 2.0 can do ? In any protocol : Meta-data operations :
→ stat, rm, mkdir,
→ mv, rmdir, etc...
→ list directory, xattr, etc...
Remote I/O in any protocol
→ open/read/write/close
→ pread/pwrite
→ etc..
In any protocol :
→ SRM
→ GRIDFTP
→ RFIO-Castor
→ RFIO-DPM
→ LFN
→ GUID
→ DCAP / GSIDCAP
→ XROOTD
→ HTTP(S) *
→ WebDav(s) *
→ local file
CERN, IT-GT-DMS

9. Feature : Protocol generic filecopy
Third party transfer copy :
gfalt_copyfile(context, NULL, « srm://monfichier », « srm://madest/higgs »)
gfalt_copyfile(context, NULL, « srm://monfichier », « gsiftp://madest/chuck_norris »)
Automatic protocol resolution :
gfalt_copyfile(context, NULL, « guid:1233 », «root://xrootd_dpm/curiosity»)
gfalt_copyfile(context, NULL, « file:///tmp/super_file», «gsidcap://42»)
CERN, IT-GT-DMS

10. Really protocol generic :
Take a, b in
srm, guid, lfn, , rfio, dcap , gsidcap, gsiftp, file, xrootd, (http, dav )
gfalt_copyfile(context, NULL, « a://src/file », «b://dst/file»)
FINISHED !
CERN, IT-GT-DMS

11. Speed Up x2-x4 for short transfers
Feature : Transparent
parallel operations
Destination exist ?
File copy operation :
SRM GET
SRM GET
Destination exist ?
SRM PUT
SRM PUT
transfer
transfer
PUTDONE
PUTDONE
Speed Up x2-x4 for short transfers
Old tools...
GFAL 2.0

12. thread-safe session reuse
Feature : Transparent
thread-safe session reuse
Pool
of Connections
SRM,
GSIFTP,
HTTP,
Etc,..
Transfer 1
Create session
End
Transfer 2
End
One server connection

13. Feature : extended attributes
GFAL 2.0 supports POSIX extended attributes ( getxattr, setxattr, etc...)
→ Clean way to uses protocol specific functionalities.
→ Simple get/list/set API
→ Easy to use.

14. NO ENVIRONMENT VAR !
Environment Variable User

15. Feature : Works out of the box :
NO ENV VAR !
→ Every parameter is in a configuration file :
→ /etc/gfal2.d/*.conf
→ No environment variable needed.
→ Tune it once, re-use everywhere.
→ Still compatible with classical environment variable of GFAL 1.0, for an easy transition.

16. Does not make the coffee
Missing Feature :
Does not make the coffee
Coming soon ….
( RFC-2324 compliant ! )

17. How to use GFAL 2.0 ? : → libgfal2 : → gfal2-python :
→ a C Library
→ a set of independent plugins
→ gfal2-python :
→ simple and pythonic python bindings
→ gfalFS fuse module for gfal 2
→ allows to mount any grid endpoint in a local folder
→ gfal-tools : experimental command line tools
→ gfal-ls, gfal-copy, gfal-mkdir, gfal-cat, ….
CERN, IT-GT-DMS

18. Already in production state !
How to get GFAL 2.0 ? :
Already in production state !
→ Part of EMI 2 release:
→ EMI 2 repository :
→ Packaged on EPEL :
→ sudo yum install gfal2-all gfal2-doc gfalFS
→ is going to be packaged on Debian ( thx Mathias )
→ Source available, designed to be portable.
CERN, IT-GT-DMS

19. Support of GFAL 2.0 : → Part of EMI
will be supported after the end of EMI
→ GFAL 2.0 is the core of FTS 3.0
will be supported in any case.
→ Long term successor to :
lcg clients : lcg-cp, lcg-cr, lcg util python
gfal 1.0, etc...
→ Development Open to everybody
CERN, IT-GT-DMS

20. From gfal1.0 to GFAL 2.0 : → GFAL 2.0 does a lot more than GFAL 1.0 :
→ a lot of Major changes :
Not 100% backward compatible.
gfal 1.0
+ lcg-util (lcg-cp, .. )
+ is-interface
+ gridftp-ifce
+ ….
CERN, IT-GT-DMS

21. From gfal1.0 to GFAL 2.0 :
But, We provides facilities for the transition
The complete POSIX API is backward compatible
We provide examples for common operations
Doxygen documentation of the API.
Any question is Welcome :
CERN, IT-GT-DMS

22. Lcg-util command line tools successor ?
gfal-tools ( ? ) :
Lcg-util command line tools successor ?
→Users wants command line tools
→ lcg-util (lcg-cp, lcg-cr ) are heavily used.
→ lcg-util suffers from his age...
→ outdated features → hard to maintain
→ lots of dependencies →
→ a GFAL 2.0 based -tools ?

23. gfal-tools :
→ Experimental tools for lcg-util replacement on the long term.
→ We need feedback and informations from the experiments :
→ Which part of lcg-util is needed :
→ lcg-cp, lcg-ls, lcg-cr, ( and ? )
→ What can be improved 
in their point of view ?:
→ It's time for suggestion !
→ How can we help to a smooth transition ?

24. Gfal-tools : Current status
→ A simple set of generic command line file utils  based on GFAL 2:
→ gfal-copy → gfal-ls
→ gfal-rm → gfal-cat
→ gfal-save → etc....
→ Prototype already available for comments ...
→ Still under development !
→ It's the right time for comments...

25. https://svnweb.cern.ch/trac/lcgutil/wiki/gfal2