WebFTS File Transfer Web Interface for FTS3 Andrea Manzi On behalf of the FTS team Workshop on Cloud Services for File Synchronisation and Sharing

Overview  The FTS3 service  WebFTS features  WebFTS cloud integrations  Dropbox  CERNBox  Ongoing development  Dropping X509  Data management operations 17/11/2014 WebFTS: File Transfer Web Interface for FTS3 2

What is WebFTS?  Web based tool to transfer files between grid/cloud storages  Modular protocol support  gsiftp, http(s), xrootd and srm  Cloud extensions: dropbox, CERNBox  Development funded by 17/11/ WebFTS: File Transfer Web Interface for FTS3

Provide access to leading technology Based on FTS3 FTS3 is the service responsible for distributing the majority of LHC data across the WLCG infrastructure Low level data movement service, responsible for moving sets of files from one site to another while allowing participating sites to control the network resource usage Used by LHC VOs + many others VOs part of EGI ~20PB monthly transfer volume / ~2.2M files per day (WLCG) /11/2014 WebFTS: File Transfer Web Interface for FTS3 4

WebFTS architecture WebFTS: File Transfer Web Interface for FTS3 BROWSER REST API GFAL2 FTS3 DAVIX GSIFTP DROP BOX … … WEBFTS.js 17/11/2014 5

Security  Simpler access while keeping the same level of security  X509 + Oauth for Dropbox  Transparent delegation of credentials  Avoid storing or transferring any sensitive data  Open access to all source code  All sensitive information is used within the browser and forgotten 17/11/ WebFTS: File Transfer Web Interface for FTS3

Delegation  Delegation is needed to let WebFTS access the grid on users behalf  Users make private key available to browser  Not available via browser API  VOMS extensions acquired by the service on users behalf  Why it’s important  Gives the users a service which can access the grid for them, from a browser, with full VOMS credentials WebFTS: File Transfer Web Interface for FTS3 17/11/2014 7

Additional Features  Check-summing and file overwriting  Possibility to resubmit transfer jobs or only-failed files transfers.  Storage Endpoints Auto-completion  For endpoints published on the BDII (EGI and WLCG Information System)  Support for LFC Registration  File catalog developed at CERN and used by EGI and WLCG 17/11/ WebFTS: File Transfer Web Interface for FTS3

Success Stories  WebFTS has been successfully tested to transfer from/to:  EUDAT B2Stage ( iRODS DSI)  Any gsiftp/webdav/xrootd aware grid storage ( DPM, dCache, Castor, EOS, Storm)  HPC Oak Ridge National Lab (ongoing)   Under evaluation by LHCb 17/11/ WebFTS: File Transfer Web Interface for FTS3

Landing page and Guided-tour 17/11/ WebFTS: File Transfer Web Interface for FTS3

Credential delegation 17/11/ WebFTS: File Transfer Web Interface for FTS3

Transfer interface 17/11/ WebFTS: File Transfer Web Interface for FTS3

Job status interface 17/11/ WebFTS: File Transfer Web Interface for FTS3

Extension for Dropbox  Nice way import/export data from the grid world  Avoid the installation of new software and uses what the user has already installed  Zero development of clients  Multiplatform is given for free  Integration with Oauth  By delegating to FTS the right to interact with dropbox on users behalf  Achieved using web tech  Which requires the interactivity of a browser WebFTS: File Transfer Web Interface for FTS3 17/11/

17/11/ WebFTS: File Transfer Web Interface for FTS3 Extension for Dropbox

Dropbox plugin  Server side the development of a plugin for the metadata management and I/O operations was needed:  FTS REST integrates the plugin to perform metadata management operations  FTS3 server uses the plugin to perform the transfers:  GridFTP dropbox  Http(s) dropbox WebFTS: File Transfer Web Interface for FTS3 17/11/

 While Dropbox has been integrated via the implementation of a plugin for CERNBox we waited for the new version with EOS as backend ( CERNBox 2.0)  We use EOS access via standard grid protocols ( e.g. xrootd)  We map user credentials to correct EOS namespace  The rest comes for free CERNBox integration 17/11/ WebFTS: File Transfer Web Interface for FTS3

WebFTS With CERNBox 17/11/2014 WebFTS: File Transfer Web Interface for FTS3 18

17/11/2014 WebFTS: File Transfer Web Interface for FTS3 19 WebFTS With CERNBox

17/11/2014 WebFTS: File Transfer Web Interface for FTS3 20 WebFTS With CERNBox

17/11/2014 WebFTS: File Transfer Web Interface for FTS3 21 WebFTS With CERNBox

17/11/2014 WebFTS: File Transfer Web Interface for FTS3 22 WebFTS With CERNBox

17/11/2014 WebFTS: File Transfer Web Interface for FTS3 23 How can we get rid of the delegation step? 1. An Identity Federation: eduGAIN  To allow identity providers to authenticate users at their own institute (SSO) 2. A token translation service : STS  To ask the CA for a certificate for the users 3. An “IOTA” Certification Authority  To grant the short lived certificate 4. VOMS  To accept the new cert as a VO member Ongoing developments: Access without X509

EDUGAIN WebFTS: File Transfer Web Interface for FTS3  Built on existing federations and infrastructures  CERN participates in eduGAIN via SWITCHaai  Many NRENs participate in eduGAIN too 24 17/11/

Security Token Service (STS) WebFTS: File Transfer Web Interface for FTS3 An EMI service SAML in, X509/VOMS out 17/11/

“IOTA” CA WebFTS: File Transfer Web Interface for FTS3 17/11/

VOMS admin WebFTS: File Transfer Web Interface for FTS3 17/11/

Architecture WebFTS CERN SSO IdP Credentials Attributes Web Redirect WAYF SAML VOMS IdP Grid Storage Element Grid Storage Element X.509 VOMS STS IOTA CA IOTA CA SAML X.509 VOMS Slide adapted from Romain Wartel, GDB Sept

Pros/Cons  X509-free access to the grid infrastructure  With VOMS support  Without modifying all the services  Federated single sign on  One password to remember  Numerous services potentially accessible  But we need Site acceptance WebFTS: File Transfer Web Interface for FTS3 17/11/

 Not only Transfers..  FTS REST API have been extended to support data management operations  Delete  Create/Remove folders  Rename  Under integration in WebFTS 17/11/ WebFTS: File Transfer Web Interface for FTS3 Ongoing developments: Data Management

 Online service accessible:   try now!  User certificate in your browser  User guide, F.A.Q:  Online guided-tour   Official support & code   Links 17/11/ WebFTS: File Transfer Web Interface for FTS3

Questions? 17/11/ WebFTS: File Transfer Web Interface for FTS3