1. HEPiX Storage, Edinburgh 27-28 May 2004 SE Experiences Supporting Multiple Interfaces to Mass Storage J Jensen j.jensen@rl.ac.uk

2. HEPiX Edinburgh May 2004 Outline u Objectives u Achievements u Experiences, lessons learned n Protocols, interfaces n Technical u Future u Conclusion

3. HEPiX Edinburgh May 2004 Objectives u Implement uniform interfaces to mass storage n Independent of underlying storage system u SRM n Uniform interface – much is optional u Develop back-end support for mass storage systems n Provide missing features – directory support? u Publish information

4. HEPiX Edinburgh May 2004 Objectives – SRM u SRM 1 provides async get, put n get (put) returns request id n getRequestStatus returns status of request n When status = Ready, status contains Transfer URL – aka TURL n Client changes status to Running n Client downloads (uploads) file from (to) TURL n Client changes status to Done u Files can be pinned and unpinned

5. HEPiX Edinburgh May 2004 Objectives – SRM u SRM 1 interface requires web services n SOAP messages via HTTP n Also SOAP via GSI-HTTP: HTTPG u Data Transfer n GridFTP mandatory s Requires GSI s Also requires Gridmap files

6. HEPiX Edinburgh May 2004 Achievements u In EDG, we developed EDG Storage Element n Uniform interface to mass storage and disk n Interfaces with EDG Replica Manager n Also client command line tools n Interface was based on SRM but simplified s Synchronous s Trade-off between getting it done soon and getting it right the first time s Additional functionality such as directory functions n Highly modular system

7. HEPiX Edinburgh May 2004 Achievements – SE TIME Look up user User database File metadata Request and handler process management Look up file data Access control MSS access Thin layer interface Mass Storage

8. HEPiX Edinburgh May 2004 Achievements - SE u The request contains the sequence of names of handlers u As each handler processes the request, it calls a library that moves the name to an audit section u The library allows easy access to global data u Handlers may also have handler- specific data in the XML u Storing the XML output from each handler as the request gets processed makes it easy to debug the SE handler3 handler4 handler5 Global Data handler1 handler2 sequence audit XML

9. HEPiX Edinburgh May 2004 Achievements - SE u The choice of architecture was right n Multiple interfaces can be supported with the same core n New functionality can be added fairly easily s Adding or removing access control is more or less a question of adding the appropriate handler n Easy to debug s Messages passed between handlers can be debugged easily n Disadvantage: not as fast as a monolithic core s Speed can be improved by having persistent handlers

10. HEPiX Edinburgh May 2004 Definition u A current definition of Storage Element u A Storage Element must provide: n GridFTP for data transfer n An SRM 1.0 or 1.1 web services interface s The difference is SRMCopy – 3 rd party copying s Which the client can always do anyway using GridFTP n Information published via MDS in GLUE schema s This includes the end point u Interoperability via WSDL document

11. HEPiX Edinburgh May 2004 Experiences – technical u Initially we had to use Java for web services (Tomcat) u Interfacing Java to Unix processes is not always painless: u Java does not appear to work well with Unix pipes u We had to write a socket-to-pipe daemon Parent process Child process New process fork() exec() pipes for std{in,out,err}

12. HEPiX Edinburgh May 2004 Experiences – technical u gSOAP is pretty good these days n Now uses std::string for strings u But how to do GSI n We use Ben Couturiers plugin s Clean s Need no knowledge of Globus API to use it n The official GSI code was recently rewritten s It now uses Globus GSSAPI s Previously it used Globus IO module s Free, but you must register your email address

13. HEPiX Edinburgh May 2004 Experiences – development u We use RFIO for data access to both CASTOR and HPSS n Unfortunately the RFIO libraries are binary incompatible – but installed in the same location n Slight differences in rfio.h, too u ADS n stress testing found limitations in the pathtape interface n All users in a VO map to same ADS user s EDG testbed: Hit limits on concurrent writes

14. HEPiX Edinburgh May 2004 Experiences - development u Look for opportunities for component reuse n Used or improved components from other EDG WPs n Almost all parts of the Data Transfer components developed externally u Prefer Open Source n Often need to look at source to debug or supplement docs u Prototype implementations live longer than expected n SEs metadata system was implemented as prototype n Now replaced with improved system

15. HEPiX Edinburgh May 2004 Experiences – protocols u SRM 1 (and the EDG SE interface) is a control interface u Separating control from data transfer is useful u Can be used for load balancing, redirection, etc u Easy to add new data transfer protocols u However, files in cache must be released by the client or time out controldata xf client

16. HEPiX Edinburgh May 2004 Experiences – SRM 1 u WSDL definitely helped interoperability u Nevertheless we often saw and still see incompatibilities between various implementations u Many parts of the protocol are open to interpretation n E.g. SRMCopy – how does client know that copy has finished? And what the error is if it fails?

17. HEPiX Edinburgh May 2004 Experiences - protocols Confusingly many names for files: u LFN: Logical File Name u GUID: used by RM u SFN: Site filename n aka PFN – physical u SURL: SFN as URL u StFN: Storage file name u TURL: Transfer URL Replica manager Replica catalogue Storage Element Mass Storage User SE disk cache LFN GUID SURL TURL StFN SFN

18. HEPiX Edinburgh May 2004 Experiences – file access u Requirement: Clients must be able to access file in MSS independently Not all access can be controlled by SE SE cannot guarantee that it has the file… …nor that file hasnt been modified s Need to keep checksums of data u Also hard to guarantee reservations… n …unless the MSS has can guarantee space for the SE

19. HEPiX Edinburgh May 2004 Experiences – GridFTP u Requires Gridmap files n Pooled accounts (contributed by Andrew McNab) help make this scalable u Observed data transfer rates: 1-6 MB/s n But seen 30 MB/s with specially tuned settings u 0.7 seconds to transfer a 200 bytes file n Time is spent negotiating secure connection

20. HEPiX Edinburgh May 2004 Experiences – NFS u Used by jobs on worker nodes n SE is NFS mounted on WNs u Who converts TURL to local file name? n E.g. put the mount point into the filename u Potential need for several copies of file in disk cache n …due to ownership and access control issues n …and who gets the file? Who releases it? How does client get the filename? What if the file times out before the job runs?

21. HEPiX Edinburgh May 2004 Experiences – protocols u SRM messages are well designed u Each has a status: n Pending, Failed, Running, Done u An array of file objects n Status: Pending, Ready, Running, Failed, Done n TURL, if applicable n File metadata (size etc) Status=Running File1: SFN=se.ac.uk/foo status=Ready TURL=gsiftp://se.ac.uk/… File2: SFN=se.ac.uk/bar status=Failed Error=Access denied File3: SFN=se.ac.uk/nain status=Pending

22. HEPiX Edinburgh May 2004 Experiences – technical u The dCache SRM server/client n Standard – an SRM SHOULD work with these n …but not Open Source n Many idiosyncrasies s Fetches WSDL file before sending commands! So the SOAP server has to be a web server as well – and HTTP/1.1 is not entirely trivial to implement correctly – and we cant just use Apache because of GSI s Doesnt check all errors – e.g. if a request fails s Doesnt set status to Running

23. HEPiX Edinburgh May 2004 Experiences – security u Delegation is not ideal n Server must be trusted fully, no fine grained capabilities n Proxy certificates incompatible with normal certificates (but on IETF track) u Incompatibility between GSI (HTTPG) and HTTPS n We used HTTPS initially, hooking the SE core into Apache s Easy to develop and debug, reliable, well understood tech s Debug with curl etc n Need special clients for GSI n Andrew McNab proposes G-HTTPS: compatible with HTTPS, can still do delegation

24. HEPiX Edinburgh May 2004 Future – SRM 2.1 u Frozen-ish as of this spring u Provides lots of new functionality… n …much of which is optional… n …e.g. directory support, space reservation n SRM 2 basic will (probably) be functionally equivalent to SRM 1 u Space types, file types: Volatile, Durable, Permanent n Slightly complicated semantics

25. HEPiX Edinburgh May 2004 Future – SRM 2.1 u Guaranteed space reservation is hard n Unless you have infinite space u Define all state transitions for the protocol n Avoid ambiguities in SRM 1 u WSRF currently being discussed in SRM community

26. HEPiX Edinburgh May 2004 Future – SRB u There exists an SRB SRM interface n Or not? n Depending on who you ask u Build an SRM SRB interface? n E.g. build a handler for the EDG SE n Need to consider data transfer: GridFTP vs SRB

27. HEPiX Edinburgh May 2004 DICOM server support The Grid Storage Element WP10 DM2 DICOM Server Metadata Encrypt, anonymise Metadata Store keyStore patient metadata Access control on metadata required; different ACLs for different types of metadata Biomed applications in EGEE: Difficult task, not done in EDG

28. HEPiX Edinburgh May 2004 Conclusion u We have built a good framework u SRM 1 is a good choice n But file access is not trivial for the client u Technology is maturing n But not as quickly as most people would like n Still frequent tech and requirements changes n Prototypes go into production n Get it right vs build it now u Lots of idiosyncrasies – some end up getting promoted to standard u Still many challenges ahead – e.g. reservations