19/05/2011 CSTS File transfer service discussions CSTS-File Transfer service discussions (2) CNES position

19/05/2011 CSTS File transfer service discussions File Content Description –Define manifest files identifying –File content (to the level that the recipient needs to know) –File format? Processing Instructions –Combine with content description? Pull Model, Push Model, or both? [Mouy Francois-Xavier] No needs identified yet for push model (but still a little bit of work to do) Subscription and automatic push? [Mouy Francois-Xavier] for push model Subscription and notification of new data? [Mouy Francois-Xavier] how send notifications if provider not bind. Bind return could notify a list of avaliable data File Management Services (see dedicated slide) File Directory / Catalogue Servcies File Management [Mouy Francois-Xavier] for push model –Create / Delete –Move –Local copy Directory Management [Mouy Francois-Xavier] for push model –Create / Delete / Copy Storage Control [Mouy Francois-Xavier] needs to be configurable –Duration of Storage –Maximum Volume (per client?)

19/05/2011 CSTS File transfer service discussions Security (see dedicated slide) Off the shelf FT protocols support security but must be configured and depending on features used may need supporting infrastructure Firewall issues Access Control –At what level? (mission, individual accounts, …) [Mouy Francois- Xavier] Mission level seem to be the right level –Access Control Groups –More complex schemes What level of privacy / access control is needed? [Mouy Francois- Xavier] We need to be able to provide this kind of features before security person's asks –E.g. can user X see that data are stored for user Y? [Mouy Francois- Xavier] he shouldn't ;-) Key management issues Others?

19/05/2011 CSTS File transfer service discussions Reading Martin initial thoughts. –Protocol choices are wide. If we use one or other protocol, it comes with all this complexity (advantages, drawbacks). –Some features of standards COTS are not useful in our context, but you have to tune and secure those features anyway. –Most of the difficulty come with the push model (security, behavior), the pull model is very near the existing offline model. Initial CNES position: –Firstly, we thought we only need the pull model which is quite simple than the push model. –Furthermore, File transfer function is already done by other ways. (ftp servers in parallel) –Therefore, why redo it ?

19/05/2011 CSTS File transfer service discussions However, Experience show us that deploying a file transfer feature with common COTS is not quite easy that it is on paper (e.g. with ftp: passive form, active, dynamic port ranges, firewalls, user management, … ). Using CSTS could simplify the integration because the network security aspects are already done. A part of the reporting too (notifications). CNES think CSTS could be a good « Secured Pipe » to the Ground center. –A binded instance is maintained (Heartbeat) –The Start/Stop mechanisms could be useful to drive providers behavior –We are able to use CSTS services via a high level of firewalls system

19/05/2011 CSTS File transfer service discussions Candidate standards (from initial thoughts) File Transfer Standards –FTP, FTPS, SFTP (SSH based FT) –HTTP / HTTPS ? –Others? CCSDS Standards –XFDU ? –Others?

19/05/2011 CSTS File transfer service discussions XFDU (xml formatted data unit) GOAL : organizing transferred data in an Open Archive environment (OAIS) A container (zip file) with a descripition of its content in a manifest file. The logical description points to the physical content which can be either local or distant

19/05/2011 CSTS File transfer service discussions XFDU description The Information Package Map contains the logical view of the package. It’s a hierarchical xml tree representing the content of the package. Each leaf of the tree is a Content Unit and can be referred to from the other parts of the package (i.e. information is linked by the Content Unit reference ID). The Data Object Section contains all the physical information needed to get the file objects out. The Metadata Section records all of the metadata for all items in the package. The Behavior Section associates executable code with the content of the package.

19/05/2011 CSTS File transfer service discussions XFDU manifest structure The structure map is a hierarchical view of logical references which are pointing to physical data, metadata and specific behaviors.

19/05/2011 CSTS File transfer service discussions XFDU : benefits Unique Identification of the data ( ContentUnit ID) Interoperability by using an XML schema description Hierarchical structure of the information Data physical access (DataObject) separated from the logical concepts (ContentUnit) Metadata associated with the ContentUnit Specific processes associated with the ContentUnit (BehaviorObject)

19/05/2011 CSTS File transfer service discussions XFDU : drawbacks XFDU developed for approaching needs but not totally the sames (multi-parts, files, cdrom) Few implementations due to standard complexity.: –NASA XFDU library, ESA SAFE’s archive for ENVISAT data, CNES’s PAIMAS Behavior function not really validated

19/05/2011 CSTS File transfer service discussions CSTS-XFDU : conclusion The data organization has been studied earlier by the MOIMS/IPR group. XFDU is an answer to csts concerns with the following conditions: –Needs of an XFDU « light » to answer the requirements. –Needs to study the behavior statement..

19/05/2011 CSTS File transfer service discussions CSTS-XFDU : conclusion XFDU (simplified one) Could be a good system to integrate the file transfer feature. The behavior side of XFDU could be a response to File transfer and more … CNES thinks it could be a possible cooperation and some people are interested to study those aspects …

19/05/2011 CSTS File transfer service discussions

19/05/2011 CSTS File transfer service discussions Some use cases at CNES Distant management of stations –Management by xfdu of TM archived in station –TM playback via CSTS-offline service Inter-facilities scheduler –Distant scripts used by the scheduler managed by xfdu –Launch via services CSTS –Re-use of CNES well-known scheduler GUI …

19/05/2011 CSTS File transfer service discussions Let’s take an example…. XFDU over CSTS for playing recorded telemetry

19/05/2011 CSTS File transfer service discussions Telemetry transfert and play A facility wants to transfert simulated telemetry to a station and to be able to play it back later. The user build an xfdu package (X1) with the TM file and two scripts (play and rewind) All the configuration and security concerns to be solved by the management layer.

19/05/2011 CSTS File transfer service discussions CSTS FT user TM 1 PLAY REW XFDU X1 CSTS FT provider bind STEP 1 CONNEXION

19/05/2011 CSTS File transfer service discussions CSTS FT user TM 1 PLAY REW XFDU X1 TM 1 PLAY REW XFDU X1 CSTS FT provider bind START PUT X1 STEP 2 XFDU TRANSFERT

19/05/2011 CSTS File transfer service discussions CSTS FT user TM 1 PLAY REW XFDU X1 TM 1 PLAY REW XFDU X1 CSTS FT provider TM 1 PLAY REW bind START PUT X1 INSTALL STEP 3 XFDU AUTOMATIC INSTALL

19/05/2011 CSTS File transfer service discussions CSTS FT user TM 1 PLAY REW XFDU X1 TM 1 PLAY REW XFDU X1 CSTS FT provider TM 1 PLAY REW bind START PUT X1 START LIST START LIST X1 PLAY, REW X1 INSTALL STEP 4 The user lists all available xfdus and their behaviors

19/05/2011 CSTS File transfer service discussions CSTS FT user TM 1 PLAY REW XFDU X1 TM 1 PLAY REW XFDU X1 CSTS FT provider TM 1 PLAY REW bind START PUT X1 START LIST START LIST X1 START RUN X1 PLAY TM 1 PLAY, REW X1 INSTALL STEP 5 The user plays the TM1 telemetry back