1. 11 Sep 2006 NVO Summer School 20061 Managing data in the VO Matthew J. Graham CACR/Caltech T HE US N ATIONAL V IRTUAL O BSERVATORY

2. 11 Sep 2006 NVO Summer School 20062 The importance of data Data is the raison dêtre of the VO LSST is the data source nonpareil –data rates of 540MB/s ~16TB in 8 hrs –final archive > 3PB of data VO Wheel Well-established ways of handling distributed data: – SRB – PVFS – OGSA-DAI

3. 11 Sep 2006 NVO Summer School 20063 Requirements A distributed storage mechanism that allows easy reference to data without concerns about physical location. Primary use cases: –User wants to easily publish and share own data –Data need to reside close to computation nodes Data use cases: –Client has data: stored locally: transfers it to service stored locally: service retrieves it stored elsewhere: service retrieves it –Service generates data: stores it locally: notifies client of location transfers it to the clients local store transfers it to a client-designated store

4. 11 Sep 2006 NVO Summer School 20064 Logical architecture User view Logical namespace Physical storage

5. 11 Sep 2006 NVO Summer School 20065 VOSpace Provides a uniform interface to existing or new data storage locations (Facade pattern) Structured/unstructured data both first level A peer network of VOSpace servers

6. 11 Sep 2006 NVO Summer School 20066 Data structures - I Each data object is represented as a node: Nodes are identified by a vos://[service]/[name] identifier: – Why not ivo://nvo.caltech/vospace/mydata1? – RFC2396 - hierarchy

7. 11 Sep 2006 NVO Summer School 20067 Unstructured DataNode Data structures - II Each node contains a map of key:value properties: 2006-09-11T13:35:51Z There are currently four types of node: Node DataNode Structured DataNode readonly=tru e

8. 11 Sep 2006 NVO Summer School 20068 Data structures - III Data nodes contain a list of data views (formats) that the node can accept and provide: …

9. 11 Sep 2006 NVO Summer School 20069 Data structures - IV … –Why not use MIME type? Easier to define new astronomy specific data types

10. 11 Sep 2006 NVO Summer School 200610 Data structures - V Data transfers are represented by transfers: The format of the data transfer is specified by a view: The protocol of the data transfer is specified by a protocol: … http://net.ivoa/vospace/protocols/http- get http://192.168.1.33:7007/vospace

11. 11 Sep 2006 NVO Summer School 200611 Data structures - VI The space has a list of which protocols the service can accept to fetch data and what protocol endpoints it provides: Why not use protocol schemes?

12. 11 Sep 2006 NVO Summer School 200612 Operations - I Service metadata: –getProtocols(): –getViews():, –getProperties():,, Creating and manipulating nodes –createNode( ): –deleteNode(uri): - –listNodes(token, limit, detail, ): token, limit, –moveNode(uri, ): –copyNode(uri, ):

13. 11 Sep 2006 NVO Summer School 200613 Operations - II Manipulating node metadata –getNode(uri): –setNode( ): Transferring data –pushToVoSpace(, ):, –pullToVoSpace(, ): –pushFromVoSpace(uri, ): - –pullFromVoSpace(uri, ):

14. 11 Sep 2006 NVO Summer School 200614 Authentication and authorization WS-Security Access policies: –No access control –No authorization but authentication –Clients may not create or change nodes –Nodes are considered to be owner by the user who created them.

15. 11 Sep 2006 NVO Summer School 200615 Forthcoming attractions Containers Links Asynchronous transfers Querying Replicas

16. 11 Sep 2006 NVO Summer School 200616 Federation by links