Presentation on theme: "11 Sep 2006 NVO Summer School 20061 Managing data in the VO Matthew J. Graham CACR/Caltech T HE US N ATIONAL V IRTUAL O BSERVATORY."— Presentation transcript:
11 Sep 2006 NVO Summer School 20061 Managing data in the VO Matthew J. Graham CACR/Caltech T HE US N ATIONAL V IRTUAL O BSERVATORY
11 Sep 2006 NVO Summer School 20062 The importance of data Data is the raison dêtre of the VO LSST is the data source nonpareil –data rates of 540MB/s ~16TB in 8 hrs –final archive > 3PB of data VO Wheel Well-established ways of handling distributed data: – SRB – PVFS – OGSA-DAI
11 Sep 2006 NVO Summer School 20063 Requirements A distributed storage mechanism that allows easy reference to data without concerns about physical location. Primary use cases: –User wants to easily publish and share own data –Data need to reside close to computation nodes Data use cases: –Client has data: stored locally: transfers it to service stored locally: service retrieves it stored elsewhere: service retrieves it –Service generates data: stores it locally: notifies client of location transfers it to the clients local store transfers it to a client-designated store
11 Sep 2006 NVO Summer School 20064 Logical architecture User view Logical namespace Physical storage
11 Sep 2006 NVO Summer School 20065 VOSpace Provides a uniform interface to existing or new data storage locations (Facade pattern) Structured/unstructured data both first level A peer network of VOSpace servers
11 Sep 2006 NVO Summer School 20066 Data structures - I Each data object is represented as a node: Nodes are identified by a vos://[service]/[name] identifier: – Why not ivo://nvo.caltech/vospace/mydata1? – RFC2396 - hierarchy
11 Sep 2006 NVO Summer School 20067 Unstructured DataNode Data structures - II Each node contains a map of key:value properties: 2006-09-11T13:35:51Z There are currently four types of node: Node DataNode Structured DataNode readonly=tru e
11 Sep 2006 NVO Summer School 20068 Data structures - III Data nodes contain a list of data views (formats) that the node can accept and provide: …
11 Sep 2006 NVO Summer School 20069 Data structures - IV … –Why not use MIME type? Easier to define new astronomy specific data types
11 Sep 2006 NVO Summer School 200610 Data structures - V Data transfers are represented by transfers: The format of the data transfer is specified by a view: The protocol of the data transfer is specified by a protocol: … http://net.ivoa/vospace/protocols/http- get http://192.168.1.33:7007/vospace
11 Sep 2006 NVO Summer School 200611 Data structures - VI The space has a list of which protocols the service can accept to fetch data and what protocol endpoints it provides: Why not use protocol schemes?
11 Sep 2006 NVO Summer School 200612 Operations - I Service metadata: –getProtocols(): –getViews():, –getProperties():,, Creating and manipulating nodes –createNode( ): –deleteNode(uri): - –listNodes(token, limit, detail, ): token, limit, –moveNode(uri, ): –copyNode(uri, ):
11 Sep 2006 NVO Summer School 200613 Operations - II Manipulating node metadata –getNode(uri): –setNode( ): Transferring data –pushToVoSpace(, ):, –pullToVoSpace(, ): –pushFromVoSpace(uri, ): - –pullFromVoSpace(uri, ):
11 Sep 2006 NVO Summer School 200614 Authentication and authorization WS-Security Access policies: –No access control –No authorization but authentication –Clients may not create or change nodes –Nodes are considered to be owner by the user who created them.