Presentation on theme: "11 Sep 2006 NVO Summer School 20061 Managing data in the VO Matthew J. Graham CACR/Caltech T HE US N ATIONAL V IRTUAL O BSERVATORY."— Presentation transcript:
11 Sep 2006 NVO Summer School 20061 Managing data in the VO Matthew J. Graham CACR/Caltech T HE US N ATIONAL V IRTUAL O BSERVATORY
11 Sep 2006 NVO Summer School 20062 The importance of data Data is the raison dêtre of the VO LSST is the data source nonpareil –data rates of 540MB/s ~16TB in 8 hrs –final archive > 3PB of data VO Wheel Well-established ways of handling distributed data: – SRB – PVFS – OGSA-DAI
11 Sep 2006 NVO Summer School 20063 Requirements A distributed storage mechanism that allows easy reference to data without concerns about physical location. Primary use cases: –User wants to easily publish and share own data –Data need to reside close to computation nodes Data use cases: –Client has data: stored locally: transfers it to service stored locally: service retrieves it stored elsewhere: service retrieves it –Service generates data: stores it locally: notifies client of location transfers it to the clients local store transfers it to a client-designated store
11 Sep 2006 NVO Summer School 20064 Logical architecture User view Logical namespace Physical storage
11 Sep 2006 NVO Summer School 20065 VOSpace Provides a uniform interface to existing or new data storage locations (Facade pattern) Structured/unstructured data both first level A peer network of VOSpace servers
11 Sep 2006 NVO Summer School 20066 Data structures - I Each data object is represented as a node: Nodes are identified by a vos://[service]/[name] identifier: – Why not ivo://nvo.caltech/vospace/mydata1? – RFC2396 - hierarchy
11 Sep 2006 NVO Summer School 20067 Unstructured DataNode Data structures - II Each node contains a map of key:value properties: 2006-09-11T13:35:51Z There are currently four types of node: Node DataNode Structured DataNode readonly=tru e
11 Sep 2006 NVO Summer School 20068 Data structures - III Data nodes contain a list of data views (formats) that the node can accept and provide: …
11 Sep 2006 NVO Summer School 20069 Data structures - IV … –Why not use MIME type? Easier to define new astronomy specific data types
11 Sep 2006 NVO Summer School 200610 Data structures - V Data transfers are represented by transfers: The format of the data transfer is specified by a view: The protocol of the data transfer is specified by a protocol: … http://net.ivoa/vospace/protocols/http- get http://192.168.1.33:7007/vospace
11 Sep 2006 NVO Summer School 200611 Data structures - VI The space has a list of which protocols the service can accept to fetch data and what protocol endpoints it provides: Why not use protocol schemes?
11 Sep 2006 NVO Summer School 200612 Operations - I Service metadata: –getProtocols(): –getViews():, –getProperties():,, Creating and manipulating nodes –createNode( ): –deleteNode(uri): - –listNodes(token, limit, detail, ): token, limit, –moveNode(uri, ): –copyNode(uri, ):
11 Sep 2006 NVO Summer School 200613 Operations - II Manipulating node metadata –getNode(uri): –setNode( ): Transferring data –pushToVoSpace(, ):, –pullToVoSpace(, ): –pushFromVoSpace(uri, ): - –pullFromVoSpace(uri, ):
11 Sep 2006 NVO Summer School 200614 Authentication and authorization WS-Security Access policies: –No access control –No authorization but authentication –Clients may not create or change nodes –Nodes are considered to be owner by the user who created them.
Your consent to our cookies if you continue to use this website.