Presentation is loading. Please wait.

Presentation is loading. Please wait.

A PPARC funded project AstroGrid and OGSA-DAI: Early experiences Kona Andrews Guy Rixon Institute of Astronomy University of Cambridge.

Similar presentations


Presentation on theme: "A PPARC funded project AstroGrid and OGSA-DAI: Early experiences Kona Andrews Guy Rixon Institute of Astronomy University of Cambridge."— Presentation transcript:

1 A PPARC funded project AstroGrid and OGSA-DAI: Early experiences Kona Andrews Guy Rixon Institute of Astronomy University of Cambridge

2 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Data Set Access: The DataCenter DataCenter component Data Access Plug-in Data Web Services Tomcat / Axis

3 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences DataCenter JDBC plug-in DataCenter component JDBC DBMS Web Services Tomcat / Axis

4 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences DataCenter custom plug-in DataCenter component File querier FITS files Web Services Tomcat / Axis

5 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences DataCenter proxy plug-in DataCenter component Service proxy Vizier Web Services Tomcat / Axis

6 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences What we imagined OGSA- DAI Web Services DataCenter Component OGSA-DAI client plug-in Tomcat / Axis Grid Service Ports Tomcat / OGSA

7 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Problem OGSA- DAI Web Services DataCenter Component OGSA-DAI client plug-in Tomcat / Axis Grid Service Ports Tomcat / OGSA

8 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Problem OGSA- DAI Web Services DataCenter Component OGSA-DAI client plug-in Tomcat / Axis Grid Service Ports Tomcat / OGSA

9 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Nasty solution OGSA- DAI Web Services DataCenter Component OGSA-DAI client plug-in Tomcat / Axis Grid Ports Tomcat / OGSA OGSA-DAI client Separate JVM

10 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Current use of OGSA-DAI activities Data Center OGSA-DAI Postgres DBMS ADQL

11 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Current use of OGSA-DAI activities Data Center OGSA-DAI Postgres DBMS ADQL to SQL ADQL SQL

12 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Current use of OGSA-DAI activities Data Center OGSA-DAI Postgres DBMS ADQL to SQL sqlQueryStatement ADQL SQL

13 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Current use of OGSA-DAI activities Data Center OGSA-DAI Postgres DBMS ADQL to SQL sqlQueryStatement ADQL SQL Result rows SQL (to XML RowSet)

14 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Current use of OGSA-DAI activities Data Center OGSA-DAI Postgres DBMS ADQL to SQL sqlQueryStatement deliverFromURL (to XML RowSet) ADQL SQL Result rows SQL

15 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Current use of OGSA-DAI activities Data Center OGSA-DAI Postgres DBMS ADQL to SQL sqlQueryStatement xslTransform deliverFromURL (to XML RowSet) ADQL SQL Result rows SQL VOTable

16 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Current use of OGSA-DAI activities Data Center OGSA-DAI Postgres DBMS ADQL to SQL sqlQueryStatement xslTransform deliverToFile / deliverToGFTP deliverFromURL (to XML RowSet) ADQL SQL VOTable Result rows SQL

17 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Current use of OGSA-DAI activities Data Center OGSA-DAI Postgres DBMS ADQL to SQL sqlQueryStatement xslTransform deliverToFile / deliverToGFTP deliverFromURL (to XML RowSet) ADQL SQL VOTable Result rows SQL Deliver results

18 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Comments Astronomers often need LARGE result sets Thousands to millions of rows Memory usage is a big issue OGSA-DAI 3.1 much better than 3.0 xslTransform activity hogging memory

19 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences OGSA-DAI speed penalty Rough figures only Measured without xslTransform activity Will likely vary with available memory

20 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Planned extension: User tables OGSA- DAI DBMS Data Center R/O Datasets QueryQuery results

21 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Planned extension: User tables OGSA- DAI DBMS Data Center R/O Datasets User tables QueryQuery resultsData flow GDS to GDS transfer Bulk load OGSA- DAI

22 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences DBMS Planned extension: User tables OGSA- DAI DBMS Data Center Data Center OGSA- DAI Table Manager R/O Datasets User tables QueryQuery resultsData flow

23 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences DBMS Planned extension: User tables OGSA- DAI DBMS Data Center Data Center OGSA- DAI Table Manager Client User data R/O Datasets User tables QueryQuery resultsData flow

24 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Comments Will be relying on sqlBulkLoadRowSet (!) Table management facilities currently thin Not using X.509 auth at user granularity Still require W/S-based access to G/S

25 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences Our needs Ability to handle very large result sets - Dont rely on loading whole result set into memory! Suite of DB/table management activities - List tables in DB - Describe table column types / space usage - Last access/modification time - Etc etc etc Ability to create and delete indices on tables Ultra-simple installation procedures

26 OGSA-DAI UG Meeting AstroGrid and OGSA-DAI: Early experiences The End


Download ppt "A PPARC funded project AstroGrid and OGSA-DAI: Early experiences Kona Andrews Guy Rixon Institute of Astronomy University of Cambridge."

Similar presentations


Ads by Google