1. EdSkyQuery-G Overview Brian Hills, December 2004 www.edikt.org

2. Contents  Edikt  Motivation & Aims  Architecture  Current Status  Results  Future Outlook

3. Edikt  “e-Science Data, Information and Knowledge Transformation” –Bridge the gap between applications and computer science:  Produce robust tools…  …for real application science problems…  …test them under extreme science conditions…  …and keep an eye on the commercial possibilities. –Projects which may be of interest to astronomers:  BinX, Eldas & EdSkyquery-G. –Visit: www.edikt.orgwww.edikt.org

4. Astronomy Requirements  Sky Surveys collect masses of data to be managed: –For example, Sloan Digital Sky Survey: 15TB. –2 * 10GB databases to be used for the project. –Edikt will have access to a 155TB SAN.  Further research by leveraging data from different surveys: –Must identify same object from different catalogues.  Require a “federated” view: –Data is distributed, homogeneous, large scale. –Building one big data warehouse isn’t feasible. –Interoperable services to combine disparate data sources. Is the middleware up to the task?

5. EdSkyQuery-G: Motivation & Aims  Support the “Open SkyQuery Initiative” –Move from a.NET-specific implementation. –Enable similar functionality on other platforms.  Extensible framework for e-science: –Handle heterogeneous archives. –‘Plug in’ algorithms e.g. Nearest Neighbour. –Interact with Astrogrid components & VO. –Leveraged for the BRIDGES project to perform simple joins.  Apply Eldas to large scale E-Science problems: –Test: functionality, scalability & performance.  Cross team collaboration: –Dr. Bob Mann (UoE, ROE), Edikt, EPCC, NeSC.

6. Portal SkyNode 1 Client SkyQuery: High Level Architecture SkyNode 2 1. User submits query. 2. Client sends query to Portal. 3. Portal invokes SkyNode with an execution plan. 4. Recursive call to next SkyNode in the execution plan. 7. Performs cross- match and returns results. 8. Performs cross- match and returns final set of results. 9. Returns results for display. SkyNode 3 5. Recursive call to last SkyNode in the execution plan. 6. Runs query and returns results.

7. EdSkyQuery-G: Architecture  Inspired by Greg Riccardi’s paper for DAIS-WG: –http://www.cs.fsu.edu/~riccardi/grid/skyquery.pdfhttp://www.cs.fsu.edu/~riccardi/grid/skyquery.pdf  Discusses two approaches: 1.Access recipes for service interactions. 2.Retained state for service interactions.  Potential benefits of #2: –Scalability –Robustness –Usability

8. EdSkyQuery-G: Architecture 6. Transfer results file. Query Builder Service Manager SkyNode 2 DB2 - Linux 1. User submits query. 2. Query Builder sends query to Service Manager (SM). 4. Handle to results. SkyNode 1 DB2 - XP 7. Handle to results. 5. SM invokes SkyNode 1 with an execution plan + results handle. 8. SM returns a handle to the results. 3. SM invokes SkyNode 1 with an execution plan. 9. Client may pull results back from SkyNode.

9. EdSkyQuery-G: Service Manager JBOSS Grid Service Layer doQuery getMetadata Distributed Query Processor Split Optimize Execute Service Manager Call to Query service Client Uses Service Generic Parser Call to Xmatch service Call to Info service

10. EdSkyQuery-G: SkyNode Architecture SkyNode DB2 Invoked via the Service Manager EldasSkyNode Grid Services Query XMatchInfo Query & Stored Proc invocations

11. EdSkyQuery-G: Stored Procedures EDR Results:.csv.xml SSA Results:.csv.xml Export Transport Import Export Results:.csv.xml Export

12. Current Status  Pre-Alpha (internal release), November ‘04.  End-to-end invocation of all components: –Client->ServiceManager->SkyNode->Database  2 * 10GB DB2 test databases: –SSA from SuperCosmos, hosted by NeSC. –EDR from SDSS, hosted by EPCC.  Limitations: –SM: No query parser/splitting. –Simple cross database join: not yet using XMatch. –No data transport, other than JDBC between databases. –Final results reside on database server.

13. EdSkyQuery-G: Pre-Alpha Stored Procedures EDRSSA Results:.csv.xml Export JDBC Query Results Import Results:.csv.xml Export

14. Results  Tested with 3 queries from ROE cookbook: –http://surveys.roe.ac.uk/ssa/sqlcookbook.htmlhttp://surveys.roe.ac.uk/ssa/sqlcookbook.html –Queries #16, #17, #19.  Results show: –Exporting data is quick. –Importing data is >10 * slower:  Should we use native DB calls rather than JDBC for import only? –Queries slow:  Need more indexes and database tuning? Query No.#Rows selected Export Time (secs) Import Time (secs) Join Query Time (secs) Total Time (secs) 16488,718130158511102825 17383,67296113814702704 194,667821514741571

15. Deliverables  Software (Internal): –Prototype client & GUI client. –Service Manager. –Eldas + Skynode Interface. –Database stored procedures (Java).  Documentation (some on NescForge): –Use Cases, Requirements, Design. –Performance Testing, Installation.  Papers –AHM 04 Poster & Paper:  http://www.allhands.org.uk/proceedings/papers/123.pdf http://www.allhands.org.uk/proceedings/papers/123.pdf –ADASS 04 Paper.

16. Short Term Focus  Enable science: –Compare different cross match algorithms. –Incorporate XMatch, CSIRO, ROE as stored procedures.  Data Transfer – SCP: –Between databases. –Pull results back to client. –Deliver to a third party.  Client & Service Manager enhancements: –Handle broader range of queries.  Performance: –Compare with pre-alpha benchmarks.  Improve test infrastructure.

17. Longer Term Focus: AstroGrid Client Registry MySpace Parser Query Builder Service Manager SkyNode 2 DB2 - Linux SkyNode 1 DB2 - XP

18. Longer Term Focus  Interaction with Astrogrid components: –Clients, Registry, ADQL Parser, MySpace.  OpenSkyQuery: –Compliance with interfaces. –Test with other DBMS and catalogues.  Query Builder GUI/Data Integration tool: –Lead user through choosing fields from different datasets.  GridFTP: –Currently unsuitable…. –NeSC course in Jan 05 may reveal more.

19. Thank you Questions?