1. 11/27/2003IVOA Small Projects Meeting 20031 China-VO Data Access Service Based on OGSA Jian Sang National Astronomical Observatory of China Chinese Virtual Observatory

2. 11/27/2003IVOA Small Projects Meeting 20032 Outline VO,Grid and OGSA Build the catalog data service Build the image mosaic service Faced technical difficulties

3. 11/27/2003IVOA Small Projects Meeting 20033 The Increase Of Astronomical Data The number of pixels and the data double every year! The total area of astro telescopes in m**2 The total Gigapixels of CCDs.

4. 11/27/2003IVOA Small Projects Meeting 20034 Challenges The quantity of data nearly amounts to PB. The data is distributed and stored in heterogeneous DBMSs in heterogeneous host environments.

5. 11/27/2003IVOA Small Projects Meeting 20035 The VO’s Goal The VO’s initial goal is to federate existing astronomical data archives and provide standard services for manipulating these data. HOW TO REACH THIS GOAL ? The Grid technology can solve the problem!

6. 11/27/2003IVOA Small Projects Meeting 20036 What is Grid Grid technology has been driven by genesis from metacomputing, but … In practice, the Grid is about resource sharing and coordinated problem solving in dynamic, multi-institutional virtual organizations Focus on how to enable, maintain and control the sharing of resources to achieve a common goal

7. 11/27/2003IVOA Small Projects Meeting 20037 What “Grid“ offers:  Resource management protocols and services that support secure remote access to shared data resources and computing and the co-allocation of multiple resources.  Security solutions that support management of credentials and policies.  Information query protocols and services that provide configuration and status information about resources,organizations and services.  Data Management services that locate and transport datasets between storage systems and applications.

8. 11/27/2003IVOA Small Projects Meeting 20038 What is OGSA The Open Grid Services Architecture (OGSA) represents an evolution towards a Grid system architecture based on Web services concepts and technologies. The OGSA integrates key Grid technologies (including the Globus Toolkit with Web services mechanisms to create a distributed system framework based around the Open Grid Services Infrastructure (OGSI). In Grids,Everything is Service

9. 11/27/2003IVOA Small Projects Meeting 20039 The Open Grid Services Architecture Service orientation to virtualize resources From Web services:( everything is service) - Standard interface definition mechanisms:multiple protocol bindings,multiple implementations,local/remote transparency Building on Globus Toolkit: -Grid service :semantics for service interactions -Management of transient instances -Factory,Registry,Discovery,other services -Reliable and secure transport Multiple host environments:J2EE,.NET,C,…

10. 11/27/2003IVOA Small Projects Meeting 200310 The Structure of Grid Service

11. 11/27/2003IVOA Small Projects Meeting 200311 Grid service interfaces

12. 11/27/2003IVOA Small Projects Meeting 200312 Construct The Astronomical Data Grid The astronomical data service is the most fundamental and important component in Virtual Observatory. In the aspect of data share, the VO can be think as a astronomical Data Grid VO=Astronomical Data Grid

13. 11/27/2003IVOA Small Projects Meeting 200313 Outline VO,Grid and OGSA Build the catalog data access service Build the image mosaic service Faced difficulties

14. 11/27/2003IVOA Small Projects Meeting 200314 The Classification of Astronomical Data Service Astronomical Catalog Service Image Mosaic Service Spectrum Data Service Simulation Data Service

15. 11/27/2003IVOA Small Projects Meeting 200315 ClassDataSet NameData Amount （ zipped ） Catalog CDS/ADC Catalog sAbout 30G Other CatalogsAbout 120G Survey RealSky5G5G ROSAT X-ray Survey10G BATC360G DSS I60G DSS IIAbout 620G SDSS EDR30G SDSS DR1 (part)65G 2dF 2003 /2QZ7G Archive ROSAT X-ray Point28G Einstein X-ray Data5G Library ADS350G Total >1700GB Existing Astronomical Datasets we have

16. 11/27/2003IVOA Small Projects Meeting 200316 Build Catalog Data Service How to federate the catalog data into VO,that is, how to build Data Service using the existing databases and programs?

17. 11/27/2003IVOA Small Projects Meeting 200317 Define Catalog Service Interface Input Query Language : SQL(now),ADQL (plan) Output Data Format: VOTable 1.0 Catalog resource metadata registry protocol: VOResource 0.9 Some standards we used: input: ADQL query sentence output: VOTable format result it makes service interface/API simple.

18. 11/27/2003IVOA Small Projects Meeting 200318 How to create a catalog data service that can understand ADQL and generate VOTable format result?? we adopt two ways! Reconstruct the existing catalog DBMS Encapsulate search program,like pmm The CDS has offered search program for big catalog like USNO A2,0….. How to use existing databases and programs to create catalog data service

19. 11/27/2003IVOA Small Projects Meeting 200319 Catalog data service based on DB Catalog/metadata VOTable Wrapper ADQL/SQL Translator GT3 Interface ADQL VOTable JDBC SQLResultSet DBMS

20. 11/27/2003IVOA Small Projects Meeting 200320 Advantage and disadvantage Can sufficiently use the functions of SQL language and implement complex query. DBMSs offer the most powerful functions for data management and maintenance. Need many works to reconstruct the DBs. To big catalogs, like USNOB1.0,2MASS PSC, query efficiency is low

22. 11/27/2003IVOA Small Projects Meeting 200322 Data service based on search program VOTable Wrapper ADQL Translator GT3 Interface ADQLVOTable JNI/ stream program Data Files parameters

23. 11/27/2003IVOA Small Projects Meeting 200323 Advantage and disadvantage Positional search is quicker than DB Only offer search functions that programs could offer. Many programs only offer position search functions,no statistical functions.

24. 11/27/2003IVOA Small Projects Meeting 200324 Catalog Access Service Provided by us BandNameNum of objectsAmount X-rayRASS-BSC188060.03GB RASS-FSC1059240.10GB optical USNO B1.0 1045913669 38 GB USNO A2.0526280881 7 GB GSC 2.2.145585123740 GB GSC 1.2252417301.4 GB UCAC 127425433>0.5 GB UCAC 2483305714.5 GB Tycho225399130.5 GB Hipparcos1182180.05GB infrared2MASS PSC470992970127 GB 2MASS ESC16475993 GB radioNVSS17734840.44 GB FIRST8111170.1 GB Total About 110 catalogs About 220GB

25. 11/27/2003IVOA Small Projects Meeting 200325 How to call a Catalog data service Grid Client Resource Registry Data Service Factory Data Service Instance Create Data service Database 1. 2. 3. 4. 5. 6.

26. 11/27/2003IVOA Small Projects Meeting 200326 Use Data Service to build www service for end user End Users Data service Services Register Services Register MySQL Oracle 9i Files Grid Client Web server Web Client Resources Register Resources Register http Data Mining Service Data processing Service Data Visualization Service End user don’t know where the data services are

27. 11/27/2003IVOA Small Projects Meeting 200327 Use data service to create other service Our next work is to build a multi-wavelength cross-identification service (MWCI)based on the catalog data service. What is multi-wavelength cross-identification ? To cross-identify datasets by positional consistency, we can understand objects from different wavelength properties.

28. 11/27/2003IVOA Small Projects Meeting 200328 The steps of multi-wavelength cross-identification Cross-identify datasets from different wavelengths within error radius. Divide the result of cross-identification into three situations: one-to-one, one-to-two, one- to-many. Choose the one-to-one entry for data mining The other two situations need statistical analysis to determine which source are the true counterpoint.

29. 11/27/2003IVOA Small Projects Meeting 200329 Requirements Locate the datasets that users want to use. (dataset discovery) How to cross-match the datasets in heterogeneous DBMSs at different locations effectively and efficiently. Find storage resource to store the results

30. 11/27/2003IVOA Small Projects Meeting 200330 User Application NVSS Storage Service Provider storage Factory MWCI Factory Registry Data Service 2MASS Data Service...... MWCI Service Provider...... storage MWCI 1 2 3 4 4 5 6 7 5 6

31. 11/27/2003IVOA Small Projects Meeting 200331 Outline VO,Grid and OGSA Build the catalog data access service Build the image mosaic service Faced technical difficulties

32. 11/27/2003IVOA Small Projects Meeting 200332 Build The Image Mosaic Service Use DSS-I sky image build our first image mosaic service.

33. 11/27/2003IVOA Small Projects Meeting 200333 the definition of interface of service Input parameters: 1.RA,2.Dec,3.image height,4.image width transport protocols :gridFTP Output Data format :fits

34. 11/27/2003IVOA Small Projects Meeting 200334 Realization of DSS-I image mosaic service GT3 Interface JNI/ Fits file GetImage parameters DSS-I Image Files GridFTP

35. 11/27/2003IVOA Small Projects Meeting 200335 Outline VO,Grid and OGSA Build the catalog data access service Build the image mosaic service Faced technical difficulties

36. 11/27/2003IVOA Small Projects Meeting 200336 Technical Difficulties service/resource registry and discovery! ADQL2SQL translator protocol shortcoming

37. 11/27/2003IVOA Small Projects Meeting 200337 protocol shortcomings The shortcomings of VOTable 1.0 protocol 1.How to encapsulate result of join query!! 2.The standard to encapsulating spectrum data 3.the definition of FIELD element is not strict and uncompleted The shortcoming of UCD 1.Can’t express concrete meaning,such as “ERROR”,Error for what?? 2. incomplete, example:HTMID has no UCD Lack of standard for Unit

38. 11/27/2003IVOA Small Projects Meeting 200338 Q & A ? www..org Thank You

39. 11/27/2003IVOA Small Projects Meeting 200339 Our provided catalogs in Catalog Service BandNameNum of objectsAmount X-rayRASS-BSC188060.03GB RASS-FSC1059240.10GB optical USNO B1.0 1045913669 38 GB USNO A2.0526280881 7 GB GSC 2.2.145585123740 GB GSC 1.2252417301.4 GB UCAC 127425433>0.5 GB UCAC 2483305714.5 GB Tycho225399130.5 GB Hipparcos1182180.05GB infrared2MASS PSC470992970127 GB 2MASS ESC16475993 GB radioNVSS17734840.44 GB FIRST8111170.1 GB Total About 110 catalogs About 220GB

40. 11/27/2003IVOA Small Projects Meeting 200340 The Step Of Calling A Data Service

41. 11/27/2003IVOA Small Projects Meeting 200341 Transparencies for Astro Data Access Heterogeneity Transparency Name Transparency Distribution Transparency

42. 11/27/2003IVOA Small Projects Meeting 200342 What is Grid Service?

43. 11/27/2003IVOA Small Projects Meeting 200343 What Is The Data Grid DataGrid : A dynamic logical namespace that enables coordinated sharing of heterogeneous distributed storage resources and digital entities based on local and global policies across administrative domains in a virtual enterprise. DataGrid –Logical name space for location independent identifiers –Abstractions for storage repositories, information repositories, and access APIs –Latency management

44. 11/27/2003IVOA Small Projects Meeting 200344 Data Grid Using a Data Grid – in Abstract Ask for data User asks for data from the data grid Data delivered The data is found and returned Where & how details are managed by data grid

45. 11/27/2003IVOA Small Projects Meeting 200345