The Japanese Virtual Observatory (JVO) Yuji Shirasaki National Astronomical Observatory of Japan

Goals of the JVO project Provides a seamless access to the distributed astronomical data (catalogs, archives) Provides a seamless access to the distributed astronomical data (catalogs, archives) NAOJ, JAXA, CDS, ESA, ESO, HESARC, SDSS, STSCI, NED, CADC … NAOJ, JAXA, CDS, ESA, ESO, HESARC, SDSS, STSCI, NED, CADC … Data discovery, Database federation  Data Grid Data discovery, Database federation  Data Grid Provides analysis environment under which multi-wavelength study can be easily conducted Provides analysis environment under which multi-wavelength study can be easily conducted An analysis tool is dedicated for each instrument An analysis tool is dedicated for each instrument A homogeneous interface to the software A homogeneous interface to the software Easy access to the software Easy access to the software Server side data analysis  Computing Grid Server side data analysis  Computing Grid

History of JVO and Grid FY2002: Prototype 1 FY2002: Prototype 1 Globus TK 2.4 Globus TK 2.4 FY2003: Prototype 2 FY2003: Prototype 2 Globus TK 3.0 Globus TK 3.0 FY2004: Prototype 3 FY2004: Prototype 3 Web Service Web Service FY2005: Operation Model FY2005: Operation Model Web Service Web Service MDS (Monitoring and Discovery Service)  Grid Computing (Subaru Data reduction, HyperZ, … ) MDS (Monitoring and Discovery Service)  Grid Computing (Subaru Data reduction, HyperZ, … )

Experience of GT 2.4 (Proto1) All the data and analysis service are implemented based on GT 2.4 All the data and analysis service are implemented based on GT 2.4 GRAM, GridFTP, GSI, Simple CA GRAM, GridFTP, GSI, Simple CA UDDI is used to resolve the service URL UDDI is used to resolve the service URL Too hard to learn the usage of this TK. Too hard to learn the usage of this TK. Slow Job submission (20 sec/submission) Slow Job submission (20 sec/submission) Needed adjustment of Poling time Needed adjustment of Poling time UDDI has no functionality to store various kind of metadata UDDI has no functionality to store various kind of metadata The most of the component was replaced by WS based component in GT 3… The most of the component was replaced by WS based component in GT 3…

Portal UDDI SimpleCA Data Service Analysis Servers User Storage User GRAM HTTP Grid FTP GSI Web Service User Request Service Discovery Data Trans. Job Submit Auth.

Experience of GT 3.0 (Proto2) All the data service and analysis service are implemented based on GT 3.0 All the data service and analysis service are implemented based on GT 3.0 Grid Service, RFTP, GSI-SFS (by Osaka univ.), SimpleCA Grid Service, RFTP, GSI-SFS (by Osaka univ.), SimpleCA Relatively easy but still too hard… Relatively easy but still too hard… Performance is good (because of no authentication is made for using Grid Service ?) Performance is good (because of no authentication is made for using Grid Service ?) Merger of Grid (OGSA) and Web Service is promising, but now it is rapidly evolving  wait until the toolkit become stable ? Merger of Grid (OGSA) and Web Service is promising, but now it is rapidly evolving  wait until the toolkit become stable ?

Portal Registry (XML DB) SimpleCA Data Sevice Analysis Servers User Storage User Grid Service HTTP RFT LDAP Data Trans. Invoke Service Discovery Auth. User Request

Database Federation with the IVOA standard (Proto 3 ~ Operational Model) First implementation of the IVOA standards First implementation of the IVOA standards All the data services are implemented based on Web Service or simply an HTTP get method. All the data services are implemented based on Web Service or simply an HTTP get method. MDS for Grid computing MDS for Grid computing Workflow Language and Workflow Engine. Workflow Language and Workflow Engine. There are plenty of information about WS implementation over the Internet.  development was rather easy and efficient. There are plenty of information about WS implementation over the Internet.  development was rather easy and efficient. Succeeded in the interoperability with the major VO projects. Succeeded in the interoperability with the major VO projects.

Portal P. Reg. S. Reg. LDAP SkyNode SIAP/SSAP SkyNode 1.0 解析 Analysis Servers MDS P. Reg. User Storage User Web Service HTTP FTP LDAP Other VOs User Request Invoke Service Discovery Auth. Metadata Harvesting

Grid Computing with MDS Parallel Data Analysis Parallel Data Analysis Data reduction of Subaru Suprime-Cam Data reduction of Subaru Suprime-Cam 250,000 Frames (~5TB) 250,000 Frames (~5TB) Analysis service for JVO users Analysis service for JVO users MDS server is a registry of computing resources MDS server is a registry of computing resources CPU type, Memory size, Disk size, … CPU type, Memory size, Disk size, … Load average, number of submitted job, … Load average, number of submitted job, … Job status listener for asynchronous job Job status listener for asynchronous job resolves the server location for a job execution considering the load balancing. resolves the server location for a job execution considering the load balancing.

Workflow Workflow enables to build a higher level service utilizing the existing services. Workflow enables to build a higher level service utilizing the existing services. Automate a typical procedure which includes data access, data retrieval, data analysis. Automate a typical procedure which includes data access, data retrieval, data analysis. Repeat the same procedure by changing parameters ’ values. Repeat the same procedure by changing parameters ’ values. Same as a shell script to use the data on local machine. Same as a shell script to use the data on local machine.

JVO Workflow Language JVO Workflow Language is defined based on the BPEL4WS. JVO Workflow Language is defined based on the BPEL4WS. Currently only a small part of BPEL4WS is implemented, and some extension is made for realizing our purposes. Currently only a small part of BPEL4WS is implemented, and some extension is made for realizing our purposes. Supported tags: Supported tags:,,,,,,,,,,,,, …,,,,,,,,,,,,, …

Application to Science Use Case Flow of the QSO-Galaxy clustering study Flow of the QSO-Galaxy clustering study 1. Get coordinates of QSOs from a catalog service 2. Get images around the QSOs from a image service 3. Measure the coordinate and color of sources on the image 4. Estimate the photometric redshift (distance) of the sources. 5. Cacluate a cross-correlation between the QSO and garaxies.

… jvoql votableForImage … <invoke identifier="ivo://jvo/tools/sextractor" protocol="" namespace="" url=" operation="performForURL"> listOfURL[_count] catalogDH … listOfCatalogName[_count] listOfObjectName[_count] …

SEQUENCE jvoql = QSOStudy.createJVOQL(qsoTable, ra, dec, radi, imageTable, imageSize, magLimit) votableForImage = executeQuery(jvoql) storeVOTable(votableForImage[0], result_file1) listOfURL = QSOStudy.getListOfURL(votableForImage[0]) listOfFileName = QSOStudy.getListOfFileName(votableForImage[0]) listOfObjectName = QSOStudy.getListOfObjectName(votableForImage[0]) listOfCatalogName = QSOStudy.getListOfCatalogNameForObject(listOfObjectName) FOR listOfURL INVOKE ivo://jvo/tools/sextractor catalogDH = performForURL(listOfURL[_count]) storeVOTable(catalogDH, listOfFileName[_count]) END wdir = getCurrentWorkDir() FOR listOfObjectName numBands = QSOStudy.prepareForHyperZ(wdir, listOfObjectName[_count], listOfFileName, listOfCatalogName[_count]) END PARFOR listOfObjectName executeHyperZ(listOfCatalogName[_count], listOfObjectName[_count]) END

Future Plan Public release at the end of this year Public release at the end of this year Increase the functionality of the workflow system Increase the functionality of the workflow system Data visualization, data analysis Data visualization, data analysis Virtual file system. Virtual file system. Single-sign on Single-sign on Globus 4 ? Globus 4 ?