1. 2018/11/27
An Application for the Data Grid : A Prototype of the Japanese Virtual Observatory System
Masatoshi Ohishi
National Astronomical Observatory of Japan
　　　　　　and Sokendai
2018/11/27
PRAGMA in Fukuoka

2. Collaborators NAOJ Y. Mizumoto N. Yasuda Y. Shirasaki M. Tanaka
Ochanomizu U.
Y. Masunaga
Fujitsu Ltd.
K. Monzen
K. Kawarai
Y. Ishihara
H. Yanaka
Y. Yamaguchi
M. Ishida
N. Saito
2018/11/27
PRAGMA in Fukuoka

3. Supported by Research Grant from the MEXT （文部科学省 科研費 特定領域研究 「情報学」
　（文部科学省　科研費
　　特定領域研究
　　「情報学」
　　松岡・下條組）
Research Grant from the Sokendai
（総合研究大学院大学　企画研究）
2018/11/27
PRAGMA in Fukuoka

4. SUBARU telescope in Hawaii (optical & IR)
2018/11/27
PRAGMA in Fukuoka

5. Orion nebula
In the IR
2018/11/27
PRAGMA in Fukuoka

6. More than
5,000 Galaxies
2018/11/27
PRAGMA in Fukuoka

7. 45m Radio Telescope at Nobeyama
2018/11/27
PRAGMA in Fukuoka

8. ALMA in Chile (sub-mm wave)
2018/11/27
PRAGMA in Fukuoka

9. Time and Spectral Dimensions The Multiwavelength Crab Nebulae
Crab star
1053 AD
X-ray,
optical,
infrared, and
radio
views of the nearby Crab Nebula, which is now in a state of chaotic expansion after a supernova explosion first sighted in 1054 A.D. by Chinese Astronomers.
2018/11/27
PRAGMA in Fukuoka
Slide courtesy of Robert CalTech.

10. Unknowns of Astronomy Physics just after the Big Bang ?
How Galaxies were formed ?
Averaged density of the Universe
etc.
→　statistical analyses for homogeneous survey data are crucial
2018/11/27
PRAGMA in Fukuoka

11. Data Productivity Nobeyama Radio ：~1TB/yr Subaru@Hawaii ：~20TB/yr
ALMA（planned）　 ：~PB/yr
Flood of excellent data (survey data)
Digitized & Archived
2018/11/27
PRAGMA in Fukuoka

12. HPC + High Speed NETWORK
+GRID Technology
→ it is possible to connect and seamlessly use remote DBs and computing facilities
→ Virtual Observatory
2018/11/27
PRAGMA in Fukuoka

13. Telescope is
behind me !
2018/11/27
PRAGMA in Fukuoka

14. 2018/11/27 PRAGMA in Fukuoka 実望遠鏡での 観測 仮想望遠鏡での アイデア 観 測 検 索 データ 解 析
研究発表
実望遠鏡での 観測
仮想望遠鏡での
2018/11/27
PRAGMA in Fukuoka

15. JVO consists of Seamless Access to Federated DBs
Various Data Analysis tools
Advanced Analysis such as Data Mining
Federation with Real Telescopes
→　crucial to control data quality in VO
2018/11/27
PRAGMA in Fukuoka

16. JVO prototype : Ver. 1
Establish seamless access to federated DBs by using the Grid technology
Define JVO Query Language, and implement its parser
Test environment is in a closed subnet
2018/11/27
PRAGMA in Fukuoka

17. Parse the JVO QL into each query sequence
Query request
Web
Broser
JVO Query Language
利用者独自
サービス
Catalog Query Service
ASTRONOMER
JVO
Browser
Viewer
Commands callable from GT2
JVO
Portal
Parse the JVO QL into each query sequence
Catalog DB
Data Archive Service
JVO
Contoroller
skycat
Globus
Toolokit V2
Data Management DB
Data
GridFTP each query
UDDI Server
（FreeSoft)
Resolve servers to request queries
　　Globus Toolkit
Service
Registry
Data Analysis Services
Security Service
Resource Management
Data Management
2018/11/27
PRAGMA in Fukuoka

18. JVO Query Language(1) Catalog Query
2018/11/27
PRAGMA in Fukuoka

19. JVO Query Language(2) Image Query
2018/11/27
PRAGMA in Fukuoka

20. UDDI Registory / Servlet Engine
JVO Prototype System Architecture
Redhat Linux
2Mass
DataBase
Redhat Linux
JVO Service
JVO
Controller
Solaris8 (Possible at Linux)
Gb Ether
GSDL
UDDI Registory / Servlet Engine
UDDI Maintenance Tool
Java2SE 1.4
Globus Toolkit (Server)
Java2SE 1.4
Hypertext
contents
Apache Tomcat
Maintenance Tool
JVO Service (X match)
soapuddi
UDDI4J
Netscape Communicator
Java2 Plugin
JVO Client
(applet)
Apache
ＳＯＡＰ
① Applet
DownLoad
JDBC
register
update
delete
JVO Service
(select)
SQL
library
③ Find Service
② Request
Observation
JVO Controller
DB Service
Image
Data
(copy)
DB Service
Java2SE 1.4
PostgreSQL (DBMS)
PostgreSQL
JVO Server
Monitoring
UDDI4J
UDDI
Data
SUBARU
SupCam
z-band
Image
Data
2Mass DB
get & parse GSDL
pre condition check
execute Service
post condition check
Redhat Linux
SUBARU
DataBase
⑦ issue SQL
JVO Service
⑤ execute Service
③ get GSDL
GSDL
(copy)
④’ Get GSDL
by GridFTP
GSDL
⑤’ execute Service
Check status
Globus Toolkit (Client)
Internet
Globus Toolkit (Server)
⑥’ spawn Service
④ Get GSDL
by GridFTP
JVO Service (image)
AVO
JVO Service
JVO Service (X match)
SUBARU
DataBase
Globus Toolkit
Globus Toolkit (Server)
GridFTP
GSDL
JVO Service
(select)
SQL
library
NVO
⑥ spawn Service
Globus Toolkit
JVO Service
(count)
SQL
library
JVO Service
(select)
SQL
library
JVO Service
(image)
JVO Service
(X match)
JVO Service
(count)
SQL
library
⑦’ issue SQL
Image
Data
Image
Data
(copy)
Copy imate data
by GridFTP
Image
Data
(copy)
DB Service
ORACLE
2018/11/27
PRAGMA in Fukuoka
The following free software is included by globus toolkit.
OpenSSL (Secure Socket Library)
OpenLDAP (LDAP Server)
wu-ftpd (ftp server)
Copy imate data
by GridFTP
SUBARU
SupCam
i-band
Free Software
In House Software
Image
Data
Commercial Software
Function

21. How it works ? Read JVO QL Create “Work Flow”, and execute each step
When necessary, access to the registry to expand the step of the work flow
Thus the steps increases dynamically
2018/11/27
PRAGMA in Fukuoka

22. 2018/11/27
PRAGMA in Fukuoka

23. 2018/11/27
PRAGMA in Fukuoka

24. 2018/11/27
PRAGMA in Fukuoka

25. create materialized view my_table as
select i.POS_EQ_RA_MAIN,
i.POS_EQ_DEC_MAIN,
i.PHOT_SDSS_I,
z.POS_EQ_RA_MAIN,
z.POS_EQ_DEC_MAIN,
z.PHOT_SDSS_Z,
img_i.BOX(POINT(ra1, dec1), 0.005, 0.005) as img_i,
img_z.BOX(POINT(ra2, dec2), 0.005, 0.005) as img_z
from Subaru.SPcam.i i,
Subaru.SPcam.z z,
Subaru.SPcam.i img_i,
Subaru.SPcam.z img_z
where XMATCH(i, z) < 10 arcsec NEAREST
and BOX(POINT(201., 27.4), 0.10, 0.10)
and i.PHOT_SDSS_I - z.PHOT_SDSS_Z > 0.0
and i.PHOT_SDSS_I < 20.0
and z.PHOT_SDSS_Z < 20.0
2018/11/27
PRAGMA in Fukuoka

26. 2018/11/27
PRAGMA in Fukuoka

27. 2018/11/27
PRAGMA in Fukuoka

28. 2018/11/27
PRAGMA in Fukuoka

29. 2018/11/27
PRAGMA in Fukuoka

30. VOTable as output
XML
Standard data exchange format among VOs defined by the IVOA (see below)
2018/11/27
PRAGMA in Fukuoka

31. 2018/11/27
PRAGMA in Fukuoka

32. 2018/11/27
PRAGMA in Fukuoka

33. FAQs on our Prototype
Q) Contents of environment variables are not passed to the executing machines.
A) Prepare a shell script to get environment variables and another script to execute the command. Then GridFTP these shell scripts , and execute the shells in the remote machine.
Q) It is impossible to prepare all users’ account to all relevant machines.
A) Map all accounts on a portal to a single representative one, such as globus.
2018/11/27
PRAGMA in Fukuoka

34. FAQs
Q) GT2 has too large overhead before starting actual jobs !! They are several seconds.
A) For batch jobs, such as database accesses, the overhead could be negligible. However users can never wait for several second for interactive data analyses.
Request) Overheads must be < 0.1 seconds
excluding the network delays.
2018/11/27
PRAGMA in Fukuoka

35. Future Prototype version 2 implement data analysis tools
　　better user interface
Interoperation with VOs in the world
We have joined the International Virtual Observatory Alliance, and are contributing standardizing protocols for interoperability.
2018/11/27
PRAGMA in Fukuoka

36. VOs in the World NVO (USA), AVO (EU),AstroGrid (UK)
Germany, Canada, Russia, Australia
India, China, JAPAN
Protocols to interconnect VOs
→　International Virtual Observatory Alliance (IVOA)
2018/11/27
PRAGMA in Fukuoka

37.
2018/11/27
PRAGMA in Fukuoka

38. 2018/11/27 PRAGMA in Fukuoka MVC GRID環境 A GRID環境 C GRID環境 B MVC MVC
天文台殿
MVC
サービス
天体カタログ
天体カタログDB　
MVC
天体カタログ
Web
サーバ
JVO
ポータル
VO用
ブラウザ
セキュリティ管理
資源管理
データ管理
仮想観測 指示
仮想観測実行
サービス
研究者
VO用
Viewer
ワーク
フロー
天文台殿
サービス
利用者独自
サービス
DB/DA
DA/DB　 DB
Data
セキュリティ管理
資源管理
データ管理
サービス
レジストリA
セキュリティ管理
資源管理
データ管理
GRID環境 A
サービス
レジストリC
サービス
レジストリB
サービス
他システム
pipeline 処理
データ管理
資源管理
セキュリティ管理
Web
サーバ VO
ポータル
サービス
他システム 処理
データ管理
資源管理
セキュリティ管理
VO用
ブラウザ
サービス
他システム 処理
データ管理
資源管理
セキュリティ管理
サービス
セキュリティ管理
研究者
VO用
Viewer
資源管理
他システム
データ管理 処理
2018/11/27
PRAGMA in Fukuoka
GRID環境 C
GRID環境 B

39. JVO Home Page
2018/11/27
PRAGMA in Fukuoka