1. Page 1 LAITS Laboratory for Advanced Information Technology and Standards Integration of Grid and OGC Technologies for Earth Science Modeling and Applications Laboratory for Advanced Information Technology and Standards George Mason University June 28, 2005 GGF14 Chicago, IL Liping Di ldi@gmu.eduldi@gmu.edu Aijun Chen achen6@gmu.eduachen6@gmu.edu

2. Page 2 LAITS Laboratory for Advanced Information Technology and Standards Contents Introduction Why Grid is useful to Earth Observation Community OGC Technology for Interoperability Objective Areas of work Current Status Future work

3. Page 3 LAITS Laboratory for Advanced Information Technology and Standards Introduction Geospatial data is the major type of data that human beings has collected. –more than 80% of the data are geospatial data. Image/gridded data is dominant form of geospatial data in terms of volume. –Most of those data are collected by the EO community. Geospatial data will grow to ~exabytes very soon. –NASA EOSDIS has more than one petabyte of data in archives; more than 2 terabytes per day of new data are added. –Application data centers: 10’s of terabytes of imagery –Tens of thousands of datasets on-line now. How to effectively, wisely, and easily use the geospatial data is the key information technology issue that we have to solve.

4. Page 4 LAITS Laboratory for Advanced Information Technology and Standards The Grid Technology The Grid technology is developed for securely sharing computational resources within an virtual organization. –Computer CPU cycles –Storage –Networks –Data, Information, algorithms, software, services. It was originally motivated and supported from sciences and engineering requiring high-end computing, for sharing geographically distributed high-end computing resources. The core of the technology is the the open source middleware called Globus Toolkit. –The latest version of Globus is version 4.0 which implements the Open Grid Service Architecture (OGSA) and converged with Web services technology.

5. Page 5 LAITS Laboratory for Advanced Information Technology and Standards Why Grid Is Useful to the EO community? Earth observation community is one of the key communities for collecting, managing, processing, archiving and distributing geospatial data and information. Because of the large volumes of EO data and geographically scattered receiving and processing facilities, the EO data and associated computational resources are naturally distributed. The multi-disciplinary nature of global change research and applications requires the integrated analysis of huge volume of multi-source data from multiple data centers. This requires sharing of both data and computing powers among data centers. Therefore, Grid is an ideal technology for EO community.

6. Page 6 LAITS Laboratory for Advanced Information Technology and Standards Why Needs the Geospatial Extensions of Grid Geospatial data and information are significantly different from those in other disciplines. –Very complex and diverse. Formats, projection, resolutions. Hyper-dimensions: spatial, temporal, spectral, thematic. Raster vs. vectors –Large data volume more than 80% of data human beings has collected is spatial data. The geospatial community has developed a set of standards specifically for geospatial data and information that users have been familiar with. (e.g., OGC, ISO, FGDC). Grid technology is developed for general sharing of computational resources and not aware of the specialty of geospatial data. In order to make Grid technology applicable to geospatial data, we have to do the geospatial domain-specific extensions.

7. Page 7 LAITS Laboratory for Advanced Information Technology and Standards The OGC Web Service Specifications The Web Coverage Services (WCS) specification: defines the standard interfaces between web-based clients and servers for accessing coverage data. –All imagery type of remote sensing data is coverage data. The Web Feature Services (WFS) specification: defines the standard interfaces between web-based clients and servers for accessing feature-based geospatial data. –vector and point data are feature data. The Web Map Services (WMS) specification: define the standard interfaces for accessing and assembling maps from multiple servers. –visualization of geospatial data The Catalog Services for Web (CSW) specification: defines the interfaces between web-based clients and servers for finding the required data or services from registries. WCS, WFS, CSW, and WMS form the foundation for the interoperable geospatial data access and service environment

8. Page 8 LAITS Laboratory for Advanced Information Technology and Standards Objectives of GMU Grid Project Making NASA EOSDIS data easily accessible to Earth science modeling and applications communities by combining the advantages of both OGC and Grid technology –Develop the geospatial extensions of Grid technology to make it geospatially enabled (Geospatial Grid). –Enable OGC geospatial clients access Grid-managed distributed geospatial resources. –Provide virtual/intelligent geospatial products in the Grid environment. –Test methods for automating the process from geospatial data to knowledge in the Grid environment. Demonstrate the geospatial Grid technology in realistic NASA EOS data environment. Contribute technology, software, and the data pool application to the CEOS Grid testbed

9. Page 9 LAITS Laboratory for Advanced Information Technology and Standards Areas of Extensions Internally in the Grid, it have to be spatially aware –Extend Globus toolkit to handle the spatial, spectral, temporal, thematic based spatial data and information management. –Develop enough Grid-enable tools for geospatial data handling/serving. Must provide data/information access and services interfaces that are standard in the geospatial community. –The Open GIS Consortium’s Web Data Access/Service interfaces (e.g., OGC WCS, WMS, WFS, and CSW).

10. Page 10 LAITS Laboratory for Advanced Information Technology and Standards Virtual Geospatial Datasets A virtual dataset is a dataset that: –not exist in a data and information system –The system knows how to create it on-demand. –A virtual dataset, once created, can be kept for fulfilling the same request from next users. The client/data user will not know the difference between a real dataset and a virtual dataset. A virtual dataset can be produced (materialized) by –running a program dedicated to the production of the virtual dataset (dedicated program approach). –running a series of service modules, each one takes care of a small step of the materialization of the virtual dataset (service approach).

11. Page 11 LAITS Laboratory for Advanced Information Technology and Standards The Service Approach to Virtual Datasets A service is defined as self-contained, self-describing, modular applications that can be published, located, and dynamically invoked across a network. –It performs functions, which can be anything from simple requests to complicated business processes. –Once a service is deployed, other applications (and other services) can discover and invoke the deployed service. A service can be implemented in the Web environment, called a web service, or in the Grid environment, called a Grid service. Standards on service discovery, declaration, binding, and invocation allow dynamically chaining individual services across a network together to fulfill a complex task. A virtual dataset, in the service environment, basically is a service chain that describes steps to be taken to produce the virtual dataset. With enough elementary service models, it is possible to provide unlimited numbers of virtual datasets by just creating the service chains.

12. Page 12 LAITS Laboratory for Advanced Information Technology and Standards Geo-object, Geo-tree, Virtual Dataset, Geospatial Models archived geo-object user geo-object Intermediate geo-object Automated data transformation service(WCS/WFS) no servicedata service modeling and virtual data services User Requested User Obtained Geospatial web/Grid services

13. Page 13 LAITS Laboratory for Advanced Information Technology and Standards User Creation of Geospatial Models A user-requested products maybe not exist both virtually and no virtually. If the user knows the thought process to create the data products from lower-level inputs step-by-step (the logical geospatial modeling) –With help of a good user interface and the availability of service modules and models/submodels, the user can construct a geospatial model/virtual data product interactively. –The system then can produce the virtual data product for the user. –The user-created model can be incorporated into the system as a part of the virtual datasets the system can provide. This allows the system to grow capabilities with time. Advantages –allows users to obtain the ready-to-use scientific information instead of the raw data, significantly reducing the data traffic between the users and the geospatial Grid. –allows users to explore huge resources available at a data Grid and to conduct tasks that they never be able to conduct before.

14. Page 14 LAITS Laboratory for Advanced Information Technology and Standards Current Status We have fulfilled all objectives of the project except for the users-defined customizable virtual geospatial products. A realistic testbed that simulates NASA EOS data environment has been created. Grid-enabled OGC web services software have been developed.

15. Page 15 LAITS Laboratory for Advanced Information Technology and Standards Grid Security (GSI) and VO Setup GMU (Solaris) (laits.gmu.edu) GT 3.2 with CEOS Certs. GMU (Mac) (geobrain.laits.gmu.edu) Globus 4.0 with Laits Certs. LAITS CA center Ames ipg05 (Linux) (ipg05.ipg.nasa.gov) Globus 3.2 with IPG Certs. GMU LAITS VO NASA IPG VO GMU (Linux) (data.laits.gmu.edu) Globus 4.0 with Laits Certs. IPG CA center NASA SGT (Linux) (arao2.sgt-inc.com) Globus 3.2 with CEOS Certs. NASA (Linux) (former.intl-interfaces.net) Globus 3.0 with CEOS Certs. CEOS VO Authentication among different VO LLNL esg2 (Linux) (esg2.llnl.gov) Globus 3.2 with ESG Certs. LLNL ESG VO ESG CA center GMU (Linux) (salmon.laits.gmu.edu) Globus 4.0 with Laits Certs.

16. Page 16 LAITS Laboratory for Advanced Information Technology and Standards The Testbed - Hardware The testbed has been created with 7 machines in 3 organizations. The flagship machine in the testbed is GMU’s Apple cluster server: – 6 Apple G5 server nodes- 3 with dual 2.5GHz CPU and 3 with dual 2.0 GHz CPU with total of 12 GB RAM. – 22.6TB RAID storage. – 1GB network to Internet II and 100 MB to Internet I. – Hosted at ESDIS network lab of GSFC.

17. Page 17 LAITS Laboratory for Advanced Information Technology and Standards The Testbed - Data Populated the G5 server with –Landsat data covering Globe for year 1975, 1990 and 2000. Currently total 7TB of data has been ingested. –Shuttle DEM data covering Globe for year 2000. The size of DEM is about 1TB –Other sample EOSDIS data (e.g., MODIS, Aster, etc). Converted part of DOE LLNL netCDF modeling data to HDF-EOS data and loaded into the LLNL node of the testbed. Replicated some typical EOSDIS data at NASA Ames node. The total size of data in the testbed is around 9 TB now.

18. Page 18 LAITS Laboratory for Advanced Information Technology and Standards The Testbed - Software Globus 3.2.1 were installed at all nodes. The geospatial Grid software developed by GMU has been installed at all nodes. Setup and issue CAs – Set up LAITS CA, issued LAITS certificates to Mac machine and Linux machine of GMU LAITS. – Set up IPG CA, issued IPG certificates to Linux machine at NASA Ames. – From CEOS CA, requested CEOS certificates for Solaris machine at GMU LAITS. – Tested and debugged the authentication between any two different CAs’ certificates among all of the above boxes.

19. Page 19 LAITS Laboratory for Advanced Information Technology and Standards Improvement of Catalog Information Model (IM) ISO 19115 Part one ISO 19115 Part two (FGDC extension) NASA ECS ISO 19119 ebRIM IM HDF-EOS metadata IM (getCapabilities and describeCoverage) GCMD Service Type IM LAITS-defined Data Type IM CSW Information Model

20. Page 20 LAITS Laboratory for Advanced Information Technology and Standards Software Development - GCSW LAITS has developed an OGC Catalog Service for Web (CSW) server. A wrapper has developed so that the catalog server can also work in the Grid environment as a service. – The Grid-enabled CSW Deployed GCSW at GeoBrain (Mac), LAITS (Solaris) and Data (Linux).

21. Page 21 LAITS Laboratory for Advanced Information Technology and Standards Software Development - GWCS/Portal OGC Web Coverage Service (WCS) specification is the fundamental specification for geospatial coverage data access WCS has been enhanced to process 4-D HDF-EOS data which is from LLNL netCDF modeling data. Enhanced the Grid enabled WCS (GWCS). Developed OGC standard compatible WCS portal which is totally supported by Grid Services to access to GWCS. Deployed and working on: Geobrain (Mac), Laits (Solaris) and Data (Linux).

22. Page 22 LAITS Laboratory for Advanced Information Technology and Standards OGC Web Map Service (WMS) specification is the fundamental specification for geospatial map data access. GMU has developed WMS server through other related project. A wrapper on top of WMS server to make WMS server a complete Grid service. – It is called Grid-enabled Web Map Service (GWMS). – It works both as a web service and a Grid service. – Both OGC and Grid clients can work with the service. Developed OGC standard compatible WMS portal which is totally supported by Grid Services to access to GWCS. Deployed WMS at Geobrain (Mac), Laits (Solaris) and Data (Linux). Software Development - GWMS/Portal

23. Page 23 LAITS Laboratory for Advanced Information Technology and Standards (iGSM: intelligent Grid Service Mediator) Support WCS portal and WMS portal to distribute their request to proper GWCS and GWMS. Software Development - iGSM WCS Portal WMS Portal GCSW GWCS GWMS iGSM ROS MDS DTS

24. Page 24 LAITS Laboratory for Advanced Information Technology and Standards Functional Overview of iGSM Managing geodata access requests from OGC WCS portal and WMS portal and transfer those requests to GWCS (Grid-enabled Web Coverage Service) or GWMS (Grid-enabled Web Map Service ) –Accepts geodata requests from default WCS portal and WMS portal. –Queries a ROS (Replica Optimization Service) for an optimized PFNInfo (Physical File Name Information) object Each PFNInfo contains a physical file name, a GridWCS service ID, and the host where the data file located –When the received PFNInfo contains a valid service ID Requests a GridCSW (Grid-enabled Catalog Service for Web) for corresponding GridWCS/WMS URL to the service ID. –When the received PFNInfo contains a null service ID Requests a GridCSW for available GridWCS(s) /WMS(s) among the Grid resources. Requests a ROS (Replica and Optimized Service) for selecting the best GridWCS/WMS among the resources returned from the GridCSW Requests a DTS (Data Transfer Service) for transferring the data to the selected system –Querying the GridCSW deployed in the selected system for the geodata URI

25. Page 25 LAITS Laboratory for Advanced Information Technology and Standards (ROS: Replica Optimization Service) Globus RLS as Grid Service Software Integration - ROS Globus Index service Globus MDS scripts modification LRC (Laits) LRC (Laits-data) LRC (Ames/LLNL) RLI (Laits) RLI (Laits-data) ROS MDS Index Service (MDS)

26. Page 26 LAITS Laboratory for Advanced Information Technology and Standards (DTS: Data Transfer Service) GridFTP as Grid Service Software Integration - DTS Machine A Globus Security Machine B Globus Security Machine B Globus Security Machine C Globus Security Machine A Globus Security Data Secure Request Data Secure Request

27. Page 27 LAITS Laboratory for Advanced Information Technology and Standards Geospatial Grid with GCSW/GWCS/GWMS/iGSM/ROS/DTS WCS Portal WMS Portal GCSW GWMS GWCS iGSM ROS MDS DTS CSW Portal User/Client Interface (Web Download & MPGC) Geospatial Catalog DB Replica DB HDF-EOS Data 1 2 2 Laits (3) Ames LLNL

28. Page 28 LAITS Laboratory for Advanced Information Technology and Standards A Data Request Scenario based on the Integration RLS LAITS WCS/WMS Portal CSW Portal ESG Catalog Client 1 2 3 Retrieval Manager 4 LAITS GridWCS Ames GridWCS LLNL GridWCS + default WCS portal IP ROS 5 2 Logical data name Physical data/service ID MDS 6 Best server ID 7 iGSM Other WCS LAITS GridCSW HDF-EOS Data Other Data Ames DTS 9 9 9 8

29. Page 29 LAITS Laboratory for Advanced Information Technology and Standards Major Work Items for the Next Year The next stage of this project be concentrated on the on-demand virtual product generation based on geospatial processing models. –Determine the requirements of metadata structures for a materialized product catalogue; Develop the metadata for few examples of ES data products and use these to adapt or implement a prototype materialized product catalogue. –Directly access to data pools through ECHO, gateway to ESG; Design a representation of the transformation prescription and use it to develop a prototype virtual data product catalogue –Investigate the emerging workflow language BPEL4WS for representing the execution of the transformation prescriptions. Build the planner that converts from the prescriptions into a workflow language. –Make LAITS BPELPower workflow engine Grid enabled and integrate it into NWGISS. –Enable OGC CSW to search and OGC WCS to retrieve virtual data the same way as non-virtual data; Demonstrate an overall virtual data scenario.

30. Page 30 LAITS Laboratory for Advanced Information Technology and Standards Acknowledgement The project team includes Prof. Liping Di (PI, GMU), Dr. Piyush Mehrotra (Co-I, NASA Ames), Dr. Dean Williams (Co-I, DOE LLNL), Dr. Chaumin Hu (NASA Ames), Dr. Aijun Chen (implementation leader, GMU), Dr. Yuqi Bai (GMU), Mr. Yang Liu (GMU), Yaxing Wei (GMU). The project is funded by NASA Advanced Information System Technology program (AIST).

31. Page 31 LAITS Laboratory for Advanced Information Technology and Standards Acknowledgement Thank You for your attention ! Any Questions ?