Presentation is loading. Please wait.

Presentation is loading. Please wait.

Craig Lee OGF-24 Mark Reichardt Grid-Enabled Geospatial Systems.

Similar presentations


Presentation on theme: "Craig Lee OGF-24 Mark Reichardt Grid-Enabled Geospatial Systems."— Presentation transcript:

1 Craig Lee OGF-24 Mark Reichardt Grid-Enabled Geospatial Systems

2 2 Geospatial Data: Immense Applicability Image: E. Gennai, C. Terbough, ESRI Estimates vary, but ~80-90% of all data collected or produced by the human race is geospatially referenced Natural exploration, e.g., oil & gas Public administration Civic planning and engineering Weather and aviation Environmental planning Disaster management Satellite ground systems

3 3 An Example: GEO Grid Global Earth Observation Grid S. Sekiguchi, AIST

4 4 The OGC-OGF Collaboration Promote best practices and international standardization for distributed geospatial data processing capabilities that is: Transparent Transparent -- users is not aware of the infrastructure Interoperable Interoperable -- the resources work together Scalable Scalable -- small local, to massive distributed platforms Promote best practices and international standardization for distributed geospatial data processing capabilities that is: Transparent Transparent -- users is not aware of the infrastructure Interoperable Interoperable -- the resources work together Scalable Scalable -- small local, to massive distributed platforms Directly Relevent to Satellite Ground Systems

5 5 Two Issues What geospatial and computing technologies must be integrated? How do we manage the process to make progress on doing so? What geospatial and computing technologies must be integrated? How do we manage the process to make progress on doing so?

6 6 Some Key OGC Standards Web Map Service (WMS) Display of registered and superimposed map-like views of information that come simultaneously from multiple remote and heterogeneous sources. Web Feature Service (WFS) Retrieval and update of digital representations of real-world objects referenced to the Earth s surface. Web Coverage Service (WCS) Access to a grid coverage, usually encoded in a binary format, and offered by a server. Catalogue Service for the Web (CSW )Catalogue Service for the Web (CSW Common interfaces to publish, discover, browse, and query metadata about data, services, and other resources Web Map Service (WMS) Display of registered and superimposed map-like views of information that come simultaneously from multiple remote and heterogeneous sources. Web Feature Service (WFS) Retrieval and update of digital representations of real-world objects referenced to the Earth s surface. Web Coverage Service (WCS) Access to a grid coverage, usually encoded in a binary format, and offered by a server. Catalogue Service for the Web (CSW )Catalogue Service for the Web (CSW Common interfaces to publish, discover, browse, and query metadata about data, services, and other resources

7 7 Key OGC Standards Sensor Observation Service (SOS) Access observations for a sensor or sensor constellation, whether in-situ or dynamic (e.g, water monitoring or satellite imaging). Optionally access associated sensor and platform data Sensor Planning Service (SPS) An interface to task sensors or models. Using SPS, sensors can be reprogrammed or reconfigured, sensor missions can be started or changed, simulation models executed and controlled. Sensor Alert Service (SAS) Defines an interface for publishing and subscribing to alerts from sensors. If an event occurs the SAS will notify all clients subscribed to this event type. Observations and Measurements Encoding Standard (O&M) Defines an abstract model and an XML schema encoding for observations and it provides support for common sampling strategies. It also provides a general framework for systems that deal in technical measurements in science and engineering. Sensor Observation Service (SOS) Access observations for a sensor or sensor constellation, whether in-situ or dynamic (e.g, water monitoring or satellite imaging). Optionally access associated sensor and platform data Sensor Planning Service (SPS) An interface to task sensors or models. Using SPS, sensors can be reprogrammed or reconfigured, sensor missions can be started or changed, simulation models executed and controlled. Sensor Alert Service (SAS) Defines an interface for publishing and subscribing to alerts from sensors. If an event occurs the SAS will notify all clients subscribed to this event type. Observations and Measurements Encoding Standard (O&M) Defines an abstract model and an XML schema encoding for observations and it provides support for common sampling strategies. It also provides a general framework for systems that deal in technical measurements in science and engineering.

8 8 Key OGC Standards Transducer Markup Language (TML) TML defines the conceptual model and XML Schema for describing transducers and supporting real-time streaming of data to and from sensor systems Web Processing Service (WPS) WPS defines a standardized interface to facilitate the publishing of geospatial processes, and the discovery of an binding to those processes by clients Transducer Markup Language (TML) TML defines the conceptual model and XML Schema for describing transducers and supporting real-time streaming of data to and from sensor systems Web Processing Service (WPS) WPS defines a standardized interface to facilitate the publishing of geospatial processes, and the discovery of an binding to those processes by clients

9 9 OGC Sensor Web Enablement Framework (SWE) SPS SOS WNS SAS CSW Mission Control System Registered sensor and observation metadata Metadata for a sensor and observations, and a URL Metadata for a sensor observations, and a URL ……… ! SWE Architecture Encodings: –Observations & Measurements –SensorML –Transducer Markup Language (TML) IEE1451

10 10 OGC Web Services Combined

11 11 Key OGF Standards HPC-Basic Profile Cluster scheduling interoperability layer supported by Microsoft, Altair Computing, Platform Computing, and others Simple API for Grid Applications (SAGA) Complete distributed programming environment with C++ and Java bindings GridRPC Grid-enabled Remote Procedure Call Data Access and Integration specs (WS-DAI-*) Set of specs for accessing remote files, databases, XML documents Web Services Resource Framework (WSRF) Allow remote data and services to be independently managed HPC-Basic Profile Cluster scheduling interoperability layer supported by Microsoft, Altair Computing, Platform Computing, and others Simple API for Grid Applications (SAGA) Complete distributed programming environment with C++ and Java bindings GridRPC Grid-enabled Remote Procedure Call Data Access and Integration specs (WS-DAI-*) Set of specs for accessing remote files, databases, XML documents Web Services Resource Framework (WSRF) Allow remote data and services to be independently managed

12 12 An Example of a Global Grid: EGEE Bob Jones, EGEE Project Director

13 13 Coalescing Service Categories to Specification Areas Infrastructure Service Categories Catalogues & Registries Discovery Applications Job Submission/Execution Svcs Workflow Management Data Product Management Storage & Archiving Messaging Event Notification User Management Information Assurance/Security Monitor & Control Enterprise Svc Mgmt/Software Mgmt APIs/Ops Systems Financial Accounting Infrastructure Service Categories Catalogues & Registries Discovery Applications Job Submission/Execution Svcs Workflow Management Data Product Management Storage & Archiving Messaging Event Notification User Management Information Assurance/Security Monitor & Control Enterprise Svc Mgmt/Software Mgmt APIs/Ops Systems Financial Accounting Specification Areas Catalogues & Registries Discovery Applications Execution Services Workflow/Transactions Data Management State Management Messaging, Routing, Addressing Event Notification Metadata Schemas & Ontologies Portals and User Interfaces Security Policy & Agreement System Architecture System Management

14 14 Populating the Specification Areas Applications (There are essentially an unbounded number of applications that could run as part of the application domain. OGC WMS, WFS & WCS are listed here since they would be part of what an end-user sees and interacts with.) Job Submission/Execution Svcs WSDL, SOAP, (WS-I Basic Profile), CORBA, JSDL, OGSA- BES (OGF HPC Basic Profile), OGC Web Processing Service Discovery UDDI, (WS-I Basic Profile), DDMS v1.3 (DoD Discovery Metadata Specification), WS-Discovery Catalogues & Registries LDAP, OpenGIS Cat 2.0, OGC CSW, ebRIM, ebXML RS, EO Products Ext., NATO DFDD (DGIWB Data Feature Data Dictionary), NAS v1.8 (NSG Application Schema), NEC V1.8 (NSG Entity Catalog) Metadata Schemas & Ontologies XML, CIM/GLUE, FGDC, GML 3.1, ISO (Dublin Core), ISO (metadata), ISO (services), ISO (sensors), OWL, OWL-S, SWRL, (many other ISO docs) Messaging, Routing, Addressing HTTP, WS-Addressing, WS-MessageDelivery, WS- Reliability, WS-ReliableMessaging Event NotificationWS-Notification, WS-Eventing, Sensor Alert Service Updated from Fox, Ho, Pierce – U. IndianaGreen: On DISR Baseline

15 15 Populating the Specification Areas Workflow/TransactionsBPEL, WS-Coordination, (Kepler, Triana, Pegasus, …) Data ManagementFTP, GridFTP, OGSA-DAI, (SRB, iRODS) State Management IETF RFC 2695 (HTTP State Management (cookies)), WS-ResourceFramework, WS-Context, WS-I+, WS-MetadataExchange Portals and User InterfacesWS-RemotePortlets Security SSL/TLS, Kerberos, WS-Security, ID-WSF 2.0 (Liberty Identity Web Services Framework), WS-Trust, SAML, GSI, Shibboleth, VOMS (Virtual Organization Management System), OpenID, OAuth Policy & AgreementWS-Policy, WS-Agreement, (iRODS), XACML, GeoXACML System Architecture OGF Open Grid Services Architecture, OGC Sensor Web Enablement (SWE), Community Sensor Model v2.A (CSM) System Management WS-DistributedManagement, WS-Management, WS-Transfer, OGF CDDLM (Configuration Description, Deployment and Lifecycle Management) Updated from Fox, Ho, Pierce – U. IndianaGreen: On DISR Baseline

16 16 How Do We Manage Progress for Distributed Geospatial Systems? Targeted Projects on Decisive Issues Targeted Projects on Decisive Issues Build Critical Mass of Key Stakeholders Build Critical Mass of Key Stakeholders Continual polling and coordination across the community Continual polling and coordination across the community They must agree on: They must agree on: Clear Goals Clear Goals Clear Schedule (time-box the process) Clear Schedule (time-box the process) Clear Responsibilities Clear Responsibilities Properly Provisioning the Effort Properly Provisioning the Effort Targeted Projects on Decisive Issues Targeted Projects on Decisive Issues Build Critical Mass of Key Stakeholders Build Critical Mass of Key Stakeholders Continual polling and coordination across the community Continual polling and coordination across the community They must agree on: They must agree on: Clear Goals Clear Goals Clear Schedule (time-box the process) Clear Schedule (time-box the process) Clear Responsibilities Clear Responsibilities Properly Provisioning the Effort Properly Provisioning the Effort

17 17 A General Process Model *RFQ/CFP = Request for Quotation/Call for Participation Develop & Test Task D Selection & Kick-off Task C RFQ/CFP*Development Task B ConceptDevelopment Task A Deploy & Persist Task E Clear Schedule, Deliverables and Project Responsibilities

18 18 Return on Investment What are the carrots to build the critical mass of stakeholders? Get early influence in specification development, early skills building, visibility, and opportunity for early market deployment of standards, but just as important… Return on Investment (ROI) Return on Investment (ROI) Investment Time, Money & People Both Monetary and In-Kind (labor & materials) Timely Connection to Concrete Results Stakeholders benefit from collaboration Get more than they put in What are the carrots to build the critical mass of stakeholders? Get early influence in specification development, early skills building, visibility, and opportunity for early market deployment of standards, but just as important… Return on Investment (ROI) Return on Investment (ROI) Investment Time, Money & People Both Monetary and In-Kind (labor & materials) Timely Connection to Concrete Results Stakeholders benefit from collaboration Get more than they put in

19 19 A Case Study: HPC Basic Profile 2006 Key stakeholders decided to demonstrate interoperability between their existing job submission infrastructures November 2006 (SC06) Prototype implementations demonstrated 28-August-2007 HPC Basic Profile, Version 1.0, published November 2007 (SC07) Interoperability demonstrated by Altair, Microsoft, Platform, OMII-UK, OMII-Europe, EGEE, UVa 21-February-2008 Interoperability Experiences with HPCBP, Version 1.0, published Commercial adoption plans by Altair, Microsoft & Platform 2006 Key stakeholders decided to demonstrate interoperability between their existing job submission infrastructures November 2006 (SC06) Prototype implementations demonstrated 28-August-2007 HPC Basic Profile, Version 1.0, published November 2007 (SC07) Interoperability demonstrated by Altair, Microsoft, Platform, OMII-UK, OMII-Europe, EGEE, UVa 21-February-2008 Interoperability Experiences with HPCBP, Version 1.0, published Commercial adoption plans by Altair, Microsoft & Platform

20 20 OGC Web Service Testbed OWS is an annual process where sponsors identify specific demonstration targets Larger number of participants supply in-kind resources OWS-5 recently finished (March 28, 2008) OWS-5 Sponsors BAE Systems - National Security Solutions Federal Geographic Data Committee (FGDC/USGS) Google Lockheed Martin Integrated Systems & Solutions (Lead Org.) Northrop Grumman US National Aeronautic and Space Administration (NASA) US National Geospatial-Intelligence Agency (NGA) ~35 Participating Organizations 3x-4x Return on Investment 3x-4x Return on Investment Strength of Collaboration Strength of Collaboration OWS is an annual process where sponsors identify specific demonstration targets Larger number of participants supply in-kind resources OWS-5 recently finished (March 28, 2008) OWS-5 Sponsors BAE Systems - National Security Solutions Federal Geographic Data Committee (FGDC/USGS) Google Lockheed Martin Integrated Systems & Solutions (Lead Org.) Northrop Grumman US National Aeronautic and Space Administration (NASA) US National Geospatial-Intelligence Agency (NGA) ~35 Participating Organizations 3x-4x Return on Investment 3x-4x Return on Investment Strength of Collaboration Strength of Collaboration

21 21 OWS-5 Five Threads Sensor Web Enablement (SWE) Geo Processing Workflow (GPW) Information Communities and Semantics (ICS) Agile Geography Compliance Testing (CITE) Outbrief Videos NASA Sensor Web 2.0 Experiments (7:43) Data Architecture Views (10:37) Conflation Processing (5:07) Web Coverage Processing Service (5:58) Five Threads Sensor Web Enablement (SWE) Geo Processing Workflow (GPW) Information Communities and Semantics (ICS) Agile Geography Compliance Testing (CITE) Outbrief Videos NASA Sensor Web 2.0 Experiments (7:43) Data Architecture Views (10:37) Conflation Processing (5:07) Web Coverage Processing Service (5:58)

22 22 Still from OWS-5 NASA Sensor Web Video Satellite EO-1 tasked to collect imagery on Northern San Diego County Wildfires that was integrated with UAV track data (red lines) in Google Earth. Demo drove issues around sensor networks, data interoperability, and command & control.

23 23 OWS-6: RFQ/CFP Topics Sensor Web Enablement (SWE) Georeferenced sensors, event notification, security Aviation Information Aeronautical Information Exchange Model (AIXM) for next generation flight control system (FAA and EuroControl) GeoProcessing Workflows (GPW) Web Processing Service (WPS), Workflow and Grids (WPS to Grid) Distributed resource management, e.g., data, services, hosts, networks, for geospatial applications Decision Support Services 3D Visualization with Fly Through Open Location Services Integrated Client for multiple OWS services Compliance, Interoperability & Test Evaluation (CITE) Suite of tests and reference implementations for all approved standards Sensor Web Enablement (SWE) Georeferenced sensors, event notification, security Aviation Information Aeronautical Information Exchange Model (AIXM) for next generation flight control system (FAA and EuroControl) GeoProcessing Workflows (GPW) Web Processing Service (WPS), Workflow and Grids (WPS to Grid) Distributed resource management, e.g., data, services, hosts, networks, for geospatial applications Decision Support Services 3D Visualization with Fly Through Open Location Services Integrated Client for multiple OWS services Compliance, Interoperability & Test Evaluation (CITE) Suite of tests and reference implementations for all approved standards

24 24 Possible Geospatial/Grid Topics Integration, interoperability of geospatial and computing resource Information models (CIM/GLUE) Distributed and federated catalogs (ebRIM, MDS, etc.) Discovery services Implementation of WPS on various grid tools SAGA, GridRPC, HPC-Basic Profile Integration of WS-Eventing, WS-Notification, INFOD Sensor Web Enablement Integration of configuration and lifecycle management tools Configuration Description, Deployment and Lifecycle Management (CDDLM) Integration of WPS with workflow management tools BPEL, Kepler, Taverna, Triana, Pegasus, etc. Workflow design tools, execution engines, planning (data virtualization), data provenance Integration of WPS with grid security models SAML, XACML, VOMS, GSI Support for virtual organizations Integration of geospatial data with rule-based, data management policy engines iRODS Integration, interoperability of geospatial and computing resource Information models (CIM/GLUE) Distributed and federated catalogs (ebRIM, MDS, etc.) Discovery services Implementation of WPS on various grid tools SAGA, GridRPC, HPC-Basic Profile Integration of WS-Eventing, WS-Notification, INFOD Sensor Web Enablement Integration of configuration and lifecycle management tools Configuration Description, Deployment and Lifecycle Management (CDDLM) Integration of WPS with workflow management tools BPEL, Kepler, Taverna, Triana, Pegasus, etc. Workflow design tools, execution engines, planning (data virtualization), data provenance Integration of WPS with grid security models SAML, XACML, VOMS, GSI Support for virtual organizations Integration of geospatial data with rule-based, data management policy engines iRODS

25 25 Digital Repositories/Libraries Digital Repositories/Libraries represent a growing part of the infomass Federated catalogues and federated storage OGF-Europe Workshop at OGF-23 4 sessions, ~70 participants from industry, academia and European Commission Proposed OGF-Europe collaboration with Global Research Library 2020 Proposed OGF-Europe involvement in GENESI-DR Digital Repositories Seminar in Autumn 2008, Frascati OGF-Europe PC membership for the DReSNet special DR session at IEEE e-Science conference, Dec 2008 Eventual development of an OGF Working Group dedicated to issues surrounding DRs Digital Repositories/Libraries represent a growing part of the infomass Federated catalogues and federated storage OGF-Europe Workshop at OGF-23 4 sessions, ~70 participants from industry, academia and European Commission Proposed OGF-Europe collaboration with Global Research Library 2020 Proposed OGF-Europe involvement in GENESI-DR Digital Repositories Seminar in Autumn 2008, Frascati OGF-Europe PC membership for the DReSNet special DR session at IEEE e-Science conference, Dec 2008 Eventual development of an OGF Working Group dedicated to issues surrounding DRs

26 26 Another Example: Green Grids Managing Energy Policy Source: Bill St. Arnaud virtualization fiber optics Extensive use of virtualization and fiber optics to put the computing where the power is Enable the workload to follow the wind or follow the sun virtualization fiber optics Extensive use of virtualization and fiber optics to put the computing where the power is Enable the workload to follow the wind or follow the sun

27 27 Another Example: Green Grids Managing Energy Policy Source: Bill St. Arnaud virtualization fiber optics Extensive use of virtualization and fiber optics to put the computing where the power is Enable the workload to follow the wind or follow the sun virtualization fiber optics Extensive use of virtualization and fiber optics to put the computing where the power is Enable the workload to follow the wind or follow the sun

28 28 Another Example: Green Grids Managing Energy Policy Source: Bill St. Arnaud virtualization fiber optics Extensive use of virtualization and fiber optics to put the computing where the power is Enable the workload to follow the wind or follow the sun A geospatial sensor network virtualization fiber optics Extensive use of virtualization and fiber optics to put the computing where the power is Enable the workload to follow the wind or follow the sun A geospatial sensor network

29 29 Summary Significant work already underway for standard, distributed geospatial computing infrastructures Highly relevant to many application domains Opportunity to leverage the work of key participants through collaboration How to engage application groups and industries needing distributed geospatial processing? Significant work already underway for standard, distributed geospatial computing infrastructures Highly relevant to many application domains Opportunity to leverage the work of key participants through collaboration How to engage application groups and industries needing distributed geospatial processing?

30 Back-Ups and Other Draft Material

31 31 A Cloudy Solution Space OS Virtualization Parallel Frameworks Software as a Service MapReduce, GFS, BigTable Amazon S3/EC2 Hadoop Hadoop over EC2 Google AppEngine Cohesive Suns Caroline MS Astoria Mesh RightScale, GigaSpaces, Elastra, 3Tera Courtesy of Dennis Gannon, map reduce Data Collection ….. Vendors of high-level application configuration tools are potential cloud framework providers

32 32 A Host of Cloud Issues Data access and interoperability Will have to be approached at the application domain level, by the domain users involved Security Can cloud providers provide sufficient security for sensitive applications? Reliability Can cloud providers provide sufficient reliability for critical applications? Frameworks How to manage sets of VMs and VOs? These are essentially cloud frameworks and call for "cloud framework providers" Performance management Clouds tend to abstract away location Managing the compute-data locality (affinity) can be important Costing models How to compare your own infrastructure costs with a cloud computer? How to compare two clouds? Data access and interoperability Will have to be approached at the application domain level, by the domain users involved Security Can cloud providers provide sufficient security for sensitive applications? Reliability Can cloud providers provide sufficient reliability for critical applications? Frameworks How to manage sets of VMs and VOs? These are essentially cloud frameworks and call for "cloud framework providers" Performance management Clouds tend to abstract away location Managing the compute-data locality (affinity) can be important Costing models How to compare your own infrastructure costs with a cloud computer? How to compare two clouds?


Download ppt "Craig Lee OGF-24 Mark Reichardt Grid-Enabled Geospatial Systems."

Similar presentations


Ads by Google