1. Deutscher Wetterdienst DWD GISC Implementation GISC Development Team

2. Page: 2GISC Audit 17.09.2010 Agenda  Welcome  Architecture –High level Architecture –Hardware  Performance Tests  International Collaborations and Co-operations  Contributions to WIS  Q&A

3. Page: 3GISC Audit 17.09.2010 Design Principles of DWD GISC  Reuse of existing well established software and implementation methodology  Use of established Interfaces – in particular to „critical components“ to allow later exchange if necessary –Metadata Providing –Authentication and Authorization  Modular concept to address performance and scalability requirements  Clear separation of responsibilities  Keep things simple  Describe external interfaces

4. Page: 4GISC Audit 17.09.2010 DWD GISC - Development  In house development team and support of business units  Professional Services –Fraunhofer – research facility –Con terra – experts on Metadata  Co-operation with companies –ask Contributes to the development with 2-3 persons Supports WMO events (EC Demo) –IBL Long term partner Supports WMO events (EC Demo) Face lift of DWD GISC Offers professional services (integration and maintenance) for DWD GISC

5. Page: 5GISC Audit 17.09.2010 High Level Architecture: Functional Blocks Security Layer Monitoring and Reporting Layer Metadata Database Subscription Management Web Interfaces –Customer –Operator –Administrator Data & Product Import Management & GISC Synchroniser DAR Components – Metadata Harvesting – Metadata Providing Data & Product Access Management 24 h Cache WIS Global Ex. File pools, Archives, Databases DAR Generator /Importer SRU Search and Retrieval by URL

6. Page: 6GISC Audit 17.09.2010 Timeline Project Start Concept Phase 11/08 01/09 03/09 05/0907/09 09/09 11/0901/10 03/10 05/10 VGISC Workshop G2G com. Workshop Impl. Start: Web Interface Subscription DAR Metadata Generator DAR Catalogue Perf. Study G2G com. Tests G2G com. Workshop MoU CMA VGISC Backup WS OAI Infrastructure SRU OAI & WIS Workshop Data & Product Management Test 06/10 EC Demo 08/10 OAI Tests 09/10 Metadata Study

7. Page: 7GISC Audit 17.09.2010 High Level Architecture: Functional Blocks Security Layer Monitoring and Reporting Layer Metadata Database Subscription Management Data & Product Import Management & GISC Synchroniser DAR Components – Metadata Harvesting – Metadata Providing Data & Product Access Management 24 h Cache WIS Global Ex. File pools, Archives, Databases DAR Generator /Importer SRU Search and Retrieval by URL Web Interfaces –Customer –Operator –Administrator

8. Page: 8GISC Audit 17.09.2010 High Level Architecture: Web Interfaces  GISC –Redesign of WebWerdis –Navigation structure –Search functions Simple Search Extended Search (e.g. Full Text Search) Expert Search –Internationalization (I18N) –Integrate logging framework (necessary for monitoring) –Code clean-up –Entry points for External Application (e.g. Online Resource)  AUST –Redesign of AUST

9. Page: 9GISC Audit 17.09.2010 High Level Architecture: Functional Blocks Security Layer Monitoring and Reporting Layer Metadata Database Subscription Management Data & Product Import Management & GISC Synchroniser Data & Product Access Management 24 h Cache WIS Global Ex. File pools, Archives, Databases SRU Search and Retrieval by URL Web Interfaces –Customer –Operator –Administrator DAR Components – Metadata Harvesting – Metadata Providing DAR Generator /Importer

10. Page: 10GISC Audit 17.09.2010 High Level Architecture: DAR Generator, Importer and Components  DAR Metadata Provider –jOAI (also used by NCAR/UCAR – reference site)  DAR Metadata Harvester –OAIHarvester2 –Modified Source Code with CMA together –In house development of a scheduler and workflow component (done by ask)  DAR Metadata Generator –In house development by DWD shared with CMA, Hong Kong, WMO, Brazil, Norway, Australia, South Africa, …  DAR Metadata Importer –In house development by DWD

11. Page: 11GISC Audit 17.09.2010 High Level Architecture: Functional Blocks Security Layer Monitoring and Reporting Layer Metadata Database DAR Components – Metadata Harvesting – Metadata Providing DAR Generator /Importer SRU Search and Retrieval by URL Web Interfaces –Customer –Operator –Administrator Subscription Management Data & Product Import Management & GISC Synchroniser Data & Product Access Management 24 h Cache WIS Global Ex. File pools, Archives, Databases

12. Page: 12GISC Audit 17.09.2010 High Level Architecture: Delivery Components  Subscription Management –Controller: time and event triggered management  AFD –Workhorse in the background –GISC-to-GISC communication –File distribution –White List  Message Switching System –REST Web-Service to manage and control Moving Weather –First proposal for generic Interface

13. Page: 13GISC Audit 17.09.2010 High Level Architecture: Functional Blocks Security Layer Monitoring and Reporting Layer Metadata Database Subscription Management Data & Product Import Management & GISC Synchroniser DAR Components – Metadata Harvesting – Metadata Providing Data & Product Access Management 24 h Cache WIS Global Ex. File pools, Archives, Databases DAR Generator /Importer Web Interfaces –Customer –Operator –Administrator SRU Search and Retrieval by URL

14. Page: 14GISC Audit 17.09.2010 High Level Architecture: SRU  SRU –WMO SRU2JDBC implementation Cooperation between WMO and DWD available as Open Source –In house development by DWD: Mapping of CQL (Contextual Query Language) to SQL (done by ask) –External configuration file for Mapping It is easy to add new indices Database specific feature (e.g. functions, special operators) are kept outside of the code –All Queries and Responses are encoded according to the SRU-Standard allowing for both interactive as well as automated usage – e.g. by catalogue systems

15. Page: 15GISC Audit 17.09.2010 Hardware Architecture –Infrastructure DMZDMZ INTRANETINTRANET Web Cache OAI Infrastructure Application Server Adapter & Scheduler Operations Application Server Adapter & Scheduler OAI Infrastructure Integration System

16. Page: 16GISC Audit 17.09.2010 Performance  05/2009 Performance Study of DAR Metadata Catalogues – Both Systems terraCatalog 2.3 and Geonetworks 2.2 do not fulfill all Performance Requirements  09/2009 CMA and DWD did performance study of GISC-to-GISC communication solutions – JMA Blog solution and AFD  03/2010 CMA and DWD did performance tests of OAI Provider solution  04/2010 INMET and DWD did performance tests of GISC-to-GISC communication solutions – AFD and Moving Weather (IBL)  07/2010 Internal performance tests of DAR Metadata Catalogue and SRU – Lessons learned –Database is hot spot, especially when we talk about Full Text Search –Oracle standard reports –Limit the result set – configuration parameter –Optimize SQL Statements Explain plan Rewrite SQL Statements Indices –Improved responses for most critical queries by an order of magnitude

17. Page: 17GISC Audit 17.09.2010 International collaboration: CMA (GISC)  GISC-to-GISC communication –Synchronization of 24 hour cache Extensively tests of JMAs Blog solution Extensively tests of AFD  GISC-to-GISC communication –Description for GISC-to-GISC backup based on XML Schema

18. Page: 18GISC Audit 17.09.2010 International collaboration: CMA (GISC)  DAR Metadata –DAR Metadata Generator –Metadata Template –Request for Clarification paper 11/2009 (CMA, JMA and DWD)  Metadata Providing / Harvesting –Functional tests of standard OAI provider –Common development of OAI Harvester –Extensively performance tests for DAR Metadata synchronization  SRU –Mapping indices to DAR Metadata elements  Workshops: –GISC-to-GISC communication workshop 04/2009 at DWD –GISC-to-GISC communication workshop 09/2009 at CMA –OAI workshop 03/2010 at CMA

19. Page: 19GISC Audit 17.09.2010 International co-operation: JMA (GISC)  DAR Metadata –Design of DAR Metadata –Request for Clarification paper 11/2009 (CMA, JMA and DWD)  Metadata Providing / Harvesting –Synchronization tests based on OAI  GISC-to-GISC communication –Knowledge exchange JMA Blog solution –JMA will take XML Schema (based on a proposal from CMA and DWD) as working draft for GISC-to-GISC backup  Workshop –WIS Workshop at JMA 03/2010

20. Page: 20GISC Audit 17.09.2010 International co-operation: Brazil (GISC)  DAR Metadata –DAR Metadata Generator  Metadata Providing / Harvesting –Synchronization tests based on OAI  GISC-to-GISC communication –Moving Weather –AFD

21. Page: 21GISC Audit 17.09.2010 International collaboration: EUMETSAT (DCPC)  DAR Metadata –Conceptual design of DAR Metadata –Knowledge exchange  DAR catalogue –Extensively knowledge exchange  VGISC ITT –Attended as expert  EUMETSAT is connected as DCPC since 23th of August 2010 –Test Phase 06/07 2010

22. Page: 22GISC Audit 17.09.2010 International co-operation: Hong Kong (DCPC)  DAR Metadata –DAR Metadata Generator  Metadata Providing / Harvesting –Synchronization tests based on OAI International co-operation: NCAR/UCAR (DCPC)  DAR Metadata –Knowledge exchange  Metadata Providing / Harvesting –Synchronization tests based on OAI

23. Page: 23GISC Audit 17.09.2010 International co-operation: VGISC  Since 2002 VGISC – Météo France, UK MetOffice, DWD, met.no, ECWMF and EUMETSAT  SIMDAT  VGISC ITT  Several workshops regarding Backup and Recovery

24. Page: 24GISC Audit 17.09.2010 Contributions to WIS  DWD participates several ETs, ICG, …  Released Metadata catalogue performance study to community  Geonetwork improvements  Shared results of DAR Metadata analysis with WIS community  DAR Metadata Generator  Released SRU2JDBC in co-operation with WMO as Open Source http://sru2jdbc.svn.sourceforge.net/ http://sru2jdbc.svn.sourceforge.net/  Active member of jOAI community  Contribute experts for WIS Jump Start Offer

25. Page: 25GISC Audit 17.09.2010 Q&A

26. Page: 26GISC Audit 17.09.2010 Live Demo http://gisc.dwd.de/