1. WISE Distributed System Koen Mertens Jan De Belder Rombout Verwimp SADL K.U. Leuven Geo Solutions NV

2. Part 2: Get it real! Overall agenda Part 1: WISE ds theory Part 2: Get it real! + demo Part 3: Results + discussion Part 4: Potential use WISE-ESTAT case + demo

3. Part 2: Get it real! Agenda Part 2 WISE ds goals & architecture WISE ds technical framework

4. Part 2: Get it real! Wise ds pilot goals 1 Evaluate the overall data flow Evaluate performance issues/requirement Evaluate INSPIRE specifications for WISE Evaluate the use of mixed WISE QA/QC engine for object and file based reporting

5. Part 2: Get it real! Evaluate metadata elements required on the data but more specific to follow up the data streams Evaluate service communication with member states To give advice on the management/coordination required by this sort of systems. WISE ds pilot goals 2

6. Part 2: Get it real! All data except WFD Art 8 data WISE data flow manager Links with REPORTNET No dissemination towards applications –(except 1H02 link to Eurostat pilot) Operational production ready systems Performance benchmarking at EEA site Multilingual applications (only English will be used) Full configurable applications -> goal is to illustrate the flow! Out of scope

7. Part 2: Get it real! Business process

8. Part 2: Get it real! Logical architecture

9. Part 2: Get it real! Based on : –Microsoft BizTalk (business processing management software – XML messaging) –Microsoft SQL server technology –C#.NET (web services) technology –ARCGIS server + desktop technology (9.3) –Postgresql/postgis –Apache Tomcat + deegree WFS –FTP WISE DS pilot solution

10. Part 2: Get it real! SQL Server 2005 LUT tables: lookup values from xsd Other tables: metadata information about contacts, reported files, file status, errors,… Link to object geodatabase: lnkFileObject: links objectids from geodatabase to fileIDs in management database Metadata management DB

11. Part 2: Get it real! Metatadata management DB

12. Part 2: Get it real! ESRI ArcGIS ArcSDE geodatabase Complex relationships due to xml structure Links: ParentTableName_Id is foreign key in childTables Physical primary keys = oid or objectid while the logical primary key = ParentTableName_Id No physical relationships using relationship classes at ArcSDE level Only 2 feature classes: Ground- and surfacewatermonitoringStation, the rest are tables Feature database

13. Part 2: Get it real! only 2 feature classes complex relationships: different ways of being child only 3 types of objects/xml Reported objects in geodatabase

14. Part 2: Get it real! Pilot components : Biztalk & web service interaction

15. Part 2: Get it real! Incoming reported file passes following stages: –File Harvesting: get & archive the file –Metadata writing –File validation Schema validation (xml schema test) Spatial validation (spatial coordinates test) Content validation (xml content test) –File approval or rejection –Storage of spatial & attribute information in spatial relational DB –Dashboard consultation Logical flow

16. Biztalk flow

17. Part 2: Get it real! UC 1: File posting: Member states report the WISE information through an FTP server or through the setup of a WFS service UC 2: File harvesting: BizTalk picks up the information from the FTP or WFS. UC 3: Controller action: The information is processed by the.NET controller web service UC 4: QA/QC: Quality control of the content of the reported data Use cases

18. Part 2: Get it real! UC 5: GIS data handling: All GIS related operations are performed by the GIS web service, e.g. object storage in the wise object geodatabase UC 6: Metadata handling: Metadata about the reported data is maintained in the management database UC 7: Dashboard usage: The dashboard facilitates information exchange from the management database Use cases

19. UC1: File Posting Ref.Use caseDescription UC.100File PostingA Member state actor posts a file UC.1.01FTP postingA Member state actor uploads a file to the ftp server UC.1.02WFS postingA Member state actor updates the WFS service

20. UC2: File Harvesting Ref.Use caseDescription UC.2File harvestingInformation is picked up from a member state reporting source UC.2.01FTP harvestingBiztalk harvest a file from a member state ftp server UC.2.02WFS harvestingBiztalk harvest information from a member state WFS service

21. UC3: Controller action Ref.Use caseDescription UC.3Controller actionThe WISE DS control service is activated to perform some actions UC. 3.01Initialize Object DB schemaRequest the GIS service to initiate the geodatabase schema based on 3 reference xml files residing on the server UC. 3.02Send mail messageSend an email message UC. 3.03Get objects to update from WFS Gets the objects to update from the WFS service, given a dataset based on the wfs response and the objects currently in the geodatabase UC. 3.04Process WFS result from fileBizTalk passes on the wfs result xml file to the controller to update the geodatabase with. This process involves the process xml use case also UC. 3.05Process XMLHandles all the business logic associated with the handling of the information reported by the MS

22. UC4: QA/QC Ref.Use caseDescription UC.4QA/QCQuality checks on the reported data UC. 4.01Schema validationCheck if the reported information is conform the xml file schema (xsd) UC. 4.02Spatial validationCheck whether reported data is inside the corresponding country boundaries UC. 4.03Content validationCheck whether the content that is reported behaves according to several rules

23. UC5: File Harvesting Ref.Use caseDescription UC.5GIS data handlingAll GIS associated operations UC. 5.01Get country codeGets the country code of the country for an x,y coordinate UC. 5.02Get objectsFetch the GIS objects of a certain type, given their ids from the geodatabase UC. 5.03Stations outside countryGets the reported station objects that are not located inside the specified country UC. 5.04Write dataset to geodatabase Writes the objects inside a.NET dataset object to the geodatabase UC. 5.05Write dataset to geodatabase schema Construct the geodatabase schema based on a reference dataset object

24. UC6: Metadata handling Ref.Use caseDescription UC.6Metadata handlingAll operations associated with the WISE management database UC. 6.01Archived fileRegister a file as archived in the database UC. 6.02AuthenticateAuthenticate a user UC. 6.03Get Article 8 testsGet the current tests for article 8 UC. 6.04Get emailGets the email address, that should be used to report about a certain file UC. 6.05Get errorsGet all the errors associated with a certain file UC. 6.06Get file IDGet the file id based on the filename UC. 6.07Get file statusGet the file status UC. 6.08Get file overviewGet an overview of all files reported by a certain contact UC. 6.09Get file overview betweenGet an overview of all files reported by a certain contact in a certain time interval UC. 6.10Get LUT values for tableGet all look up table values inside a certain table UC. 6.11Get object idsGet the object ids for a certain file and file type UC. 6.12Invalid schemaReport that a schema is invalid for a file UC. 6.13Link file to objectsLink a file to its objects stored in the geodatabase UC. 6.14Report errorReport an error for a certain file UC. 6.15Update file statusUpdate the file status for a certain file

25. UC7: Dashboard usage Ref.Use caseDescription UC.7Dashboard usageProvides the member states with metadata information about their reported files UC. 7.01Log inLog in to the dashboard website UC. 7.02Date selectionSelect a time interval in which files were reported UC. 7.03File selectionSelect a file to display its details (status and errors) UC. 7.04Object selectionSelect a reported object to view its geographic details using a wms service

26. FTP WFS Dashboard Demo

27. Part 2: Get it real! FTP upload

28. Part 2: Get it real! FTP upload

29. Part 2: Get it real! Receive email alerts

30. Part 2: Get it real! Example Email

31. Part 2: Get it real! WFS running on remote server

32. Part 2: Get it real! Object Database schema

33. Part 2: Get it real! MetaData database schema (management db)

34. Part 2: Get it real! Tbl_contacts

35. Part 2: Get it real! Tbl_errorlist

36. Part 2: Get it real! Tbl_reportedfiles

37. Part 2: Get it real! Tbl_filestatus

38. Part 2: Get it real! Object database in web viewer

39. Generic & Extensible Framework of services –Data flow governed by BizTalk (XML intermediate files) –Custom code available as web methods –Instead of “black box” of code Priorities in development –Technical issues over organisational issues Time frame too limited for extensive user testing Primary goal was to demonstrate the feasibility of the automated dataflow Technological choices –FTP: easily implementable at MS side –WFS: advanced implementations required WFS – complex features: currently only 1 solution –DEEGREE-WFS Decision steps in development

40. FTP –No complex IT requirements on the MS side –Limited reuse of ART 8 data for dissemination WFS –For serving complex features: only 1 solution: DEEGREE WFS –More complex IT architecture required at MS side Tomcat + Postgres/postgis + Deegree Datamodel according to ART 8 schema –WFS = service : reuse of ART 8 data for dissemination Findings