1. Work package in ABC III Specific Contract with PwC/interactive instruments on implementation of INSPIRE validator Clemens Portele, interactive instruments GmbH

2. WP4 – Conformance and interoperability testing  Objectives:  create a reusable open source reference tool for conformance testing for INSPIRE  contributing to wider interoperability testing relevant to ISA  Support and accelerate ongoing work in MIWP-5 to create a common agreed validator tool for data, metadata and services  identified as one of the major missing items by the MIG and other INSPIRE stakeholders  This work will build upon existing solutions in the Member States and European projects to create a ‘reusable’ testing infrastructure for INSPIRE  The tool should test the overall functioning of the infrastructure while offering configurable software and test rules for organisations to test conformance  Alignment with work related to MIWP-5 is key

3. Overview of tasks and deliverables

5. INSPIRE Testing Framework – Design  Analyse requirements  Analyse existing solutions and testing frameworks  Develop a proposal for an INSPIRE testing framework This process starts today …

6. INSPIRE Testing Framework – Requirements – Initial List (1/3)  Test objects  The testing framework should (at least) accept single requests to test the conformity of  a metadata record  a network service endpoint  a data set  a spatial data service endpoint  The framework should support the testing of large data sets (e.g. orthoimagery)  Test results  The test engine shall return a detailed test report in human-readable (e.g. HTML) or machine- readable format (e.g. XML, JSON)

7. INSPIRE Testing Framework – Requirements – Initial List (2/3)  Test projects and their management  The ETS language/format should allow easy modification of rules and documentation of a version history  It should be possible to select / define modules at individual rule level  It should be possible to test against specific versions of the rule  It should be possible to validate against specific schema versions (this may require parameterisable rules)  It should be possible to indicate the severity or weight of individual rules  ETS rules should have links to the ATS rules they implement and requirements they test

8. INSPIRE Testing Framework – Requirements – Initial List (3/3)  Architecture  The framework should be based on a generic test engine that can process executable tests specified in an ETS language/format  It should be possible to update or add rules without needing to recompile, rebuild or redeploy the test engine  It should be possible to interact with the test engine through a GUI and through an API  Licensing  The test engine and ETS language/format should be open source See also:  https://ies-svn.jrc.ec.europa.eu/projects/validation-and-conformity- testing/wiki/INSPIRE_testing_framework_-_requirements https://ies-svn.jrc.ec.europa.eu/projects/validation-and-conformity- testing/wiki/INSPIRE_testing_framework_-_requirements  https://ies-svn.jrc.ec.europa.eu/projects/validation-and-conformity- testing/wiki/INSPIRE_testing_framework_-_concepts https://ies-svn.jrc.ec.europa.eu/projects/validation-and-conformity- testing/wiki/INSPIRE_testing_framework_-_concepts

9. INSPIRE Testing Framework – Requirements – Process  Prioritise requirements following the MoScoW approach  together with JRC and the MIWP-5 sub-group (today!)  Based on the result:  review scope and approach of the work in the remaining deliverables  revise if necessary

10. INSPIRE Testing Framework – Our View (1/3)  The INSPIRE Testing Framework will in a first phase not be conceived as a single generic test engine allowing to issue a single request and receive a harmonised test report  It is our expectation that a rapid development of a “test component” as foreseen by this work package will only be possible, if the test framework will build on existing software tools, existing executable test suites and existing deployments to the extent possible  Any other solution would likely also lead to duplication of existing work

11. INSPIRE Testing Framework – Our View (2/3)  As a result, we foresee that testing an INSPIRE resource (service, data set or metadata record) will often mean that several test tools will be used to validate it. I.e., the test framework would include a number of test engines and deployments.  The ELF project follows a similar approach  A WFS 2.0 download service is expected to pass  the OGC CITE tests for WFS 2.0 (Basic WFS conformance class)  the INSPIRE Geoportal Metadata Validator  the ETF-based ELF/INSPIRE WFS 2.0 tests  the Spatineo Performance monitor QoS tests  In addition, the data set is expected to be tested using one of the data quality validators tested in the project

12. INSPIRE Testing Framework – Our View (3/3)  For cases where no existing deployment exists: set up a configuration of a test tool that meets the requirements for the executable tests of the test object  If there is a need to extend an existing test engine with additional capabilities and there is sufficient documentation and support from the developers: implement the extension  Use an agile process to document and incrementally develop the testing framework – where possible, together with the executable tests  Allows to adjust priorities as the work progresses and to focus on the most important aspects

13. INSPIRE Testing Framework – Our View (4/4)  For the executable test suites:  Executable tests will be developed where no existing tests exist to cover the abstract test suites  It is expected that these will mainly be the data interoperability requirements, as test suites already exist and have been in use for some time for INSPIRE requirements related to network services and metadata  Tests for the data interoperability requirements are in an earlier state of development  Validate draft tests with the representative samples identified in the test plan (  D4.3.1)  Consider how easy error reports can be understood  Where issues are found with existing third-party tests, we will propose changes to the providers of the test tool, test or test application

14. INSPIRE Testing Framework – ELF approach to service testing INSPIRE Geoportal Metadata Validator OGC CITE WFS 2.0 GML 3.2 WMS 1.3 WMTS 1.0 Spatineo Performance monitor, Spatineo monitor http(s) Data Set Network Service ETF, configured with INSPIRE / ELF tests Data Provider ELF Service Manager

15. Testing ELF web services – Overview TestValidatorTools usedHosted by OGC conformanceOGC CITE Tests for WFS 2.0, GML 3.2, WMS 1.3, WMTS 1.0 TEAM Engine, Test NG OGC, local tests possible Service metadataINSPIRE Geoportal Metadata Validator JRC INSPIRE/ELF requirements on services, schemas and GML encoding ETF WebApp configured with INSPIRE/ELF-specific tests ETF (with soapUI and BaseX drivers) interactive instruments, local tests possible Service capacityINSPIRE Capacity requirementsSpatineo Performance monitor NLSFI Service performanceINSPIRE performance (availability)Spatineo monitorNLSFI

16. INSPIRE Testing Framework – Potential (Future) Integration INSPIRE Geoportal Metadata Validator OGC CITE WFS 2.0 GML 3.2 WMS 1.3 WMTS 1.0 … http(s) Data Set Network Service … User INSPIRE validator Prerequisites for test components: API Parsable, well-documented and well-structured response format …

17. INSPIRE Testing Framework – Analysis (1/2)  Review and assess the re-usability of existing approaches/software components, including at least:  the list of tools collected on the MIWP-5 collaboration space  https://ies-svn.jrc.ec.europa.eu/projects/validation-and-conformity- testing/wiki/Overview_about_existing_validation_toolssolutions https://ies-svn.jrc.ec.europa.eu/projects/validation-and-conformity- testing/wiki/Overview_about_existing_validation_toolssolutions  NB: This list includes the test framework developed by ii, too (ETF)  the validation components of the EU Geoportal operational pilot and procurement version (developed by Planetek and lat/lon)  OGC TeamEngine and CITE tests relevant for network services and data interoperability

18. INSPIRE Testing Framework – Analysis (2/2)  In this process, we will analyse use constraints, intellectual property, long-term operational readiness of the existing tools, tests and deployments. Their impact and a potential need for agreements with the current owners and operators will be investigated as early as possible.  We will also explore available standards for conformance testing, such as the OASIS Test Assertion Markup Language (TAML) and the CEN Generic Interoperability Testbed (GITB) workshop agreement (  Task 4.4)

19. INSPIRE Testing Framework – Design Proposal  Based on the analysis of requirements and existing approaches/software components, we will develop a proposal for the implementation of the INSPIRE testing framework  Should be readily understandable  Include the provision of possible options for (operational) development  Present a draft of the proposal to MIWP-5 and other interested stakeholders  Incorporate feedback into the final version

20. INSPIRE Testing Framework – Abstract Test Suites  Complete the documentation of the Abstract Test Suites (ATSs) for metadata, network services, data interoperability and spatial data services, documented in the Github repository with appropriate metadata.  Review and provide comments for the existing ATSs  Add ATSs for data specifications  Propose additional metadata  Support the resolution of review comments

21. INSPIRE Testing Framework – Implementation – Test plan  Proposed set of representative samples for resources that can be used for validating the development version of the test component(s). The test plan will include:  The scope of the testing, identifying priorities  Test strategy and operational scenario plan  Test environment, including hosting options  Test schedule including organisation of the operations  Control procedures  Resources and responsibilities  Risks associated to the testing activities  Workshop with MIWP-5 and/or other stakeholders foreseen

22. INSPIRE Testing Framework – Implementation – Framework  Process discussed as part of the Design phase

23. INSPIRE Testing Framework – Implementation – ETS  Process discussed as part of the Design phase

24. INSPIRE Testing Framework – Implementation – Documentation  Test framework and executable tests will be documented with regard to installation, development and use, as applicable  Third party tools, tests and deployments will only be documented from the perspective of using the tests  Online documentation!

25. GITB integration  Explore possible synergies with the CEN Global eBusiness Interoperability Test Bed Methodologies (GITB) and ISA action 4.2.6 “Interoperable test bed”  Coordination with relevant stakeholders including ISA and MIWP-5  Compile a report with a proposal on how to achieve compliance with GITB methodologies in the conformance testing framework of INSPIRE and draft metadata following their standards