Philadelphia, May 2–4, Effective use of spatial databases for Enterprise data integration Dr Paul Watson Laser-Scan

Philadelphia, May 2–4, Overview Motivation - review key drivers for Data Integration Obstacles to Data Integration Data Quality Management Principles –Data Stewardship –Knowledge Management –Enterprise metadata –Rules based processing Methdology – Data Quality Improvement cycle Spatial Business Rules Spatial Knowledge Management – Radius Studio –Rules browser/ rules repository –Conformance check/ report –Reconciliation –Certification

Philadelphia, May 2–4, Drivers for Data Integration Agile Competitors and Virtual Organisations - Goldman, Nagel and Preiss

Philadelphia, May 2–4, Obstacles to Data Integration Proprietary Data Silos - data cul de sac Monolithic Information Systems – embedded logic Built-in, Private Data Models – structure/lifecycle Unknown/Unproven Data Quality – KR/KM

Philadelphia, May 2–4, Enablers For Enterprise Information Architectures Electronic Data Interchange Straight-through Processing Service Oriented Architectures Rules-based Processing Knowledge Representation Standards Data & service integration is the goal

Philadelphia, May 2–4, Technological Building Blocks Open data access standards Extensible, interoperable platforms Metadata publication/retrieval Data stewardship

Philadelphia, May 2–4, Data Quality Impact One third of companies have been forced to delay or scrap new systems because of faulty data, and a full 75% have experienced significant problems resulting from data quality issues - PwC 55-70% of CRM and 70% of Data Warehouse project failures are due to data quality issues - Gartner, Meta US business loses $600 billion each year due to data quality problems - Data Warehousing Institute When is data “fit for purpose”?

Philadelphia, May 2–4, Principles, Tools & Method for DQ Improvement Knowledge Management –Describe the domain/problem independently (from data and systems) Rules-based paradigm –Decouple the problem definition from problem discovery & resolution (what to do about it) Data Quality Improvement Cycle –Employ a process of continuous monitoring & improvement - sustainable interoperability ™

Philadelphia, May 2–4, Knowledge Management Where is knowledge now? –Embedded in data –Hidden in point applications –Inside people’s heads Store the knowledge/expertise of the organisation where everyone can contribute to it and share it - as enterprise metadata in a database

Philadelphia, May 2–4, Benefits of a knowledge management approach Explicit/unambiguous – not embedded Inclusive - accommodates domain experts Non-technical – not just for developers Open, distributable – location/applications Auditable – regulatory/governance issues Evolutionary – incremental acquisition, knowledge is refined/grows over time Structured – machine-readable

Philadelphia, May 2–4, Knowledge Representation RDF OWL SWRL

Philadelphia, May 2–4, Data Quality as Knowledge Express data requirements as rules (e.g. SWRL) Data quality rules – enterprise metadata - DB Rules metadata can be shared (interpreted and enforced) by many different applications Rules can be used to measure Data Quality - % conforming data instances Rules guide data reconciliation - prioritise Rules can be used to measure quality improvement reliably “Fit for Purpose” = satisfies the DQ rules

Philadelphia, May 2–4, Rules-based Processing Paradigm Fact – Pattern – Action Given some facts, if they meet any of the patterns/rules, perform the defined action –Declarative – rule separated from processing –Pluggable actions – reporting/ reconciliation DataRule A Report Action A Reconcile Action A

Philadelphia, May 2–4, Philadelphia, May 2–4, make baseline assessment refine rule metadata check data conformance perform data reconciliation data certification data publication define rule metadata define quality mission Data Quality Improvement Cycle SP ☺ e.g. address cleaning

Philadelphia, May 2–4, Enterprise Integration Gap for Location Spatial data - complex, brittle and surprising Maintenance - manual, expensive, error-prone Adherence to a spatial model is often business critical (e.g. land & property management, utilities) Tools - significant bespoke development, inflexible, built by developers not domain experts Location data - key for BI, data quality chasm, mining Rudimentary IT standards for spatial data Spatial semantics never explicit – KR/KM

Philadelphia, May 2–4, Spatial Knowledge Management Same methodology –Spatial business rules as metadata –Conformance checking/reporting –Data Reconciliation – rules driven –Certification/publication –Only detailed data operations change Business Rule - Percentage: 0 ≤ x ≤ 100 Location Rule Spatial (a,b) : coveredBy, contains, withinDistance etc. Empower the domain expert – short “dev.” cycles, stay agile

Philadelphia, May 2–4, Spatial Data Quality Rules Intra-feature Constraints physical geometric Data Specifications Ad hoc Rules Inter-feature Constraints proximity topological directional Spatial Rules Discovery

Philadelphia, May 2–4, Spatial Rule Types

Philadelphia, May 2–4, Spatial Rules Authoring curbline building

Philadelphia, May 2–4, Spatial Rule Builder

Philadelphia, May 2–4, Add Rule Clauses building zoning

Philadelphia, May 2–4, Spatial & Non-spatial Conditions Fire station building Street centreline “There’s no such thing as a spatial rule”

Philadelphia, May 2–4, r r g r n n j j a g a High St. Road Road Segment For each Road, the set of Road Segments having the same name as the Road must have the same aggregate geometry as the Road Rules as Knowledge

Philadelphia, May 2–4, Conformance Checking Spatial Rules Engine Enterprise Messaging Enterprise Spatial DB Enterprise Metadata - Spatial Rules Web Services Clients Asynchronous Messaging Client Browser Client Reporting Solution

Philadelphia, May 2–4, Data Reconciliation Spatial Rules Engine Enterprise Spatial DB Enterprise Metadata - Spatial Rules Web Services Clients Update A-B-A Clone A-B-A’ Schema Map A-B-C

Philadelphia, May 2–4, Warehouse Certification Spatial Rules Engine Enterprise Spatial DB Enterprise Metadata - Spatial Rules Web Services Clients

Philadelphia, May 2–4, Summary Data quality issues in integration are best addressed using knowledge management/rules-based approaches Spatial data quality is no different to any other data quality – standard, interoperable rules are key Enterprise spatial rules engines form a secure base from which to develop open, distributable location enabled applications

Philadelphia, May 2–4,