Algebraic relational approach to conflating images Central Washington University Ellensburg, WA 98926, USA Boris Kovalerchuk Dept of Computer Science William Sumner Dept of Geological Sciences Supported by NIMA University Research Initiative (NURI) grants ( ). Collaboration: PNNL, NGC, LSU

Agenda Introduction Example Algebraic summary Similarity measures Conflation/registration process Example Conclusions Questions

Example of matching problem: Corona and Landsat satellite images Assume that there is no meta data about scales, rotations, types and names of features. It is hard to extract tie points, but easier to extract polylines..

Corona and Landsat satellite images Feature overlap It is hard to extract tie points, but relatively easy to extract a lot of polylines (continuous or with gaps) for features like shorelines, ridges, streams Match of features provides more points than the tie point method. Key idea #1: Matching features

Algebraic invariants Algebraic invariants is a new methodology that automates the matching of raster and vector images from many sources with various resolutions and reliability, giving them common scales and coordinates. By using the term matching we mean fusion, correlation, registration and conflation of images and geospatial databases.

Terminology Registration – georeferencing Co-registration – matching images one to another. If one image registered then other one get registration via co-registration. Conflation – matching features of two images and joining attributes of matched features from both sources.

Flexibility In contrast to many traditional approaches, algebraic invariants: (1) do not rely on the identification of control points, (2) do not require common scales. (3) do not require that the orientations of individual images be known and (4) do not require that the types or names of linear features be known.

Key idea #1: Attempt to match features Key idea #2: Define features algebraically Key idea #3: Match features structurally Key idea #4: Permit gaps in features High speeds and automation are possible in principal since registration is done using only a tiny fraction of the total image data. Algebraic Invariants: Approach to matching features

Comparison of Math alternatives Geometry Distances, angles Too variant Algebra, relations between angles and distances, >,  Practically invariant Topology Topol. invariants Not robust (clouds) Too invariant Match changed Match robust Match refusal

Comparison of Math alternatives “Topological invariants such as Betti numbers are insensitive to scale, and do not distinguish between tiny holes and large ones. Moreover, features such as pockets, valleys, and ridges---which are sometimes crucial in applications--- are not usually treated as topological features at all” [Bern et al., 1999, NSF workshop]. The number of connected components (in the same drainage system) can vary significantly between different maps, aerial photographs, and sensor data for the same area. It depends on characteristics such as human error, map resolution, sensor capabilities and parameters, obstacles (e.g. clouds), data processing methods. Thus, in general, topological invariants are not invariants for the conflation problem. Support for Key idea #2: Define features algebraically.

Algebraic Invariants Provide a mathematical model where: – There is a well established math language from abstract algebra – Polylines that represent geospatial features are introduced as formal algebraic systems – Theorems on computational efficiency are proven Provide conflating algorithms – That reduce computational complexity – That are robust (practically invariant)

Definitions Definition. A pair a =  A,  is called an algebraic system if A is a set of elements and  is a set of predicates {P} and operators {F} on A and on its Cartesian products, where P: A  A ...  A  [0,1] and F: A  A ...  A  A. Further definitions complete the specification of linear features (polylines) and the model. We omit the detailed mathematics, but will show an example to explain the concepts. Note, this abstract algebraic system is not a traditional system of linear algebraic equations

Abstract Polylines a 3 a1a1 a 1 = [v 0,v 1 ], a 2 = [v 1,v 2 ],..., a n = [v n-1,v n ]... a2a2 a4a4 L1 =100 o L2=200 o L1 < L2 L1`=110 o L2`=190 o L1`< L2` v1v1 v2v2 v0v0 a5a5 a b L1` L2' Invariant relation between angles, despite variation of angles Possible gap

Structure of feature a as a matrix of relations between angles AngleL1L2L3L4L5L6 L L L31010 L4110 L511 L61 Li<Lj  P(Li,Lj)=0 A similar matrix can be constructed for relations between lengths of line segments or other task relevant characteristics of the feature. Key idea #5: Algebraic structure is expandable

Conflating algorithms Based on this algebraic model several algorithms may be designed to conflate images. One is to build matrices of relations that describe structures of two features a and b which are then searched for their largest common part. This is done by “sliding down” the diagonals of the matrices in the following way.

Illustrative matrix feature b and match with a (in grayed box) Angle L1L2L3L4L5L6L7L8L9L10 L L L L L L L71000 L8111 L911 L101

Example Corona Landsat After rotation, scaling, and translation

Theorems are proven, such as Theorem. If the number of elements in linear features a and b equals n, then their maximum co-reference subsystem e can be found in O(n 3 ) matrix comparisons for the worst-case scenario that is equivalent to O(n 5 ) binary comparisons.

Useful, but... Comparison techniques are needed for polylines that have different sampling densities. Similarity measures have been developed to do this.

Similarity Measures To define measures we consider a polyline a with two end points p 1 =(x 1,y 1,z 1 ) and p 2 =(x 2,y 2,z 2 ). The recurrent function G(n) produces a set of simpler polylines that interpolate polyline a. Arguments of the function G are n = 2 k where k = 1,2,…, that is 2, 4, 8, 16, 32…

Structural interpolations of a polyline We introduced measures of structural similarities of features based on comparison of structural interpolations and generalized this to measuring structural similarities of entire images.

Conflation/registration process

Process illustration Two vector image data sets are considered. Image 1 consists of 1497 line segments. Image 2 consists of 407 line segments. Thanks to S. Sento and P. Brennan (Northrop Grumman) for providing data.

Input vector images “Are these two images of the same scene?”

The two images conflated and combined

Features in common

Features not in common Red-Image 1, blue –Image 2

Observations The number of polylines in common supports the conclusion that these are two images of the same scene. The number of polylines not in common illustrates the need for additional information to know whether these are the result of incomplete or faulty feature information, or changes in the scene between the times of acquisition.

Key idea #1: Attempt to match features Key idea #2: Define features algebraically Key idea #3: Match features structurally Key idea #4: Permit gaps in features High speeds and automation are possible in principal since registration is done using only a tiny fraction of the total image data. Summary: Key ideas

Summary: conclusion This algebraic relational approach is a promising way of conflating both raster and vector images – with no control points – with unknown scales – with unknown orientations Can be used to automatically search for specific objects of interest by predefining their abstracted linear shapes Can be used to automate the identification of locations that change in time More work ahead, e.g., to incorporate prior imagery knowledge. Test data welcome

Questions? Boris Kovalerchuk Bill Sumner edu edu

Super features The following slides illustrate super features