1. Algebraic relational approach to conflating images Central Washington University Ellensburg, WA 98926, USA Boris Kovalerchuk Dept of Computer Science borisk@cwu.edu William Sumner Dept of Geological Sciences sumner@geology.cwu.edu Supported by NIMA University Research Initiative (NURI) grants (2002-2005). Collaboration: PNNL, NGC, LSU

2. Agenda Introduction Example Algebraic summary Similarity measures Conflation/registration process Example Conclusions Questions

3. Example of matching problem: Corona and Landsat satellite images Assume that there is no meta data about scales, rotations, types and names of features. It is hard to extract tie points, but easier to extract polylines..

4. Corona and Landsat satellite images Feature overlap It is hard to extract tie points, but relatively easy to extract a lot of polylines (continuous or with gaps) for features like shorelines, ridges, streams Match of features provides more points than the tie point method. Key idea #1: Matching features

5. Algebraic invariants Algebraic invariants is a new methodology that automates the matching of raster and vector images from many sources with various resolutions and reliability, giving them common scales and coordinates. By using the term matching we mean fusion, correlation, registration and conflation of images and geospatial databases.

6. Terminology Registration – georeferencing Co-registration – matching images one to another. If one image registered then other one get registration via co-registration. Conflation – matching features of two images and joining attributes of matched features from both sources.

7. Flexibility In contrast to many traditional approaches, algebraic invariants: (1) do not rely on the identification of control points, (2) do not require common scales. (3) do not require that the orientations of individual images be known and (4) do not require that the types or names of linear features be known.

8. Key idea #1: Attempt to match features Key idea #2: Define features algebraically Key idea #3: Match features structurally Key idea #4: Permit gaps in features High speeds and automation are possible in principal since registration is done using only a tiny fraction of the total image data. Algebraic Invariants: Approach to matching features

9. Comparison of Math alternatives Geometry Distances, angles Too variant Algebra, relations between angles and distances, >,  Practically invariant Topology Topol. invariants Not robust (clouds) Too invariant Match changed Match robust Match refusal

10. Comparison of Math alternatives “Topological invariants such as Betti numbers are insensitive to scale, and do not distinguish between tiny holes and large ones. Moreover, features such as pockets, valleys, and ridges---which are sometimes crucial in applications--- are not usually treated as topological features at all” [Bern et al., 1999, NSF workshop]. The number of connected components (in the same drainage system) can vary significantly between different maps, aerial photographs, and sensor data for the same area. It depends on characteristics such as human error, map resolution, sensor capabilities and parameters, obstacles (e.g. clouds), data processing methods. Thus, in general, topological invariants are not invariants for the conflation problem. Support for Key idea #2: Define features algebraically.

11. Algebraic Invariants Provide a mathematical model where: – There is a well established math language from abstract algebra – Polylines that represent geospatial features are introduced as formal algebraic systems – Theorems on computational efficiency are proven Provide conflating algorithms – That reduce computational complexity – That are robust (practically invariant)

12. Definitions Definition. A pair a =  A,  is called an algebraic system if A is a set of elements and  is a set of predicates {P} and operators {F} on A and on its Cartesian products, where P: A  A ...  A  [0,1] and F: A  A ...  A  A. Further definitions complete the specification of linear features (polylines) and the model. We omit the detailed mathematics, but will show an example to explain the concepts. Note, this abstract algebraic system is not a traditional system of linear algebraic equations

13. Abstract Polylines a 3 a1a1 a 1 = [v 0,v 1 ], a 2 = [v 1,v 2 ],..., a n = [v n-1,v n ]... a2a2 a4a4 L1 =100 o L2=200 o L1 < L2 L1`=110 o L2`=190 o L1`< L2` v1v1 v2v2 v0v0 a5a5 a b L1` L2' Invariant relation between angles, despite variation of angles Possible gap

14. Structure of feature a as a matrix of relations between angles AngleL1L2L3L4L5L6 L1100101 L210100 L31010 L4110 L511 L61 Li<Lj  P(Li,Lj)=0 A similar matrix can be constructed for relations between lengths of line segments or other task relevant characteristics of the feature. Key idea #5: Algebraic structure is expandable

15. Conflating algorithms Based on this algebraic model several algorithms may be designed to conflate images. One is to build matrices of relations that describe structures of two features a and b which are then searched for their largest common part. This is done by “sliding down” the diagonals of the matrices in the following way.

16. Illustrative matrix feature b and match with a (in grayed box) Angle L1L2L3L4L5L6L7L8L9L10 L11001011101 L2101000001 L310101001 L41101111 L5110000 L611100 L71000 L8111 L911 L101

17. Example Corona Landsat After rotation, scaling, and translation

18. Theorems are proven, such as Theorem. If the number of elements in linear features a and b equals n, then their maximum co-reference subsystem e can be found in O(n 3 ) matrix comparisons for the worst-case scenario that is equivalent to O(n 5 ) binary comparisons.

19. Useful, but... Comparison techniques are needed for polylines that have different sampling densities. Similarity measures have been developed to do this.

20. Similarity Measures To define measures we consider a polyline a with two end points p 1 =(x 1,y 1,z 1 ) and p 2 =(x 2,y 2,z 2 ). The recurrent function G(n) produces a set of simpler polylines that interpolate polyline a. Arguments of the function G are n = 2 k where k = 1,2,…, that is 2, 4, 8, 16, 32…

21. Structural interpolations of a polyline We introduced measures of structural similarities of features based on comparison of structural interpolations and generalized this to measuring structural similarities of entire images.

22. Conflation/registration process

23. Process illustration Two vector image data sets are considered. Image 1 consists of 1497 line segments. Image 2 consists of 407 line segments. Thanks to S. Sento and P. Brennan (Northrop Grumman) for providing data.

24. Input vector images “Are these two images of the same scene?”

25. The two images conflated and combined

26. Features in common

27. Features not in common Red-Image 1, blue –Image 2

28. Observations The number of polylines in common supports the conclusion that these are two images of the same scene. The number of polylines not in common illustrates the need for additional information to know whether these are the result of incomplete or faulty feature information, or changes in the scene between the times of acquisition.

29. Key idea #1: Attempt to match features Key idea #2: Define features algebraically Key idea #3: Match features structurally Key idea #4: Permit gaps in features High speeds and automation are possible in principal since registration is done using only a tiny fraction of the total image data. Summary: Key ideas

30. Summary: conclusion This algebraic relational approach is a promising way of conflating both raster and vector images – with no control points – with unknown scales – with unknown orientations Can be used to automatically search for specific objects of interest by predefining their abstracted linear shapes Can be used to automate the identification of locations that change in time More work ahead, e.g., to incorporate prior imagery knowledge. Test data welcome

31. Questions? Boris Kovalerchuk borisk@cwu.edu Bill Sumner sumner@geology.cwu. edu sumner@geology.cwu. edu

33. Super features The following slides illustrate super features

39. 8 538 1 897 4 284 -1 47.7897 30.7046 7.51 1 897 -1 284 2 47.7897649413 30.7049247065 0 1 897 5 -1 -1 47.7898 30.7051 7.51 1 897 6 -1 -1 47.7899 30.7053 7.51 1 897 7 -1 -1 47.79 30.7055 7.51 1 897 8 -1 -1 47.7902 30.7059 7.26 1 897 9 -1 -1 47.7905 30.7062 7.51 1 897 10 -1 -1 47.791 30.7065 7.5 1 897 11 -1 -1 47.7911 30.7066 7.38 1 897 12 -1 -1 47.7913 30.7067 7.25 1 897 13 -1 -1 47.7918 30.7069 7.25 1 897 -1 -1 -1 47.7919613336 30.7069968001 0 1 897 14 -1 -1 47.7923 30.7072 7.62 1 897 15 -1 -1 47.7928 30.7075 7.62 1 897 16 -1 -1 47.7932 30.7079 7.13 1 897 17 -1 -1 47.7937 30.7083 7.13 1 897 -1 -1 -1 47.793769313 30.7084617304 0 1 897 18 -1 -1 47.794 30.709 6.89 1 897 19 -1 -1 47.7942 30.7096 6.89