2. Turn angle function and elastic time series matching Turn angle function and elastic time series matching Presented by: Wang, Xinzhen Advisor: Dr. Longin Jan Latecki

3. Agenda Introductio n Introductio n Turn angle function Turn angle function Time series matching Time series matching Conclusio n Conclusio n Future work Future work

4. Introduction Data set: 1400 images (70 classes * 20 objects) Data set: 1400 images (70 classes * 20 objects) 1400 land mark sequences Data Preprocessing

5. Introduction An example: Original image Time series Landmark sequence

6. Introduction Time sequence matching Time sequence matching Shape Matching Sequence Matching

7. Introduction Problem definition: Problem definition: Given a query q and a distance function d, find m nearest neighbors of q by calculating the distance using d. In our case, m is equal to 40. Distance function d: a = (x 1,x 2,... x n )b = (y 1,y 2,... y n )

8. Tangent space representation Shape description in tangent space Shape description in tangent space Simplified contour Step function presentation Proble m

9. Turn angle function Modification to tangent space rep. Modification to tangent space rep. Rotation (turning angle) Rotation (turning angle) Scaling (normalization) Scaling (normalization) Starting point (double length) Starting point (double length)

10. Landmark sequence From time series to land mark sequences From time series to land mark sequences Step I : compare each point with its left neighbor Step I : compare each point with its left neighbor Step II : compare each point with its left and right neighbors Step II : compare each point with its left and right neighbors Disadvantages (ex: loss of information) Disadvantages (ex: loss of information)loss of informationloss of information

11. Landmark sequence matching Step I : Align the highest peak of the query with every peak of the object, and then align other peaks and valleys of the query accordingly. Step I : Align the highest peak of the query with every peak of the object, and then align other peaks and valleys of the query accordingly. Step II: Calculate the Euclidean distance between the peaks/valleys of query and object. As we move query along the object, we have Euclidean distance for each alignment. A smallest Euclidean distance identifies the optimized alignment. Step II: Calculate the Euclidean distance between the peaks/valleys of query and object. As we move query along the object, we have Euclidean distance for each alignment. A smallest Euclidean distance identifies the optimized alignment. Step III: In the optimized alignment, we introduce a penalty distance if either query or object has extra peaks or valleys. Step III: In the optimized alignment, we introduce a penalty distance if either query or object has extra peaks or valleys.

12. An example: Aligning query peaks and valleys with object (one optimized alignment) Landmark sequence matching Query’s peaks and valleys Object sequence (doubled)

13. Landmark sequence matching Another example : query has extra peaks and valleys Back

14. Penalty distance Penalty distance Landmark sequence matching if the query has extra peaks/valleys, a penalty distance is added to the Euclidean distance between the query and object. The penalty distance is calculated by the sum of Euclidean distances between the unmatched peaks/valleys to the closest matched peaks/valleys in the query. See exampleSee example If instead the object has extra peaks/valleys, a penalty distance is calculated by the sum of Euclidean distances between the unmatched peaks/valleys to the closest matched peaks/valleys in the object.

15. Query: The first object in the 1 st class Query: The first object in the 1 st classobject Some experimental results Search for 40 nearest neighbors in the whole dataset. Search for 40 nearest neighbors in the whole dataset. The top 40 matches found. The top 40 matches found.40 matches 40 matches Retrieval Rate : 100% Retrieval Rate : 100% Retrieval Rate Retrieval Rate

16. Back

17. Definition Retrieval Rate: Since we have the prior knowledge about those objects within the same class as the query object, we can define the retrieval rate of matching as : RetrievalRate = N / 20 ( N: number of objects in the top 40 matches that belong to the same class as the query object)

18. Part Matching --- In a primitive stage We manually select a significant part of an object, for example the leaves of an apple, and proceed sub- sequence matching and retrieval We manually select a significant part of an object, for example the leaves of an apple, and proceed sub- sequence matching and retrievalan objectan object Since our query part has only three peaks and three valleys, we define them as LeftMostPeak/Valley, MiddlePeak/Valley, RightMostPeak/Valley. See here. Since our query part has only three peaks and three valleys, we define them as LeftMostPeak/Valley, MiddlePeak/Valley, RightMostPeak/Valley. See here.See hereSee here

19. Back LeftMostPeak/Valley RightMostPeak/Valley MiddlePeak/Valley

20. Part Matching The Matching Scope in object The Matching Scope in object The closest peak/valley to the LeftMostPeak/Valley The closest peak/valley to the LeftMostPeak/Valley

21. Step I : Calculate the Euclidean distance between the peaks/valleys of the query part and object part. Only peaks/valleys fall between the matching scope in the object are considered for matching. Step I : Calculate the Euclidean distance between the peaks/valleys of the query part and object part. Only peaks/valleys fall between the matching scope in the object are considered for matching. Step II: As we move query part along the object, we have Euclidean distance for each alignment. A smallest Euclidean distance identifies the optimized alignment. Step II: As we move query part along the object, we have Euclidean distance for each alignment. A smallest Euclidean distance identifies the optimized alignment. Step III: In the optimized alignment, we introduce a penalty distance if either query part or object part has extra peaks or valleys. The penalty distance calculation would be the same as previous defined. Step III: In the optimized alignment, we introduce a penalty distance if either query part or object part has extra peaks or valleys. The penalty distance calculation would be the same as previous defined. Part Matching

22. Query part: The first object in the 1 st class Query part: The first object in the 1 st class Some experimental results Search for 40 nearest neighbors in the whole dataset. Search for 40 nearest neighbors in the whole dataset. Retrieval Rate : 60% Retrieval Rate : 60% False positives False positives False positives False positives

23. False Positives Obj626 Come pretty early in the 40 matches!

24. False Positives Looks like the leaf of an apple?

25. Conclusion It ’ s feasible to transform image contour data to time sequence. Landmark sequence can capture the important features of time series. Matching based on it is applicable and promising. Part Matching brings good result by submitting very limited query information.

26. Future work Order of Matching (Eliminate crossover) Order of Matching (Eliminate crossover)crossover Combination of global matching with part matching. Combination of global matching with part matching. Apply the technique on the whole dataset. Apply the technique on the whole dataset.