1. Graph-based Deformable Matching of 3D Line Segments with Application in Protein Fitting 12 1 HANG DOU 1, MATTHEW L BAKER 2, TAO JU 1 1 1 Washington University in St. Louis 1 2 Baylor College of Medicine 2

2. Deformation Applications of Feature Matching 2 Data alignment Co-segmentation Reconstruction

3. Our Application 3 Protein fitting Protein: building blocks of life forms. …

4. Protein 4 Atomic structureLow-res structure

5. Protein Fitting 5 Atomic structure (Cartoon Rendering) Low-res structure (Volume) Fitting result

6. Protein Fitting 6 Secondary structure (alpha helix)

7. Protein Fitting 7 Secondary structure (alpha helix)

8. Protein Fitting 8 Secondary structure (alpha helix)

9. Protein Fitting Our method Find feature (helix) correspondence Deform the protein guided by the helices correspondence 9

10. Protein Fitting Our method Find feature (helix) correspondence Deform the protein guided by the helices correspondence 10

11. Problem Statement Input: Undirected line segments |Source| <= |Target| Output: One to one correspondence Matching Criteria: Similar length As rigid as possible deformation 11 SourceTarget

12. Problem Statement Input: Undirected line segments |Source| <= |Target| Output: One to one correspondence Matching Criteria: Similar length As rigid as possible deformation 12 SourceTarget

13. Problem Statement Input: Undirected line segments |Source| <= |Target| Output: One to one correspondence Matching Criteria: Similar length As rigid as possible deformation 13 SourceTarget

14. Previous Work Matching points: Lots of work [Fitzgibbon 2003], [Angulov 2004], [Leordeanu 2005], [Chang 2009], [Chertok 2010], [Duchenne 2011], [TAM 2014], …. 14

15. Previous Work Matching line segments: very few work [Abeysinghe 2010] 15 Target Source

16. Previous Work Matching line segments: very few work [Abeysinghe 2010] Need line segments forming rigid body clusters 16 Target Source

17. Previous Work Matching line segments: very few work [Abeysinghe 2010] Need line segments forming rigid body clusters Based on a clique-finding algorithm 17 Target Source Matching Result

18. Our Work A novel method for matching 3D line segments Allowing fully non-rigid deformations Technical contributions Formulating the problem as graph matching Improved graph matching using continuous relaxation 18

19. Our Work A novel method for matching 3D line segments Allowing fully non-rigid deformations Technical contributions Formulating the problem as graph matching Improved graph matching using continuous relaxation 19

20. Background: Graph Matching 20 Source Target s2s2 s1s1 t2t2 t1t1 f node f edge

21. Our Graph Construction Naïve method Graph nodes: One undirected segment one node 21 Source Target s3s3 s2s2 s1s1 t2t2 t1t1 t4t4 t3t3

22. Our Graph Construction Naïve method Graph nodes: One undirected segment one node Node pair affinity Length similarity 22 Source Target s3s3 s2s2 s1s1 t2t2 t1t1 t4t4 t3t3

23. Our Graph Construction Naïve method Graph nodes: One undirected segment one node Node pair affinity Length similarity 23 Source Target s3s3 s2s2 s1s1 t2t2 t1t1 t4t4 t3t3

24. Our Graph Construction Naïve method Graph nodes: One segment one node Node pair affinity Length similarity Edge pair affinity Rigidity in deformation 24 Source Target s3s3 s2s2 s1s1 t2t2 t1t1 t4t4 t3t3

25. Our Graph Construction Naïve method Graph nodes: One undirected segment one node Node pair affinity Length similarity Edge pair affinity Rigidity in deformation 25 Source Target s3s3 s2s2 s1s1 t2t2 t1t1 t4t4 t3t3

26. Our Graph Construction Issue of ambiguity: 26 Matching Source Matching Result 1 Matching Result 2

27. Our Graph Construction Issue of ambiguity: 27 Matching Source Matching Result 1 Matching Result 2

28. Our Graph Construction Issue of ambiguity: 28 Matching Source Matching Result 1 Matching Result 2

29. Our Graph Construction Issue of ambiguity: 29 Matching Source Matching Result 1 Matching Result 2

30. Our Graph Construction Improved graph construction Graph nodes: Source One undirected segment one node Target One undirected segment two nodes 30 Source Target S3S3 S2S2 S1S1 T1T1 T2T2 T3T3 T4T4

31. Our Graph Construction Improved graph construction Graph nodes: Source One undirected segment one node Target One undirected segment two nodes Node Affinity Length similarity 31 Source Target S3S3 S1S1 S2S2 T1T1 T3T3 T4T4 T2T2

32. Our Graph Construction Improved graph construction Graph nodes: Source One undirected segment one node Target One undirected segment two nodes Node Affinity Length similarity Edge Affinity Rigidity in deformation 32 Source Target S3S3 S2S2 S1S1 T1T1 T2T2 T3T3 T4T4

33. Our Graph Construction Improved graph construction Graph nodes: Source One undirected segment one node Target One undirected segment two nodes Node Affinity Length similarity Edge Affinity Rigidity in deformation 33 Source Target S3S3 S2S2 S1S1 T1T1 T2T2 T3T3 T4T4

34. Rigidity Measure 34

35. Our Graph Construction No more ambiguity 35 Matching Source Matching Result

36. Our Work A novel method for matching 3D line segments Allowing fully non-rigid deformations Technical contributions Formulating the problem as graph matching Improved graph matching using continuous relaxation 36

37. Our Work A novel method for matching 3D line segments Allowing fully non-rigid deformations Technical contributions Formulating the problem as graph matching Improved graph matching using continuous relaxation 37

38. Solve Graph Matching (review) Formulate graph matching as quadratic assignment Encode graph affinities in a matrix M 38 s 1,t 1 Affinity Matrix (M) s 3,t 3 …. s3s3 s2s2 s1s1 t2t2 t1t1 t4t4 t3t3 SourceTarget

39. Solve Graph Matching (review) Formulate graph matching as quadratic assignment Encode graph affinities in a matrix M 39 s 1,t 1 Affinity Matrix (M) s 3,t 3 …. s3s3 s2s2 s1s1 t2t2 t1t1 t4t4 t3t3 SourceTarget

40. Solve Graph Matching (review) Formulate graph matching as quadratic assignment Encode graph affinities in a matrix M Matching goal: 40 s3s3 s2s2 s1s1 t2t2 t1t1 t4t4 t3t3 SourceTarget s 1,t 1 Affinity Matrix (M) s 3,t 3 ….

41. Solve Graph Matching (review) Formulate graph matching as quadratic assignment Encode graph affinities in a matrix M Matching goal: Find assignment vector x  {0,1} |S|x|T| Maximize: x T M x (equation 1) Subject to mapping constraints. 41 Mapping constraint Integer constraint

42. Solve Graph Matching (review) Solve quadratic assignment Combinatorial optimization Usually time consuming. Continuous relaxation Compute confidence vector X’  R |S|x|T| without some or all the constraints X’(a) is the confident of the match a (S i -> T j ) Binarize X’ to obtain assignment vector X  {0,1} |S|x|T| with all constraints X(a) is 1 if a is an accepted match and 0 otherwise. Widely used greedy approach [Leordeanu 2005]: sequentially pick matches with descending confidence, while avoiding conflict based on mapping constraints. 42

43. Binarize Confidence Vector (Review) 43 Target Source 0.4 0 Confidence Vector

44. Binarize Confidence Vector (Review) 44 0.4 0 Confidence Vector Target Source

45. Binarize Confidence Vector (Review) 45 0.66 0 Target Source 0.4 0 Confidence Vector

46. Binarize Confidence Vector (Review) 46 0.66 0 Target Source 0.4 0 Confidence Vector

47. Binarize Continuous Solution (Review) 47 0.66 0 Target Source 0.4 0 Confidence Vector

48. Binarize Confidence Vector (Review) 48 Correct Matches Target Source 0.66 0 0.4 0 Confidence Vector

49. Our Continuous Relaxation Key idea Only takes the assignments with high confidence Formulate a smaller matching problem, constrained by the chosen assignments Iterate until we find all the matches Compute confidence vector (with any chosen continuous method) Pick matches whose relative confidence (w.r.t. the highest confidence) value is above a threshold r Reconstruct the (smaller) affinity matrix with remaining matches Edge affinity of already picked segments is added to the diagonal of the new matrix 49

50. 0.66 0 Confidence vector in 1 st iteration Our Continuous Relaxation 50 Target Source r = 1

51. Our Continuous Relaxation 51 0.66 0 r = 1 Confidence vector in 1 st iteration Target Source

52. Our Continuous Relaxation 52 0.95 0 r = 1 Confidence vector in 2 nd iteration Target Source

53. Our Continuous Relaxation 53 0.95 0 r = 1 Confidence vector in 2 nd iteration Target Source

54. Our Continuous Relaxation 54 0.95 0 r = 1 Confidence vector in 3 rd iteration Target Source

55. Our Continuous Relaxation 55 0.95 0 r = 1 Confidence vector in 3 rd iteration Target Source

56. Our Continuous Relaxation 56 0.71 0 r = 1 Confidence vector in 4 th iteration Target Source

57. Our Continuous Relaxation 57 0.71 0 r = 1 Confidence vector in 4 th iteration Target Source

58. Our Continuous Relaxation 58 r = 1 Confidence vector in 5 th iteration Target Source 0.99 0

59. Our Continuous Relaxation 59 0.99 0 r = 1 Confidence vector in 5 th iteration Target Source

60. 0.99 0 Our Continuous Relaxation 60 r = 1 Confidence vector in 6 th iteration Target Source

61. 0.99 0 Our Continuous Relaxation 61 r = 1 Confidence vector in 6 th iteration Target Source

62. Our Continuous Relaxation 62 Correct Matching Result Target Source

63. Experiment Result Synthetic data and authentic data Synthetic data: generated by thin plate spline (TPS) deformation. Authentic data: 16 pairs of proteins from Protein Data Bank. Accuracy and function score Matching accuracy: ratio of correct matches over all matches. Total affinity ratio: Function score (x T M x) ratio of our method over a benchmark method [Leordeanu 2005]. 63

64. Experiment Result Synthetic data Pick one protein as the matching source Deform the source line segments at different level of normalized bending energies as matching targets 64 Source Bend Energy = 0.2 Bend Energy = 0.7

65. Experiment Result 65 Synthetic data Leordeanu[2005] Our, r = 0.6 Our, r = 0.3 Our, r = 1.0 Bending Energy Total Affinity Ratio

66. Experiment Result 66 Synthetic data Bending Energy Matching Accuracy Leordeanu[2005] Our, r = 0.3Our, r = 0.6 Our, r = 1.0 Abeysinghe[2010]

67. Experiment Result 67 Synthetic data Bending Energy Matching Accuracy Leordeanu[2005] Our, r = 0.3Our, r = 0.6 Our, r = 1.0 Abeysinghe[2010]

68. Experiment Result Authentic data: 68 1sx4-A 1ss8-A Source Target(Our) Target(Leordeanu) Target(Abeysinghe)

69. Experiment Result Authentic data: 69 Protein Matching Accuracy Leordeanu Our, r = 1.0, 0.6, 0.3 Abeysinghe

70. Conclusion 70 An algorithm for finding correspondence between two sets of 3D line segments Allowing fully non-rigid deformations Technical contributions Formulating the problem as graph matching Solve matching problem using a iterative continuous-discrete paradigm Validation Test on both synthetic and authentic data

71. Questions? 71