Graph-based Deformable Matching of 3D Line Segments with Application in Protein Fitting 12 1 HANG DOU 1, MATTHEW L BAKER 2, TAO JU Washington University in St. Louis 1 2 Baylor College of Medicine 2
Deformation Applications of Feature Matching 2 Data alignment Co-segmentation Reconstruction
Our Application 3 Protein fitting Protein: building blocks of life forms. …
Protein 4 Atomic structureLow-res structure
Protein Fitting 5 Atomic structure (Cartoon Rendering) Low-res structure (Volume) Fitting result
Protein Fitting 6 Secondary structure (alpha helix)
Protein Fitting 7 Secondary structure (alpha helix)
Protein Fitting 8 Secondary structure (alpha helix)
Protein Fitting Our method Find feature (helix) correspondence Deform the protein guided by the helices correspondence 9
Protein Fitting Our method Find feature (helix) correspondence Deform the protein guided by the helices correspondence 10
Problem Statement Input: Undirected line segments |Source| <= |Target| Output: One to one correspondence Matching Criteria: Similar length As rigid as possible deformation 11 SourceTarget
Problem Statement Input: Undirected line segments |Source| <= |Target| Output: One to one correspondence Matching Criteria: Similar length As rigid as possible deformation 12 SourceTarget
Problem Statement Input: Undirected line segments |Source| <= |Target| Output: One to one correspondence Matching Criteria: Similar length As rigid as possible deformation 13 SourceTarget
Previous Work Matching points: Lots of work [Fitzgibbon 2003], [Angulov 2004], [Leordeanu 2005], [Chang 2009], [Chertok 2010], [Duchenne 2011], [TAM 2014], …. 14
Previous Work Matching line segments: very few work [Abeysinghe 2010] 15 Target Source
Previous Work Matching line segments: very few work [Abeysinghe 2010] Need line segments forming rigid body clusters 16 Target Source
Previous Work Matching line segments: very few work [Abeysinghe 2010] Need line segments forming rigid body clusters Based on a clique-finding algorithm 17 Target Source Matching Result
Our Work A novel method for matching 3D line segments Allowing fully non-rigid deformations Technical contributions Formulating the problem as graph matching Improved graph matching using continuous relaxation 18
Our Work A novel method for matching 3D line segments Allowing fully non-rigid deformations Technical contributions Formulating the problem as graph matching Improved graph matching using continuous relaxation 19
Background: Graph Matching 20 Source Target s2s2 s1s1 t2t2 t1t1 f node f edge
Our Graph Construction Naïve method Graph nodes: One undirected segment one node 21 Source Target s3s3 s2s2 s1s1 t2t2 t1t1 t4t4 t3t3
Our Graph Construction Naïve method Graph nodes: One undirected segment one node Node pair affinity Length similarity 22 Source Target s3s3 s2s2 s1s1 t2t2 t1t1 t4t4 t3t3
Our Graph Construction Naïve method Graph nodes: One undirected segment one node Node pair affinity Length similarity 23 Source Target s3s3 s2s2 s1s1 t2t2 t1t1 t4t4 t3t3
Our Graph Construction Naïve method Graph nodes: One segment one node Node pair affinity Length similarity Edge pair affinity Rigidity in deformation 24 Source Target s3s3 s2s2 s1s1 t2t2 t1t1 t4t4 t3t3
Our Graph Construction Naïve method Graph nodes: One undirected segment one node Node pair affinity Length similarity Edge pair affinity Rigidity in deformation 25 Source Target s3s3 s2s2 s1s1 t2t2 t1t1 t4t4 t3t3
Our Graph Construction Issue of ambiguity: 26 Matching Source Matching Result 1 Matching Result 2
Our Graph Construction Issue of ambiguity: 27 Matching Source Matching Result 1 Matching Result 2
Our Graph Construction Issue of ambiguity: 28 Matching Source Matching Result 1 Matching Result 2
Our Graph Construction Issue of ambiguity: 29 Matching Source Matching Result 1 Matching Result 2
Our Graph Construction Improved graph construction Graph nodes: Source One undirected segment one node Target One undirected segment two nodes 30 Source Target S3S3 S2S2 S1S1 T1T1 T2T2 T3T3 T4T4
Our Graph Construction Improved graph construction Graph nodes: Source One undirected segment one node Target One undirected segment two nodes Node Affinity Length similarity 31 Source Target S3S3 S1S1 S2S2 T1T1 T3T3 T4T4 T2T2
Our Graph Construction Improved graph construction Graph nodes: Source One undirected segment one node Target One undirected segment two nodes Node Affinity Length similarity Edge Affinity Rigidity in deformation 32 Source Target S3S3 S2S2 S1S1 T1T1 T2T2 T3T3 T4T4
Our Graph Construction Improved graph construction Graph nodes: Source One undirected segment one node Target One undirected segment two nodes Node Affinity Length similarity Edge Affinity Rigidity in deformation 33 Source Target S3S3 S2S2 S1S1 T1T1 T2T2 T3T3 T4T4
Rigidity Measure 34
Our Graph Construction No more ambiguity 35 Matching Source Matching Result
Our Work A novel method for matching 3D line segments Allowing fully non-rigid deformations Technical contributions Formulating the problem as graph matching Improved graph matching using continuous relaxation 36
Our Work A novel method for matching 3D line segments Allowing fully non-rigid deformations Technical contributions Formulating the problem as graph matching Improved graph matching using continuous relaxation 37
Solve Graph Matching (review) Formulate graph matching as quadratic assignment Encode graph affinities in a matrix M 38 s 1,t 1 Affinity Matrix (M) s 3,t 3 …. s3s3 s2s2 s1s1 t2t2 t1t1 t4t4 t3t3 SourceTarget
Solve Graph Matching (review) Formulate graph matching as quadratic assignment Encode graph affinities in a matrix M 39 s 1,t 1 Affinity Matrix (M) s 3,t 3 …. s3s3 s2s2 s1s1 t2t2 t1t1 t4t4 t3t3 SourceTarget
Solve Graph Matching (review) Formulate graph matching as quadratic assignment Encode graph affinities in a matrix M Matching goal: 40 s3s3 s2s2 s1s1 t2t2 t1t1 t4t4 t3t3 SourceTarget s 1,t 1 Affinity Matrix (M) s 3,t 3 ….
Solve Graph Matching (review) Formulate graph matching as quadratic assignment Encode graph affinities in a matrix M Matching goal: Find assignment vector x {0,1} |S|x|T| Maximize: x T M x (equation 1) Subject to mapping constraints. 41 Mapping constraint Integer constraint
Solve Graph Matching (review) Solve quadratic assignment Combinatorial optimization Usually time consuming. Continuous relaxation Compute confidence vector X’ R |S|x|T| without some or all the constraints X’(a) is the confident of the match a (S i -> T j ) Binarize X’ to obtain assignment vector X {0,1} |S|x|T| with all constraints X(a) is 1 if a is an accepted match and 0 otherwise. Widely used greedy approach [Leordeanu 2005]: sequentially pick matches with descending confidence, while avoiding conflict based on mapping constraints. 42
Binarize Confidence Vector (Review) 43 Target Source Confidence Vector
Binarize Confidence Vector (Review) Confidence Vector Target Source
Binarize Confidence Vector (Review) Target Source Confidence Vector
Binarize Confidence Vector (Review) Target Source Confidence Vector
Binarize Continuous Solution (Review) Target Source Confidence Vector
Binarize Confidence Vector (Review) 48 Correct Matches Target Source Confidence Vector
Our Continuous Relaxation Key idea Only takes the assignments with high confidence Formulate a smaller matching problem, constrained by the chosen assignments Iterate until we find all the matches Compute confidence vector (with any chosen continuous method) Pick matches whose relative confidence (w.r.t. the highest confidence) value is above a threshold r Reconstruct the (smaller) affinity matrix with remaining matches Edge affinity of already picked segments is added to the diagonal of the new matrix 49
Confidence vector in 1 st iteration Our Continuous Relaxation 50 Target Source r = 1
Our Continuous Relaxation r = 1 Confidence vector in 1 st iteration Target Source
Our Continuous Relaxation r = 1 Confidence vector in 2 nd iteration Target Source
Our Continuous Relaxation r = 1 Confidence vector in 2 nd iteration Target Source
Our Continuous Relaxation r = 1 Confidence vector in 3 rd iteration Target Source
Our Continuous Relaxation r = 1 Confidence vector in 3 rd iteration Target Source
Our Continuous Relaxation r = 1 Confidence vector in 4 th iteration Target Source
Our Continuous Relaxation r = 1 Confidence vector in 4 th iteration Target Source
Our Continuous Relaxation 58 r = 1 Confidence vector in 5 th iteration Target Source
Our Continuous Relaxation r = 1 Confidence vector in 5 th iteration Target Source
Our Continuous Relaxation 60 r = 1 Confidence vector in 6 th iteration Target Source
Our Continuous Relaxation 61 r = 1 Confidence vector in 6 th iteration Target Source
Our Continuous Relaxation 62 Correct Matching Result Target Source
Experiment Result Synthetic data and authentic data Synthetic data: generated by thin plate spline (TPS) deformation. Authentic data: 16 pairs of proteins from Protein Data Bank. Accuracy and function score Matching accuracy: ratio of correct matches over all matches. Total affinity ratio: Function score (x T M x) ratio of our method over a benchmark method [Leordeanu 2005]. 63
Experiment Result Synthetic data Pick one protein as the matching source Deform the source line segments at different level of normalized bending energies as matching targets 64 Source Bend Energy = 0.2 Bend Energy = 0.7
Experiment Result 65 Synthetic data Leordeanu[2005] Our, r = 0.6 Our, r = 0.3 Our, r = 1.0 Bending Energy Total Affinity Ratio
Experiment Result 66 Synthetic data Bending Energy Matching Accuracy Leordeanu[2005] Our, r = 0.3Our, r = 0.6 Our, r = 1.0 Abeysinghe[2010]
Experiment Result 67 Synthetic data Bending Energy Matching Accuracy Leordeanu[2005] Our, r = 0.3Our, r = 0.6 Our, r = 1.0 Abeysinghe[2010]
Experiment Result Authentic data: 68 1sx4-A 1ss8-A Source Target(Our) Target(Leordeanu) Target(Abeysinghe)
Experiment Result Authentic data: 69 Protein Matching Accuracy Leordeanu Our, r = 1.0, 0.6, 0.3 Abeysinghe
Conclusion 70 An algorithm for finding correspondence between two sets of 3D line segments Allowing fully non-rigid deformations Technical contributions Formulating the problem as graph matching Solve matching problem using a iterative continuous-discrete paradigm Validation Test on both synthetic and authentic data
Questions? 71