Constraint Reasoning and Kernel Clustering for Pattern Decomposition With Scaling Ronan LeBrasComputer Science Theodoros DamoulasComputer Science Ashish SabharwalComputer Science Carla P. GomesComputer Science John M. GregoireMaterials Science / Physics Bruce van DoverMaterials Science / Physics Sept 15, 2011CP’11

Motivation Ronan Le Bras - CP’11, Sept 15, Cornell Fuel Cell Institute Mission: develop new materials for fuel cells. An Electrocatalyst must: 1) Be electronically conducting 2) Facilitate both reactions Platinum is the best known metal to fulfill that role, but: 1) The reaction rate is still considered slow (causing energy loss) 2) Platinum is fairly costly, intolerant to fuel contaminants, and has a short lifetime. Goal: Find an intermetallic compound that is a better catalyst than Pt.

Motivation 3 Recipe for finding alternatives to Platinum 1)In a vacuum chamber, place a silicon wafer. 2)Add three metals. 3)Mix until smooth, using three sputter guns. 4)Bake for 2 hours at 650ºC Ta Rh Pd (38% Ta, 45% Rh, 17% Pd) Deliberately inhomogeneous composition on Si wafer Atoms are intimately mixed [Source: Pyrotope, Sebastien Merkel] Ronan Le Bras - CP’11, Sept 15, 2011

Motivation 4 Identifying crystal structure using X-Ray Diffraction at CHESS XRD pattern characterizes the underlying crystal fairly well Expensive experimentations: Bruce van Dover’s research team has access to the facility one week every year. Ta Rh Pd (38% Ta, 45% Rh, 17% Pd) [Source: Pyrotope, Sebastien Merkel] Ronan Le Bras - CP’11, Sept 15, 2011

Motivation 5 Ta Rh Pd α β δ γ Ronan Le Bras - CP’11, Sept 15, 2011

Motivation 6 Ta Rh Pd α β α+βα+β δ γ γ+δγ+δ Ronan Le Bras - CP’11, Sept 15, 2011

Motivation 7 Ta Rh Pd α β δ γ Ronan Le Bras - CP’11, Sept 15, 2011

Motivation 8 Ta Rh Pd α β δ γ α+βα+β Ronan Le Bras - CP’11, Sept 15, 2011 PiPi PjPj

Motivation 9 Ta Rh Pd α β α+βα+β δ γ Ronan Le Bras - CP’11, Sept 15, 2011 PiPi PkPk PjPj

Motivation Fe Al Si INPUT: pure phase region Fe Al Si m phase regions  k pure regions  m-k mixed regions XRD pattern characterizing pure phases Mixed phase region OUTPUT: Additional Physical characteristics:  Peaks shift by  15% within a region  Phase Connectivity  Mixtures of  3 phases  Small peaks might be discriminative  Peak locations matter, more than peak intensities 10Ronan Le Bras - CP’11, Sept 15, 2011

Motivation 11 Ta Rh Pd Figure 1: Phase regions of Ta-Rh-PdFigure 2: Fluorescence activity of Ta-Rh-Pd Ronan Le Bras - CP’11, Sept 15, 2011

Outline 12 Motivation Problem Definition Abstraction Hardness CP Model Kernel-based Clustering Bridging CP and Machine learning Conclusion and Future work Ronan Le Bras - CP’11, Sept 15, 2011

Problem Abstraction: Pattern Decomposition with Scaling 13 Input Output Ronan Le Bras - CP’11, Sept 15, 2011

Problem Abstraction: Pattern Decomposition with Scaling 14 Input Output v1v1 v2v2 v3v3 v4v4 v5v5 Ronan Le Bras - CP’11, Sept 15, 2011

Problem Abstraction: Pattern Decomposition with Scaling 15 Input Output v1v1 v2v2 v3v3 v4v4 v5v5 P1P1 P2P2 P3P3 P4P4 P5P5 Ronan Le Bras - CP’11, Sept 15, 2011

Problem Abstraction: Pattern Decomposition with Scaling 16 Input Output v1v1 v2v2 v3v3 v4v4 v5v5 P1P1 P2P2 P3P3 P4P4 P5P5 M= K = 2, δ = 1.5 Ronan Le Bras - CP’11, Sept 15, 2011

Problem Abstraction: Pattern Decomposition with Scaling 17 Input Output v1v1 v2v2 v3v3 v4v4 v5v5 P1P1 P2P2 P3P3 P4P4 P5P5 M= K = 2, δ = 1.5 B2B2 B1B1 Ronan Le Bras - CP’11, Sept 15, 2011

Problem Abstraction: Pattern Decomposition with Scaling 18 Input Output v1v1 v2v2 v3v3 v4v4 v5v5 P1P1 P2P2 P3P3 P4P4 P5P5 M= K = 2, δ = 1.5 B2B2 B1B1 s 11 =1.00 s 12 =0.95 s 13 =0.88 s 14 =0.84 s 15 =0 s 21 =0.68 s 22 =0.78 s 23 =0.85 s 24 =0.96 s 25 =1.00 Ronan Le Bras - CP’11, Sept 15, 2011

Problem Hardness 19 Assumptions: Each B k appears by itself in some v i / No experimental noise The problem can be solved* in polynomial time. Assumption: No experimental noise The problem becomes NP-hard (reduction from the “Set Basis” problem) Assumption used in this work: Experimental noise in the form of missing elements in P i v1v1 v2v2 v3v3 v4v4 v5v5 P1P1 P2P2 P3P3 P4P4 P5P5 M= K = 2, δ = 1.5 B2B2 B1B1 s 11 =1.00 s 12 =0.95 s 13 =0.88 s 14 =0.84 s 15 =0 s 21 =0.68 s 22 =0.78 s 23 =0.85 s 24 =0.96 s 25 =1.00 Ronan Le Bras - CP’11, Sept 15, 2011 P0=P0=

Outline 20 Motivation Problem Definition CP Model Model Experimental Results Kernel-based Clustering Bridging CP and Machine learning Conclusion and Future work Ronan Le Bras - CP’11, Sept 15, 2011

CP Model 21 Advantage: Captures physical properties and relies on peak location rather than height. Drawback: Does not scale to realistic instances; poor propagation if experimental noise. Ronan Le Bras - CP’11, Sept 15, 2011

CP Model (continued) 22 Advantage: Captures physical properties and relies on peak location rather than height. Drawback: Does not scale to realistic instances; poor propagation if experimental noise. Advantage: Captures physical properties and relies on peak location rather than height. Drawback: Does not scale to realistic instances; poor propagation if experimental noise. Ronan Le Bras - CP’11, Sept 15, 2011

CP Model – Experimental Results 23 Advantage: Captures physical properties and relies on peak location rather than height. Drawback: Does not scale to realistic instances; poor propagation if experimental noise. For realistic instances, K’ = 6 and N ≈ Ronan Le Bras - CP’11, Sept 15, 2011

Outline 24 Motivation Problem Definition CP Model Kernel-based Clustering Bridging CP and Machine learning Conclusion and Future work Ronan Le Bras - CP’11, Sept 15, 2011

Kernel-based Clustering Advantage: Captures physical properties and relies on peak location rather than height. Drawback: Does not scale to realistic instances; poor propagation if experimental noise. 25 Set of features: X = Similarity matrices: [X.X T ] Method: K-means on Dynamic Time-Warping kernel Goal: Select groups of samples that belong to the same phase region to feed the CP model, in order to extract the underlying phases of these sub-problems. Better approach: take shifts into account Red: similar Blue: dissimilar ++   ++   ++   Ronan Le Bras - CP’11, Sept 15, 2011

Bridging CP and ML 26 Goal: a robust, physically meaningful, scalable, automated solution method that combines: Similarity “Kernels” & Clustering Machine Learning for a “global data-driven view” Underlying Physics A language for Constraints enforcing “local details” Constraint Programming model Ronan Le Bras - CP’11, Sept 15, 2011

Bridging Constraint Reasoning and Machine Learning: Overview of the Methodology 27 Fe Al Si INPUT: Peak detection Machine Learning: Kernel methods, Dynamic Time Wrapping Machine Learning: Partial “Clustering” Fe Al Si CP Model & Solver on sub-problems ,  + ,   only  only Full CP Model guided by partial solutions OUTPUT Fix errors in data Ronan Le Bras - CP’11, Sept 15, 2011

Experimental Validation 28 Al-Li-Fe instance with 6 phases: Ground truth (known) Our CP + ML hybrid approach is much more robust Previous work (NMF) violates many physical requirements

Conclusion 29 Hybrid approach to clustering under constraints  More robust than data-driven “global” ML approaches  More scalable than a pure CP model “locally” enforcing constraints An exciting application in close collaboration with physicists  Best inference out of expensive experiments  Towards the design of better fuel cell technology Ronan Le Bras - CP’11, Sept 15, 2011

Future work Ongoing work: Spatial Clustering, to further enhance cluster quality Bayesian Approach, to better exploit prior knowledge about local smoothness and available inorganic libraries Active learning: Where to sample next, assuming we can interfere with the sampling process? When to stop sampling if sufficient information has been obtained? Correlating catalytic properties across many thin-films: In order to understand the underlying physical mechanism of catalysis and to find promising intermettalic compounds 30Ronan Le Bras - CP’11, Sept 15, 2011

The end Thank you! 31Ronan Le Bras - CP’11, Sept 15, 2011

Extra slides 32Ronan Le Bras - CP’11, Sept 15, 2011

Experimental Sample 33 Example on Al-Li-Fe diagram: Ronan Le Bras - CP’11, Sept 15, 2011

Applications with similar structure 34 Flight Calls / Bird conservation Identifying bird population from sound recordings at night. Analogy: basis pattern = species samples = recordings physical constraints = spatial constraints, species and season specificities… Fire Detection Detecting/Locating fires. Analogy: basis pattern = warmth sources samples = temperature recordings physical constraints = gradient of temperatures, material properties… Ronan Le Bras - CP’11, Sept 15, 2011

Previous Work 1: Cluster Analysis (Long et al., 2007) 35 x i = (Feature vector)(Pearson correlation coefficients)(Distance matrix) (PCA – 3 dimensional approx)(Hierarchical Agglomerative Clustering) Drawback: Requires sampling of pure phases, detects phase regions (not phases), overlooks peak shifts, may violate physical constraints (phase continuity, etc.). Ronan Le Bras - CP’11, Sept 15, 2011

Previous Work 2: NMF (Long et al., 2009) 36 x i = (Feature vector)(Linear positive combination (A) of basis patterns (S)) (Minimizing squared Frobenius norm X = A.S + EMin ║E║ Drawback: Overlooks peak shifts (linear combination only), may violate physical constraints (phase continuity, etc.). Ronan Le Bras - CP’11, Sept 15, 2011