Multiplexed Fluorescence Unmixing Marina Alterman, Yoav Schechner Aryeh Weiss Technion, Israel Bar-Ilan, Israel

2 Natural Linear Mixing Raskar et al ImageJ image sample collection. c ci i c i

3 Natural Linear Mixing ? ImageJ image sample collection. c Raskar et al ci i How do you measure i ? c i

Single Source Excitation Multiplexed Excitation 4 demultiplex i1i1 i2i2 i3i3 1 a3a3 2 3 Beam combiner a1a1 1 2 a2a a = i a i a i

Why Multiplexing? + noise SNR Trivial Measurements SNR Multiplexed Measurements Same acquisition time 5 Intensity vector i

Multiplexing - Look closer 6 Xc i i – single source intensities η - noise estimation acquisition Minimum  W=? Estimate c not i

7 a i ˆ WiWi a i ˆ c ˆ WcWc Common ApproachThis Work c ˆ Concentrations Single source intensities Acquired multiplexed intensities efficient acquisition Alterman, Schechner & Weiss, Multiplexed Fluorescence Unmixing N dyes =3 N sources =7 size(i)=7 Wi≠WcWi≠Wc Multiplexing: a=Wi, Mixing: i=Xc

Fluorescence Cell structure and processes Corn Grain Flea Intestine Tissue Horse Dermal Fibroblast Cells Fluorescent Specimen

9 Linear Mixing Molecules per pixel More molecules per pixel Brighter pixel c i i c i = x∙c Alterman, Schechner & Weiss, Multiplexed Fluorescence Unmixing

10 Linear Mixing {c d } i vector of concentrations (spatial distribution) For each pixel: i = x x ∙ ∙ ∙ x 1 2 N dyes cc∙∙∙ccc∙∙∙c 1 2 N dyes Alterman, Schechner & Weiss, Multiplexed Fluorescence Unmixing

11 Linear Mixing s=1 i vector of intensities Mixing matrix cc∙∙∙ccc∙∙∙c 1 2 N dyes 1 s=2 i 2 i = x x ∙ ∙ ∙ x 1,1 1,2 1,N dyes 1 i = x x ∙ ∙ ∙ x 2,1 2,2 2,N dyes 2 i = x x ∙ ∙ ∙ x s,1 s,2 s,N dyes s ∙∙∙∙∙∙ {c d } vector of concentrations (spatial distribution) For each pixel:

12 Linear Mixing s=1 i vector of intensities Mixing matrix For each pixel: s=2 i 2 1 {c d } vector of concentrations (spatial distribution)

Fluorescent Microscope Fluorescent Specimen Dichroic Mirror Emission Filter Intensity image Blue L 2 (λ) λ λ Excitation Sources Excitation Filter s=1 s=2 s=3 s=4 =5s s : illumination sources e(λ) λ α(λ)α(λ)

Intensity image Fluorescent Microscope Fluorescent Specimen Dichroic Mirror Emission Filter Green L 2 (λ) λ λ Excitation Sources Excitation Filter s=1 s=2 s=3 s=4 =5s s : illumination sources λ α(λ)α(λ) e(λ) Cross-talk 14 Unmixing required Intensity image (mixed) Blue

Problem Definition 15 Unmix Intensity image (mixed) + noise noise How to multiplex for least noisy unmixing? Fluorescent specimen Alterman, Schechner & Weiss, Multiplexed Fluorescence Unmixing

Sum up the concepts c ia mixing un mixing multiplexing de multiplexing Concentrations Single source Image intensities Acquired multiplexed image intensities X X -1 W W -1 Nature Man made 16 Alterman, Schechner & Weiss, Multiplexed Fluorescence Unmixing multiplexed unmixing

Look closer - again 17 Xc i Estimate c not i i – single source intensities η - noise Alterman, Schechner & Weiss, Multiplexed Fluorescence Unmixing

Multiplexed Unmixing acquisition Minimum Variance in c  W=? For each pixel 18 i c = + a acquired measurements W multiplexing matrix X mixing matrix noise estimation OR Weighted Least Squares WX is not square Other estimators OR

Generalizations 19 var(η) = constant i =? c =? var(η) ≠ constant Image intensities concentrations Minimum Var  W=? η - noise Details in the paper Alterman, Schechner & Weiss, Multiplexed Fluorescence Unmixing

Generalized Multiplex Gain 20 What is the SNR gain for unmixing? Only Unmixing Unmixing + Multiplexing VS. Alterman, Schechner & Weiss, Multiplexed Fluorescence Unmixing

Significance of the Model N sources =N measure GAIN c a i ˆ c ˆ WcWc a i ˆ c ˆ WiWi VS. Alterman, Schechner & Weiss, Multiplexed Fluorescence Unmixing Wi≠WcWi≠Wc

Significance of the Model N sources =N measure GAIN c a i ˆ c ˆ WcWc Alterman, Schechner & Weiss, Multiplexed Fluorescence Unmixing

Significance of the Model N sources =N measure GAIN c GAIN < 1 For specific 3 dyes, camera and filter characteristics a i ˆ c ˆ WiWi a i ˆ c ˆ WcWc

24 Natural Linear Mixing ? ImageJ image sample collection. c Raskar et al ci i c i

= + a X c W Multiplexed Unmixing 25 η i Xc The goal is unmixing Efficient Acquisition Exploit all available sources SNR improvement Generalization of multiplexing theory Alterman, Schechner & Weiss, Multiplexed Fluorescence Unmixing