1. Cellular Pattern Quantification and Automatic Bench-marking Data-set Generation on confocal microscopy images Chi(Chrissie) Cui Advisor: Joseph JaJa 1Master of Engineering Thesis Defense

2. Outline Problem Definition Background Feature Extraction Feature Quantification Acceleration using CUDA Synthetic dataset generation Conclusion and Future work Master of Engineering Thesis Defense2

3. Problem Definition Thesis goal: – Extract patterns from confocal microscopy images of giloma cells and build pattern descriptors – Compute quantification measurements to profile the responses of cells to different treatments. – Focus on one cellular protein, the “F-actin” – Particular interested: Schweinfurthin A(SA), a powerful inhibitor for brain tumors Master of Engineering Thesis Defense3

4. Problem Definition Major challenges – What “features” to extract and how? – Definition: General – Extraction: Robust + Repeatable – How to quantify the features and how to validate the results – Descriptive – Intuitive – Effective for profiling Master of Engineering Thesis Defense4

5. Background Actin – A globular, roughly 42-kDa protein. – Participate in many important cellular processes, E.g. muscle contraction, cell motility, the establishment of cell shape and etc. – G-Actin: Individual subunits of actin – F-Actin: long filamentous polymers assembled by G-Actin. Glioma – a type of tumor that starts in the brain or spine Master of Engineering Thesis Defense5 Fig 1. G-Actin Fig 2. F-Actin; surface representation of 13 subunit repeat based on Ken Holmes' actin filament model

6. Background Laser Scanning Confocal Microscopy – Key Feature: Optical Sectioning (Fig. 5). – Image Formation: The out-of-focus light is suppressed by the pinhole. (Fig. 4) Master of Engineering Thesis Defense6 Fig 3. Confocal Microscopy in operation Fig 5. An example for Optical Sectioning Fig 4. Imaging Principle of Confocal Microscopy

7. Background DAPI(Nuclei), GFP(Green Fluorescent Protein), DMSO Schweinfurthin A(SA): (Fig. 6) – Isolated and identified at the National Cancer Institute – Cause dramatic changes in the organization of the actin cytoskeleton – 1000-fold selective for central nervous system tumor cells. Master of Engineering Thesis Defense7 Fig 6. Schweinfurthin A

8. Background Data Preparation – Mouse Glioma Cell Line – 6 Drugs: cytochalasin B (cytoB), Y-27632 (Y), OSW1, Schweinfurthin A (SA), cephalostatin 1 derivative (ceph), control group (DMSO) – Focus: F-actin, stained by Green Fluorescent Protein (GFP) – 2 Separate Channels: cell nuclei + F-actin – 2D Image analysis Master of Engineering Thesis Defense8

9. Previous Work “Interpretation of Protein Location Patterns from Fluorescence Microscopy” - Prof. Robert Murphy’s Group from CMU “Cell Profiler” – BROAD Institute http://www.cellprofiler.org/ Master of Engineering Thesis Defense9 Fig 6. Robert Murphy’s Group Fig 7. Cell Profiler

10. Previous Work Comparison Master of Engineering Thesis Defense10 cell profilerProtein location Pattern Our work Feature Extractioncell based image based Result FormatBinary IndicatorDiscrete Class labels Continuous Numerical Results Classification GoalTell positive from negative Different types of proteins Different organizational and morphological pattern of a single protein (F- actin) MotivationHighlight the difference Show the conditional variation trend Table 1. A comparison between our method and two previous work

11. Analysis Framework Master of Engineering Thesis Defense11 Fig 8. Flow Chart for obtaining the features regions For Ratio Scalar Quantification Ratio Scalar Quantification

12. Analysis Framework Cytoplasm Texture Analysis Quantification Vector Quantification [ Nuclei Cytoplasm Relative Distribution ] Master of Engineering Thesis Defense12 Fig 9. The flow chart for obtaining cytoplasm Texture Analysis Quantification Fig 10. The flow chart for obtaining nuclei cytoplasm relative distribution

13. Features Extraction (Top Down) Tier 1: Tri-band – Intensity based definition – WBA (Brightest Part), Cytoplasm (Medium Brightness), Background (Darkest Part) Master of Engineering Thesis Defense13 Fig 11. Red :White Bright Actin (WBA); Green: Cytoplasm; Blue: Background

14. Feature Extraction (Top Down) Tier 2-a: Sub-Cellular Structure – WBA Component Image Location Based Cortical Actin (Cell Boundary); Inner Punctuate Dots (Inside Cells) – Cytoplasm Component Image Image Enhancement + Thresholding Stress Fibers Master of Engineering Thesis Defense14 Fig 12. Examples for the images of cortical actin, inner punctuate dots and stress fibers inside the cell

15. Feature Extraction Multiscale Texture Analysis for Cytoplasm – Quad-tree decomposition – Multiscale Weighted Gray Scale Co-occurrence Matrix Master of Engineering Thesis Defense15 Fig 13. MWGLCM Formulation: k denotes the type of the measurement; S denotes the maximum decomposition depth, N s denotes the number of patches associated with scale s, W s denotes the patch size at decomposition level s and GLCM i,s k denotes the type k GLCM measurement of block i at decomposition level s

16. Tri-band Segmentation Two Steps Pixel-wise Feature Vector Generation Fuzzy C Means Multiscale Point Wise Feature Extraction – 5x5 Gaussian kernel – Filtering three times – Append the original Intensity – Length 4 feature vector per pixel Master of Engineering Thesis Defense16

17. Tri-band Segmentation Fuzzy C means Segmentation Master of Engineering Thesis Defense17

18. Feature Quantification Goal – Find the closest treatment – Show perceived changes and trend between images – Help make new discoveries Master of Engineering Thesis Defense18

19. Feature Quantification Scalar Quantification – Single Measurement per image – 10 Numbers – Ratios, resolution oblivious Master of Engineering Thesis Defense19 Fig 14. Scalar Quantification Measurement

20. Feature Quantification Nuclei Cytoplasm relative distribution (Vector) – Geodesic distance transformation based Master of Engineering Thesis Defense20 Fig 15: the visualization of cytoplasm nuclei relative distribution quantification for cytoB Fig 16. Left: the cells treated under cytoB in 10um/ml ; right: the cells treated under DMSO.

21. Feature Extraction Validation By manual labeled image 19 images and selectively labeled 2 performance metrics – Detected Rate – Mis-detected Rate Better results than general recognition testing Master of Engineering Thesis Defense21 C.A.I.P.D.S.F. Detected Rate100%95.3%98.7% Mis-detected Rate1.2%6.5%8.6% Table 2. The computed detected rate and mis-detected rate for all the images in the manual validation dataset. Here C.A. is short for cortical actin; I.P.D. is short for inner punctuate dots and S.F. is short stress fibers.

22. Feature Extraction Validation By Computer Simulated Images 5 difference noise conditions 100 image for each case Compute the ratio between the positively detected region and the ground truth as the benchmark metric Master of Engineering Thesis Defense22 Gaussian std.0.0020.004 0.0080.010.02 C.A.96.4%94.5%91.2%89.7%86.9% I.P.D88.4%86.8%86.1%83.8%78.5% S.F.89.5%87.5%86.4%85.0%82.2% Table 3. the detected rate for the computer generated images for the three sub-cellular structures.

23. Feature Quantification Validation Find closest treatment to single treatment –E.g. SA (10, 100, 1000 nm/ml) –Two Facts: 23'deoxy-ceph 1 has similar effects on F-actin as SA OSW-1 has similar effects on F-actin as SA –Visualization: Bar Graph (Next 3 Pages) –Conclusions: SA(10nm/ml) is the most similar to OSW1(1nm/ml) SA(100nm/ml) is the most similar to ceph(1nm/ml) SA(1000nm/ml) is the most similar to SA(100nm/ml) 23Master of Engineering Thesis Defense Next

24. Feature Quantification Validation Master of Engineering Thesis Defense24 Fig 17. SA(10nm/ml) is the most similar to OSW1(1nm/ml) Back

25. Feature Quantification Validation Master of Engineering Thesis Defense25 Fig 18. SA(100nm/ml) is the most similar to ceph(1nm/ml) Back

26. Feature Validation Quantification Master of Engineering Thesis Defense26 Fig 19. SA(1000nm/ml) is the most similar to SA(100nm/ml) Back

27. Feature Quantification Validation “Closeness” Extension: Pair-wise visualization Master of Engineering Thesis Defense27 Fig 20. The pair-wise distance map of all the treatment conditions in our experiment. Darker color means smaller distance, in other word, greater similarity while brighter color means less in common. The closest treatment for each row is highlighted by blue dot

28. Feature Quantification Validation Show trend in series Master of Engineering Thesis Defense28 Fig 21. The quantification result of the ratio of the stress fiber area to the cell area of SA Fig 22. The images corresponding to the quantification result Fig.6.4. (UL) DMSO (UR) SA(10nm/ml) (LL) SA(100nm/ml) (LR) SA(1000nm/ml )

29. Feature Quantification Validation Make “new discoveries” beyond observation – E.g. Weighted multi-scale Cytoplasm Texture Energy – Proved by non-optical methods Master of Engineering Thesis Defense29 Fig 23. The weighted multi-scale cytoplasm texture energy for ceph

30. CUDA Introduction Master of Engineering Thesis Defense30 Fig 24. CUDA Programming Model (Courtesy Image from NVIDIA) Fig 25. CUDA Memory Model

31. Accelerating using CUDA Master of Engineering Thesis Defense31 Table 4. the pseudo code of the Point Wise Feature Extraction Step. Fig 26. computation step and memory layout of the point wise feature extraction step.

32. Accelerating using CUDA Master of Engineering Thesis Defense32 Table 5. The pseudo code of the Fuzzy C Means Segmentation Step. Fig 27. The memory layout for the membership matrices

33. Accelerating using CUDA Experimental Hardware Environment – CPU: Intel(R) Core(TM)2 Duo T5800 2.00GHz each core + 2.00GB memory OS: 32 bit Windows Vista – GPU: NV9600 MGT 512MB device memory 1.25GHz clock rate 4 multi-core processor per die. Experimental Software Enviroment – Matlab V7.1.4 – CUDA 2.1 Master of Engineering Thesis Defense33

34. Accelerating using CUDA Master of Engineering Thesis Defense34 Table 6. the execution configurations of all the kernels. "Width" and "Height“ denotes those of the input image. Table 7. The comparison of the runtime results of the CUDA version and Matlab implementation of our tri-band Segmentation method.

35. Synthetic Dataset Generation Bottom up Approach Component Images: – Cell Bodies – Cortical Actin (No punctuate dots) – Stress Fibers Post Processing – Diffusion – Add Poisson Noise – Add Gaussian Noise – Convolute with PSF Master of Engineering Thesis Defense35 Fig 28. The computational framework of our F-actin highlighted confocal microscopy image simulation

36. Synthetic Dataset Generation Master of Engineering Thesis Defense36 Fig 29. Left to Right: 1) distance map generated from the cell nuclei mask 2)Watershed segmented color labeled image 3) image after selecting the visible part with probability p and erode the image with disk shape structuring element. Fig 30. Left to Right: 1) Generated cortical actin mask. 2) Generated stress fiber mask. 3) Final Result

37. Synthetic Image Generation Validation Synthetic Benchmark Dataset – Apply feature extraction methods to the synthetic images – Use the preciseness as a metric Master of Engineering Thesis Defense37 WBACYTOBKC.A.S.F. Average1.0050.9981.001.0231.106 s. t. d.0.020.030.0050.030.08 Table 8. the preciseness of the tri-band segmentation and sub- cellular structure extraction algorithm when applied to the artificial images. C.A. denotes cortical actin and S.F. denotes stress fibers.

38. Synthetic Image Generation Validation Synthetic Benchmark dataset Master of Engineering Thesis Defense38 Fig 31. A: Image of F- actin labeled cells. B: tri-band segmentation. C: Simulated image of F-actin in cells. D: Tri- band segmentation of the simulated image.

39. Conclusion and Future Work Introduce an image analysis framework for quantifying the F-actin organization patterns in confocal microscopy images. Decide what measures to be taken for quantification so as to make the final results more reliable and descriptive Map Tri-band Segmentation to CUDA and obtain 15 ~ 20 speed up on different sized images Propose a framework for simulating 2-D F-actin confocal microscopy images. Future work: running more validations to test the effectiveness of the quantitative measurements Master of Engineering Thesis Defense39

40. Thank you! Questions? Master of Engineering Thesis Defense40