1. CMU SCS Indexing and Mining Biological Images Christos Faloutsos CMU

2. CMU SCS CMU-052 Outline Motivation - Problem Definition Proposed method Experiments Conclusions

3. CMU SCS CMU-053 ViVo with Ambuj Singh, Vebjorn Ljosa, Arnab Bhattacharya (UCSB) Jia-Yu Tim Pan, HJ Yang (CMU)

4. CMU SCS CMU-054 Detachment Development Normal 1 day after detachment 3 days after detachment 7 days after detachment 28 days after detachment 3 months after detachment

5. CMU SCS CMU-055 Data and Problem (Data) Retinal images taken from cats (Problem) What happens in retina after detachment? –What tissues (regions) are involved? –How do they change over time? How will a program convey this info? More than classification “ we want to learn what classifier learned”

6. CMU SCS CMU-056 Why study retinal detachment Common damage to retina No effective treatment –Surgery or drugs (<100% recovery) Need to understand more about detachment development skip

7. CMU SCS CMU-057 Retina, its image, and the detachment retina Layers of tissuesstained by 3 antibodies (R,G,B) skip

8. CMU SCS CMU-058 Computer Scientist’s View of Retinal Detachment normaldetachment7 days after skip

9. CMU SCS CMU-059 Detachment Development Normal 1 day after detachment 3 days after detachment 7 days after detachment 28 days after detachment 3 months after detachment

10. CMU SCS CMU-0510 How do the treatments do? 28 days after reattachment surgery 6 days after O 2 treatment

11. CMU SCS CMU-0511 Outline Motivation - Problem Definition Proposed method Experiments Conclusions

12. CMU SCS CMU-0512 Main idea extract characteristic visual ‘words’ Equivalent to characteristic keywords, in a collection of text documents

13. CMU SCS CMU-0513 More than classification we want to learn what classifier learned Proposed method: visual vocabulary (vivo) –Try to capture local tissue texture variations –(from stage to stage) Quality of vivo –Classification; Biological meaning? Lessons learn? –Which tissue/texture is significant at stage “7-day”? –What changes between “3-day” and “7-day”? skip

14. CMU SCS CMU-0514 Visual Vocabulary (ViVo) generation Tile image Extract color structure features Independent component analysis (ICA) Visual vocabulary

15. CMU SCS CMU-0515 Proposed method: ViVo Textures are different. –Wavelet (Daubechies-4), MPEG-7 color structure Local variation: partitioned into 64x64 “tiles”. [f 1, …, f m ] “tile-vector” skip

16. CMU SCS CMU-0516 ViVos skip

17. CMU SCS CMU-0517 Outline Motivation - Problem Definition Proposed method Experiments Conclusions

18. CMU SCS CMU-0518 Evaluation of ViVo method how meaningful are the discovered ViVos? can they help in classification? generality? how else can they help biologists create hypotheses?

19. CMU SCS CMU-0519 Example ViVos vivoMeaningCondition Intact rod cell bodies Normal Outer Nuclear Layer (ONL) Intact rod cell bodies + rhodopsin labelling ONL at the beginning of detachment Degenerate rod cell bodies + rhodopsin + hypertrophied Müller cells Detached ONL Intact rod cell bodies + rhodopsin + hypertrophied Müller cells Detached ONL in oxygen treatment

20. CMU SCS CMU-0520 Goals: how meaningful are the discovered ViVos? can they help in classification? generality? how else can they help biologists create hypotheses?

21. CMU SCS CMU-0521 Quality of ViVo – by classification N1d3d7d28d28dr6dO23m N72 1d7 3d12111 7d182 28d1232 28dr121 6dO2119 3m5 Truth Predicted 86% accuracy 46 ViVos (90% energy)

22. CMU SCS CMU-0522 Goals: how meaningful are the discovered ViVos? can they help in classification? generality? how else can they help biologists create hypotheses?

23. CMU SCS CMU-0523 ViVos for protein images

24. CMU SCS CMU-0524 Protein images – MPEG7 CS GiantinHoechstLAMP2NOP4Tubulin Giantin30 Hoechst30 LAMP25091 NOP4182 Tubulin123 Truth Predicted 84% accuracy 4 ViVos (93% energy) 1-NN classifier

25. CMU SCS CMU-0525 Evaluation of ViVo method how meaningful are the discovered ViVos? can they help in classification? generality? how else can they help biologists create hypotheses? ‘ViVo-annotation’!

26. CMU SCS CMU-0526 Automatic ViVo-annotation of images A tile represents a ViVo v k if the largest coefficient of the tile is along the k th basis vector A ViVo v k represents a class c i if the majority of its tiles are in that class For each image, the representative ViVos for the class are automatically highlighted

27. CMU SCS CMU-0527 Which tissue is significant on 7- day?

28. CMU SCS CMU-0528 6 days after O2 treatment

29. CMU SCS CMU-0529 28 days after surgery

30. CMU SCS CMU-0530 Most discriminative ViVos n 1d+6dO 2

31. CMU SCS CMU-0531 Conclusions: how meaningful are the discovered ViVos? can they help in classification? generality? how else can they help biologists create hypotheses?

32. CMU SCS CMU-0532 Outcome/status What are the key results so far? –ViVos: Automatic Visual Vocabulary generation for biomedical image mining, Bhattacharya, Ljosa, Pan, Yang, Faloutsos, Singh (under review) –Software – MATLAB code Tutorial in SIGMOD’05 (Murphy+Faloutsos)