1. CMU SCS Indexing and Mining Biological Images Christos Faloutsos CMU

2. CMU SCS UCSB062 THANKS

3. CMU SCS UCSB063 Outline PART1: ViVo: Visual Vocabulary for cat retina images [PART2: other related work –FALCON: relevance feedback for image by content –Drosophila embryo image mining ]

4. CMU SCS UCSB064 PART1: ViVo with Ambuj Singh, Mark Verardo, Vebjorn Ljosa, Arnab Bhattacharya (UCSB) Jia-Yu Tim Pan, HJ Yang (CMU)

5. CMU SCS UCSB065 Detachment Development Normal 1 day after detachment 3 days after detachment 7 days after detachment 28 days after detachment 3 months after detachment

6. CMU SCS UCSB066 Data and Problem (Data) Retinal images taken from cats (Problem) What happens in retina after detachment? –What tissues (regions) are involved? –How do they change over time? How will a program convey this info? More than classification “ we want to learn what classifier learned”

7. CMU SCS UCSB067 Main idea extract characteristic visual ‘words’ Equivalent to characteristic keywords, in a collection of text documents

8. CMU SCS UCSB068 Visual Vocabulary (ViVo) generation Step 1: Tile image Step 2: Extract tile features Step 3: ViVo generation Visual vocabulary V1 V2 Feature 1 Feature 2 8x12 tiles

9. CMU SCS UCSB069 ViVos skip

10. CMU SCS UCSB0610 Evaluation of ViVo method how meaningful are the discovered ViVos? can they help in classification? generality? how else can they help biologists create hypotheses?

11. CMU SCS UCSB0611 Biological interpretation IDViVoDescriptionCondition V1 GFAP in inner retina (Müller cells)Healthy V10 Healthy outer segments of rod photoreceptors Healthy V8 Redistribution of rod opsin into cell bodies of rod photoreceptors Detached V11 Co-occurring processes: Müller cell hypertrophy and rod opsin redistribution Detached

12. CMU SCS UCSB0612 Goals: how meaningful are the discovered ViVos? can they help in classification? generality? how else can they help biologists create hypotheses?

13. CMU SCS UCSB0613 Quality of ViVo – by classification N1d3d7d28d28dr6dO23m N72 1d7 3d12111 7d182 28d1232 28dr121 6dO2119 3m5 Truth Predicted 86% accuracy 46 ViVos (90% energy)

14. CMU SCS UCSB0614 Goals: how meaningful are the discovered ViVos? can they help in classification? generality? how else can they help biologists create hypotheses?

15. CMU SCS UCSB0615 ViVos for protein images

16. CMU SCS UCSB0616 Protein images – MPEG7 CS GiantinHoechstLAMP2NOP4Tubulin Giantin30 Hoechst30 LAMP25091 NOP4182 Tubulin123 Truth Predicted 84% accuracy 4 ViVos (93% energy) 1-NN classifier

17. CMU SCS UCSB0617 Evaluation of ViVo method how meaningful are the discovered ViVos? can they help in classification? generality? how else can they help biologists create hypotheses? ‘ViVo-annotation’!

18. CMU SCS UCSB0618 Automatic ViVo-annotation of images A tile represents a ViVo v k if the largest coefficient of the tile is along the k th basis vector A ViVo v k represents a class c i if the majority of its tiles are in that class For each image, the representative ViVos for the class are automatically highlighted

19. CMU SCS UCSB0619 Which tissue is significant on 7- day?

20. CMU SCS UCSB0620 6 days after O2 treatment

21. CMU SCS UCSB0621 28 days after surgery

22. CMU SCS UCSB0622 Conclusions: how meaningful are the discovered ViVos? can they help in classification? generality? how else can they help biologists create hypotheses?

23. CMU SCS UCSB0623 Outcome/status ViVos: Automatic Visual Vocabulary generation for biomedical image mining, Bhattacharya, Ljosa, Pan, Verardo, Yang, Faloutsos, Singh; ICDM’05 (one of 5 best student paper award) Software – MATLAB code Tutorial in SIGMOD’05 (Murphy+Faloutsos)

24. CMU SCS UCSB0624 Outline PART1: ViVo: Visual Vocabulary for cat retina images PART2: FALCON: relevance feedback for image by content: SEE DEMO, later Ongoing work: Drosophila Fly Embryos

25. CMU SCS UCSB0625 FALCON - Example query: Inverted VsVs Trader wants only ‘unstable’ stocks

26. CMU SCS UCSB0626 Outline PART1: ViVo: Visual Vocabulary for cat retina images PART2: FALCON: relevance feedback for image by content: SEE DEMO, later Ongoing work: Drosophila Fly Embryos

27. CMU SCS UCSB0627 FEMine: Mining Fly Embryos

28. CMU SCS UCSB0628 FEMine: Mining Fly Embryos

29. CMU SCS UCSB0629 FEMine: Drosophila embryos Feature extraction ICA query by image content, mining, clustering with Andre Balan, Eric Xing, Tim Pan

30. CMU SCS UCSB0630 Conclusions Machine vision + Data mining + Data bases + Biology: => necessary partners