June Prof. Heikki Kälviäinen et al., ImageRet Project1 IMAGERET Detection and Decision-Support Diagnosis of Diabetic Retinopathy Using Machine Vision Professor Heikki Kälviäinen et al. Machine Vision and Pattern Recognition Laboratory (MVPR) Department of Information Technology Faculty of Technology Management Lappeenranta University of Technology (LUT), FINLAND

June Prof. Heikki Kälviäinen et al., ImageRet Project2 London Berlin Moscow St.Petersburg Tallinn Lappeenranta Oslo Stockholm Helsinki FINLAND Lappeenranta University of Technology (LUT)

June Prof. Heikki Kälviäinen et al., ImageRet Project3 Outline Machine Vision and Pattern Recognition Laboratory. Diabetes and retina. ImageRet project and the consortium. Objectives and results. On-going research and future challenges.

June Prof. Heikki Kälviäinen et al., ImageRet Project4 LUT Information Technology (LUT IT) Leader: Prof. Heikki Kälviäinen. 6 Professors, 70 members, 850 B.Sc./M.Sc./Ph.D. students in total, masters and 4-5 doctors per year. Laboratories: – Machine Vision and Pattern Recognition (MVPR). LUT Center of Excellence in Research. – Software Engineering (SWE). – Communications Software (CS). LUT IT: MVPR:

June Prof. Heikki Kälviäinen et al., ImageRet Project5 MVPR Laboratory: Research Profile

June Prof. Heikki Kälviäinen et al., ImageRet Project6 Machine Vision and Pattern Recognition Laboratory (MVPR) Leader: Prof. Heikki Kälviäinen. 2nd largest computer vision research group in Finland. Center of Excellence in Research in LUT. 24 members: 3 Professors + 3 Post docs + 2 Visiting doctors + 11 PhD students + undergraduate students + industry coordinator. Co-operation with 14 international universities and research institutes. Results: 18 Ph.D. degrees (and 3 externally produced), over 400 scientific publications, 40 research projects, and spin-off companies. Objectives: 2 PhDs/year. Annual external project funding EUR, basic funding EUR, total 1.0 million EUR.

June Prof. Heikki Kälviäinen et al., ImageRet Project7 Diabetes Diabetes is a metabolic disorder where the blood glucose level has been increased. Two types: – Type 1: insulin dependent (mostly children and young persons). – Type 2: non-insulin dependent (mostly middle-aged and elderly people). Diabetes is a serious disease: – When left untreated, diabetes can lead to serious medical complications of the kidneys, the peripheral nervous system, the eyes, and it can also cause cardiovascular diseases. Diabetes is a common and rapidly increasing disease: – For example, Finns (10 % of the population) are having Type 1 (in Finland the most common in the world) or Type 2 (In Finland the most common in the Nordic countries), with the increase of the number of Type 2 diabetics by 70 % in the next 10 years! Diabetes is an expensive disease: – 12 % of total health service costs in Finland. – Approx. 15% of all health care expenses in EU go to the treatment of diabetes and its complications.

June Prof. Heikki Kälviäinen et al., ImageRet Project8 Diabetic Retinopathy Diabetic retinopathy is a disease of the retina (the tissue responsible for vision in the eye) caused by diabetes. Without proper treatment it can lead to the loss of vision or even blindness (the leading cause of the blindness in the working age population). Early detection of the retinal complications is crucial. An ophthalmic fundus camera can be used to monitor the condition of the retina => fundus photoghaphy.

June Prof. Heikki Kälviäinen et al., ImageRet Project9 Fundus Image Acquisition The eye: Besides the vision system, an useful peephole inside a human being to see what is happening. Zeiss Fundus Camera:1500 x 1152 pixels 24 bits per pixel.

June Prof. Heikki Kälviäinen et al., ImageRet Project10

June Prof. Heikki Kälviäinen et al., ImageRet Project11 Challenges and Objectives Challenges: robust screening needed diabetes patients in Finland and the number is increasing. How to monitor the known patients and find the new ones? Not enough medical experts nor funding for applying current practices. => We must find robust automatic or semiautomatic solutions for two tasks: 1.To decide whether the eye is healthy or not (the disease present or not). 2.To find reliably abnormalities in the eye, if the eye is considered to be not healthy.

June Prof. Heikki Kälviäinen et al., ImageRet Project12 ImageRet Project FinnWell technology program was established by Finnish National Agency for Technology and Innovation (TEKES). The project called Optimal Detection and Decision-support Diagnosis of Diabetic Retinopathy (ImageRet) was established to develop reliable and accurate image processing and pattern recognition methods for automatic fundus analysis. Project of 38 months in with the several partners ( EUR): Lappeenranta University of Technology (LUT), Finland: project coordination, machine vision and pattern recognition. University of Kuopio, Finland (UKU): ophthalmology. University of Joensuu, Finland (UJO): spectral imaging. Mikkeli Polytechnics, Finland (MAMK): databases and metadata. University of Bristol, UK (UB): optic disk detection. Companies: Kuomed Oy, Mawell Oy,Perimetria Oy, Pfizer Oy, Santen Oy, VAS Oy.

June Prof. Heikki Kälviäinen et al., ImageRet Project13 ImageRet: Acknowledgements Intensive co-operation of many researchers Pauli Fält (UJO), Jari Forström (MAMK), Pertti Harju (MAMK), Dr. Jouni Hiltunen (UJO), Dr. Markku Hauta-Kasari (UJO), Valentina Kalesnykiene (UKU), Tomi Kauppi (LUT), Markku Kuivalainen (LUT), Prof. Joni Kämäräinen (LUT), Prof. Heikki Kälviäinen (LUT), Dr. Lasse Lensu (LUT), Mika Letonsaari (MAMK), Prof. Majid Mirmehdi (UB), Pekka Nikula (LUT), Prof. Jussi Parkkinen (UJO), Dr. Juhani Pietilä (Perimetria), Markku Rossi (MAMK), Prof. Iiris Sorri (UKU), Prof. Hannu Uusitalo (UKU), etc. & many company representatives.

June Prof. Heikki Kälviäinen et al., ImageRet Project14 ImageRet: Objectives Image annotation tool for medical expert annotation. – Medical experts can save and compare their diagnoses with the tool. Fundus image databases. – Expert annotations collected as ground truth in public databases. – Testing protocols for benchmarking between different methods. – Private patient databases (including temporal changes in the eye). Evaluation framework. – A solid basis for the image analysis system development and comparison. Image-based and pixel-based methods. – Image-based: Is there a healthy eye or not in an image? – Pixel-based: Detection of abnormalities related to diabetic retinopathy: hard exudate, soft exudate, hemorrhage, microaneurysm, and neovascularization. Spectral imaging. How much more it can be “seen” using spectral imaging?

June Prof. Heikki Kälviäinen et al., ImageRet Project15 Normal Fundus Image 1.Papilla (optic disk). 2.Blood vessels. 3.Macula (the area of the sharp vision)

June Prof. Heikki Kälviäinen et al., ImageRet Project16 Hard Exudate Hard exudate consists of blood plasma and lipids leaked from blood vessels. It is one of the most commonly occurring lesion in diabetic retinopathy. Yellow-white lesions. Sharp margins.

June Prof. Heikki Kälviäinen et al., ImageRet Project17 Soft Exudate Soft exudate is a micro-infarct occurring in an eye. Yellowish lesions. Fuzzy margins.

June Prof. Heikki Kälviäinen et al., ImageRet Project18 Hemorrhage Hemorrhage consists of blood leaked from vessels. Dark red lesions. The color is quite similar as in vessels.

June Prof. Heikki Kälviäinen et al., ImageRet Project19 Microaneurysm Out-pouching of capillary. Visible as a tiny red dot. The first observable type of lesion in retinopathy. Quite difficult to notice in a color fundus image.

June Prof. Heikki Kälviäinen et al., ImageRet Project20 Neovascularization Abnormal vessels growing to satisfy the lack of oxygen in a retinopathic eye. Can cause severe problems.

June Prof. Heikki Kälviäinen et al., ImageRet Project21 Medical Expert Annotations Digital fundus image. Medical expert annotations.

June Prof. Heikki Kälviäinen et al., ImageRet Project22 Image Annotation Tool

June Prof. Heikki Kälviäinen et al., ImageRet Project23 Fundus Image Databases Databases DIARETDB0 DIARETDB1 DIARETDB1 V2.1 publicly available at

June Prof. Heikki Kälviäinen et al., ImageRet Project24 Diabetic Retinopathy Database Images (89): Train images 28 Test Images 61 Med. experts 4 Findings: Haemorrhages (Ha) Microaneurysms (Ma) Hard exudates (He) Soft exudates (Se)

June Prof. Heikki Kälviäinen et al., ImageRet Project25 Evaluation Framework

June Prof. Heikki Kälviäinen et al., ImageRet Project26 Evaluation Framework – Training Uneven illumination. Colour distortions. Imaging related. Eye related. Imaging noise. Colour. Texture. Shape.

June Prof. Heikki Kälviäinen et al., ImageRet Project27 Fusing Multiple Medical Expert Annotations (b) (a)(c) Annotation fusion approaches*: a) weighted area intersection b) representative point neighbourhood c) representative point neighbourhood masked. * Tomi Kauppi, Joni-Kristian Kämäräinen, Lasse Lensu, Valentina Kalesnykiene, Iiris Sorri, Heikki Kälviäinen, Hannu Uusitalo, Juhani Pietilä, Proc. of the 16th Scandinavian Conference on Image (SCIA2009), Fusion of multiple expert annotations and overall score selection for medical diagnosis.

June Prof. Heikki Kälviäinen et al., ImageRet Project28 Feature Extraction - Colour as Feature Diabetic retinopathy colour distributions

June Prof. Heikki Kälviäinen et al., ImageRet Project29 Estimating Colour Distributions: Learning Estimating colour distributions with a Gaussian mixture model using the unsupervised Figueiredo-Jain algorithm.

June Prof. Heikki Kälviäinen et al., ImageRet Project30 Evaluation Framework – Analysis

June Prof. Heikki Kälviäinen et al., ImageRet Project31 Analysis – Classification (Colour) 1/2 Pixel-wise likelihood for hard exudates: a) original image; b) probability density map (likelihood) for colour (RGB). a)b)

June Prof. Heikki Kälviäinen et al., ImageRet Project32 Analysis – Classification 2/2 The pixel-wise probability of diabetic finding, p(finding), for image is the combination of the selected probability density maps: p(finding) = p('colour')p('texture')p('reliability' )...

June Prof. Heikki Kälviäinen et al., ImageRet Project33 Analysis – Overall Score Fusion Test hypothesis = the disease present (positive) or not (negative). Overall score = a test outcome indicator for an image (a higher value increases the certainty of the positive outcome). Overall score fusion strategies*: max, summax, mean, product. * Tomi Kauppi, Joni-Kristian Kämäräinen, Lasse Lensu, Valentina Kalesnykiene, Iiris Sorri, Heikki Kälviäinen, Hannu Uusitalo, Juhani Pietilä, Proc. of the 16th Scandinavian Conference on Image (SCIA2009), Fusion of multiple expert annotations and overall score selection for medical diagnosis.

June Prof. Heikki Kälviäinen et al., ImageRet Project34 Evaluation Framework – Evaluation  Sensitivity.  Specificity. * Tomi Kauppi, Valentina Kalesnykiene, Joni-Kristian Kämäräinen, Lasse Lensu, Iiris Sorri, Asta Raninen, Raija Voutilainen, Hannu Uusitalo, Heikki Kälviäinen, Juhani Pietilä, Proc. of the British Machine Vision Conference (BMVC2007), The DIARETDB1 diabetic retinopathy database and evaluation protocol, pp , Vol. 1.

June Prof. Heikki Kälviäinen et al., ImageRet Project35 Evaluation – Receiver Operating Curve (ROC) Parameters Sensitivity = T_P/(T_P+F_N) Specificity = T_N/(T_N+F_P) T_P = true positives (abnormal) T_N = true negatives (normal) F_P = false positives (normal as abnormal) F_N = false negatives (abnormal as normal)

June Prof. Heikki Kälviäinen et al., ImageRet Project36 Evaluation – ROC Curves = Equal error rate (EER) = Weighted error rate (WER)

June Prof. Heikki Kälviäinen et al., ImageRet Project37 Method Development So Far Automatic image analysis in human supervision is possible. Possible applications. –Medical diagnosis assistance – screening. –Fundus image sorting according to severity/certainty of the disease. –Semi-automatic tool to aid remote diagnosis. –Quality control of diagnosis work. –Patient specific image databases.

June Prof. Heikki Kälviäinen et al., ImageRet Project38 Spectral Imaging: Significantly New Information about Diabetic Retinopathy ? Grayscale images: 1 channel (e.g. fluorescein angiography). RGB images: 3 channels (colour photographs). Spectral images: Tens or hundreds of separate colour channels. => Contain significantly more colour information than RGB images. Spatial Spectrum Spectral image Spatial R/G/B RGB Spatial Grayscale

Article in the 16 th Scandinavian Conference on Image Analysis (SCIA 2009), Oslo, Norway, June 15-19, 2009: Extending Diabetic Retinopathy Imaging from Color to Spectra Pauli Fält 1, Jouni Hiltunen 1, Markku Hauta-Kasari 1, Iiris Sorri 2, Valentina Kalesnykiene 2, and Hannu Uusitalo 2,3 1 InFotonics Center Joensuu, University of Joensuu, Joensuu, Finland 2 Department of Ophthalmology, Kuopio University Hospital and University of Kuopio, Kuopio, Finland 3 Department of Ophthalmology, Tampere University Hospital, Tampere, Finland

June Prof. Heikki Kälviäinen et al., ImageRet Project40 Built by Color Vision Group, University of Joensuu, Finland based on Canon CR-45NM fundus camera. Spectral separation by 30 narrow bandpass interference filters. 400 – 700 nm at approx. 10 nm steps. A digital grayscale image for each filter separately => spectral image. Spectral Fundus Camera

June Prof. Heikki Kälviäinen et al., ImageRet Project41 Human Subjects 66 volunteers: 54 diabetic patients + 12 control subjects. The clinical trials were conducted in the Department of Ophthalmology of the Kuopio University Hospital, Finland. A corresponding spectral database will be published soon.

June Prof. Heikki Kälviäinen et al., ImageRet Project42 RGB 580 / 540 / 500 nm Optimal Colour Channels

June Prof. Heikki Kälviäinen et al., ImageRet Project43 RGB 580 / 540 / 500 nm Can We “See” More?

June Prof. Heikki Kälviäinen et al., ImageRet Project44 Yes?

June Prof. Heikki Kälviäinen et al., ImageRet Project45 Summary of Results and Future Challenges Image annotation tool for medical expert annotation. – Done. Fundus image databases. – Done as defined in the objectives. – More expert annotations to verify ground truth and a new release, if needed. – First patient databases as a function of time collected. – Spectral database to be published. Evaluation framework. – Done. Image-based and pixel-based methods. – “Semiautomatic” solution done (image-based screening). – More method development needed : spatial prior information, texture analysis, shape analysis, spectral colour information. – Other diseases than diabetes. Spectral imaging. Images taken and preliminary expert annotations marked (more needed). Feature selection to be studied and related methods to be developed.