IMAGE ARCHIVE AND LEAF CLASSIFIER SPECIFIC ENABLERS Stuart E. Middleton, Banafshe Arbab-Zavar, Stefano Modafferi, Ken Meacham and Zoheir Sabeur University of Southampton IT Innovation Centre ENVIROFI specific enabler 17 th January 2013 “ENVIROfying” the Future Internet

WP1 pilot use case Image archive Architecture User interface Leaf classifier Architecture Algorithms User interface Overview Image archive and leaf classifier specific enablers 2

WP1 pilot: Citizens in Tuscany Data sources Proof of concept CROWD SOURCING FROM SIR HAROLD HILLIER GARDENS, UK User trial CROWD SOURCING VIA WP1 PILOT IN THE TUSCANY REGION Image archive to record crowd-sourced leaf images Web portal & backend service (Italian & English) Integrated mobile phone platform Support for general public and botanical experts Leaf image + auxiliary images + geo-tag + metadata WP1 pilot use case Image archive and leaf classifier specific enablers 3

Leaf classifier to label unknown images Web portal & backend service (Italian & English) Integrated mobile phone platform Biodiversity ontology support Scientific names (Latin) Common names (Italian, English) Domain ontology URI’s (e.g. TaxMeOn) Natura 2000 habitat codes Value proposition Supporting crowd sourced leaf observations allows image data collection by volunteers at a scale beyond traditional methods WP1 pilot use case Image archive and leaf classifier specific enablers 4

Image archive architecture Image archive and leaf classifier specific enablers 5 Crowd sourcing (web upload and mobile support) Expert review of labels

Image archive user interface Image archive and leaf classifier specific enablers 6

Leaf classifier architecture Image archive and leaf classifier specific enablers 7 Users request classifications (unlabelled images) Top N matches returned (leaf classifier algorithm)

Classic benchmark datasets e.g. Swedish leaf: 1,125 images, 15 species NO SHADOWS LIMITED ROTATION Crowd-sourced datasets  challenging! e.g. Hillier Gardens (IT Innovation): 1400 images, 54 species SHADOWS NATURAL OUTDOOR LIGHTING ARBITRARY ROTATION Leaf classifier algorithms Image archive and leaf classifier specific enablers 8

Segmentation - Colour-based Expectation-Maximization HSV colour space; discard hue due to the high level of noise Colour-based EM algorithm for pixel classification using k-means clustering to initialize the EM algorithm (Belhumeur 2008) Three clusters are considered representing: leaf; shadow and background. Leaf classifier algorithms Image archive and leaf classifier specific enablers 9 P. Belhumeur, et al."Searching the World’s Herbaria: A System for Visual Identification of Plant Species." ECCV

Leaf classifier algorithms Image archive and leaf classifier specific enablers 10 Belhumeur 2008 tried segmentation with two clusters - problems handling shadows We use three clusters for leaf, shadow, background - shadows eliminated Segmentation - Colour-based Expectation-Maximization

Leaf classifier algorithms Image archive and leaf classifier specific enablers 11 ← The 3 clusters are re-classified based on cluster’s properties. Here, both leaf and shadow clusters were subsequently classified as leaf. Segmentation - Examples

Feature extraction - Inner Distance Shape Context (Ling, 2007) Matching - fusion of two matching methods based on confidence levels: Point-based IDSC matching Contour matching Leaf classifier algorithms Image archive and leaf classifier specific enablers 12 Inner-distance connections between sampled points Inner-distance shape context Point correspondence between two images of the same class H. Ling, D. W. Jacobs. Shape Classification Using the Inner-Distance. 2007, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 29, pp

Distinctive classes Leaf classifier algorithms Image archive and leaf classifier specific enablers 13 Vitex Agnus-Castus P(Best match) = 100% Confidence = 100% Quercus Polycarpa P(Best match) = 100% Confidence = 99.82% Alnus Glutinosa 'Pyramidalis‘ P(Best match) = 100% Confidence = 99.66% Platanus ’Pyramidalis’ P(Best match) = 100% Confidence = 97.60% Acer Monspessulanum P(Best match) = 100% Confidence = 97.5% Tilia Tomentosa 'Petiolaris' P(Best match) = 100% Confidence = 81.85% Populus Nigra P(Best match)=93.33% Confidence = 76.67% Rhamnus Alpina P(Best match)=92.86% Confidence = 82.28% Cornus Sanguinea P(Best match)=90.32% Confidence = 74.91% Fagus Sylvatica 'Grandidentata' P(Best match)=90.00% Confidence = 77.78% Ulmus P(Best match)=90.00% Confidence = 66.48%

Erroneous results can be caused by: Similarity between the leaf shape of different species Error in segmentation Insufficient number of training samples Leaf classifier algorithms Image archive and leaf classifier specific enablers 14 Examples of similar shapes Acer Platanoides 'Globosum ' Acer Saccharum subsp Leucoderme Platanus ’Pyramidalis’ Magnolia x Loebneri Magnolia x Soulangeana Carpinus Betulus Ostrya Carpinifolia Rhamnus Alpina Ulmus

Hillier Gardens dataset results Current dataset: 1400 images, 54 species Mean probability of correct first match: 85.18% Mean confidence in correct classification: 73.88% Leaf classifier algorithms Image archive and leaf classifier specific enablers 15

Leaf classifier user interface Image archive and leaf classifier specific enablers 16

Thank you for your attention Stuart E. Middleton twitter.com/ENVIROFI The research leading to these results has received funding from the European Community's Seventh Framework Programme (FP7/ ) under Grant Agreement Number