Interactive Visual System This presentation will probably involve audience discussion, which will create action items. Use PowerPoint to keep track of these action items during your presentation In Slide Show, click on the right mouse button Select “Meeting Minder” Select the “Action Items” tab Type in action items as they come up Click OK to dismiss this box This will automatically create an Action Item slide at the end of your presentation with your points entered. Interactive Visual System By Arthur Evans, John Sikorski and Patricia Thomas

What is Interactive Visual System Flower recognition application that runs on a handheld computer Based on CAVIAR desktop system developed by Dr George Nagy and Jie Zou at RPI Leverage the strong points of humans and computers Written in Java - runs equally well on a handheld or desktop computer IVS is a technology, not just a flower identification application – any number of object could be identified with the proper database

Motivation Image recognition is difficult Modern AI and pattern recognition aims to automate process – remove human from the equation Humans and computers have different strong points Computers excel at large scale memory and computations Humans excel at segmentation Combine the two for optimal performance with the least amount of time

Activity Diagram

Handheld Choice Sharp Zaurus SL-5500 Sharp CE-AG06 camera attachment 200 MHz Intel StrongArm processor 64 MB memory 1 Compact Flash port and 1 SD port Linux OS Personal Java support Sharp CE-AG06 camera attachment Plugs into Compact Flash port

Java 2 Micro Edition (J2ME) Provides Java functionality to a broad range of devices Uses concept of Configurations and Profiles Configurations consider system specific issues: Memory, CPU, etc Profiles sit on top of configurations and provide API used by programmer

Personal Java Predates current J2ME specifications Available on many handhelds Provides most Java 1.1.8 desktop API Most similar to Personal Profile Personal Profile is the successor to Personal Java

IVS User Interface Load Flower Select Feature Identify Previous 3 Hits Color 1 Color 2 Stamen color Petal count Petal outline Crop Identify Previous 3 Hits Next 3 Hits Store new Flower Auto feature extract List extracted features

Methods for Feature Entry Point to an object Outline an object Crop an object Enter count

Segmentation Uses Sobel edge detect algorithm Crop box around flower greatly increases accuracy

Classification Uses a k-nearest neighbor algorithm. Allows user to add new flowers to the database with no systematic changes. Handles variation within a species since sample is compared to each exemplar equally – only need a single similar exemplar in the database make an accurate classification

K-Nearest Neighbor Assumes all instances correspond to points in n-dimensional space Nearest neighbor defined as instance closest in Euclidean space (Ax-Bx)2+(Ay-By)2… D=

Calculate Distances Petal count Color Species A Species B Query

Testing IVS Users classified 20 flowers under 5 different conditions Using IVS interactively on the handheld. Using IVS interactively on a desktop. Using IVS with automatic feature extraction on the handheld. Using IVS with automatic feature extraction on the desktop. Searching through a field guide.

Results Using Handheld Using Desktop

Conclusion Designed with extensibility in mind. Only need to extend a few classes to identify other objects Written as a proof of concept, showing human interaction with an image on a handheld and porting a desktop application design to a handheld Improves both speed and accuracy of flower identification