Department of Electrical and Computer Engineering Blind Assistive Technology Bill Reading Device (BATBRD) Professor Aura Ganz Ian McAlister Colin Smith Chris Neyland Erick Drummond TEAM GANZ Midway Design Review

2Department of Electrical and Computer Engineering OUTLINE  PDR Review  MDR Deliverables  Beagle Board Testing  Open CV Template Matching  Matlab Results  Summary of Current Status  Proposed CDR Deliverables  Conclusion and Pathway to FPR

3Department of Electrical and Computer Engineering PDR Review  Population 1.8 million legally blind individuals in the U.S million reported experiencing vision loss  US Currency Problem No current identification methods Current technology is very expensive  Hardware Selection  Computer Board  BeagleBoard  Imaging Device  PS3 Eye Web Camera

4Department of Electrical and Computer Engineering Product Requirements  Low Cost  Intuitive Interface Design – 2 to 3 buttons max  Small Form Factor - <50in 3  Battery Life – One Hour Continuous Use  Performance - 90% Accuracy  Upgradable Bill Library

5Department of Electrical and Computer Engineering System Specifications: Bill Reading Device  The Bill Reader Device Estimated Specifications: 1 Processing Unit: Beagle Board Commercial Off-The-Shelf (COTS) Camera: PS3 EYE Speaker LED Lights SD Card Microcontroller and a few Buttons for User Interface Low Power Usage System standby when not in use

6Department of Electrical and Computer Engineering MDR Deliverables  Imaging Device Customizing  Learn hardware/software platform  Implement one image processing algorithm on a PC  Test the algorithm on ideal images  Evaluate image processing algorithms  Build bill library

7Department of Electrical and Computer Engineering Imaging Device Customizing Original Lens Undesirable Picture at Close Range Too Bulky Resulting Image

8Department of Electrical and Computer Engineering Imaging Device Customizing Basic Lens Great Close Range Images Compact Resulting Image

9Department of Electrical and Computer Engineering MDR Deliverables  Imaging Device Customizing  Learn hardware/software platform  Implement one image processing algorithm on a PC  Test the algorithm on ideal images  Evaluate image processing algorithms  Build bill library

10Department of Electrical and Computer Engineering  Development PC with Linux installed  Serial (null) modem  Beagleboard  DC Power supply  USB: Keyboard, Webcam, & Ethernet  Self powered USB hub  SD memory card  HDMI to DVI  Misc cables Beagle Board Development Hardware Overview

11Department of Electrical and Computer Engineering  Build an Image on the Development PC  A custom image must be created in order have appropriate drivers loaded for a our given hardware  Partition SD card and load the image  Insert SD card and “dial in” to the Beagleboard via modem  Update boot loader to use new image  Boot image and test  Two Development Methods must be implemented  Cross Compiling & Native Development Beagle Board Overall process:

12Department of Electrical and Computer Engineering  Have a working image with following supported:  USB Keyboard  Webcam Drivers  Audio Playback  Implementation of Python and Gnome C++ compilers for building and running programs  OpenCV Support  Need to:  Start implementing Erick and Ian’s code on the Beagleboard  Create (blind) user interface and button setup  Make use of the DSP to speed up program run time Beagle Board Current Standing & What’s next:

13Department of Electrical and Computer Engineering Beagle Board Demo

14Department of Electrical and Computer Engineering MDR Deliverables  Imaging Device Customizing  Learn hardware/software platform  Implement one image processing algorithm on a PC  Test the algorithm on ideal images  Evaluate image processing algorithms  Build bill library

15Department of Electrical and Computer Engineering Image Processing Algorithms - Overview  Template Matching is core of object recognition engine  Problem: computationally expensive if not optimal image - Rotation especially difficult  Solution: Pre-processing

16Department of Electrical and Computer Engineering Image Processing Algorithms

17Department of Electrical and Computer Engineering Image Processing – Autorotation

18Department of Electrical and Computer Engineering Image Processing – Autorotation  Concept: reference horizontal and vertical lines in image to correct any incidental (up to +/-16deg) rotation  Use detail-sparse binary copy of image to find H/V edges -Canny edge-detection algorithm draws outlines  Filter with 12px H/V lines to eliminate angled lines -Angles, curves composed of many small line segments -12px eliminates fine detail but preserves large features -'Image Opening' extracts features similar in shape to filter  Perform process for +/-16deg range around 0deg -Correct angle yields highest 'score' of H/V lines  Angle with highest score is chosen as Correction Angle

19Department of Electrical and Computer Engineering Image Processing – Algorithm Testing Demonstration: MATLAB Autorotation code Histogram shows occurrence of long (>12px) horizontal, vertical lines Algorithm correct to +/- 1deg (for all $1, $5 bill tests so far)

20Department of Electrical and Computer Engineering Image Processing – Algorithm Testing Demonstration: MATLAB Autorotation code Histogram shows occurrence of long (>12px) horizontal, vertical lines Algorithm correct to +/- 1deg (for all $1, $5 bill tests so far)

21Department of Electrical and Computer Engineering Image Processing – Algorithm Testing Demonstration: MATLAB Autorotation code Histogram shows occurrence of long (>12px) horizontal, vertical lines Algorithm correct to +/- 1deg (for all $1, $5 bill tests so far)

22Department of Electrical and Computer Engineering Image Processing – Template Matching

23Department of Electrical and Computer Engineering Image Processing – Template Matching Template Matching Performed – Compare Segments of Captured Image to Stored Template – “Slides” Through Captured Image Using a Function to Quantify Matches at any Given Point Template Image Captured Image

24Department of Electrical and Computer Engineering Template Matching Histograms Template Image CorrelationCorrelation Normed Correlation CoefficientCorr. Coeff. Normed

25Department of Electrical and Computer Engineering Template Matching - Poor Results  Poor Results From Various Template Matching Functions  False Positives are Common Match Found Here Template Cropped Result

26Department of Electrical and Computer Engineering Template Matching Algorithm  Find Best Match by Using Normalized Cross-Correlation Coefficients Function (NCC)  C = Correlation Matrix  I = Captured Image, = Sample Image Mean at location: u,v  T = Template Image, = Template Mean  u,v = Indices at Point of Correlation  Best Match Retrieved by OpenCV Function: cvMinMaxLoc( image, &minval, &maxval, &minloc, &maxloc, 0 ); Where maxval is a number from -1 to +1 corresponding to best match.  This Value Can then Be Used to Determine if There is a Match

27Department of Electrical and Computer Engineering Template Matching – NCC Results Match Found Here Template Cropped Result

28Department of Electrical and Computer Engineering MDR Deliverables  Imaging Device Customizing  Learn hardware/software platform  Implement one image processing algorithm on a PC  Test the algorithm on ideal images  Evaluate image processing algorithms  Build bill library

29Department of Electrical and Computer Engineering Summary of Current Status  Met MDR Deliverables  BeagleBoard Testing  Imaging Device Testing  Image Processing Algorithms  Tested Algorithms on Ideal Images  Achievements Beyond MDR Deliverables  BeagleBoard integrated with Imaging Device  Project Enclosure Researched  Fixed Camera Height for Optimal Image Capture

30Department of Electrical and Computer Engineering CDR Deliverables  Experimental Design  Extensive testing of chosen algorithms on PC  Consider Alternative Algorithms (if needed)  Determine Battery for Powering Device  Determine Lighting Setup  Implement Chosen Algorithm on BeagleBoard  Conduct time measurements  Pathway to FPR  Finalize and Fabricate Project Enclosure – March  Collaborate with Jenny on design features  Test and Data Collection Using Actual Design – April  Develop User Interface

31Department of Electrical and Computer Engineering Questions ?

32Department of Electrical and Computer Engineering CDR Deliverable Assignments  Experimental Design (Colin Smith)  Extensive testing of chosen algorithms on PC (Erick)  Consider Alternative Algorithms (if needed)  Determine Battery for Powering Device (Colin)  Determine Lighting Setup (Ian)  Implement Chosen Algorithm on BeagleBoard (Chris)  Conduct time measurements

33Department of Electrical and Computer Engineering Supporting Courses  Critical:  ECE 373 Software Intensive Engineering  ECE 353 Comp Sys I  ECE 354 Comp Sys II  ECE 313 Signals and Systems  ECE 314 Probability and Stochastic Processes  Associated Prerequisite Math and Science Courses  Helpful: Engin 351, ECE 211 and 212

34Department of Electrical and Computer Engineering Gantt Chart

35Department of Electrical and Computer Engineering Summary - Next Steps  Goals Before Beginning of Spring Semester  Power Decision (Battery)  Identify Optimal Lighting Setup  Conduct Experiments Using Algorithms  More Refined Project Enclosure Design