1. Department of Electrical and Computer Engineering Blind Assistive Technology Bill Reading Device (BATBRD) Professor Aura Ganz Ian McAlister Colin Smith Chris Neyland Erick Drummond TEAM GANZ Midway Design Review

2. 2Department of Electrical and Computer Engineering OUTLINE  PDR Review  MDR Deliverables  Beagle Board Testing  Imaging Device Testing  Mock-up  Open CV Template Matching  Matlab Results  Experimental Protocol  Summary of Current Status  Proposed CDR Deliverables  Conclusion and Pathway to FPR

3. 3Department of Electrical and Computer Engineering PDR Review  Population 1.8 million legally blind individuals in the U.S. 21.2 million reported experiencing vision loss  US Currency Problem No current identification methods Current technology is very expensive  Hardware Selection  Computer Board  BeagleBoard  Imaging Device  PS3 Eye Web Camera

4. 4Department of Electrical and Computer Engineering System Specifications: Bill Reading Device  The Bill Reader Device Estimated Specifications: 1 processing unit: Beagle Board Commercial Off-The-Shelf (COTS) Camera: PS3 EYE Speaker LED Lights SD Card Low power usage System standby when not in use  Hypothesis: Identify Bill < 10 sec. Ideal Bill Image Identification  100% Degraded or Rotated Bill Image ID  >90%

5. 5Department of Electrical and Computer Engineering  Insert the autocad picture and then show and desribe the mock-up here Mock-up

6. 6Department of Electrical and Computer Engineering MDR Deliverables  Learn hardware/software platform  Imaging Device Customizing  Implement one image processing algorithm on a PC  Test the algorithm on ideal images  Evaluate image processing algorithms  Build bill library  (DON’T FORGET TO SPRINKLE THROUGH PRESENTATION AND ADD CHECK MARKS)

7. 7Department of Electrical and Computer Engineering  Development PC with Linux installed  Serial (null) modem  Beagleboard  DC Power supply  USB Keyboard, Mouse, Webcam  A self powered USB hub  An SD memory card  HDMI to DVI  Misc cables Beagle Board Hardware Overview

8. 8Department of Electrical and Computer Engineering  Build an Image on the Development PC  A custom image must be created in order have appropriate drivers loaded for a our given hardware  Partition SD card and load the image  Insert SD card and “dial in” to the Beagleboard via modem  Update boot loader to use new image  Boot image and test  Two Development Methods must be implemented  Cross Compiling & Native Development Beagle Board Overall process:

9. 9Department of Electrical and Computer Engineering  Have a working image with following supported:  USB Keyboard, Mouse  Install webcam Drivers  Set up audio Playback  Have an implementation of python and Gnome c++ compilers for building and running programs  Set up OpenCV Support  Need to:  Start implementing Erick and Ian’s code on the Beagleboard  Create (blind) user interface and button setup  Make use of the DSP to speed up program run time Beagle Board Current Standing & What’s next:

10. 10Department of Electrical and Computer Engineering Imaging Device Customizing Original Lens Undesirable Picture at Close Range Too Bulky Resulting Image

11. 11Department of Electrical and Computer Engineering Imaging Device Customizing Basic Lens Great Close Range Images Compact Resulting Image

12. 12Department of Electrical and Computer Engineering Image Processing – Algorithms  Template Matching is core of object recognition engine - Powerful, easy to implement - Uses stored library of templates  Results for each template analyzed to determine best match  Problem: computationally expensive if not optimal image - Must optimize image before TM is performed  Choose autorotation since rotation is hardest for TM to handle - Use edges to determine and correct rotation of image

13. 13Department of Electrical and Computer Engineering Image Processing Algorithms

14. 14Department of Electrical and Computer Engineering Image Processing – Autorotation  Concept: reference horizontal and vertical lines in image to correct any incidental (up to +/-16deg) rotation  Use detail-sparse binary copy of image to find H/V edges -Canny edge-detection algorithm draws outlines  Filter with 12px H/V lines to eliminate angled lines -Angles, curves composed of many small line segments -12px eliminates fine detail but preserves large features -'Image Opening' extracts features similar in shape to filter  Perform process for +/-16deg range around 0deg -Correct angle yields highest 'score' of H/V lines  Angle with highest score is chosen as Correction Angle

15. 15Department of Electrical and Computer Engineering Image Processing – Algorithm Testing Demonstration: MATLAB Autorotation code Histogram shows occurrence of long (>12px) horizontal, vertical lines Algorithm correct to +/- 1deg (for all $1, $5 bill tests so far)

16. 16Department of Electrical and Computer Engineering Template Matching Template Matching Performed – Compare Segments of Captured Image to Stored Template – “Slides” Through Captured Image Using A Function to Quantify Matches at Any Given Point Template Image Captured Image

17. 17Department of Electrical and Computer Engineering Template Matching Testing-Poor Results  Poor Results From Various Template Matching Functions  False Positives are Common Found Match Here Template Cropped Result

18. 18Department of Electrical and Computer Engineering Template Matching Algorithm  Find Best Match by Using Normalized Cross-Correlation Coefficients Function (NCC)  Best Match Retrieved by OpenCV Function: cvMinMaxLoc( image, &minval, &maxval, &minloc, &maxloc, 0 ); Where maxval is a number from -1 to +1 corresponding to best match.  This Value Can then Be Used to Determine if There is a Match

19. 19Department of Electrical and Computer Engineering Template Matching – NCC Results Found Match Here Template Cropped Result

20. 20Department of Electrical and Computer Engineering Summary of Current Status  Met MDR Deliverables  BeagleBoard Testing  Imaging Device Testing  Image Processing Algorithms  Tested Algorithms on Ideal Images  Achievements Beyond MDR Deliverables  BeagleBoard integrated with Imaging Device  Project Enclosure Researched  Fixed Camera Height for Optimal Image Capture  Experimental Design and Evaluation Methods Established

21. 21Department of Electrical and Computer Engineering Summary - Next Steps  Goals Before Beginning of Spring Semester  Power Decision (Battery)  Identify Optimal Lighting Setup  Conduct Experiments Using Algorithms  More Finalized Project Enclosure Design

22. 22Department of Electrical and Computer Engineering CDR Deliverables  Experimental Design  Extensive testing of chosen algorithms on PC  Consider Alternative Algorithms (if needed)  Determine Battery for Powering Device  Determine Lighting Setup  Implement Chosen Algorithm on BeagleBoard  Conduct time measurements  Pathway to FPR  Finalize and Fabricate Project Enclosure – March  Ensure to ask Jenny for feedback before fabrication  Test and Data Collection Using Actual Design – April  Develop User Interface

23. 23Department of Electrical and Computer Engineering Gantt Chart

24. 24Department of Electrical and Computer Engineering Questions ?

25. 25Department of Electrical and Computer Engineering Experimental Design  Fix all controllable variables  Ideal image  100% match  Orientation Change on Ideal Image  100% match  Test different lighting schemes  Choose best. Retain only > 90%  Fixed Orientation and Choose best lighting with degraded images  Faded, Wrinkled, Markings, Partial  >90% match