1. vOICe 1.5 Enabling Technology - Final Project Gabe Su

2. Motivation Flaws of current system Flaws of current system Hard to learn Hard to learn Not how humans view images Not how humans view images Slow update time (1 sec) Slow update time (1 sec) Current vOICe system is actually pretty good if you can learn it Current vOICe system is actually pretty good if you can learn it Has been compared to learning a foreign language Has been compared to learning a foreign language My project should be considered a supplement to the vOICe system rather than a replacement My project should be considered a supplement to the vOICe system rather than a replacement

3. My Idea Do some sort of image processing to pick out important features Do some sort of image processing to pick out important features Faster updates (100 ms or less) Faster updates (100 ms or less) Constant sound coming from objects Constant sound coming from objects Let users’ own sense of motion be used to extract depth information Let users’ own sense of motion be used to extract depth information

4. What I Did Convert vertical lines in the image into sound Convert vertical lines in the image into sound For helping blind users avoid half-open doors, walls, etc. Also may help with general spatial awareness. For helping blind users avoid half-open doors, walls, etc. Also may help with general spatial awareness. I didn’t have enough time to do actual real-time system I didn’t have enough time to do actual real-time system For this project, took a video and processed it with Matlab For this project, took a video and processed it with Matlab However, algorithm should be fast enough to work in real time However, algorithm should be fast enough to work in real time

5. What I Did For each frame: For each frame: Step 1: Image processing Step 1: Image processing Find all vertical lines Find all vertical lines Find most important vertical lines Find most important vertical lines Analyze/label objects & their position, size, etc. Analyze/label objects & their position, size, etc. Step 2: Convert to sound Step 2: Convert to sound Different sound for each object Different sound for each object

6. Step 1: Image Processing Original Image Original Image

7. Step 1: Image Processing Blur/Vertical Line Filter Blur/Vertical Line Filter Blur to get rid of noise Blur to get rid of noise

8. Step 1: Image Processing Vertical Median Filter Vertical Median Filter Get rid of non-vertical lines and short vertical lines Get rid of non-vertical lines and short vertical lines

9. Step 1: Image Processing Find the x longest lines Find the x longest lines Connected Components Connected Components

10. Step 1: Image Processing Convert to 1D intensity map Convert to 1D intensity map Hard to distinguish between high and low sounds anyways Hard to distinguish between high and low sounds anyways

11. Issues When Converting to Sound Can’t use the same sound for every line Can’t use the same sound for every line Need to use different sound per line Need to use different sound per line How many sounds/positions can one hear at a time? How many sounds/positions can one hear at a time? If you want one sound to follow a particular object, how to track that object? If you want one sound to follow a particular object, how to track that object?

12. Step 2: Converting to Sound Still working on this part Still working on this part Chose to track top 5 lines per image Chose to track top 5 lines per image Assign a different sound to each line Assign a different sound to each line Why 5? Why 5? I only had 5 different “beep”-type sounds I only had 5 different “beep”-type sounds Track same object in next frame based on minimum distance moved Track same object in next frame based on minimum distance moved Won’t always work, but hopefully good enough for most cases Won’t always work, but hopefully good enough for most cases

13. Future Improvements Actually get it real-time Actually get it real-time Try it out with a webcam/headphones/laptop Try it out with a webcam/headphones/laptop Better image processing Better image processing Experiment with different filters, etc. Experiment with different filters, etc. More intuitive & accurate conversion to sound More intuitive & accurate conversion to sound In addition to sound position, assign image information to pitch, intensity, etc. In addition to sound position, assign image information to pitch, intensity, etc. Other types of image processing Other types of image processing Face/Shape/Letter/Color recognition Face/Shape/Letter/Color recognition Will get better as computers get better Will get better as computers get better

14. Demos (no sound yet) http://www.cs.unc.edu/~okkay/projectvids.htm http://www.cs.unc.edu/~okkay/projectvids.htm http://www.cs.unc.edu/~okkay/projectvids.htm