1. 37.3: Dynamic Magnification of Video for People with Visual Impairment Robert B. Goldstein, Henry Apfelbaum, Gang Luo and Eli Peli The Schepens Eye Research Institute, Harvard Medical School, Boston, MA, USA

2. Population of Visually Impaired   There are about 3 million visually impaired people in the USA   Expected to double by year 2020   People with visual impairment complain about reading or face recognition   They also watch as much television as the rest of us

3. Simulation

4. Solutions for Video Visibility  Enhancement Multiple types (next talk) Multiple types (next talk)  Substitution (voice-over description on 3rd channel)  Magnification Moving Closer Moving Closer Bigger Screen Bigger Screen Optical magnification (telescopes) Optical magnification (telescopes) Dynamic Magnification Dynamic Magnification

5. Magnification  Zoom commonly available in DVD players, TVs and Video Conferencing  Patient can dynamically control (as the video is playing)

7. Problems with Control of Magnification  Restriction of field of view means that only part of the scene is on the screen  Rapid changes in scenes in most movies does not allow for optimal manual control  Center of magnification therefore must be at the center of interest  The selection of the center of interest is of critical importance

8. POR (Point of Regard)  The problem is how to determine the center of interest and make it the center of interest (center of magnification)  It is done now for Movie-to-TV editing  We measure the eye movements of normally sighted people to determine where they are looking (Point of Regard)

9. Record Eye Movements To Get Point of Regard Subject viewing video in a comfortable seat without a bite bar Remote ISCAN Infrared pupil tracking device

10. DVD RS232 Remote ISCAN Recording Data File contains frame number and x,y coordinates Subject at 74 inches 16×9 format on a 4x3 NTSC 27” TV

11. Playback X,Y DVD Data File Zoom and Roam 27” TV Zoom

12. Video Clip Selection Table CategoryTitleTime Talk Show Quiz Show (1994) 6:40 Romance Shakespeare in Love (1998) 7:06 Sports Any Given Sunday (1999) 4:12 Documentary Blue Planet (2001) 8:14 News Network (1976) 4:02 Comedy Big (1988) 6:29 Total (min:sec) 37:29

13. We Need to Record Eye Movements from Several People  People do not blink at same time  Loss of tracking occurs at different times  Eyes “jump” (saccades) at different times  People may look at different objects  Should we use a single observer watching multiple times? different viewing strategy different viewing strategy  Merging of these multiple eye coordinate files

14. Types of Eye Movements DefinitionParameter Saccade High velocity jump from one position to another > 30 o /second Pursuit Smooth movement tracking a moving target < 30 o /second Fixation Eye position remains constant and centered on a target Segment of small motions of (max  x or  y <50 or r<0.5) terminated by Saccade, Pursuit or Artifact (blink)

15. 3 Types of Calibrations  Internal 5-Point ISCAN Calibration for POR calculations  External Calibration to equate POR Values to screen positions  POR recordings of purposeful pursuits, fixations and saccades

16. Saccade And Artifact Removal  Artifacts caused by Blinks Blinks Loss of tracking Loss of tracking Incorrectly handled timing interactions between ISCAN and DVD Incorrectly handled timing interactions between ISCAN and DVD  Filter to remove these

17. File of Fixation/Pursuit Segments After the initial filtering step, we are left with a file that defines segments of fixations Time Fixation gap

18. Detection of Time Overlapped Fixation Segments Five time overlapped segments from three observers Time Overlapped Fixation Segment Subject A Subject B Subject C Other Subjects 1 2 3 45 Arbitrary reference segment

19. Position Overlap Detection  Outlier segments determined by having mean 2 SD away from overall mean  Box represents ¼ screen dimension around the mean POR 256 0 Horizontal ISCAN “pixels” 4 Fixations Overlap Outlier Vertical ISCAN “pixels” 512

20. Smoothing Filter  Successive POR values that differ by a small amount from each other cause small shifts in center of magnification that are unnecessary and disturbing to the viewer  Smoothing Filter was implemented that eliminates small shifts in POR.  “Jump Threshold” set at 1/8 th total screen dimension  Successive PORs that move less than this jump threshold grouped together and averaged.

22. 2x Magnified

23. Rejection Statistics Gender and Age NRejected # Fixations used # Pursuits detected Male<40752%160321404 Female<40568%9644752 Female>45469%6777471 Male>45370%4914465

24. Why Not Always Use the Center?

25. Do People Look in The Same Place ? Other subject groups had similar results A single fixation segment cannot be counted multiple times Outlier

26. “Picture Over Picture” Can Address Loss of Context  Edge-detected (cartoon) image (original size) superimposed on magnified image (POP) Edge-detection of original size image in real time Edge-detection of original size image in real time  User controls level of magnification and on/off of edge-detected image

27. Playback X,Y DVD Data File Zoom and Roam 27” TV Zoom Edge Filter Video Mixer

28. Picture Over Picture  Viewer can see that there are two people in the scene  Viewer can turn edges on and off

29. POP Video

30. Issues To Address  Improved data analysis procedures  Cross-group analysis to investigate gender/age differences  Subject satisfaction experiment User control over magnification User control over magnification User control over edges User control over edges

31. Acknowledgments Shabtai Lerner Avni Vora James Barabas Dan Stringer Rob Giorgi Russell Woods Alex Bowers Jeonhoon Kim Doris Apfelbaum Morey Waltuck Supported by NIH Grants EY05957 and EY12890 Over 30 people who watched videos Lab Staff