1. Team IRALAR Breanna Heidenburg -- Michael Lenisa -- Daniel Wentzel Advisor: Dr. Malinowski

2.  The Project ◦ Why is it important  The Goals ◦ System breakdown  Image recognition  Point transformation  User Interface  The Results

3. What is our project?

4.  Track a user’s eye and use the information to control a computer cursor

5.  Enhances Human Computer Interaction ◦ Speed of use ◦ Hands-free use

6.  3 Part System ◦ Image Processing Application ◦ Calibration and Mapping system ◦ GUI designed for gaze-based interaction  Systems developed concurrently and independently  Separate Applications at run-time

7. Hardware and Image Processing Application

8.  Hardware ◦ Camera  QuickCam Pro for Notebooks  Visible Spectrum Camera ◦ Polarizer  Tiffen 25mm polarizing filter  Removes glare from eye reflections ◦ Lighting  Diffuse LEDs  Slightly distracting to the user, but necessary to provide light for the camera

9.  LitEye LE-500 ◦ High resolution (SVGA) ◦ Color Display ◦ Translucent or opaque operation ◦ Stationary relative to user’s eye

10.  Real time pupil tracking system ◦ Developed in C using OpenCV image processing libraries ◦ Traditional image processing and blob tracking  Capabilities ◦ Locate and determine center of pupil in image ◦ Low light and high reflection environments ◦ All eye colors ◦ Data logging and static test modes ◦ Packaged into self contained Windows installer for easy deployment onto any computer

11.  The Process ◦ Capture Image ◦ Extract Red Channel ◦ Smooth image ◦ Apply binary threshold ◦ Locate blobs ◦ Reject false positives ◦ Determine center of pupil blob ◦ Adapt threshold ◦ Repeat

12.  Summary ◦ The Good  Dynamically adapts to changing lighting conditions and eye types  Maintains performance in low-light and specularly noisy conditions ◦ The Bad  Still relies on Logitech camera drivers  Extreme reflections still cause problems

13. Raw CapturePre-Processed Image Completed Recognition

14.  Examples of performance in poor conditions Low Light Difficult False Positive

15. Calibration and point mapping

16.  System for mapping the location of the center of the pupil to a pixel on a computer screen  Reasons ◦ Geometry  The eye is not flat but a screen is not ◦ User Customization  All eyes are different  Everyone wears the HMD differently ◦ User Training  Calibration system also acts as a quick tutorial

17.  3 dimensional best fit plane ◦ Currently using a 4 th degree best fit X pix = A 1 + X eye *B 1 + Y eye *C 1 Y pix = A 2 + X eye *B 2 + Y eye *C 2  Calibration sub-system determines these coefficients

18.  How do we solve the problem? ◦ Multiple Variable Linear Regression – Least Squares Y = B 0 + B 1 x 1 + … + B k x k ◦ Uses matrix algebra to obtain a coefficient matrix B[] = (X’X) -1 X’Y

19.  Results ◦ pixelX = 224.9 + 4.8*eyeX – 2.9*eyeY ◦ pixelY = 1612.5 + 2.0*eyeX – 5.6*eyeY B[] = (X’X) -1 X’Y

20.  How do we click? ◦ Monitor eye movements ◦ Identify pauses ◦ When eye position is within a small percentage for a certain amount of time ◦ Currently set at 5 frames (~200 mS) ◦ Generally, it takes 230 mS for a hand to click a mouse.

21. Communications and Custom GUI interface

22.  Why is it important? ◦ Allows multiple processes to communicate ◦ Allows relay of time sensitive information  UDP vs. TCP ◦ UDP -> ‘Connectionless’ ◦ TCP -> ‘Connection oriented’  Multi-threading ◦ Necessary for running multiple pieces of code in a single process

23. Multi-threading issues present

24.  Multi-threading ◦ Issues with public variable usage ◦ Solutions we are using:  Critical Section  Raises thread priority (does not allow for interrupts) Receive request for data (over UDP) Raise Thread priority Lower Thread Priority Reply to request Read Variable

25.  Custom GUI for Gaze Tracking Applications  Why? ◦ Gaze tracking accuracy limited by inherent properties of human vision system ◦ Traditional GUI too small and intrusive for use with transparent HMD ◦ Demonstrate applications of gaze tracking

26.  Achieved via a combination of Hardware and Software  Software: WPF & C# ◦ XAML (eXtensible Application Markup Language) ◦ Very similar to XML ◦ Uses ‘code-behind’ in a similar style to javascript  ‘code-behind’ is in C#

27.  Multiple applications have been created within the interface  Multiple screens for functionality testing ◦ even games  Additional ability to minimize interface and interact with external applications is present

28. Calibration Screen Main Screen

29. What did our test results show?

34. Who helped us out?

35.  Northrop Grumman  LitEye  Dr. Malinowski and the EE faculty  Mr. Mattus & Mr. Schmidt