1. It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 1/35 Lan Zhang, Xiang-Yang Li, Wenchao Huang, Kebin Liu, Shuwei Zong, Xuesi Jian, Puchun Feng, Taeho Jung, Yunhao Liu ACM. New York, NY, USA ©2014 Advanced Topics on Internet Of Things

2. Content 1.Background 2.iGaze Idea 3.Challenges and Solutions 4.System Design 5.Empirements and Evaluation 6.Future Work It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 2/35

3. Background  Collect information sensors  Can run Mobile App  Control, or retrieve data from, other  Support wireless technologies Bluetooth, WiFi, and GPS  May also have all the features of a smartphone It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 3/35 Smart Glasses

4. Scanning Distance < 3 Scanning Angle <30° High Computation Cost Displacement > 20cm QR code Image Recognition Hand gesture It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 4/35 Background Existing Technologies

5. Voice Commands Head Gesture Control TouchPad or Buttons Smart glasses are still looking for revolutionary applications Background Human Computer Interface It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 5/35

6. New Networking Mode.  Can understand the user's visual attention using an eye camera  Camera Can automatically connect to the target of interest  Can run on top of existing networking protocols, e.g., Wi-Fi. iGaze Idea It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 6/35

7. . Museum Obtain the detailed descriptions of art works by glazing at it instead. iGaze Idea It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 7/35

8. . A house can turn on a smart appliance only by taking a glance at them Device Control iGaze Idea It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 8/35

9. . When people look at a signboard or product, information can be sent to the potential customer immediately. Advertising iGaze Idea It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 9/35

10. . iGaze Idea It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 10/35

11. iGaze Principales Unidirectional Given an observer and his/her gaze vector, only the device vector to the correct visual target’s device is consistent with the gaze vector. iGaze Idea It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 11/35

12. Bidirectional Given a pair of users who are looking at each other, their gaze vectors have opposite directions. iGaze Idea iGaze Principales It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 12/35

13. iGaze Idea It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 13/35

14. Develop a low-cost glasses hardwarembedded with attention capture software Capture the gaze direction using the orientation sensor and eye camera. Question 1 : How to accurately capture the vision plane attention of a user in real time ? Answer : Challenges and Solutions It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 14/35

15. Visual Attention Detection Visual attention detection  Fixation A fixation duration window threshold for visual attention Challenges and Solutions It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 15/35

16.  Projection on the image plane is elliptical  Look ahead: the projection looks more like a circle  Look off to one side: it looks more close to an ellipse  estimate the pose of the iris circle by back- projecting the ellipse onto a circle in 3D space Determine Gaze Direction Challenges and Solutions It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 16/35

17.  Capture the direction between glasses  Leveraging Doppler effect caused by arbitrary mild movement of head- mounted speakers. Question 2 : How to match the target on device plane according to user’s visual attention? Answer : Challenges and Solutions It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 17/35

18. Challenges and Solutions Head Node or Shake It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 18/35 Determine Device Direction

19. System Design It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 19/35

20. System Design Gaze Vector Acquisition It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 20/35

21. System Design Device Vector Estimation It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 21/35

22. System Design Matching and Networking It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 22/35

23. System Design Application Layer It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 23/35

24. System Design Visual Attention Networking (VAN) Protocol It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 24/35

25. System Design Visual Attention Networking (VAN) Protocol It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 25/35

26. System Design Visual Attention Networking (VAN) Protocol It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 26/35

27. System Design Visual Attention Networking (VAN) Protocol It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 27/35

28. Empirements and Evaluation It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 28/35

29. Attention & Gaze Direction Accuracy Attention capture accuracy: ~95% Gaze direction accuracy: < 5° (91 %) < 10° (99 %) Empirements and Evaluation It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 29/35

30. Direction estimation is robust against different head gestures and patterns. Only one mild head gesture is sufficient for highly accurate estimation. Empirements and Evaluation Device Direction Accuracy It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 30/35

31. When the distance gets larger, the accuracy decreases Device Direction Accuracy Empirements and Evaluation It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 32/35

32. Computation Cost Energy Cost Computation Cost & Energy Consumption Empirements and Evaluation It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 33/35

33.  Multiple objects in the gaze direction.  Multiple users may be interested in a same object.  User may be interested in a moving object.  How to avoid insecure connection.  Aesthetic design. Future Work It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 34/35

34. It Starts with iGaze: Visual Attention Driven Networking with Smart Glasses Presentor: Aouaidjia Kamel 35/35 Thank you for listening