1. First-Person Tele- Operation of a Humanoid Robot Lars Fritsche, Felix Unverzagt, Jan Peters and Roberto Calandra

2. OUTLINE Introduction Hardware Setting Control Architecture Operator-Robot Correspondence Experimental Results Discussion & Conclusion

3. INTRODUCTION Augmented reality: A live view of a real-world environment whose elements are augmented by computer-generated sensory input -- Provides a first-person vision for the robot operator Tele-Operation of Robots: Operating the robot at a distance. Provides the possibility of performing complex tasks in environments inaccessible or too dangerous for humans. Examples: Epidemic areas Surgical applications Damaged nuclear plants (e.g., Fukushima) Space & underwater exploration.

4. HARDWARE SETTING Microsoft Kinect v2 Collect human motion data Oculus Rift Headset Augmented reality headset SensorGlove Detect grasping motions, provide haptic feedback iCub The humanoid robot

5. Microsoft Kinect v2 Combines a camera and a depth sensor to retrieve 3D-skeleton data from the operator. From https://en.wikipedia.org/wiki/Kinect "First-person tele-operation of a humanoid robot“

6. Oculus Rift Headset A virtual reality & augmented reality headset with OLED displays and accurate, low-latency positional head tracking. Sensor Glove Track operator’s finger motions and use vibration motors to induce haptic feedback on operator’s fingertips according to the pressure measured on the fingertips of the robot. From https://www1.oculus.com/order/

7. iCub The iCub is a humanoid robot of 1 meter tall and weighting 24 kilograms. It has 53 degrees of freedom (DOFs), 30 of them are used during the tele-operation. The robot has a cameras for each eye that provide images at the maximum resolution of 640 x 480 pixels. Additionally, the robot has tactile sensors on the fingertips to measure the pressure From http://kokomad.com/icub-robot/

8. CONTROL ARCHITECTURE "First-person tele-operation of a humanoid robot“.

9. YARP: Yet Another Robot Platform YARP (Yet Another Robot Platform) is a software package for interconnecting sensors, processors, and actuators in robots. iCub Uses YARP as a way to define input and output ports for its control. Advantage: Modular and easily extensible.

10. Control Signals Filtering The signal retrieved are noisy and would lead to very jerky movements if applied directly to the robot that could endanger the motors. Use Butterworth filters Two individual filters are designed: The filter for Kinect is chosen to be of eighth-order The filter for both SensorGlove and Oculus Rift are of fourth-order

11. Safety Routines Multiple safety routines had to be implemented to guarantee a robust tele-operation in different conditions and even in the presence of non-expert operators. Thresholds are implemented for each joint to limit the workspace A maximum step size is introduced to limit the command signal such that the robot can not exceed its own bounds by force.

12. Mapping of the Arm Joints 4 DOFs on each arm Mapping of the Torso Joints 3 DOFs Mapping of the Head Joints 5 DOFs Mapping of the Hand Joints 7 DOFs on each hand Operator-Robot Correspondence

13. Experimental Results: Head Motion

14. Head Motion "First-person tele-operation of a humanoid robot“.

15. Experimental Results

16. Time delay for the Kinect Time delay for the Oculus Rift Time delay for the Sensor Glove "First-person tele-operation of a humanoid robot“.

17. DISCUSSION & CONCLUSIONS This paper presented an affordable approach to the first-person tele-operation of humanoid robot by augmented-reality motion-tracking and tactile feedback using the Microsoft Kinect, Oculus Rift and SensorGlove. Augmented reality on the humanoid robot may help to achieve a more natural and efficient tele-operation. Shortcoming: High latency for arm movement in tele-operation.

18. REFERENCE Fritsche, Lars, et al. "First-person tele-operation of a humanoid robot." Humanoid Robots (Humanoids), 2015 IEEE-RAS 15th International Conference on. IEEE, 2015. “Official Kinect website”, http://www.microsoft.com/en-us/kinectforwindows/, accessed: 2016-03-19. “First-Person Tele-Operation of the Humanoid Robot iCub (HUMANOIDS 2015)“, https://www.youtube.com/watch?v=gtkrPhcYhyI “Controlling the head of the iCub using the Oculus Rift”, https://www.youtube.com/watch?v=ZA25VHkfQjE Metta, Giorgio, Paul Fitzpatrick, and Lorenzo Natale. "YARP: yet another robot platform." International Journal on Advanced Robotics Systems 3.1 (2006): 43-48. “Official YARP website”, http://www.yarp.it/, accessed: 2016-03-21. “Wikipedia: Augmented reality”, https://en.wikipedia.org/wiki/Augmented_reality/, accessed: 2016-03-21. “Wikipedia: Butterworth filter”, https://en.wikipedia.org/wiki/Butterworth_filter/, accessed: 2016-03-21.

19. Thank you.