J. Alan Atherton Michael Goodrich Brigham Young University Department of Computer Science April 9, 2009 Funded in part by Idaho National Laboratory And Army Research Laboratory 1

 Background  Related Work  Ecological Interface  User Study  Interface Changes from Study  Conclusions and Future Work 2

 What is a remote manipulator?  Applications USAR EOD Planetary Exploration 3

 Remotely operating a robot is difficult “Soda straw” — Maintaining situation awareness Time delay Mental workload  Why is this a problem? Collisions Slow Stressful Foster-Miller Talon 4

All images adopted from Yanco, H. A.; Drury, J. L. & Scholtz, J. “Beyond usability evaluation: analysis of human- robot interaction at a major robotics competition” Hum.-Comput. Interact., L. Erlbaum Associates Inc., 2004, 19,

 Background  Related Work  Ecological Interface  User Study  Interface Changes from Study  Conclusions and Future Work 6

Idaho National LaboratoryUMass Lowell Bruemmer, D. J. et al. “Shared understanding for collaborative control.” IEEE Transactions on Systems, Man and Cybernetics, Part A, 2005, 35, Yanco, H. A. et al. “Analysis of Human-Robot Interaction for Urban Search and Rescue.” Proceedings of the IEEE International Workshop on Safety, Security and Rescue Robotics,

INL / BYU AV Interface Ferland et al. - Sherbrooke C. W. Nielsen, M. A. Goodrich, and B. Ricks. “Ecological Interfaces for Improving Mobile Robot Teleoperation.” IEEE Transactions on Robotics and Automation. Vol 23, No 5, pp , October Ferland, F.; Pomerleau, F.; Dinh, C. T. L. & Michaud, F. “Egocentric and exocentric teleoperation interface using real-time, 3D video projection.” Proceedings of the 4th ACM/IEEE international conference on Human robot interaction, ACM, 2009,

NASA Viz Nguyen, L. A.; Bualat, M.; Edwards, L. J.; Flueckiger, L.; Neveu, C.; Schwehr, K...; Wagner, M. D. & Zbinden, E. “Virtual Reality Interfaces for visualization and control of remote vehicles” Autonomous Robots, 2001, 11, Kelly, A.; Anderson, D.; Capstick, E.; Herman, H. & Rander, P. “Photogeometric Sensing for Mobile Robot Control and Visualisation Tasks” Proceedings of the AISB Symposium on New Frontiers in Human-Robot Interaction, 2009 CMU Robotics Institute

 Background  Related Work  Ecological Interface  User Study  Interface Changes from Study  Conclusions and Future Work 10

 Requirements Ecological Increase situation awareness Manage workload  Existing Interfaces Lack depth information No manipulation support Not designed for real-time operation 11

12  Real-time remote manipulation

13

 Robot Build from kit, modify Player driver Motion planning for arm Swiss Ranger driver  Communication Integrate with INL’s system Network data transfer  User Interface OpenGL display Experiment automation 14 Robot Controller User Interface

 Background  Related Work  Ecological Interface  User Study  Interface Changes from Study  Conclusions and Future Work 15

Variant 1Variant 2Variant 3 Variant 4Variant 5Variant 6 3D + Video End Effector Video3D Joint Robot Control Visualization  Task: collect yellow blocks  30 participants  Between-subject comparison 16

 Reduce memorization effects  Minimize damage to arm  Quick change 17

18

19

 Joint control  View-dependent end effector control 20

21  Robot reaches for point  User moves point with joystick  Point movement depends on view orientation

3D + Vid. End eff. 3D + Vid. Joint 3D End eff. 3D Joint Video End eff. Video Joint 22

3D + Vid. End eff. 3D + Vid. Joint 3D End eff. 3D Joint Video End eff. Video Joint 3D + Vid. End eff. 3D + Vid. Joint 3D End eff. 3D Joint Video End eff. Video Joint 23 Collisions with posts, box, table Collisions with block in final adjustments

24

 Background  Related Work  Ecological Interface  User Study  Interface Changes from Study  Conclusions and Future Work 25

 Problems Alignment Time lag Cluttered 3D scan model  Changes Stereo camera exterior orientation Interactive robot arm calibration Simple Quickening Scan Pruning 26

 Interactive stereo camera calibration  Live robot arm calibration 27

28

29

30

 Background  Related Work  Ecological Interface  User Study  Interface Changes from Study  Conclusions and Future Work 31

 3D visualization supports SA  Video is faster  3D + video is a good tradeoff  3D + video might reduce workload 32

33  Head tracking  Ecological camera video  Haptics