2. Emerging Applications of VR Electrical and Computer Engineering Dept.

3. Emerging applications of VR Emerging applications of VR In manufacturing (especially virtual prototyping, assembly verification, ergonomics, and marketing); In robotics (programming, teleoperation, space robotics); In data visualization (volume visualization, oil and gas exploration, volumetric displays); Other areas.

4. (UK VR Forum survey, 2000) VR penetration in non-medical fields VR penetration in non-medical fields % - companies using, % - companies using, Experimenting or Experimenting or Considering VR Considering VR

5. Classes of VR applications (UK VR Forum survey, 2000)

6. Main benefits of using VR (UK VR Forum survey, 2000)

7. VR in Manufacturing VR in Manufacturing

8. VIRTUAL PROTOTYPING

9. GHOST Free Form Application  Free Form sculpting is a new type of Human- Computer interaction;  It functions as a convenient GUI that sits on top of the GHOST library;  The GUI is based on static and dynamic bar menus “dynabars”;  It allows model export to CAM machines for rapid prototyping, as well as to animation packages for computer character animation.  Requires dual Pentium II (> 300 MHz), 512 MB RAM, high-end hardware graphics acceleration, 1024x768 screen resolution (or better). Free Form Sculpting GHOST SDK PHANToM Drivers

10. FreeForm Initial screen Menu bar Tool bar Dynabar Status bar Work area border border The center block is “digital clay”

11. FreeForm wire-cut mode Wire Cut Menu Sketch planes

12. Free Form wire cut Drawing on the cut plane Clay after Outside wire cut Clay after Inside wire cut Cut Outside button Cut Inside button

13. FreeForm carving tools Carving a hole Carving a corner Getting carried away…. Smoothing while carving

14. Example – Making of Saint Fruition Artist’s sketch Rough initial clay model

15. Example – Making of Saint Fruition Finished body (clay) Wire cut arrow for support (clay) Finished support (clay)

16. Example – Making of Saint Fruition 5 ft Aluminum statue 5 ft Aluminum statue Digital clay statue Digital clay statue

17. Example – Making of Saint Fruition Key frame animation of statue Key frame animation of statue Statue model used for Statue model used for animation (Maya) animation (Maya) Textured statue used for Textured statue used for animation (Maya) animation (Maya) Link to VC 9.1 on book CD

18. ASSEMBLY VERIFICATION

19. Another stage in product development when the prototype is made of several parts; University of Washington developed the “Virtual Assembly Design Environment to verify CAD design assemblies; Parts geometry and attributes are imported from CAD into VADE then the assembly is analyzed and robots are programmed Assembly verification Collision detection through swept volumes Design modification in VR

20. Parts making up car exterior have varying tolerances. Tighter tolerances are more esthetically pleasing but also cost more. What is good enough? Inspection is done in inspection rooms using stripped lights. Same can be done on a virtual car ahead of real production Assembly verification – Car body tolerances Virtual Inspection room Real inspection room

21. Assembly verification Unacceptable tolerances discovered in the virtual inspection room in the virtual inspection room Researchers in UK developed the Visualization of the Impact of Tolerance Allocation (VITAL) and tested it on a prototype Rover R75; They constructed several models with various tolerances; by shining the virtual car body with stripped light looking at discontinuities;

22. ERGONOMY ANALYSIS

23. Ergonomic Analysis Jack is a an intelligent agent humanoid used in ergonomic analysis; the Task Analysis Toolkit computes lower-back effort and energy consumption – relates to worker fatigue Link to VC 9.2 on book CD

24. Ergonomic Analysis - continued Once a prototype is done, it has to be tested for ease of use (ergonomic analysis); One such product is VirtualANTHROPOS developed in Germany to test the ease of use of tractor cabins

25. Ergonomic Analysis An avatar controlled by the user is interacting with the virtual cabin while the system computes joint discomfort levels using ERGONAUT (an ergonomic analysis tool)

26. Ergonomic Analysis Another use of VirtualANTHROPOS is to visualize reach envelopes; The user can drive the avatar in real time using a wireless body suit; Link to VC 9.3 on book CD

27. PERSONAL TRAINING

28. Personnel Training Training in airplane maintenance – task has a cognitive component (manuals) and a tool/part manipulation component – and they are sequential.

29. Personnel Training Task-related information is placed directly in the scene using augmented reality. Results in faster information retrieval and enhanced associative memory. System uses vision-based tracking to recognize worker’s view and places text in relation to objects;

30. Personnel Training Training system detect removal of cover and labels parts underneath; then it detects the cap was removed and changes dynamically the text to “4. Press to test”. If the test fails then additional areas of interest are highlighted (“Filter Bypass”)

31. Steve: Sistema de entrenamiento con un personaje de apoyo

40. Virtual Reality Welding Simulator

41. VR MARKETING APPLICATIONS

42. VR Marketing Applications Citröen uses virtual showrooms

43. VR Marketing Applications

45. ROBOTIC APPLICATIONS

46. VR applications in Robotics/manufacturing relate to several areas:  CAD design and robot programming, making the process more intuitive;  Teleoperation (control at a distance) alleviating problems related to poor visibility and large time delays;  Multiplexed teleoperation, acting as a filter of particular robot kinematics; Robots are also used in VR in haptic interfaces (discussed earlier in our course). Robotics Applications Robotics Applications

47. The “multi-modal teaching advisor helps novice operators program welding paths for industrial robots; It runs on a PC networked with trackers and laser range finder; Calculates the difference between the pre-computed (optimal) path and user’s input on the teach pendant. This is presented graphically on the user’s HMD Robot programming Robot programming

48. Robotic programming - continued Research done in Germany for off- line robot programming done in VR, with the programmer immersed in the task he is programming; The programmer specifies the trajectories, and the simulation performs optimized collision detection; Validation is done at run time when the real robot is controlled using the same computer and real sensor data is used to fine-tune the VR-generated program.

49. TELEOPERATION

50. VR Robotics Applications - continued VR Robotics Applications - continued (Burdea, 1999) Operator VR GUI Degraded video feedback Research at University of Tokyo for the teleoperation of robots in smoke-filled remote environments. Over-imposes the visual scene from the remote robot with the virtual scene of a kinematically identical robot. Thus VR acts as a guide to allow teleoperation.

51. Teleoperation with large time delays Teleoperation with large time delays Research at NASA developed a VR-based teleoperation to allow operation despite large time delays. Works by controlling a “phantom robot” which responds instantaneously to the operator. Allows preview of the move, before it is executed.

52. Teleoperation with large time delays - continued Teleoperation with large time delays - continued Research at NASA drove the Mars rover using VR-based teleprogramming; This was used to send high-level macro commands based on the simulation of a virtual rover on a virtual Mars surface. This overcame a 20 minute time delay!

53. Supervisory control Supervisory control Researchers in Germany developed a way to naturally controlling robots through avatars; The users is immersed in VR and sees a scene with avatars to which he is mapped; He interacts through gestures (measured by a sensing glove)

54. Supervisory control Supervisory control A real robot then interacts with the remote real environment; If the task is visual inspection then real images from the remote site can be overlaid on the virtual scene, and thus seen by the user; Real remote robot Real remote robot Virtual robot with viewfinder Virtual robot with viewfinder

55. VR Robotic Teleoperation VR Robotic Teleoperation Research at Jet Propulsion Lab (California) allows the teleoperation of a remote robot indirectly by controlling a motion- guide trajectory, which the robot is then constrained to follow. Motion guide Task line Remote robot arm Text mode task editor

56. VR Robotics Applications - continued VR Robotics Applications - continued Research at University of Paris to allow multiplexed (one-to-many) teleoperation of kinematically dissimilar robotic arms; VR acts as a high-level filter masking the detailed slave robot configuration; Translator then converts user actions to robot actions. Remote robot arms Operator task-level GUI

57. Supervisory control - continued Supervisory control - continued

58. INFORMATION VISUALIZATION

59. INFORMATION VISUALIZATION INFORMATION VISUALIZATION Represents the transformation of abstract data into 3D scenes; The information visualization pipeline allows the user to control the view to the scene using an input device and select an area of interest; The data extraction loop is asynchronous, so as to maintain interactivity. It reads user input from a FIFO buffer; Time-varying data represent a complex case, as user time may not coincide with the time clock used in visualizing the time-dependent data.

60. Oil and Gas Exploration and Well Management Oil and Gas Exploration and Well Management

62. VOLUME GRAPHICS

63. Volumetric graphics Volumetric graphics In this class we learned about surface-based (polygonal or spline) rendering only. This leaves the interior of virtual objects hollow; Volume graphics renders the surface as well as the interior of objects, called “voxels.” Surface rendered object Same object rendered volumetricly voxels

64. Ray casting to create a 2-D image from volumetric data Ray casting to create a 2-D image from volumetric data Rays View Plane Tri-linear interpolation Calculate gradients and lighting

65. Volumetric graphics advantages Volumetric graphics advantages Much richer dataset; Objects appear more real; Can be displayed on same displays as surface-based models Volumetric graphics disadvantages Volumetric graphics disadvantages Much larger memory requirement (order of GB); Requires special boards to be real-time; Require special displays if auto-stereoscopy is desired; Much less used today – small software support base.

66. Volumetric graphics Link to VC 9.7 on book CD

67. Volumetric Graphics Hardware Consist of volumetric rendering boards and of volumetric displays; VolumePro 1000 is a graphics accelerator sold by TeraRecon Inc. renders 512 x 512 x 512 voxels at 30 frames/second using a Mitsubishi chip. For surface geometry data it works together with the graphics board installed on the same PC

68. VolumePro Rendering Pipeline VolucePro 1000 processed the 3D data through ray casting from a view plane. Rays pick up color and opacity information by tri- linear interpolation to the nearest lattice point Gradients are then computed

69. VOLUMETRIC DISPLAYS Early models used LED matrix panel that translates back-forth on rails; User can see stereo with bare eyes. Due to eye inertia, the image appears to float in space; But they had low resolution, noisy, monochrome (red LEDs); Did not have a 360º viewing area LED Object section

70. (Favarola et al., 2001) DMD-based volumetric display Projection Projection DMD engine motor Condenser lens lens lamp Coldmirror Rotatingmirrors Translucent screen fold foldmirror The display produces 200 disk-shaped slices each refreshed at 20 Hz; Resolution 768x768, 8 colors; 10”-diameter spherical image; 360º x 180º viewing angle.

71. Actual system assembly Actual system assembly (Favarola et al., 2002)

72. Auto-stereoscopic 3-D Display produced by Actuality Systems (www.actuality.com)

73. New Interaction Techniques with Volumetric Displays New Interaction Techniques with Volumetric Displays (Balakrishnan et al, 2001)

74. Depth perception studies done at University of Toronto (2006) Depth perception studies done at University of Toronto (2006) Compared desk-top mono, stereo and stereo with head tracking with depth perception in a volumetric display Tasks were 1) judge the position of a sphere in a cube; 2) decide if two objects were going to collide or pass by each- other http://www.dgp.toronto.edu/~tovi/#Publications

75. Selection techniques study done at University of Toronto (2006) The display was this time only volumetric, but object selection technique varied Within a cluttered virtual world a ray technique will intersect several objects not just the target object; Selection by point cursor, depth ray, lock ray, flower ray and smart ray (below) Movement times were largest for the smart ray and smallest for depth ray; However error rates for new techniques were smaller (13% depth ray, 11% lock ray and flower ray and 10% smart ray) vs. point cursor (21%). http://www.dgp.toronto.edu/~tovi/#Publications

76. Aplicaciones de RV actuales u Grounds maintenance (lawn, fountain, pond, parking lot) u Environmental Health & Safety (waste removal, HAZMAT compliance, janitorial, cleaning) u Mechanical Systems (Elevators, HVAC) u Power Systems (normal power supply, emergency power supply, Generators) u Building Systems (monitoring, security, locks, access control) u Safety Systems (sprinkler, fire, evacuation, signage) u Space Management (office layout, furniture requirements) u Electrical & Lighting u Plumbing u New construction u Decoration (signage, plants, other general appearance) u Physical Asset Management

78. IEEE VR 2014 Papers u Multi-modal and Immersive Displays u Augmented Reality u 3D Capturing and Image-based Rendering u Walking and Locomotion u Usability and Performance u Virtual Humans and Avatars

79. IEEE VR 2014 u http://ieeevr.org/2014/ http://ieeevr.org/2014/ u http://ieeevr.org/2014/videos.html http://ieeevr.org/2014/videos.html

80. MiddleVR for Unity u http://www.imin-vr.com/middlevr/ http://www.imin-vr.com/middlevr/ u Middle VR añade ciertas capacidades a Unity de interaccion e inmersion u Unity cubre aspectos para desarrollo de ambientes de VR u http://www.youtube.com/watch?v=nxfBI2- 0c5o http://www.youtube.com/watch?v=nxfBI2- 0c5o

81. IEEE VR 2013 u http://ieeevr.org/2013/