2. Emerging Applications of VR Electrical and Computer Engineering Dept.

3. Emerging applications of VR Emerging applications of VR In manufacturing (especially virtual prototyping, assembly verification, ergonomics, and marketing); In robotics (programming, teleoperation, space robotics); In data visualization (volume visualization, oil and gas exploration, volumetric displays); Other areas.

4. (UK VR Forum survey, 2000) VR penetration in non-medical fields VR penetration in non-medical fields % - companies using, % - companies using, Experimenting or Experimenting or Considering VR Considering VR

5. Classes of VR applications (UK VR Forum survey, 2000)

6. Main benefits of using VR (UK VR Forum survey, 2000)

7. VR in Manufacturing VR in Manufacturing

8. GHOST Free Form Application  Free Form sculpting is a new type of Human- Computer interaction;  It functions as a convenient GUI that sits on top of the GHOST library;  The GUI is based on static and dynamic bar menus “dynabars”;  It allows model export to CAM machines for rapid prototyping, as well as to animation packages for computer character animation.  Requires two Quad Core processors, 4 to 16 GB RAM, high-end hardware graphics acceleration, 1024x768 screen resolution (or better). Free Form Sculpting GHOST SDK PHANToM Drivers

9. FreeForm Initial screen Menu bar Tool bar Dynabar Status bar Work area border border The center block is “digital clay”

10. FreeForm wire-cut mode Wire Cut Menu Sketch planes

11. Free Form wire cut Drawing on the cut plane Clay after Outside wire cut Clay after Inside wire cut Cut Outside button Cut Inside button

12. FreeForm carving tools Carving a hole Carving a corner Getting carried away…. Smoothing while carving

13. Example – Making of Saint Fruition Artist’s sketch Rough initial clay model

14. Example – Making of Saint Fruition Finished body (clay) Wire cut arrow for support (clay) Finished support (clay)

15. Example – Making of Saint Fruition 5 ft Aluminum statue 5 ft Aluminum statue Digital clay statue Digital clay statue

16. Example – Making of Saint Fruition Key frame animation of statue Key frame animation of statue Statue model used for Statue model used for animation (Maya) animation (Maya) Textured statue used for Textured statue used for animation (Maya) animation (Maya) Link to VC 9.1 on book CD

17. Another stage in product development when the prototype is made of several parts; University of Washington developed the “Virtual Assembly Design Environment” (VADE) to verify CAD design assemblies; Parts geometry and attributes are imported from CAD into VADE then the assembly is analyzed and robots are programmed Assembly verification Collision detection through swept volumes Design modification in VR

18. Parts making up car exterior have varying tolerances. Tighter tolerances are more esthetically pleasing but also cost more. What is good enough? Inspection is done in inspection rooms using stripped lights. Same can be done on a virtual car ahead of real production Assembly verification – Car body tolerances Virtual Inspection room Real inspection room

19. Assembly verification Unacceptable tolerances discovered in the virtual inspection room in the virtual inspection room Researchers in UK developed the Visualization of the Impact of Tolerance Allocation (VITAL) and tested it on a prototype Rover R75; They constructed several models with various tolerances; by shining the virtual car body with stripped light looking at discontinuities;

20. Ergonomic Analysis Jack is a an intelligent agent homanoid used in ergonomic analysis; the Task Analysis Toolkit computes lower-back effort and energy consumption – relates to worker fatigue Link to VC 9.2 on book CD

21. Ergonomic Analysis - continued Once a prototype is done, it has to be tested for ease of use (ergonomic analysis); One such product is VirtualANTHROPOS developed in Germany to test the ease of use of tractor cabins

22. Ergonomic Analysis An avatar controlled by the user is interacting with the virtual cabin while the system computes joint discomfort levels using ERGONAUT (an ergonomic analysis tool)

23. Ergonomic Analysis Another use of VirtualANTHROPOS is to visualize reach envelopes; The user can drive the avatar in real time using a wireless body suit; Link to VC 9.3 on book CD

24. Personnel Training Training in airplane maintenance – task has a cognitive component (manuals) and a tool/part manipulation component – and they are sequential.

25. Personnel Training Task-related information is placed directly in the scene using augmented reality. Results in faster information retrieval and enhanced associative memory. System uses vision-based tracking to recognize worker’s view and places text in relation to objects;

26. Personnel Training Training system detect removal of cover and labels parts underneath; then it detects the cap was removed and changes dynamically the text to “4. Press to test”. If the test fails then additional areas of interest are highlighted (“Filter Bypass”)

27. VR Marketing Applications Citröen uses virtual showrooms

28. VR Marketing Applications

30. VR applications in Robotics/manufacturing relate to several areas:  CAD design and robot programming, making the process more intuitive;  Teleoperation (control at a distance) alleviating problems related to poor visibility and large time delays;  Multiplexed teleoperation, acting as a filter of particular robot kinematics; Robots are also used in VR in haptic interfaces (discussed earlier in our course). Robotics Applications Robotics Applications

31. The “multi-modal teaching advisor helps novice operators program welding paths for industrial robots; It runs on a PC networked with trackers and laser range finder; Calculates the difference between the pre-computed (optimal) path and user’s input on the teach pendant. This is presented graphically on the user’s HMD Robot programming Robot programming

32. Robotic programming - continued Research done in Germany for off- line robot programming done in VR, with the programmer immersed in the task he is programming; The programmer specifies the trajectories, and the simulation performs optimized collision detection; Validation is done at run time when the real robot is controlled using the same computer and real sensor data is used to fine-tune the VR-generated program.

33. VR Robotics Applications - continued VR Robotics Applications - continued (Burdea, 1999) Operator VR GUI Degraded video feedback Research at University of Tokyo for the teleoperation of robots in smoke-filled remote environments. Over-imposes the visual scene from the remote robot with the virtual scene of a kinematically identical robot. Thus VR acts as a guide to allow teleoperation.

34. Teleoperation with large time delays Teleoperation with large time delays Research at NASA developed a VR-based teleoperation to allow operation despite large time delays. Works by controlling a “phantom robot” which responds instantaneously to the operator. Allows preview of the move, before it is executed.

35. Teleoperation with large time delays - continued Teleoperation with large time delays - continued Research at NASA drove the Mars rover using VR-based teleprogramming; This was used to send high-level macro commands based on the simulation of a virtual rover on a virtual Mars surface. This overcame a 20 minute time delay!

36. Supervisory control Supervisory control Researchers in Germany developed a way to naturally controlling robots through avatars; The users is immersed in VR and sees a scene with avatars to which he is mapped; He interacts through gestures (measured by a sensing glove)

37. Supervisory control Supervisory control A real robot then interacts with the remote real environment; If the task is visual inspection then real images from the remote site can be overlaid on the virtual scene, and thus seen by the user; Real remote robot Real remote robot Virtual robot with viewfinder Virtual robot with viewfinder

38. VR Robotic Teleoperation VR Robotic Teleoperation Research at Jet Propulsion Lab (California) allows the teleoperation of a remote robot indirectly by controlling a motion- guide trajectory, which the robot is then constrained to follow. Motion guide Task line Remote robot arm Text mode task editor

39. VR Robotics Applications - continued VR Robotics Applications - continued Research at University of Paris to allow multiplexed (one-to-many) teleoperation of kinematically dissimilar robotic arms; VR acts as a high-level filter masking the detailed slave robot configuration; Translator then converts user actions to robot actions. Remote robot arms Operator task-level GUI

40. Supervisory control - continued Supervisory control - continued

41. INFORMATION VISUALIZATION INFORMATION VISUALIZATION Represents the transformation of abstract data into 3D scenes; The information visualization pipeline allows the user to control the view to the scene using an input device and select an area of interest; The data extraction loop is asynchronous, so as to maintain interactivity. It reads user input from a FIFO buffer; Time-varying data represent a complex case, as user time may not coincide with the time clock used in visualizing the time-dependent data.

42. Oil and Gas Exploration and Well Management Oil and Gas Exploration and Well Management

44. Volumetric graphics Volumetric graphics In this class we learned about surface-based (polygonal or spline) rendering only. This leaves the interior of virtual objects hollow; Volume graphics renders the surface as well as the interior of objects, called “voxels.” Surface rendered object Same object rendered volumetricly voxels

45. Ray casting to create a 2-D image from volumetric data Ray casting to create a 2-D image from volumetric data Rays View Plane Tri-linear interpolation Calculate gradients and lighting

46. Volumetric graphics advantages Volumetric graphics advantages Much richer dataset; Objects appear more real; Can be displayed on same displays as surface-based models Volumetric graphics disadvantages Volumetric graphics disadvantages Much larger memory requirement (order of GB); Requires special boards to be real-time; Require special displays if auto-stereoscopy is desired; Much less used today – small software support base.

47. Volumetric graphics Link to VC 9.7 on book CD

48. Volumetric Graphics Hardware Consist of volumetric rendering boards and of volumetric displays; VolumePro 1000 is a graphics accelerator sold by TeraRecon Inc. renders 512 x 512 x 512 voxels at 30 frames/second using a Mitsubishi chip. For surface geometry data it works together with the graphics board installed on the same PC

49. VolumePro Rendering Pipeline VolucePro 1000 processed the 3D data through ray casting from a view plane. Rays pick up color and opacity information by tri- linear interpolation to the nearest lattice point Gradients are then computed

50. VOLUMETRIC DISPLAYS Early models used LED matrix panel that translates back-forth on rails; User can see stereo with bare eyes. Due to eye inertia, the image appears to float in space; But they had low resolution, noisy, monochrome (red LEDs); Did not have a 360º viewing area LED Object section

51. Volumetric Display Volumetric Display According to Wikipedia, a Volumetric Display is defined as a device that “creates 3-D images by emission, scattering or relaying of illumination from well defined regions in the (x, y, z) space.” Almost all Volumetric Displays are autostereoscopic. Currently, Volumetric Displays are only available to major companies, the military, and certain schools for academic purposes. They are still unavailable for the general public. There are two main types of Volumetric Displays: Swept-Surface Displays, and Static Volume Displays. Rely on the phenomena known as persistence of vision.

52. Persistence of Vision Persistence of Vision occurs when a human being is exposed to an image for a very short amount of time, usually nanoseconds. The result is around a millisecond of sight. This occurs because the chemical reaction time it takes for a human to process an image is much longer than the speed of light.

53. Swept-Surface Displays Also known as Swept-Volume Displays Rely on a rotating source or surface. Project a 2-D image that constantly changes based on the orientation of the rotating source or surface. An example: the Perspecta, created by Actuality Systems, Inc.

54. Perspecta by Actuality Systems, Inc.

55. Unlike Swept-Surface Displays, Static Volume Displays do not rely on motion. Some use pulsating, concentrated, Infrared Lasers that pulse around 100 times per second to establish floating balls of plasma in space which cause a clicking sound. The exact point is known as a voxel (similar to a pixel would be in 2-D, except 3-D) Others use lasers to cause visible radiation with a gas, liquid, or solid. Static Volume Displays

56. Use slideable mirrors and lenses to redirect the location of the voxel that are usually built in. Burton Inc, Keio University and AIST has made a true 3D display.

57. Disadvantages Suck up a lot of bandwidth Unable to deal with occlusion and opacity without limiting the FOV: Image appears distorted The cost of a volumetric display is much greater than that of its closest competitors, such as virtual retinal displays (retinal scanning devices). They use hundreds of times more light and power.

58. How far are we from Star Wars A volumetric display device is a graphical display device that forms a visual representation of an object in three physical dimensions, as opposed to the planar image of traditional screens that simulate depth through a number of different visual effects

59. The Graphics Lab at the University of Southern California has designed an easily reproducible, low-cost 3D display system with a form factor that offers a number of advantages for displaying 3D objects in 3D. The display is: u autostereoscopic - requires no special viewing glasses u omnidirectional - generates simultaneous views accomodating large numbers of viewers u interactive - can update content at 200Hz

60. The display consists of A high-speed video projector A spinning mirror covered by a holographic diffuser A FPGA circuitry to decode specially rendered DVI video signals A standard programmable graphics card to render over 5,000 images per second of interactive 3D graphics, projecting 360- degree views with 1.25 degree separation up to 20 updates per second

61. ANISOTROPIC SPINNING MIRROR

62. u 3D display uses an anisotropic holographic diffuser bonded onto a first surface mirror. u Horizontally, the mirror is sharply specular to maintain a 1.25 degree separation between views. u Vertically, the mirror scatters widely so the projected image can be viewed from multiple heights. u http://gl.ict.usc.edu/Research/3DDisplay/ http://gl.ict.usc.edu/Research/3DDisplay/

63. About Encoding The FPGA takes each 24-bit color frame of video and displays each bit sequentially as separate frames. Thus, if the incoming digital video signal is 60Hz, the projector displays 60×24 = 1,440 frames per second. To achieve even faster rates, we set the video card refresh to rates of 180-240Hz.

64. Sony RayModeler u http://blog.discover.sonystyle.com/360-degree-3d-prototype-takes-gaming- to-new-levels http://blog.discover.sonystyle.com/360-degree-3d-prototype-takes-gaming- to-new-levels u There is not much information and no paper for the device. However, we can see the refresh in the video, reveals that is kind of TV technique

65. We can only guess from what we can see. In the 2 snapshots from the SONY website. We can see RayModeler did not create 3D scene just 360 show pictures to 360 degree. Maybe kind of raster- based view control

66. (Favarola et al., 2001) DMD-based volumetric display Projection Projection DMD engine motor Condenser lens lens lamp Coldmirror Rotatingmirrors Translucent screen fold foldmirror The display produces 200 disk-shaped slices each refreshed at 20 Hz; Resolution 768x768, 8 colors; 10”-diameter spherical image; 360º x 180º viewing angle.

67. Actual system assembly Actual system assembly (Favarola et al., 2002)

68. Auto-stereoscopic 3-D Display produced by Actuality Systems (www.actuality.com)

69. Comparison of Volume Displays Light Field DisplaySony RayModelerPerspecta Core StructureSpinning MirrorSpecial arranged LED Rotating Screen ShapeSphereCylinderSphere resolution768*768*28896*128768*768*198 Contradictionhigh low ColorBW by 1 face mirror Multicolor by multi- face mirror Colors8 Color Fresh Rate20 Scenes/sec24 Scenes/sec InteractivehandleGesture, Game padContact, Gesture priceClaim to be a cheap Solution

70. New Interaction Techniques with Volumetric Displays New Interaction Techniques with Volumetric Displays (Balakrishnan et al, 2001)

71. Depth perception studies done at University of Toronto (2006) Depth perception studies done at University of Toronto (2006) Compared desk-top mono, stereo and stereo with head tracking with depth perception in a volumetric display Tasks were 1) judge the position of a sphere in a cube; 2) decide if two objects were going to collide or pass by each- other http://www.dgp.toronto.edu/~tovi/#Publications

72. Selection techniques study done at University of Toronto (2006) The display was this time only volumetric, but object selection technique varied Within a cluttered virtual world a ray technique will intersect several objects not just the target object; Selection by point cursor, depth ray, lock ray, flower ray and smart ray (below) Movement times were largest for the smart ray and smallest for depth ray; However error rates for new techniques were smaller (13% depth ray, 11% lock ray and flower ray and 10% smart ray) vs. point cursor (21%). http://www.dgp.toronto.edu/~tovi/#Publications

73. http://charafantah.wordpress.com/2008/01/06/running-out- of-time-%D9%85%D9%81%D9%8A%D8%B4- %D9%88%D9%82%D8%AA/ http://charafantah.wordpress.com/2008/01/06/running-out- of-time-%D9%85%D9%81%D9%8A%D8%B4- %D9%88%D9%82%D8%AA/ http://www.youtube.com/watch?v=8KaQmn2VTzs http://www.youtube.com/watch?v=He2QTpelAjE http://en.wikipedia.org/wiki/Persistence_of_vision http://en.wikipedia.org/wiki/Volumetric_display Sources