1. Immersion, Prescence, Distributed VR Bob Hobbs Staffordshire University Computing School

2. Outline –Context –Immersion –Presence –Shared Environments

3. Virtual Reality is a Tool What it is: –Use of highly interactive real-time immersive systems to convey information What it is not : –Desktop graphics –Text based –Non-interactive –Linear

4. Immersion: Realisation of an Environment generates displays ideally in all sensory systems; fully encloses the participant in those displays; tracks the body, limbs, head; determines the optical, auditory... arrays as a function of head tracking Either: –displays a Virtual Body with movements as function of the tracking. (mainly with HMD) –Participant can visualise self and world (CAVE)

5. Virtual Body At any moment there is a position in the geometry with respect to which sensory data is generated - the egocentric self-reference position. This corresponds to the place occupied by the human actor in the environment. At the self-reference position there is a functioning VB represented by the displays.

6. Cave

7. Position Tracking Systems Polhemus Inc. (http://www.polhemus.com) –3Space ISOTRAK (1 sensor) –3Space FASTRAK (many sensors) Ascension Technology Corp. (http://www.ascension-tech.com) –Flock of Birds –pcBIRD –SpacePad

8. Trackers Calibration Dynamic errors –caused by external electromagnetic fields –can be corrected by increasing measurements frequency, synchronizing the measurements with the external field source, and filtering Static errors –caused by the field distortions due to the surrounding metal and external fields –can be corrected via trackers calibration

9. Calibration Table Z X true tracked

10. Calibration Example CAVE, FoB 4 feet from the floor 1 foot grid 4 th order polynomial fit

11. Interpolation True spaceTracked space 1 d 8 d V. Kindratenko, A. Bennett, “Evaluation of Rotation Correction Techniques for Electromagnetic Position Tracking Systems”, in Proc. VE 2000, pp. 13-22

12. Data Acquisition Techniques Size and type of a calibration table depends on –Type of the calibration technique to be used –Severity of the field distortions –Required calibration quality Calibration table can be –Irregular (for high-order polynomial fit) –Regular in the true space (for interpolation) –Regular in the tracked space (for tri-linear interpolation)

13. Regular Grid in the True Space

14. An Immersive Participant A user will be head tracked Have a ‘Wand’ Stereo glasses in CAVE HMD user may have additional tracking sensors – Data Glove or Motion tracker

15. Data Glove Hand measurement devices must sense both flexing angles of fingers and position/orientation of wrist in real-time. typical example of hand measurement device: DataGlove from VPL Research. DataGlove consists of lightweight nylon glove with optical sensors mounted along fingers.

16. Each sensor: short length of fiberoptic cable, with light-emitting diode (LED) at one end and phototransistor at other end. When cable flexed, some of LED's light lost, so less light received by phototransistor. Attached to back: 3Space Isotrack system to measure orientation/position of gloved hand.

17. Data Suit Much less popular than DataGlove: allows to measure positions of body. typical example of use of datasuit: film of Fuji TV: the Dream of Mr. M. 3D character approximately performs same motion as animator. Another way of measuring positions of body just to use collection of sensors like Flock of Birds. However, needs algorithms for calibration and conversion (see paper by Molet et al.)

18. Sound Midi-equipment and workstation audio for sound generation and effects, filter processors and 3D-audio cards for spatial audio. Two categories of sound in VR can be identified: –Simulation of real world acoustics: based on our experiences in everyday life physical behavior of sound can be modeled. –comprises sound generation, e.g. caused by object collision, sound propagation and auralization. Immersive user interfaces can be used to evaluate simulation results. –Sound at user interface: sound can be applied to support user in current task or to provide information about invisible proceedings.

19. Presence Presence is a state of consciousness where the human actor has a sense of being in the location specified by the displays. We take presence as the central feature of "virtual reality": "A virtual reality is defined as a real or simulated environment in which a perceiver experiences telepresence" (Steuer). The unique feature of "virtual reality" systems is that they are general purpose presence transforming machines..

20. Meaning of Presence Presence is the psychological sense of being there in the environment specified by the displays. a high degree of presence in the VE should lead to the participant experiencing objects and processes in the virtual world as (temporarily) more the presenting reality than the real world in which the VE experience is actually embedded. A correlate of this is that the participant should exhibit behaviours that are the same as those they would carry out in similar circumstances in everyday reality. The VE experience - should be more like visiting a place, rather than like seeing images designating a place

21. Design in Immersive VEs With design in immersive virtual environments... designer shares same space as objects; a degree of evaluation can take place in the virtual space; presence leads to the designer behaving in a manner appropriate to everyday reality in similar circumstances. Special "interactive techniques" and behaviours do not have to be learned...

22. Feedback Two forms of feedbaack –Force Feedback Manipulating virtual objects Gravity Simulation –Touch (tactile) Feedback Texture appreciation Navigation Sensitive Use Haptic Devices

23. What is a haptic interface? A haptic interface is a force reflecting device which allows a user to touch, feel, manipulate, create, and/or alter simulated 3D-objects in a virtual environment. Movement trackers do not provide feedback

24. Tactile Feedback

25. Usage It could be used to train physical skills such as those jobs requiring specialized hand-help tools (e.g. surgeons, astronauts, mechanics), to provide haptic-feedback modeling of three dimensional objects without a physical medium (such as automobile body designers working with clay models), or to mock-up developmental prototypes directly from CAD databases (rather than in a machine shop).

26. Phantom Very common haptic device mainly used with augmentation on desktop systems

27. Exoskeleton

28. Actuators Electrical current drives actuators controlling individual joints Directly to motors or solenoids To valves controlling flow of fluids to hydraulic or pneumatic systemsvalves

29. Presence in Multi-participant Environments Sense of being in a place sense of sharing the same space as other individuals Sense of belonging to a totality more than just the sum of the individuals Awareness may be an important factor enhancing shared presence. Shared presence may correspondingly enhance awareness

30. Robot arm Simplest sort of robot Typical arm has 7 segments, 6 joints 6DOF Human arm 7DOF Usually driven by Step Motors Main use is in manufacturing

31. Robot Arm Fitted with end effector Usually interchangeable Artificial Hand, paint gun, welding rod Pressure sensor needed to prevent crushing Programmed by incremental steps which are then replicated ad infinitum

32. Frameworks, Chains (or Skeletons) A lot of mechanical objects in the real world consist of solid sections connected by joints Obviously robot arm but also –Creatures such as humans and animals. –Car Suspension –Ropes, string and Chains

33. Frameworks, Chains (or Skeletons) Sections and joints of robot arm are known as a 'chain‘ In creatures could be referred to as a skeleton Moveable sections correspond to bones Attachments between bones are joints.

34. Frameworks, Chains (or Skeletons) Motions of chains can be specified in terms of translations and rotations. Forward Kinematics - From the amounts of rotation and bending of each joint in an arm, for example, the position of the hand can be calculated. Inverse Kinematics - If the hand is moved, the rotation and bending of the arm is calculated, in accordance with the length and joint properties of each section of the arm.

35. Joint Translation-Rotation We can use a transform (T) to transform each point relative to the body to a position in world coordinates. If we want to model both linear and angular (rotational) motion then we need to use a 4x4 matrix to represent the transform

36. What is Inverse Kinematics? Forward Kinematics Base End Effector ?

37. What is Inverse Kinematics? Inverse Kinematics Base End Effector

38. Kinematic Chains Solid links connected at movable joints Fixed end: base Movable end: tip or end effector One degree of freedom (DOF) per joint Open chain: one fixed end, one movable end Closed chain: both ends fixed

39. Forward and Inverse Kinematics

40. Kinematic Redundancy End-effector has 6 DoFs - (x, y, z) position - (,, ) orientation Non-redundant linkage has < = 6 joints (DoFs) Redundant linkage has > 6 joints (DoFs) - Human arm has 7 DoFs » Shoulder 3 » Elbow 1 » Forearm 1 » Wrist 2 - Redundancy enables multiple solutions

41. Inverse Kinematics (IK) Non-redundant Linkages - Analytical solutions Redundant Linkages - Many techniques » Pseudo-inverse (Jacobian) » Gradient » Others IK Commonly Found in Animation Packages - 3D Studio Max

42. Redundancy A redundant system has infinite number of solutions Human skeleton has 70 DOF –Ultra-super redundant How to solve highly redundant system?

43. Iterative solution Start at end effector Move each joint so that end gets closer to target The angle of rotation for each joint is found by taking the dot product of the vectors from the joint to the current point and from the joint to the desired end point. Then taking the arcsin of this dot product. To find the sign of this angle (ie which direction to turn), take the cross product of these vectors and checking the sign of the Z element of the vector.

44. Goal Potential Function “Distance” from the end effector to the goal Function of joint angles : G(  )

45. Our Example Base End Effector Goal distance

46. Quiz Will G(  ) be always zero? –No : Unreachable Workspace Will the solution be always found? –No : Local Minima/Singular Configuration Will the solution be always unique? –No : Redundancy

47. Conflict Between Goals base ee 2 ee 1

48. Conflict Between Goals base ee 2 ee 1 Goal 1

49. Conflict Between Goals base ee 2 Goal 2 ee 1

50. Conflict Between Goals base ee 2 ee 1 Goal 1 Goal 2

51. Conflict Between Goals base ee 2 ee 1 Goal 1 Goal 2

52. Figure Modeling Many VE Applications Require Human, Animalor Robotic Actors - Team training exercises » SIMNET, DIS - Mission planning and rehearsal - Human factors studies » Boeing 777 - Walkthroughs Virtual Actors - Computational models of real-world counterparts

53. Virtual Actors: Autonomous or Guided Guided Actors are Slaved to the Motions of a Human Participant Using Body Tracking – Optical, mechanical,... – A.K.A. Avatar Autonomous Actors Are Controlled by Behavior Modeling Programs, and Can - Augment or replace human participants - Serve as surrogate instructors - Act as guides in complex synthetic worlds Hybrid Control Desirable - VRLOCO uses interaction to invoke and control locomotion behaviors

54. The Weiss 6-Level Motor Organization Hierarchy Organism Level 6. Motor Behavior 5. Motor Organ System 4. Motor Organ 3. Muscle Group 2. Muscle 1. Motor Unit Neuron Level 3. Muscle Group - Coordinated action of several muscles - Motion at one joint 2. Muscle - Muscle contraction 1. Motor Unit - Neuron + muscle fibers - Twitching, shivering

55. The Weiss 6-Level Motor Organization Hierarchy Organism Level 6. Motor Behavior 5. Motor Organ System 4. Motor Organ 3. Muscle Group 2. Muscle 1. Motor Unit Neuron Level 6. Motor Behavior - Movement of the whole organism - E.G., Goal-directed locomotion - Task manager 5. Motor Organ System - Coordinated action of several limbs - E.g., Walking - Motor programs, skills 4. Motor Organ - Coordinated action of several joints - E.G., Stepping motion of a limb - Local motor programs

56. Motion and Reaction Sensorymotor level - Levels 1 - 5 - Peripheral and proprioceptive feedback associated with reflex arcs - Motor programs and reflexes coordinate and control motion - Executes behaviors Reactive level – Level 6 and higher - Perception triggers and modulates behavior - Organism responds to environmental stimuli to select and compose behaviors - Selects behaviors

57. Organization of a Virtual Actor Organism Level 6. Motor Behavior 5. Motor Organ System 4. Motor Organ 3. Muscle Group 2. Muscle 1. Motor Unit Neuron Level Level 6 and above Reactive level Levels 1-5 Sensorymotor level

58. Virtual Actor

60. Abstraction and Interaction

61. Representation and Abstraction

63. Finite State Machines for Walking

64. Control and Abstraction

65. Avatars

66. Static Balance

67. Weight Bend: – Non-weight-bearing motion – Traverse subtree rooted at rotating joint Pivot – Weight-bearing motion – Traverse entire tree starting at root EXCEPT for subtree rooted at rotating joint Critical Element of Realism – Is the character supported by its legs, or are the legs dangling in space as the character is translated along ?

68. Non-weight-bearing motion – traverse subtree rooted at rotating joint Bend

69. Weight-bearing motion – traverse entire tree starting at root EXCEPT for subtree rooted at rotating joint Pivot

70. Gait Parameters Gait Pattern – Sequence of lifting and placing feet Gait Cycle – One repetition Of the gait pattern Period – Duration of one gait cycle Relative Phase of Leg I – Fraction of gait cycle before leg I is lifted Duty Factor – Fraction of gait cycle period a given leg spends on ground Swing Time – Time a leg spends In the air Stance Time – Time a leg spends On the ground Stroke – Distance body travels during a leg's stance time

71. Finite State Machines for Walking

73. Tele-Immersion Goal - not just making these collaborations possible, but making them convenient

74. CAVERNsoft Application Bryan Carter, Bill Plummer – ATC (Advanced Technology Center at Univ of Missouri- Columbia ) SIGGRAPH 1999 Harlem is reconstructed for an African American Literature course at MU. Instead of just reading literary works from this era, this prototype will allow students to become immersed and engaged in an interactive literature course. Jim Sosnoski, Jim Fletcher- English Dept. Univ Illinois Chicago Steve Jones- Communications Dept. Univ Illinois Chicago Virtual Harlem

75. Elements of Tele-Immersion

76. Avatars Tracking head and hand position and orientation give good cues Extendable pointing rays can be useful in large spaces Exaggerated head and hand motions give better cues than just hand

78. Shared Virtual Environments in Europe Collaborative Virtual Environments (COVEN) ACTS Develops an integrated teleworking platform that supports multi-sensory presence for collaboration in shared virtual environments. Services: mechanisms to support the presence of users in shared virtual environments. browsing and interaction facilities for large numbers of users accessing enormous quantities of remote information; synchronised multi-sensory interaction with dynamic representations of three-dimensional objects and actors; support for collaborative tasks requiring complex motor skills and shared information.

79. VR Applications Augmented Reality –Placing data in the normal workspace Data Visualisation –Explaining data through better representation Training –For dangerous/expense procedures Conferencing –Social context for telecommunication Health –Treatment of phobias/psychological disorders Entertainment