An Introduction to Virtual Reality Iman Mohammad Rezazadeh School of Biomedical Engineering Science and Research Branch Islamic Azad University Spring 2010

Virtual Environment (VE) A computer generated world with which the user can interact Interaction can vary from looking around to interactively modifying the world. Virtual Reality

Virtual Reality What is virtual reality(VR)? A computer generated,immersive (or wide field),multi-sensory information program which tracks a user in real time. Virtual Reality

Person(s) Responsible History Year Person(s) Responsible Why it was important 1965 Ivan Sutherland The beginnings of VR 1977 Dan Sandin, Richard Sayre and Thomas Defanti Interaction through body movement 1982 Bonnie MacBird (Writer) The first computer- generated movie 1983 Myron Krueger First virtual environment 1984 William Gibson (His assistant) The term 'Cyberspace' 1987 Michael Piller (Writer) The Holodeck, idea of immersive VR 1992 Stephen King (Official Website) A look at the possible negative side of VR 1995 Silicon Graphics Virtual Reality Modeling Language 1999 Larry and Andy Wachowski Virtual Reality movie grosses $750M worldwide Virtual Reality

Types Virtual Reality Immersive Augmented Text-based Video Mapping Desktop (Window on a World) Video Mapping Virtual Reality

Computer Graphics vs. VR CG is concerned with the modeling, lighting and dynamics of environments and in general how to build a medium – so forth called Virtual World. But, designing a VE is something beyond designing a CG. When we become concern about how people experience the simulated environment, (re)act within it and interact with its components through the designed human-machine interface (HMI), we are talking about VE or VR

What is Immersive VR? Virtual Reality A type of VR in which the user becomes immersed (deeply involved) in a virtual world. it is also a form of VR that uses computer related components. Virtual Reality

Types of Non-immersive VR Text-based VR: when a reader of a certain text form a mental model of this virtual world in their head from the description of people , places and things. Augmented VR: the idea of taking what is real and adding to it in some way so that user obtains more information from their environment. Virtual Reality

Applications Virtual Reality Architecture Training Medicine Engineering and Design E-Commerce Entertainment Manufacturing Virtual Reality

Devices Virtual Reality Head Mounted Displays (HMD): was the first device providing its wearer with an immersive experience. Evans and Sutherland demonstrated a head- mounted stereo display already in 1965. Virtual Reality

3D VIDEO Eyewear Virtual Reality winner of a 2005 & 2006 Consumer Electronics Show Design Award. VR Price: $499.00 Virtual Reality

Binocular Omni-Orientation Monitor (BOOM) BOOM is a head coupled stereoscopic display device. Screens and optical system are housed in a box that is attached to a multi-link arm. Virtual Reality

VR… VE…

AR in teleconferencing person works at real desk remote collaborator represented by picture or video or “talking head” objects of discussion; e.g. a patient’s brain image, might also be fused into visual field HOW IS THIS ACHIEVED? From University of Washington HIT Lab

Cave Automatic Virtual Environment (CAVE) A room with projections on all walls, floor and ceiling The users wear shutter glasses to get a 3D view of the world. The users are able to move and control the environment with some kind of input mechanism Camera Device in hand Virtual Reality

Bam and its Cultural Landscape After the earthquake of the 26th December 2003 and the immense destruction of the Bam heritage in Iran, the reconstruction of the ruins especially of the citadel of Bam has become an important task for experts. first phase of the project were to develop the 3Dimensional models of 7 case studies inside the citadel of Bam (Main gate,Bazaar,Tekkiyeh (religious plaza), Mosque, Sistani house, barrack, stable, four season reception room Virtual Reality

Cont…. The team of drawing the 3D models consisting of in Japan from Waseda University and the team of Raaz Ahang architectural firm in Iran from University of Tehran has made a tremendous effort to build the 3D models. Virtual Reality

Virtual Reality

Virtual Reality

Virtual Reality

Virtual Reality

Virtual Reality

How about… learning to hang-glide?

I’m more of the down to earth type... How about learning to drive a car? © www.ozedweb.com

Hey, that wasn’t there a moment ago... Things are easy with VR. Real Estate Agents use it every day. © www.ozedweb.com

...And Architects love it! Wait till you Walk through! © www.ozedweb.com

At last… something real. Sorry… That’s VR too. © www.ozedweb.com

Ahh... It goes here. © www.ozedweb.com

Princeton Video Wall

Augmented Reality

A nice Little AR

Share Virtual Environment You are a Avatar!!!! Wow…

Classroom is a VE Virtual Reality

VR CAVE

VR Workbench

Norman’s Affordances Affordances: Have perceived properties that may or may not exist Have suggestions or clues about to how to use these properties Can be dependent on the Experience Knowledge Culture of the actor Can make an action easy or difficult From McGrenere & Ho, Proc of Graphics Interfaces, 2000

Affordances of a Teapot?

Slide adapted from Saul Greenberg

Real vs. Perceived Affordances In product design, where one deals with real, physical objects, there can be both real and perceived affordances, and the two need not be the same.

Slide adapted from Saul Greenberg

Transfer Effects People transfer their expectations from familiar objects to similar new ones positive transfer: previous experience applies to new situation negative transfer: previous experience conflicts with new situation Based on slide by Saul Greenberg

Cultural Associations Groups of people learn idioms red = danger, green = go But these differ in different places Light switches America: down is off Britain: down is on Faucets America: counter-clockwise is on Britain: counter-clockwise is off Based on slide by Saul Greenberg

VR and Cognition

VR in Medicine

I think I’m feeling ill. Well, that’s not a problem. We’ll find a cure with the Virtual patient... © www.ozedweb.com

…now I’m sure I am. …in our virtual hospital. Why not? © www.ozedweb.com

Virtual Body vs. Cadaver Training on cadavers has several drawbacks: if trainee cuts a nerve or a blood vessel in a cadaver nothing will happen no action can be reversed on cadavers (what is cut is cut) dead tissue is harder, color is changed, arteries do not pulsate Advantages of computer simulations: procedures can be repeated many times with no damage to virtual body virtual body does not have to be dead - many functions can be simulated for realistic visualizations organs can be made transparent and modeled

VR in Endoscopy

Virtual Fighting and Rehab.

VR in Endoscopy

VR in Endoscopy

endoScope - A VR Tool EndoScope is a Motif/Inventor based model viewer that runs on SGI workstations Developed by Biomedical Imaging Resource at Mayo Clinic

VR in Colonoscopy

VR in Laparoscopy

VR in Laparoscopy

MIS Training at EPFL Group for surgical robotics and instrumentation, Swiss Gastro-intestinal organ modeled and tissue properties simulated (pushing, pulling) Force feedback generated for realistic simulation

Laparoscopic Interfaces Immersion Corp. devices for minimally invasive surgery simulators Offers tracking in 5 degrees of freedom (left-right, up-down, in-out, rotation around axis, open-close) Version with and without force- feedback are available Price range: up to $8,000

3-D Stereo Visualization Univ. of Illinois Chicago, School of Biomedical and Health Information Sciences ImmersaDesk system for temporal bone visualization

Fetus Visualization Univ. of North Carolina at Chapel Hill A VR system is developed for visualization of a fetus in a pregnant woman’s abdomen The ultrasound image of the fetus is superimposed on the video image of the woman’s abdomen The system can be used for pregnancy check-ups

Fraunhofer Institute Simulators Fraunhofer Institute medical training simulators The arthroscopy training simulator Nasal endoscopy simulator The trainee is able to practice techniques before facing a real patient

Eye Surgery Simulator Interactive Media Technology Center at Georgia Tech Eye surgery simulator for: education and training of medical students training of surgeons to cope with emergencies Simulator provides force feedback information for more realistic simulation of tissue cutting Simulation of the tissue includes elasticity of the eye surface tissue before a cut is made

Craniofacial Surgery Simulation Erlangen Institute, Germany In craniofacial surgery it is important to plan and predict the outcome of surgical intervention The face can be visualized after reconstructive plastic surgery

Augmented Reality for Surgery Julesz, Harward Medical School Augmented reality visualization has three phases: 1. 3D laser scanning of the patient’s head surface 2. 3D registration of the scanned and imaged surface 3. Augmented reality display of tumor (green)

Augmented Reality in Neurosurgery Harvard Medical School Combined neurosurgery planning and augmented reality

Phobia Projects GA Tech GVU Center Virtual Reality Exposure Therapy project Demonstrated effectiveness Advantages: cost effective effective therapy patient acceptance suitable for network delivery (telemedicine)

Spider Phobia Desensitization Univ. of Washington Human Interface Technology Lab VE environment designed that contains virtual spiders (a large brown spider with fur and a small one were used Patients are encouraged to pick up spiders with their virtual hands Spiders are unexpectedly dropped of the ceiling, patient could pull the spider legs off

Acrophobia Project Univ. of Michigan Balcony view from high floors are generated The patient gradually watches the environment from higher and higher viewpoints A multi-session therapy cures the acrophobia patient

Medical Education at UCSD UCSD Applied Technologies Lab project on VR-multimedia system for education of medical students (anatomy) UCSD Virtual Anatomy World Anatomic structures are linked to supporting multimedia contents to provide VR-MM anatomy lessons

Medical Education at Fraunhofer Institute Virtual anatomy system for medical students Virtual patient Students are able to understand complex interrelationships of anatomical structures

Remote Palpation Simulator Interactive Technology Media Center at Georgia Tech The idea is to allow doctors to examine patients at a remote location IMTC developed a haptic lens - a sensor that measures 3D surface under a specific pressure The device is pressed against an object and 3D surfaces, deformed under the pressure, are recorded in real-time

OK! OK ! No Military Use Ankle Therapy

Medical Robotics

Information flow in CIS Introduction(2) CIS Information flow in CIS

Motivation: Started with the weaknesses and strengths of minimally invasive surgery (MIS) Smaller incisions, shorter post-operative time, reduced infection, faster rehabilitation, lesser pain, better cosmetics, ... Eye-hand coordination, difficulty in moving arms, degree of motion

Classification Depending on the degree of surgeon interaction during the procedure: Supervisory-controlled; Telesurgical; Shared-control;

Application of Medical Robotics Telesurgery Surgeon sits at a console Has controls to move the robotic arms Does not operate on the patient directly Mainly used in minimally invasive surgeries

Application of Medical Robotics Telemedicine and Teleconsultation Telecommunciation channels to communicate with other physicians/patients Control an external camera which in turn controls an endoscopic camera – used to share images with a remote surgeon

Application of Medical Robotics Rehabilitation Assistive robots Wheelchair with intelligent navigational control system

Application of Medical Robotics Remote surgery Surgeon can be anywhere in the world Remotely controls the robotic surgical system Very useful for treating wounded people in battlefields

Application of Medical Robotics Laparoscopic Surgery Performed in the abdominal cavity using MIS Abdomen cavity is expanded using CO2 Uses Laparoscopic instrument Fiber optic channels to illuminate the inside of abdominal cavity Lens optics to transmit image CCD camera at the outer end Image displayed on high resolution TV

Design of Robotic Telesurgery The Concept Telesurgical system concept

Existing surgical systems Da Vinci Surgical® system by Intuitive Surgical, Inc.

Existing surgical systems Da Vinci Surgical® system by Intuitive Surgical, Inc.

VR in Brain Computer Interface

VR in BCI

Monitoring Driver’s Alterness

VR, BCI, Imagination

My Office in Future

The End of the Beginning Surgery Planning and Simulation Rehabilitation and Empowering Mind Engineering Neurofeedback and Biofeedback Mixed Reality Real Virtuality Pervasive computing Ubiquitous computing © www.ozedweb.com

Invitation for Contribution "Mixed Reality and Human-Robot Interaction” Springer

What we are doing in IAU Virtual Upper limb and Lower Limb Rehab. Mind Engineering and Neurofeedback (CARTEME) Collaborative Interfacing Affective Computing