1. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Ramesh Raskar Mitsubishi Electric Research Labs (MERL) Cambridge, MA, USA Special Effects in the Real World

2. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Parent Organization: MELCO Mitsubishi Electric (MELCO), including subsidiaries –$32B in annual sales –125,000 people –10 business sectors: Space Development and Satellite Communications Communications and Information Processing Visual Equipment and Systems Electronic Devices Home Electronics Energy Industrial Equipment Public-Use Systems Transportation

3. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 MERL Mitsubishi Electric Research Laboratories –Dick Waters, President & CEO Two labs, co-located in Cambridge, MA: –MERL Technology Lab Advanced systems development ~35 researchers –MERL Research Lab Joe Marks, Director Basic research in applied computing 23 researchers, interns, consultants CV, HCI, Multimedia, CG, AI

4. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Special Effects –Photorealism and Animation Stays on screens Yet to affect daily life in real time –Post-processed content –Unlike internet,databases, UI

5. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Changing Appearance Virtual light source

6. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Changing Virtual Illumination

7. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006

8. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Special Effects in the Real World –Photorealism and Animation Limited to display screens Yet to affect daily life in real time –Unlike other fields of info-tech –Fusion: real world with graphics Next big challenge in CG ? New opportunities in advt, gaming, animation Believable co-existence

9. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Changing Appearance Projector Virtual light source

10. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Changing Virtual Illumination Raskar, Welch, Low, Bandyopadhyay, “Shader Lamps” (2000)

11. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Virtual Motion

12. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Virtual Motion Raskar, Ziegler, Willwacher, “Cartoon Dioramas in Motion,” (NPAR 2002)

13. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Dynamic Augmentation Projecting on Tracked Objects

14. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Virtual ReflectanceVirtual Illumination InteractionVirtual Motion ShaderLamps www.ShaderLamps.com

15. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Projector-based Augmentation www.ShaderLamps.com Virtual ReflectanceVirtual Illumination InteractionVirtual Motion

16. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Poor Man’s Palace Surround CG for Daily Environment

17. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Maya: World is an Illusion Fake World –We all live in one Real-life Fakes –Not just photos and videos but surroundings Privacy

18. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Augmented Reality (AR) With See-through Head-mounted Display © Andrei State Superman X-ray vision

19. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 hand-held display spatial see-through display head- mounted display retinal display Head-attached Body-attached spatial projector Spatial Augmented Reality Classification of Augmented Reality

20. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Being There Recreate Large Environments –‘BeingThere’, walk-around –Museums, Exhibitions, Training Kok-Lim Low, Greg Welch, Anselmo Lastra, Henry Fuchs. “Life-Sized Projector-Based Dioramas,” Proc. ACM Symposium on Virtual Reality Software and Technology 2001 (VRST 2001), November 2001.

21. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Desired Virtual Model © Andrei State Projected Guidance for Placement

22. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006

23. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Head-mounted DisplayProjectors © Andrei State Augmented Reality

24. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Opportunities Entertainment -Animating objects, Museums, Special Effects, Laser Shows Advertising -Car showrooms, Furniture store Design and Ergonomics -Prototyping products, visualize material changes Information Augmentation -Office, factory or assembly Training

25. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Spatially Augmented Reality (SAR) HMD-VR Spatially Immersive-VR AR using HMD Spatially Augmented Reality e.g. CAVE Video or Optical see-through SAR, Shaderlamps

26. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Steps –Preprocessing Scan 3D object Roughly align projectors Compute correction –Run time Render images of 3D model Correct

27. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Steps –Preprocessing Scan 3D object and create virtual model Faro arm

28. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Steps –Preprocessing Scan 3D object and create model Approximately position projector(s) Calibrate by finding pose

29. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Steps –Preprocessing Scan 3D object and create model Approximately position projector(s) Calibrate by finding pose –Run-time Inverse Rendering of 3D Models

30. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Shape cue allows perceptual unwrapping

31. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 ‘Wallpaper the surface’ Shape Adaptive Projection Minimum Stretch Display

32. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Shape Adaptive Projection Minimum Stretch Display Raskar, vanBaar, Beardsley, Willwacher, Rao, Forlines ‘iLamps: Geometrically Aware and Self-Configurable Projectors’, SIGGRAPH 2003

33. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Spatially Augmented Reality Raskar, vanBaar, Beardsley, Willwacher, Rao, Forlines ‘iLamps: Geometrically Aware and Self-Configurable Projectors’, SIGGRAPH 2003

34. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Mitsubishi Pocket Projector

35. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006

36. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 A new way to play tic-tac-toe Handheld projector. Grid projects to a fixed position on the wall. Cursor is guided by pointing the projector.

37. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Pocket Projector Games

38. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Swarms

39. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Warehousing Routing Library Baggage handling Currency Livestock tracking RFID

40. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Conventional Passive RFID

41. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Tagged Books in a Library Id : List of books in RF range No Precise Location Data Are books in sorted order ? Which book is upside down ?

42. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Prototype Tag RF tag + photosensor

43. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Find tag location using handheld Projector Photosensing Wireless Tags Many geometric ops R F I R F I D Interactive stabilized projection (Radio Frequency Id & Geometry) (Radio Frequency Id & Geometry) G Siggraph 2004

44. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 AR with Photosensing RFID and Handheld Projector

45. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Visual feedback of 2D position a.Receive via RF {(x 1,y 1 ), (x 2,y 2 ), …} pixels b.Illuminate those positions a.Receive via RF {(x 1,y 1 ), (x 2,y 2 ), …} pixels b.Illuminate those positions

46. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 RFID (Radio Frequency Identification) RFIG (Radio Frequency Id and Geometry)

47. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 CurvedPlanar Non-planar Single Projector Multiple Projectors Projector j Use r : T ? Pocket-Proj Objects Projectors and Surfaces 2000 2002 2001 1999 2002 20031999 1998

48. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Augmentation of “five senses” Visual Displays Auditory Displays Haptic Displays Locomotion Interfaces Olfactory Displays Taste Displays ? Sixth Sense.. Extending sensory modalities

49. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Sense of Balance Galvanic Vestibular Stimulation Masahiko Inami (UEC)

50. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Electric Cutaneous Display Skin Nerve Electrode Dr. H. Kajimoto, U Tokyo Masahiko Inami (UEC)

51. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Smart Touch Masahiko Inami (UEC)

52. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Scent Display Yasuyuki Yanagida, ATR Spatio-Temporal Controlled Delivery of Scent

53. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Air Cannon Generating a doughnut-shaped vortex ring Aperture Push

54. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Displays Contenders Organic LED Light Emitting Polymers E-Ink

55. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Holodust Perlin et al

56. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Fake World

57. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Poor Man’s Palace Surround CG for Daily Environment

58. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Recap Special Effects in Real World –Photorealism yet to impact daily life –Poor Man’s Palace Spatial Augmented Reality –Solution for fusion: Projectors, RFID, Sensors –Games for pocket projector –Display anywhere for advertising Future Problems –Natural phenomenon, complex reflectance, other displays –All senses: haptic, olfactory, preprioception Next Challenge in Real World Animation –Movies and games in our surroundings –3D Advertising

59. Mitsubishi Electric Research LaboratoriesSpecial Effects in the Real WorldRaskar 2006 Spatial Augmented Reality Graphics –Photorealism yet to impact daily life –Special effects in real world Spatial Augmentation –Solution for fusion –Needs hardware components –Projectors, RFID, Sensors Next Challenge in Animation raskar.com