1. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Lecture 23 of 42 William H. Hsu Department of Computing and Information Sciences, KSU KSOL course pages: http://snipurl.com/1y5gchttp://snipurl.com/1y5gc Course web site: http://www.kddresearch.org/Courses/CIS636http://www.kddresearch.org/Courses/CIS636 Instructor home page: http://www.cis.ksu.edu/~bhsuhttp://www.cis.ksu.edu/~bhsu Readings: Sections 8.1, 8.4, Eberly 2 e – see http://snurl.com/1ye72http://snurl.com/1ye72 Modeling, Simulation, Virtual Reality (VR), and Virtual Environments (VE)

2. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Incorporating Dynamic Real Objects into Immersive Virtual Environments Benjamin Lok University of North Carolina at Charlotte Samir Naik Disney VR Studios Mary Whitton, Frederick P. Brooks Jr. University of North Carolina at Chapel Hill April 28 th, 2003

3. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Outline Motivation Managing Collisions Between Virtual and Dynamic Real Objects NASA Case Study Conclusion Why we need dynamic real objects in VEs How we get dynamic real objects in VEs Applying the system to a driving real world problem

4. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Assembly Verification Given a model, we would like to explore:  Can it be readily assembled?  Can repairers service it? Example:  Changing an oil filter  Attaching a cable to a payload

5. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Current Immersive VE Approaches Most objects are purely virtual  User  Tools  Parts Most virtual objects are not registered with a corresponding real object. System has limited shape and motion information of real objects.

6. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Ideally Would like:  Accurate virtual representations, or avatars, of real objects  Virtual objects responding to real objects  Haptic feedback  Correct affordances  Constrained motion Example: Unscrewing a virtual oil filter from a car engine model

7. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Dynamic Real Objects Tracking and modeling dynamic objects (change shape and appearance) would:  Improve interactivity  Enable visually faithful virtual representations

8. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Previous Work: Incorporating Real Objects into VEs Non-Real Time  Virtualized Reality (Kanade, et al.) Real Time  Image Based Visual Hulls [Matusik00, 01]  3D Tele-Immersion [Daniilidis00] How important is to get real objects into a virtual environment?

9. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Previous Work: Interaction and Collision Detection Commercial Interaction Solutions  Tracked mice, gloves, joysticks Augment specific objects for interaction  Doll’s head [Hinkley1994]  Plate [Hoffman1998] Virtual object collision detection  Traditional packages [Ehmann2000]  Hardware accelerated [Hoff2001] Virtual object – real object  a priori modeling and tracking [Breen1996]

10. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Real-time Object Reconstruction System Handle dynamic objects (generate a virtual representation) Interactive rates Bypass an explicit 3D modeling stage Inputs: outside-looking-in camera images Generate an approximation of the real objects (visual hull)

11. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Reconstruction Algorithm … 1. Start with live camera images 2. Image Subtraction 3. Use images to calculate volume intersection (visual hull) 4. Composite with the VE …

12. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Visual Hull Computation Visual hull - tightest volume given a set of object silhouettes Intersection of the projection of object pixels

13. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Visual Hull Computation Visual hull - tightest volume given a set of object silhouettes Intersection of the projection of object pixels

14. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Volume Querying in Hardware A point (P) inside the visual hull (VH real objects ) projects onto an object pixel from each camera P  VH real objects iff  i  j, P = C i -1 O i, j

15. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Implementation 1 HMD-mounted and 3 wall-mounted cameras SGI Reality Monster – handles up to 7 video feeds 15-18 fps Estimated error: 1 cm Performance will increase as graphics hardware continues to improve

16. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Managing Collisions Between Virtual and Dynamic Real Objects

17. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Approach We want virtual objects to respond to real object avatars This requires detecting when real and virtual objects intersect If intersections exist, determine plausible responses Only virtual objects can move or deform at collision. Both real and virtual objects are assumed stationary at collision.

18. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Detecting Collisions

19. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Visual Hull Computation

20. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Detecting Collisions Approach Are there real-virtual collisions? For virtual object i Done with object i Volume query each triangle Calculate plausible collision response Determine points on virtual object in collision NY

21. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Resolving Collisions Approach 1.Estimate point of deepest virtual object penetration. CP obj 2. Define plausible recovery vector V rec = RP obj - CP obj 3.Back out virtual object. CP obj = CP hull CP obj V rec CP hull

22. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Resolving Collisions Approach 1.Estimate point of deepest virtual object penetration. CP obj 2. Define plausible recovery vector V rec = RP obj - CP obj 3.Back out virtual object. CP obj = CP hull CP obj V rec CP hull

23. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Resolving Collisions Approach 1.Estimate point of deepest virtual object penetration. CP obj 2. Define plausible recovery vector V rec = RP obj - CP obj 3.Back out virtual object. CP obj = CP hull CP obj V rec CP hull

24. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Resolving Collisions Approach 1.Estimate point of deepest virtual object penetration. CP obj 2. Define plausible recovery vector V rec = RP obj - CP obj 3.Back out virtual object. CP obj = CP hull CP obj V rec CP hull

25. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Resolving Collisions Approach 1.Estimate point of deepest virtual object penetration. CP obj 2. Define plausible recovery vector V rec = RP obj - CP obj 3.Back out virtual object. CP obj = CP hull CP obj V rec CP hull

26. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Results

27. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Results

28. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Collision Detection / Response Performance Volume-query about 5000 triangles per second Error of collision points is ~0.75 cm.  Depends on average size of virtual object triangles  Tradeoff between accuracy and time  Plenty of room for optimizations

29. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Case Study: NASA Langley Research Center (LaRC) Payload Assembly Task

30. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics NASA Driving Problems Given payload models, designers and engineers want to evaluate:  Assembly feasibility  Assembly training  Repairability Current Approaches  Measurements  Design drawings  Step-by-step assembly instruction list  Low fidelity mock-ups

31. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Task Wanted a plausible task given common assembly jobs. Abstracted a payload layout task  Screw in tube  Attach power cable

32. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Task Goal Determine how much space should be allocated between the TOP of the PMT and the BOTTOM of Payload A

33. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Videos of Task

34. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Results Participant #1#2#3#4 (Pre-experience) How much space is necessary? 14 cm14.2 cm15 – 16 cm15 cm (Pre-experience) How much space would you actually allocate? 21 cm16 cm20 cm15 cm Actual space required in VE15 cm22.5 cm22.3 cm23 cm (Post-experience) How much space would you actually allocate? 18 cm16 cm (modify tool) 25 cm23 cm The tube was 14 cm long, 4cm in diameter.

35. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Results Late discovery of similar problems is not uncommon. Participant #1#2#3#4 Time cost of the spacing error days to months30 daysdays to monthsmonths Financial cost of the spacing error $100,000s - $1,000,000+ largest cost is huge hit in schedule $100,000s - $1,000,000+ $100,000s

36. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Case Study Conclusions Object reconstruction VEs benefits:  Specialized tools and parts require no modeling  Short development time to try multiple designs  Allows early testing of subassembly integration from multiple suppliers Possible to identify assembly, design, and integration issues early that results in considerable savings in time and money.

37. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Conclusions

38. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Innovations Presented algorithms for  Incorporation of real objects into VEs  Handling interactions between real and virtual objects Applied to real-world task

39. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Future Work Improved model fidelity Improved collision detection and response Apply system to upcoming NASA payload projects.

40. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Thanks Collaborators Dr. Larry F. Hodges Danette Allen (NASA LaRC) UNC-CH Effective Virtual Environments UNC-C Virtual Environments Group For more information: http://www.cs.uncc.edu/~bclok (I3D2001, VR2003) Correct Email: bclok@uncc.edu Funding Agencies The LINK Foundation NIH (Grant P41 RR02170) National Science Foundation Office of Naval Research

41. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Object Pixels Identify new objects Perform image subtraction Separate the object pixels from background pixels current image - background image = object pixels

42. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Current Projects at UNC-Charlotte with Dr. Larry Hodges Digitizing Humanity  Basic research into virtual characters  What is important?  How does personality affect interaction?  Applications:  Social situations  Human Virtual-Human Interaction Virtual Reality  Basic Research:  Incorporating Avatars  Locomotion Effect on Cognitive Performance  Applications:  Balance Disorders (w/ Univ. of Pittsburg)

43. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Research Interests Computer Graphics – computer scientists are toolsmiths  Applying graphics hardware to:  3D reconstruction  simulation  Visualization  Interactive Graphics Virtual Reality  What makes a virtual environment effective?  Applying to assembly verification & clinical psychology Human Computer Interaction  3D Interaction  Virtual Humans Assistive Technology  Computer Vision and Mobile Technology to help disabled

44. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Future Directions Long Term Goals  Help build the department into a leader in using graphics for visualization, simulation, and training.  Effective Virtual Environments (Graphics, Virtual Reality, and Psychology)  Digital Characters (Graphics & HCI)  Additional benefit of having nearby companies (Disney) and military  Assistive Technology (Graphics, VR, and Computer Vision)

45. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Occlusion

46. Computing & Information Sciences Kansas State University Lecture 23 of 42CIS 636/736: (Introduction to) Computer Graphics Occlusion