Evolution of Graphics Hardware: 40 years since Sutherland’s HMD Henry FUCHS University of North Carolina at Chapel Hill Graphics Hardware 2008 Sarajevo

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 2 Acknowledgement Assistance with preparation of this talk Hideyuki Tamura, Canon Greg Welch, UNC Mark Mine, Disney Steve Molnar, NVIDIA Adam Lake, Intel

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 3 Evolution of Graphics H/W: from what point ? Ivan Sutherland’s 1963 PhD Sketchpad Beginning of graphics, but no new hardware Sutherland’s 1965 “Ultimate Display” idea A vision, but no specific HW specified Sutherland’s D HMD System Complete working system, needed all specialized HW

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 4 Ivan E. Sutherland A Head-Mounted Three-Dimensional Display, 1968 Fall Joint Computer Conference Implemented all the components of a real-time interactive 3D graphics system: Display device: stereo, optical see-through Image generation: Real-time 3D perspective Head tracking: Real-time 6 DOF 3D model creation Hand-tracking & interaction: Real-time 3 DOF (’69 ?)

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 5 Procedure for Examining Evolution of Graphics HW For each subsystem: display, image generation, tracking : what was implemented, state of the art ~1988: sample state of the art 2008: current example subsystem Assessment: The dream in 1968/1970 Grade for where we are now: A, B, C, D

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 6 Sutherland HMD: Overview

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 7 Images from Sutherland’s ’68 HMD

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 8 Display device on ’68 HMD Optical see- through head- mounted display

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 9 HMD ca. 1985: VPL head- mount Closed display, not see-through LCD from SONY pocket TV wide angle optics (LEEP) (no distortion correction)

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 10 HMD ca. 1995: video see- through HMD (UNC) Need for video, rather than optical, see-through

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 11 Current HM Displays closed, total VR Fakespace wide5 Video see-through: Canon / Mixed Reality Labs VH2002 InnerOptic (based on UNC design)

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 12 Assessment on HM Displays The dream at 1968 / 1970: Displays built in to our ordinary eyeglasses; see virtual objects throughout the eyeglass lenses (not a ‘virtual inset’) Grade for where we are now: C significant progress not even close to adequate

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 13 Image generation: Sutherland’s 1968 system Real time Line drawing No hidden line elimination Heroic work: everything built out of gate-level chips 3D transforms Clipping divider Graphics pipeline

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 14 Image generation: ca 1988 Real-time full-color raster image generation Rapid, consistent progress throughout the 1980s and 1990s 100s of polygons /sec in 1980 Million polygons / sec early 1990s  UNC Pixel-Planes 5 at siggraph 1991: 2M polygons/sec

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 15 Image generation now Sophisticated shading effects and millions of polygons /sec Commodity graphics chips & boards Rapid, continuing progress Image generation processors becoming ubiquitous – migrating even to mobile devices

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 16 Image Generation Assessment Dream of 1968/1970: Realistic image generation Sutherland 1965 Ultimate display: can’t distinguish between virtual and real objects Grade for where we are now: A+ Are we finished? No! illuminate globally; integrate physics sim. with graphics; integrate image capture with image display,.. Feels great for the field to be doing so well.

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 17 Head tracking: Sutherland’s 1968 system Two tracking systems implemented ultrasound tracking with multiple emitters and receivers mechanical tracker

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 18 Head tracking (ultrasonic) Sutherland’s 1968 system Ultrasonic tracking 3 transmitters on head-mount 4 receivers hanging from ceiling Measure phase changes Problem with ambiguity of number of cycles of u/s signals Problem exacerbated by heating system

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 19 Head tracking (mechanical) Sutherland’s 1968 system Vertical pivot in ceiling Universal joint on top Universal joint on bottom Shaft slides in and out “Sword of Damocles” Heavy and uncomfortable Works

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 20 Head Tracking ~1988: Polhemus Magnetic Tracking Developed for head tracking in cockpit Very close range Not for walking across a room Severe warping of space with metal, with other magnetic fields. Can be reduced with calibration

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 21 Large area trackers: ~ 1991 UNC 10x12 ft space demonstrated at Siggraph 1991 Multiple optical sensors on head, LEDs in ceiling tiles Lateral effect photo-diodes, much faster than imagers One LED lighted at a time Predict most-useful LED to light each time slot

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 22 Head tracking Now Multiple good solutions Optical Magnetic Inertial / optical combination Limited to instrumented area

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 23 Head tracking assessment Dream of 1968 / 1970: Go anywhere, unencumbered Grade for where we are now: B Choice of solutions on the market if willing to live with restrictions Degraded performance if lose line of sight contact (optical trackers) or near ferrous/metal objects (magnetic trackers) Can’t go outside of instrumented area, nor outdoors. Steady progress

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 24 Interaction / Hand tracking in Sutherland’s HMD Mechanical design with three reels of fishing line mounted from ceiling All lines connected to top of (camera) hand-grip Amount of line reeled out determines 3D location of hand-grip Problem of interference between head and hand trackers

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 25 Interaction / Hand tracking ca.1988 Polhemus magnetic tracker (time-shared with a target for head-tracker) VPL DataGlove sometimes added to hand-tracker Severe warping of tracked space if user moves more than a few steps

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 26 Interaction / Hand tracking now Multiple technologies available Accuracy & speed may not be adequate for certain applications UNC AR assistance for surgical needle guidance Track ultrasound transducer Track needle

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 27 Assessment of Hand tracking Dream in 1968/ 1970: unencumbered tracking of hand Grade for where we are now: Incomplete Haven’t had sufficient number of real applications to judge --- rest of system hasn’t been ready using Fred Brooks’ denition of “What’s real about Virtual Reality?” Someone other than develops pay to use the system

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 28 3D model creation Sutherland’s 1968 system: trivially simple

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 29 3D model creation ca Manual model creation increasingly expensive / time-consuming with increasing capability of image generation systems Feasible if 3D model needed to be created as part of the application itself: CAD/CAM, selected medical applications Automatic scanning for selected objects: Cyberware 3D scanners

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 30 3D Model Creation Now Growing list of commercial and research capabilities Computer vision: multiple images / video to 3D world models Laser scanners Motion capture systems

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 31 Assessment of Model Creation Dream in 1968 / 1970: Sutherland: Building model may be as time-consuming as building the real thing Others: There’s got to be an automatic way Grade for where we are now C+ There are many tools and systems, but often still takes much tedious, manual effort Compare how much more work to digitizing this room and its contents than to rendering it

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 32 Assessment Summary Display: perennially, next model will be really good C Image generation: unalloyed triumph! (but don’t let it go to your head) A+ Head tracking: effective but in limited, prepared spaces B Interaction / Hand tracking: awaiting more real applications Incomplete Model generation: frustratingly time-consuming C+

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 33 Opportunities: Look at larger picture than a single module Head-mounted display design combined with image generation Fakespace Wide5: hardware geometric distortion correction Mark Mine (Disney) implementing more sophisticated version in GPU Image generation combined with head-tracking Model creation combined with image generation System problem spread across multiple modules: latency

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 34 Sutherland’s vision of the HMD as the penultimate display still inspires

GH2008: Evolution of Graphics Hardware (c) 2008 Henry Fuchs UNC Chapel Hill 35 In conclusion: Don’t constrain your dreams; there is so much cool graphics HW to invent Stereo and autostereo displays Multi-projector / camera systems Vision capture for image-based rendering Haptics in the small and large Cell phones w/graphics & vision: magic lenses in the real world Telepresence systems (could do an entire hour on this) Finally– Alan Kay The best way to predict the future is to invent it THE END 35