Presentation on theme: "Moveable Interactive Projected Displays Using Projector Based Tracking"— Presentation transcript:
1 Moveable Interactive Projected Displays Using Projector Based Tracking Johnny C. Lee1,2Scott E. Hudson1Jay W. Summet3Paul H. Dietz21Carnegie Mellon University2Mitsubishi Electric Research Labs3Georgia Tech UniversitySeattle, WA UIST 2005
2 UIST 2004 – Automatic Projector Calibration Embedded light sensor in surface.Project patterns to find sensor locations.Pre-warp source image to fit surface.(video clip 1)Correspondence between location data between and projected image is free (e.g. no need for calibration with external tracking system)Transforms passive surfaces into active displays in a practical manner.Variety of useful applications
4 Focus on Moveable Projected Displays Goals of this work:Achieve interactive tracking rates for hand-held surfaces.Reduce the perceptibility of the location discovery patterns.Explore interaction techniques supported by this approach.
5 Display Surface Constructed from foam core and paper Touch-sensitivity is provided by a resistive filmLighter than a legal pad
9 Difference is visible to the human eye Gray Code PatternsBlack and WhiteDifference is visible to the human eye
10 Difference is visible to the human eye Gray Code PatternsBlack and WhiteDifference is visible to the human eye
11 Difference is visible to the human eye Gray Code PatternsBlack and WhiteDifference is visible to the human eye
12 Difference is visible to the human eye Gray Code PatternsBlack and WhiteDifference is visible to the human eye
13 Difference is visible to the human eye Gray Code PatternsBlack and WhiteDifference is visible to the human eye
14 Gray Code Patterns Black and White Frequency Shift Keyed (FSK) HF LF Difference is visible to the human eyeDifference is NOT visible to the human eye
15 Gray Code Patterns Black and White Frequency Shift Keyed (FSK) HF LF Difference is visible to the human eyeDifference is NOT visible to the human eye
16 Gray Code Patterns Black and White Frequency Shift Keyed (FSK) HF LF Difference is visible to the human eyeDifference is NOT visible to the human eye
17 FSK Transmission of Patterns FSK transmission of the Gray Code patterns makes the stripped region boundaries invisible to the human eye.Patterns appear to be solid grey squares to observers.Light sensor is able to demodulate the HF and LF regions into 0’s and 1’sThis is accomplished using a modified DLP projector
18 Inside a DLP projector DLP = Digital Light Processing Many consumer projectors currently use DLP technology“DLP” is Texas Instruments marketing term for DMDDMD = Digital Micro-mirror DeviceEach mirror corresponds to a pixelBrightness corresponds to duty cycle of mirrorPictures from Texas Instruments literature
19 Inside a DLP projectorLight sourceProjector LensColor wheelDMD
21 FSK Transmission of Location Patterns Removing the color wheel flattens the color space of the projector into a monochrome scaleMultiple points in the former color space now have the same apparent intensity to a human observer, but are manifested by differing signals.The patterns formerly known as “red” and “grey” are rendered as 180Hz and 360Hz signals respectively.Monochrome projector is not ideal, but is a proof of concept device until we can build a custom DMD projector.
24 Incremental TrackingProject small tracking patterns over the last known locations of each sensor for incremental offsetsBlack masks reduce visibility of tracking patternsTracking loss strategies are needed (later)Smaller area = fewer patterns = faster updates32x32 unit grid requiring 10 images6Hz update rate
26 Latency and Interleaving Incremental tracking is a tight feedback loop:project update project update …6Hz update rate assumes 100% utilization of the frames/sec the projector can displaySystem latencies negatively impact channel utilizationAchieving 100% utilization of the projection channel requires taking advantage of the axis-independence of Gray Code patterns.
27 System Latency – Full X-Y Tracking Only 73% utilizationProjection:X patternsY patternsX patternsY patternsGraphics& VideoHardware &OS schedulingSoftware:drawX-Y patternsupdate& drawX-Y patternsTime
28 System Latency - Interleaved Tracking 100% utilization of the projection channel and 12Hz interleaved updateProjection:X patternsY patternsX patternsY patternsSoftware:drawX patternsdrawY patternsupdate& drawX patternsupdate& drawY patternsupdate& drawX patternsTime
30 Tracking Pattern Size +25% rate -75% area Tracking Area Tracking Rate 32x32 grid12Hz interleaved16x16 grid15Hz interleaved+25% rate-75% areaSmaller tracking area increases risk of losing sensor (e.g. maximum supported velocity)log2 relationship makes it hard to gain speed though the use of smaller patterns
32 Tracking Pattern Size large, coarse tracking pattern small, fine Preserves physical size of tracking pattern (cm)Preserves maximum supported velocity (m/s)Distance is approximated from screen sizeScaling factor is adjustable (precision vs. max velocity): ~2.5mm; 25cm/s
33 Motion ModelingPredicting the motion can be used to increase the range of supported movement (e.g. max acceleration vs. max velocity)Much of the work in motion modeling is applicable. But, no model is perfect and mis-predictions can lead to tracking loss potentially yielding poorer overall performance.Models are likely to be application and implementation specific.va
34 Tracking Pattern Shape We used square tracking patterns due to the axis aligned nature of Gray code patterns.Patterns with high-radial symmetry are best for general movement in two-dimensions.Pattern geometry can be optimized for specific applications.
36 Detecting Occlusions or Tracking Loss Causes of tracking loss:occlusionsexiting the projection areaexceeding the range of motion supported by the tracking patternsWith FSK transmission, tracking loss corresponds to a disappearance of the carrier signal. This allows error detection on a per-bit basis.Implemented on a low-cost PIC processor as:sudden drop in signal amplitudeinsufficient amplitudeinvalid edge count
37 Lost Tracking Behavior Single/independent sensors:Discard and hope the sensor has not movedPerform a full screen discovery process (+333ms)Grow the tracking pattern around last location until reacquiredMultiple sensors of known geometric relationship:Try the above three techniques.Compute predicted lost sensor locations using the locations of the remaining available sensors.
38 Tracking Loss With Multiple Sensors video clip 4
39 Estimating Lost Sensors Available SensorsAction4/4No estimation needed, compute the 4 point warping homography directly3/4Measure 6 offsets, compute affine transform to estimate lost sensor from last known location2/4Measure 4 offsets, compute simplified transform to estimate lost sensor locations1/4Measure 2 offsets, compute 2D translation0/4Try full screen discoveryNote: Transformations for each point cannot be implemented as a simple matrix stack because LIFO ordering of sensor loss and re-acquisition is not guaranteed.
43 ConclusionUnifying the tracking and projection technology greatly simplifies the implementation and execution of applications that combine motion tracking with projected imagery.Coherence between the location data and projected image is free.Does not require an external tracking system or calibrationSimple: Demos were created in about a weekThis approach has the potential to change the economics of interactive displaysThe marginal cost of each display can be as low as $10 USDMuseum: wireless displays could be handed out to visitors.Medical Clinic: physical organization of patient charts/folders
44 Future WorkRemoving the color wheel was a proof-of-concept work around.Construct high-speed projector using a DLP development kitExplore using infrared to project invisible patternsExplore other applications where low-speed positioning is sufficient.Achieve +18Hz (+36Hz interleaved) tracking with visible patterns and an unmodified DLP projector using RGB sections.Using multiple projectors (or steerable projectors) to increase freedom of movement.
45 AcknowledgementsFunded in part by the National Science Foundation under grants IIS and IISFunded in part by Mitsubishi Electric Research LabsJohnny Chung Lee
46 Technical Details Infocus X1 ($800) 800x600, 60Hz PIC16F819 at 20Mhz, 10bit ADCSensor package < $10 in volume4-wire resistive touch sensitive filmIF-D92 fiber optic phototransistors45Bytes/sec for location data25mW during active trackingLatency (77ms – 185ms)