1. Hybrid Infrared and Visible Light Projection for Location Tracking
Johnny Lee, Scott Hudson, Paul Dietz
Carnegie Mellon University
Mitsubishi Electric Research Labs
UIST 2007 – Newport, RI

2. Hybrid Projection
one projector
infrared
visible
But, why?

3. Projector-Based Location Discovery [Lee, UIST’04]
Light sensors
Projector
Electronics & computer

4. Projector-Based Location Discovery
calibration free:
- no computer vision
- no alignment
- no manual input
Scalable and robust

5. Moveable Surfaces calibration free no external tracker
[Lee, UIST 2005]
calibration free
no external tracker
- interactive content

6. Drawbacks Location Discovery [‘04] Moveable Surfaces [‘05]
Incremental tracking
Caustic B&W patterns
Momentary movement

7. Ideally
Full-screen
Application Content
Full-screen
Location Patterns

8. Hybrid Projection one projector infrared visible for the computer
for the human

9. Infrared & Visible Projection
If we could make a projector the can emit IR images
Could do full screen location discovery without the viewer’s knowledge
Simultaneously project visible light application content.
Chose green for infrared because that is how it is rendered in night-vision cameras.

10. Infrared & Visible Projection

11. Infrared & Visible Projection

12. Infrared & Visible Projection

13. Infrared & Visible Projection

14. Infrared & Visible Projection

15. Infrared & Visible Projection

16. Infrared & Visible Projection

17. Infrared & Visible Projection

18. Infrared & Visible Projection

19. How?

20. Light Source: Lamps Xenon Arc Lamp
Lots of IR energy, sometimes 100 times
But this is energy we can’t see so manufactures use IR cutoff filters
Reduces heat and wear on the imaging element
Even if the optics were changed
Unmodulated light is hard to use (easy interference)
Xenon Arc Lamp

21. Light Technologies You are probably familiar with these, but
LED stands for light emitting diode
Available in a wide variety of form factors, and power, and
Most importantly color.

22. More Efficient and Better Lifespan
The benefit of using an LED light source
Lower heat
Lower power
Smaller – most of a projector is power supply and cooling
Handheld projector becoming possible.
Products emerging on market this year, 2007

23. IR and Visible Light LEDs 1000 LEDs/mm2
This is our LED light source.
Notice only two color bands.
Since I’m building these by hand,
Don’t have manufacturing capability to create quad color LED arrays that are dense enough.
But a commercial manufacturer could easily do this.
So, for our proof-of-concept prototype we use
visible red light and invisible infrared.
IR and Visible Light LEDs
1000 LEDs/mm2
University of Strathclyde, Institute of Photonics

24. LED Array DMD Lens Projection optics
Inside the DLP projector
LED array
Culminating lens which focuses light onto
the DLP chip
DLP reflect light into the barrel of projection optics.
Projection optics

25. Our Dev Kit: 180 binary images/s
High-Speed Dev Kit: 16,500 binary images/s
Production Unit: +50,000 binary images/s
We use a DLP Discovery development kit to control the mirror state on the chip.
1024x768 area = 20 binary images
60Hz tracking = 2.4% duty cycle of production DMD
Required changes to commercial designs would be minimal.

26. Demo of Capability
Uses a second projector for visible content

27. Inherent Multi-Stylus Tracking
Inherent stylus support on any surface
Multi simultaneous users with user ID
Possible with a minor design change to upcoming generation of DLP projectors.

28. Non-Planar and Discontinuous Surfaces
Inherent stylus support on any surface
Multi simultaneous users with user ID
Possible with a minor design change to upcoming generation of DLP projectors.

29. Static IR Patterns

30. Concept Applications
Simulated using external tracking (calibration)

31. Hand-held projection with photosensitive tags
In siggraph 2004, explored interaction of handheld projection
Simulated using camera-based tracking, but enabled by IR projector-based tracking
Selection
Grouping
Annotating
[Siggraph 2004]

32. Foldable Interactive Displays [submitted to CHI]

33. Acknowledgements Johnny Chung Lee
Funded in part by the National Science Foundation under grants IIS and IIS
Funded in part by Mitsubishi Electric Research Labs
Johnny Chung Lee

34. Other ways to make invisible patterns
Other non-visible wavelengths
Steganography
Color shifting
Noise encoding
Bit Timing
Synchronization may be difficult

35. Camera-Based Tracking
Requires calibration
Requires markers for segmentation
IR sensors + transmitter is less power than 4 IR LEDs
Does not provide ID
Limitation on the number of points
Limitation on tracking rate
Limitation on scene/target complexity
Resolution is not as scalable
Less optically robust
Optical path geometry and variable illumination

36. Projector vs Camera Tracking
Sensors provide point ID
Independent of scene/surface complexity

37. Space-Labeling Projectors
11 Infrared LED slide projectors
Potentially Low-cost
Per axis: 500Hz tracking at 10-bits
stacked of 11 LED slide projectors
Requires cylindrical optics
Can do tracking outdoors in complex environments
And in dynamic lighting
Tracking could be boosted to 25KHz.
Outdoor motion tracking

38. Binary Gray
0000
0001
0010
0011
0100
0101
0110
0111
1000
1001
1010
1011
1100
1101
1110
1111
0000
0001
0011
0010
0110
0111
0101
0100
1100
1101
1111
1110
1010
1011
1001
1000

39. Binary Gray
0000
0001
0010
0011
0100
0101
0110
0111
1000
1001
1010
1011
1100
1101
1110
1111
0000
0001
0011
0010
0110
0111
0101
0100
1100
1101
1111
1110
1010
1011
1001
1000

40. Binary Gray

41. Binary Gray

42. Binary Gray

43. Binary Gray

44. Binary Gray

45. Binary Gray

46. Binary Gray

47. Binary Gray

48. Binary Gray

49. Binary Gray

50. Binary Gray

51. Binary Gray

52. Pixart Chip 1024x768 resolution 100Hz tracking 4 points Only 4 points
High power LED points