1. A Theory for Photometric Self-Calibration of Multiple Overlapping Projectors and Cameras
Peng Song
Tat-Jen Cham
Centre for Multimedia & Network Technology (CeMNet)
School of Computer Engineering
Nanyang Technological University
Singapore-MIT Alliance

2. Motivation Projector camera systems Photometric calibration
Oblique displays
Uncontrolled environment
Off-the-shelf commodity equipment
Quick and easy setup
Photometric calibration
Camera image
1. Need to do calibration
2. Need to self-calibrate with inexpensive equipment
Results shown are color images from our latest work on colorimetric calibration
Projector input
Camera observation
Predicted image
Calibrated model

3. Related Work
Modeling of combined projector-to-camera transfer function
Surati [99]
Jaynes et al. [03]
Photometers/spectroradiometers
Majumder et al. [00, 02]
Yang et al. [01]
High Dynamic Range imaging approaches
Debevec et al. [97]
Mitsunaga et al. [99]
Raij et al. [04]
LUT or parametric models, can not be factorized if more projectors and cameras are coming in or there is a change of environmental lightings, etc.
Very costly and not off-the-shelf equipment
Use images of the same scene but of different exposures or shutter speed to complete camera calibration first.
Requires moderately expensive cameras with a wide range of shutter speed or aperture size.
What we want is a calibration method can be factorized for more projectors, camera, changing environmental lightings; non-parametric, and do not require expensive camera with high dynamic ranges.

4. Basic Framework Multiple overlapping projectors and a camera …Pn C
Projector1
Projector pixel value I1
Intensity Transfer Function S1
Color Filter
Projector2
Go through color filter to create colored images
Try to find the blocks S1, S2, C which are different for different projectors and cameras
Projector pixel value I2
Intensity Transfer Function S2
Color Filter
Camera P’
Image Pixel Intensity Z
Intensity Transfer Function C
Color Filter
Ambient Lighting

5. Isointensity Curves Two overlapping projectors Isointensity curves
Different projector input combinations affecting a single camera pixel
Isointensity curves
Different combinations with same camera observed value
Same radiance into the camera
Any combination of projector inputs leads to the same radiance coming into the camera
Look at the slices parallel to the bottom plane, each slice will be a plane with a curve of the same camera observed intensities.
These curves called isointensity curves are labeled by camera observed intensities, but indicate the same radiance into the camera.

6. The Staircase Method
Any transition between two isointensity curves leads to the same radiance change
Consider only horizontal and vertical transitions o o
Moving from one point of isointensity curve psi(a) to another point on isointensity curve psi(b), the radiance changes are the same.
To limit the transition direction, let us look at horizontal and vertical transitions.
X, Y axis
Horizontal: from isointensity psi(a) increase I1 while keeping I2 to be the same until it reaches isointensity curve psi(b), the radiance change is shown in the right graph.
Vertical: keeping I1 the same, from isointensity curve psi(b) increase I2 until it reaches isointensity curve psi(a), the radiance change is also the same as in the horizontal case.
Each chunk in the right graph corresponds to the same radiance change by definition.
Calibration using only two isointensity curves, accumulate errors.
Trade-off: the tighter the isointensity curves are, the higher resolution the projector intensity-to-radiance curves are.

7. Recursive Bisection Method
Find a new isointensity curve between two previous isointensity curves
Radiance change from previous curves to the new curve = ½ radiance change between previous curves o o
Moving from any point from one isointensity curve to any point on another isointensity curve, the radiance change is the same.
Given one point on isointensity curve psi(a), find another point on isointensity curve psi(b) such that the other two points on their formed rectangle have the same camera observed intensity values.
The transition from one isointensity curve to the second is the same as the transition from the second isointensity curve to a third one vertically. Since the moving distance is the same for I2, while I1 does not change during the movement.
The radiance change between psi(a) and psi(b) is now an addition of both of the radiance changes induced from the transition. Vertically it is the same.
Recursively doing bisections to get finer resolution do induce some errors, but it avoid the accumulation of errors as in staircase method.

8. Gridline Optimization Method
Define vertical and horizontal gridlines
Want equal radiance change between adjacent gridlines
Diagonal gridline intersections should have the same camera observed intensity
Shift gridlines to minimize variance o o
The staircase method and recursive bisection method made use of limited number of observations, in order to make use of more observations for accurate calibration, we proposed this gridline optimization method.
By definition, vertically and horizontally, from one point on a isointensity curve to a point on another isointensity curve, the radiance change is the same, so that the isointensity curves should be on the diagonal intersection points of this grid.
In order to get the optimal grid, we employed an optimization here which is to minimize the sum of variance of diagonal intersections of this grid mesh. Graphically, it can be think of as “shifting” the gridlines to get their optimal positions.
We have used Newton based method for optimization.
Use larger number of observations for estimation

9. Further Calibration Stages
Derive additional methods for
Computing radiances contribution from projector black offsets & indoor lights
Camera photometric calibration
Extension of calibration of a pixel to all projector pixels across the display
Binary light sources include indoor lights, relative radiance contribution corresponding to projector offsets.

10. Experiments (I)
Estimating Intensity Transfer Functions

11. Experiments (I)
Combined projector-to-camera intensity transfer functions
Observed Data
Observed Vs. Computed Data Difference
Introduce axis and the difference is pixel wise, in pixel level intensity
Computed Data
Mean Absolute Difference = intensity levels

12. Experiments (II)
Prediction of camera output based on known projector input
Observation
Prediction
Observation
Prediction
Observation
Observation
Prediction
Prediction

13. Experiments (III)
Determining projector input to create desired camera observed image
Desired Image
Desired Image
Uncompensated Image
Uncompensated Image
Compensated image
Compensated Image

14. Conclusion and Future Work
A principled framework for photometric self-calibration of overlapping projectors and cameras
three methods proposed
Staircase method
Recursive bisection method
Gridline optimization method
No requirement for expensive equipment
Only needs a cheap camera with minimal controls
Extension of grayscale photometric calibration to colorimetric calibration (in progress)
In certain cases, low dynamic range camera can be used.

15. Q & A
Thank you

16. Time Analysis of Three Methods
The staircase method
Complexity
Worst case time mins
Recursive bisection method
Worst case time mins
Gridline optimization method
Data collection time 1-2 hours