1. Dorota A. Grejner-Brzezinska
Multi-sensor Integration and Calibration Aspects
Dorota A. Grejner-Brzezinska
Civil and Environmental Engineering and Geodetic Science
The Ohio State University
470 Hitchcock Hall
Columbus, OH 43210
Tel. (614)
OSU

2. Presentation Outline Multi-sensor system AIMS Calibration experiences
Camera calibration
Lever arm calibration
OTF INS calibration (GPS)
Boresighting misalignment (AT)
Summary, outlook

3. Multi-sensor System
Kinematic platform, upon which multiple sensors have been integrated and synchronized to a common time base
Provide three-dimensional near-continuous positioning of both the platform and simultaneously collected geo-spatial data
In general direct georeferencing is used
In principle, no external information, such as ground control, is needed except for the GPS base station
Calibration of the components and the link between components -- a challenging task

4. Multi-sensor System: Objectives
Automation of the map making process: the ultimate goal of digital photogrammetry
Sensors based on different physical principles record complementary and often redundant information, leading ultimately to a more consistent scene description
Offer a feasibility of automation of the photogrammetric tasks

5. Multi-sensor Systems: Design Level
Specifications of the accuracy requirements
Sensor specifications
Architecture of the sensor mount
Time synchronization circuit (crucial)
Most challenging:
Algorithms for sensor calibration
Algorithms for data processing
Methods for testing the performance

6. Multi-sensor Systems Variety of new spatial data acquisition sensors
CCD-based cameras
LIDAR (light detection and ranging)
multi/hyper-spectral sensors
SAR/IFSAR
Optimal sensor fusion
more consistent scene description
complementary information
redundant information

7. Direct georeferencing (direct platform orientation DPO)
Geometric data fusion  time-space registration (georeferencing)
Basis for higher level fusion (multi-sensor imaging systems)
GPS/INS integration
complementary and competitive data
high accuracy (5-20 cm, arcsec)
high reliability
fault-resistant
cost effective (less ground control)
mandatory for new spatial data sensors (LIDAR, SAR, multi/hyperspectral)
experimental systems (University of Calgary, OSU AIMS)
commercial (Applanix)
GPS multi-antenna systems for less demanding applications

8. Effects of Errors in Direct georeferencing on the 3D Ground Point Positioning
Depend on:
Quality of GPS/INS error modeling
Quality of boresight transformation
Quality of GPS/INS lever arm estimation
Quality of individual sensor calibration
Rigidity of the mount (platform)

9. 3D Ground Point Positioning Error Resulting from Errors in the Exterior Orientation: Simulation

10. 3D Ground Point Error as a Function of Increasing Errors in Attitude for Variable Error Levels in Exposure Center Location (300 m altitude )

11. 3D Ground Point Error as a Function of Increasing Errors in Exposure Center Location for Variable Error Levels in Attitude (300 m altitude )

12. Calibration of a Multi-sensor System Based on AIMS GPS/INS/CCD
GPS Antenna
Imaging PC
GPS Base Station
BigShot™ Hasselblad Camera
Trimble 4000SSI
LN-100
INS/GPS PC

13. OSU Center for Mapping AIMS
Over 20 airborne tests
Several land-based tests
Typical standard deviations (covariance analysis)
position 2-4 cm
attitude
5-6 arcsec for pitch and roll
7-10 arcses for heading
Typical fit to ground truth
2-20 cm for flight altitude ~ 300m
0.2-3 cm for land-based applications (10-20 m object distance)
Calibration – crucial component

14. System Calibration Concept
GPS
Rover
OTF GPS Error
Estimation and
Ambiguity Resolution
Lever Arm
Calibration
Camera
Calibration
CCD
INS
INS OTF
Calibration
Sensor Mount
Boresight
Misalignment
GPS
Base

15. Hardware Configuration for Land-based Tests
Digital camera
GPS antenna
LN 100

16. Integrated System Calibration
Camera calibration
GPS/INS lever arm calibration
Boresight calibration between the camera body and the INS body frames
linear offsets and the rotation matrix
determined from the imagery containing images of the ground control points

17. Integrated System Calibration
Camera calibration
focal length, principle point coordinates, and radial distortions
performed at the indoor test range
in-flight calibration (limited)
USGS polynomial model (3+6)
especially important for non-metric digital cameras
must to be repeated any time after the CCD was detached

18. Lever Arm Test (2)

19. Radial Symmetric Distortion Surface Carl Zeiss Distagon 4/50 mm
microns

20. Radial Distortion Difference Between Two Calibrations
microns

21. 50-mm lens equipped 4K by 4K CCD frame Camera Calibration

22. CCD Camera Calibration Experiences
Recalibration of AIMS™ digital camera
Insignificant changes in distortion surfaces
Linear parameters vary (especially principal point coordinates)
No check on changing radiometric behavior
Dust accumulates (factory cleaning)
System performance unaffected

23. Lever Arm Determination
Linear offsets from GPS phase center to INS body frame center (usually pre-surveyed)
Have to be known with sufficient accuracy
especially important for embedded systems where INS directly aids the carrier-phase-tracking loop
Could be determined by the integrated filter
speed of estimation depends on the vehicle dynamics

24. Calibration of the GPS Lever Arm Errors
GPS Lever Arm Error Estimation
GPS Lever Arm Error Estimation 10 4 5 2
Bias Estimate (x-axis, cm)
Bias Estimate (z-axis, cm) -5 -2
-10 -4
500
1000
1500
500
1000
1500
time(s)
time(s)
Standard Deviation
Standard Deviation 10 10 8 8 6 6
Standard Deviation (x-axis, cm)
Standard Deviation (z-axis, cm)) 4 4 2 2
500
1000
1500
500
1000
1500
time(s)
time(s)

25. Effects of GPS Lever Arm Errors
DD Carrier Phase Residuals: level arm errors not calibrated
DD Carrier Phase Residuals: level arm errors estimated 20 20 15 15 10 10 5 5
Residual (cm)
Residual (cm) -5 -5
-10
-10
-15
-15
-20
-20
200
400
600
800
1000
1200
1400
1600
200
400
600
800
1000
1200
1400
1600
time(s)
time(s)

26. Lever Arm Test Lever arm offsets
pre-surveyed: 0, 0.95, m
distorted: 0, 0.50, 0 m
INS x-axis oriented towards south during the initial 50-sec stationary period
200-sec vehicle maneuvering

27. Van Trajectory
start

28. Difference between solutions with correct and incorrect lever arm components
Motion starts

29. Difference between solutions with correct and incorrect lever arm components
Motion starts

30. Difference between solutions with correct and incorrect lever arm components (no initial maneuvering)

31. Difference between solutions with correct and incorrect lever arm components (no initial maneuvering)

32. Camera/INS Boresight Misalignment

33. Death Valley,
JPL Test

34. INS/Camera Mount

35. Boresight Misalignment
Since DPO rotational components are naturally related to the INS body frame, they must be transformed to the imaging sensor frame
The angular and linear misalignments between the INS body frame and the imaging sensor frame are known as boresight components
The boresight transformation must be determined with sufficiently high accuracy

36. Boresight Misalignment Calibration
Angular and linear misalignments between the INS body frame and the imaging sensor frame
Resolved by comparison of the GPS/INS positioning/orientation results with independent AT solution
or as a part of a modified bundle adjustment with constraints
Should be performed at a specialized test range
No flex or rotation of the common mount of the imaging and the georeferencing sensors can occur

37. Direct Georeferencing
ZBINS
XBINS XC YC
YBINS ZM XM YM
rM,k
rm,i,j
rM,INS
3D INS coordinates in mapping frame
3D object coordinates in model frame (derived from i,j stereo pair) attached to C-frame
3D coordinates of point k in M-frame
boresight matrix between INS body frame and camera frame C
rotation matrix between INS body frame and mapping frame M, measured by INS
boresight offset components
scaling factor s

38. Error in Object Coordinates Due to Errors in Boresighting

39. Boresight Calibration: Example
Aerotriangulation Results for Boresight Calibration
SoftPlotter data reduction and adjustment packages
2 cm accuracy assumed for control points
7  for image coordinate observations
GPS/INS Results for Boresight Calibration
5-8 arcsec estimated standard deviations of attitude components
1-2 cm estimated standard deviations of INS center position

40. Boresight Matrix Estimation: Example
RPht - photogrammetrically derived attitude matrix
RINS - GPS/INS derived attitude matrix
S - reflection matrix to approximately align INS and camera frame axes (could be absorbed directly by RB
RB - boresight matrix

41. Boresight Estimation Quality
Performed on ground control points (natural objects)
The standard deviations for the boresight components
linear displacements : 0.22, 0.08 and 0.06 m
rotation angles , , : 0.01, 0.03 and 0.04 deg
Possible reasons for modest quality:
Photogrammertic processing accuracy
affected by poor signalization of control points
Mechanical problems with the camera body/mount
Image time tagging (rather unlikely)

42. IMU OTF Error Calibration
The main sources of errors in an inertial navigation are due to the following factors:
The time rates of change of the velocity errors are driven chiefly by accelerometer errors and gravity disturbance
The attitude error rates are driven primarily by gyroscope errors
In the aligned INS the main errors in tilt are due to accelerometer bias and the gravity disturbance vector
It is important that the IMU errors are properly estimated and applied by the feedback loop
Platform maneuvers are needed to separated error sources

43. IMU Error Calibration
m/s2
Straight portion of the flight

44. IMU Error Calibration
arcsec/h

45. IMU Error Calibration – Flight Trajectory

46. Accelerometer Scaling Factor for Different Atmospheric Conditions
GPS time [s]
GPS time [s]

47. Positioning Error Growth During GPS Losses of Lock Due to Obstruction or Interference (forward direction, end of the gap)

48. Summary Direct orientation can be achieved with high accuracy
Proper system calibration is crucial
AT is needed for boresight calibration
Radio interference poses problems (AT is needed)
Continuity of the GPS lock can pose serious problems in land-based applications
Rigidity of the multi-sensor system mount -- embedded systems are preferred
Precise GPS/INS time synchronization is crucial