1. Civil and Environmental Engineering and Geodetic Science Integrated GPS/INS/CCD System for Airborne Image Data Acquisition Dorota A. Grejner-Brzezinska Civil and Environmental Engineering and Geodetic Science The Ohio State University 470 Hitchcock Hall Columbus, OH 43210 Tel. (614) 292-8787 E-mail: dorota@cfm.ohio-state.edu OSU

2. Civil and Environmental Engineering and Geodetic Science Presentation outline   Direct georeferencing concept   GPS/INS integration for positioning and orientation INS component GPS component Primary integration architectures   Summary

3. Civil and Environmental Engineering and Geodetic Science Georeferencing: the Concept (1) Sensor orientation, also called image georeferencing, is defined by a transformation between the image coordinates specified in the camera frame and the geodetic (mapping) reference frame. requires knowledge of the camera interior and exterior orientation parameters (EOP) interior orientation: principal point coordinates, focal length, and lens geometric distortion are provided by the camera calibration procedure exterior orientation: spatial coordinates of the perspective center, and three rotation angles known as , , and 

4. Civil and Environmental Engineering and Geodetic Science Georeferencing: the Concept (2)   Traditional aerial surveying EOP determined from the aerotriangulation, defining correlation between ground control points and their corresponding image representations requires scene pre-targeting high cost labor intensive

5. Civil and Environmental Engineering and Geodetic Science Georeferencing: the Concept (3)   Modern aerial surveying EOP determined directly from integrated sensors such as GPS/INS or GPS antenna array no scene pre-targeting (no ground control, except for GPS base station) no aerotriangulation low cost allows automation of the data image processing

6. Civil and Environmental Engineering and Geodetic Science Automation of Aerial Survey   System augmentation by an inertial sensor offers a number of advantages over a stand-alone GPS - - immunity to GPS outages - - continuous attitude solution - - reduced ambiguity search volume/time - - high accuracy and stability over time contributed by GPS, enabling a continuous monitoring of inertial sensor errors   Result  direct platform orientation (geo-referencing)

7. Civil and Environmental Engineering and Geodetic Science

8. Direct Geo-referencing   Increased interest in the aerial survey and remote sensing community   need to accommodate the new spatial data sensors (LIDAR, SAR, multi/hyperspectral)   cost reduction of aerial mapping decreased need for control points   maturity and cost-effectiveness of GPS/INS systems   GPS multi-antenna systems for less demanding applications   GPS/INS systems available: experimental - University of Calgary, Center for Mapping OSU commercial - Applanix

9. Civil and Environmental Engineering and Geodetic Science Direct Orientation Land-based System

10. Civil and Environmental Engineering and Geodetic Science Direct Orientation Land-based System Digital camera GPS antenna INS

11. Civil and Environmental Engineering and Geodetic Science Direct Orientation Airborne System GPS Antenna Camera INSPC Two Base Stations GPS Receiver

12. Civil and Environmental Engineering and Geodetic Science Direct Georeferencing Z BI NS X BI NS XCXC YCYC Y BI NS ZMZM XMXM YMYM r M,k r m,i,j r M,INS – – 3D INS coordinates in mapping frame – – 3D object coordinates in model frame (derived from i,j stereo pair) attached to C-frame – – 3D coordinates of point k in M-frame – – boresight matrix between INS body frame and camera frame C – – rotation matrix between INS body frame and mapping frame M, measured by INS – – boresight offset components – – scaling factor r M,INS r m,i,j r M,k s

13. Civil and Environmental Engineering and Geodetic Science GPS/INS Integration for Direct Orientation (direct geo-referencing) of the Imaging Component

14. Civil and Environmental Engineering and Geodetic Science Principles of Inertial Navigation   Principles defined in the i-frame (inertial)   Real time indication of position and velocity of a moving vehicle using sensors that react on the basis of Newton’s laws of motion   these sensors are called Inertial Measurement Units (IMU) accelerometers sense linear acceleration in inertial frame does not sense the presence of a gravitational field gyroscopes (sense rotational motion) facilitate the rotation between navigation and INS body frames (in fact rotation with respect to the inertial frame is measured)   Integration with respect to time of the sensed acceleration to obtain velocity, and subsequent integration to obtain position

15. Civil and Environmental Engineering and Geodetic Science Coordinate Frame Geometry

16. Civil and Environmental Engineering and Geodetic Science Inertial Navigation System (INS)   Provides self-contained independent means for 3-D positioning   Three gyros and three accelerometers (or less)   Accuracy degrades exponentially with time due to unbounded positioning errors caused by uncompensated gyro errors uncompensated accelerometer errors fast degradation for low cost INS   High update rate (up to 256 Hz)   Mechanical (stabilized platform) systems sense acceleration in inertial frame coordinatized in navigation frame   Strapdown systems (digital) sense acceleration in inertial frame coordinatized in body frame

17. Civil and Environmental Engineering and Geodetic Science -Y -X Z INS LN-100 Body Axes

18. Civil and Environmental Engineering and Geodetic Science Primary Error Sources   The main sources of errors in an inertial navigation are due to the following factors: The time rates of change of the velocity errors are driven chiefly by accelerometer errors and gravity anomalies The attitude error rates are driven primarily by gyroscope errors   Three basic classes of errors physical component error: deviation of inertial sensors from their design behavior (drifts, bias, scaling factors) construction errors: errors in overall system construction such as mechanical alignment errors initial conditions: errors that arise from imperfect determination of the initial position error, initial velocity error, and initial platform misalignment

19. Civil and Environmental Engineering and Geodetic Science Comparison of GPS and INS Free Navigation Trajectories (Road Test) 0.40.81.21.62.0 (Latitude = 39.99° Longitude = -83.045 ) 1.1 2.2 3.3 4.4 5.5 6.6 7.7 8.8 East Distance (km) North Distance (Km) GPS: INS: GPS start & end position Total Time 1867s 0 0 INS end position

20. Civil and Environmental Engineering and Geodetic Science Strapdown INS   Strapdown system algorithms are the mathematical definitions of processes, which convert the measured outputs of IMUs that are fixed to the vehicle body axis, into quantities that can be used to control the vehicle (attitude, velocity and positions)   The outputs are angular rates and linear velocities along the orthogonal axes   The measured angular rates are converted into changes in attitude of the vehicle with respect to its initial orientation   The resulting attitude transformation matrix is used to convert the measured velocities from body axes to the reference coordinate system   The major algorithms are: Start-up Initialization Generation of the transformation algorithm Navigation

21. Civil and Environmental Engineering and Geodetic Science Navigation Equations f - acceleration due to applied force sensed by accelerometer g(r) - gravitational acceleration r – geocentric vector of vehicle position v(t) – velocity of the vehicle relative to the earth defined in the navigation system   A general navigation equation that describes the motion of a point mass over the surface of the earth - earth rotation vector - angular rate of the navigation frame relative to the earth

22. Civil and Environmental Engineering and Geodetic Science Navigation Equations (cont) - - the Earth’s rotation rate; L is the geodetic latitude, and is the longitude.   the superscript “ n” means a vector is coordinated in the n-frame - the direction cosine matrix from body-fixed coordinates (b-frame) to navigation coordinates (n-frame)

23. Civil and Environmental Engineering and Geodetic Science Strapdown Inertial Mechanization Body-Mounted Gyroscopes Body-Mounted Accelerometers Body-Mounted Accelerometers Transformation Matrix Computation Gravity Computation Gravity Computation Earth and Vehicle Rate Computation Earth and Vehicle Rate Computation C b n    V  V ib b b n in b    V nb n  b  _ _ Euler Angle Computation Euler Angle Computation + + Position Velocity Attitude   V C ib b b b n = Delta velocity from accelerometers = Delta  from gyros (angular rates) = Direction cosine matrix from b-frame to n-frame

24. Civil and Environmental Engineering and Geodetic Science Why GPS/INS Integration?   GPS and INS have complementary operational characteristics GPS contributes its white error spectrum, high accuracy and stability over time, enabling a continuous monitoring of inertial sensor errors Calibrated INS offers high short-term accuracy and high sampling rate INS is self-contained; no outages   GPS/INS offers a number of advantages over a stand-alone GPS immunity to GPS outages and reduced ambiguity search volume/time for the closed-loop systems and more importantly, continuous attitude solution   Implementation of a closed-loop error calibration allows continuous, on-the-fly (OTF) error update bounding INS errors, leading to increased estimation accuracy   Two primary integration modes Loose coupling Tight coupling (closed-loop)

25. Civil and Environmental Engineering and Geodetic Science GPS/INS Integration   Limitations High cost of high quality INS Mission objectives (what INS is needed and what can be afforded) How complex is the integration scheme required to achieve the emission objectives

26. Civil and Environmental Engineering and Geodetic Science GPS/INS Integration   Loosely coupled mode Separate filter to process GPS data IMU measurements are processed by inertial navigation and attitude algorithms to give inertial position, velocity, and attitude solution An integrated Kalman filter is then applied to combine the GPS and inertial solutions The main disadvantage: positioning must be performed on the basis of IMU measurements alone when the number of tracked GPS satellites drops below four

27. Civil and Environmental Engineering and Geodetic Science GPS/INS Integration   Tightly coupled mode A single Kalman filter is designed to process both sets of sensor data: raw GPS observations and IMU measurements OTF IMU error calibration by a feedback loop Allows use of GPS measurements from less than four satellites High accuracy of the inertial system over short periods of time allows correction of undetected cycle slips affecting GPS measurements Makes it possible to perform a faster and more robust OTF ambiguity resolution Offers a possibility to support GPS tracking loop by a direct feedback from the integrated filter to maintain tracking during high dynamics Offers superior performance compared to loosely coupled mode

28. Civil and Environmental Engineering and Geodetic Science Architecture of Tightly Integrated GPS/INS System IMU Airborne Receiver Ground Receiver Double Differential GPS Computation IMU Error Compensation Strapdown Navigation Computation OTF Ambiguity Resolution OTF Ambiguity Resolution Error Compensation Error Compensation DD Observation Generation DD Observation Generation GPS/INS Kalman Filter Construct Model Parameters Covariance Propagation Optimal Gain Computation Covariance Update State Estimate Propagation Measurement Residual Generation Residual Testing State Estimate Update Position & Velocity Computation Coordinate Transform Quaternion Computation Euler Angle Computation

29. Civil and Environmental Engineering and Geodetic Science Airborne Integrated Mapping System (AIMS™)   Fully digital data acquisition system for large-scale mapping and other precise positioning applications   Designed for airborne applications   Incorporates state-of-the-art GPS/INS positioning and digital imaging technologies.

30. Civil and Environmental Engineering and Geodetic Science   Theoretical foundation and expertise build-up at The Center for Mapping – –GPSVan (first Mobile Mapping System) – –Real-time cm-level positioning (Construction industry)   Technology component from leading manufactures – –GPS (Trimble) – –INS (Litton) – –CCD (Lockheed Martin Fairchild)   Commercialization through industry partners AIMS ™ Resources

31. Civil and Environmental Engineering and Geodetic Science   System design and implementation   GPS/INS component   Imaging component   GPS ambiguity resolution for GPS/INS integration   Kalman filter design AIMS™ Architecture

32. Civil and Environmental Engineering and Geodetic Science AIMS™ Positioning Component   Tightly coupled GPS/INS   Target accuracy: 4-7 cm position, ~10 arcsec orientation over long baselines   Trimble 4000SSI dual-frequency GPS receivers   Litton LN-100 Strapdown Inertial Navigation System Zero-lock TM Laser Gyro (ZLG TM ) A-4 accelerometer triad 0.8 nmi/h CEP, gyro bias – 0.003  /h, accelerometer bias – 25  g

33. Civil and Environmental Engineering and Geodetic Science GPS Component   Trimble 4000SSI dual-frequency GPS receivers   Double-differenced L1 phases (optionally L2)   Initial ambiguity resolution On-The-Fly (OTF)   INS-aided OTF ambiguity resolution   Stochastic modeling of the DD ionospheric effect (random walk)   1Hz data acquisition rate   Timing provided by GPS time and   1pps signal supports GPS and INS clocks synchronization

34. Civil and Environmental Engineering and Geodetic Science   Litton LN-100 Inertial Navigation System – –Modified firmware for AIMS – –IMU data are collected   Data Collection – –IMU output data collected at 400 Hz – –INS internal timer is synchronized to GPS time – –Time-tagging INS data using both GPS and INS time – –Navigation solution provided at 256 Hz INS Component

35. Civil and Environmental Engineering and Geodetic Science AIMS Imaging Component   Digital frame camera – –based on Lockheed Martin Failrchild 4K  4K CCD sensor – – 60mm by 60mm imaging area (15-micron pixel size) – –Hasselblad 553 ELX camera body with Zeiss lens – –6 second image acquisition rate   Fully Digital Airborne Data Acquisition System – –direct connection to softcopy systems – –corridor and other large-scale mapping applications   Long-term objective – –Create a 9K by 9K advanced AIMS digital camera

36. Civil and Environmental Engineering and Geodetic Science BigShot /Hasselblad Camera

37. Civil and Environmental Engineering and Geodetic Science Digital Camera/INS Mount

38. Civil and Environmental Engineering and Geodetic Science Image Acquisition Control and Storage Image Acquisition Control and Storage Strapdown Navigation Solution Tightly Coupled GPS/INS Kalman Filter Strapdown Navigation Solution Tightly Coupled GPS/INS Kalman Filter User Interface, Control & Display Unit User Interface, Control & Display Unit 4K by 4K BigShot  Digital Camera 4K by 4K BigShot  Digital Camera Rover GPS Station Rover GPS Station LN-100 Base GPS Station Base GPS Station L1 and L2 phase observable Delta V Delta  L1 and L2 Image Data Exposure Control Time Tag/Sync EO Data Host/Slave Communication Control Signal Optimal Position, Velocity, Attitude Estimates AIMS™ Prototype Architecture

39. Civil and Environmental Engineering and Geodetic Science GPS AntennaImaging PC GPS Base Station BigShot™ Hasselblad Camera Trimble 4000SSI LN-100 INS/GPS PC AIMS™ Hardware Configuration

40. Civil and Environmental Engineering and Geodetic Science   Error states chosen (as opposed to whole-value filter states)   INS psi-angle error model adopted AIMS GPS/INS Kalman Filter Design where  v,  r, and  are the velocity, position, and attitude error vectors respectively;  is the accelerometer error vector;  g is the error in the computed gravity vector; and finally,  is the gyro drift vector.   Observations: double differenced L1/L2 phase measurements

41. Civil and Environmental Engineering and Geodetic Science   The complexity of the INS error model depends on the error models for IMU sensor measurement errors, as well as the gravity uncertainty. For example, a 24-state model may include the following variables: Subscript b stands for bias Subscript f stands for scaling factor AIMS GPS/INS Kalman Filter Design

42. Civil and Environmental Engineering and Geodetic Science Where X Nav, X Acc, X Gyro, X Grav, X Ant and X GPS are, respectively, the error vectors of the inertial navigation solution (position, velocity, attitude), the accelerometer measurement errors (bias, scaling factor), the gyro measurement error (drift; may include scaling factor and bias), the gravity anomaly and deflections errors, the antenna lever arm errors, and the GPS ionospheric errors; w Nav, w Acc, w Gyro, w Grav, w GPS and are all zero-mean Gaussian white noise vectors.  State equations based on the chosen error model AIMS GPS/INS Kalman Filter Design

43. Civil and Environmental Engineering and Geodetic Science AIMS GPS/INS Kalman Filter States

44. Civil and Environmental Engineering and Geodetic Science   Short baseline between the rover and the base assumed   Simultaneous estimation of widelane and geometry-free ambiguities using combinations of four independent GPS code and phase observations for four primary satellites   Implementation of the least-squares ambiguity search, based on the estimated widelane (N 1 - N 2 ) and geometry free (N 1 - 77/60 N 2 ) ambiguities to resolve N 1 and N 2   Positions computed based on fixed ambiguities are used to initialize the GPS/INS filter Initial Ambiguity Resolution

45. Civil and Environmental Engineering and Geodetic Science Compute DD using satellite coordinates and INS-predicted position and ionosphere,  Compute DD using satellite coordinates and INS-predicted position and ionosphere,  Form DD using measured phase  Form DD using measured phase  Check geometry-free and widelane conditions Check geometry-free and widelane conditions cycle slip ? Compute  =  p -  o Compute  =  p -  o Search First Guess   tol. Yes No Kalman Filer Update Kalman Filer Update success ? Yes Use only satellites with fixed ambiguities Use only satellites with fixed ambiguities No INS-aided OTF Ambiguity Resolution

46. Civil and Environmental Engineering and Geodetic Science AIMS  Performance Analysis   Level of double-difference residuals   Covariance matrix analysis   Comparison with traditional aerotriangulation   Ground truth test

47. Civil and Environmental Engineering and Geodetic Science Time [sec] 0100200300400500 600 700 -0.02 -0.015 -0.01 -0.005 0 0.005 0.01 0.015 0.02 DD Residuals [m] Double Difference Residuals ~ 10 km baseline

48. Civil and Environmental Engineering and Geodetic Science L1 Double Difference Residuals ~ 350 km baseline Back to base

49. Civil and Environmental Engineering and Geodetic Science Position Estimation Accuracy 0100200300400500600700 0 0.01 0.02 0.03 0.04 0.05 0.06 0.07 GPS Week Time (416455 - 417176 seconds) position errors (1 sigma, meters)

50. Civil and Environmental Engineering and Geodetic Science Velocity Estimation Accuracy time (seconds) 0100200300400500600700 0 0.001 0.002 0.003 0.004 0.005 0.006 0.007 0.008 0.009 0.01 velocity errors(1 sigma, m/s)

51. Civil and Environmental Engineering and Geodetic Science Orientation Estimation Accuracy time (seconds) 0100200300400500600700 0 10 20 30 40 50 60 orientation errors (1 sigma, arc seconds)

52. Civil and Environmental Engineering and Geodetic Science INS Positioning Errors when GPS Observations are not Used GPS Week Time (418798 - 418848 seconds) 400410420430440450 0 0.05 0.1 0.15 0.2 0.25 Positioning Errors (m)

53. Civil and Environmental Engineering and Geodetic Science Horizontal Error Growth During GPS Outage Introduced at Epoch 60 with Gravity Compensation (known deflections of the vertical)

54. Civil and Environmental Engineering and Geodetic Science Effect of (partially) Known Deflections of the Vertical (DOV)   Estimability of the attitude components improves by adding partially compensated (1 arcsec RMS) gravity information   Kalman filter converges faster and the standard deviation spectrum is less sensitive to the lack of maneuvers   Free inertial navigation during the GPS outages with DOV compensation shows slower error accumulation, especially in the horizontal components

55. Civil and Environmental Engineering and Geodetic Science Callaghan FL Test Flight   Scale 1:6,000 (altitude ~ 300 m)   GSD ~ 10cm   Ground control good to 2 cm (1 sigma)   Maximum base-rover separation ~ 350 km   Several passes with phased image capture, to resolve the conflict between the required high image acquisition rate (1.5sec) and the limited camera cycling time (6 sec)

56. Civil and Environmental Engineering and Geodetic Science Callaghan FL Test Area -81.9-81.88-81.86-81.84-81.82-81.8 30.52 30.54 30.56 30.58 30.6 longitude [deg] latitude [deg] - control points *

57. Civil and Environmental Engineering and Geodetic Science Aerotriangulation Results   SoftPlotter data reduction   SoftPlotter and OSU adjustment packages   2 cm accuracy assumed for control points   7  for image coordinate observations

58. Civil and Environmental Engineering and Geodetic Science The Purpose of Aerotriangulation   Camera/INS boresight calibration   based on the set of imagery containing a set of control points)   depends strongly on the AT quality   Direct orientation performance testing   by comparison with AT results   uses images different from those used for boresight calibration

59. Civil and Environmental Engineering and Geodetic Science Boresight Misalignment Calibration   Angular and linear misalignments between the INS body frame and the imaging sensor frame   Resolved by comparison of the GPS/INS positioning/orientation results with independent AT solution or as a part of a modified bundle adjustment with constraints   Should be performed at a specialized test range   No flex or rotation of the common mount of the imaging and the georeferencing sensors can occur

60. Civil and Environmental Engineering and Geodetic Science Error in Object Coordinates Due to Errors in Boresighting

61. Civil and Environmental Engineering and Geodetic Science GPS/INS Positions vs. AT (projection center coordinates) GPS/INS Positions vs. AT (projection center coordinates) 051015 -0.4 -0.3 -0.2 -0.1 0 0.1 0.2 0.3 0.4 control points coordinate difference [m] x difference y difference z difference

62. Civil and Environmental Engineering and Geodetic Science GPS/INS Attitude vs. AT GPS/INS Attitude vs. AT heading pitch roll

63. Civil and Environmental Engineering and Geodetic Science Boresight Estimation Quality Performed on ground control points (natural objects) The standard deviations for the boresight components · · linear displacements : 0.22, 0.08 and 0.06 m · · rotation angles , ,  : 0.01, 0.03 and 0.04 deg Possible reasons: · · Photogrammertic processing accuracy – – affected by poor signalization of control points · · Mechanical problems with the camera body/mount · · Image time tagging (rather unlikely)

64. Civil and Environmental Engineering and Geodetic Science Average differences between nominal and manual measurements of control point coordinates 3D coordinates were measured manually in four stereo models oriented by DPO, each one containing two to five control points

65. Civil and Environmental Engineering and Geodetic Science AIMS  Summary - Outlook   Promising test results   Automation and cost reduction offered by multi- sensor systems   Further refinements needed:   camera body rigidity   camera/mount/INS assembly needs improvements   boresighting should be performed at a special calibration range prior to the survey   high-resolution (9K  9K) digital camera   real-time operation capability