IGARSS 2011, Vancouver, Canada Stabilization and Georegistration of Aerial Video Over Mountain Terrain by Means of LIDAR IGARSS 2011, Vancouver, Canada July 24-29, 2011 Mark Pritt, PhD Lockheed Martin Gaithersburg, Maryland mark.pritt@lmco.com Kevin LaTourette Lockheed Martin Goodyear, Arizona kevin.j.latourette@lmco.com

Problem: Georegistration Georegistration is the assignment of 3-D geographic coordinates to the pixels of an image. It is required for many geospatial applications: Fusion of imagery with other sensor data Alignment of imagery with GIS and map graphics Accurate 3-D geolocation Inaccurate georegistration can be a major problem: Misaligned GIS Correctly aligned

Solution Our solution is image registration to a high-resolution digital elevation model (DEM): A DEM post spacing of 1 or 2 meters yields good results. It also works with 10-meter post spacing. Works with terrain data derived from many sources: LIDAR: BuckEye, ALIRT, Commercial Stereo Photogrammetry: Socet Set® DSM SAR: Stereo and Interferometry USGS DEMs

Methods Create predicted images from the DEM, illumination conditions, sensor model estimates and actual images. Register the images while refining the sensor model. Iterate. Aerial Video Sensor Image Plane Scene Occlusion Illumination Shadow Predicted Images

Methods (cont) Predicted Image from DEM Predicted Image from Aerial Image Registration Tie Point Detections The algorithm identifies tie points between the predicted and the actual images by means of NCC (normalized cross correlation) with RANSAC outlier removal.

Methods (cont) The algorithm uses the refined sensor model as the initial guess for the next video frame: The refined sensor model enables georegistration. Exterior orientation: Platform position and rotation angles Interior orientation: Focal length, pixel aspect ratio, principal point and radial distortion Initial Camera Estimate camera model Use camera focal length & platform GPS if avail. Register Predict images from DEM and camera Register images with NCC Refine Compose registration fcn & camera LS fit for better cam estimate Iterate Next Frame Register to previous frame Compose with cam of prev. frame for init. cam estimate Iterate for each video frame Finish Trajectory Propagate geo data from DEM Resample images for orthomosaic

Example 1: Aerial Motion Imagery Inputs: Aerial Motion Imagery over Arizona, U.S. 1/3 Arc-second USGS DEM Area: 64 km2 Post Spacing: 10 m 16 Mpix, 3.3 fps, panchromatic

Problem: Too shaky to find moving objects Example 1 (cont) Problem: Too shaky to find moving objects Zoomed to full resolution (1 m)

Example 1: Results Outputs: Sensor camera models Images georegistered to DEM Platform trajectory

Example 1 Results (cont) ATV Vehicle Human Pickup Truck Video is now stabilized, and as a result, moving objects are easily detected.

Example 2: Oblique Motion Imagery Inputs: Oblique Motion Imagery Over Arizona, U.S. LIDAR DEM Area: 24 km2 Post Spacing: 1 m 16 Mpix, 3.4 fps, pan

Stabilized Video Inset Example 2: Results Map coordinates Stabilized Video Inset Orthorectified Video Background LIDAR DEM Aligned Map Graphics Target Tracking

Example 2 Results (cont) How fast does the algorithm converge? The initial error is high, but it decreases after only several iterations. Subsequent frames have better initial sensor model estimates and require only 2 iterations. IMAGE 1 Camera Iteration 1 2 3 Num tie points: 319 318 282 RMSE: 17.4 4.8 2.9 Mean Δx: 1.4 -0.7 0.1 Mean Δy: -3.8 -0.1 Sigma Δx: 15.8 4 2.5 Sigma Δy: 6 2.6 1.5 IMAGE 591 Camera Iteration 1 2 3 Num tie points 681 687 RMSE 2.7 0.6 0.3 Mean Δx Mean Δy 0.9 Sigma Δx 2.1 0.5 Sigma Δy 0.2 0.1

Aerial Video Over Arizona, U.S. Example 3: Aerial Video Inputs: LIDAR DEM Area: 24 km2 Post Spacing: 1 m Aerial Video Over Arizona, U.S. 720 x 480 Color 30 fps

Background Image Draped Over DEM Example 3: Results Map coordinates Orthorectified Video Background Image Draped Over DEM Aligned Map Graphics

Example 3 Results (cont) Map Graphics Stay Aligned with Features in Video

Example 4: Thermal Infrared Video Inputs: Commercial LIDAR DEM MWIR Video Over White Tank Mountains in Arizona Post Spacing: 2 m 1 Mpix, 3.3 fps

Video Mosaic Georegistered and Draped Over Mountains in Google Earth Example 4: Results Video Mosaic Georegistered and Draped Over Mountains in Google Earth BackgroundLIDAR DEM Video Mosaic Inset: Original Video with Map Graphics Overlay

Demo Click picture to play video

Conclusion We have introduced a new method for aerial video georegistration and stabilization. It registers images to high-resolution DEMs by: Generating predicted images from the DEM and sensor model; Registering these predicted images to the actual images; Correcting the sensor model estimates with the registration results. Processing speed is 1 sec per 16-Mpix image on a PC. Absolute geospatial accuracy is about 1-2 meters. We are developing a rigorous error propagation model to quantify the accuracy. Applications: Video stabilization and mosacs Cross-sensor registration Alignment with GIS map graphics