1. MASKS © 2004 Invitation to 3D vision Lecture 11 Vision-based Landing of an Unmanned Air Vehicle

2. MASKS © 2004 Invitation to 3D vision Applications of Vision-based Control Fire Scout Global Hawk Predator SR/71 UCAV X-45

3. MASKS © 2004 Invitation to 3D vision Goal: Autonomous landing on a ship deck Challenges Hostile environments Ground effect Pitching deck High winds, etc Why vision? Passive sensor Observes relative motion

4. MASKS © 2004 Invitation to 3D vision Simulation: Vision in the loop

5. MASKS © 2004 Invitation to 3D vision Vision-Based Landing of a UAV Motion estimation algorithms Linear, nonlinear, multiple-view Error: 5cm translation, 4° rotation Real-time vision system Customized software Off-the-shelf hardware Vision in Control Loop Landing on stationary deck Tracking of pitching deck

6. MASKS © 2004 Invitation to 3D vision Vision-based Motion Estimation Pinhole Camera Landing target Image plane Feature Points Current pose

7. MASKS © 2004 Invitation to 3D vision Pose Estimation: Linear Optimization Pinhole Camera: Epipolar Constraint: Planar constraint: More than 4 feature points Solve linearly for Project onto to recover

8. MASKS © 2004 Invitation to 3D vision Pose Estimation: Nonlinear Refinement Objective: minimize error Parameterize rotation by Euler angles Minimize by Newton-Raphson iteration Initialize with linear algorithm

9. MASKS © 2004 Invitation to 3D vision Multiple-View Motion Estimation Multiple View Matrix Rank deficiency constraint Pinhole Camera

10. MASKS © 2004 Invitation to 3D vision Multiple-View Motion Estimation n points in m views Equivalent to finding s.t. Initialize with two-view linear solution Least squared solution: Use to linearly solve for Iterate until converge

11. MASKS © 2004 Invitation to 3D vision Real-time Vision System Ampro embedded Little Board PC Pentium 233MHz running LINUX 440 MB flashdisk HD robust to vibration Runs motion estimation algorithm Controls Pan/Tilt/Zoom camera Motion estimation algorithms Written and optimized in C++ using LAPACK Estimate relative position and orientation at 30 Hz UAVPan/Tilt CameraOnboard Computer

12. MASKS © 2004 Invitation to 3D vision Hardware Configuration On-board UAV Vision System Vision Computer RS232 Vision Algorithm Frame Grabber Camera WaveLAN to Ground Navigation System Navigation Computer RS232 Control & Navigation INS/GPS WaveLAN to Ground

13. MASKS © 2004 Invitation to 3D vision Feature Extraction Acquire Image Threshold Histogram Segmentation Target Detection Corner Detection Correspondence

14. MASKS © 2004 Invitation to 3D vision Pan/Tilt to keep features in image center Prevent features from leaving field of view Increased Field of View Increased range of motion of UAV Camera Control

15. MASKS © 2004 Invitation to 3D vision Ground Station Comparing Vision with INS/GPS

16. MASKS © 2004 Invitation to 3D vision Motion Estimation in Real Flight Tests

17. MASKS © 2004 Invitation to 3D vision Landing on Stationary Target

18. MASKS © 2004 Invitation to 3D vision Tracking Pitching Target

19. MASKS © 2004 Invitation to 3D vision Conclusions Contributions Vision-based motion estimation (5cm accuracy) Real-time vision system in control loop Demonstrated proof of concept prototype: first vision-based UAV landing Extensions Dynamic vision: Filtering motion estimates Symmetry-based motion estimation Fixed-wing UAVs: Vision-based landing on runways Modeling and prediction of ship deck motion Landing gear that grabs ship deck Unstructured environments: Recognizing good landing spots (grassy field, roof top etc)