1. 3D Reconstruction Using UAVs
Binocular Stereo
Structure From Motion
Introduction to Drones
Chia-Hung Yeh
葉家宏
Dept. of E.E.
National Sun Yat-sen University
Chia-Yen Chen
陳佳妍
Dept. of C.S. and I.E.
National University of Kaohsiung
Making 3D Models with A Drone

2. Binocular Stereo

3. Binocular Stereo Similar to how human eyes perceive depth
Acquire a pair of images using two cameras separated by a known base distance, or a single moving camera

4. Binocular Stereo
Finds matching points and calculates depth values from disparities
Conjugate pair is two points in different images that are the projections of he same point in the scene
Disparity is the distance between points of a conjugate pair when the two images are superimposed

5. Binocular Stereo P2 Scene object points P1 Blue + Red lines
Conjugate pair
Disparity for P1
Baseline distance b

6. Binocular Stereo P2 Scene object points P1 Disparity for P2
Baseline distance b

7. Binocular Stereo Larger disparity => Smaller distance
Disparity for P2
Disparity for P1
Disparity for P1 > Disparity for P2
Means P1 is closer than P2

8. Binocular Stereo f: focal length P1 (x,y,z) z xR xL f Disparity for P1
Baseline distance b

9. Binocular Stereo
Accuracy of depth computation can be enhanced by increasing the baseline distance, b
However, the scene seen by both cameras decreases

10. Binocular Stereo
Stereo analysis
Left image
Right image

11. Binocular Stereo Depth map Gray values indicate distances from camera
Depth with texture map

12. Binocular Stereo
What are the relative distances of the blocks?

13. Binocular Stereo
What are the relative distances of the blocks?

14. Binocular Stereo Input images Output disparity map or depth map Left
Right

15. Binocular Stereo Early satellite images
Binocular images of the pentagon

16. Binocular Stereo orthogonal map
Prof. Gimelfarb at CITR :

17. Binocular Stereo What are the limits of binocular stereo? Occlusion
Some scene points are obstructed in an image
Cannot find the conjugate pair
Resolution
Disparity values are integers
Higher accuracy achieved at cost of smaller overlapping regions

18. Binocular Stereo Consider this P2
Note the order of the points’ projections on the imaging plane P1

19. Stereo Matching Correspondence problem
Detection of conjugate pairs in stereo images
Still a challenging problem
For each point in the left image, find the corresponding point in the right image
Use similarity to determine correspondence

20. Structure from Motion

21. Structure from Motion Extension from binocular stereo
Match corresponding features and measure distances between features on the camera image plane
The Scale Invariant Feature Transform is key to matching corresponding features despite varying distances
Scale Invariant Feature Transform
SIFT (Lowe, 1999) allows corresponding features to be matched even with large variations in scale and viewpoint and under conditions of partial occlusion and changing illumination

22. Structure from Motion Given: 𝑚 images of 𝑛 fixed 3D points
𝒙 𝑖𝑗 = 𝑷 𝑖 𝑿 𝑖 , 𝑖=1,…, 𝑚, 𝑗=1,…,𝑛
Problem: estimate 𝑚 projection matrices 𝑷 𝑖 and 𝑛 3D points 𝑿 𝑖 from the 𝑚𝑛 correspondences 𝒙 𝑖𝑗

23. Structure from Motion
When we have the matching locations of multiple points on two or more photos, there is usually just one mathematical solution for where the photos were taken.
Therefore, we can calculate individual camera positions, orientations, focal lengths and relative positions of corresponding features in a single step known as “bundle adjustment”.
This is where the term Structure from Motion comes from. Scene structure refers to all these parameters; motion refers to movement of the camera

24. Structure from Motion
Next, a dense point cloud and 3D surface is determined using the known camera parameters and using the SfM points as “ground control”.
All pixels in all images are used so the dense model is similar in resolution to the raw photographs. This step is called “multiview stereo matching” (MVS)
Photographs from a range of angles and distances can be used, with no a priori knowledge of locations or orientations
Enables “unstructured” image acquisition from the ground, from legacy air-photosets, or from cheap, unmanned aerial platforms

25. Introduction to Drones

26. What are Drones?
Drones are more formally known as unmanned aerial vehicles (UAV, 無人航空載具).
Drones can be remotely controlled or can fly autonomously through software-controlled flight plans.
Drones are often associated with military but they are also used for many non-military applications, such as search and rescue, surveillance, weather monitoring, etc.
Drones can be as large as a full-sized plane or small enough to fit in the palm of your hand. By one estimate, more than 1500 different types of drones are manufactured today for the military, commercial and civilian markets.
靶機（Target Drone）是作為空中靶標的無人駕駛飛機。一般而言，靶機是用來鑑定地對空飛彈、空對空飛彈、航空機砲和高射砲等武器的效能，也可提供地對空飛彈、戰鬥機和高射砲練習射擊使用。靶機的種類很多，有的以屆退役的飛機和老舊飛彈改裝而成，有的是專門為某一特殊用途所設計、製造。
MQ-9 Reaper(MQ-9死神偵察機)是一款由通用原子航空系統為美國空軍所開發的無人機。MQ-9是由MQ-1掠奪者式改進而成的進階機種，但尺碼更大、載重更重，具有長滯空時程、高海拔監視的能力，有別於前代機種原本是開發作為偵察用途但轉用於主動攻擊目標，MQ-9是第一款專門設計作為攻擊用途的無人機。
Off-the-shelf: 現成的

27. The Evolution of Drones
1935
Reginald Denny mass produced radio-controlled target drones. He manufactured nearly fifteen thousand drones for the US Army during the World War II.
1994
The MQ-9 Reaper is the first attack drone designed for long-endurance, high-altitude surveillance.
2013
The DJI Phantom is the first quadcopter that requires no DIY assembly and is sold at an affordable price.
1960
The Ryan Firebee was one of the first jet-propelled drones, it was used for reconnaissance missions during the Vietnam War.
Early 1990s
Drones innovations are originally focused on target practice to the military.
The Austrian balloons
The earliest recorded use of an unmanned aerial vehicle for warfighting occurred on August 22, 1849, when the Austrians attacked the Italian city of Venice with unmanned balloons loaded with explosives.
World War I
The first pilotless aircraft were built during and shortly after World War I. Leading the way, using A. M. Low's radio control techniques, was the Ruston Proctor Aerial Target of 1916.
早期無人機多為軍事用途
隨著電子陀螺儀、GPS 、遙控無線電和飛行控制系統的發展，使無人飛行載具成長迅速。
Drones can be as large as a full-sized plane or small enough to fit in the palm of your hand. By one estimate, more than 1500 different types of drones are manufactured today for the military, commercial and civilian markets.
靶機（Target Drone）是作為空中靶標的無人駕駛飛機。一般而言，靶機是用來鑑定地對空飛彈、空對空飛彈、航空機砲和高射砲等武器的效能，也可提供地對空飛彈、戰鬥機和高射砲練習射擊使用。靶機的種類很多，有的以屆退役的飛機和老舊飛彈改裝而成，有的是專門為某一特殊用途所設計、製造。
MQ-9 Reaper(MQ-9死神偵察機)是一款由通用原子航空系統為美國空軍所開發的無人機。MQ-9是由MQ-1掠奪者式改進而成的進階機種，但尺碼更大、載重更重，具有長滯空時程、高海拔監視的能力，有別於前代機種原本是開發作為偵察用途但轉用於主動攻擊目標，MQ-9是第一款專門設計作為攻擊用途的無人機。
Off-the-shelf: 現成的

28. Why are Non-military Drones so Popular?
Before drones came along, the only way to get aerial footage was to hire a helicopter. With drones, anyone can use it to capture stunning aerial footage.
Aerial Footage is Breathtaking
There are no limitations on who can buy and fly a drone. Drones continue to become more affordable, accessible and easier to use.
Drones are Easy to Acquire
They are like toys for adults, which can go thousands of feet into the air with a camera .
Drones are Just So Cool
We can get creative with the use of our drones.
There are many cool things you can do with drones.

29. What can we do with Drones?
Journalism, Filming and Photography
To film sport events like skiing and snowboarding.
For film makers to capture innovative shots.
To be used in real estate market.
Search and Rescue
Efficiently and safely locate victim
Drop supply
3D Reconstruction
3D scanning of large buildings and landmarks.
Mapping difficult-to-access locations like eroded coastline or mountain tops.
Each of our aircraft have the capability of “looking down” with high resolution video cameras to survey and search an area. Via on-board video recorder or FPV (first person view) technology, we can survey extensive terrain in real time. (Imagine flying INSIDE the cockpit). Once a subject of interest is located, the exact location can be relayed back to the ground crew. Onboard video can also be downloaded for post-flight inspection. Average flight times for these all electric drones is usually between minutes giving the pilot enough time with several batteries to scan an area miles in diameter using a zig-zag sweep pattern or expanding spiral search pattern. One drone can search an extensive area in minutes that might take an entire group of ground personnel to cover in hours.
Besides locating victims, a multi-rotor could potentially be used to “drop in supplies” to an otherwise unreachable location. For example a drone might be utilized to lower a walkie talkie, gps locater, med supplies or water to a stranded victim before rescue crews are able to extract them.

30. What can we do with Drones?
Agriculture
Monitor farms
Collect crop data
Manage crop and livestock
Conservation
Track endangered species
Environmental monitoring
Monitor illegal dumping.
Shipping/Delivery
Drones can be used to deliver small packages, pizzas, medicines, beverages in short distances.
Emergency call for distribution of medication and aid.
By Daniel Kelly on November 5, 2014
Unmanned aerial drones have a lot of uses in environmental monitoring, from aiding conservationists to keeping track of agriculture. But the National Oceanic and Atmospheric Administration is using them for another purpose, according to TreeHugger: whale tracking.
Drones are a good tool for the effort because they can circle high above whales and keep out of earshot. The high-altitude approach is good for taking snapshots of whale girth, a marker of a healthy diet, without disturbing them.
Researchers with NOAA have so far used drones to successfully study killer whales off the coast of Canada. Using a custom-built hexacopter, they’ve captured 30,000 images over just 60 flights. Scientists say the technology will help them identify unhealthy whale conditions earlier, as well as aid in forming future conservation plans.
Top image: An orca jumping out of the water near East Poiont, Saturna Island. (Credit: Miles ritter, via Flickr/CC BY 2.0)

31. 可穩定停懸於空中，操控靈活，適合精細量測
Types of Drone 類型
定翼型 (Fixed-Wing)
旋翼型 (Multi-Rotor)
示意圖
滯空時間
可達60分鐘
約10~25分鐘
飛行範圍 大 小
起飛場地
需跑道供起飛降落
小型平台即可垂直起降
飛行高度
約100m
可達500m
載具尺寸
翼展1400mm
軸距350mm 特點
因浮力好，飛行時間長，適合大範圍量測
可穩定停懸於空中，操控靈活，適合精細量測
定翼型即常見的飛機，機身有一對固定的機翼，使其產生浮力滑翔於空中
旋翼型以直升機為代表，機身有至少一組得主旋翼向下產生推力，使機體能滯留於空中

32. Pros and Cons of Different Types of Drones
Typical Uses
Price ($AUD)
Multi-Rotor
Accessibility
Ease of use
VTOL and hover flight
Good camera control
Can operate in a confined area
Short flight times
Small payload capacity
Aerial Photography and Video Aerial Inspection
$5k-$65k for pro drones
Fixed-Wing
Long endurance
Large area coverage
Fast flight speed
Launch and recovery needs a lot of space
No VTOL/hover
Harder to fly, more training needed
Expensive
Aerial Mapping, Pipeline and Power line inspection
$25-$120k for pro drones
Single-Rotor
Long endurance (with gas power)
Heavier payload capability
More dangerous
Aerial LIDAR laser scanning
Fixed-Wing Hybrid
VTOL and long-endurance flight
Not perfect at either hovering or forward flight
Still in development
Drone Delivery
TBD, in development
(Vertical Take-Off and Landing)
This article appears in the Australian DRONE magazine, issue 3 (June 2016). It is a great read for hobbyists and professionals alike, with both fun and informative articles, and all the latest gear — check it out:

33. Potentials and Outlooks
Perceptual smart machine age includes more familiar technologies such as autonomous vehicles, commercial UAVs and smart dust and will be the most disruptive class of technologies over the next 10 years.

34. Cameras on Drones At first, no cameras Attach small cameras, GoPros
Built-on cameras, with 4K resolution
Drones able to carry SLR cameras

35. Cameras on Drones
Live HD video feed direct from the camera 30fps
Full camera control
Video stored on SD card
UHD: 4096x2160p 24/25, 3840x2160p 24/25/30
FHD: 1920x1080p 24/25/30/48/50/60
HD: 1280x720p 24/25/30/48/50/60

36. GIS Drone Drone acquisition & generation of raster data
Flight planning
Flight(s)
Image download & processing
Generation of orthophotos / digital surface models
Extraction of vector data
GPS recording
Geodatabases
Manual data collection
Digitisation
Web services
Drone point cloud
Import & layering of vector & raster data
GIS software
Analysis & decision making
Progress tracking, feature identification, flood simulation, project planning etc.

37. GIS Drone
Skycatch releases mobile apps built on the DJI SDK, COMMANDER iOS, compatible with the DJI Phantom 3. The app creates high-resolution maps and 3D models in just a few taps.

38. Exercise for the Next Class
Find recent applications or technologies that use UAV
Less than 3 years old
Article from reliable source
Excluding the examples that I have given
Prepare minutes presentation on the selected topic
What kind of UAV used?
Advantages over conventional methods
Difficulties encountered
Future aspects
Please include your references