1. http://www.nshorter.com 1 Clustering Features to Recognize Structure for Purposes of 3D Reconstruction from LiDAR and other sources Nicholas Shorter Machine Learning Research Group Seminar Series Email: nshorter@cfl.rr.comnshorter@cfl.rr.com Website: http://www.nshorter.comhttp://www.nshorter.com

2. 2 Presentation Layout 1.LIDAR Overview 2.Problem Statement 3.Applications 4.Existing Approaches 5.Data Features 6.Masters Research 7.Masters Research Results

3. http://www.nshorter.com3 LIDAR Overview Data Collection –Plane Equipped with GPS, INS & LIDAR –LIDAR – Light Detection and Ranging (active sensor) Collection of 3D points Laser sent out from Emitter, reflects off of Terrain, Returns to Receiver –Receiver measures back scattered electromagnetic radiation (laser intensity) Time Difference Determines Range to Target http://www.toposys.com/

4. http://www.nshorter.com4 LIDAR Captured Characteristics Range to Target (elevation from INS & LiDAR) Longitude and Latitude (GPS) First and Last Return Pulses –First – shrubbery, vegetation, power lines, birds and buildings –Last – buildings (unless vegetation is really dense, then vegetation too) Returned Laser Intensity

5. http://www.nshorter.com5 LiDAR Noise Essential to understand forms of noise corrupting data in order to realize how to deal with the noise Geolocation results from LiDAR, GPS and INS sensor systems –Accuracy Limitations –Offset and Drift in both GPS and INS –Misalignment between INS and LiDAR Atmosphere – Intensity and Path Distortion Shadowing Effect from Tall buildings Artifacts from non uniform sampling from multiple strips

6. http://www.nshorter.com6 Problem Statement Input –LIDAR (Light Detection and Ranging) Data Collection of Irregularly Distributed 3-Dimensional Points –Aerial Photograph Output –Semi to complete Automatic (minimal user intervention) development of 3-Dimensional Virtual Model

7. http://www.nshorter.com7 Applications for 3D Reconstruction Military Applications –Automatic Target Recognition –Reconstructed Models of Opponent Terrain (UAV?) Tourism/Entertainment –Virtual Walkthrough of Theme Park Commercial –Change Detection (Natural Disasters) –Network Planning for Mobile Communication –Noise Nuisance (Universal Studios, 408 Expressway - walls) –Urban Planning

8. http://www.nshorter.com8 Existing 3D Reconstruction Research Data Sources –LIDAR –Aerial Imagery –GIS Ground Plans Model Based Reconstruction –Pre-defined models with parameters –Minimize error between models and data Data Driven –Group Coplanar Pts –Identify Break Lines –Derive Model to Minimize Error

9. http://www.nshorter.com9 Triangulation Based Methods Triangular Irregular Network (TIN) –Series of Non-Overlapping Triangles Modeling given Surface TIN 3D Reconstruction Methods –Clustering approach Spherical Normal Vectors of Triangles –TIN region growing approach Merge Triangles to Same Region if Normal Vectors within Threshold

10. http://www.nshorter.com10 Triangulated Irregular Networks (TIN) Advantages of TIN –Generation Fully Automatic –Space Uniquely Defined –Cells Indexed Spatial Searches, Triangle Elevation Differences Disadvantages of TIN –Noise Causes Triangle Normal Vectors to deviate from Ideal

11. http://www.nshorter.com11 Existing 3D Reconstruction Methods Most still under development Most Methods Use ‘Grided’ (Interpolated) LIDAR Data –Advantages Less Computationally Complex DTM & DSM Thresholding to distinguish Building from Non Building Use of additional conventional methods –Disadvantages Decrease in Accuracy Uncertainty from Building and Ground Interpolation

12. http://www.nshorter.com12 Building Detection Features Investigate Returned Laser Intensity Vegetation Points –Significant first and last return elevation diff. –Corresponding green aerial image color –Nearest points have diff. elev. or adjacent triangles have significantly diff. norm vectors Common Building Points –Spatially close in terms of long. and lat. –Bounded by aerial img. edges and exterior wall tri. –Bounded building does not contain terrain pts – or - –Triangulation of all points, building points connected

13. http://www.nshorter.com13 Building Reconstruct Cluster Features Surface change - normal vector orientation difference between adjacent triangles (X,Y,Z) - Longitude, Latitude, Elevation Edge - Aerial imagery edge detection Color - Aerial imagery corresponding color (building surface differs from clutter) Triangle planar coef, pt. height diff., same normal vector, or planar equation Feedback – difference between reconstruction and raw LiDAR and Aerial Image Edges

14. http://www.nshorter.com14 Masters Research Greedy Insertion Triangulation –Implemented Noise Filtering Technique Proposed FSART normal vector clustering Proposed Planar Regression to combat Category Proliferation Realized simple planar reconstruction algorithm (MSEE Thesis)

15. http://www.nshorter.com15 Fuzzy SART Clustering Fuzzy Simplified Adaptive Resonance Theory (SART) Advantages – –Not as Sensitive to Input Order (CG FA) –No “Mandatory” Preprocessing Techniques –Activation Function Forms Spherical Arcs –Activation Function is Measure not Estimate –Long Term Weights have Intuitive Meanings –Only 2 User Defined Parameters with Clear Meanings Tau = Necessary Time to Learn Pattern VDMT = Vigilance Parameter

16. http://www.nshorter.com16 Fuzzy SART Activation Function Activation Function Module Degree of Match Angle Degree of Match Pattern Encoding Shape

17. http://www.nshorter.com17 Fuzzy SART Preprocessing Scaled Each Dimension to Same Max Value –Each Dimension Receives Equal Weight Translated Input Dimension Space to Center at Origin

18. http://www.nshorter.com18 Planar Regression Equation of a Plane: – Solving for Z: – Substitutions for Coefficients: – Use Method of Least Squares and Choose Estimates for Coefficients to minimize SSE: –

19. http://www.nshorter.com19 Planar Regression Continued… Derivative with Respect to Each Coefficient – System of Equations: –

20. http://www.nshorter.com20 Planar Regression Pictorial VDMT biased against absorbing points into an existing cluster (VDMT = 0.7) Planar Regression implemented to join like clusters below a perpendicular threshold

21. http://www.nshorter.com21 Data Set Specifications Donated by Simone Clode & Franz Rottensteiner Collected by AAMHatch LIDAR - Approximately 1 point per 1.3m 2 point density (need at least 1 pt. per 1.5m 2 for 3D rec.) –First & Last Return Pulses & Laser Intensity Recorded Aerial Photography – 15 cm pixel resolution Data depicts urban and residential area of Fairfield, Australia

22. http://www.nshorter.com22 Results 4 Actual Building Test Cases Presented –Aerial Photographs Included for Visual Comparison –Outputs for Various Stages of Algorithm presented with different levels of optimization

23. http://www.nshorter.com23 Aerial Photographs

24. http://www.nshorter.com24 3D Scatter Plots

25. http://www.nshorter.com25 Result Set 1 No Scaling, Shifting, Filtering nor Planar Regression

26. http://www.nshorter.com26 Result Set 2 Scaling, Shifting, No Filtering Nor Planar Regression

27. http://www.nshorter.com27 Result Set 3 Scaling, Shifting, Filtering, No Planar Regression

28. http://www.nshorter.com28 Result Set 4 – Roof Segmentation Scaling, Shifting, Filtering, Planar Regression

29. http://www.nshorter.com29 Result Set 5 - Final

30. http://www.nshorter.com30 Conclusions – Fuzzy SART Preprocessing Fuzzy SART Preprocessing Techniques Proposed –Scaling Dimensions to same Maximum giving each dimension equal weight –Translating Input Space to Further Separate Data –Above Measures Improved Clustering

31. http://www.nshorter.com31 Conclusions – Heuristic Processing Triangle Differences for Determining Roof Triangles Planar Regression for Triangulation Clustering Proposed –Results Show Improvement Using Method

32. http://www.nshorter.com32 Conclusions ART TIN Based Clustering Showed Promising Results for Commercial Buildings Like Other Methods, Had Difficulties with House Roof Segmentation –Houses fractions of a size of Buildings –Houses had as much as 5 times Roof Planes as Buildings

33. http://www.nshorter.com33 Acknowledgements Data Contributors –Dr. Simone Clode, Dr. Franz Rottensteiner, AAMHatch –Mr. John Ellis, AeroMap –Mr. Steffen Firchau, TopoSys –Mr. Paul Mrstik, Terra Point Advisor (PhD Committee Chair) –Dr. Takis Kasparis PhD Committee Members –Dr. Michael Georgiopoulos, Dr. Georgios Anagnostopoulos, Dr. Andy Lee, and Dr. Wasfy Mikhael