1. TKK 12.4.2007 STRS system combining LiDAR and multiple images: Multi-scale template matching and LS- adjustment of a parametric crown model with lidar data in 3D tree top positioning and estimation of the crown shape Species recognition in aerial images Ilkka Korpela (Morten Larsen)

2. Contents (Demos included?) Single-tree remote sensing (STRS) Photogrammetric 3D reconstruction and object recognition Airborne lidar-based 3D reconstruction and object reconstruction Coupling allometric constraints to the STRS problems 3D treetop positioning with template matching (TM) “ multi-scale TM Species recognition LS-adjustment of crown models with lidar points Conclusions and outlook

3. STRS Single-Tree Remote Sensing Air- or spaceborne; active and/or passive sensing 2D or 3D = with or without tree height “Direct estimation” of tree/crown position and species; indirect model-based estimation of height, dbh Restrictions: tree discernibility due to scale (detectable object size), occlusion and shading. Alternative or complement to A) field inventory, B) area-based remote sensing

4. STRS Single-Tree Remote Sensing Accuracy restricted by “allometric noise” alike in volume functions → tree and stand level bias, tree level impresicion. dbh ~ 10-12 %. Measurements subject to bias: DTM-errors, lidar does not hit apexes, Dcr underestimation Nothing can be known about quality, only quantity Unsolved issues: species recognition, regeneration stands, calibration and validation in the field, aggregated crowns result in fused trees.

5. Photogrammetric 3D reconstruction and object recognition Usually from N>1 images (multiscopic) Correspondence problem - ill-posed, perspective errors, reflectance, occlusion, scene complexity Texture in the image functions needed Relies on accurate geometry (camera interior, exterior), ray- intersection

6. Photogrammetric 3D reconstruction and object recognition Digital revolution (2000→) Aerial images upto 2GB, manageable (I/O, analysis, storage, transfer) Automatic methods in orientation, incl. DSO with GPS/INS, reduction in GCPs. Digital cameras with MS images 2005→ multiple images per target, better radiometry and geometry Automatic DSM production

7. Photogrammetric 3D reconstruction and object recognition Automation Laborious orientation tasks ± solved DSMs, DTMs using image matching e.g. Building extraction - semiautomatic 3913, 3914, 3915 (triplet matching) → 1946, 1962 Demo 1

8. Photogrammetric 3D reconstruction and object recognition Photogrammetric STRS scene and object complexity occlusion & shading scale: h = 0..40 m, Dcr 0..10 m “BDRF”-effects → automation challenging

9. Photogrammetric 3D reconstruction and object recognition Demo – manual STRS - NLS 04403 (19) treetop 3D, height, Dcr, Sp dbh = f(Sp, h, Dcr) + epsilon Image matching fails for treetop positioning unless we use a feature detector for treetops.

10. Airborne lidar-based 3D reconstruction and object reconstruction A pulse of short duration (~ 3 m, 1064 nm) Observing returned signal. Discrete data. Upto 128 samples?. Signal is reconstructed into points or samples for later waveform analysis. Intensity of the return/echo. Pros: No texture needed, active → no shading, “real ease of 3D” Cos: discrete sampling, high sampling rates costly, difficult to reconstruct “high- frequency relief”.

11. Airborne lidar-based 3D reconstruction and object reconstruction Automation DTMs – manual assistance needed – high accuracy – even under canopy Volume estimation of trees – “automatic” – e.g. using regression between lidar features and field observations (e.g. Naesset 2004. Suvanto et al. 2005) Lidar and STRS Algorithms that process point clouds directly or interpolated DSMs (CHMs) Underestimation of heights (footprint size, density, crown shape, equipment sensitivity) Species not obtained (so far)

12. Coupling allometric constraints to the STRS tasks If we know dbh, we have an idea of the height If we know dbh, species and age, we have better idea of height If we know dbh, species, and height we have a good idea of volume If we know dbh, species, height and height of crown, we have a better idea of volume. If we know height, species and crown width we can estimate dbh and volume If we know species we have an idea of the shape of the “crown envelope”. If we know species and height, can we set limits for the variation of crown width f(sp, h) => [Min, Max] of Dcrm, and assume a basic shape?

13. Coupling allometric constraints to the STRS tasks If we know species can we have an idea of the shape of the “crown envelope”?  Timo Melkas

14. Dcrm (min,max) / Shape | (Sp, height) → - Consistency of measurements (rule out impossible observations) - Initial approximations for iterative approaches in finding true Dcrm & crown shape E.g Short trees have small crowns (adjust search space accordingly, or look for small crowns from a low height) Coupling allometric constraints to the STRS tasks

15. 3D treetop positioning with template matching (TM) Demo 04402 - 3D treetop positioning using TM 1) Use, for each of the N views, a model image (template) of a crown. 2) Compute N normalized cross- correlation images (template matching). 3) Form a Cartesian 3D grid in the canopy – in the search space. 4) Aggregate 3D correlation to the grid points. 5) Process the 3D correlation into “hot-spots” – 3D treetop positions. Fine, but not invariant to object variation.

19. Cartesian grid / Search space in the upper canopy

20. Correlation > threshold

22. Multi-scale TM – Treetop positioning Can we assume that the optical properties and the “relative shape” of trees are invariant to their size? I.e. small trees appear as scaled versions of large trees in the images? (inside one species and within a restricted area)

23. Multi-scale TM – Treetop positioning Maxima at different scales, take global → (X,Y,Z)

24. Multi-scale TM – Crown size Demo 04403_19, 18, 20, 06214_3900, 3901 Near-nadir views are best for manual measurement of Dcr (crown width)

25. Species recognition Spectral values Texture Variation: - Phenology - Tree age and vigour - Image-object-sun geometry => reliable automation problematic => bottleneck

26. LS-adjustment of a crown model with lidar points Assume that 1) Photogrammetric Multi-scale TM 3D treetop position is highly accurate 2) Trees have only moderate slant 3) Crowns are ± rotation symmetric 4) We know tree height and species which give a reasonable approximaion of the crown size and shape → LiDAR hits are “observations of crown radius at a certain height below the apex” Assume a rather large crown and collect lidar hits in the visinity of the 3D treetop position, down to relative height of apprx. 60 %. Use LS-adjustment to find best set of parameters for the crown model.

27. “LiDAR hits are observations of crown radius at a certain height below the apex?”

28. LiDAR hits are observations of crown radius at a certain height below the apex – what if crowns are interlaced?”

29. Example - a 19-m high spruce: Solution in three iterations. Final RMSE 0.31 m Note apex! LiDAR did not hit the apex and the “crown width at treetop” (constant term) is negative.

30. Example - a 22-m high birch: Solution in six iterations. Final RMSE 0.47 m For some reason RMSEs are larger for birch in comparison to pine and spruce. Convergence?

31. Conclusions and outlook A - Multi-scale TM works in a manual  semi-automatic way for treetop positioning Possible to automate? Computation costs? (NCC now tried everywhere) - Multi-scale TM in crown width estimation needs comprehensive testing (Image scales, required overlaps) -Species recognition was “overlooked” here, still I think that good 3D treetop positions can be used for the purpose. - Matching LiDAR points using LS-adjustment works only if the exact treetop position is known. Aggregated crowns are problematic, but these cases are known from the tree map

32. Conclusions and outlook B - If we have a system that can be operated so that a tree measurement takes 2-3 seconds and the measurement inaccuracies are: h ~ 0.6 m Dcr ~ 10% d13 ~ 10-12 % XY-position ~ 0.3 m Sp ~ 95% Is this fast and accurate enough for sample-plot based STRS? Can we afford the images and LiDAR?

33. Calibration and validation of results?

34. Presenting method and early results in ISPRS workshop in Hannover, June 1, 2007 Presentation at Silvalaser, Espoo September 2007? Article about Multi-Scale TM in 3D treetop positioning (ISPRS JPRS?) Article about Multi-Scale TM combined with crown modeling using LS-adjustment and allometric constraints (Silva Fennica 100-yr issue?) Species recognition study remains of the Academy 3-yr resrach project to be completed before VIII/2008