2. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Bayesian Multiple Target Tracking in Forward Scan Sonar Images Using The PHD Filter Daniel Clark and Judith Bell

3. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Thesis Topic: “Tomographic Reconstruction of a Sequence of Forward Scan Sonar Images”

4. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Thesis Topic: “Tomographic Reconstruction of a Sequence of Forward Scan Sonar Images” Problems to Address:

5. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Thesis Topic: “Tomographic Reconstruction of a Sequence of Forward Scan Sonar Images” Problems to Address: Segment Sonar into Homogeneous Regions of Same Seabed Type

6. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Thesis Topic: “Tomographic Reconstruction of a Sequence of Forward Scan Sonar Images” Problems to Address: Segment Sonar into Homogeneous Regions of Same Seabed Type Locate Objects on the Seabed

7. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Thesis Topic: “Tomographic Reconstruction of a Sequence of Forward Scan Sonar Images” Problems to Address: Segment Sonar into Homogeneous Regions of Same Seabed Type Locate Objects on the Seabed Align Images onto Global Co-ordinate System

8. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Thesis Topic: “Tomographic Reconstruction of a Sequence of Forward Scan Sonar Images” Problems to Address: Segment Sonar into Homogeneous Regions of Same Seabed Type Locate Objects on the Seabed Align Images onto Global Co-ordinate System Reconstruct Sonar Data into 3D Elevation Map of Seabed

9. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Thesis Topic: “Tomographic Reconstruction of a Sequence of Forward Scan Sonar Images” Problems to Address: Segment Sonar into Homogeneous Regions of Same Seabed Type Locate Objects on the Seabed Align Images onto Global Co-ordinate System Reconstruct Sonar Data into 3D Elevation Map of Seabed Why?

10. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Thesis Topic: “Tomographic Reconstruction of a Sequence of Forward Scan Sonar Images” Problems to Address: Segment Sonar into Homogeneous Regions of Same Seabed Type Locate Objects on the Seabed Align Images onto Global Co-ordinate System Reconstruct Sonar Data into 3D Elevation Map of Seabed Why? To Aid Navigation and Path Planning

11. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Thesis Topic: “Tomographic Reconstruction of a Sequence of Forward Scan Sonar Images” Problems to Address: Segment Sonar into Homogeneous Regions of Same Seabed Type Locate Objects on the Seabed Align Images onto Global Co-ordinate System Reconstruct Sonar Data into 3D Elevation Map of Seabed Why? To Aid Navigation and Path Planning Obstacle Avoidance, mines etc.

12. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Principle of Sonar: Transmission of an acoustic pulse of energy into water and measure reflected energy:

13. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Principle of Sonar: Transmission of an acoustic pulse of energy into water and measure reflected energy: The intensity of the energy reflected is measured against time to give information on the surface below: distance to surface = (speed of sound in water)x(time to return)/2

14. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Forward Scan Sonar: The acoustic energy from the sonar is transmitted in a radial sector.

15. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Forward Scan Sonar: The acoustic energy from the sonar is transmitted in a radial sector. The backscattered energy can be shown as a sonar image:

16. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Environmental Reconstruction Methods for creating elevation maps of the seabed have been implemented using Lambert's Law and knowledge of the Sonar's Position.

17. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Environmental Reconstruction Methods for creating elevation maps of the seabed have been implemented using Lambert's Law and knowledge of the Sonar's Position. Lambert's Law relates the reflected energy from the surface to the angle between the direction of reflection and the surface normal.

18. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Local Propagation Technique: Each successive point is determined from the intersection of the circle centred at the sonar fish and the surface gradient determined from Lambert's Law.

19. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Sonar Image:

20. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Sonar Image: Reconstructed Image:

21. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Multiple Target Tracking This work will be used to address the issues of: Detecting and Locating Objects on the Seabed Aligning Images onto a Global Co-ordinate System

22. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Multiple Target Tracking This work will be used to address the issues of: Detecting and Locating Objects on the Seabed Aligning Images onto a Global Co-ordinate System Targets to be Tracked: Mines, metallic objects which have high reflectance property: Measurements obtained by thresholding.

23. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Recursive Bayesian Estimation To make inference about a dynamic system, two models are needed: Motion Model – describes evolution of state with time ie the motion of underwater vehicle.

24. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Recursive Bayesian Estimation To make inference about a dynamic system, two models are needed: Motion Model – describes evolution of state with time ie the motion of underwater vehicle. Measurement Model – relates the measurements to the state ie the objects on the seabed.

25. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Recursive Bayesian Estimation To make inference about a dynamic system, two models are needed: Motion Model – describes evolution of state with time ie the motion of underwater vehicle. Measurement Model – relates the measurements to the state ie the objects on the seabed. These correspond to prediction and update stages when tracking.

26. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Single Target Inference The tracking problem is governed by two functions: These relate to the motion and measurement models respectively.

27. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Single Target Inference The tracking problem is governed by two functions: These relate to the motion and measurement models respectively. The process noise v reflects the unknown target motion The measurement noise n reflects sensor errors

28. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Bayesian Recursion The prior distribution of a target location based on previous observations is obtained from the motion model and the posterior at time t-1:

29. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Bayesian Recursion The prior distribution of a target location based on previous observations is obtained from the motion model and the posterior at time t-1: When a new measurement is obtained, the posterior distribution at time t is obtained by Bayes' Law: where g is the likelihood of observing z given target state x.

30. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Multiple Target Inference Model The single target recursive state estimation can be directly extended to a multiple target model using Finite Set Statistics (FISST).

31. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Multiple Target Inference Model The single target recursive state estimation can be directly extended to a multiple target model using Finite Set Statistics (FISST). A Random Finite Set (RFS) is used to represent a multiple-target state. The set of objects tracked at time t is an RFS containing : Set of objects survived from time t-1.

32. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Multiple Target Inference Model The single target recursive state estimation can be directly extended to a multiple target model using Finite Set Statistics (FISST). A Random Finite Set (RFS) is used to represent a multiple-target state. The set of objects tracked at time t is an RFS containing : Set of objects survived from time t-1. Set of object appearing at time t.

33. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Multiple Target Inference Model The single target recursive state estimation can be directly extended to a multiple target model using Finite Set Statistics (FISST). A Random Finite Set (RFS) is used to represent a multiple-target state. The set of objects tracked at time t is an RFS containing : Set of objects survived from time t-1. Set of object appearing at time t. The measurements at time t are modelled by an RFS containing: Measurements generated from actual targets.

34. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Multiple Target Inference Model The single target recursive state estimation can be directly extended to a multiple target model using Finite Set Statistics (FISST). A Random Finite Set (RFS) is used to represent a multiple-target state. The set of objects tracked at time t is an RFS containing : Set of objects survived from time t-1. Set of object appearing at time t. The measurements at time t are modelled by an RFS containing: Measurements generated from actual targets. Spurious measurements from clutter.

35. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Multiple Target Bayesian Recursion The single target Bayesian recursion can be extended to the multiple target scenario using the calculus defined in FISST:

36. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk PHD Filter The Probability Hypothesis Density or PHD is defined as the first order statistical moment of the multiple target posterior distribution. The integral of the PHD in any region represents the expected number of objects in that region.

37. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk PHD Filter The Probability Hypothesis Density or PHD is defined as the first order statistical moment of the multiple target posterior distribution. The integral of the PHD in any region represents the expected number of objects in that region. Why bother? Computationally cheaper to calculate than full posterior.

38. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk PHD Filter The Probability Hypothesis Density or PHD is defined as the first order statistical moment of the multiple target posterior distribution. The integral of the PHD in any region represents the expected number of objects in that region. Why bother? Computationally cheaper to calculate than full posterior. Where are the targets? The locations of the targets can be found as peaks of this distribution.

39. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk PHD Filter Prediction Equation:

40. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk PHD Filter Prediction Equation: Data Update Equation:

41. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Particle Filter Algorithm Particle filters were designed for implementing Bayesian recursion by representing probability distributions by random samples or particles rather than in their functional forms.

42. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Particle Filter Algorithm Particle filters were designed for implementing Bayesian recursion by representing probability distributions by random samples or particles rather than in their functional forms. The areas with higher probability will have a larger number of particles.

43. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Particle Filter Algorithm (Single Target) Step 1: Initialisation. N particles are uniformly distributed across the Field of View (FoV).

44. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Particle Filter Algorithm (Single Target) Step 1: Initialisation. N particles are uniformly distributed across the Field of View (FoV). Step 2: Data Update. After a new measurement, weights are assigned to the particles according to their likelihoods.

45. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Particle Filter Algorithm (Single Target) Step 1: Initialisation. N particles are uniformly distributed across the Field of View (FoV). Step 2: Data Update. After a new measurement, weights are assigned to the particles according to their likelihoods. Step 3: Resampling An unweighted particle set is obtained by resampling from the weighted set.

46. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Particle Filter Algorithm (Single Target) Step 1: Initialisation. N particles are uniformly distributed across the Field of View (FoV). Step 2: Data Update. After a new measurement, weights are assigned to the particles according to their likelihoods. Step 3: Resampling An unweighted particle set is obtained by resampling from the weighted set. Step 4: Estimation of Target Location The location of the target is estimated by calculating the mean position of the particles.

47. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Particle Filter Algorithm (Single Target) Step 1: Initialisation. N particles are uniformly distributed across the Field of View (FoV). Step 2: Data Update. After a new measurement, weights are assigned to the particles according to their likelihoods. Step 3: Resampling An unweighted particle set is obtained by resampling from the weighted set. Step 4: Estimation of Target Location The location of the target is estimated by calculating the mean position of the particles. Step 5: Prediction The location of the next target location is estimated using the motion model.

48. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Particle Filter Algorithm (Multiple Target) Step 1: Initialisation. N particles are uniformly distributed across the Field of View (FoV).

49. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Particle Filter Algorithm (Multiple Target) Step 1: Initialisation. N particles are uniformly distributed across the Field of View (FoV). Step 2: Data Update. After a new measurement, weights are assigned to the particles according to the data update equation for the PHD.

50. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Particle Filter Algorithm (Multiple Target) Step 1: Initialisation. N particles are uniformly distributed across the Field of View (FoV). Step 2: Data Update. After a new measurement, weights are assigned to the particles according to the data update equation for the PHD. Step 3: Estimation of Target Location The locations of the targets are estimated by fitting a Gaussian mixture model to the particles where the number of targets is the sum of the weights.

51. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Particle Filter Algorithm (Multiple Target) Step 1: Initialisation. N particles are uniformly distributed across the Field of View (FoV). Step 2: Data Update. After a new measurement, weights are assigned to the particles according to the data update equation for the PHD. Step 3: Estimation of Target Location The locations of the targets are estimated by fitting a Gaussian mixture model to the particles where the number of targets is the sum of the weights. Step 4: Resampling An unweighted particle set is obtained by resampling from the weighted set.

52. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Particle Filter Algorithm (Multiple Target) Step 1: Initialisation. N particles are uniformly distributed across the Field of View (FoV). Step 2: Data Update. After a new measurement, weights are assigned to the particles according to the data update equation for the PHD. Step 3: Estimation of Target Location The locations of the targets are estimated by fitting a Gaussian mixture model to the particles where the number of targets is the sum of the weights. Step 4: Resampling An unweighted particle set is obtained by resampling from the weighted set. Step 5: Prediction The locations of the next target locations are estimated using the PHD prediction equation.

53. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Implementation on Forward Scan Sonar Data n beams separated by k degrees

54. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Implementation on Forward Scan Sonar Data n beams separated by k degrees Intensity vs time data

55. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Implementation on Forward Scan Sonar Data n beams separated by k degrees Intensity vs time data Tracks range and bearing of targets

56. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Implementation on Forward Scan Sonar Data n beams separated by k degrees Intensity vs time data Tracks range and bearing of targets Measurements are obtained by thresholding the data:

57. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Implementation on Forward Scan Sonar Data n beams separated by k degrees Intensity vs time data Tracks range and bearing of targets Measurements are obtained by thresholding the data: The motion of the sonar is assumed to be linear Field of View in the range of 20m to 60m

58. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Implementation on Forward Scan Sonar Data Sequence of Simulated Forward Scan Sonar Images with Objects

59. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Results on Simulated Data Linear Tracking in Sonar Image Reference Frame

60. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Results on Simulated Data Linear Tracking in Global Reference Frame

61. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Results on Simulated Data Sinusoidal Tracking in Global Reference Frame

62. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Implementation on Forward Scan Sonar Data Sequence of Real Forward-Scan Images

63. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Results on Real Data Tracked Cylinder in Forward Direction

64. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Results on Real Data Tracked Cylinder in Backward Direction

65. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Extensions To PHD Filter: The multi-target state set does not give individual target identities. How do we associate measurements between frames? Two possible approaches:

66. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Extensions To PHD Filter: The multi-target state set does not give individual target identities. How do we associate measurements between frames? Two possible approaches: Increase state vector with invariant attribute

67. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Extensions To PHD Filter: The multi-target state set does not give individual target identities. How do we associate measurements between frames? Two possible approaches: Increase state vector with invariant attribute Data Association

68. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk What do I do with it now? The target tracks will be used for alignment by computing a geometric transform between the frames eg affine/similarity.

69. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk What do I do with it now? The target tracks will be used for alignment by computing a geometric transform between the frames eg affine/similarity. This will be used as a preliminary step for 3D reconstruction of a sequence of images.

70. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk Future Work on Reconstruction Segmentation into different seabed types. Reconstructed 3D elevation map of sequence of images.

71. Forward-Scan Sonar Tomographic Reconstruction PHD Filter Multiple Target Tracking dec1@hw.ac.uk References R. Mahler: "Multitarget Bayes Filtering via First-Order Multitarget Moments", IEEE Transactions on Aerospace and Electronic Systems, 2003. Vo, Singh and Doucet: “Sequential Monte Carlo Implementation of the PHD Filter for Multi-target Tracking” Proc. FUSION 2003 Sidenbladh and Wirkander: “Tracking random sets of vehicles in terrain” IEEE Workshop on Multi-Object Tracking, Madison, WI, USA, 2003