Sensing for Hydrographic Autonomous Surface Vehicles Coral Moreno 2019 US Hydro Conference Hello everyone My name is Coral Moreno ASVs are becoming a popular tool in hydrography, but the reality is that they are not truly autonomous. Today my talk is going to be a prospective analysis of what is going to be required to make ASVs autonomous enough to operate safely in coastal environments while conducting a mapping mission. Center for Coastal and Ocean Mapping / Joint Hydrographic Center University of New Hampshire 3.19.2019

Coastal Mapping with ASVs Complexity of Coastal Environments Readiness Level of an ASV We at CCOM are interested in coastal mapping with ASVs So let’s see what this interest is involved with: 1- what is the complexity level of coastal environments 2- what is the readiness level of an ASV to operate autonomously in such environments on its own or with minimal supervision and intervention. _______________________ ASV for hydrography – The availability of smaller manageable size and more sophisticated ASVs made their usage more realistic for coastal surveys.

Coastal environments are challenging for ASV operations It is hard to navigate in coastal areas Because there are rocks, lobster pots, peirs, heavy boat traffic and people in the water. All those things that in the deep ocean the ASV is less likely to encounter. Close to shore, the ASV has to operate reliably and safely in the presence of humans and objects.

Autonomous Situational Awareness ASV Autonomy OPERATOR ONLY FULLY AUTONOMUS HUMAN SUPERVISED HUMAN DELEGATED 1 Level 2 3 4 Guidance & Control Safe Navigation Safe Navigation Mission Existing operational modes: Remote-control Auto heading WPT navigation Auto speed Path following Autonomous Situational Awareness Where does ASV autonomy stand now? ***************** Autonomy can be think of from 3 aspects: make the boat drive on its own make the boat aware of its environment and make navigational decisions, similarly to what a captain of a ship or a boater do; carry out the mission autonomously. (in hydrography that would be make the hydrographer job during data acquisition autonomous, e.g. supervising data acquisition, adjusting sonar parameters etc… There are different divisions of autonomy levels in the literature, but to simplify it I adopted this rough scheme of 4 levels that span from fully under human control (level 1) to fully autonomous (level 4). 1st level is the remote control mode. 2nd level is “human delegated”. This is all the low level automatic controls, like auto-heading and auto-speed. (similar to the cruise control we have in our cars) 3rd level is human supervised. ASV can perform a wide variety of more sophisticated behaviors when given top-level permission. Such as an execution of a survey plan. Human and system can initiate behaviors but ASV can do so only within the scope of its task. 4th level is full autonomy. - ASV receives mission goals, translates them into tasks to perform, and it makes all decisions. Human can enter the loop in case of an emergency To see where we are at with ASV autonomy let’s look at What are the current operational modes ? And those are: remote control, auto heading and speed, and WPT navigation. This means that now we are at levels 2-3 of autonomy. There is a huge gap between levels 3 and 4, and a critical component to bridge that gap will be to increase the autonomous situational awareness of the ASV. This is exactly what is happening in the self-driving car regime, and my thesis in part will check how well those approaches can be applied for the marine environment. For the remainder of the talk I will discuss how autonomous situational awareness can be achieved. _______________________________________________________________________________________________________ Source: “Unmanned Systems Integrated Roadmap FY2011-2036” by the DoD

Obstacles Static Dynamic Providing the ASV an environmental awareness means that the ASV identifies the objects in its surroundings, understands whether they are static or dynamic , their current location, speed and heading, and predict their future behavior All of this information may come in a form of a map that is updated close to real time and is centered around the ASV and aligned with its reference frame. To review very quickly what those obstacles are: Some of them are static (landscape, manned structures, different kinds of floats and submerged things like seaweed) and some are dynamic ( vessels of different kind and size, humans, and drifting things like floating kelp). __________________________________________________ i. Static •Landscape: rocks, shoreline (iceberg, cliffs – Val said to not mention these for the conference) •Manned structures: oil/gas platforms, piers/ports •Navigation buoys, lobster pots, floats •Submerged/underwater: seaweed, shoal zones ii. Dynamic (size and maneuverability) •Large: commercial ships, research vessel •Medium: super yachts , tag and burge etc •Small size: fishing boats, motor boats, sailboats (,manned/unmanned) •Humans: surfers, paddlers, kayakers, swimmers floating ice, floating kelp

Charts Static Dynamic Source: Sam Reed How can we gather information about those obstacles? One way would be to use charts. Charts contain information about: 1) most of the static obstacles 2) the depth 3) where it is safe to navigate. And we had a project on chart-aware navigation of an ASV but: 1) not all obstacles appear on the chart (e.g. dynamic obstacles, anchored fishing gear) 2) charts may be outdated, for example the coast line may have changed or a new pier was built. 3) Not always available. Sometimes we want to use an ASV to map uncharted waters. So in addition to charts we can use the ASV sensors for situational awareness. Source: Sam Reed

ASV Sensors for Perception: Automatic Identification System (AIS) Lawrance Radar Velodyne LIDAR FLIR camera RGB camera Automatic Identification System (AIS) For my analysis I will focus on the Sensors available on our development system, which is a CW4. Above water: AIS - AIS sends and receives the following information of a vessel within radio range: name, size, speed, heading, CPA, TCPA, LAT/LONG location Radar and Lidar – measure range and bearing to an obstacle LiDAR Color camera FLIR camera Underwater: Sonar systems for underwater detection: MBES (we have EM2040) , Forward looking sonar, and side-scan sonar I am going to focus more on the above water domain Multi-Beam Echo-Sounder: Kongsberg EM2040P

Static Dynamic Radar Let’s see what kind of obstacles each sensor can detect: Radar can detect most of the above water obstacles except for things like floating kelp.

Static Dynamic AIS AIS good only for vessels with working AIS system onboard

Static Dynamic LIDAR Lidar can be good for detection of above water objects. (Might work for kelp since it may not be fully absorbed by the water)

Static Dynamic Camera Camera is the sensor with the richest data. It can be used to identify all objects.

Static Dynamic FLIR Camera FLIR camera can detect well things with big temperature difference. So It may be good for detection of a man in the water, but not so much for floating kelp. It won’t work well underwater but will be good for most of the above water objects. ______ Time should show 5:30 or 6:00 min.

Radar AIS 10km/20km 200m 100m 100m-1km 50m-40km Camera LIDAR FLIR 20ox25o (30o-50o)x (50o-90o) 200m 100m 100m-1km Now let’s examine the range and FOV of the various sensors: the range of each sensor AIS – about 10/20 km Radar – 50m-24NM (~44km) [Spec] *depends on range settings and location of antenna Lidar - 100m Hard to specify max range for camera systems because it depends on many different factors. (It depends on object’s contrast, projected size on camera sensor, and camera focal length and algorithm used.) Color Camera- some papers claims for 100-150m, but the range can reach 500m or maybe even 1km if the weather is nice and the algorithm can identify objects of the size ~30 pixels FLIR – 50m-200m (high performance systems can detect even small 30’ outboard vessel at beyond 5NM) Zoom system can improve camera system performance and extend the detection range. The FOV of each sensor AIS radar and lidar have FOV of 360 Camera – (30-50)x(50-90) [VxH] FLIR – 20x25 [VxH] AIS is great as long as other vessel has such system working onboard. For close range (<100m) lidar and cameras provide the best solution resolution: camera>lidar>radar (radar has the lowest resolution) _____________________ Resolution V H Radar 25o±5o 5.2o±0.5o LiDAR 1.33o 0.1o-0.4o RGB Camera 448pxl 800pxl FLIR 480pxl 640pxl 50m-40km

Sensor Applicable in Bad Weather Poor Light Conditions High Waves and Water Reflectivity Poor Light Conditions Radar LIDAR Color Camera FLIR Camera AIS The sensors applicability in different conditions: 4. In Bad weather and sea state: radar and AIS work better than lidar, a color camera and a flir camera but this means that close-range detection and identification is limited 5. In poor light conditions: object detection can still be done using radar, lidar, a FLIR camera and AIS. This means visual identification is limited and means must be invested in object identification algorithms from lidar and a flir camera data. ______________________________________________________________________ NOT SAYING!! Radar – pros and cons - https://www.practical-sailor.com/issues/37_7/features/broadband_radar_10508-1.html http://support.lowrance.com/system/selfservice.controller?CONFIGURATION=1001&PARTITION_ID=1&secureFlag=false&TIMEZONE_OFFSET=&CMD=VIEW_ARTICLE&ARTICLE_ID=2335 • Less effective picking up distant storm cells. While conventional, 4kW radar will help track approaching storms from 30 miles or more, the BR24 will see strong cells at 10-12 miles. Simrad says the new latest “3G” version can see cells at 17 nautical miles or more. • Less effective at detecting difficult shorelines. The BR24 will not detect sloping beaches and shorelines as well as some conventional radar. FLIR allows to see clearly in total darkness, solar glare, and through light fog and smoke. Bad weather and sea conditions degrade the detection LIDAR sensitive to sun glare Liu, Zhixiang, et al. "Unmanned surface vehicles: An overview of developments and challenges." Annual Reviews in Control41 (2016): 71-93.

Object Identification Sensor Functionality Object Detection Object Identification Occupancy Grid Mapping Radar LIDAR Color Camera FLIR Camera AIS The functionality of the sensors: 6. Object detection can be done with all sensors , but is it easier with an AIS, radar and lidar since they are measuring it directly. And it is harder with flir and color cameras because they must employ appropriate algorithms. 7. Object identification can be done with an AIS and with lidar, vision and FLIR cameras using appropriate algorithms. These are good for relatively close ranges. 8. Occupancy grid map shows probabilistic locations of obstacles in a geo-spatial context as it was measured by sensors that provide range. Those are – an AIS, radar and lidar. Historically (Naturally), a camera does not provide range information, but new algorithms may allow us to restore range from a single camera. ______________________________________________________________________________________________________ NOT SAYING!! Other challenges for camera systems: image stabilization, water splash * Object detection and identification requires algorithms. Modern algorithms relies on machine learning (ML) and deep learning (DL). Not accurate without a lot of data Not explainable – if something goes wrong it is hard to point of where was the problem Requires strong processing units Liu, Zhixiang, et al. "Unmanned surface vehicles: An overview of developments and challenges." Annual Reviews in Control41 (2016): 71-93.

Complementary nature of sensors Objectives Complementary nature of sensors Sensor Fusion So we can see that the sensors have a complementary nature. The weak points of one sensor can be compensated by other sensors Currently, there are no algorithms that fuse all of the ASV sensors together. And there are some (partial) algorithms that were developed for non-marine environments, such as self-driving cars and drones. In my PhD I am going to adjust those algorithms to the marine environment and integrate them in order to offer a solution to the problem.

Examples of Algorithms 2D Object Identification (YOLOv3, 2018) 3D Object Identification (PointNet ,2016) Monocular Range Image (MonoDepth, 2017) Here are some examples of algorithms that can be used 1) 2D object detection – this algorithm takes photos from the camera and draws boxes around detected objects and provides classification of those objects 2) 3D object detection – this algorithms accepts point cloud from the lidar , and classify points that belongs to the same object. (This is called semantic segmentation, since it gives semantic meaning to segments.) Those can contribute to object identification from the cameras and the lidar. The third algorithm shows that there may be a way to extract range from an image. _______________________________________________________________ 3) the 3rd algorithm accepts an image from a camera as an input, and outputs a depth image, that is an image that shows distance to every object in the image - here we can see that it has some troubles dealing with splashes on the camera and reflections from water surface. - it can come as reinforcement to radar measurements in mid range (beyond the lidar range). _____________________________________________ NOT SAYING!! What is DL? Sub-field of ML. It is about learning representations from data. This is done by learning successive layers of representations that are more and more meaningful. What are NNs? Those successive layers stacked on top of each other are called Neural Networks. 3D Object Detection: - PointNet classification network on 3D point clouds taken from LIDAR, Depth cameras or CAD models They trained an MLP on each point separately (with shared weights across points). Each point was ‘projected’ to a 1024 dimension space. Then, they solved the order problem using a symmetric function (max-pool) over the points. This yielded a 1 x 1024 global feature for every point cloud which they fed into a nonlinear classifier.   They also solved the rotation problem using a ‘mini-network’ they called T-net.   It learns a transformation matrix over the points (3 x 3) and over mid-level features (64 x 64).  A multilayer perceptron (MLP) is a class of feedforward artificial neural network. It is basically a fully-connected network. *challenges include compensation from motion and tracking Depth Estimation monocular: MonoDepth: Unsupervised Monocular Depth Estimation with Left-Right Consistency Sensitive to splashes and reflection from water surface https://github.com/charlesq34/pointnet

1) Sensor Fusion for 3D Object Identification: 3D Point Cloud (nx3) RGB Image (RoI cropped) PointNet ResNet Dense Fusion 3D box corner offsets (nx8x3) Score (nx1) Global Fusion 3D box corner locations (1x8x3) Input Output Once the objects are detected and ranges are known, it is time to fuse the information. I will review 3 different types of sensor fusion algorithms: 1- Sensor Fusion for 3D object detection. This algorithm accepts 3D point cloud from a lidar and an image from the camera and outputs 3D bounding boxes around the identified objects with classification hypothesis in the point cloud and image data streams. ___________________________________________________________ NOT SAYING!! Xu, Danfei, Dragomir Anguelov, and Ashesh Jain. "Pointfusion: Deep sensor fusion for 3d bounding box estimation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018. https://arxiv.org/abs/1711.10871v2 Kocić, Jelena, Nenad Jovičić, and Vujo Drndarević. "Sensors and Sensor Fusion in Autonomous Vehicles." 2018 26th Telecommunications Forum (TELFOR). IEEE, 2018.

2) Sensor Fusion for Occupancy Grid Mapping Grid Mapping from Sensor Measurements 2D Occupancy Grid Xu, Danfei, Dragomir Anguelov, and Ashesh Jain. "Pointfusion: Deep sensor fusion for 3d bounding box estimation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018. Occupancy grid is used for navigation of autonomous vehicles in dynamic environments Sensors such as radar and lidar can be used to generate occupancy grid. The addition of camera data and object classification of 3D point cloud can add semantic information to the occupancy grid At the top we can see a car driving on a road and the generated grid map from sensors measurements. Bottom left corner – this is a 3D map of the environments that Rolls Royce produced with a Lidar and they use it for navigation of an autonomous ferry Bottom right corner is from the robotX competition of ASV. Here we can see the simulated marine environments, the 2D occupancy grid that shows the lidar readings and the interpretation of it. LIDAR https://www.red-dot.org/project/rolls-royce-intelligent-awareness-25950 Thompson, David John, "Maritime Object Detection, Tracking, and Classification Using Lidar and Vision-Based Sensor Fusion“ (2017). Dissertations and Theses. 377. https://commons.erau.edu/edt/377 Kocić, Jelena, Nenad Jovičić, and Vujo Drndarević. "Sensors and Sensor Fusion in Autonomous Vehicles." 2018 26th Telecommunications Forum (TELFOR). IEEE, 2018.

3) Sensor Fusion for Moving Object Detection and Tracking Radar Camera LiDAR AIS Vehicle State IR Camera Object Detection Image Fusion Moving Object Tracking List of Moving Objects IR Image Last kind is sensor fusion for moving object detection and tracking. This is one of the most challenging aspects of autonomous vehicle domain. There are different ways to do this, but in general, an object detection is done on each sensor separately. Then all the information is fused together , followed by tracking of moving objects And the Output is a list of moving objects that is transferred to some kind of obstacle avoidance (COLREGs compliant) algorithm. _______________________________________________________________________ NOT SAYING!! Xu, Danfei, Dragomir Anguelov, and Ashesh Jain. "Pointfusion: Deep sensor fusion for 3d bounding box estimation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018. The most common is sensor fusion of camera, lidar and radar. Kocić, Jelena, Nenad Jovičić, and Vujo Drndarević. "Sensors and Sensor Fusion in Autonomous Vehicles." 2018 26th Telecommunications Forum (TELFOR). IEEE, 2018.

ASA for ASVs - Research Workflow Semantic Occupancy Grid Chart AIS Radar Cameras LIDAR Fusion for Identification and Tracking Obstacle Avoidance SENSORS The goal is to build a semantic occupancy grid map around the ASV that updates near real time, and contains information on obstacles locations and states with semantic information of what those objects are. So first we can use the chart as a start point. Next we can utilize the sensors to extract range to objects, and to identify them according to what we discussed earlier. The sensors here are arranged according to range hierarchy. Next comes fusion for identification and tracking of the obstacles. Lastly, this information passes to an obstacle avoidance algorithm that takes into account the mapping mission. In my research I will try to build such an occupancy grid. Since the data from a chart, AIS and radar can be used directly, my work efforts will focus on object detection/identification from a camera and lidar, and then fuse it all together to obtain the semantic occupancy grid.

Conclusions Coastal areas provides unique challenges for ASVs. Sensor fusion is the key for the improvement of ASV autonomy because no single sensor provides complete solution for all obstacles. The way to achieve this is by improving algorithms for object detection and identification, and occupancy grid mapping. This is what my Ph.D. is going to be about. This topic has been under development in the academy, navy and industry in recent years but is still in early stages.

This work is supported by NOAA Grant NA15NOS4000200 Coral Moreno PhD Student, Ocean Engineering CCOM/JHC University of New Hampshire Chase Ocean Engineering Lab 24 Colovos Road Durham, NH 03824 cmoreno@ccom.unh.edu This work is supported by NOAA Grant NA15NOS4000200