Ground Truth Free Evaluation of Segment Based Maps Rolf Lakaemper Temple University, Philadelphia,PA,USA

An example for evaluation of a robot map based on mid level geometry (and an advertisement for segment based maps) Examination if evaluation can benefit from different data representation Ground truth free: we evaluate map consistency

Overview

We are evaluating GEOMETRIC MAPS (not semantic or topological maps). We are evaluating with respect to geometric consistency (not completeness) Classically there are two approaches: Pose based Grid based

Pose based Raw data grouped during data acquisition (single scan) Consistency is measured by correspondence of point groups (single scans) Groups are determined during data acquisition by visibility from similar poses Problem: grouping assumption can be violated (ground scans etc)

Grid based Consistency is measured by distribution of single points Data representation driven Problem: does not measure structural consistency – points belonging to certain structures in single scan can end in grid cells of different structures in global map (and hence might change the quality value)

Proposal: Structure Based Evaluation Data is grouped using underlying structure of single scans (similar to pose based evaluation, but a more natural grouping principle) Consistency is measured by distribution of groups (similar to grid based evaluation, but adjusted to groups in environment)

Mid Level Geometric representation is a (small) step towards semantic representation. Assuming an underlying 'form from function' paradigm, raw data points are grouped by form to clusters of similar function, i.e. groups are assumed to represent single functional entities.

Basic idea: Global map is a superimposition of single scans, consisting of segments. Single features of the environment are represented by multiple segments. We Cluster segments; single cluster for single feature. Evaluate intra cluster distance.

cluster

-- Mid Level Geometry Based Mapping -- Mid Level Geometry : e.g. geometric primitives, containing more geometric information than simple points. Line segments, Circles, Rectangles Splines (of all kinds) In this example, we use line segments

Extraction of primitives is the first step after obtaining the raw single scan. Further processing is entirely performed on primitives.

Note: We are not limited to evaluation of segment based mapping algorithms. Every mapping method using single scan matching can be evaluated, if output are corrected poses of single scans.

-- Why Segments? – 1)The segment based approach captures structural information, which goes significantly further than the information of object presence and location, contained in raw point data. Line segments combine multiple points to a single feature. This creates the properties of segment direction and feature membership.

-- Why Segments? – 2)The segment based approach is fast. A typical scan consists of <20 segments, while the number of data points is typically one to two orders of magnitude higher. This becomes important, when processing algorithms have complexity of >O(n)

-- Why Segments? – 3)The segment based approach is memory efficient, each line segment consists of two endpoints.

-- Why Segments? – 4)The segment based approach is precise. Segment endpoints don't have to be adjusted to a resolution parameter, hence there are no quantization errors. Resolution parameters are introduced in point based algorithms e.g. when occupancy grids are used.

Similarity Measure

To process segments (or any other primitives), we need a segment-distance measure, or segment-similarity measure. These measures must be carefully designed!

Examples:

note: usually, these similarity measures do not obey the metric axioms, an example case of our measure: D(s1,s2)=0 (although s1 != s2) Such a measure does not create a metric space, which limits the selection of processing algorithms.

Clustering

For the clustering, agglomerative hierarchical clustering in ’single’ mode is utilized. This method seeks to build a bottom up hierarchy of clusters, starting with each segment being a single cluster, ending in a single cluster containing all segments. Pairs of clusters are merged as one moves up the hierarchy. Cluster distances are determined by the segment similarity measure.

Hierarchical clustering is a well understood ‘classic’ algorithm. Does not need a metric space (in contrast to e.g. k- means) Problem: where to stop (where to cut links in the dendrogram)?

Confidence Measure

Confidence measure evaluates intra cluster consistency A standard measure for intra cluster consistency was already used as stop criterion for clustering Stop criterion for clustering is fast, but has certain drawbacks with respect to outliers We re-evaluate clusters using a geometrically motivated measure

Angular (C a ) and translational (C t ) confidence are computed separately Cluster confidence is C = min(C a,C t )

The resulting confidence C has same tendency as intra-cluster consistency, but is in stronger accord with human perception

Some results of the single cluster confidence

!!!

Confidence M of entire map is defined by weighted average of cluster confidence

Results

Random Distortion

Iterative Map Alignment

Map Comparison (M=0.2769)

Map Comparison (M=0.4355)

Map Comparison (M=0.6554)

FFS (0.3386) FFS & Virtual Scans (0.3876)

Outlook: Map Cleaning & Merging

Conclusion: Segment based representation and segment similarity measure allows for off the shelf clustering Clusters detect structural consistencies, advantage over grid approaches Intra cluster consistency defines consistency of feature representation Results are in accord with perception

Thanks!