Towards Performance Evaluation of Symbol Recognition & Spotting Systems in a Localization Context Mathieu Delalandre CVC, Barcelona, Spain EuroMed Meeting LORIA, Nancy city, France Monday 18th of May 2009

Introduction symbol backgrou nd text Recognition Spotting r1 r2 r3 sof a ski n tub doo r docum ent databa se learnin g datab ase Query By Exampl e (QBE) rank labels Symbol spotting: “a way to efficiently localize possible symbols and limit the computational complexity, without using full recognition methods” [Tombre2003] [Dosch2004] [Tabbone2004] [Zuwala2006] [Locteau2007] [Qureshi2007] [Rusinol2007] Symbol recognition: ““a particular application of the general problem of pattern recognition, in which an unknown input pattern (i.e. input image) is classified as belonging to one of the relevant classes (i.e. predefined symbols) in the application domain” [Chhabra1998][Cordella1999] [Llados2002] [Tombre2005] Electrical diagram Mechanical drawing Utility map scanne d CAD fileWeb image

Introduction Characterisati on Groundtruth Groundtruthin g Results Performance evaluation System Performance evaluation: Information Retrieval [Salton1992], Computer Vision [Thacker2005], CBIR [Muller2001], DIA [Haralick2000] Case of symbol recognition & spotting: [Ezra2008][Delalandre2008] Traini ng data dATA Data Spotting/Recognition System Groundtruth Mapping Region Of Interest Characterization sofa skin tub door Labels r1 r2 r3 Ranks QBE truth results Learning Performance evaluation

Plan 1.Groundtruth and test documents 2.Performance characterization 3.Conclusions and perspectives

Groundtruth and test documents Overview of approaches Real approach Document Groundtruth Groundtruthing SpeedRealismReliabilitySymbolConnectedNoise [Dosch’ manyyesno Yan’ manyyesno Rusinol’ manyyesno Aksoy’ manynoyes Zhai’ onenoyes Valveny’ onenoyes Delalandre’08++++manyyesno - - weak ++ good real appro ach synthetic approach GT groun d- truthin g validation groundtruth drawing s and alerts groundtruthed drawings validatio n and alerts evaluation test imag es recogniti on results Dosch and al connected parallel and overlapped Yan and al Overview of approaches 2. Existing datasets Rusinol and al 2009

Groundtruth and test documents Overview of approaches Synthetic approach Document Groundtruth Groundtruthin g Setting - - weak ++ good real appro ach synthetic approach Aksoy 2000 binary noise vectorial noise Valveny and al 2007 Zhai and al Overview of approaches 2. Existing datasets SpeedRealismReliabilitySymbolConnectedNoise [Dosch’ manyyesno Yan’ manyyesno Rusinol’ manyyesno Aksoy’ manynoyes Zhai’ onenoyes Valveny’ onenoyes Delalandre’08++++manyyesno

symbol backgroun d Graphical documents are composed of two layers To use a same background layer with different symbol layers Groundtruth and test documents Overview of approaches - - weak ++ good real appro ach synthetic approach Delalandre Overview of approaches 2. Existing datasets SpeedRealismReliabilitySymbolConnectedNoise [Dosch’ manyyesno Yan’ manyyesno Rusinol’ manyyesno Aksoy’ manynoyes Zhai’ onenoyes Valveny’ onenoyes Delalandre’08++++manyyesno

Delalandre2008 Groundtruth and test documents Overview of approaches c2c2 c1c1 M1M1 M2M2 M3M3 M4M4 C1C1 C2C2 C3C3 C4C4 L1L1 θ1θ1 p1p1 L2L2 θ2θ2 p2p2 p L  bounding box and control point alignm ent symbol model loaded symbol 1. Overview of approaches 2. Existing datasets - - weak ++ good real appro ach synthetic approach SpeedRealismReliabilitySymbolConnectedNoise [Dosch’ manyyesno Yan’ manyyesno Rusinol’ manyyesno Aksoy’ manynoyes Zhai’ onenoyes Valveny’ onenoyes Delalandre’08++++manyyesno

Delalandre2008 Groundtruth and test documents Overview of approaches GT Positioning Constraints Symbol Models Document Generation Symbol Positioning Symbol Models Building Engine (2) run (3) display (1) edit Backgroun d Image 1. Overview of approaches 2. Existing datasets - - weak ++ good real appro ach synthetic approach SpeedRealismReliabilitySymbolConnectedNoise [Dosch’ manyyesno Yan’ manyyesno Rusinol’ manyyesno Aksoy’ manynoyes Zhai’ onenoyes Valveny’ onenoyes Delalandre’08++++manyyesno

Groundtruth and test documents Existing datasets datasetsimagessymbols degradatio ns models GREC’03# GREC’05# GREC’07# ICPR’00# bags# none floorplans# none16 diagrams# none21 queries#66000 none16-21 Rusinol’09#142344none38 GRE C 1. Overview of approaches 2. Existing datasets ICPR SESYD Others

Groundtruth and test documents Existing datasets GRE C 1. Overview of approaches 2. Existing datasets ICPR SESYD datasetsimagessymbols degradatio ns models GREC’03# GREC’05# GREC’07# ICPR’00# bags# none floorplans# none16 diagrams# none21 queries#66000 none16-21 Rusinol’09#142344none38 Others

Groundtruth and test documents Existing datasets GRE C 1. Overview of approaches 2. Existing datasets ICPR SESYD datasetsimagessymbols degradatio ns models GREC’03# GREC’05# GREC’07# ICPR’00# bags# none floorplans# none16 diagrams# none21 queries#66000 none16-21 Rusinol’09#142344none38 Others

Groundtruth and test documents Existing datasets 1. Overview of approaches 2. Existing datasets GRE C ICPR SESYD datasetsimagessymbols degradatio ns models GREC’03# GREC’05# GREC’07# ICPR’00# bags# none floorplans# none16 diagrams# none21 queries#66000 none16-21 Rusinol’09#142344none38 Others

Groundtruth and test documents Existing datasets 1. Overview of approaches 2. Existing datasets GRE C ICPR SESYD datasetsimagessymbols degradatio ns models GREC’03# GREC’05# GREC’07# ICPR’00# bags# none floorplans# none16 diagrams# none21 queries#66000 none16-21 Rusinol’09#142344none38 Others

Groundtruth and test documents Existing datasets 1. Overview of approaches 2. Existing datasets GRE C ICPR SESYD datasetsimagessymbols degradatio ns models GREC’03# GREC’05# GREC’07# ICPR’00# bags# none floorplans# none16 diagrams# none21 queries#66000 none16-21 Rusinol’09#142344none38 Others

Groundtruth and test documents Existing datasets 1. Overview of approaches 2. Existing datasets GRE C ICPR SESYD datasetsimagessymbols degradatio ns models GREC’03# GREC’05# GREC’07# ICPR’00# bags# none floorplans# none16 diagrams# none21 queries#66000 none16-21 Rusinol’09#142344none38 Others

Groundtruth and test documents Existing datasets 1. Overview of approaches 2. Existing datasets GRE C ICPR SESYD Groundtruth Generator of queries 1. Random selection of a document 2. Radom selection of a symbol v0 x s  [0,1] y v max 3. Random crop datasetsimagessymbols degradatio ns models GREC’03# GREC’05# GREC’07# ICPR’00# bags# none floorplans# none16 diagrams# none21 queries#66000 none16-21 Rusinol’09#142344none38 Others

Groundtruth and test documents Existing datasets 1. Overview of approaches 2. Existing datasets GRE C ICPR SESYD datasetsimagessymbols degradatio ns models GREC’03# GREC’05# GREC’07# ICPR’00# bags# none floorplans# none16 diagrams# none21 queries#66000 none16-21 Rusinol’09#142344none38 Others

Plan 1.Groundtruth and test documents 2.Performance characterization 3.Conclusions and perspectives

Performance characterization Introduction Performance characterisation (segmented symbols) [Valveny2004] [Dosch2006] [Valveny2007,2008a,2008b] Recognition rate Precision/Recall Homogeneity Separability Performance characterisation (real context) Spotting/Recognition System Groundtruth Mapping Region Of Interest Characterization sofa skin tub door Labels r1 r2 r3 Ranks QBE truth results Learning Performance evaluation

Performance characterization About mapping groundtr uth segmentation Layout analysis [Antonacopoulos1999] Text/graphics separation [Wenyin1997] groundt ruth segmen tation truth results Single : a model line matches only with one detected line. Split : two model lines match with one detected line. Merge : a model line matches with two detected lines. False alarm : a detected line doesn't match with any model lines. Miss : a model line doesn't match with any detected lines. Mapping cases Symbol spotting [Rusinol2009] Groundtruth Results Mappingc1c2 g1g2 r

Performance characterization Mapping, application to symbol wrapper box, ellipsis convex polygon the precision will depend of the model could be of weak precision Which representation ?How to define the regions ? concave polygon precise but comparison is time consuming the polarized pat of the capacitor belong to the symbol ? Same for the moving area of the door ? Lot of systems use sliding windows to detect symbols providing only points [Adam2001] [Dosh2004] [Rusinol2007] point How to define local thresholds Compatibility with recognition systems ? groundtruth segmentation Lot of systems use sliding windows to detect symbols providing only points [Adam2001] [Dosh2004] [Rusinol2007] Systems providing region of interest can “tune” their results, how to limit the over segmentation cases ?

Performance characterization Work in progress Comparison of some criteria System of [Qureshi’08], 100 floorplans (2521 symbols) Domain definition of the ROI Orientation sampling [0- 2 π ] Reporting [0- 2 π ] Rate s % Region size dx × dy res ults groundt ruth Signature based characterization

Plan 1.Groundtruth and test documents 2.Performance characterization 3.Conclusions and perspectives

Conclusions and perspectives Conclusions –Large databases of segmented symbol images exist “GREC” –Synthetic databases in real context exist “SESYD” –True-life documents and groundtruth are at the corner “EPEIRES” –Characterization tools have been proposed “SymbolRec” Perspectives –Continue to produce other databases, using existing platforms –Mapping is the key problem today, to achieve a performance evaluation in real context

Thanks All the referenced papers can be found in [1]M. Delalandre, E. Valveny and J. Lladós Performance Evaluation of Symbol Recognition and Spotting Systems: A Overview. Workshop on Document Analysis Systems (DAS), pp , 2008.