1. Graphics Recognition – from Re-engineering to Retrieval Karl Tombre, Bart Lamiroy LORIA, France

2. Document Analysis in the IR era n Information is at the core of industrial strategies n A lot of digital or digitized information, but often in very “poor” formats n The challenge: not necessarily re- engineering of documents, but enrich poorly structured information, add (limited) amount of semantics, build indexes ðPurposes: browsing, navigation, indexing ðDAR methods and tools useful, but must be adapted

3. Specific challenges of large- scale IR applications n Genericity: we cannot necessarily build a complete and exhaustive a priori model of contextual knowledge (ontology) n Adaptability: various input data – scanned paper, PDF, DXF, HTML, GIF… – various resolutions n Robustness: “back-office” applications n Efficiency: online searching in heterogeneous data n Scaling: methods have to scale to increasing number of symbols/features

4. DAR and IR n Media without (or with very little) contextual knowledge ðImage-based indexing and retrieval, indexing of video sequences n Documents do explicitly convey information from one person to another person ðMuch more structure, syntax and semantics

5. DAR and IR – some examples n Indexing and/or searching scanned text without OCR ðSimilarities, signatures n Query or index on layout structure n Table spotting n Keyword spotting n …

6. What about Graphics Recognition? n Subfield of DAR, for graphics-rich documents n Numerous methods for various analysis and recognition problems u Raster-to-vector conversion u Text/graphics separation u Symbol recognition n Many specific technical areas: maps, architectural drawings, engineering drawings, diagrams and schematics, …

7. Graphics recognition methods n Text/graphics separation

8. n Vectorization Graphics recognition methods

9. Graphics recognition and IR applications n Usual text-based indexing and retrieval still useful n But need for access to other kinds of information: u Symbols u Text-drawing connections u Description-illustration connections

10. Some contributions n Syeda-Mahmood – maintenance drawings IEEE Trans. On PAMI 21(8):737-751, Aug. 1999

11. Some contributions n Arias et al., Najman et al. – use of information contained in legend / title block Proc. GREC’01, Kingston (Ontario, Canada), p.19-26, Sept. 2001

12. Some contributions n Samet & Soffer – symbols from legend IEEE Trans. On PAMI 18(8):783-798, Aug. 1996

13. Some contributions n Müller & Rigoll – graphical retrieval in database of engineering drawings Proc. ICDAR’99, Bangalore (India), pp. 697-700, Sept. 1999

14. Some contributions n Boose et al. (Boeing) – Generation of Layered Illustrated Parts Drawings (GREC’ 03) Proc. GREC’03, Barcelona, pp. 139-144

15. Wishful thinking? Symbol DB Or even better…

16. Symbol recognition n Natural features for indexing and retrieval n Most methods work with known databases of reference symbols – what about interactive querying of arbitrary symbols? n From segmentation followed by recognition, to segmentation-free recognition, or segmenting while recognizing n Scalability ûEfficiency / complexity ûDiscrimination power ðSignatures Before we move on: 1st contest on symbol recognition held last week See IAPR TC10 homepage for further details

17. Image-based signatures n Compute invariant signatures on binary document image  F -signatures (ICDAR’01) u Radon transform: R-signatures [Tabbone & Wendling] u Ridgelets [Ramos Terrades & Valveny – GREC’03] – aka wavelet transform of Radon transform

18. R-signatures Detection of arrowheads [Girardeau & Tabbone] DEA degree thesis, INPL, Nancy, Jul. 2002

19. R-signatures Another example [Girardeau & Tabbone]

20. Ridgelets [Ramos Terrades & Valveny – GREC’03] Proc. GREC’03, Barcelona, pp. 202-211

21. Vector-based signatures [Dosch & Lladós – GREC’03] n Based on set of basic graphical features: u Parallelism u Overlap u Collinearity u T- and V-junctions n Quality factor associated with the various relations n Match signatures of reference symbols with signatures of buckets

22. Vector-based signatures Proc. GREC’03, Barcelona, pp. 159-169

23. Towards symbol spotting n Pre-compute – or compute on the spot – a set of basic signatures n Can be sufficient for symbol spotting and retrieval n Followed by classical symbol recognition if more discrimination is needed

24. Symbol spotting n [Jabari & Tabbone] : graph matching through probabilistic relaxation, with nodes=segments and vertices=relations DEA degree thesis, INPL, Nancy, Jul. 2003

25. Symbol spotting n [Jabari & Tabbone] : another example

26. Combining Text and Graphics n Extracting Text/Graphics relationships within document n Using Text matching for inter-document relationships n Transitive inter-document Graphics matching n No need for complex graphics matching n Restricted to well known document types

27. Example: continuation of Wiring Diagrams (Boeing) n [Baum et al. – GREC’03] Proc. GREC’03, Barcelona, pp. 132-138

28. Scan2XML Example Proc. GREC’01, Kingston (Ontario, Canada), pp. 312-325

29. Indexing and Semantics n Signature + metric n Semantics = measured distance to signature n Applies only to homogenous contexts u Pre-segmented images u Pre-determined image classes u Implicit application of domain kowledge u... n Semantics = Syntax

30. Example Signature type A Metric M Semantics1 = (  1,  1) Semantics2 = ( ,  2)  Signature value M(  M(  semantics = measurement to reference value

31. Heterogenous Document Bases n Semantics do not have a unique syntax anymore n Syntax metrics may be context sensitive n Semantics = Syntax + Context Context needs to be considered

32. Two different contexts from the automobile industry

33. Example Context 1: Signature type A Metric M (  1,  1) = Semantics1 = (  1, 1) ( ,  2) = Semantics2 = ( , 2) Context 2: Signature type B Metric N  Signature value What if M(  and N( 

34. A step to taking into account context (while consolidating existing approaches) Component Algebra : u Image Analysis = Pipeline u Syntax + algorithm = semantics Algorithm Data (syntax) Data (semantics) Algorithm Data (semantics) Syntax and semantics need not be distinguished

35. Component Algebra n Components : Known and implemented document analysis algorithms, taking input data from one domain, and producing data into another domain. n Application Context : Set of all available Components. n Semantics : Data sets needed by or produced by Components.

36. Component Algebra is a Graph Component Data Component Data Component

37. Advantages n Each node is a semantic concept, semantic relationships are explicitly expressed. n Structure may support automatic reasoning and knowledge inference. n Context is embedded in components, different contexts give different paths in the graph. n Highly scalable and open architecture. n Bridge between signal-level document analysis and high-level document representation.

38. However... The formalism exists, the realization doesn't (yet) n What about parametrization ? n How context independant can you get ? n What about « guessing » context appropriateness ? n How to design fully interoperable components ?

39. Conclusion n A lot of DA methods – and more specifically GR methods – can be of direct use in IR, indexing and browsing applications n Specific challenges u Scaling and efficiency u Heterogeneous sets of documents u Incomplete domain knowledge u Symbol spotting u On-the-fly symbol searching n Sketch of open framework for including document semantics when context can be heterogeneous