Digital Days 29/6/2001 ISTORAMA: A Content-Based Image Search Engine and Hierarchical Triangulation of 3D Surfaces. Dr. Ioannis Kompatsiaris Centre for Research and Technology Hellas Informatics and Telematics Institute Thermi-Thessaloniki, Greece

Digital Days 29/6/2001 Outline Introduction Istorama architecture K-Means with Connectivity Constraint Algorithm (KMCC) Demo Object/model based coding Adaptive Triangulation and Progressive transmission Reduced pyramid - quincunx sampling Experimental results Conclusions

Digital Days 29/6/2001 Need for efficient image search Huge number of images or databases of images Highly visual and graphical nature of the Web Text descriptors are not always efficient Greater flexibility with “content-based” access Queries which are more natural to humans

Digital Days 29/6/2001 Proposed approach Usually a description, a “signature” or a set indexes is created for the whole image Images usually contain different objects Proposed approach: the image is first separated into objects (segmentation) Descriptors are created for each object The user can search for a specific object contained in images

Digital Days 29/6/2001 ISTORAMA architecture Server World Wide Web Data Base JDBC Java Data Base Connection User PHP Crawler - Spider Indexing - Retrieval Algorithms

Digital Days 29/6/2001 The K-Means with Connectivity Constraint Algorithm (KMCC) I Based on K-Means algorithm K-Means does not take into account spatial information In KMCC, the spatial proximity of each region is also taken into account by defining a new spatial center and by integrating the K-Means with a component labeling procedure KAutomatic correction of the number of regions K

Digital Days 29/6/2001 The K-Means with Connectivity Constraint Algorithm (KMCC) II Step1 K-Means is performed  Step2 Spatial centers are calculated Step3 Generalised distance LStep 4 Component labeling  L connected regions

Digital Days 29/6/2001 The K-Means with Connectivity Constraint Algorithm (KMCC) III

Digital Days 29/6/2001 Object descriptors Color, texture and spatial characteristics Color: histogram, 8 bins Spatial: (centroid), Shape: area, eccentricity where λ 1, λ 2 are the two first eigenvalues

Digital Days 29/6/2001 Experimental Results (Synthetic)

Digital Days 29/6/2001 Experimental Results (Synthetic)

Digital Days 29/6/2001 Experimental Results

Digital Days 29/6/2001 Experimental Results (Claire) Facial region Moving object Original sequence Frames 1-10 Segmentation

Digital Days 29/6/2001 Experimental results (Claire)

Digital Days 29/6/2001 Experimental results (table-tennis) Original sequence Frames 1-10

Digital Days 29/6/2001 Experimental results (table-tennis) Segmentation Moving objects

Digital Days 29/6/2001 Experimental results (Akiyo+Foreman) Facial region Original sequence Frames 1-10 Original sequence Frames 1-10

Digital Days 29/6/2001 Conclusions K-means with spatial proximity algorithm Multiple features segmentation Higher order segmentation Correspondence of objects between consequent frames Max-min criterion for automatic regularisation parameters

Digital Days 29/6/2001 Future work Use of texture Indexing of video Integration with text descriptors

Digital Days 29/6/2001 Triangular meshes of high quality are used in: Computer Aided Design 3D representation of objects (e.g. archaeological artifacts) Animation and visual simulation Entertainment (computer games) Digital Terrain Modelling Introduction

Digital Days 29/6/2001 Object/model-based coding

Digital Days 29/6/2001 Object/model-based coding

Digital Days 29/6/2001 Object/model-based coding

Digital Days 29/6/2001 Compression of finely detailed surfaces is necessary for: computation storage transmission display efficiency Adaptive triangulation

Digital Days 29/6/2001 Early, coarse approximations are refined though additional bits Progressive transmission

Digital Days 29/6/2001 Vertices removal and retriangulation [Schroeder] [Cohen] General mesh optimization process/function [Hoppe] Multiresolution analysis (MRA) [Lounsbery] Wavelets [Schroeder] [Gross] Progressive transmission [Schroeder] [Hoppe] Generalized triangle mesh representation [Deering] Background

Digital Days 29/6/2001 Properties of the algorithm Efficient compression of the wireframe information Simplification of the wireframe by adaptive triangulation Progressive transmission of the wireframe information Prioritised transmission of the wireframe Straightforward correspondence between successive scales

Digital Days 29/6/2001 Input surfaces Surface represented as a parametric function in the parametric space determined by the position of a set of control points or nodes It allows for arbitrary, possibly closed wire-frame surfaces to be defined.

Digital Days 29/6/2001 Input surfaces The filters are applied to the 2D parametric representation of the surface as though it were a 2D image with intensity equal to Such surfaces include also: depth images estimated from stereo pairs and every surface that is homomorphic to a plane, cylinder or torus

Digital Days 29/6/2001 Block diagram of the proposed procedure

Digital Days 29/6/2001 Reduced pyramid with quincunx sampling matrix

Digital Days 29/6/2001 Corresponding triangulation

Digital Days 29/6/2001 Optimal filters Optimal filters are determined by their Fourier transform: where is the power spectral density. Alternatively may be determined by the equation:

Digital Days 29/6/2001 Optimum bit allocation bits/vertex is assumed to be transmitted bits/vertex are allocated to each level using is the sum of error variances

Digital Days 29/6/2001 Error prioritization The prediction errors corresponding to all predicted vertices are calculated and sorted with the vertices corresponding to higher errors being put first on the list Higher Errors Lower Errors

Digital Days 29/6/2001 Entropy estimation Entropy coding is used The number of bits needed for error transmission is the entropy of the errors Using the quincunx sampling geometry at the receiver, there is no need to transmit the exact co-ordinates of the position of each transmitted vertex The final cost of the transmission is the sum of the error entropy and the position entropy

Digital Days 29/6/2001 Adaptive Triangulation Procedure Synthesis stage of the QMVINT pyramid The vertex along with the vertices used to predict it are added to the mesh Handling of cracks Triangulation of the next vertex

Digital Days 29/6/2001 Adaptive triangulation procedure

Digital Days 29/6/2001 Adaptive triangulation procedure

Digital Days 29/6/2001 Experimental results Original dense depth map and surface of the “Venus” data

Digital Days 29/6/2001 Experimental results 2569 vertices and 4006 triangles at level 2 MSE = 1.30

Digital Days 29/6/2001 Experimental results 7661 vertices and triangles at level 1 MSE = 1.30

Digital Days 29/6/2001 Experimental results vertices and triangles at level 0 MSE = 0.12

Digital Days 29/6/2001 Experimental results

Digital Days 29/6/2001 Conclusions Hierarchical representation of 3D surfaces using 3D adaptive triangular wireframes The variance of the error transmitted is minimised and therefore results to optimal compression of the wireframe information It produces a hierarchy where coarse meshes are as similar to their finer versions as is possible

Digital Days 29/6/2001 Conclusions The triangulation algorithm is integrated with a bit allocation procedure The number of nodes and triangles of the wireframe as well as the information needed for the transmission or storage of the wireframe are reduced simultaneously using a unified approach (QMVINT filtering) Precise correspondence between triangles at each level is achieved

Digital Days 29/6/2001 Future work Expansion and application directly to 3D surfaces Estimation of filters