1. CM613 Multimedia storage and retrieval Lecture: Video Compression Slide 1 CM613 Multimedia storage and retrieval Content-based image retrieval D.Miller

2. CM613 Multimedia storage and retrieval Lecture: Video Compression Slide 2 Fundamental problem of search by machine v. search by a person Sensory gap –Gap between the “object in the world” and its digital representation Semantic gap –Gap between information that a machine extract from the image and what the the image means to someone.

3. CM613 Multimedia storage and retrieval Lecture: Video Compression Slide 3 Heirinymus Bosch: The Garden of Earthly Delights (detail from the panel “Hell”) Hieronymus, or Jerome, Bosch, b. c.1450, d. August 1516, spent his entire artistic career in the small Dutch town of Hertogenbosch, from which he derived his name. At the time of his death, Bosch was internationally celebrated as an eccentric painter of religious visions who dealt in particular with the torments of hell.

4. CM613 Multimedia storage and retrieval Lecture: Video Compression Slide 4 Statistical Techniques: colour histogram Basically just a count of how many pixels of each (R, G and B) value are present –Positive »Not sensitive to orientation –Negative »Spatial information is lost. http://www.cultivate-int.org/issue7/artiste/

5. CM613 Multimedia storage and retrieval Lecture: Video Compression Slide 5 Statistical Techniques: Colour layout Divides an image into a grid and searches for similar colours in equivalent grid position. –E.g blue sky

6. CM613 Multimedia storage and retrieval Lecture: Video Compression Slide 6 Statistical Techniques: texture Using orientation and spacing of edges Also a histogram of edge features can be used Also texture layout (as colour layout) http://www.cultivate-int.org/issue7/artiste/

7. CM613 Multimedia storage and retrieval Lecture: Video Compression Slide 7 Techniques: searching for an object Much more difficult than previous techniques http://www-rocq.inria.fr/~gouet/HCP/index.htm l

8. CM613 Multimedia storage and retrieval Lecture: Video Compression Slide 8 C-Bird case study Content-Based Image Retrieval from Digital Libraries –Colour histogram –Colour Density –Colour layout –Texture layout –Illumination invariance –Object model

9. CM613 Multimedia storage and retrieval Lecture: Video Compression Slide 9 C-Bird case study: Colour histogram “coarse” histogram –3 bits for red –3 bits for green –2 bits for blue »Giving 256 “bins” for the histogram. Returns matches above a threshold, rather top few matches –So sometimes may return nothing. Similarity is measured by histogram intersection –A histogram is generated for each image in the database »Each histogram is a sequence of numbers »Each value is divided by the sum of all the values in the histogram to remove the size of the image as a factor (normalisation). »The histogram is stored in the database. »When a comparison of two histograms is made, the minimum value of equivalent bins is summed for all the bins. The closer the sum is to 1, the better the match.

10. CM613 Multimedia storage and retrieval Lecture: Video Compression Slide 10 C-Bird case study: Colour layout User sets up a scheme of how colour should appear –Coarse blocks of colour in grid sizes 1x1, 2x2, 4x4, 8x8. –Cells can be specified with any RGB colour value or not. If no value is specified that grid is ignored in the search. –Each image in the database is analysed for each of the 4 grid sizes and the 5 most frequent colours in each cell are stored.

11. CM613 Multimedia storage and retrieval Lecture: Video Compression Slide 11 C-Bird case study: Texture layout User sets up a scheme of how colour should appear –Coarse blocks grid sizes 1x1, 2x2, 4x4, 8x8. –Cells can be specified with pre-defined textures, such as: –Comparison can then be carried out in a way analogous to colour layout..

12. CM613 Multimedia storage and retrieval Lecture: Video Compression Slide 12 C-Bird case study: Illumination invariance Differences in illumination of an image can lead to RGB sensors picking up different colour values – pink under daylight might be detected as purple under fluorescent lighting, for example. It is desirable to remove the effect of differences in illumination from image when searching. This is dealt with by: – normalising each of the RGB bands of the image (analogous to the way normalisation was used with colour histogram), –Creating a 128x128 colour histogram of chromaticity (capturing only colour information, not brightness).

13. CM613 Multimedia storage and retrieval Lecture: Video Compression Slide 13 C-Bird case study: search by object model This is described by Li and Drew as “the most important search type C-BIRD” supports. The user picks a sample image and interactively selects a region for object searching – “object query-by-example”. Uses technique of “feature localization” as opposed to “image segmentation”. –Both use concept of “tiles” –However “locales” are: »collections of tiles that share similar features »and can be found in a an area of the image, »but need not necessarily be contiguous (they can be scattered about a bit) –Li and Drew outline a combination of techniques for creating the tile and locale features from raw image data.

14. CM613 Multimedia storage and retrieval Lecture: Video Compression Slide 14 Sources Li and Drew (2004) Fundamentals of Multimedia (module recommended book) http://www.cs.sfu.ca/cbird/ (demonstration site for C-BIRD)http://www.cs.sfu.ca/cbird/ Addis,M. Lewis, P. and Martinez, K. (2002) ARTISTE image retrieval system puts European galleries in the picture. http://www.cultivate- int.org/issue7/artiste/ (Accessed 4/05/06)http://www.cultivate- int.org/issue7/artiste/