Multimedia Databases (MMDB) A Content-Based Image Retrieval Perspective (CBIR)

Types of Media Files Static media: images and handwriting Dynamic media: video and sound bytes Dimensional media: 3D games or CAD)

MMDB Motivation Factors Acquisition: email, phone, web sites like FLIKR Generation: camera phones, digital cameras, Storage: databases design Processing: power and techniques more sophisticated Huge increase in multimedia data on computers and their transmission over networks.

Database Support Databases provide consistency, concurrency, integrity, security and availability of data for the large amount of multimedia data available. From a user perspective, databases provide functionalities for manipulation and querying the huge collections of stored data.

Media Data Stored Media data: actual media data representing images, audio, and video that are captured, digitized, processed, compressed and stored. Media format data: consists of media data format stored after the acquisition, processing, and encoding phases. Examples are sampling rate, resolution, frame rate, encoding scheme etc. Media keyword data: For example, for a video this might include the date, time, and place of recording , the person who recorded, the scene that is recorded. Also called content descriptive data. Media feature data: This contains the features derived from the media data. A feature characterizes the media contents. For example, this could contain information about the distribution of colors, the kinds of textures and the different shapes present in an image. This is also referred to as content dependent data.

Image Retrieval (IR) Image Retrieval is the process of searching and retrieving desired images from a large database. IR provides resourceful use of prolific image data The efficiency of implementations have increased over the past two decades

IR Methodology A simple image retrieval implementation uses individually entered keywords or descriptions of inserted images so that retrieval is performed over the annotations in normal textual forms.  If an image is poorly or incorrectly annotated, or a poor choice of arbitrary query values are given by the user then the desired output is not received even if it exists in the database. Therefore, a lot of research has gone into automatic annotation of image description.

Image Features Stored Color: Red, Blue, Green, etc Texture: Similarity in grouping of pixels Shape: Edge detection Spatial: Spacing of Features Semantic: Correlated description of image data. E.G: Color = blue, Shape = Large, Texture = smooth, Spatial = Top of image: Sky

Feature Extraction Image Segmentation: open ended topic Segment Classification: based off characteristics Filtering Techniques: Extract image features such as texture by passing images through a filter

Feature Application In DB Users could supply a range for color, texture, or shape for queries Features can be generated on a typical semantic set for automatic annotation of new pictures

Content based image Retreival (CBIR) Avoids the necessary use of textual descriptions Organizes digital archives by visual content Retrieves images based on visual similarity to a user-supplied query image or image features.

Query Types Keyword: common text searching techniques Feature: Ex. Draw area for location and size. Select color regions. Select shape. B+-tree is traversed based off given index value. Semantic: Provide words to describe feature sets that are used to query a database Composite: Index involves combination of above

Query examples from CIBR at the end of the early years

Content-Based Straight-forward implementation is each feature is used as an index. Not very efficient for querying Create an index as described earlier as a combination of region classification, spatial location, shape, and color. EX. 20-bit index key: 3bits location, 8 bits color, 4 bits size, 5 bits shape. B+-tree indexing method is used.

Relevance Feedback A query modification technique attempts to improve retrieval performance through iterative feedback and query refinement. Used in ALIPR

Data Flow From CBIR at the end of early years

IR Implementation Examples Yahoo or Google Image Searches: based mainly on annotated description and filename Automatic Linguistic Indexing of Pictures (ALIPR): learning algorithm that annotates with feedback

Future Work The open ended nature of image segmentation restricts the accuracy of object recognition. As segmenters improve so will the databases capability. The integration of image retrieval can be implemented in computer vision applications. Many researchers believer that image retrieval has grown out of its infancy and now focus will be on applications and proliferating algorithms into indivduals lives.

Bibliography Datta, R., Joshi, D., Li, J., and Wang, J. Z. 2008. Image retrieval: Ideas, influences, and trends of the new age. ACM Comput. Surv. 40, 2 (Apr. 2008), 1-60. DOI= http://doi.acm.org/10.1145/1348246.1348248 Stanchev P., Using Image Mining for Image Retrieval, IASTED International Conference “Computer Science and Technology”, May 19-21, 2003, Cancun, Mexico. "Multimedia Database." Information Technology Portal (IT Portal) - India. Web. 04 Dec. 2009. <http://www.peterindia.net/MultimediaDatabase.html>. "Image retrieval -." Wikipedia, the free encyclopedia. Web. 04 Dec. 2009. <http://en.wikipedia.org/wiki/Image_retrieval>. Smeulders, “Content-based image retrieval at the end of the early years” A.W.M.Journal:IEEE transactions on pattern analysis and machine intelligence, 2000, Vol:22, 12, 1349