1. 1 Multimedia Information Systems Vijay Atluri atluri@andromeda Office 200R Ackerson Hall Phone: 973-353-1642 Office Hours: 2 hours after the class and by appointment

2. 2 What are the most known media? n Images u 2-d color images, gray scale medical images 2-d (X-ray) or 3-d (MRI scans) u Captured images F Captured images need to be analyzed u Synthesized images / visualizations (Artificially created) F Synthetic images do not need to be analyzed ( featured information is already available). n Text u sequential vs. hypertext u semistructured F Organized into chapters, paragraphs etc. or may be indexed using HTML/SGML/XML

3. 3 What are the most known media? n Video u video clips, movies u captured u interactive n Audio u digitized voice or music n 1-d time-series u financial, marketing, production time series data such as stock prices, sales numbers n Handwritten u electronic notes n Traditional data n Scientific data u collections of sensor data, e.g.,

4. 4 MM Applications n Travel Industry u intelligent travel agent, multimedia presentation, n Entertainment Industry u Film clip database, video-on-demand, pay-per-view, interactive TV, in-flight entertainment, video game database, video dating services u Users will be able to select using a mix of query/retrieval and browsing capabilities n Education and training u classroom without walls: distance learning, teleclassrooms, interactive training, self education, employee reeducation F multimedia training is 40% more effective, retention rate is 30% higher, learning curve is 30% shorter (study conducted by DoD) F approximately US spends $56.6 billion every year (Marketing research by Training Magazine)

5. 5 MM Applications n Expert Advice u Auto repair, medical advice,.. n Home Shopping u multimedia presentation of goods sold, sales information n Medical databases n Text and photograph archives n Digital Libraries n Office automation n Electronic encyclopedia n DNA databases n Geographic Information Systems

6. 6 MMDBMS: Requirements n Ability to uniformly query multimedia data u query and seamlessly integrate data contained in different databases (that may possibly use different schema), flat files, object-oriented databases, spatial databases, arbitrary legacy sources u elicit the content of the media data (a challenge by itself) which is highly dependent on the media type and storage format u merge results from different data sources and media types n Ability to retrieve media objects from a local storage device in a smooth, jitter-free manner u considering large storage requirements, highly compressed format, secondary and tertiary storage devices, mix of storage devices with different performance characteristics

7. 7 MMDBMS: Requirements n Ability to develop a presentation in audiovisual media from the answer generated by the query n Ability to deliver the presentation that satisfies various quality of service requirements u synchronization, fidelity, temporal constraints u does not suffer from jitter and hiccups u limited buffer availability and bandwidth (output devices may reside at distributed network nodes)

8. 8 What kind of queries can users ask? J. Smith n Find all images which are created by J. Smith with the same color, shape and texture n Find all images with the same color, shape and texture which look likeimage n Find all images which look like this image which look like n Find all images which look like this sketch with the same color distribution like a sunset photograph n Find all images with the same color distribution like a sunset photograph which contains a part n Find all images which contains a part which looks like this image or sketch n Find companies whose stock prices move similarly n Find other companies that have similar sales patterns with our company n Find cases in the past that resemble last month’s sales pattern of our product

9. 9 What kind of queries can users ask? n Find past days in which the solar magnetic wind showed patterns similar to today’s pattern n Find similar music scores or video clips sunny days n Find all images of “sunny days” (we are getting into semantics) a car n Find all images which contain a car a caraman n Find all images which contain a car and a man who look like this similarobjects n Find all image pairs which contain similar objects. (data mining)

10. 10 Sample Multimedia Scenario (from the text) n Consider a police investigation of a large-scale drug operation: u Video data captured by surveillance cameras that record the activities taking place at various locations u Audio data captured by legally authorized telephone wiretaps u Image data consisting of still photographs taken by investigators u Document data seized by the police when raiding one or more places u Structured relational data containing background information, bank records, etc., of the suspects involved u Geographic information systems data containing geographic data relevant to the drug investigation being conducted

11. 11 Example of Queries for the MM Scenario n A police officer, Tom, has a photograph in front of him. He wants to find the identity of the person in picture. u Q1: “Retrieve all images from the image library in which the person appearing in the photograph appears” n Tom wants to examine pictures of a suspect Dick. u Q2: “Retrieve all images from the image library in which Dick appears” n Two types of queries u Image-based u Keyword-based

12. 12 Example of Queries for the MM Scenario n Q1: input is an image, output is a ranked list of images that are similar to the query image u Need to know what F “similarity” means F “ranking” means u Need to efficiently support these two operations n Q2: input is a keyword, output is an image whose name attribute is Dick u Need to know how to associate different attributes with images u Need to know how to effectively index and retrieve images based on such attributes

13. 13 Example of Queries for the MM Scenario n Tom is listening to an audio surveillance tape.It contains a conversation between two individuals A and B u Q3: “Find the identity of B, given that A is Dick n Tom wants to review all audio logs that Dick participated during some specified time period u Q4: “Find all audio tapes in which Dick was a participant” n Tom is browsing an archive of text documents (old newspaper archives, police dept files on old unsolved murder cases, witness statements) u Q5: “Find all documents that deal with financial transactions with ABC corporation” n Similar queries may be posed on video data

14. 14 Example of Queries for the MM Scenario n MM Query u Q6: “Find all individuals who have been photographed with Dick and who have been convicted of attempted murder in North America and who have recently had electronic fund transfers made into their bank account from ABC Corp.” F Need to access heterogeneous database systems F Need to access several MM databases: Mugshot database containing the pictures and names of individuals Surveillance photograph database of still images Surveillance video database Image processing algorithms to determine who is present in which video or still photograph

15. 15 MM Research Issues n Queries u Need a single language F with which MM data of different types can be accessed F with which one should be able to specify operations to combine different media types (just like join, union, intersection, difference, Cartesian product) F that must be able to access metadata as well as raw data F that must be able to merge, manipulate, and join together results from different media sources u After devising such a language, we need techniques to F optimize a single query F develop servers that can optimize processing of a set of queries

16. 16 MM Research Issues n Content u What is meant by content? u Under what conditions can it be described textually? u Under what conditions it must be described directly through the original media type? u How can we extract the content of F an image, a video-clip, an audio-clip, a free/structured text document? u How should we index the results of the extracted content? u What is retrieval similarity? u What algorithms can be used to efficiently retrieve media data on the basis of similarity?

17. 17 MM Research Issues n Storage u How well disks, CD-ROM, tape systems and tape libraries work? u How do we design disk/CD-ROM/tape servers so as to optimally satisfy different clients concurrently when they execute F playback, rewind, fast forward, pause, etc.

18. 18 MM Research Issues n Presentation and Delivery u How do we specify the content of MM presentations? u How do we specify the form (temporal/spatial layout, fidelity) of this content? u How do we create a presentation schedule that satisfies these temporal, spatial and fidelity requirements? u How can we deliver MM presentation to users when there is F a need to interact with other remote servers to assemble the presentation F a bound on the buffer, bandwidth, load, and other resources F a mismatch between the host server’s capabilities and the customer’s machine capabilities, preferences, etc.? u How can such presentations optimize Quality of Service?