Behrooz ChitsazLorrie Apple Johnson Microsoft ResearchU.S. Department of Energy

Multimedia Research Speech Search Face identification Object recognition Video browsing Semantic extraction (3D) Segmentation (3D) Image search

Speech as interface Speech as 1 st class content Mobile access Directory services Automation PC application Web service Text input Dictation Indexing Search Keyword extraction Transcription Meetings Voic s Closed Caption Translation Translating phone Speech Applications

Speech recognition Spectral Analysis Matching (Decoding) time alignment  most likely hypothesis W’=argmax (w 1..w N ) p(o t..o  |w 1..w N ) P(w 1..w N ) Acoustic Models p(o t..o  |phoneme) Dictionary P(phonemes|w) Grammar (Language Model) P(w 1..w N ) “Hello World” o 1..o T (w 1..w N )^

MAVIS technology Indexing automatic transcripts as text –Automatic transcription accuracy is only 50-80% MAVIS techniques –Word-level lattice indexing index word alternatives – robust to recognizer errors % accuracy improvement index timing – navigate to exact point in video –Vocabulary Adaptation Use NLP and Bing Search to expand word dictionary –Automatic keywords to expose to search engines Enables discovery of speech content through search engines Bi-product of vocabulary adaptation –See

MAVIS Architecture SQL Server(s) 1. Submit audio/video RSS 2. Retrieve AIB 3. Import AIB in SQL Web server(s) 4. Search/Retrieve results Store content to be processed in temporary Azure storage Do vocabulary adaptation using Bing Run recognition engine on content Store results or recognition process (AIB)

U.S. Department of Energy Office of Scientific and Technical Information (OSTI) Mission DOE invests > $10 billion/year in basic sciences, clean energy technology, and nuclear research. The immediate output from this investment is Information…Knowledge… R&D results OSTI’s mission is to accelerate scientific progress by accelerating access to this information.

OSTI’s Core Products Information Bridge Science Accelerator Science.gov

WorldWideScience.org

Emerging Forms of Scientific Information Require New Tools Numeric data, multimedia, and social media are emerging forms of scientific information Each form presents special opportunities and challenges

Search and Retrieval Challenges with Multimedia Science Information Lack of written transcripts, i.e. no “full text” to search Metadata, if available, is often minimal Scientific, technical, and medical terminology/vocabulary Videos can be long, often up to an hour or more

Video files collected from DOE’s National Laboratories RSS feeds with metadata and URLs sent to Microsoft Research Audio indexing performed via MAVIS Audio index blob (AIB) returned to OSTI and integrated with SQL servers Users can search for a precise term within the video, and be directed to the exact point in the video where the term was spoken OSTI and Microsoft Research Partnership

Demonstration of ScienceCinema ScienceCinema

Looking to the Future Additional content from DOE researchers Integration of multimedia searches into WorldWideScience.org by June High quality automatic closed captions Multilingual translation capabilities