1. Are We Ready? A Look at the State of the Art in Speech-to-text Applications
Marie Meteer
August 2007

2. Overview Speech Recognition: The State of the Art
A look back at where it came from
Elements of the models
State of the art performance
Applications: Making them work
Call Center Analytics
Voic Transcription
Needles in Haystacks
Multimedia search

3. BBN Technology’s Speech Milestones
Rough’ n’ Ready prototype system for browsing audio
Pioneered statistical language understanding and data extraction
Introduced context dependent phonetic units
Early adopter of statistical hidden Markov models
DARPA EARS Program Award
Exceeded DARPA EARS targets
1982
1986
1995
1998
2002
2004
1976
1992
1994
2000
2003
2005
Early continuous speech recognizer using natural language understanding
First 40,000 word real time speech recognizer
AVOKE STX 1.0 introduced
Audio Indexer System – st generation
Broadcast Monitoring System delivered to U.S. Gov’t. – 2nd generation
AVOKE STX 2.0 with Domain Development Tools
First software-only, real-time, large-vocabulary, speaker-independent, continuous speech recognizer

4. Progress in Speech Recognition 1990’s80 70
Call Home 60
SWBD Conversational Telephone 50 40
Word Error Rate (%) 30
Broadcast News 20
Resource Management
WSJ 64K Vocab 10
WSJ 5K Vocab 5
Airline Task 2
Resource Mgt Spkr Dep. 1
Connected Digits

5. DARPA EARS for ASR Performance
BBN’s 2003 Performance Exceeds
Broadcast news ceiling
Broadcast news floor
Telephony ceiling
Telephony floor
Word Error Rate Goals 60 50 40
Word error rate 30 20 10
2003
2002
2005
2007
Year

6. Elements of a Speech Model
Dictionary
List of all the words and their pronunciations, the sequence of “phonemes” that make up the word
>Real Networks R-IY-L N-EH-T-W-ER-K-S
Dictionary tool automatically creates phonetic pronunciations for most words
Acoustic Model
Captures the relationship between the sounds and the phonemes
Specific to a language (e.g. English, Spanish) and a channel (e.g. telephony, broadcast)
Domain Model
Captures the sequences of words in the language using a “tri-gram” model, that is the likelihood of a word given the two previous words
Can be as general as “Conversational” or as specific as “Technology”

7. Model Requirements Acoustic Data Domain Modeling data Dictionary
Minimum of hours transcribed data
English Broadcast News transcribed on 1600 hours of broadcast news data
Training data must be a precise transcription with corresponding audio file (including partial words, “um”, laugh, etc)
Domain Modeling data
Text data, either transcribed from audio or off the web
Does not have to be as precise as for acoustic modeling
Has to model both the vocabulary and “style” of speaking
Dictionary
Phonetic pronunciations of all of the words

8. Word Accuracy
Recognition performance varies based on audio quality and domain
Within News
Factors include
Speaker
Audio quality
Background music
Across Domains
Speaking style,
Out of vocabulary rate
SPEAKER
ACCURACY
Male Anchor 82
Female Anchor 76
Non-native over the telephone 53
Commercial 55
DOMAIN
ACCURACY
News
74.5
Movie Reviews
77.8
Technology
79.4
Gaming
59.45
Religion
68.2

9. Document Retrieval Accuracy
To correctly retrieve a document, a search term only has to be found once in the document
The table below reports on document retrieval accuracy based on words occurring 2 or more times in the document compared with overall word accuracy.

10. Markets and Applications
Consumer Search
(video search)
Government Intelligence
Call Center Recording
Broadcast Monitoring & Retrieval
(audio/video publication)
Digital Asset Production
Enterprise Search
(webcasts, corp info)

11. AVOKE Caller Experience Analytics
Breakthrough Caller Experience Analytics
The Only True End-to-End Solution
From dialing to termination
Multiple Techniques To Extract Understanding
Prompt and speech recognition, telephony data, and human annotation
Data-Driven Insights
With drill-down to listen for root cause
Zero Integration
No on-site hardware or software
To Manage & Optimize Contact Processes
Improve Operational Visibility
Reduce Agent Time by %
Boost First Call Resolution
Eliminate Customer Dis-Satisfiers

12. Full Text & Keyword Search
Search for words spoken by callers or agents
View call with full text of caller and call center – including all IVR(s), queue(s) and agent(s)

13. Voicemail Transcription
Requirements
Near real time transcription
High accuracy, especially on names
Frequently very noisy conditions (Non-native speaker calling on a cell phone from a street corner in Germany)
Solution
Speech recognition automates a “first pass”
Human correction provides accuracy
Full human transcription on poor quality calls

14. Voicemail Solution? Human in the loop
“Hi Tom. I can’t make the meeting but I’m available to call in. Give me a call at Thanks.”
Transcribers fix the output of the speech recognizer
Speech Recognizer produces a rough transcript
Phone message is left
Correct transcription goes back to the server
Result: High Quality, Lower Cost

15. Custom Applications: Broadcast Monitoring
Automatic translation of Arabic transcript from
Language Weaver MT
Automatic transcription of Arabic speech from
BBN Audio Indexer
Real-time streaming video (<5 min delay)
Continuous 24/7 video encoding and streaming
Real-time access to incoming video stream
Synchronized transcription and translation
Provides random access to spoken content in either language
30-day cache of recent video automatically maintained
Seek by date and time to any position in the cache
Search by keyword or by example in either language
Retrieve from cache and/or filter incoming stream with alerts
Export video segments and stills to PowerPoint, Word
Include selections of transcription and translation
Zero-maintenance design
No onsite administration required

16. MultiMedia Search Problem: Opportunity:
Search engines have historically had very little to work with in terms of properly discovering and indexing multimedia content:
Opportunity:
The value of multimedia content is “trapped” inside the files, out of view of search engines. Titles and tags miss key concepts within the files:
…let’s look at the overall picture not just Obama and and Clinton Brett how do you assess the overall dynamics of what's happened over the course of the last three months how big -- victory for the president how big a defeat for the Democrat well it it. He would have been a bigger defeat it was a victory. This is this is -- reprieve cents for the president it's only as bill pointed out for months worth of funding. And it's and this issue's going to come up again in the Democrats are going to continue to try to impose restrictions on the with a president for a just war -- vote to be funded completely which is what. We're just talking about so. This is just justices have a battle he wanted that's that's nice for him but there's another one coming in just a few months. And of course what we have now is this whole idea that is taken hold and it's it's out there in the in the public parlance about September being in the big month not helpful to the president's cause for prisoners efforts you know we're not going to -- all the troops on the ground until next month and then visiting get to bounce of the summer to try to fix the situation. Probably unrealistic which in September's going to be a tough month of. ...

17. Multimedia Consumption
Automatic extraction of key terms and concepts for tagging, categorization
Patent-pending “Snippet” navigation technology enables users to jump to relevant segments of the clip
Social media integrations drives RSS subscription, bookmarking, etc.
Full text output enables related content presentation

18. Multimedia Discovery Example: FoxSports.com
EveryZing Media Merchandising indexes the full contents of FoxSports Multimedia files.
As a result, EveryZing able to significantly increase the number of keyword results
Great discovery leads to increased consumption and enhanced monetization opportunities.
Search Term
EveryZing Results
FoxSports Results
EveryZing Increase
Manny Ramirez 22 7
214%
Yankees
281
111
153%
Manchester United 21 2
950%
Golf
214
170
25%
Federer 45 15
200%
David Beckham 36 17
111%
Tom Brady 53 31
71%

19. Summary
Speech recognition takes an inaccessible data structure (audio) and turns it into an accessible one (text)
It’s far from perfect, but it’s a big jump from nothing
Take away: It’s the task that matters. Find the right role, and speech recognition works
(Corollary: A good prompt is worth two years of research)

20. Media Merchandising Solutions Thank you
Media Merchandising Solutions Thank you! Marie Meteer VP of Speech and NLP