1. 1 © 2007 Avaya Inc. All rights reserved. Avaya – Proprietary & Confidential. Under NDA Tips for Writing Gooder Grammars Judi Halperin Speech Engineer

2. 2 © 2007 Avaya Inc. All rights reserved. Avaya – Proprietary & Confidential. Under NDA Overview Why does the grammar development process matter? Grammar Development Lifecycle –Best practices –What not to do –Why? Avoid the pitfalls!!

3. 3 © 2007 Avaya Inc. All rights reserved. Avaya – Proprietary & Confidential. Under NDA A successful ASR application requires… Design Application Development A good design… …efficient coding… Infrastructure …supporting infrastructure… Grammars …and effectively constrained grammars!!!

4. 4 © 2007 Avaya Inc. All rights reserved. Avaya – Proprietary & Confidential. Under NDA BOTTOM LINE… If the system does not recognize what callers say, they will not be able to complete their task in the IVR!! As a result… –Caller/Customer satisfaction will be negatively impacted –Containment rate heads south and ROI goes with it!

5. 5 © 2007 Avaya Inc. All rights reserved. Avaya – Proprietary & Confidential. Under NDA Grammar Development Lifecycle: Steps for Success…. Create TestTune Design and Plan

6. 6 © 2007 Avaya Inc. All rights reserved. Avaya – Proprietary & Confidential. Under NDA Design and Plan Should be started in parallel with VUI design –Determine feasibility Can the grammar be developed? Can the production hardware support it in real time? –Start large efforts early –Set expectations early If caller identification requires alphanumeric recognition, make sure your customer understands the recognition capabilities early on! –Host modification takes time –Dynamic and other complex grammars require more testing –New concepts and/or languages may require external resources A prescription names grammar will typically require both a large grammar AND corresponding lexicon development. –Allow time for collaboration with VUI designer and developer

7. 7 © 2007 Avaya Inc. All rights reserved. Avaya – Proprietary & Confidential. Under NDA Creating Your Menu Grammar Don’t include everything a caller might say, but rather what a cooperative caller would say –Leave the addition of possible synonyms to tuning –The more in-grammar items, the more chance of false accepts Consider logical synonyms based on the prompt –Callers have a tendency to parrot back the prompt: PROMPT: “..or say ‘main menu’ to start over.” Callers would be likely to say “main menu”, “start over” or “main menu to start over” Include the actual spelling of the word or phrase in the grammar to take advantage of the pronunciations in the system dictionary –“that’s” not “thats” –“t v” not “teevee” –“don’t” not “dont” –“f a q” not “f a queue” (it’s just silly)

8. 8 © 2007 Avaya Inc. All rights reserved. Avaya – Proprietary & Confidential. Under NDA General Grammar Development Tips Minimize the use of pre-and-post-fillers out of the box –PROMPT: “What’s your account number?” –Callers are likely to say “it’s account number...” but not “I think it might be...” –Leave the addition of these items to tuning once sufficient data proves their relevance Choose your built-ins wisely –Control your content when appropriate –Don’t use alphanumeric built-in grammar for spelling-only input Unnecessary overhead can have a latency effect Overgeneration leads to decrease in recognition accuracy (and possible latency) –Beware: Some built-ins overgenerate

9. 9 © 2007 Avaya Inc. All rights reserved. Avaya – Proprietary & Confidential. Under NDA Task: Allow spelling using military alphabet ASCII=letter.LETTER as in MIL = militaryAlpha.MIL alpha MIL="a" bravo MIL="b" …. yankee MIL="y" zulu MIL="z" a LETTER="a" b LETTER="b" ….. y LETTER="y" z LETTER="z"

10. 10 © 2007 Avaya Inc. All rights reserved. Avaya – Proprietary & Confidential. Under NDA Task: Allow spelling using military alphabet ASCII=letter.LETTER as in MIL = militaryAlpha.MIL alpha MIL="a" bravo MIL="b" …. yankee MIL="y" zulu MIL="z" This grammar generates “a as in alpha”… …but also “a as in zulu”!! a LETTER="a" b LETTER="b" ….. y LETTER="y" z LETTER="z"

11. 11 © 2007 Avaya Inc. All rights reserved. Avaya – Proprietary & Confidential. Under NDA Consider all methods of constraint Avoid overgeneration when possible –Don’t “sort it out” on the application side if you don’t have to But watch out for introduced latency –Don’t allow account numbers that are not valid –Make sure your pre-and-post-fillers make sense at every invocation: “I want my account balance” but not “I want my help” –Make sure your in-grammar items make sense at every invocation: “a as in alpha” but not “a as in zulu” Constrain birth dates to dates before today’s date and make assumptions based on caller demographics as to the earliest allowable date

12. 12 © 2007 Avaya Inc. All rights reserved. Avaya – Proprietary & Confidential. Under NDA Remove the overgeneration… a as in alpha LETTER="a" b as in bravo LETTER="b". y as in yankee LETTER="y" z as in zulu LETTER="z" This grammar generates “a as in alpha”… …but NOT “a as in zulu”!! ASCII=letter.LETTER

13. 13 © 2007 Avaya Inc. All rights reserved. Avaya – Proprietary & Confidential. Under NDA You have the power! Some platforms allow scripting within the grammar itself Grammar-internal scripting is useful in many situations –Further constrain dates, times, currency, etc –Implement a skip list or deweight a previous response that was negatively confirmed –Apply a checksum to disallow invalid digit string input –Calculate additional information based on the caller’s response Day of the week given the date Determine the time given colloquial response, ex: “in an hour”

14. 14 © 2007 Avaya Inc. All rights reserved. Avaya – Proprietary & Confidential. Under NDA Constrain using javascripting… tmpL=letter.LETTER as in tmpM=militaryAlpha.MIL <![CDATA[ if (tmpL != tmpM) { SWI_DISALLOW = 1; } else { LETTER = tmpL; } ]]> a LETTER="a" b LETTER=“b". y LETTER=“y" z LETTER="z" alpha MIL="a" bravo MIL="b". yankee MIL="y" zulu MIL="z"

15. 15 © 2007 Avaya Inc. All rights reserved. Avaya – Proprietary & Confidential. Under NDA “Simplify” Continuous String Recognition Constrain digits, alphanumeric and spelling grammars as tightly as possible –Load valid strings dynamically (especially for alphanumeric and spelling grammars) –If using a static grammar, apply appropriate patterns –Positively weight patterns that occur more often (if appropriate) –Apply logic to the task If collecting the 1 st two letters of a last name: –Use census data or other appropriate information to help constrain to the top 90+% of last names in the US –Don’t allow letter combinations that are not reasonably expected to occur given the task, or even producible by a native speaker, ex: “QB” or “CC”

16. 16 © 2007 Avaya Inc. All rights reserved. Avaya – Proprietary & Confidential. Under NDA Test Before Throwing it Over the Fence Open the grammar in a web browser or XML editor to validate Use platform tools to test the grammar –Try both valid and invalid entries –Verify correctness of the slot values Validate the mappings from raw text to slot value Validate any calculated values returned in slots –Generate a large corpus, extract unique entries and validate Load the grammar into a test application and give it a spin –Don’t worry about unit testing OOGs – leave that to tuning

17. 17 © 2007 Avaya Inc. All rights reserved. Avaya – Proprietary & Confidential. Under NDA Pilot Test and Tune Collect a sufficiently large data set over various times, dates and demographics –Not just high traffic or low traffic periods –Not during a one-time promotion –Not only when callers might have a higher tendency towards otherwise infrequently used functionality Tax season, beginning of the month, etc. Transcribe what the caller said Analyze performance and make recommendations for improvements Tuning should be performed periodically as appropriate

18. 18 © 2007 Avaya Inc. All rights reserved. Avaya – Proprietary & Confidential. Under NDA Avoid Grammar Development Pitfalls!! Plan it out… Avoid overgeneration… Constrain as tightly as possible… Start large efforts early… Include only what a cooperative caller would say… Pay attention to the prompt… Include actual spellings… Minimize out of-the-box pre-and-post-fillers Use built-ins as appropriate… Use grammar-internal scripting if appropriate… ‘Simplify’ continuous string recognition… Test, test, test…. Generate custom pronunciations if needed… Pilot test and tune…. Validate your grammar….

19. 19 © 2007 Avaya Inc. All rights reserved. Avaya – Proprietary & Confidential. Under NDA Judi Halperin jhalperin@avaya.com