2. Cortana and the Speech Platform
WinHEC 2015
12/27/2017 6:26 AM
Cortana and the Speech Platform
May Ji 吉晓茜
Principal Program Manager
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

3. Introduction and agenda
Tech Ready 15
12/27/2017
Introduction and agenda
Session introduction
Learn about the new, exciting features in Cortana and the speech platform including far- field and Wake on Voice (WoV) from Modern Standby, Cortana’s availability on Windows 10 IoT Core, and how to build high-quality innovative devices to light them up.
What is Cortana
Cortana… what can she do?
The Audio and Speech Platform
Call to Action
© 2012 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

4. Cortana 小娜 WinHEC 2015 12/27/2017 6:26 AM
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

5. What is Cortana? Cortana is your truly personal digital assistant,
12/27/2017 6:26 AM
What is Cortana?
Cortana is your truly personal digital assistant,
there for you whenever and wherever you need,
to make sure nothing slips through the cracks.
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

6. 12/27/2017 6:26 AM
Cortana Experience
Your truly personal digital assistant, there for you to make sure nothing slips through the cracks.
Natural and easy to interact with
Provides proactive and personalized assistance
Cortana remembers, so you don’t have to
Works with apps and services to help you get things done
© 2014 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

7. Cortana Momentum +400M 1M >140M #1 +13B 1000+
Ask me anything
+400M
Devices running Windows 10— and growing every day 1M
Voice Queries Daily
>140M
MAU use the Cortana Search Box
Increase in revenue per device, as compared to Windows 8. #1
Most appealing feature in consumer research
+13B
# of questions consumers have asked Cortana
1000+
Skills

8. Cortana Market Availability
Team: Microsoft Search
Template: Search Product Marketing
12/27/2017 6:26 AM
Cortana Market Availability
FRANCE
French
CANADA
English, French
GERMANY
German UK
English
ITALY
Italian
JAPAN
Japanese
USA
English
SPAIN
Spanish
CHINA
Mandarin
INDIA
English
MEXICO
Spanish
BRAZIL
Portuguese
AUSTRALIA
English
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

9. Cortana … what can she do

10. Immersive Cortana Experience
Immersive Cortana Experience
Cortana can help you with all kinds of tasks on your lock screen, even when the screen is turned off!

11. Cortana on Windows 10 devices
WinHEC 2015
12/27/2017 6:26 AM
Cortana on Windows 10 devices
Because Cortana is available across device types, she’s able to help you even more effectively
Cortana was first available on Windows phones
Then Cortana came to Windows 10 PCs, tablets, Windows Holographic and Xbox
Windows devices
With upcoming Windows 10 Creators Update, Cortana will be available on Windows 10 IoT Core devices with displays
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

12. Powered by the Audio and Speech Platform
WinHEC 2015
12/27/2017 6:26 AM
Microsoft is committed to reinventing the PC and inspiring ecosystem growth with
Better
Experience Cortana with far-field Voice and Wake On Voice (WoV) from Modern Standby
Better
Marketing will be differentiated features for 2017
Better
Device
…on innovative devices to drive excitement in the market.
Powered by the Audio and Speech Platform
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

13. The Audio and Speech Platform 音频和语音平台
WinHEC 2015
12/27/2017 6:26 AM
The Audio and Speech Platform 音频和语音平台
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

14. Audio and Speech Platform Stack
WinHEC 2015
12/27/2017 6:26 AM
Audio and Speech Platform Stack
Cortana, Shell, Bing, Office, Skype, Microsoft’s and 3rd party’s Apps and Services
Audio Pipeline, Keyword Spotter, Speech Recognition, text-to-speech
Microsoft and 3rd party PC and IoT devices with displays
Experience
Speech Platform
Hardware
High quality hardware
Performant Speech Platform
Great Cortana Experience
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

15. Wake on Voice (WoV) from Modern Standby
WinHEC 2015
12/27/2017 6:26 AM
Far-field
Interact with Cortana from 4 meters away with great user experience, even with noises and music playing
Requires high quality microphone design that meets Microsoft speech spec 2.0
Wake on Voice (WoV) from Modern Standby
Wake up a device from a screen-off state to a screen-on, user-interactive state, by saying “Hey Cortana”
Designed for Modern Standby
Far-field
Wake on Voice (WoV) from Modern Standby
Differentiation
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

16. VOICE ACTIVATION Voice Activation
WinHEC 2015
12/27/2017 6:26 AM
Voice Activation
The scenario of providing keyword detection of a predefined activation key-phrase for an application.
For example, "Hey Cortana" is the hero Microsoft Voice Activation scenario for Cortana
VOICE ACTIVATION
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

17. WAKE ON VOICE Wake on Voice (WoV)
WinHEC 2015
12/27/2017 6:26 AM
Wake on Voice (WoV)
Any technology that enables Voice Activation from a screen-off state (lower power) to a screen-on full power (S0) state.
SCREEN ON
SCREEN OFF
WAKE ON VOICE
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

18. CORTANA AND IMMERSIVE CORTANA
LEGACY SLEEP (S3) SUPPORT
MODERN STANDBY SUPPORT
WinHEC 2015
12/27/2017 6:26 AM
The Windows 10 Anniversary Update provides a checkbox to entice users to prevent automatic system sleep, such that "Hey Cortana" can listen even when the screen is off.
This is our first step on the WoV journey and provides a WoV- like experience
SCREEN ON
CORTANA AND IMMERSIVE CORTANA
SCREEN OFF
WOV-LIKE EXPERIENCE
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

19. CORTANA AND IMMERSIVE CORTANA
LEGACY SLEEP (S3) SUPPORT
MODERN STANDBY SUPPORT
WinHEC 2015
12/27/2017 6:26 AM
Wake on Voice from Modern Standby
WoV from a Modern Standby (S0ix) screen-off state to a screen-on full power (S0) state
SCREEN ON
CORTANA AND IMMERSIVE CORTANA
SCREEN OFF
WOV-LIKE EXPERIENCE
WAKE ON VOICE FROM MODERN STANDBY
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

20. CORTANA AND IMMERSIVE CORTANA
LEGACY SLEEP (S3) SUPPORT
MODERN STANDBY SUPPORT
WinHEC 2015
12/27/2017 6:26 AM
Wake on Voice with SW KWS from Modern Standby
WoV using SW KWS from a Modern Standby (S0ix) screen off state to a screen on full power (S0) state.
SCREEN ON
CORTANA AND IMMERSIVE CORTANA
SCREEN OFF
WOV-LIKE EXPERIENCE
WOV WITH SW KWS FROM MODERN STANDBY
WINDOWS 10 CREATORS UPDATE
S0 SCREEN OFF, OR
LOW POWER AUDIO
“DEEP SLEEP”
S3, OR DRIPS
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

21. CORTANA AND IMMERSIVE CORTANA
LEGACY SLEEP (S3) SUPPORT
MODERN STANDBY SUPPORT
WinHEC 2015
12/27/2017 6:26 AM
Wake on Voice with HW KWS from Modern Standby
WoV using a HW KWS from a Modern Standby (S0ix) screen-off state to a screen- on full power (S0) state.
Key Prerequisites: DSP hardware-offload
SCREEN ON
CORTANA AND IMMERSIVE CORTANA
SCREEN OFF
WOV-LIKE EXPERIENCE
WOV WITH SW KWS FROM MODERN STANDBY
WINDOWS 10 CREATORS UPDATE
S0 SCREEN OFF, OR
LOW POWER AUDIO
WOV WITH HW KWS FROM MODERN STANDBY
FUTURE
“DEEP SLEEP”
S3, OR DRIPS
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

22. Software Keyword Spotter (SW KWS) WoV from Modern Standby
WinHEC 2015
12/27/2017 6:26 AM
Software Keyword Spotter (SW KWS) WoV from Modern Standby
Algorithm Provider
Runs on
Modern Standby
Big Buffer Capture
AC Power Only
Requirements
or 3rd party APO
Audio Processing
To reduce power consumption, the audio driver is required to implement a big audio capture system buffer
Premium devices should support 100ms to 200ms audio capture buffer size
This is done by specifying MaxPacketSizeInBytes in the DEVPKEY_KsAudio_PacketSize_Constraints2 device property
KSAUDIO_PACKETSIZE_PROCESSINGMODE_CONSTRAINT structure and KSAUDIO_PACKETSIZE_CONSTRAINTS2 structure on MSDN
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

23. Standard vs. Premium Near-Field Near-Field Far-Field
WinHEC 2016
12/27/2017 6:26 AM
Standard vs. Premium
Near-Field
Near-Field
Far-Field
Works well in ambient conditions at the typical 0.5m distance.
Works well from arms-length (leaning back) 0.8m in challenging environments like a busy kitchen or a family room.
Works well from further away, up to 4m, in challenging environments like a busy kitchen or a family room
Hey Cortana,
What is the status of my flight
Hey Cortana, will the Seahawks win this weekend?
Hey Cortana…
Hey Cortana, will the Seahawks win this weekend?
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

24. “Hey Cortana” Hola Cortana 你好小娜 Ehi Cortana Hé Cortana コルタナさん
WinHEC 2015
12/27/2017 6:26 AM
“Hey Cortana”
Hola Cortana
 OS Release
Locale (BCP 47)
Selected Key-phrase
Win 10
en-US
Hey Cortana
en-GB
fr-FR
it-IT
Ehi Cortana
de-DE
es-ES
Hola Cortana
zh-CN
你好小娜
(Nǐ hǎo xiǎo nà)
Nov. (1511) Update
ja-JP
コルタナさん
(Korutana-san)
en-IN
en-CA
en-AU
Anniversary Update
pt-BR
Ei Cortana
fr-CA
Hé Cortana
es-MX
Hé Cortana
コルタナさん
你好小娜
Ehi Cortana
The Voice Activation (VA) key-phrase for Cortana to respond is “你好小娜” in Chinese
Currently supported in 14 locales for near-field
Will be supported in English (US and GB) for far-field in Windows Creators Update
To ensure a first-class, consistent experience across all Cortana devices
“Hey Cortana” key-phrases are not configurable
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

25. Take Advantage of Windows Inbox Audio Processing
Echo cancellation, noise suppression, and beam forming
It is part of Windows
No additional audio driver required
No Additional Cost
State of the art technology from years of research and development
Top performance and quality without tuning
Great Performance
No tuning or driver updates needed
Continuous improvements available in OS updates from WU
Low Maintenance
Supports both near and far-field
Shipped in multiple Microsoft devices (Surface product line, Hololens, Xbox Kinect)
Far-field Support

26. Shipped in Broad Ecosystem Devices
WinHEC 2015
12/27/2017 6:26 AM
Shipped in Broad Ecosystem Devices
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

27. Far-field Microphone Recommendations
WinHEC 2015
12/27/2017 6:26 AM
Far-field Microphone Recommendations
Far-field Mic Array Type
Number of Mics Recommendation
Inbox Audio Processing Support
Mic Array Geometry
Linear 4
Yes
Follow the above mic array geometry if using inbox audio processing
Circular 8
Coming soon
Use 3rd party solution and follow their spec recommendations. Need to meet quality bar in Microsoft speech spec 2.0
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

28. Platform algorithm providers
WinHEC 2015
12/27/2017 6:26 AM
Testing Requirements
There are separate testing requirements for platform algorithm providers and OEMs & ODMs in speech spec 2.0
Partner Type
Areas to Test
Languages to Test
Platform algorithm providers
Speech Accuracy
All 14 supported locales
Voice Activation (both correct and false accepts)
OEMs and ODMs
US English
Voice Activation (correct accept only)
US English and Japanese
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

29. Platform Quality Targets for Algorithm Providers
WinHEC 2015
12/27/2017 6:26 AM
Platform Quality Targets for Algorithm Providers
All “Hey Cortana” keyword spotting algorithms, including Microsoft SW KWS, are tested for performance measurements against the quality bar in the speech spec 2.0
Speech Accuracy
Voice Activation (Correct and False Accepts)
Release
People
Scenario
Standard
Premium
HMD
Near- field (0.5m)
Near- field (0.8m)
(4m)
2017
Male, Female, Children (5-12)
Quiet
95%
Echo
85%
90%
Noise (Medium)
Echo+Noise (Medium)
Noise (Loud)
n/a
Release
People
Scenario
Standard
Premium
HMD
Near-field (0.5m)
Near-field (0.8m)
* (4m)
2017
Male, Female, Children (5-12)
Quiet
90%
Echo
Noise (Medium)
Echo+Noise (Medium)
Noise (Loud)
n/a
Release
People
2017
<= 1 FA per 100 hours of continuous speech
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

30. Device Quality Targets for OEMs and ODMs
WinHEC 2015
12/27/2017 6:26 AM
Device Quality Targets for OEMs and ODMs
Speech Accuracy
Voice Activation (Correct Accepts)
Release
People
Scenario
Standard
Premium
HMD
Near- field (0.5m)
Near- field (0.8m)
Far-field (4m)
2017
Male, Female, Children (5-12)
Quiet
95%
Echo
85%
90%
Noise (Medium)
Echo+Noise (Medium)
Noise (Loud)
n/a
Release
People
Scenario
Standard
Premium
HMD
Near-field (0.5m)
Near-field (0.8m)
Far-field (4m)
2017
Male, Female, Children (5-12)
Quiet
90%
Echo
Noise (Medium)
Echo+Noise (Medium)
Noise (Loud)
n/a
85%
Test en-US only
Test en-US and ja-JP only
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

31. Audio Labs for Cortana and Speech Testing
WinHEC 2015
12/27/2017 6:26 AM
Audio Labs for Cortana and Speech Testing
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

32. Pairing Speech Platform Tests with HLK
WinHEC 2015
12/27/2017 6:26 AM
Pairing Speech Platform Tests with HLK
Cortana with Voice (CwV) testing done with tools provided
System submission created with Specialized PC feature
System submission packaged up with the logs from CwV logs
System submissions made to SysDev like they are today
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

33. Call to Action SYSDEV Portal Submission
WinHEC 2015
12/27/2017 6:26 AM
Call to Action
Build high-quality hardware to light up far-field and Wake on Voice (WoV) from Modern Standby
Develop Modern Standby and far-field devices for BTS/holiday 2017
Follow the technical guidance in speech specs 2.0
Work with audio driver vendors to implement larger audio capture system buffer size
Engage with Microsoft on those devices
SYSDEV Portal Submission
Submit your Speech Platform test results up to SYSDEV
© 2015 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

34. Technical Requirement References
WinHEC 2016
12/27/2017 6:26 AM
Technical Requirement References
Requirement
Detail
Reference / Spec
Speech Platform: Input Device Recommendations & Test Setup specifications 2.0
Audio input device design and development recommendations, test guidance
Speech Platform test tools
Application and content guidance for testing
Microsoft Speech Platform Test Tools
Voice Activation Test Sets (Platform provider Tests)
FAQ
Answers to FAQ
Answers to Frequently asked questions about Cortana, such as Wake-On-Voice, , etc.
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

35. Thank You 谢谢 Please follow WinHEC @ WinHEC.com 12/27/2017 6:26 AM
© 2016 Microsoft Corporation. All rights reserved. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.