1. Troubleshooting and Introduction to New Tests Yuk Lai Suen, Windows Sound Team February 26-28, 2008 1 Munich Windows Vista LogoFest

2.  What Is Fidelity Test and How Does It Work?  When Must a New Log be Exported?  Something Went Wrong--What Do I Do?  Windows  Logo Kit 1.2 Preview February 26-28, 2008 2 Munich Windows Vista LogoFest

3.  We would like to make sure that end users have an acceptable experience with the quality of audio solutions on PCs.  We have set a minimum bar for the fidelity on PCs, and we would like to drive the industry with that standard:  SYSFUND-0018 (1-Jun-2006) - Premium system audio solution delivers a high-fidelity audio experience.  SYSFUND-0127 (1-Jun-2006) - System audio record and playback meet basic performance requirements for audio experience.  AUDIO-0006 (1-Jun-2007) - Audio solution delivers a premium high- fidelity audio experience.  To test devices and systems against these requirements, we implemented fidelity test for Windows Vista  RTM:  We shipped WLK for Windows Vista RTM. February 26-28, 2008 3 Munich Windows Vista LogoFest

4.  Standard Driver Test Manager (DTM) setup:  Controller.  Studio.  System under test.  Additionally:  Audio Precision analyzer.  Audio Precision host + APIB cable.  Audio cables. February 26-28, 2008 4 Munich Windows Vista LogoFest

5. February 26-28, 2008 5 Munich Windows Vista LogoFest

6.  Same setup as “run locally”:  Test results exported to a specially-decorated log file.  This is the same as the.wtl (Windows Test Log) file on the DTM controller, but with an added hash.  The hash ensures that the file has not changed since it was created. Hash: SHA-1/Encrypted 408e96f2f46e35f916454ac3fad19567b6cb788 (from here on the file is the same as the.wtl file) (this line is important... more on this later) (basic or premium; desktop or mobile) (and so are these... more on this later) (PnP ID plus hash of topology) () February 26-28, 2008 6 Munich Windows Vista LogoFest

7. February 26-28, 2008 7 Munich Windows Vista LogoFest

8. Setup changes from “run locally”: – No AP analyzer or audio cables necessary! – “AP host” is still necessary, but for DTM Scheduler reasons only. It takes no active role in the test - it is just a replica machine. – Fidelity Test must be able to reach and read the log file from the system under test. What about the log file? – Log file must be exactly the same as when it was exported (even white space, line breaks, and case). – Content of log file must match system being tested (more on this later). – Requirements must be compatible between log file and submission (for example, can't import a Basic pass into a Premium submission). February 26-28, 2008 8 Munich Windows Vista LogoFest

9. February 26-28, 2008 9 Munich Windows Vista LogoFest

10. A premium log can be imported to both Premium and Basic submissions. The list of PnP IDs of audio devices must match between the system under test and the log file: – If you add a USB Webcam, you must export a log that includes the USB Webcam. – If the codec REV_ number is incremented, you must export a new log. – If the SUBSYS_ number changes, you must export a new log. The audio “topology” exposed by each audio device in the system under test must match the one recorded in the log file: – If you can connect a pin on the codec to either a “CD in” or an “AUX”, you must export a log each way. If you have a third-party driver for a HD Audio device, export a log with the third-party driver installed and a log with the Microsoft HD Audio class driver installed. February 26-28, 2008 10 Munich Windows Vista LogoFest

11. This is a screenshot of an audio filter, viewed in KSStudio.exe, available as part of the Windows Driver Kit. February 26-28, 2008 11 Munich Windows Vista LogoFest

12.  The topology provides a detailed map of how data flows:  For capture, from the hardware through the driver to the operating system.  For render, from the operating system through the driver to the hardware.  Topology is exposed by the audio driver.  Driver changes might change topology:  If a driver version changes, it may or may not affect the topology exposed by any given hardware. February 26-28, 2008 12 Munich Windows Vista LogoFest

13.  USB Audio Class Driver:  Uses USB descriptor stored in the device to expose topology.  If the descriptor changes, the topology might change.  HD Audio Class Driver:  Uses HD Audio pin configuration registers stored in the BIOS to expose topology.  If the BIOS changes, the topology will probably change.  Jack Presence Detection:  Some hardware can detect when a jack is plugged in.  Windows Vista allows the driver to expose filters dynamically.  Always plug in all audio jacks - including HDMI - when running any audio test.  Use Multimedia Control Panel to verify that there are no “unplugged” audio endpoints. February 26-28, 2008 13 Munich Windows Vista LogoFest

14. February 26-28, 2008 14 Munich Windows Vista LogoFest

15. February 26-28, 2008 15 Munich Windows Vista LogoFest

16. Error in log file: Could not import from file (...) (error 0x8007....). Export dry run failed (...) (error 0x8007....). – 0x80070002 is ERROR_FILE_NOT_FOUND. – 0x80070005 is ERROR_ACCESS_DENIED. Solution: Fidelity Test runs on the System Under Test... (has different C, D,... drives from the Studio machine)... as local user.\DTMLLUAdminUser (will likely not have permission to \\server\share paths)  Best practice: Place the file on the C drive of the system under test February 26-28, 2008 16 Munich Windows Vista LogoFest

17. “Requirements in the log do not match requirements for this test pass” – Importing Basic log into Premium submission. – Importing Desktop log into Mobile submission... or vice versa. – Look at REQUIREMENTS: and REPLAYED: REQUIREMENTS: log lines. – Note that the Fidelity requirements are different for Basic/Premium and Desktop/Mobile. – A Premium log can be imported into a Basic submission, but everything else must match. February 26-28, 2008 17 Munich Windows Vista LogoFest

18. “Some devices in this SUT, selected for test, do not appear in the log file” “This log file contains devices that do not exist in the SUT” – This means there is a mismatch, either in the list of PnP IDs or in some device's topology. – Make sure all jacks are plugged in; confirm via mmsys.cpl. – If that doesn't fix it, look at the DEVICE-INFO: and REPLAYED: DEVICE-INFO: lines - compare the PnP IDs. – If those all match, look at the Topology hashes and find the ones that don't match; then look at the Topology: lines and break them out. February 26-28, 2008 18 Munich Windows Vista LogoFest

19.  “This log file contains devices that do not exist in the SUT”  Compare DEVICE-INFO lines first:  DEVICE-INFO:  HDAUDIO\FUNC_01&VEN_VVVV&DEV_DDDD&SUBSYS_SSSSSSSS&REV_RRRR  Topology SHA-1 hash: f0a5d91d8a1497db91147431472a7535e5e1853  REPLAYED: DEVICE-INFO:  HDAUDIO\FUNC_01&VEN_VVVV&DEV_DDDD&SUBSYS_SSSSSSSS&REV_RRRR  Topology SHA-1 hash: 62d23834359a8c8842bdc7e468ed4b4f88591  The PnP IDs have been sanitized, but they matched (trust me.) The Topology hashes, on the other hand, don't! February 26-28, 2008 19 Munich Windows Vista LogoFest

20.  Topology in SUT Topology: HD Audio CD in capture mixer (Pins: Filter, CD Audio; Nodes: CD Volume, CD Mute; Connections: CD Audio => CD Volume(1), CD Mute(0) => Filter, CD Volume(0) => CD Mute(1); ); HD Audio CD in capture (Pins: Recording Control, Filter; Nodes: Unnamed; Connections: Filter => Unnamed(1), Unnamed(0) => Recording Control; ); HD Audio Headphone mixer (Pins: Filter, Headphones; Nodes: Speakers, Master Mute; Connections: Filter => Speakers(1), Master Mute(0) => Headphones, Speakers(0) => Master Mute(1); ); HD Audio Headphone (Pins: PC Speaker, Filter; Nodes: HD Audio line out; Connections: PC Speaker => HD Audio line out(1), HD Audio line out(0) => Filter; ); HD Audio Speaker mixer (Pins: Filter, Speakers; Nodes: Speakers, Master Mute; Connections: Filter => Speakers(1), Master Mute(0) => Speakers, Speakers(0) => Master Mute(1); ); HD Audio Microphone mixer (Pins: Filter, Microphone; Nodes: Mic Volume, Mic Mute, Microphone Boost; Connections: Microphone => Mic Volume(1), Microphone Boost(0) => Filter, Mic Mute(0) => Microphone Boost(1), Mic Volume(0) => Mic Mute(1); ); HD Audio Microphone (Pins: Recording Control, Filter; Nodes: Unnamed; Connections: Filter => Unnamed(1), Unnamed(0) => Recording Control; ); HD Audio Digital in mixer (Pins: Filter, Digital Input Device (SPDIF); Nodes: ; Connections: Digital Input Device (SPDIF) => Filter; ); HD Audio Digital in (Pins: Filter, Filter; Nodes: SPDIF Interface; Connections: Filter => SPDIF Interface(1), SPDIF Interface(0) => Filter; ); HD Audio SPDIF out mixer (Pins: Filter, Digital Output Device (SPDIF); Nodes: Master Mute; Connections: Filter => Master Mute(1), Master Mute(0) => Digital Output Device (SPDIF); ); HD Audio SPDIF out (Pins: Filter, Filter; Nodes: SPDIF Interface; Connections: Filter => SPDIF Interface(1), SPDIF Interface(0) => Filter; ); Topology from file REPLAYED: Topology: HD Audio CD in capture mixer (Pins: Filter, CD Audio; Nodes: CD Volume, CD Mute; Connections: CD Audio => CD Volume(1), CD Mute(0) => Filter, CD Volume(0) => CD Mute(1); ); HD Audio CD in capture (Pins: Recording Control, Filter; Nodes: Unnamed; Connections: Filter => Unnamed(1), Unnamed(0) => Recording Control; ); HD Audio Headphone mixer (Pins: Filter, Headphones; Nodes: Speakers, Master Mute; Connections: Filter => Speakers(1), Master Mute(0) => Headphones, Speakers(0) => Master Mute(1); ); HD Audio Headphone (Pins: PC Speaker, Filter; Nodes: HD Audio line out; Connections: PC Speaker => HD Audio line out(1), HD Audio line out(0) => Filter; ); HD Audio Speaker mixer (Pins: Filter, Speakers; Nodes: Speakers, Master Mute; Connections: Filter => Speakers(1), Master Mute(0) => Speakers, Speakers(0) => Master Mute(1); ); HD Audio Speaker (Pins: PC Speaker, Filter; Nodes: HD Audio line out; Connections: PC Speaker => HD Audio line out(1), HD Audio line out(0) => Filter; ); HD Audio Microphone mixer (Pins: Filter, Microphone; Nodes: Mic Volume, Mic Mute, Microphone Boost; Connections: Microphone => Mic Volume(1), Microphone Boost(0) => Filter, Mic Mute(0) => Microphone Boost(1), Mic Volume(0) => Mic Mute(1); ); HD Audio Microphone (Pins: Recording Control, Filter; Nodes: Unnamed; Connections: Filter => Unnamed(1), Unnamed(0) => Recording Control; ); HD Audio Digital in mixer (Pins: Filter, Digital Input Device (SPDIF); Nodes: ; Connections: Digital Input Device (SPDIF) => Filter; ); HD Audio Digital in (Pins: Filter, Filter; Nodes: SPDIF Interface; Connections: Filter => SPDIF Interface(1), SPDIF Interface(0) => Filter; ); HD Audio SPDIF out mixer (Pins: Filter, Digital Output Device (SPDIF); Nodes: Master Mute; Connections: Filter => Master Mute(1), Master Mute(0) => Digital Output Device (SPDIF); ); HD Audio SPDIF out (Pins: Filter, Filter; Nodes: SPDIF Interface; Connections: Filter => SPDIF Interface(1), SPDIF Interface(0) => Filter; ); February 26-28, 2008 20 Munich Windows Vista LogoFest

21. Each line contains several filters. Each filter looks like: – Filter name (Pins:...; Nodes:...; Connections:...; ); Each filter corresponds to a KSStudio block. Recommended legible layout: Filter name ( Pins: Pin 1, Pin 2,...; (these are the things on the outside of the block) Nodes: Node 1, Node 2,...; (these are the things on the inside of the block) Connections: (Nodes have two sides, Node(0) and Node(1); Pins have one side) Pin 1 => Node 1(1), Node 1(0) => Node 2(0),...; ); Parse both topologies into the legible layout, then use a "diff" tool to see what changed (various tools available). February 26-28, 2008 21 Munich Windows Vista LogoFest

22.  Topology in SUT HD Audio CD in capture mixer ( Pins: Filter, CD Audio; Nodes: CD Volume, CD Mute; Connections: CD Audio => CD Volume(1), CD Mute(0) => Filter, CD Volume(0) => CD Mute(1); ); HD Audio CD in capture ( Pins: Recording Control, Filter; Nodes: Unnamed; Connections: Filter => Unnamed(1), Unnamed(0) => Recording Control; );... (etc.)  Topology from file HD Audio CD in capture mixer ( Pins: Filter, CD Audio; Nodes: CD Volume, CD Mute; Connections: CD Audio => CD Volume(1), CD Mute(0) => Filter, CD Volume(0) => CD Mute(1); ); HD Audio CD in capture ( Pins: Recording Control, Filter; Nodes: Unnamed; Connections: Filter => Unnamed(1), Unnamed(0) => Recording Control; );... (etc.) February 26-28, 2008 22 Munich Windows Vista LogoFest

23. The following filter was exposed when the exported log file was created, but is not exposed in the system under test: HD Audio Speaker ( Pins: PC Speaker, Filter; Nodes: HD Audio line out; Connections: PC Speaker => HD Audio line out(1), HD Audio line out(0) => Filter; ); Diagnosis: there's something wrong with the “Speaker” endpoint, which is actually a line out. Checking mmsys.cpl confirms the diagnosis; the “Speaker” endpoint is listed as “Not plugged in.” February 26-28, 2008 23 Munich Windows Vista LogoFest

24.  Our vision:  Users have high-quality audio experience with PC entertainment and communication involving audio.  Applicable scenarios:  VoIP application, speech recognition, and dictation.  No buzzing and clicking with running graphic intensive programs, etc.  Requirements:  There are no new requirements; that is, requirements for our new tests have always been in place.  SYSFUND-0018 (1-Jun-2006) - Premium system audio solution delivers a high-fidelity audio experience.  SYSFUND-0127 (1-Jun-2006) - System audio record and playback meet Basic performance requirements for audio experience.  AUDIO-0006 (1-Jun-2007) - Audio solution delivers a Premium high- fidelity audio experience. February 26-28, 2008 24 Munich Windows Vista LogoFest

25.  Test cases:  Input Level Test  Dynamic Range Test  THD+N Test  Frequency Response Test  Skew Test  Running tests  No new hardware is required.  No new jobs or UI in DTM; just run the same jobs you were running before, and new test cases will be run:  Fidelity Test  HD Audio Class Driver Fidelity Test  Connect line-in and/or microphone jacks to Audio Precision (AP) Analyzer’s generator module through audio cables. February 26-28, 2008 25 Munich Windows Vista LogoFest

26. February 26-28, 2008 26 Munich Windows Vista LogoFest

27.  Again, no new hardware, job, or UI, same good old job.  Stress tool: RWS.exe:  Stress GPU, disk usage, memory, etc.  Fidelity test measures noise level during these system activity and makes sure that the result complies with our requirements for different programs.  Fidelity Test configures the stress profile at run time and saves the file at %temp%\fidelity.xml.  To reproduce an issue encountered during testing, you can save that file and reconfigure the stress. February 26-28, 2008 27 Munich Windows Vista LogoFest

28. 1 >1 >1 1 1 1 1 1 1 February 26-28, 2008 28 Munich Windows Vista LogoFest

29.  Export and Import for the same release  ALLOWED  Export from 1.2 – Import to 1.1  ALLOWED … as long as 1.1 is not retired.  Export from 1.1 – Import to 1.2  NOT ALLOWED February 26-28, 2008 29 Munich Windows Vista LogoFest

30.  Dialog boxes will appear when the form factor of an endpoint is “Speaker”:  If the endpoint is indeed a line-out, tester should make sure to select to test that endpoint.  Audio tests will now show up in Device Console for a submission if system does not have audio device:  Specifically, this benefits testing Windows Server  2008 systems that do not ship with audio components, so if you don’t have audio device on your system, you don’t need to run Fidelity Test with an imitation AP Host. February 26-28, 2008 30 Munich Windows Vista LogoFest

31.  Audio Fidelity Testing  http://www.microsoft.com/whdc/whql/audiofidelity.mspx http://www.microsoft.com/whdc/whql/audiofidelity.mspx  Windows Logo Kit requirements  http://www.microsoft.com/whdc/winlogo/hwrequirements.mspx http://www.microsoft.com/whdc/winlogo/hwrequirements.mspx  http://www.microsoft.com/whdc/winlogo/wlk/default.mspx http://www.microsoft.com/whdc/winlogo/wlk/default.mspx  Video for running fidelity test locally  Video demo for importing and exporting logs  Windows Driver Kit / Windows Logo Kit  https://connect.microsoft.com/site/sitehome.aspx?SiteID=148 https://connect.microsoft.com/site/sitehome.aspx?SiteID=148  System Error Codes  http://msdn2.microsoft.com/en-us/library/ms681381.aspx http://msdn2.microsoft.com/en-us/library/ms681381.aspx February 26-28, 2008 31 Munich Windows Vista LogoFest

32. © 2008 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.