1. Building Great Devices Using Windows CE.NET Multimedia

2. Agenda Vision and Goal Vision and Goal Windows CE Multimedia Windows CE Multimedia Talisker Multimedia Features Talisker Multimedia Features WM Audio/Video streaming and codecs WM Audio/Video streaming and codecs Direct3D Direct3D DVD-Video DVD-Video Unified Audio Model Unified Audio Model

3. Vision “Enabling a connected world through a web of Windows Powered devices” “Enabling a connected world through a web of Windows Powered devices” Multimedia Multimedia  It’s one reason why end users want to have a web of connected devices…  …for entertainment, communication, advertising, training, industrial control, medical, etc. …

4. Goal Supply Windows CE devices with best of breed multimedia capabilities Supply Windows CE devices with best of breed multimedia capabilities  Scalable, fully featured, small footprint (customizable), multi-CPU support  Simple, one-stop shopping for advanced multimedia on all devices  Easy for OEMs to use  Standard Windows graphics, audio, video, multimedia APIs, and WM codecs  Simple DirectShow playback  Wireless A/V streaming for mobile devices  VR, game support

5. Challenges Provide the latest and greatest WM, DX, and multimedia features Provide the latest and greatest WM, DX, and multimedia features  Support multiple CPUs/ISAs  Maximum configuration flexibility  Minimum code footprint  Minimum system resource requirements Drive the future Drive the future  Hardware acceleration, SoC  DRM, Security  Mobile QoS

6. Agenda Vision and Goal Vision and Goal Windows CE Multimedia Windows CE Multimedia WM Audio/Video streaming and codecs WM Audio/Video streaming and codecs Direct3D Direct3D DVD-Video DVD-Video Unified Audio Model Unified Audio Model

7. Windows CE Multimedia Technologies Technologies  Audio  2D and 3D Graphics  Video  Codecs  Streaming  DVD  Digital rights management and security  Communication Quality of Service Products Products  MSTV client  MSN Web Companion  Car.Net  WBT  Digital music systems  Multimedia internet appliances  Portable audio/video players

8. Technologies DirectDraw DirectDraw  Drawing surfaces, blting, flipping, overlays, alpha blend, video ports Direct3D Direct3D  Real-time 3D polygon rendering Wave/DirectSound Wave/DirectSound  Buffers, mixing, sample conversion, capture DirectMusic DirectMusic  MIDI, composition DirectShow DirectShow  Media playback DVD-Video DVD-Video  DVD navigation and playback  DVD player app Windows Media Windows Media  Audio/video/speech screen codecs  Streaming  WM Player OCX  WM Player app

9. Multimedia Architecture Graphics, Audio, DVD Drivers DirectShow Filter Graph WM Player ActiveX Control Application DirectDraw/GDI DirectSound/WaveDev Hardware Sample DX Drivers Content/HTML Page Comm Drivers IE browser Filters Codecs Internet or local file Filter Graph Manager Control Interfaces Windows Media Components DVD D3D DirectX Components COM Interfaces

10. File sys Filter Graph Manager Transformfilters RefClock Rendererfilters internet Sourcefilter Media Playback - DirectShow Display / Speakers Filter Graph WM Player ActiveX Control (Run, Pause, Stop, …) DirectDraw/GDI DirectSound/WaveDev Hardware Comm Commands Meta data Codecs

11. What We Have Shipped DXPAK 1.0, RTM 12-15-99 DXPAK 1.0, RTM 12-15-99 DXPAK 1.1, RTM 5-10-00 DXPAK 1.1, RTM 5-10-00  Add-ons to PB 2.12  DXPAK 1.0 - DirectDraw 6,1, DirectSound 6.1, DirectShow 6.1  DXPAK 1.1 - added WMP 6.4 Control, WMT 4.1  CPU’s - X86, MIPS R4300, SH-4  Sample video drivers  IGST Cyberpro 5000  S3 Virge  Sample audio drivers  IGST Cyberpro 5000  Ensoniq 1371 Cedar Add-on Pack, RTM 9-15-00 Cedar Add-on Pack, RTM 9-15-00  DXPAK 1.1 features for PB 3.0  Added new CPUs - StrongArm (SA 1100), Integer MIPS (MIPS R4100)  Added new WMT video codecs Video 7 and ISO MPEG-4 Video 7 and ISO MPEG-4  Sample WM Player application  Localized for all PB 3.0 countries Talisker Beta 1 and Beta 2 Talisker Beta 1 and Beta 2

12. Agenda Vision and Goal Vision and Goal Windows CE Multimedia Windows CE Multimedia Talisker Multimedia Features Talisker Multimedia Features WM Audio/Video streaming and codecs WM Audio/Video streaming and codecs Direct3D Direct3D DVD-Video DVD-Video Unified Audio Model Unified Audio Model

13. Talisker New features New features  Full PB 4.0 CPU support  Real-time 3D graphics (DX8 D3D)  Digital rights management (v7 & PD DRM)  DVD navigation and playback  DirectDraw and audio stack enhancements  New optimized codecs (WM v8 and MP3)  Enhanced WM player and DVD sample apps

14. CPU Support Support all Talisker CPU/BSP Platforms Support all Talisker CPU/BSP Platforms  Optimized WM and MP3 codecs  Direct3D only on FP-enabled CPU’s (x86, MIPS R4300, SH4, PPC603)  No ASM, coprocessor support  DX/Media playback performance will vary according to CPU and system!

15. Agenda Vision and Goal Vision and Goal Windows CE Multimedia Windows CE Multimedia Talisker Multimedia Features Talisker Multimedia Features WM Audio/Video streaming and codecs WM Audio/Video streaming and codecs Direct3D Direct3D DVD-Video DVD-Video Unified Audio Model Unified Audio Model

16. Overview – Windows Media on CE Windows Media functionality in Windows CE provides the ability to playback compressed audio and video content for a variety of scenarios: Windows Media functionality in Windows CE provides the ability to playback compressed audio and video content for a variety of scenarios:  Rich content experience for “Windows Powered” Internet Appliances and mobile devices  Consumer Electronics (DVD players, Internet Radio, WMA/MP3 Jukeboxes, Set-Top Boxes)  Training materials, how-to clips Digital Rights Management enables compelling digital media commerce scenarios Digital Rights Management enables compelling digital media commerce scenarios Over 50% of Internet content is now in Windows Media format Over 50% of Internet content is now in Windows Media format

17. Codecs Video: Video:  Microsoft MPEG-4 v2, v3  Microsoft MPEG-4 ISO v1  Windows Media Video v7, v8 Audio: Audio:  WMA v1, v2, v7, v8  MP3 Note: All supported codecs must be encapsulated in ASF container for streaming Note: All supported codecs must be encapsulated in ASF container for streaming

18. Streaming Protocols Multicast Multicast  Each packet is broadcast to multiple clients  Requires use of Windows Media Station (previously NetShow Channel).NSC files Microsoft Media Stream (MMS) Microsoft Media Stream (MMS)  TCP connection for commands between client and server (defaults to port 0)  UDP (defaults to port 1755) or TCP for media content  Automatic protocol roll-over (UDP, TCP) HTTP HTTP  Will automatically come through corporate firewalls  Will use Internet Explorer’s proxy-settings, or will use own settings File-based (local or network redirector) File-based (local or network redirector)

19. APIs A variety of APIs for controlling Windows Media playback: A variety of APIs for controlling Windows Media playback:  DirectShow  WMP 6.4 OCX hosted in IE4, via JScript or VBScript  WMP 6.4 OCX hosted in VisualBasic, VisualC++ application

20. Windows Media through DirectShow IGraphBuilder *pGraph; IMediaControl *pMediaControl; // Create the filter graph manager CoCreateInstance(CLSID_FilterGraph, NULL, CLSCTX_INPROC, IID_IGraphBuilder, (void **)&pGraph); // Query for IMediaControl (not shown) // Construct the filter graph pGraph->RenderFile(L"\\VideoFile.wmv", NULL); pGraph->RenderFile(L"\\VideoFile.wmv", NULL); // Play the File pMediaControl->Run(); pMediaControl->Run();

21. ID="MediaPlayer" ID="MediaPlayer" CLASSID="CLSID:22d6f312-b0f6-11d0-94ab-0080c74c7e95" CLASSID="CLSID:22d6f312-b0f6-11d0-94ab-0080c74c7e95" TYPE="application/x-oleobject" TYPE="application/x-oleobject" WIDTH="320" WIDTH="320" HEIGHT="240" HEIGHT="240" STYLE="position:absolute; STYLE="position:absolute; left:50px; left:50px; top:50px;“ top:50px;“> </Object> No CODEBASE=url attribute – WMP control must be part of CE device image Make controls visible WMP 6.4 OCX via HTML

22. Digital Rights Management Prevents unauthorized distribution and protects content owners’ rights Prevents unauthorized distribution and protects content owners’ rights Issue/Acquire/Enforce/Manage rights Issue/Acquire/Enforce/Manage rights Allows copyright owners to encrypt ASF content Allows copyright owners to encrypt ASF content  Users must acquire “license” to play  Compressed content cannot be easily emailed, transferred, or copied with associated license Core DRM is “Media Agnostic” Core DRM is “Media Agnostic” Works with ASF v1 supported codecs Works with ASF v1 supported codecs Support for DRM v7 Support for DRM v7

23. Windows CE DRM System DRM system enhancements for CE DRM system enhancements for CE  Secure OS Loader  Disables kernel and application debuggers when DRM module is loaded  Uses trusted model to verify the DRM module is signed  DirectShow security enhancements  All DRM filters are verified, non-DRM filters cannot attach to the DRM filter graph

24. Optimizing Platform Performance Display Drivers Display Drivers  Color-space conversion accounts for 15% of MPEG4 decode, when displaying content in RGB16 format.  Having native YUV (UYVY, YUY2) Overlay Surfaces exposed via a DirectDraw- enabled Display Driver will allow for the MPEG4 / Windows Media Video codec to bypass the YUV -> RGB conversion stage.

25. Optimizing Platform Performance Audio Drivers Audio Drivers  Do not perform volume adjustments for every PCM sample  Use underlying h/w support for attenuation

26. Optimizing for Footprint Windows Media has been componentized for CE, allowing for maximum flexibility during platform configuration. Windows Media has been componentized for CE, allowing for maximum flexibility during platform configuration. Componentized architecture allows selection of only the required components: Codecs, DirectShow, OCX, Player application Componentized architecture allows selection of only the required components: Codecs, DirectShow, OCX, Player application Other sub-components include: Other sub-components include:  Independently selectable streaming formats (MMS, HTTP, multicast, multi-bitrate, and file-based streaming)  Independently selectable codecs (WMA, WMV, MP3)

27. Optimizing for Footprint (cont.) DirectX Media ASF (DXMASF) (x86): DirectX Media ASF (DXMASF) (x86):  ASF, ASX parsing - 135K  File-based streaming - 40K  Network-based streaming - 125K Codec footprint (x86): Codec footprint (x86):  WMV/MPEG4 – 190K  WMA – 90K  MP3 – 50K WM Player (x86): WM Player (x86):  OCX – 275K  App – 155K

28. Unsupported Features ASX v3 format on CE does not include the PREVIEWDURATION, BANNER, and LOGO elements ASX v3 format on CE does not include the PREVIEWDURATION, BANNER, and LOGO elements MPEG-4 v1 (obsolete codec format) MPEG-4 v1 (obsolete codec format) PowerPoint support PowerPoint support JPEG/DJPEG support JPEG/DJPEG support Hotspot ASF authoring Hotspot ASF authoring Encoding Encoding ACELP.net speech codec ACELP.net speech codec

29. Sample WMP Application The CEPlayer.exe sample-app is included in Windows CE.NET The CEPlayer.exe sample-app is included in Windows CE.NET Simple host for the underlying WMP 6.4 OCX Simple host for the underlying WMP 6.4 OCX Easily expandable by OEM for customized look-and-feel, branding, service-based hooks, etc. Easily expandable by OEM for customized look-and-feel, branding, service-based hooks, etc. Provides rich playback experience to end-user, including support for media searching, playlists, and favorites Provides rich playback experience to end-user, including support for media searching, playlists, and favorites

30. CEPlayer Marcus Ash Program Manager Windows CE Multimedia Microsoft Corporation

31. Agenda Vision and Goal Vision and Goal Windows CE Multimedia Windows CE Multimedia Talisker Multimedia Features Talisker Multimedia Features WM Audio/Video streaming and codecs WM Audio/Video streaming and codecs Direct3D Direct3D DVD-Video DVD-Video Unified Audio Model Unified Audio Model

32. Real-time 3D Graphics DX 8 Direct3D API DX 8 Direct3D API  Basic primitive rendering DrawPrimitive() DrawPrimitive() Lit, textured, Gouraud shaded polygon primitives Lit, textured, Gouraud shaded polygon primitives No pixel or vertex shader commands No pixel or vertex shader commands  Hardware acceleration  OEM can optimize driver  Reference Rasterizer  Image compliance

33. D3D Real-time 3D Graphics Fixed function rendering pipeline Fixed function rendering pipeline  Vertex-state machine  Immediate mode commands - no retained scene description  Primitive (triangle) vertex lists and properties  Pipeline Stages  Application  Initialization of display and rendering pipe  Scene traversal  Set rendering state, issue draw commands  Vertex Processing  Vertex transform and lighting (T&L)  Gouraud and fixed shading lighting models  Pixel Processing  Pixel fill  Texture  Hidden surface removal  Fog, Stencil, …  Blending

34. Primitive Ops D3D Fixed Function Rendering Vertex Data Pixel Processing Tex Sample Data D3D Texture Surfaces Vertex Buffers X,YRGBU,V 1U,V 2 Output pixels Tex1 Tex0 Tex2 Vertex Transform and Lighting Vec0Vec1Vec2Vec3 PrimVectorData Multi-pass processing

35. Application Tasks Model scene description Model scene description  Polygon (triangle) primitives and properties Allocate resources Allocate resources  Texture surfaces  Vertex buffers Set up vertex state machine Set up vertex state machine  Vertex processing  Pixel processing Traverse scene description Traverse scene description Issue draw commands Issue draw commands

36. Vertex Pipeline Tasks Transform and lighting (T&L) done in driver/hardware or CPU Transform and lighting (T&L) done in driver/hardware or CPU Vertex processing Vertex processing  Vertex assembly  Vertex lighting calculation  Lighting equation  Primitive assembly  View frustum clipping  Backface cull  Perspective division  Viewport transform

37. Raster Pipeline Tasks Hardware or software rasterization in driver Hardware or software rasterization in driver Pixel processing Pixel processing  Triangle setup  Pixel fill  Iterate colors, texture coordinates, etc  Sample/interpolate textures  Texture/color blend  Stencil buffer compare  Hidden surface elimination  Depth buffer compare (z or w)  Fog blend  Frame-buffer blend

38. D3D Interfaces Interfaces Interfaces  IDirect3D8 – D3D object state  IDirect3DDevice8 – device rendering state, display state  IDirect3DResource8 – create texture and buffer resources  IDirect3DTexture8 – manage texture storage  IDirect3DSurface8 – manage drawing surface  IDirect3DSwapChain8 – manipulate flipping chain  IDirect3DVertexBuffer8 – manage vertex buffer  IDirect3DIndexBuffer8 – manage index buffer

39. D3DX Functions Math classes Math classes  D3DXMATRIX  D3DXVECTOR3  D3DXVECTOR4 Math functions Math functions  D3DXMartrix*  D3DVec3*  D3DVec4* Texture Texture  D3DXCreateTexture*  D3DXLoadSurface*  D3DXFilterTexture  D3DCheckTextureRequirements Macros Macros  D3DXToRadians  D3DXTo Degrees

40. D3D Driver 3D Labs Permedia 3 sample driver 3D Labs Permedia 3 sample driver  Support for GDI, DirectDraw, Direct3D  Uses desktop driver interfaces  Non-DDGPE driver Important optimizations Important optimizations  Do not handle individual vertices  Process complete vertex buffer  Store buffers in video memory  Use DMA whenever possible D3D requires perspectivelly-correct texture mapping D3D requires perspectivelly-correct texture mapping

41. D3D - Not Supported Excludes Excludes  IDirect3DVolumeTexture8  IDirect3DVolume8  IDirect3DCubeTexture8  Vertex and Pixel shaders  Exotic texture and processing modes

42. Direct3D Rod Deyo Program Manager Windows CE Multimedia Microsoft Corporation

43. Agenda Vision and Goal Vision and Goal Windows CE Multimedia Windows CE Multimedia Talisker Multimedia Features Talisker Multimedia Features WM Audio/Video streaming and codecs WM Audio/Video streaming and codecs Direct3D Direct3D DVD-Video DVD-Video Unified Audio Model Unified Audio Model

44. DVD Navigation / Playback Supports DVD-V video disc standard Supports DVD-V video disc standard  DVD navigation manager  Controls presentation engine  Uses playback control info in navigation data stream  Responds to user operations (“Annex J”)  Determines playback conditions, UI, search, menus/buttons, parental control, etc. - VM w/ 16- bit op code  DVD presentation engine  Control of what is displayed – based on Nav data and playback conditions  Uses audio, video, subpicture, etc. info in presentation data stream  MPEG-2/AC-3 video and audio decode and playback

45. DVD System Model Navigation Data Presentation Data DVD Disc Navigation Manager Presentation Engine DVD Player User operations Display

46. Video Title Set Individual Title Program Chain Program VOBU Video Object Unit (VOBU): List of data packs DVD Presentation Data Collection of cells Nav Pack Cell: List of VOBUs Data Packs MPEG-2 Video, Audio, and Subpicture presentation data Packs of data packets Lists of pointers to cells Control, UI, Nav, Search Cmds Information and presentation data

47. DVD Application DVDNavigator Object MetaDriver DataPort Objects Digital Media Driver DVDData Object OEM Extensions IxxxxExtensions : OSD, Zoom, Digest, etc., QI: IDVDRenderer IDVDDecoderCSS IDataPort IBitstreamProcessor IDMAChannel IUnknown IDVDUserOperations, etc. IDVDData, IDVDVolume, etc. IDVDRenderer Registry DVD Disk User Input DVD Data

48. DVD-Video DVD-Video key abstractions DVD-Video key abstractions  DVDData object– base abstraction of DVD nav and presentation data Exports IDVDData, IDVDVolume, etc. Exports IDVDData, IDVDVolume, etc.  DVDNavigator object – Navigation manager Exports IDVDUserOperations (Annex J) Exports IDVDUserOperations (Annex J) Data Port object – DMA streaming interface Data Port object – DMA streaming interface Exports IDataPort Exports IDataPort  Digital Media driver – Presentation engine Exports IDVDRenderer, IBitstreamProcessor, IDMAChannel, IDVDDecoderCSS, OEM extensions Exports IDVDRenderer, IBitstreamProcessor, IDMAChannel, IDVDDecoderCSS, OEM extensions  Meta driver – abstracts Digital Media driver  Driver for the Navigator object  Acquires correct IDVDRenderer from aggregation

49. DVD Driver Model Flexible driver model Flexible driver model  Media APIs extensible by OEM for differentiation  Mode independent - supports both hardware and software implementation  Optimized for multifunctional hardware  COM-based, with late binding through Registry (based on function, not media type)

50. Getting Started with DVD // Create the Data Object and bind to the DVDROM CoCreateInstance(CLSID_DVDData, NULL, CLSCTX_INPROC_SERVER,IID_IDVDROM, (LPVOID *)&pDisk); pDisk->Bind(TEXT("\\CDROM Drive")); // Get the VideoVolume interface and init the volume with the disk data pDisk->QueryInterface(IID_IDVDVideoVolume, (LPVOID *) &pVolume); pVolume->InitData(pDisk); // Create Navigation Manager Object and initialize to the video volume data CoCreateInstance(CLSID_DVDNavigationManager, NULL, CLSCTX_INPROC_SERVER,IID_IDVDNavigationManager, (LPVOID *)&pNavMan); pNavMan->SetVolume(pVolume); // Get Annex J interface, then play title 0 (first play) // NOW YOUR PLAYING DVD pNavMan->QueryInterface(IID_IDVDUserOperation, (LPVOID *) &pAnnexJ); pAnnexJ->TitlePlay(0);

51. DVD Richard Russell Program Manager Windows CE Multimedia Microsoft Corporation

52. Agenda Vision and Goal Vision and Goal Windows CE Multimedia Windows CE Multimedia Talisker Multimedia Features Talisker Multimedia Features WM Audio/Video streaming and codecs WM Audio/Video streaming and codecs Direct3D Direct3D DVD-Video DVD-Video Unified Audio Model Unified Audio Model

53. Audio Features Handles PCM Data Handles PCM Data Capture, playback Capture, playback Playback - static, looping, streaming, mixing, conversion Playback - static, looping, streaming, mixing, conversion WaveIn, WaveOut WaveIn, WaveOut Audio Compression Manager (ACM) Audio Compression Manager (ACM) DirectSound, DirectSoundCapture DirectSound, DirectSoundCapture Mixer API Mixer API DirectMusic DirectMusic

54. Audio Enhancements Unified audio driver model (Wav, Mix, DSHAL) Unified audio driver model (Wav, Mix, DSHAL) New sample audio drivers New sample audio drivers

55. CE 3.0 Audio Wave API (WaveDev) Wave API (WaveDev)  WaveIn, WaveOut  Single stream  Standard Win32 audio API for system and apps DirectSound (DSHAL) DirectSound (DSHAL)  Playback and capture  Multiple streams  Best performance Mixer API (MixDrv) Mixer API (MixDrv)  Connects a UI to “analog” controls  System has no visibility of intermediate data  For volume control

56. CE 2.12 / 3.0 Audio Stack Drivers Device.exe App.exe Audio Hardware DSHAL WaveDev IdsWave WaveMan WaveAPI.lib Coredll WaveBuf/ DSound Client DirectSound Server SW BufferHW Buffer SW Mixer DirectSound Client Audio Application Mixer API Lib Mixer

57. Unified Audio Model Unified Audio Model (UAM) Unified Audio Model (UAM)  New unified audio middle-layer architecture Unified Audio Driver Model (UADM) Unified Audio Driver Model (UADM)  Supports both DSound and Wave with single driver model  Power Management for DirectSound  Simpler audio capture for DirectSound  Multiple drivers/devices/streams  Software mixing and sample rate conversion for Waveform Audio

58. CE 4.0 Audio Stack Drivers Device.exe App.exe Audio Hardware Standard WaveDev Unified SW Mixer WaveMan WaveAPI (Coredll) Audio Device Manager DirectSound Server SW BufferHW Buffer DirectSound Client Audio Application Mixer API (Coredll) DSound ExtensionsMixer Extensions

59. Summary/Call to Action Design multimedia features into your product with an eye towards system capabilities Design multimedia features into your product with an eye towards system capabilities Add multimedia hardware acceleration whenever possible! Add multimedia hardware acceleration whenever possible! Optimize your multimedia drivers!! Optimize your multimedia drivers!! Ask questions on the public/beta newsgroups Ask questions on the public/beta newsgroups Let us know how we can be of help Let us know how we can be of help

60. For more information: www.microsoft.com/windows/ embedded/community Got Feedback? edevfdbk@microsoft.com

61. © 2001 Microsoft Corporation. All rights reserved.