SPACE SPlit Application architeCturE Enabling predictable SW integration Bas Engel Royal Philips / Consumer Electronics Division CELF Embedded Linux Conference 2007 April 17, 2007

Introduction

SW in TV Software size evolution Software partitioning Total SW size follows Moore’s law Philips part decreasing (not just relative) 3rd party involvement increasing

Some rationales on 3rd party SW Cost, however balance NRE and royalties vs. own development cost TTM No time to build up competence in own team “Off the shelf solution available” Critical competence available at supplier side MHEG: move to 1.06 is investment on both sides (creation vs validation) but creates healthy atmoshpere Somebody else makes a business out of supplying common features

Traditional approach: integrate process Philips 3rd party MIPS (Linux 2.x)

Integration nightmare Integration has it’s advantages Optimized resource usage Better viewing experience due to integral architecture approach Integration pitfall High effort due to increased complexity Longer TTM as there is no little change System validation requires global big bang approach No independent lifecycle, no distributed integration cycle

Create a Win-Win atmosphere with 3rd parties The challenge Create a Win-Win atmosphere with 3rd parties We have to create an environment in which 3rd parties can integrate their software Without deep system knowledge We have to enable predictable integration Preventing spaghetti SW We must cater for sharing critical resources AVG Platform General purpose infrastructure 1 Wat zijn de ervaringen (positief en negatief ) met ISVs (tav integratie, effort, communicatie, roadmapping, Market/Business aspect 1394 wrong market assesment  MHP/OCAP don’t initated yet Organizing for buy aspect Gemstar Cabot L1/L2 tipsi…. 1 een soort van do's en don't's, wat kunnen ze wel en wat kunnen ze niet goed, etc) -> kan ook al in Deel 3 WIN WIN atmosphere What when price but especially the how Gemstar NRE discussion at last moment for extra testing Mutual benefit MHEG roadmap is a cabot worry

Design objectives Fast and predictable integration of system extensions Avoid an extensive (re)validation cycle Enable convincing module test opportunities Enable PC based testing Orthogonal 3rd party SW integration Leverage standard Linux infrastructure Multi client usage of platform resources Manage independently deployable building blocks Limited building block correlation Cater for extensions without the need to know all the details Enhanced execution architecture Independent application lifecycle Enable true Multi window architecture For Video and Graphics Preserving application orthogonality

The mental model: SW bolt-ons Feat1 Feat2 Feat3 Feat4 process TV Middleware TV Platform MIPS (Linux 2.x)

SPACE

Introducing SPACE Have orthogonal applications Application Feat1 FeatN TV Middleware Platform API TV platform Application Manager Linux 2.x Have orthogonal applications The resources in the system are explicitly and centrally managed The client applications are system context unaware The lifecycle, focus and visual layout of the client applications is centrally managed

Application Manager Application lifecycle management Starting, stopping applications based on remote control keys Knows the system requirements of all applications Focus handling Determines the active window, reacts to user request Manages the application requirements to the AV resources Layout management Determines how the applications are presented

Managing resources There are implicitly managed resources by the kernel TCP, flash, USB Memory allocation There are explicitly managed resources AV platform resources Memory and CPU resources The explicitly managed resources Have a statically defined execution behavior Can by design limit the parallelism in the system Require an explicitly resource controlled system

Resource Controlled System Application Manager (amApp) Knows the resource dependencies of any client application Statically by design The platform application (plfApp) Has a static model of resource sharing By design resources are multi/single client Every application Must be resource aware Meaning the resource is or is not reserved for them

Multi Client Management tvApp otherApp SetFrequency (tuner) SetFrequency (tuner)   amApp plfApp ResourceOwner (tvApp) Linux 2.x

Resource Groups There are different resource groups offered by the platform Front-end, demux, decode, viewing mode, source selection To allow multiple applications to access different parts of the system Every resource group has a static set of interfaces The Application Manager is the main Resource controller It designates resource groups to clients By informing the platform application whom to grant access

? ? ? The platform API The Philips platform API Leverages as much as possible the current APIs Enables the resource sharing model Is supplier independent AV Interface definition Analog interfaces: Philips’ Analog TV API Digital interfaces: NXP’s Digital TV API AV Interface instance Simple header files and libraries Tool to generate proxy/stub code Graphics interface definition DirectFB ? ? ?

General Connection Management Concept Connection Management is split up in Destination setup Output: FullScreen, PIP Source setup Input: HDMI, Tuner Connection Management is distributed Application Manager is responsible for Client application lifecycle Client applications are responsible for Client applications are destination unaware Application Manager is source unaware

Example tvApp amApp plfApp Linux 2.x 3 SetSource(tuner) 6 Program Freq, PID, … 2 eventDestination(tvApp) 5 eventOnSourceSelected() tvApp 4 SelectTuner amApp plfApp Linux 2.x 1 SetDestinationFullScreen(tvApp)

Border Windows and Video Windows Border windows are linked to video windows Every application controlling video creates a border window Application manager controls visibility of such a couple Visible border windows are equal to the visible video windows There can be multiple border windows for 1 video window The border window determines the position of the video window DirectFB support this via the concept of Input Only Windows Geometry (positioning) is still available to allow focus management No client buffer is required, no scaling is done

Graphics Connection Management tvApp App2 amApp Select App1 Draw configure DirectFB infrastructure source destination

Window Management responsibilities amApp start/kill Layout Size focus tvApp App1 App2 tvApp create draw App1 App2

Traditional DirectFB Window manager Pluggable WM API Move, Resize, Raise, Lower, Hide, Show Windows Process Input, Handle Focus and Grabbing, Dispatch Events Full Cursor implementation including Show, Hide, Reshape, Move Composition of the screen or parts, e.g. triggered by Flip() on a window surface Move, Raise, Show etc. 2D composition Background, Windows, Cursor Default WM module Just follow application requests Basic focus handling App App DirectFB API WM Module SHM

SaWMan Shared Application & Window Manager New DirectFB window manager On its own like default window manager implementation Default is maintained to preserve backwards compatibility Required was a more controlled behavior Add/Remove windows, can modify initial configuration Configuration requests (move, resize, opacity), modify or reject Filter input events before being processed by SaWMan Completely overrule window layout, enable focus borders etc. Application Manager can hook into most of the API flow Application Manager App App App SaWMan API DirectFB API SHM RPC to AM SaWMan Module

Window Management Layers Z-order Clients Message Application Border Focus mgt Layers Z-order Clients Child abducted San Jose Child abducted San Jose Message high Application med Border low Video HW

DirectFB layer mapping on HW If we need fast rendering (>10FPS) and no HW acceleration available We must use direct surface drawing To achieve >25 FPS without SW scaling overhead DirectFB must allow Layers to be mixed across HW Surfaces Some Layers can have SW scaling, others may not Legend Graphics Versatile Layers Z-order Surface Layers Z-order Surface Performance Video Message GFX1 Message GFX1 high high Application GFX1 Application GFX1 med med Border GFX1 Border GFX0 low low Video Video0 Video Video0 HW HW Option 1 Option 2

Window Association Windows can be associated to another Meaning identical position and size Focus only to associated group Focus only to top window, not associated window Example use-case: teletext + menu Teletext and menu of teletext separate windows CreateAssociative Draw Client Draw window Flip time

Pixel Alignment plfApp can crop video Eg to remove DNM border artifacts plfApp can scale video within border window Eg to set the scaling to 4:3 or 16:9 Any client application can indicate ‘pixel alignment’ To assure that the graphics is pixel accurately positioned on top of the video DirectFB caters for the alignment plfApp configures the scale/crop factors Other application is pan/zoom Any application can use the SW scaling Crop & Scale Crop value VideoAligned Crop & Scale

Road to success

Planned extensions to DirectFB SawMan Shared Application Window Manager Enhanced SW up/down scaling for windows With dynamic layer reconfiguration and color key conversion FusionDale & FusionIPC Applied Fusion for Event handling and IPC Memory usage optimization Shared Memory Heap Window Association Link windows together for size and location Window mapping on HW layers Including layer specific configurations Pixel alignment Extended Key handling Sending keys to windows and return key not used Audio Nodes General ID for audio resources Multi processor SPACE Enable multiple control processors to host client applications Published To be published Plan

Main conclusion The integration nightmare can become manageable Using an open standard based multi application approach DirectFB, Linux With a clear orthogonal view No dependencies between applications Adding the necessary components for success SPACE, SaWMan SPACE Is the architectural concept for multi application systems Coping with the resource constraints in CE devices SaWMan Enables application lifecycle management, focus handling, and window management Is fully available via the DirectFB mainline

Demo: SPACE