Confidential © 2009 Teleca AB Dual Display for Android Applications Gururaj N 21st Jan, 2011 v0.2

Confidential © 2009 Teleca AB Agenda Introducing Teleca History of Multi Display Dual Display Mobile Devices Teleca’s experience in Dual Display for Android Devices QA 2

Confidential © 2009 Teleca AB 3 Teleca is the only player in segment with comprehensive onshore/offshore global delivery footprint across all major markets We have 2000 mobile software engineers available worldwide Teleca is the only player in segment with comprehensive onshore/offshore global delivery footprint across all major markets We have 2000 mobile software engineers available worldwide Introducing Teleca

Confidential © 2009 Teleca AB Introducing Teleca History of Multi Display Dual Display Mobile Devices Teleca’s experience in Dual Display for Android Devices QA Agenda 4

Confidential © 2009 Teleca AB History of Multi-Monitor (1) – PC and TV 5 Dual Monitors at Work Place Picture in Picture Multi monitor use case  Engineer needs as bigger screen possible to enhance productivity  HW Support – two video ports, or add second video card  OS Support – Adjust display properties PIP use case  Two programs side–by–side and audio output of one program going into speaker and another going to headset  Two tuner built into TV set OR one tuner being external source like DVD player.

Confidential © 2009 Teleca AB History of Multi-Monitor (2) – Mobile Phones 6 Multi display Mobile Phone usecases  The secondary display is used typically for notifications ( messages, incoming call ) and Clock  HW Support – two display drivers  OS Support – Adjust display properties  Separate applications running on two screen Disadvantages  Cannot write applications for secondary display  Only one display is active at any time.  Low resolutions

Confidential © 2009 Teleca AB Introducing Teleca History of Multi Display Dual Display Mobile Devices Teleca’s experience in Dual Display for Android Devices QA Agenda 7

Confidential © 2009 Teleca AB Dual Display Mobile Devices (1) 8 Samsung Continuum Acer Iconia Samsung Continuum Use cases  Dedicated Ticker Display for displaying Music player controls, RSS feeds, Social network site updates, Incoming call notification.  The idea seems to be use the second display for dynamically changing data without affecting the activities user does on main screen Acer Iconia  Concept laptop.  Used as e-Reader  One of the display can become Soft- Keyboard.

Confidential © 2009 Teleca AB Dual Display Mobile Devices (2) 9 HW Support  Two different displays  Two different frame buffers and drivers  One display virtually split into two displays  Single virtual frame buffer to support two independent displays SW Support  Framework changes to support dual displays  SDK changes to program applications specific to Dual display  Currently not available, mostly proprietary

Confidential © 2009 Teleca AB Introducing Teleca History of Multi Display Dual Display Mobile Devices Teleca’s experience in Dual Display for Android Devices QA Agenda 10

Confidential © 2009 Teleca AB Teleca’s experience in Dual Display for Android (1) 11 Teleca Worked on all aspects of Dual Display on Android Teleca Worked on all aspects of Dual Display on Android

Confidential © 2009 Teleca AB Teleca’s experience in Dual Display for Android (2) Teleca built Dual Display for Android with Texas Instruments Demonstrated at MWC 2009 Development Environment  TI Zoom3(OMAP 4430) with Dual Display Support (each display 864x480)  Android 2.0 ( Éclair ) 12

Confidential © 2009 Teleca AB Teleca’s experience in Dual Display for Android (3) - Components Android Components involved in Dual Display  Applications - need to define onto which display their Activities should go.  Activity Manager – launch application’s activities onto the right display.  Window Manager – needs to properly handle two Activity stacks and two window stacks  Surface Flinger – needs to handle layer composition for two displays.  OpenGL / EGL – needs to support one context for each display  Linux Kernel / Driver – support for two secondary display for output and touch input 13

Confidential © 2009 Teleca AB Teleca’s experience in Dual Display for Android (4) - Design Approaches for Window Manager Support Single Window Manager concept  This concept is based on an extension of the existing Window Manager for the second display. The WindowManagerService would be extended to support a second stack of activities and a second stack of windows.  The Activity Manager would –based- on the applications preference or the preference of an Activity create the Activity on the right stack of the Window Manager. Besides the changes to the ActivityManager the main changes need to be done in the WindowManagerService and the IWindowManager.aidl. 14 Pros Interface to only one Window Manager No additional thread Cons Very complex 11K lines of code Massive design changes to central Android component

Confidential © 2009 Teleca AB Teleca’s experience in Dual Display for Android (5) - Design Approaches for Window Manager Support Second Window Manager  The concept proposes to duplicate the WindowManagerService code, rename it and instantiate an instance of this which handles the Activities and Windows for the second display.  All Stake holders need to interface with two Window Managers 15

Confidential © 2009 Teleca AB 16 Pros Simple concept with only minor changes to the actual implementation of the WindowManagerService; Standard WindowManagerService requires nearly no changes Easy to migrate to next version of Android Cons Synchronization of both WindowManagerService instances required in some parts Code duplication Increased memory usage due to a second instance of the service A number of stakeholders of WindowManagerService need to be modified Not too attractive solution from architecture point of view

Confidential © 2009 Teleca AB Teleca’s experience in Dual Display for Android (6) - Design Approaches for Window Manager Support Second Window Manager Service Instance  This concept proposes to modify the WindowManagerService and the IWindowManager.aidl code so that it can be instantiated twice. Each instance of the service would be responsible for one display then. The required changes haven’t been analysed in detail as this seemed to be more effort compared to previous approach. However from an architecture point of view this seems to be preferred.  The problem with this approach might be that the decision which WindowManagerService needs to be used might not always be possible and thus requiring modifications in the stakeholders. As a result the other approach was chosen 17

Confidential © 2009 Teleca AB 18

Confidential © 2009 Teleca AB Teleca’s experience in Dual Display for Android (7) - Design Approaches for Window Manager Support Single Logical Display  This concept proposes a single logical display implemented by the two physical displays. Applications would be able to use both screens.  This approach was not analysed in details but seems to have some drawbacks as well such as:  No real dual display concept  Additional layout problems when landscape and portrait will be support  Complex in case the two displays have different resolution Bitmap based secondary display  This concept is a bitmap-based solution which would create images which are then drawn onto the second screen. This is a very simple solution.  Pros:  Very simple solution from WindowManagerService point of view which might even be acceptable for clamshell device with small and simple UI on outer display  Cons:  Static solution which provides only simulated user input handling  Not usable for animated/rapid changing content 19

Confidential © 2009 Teleca AB Introducing Teleca History of Multi Display Dual Display Mobile Devices Teleca’s experience in Dual Display for Android Devices  Specific Implementation Details QA Agenda 20

Confidential © 2009 Teleca AB Teleca’s experience in Dual Display for Android - High Level Block Diagram 21 Linux Kernel Libraries Application Framework Android Runtime Applications Surface Flinger OpenGL bionic Display Driver 1 Keyboard Driver Pico DLP … Activity Manager PhoneHome Home 2 Gallery App. on second display OpenGL Application … View System Window Manager … Hardware Abstraction Layer Display Driver 2 Touch Input 2 Touch Input 1 Input 1 Window Manager 2 Modified New Not feasible

Confidential © 2009 Teleca AB Teleca’s experience in Dual Display for Android - Changes across the Android Stack Applications  Extensions to the AndroidManifest.xml The AndroidManifest.xml was extended to support the attribute “android:display” for the and the tag. The possible values are integer values, usually only used when the application or an Activity of the application wants to go on the second screen  Sample : 22

Confidential © 2009 Teleca AB  Extension to the applications resources  The interface of the resources was extended to allow an application to set the display programmatically. This is usually done in the onCreate() of an Activity: protected void onCreate(Bundle icicle) { super.onCreate(icicle); getResources().setApplicationDisplay(Display.DISPLAY2); }  The class Resource was extended with two interfaces: public int getApplicationDisplay(); public void setApplicationDisplay(int display);  The display values are defined in Display.java as: Display.DEFAULT_DISPLAY Display.DISPLAY2 23

Confidential © 2009 Teleca AB Activity Manager  The Activity Manager starts an Activity and as such needs to know onto which display an application wants to go. The Activity Manager then adds the Activity onto the Window Manager’s proper Activity stack.  The ActivityManagerService was extended by a number of changes:  Additional list of history entries for the second display which stores the stack of launched activities  Reference to WindowManagerService for the second display  Support to start, stop and resume activities for the second display  Support for separate Home Screen on second display. The second Home Screen itself was not implemented during the project. 24

Confidential © 2009 Teleca AB Window Manager  The Window Manager needs extensions to support two application/Activity stacks and two Window stacks. Additionally it needs to be made second display-aware.  Touch Events  Key Events  Rendering 25

Confidential © 2009 Teleca AB Surface Flinger 26 Linux Kernel Libraries Application Framework Applications Surface Flinger Application 2 Application 1 Application 3 Hardware Abstraction Layer Display Driver 1 /dev/graphics/fb0 Surface Surface Surface Surface Surface Display Driver 2 /dev/graphics/fb1 Clamshell Driver

Confidential © 2009 Teleca AB The standard Surface Flinger was extended two handle two stacks of layers, one for each display. Internally the Layers contain the buffer to be rendered. Additionally all the calls to OpenGL set the right OpenGL context before so that the OpenGL calls go to the right display. From the applications point of view the decision onto which display is drawn is done by the display parameter set in the Surface object. The applications don’t do this directly though. They define this either in the AndroidManifest.xml or by setting the display in the associated resources by a direct API call. The Activity Manager then decides onto which display an Activity wants to go and adds it to the correct Window Manager. The Window Manager then sets the proper display in the Surface object. Access permissions  To access the Surface Flinger the following permission is defined in the platforms AndroidManifest.xml (in frameworks/base/core/res/):  android.permission.ACCESS_SURFACE_FLINGER  The following public constructor of Surface is hidden from the SDK and requires this permission:  public Surface( SurfaceSession s, int pid, int display, int w, int h, int format, int flags) throws OutOfResourcesException; 27

Confidential © 2009 Teleca AB Linux Kernel / Drivers  /dev/graphics/fb0 – Framebuffer for default display  /dev/graphics/fb1 – Framebuffer for second display  /dev/input/event0 – HW keyboard  /dev/input/event1 – Touchable first display  /dev/input/event2 – Touchable second display 28

Confidential © 2009 Teleca AB Introducing Teleca History of Multi Display Dual Display Mobile Devices Teleca’s experience in Dual Display for Android Devices QA Agenda 29

Confidential © 2009 Teleca AB Resourcefully yours