T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Java Video and Audio in Consumer Devices: JMF and MM API Pablo César and J. Luc Lamadon {pcesar, Helsinki University of Technology

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Outline Introduction Java Overview Personal Computer Digital Television Receiver Mobile Phone Conclusions References

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Introduction (1/3) What is multimedia? –Multiple media (e.g., audio, video, images, graphics, text) or multimedia objects –Synchronisation of multimedia elements –Interaction between user and system Multimedia in Consumer Devices, user expectation –Latest mobile phones with camera, MMS, Internet connection, streaming –Digital TV based on audio visual stream + interactive services –PC with 3D games, videoconference, streaming

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Introduction (2/3) Video objects in programming tools: –Synchronized Markup Integration Language (SMIL) –Flash presentations –Java Media Framework (JMF) Stand alone video player: –Proprietary players: Real One, Windows Media Player... –Open Source players: winamp, MPlayer... Audio and Video in Consumer Devices, two alternatives:

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Introduction (3/3) The focus of this presentation is video objects in a programming tool Specifically Java, why? –Consumer devices includes Java Digital TV = Multimedia Home Platform (MHP) and JDK PCs = Java 2 Standard Edition Mobile phones = Connected Limited Device Configuration (CLDC) + Information Device Profile (MIDP) –Independent of Operating System

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Java Overview (1/2)

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Java Overview (2/2) Decodes and plays the media data Retrieves the actual media data Implements the state machine

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Personal Computer (1/4) Services: –Internet, videoconference –Games –… Physical Characteristics: –Pointer device (e.g., mouse) and keyboard as major input mechanisms –Screen resolution: 640x480 to 1600x1200 pixels –Runtime memory: around 128 to 256 MB Java Environment –Java 2 Standard Edition (J2SE) including Abstract Windowing Toolkit (AWT) and Swing

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Personal Computer (2/4) JMF as an optional package either version 1.0 or 2.0 JMF relies on the mentioned classes: Manager, Player, Data Source, and Controller. Player behaviour: –Unrealised: when it does not have all the information to acquire the needed resources –Realised: when it has all the information to acquire the needed resources –Prefetched: when it has all the needed resources, and has already prefetched enough media data to start playing immediately –Started: when it is actually playing the media

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Personal Computer (3/4)

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Personal Computer (4/4) Time model: Clock defines the basic timing and synchronisation operations, which control the presentation of media data (e.g. forward, reward) Data model: DataSource encapsulates both the location and protocol of media JMF 2.0 includes as well a low level API –A Processor performs user-defined processing on the media data using JMF plug-ins (e.g. Codec, Demultiplexer, Effects, Multiplexer, Renderer) Render is done in an AWT Component: –Player.getVisualComponent() –Player.getControlPanelComponent()

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Digital Television Receiver (1/5) Multimedia platform in the living room Services: –Audio visual stream (normal passive watching) –Interactive services (active behaviour) Physical Characteristics –Remote control as major input mechanism –Screen resolution: 720x576 pixels (minimal) –Runtime memory: at least 16 MB Java Environment –MHP

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Digital Television Receiver (2/5) MHP includes JMF 1.0 (obviously, not as an optional package) with some clarifications, restrictions, and extensions The basic classes are again: –Manager, Player, DataSource, and Controller The player behaviour is the same as in JMF 1.0: –Unrealised, Realised, Prefetched, and Started JMF can be used both for broadcast stream and local files (only in high-end receivers) Video drips presents I-frame and consecutive P- frames (high quality interactive graphics)

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Digital Television Receiver (3/5) Locators (reference to a particular piece of media) basically three main types: –javax.media.MediaLocator (and subclasses): media clip –org.davic.net.Locator (and subclasses): DVB transport stream or service –javax.tv.locator.Locator: any type of digital TV system Time based (Clock) is useless in broadcast –Solution to synchronise = stream events A JMF player cannot access directly services in other transport streams (unless explicitly done) Includes new player events (e.g.,): –Media Presented: The media actually starts showing

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Digital Television Receiver (4/5) Includes a number of specific digital TV controls: –Media Select: changes the actual stream presented (e.g., change the angle of a camera) –Language: intended to control the audio and subtitles (if present) language Television display is composed of: –Background + Video + Graphics Layers Video media is, normally, hardware decoded and demultiplexed. So, rendered in video layer (no AWT container) –getVisualContent() returns null –getControlPanelComponent() returns null

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Digital Television Receiver (5/5) Still, video can be controlled: –BackgroundVideoPresentationControl: location and size –VideoFormatControl: picture format and aspect ratio (e.g., 4:3) Maybe in the future (high-end receivers) a real integration of media in graphics layer can be obtained. External audio clips can be played in JMF- MHP

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Mobile Phone (1/5) Mobile phones are becoming powerful Multimedia platforms Provide a number of services: –Internet (e.g., Nokia + Opera), MMS, Video Player –Office capabilities Physical characteristics –Key Pad as major input mechanism –Screen resolution: 84x48 to 120x130 pixels –Runtime memory: 160 to 512 KB Java Environment –Mobile Information Device Profile (MIDP) version 1.0 or 2.0

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Mobile Phone (2/5) Mobile Multimedia API (MMAPI) version 1.0 Latest release was on 26 th June, MMAPI Description: – It extends MIDP functionality by providing audio, video and other time-based multimedia support –It is a thin Java layer completely platform dependent –It is not JMF –It is an optional package MIDP 2.0 includes the audio-only subset from MMAPI (i.e. Audio Building Block)

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Mobile Phone (3/5) Same concepts as JMF: Player, Controller, Manager and DataSource. Player behaviour same (Unrealised, Realised, Prefetched, Started) as JMF but one more state: –Closed: the player cannot be used again, it has released most of the resources Controller is embedded in Player class

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Mobile Phone (4/5)

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Mobile Phone (5/5) The types of media supported depends on the controls associated to the Player –Player.getControls() returns the supported controls –Some examples: VolumeControl, ToneControl… The Player is content and protocol agnostic A Player renders media data in a component dependent on the device configuration, two options: –AWT Component –MIDP Canvas or Items VideoControl manages the location and the size of the video

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Conclusions (1/2) The actual capabilities of the targeted device is the cause of the differences between standards Low level versus high level control of the media: –In MM API and JMF (MHP) the actual control of the media is done at the native level since they are resource-constrained devices (e.g. decoding) –JMF uses two profiles, JMF plug-ins enables developers to process the data (e.g. multiplexing) Behaviour of the player –All the standards have the same player behaviour. But MM API defines a Closed state to make explicit that all the resources are freed

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Conclusions (2/2) MM API is influenced by the design of JMF, and have number of similarities: Manager, Player, Data Source, and Controller concepts. –MM API hides Controller within Player Video render –JMF = AWT Component (complete integration) –MHP = A layer (transparency can be applied) –MM API = MIDP canvas (minimal integration) Controls are different depending on the targeted device (e.g., subtitles language in television) MHP includes specific television requirements (e.g., Clock not needed in broadcast, Locators)

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) References (1/2) Multimedia –P. Vuorimaa, Multimedia Technology Course’s Slides, Helsinki University of Technology, accessed on November 7 th 2003, –S. V. Raghavan, and S. K. Tripathi, Networked Multimedia Systems: Concepts, Architecture, and design, Upper Saddle River (NJ), Prentice Hall, 1998 Digital Television –ETSI TS v1.1.1, Digital Television Broadcasting (DVB): Multimedia Home Platform (MHP) Specification 1.1, ETSI, November 2001 –Steven Morris, The Interactive TV Web, interactive.org, 2002http:// interactive.org

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) References (2/2) PC –R. Gordon, and S. Talley, Essential JMF: Java Media Framework, Upper Saddle River (NJ), Prentice Hall, 1999 –Sun Microsystems, JMF Home Page, accessed on November 7 th 2003, Mobile Phones –Sun Microsystems, J2ME: Java 2 Platform Micro Edition, accessed November 7 th 2003, –Sun Microsystems, CLDC, accessed November 7 th 2003, –Sun Microsystems, Mobile Information Device Profile (MIDP), accessed November 7 th –Sun Microsystems, Mobile Media API (MMAPI), accessed November 7 th 2003,

T Multimedia Seminar Mobile Internet Technology Architecture (MITA) Tietoliikenne- ja multimediatekniikan laboratorio (TML) Thank you! Questions? Comments?