1. MediaHub: An Intelligent MultiMedia Distributed Platform Hub
Glenn Campbell, Tom Lunney, Paul Mc Kevitt
School of Computing and Intelligent Systems
Faculty of Engineering
University of Ulster, Magee Campus
Northland Road, Derry
{Campbell-g8, TF.Lunney,
PGNET, Liverpool JMU, June 2005

2. PGNET, Liverpool JMU, June 2005
Outline
Goals and objectives
Key research problems
Distributed Processing
Distributed Platforms
Architecture of MediaHub
Tools and future development
PGNET, Liverpool JMU, June 2005

3. PGNET, Liverpool JMU, June 2005
Project goals
The primary objectives of this research are to:
Interpret/generate semantic representations of multimodal input/output
Perform decision-making (fusion and synchronisation) over multimodal data
 Implement MediaHub, a multimodal platform hub
PGNET, Liverpool JMU, June 2005

4. PGNET, Liverpool JMU, June 2005
Project objectives
Focus on following research questions:
Will MediaHub use frames for semantic representation or XML or one of its derivatives?
How will MediaHub communicate with various elements of a platform?
Will MediaHub constitute a blackboard or non-blackboard model?
What mechanism will be implemented for decision-making within MediaHub?
PGNET, Liverpool JMU, June 2005

5. PGNET, Liverpool JMU, June 2005
Key research problems
Semantic Representation
Represent language and vision
Frames or XML?
Semantic Storage
Blackboard model?
Non-blackboard model?
Decision-making
Fusion and synchronisation
AI technique
PGNET, Liverpool JMU, June 2005

6. PGNET, Liverpool JMU, June 2005
Semantic representation
XML (M3L, SmartKom)
<presentationTask>
<presentationGoal>
<inform> <informFocus> <RealizationType>list </RealizationType> </informFocus> </inform>
<abstractPresentationContent>
<discourseTopic> <goal>epg_browse</goal> </discourseTopic>
<informationSearch id="dim24"><tvProgram id="dim23">
<broadcast><timeDeictic id="dim16">now</timeDeictic>
<between> T19:42: T22:00:00</between>
<channel><channel id="dim13"/> </channel>
</broadcast></tvProgram>
</informationSearch>
<result> <event>
<pieceOfInformation>
<tvProgram id="ap_3">
<broadcast> <beginTime> T19:50:00</beginTime>
<endTime> T19:55:00</endTime>
<avMedium> <title>Today’s Stock News</title></avMedium>
<channel>ARD</channel>
</broadcast>……..
</event> </result>
</presentationGoal> </presentationTask>
Frames (CHAMELEON)
[MODULE
INPUT: input
INTENTION: intention-type
TIME: timestamp]
[SPEECH-RECOGNISER
UTTERANCE:(Point to Hanne’s office)
INTENTION: instruction!
[GESTURE
GESTURE: coordinates (3, 2)
INTENTION: pointing
PGNET, Liverpool JMU, June 2005

7. PGNET, Liverpool JMU, June 2005
Semantic storage
Blackboard or Non-blackboard?
High coupling – Blackboard?
Low coupling - distributed architecture?
Communication
Via central blackboard?
Message passing between modules?
PGNET, Liverpool JMU, June 2005

8. Decision-making (fusion & synchronisation)
Rule-based
Potential for Other AI techniques
Fuzzy Logic
Neural Networks
Genetic Algorithms
Bayesian Networks (CPNs)
PGNET, Liverpool JMU, June 2005

9. Distributed processing
PVM (Parallel Virtual Machine)
(Sunderam 1990, Fink et al. 1995)
ICE (Amtrup 1995)
DACS (Fink et al. 1995, 1996)
Open Agent Architecture (OAA)
(Cheyer et al. 1998, OAA 2004)
JATLite (Kristensen 2001, Jeon et al. 2000)
JavaSpaces (Freeman 2004)
CORBA (Vinoski 1993)
PGNET, Liverpool JMU, June 2005

10. Intelligent Multimedia Distributed Platforms
Blackboard Model:
Ymir (Thórisson 1999)
CHAMELEON (Brøndsted et al. 1998, 2001)
Smartkom
(Bühler et al. 2002, Wahlster et al. 2001, SmartKom 2004)
DARBS (Nolle et al. 2001)
DARPA Galaxy Communicator (Bayer et al. 2001)
Psyclone (Psyclone 2004)
Spoken Image/SONAS
(Ó Nualláin et al. 1994, Ó Nualláin & Smith 1994,
Kelleher et al. 2000)
PGNET, Liverpool JMU, June 2005

11. Intelligent Multimedia Distributed Platforms
Non-blackboard Model:
WAXHOLM (Carlson et al. 1996)
AESOPWORLD (Okada 1996)
COLLAGEN (Rich et al. 1997)
INTERACT (Waibel et al. 1996)
Oxygen (Oxygen 2004)
EMBASSI (Kirste 2001, EMBASSI 2004)
MIAMM (MIAMM 2004)
PGNET, Liverpool JMU, June 2005

12. PGNET, Liverpool JMU, June 2005
CHAMELEON
Language & vision integration system
consists of ten modules, mostly programmed in C and C++
DACS communication system used for communication
Blackboard stores semantic representations produced by other modules
Communication between modules achieved by exchanging semantic representations between themselves or blackboard
Semantic representation in form of input, output and integration frames
PGNET, Liverpool JMU, June 2005

13. Architecture of CHAMELEON
PGNET, Liverpool JMU, June 2005

14. PGNET, Liverpool JMU, June 2005
SmartKom
User adaptive interface for human-computer interaction
Mobile
Public
Home/Office
Facilitates speech, gestures and facial expression input
XML-based mark-up language, M3L, used for semantic representation
Distributed multiple blackboard model
PGNET, Liverpool JMU, June 2005

15. Architecture of SmartKom
PGNET, Liverpool JMU, June 2005

16. PGNET, Liverpool JMU, June 2005
Project proposal
Dialogue Manager
Acts as a blackboard module
Facilitates communication between other modules
Synchronisation
Semantic Representation Database
Provides semantic representation of language and vision data
Decision Making Module
AI technique for a unique form of decision-making
Bayesian Networks (CPNs)
Neural Networks, Genetic Algorithms, Fuzzy Logic
PGNET, Liverpool JMU, June 2005

17. Architecture of MediaHub
PGNET, Liverpool JMU, June 2005

18. Comparison of Intelligent MultiMedia Platforms
PGNET, Liverpool JMU, June 2005

19. PGNET, Liverpool JMU, June 2005
Software Analysis
Main Programming Language
Java
C++
Semantic Representation
XML
XHTML + Voice
SMIL
RDF Schema
MPEG-7
Decision Making
HUGIN (Bayesian Networks) (Hugin 2004)
FuzzyJ Toolkit (Fuzzy Logic) (NRC 2004)
- Software tools for developing the server and client sides of TeleMorph (&TeleTuras)
OUTPUT:
(1) SMIL Media Player - multimodal presentation (animations, graphics, text, audio)
(2) TTS engine (synthetic speech output) Insufficient bandwidth - audio/speech
replaced by ordinary text (->TTS). (JSML = JSAPI Markup Language)
(3) Agent - modalities that are the natural for face-to-face human communication
TeleTuras - very useful on an information aid for tourists
(about sites, points of interest, and route planning)
INPUT:
(1) speech recognition - remotely by ViaVoice
(2) Capture text & haptic deixis input (Lo / Hi-level UI API)
(3) Communicate user input data -> Media Analysis
Communicate user prefs > Constraint Info Retrieval
Communicate device status -> Constraint Info Retrieval
SERVER:
(1) XML based combination - semantic representation
(2) reasoning/decision making - Constraints Processor
(3) middleware - integrate modules
PGNET, Liverpool JMU, June 2005

20. PGNET, Liverpool JMU, June 2005
Project Schedule
PGNET, Liverpool JMU, June 2005

21. PGNET, Liverpool JMU, June 2005
Conclusion
An intelligent multimodal distributed platform hub called MediaHub will be developed
MediaHub will interpret and generate semantic representations of multimodal input and output
MediaHub will perform fusion and synchronisation of language and vision data
Unique contribution of MediaHub is to provide a new method of decision making
MediaHub will be tested within an existing multimodal platform (e.g. CONFUCIUS)
PGNET, Liverpool JMU, June 2005