1. A New Approach to Decision-making within an Intelligent MultiMedia Distributed Platform Hub Glenn Campbell, Tom Lunney, Aiden Mc Caughey, Paul Mc Kevitt School of Computing and Intelligent Systems Faculty of Engineering University of Ulster, Magee Campus Derry/Londonderry Northern Ireland {Campbell-g8, TF.Lunney, P.McKevitt} @ulster.ac.uk

2. Outline Goals and objectives Key research problems Software and tools Architecture of MediaHub Future development

3. Goals The primary objectives of this research are to: Interpret/generate semantic representations of multimodal input/output Perform fusion and synchronisation of multimodal data (decision-making) Implement and evaluate a multimodal platform hub (MediaHub)

4. Semantic representation? Semantic storage? Communication with other elements of a multimodal platform? Decision-making? Key research problems

5. Software and Tools Main Programming Language Java Communication OpenAIR Specification using Psyclone Decision-making tool HUGIN GUI / API (Hugin 2006) Semantic Representation Likely to be XML-based XML, XHTML + Voice, MPEG-7, OWL, OIL, SMIL, EMMA?

6. Psyclone Semantic Storage via Whiteboards implemented in Psyclone Psyclone uses OpenAIR Specification for communication Publish-Subscribe mechanism for communication Using Java implementation of OpenAIR specification

7. Hugin Tool for implementing Bayesian Networks as CPNs (Causal Probabilistic Networks) Hugin GUI Graphical user interface to Hugin decision engine Hugin API Library implemented in Java Allows programs to implement Bayesian Networks for decision-making

8. Bayesian Networks AKA Bayes nets, Causal Probabilistic Networks (CPNs), Bayesian Belief Networks Consists of nodes and directed edges between nodes Node represents a variable Influence between nodes represented by edges Exercise Weight Loss Diet ‘Diet’ and ‘Exercise’ nodes have influence over ‘Weight Loss’ node

9. MediaHub Example Network

10. Dialogue Manager Facilitates communication between other modules Synchronisation Whiteboard implemented in Psyclone Whiteboard and Dialogue Manager form kernel of MediaHub Decision-Making Module Bayesian Networks Semantic Representation Database Stores semantic representation of multimodal data Architecture of MediaHub

13. Decision-making in MediaHub Input: Determining semantic content of input Fusing semantics of input Resolving ambiguity at input Output: Synchronising multimodal output Best modality for output

14. Conclusion An intelligent multimodal distributed platform hub called MediaHub will be developed MediaHub will interpret/generate semantic representations of multimodal input and output MediaHub will perform fusion and synchronisation of multimodal data MediaHub will provide a new method of decision-making within a distributed platform hub

15. Future development Define necessary decisions Develop Bayesian decision-making using Hugin API for Java Finalise the semantic representation scheme Acquire (or create) multimodal corpora for testing

16. Questions ?