Magee Campus 1 Imagineering Creative Technologies Professor Paul Mc Kevitt Intelligent Systems Research Centre School.

http://www.infm.ulst.ac.uk/~paul/ Magee Campus 1 Imagineering Creative Technologies Professor Paul Mc Kevitt Intelligent Systems Research Centre School of Computing & Intelligent Systems Faculty of Computing & Engineering University of Ulster, Magee Derry/Londonderry Northern Ireland

http://www.infm.ulst.ac.uk/~paul/ Magee Campus 2 Background Intelligent MultiMedia & MultiModal Systems Integration of natural language, speech/audio, graphics/VR, vision Semantic representation, fusion & synchronisation Affective computing Integration & reuse of existing software (Thórisson, 2007) Relations with cognitive science Deployment on mobile devices (Dell Axim, Smartphones, Nintendo DS) Applications in arts: music, drama, dance, design, computer games, film, TV/radio, art, storytelling

http://www.infm.ulst.ac.uk/~paul/ Magee Campus 3

http://www.infm.ulst.ac.uk/~paul/ Magee Campus 9 CHAMELEON: a platform for developing intelligent multimedia applications Tom Brøndsted, Paul Dalsgaard, Lars Bo Larsen, Mike Manthey, Paul Mc Kevitt (CPK, Aalborg University, Denmark) Speech input/output, image processing & laser IntelliMedia Workbench campus information system application Semantic representation with frames (Minsky, 1975) Blackboard for semantic representation Bielefeld DACS distributed communications platform (Fink et al., 1996) Implemented with C, C++, DACS, Infovox TTS, grapHvite Applications in multimodal systems development

http://www.infm.ulst.ac.uk/~paul/ Magee Campus Architecture of CHAMELEON 10

http://www.infm.ulst.ac.uk/~paul/ Magee Campus IntelliMedia Workbench 11

http://www.infm.ulst.ac.uk/~paul/ Magee Campus CHAMELEON Semantic Representation Semantic Frames (Minsky, 1975) 12 Input frame: [SPEECH-RECOGNISER UTTERANCE: (Point to Hanne's office) INTENTION: instruction! TIME:timestamp] Output frame: [LASER INTENTION: description (pointing) LOCATION: coordinates (5, 2) TIME: timestamp] Integration frame: [NLP INTENTION: description (pointing) LOCATION: office (tenant Hanne) (coordinates (5, 2)) UTTERANCE: (This is Hanne's office) TIME: timestamp]

http://www.infm.ulst.ac.uk/~paul/ Magee Campus CONFUCIUS: automatic conversion of natural language to 3D animation Minhua (Eunice) Ma, Paul Mc Kevitt Lexical Visual Semantic Representation (LVSR) Use of Jackendoff’s (1990) Lexical Conceptual Structure (LCS) & Schank’s (1977) scripts Use of Fellbaum’s (1998) semantic classes in WordNet Implementation with Java, VRML, WordNet, Connexor parser, FreeTTS (Java) Applications in education, computer games, animation production, digital storytelling 13

http://www.infm.ulst.ac.uk/~paul/ Magee Campus Architecture of CONFUCIUS 14

http://www.infm.ulst.ac.uk/~paul/ Magee Campus MultiModal semantic representation 15

http://www.infm.ulst.ac.uk/~paul/ Magee Campus CONFUCIUS examples 16 Contact verb example (e.g. hit, collide, scratch, touch) “The car collided with a wall.” Collision detection Nonspeech sound effects No human role involved Human action verb examples “Ipke gave Brigitte a loaf of bread.” “Bernie put a cup of coffee on the table.” H-Anim Site node locative tags of object (on tag for table object) “Paul pushed the door.” “Gert ate the bread.”

http://www.infm.ulst.ac.uk/~paul/ Magee Campus 17 “Paul sat on the chair.” “Teri was waving goodbye.” “Helge went to the gym.” “Gert put on his hat.” States “Anke is sad.” “Anke is ashamed.”

http://www.infm.ulst.ac.uk/~paul/ Magee Campus SceneMaker: automatic visualisation of screenplays Eva Hanser, Joan Condell, Tom Lunney, Paul Mc Kevitt Automatically generates affective virtual scenes from screenplays/play scripts Realistic visualisation of emotional aspects Extraction of genre/moods from screenplays Use of OCC (Ortony-Clore-Collins) affective model, ConceptNet, Open Mind Common Sense (OMCS), WordNet-Affect Enhance believability of virtual actors and scene presentation Multimodal representation with 3D animation, speech, audio & cinematography Applications in assisting directors, actors, drama students, scriptwriters 19

http://www.infm.ulst.ac.uk/~paul/ Magee Campus Architecture of SceneMaker 20

http://www.infm.ulst.ac.uk/~paul/ Magee Campus MemoryLane: a mobile digital storytelling companion Sheila Mc Carthy, Heather Sayers, Mike McTear, Paul Mc Kevitt Multimodal stories (speech, text, video) from life-cached memory data (poems, songs, photos, home videos) Extensive HCI requirements analysis with focus on older people Multimodal preferences and abilities of users determines story modalities and user interface configuration Prudent on sensitive life-cached data Creates new stories on the fly Applications in medical informatics and digital care Implemented with Microsoft Visual Studio Deployed on Dell Axim X51v PDA 21

http://www.infm.ulst.ac.uk/~paul/ Magee Campus Architecture of MemoryLane 22

http://www.infm.ulst.ac.uk/~paul/ Magee Campus MemoryLane Prototype 23

http://www.infm.ulst.ac.uk/~paul/ Magee Campus AmbiLearn: an ambient intelligent multimodal learning environment for children Jennifer Hyndman, Tom Lunney, Paul Mc Kevitt Educational games & virtual learning environments Child - Computer Interaction (ChiCI) AmbiLearn architecture & TreasureLearn application Implementation with DevKit Pro (Nintendo homebrew) Deployment on mobile device (Nintendo DS) Application to collaborative learning for children 24

http://www.infm.ulst.ac.uk/~paul/ Magee Campus Architecture of AmbiLearn 25

http://www.infm.ulst.ac.uk/~paul/ Magee Campus PlayPhysics: an emotional games learning environment for teaching Physics Karla Munoz Esquivel, Tom Lunney, Paul Mc Kevitt Virtual Learning Environment (VLE) for teaching Physics Incorporating serious games & intelligent tutoring systems (ITS) Modelling of emotions & personality of student Use of Bayesian Networks for modelling student knowledge Affective feedback through audio-visual effects On-line web application with Student, Lecturer, Head of Department, Administrator views Implemented with Elvira Bayesian Network tool, Flash, Java, Tomcat Apache Server, MySQL database, ConceptNet, 3D Studio Max & Maya Applications in educational software & computer games 26

http://www.infm.ulst.ac.uk/~paul/ Magee Campus Olympia architecture 27

http://www.infm.ulst.ac.uk/~paul/ Magee Campus PlayPhysics interface (astronaut game) 28

http://www.infm.ulst.ac.uk/~paul/ Magee Campus TeleMorph & TeleTuras: bandwidth determined mobile multimodal presentation Tony Solon, Kevin Curran, Paul Mc Kevitt Multimedia limitations of mobile networks, mobile device capabilities User preferences on modalities Transmoding between modalities with cross-modality adaptation Use of Fuzzy (logic) Inference system (FIS) TeleMorph platform & TeleTuras tourist application Implemented with MATLAB Applications in streaming media, computer networks, multimodal presentation 29

http://www.infm.ulst.ac.uk/~paul/ Magee Campus 30 Architecture of TeleMorph Fuzzy Inference System (FIS)

http://www.infm.ulst.ac.uk/~paul/ Magee Campus 31 TeleTuras GUI

http://www.infm.ulst.ac.uk/~paul/ Magee Campus 32 Available media qualities in 6 sample test scenarios

http://www.infm.ulst.ac.uk/~paul/ Magee Campus Adaptation trace data from test scenario 6 33

http://www.infm.ulst.ac.uk/~paul/ Magee Campus MediaHub: Bayesian decision-making in an intelligent multimedia distributed platform hub Glenn Campbell, Tom Lunney, Aiden McCaughey, Paul Mc Kevitt Interpret/generate semantic representations of multimodal input/output Fusion & synchronisation of multimodal data Decision-making with Bayesian Causal Probabilistic Networks (CPNs) Implemented with Psyclone platform, HuginLite Bayesian modelling tool, Java, NetBeans IDE Applications in decision-making for key problems in multimodal systems: anaphora resolution, domain knowledge awareness, multimodal presentation, turn-taking, dialogue-act recognition & parametric learning 34

http://www.infm.ulst.ac.uk/~paul/ Magee Campus Architecture of MediaHub 35

http://www.infm.ulst.ac.uk/~paul/ Magee Campus Decision-making layers 36

http://www.infm.ulst.ac.uk/~paul/ Magee Campus psyProbe view of MediaHub Whiteboard 37

http://www.infm.ulst.ac.uk/~paul/ Magee Campus Alternative `Turn-taking' Bayesian network in Hugin 38

http://www.infm.ulst.ac.uk/~paul/ Magee Campus SoFI: SOng Form Intelligence for repairing streaming music across wireless bursty networks Jonathan Doherty, Kevin Curran, Paul Mc Kevitt Large packet loss dropouts of 5 - 15 s. Use audio pattern recognition & matching to repair gaps K-means clustering classifier determines self-similarity matches Match current sections of song received with previous sections played Typicality of Western Tonal Format (WTF) Use of MPEG-7 Audio Spectrum Envelope (ASE) multimedia content description standard giving features for extraction Implemented with G-streamer for steaming media, Ices2, Icecast2 for audio broadcast & MPEG-7 Audio Spectrum Envelope (ASE) Applications in repair of streaming music 39

http://www.infm.ulst.ac.uk/~paul/ Magee Campus Architecture of SoFI 40

http://www.infm.ulst.ac.uk/~paul/ Magee Campus Spectrogram representation 41

http://www.infm.ulst.ac.uk/~paul/ Magee Campus Steganoflage: strengthening steganography in digital images Abbas Cheddad, Joan Condell, Kevin Curran, Paul Mc Kevitt Hiding/embedding data in transmission medium Use of 1D hash algorIthm coupled with 2D iFFT (irreversible Fast Fourier Transform) Encryption of digital documents in 2D spatial domain Use of steganography to add security layer resistant to noise& JPEG compression Exploitation of Jarvis kernel (Jarvis et al., 1976) to generate error diffusion signal Use of Wavelet-based Inverse Halftoning via De-convolution (WInHD) to recover approximate original signal Implemented with MATLAB & PHP Applications in security, forensics, forgery, digital watermarking 44

http://www.infm.ulst.ac.uk/~paul/ Magee Campus Architecture of Steganoflage 45

http://www.infm.ulst.ac.uk/~paul/ Magee Campus Securing confidential data 46 The traditional method of securing individuals’ data with the notion of security by obscurity that Steganography introduces Application

http://www.infm.ulst.ac.uk/~paul/ Magee Campus National & international collaboration InstitutoTecnológico de Estudios Superiores de Monterrey, Campus Ciudad de México (ITESM-CCM), Mexico Aalborg University, Denmark CADIA, Reykjavik University, Iceland University of Bielefeld, Germany LIMSI (CNRS), Orsay, Paris, France Sheffield University, England Stanford University, California, USA 47

http://www.infm.ulst.ac.uk/~paul/ Magee Campus 50 Noel Sharkey thinks AI is an illusion

http://www.infm.ulst.ac.uk/~paul/ Magee Campus Intellectural Property (IP) Steganoflage: United Kingdom Patent Application [No. 0819982.0]: “Method for Skin Tone Detection”, University of Ulster, Abbas Cheddad, Joan Condell, Kevin Curran, Paul Mc Kevitt. SoFI: United Kingdom Patent Application [No. 0908879.0]: “A system and method for streaming music repair and error concealment”, University of Ulster, Jonathan Doherty, Kevin Curran, Paul Mc Kevitt. 52

http://www.infm.ulst.ac.uk/~paul/ Magee Campus Industrial collaboration InterTradeIreland Fusion project, 2008/10: Nvolve Ltd., Letterkenny, Co. Donegal, Ireland, Michael McNeill, School of Computing & Information Engineering, Coleraine; Prof. Paul Mc Kevitt. Enterprise Ireland Innovation Voucher, IV-2008 1047, 2008/09: ``Navigatour'', Backseat Tour Guides Ltd., Letterkenny, Co. Donegal, Ireland, Anton Hutton, School of Creative Arts & EyeSpyFX; Prof. Paul Mc Kevitt. 53

http://www.infm.ulst.ac.uk/~paul/ Magee Campus University of Ulster, Office of Innovation, Proof-of-Principle (PoP), 2008/09: ``Identity cards employing steganography``, Abbas Cheddad, Joan Condell, Kevin Curran, Paul Mc Kevitt. University of Ulster, Office of Innovation, Proof-of-Principle (PoP), 2009: ``System and method for streaming music repair and error concealment``, Jonathan Doherty, Kevin Curran, Paul Mc Kevitt. Invest NI Proof-of-Concept (PoC) project, 2009/10: ``Secure digital watermarks (SDW): secure data hiding in image frames using steganography``, Joan Condell, Kevin Curran, Paul Mc Kevitt. 54

http://www.infm.ulst.ac.uk/~paul/ Magee Campus Creative People 55 Abbas Cheddad Conrad Deighan Jonathan Doherty Eoghan Furey Eva Hanser Jennifer Hyndman Sheila McCarthy Karla Munoz-Esquivel Dr. Glenn Campbell (Nvolve Ltd., Letterkenny) Dr. Minhua (Eunice) Ma (University of Derby) Dr. Tony Solon (Bath) Dr. Joan Condell Dr. Kevin Curran Anton Hutton (Creative Arts & EyeSpyFX) Dr. Tom Lunney Aiden McCaughey Prof. Paul Mc Kevitt Dr. Michael McNeill (Coleraine) Prof. Mike McTear (Jordanstown) Dr. Heather Sayers

http://www.infm.ulst.ac.uk/~paul/ Magee Campus Conclusion & future work Imagineering creative technologies MultiModal semantic representation Affective computing Deployment on mobile devices Applications in creative industries 56

Magee Campus 1 Imagineering Creative Technologies Professor Paul Mc Kevitt Intelligent Systems Research Centre School.

Similar presentations

Presentation on theme: "Magee Campus 1 Imagineering Creative Technologies Professor Paul Mc Kevitt Intelligent Systems Research Centre School."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Magee Campus 1 Imagineering Creative Technologies Professor Paul Mc Kevitt Intelligent Systems Research Centre School.

Similar presentations

Presentation on theme: "Magee Campus 1 Imagineering Creative Technologies Professor Paul Mc Kevitt Intelligent Systems Research Centre School."— Presentation transcript:

Similar presentations

About project

Feedback