1. 1Seminar „Multimodale Räume“ Uni Karlsruhe, 14.5.2003 The FAME Project Acronym:Facilitating Agent for Multicultural Exchange Partners: Universität Karlsruhe,INPG Grenoble, UJF Grenoble,ITC-irst Trento, UPC Bacelona,SONY Europe, Stuttgart ATLAS Barcelona Project volume: 5.5 M Euro Duration: 40 months, started October 2001 More info:http://www.fame-project.org

2. 2Seminar „Multimodale Räume“ Uni Karlsruhe, 14.5.2003 The FAME Projekt Facilitating Agent for Multicultural Exchange Volume: Duration: 40 months (since October 2001) Financial Volume:  5,5 Mio. € currently approx. 30 scientists Partners: Uni Karlsruhe, INPG Grenoble, UJF Grenoble, UPC Barcelona, ITC-irst Trento, Stuttgart, Barcelona www.fame-project.org

3. 3Seminar „Multimodale Räume“ Uni Karlsruhe, 14.5.2003 Project Goals long term vision : facilitate communication between humans reduce the workload on the users of technical equipment observe humans and their activities in an intelligent room and serve as a context-aware information butler FAME project goal : provide and integrate core technologies (video and speech perception, augmented reality, translation, information retrieval) to show feasibility of the concept demonstrate system at fair scenario 1 (lecture scenario): one person is giving a talk or lecture or presentation scenario 2 (meeting scenario): several people are discussing / working on a common task

4. 4Seminar „Multimodale Räume“ Uni Karlsruhe, 14.5.2003 The FAME Showcases scenario 1 (presentation)scenario 2 (meeting) use A/V equipment intelligent cameraman presentation tracking summarisation + archiving translation, crosslingual IR augmented reality video-based activity tracking topic spotting information butler service: planning of fair visit

5. 5Seminar „Multimodale Räume“ Uni Karlsruhe, 14.5.2003 The FAME Demonstrator (at Barcelona Fair „Forum of Cultures“ 2004) FAME outside viewreception by FAME-guy meeting insidepeople mention topics room gives information about spotted topics

6. 6Seminar „Multimodale Räume“ Uni Karlsruhe, 14.5.2003 The FAME Demonstrator (at Barcelona Fair „Forum of Cultures“ 2004) at the phicon wallgesturesthe projection table multimodal input on table output also on the wall interactive visit planning borrow a camera for photographs of the visit

7. 7Seminar „Multimodale Räume“ Uni Karlsruhe, 14.5.2003 The FAME Demonstrator (at Barcelona Fair „Forum of Cultures“ 2004) Back from the visit in the FAME room: dowload...... and look at photos select, print, save photos using phicon interaction record testimony intelligent cameraman, presentation tracker take home photos and information about FAME

8. 8Seminar „Multimodale Räume“ Uni Karlsruhe, 14.5.2003 multimodal environment context-aware intelligent camera-man automatically track people and their activities augmented reality environment move physical icons (phicons) on table/wall, and interact with projection on table/wall spontaneous speech recognition (with distant microphones) translation and crosslingual information retrieval in European-English, Catalan, and Spanish dialog and context model Important Components

9. 9Seminar „Multimodale Räume“ Uni Karlsruhe, 14.5.2003 Microphon-Array (Speaker Lokalization) Microphon-Array (Speaker Lokalization) Smartboard as Projection Wall Smartboard as Projection Wall Livingroomg IR-Remote Control X-10 Illumination Audio Signals Several Beamers 4 Cameras TV/Video Loudspeakers Microphones Multimodal Environment at UKA

10. 10Seminar „Multimodale Räume“ Uni Karlsruhe, 14.5.2003 Augmented Reality Table project virtual reality on real table move around physical icons (multiple users) interact with projection select, move, rotate, resize, delete, change color write on table, pass notes to others, point to items

11. 11Seminar „Multimodale Räume“ Uni Karlsruhe, 14.5.2003 Intelligent Camera Man follow speaker while talking and moving around detect interaction from audience zoom on area of interest e.g. when pointing somewhere or showing something

12. 12Seminar „Multimodale Räume“ Uni Karlsruhe, 14.5.2003 Lecture Supporter track lecture or presentation operate FAME room equipment by speech commands automatically switch slides during presentation automatically create transcript of lecture create summary, translate to other languages record and store all lectures in searchable database retrieve and browse through previously recorded lectures

13. 13Seminar „Multimodale Räume“ Uni Karlsruhe, 14.5.2003 100 Links wichtige Wörter wichtige Wörter wichtige Wörter TO THE RECOGNITION OF THE CONTINUOUS SPEECH RECOGNITION IN NOISY AND PATTERN RECOGNITION NOT IN SECURING DUE RECOGNITION AND RESPECT CLASS 32 CLASS 14 CLASS 57 CLASS 6 CLASS 70 top n most frequent 40k words least frequent 20k words 60k vocabulary HUB-4 corpus trigramm- model P ( w n | w n -1, w n -2 ) 72 classes P ( C n | w n -1, w n -2 ) · P ( w n | C n ) add to classes estimate presentation- slides important words tf-idf 100 links ± 2 contexts perplexity scores Adaptation Overview 20% fewer errors

14. 14Seminar „Multimodale Räume“ Uni Karlsruhe, 14.5.2003 Welcome in Barcelona in Summer 2004