1. LREC 2006 workshop Genova 28 May 2006 EU-funded research initiatives: - Interactive Systems - New Interfaces - Language Technology Xavier Gros DG INFSO, European Commission, Luxembourg

2. Contents What has been done so far? Projects running so far under “FP6” 2 nd batch of proposals under negotiation “FP7” info What is it? By when? What’s in it for you?

3. Our Mission The challenge is: To make interfaces as simple as possible, and interaction as easy as possible For citizens to be able to access, receive and use information in their own language We support the research, development and integration of: Advanced technologies for interfaces & interaction Integrated multilingual information systems and services We focus on: Natural interaction between humans and physical or virtual environments Multilingual communication systems for unrestricted domains, including real-time understanding of spoken and gesture input E1 Interfaces

4. So what happened? Calls: 1st in late 2002, and 5th in mid-2005 Proposals: 169 requesting a total funding of 804 M€ High quality: 58 asking for a total funding of 296 M€ Projects: 29 projects for a total funding of ~136 M€ (15 start 2006) Participants: ~320 Type: 7 IP’s, 4 NoE’s, 17 STReP’s, 1 SSA

5. 1st Call: http://chil.server.de/ CHIL - Computers In the Human Interaction Loop http://chil.server.de/ http://chil.server.de/ http://www.amiproject.org/ AMI - Augmented Multi-party Interaction http://www.amiproject.org/ http://www.amiproject.org/ http://www.tc-star.org/ TC-STAR - Technology and Corpora for Speech to Speech Translation http://www.tc-star.org/ http://www.tc-star.org/ 5th Call: Augmented Multi-party Interaction with Distance Access Intelligent, Persistent, Personalised Multimodal Interfaces to the Internet Conveying Affectiveness in Leading-edge Living Adaptive Systems Multimodal Interfaces for Capturing and Transferring of Skills Integrated Projects

6. 1st Call: http://www.similar.cc SIMILAR - Federating the communities of human- computer interaction and signal processing http://www.similar.cc http://www.similar.cc http://emotion-research.net HUMAINE - Human-Machine Interaction Network on Emotion http://emotion-research.net http://emotion-research.net http://www.enactivenetwork.org ENACTIVE - Enactive Interfaces http://www.enactivenetwork.org http://www.enactivenetwork.org http://www.w3.org/2004/MWeb/ http://www.w3.org/2004/MWeb/ And an SSA called MWEB - Multimodal Web Interaction http://www.w3.org/2004/MWeb/http://www.w3.org/2004/MWeb/ Networks of Excellence

7. http://www.ist-matris.org/ MATRIS - Markerless Real-time Tracking for Augmented Reality Image Synthesis http://www.ist-matris.org/ http://www.ist-matris.org/ http://www.kaemart.it/touch-and- design/ T’nD - Touch and Design http://www.kaemart.it/touch-and- design/ http://www.kaemart.it/touch-and- design/http://www.kaemart.it/touch-and- design/ http://www.taichi.cf.ac.uk/ TAI-CHI - Tangible Acoustic Interfaces for Computer- Human Interaction http://www.taichi.cf.ac.uk/ http://www.taichi.cf.ac.uk/ http://www.divines-project.org/ DIVINES - Diagnostic and Intrinsic Variabilities in Natural Speech http://www.divines-project.org/ http://www.divines-project.org/ http://www.talk-project.org/ TALK - Talk and Look: Tools for Ambient Linguistic Knowledge http://www.talk-project.org/ http://www.talk-project.org/ http://www.cvsp.cs.ntua.gr/projects/bin/view/HIWIRE/Web Home HIWIRE - Human Input That Works In Real Environments http://www.cvsp.cs.ntua.gr/projects/bin/view/HIWIRE/Web Home http://www.cvsp.cs.ntua.gr/projects/bin/view/HIWIRE/Web Home http://www.cvsp.cs.ntua.gr/projects/bin/view/HIWIRE/Web Home Targeted Research 1st call

8. Statistical Multilingual Analysis for Retrieval and Translation New and more effective statistical approaches to machine translation and cross-lingual textual information access Multi-modal and Multi-Sensor Zero-distraction Interaction Interface Develop a mobile human-computer interface for eye-busy and hand-busy applications in zero- distraction mode Multimodal Interactions Analysis of Users within Controlled Environments Techniques for analysing eye gaze/fixation, eye blink and body movement in human-controlled environment interaction 3D and Speech Interfacing Technologies 3D information visualisation and manipulation based upon the fusion of holographic displays, 3D hand tracking and natural speech processing Sound and Tangible Interfaces for Novel Product Shaping Develop a new type of interface for natural hands-free interaction with virtual shapes Statistical and Hybrid Machine Translation between all European Language Combining statistical techniques with linguistic knowledge in machine translation for all EU language pairs Targeted Research 5th call subject to negotiation

9. OpenInterface OpenInterface is an open source platform for rapid development of multimodal interactive systems OpenInterface is an open source platform for rapid development of multimodal interactive systems Question Answering Learning Technologies in a Multilingual and Multimodal Environment Open domain question-answering (QA) as a new way to search, integrating multilingual open domain QA, context-aware QA, and machine learning technologies Open domain question-answering (QA) as a new way to search, integrating multilingual open domain QA, context-aware QA, and machine learning technologies Distant-talking Interfaces for Control of Interactive TV Develop hands-free voice interaction for use with a interactive-TV remote control Develop hands-free voice interaction for use with a interactive-TV remote control Action Recognition and Tracking based on time-of-flight Sensors Algorithms and prototype for a time-of-flight (ToF) camera that captures 3D information and can track head and eye movement Algorithms and prototype for a time-of-flight (ToF) camera that captures 3D information and can track head and eye movement Spoken Language Understanding in Multilingual Communication Systems New algorithms and tools for robust spoken language understanding in multilingual telephone services New algorithms and tools for robust spoken language understanding in multilingual telephone services Targeted Research 5th call subject to negotiation

10. E1 FP6 call-1 portfolio HIWIRE SIMILAR T’n D MATRISTAI-CHI DIVINES ENACTIVE TC-STAR AMI TALK HUMAINE CHIL MULTILINGUALITY SPEECH TECHNOLOGIES MULTIMODALITY – NEW INNOVATIVE INTERFACES ~73 million EURO INTERACTION IP STREP NoE MWEB SSA PASCAL

11. IST call 5 Closed: 21 September 2005 Budget: 63 M€ Number of proposals: 101 Under negotiation: 15 new projects 4 IP:6-10 MEUR 3-5 years 4 IP:6-10 MEUR 3-5 years 11 STREP:2-3 MEUR2.5-3 years 11 STREP:2-3 MEUR2.5-3 years Projects launch: around September 2006 subject to negotiation

12. FP7 (2007-2013) 4 main Specific Programmes Cooperation Implementation of ICT in the “Cooperation” part to gain leadership in key scientific and technological areas by supporting cooperation between universities, industry, research centres and public authorities across the European Union as well as the rest of the world. Ideas to support the truly creative scientists, engineers and scholars, whose curiosity and thirst for knowledge are most likely to make the unpredictable and spectacular discoveries that can change the course of human understanding and open up new vistas for technological progress and solving enduring social and environmental problems. People to strengthen, qualitatively and quantitatively, human resources in R&D in EU Capacities enhance research and innovation capacity throughout Europe http://cordis.europa.eu.int/ist/about/fp7.htm

13. Societal challenges Business and industry Content, creativity personal development Personal environments Intelligent infrastructures Robotic systems Home environments Nano-electronics, photonics and micro/ nano-systems Communication Networks Embedded systems, Computing and control Software, Grids, Security and dependability Knowledge, cognitive and learning systems Simulation, visualisation Interaction and mixed Realities New perspectives in ICT Drawing on other science and technology disciplines FET Trust and confidence Specific Programme: ICT Collaborative Research – Activities 1st activity: Technology pillars pushing the limits of performance, usability, dependability, cost- efficiency 2nd activity: Integration Environments integrating multi-technology sets that underlie new functionalities, services and applications 3rd activity: Applications research providing the knowledge and the means to develop a wide range of ICT-based services and applications and supporting research at the frontiers of knowledge

14. Draft FP7 Texts FP7 Proposal by Commission to Council & Parliament „BUILDING THE EUROPE OF KNOWLEDGE” FP7 Proposal by Commission to Council & Parliament „BUILDING THE EUROPE OF KNOWLEDGE” Information & Communication Technologies ICT Technology Pillar Simulation, visualisation, interaction and mixed realities: tools for innovative design and creativity in products, services and digital media, and for natural, language-enabled and context-rich interaction and communication. Specific Programmes Specific Programmes Co-operation – ICT Simulation, visualisation, interaction and mixed realities: tools for modelling, simulation, visualisation, interaction, virtual, augmented and mixed reality and their integration in end-to-end environments; tools for innovative design and for creativity in products, services and digital audio- visual media; more natural, intuitive and easy-to-use interfaces and new ways to interact with technology, machines, devices and other artefacts; multilingual and automatic machine translation systems.

15. “Cooperation” – Collaborative Research – Themes Health: 5984 m€ (18%) Food, agri, biotech: 1936 m€ (6%) ICT: 9080 m€ (28.5%) Nano, materials, production: 3467 m€ (11%) Energy: 2236 m€ (7%) Environment: 1886 m€ (6%) Transport: 4150 m€ (13%) Socio-econ research: 607 m€ (2%) Space and security: 2858 m€ (9%)

16. Preparing for FP7 Interactive Systems & New Interface Technologies Interactive Systems & New Interface Technologies 30-31 January 2006, Luxembourg Industrial Workshop on Simulation, Visualisation, Interaction, and Mixed Realities Industrial Workshop on Simulation, Visualisation, Interaction, and Mixed Realities 14-15 February 2006, Luxembourg Machine Translation & Human Language Technologies Machine Translation & Human Language Technologies 23 February 2006, Luxembourg

17. Some pointers … Important issues highlighted in FP7 consultations: Individual interaction modalities need to be made far more accurate and robust. One important application challenge is the lack of robust and cost-effective hands-free and eye-free interfaces (e.g. hand-busy and eye-busy situations). People want interaction and information access in their own language, and we must understand how to develop interfaces that are capable of dealing more efficiently and cost-effectively with different languages. A specific challenge is machine-assisted translation. A major research challenge involves the shift from explicit interaction to implicit human-computer interaction, where the computer adapts to the user, their language and context. In addition, we must develop ways to create adaptive interfaces that enable personalised interaction. The move towards personalised interaction requires us to address ethical and privacy issues. under discussion

18. Some pointers … Important issues highlighted in FP7 consultations: We must be able to understand and model the behaviour of users in the way they interact with machines, with services, with their surrounding environment and with each other. We must demonstrate how effective multimodal interaction helps people and we must answer the question about the benefits that multimodal interaction brings to users. One way forward is to stress-test new interaction modalities in industrial prototypes. Standards and open source software platforms are an essential prerequisite for the rapid adoption of prototype applications. We need common evaluation and benchmarking schemes, test and training data, tools, and metrics. Open competitions are an effective way to mobilise the research community. under discussion

19. Some pointers … Weak integration into real applications Language resources: lack of integration and sharing Speech understanding still difficult MT: lack of robustness, questionable quality No evaluation in specific real-life situations

20. Perspectives Resource sharing, integration of applications, collaboration across scientific domains Resource sharing, integration of applications, collaboration across scientific domains Fully automatic end-to-end MT systems Fully automatic end-to-end MT systems Easy to use and robust MT systems Easy to use and robust MT systems Text and speech recognition Text and speech recognition Linguistic infrastructure Linguistic infrastructure

21. Perspective 1: Machine Translation Fully automatic end-to-end systems and semi-automatic machine translation systems (spoken and written languages) to support a human translator/interpreter. Automatic multilingual speech-recognition toolkits. Initial objective is “high quality translation” of (semantically) domain-limited but otherwise (syntactically) unrestricted languages. Fully automatic end-to-end systems and semi-automatic machine translation systems (spoken and written languages) to support a human translator/interpreter. Automatic multilingual speech-recognition toolkits. Initial objective is “high quality translation” of (semantically) domain-limited but otherwise (syntactically) unrestricted languages.

22. Perspective 2: Robustness Easy to use machine assisted translation tools and portable and robust translation devices for text-to- text and speech-to-speech translation across multiple languages. Effective affordable, reusable systems. Interfaces and systems for on-line translation of web pages. Easy to use machine assisted translation tools and portable and robust translation devices for text-to- text and speech-to-speech translation across multiple languages. Effective affordable, reusable systems. Interfaces and systems for on-line translation of web pages.

23. Perspective 3: Resources Shared databases corpora and free distribution among research non-profit institutes Shared databases corpora and free distribution among research non-profit institutes Open source toolkits and modules Open source toolkits and modules

24. General FP6: http://europa.eu.int/comm/research/fp6/IST:http://www.cordis.lu/istFP7:http://www.cordis.lu/ist/about/fp7.htm Xavier Gros e-mail: xavier.gros@cec.eu.int tel:+352 4301 37193 tel:+352 4301 37193 Further information