1. Multi-modal Interfaces
Mats Ljungqvist INFSO E.1 – Interfaces
“The focus of IST in FP6 is on the future generation of technologies in which computers and networks will be integrated into the everyday environment, rendering accessible a multitude of services and applications through easy-to-use human interfaces“
IST WP

2. Our Mission The challenge is:
To make interfaces as simple as possible, and interaction as easy as possible
For citizens to be able to access, receive and use information in their own language
We support the research, development and integration of:
Advanced technologies for interfaces & interaction
Integrated multilingual information systems and services
We focus on:
Technology-mediated interaction and communication between people, between their devices, and between people and content
Combining traditional human language services, like human translation and interpretation, and advanced language technologies and resources
E1 Interfaces

3. Focus so far… Multilingual systems Interaction
unrestricted spontaneous speech-to-speech translation for specific application scenarios
statistical/hybrid (integrating linguistic knowledge) approaches to automatic translation
Interaction
intuitive multi-modal interfaces that are autonomous and capable of learning and adapting to the user environment in dynamically changing contexts
recognise emotive user reaction
robust dialogue capability with unconstrained speech and language input
Human-to-human: technology mediated human communication
Human-to-things: virtual and physical
Human-to-self: health, well-being
Human-to-content: information retrieval/browsing/navigation
Device-to-device: human mediated device-device communication
Human-to-embodied robots: assistance, autonomous, doing-things
Human-to-human: translation, assistive technologies, virtual meetings, telepresence.
Human-to-self: apart from “meditation”, monitoring of human activity and physical and mental state
E1 Interfaces

4. FP6 portfolio 74 millions EURO AMI CHIL TC-STAR TALK T’n D
MULTILINGUALISM
HIWIRE
TALK
DIVINES
SPEECH TECHNOLOGIES
PASCAL
INTERACTION
CHIL
AMI
SIMILAR
HUMAINE
MWEB
MULTIMODALITIES – NEW INNOVATIVE INTERFACES
TAI-CHI
T’n D
MATRIS
ENACTIVE

5. The Facts Instruments: IP’s, STReP’s
Call Title: IST Call 5
Call Identifier: FP IST-5
Strategic Objective: Multi-modal Interfaces
Open: 17 May 2005
Close: 21 September 2005 at (Brussels local time)
Budget: M€
Instruments: IP’s, STReP’s
New/Traditional Instruments: 60/40

6. Main challenges
Need to provide information artefacts to be used by increasingly diverse user groups
Expanding context of use, from scientific and business use to residential and nomadic use
Increasing variety of media and devices used to access a community-wide pool of services and information resources
Demonstrate added-value of multi-modality and performance advances through multi-modal interaction
Go beyond system integration for demonstration purposes and target real world solutions
Emphasise multi-user scenarios
Stronger focus on defining the expected impact
Need to promote cooperation between small European companies
Promote research that could lead to spin-offs

7. An R&D agenda
Key interaction technologies (speech, natural language input, haptics, vision, sensors, gesture, agents, …, including unconventional modalities) and their modelling and integration into interaction platforms
Interface architectures, tools for cost effective integration, e.g. fusion, and toolkit interoperability
Computational models of multi-modality, and advanced corpora for research
Re-develop a serious effort on automatic translation
Reinforce work on robustness and acceptability of speech recognition and research on speaker-independent systems
New learning paradigms in (spoken and written) language translation and portability to new languages, including greater focus on reusable resources
Design processes, methods and tools that support scalability and modality independence
Empirical science base for understanding end-users, e.g. computational models, test suites and benchmarks

8. Call 1 - What happened? Proposals: 68 for a total funding of 456 M€
High quality: 21 for a total funding of 164 M€
Projects: 14 projects for a total funding of ~73 M€
Type: 3 IP’s, 4 NoE’s, 6 STReP’s, 1 SSA
Main features:
Multi-modal interaction, computer mediated communication, and multilingual speech-to-speech systems
Components for tangible interfaces – speech, augmented reality, acoustics, haptics and nomadic devices
NoE’s integrate actors in multi-modal interface research

9. A real need for more product innovation and industrial impact
Future orientations?
Focus more on adaptability, learning capabilities and re-configurable interfaces
Collaborative technologies and interfaces in the arts, including games, design, new media
Less explored modalities, e.g. handwriting, haptics, bio-sensing
Affective computing, including character and facial expression recognition and animation
Innovative aspects of user centred design and ergonomics, and more generally design theory as an enabling discipline, e.g. product design
Multilingualism and machine translation
A real need for more product innovation and industrial impact
Extracted from the Call 1 evaluation, 2004

10. Multi-modal Interfaces
Multilingual communication systems
For unrestricted domains, including task-oriented, real-time understanding of spontaneous spoken and gesture input
Address novel learning paradigms that exploit contextual information, human and linguistic knowledge in a more effective way than currently done
Portability of new languages taking advantage of methods and techniques developed for languages already covered is a further challenge to be addressed, e.g. in the context of new EU languages
IST WP

11. Multi-modal Interfaces
This covers:
User modelling, system design, visual recognition and tracking, language understanding and spoken language translation
Applications should focus on proof of concepts
Includes consumer, nomadic, creative, artistic and gaming applications
IP - system-level objectives in natural interaction and multilingual communication
STReP – can also include work on language understanding and spoken language translation
IST WP

12. Instruments Integrated Project (IP):
Result driven with strong industry/academic consortium and users well integrated
Focus on impact, exploitation, and replication
Multi-faceted with R&D, demonstration, training, awareness building, standardisation, …
8-15 partners, 4+ countries, 3-4 years, 5-10 M€
Targeted Project (STReP):
Problem oriented with the right team for the S&T challenge
Single-problem R&D with high innovation/risk/reward
Investigating new research options, proof of concept, or competitive approaches
4-8 partners, 3+ countries, 2-3 years, 2-4 M€
Human-to-human: translation, assistive technologies, virtual meetings, telepresence.
Human-to-self: apart from “meditation”, monitoring of human activity and physical and mental state
draft

13. The Evaluation Criteria
Relevance:
Clarity – make it a clear yes/no decision
Reference national and European foresight results
Quality of consortium:
Tell them what your principle qualities are – quality of R&D, industrial participation, exploitation potential, SME involvement, uniqueness of consortium, track record, …
Well balanced in terms of expertise, effort and exploitation
All partners must have substantial and well defined roles to play (no window dressing, no hangers on)
Users and end-users are present and have clearly defined roles
Resources:
Clear task allocation and no unnecessary overlaps
Justify costs, effort allocation, and link explicitly to results
Human-to-human: translation, assistive technologies, virtual meetings, telepresence.
Human-to-self: apart from “meditation”, monitoring of human activity and physical and mental state
draft

14. The Evaluation Criteria
Management and organisation:
Clear work-plan and good progress indicators and tracking are key signs
Indicate some valuable intermediate milestones
Indicate how risks will be assessed and what are you back-up options or contingency plans
How decisions are made, and can they be made in a timely way – conflict resolution and consensus building
Good interlinking between objectives, activities and work-packages (Gantt)
Exploitation/dissemination:
Protect and manage IPR and provide access to partners
Start actions early and be precise about options
Go beyond just “publishing results” – make an impact and above all be visible
Human-to-human: translation, assistive technologies, virtual meetings, telepresence.
Human-to-self: apart from “meditation”, monitoring of human activity and physical and mental state
draft

15. The Evaluation Criteria
European added-value:
Show added value to existing national and European programmes
Compliment ongoing efforts of most partners
S&T excellence:
Position the project vis-à-vis wider strategic objectives
Define clear challenging but attainable objectives
Describe state-of-the-art and go beyond (no catch up)
Commit to some key deliverables/impact measures
Justify balance between research and development
Stress visible industry-scale validation through experiments, prototypes, demonstrators…
Show that you can scale to industrialisation
Don’t just be interesting, be manufacturable
Human-to-human: translation, assistive technologies, virtual meetings, telepresence.
Human-to-self: apart from “meditation”, monitoring of human activity and physical and mental state
draft

16. We welcome… STREPs and moderately sized IPs
New participants joining established consortia
Concrete (measurable) project objectives
Tangible and exploitable results
Strengthened software engineering & standards work
Spin-offs: results, products, companies, …
Joint –inter project– evaluation infrastructures (technology- and system-level evaluation)
Human-to-human: translation, assistive technologies, virtual meetings, telepresence.
Human-to-self: apart from “meditation”, monitoring of human activity and physical and mental state

17. Check your ideas with us!
Remember…
Its all about generating, demonstrating and validating new knowledge
Be ambitious, but clearly define your scientific and technological objectives – think 2010+
Be unique, innovate but be industrially relevant - include key industry players and ensure broad industrial impact
Be evaluator friendly, make it a compelling read, easy to understand, a give them a reason to remember it at the end of the week
Be credible, one proposal, no “template” proposals, do what is needed to get the result, not more, but not less – how many person-years of effort is 60 M€??
Check your ideas with us!
Human-to-human: translation, assistive technologies, virtual meetings, telepresence.
Human-to-self: apart from “meditation”, monitoring of human activity and physical and mental state

18. Further information General FP6: http://www.cordis.lu/ IST:
IST:
IST infodesk
fax:
Mats Ljungqvist