1. German Research Center for Artificial Intelligence DFKI GmbH Stuhlsatzenhausweg 3 66123 Saarbruecken, Germany phone: (+49 681) 302-5252/4162 fax: (+49 681) 302- 5383/5341 e-mail: wahlster@dfki.de WWW:http://www.dfki.de/~wahlster Wolfgang Wahlster ITC-irst, Trento Official Inauguration Wednesday, 26 January 2005 Center for the Evaluation of Language and Communication Technologies COLLATE: The German Competence Center for Language Technology at DFKI

2. © W. Wahlster Saarbrücken Trento TAMIC-P C-STAR CLASS PEACH Cooperation between ITC-IRST and DFKI: A European Success Story

3. © W. Wahlster The International Scientific Advisory Board of DFKI Prof. Dr. Donia Scott, University of Brighton, United Kingdom Prof. Dr. Oliviero Stock, Instituto Trentino Di Cultura, Italia Prof. Dr. Manfred Broy, TU Munich, Germany Prof. Dr. Horst Bunke, University of Bern, Switzerland Prof. Dr. Ronald N. Goldman, Rice University, USA Prof. Dr. Nick Jennings, University of Southampton, United Kingdom Prof. Dr. Alan Bundy, University of Edinburgh, United Kingdom Prof. Dr. Bernd Neumann (Chair), University of Hamburg, Germany

4. © W. Wahlster The European Evaluation Triangle CELCT: Component Evaluation COLLATE: End-to-End Evaluation ELDA: Evaluation Corpora

5. © W. Wahlster Project Structure of COLLATE German Competence Center for Language Technology in Saarbruecken German Competence Center for Language Technology in Saarbruecken R&D Project DFKITransfer Center Internet Services Internet Services Offers for temporal work assignments in the center Offers for temporal work assignments in the center Information & Awareness Events Information & Awareness Events Consulting DFKI, State of Saarland Share Holders Support of Spin-offs Support of Spin-offs Research Resources in Saarbruecken Services for users of LT in industry and public administration Personnel Rooms IT Investments - proactive R&D - accompanying R&D - demonstration - evaluation for transfer Personnel Consumables IT Investments LT Research Resources DFKI LT Research Resources DFKI BMBF R&D Project application-oriented basic research BMBF R&D Project application-oriented basic research

6. © W. Wahlster Important Evaluation Facts for DFKI's Performance DFKIDFKI R&D-Partners 1988-2004 2000-20042000-2004  Economic Impact Demonstrators3007520 Patent Applications12883 Spin-off Products 19014056 New Spin-off Companies382814 New High-Tech Jobs 800550205  Scientific Impact Publications 3.1851.225371 Professorships for DFKI Employees 48296 Academic Degrees (Diploma, Ph.D., Habil.)35615392 National and International Awards/Prizes3122

7. © W. Wahlster DFKI has received the first German Spin-Off Award DFKI has created 800 new jobs in IT industry and 38 start-up and spin-off companies, i.e. each DFKI researcher created 3 new jobs.

8. © W. Wahlster The Language Technology Portal: An Ontology-Driven Global Information System

9. © W. Wahlster COLLATE's Language Technology World – Some Statistics LT World has become the most comprehensive information source of Language Technology Technologies110 Systems907 Experts 2835 News&Events 1085 Projects 850 Organisations 1959 Highlights: HLT Survey, ACL Software Registry, NORFA Exchange, Glossary Patents959

10. © W. Wahlster Usage Statistics of the LT World Portal Per month we now have an average of 51.000 hits and 17.500 page calls The number of unique visitors keeps increasing Month Unique VisitorsVisits Jan 2003 8001500 Dec 2003 11502420 Apr 200415612570 Dec 200420102873

11. © W. Wahlster Vorstellung 5 Prizes in the following categories have been awarded: - The best-practice German dialogue system - The most innovative spoken dialogue application -The system with the best dialogue design -The best speech-based enterprise solution - The best voice-enabled interface to added-value services Chairman of the Prize Committee: W. Wahlster 60 deployed and fully operational spoken dialogue systems have been evaluated by the Prize Committee. 350 attendees

12. © W. Wahlster award 2005 20/21 October 2005, Bonn Second German Voice Award Contract for usability testing: DFKI: COLLATE/CELCT Sponsored by German Telekom Scansoft, Genesys, Voice Objects... - Two thirds of the tested systems provide voice-enabled access to self-service systems (e.g. for banking, brokerage, and retail). - The return of investment (ROI) for these spoken dialogue systems was within 24 months.

13. © W. Wahlster Personnel Director: Prof. Anthony Jameson Full-time researchers: Kerstin Klöckner, Mrudula Sreekanth Frequent participation of master’s students plus other COLLATE staff (10 reserachers) Hardware and software 2 ASL 501 mobile eyetrackers 1 ASL 504 stationary remote eye tracker Flock of Birds head tracker GazeTracker software (for analysis of eye tracker data) Noldus Observer Video Pro portable usability lab SPlus statistical analysis package Web site: www.lt-eval.org The COLLATE Evaluation Center for End-to-End Evaluation at DFKI in Saarbrücken

14. © W. Wahlster New York: Springer 2005 ISBN 0-387-23190-0 Dialog Evaluation Methods developed in Verbmobil and SmartKom have shaped International Evaluation Guidelines

15. © W. Wahlster Outdoor Evaluation of SmartKom Mobile In cooperation with European Media Lab, Heidelberg Subjects performed 3 typical tourist tasks They compared the interaction style of SmartKom Mobile with their preferred known methods Publication: Full paper at IUI 2005

16. © W. Wahlster Relative Ratings of Interaction Style of SmartKom Mobile

17. © W. Wahlster Multitasking with Mobile Multimodal Systems Issue How appropriately do users employ available modalities while concurrently performing nonsystem tasks? Presentation of Results Keynote address at IDS 2002 and related book chapter

18. © W. Wahlster Processing of Summaries in Search Result Pages Issue: To what extent do users look ahead before deciding whether to click on a given search result? Presentation of results: CHI 2004 (short paper) Partly breadth-first strategy:

19. © W. Wahlster Halo 2: The Digital Aristotle sponsored by Vulcan CMU: CMU: natural language understanding U Brighton: U Brighton: intelligent querying with natural language Georgia Tech: understanding diagrams and pictures DFKI: usability & intelligent interfaces Ontoprise: Ontoprise: reasoning, integration, knowledge acquisition iSOCO: PSMs, knowledge acquisition for PSMs The DarkMatter Team

20. © W. Wahlster DarkMatterStudio Querying Semantic Web Query Answer Ask How do amoebas replicate QUERY INTERFACE STUDIO

21. © W. Wahlster Embedded Speech Recognition and Speech Synthesis Tracking of the User‘s Shopping Actions Natural Language Conversation with Products in a Cybershop Most users prefer anthropomorphic interaction Chi 2 (1, N=27) = 3.00, p=0.083 This trend is clearer in men than in women Chi 2 (1, N=12) = 5.22, p=0.021 which is significant

22. © W. Wahlster Multimodal Fusion of Speech, Gesture and Physical Actions (Intra- and Extra-Gestures) Automatic Comparison Shopping with Ambient Intelligence

23. © W. Wahlster © 2005 DFKI Design by R.O. Thank you very much for your attention