project I 2 RP Intelligent Information Retrieval and Presentation in public historical multimedia databases prof. dr. L. Schomaker KI/RuG

2 ToKeN2000  grants for research between computer science, AI and cognitive science  money from Min. of Econ. affairs and Min. of Education  demonstrating that the ‘human perspective’ has an added value  demonstrating that working systems and/or models can be implemented

3 ToKeN2000 Project Title EIDETIC Intelligent Content-based Image Retrieval I2RP Intelligent IR and Presentation in public historical multimedia databases DUMPERS Distributed User Modeling and Exploration in Personalized Recommender Systems CHIME Cultural Heritage in an Interactive Multimedia Environment AUTHENTIC Knowledge discovery and disclosure for visual art: authentication and dating of graphic art and paintings ANITA Administrative Normative Information Transaction Agents VINDIT Combining visual and textual information for IR MIA Medical Information Agent DIME Distributed Interactive Medical Exploratory for 3D Medical Images TIMEBAYES Building and Using Temporal Bayesian Models in a CPR setting NARRATOR Narrative disclosure of health-care knowledge

4 I 2 RP partners  CWI  Universiteit Leiden  Universiteit Maastricht  Rijksuniversiteit Groningen  Rijksmuseum Amsterdam

5 prof. L. HardmanCWI/TUE prof. dr. H.J. van den HerikUM prof. dr. G.A.M. KempenUL prof. dr. L.R.B. SchomakerRUG dr. I. Sprinkhuizen-KuyperUM dr. J. van OssenbruggenCWI dr. N. TaatgenRUG Supervisors + Rijksmuseum: dhr. K. Schoemaker

6 Researchers drs. Stefano BocconiOIO CWI dr. Floris Wiesmanpostdoc IKAT drs. J. GrobOIO RUG drs. C. van BreugelUL + M.Sc. students

7 Intelligent Information Retrieval and Presentation Information Retrieval: searching in weakly organized multimedial databases Presentation: user and context-related rendering of retrieved results “Intelligent”, i.e., making use of methods from AI and Cognitive Science

 Upper-left picture is the query  “boy in yellow raincoat”  …yields very counter-intuitive results   What was the user’s intention?

9 Human-machine communication  Grice’s Maxims of bi-directional cooperative dialog:  quantity (adapt the size of your answer)  quality (tell the useful truth)  relation (react to what has been asked)  manner (avoid ambiguities)  Current HMC violates most of these maxims

10 Starting points in I 2 RP  Bidirectional cooperative dialog (Grice) (maxims of quantity, quality, relation, manner)  An example of ‘intelligent information retrieval and presentation’: car sales Buyer: “I’m looking for a Volvo 850 Estate for less than 5000 Euro”

11 Starting points in I 2 RP  Bidirectional cooperative dialog (Grice) (maxims of quantity, quality, relation, manner)  An example of ‘intelligent information retrieval and presentation’: car sales Buyer: “I’m looking for a Volvo 850 Estate for less than 5000 Euro” Seller: “we don’t have it” (logical response)

12 Starting points in I 2 RP  Bidirectional cooperative dialog (Grice) (maxims of quantity, quality, relation, manner)  An example of ‘intelligent information retrieval and presentation’: car sales Buyer: “I’m looking for a Volvo 850 Estate for less than 5000 Euro” Seller: “we don’t have it” (logical response) vs Seller: “we do have a Mitsubishi Station of 5500 Euro” (intelligent response)

13 Reasoning with world knowledge (1) Volvo 850 Estate(3) Mitsubishi Station (2) family car! all cars sports carsSUVs

14 Knowledge sources in I 2 RP  A bi-directional cooperative dialog (Grice)…  Requires: world knowledge  semantic web, ontologies knowledge on humans  user modeling, language

15 Project Partners  Optima: A user agent for object-based image search  Spreekbuis: A Dutch sentence generator  Cuypers: Automatic user-centric hypermedia generation  GO: Graphical Ontologies

16 Spreekbuis: a sentence generator for Dutch  UL (C. van Breugel/Arsenijevic)  Performance Grammar Workbench (PGW)

17 Optima: a user agent for object-based image search  KI/RuG, Taatgen/Grob/Schomaker  User modeling, learning in ACT-R KI RuG

18 Cuypers: user-centered hypermedia generator  CWI  Stefano Bocconi, AIO per  using knowledge on graphical design and communication in the application domain

19 GO: Graphical Ontologies  IKAT/UM (Floris Wiesman)  ‘Generic tool for searching (navigating), accessing, and editing ontologies’  MetaBrowser: a graphical browser for information retrieval

20 Goal of the meeting  a lot of mono-disciplinary research exists  … based on toy problems or artificial data (TREC, multimedia retrieval benchmark dBs)  … barely looking at the user requirements  I 2 RP  we can do it better!

21 System: application/experimentation Rendering Semantics User Modeling Speech/Language Multimedia retrieval application

22 System: application/experimentation Multimedia retrieval application dB UI Optima/ACT-R GOCuypers Spreekbuis

23 Dependencies Rendering Semantics User Modeling Speech/Language dB UI Optima/ACT-R GOCuypers Spreekbuis

24 Agenda  Group introduction  Bilateral discussions  Integration  Concrete goals: define  Milestones  Experimentation-platform specification  Demonstrable output

25 Agenda bilateral 20-min. discussions  Room C001  UM + RuG  UL + RuG  UM + UL  Room C002  UL + CWI  UM + CWI  CWI + RuG