I2RP Meeting26 Oct 2005 Temporal Aspects of Intelligent Multimedia Retrieval and Presentation Zhisheng Huang, Jacco van Ossenbruggen, and Lynda Hardman CWI The Netherlands

I2RP Meeting26 Oct 2005 Outline Motivation and Background Temporal approaches for intelligent multimedia retrieval and presentation Temporal reasoning in the Semantic Web Research Plan Past work (by Zhisheng) Demo

I2RP Meeting26 Oct 2005 Time and Multimedia Documents and Presentation Time is one of the most crucial aspects of multimedia documents and presentation. Audio, video, animation clips are presented with time goes, Image archivies are usually displayed with timelines, Text information describes events, time instants, intervals/periods, Presentation involves a series of actions which occur in time.

I2RP Meeting26 Oct 2005 Temporal Aspects and Their Description Time instant: year, date, time clock Time intervals and events: golden age, Monet’s born, Temporal Modifiers: after, before, in, during, Temporal operators: always in the past, sometimes in the past, never in the past, in the future, now, at present Action operators: doing, acting, …

I2RP Meeting26 Oct 2005 Semantics of Time Three Main Approaches: 1.The Quantitative Time Approach: time is a collection of time instants and intervals 2.The Temporal Logic Approach: time fluent is considered as state transitions 3.The Dynamic Logic Approach: time fluent is considered as results of taking actions

I2RP Meeting26 Oct 2005 Temporal Retrieval vs. Temporal Reasoning Temporal retrieval: find entities which satisfy a temporal specification. Temporal reasoning: check if a temporal property holds. Temporal retrieval can be achieved by converting answers with “yes/no” into answers with entities.

I2RP Meeting26 Oct 2005 Quantitative Time Time interval: Show all work which involve Monet in years ; Time instant: Show all Dutch impressionist work in 1926; Time instant with time modifier: Show all Dutch impressionist work after 1926; Event: Show all Dutch impressionist work in year Monet died; Event with time modifier: Show all Dutch impressionist work after Monet died;

I2RP Meeting26 Oct 2005 Temporal Logics Linear time vs. branching time Temporal operators: always-in-the-past, sometimes-in-the-past, never-in-the-past, at-present, always-in-the-future, Until Since

I2RP Meeting26 Oct 2005 Time in Temporal Logics Temporal retrieval: show the first Dutch impressionist work (in this art archive). ( Defined in terms of never-in- the-past operator and the at-present operator) Temporal reasoning: Is it never the case that a Cubist work occurred earlier than the first impressionist work ?

I2RP Meeting26 Oct 2005 Dynamic Logics State vs. Action A state corresponds with a time instant An action is a set of state pairs which correspond with the starting state and the ending state of the action. Dynamic operators: [a]p: after doing action a, the property p always holds; p: after doing action a, the property p may holds;

I2RP Meeting26 Oct 2005 Dynamic Actions Dynamic action operators: sequential actions, parallel actions, action choice, state checking conditional action

I2RP Meeting26 Oct 2005 Time in Dynamic Logics Temporal Presentation: a sequential execution of actions Temporal retrieval: a search action with temporal specification Temporal reasoning: model checking with dynamic actions

I2RP Meeting26 Oct 2005 Temporal Reasoning in the Semantic Web Temporal reasoning with RDF/RDFS data Temporal Ontologies Time in OWL

I2RP Meeting26 Oct 2005 Research Plan Compare the three temporal approaches Investigate their expressibility on intelligent multimedia retrieval and presentation Quantitative time reasoning for image retrieval and presentation Temporal retrieval and reasoning with the support of the Semantic Web technology.

I2RP Meeting26 Oct 2005 Some Past Work (by Zhisheng) Temporal reasoning for multi-version ontologies (2005) PAMELA: an intelligent presentation embodied agent (2004) Facial expressions for embodied agents in STEP (2004) XSTEP: a markup Language for embodied agents (2003) STEP: a scripting language for embodied agents (2002) WASP: Web Agent Support Program (2001)

I2RP Meeting26 Oct 2005 Pamela: Demo