Spatial reasoning in a multi-modal user guide for a complex machine Nadejda Soudzilovskaia, Rafael Bidarra, Frederik W. Jansen Delft University of Technology, Faculty of Information Technology and Systems

Contents n Problem introduction n Application n Geometry based reasoning n Techniques n Examples n Questions

Problem introduction n Multimodal interaction –Multimodal = speech + graphics + text Computer speech needs feedback of text and graphics Speech introduces geometric references, (e.g.the object in front of the building) Solving of these references needs combination of geometric and functional reasoning

? n Application: Interactive fax user guide n Speech & Reasoning & Graphics n Task: develop dialogue technologies Application

n Single questions & answers about the fax n Questions: Speech n Answers: Graphics & Speech How can I make a copy of a document? Put the document into the paper source tray Press copy button Fax encyclopedia

n Pre-programmed answers –develop set of possible user questions –develop answer + “movie” for every question Possible solutions Not an intelligent system. Only pre-cooked question and answers are possible

Possible solutions n Answers derived by reasoning Answers derived by reasoning –knowledge based system containing:knowledge based system containing: data about fax functionality data about fax construction (3D model) –Reasoning system based on:Reasoning system based on: semantic geometry n Answers derived by reasoning Answers derived by reasoning –knowledge based system containing:knowledge based system containing: data about fax functionality data about fax construction (3D model) –Reasoning system based on:Reasoning system based on: semantics geometry

System architecture Dialogue manager Speech recognizer Parser Geometry based reasoning Semantic based reasoning Graphical answer generation Textual answer generation ?

Graphics –Solve geometric references –Derive descriptions of a component location –Show components n Tasks: Geometry based reasoning n Tasks: –Navigation though the virtual world

How can I make a copy of a document? Put the document into the paper source tray Press copy button Geometry based reasoning n Show components

Geometry based reasoning –choose appropriate scale –choose appropriate view point n Show components

Where is the power slot? Power slot is on the back side of the fax Geometry based reasoning n Deriving location descriptions

What is the button left of the start button? This is a copy button Geometry based reasoning n Solve geometric references

n Data about fax construction –hierarchy graph –feature graph –geometry graph Geometry based reasoning Top face Rib Bottom face Corner vertex edge face Telefax Corpus Telephone Power plug Telephone receiver Operational panel LCD screen Start button Keypad....

Problem: search for a suitable reference object Proposal: use of hierarchy Result: reduction of reasoning complexity Techniques Telefax Corpus Telephone Power plug Telephone receiver Operational panel LCD screen Start button Keypad....

n Analyze of dimensionality in view of children location Techniques n Objects that do not have children nodes: Leaf objects n Objects that have children nodes: 3D objects: fax corpus, telephone 2D objects: operational panel, slot panel 1D objects: cables Set objects: group of slots, group of switches

n Use of applicability zone in for a geometric reference when reason about a position on a surface of another object the spatial reference differ for 3D object and 2D object use of fuzzy logic to process zones of applicability Techniques

Where is the power slot? Power slot is on the back side of the fax Examples n Deriving location descriptions

1 priority Location on the surface 1.1 on the corner 1.2 on the right / left side 1.3 on the top / bottom face 1.4 on the front / back face Search for the object parent Select meaningful descriptions 2 priority Location relatively to neighbor objects 2.1 between (two other objects) 2.2 in front of, under, above 2.3 to the left of 2.4 to the left of 2.5 behind, under 2.6. near Developed answer : The printer slot is on the back face of the fax, to the left of the printer slot Deriving location descriptions

What is the button left of the start button? This is a copy button Examples n Solve geometric references

1. Search through the same hierarchy level 1.1 Search for a parent group of operational buttons 1.2 Analyze location left of the start button 1.3. Choose suitable objects 2. Go one level higher in the hierarchy 2.1 Search for a parent of the parent operational panel 2.2 Analyze location left of the start button 2.3 Select objects 3. Go one level higher……... Solve geometric references

Summary n Role of spatial reasoning techniques in the application of multi-modal user guides n A fax machine user guide system n Untilisng 3D model of a fax n Use of hierarchy n Use of dimentionality n Use of areference applicability zone

Conclusions n Geometry based reasoning is a new functionality for such a dialogue system n 3D model of the fax-machine is used as a source of data for reasoning n Several techniques for the geometry-based reasoning were developed and implemented using Java3D and OpenGL languages

Questions ?