Enabling Natural Interaction Randall Davis, Howard Shrobe Aaron Adler, Christine Alvarado, Mark Foltz, Tracy Hammond, Mike Oltmans, Metin Sezgin,Olga Veselova MIT Artificial Intelligence Lab

Randall Davis Consider This Device...

Randall Davis Natural Interaction? What if we had to deal with one another as we deal with our software?

Randall Davis Let’s Talk

Randall Davis Our Vision

Randall Davis Our Model

Randall Davis Rationale Capture Rationale capture doesn’t happen. –Because it’s too much trouble Rationale capture tools don’t work –Because they’re too much trouble The key: it needs to be less trouble than it’s worth. The key to less trouble: break down the barriers

Randall Davis Sketch Understanding

Randall Davis Why Sketching? Informal Succinct Natural High in information content Preserve topological and spatial properties

Randall Davis Sketch Understanding Requires intelligence –at the signal level: early processing –at the interpretation level: handling ambiguity –at the engineering level: intended behavior

Randall Davis Other Research Additional domain: software diagrams Multi-modal interaction Describing and learning new icons Generating efficient recognizers

Randall Davis Sketching Software

Randall Davis Sketching Software

Randall Davis Sketching Software

Randall Davis Sketching Software

Randall Davis Sketching Software //Source file: c:\\video\\Deck.java public class Deck extends Hand implements Dealable { public Card theCard; public Game theGame; /** 3C21348C0257 */ public Deck() { } //Source file: c:\\video\\Dealable.java public interface Dealable { } //Source file: c:\\video\\Game.java public class Game { public Graphics theGraphics; public Deck theDeck; public Player thePlayer; /** 3C21348C02E3 */ public Game() { }

Randall Davis Multi-Modal Interaction “Three identical, equally spaced pendulums…” Graphical vs verbal communication: What’s best said, what’s best sketched?

Randall Davis New Icons A shape description language A drawing sequence description language Define AndGate line L1 L2 L3 arc A semi-circle A1 orientation(A1, 180) vertical L3 parallel L1 L2 same-horiz-position L1 L2 connected A.p1 L3.p1 connected A.p2 L3.p2 meets L1.p2 L3 meets L2.p2 L3

Randall Davis Sketch Understanding Architecture Mechanical Engineering devices simple compositions primitive objects geometryrectangles, curves

Randall Davis Sketch Understanding Architecture Mechanical Engineering devices simple compositions primitive objectssprings, joints geometryrectangles, curves

Randall Davis Sketch Understanding Architecture Mechanical Engineering devices simple compositionsratchet, latch primitive objectssprings, joints geometryrectangles, curves

Randall Davis Sketch Understanding Architecture Mechanical Engineering devicesmechanisms simple compositionsratchet, latch primitive objectssprings, joints geometryrectangles, curves

Randall Davis Sketch Understanding Architecture MechanicalSoftwareEngineering devicesmechanisms simple compositionsratchet, latch primitive objectssprings, jointsobjects, methods geometryrectangles, curves

Randall Davis Sketch Understanding Architecture MechanicalSoftwareEngineering devicesmechanisms simple compositionsratchet, latchhierarchies primitive objectssprings, jointsobjects, methods geometryrectangles, curves

Randall Davis Sketch Understanding Architecture MechanicalSoftwareEngineering devicesmechanismsmodules simple compositionsratchet, latchhierarchies primitive objectssprings, jointsobjects, methods geometryrectangles, curves

Randall Davis Design Rationale Capture Make it less trouble than it’s worth –Break down the barriers: eliminate interfaces –Natural interaction is enabled by intelligence About sketching, gesturing About the domain About the task