1. Semantic Parsing for Robot Commands Justin Driemeyer Jeremy Hoffman

2. Purpose STAIR (STanford AI Robot) intelligent office assistant Spoken dialogue, e.g. commands to move “Robot, could you move forward two feet?” “Please go up to my office, room two-oh-four, and wait there.” “Oh, uh, wait, hey robot, stop.” Goal: robust language understanding Extract meaning of utterance from transcription Overcome unexpected words or phrasing, stuttering, errors of the speech recognition component

3. SPEECH RECOGNIZER Audio Input DIALOG MANAGER Text Utterance/ Command Semantics Robot action (e.g., speak, move) Dialogue System SEMANTIC PARSER

4. Semantic PCFG Parser Idea: PCFG parsing with semantic grammar Tag sentences with semantic labels instead of syntactic labels, e.g., “MoveWord,” “Destination,” “ExtraWord” Train our Ass’n 3 PCFG parser from a treebank of hand- labeled utterance transcriptions From a parsed sentence, can read off the important content directly Problem: no robot-command corpus (yet) Instead, automatically generate a fake treebank of tagged sentences We hand-wrote a PCFG for commands to move or control movement Also serves as “probability prior” once we get some real data

5. Frames and Slots In Our Grammar FrameMacroMoveMicroMoveControl SlotsDestination: holds either a room number or a named place (e.g., “front lobby”) that can be converted to a room number from a building map Direction: e.g., forward, back, left, right, around Distance: holds a number and a unit of distance (e.g., feet) CommandWord: e.g., stop, wait, cancel, go, ok, continue We focused on the three commands encoding movement.

6. Command Extraction Parse tree: (ROOT (S (MacroMove (MacroMoveWords (MacroMoveWord go) (MacroMoveWord to)) (Destination (NamedPlace (WordThe the) (NamedPlaceWord lab) (NamedPlaceWord room)))))) Desired Command: (MacroMove (Floor 0) (Room 100))

7. Command Extraction Two parse trees could be different, but encode the same command: Gold Standard: (ROOT (S (MacroMove (RobotAddress (ExtraWords (ExtraWord hey)) (RobotName (RobotNameWord stair)) (PoliteWords (PoliteWord could) (PoliteWord you))) (MacroMoveWords (MacroMoveWord follow) (MacroMoveWord me) (MacroMoveWord to)) (Destination (NamedPlace (NamedPlaceWord my) (NamedPlaceWord office)))))) Extracted: (ROOT (S (MacroMove (RobotAddress (Politeness (PoliteWords (PoliteWord hey))) (RobotName (RobotNameWord stair)) (Politeness (PoliteWords (PoliteWord could) (PoliteWord you) (PoliteWord follow) (PoliteWord me)))) (MacroMoveWords (MacroMoveWord to)) (Destination (NamedPlace (NamedPlaceWord my) (NamedPlaceWord office)))))) Both encode: Command Tree: (MacroMove (Floor 2) (Room 243))