1. SmartSynth: Synthesizing Smartphone Automation Scripts from Natural Language Vu Le (UC Davis) Sumit Gulwani (MSR Redmond) Zhendong Su (UC Davis)

2. motivation

3. programming App Inventor Tasker

4. natural language When I receive a new SMS, if the phone is connected to my car’s bluetooth, it reads the message content and replies the sender “I’m driving.”

5. script when (number, content) := MessageReceived() if (IsConnectedToBTDevice(Car_BT) then Speak(content); SendMessage(number, "I'm driving");

6. SmartScript Script P ::= I E T C M Parameter I ::= input (i 1, …, i n ) | ε Event E ::= (r 1, …, r n ) := when Event() | ε Conversions T ::= F 1 ; …; F n ; Condition C ::= if (Π 1 ∧ … ∧ Π n ) then | ε Body M ::= Stmt 1 ; … ; Stmt n ; Conversion F ::= x := Convert(a) Predicate Π ::= Predicate (a 1, …, a n ) Statement Stmt ::= S | foreach x in a do S1; … ; Sn; od Atomic Stmt. S::= A | F Action A ::= (r 1, …, r n ) := Action (a 1, …, a n ) Argument a ::= x | i | r | l

7. approach Script = Components + Relations

8. approach Script = Components + Relations

9. approach Script = Components + Relations

10. approach Script = Components + Relations

11. definition o Component API: MessageReceived Entity Literal: “I’m driving” API return value: MessageReceived.Text O o Relation = E.g.,

12. component discovery DescriptionComponent When I receive a new SMSMessageReceived if the phone is connected to IsConnectedToBTDevice my car’s bluetoothCar_BT readsSpeak the message contentMessageReceived.Text O replies SendMessage the senderMessageReceived.Number O “I’m driving”"I'm driving"

13. mapping features o Regular expressions o Bag of words o Phrase length o Punctuation o Parse tree (Stanford NLP parser)

14. relation discovery o Rule-based relation detection Relative locations of APIs and entities Return Value or LiteralAPI Parameter Car_BTIsConnectedToBTDevice.Text I MessageReceived.Text O Speak.Text I MessageReceived.Number O SendMessage.Number I “I’m driving”SendMessage.Text I

15. script generation when (number, content) := MessageReceived() if (IsConnectedToBTDevice(Car_BT) then Speak(content); SendMessage(number, "I'm driving");

16. ambiguity DescriptionPossible Component Mappings When I receive a new SMSMessageReceived if the phone is connected to IsConnectedToBTDevice IsConnectedToWifiNetwork IsConnectedToDataService my car’s bluetoothCar_BT readsSpeak the message contentMessageReceived.Text O replies SendMessage SendEmail “I’m driving”"I'm driving"

17. incompletenes s when (number, content) := MessageReceived() if (IsConnectedToBTDevice(Car_BT) then Speak(content); SendMessage(number, "I'm driving");

18. insight NLP + Program Synthesis

19. ambiguity { MessagedReceived, IsConnectedToBTDevice, Car_BT, Speak, MessageReceived.Text O, (SendMessage OR SendEmail), "I'm driving“ } Ranking Script 1 (SendMessage) SendMessag e SendEmails Script 1 Script 2 Synthesizer

20. incompletenes s o Search for most likely missing relations

21. q & a o Distinguishing multiple choice questions Question: API parameter Answers: Equally-likely related entities What do you want the phone to speak? A.The received message content B.“I’m driving”

22. architecture

23. evaluation o 50 scripts collected from user forums o User study: give NL descriptions from scripts 11 students participated Each student was given 25 problems 640 correct descriptions (725 total)

24. component mapping

25. relation detection

26. completing ratio

27. synthesis time

28. related work o General purpose programming using natural language o Natural language interfaces o Specification extraction from natural language

29. thank you Script = Components + Relations NLP Program Synthesis IdentifyDetect ResolveComplet e

30. v.s. Siri Conversation Single phone API Composition of APIs Siri SmartSynth

31. v.s. Tasker