1. 1 Tutorial T4 - Programming by Example Techniques Brad A. Myers Brad A. Myers Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213-3891 (412) 268-5150 bam+@cs.cmu.edu http://www.cs.cmu.edu/~bam Brad A. Myers Brad A. Myers Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Pittsburgh, PA 15213-3891 (412) 268-5150 bam+@cs.cmu.edu http://www.cs.cmu.edu/~bam

2. 2 Contents: n Abstract & Bio n Slides from Talk n Sources for Further Information n Articles n Early VP & PBE Taxonomy n Demonstrational Interfaces paper n Gamut Paper n Intelligence in Demonstrational Interfaces n Abstract & Bio n Slides from Talk n Sources for Further Information n Articles n Early VP & PBE Taxonomy n Demonstrational Interfaces paper n Gamut Paper n Intelligence in Demonstrational Interfaces

3. 3 Abstract This tutorial covers the broad range of techniques used in programming systems that use "Programming By Example (PBE)." Sometimes called "Programming By Demonstration" or "Demonstrational Interfaces", this refers to systems that allow the user to give examples of the desired behaviors and contents, and the system generalizes from those examples so the result will work in different contexts. Some PBE systems use sophisticated artificial intelligence techniques to generalize, others use simple rule-based heuristics, and still others require the user to perform the generalization. Applications of PBE techniques have been to general- purpose programming, programming of user interfaces, defining animations, creating macros or scripts for text editing, designing business charts, desktop file manipulation, hypertext browsing, programming for kids, etc. The tutorial will include many demonstrations and videos of research and commercial uses of programming-by-example techniques.

4. 4 About the Presenter Brad Myers is a Senior Research Scientist in the Human-Computer Interaction Institute in the School of Computer Science at Carnegie Mellon University, where he is the principal investigator for various projects, including Demonstrational Interfaces, User Interface Software, Natural Programming, and the Pebbles Palm Pilot Project. He and his students have created about a dozen systems that use programming-by- example techniques. He is the author or editor of over 190 publications, including "Creating User Interfaces by Demonstration" and "Languages for Developing User Interfaces," and he is on the editorial board of five journals, including the Journal of Visual Languages and Computing. His research interests include User Interface Development Systems, user interfaces, Programming by Example, programming languages for kids, Visual Programming, interaction techniques, window management, and programming environments. He belongs to SIGCHI, ACM, IEEE Computer Society, IEEE, and Computer Professionals for Social Responsibility. He will also be a keynote speaker for VL'2000.

5. 5 Topics n What is Programming by Example? n Applications of Programming by Example n Overview of PBE Systems n Issues in PBE Systems n User Interfaces for giving examples n User Interface for feedback to show the program n Implementation architectures n What is Programming by Example? n Applications of Programming by Example n Overview of PBE Systems n Issues in PBE Systems n User Interfaces for giving examples n User Interface for feedback to show the program n Implementation architectures

6. 6 What is Programming by Example? n Direct Manipulation allows properties to be set by directly moving objects with the mouse and setting properties n Example: interface builders, Visual Basic n Limited to static parts of the interface n No way to point at objects that will be drawn by the user n How set the color of an object in Visual Basic at run time? n Direct Manipulation allows properties to be set by directly moving objects with the mouse and setting properties n Example: interface builders, Visual Basic n Limited to static parts of the interface n No way to point at objects that will be drawn by the user n How set the color of an object in Visual Basic at run time?

7. 7 Demonstrational Tools n Demonstrational Tools allow the user to operate on example objects which represent objects that are created at run-time. n Size of the boxes depends on labels n Demonstrational systems allow the user to operate on examples, and then generalize to produce a general- purpose procedure or prototype. n Demonstrational Tools allow the user to operate on example objects which represent objects that are created at run-time. n Size of the boxes depends on labels n Demonstrational systems allow the user to operate on examples, and then generalize to produce a general- purpose procedure or prototype. VL’2000 Seattle 2000 IEEE

8. 8 "Examples" n Draw an example of the objects that will be created at run time n Draw objects in approximately the right places, and systems creates general constraints n Give examples of the steps of the algorithm n Draw an example of the objects that will be created at run time n Draw objects in approximately the right places, and systems creates general constraints n Give examples of the steps of the algorithm VL’2000 Seattle 2000 Prototype IEEE

9. 9 Generalization n Is a hard problem, which is why you don't see many commercial products that do this. n Research on: n Algorithms for generalization n “Artificial Intelligence” n User Interfaces to help generalization n Is a hard problem, which is why you don't see many commercial products that do this. n Research on: n Algorithms for generalization n “Artificial Intelligence” n User Interfaces to help generalization

10. 10 Motivations n Bring programming to more people n Automate repetitive tasks n Expand how much of the interface can be specified interactively. n Interactive editors are much faster to use than programming with toolkits n Frameworks improve productivity by factors of 3 to 5, interactive tools by factors of 10 to 50! n It might take an hour to draw an interface interactively, compared to days to program it. n It is much more natural to specify the graphical parts of applications using a graphical editor. n Bring programming to more people n Automate repetitive tasks n Expand how much of the interface can be specified interactively. n Interactive editors are much faster to use than programming with toolkits n Frameworks improve productivity by factors of 3 to 5, interactive tools by factors of 10 to 50! n It might take an hour to draw an interface interactively, compared to days to program it. n It is much more natural to specify the graphical parts of applications using a graphical editor.

11. 11 Example in Emacs n Useful to create scripts for repetitive actions: n Rename all gif files to have OLD at the front of their names n Useful to create scripts for repetitive actions: n Rename all gif files to have OLD at the front of their names archesonly.gif circuitafter.gif circuitbefore.gif emacsrecord.gif generalizeloc.gif generalizeobj.gif heightgen.gif invocationdb.gif leftgen.gif moveselection.gif mv archesonly.gif OLDarchesonly.gif mv circuitafter.gif OLDcircuitafter.gif mv circuitbefore.gif OLDcircuitbefore.gif mv emacsrecord.gif OLDemacsrecord.gif mv generalizeloc.gif OLDgeneralizeloc.gif mv generalizeobj.gif OLDgeneralizeobj.gif mv heightgen.gif OLDheightgen.gif mv invocationdb.gif OLDinvocationdb.gif mv leftgen.gif OLDleftgen.gif mv moveselection.gif OLDmoveselection.gif

12. 12 Example in Microsoft Word The fillparagraphs macro changes RETURNs into spaces, unless there are 2 in a row. Lots of short ¶ lines¶ can be filled and ¶ combined. ¶ ¶ But two RETURNS¶ signal a new ¶ paragraph. ¶ Lots of short lines can be filled and combined.¶ But two RETURNS signal a new para- graph.

13. 13 Applications of Programming by Example n General-purpose programming n Programming for kids n Programming of user interfaces n Creating macros or scripts for text editing n Designing business charts n Desktop file manipulation n Hypertext browsing n General-purpose programming n Programming for kids n Programming of user interfaces n Creating macros or scripts for text editing n Designing business charts n Desktop file manipulation n Hypertext browsing

14. 14 General Issues for All PBE Systems n Amount of Inferencing n None -> Lots n Predictability vs. amount of help n If Inferencing: n Number of examples n Negative Examples n Hints or other help from user n Feedback on the Inferencing n Control structures: loops, conditionals n Variables n How show user the resulting program? n Amount of Inferencing n None -> Lots n Predictability vs. amount of help n If Inferencing: n Number of examples n Negative Examples n Hints or other help from user n Feedback on the Inferencing n Control structures: loops, conditionals n Variables n How show user the resulting program?

15. 15 Overview of Some PBE Systems n Chronological n Bias towards my systems n Easier to get pictures / video! n There are many other systems not covered here n See references n Chronological n Bias towards my systems n Easier to get pictures / video! n There are many other systems not covered here n See references

16. 16 Pygmalion n David C. Smith’s PhD thesis n Watch What I Do, chapter 1. n 1975 in Smalltalk n No inferencing n Work out program using icons and example values n Primary contributions: n Programming with icons n Working out program using examples n General motivation n David C. Smith’s PhD thesis n Watch What I Do, chapter 1. n 1975 in Smalltalk n No inferencing n Work out program using icons and example values n Primary contributions: n Programming with icons n Working out program using examples n General motivation

17. 17 Pygmalion Picture

18. 18 Emacs n R. Stallman, 1987 in Teco, Lisp, C, etc. n Easily record a sequence of keystrokes n No inferencing n Very light-weight and extremely useful n R. Stallman, 1987 in Teco, Lisp, C, etc. n Easily record a sequence of keystrokes n No inferencing n Very light-weight and extremely useful

19. 19 Emacs Example ^x( ^a ^k "mv " ^y " OLD" ^y ^n ^x) ^u1000 ^xe ^x( ^a ^k "mv " ^y " OLD" ^y ^n ^x) ^u1000 ^xe archesonly.gif circuitafter.gif circuitbefore.gif … mv archesonly.gif OLDarchesonly.gif mv circuitafter.gif OLDcircuitafter.gif mv circuitbefore.gif OLDcircuitbefore.gif …

20. 20 Emacs Contributions n Primary contributions: n Clearly demonstrate benefits of macros by example n Quick and easy macro creation for “keyboard macros” n Fully programmable using Lisp for further End-User- Programming (EUP) n Primary contributions: n Clearly demonstrate benefits of macros by example n Quick and easy macro creation for “keyboard macros” n Fully programmable using Lisp for further End-User- Programming (EUP)

21. 21 Rehearsal World n William Finzer and Laura Gould, 1984 in Smalltalk n Watch What I Do, chapter 4. n No inferencing n Metaphor of “actors” on a “stage” n Wings for off-screen computation n PBE by opening an “eye” icon n Primary Contributions: n Stage metaphor: on-stage, wings, etc. n Eye metaphor: for easy controlling of when recording n William Finzer and Laura Gould, 1984 in Smalltalk n Watch What I Do, chapter 4. n No inferencing n Metaphor of “actors” on a “stage” n Wings for off-screen computation n PBE by opening an “eye” icon n Primary Contributions: n Stage metaphor: on-stage, wings, etc. n Eye metaphor: for easy controlling of when recording

22. 22 Rehearsal World Pictures

23. 23 SmallStar n Daniel Halbert, 1980-1984, in Smalltalk n Watch What I Do, chapter 5. n Create programs in the Star office system n Desktop (file) manipulations n No inferencing n Introduced notion of “data descriptions” n Specify what data using dialog boxes n Edit straight-line program to add conditionals and loops n video (6 min) n Daniel Halbert, 1980-1984, in Smalltalk n Watch What I Do, chapter 5. n Create programs in the Star office system n Desktop (file) manipulations n No inferencing n Introduced notion of “data descriptions” n Specify what data using dialog boxes n Edit straight-line program to add conditionals and loops n video (6 min)

24. 24 SmallStar Contributions n Primary contributions n Data descriptions concept and UI n UI Techniques for control structures n Demonstration in a complex environment n Primary contributions n Data descriptions concept and UI n UI Techniques for control structures n Demonstration in a complex environment

25. 25 Peridot n Brad Myers, 1987, in Lisp n Watch What I Do, chapter 6. n Create user interface widgets (buttons, scroll bars) n Inferred: n Graphical constraints among the objects. n Control structures such as iteration over all the items in a menu n How the mouse affects the graphics, such as that the check mark should follow the mouse. n Feedback using question and answer n No representation of the program n video (8 min) n Brad Myers, 1987, in Lisp n Watch What I Do, chapter 6. n Create user interface widgets (buttons, scroll bars) n Inferred: n Graphical constraints among the objects. n Control structures such as iteration over all the items in a menu n How the mouse affects the graphics, such as that the check mark should follow the mouse. n Feedback using question and answer n No representation of the program n video (8 min)

26. 26 Peridot Contributions n Primary Contributions n Inferencing using condition-action rules n Question-and-answer feedback technique shown not successful n Demonstrational technique for graphical positions n Coverage of UI behaviors n Primary Contributions n Inferencing using condition-action rules n Question-and-answer feedback technique shown not successful n Demonstrational technique for graphical positions n Coverage of UI behaviors

27. 27 Eager n Allen Cypher, 1990 in HyperCard n Watch What I Do, chapter 9. n Inferred loops from repeated sequence of actions n Had to match actions and look for generalizations of values n “Anticipatory Feedback” of what it thought the user would do next n User can say “do the rest” n No way to know what the step after would be n Users weren’t comfortable without knowing the stopping condition n video (4 min) n Allen Cypher, 1990 in HyperCard n Watch What I Do, chapter 9. n Inferred loops from repeated sequence of actions n Had to match actions and look for generalizations of values n “Anticipatory Feedback” of what it thought the user would do next n User can say “do the rest” n No way to know what the step after would be n Users weren’t comfortable without knowing the stopping condition n video (4 min)

28. 28 Eager Contributions n Primary Contributions n Anticipatory Feedback n Algorithms for matching actions where parameters differ n Showed that people were not comfortable telling it to go without knowing what it would do and when it would stop. n Primary Contributions n Anticipatory Feedback n Algorithms for matching actions where parameters differ n Showed that people were not comfortable telling it to go without knowing what it would do and when it would stop.

29. 29 Garnet n Brad Myers, 1988 - 1992 in Lisp n Watch What I Do, chapter 10. n User interface development environment n Lapidary: Create UI elements, less inferencing n video (2.5 min) n Contributions: n UI for defining graphical constraints n More sophisticated behaviors by example n Brad Myers, 1988 - 1992 in Lisp n Watch What I Do, chapter 10. n User interface development environment n Lapidary: Create UI elements, less inferencing n video (2.5 min) n Contributions: n UI for defining graphical constraints n More sophisticated behaviors by example

30. 30 Garnet, cont. n C32 n video (2 min) n Contributions: n Simple generalizations on copy-paste n Gilt: Graphical styles for widgets n video (1.5 min) n Contributions: n Extend text metaphor for tabs and n Gilt: transformations of widget values n video (5:15 min) n Contributions: n Inferring parameters of procedures from examples n C32 n video (2 min) n Contributions: n Simple generalizations on copy-paste n Gilt: Graphical styles for widgets n video (1.5 min) n Contributions: n Extend text metaphor for tabs and n Gilt: transformations of widget values n video (5:15 min) n Contributions: n Inferring parameters of procedures from examples

31. 31 Chimera n David Kurlander, 1990, Lisp+C+Postscript n Watch What I Do, chapter 12. n Graphical search and replace n Graphical “comic strip” representation for all actions n Generalization of macros selected from the transcript n Explicit specification of control structures n video (15 min) n David Kurlander, 1990, Lisp+C+Postscript n Watch What I Do, chapter 12. n Graphical search and replace n Graphical “comic strip” representation for all actions n Generalization of macros selected from the transcript n Explicit specification of control structures n video (15 min)

32. 32 Chimera Contributions n Primary Contributions: n Graphical Search and Replace n Inferring more sophisticated constraints from examples n Comic-book metaphor for command history & programs n Editing command history as undo/redo n Primary Contributions: n Graphical Search and Replace n Inferring more sophisticated constraints from examples n Comic-book metaphor for command history & programs n Editing command history as undo/redo

33. 33 DEMO n David A. Wolber and Gene L. Fisher, 1991, 1992 n David A. Wolber and Gene L. Fisher, "Demonstrational Technique for Developing Interfaces with Dynamically Created Objects," Proceedings UIST'91, Nov, 1992, Monterey, CA, pp. 89-97. n Gene L. Fisher, Dale E. Busse, and David A. Wolber, "Adding Rule-Based Reasoning to a Demonstrational Interface Builder," Proceedings UIST'92, Nov, 1991, pp. 221-230. n Create (pieces of) user interfaces n First system to support dynamic creation of objects. n Infers graphical relationships by examples of edits n Only simple relationships provided n No feedback of code n David A. Wolber and Gene L. Fisher, 1991, 1992 n David A. Wolber and Gene L. Fisher, "Demonstrational Technique for Developing Interfaces with Dynamically Created Objects," Proceedings UIST'91, Nov, 1992, Monterey, CA, pp. 89-97. n Gene L. Fisher, Dale E. Busse, and David A. Wolber, "Adding Rule-Based Reasoning to a Demonstrational Interface Builder," Proceedings UIST'92, Nov, 1991, pp. 221-230. n Create (pieces of) user interfaces n First system to support dynamic creation of objects. n Infers graphical relationships by examples of edits n Only simple relationships provided n No feedback of code

34. 34 DEMO Contributions n Primary Contributions n Concept of “Stimulus-Response” programming n Dynamic creation of objects n Primary Contributions n Concept of “Stimulus-Response” programming n Dynamic creation of objects

35. 35 Tourmaline n Brad Myers and Andrew Werth, 1990-1992. Lisp, Ness, WordBasic n Watch What I Do, chapter 14. n Infer composite styles for text formatting n E.g., more than one format in the header n Infer roles for each piece of text, and formatting for that role n Infer tables from a drawing n video (7:22 min) n Contributions n Roles n Brad Myers and Andrew Werth, 1990-1992. Lisp, Ness, WordBasic n Watch What I Do, chapter 14. n Infer composite styles for text formatting n E.g., more than one format in the header n Infer roles for each piece of text, and formatting for that role n Infer tables from a drawing n video (7:22 min) n Contributions n Roles

36. 36 Marquise n Brad Myers, Rich McDaniel, Dave Kosbie, 1993 in Lisp (part of Garnet) n Myers B., McDaniel, R. and Kosbie, D.. "Marquise: Creating Complete User Interfaces by Demonstration," CHI'94. Amsterdam, Netherlands, April 24-29, 1993. pp. 293-300. n Do more by demonstration n just show the way that the interface should operate. n Demonstrate when the behaviors start & feedback n Mouse button does one of 10 things n Demonstrate both behavior and conditions n Built-in support for palettes and modes. n Feedback using sentences with clickable words n video (3 min) n Brad Myers, Rich McDaniel, Dave Kosbie, 1993 in Lisp (part of Garnet) n Myers B., McDaniel, R. and Kosbie, D.. "Marquise: Creating Complete User Interfaces by Demonstration," CHI'94. Amsterdam, Netherlands, April 24-29, 1993. pp. 293-300. n Do more by demonstration n just show the way that the interface should operate. n Demonstrate when the behaviors start & feedback n Mouse button does one of 10 things n Demonstrate both behavior and conditions n Built-in support for palettes and modes. n Feedback using sentences with clickable words n video (3 min)

37. 37 Marquise Contributions n Primary Contributions n Four modes n Ability to demonstrate mouse actions by changing modes in the middle using the keyboard n Introduction of icons for “mouse droppings” n Demonstration that text with pop-up embedded menus was not a great idea n Primary Contributions n Four modes n Ability to demonstrate mouse actions by changing modes in the middle using the keyboard n Introduction of icons for “mouse droppings” n Demonstration that text with pop-up embedded menus was not a great idea

38. 38 Pursuit n Francemary Modugno, 1993-1995 in Lisp n Watch What I Do, chapter 20. n Generalize desktop programs n Inferred generalizations of data descriptions n Comic book representation for programs n video (6:43 min) n Francemary Modugno, 1993-1995 in Lisp n Watch What I Do, chapter 20. n Generalize desktop programs n Inferred generalizations of data descriptions n Comic book representation for programs n video (6:43 min)

39. 39 Pursuit Contributions n Primary Contributions n Generalizing the comic-strip metaphor n Handling of errors, pop-ups in the macro n Meta-dialog boxes n Showing the importance of the representation for the program n Primary Contributions n Generalizing the comic-strip metaphor n Handling of errors, pop-ups in the macro n Meta-dialog boxes n Showing the importance of the representation for the program

40. 40 Gold n Brad Myers, 1994 in Lisp n CHI’94, pp. 106-111 n Generalize business charts n Give one or two examples of desired elements n video (7:00 min) n Primary Contributions: n Generalizing graphics with no “program” n Feedback and editing by labeling n Handling of special items n Brad Myers, 1994 in Lisp n CHI’94, pp. 106-111 n Generalize business charts n Give one or two examples of desired elements n video (7:00 min) n Primary Contributions: n Generalizing graphics with no “program” n Feedback and editing by labeling n Handling of special items

41. 41 InferenceBear & Grizzly Bear n Martin Frank, 1994-1996 n Martin Frank and Jim Foley, "Model-Based User Interface Design by Example and By Interview,” UIST'93. Atlanta, GA, Nov. 3-5, 1993, pp. 129- 137. n Create UIs by demonstration and direct edit n User control through dialog boxes, edit using textual language: EET n First UI builder with multiple examples n Snapshots of before and after n Multiple examples n Add more positive examples to cause generalization n Negative examples to specify exceptions n Martin Frank, 1994-1996 n Martin Frank and Jim Foley, "Model-Based User Interface Design by Example and By Interview,” UIST'93. Atlanta, GA, Nov. 3-5, 1993, pp. 129- 137. n Create UIs by demonstration and direct edit n User control through dialog boxes, edit using textual language: EET n First UI builder with multiple examples n Snapshots of before and after n Multiple examples n Add more positive examples to cause generalization n Negative examples to specify exceptions

42. 42 InferenceBear & Grizzly Bear

43. 43 InferenceBear Pictures

44. 44 InferenceBear Contributions n Primary Contributions n Inferencing algorithm which can handle linear equations n Use of negative examples n Studies showing negative examples are difficult n Primary Contributions n Inferencing algorithm which can handle linear equations n Use of negative examples n Studies showing negative examples are difficult

45. 45 Pavlov n David Wolber, 1995-present n David Wolber, "Pavlov: Programming by Stimulus-Response Demonstration," CHI'96. April 1996. pp. 252-259 n Single positive example n Little generalization n Score editor for feedback and editing n Stimulus from mouse or time-based n video n David Wolber, 1995-present n David Wolber, "Pavlov: Programming by Stimulus-Response Demonstration," CHI'96. April 1996. pp. 252-259 n Single positive example n Little generalization n Score editor for feedback and editing n Stimulus from mouse or time-based n video

46. 46 Pavlov Contributions n Primary Contributions n Score editor as representation n Mouse-based stimulus n Primary Contributions n Score editor as representation n Mouse-based stimulus

47. 47 StageCast Creator n (formerly called KidSim, Cocoa) n Allen Cypher & David C. Smith;1994-present n http://www.stagecast.com n Formerly research at Apple, now commercial company: n Create animations and rough simulations n Program using before and after pictures n Can add additional control to rules n No automatic generalization n Video – on computer (3 min) n (formerly called KidSim, Cocoa) n Allen Cypher & David C. Smith;1994-present n http://www.stagecast.com n Formerly research at Apple, now commercial company: n Create animations and rough simulations n Program using before and after pictures n Can add additional control to rules n No automatic generalization n Video – on computer (3 min)

48. 48 StageCast Contributions n Primary Contributions n Before-After rules by example n Grouping of rules into sets: n Random ordered, first to succeed n Bringing PBD to wider audience n Primary Contributions n Before-After rules by example n Grouping of rules into sets: n Random ordered, first to succeed n Bringing PBD to wider audience

49. 49 Gamut n Rich McDaniel, 1996 - 1999 n R.G. McDaniel and B.A. Myers, "Getting More Out Of Programming-By- Demonstration.” CHI'99. Pittsburgh, PA, May 15-20, 1999. pp. 442-449. n Domain: "board games" and educational software n Goal: new interaction techniques so can infer more complex behaviors n E.g., how a piece can move in Monopoly / Chess n Reduce number of modes in UI n “Do Something”, “Stop That” n New interaction techniques to provide hints n Better inferencing algorithms n video (4:19 min) n Rich McDaniel, 1996 - 1999 n R.G. McDaniel and B.A. Myers, "Getting More Out Of Programming-By- Demonstration.” CHI'99. Pittsburgh, PA, May 15-20, 1999. pp. 442-449. n Domain: "board games" and educational software n Goal: new interaction techniques so can infer more complex behaviors n E.g., how a piece can move in Monopoly / Chess n Reduce number of modes in UI n “Do Something”, “Stop That” n New interaction techniques to provide hints n Better inferencing algorithms n video (4:19 min)

50. 50 Gamut Contributions n Primary Contributions n UI techniques for positive and negative examples that reduce modes n Hints by asking to highlight n Most powerful inferencing algorithm n Most ambitious coverage by inference n Primary Contributions n UI techniques for positive and negative examples that reduce modes n Hints by asking to highlight n Most powerful inferencing algorithm n Most ambitious coverage by inference

51. 51 Topaz n Brad Myers, 1998, part of Amulet in C++ n B.A. Myers. "Scripting Graphical Applications by Demonstration," CHI'98. Los Angeles, CA, April 18-23, 1998. pp. 534-541. n Move Emacs techniques to graphical editing n Move selection like a cursor n Generalizations using dialog boxes n video (3:00 min) n Brad Myers, 1998, part of Amulet in C++ n B.A. Myers. "Scripting Graphical Applications by Demonstration," CHI'98. Los Angeles, CA, April 18-23, 1998. pp. 534-541. n Move Emacs techniques to graphical editing n Move selection like a cursor n Generalizations using dialog boxes n video (3:00 min)

52. 52 Topaz Contributions n Primary Contributions n Scripted macros for graphics like have for text n Moving graphics selection like text cursor n Dialog boxes for generations of position n Primary Contributions n Scripted macros for graphics like have for text n Moving graphics selection like text cursor n Dialog boxes for generations of position

53. 53 General Themes n User Interface for Giving Examples n UI for feedback of Programs n Implementation n User Interface for Giving Examples n UI for feedback of Programs n Implementation

54. 54 User Interface for Giving Examples n Issue of Modes n Hints to help with inferencing n One versus multiple examples n How specify negative examples n Issue of Modes n Hints to help with inferencing n One versus multiple examples n How specify negative examples

55. 55 User Interface for Feedback of Program n None (Peridot, Gamut, etc.) n “Real” Code (Visual Basic for Word) n Special languages n EET language for InferenceBear n Comic strip languages (Chimera, Pursuit) n “Score” (time-line) view (Pavlov) n Before-after rules (Stagecast) n How edit the resulting program? n Many only support: give up and start over n None (Peridot, Gamut, etc.) n “Real” Code (Visual Basic for Word) n Special languages n EET language for InferenceBear n Comic strip languages (Chimera, Pursuit) n “Score” (time-line) view (Pavlov) n Before-after rules (Stagecast) n How edit the resulting program? n Many only support: give up and start over

56. 56 Example in Microsoft Word The fillparagraphs macro changes RETURNs into spaces, unless there are 2 in a row. Lots of short ¶ lines¶ can be filled and ¶ combined. ¶ ¶ But two RETURNS¶ signal a new ¶ paragraph. ¶ Lots of short lines can be filled and combined.¶ But two RETURNS signal a new para- graph.

57. 57 Word Visual Basic code n For “fillparagraphs” (removes RETURNS unless there are 2 in a row) Public Sub MAIN() WordBasic.EditReplace Find:="^p^p", Replace:="%$%", Direction:=0, MatchCase:=0, WholeWord:=0, PatternMatch:=0, SoundsLike:=0, ReplaceAll:=1, Format:=0, Wrap:=0 WordBasic.EditReplace Find:="^p", Replace:=" ", Direction:=0, MatchCase:=0, WholeWord:=0, PatternMatch:=0, SoundsLike:=0, ReplaceAll:=1, Format:=0, Wrap:=0 WordBasic.EditReplace Find:="%$%", Replace:="^p", Direction:=0, MatchCase:=0, WholeWord:=0, PatternMatch:=0, SoundsLike:=0, ReplaceAll:=1, Format:=0, Wrap:=0 End Sub n For “fillparagraphs” (removes RETURNS unless there are 2 in a row) Public Sub MAIN() WordBasic.EditReplace Find:="^p^p", Replace:="%$%", Direction:=0, MatchCase:=0, WholeWord:=0, PatternMatch:=0, SoundsLike:=0, ReplaceAll:=1, Format:=0, Wrap:=0 WordBasic.EditReplace Find:="^p", Replace:=" ", Direction:=0, MatchCase:=0, WholeWord:=0, PatternMatch:=0, SoundsLike:=0, ReplaceAll:=1, Format:=0, Wrap:=0 WordBasic.EditReplace Find:="%$%", Replace:="^p", Direction:=0, MatchCase:=0, WholeWord:=0, PatternMatch:=0, SoundsLike:=0, ReplaceAll:=1, Format:=0, Wrap:=0 End Sub

58. 58 Implementation n Techniques for implementing PBE n Especially for inferencing n Rule-based heuristics n More sophisticated AI algorithms n Pattern matching n Decision trees n Techniques for implementing PBE n Especially for inferencing n Rule-based heuristics n More sophisticated AI algorithms n Pattern matching n Decision trees

59. 59 Conclusions n PBE techniques are powerful but still “researchy” n Many exciting research questions n A community of researchers addressing them n Some commercial applications now n Increasing as computers get “smarter” n PBE techniques are powerful but still “researchy” n Many exciting research questions n A community of researchers addressing them n Some commercial applications now n Increasing as computers get “smarter”

60. 60 Sources for Further Information n Special issue of Communications of the ACM on Programming by Example: March, 2000. vol. 43, no. 3. n WWW home page for programming by example: http://lieber.www.media.mit.edu/people/lieber/PBE/index.html n Watch What I Do, edited by Allen Cypher, co-edited by Daniel C. Halbert, David Kurlander, Henry Lieberman, David Maulsby, Brad A. Myers, and Alan Turransky, The MIT Press, Cambridge, Massachusetts. n New book, next year: Henry Lieberman, ed. Your Wish is My Command. 2001: Morgan Kaufmann n Special issue of Communications of the ACM on Programming by Example: March, 2000. vol. 43, no. 3. n WWW home page for programming by example: http://lieber.www.media.mit.edu/people/lieber/PBE/index.html n Watch What I Do, edited by Allen Cypher, co-edited by Daniel C. Halbert, David Kurlander, Henry Lieberman, David Maulsby, Brad A. Myers, and Alan Turransky, The MIT Press, Cambridge, Massachusetts. n New book, next year: Henry Lieberman, ed. Your Wish is My Command. 2001: Morgan Kaufmann