ACAL – Active Capture Automation Language Ana Ramírez Advisors: Marc Davis, Jen Mankoff GUIR 25 February 2004 UC Berkeley - Garage Cinema Research - Group for User Interface Research

2 10/21/2003 Overview What is Active Capture Challenges in  design  Implementation Support at toolkit level for  design  implementation

3 10/21/2003 Motivation Systems that direct human actions  Keep awake system  Sports instruction (golf swing)  Automated health screening  Video Door locks Interactions that control timing of interaction

4 10/21/2003 Active Capture CaptureInteraction Processing Direction/ Cinematography Human- Computer Interaction Computer Vision/ Audition Active Capture

5 10/21/2003 Active Capture CaptureInteraction Processing Direction/ Cinematography Human- Computer Interaction Computer Vision/ Audition Active Capture

6 10/21/2003 Active Capture CaptureInteraction Processing Direction/ Cinematography Human- Computer Interaction Computer Vision/ Audition Active Capture

7 10/21/2003 Active Capture CaptureInteraction Processing Direction/ Cinematography Human- Computer Interaction Computer Vision/ Audition Active Capture

8 10/21/2003 Active Capture CaptureInteraction Processing Direction/ Cinematography Human- Computer Interaction Computer Vision/ Audition Active Capture

9 10/21/2003 Implemented Applications See Video at:

10 10/21/2003 Head Turn Recognizer Uses  Gross motion detector  Eye detector Looks for  No eyes and No motion followed by  Motion followed by  Eyes followed by  No motion Uses mediation if something goes wrong.

11 10/21/2003 Exercise Given raw materials  User actions  System actions  Recognizers system uses Write down a representation of the interaction of the head turn recognizer. Work in groups of two

12 10/21/2003 Designing Active Capture Applications Describe path of “righteousness” Describe what to do if something goes wrong (mediation) Use good mediation techniques  Progressive assistance  Freshness  Graceful failure

13 10/21/2003 Path of “righteousness” is not obvious Tedious to write Expression of time flow cumbersome Head Turn Application

14 10/21/2003 Challenges Difficult to represent  Control process with feedback (mediation)  Timing  Strict and non strict ordering

15 10/21/2003 Goals of ACAL In general  Natural to describe Active Capture applications.  Support mediation strategies  Include time flow primitives Support brain storming  Visual “language” Support rapid prototyping  Support for Wizard-of-Oz prototypes Support implementation  Make path of “righteousness” apparent in code  Be able to prove an implementation will reach the “done” state.

16 10/21/2003 Current Status of ACAL Visual “Language”  Started with Ka-Ping Yee in Marc Davis’ class on Multimedia Information in Spring 2003 Toolkit level support  Main focus this semester Support Wizard-of-Oz protoyping  Future work Link visual language, toolkit and wizard-of-oz support together.  Future Work

17 10/21/2003 Current Research Areas Design Guidelines  Jeff Heer, Nathan Good, Ana Ramirez, Marc Davis, Jen Mankoff. “Presiding Over Accidents: System Mediation of Human Action” CHI’04 Language Support  ACAL New Application  “Say Cheese”

18 10/21/2003 Visual “Language” Path of “righteousness”  Observations  Commands  Capture  Time Constraints

19 10/21/2003 Visual “Language” Path of “righteousness”  Observations  Commands  Capture  Time Constraints

20 10/21/2003 Visual “Language” Path of “righteousness”  Observations  Commands  Capture  Time Constraints

21 10/21/2003 Visual “Language” Path of “righteousness”  Observations  Commands  Capture  Time Constraints

22 10/21/2003 Visual “Language” Path of “righteousness”  Observations  Commands  Capture  Time Constraints

23 10/21/2003 Visual “Language” Path of “righteousness”  Observations  Commands  Capture  Time Constraints

24 10/21/2003 Visual “Language” Add mediation for case when actor is looking at camera before turn.

25 10/21/2003 Visual “Language” Add freshness to mediation

26 10/21/2003 Add progressive assistance

27 10/21/2003 Language Design Process Two key challenges: 1.Control-oriented vs. time-oriented representation 2.Absolute vs. relative time relationships

28 10/21/2003 Control vs. Time State machines and procedural programs describe control flow well...but they visualize time poorly Timeline representation allows concurrency to be fully expressed...but decisions and control flow don’t fit easily on a timeline

29 10/21/2003 Control vs. Time Hybrid visual representation: timelines with flow arrows

30 10/21/2003 Absolute vs. Relative Time Horizontal scale on timeline implies particular lengths of intervals Problem: sometimes want ordering; sometimes want specific intervals Solution: arrangement on timeline yields ordering; min/max specifiers constrain time intervals

31 10/21/2003 Each point on a track specifies “true”, “false”, or “don’t care” Example: doesn’t matter whether when waving or speaking begins or ends, as long as both happen at some point within a 5-second period Flexibility in Ordering waving speaking < 5 sec

32 10/21/2003 Lessons Learned Difficult to balance between:  Control-oriented vs. time-oriented representation  Absolute vs. relative time relationships Difficult to manage complexity Important to be able to see path of “righteousness” Easy to get mediation wrong.

33 10/21/2003 Future Work Visual Language  Better solution to absolute vs. relative time relationships challenge  Better support for mediation strategies Implementation support “Say Cheese” Automated health screening

34 10/21/2003 Questions

35 10/21/2003 System Architecture

36 10/21/2003 ACAL Design Goals Natural to describe Active Capture applications Support key strategies for mediation  Progressive Assistance  Graceful Failure  Freshness Include time flow primitives Support brainstorming process

37 10/21/2003 ACAL Toolkit level support for applications with:  Mediation  Complex timing  Rich media input and output

38 10/21/2003 Future Applications “Say Cheese” Folk Computing  Support remote, more frequent medical screening.

39 10/21/2003 Methodology Theoretical  Active Capture Design Space  Mediation strategies / guidelines Practical  Reverse engineer implemented applications  Design a new application