Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 Enhancing Interactions with To-Do Lists Using Artificial Assistants Yolanda Gil, Timothy Chklovski USC/Information Sciences Institute {gil,

Similar presentations


Presentation on theme: "1 Enhancing Interactions with To-Do Lists Using Artificial Assistants Yolanda Gil, Timothy Chklovski USC/Information Sciences Institute {gil,"— Presentation transcript:

1 1 Enhancing Interactions with To-Do Lists Using Artificial Assistants Yolanda Gil, Timothy Chklovski USC/Information Sciences Institute {gil, March 26, 2007

2 2 Structured statements about tasks and objects: - setting up a videoconference camera: switch on computer: turn on, start up, link up microphone: turn on, test, adjust Problems & remedies: - during a meeting projector not available locate a portable projector Learning Common Knowledge from Volunteers to Support Assistance Learning about tasks Learning to anticipate and repair 600,000+ statements collected over 12 months Targets collection by topic and knowledge type Learning about objects Learner Proactively broadens coverage Formulates relevant followup questions in real time Validates through other users Guides knowledge entry Learning paraphrases

3 3 To Do Lists To Do lists are pervasive, and present large opportunity for assistance and learning We’ve been working with TOWEL, the To Do list manager in CALO Glimpses / hints of users’ goals To Do lists Have some regularity & structure Contents and surface form may vary widely Similar to the collected statements

4 4 To Do Lists: Related Work Ethnographic studies look at usage of To Do lists Eg: V Bellotti, B Dalal, N Good, P Flynn, D Bobrow, N Ducheneaut. What a To-Do: Studies of Task Management Towards the Design of a Personal Task List Manager. CHI 2004 Analysis of work activities and how tools may support it, cognitive aids B Harrison. An Activity-Centric Approach To Context-Sensitive Time Management. In CHI 2004: Workshop on the Temporal Aspects of Work. A Dey, G Abowd. CybreMinder: A Context-Aware System for Supporting Reminders. HUC, 2000 Some commercial tools support prioritization Based on activity type, urgency, time available Eg, Life Balance, Voo2Do, “Remember the milk”, Tada.com (provides public to do lists)http://www.llamagraphics.com/  Not focused on NL interpretation or on automation, mostly on human factors and usability

5 5 Assisting with To Do Lists: the Idea Key idea: Develop interpretation & mapping of To Do entries to assistant-supported tasks Exploit large knowledge repositories and preprocessed texts Paraphrases to help interpret text Use knowledge repositories to interpret and connect user knowledge to operationalized tasks in CALO Build on prior work on volunteer collection of paraphrases to assist speech recognition / utterance identification [Chklovski, KCAP ’04 & KCVC ’04] Build on prior work on extraction from large corpora [Chklovski & Pantel, EMNLP ’05], and volunteer validation

6 6 BEAM: “Broad-coverage Entity Analysis and Mapping” To Do interface integrating BEAM and providing task interpretation and monitoring via CALO’s TOWEL List is automatically updated when the “Plan conference travel” action is completed To Do entry made by user BEAM mapping to TOWEL task

7 User To Do Entries Find hotel w/ pool for Joe Reserve conf room for talk Buy bread on the way home Get talk abstract from Joe CALO Task Ontology: Catalog of Automated Procedures Execution and Monitoring Instrumented Desktop Task Learning Set airport pickup for Joe Announce room for talk Reserve accommodations Host a visitor Arrange meeting Opportunities for Interpretation-based Assistance Map entries to task procedures Anticipate & suggest missing Entries, sub-tasks Assist with how activities are done in the organization BEAM Initiate and report execution Group and organize entries

8 User To Do Entries Find hotel w/ pool for Joe Reserve conf room for talk Map entries to task procedures Anticipate & suggest missing Entries, sub-tasks Assist with how activities are done in the organization BEAM Buy bread on the way home Get talk abstract from Joe Initiate and report execution Group and organize entries CALO Task Ontology: Catalog of Automated Procedures Execution and Monitoring Instrumented Desktop Task Learning Set airport pickup for Joe Announce room for talk Reserve accommodations Host a visitor Arrange meeting Ontologies NGrams Repository “to schedule a meeting” Subtasks Repository: “Reserve X has-subtask find X” Organization-specific Task knowledge “Airport pickup of visitors is common, but not here” Action Paraphrases Repository: “plan X  schedule X” “lease car  rent car” Verb Relations Repository: “schedule happens-before reschedule” From Text ExtractionFrom Web VolunteersFrom Volunteers in the Organization From Knowledge Engineers BEAM Knowledge Sources for NL Interpretation and Assistance Repairs Repository: “If projector not working, try a new bulb”

9 Map entries to task procedures Anticipate & suggest missing Entries, sub-tasks Assist with how activities are done in the organization BEAM Initiate and report execution Group and organize entries User To Do Entries Find hotel w/ pool for Joe Reserve conf room for talk Added by BEAM Added by user Deleted by BEAM because unnecessary Marked by BEAM as user-only Executed & monitored Buy bread on the way home Get talk abstract from Joe Ontologies NGrams Repository “to schedule a meeting” Subtasks Repository: “Reserve X has-subtask find X” Organization-specific Task knowledge “Airport pickup of visitors is common, but not here” Action Paraphrases Repository: “plan X  schedule X” “lease car  rent car” Verb Relations Repository: “schedule happens-before reschedule” From Text ExtractionFrom Web VolunteersFrom Volunteers in the Organization From Knowledge Engineers BEAM Knowledge Sources for NL Interpretation and Assistance Repairs Repository: “If projector not working, try a new bulb” CALO Task Ontology: Catalog of Automated Procedures Execution and Monitoring Instrumented Desktop Task Learning Set airport pickup for Joe Announce room for talk Reserve accommodations Host a visitor Arrange meeting Identify Omitted Actions Identify Related Sub-tasks Tolerate syntactic variety Interpret entries

10 10 Our Approach Extended existing semantic parsing techniques to take advantage of broad-coverage knowledge repositories Based on the standard Semantic Parsing approach [Acero; Zue; Allen et al] Syntactic chunking + identification of categories of entities + semantic parsing of annotated result Our problem is simpler in some ways Allows simplifying assumptions about structure of entries, speech acts present Our problem is harder in other ways: actions may not be fully specified, new actions may be automated (learned) To assist with user requests, need to identify implied actions & sub-actions Leverage large knowledge repositories Leverages paraphrase collection for speech system [Chklovski, K- CAP05, KCVC-05]

11 11 BEAM in Year 3: What Can and Cannot Be Interpreted Statements are processed into their semantic components Some entries cannot be interpreted because content is not recognized Paraphrasing knowledge allows rewriting of entries so they can be interpreted

12 12 BEAM’s Stages of Mapping a User’s To Do Entry 1. Syntactic parsing 2. Identification of semantic components, present & implied Volunteer contributed paraphrases (also task-subtask pairs) User’s To Do entry 4. Automatic entry rewriting yes no yes no Mapping failed Rewrites available? Mapped entry 3. Ontological mapping of semantic components Mapped? Teraword textual frequency repository ontology knowledge from volunteers and text extraction

13 13 BEAM’s Stages of Mapping a User’s To Do Entry 1. Syntactic parsing 5. Identification of sub-tasks 2. Identification of semantic components, present & implied Volunteer contributed paraphrases (also task-subtask pairs) User’s To Do entry ontology knowledge from volunteers and text extraction 4. Automatic entry rewriting yes no yes no Mapping failed Rewrites available? Mapped entry 3. Ontological mapping of semantic components Mapped? Teraword textual frequency repository

14 14 Using BEAM with a To Do Manager: BEAM API

15 Recent Developments: Smarter BEAM Identifies Likely Implied Actions 15 To Do entry does not specify an action BEAM looks in word corpus for mentions of “to * a meeting”, etc, identifying actions These actions are then mapped to the specific target ontology

16 Leveraging the Teraword Ngrams Source “quarterly meeting on Monday”  “to * a meeting” “presents for John”  “to * a present” 4876 buy 642 send 3482 get 619 bring … 593 open 1751 give 499 wrap 1352 find But what if there is little or no data? (Eg, “TGW meeting”) We’re exploring backoff strategies But what if there is noise in the data? Stoplists (eg, “be”, “have”) can provide some relief Validation & feedback could also help 16

17 17 Example of Volunteer-based Validation Completed in 2006 Snapshot of validation: 1,113 harvested statements were put through context- directed validation for “likely to be purchased in an office categories of times” Most were filtered out; 107 (9.6%) passed Determined to be likely to be purchased in an office: 'office supplies' > planners 'office supplies' > equipment Business & Industrial > Food Service & Retail > Bar & Beverage Equipment > Coffee Categories determined to be unlikely to be purchased in an office: Home & Garden > Pet Supplies > Cats > Cat Toys 'building supplies' > 'concrete finishing' cards > 'racing-nascar'

18 Towards Identifying Subtasks: Start with Large Corpus “a meeting with Peter” QUERY1: “the [Y] for the [X]” QUERY2: (filter) “need the [Y]” QUERY3: “to [Z] the [X] [Y]” “approve meeting agenda”, “set/change/confirm meeting time” Similarly: for “flight to SFO” Top suggestions include: “buy/purchase flight ticket”, “make flight arrangements” 18

19 Towards Identifying Subtasks: Start with Large Corpus “a meeting with Peter” QUERY1: “the [Y] for the [X]” QUERY2: (filter) “need the [Y]” QUERY3: “to [Z] the [X] [Y]” “approve meeting agenda”, “set/change/confirm meeting time” Similarly: for “flight to SFO” Top suggestions include: “buy/purchase flight ticket”, “make flight arrangements” 19 Under Development – Stay Tuned

20 Test System: Mappings to Task Ontology Categories Early integration done to support a test question not otherwise addressed by CALO: Any small contribution to test results considered a success No lead time for targeted collection for relevant task entries PQ0166: What instances of task type (choose one) {|sc:%Communicate|, |sc:%Decide|, |sc:%Obtain|, |sc:%PlanAndSchedule|} are on user’s to-do list? Examples handled by BEAM: arrange travel  #PlanAndSchedule::Plan office supplies  #Obtain::Buy hire a car  #Obtain::Rent respond to Mary's request  #Communicate::Answer This is more powerful than straightforward application of, eg, WordNet synonyms BEAM handles some situations where there is no synonymy or is-a relation between terms, eg. (arrange travel  #PlanAndSchedule::Plan) BEAM handles some situations where action is not even present (office supplies  #Obtain::Buy)

21 21 Evaluation of Paraphrase Component Despite early integration, BEAM contributed to evaluation 582 To Do statements collected for CALO Y3 evaluation 31.1% were mappable using the paraphrase repository 24.6% without paraphrase repository Paraphrase repository was collected without focus on these items specifically Contained 3,114 items, but (we estimate) only ~100 related to the domain covered by the test question Additional knowledge sources and larger repositories will support further improvement of performance Now also have data from public online To Dos, “tada list”

22 22 Conclusions Developed BEAM, first system to demonstrate To Do list interpretation to enable automation Integrated with working ToDo list manager, CALO’s Towel Extended existing semantic parsing techniques to take advantage of preexisting large knowledge sources Leveraged volunteers-created paraphrase corpus to improve ToDo entry interpretation Identify likely implied actions Subtask suggestion in the works

23 23 Ongoing & Planned Work: New Capabilities Improve interpretation and mapping capabilities: Evaluate support of To Do entries which have no verb – ability to identify the implied actions Proactively identify automatable subtasks for To Do lists Acquire knowledge about relevant subtasks Use BEAM’s semantic frames to provide information for task arguments, Towel forms (eg, “travel to Boston”) Validate with/acquire from volunteers knowledge about sub- tasks, mappings To Do entry made by user BEAM mapping to TOWEL task


Download ppt "1 Enhancing Interactions with To-Do Lists Using Artificial Assistants Yolanda Gil, Timothy Chklovski USC/Information Sciences Institute {gil,"

Similar presentations


Ads by Google