Learning in the Wild Satanjeev “Bano” Banerjee Dialogs on Dialog March 18 th, 2005 In the Meeting Room Scenario

Introduction Project CALO: Building agents that support professionals in their work life Project CALO: Building agents that support professionals in their work life Smart management of s, calendar, etc. Smart management of s, calendar, etc. Organizing information from meetings for easier/faster access Organizing information from meetings for easier/faster access Inter-disciplinary, distributed research: Inter-disciplinary, distributed research: Speech, Vision, Multi-modal integration, Machine learning, Reasoning, etc… Speech, Vision, Multi-modal integration, Machine learning, Reasoning, etc…

CMU’s Piece of the Pie Automatic meeting understanding, specifically: Automatic meeting understanding, specifically: Perform automatic speech recognition Perform automatic speech recognition Detect different meeting states Detect different meeting states Presentations, discussions, briefings Presentations, discussions, briefings Detect major topics of discussion Detect major topics of discussion Detect decisions and action items Detect decisions and action items Detect personnel skills Detect personnel skills Vision expert vs. speech expert vs. statistics expert… Vision expert vs. speech expert vs. statistics expert… What else? What else? Program’s main aim: Learning in the Wild! Program’s main aim: Learning in the Wild!

Learning in the Wild! Improve performance with continued use and without additional engineering! Improve performance with continued use and without additional engineering! Typical learning scenario: Typical learning scenario: Engineers deliver the system to company X, and never touch the system again. Engineers deliver the system to company X, and never touch the system again. Compare % of action items automatically detected during 1 st and 10 th weeks of use. Compare % of action items automatically detected during 1 st and 10 th weeks of use. % action items detected during 10 th week must be greater than during 1 st week! % action items detected during 10 th week must be greater than during 1 st week!

How can we do LITW? Purely unsupervised learning techniques: Purely unsupervised learning techniques: Improve ASR through adaptation Improve ASR through adaptation Clustering to create better understanding of topics / personnel skills… Clustering to create better understanding of topics / personnel skills… “Implicitly supervised” learning “Implicitly supervised” learning Learn from users doing the same task (hints) Learn from users doing the same task (hints) Ask for “labels” from human in particularly “useful” cases (nudges) Ask for “labels” from human in particularly “useful” cases (nudges)

Description of Software Meeting Recorder Meeting Recorder Speech, video, notes, slides, whiteboard markings Speech, video, notes, slides, whiteboard markings All synchronized using NTP server, and auto-stored in central database All synchronized using NTP server, and auto-stored in central database Mock Brow Mock Brow Plays back media recorded using Meeting Recorder Plays back media recorded using Meeting Recorder Offline “understanding” component Offline “understanding” component Speech recognizer Speech recognizer Meeting state and topic detector Meeting state and topic detector

Out-of-the-Box Topic Detection Look for sudden drastic changes in Look for sudden drastic changes in Vocabulary Vocabulary Subset of active speakers Subset of active speakers Look for cue phrases, intonation Look for cue phrases, intonation “So…” “So…” “Moving on…” “Moving on…”

Can we Improve with Supervision? If a user was to manually mark topics, can we improve beyond a trained system? If a user was to manually mark topics, can we improve beyond a trained system? Yes, by acquiring the vocabulary of the various topics typical of this company! Yes, by acquiring the vocabulary of the various topics typical of this company! Training data may not cover company’s domain Training data may not cover company’s domain Yes, by learning the active speakers for each topic Yes, by learning the active speakers for each topic Training data cannot cover this at all Training data cannot cover this at all Yes, by learning idiosyncrasies of this group Yes, by learning idiosyncrasies of this group Topic shifts happen only when Bob is speaking Topic shifts happen only when Bob is speaking Bob always says “Now, the next thing…” before he shifts Bob always says “Now, the next thing…” before he shifts

How to Elicit Supervision? But user will not mark topics for us But user will not mark topics for us Because cost too high, and value too low! Because cost too high, and value too low! So: Can we design an interaction so topic markup falls out of other user activity? So: Can we design an interaction so topic markup falls out of other user activity? An activity with “viable” cost/value ratio An activity with “viable” cost/value ratio Idea: Better designed notes box Idea: Better designed notes box

“Smart” Notes Box First allow user to enter the agenda, one item per box First allow user to enter the agenda, one item per box User will enter notes on discussion of each agenda item in the appropriate box User will enter notes on discussion of each agenda item in the appropriate box From times of taking notes we can tell topic boundaries! From times of taking notes we can tell topic boundaries! Report on integration Inputs from sphinx ok Trouble with output of adapter Discuss web page Web space set up Bob needs access to CGI server Plan roadmap

Cost / Value of Smart Notes Box Cost of interaction: Nearly same as “flat” notes box Cost of interaction: Nearly same as “flat” notes box Immediate value: Immediate value: Users can “one click ” the notes to other participants – useful for meeting scribe Users can “one click ” the notes to other participants – useful for meeting scribe Longer term value: Longer term value: Notes will be “media augmented” – users will be able to use Mock Brow to play back the media, using the notes as an index Notes will be “media augmented” – users will be able to use Mock Brow to play back the media, using the notes as an index At future meetings, MR will “suggest” agenda items – even less typing needed than “flat” notes box! At future meetings, MR will “suggest” agenda items – even less typing needed than “flat” notes box!

Steps to Create Instances of Learning in the Wild Steps to Create Instances of Learning in the Wild 1. Identify task to be performed by system E.g.: Topic detection E.g.: Topic detection 2. Create “out-of-the-box” system E.g.: Decision trees trained on manual data E.g.: Decision trees trained on manual data 3. Show that supervision can help 4. Design interface to extract implicit supervision

My Research Goals For the chosen tasks (topic detection, action item detection, etc): For the chosen tasks (topic detection, action item detection, etc): Create out-of-the-box technologies Create out-of-the-box technologies Implement/develop algorithms to rapidly adapt using sparse implicit supervision Implement/develop algorithms to rapidly adapt using sparse implicit supervision Design interfaces that extract maximal implicit supervision Design interfaces that extract maximal implicit supervision Wish: Somehow generalize all this… Wish: Somehow generalize all this…

How to Evaluate? Out-of-the-box technology: Standard machine learning train/test evaluation Out-of-the-box technology: Standard machine learning train/test evaluation Improvement with adaptation: Show delta of performance between system with/without supervision Improvement with adaptation: Show delta of performance between system with/without supervision Evaluating the interface’s cost/value viability: Long and situated user study to show that indeed regular users will use the interface, often! Evaluating the interface’s cost/value viability: Long and situated user study to show that indeed regular users will use the interface, often!

User Study: Testing the Smart Notes Box Interface Goals: Goals: 1. Will users use the “smart” notes box? 2. Will their use indeed line up cleanly with actual topic boundaries? Cannot run a “one off” lab test Cannot run a “one off” lab test Users may use the box the first few times – but will they give up after a few meetings? Users may use the box the first few times – but will they give up after a few meetings? Will users use the notes box when they are deeply involved in the meeting? Will users use the notes box when they are deeply involved in the meeting?

User Study Design Invite multiple groups to use the Meeting Recorder for their regular meetings Invite multiple groups to use the Meeting Recorder for their regular meetings At the first meeting, describe the interface At the first meeting, describe the interface Point out the value of taking notes in the smart notes box instead of in Emacs say Point out the value of taking notes in the smart notes box instead of in Emacs say Observe the groups over a couple of months, noting their use of the text box Observe the groups over a couple of months, noting their use of the text box

User Study Success Conditions Will declare victory if users use the smart text box often Will declare victory if users use the smart text box often Often defined as some percentage of maximum use we can hope for = we get supervision for all the major topics discussed! Often defined as some percentage of maximum use we can hope for = we get supervision for all the major topics discussed! Will pop the champagne if their use of the text box increases over time Will pop the champagne if their use of the text box increases over time May happen as they experience the long term values of the interaction May happen as they experience the long term values of the interaction