Presentation is loading. Please wait.

Presentation is loading. Please wait.

Skill Acquisition via Transfer Learning and Advice Taking Lisa Torrey, Jude Shavlik, Trevor Walker University of Wisconsin-Madison, USA Richard Maclin.

Published bySara Dorsey Modified over 8 years ago

Similar presentations

Presentation on theme: "Skill Acquisition via Transfer Learning and Advice Taking Lisa Torrey, Jude Shavlik, Trevor Walker University of Wisconsin-Madison, USA Richard Maclin."— Presentation transcript:

1 Skill Acquisition via Transfer Learning and Advice Taking Lisa Torrey, Jude Shavlik, Trevor Walker University of Wisconsin-Madison, USA Richard Maclin University of Minnesota-Duluth, USA

2 Transfer Learning Agent learns Task A Agent encounters related Task B Agent discovers how tasks are related So far the user provides this info to the agent Agent uses knowledge from Task A to learn Task B faster Task A is the source. Task B is the target.

3 Transfer Learning The goal for the target task: performance training with transfer without transfer

4 Reinforcement Learning Overview Take an action Observe world state Receive a reward Policy: choose the action with the highest Q-value in the current state Use the rewards to estimate the Q- values of actions in states Described by a set of features

5 Transfer in Reinforcement Learning What knowledge will we transfer from the source? What knowledge will we transfer from the source? Q-functions (Taylor & Stone 2005) Q-functions (Taylor & Stone 2005) Policies (Torrey et al. 2005) Policies (Torrey et al. 2005) Skills (this work) Skills (this work) How will we extract that knowledge from the source? How will we extract that knowledge from the source? From Q-functions (Torrey et al. 2005) From Q-functions (Torrey et al. 2005) From observed behavior (this work) From observed behavior (this work) How will we apply that knowledge in the target? How will we apply that knowledge in the target? Model reuse (Taylor & Stone 2005) Model reuse (Taylor & Stone 2005) Advice taking (Torrey et al. 2005, this work) Advice taking (Torrey et al. 2005, this work)

6 Advice Taking Advice: instructions for the learner Advice: instructions for the learner IF: condition THEN: prefer action In these states Q action1 > Q action2 Complexity of Q-function Apply advice as soft constraints (KBKR, 2005) Apply advice as soft constraints (KBKR, 2005) For each action, find the Q-function that minimizes: Error on Training Data Disagreement with Advice + +

7 Experimental Domain: RoboCup Keep the ball Stone & Sutton 2001 KeepAway (KA/MKA) Score a goal Maclin et al. 2005 BreakAway (BA) MoveDownfield (MD) Cross the line Torrey et al. 2006 Different objectives, but a transferable skill: passing to teammates

8 A Challenge for Skill Transfer Shared skills are not exactly the same Shared skills are not exactly the same Skills have general and specific aspects Skills have general and specific aspects Aspects of the pass skill in RoboCup Aspects of the pass skill in RoboCup General: teammate must be open General: teammate must be open Game-specific: where teammate should be located Game-specific: where teammate should be located Player-specific: whether teammate is nearest or furthest Player-specific: whether teammate is nearest or furthest I’m open and near the goal. Pass to me! I’m open and far from you. Pass to me!

9 Addressing the Challenge We focus on learning general skill aspects We focus on learning general skill aspects These should transfer better These should transfer better We learn skills that apply to multiple players We learn skills that apply to multiple players This generalizes over player-specific aspects This generalizes over player-specific aspects We allow humans to provide information We allow humans to provide information They can point out game-specific aspects They can point out game-specific aspects

10 Human-Provided Information User provides a mapping to show task similarities User provides a mapping to show task similarities May also provide user advice about task differences May also provide user advice about task differences Pass Ø Pass towards goal Move towards goal Shoot at goal

11 Our Transfer Algorithm Observe source task games to learn skills Create advice for the target task Learn target task with KBKR Translate learned skills into transfer advice If there is user advice, add it in

12 Learning Skills By Observation Source-task games are sequences: (state, action) Source-task games are sequences: (state, action) Learning skills is like learning to classify states by their correct actions Learning skills is like learning to classify states by their correct actions We use Inductive Logic Programming to learn classifiers We use Inductive Logic Programming to learn classifiers State 1: distBetween(me,teammate2) = 15 distBetween(me,teammate1) = 10 distBetween(me,opponent1) = 5... action = pass(teammate2) outcome = caught(teammate2)

13 Advantages of ILP Can produce first-order rules for skills Can produce first-order rules for skills Capture only the essential aspects of the skill Capture only the essential aspects of the skill We expect these aspects to transfer better We expect these aspects to transfer better Can incorporate background knowledge Can incorporate background knowledge pass(Teammate) pass(teammate1) pass(teammateN) vs....

14 Preparing Datasets for ILP action = pass(Teammate) ? outcome = caught(Teammate) ? Q(pass) is high? Q(pass) is highest? Positive example for pass(Teammate) yes Q(other) is high? Q(pass) is lower? Negative example for pass(Teammate) no yes Reject example no

15 Example of a Skill Learned pass(Teammate) :- distBetween(me, Teammate) > 14, distBetween(me, Teammate) > 14, passAngle(Teammate) > 30, passAngle(Teammate) > 30, passAngle(Teammate) < 150, passAngle(Teammate) < 150, distBetween(me, Opponent) < 7. distBetween(me, Opponent) < 7.

16 KBKR requires propositional advice KBKR requires propositional advice We instantiate each rule head We instantiate each rule head Variables in rule bodies create disjunctions Variables in rule bodies create disjunctions We use tile features to translate them We use tile features to translate them Variables can appear multiple times Variables can appear multiple times We create new features to translate them We create new features to translate them Technical Challenges

17 Two Experimental Scenarios Pass Ø Pass towards goal Move towards goal Shoot at goal 4-on-3 MKA 3-on-2 BA 3-on-2 MD Pass MoveAhead Ø Pass MoveAhead Shoot at goal

18 Skill Transfer Results Without transfer From MKA From MD

19 Breakdown of MKA Results

20 What if User Advice is Bad?

21 Related Work Q-function transfer in RoboCup Q-function transfer in RoboCup Taylor & Stone (AAMAS 2005, AAAI 2005) Taylor & Stone (AAMAS 2005, AAAI 2005) Transfer via policy reuse Transfer via policy reuse Fernandez & Veloso (AAMAS 2006, ICML workshop 2006) Fernandez & Veloso (AAMAS 2006, ICML workshop 2006) Madden & Howley (AI Review 2004) Madden & Howley (AI Review 2004) Transfer via relational RL Transfer via relational RL Driessens et al. (ICML workshop 2006) Driessens et al. (ICML workshop 2006)

22 Summary of Contributions Transfer of shared skills in high-level logic Transfer of shared skills in high-level logic Despite differences in shared skills Despite differences in shared skills Demonstration of the value of user guidance Demonstration of the value of user guidance Easy to give and beneficial Easy to give and beneficial Effective transfer in the RoboCup domain Effective transfer in the RoboCup domain Challenging and dissimilar tasks Challenging and dissimilar tasks

23 Future Work Learn more general skills by combining multiple source tasks Learn more general skills by combining multiple source tasks Compare several transfer methods on RoboCup scenarios of varying difficulty Compare several transfer methods on RoboCup scenarios of varying difficulty Reach similar levels of transfer with less user input Reach similar levels of transfer with less user input

24 Acknowledgements DARPA Grant HR0011-04-1-0007 DARPA Grant HR0011-04-1-0007 US Naval Research Laboratory Grant N00173-06-1-G002 US Naval Research Laboratory Grant N00173-06-1-G002 Thank You

25 User Advice IF: distBetween(me,goal) < 10 AND angle(goal, me, goalie) > 40 angle(goal, me, goalie) > 40 THEN: prefer shoot IF: distBetween(me,goal) > 10 THEN: prefer move_ahead IF: [transferred conditions] AND distBetween(Teammate,goal) < distBetween(me,goal) THEN: prefer pass(Teammate) This is the part that came from transfer

26 Feature Tiling Original feature Tiling #1 Tiling #2 Tiling #8 … Tiling #9 Tiling #10 Tiling #11 … min value max value (16 tiles) (8 tiles)

27 Propositionalizing Rules pass(Teammate) :- distBetween(me, Teammate) > 14, distBetween(me, Teammate) > 14, … Step 1: rule head Step 1: rule head pass(teammate1) :- distBetween(me, teammate1) > 14, distBetween(me, teammate1) > 14, … pass(teammateN) :- distBetween(me, teammateN) > 14, distBetween(me, teammateN) > 14, … …

28 Propositionalizing Rules distBetween(me, Opponent) < 7 distBetween(me,opponent1) [0,7] + … + distBetween(me,opponentN ) [0,7] ≥ 1 Step 2: single-variable disjunctions Step 2: single-variable disjunctions distBetween(me,opponent1) < 7 OR … OR distBetween(me,opponentN) < 7

29 distBetween(me, Player) > 14, distBetween(me, Player) > 14, distBetween(Player, goal) < 10 distBetween(Player, goal) < 10 newFeature(player1) + … + newFeature(playerN) ≥ 1 newFeature(Player) :- Dist1 is distBetween(me, Player), Dist1 is distBetween(me, Player), Dist2 is distBetween(Player, goal), Dist2 is distBetween(Player, goal), Dist1 > 14, Dist2 14, Dist2 < 10. Add to target task feature space: Step 3: linked-variable disjunctions Step 3: linked-variable disjunctions Propositionalizing Rules

Download ppt "Skill Acquisition via Transfer Learning and Advice Taking Lisa Torrey, Jude Shavlik, Trevor Walker University of Wisconsin-Madison, USA Richard Maclin."

Similar presentations

Explanation-Based Learning (borrowed from mooney et al)

Explanation-Based Learning (borrowed from mooney et al)
Information Processing Technology Office Learning Workshop April 12, 2004 Seedling Overview Learning Hierarchical Reactive Skills from Reasoning and Experience.

Information Processing Technology Office Learning Workshop April 12, 2004 Seedling Overview Learning Hierarchical Reactive Skills from Reasoning and Experience.
Learning Procedural Planning Knowledge in Complex Environments Douglas Pearson March 2004.

Learning Procedural Planning Knowledge in Complex Environments Douglas Pearson March 2004.
Galit Haim, Ya'akov Gal, Sarit Kraus and Michele J. Gelfand A Cultural Sensitive Agent for Human-Computer Negotiation 1.

Galit Haim, Ya'akov Gal, Sarit Kraus and Michele J. Gelfand A Cultural Sensitive Agent for Human-Computer Negotiation 1.
Perceptron Learning Rule

Perceptron Learning Rule
1 Update on Learning By Observation Learning from Positive Examples Only Tolga Konik University of Michigan.

1 Update on Learning By Observation Learning from Positive Examples Only Tolga Konik University of Michigan.
RL for Large State Spaces: Value Function Approximation

RL for Large State Spaces: Value Function Approximation
Reinforcement Learning in Simulated Soccer with Kohonen Networks Chris White and David Brogan University of Virginia Department of Computer Science.

Reinforcement Learning in Simulated Soccer with Kohonen Networks Chris White and David Brogan University of Virginia Department of Computer Science.
Multi-Agent Strategic Modeling in a Robotic Soccer Domain Andraz Bezek, Matjaz Gams Department of Intelligent Systems, Jozef Stefan Institute {andraz.bezek,

Multi-Agent Strategic Modeling in a Robotic Soccer Domain Andraz Bezek, Matjaz Gams Department of Intelligent Systems, Jozef Stefan Institute {andraz.bezek,
Team Games Target level 3>4 My target is to…. Improve my basic knowledge of positions and their jobs within the game.

Team Games Target level 3>4 My target is to…. Improve my basic knowledge of positions and their jobs within the game.
Artificial Intelligence in Game Design Introduction to Learning.

Artificial Intelligence in Game Design Introduction to Learning.
Training complex judgment The effects of critical thinking instruction and contextual interference Anne Helsdingen 1, Tamara van Gog 1, Jeroen van Merriёnboer.

Training complex judgment The effects of critical thinking instruction and contextual interference Anne Helsdingen 1, Tamara van Gog 1, Jeroen van Merriёnboer.
Relational Data Mining in Finance Haonan Zhang CFWin03-37 03/04/2003.

Relational Data Mining in Finance Haonan Zhang CFWin /04/2003.
1 Learning from Behavior Performances vs Abstract Behavior Descriptions Tolga Konik University of Michigan.

1 Learning from Behavior Performances vs Abstract Behavior Descriptions Tolga Konik University of Michigan.
1 MACHINE LEARNING TECHNIQUES IN IMAGE PROCESSING By Kaan Tariman M.S. in Computer Science CSCI 8810 Course Project.

1 MACHINE LEARNING TECHNIQUES IN IMAGE PROCESSING By Kaan Tariman M.S. in Computer Science CSCI 8810 Course Project.
Ai in game programming it university of copenhagen Reinforcement Learning [Intro] Marco Loog.

Ai in game programming it university of copenhagen Reinforcement Learning [Intro] Marco Loog.
Machine Learning via Advice Taking Jude Shavlik. Thanks To... Rich Maclin Lisa Torrey Trevor Walker Prof. Olvi Mangasarian Glenn Fung Ted Wild DARPA.

Machine Learning via Advice Taking Jude Shavlik. Thanks To... Rich Maclin Lisa Torrey Trevor Walker Prof. Olvi Mangasarian Glenn Fung Ted Wild DARPA.
© Jesse Davis 2006 View Learning Extended: Learning New Tables Jesse Davis 1, Elizabeth Burnside 1, David Page 1, Vítor Santos Costa 2 1 University of.

© Jesse Davis 2006 View Learning Extended: Learning New Tables Jesse Davis 1, Elizabeth Burnside 1, David Page 1, Vítor Santos Costa 2 1 University of.
WISCONSIN UNIVERSITY OF WISCONSIN - MADISON Integrating Knowledge Capture and Supervised Learning through a Human-Computer Interface Trevor Walker, Gautam.

WISCONSIN UNIVERSITY OF WISCONSIN - MADISON Integrating Knowledge Capture and Supervised Learning through a Human-Computer Interface Trevor Walker, Gautam.
RL via Practice and Critique Advice Kshitij Judah, Saikat Roy, Alan Fern and Tom Dietterich PROBLEM: RL takes a long time to learn a good policy. Teacher.

RL via Practice and Critique Advice Kshitij Judah, Saikat Roy, Alan Fern and Tom Dietterich PROBLEM: RL takes a long time to learn a good policy. Teacher.

Similar presentations

Ads by Google