1. Inverse Reinforcement Learning Pieter Abbeel UC Berkeley EECS
TexPoint fonts used in EMF.
Read the TexPoint manual before you delete this box.: AAAAAAAAAA

2. High-level picture Dynamics Model T Reinforcement Reward Function R
Probability distribution over next states given current state and action
Describes desirability of being in a state.
Reinforcement
Learning / Optimal Control
Reward Function R
Controller/ Policy p*
Prescribes action to take for each state
What’s high level picture here: need 1,2,3.
load balancing, pricing, ad placement, …
Many problems in robotics have unknown, stochastic, high-dimensional,
and highly non-linear dynamics, and offer significant challenges to
classical control methods. Some of the key difficulties in these
problems are that (i) It is often hard to write down, in closed form,
a formal specification of the control task (for example, what is the
objective function for "flying well"?), (ii) It is difficult to build
a good dynamics model because of both data collection and data
modeling challenges (similar to the "exploration problem" in
reinforcement learning), and (iii) It is expensive to find closed-loop
controllers for high dimensional, stochastic domains. In this
talk, I will present learning algorithms with formal performance
guarantees which show that these problems can be efficiently addressed
in the apprenticeship learning setting---the setting when expert
demonstrations of the task are available.
Car example:
What are we trying to learn here:
Current state  correct action = (throttle/brake, steering).
Let’s consider the fairly simple task of driving a car on a highway (in that, much simpler than general traffic situation). Keep in mind, just iconic example to get the point across. I will look at more complex, real-world problems (which we solved for the first time) later.
Build up policy complexity: try to maintain desired speed; if car ahead, switch to left lane; if car there, switch to right lane;
What if we want to take an exit soon? – Don’t switch into the left lane, except if you see a clear path to get back (what does that even mean, a clear path) etc.
Quickly becomes very complex. For supervised learning to work we need to
(a) Encounter all these different situations, so we can learn the appropriate action for them. Thus, e.g., our demonstration needs to include some pretty dangerous situations …
(b) We need to have a learning architecture (SVM, deep belief net, neural net, decision tree, …) capable of
capturing the extreme richness of our decisions.
Making decisions with so many possible actions (rather than 0/1 etc.)
Really, only will work in case of policy simplicity.
Examples include: bla …; these examples are extremely restrictive
Ignoring the structure of the problem.
When we drive a car we have
(a) specific objectives:
Avoid other cars
Get to our goal
Maybe talk on our cell phone, curse at other drivers, etc. ?
(b) an idea of how our car and other cars behave
== dynamics model
Key contribution: exploit this structure, and learn reward function and dynamics model from demonstrations rather than directly learn a policy. The controller that falls out generalizes much better b/c it “understands” the nature of the problem. Rather than “blindly” learned to mimick the teacher.
Step back here: I am not so much concerned about driving the car in the highway simulator; this is just an illustrative toy example. This idea applies generally to problems I mentioned before:
Inverse RL:
Given ¼*and T, can we recover R?
More generally, given execution traces, can we recover R?

3. Motivation for inverse RL
Scientific inquiry
Model animal and human behavior
E.g., bee foraging, songbird vocalization. [See intro of Ng and Russell, 2000 for a brief overview.]
Apprenticeship learning/Imitation learning through inverse RL
Presupposition: reward function provides the most succinct and transferable definition of the task
Has enabled advancing the state of the art in various robotic domains
Modeling of other agents, both adversarial and cooperative
Potential use of RL and related methods as computational models for animal and human learning (Watkins, 1989; Schamjuk and Zanuttoa, 1997; Touretzky and Saksida, 1997). Such models are supported both by behavioral studies and by neurophysiological evidence that RL occurs in bee foraging (Montague et al., 1995) and in songbird vocalization (Doya and Sejnowski, 1995). This literature assumes, however, that the reward function is fixed and known---for example, models of bee foraging assume that the reward at each flower is a simple saturating function of nectar content. Yet it seems clear that in examining animal and human behavior we must consider the reward function as an unknown to be ascertained through empirical investigation. This is particularly true of multiattribute reward functions. Consider, for example, that the bee might weigh nectar ingestion against flight distance, time, and risk from wind and predators. It is hard to see how one could determine the relative weights of these terms a priori. Similar considerations apply to human economic behavior, for example. Hence inverse RL is a fundamental problem of theoretical biology, econometrics, and other fields.

4. Lecture outline Examples of apprenticeship learning via inverse RL
Inverse RL vs. behavioral cloning
Historical sketch of inverse RL
Mathematical formulations for inverse RL
Case studies
Trajectory-based reward, with application to autonomous helicopter flight

5. Examples Simulated highway driving Aerial imagery based navigation
Abbeel and Ng, ICML 2004,
Syed and Schapire, NIPS 2007
Aerial imagery based navigation
Ratliff, Bagnell and Zinkevich, ICML 2006
Parking lot navigation
Abbeel, Dolgov, Ng and Thrun, IROS 2008
Urban navigation
Ziebart, Maas, Bagnell and Dey, AAAI 2008
Quadruped locomotion
Ratliff, Bradley, Bagnell and Chestnutt, NIPS 2007
Kolter, Abbeel and Ng, NIPS 2008

6. Urban navigation Reward function for urban navigation?
 destination prediction
Distance Speed Type Lanes Turns Context Time Fuel Safety Stress Skill Mood
Our view is that, like a fingerprint, each driver has a unique way in which they trade off these various factors under different contextual situations. Unfortunately, we have no way of knowing the perceived time, safety, stress, or skill requirements that a driver has for different routes. <click>
Instead we view drivers preferences based on the characteristics of the road network that we do have, and learn to “take a driver’s fingerprint” by analyzing their previously driven routes with respect to these road network features.
Ziebart, Maas, Bagnell and Dey AAAI 2008

7. Lecture outline Examples of apprenticeship learning via inverse RL
Inverse RL vs. behavioral cloning
Historical sketch of inverse RL
Mathematical formulations for inverse RL
Case studies
Trajectory-based reward, with application to autonomous helicopter flight

8. Problem setup Input: Inverse RL:
State space, action space
Transition model Psa(st+1 | st, at)
No reward function
Teacher’s demonstration: s0, a0, s1, a1, s2, a2, …
(= trace of the teacher’s policy *)
Inverse RL:
Can we recover R ?
Apprenticeship learning via inverse RL
Can we then use this R to find a good policy ?
Behavioral cloning
Can we directly learn the teacher’s policy using supervised learning?

9. Behavioral cloning Formulate as standard machine learning problem
Fix a policy class
E.g., support vector machine, neural network, decision tree, deep belief net, …
Estimate a policy (=mapping from states to actions) from the training examples (s0, a0), (s1, a1), (s2, a2), …
Two of the most notable success stories:
Pomerleau, NIPS 1989: ALVINN
Sammut et al., ICML 1992: Learning to fly (flight sim)
Kuniyoshi et al., 1994; Demiris & Hayes, 1994; Amit & Mataric, 2002.

10. Inverse RL vs. behavioral cloning
Which has the most succinct description: ¼* vs. R*?
Especially in planning oriented tasks, the reward function is often much more succinct than the optimal policy.

11. Lecture outline Examples of apprenticeship learning via inverse RL
Inverse RL vs. behavioral cloning
Historical sketch of inverse RL
Mathematical formulations for inverse RL
Case studies
Trajectory-based reward, with application to autonomous helicopter flight

12. Inverse RL history
1964, Kalman posed the inverse optimal control problem and solved it in the 1D input case
1994, Boyd+al.: a linear matrix inequality (LMI) characterization for the general linear quadratic setting
2000, Ng and Russell: first MDP formulation, reward function ambiguity pointed out and a few solutions suggested
2004, Abbeel and Ng: inverse RL for apprenticeship learning---reward feature matching
2006, Ratliff+al: max margin formulation

13. Inverse RL history
2007, Ratliff+al: max margin with boosting---enables large vocabulary of reward features
2007, Ramachandran and Amir, and Neu and Szepesvari: reward function as characterization of policy class
2008, Kolter, Abbeel and Ng: hierarchical max-margin
2008, Syed and Schapire: feature matching + game theoretic formulation
2008, Ziebart+al: feature matching + max entropy
2008, Abbeel+al: feature matching -- application to learning parking lot navigation style
Active inverse RL? Inverse RL w.r.t. minmax control, partial observability, learning stage (rather than observing optimal policy), … ?

14. Lecture outline Examples of apprenticeship learning via inverse RL
Inverse RL vs. behavioral cloning
Historical sketch of inverse RL
Mathematical formulations for inverse RL
Case studies
Trajectory-based reward, with application to autonomous helicopter flight

15. Three broad categories of formalizations
Max margin
Feature expectation matching
Interpret reward function as parameterization of a policy class

16. Basic principle
Find a reward function R* which explains the expert behaviour.
Find R* such that
In fact a convex feasibility problem, but many challenges:
R=0 is a solution, more generally: reward function ambiguity
We typically only observe expert traces rather than the entire expert policy ¼* --- how to compute LHS?
Assumes the expert is indeed optimal --- otherwise infeasible
Computationally: assumes we can enumerate all policies
Note: it is a convex problem (but with many challenges)

17. Feature based reward functionff
Subbing into
gives us:
Expected cumulative discounted sum of feature values or “feature expectations”
Give car driving example here to illustrate the sample complexity issue.

18. Feature based reward function
Feature expectations can be readily estimated from sample trajectories.
The number of expert demonstrations required scales with the number of features in the reward function.
The number of expert demonstration required does not depend on
Complexity of the expert’s optimal policy ¼*
Size of the state space
Give car driving example here to illustrate the sample complexity issue.

19. Recap of challenges Challenges:
Assumes we know the entire expert policy ¼*  assumes we can estimate expert feature expectations
R=0 is a solution (now: w=0), more generally: reward function ambiguity
Assumes the expert is indeed optimal---became even more of an issue with the more limited reward function expressiveness!
Computationally: assumes we can enumerate all policies

20. Ambiguity Standard max margin: “Structured prediction” max margin:
Justification: margin should be larger for policies that are very different from ¼*.
Example: m(¼, ¼*) = number of states in which ¼* was observed and in which ¼ and ¼* disagree

21. Expert suboptimality
Structured prediction max margin with slack variables:
Can be generalized to multiple MDPs (could also be same MDP with different initial state)

22. Complete max-margin formulation
[Ratliff, Zinkevich and Bagnell, 2006]
Resolved: access to ¼*, ambiguity, expert suboptimality
One challenge remains: very large number of constraints
Ratliff+al use subgradient methods.
In this lecture: constraint generation

23. Constraint generation
Initialize ¦(i) = {} for all i and then iterate
Solve
For current value of w, find the most violated constraint for all i by solving:
= find the optimal policy for the current estimate of the reward function (+ loss augmentation m)
For all i add ¼(i) to ¦(i)
If no constraint violations were found, we are done.
[Note: many other approaches exist, this one is particularly simple and seems to work well in practice.]

24. Visualization in feature expectation space
Every policy ¼ has a corresponding feature expectation vector ¹(¼), which for visualization purposes we assume to be 2D 2
structured max margin (?)
max margin
(*)
wmm
Draw out some w vectors; show which are the optimal policy for it
wsmm 1

25. Constraint generation
Every policy ¼ has a corresponding feature expectation vector ¹(¼), which for visualization purposes we assume to be 2D
constraint generation: 2
(2)
(*)
(1)
Draw out some w vectors; show which are the optimal policy for it
(0) 1

26. Three broad categories of formalizations
Max margin (Ratliff+al, 2006)
Feature boosting [Ratliff+al, 2007]
Hierarchical formulation [Kolter+al, 2008]
Feature expectation matching (Abbeel+Ng, 2004)
Two player game formulation of feature matching (Syed+Schapire, 2008)
Max entropy formulation of feature matching (Ziebart+al,2008)
Interpret reward function as parameterization of a policy class. (Neu+Szepesvari, 2007; Ramachandran+Amir, 2007)

27. Feature matching
Inverse RL starting point: find a reward function such that the expert outperforms other policies
Observation in Abbeel and Ng, 2004: for a policy ¼ to be guaranteed to perform as well as the expert policy ¼*, it suffices that the feature expectations match:
implies that for all w with

28. Theoretical guarantees
I don’t want to parse this, but happy to give further details offline if anyone would like me to.
Our algorithm does not necessarily recover the teacher’s reward function R_w^* which is impossible to recover.
TODO: add slide with theorem that includes the sample complexity in the statement.
Guarantee w.r.t. unrecoverable reward function of teacher.
Sample complexity does not depend on complexity of teacher’s policy *.

29. Apprenticeship learning [Abbeel & Ng, 2004]
Assume
Initialize: pick some controller 0.
Iterate for i = 1, 2, … :
“Guess” the reward function:
Find a reward function such that the teacher maximally outperforms all previously found controllers.
Find optimal control policy i for the current guess of the reward function Rw.
If , exit the algorithm.
Learning through reward functions rather than directly learning policies.
Say: providing the appropriate is typically fairly straightforward: we know the task well enough. The real challenge is to identify the trade-offs: the number of possibilities is combinatorial in the number of features, making it near-impossible for our expert to come up with.
Convey intuition about what the algorithm is doing. Find reward that makes teacher look much better; and then hypothesize that reward and find the policy.
The intuition behind the algorithm, is that the set of policies generated gets closer (in some sense) to the expert at every iteration.
RL black box.
Inverse RL: explain the details now.
There is no reward function for which the teacher significantly outperforms thus-far found policies.

30. Algorithm example run 2 (E) (2) w(3) (1) w(2) w(1) (0) 1
At the end, if we properly mix 0, 1 and 2 (really 1 and 2) we can get quite close to the expert E.
(0) 1

31. Suboptimal expert case
Can match expert by stochastically mixing between 3 policies
In practice: for any w* one of ¼0, ¼1, ¼2 outperforms ¼*  pick one of them.
Generally: for k-dimensional feature space left to pick between k+1 policies 2
(2)
(*)
(1)
How to proceed in this case?
(0) 1

32. Feature expectation matching
If expert suboptimal then the resulting policy is a mixture of somewhat arbitrary policies which have expert in their convex hull.
In practice: pick the best one of this set and pick the corresponding reward function.
Next:
Syed and Schapire, 2008.
Ziebart+al, 2008.

33. Min-Max feature expectation matching Syed and Schapire (2008)2
Any policy in this area performs at least as well as expert.
How to find policy on pareto optimal curve in this area + corresponding reward function?
(*)
Intuition about feature expectation space here 1

34. Min-Max feature expectation matching Syed and Schapire (2008)2
(*)
Draw out w’s that would be OK. 1

35. Min max games minimizer pay-off matrix G
Example of standard min-max game setting:
rock-paper-scissors pay-off matrix:
maximizer
rock paper scissors
rock
paper
scissors
Nash equilibrium solution is mixed strategy: (1/3,1/3,1/3) for both players
minimizer
pay-off matrix G

36. Min-Max feature expectation matching Syed and Schapire (2008)
Standard min-max game:
Min-max inverse RL:
Solution: maximize over weights ¸ which weigh the contribution of all policies ¼1, ¼2, …, ¼N to the mixed policy.
Formally:
Remaining challenge: G very large! See paper for algorithm that only uses relevant parts of G. [Strong similarity with constraint generation schemes we have seen.]

37. Maximum-entropy feature expectation matching --- Ziebart+al, 2008
Recall feature matching in suboptimal expert case: 2
(3)
(2)
(*)
(1)
Can we actually capture a suboptimal expert?
(5)
(0)
(4) 1

38. Maximum-entropy feature expectation matching --- Ziebart+al, 2008
Maximize entropy of distributions over paths followed while satisfying the constraint of feature expectation matching:
This turns out to imply that P is of the form:
See paper for algorithmic details.
Can we actually capture a suboptimal expert?

39. Feature expectation matching
If expert suboptimal:
Abbeel and Ng, 2004: resulting policy is a mixture of policies which have expert in their convex hull---In practice: pick the best one of this set and pick the corresponding reward function.
Syed and Schapire, 2008 recast the same problem in game theoretic form which, at cost of adding in some prior knowledge, results in having a unique solution for policy and reward function.
Ziebart+al, 2008 assume the expert stochastically chooses between paths where each path’s log probability is given by its expected sum of rewards.

40. Lecture outline Examples of apprenticeship learning via inverse RL
Inverse RL vs. behavioral cloning
Historical sketch of inverse RL
Mathematical formulations for inverse RL
Max-margin
Feature matching
Reward function parameterizing the policy class
Case studies
Trajectory-based reward, with application to autonomous helicopter flight

41. Reward function parameterizing the policy class
Recall:
Let’s assume our expert acts according to:
Then for any R and ®, we can evaluate the likelihood of seeing a set of state-action pairs as follows:

42. Reward function parameterizing the policy class
Assume our expert acts according to:
Then for any R and ®, we can evaluate the likelihood of seeing a set of state-action pairs as follows:
Ramachandran and Amir, AAAI2007: MCMC method to sample from this distribution
Neu and Szepesvari, UAI2007: gradient method to optimize the likelihood

43. Lecture outline Examples of apprenticeship learning via inverse RL
Inverse RL vs. behavioral cloning
History of inverse RL
Mathematical formulations for inverse RL
Case studies: (1) Highway driving, (2) Crusher, (3) Parking lot navigation, (4) Route inference, (5) Quadruped locomotion
Trajectory-based reward, with application to autonomous helicopter flight

44. Simulated highway driving
Abbeel and Ng, ICML 2004; Syed and Schapire, NIPS 2007

45. Highway driving [Abbeel and Ng 2004] Input:
Teacher in Training World
Learned Policy in Testing World
Reward features: 0/1 features corresponding to the car being in each lane, each shoulder; 0/1 features corresponding to the closest car in the current lane discretized into 10 distances (total: 15 features).
Input:
Dynamics model / Simulator Psa(st+1 | st, at)
Teacher’s demonstration: 1 minute in “training world”
Note: R* is unknown.
Reward features: 5 features corresponding to lanes/shoulders; 10 features corresponding to presence of other car in current lane at different distances

46. More driving examples [Abbeel and Ng 2004]
Driving demonstration
Learned behavior
Driving demonstration
Learned behavior
In each video, the left sub-panel shows a demonstration of a different driving “style”, and the right sub-panel shows the behavior learned from watching the demonstration.

47. Crusher RSS 2008: Dave Silver and Drew Bagnell
Crusher outfitted for teleoperation
Dave Silver and Drew Bagnell drove this robot around to generate demostrations;
Learned using these functional gradient procedures; particularly a form of exponentiated functional gradient descent which we’ve been developing
RSS 2008: Dave Silver and Drew Bagnell

48. Learning movie Shows only one of the examples
Readily converges to desired trajectory

49. Max margin [Ratliff + al, 2006/7/8]
Side-by-side: hold out terrain and the learned cost function
Low cost: blue - road
High cost: redish tints - Steeply sloped mountainous regions
[Ratliff + al, 2006/7/8]

50. Max-margin [Ratliff + al, 2006/7/8]
Implemented learned cost function on the robot
playback of actual data from robot
[Ratliff + al, 2006/7/8]

51. Parking lot navigation
Reward function trades off:
Staying “on-road,”
Forward vs. reverse driving,
Amount of switching between forward and reverse,
Lane keeping,
On-road vs. off-road,
Curvature of paths.
[Abbeel et al., IROS 08]

52. Experimental setup
Demonstrate parking lot navigation on “train parking lots.”
Run our apprenticeship learning algorithm to find the reward function.
Receive “test parking lot” map + starting point and destination.
Find the trajectory that maximizes the learned reward function for navigating the test parking lot.

53. Nice driving style

54. Sloppy driving-style

55. “Don’t mind reverse” driving-style

56. Only 35% of routes are “fastest” (Letchner, Krumm, & Horvitz 2006)
Since people often care about getting to places quickly, we could assume they are taking the time-optimal route <click>
But some folks at Microsoft found that this isn’t that great of a predictor of actual behavior

57. Time Money Stress Skill
And if you think about why this might be, there are a lot of different factors that may influence our route decisions. How much will the route cost in terms of gas and tolls? How stressful will it be? Does the route match my skill level?
Ziebart+al, 2007/8/9

58. Time Distance Fuel Speed Safety Type Stress Lanes Skill Turns Mood
Context
Our view is that, like a fingerprint, each driver has a unique way in which they trade off these various factors under different contextual situations. Unfortunately, we have no way of knowing the perceived time, safety, stress, or skill requirements that a driver has for different routes. <click>
Instead we view drivers preferences based on the characteristics of the road network that we do have, and learn to “take a driver’s fingerprint” by analyzing their previously driven routes with respect to these road network features.
Ziebart+al, 2007/8/9

59. Data Collection 25 Taxi Drivers Over 100,000 miles Length Speed
Now I’ll talk about how we applied our approach for modeling the driving decisions of taxi drivers.
We obtained a road network dataset with features describing each road segments <click>
We additionally obtained contextual features based on accident reports, construction zones, congestion, and time of day <click>
We provided GPS loggers to 25 taxi drivers <click> And collected over 100,000 miles of their driving data
Length
Speed
Road Type
Lanes
Accidents
Construction
Congestion
Time of day
Over 100,000 miles
Ziebart+al, 2007/8/9

60. Destination Prediction
Here’s a video of how this works in practice. A driver is starting over here in the northeast and as he travels, the posterior distribution over destinations changes…
It may seem like you can do something more naïve here, and just “block off” sections that are passed, but drivers may be going out of their way a bit to hop on the interstate and you really need a more sophisticated approach like ours to handle such a case.

61. Quadruped Reward function trades off 25 features.
Height differential of terrain.
Gradient of terrain around each foot.
Height differential between feet.
… (25 features total for our setup)
Reward function trades off 25 features.
Hierarchical max margin [Kolter, Abbeel & Ng, 2008]

62. Experimental setup Demonstrate path across the “training terrain”
Run our apprenticeship learning algorithm to find the reward function
Receive “testing terrain”---height map.
Find the optimal policy with respect to the learned reward function for crossing the testing terrain.
Hierarchical max margin [Kolter, Abbeel & Ng, 2008]

63. Without learning

64. With learned reward function

65. Quadruped: Ratliff + al, 2007
Run footstep planner as expert (slow!)
Run boosted max margin to find a reward function that explains the center of gravity path of the robot (smaller state space)
At control time: use the learned reward function as a heuristic for A* search when performing footstep-level planning

66. Lecture outline Example of apprenticeship learning via inverse RL
Inverse RL vs. behavioral cloning
Sketch of history of inverse RL
Mathematical formulations for inverse RL
Case studies
Trajectory-based reward, with application to autonomous helicopter flight

67. Remainder of lecture: extreme helicopter flight
How does helicopter dynamics work
Autonomous helicopter setup
Application of inverse RL to autonomous helicopter flight

68. Helicopter dynamics 4 control inputs: Main rotor collective pitch
Main rotor cyclic pitch (roll and pitch)
Tail rotor collective pitch

69. Autonomous helicopter setup
On-Board Inertial Measurements Unit (IMU) data
Position data
Send out controls to helicopter
Kalman filter
Feedback controller

70. Motivating example How do we specify a task like this???
I want to start off with a motivating example.
Back in December 2006, we were working on getting our autonomous helicopter to perform expert-level aerobatics.
In our previous work we managed to perform some challenging aerobatic maneuvers, like flipping the helicopter over in place, and flying at high speeds.
It took some time to get things right, but we could achieve pretty reasonable performance on these tasks.
{play video}
Then we wanted to do a new maneuver---and this maneuver is called tic-toc.
So we spent several weeks working on this maneuver, trying to specify this trajectory by hand and fly it using our existing approach.
But we kept running into the same problem. It’s actually very hard to articulate just how the helicopter should behave during this maneuver.
Now, maybe, if we spent enough time, we could have come up with a trajectory that worked---but if we expect to solve autonomous aerobatics soon,
It seemed like we needed a much more automated procedure.
How do we specify a task like this???

71. Key difficulties Often very difficult to specify trajectory by hand.
Difficult to articulate exactly how a task is performed.
The trajectory should obey the system dynamics.
Use an expert demonstration as trajectory.
But, getting perfect demonstrations is hard.
Use multiple suboptimal demonstrations.
Simply average them?
How to find the experts “intended” trajectory from suboptimal demonstrations?
Unfortunately, it can be very difficult to specify the trajectory by hand.
(1) We may not be able to really explain how task is performed.
(2) Trajectory should obey the true system dynamics. Not strictly necessary, but it makes control more difficult otherwise.
(3) We had this idea: one way to satisfy these requirements is to use an expert demonstration.
(4) but, getting a perfect demonstration is far from easy.
After watching several suboptimal demonstrations though, it becomes very clear to a human spectator---even a spectator with no knowledge of helicopters---what the expert intended to do.
(5) So, maybe, if we use multiple demonstrations, even suboptimal ones, we can create an algorithm that extracts the trajectory that the expert was trying to demonstrate.

72. Expert demonstrations: Airshow
So, here is what our demonstration data looks like.
We asked our human pilot to fly our helicopter through the same aerobatic sequence several times.
And you’ll notice some clear suboptimalities; they’re all flying different paths, and at different speeds.
But even though they’re all different, you can clearly see that there is some intended trajectory that the pilot is trying to follow.

73. Expert demonstrations
Graphical model
Intended trajectory
Expert demonstrations
Time indices
To address these problems, we’re first going to propose a generative model for the expert’s suboptimal demonstrations.
We assume that that is some “hidden” intended trajectory, represented here by the variables ‘z’
(1) and that this trajectory follows a nonlinear dynamical system with gaussian noise.
(2) We also have a sequence of states from each expert demonstration, which we model as independent observations of one of the hidden trajectory states.
(3) Unfortunately, we don’t know _which_ hidden state corresponds to each of the demonstration states because the demonstrations occur at different rates.
So, we assume that there is some unknown set of time indices, the taus, that match these observations to the hidden states.
For instance, y1 could be an observation of z1, z2, or z3. And the variable tau1 tells us which of these is the case.
Intended trajectory satisfies dynamics.
Expert trajectory is a noisy observation of one of the hidden states.
But we don’t know exactly which one.
[ICML 2008]

74. Learning algorithm
Similar models appear in speech processing, genetic sequence alignment.
See, e.g., Listgarten et. al., 2005
Maximize likelihood of the demonstration data over:
Intended trajectory states
Time index values
Variance parameters for noise terms
Time index distribution parameters
(1) We note that this type of model appears in other fields. For instance, speech recognition, and genetic sequence alignment. And while the details of our inference algorithm differ slightly, there are some existing inference methods for models of this kind.
One example is the work of Listgarten, et. al., though the particulars there are somewhat different because they are working in a different setting.
As is common, we use a maximum likelihood approach where
we maximize the likelihood of all the demonstration data over
the intended trajectory states
the time index values
the variance parameters for the gaussian noise terms,
and the distribution parameters for the time indices

75. Learning algorithm If ¿ is unknown, inference is hard.
If ¿ is known, we have a standard HMM.
Make an initial guess for ¿.
Alternate between:
Fix ¿. Run EM on resulting HMM.
Choose new ¿ using dynamic programming.
How do we do this?
First, looking back at our model, we can see that inference when the time indices, \tau, are unknown inference is difficult because there are a large number of dependencies.
On the other hand, if we know the time indices then the model is simpler.
For instance, if we know that y1 corresponds to the intended state z2, then we don’t have to consider the direct dependence between z1 and y1, or z3 and y1.
In this case we just have a standard hidden markov model for a nonlinear-gaussian system.
So we use an alternating procedure.
We start out with a guess for tau.
Then we run EM on the resulting HMM to find the intended trajectory and distribution parameters.
And then we choose a new value for tau using a dynamic programming algorithm.
We give details for both of these steps in the full version of the paper on our website.

76. Details: Incorporating prior knowledge
Might have some limited knowledge about how the trajectory should look.
Flips and rolls should stay in place.
Vertical loops should lie in a vertical plane.
Pilot tends to “drift” away from intended trajectory.
It would be nice if we could incorporate the expert’s advice into our trajectory learning.
Some of this knowledge is very specific in nature:
We know that loops should look flat when you view them from the edge,
And we know that flips and rolls should stay roughly in place.
But some of it is more dynamic. For instance, we know that the pilot tends to “drift”
away from the intended trajectory slowly over time.
Because we have a very rich model representation we can actually incorporate this knowledge to improve the trajectories,
And we give examples of how to do that in the paper.

77. Results: Time-aligned demonstrations
White helicopter is inferred “intended” trajectory.
(start video)
So, what does the result look like?
The white helicopter is the resulting inferred trajectory that we think is the expert’s intended trajectory.
The demonstrations here are the same as before, but now they’re being shown after time-warping them to match up with the intended trajectory, which is shown in white. You can do this using the time indices computed by the algorithm.
You can see that the white trajectory really does capture what we might intuitively describe as the “typical” or “intended” behavior of these trajectories.

78. Results: Loops
Here’s a plot of the loops that were flown by our pilot (only 3 are shown).
(1) And here’s what the algorithm gives as the most likely intended path.
No prior knowledge of the loop’s structure was used, but you can see that the inferred trajectory is much closer to an ideal loop.

79. Learning / Optimal Control
High-level picture
Data
Dynamics Model
Trajectory +
Penalty Function
Reinforcement
Learning / Optimal Control
Reward Function R
Controller/ Policy p
So, let’s go back now.
We’ve just presented an algorithm for extracting a good trajectory that we can use for our reward function.
But if we actually want to fly this trajectory, we need to have a good dynamics model.
What’s high level picture here: need 1,2,3.
load balancing, pricing, ad placement, …
Many problems in robotics have unknown, stochastic, high-dimensional,
and highly non-linear dynamics, and offer significant challenges to
classical control methods. Some of the key difficulties in these
problems are that (i) It is often hard to write down, in closed form,
a formal specification of the control task (for example, what is the
objective function for "flying well"?), (ii) It is difficult to build
a good dynamics model because of both data collection and data
modeling challenges (similar to the "exploration problem" in
reinforcement learning), and (iii) It is expensive to find closed-loop
controllers for high dimensional, stochastic domains. In this
talk, I will present learning algorithms with formal performance
guarantees which show that these problems can be efficiently addressed
in the apprenticeship learning setting---the setting when expert
demonstrations of the task are available.
Car example:
What are we trying to learn here:
Current state  correct action = (throttle/brake, steering).
Let’s consider the fairly simple task of driving a car on a highway (in that, much simpler than general traffic situation). Keep in mind, just iconic example to get the point across. I will look at more complex, real-world problems (which we solved for the first time) later.
Build up policy complexity: try to maintain desired speed; if car ahead, switch to left lane; if car there, switch to right lane;
What if we want to take an exit soon? – Don’t switch into the left lane, except if you see a clear path to get back (what does that even mean, a clear path) etc.
Quickly becomes very complex. For supervised learning to work we need to
(a) Encounter all these different situations, so we can learn the appropriate action for them. Thus, e.g., our demonstration needs to include some pretty dangerous situations …
(b) We need to have a learning architecture (SVM, deep belief net, neural net, decision tree, …) capable of
capturing the extreme richness of our decisions.
Making decisions with so many possible actions (rather than 0/1 etc.)
Really, only will work in case of policy simplicity.
Examples include: bla …; these examples are extremely restrictive
Ignoring the structure of the problem.
When we drive a car we have
(a) specific objectives:
Avoid other cars
Get to our goal
Maybe talk on our cell phone, curse at other drivers, etc. ?
(b) an idea of how our car and other cars behave
== dynamics model
Key contribution: exploit this structure, and learn reward function and dynamics model from demonstrations rather than directly learn a policy. The controller that falls out generalizes much better b/c it “understands” the nature of the problem. Rather than “blindly” learned to mimick the teacher.
Step back here: I am not so much concerned about driving the car in the highway simulator; this is just an illustrative toy example. This idea applies generally to problems I mentioned before:

80. Experimental setup for helicopter
Our expert pilot demonstrates the airshow several times.
Learn (by solving a joint optimization problem):
Reward function---trajectory.
Dynamics model---trajectory-specific local model.
Fly autonomously:
Inertial sensing + vision-based position sensing  (extended) Kalman filter
Receding horizon differential dynamic programming (DDP) feedback controller (20Hz)
Learning to fly new aerobatics takes < 1 hour

81. Related work
Bagnell & Schneider, 2001; LaCivita, Papageorgiou, Messner & Kanade, 2002; Ng, Kim, Jordan & Sastry 2004a (2001);
Roberts, Corke & Buskey, 2003; Saripalli, Montgomery & Sukhatme, 2003; Shim, Chung, Kim & Sastry, 2003; Doherty et al., 2004.
Gavrilets, Martinos, Mettler and Feron, 2002; Ng et al., 2004b.
Maneuvers presented here are significantly more challenging and more diverse than those performed by any other autonomous helicopter.
Before we show our final results, we note that there has been quite a lot of excellent work on autonomous helicopter flight.
Bagnell & Schneider, etc. all developed helicopters that performed well during hover and low-speed flight. And these results were certainly ground-breaking.
Since then there have been a number of other impressive applications; but we note that most of them are restricted still to low speed, non-aerobatic flight.
In contrast, the work of Gavrilets et. al. in 2002, as well as Andrew Ng’s group at Stanford really pushed the envelope of helicopter control---
demonstrating that with some work, we could get autonomous helicopters to fly a limited range of very impressive aerobatic maneuvers.
But we want to point out that the maneuvers we’re now able to perform are significantly more challenging than those flown by any other autonomous helicopter, and not only that, but we can fly maneuvers in a truly vast variety.

82. Autonomous aerobatic flips (attempt) before apprenticeship learning
Task description: meticulously hand-engineered
Model: learned from (commonly used) frequency sweeps data

83. Results: Autonomous airshow
Let’s see the result.
So, while you’re watching this, feel free to think about how long it would take you to code the entire trajectory by hand.
{pause}
We’ve shown this video to a lot of people, and a common question is,
How long did this take you? How long would it take to put together a new airshow?
On typical day, we spend about 30 minutes collecting demonstrations from our pilot,
We spend another 30 minutes or so running our learning algorithm.
Then, it only takes about a minute to generate our diff. dyn. Prog. Controllers,
And shortly thereafter we’re flying the new airshow as well as you see here.
It’s a very routine procedure now; and you can come to the poster to see some of the other shows and maneuvers that we’ve done.

84. Summary Example of apprenticeship learning via inverse RL
Inverse RL vs. behavioral cloning
Sketch of history of inverse RL
Mathematical formulations for inverse RL
Case studies
Trajectory-based reward, with application to autonomous helicopter flight

86. Thank you.

87. Questions?

88. More materials
Lecture based upon these slides (by Pieter Abbeel) to appear on VideoLectures.net [part of Robot Learning Summer School, 2009]
Helicopter work:
for helicopter videos and video lecture (ICML 2008, Coates, Abbeel and Ng) and thesis defense (Abbeel)

89. More materials / References
Abbeel and Ng. Apprenticeship learning via inverse reinforcement learning. (ICML 2004)
Abbeel, Dolgov, Ng and Thrun. Apprenticeship learning for motion planning with application to parking lot navigation.
Boyd, Ghaoui, Feron and Balakrishnan. (1994). Linear matrix inequalities in system and control theory. SIAM.
Coates, Abbeel and Ng. Learning for control from multiple demonstrations. (ICML 2008)
Kolter, Abbeel and Ng. Hierarchical apprenticeship learning with application to quadruped locomotion. (NIPS 2008)
Neu and Szepesvari. Apprenticeship learning using inverse reinforcement learning and gradient methods. (UAI 2007)
Ng and Russell. Algorithms for inverse reinforcement learning. (ICML 2000)
Ratliff, Bagnell and Zinkevich. Maximum margin planning. (ICML 2006)
Ratliff, Bradley, Bagnell and Chestnutt. Boosting structured prediction for imitation learning. (NIPS 2007)
Ratliff, Bagnell and Srinivasa. Imitation learning for locomotoin and manipulation. (I. Conf. on Humanoid Robotics, 2007)
Ramachandran and Amir. Bayesian inverse reinforcement learning. (IJCAI 2007)
Syed and Schapire. A game-theoretic approach to apprenticeship learning. (NIPS 2008)
Ziebart, Maas, Bagnell and Dey. Maximum entropy inverse reinforcement learning. (AAAI 2008)