1. ReinforcementLearning: A package for replicating human behavior in R
University of Freiburg, Germany
Nicolas Pröllochs, Stefan Feuerriegel & Dirk Neumann
R User Conference
July 6, 2017

2. Pröllochs – Reinforcement learning in R
Motivation
Reinforcement learning
Recently gained a great deal of traction in studies that perform human-like learning
Learns optimal behavior through trial-and-error interactions with a dynamic environment
Approach appears quite natural by mimicking the fundamental way humans learn
Applications
Robotics and production control
Finance, economics & behavioral research
Artificial intelligence (e.g. learn playing Atari games based on input pixels (Mnih et al., 2015)) …
Trial-and-error learning based on continuous interactions between an agent an its environment
Pröllochs – Reinforcement learning in R

3. Reinforcement learning
Reinforcement learning problem
Feedback of the agent is restricted to a reward signal that indicates how well the agent is behaving
Any instruction concerning how to improve its behavior is absent
Agent-environment interface
Model consists of environment states S, agent actions A, and rewards R
At each time step t, the agent observes a representation of the environment’s state st and selects an action at
Subsequently, it receives a reward rt+1 and observes the next state st+1
Result: State-action function and optimal policy that specifies the best possible action in each state
Pröllochs – Reinforcement learning in R

4. Pröllochs – Reinforcement learning in R
Experience replay
Experience replay
Allows reinforcement learning agents to remember and reuse experiences from the past
Speed up convergence by replaying observed state transitions repeatedly to the agent, as if they were new observations collected while interacting with a system
Only requires input data in the form of sample sequences consisting of states, actions and rewards
Batch reinforcement learning
Collect sample sequences from a running system without direct interaction
Use the collected sample to learn an optimal policy for every state transition in the input data
Resulting policy can be applied to the system for validation purposes or to collect new data points (e.g. in order to iteratively improve the current policy).
Pröllochs – Reinforcement learning in R

5. Package: ReinforcementLearning
Motivation
Reinforcement learning results in a highly interpretable policy that allows to generate new insights for research and practice
The available tools in R are not living up to the needs of researcher and practitioners
ReinforcementLearning package is intended to partially close this gap
Main features
Learn an optimal policy from a fixed set of a priori known transition samples
Predefined learning rules and action selection modes
A highly customizable framework for model-free reinforcement learning tasks
Pröllochs – Reinforcement learning in R

6. Pröllochs – Reinforcement learning in R
Installation
Installation via CRAN install.packages("ReinforcementLearning") library(ReinforcementLearning) Installation via GitHub devtools::install_github("nproellochs/ReinforcementLearning")
Pröllochs – Reinforcement learning in R

7. Pröllochs – Reinforcement learning in R
Data format
Each training example consists of a state transition tuple (s,a,r,s_new):
s The current environment state.
a The selected action in the current state.
r The immediate reward received after transitioning from the current state to the next state.
s_new The next environment state.
Input data must be a dataframe in which each row represents a state transition tuple (s,a,r,s_new)
Pröllochs – Reinforcement learning in R

8. Generating experience
Read-in sample experience
data("tictactoe")
head(tictactoe, 5)
## State Action NextState Reward
## c X.B
## X.B c6 ...B.XX.B
## 3 ...B.XX.B c2 .XBB.XX.B
## 4 .XBB.XX.B c8 .XBBBXXXB
## 5 .XBBBXXXB c1 XXBBBXXXB B
Experience sampling using an environment function
environment <- function(state, action) {
...
return(list("NextState" = newState,
"Reward" = reward)) }
Pröllochs – Reinforcement learning in R

9. Pröllochs – Reinforcement learning in R
Case study: Gridworld
Learning problem
Agent navigates from a random starting position to a goal position on a 2x2 grid
Each cell represents one state (4 states)
Wall around the grid and between s1 and s4 (agent cannot move off the grid)
Crossing each square leads to a reward of -1
Reaching the goal position (s4) results in a reward of 10
|—————————|
| s1 | s4 |
| s2 s3 |
# Define state and action sets states <- c("s1", "s2", "s3", "s4") actions <- c("up", "down", "left", "right")
Pröllochs – Reinforcement learning in R

10. Pröllochs – Reinforcement learning in R
Case study: Gridworld
function (state, action) { next_state <- state if (state == state("s1") && action == "down") next_state <- state("s2") if (state == state("s2") && action == "up") next_state <- state("s1") if (state == state("s2") && action == "right") next_state <- state("s3") if (state == state("s3") && action == "left") if (state == state("s3") && action == "up") next_state <- state("s4") if (next_state == state("s4") && state != state("s4")) { reward <- 10 } else { reward <- -1 return(list(NextState = next_state, Reward = reward))
Environment function
A function that defines the dynamics of the environment
Function has to be manually implemented and must take a state and an action as input
Return value must be a list containing the name of the next state and the reward
|—————————|
| s1 | s4 |
| s2 s3 |
Pröllochs – Reinforcement learning in R

11. Pröllochs – Reinforcement learning in R
Case study: Gridworld
Generate sample experience
Package is shipped with the built-in capability to sample experience from a function that defines the dynamics of the environment
Allows to easily validate the performance of reinforcement learning, by applying the learned policy to newly generated samples
# Define state and action sets states <- c("s1", "s2", "s3", "s4") actions <- c("up", "down", "left", "right") # Sample N = 1000 random sequences data <- sampleExperience(N = 1000, env = env, states = states, actions = actions) head(data) ## State Action Reward NextState ## 1 s4 left -1 s4 ## 2 s2 right -1 s3 ## 3 s2 right -1 s3 ## 4 s3 left -1 s2 ## 5 s4 up -1 s4 ## 6 s1 down -1 s2
|—————————|
| s1 | s4 |
| s2 s3 |
Pröllochs – Reinforcement learning in R

12. Pröllochs – Reinforcement learning in R
Case study: Gridworld
Performing reinforcement learning
ReinforcementLearning(…) learns an optimal policy based on the input data
User is required to specify the column names of the individual tuple elements in 'data‘
Control parameters customize the learning behavior of the agent
# Define reinforcement learning parameters control <- list(alpha = 0.1, gamma = 0.5, epsilon = 0.1) # Perform reinforcement learning model <- ReinforcementLearning( data, s = "State", a = "Action", r = "Reward", s_new = "NextState", control = control )
Learning parameters
alpha Learning rate, set between 0 and 1
gamma Discount factor, set between 0 and 1
epsilon Exploration parameter, set between 0 and 1
|—————————|
| s1 | s4 |
| s2 s3 |
See package manual for details
Pröllochs – Reinforcement learning in R

13. Pröllochs – Reinforcement learning in R
Case study: Gridworld
# Print result print(model) ## State-Action function Q ## right up down left ## s ## s ## s ## s ## ## Policy ## s1 s2 s3 s4 ## "down" "right" "up" "right" ## Reward (last iteration) ## [1] -263
Inspect results
print(model) shows the state-action table and the optimal (i.e. the best possible action in each state)
|—————————|
| s1 | s4 |
| s2 s3 |
Pröllochs – Reinforcement learning in R

14. Pröllochs – Reinforcement learning in R
Case study – next steps
# Sample N = 1000 sequences using epsilon-greedy action selection
data_new <-
sampleExperience(
N = 1000, env = env,
states = states, actions = actions,
model = model,
actionSelection = "epsilon-greedy",
control = control ) A
Sample new experience using learned policy B
Update the existing policy
model_new <-
ReinforcementLearning(
data_new,
s = "State", a = "Action",
r = "Reward", s_new = "NextState",
control = control, model = model )
Pröllochs – Reinforcement learning in R

15. Pröllochs – Reinforcement learning in R
Case study – next steps C
Inspect results from updated policy and plot learning curve D
Plot learning curve
summary(model_new)
Model details
Learning rule: experienceReplay
Learning iterations: 2
Number of states:
Number of actions:
Total Reward:
Reward details (per iteration)
Min:
Max:
Average:
Median:
Standard deviation:
plot(model_new)
Pröllochs – Reinforcement learning in R

16. Pröllochs – Reinforcement learning in R
Summary
Reinforcement learning teaches an agent via interaction with its environment without any supervision other than its own decision-making policy
ReinforcementLearning R package allows an agent to learn optimal behavior based on sample experience consisting of states, actions and rewards (experience replay)
The package provides a remarkably flexible framework and is easily applied to a wide range of different problems
Package ReinforcementLearning is available on CRAN
Pröllochs – Reinforcement learning in R

17. Pröllochs – Reinforcement learning in R