Students: Lior Kupfer Pavel Lifshits Supervisor: Andrey Bernstein Advisor: Prof. Nahum Shimkin Technion – Israel Institute of Technology Faculty of Electrical Engineering Control and Robotics Laboratory

 Introduction  Notations  The System  The Learning Algorithm  Project Goals  Results ◦ Artificial Time Series (the AR case) ◦ Real Foreign Exchange / Stock Data  Conclusions  Future work Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 2 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.

 Using Machine Learning methods for trading ◦ One relatively new approach to financial trading ◦ Using learning algorithms to predict the rise and fall of asset prices before they occur ◦ An optimal trader would buy an asset before the price rises, and sell the asset before its value declines Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 3 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 4 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.  Trading technique ◦ An asset trader was implemented using recurrent reinforcement learning (RRL) suggest by Moody and Saffell (2001) ◦ It is a gradient ascent algorithm which attempts to maximize a utility function known as Sharpe’s ratio. ◦ We denote a parameter vector which completely defines the actions of the trader. ◦ By choosing an optimal parameter for the trader, we attempt to take advantage of asset price changes.

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 5 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.  Due to transactions costs which include ◦ Commissions ◦ Bid/Ask spreads ◦ Price slippage ◦ Market impact  Our constrains ◦ Can’t make arbitrarily frequent trade ◦ Can’t make large changes in portfolio composition.  Model assumptions ◦ Fixed position size ◦ Single security

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 6 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.  – Fixed quantities of security ◦ The price series is ◦ - The corresponding price changes  - Out position in each time step ◦  - System return in each time step ◦ 

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 7 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.  - Add i tive profit accumulated over T time periods ◦  - Performance criterion ◦ ◦ Is the marginal increase in the performance

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 8 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.  ◦ - Parameters vector (which we attempt to learn) ◦ - Information available at time t (in our case - the price changes) ◦ - Stochastic extension (noise) which level can be varied to control “exploration vs. exploitation”.  Our system is a single layer recurrent neural network:  Formally: ◦

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 9 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 10 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.  We use reinforcement learning (RL) to adjust the parameters of the system to maximize our performance criteria of choice  RL – an alternative between supervised & unsupervised learning  RL Framework: ◦ Agent  Environment ◦ Reward ◦ Expected Return ◦ Policy Learning  RL modus operandi ◦ Agent perceives the state of the environment s t and chooses an action a t. It subsequently observes the new state of the environment s t+1 and receives a reward r t. ◦ Aim : Learn a policy π (mapping from states to actions), which optimizes the expected return

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 11 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.  RL approaches ◦ Direct RL  In this approach, the policy is represented directly. The reward function (immediate feedback) is used to adjust the policy on the fly.  e.g. policy search ◦ Value function RL  In this approach,values are assigned to each state (or state‐action pair). Values correspond to estimates of future expected returns, or in other words, to the long‐term desirability of states. These values help guide the agent towards the optimal policy  e.g. TD-Learning, Q-Learning. ◦ Actor-Critic  The model is split into two parts: the critic, which maintains the state value estimate V, and the actor, which is responsible for choosing the appropriate actions at each state.

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 12 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.  In RRL we learn the policy by gradient ascent in the performance function  Performance function can be ◦ Profits ◦ Sharpe’s ratio ◦ Sterling ratio ◦ Double deviation  Moody suggests an additive and differentiable approximation for Sharpe’s ratio – the differential Sharpe’s ratio

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 13 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.  Now we develop ◦ Note: 

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 14 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.  Investigate Reinforcement Learning by policy gradient  Implement an automated trading system which learns it’s trading strategy by Recurrent Reinforcement Learning algorithm  Analyze the system’s results & structure  Suggest and examine improvement methods

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 15 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory. DataSetGoals Artificial Time Series Show the system can learn Analyze parameters effect Validate various model approximations Real Foreign Exchange EUR/USD Data Show the system can learn a profitable strategy on real data Search for possible improvements

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 16 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.  The challenges we face ◦ Model parameters: ◦ If and how to normalize the learned weights? ◦ How to normalize the input?  The averages changes over time (non stationary) – we assume that the change is slower than “how far back we look”

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 17 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.  r t – the return series are generated by AR(p) process  We analyze the effect of ◦ transaction costs ◦ quantization levels ◦ number of autoregressive inputs On ◦ Sharpe’s ratio ◦ trading frequency ◦ Profits  Effect of initial conditions

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 18 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 19 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 20 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 21 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 22 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory. % Long positions % Neutral positions % Short positions 2 Positions trader 51.2%0%48.8% 3 Positions trader 40%25.2%34.8% 3 Positions conservative trader 31.2%48.8%20%

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 23 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.  The prices series are of US Dollar vs. Euro exchange rate between 21/05/2007 until 15/01/2010 on 15 minutes data points  We compare our trader with ◦ Random strategy of Uniform distribution ◦ Buy and Hold strategy of Euro against US Dollar.

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 24 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.  No commissions

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 25 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.  With commissions (0.1%)

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 26 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.  RRL performs better than the random strategy  Positive Sharpe Ratios achieved in most cases  RRL seems to struggle during volatile periods  Large variance is a major cause for concern  Can’t unravel complex relationships in the data  Changes in market condition lead to waste of all the system’s learning during the training phase (but most learning systems suffer from this).

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 27 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.  When trading real data - the transaction cost is a killer  Normalizing the input series can be a real challenge ◦ The input series are non-stationary ◦ We assume the rate of change of average number of AR inputs to the system  Normalizing the weights – heuristically ◦ Threshold method leads to best results on both artificial & real data  Redundancy when input series are ARMA processes  Large training sessions under constant market conditions lead to overfitting

Outline Introduction Notations The system The Learning Algorithm Project Goals Results Artificial Series Real Forex Data Conclusions Future work 28 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.  Wrapping the system with risk management layer (e.g. Stop-Loss, retraining trigger, shut down the system under anomalous behavior)  Dynamical adjustment of external parameters (such as learning-rate)  Working with more than one security  Working with variable size positions  Working with coordination with another expert system (based on other algorithms)

29 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.  We would like to thank our project supervisor Andrey Bernstein for the guidance, Prof. Nahum Shimkin for advising us and allowing us to pursue a research project of our interest and sharing his experience with us.  Additionally we would like to thank Prof. Ron Meir & Prof. Neri Merhav for their time spent consulting us.  Special warm thanks to Gabriel Molina from Stanford university and Tikesh Ramtohul from University of Basel for their priceless help.

30 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.

31 L. Kupfer & P.Lifshits : “An automated trading system based on Recurrent Reinforcement Learning”, Technion - Israel Institute of Technology, Faculty of Electrical Engineering, Control and Robotics Laboratory.  [1] J Moody, M Saffell, Learning to Trade via Direct Reinforcement, IEEE Transactions on Neural Networks,Vol 12, No 4, July 2001  [2] Carl Gold, FX Trading via Recurrent Reinforcement Learning, CIFE, Hong Kong, 2003  [3] M.A.H. Dempster, V. Leemans, An Automated FX trading system using adaptive reinforcement learning, Expert Systems with Applications 30, pp , 2006