Presentation on theme: "Traffic Light Control Using Reinforcement Learning"— Presentation transcript:
1 Traffic Light Control Using Reinforcement Learning Final PresentationTraffic Light Control Using Reinforcement LearningDaniel GoldbergAndrew Elstein
2 The ProblemTraffic congestion is estimated to cost Americans $121 billion in lost productivity fuel, and other costs.Traffic Lights are imperfect and contribute to thisUsually statically controlledA better method of controlling them can reduce waiting times significantly
3 ApproachImplement a “Reinforcement Learning” (RL) algorithm to control traffic lightsCreate a simulation of traffic to tweak and test traffic light optimizations
4 ImplementationIf minor adjustments were made to the algorithm, it could operate within existing infrastructureOptimally, a camera system and would be added
6 Simulation StructureTo build the simulation we created the follow Data Structures:CarsPosition, Destination, Velocity, Map, ColorRoadsLanesIndividual CellsIntersection location matrixIntersectionsPosition, Traffic LightsIn total, the simulation is coded inMATLAB with 3100 lines of codeCars Struct
7 Simulation Dynamics Cars are spawned randomly They follow an randomly generated path to destinationCars follow normal traffic rulesRoad Cells are discretized to easily simulate traffic, only one car can exist in each road cell. Cars move ahead one or two cells in each time-step, depending on the car's max velocity and whether there is an open spot.
10 Reinforcement Learning Theory Coordinating a system of lights to respond to current conditions can reap exceptional benefitThe theory cleverly merges probability, game theory and machine learning to efficiently control trafficIn our case, the expected value of each of a light’s possible states are calculatedWith this value function a game is played to maximize it, in turn minimizing waiting time
11 ResultsWrote a script to compare the smart algorithm to static On-Off-On-Off lights. Our algorithm reduced average waiting time—and thus traveling time— for a system with any number of cars Travelling time for our implementation was reduced by an average of 10%. There was a 15% reduction for sparse traffic systems from a static control, but only a 3% decrease for heavy congestion.
13 Extensions Fairness-weighted objective: ω = weighting constant ω = weighting constantt = current timeti = time of arrival for car iIf F(t) > 1, cars on road 1 get to goIf F(t) < 1, cars on road 2 get to go
14 Further Extensions Car Path optimization and rerouting Model expansion to traverse an entire cityInter-traffic-light communicationRetesting with increased computational resources for modeling accuracy and robustness
15 RL In the NewsSamah El-Tantawy, 2012 PhD recipient from the University of Toronto, won the 2013 IEEE best dissertation award for her research in RL.Her RL traffic model showed reduced rush-hour travel times by as much as 26 percent and is working on monetizing her research with small MARLIN-ATSC (Multi-agent Reinforcement Learning for Integrated Network of Adaptive Traffic Signal Controllers) computers.Multi-agent Reinforcement Learning for Integrated Network of Adaptive Traffic Signal Controllers.
16 ChallengesDifficult to understand data structures and how they would interactObject Oriented Approach vs. MATLAB’s index-based structuresUnderstand how cars would interact with each otherUnderstanding RL algorithmAdapting our model to use RL algorithmLimited computational resources