1. Inventory Routing for Dynamic Waste Collection from Underground Containers Martijn Mes Department of Operational Methods for Production and Logistics University of Twente The Netherlands Monday, November 14, 2011 INFORMS Annual Meeting 2011, Charlotte, NC

2. INFORMS Annual Meeting 2011 OUTLINE  Case introduction  The company  The underground container project  Dynamic collection policies  The Inventory Routing Problem  Heuristic approach  Optimization approach  Conclusions 2/41

3. THE COMPANY  Twente Milieu: a waste collection company located in the Netherlands.  Main activity: collection and processing of waste.  But also: cleaning of streets and sewers, mowing of verges, road ice control, and the control of plague animals.  One of the largest waste collectors in the Netherlands when it comes to the #households connected to their network.  Yearly collection of around 225,000,000 kg of waste from a population of around 400,000 inhabitants. INFORMS Annual Meeting 2011 3/41

4. INFORMS Annual Meeting 2011 4/41

5. TYPE OF CONTAINERS Mini containersBlock containers One per household; have to be put along the side of the road on pre-defined days. One for multiple households; mostly located at apartment buildings; freely accessible. INFORMS Annual Meeting 2011 5/41

6. UNDERGROUND CONTAINERS INFORMS Annual Meeting 2011 6/41

7. INFORMS Annual Meeting 2011 7/41

8. ADVANTAGES UNDERGROUND CONTAINERS  Can be used at all places: apartments, houses, business parks, within the city centre etc. (≠ mini containers)  Don’t have to be emptied on pre-defined days (≠ mini containers)  Much larger then the block containers (typically 5m 3 which is 5 times the volume of a block container)  Only accessible with a personal card  Avoids illegal waste deposits (≠ block containers)  Enables the introduction of ‘Diftar’: charging waste disposal at different rates per kg depending on the type of garbage  Less odour nuisance due to solid locking (≠ block containers)  Contributes to an attractive environment (≠ block containers) INFORMS Annual Meeting 2011 8/41

9. INFORMS Annual Meeting 2011 9/41

10. USING THE UNDERGROUND CONTAINERS  Between 2009 and 2011, around 700 underground containers have been installed; 800 new containers will be added soon.  Containers are equipped with a motion sensor: the number of lid openings are communicated to Twente Milieu.  There is a static cyclic schedule that states which containers have to be emptied on what day. For example: container X has to be emptied every Tuesday and container Y has to be emptied on Friday once in the two weeks.  Every workday, a planning employee assigns trucks and drivers to the pre-defined containers. On Fridays, the planner uses the sensor information to include some additional urgent containers, thereby slightly deviating from the static cyclic schedule.  Why not using this sensor information for the whole selection process? INFORMS Annual Meeting 2011 10/41

11. DYNAMIC WASTE COLLECTION  Dynamic planning methodology: each day, select the containers to be emptied based on their estimated fill levels (using sensor information).  Research objective: To asses in what way and up to what degree a dynamic planning methodology can be used by Twente Milieu to increase efficiency in the emptying process of underground containers in terms of logistical costs, customer satisfaction, and CO 2 emissions. INFORMS Annual Meeting 2011 11/41

12. INVENTORY ROUTING PROBLEM  In the literature, our problem is known as a Inventory Routing Problem (IRP) which combines:  The vehicle routing problem (VRP)  Inventory Management \ Vendor Managed Inventory (VMI)  Trade-off decisions:  When to deliver a customer?  How much to deliver a customer?  Which delivery routes to use?  The current cyclic planning approach relates to the Periodic Vehicle Routing Problem (PVRP):  A multi-period VRP where customers have to be visited a given number of times within a given planning horizon (decision on visit combinations and routes). INFORMS Annual Meeting 2011 12/41

13. ILLUSTRATION OF THE IRP INFORMS Annual Meeting 2011  Basic question for IRPs: which customers to serve today and how to route our trucks? Parking Depot Enough empty space left Empty space needs to be delivered soon 13/41

14. SOLUTION METHODOLOGIES FOR IRPs  ILP\SDP\MDP\Heuristics:  Federgruen and Zipkin (1984), A Combined Vehicle Routing and Inventory Allocation Problem.  Campbell et al. (1997), The Inventory Routing Problem.  Bard et al. (1998), A Decomposition Approach to the Inventory Routing Problem with Satellite Facilities.  Chan et al. (1998), Probabilistic Analyses and Practical Algorithms for Inventory-Routing Models.  Berman et al. (2001), Deliveries in an inventory/routing problem using stochastic dynamic programming.  Kleywegt et al. (2002), The Stochastic inventory routing problem with direct deliveries.  Adelman (2004), A Price-Directed Approach to Stochastic Inventory/Routing.  Campbell et al. (2004), A decomposition approach for the inventory-routing problem.  Kleywegt et al. (2004), Dynamic programming approximations for a stochastic inventory routing problem.  Archetti et al. (2007), A branch-and-cut algorithm for a vendor-managed inventory-routing problem.  Bard et al. (2009), The integrated production–inventory–distribution–routing problem. INFORMS Annual Meeting 2011 14/41

15. OUR SOLUTION METHODOLOGY  Some characteristics of our problem:  Multi-vehicle: up to 7 trucks.  Multi-depot: 2 parking areas and 1 waste processing center.  Large-scale: expanding to 1500 customers (containers), which requires > 300 visits per day.  Long planning horizon: a short-term planning approach will postpone deliveries to the next period.  Dynamic environment: stochastic travel times and waste disposals → we have to be able to do replanning.  Changing environment: seasonal patters and special days.  To cope with these characteristics, we use a fast heuristic.  To anticipate changes in waste disposal, we equip our heuristic with a number of tunable parameters and optimize over these parameters. INFORMS Annual Meeting 2011 15/41

16. BASIC IDEA OF THE HEURISTIC  C reate initial routes based on MustGo’s (seed customers and workload balancing) and extend these routes with MayGo’s. INFORMS Annual Meeting 2011 Parking Depot MayGo MustGo 16/41

17. BASIC IDEA OF THE HEURISTIC INFORMS Annual Meeting 2011 Parking Depot Seed  C reate initial routes based on MustGo’s (seed customers and workload balancing) and extend these routes with MayGo’s. 17/41

18. BASIC IDEA OF THE HEURISTIC INFORMS Annual Meeting 2011 Parking Depot  C reate initial routes based on MustGo’s (seed customers and workload balancing) and extend these routes with MayGo’s. 18/41

19. BASIC IDEA OF THE HEURISTIC INFORMS Annual Meeting 2011 Parking Depot  C reate initial routes based on MustGo’s (seed customers and workload balancing) and extend these routes with MayGo’s. 19/41

20. BASIC IDEA OF THE HEURISTIC INFORMS Annual Meeting 2011 Parking Depot  C reate initial routes based on MustGo’s (seed customers and workload balancing) and extend these routes with MayGo’s. 20/41

21. BASIC IDEA OF THE HEURISTIC INFORMS Annual Meeting 2011 Parking Depot  C reate initial routes based on MustGo’s (seed customers and workload balancing) and extend these routes with MayGo’s. 21/41

22. BASIC IDEA OF THE HEURISTIC INFORMS Annual Meeting 2011 Parking Depot  C reate initial routes based on MustGo’s (seed customers and workload balancing) and extend these routes with MayGo’s. 22/41

23. BASIC IDEA OF THE HEURISTIC INFORMS Annual Meeting 2011 Parking Depot Extended with MayGo’s  C reate initial routes based on MustGo’s (seed customers and workload balancing) and extend these routes with MayGo’s. 23/41

24. ALGORITHM OUTLINE 1.Initial planning in the morning and replanning during the day. 2.Empty schedules in a non-preemtive way and keep them feasible. 3.Estimate the days left; MustGo’s (days left < MustGoDay); optional workload balancing (to avoid peaks on Mondays and Fridays); trucks to use; lower bound on the number of routes to use. 4.One seed per truck to (i) spread trucks across the area, (ii) realize container insertions both close and far from the depot, and (iii) balance the workload per route to anticipate later MayGo insertions; seeds based on largest minimum distance from the depot and other seeds; Assign routes to trucks. 5.Optionally, assign MustGo’s to trucks or routes in a balanced way (in anticipation of MayGo insertions). 6.Plan all remaining MustGo’s based on cheapest insertion costs. 7.Play MayGo’s: see next sheet. 8.Execute planning and perform replanning when needed. 1. Start 2. Initialize schedules 3. Initial computations 4. Plan seeds 5. Balance workload 6. Plan MustGo’s 7. Plan MayGo’s 8. End INFORMS Annual Meeting 2011 24/41

25. ADDING MAYGO CONTAINERS  MayGo’s: days left < MustGoDay+MayGoDay.  Planning extremes:  Wait first: MayGoDay=0  Drive first: MayGoDay=∞  The best option would be somewhere in between.  Selection of MayGo’s depend on the additional travel time (insertion costs) as well as the inventory (volume garbage).  Options:  Ratio insertion costs / inventory.  Relative improvement of this ratio compared to a smoothed historical ratio. A large positive value indicates an opportunity we should take.  Use (optional) limit on the number of MayGo’s. INFORMS Annual Meeting 2011 25/41

26. WILL IT WORK? A SIMULATION STUDY  Benchmark the current way of working and gain insight in the performance of our heuristic INFORMS Annual Meeting 2011 26/41

27. NUMERICAL RESULTS  Based on current deposit volumes and truck capacity, savings of 14.6% can be achieved, which consists of 40% reduction of penalty costs and 18% less travel distance.  Savings increase with decreasing truck capacities. INFORMS Annual Meeting 2011 27/41

28. OBSERVATIONS  Performance heavily depends on the parameter settings:  MustGoDay  MayGoDay  MaxPerDay (to limit MayGo’s)  NrTrucks  Slack capacity in trucks (to avoid replanning)  Etc.  Moreover, the “right settings” for these parameters heavily depend on the day of the week.  We could learn these parameters  Through experimentation in practice (online learning)  Through simulation experiments (offline learning) INFORMS Annual Meeting 2011 28/41

29. STOCHASTIC SEARCH  Where is the min\max of some multi-dimensional function when the surface is measured with noise?  In our case: at least a 10 dimensional function (using only the parameters MustGoDay and MayGoDay for 5 workdays). INFORMS Annual Meeting 2011 29/41

30. SIMULATION OPTIMIZATION  The optimization problem:  Simulation optimization:  The measurements follow from a simulation run.  Hence, these measurements are expensive.  Hence, we aim to reduce the required number of measurements.  Approaches: Heuristic methods (genetic algorithms, simulated annealing, tabu search etc.); Response Surface Methods (RSM); Stochastic Approximation (SA) methods; Bayesian Global Optimization (BGO). INFORMS Annual Meeting 2011 Vector or parameters to be adjusted (MustGoDay, MayGoDay, NrTrucks, etc., for all working days) Set of all parameter combinations Unknown function (no closed-form formulation) We can measure it Measurement will not be exact (we measure with noise y=f(x)+ε) 30/41

31. BAYESIAN GLOBAL OPTIMIZATION  Bayesian optimization involves three stages: 1.Designing the prior distribution (belief about f) 2.Updating this distribution using Bayes' rule 3.Deciding what values to sample next  Often, the belief about f conforms to a Gaussian process.  A Gaussian process is a collection of random variables {y x1, y x2,…} for which any finite subset has a joint multivariate Gaussian (Normal) distribution: INFORMS Annual Meeting 2011 Measurements Mean Kernel function (covariance between two variables) 31/41

32. MORE INFORMATION ON BGO  Daniel Lizotte (2008) Practical Bayesian Optimization, PhD Thesis.  Eric Brochu, Mike Cora and Nando de Freitas (2009) A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning.  INFORMS Tutorial by Peter Frazier today from 16:30-18:00 Bayesian Methods for Global and Simulation Optimization. INFORMS Annual Meeting 2011 32/41

33. OPTIMIZATION POLICIES WE CONSIDER  Sequential Kriging Optimization (SKO) by Huang et al. (2006) which is an extension of Efficient global optimization (EGO) by Jones et al. (1998) for noisy measurements. EGO: new points to be measured are selected based on “expected improvement” which strikes a balance between exploitation and exploration.  Knowledge Gradient for Correlated Beliefs (KGCB) by Frazier et al. (2009). KG: best we can do given we if there is only one measurement left to make.  Hierarchical Knowledge Gradient (HKG) by Mes et al. (2011). HKG: hierarchical aggregation technique that uses the common features shared by alternatives to learn about many alternatives from even a single measurement. INFORMS Annual Meeting 2011 33/41

34. ILLUSTRATION OF EGO: N=2 INFORMS Annual Meeting 2011 Source: Brochu et al. (2009) 34/41

35. ILLUSTRATION OF EGO: N=3 INFORMS Annual Meeting 2011 Source: Brochu et al. (2009) 35/41

36. ILLUSTRATION OF EGO: N=4 INFORMS Annual Meeting 2011 Source: Brochu et al. (2009) 36/41

37. ILLUSTRATION OF EGO: N=5 INFORMS Annual Meeting 2011 Source: Brochu et al. (2009) 37/41

38. ILLUSTRATION OF HKG [EXCEL DEMO] INFORMS Annual Meeting 2011 38/41

39. APPLICABILITY OF THESE POLICIES INFORMS Annual Meeting 2011  39/41

40. EXPERIMENTS WITH SKO  Experiment 1: 378 containers with 3 trucks: with a maximum of 113 emptying's per day.  Experiment 2: 700 containers, 50% higher deposit volumes and 2 trucks: with a maximum of 672 emptying’s per day.  Results are counterintuitive at first sight. Still, they result in additional savings of around 10%. INFORMS Annual Meeting 2011 MonTueWedThuFri MustGoDay4.00.0 1.20.0 MayGoDay4.0XX3.5X MonTueWedThuFri MustGoDay1.01.11.52.72.1 MayGoDay0.0 4.0 40/41

41. CONCLUSIONS  We proposed a fast heuristic suitable for Inventory Routing Problems involving a large number of customers.  Application of this heuristic to the waste collection problem is expected to result in a reduction of 18% in travel costs and 40% in penalty costs (due to waste overflow).  An optimization approach is preferred to anticipate changes in waste disposals. To enable this, we equipped our heuristic with several tunable parameters.  To optimize over these parameters we used techniques from Simulation Optimization and Bayesian Global Optimization (SKO, KGCB, HKG).  For our waste collection problem, this will result in additional savings of 10% in total costs (travel costs and penalty costs). INFORMS Annual Meeting 2011 41/41

42. QUESTIONS? Martijn Mes Assistant professor University of Twente School of Management and Governance Operational Methods for Production and Logistics The Netherlands Contact Phone:+31-534894062 Email: m.r.k.mes@utwente.nl Web: http://www.utwente.nl/mb/ompl/staff/Mes/