Mathematical Modeling and Optimization: Summary of “Big Ideas”

A schematic view of modeling/optimization process Real-world problem Mathematical model Solution to model Solution to real-world problem assumptions, abstraction,data, simplifications optimization algorithm interpretation makes sense? change the model, assumptions?

Mathematical models in Optimization The general form of an optimization model: min or max f(x 1,…,x n ) (objective function) subject to g i (x 1,…,x n ) ≥ 0 (functional constraints) x 1,…,x n  S (set constraints) x 1,…,x n are called decision variables In words, the goal is to find x 1,…,x n that –satisfy the constraints; –achieve min (max) objective function value.

Types of Optimization Models Stochastic (probabilistic information on data) Deterministic (data are certain) Discrete, Integer (S = Z n ) Continuous (S = R n ) Linear (f and g are linear) Nonlinear (f and g are nonlinear)

What is Discrete Optimization? Discrete Optimization is a field of applied mathematics, combining techniques from combinatorics and graph theory, linear programming, theory of algorithms, to solve optimization problems over discrete structures.

Solution Methods for Discrete Optimization Problems Integer Programming Network Algorithms Dynamic Programming Approximation Algorithms

Integer Programming Programming = Planning in this context In a survey of Fortune 500 firms, 85% of those responding said that they had used linear or integer programming. Why is it so popular? –Many different real-life situations can be modeled as IPs. –There are efficient algorithms to solve IPs.

Topics in this class about Integer Programming Modeling real-life situations as integer programs Applications of integer programming Solution methods (algorithms) for integer programs (maybe) Using software (called AMPL) to solve integer programs

IP modeling techniques  Using binary variables  Restrictions on number of options  Contingent decisions  Variables (functions) with k possible values  Either-Or Constraints  Big M method  Balance constraints  Fixed Charges  Making choices with non-binary variables  Piecewise linear functions

IP applications  Facility Location Problem  Knapsack Problem  Multi-period production planning  Inventory management  Fair representation in electoral systems  Consultant hiring  Bin Packing Problem  Pairing Problem  Traveling Salesman Problem

Difficulties of real-life modeling The problems that you will encounter in the real world are always a lot messier than the ``clean'' problems we looked at in this class; there are always side constraints that complicate getting even a feasible solution. Most real world problems have multiple objectives, and it is hard to choose from among them. In the real world you must gain experience with how to adapt the idealized models of academia to each new problem you are asked to solve.

Utilizing the relationship between problems  Important modeling skill: –Suppose you know how to model Problems A 1,…,A p ; –You need to solve Problem B; –Notice the similarities between Problems A i and B; –Build a model for Problem B, using the model for Problem A i as a prototype. Examples: –The version of the facility location problem as a special case of the knapsack problem –Transshipment and Assignment problems as special cases of Maximum Flow Problem

Complexity of Solving Discrete Optimization Problems Two classes:  Class 1 problems have polynomial-time algorithms for solving the problems optimally. Examples: Min. Spanning Tree Problem, Maximum Flow Problem  For Class 2 problems (NP-hard problems) No polynomial-time algorithm is known; And more likely there is no one. Example: Traveling Salesman Problem Most discrete optimization problems are in the second class.

Three main directions to solve NP-hard discrete optimization problems: Integer programming techniques Approximation algorithms Heuristics  Important Observation: Any solution method suggests a tradeoff between time and accuracy. On time-accuracy tradeoff schedule: Brute force Integer programming Approximation algorithms Heuristics Least accuracy Most accuracy Worst timeBest time

Solving Integer Programs (IP) vs solving Linear Programs (LP)  LP algorithms – Simplex Method – Interior-point methods  IP algorithms use the above-mentioned LP algorithms as subroutines.  The algorithms for solving LPs are much more time- efficient than the algorithms for IPs.  Important modeling consideration: Whenever possible avoid integer variables in your model.

LP-relaxation-based solution methods for Integer Programs  Branch-and-Bound Technique  Cutting Plane Algorithms

Basic Concepts of Branch-and-Bound The basic concept underlying the branch-and-bound technique is to divide and conquer. Since the original “large” problem is hard to solve directly, it is divided into smaller and smaller subproblems until these subproblems can be conquered. The dividing (branching) is done by partitioning the entire set of feasible solutions into smaller and smaller subsets. The conquering (fathoming) is done partially by (i) giving a bound for the best solution in the subset; (ii) discarding the subset if the bound indicates that it can’t contain an optimal solution.

Summary of branch-and-bound Steps for each iteration: 1)Branching: Among the unfathomed subproblems, select the one that was created most recently. (Break ties according to which has larger LP value.) Choose a variable x i which has a noninteger value x i * in the LP solution of the subproblem. Create two new subproblems by adding the respective constraints x i   x i *  and x i ≥  x i * . 2)Bounding: Solve the new subproblems, record their LP solutions. Based on the LP values, update the incumbent, and the lower and upper bounds for OPT(IP) if necessary. 3)Fathoming: For each new subproblem, apply the three fathoming tests. Discard the subproblems that are fathomed. 4)Optimality test: If there are no unfathomed subproblems left then return the current incumbent as optimal solution (if there is no incumbent then IP is infeasible.) Otherwise, perform another iteration.

Importance of tight lower and upper bounds in branch-and-bound  Having tight lower and upper bounds on the IP optimal value might significantly reduce the number of branch-and-bound iterations.  For maximization problem, A lower bound can be found by applying a fast heuristic algorithm to the problem. An upper bound can be found by solving a relaxation of the problem (e.g., LP-relaxation).  If the current lower and upper bounds are close enough, we can stop the branch-and-bound algorithm and return the current incumbent solution (it can’t be too far from the optimum).

General Idea of Cutting Plane Technique  Add new constraints (cutting planes) to the problem such that (i) the set of feasible integer solutions remains the same, i.e., we still have the same integer program. (ii) the new constraints cut off some of the fractional solutions making the feasible region of the LP-relaxation smaller.  Smaller feasible region might result in a better LP value (i.e., closer to the IP value), thus making the search for the optimal IP solution more efficient.  Each integer program might have many different formulations. Important modeling skill: Give as tight formulation as possible. How? Find cutting planes that make the formulation of the original IP tighter.

Methods of getting Cutting Planes 1)Exploit the special structure of the problem to get cutting planes (e.g., bin packing problem, pairing problem)  Often can be hard to get  Topic of intensive research 2)More general methods are also available  Can be used automatically for many problems (e.g., knapsack-type constraints)

Branch-and-cut algorithms Integer programs are rarely solved based solely on cutting plane method. More often cutting planes and branch-and-bound are combined to provide a powerful algorithmic approach for solving integer programs. Cutting planes are added to the subproblems created in branch-and-bound to achieve tighter bounds and thus to accelerate the solution process. This kind of methods are known as branch-and-cut algorithms.

Network Models Minimum Spanning Tree Problem Traveling Salesman Problem Maximum Flow Problem Min. Spanning Tree and Maximum Flow Problems are in Class 1 (have polynomial-time algorithms for solving the problems optimally). Traveling Salesman Problem is in Class 2 (NP-hard problem).

Methods for solving NP-hard problems Three main directions to solve NP-hard discrete optimization problems: Integer programming techniques Heuristics Approximation algorithms We gave examples of all three methods for TSP. 2-approximation algorithm for TSP was given and analyzed in details.

Some Characteristics of Approximation Algorithms Time-efficient (sometimes not as efficient as heuristics) Don’t guarantee optimal solution Guarantee good solution within some factor of the optimum Vigorous mathematical analysis to prove the approximation guarantee Often use algorithms for related problems as subroutines The 2-approximation algorithm for TSP used the algorithm of finding a minimum spanning tree as subroutine.

Performance of TSP algorithms in practice A more sophisticated algorithm (which again uses the MST algorithm as a subroutine) guarantees a solution within factor of 1.5 of the optimum (Christofides). For many discrete optimization problems, there are benchmarks of instances on which algorithms are tested. For TSP, such a benchmark is TSPLIB. On TSPLIB instances, the Christofides’ algorithm outputs solutions which are on average 1.09 times the optimum. For comparison, the nearest neighbor algorithm outputs solutions which are on average 1.26 times the optimum. A good approximation factor often leads to good performance in practice.

Maximum Flow Problem We gave a linear program for the problem which can be solved by a special type of the Simplex method, called Network Simplex. (covered in Math 442/542) In this class, we covered a more efficient algorithm for the Maximum Flow Problem, the Augmenting Path Algorithm. Features of the algorithm: –Finding augmenting paths; –Residual capacities and residual networks; –Reversing flow when necessary.

Primal and Dual Problems in Optimization  For many optimization problems, there is a dual problem associated with the primal (original) problem.  The relationship between the primal and dual problems proves to be extremely useful in a variety of ways. Particularly, it  helps to find efficient algorithms for solving the primal problem;  provides arguments for proving the optimality (or close-to- optimality) of a primal solution.  There is a dual linear program associated with each primal linear program. One of key uses of LP duality theory is Sensitivity Analysis (discussed in Math442/542).  In this class, we discussed the dual of Maximum Flow Problem, known as Minimum Cut Problem.

Minimum Cut Problem A cut may be defined as a set of directed arcs such that if we remove the arcs of that set, no directed path from the source to the sink will be left. Weak duality property: Maximum flow value  capacity of any cut, i.e., capacity of any cut is an upper bound on the maximum flow. Strong Duality Property: Maximum flow value = Capacity of minimum cut Why do we care about minimum cuts? Minimum cuts provide an intuitive argument for proving that our current flow is maximum. There are many applications where we need to find not a maximum flow but a minimum cut. Minimum cuts give the bottleneck (weakest links) of a system. Observation: Identifying the bottleneck of a system (industrial, transportation, etc.) is an important goal in Engineering.

Applications of Maximum Flow and Minimum Cut Problems Transshipment Problem Assignment Problem Baseball Elimination Problem Each of these problems can be solved by creating and solving a related instance of maximum flow problem. On the other hand, the infeasibility of these problems can be shown using minimum-cut-based intuitive arguments.