UMass Lowell Computer Science Analysis of Algorithms Prof. Karen Daniels Fall, 2008 Lecture 9 Tuesday, 11/18/08 Linear Programming

Overview ä Motivation & Basics ä Standard & Slack Forms ä Formulating Problems as Linear Programs ä Simplex Algorithm ä Example ä High-Level Algorithm ä Correctness ä Roadmap ä Key Concepts ä Initial Basic Feasible Solution ä Duality ä Literature Case Study

Motivation & Basics

Motivation: A Political Problem source: textbook Cormen et al. Goal: Win election by winning majority of votes in each region. Subgoal: Win majority of votes in each region while minimizing advertising cost. 100,000 voters 200,000 voters 50,000 voters Thousands of voters who could be won with $1,000 of ads

Motivation: A Political Problem (continued) Thousands of voters representing majority. urban suburban rural source: textbook Cormen et al.

General Linear Programs real numbers real-valued variables Linear function Linear inequalities Linear constraints source: textbook Cormen et al. =

Overview of Linear Programming Convex feasible region Objective function Objective value source: textbook Cormen et al.

Standard & Slack Forms

Standard Form objective function constraints source: textbook Cormen et al.

Standard Form (compact) mxn matrix m-dimensional vector n-dimensional vectors source: textbook Cormen et al. Can specify linear program in standard form by (A,b,c).

Converting to Standard Form source: textbook Cormen et al.

Converting to Standard Form (continued) source: textbook Cormen et al. Negate coefficients Transforming minimization to maximization

Converting to Standard Form (continued) source: textbook Cormen et al. If x j has no non-negativity constraint, replace each occurrence of x j with x j ’ – x j ”. Giving each variable a non-negativity constraint New non-negativity constraints - 3(x 2 ’ – x 2 ”) x 2 has no non-negativity constraint

Converting to Standard Form (continued) source: textbook Cormen et al. Transforming equality constraints to inequality constraints

Converting to Standard Form (continued) source: textbook Cormen et al. Changing sense of an inequality constraint Rationale: (Rename variables for notational consistency.)

Converting Linear Programs into Slack Form source: textbook Cormen et al. for algorithmic ease, transform all constraints except non-negativity ones into equalities for inequality constraint: define slack slack variable instead of s basic variables non-basic variables

Converting Linear Programs into Slack Form (continued) source: textbook Cormen et al. objective function

Converting Linear Programs into Slack Form (continued) source: textbook Cormen et al. Compact Form: (N, B, A, b, c, v) set of indices of non-basic variables set of indices of basic variables Slack Form Example Compact Form negative of slack form coefficients basic variables

Formulating Problems as Linear Programs

Shortest Paths source: textbook Cormen et al. Single-pair shortest path: minimize “distance” from source s to sink t. Can we replace maximize with minimize here? Why or why not?

Maximum Flow source: textbook Cormen et al. Flow conservation Skew symmetry Capacity constraints

Minimum Cost Flow source: textbook Cormen et al. Flow target is prespecified. cost = 4 cost = 7 cost = 3 cost = 10 cost = 3 Total weighted flow cost = 27

Multicommodity Flow source: textbook Cormen et al. should be s i s i is source for commodity i t i is sink for commodity i

Simplex Method

Solving a Linear Program source: textbook Cormen et al. ä Simplex algorithm ä Geometric interpretation ä Visit vertices on the boundary of the simplex representing the convex feasible region ä Transforms set of inequalities using process similar to Gaussian elimination ä Run-time ä not polynomial in worst-case ä often very fast in practice ä Ellipsoid method ä Run-time ä polynomial ä slow in practice ä Interior-Point methods ä Run-time ä polynomial ä for large inputs, performance can be competitive with simplex method ä Moves through interior of feasible region

Simplex Algorithm: Example Basic Solution source: textbook Cormen et al. Standard Form Slack Form Basic Solution: set each nonbasic variable to 0. Basic Solution:

Simplex Algorithm: Example Reformulating the LP Model source: textbook Cormen et al. Main Idea: In each iteration, reformulate the LP model so basic solution has larger objective value Select a nonbasic variable whose objective coefficient is positive: x 1 Increase its value as much as possible. Identify tightest constraint on increase. For basic variable x 6 of that constraint, swap role with x 1. Rewrite other equations with x 6 on RHS. PIVOT leaving variable entering variable new objective value

Simplex Algorithm: Example Reformulating the LP Model source: textbook Cormen et al. Next Iteration: select x 3 as entering variable. PIVOT leaving variable entering variable New Basic Solution: new objective value

Simplex Algorithm: Example Reformulating the LP Model source: textbook Cormen et al. Next Iteration: select x 2 as entering variable. PIVOT leaving variable entering variable New Basic Solution: new objective value

Simplex Algorithm: Pivoting source: textbook Cormen et al. leaving variable entering variable Rewrite the equation that has x l on LHS to have x e on LHS Update remaining equations by substituting RHS of new equation for each occurrence of x e. Do the same for objective function. Update sets of nonbasic, basic variables.

Simplex Algorithm: Pseudocode source: textbook Cormen et al. to be defined later (detects infeasibility) initial basic solution optimal solution detects unboundedness

Correctness: Roadmap (Key Pieces) Theorem 29.13: Fundamental Theorem of Linear Programming For LP model in standard form, either: 1. exists optimal solution with finite objective function value & SIMPLEX returns one, or 2. infeasible & SIMPLEX returns INFEASIBLE, or 3. Unbounded & SIMPLEX returns UNBOUNDED Lemma 29.2: Basic solution feasible -> if SIMPLEX finds solution it is feasible; if reports unbounded, then model is unbounded Lemma 29.3: Algebraic lemma Theorem 29.10: LP duality: SIMPLEX primal result is optimal & dual is optimal Lemma 29.5: Iteration bound, cycling Lemma 29.4: Slack form uniqueness Lemma 29.6: Tie-breaking Lemma 29.8: Weak LP duality Corollary 29.9: Conditions for which feasible solutions for primal, dual programs are optimal Lemma 29.1: Pivot results Lemma 29.12: Infeasibility detection Lemma 29.11: L aux Lemma 29.7: Basic solution feasible -> SIMPLEX either reports unbounded or finds feasible solution in iterations

Proof of Correctness Key Concepts Initial Basic Feasible Solution

Finding an Initial Solution source: textbook Cormen et al. An LP model whose initial basic solution is not feasible

Finding an Initial Solution (continued) source: textbook Cormen et al. Auxiliary LP model L aux : 0 L aux L feasible -> 0 optimal for L aux L feasible <- 0 optimal for L aux

Finding an Initial Solution (continued) source: textbook Cormen et al.

Finding an Initial Solution (continued) source: textbook Cormen et al. Original LP model L aux L aux in slack form

Finding an Initial Solution (continued) source: textbook Cormen et al. PIVOT entering leaving most negative (restrictive) Feasible Basic Solution: (0, 4/5, 14/5, 0)

Finding an Initial Solution (continued) source: textbook Cormen et al. See textbook for proof (p. 815).

Duality Proof of Correctness Key Concepts

Linear Programming Duality source: textbook Cormen et al. max becomes min RHS coefficients swap places with objective function coefficients sense changes x variables go away y variables appear

Duality Example source: textbook Cormen et al.

Weak Linear Programming Duality source: textbook Cormen et al. Any feasible solution to primal LP has value no greater than that of any feasible solution to the dual LP. standard form

Weak Linear Programming Duality (continued) source: textbook Cormen et al.

Finding a Dual Solution source: textbook Cormen et al. Finding a dual solution whose value is equal to that of an optimal primal solution…

Optimality source: textbook Cormen et al.

Beyond Chapter 29…

Some Variations on Linear Programming ä Integer Programming ä Variables are restricted to integer values. ä Feasibility is NP-complete. ä Mixed Integer Programming ä Some variable(s) are restricted to integer values. ä Some variables are not restricted to integers. ä Quadratic Programming ä Quadratic objective programming ä Linear constraints ä Real-valued variables ä Convex Programming ä Generalization of Linear Programming

A Hierarchy of Related Types of Mathematical Programming

See docs portion of web site for additional information on Karp’s work. (courtesy of Nate)

My Applied Mathematical Programming Research ä Lagrangian Relaxation ä Apparel trim placement (Grinde & Daniels 2000) ä Polygon covering (Daniels et al. 2003) ä Box covering (England, Daniels 2008) ä Mixed Integer Programming ä Dynamic Channel assignment (Liu, Daniels, Chandra 2001, 2004) ä Quadratic Programming ä Support vector clustering enhancements (Lee, Daniels 2005, 2006) See

Linear Programming Resource Elementary Linear Programming with Applications (2 nd edition), by Kolman and Beck, Academic Press, 1980.