2. Introducation Defu Zhang Tel: 18959217108 Email: dfzhang@xmu.edu.cndfzhang@xmu.edu.cn ppt download : http://algorithm.xmu.edu.cn:10000/Download.aspx# p3 http://algorithm.xmu.edu.cn:10000/Download.aspx# p3 QQ: 366593121, share with your idea. Test methods: 50% experiment+50%paper Reference Books: The lastest paper 1

3. Introduction Optimization models Formulate the problem Model the problem Optimize the problem Implement a solution 2

4. Introduction Optimization problems Combinatorial Optimization problems (Discrete Optimization problems) involve finding an ordering, or assignment of a discrete set of objects which satisfies certain constraints arise in many domains of computer science and various application areas have high computational complexity (NP-hard) are solved in practice by searching an exponentially large space of candidate / partial solutions

5. Examples for combinatorial problems finding shortest/cheapest route (TSP) finding models of propositional formulae (SAT) Routing, Job shop scheduling, timetabling resource allocation it determines the allocation of a fixed amount of resources to a given number of activities in order to achieve the most effective results. Cutting and Packing Protein structure prediction

6. Continuous optimization Many machine learning problems can be cast as continuous optimization Classification where observations are inherently organized into labelled groups (classes) and a supervised process models an underlying discrimination function to classify unobserved samples. Example: Training data D = {(x 1,c 1 ), ……… (x n, c n )} where x i = feature or attribute vector and c i = class label (say binary-valued)

7. We have a model (a function or classifier) that maps from x to c e.g., sign( w. x ’ )  {-1, +1} We can measure the error E(w) for any setting of the weights w, and given a training data set D Optimization problem: find the weight vector that minimizes E(w) (general idea is “ empirical error minimization ” )

8. Feature Selection where a feature is considered an aggregation of one-or-more attributes, where only those features that have meaning in the context of the target function are necessary to the modeling function Clustering requiring a process to model an underlying where observations may be organized into groups based on underlying common features, although the groups are unlabeled discrimination function without corrective feedback. 7

9. Learning a minimum error decision boundary

10. Well-known problems Continue optimization problems Credit scoring problem Stock forecasting problem … Discrete optimization problems 3-CNF satisfiability Strip packing problem Vehicle routing problem … 9

11. Credit scoring problem The risk for financial institutions depends on how well they distinguish the good credit applicants from the bad ones. A lender commonly makes two types of decisions: first, whether to grant credit to a new applicant, and second, how to deal with existing applicants, including whether to increase their credit limits whatever the techniques used, it is critical that there is a large sample of previous customers with their application details, behavioral patterns, and subsequent credit history available (annual income, age, number of years in employment with their current employer, etc.). 10

12. Stock forecasting problem 11

13. 12 3-CNF satisfiability A literal in a boolean formula is an occurrence of a variable or its negation. A boolean formula is in conjunctive normal form, or CNF, if it is expressed as an AND of clauses, each of which is the OR of one or more literals. A boolean formula is in 3-conjunctive normal form, or 3-CNF, if each clause has exactly three distinct literals. In 3-CNF-SAT, we are asked whether a given boolean formula φ in 3-CNF is satisfiable.

14. Strip Packing problem In wood or glass industries, rectangular components have to be cut from large sheets of material. Given a set of rectangular objects (w i,h i ), 0≤i≤n and a large rectangular sheet with width W and unlimited height,the objective is to minimize the height (H) of sheet. For example, given 5 rectangular objects, how to cut them from large rectangular sheet such that H is minimum (see Fig. 1) and every packing rectangles have a fix direction.

16. Vehicle routing problem Objective: find the minimum traveling cost of K vehicles constraints: 1. Every vehicle starts from, and ends at, the central depot 2. The weight of the items loaded in a vehicle must not exceed the capacity of the vehicle D 3. each customer must be visited by one, and only one vehicle, once 4. Meet the need of customer 5. One central depot

17. central depot custom depot Vehicle route

18. Complexity Theory Undecidable problems could never have any algorithm to solve them even with unlimited time and space resources Decidable problems always has a yes or no answer. 17

19. Complexity Theory Complexity of Algorithms An algorithm needs two important resources to solve a problem: time and space. Polynomial-time algorithm Exponential-time algorithm 18

20. Complexity Theory Complexity of Problems is equivalent to the complexity of the best algorithm solving that problem. Tractable (or easy) problem if there exists a polynomial-time algorithm to solve it. Intractable(or difficult) problem if no polynomial-time algorithm exists to solve the problem. P,NP,NPC A decision problem A ∈ NP is NP-complete if all other problems of class NP are reduced polynomially to the problem A. 19

21. NP-hard NP-hard problems are optimization problems whose associated decision problems are NP-complete. 20

22. OPTIMIZATION METHODS (Algorithms)

23. Types of search methods: systematic ←→ local search deterministic ←→ stochastic sequential ←→ parallel Heuristic algorithms Problem-special heuristics Meta-heuristics(Local search, stochastic) Single-solution-based metaheuristics Population-based metaheuristics Hybrid heuristics Hyper Heuristics

24. Hybrid? If a hybrid makes sense then it is worth considering probably someone has already tried it E.g. SA & TS We could use both randomness of SA tabu ’ d moves of TS It is quite common that some kind of tabu list is added to other meta-heuristics

25. How to evaluate algorithm? Empirical analysis is most useful for heuristic methods Data generation (Benchmark) Public benchmark; Randomly generated benchmark; Algorithm implement (Software and hardware) Result analysis Running time Solution quality 24

26. Challenges Parameters settings You will probably want to experiment with your own meta-heuristics for a specific problem, because there is no general theory available that would help you to tune meta-heuristic parameters for any problem. No free lunch Tradeoff between time and solution quality

27. 26 Conclusions Heuristics and rules-of-thumb can vastly improve basic algorithms: They guide simple behaviour with application specific rules. The rules are not guess work but derived from understanding the problem. You can apply this sort of principle to all sorts of problem. Stochastic methods help to avoid some problems using randomisation based on physical phenomena: By randomly tweaking a solution and maybe making it slight worst in the short term, we may allow the possibility of improving it in the long term.

28. 27 Homework Download the following paper to read: Leung SCH, Zhang D, Sim KM. A two-stage intelligent search algorithm for the two-dimensional strip packing problem. European Journal of Operational Research 215(1) (2011) 57-69. Leung Stephen C.H. Zhou, Xiyue Zhang Defu Zheng, Jiemin. Extended guided tabu search and a new packing algorithm for the two-dimensional loading vehicle routing problem. Computers and Operations Research 38(1) (2010) 205-215.