1. Genetic Algorithms Schematic of neural network application to identify metabolites by mass spectrometry (MS) Developed by Dr. Lars Kangas Input to Genetic Algorithm is measure of fitness from comparison of in silico and experimental MS Output are “chromosomes” translated into weights for neural network that is part of model for metabolite MS

2. Very Brief History of genetic algorithms: Genetic Algorithm were developed by John Holland in 60’s and 70’s Author of “Adaption in Natural and Artificial Systems” More recent book on the subject “An Introduction to Genetic Algorithms” by Melanie Mitchell (MIT Press, Cambridge, MA, 2002)

3. Natural adaption: Populations of organisms are subjected to environmental stress. Fitness is manifest by ability to survive and reproduce Fitness is passed to offspring by genes that are organized on chromosomes. If environmental conditions change, evolution creates a new population with different characteristics that optimize fitness under the new conditions

4. Basic tools of evolution Recombination (crossover) occurs during reproduction. Chromosome of offspring is a mixture of chromosomes from parents Mutation changes a single gene within a chromosome. To be expressed, organism must survive and pass modified chromosome to offspring

5. Artificial adaptation : Represent a candidate solution to a problem by a chromosome Define a fitness function on the domain of all chromosomes Define the probabilities of crossover and mutation. Select 2 chromosomes for reproduction based on their fitness Produce new chromosomes by crossover and mutation Evaluate fitness of new chromosomes Completes a “generation”

6. Artificial adaptation continued: In 50-500 generations create a population of solutions with high fitness Repeat whole process several times and merge best solutions Simple example: Find the position of the maximum of a normal distribution with mean of 16 and standard deviation of 4

7. Fitness function

8. Chromosome = binary representation of integers between 0 and 31 (requires 5 bits) 0 to 31 covers the range where fitness is significantly different from zero Fitness of chromosome = value of fitness function f(x) where x is decimal equivalent of a 5-bit binary Crossover probability (rate) = 0.75 Mutation probability (rate) = 0.002 Size of population, n = 4 Problem set up

9. Method to select chromosomes for refinement Calculate fitness f(x i ) for each chromosome in population Assigned each chromosome a discrete probability by Use p i to design a roulette wheel Divide number line between 0 and 1 into segments of length p i in a specified order Get r, random number uniformly distributed between 0 and 1 Choose the chromosome of the line segment containing r

10. 00100 = 4fitness = 0.0011pi = 0.044 01001 = 9fitness = 0.0216pi = 0.861 11011 = 27fitness = 0.0023pi = 0.091 11111 = 31fitness = 0.0001pi = 0.004  i f(x i ) = 0.0251 1 st generation: 5-bit binary numbers chosen randomly Assume the pair with largest 2 probabilites (01001 and 11011) are selected for replication

11. Assume a mixing point (locus) is chosen between first and second bit. Crossover selected to induce change Mutation is rejected as method to induce change

12. Evaluate fitness of new population 00100 = 4fitness = 0.0011pi = 0.015 01011 = 11fitness = 0.0457pi = 0.283 11001 = 25fitness = 0.0079pi = 0.599 11111 = 31fitness = 0.0001pi = 0.104  i f(x i ) = 0.0548 about 2 times that of the 1 st generation Repeat until fitness of population is almost uniform Values of all chromosomes should be near 16

13. Crowding: In the initial chromosome population of this example 01001 has 86% of the selection probability. Potentially can lead to imbalance of fitness over diversity Limit the ability of GA to explore new regions of search space Solution: penalize choice of similar chromosomes for mating

14.  and  are the mean and standard deviation of fitness in the population In early generations, selection pressure should be low to enable wider coverage of search space (large  ) In later generations selection pressure should be higher to encourage convergence to optimum solution (small  ) Sigma scaling allows variable selection pressure Sigma scaling of fitness f(x)

15. Positional bias: Single-point crossover lets near-by loci stay together in children One of several methods to avoid positional bias

16. Genetic Algorithm for real-valued variables Real-valued variables can be converted to binary representation as in example of finding maximum of normal distribution. Results in loss of significance unless one uses a large number of bits Arithmetic crossover Parents and Choose k th gene at random Children 0 <  <1

17. Discrete crossover: With uniform probability, each gene of child chromosome chosen to be a gene in one or the other parent chromosomes at the same locus. Parents and Child Normally distributed mutation: Choose random number from normal distribution with zero mean and standard deviation comparable to size of genes (e.g.  = 1 for genes scaled between -1 and +1). Add to randomly chosen gene. Re-scale if needed. More methods for real-valued variables

18. Using GA in training of ANN ANN with 11 weights: 8 to hidden layer, 3 to output w 1A w 1B w 2A w 2B w 3A w 3B w 0A w 0B w AZ w BZ w 0Z

19. Chromosome for weight optimization by GA Scaled to values between -1 and +1 Use methods crossover and mutation for real numbers to modify chromosome Fitness function: mean squared deviation between output and target

20. Use feed forward to determine the fitness of this new chromosome

22. Genetic algorithm for attribute selection Find the best subset of attributes for data mining GA is well suited to this task since, with diversity, it can explore many combinations of attributes.

23. WEKA’s GA applied to attribute selection Default values: Population size = 20 Crossover probability = 0.6 Mutation probability = 0.033 Example: breast-cancer classification Wisconsin Breast Cancer Database Breast-cancer.arff 683 instances 9 numerical attributes 2 target classes benign=2 malignant=4

24. Tumor characteristics 1.clump-thickness 2.uniform-cell size 3.uniform-cell shape 4.marg-adhesion 5.single-cell size 6.bare-nuclei 7.bland-chomatin 8.normal-nucleoli 9.mitoses Severity scores 5,1,1,1,2,1,3,1,1,2 5,4,4,5,7,10,3,2,1,2 3,1,1,1,2,2,3,1,1,2 6,8,8,1,3,4,3,7,1,2 4,1,1,3,2,1,3,1,1,2 8,10,10,8,7,10,9,7,1,4 1,1,1,1,2,10,3,1,1,2 2,1,2,1,2,1,3,1,1,2 2,1,1,1,2,1,1,1,5,2 4,2,1,1,2,1,2,1,1,2 Severity scores are attributes Last number in a row is class label Examples from dataset

25. Characteristic 1.clump-thickness 2.uniform-cell size 3.uniform-cell shape 4.marg-adhesion 5.single-cell size 6.bare-nuclei 7.bland-chomatin 8.normal-nucleoli 9.mitoses Severity score 5,1,1,1,2,1,3,1,1,2 5,4,4,5,7,10,3,2,1,2 3,1,1,1,2,2,3,1,1,2 6,8,8,1,3,4,3,7,1,2 4,1,1,3,2,1,3,1,1,2 8,10,10,8,7,10,9,7,1,4 1,1,1,1,2,10,3,1,1,2 2,1,2,1,2,1,3,1,1,2 2,1,1,1,2,1,1,1,5,2 4,2,1,1,2,1,2,1,1,2 Chromosomes have 9 binary genes gene k = 1 means k th severity score included Fitness: accuracy of naïve Bayes classification

26. Background on Bayesian classification

27. 27 posterior Class likelihoodprior normalization Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) Assign client to class with higher posterior With normalized, assign to class with P(C|x) > 0.5 P(C|x) = 0.5 is a discriminant in attribute space Bayes’ Rule for binary classification

28. 28 posterior Class likelihoodprior normalization Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) Prior is information relevant to classifying that is independent of attributes Class likelihood is probability that member of class C will have attribute x Bayes’ Rule for binary classification

29. 29 posterior Class likelihoodprior normalization Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) Prior = risk tolerance of bank (determined from loan-approval history) Class likelihood = is x like other high-risk applications? Example: Bayes’ Rule for loan approval

30. 30 posterior Class likelihoodprior normalization Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) Normalization is generally not necessary for classification Normalized Bayes’ rule for binary classification

31. Bayes’ Rule: K>2 Classes 31 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0)

32. 32 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) With class labels r i t, estimators are Estimate priors and class likelihoods from data set Number of examples in class is estimate of its prior. Assume members of class are Gaussian distributed. Mean and covariance parameterize class likelihood.

33. Assume x i are independent, offdiagonals of ∑ are 0, p(x|C) is product of probabilities for each component of x 33 Lecture Notes for E Alpaydın 2010 Introduction to Machine Learning 2e © The MIT Press (V1.0) naïve Bayes classification Each class has set of means and variances for the components of the attributes in that class

34. Open file breast-cancer.arff Check attribute 10 (class) to see the number of examples in each class Attribute selection using WEKA’s genetic algorithm method

36. benign malignant

37. Open file breast-cancer.arff Click on attribute 10 (class) to see the number of examples in each class Attribute selection using WEKA’s genetic algorithm method Click on any other attribute.

38. clump thickness Distribution of attribute scores (1 – 10) over examples in dataset Severity of clump thickness positively correlated with malignancy increasing severity 

39. Baseline performance measures use naïve Bayes classifier

40. Under the Select-Attributes tab of Weka Explorer Press choose button under Attribute Evaluator Under Attribute Selection find WrapperSubsetEval

41. Click on WrapperSubsetEval to bring up dialog box which shows ZeroR as the default classifier Find the Naïve Bayes classifier, click OK Evaluator has been selected

42. Under the Select-Attribute tab of Weka Explorer Press choose button under Search Method Find Genetic Search (see package manager in Weka 3.7) Start search with default settings including “Use full training set”

43. Fitness function: linear scaling of the error rate of naïve Bayes classification such that the highest error rate corresponds to a fitness of zero How is subset related to chromosome?

44. Any subset that includes 9 th attribute has low fitness Results with Weka 3.6

45. Increasing the number of generations to 100 does not change the attributes selected 9 th attribute “mitoses” has been deselected Return to Preprocess tab, remove “mitoses” and reclassify

46. Performance with reduced attribute set is slightly improved Slight improvement Misclassified malignant cases decreased by 2

47. Weka has other attribute selection techniques For theory see http://en.wikipedia.org/wiki/Feature_selection “information gained” is alternative to SubSetEval with GA search Ranker is the only Search Method that can be used with InfoGainAttributeEval