1. Extending the Multi- Instance Problem to Model Instance Collaboration Anjali Koppal Advanced Machine Learning December 11, 2007

2. Multi-Instance Learning: Set-up Given:A set of labeled data points (training set) and unlabeled data points (test set) Bag:Each data point is called a bag and is associated with a label (‘+’ or ‘-’) Instance: A Bag is described by a set of Instances. Each instance is described by a vector of features. Every instance also has a (hidden/unknown) label. Problem: Predict the class of an unlabelled bag. +

3. Multi-Instance Learning: Set-up Given:A set of labeled data points (training set) and unlabeled data points (test set) Bag:Each data point is called a bag and is associated with a label (‘+’ or ‘-’) Instance: A Bag is described by a set of Instances. Each instance is described by a vector of features. Every instance also has a (hidden/unknown) label. Problem: Predict the class of an unlabelled bag. +

4. Multi-Instance Learning: Set-up Given:A set of labeled data points (training set) and unlabeled data points (test set) Bag:Each data point is called a bag and is associated with a label (‘+’ or ‘-’) Instance: A Bag is described by a set of Instances. Each instance is described by a vector of features. Every instance also has a (hidden/unknown) label. Problem: Predict the class of an unlabelled bag. +

5. Multi-Instance Learning: Example Bag -> An image Instance -> A ‘region’ in the image Label -> {‘beach’ ‘not beach’} beach sand http://www.adrhi.com/Waimanalo-Beach.jpg water sky

6. Framework for Solving MI Problems Key Assumption: A bag is positive if at least one of its instances is predicted to be positive. A bag is negative if all its instances are predicted to be negative. + + –– –– – – – – – –– –– – – –

7. Framework for Solving MI Problems: Diversity Density Diversity Density of an instance: A measure of how close an instance is to instances of different positive bags while also being far away from all instances of negative bags. The instance with the maximum DD value is the most positive instance. O. Maron and T. Lozano-Pérez. A framework for multiple-instance learning. In Advances in Neural Information Processing Systems 10, pages 570-576. Cambridge, MA: MIT Press, 1998

8. Using DD to Solve MIL Problems Approach #1: Maron et al: Find the point in instance space with the maximum DD value. Call this p. Use the max DD value to compute a distance threshold t. A test bag b with instances x_i is classified as positive if min_i(distance(x_i, p)) < t ? ? ? ? p d1d1 d2d2 d3d3 d4d4

9. Using DD to Solve MIL Problems Approach #1: Maron et al: Find the point in instance space with the maximum DD value. Call this p. Use the max DD value to compute a distance threshold t. A test bag b with instances x_i is classified as positive if min_i(distance(x_i p)) < t + – – – p d 1 < t d 3 > t d 2 > t d 4 > t

10. DD-SVM Approach #2: (Y. Chen and J. Wang, image categorization by learning and reasoning with regions, The Journal of Machine Learning Research, 5, p.913-939) The previous approach uses one instance point (the one with the globally maximal DD value) as a prototype for all positive instances. Idea: Represent the positive instance space by a set of instance prototypes (locally max DD values). A bag is represented by a vector of minimum distances to each of the prototypes. d1d1 d2d2 d3d3 d4d4 d5d5 d1d2d3d4d5d1d2d3d4d5 Now train a SVM in the “bag-feature” space.

11. Extending DD-SVM : Motivation Instance Prototypes (Approach #2) are an improvement over single prototypes (Approach #1) because they allow for diversity in positive instances. e.g: A ‘beach scene’ could contain “water” or “sand” or “sky”. sand water sky

12. Extending DD-SVM : Motivation However Approach #2 assumes to some extent that the instances contribute independently to the bag’s class. e.g: The classification rule for ‘beach scene’ might be: “has water and sand but not only one”. For what is a beach without sand…

13. Extending DD-SVM : Proposal Instead of computing the minimum distance between an instance prototype and any instance in a bag compute the minimum distance between an instance prototype and a linear combination of all instances in the bag. α1α1 α2α2 α3α3 α4α4 α5α5 Instance Prototype For bag B with instances b i find the values of α i s such that dist( Σα i b i instancePrototype) is minimized – this is a standard QP Problem.

14. Extending DD-SVM : Initial Results Artificial Data Set Instances come from two normal distributions N1 and N2. number of bags = 200 with each bag containing 2 instances. Positive bags contain one instance from each of N1 and N2. Negative bags contain two instances from N2 (with probability t) +-- Train6061 Test3940 Total99101

15. Extending DD-SVM: Initial Results Classifier10-fold Cross-validation # SVsPrediction accuracy DD-SVM63.63%10262.0253% EDD-SVM65.29%9570.8861% SVM used: LIBSVM RBF Kernel 1-10.548310.5000 1-10.622010.5868 1-10.559310.5535 1-10.532910.5000 110.6002-10.5396 1-10.520510.6843 1-10.537910.5199 1-10.515210.5817 -110.5442-10.5075 -1-10.583410.5414 -110.5109-10.5146 -110.5388-10.5071 -1-10.510610.5661 Correct_LabelDD_labelDD_confEDD_labelEDD_conf

16. Extending DD-SVM: Initial Results Musk 2 Data Set (UCI) -102 bags (molecules) containing instances represented by 166 dimensions (conformations). The 2 classes are ‘musk’ and ‘non-musk’ - Average number of instances per bag ~ 64 – computationally very slow to calculate instance prototypes. Worked only with bags containing fewer than 20 instances (there were 63 such bags). Data+-- Train1411 Test2810 Total4221 Classifier10-fold Cross-validation # SVsPrediction accuracy DD-SVM71.4286%4066.6667% (14/21) EDD-SVM76.1905%4061.9048% (13/21)

17. Conclusions & Future Work 1.The Extended DD-SVM is a modification of the DD-SVM algorithm to model the influence of groups of instances instead of single, independent instances in multi instance labeling. 2.Initial results were interesting, but need to run more expansive tests to better understand its performance. 3.Useful to find datasets where a collaborative model is intuitively a good one (unclear if that is true for the MUSK dataset). 4.Other possible directions: -- Another way of extending DD-SVM: use the average distance of k closest instances to a prototype. -- Another way to model collaborations: convex combinations of nearest neighbours (instead of all instances).

18. Application mRNATS3TS1TS2 µ-rNA Bag ~ mRNA Instance ~ target sites (TS) Positive instance ~ legitimate TS Positive bag ~ protein expression