1. Yukun Chen Subramani Mani Discovery Systems Lab (DSL) Department of Biomedical Informatics Vanderbilt University May 2010

2.  Introduction  Datasets in the challenge  Probabilistic models  Querying methods  Other methods for active learning  Experiments and Results  Conclusion 2

3.  Active learning challenge is based on the pool-based active learning model.  Practically, labeling is costly but observational data is abundantly available at low cost.  Active learner could find the most informative instance and perform high learning accuracy with minimal querying cost.  In the challenge, we need to optimize the global score (ALC score) by implementing probabilistic prediction model, querying strategy, and more.  Learning from datasets in the challenge is not easy because the data is very sparse, is unbalanced for class label, has high dimensional feature space, and has missing values.  Uncertainty sampling with biasing consensus (USBC) is our basic active learning strategy for prediction and querying for labels. 3

4. 4 Development DatasetsFinal Datasets

5.  Random Forests (RF) classifier is the basic prediction model we used in this challenge.  We built a multi-model committee with multiple RF classifiers.  The final prediction was based on consensus posterior probability (CPP):  We also considered the variance of posterior probabilities from multiple models. The high-variance filter was used in querying method. 5

6.  Querying method ranks the samples based on the informative values, and outputs the most informative sample(s) to query.  Least confidence with bias (LCB) was our basic querying method.  The informative value of sample is a function of CPP and bias factor pp (the positive fraction for the current training set in active learning). 6 Pmax

7.  It is very important to have a good starting point on the learning curve in active learning, which is the prediction performance by knowing just one positive label.  Pure unsupervised learning method (for example the metric based on distance, similarity, clustering result) might not be good enough to make prediction.  We combined unsupervised and supervised learning:  (1) For all samples, compute the cosine similarity to the positive-labeled seed;  (2) Assign negative labels to K samples with smallest cosine similarity values;  (3) Train the training set with one given positive sample and K predicted negative samples by our multiple models, and predict for other samples.  Here is our comparison result between cosine similarity function and semi-supervised learning method for the initial AUC: 7 Dataset Name Initial AUC by Cosine Similarity Function Initial AUC by semi-supervised learning HIVA0.5441 +/– 0.41%0.6502 +/– 0.65% IBN_SINA0.8335 +/– 0.28%0.7900 +/– 0.28% NOVA0.5618 +/– 0.39%0.6853 +/– 0.38% ORANGE0.5661 +/– 0.51%0.5170 +/– 0.78% SYLVA0.6709 +/– 0.27%0.8958 +/– 0.22% ZEBRA0.3758 +/– 0.27%0.6751 +/– 0.48%

8.  For some datasets (ZEBRA, ORANGE, HIVA, and NOVA), our models did not have a good prediction when the size of training set is small. The bad initial performance could badly affect the global score based on learning curve in Log2 space (see the learning curves with respect to initial batch size).  We ran batch size validation to search for the minimal sufficient size of initial training set.  This prevented a significant drop in performance at the beginning for our prediction model.  Batch size validation result figure for ZEBRA, IBN_SINA and NOVA: 8

9. 9 051015 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 ZEBRA USBC IBATCH 2^14 (16384): Global score=0.5199 Log 2 (Number of labels queried) Area under the ROC curve 051015 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 ZEBRA USBC IBATCH 2^12 (4096): Global score=0.4218 Log 2 (Number of labels queried) Area under the ROC curve 051015 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 0.95 1 ZEBRA USBC IBATCH 2^10 (1024): Global score=0.3846 Log 2 (Number of labels queried) Area under the ROC curve 051015 0.4 0.5 0.6 0.7 0.8 0.9 1 ZEBRA USBC IBATCH 2^1 (2): Global score=0.2876 Log 2 (Number of labels queried) Area under the ROC curve 051015 0.4 0.5 0.6 0.7 0.8 0.9 1 ZEBRA USBC IBATCH 2^4 (16): Global score=0.3391 Log 2 (Number of labels queried) Area under the ROC curve 051015 0.4 0.5 0.6 0.7 0.8 0.9 1 ZEBRA USBC IBATCH 2^8 (256): Global score=0.3164 Log 2 (Number of labels queried) Area under the ROC curve Initial Batch: 256 Initial Batch: 4096Initial Batch: 1024Initial Batch: 16384 Initial Batch: 16Initial Batch: 2

10.  (1) Initialization:  (1.1) Run preprocessing steps (missing value imputation, PCA, etc) if needed.  (1.2) Assign batch size as the function of iteration, depending on the batch size validation result.  (2) Run semi-supervised learning for initial prediction and basic uncertainty sampling to rank and query samples.  (3) Run uncertainty sampling with biasing consensus (USBC) in the iterations of active learning:  (3.1) Add predicted negative samples into the training sets (if activated).  (3.2) Train by 5 RF models and predict for all unlabeled samples.  (3.3) Run high-variance filter (if activated).  (3.4) Run uncertainty sampling with bias to rank and query samples (Bias factor is the function of positive fraction and the size of training set).  (4) Output learning curves and global ALC score. 10

11. Dataset NameALCAUCInitial AUC Initial Batch Size Use Filter Use Predicted Negative HIVA 0.32330.7468 +/– 0.79%0.6502 +/– 0.65%1No IBN_SINA 0.87050.9960 +/– 0.09%0.7900 +/– 0.28%1NoYes NOVA 0.76750.9940 +/– 0.14%0.6853 +/– 0.38%16Yes ORANGE 0.20370.7630 +/– 1.11%0.5170 +/– 0.78%1NoYes SYLVA 0.94840.9990 +/– 0.04%0.8958 +/– 0.22%1No ZEBRA 0.51990.8318 +/– 0.56%0.6751 +/– 0.48%16384No 11 Dataset NameALCAUCInitial AUC Initial Batch Size Use Filter Use Predicted Negative Rank A0.36090.9615 +/– 0.39%0.75001NoYes9 B0.12970.6484 +/– 0.44%0.50001NoYes12 C0.18760.7715 +/– 0.52%0.45001No 12 D0.53900.9554 +/– 0.33%0.450016Yes 12 E0.62660.8939 +/– 0.39%0.730030000No 1 F0.78530.9976 +/– 0.09%0.55001No 3 The results for development datasets The results for final datasets

12. 12 Dataset: D; Global score: 0.54 Dataset: B; Global score: 0.13Dataset: C; Global score: 0.19Dataset: A; Global score: 0.36 Dataset: E; Global score: 0.63Dataset: F; Global score: 0.79

13.  For dataset E, the global score is benefited by the batch size validation. Semi-supervised learning generates a good starting point. We won on dataset E.  For dataset F, the learning curve based on USBC is acceptable except that the initial performance is not stable. We were ranked 3 rd on F.  For dataset D also the batch size validation was effective. The high- variance filter successfully helped prevent a significant drop in the curve. But the starting point is quite low.  For dataset A, USBC worked well when the size of training set was at least 64. However, the initial low performance hurt our global score.  Datasets B and C are the hardest datasets like HIVA and ORANGE. Our prediction models were not effective in these datasets. 13

14.  Our strategies consider more than prediction model and query model. Semi-supervised learning and batch size validation are also important parts of the active learning process.  Our methods need further evaluation using additional datasets.  The active learning challenge is still a very open problem to solve.  One possible future direction to explore is to automatically assign batch size as a function of predictive performance and informativeness. 14