1 Blockwise Coordinate Descent Procedures for the Multi-task Lasso with Applications to Neural Semantic Basis Discovery ICML 2009 Han Liu, Mark Palatucci, Jian Zhang Carnegie Mellon University Purdue University June 16, 2009 TexPoint fonts used in EMF: AAAAA A A A A AA A

2 Overview Sparsity and Multi-task Lasso Review Lasso and Multi-task Lasso (MTL) Discuss a new technique for solving MTL that is very fast and highly scalable Cognitive Neuroscience Goal: Want to predict the brain’s neural activity in response to some stimulus Show how we can use MTL to learn good features for prediction

3 Sparse Models Model Output Prediction Features: Relevant Features

4 Sparsity through Regularization Model: Loss Function + Penalty Term Squared Loss Sum of Absolute Weights Lasso

5 Joint Sparsity of Related Tasks Model Output Prediction Relevant Features Output Prediction-2 Output Prediction-3 Model

6 Joint Sparsity: Shared Features Model Output Prediction Shared Features: Relevant Features Output Prediction-2 Output Prediction-3

7 Lasso Model Multi-Task Lasso Model a.k.a Sum of sup-norm Sum over K tasks

8 Lasso Penalty Multi-Task Lasso Penalty Learns sparse solution: most elements zero, only relevant features non-zero Learns row-sparse solution: most rows zero, some rows have non-zero elements in each column Features Tasks

9 Solving the Lasso and Multi-task Lasso Lasso: LARS, Interior Point, Primal/Dual, Gradient Projection, Coordinate Descent Multi-task Lasso Interior Point (Turlach et al. 2004) Gradient Projection (Quattoni et al. 2009) Coordinate Descent (this work 2009)

10 What’s the best method? Single-task Lasso: Coordinate Descent Friedman, Hastie, Tibshirani (2008) Wu, Lange (2008) Duchi, Shalev-Shwartz, Singer, and Chandra (2008)

11 Coordinate Descent 1.Each iteration can be computed using a closed-form soft- thresholding operator 2.If solution vector is truly sparse, it can avoid updating irrelevant parameters through a simple check 3.Many computational tricks – warm start, covariance pre- computing, adaptive/greedy updates Why is coordinate descent so good for the Lasso? Can we develop a coordinate descent procedure for the Multi-task Lasso that has a similar closed-form update for each iteration?

12 Coordinate Descent for Multi-task Lasso Yes we can! Lasso: Multi-task Lasso: Main result: Can generalize soft-thresholding to multiple tasks using a Winsorization operator

13 Take Out Points from Part I Can efficiently solve the multi-task Lasso using a closed-form Winsorization operator that’s easy-to-implement This leads to a Coordinate Descent method that can take advantage of all the usual computational tricks

14 Overview Sparsity and Multi-task Lasso Review Lasso and Multi-task Lasso (MTL) Discuss a new technique for solving MTL that is very fast and highly scalable Cognitive Neuroscience Goal: Want to predict the brain’s neural activity in response to some stimulus Show how we can use MTL to learn good features for prediction

15 Neural Activity Prediction Model “apple” Mitchell et al. (2008) 0.836, 0.346, 0.000, … “airplane” eat | taste | ride, … 0.01, 0.001, , … Predicted activity

16 Neural Activity Prediction Can we train a model to automatically discover a good set of semantic features?

17 Neural Activity Prediction Model “apple” Large number of possible features This is a multi-task Lasso problem! Each “task” is the neural activity of a voxel

18 Evaluation Data Response: Collected fMRI images of neural activity for 60 words (5 examples from 12 categories)

19 BODY PARTSlegarmeyefoothand FURNITURE chairtablebeddeskdresser VEHICLES carairplanetraintruckbicycle ANIMALS horsedogbearcowcat KITCHEN UTENSILSglassknifebottlecupspoon TOOLS chiselhammerscrewdriverplierssaw BUILDINGS apartmentbarnhousechurchigloo PART OF A BUILDINGwindowdoorchimneyclosetarch CLOTHING coatdressshirtskirtpants INSECTS flyantbeebutterflybeetle VEGETABLESlettucetomatocarrotcorncelery MAN MADE OBJECTSrefrigeratorkeytelephonewatchbell Categories 60 Exemplars Data – fMRI Examples

20 Evaluation Data Response: Collected fMRI images of neural activity for 60 words (5 examples from 12 categories) Design: Each word represented as co-occurrence vector with 5,000 most frequent words in English

21 Evaluation Experiment Train model using 58 of 60 word stimuli (leave-two-out-cross- validation) Run Multi-task Lasso to select features from co-occurrences with 5,000 most frequent words Use those features in same linear model in Mitchell et al. (2008) Apply to predict activation for 2 held-out-words Label 2 held-out-words using cosine similarity with prediction Repeat for all (60 choose 2) = 1,770 iterations

22 “celery” “airplane” Predicted:Observed: fMRI activation high below average average Predicted and observed fMRI images for “celery” and “airplane” after training on 58 other words. [Mitchell et al. 08]

23 Results Random 25 features (words) Handcrafted 25 features from domain experts Learned features using Multi-task Lasso (set size of 25, 140, 350 features)

24 Results

25 Interpreting the Model 25 Top Features (Words) Selected by Multi-task Lasso ToolsCarDogWinePottery ModelStationBedroomBreakfastCup MadRentalsFishingCakeTip ArmsWalkCleaningCheeseGay RightWhiteFrontContentsResult

26 Interpreting the Model How does a feature contribute to observed neural activation?

27 Interpreting the Model Analyzing the weights learned for the “Tools” feature: Postcentral gyrus believed associated with pre-motor planning Superior temporal sulcus believed associated with perception of biological motion

28 Take Out Points: Multi-task Lasso Can learn common features of related tasks Scalable to thousands of features and tasks found with our coordinate descent procedure. Can build interpretable models useful for natural science applications Can learn features for neural prediction that perform better than handcrafted features on majority of fMRI subjects See Liu, Palatucci, and Zhang 2009: Blockwise Coordinate Descent Procedures for the Multi-task Lasso

29 Thanks to: ICML 2009 TexPoint fonts used in EMF: AAAAA A A A A AA Tom Mitchell W.M. Keck Foundation National Science Foundation Intel Research Google