Sofus A. Macskassy Fetch Technologies sofmac@fetch.com Using Graph-based Metrics with Empirical Risk Minimization to Speed Up Active Learning on Networked Data Sofus A. Macskassy Fetch Technologies sofmac@fetch.com

Context Types of learning with labeled data Supervised learning is given a training set Semi-supervised (or transductive learning) is given a partially labeled data set Both learning methodologies seeks to induce a model to predict labels on unlabeled instances Active Learning seeks to help the learner induce the best model with the fewest labeled instances Picks the next training instance which will (probably) get the most boost in performance KDD 2009 - Sofus Macskassy

Motivation One well-known and popular active learning strategy iterates through all instances and computes the likely boost in performance if one were to label that particular instance. This is known as empirical risk minimization (ERM). Problem: ERM is very costly (induce and evaluate a new classifier for each class for each unlabeled instance) Proposed solution: Identify a small set of candidate instances on which to compute ERM KDD 2009 - Sofus Macskassy

Key Observation from Prior Work Prior work on pairing a graph-based semi-supervised learning method with active learning observed that ERM tended to pick instances that were in the center of clusters  Can we leverage this observation? KDD 2009 - Sofus Macskassy

Can we improve running time of ERM? Idea: Keep ERM, but limit ERM computation to the “best” candidates rather than all How? Use clustering and pick among the most central instances in each cluster? Closest in spirit to prior key observation Pick from the top-K most uncertain instances? Prior work on uncertainty sampling Use graph-based metrics to identify central instances and pick among the most central? Global and possibly a more consistent metric KDD 2009 - Sofus Macskassy

Selecting Best Candidates (1) Uncertainty labeling Use the current model to identify the unlabeled instances it is most uncertain of Uncertainty of vertex v: KDD 2009 - Sofus Macskassy

Selecting Best Candidates (2) Highest betweenness Betweenness centrality which instance has the most information flow = number of shortest paths between s and t = number paths that go through v Note: Need to compute all shortest-paths – can do this efficiently: O(nE) ~ O(n2) for sparse graphs KDD 2009 - Sofus Macskassy

Selecting Best Candidates (3) Highest closeness Closeness centrality which instance is “closest” to all other instances Note: Need to compute all pairwise distances – can do this efficiently: O(nE) ~ O(n2) for sparse graphs KDD 2009 - Sofus Macskassy

Selecting Best Candidates (4) Highest cluster closeness Central nodes in a cluster Cluster the graph Chose most central instances in cluster: = vertices in cluster that v belongs to Clustering details in paper KDD 2009 - Sofus Macskassy

Real world data is not “clean” (from CoRA data) all edges prob meth. (yellow) theory (green) genetic algs (red) rule learning (blue) neural nets (pink) RL (white) case-based (orange)

Empirical Study: Which method is best? Compare strategies (uncertainty, betweenness) to: Full ERM (current optimal) Random sampling (baseline) Metrics: accuracy and time-to-run Methodology Initialize: Randomly pick 1 instance per class Iteratively pick next instance using each methodology and record accuracy on remaining instances until 100 instances have been picked Repeat 10 times, record average accuracy KDD 2009 - Sofus Macskassy

11 Benchmark Data Sets WebKB [Craven 1998] (8 data sets) 4 computer science websites (sizes: 338-434) Each graph had a 6 class problem and a 2 class problem Industry classification [Bernstein et al. 2003] (2 data sets) 2 sources (prnews , Yahoo!) – sizes: 1798/218 Network = companies that co-occur in financial news stories 12 class problem CoRA [McCallum et al. 2000] (1 data set) 4240 academic papers Network = citations 6 class problem KDD 2009 - Sofus Macskassy

erm vs. betweenness vs. uncertainty KDD 2009 - Sofus Macskassy

Combine new strategies? None of the new strategies worked very well by themselves. However, the top ERM pick was often at the top of at least one strategy’s picks… New hybrid approach: Pick the top-K instances from each strategy (uncertainty, cluster closeness, betweenness) Pick the instance with the highest ERM score KDD 2009 - Sofus Macskassy

erm vs hybrid KDD 2009 - Sofus Macskassy

Which strategies were used in the hybrid? Dataset Cluster Closeness Uncertainty Sampling Betweenness Centrality Number of ties cora 11.00% 12.30% 76.90% 1 industry-pr 9.00% 36.80% 54.40% 2 industry-yh 1.90% 30.40% 68.90% 12 cornell-binary 12.60% 62.30% 36.90% 118 cornell-multi 3.50% 75.50% 29.90% 89 texas-binary 9.80% 40.10% 59.20% 91 texas-multi 16.30% 68.50% 28.50% 113 washington-binary 20.60% 72.50% 28.00% 211 washington-multi 25.70% 72.00% 25.10% 228 wisconsin-binary 8.80% 62.00% 27.40% 282 wisconsin-multi 24.00% 69.80% 25.40% 192 “larger” graphs KDD 2009 - Sofus Macskassy

Conclusions Empirical Risk Minimization is a strong active learning strategy but it is too slow We have shown that we can efficiently identify a small set of candidate nodes that contain the (close to) best instance as defined by ERM If the data is relational, we found that using graph metrics such as clustering, closeness and betweenness can identify a good candidate set Performs comparably to full blown ERM but runs an order of magnitude faster Potential for great speedups in larger graphs KDD 2009 - Sofus Macskassy

Future Work Using graph metrics to identify a candidate set seems like it has a lot of potential but it needs more work Need better understanding of metric behavior and how it relates to ERM For example, why did cluster closeness not do so well in practice? How do we incorporate labeled information into metrics? More work on network metrics needed KDD 2009 - Sofus Macskassy

Thank you Sofus Macskassy KDD 2009 - Sofus Macskassy