1. Maximizing long-term ROI for Active Learning Systems
Thesis Proposal
Maximizing long-term ROI for Active Learning Systems

2. Interactive Classification Goal: Optimize life-time Return On Investment
Majority transactions automatically cleared
Learning Model to Flag Transactions for Manual Intervention
Large volume (in millions) of transactions coming in
Domain specific transaction processing
Machine Learning model
Transactions processed successfully
Minority transactions flagged for auditing
Defining Characteristics
Expensive domain experts
Skewed class distribution (minority events)
Concept/Feature drift
Biased sampling of labeled historical data
Lots of unlabeled data
Goal: optimize roi over long term
Lower false positive rates based on learning model

3. Interactive Classification Applications
Fraud detection (Credit Card, Healthcare)
Network Intrusion detection
Video Surveillance
Information Filtering / Recommender Systems
Error prediction/Quality Control
Health Insurance Claims Rework
Characteristics
Skewed class distribution (Rare events)
Biased sampling of labeled data
Concept drift
Expensive domain experts

4. Health Insurance Claim Process - Rework
Underpayments
Overpayments
Why does rework happen: Complexity in the system, provider contract, member benefits, 20% of all claims need manual pricing

5. Why is solving Claims Rework important?
Inefficiencies in the healthcare process result in large monetary losses affecting corporations and consumers
For large (10 million+) insurance plan, estimated $1 billion in loss of revenue
$91 billion over-spent in US every year on Health Administration and Insurance (McKinsey study’ Nov 2008)
131 percent increase in insurance premiums over past 10 years
Claim payment errors drive a significant portion of these inefficiencies
Increased administrative costs and service issues of health plans
Overpayment of Claims - direct loss
Underpayment of Claims – loss in interest payment for insurer, loss in revenue for provider
Some statistics
33% of all the workforce is involved in taking care of these errors
For 6 million member insurance plan, $400 million identified overpayments
Source: [Anand and Khots, 2008]
For large (10 million+) insurance plan, estimated $1 billion in loss of revenue
Source: Discussion with domain experts

6. Interactive Classification Setting – Machine Learning Setup
Unlabeled + Labeled Data
Ranked List scored by classifier
Trained Classifier
Classifier trained from labeled data
Human (user/expert) in the loop using the results but also providing feedback at a cost
Goal: Maximize long-term Return on Investment (equivalent to the productivity of the entire system)

7. Factorization of the problem
Cost-Sensitive Exploitation
Factorization of the problem
Cost
(Time of human expert)
Exploration (Future classifier performance)
Exploitation (Relevancy to the expert)
Cost-Sensitive Active Learning
Exploration-Exploitation Tradeoffs
Related Work
Exploitation: learning to rank
Exploration: All active learning literature
Exploration+Exploitation: Reinforcement learning, has a notion of cost or budget but doesn’t manipulate cost proactively; the notion of showing similar examples to reduce the cost
Cost-sensitive active learning: budgeted active learning; feature-based active learning; multiple & noisy oracles;
Standard Ranking / Relevance Feedback
Active Learning

8. Factorization of the problem – characterization of the models
Cost (Time of human expert)
Exploration (Future classifier performance)
Exploitation (Relevancy to the expert)
Uniform
Each instance has same value
Variable
Each instance has different value which is dependent on the properties of the instance
Markovian
Each instance has dynamically changing value depending on the (ordered) history of instances already observed, in addition to the factors for Variable model

9. Example Cases for Factorization of Cost Model
Uniform: Speculative versus definitive language usage distinction for biomedical abstracts
[Settles et al., 2008]
Variable: Part Of Speech tagging
Annotation time dependent on the sentence length with longer documents taking more time to label [Ringger et al., 2008]
Markovian: Claims Rework Error Prediction
If similar claims are shown to the auditors in sequence reducing the cognitive switching costs, the time taken to label reduces [Ghani and Kumar, 2011]

10. Example Cases for Factorization of Exploitation Model
Uniform: Claims Rework Error Prediction
If we only account for the administrative overhead of fixing a claim [Kumar et al., 2010]
Variable: Claims Rework Error Prediction
If we take into account the savings based on the adjustment amount of the claim [Kumar et al., 2010]
Markovian: Claims Rework Error Prediction
Root cause detection [Kumar et al., 2010]

11. Example Cases for Factorization of Exploration Model
Uniform: Extracting contact details from signature lines
Random strategy gives results comparable to other strategies [Settles et al., 2008]
Variable: KDD Cup 1999, Network Intrusion detection
Sparsity based strategy gives good performance [Ferdowsi et al., 2011]
Dependent on the properties of the examples (or population) which can be pre-determined.
Markovian: Uncertainty based active sampling strategy
Most commonly used strategy

12. Problem Statement
How can we maximize long term ROI of active learning systems for interactive classification problems?

13. Proposed Hypothesis
Jointly managing the cost, exploitation and exploration factors will lead to increased long term ROI compared to managing them independently

14. Proposed Contributions
A framework to jointly manage cost, exploitation and exploration
Extensions of Active Learning along the following dimensions
Differential utility of a labeled example
Dynamic cost of labeling an example
Tackling concept drift

15. Proposed Framework Choice of Cost model Choice of Exploitation model
Choice of Exploration model
Utility metric
Algorithms to optimize the utility metric

16. Choice of Models Exploitation Model Cost Model Exploration Model
Markovian
Exploitation Model
Variable
Uniform
Complexity increases across the faces of the cube – color coding for the same
Uniform
Variable
Markovian
Variable
Markovian
Cost Model
Uniform
Exploration Model

17. Utility Metric Domain dependent
May or may not have a simple instantiation in the domain
Possible instantiations for Claims Rework domain
Return on Investment (Haertal et al, 2008)
Corresponds to the business goal of the deployed systems
Return: Cumulative dollar value of claims adjusted
Investment: Cumulative time (equivalent dollar amount) for auditing the claims
Does not take into account the classifier improvement/degradation
Amortized Return on Investment
Amortized return: Calculate the net present value of the returns based on the expected future classifier improvement
Return: Cumulative dollar value of claims adjusted + net present value of the increased returns due to future classifier improvement
Takes into account exploration and exploitation
Evaluation metric is given by the business goal
For Claims Rework, return on the claims saved wrt the cost

18. Algorithm to optimize the utility metric
Optimization straightforward if a well defined utility metric exists for the domain
Computational approximations may still be required for practical feasibility
Cases where a utility metric is not well defined based on the constituent cost/exploration/exploitation models, approaches to explore
Rank fusion based approach
Each model provides a ranking which are combined to get a final ranking
Explore relevant approaches from reinforcement learning
Upper Confidence Bounds for Trees (Kocsis and Szepesvári, 2006)
Multi-armed bandit with dependent arms (Pandey et al, 2007)
In Claims Rework domain, if cost model is defined in terms of audit complexity and the equivalent dollar conversion is not available/feasible

19. Interactive Classification Framework-Experimental Setup
Cost (Time of human expert)
Exploration (Future classifier performance)
Exploitation (Relevancy to the expert)
Ranked List
Trained Classifier (1,…,t-1)
Unlabeled Data (t)
Labeled Data (1,…,t-1)
Labeled Data (t)
Performance evaluation done on the set of labeled instances obtained at each iteration

20. Evaluation Compare various approaches with multiple baselines Random
Pure Exploitation
Exploitation=Var/Mar; Exploration=Uniform; Cost=Uniform
Pure Exploration
Exploration=Var/Mar; Exploitation=Uniform; Cost=Uniform
Pure Cost sensitive
Cost=Var/Mar; Exploitation=Uniform; Exploration=Uniform
Evaluation metric
Domain dependent
Claims Rework
Cumulative Return on Investment
Aligned with business goal
Based on the set of instances labeled in each iteration, obtain true return (dollar value saved) and true investment (cost of auditor’s time)

21. Preliminary results
Graph with results from framework

22. Generalizing Active Learning for Handling Temporal Drift
What is temporal drift?
Changing data distribution
Changing nature of classification problem
Adversarial actions
Related Work
Traditional active learning assumes static unlabeled pool
Stream-based active learning (Chu et al., 2011) assumes no memory to store the instances and makes online decisions to request labels
Not completely realistic as labeling requires human effort and is usually not real-time
Learning approaches from data streams with concept drift predominantly use ensembles over different time period (Kolter and Maloof, 2007)

23. Proposed Setup for Temporal Active Learning
Periodically changing unlabeled pool, corresponding to the experimental setup for interactive framework
Cumulative streaming pool
Recent streaming pool
Novel setup
Three components for handling temporal drift
Instance selection strategy
Type of model: Ensemble or Single
Instance or model weighing scheme

24. Proposed Instance Selection Strategies
Model Weight Drift Strategy
Feature Weight Drift Strategy
Feature Distribution Drift Strategy

25. Detecting Drift – Change in Models over Time
Claims rework domain
15 models built over 15 time periods
Similarity between the models based on cosine measure

26. Preliminary results Evaluation metric: Precision at 5 percentile
Represented in graph as percentage of the best strategy at a given iteration to give a sense that the mentioned strategies are not the best strategies at all iterations
Uncertainty begins to perform poorly at later iterations and feature drift based strategy starts performing better

27. Proposed Work
More experiments and analysis for claims rework data with data from different clients
More experiments based on synthetic dataset with longer observation sequence to analyze the performance of sampling strategies
Generation of synthetic data based on Gaussian Mixture models to mimic real data

28. Cost-Sensitive Exploitation
(Time of human expert)
Exploration (Future classifier performance)
Exploitation (Relevancy to the expert)
Also related work

29. More Like This strategy
Select Top m% claims
Labeled Data
Cluster
Rank
Ranked List scored by classifier
Online Strategy

30. Online “More-like-this” Algorithm
Require a labeled set L and an unlabeled set U
Train classifier C on L
Label U using C
Select top m% scored unlabeled examples UT
Cluster the examples UT U L into k clusters
Rank the k clusters using a exploitation metric
For each cluster ki in k
Rank examples in ki
For each example in ki
Query expert for label of
If precision of cluster ki is < Pmin and number of labels > Nmin, Next

31. Offline Comparison – MLT vs Baseline
9% relative improvement over baseline for Precision at 2nd percentile metric

32. Live System Deployment
~$10 Million savings /year for a typical insurance company
Number of claims audited:
Baseline system: 200
More-Like-This: 307
90% relative improvement over baseline
27% reduction in audit time over baseline

33. Summary Problem Statement
How to maximize long term ROI of active learning systems for interactive classification problems

34. Summary Thesis Contributions
Characterization of the interactive classification problem
Defining the cost/exploration/exploitation models
Uniform
Variable
Markovian
Generalization (Extensions?) of Active Learning along the following dimensions
Differential utility of a labeled example
Dynamic cost of labeling an example
Tackling concept drift
A framework to jointly manage these considerations

35. Summary Evaluation Empirical Evaluation of the proposed framework
Using evaluation metric motivated by real business tasks
Datasets
Real world dataset: Health Insurance Claims Rework
Synthetic dataset
Comparison with multiple baselines based on underlying cost/exploitation/exploration models
Methodological contribution
Novel experimental setup
Intend to make the synthetic dataset and its generators public

36. Summary Proposed Work: Temporal Active Learning
Creation of synthetic datasets
Evaluation and analysis of proposed strategies on synthetic and claims rework dataset

37. Summary Proposed Work: Framework for interactive classification
Evaluate multiple utility metrics/optimization algorithm for Claims Rework domain
Augment temporal drift synthetic data for evaluating framework
Evaluate multiple utility metrics/optimization algorithm for synthetic dataset
Cost Model
Exploitation Model
Exploration Model
Uniform
Variable
Markovian

38. Thanks

39. Our proposed approach – framework
Problem Description
High level factorization of the problem
Related Work
Triangle
Our proposed approach – framework
Broad categorization of the models
Choice of models
Choice of utility metric
Choice of optimization
Proposed work (various aproaches)
Temporal active learning
Some initial results
Cost sensitive exploitation
Summary
Problem statemnt
Contributions
Evaluation

40. Thesis Contributions
Problem Statement: How to generalize active learning to incorporate differential utility of a labeled example(dynamic/variable exploitation), dynamic cost of labeling an example, concept drift in a unified framework that makes the deployment of such learning systems practical
Contributions
Characterization of the interactive learning problem
Generalization of Active Learning along the following dimensions
Differential utility of a labeled example
Dynamic cost of labeling an example
Tackling concept drift
Cost-Sensitive Exploitation
A unified framework to solve these considerations jointly
First solution: Optimizing joint utility function based on cost, exploration utility and exploitation utility
Second solution: Using Upper Confidence Bound approach with contextual multi-armed bandit setup to incorporate the different factors
Empirical Evaluation of the proposed framework
Using evaluation metric motivated by real business tasks
Datasets
Synthetic dataset
Real world dataset: Health Insurance Claims Rework
Comparison with multiple baselines based on underlying factors

41. Situating the thesis work wrt related work
Efficiency & Representation
Feature level feedback
Feature acquisition
Batch active learning
Cost-sensitive
Active Learning
PrActive
Learning
Differential Utility
Dynamic cost
Concept Drift
Proactive
Learning
Unreliable Oracle
Oracle variation

42. Problem Statement
How to generalize active learning to incorporate differential utility of a labeled example(dynamic/variable exploitation), dynamic cost of labeling an example, concept drift in a unified framework that makes the deployment of such learning systems practical