1. © 2014 Fair Isaac Corporation. Confidential. This presentation is provided for the recipient only and cannot be reproduced or shared without Fair Isaac Corporation’s express consent. Exploring Predictions with a Powerful Research Tool Research Showcase Waley Liang Analytic Science, Lead Scientist FICO

2. © 2014 Fair Isaac Corporation. Confidential. ► Direct Benefits: functionalities migrate to products that you can use, e.g., FICO ® Model Builder, FICO ® Analytic Modeler Scorecard ► Indirect Benefits: FICO data analysts are using this tool to deliver improved analytic solutions and services for our customers Why Should You be Interested in an Internal Tool? 2

3. Agenda © 2014 Fair Isaac Corporation. Confidential. ► Prediction Exploration Platform (PEP) ► Motivation ► Key Analytic Capabilities ► Key Benefits ► Use Cases ► Demonstration ► Discussion 3

4. © 2014 Fair Isaac Corporation. Confidential. Universe of Machine Learning Algorithms PEP FICO Products Motivation 4 FICO Data Scientists

5. © 2014 Fair Isaac Corporation. Confidential. Key Analytic Capabilities of PEP 5 Predictive Algorithms ► Random Forest (RF) ► Stochastic Gradient Boosting (SGB) ► Fuzzy Segmented SGB (FSSGB) ► Sparse Logistic Regression (SLR) Support for Experiments ► Algorithm/methods comparisons ► Hyper-parameter grid searches Model Training Acceleration ► Via parallel/distributed computing Black-box Buster Reports ► Understanding variable importance and interactions ► Visualization of complex variable behaviors

6. © 2014 Fair Isaac Corporation. Confidential. Key Benefits of PEP ► ML algorithms match or beat traditional models Improves Prediction ► Importance of 100’s of variables for prediction quickly assessed ► Complex relations can be visualized ► Inform Scorecard segmentation decisions Generates insights ► Fast from data to powerful predictions, insights ► GUI based, no need for coding Boosts productivity ► User feedback guides addition of new algorithms Extensible for the future 6

7. © 2014 Fair Isaac Corporation. Confidential. Use Cases

8. © 2014 Fair Isaac Corporation. Confidential. ► Boost fraud detection rate at 1% review rate Challenge ► Developed several TEM and a baseline Logistic Regression model in PEP ► Compared models on a hold- out dataset Experiment ► Boosted fraud detection rate from 7% up to 14% ► Sped up modeling with hyperparameter search Results/Benefit s Auto Insurance Claim Frauds 8

9. © 2014 Fair Isaac Corporation. Confidential. ► Assess the benefit of a custom model versus a pooled model. Challenge ► Trained 2 models using PEP ► Model 1 on Client data ► Model 2 on Pooled data ► Compared performance (AUC, KS) on the client’s out- of-time data Experiment ► Custom model KS = 0.713 ► Pooled model KS = 0.709 ► Confirmed the pooled segmentation is effective ► Used ~1 week with PEP vs. ~9 weeks with traditional approaches Results/Benefit s Credit Card Behavior Scoring 9

10. © 2014 Fair Isaac Corporation. Confidential. ► Understand the effect of temporary inactivity on future attrition risk. Challenge ► Trained 3 models in PEP based on recency and frequency w/ and w/o interaction ► 2 predictors (additive) ► 2 predictors (interaction) ► 1000’s predictors (interaction) ► Compared lift and profits at top 5% of population Experiment ► Higher lift (6  6.8  7.5) and profits due to interaction and additional predictors ► Gained understanding of interactions via visualizations Results/Benefit s Credit Card Attrition 10

11. © 2014 Fair Isaac Corporation. Confidential. Demonstration

12. © 2014 Fair Isaac Corporation. Confidential. ► Are you using these or similar ML algorithms today? ► Do you see a role for increase in ML usage in your organization in the next 2 years? ► Which is the most important problem to tackle with ML? ► Credit scoring ► Fraud ► Marketing ► Other (what?) ► Are you interested in using ML as an exploratory tool only? ► Are you interested in implementing the analytic solution from ML in a production environment? ► Would you prefer to carry out ML using a GUI? ► With coding? We’d Love to Hear Your Feedback 12

13. © 2014 Fair Isaac Corporation. Confidential. This presentation is provided for the recipient only and cannot be reproduced or shared without Fair Isaac Corporation’s express consent. Thank You! Waley Liang waleyliang@fico.com

14. © 2014 Fair Isaac Corporation. Confidential. Learn More at FICO World Related Sessions ► Improvements in Recommendation Systems ► Addressing Attrition: Ultra-Dynamic Multi-Dimensional Attrition Analytics with Tree Ensemble Models Products in Solution Center ► Analytic Capabilities: Modeling Techniques and Innovations ► FICO ® Model Builder Experts at FICO World ► Gerald Fahner ► Michael Cohen Blogs ► http://ficolabsblog.fico.com/ 14

15. © 2014 Fair Isaac Corporation. Confidential. Please rate this session online! Waley Liang waleyliang@fico.com 15