Presentation on theme: "The Business of Predictive Modeling December 17, 2013 Christine Hofbeck, FSA, MAAA Centroid Analytics, LLC."— Presentation transcript:
The Business of Predictive Modeling December 17, 2013 Christine Hofbeck, FSA, MAAA Centroid Analytics, LLC
AGENDA PART I -- INTRODUCTION PART II – MODELING 101 (Basic Steps) PART III – “GOLDEN QUESTION” PART IV – OPERATIONAL CONSIDERATIONS 2
INTRODUCTION Predictive modelling [sic] is the process by which a model is created or chosen to try to best predict the probability of an outcome. -- Wikipedia In practice: 3 VISUALIZE Customers Most Profitable Lines OTHER you are only limited by your creativity Identify patterns/ segment risks Develop business rules Improved decision making
Potential Applications 4 OPTIMIZE Operational Efficiency Distribution Channels Claims Management Pricing / Reserves DATA VISUALIZE Customers Most Profitable Lines or Products Target Marketing OTHER you are only limited by your creativity MINIMIZE Risk Fraud
Potential Applications (Life) 5 1.Triage UW decisions; implement STP for (more) applicants 2.Decrease purchase of traditional UW requirements by determining when they may not be necessary 3.Identify & target customers more likely to buy 4.Identify customers more likely to lapse – intervene if profitable, allow unhealthies to lapse 5.Inforce book management 6.Identify most desirable agents 7.Smart customer handling
Predictors A predictive model is made up of a number of predictors (“independent variables”), which are data elements likely to influence future behavior or results (“dependent variable”). 6 SEEK PARSIMONY DON’T USE ONE VARIABLEDON’T USE ALL VARIABLES the mean predicts the future but doesn’t tell us why… (“underfit”) exactly replicates the past… cannot predict the future (“overfit”)
BASIC STEPS (Modeling 101) 7 Define & Scope Data Prep Model Build Model Validation Implementation Review & Refine
Define & Scope 8 What is the budget? Consider IT, staff, data purchase, training, etc. Do we have the systems capacity to implement? How long do we have to build? To implement? Insource or outsource? How will the results be used? For whom/what are we trying to predict this? (“unit of exposure”) Exactly what are we trying to predict?
Data Prep * sometimes the most time intensive step of modeling 9 MODELING DATASET INTERNAL DATA o # years o accuracy o ability to access o primary key EXTERNAL DATA o match rate o cost – to model o cost – to use o frequency of update Consider both expected & unexpected relationships – creativity in data exploration can be the key to your competitive edge!
Data Prep (cont’d) 1.COMBINE various data sources 2.CONVERT to desired exposure unit or format 3.CORRECT inaccurate data 4.INSPECT to remove variables: - Too many blank values that cannot be imputed - All/most values the same - Data cannot be relied upon - Data will not be captured going forward - Legal advice not to use 5.BUCKET (“bin”) values 10
Model Build (cont’d) UNIVARIATE ANALYSIS – test each variable one by one to see which ones may be predictive. 11 MULTIVARIATE ANALYSIS – examine multiple variables in different groups to obtain the best, USABLE results – remember parsimony! INTERACTIONS – which variables can be combined into a “mega variable” to improve results (i.e., does 1+1 = 1.5? does 1+1 = 3?) Complicate the model (add variables, interactions) and simplify the model (remove variables, bin) to find the preferred combination.
Model Build (cont’d) Various tests can be used to determine variable inclusion: 12 STATISTICAL CONSISTENCY JUDGMENT P-values Cramer’s V Confidence intervals Type III tests Apply business knowledge to assess whether suggested relationships make sense Of patterns - Over time Over random parts of a dataset
Model Validation ACTUAL vs. EXPECTED-- how close did we get? Generally, a subset of the data is withheld during the modeling process for validation: Model validation graphs are useful for communicating model performance to non-technical audiences. 13 OUT OF TIME withhold most recent data OUT OF SAMPLE withhold randomly generated % of records
Model Validation – Sample Chart 14 Outcome Decile
Implementation BUSINESS RULES What decisions will be made based on the prediction? May vary by location, business, rate group, etc. SYSTEM BUILD Scoring engine (collects data & calculates predictions) Decision tool (executes business rules) User interface TRAINING Anyone who will interact with the model must understand what it does and why 15
Review & Refine REPORTING How close did we get to the goal? How far did we exceed it? Multiple reporting packages required for varied audiences, for example: Executives – highlights in aggregate by zone, business unit, product Actuaries – detailed results by variable, state, rate group Marketing – by broker/agent, location Underwriting – by underwriter as a performance measure Frequency of update – weekly, monthly, quarterly, yearly? Method of calculation – automated? ad hoc?
Review & Refine (cont’d) MODEL UPDATES WHY? As target customer is attained, characteristics of inforce book will change Business goals/strategies may change New data may become available Tolerance for certain characteristics may change HOW? Update current variable relativities (“recalibrate”) Start over - search for more predictive variables (“recast”) HOW OFTEN? 17
Advantages of Modeling Over 18 1.Many additional and often unconventional variables may be examined 2.Modeling a particular variable controls for the effects of other included variables – we don’t risk double counting or attributing effects to the wrong variables 3.Traditional approaches segment data into smaller categories which impact credibility 4.Interactions are introduced The above advantages can lead to improved accuracy, enhanced business and strategic benefits, more reliable assumptions, improved risk mitigation, etc. Traditional Approaches
THE “GOLDEN QUESTION” Through brainstorming, feedback loops, and data review, determine what single characteristic (“golden question”) will define your target 19
OPERATIONAL CONSIDERATIONS 1.Executive & cross-functional support 2.Time/cost versus depth of investigation 3.Strategic modeling process 4.Cross-functional involvement throughout build 5.Thorough training 20
Executive & Cross-Functional Support 21 If target users don’t support the model, they will resist using it. Gaining complete support can be difficult: 1.Resistance to change 2.Concern that model results will highlight current deficiencies 3.Lack of understanding of predictive models
Support (cont’d) 22 We’ve always done it this way, and it’s worked. I don’t see a reason to change anything. I will have to take on additional work associated with new processes. My workflow will double (triple). I don’t know how to explain this to a broker/agent so I don’t want to use it. I found one outlier so the model must be wrong. I already have an established plan. I know who our target customer is. The model will suggest that my current method is incorrect, which will reflect poorly on my performance/reputation. My position will be eliminated if a model is now used to select risks. My expertise must not be important to the company.
Time/Cost vs. Depth of Investigation 23 The process of building and implementing a model can typically be quite lengthy – longer than most expect OR Remember that a simple model does not necessarily indicate a simple study! Simpler Study (3-12 months) Results more conservative Perhaps internal data only Generous binning Limited interactions May be appropriate if goal is a general sense of direction More thorough investigation Additional time Additional development cost Possible greater payoff through enhanced segmentation and data exploration
Strategic Modeling Process TARGET PREDICTION/USE o Ensure target is appropriate for the intended use o While many ideas are interesting, you may wish to focus on those which are actionable STATISTICAL SIGNIFICANCE vs. ULTIMATE IMPACT o The most statistically significant model may not be the most impactful o Consider ease of implementation, repeatability, updates o Identify when “less is more”! FLEXIBILITY o Allow for unexpected insights which could lead to unanticipated changes in business strategy or process o Sometimes the insights gained from the journey will prove more important than the planned goal
Cross-Functional Involvement Data, product & IT experts, legal advisors, and model users must remain engaged throughout the model build 25 Insight from functionsInsight to functions Eases training and implementation Keep modelers apprised of changes in strategyLegal considerations around certain variables
Thorough Training 26 The model isn’t done when it’s done. Who will provide the training? Who is most appropriate to provide training? Modeling team General training team Functional experts Consulting team* Other No clear answer – but this must be thoughtfully considered and appropriately executed to reap the full benefits of the model which was built *Consider what information may be shared (non-proprietary)
Discussion/Q&A 27 Remember… Modeling is a complete business strategy NOT just a mathematical process So how will YOU use predictive modeling to improve your business?