Machine Learning Interpretability Thuy Nguyen Mihir Jain Edward Adcock Toby Alfred-Jones © COPYRIGHT | Delta Capita | CONFIDENTIAL

Contents Project objective Use cases Data overview Neural Network model description Machine Learning Interpretability Results  Visualization Next steps 01 02 03 04 05 06 07 08

01. Project objective

Project objective Delta Capita has developed a Neural Network model to determine the likelihood of mortgage default given a set of information that represents an individual loan The aim of the project is to develop knowledge extraction techniques which interpret the mortgage defaulting model

02. Use cases

Use cases The ability to accurately determine loan default is especially useful in two domains: Mortgage-backed security investing Risk Management

03. Data overview Overview of data 8 Data features 9 Data features (provided) 10 Data features (created) 11 Data features (added) 12 Model Preparation 13

Overview of data Main Dataset Freddy Mac Single-Family Loans Time frame: 1999 – 2016 Size: 15.3m unique loans with 326m performance updates (monthly) Types of Loans: Default : 85k Fully Paid : 15.2m Ratio of Default vs Fully Paid loans: 0.6 : 99.4 Additional Dataset Average National Mortgage Interest Rate Monthly national interest rate for standard mortgages from January 1999 to July 2016 Housing Price Index Per State Monthly House Price Index in each U.S. state from January 1999 to July 2016 Unemployment Rate Per State Seasonally adjusted unemployment rate by each U.S. state from January 1999 to July 2016

Data features We split the following section into 3 parts: Data Features provided in the main dataset Data Features created from the main dataset Data Features added from external sources We will later evaluate the models performance on data from Feature Sets 1, 2 & 3 (combined)

Data features (provided) Monthly Performance Update Features Evaluation of on-going loans on a monthly basis Origination Features Assessment at the time of the loan application Origination Features Credit Score Original Unpaid Principal Balance First Payment Date Loan-To-Value Ratio First Time Home Buyer Flag Interest Rate Maturity Date Channel of origination of a loan Metropolitan Statistical Area Prepayment Penalty Mortgage Flag Mortgage Insurance Percentage Product Type Number of Units in a Property Property Type Occupancy Status Property State Combined Loan-To-Value Ratio Loan Purpose Debt-To-Income (DTI) Ratio Original Loan Term Number of Borrowers Performance Features Monthly Reporting Period Current Actual Unpaid Principal Balance Loan Age Remaining Months to Legal Maturity Current Interest Rate

Data features (created) Based on history of current loan: Occurrence of: Loan Status (30-dd, 60-dd, 90-dd, foreclosed, etc ...) Occurrences of Loan Status in the last 12 months Percentage change between Last Balance and Current Balance Based on history of all loans: Number of Loans (active) per State/Zip-code Number of Loans (taken out) per State/Zip-code Number of Loans (taken out) per State/Zip-code in the last 12 months Default Rate per State/Zip-code Default Rate per State/Zip-code in the last 12 months Occurrences of ‘Paid Off’ & ‘Default’ per State/Zip-code Occurrences of ‘Paid Off’ & ‘Default’ per State/Zip-code in the last 12 months

Data features (added) Economic Features: Monthly Unemployment Rate per State Monthly Housing Price Index Per State Monthly National Interest Rate Extra features created from added datasets: Difference between Current Interest Rate and National Interest Rate Number of Months that Mortgage Interest Rate is less than National Interest Rate

Model preparation Class imbalance Categorical data: One hot encoding Using under-sampling technique on the training set New ratio of Default vs Fully Paid loans: 15 : 85 Categorical data: One hot encoding For example, if the property is in New York, the value is 1, otherwise 0 Randomly shuffle data

04. Neural Network Model Description Model Architecture                                     15 Performance Evaluation Metric                           17 Model performance                                                      19

Model Architecture We use Neural Network to create the Mortgage Classification model Model classes: Default or Fully Paid Model output: Any value between 0 and 1, which represents the probability of Default Threshold value of 0.5 (Value above 0.5 predicts a Default Loan) Model architecture: Layers Number of layers Number of Neurons Input layer (Number of loan features) 1 133* Hidden layer 2 100 : 100 Output layer (Number of classes) * means out of 133 input features, there are only 64 unique loan features

Performance Evaluation Metrics We use 4 performance metrics: Accuracy - Overall classification accuracy True Positive Rate - Classification accuracy of ‘Default’ loans  True Negative Rate - Classification accuracy of ‘Fully Paid’ loans AUC - ( True Positive Rate + True Negative Rate ) / 2

Model performance Using the data from Feature Sets 1, 2 & 3 (combined): Performance Metric % Accuracy 98.3 % Correct Default (True Positive Rate) 98.5 % Correct ‘Fully Paid’ (True Negative Rate) 98.2 AUC 98.4

05. Machine Learning Interpretability Knowledge Extraction                                   22 TREPAN                                                                                       23 Distilling Soft Decision Tree                                      24 LIME                                                                25

Knowledge extraction Problems: Methods: Neural Networks: high performance, but black-box Decision Tree: high representation, but low performance Combine Neural Networks & Decision Tree to create rules that are human-comprehensible Methods: Global: TREPAN Distilling Soft Decision Tree Local: LIME

TREPAN Key features: Neural Networks serve as an oracle that returns class labels Construct models of the underlying distribution of data Tree expansion: best-first expansion to increase fidelity Splitting tests: m-of-n Stopping criteria: Global criteria: size of the tree, highest fidelity tree Local criterion: stopping the tree Key metrics: Accuracy Fidelity Comprehensibility

Distilling soft decision tree Key features: Mimic the input– output function from the Neural Networks Soft targets: true label, predictions of Neural Networks Trained with mini-batch-gradient descent Uses learned filters to make hierarchical decisions Selects a particular static probability distribution over classes as output Key metrics: Accuracy Comprehensibility: complexity of the tree

LIME Key features: Create a local linear model around the prediction Assign weights to different features in the dataset Compute the class probability Predict the class having the highest probability Key metrics: Accuracy

06. Results TREPAN                                                                27 Distilling Soft Decision Tree          28 LIME                                                 29

TREPAN Use 400 data points Conditions: Model performance: Maximum of nodes: 10 Minimum sample: 100 Model performance: Accuracy: 80% Fidelity: 88%

Distilling Soft Decision Tree Use 400 data points Condition: Maximum of tree depth: 10 Accuracy: 95%

LIME Use 400 data points Example: Prediction of a loan for the 5th customer

07. Visualisation

Visualization - Dashboard

Visualization - Dashboard

08. Next Steps

Next steps Interpretability Model Use the entire dataset to validate all interpretability models  Try to interpret different Machine Learning models such as Random Forest, SVM Commercial products Develop a front-end app which is easier for people with no data science background to use Provide the tool to work irrespective of dataset or Python libraries Suggest recommendation from results of interpretability models