1. Impact of Sales Force Structure Change on Products Performance Pilot Study Business Intelligence Solutions June, 2015 1

2. Objectives/Business Questions Does 2-up promotion of Product A have a positive impact on its sales relative to 1-up promotion? The hypothesis behind 2-up promotion: Engaging a 2 nd representative in the promotion will accelerate product adoption and have a positive impact on product performance of relative to 1-up promotion Testing 1-up versus 2-up promotion will allow to assess the impact and relative value of a 2nd representative engaged in active promotion of Product A within selected customer segment Does the incremental revenue associated with the 2nd sale representative actively promoting Product A provide an acceptable return on the investment? Is promoting Product A the best short-term use of the SF1 sales force capacity? 2

3. Findings / Conclusions There is no statistically significant or practically important difference in Product A sales between Test 1 and Control 1 groups Promotion cost for Control 1 group is two times higher than for Test 1 group The 2-up structure does not produce desired/expected outcome for Product A 3

4. Structure of Test – Control Groups Test 1: Product B and Product C Test 2: Product B, Product C, and Product A Control: Product B, Product C Control groups are formed on the basis of the last 2014 quarter sales data The Test and Control groups were selected to allow for a sufficient number of matched customers across the two groups to account for other variables that may impact Product A sales By matching locations with respect to other variables (DTC, business size, geography, etc.) we can effectively isolate the number of representatives actively promoting Product A as the differentiating factor between the groups 4 Product B Product A Product B Product A Product B Product C Product B Product C SF2SF2SF1SF1 Product B Product A Product B Product A Product B Product A Product B Product A SF2SF2 SF1SF1 Product B Product A Product B Product A Product B Product A Product C Product B Product A Product C SF2SF2 SF1SF1 Test 1 Test 2 Control

5. Methodology Form Test1- Control1 and Test2 - Control2 groups, using the data of the last quarter of 2014 and propensity score technique with: – nonparametric nonlinear logistic model – greedy one-to-one matching technique Develop Stochastic Gradient Boosting regression models for the first quarter of 2015 for each pair of Test – Control groups, using the following dependent variables: – Product B sales – Product A sales – Product C Sales controlling for all – “User demographics” variables (sales potential, milestone, state, business size, etc.) – promotion variables in last quarter of 2014 Estimate the difference in sales for different sales team 5

6. One-to-one Matching on Propensity Score Propensity Score Basics Propensity score – is the predicted probability of receiving the treatment (probability of belonging to a test group) – is a function of several differently scaled covariates Propensity_Score = f (Product_B_Sales_Pre, Product_A_Sales_Pre, Product B_Sales_Potential, State, Product A_Sales_Potential, Product B_Potential_Decile, Promotion variables, etc.) where f is a non-parametric non-linear multivariate function, unique for each pair of Test – Control study – If State in ('MA', 'MI', 'MN', 'IL', 'FL', 'NJ') then DTC_Indicator = 1; else DTC_Indicator=0; – If State in ('NC', 'CA', 'NY', 'GA', 'VA') then Paper_Indicator = 1; else Paper_Indicator = 0; A sample matched on propensity score will be similar across all covariates used to calculate propensity score 6

7. Control Groups Control groups are formed on the base of propensity score methodology, using only the last 2014 quarter data Control1 (for Test1 group with 547 Users): – Users are from Product A 1 – 8 deciles and from the following States: AL, FL, MI, MN, NC, NJ, WI Total Unmatched Number of Users: 4,244 Matched Number of Users: 543 Control2 (for Test2 group with 717 Users): – Users are from Product A 1 – 8 deciles and from the following States: AL, FL, MA, MN, NC, NJ, TN, WI Total Unmatched Number of Users: 6,784 Matched Number of Users: 717 7

8. Propensity Scores Calculation Approaches/software on non-parametric logistic regression: – SAS SEMMA (Sample, Explore, Modify, Model, Assess) methodology within SAS Enterprise Miner – SPSS CRISP (Cross Industry Standard Process for Data Mining) – Salford Systems CART, MARS, TreeNet, and Random Forest Approach selected: SAS SEMMA within SAS Enterprise Miner and Stochastic Garadient Boosting of Salford Systems – Test1 – Control1: (543 Product Users per group) Best model: Funnel architecture of Neural Net – Test2 – Control2: (717 Product Users per group) Best model: Cascade Correlation architecture of Neural Net 8

9. Propensity Score: Selection the Best Modeling Paradigm 9 Neural Net was the best modeling paradigm

10. Propensity Score for Test1 – Control1 Groups: Selection the Best Modeling Method 10 Neural Net with Funnel architecture was the best modeling method Misclassification Rate: Train Validation 0.11 0.12

11. Propensity Score for Test2 – Control2 Groups: Selection the Best Modeling Method 11 Neural Net with Cascade architecture was the best modeling method Misclassification Rate: Train Validation 0.09 0.10

12. Matched-Pair Samples Comparison Non-parametric tests: – For interval variables: Kolmogorov-Smirnov Two-Sample Test – For nominal variables: Chi-square test Before matching there was a significant difference in predictor distribution across all variables for – Test1 – Control1 – Test2 – Control2 After matching there was no significant difference in predictor distribution across all variables for – Test1 – Control1 – Test2 – Control2 12

13. Sales Analysis by Group TreeNet/Stochastic Gradient Boosting Modeling Total number of predictors: 42 Non-parametric model structure: Dep_var_Post = f(Dep_var_Pre, Promo_vars_Pre, … User_demographics_vars) 13

14. Dependent Variable: Product B Sales Post 14 Control1 Test1 Difference is staistically significant but practically not important Product B Sales Post GroupProduct B Mean Visits Post Test 18.3 Cntrl 17.8 Product B Sales Post GroupProduct B Mean Visits Post Test 27.20 Cntrl 28.17 Control2 Test2

15. Dependent Variable: Product B Sales Post for Test1 – Control1 15 GroupProduct B Mean Visits 2011 Test 18.3 Cntrl 17.8 The most important 5 predictors of Product B Sales Post: Product_B_Sales_Pre Product_B_Sales_Potential State Product_B_Visits_Pre Product _A_Sales_Potential Control1 Test1 Difference is practically not important Product B Sales Post

16. Dependent Variable: Product C Sales Post for Test1 – Control1 16 GroupProduct C Mean Visits 2011 Test 17.6 Cntrl 13.4 Product C Sales Post Difference is practically not important Control1 Test1 The most important 5 predictors of Product C Sales Post: Product_C_Sales_Pre Product_A_Sales_Potential Product_B_Sales_Potential State Product_B_Visits_Pre

17. Dependent Variable: Product A Sales Post for Test1 – Control1 17 GroupProduct A Mean Visits 2011 Test 13.8 Cntrl 17.7 Conclusions There is no statistically significant or practically important difference in Product A sales between Test 1 and Control 1 groups, but promotion cost for Control 1 group is two times higher than for Test 1 group. In other words, 2-up structure does not produce desired/expected outcome for Product A The most important 5 predictors of Product A Sales Post: Product_A_Sales_Pre Product_A_Sales_Potential State Product_B_Sales_Pre Product _C_Sales_Pre