1. Marketing Engineering Class Gary L. Lilien, Arvind Rangaswamy, Katrin Starke, and Gerrit H. van Bruggen How and Why Decision Models Influence Resource Allocation Decisions

2. Marketing Engineering Class Outline G Background G Issues Addressed G Methodology G Results G What Next?

3. Marketing Engineering Class Background G Through IT, there is greater deployment of marketing decision models (DSS). G More managers, not just analysts, are using decision models. G A lot of research is currently focused on model development, and their underlying methods. We know little about the decision contexts and model designs that favor model effectiveness and impact.

4. Marketing Dept Working Series

9. Marketing Engineering Class Issues Addressed G What makes a good DSS? What factors influence the decision process and actual outcomes associated with a DSS? G To what extent could a DSS overcome anchors (e.g., past decision rules)? G To what extent does process influence outcomes?

10. Marketing Engineering Class Why Another Study? G There is no comprehensive study of DSS effectiveness that explores both decision process and outcomes, and the effect of process on outcomes. Also past studies give contradictory results, which are difficult to explain because they do not typically measure the effects of models on decision processes. G Not many studies have looked at resource allocation decisions (they typically focus on forecasting problems, for which there is an ex-post “right answer.”). G Past studies typically compare model versus no model (human judgment). We compare explicit decision models against availability and a generic decision tool (Excel). G We explore effects of various process variables such as de-anchoring, discussion quality, etc. by simulating a decision process, not just use of a model.

11. Marketing Engineering Class Specific Areas Explored Context u Nature and type of model available for decision making u Prior experience (in using models; exposure to Excel tools) Outcomes u Profit realized (as computed by model; as compared to reality) u Perceptual measures of performance (e.g., satisfaction, learning) u Decision quality as judged by experts u Decision patterns (e.g., resources allocation to different entities )

12. Marketing Engineering Class Specific Areas Explored Process u Amount of effort u Complexity u Generation of alternatives u Departures from anchor points u Quality of discussions u Problem solving approach (analysis and intuition)

13. Marketing Engineering Class Conceptual Framework G Decision context and process, together determine outcomes Context Decision Process Outcomes

14. Marketing Engineering Class The Conceptual Model for Evaluation Model type Order in which models are used Problem Complexity Cognitive Effort/Time Discussion Quality Decision Alternatives De-Anchoring Context Process Outcomes Profit Satisfaction Learning Usefulness Expert evaluation

15. Marketing Engineering Class Hypotheses Decision models Proposition 1: will improve objective and subjective outcomes associated with resource allocation decisions. Specifically, we expect it to improve: (1) realized “objective” profits, (2) the perceived satisfaction with the decision, the perceived learning from problem solving, and usefulness ascribed to the DSS, and (3) independent expert rater’s assessment of the decision quality. Proposition 2: will improve the process of decision making (e.g., reduce problem complexity and enhance cognitive effort deployment) in resource allocation tasks. Proposition 3: improved decision processes will improve outcomes. Proposition 4: The patterns of DSS impact articulated in the above propositions hold independent of marketing resource allocation decision context and DSS design.

16. Marketing Engineering Class Conceptual Framework G Decision models reduce cognitive effort -- for tasks supported by the models (e.g., Payne 1982; Payne, Bettman, and Johnson 1988). G If saved effort is not used for doing additional analyses (e.g., exploring options in greater depth, exploring new options), then model use does not necessarily improve outcomes (e.g., Todd and Benbasat, 1992) G If saved effort is used for additional analyses and explorations, then model use will improve outcomes

17. Marketing Engineering Class Criteria for Study Design G Replicable situations, to permit statistical model building and hypothesis testing. G Realistic decision context. G Participants should have some exposure/ understanding of issues associated with resource allocation decisions. G Identify results that are robust to decision contexts and DSS designs G Participants should have the background and capability to understand and use spreadsheet models and market response models. G Participants must not be experts (e.g., Analysts).

18. Marketing Engineering Class Method G Experimental study u Two real business cases (ABB, Syntex; Winners of Edelman prize) u 56 pairs of students in eight experimental conditions v Subjective measures at individual level v Objective measures for team performance u All subjects had access to spreadsheet; those in the treatment conditions had additional built-in modeling capabilities u Clearly specified anchor points (status quo) in both cases

19. Marketing Engineering Class Method Experiment with both within- and between-subject measurements Note: Order of cases were also randomized.

20. Marketing Engineering Class Decision Models Selected for Study G ABB: Select 20 customers to focus incremental marketing effort (model helps users to move toward segmentation based on switchability, but does not provide specific feedback on performance) G Syntex: Determine level of sales effort, and allocation of effort across products (Model allows users to optimize contribution under various user-imposed constraints, and provides feedback on profitability of alternate actions) u Both cases are based on real decision situations where we know the actual outcomes u Both cases are part of the Marketing Engineering suite of software programs -- we can control the kind of decision aids given to subjects

21. Marketing Engineering Class ABB Raw Data (Gensch et al.)

22. Marketing Engineering Class ABB Choice Model (Gensch et al.)

23. Marketing Engineering Class Syntex Product Model (Lodish et al.)

24. Marketing Engineering Class Anchor Points G ABB: A senior district sales force manager makes the following recommendation in the case, “Our goal is to grow the company by landing more big contracts. You’ve got to fish where the big fish are, so the answer is easy. Let’s pick the 20 biggest contract- proposals and go after those folks with the new program. If we can get a few more of those big fish to bite, Elwing [the President] and the board will be really happy!” G Syntex: The case describes the current management plan: Robert Nelson, the VP for Sales says, “Don’t change a winning game plan.” The current plan called for maintaining the same allocation (as specified in the “Base Selling Effort” column), while increasing sales effort by about the same level (40 reps) per year.

25. Marketing Engineering Class Experimental Procedure G MBA students G Two-member groups G All groups provided spreadsheets (those in treatment conditions also got a built-in model) G Paid for their participation ($30 per participant) G Prizes for best-performing groups

26. Marketing Engineering Class Experimental Procedure 1.Pre-Experimental Questionnaire 2.Case 1 u Case Description u Model Tutorial u Recommendation forms 3.Post-Experimental Questionnaire 1 4.Case 2 u Case Description u Model Tutorial u Recommendation forms 5.Post-Experimental Questionnaire 2

27. Marketing Engineering Class Context Measures G Experimental Factors u ABB model u Syntex model u Order in which cases were evaluated by subjects (ABB/Syntex =1 and Syntex/ABB = 2) G Covariates u Systematic Approach to Problems (Self assessment, single item, 5-point agree/disagree scale), mean = 4.16 u Excel Skills (6 items, 0-6 scale, ability to plot graphs, sort data, run data analysis tools, use solver, use functions, write macros), mean = 3.80

28. Marketing Engineering Class Outcome Measures G Performance u Mean Incremental sales for ABB: $4,135K u Mean Incremental profit contribution for Syntex: $260,638K G Satisfaction with the Decision; 5 items, 5-point Likert scale u I am satisfied with it u it is of high quality u I am in full agreement with it u I like it u I am confident that it will work out well  ABB = 0.90 Mean ABB = 3.94;  Syntex = 0.94 Mean Syntex = 3.16

29. Marketing Engineering Class Outcome Measures G Perceived Learning; 3 items, 5-point Likert scale u it increased my skills in critical thinking u it increased my ability to integrate facts u it showed me how to focus on identifying the central issues  ABB = 0.82 Mean ABB = 3.61;  Syntex = 0.86 Mean Syntex = 3.33 G Perceived Usefulness of the Excel Tool; 3 items, 5-point Likert scale u enabled us to make decisions more quickly u increased productivity u improved our performance  ABB = 0.91 Mean ABB = 4.23;  Syntex = 0.96 Mean Syntex = 3.59

30. Marketing Engineering Class Outcome Measures G Expert rater’s evaluation (3 judges; 5-item-scale plus one overall score)  ABB = 0.61  Syntex = 0.83 (across judges) Mean of overall scale: ABB 57.6; Syntex 48.9

31. Marketing Engineering Class Process Measures G Amount of Time u Number of minutes used for case evaluation Mean ABB = 77, Mean Syntex = 85 G Amount of De-Anchoring u Deviation of decision from current practice v ABB: Lack of overlap with management plan v Syntex: distance from current management plan G Cognitive Effort Spent; 3 items, 5-point Likert scale u we were totally immersed in resolving this problem u we took this task seriously u we put in a lot of effort  ABB = 0.73 Mean ABB = 4.32;  Syntex = 0.79 Mean Syntex = 4.11

32. Marketing Engineering Class Process Measures … Contd G Complexity (Index of required cognitive effort); 3 items, 5-point Likert scale u it was a complex process u it was a challenging process u it was a difficult process  ABB = 0.87 Mean ABB = 3.55;  Syntex = 0.91 Mean Syntex = 3.94 G Discussion Quality; 3 items, 5-point Likert scale u our discussions were well organized u we had discussions about what criteria to use to select amongst the various decision alternatives u we both participated actively in our deliberations  ABB = 0.58 Mean ABB = 4.29;  Syntex = 0.59 Mean Syntex = 3.92

33. Marketing Engineering Class Process Measures …Contd G Quantity of Decision Alternatives Generated (2 items, 5 point agree/disagree) u we had discussions about many decision alternatives that were not part of the final recommendation u we considered several alternatives carefully  ABB = 0.56 Mean ABB = 3.51;  Syntex = 0.65 Mean Syntex = 3.54

34. Test of Proposition 1

35. Marketing Engineering Class ABB Case: Gains from Resource Allocation Note: Incremental profit based on scoring rule calibrated from actual sales.

36. Marketing Engineering Class Syntex Case: Profit Outcomes One outlier was dropped from the analysis

37. Marketing Engineering Class Syntex Case: Comparison of Allocations Base UnaidedModel p Incremental Number 0 177 (145)272 (150).02 of salespersons Allocation (proportion of total salespersons) Naprosyn.23.30 (.09).38 (.10).00 Anaprox.33.27 (.08).25 (.07).36 Norinyl 135.12.12 (.02).11 (.02).05 Norinyl 150.06.07 (.04).05 (.01).06 Lidex.06.07 (.02).06 (.01).20 Synalar.07.06 (.01).05 (.01).01 Nasalide.13.12 (.04).10 (.10).02

38. Marketing Engineering Class ABB - Subjective Perceptions of Outcomes Significance at p < 0.1 highlighted in green

39. Marketing Engineering Class Syntex - Subjective Perceptions of Outcomes Significance at p < 0.1 highlighted in green

40. Marketing Engineering Class Determinants of Expert Ratings Significance at p < 0.1 highlighted in green

41. Marketing Engineering Class Sample of Report Given to Expert Raters

42. Marketing Engineering Class Sample of Report Given to Expert Raters

43. Marketing Engineering Class Determinants of Report Length Significance at p < 0.1 highlighted in green

44. Test of Proposition 2

45. Marketing Engineering Class ABB - Process Variables Significance at p < 0.1 highlighted in green

46. Marketing Engineering Class Syntex -- Process Variables Significance at p < 0.1 highlighted in green

47. Test of Proposition 3 and 4

48. Marketing Engineering Class Simultaneous Equation Model Evaluated structural equations model -- one for each outcome measure for each case Y: Process variables (endogenous) X: Context variables (exogenous) Y: Subset of process variables

49. Marketing Engineering Class Structure of Model ABB ABB Process ABB Outcome Syntex Syntex Process Syntex Outcome

50. Marketing Engineering Class Summary of Process Impact in ABB and Syntex Process Outcomes Model Order Decision Model Process Complexity Cognitive Effort Discussion Quality # of Alternatives De- Anchoring Incremental Revenue Satisfaction Learning Usefulness

51. Marketing Engineering Class ABB -- Direct and Indirect Effects Indirect effects are those mediated by the decision process.

52. Marketing Engineering Class Syntex -- Direct and Indirect Effects Indirect effects are those mediated by the decision process.

53. Marketing Engineering Class Proposition 4 Impact of ABB and Syntex are different. In Syntex, the process effects are directionally stronger.

54. Marketing Engineering Class Summary of Findings G The availability of models (both ABB and Syntex) improve objective resource allocation performance u Can change the basis of allocation decisions (e.g., shift focus to growing products, profitable products, switchable customers, etc.) G Availability of model enhances decision process (e.g., reduces process complexity, help users move farther away from anchor points) G Process has stronger influence on subjective outcomes than on objective outcomes.

55. Marketing Engineering Class Summary of Some Findings..Contd G Of particular interest are the following process effects: Model availability increases cognitive effort, which improves discussion quality, leading to consideration of more alternatives. Discussion quality had no impact on incremental returns, but enhanced satisfaction, learning, and perceived usefulness of the system. G Model with feedback (Syntex) seems to have stronger influence on process.

56. Marketing Engineering Class Managerial Implications G DSS should be designed and tested to improve objective outcomes. G Just the promise of improved objective outcomes by using a DSS is not enough – DSS’s should be designed to enhance the process by increasing cognitive effort devoted to the task, facilitating discussion, and encouraging consideration of more alternatives. Otherwise, all that they might accomplish is increase efficiency of decisions (i.e., reduce effort). G Senior managers (expert raters) are unlikely to be able to distinguish between DSS supported recommendations (which our research suggests are superior) and non-DSS supported recommendations, especially when potentially biasing cues (e.g., length, format of the presentation) are readily available. G Include feedback in the model to enhance process effects.

57. Marketing Engineering Class What’s Next? G Analyze decision process protocols u evaluation of process recorded by the computer G Analyze factors that encourage the Use of decision models G Assess effects of training in models and modeling on decision processes and outcomes G Explore the effects of decision models on group processes