1. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Add Conjoint Analysis to your Quant Toolbox

2. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Agenda 2  What is MaxDiff Analysis?  What is Conjoint Analysis?  What is Choice-Based Conjoint Analysis?  Demo of Discover  Analysis with Discover and online simulator  How can I teach this in my classroom?  Questions?

3. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session WHAT IS MAXDIFF? Section 1 3

4. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session 4 The Ubiquitous Monadic Rating Scale

5. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Possible Solutions  Ranking exercises  Becomes impractical with >7 items  Ordinal scale results  Constant-Sum Allocation  Becomes impractical with >7 items  Allocation may not be independent  Making answers to sum to a particular value is difficult 5

6. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session MaxDiff (n). Maximum difference scaling, also known as best/worst item scaling, is used for measuring the importance or preference within a list of items. 6

7. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Quick Overview  Respondents typically shown between 2-6 items at a time, asked to indicate which is best and/or which is worst  Task is repeated many times, showing a different set of items in each task  Resulting model provides ratio-scaled scores, or utilities, for each item 7

8. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Relationship to Conjoint Analysis 8  MaxDiff is similar in many ways to conjoint analysis, which also forces tradeoffs  Rather than asking respondents to directly score the items, we observe these tradeoffs and derive the scores  The derived scores have interval scale qualities  Scores can be derived at the individual or group level

9. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Example http://www.sawtoothsoftware.com/surveys/5fact 9

10. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Example 10

11. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session  4 to 5 items per set  Don’t show more than half of the total items in a set  More than 5 items is detrimental 4 5 Number of Items Per Set 11

12. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Number of Sets per Respondent ?? x 4 10 >3 # tasks X # items per task > 3 Total # items 12

13. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Frequency Balance (n). Each item appears an equal number of times as every other item. 13

14. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Orthogonality (n). Each item appears an equal number of times with every other item. 14

15. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Positional Balance (n). Each item appears an equal number of times in the first, second, third, etc., positions within the set. 15

16. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Connectivity (n). Each item should be directly or indirectly compared to every other item in the study. It allows all items to be placed on a common scale. 16

17. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session 17 Methods of Analysis Item Number Times Shown Times Selected Best Best Count Proportion 20100055556% 1999938639% 5100036136% 199931031% 9100029429% 18100027628% 7100025225% … Item NumberAverage 2010.16729 57.42383 197.25356 16.57479 95.80929 185.63295 114.96448 74.96292 164.79268 154.62568 104.5877 134.29381 174.23938 84.21049 33.93688 63.91944 43.90014 123.68288 143.65859 21.36321 Counts Aggregate Logit Latent Class Hierarchical Bayes

18. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Total Unduplicated Reach and Frequency (TURF)  Finds the items that will reach the maximum number of people 18

19. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session 19 Case Study – Clinical Research MaxDiff Business Objective A clinical research facility was interested in prioritizing the 30+ possible claims they could make about their new drug for marketing and communication Our Solution A MaxDiff study was done to prioritize these claims and results were also entered into a TURF simulator The Results Three tiers of claims emerged, with four claims very far above the rest of the pack. The client was then able to allocate clinical research funds by trading off the value of the claims and their likelihood of success in clinical studies.

20. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session 20 Case Study – Website Optimization MaxDiff Business Objective A startup had +20 ideas of content for their landing page. Some of the content would require more real estate than others. In addition, the lower prioritized content could show up on subsequent pages. Our Solution An anchored MaxDiff exercise was conducted to prioritize the content and find the optimal combination of that content for the landing page. The Results While the most valuable top 10 ideas could not fit on one landing page, a set of 11 ideas, swapping out a top 10, gave them a higher value score. A utility correlation matrix also helped determine which ideas should appear together on the subsequent pages.

21. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session WHAT IS CONJOINT? Section 2 21

22. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Different Perspectives / Different Goals 22  Buyers want all of the most desirable features at lowest possible price.  Sellers want to maximize profits by:  Minimizing the costs of providing features  Providing products that offer greater overall value than the competition

23. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session  If we learn how buyers VALUE the components of a product, we are in a better position to design those that improve profitability.  BUT, how do we learn what the buyer wants? Breaking Down the Problem 23

24. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session  Ask Direct Questions about preference:  What brand do you prefer?  How much would you pay for it?  What color/flavor do you prefer?  Answers often trivial and unenlightening (e.g. respondents prefer low prices to high prices, etc.) Age Old Approaches 24

25. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session  Research technique developed in early 70s  Measures how buyers value individual components of a product/service bundle  Dictionary definition -- “Conjoint: Joined together, combined” Insert – Conjoint Analysis! 25

26. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session  We vary the product features (independent variables) to build many product concepts.  We ask respondents to rate, rank or choose among a subset of those product concepts (dependent variable).  Based on the respondents’ evaluations of the product concepts, we figure out how much unique value (utility) each of the features added. How Does Conjoint Analysis Work? 26

27. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session You can’t have the highest fuel efficiency and the best performance 27

28. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session 28 When respondents are forced to make difficult tradeoffs, we learn what they truly value

29. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session WHAT IS CHOICE- BASED CONJOINT? Section 3

30. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session  Rather than rate options, respondents are simply asked to choose which they prefer. What Does a CBC Experiment Look Like? 30 Income StatusRichPoorMiddle ClassNone: I would rather die than end up like any of these people. Hair StatusBaldFull head of hairReceding hair line FamilyNo familyLarge family (10+ children) One ex, but two great kids Which would you rather be in 20 years? (Task 1 of 12) Concept 1Concept 2Concept 3“None” Option (Concept 4) Attributes

31. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session 12% No Fee $5000 limit 9.5% No Fee $3000 limit 7.5% $75 $9000 limit 11.5% $50 $7500 limit

32. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session http://www.sawtoothsoftware.com/baseball

33. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session 33

34. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session 34

35. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session DISCOVER DEMO Section 4 35

36. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Cell Phone Demo 36 BrandBattery LifeScreen Size CameraPrice Apple iPhone1 day battery4” screen5 mega pixels$200 Microsoft Windows Phone2 day battery5” screen8 mega pixels$300 Google Nexus (Android)3 day battery6” screen15 mega pixels $400 Samsung Galaxy (Android) $500 $600

37. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session 37 Creating a CBC Exercise in Discover

38. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session 38 Adding Attributes & Denoting Level Preference Order in Discover-CBC Upfront, you must specify which attributes have known (a priori) utility order (such as higher speeds always being preferred to lower speeds or lower prices always preferred to higher prices).

39. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session  By default, a “None” concept is added to each choice task. The questionnaire author may remove the None option if desired. Including a “None” Option 39

40. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Why Include a “None”? 40  We may not want to assume that the respondent MUST choose an alternative. Therefore, we can include a “None” choice for respondents to tell us they would NOT choose any concepts in the task.  Why include a none option?  Allows respondents to avoid making uncomfortable choices  May provide usable information about whether our concepts are acceptable, how acceptable they need to be  Remove respondents for whom product category does not apply  Earlier screening also helps!  Estimate utility for “None”

41. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session 41 How Many Tasks? How Many Concepts?

42. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Discover-CBC’s recommendation wizard will help you with that! 42  Discover-CBC’s recommendation wizard will suggest an appropriate number of tasks and concepts per to ask, given the specific attribute and levels of your exercise.  We also warn the user if too few tasks are specified. Essentially, we make it nearly impossible for users to field a study that would yield poor quality utility estimates.  The recommendations are based on Logit theory but the author may override the suggestions.

43. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Additional Recommendations 43  Generally, 3 or more concepts are used. Showing just 2 concepts is typically considered sub-optimal.  It doesn’t take respondents twice as much time to respond to four concepts per task than two.  Attribute text length, graphical representation affect the decision  Notable exception: highly emotional decisions such as cancer treatment, where patients are unfamiliar with attributes and must make extremely difficult choices.  As a rule of thumb, for individual-level estimation, we recommend that each level occur approximately 6x or more for each respondent.

44. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session 44 Adding Prohibitions

45. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session 45 So, What Does the Final CBC Exercise Look Like?

46. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session ANALYZING CBC DATA Section 5 46

47. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session 47 Discover-CBC’s Empirical Bayes Model

48. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session 48 Three Conjoint Analysis Outputs Utilities Market Simulations Importance s

49. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session  “Utilities” are developed for each level such that the following model produces “maximum likelihood fit” to the actual choices.  Consider three product alternatives (A, B, C) in a choice set: P A = exp(U A ) / [exp(U A )+exp(U B )+exp(U C )] where: P A = “Probability of choosing alternative A” exp means “exponentiate” or raising e to the power of the total utility U A = Total utility for alternative A, etc. Utility Estimation 49

50. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session  Interval scaled data (no ratio operations!)  You CANNOT compare one level from one attribute with one level from another attribute, since conjoint utilities are scaled to an arbitrary constant within each attribute (often zero-centered)  You CAN compare differences between two levels of one attribute versus two levels of another attribute (an addition operation) Interpreting Conjoint Utilities 50

51. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Utility Examples 51

52. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session  Ratio scaled data  Measure of how much influence each attribute has on people’s choices  Best minus worst level of each attribute, percentaged  Importances are directly affected by the range of levels you choose for each attribute Conjoint Importances 52

53. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session 53 Importances Example

54. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session  How much a respondent likes a total product is simply the total of the utility values for that product. Conjoint Analysis is an Additive Model 54

55. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Market Simulation Example 55  Create competitive market scenarios to predict which products respondents would choose.

56. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session 56 Case Study – Cancer Treatment CBC Business Objective Cancer treatments can come with harmful side effects that can cause a patient to quit treatment. Research was sought to help increase the treatment completion rate and ultimately lead to higher survival rates. Our Solution Patients were asked to complete a choice-based conjoint exercise that would help them choose the proper course of treatment based on their individual preferences found in the conjoint analysis. The Results By studying preference (or tolerance) of side effects, effectiveness and length of treatment, doctors could recommend an appropriate treatment method that would encourage treatment completion.

57. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session 57 Case Study – Employee Benefits CBC Business Objective A large corporation knew how important good employees were to their success and they wanted to understand how to keep them satisfied with their compensation and reduce turnover, all while keeping down costs to provide these benefits. Our Solution A choice-based conjoint exercise was designed that simulated different employee packages. Respondents were asked to choose which one they preferred the most. The Results The company was able to identify and quantify the drivers of job satisfaction. In addition, they were able to use cost information of each of the programs to optimize the benefits packages.

58. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session 58 Case Study – Household Electronics CBC Business Objective A household electronics manufacturer wanted to test features that were popular in foreign countries for possible inclusion in a U.S. based product line. Our Solution A choice-based conjoint experiment was conducted incorporating ideas from marketing, sales, and engineering departments. The Results While none of the new-to-the-U.S. features appealed much, the results showed two existing features deserved much more play. By emphasizing these features, the client had an unexpected advantage over its competitors until its competitors matched the offerings.

59. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session HOW CAN I TEACH THIS IN MY CLASSROOM? Section 6 59

60. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Up to 50 users of the software Two interviewing platforms SSI Web (desktop- based) and Discover (web-based) Discrete Choice Components including: MaxDiff Choice Based Conjoint Adaptive Choice Based Conjoint Analytical Tools Hierarchical Bayes Estimation Latent Class Analysis Convergent Cluster Ensemble Analysis Online Simulator & MaxDiff Analyzer Free Survey Hosting Allows for n=250 completes/survey Annual lab license - $1,000 Annual research license - $3,000 60

61. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session  Discover includes:  Standard survey question types: select (radio or multi- check), numeric, grid, ranking, constant sum, open-ends  Skip patterns, styles, progress bar options  CBC designer up to 8 attributes and 15 levels per attribute; No more than 8 concepts per task; No more than 30 tasks per respondent  Near orthogonal CBC plans, avoiding dominated concepts and adjustable level overlap  Individual-level utility estimation with empirical Bayes  Online Market Simulator  Discover-CBC: How and Why It Differs from Lighthouse Studio’s CBC Software Discover-CBC: How and Why It Differs from Lighthouse Studio’s CBC Software 61

62. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session QUESTIONS? 62 Justin Luster Product Manager justin@sawtoothsoftware.com Megan Peitz Ingenuity Ambassador megan@sawtoothsoftware.com www.sawtoothsoftware.com +1 801 477 4700 @sawtoothsoft

63. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session USE CASES Appendix 63

64. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Marketing  Set pricing  Evaluate the impact of a product redesign  Estimate brand equity  Predict effect of line extensions  Identify market segments 64

65. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Health Care  Evaluate doctor to patent interactions  Understand doctor prescription preferences  Design effective treatment strategies  Develop new drugs  Identify ideal benefit programs 65

66. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Economics  Evaluate transportation alternatives  Compare energy alternatives  Measure environmental impact 66

67. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Law  Measure effects of litigation  Damage assessment  Identify boundaries between firms  Evaluate punishment alternatives  Select jury members 67

68. © 2016 Sawtooth Software, Inc. | www.sawtoothsoftware.com AMA Winter Learning Session Human Resources  Screen potential employees  Design compensation packages  Select health care plans  Evaluate performance  Predict employee responses 68