1. Data Mining & Knowledge Discovery: Personalization Technologies for One to One Marketing Bhagi Narahari

2. Outline of Lecture l What and Why of Data Mining and KDD? ] Importance and Applications to E-commerce l How ? l Personalization ] personalized one-to-one business on the internet l Part I: Overview of Personalization l Part 2: The Data Mining Process

3. Predictive Modelling l A “black box” that makes predictions about the future based on information from the past and present Age balance income How much will customer spend on next catalog order ? Model (Crystal ball?)

4. What is Data Mining? l It is the exploration and analysis by automatic or semiautomatic means, of large quantities of data in order to discover meaningful patterns and rules.

5. Why now? (A historical perspective) l Because data is now available (wasn’t always) l Distributed sources l Technology evolution l Competition (do what you can to outdo)

6. Why DM? l CRM (Customer Relationship Management) - important success factor in E-commerce ] price differentiation no longer enough ] customer service more important l Links with suppliers already exist (B2B) - JIT, joint forecasting, planning, procurement l Current emphasis on links with customers - feedback, input in design, etc.

7. CRM l Identifying profitable customers l Better service for more valued customers l Retaining profitable customers ] Getting a new customer costs a lot more than retaining an existing one ] takes 5X to acquire new customers (Peppers&Rogers) ] An increase from 75% to 80% in retention reduces costs by about 10% l Larger share of customer pool

8. CRM l Product differentiations based on “price” and “quality” are increasingly difficult ] need to differentiate based on relationships l Increasingly sophisticated mass marketing increases probability of success ] cost of mass marketing is driven down by internet (reach)

9. CRM l Goal: Positively interact with your customers and prospects ] define customer segments ] lights out execution of campaigns against segments ] attribution and evaluation of responses

10. Personalization in Ecommerce l Positive: ] much better chance of personalization b customer identification b tracking across visits and within visit ] ability to do ‘what if’ experiments l Negative: ] cost of switching is much less ] is web based shopping good for ‘touchy feely’ things ] price differentiation across geographies not easy

11. Personalization Product Discovery Product Evaluation Terms Negotiation Order Placement Order Payment Customer Service & Support Market Research Market Stimulation/ Education Terms Negotiations Order Receipt Order billing and payment management Customer Service & Support Producer Chain Customer Chain

12. B2C Personalization Objectives l Know the customer ] profile - registration, cookies l Determine what the customer wants ] Ask: Questionnaires b what is the incentive for truthfulness ] Deduce: click streams, history, collaborative filtering (Amazon!!) l Deliver ] Customize the look and feel ] offer special promotions ] offer customized products (Holy Grail)

13. Use of Personalization l In addition to storing and retrieving information on the individual’s profile “on the fly” ] can also use mining software to analyze the information in the database to make recommendations or comments specific to the individual

14. Impact of Personalization l Customer relationship l Learn more about customers ] learn and understand the why and how they prefer to do business with your organization l In tandem with tracking provides you with a tool to monitor your website ] what works, what does’nt, what makes your audience “click”

15. Security and Privacy as Barrier to Personalization l Large number of customers concerned about personalization (double click!) l will they pay more to preserve privacy? l Some falsify info to preserve privacy l customers give more info to trusted site l need secure site with clear privacy policies stated at site

16. Personalization Know the Customer Identify Give the customer his/her wants Questionnaires Past history Click Streams Profile Login Credit Card# Predicting the wants Mapping to “peers” Extrapolation from past Extrapolation from peers (firefly.com) Look &feel Product selection& promotions New Product

17. Know the customer l Cookies ] backlash (users do not trust them) l OPS: Open Profiling Standard ] combined with eTrust certification l Registration ] User certificates: logons l Key Question: ] how do you know that this customer is same as that goes to your storefront ] need standard warehouse techniques like address resolution, cred.card resolution etc.

18. Know the Customer:OPS l Two drivers ] user should not retype again & again basic info ] data is used in a trusted fashion (not leaked, other data not see etc.) by users l Two parts ] Common data b demographics (country,zip,age,gender) b Contact (name, address, CreditCard…) b User agent preferences ] Per-site Sections (can be shared across sites, if user allows)

19. What if no profile??? l Deduce ] collect information: history of purchases, time spent on pages ] ask questions (offer rewards) ] combine with database marketing data l Predict behaviour ] buy probabilities ] build customer relationship l mining is key!

20. Personalization: Actions to take- Look and feel l Personalized pages ] specific data ] specific presentation and design ] sent through various mediums l Manage Customers not products: 1-1 marketing l Strategy.com ] deliver personalized pages b eg: stock portfolio, personal info including alarm, travel reservations ] use different mediums b WAP enable phones (eg: Sprint PCS Web)

21. Storefront Personalization l Customers visit Store Website ] Howard buys ties ] Rob buys Baby Products ] Ray buys toys ] Amy buys clothes l Provide a view of the store to these customers ] present them with what they are likely to buy? b Howard: ties, and men’s formal wear b Ray: Toys and gadgets b Rob: Infant, Toddler section b Amy: Women’s Clothes section

22. More Actions: Product Presentations & Promotions Basic Storefront Product Hierarchy Clothes Men’sWomen’sChildren’s ShirtsPants CasualsEvening InfantsKids John’s View Mary’s View

23. BroadVision.com l BroadVision One-to-One application ] allows businesses to develop and manage personalized web sites ] interactively profile each visitor and dynamically match info based on their profile and business rules specified by providers of site & services b users do not go through hoops finding relevant data

24. DM Terminology OLAP ROLAP Data Warehouse Data Marts Data Stores Neural Networks Genetic Algorithms Data Mining Rule Based Systems SQL

25. How? l Determine probability of buying as a function of customer attributes such as age, income, past buying patterns,.. l Target customers by ranking from highest to lowest probabilities l Other techniques: Decision Trees, Neural Networks, ….

26. KDD l Knowledge Discovery in Databases l It is the process of identifying valid, novel, potentially useful, and understandable patterns in data (Fayyad, Piatesky-Shapiro, and Smyth) l It involves data preparation, pattern extraction, knowledge evaluation, and refinement, in iteration

27. KDD l Data mining is a step in the KDD process that involves the application of certain algorithms to extract patterns l Steps in the KDD process: b Select Data b Data Cleansing and Pre-processing b Data Mining b Results interpretation b Implementation

28. Pre-processing in KDD l 80-90% of KDD process is spent here l Why? b Operational data is incomplete, inconsistent, in different formats across systems b DM techniques might require data in a specific format

29. Data Mining Problems l Classification/Segmentation ] Binary (Yes/No) ] Multiple Category (Large/Medium/Small) l Forecasting (how much) l Association Rule extraction (market basket analysis) l Sequence detection ] balance increase -> missed payment -> default

30. Typical DM tasks l Prediction and Classification ] Directed ] Decision trees, Neural networks, memory based reasoning, logistic regression ] Examples: b How many units will be sold on a given day? b What will be the stock price on a given day? b Will a customer buy the product or not?

31. DM tasks l Affinity grouping ] Undirected ] Which products go together naturally? ] The beer-diaper syndrome? ] Market basket analysis ] Examples: b Which products peak in demand simultaneously?

32. DM tasks l Clustering task ] Undirected ] Segmenting into similar clusters ] Different from classification ] Examples b Customers with similar buying profiles b Products with similar demand patterns

33. DM success factors l Integration with data warehouses and DSS l Users should develop a good understanding of techniques l Recognize that these tools cannot automatically find patterns without being told what to do l Most methods now used are extensions of analytical methods that have been around for decades

34. Legal and Ethical Issues l Privacy concerns ] becoming more important ] will impact the way that data can be used and analyzed ] ownership issues ] European data laws have implications on US l Often data included in the data warehouse cannot legally be used in decision making process ] Race, Gender, Age l Data contamination will become critical

35. Making Decisions Data Warehouse? Models Decisions Data

36. Data Warehouse l Bill Inmon: “A data warehouse is a subject- oriented, integrated, time-variant, non- volatile collection of data in support of management decisions.” l is managed data that is situated after and outside the operational systems

37. Data Warehousing l Increasing need to find, summarize, and interpret large amounts of data effectively ] Especially when data is distributed across many different databases l Transaction processing systems not easily accessible to other systems ] Plus TP systems have time constraints

38. Enter the Data Warehouse l To deliver decision data to decision makers l by integrating data from various TPS to a single storage which can then l feed a range of decision support applications l through an OLAP interface!

39. Data Complications l Noise l Missing data l Transformation ] numeric data ] text l Need to differentiate between variables you can control and those you cannot ] Actionable: size of discount, number of offers etc. ] Non-actionable: age, income..

40. Data Mining Techniques l Market Basket Analysis l Memory Based Reasoning l Cluster Detection l Link Analysis l Decision Trees and Rule Induction l Neural Networks l Genetic Algorithms l OLAP

41. OLAP: On Line Analytical Processing l While a data warehouse brings data together, OLAP lets you look at data and manipulate interactively l OLAP allows users to “slice and dice” data l Allows user to drill-down into detail data

42. Relational vs Multidimensional

43. Consolidations

44. Multidimensional Terminology l East, West, Central are input members of the Region dimension. Total Region is an output member of the Region dimension. Similarly, Nuts, Screws, Bolts, Washers, and Total are members of the Product dimension. l Variables are typically numerical measures like Sales, Costs, Profits, Expenses, and so forth. l Dimensions are roughly equivalent to Fields in a relational database. Cells are roughly equivalent to Records.

45. Steps in DW and OLAP Data Loader Data Converter Data Scrubber Data Transformer Data Warehouse OLAP ServerOLAP Interface Data

47. Cluster Detection l Undirected data mining l Finds records that are similar to each other (clusters) l Clusters are found using geometric methods, statistical methods, and neural networks l Good way to start any analysis

48. Market Basket Analysis l Form of clustering used for finding items that occur together (in a transaction or market basket) l Likelihood of different products being purchased together as rules l Planning store layouts, limiting specials to one of the products in a set,...

49. Transaction data

50. Co-occurrence matrix

51. Support and confidence l For a rule that says: If A then B l Support is defined as the ratio of number of transactions that include both A and B to total number of transactions l Confidence is defined by the ratio of the number of transactions that include both A and B to the number of transactions that include A. l How do you specify ‘significant’ support and confidence ?

52. Algorithm for Finding Association Rules l Input is Min-Support and Min-Confidence l Find all sets of items with Min-Support (frequent itemsets) ] Frequent Itemsets Property: Every subset of a frequent itemset must also be a frequent itemset b iterative algorithm: start with frequent itemsets with one item, and construct larger itemsets using only smaller frequent itemsets.

53. MBA example l Using the sample data create a co- occurrence table l Let relevant Support = 25% and Confidence= 50%: ] Beer and Diapers appear in 3/5= 60% ] If beer then diapers has confidence of 2/3=67% ] Thus, “If customer buys beer then customer buys diapers” satisfies 25% support & 50% confidence l Conclusion drawn by mining system: ] Customers who buy beer also buy diapers

54. Applying MBA Results l Is the relationship useful ? ] Beer and Diapers may not be of use ] Victoria’s Secret transaction mining led to specific apparel sent to specific stores -- Microstrategy software l Who defines “usefullness” ] only as good as rules specified by humans/marketing workforce ] NBA mining: designers of s/w did not include height mismatches at first…coaches made the correction

55. Data Mining Algorithms l Four algorithms commonly cited ] Association Rule (used in over 90% of the cases!) ] Nearest Neighbor b quick and easy but models get large ] Decision Tree ] Neural Network b difficult to interpret and large time

56. Decision Trees l Series of if/then rules ] easy to understand, complexity in implementation No yes Balance<10K Balance > 10K Age > 48 Age< 48 yes

57. CRM and Data Mining l Recall:customer segmentation is key in CRM ] data mining can help improve understanding of customer behaviour b helps located meaningful segments from customer data ] users want to turn that understanding into an automated interactions with their customers

58. Integrating Data Mining & CRM l Data mining application owns the modelling process l CRM application owns the campaign execution process l Goals: ] minimize pain involved with using models in campaigns ] score records only when and where necessary

59. Integrating Mining & CRM l Step 1: ] analytic user creates model using mining system ] model is then exported into campaign management system l Step 2: ] Marketing user creates campaign that includes predictive models ] when campaign executes, data mining engine scores customers dynamically

60. Benefits of Integration l Pre-generated model selection l Score defined segments “on the fly” ] eliminates need to score entire database ] improve efficiency of campaigns l Reduces manual intervention and error l Accelerates the market cycle ] increases likelihood of reaching customers before competitors ] improves campaign results and lower costs

61. Summary l “Using the new media of the one-to-one future, you will be able to communicate directly with customers individually…..” - Don Peppers & Martha Rogers (One-to-One Future) l “What are you afraid of?…..Even if you’re not afraid of these things, the beauty is,with proper marketing, we can make you afraid”-- Michael Saylor, CEO Microstrategy.