Processing and Analyzing Large log from Search Engine Meng Dou 13/9/2012

2 Web-browsing data social network communications sensor data ->Behavior data Google and Facebook, for example, are Big Data companies. Big data processing Extracting useful information that reflects user behavior from massive log Instance data management Data analysis Behavior data (like web log) can be used for improving and supporting business processes. Data mining, process mining and so on Big data Challenges Opportuni ties

3 Distributed File System(HDFS) Key-value Database(HBase,Cassandra, MongoDB) Unstructured Data Cloud Storage Big Data processing BI/ Reporting BI/ Reporting Data Mining Machine Learning Analytic applications Cloud computing (Map/Reduce Framework) Cloud computing (Map/Reduce Framework) Big Data Access Hive NoSQL Raw data Instance data Distributed File System(HDFS) NoSQL Cloud computing (Map/Reduce Framework) Cloud computing (Map/Reduce Framework) Cassandra Process Mining Process Mining Process Mining Process Mining

Case study: Search Engine Company 4 News, Page, Image, Maps, Music, navigation Dataset: 66 million clicks in one month, 2.2 million clicks per day ->generate behavior in 10 minutes User Behavior: Visiting path (Referer) Searching result effectiveness Abs Clicking Behavior Source and Destination of User visiting Robot Behavior Reorganization and Analysis Visiting page layout Behavior comparison and product improvement User grouping and recommendation

Data features 5 It contains massive information in a well recorded format Large scale with big growing potential Real-time analysis

existing tools 6 Data extracting: XESame ， Prom Import Process Mining : ProM 1)Due to large data set, analysing has low speed and in most situations it got crash 2)Offline analysis-> real-time analysis Cloud Storage /no rational DB Instance data(XES) Extracting data from cloud

System Structure 7 Log processing Understandable model Extracting useful information that reflects user behavior from massive log

Convert raw log to instance data(event log) with Map/Reduce 8

9

10 fileSizelogNumOnePCTimeMapReduceTimeMapNumReduceNum 8.84 MB s, 921 ms7s M s, 846 ms25s M s, 559 ms30s315 One day(371M)2,200,0002.5minutes1.3minutes4015 One week15,000,00020 Minutes (Expected ) 2.5minutes28015 One month66,000,0002 hours (Expected ) 6 minutes CPU: Intel Xeon 2.40GHZ RAM:2GB 14Nodes

Process Discovery 11 Alpha miner Heuristic miner Fuzzy miner Sequence model One instance/case is defined as one visitor’s one time visiting. IP+UA CookieID Activity varies based on different requirements

Behavior analysis 12 User behavior pattern rangeactivityData selection Interaction between channels allContentType Web Map vising path allReferer/URL webpage layoutnewsContentType+Page Type+Block (Channel =news)AND( PageType=19 5) image ContentType+Page Type+Block (Channel =image)AND( PageType=43 5) Searching result all Behavior grouping all Registration

13 User behavior pattern rangeactivityData selection Interaction between channels allContentType Web Map vising path allReferer/URL webpage layoutnewsContentType+Page Type+Block (Channel =news)AND( PageType=19 5) image ContentType+Page Type+Block (Channel =image)AND( PageType=43 5) Searching result all Behavior grouping all Registration

Behavior analysis 14 User behavior pattern rangeactivityData selection Interaction between channels allContentType Web Map vising path allReferer/URL webpage layoutnewsContentType+Page Type+Block (Channel =news)AND( PageType=19 5) image ContentType+Page Type+Block (Channel =image)AND( PageType=43 5) Searching result all Behavior grouping all Registration

Active visitor’s visiting path 15

Behavior analysis 16 User behavior pattern rangeactivityData selection Interaction between channels allContentType Web Map vising path allReferer/URL webpage layoutnewsContentType+Page Type+Block (Channel =news)AND( PageType=19 5) image ContentType+Page Type+Block (Channel =image)AND( PageType=43 5) Searching result all Behavior grouping all Registration

Main page 17

18

Sequence model 19

` 20

XES statistics 21

Conclusion 22 It is a nice project to get into data analysis field,with the combination of web data analysis, process mining and cloud computing technology. Future work: 1 More algorithms and technologies should be applied to this data set. 2 Behavior comparison and user recommendation still need to be accomplished. 3 Can process mining analyze the behavior that does not have a certain pattern. 1 Log Sampling 2 Detect the incorrectness from logs before applying log to analysis technologies. 3 Extend function of “converting data from key-value database or cloud storage to event log” in Prom or XESame.

feedback 23 1 What is the real questions? 2 Why process mining?

Thank you ! Meng Dou 13/9/2012