1. 巨量資料基礎： MapReduce典範、Hadoop與Spark生態系統
Tamkang University
Tamkang University
Big Data Mining 巨量資料探勘
巨量資料基礎： MapReduce典範、Hadoop與Spark生態系統
(Fundamental Big Data: MapReduce Paradigm, Hadoop and Spark Ecosystem)
1052DM02
MI4 (M2244) (3069)
Thu, 8, 9 (15:10-17:00) (B130)
Min-Yuh Day
戴敏育
Assistant Professor
專任助理教授
Dept. of Information Management, Tamkang University
淡江大學 資訊管理學系
tku.edu.tw/myday/

2. 課程大綱 (Syllabus)
週次 (Week) 日期 (Date) 內容 (Subject/Topics) /02/16 巨量資料探勘課程介紹 (Course Orientation for Big Data Mining) /02/23 巨量資料基礎：MapReduce典範、Hadoop與Spark生態系統 (Fundamental Big Data: MapReduce Paradigm, Hadoop and Spark Ecosystem) /03/02 關連分析 (Association Analysis) /03/09 分類與預測 (Classification and Prediction) /03/16 分群分析 (Cluster Analysis) /03/23 個案分析與實作一 (SAS EM 分群分析)： Case Study 1 (Cluster Analysis – K-Means using SAS EM) /03/30 個案分析與實作二 (SAS EM 關連分析)： Case Study 2 (Association Analysis using SAS EM)

3. 課程大綱 (Syllabus)
週次 (Week) 日期 (Date) 內容 (Subject/Topics) /04/06 教學行政觀摩日 (Off-campus study) /04/13 期中報告 (Midterm Project Presentation) /04/20 期中考試週 (Midterm Exam) /04/27 個案分析與實作三 (SAS EM 決策樹、模型評估)： Case Study 3 (Decision Tree, Model Evaluation using SAS EM) /05/04 個案分析與實作四 (SAS EM 迴歸分析、類神經網路)： Case Study 4 (Regression Analysis, Artificial Neural Network using SAS EM) /05/11 Google TensorFlow 深度學習 (Deep Learning with Google TensorFlow) /05/18 期末報告 (Final Project Presentation) /05/25 畢業班考試 (Final Exam)

4. 2017/02/ 巨量資料基礎： MapReduce典範、 Hadoop與Spark生態系統 (Fundamental Big Data: MapReduce Paradigm, Hadoop and Spark Ecosystem)

5. Big Data Analytics and Data Mining

6. Architectures of Big Data Analytics

7. Architecture of Big Data Analytics
Big Data Analytics Applications
Big Data Sources
Big Data Transformation
Big Data Platforms & Tools
* Internal
* External
* Multiple formats
* Multiple locations
* Multiple applications
Middleware
Hadoop
MapReduce
Pig
Hive
Jaql
Zookeeper
Hbase
Cassandra
Oozie
Avro
Mahout
Others
Queries
Transformed Data
Big Data Analytics
Raw Data
Extract Transform Load
Reports
Data Warehouse
OLAP
Traditional Format CSV, Tables
Data Mining
Source: Stephan Kudyba (2014), Big Data, Mining, and Analytics: Components of Strategic Decision Making, Auerbach Publications

8. Architecture of Big Data Analytics
Big Data Analytics Applications
Big Data Sources
Big Data Transformation
Big Data Platforms & Tools
* Internal
* External
* Multiple formats
* Multiple locations
* Multiple applications
Data Mining
Big Data Analytics Applications
Middleware
Hadoop
MapReduce
Pig
Hive
Jaql
Zookeeper
Hbase
Cassandra
Oozie
Avro
Mahout
Others
Queries
Transformed Data
Big Data Analytics
Raw Data
Extract Transform Load
Reports
Data Warehouse
OLAP
Traditional Format CSV, Tables
Data Mining
Source: Stephan Kudyba (2014), Big Data, Mining, and Analytics: Components of Strategic Decision Making, Auerbach Publications

9. Architecture for Social Big Data Mining (Hiroshi Ishikawa, 2015)
Enabling Technologies
Analysts
Integrated analysis
Model Construction
Explanation by Model
Integrated analysis model
Conceptual Layer
Construction and confirmation of individual hypothesis
Description and execution of application-specific task
Natural Language Processing
Information Extraction
Anomaly Detection
Discovery of relationships among heterogeneous data
Large-scale visualization
Data Mining
Application specific task
Multivariate analysis
Logical Layer
Parallel distrusted processing
Social Data
Software
Hardware
Physical Layer
Source: Hiroshi Ishikawa (2015), Social Big Data Mining, CRC Press

10. Business Intelligence (BI) Infrastructure
Source: Kenneth C. Laudon & Jane P. Laudon (2014), Management Information Systems: Managing the Digital Firm, Thirteenth Edition, Pearson.

11. Data Warehouse Data Mining and Business Intelligence
Increasing potential
to support
business decisions
End User
Decision Making
Data Presentation
Business
Analyst
Visualization Techniques
Data Mining
Data
Analyst
Information Discovery
Data Exploration
Statistical Summary, Querying, and Reporting
Data Preprocessing/Integration, Data Warehouses
DBA
Data Sources
Paper, Files, Web documents, Scientific experiments, Database Systems
Source: Jiawei Han and Micheline Kamber (2006), Data Mining: Concepts and Techniques, Second Edition, Elsevier

12. The Evolution of BI Capabilities
Source: Turban et al. (2011), Decision Support and Business Intelligence Systems

13. Data Science and Business Intelligence
Source: EMC Education Services, Data Science and Big Data Analytics: Discovering, Analyzing, Visualizing and Presenting Data, Wiley, 2015

14. Data Science and Business Intelligence
Predictive Analytics and Data Mining
(Data Science)
Source: EMC Education Services, Data Science and Big Data Analytics: Discovering, Analyzing, Visualizing and Presenting Data, Wiley, 2015

15. Data Science and Business Intelligence
Predictive Analytics and Data Mining
(Data Science)
Data Science and Business Intelligence
Structured/unstructured data, many types of sources, very large datasets
Optimization, predictive modeling, forecasting statistical analysis
What if…?
What’s the optimal scenario for our business?
What will happen next?
What if these trends countinue?
Why is this happening?
Source: EMC Education Services, Data Science and Big Data Analytics: Discovering, Analyzing, Visualizing and Presenting Data, Wiley, 2015

16. Data Mining at the Intersection of Many Disciplines
Source: Turban et al. (2011), Decision Support and Business Intelligence Systems

17. A Taxonomy for Data Mining Tasks
Source: Turban et al. (2011), Decision Support and Business Intelligence Systems

18. Traditional Analytics
BI and Analytics
Data Mart
Operational Data Sources
Data Mart
EDW
Analytic Mart
Analytic Mart
Unstructured, Semi-structured and Streaming data (i.e. sensor data) handled often outside the Warehouse flow
Source: Deepak Ramanathan (2014), SAS Modernization architectures - Big Data Analytics

19. Hadoop as a “new data” Store
BI and Analytics
Data Mart
Operational Data Sources
Data Mart
EDW
Analytic Mart
Analytic Mart
Source: Deepak Ramanathan (2014), SAS Modernization architectures - Big Data Analytics

20. Hadoop as an additional input to the EDW
BI and Analytics
Data Mart
Operational Data Sources
Data Mart
EDW
Analytic Mart
Analytic Mart
Analytic Mart
Data Mart
Source: Deepak Ramanathan (2014), SAS Modernization architectures - Big Data Analytics

21. Operational Data Sources
Hadoop Data Platform As a “staging Layer” as part of a “data Lake” – Downstream stores could be Hadoop, data appliances or an RDBMS
BI and Analytics
Operational Data Sources
EDW
Data Mart
Data Mart
Analytic Mart
Analytic Mart
Source: Deepak Ramanathan (2014), SAS Modernization architectures - Big Data Analytics

22. SAS Big data Strategy – SAS areas
Source: Deepak Ramanathan (2014), SAS Modernization architectures - Big Data Analytics

23. SAS Big data Strategy – SAS areas
Source: Deepak Ramanathan (2014), SAS Modernization architectures - Big Data Analytics

24. SAS® Within the HADOOP ECOSYSTEMEG EM VA
SAS® Enterprise
Guide®
SAS® Data Integration
SAS® Enterprise Miner™
SAS® Visual Analytics
SAS® In-Memory Statistics for Haodop
User Interface
SAS Metadata
Metadata
Next-Gen
SAS® User
SAS® User
Base SAS & SAS/ACCESS® to Hadoop™
In-Memory
Data Access
Data Access
Impala
Pig
Hive
SAS Embedded
Process
Accelerators
SAS® LASR™ Analytic
Server
SAS® High-
Performance
Analytic Procedures
Data Processing
Map Reduce
MPI Based
HDFS
File System
Source: Deepak Ramanathan (2014), SAS Modernization architectures - Big Data Analytics

25. SAS enables the entire lifecycle around HADOOP
Done using either the Data Preparation, Data Exploration or Build Model Tools
SAS Visual Analytics
Decision Manager
IDENTIFY /
FORMULATE
PROBLEM
DATA
PREPARATION
EXPLORATION
TRANSFORM
& SELECT
BUILD
MODEL
VALIDATE
DEPLOY
EVALUATE /
MONITOR
RESULTS
SAS Visual Analytics SAS Visual Statistics
SAS In-Memory Statistics for Hadoop
SAS Scoring Accelerator for Hadoop
SAS Code Accelerator for Hadoop
Done using either the Data Preparation, Data Exploration or Build Model Tools
Decision Manager
SAS High Performance Analytics Offerings supported by relevant clients like SAS Enterprise Miner, SAS/STAT etc.
Source: Deepak Ramanathan (2014), SAS Modernization architectures - Big Data Analytics

26. Data Mining Process

27. Data Mining Process A manifestation of best practices
A systematic way to conduct DM projects
Different groups has different versions
Most common standard processes:
CRISP-DM (Cross-Industry Standard Process for Data Mining)
SEMMA (Sample, Explore, Modify, Model, and Assess)
KDD (Knowledge Discovery in Databases)
Source: Turban et al. (2011), Decision Support and Business Intelligence Systems

28. Data Mining Process (SOP of DM)
What main methodology are you using for your analytics, data mining, or data science projects ?
Source:

29. Data Mining Process
Source:

30. Data Mining: Core Analytics Process The KDD Process for Extracting Useful Knowledge from Volumes of Data
Source: Fayyad, U., Piatetsky-Shapiro, G., & Smyth, P. (1996). The KDD Process for Extracting Useful Knowledge from Volumes of Data. Communications of the ACM, 39(11),

31. Fayyad, U. , Piatetsky-Shapiro, G. , & Smyth, P. (1996)
Fayyad, U., Piatetsky-Shapiro, G., & Smyth, P. (1996). The KDD Process for Extracting Useful Knowledge from Volumes of Data. Communications of the ACM, 39(11),

32. Data Mining Knowledge Discovery in Databases (KDD) Process (Fayyad et al., 1996)
Source: Fayyad, U., Piatetsky-Shapiro, G., & Smyth, P. (1996). The KDD Process for Extracting Useful Knowledge from Volumes of Data. Communications of the ACM, 39(11),

33. Knowledge Discovery in Databases (KDD) Process
Data mining: core of knowledge discovery process
Knowledge
Evaluation and Presentation
Data Mining
Patterns
Selection and Transformation
Task-relevant Data
Data Warehouse
Cleaning and
Integration
Databases
Flat files
Source: Jiawei Han and Micheline Kamber (2006), Data Mining: Concepts and Techniques, Second Edition, Elsevier

34. Data Mining Process: CRISP-DM
Source: Turban et al. (2011), Decision Support and Business Intelligence Systems

35. Data Mining Process: CRISP-DM
Step 1: Business Understanding
Step 2: Data Understanding
Step 3: Data Preparation (!)
Step 4: Model Building
Step 5: Testing and Evaluation
Step 6: Deployment
The process is highly repetitive and experimental (DM: art versus science?)
Accounts for ~85% of total project time
Source: Turban et al. (2011), Decision Support and Business Intelligence Systems

36. Data Preparation – A Critical DM Task
Source: Turban et al. (2011), Decision Support and Business Intelligence Systems

37. Data Mining Process: SEMMA
Source: Turban et al. (2011), Decision Support and Business Intelligence Systems

38. Data Mining Processing Pipeline (Charu Aggarwal, 2015)
Data Collection
Data Preprocessing
Analytical Processing
Output for Analyst
Feature Extraction
Cleaning and Integration
Building Block 1
Building Block 2
Feedback (Optional)
Feedback (Optional)
Source: Charu Aggarwal (2015), Data Mining: The Textbook Hardcover, Springer

39. Fundamental Big Data: MapReduce Paradigm, Hadoop and Spark Ecosystem

40. Source: https://www. thalesgroup

41. MapReduce Paradigm

42. MapReduce Paradigm Map Reduce Big Data Map0 Map1 Map2 Map3 Reduce0
MapReduce Data
Output Data

43. Source: https://www.edureka.co/blog/mapreduce-tutorial/
MapReduce Word Count
Input
Dog Love Cat
Bird Love Bird
Dog Bird Cat
Source:

44. Source: https://www.edureka.co/blog/mapreduce-tutorial/
MapReduce Word Count
Input
Output
Dog Love Cat
Bird Love Bird
Dog Bird Cat
Bird, 3
Cat, 2
Dog, 2
Love, 2
Source:

45. Source: https://www.edureka.co/blog/mapreduce-tutorial/
MapReduce Word Count
Input
Split
Map
Shuffle
Reduce
Output
Bird, (1, 1, 1)
Bird, 3
Dog, 1
Love, 1
Cat, 1
Dog Love Cat
Cat, (1, 1)
Cat, 2
Dog Love Cat
Bird Love Bird
Dog Bird Cat
Bird, 3
Cat, 2
Dog, 2
Love, 2
Bird, 1
Love, 1
Bird Love Bird
Dog, (1, 1)
Dog, 2
Dog, 1
Bird, 1
Cat, 1
Dog Bird Cat
Love, (1, 1)
Love, 2
Source:

46. Hadoop Ecosystem

47. The Apache™ Hadoop® project develops open-source software for reliable, scalable, distributed computing.
Source:

48. Processing
MapReduce
HDFS
Storage
Source:

49. Big Data with Hadoop Architecture
Source:

50. Big Data with Hadoop Architecture Logical Architecture Processing: MapReduce
Source:

51. Big Data with Hadoop Architecture Logical Architecture Storage: HDFS
Source:

52. Big Data with Hadoop Architecture Process Flow
Source:

53. Big Data with Hadoop Architecture Hadoop Cluster
Source:

54. Hadoop Ecosystem
Source: Shiva Achari (2015), Hadoop Essentials - Tackling the Challenges of Big Data with Hadoop, Packt Publishing

55. HDP (Hortonworks Data Platform) A Complete Enterprise Hadoop Data Platform
Source:

56. Apache Hadoop Hortonworks Data Platform
Source:

57. Hadoop and Data Analytics Tools
Source:

58. Hadoop 1  Hadoop 2
Source:

59. Source: http://www.newera-technologies.com/big-data-solution.html
Big Data Solution EG EM VA
Source:

60. Traditional ETL Architecture
Source:

61. Offload ETL with Hadoop (Big Data Architecture)
Source:

62. Spark Ecosystem

63. Lightning-fast cluster computing
Apache Spark is a fast and general engine for large-scale data processing.
Source:

64. Logistic regression in Hadoop and Spark
Run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk.
Source:

65. Ease of Use Write applications quickly in Java, Scala, Python, R.
Source:

66. Word count in Spark's Python API
text_file = spark.textFile("hdfs://...") text_file.flatMap(lambda line: line.split()) .map(lambda word: (word, 1)) .reduceByKey(lambda a, b: a+b)
Source:

67. Spark and Hadoop
Source:

68. Spark Ecosystem
Source:

69. MLlib (machine learning)
Spark Ecosystem
Spark
Spark
Streaming
MLlib (machine learning)
Spark
SQL
GraphX
(graph)
Kafka
Flume
H2O
Hive
Titan
HBase
Cassandra
HDFS
Source: Mike Frampton (2015), Mastering Apache Spark, Packt Publishing

70. Hadoop vs. Spark HDFS read HDFS write HDFS read HDFS write Iter. 1
Input
Iter. 1
Iter. 2
Input
Source: Shiva Achari (2015), Hadoop Essentials - Tackling the Challenges of Big Data with Hadoop, Packt Publishing

71. Steps to Install Hadoop on a Personal Computer (Windows/OS X)
Source:

72. Hodoop: Linux Based Software
Source:

73. Appliance Hadoop Personal Computer (Windows / OS X)
Virtual Machine (VirtualBox / VMWare)
Linux
Hadoop
Source:

74. Connection to Hadoop Hadoop Access from host
Personal Computer (Windows / OS X)
Browser
Virtual Machine (VirtualBox / VMWare)
Linux
Hadoop
Source:

75. Steps to Install Hadoop on a Personal Computer (Windows/OS X)
Step 1. Download and Install VirtualBox
Step 2. Download Appliance
Step 3. Import Appliance
Step 4. Configure Virtual Machine (VM)
Step 5. Start Virtual Machine (VM)
Step 6. Test Connection From Host
Source:

76. Virtual Box

77. Steps to Install Hadoop on a Personal Computer (Windows/OS X)
Step 1. Download and Install VirtualBox
Step 2. Download Appliance
Hortonworks Sandbox
Step 3. Import Appliance
Step 4. Configure Virtual Machine (VM)
Step 5. Start Virtual Machine (VM)
Step 6. Test Connection From Host
Source:

78. Hortonworks Sandbox The easiest way to get started with Enterprise Hadoop

79. Get started on Hadoop with these tutorials based on the Hortonworks Sandbox

80. Apache Hadoop

81. Apache Hadoop

82. Apache Hadoop YARN
Source:

83. Apache Spark

84. References
EMC Education Services (2015), Data Science and Big Data Analytics: Discovering, Analyzing, Visualizing and Presenting Data, Wiley
Shiva Achari (2015), Hadoop Essentials - Tackling the Challenges of Big Data with Hadoop, Packt Publishing
Mike Frampton (2015), Mastering Apache Spark, Packt Publishing
Deepak Ramanathan (2014), SAS Modernization architectures - Big Data Analytics,