Presentation is loading. Please wait.

Presentation is loading. Please wait.

Berkeley Data Analytics Stack - Apache Spark

Similar presentations


Presentation on theme: "Berkeley Data Analytics Stack - Apache Spark"— Presentation transcript:

1 Berkeley Data Analytics Stack - Apache Spark
Prof. Chi (Harold) Liu October 2016

2 2018/2/15 Data Processing Goals Low latency (interactive) queries on historical data: enable faster decisions e.g., identify why a site is slow and fix it Low latency queries on live data (streaming): enable decisions on real-time data e.g., detect & block worms in real-time (a worm may infect 1mil hosts in 1.3sec) Sophisticated data processing: enable “better” decisions e.g., anomaly detection, trend analysis So what does this mean? Well, this means that we want low response-time on historical data since the faster we can make a decision the better. We want the ability to perform queries on live data since decisions on real-time data are better than on stale data. Finally, we want to perform sophisticated processing on massive data as, in principle, processing more data will lead to better decisions. 2

3 Today’s Open Analytics Stack…
2018/2/15 Today’s Open Analytics Stack… ..mostly focused on large on-disk datasets: great for batch but slow Data Processing Application Storage Today’s open analytics stack is mostly focused on large on-disk datasets. As a result, it provides sophisticated processing on massive data but it is slow.. This stack consists of roughly four layers… Except Google, Microsoft, and at some extend Amazon, this is the de-facto standarrd stack for data analysis today, and it is used by Yahoo!, Facebook, Twitter, to name a few. Infrastructure 3

4 Goals Batch Interactive Streaming One stack to rule them all!
Easy to combine batch, streaming, and interactive computations Easy to develop sophisticated algorithms Compatible with existing open source ecosystem (Hadoop/HDFS)

5 Support Interactive and Streaming Comp.
Aggressive use of memory Why? 1. Memory transfer rates >> disk or SSDs 2. Many datasets already fit into memory Inputs of over 90% of jobs in Facebook, Yahoo!, and Bing clusters fit into memory e.g., 1TB = 1 billion 1KB each 3. Memory density (still) grows with Moore’s law RAM/SSD hybrid memories at horizon High end datacenter node 10Gbps GB 40-60GB/s 16 cores 0.2-1GB/s (x10 disks) 1-4GB/s (x4 disks) What do people use big data for? First, they use it to generate reports to track and better understand business processes, ransactions Second, they use it to diagnose and answer questions such as Why the user engagement dropping?, why is the system slow? Or to detect spam, worms, or DDoS attacks But most importantly they use it to make decisions, such us improving the business process, deciding what features to add to the product, deciding what ad to show, or, once it identifies a spam, to block it. Thus, the development of the BDAS stack is driven by the believe that “data is as useful as the decisions you can take based on that data” 10-30TB 1-4TB 5

6 Support Interactive and Streaming Comp.
Increase parallelism Why? Reduce work per node  improve latency Techniques: Low latency parallel scheduler that achieve high locality Optimized parallel communication patterns (e.g., shuffle, broadcast) Efficient recovery from failures and straggler mitigation result T What do people use big data for? First, they use it to generate reports to track and better understand business processes, ransactions Second, they use it to diagnose and answer questions such as Why the user engagement dropping?, why is the system slow? Or to detect spam, worms, or DDoS attacks But most importantly they use it to make decisions, such us improving the business process, deciding what features to add to the product, deciding what ad to show, or, once it identifies a spam, to block it. Thus, the development of the BDAS stack is driven by the believe that “data is as useful as the decisions you can take based on that data” result Tnew (< T) 6

7 Support Interactive and Streaming Comp.
Trade between result accuracy and response times Why? In-memory processing does not guarantee interactive query processing e.g., ~10’s sec just to scan 512 GB RAM! Gap between memory capacity and transfer rate increasing Challenges: accurately estimate error and running time for… … arbitrary computations GB 16 cores 40-60GB/s doubles every 18 months doubles every 36 months What do people use big data for? First, they use it to generate reports to track and better understand business processes, ransactions Second, they use it to diagnose and answer questions such as Why the user engagement dropping?, why is the system slow? Or to detect spam, worms, or DDoS attacks But most importantly they use it to make decisions, such us improving the business process, deciding what features to add to the product, deciding what ad to show, or, once it identifies a spam, to block it. Thus, the development of the BDAS stack is driven by the believe that “data is as useful as the decisions you can take based on that data” 7

8 Berkeley Data Analytics Stack (BDAS)
New apps: AMP-Genomics, Carat, … Application Application in-memory processing trade between time, quality, and cost Data Processing Data Processing Data Management Storage Efficient data sharing across frameworks … in two fundamental aspects… At the application layer we build new, real applications such a AMP-Genomics a genomics pipeline, and Carat, an application I’m going to talk about soon. Building real applications allows us to drive the features and design of the lower layers. Infrastructure Resource Management Share infrastructure across frameworks (multi-programming for datacenters) 8

9 AMP Berkeley AMPLab Algorithms Machines People
“Launched” January 2011: 6 Year Plan 8 CS Faculty ~40 students 3 software engineers Organized for collaboration:

10 Berkeley Funding: XData, CISE Expedition Grant
Industrial, founding sponsors 18 other sponsors, including Goal: Next Generation of Analytics Data Stack for Industry & Research: Berkeley Data Analytics Stack (BDAS) Release as Open Source

11 Berkeley Data Analytics Stack (BDAS)
Existing stack components…. Data Management Data Processing Resource Management HDFS MPI Resource Mgmnt. Data Processing Hadoop HIVE Pig HBase Storm Our instantiation of the resource management is Mesos 11

12 Mesos Management platform that allows multiple framework to share cluster Compatible with existing open analytics stack Deployed in production at Twitter on 3,500+ servers HIVE Pig HBase Storm MPI Data Processing Hadoop Our instantiation of the resource management is Mesos HDFS Data Mgmnt. Mesos Resource Mgmnt. 12

13 Spark In-memory framework for interactive and iterative computations
Resilient Distributed Dataset (RDD): fault-tolerance, in-memory storage abstraction Scala interface, Java and Python APIs HIVE Pig Storm MPI Data Processing Spark Hadoop Our instantiation of the resource management is Mesos HDFS Data Mgmnt. Mesos Resource Mgmnt. 13

14 Spark Streaming [Alpha Release]
Large scale streaming computation Ensure exactly one semantics Integrated with Spark  unifies batch, interactive, and streaming computations! Spark Streaming HIVE Pig Storm MPI Data Processing Spark Hadoop Our instantiation of the resource management is Mesos HDFS Data Mgmnt. Mesos Resource Mgmnt. 14

15 Shark  Spark SQL HIVE over Spark: SQL-like interface (supports Hive 0.9) up to 100x faster for in-memory data, and 5-10x for disk In tests on hundreds node cluster at Spark Streaming HIVE Pig Storm MPI Data Processing Shark Spark Hadoop Our instantiation of the resource management is Mesos HDFS Data Mgmnt. Mesos Resource Mgmnt. 15

16 Tachyon High-throughput, fault-tolerant in-memory storage
Interface compatible to HDFS Support for Spark and Hadoop Spark Streaming HIVE Pig Storm MPI Data Processing Shark Spark Hadoop Our instantiation of the resource management is Mesos HDFS Tachyon Data Mgmnt. Resource Mgmnt. Mesos 16

17 BlinkDB Large scale approximate query engine
Allow users to specify error or time bounds Preliminary prototype starting being tested at Facebook Spark Streaming BlinkDB Pig Storm MPI Data Processing Shark HIVE Spark Hadoop Our instantiation of the resource management is Mesos HDFS Tachyon Data Mgmnt. Mesos Resource Mgmnt. 17

18 SparkGraph GraphLab API and Toolkits on top of Spark
Fault tolerance by leveraging Spark Spark Streaming Spark Graph BlinkDB Pig Storm MPI Data Processing Shark HIVE Spark Hadoop HDFS Our instantiation of the resource management is Mesos Tachyon Data Mgmnt. Resource Mgmnt. Mesos 18

19 MLlib Declarative approach to ML Develop scalable ML algorithms
Make ML accessible to non-experts Spark Streaming Spark Graph MLbase BlinkDB Pig Storm MPI Data Processing Shark HIVE Spark Hadoop Our instantiation of the resource management is Mesos HDFS Tachyon Data Mgmnt. Resource Mgmnt. Mesos 19

20 Compatible with Open Source Ecosystem
Support existing interfaces whenever possible GraphLab API Hive Interface and Shell Spark Streaming Spark Graph MLbase BlinkDB Pig Storm MPI Data Processing Shark HIVE Spark Hadoop HDFS Our instantiation of the resource management is Mesos Tachyon Compatibility layer for Hadoop, Storm, MPI, etc to run over Mesos Data Mgmnt. HDFS API Resource Mgmnt. Mesos 20

21 Compatible with Open Source Ecosystem
Use existing interfaces whenever possible Accept inputs from Kafka, Flume, Twitter, TCP Sockets, … Support Hive API Spark Streaming Spark Graph MLbase BlinkDB Pig Storm MPI Data Processing Shark HIVE Support HDFS API, S3 API, and Hive metadata Spark Hadoop Our instantiation of the resource management is Mesos HDFS Tachyon Data Mgmnt. Resource Mgmnt. Mesos 21

22 Summary Support interactive and streaming computations
In-memory, fault-tolerant storage abstraction, low-latency scheduling,... Easy to combine batch, streaming, and interactive computations Spark execution engine supports all comp. models Easy to develop sophisticated algorithms Scala interface, APIs for Java, Python, Hive QL, … New frameworks targeted to graph based and ML algorithms Compatible with existing open source ecosystem Open source (Apache/BSD) and fully committed to release high quality software Three-person software engineering team lead by Matt Massie (creator of Ganglia, 5th Cloudera engineer) Batch Interactive Streaming Spark

23 In-Memory Cluster Computing for Iterative and Interactive Applications
Spark In-Memory Cluster Computing for Iterative and Interactive Applications UC Berkeley 23

24 Background Commodity clusters have become an important computing platform for a variety of applications In industry: search, machine translation, ad targeting, … In research: bioinformatics, NLP, climate simulation, … High-level cluster programming models like MapReduce power many of these apps Theme of this work: provide similarly powerful abstractions for a broader class of applications Explain that trends in data collection rates vs processor and IO speeds are driving this 24

25 Motivation Current popular programming models for clusters transform data flowing from stable storage to stable storage e.g., MapReduce: Map Reduce Input Output Also applies to Dryad, SQL, etc Benefits: easy to do fault tolerance and 25

26 Motivation Acyclic data flow is a powerful abstraction, but is not efficient for applications that repeatedly reuse a working set of data: Iterative algorithms (many in machine learning) Interactive data mining tools (R, Excel, Python) Spark makes working sets a first-class concept to efficiently support these apps

27 Spark Goal Provide distributed memory abstractions for clusters to support apps with working sets Retain the attractive properties of MapReduce: Fault tolerance (for crashes & stragglers) Data locality Scalability RDDs = first-class way to manipulate and persist intermediate datasets Solution: augment data flow model with “resilient distributed datasets” (RDDs) 27

28 Example: Log Mining Load error messages from a log into memory, then interactively search for various patterns Base RDD Cache 1 Transformed RDD lines = spark.textFile(“hdfs://...”) errors = lines.filter(_.startsWith(“ERROR”)) messages = errors.map(_.split(‘\t’)(2)) cachedMsgs = messages.cache() Worker Driver results tasks Block 1 Cached RDD Parallel operation cachedMsgs.filter(_.contains(“foo”)).count cachedMsgs.filter(_.contains(“bar”)).count Cache 2 . . . Cache 3 Block 2 Result: full-text search of Wikipedia in <1 sec (vs 20 sec for on-disk data) Block 3 28

29 Spark Components

30

31 Programming Model by RDD
Resilient distributed datasets (RDDs) Immutable collections partitioned across cluster that can be rebuilt if a partition is lost Created by transforming data in stable storage using data flow operators (map, filter, group-by, …) Can be cached across parallel operations Parallel operations on RDDs Reduce, collect, count, save, … You write a single program  similar to DryadLINQ Distributed data sets with parallel operations on them are pretty standard; the new thing is that they can be reused across operations Variables in the driver program can be used in parallel ops; accumulators useful for sending information back, cached vars are an optimization Mention cached vars useful for some workloads that won’t be shown here Mention it’s all designed to be easy to distribute in a fault-tolerant fashion 31

32 RDDs in More Detail An RDD is an immutable, partitioned, logical collection of records Need not be materialized, but rather contains information to rebuild a dataset from stable storage Partitioning can be based on a key in each record (using hash or range partitioning) Built using bulk transformations on other RDDs Can be cached for future reuse

33

34 RDD Operations Transformations (define a new RDD) map filter sample union groupByKey reduceByKey join cache … Parallel operations (Actions) (return a result to driver) reduce collect count save lookupKey …

35

36

37

38

39

40 RDD Fault Tolerance RDDs maintain lineage information that can be used to reconstruct lost partitions e.g.: cachedMsgs = textFile(...).filter(_.contains(“error”)) .map(_.split(‘\t’)(2)) .cache() HdfsRDD path: hdfs://… FilteredRDD func: contains(...) MappedRDD func: split(…) CachedRDD 40

41 Example 1: Logistic Regression
Goal: find best line separating two sets of points random initial line + + + + + + + + Note that dataset is reused on each gradient computation + + target 41

42 Logistic Regression Code
val data = spark.textFile(...).map(readPoint).cache() var w = Vector.random(D) for (i <- 1 to ITERATIONS) { val gradient = data.map(p => (1 / (1 + exp(-p.y*(w dot p.x))) - 1) * p.y * p.x ).reduce(_ + _) w -= gradient } println("Final w: " + w) 42

43 Logistic Regression Performance
127 s / iteration first iteration 174 s further iterations 6 s This is for a 29 GB dataset on 20 EC2 m1.xlarge machines (4 cores each) 43

44 Example 2: MapReduce Or with combiners:
MapReduce data flow can be expressed using RDD transformations res = data.flatMap(rec => myMapFunc(rec)) .groupByKey() .map((key, vals) => myReduceFunc(key, vals)) Or with combiners: res = data.flatMap(rec => myMapFunc(rec)) .reduceByKey(myCombiner) .map((key, val) => myReduceFunc(key, val)) 44

45 Example 3 45

46

47

48 RDD Graph DAG, scheduler, lineage

49 RDD Dependency Types RDD Partition

50 Scheduling

51 Scheduler Optimization

52 Event Flow (Direct Acycle Graph)

53 Conclusion


Download ppt "Berkeley Data Analytics Stack - Apache Spark"

Similar presentations


Ads by Google