1. Presented by Peifeng Yu
Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing
EECS 582 – F16
Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael J. Franklin, Scott Shenker, Ion Stoica
NSDI 12’
Presented by Peifeng Yu

2. Table of Contents Background RDD in a nutshell
EECS 582 – F16
Background
RDD in a nutshell
Spark the implementation
Evaluation
What’s new in Spark

3. Yet Another Cluster Computing Frameworks?

4. Related Works Problem General purpose Specialized In-memory Storage
MapReduce, Dryad
Specialized
Pregel: iterative graph compute
HaLoop: loop of MapReduce steps
In-memory Storage
Distributed shared memory
Key Value Storage / Databases
Piccolo
External storage needed for data reuse cross computations
Iterative algorithm
Ad-hoc query
Don’t generalize
Hard to implement efficient fault tolerance
- why in-memory computation?
* bottleneck in disk based (MapReduce) systems
* iterative algorithms and interactive data mining tools: in-memory can boost by an order of magnitude
* Data reuse!
- related
* MapReduce and Dryad: need to write result to external storage to enable data reuse between computations
* Specialized system: Pregel, HaLoop
+ doesn't generalize
* In memory storage on cluster
+ distributed shared memory
+ key-value stores
+ databases
+ Piccolo: share distributed, mutable state via a key-value table interface
+ fine-grained updates to mutable state, only way to fault tolerance is to replicate/log

5. RDD in a nutshell Resilient Distributed Dataset
EECS 582 – F16
Resilient Distributed Dataset
General purpose distributed memory abstraction
In-memory
Immutable
Can only be created through deterministic operations (Transformations)
Atomic piece of data: partition
Fault-tolerant
* Only suitable for batch analytics
+ asynchronous applications are not suitable

6. RDD - Operations Transformations Actions
EECS 582 – F16
Transformations
map, filter, union, join, etc.
Actions
count, collect, reduce, lookup, save
+ can only be created through deterministic operations (transformations) on data in stable storage or other RDDs
- map, filter, join
+ Actions are used to get the final result out (More detail in Spark)
+ immutable is not a problem: possible to implement mutable state by having multiple RDDs to represent multiple versions of a dataset.
External Source
RDD1
RDD2
External Result
Transformation
Transformation
Action

7. RDD - Fault Tolerance
EECS 582 – F16
Store actual data => Lineage: how the partitions were derived from other datasets
Checkpoint if the lineage is too long
+ Thanks to the coarse-grained operations, the operation log size is significantly smaller than the actual data
+ no need to materialized at all times, only need to know how it was derived from other datasets
* persistance and partitioning can be controled
* some node (dataset) in the lineage can be persistant either by user request or in large lineage case, automatically by runtime

8. RDD - Persistence & Partitioning
EECS 582 – F16
Persistence Level
In-memory
Disk backed
Replica
Partitioning
Default hashing
User defined
Persistence Level: 7 in total
MEMORY_ONLY: Store RDD as deserialized Java objects in the JVM. If the RDD does not fit in memory, some partitions will not be cached and will be recomputed on the fly each time they're needed. This is the default level.
MEMORY_AND_DISK: Store RDD as deserialized Java objects in the JVM. If the RDD does not fit in memory, store the partitions that don't fit on disk, and read them from there when they're needed.
MEMORY_ONLY_SER: Store RDD as serialized Java objects (one byte array per partition). This is generally more space-efficient than deserialized objects, especially when using a fast serializer, but more CPU-intensive
MEMORY_AND_DISK_SER: Similar to MEMORY_ONLY_SER, but spill partitions that don't fit in memory to disk instead of recomputing them on the fly each time they're needed.
DISK_ONLY : Store the RDD partitions only on disk.
MEMORY_ONLY_2, MEMORY_AND_DISK_2, etc.: Same as the levels above, but replicate each partition on two cluster nodes.
Advantages
In case when re-computation is more costly than storage IO
Help improve data locality

9. RDD vs. DSM Aspect RDDs Distributed Shared Memory Reads
EECS 582 – F16
Aspect
RDDs
Distributed Shared Memory
Reads
Coarse- or fine-grained
Fine-grained
Writes
Coarse-grained
Consistency
Trivial (immutable)
Up to app / runtime
Fault recovery
Fine-grained and low-overhead using lineage
Requires checkpoints and program rollback
Straggler mitigation
Possible using backup tasks
Difficult
Work placement
Automatic based on data locality
Up to app (runtimes aim for transparency)
Behavior if no enough RAM
Similar to existing data flow systems
Poor performance (swapping)
* Comparsion with DSM
+ reads
+ __writes__
+ consistency
+ __fault recovery__
+ __straggler__ (immutable)
+ work placement: runtime can do scheduling to improve data locality
+ OOM: degrade gracefully to current data-parallel system
* Only suitable for batch analytics
+ asynchronous applications are not suitable

10. Spark the implementation
EECS 582 – F16
- Spark: exposes RDDs through a language-integrated API (dataset as object, transformations as methods)
* Actions: operations that return a value to the application or export data to a storage system. RDDs are Lazily computed
* Persistance: in-memory, spill to disk, replica, priority
* closure: transformation that require user functions are passed through network as closure. Instead of transfer data around, the code is transferred, which is relatively small than data

11. Spark the implementation
EECS 582 – F16
Job scheduler
Memory manager
Interactive interpreter (shipping closure around)
Not a cluster manager
Mesos
YARN
Standalone (added later)
Job scheduler: Whenever a user runs an action (e.g., count or save) on an RDD, the scheduler examines that RDD’s lineage graph to build a DAG of stages to execute
* Actions: operations that return a value to the application or export data to a storage system. RDDs are Lazily computed
Memory manager
Interactive interpreter
* Persistence: in-memory, spill to disk, replica, priority
* closure: transformation that require user functions are passed through network as closure. Instead of transfer data around, the code is transferred, which is relatively small than data

12. Spark The Scheduler
EECS 582 – F16
Mapping high-level logical representation to low-level tasks
Build a DAG of stages to execute based on the RDD’s lineage
Transformation are lazily computed
Factors
Data locality
Pipeline
Worker fault tolerance
Mapping from high level logical RDD representation to low level tasks
Data locality
If a task needs to process a partition that is available in memory on a node, we send it to that node.
Otherwise, if a task processes a partition for which the containing RDD provides preferred locations (e.g., an HDFS file), we send it to those.

13. Revisit RDD Dependencies
EECS 582 – F16
Narrow
Pipeline execution
Partition-wise
Easy recover
Wide
All parents must be present to compute any partition
Full re-computation needed for recovering
Narrow Dependencies
Wide Dependencies

14. Spark The Scheduler Stage: pipelined op with narrow dependencies
EECS 582 – F16
Stage: pipelined op with narrow dependencies
Boundaries
shuffle operations required by wide dependencies
Already computed partitions
Stage: pipelined transformations with narrow dependencies
Boundaries
shuffle operations required by wide dependencies
Already computed partitions

15. Spark The Scheduler Fault tolerance
EECS 582 – F16
Fault tolerance
Re-run on another node in case a task fails
Resubmit tasks for missing partitions in parallel
Only worker failures are tolerated
Scheduler (master) failure can be recovered by using additional service like zookeeper or simple local filesystem based checkpoint
Optimization for long lineage: checkpointing
Leave to the user to decide which RDD to checkpoint
If a task fails, we re-run it on another node as long as its stage’s parents are still available.
If some stages have become unavailable (e.g., because an output from the “map side” of a shuffle was lost), we resubmit tasks to compute the missing partitions in parallel.
We do not yet tolerate scheduler failures, though replicating the RDD lineage graph would be straightforward.
Optimization for long lineage
For RDDs with narrow dependencies on data in stable storage, such as the points in our logistic regression example (§3.2.1) and the link lists in PageRank, checkpointing may never be worthwhile.

16. Spark the Memory Manager
EECS 582 – F16
Options for storage of persistent RDDs
In-memory vs. on-disk
Deserialized vs. serialized
Single copy vs. replica
Insufficient Memory
LRU (skipping the RDD currently operating on)
User defined priority
LRU
When a new RDD partition is computed but there is not enough space to store it, we evict a partition from the least recently accessed RDD, unless this is the same RDD as the one with the new partition.
In that case, we keep the old partition in memory to prevent cycling partitions from the same RDD in and out.

17. Evaluation Iterative machine learning Limited memory
EECS 582 – F16
Iterative machine learning
Limited memory
Interactive data mining
Others, refer to paper for details
Go through most of them quickly, only highlight a few interesting points

18. Evaluation Iterative machine learning
EECS 582 – F16
Iterative machine learning
Spark : first vs. later iterations
Hadoop, Spark: first iteration
HadoopBinMem, Spark: later iteration
Logistic regression
K-means
- Hadoop
- HadoopBinMem: A Hadoop deployment that converts the input data into a low-overhead binary format in the first iteration to eliminate text parsing in later ones, stores it in an in-memory HDFS instance
Spark
Spark: First vs later iteration
read text input from HDFS in their first iterations.
Hadoop, Spark: first iteration
Overheads in Hadoop’s heartbeat between worker and master
HadoopBinMem, Spark: later iteration
Overhead of Hadoop software stack
Overhead of HDFS
Deserialization cost to convert binary records to usable in-memory Java objects

19. Evaluation Limited memory Logistic regression Graceful degradation
EECS 582 – F16
Limited memory
Logistic regression
Graceful degradation
With less memory available, more partitions are saved to disk, resulting in longer execution time.

20. Evaluation Interactive data mining
EECS 582 – F16
Interactive data mining
Not really the kind of “instant feedback” you would get like in Google Instant Search
Still quite usable, compared to several hundreds second when working from disk

21. Take Away RDD Spark Immutable in-memory data partitions
EECS 582 – F16
RDD
Immutable in-memory data partitions
Fault tolerance using lineage, with optional checkpoint
Lazily computed until user requested
Limited operation, but still quite expressive
Spark
Schedule computation task
Move data and code around in cluster
Interactive interpreter

22. What’s new In Spark Language bindings Libraries built on top of Spark
EECS 582 – F16
Language bindings
Java, Scala, Python, R
Libraries built on top of Spark
Spark SQL: working with structured data, mix SQL queries with Spark programs
Spark Streaming: build scalable fault-tolerant streaming application
MLlib: scalable machine learning library
GraphX: API for graphs and graph-parallel computation
SparkNet: distributed neural networks for Spark. Paper
Accepted to Apache Incubator in 2013
Spark Streaming: use sliding to form mini batch, operation on higher level concept called DStream
MLlib: high performance, because of Spark excels at iterative computation, announced to be 100x faster than MapReduce

23. Code Example
EECS 582 – F16

24. Example - Inverted Index
EECS 582 – F16
Spark version (python)
"""InvertedIndex.py""”
from pyspark import SparkContext
sc = SparkContext("local", ”Inverted Index")
docFile = ”path/to/input/file"  # Should be some file on your system
# each record is <docId, docContent>
docData = sc.textFile(docFile)
# split words, of type <word, docId>
docWords = docData.flatMap(lambda k, v: [(wd, docId) for wd in v.split()])
# sort and then group by key, invIndex is of type <word, list<docId> >
invIndex = docWords.sortByKey().groupByKey()
# persist
invIndex.save(‘path/to/output/file’)

25. Example - Inverted Index
EECS 582 – F16
MapReduce version (pseudo code)
map(String key, String value):
// key: document id // value: document contents
for each word w in value:
EmitIntermediate(w, key);
reduce(String key, Iterator values):
// key: a word // values: a list of document ids sort(values)
Emit(key, values)

26. What if
EECS 582 – F16
What if we want to do streaming data analytic, what’s the best way given a batch processing system like Spark?
Optimal partition function? Application specific?