Spark Streaming Large-scale near-real-time stream processing Tathagata Das (TD) UC Berkeley UC BERKELEY

What is Spark Streaming? Framework for large scale stream processing Scales to 100s of nodes Can achieve second scale latencies Integrates with Spark’s batch and interactive processing Provides a simple batch-like API for implementing complex algorithm Can absorb live data streams from Kafka, Flume, ZeroMQ, etc.

Motivation Many important applications must process large streams of live data and provide results in near-real-time Social network trends Website statistics Intrustion detection systems etc. Distributed stream processing framework is required to Scale to large clusters (100s of machines) Achieve low latency (few seconds)

Integration with Batch Processing Many environments require processing same data in live streaming as well as batch post processing Existing framework cannot do both Either do stream processing of 100s of MB/s with low latency Or do batch processing of TBs / PBs of data with high latency Extremely painful to maintain two different stacks Different programming models 2X implementation effort 2X bugs

Stateful Stream Processing Traditional streaming systems have a event-driven record-at-a-time processing model Each node has mutable state For each record, update state & send new records State is lost if node dies! Making stateful stream processing be fault-tolerant is challenging mutable state node 1 node 3 input records node 2 Traditional streaming systems have what we call a “record-at-a-time” processing model. Each node in the cluster processing a stream has a mutable state. As records arrive one at a time, the mutable state is updated, and a new generated record is pushed to downstream nodes. Now making this mutable state fault-tolerant is hard.

Existing Streaming Systems Storm Replays record if not processed by a node Processes each record at least once Double counting - may update mutable state twice! Not fault-tolerant - mutable state can be lost due to failure! Trident – Use transactions to update state Processes each record exactly once Transactions to external database for state updates reduces throughput

Spark Streaming

Discretized Stream Processing Run a streaming computation as a series of very small, deterministic batch jobs live data stream Chop up the live data stream into batches of X seconds Spark treats each batch of data as RDDs and processes them using RDD operations Finally, the processed results of the RDD operations are returned in batches Spark Streaming batches of X seconds Spark processed results

Discretized Stream Processing Run a streaming computation as a series of very small, deterministic batch jobs live data stream Batch sizes as low as ½ second, latency ~ 1 second Potential for combining batch processing and streaming processing in the same system Spark Streaming batches of X seconds Spark processed results

Example – Get hashtags from Twitter val tweets = ssc.twitterStream(authorization) DStream: a sequence of RDD representing a stream of data batch @ t+1 batch @ t batch @ t+2 Twitter Streaming API tweets DStream stored in memory as an RDD (immutable, distributed dataset)

Example – Get hashtags from Twitter val tweets = ssc.twitterStream(authorization) val hashTags = tweets.flatMap(status => getTags(status)) new DStream transformation: modify data in one Dstream to create another DStream batch @ t+1 batch @ t batch @ t+2 tweets DStream Twitter Streaming API flatMap flatMap flatMap … hashTags Dstream [#cat, #dog, … ] new RDDs created for every batch

Example – Get hashtags from Twitter val tweets = ssc.twitterStream(authorization) val hashTags = tweets.flatMap(status => getTags(status)) hashTags.saveAsHadoopFiles("hdfs://...") output operation: to push data to external storage RDD @ t RDD @ t+1 RDD @ t+2 tweets DStream flatMap flatMap flatMap hashTags DStream save every batch saved to HDFS

Example 1 – Get hashtags from Twitter val tweets = ssc.twitterStream(authorization) val hashTags = tweets.flatMap(status => getTags(status)) hashTags.foreach(rdd => { ... }) foreach: do anything with the processed data RDD @ t RDD @ t+1 RDD @ t+2 tweets DStream flatMap flatMap flatMap hashTags DStream foreach foreach foreach Write to database, update analytics UI, do whatever you want

Function object to define the transformation Java Example Scala val tweets = ssc.twitterStream(authorization) val hashTags = tweets.flatMap(status => getTags(status)) hashTags.saveAsHadoopFiles("hdfs://...") Java JavaDStream<Status> tweets = ssc.twitterStream(authorization) JavaDstream<String> hashTags = tweets.flatMap(new Function<...> { }) Function object to define the transformation

Window-based Transformations val tweets = ssc.twitterStream(authorization) val hashTags = tweets.flatMap(status => getTags(status)) val tagCounts = hashTags.window(Minutes(1), Seconds(5)).countByValue() sliding window operation window length sliding interval window length DStream of data sliding interval

Demo

lost partitions recomputed in parallel on other workers Fault-tolerance RDDs remember the sequence of operations that created it from the original fault-tolerant input data Batches of input data are replicated in memory of multiple worker nodes to make them fault-tolerant Data lost due to worker failure, can be recomputed from input data tweets RDD input data replicated in memory flatMap hashTags RDD lost partitions recomputed in parallel on other workers

Arbitrary Stateful Computations Maintain arbitrary per-key state across time Specify function to generate new state based on previous state and new data Example: Maintain per-user mood as state, and update it with their tweets def updateMood(newTweets, lastMood) => newMood moods = tweets.updateStateByKey(updateMood _)

Arbitrary Combos of Batch and Streaming Computations Inter-mix RDD and DStream operations! Example: Join incoming tweets with a spam HDFS file to filter out bad tweets tweets.transform(tweetsRDD => { tweetsRDD.join(spamHDFSFile).filter(...) })

DStream Input Sources Out of the box we provide Kafka Flume HDFS Raw TCP sockets Etc. Simple interface to implement your own data receiver What to do when a receiver is started (create TCP connection, etc.) What to do when a received is stopped (close TCP connection, etc.)

Performance Can process 6 GB/sec (60M records/sec) of data on 100 nodes at sub-second latency Tested with 100 text streams on 100 EC2 instances

Comparison with Storm and S4 Higher throughput than Storm Spark Streaming: 670k records/second/node Storm: 115k records/second/node Apache S4: 7.5k records/second/node Streaming Spark offers similar speed while providing FT and consistency guarantees that these systems lack

Fast Fault Recovery Recovers from faults/stragglers within 1 sec

Real Applications: Mobile Millennium Project Traffic transit time estimation using online machine learning on GPS observations Markov chain Monte Carlo simulations on GPS observations Very CPU intensive, requires dozens of machines for useful computation Scales linearly with cluster size

Unifying Batch and Stream Processing Models Spark program on Twitter log file using RDDs val tweets = sc.hadoopFile("hdfs://...") val hashTags = tweets.flatMap(status => getTags(status)) hashTags.saveAsHadoopFile("hdfs://...") Spark Streaming program on Twitter stream using DStreams val tweets = ssc.twitterStream() hashTags.saveAsHadoopFiles("hdfs://...")

Vision - one stack to rule them all Explore data interactively using Spark Shell to identify problems Use same code in Spark stand- alone programs to identify problems in production logs Use similar code in Spark Streaming to identify problems in live log streams $ ./spark-shell scala> val file = sc.hadoopFile(“smallLogs”) ... scala> val filtered = file.filter(_.contains(“ERROR”)) scala> val mapped = filtered.map(...) object ProcessProductionData { def main(args: Array[String]) { val sc = new SparkContext(...) val file = sc.hadoopFile(“productionLogs”) val filtered = file.filter(_.contains(“ERROR”)) val mapped = filtered.map(...) ... } object ProcessLiveStream { def main(args: Array[String]) { val sc = new StreamingContext(...) val stream = sc.kafkaStream(...) val filtered = file.filter(_.contains(“ERROR”)) val mapped = filtered.map(...) ... }

Our Vision - one stack to rule them all Interactive Processing Batch Processing Stream Processing Spark + Shark BlinkDB Spark Streaming Tachyon A single, unified programming model for all computations Allows inter-operation between all the frameworks Save RDDs from streaming to Tachyon Run Shark SQL queries on streaming RDDs Run direct Spark batch and interactive jobs on streaming RDDs

Thank you! Conclusion Integrated with Spark as an extension Run it locally or EC2 Spark Cluster (takes 5 minute to setup) Streaming programming guide – http://spark.incubator.apache.org/docs/latest/streaming-programming-guide.html Paper – tinyurl.com/dstreams Thank you!