A high-throughput distributed messaging system Apache Kafka A high-throughput distributed messaging system Johan Lundahl

Agenda Kafka overview Features, strengths and tradeoffs Main concepts and comparisons to other messaging systems Features, strengths and tradeoffs Message format and broker concepts Partitioning, Keyed messages, Replication Producer / Consumer APIs Operation considerations Kafka ecosystem If time permits: Kafka as a real-time processing backbone Brief intro to Storm Kafka-Storm wordcount demo

What is Apache Kafka? Distributed, high-throughput, pub-sub messaging system Fast, Scalable, Durable Main use cases: log aggregation, real-time processing, monitoring, queueing Originally developed by LinkedIn Implemented in Scala/Java Top level Apache project since 2012: http://kafka.apache.org/

Comparison to other messaging systems Traditional: JMS, xxxMQ/AMQP New gen: Kestrel, Scribe, Flume, Kafka Message queues Low throughput, low latency JMS ActiveMQ Qpid RabbitMQ Log aggregators High throughput, high latency Kestrel Scribe Flume Hedwig Batch jobs Kafka

Kafka concepts Producers Broker Consumers Frontend Frontend Service Topic1 Topic1 Topic3 Topic2 Push Broker Kafka Pull Topic3 Topic3 Topic2 Topic1 Topic3 Topic2 Topic1 Consumers Monitoring Stream processing Batch processing Data warehouse

Distributed model Zookeeper Producer Producer Producer KAFKA-156 Producer Producer Producer Producer persistence Partitioned Data Publication Broker Broker Broker Intra cluster replication Zookeeper Ordered subscription Topic1 consumer group Topic2 consumer group

Agenda Kafka overview Features, strengths and tradeoffs Main concepts and comparisons to other messaging systems Features, strengths and tradeoffs Message format and broker concepts Partitioning, Keyed messages, Replication Producer / Consumer APIs Operation considerations Kafka ecosystem If time permits: Kafka as a real-time processing backbone Brief intro to Storm Kafka-Storm wordcount demo

Performance factors Broker doesn’t track consumer state Like a distributed commit log Everything is distributed Low overhead protocol Zero-copy (sendfile) reads/writes Message batching (Producer & Consumer) Usage of page cache backed by sequential disk allocation Compression (End to end) Configurable ack levels From: http://queue.acm.org/detail.cfm?id=1563874

Kafka features and strengths Simple model, focused on high throughput and durability O(1) time persistence on disk Horizontally scalable by design (broker and consumers) Push - pull => consumer burst tolerance Replay messages Multiple independent subscribes per topic Configurable batching, compression, serialization Online upgrades

Tradeoffs Not optimized for millisecond latencies Have not beaten CAP Simple messaging system, no processing Zookeeper becomes a bottleneck when using too many topics/partitions (>>10000) Not designed for very large payloads (full HD movie etc.) Helps to know your data in advance

Agenda Kafka overview Features, strengths and tradeoffs Main concepts and comparisons to other messaging systems Features, strengths and tradeoffs Message format and broker concepts Partitioning, Keyed messages, Replication Producer / Consumer APIs Operation considerations Kafka ecosystem If time permits: Kafka as a real-time processing backbone Brief intro to Storm Kafka-Storm wordcount demo

Message/Log Format Message Length Version Checksum Payload

Log based queue (Simplified model) Broker Producer API used directly by application or through one of the contributed implementations, e.g. log4j/logback appender Topic1 Topic2 Consumer1 Message1 Message1 Message2 Message2 Producer1 Message3 Message3 Consumer2 Message4 Message4 Message5 Message5 Producer2 Message6 Message6 Message7 Message7 Message8 Consumer3 ConsumerGroup1 Message9 Batching Compression Serialization Message10

Partitioning Broker No partition for this guy Group1 Partitions Topic1 Consumer Group1 Broker Partitions Topic1 Producer Producer Producer Consumer Group2 Topic2 Producer Consumer Group3 Producer No partition for this guy Consumer

Keyed messages #partitions=3 hash(key) % #partitions Message1 Message2 BrokerId=1 BrokerId=2 BrokerId=3 Topic1 Topic1 Topic1 Message1 Message2 Message3 Message5 Message4 Message7 Message9 Message6 Message11 Message13 Message8 Message15 Message17 Message10 Message12 Message14 Producer Message16 Message18

Intra cluster replication Replication factor = 3 Broker1 Broker2 Broker3 InSyncReplicas Follower fails: Follower dropped from ISR When follower comes online again: fetch data from leader, then ISR gets updated Topic1 leader Topic1 follower Topic1 follower Message1 Message1 Message1 Message2 Message2 Message2 Message3 Message3 Message3 Leader fails: Detected via Zookeeper from ISR New leader gets elected Message4 Message4 Message4 Message5 Message5 Message5 Message6 Message6 Message6 Message7 Message7 Message7 Message8 Message8 Message8 Producer Message9 ack ack ack Message9 Message9 ack Message10 Message10 Message10 3 commit modes: Commit mode Latency Durability Fire & Forget “none” Weak Leader ack 1 roundtrip Medium Full replication 2 roundtrips Strong

Agenda Kafka overview Features, strengths and tradeoffs Main concepts and comparisons to other messaging systems Features, strengths and tradeoffs Message format and broker concepts Partitioning, Keyed messages, Replication Producer / Consumer APIs Operation considerations Kafka ecosystem If time permits: Kafka as a real-time processing backbone Brief intro to Storm Kafka-Storm wordcount demo

Producer API …or for log aggregation: Configuration parameters: ProducerType (sync/async) CompressionCodec (none/snappy/gzip) BatchSize EnqueueSize/Time Encoder/Serializer Partitioner #Retries MaxMessageSize …

Consumer API(s) High-level (consumer group, auto-commit) Low-level (simple consumer, manual commit)

Agenda Kafka overview Features, strengths and tradeoffs Main concepts and comparisons to other messaging systems Features, strengths and tradeoffs Message format and broker concepts Partitioning, Keyed messages, Replication Producer / Consumer APIs Operation considerations Kafka ecosystem If time permits: Kafka as a real-time processing backbone Brief intro to Storm Kafka-Storm wordcount demo

Broker Protips Reasonable number of partitions – will affect performance Reasonable number of topics – will affect performance Performance decrease with larger Zookeeper ensembles Disk flush rate settings message.max.bytes – max accept size, should be smaller than the heap socket.request.max.bytes – max fetch size, should be smaller than the heap log.retention.bytes – don’t want to run out of disk space… Keep Zookeeper logs under control for same reason as above Kafka brokers have been tested on Linux and Solaris

Operating Kafka Zookeeper usage Distribution Tools: Monitoring Producer loadbalancing Broker ISR Consumer tracking Monitoring JMX Audit trail/console in the making Distribution Tools: Controlled shutdown tool Preferred replica leader election tool List topic tool Create topic tool Add partition tool Reassign partitions tool MirrorMaker

Multi-datacenter replication

Agenda Kafka overview Features, strengths and tradeoffs Main concepts and comparisons to other messaging systems Features, strengths and tradeoffs Message format and broker concepts Partitioning, Keyed messages, Replication Producer / Consumer APIs Operation considerations Kafka ecosystem If time permits: Kafka as a real-time processing backbone Brief intro to Storm Kafka-Storm wordcount demo

Ecosystem Producers: Consumers: Java (in standard dist) Scala (in standard dist) Log4j (in standard dist) Logback: logback-kafka Udp-kafka-bridge Python: kafka-python Python: pykafka Python: samsa Python: pykafkap Python: brod Go: Sarama Go: kafka.go C: librdkafka C/C++: libkafka Clojure: clj-kafka Clojure: kafka-clj Ruby: Poseidon Ruby: kafka-rb Ruby: em-kafka PHP: kafka-php(1) PHP: kafka-php(2) PHP: log4php Node.js: Prozess Node.js: node-kafka Node.js: franz-kafka Erlang: erlkafka Consumers: Java (in standard dist) Scala (in standard dist) Python: kafka-python Python: samsa Python: brod Go: Sarama Go: nuance Go: kafka.go C/C++: libkafka Clojure: clj-kafka Clojure: kafka-clj Ruby: Poseidon Ruby: kafka-rb Ruby: Kafkaesque Jruby::Kafka PHP: kafka-php(1) PHP: kafka-php(2) Node.js: Prozess Node.js: node-kafka Node.js: franz-kafka Erlang: erlkafka Erlang: kafka-erlang Common integration points: Stream Processing Storm - A stream-processing framework. Samza - A YARN-based stream processing framework. Hadoop Integration Camus - LinkedIn's Kafka=>HDFS pipeline. This one is used for all data at LinkedIn, and works great. Kafka Hadoop Loader A different take on Hadoop loading functionality from what is included in the main distribution. AWS Integration Automated AWS deployment Kafka->S3 Mirroring Logging klogd - A python syslog publisher klogd2 - A java syslog publisher Tail2Kafka - A simple log tailing utility Fluentd plugin - Integration with Fluentd Flume Kafka Plugin - Integration with Flume Remote log viewer LogStash integration - Integration with LogStash and Fluentd Official logstash integration Metrics Mozilla Metrics Service - A Kafka and Protocol Buffers based metrics and logging system Ganglia Integration Packing and Deployment RPM packaging Debian packaginghttps://github.com/tomdz/kafka-deb-packaging Puppet integration Dropwizard packaging Misc. Kafka Mirror - An alternative to the built-in mirroring tool Ruby Demo App  Apache Camel Integration Infobright integration

What’s in the future? Topic and transient consumer garbage collection (KAFKA-560/KAFKA-559) Producer side persistence (KAFKA-156/KAFKA-789) Exact mirroring (KAFKA-658) Quotas (KAFKA-656) YARN integration (KAFKA-949) RESTful proxy (KAFKA-639) New build system? (KAFKA-855) More tooling (Console, Audit trail) (KAFKA-266/KAFKA-260) Client API rewrite (Proposal) Application level security (Proposal)

Agenda Kafka overview Features, strengths and tradeoffs Main concepts and comparisons to other messaging systems Features, strengths and tradeoffs Message format and broker concepts Partitioning, Keyed messages, Replication Producer / Consumer APIs Operation considerations Kafka ecosystem If time permits: Kafka as a real-time processing backbone Brief intro to Storm Kafka-Storm wordcount demo

Kafka as a processing pipeline backbone Stream processing Kafka as a processing pipeline backbone Producer Process1 Process2 Kafka topic1 Kafka topic2 Producer Process1 Process2 Producer Process1 Process2 System1 System2

What is Storm? Distributed real-time computation system with design goals: Guaranteed processing No orphaned tasks Horizontally scalable Fault tolerant Fast Use cases: Stream processing, DRPC, Continuous computation 4 basic concepts: streams, spouts, bolts, topologies In Apache incubator Implemented in Clojure

Streams Spouts (t4,s2,e2) (t3,s3) (t2,s1,e2) (t1,s1,e1) (t4,s2,e2) an [infinite] sequence (of tuples) (timestamp,sessionid,exception stacktrace) (t4,s2,e2) (t3,s3) (t2,s1,e2) (t1,s1,e1) Spouts a source of streams (t4,s2,e2) (t3,s3) (t2,s1,e2) (t1,s1,e1) Connects to queues, logs, API calls, event data. Some features like transactional topologies (which gives exactly-once messaging semantics) is only possible using the Kafka-TransactionalSpout-consumer

Bolts (t2,s1,h2) (t1,s1,h1) (t3,s3) (t4,s2,e2) (t5,s4) Filters Transformations Apply functions Aggregations Access DB, APIs etc. Emitting new streams Trident = a high level abstraction on top of Storm

Topologies (t2,s1,h2) (t1,s1,h1) (t4,s2,e2) (t5,s4) (t3,s3) (t6,s6)

Storm cluster Nimbus Compare with Hadoop: (JobTracker) Zookeeper Deploy Topology Compare with Hadoop: Nimbus (JobTracker) Zookeeper Supervisor Supervisor Supervisor Supervisor Supervisor (TaskTrackers) Mesos/YARN

Links Apache Kafka: Papers and presentations Main project page Small Mediawiki case study Storm: Introductory article Realtime discussing blog post Kafka+Storm for realtime BigData Trifecta blog post: Kafka+Storm+Cassandra IBM developer article Kafka+Storm@Twitter BigData Quadfecta blog post