Hadoop Ecosystem Overview CMSC 491 Hadoop-Based Distributed Computing Spring 2015 Adam Shook

Agenda Introduce Hadoop projects to prepare you for your group work Intimate detail will be provided in future lectures Discuss potential use cases for each project

Topics HDFS MapReduce YARN Sqoop Flume NiFi Pig Hive Streaming HBase Accumulo Avro Parquet Mahout Oozie Storm ZooKeeper Spark SQL-on-Hadoop In-Memory Stores Cassandra Kafka Crunch Azkaban

HDFS Hadoop Distributed File System We’ve talked about this enough High-performance file system for storing data We’ve talked about this enough

Hadoop MapReduce High-performance fault-tolerance data processing system We’ve also talked about this enough

YARN Abstract framework for distributed application development Split functionality of JobTracker into two components ResourceManager ApplicationMaster TaskTracker becomes NodeManager Containers instead of map and reduce slots Configurable amount of memory per NodeManager RM manages global assignment of compute resources to applications AM manages application life cycle – tasked to negotiate resources form the RM and works with NM to execute and monitor tasks NodeManager executes containers

MapReduce 2.x on YARN MapReduce API has not changed Binary-level backwards compatible (no recompile) Application Master launches and monitors job via YARN MapReduce History Server to store… history Enabled Yahoo! to scale beyond 4,000 nodes In YARN, a MapReduce application is equivalent to a job, executed by the MapReduce AM

Hadoop Ecosystem Core Technologies Many other tools… Hadoop Distributed File System Hadoop MapReduce Many other tools… Which we will be discussing… now

Apache Sqoop Apache project designed for efficient transfer between Apache Hadoop and structured data stores Use through CLI and extendable Use cases? Migrating off EDWs Pushing MapReduce output to EDW for application consumption or analytics

Apache Flume Distributed, reliable, available service for collecting, aggregating, and moving large amounts of log data Configure agents using simple files, extendable Use cases? Capturing log files from web servers Creating complex topologies for moving data Can support ‘events’ instead of just log data

Apache NiFi A service to reliably move and manipulate files between clusters using a web front-end Uses a GUI to drop processors and connect them to build workflows Use cases? Reliably push files between HDFS clusters

Apache Pig Platform for analyzing large data sets that consists of a high-level language for expressing data analysis programs Infrastructure compiles language to a sequence of MapReduce programs Use cases? Enabling MR analytics to those who don’t know Java Rapid prototyping of future Java analytics High-level languages improve over time

Apache Hive Data warehouse facilitating querying and managing large datasets Compiles SQL-like queries into MapReduce programs Use cases? Enabling MR analytics to those who don’t know Java Rapid prototyping of future Java analytics High-level languages improve over time

Hadoop Streaming Utility to create and run MapReduce jobs with any executable or script as the mapper or reducer Just a jar file, not a real project Use cases? Enabling MR analytics to those who don’t know Java Rapid prototyping of future Java analytics High-level languages improve over time

Which high-level API is for you? What are you comfortable with? What are you being told to use? Boils down to what you are comfortable with and what are you being told to use. All of them are highly extensible and

Apache HBase Distributed, scalable, big data store Data stored as sorted key/value pairs, with the key consisting of a row and column Use cases? Sub-second fetches of key/value pairs and fast small range scans CRUD operations Generally faster at reads than Accumulo

Apache Accumulo Robust, scalable, high-performance data storage and retrieval key/value store Cell-based access controls i.e. cell-level security Use cases? Sub-second fetches of key/value pairs and fast small range scans CRUD operations Cell-level security is built into the system Back-end iterators give automated operations on tables Generally faster at writes than HBase

Apache Avro Data serialization system for the Hadoop ecosystem Use cases? A unified data format across all Hadoop ecosystem projects Built for Hadoop

Apache Parquet Columnar storage format for Hadoop Use cases? A unified data format across all Hadoop ecosystem projects Built for Hadoop Columnar storage format can enhance how some projects access data for partition pruning (SQL on Hadoop projects, specifically)

Apache Mahout Machine learning library to build scalable machine learning algorithms implemented on top of Hadoop MapReduce Use cases? Do cool and scalable analytics

Apache Oozie Workflow scheduler system to manage Apache Hadoop jobs Use cases? Can create complex application workflows and have them scheduled via Oozie

Apache Storm Distributed real-time computation system Didn’t have a logo until June 2014 How is this different than MapReduce? Use cases? Scales to hundreds of thousands of IOPs per node Perform functions on streaming data to get immediate value rather than using MR for batch processes

Apache ZooKeeper Effort to develop and maintain and open-source server enabling highly reliable distributed coordination Use cases? Solves a lot of problems you generally have when building distributed systems Can be used for configuration, group services, pretty much anything you want it to do. It’ll remember your birthday without being told by Facebook that it was your birthday. Really great guy.

Apache Spark Fast and general engine for large-scale data processing Write applications in Java, Scala, or Python Use cases? It’s can do in-memory MR, removing the costly “writing out to HDFS between each phase” Has libraries for streaming, SQL, machine learning, and graphs

SQL on Hadoop Apache Drill, Cloudera Impala, Facebook’s Presto, Hortonworks’s Hive Stinger, Pivotal HAWQ, etc. SQL-like or ANSI SQL compliant MPP execution engines using HDFS as a data store Use cases? Non use cases? No overhead with MR gives faster execution times Opens a new paradigm in working with large data sets Can run an MPP database as part of your Hadoop cluster, rather than using an expensive process to move data from HDFS to an EDW

Sample Architecture SQL Pig HBase Storm HDFS SQL Flume Agent Oozie Webserver Website Flume Agent Sales MapReduce Pig HBase Storm HDFS Flume Agent Call Center SQL

We [maybe] won’t be covering these in detail later on These ones we won’t be covering in detail in the class (either less common, mildly unrelated, or fairly new). Students can use them with permission Other Hadoop Projects

Redis, Memcached, etc. Open-source in-memory key/value stores Use cases? Very helpful in supplementing MapReduce analytics or otherwise giving you fast data lookups

Apache Cassandra NoSQL database for managing large amounts of structured, semi-structured, and unstructured data Support for clusters spanning multiple datacenters Unlike HBase and Accumulo, data is not stored on HDFS Use cases? Non use cases? More performant system than HBase and Accumulo since data is not stored on HDFS Built-in WAN replication

Apache Crunch Java framework for writing, testing, and running MapReduce pipelines with a simple API Same code executes as a local job, as a MapReduce job, or as a streaming Spark job Use cases? * *Not the real logo, but truly fantastic

Apache Kafka High-throughput distributed publish-subscribe message service Use cases? Have multiple brokers publish messages to hundreds of ‘topics’ of interest Clients subscribe to these messages and pull them off the queue for consumption

Azkaban Batch workflow job scheduler to run Hadoop jobs Use cases? Same as Oozie but apparently has a cool web UI

Review A lot of projects available to you for your grou project Think of a problem you are interested in, then choose the appropriate projects to solve it Keep in mind data ingest, storage, processing, and egress Feel free to explore and use other projects than the ones I have listed here Get permission if you plan on using it as part of your project quota

References All those logos are the property of their owners *.apache.org redis.io