BigData Tools Seyyed mohammad Razavi

Outline  Introduction  Hbase  Cassandra  Spark  Acumulo  Blur  MongoDB  Hive  Giraph  Pig

Introduction Hadoop consists of three primary resources:  The Hadoop Distributed File System (HDFS)  The MapReduce Programming platform  The Hadoop ecosystem, a collection of tools that use of sit beside MapReduce and HDFS

FULLY INTEGERATED Hbase  NoSQL database system included in the standard Hadoop distributions.  It is a key-value store.  Rows are defined by a key, and have associated with them a number of columns where the associated values are stored.  The only data type is the byte string.

Hbase  Hbase is accessed via java code, but APIs exist for using Hbase with pig, Thrift, Jython, …  Hbase is often used for applications that may require sparse rows. That is, each row may use only a few of the defined columns.  Unlike traditional HDFS applications, it permits random access to rows, rather than sequential searches.

Cassandra  Distributed key-value database designed with simplicity and scalability in mind. API COMPATIBLE

Cassandra VS HBase  Cassandra is an all-inclusive system, which means it does not require Hadoop environment or any other big data tools.  Cassandra is completely master less: it operates as a peer-to-peer system. This make it easier to configure and highly resilient.

Cassandra  Oftentimes you may need to simply organize some of your big data for easy retrieval.

Cassandra  You need to create a keyspace. Keyspaces are similar to schemas in traditional relational databases.

Cassandra

Spark  MapReduce has proven to be suboptimal for applications like graph analysis that require iterative processing and data sharing.  Spark is designed to provide a more flexible model that supports many of the multipass applications that falter in MapReduce. API COMPATIBLE

Spark  Spark is a fast and general engine for large- scale data processing.  Spark has an advanced DAG execution engine that supports cyclic data flow and in-memory computing

Spark  Unlike Pig and Hive, Spark is not a tool for making MapReduce easier to use.  It is a complete replacement for MapReduce that includes its own work execution engine.

Spark  Spark Operates with three core ideas:  Resilient Distributed Dataset (RDD)  Transformation  Action

Spark VS Hadoop  Spark is a fast and general processing engine compatible with Hadoop data.  It is designed to perform both batch processing (similar to MapReduce) and new workloads like streaming, interactive queries, and machine learning.

Spark VS Hadoop  It has been used to sort 100 TB of data 3X faster than Hadoop MapReduce on 1/10th of the machines.

Accumulo  Control which users can see which cells in your data.  U.S. National Security Agency (NSA) developed Accumulo and then donated the code to the Apache foundation. FULLY INTEGERATED

Accumulo VS HBase  Accumulo improves on that model with its focus on security and cell-based access control.  With security labels and Boolean expressions.

Blur  Tool for indexing and searching text with Hadoop.  Because it has Lucene at its core, it has many useful features, including fuzzy matching, wildcard searches, and paged results.  It allows you to search through unstructured data in a way that would otherwise be very difficult. FULLY INTEGERATED

mongoDB  If you have large number of JSON documents in your Hadoop cluster and need some data management tool to effectively use them consider mongoDB API COMPATIBLE

mongoDB  In relational databases, you have tables and rows. In MongoDB, the equivalent of a row is a JSON document, and the analog to a table is a collection, a set of JSON documents.

Hive  The goal of Hive is to allow SQL access to data in the HDFS.  The Apache Hive data-warehouse software facilitates querying and managing large datasets residing in HDFS.  Hive defines a simple SQL-like query language, called HQL, that enables users familiar with SQL to query the data. FULLY INTEGERATED

Hive  Queries written in HQL are converted into MapReduce code by Hive and executed by Hadoop.

Giraph  Graph Database  Graphs useful in describing relationships between entities FULLY INTEGERATED

Giraph  Apache Giraph is derived from a Google project called Pregel and has been used by Facebook to build and analyze a graph with a trillion nodes, admittedly on a very large Hadoop cluster.  It is built using a technology called Bulk Synchronous Parallel (BSP).

Pig  A data flow language in which datasets are read in and transformed to other data sets.  Pig is so called because “pigs eat everything,” meaning that Pig can accommodate many different forms of input, but is frequently used for transforming text datasets. FULLY INTEGERATED

Pig  Pig is an admirable extract, transform, and load (ETL) tool.  There is a library of shared Pig routines available in the Piggy Bank.

CORE TECHNOLOGIES Hadoop Distributed File System (HDFS) MapReduce YARNSpark DATABASE AND DATA MANAGEMENT CassandraHBase AccumuloMemcached BlurSolr MongoDBHive Spark SQL (formerly Shark) Giraph SERIALIZATION AvroJSON Protocol Buffers (protobuf) Parquet

MANAGEMENT AND MONITORING AmbariHcatalog NAGIOSPuppet ChefZOOKEEPER OozieGanglia ANALYTIC HELPERS PigHadoop Streaming MahoutMLLib Hadoop Image Processing Interface (HIPI) Spatial Hadoop DATA TRANSFER SqoopFlume DistCpStorm SECURITY, ACCESS CONTROL, AND AUDITING SentryKerberos Knox CLOUD COMPUTING AND VIRTUALIZATION SerengetiDocker Whirr

?