Yarn

YARN (Hadoop v2) Difficulties with the original Hadoop implementation lead the development of a successor to Hadoop called YARN (Yet Another Resource Negotiator). As the name might indicate, YARN is meant to handle resource management. https://hadoop.apache.org/docs/r2.7.1/hadoop-yarn/hadoop-yarn-site/YARN.html http://www.iowapark.com/event/yarn-therapy-for-charity/

Problems with Hadoop v1 As cluster sizes and the number of users grew, the Job Tracker became a bottleneck. The static allocation of resources to map/reduce functions lead to poor utilization. HDFS was the only storage system that many enterprise applications could use, leading to the creation of jobs not suited for MapReduce. On large clusters, upgrading the version of Hadoop running on each machine became problematic.

YARN Components ApplicationMaster: Responsible for managing the work needed to be done. MapReduce ApplicationMaster is one example NodeManager: Each worker node has a manager responsible for gathering the required resources. ResourceManager: Managers the NodeManagers and schedules resources with the ApplicationMaster. The big change from Hadoop v1 is the separation of the Resource Management from the Application Management. https://yarn-book.com/

Benefits of YARN A rich diversity of data services, each with its own programming model (not all MapReduce). Application Masters can negotiate for resources in patterns optimal for them (duration and size). Per node Resource Managers allow for nodes to be utilized in a dynamic manner (used when needed). ResourceManager does just one thing (manage resources) so it can scale to tens of thousands of nodes. With ApplicationMaster managing jobs, you can have multiple versions of an application, which doesn't require a global cluster update (and the need to halt the cluster).

What do you call a YARN script? 1. Knot 2. CatToy 3. Fabric 4. Sweater

Frameworks Built On YARN Apache Tez Meant to handle datasets in the petabyte range. Workflows are modeled as a directed acyclic graph (DAG) where vertices are tasks and edges are interoperational dependencies or flows of data. This model is a better fit for many jobs, as such, Pig and Hive can run atop Tez for improved performance. Apache Giraph Large graph processing system (similar to Neo4j) Hoya: Hbase on YARN Just what it sounds like https://svn.apache.org/repos/asf/giraph/logo/

The Cloud Cloud - Making managing the servers someone else's problem. Lots of benefits: Much easier to "scale out" add nodes to distributed database Interchangeable Resources - servers can fail and be replaced, virtualization makes it easy Peak Usage - Additional resources can be requested when needed to deal with surges Large Volume and Velocity - Cloud has huge storage and very rapid data transfer rates Low initial investment - Trading capital for operational costs Globally distributed - You can put a server close to your clients http://www.pcmag.com/article2/0,2817,2372163,00.asp