Apache YARN: How Hadoop's Resource Manager Works

Now we will go forth explaining YARN with three parts. This posting is the first time that describes an overview and features of YARN.

Overview

YARN is the abbreviation for Yet Another Resource Negotiator and is a resource manager of Hadoop and a module newly added in Hadoop 2.0 version. YARN is the framework that works the scheduling with Cluster Resources (CPU, memory, disk, etc.) for an operation of the data processing, so it is a project started to overcome defect of MapReduce that is the data processing framework of existing Hadoop.

For example, MapReduce creates bottlenecks in the process of expanding over 4,000 nodes of very large cluster. The various applications such as MapReduce, Hive, Impala, Tajo, and Spark perform the operation for being allocated the resource from YARN.

The below is the figures comparing Hadoop 1.x and 2.x version.

The job tracker of Hadoop 1.0 MapReduce performs two functions. One is the management for the resource of total cluster, and another is to manage until jobs succeed by performing them. Likewise, the resource manager and application master are separated in YARN, so there is the resource manager for each YARN cluster and there is the node manager for each server in the cluster.

Each job is operated on the distributed environment, and a server is performed as a master. Therefore, they receive the necessary resource from the resource manager and manage tasks in the job.

The tasks are run in the module called ‘Container’, and at Hadoop 1.0 MapReduce, it is a task in the task tracker. Container as the universal module can very simply execute shell script and it regularly reports the condition of running task to the application master. (i.e. a task tracker report the situation to the job tracker at Hadoop 1.0 MapReduce.)

The mapper shown at Hadoop 1.0 MapReduce and the number of slots of Reducer don’t exist since the node managers of each server execute the tasks and manage the necessary resources. The problem of Hadoop 1.0 MapReduce is to set the mapper and reducer separately – the mapper run but reducer doesn’t and the reverse is too.

In other words, the utilization of aggregate cluster is extremely lower. According to the resource condition of aggregate cluster and the request of required resource, it is determined the number of executed containers, not exist slots separately in containers.

In YARN, the node managers run tasks as needed. For example, the default value – a parameter called yarn.nodemanager.resource-md determines the memory allocated in node manager – is 8,192MB.

Hadoop 1.0 Job Tracker is a node and has limits because it manages the resources of whole clusters and running jobs. For example, there are some problems that don’t implement more than 4,000 clusters or more than 40,000 jobs at the same time. However, YARN can solve the problems by dividing the resource manager and application master.

It is possible to develop new distributed processing environment by using YARN API, but never easy. The distributed processing environments operate as an application on YARN, but not an installation service. For example, it operates as an application and runs wordcount on that. And when the run ends, MapReduce also disappear on YARN.

YARN has a self-environment settings file (yarn-site.xml), and the applications operated on it have the file. In case of MapReduce, mapred-site.xml is just the environment settings file as shown in Hadoop 1.0.