Apache Oozie Workflow: How It Works and When to Use It

In Part 1, we looked at the background, basic concepts, and architecture of Oozie. This time, we will cover the workflow.

As we’ve seen in the previous part 1, Oozie’s workflow consists of the action node and the control node. The action node is the actual workflow task and the control node designates the frequency of actions performed and the order in which it is performed.

Oozie’s workflow action performs the job from the remote system, and when the job is complete, the remote system calls back Oozie and announces that the job is complete. Upon receiving the job completion, Oozie advances to the next job.

The action node includes the following items:

– Hadoop MapReduce job execution

– Move files to HDFS

– Run Pig program

– Run Java program

– Data import using Sqoop

– Various sub-workflows and custom Java code

The control node is as follows:

– Start node: Specify the start of workflow job

– End node: Specify the end of workflow job

– Decision node: The regulation of conditions for branch while performing workflow job

– Kill node: Stop progressing a job

– Error node: When a problem occurs in performing a job, this is defined with an error and notifies the contents about the error

The control node definition

 . . . ${wf:actionData(‘hiveSwitch’)[‘paramNum’] eq 1} . . . 

and tags

– Indicate the start and end of the configuration file.

tag

– Start node

– Display the start point of Oozie workflow by this configuration file.

– In the above, we set that the action node named startdatafeed run primarily.

tag

– End node

– When this stage is reached, this means that the process has been successfully completed. All workflow definition files must always be finished with this tag

tag

– Decision node

– Specify the conditions for determining the path of ongoing work. This is similar to the switch-case of any program, i.e. Word count workflow

The picture below is word count which is an example of Oozie workflow.

 ${jobTracker} ${nameNode} mapred.mapper.class org.myorg.WordCount.Map mapred.reducer.class org.myorg.WordCount.Reduce mapred.input.dir ${inputDir} mapred.output.dir ${outputDir} Something went wrong: ${wf:errorCode('wordcount')} font-size:11pt;