How Apache Spark Works, and How It Fits with HDFS and HBase

RDD (Resilient Distributed Datasets) was designed based on a thesis which was published at NSDI (USENIX Symposium on Networked Systems Design & Implementation) under the sponsorship of USENIX in 2012. In the case of frequently recycling data among applications such as ‘Iterative machine learning’ or ‘multiple ad-hoc queries’, Data has been utilized only by storage just like the distributed file system. ‘Disc I/O’ and ‘setting up in series’ are much ineffective for the system. To solve this problem, the concept of RDD appeared for generally applicable recycling-Data due to ‘Pregel & Haloop’ are limited to specific conditions.

Each word means by below

• Resilient : Data is able to be restored as RDD stores Lineage when Data has lost on memory or error has occurred in a node.
• Distributed : Data is stored to several nodes in a cluster.
• Dataset : The first data can be produced from files or by programming.

RDD which Spark consists of is a group of unchanged data stored throughout several distributed nodes and so it’s the only way to change data to create new RDD.

Transformation

• The attribute of RDD data is unchangeable, so existing RDD should be filtered out or converted to make new RDD which comes with new Data. These operations above are called ‘Transformations’

Actions

• The operation is to put the result on storage or to get desired data.

There are queries for each operation as referred to hereinafter.

RDD data is unable to be used or converted before the process for the ‘Actions’ as referred above. It’s hereby called ‘Lazy Execution’.

Here is an example summarized as follows ; After reading American poem of ‘A Thousand Winds’, capitalize all the letters and get all the sentences included in ‘I am’

1. Read and save “a_thousand_winds.txt’ stored in HDFS

2. Make new RDD converted all the letters in the poem into capitalization.

3. Make another RDD consist of sentences included in “I AM”

4. the result comes out

While processing ④ / “collect()”, ‘ ① ~ ③ ‘ proceed in numerical order.

Scala Spark Shell regarding REPL(Read-eval-print loop) was used for the example above.

For the next time, ‘Persistence of RDD’ and ‘programming of Spark’ will be mentioned about.

[Reference] Matei Zaharia(2012), “Resilient Distributed Datasets :A Fault-Tolerant Abstraction for In-Memory Cluster Computing”