Big data systems
HDFS, Kafka, Spark and the rest of the stack, and how the pieces fit.
- How Apache Spark Works, and How It Fits with HDFS and HBaseApache Spark is a distributed engine that processes data in memory across a cluster of machines, built as a faster alternative to Hadoop's original MapReduce.
- HBase Functionality: Tables, Schema, and Cluster InternalsHBase is a distributed, column-oriented database built on HDFS, modeled on Google's Bigtable, for sparse datasets that need fast random reads and writes at scale.
- Apache Hive: How It Works and When to Use ItApache Hive is a data warehouse layer built on Hadoop that lets you query large, distributed datasets using SQL-like syntax called HiveQL.
- Hive and HiveQL: SQL on Hadoop ExplainedApache Hive is a data warehouse layer on top of Hadoop-compatible storage that lets you query large files with HiveQL, a SQL-like language.
- Apache Spark Explained: How It Works With HDFS, Sqoop, and Graph DataApache Spark is an open-source engine for processing large datasets across a cluster of machines, keeping intermediate data in memory instead of writing it to disk between every step.
- What Is Apache Oozie? Workflow Scheduling for HadoopApache Oozie is a server-based workflow scheduler for Hadoop jobs.
- Apache Oozie Workflow: How It Works and When to Use ItAn Oozie workflow is a directed acyclic graph of actions, written in XML, that controls the order in which Hadoop jobs such as MapReduce, Hive, Pig, or shell scripts run.
- What Is Sqoop and How Does It Move Data Between Hadoop and Relational Databases?Apache Sqoop is a command-line tool that bulk-transfers structured data between relational databases (or other JDBC sources) and Hadoop storage such as HDFS, Hive, or HBase.
- Understanding Apache Kafka: Architecture, Internals, and Trade-offsApache Kafka is a distributed event streaming platform that stores records as ordered, append-only logs and lets many producers and consumers read and write that data independently.
- What Is HBase and How Does It Work?HBase is an open-source, column-oriented database that runs on top of HDFS, modeled after Google's Bigtable, built for fast random read and write access to tables with billions of rows.
- Understanding HDFS: Architecture, Data Flow, and Trade-offsHDFS (Hadoop Distributed File System) is a distributed, fault-tolerant file system that splits large files into blocks and spreads replicated copies of those blocks across many machines.
- What Is MapReduce and How Does It Work?MapReduce is a programming model that splits data processing into a map step, which transforms records into key-value pairs, and a reduce step, which aggregates values sharing the same key.
- Apache YARN: How Hadoop's Resource Manager WorksYARN, short for Yet Another Resource Negotiator, is the resource management and scheduling layer of Apache Hadoop.
- Apache ZooKeeper Explained: Coordination for Distributed SystemsApache ZooKeeper is a coordination service that distributed systems use to agree on shared state, such as configuration, service membership, and which node is currently the leader.
- About Hive 3 – Hive 2.0Previous blog Hive 1: http://bitnine.net/blog-computing/about-hive-1-preview/ Hive 2: http://bitnine.net/blog-computing/about-hive-2-hiveql/ The latest blog, we looked into a feature, structure, operator and example of actual query for Hive version 1.x.