From now on, we will introduce Sqoop and the methods transferred data of RDBMS to Hadoop by using Sqoop. And this topic is divided into three parts. Well, first of all, let’s look at the introduction, comparison between Sqoop1 and Sqoop2, and the service environment.
What is Sqoop?
Sqoop stands for “SQL to Hadoop” and Apache Sqoop is an open-source tool designed for efficiently transferring bulk data between Apache Hadoop and structured data stores such as relational databases.
Data processed by Scoop can be used for MapReduce or Hive. After launching the first version in 2009, it became Apache Top Level Project and is developing constantly. Now Sqoop has two versions with Sqoop1 of client type and Sqoop2 added Server side to Sqoo1.
Compare between Sqoop1 and Sqoop2The biggest difference is an addition for Server side and the convenience of Apache Oozie and the integration by using HTTP REST.
| Sqoop1 | Sqoop2 |
| ■ Client-side Install ■ Connector must be installed at Local ■ JDBC Driver needs the install each conneted Local ■ Offer CLI(Command Line Interface) | ■ Client-side Install ■ Possible to install and connect on only one server that needs Connector ■ i.e., JDBC Driver installs one place only ■ Possible to connect through WEB and REST API besides CLI conneting ■ Easy to combine by using Apache Oozie and REST API as Workflow Manager |
The Service Environment
- If store a log in RDBMS such as Oracle, MySQL because it doesn’t have the extra log acquisition system and data storage, that is, need to transfer the overlarge data to storage of the same distribute environment and to analyze it because of cost and time involved.
- Not only a log, but meta-data is mostly stored in RDBMS. If transfer meta-data to Hadoop, Hive, etc.
- Otherwise, if transfer the result of analysis by Hadoop, Hive, etc. to the remote RDBMS, not API.
[Reference] http://www.dbguide.net/knowledge.db?cmd=specialist_view&boardUid=176552&boardConfigUid=108&boardStep=0&categoryUid =
Common questions
Is Sqoop still actively maintained?
No. The project was retired and moved to the Apache Attic, so there are no new releases or active maintainers. Existing installations keep working, but check the project's current status before relying on it for a new pipeline.
What is the difference between Sqoop import and export?
Import pulls data from a relational database into Hadoop (HDFS, Hive, or HBase). Export pushes data from HDFS back into a relational database table. Both run as map-only MapReduce jobs.
Does Sqoop need a running Hadoop cluster?
Yes. Sqoop submits its work as a MapReduce job, so it needs a working Hadoop cluster with YARN available to run on, plus a JDBC driver for whatever database it connects to.
Why does Sqoop ask for a --split-by column?
Sqoop parallelizes an import by dividing the table into row ranges across mappers. It needs a column, ideally an indexed numeric primary key, to calculate those ranges. Without one, the import effectively runs on a single mapper.
Can Sqoop be used for real-time data replication?
Not well. Sqoop is a batch tool meant to run on demand or on a schedule, not a continuous change-data-capture pipeline. For near-real-time replication, a tool built for CDC, such as Debezium, is a better fit.
Sources
About this page
Last reviewed October 30, 2026 by the Nine Bits editors. Read our editorial policy or report an error.