We’ll explain Apache Kafka and the first part deals with the description, operation principle, and features of Kafka.
What is Apache Kafka?Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system, but with a unique design. The lists of Kafka’s producers are an application, server, messages of various services, and etc. and the list of its customers are real time processing applications, NoSQL, Hadoop, DW, and etc.
Kafka is a solution to collect big data. The message publishing solutions such as Kafka are divided into producers publishing massages and consumers processing the feed of published messages, and also several message publishing solutions operates by synchronizing the producers and customers. Therefore, they need to be developed again in case of changing producers or customers, not reuse the existing codes. However, the solutions including Kafka ends by just connecting with producers or customers.
The operation principle of KafkaFirst let’s review some basic messaging terminology:
- Kafka maintains feeds of messages in categories called topics.
- We’ll call processes that publish messages to Kafka topic producers.
- We’ll call processes that subscribe to topics and process the feed of published messages consumers.
- Kafka is run as a cluster comprised of one or more servers each of which is called a broker.
So, at a high level, producers send messages over the network to the Kafka cluster which in turn serves them up to consumers like this:
The messages published by producers deliver through the broker of cluster, and customers can receive and process the forwarded messages immediately.
Various features of KafkaKafka uses TCP (Transmission Control Protocol) and supports many languages. The main features of Kafka are as follows:
- Nonvolatile messaging: It shouldn’t happen to an information loss in order to derive the value from big data. Apache Kafka enables you to produce the performance of constant time, even multi-terabyte message.
- High through-put: It was designed as to process millions of messages per second on the normal commodity hardware.
- Distribution: Apache Kafka supports a message partitioning to Kafka servers, and the partitioning offers the distributed consumption by the cluster including customer’s equipment and has an order only by the partition unit.
- Real time: A message published in the producer thread that is the required attribute of the system based on event such as Complex Event Processing can be directly showed on the customer thread.
Kafka uses Zookeeper to manage the high through-put and distribution from among the above features. By Zookeeper, it manages resources for processing each partitioned message and for the customers.
Another feature of Kafka is a message transfer according to the transfer method of message. Even if the transfer has failed with a general error during data transfer, the replication is carried out successfully in case a replica exists. If it doesn’t even have a replica, the data transfer is regarded as a complete failure.
Kafka transfers data by partitioning it when transferring data, so the fault tolerance is possible for the data. The data transmission structure of Kafka is as follows:
Above figure for example, the messages are input on three partitions, and if an error occurs on two partitions and a partition transfers successfully when transferring the contents of each partition, the messages are transferred without loss of error.
Currently the above advantages of Kafka are reflected in a large of part for lately updating Flume version. The particular solutions of Hadoop Ecosystem are not perfect. Even if the other solution has the same function, do not use only one because you need to suitably use the solution for each situation.
This concludes the simple description of Apache Kafka. The next part will deal with the methods that install and use Kafka.
[Reference] http://kafka.apache.org/documentation.html#introduction
Common questions
Is Apache Kafka a message queue?
Not in the traditional sense. A classic message queue removes a message once it is consumed, while Kafka keeps records in a partition for a configured retention period regardless of how many consumers have read them. This lets multiple independent applications re-read or replay the same stream.
Do I need ZooKeeper to run Kafka?
Not with current releases. Kafka depended on a ZooKeeper ensemble for cluster coordination for most of its history, but current major versions use a built-in KRaft mode instead, and recent releases have dropped ZooKeeper-based clusters for new deployments. If you are running or planning an older cluster, check the Apache Kafka documentation for that specific release to confirm which mode it uses and what migration options exist.
How does Kafka guarantee message order?
Order is guaranteed only within a single partition, not across an entire topic. Producers typically use a record key so that related records are routed to the same partition, which keeps their relative order intact.
What is the difference between Kafka and Hadoop?
Kafka is a real-time event streaming system for moving and buffering continuous data, while Hadoop is a storage and batch-processing ecosystem built around HDFS and tools like Hive and Spark for analyzing large, accumulated datasets. They are commonly used together, with Kafka feeding data into a Hadoop-based or cloud data lake.
How many partitions should a Kafka topic have?
There is no universal number; it depends on the throughput you need and how many consumers should be able to read the topic in parallel within a single consumer group. More partitions increase parallelism but also add coordination and resource overhead on the brokers, so the right count is workload-specific and should be measured rather than guessed.
Sources
About this page
Last reviewed October 30, 2026 by the Nine Bits editors. Read our editorial policy or report an error.