A message is simply an array of bytes as far as Kafka is concerned, so the data contained within it does not have a specific format or meaning to Kafka. A message can have an optional piece of metadata, which is referred to as a key. The key is also a byte array and, as with the message, has no specific meaning to Kafka. Keys are used when messages are to be written to partitions in a more controlled manner.
The simplest such scheme is to generate a consistent hash of the key and then select the partition number for that message by taking the result of the hash modulo the total number of partitions in the topic.
This ensures that messages with the same key are always written to the same partition (provided that the partition count does not change).
For efficiency, messages are written into Kafka in batches.
A batch is just a collection of messages, all of which are being produced to the same topic and partition .
Of course, this is a trade-off between latency and throughput: the larger the batches, the more messages that can be handled per unit of time, but the longer it takes an individual message to propagate. Batches are also typically compressed, providing more efficient data transfer and storage at the cost of some processing power.
While messages are opaque byte arrays to Kafka itself, it is recommended that additional structure, or schema, be imposed on the message content so that it can be easily understood. There are many options available for message schema, depending on your application’s individual needs. Simplistic systems, such as JavaScript Object Notation (JSON) and Extensible Markup Language (XML), are easy to use and human readable. However, they lack features such as robust type handling and compatibility between schema versions. Many Kafka developers favor the use of Apache Avro, which is a serialization framework originally developed for Hadoop. Avro provides a compact serialization format, schemas that are separate from the message payloads and that do not require code to be generated when they change, and strong data typing and schema evolution, with both backward and forward compatibility.
A consistent data format is important in Kafka, as it allows writing and reading messages to be decoupled. When these tasks are tightly coupled, applications that subscribe to messages must be updated to handle the new data format, in parallel with the old format. Only then can the applications that publish the messages be updated to utilize the new format. By using well-defined schemas and storing them in a common repository, the messages in Kafka can be understood without coordination.
Messages in Kafka are categorized into topics. The closest analogies for a topic are a database table or a folder in a filesystem. Topics are additionally broken down into a number of partitions. Going back to the “commit log” description, a partition is a single log. Messages are written to it in an append-only fashion and are read in order from beginning to end. Note that as a topic typically has multiple partitions, there is no guarantee of message ordering across the entire topic, just within a single partition. Figure 1-5 shows a topic with four partitions, with writes being appended to the end of each one. Partitions are also the way that Kafka provides redundancy and scalability. Each partition can be hosted on a different server, which means that a single topic can be scaled horizontally across multiple servers to provide performance far beyond the ability of a single server. Additionally, partitions can be replicated, such that different servers will store a copy of the same partition in case one server fails.
The term stream is often used when discussing data within systems like Kafka. Most often, a stream is considered to be a single topic of data, regardless of the number of partitions. This represents a single stream of data moving from the producers to the consumers. This way of referring to messages is most common when discussing stream processing, which is when frameworks—some of which are Kafka Streams, Apache Samza, and Storm—operate on the messages in real time. This method of operation can be compared to the way offline frameworks, namely Hadoop, are designed to work on bulk data at a later time.
Kafka clients are users of the system, and there are two basic types: producers and consumers. There are also advanced client APIs—Kafka Connect API for data integration and Kafka Streams for stream processing. The advanced clients use producers and consumers as building blocks and provide higher-level functionality on top.
In other publish/subscribe systems, these may be called publishers or writers. A message will be produced to a specific topic. By default, the producer will balance messages over all partitions of a topic evenly. In some cases, the producer will direct messages to specific partitions. This is typically done using the message key and a partitioner that will generate a hash of the key and map it to a specific partition. This ensures that all messages produced with a given key will get written to the same partition. The producer could also use a custom partitioner that follows other business rules for mapping messages to partitions.
In other publish/subscribe systems, these clients may be called subscribers or readers. The consumer subscribes to one or more topics and reads the messages in the order in which they were produced to each partition. The consumer keeps track of which messages it has already consumed by keeping track of the offset of messages. The offset—an integer value that continually increases—is another piece of metadata that Kafka adds to each message as it is produced. Each message in a given partition has a unique offset, and the following message has a greater offset (though not necessarily monotonically greater). By storing the next possible offset for each partition, typically in Kafka itself, a consumer can stop and restart without losing its place. Consumers work as part of a consumer group, which is one or more consumers that work together to consume a topic. The group ensures that each partition is only consumed by one member. In Figure 1-6, there are three consumers in a single group consuming a topic. Two of the consumers are working from one partition each, while the third consumer is working from two partitions. The mapping of a consumer to a partition is often called ownership of the partition by the consumer. In this way, consumers can horizontally scale to consume topics with a large number of messages. Additionally, if a single consumer fails, the remaining members of the group will reassign the partitions being consumed to take over for the missing member.
A single Kafka server is called a broker. The broker receives messages from producers, assigns offsets to them, and writes the messages to storage on disk. It also services consumers, responding to fetch requests for partitions and responding with the messages that have been published. Depending on the specific hardware and its performance characteristics, a single broker can easily handle thousands of partitions and millions of messages per second. Kafka brokers are designed to operate as part of a cluster. Within a cluster of brokers, one broker will also function as the cluster controller (elected automatically from the live members of the cluster). The controller is responsible for administrative operations, including assigning partitions to brokers and monitoring for broker failures. A partition is owned by a single broker in the cluster, and that broker is called the leader of the partition. A replicated partition (as seen in Figure 1-7) is assigned to additional brokers, called followers of the partition. Replication provides redundancy of messages in the partition, such that one of the followers can take over leadership if there is a broker failure. All producers must connect to the leader in order to publish messages, but consumers may fetch from either the leader or one of the followers.
A key feature of Apache Kafka is that of retention, which is the durable storage of messages for some period of time. Kafka brokers are configured with a default retention setting for topics, either retaining messages for some period of time (e.g., 7 days) or until the partition reaches a certain size in bytes (e.g., 1 GB). Once these limits are reached, messages are expired and deleted. In this way, the retention configuration defines a minimum amount of data available at any time. Individual topics can also be configured with their own retention settings so that messages are stored for only as long as they are useful. For example, a tracking topic might be retained for several days, whereas application metrics might be retained for only a few hours. Topics can also be configured as log compacted, which means that Kafka will retain only the last message produced with a specific key. This can be useful for changelog-type data, where only the last update is interesting.
As Kafka deployments grow, using multiple clusters can become beneficial. There are several reasons for having multiple Kafka clusters:
Segregation of different types of data
Isolation for security requirements
Support for multiple datacenters and disaster recovery
When multiple datacenters are used, messages often need to be copied between clusters. This allows applications in different locations to access the same data.
For example, if a user updates public information in their profile, that change should be available regardless of which datacenter handles the search request. Similarly, monitoring data from multiple locations can be collected into a central cluster where analysis and alerting systems run.
Kafka’s built-in replication mechanism only works within a single cluster and does not replicate data between different clusters.
To solve this problem, Kafka provides a tool called MirrorMaker. MirrorMaker is used to replicate messages between Kafka clusters. It works by acting as both a Kafka consumer and a Kafka producer connected through a queue:
It consumes messages from one Kafka cluster.
It produces those messages to another Kafka cluster.
MirrorMaker can create advanced data pipelines, such as collecting data from multiple local clusters into a central cluster and then replicating that data to other datacenters.
Although its design is simple, MirrorMaker enables powerful architectures for multi-cluster Kafka environments.
Kafka can handle multiple producers easily, whether they are producing messages to different topics or the same topic.
This makes Kafka useful for collecting data from many frontend systems and standardizing it into a consistent format.
For example, a website built with multiple microservices can use a single topic for page view events. All services can write their page view data to this topic using the same format.
Consumer applications can then read one unified stream of page views from all services instead of managing multiple topics, one for each application.
Kafka is also designed to support multiple consumers reading the same stream of messages without affecting each other.
This is different from traditional messaging systems, where once a message is consumed by one client, it is usually unavailable to other clients.
In Kafka, multiple consumers can independently read the same messages. They can also work together as part of a consumer group, where the messages are distributed among the group members, ensuring that each message is processed only once by the entire group.
Many applications participate in data processing environments. Some applications create data or introduce it into the system as inputs, while others produce outputs such as metrics, reports, and other data products.
Data flows in loops where some components read data, transform it using information from other sources, and send it back into the data infrastructure for further use. Different types of data have different characteristics in terms of content, size, and usage.
Apache Kafka acts as the circulatory system of the data ecosystem. It transfers messages between different parts of the infrastructure and provides a consistent interface for clients. When combined with a message schema system, producers and consumers no longer need tight coupling or direct connections. Components can be added or removed easily, and producers do not need to know who consumes the data or how many consumers exist.
Kafka was originally designed at LinkedIn for user activity tracking. Users interact with frontend applications, which generate messages about their actions, such as page views, clicks, or profile updates.
These messages are published to one or more topics and consumed by backend applications. These applications can generate reports, feed machine learning systems, update search results, or perform other operations required to improve user experience.
Kafka is also used for messaging systems where applications need to send notifications such as emails.
Applications can produce messages without worrying about message formatting or delivery details. A centralized application can consume these messages and handle them consistently by:
Formatting messages with a common style and structure
Combining multiple messages into a single notification
Applying user preferences for notification delivery
This approach avoids duplicating functionality across multiple applications and enables operations such as aggregation.
Kafka is suitable for collecting application metrics and logs.
Applications can publish metrics regularly to Kafka topics, and monitoring or alerting systems can consume them. These metrics can also be stored in offline systems such as Hadoop for long-term analysis, such as growth prediction.
Logs can be published in the same way and routed to systems like Elasticsearch or security analysis platforms. Kafka also allows destination systems to change without modifying frontend applications or aggregation logic.
Kafka is based on the concept of a commit log, allowing database changes to be published as streams of events.
Applications can monitor these streams to receive real-time updates. The changelog stream can also be used for:
Replicating database updates to remote systems
Combining changes from multiple applications into a single database view
Durable retention allows changelog data to be stored and replayed if consumer applications fail. Log-compacted topics can provide longer retention by keeping only the latest change for each key.
Stream processing is another major Kafka use case. Although many Kafka operations can be considered stream processing, the term usually refers to applications that provide functionality similar to Hadoop MapReduce.
Unlike Hadoop, which typically processes aggregated data over long periods such as hours or days, stream processing works on data in real time as messages are produced.
Stream processing frameworks allow developers to create applications that process Kafka messages by performing tasks such as:
Counting metrics
Partitioning messages for efficient processing
Transforming messages using data from multiple sources
Stream processing enables real-time data processing and analysis using Kafka.
Apache Kafka uses Apache ZooKeeper to store metadata about the Kafka cluster, as well as consumer client details, as shown in Figure 2-1. ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
KRaft Controller nodes form a Raft Quorum responsible for managing the Kafka Metadata Log. This log maintains all cluster metadata changes such as Topic, Partition, ISR, and Configuration.
Using Raft, controllers handleLeader Election and Consistency without requiring an external system. The metadata leader is called the Active Controller and processes all RPC requests from Brokers. Other controllers act as Follower, replicating data and serving as Hot Standby ready to take over in case the Active Controller fails.
Brokers track the latest metadata version using Offset, enabling faster metadata propagation and quicker Controller Failover.
KRaft requires a Majority of controllers to continue operating; for example, a 3-controller cluster tolerates one failure, while a 5-controller cluster tolerates two failures.
Controllers periodically store a Snapsho of metadata on disk, which is conceptually similar to Log Compaction, with the difference that information is read from memory.
The maximum number of Partitions in a cluster depends on two factors:
* The partition limit per Broker
* The partition limit for the entire cluster
KRaft is designed to manage a very large number of Partitions, but Kafka's scalability still increases with the addition of new Brokers; therefore, the overall cluster limit remains the ultimate scalability ceiling.
Additionally, the Quorum Controller significantly reduces Controller Failover time, enabling near-instantaneous controller recovery. Confluent's tests on a cluster with 2 million Partitions have shown that controlled shutdown time and recovery after failure have improved significantly compared to ZooKeeper.
In KRaft, new Topics can be assigned to Brokers that are offline as well. This capability allows performing Rolling Upgrades or reducing the number of Brokers (e.g., from 4 to 3) without encountering issues.
If an offline Broker should no longer be a member of the cluster, it must be removed (Unregistered) from the cluster metadata using the kafka-cluster unregister command. This command invokes the UnregisterBroker API and removes all information related to that Broker from the Cluster Metadata.
To view Brokers that may need removal, you can also use the kafka-cluster list-endpoints command.
Kafka uses Apache ZooKeeper to maintain the list of brokers that belong to a cluster. Each broker has a unique broker ID, configured manually or generated automatically. When a broker starts, it registers itself in ZooKeeper by creating an ephemeral node under /brokers/ids.
Kafka brokers, the controller, and other Kafka tools monitor /brokers/ids to detect when brokers join or leave the cluster.
If another broker starts with the same broker ID, registration fails because that ID already exists in ZooKeeper.
If a broker loses its connection to ZooKeeper (due to shutdown, network partition, or a long garbage collection pause), its ephemeral node is automatically removed. All components watching ZooKeeper are notified that the broker has left the cluster.
Although the ZooKeeper node disappears, the broker ID is still stored in Kafka metadata, such as the replica assignments of topics. Therefore, if a new broker starts with the same broker ID, it automatically replaces the old broker and takes ownership of the same partitions and replicas.
The controller is a Kafka broker responsible for partition leader election in addition to normal broker tasks. The first broker that starts becomes the controller by creating the /controller ephemeral node in ZooKeeper. Other brokers detect that the node already exists, watch it for changes, and ensure that only one controller exists at a time.
If the controller stops or loses its ZooKeeper connection (e.g., shutdown, network partition, or missing heartbeats for longer than zookeeper.session.timeout.ms), its ephemeral node is removed. Other brokers detect this through their ZooKeeper watch and compete to become the new controller. The first broker to create the /controller node becomes the new controller, while the others continue watching it.
Each new controller receives a higher controller epoch. Brokers track the current epoch and ignore messages from controllers with an older epoch. This prevents zombie controllers—controllers that resume after a long GC pause—from making outdated cluster changes (zombie fencing).
When a controller starts, it first loads the latest replica state from ZooKeeper before managing metadata or performing leader elections. This loading process uses asynchronous, pipelined ZooKeeper requests, but in clusters with many partitions it can still take several seconds.
When the controller detects that a broker has left the cluster (through ZooKeeper notifications or a ControlledShutdownRequest), it:
Identifies all partitions whose leader was on the failed broker.
Selects a new leader (typically the next replica in the replica list).
Persists the new leader information to ZooKeeper.
Sends LeaderAndISR requests to brokers hosting replicas so new leaders start serving clients and followers begin replicating from the new leader.
Sends UpdateMetadata requests to all brokers so they update their MetadataCache with the latest leader information.
When a broker rejoins the cluster, its replicas initially become followers and must catch up with the current leader before they are eligible to become leaders.
In summary, Kafka uses ZooKeeper ephemeral nodes to elect a single controller and detect broker membership changes. The controller manages leader elections, updates cluster metadata, and uses the controller epoch to prevent split-brain and zombie-controller scenarios.
Starting in 2019, the Apache Kafka community began working on KRaft, a new controller architecture based on the Raft consensus algorithm to replace the existing ZooKeeper-based controller. KRaft was introduced as a preview feature in Kafka 2.8, and Kafka 3.0 included the first production version. This allowed Kafka clusters to run either with the traditional ZooKeeper-based controller or the new KRaft controller.
The main reason for replacing the existing controller was scalability. Although Kafka’s controller had gone through several improvements, the ZooKeeper-based design could not scale well with the increasing number of partitions and brokers. Several problems motivated this change:
Metadata changes were written synchronously to ZooKeeper but propagated asynchronously to brokers. Also, brokers received updates from ZooKeeper asynchronously, which could create situations where metadata became inconsistent between ZooKeeper, the controller, and brokers. These inconsistencies were difficult to detect and troubleshoot.
Another major issue was controller restart time. Whenever the controller restarted, it had to reload all metadata about brokers, partitions, and replicas from ZooKeeper and then send this information to all brokers. As Kafka clusters grew, this process became a significant performance bottleneck.
The metadata management architecture was also complicated because some operations were handled by the controller, some by brokers, and some directly through ZooKeeper. In addition, ZooKeeper is itself a distributed system, meaning Kafka operators had to manage and understand two separate distributed systems instead of only Kafka.
In the existing architecture, ZooKeeper had two main responsibilities: electing the Kafka controller and storing cluster metadata such as brokers, topics, partitions, replicas, and configurations. The controller was responsible for managing this metadata, including leader election, topic creation and deletion, and replica reassignment. All these responsibilities needed to be redesigned in the new controller architecture.
The main idea behind KRaft is based on Kafka’s existing log-based design. Kafka already represents user data as a stream of events, where consumers can rebuild the latest state by replaying events from the log. This approach provides a clear ordering of changes and ensures that all consumers follow the same timeline. KRaft applies the same idea to Kafka’s internal metadata management.
In the KRaft architecture, controller nodes form a Raft quorum that manages a metadata event log. This log contains every change made to cluster metadata, including topics, partitions, replicas, ISRs, configurations, and other information previously stored in ZooKeeper.
Using the Raft algorithm, controller nodes elect a leader among themselves without depending on an external system. The leader of the metadata log is called the active controller. It handles requests from brokers, while follower controllers replicate the metadata and act as hot standbys. Because all controllers maintain the latest metadata state, failover does not require a long process of reloading and distributing metadata.
Instead of the controller pushing metadata updates to brokers, brokers fetch updates from the active controller using the new MetadataFetch API. Brokers keep track of the offset of the latest metadata change they received and only request newer updates. Brokers also persist metadata locally on disk, allowing them to restart quickly even in clusters with millions of partitions.
In KRaft, brokers register with the controller quorum and remain registered until an administrator removes them. If a broker shuts down, it becomes offline but remains known to the cluster. Brokers that are running but have outdated metadata are fenced, meaning they cannot serve client requests. This prevents situations where clients send data to a broker that is no longer the leader but is unaware of the leadership change.
During migration to KRaft, all operations that previously required clients or brokers to communicate directly with ZooKeeper are redirected through the controller. This allows Kafka to replace the ZooKeeper-based controller without requiring changes to brokers.
The overall design of KRaft is described in KIP-500. The adaptation of the Raft protocol for Kafka is explained in KIP-595, and the detailed design of the controller quorum, configuration options, and new metadata management tools are covered in KIP-631.
Replication is a core part of Kafka’s architecture. Kafka is often described as a distributed, partitioned, replicated commit log service because replication is responsible for providing availability and durability when brokers fail.
Kafka data is organized into topics. Each topic is divided into partitions, and each partition can have multiple replicas stored across different brokers. A broker usually stores hundreds or thousands of replicas from different topics and partitions.
There are two main types of replicas:
Leader replica
Each partition has one replica that acts as the leader. All produce requests are handled by the leader to maintain consistency. Consumers can read data from either the leader replica or follower replicas.
Follower replica
All replicas that are not leaders are called followers. By default, followers do not serve client requests. Their main responsibility is to replicate messages from the leader and remain up-to-date. If the leader fails, one of the follower replicas can be promoted to become the new leader.
The ability to consume messages directly from follower replicas was introduced in KIP-392. The main purpose of this feature is to reduce network costs by allowing consumers to read from the nearest in-sync replica instead of always reading from the leader.
To enable this feature, consumers need to configure client.rack to identify their location. Brokers need to configure replica.selector.class. By default, Kafka uses LeaderSelector, which always selects the leader replica, but it can be changed to RackAwareReplicaSelector, which selects a replica located on a broker with a matching rack ID. Kafka also allows custom replica selection logic by implementing the ReplicaSelector interface.
The replication protocol was extended to ensure that only committed messages are available when consuming from follower replicas. This keeps the same reliability guarantees as consuming from the leader.
To achieve this, the leader sends the current high-water mark (the latest committed offset) to followers. However, propagating the high-water mark introduces a small delay, meaning messages may become available for consumption from the leader slightly earlier than from followers. Therefore, reading from followers does not always reduce consumer latency.
The leader is responsible for tracking which follower replicas are synchronized with it. Followers replicate messages continuously, but they can fall behind because of issues such as network congestion or broker failures.
Followers stay synchronized by sending Fetch requests to the leader, similar to the requests consumers use to retrieve messages. These requests include the offset of the next message the follower wants to receive. Since requests are always ordered, the leader can determine how many messages each follower has received and how far behind each replica is.
If a follower has not requested messages for more than 10 seconds, or if it has requested messages but has not caught up with the latest messages within that time, it is considered out of sync. A replica that falls behind cannot become the leader during a failure because it may not contain all messages.
Replicas that continuously receive the latest messages are called in-sync replicas (ISR). Only replicas in the ISR set are eligible to become partition leaders when the current leader fails.
The maximum allowed time that a follower can be inactive or behind is controlled by the replica.lag.time.max.ms configuration parameter. This setting affects client behavior and data availability during leader elections.
Besides the current leader, each partition also has a preferred leader. The preferred leader is the replica that was originally selected as the leader when the topic was created.
Kafka tries to distribute leaders evenly across brokers when partitions are created. If the preferred leader is active for all partitions, the workload should remain balanced between brokers.
By default, Kafka enables auto.leader.rebalance.enable=true. This feature checks whether the preferred leader is not currently the leader but is still in sync. If so, Kafka triggers a leader election to move leadership back to the preferred leader.
The easiest way to identify the preferred leader is by checking the replica list of a partition. The first replica in the list is always the preferred leader, regardless of which replica is currently the leader.
This remains true even if replicas have been reassigned to different brokers using the replica reassignment tool. When manually reassigning replicas, the first replica specified becomes the preferred replica. Therefore, preferred replicas should be distributed carefully across brokers to prevent some brokers from handling too many leaders while others remain underutilized.
A Kafka broker mainly processes requests sent by clients, partition replicas, and the controller. Kafka uses a binary TCP protocol that defines request and response formats, including error handling. Kafka clients (Java, C, Python, Go, and others) communicate with brokers using this protocol.
Clients always initiate connections and send requests, while brokers process and respond to them. Requests from the same client are handled in the order they are received, which allows Kafka to provide message ordering guarantees.
Each request contains a standard header including the request type (API key), request version, correlation ID for tracking and troubleshooting, and client ID for identifying the application.
For each broker port, an acceptor thread accepts connections and passes them to network threads (processor threads). Network threads receive requests, place them in the request queue, and send responses back to clients. Some responses may be delayed and are stored in a purgatory until they can be completed.
Request handler threads (I/O threads) take requests from the queue and process them. The most common request types are:
Produce requests: Sent by producers to write messages to Kafka.
Fetch requests: Sent by consumers and follower replicas to read messages.
Admin requests: Used for administrative operations such as creating or deleting topics.
Produce and fetch requests must always be sent to the leader replica of a partition. If a request reaches a broker that is not the leader, Kafka returns a “Not a Leader for Partition” error. Kafka clients are responsible for finding the correct broker.
Clients discover partition leaders by sending metadata requests. The response contains information about topics, partitions, replicas, and current leaders. Since all brokers have metadata caches, metadata requests can be sent to any broker.
Clients cache metadata and use it to route produce and fetch requests to the correct brokers. Because metadata can change, clients periodically refresh it using the metadata.max.age.ms configuration. They also refresh metadata immediately after receiving a “Not a Leader” error, which indicates that their metadata is outdated.
When a producer sends a message to Kafka, the acks configuration determines how many brokers must confirm receiving the message before the write is considered successful. Producers can use different levels of acknowledgment:
acks=0: The producer considers the message successful immediately without waiting for a broker response.
acks=1: Only the leader replica needs to accept the message.
acks=all: All in-sync replicas (ISR) must acknowledge the message.
When the broker containing the leader replica receives a produce request, it performs several validations:
Checks whether the producer has permission to write to the topic.
Verifies that the acks value is valid.
If acks=all, checks whether enough in-sync replicas exist to safely store the message.
After validation, the broker writes the message to local storage. On Linux, data is usually written to the filesystem cache, and Kafka does not wait for it to be physically written to disk. Kafka relies mainly on replication to provide durability.
After writing the message to the leader replica, Kafka checks the acks setting. If acks=0 or acks=1, the broker responds immediately. If acks=all, the request is stored in a buffer called purgatory until follower replicas replicate the message. Once replication is confirmed, the broker sends a response to the producer.
Fetch requests are used by consumers and follower replicas to read messages from Kafka. The client sends a request containing the topics, partitions, and offsets it wants to read, for example requesting messages starting from a specific offset in a partition.
Clients also specify a maximum amount of data the broker can return. This prevents brokers from sending very large responses that could cause clients to run out of memory.
Fetch requests must be sent to the leader replicas of the requested partitions. Clients use metadata requests to find the correct brokers.
When the leader receives a fetch request, it first validates the requested offset. If the offset does not exist or the requested data has already been deleted, the broker returns an error. Otherwise, the broker reads messages up to the requested limit and sends them to the client.
Kafka uses a zero-copy technique when sending messages. Instead of copying data through multiple memory buffers, Kafka sends data directly from the file or Linux filesystem cache to the network connection. This reduces memory overhead and improves performance.
Clients can also specify a minimum amount of data they want to receive. For example, a consumer can request that the broker waits until at least 10 KB of data is available before responding. This reduces unnecessary requests, CPU usage, and network overhead for low-traffic topics.
Consumers can also define a timeout. If the minimum amount of data is not available within the specified time, the broker returns whatever data is currently available.
Not all messages stored on the leader are immediately available to consumers. Kafka only allows consumers to read messages that have been replicated to all in-sync replicas. Messages that exist only on the leader are considered unsafe because if the leader fails before replication, those messages could be lost.
Waiting for replication prevents inconsistent behavior. For example, if a consumer reads a message and then the leader fails before the message is replicated, other consumers might never see that message. Therefore, Kafka waits until the message is safely replicated before allowing consumers to read it.
This also means that slow replication between brokers can increase consumer latency because messages must be replicated before they become visible to consumers. The allowed replication delay is controlled by replica.lag.time.max.ms
When consumers read from many partitions, sending the complete list of partitions and metadata with every fetch request creates unnecessary overhead because partition information usually changes rarely.
To reduce this overhead, Kafka provides a fetch session cache. Consumers can create a cached session that stores the partitions they consume from and their metadata. After creating a session, consumers can send incremental fetch requests instead of sending the full partition list every time.
The broker only returns metadata changes instead of sending all metadata repeatedly. This improves efficiency and reduces network traffic.
The fetch session cache has limited capacity. Kafka prioritizes follower replicas and consumers with many partitions, so some sessions may not be created or may be removed. In these cases, the broker returns an error, and the consumer automatically falls back to normal full fetch requests.