Producer Overview
To produce messages to Kafka, the application first creates a ProducerRecord containing at least the target topic and the message value. Optionally, it can also include a key, partition, timestamp, and headers. Before transmission, the Producer serializes the key and value into byte arrays. If no partition is specified, the partitioner selects one, typically based on the message key. The record is then added to a batch of messages for the same topic and partition, which is sent to the appropriate broker by a background thread. After processing the request, the broker returns either a RecordMetadata object containing the topic, partition, and offset if the write succeeds, or an error if it fails. In the event of a failure, the Producer may automatically retry sending the message before ultimately reporting an exception if all retry attempts are exhausted.
The Kafka Producer is the component responsible for producing and sending messages to a Kafka Cluster. Before sending any messages, a Producer must be created with a set of configuration properties that enable it to connect to Kafka brokers and publish data to the desired topics. The most essential configuration properties required to create a Producer are bootstrap.servers, key.serializer, and value.serializer.
The bootstrap.servers property specifies a list of brokers that the Producer uses to establish its initial connection to the Kafka cluster. This list does not need to include every broker in the cluster because, after the initial connection is established, the Producer retrieves complete cluster metadata, including information about all available brokers. However, it is recommended to specify at least two brokers so that if one broker becomes unavailable, the Producer can still connect to the cluster through another broker.
Since Kafka accepts messages only as byte arrays, the Producer must serialize both the key and the value of each message before sending it. This is accomplished through the key.serializer and value.serializer configuration properties. The values of these properties must be the names of classes that implement the org.apache.kafka.common.serialization.Serializer interface. Kafka provides several built-in serializers, such as StringSerializer, IntegerSerializer, LongSerializer, and ByteArraySerializer, which are sufficient for most use cases, eliminating the need for custom serializer implementations. An important point to note is that even if the Producer sends only a value and does not use a key, the key.serializer configuration is still mandatory.
Once the configuration is complete, a Producer is created using the KafkaProducer class. The generic types specified for the key and value when creating the Producer must match the selected serializers. For example, if StringSerializer is used, both the key and value types should be String.
To send a message, a ProducerRecord object must first be created. This object contains information such as the topic name, the message key, and the message value. The record is then passed to the Producer using the send() method. Contrary to a common misconception, the send() method is inherently asynchronous. This means that after it is invoked, the message is first placed into the Producer's internal buffer and is subsequently transmitted to the broker by a separate thread. Therefore, calling send() alone does not guarantee that the message has already been delivered; it merely initiates the sending process.
The simplest message delivery approach is the Fire-and-Forget pattern. In this approach, the application simply invokes the send() method without checking the outcome of the operation. Under normal circumstances, thanks to Kafka's high reliability and the Producer's built-in retry mechanism, most messages are successfully delivered. However, if a non-retriable error or a timeout occurs, the message may be lost without the application being aware of the failure. Therefore, this approach is suitable only for scenarios where occasional message loss is acceptable.
The second approach is Synchronous Send. Since the send() method returns a Future<RecordMetadata> object, calling the get() method on this object causes the sending thread to block until a response is received from Kafka. If the operation succeeds, the returned RecordMetadata object provides information such as the topic, partition, and offset of the message. If an error occurs, an appropriate exception is thrown. Although this approach guarantees that the application knows whether the message was successfully delivered, waiting for the broker's response significantly reduces throughput and is therefore rarely used in production environments.
The third approach is Asynchronous Send with Callback, which is the most commonly used method in production systems. In this approach, a Callback object is passed along with the send() method. Once Kafka responds, the callback's onCompletion() method is invoked. If the message is successfully delivered, the Exception parameter is null; otherwise, it contains information about the failure, allowing the application to perform actions such as logging the error, implementing custom retry logic, or storing failed messages in monitoring systems.
An important aspect of callbacks is that they execute on the Producer's internal I/O thread. As a result, callback execution order is preserved for messages sent to the same partition. However, performing heavy or blocking operations inside a callback—such as database access, HTTP requests, or file I/O—can negatively impact the Producer's performance and delay the delivery of subsequent messages. Therefore, time-consuming tasks should be delegated to a separate thread or executor.
KafkaProducer classifies errors into two categories. The first category consists of Retriable Errors, which can be resolved by automatically resending the message. Examples include temporary broker connection failures or leader changes for a partition. In such cases, the Producer retries the operation automatically. Only after all configured retry attempts have been exhausted does the exception propagate back to the application. The second category consists of Non-Retriable Errors, which cannot be resolved through retries. An example is the Message size too large error. In these situations, the Producer immediately returns the exception without attempting any retries.
In addition to broker-related failures, the Producer may encounter exceptions before the message is even transmitted. Common examples include SerializationException when the message cannot be serialized, BufferExhaustedException or TimeoutException when the Producer's internal buffer is full, and InterruptException when the sending thread is interrupted.
Finally, one of the key characteristics of KafkaProducer is that it is thread-safe. A single Producer instance can be safely shared by multiple threads concurrently, eliminating the need to create a separate Producer for each thread. This design reduces resource consumption, improves performance, and enables more efficient utilization of the connections established with the Kafka cluster.
So far we’ve seen very few configuration parameters for the producers—just the mandatory bootstrap.servers URI and serializers.
The producer has a large number of configuration parameters documented in Apache Kafka documentation, and many have reasonable defaults, so there is no need to tune everything. However, some parameters significantly impact memory usage, performance, and reliability, and these are the most important ones.
client.id
client.id is a logical identifier for the client and the application it is used in. It is used by brokers for logging, metrics, and quota tracking.
A meaningful client.id greatly improves debugging and operational visibility, making it easier to identify which service is generating traffic or errors instead of relying on raw IP addresses.
acks
The acks parameter controls how many replicas must acknowledge a write before it is considered successful. It directly affects durability and reliability.
acks=0
The producer does not wait for any response. This gives maximum throughput but no guarantee of delivery. Messages can be lost without detection.
acks=1
The producer waits for the leader to acknowledge the message. This provides a balance between performance and reliability, but data can still be lost if the leader fails before replication.
acks=all
The producer waits until all in-sync replicas acknowledge the message. This provides the highest durability guarantee but increases latency.
TIP:
Lower ack values improve throughput but reduce reliability. However, end-to-end latency remains the same across all modes because Kafka only exposes messages to consumers after replication to all in-sync replicas is complete. Therefore, for end-to-end latency, there is no real trade-off—only for producer latency and durability.
Message Delivery Time
Message delivery time refers to how long a send() call takes to either succeed or fail. This includes waiting for buffer space, sending data, and receiving broker response.
Since Kafka 2.1, delivery time is split into two parts:
Time spent inside send() before it returns (blocking time)
Time from message batching until callback execution (success or failure)
If send() is synchronous, both phases are combined into one blocking period.
This parameter defines how long the producer can block when calling send() or requesting metadata.
It applies when:
The send buffer is full
Metadata is unavailable
If exceeded, a timeout exception is thrown.
This is the total time allowed for a message from the moment it is ready to be sent until final success or failure, including retries.
Must be greater than linger.ms and request.timeout.ms
If exceeded during retries, callback returns the last broker error
If exceeded before sending, callback returns a timeout error
TIP:
A practical approach is to set delivery.timeout.ms to a fixed upper bound (e.g., a few minutes) and allow retries to continue within that window. This avoids complex retry tuning and ensures Kafka retries automatically until success or timeout.
request.timeout.ms
This parameter controls how long the producer waits for a response from the broker for each individual request. It applies per request only and does not include retries or time spent waiting before sending.
If the broker does not respond within this time, the request is considered failed. At that point, the producer may either retry the request (if retries are enabled) or fail the operation and trigger a TimeoutException in the callback.
retry.backoff.ms
The retries parameter defines how many times the producer will retry sending a message when it receives a retryable (transient) error from the broker, such as missing leader or temporary unavailability of a partition.
Between each retry attempt, the producer waits for the duration defined by retry.backoff.ms (default is 100 ms). This prevents tight retry loops and gives the cluster time to recover.
However, modern Kafka usage generally discourages fine-tuning retries manually. Instead, it is recommended to rely on delivery.timeout.ms as the overall control mechanism for retry duration. The idea is to configure a total time window large enough to survive typical broker failures (for example, leader election delays), and let Kafka handle retries automatically within that window.
Not all errors are retriable. Some errors, such as “message too large,” are permanent and will immediately fail without retries. Because retries are handled internally by the producer, application-level retry logic is usually unnecessary and should only focus on non-retriable or exhausted-retry cases.
TIP:
Setting retries=0 completely disables retry behavior.
linger.ms
linger.ms controls how long the producer waits to accumulate additional messages before sending a batch.
By default, Kafka sends a batch immediately when a sender thread is available, even if the batch contains only one message. However, setting linger.ms to a value greater than zero allows the producer to wait briefly to collect more messages into the same batch before sending.
This slightly increases latency but significantly improves throughput because:
batching reduces per-message overhead
compression becomes more efficient
buffer.memory
This parameter defines the total memory available for buffering records waiting to be sent to the broker.
If the application produces messages faster than Kafka can send them, the buffer fills up. When this happens, calls to send() will block for up to max.block.ms while waiting for buffer space to free up. If space is still unavailable after this period, a timeout exception is thrown.
Unlike most producer errors, this timeout occurs directly in send() rather than in the returned Future, making it a blocking-level failure rather than an async callback failure.
compression.type
By default, Kafka sends messages uncompressed. This parameter enables compression before data is sent to brokers, using algorithms such as snappy, gzip, lz4, or zstd.
snappy: Designed by Google, offers good balance between CPU usage and compression speed; recommended when both performance and bandwidth matter.
gzip: Provides better compression ratios but uses more CPU and is slower; useful when network bandwidth is limited.
lz4 / zstd: Typically offer strong performance and good compression efficiency depending on workload.
Compression reduces network usage and storage cost, which are often major bottlenecks in Kafka systems.
batch.size
This parameter defines the maximum memory (in bytes) allocated for a batch of messages per partition.
When multiple records are sent to the same partition, they are grouped into batches. Once a batch is full, it is sent immediately. However, Kafka does not wait for batches to fill completely—partial batches or even single-message batches can still be sent.
A larger batch size does not introduce delay but increases memory usage and can improve throughput due to better batching and compression. A smaller batch size reduces memory usage but increases overhead due to more frequent sends.
max.in.flight.requests.per.connection
This controls how many request batches can be sent to the broker without waiting for responses.
Higher values improve throughput but increase memory usage and can introduce ordering risks when retries are enabled.
Kafka preserves message order within a partition, but with retries enabled and multiple in-flight requests, ordering can be broken if an earlier batch fails and is retried after a later batch has already been successfully written.
To balance performance and safety, Kafka recommends enabling:
enable.idempotence=true
This ensures:
Message ordering is preserved (even with retries)
No duplicate messages are introduced
Safe operation with up to multiple in-flight requests (typically up to 5)
Ordering Guarantees
Kafka guarantees ordering within a partition: messages are written and read in the same order they are sent.
However, ordering can break in certain cases when:
retries are enabled
max.in.flight.requests.per.connection is greater than 1
In such cases, a later batch may succeed before an earlier failed batch is retried, causing reordering.
The recommended solution is enabling idempotent producer mode, which preserves ordering and prevents duplicates while still allowing retries and multiple in-flight requests for performance.
max.request.size
This parameter defines the maximum size of a single produce request sent to the broker. It limits:
the maximum size of an individual message
the total batch size of messages in one request
For example, with a 1 MB limit:
one 1 MB message can be sent, or
1024 messages of 1 KB each can be batched
The broker also has its own limit (message.max.bytes). These should be aligned to avoid producing messages that will be rejected by the broker.
receive.buffer.bytes and send.buffer.bytes
These parameters control the size of the TCP socket buffers used for receiving and sending data between the Kafka client and brokers.
send.buffer.bytes: size of the TCP send buffer
receive.buffer.bytes: size of the TCP receive buffer
If set to -1, the operating system default values are used
Increasing these values is beneficial when clients communicate with brokers over high-latency or low-bandwidth networks, such as across different datacenters, because larger buffers help smooth network transmission and improve throughput.
enable.idempotence
Starting from Kafka 0.11, Kafka supports idempotent producers, which are a key part of achieving exactly-once semantics (though full exactly-once behavior is broader and more complex).
When a producer is configured for maximum reliability (for example, acks=all with retries enabled and a sufficiently large delivery.timeout.ms), Kafka guarantees that messages are delivered at least once. However, in failure scenarios, this can lead to duplicate messages.
For example, if a broker successfully writes a message and replicates it, but crashes before sending an acknowledgment, the producer will retry. The retry may land on a new leader that already has the message, resulting in a duplicate.
To solve this, setting enable.idempotence=true ensures that Kafka eliminates duplicates. The producer attaches a sequence number to each message, and brokers detect duplicate sequences and discard them. In this case, duplicates do not reach the log, and the producer receives a harmless DuplicateSequenceException.
Enabling idempotence imposes strict configuration constraints to ensure correctness:
acks must be set to all
retries must be greater than 0
max.in.flight.requests.per.connection must be ≤ 5
If these conditions are not met, Kafka will throw a ConfigException at startup.
Idempotent producer is a lightweight but very important mechanism because it allows Kafka to safely handle retries while preserving correctness, preventing duplicates without requiring application-level logic.
Every Kafka producer must be configured with serializers to convert Java objects into byte arrays before sending them to Kafka. Kafka provides built-in serializers for common data types such as String, Integer, and ByteArray; however, real-world applications typically exchange complex business objects. In these scenarios, developers must either implement a custom serializer or use a standard serialization framework such as Apache Avro, Protocol Buffers (Protobuf), or Apache Thrift. Among these options, Apache Avro is generally recommended because it simplifies schema management and provides excellent support for schema evolution.
A custom serializer is required when an application needs to serialize domain-specific objects that are not supported by Kafka's built-in serializers. A serializer implements Kafka's Serializer<T> interface and is responsible for converting an object into a sequence of bytes.
For example, a Customer object can be serialized by manually writing each field into a ByteBuffer. The serializer converts the customer identifier, the length of the customer's name, and finally the UTF-8 encoded name into a byte array that Kafka can transmit.
Once configured in the producer, the custom serializer allows applications to publish records using strongly typed objects such as:
ProducerRecord<String, Customer>
instead of manually converting objects before sending them.
Although custom serializers provide full control over the serialization process, they introduce several long-term maintenance challenges.
The primary issue is schema evolution. Any modification to the object structure—such as changing a field type from int to long or introducing a new field—can break compatibility between producers and consumers using different versions of the serializer.
Debugging these compatibility problems is also difficult because developers often have to inspect raw byte arrays to determine how objects were serialized.
In distributed environments, the problem becomes even more significant. If multiple teams produce the same type of data, they must all use identical serialization logic and coordinate serializer updates simultaneously. This creates operational complexity and increases the risk of incompatible deployments.
For these reasons, Kafka documentation recommends using established serialization frameworks such as Apache Avro, JSON, Protocol Buffers, or Thrift instead of implementing custom serializers.
Apache Avro is a language-independent serialization framework designed for efficient data exchange. Instead of relying on Java classes alone, Avro defines the structure of data using a JSON schema.
Data is typically serialized into a compact binary format, although JSON serialization is also supported. During serialization and deserialization, both the producer and consumer rely on the schema to correctly interpret the binary data.
One of Avro's greatest advantages is its support for schema evolution, making it particularly suitable for messaging systems such as Kafka.
Schema evolution allows applications to change the structure of their data without breaking existing producers or consumers.
For example, an initial Customer schema may contain the fields:
id
name
faxNumber
Later, the application may replace faxNumber with a new optional field named email.
Because Avro resolves differences between compatible schemas, older consumers can continue reading new messages, while newer consumers can still process older records.
If an older application attempts to read a message containing the new schema, the missing faxNumber simply returns null.
Likewise, if a newer application reads an older message, the missing email field also resolves to null.
This behavior enables gradual system upgrades without requiring all applications to be updated simultaneously or existing Kafka data to be rewritten.
Schema evolution is only possible when the new schema remains compatible with the previous version.
Avro defines explicit compatibility rules that determine whether schema changes are considered backward or forward compatible.
In addition, the consumer must always have access to the writer's schema, even if it differs from the schema currently used by the application. Without access to the original schema, correct deserialization cannot be guaranteed.
Unlike Avro data files, Kafka messages should not include the complete schema within every record because doing so would significantly increase message size.
Instead, Kafka applications typically use a Schema Registry.
The Schema Registry stores every schema used by producers and assigns a unique identifier to each version.
When a producer publishes a message, it stores only the schema identifier inside the Kafka record rather than embedding the full schema.
During deserialization, the consumer retrieves the corresponding schema from the Schema Registry using this identifier and reconstructs the original object.
This approach dramatically reduces message size while maintaining support for schema evolution.
Although Schema Registry is not part of Apache Kafka itself, the Confluent Schema Registry is the most widely adopted implementation.
Kafka producers serialize Avro objects using the KafkaAvroSerializer.
The producer configuration specifies:
Kafka bootstrap servers
KafkaAvroSerializer for both keys and values
The URL of the Schema Registry
Once configured, producers can send Avro-generated objects exactly like any other Kafka record. The serializer automatically registers new schemas when necessary, stores the schema identifier in each record, and serializes the object into Avro binary format.
This automation removes the need for developers to manually manage serialization or schema storage.
Avro-generated classes are not ordinary Java POJOs.
Instead, they are specialized classes generated directly from an Avro schema using tools such as:
avro-tools.jar
Avro Maven Plugin
The KafkaAvroSerializer only supports these generated Avro classes and cannot serialize arbitrary Java objects.
Applications therefore generate Java classes from their Avro schemas during the build process before producing Kafka messages.
Besides generated classes, Avro also supports GenericRecord, which represents data as a flexible key-value structure.
Unlike generated classes, GenericRecord requires the schema to be provided explicitly at runtime.
The application creates the schema, constructs a GenericRecord, populates its fields, and sends it through the same KafkaAvroSerializer.
The serializer extracts the schema from the GenericRecord, registers it if necessary, stores the schema identifier in Kafka, and serializes the associated data.
GenericRecord is especially useful when schemas are dynamic or generated at runtime and Java code generation is undesirable.
Every Kafka record consists of a topic, a key, and a value. While the key is optional, it plays a crucial role in determining which partition a record is written to. In addition to serving as business metadata, the record key is primarily used by Kafka's partitioning mechanism to ensure consistent data placement.
When records share the same key, Kafka guarantees that they are always written to the same partition (as long as the partition count remains unchanged). This behavior ensures that all events related to a specific entity, such as a customer or user, are processed in order by the same consumer, simplifying stateful processing and maintaining event ordering.
If a producer sends records without specifying a key, the default Kafka partitioner distributes messages across the available partitions.
Beginning with Apache Kafka 2.4, the default partitioner uses a Sticky Round-Robin strategy. Instead of sending every consecutive message to a different partition, Kafka temporarily selects one partition and fills an entire batch before moving to another partition.
This optimization significantly reduces the number of producer requests, decreases network overhead, lowers broker CPU utilization, and improves overall throughput while still maintaining an even distribution of records.
When a key is present, Kafka uses its internal hashing algorithm to calculate the destination partition.
The hashing algorithm is deterministic, meaning that the same key will always produce the same partition assignment. Kafka uses its own implementation of the Murmur2 hash algorithm to guarantee consistent partition mapping across different Java versions.
This deterministic mapping allows applications to rely on partition-level ordering, ensuring that all events associated with a particular key are processed sequentially.
Besides the default partitioner, Kafka provides additional partitioning strategies to support different workload characteristics.
The RoundRobinPartitioner distributes records randomly across all partitions, regardless of whether a key exists.
The UniformStickyPartitioner combines sticky batching with uniform workload distribution. Even when messages contain keys, this partitioner attempts to spread traffic evenly across partitions, preventing workload imbalance caused by highly active keys.
These partitioners are particularly useful in applications where the record key is required only as metadata—for example, when ETL pipelines use Kafka record keys as primary keys while loading data into relational databases—but message ordering by key is not required.
The relationship between a key and its partition remains stable only while the number of partitions in the topic remains unchanged.
If additional partitions are added later, Kafka recalculates the hash mapping. As a result, new records for the same key may be written to different partitions, while historical records remain in their original partitions.
This breaks the guarantee that all records for a particular key reside in the same partition.
For applications that rely on partition affinity or strict ordering, it is recommended to estimate future scalability requirements, create topics with a sufficient number of partitions from the beginning, and avoid increasing the partition count after production deployment.
Kafka allows developers to implement custom partitioning logic by creating a class that implements the Partitioner interface.
A custom partitioner can route records according to business requirements instead of relying solely on hash-based partitioning.
For example, if one customer generates a disproportionately large percentage of system traffic, assigning all of its records to a dedicated partition can prevent workload imbalance across the cluster.
The remaining customers can continue using hash-based partitioning across the other partitions.
A custom partitioner typically overrides the following methods:
configure() for initialization and configuration.
partition() to determine the destination partition for each record.
close() for cleanup when the producer shuts down.
The partition() method contains the core routing logic and can validate record keys, apply business-specific rules, or perform any desired partition selection algorithm.
Custom partitioning provides greater flexibility for optimizing data distribution based on business requirements.
Common use cases include:
Isolating high-volume customers or tenants.
Balancing uneven workloads across partitions.
Directing records to dedicated partitions for performance or compliance reasons.
Implementing domain-specific routing strategies that cannot be achieved through simple hash partitioning.
Although powerful, custom partitioners should be carefully designed because they directly affect data distribution, consumer parallelism, and system scalability.
In addition to the key and value, Kafka records can include headers.
Headers provide a mechanism for attaching metadata to a record without modifying the message payload. This allows infrastructure components and applications to exchange contextual information independently of the business data.
A Kafka header consists of an ordered collection of key-value pairs. Header keys are always strings, while header values are stored as serialized byte arrays, allowing virtually any type of information to be attached.
Headers are commonly used to transport technical metadata rather than business data.
Typical use cases include:
Recording the origin or lineage of a message.
Identifying the producing application or service.
Carrying routing information.
Supporting distributed tracing and correlation IDs.
Storing security or privacy classifications.
Passing versioning or processing metadata between services.
Because consumers and middleware can inspect headers without parsing the message payload, headers improve routing efficiency and are especially useful when message bodies are encrypted or encoded in binary formats.
Kafka provides a simple API for attaching headers to a ProducerRecord.
Headers can be added before sending the record by specifying a header name and its serialized value.
For example, a producer may attach a header such as privacy-level to indicate the classification of the data without modifying the record key or value.
This separation of metadata from business data results in cleaner message design and enables middleware, gateways, and consumers to make routing or processing decisions based solely on header information.
Kafka provides Producer Interceptors as a mechanism for extending or modifying producer behavior without changing the application's source code. This feature is particularly useful when organizations need to apply common functionality across multiple producer applications or when the original application code cannot be modified.
Producer interceptors execute automatically during the producer lifecycle and allow developers to inspect, enrich, monitor, or transform records before they are sent to Kafka, as well as observe the broker's response after the records have been acknowledged.
The ProducerInterceptor interface defines four primary methods that control the interceptor lifecycle:
configure() – Initializes the interceptor using the producer configuration before any records are processed.
onSend() – Invoked immediately before a record is serialized and sent to Kafka. The interceptor can inspect or modify the ProducerRecord and must return the record that will actually be transmitted.
onAcknowledgement() – Called after Kafka acknowledges a produced record. This method provides access to the metadata or any exception that occurred during delivery but does not allow modifying the broker's response.
close() – Executed when the producer shuts down, allowing the interceptor to release resources such as threads, network connections, or file handles.
These methods provide well-defined extension points for implementing custom producer behavior.
Producer interceptors are commonly used to implement cross-cutting concerns that should be applied consistently across producer applications.
Typical use cases include:
Collecting monitoring and performance metrics.
Implementing distributed tracing.
Automatically adding standard record headers.
Recording data lineage information.
Masking or removing sensitive information before transmission.
Logging producer activity for auditing purposes.
Because interceptors operate transparently, these capabilities can be introduced without modifying the business logic of existing producer applications.
A simple example of a producer interceptor counts the number of records sent and the number of acknowledgments received during fixed time intervals.
When a producer sends a record, the onSend() method increments a message counter before returning the original record unchanged.
Whenever Kafka acknowledges a record, the onAcknowledgement() method increments a separate acknowledgment counter.
A scheduled task periodically prints both counters, allowing developers or operators to monitor producer activity in real time.
This example demonstrates that interceptors can collect operational metrics without affecting normal message production.
Producer interceptors are configured entirely through producer configuration and do not require changes to application code.
To enable an interceptor:
Add the interceptor implementation JAR to the application's classpath.
Specify the interceptor class using the interceptor.classes configuration property.
Provide any custom configuration parameters required by the interceptor.
Because interceptors are configured externally, the same interceptor implementation can easily be deployed across multiple producer applications, including Kafka's command-line producer tools.
Kafka brokers support quotas, which limit the amount of resources individual clients are allowed to consume. Quotas protect the cluster from overloaded or misbehaving clients and help ensure fair resource allocation among multiple applications.
Kafka supports three types of quotas:
Produce Quotas – Limit the rate at which producers can send data, measured in bytes per second.
Consume Quotas – Limit the rate at which consumers can receive data.
Request Quotas – Limit the percentage of broker processing time that a client may consume.
These quotas improve cluster stability by preventing a single client from monopolizing broker resources.
Quotas can be configured at multiple levels depending on administrative requirements.
Kafka supports quotas for:
All clients (default quotas).
Individual client IDs.
Individual authenticated users.
Specific combinations of users and client IDs.
Default quotas are typically defined in the broker configuration file and apply to every client unless overridden.
For more flexible administration, Kafka also supports dynamic quota configuration, allowing administrators to modify quotas without restarting brokers. Dynamic quotas can be managed using the kafka-configs command-line utility or the Kafka AdminClient API.
This approach enables administrators to quickly adjust client limits as workloads change.
When a client exceeds its assigned quota, Kafka automatically applies throttling.
Rather than rejecting requests immediately, the broker intentionally delays responses to slow the client's effective throughput. Since Kafka clients maintain only a limited number of concurrent requests, these delays naturally reduce the rate at which new requests are sent.
If a client continues sending requests aggressively while being throttled, Kafka may temporarily mute communication with that client until its traffic returns to the permitted rate.
This mechanism protects broker resources while avoiding unnecessary request failures.
Kafka exposes several metrics that allow administrators to monitor throttling behavior.
Examples include:
produce-throttle-time-avg
produce-throttle-time-max
fetch-throttle-time-avg
fetch-throttle-time-max
These metrics report the average and maximum amount of time producer or consumer requests were delayed due to quota enforcement.
Monitoring these values helps identify clients that consistently exceed configured quotas and may require optimization or quota adjustments.
If producers generate messages faster than brokers can accept them—whether due to broker capacity limits or enforced quotas—the buffered messages begin to accumulate in memory.
As long as buffer space remains available, the producer continues accepting new messages. However, if the sending rate continues to exceed the broker's processing capacity, the producer eventually exhausts its available buffer space.
At that point, subsequent send() operations may block until buffer space becomes available. If the waiting time exceeds the configured timeout, Kafka throws a TimeoutException.
Additionally, records already waiting inside producer batches may exceed the configured delivery.timeout.ms value before reaching the broker. These expired records are discarded, and their callbacks are invoked with a TimeoutException.
For this reason, production systems should continuously monitor producer throughput, broker capacity, buffer utilization, and timeout metrics to ensure that Kafka infrastructure can sustain the expected workload without causing excessive buffering or message expiration.
Use producer interceptors for monitoring, tracing, and metadata enrichment instead of modifying application code.
Configure quotas to protect shared Kafka clusters from resource abuse.
Prefer dynamic quotas over static broker configuration whenever possible.
Continuously monitor throttling metrics to identify overloaded clients.
Ensure broker throughput matches producer traffic to prevent excessive buffering and timeout exceptions.
Tune producer timeout and buffer configurations according to expected workload characteristics.
