The labs on this site need JavaScript to run. The same content is written out below.

Learn › Architecture and operations › Messaging and asynchronous processing

Kafka partitions and consumers

I added consumers, so why is the throughput the same?

A topic is split into partitions, and the consumers of a consumer group divide the partitions among themselves. Within a group, a partition can be read by only one consumer.

Checked against documentation and standards · Checked against the official Apache Kafka documentation

Concepts in this lab

Kafka topics, partitions, and offsets

A topic is a name that separates messages by kind. A topic is divided into one or more partitions, and a partition is a record (log) to which messages are appended at the end in the order they arrive. An offset is a number that shows the position of each message within a partition, and a consumer uses offsets to record how far it has read. Order is guaranteed only within the same partition.

Consumer groups and rebalancing

A consumer group is a set of consumers that share the work of processing one topic. Within the group, each partition is assigned to only one consumer, so the consumers divide the messages among them. Rebalancing is the process of dividing up the partition assignments again when a consumer is added or stops, and processing can be delayed briefly while it is in progress.

Consumer lag

The difference between the position of the last message that producers wrote and the position that the consumer has processed. In other words, it is the amount of messages that are piled up and not yet processed. If the value keeps growing, the consuming speed is not keeping up with the producing speed.

Message delivery guarantees

The level of guarantee about how many times a message is delivered. With at-most-once, a message is delivered at most one time, so there are no duplicates but messages can be lost. With at-least-once, a message is delivered at least one time, so nothing is lost but duplicates can occur. With exactly-once, a message is delivered exactly one time. In practice, systems often use at-least-once and make the consumer handle duplicates (idempotent).

Message queue

An intermediate store that sits between the sending side (the producer) and the receiving side (the consumer), holds messages, and then delivers them. The producer puts a message in and moves straight on to its next work, and the consumer takes messages out and processes them at its own pace. The two systems do not call each other directly, so coupling is lower, and a sudden burst of requests can be piled up and processed later.

Idempotency and idempotency keys

Idempotency is the property that performing the same request several times gives the same result as performing it once. A network problem can cause the same request to be sent again, so operations that must not be duplicated, such as payments, must be made idempotent. An idempotency key is a unique value that the client attaches to each request. When a request arrives with a key that the server has already processed, the server does not process it again and returns the earlier result.

Try these

• Default settings — 2 consumers for 3 partitions. The processing capacity is too low, so messages pile up.
• With 3 consumers — Each partition gets one consumer, and the pile-up disappears.
• Even with 6 consumers — There are only 3 partitions, so 3 consumers have nothing to do.
• With 6 partitions too — Now all 6 consumers work.
• A consumer dies while processing (commit after) — Another consumer takes over, and it processes some messages again.
• A consumer dies while processing (commit before) — Messages that were already committed are skipped without being processed.

Running with the default settings

1. 2 consumers subscribe to the topic as one group. The 3 partitions are divided among the consumers (the group coordinator coordinates the assignment). Within a group, a partition is handled by only one consumer.
2. Messages arrive at 120 per second, and 40 per second go into each partition (assuming the keys are spread evenly). Each consumer processes 50 messages per second, and a consumer that handles several partitions shares that capacity among them. The throughput of the whole group is 90 messages per second.
3. 10 seconds have passed. Processing cannot keep up with the inflow, so 300 messages are waiting unread (consumer lag). If this continues, the lag grows without limit.

Result of this run

• Lag builds up. You must increase the consumers up to the number of partitions, or make each consumer process faster. Lag is the key metric that shows whether the consumers are keeping up.
• Order is guaranteed only within the same partition. If the events of the same order must be processed in order, give the order ID as the key so that they go to the same partition.

Code

// Topic orders: 3 partitions
// Consumer group order-service: 2 consumers

while (true) {
    ConsumerRecords<String, Order> records = consumer.poll(Duration.ofMillis(500));
    for (var r : records) {
        handle(r.value());               // first, process
    }
    consumer.commitSync();               // then, record "read up to here"
}

Assumptions of this model

It is assumed that the message keys are spread evenly so that each partition receives the same amount, and partitions are simplified to be assigned to the consumers one by one in turn (the actual assignment strategy depends on the configuration). The time a rebalance takes and the delay during it are not given as numbers. The example of the dead consumer (three of five messages processed) is a fixed situation to show the difference made by the commit timing. The rule that only one consumer in a group reads a partition and the definitions of the delivery guarantees follow the official Kafka documentation.

Previous lab: Message queue basics · Next lab: Circuit breaker