Your application sends order events to a message broker, and one background worker updates a reporting database. At first, the worker keeps up comfortably. As the shop grows, events arrive faster than the worker can process them, and reports begin showing yesterday's information.
Starting another worker sounds like the obvious fix. But should both workers receive every event, or should they divide the work? What happens if one stops halfway through an update? The answers depend on how the workers subscribe, how the stream is divided and when progress is saved.
Introduction
A consumer is a program that receives messages and does something with them. A consumer group is a set of consumers cooperating as one logical subscriber. In a partitioned event log such as Apache Kafka, members of the same group share responsibility for reading the log's partitions.
We will follow a small order-reporting application as it grows from one worker to several. Along the way, we will introduce partitions, offsets, rebalancing and lag using concrete examples. These terms describe where work lives, who currently owns it and how far processing has progressed.
The important goal is useful completed work. Adding processes can improve throughput and recovery, but it does not automatically prevent duplicate database updates or preserve the order of asynchronous tasks. The worker's processing rules must fit the broker's delivery contract.
Start with One Event and Two Different Jobs
An OrderPlaced event says that an order was accepted. The reporting service wants to add it to a sales dashboard. A separate notification service wants to send the customer a confirmation. Both services need to learn about that event, because they perform different jobs.
Those two services should generally use separate logical subscriptions. Each subscription keeps its own progress. Reporting can be temporarily behind without preventing notifications from moving forward, and notification retries should not reset the reporting service's position.
Now imagine two instances of the reporting service. They perform the same job against the same reporting model. Putting them in the same consumer group allows them to share that subscription's work instead of both deliberately processing its entire stream.
Order events
-> reporting group: worker A and worker B share the work
-> notification group: worker C handles notifications
The difference between a service and an instance matters. A new service with a new responsibility normally needs its own subscription. Another instance of an existing service normally joins that service's existing work-sharing group. Confusing these cases is a common cause of missing actions or unexpected duplicates.
Understand a Partitioned Log
A topic is a named stream of records, such as orders. To spread storage and processing, a topic can be divided into partitions. Each partition is an ordered sequence with its own positions, called offsets.
Think of four separate numbered workbooks rather than one giant workbook. A record at offset 50 in partition 0 has no direct ordering relationship with offset 50 in partition 1. The numbers identify positions inside their own books.
A producer commonly chooses a partition using a message key. Events for the same order can use the order identifier as their key, keeping them together under a stable partitioning scheme. This makes it possible to process that order's events in their recorded order.
The partition key is therefore more than a performance setting. It helps define the scope of order. Choosing a random key for every event may spread traffic well while scattering one order's changes across independent sequences.
The Kafka consumer documentation explains partitions, group membership and committed positions. We will use Kafka's ordinary partition-assigned consumer groups as the main model; other broker features can offer different work-sharing semantics.
See How Two Workers Share Four Partitions
Suppose the orders topic has four partitions and the reporting group initially contains one worker. Worker A reads all four. Adding worker B allows the group coordinator to assign some partitions to each member.
One possible assignment is:
Worker A: partitions 0 and 1
Worker B: partitions 2 and 3
The exact assignment depends on the broker and assignment strategy. The essential rule in this model is that a partition has one assigned consumer within that group at a time. A different group can independently read the same partition for another purpose.
Adding workers C and D can give each worker one partition. Adding a fifth worker does not create a fifth partition. At least one member will have no partition to read, assuming this is the group's only subscribed topic.
This explains why doubling workers sometimes changes nothing. The available partition assignments place a limit on parallel reading through this group model. The application may perform additional internal parallel processing, but that introduces separate ordering and checkpointing responsibilities.
Do not assume every broker uses this exact limit. Queue-based systems can distribute individual messages differently, and newer features may use another assignment model. Start by naming the specific subscription mode rather than applying one product's rules to everything called a consumer.
Separate Reading from Completing the Job
Receiving an event does not mean its business processing has finished. A worker may have fetched it into memory, started a database transaction or sent a request to another service. These are different stages.
An offset is a bookmark in one partition. The worker's current read position can move ahead when it fetches records. Its committed position is the durable bookmark used when processing restarts under the broker's checkpoint contract.
In Kafka, a committed offset normally identifies the next record to consume, not the last record already handled. If records through offset 42 have been completed in a simple consecutive example, the next position is 43. Actual logs can contain gaps, so use the client's supported position information rather than assuming every integer corresponds to an application record.
The worker should save progress only when the work covered by that bookmark is safely complete according to its design. Committing as soon as records enter memory can skip unfinished work after a crash.
Automatic commit settings need to be understood alongside the processing loop. They can be appropriate for some patterns, but handing fetched records to background tasks and continuing to poll can let automatic progress move beyond those tasks. The broker cannot infer that an arbitrary task has finished updating your database.
Walk Through a Crash after a Database Update
Worker A reads OrderPlaced for order 842 and inserts the order into the reporting database. The database commits successfully. Before the worker saves its new offset, the process crashes.
The replacement starts from the last committed bookmark, so it reads order 842 again. This is a normal consequence of saving progress after processing: completed work can be repeated when acknowledgement is lost.
The reporting handler needs an idempotent operation, meaning that repeating the same logical event does not apply its effect twice. A unique event identifier recorded with the database update can prevent a duplicate insertion or double increase in a sales total.
Begin database transaction
If this event was already applied, return its saved outcome
Apply the reporting change
Record this event as applied
Commit database transaction
Then save the broker checkpoint
The reporting change and processed-event record belong in the same transaction. If they are saved separately, a crash between them can still produce a duplicate or incorrectly suppress unfinished work.
This protection is separate from consumer-group assignment. The group normally chooses one active reader per partition, but recovery can deliver a record again. Work sharing reduces simultaneous duplication; it does not remove the need to handle repeated delivery.
Consider Keeping State and Progress Together
When the worker's entire result lives in one database, it can sometimes store its processing position in that database alongside the result. One transaction then commits both the business update and the bookmark.
On restart, the worker resumes from that stored position. This closes the gap between saving the reporting change and saving progress in another system. It also means the application owns more of the recovery and partition-transfer protocol.
The record must identify the correct topic, partition, consumer purpose and source history. A position from a recreated topic or a different dataset is not interchangeable merely because its number looks plausible.
If partition ownership changes, the replacement must load the authoritative database checkpoint and ensure an old worker cannot continue applying conflicting updates. Use the client framework's supported handoff mechanisms and appropriate database conditions. Do not assume storing a number in SQL solves ownership automatically.
External effects remain outside that database transaction. A confirmation email accepted by an email provider cannot become atomic with the reporting bookmark through an ordinary SQL commit. Such operations require stable effect identities and provider-aware recovery.
For a beginner implementation, manual checkpointing after a transaction plus idempotent processing is often easier to operate. More integrated designs can be worthwhile when their stronger boundary addresses a measured requirement.
Understand What Happens When a Worker Leaves
When a worker shuts down or stops communicating, the group must assign its partitions to surviving members. This redistribution is called a rebalance. Adding a worker or changing subscribed partitions can also trigger assignment changes.
Suppose worker B owns partitions 2 and 3 and disappears. Worker A may temporarily take them both. It resumes from the group's saved progress, so recently processed but uncommitted records can be delivered again.
Failure detection is not instantaneous. The broker or coordination mechanism waits according to its membership rules before deciding the worker is unavailable. A shorter detection interval can improve takeover time but can also react to ordinary pauses or network delays.
During a planned shutdown, stop accepting additional processing, finish bounded in-flight work where appropriate and save only completed progress. Then leave the group through the supported client shutdown path. This gives the system a cleaner handoff than abruptly terminating every process.
The old worker may not actually be dead. It could be paused or disconnected from the coordinator while still able to reach the reporting database. Cancellation, assignment-generation checks and idempotent or conditionally ordered database writes help make that stale work harmless.
Keep Ownership Changes Separate from Business Ordering
A partition's log order tells the worker which record came first in that partition. It does not force application tasks to complete in that order. A worker that launches every event as an independent task can let a later update finish before an earlier one.
Imagine OrderPlaced followed by OrderCancelled for the same order. If cancellation finishes first and creation later overwrites it with an unconditional insert-or-update, the report can incorrectly show an active order.
The simplest approach is sequential processing within each partition while processing different partitions concurrently. This preserves order at the cost of letting one slow record delay later records in that partition.
More advanced designs can process independent keys concurrently while preserving each key's sequence. They must also track completion carefully so the committed partition bookmark does not pass an unfinished earlier record.
For state-replacement events, a source version can reject older updates at the database boundary. That can improve safety, but not every event can be skipped because a newer version exists. Two separate payment facts are not interchangeable with two snapshots of a product description.
Start with the ordering requirement of the business effect, then choose concurrency. More tasks are useful only if they preserve the meaning of the events they process.
Do Not Commit Past a Gap
Suppose a worker fetches records at offsets 100, 101 and 102. It processes them concurrently. Records 101 and 102 finish, but 100 is still running. Saving progress beyond 102 would tell a replacement to skip 100 after a crash.
The safe bookmark can advance only through the completed prefix of work that the application has accepted responsibility for. A completed prefix means there is no unfinished earlier record inside the range being acknowledged.
100 unfinished, 101 done, 102 done
Safe progress cannot move past 100
100 done, 101 done, 102 done
Progress may advance to the next supported position
This bookkeeping becomes more complicated with retries, cancellation and partition reassignment. Completed tasks must report back to the component that owns checkpoint decisions, and results from an obsolete assignment must not accidentally advance a new owner's progress.
Batch size also matters. A large batch can improve throughput but increase the amount of repeated work after failure. It can delay checkpoint advancement behind one slow record and require more memory for outstanding tasks.
If the application does not need that complexity, process bounded batches sequentially per partition. A simpler worker that can explain every saved bookmark is often more dependable than a highly parallel worker whose recovery behaviour is uncertain.
Diagnose Why Another Worker Did Not Help
First check assignments. If all partitions already have consumers, another group member may be idle. If one worker owns several partitions, additional members may help, provided the downstream system has spare capacity.
Next check distribution. One partition might receive most of the traffic because a popular customer or an unfortunate key choice dominates the stream. Three nearly idle partitions cannot compensate for one overloaded partition when its required order is serial.
Then inspect processing time. A database lock, slow external call or expensive document may consume most of the worker's time. Adding consumers can increase pressure on the same shared bottleneck rather than improve useful completions.
Measure arrival rate and completion rate for each partition. If the hot partition receives 150 records per second but its worker completes 100, backlog grows by fifty per second. Adding an idle worker cannot change that serial path without changing how the work is divided or processed.
Finally, check the load generator and operational environment if this is a test. A local development database, tiny dataset or unrealistic message keys can hide the bottleneck that matters in production. Record the workload and configuration with the result.
Choose Partition Keys with Room to Grow
A key should preserve the smallest meaningful ordering scope. Using an entire tenant as the key may unnecessarily serialise thousands of independent orders. Using the order identifier may allow those orders to progress independently while keeping each order's events together.
Some operations genuinely need broader order. If all changes update one shared account balance, splitting them arbitrarily across partitions creates a coordination problem elsewhere. The design cannot remove a business dependency simply by selecting a different hash function.
Increasing a topic's partition count can change where future keyed records are placed under common partitioning strategies. Existing records do not automatically move into a new order with the future ones. Plan this transition if consumers rely on a key always belonging to one partition.
One option is a new topic with an explicit migration and cutover. Another is an application partitioning scheme whose compatibility rules are designed in advance. The right approach depends on the platform, so do not treat partition expansion as a harmless setting change.
Avoid creating an enormous number of partitions merely because future load is unknown. Partitions consume broker, client and operational resources. Estimate plausible concurrency needs, measure actual skew and choose a manageable growth strategy.
Give Each Independent Consumer Purpose Its Own Progress
A new fraud-analysis service needs to read the order events independently of reporting. Give it a distinct group identity and decide where its initial position should begin. It might need retained history, or only events arriving after the service is enabled.
Accidentally reusing the reporting group name would cause fraud and reporting workers to share assignments. Some events would be handled by a fraud worker and never reach a reporting worker in that group, even though both services expected every event for their own purpose.
The opposite mistake is giving every reporting instance a random group name. Each becomes an independent subscriber and reads the stream again, potentially multiplying database work and notifications.
Use stable group names that identify environment and purpose, such as production.order-reporting. Keep test and production permissions separate so a development process cannot join a live work-sharing group accidentally.
Azure Event Hubs also describes consumer groups as independent views of a stream, with checkpoint and processing responsibilities described in its features and terminology guide. Follow its supported processor ownership model rather than assuming Kafka client settings apply directly.
Compare Consumer Groups with Queue Consumers
A traditional work queue can distribute individual messages among competing consumers. Acknowledging a message normally removes that delivery obligation from the queue under the broker's contract. A retained event log instead lets each group move its own bookmark through shared history.
Both models can support multiple workers, but their replay and ordering properties differ. With a log, a new group can often read older retained events without creating a new copy of every record. With a queue, completed messages may no longer be available unless the application archived them elsewhere.
RabbitMQ's consumer documentation explains acknowledgement, competing consumers and consumer behaviour for its queue model. Features such as single-active-consumer mode or streams introduce additional choices, so identify the exact feature in use.
Do not choose solely from terminology. A thumbnail-generation task might fit an individually acknowledged queue well. A stream of order facts used independently by reporting, fraud and analytics may benefit from retained logs and separate progress.
The worker's business effect needs safe repetition in either case. A queue can redeliver after lost acknowledgement, and a log consumer can replay after an offset was not committed. Different transport mechanisms still expose the same gap between doing work and recording completion.
Keep Slow Processing from Breaking Group Membership
A consumer has operational responsibilities as well as business work. Depending on the client protocol, it must poll, send heartbeats or maintain an ownership lease within configured limits. Long blocking work can interfere with those responsibilities.
Do not simply increase every timeout until rebalances disappear. A larger timeout can delay recovery from real failures. Find whether the problem is an oversized batch, a blocked thread, an external call or an inappropriate processing model.
Keep batches bounded and make slow operations cancellable where possible. If processing is moved to separate workers, keep fetched-but-unfinished work bounded and preserve the checkpoint rules described earlier. Background polling must not turn into unlimited prefetch into memory.
Use the documented thread-safety rules of the consumer library. For example, Kafka's Java consumer is not generally thread-safe. An application should not have several arbitrary tasks concurrently calling its poll and commit methods without an appropriate supported coordination design.
Watch for repeated membership changes during ordinary traffic. Frequent rebalances can indicate an unstable deployment, processing that exceeds membership deadlines or resource starvation. They are a symptom to explain, not just noise to suppress in logs.
Handle Failed Messages without Blocking Everything Forever
A malformed or unsupported event can repeatedly fail at one position. If strict partition order is required, later events may remain blocked behind it. Retrying forever can make a healthy worker look busy while its useful progress is zero.
Classify the failure. A temporary database outage may justify delayed retry. An invalid schema needs a code or data decision. An event that cannot be processed under the current contract should become a visible diagnostic case rather than an endless tight loop.
A dead-letter or quarantine path stores the event and failure context for investigation. Moving past it is a business decision: skipping an inventory update can make later state wrong. Some projections can rebuild from authoritative state; other consumers must wait until the missing event is resolved.
If the event is quarantined, make the transfer and checkpoint decision recoverable. A crash must not both lose the event from the original path and fail to preserve it in the investigation store. Duplicate quarantine records should be recognised by stable event identity.
Document how repaired events return to processing and how their original order is handled. Blindly appending an old event at the end of a stream may not recreate the state that would have resulted from processing it in its original position.
Observe the Group as a Working System
Track lag per partition, completed-event rate, processing duration, errors, assignment changes and checkpoint age. Lag measures how far processing is behind according to the platform's position model; oldest unprocessed event age adds a useful time-based view.
A large offset difference is not always the same as a large count of visible application events, because logs can contain gaps or internal records. Use the broker's metrics with an understanding of what they measure.
Distinguish fetched progress from durably completed progress. A consumer that reads rapidly into a local queue can appear close to the log's end while the reporting database remains far behind. The user cares about completed reporting, not the contents of worker memory.
Keep metric labels bounded. Group and partition identities are usually manageable, while arbitrary order identifiers belong in logs or traces. Correlate a specific event across receipt, processing and saved outcome when investigating a customer issue.
Alert on sustained failure to make progress and on increasing age, not only on process availability. A worker can be alive, connected and sending heartbeats while repeatedly failing the same record. That is a working connection attached to a broken business process.
Test Adding and Removing Workers
Begin in an isolated environment with four partitions and one worker. Send events with known keys and verify which partition receives each key. Add a second worker and inspect the assignment change rather than assuming an even split.
Pause a worker after its database transaction commits but before its checkpoint is saved. Let another worker take over and verify that the repeated event does not duplicate the reporting effect.
Test a slow earlier event while later tasks finish. Confirm that progress never moves beyond unfinished work. Then terminate the process and check that recovery resumes from a position that preserves every required update.
Try a deliberately skewed key distribution, a failed event and a downstream connection limit. These experiments show when additional consumers help and when the bottleneck belongs elsewhere.
Finally, test a rolling deployment with old and new consumer versions in the same group. Both versions must understand the messages they can receive during the overlap. Scaling and deployment share the same assignment mechanisms, so a safe worker must handle both.
Summary
Consumer groups let several workers act as one logical subscriber. In a partition-assigned log, workers share partitions, separate groups keep independent progress and adding more workers helps only when work and downstream capacity can actually be divided.
Offsets are bookmarks, not proof that business processing finished. Save progress after the relevant work is durable, handle repeated events safely and never commit past an unfinished earlier record. Rebalances and crashes make these rules part of ordinary operation.
Start with clear group identity, an appropriate partition key and simple bounded processing. Then measure lag, useful completions and ownership changes. That gives you a reliable basis for deciding whether to add workers, change the partitioning or fix a bottleneck outside the consumer group.