Software Architecture & Design

Why 2024 Architecture Advice Fails DevOps in Production

A service can deploy cleanly and still leave DevOps with a backlog nobody knows how to drain. Over the last two years, broker upgrades and maturing telemetry have made old advice about “just add a consumer” less defensible in production. My position: choose service and event boundaries by who can recover failed work, not by how independently teams want to deploy.

Kafka 4.0 makes broker-era runbooks a liability

Apache Kafka 4.0, released in 2025, removed ZooKeeper support and requires KRaft. That is a concrete change for an on-call team: a runbook that tells an operator to inspect ZooKeeper or change broker state through ZooKeeper no longer describes the system being operated. Inventory those commands before an upgrade, and replace them with procedures tested against the cluster’s actual Kafka version.

Kafka 4.0 also offers a newer, broker-driven consumer rebalance protocol that clients can opt into with group.protocol=consumer. Older advice to treat every rebalance as an unavoidable, group-wide pause needs qualification because the new protocol can reduce disruption during membership changes. I would not enable it across every consumer during the broker upgrade, because changing broker coordination and client behavior in one maintenance window makes a stalled partition harder to diagnose. Test one consumer group, including a rolling restart and a slow handler, before widening the change.

Read Microservices vs Monolith Choosing the Right Architecture critically, because deployment independence says little about who repairs a service that has stopped making progress. A modular monolith can keep a database transaction and its failure visible to one team; splitting the same operation across services adds network, broker, and consumer states that someone must observe. I would keep a workflow together when one on-call team owns its data and release schedule, because a new service boundary would add recovery steps without providing an independent operational owner.

That does not mean avoiding Kafka or microservices. It means requiring an answer to three questions before creating another consumer group: Who can pause it? Who can replay its work? Who decides whether a duplicate side effect is safe? If those answers span teams, record the handoff in the runbook before the first production event, because the broker cannot resolve an ownership dispute during an incident.

Retries are a delivery contract, not an invisible queue setting

“Put it on a queue and retry” is outdated advice when the side effect can be charged, emailed, or applied twice. Kafka consumers may see a record again after a crash or offset-commit failure, so successful publication is not proof of exactly-once business execution. Make the consumer’s idempotency rule explicit: for a CloudEvents 1.0 message, source and id can identify an event, while a business operation identifier may be needed if different events can request the same side effect.

Read Event Driven Architecture Patterns for Scalable Systems critically, because a fan-out diagram does not show which consumer is allowed to execute twice. For each handler, document whether it deduplicates, overwrites a deterministic result, or requires a compensating action. Put the deduplication record and the protected database change in the same PostgreSQL transaction when possible; otherwise a crash between those writes can defeat the safeguard.

Retry timing should match the failure. A brief network error might justify a backoff whose initial 5 seconds is a value to tune under load; a schema-invalid event will not become valid by waiting. Send irrecoverable records to a dead-letter destination with the original payload, error category, and event identifier, because an error message alone is insufficient for a safe replay. Set a retention period deliberately rather than treating the dead-letter queue as permanent storage.

W3C Trace Context headers such as traceparent help connect publication to consumption, but a trace is not a durable audit log because sampling and retention can remove it. Carry a stable event ID through logs and metrics as well. Avoid putting event IDs in Prometheus label values, however, because one label value per event creates unbounded series cardinality; search for individual IDs in logs or a dedicated event ledger instead.

The transactional outbox beats dual writes when rollback is real

Compare direct Kafka publishing with a PostgreSQL transactional outbox plus Debezium. Direct publishing wins when the producer does not need to make a database change atomically with the event: it has fewer components, but the application must still handle publish failures and retries. The outbox wins when a committed database change must eventually produce an event, because the application writes both its business row and an outbox row in one transaction. Its cost is operational: Debezium needs a replication slot, change-stream monitoring, and a plan for clearing old outbox rows.

I would not write the database, publish to Kafka, and call the sequence “reliable” because a process crash between those operations leaves either an unpublished change or an event for a rolled-back change. Nor does the outbox make the consumer’s side effect exactly once: Debezium or the consumer can retry, so the receiving application still needs an idempotency rule.

A replication slot is not free storage. Monitor PostgreSQL’s pg_replication_slots and the WAL retained behind a slot; if a connector stops advancing, retained WAL can consume disk while the application appears healthy. Set a WAL-retention alert at a locally tested capacity threshold rather than copying a generic number, because disk size, write rate, and recovery time determine how long an operator has to act. Test the response—restart the connector, investigate its error, or intentionally rebuild it—before deleting a slot, because deleting it can forfeit unread changes.

The choice between Kafka and RabbitMQ also depends on the work. Kafka wins when multiple consumer groups need independent replay of a retained log, at the cost of partition, offset, and retention operations. RabbitMQ quorum queues win for work distribution where acknowledgments and queue-oriented routing fit the task, at the cost of a different replay model and quorum capacity planning. Neither broker removes the need to bound retries and test duplicates, because acknowledgment and application side effects cannot generally be committed in one transaction.

A consumer is healthy only when its oldest work is young

Replica count and HTTP health checks are weak evidence of event-processing health because a ready pod can keep accepting records while its oldest work ages. Track Kafka consumer lag with a metric such as kafka_consumergroup_lag, but pair it with the age of the oldest uncompleted business event: a low record count can still contain one expensive, stuck operation. For a proposed service objective of 99.9% of eligible events completed within its agreed deadline, measure completion from event creation to the durable side effect, not merely to the consumer’s poll.

Use OpenTelemetry and OTLP to connect producer and consumer spans, then inspect actual attribute names after instrumentation upgrades; mixed messaging semantic conventions can split dashboards even when traffic is unchanged. Add Kubernetes signals such as kube_pod_container_status_restarts_total and container_memory_working_set_bytes, because repeated restarts or memory pressure can explain a lag increase. Keep the alerts distinct: rising lag with stable event age may be a burst the system is draining, while rising age after input falls suggests progress has stopped.

A small duplicate-event lab is more useful than a diagram when checking a handler’s claim of idempotency. This shell example starts the official Kafka 4.0.0 image, waits for its broker, and publishes the same sample line twice:

docker run -d --name kafka-lab -p 9092:9092 apache/kafka:4.0.0
until docker exec kafka-lab /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 --list >/dev/null 2>&1; do sleep 1; done
docker exec kafka-lab /opt/kafka/bin/kafka-topics.sh \
  --bootstrap-server localhost:9092 --create --topic replay-test \
  --partitions 3 --replication-factor 1
printf 'order-1\norder-1\n' | docker exec -i kafka-lab \
  /opt/kafka/bin/kafka-console-producer.sh \
  --bootstrap-server localhost:9092 --topic replay-test

The 3 partitions and single replica are lab settings, not production recommendations. Connect a staging consumer and check its durable result after both records; then repeat with a crash after the side effect but before offset commit. Record the observed duplicate count and recovery time in the runbook, because only that measured behavior can validate the handler’s stated guarantee.

Run the first replay before the first incident

Pick one production consumer group and schedule a staging replay of an already-processed event this week. Before running it, identify the side effect, deduplication key, offset-reset procedure, and person authorized to approve replay. Capture event age and the durable result before and after. If the team cannot prove that replay is safe, pause the next service split: another deployment boundary will make that unanswered recovery question harder to own.