ExpertWeb & Load Balancing
Apache Kafka Exactly-Once Processing
Solve the two generals problem in distributed systems by configuring Kafka for exactly-once semantics, ensuring no message duplication during network retries.
KafkaJava
Prerequisites
- Kafka Cluster (Broker 0.11+)
Configuration Files
producer.properties./config/producer.properties
enable.idempotence=true\nacks=all\nmax.in.flight.requests.per.connection=5\nretries=2147483647\ntransactional.id=order-processor-1Explanation:Enables idempotence (deduplication) and defines a transactional ID to group operations atomically.
Verification Steps
1
Ensure topics used in transactions have appropriate replication.
$kafka-topics.sh --describe --topic transactions
Expected OutputReplicationFactor: 3
Production Gotchas
- Consumers must set isolation.level=read_committed to ensure they do not read uncommitted transactional messages.
Frequently Asked Questions
How does idempotence work?
The producer assigns a sequence number to each message. The broker uses this to discard duplicates caused by network retries.