Kafka, the log
at the heart of your system.

Strip away the marketing and Kafka is one thing: a distributed, append-only, durable log of events that many independent consumers can read at their own pace. Not a queue, not a database, not a message bus in the classic sense — a log. Producers append records to the end; consumers read forward from a position they track themselves; and the records stay after being read, for a configured retention. Once that single picture is in your head, every feature in this manual is a consequence of it.

The official description is "a distributed event streaming platform," and each word earns its place. Distributed: it runs as a cluster of brokers, replicating data for fault tolerance and scaling horizontally. Event streaming: it's built around an unbounded, continuous flow of immutable events (something happened: an order was placed, a payment cleared, a sensor read), not request/response calls. Platform: producers, consumers, a connector ecosystem, and stream processing all sit on the same log.

This is the most important section in the manual, because conflating Kafka with a traditional message queue is the root of most confusion. The task-queues chapter covered the queue model in depth — producers, workers, brokers like RabbitMQ and SQS, retries, visibility timeouts. Kafka is a different shape of thing, and the difference is not a detail; it changes what problems it solves.

Everything Kafka does rests on the humblest data structure in computing: the append-only log — an ordered sequence of records you can only add to the end of, never insert into the middle of or modify. Each record gets a monotonically increasing offset (0, 1, 2, …) that is its permanent address. This is the same structure a database uses for its write-ahead log and durability; Kafka's insight was to make the log itself the primary, public abstraction rather than a hidden internal detail.

Why an append-only log is so fast and so useful

• Sequential I/O is cheap. Appending to the end of a file and reading it forward are sequential disk operations — dramatically faster than random access, even on SSDs. Kafka leans on the OS page cache and zero-copy transfer to move data from disk to network without passing through application memory, which is how it sustains millions of messages per second on modest hardware.
• Immutability is simple & safe. Records never change, so there are no locks for updates, no read/write conflicts, and concurrent readers never interfere. An immutable history is also trivially cacheable and easy to reason about.
• The log is the history. Because nothing is overwritten, the log is a complete, ordered record of everything that happened — which is exactly what makes replay (§11), event sourcing (§14), and feeding new consumers from the beginning possible.

A topic is a named stream of related events — orders, payments, page-views. It's the logical channel producers write to and consumers read from. But a topic is not a single log; it's split into partitions, and the partition is where all the real mechanics live. The partition is simultaneously Kafka's unit of parallelism, of ordering, and of storage — three of the most important properties in the system, all riding on this one concept.

The three roles of a partition

• Parallelism. Partitions can live on different brokers and be consumed by different consumers at once, so a topic's throughput scales with its partition count. More partitions → more parallel consumption (up to the consumer-group limit — §7).
• Ordering. Kafka guarantees order within a single partition and makes no promise across partitions. This is the central trade-off (§10): to keep related events ordered, you route them to the same partition via a shared key (all events for order-42 use key "order-42", so they're strictly ordered relative to each other).
• Storage. Each partition is physically a set of segment files on a broker's disk — the actual append-only log from §3. Retention and compaction (§11) operate per partition.

A Kafka cluster is a set of brokers (servers); partitions are spread across them, and that's how the cluster scales and survives failure. Durability comes from replication: each partition has one leader replica and some follower replicas on other brokers. All reads and writes go to the leader; followers continuously copy from it. If the leader's broker dies, a follower is promoted — no data lost, brief unavailability. This is the same desired-state/failover thinking as the Kubernetes chapter, applied to data.

The durability dials: acks & ISR

How safe a write is depends on two settings. The producer's acks says how many replicas must confirm before the write is considered done: acks=0 (fire-and-forget, may lose data), acks=1 (leader only — lost if the leader dies before followers copy it), or acks=all (the leader plus all in-sync replicas — the safe choice). The ISR (in-sync replica set) is the followers currently caught up with the leader; pairing acks=all with min.insync.replicas=2 means a write only succeeds if at least two replicas have it — surviving a single broker loss with zero data loss.

A producer publishes records to a topic. Three producer concerns shape correctness and performance: the key (which decides the partition and thus ordering — §4), the acks (durability — §5), and batching (the producer groups records and compresses them for throughput). A record is a key/value pair plus optional headers and a timestamp; the value is your event payload (often Avro/Protobuf — §12).

// Using confluent-kafka-go (librdkafka). kafka-go (segmentio) is a pure-Go alternative.
p, _ := kafka.NewProducer(&kafka.ConfigMap{
    "bootstrap.servers": "localhost:9092",
    "acks":              "all",   // durability: leader + in-sync replicas (§5)
    "enable.idempotence": true,   // no duplicates on producer retry (§9)
    "linger.ms":         5,       // wait up to 5ms to batch records (throughput)
    "compression.type":  "zstd",
})
defer p.Close()

topic := "orders"
// The KEY decides the partition → all events for one order stay ordered (§4, §10).
err := p.Produce(&kafka.Message{
    TopicPartition: kafka.TopicPartition{Topic: &topic, Partition: kafka.PartitionAny},
    Key:            []byte("order-42"),
    Value:          []byte(`{"event":"placed","amount":1999}`),
}, nil)
if err != nil { log.Fatal(err) }

// Producing is async + batched; Flush blocks until queued messages are delivered.
p.Flush(5000)

// Delivery reports tell you the final partition/offset (or an error) per message:
//   for e := range p.Events() { if m, ok := e.(*kafka.Message); ok { ... } }

# Using confluent-kafka (librdkafka). kafka-python / aiokafka are alternatives.
from confluent_kafka import Producer

p = Producer({
    "bootstrap.servers": "localhost:9092",
    "acks": "all",                 # durability: leader + in-sync replicas (§5)
    "enable.idempotence": True,    # no duplicates on producer retry (§9)
    "linger.ms": 5,                # batch window for throughput
    "compression.type": "zstd",
})

def on_delivery(err, msg):         # async callback: final partition/offset or error
    if err:
        print("delivery failed:", err)
    else:
        print(f"delivered to {msg.topic()}[{msg.partition()}]@{msg.offset()}")

# The KEY decides the partition → all events for one order stay ordered (§4, §10).
p.produce(
    "orders",
    key=b"order-42",
    value=b'{"event":"placed","amount":1999}',
    on_delivery=on_delivery,
)
p.flush()                          # block until queued messages are delivered

A consumer reads records from partitions. The powerful abstraction is the consumer group: a set of consumers sharing a group ID that cooperatively divide a topic's partitions among themselves so each partition is read by exactly one consumer in the group. Add consumers to the group and Kafka rebalances, reassigning partitions to spread the load. This is how you scale consumption horizontally — and it's the mechanism behind both work-sharing and broadcast.

c, _ := kafka.NewConsumer(&kafka.ConfigMap{
    "bootstrap.servers": "localhost:9092",
    "group.id":          "billing",        // the consumer GROUP — scale by adding instances
    "auto.offset.reset": "earliest",       // where to start if no committed offset (§8)
    "enable.auto.commit": false,           // we'll commit manually for at-least-once (§9)
})
defer c.Close()
c.Subscribe("orders", nil)                 // Kafka assigns this instance some partitions

for {
    msg, err := c.ReadMessage(-1)          // blocks for the next record
    if err != nil { continue }

    process(msg.Value)                     // do the work FIRST...
    c.CommitMessage(msg)                   // ...THEN commit the offset (at-least-once, §9)
}

from confluent_kafka import Consumer

c = Consumer({
    "bootstrap.servers": "localhost:9092",
    "group.id": "billing",                 # the consumer GROUP — scale by adding instances
    "auto.offset.reset": "earliest",       # where to start if no committed offset (§8)
    "enable.auto.commit": False,           # commit manually for at-least-once (§9)
})
c.subscribe(["orders"])                    # Kafka assigns this instance some partitions

try:
    while True:
        msg = c.poll(1.0)                  # wait up to 1s for the next record
        if msg is None:
            continue
        if msg.error():
            print("error:", msg.error()); continue

        process(msg.value())               # do the work FIRST...
        c.commit(msg)                      # ...THEN commit the offset (at-least-once, §9)
finally:
    c.close()                              # leave the group cleanly → triggers a rebalance

An offset is a record's position in its partition (§3). The consumer's job is to track which offset it has processed up to — its committed offset — so that on restart or rebalance it resumes from the right place instead of re-reading everything or skipping records. Kafka stores these committed offsets in an internal topic (__consumer_offsets), keyed by group + partition. The gap between the latest offset in a partition and a group's committed offset is consumer lag — the single most important health metric for a consumer (§19).

When you commit decides your guarantee

The choice between auto-commit (Kafka periodically commits in the background) and manual commit (you commit explicitly) is really a choice about delivery semantics (§9). The ordering of "process the message" versus "commit the offset" is everything: commit after processing and a crash in between means you'll re-process (at-least-once); commit before processing and a crash means you'll skip (at-most-once). Auto-commit on a timer can do either depending on timing, which is why correctness-sensitive consumers commit manually after the work is done (as in §7's code).

The question every distributed messaging system must answer: if things fail and retry, how many times might a message be delivered? There are three possible guarantees, and Kafka can give you any of them depending on configuration — so you must choose consciously based on what your processing can tolerate.

-- The consumer may see the same event twice on retry/rebalance.
-- Make the write idempotent so a duplicate is a no-op:

-- (a) dedupe by a unique event id
INSERT INTO processed_events (event_id) VALUES ($1)
ON CONFLICT (event_id) DO NOTHING;        -- second delivery inserts nothing
-- ...only do the side effect if the insert affected a row.

-- (b) or make the effect itself idempotent — upsert the end state, don't increment
INSERT INTO account_balance (account_id, balance) VALUES ($1, $2)
ON CONFLICT (account_id) DO UPDATE SET balance = EXCLUDED.balance;
-- applying the same "balance = 100" twice lands on the same state.

One sentence to memorize: Kafka guarantees ordering within a partition, and nowhere else. Records in partition P are delivered to a consumer in the exact offset order they were written. Across partitions of the same topic, there is no global ordering — records interleave arbitrarily. This isn't a limitation to work around so much as the direct price of partition-level parallelism (§4): you can have total ordering, or you can have horizontal scale, but not both at once for the same stream.

Since records are retained after reading (§2), Kafka needs rules for when to remove them — otherwise the log grows forever. There are two cleanup policies, and they answer different needs. Time/size retention (the default) keeps records for a window — say 7 days or 50 GB per partition — then deletes the oldest. Log compaction is cleverer: it keeps the latest record for each key and garbage-collects older versions of that key, turning the log into a continuously-updated snapshot of "the current value per key."

Kafka itself treats record values as opaque bytes — it neither knows nor cares what's inside. That freedom is a trap at scale: if producers and consumers don't agree on the payload's shape, a producer changing a field silently breaks every downstream consumer (the same contract-drift danger the gRPC and serialization chapters warned about, now spread across many independent teams). The fix is a schema for each topic plus a Schema Registry that stores schemas, assigns them IDs, and enforces compatibility as they evolve.

Formats & compatibility

The common serialization formats are Avro (the traditional Kafka pairing, compact with a rich schema-evolution story), Protobuf (the same IDL as the gRPC chapter, increasingly popular), and JSON Schema (readable, larger). The registry enforces a compatibility mode — typically backward compatibility, meaning new consumers can read old data: you may add optional fields and remove ones with defaults, but you may not rename or retype a field. This is precisely the additive-change discipline from the gRPC chapter (§04 there: field numbers, never reuse) and REST versioning, enforced automatically by a server.

Kafka is the substrate for event-driven architecture (EDA), where services communicate by emitting and reacting to events rather than calling each other directly. Instead of the order service commanding the email, inventory, and analytics services (tight coupling, the order service must know them all), it simply announces "OrderPlaced" to a topic, and any interested service subscribes. The producer doesn't know — or care — who consumes. This is the deep payoff of the log's decoupling (§1): you add a new consumer without touching the producer.

Two advanced patterns the log enables, often confused, frequently paired. Event sourcing stores state as the sequence of events that produced it rather than just the current snapshot: instead of a row balance = 100, you keep Deposited 60, Deposited 50, Withdrew 10 — and derive the balance by replaying them. Kafka's immutable, ordered, retained log (§3) is a natural event store. CQRS (Command Query Responsibility Segregation) separates the write model from one or more read models optimized for querying.

The benefits are real: a complete audit trail (every change is a recorded fact — gold for finance and compliance), the ability to reconstruct state at any past point by replaying to an offset, and multiple specialized read models each rebuildable from the same log (compaction — §11 — provides the snapshot to avoid replaying from the dawn of time).

A subtle, important problem in event-driven systems: when a service must both update its database and publish an event, doing them as two separate operations is unsafe. If the DB commit succeeds but the Kafka publish fails (or vice versa), your systems diverge — the order exists but no "OrderPlaced" event fired, or an event fired for an order that rolled back. This is the dual-write problem, and you can't fix it by reordering the two writes — there's always a crash window between them.

The outbox pattern solves it with a single local transaction: write your business change and an "outbox" row describing the event into the same database transaction — so they commit atomically. A separate relay process then reads new outbox rows and publishes them to Kafka (retrying until success, at-least-once — hence consumers stay idempotent, §9). The modern, low-friction way to run that relay is Change Data Capture (CDC): a tool like Debezium tails the database's transaction log and streams committed changes straight into Kafka — no application polling required.

BEGIN;

-- 1) the actual business change
INSERT INTO orders (id, customer_id, amount, status)
VALUES ('order-42', 'cust-7', 1999, 'PLACED');

-- 2) the event to publish, written in the SAME transaction → atomic with (1)
INSERT INTO outbox (id, aggregate, event_type, payload, created_at)
VALUES (gen_random_uuid(), 'order-42', 'OrderPlaced',
        '{"id":"order-42","amount":1999}', now());

COMMIT;   -- both succeed together, or neither does — no dual-write gap

-- A relay (or Debezium CDC on the transaction log) then reads new `outbox`
-- rows and produces them to Kafka, marking each published. Retries are safe
-- because downstream consumers are idempotent (§9).

So far consumers read events one at a time. Stream processing is the discipline of computing continuously over the stream — filtering, transforming, aggregating, joining — producing new derived streams. Instead of "read event, do a thing," you express a standing query like "a 5-minute rolling count of failed payments per merchant" that updates forever as events arrive. Kafka's own library for this is Kafka Streams (JVM); ksqlDB offers a SQL-like layer, and Apache Flink is the heavyweight for large, complex jobs.

Returning to the §2 distinction with enough context to decide well. This is the question that sends people to the task-queues chapter and back: do I want Kafka or a message queue (RabbitMQ, SQS, the Celery/Sidekiq world)? They overlap enough to confuse and differ enough to matter.

Most production Kafka pain shows up on the consumer side. These are the failure modes that bite real teams — worth recognizing before they page you at 2am.

Running Kafka well comes down to watching a few signals and sizing a few things correctly. You don't need to memorize every metric, but you must know the ones that predict trouble.

Sizing decisions you actually make

• Partition count — sets max consumer parallelism and throughput (§4, §7). Estimate from target throughput and peak consumer count; err slightly high, but not wildly (each partition has overhead, and thousands per broker hurt). Hard to change later, so think ahead (§20).
• Replication factor — usually 3 in production: tolerates one broker loss with room to spare. Pair with min.insync.replicas=2 and acks=all (§5).
• Retention — how long events live (and thus disk usage and replay window). Set per topic by how far back consumers might need to read.

Pulling the manual into design judgment. Kafka is a powerful, sharp tool; used where it fits it's transformative, used as a generic message bus it's a liability. A few principles separate good Kafka designs from painful ones.

Design principles

• Model topics around events/facts, not commands or RPC. A topic is a stream of "things that happened" (PaymentSettled), consumed by whoever cares — not a way to tell a specific service what to do. If you find yourself wanting request/response, you want gRPC/REST, not Kafka.
• Choose the partition key by your ordering boundary. Key by the entity whose events must stay ordered (account, order, user) — but at a granularity that still spreads load, to avoid hot partitions (§10, §18). The key is the single most consequential per-topic decision.
• Assume at-least-once; build idempotent consumers. Don't chase end-to-end exactly-once across external systems (§9). Dedupe by event ID or use upserts so duplicates are harmless — cheaper and more robust.
• Govern schemas from day one. Use the Schema Registry with backward compatibility (§12) so independent teams can evolve safely. A topic's schema is a contract.
• Use the outbox for DB-and-event writes. Never dual-write; make the event atomic with the data and relay it (§15).
• Size partitions for peak up front. Repartitioning breaks key ordering (§4), so plan for growth rather than reacting to it.

The whole manual compressed to what you reach for under pressure.