Kafka and Streaming Platforms

A commit log is the single idea that everything else on this day is built from. It is an ordered file that you only ever append to — new records go on the end, and nothing already written is edited or moved. Reading it means starting at some position and walking forward. That is the whole data structure: a growing, numbered list of records that is never rewritten.

To see why that is powerful, contrast it with a plain queue — a middle store that hands each message to one worker and then deletes it once the worker confirms success. A queue is a to-do inbox: an item is taken, done, and thrown away, so it can be read exactly once and then it is gone. A log is the opposite: reading does not consume or remove anything. Each reader simply remembers how far it has read (a number called an offset, defined properly in the next section), and many independent readers can each walk the same log at their own pace. A brand-new reader can start at the beginning and replay the entire history.

That one difference — retain instead of delete on read — is what turns a messaging system into a streaming platform. Because the data stays, the log becomes a replayable buffer of history that several unrelated systems can each read in full, rewind, and re-process. Apache Kafka is the dominant open-source implementation of this idea, so the rest of the day uses Kafka's vocabulary; the concepts carry over to the Kafka-like systems covered later.

Rule of thumb: if the same records must feed several independent systems, be replayable, or preserve per-key order, you want a log; if each item is a one-off job to do once and forget, a plain queue is simpler.

Before any moving parts, fix the three nouns that name where data lives in Kafka. A topic is a named stream of records — the logical category you publish to and read from, like orders or page-views. A topic is not one physical log, though; it is split into several partitions, each of which is an independent append-only commit log. An offset is the position of a record within one partition: partition 0 has records at offset 0, 1, 2, and so on, counting up forever.

Partitions are the single most important concept on this day, because a partition is both the unit of parallelism and the unit of ordering. Records within one partition are strictly ordered by offset; records in different partitions have no defined order relative to each other. Splitting a topic into N partitions lets N machines store and serve it in parallel, but it means the topic as a whole has no single global order.

An offset is just a number that a reader tracks, so "where am I" is cheap to store and cheap to move. Written out for one partition, with no arrows, it is simply positions filling up on the right:

partition 2:  [0][1][2][3][4][5]   next append lands at offset 6
              oldest        newest
reader remembers: "next offset to read = 4"

Because a reader only stores a small integer per partition, rewinding is trivial: set the offset back to 0 and read the whole partition again, or set it to a saved value to reprocess from a point in time. That property — seek to any offset — is what makes a log replayable where a queue is not.

Rule of thumb: choose the partition count deliberately, because it fixes both how much parallelism the topic can ever have and which records are ordered relative to each other.

Now the three actors that move data. A producer is any program that appends records to a topic. A consumer is any program that reads records from a topic's partitions. Between them sits the log, so a producer never waits on a consumer and a consumer never blocks a producer — they are decoupled through the stored partitions.

The concept that makes reading scale is the consumer group: a named set of consumers that cooperate to read a topic once between them. Kafka divides the topic's partitions among the group's members so that, within one group, each partition is read by exactly one consumer at a time. That rule is what preserves per-partition order — no two members ever touch the same partition simultaneously. Different groups are fully independent: each group gets every record and keeps its own offsets, so the email-service group and the analytics group both read all of orders without affecting each other.

Two consequences follow from "one partition, one consumer per group." First, the partition count is the parallelism ceiling for a group: a group can usefully run at most as many consumers as there are partitions, and any extra consumers sit idle. If you might need eight parallel workers later, create eight partitions now, because raising the count afterward reshuffles which records land where. Second, when a consumer joins or dies, the group rebalances — Kafka reassigns partitions across the surviving members, processing pauses briefly while the group agrees on who owns what, then each member resumes from its partition's last saved offset.

Rule of thumb: use one consumer group per independent system that needs the whole stream, and size the partition count for the peak number of workers a single group will ever need.

So far the log has been abstract; here is the machinery that actually stores it. A broker is one Kafka server — a single machine (or process) that holds partitions on its disk and serves reads and writes for them. A Kafka cluster is a group of brokers working together; the partitions of a topic are spread across the brokers so no one machine holds everything, which is how the cluster scales storage and throughput.

Something has to track the shared facts every broker must agree on: which topics exist, how many partitions each has, which broker holds which partition, and who leads each one. That shared bookkeeping is cluster metadata, and one elected broker, the controller, is responsible for maintaining it and reacting when a broker fails (reassigning leadership, updating the metadata).

Where that metadata is stored is the one piece of Kafka history worth knowing. Older Kafka kept metadata in ZooKeeper — a separate cluster whose only job was storing small shared configuration and coordinating who is leader. Modern Kafka replaced it with KRaft (Kafka Raft), a mode where the metadata lives inside Kafka itself as its own internal commit log managed by the controller, so no separate ZooKeeper system is needed. Practically: KRaft means one fewer thing to run and simpler operations; when you read older docs mentioning ZooKeeper, KRaft is the current answer.

Rule of thumb: treat brokers as interchangeable storage-and-serving nodes, and remember that exactly one controller keeps the metadata straight — via KRaft in current versions, via ZooKeeper in older ones.

Ordering is the promise that records are read in the sequence they were written. On a log that promise holds only within a partition, so the practical question is: which records land in the same partition? The answer is the key. Every record can carry an optional key (an order id, a user id, an account id), and the producer decides a record's partition by hashing that key.

The routing rule is simple arithmetic — the hash of the key, taken modulo the number of partitions, with no arrows involved:

partition = hash(key) % number_of_partitions

Because the same key always hashes to the same partition, all records for account-123 land on one partition and are therefore read in strict order relative to each other. Records for account-777 may land on a different partition and flow in parallel. This gives per-key ordering: everything about one entity stays ordered, while unrelated entities are processed concurrently. There is deliberately no global order across the whole topic, because guaranteeing that would require funneling every record through a single partition and a single consumer, capping throughput at what one worker can do.

The catch is key choice. If one key is far more active than the rest — say every event carries the same country=US key — its partition becomes a hot partition that one consumer cannot keep up with, while other partitions sit nearly idle. A good key has many distinct values and no single dominant one. If you send records with no key, Kafka spreads them across partitions for balance, but you give up per-key ordering.

Rule of thumb: pick a key so that exactly the records which must stay in order share it, and check that no single key value dominates the traffic.

Storing a partition on one broker is fragile — if that disk dies, the data is gone. Kafka's answer is replication: each partition is copied onto several brokers. One copy is the leader (the copy that handles all reads and writes for that partition) and the others are followers (copies on other brokers that continuously fetch from the leader to stay current). The replication factor is how many copies exist in total; three is the common choice, meaning one leader plus two followers.

The key concept for durability is the in-sync replica (ISR) set: the subset of replicas that are fully caught up with the leader right now. A follower that has fetched everything up to the leader's latest offset is in the ISR; one that has fallen behind (slow disk, network lag) drops out until it catches up. When the leader fails, the controller promotes a replica from the ISR to be the new leader — which is safe, because an in-sync replica has all the acknowledged data.

How much durability you get is a knob the producer sets with acks, and it trades directly against latency:

acks	Producer waits for	Durability	Latency	Failure it survives
0	Nothing (fire and forget)	Lowest — record can vanish	Lowest	None; loses in-flight data on any hiccup
1	Leader has written it	Medium — safe unless leader dies before replicating	Low	Leader stays up
all	Every in-sync replica has it	Highest — survives leader loss	Highest	Any single broker (with RF ≥ 3)

acks=all alone is not enough, because if the ISR has shrunk to just the leader, "all in-sync replicas" means only the leader, and you are back to acks=1 durability without noticing. The guard is min.insync.replicas: the minimum ISR size the leader requires before it accepts an acks=all write. Set it to 2 with replication factor 3, and a write is refused unless at least two copies will hold it — so no single broker loss can lose an acknowledged record. Written as producer/broker settings:

acks=all
min.insync.replicas=2
replication.factor=3

Rule of thumb: for data that matters, use replication factor 3, acks=all, and min.insync.replicas=2 — accept the extra write latency in exchange for surviving any single broker failure without data loss.

A log keeps records after they are read, but it cannot keep them forever — disks are finite. Kafka offers two different policies for deciding what stays, and they serve opposite purposes. The first is retention: keep every record for a while, then delete the oldest. The second is compaction: keep, per key, only the most recent record, deleting older values for the same key.

Retention treats the partition as a time-bounded replayable buffer. You configure it by time (keep 7 days) or by size (keep the last 50 GB per partition), and once a record ages out or the partition exceeds the size, the oldest segments are deleted. During the window, any consumer can rewind and re-read. This is the default and the right choice for event streams — clicks, orders, metrics — where each record is a distinct fact and you want history for a bounded period.

Compaction treats the partition as a changelog that can be collapsed into a table. For a compacted topic keyed by, say, user-id, Kafka guarantees to retain at least the latest record for every key, discarding superseded older values in the background. Read the whole compacted topic and you reconstruct the current value of every key — effectively a snapshot of the latest state. This is how a stream of updates becomes a queryable table.

The two are about what a record means. If each record is an independent event you might replay, use time/size retention. If each record is a new value for a keyed entity and you only care about the latest, use compaction — it is exactly how a Kafka topic can back a lookup table or feed the current state of a cache.

	Time/size retention	Log compaction
Keeps	All records within a window	Latest record per key
Deletes	Oldest, by age or total size	Superseded older values for a key
Reading the whole topic gives	A history of events	The current value of every key
Mental model	Replayable event buffer	A table / changelog
Use for	Events, metrics, audit trails	State snapshots, config, materialized tables

Rule of thumb: use retention when records are events you want to replay for a bounded time, and compaction when records are updates to keyed state and you want the latest value per key to survive indefinitely.

"Delivery semantics" is the promise about how many times a record reaches its consumer, and Kafka's honest default is at-least-once: a record is never lost, but it may be processed more than once. Understanding why — and what Kafka's "exactly-once" actually covers — prevents both data loss and double-counting.

Two mechanics create the at-least-once behavior. On the write side, a producer that does not get an ack (maybe the ack was lost on the network) retries, which can append the same record twice. On the read side, a consumer records its progress by committing offsets — telling Kafka "I have processed up to offset N in this partition" — and the timing of that commit sets the guarantee:

• Commit after processing (process, then commit): a crash between the two means the offset was not saved, so the record is read again on restart — at-least-once, duplicates possible, nothing lost. This is the safe default.
• Commit before processing (or auto-commit on a timer, which can fire before your work finishes): a crash means the offset moved past work that never completed — at-most-once, records can be silently skipped.

Kafka can give you exactly-once within Kafka through two features. The idempotent producer tags each record so the broker discards a retry that would duplicate an already-stored record, removing producer-side double-appends. Transactions let a producer write to several partitions and commit the consumer's offsets as one atomic unit, so a read-process-write pipeline that stays inside Kafka either fully happens or fully does not. This is often called effectively-once.

The crucial limit: this covers effects inside Kafka. The moment your consumer writes to an outside system — a database, a payment API, an email — Kafka's transaction cannot reach it, so the end-to-end guarantee is still at-least-once and you must make the consumer idempotent (safe to run twice for the same record, e.g. by deduping on a stable id). Exactly-once effect end to end is something you engineer at the consumer, not a switch on the broker.

Rule of thumb: default to at-least-once (process, then commit offsets) plus idempotent consumers; reach for Kafka transactions only for Kafka-to-Kafka pipelines, and never assume "exactly-once" extends to systems outside Kafka.

A fair question is how appending everything to a file and keeping it can possibly be fast — surely a database with indexes is cleverer. The answer is that Kafka is fast because it is a dumb append-only log, and four mechanics compound to make it handle millions of records per second on ordinary hardware.

• Sequential disk I/O. Appending to the end of a file and reading forward are sequential access patterns. Even spinning disks, and especially SSDs, are dramatically faster at sequential I/O than at the random seeks a general database does — there is no index to update in place, just bytes written in order.
• The page cache. The page cache is the operating system's use of free RAM to hold recently accessed file data. Because consumers usually read records soon after they are written, those bytes are still in RAM, so most reads never touch the disk at all — Kafka leans on the OS cache instead of maintaining its own.
• Zero-copy. Zero-copy is a technique where the OS sends file data straight from the page cache to the network socket without copying it through the application's memory first. Since Kafka mostly ships stored bytes to consumers unchanged, it hands the OS the "send this file range to this socket" instruction and skips several memory copies and context switches.
• Batching and compression. Producers group many records into one batch before sending, and Kafka can compress the whole batch (gzip, snappy, lz4, zstd). Fewer, larger, compressed network and disk operations amortize per-record overhead and cut the bytes moved.

None of these is exotic; the point is that the log's simplicity is what unlocks them. A structure that only appends and reads forward is exactly the structure the disk, the OS cache, and the network card are all optimized for.

Rule of thumb: expect a log to sustain far higher throughput than a comparable database because sequential I/O, the page cache, zero-copy, and batching all reward its append-only shape — and preserve that by producing in batches rather than one record per request.

Kafka the broker moves records; the value in practice comes from the tools built around it that get data in, transform it in flight, and keep its shape sane. Three components matter most, and knowing what each does tells you what you should never hand-write.

Kafka Connect is a framework for moving data between Kafka and other systems without writing custom code. A source connector pulls data into Kafka (from a database, a file, an API); a sink connector pushes data out of Kafka (into a warehouse, a search index, object storage). Its headline use is change data capture (CDC) — reading a database's change log so every insert, update, and delete becomes a Kafka record, turning an ordinary database into a live event stream without touching the application that owns it.

Kafka Streams and ksqlDB are for stream processing — transforming, filtering, joining, and aggregating records as they flow, rather than in a nightly batch. Kafka Streams is a Java library you embed in your own service; ksqlDB lets you express the same continuous computations in SQL-like queries. Both consume from topics and produce derived topics — a running count, a joined stream, a filtered view — that other consumers read like any other topic.

The Schema Registry governs the shape of records. Producers and consumers agree on a schema (a formal description of a record's fields and types), commonly written in Avro (a compact binary serialization format that stores data alongside its schema) or Protobuf. The registry stores these schemas centrally and enforces compatibility rules, so a producer cannot roll out a change that would break existing consumers — a new field must be added in a way old readers tolerate. It is what keeps a topic readable as the data evolves over months.

Rule of thumb: use Connect (and CDC) instead of writing bespoke import/export jobs, use Kafka Streams or ksqlDB for in-flight transformation instead of batch reprocessing, and put a Schema Registry in front of any topic that more than one team reads.

Kafka is the default log, but it is not the only one, and several systems implement the same commit-log idea with different trade-offs. The point here is not to memorize each product but to know what distinguishes it, so you can reach for the right one. Many of these speak the Kafka protocol (the wire format Kafka clients use), meaning existing Kafka tools work against them unchanged.

• Apache Pulsar separates serving from storage: brokers are stateless and the actual log lives in a separate storage layer (Apache BookKeeper) as segments. This makes scaling storage and scaling serving independent, and Pulsar leans into multi-tenancy — many teams sharing one cluster with isolation — plus built-in geo-replication and a queue mode alongside the log.
• Redpanda is a from-scratch reimplementation of the Kafka API in C++ with no JVM and no ZooKeeper — a single self-contained binary that is Kafka-API compatible. It targets lower and more predictable latency and simpler operations, so existing Kafka clients connect to it without changes.
• AWS Kinesis and Google Cloud Pub/Sub are managed cloud streaming services: the provider runs the cluster, and you pay per throughput. You give up some control and portability in exchange for no brokers to operate, which suits teams that want streaming without owning the infrastructure.
• NATS JetStream adds persistence and replay to NATS, a very lightweight messaging system. It is small, fast, and simple to run — a good fit for edge, IoT, and internal service messaging where Kafka's operational weight is overkill.
• RabbitMQ Streams is a log-style feature added to RabbitMQ (traditionally a queue broker), giving replay and multiple independent readers to teams already running RabbitMQ who need a stream without adopting a whole new platform.
• Redis Streams is a log data type inside Redis, with consumer groups and offsets. It is ideal when you already run Redis and need a modest, low-latency stream, but it is bounded by memory and is not built for Kafka-scale retention.

System	Distinguishing trait	Reach for it when
Apache Kafka	The de-facto standard log; huge ecosystem	You want the mainstream choice and the richest tooling
Apache Pulsar	Segmented storage split from serving; strong multi-tenancy	Many tenants share a cluster; storage and serving must scale apart
Redpanda	C++, Kafka-API compatible, no JVM/ZooKeeper	You want Kafka semantics with lower latency and simpler ops
AWS Kinesis / Google Pub/Sub	Fully managed by the cloud provider	You want streaming without running any brokers
NATS JetStream	Very lightweight, simple to operate	Edge/IoT or internal messaging; Kafka is too heavy
RabbitMQ Streams	Log feature bolted onto a queue broker	You already run RabbitMQ and need replay
Redis Streams	A stream type inside Redis, memory-bound	You already run Redis and need a small, fast stream

Rule of thumb: default to Kafka (or a Kafka-API-compatible option like Redpanda) for the ecosystem, choose a managed service to avoid operating brokers, and drop down to a lightweight option like NATS JetStream or Redis Streams when Kafka's scale and operational cost are more than the problem needs.

A log is powerful and also heavy, so the sharpest skill is knowing when not to reach for it. Three tools sit near each other — a plain queue, a log, and a database — and each is right for a different shape of problem. Here is a one-line recap so this stands alone: a queue hands each message to one worker and deletes it (do-once jobs), a log retains records so many independent readers can each read and replay them (event streams), and a database stores current state you query and update in place.

The log wins specifically when: several independent consumers each need the whole stream (fan-out to email, shipping, analytics from one topic); you need to replay history to rebuild a downstream store or backfill a new service; per-key ordering matters at high throughput; or you are doing stream processing (CDC, running aggregates). It is the wrong tool when each message is an isolated job with no replay need — a queue is simpler and cheaper — or when you need to look up and mutate current state by key, which is a database's job. A log is an append-only history, not a place to update a row.

Rule of thumb: reach for a log when the same events must feed multiple systems or be replayable; use a plain queue for one-off work distribution and a database for queryable, updatable current state.

A log is not free to run, and most Kafka pain comes from a handful of predictable operational realities. Knowing them upfront is the difference between a platform that hums and one that pages people at night.

• Partitions are not free. Every partition costs open file handles, memory, and replication traffic on the brokers, and it is a unit the controller must track. Tens of thousands of partitions across a cluster slow metadata operations and lengthen recovery. Over-partitioning "just in case" is a real cost — size for peak need, not for imagination.
• Rebalancing storms. Recall that a group rebalances (reassigns partitions) whenever a member joins or leaves. If consumers are flapping — restarting, timing out, hitting long processing pauses that look like death — the group can rebalance repeatedly, and during each rebalance processing stops. A cluster stuck in a rebalance loop makes no progress; tuning session timeouts and using cooperative rebalancing calms it.
• Hot partitions. Because a record's partition comes from hash(key) % N, a skewed key (one value far more common than the rest) piles disproportionate traffic onto one partition. Its single consumer falls behind while others idle, and no amount of extra consumers helps — the fix is a better-distributed key or more partitions, not more workers.
• Consumer lag. Consumer lag is the gap between the newest offset in a partition and the offset a consumer group has committed — how far behind real time a reader is. Growing lag means consumers cannot keep up with producers, and it is the primary health signal for a streaming system. Monitoring and alerting on lag per group per partition is non-negotiable; without it, a slow consumer silently drifts hours behind before anyone notices.

The broader cost is operational: you are running a stateful, replicated, disk-backed distributed system with its own tuning, capacity planning, and upgrade dance. That is exactly why managed services and simpler alternatives exist — the platform earns its keep only when you genuinely need replay, fan-out, or high-throughput ordered streams.

Rule of thumb: monitor consumer lag as your top-line health metric, keep partition counts as low as your parallelism truly requires, and treat repeated rebalancing and hot partitions as design smells to fix at the key or capacity level.

This table collapses the day into concept-to-purpose pairs — the fast reference for what each Kafka piece is for.

Concept	What it is	What it buys you
Commit log	Append-only, ordered, retained record file	Replay and multi-reader access, unlike a delete-on-read queue
Topic	Named stream, split into partitions	The logical thing you produce to and consume from
Partition	One independent ordered sub-log	Unit of both parallelism and ordering
Offset	A record's position in a partition	Cheap, seekable progress marker → replay
Key	Field hashed to pick a partition	Per-key ordering; beware hot partitions
Producer / consumer	Writer / reader of a topic	Decoupled through the stored log
Consumer group	Consumers sharing a topic once between them	Scale reads; each group gets the full stream independently
Broker / controller	A server / the metadata-owning broker	Distributed storage; one keeper of cluster state
KRaft (vs ZooKeeper)	Metadata as Kafka's own internal log	One fewer system to run vs the old ZooKeeper dependency
Replication + ISR	Copies of a partition; the caught-up set	Survive broker loss; ISR is who can safely lead
acks + min.insync.replicas	Durability knob on writes	Trade latency for surviving failures without loss
Retention	Keep all records for a time/size window	A replayable buffer of events
Compaction	Keep latest record per key	A topic that acts as a table / changelog
At-least-once + idempotency	Default delivery + safe-to-repeat consumer	Exactly-once effect end to end
Idempotent producer + transactions	Dedup writes + atomic multi-partition writes	Effectively-once within Kafka
Sequential I/O, page cache, zero-copy, batching	Why the log is fast	High throughput on ordinary hardware
Connect / Streams / Schema Registry	Import-export / stream processing / schema governance	Get data in, transform in flight, keep shape sane
Consumer lag	Newest offset minus committed offset	The top-line health signal for a stream

• A commit log retains records instead of deleting them on read, so many independent consumers can each read and replay the same stream by tracking an offset — the core difference from a queue.
• A topic is split into partitions; a partition is both the unit of parallelism and the unit of ordering, and there is no global order across partitions.
• A record's partition is hash(key) % N, giving strict per-key ordering while unrelated keys flow in parallel — but a dominant key creates a hot partition.
• A consumer group reads a topic once between its members (one partition per consumer), so the partition count is the parallelism ceiling; groups are independent and each sees the whole stream.
• Brokers store partitions; one controller owns cluster metadata, held in KRaft (modern) rather than the old ZooKeeper dependency.
• Durability comes from replication plus the in-sync-replica set; acks=all with min.insync.replicas=2 and replication factor 3 survives any single broker loss without losing acknowledged data.
• Use time/size retention for replayable event streams and log compaction to keep the latest value per key, turning a topic into a table.
• Delivery is at-least-once by default (process, then commit offsets); idempotent producers and transactions give effectively-once within Kafka, but end-to-end still needs idempotent consumers.
• A log is fast because its append-only shape rewards sequential I/O, the OS page cache, zero-copy, and batching.
• Connect (and CDC) move data in and out, Kafka Streams/ksqlDB transform it in flight, and the Schema Registry keeps record shapes compatible as data evolves.
• Reach for a log when the same events feed many systems, need replay, or need per-key ordering at scale; use a plain queue for one-off jobs and a database for queryable current state.
• Partitions are not free, rebalancing storms and hot partitions are design smells, and consumer lag is the health metric to monitor above all others.