Streaming Data Platforms

How log-based streaming platforms like Kafka work — topics, partitions, offsets, replication, retention and compaction — and the real-world problems they create (rebalancing storms, hot partitions, consumer lag, duplicate and out-of-order delivery, exactly-once pitfalls, schema evolution) and how engineers fix them.

1. 01Kafka and Streaming PlatformsThe distributed commit log: topics, partitions, offsets, durability, and the Kafka-like ecosystem.15 sections
2. 02Streaming Problems and How They Are FixedThe real failures of running Kafka-style platforms at scale — lag, rebalances, hot partitions, duplicates — and the fixes.15 sections