Why every Kafka tutorial starts in the wrong place

Open the official quickstart and you will be sending messages in about ten minutes. Start the broker. Create a topic. Run the console producer in one terminal, the console consumer in another, type “hello” and watch it appear.

It works every time, and it teaches almost nothing.

Ten minutes later you can operate Kafka. You still cannot design anything with it, and — this is the part that stings six months in — you don’t yet know that those are different.

The API is the easy part

Kafka’s surface area is genuinely small. You append records to the end of a log and you read records from a position in that log. That is the whole thing. Producers append. Consumers read. There is not much API to learn because there is not much API.

So a tutorial that teaches the API finishes quickly, feels successful, and leaves the actual difficulty entirely untouched. The hard part of Kafka was never the method names. It is the model — and specifically these four ideas, which arrive nowhere in most quickstarts:

• The log is the source of truth, not a delivery mechanism. Messages are not consumed in the sense of being removed. They sit there until a retention policy deletes them, whether anybody read them or not.
• A partition is both the unit of ordering and the unit of parallelism, and those two purposes are in direct conflict. Ordering is guaranteed inside a partition and nowhere else. More partitions means more consumers can work at once and less of your data is ordered.
• The broker does not track what you have read. You commit your own offset. Where you commit it, relative to doing the work, is what decides whether you get at-least-once or at-most-once delivery — and nobody hands you exactly-once for free.
• Rebalancing is your problem. Consumers join and leave a group, partitions get reassigned, and the code you wrote while thinking of one consumer now runs in five, in an order you do not control.

None of those are advanced topics. Every one of them will decide whether your first Kafka system works. All four are routinely below the fold.

The question the tutorial skips

Here is what a quickstart cannot ask, because it has already assumed the answer: why are you using this instead of a queue or a database table?

That is not a rhetorical question. It has a real answer, and the answer is Kafka. You use a log rather than a queue when more than one independent thing wants to read the same stream, at different speeds, keeping its own position — including things that do not exist yet and will want to read from the beginning next year. Replay and multiple independent readers. That is the trade the whole design exists to make, and it is why you accept the operational weight.

If you do not need that, you wanted a queue, and RabbitMQ or SQS will make your life shorter and happier.

Because the quickstart never poses the question, a predictable set of systems get built. Kafka used as a task queue with exactly one consumer group, which is strictly harder than SQS and buys nothing. A topic created with one partition because ordering seemed obviously desirable, and then a team discovering they cannot add a second consumer without giving that up. Retention set to seven days by whoever copied the config, on a topic somebody is about to treat as a system of record.

Each of those is a design decision made by default, by somebody who did not know a decision was being made.

Why it happens

Not laziness. The quickstart is doing its actual job well.

It is written for a reader who has already decided to use Kafka and needs to get it running — and it is written by people who understand the system so thoroughly that the model is invisible to them. It is onboarding documentation. It optimises for time-to-first-message, which is a metric about the document.

The problem is that it is the first result for “learn Kafka”, so it gets read by people who are not that reader at all.

What starting in the right place looks like

Not with a broker. With a shape of problem:

You have a stream of events. More than one system wants to react to them, at different speeds. One of those systems does not exist yet, and when it is built it will want to see everything from the beginning.

Sit with that for a page and the log stops being an implementation detail and becomes the obvious answer. Then partitions arrive as the consequence of that log outgrowing one machine — and the loss of total ordering lands as a price you agreed to pay, rather than a footnote you will rediscover during an incident. Then consumer groups, because now you have to decide who reads what. Then offsets, because now you have to decide what “done” means.

The CLI goes last. It takes ten minutes and it always did.

And even that has three versions

Here is the part no tutorial can solve, however well written.

Someone arriving from RabbitMQ needs one sentence more than anything else in the document: the broker does not track what you have read. Their entire mental model of a message broker is built on acknowledgement and removal, and until that assumption is explicitly broken, every subsequent page is being quietly mistranslated.

Someone arriving from Postgres needs a different sentence: this is a table you may only append to and read in order, and that restriction is precisely what buys the throughput.

Someone who has done neither needs the event-streaming problem first, slowly, because they have no existing model to correct — they need one built.

Three readers. Three genuinely different first chapters. One tutorial, which must pick one and will therefore be wrong for two.

That constraint is not a failure of care by the people writing tutorials. It is arithmetic. Writing once and publishing to everybody means writing for an average reader, and on the one question that matters most here — what do you already believe about how message systems work — there is no average reader to write for.

Which is a decent argument for a course that gets to ask first.