r/apachekafka • • Mar 18 '25

Apache Kafka 4.0 released 🎉

201 Upvotes

Quoting from the release blog:

Apache Kafka 4.0 is a significant milestone, marking the first major release to operate entirely without Apache ZooKeeper¼. By running in KRaft mode by default, Kafka simplifies deployment and management, eliminating the complexity of maintaining a separate ZooKeeper ensemble. This change significantly reduces operational overhead, enhances scalability, and streamlines administrative tasks. We want to take this as an opportunity to express our gratitude to the ZooKeeper community and say thank you! ZooKeeper was the backbone of Kafka for more than 10 years, and it did serve Kafka very well. Kafka would most likely not be what it is today without it. We don’t take this for granted, and highly appreciate all of the hard work the community invested to build ZooKeeper. Thank you!

Kafka 4.0 also brings the general availability of KIP-848, introducing a powerful new consumer group protocol designed to dramatically improve rebalance performance. This optimization significantly reduces downtime and latency, enhancing the reliability and responsiveness of consumer groups, especially in large-scale deployments.

Additionally, we are excited to offer early access to Queues for Kafka (KIP-932), enabling Kafka to support traditional queue semantics directly. This feature extends Kafka’s versatility, making it an ideal messaging platform for a wider range of use cases, particularly those requiring point-to-point messaging patterns.


r/apachekafka • • Jun 17 '21

A new children's book / illustrated guide to Apache Kafka

180 Upvotes

hi all, just wanted to share a beginner-friendly resource that I've been working on for Apache Kafka. I spent 3 months illustrating this digital book, which I'm calling Gently Down the Stream: http://www.gentlydownthe.stream/

I know that normal tech books that span hundreds of pages are often intimidating for beginners (I wrote one of those too, btw: http://kafka-streams-book.com/). So I just wanted to create something a little simpler, more colorful, playful, and fun. Hope you enjoy it!


r/apachekafka • • Nov 16 '25

Blog The Floor Price of Kafka (in the cloud)

Post image
152 Upvotes

EDIT (Nov 25, 2025): I learned the Confluent BASIC tier used here is somewhat of an unfair comparison to the rest, because it is single AZ (99.95% availability)

I thought I'd share a recent calculation I did - here is the entry-level price of Kafka in the cloud.

Here are the assumptions I used:

  • must be some form of a managed service (not BYOC and not something you have to deploy yourself)

  • must use the major three clouds (obviously something like OVHcloud will be substantially cheaper)

  • 250 KiB/s of avg producer traffic

  • 750 KiB/s of avg consumer traffic (3x fanout)

  • 7 day data retention

  • 3x replication for availability and durability

  • KIP-392 not explicitly enabled

  • KIP-405 not explicitly enabled (some vendors enable it and abstract it away frmo you; others don't support it)

Confluent tops the chart as the cheapest entry-level Kafka.

Despite having a reputation of premium prices in this sub, at low scale they beat everybody. This is mainly because the first eCKU compute unit in their Basic multi-tenant offering comes for free.

Another reason they outperform is their usage-based pricing. As you can see from the chart, there is a wide difference in pricing between providers with up to 5x of a difference. I didn't even include the most expensive options of:

  • Instaclustr Kafka - ~$20k/yr

  • Heroku Kafka - ~$39k/yr đŸ€Ż

Some of these products (Instaclustr, Event Hubs, Heroku, Aiven) use a tiered pricing model, where for a certain price you buy X,Y,Z of CPU, RAM and Storage. This screws storage-heavy workloads like the 7-day one I used, because it forces them to overprovision compute. So in my analysis I picked a higher tier and overpaid for (unused) compute.

It's noteworthy that Kafka solves this problem by separating compute from storage via KIP-405, but these vendors either aren't running Kafka (e.g Event Hubs which simply provides a Kafka API translation layer), do not enable the feature in their budget plans (Aiven) or do not support the feature at all (Heroku).

Through this analysis I realized another critical gap: no free tier exists anywhere.

At best, some vendors offer time-based credits. Confluent has 30 days worth and Redpanda 14 days worth of credits.

It would be awesome if somebody offered a perpetually-free tier. Databases like Postgres are filled to the brim with high-quality free services (Supabase, Neon, even Aiven has one). These are awesome for hobbyist developers and students. I personally use Supabase's free tier and love it - it's my preferred way of running Postgres.

What are your thoughts on somebody offering a single-click free Kafka in the cloud? Would you use it, or do you think Kafka isn't a fit for hobby projects to begin with?


r/apachekafka • • 5d ago

Tool (AI) 📣 Announcing kafuck: Apache Kafka, rewritten in Brainfuck. Blazingly fast. Memory safe. Diskless.

Thumbnail
github.com
137 Upvotes

r/apachekafka • • Mar 18 '25

Blog A 2 minute overview of Apache Kafka 4.0, the past and the future

135 Upvotes

Apache Kafka 4.0 just released!

3.0 released in September 2021. It’s been exactly 3.5 years since then.

Here is a quick summary of the top features from 4.0, as well as a little retrospection and futurespection

1. KIP-848 (the new Consumer Group protocol) is GA

The new consumer group protocol is officially production-ready.

It completely overhauls consumer rebalances by:

  • reducing consumer disruption during rebalances - it removes the stop-the-world effect where all consumers had to pause when a new consumer came in (or any other reason for a rebalance)

  • moving the partition assignment logic from the clients to the coordinator broker

  • adding a push-based heartbeat model, where the broker pushes the new partition assignment info to the consumers as part of the heartbeat (previously, it was done by a complicated join group and sync group dance)

I have covered the protocol in greater detail, including a step-by-step video, in my blog here.

Noteworthy is that in 4.0, the feature is GA and enabled in the broker by default. The consumer client default is still the old one, though. To opt-in to it, the consumer needs to set group.protocol=consumer

2. KIP-932 (Queues for Kafka) is EA

Perhaps the hottest new feature (I see a ton of interest for it).

KIP-932 introduces a new type of consumer group - the Share Consumer - that gives you queue-like semantics:

  1. per-message acknowledgement/retries

  2. ability to have many consumers collaboratively share progress reading from the same partition (previously, only one consumer per consumer group could read a partition at any time)

This allows you to have a job queue with the extra Kafka benefits of:

  • no max queue depth

  • the ability to replay records

  • Kafka’s greater ecosystem

The way it basically works is that all the consumers read from all of the partitions - there is no sticky mapping.

These queues have at least once semantics - i.e. a message can be read twice (or thrice). There is also no order guarantee.

I’ve also blogged about it (with rich picture examples).

3. Goodbye ZooKeeper

After some faithful 14 years of service (not without its issues, of course), ZooKeeper is officially gone from Apache Kafka.

KRaft (KIP-500) completely replaces it. It’s been production ready since October 2022 (Kafka 3.3), and going forward, you have no choice but to use it :) The good news is that it appears very stable. Despite some complaints about earlier versions, Confluent recently blogged about how they were able to migrate all of their cloud fleet (thousands of clusters) to KRaft without any downtime.

Others

  • the MirrorMaker1 code is removed (it was deprecated in 3.0)

  • The Transaction Protocol is strengthened

  • KRaft is strengthened via Pre-Vote

  • Java 8 support is removed for the whole project

  • Log4j was updated to v2

  • The log message format config (message.format.version) and versions v0 and v1 are finally deleted

Retrospection

A major release is a rare event, worthy of celebration and retrospection. It prompted me to look back at the previous major releases. I did a longer overview in my blog, but I wanted to call out perhaps the most important metric going up - number of contributors:

  1. Kafka 1.0 (Nov 2017) had 108 contributors

  2. Kafka 2.0 (July 2018) had 131 contributors

  3. Kafka 3.0 (September 2021) had 141 contributors

  4. Kafka 4.0 (March 2025) had 175 contributors

The trend shows a strong growth in community and network effect. It’s very promising to see, especially at a time where so many alternative Kafka systems have popped up and compete with the open source project.

The Future

Things have changed a lot since 2021 (Kafka 3.0). We’ve had the following major features go GA:

  • Tiered Storage (KIP-405)

  • KRaft (KIP-500)

  • The new consumer group protocol (KIP-848)

Looking forward at our next chapter - Apache Kafka 4.x - there are two major features already being worked on:

And other interesting features being discussed:

Kafka keeps evolving thanks to its incredible community. Special thanks to David Jacot for driving this milestone release and to the 175 contributors who made it happen!


r/apachekafka • • Oct 02 '25

Question The kafka book by Gwen Shapiro

Post image
133 Upvotes

I have started reading this book this week,

is it worth it?


r/apachekafka • • Oct 18 '25

Question Kafka's 60% problem

124 Upvotes

I recently blogged that Kafka has a problem - and it’s not the one most people point to.

Kafka was built for big data, but the majority use it for small data. I believe this is probably the costliest mismatch in modern data streaming.

Consider a few facts:

- A 2023 Redpanda report shows that 60% of surveyed Kafka clusters are sub-1 MB/s.

- Our own 4,000+ cluster fleet at Aiven shows 50% of clusters are below 10 MB/s ingest.

- My conversations with industry experts confirm it: most clusters are not “big data.”

Let’s make the 60% problem concrete: 1 MB/s is 86 GB/day. With 2.5 KB events, that’s ~390 msg/s. A typical e-commerce flow—say 5 orders/sec—is 12.5 KB/s. To reach even just 1 MB/s (roughly 10× below the median), you’d need ~80× more growth.

Most businesses simply aren’t big data. So why not just run PostgreSQL, or a one-broker Kafka? Because a single node can’t offer high availability or durability. If the disk dies—you lose data; if the node dies—you lose availability. A distributed system is the right answer for today’s workloads, but Kafka has an Achilles’ heel: a high entry threshold. You need 3 brokers, 3 controllers, a schema registry, and maybe even a Connect cluster—to do what? Push a few kilobytes? Additionally you need a Frankenstack of UIs, scripts and sidecars, spending weeks just to make the cluster work as advertised.

I’ve been in the industry for 11 years, and getting a production-ready Kafka costs basically the same as when I started out—a five- to six-figure annual spend once infra + people are counted. Managed offerings have lowered the barrier to entry, but they get really expensive really fast as you grow, essentially shifting those startup costs down the line.

I strongly believe the way forward for Apache Kafka is topic mixes—i.e., tri-node topics vs. 3AZ topics vs. Diskless topics—and, in the future, other goodies like lakehouse in the same cluster, so engineers, execs, and other teams have the right topic for the right deployment. The community doesn't yet solve for the tiniest single-node footprints. If you truly don’t need coordination or HA, Kafka isn’t there (yet). At Aiven, we’re cooking a path for that tier as well - but can we have the Open Source Apache Kafka API on S3, minus all the complexity?

But i'm not here to market Aiven and I may be wrong!

So I'm here to ask: how do we solve Kafka's 60% Problem?


r/apachekafka • • Sep 19 '26

Tool After 10 years working with Kafka, I built the data tool I wanted

Post image
123 Upvotes

I built Kafka Streamyard to make Kafka data investigations and manual testing less repetitive. It preserves context between topics and environments, keeps workflow-related topics and filters organised, combines related topics in Live Views, and supports reusable test-data samples and development playbooks. It is a local-first desktop app for macOS, Windows, and Linux.

I've been developing Kafka applications for about 10 years, and I started an early version of Kafka Streamyard about six or seven years ago. It is an Electron application built with React, Websockets and a local Java backend.

After working closely with developers, testers, and DevOps engineers, I found that the Kafka UIs we used were generally designed around administration. They did not fit our day-to-day data investigation and manual testing workflows particularly well. Lots of individually small inconveniences added up to a significant amount of repeated work.

One example is simply finding all the topics involved in a workflow. In a busy cluster, I have spent far too much time searching for the same groups of topics again and again. Kafka Streamyard lets me create and save reusable topic-name filters using wildcards, nested Boolean logic, and some Kafka metadata. I can tag each filter for the connections where it is relevant and quickly switch between different parts of a workflow during testing, investigations, or team workshops. AI assistance is also available when I need help building a filter.

I have also put a lot of effort into preserving context. I can investigate something in UAT, switch to Dev, and later return with my selected topic, results, filter, and even the selected record still available. That context is restored after restarting the application too. Optional Cloud Sync lets me reuse filters, custom columns, Live Views, Data Samples, and Playbooks when moving between my laptop and a VM, while Kafka connections and credentials stay local.

Live Views are probably my favourite part when working with a team. They combine related topics into one chronological view, so everyone can watch a process move from a UI to a database, through CDC, and into the topics that trigger the rest of a workflow. Each topic can use its own custom columns, making it easier to confirm that the important fields were set correctly at each stage. A Live View can also include a session filter, message counts, and an optional flowchart showing the process and data flow.

For repetitive manual testing, Data Samples can keep related keys, headers, and messages together. Session variables can carry a correlation ID across several samples, increment sequence values, generate timestamps or identifiers, and reset everything for the next test. Samples can then be sent in a controlled order, in bursts, or with delays between them.

Message filters use SL, or Streamyard Language, which is yet another Java-like filter language to learn. That is why I added autocomplete and optional AI assistance. SL can filter the message, key, headers, and Kafka metadata, including fields inside JSON, XML, and embedded JSON. The AI-generated filter remains visible and editable, so it can be checked before use. Users provide their own API key and can choose from several common providers.

Playbooks help me reset a narrow part of a development or local Docker environment into a known state before testing again. They are line-based scripts that can create, configure, clear, delete, or copy topics and import test data. They support variables, repeat blocks, topic patterns, validation, Dry Run, a builder wizard, and AI assistance. They are intended for development and testing, not as a replacement for production CI/CD or Infrastructure as Code.

There are also precise offset and time controls for historical investigations, background Live Feeds for catching infrequent records, automatic format detection, Schema Registry support, a hex editor for raw bytes, custom calculated columns, and CSV export when somebody needs a report or evidence from a set of topics.

Kafka Streamyard is not intended to replace a typical multi-user Kafka management platform. It includes some familiar management features, but its main focus is working directly with Kafka data during investigations, testing sessions, support work, and team workshops.

The website is still a work in progress, but you can try Kafka Streamyard on macOS with M1 or newer, Windows, or Linux: https://www.kafka-streamyard.com. The application works consistently across the three platforms and updates automatically when a new version is available.

If you try it, I would genuinely appreciate feedback and criticism. I am particularly interested in hearing how other people investigate Kafka data, prepare test scenarios, and follow workflows across topics. If there is a specific problem you would like it to solve, let me know. I enjoy turning useful feature requests around quickly.

**AI disclosure:** This is mostly in my own words, but I used AI to correct my grammar, spelling, structure, and word order.


r/apachekafka • • Aug 25 '25

Blog Top 5 largest Kafka deployments

Post image
98 Upvotes

These are the largest Kafka deployments I’ve found numbers for. I’m aware of other large deployments (datadog, twitter) but have not been able to find publicly accessible numbers about their scale


r/apachekafka • • Nov 10 '25

Tool I’ve built an interactive simulation of Kafka Streams’ architecture!

89 Upvotes

This tool makes the inner workings of Kafka Streams tangible — see messages flow through the simulation, change partition and thread counts, play with the throughput and see how it impacts message processing.

A great way to deepen your understanding or explain the architecture to your team.

Try it here: https://kafkastreamsfieldguide.com/tools/interactive-architecture


r/apachekafka • • May 09 '25

AMA We’re the co-founders of WarpStream. Ask Us Anything.

79 Upvotes

Hey, everyone. We are Richie Artoul and Ryan Worl, co-founders and engineers at WarpStream, a stateless, drop-in replacement for Apache Kafka that uses S3-compatible object storage. We're doing an AMA to answer any engineering or other questions you have about WarpStream; why and how it was created, how it works, our product roadmap, etc.

Before WarpStream, we both worked at Datadog and collaborated on building Husky, a distributed event storage system.

Per AMA and this subreddit's specific rules:

  • We’re not here to sell WarpStream. The point of this AMA is to answer engineering and technical questions about WarpStream.

  • We’re happy to chat about WarpStream pricing if you have specific questions, but we’re not going to get into any mud-slinging with comparisons to other vendors 😁.

The AMA will be on Wednesday, May 14, at 10:30 a.m. Eastern Time (United States). You can RSVP and submit questions ahead of time.

See here for our AMA selfie:

Thank you!


r/apachekafka • • Nov 17 '20

🎉[BIG NEWS] 🎉 Redpanda is now FREE & Source Available

Thumbnail
vectorized.io
77 Upvotes

r/apachekafka • • Dec 08 '24

Blog Exploring Apache Kafka Internals and Codebase

74 Upvotes

Hey all,

I've recently begun exploring the Kafka codebase and wanted to share some of my insights. I wrote a blog post to share some of my learnings so far and would love to hear about others' experiences working with the codebase. Here's what I've written so far. Any feedback or thoughts are appreciated.

Entrypoint: kafka-server-start.sh and kafka.Kafka

A natural starting point is kafka-server-start.sh (the script used to spin up a broker) which fundamentally invokes kafka-run-class.sh to run kafka.Kafka class.

kafka-run-class.sh, at its core, is nothing other than a wrapper around the java command supplemented with all those nice Kafka options.

exec "$JAVA" $KAFKA_HEAP_OPTS $KAFKA_JVM_PERFORMANCE_OPTS $KAFKA_GC_LOG_OPTS $KAFKA_JMX_OPTS $KAFKA_LOG4J_CMD_OPTS -cp "$CLASSPATH" $KAFKA_OPTS "$@"

And the entrypoint to the magic powering modern data streaming? The following main method situated in Kafka.scala i.e. kafka.Kafka

  try {
      val serverProps = getPropsFromArgs(args)
      val server = buildServer(serverProps)

      // ... omitted ....

      // attach shutdown handler to catch terminating signals as well as normal termination
      Exit.addShutdownHook("kafka-shutdown-hook", () => {
        try server.shutdown()
        catch {
          // ... omitted ....
        }
      })

      try server.startup()
      catch {
       // ... omitted ....
      }
      server.awaitShutdown()
    }
    // ... omitted ....

That’s it. Parse the properties, build the server, register a shutdown hook, and then start up the server.

The first time I looked at this, it felt like peeking behind the curtain. At the end of the day, the whole magic that is Kafka is just a normal JVM program. But a magnificent one. It’s incredible that this astonishing piece of engineering is open source, ready to be explored and experimented with.

And one more fun bit: buildServer is defined just above main. This where the timeline splits between Zookeeper and KRaft.

    val config = KafkaConfig.fromProps(props, doLog = false)
    if (config.requiresZookeeper) {
      new KafkaServer(
        config,
        Time.SYSTEM,
        threadNamePrefix = None,
        enableForwarding = enableApiForwarding(config)
      )
    } else {
      new KafkaRaftServer(
        config,
        Time.SYSTEM,
      )
    }

How is config.requiresZookeeper determined? it is simply a result of the presence of the process.roles property in the configuration, which is only present in the Kraft installation.

Zookepeer connection

Kafka has historically relied on Zookeeper for cluster metadata and coordination. This, of course, has changed with the famous KIP-500, which outlined the transition of metadata management into Kafka itself by using Raft (a well-known consensus algorithm designed to manage a replicated log across a distributed system, also used by Kubernetes). This new approach is called KRaft (who doesn't love mac & cheese?).

If you are unfamiliar with Zookeeper, think of it as the place where the Kafka cluster (multiple brokers/servers) stores the shared state of the cluster (e.g., topics, leaders, ACLs, ISR, etc.). It is a remote, filesystem-like entity that stores data. One interesting functionality Zookeeper offers is Watcher callbacks. Whenever the value of the data changes, all subscribed Zookeeper clients (brokers, in this case) are notified of the change. For example, when a new topic is created, all brokers, which are subscribed to the /brokers/topics Znode (Zookeeper’s equivalent of a directory/file), are alerted to the change in topics and act accordingly.

Why the move? The KIP goes into detail, but the main points are:

  1. Zookeeper has its own way of doing things (security, monitoring, API, etc) on top of Kafka's, this results in a operational overhead (I need to manage two distinct components) but also a cognitive one (I need to know about Zookeeper to work with Kafka).

  2. The Kafka Controller has to load the full state (topics, partitions, etc) from Zookeeper over the network. Beyond a certain threshold (~200k partitions), this became a scalability bottleneck for Kafka.

  3. A love of mac & cheese.

Anyway, all that fun aside, it is amazing how simple and elegant the Kafka codebase interacts and leverages Zookeeper. The journey starts in initZkClient function inside the server.startup() mentioned in the previous section.

  private def initZkClient(time: Time): Unit = {
    info(s"Connecting to zookeeper on ${config.zkConnect}")
    _zkClient = KafkaZkClient.createZkClient("Kafka server", time, config, zkClientConfig)
    _zkClient.createTopLevelPaths()
  }

KafkaZkClient is essentially a wrapper around the Zookeeper java client that offers Kafka-specific operations. CreateTopLevelPaths ensures all the configuration exist so they can hold Kafka's metadata. Notably:

    BrokerIdsZNode.path, // /brokers/ids
    TopicsZNode.path, // /brokers/topics
    IsrChangeNotificationZNode.path, // /isr_change_notification

One simple example of Zookeeper use is createTopicWithAssignment which is used by the topic creation command. It has the following line:

zkClient.setOrCreateEntityConfigs(ConfigType.TOPIC, topic, config)

which creates the topic Znode with its configuration.

Other data is also stored in Zookeeper and a lot of clever things are implemented. Ultimately, Kafka is just a Zookeeper client that uses its hierarchical filesystem to store metadata such as topics and broker information in Znodes and registers watchers to be notified of changes.

Networking: SocketServer, Acceptor, Processor, Handler

A fascinating aspect of the Kafka codebase is how it handles networking. At its core, Kafka is about processing a massive number of Fetch and Produce requests efficiently.

I like to think about it from its basic building blocks. Kafka builds on top of java.nio.Channels. Much like goroutines, multiple channels or requests can be handled in a non-blocking manner within a single thread. A sockechannel listens of on a TCP port, multiple channels/requests registered with a selector which polls continuously waiting for connections to be accepted or data to be read.

As explained in the Primer section, Kafka has its own TCP protocol that brokers and clients (consumers, produces) use to communicate with each other. A broker can have multiple listeners (PLAINTEXT, SSL, SASL_SSL), each with its own TCP port. This is managed by the SockerServer which is instantiated in the KafkaServer.startup method. Part of documentation for the SocketServer reads :

 *    - Handles requests from clients and other brokers in the cluster.
 *    - The threading model is
 *      1 Acceptor thread per listener, that handles new connections.
 *      It is possible to configure multiple data-planes by specifying multiple "," separated endpoints for "listeners" in KafkaConfig.
 *      Acceptor has N Processor threads that each have their own selector and read requests from sockets
 *      M Handler threads that handle requests and produce responses back to the processor threads for writing.

This sums it up well. Each Acceptor thread listens on a socket and accepts new requests. Here is the part where the listening starts:

  val socketAddress = if (Utils.isBlank(host)) {
      new InetSocketAddress(port)
    } else {
      new InetSocketAddress(host, port)
    }
    val serverChannel = socketServer.socketFactory.openServerSocket(
      endPoint.listenerName.value(),
      socketAddress,
      listenBacklogSize, // `socket.listen.backlog.size` property which determines the number of pending connections
      recvBufferSize)   // `socket.receive.buffer.bytes` property which determines the size of SO_RCVBUF (size of the socket's receive buffer)
    info(s"Awaiting socket connections on ${socketAddress.getHostString}:${serverChannel.socket.getLocalPort}.")

Each Acceptor thread is paired with num.network.threads processor thread.

 override def configure(configs: util.Map[String, _]): Unit = {
    addProcessors(configs.get(SocketServerConfigs.NUM_NETWORK_THREADS_CONFIG).asInstanceOf[Int])
  }

The Acceptor thread's run method is beautifully concise. It accepts new connections and closes throttled ones:

  override def run(): Unit = {
    serverChannel.register(nioSelector, SelectionKey.OP_ACCEPT)
    try {
      while (shouldRun.get()) {
        try {
          acceptNewConnections()
          closeThrottledConnections()
        }
        catch {
          // omitted
        }
      }
    } finally {
      closeAll()
    }
  }

acceptNewConnections TCP accepts the connect then assigns it to one the acceptor's Processor threads in a round-robin manner. Each Processor has a newConnections queue.

private val newConnections = new ArrayBlockingQueue[SocketChannel](connectionQueueSize)

it is an ArrayBlockingQueue which is a java.util.concurrent thread-safe, FIFO queue.

The Processor's accept method can add a new request from the Acceptor thread if there is enough space in the queue. If all processors' queues are full, we block until a spot clears up.

The Processor registers new connections with its Selector, which is a instance of org.apache.kafka.common.network.Selector, a custom Kafka nioSelector to handle non-blocking multi-connection networking (sending and receiving data across multiple requests without blocking). Each connection is uniquely identified using a ConnectionId

localHost + ":" + localPort + "-" + remoteHost + ":" + remotePort + "-" + processorId + "-" + connectionIndex

The Processor continuously polls the Selector which is waiting for the receive to complete (data sent by the client is ready to be read), then once it is, the Processor's processCompletedReceives processes (validates and authenticates) the request. The Acceptor and Processors share a reference to RequestChannel. It is actually shared with other Acceptor and Processor threads from other listeners. This RequestChannel object is a central place through which all requests and responses transit. It is actually the way cross-thread settings such as queued.max.requests (max number of requests across all network threads) is enforced. Once the Processor has authenticated and validated it, it passes it to the requestChannel's queue.

Enter a new component: the Handler. KafkaRequestHandler takes over from the Processor, handling requests based on their type (e.g., Fetch, Produce).

A pool of num.io.threads handlers is instantiated during KafkaServer.startup, with each handler having access to the request queue via the requestChannel in the SocketServer.

        dataPlaneRequestHandlerPool = new KafkaRequestHandlerPool(config.brokerId, socketServer.dataPlaneRequestChannel, dataPlaneRequestProcessor, time,
          config.numIoThreads, s"${DataPlaneAcceptor.MetricPrefix}RequestHandlerAvgIdlePercent", DataPlaneAcceptor.ThreadPrefix)

Once handled, responses are queued and sent back to the client by the processor.

That's just a glimpse of the happy path of a simple request. A lot of complexity is still hiding but I hope this short explanation give a sense of what is going on.


r/apachekafka • • Oct 03 '22

Blog Apache Kafka 3.3 has been released (including KRaft is Production Ready 🎉)

71 Upvotes

Download: https://kafka.apache.org/downloads

Release notes: https://archive.apache.org/dist/kafka/3.3.0/RELEASE_NOTES.html

Blog: https://blogs.apache.org/kafka/entry/what-rsquo-s-new-in

Video: https://www.youtube.com/watch?v=EUwwNnVyc4c

Some of the notable changes:

NB the version released is 3.3.1. Per the Apache Kafka site:

A significant bug was found in the 3.3.0 release after artifacts were pushed to Apache and Maven central but prior to the release announcement. As a result, the decision was made to not announce 3.3.0 and instead release 3.3.1 with the fix. It is recommended that 3.3.0 not be used.


r/apachekafka • • Sep 26 '24

Blog Kafka Has Reached a Turning Point

69 Upvotes

https://medium.com/p/649bd18b967f

Kafka will inevitably become 10x cheaper. It's time to dream big and create even more.


r/apachekafka • • Oct 27 '25

Tool My Core Insights dashboard for Kafka Streams

Post image
67 Upvotes

I’ve built a Core Insights dashboard for Kafka Streams!

This Prometheus-based Grafana dashboard brings together the metrics that actually matter: processing latency, throughput, state store health, and thread utilization. One view to spot issues before they become incidents.
It shows you processing latency, message flow per topic, tracks RocksDB activity, breaks down exactly how each thread spends its time (processing, punctuating, committing, or polling), and more


Explore all its features and learn how to interpret and use the dashboard: https://kafkastreamsfieldguide.com/articles/kafka-streams-grafana-dashboard


r/apachekafka • • Dec 13 '24

Blog Cheaper Kafka? Check Again.

65 Upvotes

I see the narrative repeated all the time on this subreddit - WarpStream is a cheaper Apache Kafka.

Today I expose this to be false.

The problem is that people repeat marketing narratives without doing a deep dive investigation into how true they are.

WarpStream does have an innovative design tha reduces the main drivers that rack up Kafka costs (network, storage and instances indirectly).

And they have a calculator that allegedly proves this by comparing the costs.

But the problem is that it’s extremely inaccurate, to the point of suspicion. Despite claiming in multiple places that it goes “out of its way” to model realistic parameters, that its objective is “to not skew the results in WarpStream’s favor” and that that it makes “a ton” of assumptions in Kafka’s favor
 it seems to do the exact opposite.

I posted a 30-minute read about this in my newsletter.

Some of the things are nuanced, but let me attempt to summarize it here.

The WarpStream cost comparison calculator:

  • inaccurately inflates Kafka costs by 3.5x to begin with

    • its instances are 5x larger cost-wise than what they should be - a 16 vCPU / 122 GiB r4.4xlarge VM to handle 3.7 MiB/s of producer traffic

    • uses 4x more expensive SSDs rather than HDDs, again to handle just 3.7 MiB/s of producer traffic per broker. (Kafka was made to run on HDDs)

    • uses too much spare disk capacity for large deployments, which not only racks up said expensive storage, but also forces you to deploy more of those overpriced instances to accommodate disk

  • had the WarpStream price increase by 2.2x post the Confluent acquisition, but the percentage savings against Kafka changed by just -1% for the same calculator input.

    • This must mean that Kafka’s cost increased 2.2x too.

  • the calculator’s compression ratio changed, and due to the way it works - it increased Kafka’s costs by 25% while keeping the WarpStream cost the same (for the same input)

    • The calculator counter-intuitively lets you configure the pre-compression throughput, which allows it to subtly change the underlying post-compression values to higher ones. This positions Kafka disfavorably, because it increases the dimension Kafka is billed on but keeps the WarpStream dimension the same. (WarpStream is billed on the uncompressed data)

    • Due to their architectural differences, Kafka costs already grow at a faster rate than WarpStream, so the higher the Kafka throughput, the more WarpStream saves you.

    • This pre-compression thing is a gotcha that I and everybody else I talked to fell for - it’s just easy to see a big throughput number and assume that’s what you’re comparing against. “5 GiB/s for so cheap?” (when in fact it’s 1 GiB/s)

  • The calculator was then further changed to deploy 3x as many instances, account for 2x the replication networking cost and charge 2x more for storage. Since the calculator is in Javascript ran on the browser, I reviewed the diff. These changes were done by

    • introducing an obvious bug that 2x the replication network cost (literallly a * 2 in the code)

    • deploy 10% more free disk capacity without updating the documented assumptions which still referenced the old number (apart from paying for more expensive unused SSD space, this has the costly side-effect of deploying more of the expensive instances)

    • increasing the EBS storage costs by 25% by hardcoding a higher volume price, quoted “for simplicity”

The end result?

It tells you that a 1 GiB/s Kafka deployment costs $12.12M a year, when it should be at most $4.06M under my calculations.

With optimizations enabled (KIP-392 and KIP-405), I think it should be $2M a year.

So it inflates the Kafka cost by a factor of 3-6x.

And with that that inflated number it tells you that WarpStream is cheaper than Kafka.

Under my calculations - it’s not cheaper in two of the three clouds:

  • AWS - WarpStream is 32% cheaper

  • GCP - Apache Kafka is 21% cheaper

  • Azure - Apache Kafka is 77% cheaper

Now, I acknowledge that the personnel cost is not accounted for (so-called TCO).

That’s a separate topic in of itself. But the claim was that WarpStream is 10x cheaper without even accounting for the operational cost.

Further - the production tiers (the ones that have SLAs) actually don’t have public pricing - so it’s probably more expensive to run in production that the calculator shows you.

I don’t mean to say that the product isn’t without its merits. It is a simpler model. It is innovative.

But it would be much better if we were transparent about open source Kafka's pricing and not disparage it.

</rant>

I wrote a lot more about this in my long-form blog.

It’s a 30-minute read with the full story. If you feel like it, set aside a moment this Christmas time, snuggle up with a hot cocoa/coffee/tea and read it.

I’ll announce in a proper post later, but I’m also releasing a free Apache Kafka cost calculator so you can calculate your Apache Kafka costs more accurately yourself.

I’ve been heads down developing this for the past two months and can attest first-hard how easy it is to make mistakes regarding your Kafka deployment costs and setup. (and I’ve worked on Kafka in the cloud for 6 years)


r/apachekafka • • Apr 28 '21

How Kafka works visualization tool, from Software Mill

Thumbnail
softwaremill.com
65 Upvotes

r/apachekafka • • Nov 30 '25

Tool KafkIO 2.1.0 released (macOS, Windows and Linux)

Post image
60 Upvotes

KafkIO 2.1.0 was just released, grab it here: https://www.kafkio.com. There has been a lot of new features and improvements added since our last post.

To those new to KafkIO: it's a client-side native Kafka GUI, for engineers and administrators (macOS, Windows and Linux), easy to setup. It handles management of brokers, topics, offsets, dumping/searching topics, consumers, schemas, ACLs, connectors and their lifecycles, ksqlDB with an advanced KSQL editor, and contains a bunch of utilities and productivity features. It handles all the usual security mechanisms and various proxy configurations necessary. It tries to make working with Kafka easy and enjoyable.

If you want to get away from Docker, web servers, complex configuration, and get back to reliable multi-tabbed desktop UIs, this is the tool for you.


r/apachekafka • • Oct 10 '24

Blog The Numbers behind Uber's Kafka (& rest of their data infra stack)

60 Upvotes

I thought this would be interesting to the audience here.

Uber is well known for its scale in the industry.

Here are the latest numbers I compiled from a plethora of official sources:

  • Apache Kafka:

    • 138 million messages a second

    • 89GB/s (7.7 Petabytes a day)

    • 38 clusters

This is 2024 data.

They use it for service-to-service communication, mobile app notifications, general plumbing of data into HDFS and sorts, and general short-term durable storage.

It's kind of insane how much data is moving through there - this might be the largest Kafka deployment in the world.

Do you have any guesses as to how they're managing to collect so much data off of just taxis and food orders? They have always been known to collect a lot of data afaik.

As for Kafka - the closest other deployment I know of is NewRelic's with 60GB/s across 35 clusters (2023 data). I wonder what DataDog's scale is.

Anyway. The rest of Uber's data infra stack is interesting enough to share too:

  • Apache Pinot:

    • 170k+ peak queries per second

    • 1m+ events a second

    • 800+ nodes

  • Apache Flink:

    • 4000 jobs

    • processing 75 GB/s

  • Presto:

    • 500k+ queries a day

    • reading 90PB a day

    • 12k nodes over 20 clusters

  • Apache Spark:

    • 400k+ apps ran every day

    • 10k+ nodes that use >95% of analytics’ compute resources in Uber

    • processing hundreds of petabytes a day

  • HDFS:

    • Exabytes of data

    • 150k peak requests per second

    • tens of clusters, 11k+ nodes

  • Apache Hive:

    • 2 million queries a day

    • 500k+ tables

They leverage a Lambda Architecture that separates it into two stacks - a real time infrastructure and batch infrastructure.

Presto is then used to bridge the gap between both, allowing users to write SQL to query and join data across all stores, as well as even create and deploy jobs to production!

A lot of thought has been put behind this data infrastructure, particularly driven by their complex requirements which grow in opposite directions:

  1. 1. Scaling Data - total incoming data volume is growing at an exponential rate

    1. Replication factor & several geo regions copy data.

    2. Can’t afford to regress on data freshness, e2e latency & availability while growing.

  2. Scaling Use Cases - new use cases arise from various verticals & groups, each with competing requirements.

  3. Scaling Users - the diverse users fall on a big spectrum of technical skills. (some none, some a lot)

If you're in particular interested about more of Uber's infra, including nice illustrations and use cases for each technology, I covered it in my 2-minute-read newsletter where I concisely write interesting Kafka/Big Data content.


r/apachekafka • • Aug 10 '26

Blog AI Generated Projects

Post image
59 Upvotes

I posted this over on LI last week because I've seen a lot of solo AI-assisted projects here recently. Cool to see people building their own tools but I think the best part of OSS is people actually collaborating and building together rather than working on our own thing in a corner.

To avoid just being a grump though I thought I'd shout out just a few great human-made projects and blogs I like, many of which I came across here:

  • ShadowTraffic brilliant tool from Michael Drogalis for rapidly simulating production traffic, he's been building this in the open and with the community.

  • Fresha Data Engineering the Fresha team are doing some ground-breaking stuff, a shout out to Nicoleta in particular who's done some great work on Fluss in prod.

  • Leo Delmouly's Medium, Leo has done a great series on Kafka + Iceberg which helped me understand why this is such a hard problem.

  • Michael Maison's monthly Kafka digest which is my go to resource for understanding what's coming up for the Apache Kafka project.


r/apachekafka • • Sep 04 '25

Blog Apache Kafka 4.1 Released đŸ”„

57 Upvotes

Here's to another release 🎉

The top noteworthy features in my opinion are:

KIP-932 Queues go from EA -> Preview

KIP-932 graduated from Early Access to Preview. It is still not recommended for Production, but now has a stable API. It bumped its share.version=1 and is ready to develop and test against.

As a reminder, KIP-932 is a much anticipated feature which introduces first-class support for queue-like semantics through Share Consumer Groups. It offers the ability for many consumers to read from the same partition out of order with individual message acknowledgements and retries.

We're now one step closer to it being production-ready!

Unfortunately the Kafka project has not yet clearly defined what Early Access nor Preview mean, although there is an under discussion KIP for that.

KIP-1071 - Stream Groups

Not to be confused with share groups, this is a KIP that introduces a Kafka Streams rebalance protocol. It piggybacks on the new consumer group protocol (KIP-848), extending it for Kafka Streams via a dedicated API for rebalancing.

This should help make Kafka Streams app scale smoother, make their coordination simpler and aid in debugging.

Others

  • KIP-877 introduces a standardized API to register metrics for all pluggable interfaces in Kafka. It captures things like the CreateTopicPolicy, the producer's Partitioner, Connect's Task, and many others.

  • KIP-891 adds support for running multiple plugin versions in Kafka Connect. This makes upgrades & downgrades way easier, as well as helps consolidate Connect clusters

  • KIP-1050 simplifies the error handling for Transactional Producers. It adds 4 clear categories of exceptions - retriable, abortable, app-recoverable and invalid-config. It also clears up the documentation. This should lead to more robust third-party clients, and generally make it easier to write robust apps against the API.

  • KIP-1139 adds support for the jwt_bearer OAuth 2.0 grant type (RFC 7523). It's much more secure because it doesn't use a static plaintext client secret and is a lot easier to rotate hence can be made to expire more quickly.


Thanks to Mickael Maison for driving the release, and to the 167 contributors that took part in shipping code for this release.


r/apachekafka • • Sep 14 '25

Tool End-to-End Data Lineage with Kafka, Flink, Spark, and Iceberg using OpenLineage

56 Upvotes

I've created a complete, hands-on tutorial that shows how to capture and visualize data lineage from the source all the way through to downstream analytics. The project follows data from a single Apache Kafka topic as it branches into multiple parallel pipelines, with the entire journey visualized in Marquez.

The guide walks through a modern, production-style stack:

  • Apache Kafka - Using Kafka Connect with a custom OpenLineage SMT for both source and S3 sink connectors.

  • Apache Flink - Showcasing two OpenLineage integration patterns:

    • DataStream API for real-time analytics.

    • Table API for data integration jobs.

  • Apache Iceberg - Ingesting streaming data from Flink into a modern lakehouse table.

  • Apache Spark - Running a batch aggregation job that consumes from the Iceberg table, completing the lineage graph.

This project demonstrates how to build a holistic view of your pipelines, helping answer questions like:

  • Which applications are consuming this topic?

  • What's the downstream impact if the topic schema changes?

The entire setup is fully containerized, making it easy to spin up and explore.

Want to see it in action? The full source code and a detailed walkthrough are available on GitHub.


r/apachekafka • • Mar 31 '21

Apache Kafka Made Simple: A First Glimpse of a Kafka Without ZooKeeper

Thumbnail
confluent.io
56 Upvotes