r/apachekafka • • 4h ago

Tool (AI) In Flight: an open-source console that shows what's stuck in your dead-letter queues as orders, customers and money

4 Upvotes

I am building this tool (ProvenMap's open-source project, Apache-2.0), and most of the code was written with Claude Code.

When a consumer fails, its events go to a dead-letter queue, and the dashboards show lag and offsets. Whoever owns the flow wants to know which orders are stuck, for which customers, worth how much. "In Flight" is for that person where "to learn Kafka first" is not required.

You point it at a cluster and it finds the flow's topics, consumer groups and dead-letter queues (by name, or by Kafka Connect's or Spring's headers on the records). It drafts a catalog from the records: which field is the amount and whether it's in cents, which is the order and which the customer. confirm it, and from then on it shows money at risk per consumer group and per customer, grouped by error.

All these can be configured through built-in mcp server

Welcome any comments.

Code: https://github.com/provenmap/in-flight

Its architecture as a board: https://provenmap.com/boards/paws/in-flight-the-money-moving-through-your-kafka


r/apachekafka • • 1d ago

Event Redpanda archived its GitHub repo

Thumbnail
github.com
50 Upvotes

r/apachekafka • • 20h ago

Blog Change Data Capture (CDC) Explained: Stop Dual Writes with Debezium + Kafka

Thumbnail
youtube.com
2 Upvotes

r/apachekafka • • 2d ago

Question What are you using as a desktop Kafka client now that Conduktor Desktop is gone?

6 Upvotes

Since Conduktor Desktop was discontinued, the options for just poking at topics locally seem kind of thin: Offset Explorer works but feels dated, Kafbat/kafka-ui is solid but I don't want to spin up a container just to peek at a topic, and kcat is great until you need Avro/Protobuf decoding.

Curious what everyone ended up with:

- What do you use day to day to inspect topics and messages?

- What's the most annoying part — Schema Registry decoding, searching/filtering messages, consumer lag, auth (SASL/IAM/mTLS), or being scared of touching prod?

- Would a lightweight native desktop app (read-only by default, Schema Registry-aware) be something you'd actually install, or are CLI + web UI good enough?


r/apachekafka • • 2d ago

Tool Building an experimental Go stream processor: what recovery failure would you test first?

2 Upvotes

A question about the project got straight to the difficult part: after a crash, where does the state come from?

kstream-go is an independent Go stream-processing core using Kafka Streams as a semantic reference.

The recovery design uses committed Kafka changelogs as the durable authority. The experimental Kafka adapters commit state changes, output, and input positions together. Recovery replays committed changes into task-owned stores.

An optional Pebble adapter provides persistent local caches. Validated local checkpoint reuse is still unimplemented. A local store surviving a restart does not, by itself, prove that it matches the committed Kafka state.

Other remaining gaps include broader consumer-group fault-tolerance qualification and heterogeneous repartition scheduling. Differential tests cover implemented scenarios against Kafka Streams 3.9.1; they do not establish full compatibility or production readiness.

Repo, examples, and validation boundaries: https://github.com/SoraDaibu/kstream-go

For people operating stateful Kafka applications: which failure would you test first before trusting this recovery design?

  • An ambiguous transaction outcome

  • Ownership loss during processing

  • A local cache-apply failure after the Kafka commit

  • Something else we’re overlooking

Concrete failure sequences or lessons from operating similar systems would be especially helpful.


r/apachekafka • • 5d ago

Tool (AI) 📣 Announcing kafuck: Apache Kafka, rewritten in Brainfuck. Blazingly fast. Memory safe. Diskless.

Thumbnail
github.com
137 Upvotes

r/apachekafka • • 5d ago

Event We're hosting a webinar around consumer group latency and kafka troubleshooting

Post image
2 Upvotes

Hi chums!

We're hosting a webinar around kafka troubleshooting live next week on November 18.

We'll be debugging a cluster live and talking through the issues that can occur when you get far more messages than expected!

Disclose : we're from lenses!

sign in here


r/apachekafka • • 7d ago

Blog bus: an opinionated Go layer over sarama with protobuf event contracts, typed handlers and built-in commit/retry policy

2 Upvotes

Hi all,

I've been building bus, a Go library on top of IBM/sarama. It packages the commit, retry and serialization decisions I kept re-implementing in every service into one place.

Event contracts

Events are protobuf messages. The topic and the partition key are declared in the .proto file via custom options, so the contract travels with the schema:

message OrderCreated {
  option (korableg.protobuf.event.topic) = "orders";
  string id = 1 [(korableg.protobuf.event.key) = true];
}

Several event types can share one topic. Each message carries its type in a header, and a handler only receives the type it's registered for.

Feedback from people who've run Kafka in production is very welcome, especially on the commit semantics and whether a built-in DLT is worth adding.


r/apachekafka • • 7d ago

Question How do you get third-party webhooks (Stripe, GitHub, Shopify) into Kafka today?

10 Upvotes

Hi all,

I'm curious how teams here handle incoming webhooks when the rest of their system runs on Kafka.

From what I've seen, most people end up writing a small receiver service that:

- verifies the provider's signature (different for every provider)

- deduplicates events, since providers sometimes send the same one twice

- buffers or retries when Kafka is slow or down

- produces the event to a topic

And then there's usually no easy way to see what arrived, what failed, or to replay an event.

A few questions:

  1. Do you run your own receiver, use a service (Hookdeck, Airbyte, something else), or skip Kafka for webhooks entirely?

  2. What's been the most annoying part: signatures, duplicates, lost events, debugging?

  3. Have you ever lost or double-processed a webhook event because of this?

  4. If a hosted service did this reliably and pushed events into your own Kafka cluster, would that be useful, or is it simple enough to keep in-house?

I'm thinking about building something in this space, so honest answers (including "not worth it") are very welcome. Thanks!


r/apachekafka • • 9d ago

Blog System Design - Apache Kafka Basics

Thumbnail
javarevisited.substack.com
12 Upvotes

r/apachekafka • • 9d ago

Tool I built an open-source tool for exploring Kafka compacted topics

3 Upvotes

Hi there! I've been working on an open-source project called KGazer, designed to make working with Kafka compacted topics a bit easier.

The idea came from a problem I've run into myself: sometimes you just want to find the current value of a particular key, see how it's changed over time or investigate its previous versions. Doing this often means writing a temporary Kafka consumer or putting together a few scripts.

I wanted to make that process simpler.

- KGazer provides a web interface where you can:

- Browse keys and search message values.

- Inspect the history of individual keys.

- Compare different message versions using JSON diffs.

- Visualise how values change over time.

- Monitor consumer groups and consumer lag.

It also supports multiple Kafka clusters, Avro and Protobuf deserialisation, and Kubernetes deployments.

Under the hood, it's built with Go and React, uses PostgreSQL for storage, and is released under the MIT licence.

GitHub: https://github.com/alfonsojimenez/kgazer

It's still an evolving project, so I'd genuinely appreciate feedback from anyone working with Kafka, particularly those running it in production.

I'm curious: how do you currently inspect compacted topics and investigate changes to individual keys? Is this something you find yourself writing scripts for or do you use another tool?

I'd be interested to hear what you'd find useful in a tool like this.


r/apachekafka • • 10d ago

Question Apache kafka ->recent interview questions

6 Upvotes

HI there, if anyone has attended interviews for BE + kafka for 5+ yoe, can anyone share the interview questions, situation based questions related to kafka?


r/apachekafka • • 11d ago

Tool Your Kafka consumer can just be an HTTP endpoint now: retries, DLQ and offsets handled for you

4 Upvotes

I work at wolveflake.io, this came out of our internal infra and we just open-sourced it (Apache-2.0).

We kept rewriting the same consumer boilerplate in every service: commit after processing, retry with backoff, send bad records to a DLQ, add metrics. Then a Python team needed it too and we did it all again.

So we pulled it into a small Go service. It runs the consumer groups and POSTs batches to your endpoint. Your handler returns SUCCESS / RETRY / FAILED per record, and WKS commits, retries, or writes to the DLQ. At-least-once, config hot-reloads, Grafana dashboard included.

So It's not a stream processor or a Connect replacement, just Kafka in, HTTP out. If your logic fits in a Redpanda Connect config, use that. This is for when your logic is already a service.

https://github.com/wolveflake/wolvekafka

v0.1.0, so it's early and the roadmap is wide open. What would you need before you'd actually try this? Or tell me why you'd never use it, that's useful too.


r/apachekafka • • 12d ago

Tool Kafkaesque: A lightweight Kafka monitoring dashboard, API, and CLI.

10 Upvotes

If you are using Kafka in your company, startup, or personal project, you need to try this tool: Kafkaesque

Monitoring Kafka is tedious. We either have to rely on the CLI or observability platforms. Built Kafkaesque to solve this problem: a dedicated, read-only Kafka monitoring tool.

Provides a web UI and CLI to inspect cluster health, brokers, topics, hourly message rates, consumer groups, and partition lag.

Synopsis & relevance to Apache Kafka

Kafkaesque connects directly to Apache Kafka clusters to collect and present operational metadata and monitoring information. It helps developers and engineering teams understand their Kafka cluster's health, identify consumer lag, inspect topic activity, and troubleshoot common operational issues without relying entirely on CLI commands or deploying a large observability stack.

It is read-only by design and does not modify Kafka topics, messages, or cluster configurations.

Stack: Go + PostgreSQL + HTMX

Single binary, easy to deploy, and read-only (safe by design).

Try it out: https://github.com/nikhil25803/kafkaesque/pkgs/container/kafkaesque

Installation and setup guide: https://github.com/nikhil25803/kafkaesque#run-the-web-ui-with-docker

The project is open source. Feedback, suggestions, issues, and contributions are always welcome!

Highlighted feature

ConcernDetails
Web monitoringReal-time Kafka cluster, broker, topic, partition, and consumer-group visibility with message-rate and consumer-lag metrics.
Kafka connectivitySupport for plaintext, TLS, mTLS, SASL authentication, and flexible broker address overrides.
Authentication and accessPostgreSQL-backed user management, sessions, built-in roles, custom RBAC, secure cookies, and API tokens.
Slack lag alertsConfigurable average or maximum partition-lag thresholds with Slack Webhook and Workflow Builder integrations.
API and CLIAuthenticated JSON APIs and standalone CLI with text/JSON output and watch mode.
DeploymentSingle Go binary with embedded UI, non-root AMD64/ARM64 containers, and automated PostgreSQL migrations.

Gallery: UI Walkthrough


r/apachekafka • • 12d ago

Tool Kurkik and AsyncAPI JetBrains plugin for self-maintaining APIs

Thumbnail
gallery
1 Upvotes

Kurkik is a CLI with a terminal UI that reads a Kafka cluster and works with it through AsyncAPI 3.0 documents. It is in Early Access, free to use (also at work), and I'd like feedback from people who run Kafka day to day

Terminal UI

Run kurkik in a terminal and you get a UI to browse your connections and what is on them: topics, partitions, replication, changed configs, consumer groups, ACLs. Connections are named, and secrets stay in the OS keychain or env/file references instead of in the config. You can import connections from IntelliJ IDEA.

MCP

Ships with tools to integrate with your agent of choice

claude mcp add kurkik -- kurkik mcp
> Orders stopped reaching the billing service on prod. Compare prod with staging and tell me what drifted.
  asyncapi_connections  asyncapi_topology prod asyncapi_compare staging prod
  
  The evidence: the topic and its consumers, side by side, read-only

Diff environments

kurkik diff staging-kafka.yaml prod        # a document against a cluster
kurkik diff dev prod                       # one environment against another

Exit code 0 means no drift, 2 means drift, 3 means it couldn't read everything (missing permissions, truncated listing). That makes it usable as a CI check. Whatever it could not read is said in the output rather than guessed.

Export to AsyncAPI, and sync from it

kurkik export prod                         # prod-kafka.yaml

The document has channels for topics with their settings, servers, consumer groups as receive operations and, optionally, the real Avro/JSON Schema/Protobuf schemas from a Confluent-compatible registry (Confluent, Karapace, Redpanda, Apicurio). It never invents send operations, because a broker doesn't know who produces to a topic.

The same document can then go the other way:

kurkik kafka sync staging-kafka.yaml prod

It shows a plan, asks for your consent, takes a snapshot, then applies the change. kurkik kafka sync revert <snapshot> puts it back. It deletes or recreates only what you asked for, and tells you who reads what it would destroy before you agree.

Because the result is plain AsyncAPI, the same document can also configure API gateways or feed documentation tools such as EventCatalog. Pulsar, SNS and SQS work the same way, if that is useful to you.

Install:

curl -fsSL https://raw.githubusercontent.com/Pakisan/kurkik-app/master/install.sh | sh

If you work in JetBrains IDEs

The same engine powers the messaging features of the AsyncAPI plugin for JetBrains IDEs: previewing a connection, diffing it against another environment or your source code, exporting it, and integration with Spring messaging (it finds your Spring Kafka listeners and maps them to AsyncAPI). Plugin: https://plugins.jetbrains.com/plugin/15673-asyncapi

Kurkik: https://kurkik.pavelon.dev (there is a feedback form on the site)

Questions I'd like your help with

  • What drift between environments actually hurts you: topic configs, partitions, ACLs, consumer groups, schemas?

  • Would you run kurkik diff in CI? What would make you trust the exit code?

  • For sync, what do you need to see in a plan before you'd approve it on a production cluster?

  • Do you keep AsyncAPI documents today? If not, what do you use to document topics and who uses them?

  • Which would you want next: another broker, more checks, or something else?


r/apachekafka • • 13d ago

Blog Cruise Control 3.1.0: A New Chapter Under the Linux Foundation

Thumbnail
linkedin.com
14 Upvotes

r/apachekafka • • 14d ago

Question Strimzi 1.3.0 is out!

11 Upvotes

Strimzi 1.3.0 is out!

3 things stood out to us in this release:

Configurable internal cluster security (Early Access)
Until now, broker-to-broker, controller and operator traffic always used TLS + mTLS. You can now choose your encryption (tls or none) and your authentication (mtls, none, or Kubernetes service-account tokens). If you already run a service mesh, that means you can stop encrypting the same traffic twice.

cert-manager integration (Early Access)
Set type: cert-manager on clusterCa / clientsCa, point it at your Issuer or ClusterIssuer, and cert-manager issues and renews the component certificates.

Graceful rolling restarts on Cilium & co.
UseBackgroundPodDeletion is now beta and on by default. Some CNIs tear down a pod's network the moment it's deleted, before the broker has finished shutting down. With background deletion, the network stays up until the containers exit, so the cluster recovers faster after rolls.

Personally, the cert-manager integration is the one I'm most excited about. Handing certificate issuing and renewal to cert-manager fits right into how a lot of platforms already manage their certificates. Since it's Early Access, it's going on a test cluster first, and I'm looking forward to taking it for a spin!

What's your favorite new feature?


r/apachekafka • • 13d ago

Tool Combining gRPC and Kafka in Java Microservices with Micronaut 4

2 Upvotes

One interesting challenge when designing Java microservices is choosing the right communication pattern.

gRPC and Apache Kafka address different requirements: synchronous RPC communication and asynchronous event-driven messaging.

With gRPC, the service contract is defined using Protocol Buffers. The .proto file describes the service methods, request messages and response messages. Maven and the Protocol Buffers compiler generate the corresponding Java classes and gRPC stubs.

For example, a Greeter service can expose a SayHello RPC method. A client sends a HelloRequest, the server processes it and returns a HelloReply through a generated gRPC stub.

With Micronaut 4 and Java 21, this provides a strongly typed communication layer without manually managing HTTP endpoints or JSON serialization.

However, synchronous RPC is not always the best choice when multiple services need to react to the same operation.

This is where Kafka comes in.

After handling a gRPC request, a service can publish a domain event, such as GreetingRequested, to a Kafka topic. Independent consumers can subscribe to that topic and process the event without being directly coupled to the original service.

This creates two distinct communication flows:

gRPC: request/response, synchronous execution, explicit service contracts.

Kafka: asynchronous message delivery, independent consumers, event-driven processing.

Combining them requires attention to several engineering details: event serialization, schema evolution, consumer groups, retry policies, idempotent processing, dead-letter topics and observability.

Another important issue is consistency between the RPC response and event publication. If an operation succeeds but publishing the event fails, the system can end up with inconsistent state. Depending on the use case, patterns such as the Transactional Outbox can help address this problem.

I'm exploring this architecture using Java 21, Micronaut 4, gRPC, Protocol Buffers, Maven and Kafka, starting from a simple Greeter service.

The gRPC implementation is available here: https://github.com/frekost/grpc-micronaut-starter-kit

I'm curious about how others approach this: do you use gRPC for synchronous communication and Kafka for domain events, or do you prefer a different architecture?


r/apachekafka • • 16d ago

Blog Read-Only Apache Iceberg Catalog for Kafka

6 Upvotes

Re-introducing https://minkdb.com, a read-only Apache Iceberg catalog for Kafka.

Iceberg engines query topics as tables, while the records stay in Kafka. Mink treats partition + offset ranges as virtual Iceberg data files and builds each one from Kafka when an engine reads it. It keeps file-level stats, such as min/max per column, so engines can skip files. A read-through cache speeds up repeat queries.

That makes Mink stateless, zero-copy and true real-time. And it's easy to onboard, since teams keep their existing Kafka vendor and pipelines.

Disclaimer: Mink was previously closer to the Fluss model. I changed direction for easier onboarding and more flexibility.

https://github.com/PicoMQ/mink


r/apachekafka • • 17d ago

Blog Interesting Kafka Links: October 2026

Thumbnail
rmoff.net
21 Upvotes

r/apachekafka • • 17d ago

Tool desflow: realistic Kafka event streams from a simulation

13 Upvotes

desflow is an open source Python package that publishes Kafka events from a discrete-event simulation of a process, built on SimPy. I shared an early version here under its old name, dynamic-des. This post shows what it does now.

If you only need messages with the right shape, Faker or a custom script is enough. When the stream has to behave like a real system, with bursts, queues that build up and slow periods, a simulation of that system produces it. desflow wraps SimPy and adds:

  • Three ways to write a model: SimPy-style process code, a builder API that modernizes SimPy code, and a YAML blueprint that needs little or no Python.

  • Parameters that change while the model runs: send a new capacity or arrival rate to a config topic, and the next events reflect it.

  • History, then live: one run writes past data to Parquet as fast as the machine allows, then publishes to Kafka in real time.

  • Other sinks: besides Kafka, PostgreSQL, Redis, Parquet and JSON Lines files on S3, GCS or Azure, and Iceberg tables.

  • Schemas: Avro, JSON Schema and Protobuf through the Confluent Schema Registry or the AWS Glue Schema Registry, with no dependency on confluent-kafka.

pip install "desflow[kafka]"
odctl up kafka-lite
desflow examples get kafka --api yaml
cd kafka/yaml
desflow run kafka.yaml

To send the events to your own cluster, set KAFKA_BOOTSTRAP_SERVERS before desflow run. Secured clusters work too: the connectors pass security settings to the Kafka client, and the docs show Confluent Cloud (SASL), mTLS and Amazon MSK with IAM.

The docs start with a tutorial for readers new to simulation, and include ready-made examples. A skill for AI coding assistants lets an assistant write, check and run a model with you.

On the roadmap: building a simulation from a process's event history with process mining techniques, so a model can start from the data your systems already record.


r/apachekafka • • 17d ago

Tool [Project] Kawa 0.1.0 - An open-source Kafka protocol gateway

6 Upvotes

I've just released Kawa v0.1.0, an open-source Kafka gateway I've been working on.

Kawa sits between Kafka clients and brokers, introducing a programmable layer without requiring changes to existing Kafka clients.

Some of its features:

  • Virtual topics

  • CEL-based message filtering

  • Centralized access control and RBAC

  • Topic governance and policy enforcement

  • Kubernetes integration through

    • Helm Charts

    • Operator

  • Kawa Studio; a web based UI

The goal is to explore what becomes possible by introducing a programmable layer at the Kafka protocol boundary.

This is the first public release of Kawa. I'm looking forward to getting it out there, hearing how others might use it, and getting feedback from the Kafka community.

Feedback, criticism and contributions are welcome!


r/apachekafka • • 17d ago

Question Anyone using StreamNative

3 Upvotes

Hi everyone,

We are currently exploring enterprise messaging platforms, with the long-term goal of consolidating Kafka and traditional MQ messaging into a single platform.

One of the vendors we are looking into is StreamNative (Apache Pulsar) mainly because of its multi-protocol capabilities and potential to support different messaging patterns within one platform.

Our goal is not simply to replace Kafka or MQ We are looking for a commercially supported solution that could potentially serve both Kafka and MQ workloads, without requiring extensive application changes or building and maintaining everything ourselves.

I would be interested to hear from anyone who has worked with StreamNative, either in production or during a PoC/vendor evaluation.

Thank you


r/apachekafka • • 19d ago

Question Need help with alternatives for dropMalformed and Schema Registry URL not supported for Kafka source on a Shared cluster in Databricks

Thumbnail reddit.com
2 Upvotes