r/kubernetes • • 4h ago

How are you handling multi-cluster Kubernetes networking and traffic routing in a hybrid cloud environment?

11 Upvotes

Hey folks,

We're running multiple Kubernetes clusters across on-prem (OpenShift) and GCP (GKE), and I'm curious how others are handling networking and traffic routing in a similar setup.

Some of our services run in multiple clusters, while others are only deployed in specific clusters.

A few things I'm wondering about:

  1. How do you handle ingress and route traffic to the right cluster?

  2. How do services communicate across clusters?

  3. Are you using global load balancers, API gateways, service mesh, or something else?

  4. How do you handle failover when one cluster goes down?

Would love to hear what your setup looks like, what tools you're using, and any pain points you've run into.


r/kubernetes • • 56m ago

Do we have enough hands-on practice for real-world DevOps incidents?

2 Upvotes

We have no shortage of Kubernetes tutorials, cloud certifications, courses, and documentation. You can learn almost any DevOps tool online today.

But I'm curious about something: how do we get better at troubleshooting when things don't go as expected?

In a real engineering environment, these situations can take hours to diagnose. Yet most learning resources tend to focus on setting things up correctly rather than dealing with things when they go wrong.

I'm curious how other engineers approach this.

Do you create your own broken environments to practice? Do you rely on production incidents and experience? Or do you feel there's a gap in the way we currently learn troubleshooting?

I've been exploring this space and working on an open-source project called SnowOps Labs, which provides hands-on scenarios where engineers can reproduce failures, investigate incidents, and practice fixing them.

I'd particularly like some outside perspectives on the concept and the implementation.

Project: https://github.com/snowops-oss/snowopslabs

A few questions I'd love to hear your thoughts on:

  1. Is this a problem worth solving, or am I overestimating the gap?

  2. What would make a hands-on incident simulation platform genuinely useful?

  3. Would you use something like this for personal learning, interview preparation, or team training?

  4. What existing tools or platforms do you think already solve this problem well?

Feel free to be critical. I'd much rather hear why this isn't useful than get generic encouragement.

Interested to hear how others think about this.

Not looking for stars or promotion. Just genuinely interested in hearing what other engineers think, including criticism.


r/kubernetes • • 4h ago

IBM core-dump-handler removed?

1 Upvotes

Did IBM remove core-dump-handler? The GitHub page is returning a 404: https://github.com/IBM/core-dump-handler


r/kubernetes • • 21h ago

How do you keep Kubernetes rightsizing and autoscaling configs from becoming a permanent manual job?

16 Upvotes

When running different types of clusters (prod, staging, dev), how do you keep resource allocation and autoscaling settings aligned with workload requirements as they change, to avoid unnecessary increases in the cloud bill?

Over the lifetime of a product, new services get added in a cluster, traffic patterns would change, requirements evolve, etc. CPU/memory requests could get padded, limits and replica counts can become outdated, and the node types originally chosen may no longer fit the current workload mix particularly well.

I'm particularly interested in continuous pod rightsizing alongside tools like Karpenter and KEDA. As a simple example, cpu/mem requests could remain significantly higher than workloads actually need over time. This usually leaves the cluster running far more nodes than necessary (even for smaller workloads with a few nodes, even 1 or 2 of overprovisioned and unutilized nodes could cause a significant dent in the final bill). This can be overprovisioned intentionally, but over time it does become a bottleneck in a sense that there's not enough time to address everything.

How do you keep those settings appropriate as workloads evolve? How are you handling this in practice, and lately with AI?

- Do you review the full cluster regularly, leave it to application teams to right size (where applicable), or just accept the overprovisioned capacity?

- How do you distinguish necessary headroom from avoidable waste?

- Are you using any tools that recommend specific changes and explain the expected savings and risks, ideally across the major clouds? I know this can get very workload-specific, but still would like to hear general thoughts.

- Does anyone use automated tools that continuously adjust pod resources, scaling settings, or node selection and placement? Can you share your experience with such tools? Does any of these run internally within own clusters, or do they all offer managed control plane?

Thanks


r/kubernetes • • 1h ago

So who else came home from Kubecon with the flu

0 Upvotes

Least favorite thing I brought home. Feelsbadman.


r/kubernetes • • 7h ago

Running ZAP as an in-cluster Job against one microservice change instead of shared staging

1 Upvotes

Hey folks, we just published a tutorial on running ZAP as an in-cluster Job against one microservice change. Might be helpful if your security scans only ever see shared staging.

ZAP's official image runs as a Job and scans only the changed service, so a finding belongs to that change rather than to whatever else shipped to staging. The scan itself works against any in-cluster URL. Signadot is what runs the changed service next to the shared ones without copying the stack. The tradeoffs, like real attack payloads reaching shared databases, are covered in the write-up.

Hope it helps. Happy to answer any questions. Full transparency, I work for Signadot.

https://www.signadot.com/docs/tutorials/zap-security-testing


r/kubernetes • • 6h ago

Criei um cliente Kubernetes para desktop porque o Lens e o Headlamp não se adaptavam à minha forma de depurar.

Thumbnail
reddit.com
0 Upvotes

r/kubernetes • • 1d ago

Hows everyone handling secret zero outside kubernetes?

37 Upvotes

vault or openbao handing out short lived postgres logins is ok and nothing sits in a k8s secret anymore. but the app still has to log into vault first

inside the cluster thats mostly solved, vault checks the pods service account token against the TokenReview api and theres even a patch on pgsql-hackers so postgres cert auth can match spiffe ids. outside kubernetes its murkier. cloud instance identity covers vms on aws or gcp, spiffe setups like spire or hush security go wider and a lot of the rest ends up with an approle secret id baked in somewhere which is the same problem one step back

whats ideal to use for that first login outside the cluster?


r/kubernetes • • 1d ago

How do you handle prod promotion with Argo CD + Image Updater when CI is triggered by a git tag?

Thumbnail reddit.com
3 Upvotes

r/kubernetes • • 1d ago

Anyone running kubernetes-sigs/agent-sandbox in prod or lab for coding agents? Looking for real-world feedback on isolation, warm pools & latency

28 Upvotes

I’m currently evaluating kubernetes-sigs/agent-sandbox to run autonomous coding agents (like Claude Code, custom harness ,etc) on our own Kubernetes cluster, and I’d love to get feedback from anyone who has deployed it beyond a quick local test.

​Context & Current Setup:

  • ​Cluster: AWS EKS

  • ​Workload: Ephemeral dev/test environments where agents clone repos, run builds/tests, and execute arbitrary shell commands via MCP or tool calls

  • ​Isolation: Evaluating gVisor vs. Kata Containers vs recently introduced AWS nitro sandboxes

​Questions for folks actually running this:

  1. ​Cold-start latency & SandboxWarmPool: What kind of pod startup and readiness times are you seeing in practice? Are warm pools reliable enough to make interactive/agentic loops feel snappy, or are you seeing churn issues?

  2. ​Runtime performance hit: If you’re pairing agent-sandbox with gVisor or Kata, how noticeable is the performance hits

  3. ​Egress & network isolation: How are you locking down outbound network access (e.g., allowing git clone and package registries while blocking internal VPC/metadata endpoints)?

  4. ​Workspace caching: Are you doing anything clever to cache dependencies across ephemeral sandbox runs without compromising isolation between sessions?

  5. ​Hidden gotchas: Any rough edges around CRD lifecycle cleanup, orphaned PVCs/volumes, or resource quotas when an agent loop goes rogue?

​If you looked at agent-sandbox and decided to go a different route—like self-hosting E2B, rolling custom Firecracker microVMs, or just using hardened Kubernetes Jobs—I’d also love to hear what pushed you away from it.


r/kubernetes • • 2d ago

Kubefactory - Container Management Gaming Interface

472 Upvotes

A first-person factory game that views and manages your Kubernetes clusters and Docker containers.

In-browser (sandbox) demo: https://dascondor.github.io/kubefactory/
GitHub Repo: https://github.com/DasCondor/kubefactory

Edit: Thank you all for the support & feedback, if I haven't answered your question in the comments I am sorry - I will try and get with you over this coming week.

Different versions of Kubefactory 1.0.1-release vs 2.0-beta & Coming 2.1;

1.0.1 — Original release

  • Windows desktop

  • Browser demo

  • Mobile/tablet access through the browser

2.0.0-beta.2 — Available now

  • Windows, Linux, and macOS builds

  • Browser demo with phone and touch support

  • Native Kubernetes and Docker connectivity

  • Model library, LiveView, storage inspection, Krew, Porter, Bob, Gauge, and Foreman

  • VPN registry, exit-IP checks, direct pipes, VPN sidecars, and gateway views

  • Service pipelines and Jellyfin + VPN media stack

  • Text editor with Standard, Vim, and Nano modes

  • High-graphics browser fix, updated docs, screenshots, tests, and release attestations

  • Windows signing workflow wired in, pending activation

2.1 — Coming next

Further polish, fixes, testing, and features building on the 2.0 beta.

Note: beta.2 is a pre-release, and its packages are not yet code-signed.


r/kubernetes • • 1d ago

Prune=false on every Namespace with Argo CD auto-prune, is this what people actually do?

2 Upvotes

Sorry first, english is not my first language and I write this with a translator.

I never worked in a big SRE or platform team, so I want to check if what I did is normal. I run a small k3s cluster at home, plus EKS at work and a small AKS for a client. A read-only audit tool I made for my own clusters (kubegap) flagged that my Argo CD apps with automated prune also track their own Namespace.

So I tested it with throwaway apps on Argo CD 3.2.0. I removed namespace.yaml from kustomization.yaml and pushed. 2 min 48 s later the Namespace was pruned, and a Secret I had created by hand inside it was gone too.

Removing an app's directory under an ApplicationSet git generator is a different path. The Application itself gets deleted and the finalizer cascades. Prune=false does nothing there, Delete=false does.

What I changed: every Namespace manifest now has the sync-options annotation Prune=false,Delete=false (8 of them). The istio app owns 23 CRDs, so I turned automated prune off for that one. And I added an alert for apps that stay OutOfSync for 30 minutes, because a skipped prune just leaves the app OutOfSync and nobody looks.

Questions for people who run this in production:

Do you keep Namespaces inside the app at all, or use CreateNamespace=true or a separate app just for namespaces?

Is per-resource Prune=false normal, or do teams just turn automated prune off?

I read that a force sync can still delete a namespace even with Prune=false. Do you guard against that somehow?


r/kubernetes • • 1d ago

Does GKE's node kernel have the SCTP `sctp_init_cause` padding fix, or is it still shipping unpatched?

Thumbnail reddit.com
3 Upvotes

I've been checking which Linux kernels still carry an SCTP bug that panics the host: `sctp_init_cause()` checks buffer room for an error cause but not for the up-to-3 bytes of 4-byte alignment padding the write adds, so a crafted INIT (listener) or INIT-ACK (outbound association) can walk an skb one byte past its end and hit `skb_over_panic()`. It's fixed upstream in `ed6c53b27be4` (mainline 7.4-rc2, Cc: stable), introduced back in 2018.

As of 14 Nov I've read the guard in the shipped source of every maintained stable branch and several distro kernels, and none of them had the fix yet. That includes the current Ubuntu kernels (jammy 5.15, noble 6.8, resolute 7.0). GKE Standard supports SCTP on Ubuntu nodes with Dataplane V2, and Google documents it as on by default from `1.32.2-gke.1297000`. Since GKE's Ubuntu node images are built from Ubuntu kernels, my working assumption is that most GKE Ubuntu nodes are running the unpatched code. I haven't confirmed a specific GKE node-image build, which is where I could use help.

If you run SCTP workloads on GKE (telecom CNFs especially), could you check a non-production node and report back:

- `uname -r` on the node, and whether `ed6c53b27be4` is backported (the node's kernel source/changelog, not the version string, since backports move independently)

- whether `lsmod | grep '^sctp '` shows the module loaded, i.e. a workload is actually using it

- the GKE version and node image

All read-only. Please don't send anything at a listener or a live service; I'm after kernel/config state, not a crash.

If it turns out a GKE configuration is unpatched and reachable, that belongs in a report to Google's Vulnerability Reward Program, not in this thread. I'll help put that together rather than post specifics here.

Mechanism and the full affected-version matrix (no exploit) are written up here:

Here

Mainly I want to nail down one thing: is a default-config GKE Ubuntu node running the fixed kernel yet, or not?


r/kubernetes • • 2d ago

Metrics, logs and alerts for my 3-node Talos homelab

8 Upvotes

My 3-node Talos homelab had no monitoring, so I only found out something was wrong when an app stopped working.

So I added VictoriaMetrics for metrics and VictoriaLogs for logs from every node and pod. Both show up in Grafana, and alerts go to my phone on Telegram.

Healthchecks[.]io also watches the alerting itself, so I hear about it if the alerts stop. The whole stack uses about 1.3 GiB of RAM across the three nodes.

Repo: https://github.com/prateekjaindev/homelab

Write-up: https://blog.prateekjain.dev/metrics-logs-and-alerts-for-a-3-node-talos-homelab-with-victoriametrics-0f4a91b9d44e?sk=751eedebb5d54cc15561d59abb7e52d8


r/kubernetes • • 1d ago

Reviewing agent written Helm changes when the problem is cluster wide

0 Upvotes

We run about 30 services on two EKS clusters and a lot of our Helm values changes now come from agents. They're usually small and they usually look fine.

Last week one changed the default resource requests in our shared chart from 250m to 500m CPU "to reduce throttling". Review was clean. coderabbit reviewed the values file, it catches things like missing limits or a probe pointing at the wrong port, and there was nothing wrong with the YAML itself.

The problem was everything else. 30 services picked up the new default on their next deploy, the cluster autoscaler added 6 nodes overnight, and our bill for that week jumped by about $1,900 before anyone noticed.

The diff was 2 lines. The impact was the whole cluster, and nothing in the PR showed that.

What we're looking at now:

- running helm diff for every release that uses the shared chart in CI and posting the summary on the PR
- requiring a second reviewer from platform for any change to shared chart defaults
- some kind of cost estimate on PRs that change requests

How do you review changes to shared charts? Do you diff every consumer, or is it mostly "be careful with the shared one"?


r/kubernetes • • 1d ago

Kubernetes Scheduling: Can You Move an Existing Pod to Another Node Without Touching Its YAML? 🤔

Thumbnail reddit.com
0 Upvotes

r/kubernetes • • 2d ago

GitHub - danielecr/tetragon-outgoing-guard: block any process with outgoing connection (run only on k8s or docker swarm vms)

Thumbnail
github.com
4 Upvotes

How I locked down my Docker host against GSocket-style C2 implants using a 50-line Cilium Tetragon eBPF policy.

Also in docker swarm, I mean, the only services are supposed to be in container


r/kubernetes • • 2d ago

I have my KCSA day after tomorrow

1 Upvotes

as the title suggests, can I have any last day tips. Any question which you remember could be of help.
thanks


r/kubernetes • • 3d ago

Best resource to learn hands on Kubernetes

76 Upvotes

Hello, I am preparing for an interview. I have limited experience in Kubernetes with some in open shift. But most of my experience into terraform, Python, Jenkins, ansible and cloud architecture. I have interviews scheduled next week and I am looking for good resources to learn and practice Kubernetes over the weekend. Can someone suggest a good course/resource for the same? I want to do hands on over the weekend and can set up a lab in my personal laptop


r/kubernetes • • 2d ago

Free browser-based Kubernetes manifest generator (Deployment, StatefulSet, CronJob, RBAC, NetworkPolicy…). Feedback welcome

Thumbnail
reddit.com
1 Upvotes

r/kubernetes • • 3d ago

Tipps on migrating TLS Cert Management from Traefik to cert-manager

9 Upvotes

Maybe someone here has done this or something similar in the past. This is the situation right now:

I have about 20 services on a HA k3s cluster with the bundled Traefik as a single instance deployment. TLS is currently done with the basic Traefik letsencrypt certificate resolver with a http challenge.

I now want (or rather need) to move to cert-manager, to enable a HA traefik setup. The migration should happen without downtime.

If both Traefik and cert-manager use the HTTP challenge method to generate a cert, cert-manager could never complete the challenge, as Traefik would intercept the challenge call. I was thinking to switch Traefik to the TLS-ALPN challenge temporarily, so that cert-manager can generate the certs, and then switch over to the HA deployment with only cert-manager.

I guess it should work, but are there better options? Any pitfalls to watch out for? I'm thankful for any input!

Edit: I forgot to add that some domains are "customer domains" and they just point their A records to my cluster.


r/kubernetes • • 2d ago

Agent skill to create Kubernetes concepts explanation video

0 Upvotes

Hi everyone,

I made a small skill that lets your AI agent create 3Blue1Brown-style videos explaining Kubernetes concepts. The demo is below.

Tested on Claude Code Sonnet 5 and Opus 5.5

If you have more interesting ideas, feel free to contribute!

Link to GitHub: https://github.com/wielenmaker/k8s-animate


r/kubernetes • • 3d ago

Two-node Kubernetes on openRuyi RISC-V with Kata and Cloud Hypervisor

15 Upvotes