r/devops • u/Obvious-Jacket-3770 • May 29 '24
Kubernetes is not DevOps.
See it time and time again here with people asking what to learn or creating roadmaps for DevOps saying you "need" Kubernetes (K8S) but that's such an incorrect statement.
Are there companies out there that use it and want you to have it? Sure. Many of those will never actually leverage it in the way it is constructed. Often times you will spend more time fixing issues with Kubernetes than you would other solutions. Kubernetes is container orchestration at scale, At Scale means "large volumes", which many companies don't have. I have worked for larger companies whose clusters almost never scaled and when they did they were minor. To the point that they spent more on K8S management, upkeep, and spending (servers on prem or cloud cost) than they made with systems on K8S.
Does this mean those companies built it incorrectly? Not all the time, no, it more often mean s that they were sold a solution for a problem they didn't have.
I want to be clear, you should ensure you learn Containerization. That is a modern framework for handling any application that's been made for or updated to handle it. Containers can be deployed on a variety of systems, both Kubernetes and not, and should be leveraged as such.
This ultimately comes down to a business case. Are you scaling, or planning to scale, at high levels? Say over 100 or 500 or greater? Yep Kubernetes is probably the solution you want at that point. Are you scaling less than 100, or 50, or 10? Other solutions are capable to assist you here for less cost and work.
From the azure end alone you have Container Instances and App Services which can host containers natively for both Windows and Linux. They have scale settings to allow up or out. They also have a much lower management overhead and cost overhead associated to them. AWS and GCP both have similar offerings that can be used instead.
Should you learn Version Control (git in most cases), a programming language (optional usually depending on the role you want), monitoring, log handling, IaC, Containerization, scripting, CI/CD? Yes to almost all. Should you learn Kubernetes? Only if the jobs you want need it.
22
u/durple Cloud Whisperer May 29 '24
Honestly, where I am we are using kubernetes in some unconventional ways. It fits our needs very well, and we couldn’t have done it this way without deep understanding of everything underneath. The Kubernetes layer just replaces what would otherwise be an effort-intensive internal tooling solution which reinvents various wheels and levers. It is less to learn than learning what all those levers are connected to, which is the real meat of devops. Kubernetes is just the current API.
Not to dismiss its importance. It’s likely to be relevant for a long time still. I can only wonder what new paradigm will emerge eventually, replacing or becoming yet another layer of abstraction.
7
u/Agreeable-Archer-461 May 29 '24
i'm kinda expecting it to just become more polished. It's basically doing at the process layer what VMs did for us at the host layer. You can't really go any lower down, so i'm kinda thinking it'll just become a lot easier to use and more polished. Eventually it'll be forgotten about entirely and just be the way things are.
2
u/p9-joe May 30 '24
Kubernetes is just the current API.
I think this is a big part of it, and something people like Kelsey Hightower were calling out years ago: the real value in Kubernetes isn't the orchestration per se -- there are a ton of process and container orchestrators out there, and even more that used to be popular but have faded away. What Kubernetes has going for it besides an enormous community is the extensible API that provides a consistent surface to build entire platforms on top of. In the future the idea of "managing the Kubernetes cluster" that your apps run on may be similar to the idea of managing the hypervisor your cloud instances run on -- "what's to manage, I can't see things at that level and I don't even want to".
15
u/ThatSituation9908 May 29 '24
Anecdote of DevOps lacking K8s familiarity hurts the project.
We have autoscaling K8s up on every environment. I want to experiment with monitoring tools. I asked DevOps to deploy one that I've got working in dev env. I suggest using K8s and Helm to deploy the service because IaC is very simple.
DevOps insist on using Ansible on an EC2. They lack knowing how Helm or K8s work. Now they have to write IaC scripts for both the EC2 and monitoring daemon.
In my view, if you're already maintaining K8s for other purposes needing scalability. Then do also use it for other services that do not need scalability because it has a smaller marginal cost than deploying it elsewhere.
13
u/fangisland May 29 '24
One of the things that tend to get missed in these types of convos is the benefits of K8s (or a managed platform of any sort) is it cleanly defines the software boundaries w/ interfaces that allow diff types of teams to be autonomous. Is it overkill in most use cases, esp when the business is new? Of course, you don't have a scaling problem until you have a scaling problem. But k8s sets up a good platform-centric framework that lets the underlying infra "means to an end" be separated from the business-delivery teams but still interact with each other through code and the components themselves. Which is what you want even at a small scale. You can absolutely set this type of framework up with very simple things and a document describing it that lets people self-serve. K8s particularly when coupled with gitops practices like Helm charts just delivers a lot of that without needing to make architectural decisions on a bunch of individual components. The tradeoff there is obviously, complexity - the context of every org will be different.
15
u/Agreeable-Archer-461 May 29 '24
It's just a tool, like any other. Fwiw though i've seen a lot more places try to build everything out on ecs and then end up reinventing the wheel further down the line when they could have just used k8s, than i have the other way around.
0
u/Glebk0 May 29 '24
Funny thing is you don't even build on ecs, it's almost as just running your containers. You don't need to bother with helm, argo, loki, service discovery, ways to get your secrets, integrations with other aws resources, etc. It's all just easily accessible straight from your containers, logs are shipped in cloudwatch(or you can setup your own logging if you really want to, like in k8s), secrets manager secrets can be easily accessed, you can work with queues, databases, whatever you need. Just assign the iam role with proper permissions to your tasks. Autoscaling and automated rollbacks in case of broken images is also there. It has anything you need. And if it is somehow not enough, you didn't actually waste any time setting ecs up, because it barely needs configuring compared to EKS or just regular k8s.
3
u/realitythreek May 29 '24 edited May 30 '24
I think this is a very specific AWS perspective. From my standpoint, a lot of the problems I have to solve for EKS, I still have to solve for ECS. The main difference for me is whether I use an AWS-centric grammar or a more portable k8s one. And the answer to that depends from company to company. OR, if you’re already tightly integrated with AWS and you’d be using their solutions for everything anyway.
0
May 29 '24
[deleted]
7
u/tr_thrwy_588 May 29 '24
reinventing the wheel with putting one container on a vm because "we'll never need more", then a second one, then a third. cue in 6 months later and its 250 containers all running on a single vm (!?). Container #132 ate up all the memory of the machine! Oops? true story.
5
u/Lifaux May 29 '24
Likely versioning deployments, handling multiple environments, etc. Things that are simpler with kustomize.
1
u/Glebk0 May 29 '24
All this is easily done in ecs, you don't even need to invent anything. Tasks have versioning by default, about environments, you just create more clusters(which are free), or use different aws accounts like a normal person. Ecs is really good platform, which is easy to use(not saying that k8s is hard, but nobody can argue against k8s having more overhead) within this whole aws ecosystem.
3
May 30 '24
We came from ECS and found a lot more tooling out there for gitops workflows that “just work” with kubernetes like argocd and Kyverno. Those two alone and kustomize have been huge time savers vs ECS, but it’s fair to say our original ECS setup was pretty rough
1
u/emcell May 30 '24
I'm coming more from a developer scope.
For me, the best thing about kubernetes is the standardization of the deployments. before kubernetes the questions were always:
on which server is this app deployed
are the domains pointing to the correct load balancers?
how do i access the server to update the software?
which reverse proxy is running on the server?
now the answer is always: look into the kubernetes api
1
u/Obvious-Jacket-3770 May 30 '24
You don't need Kubernetes for deployment standards, Containerization as a whole covers that.
1
u/emcell May 30 '24
containerization is just part of problem. where should i run those containers, how do i deploy those docker compose files? is there a reverse proxy running and i need some attributes in the compose file to configure it? do i need to change some config files in /etc/.
i'm with you that i don't need kubernetes to standardize this across the company. but kubernetes is already sone kind of standard we can just follow.
1
u/Obvious-Jacket-3770 May 30 '24
It's a forced standard by marketing. You also largely need to add many of those items in a dockerfile or docker-compose file.
3
u/the_moooch May 29 '24
Its standardized, opensource, extendable and have a myriad of plugins and ecosystem around it. Many would probably find other tools which fits their CURRENT needs but when it didn’t migrating an existing infrastructure later is a fucking nightmare nobody want to unravel.
3
u/khobbits Systems Infrastructure Engineer May 30 '24
Does Kubernetes have overheads? Yes.
There is a CPU/RAM and hosting cost related to run it, the amount varies on how resilient you opt in for, but the overhead exists.
You can minimise this a bit, by reusing the cluster for other apps, teams, projects that have a similar risk profile.
There is also a man hour overhead. This one however can pay itself off quite quickly. I have a scattering of different application platforms, some strung together by bash scripts, some by configuration management (Chef & Puppet), some by cloud-init scripts, some with docker-compose files. The easiest machines to patch, are the Kubernetes worker nodes.
Is it the right platform for everything, hell no. But let's say you needed to spin up a simple (dynamic) website, low throughput, not likely to require any scaling, but it can be a bit flaky, say has a memory leak, or crashes randomly every few days. I'd strongly consider throwing it on Kubernetes. The out of the box tooling for handling and monitoring application state, recovering from failures, and integration with things like load balancers, means that all will stay up 24/7, and weather OS patching, and upgrades without an issue.
And all of that will be done using industry standing tooling, that the next engineer will be able to glance at the manifests in git, and fully understand the infrastructure. No hidden bits in instance metadata, no floating files on nfs servers, no hidden nagios servers triggering reboots.
1
u/nwmcsween May 30 '24
Have you worked with k8s? Many Azure ARC offerings is just k8s covered in some Azure lipstick. The problem with container instances, container apps, etc is really vendor lock in, in order to get the same metrics and functionality you can from a well designed k8s "package" (helm, kcl, kustomize, w/e) you need and pay to play at $CLOUD which is as designed.
1
u/Obvious-Jacket-3770 May 30 '24
Yes and I've watched it not scale or be utilized in a way that's worth the expense vs other options for hosting containers.
2
u/jaybutts May 29 '24
Well its definitely devops, it makes it possible, (keyword possible) to make development and operations better, so its devops. A lot of the times its over the top, more work than necessary, etc..do we really need to spend extra time to deploy something that is only potentially slightly better for deployments, configurations, resources , etc than just ansible/terraform and vms ? Often not…this is the age old problem of prioritizing code, like when you get a new ops guy and he spends 40 hours to code something that just takes 5 minutes once a year and your like “ what the hell dude”
2
2
2
u/gowithflow192 May 30 '24
I'd argue that actually Kubernetes is becoming more accessible at smaller scale (look at k0s for example). I would rather a career using Kubernetes than Rancher or Nomad or ECS and yes, it is better "resume development" for me but we have to be mindful of that to stay employed and have good, sustainable careers.
You also completely forget the entire ecosystem you get with Kubernetes. I think this is actually becoming more important than the basic technical requirements of the app.
3
May 29 '24 edited Jan 26 '26
This post was mass deleted and anonymized with Redact
hospital imagine alive soup consist touch wide resolute sleep fly
1
u/glotzerhotze May 30 '24
Sure, kubernetes is a lot of complexity made visible right from the start. It forces you to think about problem-spaces you‘d never thought of, you‘ll never deem relevant for your system at hands.
But let me tell you: every system you run - even without k8s - will inherently exhibit all the same problem-spaces you need to get familiar with when starting to use k8s.
The point I‘m trying to make is this: just because you don‘t currently see the problems k8s forces you to deal with in your non-k8s-setup, does NOT mean you won‘t face these problems in the future.
Now you can deal with them while production grinds to a halt - which probably is a very stressful experience for OPS people working to get stuff back online. Or you can solve those problems ahead of time by using k8s and have a nice sleep at night.
I think people need to live through such an experience to fully understand the benefits k8s offers for every stage of an org.
1
79
u/snarkhunter Lead DevOps Engineer May 29 '24
I think the best thing about a very junior person learning k8s is that it will expose them to so many of the aspects and components of running a live service in a cohesive coherent framework. If you learn K8s you learn about networking, storage, load balancing, running updates, etc.