r/sysadmin • u/syscoderadmin • 2d ago
Linux Managing multiple linux servers
Hi guys,
currently i'm working as junior linux sysadmin and managing about 50+ linux servers (different distros, services etc.). For every task i connect to servers with SSH and in my opinion its really inefficient. My main task is patching the servers and this would be great to implement ansible.
Do you have any recommendation about best practises with ansible? Like good articles, books, popular YouTube vids?
ps: its my first post đ
6
u/feedmittens Sysadmin 2d ago
Learn Linux TV (https://www.youtube.com/@LearnLinuxTV) has a great Ansible tutorial series, and this guy explains things well in a way that makes it easy to apply to your day-to-day tasks.
Patching is very easy to implement in Ansible. AI can build all this stuff for you, but if you don't understand the fundamentals, you will not be able to troubleshoot properly and/or will end up with spaghetti code you can't read.
If you haven't done this already, make sure you have a DEV/TEST/PROD workflow, if you can, because recent Red Hat kernel updates have frozen systems, and that's why you want to test changes and control changes in general.
1
u/syscoderadmin 2d ago
Didn't know about Learn Linux TV. I'll look into it đ
"AI can build all this stuff for you, but if you don't understand the fundamentals, you will not be able to troubleshoot properly and/or will end up with spaghetti code you can't read."
- Absolutely true!
9
u/Steerable-Octopus Sr. Infrastructure Engineer 2d ago
The official ansible documentation is good in its own right. But here's a freebie best practice tip: Don't make the mistake of making an imperative patching script in ansible, use ansible to configure up declarative automatic patching like unattended-upgrades or dnf-automatic or os-update, etc. and then have those tools do the patching for you instead.
3
1
u/Exzellius2 1d ago
Can you elaborate why?
1
u/Steerable-Octopus Sr. Infrastructure Engineer 1d ago
If you have a patching ansible script it means that it requires unattended remote root access to all these secondary nodes to run the patching process and this is an unnecessary cybersecurity risk.
It's an external dependency for the local patching which is unnecessarily fragile. If the connection drops it means stuck user sessions.
If you're an admin looking at the end nodes you might not realize that patching happens externally through ansible. It adds unnecessary obfuscation.
4
u/ABotelho23 DevOps 2d ago
Why do you have a mix of distributions?
5
u/syscoderadmin 2d ago
History.
I "inherited" the linux servers..
6
u/ABotelho23 DevOps 2d ago
So I would make it a project to distill it down to 2 distributions at most (with a justification like server vs client, etc). If applications absolutely must run in a specific distribution, use containers. Puppet/OpenVox would be your best bet for configuration management, and Ansible is best for task automation.
2
1
u/AcreMakeover 2d ago
Because the 5 people before OP each had a different opinion on which was best.
2
u/idrac1966 1d ago
Is ansible still the 'right' tool for the job in this day and age to keep a bunch of Linux boxes configured similarly or is there other better stuff out there now?
1
u/syscoderadmin 1d ago
Good question! I read in the answer about combination instead of only one tool.
2
49
u/farallon1 2d ago
I would start with Jeff Geerling videos as a starting point. His book is also good, which he open sourced I believe. That'll get you started.
Rest is learning through pain and/or reading docs.
15
u/moanos 2d ago
Just FYI before you spend money on his book: Jeff is a vocal homophobe and has compared being gay to being alcoholic (among other things). Details: https://drewdevault.com/weird-guys/
19
u/farallon1 2d ago
While I like what he does, I never cared enough to go look and find reasons to dislike him. Thanks for the heads up.
4
u/current_thread 1d ago
Oh man, that is so disappointing
4
u/moanos 1d ago
Yeah, I was also really bummed when I found out. I really hoped he changed over the years but he clearly didn't đŤ
â˘
u/farallon1 23h ago
TBH, you're overstating the situation when you say vocal homophobe. I can't find any recent content where he is discussing this. I can bet you can't find either. His older posts paints a homophobic picture, yes. But his content isn't really about sexual education. If I were looking for such content he won't be in my list.
As much as I like activism, from someone who spent a lot of time doing activism to change mind of strangers on internet, I suggest you spend your time and energy into something more productive. If you really care about these issues.
4
u/SirTwitchALot 1d ago
I had no idea. I've watched some of his 3d printing content. As a gay man I'll be disliking any future videos the algorithm feeds me. Thanks for the heads up!
5
4
-6
6
u/One-Department1551 2d ago
Avoid the shell module as much as possible, always look for either official or well supported community modules first.
Thereâs a tendency that âall can be solved with some bashâ that is hard to get rid off later.
6
u/Dynamic_Quest_Mod 1d ago
Once you're managing more than a handful, I'd stop treating them as individual machines. Ansible for configuration, centralised logging, proper SSH key management and a consistent patching process will save you a lot of pain.
The trick is getting everything into a predictable state before you start automating it. Otherwise you're just automating 50 slightly different problems.
4
u/Adam_Kearn 1d ago
Look into getting an RMM platform or even just create your own system that audits all your servers on a schedule so you know thing like
What distribution
What version
What packages/version
What hardware
EtcâŚ
Then when you need todo patches or changes itâs easy to see what is missing.
â˘
u/Optanik 11h ago
Ansible is the right call for this, and patching is a good first project because it's mostly a few modules and an inventory.
Build the inventory first and group hosts by distro family and by role (debian, rhel, db, web). Patching logic splits on distro anyway, and role groups let you stagger reboots.
Use the package modules instead of shell: ansible.builtin.apt with upgrade=dist and update_cache, ansible.builtin.dnf with name='*' state=latest. Branch on ansible_os_family, or just target the groups.
Check reboot-required (/var/run/reboot-required on Debian/Ubuntu, needs-restarting -r on RHEL), then use ansible.builtin.reboot so it waits for the host to come back.
Set serial: 1 or a small percentage on anything clustered, so you never take down a whole service at once.
Always run with --check --diff first and use --limit on a test host before hitting the full fleet.
For learning, the official docs' getting started guide and the 'Intro to playbooks' section cover most of this. Jeff Geerling's Ansible 101 series on YouTube and his book Ansible for DevOps are what most people point to. Put the playbooks in git from day one.
9
u/MeetJoan 2d ago
Jeff Geerlingâs Ansible for DevOps plus his free Ansible 101 YouTube series. Appendix B of the book is the best-practices cheat sheet.
For patching: group by distro and use apt/dnf with name: "*" and state: latest, roll it out with serial:, bump --forks past the default 5, dry-run with --check --diff, and let ansible.builtin.reboot handle restarts. Keep it in git.
2
u/OhBeeOneKenOhBee 2d ago
Weve started using patchmon a bit lately, it's pretty good but still quite young
2
u/Darthvaderisnotme 2d ago
Create your own servers, one for each distro
test there.
go veeery very slow, test and test again.
2
u/KN4SKY Linux Admin/Backup Guy 2d ago edited 2d ago
Beanologi's RHCE playlist on YouTube got me most of the way there. It's free but hasn't been updated since late 2023, so you'll want to supplement it with other resources: https://www.youtube.com/playlist?list=PLTY9BjMMGESHhzxfN6dcyYen8ZqUsxaLq
Make sure you use use --check before running a playbook. I'd also recommend having a few different inventory groups (such as a limited group that only touches 2-3 servers, then 10-20, then the rest.) This is kinda like Update Rings in Windows.
The RHCE official cert guide is good, since the exam is now entirely Ansible. You can get it for under $50 and it comes with practice exams if you ever want to take the exam.
Also, make sure you have good logging configured. You can specify a log file in ansible.cfg or via environment variables. This is especially important if you have multiple people running playbooks.
2
2
u/Open-Adhesiveness-86 2d ago
One thing that'll bite you with mixed distros: don't branch on ansible_distribution for patching, branch on ansible_pkg_mgr, since you'll hit things like Amazon Linux reporting yum vs dnf and SLES showing up as zypper. Also set serial: 5 and a max_fail_percentage so a bad mirror doesn't take out all 50 at once, and gather package facts before and after so you have a diff when someone asks what changed.
2
u/josemcornynetoperek 1d ago
Standarise them to one or two distributions.
Standarise configuration of common services as sshd, firewall or so.
Start using ansible or other solution for save server state.
It will be more work on start but in time you will have a lot more knowledge what exactly is running on servers and any operation will be faster and easier.
2
u/Forward-Outside-9911 Linux Admin 1d ago
+1 - weâve had manager support to standardise on just one main distro. Itâs easier to work with a single maintainable deployment.
And then while youâre at it make it all managed by Ansible or at least using a template ISO with a hardened/security preset.
4
u/databeestjenl 2d ago
use ansbile, see if you can get a CLI LLM tool on there to help make the playbooks, they are pretty easy to read.
3
u/Lost_Term_8080 2d ago
This is the reality of managing linux. There are management tools that help, but the management tools themselves necessitate extensive management. there are tools like saltstack or puppet, you'll add a few more servers to run that as well as add database administration to you list. If its only patching, check out azure update manager. Its by far the most mature linux patch management solution there is and it costs a dollar a month per server. However, once you get patching taken care of, I suspect so much time will be freed up you'll be looking for other things to do.
2
u/Ramiraz80 2d ago
Go buy Jeff Geerlings Ansible book, it is really good :) and he keeps it updated :)
Also watch some of Learnlinux.tv YouTube videos about ansible
1
u/1saltypistachio 2d ago
You can take a look at something like ansible. Should be simple enough to create a unified playbook for maintenance if you do the same thing on each server; or a different playbook for a family of servers.
1
1
1
u/Incrediblecodeman 1d ago
You should get really good at doing this way to be good at your job then yea in your off time master Ansible. Then present to your boss how youâd use it. Better then a jr being like âooOo the way YOU work isnât good. Can i try out this thing I heard about called ansivle on your production server? Nurrr errrr nnuuuurrrrr đ¤Ş
Innovation requires trust and itâs probably possible in a couple of months. 3 tops. To really make a convincing case. But you every donate 2 weekends a month to this passion project. Maybe find a few old boxes to make a home lab? As in you canât âlearnâ but watching YouTube and chatting with ai.
Good luck man boy I wish I was a Linux dude.
Btw is overthewire still cool?
1
u/MinimallyLoquacious HomeAdmin-Retired-Dev 2d ago
If it were me, Ansible. I don't use it on my home machines but it's on my list one of these days. If I were a FT Linux Admin though, I'd be using Ansible.
Also, and not sure why it's not listed under COMPANION SUBS but this is a good subreddit: https://www.reddit.com/r/linuxadmin/
0
u/Lonely-Abalone-5104 2d ago
This might not be a popular opinion but have AI like Claude teach you it. I find it to be a very good teaching tool
0
u/masterofrants Jr. Sysadmin 1d ago
You should install the Semaphore and Ansible UI as well. It'll help you graphically manage Ansible tasks then you can simply create the playbooks from ai use them via github and semaphore.
-1
u/CoreAndCardio 2d ago
Just use InTune. /s
1
u/idrac1966 1d ago
Can it actually do decent like 'desired state' configuration in Linux
1
u/CoreAndCardio 1d ago
No- it was a total joke hence the /s.
Based on the downvotes I see no one quite got that. Bad joke I guess
-1
u/ElevenB2002 2d ago
Automox is a great, relatively cheap option to Cloud manage Linux (and Windows/MacOS). You can patch, check posture, run "worklets" which can be any type of automated tooling you'd like, etc. It's a few bucks a month per managed node.
-10
u/Evening_Lavishness58 2d ago
just use the AI write you a script to automate the ssh connections and tasks you run
7
-3
u/Evening_Lavishness58 2d ago
crazy how sysadmins can't handle a solution that gets the job done
3
u/Hotshot55 Linux Engineer 2d ago
Does it really "get the job done", though? An AI script to automate ssh connections is basically just a shitty attempt at rewriting Ansible with less than 1% of the actual functionality.
-11
0
2d ago
[deleted]
4
u/pacmanlives Alcoholism as a Service 2d ago
I just write my playbooks to be the app tiers and then have logic in my playbook to run commands if they are different for each distro. Things like an update playbook. All the update commands are in the same playbook and let ansible handle the logic if itâs a Debian, SuSE, RHEL server. I just point it at the inventory and group I want to patch. I will also have another playbook that has no reboot baked in.
19
u/dwm 2d ago
Two thoughts:
Before you start doing much with Ansible, I would start with revision control as a foundational layer. `git init` is quick. Be disciplined about committing significant changes effectively â which includes writing good commit messages explaining why you're making the change, and how to back it out if it doesn't go well.
This is an old paper, and isn't Ansible-specific, but I think is really helpful for explaining infrastructure-oriented thinking:
http://www.infrastructures.org/papers/bootstrap/bootstrap.html
(It's more than 25 years old; many of the specific details have significantly changed since this was last updated, so don't rely on it to tell you which technologies to use â it predates git, containers, using virtual machines in production, and Google.)