r/sysadmin • • 2d ago

Linux Managing multiple linux servers

Hi guys,

currently i'm working as junior linux sysadmin and managing about 50+ linux servers (different distros, services etc.). For every task i connect to servers with SSH and in my opinion its really inefficient. My main task is patching the servers and this would be great to implement ansible.

Do you have any recommendation about best practises with ansible? Like good articles, books, popular YouTube vids?

ps: its my first post 😄

37 Upvotes

72 comments sorted by

19

u/dwm 2d ago

Two thoughts:

  1. Before you start doing much with Ansible, I would start with revision control as a foundational layer. `git init` is quick. Be disciplined about committing significant changes effectively — which includes writing good commit messages explaining why you're making the change, and how to back it out if it doesn't go well.

  2. This is an old paper, and isn't Ansible-specific, but I think is really helpful for explaining infrastructure-oriented thinking:

http://www.infrastructures.org/papers/bootstrap/bootstrap.html

(It's more than 25 years old; many of the specific details have significantly changed since this was last updated, so don't rely on it to tell you which technologies to use — it predates git, containers, using virtual machines in production, and Google.)

3

u/syscoderadmin 2d ago

I'll look into it 👍

2

u/Exzellius2 1d ago

Where would you set the base for the git repo? /etc?

3

u/dwm 1d ago edited 1d ago

The base of the Ansible (or other tool) files that I'm writing, rather than any particular system folder on my managed computer(s).

There will be things other than my chosen change management tool that will also be updating the configuration state of my systems. For example, package additions/updates.

Therefore, trying to capture any subset of the live server filesystem in git directly will result in my change management system and those other sources of configuration change fighting with each other.

Even if that weren't the case, the contents of the filesystem to be written out will need to vary from system to system. (/etc/hosts and /etc/hostname need to contain the name of the server, for example.)

So I'd recommend against trying to do that at all, and instead use some kind of executable domain-specific language, such as Ansible (or Puppet or a bunch of scripts or..) that is designed to run over a system and configure various aspects of a machine that I want to be consistently set a particular way.

And those scripts should be kept in version control.

6

u/feedmittens Sysadmin 2d ago

Learn Linux TV (https://www.youtube.com/@LearnLinuxTV) has a great Ansible tutorial series, and this guy explains things well in a way that makes it easy to apply to your day-to-day tasks.

Patching is very easy to implement in Ansible. AI can build all this stuff for you, but if you don't understand the fundamentals, you will not be able to troubleshoot properly and/or will end up with spaghetti code you can't read.

If you haven't done this already, make sure you have a DEV/TEST/PROD workflow, if you can, because recent Red Hat kernel updates have frozen systems, and that's why you want to test changes and control changes in general.

1

u/syscoderadmin 2d ago

Didn't know about Learn Linux TV. I'll look into it 😄

"AI can build all this stuff for you, but if you don't understand the fundamentals, you will not be able to troubleshoot properly and/or will end up with spaghetti code you can't read."

- Absolutely true!

9

u/Steerable-Octopus Sr. Infrastructure Engineer 2d ago

The official ansible documentation is good in its own right. But here's a freebie best practice tip: Don't make the mistake of making an imperative patching script in ansible, use ansible to configure up declarative automatic patching like unattended-upgrades or dnf-automatic or os-update, etc. and then have those tools do the patching for you instead.

3

u/syscoderadmin 2d ago

I take that freebie best practise tip :D

1

u/Exzellius2 1d ago

Can you elaborate why?

1

u/Steerable-Octopus Sr. Infrastructure Engineer 1d ago

If you have a patching ansible script it means that it requires unattended remote root access to all these secondary nodes to run the patching process and this is an unnecessary cybersecurity risk.

It's an external dependency for the local patching which is unnecessarily fragile. If the connection drops it means stuck user sessions.

If you're an admin looking at the end nodes you might not realize that patching happens externally through ansible. It adds unnecessary obfuscation.

4

u/ABotelho23 DevOps 2d ago

Why do you have a mix of distributions?

5

u/syscoderadmin 2d ago

History.

I "inherited" the linux servers..

6

u/ABotelho23 DevOps 2d ago

So I would make it a project to distill it down to 2 distributions at most (with a justification like server vs client, etc). If applications absolutely must run in a specific distribution, use containers. Puppet/OpenVox would be your best bet for configuration management, and Ansible is best for task automation.

2

u/syscoderadmin 2d ago

Same thought 👍

1

u/AcreMakeover 2d ago

Because the 5 people before OP each had a different opinion on which was best.

2

u/idrac1966 1d ago

Is ansible still the 'right' tool for the job in this day and age to keep a bunch of Linux boxes configured similarly or is there other better stuff out there now?

1

u/syscoderadmin 1d ago

Good question! I read in the answer about combination instead of only one tool.

2

u/Runnergeek DevOps 1d ago

Yes very much the right tool.

49

u/farallon1 2d ago

I would start with Jeff Geerling videos as a starting point. His book is also good, which he open sourced I believe. That'll get you started.

Rest is learning through pain and/or reading docs.

15

u/moanos 2d ago

Just FYI before you spend money on his book: Jeff is a vocal homophobe and has compared being gay to being alcoholic (among other things). Details: https://drewdevault.com/weird-guys/

19

u/farallon1 2d ago

While I like what he does, I never cared enough to go look and find reasons to dislike him. Thanks for the heads up.

4

u/current_thread 1d ago

Oh man, that is so disappointing

4

u/moanos 1d ago

Yeah, I was also really bummed when I found out. I really hoped he changed over the years but he clearly didn't 🫠

•

u/farallon1 23h ago

TBH, you're overstating the situation when you say vocal homophobe. I can't find any recent content where he is discussing this. I can bet you can't find either. His older posts paints a homophobic picture, yes. But his content isn't really about sexual education. If I were looking for such content he won't be in my list.

As much as I like activism, from someone who spent a lot of time doing activism to change mind of strangers on internet, I suggest you spend your time and energy into something more productive. If you really care about these issues.

4

u/SirTwitchALot 1d ago

I had no idea. I've watched some of his 3d printing content. As a gay man I'll be disliking any future videos the algorithm feeds me. Thanks for the heads up!

5

u/Capable-Ad-5344 2d ago

And.... doesn't mean his free book doesn't have good points in it.

4

u/Low_Independent766 2d ago

No one is suggesting OP hang out with him. 

-6

u/Signal_Accident_9398 2d ago

Only separate the book from the author. Simple.

6

u/One-Department1551 2d ago

Avoid the shell module as much as possible, always look for either official or well supported community modules first.

There’s a tendency that “all can be solved with some bash” that is hard to get rid off later.

6

u/roiki11 2d ago

Standardize on a single distro and set up something like foreman or it's derivatives.

Really simplifies managing hundreds of vms.

6

u/Dynamic_Quest_Mod 1d ago

Once you're managing more than a handful, I'd stop treating them as individual machines. Ansible for configuration, centralised logging, proper SSH key management and a consistent patching process will save you a lot of pain.

The trick is getting everything into a predictable state before you start automating it. Otherwise you're just automating 50 slightly different problems.

4

u/Adam_Kearn 1d ago

Look into getting an RMM platform or even just create your own system that audits all your servers on a schedule so you know thing like

What distribution
What version
What packages/version
What hardware
Etc…

Then when you need todo patches or changes it’s easy to see what is missing.

•

u/Optanik 11h ago

Ansible is the right call for this, and patching is a good first project because it's mostly a few modules and an inventory.

  1. Build the inventory first and group hosts by distro family and by role (debian, rhel, db, web). Patching logic splits on distro anyway, and role groups let you stagger reboots.

  2. Use the package modules instead of shell: ansible.builtin.apt with upgrade=dist and update_cache, ansible.builtin.dnf with name='*' state=latest. Branch on ansible_os_family, or just target the groups.

  3. Check reboot-required (/var/run/reboot-required on Debian/Ubuntu, needs-restarting -r on RHEL), then use ansible.builtin.reboot so it waits for the host to come back.

  4. Set serial: 1 or a small percentage on anything clustered, so you never take down a whole service at once.

  5. Always run with --check --diff first and use --limit on a test host before hitting the full fleet.

For learning, the official docs' getting started guide and the 'Intro to playbooks' section cover most of this. Jeff Geerling's Ansible 101 series on YouTube and his book Ansible for DevOps are what most people point to. Put the playbooks in git from day one.

9

u/MeetJoan 2d ago

Jeff Geerling’s Ansible for DevOps plus his free Ansible 101 YouTube series. Appendix B of the book is the best-practices cheat sheet.

For patching: group by distro and use apt/dnf with name: "*" and state: latest, roll it out with serial:, bump --forks past the default 5, dry-run with --check --diff, and let ansible.builtin.reboot handle restarts. Keep it in git.

2

u/OhBeeOneKenOhBee 2d ago

Weve started using patchmon a bit lately, it's pretty good but still quite young

2

u/Darthvaderisnotme 2d ago

Create your own servers, one for each distro
test there.
go veeery very slow, test and test again.

2

u/KN4SKY Linux Admin/Backup Guy 2d ago edited 2d ago

Beanologi's RHCE playlist on YouTube got me most of the way there. It's free but hasn't been updated since late 2023, so you'll want to supplement it with other resources: https://www.youtube.com/playlist?list=PLTY9BjMMGESHhzxfN6dcyYen8ZqUsxaLq

Make sure you use use --check before running a playbook. I'd also recommend having a few different inventory groups (such as a limited group that only touches 2-3 servers, then 10-20, then the rest.) This is kinda like Update Rings in Windows.

The RHCE official cert guide is good, since the exam is now entirely Ansible. You can get it for under $50 and it comes with practice exams if you ever want to take the exam.

Also, make sure you have good logging configured. You can specify a log file in ansible.cfg or via environment variables. This is especially important if you have multiple people running playbooks.

2

u/FreddyOny 2d ago

Use Foreman for Updates. Look intro it. Use gitlab for config and Scripts

2

u/Open-Adhesiveness-86 2d ago

One thing that'll bite you with mixed distros: don't branch on ansible_distribution for patching, branch on ansible_pkg_mgr, since you'll hit things like Amazon Linux reporting yum vs dnf and SLES showing up as zypper. Also set serial: 5 and a max_fail_percentage so a bad mirror doesn't take out all 50 at once, and gather package facts before and after so you have a diff when someone asks what changed.

2

u/josemcornynetoperek 1d ago
  1. Standarise them to one or two distributions.

  2. Standarise configuration of common services as sshd, firewall or so.

  3. Start using ansible or other solution for save server state.

It will be more work on start but in time you will have a lot more knowledge what exactly is running on servers and any operation will be faster and easier.

2

u/Forward-Outside-9911 Linux Admin 1d ago

+1 - we’ve had manager support to standardise on just one main distro. It’s easier to work with a single maintainable deployment.

And then while you’re at it make it all managed by Ansible or at least using a template ISO with a hardened/security preset.

4

u/databeestjenl 2d ago

use ansbile, see if you can get a CLI LLM tool on there to help make the playbooks, they are pretty easy to read.

3

u/Lost_Term_8080 2d ago

This is the reality of managing linux. There are management tools that help, but the management tools themselves necessitate extensive management. there are tools like saltstack or puppet, you'll add a few more servers to run that as well as add database administration to you list. If its only patching, check out azure update manager. Its by far the most mature linux patch management solution there is and it costs a dollar a month per server. However, once you get patching taken care of, I suspect so much time will be freed up you'll be looking for other things to do.

2

u/Ramiraz80 2d ago

Go buy Jeff Geerlings Ansible book, it is really good :) and he keeps it updated :)

Also watch some of Learnlinux.tv YouTube videos about ansible

1

u/1saltypistachio 2d ago

You can take a look at something like ansible. Should be simple enough to create a unified playbook for maintenance if you do the same thing on each server; or a different playbook for a family of servers.

1

u/wowsomuchempty 2d ago

pdsh old skool, ansible new.

1

u/Sufficient_Job7779 1d ago

Try OpsFabric. Ideal for multios environments. https://opsfabric.io

1

u/Incrediblecodeman 1d ago

You should get really good at doing this way to be good at your job then yea in your off time master Ansible. Then present to your boss how you’d use it. Better then a jr being like “ooOo the way YOU work isn’t good. Can i try out this thing I heard about called ansivle on your production server? Nurrr errrr nnuuuurrrrr 🤪

Innovation requires trust and it’s probably possible in a couple of months. 3 tops. To really make a convincing case. But you every donate 2 weekends a month to this passion project. Maybe find a few old boxes to make a home lab? As in you can’t “learn” but watching YouTube and chatting with ai.

Good luck man boy I wish I was a Linux dude.

Btw is overthewire still cool?

1

u/MinimallyLoquacious HomeAdmin-Retired-Dev 2d ago

If it were me, Ansible. I don't use it on my home machines but it's on my list one of these days. If I were a FT Linux Admin though, I'd be using Ansible.

Also, and not sure why it's not listed under COMPANION SUBS but this is a good subreddit: https://www.reddit.com/r/linuxadmin/

0

u/Lonely-Abalone-5104 2d ago

This might not be a popular opinion but have AI like Claude teach you it. I find it to be a very good teaching tool

0

u/masterofrants Jr. Sysadmin 1d ago

You should install the Semaphore and Ansible UI as well. It'll help you graphically manage Ansible tasks then you can simply create the playbooks from ai use them via github and semaphore.

-1

u/CoreAndCardio 2d ago

Just use InTune. /s

1

u/idrac1966 1d ago

Can it actually do decent like 'desired state' configuration in Linux

1

u/CoreAndCardio 1d ago

No- it was a total joke hence the /s.

Based on the downvotes I see no one quite got that. Bad joke I guess

•

u/raul824 13h ago

if you can atandardize on one distro then nix os also looks like a good fit for this.

-1

u/ElevenB2002 2d ago

Automox is a great, relatively cheap option to Cloud manage Linux (and Windows/MacOS). You can patch, check posture, run "worklets" which can be any type of automated tooling you'd like, etc. It's a few bucks a month per managed node.

-10

u/Evening_Lavishness58 2d ago

just use the AI write you a script to automate the ssh connections and tasks you run

7

u/therealviiru 2d ago

Well hello satan.

-3

u/Evening_Lavishness58 2d ago

crazy how sysadmins can't handle a solution that gets the job done

3

u/Hotshot55 Linux Engineer 2d ago

Does it really "get the job done", though? An AI script to automate ssh connections is basically just a shitty attempt at rewriting Ansible with less than 1% of the actual functionality.

-11

u/alpha417 _ 2d ago

do you have access to Google?

9

u/GhostNode 2d ago

Be a better human.

0

u/[deleted] 2d ago

[deleted]

4

u/pacmanlives Alcoholism as a Service 2d ago

I just write my playbooks to be the app tiers and then have logic in my playbook to run commands if they are different for each distro. Things like an update playbook. All the update commands are in the same playbook and let ansible handle the logic if it’s a Debian, SuSE, RHEL server. I just point it at the inventory and group I want to patch. I will also have another playbook that has no reboot baked in.