r/sysadmin • • 13m ago

General Discussion It seems there are others who think thin client OSes are cool ...

0 Upvotes

Over the last few weeks I gave my pet OSS thin client project a massive upgrade and changed its name: Say hi to ThinFloe.

ThinFloe is ubuntu based, can live entirely in ram, boot of a stick or from the network. It uses wayland as its graphics subsystem and it comes with a custom menu application written in rust. And it looks pretty stylish I think :)

Details on github: https://oposs.github.io/thinfloe Enjoy!

ps: if you wonder what oxulnk is, ask me :)


r/sysadmin • • 37m ago

Question Uninstalling SCCM on AD server question

1 Upvotes

I inherited some servers. Two are an AD server replicating each other. During this time we have migrated from VMware to proxmox. Since then, we have had an issue where logging into the server will take roughly 2-3 minutes with the screen displaying "Applying Configmgr User State Management Extension. policy" I have discovered it is SCCM causing this issue. Talking to other users they have never used SCCM but apparently it was setup and pointed to a server that wasn't migrated.

I tested removing it from another server that is no longer is use but thats getting the same issue. I ran ccmsetup.exe /uninstall with success and error goes away.

Question is, does anyone know if I run this on the AD servers what issues could happen? Never used SCCM so would like to try and be ready for anything.


r/sysadmin • • 3h ago Popcorn

Root cause of FAA shutdown: Newly installed HPE Juniper router with misconfiguration that broadcast data with a thousand connections every 4-5 seconds

2.2k Upvotes

They didn't mention which configuration this was specifically, but now we're actually getting more specific detail and this is worth a read if you're interested. Here's the crux of it via WSJ through MSN to avoid the paywall:

"Data from that router and another at the Texas location overwhelmed the system, government and industry officials said. It led to 17 out of the FAA’s 22 centers that oversee high-altitude flying to experience disruptions in communications, Bedford said, resulting in the FAA ordering the ground stop.

There are about 60 Juniper routers active and operating normally in the FAA’s air-traffic network, said a government official familiar with the system. It wasn’t clear if the problems Friday resulted from a bad router or if L3Harris didn’t configure it properly.

“Was it a faulty installation? Was it a faulty configuration? Or was it just a bad piece of router technology that we put into this system that propagated these errors?” Bedford said Saturday, noting the FAA is still investigating the situation.

An HPE spokesman said the Juniper devices appeared to have been working as intended. “They did not fail,” he said.

FAA officials identified a feature on the router that appeared to be resulting in the data overload, according to government and industry officials. A government official described it as a broadcasting feature that sent data more broadly than necessary, and said the agency has since disabled it on all the Juniper routers."

https://www.msn.com/en-us/news/other/the-single-router-in-southern-arkansas-that-ground-air-travel-to-a-halt/ar-AA2hs7wV


r/sysadmin • • 6h ago

Question How do people handle updating their security baselines?

6 Upvotes

So this thing has been keeping me busy for a while now.

We manage a fleet of devices and deploy sets of policies on them. Now one of the sets of policies is a security baseline which regularly gets updated.

At the moment when the new set gets published we deploy the baseline through a build kit and go through it by hand to see what exceptions we have to put back, find out which settings are new and which are obsolete.

It is doable, however each time it takes a lot of time and mental gymnastics to find out the aforementioned diffs.

After we're done we can push the new config for testing and deploy it at other tenants as well.

What do people do / use to smooth out this process? I am fairly certain there must be a better way.


r/sysadmin • • 14h ago

Question Boardroom Cam under $1500 AUD

0 Upvotes

I see alot of reccomendations for the Logutech Rally but its well beyond our price range.

What is a good alternative, sound not an issue we have that covered already.

TIA


r/sysadmin • • 14h ago

ChatGPT Detect AI generated text in email drafts?

0 Upvotes

Recently discovered we have employees using personal AI accounts by putting company data into an email draft and then opening the draft on their personal device.

From there they copy it into ChatGPT or Claude and do what they do. Sometimes they copy the results back into a new draft email and retrieve it on their work PC.

Management of course wants to maintain access to email from personal devices.

Suggestions? I know that AI generated text is watermarked or can otherwise be detected with a degree of confidence, so this is technically possible, but I have not found any tools which can do so.


r/sysadmin • • 16h ago

US aviation shutdown blamed on L3 Harris installed router

844 Upvotes

r/sysadmin • • 17h ago

General Discussion Am I looking for a unicorn or are these skill requirements a reasonable ask for a job posting?

19 Upvotes

Howdy all. I'm making this post in hopes of affirmation or a reality check. I run a small visual effects studio and through self-teaching over the last six years have implemented a fully remote cloud based infrastructure. There is some growth on the horizon and it will likely be happening at a speed that I will not have time to figure out myself. Thankfully the growth will come with a budget to find people to support that growth much better than I would probably ever be able to.

What I want to know is the idea of a single person having the knowledge and ability to implement the below crazy? Is this job posting describing multiple roles? I can't state enough that I am NOT an IT professional and have only learned what I have needed in very isolated ways in order to accomplish our goals as a studio so I am not actually sure what role/responsibility split should be.

I put 7+ years based on the time it took me to build out my own knowledge of the posting contents. For me, this isn't about "cheaping out" by combining it all so much as I'm not sure if this needs to be broken up into multiple hires due to my own knowledge gaps.

The short term goal is to go through some security audit processes and modify/build out our current infrastructure as needed. Ideally there would be enough work coming down the pipe that we would bring someone on to build a small team dedicated to it.

Any advice is appreciated. Post is below, thanks!

About Us

We are a boutique, cloud-forward VFX studio based in Vancouver delivering visual effects for film and episodic television.

The Role

We’re looking for an experienced CloudOps Architect with a lean towards SecOps. You will lead the analysis, architecture, implementation, and documentation of our growing cloud infrastructure. In your estimation, if required, you will outline additional roles needed to support this process.

Requirements

• 7+ years in IT with direct, practical experience in CloudOps and SecOps within AWS.

• Demonstrable experience utilizing principles of least privilege operating in Windows and Linux environments.

• Knowledge of principals and implementation of Cloudflare Zero Trust network/VPN policies and associated configurations.

• Demonstrable knowledge of AWS VPC, EC2, AMI configuration, operation, and maintenance.

• Experience with AWS Network Firewall services

• Demonstrable knowledge of AWS ACL/IAM policy and role implementation.

• Understanding of network segmentation as it relates to different aspects of the overall network infrastructure (security/logging, staging, production, etc.)

• Great written and verbal communicator. Able to translate complex ideas into practical targets and goals for a small team.

Nice to Have

• Experience supporting Trusted Partner Network implementations and Content Security Best Practices.

• Experience in VFX, post-production, or media-based pipeline environments.

• Knowledge of Amazon FSX Lustre and SMB file servers

• Knowledge of Google org administration and GCPW

• Knowledge of LDAP services/implementation

• Knowledge of Parsec remote desktop client and its associated API

• A dog or cat that you can wave around in front of the camera during company video calls.

---------------------------------------------

**EDIT**
Based on some of the quick responses-- yes I'm not surprised that the ask for someone like this in a single role would be in the $200K+ range. Ultimately I don't want the real job posting to be ignored and I don't want to look ignorant as the potential employer. Thanks again.

**EDIT EDIT**
If you were going to break these responsibilities into separate roles, what would you expect the salary to be for each role?

**EDIT EDIT EDIT**
I'm not sure what to expect exactly timeline wise but I was picturing a 3-ish month contract-- evaluate/design/implement and it would be fully remote. Of course if someone gets in and it looks like it's going to take longer than that so be it but I don't know what I don't know.

-------------------------------------------------

** EDIT EDIT EDIT EDIT**
Have modified things a bit based on feedback from folks, please continue to chime in if you'd like... very much appreciate everyone's thoughts-- we're trying our best to get this right in the current hellscape that is the VFX and tech job market...

The Role
We’re looking for an experienced CloudOps Architect with a lean towards SecOps. You will lead the analysis, architecture, implementation, and documentation of our growing cloud infrastructure. After initial evaluation and consult, in your estimate, if required, you will outline additional roles required to support this process if the scope is deemed too large.

Desired Qualifications
• 7+ years in IT with direct, practical experience in CloudOps and SecOps within AWS.
• Demonstrable experience utilizing principles of least privilege operating in Windows and Linux environments.
• Knowledge of principals and implementation of Cloudflare Zero Trust network/VPN policies and associated configurations.
• Great written and verbal communicator. Able to translate complex concepts and tasks into practical targets and goals for a small team.

Nice to Have
• Experience supporting Trusted Partner Network implementations and Content Security Best Practices.
• Experience in VFX, post-production, or media-based pipeline environments.
• Knowledge of Amazon FSX Lustre and SMB file servers
• Knowledge of Google org administration and GCPW
• Knowledge of LDAP services/implementation
• Knowledge of Parsec remote desktop client API
• A dog or cat that you can wave around in front of the camera during company video calls.


r/sysadmin • • 18h ago

Windows Hello Passkeys - Unable to register

0 Upvotes

Hello,

Windows Hello sign-in to M365 on my laptop was working until a few days ago. Then it started looping. My password and the Authenticator passkey on my phone (QR code) still work, but I want Windows Hello back.

Laptop is Entra joined and Intune managed.

What I’ve tried:

  • Required MFA re-registration and removed my methods

  • Ran certutil -DeleteHelloContainer, rebooted, set up a new PIN

  • Settings > Passkeys shows one passkey for my account. It can’t be removed (“used by the system”)

  • Adding a passkey from Security info fails with “You already registered this device”

  • Saving it to my password manager fails with “Passkey not registered”

dsregcmd shows AzureAdJoined YES, NgcSet YES, PRT YES.

My guess is that the MFA reset removed the WHfB registration in Entra and left the local key orphaned, but a new PIN should have fixed that. Anyone run into this?


r/sysadmin • • 20h ago

Rant Who installs a router in a late Friday afternoon?

1.8k Upvotes

So it looks like L3Harris installed a new router and didn’t enable Spanning Tree Protocol.
That took down air traffic control yesterday


r/sysadmin • • 22h ago

General Discussion drowning in API access requests

24 Upvotes

I admin a saas PPM that is used by the entire enterprise, more than 10K active users.

It has a REST API and is fairly extensible. Historically, only developers in our org have requested API access which, for this product, requires creating a distinct API account not associated with the user. So we have, let's say, 40 of these API accounts that have accumulated over the past few years.

Our company is fairly generous with AI access and credits. What that has meant for this application is a huge influx of users who are now developing their own micro front ends or custom HTML dashboards to produce highly customized experiences. Which they love because generally speaking, no user has ever loved the out of the box experience.

Now that users can leverage AI to improve their experience with the app, we are getting hit up daily for API account requests. Hundreds of users are just vibe coding their own front end in Claude, which works great for them as a local artifact, but gets all of its data via the user’s locally auth’d MCP.

Once they realize that what they've built is not easily distributed, their AI gives them directions for how to request or procure the developer API access that will unblock them.

That's the point at which they contact my team and casually demand “oh yeah, just give me my API account now, thanks!” We get these requests almost hourly now. Sometimes this is for team-based development, and the user requesting this is working on something that they were told to develop by their manager. I have no interest in getting in the way of those use cases.

But 3 times out of 4, this is just an individual who thinks they are a superior product designer and become nearly belligerent if you dare to challenge the appropriateness of giving them developer roles or developer access when they themselves have never coded anything outside of an AI harness.

Is everyone getting hit up with these types of requests? How do you effectively say no to anyone? In our case, when we reject a request, the user escalates through their own management and emphasizes how we are not supporting the company's effort to use AI to do better work faster. Then we look like the asshole. Never mind the fact that these hundreds of users have never used the API and have no idea if what they're producing actually shows correct data or not while fragmenting the UX in such a way that the original app cannot even be recognized.

And of course, no one outside of my team cares that we're responsible for maintaining the documentation on these API accounts so that we understand exactly what every non-human account is doing, where it does it within the application, and which users or teams are responsible for maintaining the solutions it supports.

What are effective and appropriate barriers we can stand up so the application is only extended by knowledgeable individuals and without creating hundreds of throwaway API accounts that can never be removed from the system?


r/sysadmin • • 23h ago

Question - Solved Idrac9 just stopped sending e-mail alerts

13 Upvotes

Got a head scratcher for you.

1 of my 3 Idrac9 equipped servers just up and quit sending alerts automatically. Near as I remember, it happened around the time Google enforced Oauth, but switched them all to Smtp2go.

Test email: passes and received
Test Event: passes and received
Normal chatter on monthly patching reboots: Crickets

Alerts - e-mail is checked

Give me the off the wall stuff to look at, I’m fresh out of normal.


r/sysadmin • • 23h ago

Career / Job Related Career Advice - SysAdmin to Management - Worth it?

3 Upvotes

Hi r/Syadmin haven't posted here a ton lately. However, always looking at sub for latest news/outages/rants/etc. so thank for everyone contributions. I was hoping to pick the brains of people that have been in a situation where they're been asked to transition from a systems administrator role to a managerial role.

In my own case, I'm a senior systems administrator who is involved heavily in endpoint architecture/engineering, M365, and serve as backup on the infrastructure side and still try to contribute to other buckets as time permits. Recently, I was asked by management to help cover one of our management roles that looks after technicians, service desk, and procurement. The reasoning was since internally to organization I know so much and serve as the top escalation tier I could help cover and bring with some fresh new ideas on things that could be streamlined and/or potentially automated. The previous manager wasn't very technical.

I helped cover the role for the sake a being a team player and did enjoy it. The person I was covering for was out on medical leave. When they came back from medical they submitted their two weeks. So I was asked again to cover for role again until they found somebody. They're asking me if I would be interested and what I would change. My main piece is I would really still like to retain is staying technical but not really too sure if I would be able to do that while having to do tier 1/2 plus meeting with stakeholders, budget, etc. and other managerial stuff.

Have any of you here in r/sysadmin been in a similar predicament? Did you still feel like you were a sysadmin while being manager or did the managerial side completely take over any technical piece. I'm honestly 70% don't do it I'm happy to fill in but not really thinking I'm going to remain technical and that is what I enjoy. However, I do like training and coaching people technically. The reprimanding, budget, politics not so much.


r/sysadmin • • 1d ago

Zyxel USG Flex 100 - Force all LAN DNS requests to ZyWALL DNS instead of blocking them?

2 Upvotes

I have a USG FLEX 100 running firmware V5.45(ABUH.0).

My LAN clients receive the ZyWALL as their DNS server via DHCP, and I've already blocked outbound DNS (TCP/UDP 53) to the Internet.

What I'd like to do is transparently redirect any client that tries to use an external DNS server (e.g. 8.8.8.8, 1.1.1.1, 9.9.9.9) back to the ZyWALL's DNS service a , rather than simply blocking the request.

Desired behavior:

I've looked through NAT, Interface, and other network settings but haven't found an obvious DNS hijack/redirect feature.

Has anyone successfully configured DNS interception/redirection on a USG FLEX 100, or is blocking external DNS the only supported option?


r/sysadmin • • 1d ago

Microsoft status page

15 Upvotes

Their page currently shows “all products are operational”. So, everything is good now? I haven’t experienced any issues in the past day now but last I checked they were still monitoring.


r/sysadmin • • 1d ago

Linux Managing multiple linux servers

31 Upvotes

Hi guys,

currently i'm working as junior linux sysadmin and managing about 50+ linux servers (different distros, services etc.). For every task i connect to servers with SSH and in my opinion its really inefficient. My main task is patching the servers and this would be great to implement ansible.

Do you have any recommendation about best practises with ansible? Like good articles, books, popular YouTube vids?

ps: its my first post 😄


r/sysadmin • • 1d ago

Parker Colorado Area - Comcast Business

3 Upvotes

Is anyone else experiencing a problem with their L2TP-IPSEC tunnel not establishing connection after Comcast doing maintenance last night!?

I tried to be proactive when I saw the internet go down at my Parker site at 2AM last night; gave Comcast a call and they said they’re doing maintenance.
They told me it would come back up at 7AM mountain time, well, the internet is back up but our ERP cloud SaaS is not working, because our site to site is down!

Comcast fiber guy said he doesn’t know if the SDWAN Folks work on the weekend, WTF XD

just wondering if anyone else in that area is experiencing a similar issue.


r/sysadmin • • 1d ago

FAA and satellite

0 Upvotes

So why can’t the pilots and towers use satellite or different modes of communication? Why does it all rely on a router at ohare?


r/sysadmin • • 1d ago

How do you stay on top of CVE (vulnerability) information without spending your entire day just monitoring security alerts?

65 Upvotes

It feels like a constant stream of new vulnerabilities and attack techniques often fueled by AI is emerging these days.

As an IT administrator, I find myself responding to these issues far more frequently than in previous years often having to handle urgent matters during weekends or after hours. How do you decide which vulnerabilities require immediate patching and which ones can wait? Do you primarily follow vendor recommendations, or do you use tools that validate these alerts against your actual operational environment?

I’m also curious: are there any specific sources of information that have helped you identify critical vulnerabilities before they escalate into major problems?

I imagine many of you are facing similar challenges I’d love to hear how you handle this.

* I apologize for not being clear. I would like to know if there is a specific category for vulnerabilities that require immediate action such as those for which patches must be applied right away and what procedures you have in place to handle vulnerability patching (for instance, whether all necessary patches are applied immediately as they arise).


r/sysadmin • • 1d ago

Question How much do we depend on Microsoft 365 for critical services?

0 Upvotes

Following yesterday's Microsoft 365 outage, I was wondering how others see the risks of relying on SaaS services.

I work with several small hospitals using Microsoft 365 for their email. This outage was a good example I could use to make management understand the risks of depending on an external provider for such an essential service.

Email is also used by many applications to send OTP codes for authentication. When the email service goes down, users may lose access to applications that are working perfectly fine.

Have you been impacted too? Are you considering moving away from Microsoft 365, or even hosting your own email servers?

What's your position on this?


r/sysadmin • • 1d ago

General Discussion Tailscale for NAS-to-NAS backups behind Bell GigaHub?

0 Upvotes

Working with a client who wants to back up their office UniFi UNAS to another NAS at a separate location rather than use cloud backup.

The issue is that the remote location uses a Bell GigaHub, which doesn't support traditional bridge mode, making a conventional site-to-site VPN difficult.

I've successfully tested Tailscale using a Windows PC as a subnet router with a /32 route to work around overlapping subnets. Transfers are working at approximately 20 MB/s.

My question: Is Tailscale a reliable and appropriate long-term solution for automated business backups in this situation, or would you recommend another approach?

The client wants to proceed without paying for cloud backup storage. Looking for advice on reliability, security, and potential issues.


r/sysadmin • • 1d ago

Windows 0-day "SignDrop" published, "this makes this vulnerability unique as it can be used to turn off any antivirus/EDR."

688 Upvotes

https://arstechnica.com/security/2026/11/windows-0-day-dropped-by-disgruntled-researcher-hands-hackers-system-rights/

https://github.com/MSNightmare/SignDrop

If everything is chained correctly, funnily this legit security mitigation can be used against itself and allow an elevation if privileges. Everything is done properly by Microsoft here but the logic allows abuse. What makes it funnier is the fact that it can be used against any antimalware engine to bypass their self defense (or tamper protection) mechanism as you can trigger arbitrary file creation in the context of their own antimalware process so from the perspective of their kernel drivers, the file creation request is coming from their own legit process so it passes, this makes this vulnerability unique as it can be used to turn off any antivirus/EDR.

11th zero day dropped by this person, in what 6 months? I'm tired boss.


r/sysadmin • • 1d ago

Is anyone using AI as an assistant? Giving it access to systems (monitored) and giving them tasks? like create this GPO, add these accounts, etc?

0 Upvotes

I am self-employed responsible for about 600 workstations and servers across 12 orgs. I use AI each day to create PS scripts and to guide me with best practices administering and securing client systems. I copy/paste the scripts and share the results with Claude. I do have over 30 years of experience in IT so I am not just blindly allowing it to do what it wants. I feel a next step might be giving it some limited access to check backups for me, to run certain approved scripts, and perform routine tasks across all clients. Thinking along the lines of having an assistant. I also know it can be risky. Just wondering where others are with this and if they are doing it or considering it.


r/sysadmin • • 1d ago

General Discussion With all the outages the past few months, what do you actually think the future of system administration looks like?

75 Upvotes

I'm wanting to make this post out of an interest as to what the future of system administration could look like with all the outages.

I want to start by highlighting the major outages the past few months:

- Azure West US went down in July

- The next day, AWS took out Apple Pay, DoorDash, Reddit and other platforms

- Google's new region in Melbourne fell over about a month after it launched

- PHP had a bug last month which impacted many things across the globe

- Microsoft's outage over the past few days

It feels like the answer from Managers to employee burnout is "use AI", whilst also cutting headcount because of AI.

As someone who isn't in the industry but does know a teeny weeny bit about it, I was wondering, what could the future of being a system admin look like?