2
u/evolutionxtinct Jan 02 '22
Well for once I’m glad we are behind a bit on patches, that we are behind a 3rd party service and we don’t use anti-malware feature! Lol will deal with this on Monday.
1
u/dickielaw88 Jan 02 '22
We use Sophos Email Advanced for spam filtering and such and Sophos Intercept X. Guess I can maybe just leave this setting off now ha. This wasn't a patch that you installed however, it was an automatic update that gets applied without approval since its related to malware definitions.
1
u/evolutionxtinct Jan 02 '22
Thanks that’s what I was trying to figure out as I thought we pushed definitions through SCCM.
9
u/FFSFuse Jan 01 '22
Oh, God. I’m not looking until Monday…..
17
u/ipsok Jan 01 '22
Run the bypass command against each exch server, restart the transport service on each server... scream "fuckyoumicrosoft!!!" and call it a night.
Cheers fellow denizens of admin hell!
1
1
5
4
3
u/XS4Me Jan 01 '22
If you are fixing exchange tonight your next year goal should be migrating to exol.
2
u/dickielaw88 Jan 01 '22
Eh for our size the cost is too much, at least at this point. If we do finally switch over we would need to find a solution to have everything backed up locally to tape every night due to pharma regulations.
2
u/WalksAllRoads Jan 01 '22
Tape? Ewww. Do your regs literally require tape? Or can you back up to an appliance like a Datto or a different medium like RDX drives
4
u/dickielaw88 Jan 01 '22
Requires us to keep all data for 10+ years, so tape is a good, fast, and inexpensive option. Our lto-8 tape drives read/write at 380MB/s, so backing up TBs of data each night is easy. We currently use Acronis cyber protect to do a combination of tape / cloud / local hdd backup.
2
u/WalksAllRoads Jan 01 '22
I felt I was late jumping off the tape wagon with lto-6. In our environment, 1.5TB was taking 18+ hours. BackupExec (I inherited the setup) was never able give an explanation why it was not performing at the speeds of the tape spec. Though I suspect it was environmental because a) lots of tiny files vs fewer larger files and b) DFSr replication of said tiny files. My issue with tape is it might say the backup completed successfully, but actually doing the restore could yield a different result. At least from a full rebuild perspective.
2
u/dickielaw88 Jan 01 '22
Just to add, there was a almost double speed increase between 6 and 7. And yea....backup exec sucks!
1
1
u/dickielaw88 Jan 01 '22
Ah yea LTO-8 is much faster. I think that is the problem most people make with tape backup is not having the right generation. https://en.wikipedia.org/wiki/Linear_Tape-Open
1
u/homing-duck Jan 01 '22
Tape has two speeds, fast and faster. If your LTO6 is not writing at at least 160MB/s, your backup environment can’t feed the drive quick enough.
LTO6 should get 1.5 TB written in just under 3 hours with hours with out compression.
If you had lots of little files, that was more than likely the cause, if you disks can’t keep up with all the random reads.
We go disk to disk to tape with full image backups of servers and never get less than the max uncompressed speed of the LTO tape we using (mix of LTO8 or LTO M8)
2
u/bagaudin Acronis Community Manager Jan 12 '22
Thanks for using our software /u/dickielaw88! Come visit us at r/Acronis if you'll need any assistance!
2
u/carp3tguy Jan 01 '22
I know a lot of people aren’t a fan but one plus about tapes is you have a contingency in the event of a ransomware attack. Some cloud platforms are resilient to this but nothing can really beat an offline backup of some kind.
4
Jan 01 '22
Are you personally paying for it?
Nah. Then it’s the cost of business. You deserve to sleep too king.
11
u/disclosure5 Jan 01 '22
I've got 300+ printers doing scan to email to a hybrid Exchange server. This is documented as recommended by Microsoft even for Exchange Online users as the best way to manage mail relay on a large scale. I'm not walking around reconfiguring these to use an Exchange Online connector when half the time it'll start a discussion about brand new devices that don't talk TLS 1.2 and are going to be dicey using with Exchange Online.
I'm fixing Exchange tonight, and I don't feel the above is applies to every case.
4
4
u/WJRobbins Jan 01 '22
Because there are never problems there...
2
u/Teriyaki350z Jan 01 '22
Until a cert expires on Microsoft's side.
XD Good times3
1
u/dawho1 MCSE: Messaging/Productivity - @InvalidCanary Jan 01 '22
I ran IRM in several environments and had occasional hiccups.
The first time I had a client that was in the cloud for IRM and MS botched a cert renewal I felt for the first time that I, too, could work for a big company and still fuck up once in a while!
1
Jan 01 '22
Maybe MS shouldn’t break it on new year’s with their shit updates? The next mail migration I do will be off of Microsoft altogether.
3
u/WalksAllRoads Jan 01 '22
I always knew about the 2043 coding timebomb, but assumed after Y2K everybody bypassed signed longs and went straight to using unsigned longs for anything involving storing a date as an integer. Someone whose job it is to review Exchange's code should be sacked. Or at least lose their license to code. Wondering if there's any fun nuggets like this hiding in other critical software code?
2
u/SithLordDooku Jan 01 '22
I've disabled the filter about an hour ago and i still have 15k messages in my submission queue. I've restarted all the transport servers (4) multiple times and still no movement. I've even added the servers to the bypass. Any suggestions?
2
u/YouDamnRightItIs Jan 01 '22
Disabling actually took effect right? I had to run the disable command a couple times after running a get and seeing the bypass still showing false. It can take 10/15 mins to start clearing after the service restart too - not being impatient and restarting the service too quickly? Either way you posted 4 hours ago so hope all is fine now!
2
u/SithLordDooku Jan 02 '22
Ended up being a transport rule that needed to be disabled. All good now!!
1
u/WalksAllRoads Jan 01 '22
Also, as an aside, did this effect 365 in any way? I assume if it did, perhaps their admins bypassed the antimalware engine before anyone noticed.
3
2
u/YouDamnRightItIs Jan 01 '22
We have on prem 2016 servers for relay & management purposes so it did with us. Our 365 side has been fine though.
1
u/ddildine Jan 02 '22
If you see the 5300 event log, but do not have a backed up queue, is there any reason to not run their provided update script anyways? (one of three servers did have the malware agent enabled)
Thanks
9
u/dickielaw88 Jan 01 '22
https://www.reddit.com/r/sysadmin/comments/rt91z6/exchange_2019_antimalware_bad_update/?user_id=411891742&web_redirect=true