r/linuxadmin • • Jul 23 '18

Schrodinger’s Backup

Post image
1.1k Upvotes

59 comments sorted by

23

u/ahandle Jul 23 '18

If the emphasis is on backup rather than restore, you're doing it wrong.

13

u/nixcraft Jul 24 '18

Exactly. BTW I NEVER thought that this image would make it too Reddit. is weird to see my images here. Hehe

45

u/Travis_McGee Jul 23 '18

This seems more like quantum backups. "The value of a backup is unknown until observed"

30

u/Dasbufort Jul 23 '18

Quantum mechanics is what the OP is referring to: https://en.wikipedia.org/wiki/Schr%C3%B6dinger%27s_cat

14

u/Travis_McGee Jul 23 '18

Oh that's right. I forgot about the origin of Schrodinger's Cat

0

u/[deleted] Jul 23 '18

[deleted]

2

u/2drawnonward5 Jul 24 '18

Alright, how are we gonna figure this out?

8

u/TONKAHANAH Jul 23 '18

yup. I do "support" for a big phone company and before replacement devices can be issued people have to a do a factory reset to make sure they're they just dont suck at following previous directions. They almost NEVER verify backed up data before just erasing everything under the assumption "well you guys back it up for me right?" no, fuck off your TOS actually specifically says we're not responsible for your shit if its lost, only you're responsible for your fuck'n baby pictures and shit, backup your data and verify its backed up

4

u/[deleted] Jul 23 '18

'I know exactly where the backup is, but I don't know the speed at which it is traveling.'

1

u/vim_for_life Jul 24 '18

You observed it's in a speeding station wagon ?

3

u/[deleted] Jul 24 '18

The plate said 'HSNBURG01'.

11

u/swampmeister Jul 23 '18

I don't always test my restores in a dev/test environment; I prefer to save that glory to after the storm/ tornado/ hurricane/ earthquake, when the CEO and CIO are standing behind me watching...

16

u/hawerner Jul 23 '18

That's why you should test your backup setup by restoring it to another computer, before you implement that setup in production

17

u/bluehavana Jul 23 '18

And also schedule test restores regularly.

1

u/bluehavana Jul 23 '18

But backups can still feel uncertain, even under verified processes.

7

u/ahandle Jul 23 '18

Keep doing it until they don't.

2

u/bluehavana Jul 23 '18

Sure, I guess it really goes into how far management is willing to take business continuity exercises. I guess my major consideration goes along with my reply to u/cheeseboythrowaway in users, developers and engineers not communicating/understanding the entirety if their data. Processes that happen once a month+ can be difficult to discover in disaster recovery exercises that only last a day/week.

0

u/[deleted] Jul 23 '18

[deleted]

3

u/bluehavana Jul 23 '18 edited Jul 23 '18

No stories in particular; I just know systems get added sometimes without people considering concrete backup plans ("it happens automatically for the rest of the infrastructure, why wouldn't it happen for this") or not coordinating across silos.

Edit: one also can't setup tests for processes outside normal purview. But you're point is valid. The test for such a thing would be an organizational check list for new features/systems.

Edit 2: or software updates where path changes in critical files are overlooked.

1

u/ahandle Jul 23 '18

This is an HR problem, not IT.

2

u/macboost84 Jul 24 '18

Only issue I have with restores is figuring out if everything was restored - especially file servers where you have millions of files, some of which haven’t been touched in a decade.

1

u/mc_dog Jun 17 '24

One could generate a file listing of the original data, (optionally hash it), and compare with a file listing from the restored data.

1

u/GlennHD Jul 23 '18

A backup system... pshh I wish I had working production systems!

7

u/MattTheFlash Jul 23 '18

Backups have no business value.

Only restoration has business value.

3

u/[deleted] Jul 24 '18

More like: The condition of the backup is always less than preferable for the situation in which the backup is required. I.E., "we have all the other user's files...but not yours." or "ohhh.. you need the 11th? Well we got the 10th and the 12th. The 13th? Nah not that either, we have the 14th though!"

1

u/themacman2 Jul 24 '18

Is the tao of backup still around? It was a pretty wise website.

1

u/Slam_Makanen Jul 24 '18

Could also avoid this by using a Datto appliance that automatically verifies bootability of each backup..

1

u/elleGeneralisimo Jul 24 '18

As a person working on improving the almost non-existant backup system right now, this is too real.

1

u/LyokoMan95 Jul 24 '18

I’m definitely printing this out and putting it in our DR center...

1

u/Schr0ding3r_ Jul 23 '18

Haha I deal with this

0

u/Sjeiken Jul 24 '18

Schrodinger’s sub either you get god quality Linux stuff or you get shit like this.

-4

u/mercenary_sysadmin Jul 23 '18

No longer actually true, if you use a modern filesystem with snapshots, per-block cryptographic integrity hashing, and replication.

me@source:~$ ssh me@target
me@target:~$ sudo zpool status backups
  pool: backups
 state: ONLINE
  scan: scrub repaired 0 in 82h1m with 0 errors on Wed Jul 11 10:25:01 2018
config:

	NAME                       STATE     READ WRITE CKSUM
	data                       ONLINE       0     0     0
	  mirror-0                 ONLINE       0     0     0
	    ata-WDC_WD40EFRX-$id1  ONLINE       0     0     0
	    ata-WDC_WD40EFRX-$id2  ONLINE       0     0     0

errors: No known data errors

Yep. It's all good.

18

u/sfrazer Jul 23 '18

Unless you've actually restored the data and verified it's usefulness, what you may have there is a big pile of checksum-verified useless data

-4

u/mercenary_sysadmin Jul 23 '18 edited Jul 23 '18

It's snapshots of the data at hourly-separated intervals over the past weeks. Each one is already guaranteed to be the exact condition of the data at that specific instant in time - every single block of the data, bar none. There's not really any "restore" required in the sense of recovering from a tape drive or a monolithic archive backup like a .tar.gz or what have you; that is the data, in precisely the condition it was in at every moment you have a snapshot for in the stream.

If your data was in working condition at the time the snapshot was taken, it's in working condition in the snapshot; replication guarantees that it's block-for-block correct on the remote end as it was on the source end; a scrub verified that it's still consistent at the time you ran the scrub.

Granted, I should really have shown a list of snapshots as well; with nothing but the scrub you don't know that you have the snapshots you're looking for. Which looks like this:

root@bittybox:~# zfs list -rt snap data/home | head -n 10
NAME                                              USED  AVAIL  REFER  MOUNTPOINT
data/home@autosnap_2018-05-04_19:29:16_monthly    915M      -  2.91G  -
data/home@autosnap_2018-06-01_00:00:01_monthly    737M      -  3.51G  -
data/home@autosnap_2018-06-24_00:00:01_daily      555M      -  4.57G  -
data/home@autosnap_2018-06-25_00:00:01_daily     93.0M      -  4.53G  -
data/home@autosnap_2018-06-26_00:00:01_daily      160M      -  4.54G  -
data/home@autosnap_2018-06-27_00:00:01_daily      353M      -  4.48G  -
data/home@autosnap_2018-06-28_00:00:01_daily      283M      -  4.54G  -
data/home@autosnap_2018-06-29_00:00:01_daily      236M      -  4.58G  -

If I want to know if I've got the snapshot for a particular day (or hour; I spared you the whole list but I keep hourlies for the past 30 hours as well) I'm interested in on the remote machine, I can just do a quick lookup:

root@archivist:~# zfs list -t snap | grep bittybox/data/home | grep 2018-06-25 | grep daily
backups/bittybox/data/home@autosnap_2018-06-25_00:00:01_daily                  304K      -  3.96G  -

Yep, it's present. There is no transformation involved - it's the exact same collection of blocks, in the same condition, each cryptographically hash verified - as it was on the source at that exact instant in time. So it's good. There's no restore transformation required to verify it since it hasn't been transformed in the first place during the backup process.

12

u/sfrazer Jul 23 '18

Have you connected one of those snapshots up to a live system and read the data off it? For example: if that’s a database volume, have you launched the database pointed at that volume? If not, you don’t know if you have a good backup.

By comparison, I restore my database backups to stage every morning. I know that those backups work as restores because I’ve restored them.

That’s what the image is getting at.

“If the data is working at the time of the snapshot” — if you haven’t suspended activity in your database it probably won’t like you making a copy of it while it’s running. Doing a restore tells you if that’s a problem or not.

0

u/mercenary_sysadmin Jul 23 '18

Have you connected one of those snapshots up to a live system

It is a live system. They're connected 24/7.

By comparison, I restore my database backups to stage every morning.

In this particular case - my home directory - there's nothing really to boot up; it's static files. For work-y production stuff, it's usually VM images, and they can be booted instantly on demand, directly on the backup machine.

if you haven’t suspended activity in your database it probably won’t like you making a copy of it while it’s running

Are you still using non-transactional, non-journaled databases? That's a pretty big bug. None of my production stuff has any issues with crash consistency. Guest filesystems are journaling, database engines are journaling, and anything that matters in the way of database applications is transactional.

3

u/vim_for_life Jul 24 '18

So... What are you doing if the data center your in is blown away by a tornado?

Or say chaos monkey accidentally corrupts all love file systems?(even if chaos monkey is a Russian with a crypolocker?)

2

u/mercenary_sysadmin Jul 24 '18

So... What are you doing if the data center your in is blown away by a tornado?

For work stuff, it's not all in one datacenter. The exact breakdown depends on client, but typically production is in one facility and offsite DR is either in another facility or in an owner or VIP's house. There's usually not a "datacenter" involved at all, but there are multiple physical sites with significant geographic separation.

For most of the clients I work with, if a hurricane wipes out the entire city, it'll nail these backups. But not all. For example one client has offices in Miami and New York; each office has its own production and hotspare, and also hosts the offsite DR for the other office. For them, if the entire east coast goes up in a fireball, they'll lose their data... but we've all got worse problems if that happens!

Or say chaos monkey accidentally corrupts all love file systems?(even if chaos monkey is a Russian with a crypolocker?)

Crypto isn't generally an issue because the offsite DR is insanely difficult to access. The DR pulls backups rather than accepting push backups; depending on level of client paranoia you need anything ranging from an OpenVPN key with passphrase all the way up to physical access PERIOD to be able to access them. The ones that require physical access aren't exposing anything at all to the world but Nagios NRPE over an OpenVPN tunnel to a Nagios server, to monitor that new snapshots are arriving as they should, that the storage is healthy, and a few other obvious "is server working right" type things.

For work stuff, this is generally for VM images, and the production and hotspare hosts are also pretty damn difficult targets; the VMs have services available on the network but the hosts do not. My most vulnerable stuff has production and hotspare hosts' SSH open to the LAN, and across an OpenVPN tunnel to the monitoring network, with keypair-and-passphrase required to gain access even if you can get to the port.

I have had to deal with ransomware quite a few times for clients, but the ransomware hits the VMs, not the hosts; the answer there is:

  1. identify the time of the intrusion

  2. identify the source machine(s) for the intrusion (typically somebody's Windows laptop or workstation, which encrypts everything on Windows shares on a fileserver)

  3. ensure source of intrusion is completely disabled (pull network cord, power off completely to make sure wifi not connected in case of laptops)

  4. roll back affected VM(s) to most recent snapshot prior to intrusion time identified in step 1

  5. reboot VM(s)

This is usually about a 10 minute process, at worst.

You didn't ask, exactly, but the remaining possible scenario is an APT who manages to get access to the host(s) despite them being hardened targets with almost nothing exposed to the LAN, let alone the internet. You're basically fucked if you've got that level of attacker who's willing and patient enough to spend months or years on owning you. You can send archives off to cold storage at Iron Mountain or what have you if you like, but honestly if they can penetrate you that thoroughly in the office they can most likely own whatever your offsite cold storage setup is also.

Targets actually worth that level of investment by an APT are kind of outside my experience; I can hypothesize, but you're honestly kind of out of the "general infosec" realm of knowledge and into the "nation-state level opsec" realm at that point.

8

u/[deleted] Jul 23 '18 edited Jul 23 '18

Yeah that's not a backup. ZFS is awesome and all but snapshots are not backups. Faulty hardware can destroy your zpool including snapshots. I don't think even the biggest ZFS fanboys will tell you that you don't need backups.

3

u/koera Jul 23 '18

A zfs fan boy that dispute the relevance of backups have 100% missed the whole point of zfs: data safety.

1

u/mercenary_sysadmin Jul 23 '18 edited Jul 23 '18

I don't think you read that carefully enough. The source machine (where my home directory is, and the snapshots are taken) is bittybox; the backup machine (where they're replicated to, and where I ssh to check the presence of a snapshot at the end of that comment) is archivist. It's another physical machine in a physically separate location.

I don't think even the biggest ZFS fanboys will tell you that you don't need backups.

You're unfortunately incorrect about that one, my role in /r/zfs and /r/datahoarder is very frequently "person who keeps hammering home that redundant storage is not a backup". =)

2

u/[deleted] Jul 23 '18

So the way you were presenting it, it sounded like you were on one machines and calling your snaps a backup.

Beyond that I don't have enough info about your setup and how you are using this all this to agree or disagree with you.

2

u/mercenary_sysadmin Jul 23 '18 edited Jul 23 '18

I think people are glossing over the first two lines of my OP in this thread:

me@source:~$ ssh me@target

me@target:~$ sudo zpool status backups

The actual source is bittybox and target is archivist, but yeah... separate machines, that was kinda my whole point. Beyond the fact that there's no transformative element in the backup, and that the target is live, so there's no transformative "restore" operation needed either.

7

u/crypticedge Jul 23 '18

I've had array backups that "checked out as all good" be in reality entirely unreadable.

If that's all you're doing to validate, you're playing with fire.

4

u/[deleted] Jul 23 '18

Nothing about this is backups.

0

u/mercenary_sysadmin Jul 23 '18

Sure it is. It's another copy of the data, exactly as it was in an atomic instant, on a separate machine in a separate location on separate media. That's exactly what a backup is.

1

u/RagingAnemone Jul 23 '18

What if the house burns down?

2

u/mercenary_sysadmin Jul 23 '18

In this particular case, if the house burns down, the data goes away with it. The workstation and server in question are both in the house.

For work stuff, it's usually a trio of machines. Production, onsite hotspare, offsite disaster recovery. Production replicates to onsite hotspare hourly, production replicates to offsite DR once daily.

The breakfix flowchart looks basically like this:

  • production machine operational? rollback to snapshot.

  • production machine offline? fire up VM on onsite hotspare from most recent hourly (or swap in disks from hot-swap cage on production, then boot from there).

  • production site destroyed? fire up VM on offsite DR.

-5

u/[deleted] Jul 23 '18

is that a windows backup? because I'm fairly certain my backup script on linux has checksums

7

u/crypticedge Jul 23 '18

So does windows backups.

Checksums don't mean restorable. They just mean unaltered.

Test your backups, we run daily virtual boot tests of every single system backed up.

1

u/[deleted] Jul 23 '18

But if a backup is accurate, then how can a restore fail?

8

u/hbdgas Jul 23 '18

You're testing that it was accurate. e.g. nobody else messed up the script, removed the target, etc. And that you didn't forget to include anything. But here's a good one even if everything did save properly:

"Oops, the decryption key for the backup was only on the machine that failed."

5

u/RagingAnemone Jul 23 '18

In many, many ways, but the worst ones are the foreseeable ones (aka human error).

To answer your question directly, how about if the file is corrupted. The backup may be accurate, but the file itself is invalid. "Get me the last backup before it was corrupted".

3

u/crypticedge Jul 23 '18

In a thousand different ways. A simple read error due to the wrong bad block had caused more failed backups than you'd ever realize, while still passing that checksum.

I've seen backups pass checksums and be unusable, and I've seen them fail checksums and be accurate all because the system running the check had a processor issue.

Checksums don't mean the backup is good, and if you treat it like it does, then you're in for a disaster and I hope someone more competent is running a second backup scheme behind you.

0

u/[deleted] Jul 23 '18

so how would you go about ensuring that the data you copied has been copied correctly? I highly doubt you can do much more than comparing the data.

4

u/crypticedge Jul 24 '18

Virtual boot test of image based backups. Manual periodic test restores

4

u/sfrazer Jul 24 '18

Test. Your. Restores.

What would you do in the event a failure that required you to go back to your backup? Do that.

Do it on a staging box, do it on a VM, do it on a spare workstation if that’s all you’ve got, but do the restore. Do it when it’s not an emergency. Do it often enough that when the emergency happens, not only are you ready for it, you’re calm about it.

1

u/[deleted] Jul 23 '18

Either way, Windows 10 Backups reliably fail to restore.

1

u/ahandle Jul 23 '18

now compare checksum on restores