I am using OpenZFS 2.3 and have been taking multiple snapshots of my root-on-ZFS pools and sending them to external drives for backup.
As the host SSD was running low on storage, I read the documentation about bookmarks for the first time and created some for older snapshots of said pools, so I could destroy the oldest snapshots and continue performing incremental send streams.
As mentioned, it was my first time using bookmarks, and when I started reading the documentation, I was thinking about the importance of having some -r option, like with snapshots, but forgot about it while reading. Anyway, I created the bookmarks, destroyed the old snapshots, and had space again (I had like 5 GB left on a 4 TB drive, with no quotas/reservations etc. — only the default slop space was left). I created a new snapshot and wanted to incrementally send the latest snapshot to the backup drives, only to receive some funny error: Error: multiple snapshots cannot be sent from a bookmark.
Then it hit me. Fvvvvvvvck.
zfs list -t snapshot,bookmark shows: I only have a bookmark of the pool — not its children.
So I went back to the ZFS documentation and noticed that bookmark -r was only recently added to the master branch. Welp, that's maybe why my brain shut off when I didn't see it in the v2.3 documentation.
Consulted some LLMs to see if I could fix the situation, but it seems there is no realistic option. I guess I'll have to break the chain and send a new backup to the drives, losing the incremental history :(
On the one hand, I'm kinda frustrated because I've spent my entire weekend since Friday doing nothing but ZFS-related stuff, with no other use of the machine. On the other hand, I'm really glad I've learned this lesson now, while it's non-critical.
tl;dr:
zfs bookmark -r is only available in future releases (not up until v2.4)
Never use zfs send -i/-I from a bookmark (even if theoretically possible), but rather from a snapshot, circumventing the lack ofzfs bookmark -r and the lack ofzfs send -Rfor bookmarks (Error: multiple snapshots cannot be sent from a bookmark)
If anybody knows a way to fix my issue using the existing snapshots on the external drive and the pool#bookmark (no child bookmarks for the pool's datasets) on my internal drive, I'd be glad to hear it.
But first: for the next 24 hours, I'm taking a break from managing ZFS 🐼