r/zfs • • Jun 23 '21

WARNING: deleting posts == instaban

434 Upvotes

No dirty deletes.

If I catch anybody else deleting their question and all their comments on it immediately after getting an answer, they're getting an instant banhammer.

Half the point of asking questions in a public sub is so that everyone can benefit from the answers—which is impossible if you go deleting everything behind yourself once you've gotten yours.

It's been a rule for months now.

This rule has been in the sidebar for months now, but apparently people aren't noticing it. So here it is in a big ol' ugly sticky. Yes, we mean it, yes, you will get banned. You have been warned.


r/zfs • • Mar 22 '20

ZFS topology FAQ: What's a zpool? What's a vdev?

292 Upvotes

Hi, ZFSers!

If you're new to ZFS, you're probably not entirely solid on how the structure of the system works. This post is here to help, complete with lovely ASCII diagrams (that don't require loading other links, since Reddit can't embed images without use of browser extensions).

In word form, it's fairly simple: The top level structure is the zpool. A zpool consists of one or more vdevs. Each vdev consists of one or more disks.

Standard vdevs are where ZFS stores your files. Support vdevs (all of which are optional) may accelerate your zpool by offering pool-wide services which decrease latency and increase throughput in various ways.

ASCII art time!

Before we go any further, let's look at that promised ASCII diagram.

—————————————————————————————————————————————————————————
|   top level: the zpool. zpools are self-contained.    |
|  A zpool contains zero or more support vdevs, which   |
|  service the entire pool as a whole, and one or more  |
|    standard vdevs, which contain the pool's data.     |
|                                                       |
| Each vdev consists of one or more disks, and can have |
| one of several topologies: single disk, n-way mirror, |
| RAIDz1/2/3 (stripe with 1, 2, or three parity blocks) |
|                                                       |
|              Support (optional) vdevs                 |
|    --------   ---------   ----------------------      |
|    ! SLOG !   ! L2ARC !   ! special allocation !      |
|    --------   ---------   ---------------------!      |
|                                                       |
|            Standard (main storage) vdevs              |
|  ————————————————————————  ------------------------   |
|  | —————— ------ ------ |  ! —————— ------ ------ !   |
|  | |disk| !disk! !disk! |  ! |disk| !disk! !disk! !   |
|  | —————— ------ ------ |  ! —————— ------ ------ !   |
|  ————————————————————————  ------------------------   |
|  ------------------------  ------------------------   |
|  ! —————— ------ ------ !  ! —————— ------ ------ !   |
|  ! |disk| !disk! !disk! !  ! |disk| !disk! !disk! !   |
|  ! —————— ------ ------ !  ! —————— ------ ------ !   |
|  ------------------------  ------------------------   |
—————————————————————————————————————————————————————————

Beautiful, right? A box enclosed by |———| is mandatory; one enclosed by !----! is optional. So we can see the visually reinforced concepts outlined above: a pool contains at least one storage vdev, which contains at least one disk. And we might or might not have support vdevs.

Note: the diagram only shows up to three disks in each vdev and four vdevs in its pool, but that's a visual limitation of the diagram, not an actual ZFS limitation. There are no hard limits to disks-per-vdev or vdevs-per-pool; although there are some practical limitations in real life, they're nowhere near this small.

Fault tolerance: zpool level

There is no parity or redundancy at the pool level; all fault tolerance is at the vdev level. If you lose any storage vdev (or if you lose a special allocation vdev), you lose the entire pool along with it.

Let's repeat that one more time: if you lose a vdev, you lose the whole pool! Think about that before you decide to skimp on parity or redundancy in any single vdev.

The exceptions are L2ARC and SLOG vdevs. The pool will still be functional after the loss of either L2ARC or SLOG (although if the SLOG contained data, and was lost after a system crash, the data in the SLOG will be lost).

NOTE: for the most part, writes are distributed among vdevs in proportion to the amount of free space each has available. This allows the pool to continue operating until all vdevs have become full, even when they are differently sized or unevenly balanced. There is some provision in newer ZFS versions for writing preferentially to lower-latency vdevs in some cases, but it should not be relied upon to materially alter the general write distribution behavior.

Fault tolerance: vdev level

Any vdev—including the support vdev types—can be fault tolerant, or non-fault-tolerant. It's up to you, the zfs admin, to make good choices here.

These are your vdev topologies:

  • Single disk vdev—no fault tolerance. Lose the disk, lose the vdev; if that vdev was storage or special allocation, lose the zpool with it.

  • n-way mirror vdev—fault tolerance through redundance. Each disk in a mirror vdev has the same data on it as every other disk in the mirror vdev. Usually seen in 2-way mirrors, but 3-way and up are entirely possible. The vdev survives as long as any single disk survives—but once it's down to one disk only, it's "uncovered", meaning that any existing corrupt blocks are permanently irreparable, as well as any blocks which become corrupted while the mirror is down to a single disk.

  • RAIDzn vdev—all disks in the vdev are striped, with n blocks of parity per stripe, where n=1, 2, or 3. If you lose n disks in this vdev, it becomes "uncovered". Losing n+1 disks makes the vdev entirely inoperable.

Support vdev types

  • L2ARC: The Layer 2 ARC is a simple read cache, which operates a step below the ARC (Adaptive Replacement Cache, which is in RAM). This vdev type is nowhere near as useful as many new users expect it to be—even a fast SSD is tremendously slower than RAM, and the L2ARC isn't actually an ARC at all, it's a simple ring buffer populated with blocks evicted from the ARC. While an L2ARC can be useful in certain circumstances, the odds are extremely high that it's not useful in your circumstances. WARNING: indexing L2ARC eats system RAM; an over-large L2ARC can end up costing you far more RAM (which then can't be used for ARC, standard filesystem cache, or applications) than it's worth. Caution is strongly advised here.

  • SLOG—the Secondary LOg Device is not, as commonly misunderstood, really a write cache. It's just a really fast place to store the ZIL (Zfs Intent Log). The ZIL is a place on non-volatile storage where ZFS can immediately dump a copy of any dirty data present when sync() is called, in order to honor sync()'s request to ensure data in flight is crash-safe before the system does anything else. This only affects synchronous writes, and the data in ZIL is only read from after a system crash. In normal operation, sync writes are dumped immediately in the ZIL, then written out as normal, later, in TXGs (TransaXtion Groups) onto main storage along with async writes. Having a SLOG on fast SSD storage can dramatically increase throughput and decrease latency in a workload with many sync writes; it won't have any effect at all on a workload that's already asynchronous. NOTE: a SLOG does not need to be any larger than the amount of data which might flow through the system for vfs.zfs.txg.timeout seconds—5 seconds by default. But SSD endurance is measured in total drive writes per day, and a SLOG sees an enormous amount of writes on systems that need it: so you'll want bare minimum 256GB for any non-Optane SLOG. (Optane can be smaller because it has massively higher write endurance than "normal" NAND flash SSDs.)

  • special (allocation) vdev—the new special vdev class, only present in ZFS 0.8 and above, allows for storage of system metadata (and, optionally, very small writes) separately from main storage. This allows extremely latency-sensitive operations (file and directory metadata, small database operations, etc) to be stored on the special type vdev, which should be constructed of very fast SSDs. WARNING: if you lose a special allocation vdev, you lose the whole pool with it—we do not recommend using a single disk topology here!


r/zfs • • Feb 18 '22

A simple (real world) ZFS compression speed an compression ratio benchmark

226 Upvotes

Introduction:

As we all know ZFS is an awesome filesystem. It is for a reason the default filesystem in Proxmox and TrueNAS and also commonly used in OMV, Ubuntu and others.

As a result it is often used as a storage backend. It seems to be common knowledge, that it is good to turn compression on, but bad to turn deduplication on. A lot of these recommendations come from the server environment where there is a ton of memory and even higher demand on performance.Since I am in the progress of setting up a new home server I wanted to investigate these claims a little bit further for end user friendly tasks.The new server – like the old one – is used as a central storage and backup location. Most of the data on it is written rarely but read a bit more often. So write speed is (within reason) not the most important figure of merit.

I am interested to see how ZFS is performing on rust disks and how efficient it can handle different types of data.In a first set of measurements I compare the performance and efficiency for the most common used compression algorithms: off, on, gzip, lz4, zstd.

Hardware:

The hardware is my old (now retired) fileserver.

CPU: i7 2600 u/3.8GHzRam 8GB DDR3

1x 750GB WD Blue 2.5” Drive (OS)5x 3TB WD Green 3.5”

This server has aged quite well for a 10 year old machine. The CPU is still reasonably fast. The two big downsides is the (relative) low amount of RAM and no SSD as a boot drive. Also the storage HDD are not up to current standards and the power consumption is to high for the performance provided…

Still it is fast enough to give some reasonable indicators. Any newer file server with more RAM and an SSD – maybe ZLOG or L2ARC device will perform better. On a positive note: If I don’t see a performance impact on this machine I consider the tested configuration as “suitable for use”.

Software:

Fresh install of Proxmox 7.1 kernel 5.13.19-4-pve with OpenZFS 2.1.2-pve . All Packages updated 15.02.2022.

I chose Proxmox over TrueNAS because I am a bit more familiar with the distribution and had the Install media ready to go. There were no VMs running on the system.

The ZFS pools and benchmarks were all created using the command line. This should eliminate most bottlenecks and latencies associated with web interfaces (not that it is relevant for this test) …

Preparation:

Before the test runs Proxmox was installed freshly on the OS drive. The OS drive is compressed via ZSTD. Proxmox is than updated to the current package versions (current at 15.02.2022).

The 5 identical WD green drives (that used to be a raid5 for storage) have been cleaned and checked for smart errors. For the tests each of the drives will be hosting one pool that is completely re initialized freshly for each test data set. All pools have the default blocksize of 128k, ashift of 12 and deduplication turned off. The only different setting is the compression.

All the test data is copied onto the internal 2.5” rust drive compressed via ZSTD (yes I changed it from default LZ4 to ZSTD). Please note, that this drive also limits the total transfer speed in some cases! However I consider the achieved speed as a useable lower limit.

The first dataset I tested are 25739 default documents. (Actually it is my documents folder from my laptop.) This is a wild mix of Word Excel, Powerpoint, pdf and source code with the occasional jpg or png in the mix. Some of these documents are duplicates (aka. filename version control).This is a reasonably good representation of default office documents.

The second dataset is uncompressible data. For this purpose I used my picture library of 27066 files. All JPGs either from lightroom or directly out of camera. There might be duplicates of the same picture in there if have sorted it into several subfolders .This is a reasonable representation of any kind of uncompressible sorted data.

The third dataset are VM Images. The VMs are a mix of 6 different Linux VMs. Some of the VMs have been running for 3years+ with regular updates and other activities. Some of the VMs have been based of the same master image but diverged a lot from there over time.This dataset is to estimate the efficiency of ZFS for a small scale VM based home server.

Tests:

The first dataset consisting of documents has been transferred onto the pools. The total transfer time (and thus speed) is monitored via the time command. Please Note that this speed is limited by the read speed of the ZSTD compressed donator disk. For my testing purposes reaching this speed is “good enough”

offongziplz4zstd
Data [MiB]58163.247923.245465.647923.245363.1
Compression1.001.211.271.211.28
Time [s]794756780753750
Speed [MB/s]73.376.974.677.277.6

From this several things can be seen: The default compression of ZFS in this version is lz4. Compressing the data is definitely worth it since there is no speed penalty. The compression ratio of gzip and zstd is a bit higher while the write speed of lz4 and zstd is a bit higher.This results in the clear conclusion that for this data zstd is optimal since it saves 13GB of space while increasing the write speed slightly.

Now to see how ZFS deals with the second dataset of uncompressible data. As before the pools are freshly initialized and the transfer time is logged.

offongziplz4zstd
Data [MiB]139264139264139264139264139264
Compression1.001.011.011.011.01
Time [s]24072407240724032412
Speed [MB/s]57.957.957.958.057.7

As we can see the data relay has been uncompressible. Also the compression algorithms did not change the outcome speed wise. All is identical within the margin of error.

The third dataset of VM Images is important for home lab enthusiast like me. Off all of these datasets this is the most speed and efficiency critical dataset.

offongziplz4zstd
Data [MiB]1013766297652940.86297654067.2
Compression1.001.611.911.611.87
Time [s]1005611973616552
Speed [MB/s]100.9165.9104.2164.6183.7

Again the general recommendation that you should turn on compression turns out to be true. You always will save space. With the lz4 and zstd algorithm you also increase the write speed over the speed of the underlying rust disks (that caps at 130MB/s). Although gzip achieved a slightly higher compression ratio my personal winner is again the zstd algorithm with its stellar write speed.

Conclusion part 1:

The recommendation to turn on compression is true for three different kinds of datasets. Even for uncompressible data it does not hurt the speeds. I am a bit surprised that this first round has a clear winner: ZSTD.It achieves a good compression (tied to the best) at high write speeds.

This warrants that in part 2 I examine this algorithm a bit closer. ZFS enables different compression parameters for this algorithm. There is a zstd-fast implementation, but also levels between 1 and 19 (with higher being stronger compression).Keep your eyes peeled for part 2…


r/zfs • • Sep 12 '25

Gotta give a shoutout to the robustness of ZFS

Post image
191 Upvotes

Recently moved my kit into a new home and probably wasn't as careful and methodical as I should have been. Not only a new physical location, but new HBAs. Ended up with multiple faults due to bad data and power cables, and trouble getting the HBAs to play nice...and even a failed disk during the process.

The pool wouldn't even import at first. Along the way, I worked through the problems, and ended up with even more faulted disks before it was over.

Ended up with 33/40 disks resilvering by the time it was all said and done. But the pool survived. Not a single corrupted file. In the past, I had hardware RAID arrays fail for much less. I'm thoroughly convinced that you couldn't kill a zpool if you tried.

Even now, it's limping through the resilver process, but the pool is available. All of my services are still running (though I did lighten the load a bit for now to let it finish). I even had to rely on it for a syncoid backup to restore something on my root pool -- not a single bit was out of place.

This is beyond impressive.


r/zfs • • Nov 30 '20

OpenZFS 2.0 Released!

Thumbnail
github.com
190 Upvotes

r/zfs • • Jul 22 '26

ZFS is incredible

168 Upvotes

Just felt the need to express somewhere how impressed I am. I’m using too many drives in a homelab setting, so there are SATA power splitters involved. When I added zlogslog devices, I had one drive fault out because I ended up with too many drives on a five-way splitter (which is probably something we shouldn’t even manufacture… but my fault for buying), rebalanced the drives and restarted, no issues since nothing was being written. Did my first major transfer to the server today and a drive faulted out again (just resetting repeatedly), so I spent an hour making another modular cable for my power supply and eliminated that splitter entirely.

I expected a corrupt drive and a full resilver; what I got was only 150GB out of a 450GB transfer needing resilvered. It only had to recalculate the data since the drive faulted out.

Amazing engineering, that’s all I’ve got to say. Have a good day, everyone!

EDIT: not everything is a z


r/zfs • • Apr 29 '20

Linux ZFS GUI

161 Upvotes

I start moving my personal backup system to ZFS a few weeks ago. And IMHO, the best way to learn it is to write a software using it. Command line is good but GUI can help when you have many pools/datasets. Right now, it's an alpha version, only tested on my computer. This is only a viewer, you can't do any ZFS action with it yet. But if many people are interested in this software, I will test it for other OS/ZFS/Go version.

This is posted here with a new account to dissociate it from my personal account activity.

https://gitlab.com/beteras/zgui


r/zfs • • May 23 '25

Introducing ZFS AnyRaid

Thumbnail
hexos.com
157 Upvotes

r/zfs • • Jan 14 '25

OpenZFS 2.3.0 released

Thumbnail
github.com
155 Upvotes

r/zfs • • Sep 18 '25

Lesson Learned - Make sure your write caches are all enabled

Post image
140 Upvotes

So I recently had the massive multi-disk/multi-vdev fault from my last post, and when I finally got the pool back online, I noticed the resilver speed was crawling. I don't recall what caused me to think of it, but I found myself wondering "I wonder if all the disk write caches are enabled?" As it turns out -- they weren't (this was taken after -- sde/sdu were previously set to 'off'). Here's a handy little script to check that and get the output above:

for d in /dev/sd*; do

# Only block devices with names starting with "sd" followed by letters, and no partition numbers

[[ -b $d ]] || continue

if [[ $d =~ ^/dev/sd[a-z]+$ ]]; then

fw=$(sudo smartctl -i "$d" 2>/dev/null | awk -F: '/Firmware Version/{gsub(/ /,"",$2); print $2}')

wc=$(sudo hdparm -W "$d" 2>/dev/null | awk -F= '/write-caching/{gsub(/ /,"",$2); print $2}')

printf "%-6s Firmware:%-6s WriteCache:%s\n" "$d" "$fw" "$wc"

fi

done

Two new disks I just bought had their write caches disabled on arrival. Also had a tough time getting them to flip, but this was the command that finally did it: "smartctl -s wcache-sct,on,p /dev/sdX". I had only added one to the pool as a replacement so far, and it was choking the entire resilver process. My scan speed shot up 10x, and issue speed jumped like 40x.


r/zfs • • Jan 29 '22

Linus Tech Tips fails at using ZFS properly, loses data

Thumbnail
youtube.com
131 Upvotes

r/zfs • • Oct 28 '24

OpenZFS deduplication is good now and you shouldn't use it

Thumbnail
despairlabs.com
124 Upvotes

r/zfs • • 15d ago

You will use SMR and be happy! (if you are allowed to own a computer)

Post image
123 Upvotes

Fresh off the "OpenZFS User and Developer Summit 2026" presses - scary WD guy says industry is moving entirely towards host managed SMR HAMR drives and CMR is going away. "Host managed" is going to be implied when talking about SMR and "drive managed" is not going to be a thing. No CMR areas either - heads can't support both.

There's already work underway to make ZFS support this, but at least from what I understood, you're going to need some sort of additional non-SMR vdev (special) to store bootstrapping things on.

All of you who wanted 100M AI generated pictures of cat-octopus hybrids in the clouds - this is your fault!

Two back to back presentations about this. Overall, very interesting.

https://youtu.be/s1L1VxASNfw?t=23503


r/zfs • • Aug 22 '26

OpenZFS 2.4.4 has been released!

Thumbnail
github.com
113 Upvotes

r/zfs • • Feb 21 '22

A simple (real world) ZFS compression speed an compression ratio benchmark - Part 2

106 Upvotes

Introduction:

This is a continuation of my first ZFS compression benchmark. If you haven’t read part 1 yet, please start there: https://www.reddit.com/r/zfs/comments/svnycx/a_simple_real_world_zfs_compression_speed_an/

In the first round ZSTD came out on top over GZIP, LZ4 and the default compression (also LZ4). As a result I decided to investigate it a bit further.

ZSTD background:

ZSTD is a modern compression algorithm originally developed by Facebook. It has several compression levels from 1 to 19. The default in ZFS is level 3 (zstd-3). As seen in the previous test there it achieves speeds similar to LZ4 while getting compression rations close to the default gzip level.

Since decompressing speed is similar (blazing fast) for all levels in some usage scenarios a higher compression level might be desirable for archiving purposes. It is supposed to achieve similar compression ratios to LZMA on high levels…

There is also an implementation of zstd-fast that supports even faster compression and decompression speeds at the cost of compression ratio.

Hardware/Software/preparation:

This is all done identical to the first part.The Machine is an i7 2600 with 8GB RAM running Proxmox V7.1.Each test is run on its own freshly created pool on a 3TB WD green drive. The test data is identical to the first round

The compression algorithms this time are zstd-fast, zstd-5, zstd-10, zstd-15 and zstd-19. Again the transfer speed is monitored by the time command but ultimately limited by the speed of the 2.5” source drive.

In the first round the dataset consisting of mixed documents, zstd aka. zstd-3 was the best algorithm out there. I was curious to see how much compression it could achieve and if it is write speed limited at higher levels.

zstd-fastzstdzstd-5zstd-10zstd-15zstd-19
Data [MiB]46489.645363.245260.844953.644646.444544
Compression1.251.281.281.291.301.30
Time [s]75375074979613663220
Speed [Mib/s]77.277.677.773.142.618.1

As we can see in this comparison higher compression levels result in diminishing returns. I am a bit surprised of how well zstd-fast is doing since it is supposed to be trading ratio for speed. It can also be seen that for the higher levels starting around zstd-10 the compression is getting so CPU intensive that it is limiting the overall speed. Since it only manages to squeeze out another 2% compression ratio, this is only suitable when archiving stuff.

Also I notices starting at zst-10 that the CPU was at 100% (on 6 of 8 cores) while writing to disk. The task becomes CPU limited not IO limited...

The second dataset was Incompressible data (jpg pictures). There compression should not make a difference.

zstd-fastzstdzstd-5zstd-10zstd-15zstd-19
Data [MiB]139264139264139264139264139264139264
Compression1.011.011.011.011.011.01
Time [s]241824122417241130025255
Speed [Mib/s]57.657.757.657.846.426.5

As we can see data wise the higher levels where – as expected – not able to achieve any better compression. We can also observe a speed impact for higher compression levels. However the speed impact is not as severe as with compressible data. This is a result of the intelligent management of ZFS. If the data is uncompressible it aborts after some testing.

But still for high compression levels the speed is impacted significantly without yielding any benifits in compression. The only redeeming fact is taht the data slill coulb be read at drive speeds.

The final test was done with VM images, that proved to do quite well with compression.

zstd-fastzstdzstd-5zstd-10zstd-15zstd-19
Data [MiB]57753.654067.253248.052633.65068850483.2
Compression1.751.871.901.921.992.00
Time [s]55855257876126686594
Speed [Mib/s]181.7183.7175.4133.238.015.4

In the results we see a noticeable increase in the compression ratio with higher levels. However starting at zstd-10 this is accompanied with a dramatic decrease in write performance. This might be OK for archiving purposes, but for live operation a lower compression level seems better suited…

Since there is so much data that goes trough the Compression algorythm the speed is worse eaven compared to the other data at these high levels.

Conclusion:

Time for the conclusion. To make it a bit clearer here are two diagrams of compression ratio and speed.

As can be clearly seen the compression ratio largely depends on the type of data and only marginally on the actual level. However the speed is dramatically impacted by the level. There is a regime for low levels where the overall speed is limited by the speed of the underlying drive. The speed achieved with the Compression can be higher than the actual trughpput of the drive - a cler win for compression. With my case the sweet spot is somewhee between zst-5 and zstd-10. Note that this cap largely depends on your drive speed and CPU performance. For some combinations (fast NVME drive / RAID and a low power atom CPU) there might even be no performance gain from compression.

But at least for this combination the default option of zstd that is equivalent to zstd-3 turns out to be surprisingly close to the optimum. Like always this should not be considered as the only valid option, but as a starting point for your own investigation. I would recommend testing out zstd-fast on SSDs and higher levels if you have CPU power to spare.To squeeze out the last couple percent of storage efficiency for archiving purposes the higher levels could be useful, maybe in combination with an archiving cron job running in off peak hrs.

Since this concludes the Compression algrythm tests it is time to give my 2ct:

Compression if used within default parameters is mostly usefull and in the worst case not harmful. On spinning disks zstd is the best algorythm. On fast (big) RAID systems and SSDs LZ4 might be a better choice. Only on datasets that are not compressible turning it off is benificial.

But as always you should test your pool on your hardware with your data to get the best indicators ;)

Outlook:

Since (default) compression largely turns out to be positive or at least not harmful, but the higher levels have diminishing returns I will investigate deduplication in the next part. This might be a good way to trade RAM consumption for efficiency.


r/zfs • • 18d ago

After 12 years I finally got my data back from a pool that wouldn't open - led to a small read-only POC tool

105 Upvotes

this is mostly a "don't give up on your old drives" post, maybe it helps someone who has a dead pool sitting in a drawer like I did.

oct 2014, my home NAS (gentoo, zfs 0.6.3, mirror on LUKS) had a kernel panic and afterwards rpool/home just wouldn't open anymore: can't open objset for rpool/home. I tried everything people suggested in openzfs #2831 back then, -F, -X, -T, patched txgs, looping over every txg for weeks... best I got was "cannot iterate filesystems". a recovery company gave up too, so the drives went into the drawer.

this year I tried again with current openzfs master (a zdb crash on exactly that pool had just been fixed). zdb could finally open the pool and showed what was wrong: everything was fine except one single block, the objset block of home. both copies overwritten, on both disks. one block, 360 GB unreachable, and every older txg pointed to the same broken block, so no rewind could ever have worked.

but the data underneath was still there. so I scanned the raw disk for the dataset's dnode tree, checked every piece against the checksums zfs stores anyway, and found a complete state from ~16 hours before the crash. a few days ago my photos from 2014 opened again, byte for byte. pretty emotional tbh.

I put what I hacked together into a small read-only tool, more a proof of concept than a product: https://github.com/simonbuehler/zfs-salvage

it runs from an ubuntu live usb with a little web UI, works for single disks, mirrors and stripes, and can also go back to older states for deleted files or destroyed datasets. no raidz yet, no zstd, no native encryption.

two things I wish someone had told me back then:

  • never open the disks writable, not even for a "readonly" import. my old attempts kept writing new uberblocks. cryptsetup open --readonly / blockdev --setro, or just work on images

  • try zdb from current master first, even if it can't import it can tell you which block is actually broken

if you're stuck with a pool like that, or have a raidz one that won't import, feel free to try it or just ask


r/zfs • • Nov 24 '21

OpenZFS 3.0 Announced- online zfs expansion

Thumbnail
ixsystems.com
105 Upvotes

r/zfs • • 25d ago

An interactive museum of OpenZFS internals

104 Upvotes

Uberblok is a website that walks through OpenZFS one idea per room: the uberblock and the block tree, copy-on-write, checksums and self-healing, transaction groups, the vdev tree, RAID-Z, snapshots and deadlists, the ARC, metaslabs and space maps, ZIL and SLOG, send/receive, and compression, dedup and encryption. Plus a history wing from Sun to OpenZFS.

Each room starts from a problem (silent corruption, the write hole, consistency without fsck), then shows the idea with a figure you can step through and short annotated excerpts of the real code. A few things you can do:

- pull the plug at any of the four beats of vdev_config_sync, then import the pool

- flip a bit in one side of a mirror, read the file, and watch the checksum in the parent catch it and the mirror repair it

- lay out one block on a RAID-Z vdev for any size, parity and width, then reconstruct it

- follow a synchronous write through the ZIL, crash, then claim and replay

How it is kept honest: everything reads one frozen commit (zfs-2.4.4, 71a9f95). The 116 excerpts are checked byte for byte against that commit by CI, every code link points to lines at that commit, and every history fact has a row in SOURCES.md.

How it was built : using Algotitan, an agent swarm wrote the site and the code; Human set the mission, steered it and validated each version.

If a room says something the code does not, please tell !

Site: https://uberblok.storagebits.workers.dev

Code, sources and every design decision: https://github.com/algotitanhq/uberblok


r/zfs • • Jan 09 '23

A detailed guide to OpenZFS - Understanding important ZFS concepts to help with system design and administration

Thumbnail jro.io
104 Upvotes

r/zfs • • Mar 05 '21

ZFS recommendations for new users.

104 Upvotes

To every new ZFS user, or someone who wants to get started with ZFS, here are my own "general" recommendations.

RAM

ECC RAM

If you have a backup of your data, or if the data solution will be "Write once, read multiple times", then you probably don't need ECC RAM. Keep in mind, non-ECC memory won't damage your data when you scrub your zpool (source).

Yes, I am an advocate of using ECC RAM with ZFS, but I have my own opinions (stated in the paragraph below) and it's not necessary for you to follow them to have your data integrity remain intact with ZFS. Most people download rip blu rays once and store it on their NAS, they click photos once and store store it on the NAS. And that is accessed frequently, so write once, read multiple times applies to the most of consumers/prosumers (including my own personal usage).

But considering that you're using non-ECC memory with ZFS, the only thing prone to errors (except for in-flight data over network, but that too has it's own integrity protections and I don't want to get into a networking rabbit hole right now) is your non-ECC RAM; It is the only place left where errors can occur. And, ECC RAM sure is a nice to have. If you're financially restricted, don't hesitate to go with [a] good non-ECC RAM module(s). If you really want to, read below section for how much do you really require.

How much RAM do I need?

YOU DON'T NEED 1GB OF RAM PER 1TB OF STORAGE. ZFS was previously used for storing databases like MySQL/MariaDB or whatever you had in the days when ZFS was new; SSDs were not widespread - not even in a datacenter. So to get higher IOPS (higher IOPS are better for database related workloads) multiple HDDs were pooled together to increase IOPS. That is not a case anymore, at least I hope it is not. You host your database on a SSD now. A single SATA SSD will be better than multiple HDDs, nevermind using a good NVMe lol

Hence the 1GB of RAM for 1TB of disk storage is the rule when you host your database on spinning rust. If you're using a SSD, having a lot of RAM won't help you as much (more memory will surely help you regardless of having a HDD or Optane, but it won't be as drastic as using more RAM with HDDs).

If you just want a bulk storage solution and are not going to access your NAS frequently ("frequently" is loosely used here - a home user's frequent usage will be equivalent or even lower than an enterprise's least frequent usage), 8GB is the sweet "starting point". Although you can go as low as 4GB RAM with FreeBSD. Go nuts if you can afford more. Unlike L2ARC, more RAM will almost never hurt you (more on L2ARC/caching later).

Compression

Turn on compression. Please. Your 1st gen Core i3 can handle gzip-6 (refer here) so I don't see any reason why not to turn it on.

People argue that your speeds decrease but it's actually false. Say you have a 100MB file. Below are a few scenarios I can currently think of off the top of my head -

  • The file is compressible and gzip compresses it to a 50MB file. Now you only need to write half of what you would have to (you're only writing 50MB instead of 100MB) write to the disk. And since it's a 50MB file now, it's much faster in reads as well.

  • The file is not compressible. gzip tries to compress it but as the file is not compressible, it is taxing the CPU for a useless workload. ZFS has something called 'early abort'. While your file is being compressed, it will check if the compression ratio is good enough. If the ratio is not good enough, it will skip compression and write the file to disk without compression.

Either way, you're either saving disk space and getting better sequential Read+Write speeds. Or, you're intelligently using your CPU and preventing CPU cycles from being put to waste and store the data.

If you're wondering if a file is compressible is or not; IN VERY SIMPLE WORDS; Text files can be compressed very well i.e. you can compress a 100MB text file in 10MB (theoretically); but video/images don't compress very well. But, turning on compression will be the best for your storage space regardless of the file's nature of being compressed.

Which compression algorithm to choose?

  • zstd: this is a "relatively" new compression algorithm used in ZFS and it has compression ratio that matches/beates gzip-6 at the speed of lz4 i.e. it can compress files to a very low size and it can do it very fast. zstd go brrr

  • lz4: it has excellent compression speed but doesn't compress very well. Use it where you can do without compression but where compression is nice to have.

  • gzip: it is used where the 'time taken to compress' is not a problem - where you need the lowest file size e.g. off-location backup servers

For new users wondering what gzip-6 is, gzip is a compression algorithm and has 9 compression levels; 1 (gzip-1) being - do it as fast as you can and 9 (gzip-9) being - reduce it down to an atom; gzip-6 is the default when you set compression to use gzip without explicitly stating the compression level, and it is a sane default.

zstd has similar compression levels, from 1 to 19, like you have in gzip.

ZFS RAID types

RAIDZ(1/2/3)

The only thing I would want to add to this section is that you don't need to follow the odd-even rule. And if you don't know about the odd-even rule, congratulations, you didn't receive useless false information in the first place.

When using raidz1/2/3, there is no technical (or even a practical) limit of how many drives are in the vdev. You can have 100 drives in raidz1 if your heart desires. However, since you're uing ZFS I assume you worry about your data. So please keep sane configurations - like not using more than 5 drives in raidz1, 10 drives in raidz2 and 15 in raidz3. You should have your vdev at at least 20% fault tolerance. So in raidz1 you can lose 1 drive and as my advise you shouldn't go (although you definately can, but shouldn't) over 5 drives in total per a raidz1 vdev, you get 20% fault tolerance (1 drive out failing out of 5 total per vdev is 20% fault tolerance - if my math is right). 5, 10 and 15 drives per one raidz1, raidz2 and raidz3 vdev are as extreme as you should get before you start worrying before going to bed. Although the "recommended" number are as below:

  • Between 4 drives per a single raidz1 vdev.

  • Between 6 to 9 (nice) drives per a single raidz2 vdev.

  • Between 9 to 12 drives per a single raidz3 vdev.

Mirror

My understanding of your choice of using/opting for zfs mirror is that you either want the most redudant array or you want the most IOPS possible or a combination of both. I would advise a 3-way mirror over a 2-way mirror where your critical data is stored. I advised 3-way over 2-way because the primary reason to choose a mirror over raidz1/2/3 is mainly higher redudancy or more IOPS (or you have less drives to get started with/dedicate to ZFS) in which case in a 2-way mirror if one drive fails, you're good, but in a vulnerable state, if one more second drive fails, you're done. So go with a 3-way mirror if higher data redudancy is your primary goal. And if you go with a 3-way mirror, more drives in the mirror will naturally increase your IOPS.

Deduplication

Short answer, no. Long answer, for something where you have multiple users storing the same damn file with a differnt location, or you're storing multiple VMs with similar config; Sure, turn it on, but be warned, turning it on without any prior thought will have a negative impact. An example of why I myself would (but shouldn't) turn it on will be to store Linux ISOs on a different location for seeding and copying them to a different location for where only Linux ISOs live. But still. using a symlink is a far better option in my use case, and that is what I do.

Misc

What OS should I go with?

  • If you're not comfortable with command line, go with (formerly named FreeNAS) TrueNAS. It has a GUI for almost everying ZFS related.

  • If you want a OS that has ZFS as it's first class citizen, go with FreeBSD. FreeBSD has ZFS based boot environments. So before an update is applied, it takes a snapshot. And should the update go wrong and you're unable to boot back up, you just boot with the previous boot environment and everything is exactly as it was before the update. I would highly recommend FreeBSD.

  • If you want a Linux distro with support for ZFS-on-root (which FreeBSD has had since years) out of the box, go with Ubuntu (Server) 20.04 or newer. Arch might support ZFS-on-root but I have not tested it personally so cannot say for sure.

  • If you want to use ZFS as just a storage file system IN your current computer (not as a standalone NAS), go with any Linux mainstream distro that has official documentaion for ZFS. I personally have used Pop_OS 20.04 and can say ZFS doesn't have any weird issues on Pop_OS as a separate file system. And considering Pop_OS is based on Ubuntu, Ubuntu might stable too.

  • Do yourself a favor and ignore Windows as NT and POSIX don't mix well. macOS doesn't have a rich community in terms of ZFS (even though BSD based) so that is out of the equation too. But that said, you can obviously test ZFS on Windows and on macOS too.

Misc misc

Say you're low on ports or low on money and can only afford a single drive for your OS, fear not. Setup your OS with ZFS-on-root (FreeBSD comes to mind; versions of Ubuntu after but not excluding 20.04 support ZFS-on-root as well; Arch might be too, not sure about Arch) and send periodic snapshots of your OS pool to the storage pool. And when your OS drive fails, plug a live USB and a new OS drive in. Boot into a shell, restore the new drive with a snapshot of OS from the storage pool and voila, you're back up. Although, I would highly recommend at least a 2-way mirror for your OS drive.

Let me know your recommendations or modifications I should make :')

Edit: Replaced "SATA" with "SATA SSDs" as per u/kaihp pointed out.


r/zfs • • Feb 23 '22

Working smarter not harder: why ZFS performance advantages cannot be measured by fio and other disk benchmarking tools.

98 Upvotes

tl;dr ZFS maximises performance through clever ways of avoiding disk access altogether. This can make ZFS much faster than other file systems when they need to access disk but ZFS doesn’t, however this advantage is by definition invisible to disk benchmarking tools.

——————

Every now and then I see a post or article where someone has run fio or another benchmark to measure zfs performance vs other file systems. Sometimes this produces numbers that seem quite unfavourable to zfs. 

It seems to me however that the various performance optimisations of ZFS aim to minimise the need to access disk at all; and this makes a lot of sense, since the fastest disk access is always the one you didn’t have to make. This advantage is however something that disk benchmarks, by definition, will never be able to measure.

Also given that ZFS is a transactional copy-on-write filesystem that uses block pointers and checksums, when it DOES have to access disk there will always be some overhead compared to simpler filesystems, which is something that disk benchmarking tools CAN measure. Disk benchmarks are thus blind to the performance advantages that ZFS provides, but are quite capable of reporting on the extra disk i/o that makes these advantages possible. This can give a profoundly distorted picture of how ZFS is likely to perform in practice compared to other filesystems.

It also seems to me that the design of ZFS is a reflection of the history of ZFS, and as a result it continues to provide the greatest performance advantages for the same types of workload and on the same category of hardware for which it was initially conceived. 

According to Jeff Bonwick, he first conceived of the design for ZFS out of anger, while standing around some time during 1995 or 1996 with nothing to do after a 1-character administrator typo took out the home directory storage server for 1,000 Sun Microsystems employees in his building, costing the company 1,000 person days of work.

This origin story seems to have produced a design that maximises real-world performance in three ways: by eliminating minimum performance scenarios, by eliminating unnecessary background workloads, and by leveraging other parts of the computer system to take reads and writes away from main pool storage wherever possible.

Firstly, a key part of achieving high performance is avoiding low performance. And there is no performance lower than an outage. IOPS drop to zero when they are queued behind a sysadmin team that is trying to work out how best to recover from a data corruption, and no database commit is slower than one that is blocked waiting for the courier to arrive with the backup tapes.

Furthermore, outages have cascading effects on the throughput of other systems, both automated and human. Performance only matters because somewhere there is a human waiting for the data, and the human suffering caused by poor performance scales exponentially with the length of the delay and the number of people affected, as well as the uncertainty of the outage. Extra milliseconds for every web form submit or a couple of hours extra for a batch job every night is inconvenient, but it is manageable and can be mitigated with workarounds or hardware. Unexpected multi-day outages can ruin the holidays, or in extremis, businesses and livelihoods.

ZFS has a focus on data integrity and administrative ease of use that tends to minimise the need for these ruinously low performance types of disk access (commonly referred to as “data recovery” or “restore from tape” or even “update resume…”). This avoidance of tail risk scenarios is something that benchmarking tools are not particularly good at capturing (although chaos engineering tools are, and it might be informative to test zfs vs other file systems using those).

Secondly, ZFS avoids the need for a few types of disk access that occur regularly with the kind of workloads it was initially designed for: massively multi-user, high volume, mission critical, mixed read/write workloads, overseen by expensive system administrators. In this environment there are frequent backups, occasional restores, and regular copying of large volumes of data between different environments.

ZFS can perform these kinds of tasks with enormously reduced disk access due to it’s snapshot, send/receive, and cloning features. From an administrator’s point of view the performance is orders of magnitude faster on the tasks that make up the bulk of their data access, and this reduction in disk access also improves performance for other processes and users who don’t need to contend with concurrent administration tasks.

Lastly, ZFS can be configured to achieve high performance by offloading some or all of the work of your main storage drives to other parts of the system, such as RAM, CPU, or ancillary storage devices such as high speed Optane or NVMe storage. Implicit in this design is the assumption that you will actually have sufficient resources in these other parts of the system to handle the load. This assumption seems reasonable in the world that ZFS came from, where storage hardware is built to spec for the workload it will have to support. For single user personal computing however, this might often not be the case.

Running a ZFS benchmark on a single drive or vdev, particularly with fsync on, does not capture the benefit that these other resources can provide to a dedicated ZFS system with adequate RAM and fast supplemental drives. But even if one does run synthetic benchmarks on a pool backed by lots of RAM, CPU, and fast cache, log, and special devices, most benchmarks will still not generate the kind of traffic that can benefit from these features (with the likely exception of tools like tpc).

The flagship performance optimisation in ZFS is the Adaptive Replacement Cache, or ARC. Just like the cache in other file systems, the ARC serves previously accessed files from RAM rather than disk, giving an orders of magnitude improvement in latency and throughput. It’s claim to fame however is that it is much more likely to actually have the data you want, due to it’s incorporation of a “most frequently used” cache as well as the traditional “most recently used” cache that other file systems have, as well as the way ZFS dynamically tunes the relative size of these two caches in response to your actual data usage over time.

For most real world workloads this improvement in cache hit ratio will have a significant impact on the performance of the file system. Most disk benchmarking tools however seek to avoid accessing cached data altogether, given that their one job is to measure disk, not memory. Even if a disk benchmark does allow reading of cached data, it is still not going to capture the incremental cache hit rate the ARC provides unless it specifically mirrors your real world workload, e.g. you are doing something like replaying your actual database log files with a tool that recreates the timing of the original read and write transactions, or replaying your production web server logs into your test system in a similarly timed manner (in which case you might want to build that test system from a ZFS clone - you know, to avoid needing to access disk…)

ZFS also provides transparent in-line compression of data that is written to appropriately configured datasets. This is very useful for textual data like database records, email, word documents, time series observations, log files, genomic data, etc. etc. etc., but not at all useful for media data like photos, video, or audio streams, which are generally already compressed by their codec.

ZFS compression can offload work from your disks to your CPU. Compressing data to a fraction of it’s original size gives a proportional boost to disk performance, as there is now less data to be written or read. The amount of compression that can be achieved however depends strongly on the type of data. Benchmarking tools that primarily use random data will not show any benefit from compression and will only be relevant to incompressible media workloads. Any benchmarking tool that uses a string of zeros on the other hand would show nonsensically high compression and performance.

Compression ratios are also limited by the size of the records being written relative to the underlying block size of the device they are written to. Because space can only be allocated in whole blocks, compression only produces a benefit when it is sufficiently effective so as to reduce the number of blocks required. This becomes less likely when smaller records are used (as is often the case with benchmarking tools). For example, a 128kb record with a compression ratio of 0.75 will only need 96kb of storage. On a 4k block device (ashift=12), this saves 8 whole blocks (32kb), and will increase perceived disk performance accordingly. An 8kb record with the same compression ratio on the same device however needs 6kb, or 2 whole blocks, which is the same as the uncompressed data and means that the effective benefit from compression is zero. If a benchmarking tool uses small records on a compressed dataset, it will not show the full benefit of compression even if it uses sensibly compressible data.

An exciting new feature of recent OpenZFS versions is “compressed ARC”. Blocks that are compressed on disk are now stored in ARC in exactly the same compressed format, and are only decompressed when read. This means that with compressible data, The ARC can now hold an increased amount of data that is proportional to the compression ratio of that data.

This feature is of particular benefit when it allows a database designer to configure their machine to fit the whole database into ARC, and therefore get memory-speed data access for all queries. Database records often have compression ratios exceeding 3:1 or 4:1, which can make an in-memory database cache practical in many cases where it would not be without compression. Synthetic benchmarks do nothing to capture this tipping point effect of compressing a database cache and eliminating disk reads altogether (again with the exception of tpc or log replay).

Of course, being able to fit more data into ARC will benefit non-database compressible workloads as well, and will magnify the benefit of the ARC’s improved cache hit ratio and therefore the size of the performance benefit that disk benchmarks cannot measure.

Another recent feature of OpenZFS is “persistent L2ARC”. This preserves the content of Level 2 ARC across machine reboots, so that ZFS can start with a warm L2 cache. L2ARC is an optional feature that allows ZFS to use a fast storage device such as an Optane card or NVMe SSD to hold entries that are soon to be evicted from regular ARC. Although L2ARC devices are generally much slower than RAM based ARC, they are also generally much larger, and are faster than the main disk pool they have been provisioned to support.

This is one if the few performance features that is just as likely to benefit a single user desktop machine that is powered off every night, as it is to benefit an enterprise storage server which needs a warm cache to meet service level metrics immediately following a kernel update and restart. In either case though the benefit of persisting cache across reboots will not show up in disk benchmarks, which (hopefully) don’t trigger reboots in the middle of their run.

As well as L2ARC, ZFS has two other ways to offload disk activity from the main storage drives to fast ancillary devices. These are SLOG devices, and Special VDEVs. These can deliver dramatic performance improvements, particularly for the synchronous writes favoured by benchmarking tools. While this is something that benchmarking tools CAN measure, it is often the case that benchmarks are run on systems that lack these features, thus overlooking performance advantages that well configured ZFS pools can provide relative to other filesystems.

So to summarise:

ZFS has three main categories of performance optimisation that are invisible to disk benchmarking tools: outage prevention and recovery, snapshot based administration, and offloading disk access to more capable parts of a well-provisioned server.

These are likely to provide a performance benefit far in excess of that which would be indicated by synthetic benchmarks, unless your use case is particularly close to that of a synthetic benchmark, i.e. no hardware to offload to, no administration to perform, and no need to restore the system state in case of outage or failure.

Most real world filesystem use cases at the very least require the ability to restore state in case of outage, generally require regular backups, and may often have spare RAM and CPU cycles available, and are likely to perform far better on ZFS than benchmarks would indicate.


r/zfs • • May 08 '20

ZFS 101—Understanding ZFS storage and performance | Illustrated article on Ars Technica

Thumbnail
arstechnica.com
97 Upvotes

r/zfs • • May 23 '19

ZFS on Linux 0.8.0 released!

97 Upvotes

New Features

Native encryption #5769 - The encryption property enables the creation of encrypted filesystems and volumes. The aes-256-ccm algorithm is used by default. Per-dataset keys are managed with zfs load-key and associated subcommands.

Raw encrypted 'zfs send/receive' #5769 - The zfs send -w option allows an encrypted dataset to be sent and received to another pool without decryption. The received dataset is protected by the original user key from the sending side. This allows datasets to be efficiently backed up to an untrusted system without fear of the data being compromised.

Device removal #6900 - This feature allows single and mirrored top-level devices to be removed from the storage pool with zpool remove. All data is copied in the background to the remaining top-level devices and the pool capacity is reduced accordingly.

Pool checkpoints #7570 - The zpool checkpoint subcommand allows you to preserve the entire state of a pool and optionally revert back to that exact state. It can be thought of as a pool wide snapshot. This is useful when performing complex administrative actions which are otherwise irreversible (e.g. enabling a new feature flag, destroying a dataset, etc).

Pool TRIM #8419 - The zpool trim subcommand provides a way to notify the underlying devices which sectors are no longer allocated. This allows an SSD to more efficiently manage itself and helps prevent performance from degrading. Continuous background trimming can be enabled via the new autotrim pool property.

Pool initialization #8230 - The zpool initialize subcommand writes a pattern to all the unallocated space. This eliminates the first access performance penalty, which may exist on some virtualized storage (e.g. VMware VMDKs).

Project accounting and quota #6290 - This features adds project based usage accounting and quota enforcement to the existing space accounting and quota functionality. Project quotas add an additional dimension to traditional user/group quotas. The zfs project and zfs projectspace subcommands have been added to manage projects, set quota limits and report on usage.

Channel programs #6558 - The zpool program subcommand can be used to perform compound ZFS administrative actions via Lua scripts in a sandboxed environment (with time and memory limits).

Pyzfs #7230 - The new pyzfs library is intended to provide a stable interface for the programmatic administration of ZFS. This wrapper provides a one-to-one mapping for the libzfs_core API functions, but the signatures and types are more natural to Python.

Python 3 compatibility #8096 - The arcstat, arcsummary, and dbufstat utilities have been updated to be compatible with Python 3.

Direct IO #7823 - Adds support for Linux's direct IO interface.

Performance Sequential scrub and resilver #6256 - When scrubbing or resilvering a pool the process has been split into two phases. The first phase scans the pool metadata in order to determine where the data blocks are stored on disk. This allows the second phase to issue scrub I/O as sequentially as possible, greatly improving performance.

Allocation classes #5182 - Allows a pool to include a small number of high-performance SSD devices that are dedicated to storing specific types of frequently accessed blocks (e.g. metadata, DDT data, or small file blocks). A pool can opt-in to this feature by adding a special or dedup top-level device.

Administrative commands #7668 - Improved performance due to targeted caching of the metadata required for administrative commands like zfs list and zfs get.

Parallel allocation #7682 - The allocation process has been parallelized by creating multiple "allocators" per-metaslab group. This results in improved allocation performance on high-end systems.

Deferred resilvers #7732 - This feature allows new resilvers to be postponed if an existing one is already in progress. By waiting for the running resilver to complete redundancy is restored as quickly as possible.

ZFS Intent Log (ZIL) #6566 - New log blocks are created and issued while there are still outstanding blocks being serviced by the storage, effectively reducing the overall latency observed by the application.

Volumes #8615 - When a pool contains a large number of volumes they are more promptly registered with the system and made available for use after a zpool import.

QAT #7295 #7282 #6767 - Support for accelerated SHA256 checksums, AES-GSM encryption, and the new QAT Intel(R) C62x Chipset / Atom(R) C3000 Processor Product Family SoC.

https://github.com/zfsonlinux/zfs/releases/tag/zfs-0.8.0