tl;dr ZFS maximises performance through clever ways of avoiding disk access altogether. This can make ZFS much faster than other file systems when they need to access disk but ZFS doesn’t, however this advantage is by definition invisible to disk benchmarking tools.
——————
Every now and then I see a post or article where someone has run fio or another benchmark to measure zfs performance vs other file systems. Sometimes this produces numbers that seem quite unfavourable to zfs.
It seems to me however that the various performance optimisations of ZFS aim to minimise the need to access disk at all; and this makes a lot of sense, since the fastest disk access is always the one you didn’t have to make. This advantage is however something that disk benchmarks, by definition, will never be able to measure.
Also given that ZFS is a transactional copy-on-write filesystem that uses block pointers and checksums, when it DOES have to access disk there will always be some overhead compared to simpler filesystems, which is something that disk benchmarking tools CAN measure. Disk benchmarks are thus blind to the performance advantages that ZFS provides, but are quite capable of reporting on the extra disk i/o that makes these advantages possible. This can give a profoundly distorted picture of how ZFS is likely to perform in practice compared to other filesystems.
It also seems to me that the design of ZFS is a reflection of the history of ZFS, and as a result it continues to provide the greatest performance advantages for the same types of workload and on the same category of hardware for which it was initially conceived.
According to Jeff Bonwick, he first conceived of the design for ZFS out of anger, while standing around some time during 1995 or 1996 with nothing to do after a 1-character administrator typo took out the home directory storage server for 1,000 Sun Microsystems employees in his building, costing the company 1,000 person days of work.
This origin story seems to have produced a design that maximises real-world performance in three ways: by eliminating minimum performance scenarios, by eliminating unnecessary background workloads, and by leveraging other parts of the computer system to take reads and writes away from main pool storage wherever possible.
Firstly, a key part of achieving high performance is avoiding low performance. And there is no performance lower than an outage. IOPS drop to zero when they are queued behind a sysadmin team that is trying to work out how best to recover from a data corruption, and no database commit is slower than one that is blocked waiting for the courier to arrive with the backup tapes.
Furthermore, outages have cascading effects on the throughput of other systems, both automated and human. Performance only matters because somewhere there is a human waiting for the data, and the human suffering caused by poor performance scales exponentially with the length of the delay and the number of people affected, as well as the uncertainty of the outage. Extra milliseconds for every web form submit or a couple of hours extra for a batch job every night is inconvenient, but it is manageable and can be mitigated with workarounds or hardware. Unexpected multi-day outages can ruin the holidays, or in extremis, businesses and livelihoods.
ZFS has a focus on data integrity and administrative ease of use that tends to minimise the need for these ruinously low performance types of disk access (commonly referred to as “data recovery” or “restore from tape” or even “update resume…”). This avoidance of tail risk scenarios is something that benchmarking tools are not particularly good at capturing (although chaos engineering tools are, and it might be informative to test zfs vs other file systems using those).
Secondly, ZFS avoids the need for a few types of disk access that occur regularly with the kind of workloads it was initially designed for: massively multi-user, high volume, mission critical, mixed read/write workloads, overseen by expensive system administrators. In this environment there are frequent backups, occasional restores, and regular copying of large volumes of data between different environments.
ZFS can perform these kinds of tasks with enormously reduced disk access due to it’s snapshot, send/receive, and cloning features. From an administrator’s point of view the performance is orders of magnitude faster on the tasks that make up the bulk of their data access, and this reduction in disk access also improves performance for other processes and users who don’t need to contend with concurrent administration tasks.
Lastly, ZFS can be configured to achieve high performance by offloading some or all of the work of your main storage drives to other parts of the system, such as RAM, CPU, or ancillary storage devices such as high speed Optane or NVMe storage. Implicit in this design is the assumption that you will actually have sufficient resources in these other parts of the system to handle the load. This assumption seems reasonable in the world that ZFS came from, where storage hardware is built to spec for the workload it will have to support. For single user personal computing however, this might often not be the case.
Running a ZFS benchmark on a single drive or vdev, particularly with fsync on, does not capture the benefit that these other resources can provide to a dedicated ZFS system with adequate RAM and fast supplemental drives. But even if one does run synthetic benchmarks on a pool backed by lots of RAM, CPU, and fast cache, log, and special devices, most benchmarks will still not generate the kind of traffic that can benefit from these features (with the likely exception of tools like tpc).
The flagship performance optimisation in ZFS is the Adaptive Replacement Cache, or ARC. Just like the cache in other file systems, the ARC serves previously accessed files from RAM rather than disk, giving an orders of magnitude improvement in latency and throughput. It’s claim to fame however is that it is much more likely to actually have the data you want, due to it’s incorporation of a “most frequently used” cache as well as the traditional “most recently used” cache that other file systems have, as well as the way ZFS dynamically tunes the relative size of these two caches in response to your actual data usage over time.
For most real world workloads this improvement in cache hit ratio will have a significant impact on the performance of the file system. Most disk benchmarking tools however seek to avoid accessing cached data altogether, given that their one job is to measure disk, not memory. Even if a disk benchmark does allow reading of cached data, it is still not going to capture the incremental cache hit rate the ARC provides unless it specifically mirrors your real world workload, e.g. you are doing something like replaying your actual database log files with a tool that recreates the timing of the original read and write transactions, or replaying your production web server logs into your test system in a similarly timed manner (in which case you might want to build that test system from a ZFS clone - you know, to avoid needing to access disk…)
ZFS also provides transparent in-line compression of data that is written to appropriately configured datasets. This is very useful for textual data like database records, email, word documents, time series observations, log files, genomic data, etc. etc. etc., but not at all useful for media data like photos, video, or audio streams, which are generally already compressed by their codec.
ZFS compression can offload work from your disks to your CPU. Compressing data to a fraction of it’s original size gives a proportional boost to disk performance, as there is now less data to be written or read. The amount of compression that can be achieved however depends strongly on the type of data. Benchmarking tools that primarily use random data will not show any benefit from compression and will only be relevant to incompressible media workloads. Any benchmarking tool that uses a string of zeros on the other hand would show nonsensically high compression and performance.
Compression ratios are also limited by the size of the records being written relative to the underlying block size of the device they are written to. Because space can only be allocated in whole blocks, compression only produces a benefit when it is sufficiently effective so as to reduce the number of blocks required. This becomes less likely when smaller records are used (as is often the case with benchmarking tools). For example, a 128kb record with a compression ratio of 0.75 will only need 96kb of storage. On a 4k block device (ashift=12), this saves 8 whole blocks (32kb), and will increase perceived disk performance accordingly. An 8kb record with the same compression ratio on the same device however needs 6kb, or 2 whole blocks, which is the same as the uncompressed data and means that the effective benefit from compression is zero. If a benchmarking tool uses small records on a compressed dataset, it will not show the full benefit of compression even if it uses sensibly compressible data.
An exciting new feature of recent OpenZFS versions is “compressed ARC”. Blocks that are compressed on disk are now stored in ARC in exactly the same compressed format, and are only decompressed when read. This means that with compressible data, The ARC can now hold an increased amount of data that is proportional to the compression ratio of that data.
This feature is of particular benefit when it allows a database designer to configure their machine to fit the whole database into ARC, and therefore get memory-speed data access for all queries. Database records often have compression ratios exceeding 3:1 or 4:1, which can make an in-memory database cache practical in many cases where it would not be without compression. Synthetic benchmarks do nothing to capture this tipping point effect of compressing a database cache and eliminating disk reads altogether (again with the exception of tpc or log replay).
Of course, being able to fit more data into ARC will benefit non-database compressible workloads as well, and will magnify the benefit of the ARC’s improved cache hit ratio and therefore the size of the performance benefit that disk benchmarks cannot measure.
Another recent feature of OpenZFS is “persistent L2ARC”. This preserves the content of Level 2 ARC across machine reboots, so that ZFS can start with a warm L2 cache. L2ARC is an optional feature that allows ZFS to use a fast storage device such as an Optane card or NVMe SSD to hold entries that are soon to be evicted from regular ARC. Although L2ARC devices are generally much slower than RAM based ARC, they are also generally much larger, and are faster than the main disk pool they have been provisioned to support.
This is one if the few performance features that is just as likely to benefit a single user desktop machine that is powered off every night, as it is to benefit an enterprise storage server which needs a warm cache to meet service level metrics immediately following a kernel update and restart. In either case though the benefit of persisting cache across reboots will not show up in disk benchmarks, which (hopefully) don’t trigger reboots in the middle of their run.
As well as L2ARC, ZFS has two other ways to offload disk activity from the main storage drives to fast ancillary devices. These are SLOG devices, and Special VDEVs. These can deliver dramatic performance improvements, particularly for the synchronous writes favoured by benchmarking tools. While this is something that benchmarking tools CAN measure, it is often the case that benchmarks are run on systems that lack these features, thus overlooking performance advantages that well configured ZFS pools can provide relative to other filesystems.
So to summarise:
ZFS has three main categories of performance optimisation that are invisible to disk benchmarking tools: outage prevention and recovery, snapshot based administration, and offloading disk access to more capable parts of a well-provisioned server.
These are likely to provide a performance benefit far in excess of that which would be indicated by synthetic benchmarks, unless your use case is particularly close to that of a synthetic benchmark, i.e. no hardware to offload to, no administration to perform, and no need to restore the system state in case of outage or failure.
Most real world filesystem use cases at the very least require the ability to restore state in case of outage, generally require regular backups, and may often have spare RAM and CPU cycles available, and are likely to perform far better on ZFS than benchmarks would indicate.