ZFS encryption bug

Disclaimer: The following should not be construed as professional advice. Avid Andrew is not liable for any damages arising from your use of ZFS native encryption. You should consult with a storage professional when configuring your filesystem.

ZFS is a powerful filesystem with many advanced features and two decades of reliability. However, recently concerns have arisen about the new ZFS native encryption functionality. Let's examine this in more detail to understand what the known risks are and possible workarounds for this bug.

This bug report, opened in 2021, details a bug with the new ZFS native encryption functionality that results in corruption in snapshots. Reproducing this bug has been particularly difficult, and it's possible that there are multiple bugs related to this behavior. Recently, there have been calls to add a warning about using native ZFS encryption. However, according to the bug reports, it appears that corruption only occurs when using zfs send from an encrypted dataset, and even then perhaps only when not using zfs send -R. This is important because many use cases of ZFS native encryption may not involve zfs send at all, or you may be able to change how you use zfs send to avoid this bug and still take advantage of native encryption.

Let's consider the following scenarios:

ZFS native encryption scenarios

In Scenario 1, the primary dataset is unencrypted and data is sent via zfs send to a backup dataset which is encrypted. In this setup, there shouldn't be any danger of hitting the bug since the source dataset is unencrypted. In Scenario 2 and Scenario 3, the primary dataset is encrypted and zfs send is used to send data to the backup dataset so there is a risk of corruption. However, there are some options for how to avoid the behavior that triggers this bug:

  • Use Non-ZFS Replication: Rather than using zfs send, you could replicate the ZFS dataset to the backup dataset using another tool such as rsync. Yes, this forefits the benefits of block-based replication, but if you combine rsync with sanoid on the destination, you could sync data and create snapshots on the destination while avoiding the scenario that triggers this bug
  • Tweak ZFS Send Options: Using zfs send -w (a "raw" send) appears to prevent this issue because it sends the data encrypted, exactly how it already exists on the source disk. If you're using syncoid instead of zfs send directly, it's also important to pass the -R flag to zfs send when syncoid calls it by invoking syncoid with --sendoptions=R instead of --recursive; note that the developer of syncoid warns of sync errors when using this option on a zpool where the list of datasets is actively changing during the syncoid run.

A Possible Resolution

The good news is the developers may be close to isolating at least one of the bugs, which will hopefully lead to a permanent fix to this problem. In the meantime, it should be possible to use ZFS native encryption safely as long as caution is used when sending snapshots from an encrypted dataset.

Support Us

If you found this article helpful, please subscribe to our newsletter or support us on Patreon and get access to bonus features!

Questions? Comments?

Do you have questions or comments about this article? Please contact us via Mastodon or Email - we want to hear from you!