Ctrlk
VitePress 镜像站QQ 群 787969044视频教程Ⅰ视频教程Ⅱ
discord
For the complete documentation index, see llms.txt. This page is also available as Markdown.

28.3 ZFS Storage Pool Management

This section introduces several ZFS storage pool management methods.

ZFS Storage Pool Expansion

ZFS supports online expansion of storage pools without service interruption. The expansion process consists of two steps: first, adjust the underlying partition size to utilize the newly added space, then use ZFS commands to make the storage pool recognize the new capacity of the device.

The available capacity of a redundant pool depends on the smallest device in each vdev. After replacing the smallest device with a larger-capacity device, the pool capacity can be expanded.

After a replace or resilver operation completes, the pool's available space will grow to the capacity of the new device.

Take a mirror pool consisting of 1TB and 2TB hard drives as an example: the available space is 1TB. After replacing the 1TB hard drive with another 2TB hard drive, the resilver operation will copy existing data to the new hard drive.

Both hard drives now have a capacity of 2TB, so the available space of the mirror pool will increase to 2TB.

Warning

ZFS storage pools can only be expanded, not shrunk. Therefore, changes cannot be undone.

Resizing Partitions

Display the current disk partition table and partition information:

# gpart show
=>       40  167772087  nda0  GPT  (80G)
         40     532480     1  efi  (260M)
     532520       1024     2  freebsd-boot  (512K)
     533544        984        - free -  (492K)
     534528    4194304     3  freebsd-swap  (2.0G)
    4728832  142071775     4  freebsd-zfs  (68G)
  146800607   20971520        - free -  (10G)

From the output, the free space is 10 GB.

Based on the partition table information, select the freebsd-zfs partition located before the free space (partition 4 in this example) to expand. Confirm the partition number before proceeding:

Display the current disk partition table and partition information:

List all ZFS pools in the system and their status information:

Display detailed status information of the ZFS pool, including health status and error information:

Expanding the ZFS Pool

Expand the ZFS pool:

View all ZFS pools and their capacity, usage, and health status after expansion:

The expansion operation is complete. Additionally, by setting the pool property autoexpand, the pool can automatically expand after disk replacement or underlying device expansion, without the need to manually run zpool online -e each time:

Appendix

This appendix introduces additional usage of the gpart command. You can obtain partition numbers using the gpart show command, or use the -p parameter to display all disk and partition information with full paths:

  • Print partition type GUIDs (for GPT) or raw partition types (for MBR):

Display disk and partition information in a reproducible and complete path format:

  • Display detailed partition information for disk mmcsd0:

Troubleshooting

gpart: table 'ada0' is corrupt: Operation not permitted

This indicates a partition table error and requires resetting the GPT partition table. The gpart recover command restores the main partition table information from the backup GPT header. This issue commonly occurs with raw disk images imported directly.

References

Creating and Destroying Storage Pools

When creating a ZFS storage pool, some decisions cannot be changed, the most important being how physical disks are grouped by vdev type. After pool creation, mirrors allow adding new disks to the vdev; stripes can also be upgraded to mirrors by attaching a new disk to the vdev. With the raidz_expansion feature flag enabled, RAID-Z vdevs can also be expanded. Although adding new vdevs can expand a pool, the vdev type cannot be changed after creation. The only option is to back up data, destroy the pool, and then recreate it.

Creating a Simple Mirror Pool

Create a simple mirror pool:

View the pool status:

Creating Multiple vdevs

To create multiple vdevs in a single command, specify groups of disks separated by vdev type keywords (mirror in this example):

View the pool status:

Creating a Pool Using Partitions

Pools can also use partitions rather than entire disks. By placing ZFS on a separate partition, the remaining space on the same disk is available for other purposes. In particular, partitions containing boot code and a boot filesystem can be added, enabling the pool member disks to also serve as boot disks. Using partitions instead of entire disks does not impose additional performance overhead on FreeBSD. Additionally, partitions support underprovisioning — allocating less space than the disk's total capacity. If a replacement disk has the same nominal size as the original but slightly less actual capacity, the smaller partition can still fit on the replacement disk.

Create a RAID-Z2 pool using partitions:

dRAID Parameters and IO Formulas

  • Distributed RAID-Z (dRAID): dRAID evolved from the RAID-Z concept and is a standalone vdev type that provides integrated distributed hot spares and supports faster sequential resilvering. dRAID consists of multiple internal raidz groups, each containing D data devices and P parity devices. These groups are distributed across all child devices to fully utilize disk performance — a technique known as parity declustering. Additionally, dRAID randomly arranges child vdevs through carefully chosen precomputed permutation maps, ensuring that resilver I/O (both reads and writes) is evenly distributed across all surviving disks regardless of which disk fails, thereby significantly accelerating the resilver process. dRAID uses a fixed stripe width (padding with zeros when necessary), which enables sequential resilvering, but the fixed stripe width significantly affects available capacity and IOPS. For example, with the default D=8 and 4 KiB disk sectors, the minimum allocation size is 32 KiB; when using compression, larger allocation sizes may reduce the effective compression ratio. The random IOPS of dRAID can be approximated as floor((N-S)/(D+P)) × single-disk IOPS. If a dRAID pool will store a large amount of small-block data, it is recommended to additionally add a mirror special vdev. When using ZFS volumes with dRAID, the default volblocksize property is automatically increased to accommodate the allocation size. draid1 (single parity), draid2 (double parity), and draid3 (triple parity) are supported; draid is an alias for draid1.

    Syntax for creating a dRAID vdev:

    dRAID parameter descriptions:

    Parameter
    Description
    Default

    parity

    Parity level (1-3)

    1

    data

    Number of data devices per redundancy group. A smaller D value improves IOPS, compression ratio, and resilver speed, at the cost of reduced total available capacity

    8 (unless N-P-S is less than 8)

    children

    Expected number of child devices, used for cross-validation

    -

    spares

    Number of distributed hot spares

    0

    For example, create an 11-disk dRAID pool with 4+1 redundancy and 1 distributed hot spare:

    The main advantage of dRAID is its support for sequential resilvering: resilver performance scales with the number of disks divided by the stripe width (D+P), which can significantly reduce resilver time and restore full redundancy more quickly. When a disk failure is detected, the ZFS Event Daemon (ZED) automatically begins resilvering to the distributed hot spare — using sequential resilvering for dRAID, while RAID-Z must use traditional healing resilvering.

Destroying Storage Pools

Destroy unneeded pools to reclaim disks. zpool destroy automatically unmounts all datasets; if a dataset is in use (with processes having open files), the destroy operation will fail. In this case, use -f to force the destruction.

Adding and Removing Devices

There are two ways to add disks to a pool: use zpool attach to attach a disk to an existing vdev, or use zpool add to add a new vdev to the pool. Some vdev types allow adding disks after creation.

Attaching Disks to an Existing vdev

A pool created from a single disk has no redundancy: corruption can be detected, but without data copies, it cannot be repaired. The copies property may recover data from minor failures (such as bad sectors), but its protection level is inferior to mirroring or RAID-Z. If the current pool contains only a single-disk vdev, zpool attach can be used to attach a new disk to the vdev to create a mirror. zpool attach can also be used to add a new disk to an existing mirror group, thereby enhancing redundancy and improving read performance. If the disks use partitions, the partition layout of the first disk should be replicated to the second disk. Using gpart backup and gpart restore can simplify this process.

Upgrade the single-disk (stripe) vdev nda0p3 to a mirror by attaching nda1p1:

Warning

gpart create -s gpt will erase the existing partition table on the target disk, and all original partitions and data will be unrecoverable. Make sure to confirm that /dev/nda1 is the correct target disk, not the system disk or another data disk.

First, view the current pool configuration to confirm there is only a single-disk vdev:

Create a GPT partition table on the new disk and add a ZFS partition:

Attach the new partition to the existing vdev to create a mirror:

View the resilver progress; the mirror is currently synchronizing data:

After resilvering completes, confirm the mirror has returned to normal status:

Adding a New vdev to the Pool

If adding disks to an existing vdev is not desired, the alternative is to add another vdev to the pool. Adding vdevs can distribute writes across multiple vdevs, thereby improving performance. Each vdev provides its own redundancy. mirror and RAID-Z type vdevs can be mixed, but this practice is not recommended. Adding a non-redundant vdev to a pool containing mirror or RAID-Z vdevs compromises the data safety of the entire pool — since writes are distributed, a failure of the non-redundant disk would result in partial data loss for every data block in the pool.

ZFS stripes data across vdevs. For example, using two mirror vdevs effectively constitutes RAID 10, with writes striped across both mirror groups. ZFS attempts to balance space allocation so that all vdevs reach 100% utilization simultaneously. Since more writes are directed to vdevs with more free space, imbalanced free space among vdevs can degrade performance.

When adding new devices to a boot pool, be sure to update the boot code.

Add the second mirror group (nda2p1 and nda3p1) as a new vdev to the pool:

First, view the current pool configuration:

Create GPT partition tables for the new disks and add ZFS partitions:

Add the new disks as a second mirror vdev to the pool:

After adding, confirm the pool now contains two mirror vdevs:

Removing Top-Level vdevs

After enabling the device_removal feature flag, top-level vdevs can be removed from the pool.

Check whether the current pool has this feature enabled:

Removing Disks from a Mirror

Disks can also be removed from a mirror, provided sufficient redundancy remains. If only one disk remains in a mirror group, the group will no longer operate as a mirror and will degrade to a stripe; if this disk fails, the entire pool's data is at risk.

Remove a disk from the mirror group:

First, view the current pool configuration:

Remove a disk from the second mirror group and view the status after removal:

Remove the remaining one:

Removal in progress; view the data evacuation progress:

Removal complete; the pool returns to a single mirror group:

Checking Pool Status

Monitoring pool status is important. If a hard drive goes offline, or ZFS detects read, write, or checksum errors, the corresponding error counts will increase. The status output displays the configuration and state of each device in the pool, as well as the overall pool state, and also shows details of the most recent scrub operation.

Clearing Errors

After detecting errors, ZFS increments the corresponding read, write, or checksum error counts. Use zpool clear <mypool> to clear error messages and reset the counts. Clearing error states is critical for automated scripts that alert administrators when a pool encounters errors. If old errors are not cleared, scripts may fail to report subsequent errors.

Replacing Healthy Devices

Sometimes it is necessary to replace an existing disk with a different physical disk. When replacing a healthy disk, the old disk remains online during the operation, and the pool does not enter a degraded state, thereby reducing the risk of data loss. Running zpool replace copies data from the old disk to the new disk. Once the operation completes, ZFS disconnects the old disk from the vdev.

Replace a healthy device in the pool:

First, view the current pool configuration:

Execute the replacement operation, replacing nda0p3 with the new disk nda2p1:

View the replacement progress; data is currently being resilvered:

After resilvering completes, confirm the new disk is online and the old disk has been removed:

Update the EFI boot entry so the new disk can also boot:

Handling Failed Devices

When a disk in a pool fails, the vdev containing that disk enters a degraded state. Data remains accessible, but performance may degrade because ZFS must compute missing data from redundant copies. To restore the vdev to a fully operational state, the failed physical device must be replaced, and ZFS must then be instructed to begin a resilver operation. ZFS recalculates the data from the failed device using available redundant data and writes it to the replacement device. Once the operation completes, the vdev returns to an online state.

If the vdev has no redundancy, or if devices have failed and redundancy is insufficient to compensate, the pool enters a faulted state. At this point, the pool is unavailable unless enough devices are reattached, and data must be restored from backups.

After replacing a failed disk, the failed disk's name changes to the new disk's GUID. If the replacement device has the same name as the original device, zpool replace does not need to specify the new device name.

Use the zpool replace command to replace a failed disk:

First, view the pool status to confirm the failed device and its GUID:

Execute the replacement using the failed device's GUID and the new disk:

View the resilver progress:

Resilvering complete; the pool has returned to the ONLINE state:

Taking Devices Offline and Online

zpool offline takes a physical device offline — while offline, ZFS performs no reads or writes to the device. This operation does not apply to spare disks. If there is only one device, an error will be reported: cannot offline nda1: no valid replicas.

Create 2 virtual disks of 1G each under /tmp for demonstration:

Create the mirror pool mypool, then view its status:

Take the device /tmp/vdisk1 offline from the storage pool mypool:

View the pool status to confirm the device is offline:

  • The -t option makes the offline state temporary — the device will revert to its previous state after a system reboot.

  • The -f option forcibly marks the device as FAULTED rather than offline, and the faulted state persists across import/export (unless -t is also used).

zpool online brings an offline device back online:

After bringing the device online, if there is unwritten data from when the device was offline, ZFS automatically triggers a resilver to synchronize the data. For devices in a mirror or RAID-Z, new space only becomes available to the pool after all devices have been expanded.

If a device is only temporarily disconnected, it can be brought back online directly without performing a full replacement procedure.

Data Integrity Checking (scrub)

It is recommended to regularly scrub pools, preferably at least once per month.

The scrub operation consumes disk resources, so it can degrade performance while running. Avoid scheduling scrub operations during periods of high load, or use vfs.zfs.scrub_delay to adjust the relative priority of scrub to avoid impacting other workloads.

Perform a data integrity check on the pool:

View the scrub progress:

If you need to cancel a scrub operation, run zpool scrub -s pool_name.

Disk Initialization

Before new disks are added to a storage pool, a disk initialization operation can be performed first.

zpool initialize writes data (zeros by default) to all unallocated regions on the disk, enabling ZFS to detect potential bad blocks in advance — if an unrecoverable read or write error is encountered during initialization, the disk immediately reports a failure, thereby avoiding the risk of writing data to bad blocks. Only data devices and log devices can be initialized.

Initialize all devices in the pool:

Initialize a single disk in the mypool pool:

Initialization progress can be viewed with zpool status:

The -w option makes the command wait until initialization completes before returning:

Initialization can be canceled (-c), suspended (-s), or the initialization state can be cleared (-u):

After clearing the initialization state, the device can be initialized again from scratch. Unlike zpool trim, initialize is a one-time operation, while trim is used to notify SSDs to reclaim released blocks.

Storage Pool Checkpoints

zpool checkpoint creates an instantaneous checkpoint of the entire storage pool, preserving the pool's complete consistent state. If a destructive operation (such as zpool upgrade or accidental dataset deletion) is performed afterwards, recovery is possible by exporting and then importing the pool with a rollback to the checkpoint.

Create a checkpoint:

While a checkpoint exists, the following commands cannot be executed: zpool remove, zpool attach, zpool detach, zpool split, zpool reguid. Additionally, if the pool has insufficient free space, the checkpoint may break reserved space limits.

View the checkpoint status:

View the checkpoint space usage (CKPOINT column) via zpool list:

Discard the checkpoint to free space:

The -w option makes the command wait until the checkpoint discard completes before returning:

Rolling back to a checkpoint requires exporting the pool first, then importing with --rewind-to-checkpoint:

Warning

If zroot is used as the root partition and is currently in use, exporting is prohibited.

A typical use case for checkpoints is establishing a rollback point before pool-level upgrades or other high-risk administrative operations.

Importing and Exporting Pools

Before migrating a pool to another system, the pool should be exported first. ZFS unmounts all datasets, marks each device as exported, and locks the devices to prevent use by other disk subsystems. Exported pools can be imported on other ZFS-supported computers, operating systems, and even different hardware architectures (with some caveats). If datasets have open files, zpool export -f can be used to force-export the pool, but exercise caution. Forced export forcibly unmounts datasets, which may cause unexpected behavior if applications still hold open files at the time of unmounting.

Export an unused pool:

Importing a pool automatically mounts datasets. To prevent automatic mounting, use zpool import -N. zpool import -o can set temporary properties for the current import. zpool import -R can specify an alternate mount point for the imported pool, replacing the filesystem root. This can also be set using zpool import -o altroot=. If the pool was previously running on another system and was not properly exported, zpool import -f can force the import. zpool import -a imports all pools not in use by other systems.

List all available pools for import:

Import the pool using an alternate root directory:

Confirm the dataset is mounted under the alternate root directory:

Self-Healing

Self-Healing Principles

ZFS's self-healing capability stems from the combination of checksums and storage redundancy. Each data block is accompanied by a checksum, with the algorithm determined by the dataset property. Checksums are automatically and transparently verified on read, enabling ZFS to detect silent data corruption. If the data does not match the expected checksum, ZFS attempts to recover using available redundancy (mirror or RAID-Z). The checksum algorithm used for each block is stored in the block pointer (metadata) and is computed when the block is written, so changing the algorithm only affects writes that occur after the change.

Change the checksum algorithm for a dataset:

Supported checksum algorithms:

Algorithm
Suitable for dedup and nopwrite?
Compatible with other ZFS implementations?
Description

on

See description

Yes

fletcher4 for non-dedup datasets, sha256 for dedup datasets

off

No

Yes

Checksums should not be disabled

fletcher2

No

Yes

Deprecated, retained only for backward compatibility; should not be used on new datasets; use fletcher4 instead

fletcher4

No

Yes

Fletcher algorithm, also used for zfs send streams

sha256

Yes

Yes

Default algorithm for dedup datasets

noparity

No

Yes

noparity should not be used; this option is retained only for backward compatibility

sha512

Yes

Requires org.illumos:sha512 to be enabled

Salted sha512; currently not supported for any filesystem on a boot pool

skein

Yes

Requires org.illumos:skein to be enabled

Salted skein; currently not supported for any filesystem on a boot pool

edonr

See description

Requires org.illumos:edonr to be enabled

Salted edonr; when used with dedup, verify is automatically enabled as a precaution; currently not supported for any filesystem on a boot pool

blake3

Yes

Requires org.openzfs:blake3 to be enabled

Salted blake3; currently not supported for any filesystem on a boot pool

Although disabling checksums to improve CPU performance may seem tempting, the ZFS community generally considers this an extremely unwise practice. Checksums should not be disabled.

Self-Healing Demonstration

The following example uses two md(4) memory disks (md0, md1) to build a mirror pool healer, safely demonstrating self-healing behavior — md pseudo-devices are created only in memory and automatically disappear after reboot, with absolutely no risk of damaging real disks.

Create two 128 MB memory disks:

Create a mirror pool using memory disks:

View the status of the newly created mirror pool:

View the pool's capacity and allocation:

Copy some non-critical data to the healer storage pool to guard against data corruption due to errors, then generate a checksum of the pool for later comparison.

This example uses the Philosophische-untersuchungen.pdf file as a sample.

Confirm the file has been written to the pool:

Generate a SHA-256 checksum of the file for later comparison:

View the checksum file contents:

Writing random data to one disk in the mirror can simulate data corruption. To prevent ZFS from automatically repairing data after detecting corruption, export the pool first, then re-import it after the corruption is done.

Safety Note

This example uses md memory disks — writing to /dev/md1 only affects the in-memory pseudo-device and poses absolutely no threat to physical disks. md devices completely disappear after a system reboot or executing mdconfig -d. The entire demonstration is completely independent of real storage hardware. If simulating in other environments, always use md disks or file vdevs (created with truncate -s); never perform the following operations on real pool member disks.

First, export the pool to simulate data corruption in an offline state:

Write random data to one memory disk in the mirror to simulate disk corruption:

Re-import the pool; ZFS will attempt to identify and repair detected data errors:

The pool status shows that the md1 device experienced checksum errors, but the data read by applications is correct — ZFS served the data from the healthy md0 mirror disk:

ZFS detected the error and repaired it using redundant data from the unaffected md0 mirror disk. By comparing with the original checksum, it can be confirmed whether the pool data is consistent.

Comparing the checksums generated before and after the intentional data corruption, the data remains consistent. This demonstrates that ZFS can automatically detect and repair errors even when checksums are inconsistent.

Note that this capability only holds when the pool has sufficient redundancy — a pool composed of a single device has no self-healing capability. This is precisely why checksums are indispensable in ZFS and should never be disabled for any reason.

Through this mechanism, ZFS can detect and repair these issues without relying on traditional filesystem consistency check programs (fsck), while maintaining pool availability when problems occur.

Use zpool clear to clear error messages in the pool status.

After clearing the errors, confirm the pool has returned to a fully normal state with all error counts reset to zero:

At this point, the pool has returned to a fully normal operating state, with all error counts reset to zero.

After the demonstration, use the following commands to reclaim resources:

Warning

The following commands will permanently destroy the ZFS storage pool healer and delete the md memory disk devices; all data in the pool will be unrecoverable. Confirm the target is correct before executing.

Displaying Recorded Pool History

ZFS records all commands that change pool state, including creating datasets, changing properties, or replacing disks. Viewing the pool's creation history helps trace administrative operations and confirm what actions a specific user performed and when. The history is not stored in a log file but is part of the pool structure. The command to view the history of pool zroot is zpool history:

The output shows how zpool and zfs commands record pool-state-changing operations with timestamps. Commands like zfs list are not included. If no pool name is specified, ZFS displays the history of all pools.

If the -i or -l option is specified, zpool history can display more information. -i shows user-initiated events as well as internally recorded ZFS events.

Adding -l shows more details. The long-format history displays the user who executed the command and the hostname where the command occurred.

The output shows that the root user created a single-disk pool zroot using /dev/nda0p3, and subsequently created the ROOT and ROOT/default datasets. The hostname freebsd also appears in the command records after pool creation (see ykla:freebsd in the -l output). Hostname information is particularly important when a pool is exported from one system and imported into another — administrators can use the hostname to distinguish commands executed on different systems.

Combining both options, zpool history can provide the most detailed information about a pool. Pool history records can be used for tracking operations and debugging.

Performance Monitoring

ZFS has a built-in monitoring system that can display pool I/O statistics in real time, including used and free space, read and write operations per second, and I/O bandwidth.

By default, ZFS monitors and displays the status of all pools in the system. To monitor only a specific pool, specify its name. Here is a basic example:

To continuously view I/O activity, specify a numeric argument at the end of the command representing the update interval in seconds. At each interval, the system outputs a set of statistics. Press Ctrl + C to stop continuous monitoring. To also specify the number of statistics displays, add a second number on the command line.

Using -v displays more detailed I/O statistics, with one line per device in the pool. By examining the read and write operations of each device, you can determine whether a particular device is slowing down pool performance. Here is an example of a mirror pool with two devices:

Splitting Storage Pools

ZFS can split a pool containing one or more mirror vdevs into two pools. Unless otherwise specified, ZFS detaches the last member from each mirror and creates a new pool containing identical data. Before executing, it is recommended to first use the -n option for a dry run — it displays the detailed process of the requested operation without actually executing it. This helps confirm the operation will proceed as expected.

Previous28.2 ZFS Features and TerminologyNext28.4 Updating ZFS Storage Pools

Last updated