> For the complete documentation index, see [llms.txt](https://book.bsdcn.org/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://book.bsdcn.org/ask/flat/chapter-28-the-zfs-file-system/di-28.3-jie-zfs-cun-chu-chi-guan-li.md).
# 28.3 ZFS Storage Pool Management
This section introduces several ZFS storage pool management methods.
## ZFS Storage Pool Expansion
ZFS supports online expansion of storage pools without service interruption. The expansion process consists of two steps: first, adjust the underlying partition size to utilize the newly added space, then use ZFS commands to make the storage pool recognize the new capacity of the device.
The available capacity of a redundant pool depends on the smallest device in each vdev. After replacing the smallest device with a larger-capacity device, the pool capacity can be expanded.
After a replace or resilver operation completes, the pool's available space will grow to the capacity of the new device.
Take a mirror pool consisting of 1TB and 2TB hard drives as an example: the available space is 1TB. After replacing the 1TB hard drive with another 2TB hard drive, the resilver operation will copy existing data to the new hard drive.
Both hard drives now have a capacity of 2TB, so the available space of the mirror pool will increase to 2TB.
> **Warning**
>
> ZFS storage pools can only be expanded, not shrunk. Therefore, changes cannot be undone.
### Resizing Partitions
Display the current disk partition table and partition information:
```sh
# gpart show
=> 40 167772087 nda0 GPT (80G)
40 532480 1 efi (260M)
532520 1024 2 freebsd-boot (512K)
533544 984 - free - (492K)
534528 4194304 3 freebsd-swap (2.0G)
4728832 142071775 4 freebsd-zfs (68G)
146800607 20971520 - free - (10G)
```
From the output, the `free` space is 10 GB.
Based on the partition table information, select the freebsd-zfs partition located before the free space (partition 4 in this example) to expand. Confirm the partition number before proceeding:
```sh
# gpart resize -i 4 nda0 # Expand partition 4 on disk nda0 to all available space
nda0p4 resized
```
Display the current disk partition table and partition information:
```sh
# gpart show
=> 40 167772087 nda0 GPT (80G)
40 532480 1 efi (260M)
532520 1024 2 freebsd-boot (512K)
533544 984 - free - (492K)
534528 4194304 3 freebsd-swap (2.0G)
4728832 163043295 4 freebsd-zfs (78G)
```
List all ZFS pools in the system and their status information:
```sh
# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zroot 67.5G 2.20G 65.3G - - 2% 3% 1.00x ONLINE - # Partition expansion only adjusted the partition table; the ZFS pool has not yet detected the underlying device's space change and needs to be expanded separately
```
Display detailed status information of the ZFS pool, including health status and error information:
```sh
# zpool status
pool: zroot
state: ONLINE
status: Some supported and requested features are not enabled on the pool.
The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
the pool may no longer be accessible by software that does not support
the features. See zpool-features(7) for details.
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0 # Default pool name
nda0p4 ONLINE 0 0 0 # Corresponding partition
errors: No known data errors
```
### Expanding the ZFS Pool
Expand the ZFS pool:
```sh
# zpool online -e zroot nda0p4 # Online expand the nda0p4 partition in ZFS pool zroot (-e means expand)
```
View all ZFS pools and their capacity, usage, and health status after expansion:
```sh
# zpool list
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zroot 77.5G 2.20G 75.3G - - 2% 2% 1.00x ONLINE - # Available capacity after expansion is 75.3G, as expected
```
The expansion operation is complete. Additionally, by setting the pool property `autoexpand`, the pool can automatically expand after disk replacement or underlying device expansion, without the need to manually run `zpool online -e` each time:
```sh
# zpool set autoexpand=on zroot
```
### Appendix
This appendix introduces additional usage of the `gpart` command. You can obtain partition numbers using the `gpart show` command, or use the `-p` parameter to display all disk and partition information with full paths:
```sh
# gpart show -p
=> 40 244277168 mmcsd0 GPT (116G)
40 532480 mmcsd0p1 efi (260M)
532520 2008 - free - (1.0M)
534528 243740672 mmcsd0p2 freebsd-zfs (116G)
244275200 2008 - free - (1.0M)
=> 34 976773101 nda0 GPT (466G)
34 6 - free - (3.0K)
40 567256 nda0p1 efi (277M)
567296 419436064 nda0p2 ms-basic-data (200G)
420003360 310592132 nda0p3 ms-basic-data (148G)
730595492 4 - free - (2.0K)
730595496 177626968 nda0p4 ms-basic-data (85G)
908222464 67100672 nda0p5 freebsd-swap (32G)
975323136 1445937 nda0p6 ms-recovery (706M)
976769073 4062 - free - (2.0M)
```
* Print partition type GUIDs (for GPT) or raw partition types (for MBR):
Display disk and partition information in a reproducible and complete path format:
```sh
# gpart show -rp
=> 40 244277168 mmcsd0 GPT (116G)
40 532480 mmcsd0p1 c12a7328-f81f-11d2-ba4b-00a0c93ec93b (260M)
532520 2008 - free - (1.0M)
534528 243740672 mmcsd0p2 516e7cba-6ecf-11d6-8ff8-00022d09712b (116G)
244275200 2008 - free - (1.0M)
=> 34 976773101 nda0 GPT (466G)
34 6 - free - (3.0K)
40 567256 nda0p1 c12a7328-f81f-11d2-ba4b-00a0c93ec93b (277M)
567296 419436064 nda0p2 ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 (200G)
420003360 310592132 nda0p3 ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 (148G)
730595492 4 - free - (2.0K)
730595496 177626968 nda0p4 ebd0a0a2-b9e5-4433-87c0-68b6b72699c7 (85G)
908222464 67100672 nda0p5 516e7cb5-6ecf-11d6-8ff8-00022d09712b (32G)
975323136 1445937 nda0p6 de94bba4-06d1-4d40-a16a-bfd50179d6ac (706M)
976769073 4062 - free - (2.0M)
```
* Display detailed partition information for disk `mmcsd0`:
```sh
# gpart list mmcsd0
```
### Troubleshooting
#### `gpart: table 'ada0' is corrupt: Operation not permitted`
This indicates a partition table error and requires resetting the GPT partition table. The `gpart recover` command restores the main partition table information from the backup GPT header. This issue commonly occurs with raw disk images imported directly.
```sh
# gpart recover ada0
```
### References
* Yubao Liu. FreeBSD root on ZFS (Notorious Pitfalls)\[EB/OL]. (2019-02-18)\[2026-04-02]. . Documents similar ZFS boot error cases.
* FreeBSD Forums. 10.1 doesn't boot anymore from zroot after applying p25\[EB/OL]. \[2026-04-02]. . Related discussion on the FreeBSD Forums.
* FreeBSD Forums. Solved-extend ZFS partition\[EB/OL]. \[2026-04-02]. . ZFS partition expansion solution on the FreeBSD Forums.
## Creating and Destroying Storage Pools
When creating a ZFS storage pool, some decisions cannot be changed, the most important being how physical disks are grouped by vdev type. After pool creation, mirrors allow adding new disks to the vdev; stripes can also be upgraded to mirrors by attaching a new disk to the vdev. With the `raidz_expansion` feature flag enabled, RAID-Z vdevs can also be expanded. Although adding new vdevs can expand a pool, the vdev type cannot be changed after creation. The only option is to back up data, destroy the pool, and then recreate it.
### Creating a Simple Mirror Pool
Create a simple mirror pool:
```sh
# zpool create mypool mirror /dev/nda1 /dev/nda2
```
View the pool status:
```sh
# zpool status mypool
pool: mypool
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
mypool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nda1 ONLINE 0 0 0
nda2 ONLINE 0 0 0
errors: No known data errors
```
### Creating Multiple vdevs
To create multiple vdevs in a single command, specify groups of disks separated by vdev type keywords (`mirror` in this example):
```sh
# zpool create mypool mirror /dev/nda1 /dev/nda2 mirror /dev/nda3 /dev/nda4
```
View the pool status:
```sh
# zpool status mypool
pool: mypool
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
mypool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nda1 ONLINE 0 0 0
nda2 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
nda3 ONLINE 0 0 0
nda4 ONLINE 0 0 0
errors: No known data errors
```
### Creating a Pool Using Partitions
Pools can also use partitions rather than entire disks. By placing ZFS on a separate partition, the remaining space on the same disk is available for other purposes. In particular, partitions containing boot code and a boot filesystem can be added, enabling the pool member disks to also serve as boot disks. Using partitions instead of entire disks does not impose additional performance overhead on FreeBSD. Additionally, partitions support **underprovisioning** — allocating less space than the disk's total capacity. If a replacement disk has the same nominal size as the original but slightly less actual capacity, the smaller partition can still fit on the replacement disk.
Create a RAID-Z2 pool using partitions:
```sh
# zpool create mypool raidz2 /dev/ada0p3 /dev/ada1p3 /dev/ada2p3 /dev/ada3p3 /dev/ada4p3 /dev/ada5p3
# zpool status
pool: mypool
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
mypool ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
ada0p3 ONLINE 0 0 0
ada1p3 ONLINE 0 0 0
ada2p3 ONLINE 0 0 0
ada3p3 ONLINE 0 0 0
ada4p3 ONLINE 0 0 0
ada5p3 ONLINE 0 0 0
errors: No known data errors
```
### dRAID Parameters and IO Formulas
* **Distributed RAID-Z (dRAID)**: dRAID evolved from the RAID-Z concept and is a standalone vdev type that provides integrated distributed hot spares and supports faster sequential resilvering. dRAID consists of multiple internal raidz groups, each containing D data devices and P parity devices. These groups are distributed across all child devices to fully utilize disk performance — a technique known as parity declustering. Additionally, dRAID randomly arranges child vdevs through carefully chosen precomputed permutation maps, ensuring that resilver I/O (both reads and writes) is evenly distributed across all surviving disks regardless of which disk fails, thereby significantly accelerating the resilver process. dRAID uses a fixed stripe width (padding with zeros when necessary), which enables sequential resilvering, but the fixed stripe width significantly affects available capacity and IOPS. For example, with the default D=8 and 4 KiB disk sectors, the minimum allocation size is 32 KiB; when using compression, larger allocation sizes may reduce the effective compression ratio. The random IOPS of dRAID can be approximated as `floor((N-S)/(D+P)) × single-disk IOPS`. If a dRAID pool will store a large amount of small-block data, it is recommended to additionally add a mirror special vdev. When using ZFS volumes with dRAID, the default `volblocksize` property is automatically increased to accommodate the allocation size. draid1 (single parity), draid2 (double parity), and draid3 (triple parity) are supported; draid is an alias for draid1.
Syntax for creating a dRAID vdev:
```sh
# zpool create draid[][:d][:c][:s]
```
dRAID parameter descriptions:
| Parameter | Description | Default |
| --------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- |
| parity | Parity level (1-3) | 1 |
| data | Number of data devices per redundancy group. A smaller D value improves IOPS, compression ratio, and resilver speed, at the cost of reduced total available capacity | 8 (unless N-P-S is less than 8) |
| children | Expected number of child devices, used for cross-validation | - |
| spares | Number of distributed hot spares | 0 |
For example, create an 11-disk dRAID pool with 4+1 redundancy and 1 distributed hot spare:
```sh
# zpool create tank draid:4d:1s:11c /dev/ada0 /dev/ada1 /dev/ada2 /dev/ada3 /dev/ada4 /dev/ada5 /dev/ada6 /dev/ada7 /dev/ada8 /dev/ada9 /dev/ada10
```
The main advantage of dRAID is its support for sequential resilvering: resilver performance scales with the number of disks divided by the stripe width (D+P), which can significantly reduce resilver time and restore full redundancy more quickly. When a disk failure is detected, the ZFS Event Daemon (ZED) automatically begins resilvering to the distributed hot spare — using sequential resilvering for dRAID, while RAID-Z must use traditional healing resilvering.
### Destroying Storage Pools
Destroy unneeded pools to reclaim disks. `zpool destroy` automatically unmounts all datasets; if a dataset is in use (with processes having open files), the destroy operation will fail. In this case, use `-f` to force the destruction.
## Adding and Removing Devices
There are two ways to add disks to a pool: use `zpool attach` to attach a disk to an existing vdev, or use `zpool add` to add a new vdev to the pool. Some vdev types allow adding disks after creation.
### Attaching Disks to an Existing vdev
A pool created from a single disk has no redundancy: corruption can be detected, but without data copies, it cannot be repaired. The copies property may recover data from minor failures (such as bad sectors), but its protection level is inferior to mirroring or RAID-Z. If the current pool contains only a single-disk vdev, `zpool attach` can be used to attach a new disk to the vdev to create a mirror. `zpool attach` can also be used to add a new disk to an existing mirror group, thereby enhancing redundancy and improving read performance. If the disks use partitions, the partition layout of the first disk should be replicated to the second disk. Using `gpart backup` and `gpart restore` can simplify this process.
Upgrade the single-disk (stripe) vdev **nda0p3** to a mirror by attaching **nda1p1**:
> **Warning**
>
> `gpart create -s gpt` will erase the existing partition table on the target disk, and all original partitions and data will be unrecoverable. Make sure to confirm that **/dev/nda1** is the correct target disk, not the system disk or another data disk.
First, view the current pool configuration to confirm there is only a single-disk vdev:
```sh
# zpool status
pool: zroot
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
nda0p3 ONLINE 0 0 0
errors: No known data errors
```
Create a GPT partition table on the new disk and add a ZFS partition:
```sh
# gpart create -s gpt /dev/nda1
nda1 created
# gpart add -t freebsd-zfs /dev/nda1
nda1p1 added
```
Attach the new partition to the existing vdev to create a mirror:
```sh
# zpool attach zroot nda0p3 nda1p1
```
View the resilver progress; the mirror is currently synchronizing data:
```sh
# zpool status
pool: zroot
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue May 12 13:03:48 2026
888M / 888M scanned, 843M / 888M issued at 421M/s
848M resilvered, 94.90% done, 00:00:00 to go
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nda0p3 ONLINE 0 0 0
nda1p1 ONLINE 0 0 0 (resilvering)
errors: No known data errors
```
After resilvering completes, confirm the mirror has returned to normal status:
```sh
# zpool status zroot
pool: zroot
state: ONLINE
scan: resilvered 912M in 00:00:33 with 0 errors on Tue May 12 13:04:21 2026
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nda0p3 ONLINE 0 0 0
nda1p1 ONLINE 0 0 0
errors: No known data errors
```
### Adding a New vdev to the Pool
If adding disks to an existing vdev is not desired, the alternative is to add another vdev to the pool. Adding vdevs can distribute writes across multiple vdevs, thereby improving performance. Each vdev provides its own redundancy. `mirror` and `RAID-Z` type vdevs can be mixed, but this practice is not recommended. Adding a non-redundant vdev to a pool containing mirror or RAID-Z vdevs compromises the data safety of the entire pool — since writes are distributed, a failure of the non-redundant disk would result in partial data loss for every data block in the pool.
ZFS stripes data across vdevs. For example, using two mirror vdevs effectively constitutes RAID 10, with writes striped across both mirror groups. ZFS attempts to balance space allocation so that all vdevs reach 100% utilization simultaneously. Since more writes are directed to vdevs with more free space, imbalanced free space among vdevs can degrade performance.
When adding new devices to a boot pool, be sure to update the boot code.
Add the second mirror group (**nda2p1** and **nda3p1**) as a new vdev to the pool:
First, view the current pool configuration:
```sh
# zpool status
pool: zroot
state: ONLINE
scan: resilvered 912M in 00:00:33 with 0 errors on Tue May 12 13:04:21 2026
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nda0p3 ONLINE 0 0 0
nda1p1 ONLINE 0 0 0
errors: No known data errors
```
Create GPT partition tables for the new disks and add ZFS partitions:
```sh
# gpart create -s gpt /dev/nda2
nda2 created
# gpart add -t freebsd-zfs /dev/nda2
nda2p1 added
# gpart create -s gpt /dev/nda3
nda3 created
# gpart add -t freebsd-zfs /dev/nda3
nda3p1 added
```
Add the new disks as a second mirror vdev to the pool:
```sh
# zpool add zroot mirror nda2p1 nda3p1
```
After adding, confirm the pool now contains two mirror vdevs:
```sh
# zpool status
pool: zroot
state: ONLINE
scan: resilvered 912M in 00:00:33 with 0 errors on Tue May 12 13:04:21 2026
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nda0p3 ONLINE 0 0 0
nda1p1 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
nda2p1 ONLINE 0 0 0
nda3p1 ONLINE 0 0 0
errors: No known data errors
```
### Removing Top-Level vdevs
After enabling the `device_removal` feature flag, top-level vdevs can be removed from the pool.
Check whether the current pool has this feature enabled:
```sh
# zpool get feature@device_removal zroot
NAME PROPERTY VALUE SOURCE
zroot feature@device_removal enabled local
```
### Removing Disks from a Mirror
Disks can also be removed from a mirror, provided sufficient redundancy remains. If only one disk remains in a mirror group, the group will no longer operate as a mirror and will degrade to a stripe; if this disk fails, the entire pool's data is at risk.
Remove a disk from the mirror group:
First, view the current pool configuration:
```sh
# zpool status
pool: zroot
state: ONLINE
scan: resilvered 912M in 00:00:33 with 0 errors on Tue May 12 13:04:21 2026
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nda0p3 ONLINE 0 0 0
nda1p1 ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
nda2p1 ONLINE 0 0 0
nda3p1 ONLINE 0 0 0
errors: No known data errors
```
Remove a disk from the second mirror group and view the status after removal:
```sh
# zpool detach zroot nda3p1
# zpool status
pool: zroot
state: ONLINE
scan: resilvered 912M in 00:00:33 with 0 errors on Tue May 12 13:04:21 2026
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nda0p3 ONLINE 0 0 0
nda1p1 ONLINE 0 0 0
nda2p1 ONLINE 0 0 0
errors: No known data errors
```
Remove the remaining one:
```sh
zpool detach zroot nda2p1
```
Removal in progress; view the data evacuation progress:
```sh
# zpool status
pool: zroot
state: ONLINE
scan: resilvered 912M in 00:00:33 with 0 errors on Tue May 12 13:04:21 2026
remove: Evacuation of /dev/nda2p1 in progress since Tue May 12 13:21:33 2026
444K copied out of 444K at 14.8K/s, 100.00% done, 0h0m to go
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nda0p3 ONLINE 0 0 0
nda1p1 ONLINE 0 0 0
nda2p1 ONLINE 0 0 0 (removing)
```
Removal complete; the pool returns to a single mirror group:
```sh
pool: zroot
state: ONLINE
scan: resilvered 912M in 00:00:33 with 0 errors on Tue May 12 13:04:21 2026
remove: Removal of vdev 1 copied 444K in 0h0m, completed on Tue May 12 13:22:03 2026
648 memory used for removed device mappings
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nda0p3 ONLINE 0 0 0
nda1p1 ONLINE 0 0 0
errors: No known data errors
```
## Checking Pool Status
Monitoring pool status is important. If a hard drive goes offline, or ZFS detects read, write, or checksum errors, the corresponding error counts will increase. The `status` output displays the configuration and state of each device in the pool, as well as the overall pool state, and also shows details of the most recent `scrub` operation.
```sh
# zpool status
pool: zroot
state: ONLINE
scan: resilvered 912M in 00:00:33 with 0 errors on Tue May 12 13:04:21 2026
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nda0p3 ONLINE 0 0 0
nda1p1 ONLINE 0 0 0
nda2p1 ONLINE 0 0 0
errors: No known data errors
```
## Clearing Errors
After detecting errors, ZFS increments the corresponding read, write, or checksum error counts. Use `zpool clear ` to clear error messages and reset the counts. Clearing error states is critical for automated scripts that alert administrators when a pool encounters errors. If old errors are not cleared, scripts may fail to report subsequent errors.
## Replacing Healthy Devices
Sometimes it is necessary to replace an existing disk with a different physical disk. When replacing a healthy disk, the old disk remains online during the operation, and the pool does not enter a degraded state, thereby reducing the risk of data loss. Running `zpool replace` copies data from the old disk to the new disk. Once the operation completes, ZFS disconnects the old disk from the vdev.
Replace a healthy device in the pool:
First, view the current pool configuration:
```sh
# zpool status
pool: zroot
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nda0p3 ONLINE 0 0 0
nda1p1 ONLINE 0 0 0
errors: No known data errors
```
Execute the replacement operation, replacing nda0p3 with the new disk nda2p1:
```sh
# zpool replace zroot nda0p3 nda2p1
```
View the replacement progress; data is currently being resilvered:
```sh
# zpool status
pool: zroot
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue May 12 13:27:31 2026
889M / 889M scanned, 889M / 889M issued at 444M/s
912M resilvered, 100.00% done, 00:00:00 to go
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
replacing-0 ONLINE 0 0 0
nda0p3 ONLINE 0 0 0
nda2p1 ONLINE 0 0 0 (resilvering)
nda1p1 ONLINE 0 0 0
errors: No known data errors
```
After resilvering completes, confirm the new disk is online and the old disk has been removed:
```sh
# zpool status
pool: zroot
state: ONLINE
scan: resilvered 913M in 00:00:33 with 0 errors on Tue May 12 13:28:04 2026
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nda2p1 ONLINE 0 0 0
nda1p1 ONLINE 0 0 0
errors: No known data errors
```
Update the EFI boot entry so the new disk can also boot:
```sh
# efibootmgr -a -c -l /boot/efi/efi/freebsd/loader.efi -L "FreeBSD 15.0"
```
## Handling Failed Devices
When a disk in a pool fails, the vdev containing that disk enters a degraded state. Data remains accessible, but performance may degrade because ZFS must compute missing data from redundant copies. To restore the vdev to a fully operational state, the failed physical device must be replaced, and ZFS must then be instructed to begin a resilver operation. ZFS recalculates the data from the failed device using available redundant data and writes it to the replacement device. Once the operation completes, the vdev returns to an online state.
If the vdev has no redundancy, or if devices have failed and redundancy is insufficient to compensate, the pool enters a faulted state. At this point, the pool is unavailable unless enough devices are reattached, and data must be restored from backups.
After replacing a failed disk, the failed disk's name changes to the new disk's GUID. If the replacement device has the same name as the original device, `zpool replace` does not need to specify the new device name.
Use the `zpool replace` command to replace a failed disk:
First, view the pool status to confirm the failed device and its GUID:
```sh
# zpool status
pool: mypool
state: DEGRADED
status: One or more devices could not be opened. Sufficient replicas exist for
the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
see: http://illumos.org/msg/ZFS-8000-2Q
scan: none requested
config:
NAME STATE READ WRITE CKSUM
mypool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
ada0p3 ONLINE 0 0 0
316502962686821739 UNAVAIL 0 0 0 was /dev/ada1p3
errors: No known data errors
```
Execute the replacement using the failed device's GUID and the new disk:
```sh
# zpool replace mypool 316502962686821739 ada2p3
```
View the resilver progress:
```sh
# zpool status
pool: mypool
state: DEGRADED
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Mon Jun 2 14:52:21 2014
641M scanned out of 781M at 49.3M/s, 0h0m to go
640M resilvered, 82.04% done
config:
NAME STATE READ WRITE CKSUM
mypool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
ada0p3 ONLINE 0 0 0
replacing-1 UNAVAIL 0 0 0
15732067398082357289 UNAVAIL 0 0 0 was /dev/ada1p3/old
ada2p3 ONLINE 0 0 0 (resilvering)
errors: No known data errors
```
Resilvering complete; the pool has returned to the ONLINE state:
```sh
# zpool status
pool: mypool
state: ONLINE
scan: resilvered 781M in 0h0m with 0 errors on Mon Jun 2 14:52:38 2014
config:
NAME STATE READ WRITE CKSUM
mypool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0p3 ONLINE 0 0 0
ada2p3 ONLINE 0 0 0
errors: No known data errors
```
## Taking Devices Offline and Online
`zpool offline` takes a physical device offline — while offline, ZFS performs no reads or writes to the device. This operation does not apply to spare disks. If there is only one device, an error will be reported: `cannot offline nda1: no valid replicas`.
Create 2 virtual disks of 1G each under **/tmp** for demonstration:
```sh
# truncate -s 1G /tmp/vdisk1
# truncate -s 1G /tmp/vdisk2
```
Create the mirror pool **mypool**, then view its status:
```sh
# zpool create mypool mirror /tmp/vdisk1 /tmp/vdisk2
# zpool status mypool
pool: mypool
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
mypool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
/tmp/vdisk1 ONLINE 0 0 0
/tmp/vdisk2 ONLINE 0 0 0
errors: No known data errors
```
Take the device **/tmp/vdisk1** offline from the storage pool **mypool**:
```sh
# zpool offline mypool /tmp/vdisk1
```
View the pool status to confirm the device is offline:
```sh
# zpool status mypool
pool: mypool
state: DEGRADED
status: One or more devices has been taken offline by the administrator.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Online the device using 'zpool online' or replace the device with
'zpool replace'.
config:
NAME STATE READ WRITE CKSUM
mypool DEGRADED 0 0 0
mirror-0 DEGRADED 0 0 0
/tmp/vdisk1 OFFLINE 0 0 0
/tmp/vdisk2 ONLINE 0 0 0
errors: No known data errors
```
* The `-t` option makes the offline state temporary — the device will revert to its previous state after a system reboot.
* The `-f` option forcibly marks the device as FAULTED rather than offline, and the faulted state persists across import/export (unless `-t` is also used).
`zpool online` brings an offline device back online:
```sh
# zpool online mypool /tmp/vdisk1
```
After bringing the device online, if there is unwritten data from when the device was offline, ZFS automatically triggers a resilver to synchronize the data. For devices in a mirror or RAID-Z, new space only becomes available to the pool after all devices have been expanded.
If a device is only temporarily disconnected, it can be brought back online directly without performing a full replacement procedure.
## Data Integrity Checking (scrub)
It is recommended to regularly scrub pools, preferably at least once per month.
The `scrub` operation consumes disk resources, so it can degrade performance while running. Avoid scheduling `scrub` operations during periods of high load, or use `vfs.zfs.scrub_delay` to adjust the relative priority of `scrub` to avoid impacting other workloads.
Perform a data integrity check on the pool:
```sh
# zpool scrub zroot
```
View the scrub progress:
```sh
# zpool status
pool: zroot
state: ONLINE
scan: scrub in progress since Tue May 12 13:38:00 2026
889M / 889M scanned, 889M / 889M issued at 98.8M/s
0B repaired, 100.00% done, 00:00:00 to go
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
nda2p1 ONLINE 0 0 0
nda1p1 ONLINE 0 0 0
errors: No known data errors
```
If you need to cancel a scrub operation, run `zpool scrub -s pool_name`.
## Disk Initialization
Before new disks are added to a storage pool, a disk initialization operation can be performed first.
`zpool initialize` writes data (zeros by default) to all unallocated regions on the disk, enabling ZFS to detect potential bad blocks in advance — if an unrecoverable read or write error is encountered during initialization, the disk immediately reports a failure, thereby avoiding the risk of writing data to bad blocks. Only data devices and log devices can be initialized.
Initialize all devices in the pool:
```sh
# zpool initialize mypool
```
Initialize a single disk in the **mypool** pool:
```sh
# zpool initialize mypool nda1
```
Initialization progress can be viewed with `zpool status`:
```sh
# zpool status mypool
pool: mypool
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
mypool ONLINE 0 0 0
nda1 ONLINE 0 0 0 (initializing)
errors: No known data errors
```
The `-w` option makes the command wait until initialization completes before returning:
```sh
# zpool initialize -w mypool
```
Initialization can be canceled (`-c`), suspended (`-s`), or the initialization state can be cleared (`-u`):
```sh
# zpool initialize -c mypool
```
After clearing the initialization state, the device can be initialized again from scratch. Unlike `zpool trim`, `initialize` is a one-time operation, while trim is used to notify SSDs to reclaim released blocks.
## Storage Pool Checkpoints
`zpool checkpoint` creates an instantaneous checkpoint of the entire storage pool, preserving the pool's complete consistent state. If a destructive operation (such as `zpool upgrade` or accidental dataset deletion) is performed afterwards, recovery is possible by exporting and then importing the pool with a rollback to the checkpoint.
Create a checkpoint:
```sh
# zpool checkpoint zroot
```
While a checkpoint exists, the following commands cannot be executed: `zpool remove`, `zpool attach`, `zpool detach`, `zpool split`, `zpool reguid`. Additionally, if the pool has insufficient free space, the checkpoint may break reserved space limits.
View the checkpoint status:
```sh
# zpool status zroot
pool: zroot
state: ONLINE
checkpoint: created Wed May 13 13:26:06 2026, consumes 288K # Note this line
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
nda0p3 ONLINE 0 0 0
errors: No known data errors
```
View the checkpoint space usage (CKPOINT column) via `zpool list`:
```sh
# zpool list zroot
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zroot 51.5G 890M 50.6G 288K - 0% 1% 1.00x ONLINE -
```
Discard the checkpoint to free space:
```sh
# zpool checkpoint -d zroot
```
The `-w` option makes the command wait until the checkpoint discard completes before returning:
```sh
# zpool checkpoint -d -w zroot
```
Rolling back to a checkpoint requires exporting the pool first, then importing with `--rewind-to-checkpoint`:
```sh
# zpool export pool_name
# zpool import --rewind-to-checkpoint pool_name
```
> **Warning**
>
> If zroot is used as the root partition and is currently in use, exporting is prohibited.
A typical use case for checkpoints is establishing a rollback point before pool-level upgrades or other high-risk administrative operations.
## Importing and Exporting Pools
Before migrating a pool to another system, the pool should be **exported** first. ZFS unmounts all datasets, marks each device as exported, and locks the devices to prevent use by other disk subsystems. Exported pools can be **imported** on other ZFS-supported computers, operating systems, and even different hardware architectures (with some caveats). If datasets have open files, `zpool export -f` can be used to force-export the pool, but exercise caution. Forced export forcibly unmounts datasets, which may cause unexpected behavior if applications still hold open files at the time of unmounting.
Export an unused pool:
```sh
# zpool export mypool
```
Importing a pool automatically mounts datasets. To prevent automatic mounting, use `zpool import -N`. `zpool import -o` can set temporary properties for the current import. `zpool import -R` can specify an alternate mount point for the imported pool, replacing the filesystem root. This can also be set using `zpool import -o altroot=`. If the pool was previously running on another system and was not properly exported, `zpool import -f` can force the import. `zpool import -a` imports all pools not in use by other systems.
List all available pools for import:
```sh
# zpool import
pool: healer
id: 16015915091752228231
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:
healer ONLINE
mirror-0 ONLINE
nda2 ONLINE
nda3 ONLINE
```
Import the pool using an alternate root directory:
```sh
# zpool import -o altroot=/mnt healer
```
Confirm the dataset is mounted under the alternate root directory:
```sh
# zfs list healer
NAME USED AVAIL REFER MOUNTPOINT
healer 736K 57.7G 100K /mnt/healer
```
## Self-Healing
### Self-Healing Principles
ZFS's self-healing capability stems from the combination of checksums and storage redundancy. Each data block is accompanied by a checksum, with the algorithm determined by the dataset property. Checksums are automatically and transparently verified on read, enabling ZFS to detect silent data corruption. If the data does not match the expected checksum, ZFS attempts to recover using available redundancy (mirror or RAID-Z). The checksum algorithm used for each block is stored in the block pointer (metadata) and is computed when the block is written, so changing the algorithm only affects writes that occur after the change.
Change the checksum algorithm for a dataset:
```sh
# zfs set checksum=sha256 pool_name/dataset
```
Supported checksum algorithms:
| Algorithm | Suitable for dedup and nopwrite? | Compatible with other ZFS implementations? | Description |
| --------- | -------------------------------- | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------ |
| on | See description | Yes | `fletcher4` for non-dedup datasets, `sha256` for dedup datasets |
| off | No | Yes | Checksums should not be disabled |
| fletcher2 | No | Yes | Deprecated, retained only for backward compatibility; should not be used on new datasets; use `fletcher4` instead |
| fletcher4 | No | Yes | Fletcher algorithm, also used for `zfs send` streams |
| sha256 | Yes | Yes | Default algorithm for dedup datasets |
| noparity | No | Yes | `noparity` should not be used; this option is retained only for backward compatibility |
| sha512 | Yes | Requires `org.illumos:sha512` to be enabled | Salted sha512; currently not supported for any filesystem on a boot pool |
| skein | Yes | Requires `org.illumos:skein` to be enabled | Salted skein; currently not supported for any filesystem on a boot pool |
| edonr | See description | Requires `org.illumos:edonr` to be enabled | Salted edonr; when used with dedup, `verify` is automatically enabled as a precaution; currently not supported for any filesystem on a boot pool |
| blake3 | Yes | Requires `org.openzfs:blake3` to be enabled | Salted blake3; currently not supported for any filesystem on a boot pool |
Although disabling checksums to improve CPU performance may seem tempting, the ZFS community generally considers this an extremely unwise practice. Checksums should not be disabled.
### Self-Healing Demonstration
The following example uses two md(4) memory disks (`md0`, `md1`) to build a mirror pool `healer`, safely demonstrating self-healing behavior — `md` pseudo-devices are created only in memory and automatically disappear after reboot, with absolutely no risk of damaging real disks.
Create two 128 MB memory disks:
```sh
# mdconfig -a -t swap -s 128m
md0
# mdconfig -a -t swap -s 128m
md1
```
Create a mirror pool using memory disks:
```sh
# zpool create healer mirror /dev/md0 /dev/md1
```
View the status of the newly created mirror pool:
```sh
# zpool status healer
pool: healer
state: ONLINE
config:
NAME STATE READ WRITE CKSUM
healer ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
md0 ONLINE 0 0 0
md1 ONLINE 0 0 0
errors: No known data errors
```
View the pool's capacity and allocation:
```sh
# zpool list healer
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
healer 112M 372K 112M - - 23% 0% 1.00x ONLINE -
```
Copy some **non-critical data** to the `healer` storage pool to guard against data corruption due to errors, then generate a checksum of the pool for later comparison.
This example uses the **Philosophische-untersuchungen.pdf** file as a sample.
```sh
# cp /home/ykla/Philosophische-untersuchungen.pdf /healer/
```
Confirm the file has been written to the pool:
```sh
# zfs list healer
NAME USED AVAIL REFER MOUNTPOINT
healer 28.0M 28.0M 27.6M /healer
```
Generate a SHA-256 checksum of the file for later comparison:
```sh
# sha256 /healer/Philosophische-untersuchungen.pdf > checksum.txt
```
View the checksum file contents:
```sh
# cat checksum.txt
SHA256 (/healer/Philosophische-untersuchungen.pdf) = 7fb17d5a0554bad9eef45803f3b24a50e0aadcaccee122037f3dfd54ef586436
```
Writing random data to one disk in the mirror can simulate data corruption. To prevent ZFS from automatically repairing data after detecting corruption, export the pool first, then re-import it after the corruption is done.
> **Safety Note**
>
> This example uses `md` memory disks — writing to `/dev/md1` only affects the in-memory pseudo-device and **poses absolutely no threat to physical disks**. `md` devices completely disappear after a system reboot or executing `mdconfig -d`. The entire demonstration is completely independent of real storage hardware. If simulating in other environments, always use `md` disks or file vdevs (created with `truncate -s`); **never** perform the following operations on real pool member disks.
First, export the pool to simulate data corruption in an offline state:
```sh
# zpool export healer
```
Write random data to one memory disk in the mirror to simulate disk corruption:
```sh
# dd if=/dev/random of=/dev/md1 bs=1m count=20
20+0 records in
20+0 records out
20971520 bytes transferred in 0.089384 secs (234622722 bytes/sec)
```
Re-import the pool; ZFS will attempt to identify and repair detected data errors:
```sh
# zpool import healer
```
The pool status shows that the md1 device experienced checksum errors, but the data read by applications is correct — ZFS served the data from the healthy md0 mirror disk:
```sh
# zpool status healer
pool: healer
state: ONLINE
status: One or more devices has experienced an unrecoverable error. An
attempt was made to correct the error. Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
using 'zpool clear' or replace the device with 'zpool replace'.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
config:
NAME STATE READ WRITE CKSUM
healer ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
md0 ONLINE 0 0 0
md1 ONLINE 0 0 1 # Note here
errors: No known data errors
```
ZFS detected the error and repaired it using redundant data from the unaffected **md0** mirror disk. By comparing with the original checksum, it can be confirmed whether the pool data is consistent.
```sh
# sha256 /healer/Philosophische-untersuchungen.pdf >> checksum.txt
# cat checksum.txt
SHA256 (/healer/Philosophische-untersuchungen.pdf) = 7fb17d5a0554bad9eef45803f3b24a50e0aadcaccee122037f3dfd54ef586436
SHA256 (/healer/Philosophische-untersuchungen.pdf) = 7fb17d5a0554bad9eef45803f3b24a50e0aadcaccee122037f3dfd54ef586436
```
Comparing the checksums generated before and after the intentional data corruption, the data remains consistent. This demonstrates that ZFS can automatically detect and repair errors even when checksums are inconsistent.
Note that this capability only holds when the pool has sufficient redundancy — a pool composed of a single device has no self-healing capability. This is precisely why checksums are indispensable in ZFS and should never be disabled for any reason.
Through this mechanism, ZFS can detect and repair these issues without relying on traditional filesystem consistency check programs (fsck), while maintaining pool availability when problems occur.
Use `zpool clear` to clear error messages in the pool status.
```sh
# zpool clear healer
```
After clearing the errors, confirm the pool has returned to a fully normal state with all error counts reset to zero:
```sh
# zpool status healer
pool: healer
state: ONLINE
scan: scrub repaired 4.33M in 00:00:00 with 0 errors on Thu Apr 16 11:59:56 2026
config:
NAME STATE READ WRITE CKSUM
healer ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
md0 ONLINE 0 0 0
md1 ONLINE 0 0 0
errors: No known data errors
```
At this point, the pool has returned to a fully normal operating state, with all error counts reset to zero.
After the demonstration, use the following commands to reclaim resources:
> **Warning**
>
> The following commands will permanently destroy the ZFS storage pool `healer` and delete the `md` memory disk devices; all data in the pool will be unrecoverable. Confirm the target is correct before executing.
```sh
# zpool destroy healer
# mdconfig -d -u 0
# mdconfig -d -u 1
```
## Displaying Recorded Pool History
ZFS records all commands that change pool state, including creating datasets, changing properties, or replacing disks. Viewing the pool's creation history helps trace administrative operations and confirm what actions a specific user performed and when. The history is not stored in a log file but is part of the pool structure. The command to view the history of pool `zroot` is `zpool history`:
```sh
# zpool history zroot
History for 'zroot':
2026-04-16.19:46:03 zpool create -o altroot=/mnt -O compression=on -O atime=off -m none -f zroot nda0p3
2026-04-16.19:46:03 zfs create -o mountpoint=none zroot/ROOT
2026-04-16.19:46:03 zfs create -o mountpoint=/ zroot/ROOT/default
...output omitted...
```
The output shows how `zpool` and `zfs` commands record pool-state-changing operations with timestamps. Commands like `zfs list` are not included. If no pool name is specified, ZFS displays the history of all pools.
If the `-i` or `-l` option is specified, `zpool history` can display more information. `-i` shows user-initiated events as well as internally recorded ZFS events.
```sh
# zpool history -i zroot
History for 'zroot':
2026-04-16.19:51:01 [txg:75] permission update zroot/home/ykla (1668) ud$1001 mount
2026-04-16.19:51:01 [txg:75] permission update zroot/home/ykla (1668) ud$1001 snapshot
2026-04-16.19:51:01 zfs allow ykla create,destroy,mount,snapshot zroot/home/ykla
2026-04-16.19:51:01 [txg:78] inherit zroot/home/ykla (1668) mountpoint=/home
2026-04-16.19:51:01 zfs inherit mountpoint zroot/home/ykla
1970-01-01.08:00:04 [txg:82] open pool version 5000; software version zfs-2.4.99-468-g3ee08abd2; uts 16.0-CURRENT 1600015 amd64
...output omitted...
```
Adding `-l` shows more details. The long-format history displays the user who executed the command and the hostname where the command occurred.
```sh
# zpool history -l zroot
History for 'zroot':
2026-04-16.19:46:34 zpool set cachefile=/mnt/boot/zfs/zpool.cache zroot [user 0 (root) on ykla:freebsd]
2026-04-16.19:46:34 zfs set canmount=noauto zroot/ROOT/default [user 0 (root) on ykla:freebsd]
2026-04-16.19:50:30 zfs create -u zroot/home/ykla [user 0 (root) on ykla:freebsd]
2026-04-16.19:50:30 zfs set mountpoint=legacy zroot/home/ykla [user 0 (root) on ykla:freebsd]
...output omitted...
```
The output shows that the `root` user created a single-disk pool `zroot` using **/dev/nda0p3**, and subsequently created the `ROOT` and `ROOT/default` datasets. The hostname `freebsd` also appears in the command records after pool creation (see `ykla:freebsd` in the `-l` output). Hostname information is particularly important when a pool is exported from one system and imported into another — administrators can use the hostname to distinguish commands executed on different systems.
Combining both options, `zpool history` can provide the most detailed information about a pool. Pool history records can be used for tracking operations and debugging.
## Performance Monitoring
ZFS has a built-in monitoring system that can display pool I/O statistics in real time, including used and free space, read and write operations per second, and I/O bandwidth.
By default, ZFS monitors and displays the status of all pools in the system. To monitor only a specific pool, specify its name. Here is a basic example:
```sh
# zpool iostat
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
healer 736K 59.5G 0 1 5.57K 9.53K
zroot 889M 50.6G 0 2 13.0K 11.0K
---------- ----- ----- ----- ----- ----- -----
```
To continuously view I/O activity, specify a numeric argument at the end of the command representing the update interval in seconds. At each interval, the system outputs a set of statistics. Press **Ctrl** + **C** to stop continuous monitoring. To also specify the number of statistics displays, add a second number on the command line.
Using `-v` displays more detailed I/O statistics, with one line per device in the pool. By examining the read and write operations of each device, you can determine whether a particular device is slowing down pool performance. Here is an example of a mirror pool with two devices:
```sh
# zpool iostat -v
capacity operations bandwidth
pool alloc free read write read write
---------- ----- ----- ----- ----- ----- -----
healer 736K 59.5G 0 1 5.03K 8.61K
mirror-0 736K 59.5G 0 1 5.03K 8.61K
nda2 - - 0 0 2.69K 4.30K
nda3 - - 0 0 2.34K 4.30K
---------- ----- ----- ----- ----- ----- -----
zroot 889M 50.6G 0 2 12.7K 11.1K
nda0p3 889M 50.6G 0 2 12.7K 11.1K
---------- ----- ----- ----- ----- ----- -----
```
## Splitting Storage Pools
ZFS can split a pool containing one or more mirror vdevs into two pools. Unless otherwise specified, ZFS detaches the last member from each mirror and creates a new pool containing identical data. Before executing, it is recommended to first use the `-n` option for a dry run — it displays the detailed process of the requested operation without actually executing it. This helps confirm the operation will proceed as expected.
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter:
```
GET https://book.bsdcn.org/ask/flat/chapter-28-the-zfs-file-system/di-28.3-jie-zfs-cun-chu-chi-guan-li.md?ask=
```
The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.