Recommended ZFS Storage Pool Practices

ZFS uses a pool storage model where storage devices are aggregated into a storage pool. File systems within the storage pool use all the storage in the pool.

The following sections provide recommended practices for creating, monitoring, and troubleshooting ZFS storage pools.

ZFS Storage Pool Creation Practices

• Specific root pool device and boot disk requirements

  • ZFS Root Pool Device Requirements

  • ZFS Root Pool Disk and Boot Administration

• General root pool creation practices

  • The root pool must be created as a mirrored configuration or as a single-disk configuration. Neither a RAID-Z nor a striped configuration is supported. You cannot add additional disks to create multiple mirrored top-level virtual devices by using the zpool add command, but you can expand a mirrored virtual device by using the zpool attach command.

  • The root pool cannot have a separate log device.

  • Pool properties can be set during an AI installation by using the pool_options keyword syntax, but the gzip compression algorithm is not supported on root pools.

  • Do not rename the root pool after it is created by an initial installation. Renaming the root pool might cause an unbootable system.

  • Do not create a root pool on a USB stick for a production system because root pool disks are critical for continuous operation, particularly in an enterprise environment. Consider using a system's internal disks for the root pool, or at least use, the same quality disks that you would use for your non-root data. In addition, a USB stick might not be large enough to support a dump volume size that is equivalent to at least 1/2 the size of physical memory.

  • Consider keeping root pool components separate from non-root pool data.

• Non-root pool creation practices – Create non-root pools with whole disks by using the d* identifier. Do not use the p* identifier.

  • ZFS works best without any additional volume management software.

  • For better performance, use individual disks or at least LUNs made up of just a few disks. By providing ZFS with more visibility into the LUN setup, ZFS is able to make better I/O scheduling decisions.

  • Mirrored storage pools – Consume more disk space but generally perform better with small random reads. For example:

    # zpool create tank mirror c1d0 c2d0 mirror c3d0 c4d0

    Mirrored storage pools are also more flexible in that you can detach, attach, and replace existing devices in the pool.

  • RAID-Z storage pools – Can be created with 3 parity strategies, where parity equals 1 (raidz), 2 (raidz2), or 3 (raidz3).

    • A RAID-Z configuration maximizes disk space and generally performs well when data is written and read in large chunks (128K or more). Create a single-parity RAIDZ (raidz) configuration at 3 disks (2+1).

    • A RAIDZ-2 configuration offers better data availability, and performs similarly to RAID-Z. RAIDZ-2 has significantly better mean time to data loss (MTTDL) than either RAID-Z or 2-way mirrors. Create a double-parity RAID-Z (raidz2) configuration at 6 disks (4+2).

    • A RAIDZ-3 configuration maximizes disk space and offers excellent availability because it can withstand 3 disk failures. Create a triple-parity RAID-Z (raidz3) configuration at 8 disks (5+3).

• Non-redundant pools – If you create a non-redundant pool, you will see a message similar to the following:

  # zpool create pond c8t2d0 c8t3d0
  'pond' successfully created, but with no redundancy; failure of one
  device will cause loss of the pool

  Creating a pool with no redundancy is not recommended because a device failure could mean that the data is unrecoverable. Consider creating a ZFS storage pool with redundancy. For example:

  # zpool create pond mirror c8t2d0 c8t3d0

ZFS Storage Pool Monitoring Practices

• Make sure that pool capacity is below 90% for best performance. Monitor pool and file system space to make sure that they are not full. Consider using ZFS quotas and reservations to make sure file system space does not exceed 90% pool capacity.

• Run zpool scrub on a regular basis to identify data integrity problems:

  • If you have consumer-quality drives, consider a weekly scrubbing schedule.

  • If you have datacenter-quality drives, consider a monthly scrubbing schedule.

  • You should also run a scrub prior to replacing devices to ensure that all devices are currently operational.

• Use zpool status on a weekly basis to monitor pool and pool device status. Also use fmdump or fmdump -eV to see if any device faults or errors have occurred.

ZFS Storage Pool Troubleshooting Practices

Troubleshooting pool problems in Oracle Solaris 11 is similar to diagnosing problems in the Oracle Solaris 10 release, but review the following new diagnostic descriptions and features:

• Failed devices – Review the zpool status -l output to identify the physical location of the failed device and replace it. For information about replacing a failed disk, see Replacing or Repairing a Damaged Device in Oracle Solaris 11.1 Administration: ZFS File Systems.

• Failed device notification – The smtp-notify service can be configured to send electronic mail notifications in response to various fault management events, such as when a hardware component has been diagnosed as faulty. For more information, see the notification parameters section of smf(5).

  By default, some notifications are set up automatically to be sent to the root user. If you add an alias for your user account as root in the /etc/aliases file, you will receive electronic mail notifications, similar to the following:

  -------- Original Message --------
  Subject: Fault Management Event: tardis:SMF-8000-YX
  Date: Wed, 21 Sep 2011 11:11:27 GMT
  From: No Access User <noaccess@tardis.drwho.COM>
  Reply-To: root@tardis.drwho.COM
  To: root@tardis.drwho.COM
  
  SUNW-MSG-ID: ZFS-8000-D3, TYPE: Fault, VER: 1, SEVERITY: Major
  EVENT-TIME: Wed Sep 21 11:11:27 GMT 2011
  PLATFORM: Sun-Fire-X4140, CSN: 0904QAD02C, HOSTNAME: tardis
  SOURCE: zfs-diagnosis, REV: 1.0
  EVENT-ID: d9e3469f-8d84-4a03-b8a3-d0beb178c017
  DESC: A ZFS device failed.  Refer to http://sun.com/msg/ZFS-8000-D3 
  for more information.
  AUTO-RESPONSE: No automated response will occur.
  IMPACT: Fault tolerance of the pool may be compromised.
  REC-ACTION: Run 'zpool status -x' and replace the bad device.

• Moving devices – Devices that are part of a ZFS storage pool contain a device ID if the device driver creates or fabricates device IDs. Like all file systems, ZFS has a very close relationship with its underlying devices so if you attempt to upgrade a system's firmware, move a pool device to a different controller, or change a device's cabling, you might consider exporting the pool first. If the device ID does not follow the device change and this can happen with non-Oracle hardware, then the pool and pool data might become unavailable. In general, Oracle's Sun hardware can recover if a device is changed under a live pool because our drivers fully support device IDs, but you might consider exporting the pool before making any hardware changes.

For a full description of troubleshooting pool problems, see Chapter 10, Oracle Solaris ZFS Troubleshooting and Pool Recovery, in Oracle Solaris 11.1 Administration: ZFS File Systems.