Dynamic Reconfiguration and Hot-Plugging

The following information is described in this section:

• Attachment Points

• Detaching PCI or PCIe Adapter Cards

• Attaching PCI or PCIe Adapter Cards

• PCIe Hot-Plugging With the (hotplug) Command

Hot-plugging is the ability to physically add, remove, or replace system components while the system is running. Dynamic reconfiguration refers to the ability to hot-plug system components. This term also refers to the general ability to move system resources (both hardware and software) around in the system or to disable them in some way without physically removing them from the system.

Generally, you can hot-plug the following bus types:

• USB

• Fibre Channel

• 1394

• ATA

• SCSI

• Infiniband

In addition, you can hot-plug the following devices with the cfgadm command on both SPARC and x86 platforms:

• USB devices

• SCSI or SAS devices

• PCI devices

• PCIe devices

• SATA devices

• InfiniBand devices

Features of the cfgadm command include the following:

• Displaying system component status

• Testing system components

• Changing component configurations

• Displaying configuration help messages

The benefit of using the cfgadm command to reconfigure systems components is that you can add, remove, or replace components while the system is running. An added benefit is that the cfgadm command guides you through the steps needed to add, remove, or replace system components.

For step-by-step instructions on hot-plugging components, see the following:

• SCSI Hot-Plugging With the cfgadm Command

• PCI or PCIe Hot-Plugging With the cfgadm Command

• SATA Hot-Plugging With the cfgadm Command

• cfgadm(1M)

As part of Oracle's high availability strategy, dynamic reconfiguration is expected to be used in conjunction with additional layered products, such as alternate pathing or fail over software. Both products provide fault tolerance in the event of a device failure.

Without any high availability software, you can replace a failed device by manually stopping the appropriate applications, unmounting noncritical file systems, and then proceeding with the add or remove operations.

Attachment Points

The cfgadm command displays information about attachment points, which are locations in the system where dynamic reconfiguration operations can occur.

An attachment point consists of the following:

• An occupant, which represents a hardware component that can be configured into the system

• A receptacle, which is the location that accepts the occupant

Attachment points are represented by logical and physical attachment point IDs (Ap_Ids). The physical Ap_Id is the physical path name of the attachment point. The logical Ap_Id is a user-friendly alternative for the physical Ap_Id. For more information on Ap_Ids, refer to cfgadm(1M).

The logical Ap_Id for a SCSI Host Bus Adapter (HBA), or SCSI controller, is usually represented by the controller number, such as c0.

In cases where no controller number has been assigned to a SCSI HBA, then an internally generated unique identifier is provided. An example of a unique identifier for a SCSI controller is the following:

fas1:scsi

The logical Ap_Id for a SCSI device usually has this format:

HBA-logical-apid::device-identifier

In the following example, c0 is the logical Ap_Id for the SCSI HBA:

c0::dsk/c0t3d0

The device identifier is typically derived from the logical device name for the device in the /dev directory. For example, a tape device with logical device name, /dev/rmt/1, has the following logical Ap_Id:

c0::rmt/1

If a logical Ap_Id of a SCSI device cannot be derived from the logical name in the /dev directory, then an internally generated unique identifier is provided. An example of an identifier for the /dev/rmt/1 tape device is the following:

c0::st4

For more information on SCSI Ap_Ids, refer to cfgadm_scsi(1M).

The cfgadm command represents all resources and dynamic reconfiguration operations in terms of a common set of states (such as configured and unconfigured) and operations (such as connect, configure, unconfigure, and so on). For more information on these common states and operations, see cfgadm(1M).

The following table shows the receptacle and occupant states for the SCSI HBA attachment points.

The following table shows the receptacle and occupant states for SCSI device attachment points.

The state of SCSI attachment points is unknown unless special hardware indicates otherwise. For instructions on displaying SCSI component information, see How to Display Information About SCSI Devices.

Detaching PCI or PCIe Adapter Cards

A PCI adapter card that is hosting nonvital system resources can be removed if the device driver supports hot-plugging. A PCI adapter card is not detachable if it is a vital system resource.

For a PCI adapter card to be detachable, the following conditions must be met:

• The device driver must support hot-plugging.

• Critical resources must be accessible through an alternate pathway.

For example, if a system has only one Ethernet card installed in it, the Ethernet card cannot be detached without losing the network connection. This detachment requires additional layered software support to keep the network connection active.

Attaching PCI or PCIe Adapter Cards

A PCI adapter card can be added to the system as long as the following conditions are met:

• There are slots available.

• The device driver supports hot-plugging for this adapter card.

For step-by-step instructions on adding or removing a PCI adapter card, see PCI or PCIe Hot-Plugging With the cfgadm Command.

PCIe Hot-Plugging With the (hotplug) Command

You can use the hotplug command to manage hot pluggable connections, where a connection can be a connector or port, on PCI Express (PCIe) and PCI SHPC devices only. A hotplug connector is a physical representation in the system where a component is inserted or removed. A hotplug port is a logical representation in the system device tree where the connection of a device to the system is managed.

You can use the hotplug features to online or offline a device, even an on-board device, without physically adding or removing the device from the system.

You must enable the hotplug service to manage devices with the hotplug command.

# svcadm enable svc:/system/hotplug:default

The following examples show how to use the hotplug command:

Display all the PCI/PCIe hot-pluggable connectors/ports (virtual and physical) in the system as follows:

# hotplug list -lv

Configure an Ethernet card in a PCIe slot. For example:

# hotplug enable /pci0,0 pcie0

Unconfigure an Ethernet card in a PCIe slot. For example:

# hotplug disable /pci0,0 pcie0

Offline a PCI device node, which means detach the device driver for that node. For example:

# hotplug offline /pci0,0/pci1 pci.0,2

Online a PCI device node, which means attach the driver for that node. For example:

# hotplug online /pci0,0/pci1 pci.0,2

Install dependent ports of an IOV physical function. For example:

# hotplug install /pci@400/pci@1/pci@0/pci@4 pci.0,1

Then, display the resulting IOV virtual functions that were probed. For example:

# hotplug list -v /pci@400/pci@1/pci@0/pci@4 pci.0,1
<pci.0,1>  (ONLINE)
     { IOV physical function }
     { IOV virtual function 'pci.0,81' }
     { IOV virtual function 'pci.0,83' }
     { IOV virtual function 'pci.0,85' }
     { IOV virtual function 'pci.0,87' }
<pci.0,81>  (OFFLINE)
ethernet@0,81
<pci.0,83>  (OFFLINE)
ethernet@0,83
<pci.0,85>  (OFFLINE)
ethernet@0,85
<pci.0,87>  (OFFLINE)
ethernet@0,87

Uninstall the dependent ports of an IOV physical function. For example:

# hotplug uninstall /pci@400/pci@1/pci@0/pci@4 pci.0,0

This operation fails if a dependent IOV virtual function is busy. For example:

# hotplug uninstall /pci@400/pci@1/pci@0/pci@4 pci.0,0
ERROR: devices or resources are busy.
ethernet@0,81:
    { Network interface igbvf1 }
    { igbvf1: hosts IP addresses: 10.0.0.1 }
    { Plumbed IP Address }

Troubleshooting PCI Hot Plug Operations (hotplug)

You might see the following maintenance states for an attached device in a hot-pluggable port.

/pci@0,0/pci10de,5d@e <pci.a,1> (MAINTENANCE)
.
.
./pci@0,0/pci108e,534a@d <pci.0,0> (MAINTENANCE-SUSPENDED)

These messages indicate that a fault event or a maintenance operation occurred. The MAINTENANCE states mean that a device is in use, but it is not fully operational. The MAINTENANCE-SUSPENDED state means that the device is live suspended, due to a maintenance operation. For example, reconfiguring the device hardware.

The following service must be running to use the hotplug command.

svc:/system/hotplug:default

Otherwise, you will see the following message:

ERROR: hotplug service is not available.

The following error message is displayed on systems that do not have any supported I/O buses:

ERROR: there are no connections to display.
       (See hotplug(1m) for more information.)

The above message could mean that the system might have other hot-pluggable I/O devices, but you need to use the cfgadm command rather than the hotplug command to manage these devices.