JavaScript is required to for searching.
Skip Navigation Links
Exit Print View
Oracle Solaris 11.1 Administration: Devices and File Systems     Oracle Solaris 11.1 Information Library
search filter icon
search icon

Document Information

Preface

1.  Managing Removable Media (Tasks)

2.  Writing CDs and DVDs (Tasks)

3.  Managing Devices (Tasks)

4.  Dynamically Configuring Devices (Tasks)

5.  Managing USB Devices (Tasks)

6.  Using InfiniBand Devices (Overview/Tasks)

Overview of InfiniBand Devices

InfiniBand Software Packages

Dynamically Reconfiguring IB Devices (Task Map)

Dynamically Reconfiguring IB Devices (cfgadm)

Unconfiguring IB Device Considerations

How to Display IB Device Information

How to Unconfigure an IB Port, HCA_SVC, or a VPPA Device

How to Configure a IB Port, HCA_SVC, or a VPPA Device

How to Unconfigure an IB Pseudo Device

How to Configure an IB Pseudo Device

How to Display Kernel IB Clients of an HCA

How to Dynamically Reconfigure an HCA With Active EoIB Devices

How to Reconfigure and Restore an EoIB Interface After Hot Removal

Configuring an IB HCA

How to Update the IB P_key Tables

How to Display IB Communication Services

How to Add a VPPA Communication Service

How to Remove an Existing IB Port, HCA_SVC, or a VPPA Communication Service

Using the uDAPL Application Interface With InfiniBand Devices

How to Enable uDAPL

Updating the DAT Static Registry

How to Update the DAT Static Registry

How to Register a Service Provider in the DAT Static Registry

How to Unregister a Service Provider from the DAT Static Registry

Administering IPoIB Devices (dladm)

How to Display Physical Data Link Information

How to Create IB Partition Links

How to Display IB Partition Link Information

How to Remove an IB Partition Link

Monitoring and Troubleshooting IB Devices

7.  Managing Disks (Overview)

8.  Managing Disk Use (Tasks)

9.  Administering Disks (Tasks)

10.  Setting Up Disks (Tasks)

11.  Configuring Storage Devices With COMSTAR (Tasks)

12.  Configuring and Managing the Oracle Solaris Internet Storage Name Service (iSNS)

13.  The format Utility (Reference)

14.  Managing File Systems (Overview)

15.  Creating and Mounting File Systems (Tasks)

16.  Configuring Additional Swap Space (Tasks)

17.  Copying Files and File Systems (Tasks)

18.  Managing Tape Drives (Tasks)

Index

Dynamically Reconfiguring IB Devices (cfgadm)

One can configure or unconfigure an IB device from a running system by using the cfgadm CLI only. This command also provides a way to display the IB fabric, manage communication services, and update P_key table databases. For more information, see cfgadm_ib(1M).

The cfgadm CLI manages dynamic reconfiguration, referred to in this guide as DR, of the entire IB fabric as seen by a host. The cfgadm operations are supported on all the IB devices, such as Port, VPPA, HCA_SVC, IOC, and pseudo devices.

The cfgadm command displays information about attachment points (Ap_Ids), which are locations in the system where DR operations can occur. For details on the Ap_Ids that cfgadm supports, see cfgadm_ib.1M. Note that all IB Ap_Ids are shown as connected.

The cfgadm command provides the following IB device status information.

Receptacle State
Description
connected/configured/ok
The device is connected and available. The devinfo node is present.
connected/unconfigured/unknown
The device is unavailable and no devinfo node or device driver exists for this device. Or, the device was never configured for use by ib nexus driver. The device might be known to the IB Device Manager.

All of the procedures in this section require administrative privileges that are not generally granted to user accounts. For more information, see How to Use Your Assigned Administrative Rights in Oracle Solaris 11.1 Administration: Security Services.

The following sections describe how to dynamically reconfigure (DR) IB devices with the cfgadm command. All of the sample IB device information in these sections has been truncated to focus on relevant information.

Unconfiguring IB Device Considerations

An actual dynamic reconfiguration (DR) of an HCA is beyond the scope of the IB cfgadm plugin. Although DR of an HCA can be achieved by using the plugin of the underlying bus. For example, a PCI based HCA can use the cfgadm_pci command. For more information, see cfgadm_pci(1M).


Note - RDSv3 does not support unconfiguring a HCA. If the system has a RDSv3 driver installed at the time of DR, unconfiguring the HCA fails as shown below.

# cfgadm -c unconfigure ib::rdsv3,0
This operation will suspend activity on the IB device
Continue (yes/no)? yes
cfgadm: Hardware specific failure: unconfigure operation failed ap_id: /devices/ib:fabric::rdsv3,0
 
# cfgadm -c unconfigure PCI-EM0
cfgadm: Component system is busy, try again: unconfigure failed 

Workaround:

On a production system, you must remove the RDSv3 driver before the HCA DR operation and reboot the system.

# rem_drv rdsv3
Device busy
Cannot unload module: rdsv3
Will be unloaded upon reboot.
 
# init 6

How to Display IB Device Information

You can use the prtconf command to display general information about IB devices. For example:

$ prtconf
.
.
.
    ib, instance #0
        rpcib, instance #0
        rdsib, instance #0
        daplt, instance #0
        rdsv3, instance #0
        sdpib, instance #0
        eibnx, instance #0
        sol_umad, instance #0
        sol_uverbs, instance #0
        iser, instance #0
.
.
.
    pci15b3,673c, instance #0
                 ibport, instance #0
                 ibport, instance #1 

In the above example, pci15b3,673c refers to an IB HCA.

Use the following steps to display specific IB device information.

  1. Become an administrator.
  2. Display IB fabric information.

    For example:

    # cfgadm -al  
    Ap_Id                       Type         Receptacle     Occupant     Condition
    ib                          IB-Fabric    connected      configured   ok
    hca:21280001A0A478          IB-HCA       connected      configured   ok
    ib::21280001A0A47A,0,ipib   IB-PORT      connected      configured   ok
    ib::21280001A0A479,0,ipib   IB-PORT      connected      configured   ok
    ib::1730000008070,0,hnfs    IB-HCA_SVC   connected      configured   ok
    ib::daplt,0                 IB-PSEUDO    connected      configured   ok
    ib::eibnx,0                 IB-PSEUDO    connected      configured   ok
    ib::iser,0                  IB-PSEUDO    connected      configured   ok
    ib::rdsib,0                 IB-PSEUDO    connected      configured   ok
    ib::rdsv3,0                 IB-PSEUDO    connected      configured   ok
    ib::rpcib,0                 IB-PSEUDO    connected      configured   ok
    ib::sdpib,0                 IB-PSEUDO    connected      configured   ok
    ib::sol_umad,0              IB-PSEUDO    connected      configured   ok
    ib::sol_uverbs,0            IB-PSEUDO    connected      configured   ok

    In the above example output, the components are described as follows:

    Ap_Id ib::21280001A0A47A,0,ipib

    Identifies an IB port device that is connected to port GUID and is bound to the ipib service.

    Ap_ID ib::sdpib,0

    Identifies a pseudo device.

    Ap_ID hca:21280001A0A478

    Identifies an HCA device.

    Ap_ID ib::1730000008070,0,hnfs

    Identifies an IB HCA_SVC device that is bound to the hnfs service.

    Ap_Id ib::ibgen,0

    Identifies a pseudo device.

  3. Display specific IB device information.

    For example, for an IB port device:

    # cfgadm -al -s "cols=ap_id:info" ib::21280001A0A47A,0,ipib
    Ap_Id                          Information
    ib::21280001A0A47A,0,ipib      ipib

    For example, for an IB HCA device:

    # cfgadm -al -s "cols=ap_id:info" hca::1730000008070
    Ap_Id                          Information
    hca::1730000008070             VID: 0x15b3, PID: 0x5a44, #ports: 0x2, 
    port1 GUID: 0x1730000008071, port2 GUID: 0x1730000008072

    The preceding output displays the number of ports and their GUIDs.

How to Unconfigure an IB Port, HCA_SVC, or a VPPA Device

Use the following steps if you want to remove an IB Port, HCA_SVC, or a VPPA device from the system.

The example below illustrates how to unconfigure an IB Port device, but the same procedure applies to VPPA and HCA_SVC devices as well.

  1. Become an administrator.
  2. Unconfigure virtual IB port devices.

    For example:

    # cfgadm -c unconfigure ib::1730000007F51,*0*,ipib
    Unconfigure the device: /devices/ib:fabric::1730000007F51,*0*,ipib
    This operation will suspend activity on the IB device
    Continue (yes/no)? Y
  3. Verify that the device is disconnected.

    For example:

    # cfgadm -a ib::1730000007F51,*0*,ipib
    Ap_Id                       Type    Receptacle Occupant     Condition
    ib::1730000007F51,*0*,ipib IB-VPPA  connected  unconfigured unknown

How to Configure a IB Port, HCA_SVC, or a VPPA Device

Use the following steps if you want to configure an IB Port, HCA_SVC, or a VPPA device on the system.

The example below illustrates how to configure a VPPA device, but similar steps can be used to configure an IB Port and HCA_SVC devices as well.

  1. Become an administrator.
  2. Configure the virtual IB port devices.

    For example:

    # cfgadm -c configure ib::1730000007F51,*0*,ipib
  3. Verify that the device is connected.

    For example:

    # cfgadm -a ib::1730000007F51,*0*,ipib   
    Ap_Id                       Type      Receptacle Occupant   Condition
    ib::1730000007F51,*0*,ipib  IB-VPPA   connected  configured ok

    Note - A cfgadm based configure or unconfigure operation of IB Port and HCA_SVC devices is similar to the preceding examples for an IB VPPA device.


How to Unconfigure an IB Pseudo Device

Use the following steps if you want to remove an IB pseudo device from the system.

  1. Become an administrator.
  2. Unconfigure the IB pseudo device.

    For example:

    # cfgadm -c unconfigure ib::ibgen,0
    Unconfigure the device: /devices/ib:fabric::ibgen,0
    This operation will suspend activity on the IB device
    Continue (yes/no)? Y
  3. Verify that the device is disconnected.
    # cfgadm -a ib::ibgen,0
    Ap_Id                  Type      Receptacle Occupant     Condition
    ib::ibgen,0            IB-PSEUDO connected  unconfigured unknown

How to Configure an IB Pseudo Device

Use the following steps to configure an IB pseudo device.

  1. Become an administrator.
  2. Configure the IB pseudo device.

    For example:

    # cfgadm -yc configure ib::ibgen,0
  3. Verify that the device is connected.

    For example:

    # cfgadm -a ib::ibgen,0
    Ap_Id                  Type       Receptacle Occupant   Condition
    ib::ibgen,0            IB-PSEUDO  connected  configured ok

How to Display Kernel IB Clients of an HCA

The following IB cfgadm plugin command can be invoked to list kernel IB clients using this HCA. Note that the last column would show a “yes” if a kernel IB client uses another HCA. IB Managers and kernel clients that do not use the HCA are shown with an Ap_Id of “-”.

How to Dynamically Reconfigure an HCA With Active EoIB Devices

When a dynamic reconfiguration (DR) operation is attempted on a system and a Connect-X family IB HCA is in use with active EoIB interfaces, (for example, EoIB datalinks are plumbed or VNICs are created over the datalinks on the HCA), the DR operation succeeds if either of the following conditions is true:

Under any other condition, a dynamic reconfiguration (DR) operation on an IB HCA with active EoIB interfaces fails with a cfgadm error message.

If the DR operation fails, you must unplumb the active EoIB interfaces and delete any VNICs on the datalink and retry the DR operation. In addition, in such a case, if a Connect-X family HCA is replaced in the same slot after the DR operation and configured again using the cfgadm or hotplug commands, you must replumb the EoIB datalinks that were unplumbed earlier and recreate any VNICs that were previously deleted.

Note that a functioning EoIB interface requires that the HCA port GUID be listed in the appropriate config file on the Sun Network QDR Infiniband Gateway Switch. So, after a hot-removal and before a re-insertion, you will need to update the new HCA port GUID information in the Gateway Switch config file. For information on how to update the configuration, see the Sun Network QDR Infiniband Gateway Switch Administration Guide.

  1. Become an administrator on the system with multiple IB HCAs.
  2. Attempt to unconfigure the attachment point associated with the EoIB datalink.
    # cfgadm -c unconfigure PCI-EM0
    cfgadm: Component system is busy, try again: unconfigure failed
  3. Determine the EoIB interface that is preventing the unconfigure operation.
    # dmesg | tail | grep 'failing HCA detach'
    Aug 23 12:37:20  eoib: [ID 530795 kern.warning] WARNING: eoib0 still in use, 
    failing HCA detach
  4. Determine if eoib0 is in use because an IP interface exists on it.
    # ipadm show-if eoib0
    ipadm: cannot get information for interface(s): No such interface
  5. If no IP interface over eoib0 exists, check to see if any VNIC is present that is causing eoib0 to be busy.
    # dladm show-vnic
        LINK                OVER         SPEED  MACADDRESS        MACADDRTYPE       VID
        evnic0              eoib0        10000  2:8:20:e5:56:99   random            0
  6. Delete the VNIC over eoib0.
    # dladm delete-vnic evnic0
  7. Retry the unconfigure operation.
    # cfgadm -c unconfigure PCI-EM0

How to Reconfigure and Restore an EoIB Interface After Hot Removal

  1. Become an administrator on the Sun Network QDR Gateway Switch.
  2. Edit the BXM config file on the Sun Network QDR Gateway Switch and replace all occurrences of the old-HCA port GUIDs with the port GUIDs of the new HCA replacing it.
  3. Restart BXM.
  4. Become an administrator on the Oracle Solaris system.
  5. Hot insert the new HCA on the Solaris system and use the cfgadm command to reconfigure the attachment point originally associated with the EoIB datalink.
  6. Confirm the restoration of the original EoIB datalink.
    # dladm show-link | grep eoib0
    eoib0               phys      1500   unknown       --
  7. If any VNICs had to be explicitly deleted during a previous unconfigure operation, re-create them now.
    # dladm create-vnic -l eoib0 evnic0
  8. Confirm the presence of the VNIC over eoib0.
    # dladm show-vnic
    LINK         OVER     SPEED  MACADDRESS        MACADDRTYPE       VID
    evnic0       eoib0    10000  2:8:20:e5:56:99   random              0
  9. If any IP addresses had to be explicitly deleted on eoib0 during a previous unconfigure operation, recreate them by using the ipadm command.

Configuring an IB HCA

Invoke the bus-specific cfgadm plugin to configure the HCA. The exact details are beyond the scope of this chapter.

How to Update the IB P_key Tables

If the P_key table information of an HCA ports changes, for example, additional P_keys are enabled or disabled, InfiniBand Transport Framework (IBTF) and IBDM need to be notified so that their internal P_key databases are updated. The cfgadm command helps update the P_key databases of IBTF and IBDM. For more information, see ibtl(7D) and ibdm(7D).

  1. Become an administrator.
  2. Update the P_key tables.

    For example:

    # cfgadm -x update_pkey_tbls -y ib

How to Display IB Communication Services

Use the following steps to display the communication services that are currently in use by the IBTF.

  1. Become an administrator.
  2. Display IB communication services.

    For example:

    # cfgadm -x list_services ib
    Port communication services:
                    srp
    VPPA communication services:
                    ibd
    HCA_SVC communication services:
                    hnfs

How to Add a VPPA Communication Service

Use the following steps to add a new VPPA communication service.

Similar steps can be used to add a new HCA_SVC or a port communication service.

  1. Become an administrator.
  2. Add a new VPPA communication service.

    For example:

    # cfgadm -o comm=vppa,service=new -x add_service ib
  3. Verify that the new service has been added.

    For example:

    # cfgadm -x list_services ib
    Port communication services:
                    srp
    VPPA communication services:
                    ibd
                    new
    HCA_SVC communication services:
                    nfs_service

How to Remove an Existing IB Port, HCA_SVC, or a VPPA Communication Service

Use the following steps to delete an existing IB Port, HCA_SVC, or a VPPA communication service.

  1. Become an administrator.
  2. Remove a VPPA communication service.

    For example:

    # cfgadm -o comm=vppa,service=new -x delete_service ib
  3. Verify that the communication service has been removed.

    For example:

    # cfgadm -x list_services ib
    Port communication services:
                    srp
    VPPA communication services:
                    ibd
    HCA_SVC communication services:
                    hnfs