Skip Navigation Links | |
Exit Print View | |
Oracle Solaris 11.1 Administration: Devices and File Systems Oracle Solaris 11.1 Information Library |
1. Managing Removable Media (Tasks)
2. Writing CDs and DVDs (Tasks)
4. Dynamically Configuring Devices (Tasks)
5. Managing USB Devices (Tasks)
6. Using InfiniBand Devices (Overview/Tasks)
Overview of InfiniBand Devices
Dynamically Reconfiguring IB Devices (Task Map)
Dynamically Reconfiguring IB Devices (cfgadm)
Unconfiguring IB Device Considerations
How to Display IB Device Information
How to Unconfigure an IB Port, HCA_SVC, or a VPPA Device
How to Configure a IB Port, HCA_SVC, or a VPPA Device
How to Unconfigure an IB Pseudo Device
How to Configure an IB Pseudo Device
How to Display Kernel IB Clients of an HCA
How to Dynamically Reconfigure an HCA With Active EoIB Devices
How to Reconfigure and Restore an EoIB Interface After Hot Removal
How to Update the IB P_key Tables
How to Display IB Communication Services
How to Add a VPPA Communication Service
How to Remove an Existing IB Port, HCA_SVC, or a VPPA Communication Service
Using the uDAPL Application Interface With InfiniBand Devices
Updating the DAT Static Registry
How to Update the DAT Static Registry
How to Register a Service Provider in the DAT Static Registry
How to Unregister a Service Provider from the DAT Static Registry
Administering IPoIB Devices (dladm)
How to Display Physical Data Link Information
How to Create IB Partition Links
How to Display IB Partition Link Information
How to Remove an IB Partition Link
Monitoring and Troubleshooting IB Devices
9. Administering Disks (Tasks)
11. Configuring Storage Devices With COMSTAR (Tasks)
12. Configuring and Managing the Oracle Solaris Internet Storage Name Service (iSNS)
13. The format Utility (Reference)
14. Managing File Systems (Overview)
15. Creating and Mounting File Systems (Tasks)
16. Configuring Additional Swap Space (Tasks)
17. Copying Files and File Systems (Tasks)
One can configure or unconfigure an IB device from a running system by using the cfgadm CLI only. This command also provides a way to display the IB fabric, manage communication services, and update P_key table databases. For more information, see cfgadm_ib(1M).
The cfgadm CLI manages dynamic reconfiguration, referred to in this guide as DR, of the entire IB fabric as seen by a host. The cfgadm operations are supported on all the IB devices, such as Port, VPPA, HCA_SVC, IOC, and pseudo devices.
The cfgadm command displays information about attachment points (Ap_Ids), which are locations in the system where DR operations can occur. For details on the Ap_Ids that cfgadm supports, see cfgadm_ib.1M. Note that all IB Ap_Ids are shown as connected.
The cfgadm command provides the following IB device status information.
|
All of the procedures in this section require administrative privileges that are not generally granted to user accounts. For more information, see How to Use Your Assigned Administrative Rights in Oracle Solaris 11.1 Administration: Security Services.
The following sections describe how to dynamically reconfigure (DR) IB devices with the cfgadm command. All of the sample IB device information in these sections has been truncated to focus on relevant information.
An actual dynamic reconfiguration (DR) of an HCA is beyond the scope of the IB cfgadm plugin. Although DR of an HCA can be achieved by using the plugin of the underlying bus. For example, a PCI based HCA can use the cfgadm_pci command. For more information, see cfgadm_pci(1M).
Note - RDSv3 does not support unconfiguring a HCA. If the system has a RDSv3 driver installed at the time of DR, unconfiguring the HCA fails as shown below.
# cfgadm -c unconfigure ib::rdsv3,0 This operation will suspend activity on the IB device Continue (yes/no)? yes cfgadm: Hardware specific failure: unconfigure operation failed ap_id: /devices/ib:fabric::rdsv3,0 # cfgadm -c unconfigure PCI-EM0 cfgadm: Component system is busy, try again: unconfigure failed
Workaround:
On a production system, you must remove the RDSv3 driver before the HCA DR operation and reboot the system.
# rem_drv rdsv3 Device busy Cannot unload module: rdsv3 Will be unloaded upon reboot. # init 6
You can use the prtconf command to display general information about IB devices. For example:
$ prtconf . . . ib, instance #0 rpcib, instance #0 rdsib, instance #0 daplt, instance #0 rdsv3, instance #0 sdpib, instance #0 eibnx, instance #0 sol_umad, instance #0 sol_uverbs, instance #0 iser, instance #0 . . . pci15b3,673c, instance #0 ibport, instance #0 ibport, instance #1
In the above example, pci15b3,673c refers to an IB HCA.
Use the following steps to display specific IB device information.
For example:
# cfgadm -al Ap_Id Type Receptacle Occupant Condition ib IB-Fabric connected configured ok hca:21280001A0A478 IB-HCA connected configured ok ib::21280001A0A47A,0,ipib IB-PORT connected configured ok ib::21280001A0A479,0,ipib IB-PORT connected configured ok ib::1730000008070,0,hnfs IB-HCA_SVC connected configured ok ib::daplt,0 IB-PSEUDO connected configured ok ib::eibnx,0 IB-PSEUDO connected configured ok ib::iser,0 IB-PSEUDO connected configured ok ib::rdsib,0 IB-PSEUDO connected configured ok ib::rdsv3,0 IB-PSEUDO connected configured ok ib::rpcib,0 IB-PSEUDO connected configured ok ib::sdpib,0 IB-PSEUDO connected configured ok ib::sol_umad,0 IB-PSEUDO connected configured ok ib::sol_uverbs,0 IB-PSEUDO connected configured ok
In the above example output, the components are described as follows:
Identifies an IB port device that is connected to port GUID and is bound to the ipib service.
Identifies a pseudo device.
Identifies an HCA device.
Identifies an IB HCA_SVC device that is bound to the hnfs service.
Identifies a pseudo device.
For example, for an IB port device:
# cfgadm -al -s "cols=ap_id:info" ib::21280001A0A47A,0,ipib Ap_Id Information ib::21280001A0A47A,0,ipib ipib
For example, for an IB HCA device:
# cfgadm -al -s "cols=ap_id:info" hca::1730000008070 Ap_Id Information hca::1730000008070 VID: 0x15b3, PID: 0x5a44, #ports: 0x2, port1 GUID: 0x1730000008071, port2 GUID: 0x1730000008072
The preceding output displays the number of ports and their GUIDs.
Use the following steps if you want to remove an IB Port, HCA_SVC, or a VPPA device from the system.
The example below illustrates how to unconfigure an IB Port device, but the same procedure applies to VPPA and HCA_SVC devices as well.
For example:
# cfgadm -c unconfigure ib::1730000007F51,*0*,ipib Unconfigure the device: /devices/ib:fabric::1730000007F51,*0*,ipib This operation will suspend activity on the IB device Continue (yes/no)? Y
For example:
# cfgadm -a ib::1730000007F51,*0*,ipib Ap_Id Type Receptacle Occupant Condition ib::1730000007F51,*0*,ipib IB-VPPA connected unconfigured unknown
Use the following steps if you want to configure an IB Port, HCA_SVC, or a VPPA device on the system.
The example below illustrates how to configure a VPPA device, but similar steps can be used to configure an IB Port and HCA_SVC devices as well.
For example:
# cfgadm -c configure ib::1730000007F51,*0*,ipib
For example:
# cfgadm -a ib::1730000007F51,*0*,ipib Ap_Id Type Receptacle Occupant Condition ib::1730000007F51,*0*,ipib IB-VPPA connected configured ok
Note - A cfgadm based configure or unconfigure operation of IB Port and HCA_SVC devices is similar to the preceding examples for an IB VPPA device.
Use the following steps if you want to remove an IB pseudo device from the system.
For example:
# cfgadm -c unconfigure ib::ibgen,0 Unconfigure the device: /devices/ib:fabric::ibgen,0 This operation will suspend activity on the IB device Continue (yes/no)? Y
# cfgadm -a ib::ibgen,0 Ap_Id Type Receptacle Occupant Condition ib::ibgen,0 IB-PSEUDO connected unconfigured unknown
Use the following steps to configure an IB pseudo device.
For example:
# cfgadm -yc configure ib::ibgen,0
For example:
# cfgadm -a ib::ibgen,0 Ap_Id Type Receptacle Occupant Condition ib::ibgen,0 IB-PSEUDO connected configured ok
The following IB cfgadm plugin command can be invoked to list kernel IB clients using this HCA. Note that the last column would show a “yes” if a kernel IB client uses another HCA. IB Managers and kernel clients that do not use the HCA are shown with an Ap_Id of “-”.
For example:
$ cfgadm -x list_clients hca:173000007F50 Ap_Id IB Client Alternate HCA ib::1730000007F51D0 ibgen no ib::1730000007F51D1 ibgen no ib::1730000007F51,0,ipib ibd no ib::ibgen,0 ibgen no - ibdm no - ibmf no - nfs/ib no
When a dynamic reconfiguration (DR) operation is attempted on a system and a Connect-X family IB HCA is in use with active EoIB interfaces, (for example, EoIB datalinks are plumbed or VNICs are created over the datalinks on the HCA), the DR operation succeeds if either of the following conditions is true:
This is the only Connect-X family HCA on the system
Multiple Connect-X HCAs exist on the system, but this adapter is the last one to be unconfigured
Under any other condition, a dynamic reconfiguration (DR) operation on an IB HCA with active EoIB interfaces fails with a cfgadm error message.
If the DR operation fails, you must unplumb the active EoIB interfaces and delete any VNICs on the datalink and retry the DR operation. In addition, in such a case, if a Connect-X family HCA is replaced in the same slot after the DR operation and configured again using the cfgadm or hotplug commands, you must replumb the EoIB datalinks that were unplumbed earlier and recreate any VNICs that were previously deleted.
Note that a functioning EoIB interface requires that the HCA port GUID be listed in the appropriate config file on the Sun Network QDR Infiniband Gateway Switch. So, after a hot-removal and before a re-insertion, you will need to update the new HCA port GUID information in the Gateway Switch config file. For information on how to update the configuration, see the Sun Network QDR Infiniband Gateway Switch Administration Guide.
# cfgadm -c unconfigure PCI-EM0 cfgadm: Component system is busy, try again: unconfigure failed
# dmesg | tail | grep 'failing HCA detach' Aug 23 12:37:20 eoib: [ID 530795 kern.warning] WARNING: eoib0 still in use, failing HCA detach
# ipadm show-if eoib0 ipadm: cannot get information for interface(s): No such interface
# dladm show-vnic LINK OVER SPEED MACADDRESS MACADDRTYPE VID evnic0 eoib0 10000 2:8:20:e5:56:99 random 0
# dladm delete-vnic evnic0
# cfgadm -c unconfigure PCI-EM0
# dladm show-link | grep eoib0 eoib0 phys 1500 unknown --
# dladm create-vnic -l eoib0 evnic0
# dladm show-vnic LINK OVER SPEED MACADDRESS MACADDRTYPE VID evnic0 eoib0 10000 2:8:20:e5:56:99 random 0
Invoke the bus-specific cfgadm plugin to configure the HCA. The exact details are beyond the scope of this chapter.
If the P_key table information of an HCA ports changes, for example, additional P_keys are enabled or disabled, InfiniBand Transport Framework (IBTF) and IBDM need to be notified so that their internal P_key databases are updated. The cfgadm command helps update the P_key databases of IBTF and IBDM. For more information, see ibtl(7D) and ibdm(7D).
Use the following steps to display the communication services that are currently in use by the IBTF.
For example:
# cfgadm -x list_services ib Port communication services: srp VPPA communication services: ibd HCA_SVC communication services: hnfs
Use the following steps to add a new VPPA communication service.
Similar steps can be used to add a new HCA_SVC or a port communication service.
For example:
# cfgadm -o comm=vppa,service=new -x add_service ib
For example:
# cfgadm -x list_services ib Port communication services: srp VPPA communication services: ibd new HCA_SVC communication services: nfs_service
Use the following steps to delete an existing IB Port, HCA_SVC, or a VPPA communication service.
For example:
# cfgadm -o comm=vppa,service=new -x delete_service ib
For example:
# cfgadm -x list_services ib Port communication services: srp VPPA communication services: ibd HCA_SVC communication services: hnfs