Overview of InfiniBand Devices

InfiniBand (IB) is a new I/O technology based on switched fabrics. It provides a high bandwidth, low latency interconnect for attaching I/O devices to hosts and for host-to-host communication. IB devices are managed by the Solaris IB nexus driver.

Support for the following devices is provided in Oracle Solaris 11:

• IP over IB (IPoIB) devices – Enables the ability to transport the IP packets over IB connections. This feature is implemented by the ibp(7D)) driver

• Socket Direct Protocol (SDP) – Provides support for sockets over IB

• Reliable Datagram Service version 3 (RDSv3) and Reliable Datagram Service (RDS)

• NFS over Remote Direct Memory Access (NFSoRDMA) - Provides NFS services over IB using RDMA

• iSCSI Extension for RDMA (iSER) – Provides the RDMA data transfer capability to the iSCSI protocol

• User Direct Access Programming Language (uDAPL)

• Open Fabric User Verb (OFUV)

• Ethernet over IB (EoIB)

The IB nexus driver queries the Solaris IB Device Manager (IBDM) for services, referred in this guide as communication services, to enumerate the IB Port, HCA_SVC, and IB VPPA devices.

The IB partition link represents a new part class of data link and this object is managed by using the new dladm subcommands. An IB partition link can be created on top of an IB physical link, one per each P_Key on the port. The partition links are used for data transfers.

The Port devices bind a communication service to a given port# of a Host Channel Adapter (HCA). The VPPA devices bind a communication service to a port#, P_key# combination instead. The HCA_SVC devices bind a communication service to a given HCA. Note that the Port devices and the HCA_SVC devices always use a P_key (partition key) whose value is zero. The Port, HCA_SVC, and VPPA devices are children of the HCA and are enumerated through the ib.conf file. For more information, see ib(7D).

The Input Output Controller (IOC) devices are children of the IB nexus driver and are part of an I/O unit. The pseudo devices are also children of the IB nexus driver and refer to all other devices that provide their own configuration files to enumerate. For more information, see ib(4).

The possible IB device tree path name(s) are listed in the following table.

Note that the IB HCA_SVC devices have zero as the port# and the P_key.

The IB components in the preceding table are described as follows:

services

Is a communication service. For example, ipib is the communication service used by the ibd kernel client driver.

P_key

Is the partition link key value being used.

port

Is the port number.

unit-address

Refers to IB kernel client driver's property by this name specified in its driver.conf file. For more information, see driver.conf(4).

For information about using IB diagnostic commands and utilities, see Monitoring and Troubleshooting IB Devices.

InfiniBand Software Packages

The IB related software packages are as follows:

• driver/infiniband/connectx – Mellanox ConnectX Family InfiniBand HCA and 10GbE NIC drivers

• system/io/infiniband/ethernet-over-ib – InfiniBand device driver implementing Ethernet over InfiniBand

• system/io/infiniband/ib-device-mgt-agent – InfiniBand Device Manager Agent

• system/io/infiniband/ib-sockets-direct – InfiniBand layered Sockets Direct Protocol

• system/io/infiniband/ip-over-ib – Network device driver supporting the IP over InfiniBand (IPoIB) protocol

• system/io/infiniband/open-fabrics – Open Fabrics kernel components

• system/io/infiniband/reliable-datagram-sockets-v3 – Reliable Datagram Sockets (RDSv3)

• system/io/infiniband/reliable-datagram-sockets – Reliable Datagram Sockets

• system/io/infiniband/rpc-over-rdma – InfiniBand RPC over RDMA Driver

• system/io/infiniband/udapl – UDAPL library and commands

• system/io/infiniband – InfiniBand Framework