Synopsis

The ibp driver implements the IETF IP over Infiniband protocol and provides IPoIB service for all IBA ports present in the system. For more information about managing the data-links created by the ibp driver, see dladm(1M) manual page.

The ibp driver is a multi-threaded, loadable, clonable, STREAMS hardware driver supporting the connectionless Data Link Provider Interface, dlpi(7P)).

The ibp driver provides basic support for both the IBA Unreliable Datagram Queue Pair hardware and the IBA Reliable Connected Queue Pair hardware. Functions include QP initialization, frame transmit and receive, multicast and promiscuous mode support, and statistics reporting.

By default, Connected Mode will be used by the each IB link. This behavior can be modified by changing the linkmode property of the data link. See the EXAMPLES section of the dladm(1M) manual page for information .

Because ibp over connected mode attempts to use a large MTU (65520 bytes), applications should adapt to the large MTU to get better performance, for example, adopting a large TCP window size.

Use the cloning, character-special device /dev/ibp to access all ibp devices installed within the system.

The ibp driver is dependent on GLD, a loadable kernel module that provides the ibp driver with the DLPI and STREAMS functionality required of a LAN driver. Except as noted in the Application Programming Interface section of this man page, see gld(7D) for more details on the primitives supported by the driver. The GLD module is located at /kernel/misc/sparcv9/gld on 64 bit systems and at /kernel/misc/gld on 32 bit systems.

The ibp driver expects certain configuration of the IBA fabric prior to operation (which also implies the SM must be active and managing the fabric). Specifically, the IBA multicast group representing the IPv4 limited broadcast address 255.255.255.255 (also defined as broadcast-GID in IETF documents) should be created prior to initializing the device. IBA properties (including mtu, qkey and sl) of this group is used by the driver to create any other IBA multicast group as instructed by higher level (IP) software. The driver probes for the existance of this broadcast-GID during attach(9E).

APPLICATION PROGRAMMING INTERFACE (DLPI)

The values returned by the driver in the DL_INFO_ACK primitive in response to your DL_INFO_REQ are:

Maximum SDU is the MTU associated with the broadcast-GID group, less the 4 byte IPoIB header for UD mode and 65520 for CM mode.
Minimum SDU is 0.
dlsap address length is 22.
MAC type is DL_IB.
The sap length value is -2, meaning the physical address component is followed immediately by a 2-byte sap component within the DLSAP address.
Broadcast address value is the MAC address consisting of the 4 bytes of QPN 00:FF:FF:FF prepended to the IBA multicast address of the broadcast-GID.
Due to the nature of link address definition for IPoIB, the DL_SET_PHYS_ADDR_REQ DLPI primitive is not supported.
In the transmit case for streams that have been put in raw mode via the DLIOCRAW ioctl, the DLPI application must prepend the 20 byte IPoIB destination address to the data it wants to transmit over-the-wire. In the receive case, applications receive the IP/ARP datagram along with the IETF defined 4 byte header.

WARNING

This section describes warning messages that might be generated by the driver. Please note that while the format of these messages can be modified in future versions, the same general information is provided.

While joining IBA multicast groups corresponding to IP multicast groups as part of multicast promiscuous operations as required by IP multicast routers, or as part of running snoop(1M), it is possible that joins to some multicast groups can fail due to inherent resource constraints in the IBA components. In such cases, warning message similar to the following appear in the system log, indicating the interface on which the failure occurred:

NOTICE: ibp: Could not get list of IBA multicast groups
NOTICE: ibp: IBA promiscuous mode missed multicast group
NOTICE: ibp: IBA promiscuous mode missed new multicast gid

Additionally, if the IBA link transitions to an unavailable state (that is, the IBA link state becomes Down, Initialize or Armed) and then becomes active again, the driver tries to rejoin previously joined groups if required. Failure to rejoin multicast groups triggers messages such as:

NOTICE: ibp: Failure on port up to rejoin multicast gid

Further, as described above, if the broadcast-GID is not found or could not be created, or the associated MTU is higher than what the HCA port can support, the following messages are printed to the system log:

NOTICE: ibp: IPoIB broadcast group absent
NOTICE: ibp: IPoIB broadcast group MTU 4096 greater than port's
maximum MTU 2048

In all cases of these reported problems when running ifconfig(1M), it should be checked that IBA cabling is intact, an SM is running on the fabric, and the broadcast-GID with appropriate properties has been created in the IBA partition.

The MTU of Reliable Connected mode can be larger than the MTU of Unreliable Datagram mode.

When Reliable Connected mode is enabled, ibp still uses Unreliable Datagram mode to transmit and receive multicast packets.

If only one side has enabled Reliable Connected mode, communication falls back to datagram mode. The connected mode instance uses Path MTU discovery to automatically adjust the MTU of a unicast packet if an MTU difference exists. Before Path MTU discovery reduces the MTU for a specific destination, several packets whose size exceeds the MTU of Unreliable Datagram mode is dropped.

Examples

Example 1 Modifying the linkmode Property

# Below example modify the 'linkmode' to ud

# dladm show-linkprop pffff.ibp0 | grep linkmode
LINK        PROPERTY   PERM VALUE  DEFAULT    POSSIBLE
pffff.ibp0  linkmode    rw   cm       cm        cm,ud 

# dladm set-linkprop -p linkmode=ud pffff.ibp0

# dladm show-linkprop pffff.ibp0 | grep linkmode
LINK        PROPERTY   PERM VALUE  DEFAULT    POSSIBLE
pffff.ibp0  linkmode    rw   ud      cm        cm,ud 
#

Files

/dev/ibp*: Special character device
/kernel/drv/ib.conf: Configuration file to start IPoIB service
/kernel/drv/sparcv9/ibp: 64–bit SPARC device driver
/kernel/drv/amd64/ibp: 64–bit x86 device driver

Notes

IBP is a GLD-based driver and provides the statistics described by gld(7D). Valid received packets not accepted by any stream (long) increases when IBP transmits broadcast IP packets. This happens because the infiniband hardware copies and loops back the transmitted broadcast packets to the source. These packets are discarded by GLD and are recorded as unknowns.

Skip Navigation Links
Exit Print View
	man pages section 7: Device and Network Interfaces Oracle Solaris 11.1 Information Library

ibp