Device Interrupts

I/O buses implement interrupts in two common ways: vectored and polled. Both methods commonly supply a bus-interrupt priority level. Vectored devices also supply an interrupt vector. Polled devices do not supply interrupt vectors.

To stay current with changing bus technologies, the Oracle Solaris OS has been enhanced to accommodate both newer types of interrupts and more traditional interrupts that have been in use for many years. Specifically, the operating system now recognizes three types of interrupts:

• Legacy interrupts – Legacy or fixed interrupts refer to interrupts that use older bus technologies. With these technologies, interrupts are signaled by using one or more external pins that are wired “out-of-band,” that is, separately from the main lines of the bus. Newer bus technologies such as PCI Express maintain software compatibility by emulating legacy interrupts through in-band mechanisms. These emulated interrupts are treated as legacy interrupts by the host OS.

• Message-signaled interrupts – Instead of using pins, message-signaled interrupts (MSI) are in-band messages and can target addresses in the host bridge. (See PCI Local Bus for more information on host bridges.) MSIs can send data along with the interrupt message. Each MSI is unshared so that an MSI that is assigned to a device is guaranteed to be unique within the system. A PCI function can request up to 32 MSI messages.

• Extended message-signaled interrupts – Extended message-signaled interrupts (MSI-X) are an enhanced version of MSIs. MSI-X interrupts have the following added advantages:

  • Support 2048 messages rather than 32 messages

  • Support independent message address and message data for each message

  • Support per-message masking

  • Enable more flexibility when software allocates fewer vectors than hardware requests. The software can reuse the same MSI-X address and data in multiple MSI-X slots.

High-Level Interrupts

A bus prioritizes a device interrupt at a bus-interrupt level. The bus interrupt level is then mapped to a processor-interrupt level. A bus interrupt level that maps to a CPU interrupt priority above the scheduler priority level is called a high-level interrupt. High-level interrupt handlers are restricted to calling the following DDI interfaces:

• mutex_enter(9F) and mutex_exit(9F) on a mutex that is initialized with an interrupt priority associated with the high-level interrupt

• ddi_intr_trigger_softint(9F)

• The following DDI get and put routines: ddi_get8(9F), ddi_put8(9F), ddi_get16(9F), ddi_put16(9F), ddi_get32(9F), ddi_put32(9F), ddi_get64(9F), and ddi_put64(9F).

A bus-interrupt level by itself does not determine whether a device interrupts at a high level. A particular bus-interrupt level can map to a high-level interrupt on one platform, but map to an ordinary interrupt on another platform.

A driver is not required to support devices that have high-level interrupts. However, the driver is required to check the interrupt level. If the interrupt priority is greater than or equal to the highest system priority, the interrupt handler runs in high-level interrupt context. In this case, the driver can fail to attach, or the driver can use a two-level scheme to handle interrupts. For more information, see Handling High-Level Interrupts.

Legacy Interrupts

The only information that the system has about a device interrupt is the priority level of the bus interrupt and the interrupt request number. An example of the priority level for a bus interrupt is the IPL on an SBus in a SPARC machine. An example of an interrupt request number is the IRQ on an ISA bus in an x86 machine.

When an interrupt handler is registered, the system adds the handler to a list of potential interrupt handlers for each IPL or IRQ. When the interrupt occurs, the system must determine which device actually caused the interrupt, among all devices that are associated with a given IPL or IRQ. The system calls all the interrupt handlers for the designated IPL or IRQ until one handler claims the interrupt.

The following buses are capable of supporting polled interrupts:

• SBus

• ISA

• PCI

Standard and Extended Message-Signaled Interrupts

Both standard (MSI) and extended (MSI-X) message-signaled interrupts are implemented as in-band messages. A message-signaled interrupt is posted as a write with an address and value that are specified by the software.

MSI Interrupts

Conventional PCI specifications include optional support for Message Signaled Interrupts (MSI). An MSI is an in-band message that is implemented as a posted write. The address and the data for the MSI are specified by software and are specific to the host bridge. Because the messages are in-band, the receipt of the message can be used to “push” data that is associated with the interrupt. By definition, MSI interrupts are unshared. Each MSI message that is assigned to a device is guaranteed to be a unique message in the system. PCI functions can request 1, 2, 4, 8, 16, or 32 MSI messages. Note that the system software can allocate fewer MSI messages to a function than the function requested. The host bridge can be limited in the number of unique MSI messages that are allocated for devices.

MSI-X Interrupts

MSI-X interrupts are enhanced versions of MSI interrupts that have the same features as MSI interrupts with the following key differences:

• A maximum of 2048 MSI-X interrupt vectors are supported per device.

• Address and data entries are unique per interrupt vector.

• MSI-X supports per function masking and per vector masking.

With MSI-X interrupts, an unallocated interrupt vector of a device can use a previously added or initialized MSI-X interrupt vector to share the same vector address, vector data, interrupt handler, and handler arguments. Use the ddi_intr_dup_handler(9F) function to alias the resources provided by the Oracle Solaris OS to the unallocated interrupt vectors on an associated device. For example, if 2 MSI-X interrupts are allocated to a driver and 32 interrupts are supported on the device, then the driver can use ddi_intr_dup_handler() to alias the 2 interrupts it received to the 30 additional interrupts on the device.

The ddi_intr_dup_handler() function can duplicate interrupts that were added with ddi_intr_add_handler(9F) or initialized with ddi_intr_enable(9F).

A duplicated interrupt is disabled initially. Use ddi_intr_enable() to enable the duplicated interrupt. You cannot remove the original MSI-X interrupt handler until all duplicated interrupt handlers that are associated with this original interrupt handler are removed. To remove a duplicated interrupt handler, first call ddi_intr_disable(9F), and then call ddi_intr_free(9F). When all duplicated interrupt handlers that are associated with this original interrupt handler are removed, then you can use ddi_intr_remove_handler(9F) to remove the original MSI-X interrupt handler. See the ddi_intr_dup_handler(9F) man page for examples.

Software Interrupts

The Oracle Solaris DDI/DKI supports software interrupts, also known as soft interrupts. Soft interrupts are initiated by software rather than by a hardware device. Handlers for these interrupts must also be added to and removed from the system. Soft interrupt handlers run in interrupt context and therefore can be used to do many of the tasks that belong to an interrupt handler.

Hardware interrupt handlers must perform their tasks quickly, because the handlers might have to suspend other system activity while doing these tasks. This requirement is particularly true for high-level interrupt handlers, which operate at priority levels greater than the priority level of the system scheduler. High-level interrupt handlers mask the operations of all lower-priority interrupts, including the interrupt operations of the system clock. Consequently, the interrupt handler must avoid involvement in activities that might cause it to sleep, such as acquiring a mutex.

If the handler sleeps, then the system might hang because the clock is masked and incapable of scheduling the sleeping thread. For this reason, high-level interrupt handlers normally perform a minimum amount of work at high-priority levels and delegate other tasks to software interrupts, which run below the priority level of the high-level interrupt handler. Because software interrupt handlers run below the priority level of the system scheduler, software interrupt handlers can do the work that the high-level interrupt handler was incapable of doing.