powerpc / cxl: Add support for the Mellanox CX4 in cxl mode

Ian Munsie imunsie at au1.ibm.com
Mon Jul 4 23:21:58 AEST 2016


This series adds support for the Mellanox CX4 network adapter operating in cxl
mode to the cxl driver and the PowerNV PHB code. The Mellanox developers will
submit a separate patch series that makes use of this in the mlx5 driver.

The CX4 card can operate in either pci mode, or cxl mode. In cxl mode, memory
accesses from the card go through the XSL (Translation Service Layer,
essentially a stripped down version of the Power Service Layer), allowing it to
transparently access unpinned memory with the cxl driver handling faulting in
pages as necessary, etc. Most of the support for the XSL is already upstream,
though this series does include a bug fix to enable bus mastering for this
(patch 3).

Patch 2 in this series provides an API which the mlx5 driver can query to check
if it is in a cxl capable slot. The card will come up in pci mode, and the mlx5
driver can choose to switch it to cxl mode, wherein it will reappear with an
additional physical function representing the XSL that the cxl driver will bind
to. Patches 12-14 add support for switching the card's mode, including using
the PCI hotplug support to re-enumerate the device tree and re-probind the
card.

Unlike previous users of the cxl kernel API where we used a virtual PHB and
exposed PCI devices under it, the Mellanox CX4 uses a peer model where cxl
binds to one of the physical functions of the card and the mlx5_core driver
binds to the other networking physical functions. Patches 6 and 7 add support
for using the cxl kernel API with the real PHB to enable this peer model.
Patches 4 and 5 are prepatory patches exposing some APIs that the PHB will need
to call.

While in cxl mode, interrupts from the CX4 are a little unusual - they are
neither pci interrupts, nor cxl interrutps, but rather a hybrid of the two. The
interrupts are passed from the networking hardware to the XSL using a custom
format in the MSIX table, and from there are treated as cxl interrupts. These
are configured mostly transparently using the standard msix APIs - the PHB
handles allocating and configuring the cxl interrupts, associating them with
the default context, and the mlx5 driver handles filling out the MSIX table
with their custom format (not included in this series). See patch 10.

Additionally, the CX4 has a hard limitation of the number of interrupts that
can be associated with a given context, so to overcome this patches 8 and 9
expose an API to allow the mlx5 driver to inform us of the limit, and the
interrupt allocation code in patch 10 will allocate additional contexts to
associate these with.

Patch 1 is a prepatory cleanup patch to reorganise cxl code in arch/powerpc
into a separate file.

Patch 11 is a workaround for a hardware limitation in the CX4 where a context
with PE=0 cannot be used.

Note that patch 2 depends on "cxl: Ignore CAPI adapters misplaced in switched
slot" by Philippe Bergheaud:
http://patchwork.ozlabs.org/patch/642920/

Additionally, the following stand-alone patches related to the CX4 are also
pending on the mainling list, but are *not* dependencies of this series:
- cxl: Fix bug where AFU disable operation had no effect
- cxl: Workaround XSL bug that does not clear the RA bit after a reset
- cxl: Fix NULL pointer dereference on kernel contexts with no AFU interrupts

The entire series is bisectable.



More information about the Linuxppc-dev mailing list