Using VFIO vs. developing a kernel module

[Dhananjay Phadke]

On Fri, 28 May 2021, Andrew Jeffery wrote:
>
> On Thu, 27 May 2021, at 22:53, sainath grandhi wrote:
>> Hello,
>> Our project has an FPGA connected to BMC as a PCIe endpoint. This
>> endpoint provides a set of registers via MMIO and an interrupt for
>> notifying completion of work. This endpoint also implements AER
>> capability.
>>
>> We have two options to enable this endpoint.
>> 1) Write a new kernel module with a character device interface for
>> user-space interaction.
>> 2) Use VFIO infrastructure provided by Linux and write an user-space
>> application.
>>
>> I am reaching out to the community to check if there is any
>> recommended option, using VFIO vs. implementing a new kernel module,
>> or any previous experiences weighing in one option over the other.
>
> I don't have any experience with VFIO, so take this with a grain of salt.
>
> Generally you should write an in-kernel driver for it. The reason you
> might not want to do so is if the device's register interface changes
> frequently, as it's more pain to update the kernel than some userspace
> application, which slows iteration. But handling DMAs and interrupts
> make userspace more painful, so unless VFIO helps there (I assume it
> does), then that would push the implementation back towards the kernel.

VFIO requires IOMMU to protect kernel memory corruption by device DMA,
which is programmed by (untrusted) userspace app. Unless the BMC SoC
implements IOMMU (SMMU) for I/O virtualization (I/O page tables), it
would not be possible/safe to use VFIO for DMA/interrupts.

See https://www.kernel.org/doc/Documentation/vfio.txt

If you just want to program MMIO registers, you could mmap() PCIe sysfs
resource files that represent MMIO bars and let application write and
read registers, including polling for completion.

See https://www.kernel.org/doc/Documentation/PCI/sysfs-pci.rst

If none of these options are viable, then a custom kernel driver would
be necessary.

Regards,
Dhananjay