Using VFIO vs. developing a kernel module

[sainath grandhi]

On Fri, May 28, 2021, 6:12 PM Dhananjay Phadke <dphadke at linux.microsoft.com>
wrote:

> On Fri, 28 May 2021, Andrew Jeffery wrote:
> >
> > On Thu, 27 May 2021, at 22:53, sainath grandhi wrote:
> >> Hello,
> >> Our project has an FPGA connected to BMC as a PCIe endpoint. This
> >> endpoint provides a set of registers via MMIO and an interrupt for
> >> notifying completion of work. This endpoint also implements AER
> >> capability.
> >>
> >> We have two options to enable this endpoint.
> >> 1) Write a new kernel module with a character device interface for
> >> user-space interaction.
> >> 2) Use VFIO infrastructure provided by Linux and write an user-space
> >> application.
> >>
> >> I am reaching out to the community to check if there is any
> >> recommended option, using VFIO vs. implementing a new kernel module,
> >> or any previous experiences weighing in one option over the other.
> >
> > I don't have any experience with VFIO, so take this with a grain of salt.
> >
> > Generally you should write an in-kernel driver for it. The reason you
> > might not want to do so is if the device's register interface changes
> > frequently, as it's more pain to update the kernel than some userspace
> > application, which slows iteration. But handling DMAs and interrupts
> > make userspace more painful, so unless VFIO helps there (I assume it
> > does), then that would push the implementation back towards the kernel.
>
> VFIO requires IOMMU to protect kernel memory corruption by device DMA,
> which is programmed by (untrusted) userspace app. Unless the BMC SoC
> implements IOMMU (SMMU) for I/O virtualization (I/O page tables), it
> would not be possible/safe to use VFIO for DMA/interrupts.
>
> See https://www.kernel.org/doc/Documentation/vfio.txt
>
> Yeah. In our case the endpoint does not have DMA.  And uses INTx for
> interrupt to avoid memory corruption issues.


Our BMC does not support SMMU, so I am using VFIO with no IOMMU
> configuration. Linux kernel introduced VFIO without IOMMU for such
> platforms so that userspace applications can still take advantage of the
> VFIO facilities.
>

If you just want to program MMIO registers, you could mmap() PCIe sysfs
> resource files that represent MMIO bars and let application write and
> read registers, including polling for completion.
>
> See https://www.kernel.org/doc/Documentation/PCI/sysfs-pci.rst
>
> If none of these options are viable, then a custom kernel driver would
> be necessary.
>
> Regards,
> Dhananjay
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ozlabs.org/pipermail/openbmc/attachments/20210529/89be9820/attachment.htm>