[PATCH v6 1/3] uacce: Add documents for uacce

[Kenneth Lee]

On Mon, Oct 21, 2019 at 03:34:55PM +0200, Jean-Philippe Brucker wrote:
> Date: Mon, 21 Oct 2019 15:34:55 +0200
> From: Jean-Philippe Brucker <jean-philippe at linaro.org>
> To: Kenneth Lee <Kenneth-Lee-2012 at foxmail.com>
> Cc: Zhangfei Gao <zhangfei.gao at linaro.org>, Greg Kroah-Hartman
>  <gregkh at linuxfoundation.org>, Arnd Bergmann <arnd at arndb.de>, Herbert Xu
>  <herbert at gondor.apana.org.au>, jonathan.cameron at huawei.com,
>  grant.likely at arm.com, ilias.apalodimas at linaro.org,
>  francois.ozog at linaro.org, Wangzhou <wangzhou1 at hisilicon.com>, "haojian .
>  zhuang" <haojian.zhuang at linaro.org>, linux-accelerators at lists.ozlabs.org,
>  linux-kernel at vger.kernel.org, linux-crypto at vger.kernel.org, Kenneth Lee
>  <liguozhu at hisilicon.com>, Zaibo Xu <xuzaibo at huawei.com>
> Subject: Re: [PATCH v6 1/3] uacce: Add documents for uacce
> Message-ID: <20191021133455.GC117664 at lophozonia>
> 
> Hi Kenneth,
> 
> On Thu, Oct 17, 2019 at 03:49:07PM +0800, Kenneth Lee wrote:
> > Dear Jean,
> > 
> > Please let me answer your question about why we build another subsystem
> > other than use vfio-mdev.
> > 
> > I think you might remember that we did build WarpDrive on top of
> > vfio-mdev from the very beginning.
> 
> Right thanks for reminding me, I had forgotten about the first RFCs.
> 
> > Both RFCv1, Share Parent IOMMU mdev, and the RFCv2, Share Domain mdev,
> > are based on mdev. We got many comments and we finally felt we could not
> > solve all of them if we continued the mdev directory.
> > 
> > I think the key problem here is that mdev is a virtual *device*. So, 
> > 
> > 1. As you have said, this creates more logic which is useless for
> >    accelerator. For example, it gives the user the full control of DMA
> >    and irq, it replays the dma mapping for new attach device and it
> >    create VFIO IOMMU drivers... These are necessary to simulate a raw
> >    device to a virtual machine. But it is not necessary to an
> >    accelerator.
> > 
> > 2. You are forced to separate the resource to a device before use it.
> >    And if the user process crash, we need extra facility to put it back
> >    to the resource pool.
> 
> I don't understand the difference between vfio-mdev and uacce in this
> context. An example may help. If you want to give direct access of a bit
> of hardware to userspace, you necessarily need to isolate any resource
> associated to that partition from other processes and from the kernel.
> Namely create a DMA address space (SVA or AUXD), allocate an MMIO frame
> and an interrupt (although IRQs are still handled by the kernel and could
> be shared). And then you need to release those resources back into the
> pool when the process exits or crashes.

But the logic is different here:). VFIO creates a device for particular
application, such as qemu or dpdk, with promised resource. While uacce
manage some computing resource for the whole system. Accelerator is a
heterogeneous cpu. It is good at some particular calculation. So the
caller just use the computing power for some time. It need not to keep
it for long. We can certainly add quota feature in the future. But it
need not to be per-deviced, needn't it?

As a example, if I have 3 zip-cores in my accelerator, I don't need to
keep them in 3 virtual device. And I don't need to do all kinds of DMA
from user space. I hope I just send a pointer to the hardware and tell:
"I need some work done, please update these data for me". It is not
about DMA, it is about heterogeneous core calling from my own process
space.

> 
> > 3. Though Alex Williamson argues that vfio is not just used for
> >    virtualisation. But it is indeed used only by virtualisation for the
> >    time being.
> 
> There is DPDK, that implements userspace drivers for net and crypto using
> vfio-pci, and will likely gain support for vfio-mdev soon. However
> similarly to Qemu they are self-contained and can easily deal with the
> fork problem, unlike a decompression library for example, that could be
> included by any application.

Yes. Both DPDK and qemu know they are using a device and they do DMA on
it. VFIO should keep this promise. So it is not easy to solve the COW
problem and even it is not necessary.

With uacce, we don't have that promise. It is not hard to clone the
device context and memory to the child if necessary.

> 
> In any case I agree with your point 1. that a simpler user interface might
> be beneficial. Perhaps DPDK could support uacce as well later.
> 
> Thanks,
> Jean
> 
> >    And Jerome Glisse （also from Redhat) said he could
> >    accept some problem from a virtual machine because we can constrain
> >    the behavior of virtual machine program (such as qemu) . But he could
> >    not accept that happened in an general application. For example, if
> >    you pin the memory in a process and then fork, you may lost the
> >    physical page due to COW. We can solve the problem in uacce by
> >    letting go the shared pages in the child.
> > 
> > So we think we should not continue the mdev direction for uacce. Even we
> > can merge the logic together. It we become a burden for both vfio and
> > uacce. Both of them make use of IOMMU, so of course they will have some
> > code similar. But they go to different direction. VFIO is trying to
> > export virtual DMA and irq to the user space. While uacce is trying to
> > let the accelerator share address with the process itself. Many tricks
> > can be made between the final user interface and the address translation
> > framework. Merge them together will not be good to both of them. 
> > 
> > Hope this answer some of your question.
> > 
> > Cheers:)

-- 
			-Kenneth Lee (Hisilicon)