To extend the feature of vfio-mdev

Jean-Philippe Brucker jean-philippe.brucker at
Mon Oct 23 23:03:37 AEDT 2017

On 23/10/17 11:37, Kenneth Lee wrote:
> On Mon, Oct 23, 2017 at 09:31:25AM +0100, Jean-Philippe Brucker wrote:
>> Date: Mon, 23 Oct 2017 09:31:25 +0100
>> From: Jean-Philippe Brucker <jean-philippe.brucker at>
>> To: Kenneth Lee <liguozhu at>, Alex Williamson
>>  <alex.williamson at>
>> CC: "kenneth-lee-2012 at" <kenneth-lee-2012 at>, Jon
>>  Masters <jcm at>, "jcm at" <jcm at>,
>>  "xuzaibo at" <xuzaibo at>,
>>  "linux-accelerators at"
>>  <linux-accelerators at>
>> Subject: Re: To extend the feature of vfio-mdev
>> User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
>>  Thunderbird/52.4.0
>> Message-ID: <6fed6c27-4b38-7d65-ee5d-63d6bb283e95 at>
>> On 23/10/17 06:18, Kenneth Lee wrote:
>>>>>>> 3. Let iommu driver accept more than one iommu_domain for the same
>>>>>>>    device. The substream id or pasid should be support for that  
>>>>>> You're really extending the definition of an iommu_domain to include
>>>>>> PASID to do this, I don't think it makes sense in the general case.  So
>>>>>> perhaps you're talking about a PASID management layer sitting on top of
>>>>>> an iommu_domain.  AIUI for PCIe, a device has a requester ID which is
>>>>>> used to find the context entry for that device.  The IOMMU may support
>>>>>> PASID, which would cause a first level lookup via those set of page
>>>>>> tables, or it might only support second level translation.  The
>>>>>> iommu_domain is a reflection of that initial, single requester ID.  
>>>>> Maybe I misunderstand this. But the IOMMU hardware, such as SMMU for ARM,
>>>>> support multiple page table and is referred by something like ASID. If we should
>>>>> support it in Linux, iommu_domain should be the best choice (no matter you call
>>>>> it cookie or id or something else). Or where you can get a object referring to it?
>>>> For PASID, a PASID is unique only within the requester ID.  I don't
>>>> know of anything equivalent to your ASID within PCIe.
>>> As my understanding to the ARM Spec:
>>> 	PCIE[RequestID]=ARM_SMMU[StreamID]
>> In the SMMUv3 spec, the PASID is dissociated from the ASID. We set the
>> ASID in the context descriptor indexed by PASID, which provides more
>> flexibility when the CPU's ASID capacity and endpoint's PASID capacity differ.
> Yes. Sorry, I tried to make the concept simply. But it is wrong slightly.

That's all right :) I'm still not sure if we should implement ASID=PASID
for arm. I think it would be overly complicated to implement, because of
the capacity difference and because it prevents users from allocating and
controlling PASIDs contexts on their own with map/unmap, without attaching
to a Linux process (some users want this).

>>> For ARM SMMU, the Stream ID is used to index a Context Descriptors
>>> Table while and Sub-stream ID is used to index a Descriptor which refer
>>> to a general page table in the same format as MMU.
>>> So both RequestID and PASID uniquely identify a address space. Then
>>> every iommu-enabled device can service more than one user process at the
>>> same time.
>>> So the IOMMU should support more than one page table, which in turn
>>> should be added to somewhere in Linux. If the iommu_domain refer to one
>>> address space, the iommu driver should accept more than one
>>> iommu_domain.
>> The current design choice is to have multiple address space per
>> iommu_domain (and one PASID table per domain). It fits better with
>> existing IOMMU and VFIO APIs.
> I am still reading the patchset. But if the PASID is used to index the address
> space table in iommu_domain, it will become a general concept in the whole
> system. But if we would bind the address space with iommu_domain, the
> iommu_domain become a handle for any process to control the device dma. It will
> be easier for one process with two or more domains to control the device for
> different purpose. And it is also easier for a process hand over the domain to
> another without change anything. And the process need not to know anything about
> pasid.

Unfortunately this cannot work with vfio-pci, where one userspace process
owns the whole function. Userspace has to obtain the PASID from the kernel
allocator and somehow program it into the device, with an
implementation-defined method.

> A domain is a handle to control the device separately. I think this make
> thing simpler.

Similarly, how different parts of a function can be controlled and
assigned separate PASIDs is implementation-defined in PCI. The
userspace driver is given a single device fd in a VFIO container, and
has to do the partitioning itself.


More information about the Linux-accelerators mailing list