[PATCH 5/5 v11] iommu/fsl: Freescale PAMU driver and iommu implementation.

[Alex Williamson]

On Thu, 2013-04-04 at 16:35 +0000, Sethi Varun-B16395 wrote:
> 
> > -----Original Message-----
> > From: Alex Williamson [mailto:alex.williamson at redhat.com]
> > Sent: Thursday, April 04, 2013 8:52 PM
> > To: Sethi Varun-B16395
> > Cc: Joerg Roedel; Yoder Stuart-B08248; Wood Scott-B07421;
> > iommu at lists.linux-foundation.org; linuxppc-dev at lists.ozlabs.org; linux-
> > kernel at vger.kernel.org; galak at kernel.crashing.org;
> > benh at kernel.crashing.org
> > Subject: Re: [PATCH 5/5 v11] iommu/fsl: Freescale PAMU driver and iommu
> > implementation.
> > 
> > On Thu, 2013-04-04 at 13:00 +0000, Sethi Varun-B16395 wrote:
> > >
> > > > -----Original Message-----
> > > > From: Alex Williamson [mailto:alex.williamson at redhat.com]
> > > > Sent: Wednesday, April 03, 2013 11:32 PM
> > > > To: Joerg Roedel
> > > > Cc: Sethi Varun-B16395; Yoder Stuart-B08248; Wood Scott-B07421;
> > > > iommu at lists.linux-foundation.org; linuxppc-dev at lists.ozlabs.org;
> > > > linux- kernel at vger.kernel.org; galak at kernel.crashing.org;
> > > > benh at kernel.crashing.org
> > > > Subject: Re: [PATCH 5/5 v11] iommu/fsl: Freescale PAMU driver and
> > > > iommu implementation.
> > > >
> > > > On Tue, 2013-04-02 at 18:18 +0200, Joerg Roedel wrote:
> > > > > Cc'ing Alex Williamson
> > > > >
> > > > > Alex, can you please review the iommu-group part of this patch?
> > > >
> > > > Sure, it looks pretty reasonable.  AIUI, all PCI devices are below
> > > > some kind of host bridge that is either new and supports
> > > > partitioning or old and doesn't.  I don't know if that's a
> > > > visibility or isolation requirement, perhaps PCI ACS-ish.  In the
> > > > new host bridge case, each device gets a group.  This seems not to
> > > > have any quirks for multifunction devices though.  On AMD and Intel
> > > > IOMMUs we test multifunction device ACS support to determine whether
> > > > all the functions should be in the same group.  Is there any reason
> > to trust multifunction devices on PAMU?
> > > >
> > > [Sethi Varun-B16395] In the case where we can partition endpoints we
> > > can distinguish transactions based on the bus,device,function number
> > > combination. This support is available in the PCIe controller (host
> > > bridge).
> > 
> > So can x86 IOMMUs, that's the visibility aspect of IOMMU groups.
> > Visibility alone doesn't necessarily imply that a device is isolated
> > though.  A multifunction PCI device that doesn't expose ACS support may
> > not isolate functions from each other.  For example a peer-to-peer DMA
> > between functions may not be translated by the upstream IOMMU.  IOMMU
> > groups should encompass both visibility and isolation.
> [Sethi Varun-B16395] We can isolate the DMA access to the host based
> on the to the pci bus,device,function number.

The IOMMU can only isolate DMA that it can see.  A multifunction device
may never expose peer-to-peer DMA to the upstream device, it's
implementation specific.  The ACS flags allow that possibility to be
controlled and prevented.

> I thought that was enough to put devices in to separate iommu groups.
> This is a PCIe controller property which allows us to partition PCIe
> devices. But, what I can understand from your point is that we also
> need to consider isolation at PCIe device level as well. I will check
> for the case of multifunction devices.
> 
> > 
> > > > I also find it curious what happens to the iommu group of the host
> > > > bridge.  In the partitionable case the host bridge group is removed,
> > > > in the non-partitionable case the host bridge group becomes the
> > > > group for the children, removing the host bridge.  It's unique to
> > > > PAMU so far that these host bridges are even in an iommu group (x86
> > > > only adds pci devices), but I don't see it as necessarily wrong
> > > > leaving it in either scenario.  Does it solve some problem to remove
> > them from the groups?
> > > > Thanks,
> > > [Sethi Varun-B16395] The PCIe controller isn't a partitionable entity,
> > > it would always be owned by the host.
> > 
> > Ownership of a device shouldn't play into the group context.  An IOMMU
> > group should be defined by it's visibility and isolation from other
> > devices.  Whether the PCIe controller is allowed to be handed to
> > userspace is a question for VFIO.
> [Sethi Varun-B16395] The problem is in the case, where we can't
> partition PCIe devices. PCIe devices share the same device group as
> the PCI controller. This becomes a problem while assigning the devices
> to the guest, as you are required to unbind all the PCIe devices
> including the controller from the host. PCIe controller can't be
> unbound from the host, so we simply delete the controller iommu_group.

Unbinding devices is a VFIO implementation, it shouldn't leak into IOMMU
groups.  Also note that VFIO has a driver white list where we can have
exceptions to the rule.  I recently added pciehp to that list because
the host driver provides functionality.  Being attached to the host
driver means the device is not accessible to the user through VFIO, but
other devices in the group are.  Thanks,

Alex