[PATCH/RFC] PCI Error Recovery

[Benjamin Herrenschmidt]

> Why not?
> Can't we do that today with various PCI initialization
> routines that provide arch (pcibios) specific hooks?
> e.g. pci_set_master vs pci_enable_device

Well, it gets complicated. For example, the driver may try to re-enable
IO to check for some status, but that itself triggers a new error right
away because the HW is dead ...

Maybe the driver should "assume" by default indeed that the slot is
isolated by default (even if it's not on non-ppc64 archs) and thus has
to call pci_enable_device() and pci_set_master() again. It adds some
burden to the underlying code though to figure out if those calls are
emitted in the context of an error or not, since the operations are
completely different at the firmware level.

Then, there is need to inform the driver as well of the capability to
reset the slot, to be used if the driver decides it can't recover.

Finally, I'm not fan at _ALL_ of providing synchronous APIs like
pci_enable_device() or pci_set_master(). In fact, those two would be not
_too_ bad, but the slot reset is more nasty. The problem is that we have
potentially more than one driver affected. Even if the error was
triggered by one card/function, several cards/functions may have been
isolated etc...

We need to "notify" all drivers, give them a chance to re-enable device
& gather diagnostic data, etc... before we try to reset the slot if a
driver decides it requires that to happen. Also, if a driver is ok after
just enabling the device() re-initializes itself, but it's sibling
decides it needs to reset the slot ?

This is why I'm more inclined toward a callback that acts like a state
machine.
 
> I'm wondering if the second part of the error recovery path in
> the driver can use it's "normal" initialization sequence.
> Proably needs adjusting to look for error states and the first
> part will need to clean up pending IO requests.

Oh it could, but I wouldn't make it mandatory by calling probe() or
whatevre. It's up to each driver to decide, easy enough to move their
init code into a function called by both code path.

> >  but I need
> > to find the right "cutting point". Just re-enabling IOs is useful for
> > drivers who can extract diagnostic infos from the device, for example
> > after a DMA error.
> 
> By "IO", I'm guessing you mean MMIO or IO Port space access.

Yes.

> This implies only the device driver knows what/where any diag info lives.

Yes.

> But some of the info is architected in PCI: SERR and PERR status bits.
> PCIe seems to be richer in error reporting but I don't know details.

Oh, sure, and that's why it may not be worth bothering about this "step"
and just always reset the slot when we can. But we then need to inform
the driver of what happened since not all platforms will be able to do
that. That would definitely simplify the above problem, and this is what
I meant by "I may not offer that rich functionality in the generic API"

> I think the majority of the error info is much more likely to be held
> in driver state and platform chipset state. E.g. only the driver will
> be able to associate a particular IO request with the invalid DMA or
> MMIO address that the chipset captured. The driver can reject that IO
> (with extreme prejudice so it doesn't get retried) and restart the PCI
> device.
>
> In case it's not obvious, this is all just hand waving and maybe
> it will inspire something more realistic...
> 
> > Resetting the slot may be necessary to get some devices back.
> 
> *nod*  Or even several slots.
>
> > > I don't get why the driver even needs to know about isolation
> > > or not. It's not fundamentally different from an bus abort
> > > on other systems, just that it lasts longer. 
> 
> I think the driver just needs to know if it's ok to do MMIO/IO Port
> access to the device or not at any given point in time.
> 
> A simpler strategy could be to just blow away (PCI Bus reset) the failed
> device(s) and reconfigure the PCI bus. Then call back into the drivers
> to tell them their devices suffered an "event". But then finer grain
> recovery isn't really possible.

Yes, but I think fine grained recovery ends up beeing an API nightmare
when you start dealing with several drivers on the same segment with
conflicting requirements for recovery.

Now, the problem is that we have to provide both approaches anway, as a
lot of platforms can't do anything but clear the SERR/PERR state and
hope the driver can go on. So we need to inform the driver of the
platform capability in a way.

Ben.