[PATCH/RFC] ppc64: EEH + SCSI recovery (IPR only)]

[Linas Vepstas]

Hi Ben,

On Thu, Feb 24, 2005 at 12:16:11PM +1100, Benjamin Herrenschmidt was heard to remark:
> 
> > The next patch will put the recovery routines into 
> > struct pci_driver.  And we'll go from there... 
> 
> "recovery routines" isn't quite clear... what do you have in mind
> exactly ? 

OK, see below.

>Have you read my previous mails on this issue a couple of
> weeks ago on this list ?

Yes. And I liked them ... 

> There are various issues there, like properly synchronizing drivers
> sharing the same bus segment, dealing with drivers that don't support
> recovery on the same segment, API suitable for other platforms that do
> things differently, etc... etc...

Yes, well, these are the same old issues since the dawn of time.  
When I read your emails, I sensed that you were heading back to
the "original" design point, which had been shredded , but was the 
one that made the most sense to me.  So I took your emails in a 
very positive light :)

---------------
So, here was the general thinking:

> properly synchronizing drivers
> sharing the same bus segment,

This is done by having a "master eeh reset/recovery thread", 
which synchronizes the recovery across any/all device drivers 
that are affected.

It does this by calling back into each device driver to tell 
them things like, "now the bus is frozen", "now the bus is being 
reset", "now the bus is finsihed being reset".  Each device
driver can handle these as desired.

In the most recent patch, I put these callbacks into a 
"struct eeh_recovery_ops" for now, but these should probably  
go into "struct pci_driver".  Right now, the names and purpose 
of these callbacks are a rough "first cut" and subject to change.

-----------
Historically, one of the issues that was repeatedly argued was whether
the "master eeh reset/recovery thread" should run in the kernel 
or in user space.  I think things are a lot simpler if this 
"master recovery thread" runs in the kernel.  

> API suitable for other platforms that do things differently, 

I'm currently envisioning  the "master recovery thread" to be 
arch-specific and/or pci-bus-controller+bios/firmware-specific.  
This avoids having to figure out how to design a generic recovery 
procedure that would be arch-indep, and yet still capable of 
reseting a pci bus for any weird current or future pci bus design. 

In the current patch I sent out, this "master recovery thread" 
is implemented in the "rpaphp" code, and thus is specific to 
the rpa pci bus.

-------------

> dealing with drivers that don't support
> recovery on the same segment, 

In the current patch, if a driver failed to register a "struct
eeh_recovery_ops", then that driver would get whacked by a 
rpaphp_unconfig_pci_adapter()  This works, because, ...

    rpaphp_unconfig_pci_adapter (struct slot *)  // in rpaphp_pci.c
    {
      calls
      pci_remove_bus_device (struct pci_dev *) // in /drivers/pci/remove.c
      {
        calls
        pci_destroy_dev (struct pci_dev *)
        {

etc ...
                   calls
                    struct pci_driver->remove()

which frees memory .. etc.
also generates user-land events to udev, and so on.

-------------
The bonus of using hotplug to recover is that any hotplug-capble driver
can be recovered, (although this is a heavy-handed aproach).  This
works almost everywhere, except for anything that holds a block device,
where we can't just unmount filesystems after the fact.


--linas