smp: Start up non-boot CPUs asynchronously

[Srivatsa S. Bhat]

[Small note, it appears as if the last 2 of your replies to this thread
 didn't reach LKML.]

On 02/14/2012 08:01 PM, Arjan van de Ven wrote:

> one coments; will comment more when I get to work
> 
> On Tue, Feb 14, 2012 at 1:48 AM, Srivatsa S. Bhat 
> 
> 7. And whichever code between smp_init() and async_synchronize_full() didn't
> 
>     care about CPU hotplug till today but depended on all cpus being
>     online must
>     suddenly start worrying about CPU Hotplug. They must register a cpu
>     notifier
>     and handle callbacks etc etc.. Or if they are not worth that
>     complexity, they
>     should atleast be redesigned or moved around - like the print
>     statements that
>     tell how many cpus came up, for example.
> 
> 
> frankly, such code HAS to worry about cpus going online and offline even
> today; the firmware, at least on X86, can start taking cores
> offline/online once ACPI is initialized....
> (as controlled by a data center manager from outside the box, usually
> done based on thermal or power conditions on a datacenter level).
> Now, no doubt that we have bugs in this space, since this only happened
> very rarely before.
> 
> Question is what to do from a longer term strategy:
> Either we declare the number of online CPUs invariant during a certain
> phase of the boot (and make ACPI and co honor this as well somehow)
> or
> We decide to go about fixing these (maybe with the help of lockdep?)
> 
> In addition to this, the reality is that the whole "bring cpus up"
> sequence needs to be changed; the current one is very messy and requires
> the hotplug lock for the whole bring up of each individual cpu... which
> is a very unfortunate design; a much better design would be to only take
> the lock for the actual registration of the newly brought up CPU to the
> kernel, while running the physical bringup without the global lock.
> If/when that change gets made, we can do the physical bring up in
> parallel (with each other, but also with the rest of the kernel boot),
> and do the registration en-mass at some convenient time in the boot,
> potentially late.
> 


Sounds like a good idea, but how will we take care of CPU_UP_PREPARE and
CPU_STARTING callbacks then? Because, CPU_UP_PREPARE callbacks are run
before bringing up the cpu and CPU_STARTING is called from the cpu that is
coming up. Also, CPU_UP_PREPARE callbacks can be failed, which can lead
to that particular cpu boot getting aborted. With the "late commissioning
of CPUs" idea you proposed above, retaining such semantics could become
very challenging.

Regards,
Srivatsa S. Bhat