[PATCH v17 5/6] powerpc/crash: add crash CPU hotplug support
Hari Bathini
hbathini at linux.ibm.com
Sun Mar 3 00:18:58 AEDT 2024
On 26/02/24 2:11 pm, Sourabh Jain wrote:
> Due to CPU/Memory hotplug or online/offline events, the elfcorehdr
> (which describes the CPUs and memory of the crashed kernel) and FDT
> (Flattened Device Tree) of kdump image becomes outdated. Consequently,
> attempting dump collection with an outdated elfcorehdr or FDT can lead
> to failed or inaccurate dump collection.
>
> Going forward, CPU hotplug or online/offline events are referred as
> CPU/Memory add/remove events.
>
> The current solution to address the above issue involves monitoring the
> CPU/Memory add/remove events in userspace using udev rules and whenever
> there are changes in CPU and memory resources, the entire kdump image
> is loaded again. The kdump image includes kernel, initrd, elfcorehdr,
> FDT, purgatory. Given that only elfcorehdr and FDT get outdated due to
> CPU/Memory add/remove events, reloading the entire kdump image is
> inefficient. More importantly, kdump remains inactive for a substantial
> amount of time until the kdump reload completes.
>
> To address the aforementioned issue, commit 247262756121 ("crash: add
> generic infrastructure for crash hotplug support") added a generic
> infrastructure that allows architectures to selectively update the kdump
> image component during CPU or memory add/remove events within the kernel
> itself.
>
> In the event of a CPU or memory add/remove events, the generic crash
> hotplug event handler, `crash_handle_hotplug_event()`, is triggered. It
> then acquires the necessary locks to update the kdump image and invokes
> the architecture-specific crash hotplug handler,
> `arch_crash_handle_hotplug_event()`, to update the required kdump image
> components.
>
> This patch adds crash hotplug handler for PowerPC and enable support to
> update the kdump image on CPU add/remove events. Support for memory
> add/remove events is added in a subsequent patch with the title
> "powerpc: add crash memory hotplug support"
>
> As mentioned earlier, only the elfcorehdr and FDT kdump image components
> need to be updated in the event of CPU or memory add/remove events.
> However, on PowerPC architecture crash hotplug handler only updates the
> FDT to enable crash hotplug support for CPU add/remove events. Here's
> why.
>
> The elfcorehdr on PowerPC is built with possible CPUs, and thus, it does
> not need an update on CPU add/remove events. On the other hand, the FDT
> needs to be updated on CPU add events to include the newly added CPU. If
> the FDT is not updated and the kernel crashes on a newly added CPU, the
> kdump kernel will fail to boot due to the unavailability of the crashing
> CPU in the FDT. During the early boot, it is expected that the boot CPU
> must be a part of the FDT; otherwise, the kernel will raise a BUG and
> fail to boot. For more information, refer to commit 36ae37e3436b0
> ("powerpc: Make boot_cpuid common between 32 and 64-bit"). Since it is
> okay to have an offline CPU in the kdump FDT, no action is taken in case
> of CPU removal.
>
> There are two system calls, `kexec_file_load` and `kexec_load`, used to
> load the kdump image. Few changes have been made to ensure kernel can
> safely update the FDT of kdump image loaded using both system calls.
>
> For kexec_file_load syscall the kdump image is prepared in kernel. So to
> support an increasing number of CPUs, the FDT is constructed with extra
> buffer space to ensure it can accommodate a possible number of CPU
> nodes. Additionally, a call to fdt_pack (which trims the unused space
> once the FDT is prepared) is avoided if this feature is enabled.
>
> For the kexec_load syscall, the FDT is updated only if the
> KEXEC_CRASH_HOTPLUG_SUPPORT kexec flag is passed to the kernel by
> userspace (kexec tools). When userspace passes this flag to the kernel,
> it indicates that the FDT is built to accommodate possible CPUs, and the
> FDT segment is excluded from SHA calculation, making it safe to update.
>
> The changes related to this feature are kept under the CRASH_HOTPLUG
> config, and it is enabled by default.
>
Looks good.
Acked-by: Hari Bathini <hbathini at linux.ibm.com>
> Signed-off-by: Sourabh Jain <sourabhjain at linux.ibm.com>
> Cc: Akhil Raj <lf32.dev at gmail.com>
> Cc: Andrew Morton <akpm at linux-foundation.org>
> Cc: Aneesh Kumar K.V <aneesh.kumar at kernel.org>
> Cc: Baoquan He <bhe at redhat.com>
> Cc: Borislav Petkov (AMD) <bp at alien8.de>
> Cc: Boris Ostrovsky <boris.ostrovsky at oracle.com>
> Cc: Christophe Leroy <christophe.leroy at csgroup.eu>
> Cc: Dave Hansen <dave.hansen at linux.intel.com>
> Cc: Dave Young <dyoung at redhat.com>
> Cc: David Hildenbrand <david at redhat.com>
> Cc: Greg Kroah-Hartman <gregkh at linuxfoundation.org>
> Cc: Hari Bathini <hbathini at linux.ibm.com>
> Cc: Laurent Dufour <laurent.dufour at fr.ibm.com>
> Cc: Mahesh Salgaonkar <mahesh at linux.ibm.com>
> Cc: Michael Ellerman <mpe at ellerman.id.au>
> Cc: Mimi Zohar <zohar at linux.ibm.com>
> Cc: Naveen N Rao <naveen at kernel.org>
> Cc: Oscar Salvador <osalvador at suse.de>
> Cc: Thomas Gleixner <tglx at linutronix.de>
> Cc: Valentin Schneider <vschneid at redhat.com>
> Cc: Vivek Goyal <vgoyal at redhat.com>
> Cc: kexec at lists.infradead.org
> Cc: x86 at kernel.org
> ---
> arch/powerpc/Kconfig | 4 ++
> arch/powerpc/include/asm/kexec.h | 8 +++
> arch/powerpc/kexec/crash.c | 103 ++++++++++++++++++++++++++++++
> arch/powerpc/kexec/elf_64.c | 3 +-
> arch/powerpc/kexec/file_load_64.c | 17 +++++
> 5 files changed, 134 insertions(+), 1 deletion(-)
>
> diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
> index e377deefa2dc..16d2b20574c4 100644
> --- a/arch/powerpc/Kconfig
> +++ b/arch/powerpc/Kconfig
> @@ -686,6 +686,10 @@ config ARCH_SELECTS_CRASH_DUMP
> depends on CRASH_DUMP
> select RELOCATABLE if PPC64 || 44x || PPC_85xx
>
> +config ARCH_SUPPORTS_CRASH_HOTPLUG
> + def_bool y
> + depends on PPC64
> +
> config FA_DUMP
> bool "Firmware-assisted dump"
> depends on CRASH_DUMP && PPC64 && (PPC_RTAS || PPC_POWERNV)
> diff --git a/arch/powerpc/include/asm/kexec.h b/arch/powerpc/include/asm/kexec.h
> index d9ff4d0e392d..e75970351bcd 100644
> --- a/arch/powerpc/include/asm/kexec.h
> +++ b/arch/powerpc/include/asm/kexec.h
> @@ -135,6 +135,14 @@ static inline void crash_setup_regs(struct pt_regs *newregs,
> ppc_save_regs(newregs);
> }
>
> +#ifdef CONFIG_CRASH_HOTPLUG
> +void arch_crash_handle_hotplug_event(struct kimage *image, void *arg);
> +#define arch_crash_handle_hotplug_event arch_crash_handle_hotplug_event
> +
> +int arch_crash_hotplug_support(struct kimage *image, unsigned long kexec_flags);
> +#define arch_crash_hotplug_support arch_crash_hotplug_support
> +#endif /* CONFIG_CRASH_HOTPLUG */
> +
> extern int crashing_cpu;
> extern void crash_send_ipi(void (*crash_ipi_callback)(struct pt_regs *));
> extern void crash_ipi_callback(struct pt_regs *regs);
> diff --git a/arch/powerpc/kexec/crash.c b/arch/powerpc/kexec/crash.c
> index ef5c2d25ec39..8938a19af12f 100644
> --- a/arch/powerpc/kexec/crash.c
> +++ b/arch/powerpc/kexec/crash.c
> @@ -16,6 +16,7 @@
> #include <linux/delay.h>
> #include <linux/irq.h>
> #include <linux/types.h>
> +#include <linux/libfdt.h>
>
> #include <asm/processor.h>
> #include <asm/machdep.h>
> @@ -392,3 +393,105 @@ void default_machine_crash_shutdown(struct pt_regs *regs)
> if (ppc_md.kexec_cpu_down)
> ppc_md.kexec_cpu_down(1, 0);
> }
> +
> +#ifdef CONFIG_CRASH_HOTPLUG
> +#undef pr_fmt
> +#define pr_fmt(fmt) "crash hp: " fmt
> +
> +/**
> + * get_fdt_index - Loop through the kexec segment array and find
> + * the index of the FDT segment.
> + * @image: a pointer to kexec_crash_image
> + *
> + * Returns the index of FDT segment in the kexec segment array
> + * if found; otherwise -1.
> + */
> +static int get_fdt_index(struct kimage *image)
> +{
> + void *ptr;
> + unsigned long mem;
> + int i, fdt_index = -1;
> +
> + /* Find the FDT segment index in kexec segment array. */
> + for (i = 0; i < image->nr_segments; i++) {
> + mem = image->segment[i].mem;
> + ptr = __va(mem);
> +
> + if (ptr && fdt_magic(ptr) == FDT_MAGIC) {
> + fdt_index = i;
> + break;
> + }
> + }
> +
> + return fdt_index;
> +}
> +
> +/**
> + * update_crash_fdt - updates the cpus node of the crash FDT.
> + *
> + * @image: a pointer to kexec_crash_image
> + */
> +static void update_crash_fdt(struct kimage *image)
> +{
> + void *fdt;
> + int fdt_index;
> +
> + fdt_index = get_fdt_index(image);
> + if (fdt_index < 0) {
> + pr_err("Unable to locate FDT segment.\n");
> + return;
> + }
> +
> + fdt = __va((void *)image->segment[fdt_index].mem);
> +
> + /* Temporarily invalidate the crash image while it is replaced */
> + xchg(&kexec_crash_image, NULL);
> +
> + /* update FDT to reflect changes in CPU resources */
> + if (update_cpus_node(fdt))
> + pr_err("Failed to update crash FDT");
> +
> + /* The crash image is now valid once again */
> + xchg(&kexec_crash_image, image);
> +}
> +
> +int arch_crash_hotplug_support(struct kimage *image, unsigned long kexec_flags)
> +{
> +#ifdef CONFIG_KEXEC_FILE
> + if (image->file_mode)
> + return 1;
> +#endif
> + return kexec_flags & KEXEC_CRASH_HOTPLUG_SUPPORT;
> +}
> +
> +/**
> + * arch_crash_handle_hotplug_event - Handle crash CPU/Memory hotplug events to update the
> + * necessary kexec segments based on the hotplug event.
> + * @image: a pointer to kexec_crash_image
> + * @arg: struct memory_notify handler for memory hotplug case and NULL for CPU hotplug case.
> + *
> + * Update the kdump image based on the type of hotplug event, represented by image->hp_action.
> + * CPU add: Update the FDT segment to include the newly added CPU.
> + * CPU remove: No action is needed, with the assumption that it's okay to have offline CPUs
> + * part of the FDT.
> + * Memory add/remove: No action is taken as this is not yet supported.
> + */
> +void arch_crash_handle_hotplug_event(struct kimage *image, void *arg)
> +{
> + switch (image->hp_action) {
> + case KEXEC_CRASH_HP_REMOVE_CPU:
> + return;
> +
> + case KEXEC_CRASH_HP_ADD_CPU:
> + update_crash_fdt(image);
> + break;
> +
> + case KEXEC_CRASH_HP_REMOVE_MEMORY:
> + case KEXEC_CRASH_HP_ADD_MEMORY:
> + pr_info_once("Crash update is not supported for memory hotplug\n");
> + return;
> + default:
> + pr_warn_once("Unknown hotplug action\n");
> + }
> +}
> +#endif /* CONFIG_CRASH_HOTPLUG */
> diff --git a/arch/powerpc/kexec/elf_64.c b/arch/powerpc/kexec/elf_64.c
> index 6d8951e8e966..214c071c58ed 100644
> --- a/arch/powerpc/kexec/elf_64.c
> +++ b/arch/powerpc/kexec/elf_64.c
> @@ -116,7 +116,8 @@ static void *elf64_load(struct kimage *image, char *kernel_buf,
> if (ret)
> goto out_free_fdt;
>
> - fdt_pack(fdt);
> + if (!IS_ENABLED(CONFIG_CRASH_HOTPLUG) || image->type != KEXEC_TYPE_CRASH)
> + fdt_pack(fdt);
>
> kbuf.buffer = fdt;
> kbuf.bufsz = kbuf.memsz = fdt_totalsize(fdt);
> diff --git a/arch/powerpc/kexec/file_load_64.c b/arch/powerpc/kexec/file_load_64.c
> index 4b94c31e3172..2640a804fcdf 100644
> --- a/arch/powerpc/kexec/file_load_64.c
> +++ b/arch/powerpc/kexec/file_load_64.c
> @@ -30,6 +30,7 @@
> #include <asm/iommu.h>
> #include <asm/prom.h>
> #include <asm/plpks.h>
> +#include <asm/cputhreads.h>
>
> struct umem_info {
> __be64 *buf; /* data buffer for usable-memory property */
> @@ -789,6 +790,9 @@ static unsigned int kdump_extra_fdt_size_ppc64(struct kimage *image)
> unsigned int cpu_nodes, extra_size = 0;
> struct device_node *dn;
> u64 usm_entries;
> +#ifdef CONFIG_CRASH_HOTPLUG
> + unsigned int possible_cpu_nodes;
> +#endif
>
> if (!IS_ENABLED(CONFIG_CRASH_DUMP) || image->type != KEXEC_TYPE_CRASH)
> return 0;
> @@ -816,6 +820,19 @@ static unsigned int kdump_extra_fdt_size_ppc64(struct kimage *image)
> if (cpu_nodes > boot_cpu_node_count)
> extra_size += (cpu_nodes - boot_cpu_node_count) * cpu_node_size();
>
> +#ifdef CONFIG_CRASH_HOTPLUG
> + /*
> + * Make sure enough space is reserved to accommodate possible CPU nodes
> + * in the crash FDT. This allows packing possible CPU nodes which are
> + * not yet present in the system without regenerating the entire FDT.
> + */
> + if (image->type == KEXEC_TYPE_CRASH) {
> + possible_cpu_nodes = num_possible_cpus() / threads_per_core;
> + if (possible_cpu_nodes > cpu_nodes)
> + extra_size += (possible_cpu_nodes - cpu_nodes) * cpu_node_size();
> + }
> +#endif
> +
> return extra_size;
> }
>
More information about the Linuxppc-dev
mailing list