[Skiboot] [PATCH skiboot v3] npu2: Disable Probe-to-Invalid-Return-Modified-or-Owned snarfing by default
Michael Neuling
mikey at neuling.org
Tue Apr 30 15:57:58 AEST 2019
On Mon, 2019-04-29 at 19:12 +1000, Alexey Kardashevskiy wrote:
> V100 GPUs are known to violate NVLink2 protocol in some cases (one is when
> memory was accessed by the CPU and they by GPU using so called block
> linear mapping) and issue double probes to NPU which can cope with this
> problem only if CONFIG_ENABLE_SNARF_CPM ("disable/enable Probe.I.MO
> snarfing a cp_m") is not set in the CQ_SM Misc Config register #0.
> If the bit is set (which is the case today), NPU issues the machine
> check stop.
>
> The snarfing feature is designed to detect 2 probes in flight and combine
> them into one.
>
> This adds a new "opal-npu2-snarf-cpm" nvram variable which controls
> CONFIG_ENABLE_SNARF_CPM for all NVLinks to prevent the machine check
> stop from happening.
>
> This disables snarfing by default as otherwise a broken GPU driver can
> crash the entire box even when a GPU is passed through to a guest.
> This provides a dial to allow regression tests (might be useful for
> a bare metal). To enable snarfing, the user needs to run:
>
> sudo nvram -p ibm,skiboot --update-config opal-npu2-snarf-cpm=enable
>
> and reboot the host system.
>
> While at this, define macros for register names as well to avoid touching
> same lines over and over again.
>
> Signed-off-by: Alexey Kardashevskiy <aik at ozlabs.ru>
Thanks, I think this looks better.
Mikey
> ---
> Changes:
> v3:
> * reworded the commit log
> * disabled by default as it was in v1
> * instead of living the default value (whatever it is), this explicitly
> enabled snarfing if requested
>
> v2:
> * disable sharfing by default
> * add a comment block
> ---
> include/npu2-regs.h | 14 ++++++++++++
> hw/npu2.c | 56 ++++++++++++++++++++++++++++++++++-----------
> 2 files changed, 57 insertions(+), 13 deletions(-)
>
> diff --git a/include/npu2-regs.h b/include/npu2-regs.h
> index ba10b8eaf88d..61e8ea8615f8 100644
> --- a/include/npu2-regs.h
> +++ b/include/npu2-regs.h
> @@ -791,4 +791,18 @@ void npu2_scom_write(uint64_t gcid, uint64_t scom_base,
> #define L3_PRD_PURGE_TTYPE_MASK PPC_BIT(1) | PPC_BIT(2) |
> PPC_BIT(3) | PPC_BIT(4)
> #define L3_FULL_PURGE 0x0
>
> +/* Config registers for NPU2 */
> +#define NPU_STCK0_CS_SM0_MISC_CONFIG0 0x5011000
> +#define NPU_STCK0_CS_SM1_MISC_CONFIG0 0x5011030
> +#define NPU_STCK0_CS_SM2_MISC_CONFIG0 0x5011060
> +#define NPU_STCK0_CS_SM3_MISC_CONFIG0 0x5011090
> +#define NPU_STCK1_CS_SM0_MISC_CONFIG0 0x5011200
> +#define NPU_STCK1_CS_SM1_MISC_CONFIG0 0x5011230
> +#define NPU_STCK1_CS_SM2_MISC_CONFIG0 0x5011260
> +#define NPU_STCK1_CS_SM3_MISC_CONFIG0 0x5011290
> +#define NPU_STCK2_CS_SM0_MISC_CONFIG0 0x5011400
> +#define NPU_STCK2_CS_SM1_MISC_CONFIG0 0x5011430
> +#define NPU_STCK2_CS_SM2_MISC_CONFIG0 0x5011460
> +#define NPU_STCK2_CS_SM3_MISC_CONFIG0 0x5011490
> +
> #endif /* __NPU2_REGS_H */
> diff --git a/hw/npu2.c b/hw/npu2.c
> index d532c4da3532..0d79d8a50845 100644
> --- a/hw/npu2.c
> +++ b/hw/npu2.c
> @@ -1452,7 +1452,7 @@ static void assign_mmio_bars(uint64_t gcid, uint32_t
> scom, uint64_t reg[2], uint
> int npu2_nvlink_init_npu(struct npu2 *npu)
> {
> struct dt_node *np;
> - uint64_t reg[2], mm_win[2], val;
> + uint64_t reg[2], mm_win[2], val, mask;
>
> /* TODO: Clean this up with register names, etc. when we get
> * time. This just turns NVLink mode on in each brick and should
> @@ -1461,18 +1461,48 @@ int npu2_nvlink_init_npu(struct npu2 *npu)
> *
> * Obviously if the year is now 2020 that didn't happen and you
> * should fix this :-) */
> - xscom_write_mask(npu->chip_id, 0x5011000, PPC_BIT(58), PPC_BIT(58));
> - xscom_write_mask(npu->chip_id, 0x5011030, PPC_BIT(58), PPC_BIT(58));
> - xscom_write_mask(npu->chip_id, 0x5011060, PPC_BIT(58), PPC_BIT(58));
> - xscom_write_mask(npu->chip_id, 0x5011090, PPC_BIT(58), PPC_BIT(58));
> - xscom_write_mask(npu->chip_id, 0x5011200, PPC_BIT(58), PPC_BIT(58));
> - xscom_write_mask(npu->chip_id, 0x5011230, PPC_BIT(58), PPC_BIT(58));
> - xscom_write_mask(npu->chip_id, 0x5011260, PPC_BIT(58), PPC_BIT(58));
> - xscom_write_mask(npu->chip_id, 0x5011290, PPC_BIT(58), PPC_BIT(58));
> - xscom_write_mask(npu->chip_id, 0x5011400, PPC_BIT(58), PPC_BIT(58));
> - xscom_write_mask(npu->chip_id, 0x5011430, PPC_BIT(58), PPC_BIT(58));
> - xscom_write_mask(npu->chip_id, 0x5011460, PPC_BIT(58), PPC_BIT(58));
> - xscom_write_mask(npu->chip_id, 0x5011490, PPC_BIT(58), PPC_BIT(58));
> +
> + val = PPC_BIT(58);
> + mask = PPC_BIT(58) | /* CONFIG_NVLINK_MODE */
> + PPC_BIT(40); /* CONFIG_ENABLE_SNARF_CPM */
> +
> + /*
> + * V100 GPUs are known to violate NVLink2 protocol if some GPU memory
> + * mapped by a CPU was also "linear-block" mapped by a GPU. When this
> + * happens, it breaks the NPU2 cache coherency state machine and
> + * it throws machine checkstop. Disabling snarfing fixes this so let's
> + * disable it by default.
> + */
> + if (nvram_query_eq("opal-npu2-snarf-cpm", "enable")) {
> + prlog(PR_WARNING, "NPU2#%d: enabling Probe.I.MO snarfing, a bad
> GPU driver may crash the system!\n",
> + npu->index);
> + val |= PPC_BIT(40); /* CONFIG_ENABLE_SNARF_CPM */
> + }
> +
> + xscom_write_mask(npu->chip_id, NPU_STCK0_CS_SM0_MISC_CONFIG0,
> + val, mask);
> + xscom_write_mask(npu->chip_id, NPU_STCK0_CS_SM1_MISC_CONFIG0,
> + val, mask);
> + xscom_write_mask(npu->chip_id, NPU_STCK0_CS_SM2_MISC_CONFIG0,
> + val, mask);
> + xscom_write_mask(npu->chip_id, NPU_STCK0_CS_SM3_MISC_CONFIG0,
> + val, mask);
> + xscom_write_mask(npu->chip_id, NPU_STCK1_CS_SM0_MISC_CONFIG0,
> + val, mask);
> + xscom_write_mask(npu->chip_id, NPU_STCK1_CS_SM1_MISC_CONFIG0,
> + val, mask);
> + xscom_write_mask(npu->chip_id, NPU_STCK1_CS_SM2_MISC_CONFIG0,
> + val, mask);
> + xscom_write_mask(npu->chip_id, NPU_STCK1_CS_SM3_MISC_CONFIG0,
> + val, mask);
> + xscom_write_mask(npu->chip_id, NPU_STCK2_CS_SM0_MISC_CONFIG0,
> + val, mask);
> + xscom_write_mask(npu->chip_id, NPU_STCK2_CS_SM1_MISC_CONFIG0,
> + val, mask);
> + xscom_write_mask(npu->chip_id, NPU_STCK2_CS_SM2_MISC_CONFIG0,
> + val, mask);
> + xscom_write_mask(npu->chip_id, NPU_STCK2_CS_SM3_MISC_CONFIG0,
> + val, mask);
>
> xscom_write_mask(npu->chip_id, 0x50110c0, PPC_BIT(53), PPC_BIT(53));
> xscom_write_mask(npu->chip_id, 0x50112c0, PPC_BIT(53), PPC_BIT(53));
More information about the Skiboot
mailing list