[Skiboot] [PATCH 5/9] npu2: Add NPU2_GPU1_MEM_BAR

[Alistair Popple]

Couple of comments below...

On Fri, 14 Jul 2017 10:39:24 AM Reza Arbab wrote:
> POWER9 DD2 has added a second GPU memory BAR. Use it, but continue to
> program things the old way on DD1 systems.
> 
> Cc: Alistair Popple <alistair at popple.id.au>
> Signed-off-by: Reza Arbab <arbab at linux.vnet.ibm.com>
> ---
>  hw/npu2.c           | 26 +++++++++++++++-----------
>  include/npu2-regs.h |  1 +
>  2 files changed, 16 insertions(+), 11 deletions(-)
> 
> diff --git a/hw/npu2.c b/hw/npu2.c
> index e44fc8a..e6bb3c3 100644
> --- a/hw/npu2.c
> +++ b/hw/npu2.c
> @@ -579,7 +579,7 @@ static int npu2_assign_gmb(struct npu2_dev *ndev)
>  	struct npu2 *p = ndev->npu;
>  	int peers, mode;
>  	uint32_t bdfn;
> -	uint64_t base, size, reg, val, old_val;
> +	uint64_t base, size, reg, val, old_val, gmb;
>  
>  	/* Need to work out number of link peers. This amount to
>  	 * working out the maximum function number. So work start at
> @@ -628,27 +628,31 @@ static int npu2_assign_gmb(struct npu2_dev *ndev)
>  
>  	mode += ndev->bdfn & 0x7;
>  	val = SETFIELD(NPU2_MEM_BAR_MODE, val, mode);
> -	if (NPU2DEV_BRICK(ndev))
> -		val >>= 32;
> +	gmb = NPU2_GPU0_MEM_BAR;
> +
> +	if (NPU2DEV_BRICK(ndev)) {
> +		if (is_p9dd1())
> +			/* In DD1, each brick's GPU is set via the same BAR */
> +			val >>= 32;
> +		else
> +			gmb = NPU2_GPU1_MEM_BAR;
> +	}
> +
>  	reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + NPU2DEV_STACK(ndev),
> -			      NPU2_BLOCK_SM_0,
> -			      NPU2_GPU0_MEM_BAR);
> +			      NPU2_BLOCK_SM_0, gmb);
>  
>  	old_val = npu2_read(p, reg);

We should really mask old_val here before OR'ing it in in case Hostboot or other
system boot software has assigned this to some other value.

>  	val |= old_val;

On DD2 we also need to set GPUx_BAR_POISON for the GPU BARs. I couldn't see it
being set later in the series so we should add it here for DD2.

Thanks.

- Alistair

>  	npu2_write(p, reg, val);
>  	reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + NPU2DEV_STACK(ndev),
> -			      NPU2_BLOCK_SM_1,
> -			      NPU2_GPU0_MEM_BAR);
> +			      NPU2_BLOCK_SM_1, gmb);
>  	npu2_write(p, reg, val);
>  	reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + NPU2DEV_STACK(ndev),
> -			      NPU2_BLOCK_SM_2,
> -			      NPU2_GPU0_MEM_BAR);
> +			      NPU2_BLOCK_SM_2, gmb);
>  	npu2_write(p, reg, val);
>  	reg = NPU2_REG_OFFSET(NPU2_STACK_STCK_0 + NPU2DEV_STACK(ndev),
> -			      NPU2_BLOCK_SM_3,
> -			      NPU2_GPU0_MEM_BAR);
> +			      NPU2_BLOCK_SM_3, gmb);
>  	npu2_write(p, reg, val);
>  
>  	return 0;
> diff --git a/include/npu2-regs.h b/include/npu2-regs.h
> index 4b86f6f..fb85729 100644
> --- a/include/npu2-regs.h
> +++ b/include/npu2-regs.h
> @@ -101,6 +101,7 @@ void npu2_write_mask(struct npu2 *p, uint64_t reg, uint64_t val, uint64_t mask);
>  #define NPU2_PB_EPSILON				0x010
>  #define NPU2_TIMER_CFG				0x018
>  #define NPU2_GPU0_MEM_BAR			0x020
> +#define NPU2_GPU1_MEM_BAR			0x028
>  #define   NPU2_MEM_BAR_SEL_MEM			PPC_BITMASK(0,2)
>  #define   NPU2_MEM_BAR_GROUP			PPC_BITMASK(3,6)
>  #define   NPU2_MEM_BAR_CHIP			PPC_BITMASK(7,9)
>