[Skiboot] [PATCH] phb4/capp: Calculate STQ/DMA read engines based on link-width for PEC2

[Andrew Donnellan]

On 03/07/18 20:24, Vaibhav Jain wrote:
> Presently in CAPI mode the number of STQ/DMA-read engines allocated on
> PEC2 for CAPP is fixed to 6 and 0-30 respectively irrespective of the
> PCI link width. These values are only suitable for x8 cards and
> quickly run out if a x16 card is plugged to a PEC2 attached slot. This
> usually manifests as CAPP reporting TLBI timeout due to these messages
> getting stalled due to insufficient STQs.
> 
> To fix this we update enable_capi_mode() to check if PEC2 chiplet is
> in x16 mode and if yes then we allocate 4/0-47 STQ/DMA-read engines
> for the CAPP traffic.
> 
> Fixes: 37ea3cfdc852("capi: Enable capi mode for PHB4")
> Signed-off-by: Vaibhav Jain <vaibhav at linux.ibm.com>

This should probably head to stable, right?

One minor nitpick below

Reviewed-by: Andrew Donnellan <andrew.donnellan at au1.ibm.com>

> ---
>   hw/phb4.c           | 35 ++++++++++++++++++++++++++---------
>   include/phb4-regs.h |  6 ++++++
>   2 files changed, 32 insertions(+), 9 deletions(-)
> 
> diff --git a/hw/phb4.c b/hw/phb4.c
> index ae584d67..62443d7e 100644
> --- a/hw/phb4.c
> +++ b/hw/phb4.c
> @@ -3918,8 +3918,6 @@ static int64_t enable_capi_mode(struct phb4 *p, uint64_t pe_number,
>   		return OPAL_HARDWARE;
>   	}
>   
> -	/* CAPP Control Register. Enable CAPP Mode */
> -	reg = 0x8000000000000000ULL; /* PEC works in CAPP Mode */
>   	if (p->index == CAPP0_PHB_INDEX) {
>   		/* PBCQ is operating as a x16 stack
>   		 * - The maximum number of engines give to CAPP will be
> @@ -3929,17 +3927,36 @@ static int64_t enable_capi_mode(struct phb4 *p, uint64_t pe_number,
>   		stq_eng = 0x000E000000000000ULL; /* 14 CAPP msg engines */
>   		dma_eng = 0x0000FFFFFFFFFFFFULL; /* 48 CAPP Read machines */
>   	}
> +
>   	if (p->index == CAPP1_PHB_INDEX) {
> -		/* PBCQ is operating as a x8 stack
> -		 * - The maximum number of engines given to CAPP should
> -		 * be 6 and will be assigned in the order of 7 to 2.
> -		 * - 0-30 (Read machines) are available for capp use.
> -		 */
> -		stq_eng = 0x0006000000000000ULL; /* 6 CAPP msg engines */
> -		dma_eng = 0x0000FFFFF00E0000ULL; /* 30 Read machines for CAPP Minus 20-27 for DMA */
> +		/* Check if PEC is in x8 or x16 mode */
> +		xscom_read(p->chip_id, XPEC_PCI2_CPLT_CONF1, &reg);
> +
> +		if ((reg & XPEC_PCI2_IOVALID_MASK) == XPEC_PCI2_IOVALID_X16) {
> +			/* PBCQ is operating as a x16 stack
> +			 * - The maximum number of engines give to CAPP will be
> +			 * 14 and will be assigned in the order of STQ 15 to 2.
> +			 * - 0-47 (Read machines) are available for capp use.
> +			 */
> +			stq_eng = 0x000E000000000000ULL;
> +			dma_eng = 0x0000FFFFFFFFFFFFULL;
> +		} else {
> +
> +			/* PBCQ is operating as a x8 stack
> +			 * - The maximum number of engines given to CAPP should
> +			 * be 6 and will be assigned in the order of 7 to 2.
> +			 * - 0-30 (Read machines) are available for capp use.

is it worth keeping the "minus 20-27 for DMA" comment in here?

> +			 */
> +			stq_eng = 0x0006000000000000ULL;
> +			dma_eng = 0x0000FFFFF00E0000ULL;
> +		}
>   	}
> +
>   	if (capp_eng & CAPP_MIN_STQ_ENGINES)
>   		stq_eng = 0x0002000000000000ULL; /* 2 capp msg engines */
> +
> +	/* CAPP Control Register. Enable CAPP Mode */
> +	reg = 0x8000000000000000ULL; /* PEC works in CAPP Mode */
>   	reg |= stq_eng;
>   	if (capp_eng & CAPP_MAX_DMA_READ_ENGINES)
>   		dma_eng = 0x0000FF0000000000ULL; /* 16 CAPP Read machines */
> diff --git a/include/phb4-regs.h b/include/phb4-regs.h
> index 3f87ddcd..e7a190ee 100644
> --- a/include/phb4-regs.h
> +++ b/include/phb4-regs.h
> @@ -400,6 +400,12 @@
>   #define   XETU_HV_IND_ADDR_AUTOINC		PPC_BIT(2)
>   #define XETU_HV_IND_DATA			0x1
>   
> +
> +/* PCI Chiplet Config Register */
> +#define XPEC_PCI2_CPLT_CONF1			0x000000000F000009ULL
> +#define XPEC_PCI2_IOVALID_MASK			PPC_BITMASK(4, 6)
> +#define XPEC_PCI2_IOVALID_X16			PPC_BIT(4)
> +
>   /*
>    * IODA3 on-chip tables
>    */
> 

-- 
Andrew Donnellan              OzLabs, ADL Canberra
andrew.donnellan at au1.ibm.com  IBM Australia Limited