[Skiboot] [PATCH 1/2] phb4: Reallocate PEC2 DMA-Read engines to improve GPU-Direct bandwidth

[Andrew Donnellan]

On 19/07/18 19:48, Stewart Smith wrote:
> Vaibhav Jain <vaibhav at linux.ibm.com> writes:
>> We reallocate additional 16/8 DMA-Read engines allocated to stack0/1
>> on PEC2 respectively. This is needed to improve bandwidth available to
>> the Mellanox CX5 adapter when trying to read GPU memory (GPU-Direct).
>>
>> If kernel cxl driver indicates a request to allocate maximum possible
>> DMA read engines when calling enable_capi_mode() and card is attached
>> to PEC2/stack0 slot then we assume its a Mellanox CX5 adapter. We then
>> allocate additional 16/8 extra DMA read engines to stack0 and stack1
>> respectively on PEC2. This is done by populating the
>> XPEC_PCI_PRDSTKOVR and XPEC_NEST_READ_STACK_OVERRIDE as suggested by
>> the h/w team.
>>
>> Signed-off-by: Christophe Lombard <clombard at linux.vnet.ibm.com>
>> Signed-off-by: Vaibhav Jain <vaibhav at linux.ibm.com>
>> ---
>>   hw/phb4.c           | 40 +++++++++++++++++++++++++++++++++++++---
>>   include/phb4-regs.h |  2 ++
>>   2 files changed, 39 insertions(+), 3 deletions(-)
> 
> thanks, and thanks for addressing things Andrew and I brought up in that
> random internal Slack thread to get this ready to go in so quickly.
> 
> Merged to master as of 5690c5a8980faf9e528df65dd95535e21c2c868f
> and to 6.0.x as of b50653f4c4707f3406269573bc421df3cffce950
> 
> I fixed the obvious nitpicks in the doc with merging... although I
> didn't do the "nor on P8 for set_mode_pcie" as I'm not 100% sure of that
> and wanted to check (I thought we did that... or maybe we just *think*
> we can and reset things appropriately in fast-reboot - or I'm just plain
> wrong and shouldn't be allowed near computers).
> 

Thanks.

As for pcie mode on p8, see phb3.c:3695. To switch back, we have to do a 
CRESET on the PHB instead.

-- 
Andrew Donnellan              OzLabs, ADL Canberra
andrew.donnellan at au1.ibm.com  IBM Australia Limited