[Skiboot] [PATCH v2 0/3] Support PCI Virtual Device

[Alistair Popple]

On Thu, 8 Sep 2016 03:46:45 PM Gavin Shan wrote:
> On Wed, Sep 07, 2016 at 06:04:16PM +1000, Gavin Shan wrote:
> >On Wed, Sep 07, 2016 at 04:50:10PM +1000, Alistair Popple wrote:
> >>Hi Gavin,
> >>
> >>On Thu, 11 Aug 2016 12:12:35 PM Gavin Shan wrote:
> >>> The patches support PCI virtual device. The NVLinks are presented to 
Linux
> >>> kernel via PCI virtual (emulated) device. As we're going to support 
NVLink2
> >>> pretty soon, we need support PCI virtual device in common layer so that 
it
> >>> can be shared by NVLINK1/2.
> >>> 
> >>> Changelog
> >>> =========
> >>> v1 -> v2:
> >>>    * Rebased to master branch per requested by Stewart. No code changes
> >>>      since v1. The last commit is f102c9d19078 ("Merge skiboot-5.3.1
> >>>      release notes").
> >>>    * Had quick test on Garrison1 and no obvious issues found.
> >>
> >>We should run actual GPU workloads to test any NPU changes (eg. a CUDA 
> >>bandwidth test - let me know if you're not sure how). A simple boot test 
is 
> >>not enough as it won't exercise many of the code paths that have been 
changed 
> >>here. It seems patch 3 breaks something as the nVidia driver no longer 
works 
> >>with nvlink on master - could you please take a look at this? Thanks!
> >>
> >>Stewart - we may need to revert "[PATCH v2 3/3] hw/npu: Use PCI virtual 
> >>device" if we can't get this fixed soon - will let you know.
> >>
> >
> >Yeah, agree. I will talk to you about how to run CUDA bandwidth
> >test tomorrow morning when you have available time. After that,
> >I will investigate what's broken by PATCH[3/3]. I assume the
> >issue is existing on Garrison only.
> >
> 
> Another patch was sent to fix something incorrect in PCI config filter. With
> it's applied, the bandwidth testing succeeded. Alistair, please help confirm
> the result is close to the expected one? It seems we have some performance
> lost and I can sort it out later.

That's strange. There's no obvious reason why these patches should impact 
nvlink bandwidth at all.

> https://patchwork.ozlabs.org/patch/667255/
> 
> ---
> 
> garrison1# 
/home/alistair/NVIDIA_CUDA-7.5_Samples/1_Utilities/bandwidthTest/bandwidthTest 
--memory=pinned
> [CUDA Bandwidth Test] - Starting...
> Running on...
> 
>  Device 0: Graphics Device
>  Quick Mode
> 
>  Host to Device Bandwidth, 1 Device(s)
>  PINNED Memory Transfers
>    Transfer Size (Bytes)	Bandwidth(MB/s)
>    33554432			29467.5
> 
>  Device to Host Bandwidth, 1 Device(s)
>  PINNED Memory Transfers
>    Transfer Size (Bytes)	Bandwidth(MB/s)
>    33554432			21323.7
> 
>  Device to Device Bandwidth, 1 Device(s)
>  PINNED Memory Transfers
>    Transfer Size (Bytes)	Bandwidth(MB/s)
>    33554432			345036.7
> 
> Result = PASS
> 
> NOTE: The CUDA Samples are not meant for performance measurements. Results 
may vary when GPU Boost is enabled.
> 
> Thanks,
> Gavin
>