[Skiboot] [PATCH v2 0/3] Support PCI Virtual Device

[Gavin Shan]

On Wed, Sep 07, 2016 at 06:04:16PM +1000, Gavin Shan wrote:
>On Wed, Sep 07, 2016 at 04:50:10PM +1000, Alistair Popple wrote:
>>Hi Gavin,
>>
>>On Thu, 11 Aug 2016 12:12:35 PM Gavin Shan wrote:
>>> The patches support PCI virtual device. The NVLinks are presented to Linux
>>> kernel via PCI virtual (emulated) device. As we're going to support NVLink2
>>> pretty soon, we need support PCI virtual device in common layer so that it
>>> can be shared by NVLINK1/2.
>>> 
>>> Changelog
>>> =========
>>> v1 -> v2:
>>>    * Rebased to master branch per requested by Stewart. No code changes
>>>      since v1. The last commit is f102c9d19078 ("Merge skiboot-5.3.1
>>>      release notes").
>>>    * Had quick test on Garrison1 and no obvious issues found.
>>
>>We should run actual GPU workloads to test any NPU changes (eg. a CUDA 
>>bandwidth test - let me know if you're not sure how). A simple boot test is 
>>not enough as it won't exercise many of the code paths that have been changed 
>>here. It seems patch 3 breaks something as the nVidia driver no longer works 
>>with nvlink on master - could you please take a look at this? Thanks!
>>
>>Stewart - we may need to revert "[PATCH v2 3/3] hw/npu: Use PCI virtual 
>>device" if we can't get this fixed soon - will let you know.
>>
>
>Yeah, agree. I will talk to you about how to run CUDA bandwidth
>test tomorrow morning when you have available time. After that,
>I will investigate what's broken by PATCH[3/3]. I assume the
>issue is existing on Garrison only.
>

Another patch was sent to fix something incorrect in PCI config filter. With
it's applied, the bandwidth testing succeeded. Alistair, please help confirm
the result is close to the expected one? It seems we have some performance
lost and I can sort it out later.

https://patchwork.ozlabs.org/patch/667255/

---

garrison1# /home/alistair/NVIDIA_CUDA-7.5_Samples/1_Utilities/bandwidthTest/bandwidthTest --memory=pinned
[CUDA Bandwidth Test] - Starting...
Running on...

 Device 0: Graphics Device
 Quick Mode

 Host to Device Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)	Bandwidth(MB/s)
   33554432			29467.5

 Device to Host Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)	Bandwidth(MB/s)
   33554432			21323.7

 Device to Device Bandwidth, 1 Device(s)
 PINNED Memory Transfers
   Transfer Size (Bytes)	Bandwidth(MB/s)
   33554432			345036.7

Result = PASS

NOTE: The CUDA Samples are not meant for performance measurements. Results may vary when GPU Boost is enabled.

Thanks,
Gavin