[SLOF] [PATCH slof] pci: Put non-prefetchable 64bit BARs into 32bit MMIO window

[Alexey Kardashevskiy]

On 26/04/17 01:06, Laurent Vivier wrote:
> On 25/04/2017 15:44, Alexey Kardashevskiy wrote:
>> On 25/04/17 21:00, Laurent Vivier wrote:
>>> On 25/04/2017 12:36, Laurent Vivier wrote:
>>>> On 24/04/2017 05:01, Alexey Kardashevskiy wrote:
>>>>> At the moment 64bit non-prefetchable BARs of devices behind PCI p2p
>>>>> bridge go to a 64bit prefetchable windows which is not correct and
>>>>> causes linux guests to fail to ioremap() such resources.
>>>>>
>>>>> This moves 64bit non-prefetchable BARs 32bit non-prefetchable window.
>>>>>
>>>>> Note that this does not make distinction between P2P and PHB so
>>>>> from now on XHCI BARs will be allocated from 32bit MMIO space.
>>>>> However since most 64bit-MMIO-capable devices have prefetchable BARs,
>>>>> and XHCI BAR is just 4K (so it is unlikely to cause any space problems),
>>>>> this should not affect usual behavior.
>>>>>
>>>>> Signed-off-by: Alexey Kardashevskiy <aik at ozlabs.ru>
>>>>> ---
>>>>>
>>>>> This fixes QEMU's XHCI when it is put on a P2P PCI bridge.
>>>>>
>>>>> There is a little naming confusion as it may look like the patch
>>>>> is changing assignment for all 64bit BAR but it does not as:
>>>>> - "mmio" is used for non-prefetchable memory,
>>>>> - "mem" is used for prefetchable memory.
>>>>> ---
>>>>>  slof/fs/pci-properties.fs | 6 +-----
>>>>>  1 file changed, 1 insertion(+), 5 deletions(-)
>>>>>
>>>>> diff --git a/slof/fs/pci-properties.fs b/slof/fs/pci-properties.fs
>>>>> index e6fd843..8594e5d 100644
>>>>> --- a/slof/fs/pci-properties.fs
>>>>> +++ b/slof/fs/pci-properties.fs
>>>>> @@ -166,11 +166,7 @@
>>>>>  \ Setup a non-prefetchable 64bit BAR and return its size
>>>>>  : assign-mmio64-bar ( bar-addr -- 8 )
>>>>>          dup pci-bar-size-mem64          \ fetch size
>>>>> -        pci-next-mem64 @ 0 = IF          \ Check if we have 64-bit memory range
>>>>> -	    pci-next-mmio
>>>>> -	ELSE
>>>>> -	    pci-next-mem64              \ for board-qemu we will use same range
>>>>> -	THEN
>>>>> +        pci-next-mmio
>>>>>          assign-bar-value64              \ and set it all
>>>>>  ;
>>>>>  
>>>>>
>>>> Tested-by: Laurent Vivier <lvivier at redhat.com>
>>>
>>> This fixes the problem when the BAR is mem, not when it is io.
>>>
>>> I have the same problem as with XHCI as with virtio-blk devices behind a
>>> pci bridge, and the error is:
>>>
>>> ...
>>> PCI: Cannot allocate resource region 0 of device 0000:01:01.0, will remap
>>> ...
>>> virtio-pci 0000:01:01.0: can't enable device: BAR 0 [io  size 0x0040]
>>> not assigned
>>> ...
>>>
>>> And this patch doesn't fix it.
>>
>>
>> Correct, it did not try to fix it. But afaict virtio uses IO in legacy
>> drivers and also implements mmio BARs which are actually used in guest
>> kernels we care about. Or there is a problem and something does not work?
> 
> You're right, adding "disable-legacy=on" to the virtio-blk-pci device
> allows to use the disk and remove the errors from the guest kernel logs.
> 
> What I don't understand is why "disable-legacy=on" fixes the problem
> even without your patch for SLOF...


The patch changes handling of MMIO BARs (up to 64bit of PCI address space)
and virtio problem is caused by an IO BAR (65536 IO ports, x86 legacy) -
this is what "disable-legacy=on" removes.


-- 
Alexey