CONFIG_PPC_VAS depends on 64k pages...?

[Will Springer]

On Tuesday, December 1, 2020 5:16:51 AM PST Bulent Abali wrote:
> I don't know anything about VAS page size requirements in the kernel.  I
> checked the user compression library and saw that we do a sysconf to
> get the page size; so the library should be immune to page size by
> design. But it wouldn't surprise me if a 64KB constant is inadvertently
> hardcoded somewhere else in the library.  Giving heads up to Tulio and
> Raphael who are owners of the github repo.
> 
> https://github.com/libnxz/power-gzip/blob/master/lib/nx_zlib.c#L922
> 
> If we got this wrong in the library it might manifest itself as an error
> message of the sort "excessive page faults".  The library must touch
> pages ahead to make them present in the memory; occasional page faults
> is acceptable. It will retry.

Hm, good to know. As I said I haven't noticed any problems so far, over a 
few different days of testing. My change is now in the Void Linux kernel 
package, and is working for others as well (including the Void maintainer 
Daniel/q66 who I CC'd initially).

> 
> Bulent
> 
> 
> 
> 
> From:        "Sukadev Bhattiprolu" <sukadev at linux.ibm.com>
> To:        "Christophe Leroy" <christophe.leroy at csgroup.eu>
> Cc:        "Will Springer" <skirmisher at protonmail.com>,
> linuxppc-dev at lists.ozlabs.org, daniel at octaforge.org, Bulent
> Abali/Watson/IBM at IBM, haren at linux.ibm.com Date:        12/01/2020 12:53
> AM
> Subject:        Re: CONFIG_PPC_VAS depends on 64k pages...?
> 
> Christophe Leroy [christophe.leroy at csgroup.eu] wrote:
> > Hi,
> > 
> > Le 19/11/2020 à 11:58, Will Springer a écrit :
> > > I learned about the POWER9 gzip accelerator a few months ago when
> > > the
> > > support hit upstream Linux 5.8. However, for some reason the Kconfig
> > > dictates that VAS depends on a 64k page size, which is problematic
> > > as I
> > > run Void Linux, which uses a 4k-page kernel.
> > > 
> > > Some early poking by others indicated there wasn't an obvious page
> > > size
> > > dependency in the code, and suggested I try modifying the config to
> > > switch it on. I did so, but was stopped by a minor complaint of an
> > > "unexpected DT configuration" by the VAS code. I wasn't equipped to
> > > figure out exactly what this meant, even after finding the
> > > offending condition, so after writing a very drawn-out forum post
> > > asking for help, I dropped the subject.
> > > 
> > > Fast forward to today, when I was reminded of the whole thing again,
> > > and decided to debug a bit further. Apparently the VAS platform
> > > device (derived from the DT node) has 5 resources on my 4k kernel,
> > > instead of 4 (which evidently works for others who have had success
> > > on 64k kernels). I have no idea what this means in practice (I
> > > don't know how to introspect it), but after making a tiny patch[1],
> > > everything came up smoothly and I was doing blazing-fast gzip
> > > (de)compression in no time.
> > > 
> > > Everything seems to work fine on 4k pages. So, what's up? Are there
> > > pitfalls lurking around that I've yet to stumble over? More
> > > reasonably,
> > > I'm curious as to why the feature supposedly depends on 64k pages,
> > > or if there's anything else I should be concerned about.
> 
> Will,
> 
> The reason I put in that config check is because we were only able to
> test 64K pages at that point.
> 
> It is interesting that it is working for you. Following code in skiboot
> https://github.com/open-power/skiboot/blob/master/hw/vas.cshould
> restrict it to 64K pages. IIRC there is also a corresponding change in
> some NX registers that should also be configured to allow 4K pages. 

Huh, that is interesting indeed. As far as the kernel code, the only thing 
specific to 64k pages I could find was in [1], where 
VAS_XLATE_LPCR_PAGE_SIZE is set. There is also NX_PAGE_SIZE in drivers/
crypto/nx/nx.h, which is set to 4096, but I don't know if that's related to 
kernel page size at all. Without a better idea of the code base, I didn't
examine more thoroughly.

[1]: https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git/tree/arch/powerpc/platforms/powernv/vas-window.c#n293

>                 static int init_north_ctl(struct proc_chip *chip)
>                 {
>                                  uint64_t val = 0ULL;
> 
>                                  val = SETFIELD(VAS_64K_MODE_MASK, val,
> true); val = SETFIELD(VAS_ACCEPT_PASTE_MASK, val, true); val =
> SETFIELD(VAS_ENABLE_WC_MMIO_BAR, val, true); val =
> SETFIELD(VAS_ENABLE_UWC_MMIO_BAR, val, true); val =
> SETFIELD(VAS_ENABLE_RMA_MMIO_BAR, val, true);
> 
>                                  return vas_scom_write(chip,
> VAS_MISC_N_CTL, val); }
> 
> I am copying Bulent Albali and Haren Myneni who have been working with
> VAS/NX for their thoughts/experience.

Thanks for this and for your input, by the way.

> 
> > Maybe ask Sukadev who did the implementation and is maintaining it ?
> > 
> > > I do have to say I'm quite satisfied with the results of the NX
> > > accelerator, though. Being able to shuffle data to a RaptorCS box
> > > over gigE and get compressed data back faster than most software
> > > gzip could ever hope to achieve is no small feat, let alone the
> > > instantaneous results locally.> > 
> > > :)
> > > 
> > > Cheers,
> > > Will Springer [she/her]
> > > 
> > > [1]:
> > > https://github.com/Skirmisher/void-packages/blob/vas-4k-pages/srcpkgs/linux5.9/patches/ppc-vas-on-4k.patch
> > Christophe

Will [she/her]