[Patchwork-maintainers] patchwork.ozlabs.org downtime for maintenance - 15/16 August

[Stephen Finucane]

On Mon, 2024-01-22 at 15:52 +0800, Jeremy Kerr wrote:
> Hi all,
> > I'll try and get to that over the weekend.
> 
> Looks like the heaviest database load is due to API request to the
> global patches view, which is a bit of an odd use-case; that all
> appears to be mostly spider traffic.

Just as an aside, many if not all of the performance issues related to this API
in particular should be resolved in the 3.0 release, owing to the removal of the
Submission table. The DPDK folks are running 3.x in production for a couple of
months now (at https://patches.dpdk.org/) and I'm only aware of one minor issue
[1] that they've encountered. Could be worth lining up the upgrade at some
point...

Cheers,
Stephen

PS: URLs API v2.0 will be almost entirely project-oriented (e.g.
'/project/{projectID}/patches'), but I haven't got there yet.

[1] https://github.com/getpatchwork/patchwork/issues/556

> 
> Konstantin: I'm not sure your new index would help in that case, we're
> not looking up delegates for those views.
> 
> Looking through the access logs, there seem to be three clients that
> are causing around 40-50% of patchwork load:
> 
>  - one IP from an "Alibaba Cloud HK" AS, various UAs
>  - one IP from a Red Hat AS, curl/7.61.1 UA
>  - the Bytedance "Bytespider" UA
> 
> All three seem to be scraping the patchwork site.
> 
> I have blocked all three for now, but it would be worthwhile setting up
> a more fair robots.txt and/or a reasonable ratelimit for the latter
> case.
> 
> If anyone knows what might be up with that Red Hat crawler, please get
> in touch with me.
> 
> I'll keep an eye on things here; there's still likely a bunch of
> potential configuration optimisation we can do too. Let me know if your
> observations change though.
> 
> Cheers,
> 
> 
> Jeremy
> _______________________________________________
> Patchwork mailing list
> Patchwork at lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/patchwork