[Patchwork-maintainers] patchwork.ozlabs.org downtime for maintenance - 15/16 August

[Aaron Conole]

Ilya Maximets <i.maximets at ovn.org> writes:

> On 1/22/24 14:59, Jeremy Kerr wrote:
>> Hi Ilya,
>> 
>>> We have a robot running in the RH network that pushes checks and
>>> updates statuses for openvswitch and ovn projects, but it shouldn't
>>> really make any "global patches view" types of requests.  All the
>>> patches it looks at supposed to be only from these two projects.  And
>>> it should not make more than one concurrent request.
>> 
>> OK, that looks like it then; this isn't one of the spiders crawling
>> through *all* patches (and does seem to be contained to OVS & OVN) but
>> it's certainly a major contributor to load.
>> 
>> It seems to be re-requesting the same view hundreds of times. From one
>> day's worth of log, the top 10 URLs from that IP:
>>    
>>        399 /api/patches/1887072/checks/
>>        285 /api/patches/1888116/checks/
>>        285 /api/patches/1888115/checks/
>>        285 /api/patches/1888111/checks/
>>        228 /api/patches/1888114/checks/
>>        228 /api/patches/1888112/checks/
>>        228 /api/patches/1887464/checks/
>>        228 /api/patches/1887463/checks/
>>        228 /api/patches/1884952/checks/
>>        228 /api/patches/1884950/checks/
>> 
>> - totalling 43,786 requests for that day.
>
> Ugh.  Yeah, the checks/ requests are definitely something we can improve.
> Aaron is working on removing vast majority of this type of requests as we
> speak.  Hopefully, that will be done soon.
>
> Do you think it'll be fine to unban the robot once it doesn't run that
> many requests on the checks/ API in particular?  (I expect the number of
> requests to be less than a 100-ish per day after the fix.)

Yes - we are now going to store the checks that have already been
submitted (I'm going to try and pre-load the requests).

That should significantly reduce the amount of churn through the
requests.

> Robot will also have an updated UA, so it will be easier to identify in
> case of any issues in the future.

Yes, this change was already completed.

I'm testing out things right now.  When it is reeady, I'll post a series
and CC you.

>> 
>>> Could you provide some examples of requests that are heavy (maybe
>>> off-list), so we can take a look?
>> 
>> I can send you a log over a day if that's helpful.
>
> That might be useful, thanks!
>
>> 
>> Cheers,
>> 
>> 
>> Jeremy