Rate Limiting

[Stephen Finucane]

On Wed, 2017-05-03 at 23:51 +1000, Andrew Donnellan wrote:
> On 03/05/17 23:15, Stephen Finucane wrote:
> > I spent a couple of hours over the long weekend drafting a patch to
> > add rate limiting to the REST API. It's a customized variant of
> > what DRF provides by default and sets three unstandardized yet
> > widely used headers [1] in responses:
> > 
> >   X-RateLimit-Remaining
> >   X-RateLimit-Limit
> >   X-RateLimit-Reset
> > 
> > As I wrapped this up, I spent a long time trying to determine some
> > sane default limits to set for anonymous users and authenticated
> > users. In doing so, I realized they'd have to be very high because
> > we don't provide anything to minimize the need for polling
> > (webhooks, ETags, etc.) and that any rate limiting would negatively
> > impact REST API users alone, because we don't rate limit the XML-
> > RPC API or the web UI.
> > 
> > Despite my suggestions to the contrary [2], I began to think that
> > maybe, just maybe, we _shouldn't_ include rate limiting in 2.0 and
> > should instead get that release out the door and work on
> > implementing a combined caching (via. ETags) and rate limiting
> > solution for 2.1.
> > 
> > Does this sound rational, or am I talking rubbish? Anyone have any
> > other thoughts on the matter?
> 
> As I once heard a wise person say, "solve the problems you have, not
> the problems you wish you had" - I think we can probably survive
> without rate limiting until there's a non-negligible number of REST
> API consumers out there / someone starts complaining. :)

+1. A very wise person indeed :)

> But when you say "very high" default limits, how high are you
> thinking?

I think I settled on 600/hour for anonymous users, and 60000/hour for
authenticated users. That was completely arbitrary though: GitHub use
60 and 5000 respectively, but they provide both webhook functionality
(to avoid polling) and ETags (to check if anything has changed before
making a full new request).

Stephen