[PATCH 0/4] Add 'Event.payload' field

[Stephen Finucane]

On Wed, 2018-03-28 at 10:02 +1100, Daniel Axtens wrote:
> Stephen Finucane <stephen at that.guru> writes:
> 
> Hi Stephen,
> 
> > The '/event' API endpoint is really slow due to the amount of JOINs
> > necessary to retrieve records from the database. Resolve this by
> > a static JSON representation of any embedded data in the database. This
> > has some disadvantages, noted in the patches, but the performance
> > improvement is huge and users will not notice a thing.
> > 
> > As a necessary side-effect of this change, events now require the REST
> > API be enabled. To be honest, this should have been the case from day 1
> > as events are currently only exposed over this API.
> 
> How does perfomance of this compare to
> https://patchwork.ozlabs.org/project/patchwork/list/?series=35029 ?

I've no idea. I know that it removes all but one (I think) JOIN so in
theory it should massively improve performance. It's impossible for me
to test this stuff as I simply can't get my hands on a big enough
archive to test it. If you have an idea on how to test this, I'd
appreciate the help.

> This is a much bigger change than that series, so if the performance
> impact is similar perhaps the smaller, migration-free change would be
> preferable?

>From what I'm hearing, these large JOINs are generally killing us and
only get worse the larger the instance is. I was envisioning the above
series as something that would fix performance for 2.0, while this as a
longer term solution than the above series that could be applied for
2.1. As noted above though, I need some way to test this so any help
here would be appreciated.

Stephen

> Regards,
> Daniel
> 
> > 
> > Stephen Finucane (4):
> >   REST: Support embedded serializers without context
> >   signals: Only enable events when REST API enabled
> >   models: Migrate event fields to JSON field
> >   REST: Only fetch required fields event filtering
> > 
> >  patchwork/api/embedded.py                          |  60 ++++++++---
> >  patchwork/api/event.py                             | 110 +++++++++++++--------
> >  patchwork/api/filters.py                           |   7 ++
> >  patchwork/fields.py                                |  32 ++++++
> >  .../migrations/0025_add_event_payload_field.py     |  21 ++++
> >  ...26_migrate_data_from_event_fields_to_payload.py |  63 ++++++++++++
> >  .../migrations/0027_remove_old_event_fields.py     |  34 +++++++
> >  patchwork/models.py                                |  28 +-----
> >  patchwork/signals.py                               |  70 +++++++++++--
> >  patchwork/tests/test_events.py                     | 110 +++++++++++++--------
> >  .../events-require-rest-api-47eab4a3be745f75.yaml  |   5 +
> >  11 files changed, 413 insertions(+), 127 deletions(-)
> >  create mode 100644 patchwork/migrations/0025_add_event_payload_field.py
> >  create mode 100644 patchwork/migrations/0026_migrate_data_from_event_fields_to_payload.py
> >  create mode 100644 patchwork/migrations/0027_remove_old_event_fields.py
> >  create mode 100644 releasenotes/notes/events-require-rest-api-47eab4a3be745f75.yaml
> > 
> > -- 
> > 2.14.3
> > 
> > _______________________________________________
> > Patchwork mailing list
> > Patchwork at lists.ozlabs.org
> > https://lists.ozlabs.org/listinfo/patchwork