[PATCH] Implement list filtering
Don Zickus
dzickus at redhat.com
Mon Jan 29 22:09:16 AEDT 2018
On Thu, Jan 25, 2018 at 05:57:47PM +1100, Daniel Axtens wrote:
> vkabatov at redhat.com writes:
>
> > From: Veronika Kabatova <vkabatov at redhat.com>
> >
> > Sometimes, multiple projects reside at the same mailing list. So far,
> > Patchwork only allowed a single project per mailing list, which made it
> > impossible for these projects to use Patchwork (unless they did some
> > dirty hacks).
> >
> > Add a new property `subject_match` to projects and implement filtering
> > on (list_id, subject_match) match instead of solely list_id. Instance
> > admin can specify a regex on a per-project basis when the project is
> > created.
>
Hi Daniel,
Thanks for the reply.
> (I haven't thought a great deal about this, so don't take this as a nack)
>
> Two general thoughts:
>
> - netdev has several projects sharing a mailing list and they seem to
> get along fine...
>
> - I am pretty sure Snowpatch deals with this somehow - Snowpatch does
> CI on patchwork, so they need to know what project is used. Andrew,
> Russell - how do you deal with this there?
I am not familiar with how they do their parsing.
>
> I suppose to put a finer point on it - what is your usecase here? What
> are you trying to achieve, and can we help you do that in a way that
> requires smaller changes to Patchwork, and is less fragile? (For example
> this is going to break if someone mis-spells the keyword you're looking
> for in the subject_match.)
So here is our use case. Internally at Red Hat we use one mailing list to
post all of our kernel patches. However, being a distro company, patches
can be applied to either RHEL-6,7,X. For the last 15 years we have always
used the method:
[RHEL-7 PATCH] instead of [PATCH].
The project inside the []s has been what we filter through our regex. Is it
prone to human typos? Yes. Most folks have stuck this in the .git/config
subjectprefix option. That limited the typos. It isn't perfect.
We have been using a hacked up PatchWork1 for the last 7 or so years. This
is one of those features we need (or something to solve our problem) if we
want to migrate to a PatchWork2 instance.
I hope that adds a little bit of context on our thinking. Thoughts?
Cheers,
Don
>
> Regards,
> Daniel
>
> >
> > Signed-off-by: Veronika Kabatova <vkabatov at redhat.com>
> > ---
> > docs/api.yaml | 3 +++
> > docs/deployment/management.rst | 4 +--
> > patchwork/api/project.py | 5 ++--
> > .../0021_add_subject_match_to_project.py | 29 ++++++++++++++++++++++
> > patchwork/models.py | 9 ++++++-
> > patchwork/parser.py | 19 +++++++++++++-
> > .../notes/list-filtering-4643d98b4064367a.yaml | 10 ++++++++
> > 7 files changed, 73 insertions(+), 6 deletions(-)
> > create mode 100644 patchwork/migrations/0021_add_subject_match_to_project.py
> > create mode 100644 releasenotes/notes/list-filtering-4643d98b4064367a.yaml
> >
> > diff --git a/docs/api.yaml b/docs/api.yaml
> > index 3e79f0b..3373226 100644
> > --- a/docs/api.yaml
> > +++ b/docs/api.yaml
> > @@ -374,6 +374,9 @@ definitions:
> > list_id:
> > type: string
> > description: Mailing list identifier for project.
> > + subject_match:
> > + type: string
> > + description: Regex used for email filtering.
> > list_email:
> > type: string
> > description: Mailing list email address for project.
> > diff --git a/docs/deployment/management.rst b/docs/deployment/management.rst
> > index c50b7b6..870d7ee 100644
> > --- a/docs/deployment/management.rst
> > +++ b/docs/deployment/management.rst
> > @@ -63,7 +63,7 @@ due to, for example, an outage.
> > .. option:: --list-id <list-id>
> >
> > mailing list ID. If not supplied, this will be extracted from the mail
> > - headers.
> > + headers. Don't use this option if you require filtering based on subjects.
> >
> > .. option:: infile
> >
> > @@ -88,7 +88,7 @@ the :ref:`deployment installation guide <deployment-parsemail>`.
> > .. option:: --list-id <list-id>
> >
> > mailing list ID. If not supplied, this will be extracted from the mail
> > - headers.
> > + headers. Don't use this option if you require filtering based on subjects.
> >
> > .. option:: infile
> >
> > diff --git a/patchwork/api/project.py b/patchwork/api/project.py
> > index 446c473..597f605 100644
> > --- a/patchwork/api/project.py
> > +++ b/patchwork/api/project.py
> > @@ -39,8 +39,9 @@ class ProjectSerializer(HyperlinkedModelSerializer):
> > class Meta:
> > model = Project
> > fields = ('id', 'url', 'name', 'link_name', 'list_id', 'list_email',
> > - 'web_url', 'scm_url', 'webscm_url', 'maintainers')
> > - read_only_fields = ('name', 'maintainers')
> > + 'web_url', 'scm_url', 'webscm_url', 'maintainers',
> > + 'subject_match')
> > + read_only_fields = ('name', 'maintainers', 'subject_match')
> > extra_kwargs = {
> > 'url': {'view_name': 'api-project-detail'},
> > }
> > diff --git a/patchwork/migrations/0021_add_subject_match_to_project.py b/patchwork/migrations/0021_add_subject_match_to_project.py
> > new file mode 100644
> > index 0000000..6066266
> > --- /dev/null
> > +++ b/patchwork/migrations/0021_add_subject_match_to_project.py
> > @@ -0,0 +1,29 @@
> > +# -*- coding: utf-8 -*-
> > +# Generated by Django 1.10.8 on 2018-01-19 18:16
> > +from __future__ import unicode_literals
> > +
> > +from django.db import migrations, models
> > +
> > +
> > +class Migration(migrations.Migration):
> > +
> > + dependencies = [
> > + ('patchwork', '0020_tag_show_column'),
> > + ]
> > +
> > + operations = [
> > + migrations.AddField(
> > + model_name='project',
> > + name='subject_match',
> > + field=models.CharField(blank=True, default=b'', help_text=b'Regex to match the subject against if only part of emails sent to the list belongs to this project. Will be used with IGNORECASE and MULTILINE flags. If rules for more projects match the first one returned from DB is chosen.', max_length=64),
> > + ),
> > + migrations.AlterField(
> > + model_name='project',
> > + name='listid',
> > + field=models.CharField(max_length=255),
> > + ),
> > + migrations.AlterUniqueTogether(
> > + name='project',
> > + unique_together=set([('listid', 'subject_match')]),
> > + ),
> > + ]
> > diff --git a/patchwork/models.py b/patchwork/models.py
> > index 11886f1..907707f 100644
> > --- a/patchwork/models.py
> > +++ b/patchwork/models.py
> > @@ -71,8 +71,14 @@ class Project(models.Model):
> >
> > linkname = models.CharField(max_length=255, unique=True)
> > name = models.CharField(max_length=255, unique=True)
> > - listid = models.CharField(max_length=255, unique=True)
> > + listid = models.CharField(max_length=255)
> > listemail = models.CharField(max_length=200)
> > + subject_match = models.CharField(
> > + max_length=64, blank=True, default='', help_text='Regex to match the '
> > + 'subject against if only part of emails sent to the list belongs to '
> > + 'this project. Will be used with IGNORECASE and MULTILINE flags. If '
> > + 'rules for more projects match the first one returned from DB is '
> > + 'chosen.')
> >
> > # url metadata
> >
> > @@ -100,6 +106,7 @@ class Project(models.Model):
> > return self.name
> >
> > class Meta:
> > + unique_together = (('listid', 'subject_match'),)
> > ordering = ['linkname']
> >
> >
> > diff --git a/patchwork/parser.py b/patchwork/parser.py
> > index ac7dc5f..015f709 100644
> > --- a/patchwork/parser.py
> > +++ b/patchwork/parser.py
> > @@ -157,9 +157,25 @@ def find_project_by_id(list_id):
> > project = Project.objects.get(listid=list_id)
> > except Project.DoesNotExist:
> > logger.debug("'%s' if not a valid project list-id", list_id)
> > + except Project.MultipleObjectsReturned:
> > + logger.debug("Multiple projects with list-id '%s' found", list_id)
> > return project
> >
> >
> > +def find_project_by_id_and_subject(list_id, subject):
> > + """Find a `project` object based on `list_id` and subject prefix match."""
> > + projects = Project.objects.filter(listid=list_id)
> > + for project in projects:
> > + if (not project.subject_match or
> > + re.search(project.subject_match, subject,
> > + re.MULTILINE | re.IGNORECASE)):
> > + return project
> > +
> > + logger.debug("No project to match email with list-id '%s', subject '%s' "
> > + "found", list_id, subject)
> > + return None
> > +
> > +
> > def find_project_by_header(mail):
> > project = None
> > listid_res = [re.compile(r'.*<([^>]+)>.*', re.S),
> > @@ -181,7 +197,8 @@ def find_project_by_header(mail):
> >
> > listid = match.group(1)
> >
> > - project = find_project_by_id(listid)
> > + project = find_project_by_id_and_subject(listid,
> > + mail.get('Subject'))
> > if project:
> > break
> >
> > diff --git a/releasenotes/notes/list-filtering-4643d98b4064367a.yaml b/releasenotes/notes/list-filtering-4643d98b4064367a.yaml
> > new file mode 100644
> > index 0000000..21c1680
> > --- /dev/null
> > +++ b/releasenotes/notes/list-filtering-4643d98b4064367a.yaml
> > @@ -0,0 +1,10 @@
> > +---
> > +features:
> > + - |
> > + Allow list filtering into multiple projects (and email dropping) based on
> > + subject prefixes. Disabled by default, enable by specifying a regular
> > + expression which needs to be matched in the subject on a per-project basis
> > + (field `subject_match`).
> > +api:
> > + - |
> > + Expose `subject_match` in REST API.
> > --
> > 2.13.6
> >
> > _______________________________________________
> > Patchwork mailing list
> > Patchwork at lists.ozlabs.org
> > https://lists.ozlabs.org/listinfo/patchwork
> _______________________________________________
> Patchwork mailing list
> Patchwork at lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/patchwork
More information about the Patchwork
mailing list