> Patch > - hide > - | > - - >download patch > - | > - - >download mbox > +

> + + class="btn btn-default" role="button" data-toggle="tooltip" I have grepped and googled and I cannot find anything in this project that would make use of the data-toggle attribute. I also couldn't observe any difference in behaviour without it. Am I missing something? Apart from that: Reviewed-by: Daniel Axtens Regards, Daniel > + title="Download patch diff">diff > + + class="btn btn-default" role="button" data-toggle="tooltip" > + title="Download patch mbox">mbox > +

>  {{ submission|patchsyntax }}
> -- 
> 2.9.3
>
> _______________________________________________
> Patchwork mailing list
> Patchwork at lists.ozlabs.org
> https://lists.ozlabs.org/listinfo/patchwork

From dja at axtens.net  Mon Apr  3 07:25:24 2017
From: dja at axtens.net (Daniel Axtens)
Date: Mon, 03 Apr 2017 07:25:24 +1000
Subject: [PATCH v4 9/9] templates: Add "download series" links
In-Reply-To: <20170310173807.13407-10-stephen@that.guru>
References: <20170310173807.13407-1-stephen@that.guru>
 <20170310173807.13407-10-stephen@that.guru>
Message-ID: <87a87ymri3.fsf@possimpible.ozlabs.ibm.com>

Stephen Finucane  writes:

> This will allow users to download entire series including the current
> patch. Since a patch can belong to many series a dropdown is used.
>
> Signed-off-by: Stephen Finucane 
> ---
> v4:
> - Don't use a dropdown when only a single series is available
> v3:
> - Add unit tests
> ---
>  patchwork/templates/patchwork/submission.html | 16 ++++++++++++++++
>  patchwork/tests/test_detail.py                | 16 ++++++++++++++++
>  2 files changed, 32 insertions(+)
>
> diff --git a/patchwork/templates/patchwork/submission.html b/patchwork/templates/patchwork/submission.html
> index 66c1159..e6b78d8 100644
> --- a/patchwork/templates/patchwork/submission.html
> +++ b/patchwork/templates/patchwork/submission.html
> @@ -288,6 +288,22 @@ function toggle_div(link_id, headers_id)
>         class="btn btn-default" role="button" data-toggle="tooltip"
>     title="Download patch mbox">mbox
> +  {% if submission.series.all|length == 1 %}
> +   +   class="btn btn-default" role="button" data-toggle="tooltip"
Again, not sure what data-toggle=tooltip does. A git grep shows
data-toggle=button, tab, pil, collapse, checkboxes, but I can't find an
=tooltip in CSS or JS.

Again, apart from that:
Reviewed-by: Daniel Axtens 

Regards,
Daniel

> +   title="Download patch mbox">series
> +  {% elif submission.series.all|length > 1 %}
> +  
> +  
> +  {% for series in submission.series.all %}
> +    +    >{{ series }}
> +  {% endfor %}
> +  
> +  {% endif %}
>

writes: > Hi, > > If I try to run patchwork with system versions of packages on Xenial, I > run in to problems. > > Specifically, when I run with ENABLE_REST_API=True, I see the following > error: > > ImportError: Could not import > 'django_filters.rest_framework.DjangoFilterBackend' for API setting > 'DEFAULT_FILTER_BACKENDS'. ImportError: No module named rest_framework. > > I suspect this is due to the version of DRF - apt search shows: > > python-djangorestframework/xenial 3.3.2-2 all > > requirements-prod.txt lists the required version as 3.5 to 3.6. > Sorry, I should have said: I think this is an interaction between DRF and django-filter: django-filter is 0.11: python-django-filter/xenial 0.11.0-2 all and requirements-prod.txt has 1.0-1.1 I was pointed towards the filters/DRF interaction by: http://stackoverflow.com/a/41477856/463510 I'm still digging in to this but I wanted to flag it in case anyone had any bright ideas. We still need to fix this for the reasons in my original email. I also can't find a system package for drf-nested-router, which will also be an issue for deployment on sites that want to use system packages for stability and security update reasons. Regards, Daniel > This is something we really need to fix: ozlabs.org runs Debian, and if > it doesn't run on Xenial it's not likely to run on Debian. And we > really, really want ozlabs.org to be able to support the REST API. > > Can we relax the drf requirement? > > Regards, > Daniel From stephen at that.guru Wed Apr 5 02:22:06 2017 From: stephen at that.guru (Stephen Finucane) Date: Tue, 04 Apr 2017 17:22:06 +0100 Subject: [PATCH v4 0/9] Add support for download series mboxes In-Reply-To: <20170310173807.13407-1-stephen@that.guru> References: <20170310173807.13407-1-stephen@that.guru> Message-ID: <1491322926.15617.1.camel@that.guru> On Fri, 2017-03-10 at 17:37 +0000, Stephen Finucane wrote: > We now support series, but we don't support any way of making use of > these from within the UI. Correct this oversight by integrating a > "download series" button on all patch pages along with ways to > download > both entire series and individual patches with dependencies. Applied this series. Thanks for the reviews, Daniel. Stephen From stephen at that.guru Wed Apr 5 07:32:26 2017 From: stephen at that.guru (Stephen Finucane) Date: Tue, 04 Apr 2017 22:32:26 +0100 Subject: Patchwork doesn't run on Xenial with system packages In-Reply-To: <87zifwjt0g.fsf@possimpible.ozlabs.ibm.com> References: <8737dol8b0.fsf@possimpible.ozlabs.ibm.com> <87zifwjt0g.fsf@possimpible.ozlabs.ibm.com> Message-ID: <1491341546.22202.2.camel@that.guru> On Tue, 2017-04-04 at 21:45 +1000, Daniel Axtens wrote: > Daniel Axtens writes: I could have sworn I sent a reply to this a few hours ago, but here we go again. > > Hi, > > > > If I try to run patchwork with system versions of packages on > > Xenial, I > > run in to problems. > > > > Specifically, when I run with ENABLE_REST_API=True, I see the > > following > > error: > > > > ImportError: Could not import > > 'django_filters.rest_framework.DjangoFilterBackend' for API setting > > 'DEFAULT_FILTER_BACKENDS'. ImportError: No module named > > rest_framework. > > > > I suspect this is due to the version of DRF - apt search shows: > > > > python-djangorestframework/xenial 3.3.2-2 all > > > > requirements-prod.txt lists the required version as 3.5 to 3.6. If you're installing using system packages, you shouldn't be looking at requirements.txt - that's for pip-based installation. However, there are issues which need to be resolved. > Sorry, I should have said: I think this is an interaction between DRF > and django-filter: django-filter is 0.11: > > python-django-filter/xenial 0.11.0-2 all > > and requirements-prod.txt has 1.0-1.1 > > I was pointed towards the filters/DRF interaction by: > http://stackoverflow.com/a/41477856/463510 Correct. I had previously tried to fix this but got bogged down in django-rest-framework and django-filter issues. I think I have these resolved now. > I'm still digging in to this but I wanted to flag it in case anyone > had > any bright ideas. I do! :) Fixes should be on the list shortly. > We still need to fix this for the reasons in my original email. I > also > can't find a system package for drf-nested-router, which will also be > an > issue for deployment on sites that want to use system packages for > stability and security update reasons. drf-nested-router isn't required any more. Where have you seen this dependency? > Regards, > Daniel > > > This is something we really need to fix: ozlabs.org runs Debian, > > and if > > it doesn't run on Xenial it's not likely to run on Debian. And we > > really, really want ozlabs.org to be able to support the REST API. Completely agree. > > Can we relax the drf requirement? As above, you can ignore 'requirements.txt' for installation from packages. I have updated the README, however, to reflect the variety of packages we support. Cheers, Stephen PS: We _really_ need to drop Django 1.6, 1.7 support :( From stephen at that.guru Wed Apr 5 07:38:19 2017 From: stephen at that.guru (Stephen Finucane) Date: Tue, 4 Apr 2017 22:38:19 +0100 Subject: [PATCH 3/4] requirements: Test older versions of DRF (v2) In-Reply-To: <20170404213820.13350-1-stephen@that.guru> References: <20170404213820.13350-1-stephen@that.guru> Message-ID: <20170404213820.13350-3-stephen@that.guru> In commit febad055fb6609369f1a465a5eec323549c5c065, we attempted to start testing older versions of Django REST Framework for use with older versions of Django. This work was later reverted in commit 6f2ddab6e60aac91dac782aeccd2ecd35416c271 due to perceived issues with django-filter versions. There are additional reasons for validating older versions of DRF, such as supporting the versions provide as part of distributions. It turns out that those issues aren't such a big deal and can be handled with a shim. All in all, this means we can and should support these older versions of Django REST Framework, and that is what we'll do. Note that the minimum version supported in requirements.txt is not bumped as it's good practive to use the latest available version. However, that doesn't make it any less supported. Signed-off-by: Stephen Finucane Cc: Daniel Axtens --- README.md | 9 +++++++++ patchwork/settings/dev.py | 3 +-- tox.ini | 4 ++++ 3 files changed, 14 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 488dad4..a3c66bf 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,15 @@ subsystems of the Linux kernel. Although Patchwork has been developed with the kernel workflow in mind, the aim is to be flexible enough to suit the majority of community projects. +# Requirements + +- Python (2.7, 3.3 - 3.5) +- Django (1.6 - 1.10) +- Django REST Framework (3.2 - 3.6) (\*) +- Django Filters (0.11 - 1.0) (\*\*) + +Note that various dependencies have their own requirements + # Development Installation [Docker][ref-docker] is the recommended installation methods for a Patchwork diff --git a/patchwork/settings/dev.py b/patchwork/settings/dev.py index ee5b203..86b0f1b 100644 --- a/patchwork/settings/dev.py +++ b/patchwork/settings/dev.py @@ -92,5 +92,4 @@ if django.VERSION >= (1, 8): ENABLE_XMLRPC = True -if django.VERSION >= (1, 8): - ENABLE_REST_API = True +ENABLE_REST_API = True diff --git a/tox.ini b/tox.ini index bf38f8f..8cf23f6 100644 --- a/tox.ini +++ b/tox.ini @@ -7,7 +7,11 @@ skipsdist = True deps = -r{toxinidir}/requirements-test.txt django16: django>=1.6,<1.7 + django16: djangorestframework>=3.2,<3.3 + django16: django-filter>=0.11,<0.12 django17: django>=1.7,<1.8 + django17: djangorestframework>=3.3,<3.4 + django17: django-filter>=0.11,<0.12 django18: django>=1.8,<1.9 django19: django>=1.9,<1.10 django110: django>=1.10,<1.11 -- 2.9.3 From stephen at that.guru Wed Apr 5 07:38:17 2017 From: stephen at that.guru (Stephen Finucane) Date: Tue, 4 Apr 2017 22:38:17 +0100 Subject: [PATCH 1/4] compat: Handle older versions of django-filter Message-ID: <20170404213820.13350-1-stephen@that.guru> Recent versions of Django REST Framework (DRF) have deprecated the 'DjangoFilterBackend' filter found in-tree, in favour of an equivalent implementation found in django-filter. However, we need to support older versions of DRF for users who want to use system packages. Seeing as the two implementations are, for all intents and purposes, essentially the same thing, provide a shim that will allow us to use both together. Signed-off-by: Stephen Finucane Cc: Daniel Axtens Partial-bug: #94 --- patchwork/api/filters.py | 5 +++-- patchwork/compat.py | 37 +++++++++++++++++++++++++++++++++++++ patchwork/settings/base.py | 2 +- 3 files changed, 41 insertions(+), 3 deletions(-) diff --git a/patchwork/api/filters.py b/patchwork/api/filters.py index eff7ceb..1573416 100644 --- a/patchwork/api/filters.py +++ b/patchwork/api/filters.py @@ -20,6 +20,7 @@ from django_filters import FilterSet from django_filters import IsoDateTimeFilter +from patchwork.compat import LOOKUP_FIELD from patchwork.models import Bundle from patchwork.models import Check from patchwork.models import CoverLetter @@ -31,8 +32,8 @@ from patchwork.models import Series class TimestampMixin(FilterSet): # TODO(stephenfin): These should filter on a 'updated_at' field instead - before = IsoDateTimeFilter(name='date', lookup_expr='lt') - since = IsoDateTimeFilter(name='date', lookup_expr='gte') + before = IsoDateTimeFilter(name='date', **{LOOKUP_FIELD: 'lt'}) + since = IsoDateTimeFilter(name='date', **{LOOKUP_FIELD: 'gte'}) class SeriesFilter(TimestampMixin, FilterSet): diff --git a/patchwork/compat.py b/patchwork/compat.py index 5c8ca23..af7d684 100644 --- a/patchwork/compat.py +++ b/patchwork/compat.py @@ -20,6 +20,7 @@ """Compatibility wrappers for various Django versions.""" import django +from django.conf import settings # render_to_string @@ -39,3 +40,39 @@ else: context_instance = RequestContext(request) if request else None return loader.render_to_string(template_name, context, context_instance) + + +# DjangoFilterBackend +# +# The DjangoFilterBackend was provided in Django REST Framework from 3.0 to +# 3.4, was marked as pending deprecation in 3.5, was deprecated in 3.6 and will +# be removed in 3.7. However, the equivalent DjangoFilterBackend found in +# django-filter is only available since 1.0 of that package. +# +# http://www.django-rest-framework.org/topics/3.6-announcement/ + +if settings.ENABLE_REST_API: + import rest_framework # noqa + + if rest_framework.VERSION >= '3.5': + from django_filters.rest_framework import DjangoFilterBackend # noqa + else: + from rest_framework.filters import DjangoFilterBackend # noqa + + +# LOOKUP_FIELD +# +# The django-filter library uses the 'lookup_expr' attribute to determine which +# lookup type to use, e.g. exact, gt, lt etc. However, until 0.13 this was +# called 'lookup_type', and 0.13 supported both but gave a deprecation warning. +# We need to support these versions for use with older versions of DRF. +# +# https://github.com/carltongibson/django-filter/blob/v0.13/django_filters\ +# /filters.py#L35-L36 +if settings.ENABLE_REST_API: + import django_filters # noqa + + if django_filters.VERSION >= (1, 0): + LOOKUP_FIELD = 'lookup_expr' + else: + LOOKUP_FIELD = 'lookup_type' diff --git a/patchwork/settings/base.py b/patchwork/settings/base.py index 180a469..5d5caf1 100644 --- a/patchwork/settings/base.py +++ b/patchwork/settings/base.py @@ -147,7 +147,7 @@ REST_FRAMEWORK = { 'rest_framework.versioning.NamespaceVersioning', 'DEFAULT_PAGINATION_CLASS': 'patchwork.api.base.LinkHeaderPagination', 'DEFAULT_FILTER_BACKENDS': ( - 'django_filters.rest_framework.DjangoFilterBackend', + 'patchwork.compat.DjangoFilterBackend', 'rest_framework.filters.SearchFilter', 'rest_framework.filters.OrderingFilter', ), -- 2.9.3 From stephen at that.guru Wed Apr 5 07:38:18 2017 From: stephen at that.guru (Stephen Finucane) Date: Tue, 4 Apr 2017 22:38:18 +0100 Subject: [PATCH 2/4] compat: Add non-None queryset attribute In-Reply-To: <20170404213820.13350-1-stephen@that.guru> References: <20170404213820.13350-1-stephen@that.guru> Message-ID: <20170404213820.13350-2-stephen@that.guru> Older versions of Django REST Framework's 'RelatedField' did not check for the presence 'get_queryset' before checking for a 'queryset' attribute. This was fixed in 3.4.0 [1] and backported to the 3.3 series in 3.3.3 [2]. However, the fix was not backported to the last release of 3.2 series [3], which is the last release to support Django 1.6. As such, we must set a 'queryset' attribute, even if it's set to a garbage value. [1] ../blob/3.4.0/rest_framework/relations.py#L78-L82 [2] ../blob/3.3.3/rest_framework/relations.py#L78-L82 [3] ../blob/3.2.5/rest_framework/relations.py#L64-L67 Signed-off-by: Stephen Finucane Cc: Daniel Axtens Closes-bug: #94 --- patchwork/api/patch.py | 1 + 1 file changed, 1 insertion(+) diff --git a/patchwork/api/patch.py b/patchwork/api/patch.py index ded4910..fa6fb46 100644 --- a/patchwork/api/patch.py +++ b/patchwork/api/patch.py @@ -53,6 +53,7 @@ class StateField(RelatedField): 'incorrect_type': _('Incorrect type. Expected string value, received ' '{data_type}.'), } + queryset = '' # django 1.6, rest_framework 3.2 require this def to_internal_value(self, data): try: -- 2.9.3 From stephen at that.guru Wed Apr 5 07:38:20 2017 From: stephen at that.guru (Stephen Finucane) Date: Tue, 4 Apr 2017 22:38:20 +0100 Subject: [PATCH 4/4] tox: Test all supported versions of DRF In-Reply-To: <20170404213820.13350-1-stephen@that.guru> References: <20170404213820.13350-1-stephen@that.guru> Message-ID: <20170404213820.13350-4-stephen@that.guru> We should validate against the various DRF versions that support the Django versions we support. Signed-off-by: Stephen Finucane Cc: Daniel Axtens --- I'm not sure about this one: it's a _lot_ of extra testing that I'm not so sure people will run. I wonder if it's even worth it? Perhaps Travis could test using system packages in addition to pip? --- tox.ini | 19 ++++++++++++------- 1 file changed, 12 insertions(+), 7 deletions(-) diff --git a/tox.ini b/tox.ini index 8cf23f6..72ec5c9 100644 --- a/tox.ini +++ b/tox.ini @@ -1,22 +1,27 @@ [tox] minversion = 2.0 -envlist = pep8,py{27,34}-django{16,17,18,19,110},py35-django{18,19,110} +envlist = + pep8 + py{27,34}-django16-drf32 + py{27,34}-django17-drf{32,33} + py{27,34,35}-django{18,19,110}-drf{34,35,36} skipsdist = True [testenv] deps = -r{toxinidir}/requirements-test.txt django16: django>=1.6,<1.7 - django16: djangorestframework>=3.2,<3.3 - django16: django-filter>=0.11,<0.12 django17: django>=1.7,<1.8 - django17: djangorestframework>=3.3,<3.4 - django17: django-filter>=0.11,<0.12 django18: django>=1.8,<1.9 django19: django>=1.9,<1.10 django110: django>=1.10,<1.11 - django{18,19,110}: djangorestframework>=3.5,<3.6 - django{18,19,110}: django-filter>=1.0,<1.1 + drf32: djangorestframework>=3.2,<3.3 + drf33: djangorestframework>=3.3,<3.4 + drf34: djangorestframework>=3.4,<3.5 + drf35: djangorestframework>=3.5,<3.6 + drf36: djangorestframework>=3.6,<3.7 + drf{32,33,34}: django-filter>=0.11,<0.12 + drf{35,36}: django-filter>=1.0,<1.1 setenv = DJANGO_SETTINGS_MODULE = patchwork.settings.dev PYTHONDONTWRITEBYTECODE = 1 -- 2.9.3 From dja at axtens.net Wed Apr 5 14:29:20 2017 From: dja at axtens.net (Daniel Axtens) Date: Wed, 05 Apr 2017 14:29:20 +1000 Subject: [PATCH 4/4] tox: Test all supported versions of DRF In-Reply-To: <20170404213820.13350-4-stephen@that.guru> References: <20170404213820.13350-1-stephen@that.guru> <20170404213820.13350-4-stephen@that.guru> Message-ID: <87vaqjjx3z.fsf@possimpible.ozlabs.ibm.com> Hi Stephen, Thanks for doing all of these - I will test them soon. > I'm not sure about this one: it's a _lot_ of extra testing that I'm not > so sure people will run. I wonder if it's even worth it? Perhaps Travis > could test using system packages in addition to pip? I have a partial patch in this direction - I suggest adding things like py{27,35}-{xenial,jessie,etc} that install the versions known to be available on those platforms. Regards, Daniel > --- > tox.ini | 19 ++++++++++++------- > 1 file changed, 12 insertions(+), 7 deletions(-) > > diff --git a/tox.ini b/tox.ini > index 8cf23f6..72ec5c9 100644 > --- a/tox.ini > +++ b/tox.ini > @@ -1,22 +1,27 @@ > [tox] > minversion = 2.0 > -envlist = pep8,py{27,34}-django{16,17,18,19,110},py35-django{18,19,110} > +envlist = > + pep8 > + py{27,34}-django16-drf32 > + py{27,34}-django17-drf{32,33} > + py{27,34,35}-django{18,19,110}-drf{34,35,36} > skipsdist = True > > [testenv] > deps = > -r{toxinidir}/requirements-test.txt > django16: django>=1.6,<1.7 > - django16: djangorestframework>=3.2,<3.3 > - django16: django-filter>=0.11,<0.12 > django17: django>=1.7,<1.8 > - django17: djangorestframework>=3.3,<3.4 > - django17: django-filter>=0.11,<0.12 > django18: django>=1.8,<1.9 > django19: django>=1.9,<1.10 > django110: django>=1.10,<1.11 > - django{18,19,110}: djangorestframework>=3.5,<3.6 > - django{18,19,110}: django-filter>=1.0,<1.1 > + drf32: djangorestframework>=3.2,<3.3 > + drf33: djangorestframework>=3.3,<3.4 > + drf34: djangorestframework>=3.4,<3.5 > + drf35: djangorestframework>=3.5,<3.6 > + drf36: djangorestframework>=3.6,<3.7 > + drf{32,33,34}: django-filter>=0.11,<0.12 > + drf{35,36}: django-filter>=1.0,<1.1 > setenv = > DJANGO_SETTINGS_MODULE = patchwork.settings.dev > PYTHONDONTWRITEBYTECODE = 1 > -- > 2.9.3 From stephen at that.guru Wed Apr 5 19:47:37 2017 From: stephen at that.guru (Stephen Finucane) Date: Wed, 05 Apr 2017 10:47:37 +0100 Subject: [PATCH 4/4] tox: Test all supported versions of DRF In-Reply-To: <87vaqjjx3z.fsf@possimpible.ozlabs.ibm.com> References: <20170404213820.13350-1-stephen@that.guru> <20170404213820.13350-4-stephen@that.guru> <87vaqjjx3z.fsf@possimpible.ozlabs.ibm.com> Message-ID: <1491385657.22202.5.camel@that.guru> On Wed, 2017-04-05 at 14:29 +1000, Daniel Axtens wrote: > Hi Stephen, > > Thanks for doing all of these - I will test them soon. No problem. Do note that there's an apparent issue with DRF 3.4 and Django 1.10. It _should_ work, but it gives an odd warning. You can probably ignore that particular combination given the age gap between them. > > I'm not sure about this one: it's a _lot_ of extra testing that I'm > > not > > so sure people will run. I wonder if it's even worth it? Perhaps > > Travis > > could test using system packages in addition to pip? > > I have a partial patch in this direction - I suggest adding things > like > py{27,35}-{xenial,jessie,etc} that install the versions known to be > available on those platforms. Do be aware that those packages frequently carry backports from newer versions and might be different to the version found in pip. Testing with _actual_ system packages would be a far better indicator. Also - I ran these tests last night and it took over an hour. That's a bit long for anything non-automated, I think. Must find a better way. Cheers, Stephen From stephen at that.guru Wed Apr 5 20:11:28 2017 From: stephen at that.guru (Stephen Finucane) Date: Wed, 05 Apr 2017 11:11:28 +0100 Subject: [PATCH 1/3] xmlrpc: Remove 'bundle_to_dict' In-Reply-To: <20170323132501.324-1-stephen@that.guru> References: <20170323132501.324-1-stephen@that.guru> Message-ID: <1491387088.22202.6.camel@that.guru> On Thu, 2017-03-23 at 13:24 +0000, Stephen Finucane wrote: > Turns out this function is not called anywhere nor has it been called > since being introduced in '83964878'. Given that we're now focused on > the REST API, we can simply remove this. > > Signed-off-by: Stephen Finucane > Cc: Daniel Axtens Trivial fix. Merged. From stephen at that.guru Wed Apr 5 20:11:45 2017 From: stephen at that.guru (Stephen Finucane) Date: Wed, 05 Apr 2017 11:11:45 +0100 Subject: [PATCH 2/3] views: Use bundle-detail for public URLs In-Reply-To: <20170323132501.324-2-stephen@that.guru> References: <20170323132501.324-1-stephen@that.guru> <20170323132501.324-2-stephen@that.guru> Message-ID: <1491387105.22202.7.camel@that.guru> On Thu, 2017-03-23 at 13:25 +0000, Stephen Finucane wrote: > In 'd1c605f', we reworked the 'bundle-list' view to use the new > 'Bundle.get_mbox_url' function instead of the 'Bundle.public_url'. > However, these are not the same thing. The latter referred to the > 'bundle-detail' view, while the former referred to the 'bundle-mbox' > view. > > The easiest fix would be to simply revert that patch. However, it > turns > out that 'public_url' isn't actually needed. Commit '5d0140ef' > removed a > divide between public and non-public URLs for bundles, meaning we can > actually use an existing function - 'get_absolute_url' - instead. > > This also presents the opportunity to clean up the 'bundle-list' > page, > favouring a simple public/is-not-public marker and only a single URL. > > Signed-off-by: Stephen Finucane > Fixes: d1c605f9 ("views: Stop using Bundle.public_url") > Closes-bug: #92 > Cc: Daniel Axtens > Cc: Andy Doan Applied. From stephen at that.guru Wed Apr 5 20:12:00 2017 From: stephen at that.guru (Stephen Finucane) Date: Wed, 05 Apr 2017 11:12:00 +0100 Subject: [PATCH 3/3] views: Link to project from bundle-list In-Reply-To: <20170323132501.324-3-stephen@that.guru> References: <20170323132501.324-1-stephen@that.guru> <20170323132501.324-3-stephen@that.guru> Message-ID: <1491387120.22202.8.camel@that.guru> On Thu, 2017-03-23 at 13:25 +0000, Stephen Finucane wrote: > Simple usability improvement. > > Signed-off-by: Stephen Finucane > Cc: Daniel Axtens Trivial fix. Applied. From stephen at that.guru Wed Apr 5 20:15:31 2017 From: stephen at that.guru (Stephen Finucane) Date: Wed, 05 Apr 2017 11:15:31 +0100 Subject: [PATCH] pwclient: Fix silent crash on python 2 In-Reply-To: <20170328092624.21943-1-robin.jarry@6wind.com> References: <20170328092624.21943-1-robin.jarry@6wind.com> Message-ID: <1491387331.22202.9.camel@that.guru> On Tue, 2017-03-28 at 11:26 +0200, Robin Jarry wrote: > Replacing sys.stdout and sys.stderr can cause obscure crashes when > trying to write non unicode data. The interpreter is terminated with > SIGINT without any specific error writen on the console. > > rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7f964820e8d0}, > {0x559f50, [], SA_RESTORER, 0x7f964820e8d0}, 8) = 0 > > This happens easily when there is an untrapped exception which should > lead to printing a traceback on stderr. > > To fix this, the only way is to make sure that the PYTHONIOENCODING > env > variable is set *before* starting the interpreter as the > initialization > is made very early on and the encoding cannot be set or modified > after. > > On python 3, the default IO encoding is already properly set > according > to locale. > > Fixes: 046419a3bf8f ("pwclient: Fix encoding problems") > Signed-off-by: Robin Jarry I've no idea how I could even go about testing this. However, it looks sane and you've tested it locally so... Acked-by: Stephen Finucane ...and applied. From stephen at that.guru Wed Apr 5 20:20:29 2017 From: stephen at that.guru (Stephen Finucane) Date: Wed, 05 Apr 2017 11:20:29 +0100 Subject: [PATCH] pwclient: Fix silent crash on python 2 In-Reply-To: <1491387331.22202.9.camel@that.guru> References: <20170328092624.21943-1-robin.jarry@6wind.com> <1491387331.22202.9.camel@that.guru> Message-ID: <1491387629.22202.11.camel@that.guru> On Wed, 2017-04-05 at 11:15 +0100, Stephen Finucane wrote: > On Tue, 2017-03-28 at 11:26 +0200, Robin Jarry wrote: > > Replacing sys.stdout and sys.stderr can cause obscure crashes when > > trying to write non unicode data. The interpreter is terminated > > with > > SIGINT without any specific error writen on the console. > > > > rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7f964820e8d0}, > > {0x559f50, [], SA_RESTORER, 0x7f964820e8d0}, 8) = 0 > > > > This happens easily when there is an untrapped exception which > > should > > lead to printing a traceback on stderr. > > > > To fix this, the only way is to make sure that the PYTHONIOENCODING > > env > > variable is set *before* starting the interpreter as the > > initialization > > is made very early on and the encoding cannot be set or modified > > after. > > > > On python 3, the default IO encoding is already properly set > > according > > to locale. > > > > Fixes: 046419a3bf8f ("pwclient: Fix encoding problems") > > Signed-off-by: Robin Jarry > > I've no idea how I could even go about testing this. However, it > looks > sane and you've tested it locally so... > > Acked-by: Stephen Finucane > > ...and applied. Actually, I take that back. Let's see if there's a less hacky way to do this. Seems this is a fairly common problem with a few fixes available. Would a fix like [1] do the job? Also, could you provide some guidance as to how I could reproduce the issue myself? Is the Patchwork instance using Python 2 or 3? What environment is pwclient running in? Stephen [1] http://stackoverflow.com/a/11742574/613428 > _______________________________________________ > Patchwork mailing list > Patchwork at lists.ozlabs.org > https://lists.ozlabs.org/listinfo/patchwork From robin.jarry at 6wind.com Wed Apr 5 21:35:52 2017 From: robin.jarry at 6wind.com (Robin Jarry) Date: Wed, 5 Apr 2017 13:35:52 +0200 Subject: [PATCH] pwclient: Fix silent crash on python 2 In-Reply-To: <1491387629.22202.11.camel@that.guru> References: <20170328092624.21943-1-robin.jarry@6wind.com> <1491387331.22202.9.camel@that.guru> <1491387629.22202.11.camel@that.guru> Message-ID: <20170405113552.7vbiyjgnfykoojmn@6wind.com> Hi Stephen, thanks for your reply. It has made me look deeper and I found that my fix is not completely satisfying. Python 3 is also concerned. I will submit another version ASAP. The problem that I was trying to fix with my previous patch only appears when neither the LANG nor LC_ALL env vars are set. Python (2 and 3) assumes ASCII encoding by default which causes nasty errors like the following: $ env -u LC_ALL -u LANG python2 -c "print u's\u00e9duisante'" Traceback (most recent call last): File "", line 1, in UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' $ env -u LC_ALL -u LANG python3 -c "print('s\u00e9duisante')" Traceback (most recent call last): File "", line 1, in UnicodeEncodeError: 'ascii' codec can't encode character '\xe9' When either LANG or LC_ALL are set to something else than 'C', python tries to encode accordingly. $ env -u LC_ALL - LANG=C.UTF-8 python3 -c "print(u's\u00e9duisante')" séduisante $ env -u LC_ALL - LANG=français python3 -c "print(u's\u00e9duisante')" s�duisante My new patch makes sure that the default encoding is never 'ascii' to avoid the previous problems. But without the nasty hack that replaces stdout and stderr. 2017-04-05, Stephen Finucane: > Actually, I take that back. Let's see if there's a less hacky way to do > this. Seems this is a fairly common problem with a few fixes available. > Would a fix like [1] do the job? > > [1] http://stackoverflow.com/a/11742574/613428 This approach is risky. sys.defaultencoding is used to decode python source. Have a look at these links: http://stackoverflow.com/questions/3828723/why-should-we-not-use-sys-setdefaultencodingutf-8-in-a-py-script https://anonbadger.wordpress.com/2015/06/16/why-sys-setdefaultencoding-will-break-code/ > Also, could you provide some guidance as to how I could reproduce the > issue myself? Is the Patchwork instance using Python 2 or 3? What > environment is pwclient running in? The issue can be reproduced with pwclient executed with python 2 (server python version does not matter). Simply put an invalid URL in ~/.pwclientrc (note the missing /): $ cat ~/.pwclientrc [options] default = dpdk [dpdk] url= http:/dpdk.org/dev/patchwork/xmlrpc/ Then run this: $ git describe v1.1.0-384-g21e2926743a1 $ python2 ./patchwork/bin/pwclient projects $ echo $? 1 -- Robin From robin.jarry at 6wind.com Wed Apr 5 22:46:04 2017 From: robin.jarry at 6wind.com (Robin Jarry) Date: Wed, 5 Apr 2017 14:46:04 +0200 Subject: [PATCH v2] pwclient: Fix silent crash on python 2 In-Reply-To: <20170405113552.7vbiyjgnfykoojmn@6wind.com> References: <20170405113552.7vbiyjgnfykoojmn@6wind.com> Message-ID: <20170405124604.28022-1-robin.jarry@6wind.com> Replacing sys.stdout and sys.stderr can cause obscure crashes when trying to write non unicode data. The interpreter is terminated with SIGINT without any specific error writen on the console. rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7f964820e8d0}, {0x559f50, [], SA_RESTORER, 0x7f964820e8d0}, 8) = 0 This happens easily when there is an untrapped exception which should lead to printing a traceback on stderr. The only way to prevent UnicodeEncodeErrors is to make sure that PYTHONIOENCODING is set with the ':replace' suffix and this can only be done *before* starting the interpreter as the initialization is made very early on and the encoding cannot be set or modified after. >From the official documentation: PYTHONIOENCODING Overrides the encoding used for stdin/stdout/stderr, in the syntax encodingname:errorhandler. The :errorhandler part is optional and has the same meaning as in str.encode(). https://docs.python.org/2/using/cmdline.html https://docs.python.org/3/using/cmdline.html Of course, for proper encoding of unicode characters, one of the locale-related environment variables (LC_ALL, LANG, LANGUAGE, etc.) must be set. Python will use the correct encoding accordingly and PYTHONIOENCODING will be set to "encoding:replace". Examples: $ grep utf8 ~/.Xresources xterm*utf8: 2 $ env - PYTHONIOENCODING=utf-8:replace python2 -c "print u's\u00e9duisante'" séduisante $ env - PYTHONIOENCODING=utf-8:replace python3 -c "print('s\u00e9duisante')" séduisante $ env - PYTHONIOENCODING=ascii:replace python2 -c "print u's\u00e9duisante'" s?duisante $ env - PYTHONIOENCODING=ascii:replace python3 -c "print('s\u00e9duisante')" s?duisante $ env - PYTHONIOENCODING=ISO-8859-1:replace python2 -c "print u's\u00e9duisante'" s�duisante $ env - PYTHONIOENCODING=ISO-8859-1:replace python3 -c "print('s\u00e9duisante')" s�duisante Fixes: 046419a3bf8f ("pwclient: Fix encoding problems") Signed-off-by: Robin Jarry --- v2: - Always set PYTHONIOENCODING=:replace (on python 2 and 3) to prevent from UnicodeEncodeErrors patchwork/bin/pwclient | 35 ++++++++++++++++++++++++++--------- 1 file changed, 26 insertions(+), 9 deletions(-) diff --git a/patchwork/bin/pwclient b/patchwork/bin/pwclient index ed0351bf5288..5a7b6723afe3 100755 --- a/patchwork/bin/pwclient +++ b/patchwork/bin/pwclient @@ -41,16 +41,7 @@ except ImportError: import shutil import re import io -import locale -if sys.version_info.major == 2: - # hack to make writing unicode to standard output/error work on Python 2 - OUT_ENCODING = (sys.stdout.encoding or locale.getpreferredencoding() or - os.getenv('PYTHONIOENCODING', 'utf-8')) - sys.stdout = io.open(sys.stdout.fileno(), mode='w', - encoding=OUT_ENCODING, errors='replace') - sys.stderr = io.open(sys.stderr.fileno(), mode='w', - encoding=OUT_ENCODING, errors='replace') # Default Patchwork remote XML-RPC server URL # This script will check the PW_XMLRPC_URL environment variable @@ -821,5 +812,31 @@ def main(): sys.exit(1) +def force_io_encoding(): + """ + Force PYTHONIOENCODING ":errorhandler" to avoid UnicodeEncodeErrors. The + only way to do it is to set the environment variable *before* starting the + interpreter. From the python docs: + + PYTHONIOENCODING + + Overrides the encoding used for stdin/stdout/stderr, in the syntax + encodingname:errorhandler. The :errorhandler part is optional and has the + same meaning as in str.encode(). + + Note that this only prevents interpreter crashes, it does not exempt from + correctly setting the LANG or LC_ALL variables in order to have valid + output. + """ + if 'PYTHONIOENCODING' in os.environ: + return + + encoding = sys.stdout.encoding or 'utf-8' + + os.environ['PYTHONIOENCODING'] = encoding + ':replace' + os.execvp(sys.executable, [sys.executable] + sys.argv) # no return + + if __name__ == "__main__": + force_io_encoding() main() -- 2.11.0.193.g1d1bdafd6426 From ralf at linux-mips.org Thu Apr 6 00:06:03 2017 From: ralf at linux-mips.org (Ralf Baechle) Date: Wed, 5 Apr 2017 16:06:03 +0200 Subject: Issue with patch in a reply to a patch Message-ID: <20170405140603.GA5608@linux-mips.org> Hi, since a recent upgrade of patchwork in February I've observed patchwork misshandling patches sent as reply to another patch. An example is https://patchwork.linux-mips.org/patch/15887/ where James' reply (first reply) was submitted like: [...] So I think something like this would be more correct. Does that fix your problem? diff --git a/arch/mips/kernel/cevt-r4k.c b/arch/mips/kernel/cevt-r4k.c index 804d2a2a19fe..dd6a18bc10ab 100644 --- a/arch/mips/kernel/cevt-r4k.c +++ b/arch/mips/kernel/cevt-r4k.c @@ -80,7 +80,7 @@ static unsigned int calculate_min_delta(void) } /* Sorted insert of 75th percentile into buf2 */ - for (k = 0; k < i; ++k) { + for (k = 0; k < i && k < ARRAY_SIZE(buf2); ++k) { if (buf1[ARRAY_SIZE(buf1) - 1] < buf2[k]) { l = min_t(unsigned int, i, ARRAY_SIZE(buf2) - 1); Thanks James [...] But then mangled into: [...] Thanks Jamesdiff --git a/arch/mips/kernel/cevt-r4k.c b/arch/mips/kernel/cevt-r4k.c ^^^^^^^^^ [...] James' archived original mail in mbox format is available at: https://www.linux-mips.org/cgi-bin/extract-mesg.cgi?a=linux-mips&m=2017-04&i=20170404203219.GK31606%40jhogan-linux.le.imgtec.org the whole thread at https://www.linux-mips.org/archives/linux-mips/2017-04/msg00035.html Ralf From dja at axtens.net Thu Apr 6 07:04:44 2017 From: dja at axtens.net (Daniel Axtens) Date: Thu, 06 Apr 2017 07:04:44 +1000 Subject: [PATCH 4/4] tox: Test all supported versions of DRF In-Reply-To: <1491385657.22202.5.camel@that.guru> References: <20170404213820.13350-1-stephen@that.guru> <20170404213820.13350-4-stephen@that.guru> <87vaqjjx3z.fsf@possimpible.ozlabs.ibm.com> <1491385657.22202.5.camel@that.guru> Message-ID: <87shlmk1lf.fsf@possimpible.ozlabs.ibm.com> Stephen Finucane writes: > On Wed, 2017-04-05 at 14:29 +1000, Daniel Axtens wrote: >> Hi Stephen, >> >> Thanks for doing all of these - I will test them soon. > > No problem. Do note that there's an apparent issue with DRF 3.4 and > Django 1.10. It _should_ work, but it gives an odd warning. You can > probably ignore that particular combination given the age gap between > them. > >> > I'm not sure about this one: it's a _lot_ of extra testing that I'm >> > not >> > so sure people will run. I wonder if it's even worth it? Perhaps >> > Travis >> > could test using system packages in addition to pip? >> >> I have a partial patch in this direction - I suggest adding things >> like >> py{27,35}-{xenial,jessie,etc} that install the versions known to be >> available on those platforms. > > Do be aware that those packages frequently carry backports from newer > versions and might be different to the version found in pip. Testing > with _actual_ system packages would be a far better indicator. > > Also - I ran these tests last night and it took over an hour. That's a > bit long for anything non-automated, I think. Must find a better way. OK - I have been building docker files for production deployment*: they use the system packages. I will rework those and send them. Note that I haven't been able to find a system package for drf-nested-routers. Perhaps we could 'vendor' that? Regards, Daniel * see patchwork.dja.id.au > > Cheers, > Stephen From stephen at that.guru Thu Apr 6 09:02:55 2017 From: stephen at that.guru (Stephen Finucane) Date: Thu, 06 Apr 2017 00:02:55 +0100 Subject: [PATCH 4/4] tox: Test all supported versions of DRF In-Reply-To: <87shlmk1lf.fsf@possimpible.ozlabs.ibm.com> References: <20170404213820.13350-1-stephen@that.guru> <20170404213820.13350-4-stephen@that.guru> <87vaqjjx3z.fsf@possimpible.ozlabs.ibm.com> <1491385657.22202.5.camel@that.guru> <87shlmk1lf.fsf@possimpible.ozlabs.ibm.com> Message-ID: <1491433375.26980.1.camel@that.guru> On Thu, 2017-04-06 at 07:04 +1000, Daniel Axtens wrote: > Stephen Finucane writes: > > > On Wed, 2017-04-05 at 14:29 +1000, Daniel Axtens wrote: > > > Hi Stephen, > > > > > > Thanks for doing all of these - I will test them soon. > > > > No problem. Do note that there's an apparent issue with DRF 3.4 and > > Django 1.10. It _should_ work, but it gives an odd warning. You can > > probably ignore that particular combination given the age gap > > between > > them. > > > > > > I'm not sure about this one: it's a _lot_ of extra testing that > > > > I'm > > > > not > > > > so sure people will run. I wonder if it's even worth it? > > > > Perhaps > > > > Travis > > > > could test using system packages in addition to pip? > > > > > > I have a partial patch in this direction - I suggest adding > > > things > > > like > > > py{27,35}-{xenial,jessie,etc} that install the versions known to > > > be > > > available on those platforms. > > > > Do be aware that those packages frequently carry backports from > > newer > > versions and might be different to the version found in pip. > > Testing > > with _actual_ system packages would be a far better indicator. > > > > Also - I ran these tests last night and it took over an hour. > > That's a > > bit long for anything non-automated, I think. Must find a better > > way. > > OK - I have been building docker files for production deployment*: > they use > the system packages. I will rework those and send them. Sounds like a plan. I'm not sure if this should be in the Patchwork project or maintained separately though? > Note that I haven't been able to find a system package for > drf-nested-routers. Perhaps we could 'vendor' that? This dependency isn't needed any more - what made you think it was? Stephen > > Regards, > Daniel > > * see patchwork.dja.id.au PS: Very nice :) > > > > Cheers, > > Stephen From dja at axtens.net Thu Apr 6 11:47:06 2017 From: dja at axtens.net (Daniel Axtens) Date: Thu, 06 Apr 2017 11:47:06 +1000 Subject: [PATCH 4/4] tox: Test all supported versions of DRF In-Reply-To: <1491433375.26980.1.camel@that.guru> References: <20170404213820.13350-1-stephen@that.guru> <20170404213820.13350-4-stephen@that.guru> <87vaqjjx3z.fsf@possimpible.ozlabs.ibm.com> <1491385657.22202.5.camel@that.guru> <87shlmk1lf.fsf@possimpible.ozlabs.ibm.com> <1491433375.26980.1.camel@that.guru> Message-ID: <87pogqjoit.fsf@possimpible.ozlabs.ibm.com> >> OK - I have been building docker files for production deployment*: >> they use >> the system packages. I will rework those and send them. > > Sounds like a plan. I'm not sure if this should be in the Patchwork > project or maintained separately though? We should probably keep it in Patchwork - it will help us make sure we don't stray too far from compatibility with major distros. I'll try to get it easy to use interactively and wired up in Travis CI. > >> Note that I haven't been able to find a system package for >> drf-nested-routers. Perhaps we could 'vendor' that? > > This dependency isn't needed any more - what made you think it was? That's an excellent question. I feel like I saw it in a requrements file but I can't find it now. Excellent! Regards, Daniel > > Stephen > >> >> Regards, >> Daniel >> >> * see patchwork.dja.id.au > > PS: Very nice :) > >> > >> > Cheers, >> > Stephen From stephen at that.guru Mon Apr 10 01:23:55 2017 From: stephen at that.guru (Stephen Finucane) Date: Sun, 9 Apr 2017 16:23:55 +0100 Subject: [PATCH] parser: Don't extract diffs from replies Message-ID: <20170409152355.17160-1-stephen@that.guru> In '2a915efd', a check was added to ensure mails prefixed with 'RE:' or similar would not be parsed as patches. By time this check actually happens, any patches had already been extracted from the mail thus these patches were re-added to the mail content before saving the comment. Unfortunately, this didn't take into account cases where a patch or diff was not the last part of a mail but rather located somewhere in the middle of the content: Introduction content Diff or patch content *** Additional content This would result in mangling of the mail as the patch would _always_ be appended to the end: Introduction content Additional content Diff or patch content *** Handle this by only breaking a mail into a comment and a diff if there is any possibility that we might want to use that diff. Signed-off-by: Stephen Finucane Fixes: 2a915efd ("parser: fix wrong parsing of diff comments") Closes-bug: #95 Cc: Ralf Baechle --- patchwork/parser.py | 63 +++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 49 insertions(+), 14 deletions(-) diff --git a/patchwork/parser.py b/patchwork/parser.py index 7e2067e..3cd0f97 100644 --- a/patchwork/parser.py +++ b/patchwork/parser.py @@ -342,10 +342,15 @@ def parse_version(subject, subject_prefixes): return 1 -def find_content(mail): - """Extract a comment and potential diff from a mail.""" - patchbuf = None - commentbuf = '' +def _find_content(mail): + """Extract the payload(s) from a mail. + + Handles various payload types. + + :returns: A list of tuples, corresponding the payload and subtype + of payload. + """ + results = [] for part in mail.walk(): if part.get_content_maintype() != 'text': @@ -381,8 +386,19 @@ def find_content(mail): # Could not find a valid decoded payload. Fail. if payload is None: - return None, None + continue + + results.append((payload, subtype)) + + return results + +def find_patch_content(mail): + """Extract a comment and potential diff from a mail.""" + patchbuf = None + commentbuf = '' + + for payload, subtype in _find_content(mail): if subtype in ['x-patch', 'x-diff']: patchbuf = payload elif subtype == 'plain': @@ -399,6 +415,22 @@ def find_content(mail): return patchbuf, commentbuf +def find_comment_content(mail): + """Extract content from a mail.""" + commentbuf = '' + + for payload, _ in _find_content(mail): + if not payload: + continue + + commentbuf += payload.strip() + '\n' + + commentbuf = clean_content(commentbuf) + + # keep the method signature the same as find_patch_content + return None, commentbuf + + def find_submission_for_comment(project, refs): for ref in refs: # first, check for a direct reply @@ -759,12 +791,7 @@ def parse_mail(mail, list_id=None): logger.error('Failed to find a project for email') return - # parse content - - diff, message = find_content(mail) - - if not (diff or message): - return # nothing to work with + # parse metadata msgid = mail.get('Message-Id').strip() author = find_author(mail) @@ -776,6 +803,17 @@ def parse_mail(mail, list_id=None): refs = find_references(mail) date = find_date(mail) headers = find_headers(mail) + + # parse content + + if not is_comment: + diff, message = find_patch_content(mail) + else: + diff, message = find_comment_content(mail) + + if not (diff or message): + return # nothing to work with + pull_url = parse_pull_request(message) # build objects @@ -914,9 +952,6 @@ def parse_mail(mail, list_id=None): if not submission: return - if is_comment and diff: - message += diff - author.save() comment = Comment( -- 2.9.3 From stephen at that.guru Mon Apr 10 02:13:06 2017 From: stephen at that.guru (Stephen Finucane) Date: Sun, 09 Apr 2017 17:13:06 +0100 Subject: Issue with patch in a reply to a patch In-Reply-To: <20170405140603.GA5608@linux-mips.org> References: <20170405140603.GA5608@linux-mips.org> Message-ID: <1491754386.5029.2.camel@that.guru> On Wed, 2017-04-05 at 16:06 +0200, Ralf Baechle wrote: > Hi, > > since a recent upgrade of patchwork in February I've observed > patchwork > misshandling patches sent as reply to another patch. An example is > > https://patchwork.linux-mips.org/patch/15887/ > > where James' reply (first reply) was submitted like: > > [...] > So I think something like this would be more correct. Does that fix > your > problem? > > diff --git a/arch/mips/kernel/cevt-r4k.c b/arch/mips/kernel/cevt- > r4k.c > index 804d2a2a19fe..dd6a18bc10ab 100644 > --- a/arch/mips/kernel/cevt-r4k.c > +++ b/arch/mips/kernel/cevt-r4k.c > @@ -80,7 +80,7 @@ static unsigned int calculate_min_delta(void) > } > > /* Sorted insert of 75th percentile into buf2 */ > - for (k = 0; k < i; ++k) { > + for (k = 0; k < i && k < ARRAY_SIZE(buf2); ++k) { > if (buf1[ARRAY_SIZE(buf1) - 1] < buf2[k]) { > l = min_t(unsigned int, > i, ARRAY_SIZE(buf2) - 1); > > Thanks > James > [...] > > But then mangled into: > > [...] > Thanks > Jamesdiff --git a/arch/mips/kernel/cevt-r4k.c > b/arch/mips/kernel/cevt-r4k.c > ^^^^^^^^^ > [...] > > James' archived original mail in mbox format is available at: > > https://www.linux-mips.org/cgi-bin/extract-mesg.cgi?a=linux-mips&m=20 > 17-04&i=20170404203219.GK31606%40jhogan-linux.le.imgtec.org > > the whole thread at > > https://www.linux-mips.org/archives/linux-mips/2017-04/msg00035.html > > Ralf Aye, this looks like a bug introduced with some recent changes around parsing replies. I've a fix up [1] and would appreciate your reviews :) Cheers, Stephen [1] https://patchwork.ozlabs.org/patch/748756/ From stephen at that.guru Mon Apr 10 02:13:55 2017 From: stephen at that.guru (Stephen Finucane) Date: Sun, 09 Apr 2017 17:13:55 +0100 Subject: [PATCH 1/3] Put download buttons at the top as well In-Reply-To: <20170327054252.24337-1-dja@axtens.net> References: <20170327054252.24337-1-dja@axtens.net> Message-ID: <1491754435.5029.3.camel@that.guru> On Mon, 2017-03-27 at 16:42 +1100, Daniel Axtens wrote: > Suggested by a few people. > > Signed-off-by: Daniel Axtens Reviewed-by: Stephen Finucane From stephen at that.guru Mon Apr 10 02:14:05 2017 From: stephen at that.guru (Stephen Finucane) Date: Sun, 09 Apr 2017 17:14:05 +0100 Subject: [PATCH 3/3] Display count of patches In-Reply-To: <20170327054252.24337-3-dja@axtens.net> References: <20170327054252.24337-1-dja@axtens.net> <20170327054252.24337-3-dja@axtens.net> Message-ID: <1491754445.5029.4.camel@that.guru> On Mon, 2017-03-27 at 16:42 +1100, Daniel Axtens wrote: > Minor UI tweak - show the number of patches left in the current > view. > > Suggested-by: Stewart Smith > Signed-off-by: Daniel Axtens Reviewed-by: Stephen Finucane From stephen at that.guru Mon Apr 10 02:16:18 2017 From: stephen at that.guru (Stephen Finucane) Date: Sun, 09 Apr 2017 17:16:18 +0100 Subject: [PATCH 2/3] 'mpe mode': click to copy patch IDs In-Reply-To: <20170327054252.24337-2-dja@axtens.net> References: <20170327054252.24337-1-dja@axtens.net> <20170327054252.24337-2-dja@axtens.net> Message-ID: <1491754578.5029.7.camel@that.guru> On Mon, 2017-03-27 at 16:42 +1100, Daniel Axtens wrote: > If 'Show Patch IDs' is turned on in settings, add an extra column > to the patch list, with buttons showing the patch IDs. The buttons > copy the patch IDs to the clipboard. > > JavaScript inspired by https://github.com/Triforcey/clip-j and many > many StackOverflow answers. > > Suggested-by: Michael Ellerman > Signed-off-by: Daniel Axtens This looks good to me. I do wonder if there's any reason not to turn this on by default though? Stephen From stephen at that.guru Mon Apr 10 02:21:14 2017 From: stephen at that.guru (Stephen Finucane) Date: Sun, 09 Apr 2017 17:21:14 +0100 Subject: [PATCH] parser: cut off patch names after 255 characters In-Reply-To: <20170327050113.22312-1-dja@axtens.net> References: <20170327050113.22312-1-dja@axtens.net> Message-ID: <1491754874.5029.9.camel@that.guru> On Mon, 2017-03-27 at 16:01 +1100, Daniel Axtens wrote: > Our model limits the length of patch names to 255 characters. > Enforce this cutoff in the parser, otherwise we throw an > exception and fail to store patches with stupidly long titles. > > Reported-by: Russell Currey > Signed-off-by: Daniel Axtens Applied after adding the following fix to also handle long series names. diff --git a/patchwork/parser.py b/patchwork/parser.py index 7ec024c..b81f5b1 100644 --- a/patchwork/parser.py +++ b/patchwork/parser.py @@ -895,7 +895,7 @@ def parse_mail(mail, list_id=None): cover_letter = CoverLetter( msgid=msgid, project=project, - name=name, + name=name[:255], date=date, headers=headers, submitter=author, --- Stephen From stephen at that.guru Mon Apr 10 03:14:03 2017 From: stephen at that.guru (Stephen Finucane) Date: Sun, 9 Apr 2017 18:14:03 +0100 Subject: [PATCH] parsearchive: Support maildirs Message-ID: <20170409171403.16836-1-stephen@that.guru> At present, the 'parsearchive' command only supports parsing of mboxes. Expand this to support maildirs. This allows us to rewrite the 'parsemail-bulk' script to deliver much improved performance. Signed-off-by: Stephen Finucane Suggested-by: Daniel Axtens --- This is an alternative to [1] that avoids us having to add yet another management command. [1] https://patchwork.ozlabs.org/patch/731892/ --- patchwork/bin/parsemail-batch.sh | 27 +++++++++------------------ patchwork/management/commands/parsearchive.py | 7 ++++++- 2 files changed, 15 insertions(+), 19 deletions(-) diff --git a/patchwork/bin/parsemail-batch.sh b/patchwork/bin/parsemail-batch.sh index d42712e..3d3725c 100755 --- a/patchwork/bin/parsemail-batch.sh +++ b/patchwork/bin/parsemail-batch.sh @@ -1,7 +1,7 @@ #!/bin/sh # # Patchwork - automated patch tracking system -# Copyright (C) 2008 Jeremy Kerr +# Copyright (C) 2017 Stephen Finucane # # This file is part of the Patchwork package. # @@ -20,25 +20,16 @@ # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA BIN_DIR=$(dirname "$0") +PATCHWORK_BASE=$(readlink -e "$BIN_DIR/../..") -if [ $# -lt 1 ]; then - echo "usage: $0 [options]" >&2 - exit 1 +if [ -z "$PW_PYTHON" ]; then + PW_PYTHON=python2 fi -mail_dir="$1" - -echo "dir: $mail_dir" - -if [ ! -d "$mail_dir" ]; then - echo "$mail_dir should be a directory"? >&2 - exit 1 +if [ -z "$DJANGO_SETTINGS_MODULE" ]; then + DJANGO_SETTINGS_MODULE=patchwork.settings.production fi -shift - -find "$mail_dir" -maxdepth 1 | -while read -r line; do - echo "$line" - "$BIN_DIR/parsemail.sh" "$@" < "$line" -done +PYTHONPATH="${PATCHWORK_BASE}:${PATCHWORK_BASE}/lib/python:$PYTHONPATH" \ + DJANGO_SETTINGS_MODULE="$DJANGO_SETTINGS_MODULE" \ + "$PW_PYTHON" "$PATCHWORK_BASE/manage.py" parsearchive "$@" diff --git a/patchwork/management/commands/parsearchive.py b/patchwork/management/commands/parsearchive.py index 40b2cc0..a3c8360 100644 --- a/patchwork/management/commands/parsearchive.py +++ b/patchwork/management/commands/parsearchive.py @@ -69,7 +69,12 @@ class Command(BaseCommand): self.stdout.write('Invalid path: %s' % path) sys.exit(1) - mbox = mailbox.mbox(path) + # assume if is a directory, then we're passing a maildir + if os.path.isfile(path): + mbox = mailbox.mbox(path) + else: + mbox = mailbox.Maildir(path) + count = len(mbox) logger.info('Parsing %d mails', count) -- 2.9.3 From stephen at that.guru Mon Apr 10 04:08:49 2017 From: stephen at that.guru (Stephen Finucane) Date: Sun, 9 Apr 2017 19:08:49 +0100 Subject: [PATCH] sql: Update permissions on grant-all scripts Message-ID: <20170409180849.26170-1-stephen@that.guru> Allow access to Submission, CoverLetter, Series, SeriesReference and SeriesPatch. Signed-off-by: Stephen Finucane Cc: Jeremy Kerr --- lib/sql/grant-all.mysql.sql | 14 ++++++++++++-- lib/sql/grant-all.postgres.sql | 23 ++++++++++++++++++++++- 2 files changed, 34 insertions(+), 3 deletions(-) diff --git a/lib/sql/grant-all.mysql.sql b/lib/sql/grant-all.mysql.sql index 12dc5ca..5a297b8 100644 --- a/lib/sql/grant-all.mysql.sql +++ b/lib/sql/grant-all.mysql.sql @@ -20,7 +20,12 @@ GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_userprofile_maintainer_project GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_project TO 'www-data'@localhost; GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_bundle TO 'www-data'@localhost; GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_bundlepatch TO 'www-data'@localhost; +GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_submission TO 'www-data'@localhost; GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_patch TO 'www-data'@localhost; +GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_coverletter TO 'www-data'@localhost; +GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_series TO 'www-data'@localhost; +GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_seriespatch TO 'www-data'@localhost; +GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_seriesreference TO 'www-data'@localhost; GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_emailoptout TO 'www-data'@localhost; GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_patchchangenotification TO 'www-data'@localhost; GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_tag TO 'www-data'@localhost; @@ -28,8 +33,14 @@ GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_patchtag TO 'www-data'@localho GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_check TO 'www-data'@localhost; GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_delegationrule TO 'www-data'@localhost; --- allow the mail user (in this case, 'nobody') to add patches +-- allow the mail user (in this case, 'nobody') to add submissions (patches, +-- cover letters) and series +GRANT INSERT, SELECT ON patchwork_submission TO 'nobody'@localhost; GRANT INSERT, SELECT ON patchwork_patch TO 'nobody'@localhost; +GRANT INSERT, SELECT ON patchwork_coverletter TO 'nobody'@localhost; +GRANT INSERT, SELECT ON patchwork_series TO 'nobody'@localhost; +GRANT INSERT, SELECT ON patchwork_seriespatch TO 'nobody'@localhost; +GRANT INSERT, SELECT ON patchwork_seriesreference TO 'nobody'@localhost; GRANT INSERT, SELECT ON patchwork_comment TO 'nobody'@localhost; GRANT INSERT, SELECT ON patchwork_person TO 'nobody'@localhost; GRANT INSERT, SELECT, UPDATE, DELETE ON patchwork_patchtag TO 'nobody'@localhost; @@ -39,4 +50,3 @@ GRANT SELECT ON patchwork_tag TO 'nobody'@localhost; GRANT SELECT ON patchwork_delegationrule TO 'nobody'@localhost; COMMIT; - diff --git a/lib/sql/grant-all.postgres.sql b/lib/sql/grant-all.postgres.sql index 7cc396e..405ba44 100644 --- a/lib/sql/grant-all.postgres.sql +++ b/lib/sql/grant-all.postgres.sql @@ -21,7 +21,12 @@ GRANT SELECT, UPDATE, INSERT, DELETE ON patchwork_project, patchwork_bundle, patchwork_bundlepatch, + patchwork_submission, patchwork_patch, + patchwork_coverletter, + patchwork_series, + patchwork_seriespatch, + patchwork_seriesreference, patchwork_emailoptout, patchwork_patchchangenotification, patchwork_tag, @@ -42,7 +47,12 @@ GRANT SELECT, UPDATE ON patchwork_bundle_id_seq, patchwork_bundlepatch_id_seq, patchwork_comment_id_seq, + patchwork_submission_id_seq, patchwork_patch_id_seq, + patchwork_coverletter_id_seq, + patchwork_series_id_seq, + patchwork_seriespatch_id_seq, + patchwork_seriesreference_id_seq, patchwork_person_id_seq, patchwork_project_id_seq, patchwork_state_id_seq, @@ -55,9 +65,15 @@ GRANT SELECT, UPDATE ON patchwork_delegationrule_id_seq TO "www-data"; --- allow the mail user (in this case, 'nobody') to add patches +-- allow the mail user (in this case, 'nobody') to add submissions (patches, +-- cover letters) and series GRANT INSERT, SELECT ON + patchwork_submission, patchwork_patch, + patchwork_coverletter, + patchwork_series, + patchwork_seriespatch, + patchwork_seriesreference, patchwork_comment, patchwork_person TO "nobody"; @@ -71,7 +87,12 @@ GRANT SELECT ON patchwork_delegationrule TO "nobody"; GRANT UPDATE, SELECT ON + patchwork_submission_id_seq, patchwork_patch_id_seq, + patchwork_coverletter_id_seq, + patchwork_series_id_seq, + patchwork_seriespatch_id_seq, + patchwork_seriesreference_id_seq, patchwork_person_id_seq, patchwork_comment_id_seq, patchwork_patchtag_id_seq -- 2.9.3 From dja at axtens.net Mon Apr 10 08:25:07 2017 From: dja at axtens.net (Daniel Axtens) Date: Mon, 10 Apr 2017 08:25:07 +1000 Subject: [PATCH 2/3] 'mpe mode': click to copy patch IDs In-Reply-To: <1491754578.5029.7.camel@that.guru> References: <20170327054252.24337-1-dja@axtens.net> <20170327054252.24337-2-dja@axtens.net> <1491754578.5029.7.camel@that.guru> Message-ID: <8760idz0ak.fsf@possimpible.ozlabs.ibm.com> Stephen Finucane writes: > On Mon, 2017-03-27 at 16:42 +1100, Daniel Axtens wrote: >> If 'Show Patch IDs' is turned on in settings, add an extra column >> to the patch list, with buttons showing the patch IDs. The buttons >> copy the patch IDs to the clipboard. >> >> JavaScript inspired by https://github.com/Triforcey/clip-j and many >> many StackOverflow answers. >> >> Suggested-by: Michael Ellerman >> Signed-off-by: Daniel Axtens > > This looks good to me. I do wonder if there's any reason not to turn > this on by default though? My thought was just to avoid messing around too much with the default UI that people are used to, especially since it's not entirely obvious what the buttons do just by looking at them. Regards, Daniel > > Stephen > _______________________________________________ > Patchwork mailing list > Patchwork at lists.ozlabs.org > https://lists.ozlabs.org/listinfo/patchwork From stephen at that.guru Tue Apr 11 09:27:44 2017 From: stephen at that.guru (Stephen Finucane) Date: Tue, 11 Apr 2017 00:27:44 +0100 Subject: [PATCH 2/9] docs: Convert README, index In-Reply-To: <20170410232751.8489-1-stephen@that.guru> References: <20170410232751.8489-1-stephen@that.guru> Message-ID: <20170410232751.8489-3-stephen@that.guru> We can remove the Markdown README but must keep the index page until all docs are migrated. Signed-off-by: Stephen Finucane --- README.md | 88 -------------------------------------------------------- README.rst | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ docs/index.rst | 80 ++++++++++++++++++++++++++++++++++++++++++++++----- 3 files changed, 164 insertions(+), 95 deletions(-) delete mode 100644 README.md create mode 100644 README.rst diff --git a/README.md b/README.md deleted file mode 100644 index 488dad4..0000000 --- a/README.md +++ /dev/null @@ -1,88 +0,0 @@ -# Patchwork - -[![Requirements Status][badge-req-img]][badge-req-ref] -[![Build Status][badge-travis-img]][badge-travis-ref] -[![Code Health][badge-landscape-img]][badge-landscape-ref] -[![Codecov][badge-codecov-img]][badge-codecov-ref] -[![Docs Status][badge-doc-img]][badge-doc-ref] -[![Stories in Ready][badge-waffle-img]][badge-waffle-ref] - -**Patchwork** is a patch tracking system for community-based projects. It is -intended to make the patch management process easier for both the project's -contributors and maintainers, leaving time for the more important (and more -interesting) stuff. - -Patches that have been sent to a mailing list are "caught" by the system, and -appear on a web page. Any comments posted that reference the patch are appended -to the patch page too. The project's maintainer can then scan through the list -of patches, marking each with a certain state, such as Accepted, Rejected or -Under Review. Old patches can be sent to the archive or deleted. - -Currently, Patchwork is being used for a number of open-source projects, mostly -subsystems of the Linux kernel. Although Patchwork has been developed with the -kernel workflow in mind, the aim is to be flexible enough to suit the majority -of community projects. - -# Development Installation - -[Docker][ref-docker] is the recommended installation methods for a Patchwork -development environment. To install Patchwork: - -1. Install [**Docker**][ref-docker] and [**docker-compose**][ref-compose]. -2. Clone the Patchwork repo: - - $ git clone https://github.com/getpatchwork/patchwork.git - -3. Build the images. This will download over 200MB from the internet: - - $ docker-compose build - -4. Run `docker-compose up`: - - $ docker-compose up - -The Patchwork instance will now be deployed at `http://localhost:8000/`. - -For more information, including helpful command line options and alternative -installation methods, refer to the [development installation -guide][docs-development]. - -# Talks and Presentations - -* [**Mailing List, Meet CI**][pdf-fosdem-2] - FOSDEM 2017 -* [**A New Patchwork**][pdf-fosdem-1] - FOSDEM 2016 -* [**Patches carved into stone tablets**][pdf-stone-tools] - Kernel Recipes - Conference 2016 -* [**Patchwork: reducing your patch workload**][pdf-plumbers] - Linux Plumbers - Conference 2011 - -# Additional Information - -For further information, please refer to the [docs][docs]. - -# Contact - -For bug reports, patch submissions or other questions, please use the -[Patchwork mailing list][pw-ml]. - -[badge-codecov-ref]: https://codecov.io/gh/getpatchwork/patchwork -[badge-codecov-img]: https://codecov.io/gh/getpatchwork/patchwork/branch/master/graph/badge.svg -[badge-doc-ref]: https://patchwork.readthedocs.io/en/latest/ -[badge-doc-img]: https://readthedocs.org/projects/patchwork/badge/?version=latest -[badge-landscape-ref]: https://landscape.io/github/getpatchwork/patchwork/master -[badge-landscape-img]: https://landscape.io/github/getpatchwork/patchwork/master/landscape.svg?style=flat -[badge-req-ref]: https://requires.io/github/getpatchwork/patchwork/requirements/?branch=master -[badge-req-img]: https://requires.io/github/getpatchwork/patchwork/requirements.svg?branch=master -[badge-travis-ref]: https://travis-ci.org/getpatchwork/patchwork -[badge-travis-img]: https://travis-ci.org/getpatchwork/patchwork.svg?branch=master -[badge-waffle-ref]: https://waffle.io/getpatchwork/patchwork -[badge-waffle-img]: https://badge.waffle.io/getpatchwork/patchwork.svg?label=ready&title=Ready -[docs]: https://patchwork.readthedocs.io/en/latest/ -[docs-development]: https://patchwork.readthedocs.io/en/latest/development/ -[pdf-fosdem-1]: https://speakerdeck.com/stephenfin/a-new-patchwork-bringing-ci-patch-tracking-and-more-to-the-mailing-list -[pdf-fosdem-2]: https://speakerdeck.com/stephenfin/mailing-list-meet-ci -[pdf-plumbers]: https://www.linuxplumbersconf.org/2011/ocw/system/presentations/255/original/patchwork.pdf -[pdf-stone-tools]: https://github.com/gregkh/presentation-stone-tools/blob/34a3963/stone-tools.pdf -[pw-ml]: https://ozlabs.org/mailman/listinfo/patchwork -[ref-compose]: https://docs.docker.com/compose/install/ -[ref-docker]: https://docs.docker.com/engine/installation/linux/ diff --git a/README.rst b/README.rst new file mode 100644 index 0000000..5a94a77 --- /dev/null +++ b/README.rst @@ -0,0 +1,91 @@ +========= +Patchwork +========= + +.. image:: https://requires.io/github/getpatchwork/patchwork/requirements/?branch=master + :target: https://requires.io/github/getpatchwork/patchwork/requirements.svg?branch=master + :alt: Requirements Status + +.. image:: https://codecov.io/gh/getpatchwork/patchwork/branch/master/graph/badge.svg + :target: https://codecov.io/gh/getpatchwork/patchwork + :alt: Codecov + +.. image:: https://landscape.io/github/getpatchwork/patchwork/master + :target: https://landscape.io/github/getpatchwork/patchwork/master/landscape.svg?style=flat + :alt: Code Health + +.. image:: https://travis-ci.org/getpatchwork/patchwork + :target: https://travis-ci.org/getpatchwork/patchwork.svg?branch=master + :alt: Build Status + +.. image:: https://readthedocs.org/projects/patchwork/badge/?version=latest + :target: http://patchwork.readthedocs.io/en/latest/?badge=latest + :alt: Documentation Status + +**Patchwork** is a patch tracking system for community-based projects. It is +intended to make the patch management process easier for both the project's +contributors and maintainers, leaving time for the more important (and more +interesting) stuff. + +Patches that have been sent to a mailing list are "caught" by the system, and +appear on a web page. Any comments posted that reference the patch are appended +to the patch page too. The project's maintainer can then scan through the list +of patches, marking each with a certain state, such as Accepted, Rejected or +Under Review. Old patches can be sent to the archive or deleted. + +Currently, Patchwork is being used for a number of open-source projects, mostly +subsystems of the Linux kernel. Although Patchwork has been developed with the +kernel workflow in mind, the aim is to be flexible enough to suit the majority +of community projects. + +Development Installation +------------------------ + +`Docker`_ is the recommended installation methods for a Patchwork +development environment. To install Patchwork: + +1. Install **`Docker`_** and **`docker-compose`_**. + +2. Clone the Patchwork repo:: + + $ git clone https://github.com/getpatchwork/patchwork.git + +3. Build the images. This will download over 200MB from the internet:: + + $ docker-compose build + +4. Run `docker-compose up`:: + + $ docker-compose up + +The Patchwork instance will now be deployed at `http://localhost:8000/`. + +For more information, including helpful command line options and alternative +installation methods, refer to the `documentation`_. + +Talks and Presentations +----------------------- + +* **`Mailing List, Meet CI`_** - FOSDEM 2017 + +* **`Patches carved into stone tablets`_** - Kernel Recipes Conference 2016 + +* **`A New Patchwork`_** - FOSDEM 2016 + +* **`Patchwork: reducing your patch workload`_** - Linux Plumbers Conference + 2011 + +Additional Information +---------------------- + +For further information, refer to the `documentation`_. + +Contact +------- + +For bug reports, patch submissions or other questions, use the `mailing list`_. + +.. _docker-compose: https://docs.docker.com/compose/install/ +.. _Docker: https://docs.docker.com/engine/installation/linux/ +.. _documentation: https://patchwork.readthedocs.io/ +.. _mailing list: https://ozlabs.org/mailman/listinfo/patchwork diff --git a/docs/index.rst b/docs/index.rst index 2a9ce05..7a9ce09 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -1,15 +1,81 @@ -Welcome to Patchwork's documentation! -===================================== +Patchwork +========= .. toctree:: :maxdepth: 2 :caption: Contents: +Patchwork is a patch tracking system for community-based projects. It is +intended to make the patch management process easier for both the project's +contributors and maintainers, leaving time for the more important (and more +interesting) stuff. +Patches that have been sent to a mailing list are 'caught' by the system, and +appear on a web page. Any comments posted that reference the patch are appended +to the patch page too. The project's maintainer can then scan through the list +of patches, marking each with a certain state, such as Accepted, Rejected or +Under Review. Old patches can be sent to the archive or deleted. -Indices and tables -================== +Currently, Patchwork is being used for a number of open-source projects, mostly +subsystems of the Linux kernel. Although Patchwork has been developed with the +kernel workflow in mind, the aim is to be flexible enough to suit the majority +of community projects. -* :ref:`genindex` -* :ref:`modindex` -* :ref:`search` +Download +-------- + +The latest version of Patchwork is available with git. To download: + +.. code-block:: shell + + $ git clone git://github.com/getpatchwork/patchwork + +Patchwork is distributed under the `GNU General Public License (v2)`__. + +__ http://www.gnu.org/licenses/gpl-2.0.html + +Design +------ + +Patchwork should supplement mailing lists, not replace them +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Patchwork isn't intended to replace a community mailing list; that's why you +can't comment on a patch in Patchwork. If this were the case, then there would +be two forums of discussion on patches, which fragments the patch review +process. Developers who don't use Patchwork would get left out of the +discussion. + +However, a future development item for Patchwork is to facilitate on-list +commenting, by providing a "send a reply to the list" feature for logged-in +users. + +Don't pollute the project's changelogs with Patchwork poop +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A project's changelogs are valuable - we don't want to add Patchwork-specific +metadata. + +Patchwork users shouldn't require a specific version control system +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Not everyone uses git for kernel development, and not everyone uses git for +Patchwork-tracked projects. + +It's still possible to hook other programs into Patchwork, using the pwclient +command-line client for Patchwork, or directly to the XML RPC interface. + +Getting Started +--------------- + +You should check out the `deployment/installation` and +`development/installation` guides for information on how to configure +Patchwork for production and development environments, respectively. + +Support +------- + +All questions and contributions should be sent to the +`Patchwork mailing list`__ + +__ https://ozlabs.org/mailman/listinfo/patchwork -- 2.9.3 From stephen at that.guru Tue Apr 11 09:27:42 2017 From: stephen at that.guru (Stephen Finucane) Date: Tue, 11 Apr 2017 00:27:42 +0100 Subject: [PATCH 0/9] Convert documentation to rST Message-ID: <20170410232751.8489-1-stephen@that.guru> This was suggested before but there wasn't any good reason to do it at the time. There is now: reno. reno is a tool that allows us to provide rich release notes alongside the code itself. It's used extensively in the OpenStack community and I'd like to introduce it here too to allow us to provide detailed release notes for the upcoming 2.0 release. This is a manual conversion to rST that should result in basically the same output. Stephen Finucane (9): docs: Add skeleton for Sphinx docs docs: Convert README, index docs: Covert usage guide docs: Convert deployment guide docs: Convert development guide docs: Remove old Markdown files Add reno for release notes management docs: Use ReadTheDocs theme docs: Migrate release notes to rST .gitignore | 3 + CHANGELOG.md | 189 ------- README.md | 88 --- README.rst | 91 +++ UPGRADING.md | 4 - docs/conf.py | 100 ++++ docs/deployment/index.rst | 8 + docs/deployment/installation.md | 489 ---------------- docs/deployment/installation.rst | 614 +++++++++++++++++++++ docs/deployment/{upgrading.md => upgrading.rst} | 104 ++-- docs/development/contributing.md | 59 -- docs/development/contributing.rst | 73 +++ docs/development/index.rst | 10 + docs/development/installation.md | 379 ------------- docs/development/installation.rst | 488 ++++++++++++++++ docs/development/{releasing.md => releasing.rst} | 49 +- docs/development/xmlrpc.md | 54 -- docs/development/xmlrpc.rst | 58 ++ docs/{index.md => index.rst} | 51 +- docs/releases/alpaca.rst | 10 + docs/releases/burlap.rst | 99 ++++ docs/releases/cashmere.rst | 82 +++ docs/releases/index.rst | 10 + docs/releases/unreleased.rst | 4 + docs/usage/delegation.md | 45 -- docs/usage/delegation.rst | 52 ++ docs/usage/headers.md | 34 -- docs/usage/headers.rst | 41 ++ docs/usage/index.rst | 21 + docs/usage/{overview.md => overview.rst} | 315 ++++++----- docs/usage/{rest.md => rest.rst} | 31 +- docs/usage/{xmlrpc.md => xmlrpc.rst} | 31 +- mkdocs.yml | 22 - releasenotes/config.yaml | 3 + .../notes/initial-reno-68c116ae9c5259a3.yaml | 107 ++++ tox.ini | 8 + 36 files changed, 2228 insertions(+), 1598 deletions(-) delete mode 100644 CHANGELOG.md delete mode 100644 README.md create mode 100644 README.rst delete mode 100644 UPGRADING.md create mode 100644 docs/conf.py create mode 100644 docs/deployment/index.rst delete mode 100644 docs/deployment/installation.md create mode 100644 docs/deployment/installation.rst rename docs/deployment/{upgrading.md => upgrading.rst} (60%) delete mode 100644 docs/development/contributing.md create mode 100644 docs/development/contributing.rst create mode 100644 docs/development/index.rst delete mode 100644 docs/development/installation.md create mode 100644 docs/development/installation.rst rename docs/development/{releasing.md => releasing.rst} (61%) delete mode 100644 docs/development/xmlrpc.md create mode 100644 docs/development/xmlrpc.rst rename docs/{index.md => index.rst} (66%) create mode 100644 docs/releases/alpaca.rst create mode 100644 docs/releases/burlap.rst create mode 100644 docs/releases/cashmere.rst create mode 100644 docs/releases/index.rst create mode 100644 docs/releases/unreleased.rst delete mode 100644 docs/usage/delegation.md create mode 100644 docs/usage/delegation.rst delete mode 100644 docs/usage/headers.md create mode 100644 docs/usage/headers.rst create mode 100644 docs/usage/index.rst rename docs/usage/{overview.md => overview.rst} (62%) rename docs/usage/{rest.md => rest.rst} (73%) rename docs/usage/{xmlrpc.md => xmlrpc.rst} (52%) delete mode 100644 mkdocs.yml create mode 100644 releasenotes/config.yaml create mode 100644 releasenotes/notes/initial-reno-68c116ae9c5259a3.yaml -- 2.9.3 From stephen at that.guru Tue Apr 11 09:27:43 2017 From: stephen at that.guru (Stephen Finucane) Date: Tue, 11 Apr 2017 00:27:43 +0100 Subject: [PATCH 1/9] docs: Add skeleton for Sphinx docs In-Reply-To: <20170410232751.8489-1-stephen@that.guru> References: <20170410232751.8489-1-stephen@that.guru> Message-ID: <20170410232751.8489-2-stephen@that.guru> This is mostly the output of 'sphinx-quickstart' with all non-HTML build cruft removed and Sphinx minimum version set to 1.5. A tox target is included and the output of the docs build ignored. Signed-off-by: Stephen Finucane --- .gitignore | 3 ++ docs/conf.py | 97 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ docs/index.rst | 15 +++++++++ tox.ini | 5 +++ 4 files changed, 120 insertions(+) create mode 100644 docs/conf.py create mode 100644 docs/index.rst diff --git a/.gitignore b/.gitignore index d295c3c..31fea23 100644 --- a/.gitignore +++ b/.gitignore @@ -40,6 +40,9 @@ htmlcov/ # Django stuff *.log +# Sphinx stuff +/docs/_build + # Vagrant stuff /.vagrant diff --git a/docs/conf.py b/docs/conf.py new file mode 100644 index 0000000..88cadea --- /dev/null +++ b/docs/conf.py @@ -0,0 +1,97 @@ +# -*- coding: utf-8 -*- +# +# Patchwork documentation build configuration file +# +# This file is execfile()d with the current directory set to its +# containing dir. +# +# Note that not all possible configuration values are present in this +# autogenerated file. +# +# All configuration values have a default; values that are commented out +# serve to show the default. + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +# import os +# import sys +# sys.path.insert(0, os.path.abspath('.')) + + +# -- General configuration ------------------------------------------------ + +# If your documentation needs a minimal Sphinx version, state it here. +# +needs_sphinx = '1.5' + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [] + +# Add any paths that contain templates here, relative to this directory. +templates_path = [] + +# The suffix(es) of source filenames. +# You can specify multiple suffix as a list of string: +# +# source_suffix = ['.rst', '.md'] +source_suffix = '.rst' + +# The master toctree document. +master_doc = 'index' + +# General information about the project. +project = u'Patchwork' +copyright = u'2017, Stephen Finucane' +author = u'Stephen Finucane' + +# The version info for the project you're documenting, acts as replacement for +# |version| and |release|, also used in various other places throughout the +# built documents. +# +# TODO(stephenfin): Eventually this should read the version programmatically +# +# The short X.Y version. +version = u'2.0' +# The full version, including alpha/beta/rc tags. +release = u'2.0' + +# The language for content autogenerated by Sphinx. Refer to documentation +# for a list of supported languages. +# +# This is also used if you do content translation via gettext catalogs. +# Usually you set "language" from the command line for these cases. +language = None + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This patterns also effect to html_static_path and html_extra_path +exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store'] + +# The name of the Pygments (syntax highlighting) style to use. +pygments_style = 'sphinx' + +# If true, `todo` and `todoList` produce output, else they produce nothing. +todo_include_todos = False + + +# -- Options for HTML output ---------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'alabaster' + +# Theme options are theme-specific and customize the look and feel of a theme +# further. For a list of options available for each theme, see the +# documentation. +# +# html_theme_options = {} + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = [] diff --git a/docs/index.rst b/docs/index.rst new file mode 100644 index 0000000..2a9ce05 --- /dev/null +++ b/docs/index.rst @@ -0,0 +1,15 @@ +Welcome to Patchwork's documentation! +===================================== + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + + +Indices and tables +================== + +* :ref:`genindex` +* :ref:`modindex` +* :ref:`search` diff --git a/tox.ini b/tox.ini index bf38f8f..2d195cd 100644 --- a/tox.ini +++ b/tox.ini @@ -43,6 +43,11 @@ commands = flake8 {posargs} patchwork patchwork/bin/pwclient ignore = E129, F405 exclude = ./patchwork/migrations +[testenv:docs] +deps = sphinx>=1.5 +commands = + sphinx-build -W -b html docs docs/_build/html + [testenv:lint] basepython = python2.7 deps = -- 2.9.3 From stephen at that.guru Tue Apr 11 09:27:45 2017 From: stephen at that.guru (Stephen Finucane) Date: Tue, 11 Apr 2017 00:27:45 +0100 Subject: [PATCH 3/9] docs: Covert usage guide In-Reply-To: <20170410232751.8489-1-stephen@that.guru> References: <20170410232751.8489-1-stephen@that.guru> Message-ID: <20170410232751.8489-4-stephen@that.guru> Signed-off-by: Stephen Finucane --- docs/conf.py | 2 +- docs/index.rst | 3 +- docs/usage/delegation.rst | 52 +++++++ docs/usage/headers.rst | 41 ++++++ docs/usage/index.rst | 21 +++ docs/usage/overview.rst | 368 ++++++++++++++++++++++++++++++++++++++++++++++ docs/usage/rest.rst | 63 ++++++++ docs/usage/xmlrpc.rst | 46 ++++++ 8 files changed, 594 insertions(+), 2 deletions(-) create mode 100644 docs/usage/delegation.rst create mode 100644 docs/usage/headers.rst create mode 100644 docs/usage/index.rst create mode 100644 docs/usage/overview.rst create mode 100644 docs/usage/rest.rst create mode 100644 docs/usage/xmlrpc.rst diff --git a/docs/conf.py b/docs/conf.py index 88cadea..b23d033 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -29,7 +29,7 @@ needs_sphinx = '1.5' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. -extensions = [] +extensions = ['sphinx.ext.todo'] # Add any paths that contain templates here, relative to this directory. templates_path = [] diff --git a/docs/index.rst b/docs/index.rst index 7a9ce09..7b4d7c6 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -3,7 +3,8 @@ Patchwork .. toctree:: :maxdepth: 2 - :caption: Contents: + + usage/index Patchwork is a patch tracking system for community-based projects. It is intended to make the patch management process easier for both the project's diff --git a/docs/usage/delegation.rst b/docs/usage/delegation.rst new file mode 100644 index 0000000..eaabc20 --- /dev/null +++ b/docs/usage/delegation.rst @@ -0,0 +1,52 @@ +Autodelegation +============== + +Autodelegation allows patches to be automatically delegated to a user based on +the files modified by the patch. To do this, a number of rules can be +configured in the project administration page. This can usually be found at: + + /admin/patchwork/project//change + +.. note:: + + Autodelegation can only be configured by Patchwork administrators, i.e. + those that can access the 'admin' panel. If you require configuration of + autodelegation rules on a local instance, contact your Patchwork + administrator. + +In this section there are the following fields: + +User + + The patchwork user that should be autodelegated to the patch + +Priority + + The priority of the rule relative to other patches. Higher values indicate + higher priority. If two rules have the same priority, ordering will be based + on the path. + +Path + + A path in `fnmatch`__ format. The fnmatch library allows for limited, Unix + shell-style wildcarding. Filenames are extracted from patch lines beginning + with ``---`` or ``+++``. Note that for projects using Git or Mercurial, the + tools these VCS provide for producing patches are prefixed with `a` or `b`. + You should account for this in your path. For example, to match the path + `patchwork/views` (relative to the top of a Git repo) your pattern should + be:: + + ?/patchwork/views/* + + It is also possible to use relative paths, such as:: + + */manage.py + + For projects using other VCSs like Subversion can simply use a bare path:: + + patchwork/views/* + +Rules are configured by setting the above fields and saving the rules. These +rules will be applied at patch parse time. + +__ https://docs.python.org/2/library/fnmatch.html diff --git a/docs/usage/headers.rst b/docs/usage/headers.rst new file mode 100644 index 0000000..8741d9c --- /dev/null +++ b/docs/usage/headers.rst @@ -0,0 +1,41 @@ +Hint Headers +============ + +Patchwork provides a number of special email headers to control how a patch is +handled when it is received. The examples provided below use `git-send-email`, +but custom headers can also be set when using tools like `mutt`. + +`X-Patchwork-Ignore` + + Valid values: * + + When set, the mere presence of this header will ensure the provided email is + not parsed by Patchwork. For example: + + .. code-block:: shell + + $ git send-email --add-header="X-Patchwork-Ignore: test" master + +`X-Patchwork-Delegate` + + Valid values: An email address associated with a Patchwork user + + If set and valid, the user corresponding to the provided email address will + be assigned as the delegate of any patch parsed. If invalid, it will be + ignored. For example: + + .. code-block:: shell + + $ git send-email --add-header="X-Patchwork-Delegate: a at example.com" master + +`X-Patchwork-State` + + Valid values: Varies between deployments. This can usually be one of + "Accepted", "Rejected", "RFC" or "Awaiting Upstream", among others. + + If set and valid, the state provided will be assigned as the state of any + patch parsed. If invalid, it will be ignored. For example: + + .. code-block:: shell + + $ git send-email --add-header="X-Patchwork-State: RFC" master diff --git a/docs/usage/index.rst b/docs/usage/index.rst new file mode 100644 index 0000000..d79f6c8 --- /dev/null +++ b/docs/usage/index.rst @@ -0,0 +1,21 @@ +User Guide +========== + +Overview +-------- + +.. toctree:: + :maxdepth: 2 + + overview + delegation + headers + +APIs +---- + +.. toctree:: + :maxdepth: 2 + + rest + xmlrpc diff --git a/docs/usage/overview.rst b/docs/usage/overview.rst new file mode 100644 index 0000000..40f49e1 --- /dev/null +++ b/docs/usage/overview.rst @@ -0,0 +1,368 @@ +Overview +======== + +The key concepts or models of Patchwork are outlined below. + +Projects +-------- + +Projects typically represent a software project or sub-project. A Patchwork +server can host multiple projects. Each project can have multiple maintainers. +Projects usually have a 1:1 mapping with a mailing list, though it's also +possible to have multiple projects in the same list using the subject as +filter. Patches, cover letters, and series are all associated with a single +project. + +People +------ + +People are anyone who has submitted a patch, cover letter, or comment to a +Patchwork instance. + +Users +----- + +Users are anyone who has created an account on a given Patchwork instance. + +Standard Users +~~~~~~~~~~~~~~ + +A standard user can associate multiple email addresses with their user account, +create bundles and store TODO lists. + +Maintainers +~~~~~~~~~~~ + +Maintainers are a special type of user that with permissions to do certain +operations that regular Patchwork users can't. Patchwork maintainers usually +have a 1:1 mapping with a project's code maintainers though this is not +necessary. + +The operations that a maintainer can invoke include: + +- Change the state of a patch +- Archive a patch +- Delegate a patch, or be delegated a patch + +Submissions +----------- + +Patchwork captures three types of mail to mailing lists: patches, cover +letters, and replies to either patches or cover letters, a.k.a. comments. Any +mail that does not fit one of these categories is ignored. + +Patches +~~~~~~~ + +Patches are the central object in Patchwork structure. A patch contains both a +diff and some metadata, such as the name, the description, the author, the +version of the patch etc. Patchwork stores not only the patch itself but also +various metadata associated with the email that the patch was parsed from, such +as the message headers or the date the message itself was received. + +Cover Letters +~~~~~~~~~~~~~ + +Cover letters provide a way to offer a "big picture" overview of a series of +patches. When using Git, these mails can be recognised by way of their `0/N` +subject prefix, e.g. `[00/11] A sample series`. Like patches, Patchwork stores +not only the various aspects of the cover letter itself, such as the name and +body of the cover letter, but also various metadata associated with the email +that the cover letter was parsed from. + +Comments +~~~~~~~~ + +Comments are replies to a submission - either a patch or a cover letter. Unlike +a Mail User Agent (MUA) like Gmail, Patchwork does not thread comments. +Instead, every comment is associated with either a patch or a cover letter, and +organized by date. + +Patch Metadata +-------------- + +Patchwork allows users to store various metadata against patches. This metadata +is only configurable by a maintainer. + +States +~~~~~~ + +States track the state of patch in its lifecycle. States vary from project to +project, but generally a minimum subset of "new", "rejected" and "accepted" +will exist. + +Delegates +~~~~~~~~~ + +Delegates are Patchwork users who are responsible for both reviewing a patch +and setting its eventual state in Patchwork. This makes them akin to reviewers +in other tools. Delegation works particularly well for larger projects where +various subsystems, each with their own maintainer(s), can be identified. Only +one delegate can be assigned to a patch. + +.. note:: + + Patchwork supports automatic delegation of patches. Refer to + :doc:`delegation` for more information. + +Tags +~~~~ + +Tags are specially formatted metadata appended to the foot the body of a patch +or a comment on a patch. Patchwork extracts these tags at parse time and +associates them with the patch. You add extra tags to an email by replying to +the email. The following tags are available on a standard Patchwork install: + +Acked-by: + + For example:: + + Acked-by: Stephen Finucane + +Tested-by: + + For example:: + + Tested-by: Stephen Finucane + +Reviewed-by: + + For example:: + + Tested-by: Stephen Finucane + +The available tags, along with the significance of said tags, varies from +project to project and Patchwork instance to Patchwork instance. The `kernel +project documentation`__ provides an overview of the supported tags for the +Linux kernel project. + +__ https://www.kernel.org/doc/Documentation/SubmittingPatches + +Checks +~~~~~~ + +Checks store the results of any tests executed (or executing) for a given +patch. This is useful, for example, when using a continuous integration (CI) +system to test patches. Checks have a number of fields associated with them: + +Context + + A label to discern check from the checks of other testing systems + +Description + + A brief, optional description of the check + +Target URL + + A target URL where a user can find information related to this check, such as + test logs. + +State + + The state of the check. One of: pending, success, warning, fail + +User + + The user creating the check + +.. note:: + + Checks can only be created through the Patchwork APIs. Refer to `../api` + for more information. + +.. todo:: + + Provide information on building a CI system that reports check results back + to Patchwork. + +Collections +----------- + +Patchwork provides a number of ways to store groups of patches. Some of these +are automatically generated, while others are user-defined. + +Series +~~~~~~ + +Series are groups of patches, along with an optional cover letter. Series are +mostly dumb containers, though they also contain some metadata themselves such +as a version (which is inherited by the patches and cover letter) and a count +of the number of patches found in the series. + +Bundles +~~~~~~~ + +Bundles are custom, user-defined groups of patches. Bundles can be used to keep +patch lists, preserving order, for future inclusion in a tree. There's no +restriction of number of patches and they don't even need to be in the same +project. A single patch also can be part of multiple bundles at the same time. +An example of Bundle usage would be keeping track of the Patches that are ready +for merge to the tree. + +To-do Lists +~~~~~~~~~~~ + +Patchwork users can store a to-do list of patches. + +Events +------ + +Events are raised whenever patches are created or modified. + +All events have a number of common properties, along with some event-specific +properties: + +`category` + + The type of event + +`project` + + The project this event belongs to + +`date` + + When this event was created + +.. note:: + + Checks can only be created and read through the Patchwork APIs. Refer to + `../api/index` for more information. + +Cover Letter Created +~~~~~~~~~~~~~~~~~~~~ + +Sent when a cover letter is created. + +`category` + + ``cover-created`` + +`cover` + + Created cover letter + +Patch Created +~~~~~~~~~~~~~ + +Sent when a patch is created. + +`category` + + ``patch-created`` + +`patch` + + Created patch + +Patch Completed +~~~~~~~~~~~~~~~ + +Sent when a patch in a series has its dependencies met, or when a patch that is +not in a series is created (since that patch has no dependencies). + +`category` + + ``patch-completed`` + +`patch` + + Completed patch + +`series` + + Series from which patch dependencies were extracted, if any + +Patch Delegated +~~~~~~~~~~~~~~~ + +Sent when a patch's delegate is changed. + +`category` + + ``patch-delegated`` + +`patch` + + Updated patch + +`previous` + + Previous delegate, if any + +`current` + + Current delegate, if any + +Patch State Changed +~~~~~~~~~~~~~~~~~~~ + +Sent when a patch's state is changed. + +`category` + + ``patch-state-changed`` + +`patch` + + Updated patch + +`previous` + + Previous state + +`current` + + Current state + +Check Created +~~~~~~~~~~~~~ + +Sent when a patch check is created. + +`category` + + ``check-created`` + +`check` + + Created check + +Series Created +~~~~~~~~~~~~~~ + +Sent when a series is created. + +`category` + + ``series-created`` + +`series` + + Created series + +Series Completed +~~~~~~~~~~~~~~~~ + +Sent when a series is completed. + +`category` + + ``series-completed`` + +`series` + + Completed series + +What's Not Exposed +~~~~~~~~~~~~~~~~~~ + +* Bundles + + We don't expose an "added to bundle" event as it's unlikely that this will + be useful to either users or CI setters. + +* Comments + + Like Bundles, there likely isn't much value in exposing these via the API. diff --git a/docs/usage/rest.rst b/docs/usage/rest.rst new file mode 100644 index 0000000..8d6e036 --- /dev/null +++ b/docs/usage/rest.rst @@ -0,0 +1,63 @@ +The REST API +============ + +Patchwork provides a REST API. This API can be used to retrieve and modify +information about patches, projects and more. + +.. note:: + + The REST API was introduced in Patchwork v2.0. Users of earlier Patchwork + versions should instead refer to :doc:`xmlrpc`. + +Documentation +------------- + +Patchwork provides automatically generated documentation for the RESET API. +You can find this at the following URL: + + http://patchwork.example.com/api/ + +where `patchwork.example.com` refers to the URL of your Patchwork instance. + +Interacting with the API +------------------------ + +REST APIs run over plain HTTP(S), thus, the API can be interfaced using +applications or libraries that support this widespread protocol. One such +application is `curl`__, which can be used to both retrieve and send +information to the REST API. For example, to get the version of the REST API +for a Patchwork instance hosted at `patchwork.example.com`, run: + +.. code-block:: shell + + $ curl -s http://localhost:8000/api/1.0/ | python -m json.tool + { + "patches": "http://localhost:8000/api/1.0/patches/", + "people": "http://localhost:8000/api/1.0/people/", + "projects": "http://localhost:8000/api/1.0/projects/", + "users": "http://localhost:8000/api/1.0/users/" + } + +In addition, a huge variety of libraries are avaiable for interacting with and +parsing the output of REST APIs. The `requests`__ library is wide-spread and +well-supported. To repeat the above example using `requests`: + +.. code-block:: pycon + + $ python + >>> import json + >>> import requests + >>> r = requests.get('http://patchwork.example.com/api/1.0/') + >>> print(json.dumps(r.json(), indent=2)) + { + "users": "http://localhost:8000/api/1.0/users/", + "patches": "http://localhost:8000/api/1.0/patches/", + "projects": "http://localhost:8000/api/1.0/projects/", + "people": "http://localhost:8000/api/1.0/people/" + } + +Tools like `curl` and libraries like `requests` can be used to build anything +from small utilities to full-fledged clients targeting the REST API. + +__ https://curl.haxx.se/ +__ http://docs.python-requests.org/en/master/ diff --git a/docs/usage/xmlrpc.rst b/docs/usage/xmlrpc.rst new file mode 100644 index 0000000..644e8e2 --- /dev/null +++ b/docs/usage/xmlrpc.rst @@ -0,0 +1,46 @@ +The XML-RPC API +=============== + +.. note:: + + This guide covers usage information for the Patchwork XML-RPC API. For + information on using the REST API, refer to :doc:`rest`. For information on + developing custom applications or clients for this API, refer to the + `../development/xmlrpc`. + +Patchwork provides an XML-RPC API. This API can be used to be used to retrieve +and modify information about patches, projects and more. + +.. note:: + + The XML-RPC API can be enabled/disabled by the administrator: it may not be + available in every instance. + +pwclient +-------- + +The `pwclient` application, provided with Patchwork, can be used to interact +with Patchwork from the command line. Functionality provided by `pwclient` +includes: + +* Listing patches, projects, and checks +* Downloading and applying patches to a local code base +* Modifying the status of patches +* Creating new checks + +pwclient can be downloaded from the `Ozlabs Patchwork instance`__, or at the +following path for other Patchwork instances: + + http://patchwork.example.com/pwclient/ + +where `patchwork.example.com` corresponds to the URL a Patchwork instance is +hosted at. + +Once downloaded, to view information about all the operations supported by +`pwclient`, run: + +.. code-block:: shell + + $ pwclient --help + +__ https://patchwork.ozlabs.org/pwclient/ -- 2.9.3 From stephen at that.guru Tue Apr 11 09:27:47 2017 From: stephen at that.guru (Stephen Finucane) Date: Tue, 11 Apr 2017 00:27:47 +0100 Subject: [PATCH 5/9] docs: Convert development guide In-Reply-To: <20170410232751.8489-1-stephen@that.guru> References: <20170410232751.8489-1-stephen@that.guru> Message-ID: <20170410232751.8489-6-stephen@that.guru> Signed-off-by: Stephen Finucane --- docs/development/contributing.rst | 73 ++++++ docs/development/index.rst | 10 + docs/development/installation.rst | 488 ++++++++++++++++++++++++++++++++++++++ docs/development/releasing.rst | 60 +++++ docs/development/xmlrpc.rst | 58 +++++ docs/index.rst | 1 + 6 files changed, 690 insertions(+) create mode 100644 docs/development/contributing.rst create mode 100644 docs/development/index.rst create mode 100644 docs/development/installation.rst create mode 100644 docs/development/releasing.rst create mode 100644 docs/development/xmlrpc.rst diff --git a/docs/development/contributing.rst b/docs/development/contributing.rst new file mode 100644 index 0000000..989bc66 --- /dev/null +++ b/docs/development/contributing.rst @@ -0,0 +1,73 @@ +Contributing +============ + +Coding Standards +---------------- + +**Follow PEP8**. All code is currently PEP8 compliant and it should stay this +way. + +Changes that fix semantic issues will be generally be happily received, but +please keep such changes separate from functional changes. + +`pep8` targets are provided via tox. Refer to the :ref:`testing` section +below for more information on usage of this tool. + +.. _testing: + +Testing +------- + +Patchwork includes a `tox`__ script to automate testing. This requires a +functional database and some Python requirements like `tox`. Refer to the +:doc:`index` for information on how to configure these. + +Assuming these requirements are met, actually testing Patchwork is quite easy +to do. To start, you can show the default targets like so: + +.. code-block:: shell + + $ tox --list + +You'll see that this includes a number of targets to run unit tests against the +different versions of Django supported, along with some other targets related +to code coverage and code quality. To run one of these, use the `-e` parameter: + +.. code-block:: shell + + $ tox -e py27-django18 + +In the case of the unit tests targets, you can also run specific tests by +passing the fully qualified test name as an additional argument to this +command: + +.. code-block:: shell + + $ tox -e py27-django18 patchwork.tests.SubjectCleanUpTest + +Because Patchwork support multiple versions of Django, it's very important that +you test against all supported versions. When run without argument, tox will do +this: + +.. code-block:: shell + + $ tox + +__ https://tox.readthedocs.io/en/latest/ + +Submitting Changes +------------------ + +All patches should be sent to the `mailing list`__. When doing so, please abide +by the `QEMU guidelines`__ on contributing or submitting patches. This covers +both the initial submission and any follow up to the patches. In particular, +ensure: + +* :ref:`All tests pass ` + +* Documentation has been updated with new requirements, new script names etc. + +* The `CHANGES` file has been updated with any added or removed features + +__ https://ozlabs.org/mailman/listinfo/patchwork +__ http://wiki.qemu.org/Contribute/SubmitAPatch diff --git a/docs/development/index.rst b/docs/development/index.rst new file mode 100644 index 0000000..74857f6 --- /dev/null +++ b/docs/development/index.rst @@ -0,0 +1,10 @@ +Development Guide +================= + +.. toctree:: + :maxdepth: 2 + + installation + contributing + releasing + xmlrpc diff --git a/docs/development/installation.rst b/docs/development/installation.rst new file mode 100644 index 0000000..c8049af --- /dev/null +++ b/docs/development/installation.rst @@ -0,0 +1,488 @@ +Installation +============ + +This document describes the necessary steps to configure Patchwork in a +development environment. If you are interested in deploying Patchwork in a +production environment, please refer to [the deployment guide][doc-deployment] +instead. + +To begin, you should clone Patchwork: + +.. code-block:: shell + + $ git clone git://github.com/getpatchwork/patchwork.git + +Docker-Based Installation +------------------------- + +Patchwork provides a Docker-based environment for quick configuration of a +development environment. This is the preferred installation method. To +configure Patchwork using Docker: + +1. Install **`docker`_** and **`docker-compose`_**. + +2. Build the images. This will download over 200MB from the internet: + + .. code-block:: shell + + $ docker-compose build + +3. Run `docker-compose up`: + + .. code-block:: shell + + $ docker-compose up + + This will be visible at http://localhost:8000/. + +To run a shell within this environment, run: + +.. code-block:: shell + + $ docker-compose run --rm web --shell + +To run `django-manage` commands, such as `createsuperuser` or `migrate`, run: + +.. code-block:: shell + + $ docker-compose run --rm web python manage.py createsuperuser + +To run unit tests, excluding Selenium UI interaction tests, using only the +package versions installed during container initialization, run: + +.. code-block:: shell + + $ docker-compose run --rm web --quick-test + +To run the same against all supported versions of Django (via tox), run: + +.. code-block:: shell + + $ docker-compose run --rm web --quick-tox + +To run specific tox targets or tests, pass arguments to the above: + +.. code-block:: shell + + $ docker-compose run --rm web --quick-tox -e py27-django17 \ + patchwork.tests.test_bundles + +To run all tests, including Selenium UI interaction tests, using only the +package versions installed container initialization, run: + +.. code-block:: shell + + $ docker-compose run --rm web --test + +To run the same against all supported versions of Django (via tox), run: + +.. code-block:: shell + + $ docker-compose run --rm web --tox + +To run all tests, including Selenium UI interaction tests in non-headless mode, +run: + +.. code-block:: shell + + $ docker run -it --rm -v (pwd):/home/patchwork/patchwork/ \ + --link patchwork_db_1:db -p 8000:8000 \ + -v /tmp/.X11-unix:/tmp/.X11-unix \ + -e PW_TEST_DB_HOST=db -e DISPLAY patchwork_web bash + +To reset the database before any of these commands, add `--reset` to the +command line after `web` and before any other arguments. + +Any local edits to the project files made locally are immediately visible to +the Docker container, and so should be picked up by the Django auto-reloader. + +For more information on Docker itself, please refer to the `docker`_ and +`docker-compose`_ documentation. + +.. note:: + + If using SELinux, you will need to create a custom SELinux rule to allow the + Docker process to access your working directory. Run: + + .. code-block:: shell + + $ chcon -RT svirt_sandbox_file_t $PATCHWORK_DIR + + where `$PATCHWORK_DIR` is the absolute patch to the `patchwork` folder + created when you cloned the repo. For more information, see `man docker + run`. + +.. note:: + + If you see an error like the below:: + + ERROR: Couldn't connect to the Docker daemon at http+docker://localunixsocket - is it running? + + ensure you have correctly installed Docker, added your user to the `docker` + group, and started the daemon, per the `docker documentation `_. + +.. note:: + + If you see an error like the below:: + + py.error.EACCES: [Permission denied]: open('/home/patchwork/patchwork/.tox/py27-django18/.tox-config1', 'w') + + your host user account is likely using a different UID to the one hardcoded + in the Dockerfile. You can confirm this like so: + + .. code-block:: shell + + $ echo $UID + 1234 + + If this is anything other than `1000`, you must must modify the `Dockerfile` + found in `tools/docker` to use your UID and then rebuild: + + .. code-block:: shell + + $ sed -i "/ARG UID=/c\ARG UID=$(echo $UID)" tools/docker/Dockerfile + $ docker-compose build web + + This change must be retained in the event that you rebuild the container. + You can "hide" the change from Git like so: + + .. code-block:: shell + + $ git update-index --assume-unchanged tools/docker/Dockerfile + $ git update-index --skip-worktree tools/docker/Dockerfile + + This should be resolved in a future release when we support docker-compose + 2.1 syntax in `docker-compose.yml`. + +.. _docker: https://docs.docker.com/compose/install/ +.. _docker-compose: https://docs.docker.com/engine/installation/linux/ + +Vagrant-Based Installation +-------------------------- + +Patchwork provides a Vagrant-based environment as an alternative to Docker. +Like Docker, Vagrant can be used to quickly configure Patchwork in a +development environment. To configure Patchwork using Vagrant: + +1. Install `**Vagrant** `_ + +2. Run `vagrant up` from the project directory: + + .. code-block:: shell + + $ cd patchwork + $ vagrant up + +Once stacked, follow the on-screen instructions. For more information on +Vagrant itself, refer to the `Vagrant documentation `_. + +Manual Installation +------------------- + +Manual installation can be used where use of Docker or Vagrant is not possible +or desired. + +Install Required Packages +~~~~~~~~~~~~~~~~~~~~~~~~~ + +There are a number of different requirements for developing Patchwork: + +* Python and libraries + +* A supported database (RDBMS) + +These are detailed below. + +Python Requirements +^^^^^^^^^^^^^^^^^^^ + +To develop Python-based software you first need Python. Patchwork supports both +Python 2.7 and Python 3.3+. One of these will be installed by default on many +installations, though they can also be installed manually using the `python` or +`python3` packages. + +It's a good idea to use `virtual environments`__ to develop Python software. +Virtual environments are "instances" of your system Python without any of the +additional Python packages installed. They are useful to develop and possibly +deploy Patchwork against a "well known" set of dependencies, but they can also +be used to test Patchwork against several versions of Django. + +If you do not have `virtualenv` installed then you should install it now. This +can be installed using the `python-virtualenv` or `python3-virtualenv` +packages. Alternatively you can install these using `pip`. + +It is also helpful to install [`tox`][ref-tox] which is used for running tests +in Patchwork. This can be installed using the `python-tox` or `python3-tox` +packages, or via `pip`. + +__ https://virtualenv.readthedocs.io/en/latest/ + +Database Requirements +^^^^^^^^^^^^^^^^^^^^^ + +If not already installed, you may need to install an RDBMS. You can use either +MariaDB/MySQL or PostgreSQL for this purpose. You should also install the +development headers, known as `libmysqlclient-dev` or `libpq-dev` respectively +on Debian-based Debian-based distros like Ubuntu and `mysql-devel` or +`postgresql-devel` on RHEL-based distros. + +.. note:: + + While Django provides support for `multiple database backends`__, Patchwork + itself is only tested against MySQL/MariaDB and PostgreSQL. Should you wish + to use a different backend, ensure you validate this first (and perhaps + upstream any changes you may find necessary). + +.. note:: + + You may be tempted to use SQLite to develop Patchwork. We'd advise against + doing this. SQLite supports a subset of the functionality of "full" RDBMS + like MySQL: for example, case-sensitive matching of Unicode `is not + supported`__. You will find some tests provided by Patchwork fail and some + patches you develop may fail in production due to these differences. + +__ https://docs.djangoproject.com/en/1.8/ref/databases/ +__ https://www.sqlite.org/faq.html#q18 + +Example Installation +^^^^^^^^^^^^^^^^^^^^ + +An example for installing all these packages and the MySQL RDBMS on Ubuntu +15.04 is given below: + +.. code-block:: shell + + $ sudo apt-get install python python-pip python-dev python-virtualenv \ + python-tox mysql-server libmysqlclient-dev + +If you have an existing MariaDB/MySQL installation and have installed `pip` +already/are using Python 3.4+ then you can install all packages using `pip`: + +.. code-block:: shell + + $ sudo pip install virtualenv tox + +If you wish to use Python 3 then simply replace `python` with `python3` in the +above command. + +Configure Virtual Environment +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +.. note:: + + If you are interested in simply :ref:`testing Patchwork `, many of + the below steps are not required. tox will automatically install + dependencies and use virtual environments when testing. + +Once these requirements are installed, you should create and activate a new +virtual environment. This can be done like so: + +.. code-block:: shell + + $ virtualenv .venv + $ source .venv/bin/activate + (.venv)$ + +.. note:: + + If you installed a Python 3.x-based virtual environment package, adjust the + executable indicated above as necessary, e.g. `virtualenv-3.4`. + +Now install the packages. Patchwork provides three requirements files. + +`requirements-dev.txt` + + Packages required to configure a development environment + +`requirements-prod.txt` + + Packages required for deploying Patchwork in production + +`requirements-test.txt` + + Packages required to run tests + +We're going to install the first of these, which can be done like so: + +.. code-block:: shell + + (.venv)$ cd patchwork + (.venv)$ pip install -r requirements-dev.txt + +.. note:: + + Once configured this does not need to be done again *unless* the + requirements change, e.g. Patchwork requires an updated version of Django. + +Initialize the Database +~~~~~~~~~~~~~~~~~~~~~~~ + +One installed, the database must be configured. We will assume you have root +access to the database for these steps. + +To begin, export your database credentials as follows: + +.. code-block:: shell + + (.venv)$ db_user=root + (.venv)$ db_pass=password + +Now, create the database. If this is your first time configuring the database, +you must create a `patchwork` user (or similar) along with the database +instance itself. The commands below will do this, dropping existing databases +if necessary: + +.. code-block:: shell + + (.venv)$ mysql -u$db_user -p$db_pass << EOF + DROP DATABASE IF EXISTS patchwork; + CREATE DATABASE patchwork CHARACTER SET utf8; + GRANT ALL PRIVILEGES ON patchwork.* TO 'patchwork'@'localhost' + IDENTIFIED BY 'password'; + EOF + +.. note:: + + The `patchwork` username and `password` password are the defaults expected + by the provided `dev` settings files. If using something different, + export the `PW_TEST_DB_USER` and `PW_TEST_DB_PASS` variables described in + the :ref:`Environment Variables ` section below. Alternatively, + you can create your own settings file with these variables hardcoded and + change the value of `DJANGO_SETTINGS_MODULE` as described below. + +Load Initial Data +~~~~~~~~~~~~~~~~~ + +Before continuing, we need to tell Django where it can find our configuration. +Patchwork provides a default development `settings.py` file for this purpose. +To use this, export the `DJANGO_SETTINGS_MODULE` environment variable as +described below: + +.. code-block:: shell + + (.venv)$ export DJANGO_SETTINGS_MODULE=patchwork.settings.dev + +Alternatively you can provide your own `settings.py` file and provide the path +to that instead. + +Once done, we need to create the tables in the database. This can be done using +the `migrate` command of the `manage.py` executable: + +.. code-block:: shell + + (.venv)$ ./manage.py migrate + +Next, you should load the initial fixtures into Patchwork. These initial +fixtures provide. + +`default_tags.xml` + + The tags that Patchwork will extract from mails. For example: `Acked-By`, + `Reviewed-By` + +`default_states.xml` + + The states that a patch can be in. For example: `Accepted`, `Rejected` + +`default_projects.xml` + + A default project that you can then upload patches for + +These can be loaded using the `loaddata` command: + +.. code-block:: shell + + (.venv)$ ./manage.py loaddata default_tags default_states default_projects + +You should also take the opportunity to create a "superuser". You can do this +using the aptly-named `createsuperuser` command: + +.. code-block:: shell + + (.venv)$ ./manage.py createsuperuser + +Import Mailing List Archives +---------------------------- + +Regardless of your installation method of choice, you will probably want to +load some real emails into the system. This can be done manually, however it's +generally much easier to download an archive from a Mailman instance and load +these using the `parsearchive` command. You can do this like so: + +.. code-block:: shell + + (.venv)$ mm_user= + (.venv)$ mm_pass= + (.venv)$ mm_host=https://lists.ozlabs.org + (.venv)$ mm_url=$mm_host/private/patchwork.mbox/patchwork.mbox + (.venv)$ curl -F username=$mm_user -F password=$mm_pass -k -O $mm_url + +where `mm_user` and `mm_pass` are the username and password you have registered +with on the Mailman instance found at `mm_host`. + +.. note:: + + We provide instructions for downloading archives from the Patchwork mailing + list, but almost any instance of Mailman will allow downloading of archives + as seen above; simply change the `pw_url` variable defined. You can find + more informations about this `here`__. + +Load these archives into Patchwork. Depending on the size of the downloaded +archives this may take some time: + +.. code-block:: shell + + (.venv)$ ./manage.py parsearchive --list-id=patchwork.ozlabs.org \ + patchwork.mbox + +Finally, run the server and browse to the IP address of your board using your +browser of choice: + +.. code-block:: shell + + (.venv)$ ./manage.py runserver 0.0.0.0:8000 + +Once finished, you can kill the server (`Ctrl` + `C`) and exit the virtual +environment: + +.. code-block:: shell + + (.venv)$ deactivate + $ + +Should you wish to re-enter this environment, simply source the `activate` +script again. + +__ http://blog.behnel.de/posts/indexp118.html + +Django Debug Toolbar +-------------------- + +Patchwork installs and enables the 'Django Debug Toolbar' by default. However, +by default this is only displayed if you are developing on localhost. If +developing on a different machine, you should configure an SSH tunnel such +that, for example, `localhost:8000` points to `[DEV_MACHINE_IP]:8000`. + +.. _dev-envvar: + +Environment Variables +--------------------- + +The following environment variables are available to configure settings when +using the provided `dev` settings file. + +`PW_TEST_DB_NAME=patchwork` + + Name of the database + +`PW_TEST_DB_USER=patchwork` + + Username to access the database with + +`PW_TEST_DB_PASS=password` + + Password to access the database with< + +`PW_TEST_DB_TYPE=mysql` + + Type of database to use. Options: 'mysql', 'postgres' diff --git a/docs/development/releasing.rst b/docs/development/releasing.rst new file mode 100644 index 0000000..3f3a8ee --- /dev/null +++ b/docs/development/releasing.rst @@ -0,0 +1,60 @@ +Release Process +=============== + +Versioning +---------- + +Since version 1.0, Patchwork has implemented a version of `Semantic +Versioning`__ . To summarise, releases take the format **MAJOR.MINOR.PATCH** +(or just **MAJOR.MINOR**). We increment: + +1. **MAJOR** version when we make major UI changes or functionality updates + +2. **MINOR** version when we make minor UI changes or functionality updates + +3. **PATCH** version when we make make bug fixes, dependency updates etc. + +In Git, each release will have a tag indicating the version number. In +addition, each release series has it's own branch called `stable/MAJOR.MINOR` +to allow backporting of bugfixes or security updates to older versions. + +__ http://semver.org/ + +Release Cycle +------------- + +There is no cadence for releases: they are made available as necessary. + +Supported Versions +------------------ + +Typically all development should occur on `master`. While we will backport +bugfixes and security updates, we will not backport any new features. This is +to ensure stability for users of these versions of Patchwork. + +Release Checklist +----------------- + +* Documentation has been updated with latest release version + +* Documentation references latest supported version of Django + +Backporting +----------- + +We will occasionally backport bugfixes and security updates. When backporting a +patch, said patch should first be merged into `master`. Once merged, you can +backport by cherry-picking commits, using the `-x` flag for posterity: + +.. code-block:: shell + + $ git cherry-pick -x + +There may be some conflicts; resolve these, uncommenting the `Conflicts` line +when commiting:: + + Conflicts + patchwork/bin/pwclient + +When enough patches have been backported, you should release a new `PATCH` +release. diff --git a/docs/development/xmlrpc.rst b/docs/development/xmlrpc.rst new file mode 100644 index 0000000..9b98767 --- /dev/null +++ b/docs/development/xmlrpc.rst @@ -0,0 +1,58 @@ +The XML-RPC API +=============== + +Patchwork provides an XML-RPC API. This API can be used to be used to retrieve +and modify information about patches, projects and more. + +This guide covers development information for the Patchwork XML-RPC API. For +information on using the REST API, refer to `rest`. For information on general +usage of the XML-RPC API, refer to :doc:`../usage/index`. + +.. note:: + + The XML-RPC API can be enabled/disabled by the administrator: it may not be + available in every instance. + +Documentation +------------- + +Patchwork provides automatically generated documentation for the XML-RPC API. +You can find this at the following URL: + + http://patchwork.example.com/xmlrpc/ + +where `patchwork.example.com` refers to the URL of your Patchwork instance. + +.. note:: + + Automatic documentation generation for the Patchwork API was introduced in + Patchwork v1.1. Prior versions of Patchwork do not offer this functionality. + +Interacting with the API +------------------------ + +The Patchwork XML-RPC API provides a number of "methods". Some methods require +authentication (via HTTP Basic Auth) while others do not. Authentication uses +your Patchwork account and the on-server documentation will indicate where it +is necessary. We will only cover the unauthenticated method here for brevity - +consult the `xmlrpclib`__ documentation for more detailed examples: + +To interact with the Patchwork XML-RPC API, a XML-RPC library should be used. +Python provides such a library - `xmlrpclib` - in its standard library. For +example, to get the version of the XML-RPC API for a Patchwork instance hosted +at `patchwork.example.com`, run: + +.. code-block:: pycon + + $ python + >>> import xmlrpclib # or 'xmlrpc.client' for Python 3 + >>> rpc = xmlrpclib.ServerProxy('http://patchwork.example.com/xmlrpc/') + >>> rpc.pw_rpc_version() + 1.1 + +Once connected, the `rpc` object will be populated with a list of available +functions (or procedures, in RPC terminology). In the above example, we used +the `pw_rpc_version` method, however, it should be possible to use all the +methods listed in the server documentation. + +__ https://docs.python.org/2/library/xmlrpclib.html diff --git a/docs/index.rst b/docs/index.rst index 7989b4e..f2e7fcd 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -6,6 +6,7 @@ Patchwork usage/index deployment/index + development/index Patchwork is a patch tracking system for community-based projects. It is intended to make the patch management process easier for both the project's -- 2.9.3 From stephen at that.guru Tue Apr 11 09:27:46 2017 From: stephen at that.guru (Stephen Finucane) Date: Tue, 11 Apr 2017 00:27:46 +0100 Subject: [PATCH 4/9] docs: Convert deployment guide In-Reply-To: <20170410232751.8489-1-stephen@that.guru> References: <20170410232751.8489-1-stephen@that.guru> Message-ID: <20170410232751.8489-5-stephen@that.guru> Signed-off-by: Stephen Finucane --- docs/deployment/index.rst | 8 + docs/deployment/installation.rst | 614 +++++++++++++++++++++++++++++++++++++++ docs/deployment/upgrading.rst | 152 ++++++++++ docs/index.rst | 1 + 4 files changed, 775 insertions(+) create mode 100644 docs/deployment/index.rst create mode 100644 docs/deployment/installation.rst create mode 100644 docs/deployment/upgrading.rst diff --git a/docs/deployment/index.rst b/docs/deployment/index.rst new file mode 100644 index 0000000..04ff4f4 --- /dev/null +++ b/docs/deployment/index.rst @@ -0,0 +1,8 @@ +Deployment Guide +================ + +.. toctree:: + :maxdepth: 2 + + installation + upgrading diff --git a/docs/deployment/installation.rst b/docs/deployment/installation.rst new file mode 100644 index 0000000..6ca5785 --- /dev/null +++ b/docs/deployment/installation.rst @@ -0,0 +1,614 @@ +Installation +============ + +This document describes the necessary steps to configure Patchwork in a +production environment. This requires a significantly "harder" deployment than +the one used for development. If you are interested in developing Patchwork, +refer to :doc:`../deployment/index` instead. + +This document describes a two-node installation of Patchwork, consisting of a +database sever and an application server. It should be possible to combine +these machines for smaller Patchwork instances. It should also be possible to +configure high availability deployment through use of additional database and +application machines, though this is out of the scope of this document. + +.. todo:: + + Update this guide to the latest Ubuntu LTS (16.04) + +Deployment Guides, Provisioning Tools and Platform-as-a-Service +--------------------------------------------------------------- + +Before continuing, it's worth noting that Patchwork is a Django application. +With the exception of the handling of incoming mail (described below), it +can be deployed like any other Django application. This means there are tens, +if not hundreds, of existing articles and blogs detailing how to deploy an +application like this. As such, if any of the below information is unclear +then we'd suggest you go search for "Django deployment guide" or similar, +deploy your application, and submit a patch for this guide to clear up that +confusion for others. + +You'll also find that the same search reveals a significant number of existing +deployment tools aimed at Django. These tools, be they written in Ansible, +Puppet, Chef or something else entirely, can be used to avoid much of the +manual configuration described below. If possible, embrace these tools to make +your life easier. + +Finally, many Platform-as-a-Service (PaaS) providers and tools support +deployment of Django applications with minimal effort. Should you wish to avoid +much of the manual configuration, we suggest you investigate the many options +available to find one that best suits your requirements. The only issue here +will likely be the handling of incoming mail - something which many of these +providers don't support. We address this in the appropriate section below. + +Requirements +------------ + +For the purpose of this guide, we will assume the following machines: + ++-------------+------------+ +| server role | IP address | ++-------------+------------+ +| database | 10.1.1.1 | +| application | 10.1.1.2 | ++-------------+------------+ + +We will use the database server to, ostensibly enough, host the database for +the Patchwork instance. The application server, on the other hand, will host +the Patchwork instance along with the required reverse proxy and WSGI HTTP +servers. + +We expect a Ubuntu 15.04 installation on each of these hosts: commands, package +names and/or package versions will likely change if using a different distro or +release. In addition, usage of different package versions to the ones suggested +may require slightly different configuration. + +Before beginning, you should update these systems: + +.. code-block:: shell + + $ sudo apt-get update + $ sudo apt-get upgrade + +We also need to configure some environment variables to ease deployment. These +should be exported on all systems: + +`PW_HOST_DB=10.1.1.1` + + IP of the database host + +`PW_HOST_APP=10.1.1.2` + + IP of the application host + +`PW_DB_NAME=patchwork` + + Name of the database + +`PW_DB_USER=www-data` + + Username that the Patchwork app will access the database with + +Database +-------- + +These steps should be run on the database server. + +.. note:: + + If you already have a database server on site, you can skip much of this + section. + +Install Requirements +~~~~~~~~~~~~~~~~~~~~ + +We're going to rely on PostgreSQL. You can adjust the below steps if using a +different RDBMS. Install the required packages. + +.. code-block:: shell + + $ sudo apt-get install -y postgresql postgresql-contrib + +Configure Database +~~~~~~~~~~~~~~~~~~ + +PostgreSQL created a user account called `postgres`; you will need to run +commands as this user. Use this account to create the database that Patchwork +will use, using the credentials we configured earlier. + +.. code-block:: shell + + $ sudo -u postgres createdb $PW_DB_NAME + $ sudo -u postgres createuser $PW_DB_USER + +We will also need to apply permissions to the tables in this database but +seeing as the tables haven't actually been created yet this will have to be +done later. + +.. todo:: + + Add `pg_hba.conf` configuration + +Patchwork +--------- + +These steps should be run on the application server. + +Install Packages +~~~~~~~~~~~~~~~~ + +The first requirement is Patchwork itself. It can be downloaded like so: + +.. code-block:: shell + + $ wget https://github.com/getpatchwork/patchwork/archive/v1.1.0.tar.gz + +We will install this under `/opt`, though this is only a suggestion: + +.. code-block:: shell + + $ tar -xvzf v1.1.0.tar.gz + $ sudo mv v1.1.0 /opt/patchwork + +.. important:: + + Per the `Django documentation`__, source code should not be placed in your + web server's document root as this risks the possibility that people may be + able to view your code over the Web. This is a security risk. + +__ https://docs.djangoproject.com/en/dev/intro/tutorial01/#creating-a-project + +Next we require Python. If not already installed, then you should do so now. +Patchwork supports both Python 2.7 and Python 3.3+, though we would suggest +using the latter to ease future upgrades: + +.. code-block:: shell + + $ sudo apt-get install python3 # or 'python' if using Python 2.7 + +We require a number of Python packages. These can be installed using `pip`: + +.. code-block:: shell + + $ sudo pip install -r /opt/patchwork/requirements-prod.txt + +If you're not using `pip`, you will need to identify and install the +corresponding distro package for each of these requirements. For example: + +.. code-block:: shell + + $ sudo apt-get install python3-django + +.. tip:: + + The `pkgs.org`__ website provides a great reference for identifying the name + of these dependencies. + +__ https://pkgs.org/ + +Configure Patchwork +~~~~~~~~~~~~~~~~~~~ + +You will also need to configure a `settings file`__ for Django. A sample +settings file is provided that defines default settings for Patchwork. You'll +need to configure settings for your own setup and save this as `production.py`. + +.. code-block:: shell + + $ cp patchwork/settings/production.example.py \ + patchwork/settings/production.py + +Alternatively, you can override the `DJANGO_SETTINGS_MODULE` environment +variable and provide a completely custom settings file. + +.. important:: + + You should not include shell variables in settings but rather hardcoded + values. These settings files are evaluated in Python - not a shell. + +__ https://docs.djangoproject.com/en/1.8/ref/settings/ + +Databases +^^^^^^^^^ + +You can configure the `DATABASES` setting using the variables we set earlier. + +.. code-block:: python + + DATABASES = { + 'default': { + 'ENGINE': 'django.db.backends.postgresql_psycopg2', + 'HOST': '$PW_HOST_DB', # don't use sh variables but actual values + 'PORT': '', + 'NAME': '$PW_DB_NAME', + 'USER': '$PW_DB_USER', + 'PASSWORD': '$PW_DB_PASS', + 'TEST': { + 'CHARSET': 'utf8', + }, + }, + } + +.. note:: + + `TEST/CHARSET` is used when creating tables for the test suite. Without it, + tests checking for the correct handling of non-ASCII characters fail. It is + not necessary if you don't plan to run tests, however. + +Static Files +^^^^^^^^^^^^ + +While we have not yet configured our proxy server, we do need to configure the +location that these files will be stored in. We will install these under +`/var/www/patchwork`, though this is only a suggestion and can be changed. + +.. code-block:: shell + + $ mkdir /var/www/patchwork + +You can configure this by setting the `STATIC_ROOT` variable. + +.. code-block:: shell + + STATIC_ROOT = '/var/www/patchwork' + +Other Options +^^^^^^^^^^^^^ + +Finally, the following settings need to be configured. The purpose of these +variables is described in the `Django documentation`__: + +* `SECRET_KEY` +* `ADMINS` +* `TIME_ZONE` +* `LANGUAGE_CODE` +* `DEFAULT_FROM_EMAIL` +* `NOTIFICATION_FROM_EMAIL` + +You can generate the `SECRET_KEY` with the following python code: + +.. code-block:: python + + import string, random + chars = string.letters + string.digits + string.punctuation + print repr("".join([random.choice(chars) for i in range(0,50)])) + +If you wish to enable the XML-RPC interface, you should add the following to +the file: + +.. code-block:: python + + ENABLE_XMLRPC = True + +__ https://docs.djangoproject.com/en/1.8/ref/settings/ + +Final Steps +~~~~~~~~~~~ + +Once done, we should be able to check that all requirements are met using the +`check` command of the `manage.py` executable: + +.. code-block:: shell + + $ /opt/patchwork/manage.py check + +We should also take this opportunity to both configure the database and static +files: + +.. code-block:: shell + + $ /opt/patchwork/manage.py migrate + $ /opt/patchwork/manage.py loaddata \ + /opt/patchwork/patchwork/fixtures/default_tags.xml + $ /opt/patchwork/manage.py loaddata \ + /opt/patchwork/patchwork/fixtures/default_states.xml + $ /opt/patchwork/manage.py collectstatic + +.. note:: + + The above `default_tags` and `default_states` are just that: defaults. You + can modify these to fit your own requirements. + +Finally, it may be helpful to start the development server quickly to ensure +you can see *something*: + +.. code-block:: shell + + $ /opt/patchwork/manage.py runserver 0.0.0.0:8080 + +Browse this instance at `http://[your_server_ip]:8000`. If everything is +working, kill the development server using `Ctrl`+`C`. + +Reverse Proxy and WSGI HTTP Servers +----------------------------------- + +These steps should be run on the application server. + +Install Packages +~~~~~~~~~~~~~~~~ + +We will use nginx and uWSGI to deploy Patchwork, acting as reverse proxy server +and WSGI HTTP server respectively. Other options are available, such as +Apache+mod_wsgi or nginx+Gunicorn. While we don't document these, sample +configuration files for the former case are provided in `lib/apache2/`. + +.. code-block:: shell + + $ sudo apt-get install nginx-full uwsgi uwsgi-plugin-python + +Configure nginx and uWSGI +~~~~~~~~~~~~~~~~~~~~~~~~~ + +Configuration files for nginx and uWSGI are provided in the `lib` subdirectory +of the Patchwork source code. These can be modified as necessary, but for now +we will simply copy them. + +First, let's load the provided configuration for nginx: + +.. code-block:: shell + + $ sudo cp /opt/patchwork/lib/nginx/patchwork.conf \ + /etc/nginx/sites-available/ + +If you wish to modify this configuration, now is the time to do so. Once done, +validate and enable your configuration: + +.. code-block:: shell + + $ sudo nginx -t + $ sudo ln -s /etc/nginx/sites-available/patchwork.conf \ + /etc/nginx/sites-enabled/patchwork.conf + +Now use the provided configuration for uWSGI: + +.. code-block:: shell + + $ sudo mkdir -p /etc/uwsgi/sites + $ sudo cp /opt/patchwork/lib/uwsgi/patchwork.ini \ + /etc/uwsgi/sites/patchwork.ini + +.. note:: + + We created the `/etc/uwsgi` directory above because we're going to run uWSGI + in `emperor mode`__. This has benefits for multi-app deployments. + +__ https://uwsgi-docs.readthedocs.io/en/latest/Emperor.html + +Create systemd Unit File +~~~~~~~~~~~~~~~~~~~~~~~~ + +As things stand, uWSGI will need to be started manually every time the system +boots, in addition to any time it may fail. We can automate this process using +systemd. To this end a `systemd unit file`__ should be created to start uWSGI +at boot: + +.. code-block:: shell + + $ sudo cat << EOF > /etc/systemd/system/uwsgi.service + [Unit] + Description=uWSGI Emperor service + + [Service] + ExecStartPre=/usr/bin/bash -c 'mkdir -p /run/uwsgi; chown user:nginx /run/uwsgi' + ExecStart=/usr/bin/uwsgi --emperor /etc/uwsgi/sites + Restart=always + KillSignal=SIGQUIT + Type=notify + NotifyAccess=all + + [Install] + WantedBy=multi-user.target + EOF + +.. note:: + + On older version of Ubuntu you may need to tweak these steps to use + `upstart`__ instead. + +__ https://uwsgi-docs.readthedocs.io/en/latest/Systemd.html +__ https://uwsgi-docs.readthedocs.io/en/latest/Upstart.html + +Final Steps +~~~~~~~~~~~ + +Start the uWSGI service we created above: + +.. code-block:: shell + + $ sudo systemctl start uwsgi + $ sudo systemctl status uwsgi + +Next up, restart the nginx service: + +.. code-block:: shell + + $ sudo systemctl restart nginx + $ sudo systemctl status nginx + +Patchwork uses a cron script to clean up expired registrations and send +notifications of patch changes (for projects with this enabled). Something like +this in your crontab should work. + +:: + + # m h dom mon dow command + */10 * * * * cd patchwork; ./manage.py cron + +.. note:: + + The frequency should be the same as the `NOTIFICATION_DELAY_MINUTES` + setting, which defaults to 10 minutes. + +Finally, browse to the instance using your browser of choice. + +You may wish to take this opportunity to setup your projects and configure your +website address (in the Sites section of the admin console, found at `/admin`). + +Django administrative console +----------------------------- + +In order to access the administrative console at `/admin`, you need at least +one user account to be registered and configured as a super user or staff +account to access the Django administrative console. This can be achieved by +doing the following: + +.. code-block:: shell + + $ /opt/patchwork/manage.py createsuperuser + +Once the administrative console is accessible, you would want to configure your +different sites and their corresponding domain names, which is required for the +different emails sent by patchwork (registration, password recovery) as well as +the sample `pwclientrc` files provided by your project's page. + +Incoming Email +-------------- + +Patchwork is designed to parse incoming mails which means you need an address +to receive email at. This is a problem that has been solved for many webapps, +thus there are many ways to go about this. Some of these ways are discussed +below. + +IMAP/POP3 +~~~~~~~~~ + +The easiest option for getting mail into Patchwork is to use an existing email +address in combination with a mail retriever like `getmail`__, which will +download mails from your inbox and pass them to Patchwork for processing. +getmail is easy to set up and configure: to begin, you need to install it: + +.. code-block:: shell + + $ sudo apt-get install getmail4 + +Once installed, you should configure it, substituting your own configuration +details where required below: + +.. code-block:: shell + + $ sudo cat << EOF > /etc/getmail/user at example.com/getmailrc + [retriever] + type = SimpleIMAPSSLRetriever + server = imap.example.com + port = 993 + username = XXX + password = XXX + mailboxes = ALL + + [destination] + # we configure Patchwork as a "mail delivery agent", in that it will + # handle our mails + type = MDA_external + path = /opt/patchwork/patchwork/bin/parsemail.sh + + [options] + # retrieve only new emails + read_all = false + # do not add a Delivered-To: header field + delivered_to = false + # do not add a Received: header field + received = false + EOF + +Validate that this works as expected by starting `getmail`: + +.. code-block:: shell + + $ getmail --getmaildir=/etc/getmail/user at example.com --idle INBOX + +If everything works as expected, you can create a systemd script to ensure this +starts on boot: + +.. code-block:: shell + + $ sudo cat << EOF > /etc/systemd/system/getmail.service + [Unit] + Description=Getmail for user at example.com + + [Service] + User=pathwork + ExecStart=/usr/bin/getmail --getmaildir=/etc/getmail/user at example.com --idle INBOX + Restart=always + + [Install] + WantedBy=multi-user.target + EOF + +And start the service: + +.. code-block:: shell + + $ sudo systemctl start getmail + $ sudo systemctl status getmail + +__ http://pyropus.ca/software/getmail/ + +Mail Transfer Agent (MTA) +~~~~~~~~~~~~~~~~~~~~~~~~~ + +The most flexible option is to configure our own mail transfer agent (MTA) or +"email server". There are many options, of which `Postfix`__ is one. While we +don't cover setting up Postfix here (it's complicated and there are many guides +already available), Patchwork does include a script to take received mails and +create the relevant entries in Patchwork for you. To use this, you should +configure your system to forward all emails to a given localpart (the bit +before the `@`) to this script. Using the `patchwork` localpart (e.g. +`patchwork at example.com`) you can do this like so: + +.. code-block:: shell + + $ sudo cat << EOF > /etc/aliases + patchwork: "|/opt/patchwork/patchwork/bin/parsemail.sh" + EOF + +You should ensure the appropriate user is created in PostgreSQL and that it has +(minimal) access to the database. Patchwork provides scripts for the latter and +they can be loaded as seen below: + +.. code-block:: shell + + $ sudo -u postgres createuser nobody + $ sudo -u postgre psql -f \ + /opt/patchwork/lib/sql/grant-all.postgres.sql patchwork + +.. note:: + + This assumes your Postfix process is running as the `nobody` user. If this + is not correct (use of `postfix` user is also common), you should change + both the username in the `createuser` command above and substitute the + username in the `grant-all-postgres.sql` script with the appropriate + alternative. + +__ http://www.postfix.org/ + +Use a Email-as-a-Service Provider +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Setting up an email server can be a difficult task and, in the case of +deployment on PaaS provider, may not even be an option. In this case, there +are a variety of web services available that offer "Email-as-as-Service". +These services typically convert received emails into HTTP POST requests to +your endpoint of choice, allowing you to sidestep configuration issues. We +don't cover this here, but a simple wrapper script coupled with one of these +services can be more than to get email into Patchwork. + +You can also create such as service yourself using a PaaS provider that +supports incoming mail and writing a little web app. + +(Optional) Configure your VCS to Automatically Update Patches +------------------------------------------------------------- + +The `tools` directory of the Patchwork distribution contains a file named +`post-receive.hook` which is a sample git hook that can be used to +automatically update patches to the `Accepted` state when corresponding +commits are pushed via git. + +To install this hook, simply copy it to the `.git/hooks` directory on your +server, name it `post-receive`, and make it executable. + +This sample hook has support to update patches to different states depending +on which branch is being pushed to. See the `STATE_MAP` setting in that file. + +If you are using a system other than git, you can likely write a similar hook +using `pwclient` to update patch state. If you do write one, please contribute +it. diff --git a/docs/deployment/upgrading.rst b/docs/deployment/upgrading.rst new file mode 100644 index 0000000..298db11 --- /dev/null +++ b/docs/deployment/upgrading.rst @@ -0,0 +1,152 @@ +Upgrading +========= + +This document provides some general tips and tricks that one can use when +upgrading an existing, production installation of Patchwork. If you are +interested in the specific changes between each release, refer to the +`UPGRADING` document instead. If this is your first time installing Patchwork, +refer to the :doc:`installation` instead. + +Before You Start +---------------- + +Before doing anything, always **backup your data**. This generally means +backing up your database, but it might also be a good idea to backup your +environment in case you encounter issues during the upgrade process. + +While Patchwork won't explicitly prevent it, it's generally wise to avoid +upgrades spanning multiple releases in one go. An iterative upgrade approach +will provide an easier, if slower, upgrade process. + +Identify Changed Scripts, Requirements, etc. +-------------------------------------------- + +The `CHANGELOG` document provides a comprehensive listing of all +backwards-incompatible changes that occur between releases of Patchwork. +Examples of such changes include: + +* Moved/removed scripts and files + +* Changes to the requirements, e.g. supported Django versions + +* Changes to API that may affect, for example, third-party tools + +It is important that you understand these changes and ensure any scripts you +may have, such as systemd scripts, are modified accordingly. + +Understand What Requirements Have Changed +----------------------------------------- + +New versions of Patchwork can often require additional or updated version of +dependencies, e.g. newer versions of Django. It is important that you +understand these requirements and can fulfil them. This is particularly true +for users relying on distro-provided packages, who may have to deal with older +versions of a package or may be missing a package altogether (though we try to +avoid this). Such changes are usually listed in the `UPGRADING` document, but +you can also diff the `requirements.txt` files in each release for comparison. + +Collect Static Files +-------------------- + +New versions of Patchwork generally contain changes to the additional files +like images, CSS and JavaScript. To do this, run the `collectstatic` +management commands: + +.. code-block:: shell + + $ ./manage.py collectstatic + +Upgrade Your Database +--------------------- + +Migrations of the database can be tricky. Prior to `v1.0.0`__, database +migrations were provided by way of manual, SQL migration scripts. After this +release, Patchwork moved to support `Django migrations`__. If you are +upgrading from `v1.0.0` or later, it is likely that you can rely entirely on +the later to bring your database up-to-date. This can be done like so: + +.. code-block:: shell + + $ ./manage.py migrate + +However, there are a number of scenarios in which you may need to fall back to +the provided SQL migrations or provide your own: + +* You are using Django < 1.6 + + Patchwork supports Django 1.6. However, Django Migrations was added in 1.7 + and is `not available for previous versions`__. As such, you must continue to + use manual migrations or upgrade your version of Django. For many of the + migrations, this can be done automatically: + + .. code-block:: shell + + $ ./manage.py sqlmigrate patchwork 0003_add_check_model + + However, this only works for schema migrations. For data migrations, + however, this will fail. In this cases, these migrations will need to be + handwritten. + +* You are using Django > 1.6, but upgrading from Patchwork < 1.0.0 + + Patchwork only started providing migrations in `v1.0.0`. SQL migrations are + provided for versions prior to this and must be applied to get the database + to the "initial" state that Django migrations expects. + +* You have diverged from upstream Patchwork + + If you have applied custom patches that change the database models, the + database in an "inconsistent state" and the provided migrations will likely + fail to apply. + +Steps to handle the latter two of these are described below. + +__ https://github.com/getpatchwork/patchwork/releases/tag/v1.0.0 +__ https://docs.djangoproject.com/en/1.8/topics/migrations/ +__ http://blog.allenap.me/2015/05/south-south-2-and-django-migrations.html + +Upgrading a pre-v1.0.0 Patchwork instance +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +The process for this type of upgrade is quite simple: upgrade using manual SQL +upgrades until better options become available. As such, you should apply all +unapplied SQL migrations that are not duplicated by Django migrations. Once +such duplication occurs, rely on the Django migrations only and continue to do +so going forward. + +Upgrading a "diverged" Patchwork instance +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +This type of upgrade is a little trickier. There are two options you can take: + +1. Bring your Patchwork instance back in sync with upstream + +2. Provide your own migrations + +The former option is particularly suitable if you decide to upstream your +change or decide it's not valuable enough to retain. This will require either +reworking any migrations that exist prior to your feature being upstreamed, or +deleting any added database fields and tables, respectively. In both cases, +manually, hand-written SQL migrations will be required to get the databse into +a consistent state (remember: **backup**!). Once this is done, you can resume +using the upstream-provided migrations, ensuring any Django migrations that you +may have skipped are not applied again: + +.. code-block:: shell + + $ ./manage.py migrate 000x-abc --fake # when 000x-abc is last "skippable" + +It's worth adding that with the databases now back in sync it should be +possible to return to using upstream code rather than maintaining a fork. + +The latter option is best chosen if you wish to retain the aforementioned fork. +How you do this depends on the extensiveness of your changes, but getting the +latest version of Patchwork, deleting the provided migrations, applying any +patches you may have and regenerating the migrations seems like the best +option. + +.. note:: + + To prevent the latter case above from occurring, we'd ask that you submit + any patches you may have to the upstream Patchwork so that the wider + community can benefit from this new functionality. diff --git a/docs/index.rst b/docs/index.rst index 7b4d7c6..7989b4e 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -5,6 +5,7 @@ Patchwork :maxdepth: 2 usage/index + deployment/index Patchwork is a patch tracking system for community-based projects. It is intended to make the patch management process easier for both the project's -- 2.9.3 From stephen at that.guru Tue Apr 11 09:27:50 2017 From: stephen at that.guru (Stephen Finucane) Date: Tue, 11 Apr 2017 00:27:50 +0100 Subject: [PATCH 8/9] docs: Use ReadTheDocs theme In-Reply-To: <20170410232751.8489-1-stephen@that.guru> References: <20170410232751.8489-1-stephen@that.guru> Message-ID: <20170410232751.8489-9-stephen@that.guru> Signed-off-by: Stephen Finucane --- docs/conf.py | 5 ++++- tox.ini | 1 + 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/conf.py b/docs/conf.py index 6b978d2..06c1b4d 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -19,6 +19,7 @@ # import sys # sys.path.insert(0, os.path.abspath('.')) +import sphinx_rtd_theme # -- General configuration ------------------------------------------------ @@ -83,7 +84,7 @@ todo_include_todos = False # The theme to use for HTML and HTML Help pages. See the documentation for # a list of builtin themes. # -html_theme = 'alabaster' +html_theme = 'sphinx_rtd_theme' # Theme options are theme-specific and customize the look and feel of a theme # further. For a list of options available for each theme, see the @@ -95,3 +96,5 @@ html_theme = 'alabaster' # relative to this directory. They are copied after the builtin static files, # so a file named "default.css" will overwrite the builtin "default.css". html_static_path = [] + +html_theme_path = [sphinx_rtd_theme.get_html_theme_path()] diff --git a/tox.ini b/tox.ini index 78c1dd1..0392641 100644 --- a/tox.ini +++ b/tox.ini @@ -47,6 +47,7 @@ exclude = ./patchwork/migrations deps = sphinx>=1.5 reno>=2.2 + sphinx_rtd_theme>=0.2.0,<0.3.0 commands = sphinx-build -W -b html docs docs/_build/html -- 2.9.3 From stephen at that.guru Tue Apr 11 09:27:49 2017 From: stephen at that.guru (Stephen Finucane) Date: Tue, 11 Apr 2017 00:27:49 +0100 Subject: [PATCH 7/9] Add reno for release notes management In-Reply-To: <20170410232751.8489-1-stephen@that.guru> References: <20170410232751.8489-1-stephen@that.guru> Message-ID: <20170410232751.8489-8-stephen@that.guru> An initial patch to add reno and create a base directory for release notes. Signed-off-by: Stephen Finucane --- docs/conf.py | 2 +- docs/index.rst | 1 + docs/releases/index.rst | 7 ++ docs/releases/unreleased.rst | 5 + releasenotes/config.yaml | 3 + .../notes/initial-reno-68c116ae9c5259a3.yaml | 107 +++++++++++++++++++++ tox.ini | 4 +- 7 files changed, 127 insertions(+), 2 deletions(-) create mode 100644 docs/releases/index.rst create mode 100644 docs/releases/unreleased.rst create mode 100644 releasenotes/config.yaml create mode 100644 releasenotes/notes/initial-reno-68c116ae9c5259a3.yaml diff --git a/docs/conf.py b/docs/conf.py index b23d033..6b978d2 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -29,7 +29,7 @@ needs_sphinx = '1.5' # Add any Sphinx extension module names here, as strings. They can be # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom # ones. -extensions = ['sphinx.ext.todo'] +extensions = ['sphinx.ext.todo', 'reno.sphinxext'] # Add any paths that contain templates here, relative to this directory. templates_path = [] diff --git a/docs/index.rst b/docs/index.rst index f2e7fcd..e1d60f3 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -7,6 +7,7 @@ Patchwork usage/index deployment/index development/index + releases/index Patchwork is a patch tracking system for community-based projects. It is intended to make the patch management process easier for both the project's diff --git a/docs/releases/index.rst b/docs/releases/index.rst new file mode 100644 index 0000000..abc878b --- /dev/null +++ b/docs/releases/index.rst @@ -0,0 +1,7 @@ +Release Notes +============= + + .. toctree:: + :maxdepth: 1 + + unreleased diff --git a/docs/releases/unreleased.rst b/docs/releases/unreleased.rst new file mode 100644 index 0000000..720c149 --- /dev/null +++ b/docs/releases/unreleased.rst @@ -0,0 +1,5 @@ +Unreleased +========== + +.. release-notes:: + diff --git a/releasenotes/config.yaml b/releasenotes/config.yaml new file mode 100644 index 0000000..4f10ec1 --- /dev/null +++ b/releasenotes/config.yaml @@ -0,0 +1,3 @@ +--- +earliest_version: v1.1.0 +release_tag_re: 'v\d\.\d\.\d(rc\d+)?' diff --git a/releasenotes/notes/initial-reno-68c116ae9c5259a3.yaml b/releasenotes/notes/initial-reno-68c116ae9c5259a3.yaml new file mode 100644 index 0000000..5e1e94d --- /dev/null +++ b/releasenotes/notes/initial-reno-68c116ae9c5259a3.yaml @@ -0,0 +1,107 @@ +--- +features: + - | + REST API. + + Previous versions of Patchwork provided an XML-RPC API. This was functional + but there were a couple of issues around usability and general design. This + API also provided basic versioning information but the existing clients, + mostly `pwclient` variants, did not validate this version. Together, this + left us with an API that needed work but no way to fix it without breaking + every client out there. + + Rather than breaking all those users, make a clean break and provide + another API method. REST APIs are the API method de jour providing a number + of advantages over XML-RPC APIs, thus, a REST API is chosen. The following + resources are exposed over this new API: + + - Bundles + - Checks + - Projects + - People + - Users + - Patches + - Series + - Cover letters + + For information on the usage of the API, refer to the documentation. + + Note that this feature depends on a combination of Django REST framework + and Django Filter, which in turn require Django 1.8+. As a result, Django + 1.8 or greater must be used when the REST API is enabled. + - | + Cover letters are now supported. + + Cover letters are often sent in addition to a series of patches. They do + not contain a diff and can generally be identified as number 0 of a series. + For example:: + + [PATCH 0/3] A cover letter + + Cover letters contain useful information that should not be discarded. + Both cover letters and replies to these mails are now stored for use with + series. + - | + Series are now supported. + + Series are groups of patches sent as one bundle. For example:: + + [PATCH 0/3] A cover letter + [PATCH 1/3] The first patch + [PATCH 2/3] The second patch + [PATCH 3/3] The third patch + + While Patchwork already supports bundles, these must be created manually, + defeating the purpose of using series in the first place. Series make use + of the information provided in the emails themselves, avoiding this manual + step. The series support implemented is basic and does not support + versioning. This will be added in a future release. + - | + All comments now have a permalink which can be used to reference individual + replies to patches and cover letters. + - | + `Django Debug Toolbar `_ + is now enabled by defaut when using development settings. + - | + `Django 1.9 `_ and + `1.10 `_ are now + supported. + - | + `Python 3.5 `_ is now + supported. + - | + `Docker `_ support is now + integrated for development usage. To use this, refer to the documentation. +upgrade: + - | + The REST API is enabled by default. + + The REST API is enabled by default when using Django 1.8. It is possible to + disable this API, though this functionality may be removed in a future + release. Should you wish to disable this feature, configure the + `ENABLE_REST_API` setting to `False`. + - | + The `parsemail.py` and `parsearchive.py` scripts have been replaced by the + `parsemail` and `parsearchive` management commands. These can be called + like any other management commands. For example:: + + $ ./manage.py parsemail [args...] +fixes: + - | + When downloading an mbox, a user's name will now be set to the name used in + the last email recieved from them. Previously, the name used in the first + email received from a user was used. + - | + `user at domain`-style email addresses, commonly found in Mailman archives, + are now handled correctly. + - | + Unicode characters transmitted over the XML-RPC API are now handled + correctly under Python 3 +deprecated: + - | + Django 1.6 and Django 1.7 are no longer supported upstream and will not be + supported in a future release. +other: + - | + `reno `_ is now used for release note + management. diff --git a/tox.ini b/tox.ini index 2d195cd..78c1dd1 100644 --- a/tox.ini +++ b/tox.ini @@ -44,7 +44,9 @@ ignore = E129, F405 exclude = ./patchwork/migrations [testenv:docs] -deps = sphinx>=1.5 +deps = + sphinx>=1.5 + reno>=2.2 commands = sphinx-build -W -b html docs docs/_build/html -- 2.9.3 From stephen at that.guru Tue Apr 11 09:27:51 2017 From: stephen at that.guru (Stephen Finucane) Date: Tue, 11 Apr 2017 00:27:51 +0100 Subject: [PATCH 9/9] docs: Migrate release notes to rST In-Reply-To: <20170410232751.8489-1-stephen@that.guru> References: <20170410232751.8489-1-stephen@that.guru> Message-ID: <20170410232751.8489-10-stephen@that.guru> This allows us to remove both UPGRADING and CHANGELOG. Signed-off-by: Stephen Finucane --- CHANGELOG.md | 189 ------------------------------------------- UPGRADING.md | 4 - docs/releases/alpaca.rst | 10 +++ docs/releases/burlap.rst | 99 +++++++++++++++++++++++ docs/releases/cashmere.rst | 82 +++++++++++++++++++ docs/releases/index.rst | 3 + docs/releases/unreleased.rst | 1 - 7 files changed, 194 insertions(+), 194 deletions(-) delete mode 100644 CHANGELOG.md delete mode 100644 UPGRADING.md create mode 100644 docs/releases/alpaca.rst create mode 100644 docs/releases/burlap.rst create mode 100644 docs/releases/cashmere.rst diff --git a/CHANGELOG.md b/CHANGELOG.md deleted file mode 100644 index 9dac8a4..0000000 --- a/CHANGELOG.md +++ /dev/null @@ -1,189 +0,0 @@ -# Change Log - -All notable changes to this project will be documented in this file. Please -refer to the release notes for more detailed information, e.g. how to upgrade. - -This project adheres to [Semantic Versioning](http://semver.org/). - -## [Unreleased] - -This release added support for a REST API, cover letter parsing, and the latest -versions of both Django and Python. - -### Added - -- REST API support (Django >= 1.8 only) -- Cover letter support -- Series support -- Comment permalinks -- Django debug toolbar support for developers -- Django 1.9 and 1.10 support -- Python 3.5 support -- Docker support for developers -- Sample deployment documentation -- User documentation - -### Changed - -- The `parsemail.py` and `parsearchive.py` scripts have been replaced by the - `parsemail` and `parsearchive` management commands -- Significant rework of tests -- A user's name will now be updated on each email received from them - -### Fixed - -- Numerous typos and other issues in both documentation and code -- Various UI and performance improvements -- Checks are now displayed with their associated owner, thus preventing - spoofing -- `user at domain`-style email address, commonly found in Mailman archives, are - now handled correctly -- Unicode characters transmitted over the XML-RPC API are now handled correctly - under Python 3 - -### Removed - -- `parser.py` can no longer be exected as a script - -### Deprecated - -- Django 1.7 support will be removed in a future release - -## [1.1.1] - 2016-03-29 - -This release fixed a number of issues with the [1.1.0] release. - -## Fixed - -- Numerous issues in the `parsemail.py`, `parsearchive.py` and `parsemail.sh` - scripts -- Permissions of database tables as set by `grant-all` SQL scripts -- Misc. UI and performance improvements - -## [1.1.0] - 2016-03-03 - -This release overhauled the web UI and added support for automatic delegation -of patches, test result capture, and Python 3.4. - -### Added - -- Python 3.4 support -- Check feature, which can be used to report the status of tests -- Automatic delegation of patches based on file path -- Automated documentation for the XML-RPC API. This can be found at the - '/xmlrpc' in most Patchwork deployments -- Vagrant support for developers -- Selenium-based UI tests for developers - -### Changed - -- Overhauled the web UI to reflect modern web standards -- Patches can now be delegated to any Patchwork user -- Significant updates to the documentation -- Assorted cleanup tasks and bug fixes - -## [1.0.0] - 2015-10-26 - -### Added - -- Patch tag infrastructure feature, which provides a quick summary of patch - "tags" (e.g. `Acked-by`, `Reviewed-by`, ...) found in a patch and its replies -- Django 1.7 and Django 1.8 support -- Django `staticfiles` support, which should be used to gather static files for - for serving via a web server -- tox support for developers - -### Changed - -- Migrations are now executed using the Django migrations framework -- Moved Patchwork source from the `apps` directory to the top level directory -- The `cron` Django management command has replaced the `bin/patchwork-cron` - script -- Rewrote documentation to reflect changes in development and deployment best - practices over the past few years -- Reworked `requirements.txt` and `settings.py` files - -### Removed - -- Django 1.5 support -- Defunct Python 2.5 code -- Numerous dead files/code - -### Deprecated - -- Django 1.6 support will be removed in a future release -- Raw SQL migration scripts, previously found at `lib/sql/migration`, will no - longer be provided. Use the Django Migration framework found in Django 1.7 - and above - -### Additional notes - -This version changes a few admin-visible components of Patchwork, so upgrading -involves a few steps. - -#### Update settings - -By default, settings are read from `patchwork/settings/production.py`. To -migrate, use the provided template: - - $ cp patchwork/settings/production{.example,}.py - -Merge your previous settings, usually located in `apps/local_settings.py`, to -this file. - -#### Fix external references - -Any scripts or tools that call Patchwork applications, such as `parsemail.sh`, -must be updated to reference the new location of these scripts. To do this, -simply remove `apps/` from the path, i.e. `apps/patchwork/` becomes -`patchwork`. - -In addition, any scripts that set the `DJANGO_SETTINGS_MODULE` environment -variable will need to be updated to reflect the new location, typically: - - DJANGO_SETTINGS_MODULE=patchwork.settings.production - -Finally, as the `patchwork-cron.py` script has been replaced by the `cron` -management command, any references to the former should be updated to the -latter. The `cron` management command can be called like so: - - $ ./manage.py cron - -#### Migrate to Django Migrations - -Future database migrations will be implemented using Django Migrations, rather -than raw SQL scripts. Before switching to Django migrations, first apply any -unapplied migrations in the `lib/sql/migration` folder. For example, on -postgres: - - $ psql -f lib/sql/migration/015-add-patch-tags.sql patchwork - $ psql -f lib/sql/grant-all.postgres.sql patchwork - -Once applied, configure the required Django Migration tables using the -`migrate` managment command: - - $ ./manage.py migrate --fake-initial - -#### Migrate to Django Staticfiles - -Static content should now be located in the folder indicated by `STATIC_ROOT`. -This should point to somewhere sensible, such as the absolute path of -`htdocs/static` in the Patchwork tree. Configure the `STATIC_ROOT` setting in -your settings file, then run the `collectstatic` management command: - - $ ./manage.py collectstatic - -Finally, update your webserver's configuration to serve the static content from -this new location. Refer to the sample web configuration files provided in -`lib` for more information. - -## [0.9.0] - 2015-03-22 - -**NOTE:** 1.0.0 was the first release of Patchwork adopting semantic versioning. -For information on *"0.9.0"* and before, please refer to Git logs. - -[Unreleased]: https://github.com/getpatchwork/patchwork/compare/v1.1.0...HEAD -[1.1.0]: https://github.com/getpatchwork/patchwork/compare/v1.0.0...v1.1.0 -[1.0.0]: https://github.com/getpatchwork/patchwork/compare/v0.9.0...v1.0.0 -[0.9.0]: https://github.com/getpatchwork/patchwork/compare/c561ebe...v0.9.0 - diff --git a/UPGRADING.md b/UPGRADING.md deleted file mode 100644 index 1089194..0000000 --- a/UPGRADING.md +++ /dev/null @@ -1,4 +0,0 @@ -# Patchwork Upgrade Guide - -For information on upgrading Patchwork instances, refer to the -[changelog](./CHANGELOG.md). diff --git a/docs/releases/alpaca.rst b/docs/releases/alpaca.rst new file mode 100644 index 0000000..e4b310d --- /dev/null +++ b/docs/releases/alpaca.rst @@ -0,0 +1,10 @@ +v0.9 Series ("Alpaca") +====================== + +This represents the state of the Patchwork code before adopting `semantic +versioning`__, along with `fabric-inspired release names`__. For information on +the features available in this release, refer to the `git logs`__. + +__ http://semver.org/ +__ https://en.wikipedia.org/wiki/List_of_fabrics +__ https://github.com/getpatchwork/patchwork/commits/v0.9.0 diff --git a/docs/releases/burlap.rst b/docs/releases/burlap.rst new file mode 100644 index 0000000..8477e88 --- /dev/null +++ b/docs/releases/burlap.rst @@ -0,0 +1,99 @@ +v1.0 Series ("Burlap") +====================== + +1.0.0 +----- + +This release changes a few admin-visible components of Patchwork, so upgrading +involves a few steps. + +New Features +~~~~~~~~~~~~ + +- Patch tags are now supported + + Patch "tags", such as `Acked-by`, `Reviewed-by`, are typically included in + patches and replies. They provide important information as to the activity + and "mergability" of a patch. These tags are now extracted from patches and + included in the patch list. + +- Django 1.7 and Django 1.8 are now supported + +- tox support is integrated for use by developers + +Upgrade Notes +~~~~~~~~~~~~~ + +- Migrations are now executed using the Django migrations framework. + + Future database migrations will be implemented using Django Migrations, + rather than raw SQL scripts. Before switching to Django migrations, first + apply any unapplied migrations in the `lib/sql/migration` folder. For + example, on postgres:: + + $ psql -f lib/sql/migration/015-add-patch-tags.sql patchwork + $ psql -f lib/sql/grant-all.postgres.sql patchwork + + Once applied, configure the required Django Migration tables using the + `migrate` managment command:: + + $ ./manage.py migrate --fake-initial + +- Moved Patchwork source from the `apps` directory to the top level directory. + + Any scripts or tools that call Patchwork applications, such as + `parsemail.sh`, must be updated to reference the new location of these + scripts. To do this, simply remove `apps/` from the path, i.e. + `apps/patchwork/` becomes `patchwork`. + +- The `patchwork-cron.py` script has been replaced by the `cron` management + command. + + Any references to the former should be updated to the latter. The `cron` + management command can be called like so:: + + $ ./manage.py cron + +- The `settings.py` file has been updated to reflect modern Django practices. + + You may need to manually migrate your existing configuration to the new + settings file(s). By default, settings are read from + `patchwork/settings/production.py`. To migrate, use the provided template:: + + $ cp patchwork/settings/production{.example,}.py + + Merge your previous settings, usually located in `apps/local_settings.py`, to + this file. + + In addition, any scripts that set the `DJANGO_SETTINGS_MODULE` environment + variable will need to be updated to reflect the new location, typically:: + + DJANGO_SETTINGS_MODULE=patchwork.settings.production + +- Django `staticfiles` is now used to to gather static files for for serving + via a web server + + Static content should now be located in the folder indicated by + `STATIC_ROOT`. This should point to somewhere sensible, such as the absolute + path of `htdocs/static` in the Patchwork tree. Configure the `STATIC_ROOT` + setting in your settings file, then run the `collectstatic` management + command:: + + $ ./manage.py collectstatic + + Finally, update your webserver configuration to serve the static content from + this new location. Refer to the sample web configuration files provided in + `lib` for more information. + +- Django 1.5 is no longer supported + +- Python 2.5 support was broken and is officially no longer supported + +Deprecation Notes +~~~~~~~~~~~~~~~~~ + +- Django 1.6 support will be removed in a future release + +- Raw SQL migration scripts, currently found at `lib/sql/migration`, will no + longer be updated and will be removed in a future release. The Django + Migration framework, found in Django 1.7 and above, should be used instead. diff --git a/docs/releases/cashmere.rst b/docs/releases/cashmere.rst new file mode 100644 index 0000000..9db9f8f --- /dev/null +++ b/docs/releases/cashmere.rst @@ -0,0 +1,82 @@ +v1.1 Series ("Cashmere") +======================== + +1.1.3 +----- + +This release fixes a number of issues with the 1.1.2 release. + +Bug Fixes +~~~~~~~~~ + +- Some Python 3 issues are resolved in `pwclient` + +- `pwclient` now functions as expected behind a proxy + +1.1.2 +----- + +This release fixed a number of issues with the 1.1.1 release. + +Bug Fixes +~~~~~~~~~ + +- Headers containing invalid characters or codings are now parsed correctly + +- Patches can no longer be delegated to any user + + This had significant performance impacts and has been reverted. + +1.1.1 +----- + +This release fixed a number of issues with the 1.1.0 release. + +Bug Fixes +~~~~~~~~~ + +- Numerous issues in the `parsemail.py`, `parsearchive.py` and `parsemail.sh` + scripts are resolved + +- Permissions of database tables, as set by `grant-all` SQL scripts, are now + set for tables added in Patchwork 1.1.0 + +- Some performance and usability regressions in the UI are resolved + +1.1.0 +----- + +This release focuses on usability and maintainability, and sets us up nicely +for a v2.0.0 release in the near future. Feature highlights of v1.1.0 include: + +- Automated delegation of patches, based on the files modified in said patches. + +- Storing of test results, a.k.a. "checks", on a patch-by-patch basis. + +- Delegation of patches to any registered Patchwork user (previously one had to + be a registered maintainer). + +- Overhaul of the web UI, which is now based on Bootstrap. + +- Python 3 support. + +New Features +~~~~~~~~~~~~ + +- The web UI is updated to reflect modern web standards. Bootstrap 3.x is used. + +- Python 3.4 is now supported + +- Checks, which can be used to report the status of tests, have been added + +- Automatic delegation of patches based on file path + +- Automated documentation for the XML-RPC API. This can be found at the + '/xmlrpc' in most Patchwork deployments + +- Vagrant is now integrated for use during development + +Upgrade Notes +~~~~~~~~~~~~~ + +- Patches can now be delegated to any Patchwork user. diff --git a/docs/releases/index.rst b/docs/releases/index.rst index abc878b..8d76f68 100644 --- a/docs/releases/index.rst +++ b/docs/releases/index.rst @@ -5,3 +5,6 @@ Release Notes :maxdepth: 1 unreleased + cashmere + burlap + alpaca diff --git a/docs/releases/unreleased.rst b/docs/releases/unreleased.rst index 720c149..23caa95 100644 --- a/docs/releases/unreleased.rst +++ b/docs/releases/unreleased.rst @@ -2,4 +2,3 @@ Unreleased ========== .. release-notes:: - -- 2.9.3 From stephen at that.guru Fri Apr 14 06:48:21 2017 From: stephen at that.guru (Stephen Finucane) Date: Thu, 13 Apr 2017 21:48:21 +0100 Subject: [PATCH] Capture stderr for 'git describe' Message-ID: <20170413204821.17445-1-stephen@that.guru> Capture the error messages thrown if git is not installed. Signed-off-by: Stephen Finucane Fixes: 7b07b71e ("templates: Add Patchwork version") --- patchwork/version.py | 1 + 1 file changed, 1 insertion(+) diff --git a/patchwork/version.py b/patchwork/version.py index e38f50c..16b7f61 100644 --- a/patchwork/version.py +++ b/patchwork/version.py @@ -56,6 +56,7 @@ def get_raw_git_version(): """Returns the raw git version via 'git-describe'.""" try: git_version = subprocess.check_output(['git', 'describe'], + stderr=subprocess.STDOUT, cwd=ROOT_DIR) except (OSError, subprocess.CalledProcessError): return '' -- 2.9.3 From stephen at that.guru Sat Apr 15 04:58:21 2017 From: stephen at that.guru (Stephen Finucane) Date: Fri, 14 Apr 2017 19:58:21 +0100 Subject: [PATCH 2/3] 'mpe mode': click to copy patch IDs In-Reply-To: <8760idz0ak.fsf@possimpible.ozlabs.ibm.com> References: <20170327054252.24337-1-dja@axtens.net> <20170327054252.24337-2-dja@axtens.net> <1491754578.5029.7.camel@that.guru> <8760idz0ak.fsf@possimpible.ozlabs.ibm.com> Message-ID: <1492196301.10348.13.camel@that.guru> On Mon, 2017-04-10 at 08:25 +1000, Daniel Axtens wrote: > Stephen Finucane writes: > > > On Mon, 2017-03-27 at 16:42 +1100, Daniel Axtens wrote: > > > If 'Show Patch IDs' is turned on in settings, add an extra column > > > to the patch list, with buttons showing the patch IDs. The > > > buttons > > > copy the patch IDs to the clipboard. > > > > > > JavaScript inspired by https://github.com/Triforcey/clip-j and > > > many > > > many StackOverflow answers. > > > > > > Suggested-by: Michael Ellerman > > > Signed-off-by: Daniel Axtens > > > > This looks good to me. I do wonder if there's any reason not to > > turn > > this on by default though? > > My thought was just to avoid messing around too much with the default > UI > that people are used to, especially since it's not entirely obvious > what > the buttons do just by looking at them. Sounds fair. We can make this the default in the future if required. Reviewed-by: Stephen Finucane Stephen From stephen at that.guru Sat Apr 15 10:34:59 2017 From: stephen at that.guru (Stephen Finucane) Date: Sat, 15 Apr 2017 01:34:59 +0100 Subject: [PATCH] Capture stderr for 'git describe' In-Reply-To: <20170413204821.17445-1-stephen@that.guru> References: <20170413204821.17445-1-stephen@that.guru> Message-ID: <1492216499.4647.0.camel@that.guru> On Thu, 2017-04-13 at 21:48 +0100, Stephen Finucane wrote: > Capture the error messages thrown if git is not installed. > > Signed-off-by: Stephen Finucane > Fixes: 7b07b71e ("templates: Add Patchwork version") Trivial fix. Applied. From stephen at that.guru Sat Apr 15 10:35:41 2017 From: stephen at that.guru (Stephen Finucane) Date: Sat, 15 Apr 2017 01:35:41 +0100 Subject: [PATCH] parser: Don't extract diffs from replies In-Reply-To: <20170409152355.17160-1-stephen@that.guru> References: <20170409152355.17160-1-stephen@that.guru> Message-ID: <1492216541.4647.1.camel@that.guru> On Sun, 2017-04-09 at 16:23 +0100, Stephen Finucane wrote: > In '2a915efd', a check was added to ensure mails prefixed with 'RE:' > or > similar would not be parsed as patches. By time this check actually > happens, any patches had already been extracted from the mail thus > these > patches were re-added to the mail content before saving the comment. > Unfortunately, this didn't take into account cases where a patch or > diff > was not the last part of a mail but rather located somewhere in the > middle of the content: > > Introduction content > Diff or patch content *** > Additional content > > This would result in mangling of the mail as the patch would _always_ > be > appended to the end: > > Introduction content > Additional content > Diff or patch content *** > > Handle this by only breaking a mail into a comment and a diff if > there > is any possibility that we might want to use that diff. > > Signed-off-by: Stephen Finucane > Fixes: 2a915efd ("parser: fix wrong parsing of diff comments") > Closes-bug: #95 > Cc: Ralf Baechle I'm pretty confident with this fix. Applied so I can work on finishing touches for 2.0. Stephen From stephen at that.guru Sat Apr 15 10:35:52 2017 From: stephen at that.guru (Stephen Finucane) Date: Sat, 15 Apr 2017 01:35:52 +0100 Subject: [PATCH] sql: Update permissions on grant-all scripts In-Reply-To: <20170409180849.26170-1-stephen@that.guru> References: <20170409180849.26170-1-stephen@that.guru> Message-ID: <1492216552.4647.2.camel@that.guru> On Sun, 2017-04-09 at 19:08 +0100, Stephen Finucane wrote: > Allow access to Submission, CoverLetter, Series, SeriesReference and > SeriesPatch. > > Signed-off-by: Stephen Finucane > Cc: Jeremy Kerr Applied. From stephen at that.guru Sat Apr 15 10:36:04 2017 From: stephen at that.guru (Stephen Finucane) Date: Sat, 15 Apr 2017 01:36:04 +0100 Subject: [PATCH] parsearchive: Support maildirs In-Reply-To: <20170409171403.16836-1-stephen@that.guru> References: <20170409171403.16836-1-stephen@that.guru> Message-ID: <1492216564.4647.3.camel@that.guru> On Sun, 2017-04-09 at 18:14 +0100, Stephen Finucane wrote: > At present, the 'parsearchive' command only supports parsing of > mboxes. > Expand this to support maildirs. This allows us to rewrite the > 'parsemail-bulk' script to deliver much improved performance. > > Signed-off-by: Stephen Finucane > Suggested-by: Daniel Axtens Applied. From stephen at that.guru Sat Apr 15 22:06:23 2017 From: stephen at that.guru (Stephen Finucane) Date: Sat, 15 Apr 2017 13:06:23 +0100 Subject: [PATCH 1/3] Put download buttons at the top as well In-Reply-To: <1491754435.5029.3.camel@that.guru> References: <20170327054252.24337-1-dja@axtens.net> <1491754435.5029.3.camel@that.guru> Message-ID: <1492257983.2544.0.camel@that.guru> On Sun, 2017-04-09 at 17:13 +0100, Stephen Finucane wrote: > On Mon, 2017-03-27 at 16:42 +1100, Daniel Axtens wrote: > > Suggested by a few people. > > > > Signed-off-by: Daniel Axtens > > Reviewed-by: Stephen Finucane Applied all of these. From robin.jarry at 6wind.com Sat Apr 22 21:32:23 2017 From: robin.jarry at 6wind.com (Robin Jarry) Date: Sat, 22 Apr 2017 13:32:23 +0200 Subject: [PATCH v2] pwclient: Fix silent crash on python 2 In-Reply-To: <20170405124604.28022-1-robin.jarry@6wind.com> References: <20170405113552.7vbiyjgnfykoojmn@6wind.com> <20170405124604.28022-1-robin.jarry@6wind.com> Message-ID: Gentle bump :-) Robin -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephen at that.guru Thu Apr 27 05:49:47 2017 From: stephen at that.guru (Stephen Finucane) Date: Wed, 26 Apr 2017 20:49:47 +0100 Subject: [PATCH] requirements: Remove docs requirements Message-ID: <20170426194947.6862-1-stephen@that.guru> This was causing issues in the Docker build and is unnecessary with tox around. Simply drop this. Signed-off-by: Stephen Finucane --- requirements-dev.txt | 1 - 1 file changed, 1 deletion(-) diff --git a/requirements-dev.txt b/requirements-dev.txt index ea548c0..8f88605 100644 --- a/requirements-dev.txt +++ b/requirements-dev.txt @@ -2,4 +2,3 @@ Django>=1.8,<1.11 djangorestframework>=3.4,<3.7 django-filter>=1.0,<1.1 -r requirements-test.txt --r docs/requirements.txt -- 2.9.3 From stephen at that.guru Thu Apr 27 09:13:42 2017 From: stephen at that.guru (Stephen Finucane) Date: Thu, 27 Apr 2017 00:13:42 +0100 Subject: [PATCH] requirements: Remove docs requirements In-Reply-To: <20170426194947.6862-1-stephen@that.guru> References: <20170426194947.6862-1-stephen@that.guru> Message-ID: <1493248422.2661.0.camel@that.guru> On Wed, 2017-04-26 at 20:49 +0100, Stephen Finucane wrote: > This was causing issues in the Docker build and is unnecessary with > tox around. Simply drop this. > > Signed-off-by: Stephen Finucane Trivial fix. Merged. From stephen at that.guru Thu Apr 27 09:58:03 2017 From: stephen at that.guru (Stephen Finucane) Date: Thu, 27 Apr 2017 00:58:03 +0100 Subject: [PATCH 1/2] REST: Fix versioning Message-ID: <20170426235804.31181-1-stephen@that.guru> One of the few remaining warts in the API is versioning. There is some basic versioning there, but it doesn't work properly as-is. Fix this by correcting the index endpoint ('/') to use Django REST Framework's 'reverse' function [1], which handles versioning for us [2] and switching from 'NamespaceVersioning' to 'URLPathVersioning', the latter of which does the same thing but in a different way. [1] http://www.django-rest-framework.org/api-guide/reverse/#reverse [2] http://www.django-rest-framework.org/api-guide/versioning/#reversing-urls-for-versioned-apis [3] http://www.django-rest-framework.org/api-guide/versioning/#urlpathversioning Signed-off-by: Stephen Finucane Cc: Andy Doan --- patchwork/api/event.py | 2 +- patchwork/api/index.py | 21 ++++++++++----------- patchwork/api/patch.py | 2 +- patchwork/settings/base.py | 2 +- patchwork/urls.py | 2 +- 5 files changed, 14 insertions(+), 15 deletions(-) diff --git a/patchwork/api/event.py b/patchwork/api/event.py index 32b3bde..a060022 100644 --- a/patchwork/api/event.py +++ b/patchwork/api/event.py @@ -19,8 +19,8 @@ from collections import OrderedDict -from django.core.urlresolvers import reverse from rest_framework.generics import ListAPIView +from rest_framework.reverse import reverse from rest_framework.serializers import HyperlinkedModelSerializer from rest_framework.serializers import SerializerMethodField diff --git a/patchwork/api/index.py b/patchwork/api/index.py index 513e8b6..1cdd379 100644 --- a/patchwork/api/index.py +++ b/patchwork/api/index.py @@ -17,22 +17,21 @@ # along with Patchwork; if not, write to the Free Software # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA -from django.core.urlresolvers import reverse from rest_framework.response import Response +from rest_framework.reverse import reverse from rest_framework.views import APIView class IndexView(APIView): - def get(self, request, format=None): + def get(self, request, version=None): return Response({ - 'projects': request.build_absolute_uri( - reverse('api-project-list')), - 'users': request.build_absolute_uri(reverse('api-user-list')), - 'people': request.build_absolute_uri(reverse('api-person-list')), - 'patches': request.build_absolute_uri(reverse('api-patch-list')), - 'covers': request.build_absolute_uri(reverse('api-cover-list')), - 'series': request.build_absolute_uri(reverse('api-series-list')), - 'events': request.build_absolute_uri(reverse('api-event-list')), - 'bundles': request.build_absolute_uri(reverse('api-bundle-list')), + 'projects': reverse('api-project-list', request=request), + 'users': reverse('api-user-list', request=request), + 'people': reverse('api-person-list', request=request), + 'patches': reverse('api-patch-list', request=request), + 'covers': reverse('api-cover-list', request=request), + 'series': reverse('api-series-list', request=request), + 'events': reverse('api-event-list', request=request), + 'bundles': reverse('api-bundle-list', request=request), }) diff --git a/patchwork/api/patch.py b/patchwork/api/patch.py index ded4910..a231524 100644 --- a/patchwork/api/patch.py +++ b/patchwork/api/patch.py @@ -19,11 +19,11 @@ import email.parser -from django.core.urlresolvers import reverse from django.utils.translation import ugettext_lazy as _ from rest_framework.generics import ListAPIView from rest_framework.generics import RetrieveUpdateAPIView from rest_framework.relations import RelatedField +from rest_framework.reverse import reverse from rest_framework.serializers import HyperlinkedModelSerializer from rest_framework.serializers import SerializerMethodField diff --git a/patchwork/settings/base.py b/patchwork/settings/base.py index 180a469..cafb4a6 100644 --- a/patchwork/settings/base.py +++ b/patchwork/settings/base.py @@ -144,7 +144,7 @@ except ImportError: REST_FRAMEWORK = { 'DEFAULT_VERSIONING_CLASS': - 'rest_framework.versioning.NamespaceVersioning', + 'rest_framework.versioning.URLPathVersioning', 'DEFAULT_PAGINATION_CLASS': 'patchwork.api.base.LinkHeaderPagination', 'DEFAULT_FILTER_BACKENDS': ( 'django_filters.rest_framework.DjangoFilterBackend', diff --git a/patchwork/urls.py b/patchwork/urls.py index e888386..8c8c172 100644 --- a/patchwork/urls.py +++ b/patchwork/urls.py @@ -232,7 +232,7 @@ if settings.ENABLE_REST_API: ] urlpatterns += [ - url(r'^api/1.0/', include(api_patterns)), + url(r'^api/(?P(1.0))/', include(api_patterns)), ] -- 2.9.3 From stephen at that.guru Thu Apr 27 09:58:04 2017 From: stephen at that.guru (Stephen Finucane) Date: Thu, 27 Apr 2017 00:58:04 +0100 Subject: [PATCH 2/2] REST: Allow users to omit version entirely In-Reply-To: <20170426235804.31181-1-stephen@that.guru> References: <20170426235804.31181-1-stephen@that.guru> Message-ID: <20170426235804.31181-2-stephen@that.guru> This is super handy for debugging/development of Patchwork and Patchwork clients alike. Signed-off-by: Stephen Finucane --- docs/api/rest.rst | 17 +++++++++++++++++ patchwork/urls.py | 2 +- 2 files changed, 18 insertions(+), 1 deletion(-) diff --git a/docs/api/rest.rst b/docs/api/rest.rst index 75e31f6..1750913 100644 --- a/docs/api/rest.rst +++ b/docs/api/rest.rst @@ -86,6 +86,23 @@ overview of existing API clients, refer to :doc:`../usage/clients`. own instance of Patchwork locally <../development/installation>` and experiment with that instead. +Versioning +---------- + +By default, all requests will receive the latest version of the API: currently +``1.0``: + +.. code-block:: http + + GET /api HTTP/1.1 + +You should explicitly request this version through the URL to prevent API +changes breaking your application: + +.. code-block:: http + + GET /api/1.0 HTTP/1.1 + Schema ------ diff --git a/patchwork/urls.py b/patchwork/urls.py index 8c8c172..f700f38 100644 --- a/patchwork/urls.py +++ b/patchwork/urls.py @@ -232,7 +232,7 @@ if settings.ENABLE_REST_API: ] urlpatterns += [ - url(r'^api/(?P(1.0))/', include(api_patterns)), + url(r'^api/(?:(?P(1.0))/)?', include(api_patterns)), ] -- 2.9.3 From philippe.pepiot at logilab.fr Sat Apr 29 02:14:42 2017 From: philippe.pepiot at logilab.fr (Philippe Pepiot) Date: Fri, 28 Apr 2017 18:14:42 +0200 Subject: [PATCH] parser: detect mercurial series markers Message-ID: <1493396082-28383-1-git-send-email-philippe.pepiot@logilab.fr> Unlike git, Mercurial series marker uses the format "M of N", handle this in the subject parser. Test code original author is Andrew Shadura Signed-off-by: Philippe Pepiot --- patchwork/parser.py | 13 +- patchwork/tests/series/mercurial-cover-letter.mbox | 194 +++++++++++++++++++++ .../tests/series/mercurial-no-cover-letter.mbox | 154 ++++++++++++++++ patchwork/tests/test_parser.py | 2 + patchwork/tests/test_series.py | 44 +++++ 5 files changed, 404 insertions(+), 3 deletions(-) create mode 100644 patchwork/tests/series/mercurial-cover-letter.mbox create mode 100644 patchwork/tests/series/mercurial-no-cover-letter.mbox diff --git a/patchwork/parser.py b/patchwork/parser.py index e03cc4b..6d4640a 100644 --- a/patchwork/parser.py +++ b/patchwork/parser.py @@ -311,7 +311,7 @@ def parse_series_marker(subject_prefixes): (x, n) if markers found, else (None, None) """ - regex = re.compile('^([0-9]+)/([0-9]+)$') + regex = re.compile(r'^([0-9]+)(?:/| of )([0-9]+)$') m = _find_matching_prefix(subject_prefixes, regex) if m: return (int(m.group(1)), int(m.group(2))) @@ -471,10 +471,17 @@ def find_submission_for_comment(project, refs): def split_prefixes(prefix): """Turn a prefix string into a list of prefix tokens.""" + tokens = [] + # detect mercurial series marker (M of N) + series_re = re.compile(r'^PATCH (\d+ of \d+)(.*)$') + match = series_re.match(prefix) + if match is not None: + series, prefix = match.groups() + tokens.extend(['PATCH', series]) split_re = re.compile(r'[,\s]+') matches = split_re.split(prefix) - - return [s for s in matches if s != ''] + tokens.extend([s for s in matches if s != '']) + return tokens def clean_subject(subject, drop_prefixes=None): diff --git a/patchwork/tests/series/mercurial-cover-letter.mbox b/patchwork/tests/series/mercurial-cover-letter.mbox new file mode 100644 index 0000000..34cd6f8 --- /dev/null +++ b/patchwork/tests/series/mercurial-cover-letter.mbox @@ -0,0 +1,194 @@ +From andrew at shadura.me Mon Jan 23 08:18:28 2017 +Return-Path: +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Subject: [PATCH 0 of 2] Sample Mercurial patches +Message-Id: +User-Agent: Mercurial-patchbomb/3.9.1 +Date: Mon, 23 Jan 2017 09:16:25 +0100 +From: Andrew Shadura +To: Stephen Finucane +Cc: andrew.shadura at collabora.co.uk +Content-Transfer-Encoding: 8bit + +Hi, + +On 22/01/17 22:28, Stephen Finucane wrote: +> On Sun, 2017-01-22 at 01:44 +0100, Andrew Shadura wrote: +>> Unlike Git, Mercurial produces email subject lines of the +>> following format: "[PATCH M of N] ...". '(?:/| of )' pattern +>> matches both 'M/N' and 'M of N' formats. +>> +>> Signed-off-by: Andrew Shadura +> +> This looks a-OK to me, but I would like to add an additional test case +> in patchwork/tests/test_series.py to prevent this regressing if we +> could. I can write the tests but I need some mbox files to validate +> against. Could you supply me with some sample, Mercurial-produced patch +> emails for this (email me directly)? A two patch series, with and +> without a cover letter, would suffice (look at the files in +> patchwork/tests/series for examples). + +Sure, here you go. + +P.S. Sorry for (possibly) spamming the mailing list by multiple patch submissions. +It seems the list is moderated, right? + +From andrew at shadura.me Mon Jan 23 08:18:29 2017 +Return-Path: +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Subject: [PATCH 1 of 2] contrib: fix check-commit to not reject commits + from `hg sign` and `hg tag` +X-Mercurial-Node: 2fb3ae89e4e1992e93826c0e3f04e4f965edb0ab +X-Mercurial-Series-Index: 1 +X-Mercurial-Series-Total: 2 +Message-Id: <2fb3ae89e4e1992e9382.1485159386 at nuevo> +X-Mercurial-Series-Id: <2fb3ae89e4e1992e9382.1485159386 at nuevo> +In-Reply-To: +References: +User-Agent: Mercurial-patchbomb/3.9.1 +Date: Mon, 23 Jan 2017 09:16:26 +0100 +From: Andrew Shadura +To: Stephen Finucane +Cc: andrew.shadura at collabora.co.uk +Content-Transfer-Encoding: 8bit + +# HG changeset patch +# User Augie Fackler +# Date 1484800475 18000 +# Wed Jan 18 23:34:35 2017 -0500 +# Branch stable +# Node ID 2fb3ae89e4e1992e93826c0e3f04e4f965edb0ab +# Parent 94af7d0c812fe7d3a5651191685ca43e1a331814 +contrib: fix check-commit to not reject commits from `hg sign` and `hg tag` + +I'm tired of having a spurious red build every time we do a +release. Fix it once and for all. + +diff --git a/contrib/check-commit b/contrib/check-commit +--- a/contrib/check-commit ++++ b/contrib/check-commit +@@ -59,6 +59,10 @@ def checkcommit(commit, node=None): + exitcode = 0 + printed = node is None + hits = [] ++ signtag = (afterheader + ++ r'Added (tag [^ ]+|signature) for changeset [a-f0-9]{12}') ++ if re.search(signtag, commit): ++ return 0 + for exp, msg in errors: + for m in re.finditer(exp, commit): + end = m.end() +diff --git a/tests/test-contrib-check-commit.t b/tests/test-contrib-check-commit.t +--- a/tests/test-contrib-check-commit.t ++++ b/tests/test-contrib-check-commit.t +@@ -30,6 +30,30 @@ A fine patch: + > EOF + $ cat patch-with-long-header.diff | $TESTDIR/../contrib/check-commit + ++This would normally be against the rules, but it's okay because that's ++what tagging and signing looks like: ++ ++ $ cat > creates-a-tag.diff << EOF ++ > # HG changeset patch ++ > # User Augie Fackler ++ > # Date 1484787778 18000 ++ > # Wed Jan 18 20:02:58 2017 -0500 ++ > # Branch stable ++ > # Node ID c177635e4acf52923bc3aa9f72a5b1ad1197b173 ++ > # Parent a1dd2c0c479e0550040542e392e87bc91262517e ++ > Added tag 4.1-rc for changeset a1dd2c0c479e ++ > ++ > diff --git a/.hgtags b/.hgtags ++ > --- a/.hgtags ++ > +++ b/.hgtags ++ > @@ -150,3 +150,4 @@ 438173c415874f6ac653efc1099dec9c9150e90f ++ > eab27446995210c334c3d06f1a659e3b9b5da769 4.0 ++ > b3b1ae98f6a0e14c1e1ba806a6c18e193b6dae5c 4.0.1 ++ > e69874dc1f4e142746ff3df91e678a09c6fc208c 4.0.2 ++ > +a1dd2c0c479e0550040542e392e87bc91262517e 4.1-rc ++ > EOF ++ $ $TESTDIR/../contrib/check-commit < creates-a-tag.diff ++ + A patch with lots of errors: + + $ cat > patch-with-long-header.diff << EOF + +From andrew at shadura.me Mon Jan 23 08:18:35 2017 +Return-Path: +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Subject: [PATCH 2 of 2] tests: work around FreeBSD's unzip having slightly + different output +X-Mercurial-Node: b3d2e8cce78c04dbf6f20f0846b6ea8b622983c4 +X-Mercurial-Series-Index: 2 +X-Mercurial-Series-Total: 2 +Message-Id: +X-Mercurial-Series-Id: <2fb3ae89e4e1992e9382.1485159386 at nuevo> +In-Reply-To: +References: +User-Agent: Mercurial-patchbomb/3.9.1 +Date: Mon, 23 Jan 2017 09:16:27 +0100 +From: Andrew Shadura +To: Stephen Finucane +Cc: andrew.shadura at collabora.co.uk +Content-Transfer-Encoding: 8bit + +# HG changeset patch +# User Augie Fackler +# Date 1484801021 18000 +# Wed Jan 18 23:43:41 2017 -0500 +# Branch stable +# Node ID b3d2e8cce78c04dbf6f20f0846b6ea8b622983c4 +# Parent 2fb3ae89e4e1992e93826c0e3f04e4f965edb0ab +tests: work around FreeBSD's unzip having slightly different output + +According to man 1 unzip, this unzip appeared in FreeBSD 8.0. It's +what comes as /usr/bin/unzip, so we may as well cater to it since it's +easy. + +diff --git a/tests/test-archive.t b/tests/test-archive.t +--- a/tests/test-archive.t ++++ b/tests/test-archive.t +@@ -327,14 +327,11 @@ old file -- date clamped to 1980 + $ hg add old + $ hg commit -m old + $ hg archive ../old.zip +- $ unzip -l ../old.zip ++ $ unzip -l ../old.zip | grep -v -- ----- | egrep -v files$ + Archive: ../old.zip + \s*Length.* (re) +- *-----* (glob) + *172*80*00:00*old/.hg_archival.txt (glob) + *0*80*00:00*old/old (glob) +- *-----* (glob) +- \s*172\s+2 files (re) + + show an error when a provided pattern matches no files + +diff --git a/tests/test-subrepo-recursion.t b/tests/test-subrepo-recursion.t +--- a/tests/test-subrepo-recursion.t ++++ b/tests/test-subrepo-recursion.t +@@ -321,10 +321,9 @@ Test archiving to zip file (unzip output + + (unzip date formating is unstable, we do not care about it and glob it out) + +- $ unzip -l ../archive.zip ++ $ unzip -l ../archive.zip | grep -v -- ----- | egrep -v files$ + Archive: ../archive.zip + Length [ ]* Date [ ]* Time [ ]* Name (re) +- [\- ]* (re) + 172 [0-9:\- ]* .hg_archival.txt (re) + 10 [0-9:\- ]* .hgsub (re) + 45 [0-9:\- ]* .hgsubstate (re) +@@ -333,8 +332,6 @@ Test archiving to zip file (unzip output + 45 [0-9:\- ]* foo/.hgsubstate (re) + 9 [0-9:\- ]* foo/y.txt (re) + 9 [0-9:\- ]* foo/bar/z.txt (re) +- [\- ]* (re) +- 303 [ ]* 8 files (re) + + Test archiving a revision that references a subrepo that is not yet + cloned: + diff --git a/patchwork/tests/series/mercurial-no-cover-letter.mbox b/patchwork/tests/series/mercurial-no-cover-letter.mbox new file mode 100644 index 0000000..9006ab8 --- /dev/null +++ b/patchwork/tests/series/mercurial-no-cover-letter.mbox @@ -0,0 +1,154 @@ +From andrew at shadura.me Mon Jan 23 08:19:08 2017 +Return-Path: +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Subject: [PATCH 1 of 2] contrib: fix check-commit to not reject commits + from `hg sign` and `hg tag` +X-Mercurial-Node: 2fb3ae89e4e1992e93826c0e3f04e4f965edb0ab +X-Mercurial-Series-Index: 1 +X-Mercurial-Series-Total: 2 +Message-Id: <2fb3ae89e4e1992e9382.1485159499 at nuevo> +X-Mercurial-Series-Id: <2fb3ae89e4e1992e9382.1485159499 at nuevo> +User-Agent: Mercurial-patchbomb/3.9.1 +Date: Mon, 23 Jan 2017 09:18:19 +0100 +From: Andrew Shadura +To: Stephen Finucane +Content-Transfer-Encoding: 8bit + +# HG changeset patch +# User Augie Fackler +# Date 1484800475 18000 +# Wed Jan 18 23:34:35 2017 -0500 +# Branch stable +# Node ID 2fb3ae89e4e1992e93826c0e3f04e4f965edb0ab +# Parent 94af7d0c812fe7d3a5651191685ca43e1a331814 +contrib: fix check-commit to not reject commits from `hg sign` and `hg tag` + +I'm tired of having a spurious red build every time we do a +release. Fix it once and for all. + +diff --git a/contrib/check-commit b/contrib/check-commit +--- a/contrib/check-commit ++++ b/contrib/check-commit +@@ -59,6 +59,10 @@ def checkcommit(commit, node=None): + exitcode = 0 + printed = node is None + hits = [] ++ signtag = (afterheader + ++ r'Added (tag [^ ]+|signature) for changeset [a-f0-9]{12}') ++ if re.search(signtag, commit): ++ return 0 + for exp, msg in errors: + for m in re.finditer(exp, commit): + end = m.end() +diff --git a/tests/test-contrib-check-commit.t b/tests/test-contrib-check-commit.t +--- a/tests/test-contrib-check-commit.t ++++ b/tests/test-contrib-check-commit.t +@@ -30,6 +30,30 @@ A fine patch: + > EOF + $ cat patch-with-long-header.diff | $TESTDIR/../contrib/check-commit + ++This would normally be against the rules, but it's okay because that's ++what tagging and signing looks like: ++ ++ $ cat > creates-a-tag.diff << EOF ++ > # HG changeset patch ++ > # User Augie Fackler ++ > # Date 1484787778 18000 ++ > # Wed Jan 18 20:02:58 2017 -0500 ++ > # Branch stable ++ > # Node ID c177635e4acf52923bc3aa9f72a5b1ad1197b173 ++ > # Parent a1dd2c0c479e0550040542e392e87bc91262517e ++ > Added tag 4.1-rc for changeset a1dd2c0c479e ++ > ++ > diff --git a/.hgtags b/.hgtags ++ > --- a/.hgtags ++ > +++ b/.hgtags ++ > @@ -150,3 +150,4 @@ 438173c415874f6ac653efc1099dec9c9150e90f ++ > eab27446995210c334c3d06f1a659e3b9b5da769 4.0 ++ > b3b1ae98f6a0e14c1e1ba806a6c18e193b6dae5c 4.0.1 ++ > e69874dc1f4e142746ff3df91e678a09c6fc208c 4.0.2 ++ > +a1dd2c0c479e0550040542e392e87bc91262517e 4.1-rc ++ > EOF ++ $ $TESTDIR/../contrib/check-commit < creates-a-tag.diff ++ + A patch with lots of errors: + + $ cat > patch-with-long-header.diff << EOF + +From andrew at shadura.me Mon Jan 23 08:19:08 2017 +Return-Path: +Content-Type: text/plain; charset="us-ascii" +MIME-Version: 1.0 +Subject: [PATCH 2 of 2] tests: work around FreeBSD's unzip having slightly + different output +X-Mercurial-Node: b3d2e8cce78c04dbf6f20f0846b6ea8b622983c4 +X-Mercurial-Series-Index: 2 +X-Mercurial-Series-Total: 2 +Message-Id: +X-Mercurial-Series-Id: <2fb3ae89e4e1992e9382.1485159499 at nuevo> +In-Reply-To: <2fb3ae89e4e1992e9382.1485159499 at nuevo> +References: <2fb3ae89e4e1992e9382.1485159499 at nuevo> +User-Agent: Mercurial-patchbomb/3.9.1 +Date: Mon, 23 Jan 2017 09:18:20 +0100 +From: Andrew Shadura +To: Stephen Finucane +Content-Transfer-Encoding: 8bit + +# HG changeset patch +# User Augie Fackler +# Date 1484801021 18000 +# Wed Jan 18 23:43:41 2017 -0500 +# Branch stable +# Node ID b3d2e8cce78c04dbf6f20f0846b6ea8b622983c4 +# Parent 2fb3ae89e4e1992e93826c0e3f04e4f965edb0ab +tests: work around FreeBSD's unzip having slightly different output + +According to man 1 unzip, this unzip appeared in FreeBSD 8.0. It's +what comes as /usr/bin/unzip, so we may as well cater to it since it's +easy. + +diff --git a/tests/test-archive.t b/tests/test-archive.t +--- a/tests/test-archive.t ++++ b/tests/test-archive.t +@@ -327,14 +327,11 @@ old file -- date clamped to 1980 + $ hg add old + $ hg commit -m old + $ hg archive ../old.zip +- $ unzip -l ../old.zip ++ $ unzip -l ../old.zip | grep -v -- ----- | egrep -v files$ + Archive: ../old.zip + \s*Length.* (re) +- *-----* (glob) + *172*80*00:00*old/.hg_archival.txt (glob) + *0*80*00:00*old/old (glob) +- *-----* (glob) +- \s*172\s+2 files (re) + + show an error when a provided pattern matches no files + +diff --git a/tests/test-subrepo-recursion.t b/tests/test-subrepo-recursion.t +--- a/tests/test-subrepo-recursion.t ++++ b/tests/test-subrepo-recursion.t +@@ -321,10 +321,9 @@ Test archiving to zip file (unzip output + + (unzip date formating is unstable, we do not care about it and glob it out) + +- $ unzip -l ../archive.zip ++ $ unzip -l ../archive.zip | grep -v -- ----- | egrep -v files$ + Archive: ../archive.zip + Length [ ]* Date [ ]* Time [ ]* Name (re) +- [\- ]* (re) + 172 [0-9:\- ]* .hg_archival.txt (re) + 10 [0-9:\- ]* .hgsub (re) + 45 [0-9:\- ]* .hgsubstate (re) +@@ -333,8 +332,6 @@ Test archiving to zip file (unzip output + 45 [0-9:\- ]* foo/.hgsubstate (re) + 9 [0-9:\- ]* foo/y.txt (re) + 9 [0-9:\- ]* foo/bar/z.txt (re) +- [\- ]* (re) +- 303 [ ]* 8 files (re) + + Test archiving a revision that references a subrepo that is not yet + cloned: + diff --git a/patchwork/tests/test_parser.py b/patchwork/tests/test_parser.py index 61d8d80..57a3f0d 100644 --- a/patchwork/tests/test_parser.py +++ b/patchwork/tests/test_parser.py @@ -807,6 +807,8 @@ class SubjectTest(TestCase): self.assertEqual(parse_series_marker(['bar']), (None, None)) self.assertEqual(parse_series_marker(['bar', '1/2']), (1, 2)) self.assertEqual(parse_series_marker(['bar', '0/12']), (0, 12)) + self.assertEqual(parse_series_marker(['bar', '1 of 2']), (1, 2)) + self.assertEqual(parse_series_marker(['bar', '0 of 12']), (0, 12)) def test_version(self): self.assertEqual(parse_version('', []), 1) diff --git a/patchwork/tests/test_series.py b/patchwork/tests/test_series.py index d2514c6..8de94a0 100644 --- a/patchwork/tests/test_series.py +++ b/patchwork/tests/test_series.py @@ -438,6 +438,50 @@ class SeriesTotalTest(_BaseTestCase): self.assertTrue(series.received_all) +class MercurialSeriesTest(_BaseTestCase): + """Tests for a series without any revisions. + + All patches are generated using hg(1) email, provided via the Patchbomb + extension. + """ + + def test_cover_letter(self): + """Series with a cover letter. + + Parse a series with a cover letter and two patches. + + Input: + + - [PATCH 0 of 2] Sample Mercurial patches + - [PATCH 1 of 2] contrib: fix check-commit to not reject + commits from `hg sign` and `hg tag` + - [PATCH 2 of 2] tests: work around FreeBSD's unzip having + slightly different output + """ + covers, patches, comments = self._parse_mbox( + 'mercurial-cover-letter.mbox', [1, 2, 0]) + + self.assertSerialized(patches, [2]) + self.assertSerialized(covers, [1]) + + def test_no_cover_letter(self): + """Series without a cover letter. + + Parse a series with two patches but no cover letter. + + Input: + + - [PATCH 1 of 2] contrib: fix check-commit to not reject + commits from `hg sign` and `hg tag` + - [PATCH 2 of 2] tests: work around FreeBSD's unzip having + slightly different output + """ + _, patches, _ = self._parse_mbox( + 'mercurial-no-cover-letter.mbox', [0, 2, 0]) + + self.assertSerialized(patches, [2]) + + class SeriesNameTestCase(TestCase): def setUp(self): -- 2.1.4 From sean at farley.io Sat Apr 29 03:50:48 2017 From: sean at farley.io (Sean Farley) Date: Fri, 28 Apr 2017 10:50:48 -0700 Subject: [PATCH] parser: detect mercurial series markers In-Reply-To: <1493396082-28383-1-git-send-email-philippe.pepiot@logilab.fr> References: <1493396082-28383-1-git-send-email-philippe.pepiot@logilab.fr> Message-ID: Philippe Pepiot writes: > Unlike git, Mercurial series marker uses the format "M of N", handle > this in the subject parser. > > Test code original author is Andrew Shadura Oh snap! You beat me to cleaning up and submitting mine! -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 800 bytes Desc: not available URL: From stephen at that.guru Sat Apr 29 07:46:03 2017 From: stephen at that.guru (Stephen Finucane) Date: Fri, 28 Apr 2017 22:46:03 +0100 Subject: [PATCH v2] pwclient: Fix silent crash on python 2 In-Reply-To: <20170405124604.28022-1-robin.jarry@6wind.com> References: <20170405113552.7vbiyjgnfykoojmn@6wind.com> <20170405124604.28022-1-robin.jarry@6wind.com> Message-ID: <1493415963.3875.5.camel@that.guru> On Wed, 2017-04-05 at 14:46 +0200, Robin Jarry wrote: > Replacing sys.stdout and sys.stderr can cause obscure crashes when > trying to write non unicode data. The interpreter is terminated with > SIGINT without any specific error writen on the console. > > rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7f964820e8d0}, > {0x559f50, [], SA_RESTORER, 0x7f964820e8d0}, 8) = 0 > > This happens easily when there is an untrapped exception which should > lead to printing a traceback on stderr. > > The only way to prevent UnicodeEncodeErrors is to make sure that > PYTHONIOENCODING is set with the ':replace' suffix and this can only > be > done *before* starting the interpreter as the initialization is made > very early on and the encoding cannot be set or modified after. > > From the official documentation: > > PYTHONIOENCODING > > Overrides the encoding used for stdin/stdout/stderr, in the syntax > encodingname:errorhandler. The :errorhandler part is optional and > has the same meaning as in str.encode(). > > https://docs.python.org/2/using/cmdline.html > https://docs.python.org/3/using/cmdline.html > > Of course, for proper encoding of unicode characters, one of the > locale-related environment variables (LC_ALL, LANG, LANGUAGE, etc.) > must > be set. Python will use the correct encoding accordingly and > PYTHONIOENCODING will be set to "encoding:replace". > > Examples: > > $ grep utf8 ~/.Xresources > xterm*utf8: 2 > > $ env - PYTHONIOENCODING=utf-8:replace python2 -c "print > u's\u00e9duisante'" > séduisante > $ env - PYTHONIOENCODING=utf-8:replace python3 -c > "print('s\u00e9duisante')" > séduisante > > $ env - PYTHONIOENCODING=ascii:replace python2 -c "print > u's\u00e9duisante'" > s?duisante > $ env - PYTHONIOENCODING=ascii:replace python3 -c > "print('s\u00e9duisante')" > s?duisante > > $ env - PYTHONIOENCODING=ISO-8859-1:replace python2 -c "print > u's\u00e9duisante'" > s�duisante > $ env - PYTHONIOENCODING=ISO-8859-1:replace python3 -c > "print('s\u00e9duisante')" > s�duisante > > Fixes: 046419a3bf8f ("pwclient: Fix encoding problems") > Signed-off-by: Robin Jarry I understand the issue now - thanks for clarifying :) Sorry for the delay in replying also. Now, while I understand what you're doing here, I'm wondering if we could simply revert the original patch instead. I clearly merged that without fully understanding the issue and not only does it cause the issues you've trying to resolve here but it seems it's not necessary for a lot of people (including me. Fedora 24). For those users that are affected, there are less hacky workarounds. For example, a simple script like so would so the job, I imagine. $ cat pwclient2 #!/usr/bin/env bash PYTHONIOENCODING=UTF-8 pwclient I'd be happy to include an epilog detailing this issue that would appear when you'd run '--help'. I realize that were still be an extra step for affected users, but the vast majority aren't and wouldn't be affected. Would this be acceptable? Stephen PS: What kind of environment are you encountering this issue in? > --- > v2: > > - Always set PYTHONIOENCODING=:replace (on python 2 and 3) to > prevent from > UnicodeEncodeErrors > > patchwork/bin/pwclient | 35 ++++++++++++++++++++++++++--------- > 1 file changed, 26 insertions(+), 9 deletions(-) > > diff --git a/patchwork/bin/pwclient b/patchwork/bin/pwclient > index ed0351bf5288..5a7b6723afe3 100755 > --- a/patchwork/bin/pwclient > +++ b/patchwork/bin/pwclient > @@ -41,16 +41,7 @@ except ImportError: > import shutil > import re > import io > -import locale > > -if sys.version_info.major == 2: > - # hack to make writing unicode to standard output/error work on > Python 2 > - OUT_ENCODING = (sys.stdout.encoding or > locale.getpreferredencoding() or > - os.getenv('PYTHONIOENCODING', 'utf-8')) > - sys.stdout = io.open(sys.stdout.fileno(), mode='w', > - encoding=OUT_ENCODING, errors='replace') > - sys.stderr = io.open(sys.stderr.fileno(), mode='w', > - encoding=OUT_ENCODING, errors='replace') > > # Default Patchwork remote XML-RPC server URL > # This script will check the PW_XMLRPC_URL environment variable > @@ -821,5 +812,31 @@ def main(): > sys.exit(1) > > > +def force_io_encoding(): > + """ > + Force PYTHONIOENCODING ":errorhandler" to avoid > UnicodeEncodeErrors. The > + only way to do it is to set the environment variable *before* > starting the > + interpreter. From the python docs: > + > + PYTHONIOENCODING > + > + Overrides the encoding used for stdin/stdout/stderr, in the > syntax > + encodingname:errorhandler. The :errorhandler part is optional > and has the > + same meaning as in str.encode(). > + > + Note that this only prevents interpreter crashes, it does not > exempt from > + correctly setting the LANG or LC_ALL variables in order to have > valid > + output. > + """ > + if 'PYTHONIOENCODING' in os.environ: > + return > + > + encoding = sys.stdout.encoding or 'utf-8' > + > + os.environ['PYTHONIOENCODING'] = encoding + ':replace' > + os.execvp(sys.executable, [sys.executable] + sys.argv) # no > return > + > + > if __name__ == "__main__": > + force_io_encoding() > main() From stephen at that.guru Sat Apr 29 07:47:17 2017 From: stephen at that.guru (Stephen Finucane) Date: Fri, 28 Apr 2017 22:47:17 +0100 Subject: [PATCH 1/4] compat: Handle older versions of django-filter In-Reply-To: <20170404213820.13350-1-stephen@that.guru> References: <20170404213820.13350-1-stephen@that.guru> Message-ID: <1493416037.3875.6.camel@that.guru> On Tue, 2017-04-04 at 22:38 +0100, Stephen Finucane wrote: > Recent versions of Django REST Framework (DRF) have deprecated the > 'DjangoFilterBackend' filter found in-tree, in favour of an > equivalent > implementation found in django-filter. However, we need to support > older > versions of DRF for users who want to use system packages. > > Seeing as the two implementations are, for all intents and purposes, > essentially the same thing, provide a shim that will allow us to use > both together. > > Signed-off-by: Stephen Finucane > Cc: Daniel Axtens > Partial-bug: #94 I've gone ahead and merged all but the last of these (which served well enough as a temporary test, but is tediously slow to keep around long term). Stephen From stephen at that.guru Sat Apr 29 08:26:47 2017 From: stephen at that.guru (Stephen Finucane) Date: Fri, 28 Apr 2017 23:26:47 +0100 Subject: [PATCH] parser: detect mercurial series markers In-Reply-To: <1493396082-28383-1-git-send-email-philippe.pepiot@logilab.fr> References: <1493396082-28383-1-git-send-email-philippe.pepiot@logilab.fr> Message-ID: <1493418407.3875.8.camel@that.guru> On Fri, 2017-04-28 at 18:14 +0200, Philippe Pepiot wrote: > Unlike git, Mercurial series marker uses the format "M of N", handle > this in the subject parser. > > Test code original author is Andrew Shadura .co.uk> > > Signed-off-by: Philippe Pepiot Spot on, Philippe. Merged. Thanks, Stephen > --- > patchwork/parser.py | 13 +- > patchwork/tests/series/mercurial-cover-letter.mbox | 194 > +++++++++++++++++++++ > .../tests/series/mercurial-no-cover-letter.mbox | 154 > ++++++++++++++++ > patchwork/tests/test_parser.py | 2 + > patchwork/tests/test_series.py | 44 +++++ > 5 files changed, 404 insertions(+), 3 deletions(-) > create mode 100644 patchwork/tests/series/mercurial-cover- > letter.mbox > create mode 100644 patchwork/tests/series/mercurial-no-cover- > letter.mbox > > diff --git a/patchwork/parser.py b/patchwork/parser.py > index e03cc4b..6d4640a 100644 > --- a/patchwork/parser.py > +++ b/patchwork/parser.py > @@ -311,7 +311,7 @@ def parse_series_marker(subject_prefixes): > (x, n) if markers found, else (None, None) > """ > > - regex = re.compile('^([0-9]+)/([0-9]+)$') > + regex = re.compile(r'^([0-9]+)(?:/| of )([0-9]+)$') > m = _find_matching_prefix(subject_prefixes, regex) > if m: > return (int(m.group(1)), int(m.group(2))) > @@ -471,10 +471,17 @@ def find_submission_for_comment(project, refs): > > def split_prefixes(prefix): > """Turn a prefix string into a list of prefix tokens.""" > + tokens = [] > + # detect mercurial series marker (M of N) > + series_re = re.compile(r'^PATCH (\d+ of \d+)(.*)$') > + match = series_re.match(prefix) > + if match is not None: > + series, prefix = match.groups() > + tokens.extend(['PATCH', series]) > split_re = re.compile(r'[,\s]+') > matches = split_re.split(prefix) > - > - return [s for s in matches if s != ''] > + tokens.extend([s for s in matches if s != '']) > + return tokens > > > def clean_subject(subject, drop_prefixes=None): > diff --git a/patchwork/tests/series/mercurial-cover-letter.mbox > b/patchwork/tests/series/mercurial-cover-letter.mbox > new file mode 100644 > index 0000000..34cd6f8 > --- /dev/null > +++ b/patchwork/tests/series/mercurial-cover-letter.mbox > @@ -0,0 +1,194 @@ > +From andrew at shadura.me Mon Jan 23 08:18:28 2017 > +Return-Path: > +Content-Type: text/plain; charset="us-ascii" > +MIME-Version: 1.0 > +Subject: [PATCH 0 of 2] Sample Mercurial patches > +Message-Id: > +User-Agent: Mercurial-patchbomb/3.9.1 > +Date: Mon, 23 Jan 2017 09:16:25 +0100 > +From: Andrew Shadura > +To: Stephen Finucane > +Cc: andrew.shadura at collabora.co.uk > +Content-Transfer-Encoding: 8bit > + > +Hi, > + > +On 22/01/17 22:28, Stephen Finucane wrote: > +> On Sun, 2017-01-22 at 01:44 +0100, Andrew Shadura wrote: > +>> Unlike Git, Mercurial produces email subject lines of the > +>> following format: "[PATCH M of N] ...". '(?:/| of )' pattern > +>> matches both 'M/N' and 'M of N' formats. > +>> > +>> Signed-off-by: Andrew Shadura > +> > +> This looks a-OK to me, but I would like to add an additional test > case > +> in patchwork/tests/test_series.py to prevent this regressing if we > +> could. I can write the tests but I need some mbox files to > validate > +> against. Could you supply me with some sample, Mercurial-produced > patch > +> emails for this (email me directly)? A two patch series, with and > +> without a cover letter, would suffice (look at the files in > +> patchwork/tests/series for examples). > + > +Sure, here you go. > + > +P.S. Sorry for (possibly) spamming the mailing list by multiple > patch submissions. > +It seems the list is moderated, right? > + > +From andrew at shadura.me Mon Jan 23 08:18:29 2017 > +Return-Path: > +Content-Type: text/plain; charset="us-ascii" > +MIME-Version: 1.0 > +Subject: [PATCH 1 of 2] contrib: fix check-commit to not reject > commits > + from `hg sign` and `hg tag` > +X-Mercurial-Node: 2fb3ae89e4e1992e93826c0e3f04e4f965edb0ab > +X-Mercurial-Series-Index: 1 > +X-Mercurial-Series-Total: 2 These look useful. We could probably use them in the future to validate stuff. I wish Git would simply include these itself. > +Message-Id: <2fb3ae89e4e1992e9382.1485159386 at nuevo> > +X-Mercurial-Series-Id: <2fb3ae89e4e1992e9382.1485159386 at nuevo> > +In-Reply-To: > +References: > +User-Agent: Mercurial-patchbomb/3.9.1 > +Date: Mon, 23 Jan 2017 09:16:26 +0100 > +From: Andrew Shadura > +To: Stephen Finucane > +Cc: andrew.shadura at collabora.co.uk > +Content-Transfer-Encoding: 8bit > + > +# HG changeset patch > +# User Augie Fackler > +# Date 1484800475 18000 > +# Wed Jan 18 23:34:35 2017 -0500 > +# Branch stable > +# Node ID 2fb3ae89e4e1992e93826c0e3f04e4f965edb0ab > +# Parent 94af7d0c812fe7d3a5651191685ca43e1a331814 > +contrib: fix check-commit to not reject commits from `hg sign` and > `hg tag` > + > +I'm tired of having a spurious red build every time we do a > +release. Fix it once and for all. > + > +diff --git a/contrib/check-commit b/contrib/check-commit > +--- a/contrib/check-commit > ++++ b/contrib/check-commit > +@@ -59,6 +59,10 @@ def checkcommit(commit, node=None): > + exitcode = 0 > + printed = node is None > + hits = [] > ++ signtag = (afterheader + > ++ r'Added (tag [^ ]+|signature) for changeset [a-f0- > 9]{12}') > ++ if re.search(signtag, commit): > ++ return 0 > + for exp, msg in errors: > + for m in re.finditer(exp, commit): > + end = m.end() > +diff --git a/tests/test-contrib-check-commit.t b/tests/test-contrib- > check-commit.t > +--- a/tests/test-contrib-check-commit.t > ++++ b/tests/test-contrib-check-commit.t > +@@ -30,6 +30,30 @@ A fine patch: > + > EOF > + $ cat patch-with-long-header.diff | $TESTDIR/../contrib/check- > commit > + > ++This would normally be against the rules, but it's okay because > that's > ++what tagging and signing looks like: > ++ > ++ $ cat > creates-a-tag.diff << EOF > ++ > # HG changeset patch > ++ > # User Augie Fackler > ++ > # Date 1484787778 18000 > ++ > # Wed Jan 18 20:02:58 2017 -0500 > ++ > # Branch stable > ++ > # Node ID c177635e4acf52923bc3aa9f72a5b1ad1197b173 > ++ > # Parent a1dd2c0c479e0550040542e392e87bc91262517e > ++ > Added tag 4.1-rc for changeset a1dd2c0c479e > ++ > > ++ > diff --git a/.hgtags b/.hgtags > ++ > --- a/.hgtags > ++ > +++ b/.hgtags > ++ > @@ -150,3 +150,4 @@ 438173c415874f6ac653efc1099dec9c9150e90f > ++ > eab27446995210c334c3d06f1a659e3b9b5da769 4.0 > ++ > b3b1ae98f6a0e14c1e1ba806a6c18e193b6dae5c 4.0.1 > ++ > e69874dc1f4e142746ff3df91e678a09c6fc208c 4.0.2 > ++ > +a1dd2c0c479e0550040542e392e87bc91262517e 4.1-rc > ++ > EOF > ++ $ $TESTDIR/../contrib/check-commit < creates-a-tag.diff > ++ > + A patch with lots of errors: > + > + $ cat > patch-with-long-header.diff << EOF > + > +From andrew at shadura.me Mon Jan 23 08:18:35 2017 > +Return-Path: > +Content-Type: text/plain; charset="us-ascii" > +MIME-Version: 1.0 > +Subject: [PATCH 2 of 2] tests: work around FreeBSD's unzip having > slightly > + different output > +X-Mercurial-Node: b3d2e8cce78c04dbf6f20f0846b6ea8b622983c4 > +X-Mercurial-Series-Index: 2 > +X-Mercurial-Series-Total: 2 > +Message-Id: > +X-Mercurial-Series-Id: <2fb3ae89e4e1992e9382.1485159386 at nuevo> > +In-Reply-To: > +References: > +User-Agent: Mercurial-patchbomb/3.9.1 > +Date: Mon, 23 Jan 2017 09:16:27 +0100 > +From: Andrew Shadura > +To: Stephen Finucane > +Cc: andrew.shadura at collabora.co.uk > +Content-Transfer-Encoding: 8bit > + > +# HG changeset patch > +# User Augie Fackler > +# Date 1484801021 18000 > +# Wed Jan 18 23:43:41 2017 -0500 > +# Branch stable > +# Node ID b3d2e8cce78c04dbf6f20f0846b6ea8b622983c4 > +# Parent 2fb3ae89e4e1992e93826c0e3f04e4f965edb0ab > +tests: work around FreeBSD's unzip having slightly different output > + > +According to man 1 unzip, this unzip appeared in FreeBSD 8.0. It's > +what comes as /usr/bin/unzip, so we may as well cater to it since > it's > +easy. > + > +diff --git a/tests/test-archive.t b/tests/test-archive.t > +--- a/tests/test-archive.t > ++++ b/tests/test-archive.t > +@@ -327,14 +327,11 @@ old file -- date clamped to 1980 > + $ hg add old > + $ hg commit -m old > + $ hg archive ../old.zip > +- $ unzip -l ../old.zip > ++ $ unzip -l ../old.zip | grep -v -- ----- | egrep -v files$ > + Archive: ../old.zip > + \s*Length.* (re) > +- *-----* (glob) > + *172*80*00:00*old/.hg_archival.txt (glob) > + *0*80*00:00*old/old (glob) > +- *-----* (glob) > +- \s*172\s+2 files (re) > + > + show an error when a provided pattern matches no files > + > +diff --git a/tests/test-subrepo-recursion.t b/tests/test-subrepo- > recursion.t > +--- a/tests/test-subrepo-recursion.t > ++++ b/tests/test-subrepo-recursion.t > +@@ -321,10 +321,9 @@ Test archiving to zip file (unzip output > + > + (unzip date formating is unstable, we do not care about it and glob > it out) > + > +- $ unzip -l ../archive.zip > ++ $ unzip -l ../archive.zip | grep -v -- ----- | egrep -v files$ > + Archive: ../archive.zip > + Length [ ]* Date [ ]* Time [ ]* Name (re) > +- [\- ]* (re) > + 172 [0-9:\- ]* .hg_archival.txt (re) > + 10 [0-9:\- ]* .hgsub (re) > + 45 [0-9:\- ]* .hgsubstate (re) > +@@ -333,8 +332,6 @@ Test archiving to zip file (unzip output > + 45 [0-9:\- ]* foo/.hgsubstate (re) > + 9 [0-9:\- ]* foo/y.txt (re) > + 9 [0-9:\- ]* foo/bar/z.txt (re) > +- [\- ]* (re) > +- 303 [ ]* 8 files (re) > + > + Test archiving a revision that references a subrepo that is not yet > + cloned: > + > diff --git a/patchwork/tests/series/mercurial-no-cover-letter.mbox > b/patchwork/tests/series/mercurial-no-cover-letter.mbox > new file mode 100644 > index 0000000..9006ab8 > --- /dev/null > +++ b/patchwork/tests/series/mercurial-no-cover-letter.mbox > @@ -0,0 +1,154 @@ > +From andrew at shadura.me Mon Jan 23 08:19:08 2017 > +Return-Path: > +Content-Type: text/plain; charset="us-ascii" > +MIME-Version: 1.0 > +Subject: [PATCH 1 of 2] contrib: fix check-commit to not reject > commits > + from `hg sign` and `hg tag` > +X-Mercurial-Node: 2fb3ae89e4e1992e93826c0e3f04e4f965edb0ab > +X-Mercurial-Series-Index: 1 > +X-Mercurial-Series-Total: 2 > +Message-Id: <2fb3ae89e4e1992e9382.1485159499 at nuevo> > +X-Mercurial-Series-Id: <2fb3ae89e4e1992e9382.1485159499 at nuevo> > +User-Agent: Mercurial-patchbomb/3.9.1 > +Date: Mon, 23 Jan 2017 09:18:19 +0100 > +From: Andrew Shadura > +To: Stephen Finucane > +Content-Transfer-Encoding: 8bit > + > +# HG changeset patch > +# User Augie Fackler > +# Date 1484800475 18000 > +# Wed Jan 18 23:34:35 2017 -0500 > +# Branch stable > +# Node ID 2fb3ae89e4e1992e93826c0e3f04e4f965edb0ab > +# Parent 94af7d0c812fe7d3a5651191685ca43e1a331814 > +contrib: fix check-commit to not reject commits from `hg sign` and > `hg tag` > + > +I'm tired of having a spurious red build every time we do a > +release. Fix it once and for all. > + > +diff --git a/contrib/check-commit b/contrib/check-commit > +--- a/contrib/check-commit > ++++ b/contrib/check-commit > +@@ -59,6 +59,10 @@ def checkcommit(commit, node=None): > + exitcode = 0 > + printed = node is None > + hits = [] > ++ signtag = (afterheader + > ++ r'Added (tag [^ ]+|signature) for changeset [a-f0- > 9]{12}') > ++ if re.search(signtag, commit): > ++ return 0 > + for exp, msg in errors: > + for m in re.finditer(exp, commit): > + end = m.end() > +diff --git a/tests/test-contrib-check-commit.t b/tests/test-contrib- > check-commit.t > +--- a/tests/test-contrib-check-commit.t > ++++ b/tests/test-contrib-check-commit.t > +@@ -30,6 +30,30 @@ A fine patch: > + > EOF > + $ cat patch-with-long-header.diff | $TESTDIR/../contrib/check- > commit > + > ++This would normally be against the rules, but it's okay because > that's > ++what tagging and signing looks like: > ++ > ++ $ cat > creates-a-tag.diff << EOF > ++ > # HG changeset patch > ++ > # User Augie Fackler > ++ > # Date 1484787778 18000 > ++ > # Wed Jan 18 20:02:58 2017 -0500 > ++ > # Branch stable > ++ > # Node ID c177635e4acf52923bc3aa9f72a5b1ad1197b173 > ++ > # Parent a1dd2c0c479e0550040542e392e87bc91262517e > ++ > Added tag 4.1-rc for changeset a1dd2c0c479e > ++ > > ++ > diff --git a/.hgtags b/.hgtags > ++ > --- a/.hgtags > ++ > +++ b/.hgtags > ++ > @@ -150,3 +150,4 @@ 438173c415874f6ac653efc1099dec9c9150e90f > ++ > eab27446995210c334c3d06f1a659e3b9b5da769 4.0 > ++ > b3b1ae98f6a0e14c1e1ba806a6c18e193b6dae5c 4.0.1 > ++ > e69874dc1f4e142746ff3df91e678a09c6fc208c 4.0.2 > ++ > +a1dd2c0c479e0550040542e392e87bc91262517e 4.1-rc > ++ > EOF > ++ $ $TESTDIR/../contrib/check-commit < creates-a-tag.diff > ++ > + A patch with lots of errors: > + > + $ cat > patch-with-long-header.diff << EOF > + > +From andrew at shadura.me Mon Jan 23 08:19:08 2017 > +Return-Path: > +Content-Type: text/plain; charset="us-ascii" > +MIME-Version: 1.0 > +Subject: [PATCH 2 of 2] tests: work around FreeBSD's unzip having > slightly > + different output > +X-Mercurial-Node: b3d2e8cce78c04dbf6f20f0846b6ea8b622983c4 > +X-Mercurial-Series-Index: 2 > +X-Mercurial-Series-Total: 2 > +Message-Id: > +X-Mercurial-Series-Id: <2fb3ae89e4e1992e9382.1485159499 at nuevo> > +In-Reply-To: <2fb3ae89e4e1992e9382.1485159499 at nuevo> > +References: <2fb3ae89e4e1992e9382.1485159499 at nuevo> > +User-Agent: Mercurial-patchbomb/3.9.1 > +Date: Mon, 23 Jan 2017 09:18:20 +0100 > +From: Andrew Shadura > +To: Stephen Finucane > +Content-Transfer-Encoding: 8bit > + > +# HG changeset patch > +# User Augie Fackler > +# Date 1484801021 18000 > +# Wed Jan 18 23:43:41 2017 -0500 > +# Branch stable > +# Node ID b3d2e8cce78c04dbf6f20f0846b6ea8b622983c4 > +# Parent 2fb3ae89e4e1992e93826c0e3f04e4f965edb0ab > +tests: work around FreeBSD's unzip having slightly different output > + > +According to man 1 unzip, this unzip appeared in FreeBSD 8.0. It's > +what comes as /usr/bin/unzip, so we may as well cater to it since > it's > +easy. > + > +diff --git a/tests/test-archive.t b/tests/test-archive.t > +--- a/tests/test-archive.t > ++++ b/tests/test-archive.t > +@@ -327,14 +327,11 @@ old file -- date clamped to 1980 > + $ hg add old > + $ hg commit -m old > + $ hg archive ../old.zip > +- $ unzip -l ../old.zip > ++ $ unzip -l ../old.zip | grep -v -- ----- | egrep -v files$ > + Archive: ../old.zip > + \s*Length.* (re) > +- *-----* (glob) > + *172*80*00:00*old/.hg_archival.txt (glob) > + *0*80*00:00*old/old (glob) > +- *-----* (glob) > +- \s*172\s+2 files (re) > + > + show an error when a provided pattern matches no files > + > +diff --git a/tests/test-subrepo-recursion.t b/tests/test-subrepo- > recursion.t > +--- a/tests/test-subrepo-recursion.t > ++++ b/tests/test-subrepo-recursion.t > +@@ -321,10 +321,9 @@ Test archiving to zip file (unzip output > + > + (unzip date formating is unstable, we do not care about it and glob > it out) > + > +- $ unzip -l ../archive.zip > ++ $ unzip -l ../archive.zip | grep -v -- ----- | egrep -v files$ > + Archive: ../archive.zip > + Length [ ]* Date [ ]* Time [ ]* Name (re) > +- [\- ]* (re) > + 172 [0-9:\- ]* .hg_archival.txt (re) > + 10 [0-9:\- ]* .hgsub (re) > + 45 [0-9:\- ]* .hgsubstate (re) > +@@ -333,8 +332,6 @@ Test archiving to zip file (unzip output > + 45 [0-9:\- ]* foo/.hgsubstate (re) > + 9 [0-9:\- ]* foo/y.txt (re) > + 9 [0-9:\- ]* foo/bar/z.txt (re) > +- [\- ]* (re) > +- 303 [ ]* 8 files (re) > + > + Test archiving a revision that references a subrepo that is not yet > + cloned: > + > diff --git a/patchwork/tests/test_parser.py > b/patchwork/tests/test_parser.py > index 61d8d80..57a3f0d 100644 > --- a/patchwork/tests/test_parser.py > +++ b/patchwork/tests/test_parser.py > @@ -807,6 +807,8 @@ class SubjectTest(TestCase): > self.assertEqual(parse_series_marker(['bar']), (None, None)) > self.assertEqual(parse_series_marker(['bar', '1/2']), (1, > 2)) > self.assertEqual(parse_series_marker(['bar', '0/12']), (0, > 12)) > + self.assertEqual(parse_series_marker(['bar', '1 of 2']), (1, > 2)) > + self.assertEqual(parse_series_marker(['bar', '0 of 12']), > (0, 12)) > > def test_version(self): > self.assertEqual(parse_version('', []), 1) > diff --git a/patchwork/tests/test_series.py > b/patchwork/tests/test_series.py > index d2514c6..8de94a0 100644 > --- a/patchwork/tests/test_series.py > +++ b/patchwork/tests/test_series.py > @@ -438,6 +438,50 @@ class SeriesTotalTest(_BaseTestCase): > self.assertTrue(series.received_all) > > > +class MercurialSeriesTest(_BaseTestCase): > + """Tests for a series without any revisions. > + > + All patches are generated using hg(1) email, provided via the > Patchbomb > + extension. > + """ > + > + def test_cover_letter(self): > + """Series with a cover letter. > + > + Parse a series with a cover letter and two patches. > + > + Input: > + > + - [PATCH 0 of 2] Sample Mercurial patches > + - [PATCH 1 of 2] contrib: fix check-commit to not reject > + commits from `hg sign` and `hg tag` > + - [PATCH 2 of 2] tests: work around FreeBSD's unzip > having > + slightly different output > + """ > + covers, patches, comments = self._parse_mbox( > + 'mercurial-cover-letter.mbox', [1, 2, 0]) > + > + self.assertSerialized(patches, [2]) > + self.assertSerialized(covers, [1]) > + > + def test_no_cover_letter(self): > + """Series without a cover letter. > + > + Parse a series with two patches but no cover letter. > + > + Input: > + > + - [PATCH 1 of 2] contrib: fix check-commit to not reject > + commits from `hg sign` and `hg tag` > + - [PATCH 2 of 2] tests: work around FreeBSD's unzip > having > + slightly different output > + """ > + _, patches, _ = self._parse_mbox( > + 'mercurial-no-cover-letter.mbox', [0, 2, 0]) > + > + self.assertSerialized(patches, [2]) > + > + > class SeriesNameTestCase(TestCase): > > def setUp(self): From stephen at that.guru Sat Apr 29 08:59:59 2017 From: stephen at that.guru (Stephen Finucane) Date: Fri, 28 Apr 2017 23:59:59 +0100 Subject: Patchwork 2.0 ready for takeoff? Message-ID: <1493420399.3875.12.camel@that.guru> You may have noticed a small bump in activity in the last day or two as I've merged a few patches and done some (significant) rework of the docs. This is prep for the 2.0 release, which I hope to make next week. As things stand I'm very happy with the current state of the REST API, series support is working as expected, and the check API looks fit for purpose. Daniel has been testing this locally and the couple of folks that have been using the 'master' code [1] have reported the odd issue but nothing calamitous. I think we're all set. Now, there are a couple of things that I'd like to work on before the release [2] but nothing that should block the release. Assuming I get those out of the door, is there any reason that we can't (a) enable the snazzy new REST API by default, (b) tag a 2.0 release some time next week and (c) release and celebrate? Cheers, Stephen [1] I've seen at the least the following: - patches.linaro.org - patches.opendataplane.org (same as above) - patchwork.linux-mips.org There are probably more. I should really index these. [2] I really want to re-do the deployment guide for Xenial. 2.0 is a pretty significant change and not only is Trusty old news now but I imagine additional steps will be required to deploy it (DRF, at a minimum). In addition, I've found myself unable to filter patches uses the slugified states (i.e. '/patches/?state=new') yesterday using master. I don't know if this is a regression or not, but I really want this for 'git-pw'. Finally, we still don't have anything done for rate limiting in the REST API. Personally, I wouldn't consider this a blocker as it's easy to enable (it's simply not done by default). However, we might want to at least set a somewhat safe default. From stephen at that.guru Sat Apr 29 09:07:35 2017 From: stephen at that.guru (Stephen Finucane) Date: Sat, 29 Apr 2017 00:07:35 +0100 Subject: [PATCH 1/2] REST: Fix versioning In-Reply-To: <20170426235804.31181-1-stephen@that.guru> References: <20170426235804.31181-1-stephen@that.guru> Message-ID: <1493420855.3875.14.camel@that.guru> On Thu, 2017-04-27 at 00:58 +0100, Stephen Finucane wrote: > One of the few remaining warts in the API is versioning. There is > some > basic versioning there, but it doesn't work properly as-is. Fix this > by > correcting the index endpoint ('/') to use Django REST Framework's > 'reverse' function [1], which handles versioning for us [2] and > switching from 'NamespaceVersioning' to 'URLPathVersioning', the > latter > of which does the same thing but in a different way. > > [1] http://www.django-rest-framework.org/api-guide/reverse/#reverse > [2] http://www.django-rest-framework.org/api-guide/versioning/#revers > ing-urls-for-versioned-apis > [3] http://www.django-rest-framework.org/api-guide/versioning/#urlpat > hversioning > > Signed-off-by: Stephen Finucane > Cc: Andy Doan The first one of these is a clear bug fix while the second is a bit contentious but is a clear usability win. I've bit the bullet and just applied both. Stephen