[PATCH 5/9] docs: Convert development guide
Stephen Finucane
stephen at that.guru
Tue Apr 11 09:27:47 AEST 2017
Signed-off-by: Stephen Finucane <stephen at that.guru>
---
docs/development/contributing.rst | 73 ++++++
docs/development/index.rst | 10 +
docs/development/installation.rst | 488 ++++++++++++++++++++++++++++++++++++++
docs/development/releasing.rst | 60 +++++
docs/development/xmlrpc.rst | 58 +++++
docs/index.rst | 1 +
6 files changed, 690 insertions(+)
create mode 100644 docs/development/contributing.rst
create mode 100644 docs/development/index.rst
create mode 100644 docs/development/installation.rst
create mode 100644 docs/development/releasing.rst
create mode 100644 docs/development/xmlrpc.rst
diff --git a/docs/development/contributing.rst b/docs/development/contributing.rst
new file mode 100644
index 0000000..989bc66
--- /dev/null
+++ b/docs/development/contributing.rst
@@ -0,0 +1,73 @@
+Contributing
+============
+
+Coding Standards
+----------------
+
+**Follow PEP8**. All code is currently PEP8 compliant and it should stay this
+way.
+
+Changes that fix semantic issues will be generally be happily received, but
+please keep such changes separate from functional changes.
+
+`pep8` targets are provided via tox. Refer to the :ref:`testing` section
+below for more information on usage of this tool.
+
+.. _testing:
+
+Testing
+-------
+
+Patchwork includes a `tox`__ script to automate testing. This requires a
+functional database and some Python requirements like `tox`. Refer to the
+:doc:`index` for information on how to configure these.
+
+Assuming these requirements are met, actually testing Patchwork is quite easy
+to do. To start, you can show the default targets like so:
+
+.. code-block:: shell
+
+ $ tox --list
+
+You'll see that this includes a number of targets to run unit tests against the
+different versions of Django supported, along with some other targets related
+to code coverage and code quality. To run one of these, use the `-e` parameter:
+
+.. code-block:: shell
+
+ $ tox -e py27-django18
+
+In the case of the unit tests targets, you can also run specific tests by
+passing the fully qualified test name as an additional argument to this
+command:
+
+.. code-block:: shell
+
+ $ tox -e py27-django18 patchwork.tests.SubjectCleanUpTest
+
+Because Patchwork support multiple versions of Django, it's very important that
+you test against all supported versions. When run without argument, tox will do
+this:
+
+.. code-block:: shell
+
+ $ tox
+
+__ https://tox.readthedocs.io/en/latest/
+
+Submitting Changes
+------------------
+
+All patches should be sent to the `mailing list`__. When doing so, please abide
+by the `QEMU guidelines`__ on contributing or submitting patches. This covers
+both the initial submission and any follow up to the patches. In particular,
+ensure:
+
+* :ref:`All tests pass <testing>`
+
+* Documentation has been updated with new requirements, new script names etc.
+
+* The `CHANGES` file has been updated with any added or removed features
+
+__ https://ozlabs.org/mailman/listinfo/patchwork
+__ http://wiki.qemu.org/Contribute/SubmitAPatch
diff --git a/docs/development/index.rst b/docs/development/index.rst
new file mode 100644
index 0000000..74857f6
--- /dev/null
+++ b/docs/development/index.rst
@@ -0,0 +1,10 @@
+Development Guide
+=================
+
+.. toctree::
+ :maxdepth: 2
+
+ installation
+ contributing
+ releasing
+ xmlrpc
diff --git a/docs/development/installation.rst b/docs/development/installation.rst
new file mode 100644
index 0000000..c8049af
--- /dev/null
+++ b/docs/development/installation.rst
@@ -0,0 +1,488 @@
+Installation
+============
+
+This document describes the necessary steps to configure Patchwork in a
+development environment. If you are interested in deploying Patchwork in a
+production environment, please refer to [the deployment guide][doc-deployment]
+instead.
+
+To begin, you should clone Patchwork:
+
+.. code-block:: shell
+
+ $ git clone git://github.com/getpatchwork/patchwork.git
+
+Docker-Based Installation
+-------------------------
+
+Patchwork provides a Docker-based environment for quick configuration of a
+development environment. This is the preferred installation method. To
+configure Patchwork using Docker:
+
+1. Install **`docker`_** and **`docker-compose`_**.
+
+2. Build the images. This will download over 200MB from the internet:
+
+ .. code-block:: shell
+
+ $ docker-compose build
+
+3. Run `docker-compose up`:
+
+ .. code-block:: shell
+
+ $ docker-compose up
+
+ This will be visible at http://localhost:8000/.
+
+To run a shell within this environment, run:
+
+.. code-block:: shell
+
+ $ docker-compose run --rm web --shell
+
+To run `django-manage` commands, such as `createsuperuser` or `migrate`, run:
+
+.. code-block:: shell
+
+ $ docker-compose run --rm web python manage.py createsuperuser
+
+To run unit tests, excluding Selenium UI interaction tests, using only the
+package versions installed during container initialization, run:
+
+.. code-block:: shell
+
+ $ docker-compose run --rm web --quick-test
+
+To run the same against all supported versions of Django (via tox), run:
+
+.. code-block:: shell
+
+ $ docker-compose run --rm web --quick-tox
+
+To run specific tox targets or tests, pass arguments to the above:
+
+.. code-block:: shell
+
+ $ docker-compose run --rm web --quick-tox -e py27-django17 \
+ patchwork.tests.test_bundles
+
+To run all tests, including Selenium UI interaction tests, using only the
+package versions installed container initialization, run:
+
+.. code-block:: shell
+
+ $ docker-compose run --rm web --test
+
+To run the same against all supported versions of Django (via tox), run:
+
+.. code-block:: shell
+
+ $ docker-compose run --rm web --tox
+
+To run all tests, including Selenium UI interaction tests in non-headless mode,
+run:
+
+.. code-block:: shell
+
+ $ docker run -it --rm -v (pwd):/home/patchwork/patchwork/ \
+ --link patchwork_db_1:db -p 8000:8000 \
+ -v /tmp/.X11-unix:/tmp/.X11-unix \
+ -e PW_TEST_DB_HOST=db -e DISPLAY patchwork_web bash
+
+To reset the database before any of these commands, add `--reset` to the
+command line after `web` and before any other arguments.
+
+Any local edits to the project files made locally are immediately visible to
+the Docker container, and so should be picked up by the Django auto-reloader.
+
+For more information on Docker itself, please refer to the `docker`_ and
+`docker-compose`_ documentation.
+
+.. note::
+
+ If using SELinux, you will need to create a custom SELinux rule to allow the
+ Docker process to access your working directory. Run:
+
+ .. code-block:: shell
+
+ $ chcon -RT svirt_sandbox_file_t $PATCHWORK_DIR
+
+ where `$PATCHWORK_DIR` is the absolute patch to the `patchwork` folder
+ created when you cloned the repo. For more information, see `man docker
+ run`.
+
+.. note::
+
+ If you see an error like the below::
+
+ ERROR: Couldn't connect to the Docker daemon at http+docker://localunixsocket - is it running?
+
+ ensure you have correctly installed Docker, added your user to the `docker`
+ group, and started the daemon, per the `docker documentation <docker>`_.
+
+.. note::
+
+ If you see an error like the below::
+
+ py.error.EACCES: [Permission denied]: open('/home/patchwork/patchwork/.tox/py27-django18/.tox-config1', 'w')
+
+ your host user account is likely using a different UID to the one hardcoded
+ in the Dockerfile. You can confirm this like so:
+
+ .. code-block:: shell
+
+ $ echo $UID
+ 1234
+
+ If this is anything other than `1000`, you must must modify the `Dockerfile`
+ found in `tools/docker` to use your UID and then rebuild:
+
+ .. code-block:: shell
+
+ $ sed -i "/ARG UID=/c\ARG UID=$(echo $UID)" tools/docker/Dockerfile
+ $ docker-compose build web
+
+ This change must be retained in the event that you rebuild the container.
+ You can "hide" the change from Git like so:
+
+ .. code-block:: shell
+
+ $ git update-index --assume-unchanged tools/docker/Dockerfile
+ $ git update-index --skip-worktree tools/docker/Dockerfile
+
+ This should be resolved in a future release when we support docker-compose
+ 2.1 syntax in `docker-compose.yml`.
+
+.. _docker: https://docs.docker.com/compose/install/
+.. _docker-compose: https://docs.docker.com/engine/installation/linux/
+
+Vagrant-Based Installation
+--------------------------
+
+Patchwork provides a Vagrant-based environment as an alternative to Docker.
+Like Docker, Vagrant can be used to quickly configure Patchwork in a
+development environment. To configure Patchwork using Vagrant:
+
+1. Install `**Vagrant** <vagrant>`_
+
+2. Run `vagrant up` from the project directory:
+
+ .. code-block:: shell
+
+ $ cd patchwork
+ $ vagrant up
+
+Once stacked, follow the on-screen instructions. For more information on
+Vagrant itself, refer to the `Vagrant documentation <vagrant>`_.
+
+Manual Installation
+-------------------
+
+Manual installation can be used where use of Docker or Vagrant is not possible
+or desired.
+
+Install Required Packages
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are a number of different requirements for developing Patchwork:
+
+* Python and libraries
+
+* A supported database (RDBMS)
+
+These are detailed below.
+
+Python Requirements
+^^^^^^^^^^^^^^^^^^^
+
+To develop Python-based software you first need Python. Patchwork supports both
+Python 2.7 and Python 3.3+. One of these will be installed by default on many
+installations, though they can also be installed manually using the `python` or
+`python3` packages.
+
+It's a good idea to use `virtual environments`__ to develop Python software.
+Virtual environments are "instances" of your system Python without any of the
+additional Python packages installed. They are useful to develop and possibly
+deploy Patchwork against a "well known" set of dependencies, but they can also
+be used to test Patchwork against several versions of Django.
+
+If you do not have `virtualenv` installed then you should install it now. This
+can be installed using the `python-virtualenv` or `python3-virtualenv`
+packages. Alternatively you can install these using `pip`.
+
+It is also helpful to install [`tox`][ref-tox] which is used for running tests
+in Patchwork. This can be installed using the `python-tox` or `python3-tox`
+packages, or via `pip`.
+
+__ https://virtualenv.readthedocs.io/en/latest/
+
+Database Requirements
+^^^^^^^^^^^^^^^^^^^^^
+
+If not already installed, you may need to install an RDBMS. You can use either
+MariaDB/MySQL or PostgreSQL for this purpose. You should also install the
+development headers, known as `libmysqlclient-dev` or `libpq-dev` respectively
+on Debian-based Debian-based distros like Ubuntu and `mysql-devel` or
+`postgresql-devel` on RHEL-based distros.
+
+.. note::
+
+ While Django provides support for `multiple database backends`__, Patchwork
+ itself is only tested against MySQL/MariaDB and PostgreSQL. Should you wish
+ to use a different backend, ensure you validate this first (and perhaps
+ upstream any changes you may find necessary).
+
+.. note::
+
+ You may be tempted to use SQLite to develop Patchwork. We'd advise against
+ doing this. SQLite supports a subset of the functionality of "full" RDBMS
+ like MySQL: for example, case-sensitive matching of Unicode `is not
+ supported`__. You will find some tests provided by Patchwork fail and some
+ patches you develop may fail in production due to these differences.
+
+__ https://docs.djangoproject.com/en/1.8/ref/databases/
+__ https://www.sqlite.org/faq.html#q18
+
+Example Installation
+^^^^^^^^^^^^^^^^^^^^
+
+An example for installing all these packages and the MySQL RDBMS on Ubuntu
+15.04 is given below:
+
+.. code-block:: shell
+
+ $ sudo apt-get install python python-pip python-dev python-virtualenv \
+ python-tox mysql-server libmysqlclient-dev
+
+If you have an existing MariaDB/MySQL installation and have installed `pip`
+already/are using Python 3.4+ then you can install all packages using `pip`:
+
+.. code-block:: shell
+
+ $ sudo pip install virtualenv tox
+
+If you wish to use Python 3 then simply replace `python` with `python3` in the
+above command.
+
+Configure Virtual Environment
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+.. note::
+
+ If you are interested in simply :ref:`testing Patchwork <testing>`, many of
+ the below steps are not required. tox will automatically install
+ dependencies and use virtual environments when testing.
+
+Once these requirements are installed, you should create and activate a new
+virtual environment. This can be done like so:
+
+.. code-block:: shell
+
+ $ virtualenv .venv
+ $ source .venv/bin/activate
+ (.venv)$
+
+.. note::
+
+ If you installed a Python 3.x-based virtual environment package, adjust the
+ executable indicated above as necessary, e.g. `virtualenv-3.4`.
+
+Now install the packages. Patchwork provides three requirements files.
+
+`requirements-dev.txt`
+
+ Packages required to configure a development environment
+
+`requirements-prod.txt`
+
+ Packages required for deploying Patchwork in production
+
+`requirements-test.txt`
+
+ Packages required to run tests
+
+We're going to install the first of these, which can be done like so:
+
+.. code-block:: shell
+
+ (.venv)$ cd patchwork
+ (.venv)$ pip install -r requirements-dev.txt
+
+.. note::
+
+ Once configured this does not need to be done again *unless* the
+ requirements change, e.g. Patchwork requires an updated version of Django.
+
+Initialize the Database
+~~~~~~~~~~~~~~~~~~~~~~~
+
+One installed, the database must be configured. We will assume you have root
+access to the database for these steps.
+
+To begin, export your database credentials as follows:
+
+.. code-block:: shell
+
+ (.venv)$ db_user=root
+ (.venv)$ db_pass=password
+
+Now, create the database. If this is your first time configuring the database,
+you must create a `patchwork` user (or similar) along with the database
+instance itself. The commands below will do this, dropping existing databases
+if necessary:
+
+.. code-block:: shell
+
+ (.venv)$ mysql -u$db_user -p$db_pass << EOF
+ DROP DATABASE IF EXISTS patchwork;
+ CREATE DATABASE patchwork CHARACTER SET utf8;
+ GRANT ALL PRIVILEGES ON patchwork.* TO 'patchwork'@'localhost'
+ IDENTIFIED BY 'password';
+ EOF
+
+.. note::
+
+ The `patchwork` username and `password` password are the defaults expected
+ by the provided `dev` settings files. If using something different,
+ export the `PW_TEST_DB_USER` and `PW_TEST_DB_PASS` variables described in
+ the :ref:`Environment Variables <dev-envvar>` section below. Alternatively,
+ you can create your own settings file with these variables hardcoded and
+ change the value of `DJANGO_SETTINGS_MODULE` as described below.
+
+Load Initial Data
+~~~~~~~~~~~~~~~~~
+
+Before continuing, we need to tell Django where it can find our configuration.
+Patchwork provides a default development `settings.py` file for this purpose.
+To use this, export the `DJANGO_SETTINGS_MODULE` environment variable as
+described below:
+
+.. code-block:: shell
+
+ (.venv)$ export DJANGO_SETTINGS_MODULE=patchwork.settings.dev
+
+Alternatively you can provide your own `settings.py` file and provide the path
+to that instead.
+
+Once done, we need to create the tables in the database. This can be done using
+the `migrate` command of the `manage.py` executable:
+
+.. code-block:: shell
+
+ (.venv)$ ./manage.py migrate
+
+Next, you should load the initial fixtures into Patchwork. These initial
+fixtures provide.
+
+`default_tags.xml`
+
+ The tags that Patchwork will extract from mails. For example: `Acked-By`,
+ `Reviewed-By`
+
+`default_states.xml`
+
+ The states that a patch can be in. For example: `Accepted`, `Rejected`
+
+`default_projects.xml`
+
+ A default project that you can then upload patches for
+
+These can be loaded using the `loaddata` command:
+
+.. code-block:: shell
+
+ (.venv)$ ./manage.py loaddata default_tags default_states default_projects
+
+You should also take the opportunity to create a "superuser". You can do this
+using the aptly-named `createsuperuser` command:
+
+.. code-block:: shell
+
+ (.venv)$ ./manage.py createsuperuser
+
+Import Mailing List Archives
+----------------------------
+
+Regardless of your installation method of choice, you will probably want to
+load some real emails into the system. This can be done manually, however it's
+generally much easier to download an archive from a Mailman instance and load
+these using the `parsearchive` command. You can do this like so:
+
+.. code-block:: shell
+
+ (.venv)$ mm_user=<myusername>
+ (.venv)$ mm_pass=<mypassword>
+ (.venv)$ mm_host=https://lists.ozlabs.org
+ (.venv)$ mm_url=$mm_host/private/patchwork.mbox/patchwork.mbox
+ (.venv)$ curl -F username=$mm_user -F password=$mm_pass -k -O $mm_url
+
+where `mm_user` and `mm_pass` are the username and password you have registered
+with on the Mailman instance found at `mm_host`.
+
+.. note::
+
+ We provide instructions for downloading archives from the Patchwork mailing
+ list, but almost any instance of Mailman will allow downloading of archives
+ as seen above; simply change the `pw_url` variable defined. You can find
+ more informations about this `here`__.
+
+Load these archives into Patchwork. Depending on the size of the downloaded
+archives this may take some time:
+
+.. code-block:: shell
+
+ (.venv)$ ./manage.py parsearchive --list-id=patchwork.ozlabs.org \
+ patchwork.mbox
+
+Finally, run the server and browse to the IP address of your board using your
+browser of choice:
+
+.. code-block:: shell
+
+ (.venv)$ ./manage.py runserver 0.0.0.0:8000
+
+Once finished, you can kill the server (`Ctrl` + `C`) and exit the virtual
+environment:
+
+.. code-block:: shell
+
+ (.venv)$ deactivate
+ $
+
+Should you wish to re-enter this environment, simply source the `activate`
+script again.
+
+__ http://blog.behnel.de/posts/indexp118.html
+
+Django Debug Toolbar
+--------------------
+
+Patchwork installs and enables the 'Django Debug Toolbar' by default. However,
+by default this is only displayed if you are developing on localhost. If
+developing on a different machine, you should configure an SSH tunnel such
+that, for example, `localhost:8000` points to `[DEV_MACHINE_IP]:8000`.
+
+.. _dev-envvar:
+
+Environment Variables
+---------------------
+
+The following environment variables are available to configure settings when
+using the provided `dev` settings file.
+
+`PW_TEST_DB_NAME=patchwork`
+
+ Name of the database
+
+`PW_TEST_DB_USER=patchwork`
+
+ Username to access the database with
+
+`PW_TEST_DB_PASS=password`
+
+ Password to access the database with<
+
+`PW_TEST_DB_TYPE=mysql`
+
+ Type of database to use. Options: 'mysql', 'postgres'
diff --git a/docs/development/releasing.rst b/docs/development/releasing.rst
new file mode 100644
index 0000000..3f3a8ee
--- /dev/null
+++ b/docs/development/releasing.rst
@@ -0,0 +1,60 @@
+Release Process
+===============
+
+Versioning
+----------
+
+Since version 1.0, Patchwork has implemented a version of `Semantic
+Versioning`__ . To summarise, releases take the format **MAJOR.MINOR.PATCH**
+(or just **MAJOR.MINOR**). We increment:
+
+1. **MAJOR** version when we make major UI changes or functionality updates
+
+2. **MINOR** version when we make minor UI changes or functionality updates
+
+3. **PATCH** version when we make make bug fixes, dependency updates etc.
+
+In Git, each release will have a tag indicating the version number. In
+addition, each release series has it's own branch called `stable/MAJOR.MINOR`
+to allow backporting of bugfixes or security updates to older versions.
+
+__ http://semver.org/
+
+Release Cycle
+-------------
+
+There is no cadence for releases: they are made available as necessary.
+
+Supported Versions
+------------------
+
+Typically all development should occur on `master`. While we will backport
+bugfixes and security updates, we will not backport any new features. This is
+to ensure stability for users of these versions of Patchwork.
+
+Release Checklist
+-----------------
+
+* Documentation has been updated with latest release version
+
+* Documentation references latest supported version of Django
+
+Backporting
+-----------
+
+We will occasionally backport bugfixes and security updates. When backporting a
+patch, said patch should first be merged into `master`. Once merged, you can
+backport by cherry-picking commits, using the `-x` flag for posterity:
+
+.. code-block:: shell
+
+ $ git cherry-pick <master_commit> -x
+
+There may be some conflicts; resolve these, uncommenting the `Conflicts` line
+when commiting::
+
+ Conflicts
+ patchwork/bin/pwclient
+
+When enough patches have been backported, you should release a new `PATCH`
+release.
diff --git a/docs/development/xmlrpc.rst b/docs/development/xmlrpc.rst
new file mode 100644
index 0000000..9b98767
--- /dev/null
+++ b/docs/development/xmlrpc.rst
@@ -0,0 +1,58 @@
+The XML-RPC API
+===============
+
+Patchwork provides an XML-RPC API. This API can be used to be used to retrieve
+and modify information about patches, projects and more.
+
+This guide covers development information for the Patchwork XML-RPC API. For
+information on using the REST API, refer to `rest`. For information on general
+usage of the XML-RPC API, refer to :doc:`../usage/index`.
+
+.. note::
+
+ The XML-RPC API can be enabled/disabled by the administrator: it may not be
+ available in every instance.
+
+Documentation
+-------------
+
+Patchwork provides automatically generated documentation for the XML-RPC API.
+You can find this at the following URL:
+
+ http://patchwork.example.com/xmlrpc/
+
+where `patchwork.example.com` refers to the URL of your Patchwork instance.
+
+.. note::
+
+ Automatic documentation generation for the Patchwork API was introduced in
+ Patchwork v1.1. Prior versions of Patchwork do not offer this functionality.
+
+Interacting with the API
+------------------------
+
+The Patchwork XML-RPC API provides a number of "methods". Some methods require
+authentication (via HTTP Basic Auth) while others do not. Authentication uses
+your Patchwork account and the on-server documentation will indicate where it
+is necessary. We will only cover the unauthenticated method here for brevity -
+consult the `xmlrpclib`__ documentation for more detailed examples:
+
+To interact with the Patchwork XML-RPC API, a XML-RPC library should be used.
+Python provides such a library - `xmlrpclib` - in its standard library. For
+example, to get the version of the XML-RPC API for a Patchwork instance hosted
+at `patchwork.example.com`, run:
+
+.. code-block:: pycon
+
+ $ python
+ >>> import xmlrpclib # or 'xmlrpc.client' for Python 3
+ >>> rpc = xmlrpclib.ServerProxy('http://patchwork.example.com/xmlrpc/')
+ >>> rpc.pw_rpc_version()
+ 1.1
+
+Once connected, the `rpc` object will be populated with a list of available
+functions (or procedures, in RPC terminology). In the above example, we used
+the `pw_rpc_version` method, however, it should be possible to use all the
+methods listed in the server documentation.
+
+__ https://docs.python.org/2/library/xmlrpclib.html
diff --git a/docs/index.rst b/docs/index.rst
index 7989b4e..f2e7fcd 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -6,6 +6,7 @@ Patchwork
usage/index
deployment/index
+ development/index
Patchwork is a patch tracking system for community-based projects. It is
intended to make the patch management process easier for both the project's
--
2.9.3
More information about the Patchwork
mailing list