[PATCH 5/7] docs: Rework the development guide
Stephen Finucane
stephen.finucane at intel.com
Thu Feb 4 08:22:33 AEDT 2016
patchwork has undergone a number of changes recently that can make
deployment of a development environment much simpler than it was
previously. Unfortunately, these were not illustrated in the
documentation. Resolve this oversight by rewriting the developer guide
to reflect, for example, the usefulness of the 'parsearchive' script or
the fact that environment variables like 'PW_TEST_DB_USER' do not
always need to be exported.
This includes changes to the provided 'default_projects' fixture: the
existing projects weren't chosen for any particular reason and the
rewritten guide expects a 'patchwork' project to exist.
Signed-off-by: Stephen Finucane <stephen.finucane at intel.com>
---
docs/development.md | 251 ++++++++++++++++++++++++++------
patchwork/fixtures/default_projects.xml | 17 +--
2 files changed, 209 insertions(+), 59 deletions(-)
diff --git a/docs/development.md b/docs/development.md
index 3868adf..12a0edc 100644
--- a/docs/development.md
+++ b/docs/development.md
@@ -1,84 +1,233 @@
-# Developing patchwork
+# Development
-## Using virtualenv
+This document describes the necessary steps to configure patchwork in a
+development environment. If you are interested in deploying patchwork in a
+production environment, please refer to [the deployment guide][doc-deployment]
+instead.
-It's a good idea to use virtualenv to develop Python software. Virtual
-environments are "instances" of your system Python, without any of the
-additional Python packages installed. They are useful to develop and deploy
-patchwork against a "well known" set of dependencies, but they can also be
-used to test patchwork against several versions of Django.
+## Download Patchwork
-1. Install pip, virtualenv (python-pip, python-virtualenv packages)
+The latest version of Patchwork is available with git. To download:
- Because we're going to recompile our dependencies, we'll also need
- development headers. For the MySQL/MariaDB setups these are
- `mariadb-devel` (Fedora), `libmysqlclient-dev` (Debian)
+ $ git clone git://github.com/getpatchwork/patchwork.git
-2. Create a new virtual environement.
+## Install Required Packages
- Inside a virtual env, we'll just install the dependencies needed for
- patchwork and run it from there.
+There are a number of different requirements for developing patchwork:
- $ virtualenv django-1.8
+* Python and libraries
+* A supported RDBMS
- This will create a virtual env called 'django-1.8' in eponymous directory.
+These are detailed below.
-3. Activate a virtual environment
+### Python Requirements
- $ source django-1.8/bin/activate
- (django-1.8)$
+To develop Python-based software you first need Python. Patchwork supports
+both Python 2.7 and Python 3.3+. One of these will be installed by default
+on many installations, though they can also be installed manually using the
+`python` or `python3` packages.
- The shell prompt is preprended with the virtual env name.
+It's a good idea to use [virtual environments][ref-venv] to develop Python
+software. Virtual environments are "instances" of your system Python without
+any of the additional Python packages installed. They are useful to develop and
+possibly deploy patchwork against a "well known" set of dependencies, but they
+can also be used to test patchwork against several versions of Django.
-4. Install the required dependencies
+If you do not have `virtualenv` installed then you should install it now. This
+can be installed using the `python-virtualenv` or `python3-virtualenv`
+packages. Alternatively you can install these using `pip`.
- To ease this task, it's customary to maintain a list of dependencies in a
- text file and install them in one go. One can maintain such a list of
- dependencies per interesting configuration.
+It is also helpful to install [`tox`][ref-tox] which is used for running tests
+in patchwork. This can be installed using the `python-tox` or `python3-tox`
+packages, or via `pip`.
- (django-1.8)$ pip install -r requirements-dev.txt
+### Database Requirements
- You will also need to install a version of Django - we don't install this
- by default to allow development against multiple versions of Django. This
- can be installed like so (assuming Django 1.8):
+If not already installed, you may need to install an RDBMS. You can use either
+MariaDB/MySQL or PostgreSQL for this purpose. You should also install the
+development headers, known as `libmysqlclient-dev` or `libpq-dev` respectively
+on Debian-based Debian-based distros like Ubuntu and `mysql-devel` or
+`postgresql-devel` on RHEL-based distros.
- (django-1.8)$ pip install 'django<1.9,>=1.8'
+**NOTE:** While Django provides support for
+[multiple database backends][ref-django-db], patchwork itself is only tested
+against MySQL/MariaDB and PostgreSQL. Should you wish to use a different
+backend, ensure you validate this first (and perhaps
+[upstream][doc-contributing] any changes you may find necessary).
- Of course, this is a one-time step: once installed in the virtual
- environment there is no need to to install requirements again.
+**NOTE:** You may be tempted to use SQLite to develop patchwork. We'd advise
+against doing this. SQLite supports a subset of the functionality of "full"
+RDBMS like MySQL: for example, case-sensitive matching of Unicode
+[is not supported][ref-sqlite-utf8]. You will find some tests provided by
+patchwork fail and some patches you develop may fail in production due to these
+differences.
-5. Export the `DJANGO_SETTINGS_MODULE` path
+### Example Installations
- If you are using the provided `settings/dev.py` file, you can simply export
- the path to this (in Python module format) like so:
+An example for installing all these packages and the MySQL RDBMS on Ubuntu
+15.04 is given below:
- (django-1.8)$ export DJANGO_SETTINGS_MODULE=patchwork.settings.dev
+ $ sudo apt-get install python python-pip python-dev python-virtualenv \
+ python-tox mysql-server libmysqlclient-dev
- If you do so, you may also need to configure you database configuration.
- See the [Environmental Configuration](#environmental-configuration) section
- below for details on the specific variables to export. For example:
+If you have an existing MariaDB/MySQL installation and have installed `pip`
+already/are using [Python 3.4+][ref-py34-pip] then you can install all
+packages using `pip`:
- (django-1.8)$ export PW_TEST_DB_USER=root
+ $ sudo pip install virtualenv tox
- You can also provide your own `settings.py` file. Simply change the path
- used for `DJANGO_SETTINGS_MODULE` above and omit the `PW_` related steps.
+If you wish to use Python 3 then simply replace 'python' with 'python3' in
+the above command.
-6. Run the development server
+## Configure Virtual Environment
- (django-1.8)$ ./manage.py runserver
+**NOTE:** If you are interested in simply [testing patchwork][doc-testing],
+many of the the below steps are not required. tox will automatically install
+dependencies and use virtual environments when testing.
+
+Once these requirements are installed, you should create and activate a new
+virtual environment. This can be done like so:
+
+ $ virtualenv .venv
+ $ source .venv/bin/activate
+ (.venv)$
+
+**NOTE:** It you installed a Python 3.x-based virtual environment package,
+adjust the executable indicated above as necessary, e.g. `virtualenv-3.4`.
+
+Now install the packages. Patchwork provides three requirements files.
+
+* `requirements-dev.txt`: Packages required to configure a development
+ environment
+* `requirements-prod.txt`: Packages required for deploying patchwork in
+ production
+* `requirements-test.txt`: Packages required to run tests
+
+We're going to install the first of these, which can be done like so:
+
+ (.venv)$ cd patchwork
+ (.venv)$ pip install -r requirements-dev.txt
+
+**NOTE:** Once configured this does not need to be done again *unless* the
+requirements change, e.g. patchwork requires an updated version of Django.
+
+## Initialize the Database
+
+One installed, the database must be configured. We will assume you have root
+access to the database for these steps.
+
+To begin, export your database credentials as follows:
+
+ (.venv)$ db_user=root
+ (.venv)$ db_pass=password
+
+Now, create the database. If this is your first time configuring the database,
+you must create a `patchwork` user (or similar) along with the database
+instance itself. You can do this like so:
+
+ (.venv)$ mysql -u$db_user -p$db_pass << EOF
+ CREATE DATABASE patchwork CHARACTER SET utf8;
+ CREATE USER 'patchwork'@'localhost' IDENTIFIED BY 'password';
+ GRANT ALL PRIVILEGES ON patchwork.* TO 'patchwork'@'localhost';
+ EOF
+
+If you're resetting your database instead, then you must drop the existing
+database and recreate it. You can do this like so:
+
+ (.venv)$ mysql -u$db_user -p$db_pass << EOF
+ DROP DATABASE patchwork;
+ CREATE DATABASE patchwork CHARACTER SET utf8;
+ GRANT ALL PRIVILEGES ON patchwork.* TO 'patchwork'@'localhost';
+ EOF
+
+**NOTE:** The `patchwork` username and `password` password are the defaults
+expected by the provided `dev` settings files. If using something different,
+please export the `PW_TEST_DB_USER` and `PW_TEST_DB_PASS` variables described
+in the [Environment Variables](#environment-variables) section below.
+Alternatively, you can create your own settings file with these variables
+hardcoded and change the value of `DJANGO_SETTINGS_MODULE` as described below.
+
+## Load Initial Data
+
+Before continuing, we need to tell Django where it can find our configuration.
+patchwork provides a default development `settings.py` file for this purpose.
+To use this, export the `DJANGO_SETTINGS_MODULE` environment variable as
+described below:
+
+ (.venv)$ export DJANGO_SETTINGS_MODULE=patchwork.settings.dev
+
+Alternatively you can provide your own `settings.py` file and provide the path
+to that instead.
+
+Once done, we need to create the tables in the database. This can be done using
+the `migrate` command of the `manage.py` executable:
+
+ (.venv)$ ./manage.py migrate
+
+Next, you should load the initial fixtures into patchwork. These initial
+fixtures provide.
+
+* `default_tags.xml`: The tags that patchwork will extract from mails.
+ Examples: `Acked-By`, `Reviewed-By`
+* `default_states.xml`: The states that a patch can be in. Examples:
+ `Accepted`, `Rejected`
+* `default_projects.xml`: A default project that you can then upload patches
+ for
+
+These can be loaded using the `loaddata` command:
+
+ (.venv)$ ./manage.py loaddata patchwork/fixtures/default_tags.xml
+ (.venv)$ ./manage.py loaddata patchwork/fixtures/default_states.xml
+ (.venv)$ ./manage.py loaddata patchwork/fixtures/default_projects.xml
+
+You should also take the opportunity to create a "superuser". You can do this
+using the aptly-named `createsuperuser` command:
+
+ (.venv)$ ./manage.py createsuperuser
+
+Once this is done, it's beneficial to load some real emails into the system.
+This can be done manually, however it's generally much easier to download an
+an archive from a Mailman instance and load these using the `parsearchive.py`
+tool. You can do this like so:
+
+ (.venv)$ mm_user=myusername
+ (.venv)$ mm_pass=mypassword
+ (.venv)$ mm_host=https://lists.ozlabs.org
+ (.venv)$ mm_url=$mm_host/private/patchwork.mbox/patchwork.mbox
+ (.venv)$ curl -F username=$mm_user -F password=$mm_pass -k -O $mm_url
+
+Where `mm_user` and `mm_pass` are the username and password you have registered
+with on the Mailman instance found at `mm_host`.
+
+**NOTE:** We provide instructions for downloading archives from the patchwork
+mailing list, but almost any instance of Mailman will allow downloading of
+archives as seen above; simply change the `pw_url` variable defined. You can
+find more informations about this [here][ref-mman-bulk].
+
+Load these archives into patchwork. Depending on the size of the downloaded
+archives this may take some time:
+
+ (.venv)$ PYTHONPATH=. ./patchwork/bin/parsearchive.py \
+ --list-id=patchwork.ozlabs.org patchwork.mbox
+
+Finally, run the server and browse to the IP address of your board using your
+browser of choice:
+
+ (.venv)$ ./manage.py runserver 0.0.0.0:8000
Once finished, you can kill the server (`Ctrl` + `C`) and exit the virtual
environment:
- (django-1.8)$ deactivate
+ (.venv)$ deactivate
$
Should you wish to re-enter this environment, simply source the `activate`
script again.
-## Environmental Variables
+## Environment Variables
-The following environmental variables are available to configure settings:
+The following environment variables are available to configure settings when
+using the provided `dev` settings file.
<dl>
<dt>PW_TEST_DB_NAME = 'patchwork'</dt>
@@ -90,3 +239,13 @@ The following environmental variables are available to configure settings:
<dt>PW_TEST_DB_TYPE = 'mysql'</dt>
<dd>Type of database to use. Options: 'mysql', 'postgresql'</dd>
</dl>
+
+[doc-contributing]: ../CONTRIBUTING.md
+[doc-deployment]: installation.md
+[doc-testing]: testing.md
+[ref-django-db]: https://docs.djangoproject.com/en/1.8/ref/databases/
+[ref-mman-bulk]: http://blog.behnel.de/posts/indexp118.html
+[ref-py34-pip]: http://legacy.python.org/dev/peps/pep-0453/
+[ref-sqlite-utf8]: https://www.sqlite.org/faq.html#q18
+[ref-tox]: https://tox.readthedocs.org/en/latest/
+[ref-venv]: https://virtualenv.readthedocs.org/en/latest/
diff --git a/patchwork/fixtures/default_projects.xml b/patchwork/fixtures/default_projects.xml
index c67fa56..30d3461 100644
--- a/patchwork/fixtures/default_projects.xml
+++ b/patchwork/fixtures/default_projects.xml
@@ -1,18 +1,9 @@
<?xml version="1.0" encoding="utf-8"?>
<django-objects version="1.0">
-
- <!-- some default projects -->
<object pk="1" model="patchwork.project">
- <field type="CharField" name="linkname">cbe-oss-dev</field>
- <field type="CharField" name="name">Cell Broadband Engine development</field>
- <field type="CharField" name="listid">cbe-oss-dev.ozlabs.org</field>
- <field type="CharField" name="listemail">cbe-oss-dev at ozlabs.org</field>
+ <field type="CharField" name="linkname">patchwork</field>
+ <field type="CharField" name="name">Patchwork</field>
+ <field type="CharField" name="listid">patchwork.ozlabs.org</field>
+ <field type="CharField" name="listemail">patchwork at lists.ozlabs.org</field>
</object>
- <object pk="2" model="patchwork.project">
- <field type="CharField" name="linkname">linuxppc-dev</field>
- <field type="CharField" name="name">Linux PPC development</field>
- <field type="CharField" name="listid">linuxppc-dev.ozlabs.org</field>
- <field type="CharField" name="listemail">linuxppc-dev at ozlabs.org</field>
- </object>
-
</django-objects>
--
2.0.0
More information about the Patchwork
mailing list