[PATCH v3 5/6] Allow use of Docker for development
Stephen Finucane
stephenfinucane at hotmail.com
Sun Aug 14 09:31:09 AEST 2016
On 09 Aug 14:27, Daniel Axtens wrote:
> This makes it possible to use Docker and docker-compose for development
> as an alternative to Vagrant.
>
> I quite liked vagrant a couple of years ago, but currently:
>
> * Trying to install VirtualBox on Ubuntu wants me to disable
> Secure Boot, and I don't want to do that.
>
> * Trying to use the libvirt plugin for vagrant requires I pick
> from a very small set of possible images, and requires that I
> install the upstream vagrant rather than the vagrant shipped
> with Ubuntu 16.04
>
> * I find docker containers faster to work with and more transparent.
>
> So I've done the work to make docker work for Patchwork development.
> This doesn't break or in any way interfere with using Vagrant, it just
> provides an alternative.
>
> It includes support for headless selenium tests using Chromium.
Excellent work. I've a couple of minor requests below, then we're good
to go I think.
> Signed-off-by: Daniel Axtens <dja at axtens.net>
>
> ---
>
> v2: chrome -> chromium
> v3: The following, thanks largely to Andrew:
> - various minor tweaks
> - wait longer for db to come up
> - grant patchwork user permissions to test db
> ---
> .dockerignore | 3 ++
> README.md | 38 +++++++++++++++++++++-
> docker-compose.yml | 23 +++++++++++++
> docker/Dockerfile | 46 ++++++++++++++++++++++++++
> docker/bashrc | 5 +++
> docker/db/.dockerignore | 1 +
> docker/db/.gitignore | 1 +
> docker/db/Dockerfile | 10 ++++++
> docker/entrypoint.sh | 86 +++++++++++++++++++++++++++++++++++++++++++++++++
> 9 files changed, 212 insertions(+), 1 deletion(-)
> create mode 100644 .dockerignore
> create mode 100644 docker-compose.yml
> create mode 100644 docker/Dockerfile
> create mode 100644 docker/bashrc
> create mode 100644 docker/db/.dockerignore
> create mode 100644 docker/db/.gitignore
> create mode 100644 docker/db/Dockerfile
> create mode 100755 docker/entrypoint.sh
>
> diff --git a/.dockerignore b/.dockerignore
> new file mode 100644
> index 000000000000..24473bbe1237
> --- /dev/null
> +++ b/.dockerignore
> @@ -0,0 +1,3 @@
> +.vagrant
> +docker/db
> +
> diff --git a/README.md b/README.md
> index c83e7d31f1db..3c25f04d7c8d 100644
> --- a/README.md
> +++ b/README.md
> @@ -20,7 +20,7 @@ subsystems of the Linux kernel. Although Patchwork has been developed with the
> kernel workflow in mind, the aim is to be flexible enough to suit the majority
> of community projects.
>
> -# Development Installation
> +# Development Installation using Vagrant
>
> 1. Install [**Vagrant**][ref-vagrant]
> 2. Clone this repo:
> @@ -32,6 +32,42 @@ of community projects.
> $ cd patchwork
> $ vagrant up
>
> +# Development Installation using Docker
> +
> +1. Install Docker and docker-compose.
> +2. Clone this repo, as with vagrant.
> +3. Build the images. This will download over 200MB from the internet:
> +
> + $ docker-compose build
> +
> +4. Run as follows:
> +
> + * Regular server:
> +
> + $ docker-compose up
Can you include only up to here in the README and move the rest to
'docs/development.md'? I'd like to keep the README short and sweet. A
reference to the docs (perhaps the readthedocs hosted version) might be
helpful.
http://patchwork.readthedocs.io/en/latest/development/
> +
> + This will be visible on http://localhost:8000/.
> +
> + * Shell:
> +
> + $ docker-compose run --rm web --shell
> +
> + * Quick test (not including selenium UI interaction tests):
> +
> + $ docker-compose run --rm web --quick-test
> +
> + * Full tests, including selenium, run headlessly:
> +
> + $ docker-compose run --rm web --test
> +
> + * To reset the database before beginning, add `--reset` to the command line after `web` and before any other arguments.
> +
> + * If you want to run non-headless tests, you'll need something like this ugly hack:
> +
> + $ docker run -it --rm -v (pwd):/home/patchwork/patchwork/ --link patchwork_db_1:db -p 8000:8000 -v /tmp/.X11-unix:/tmp/.X11-unix -e PW_TEST_DB_HOST=db -e DISPLAY patchwork_web bash
> +
> +With both vagrant and docker, any edits to the project files made locally are immediately visible to the VM/container, and so should be picked up by the Django auto-reloader.
Wrap text and commands (with slashes) at 80 characters.
> +
> # Talks and Presentations
>
> * [**A New Patchwork**][pdf-fosdem] - FOSDEM 2016
> diff --git a/docker-compose.yml b/docker-compose.yml
> new file mode 100644
> index 000000000000..881a72ad6de1
> --- /dev/null
> +++ b/docker-compose.yml
> @@ -0,0 +1,23 @@
> +# the version of docker-compose shipped in ubuntu 16.04 is
> +# 1.5.2, which doesn't support version 2 syntax. Yay!
> +# also, v1 doesn't support explicit build args, so if you're not
> +# uid 1000, you will either need to manually hack the Dockerfile
> +# or upgrade to v2 and use the build-arg to override it.
> +
> +db:
> + build: docker/db
> + volumes:
> + - ./docker/db/data:/var/lib/mysql
> +web:
> + build: .
> + dockerfile: ./docker/Dockerfile
> + command: python3 manage.py runserver 0.0.0.0:8000
> + volumes:
> + - .:/home/patchwork/patchwork/
> + ports:
> + - "8000:8000"
> + links:
> + - db
> + environment:
> + - PW_TEST_DB_HOST=db
> + - PW_TEST_DB_PORT=3306
> diff --git a/docker/Dockerfile b/docker/Dockerfile
> new file mode 100644
> index 000000000000..ea2a8fc5ab77
> --- /dev/null
> +++ b/docker/Dockerfile
Could you move the 'docker' folder to 'tools/docker', please? Perhaps
rename 'tools/install' to 'tools/vagrant' while at it? I think this
folder is a natural fit for these scripts.
> @@ -0,0 +1,46 @@
> +FROM ubuntu
> +
> +ARG UID=1000
> +
> +ENV PROJECT_HOME /home/patchwork/patchwork
> +
> +ENV db_user root
> +ENV db_pass password
> +
> +ENV DJANGO_SETTINGS_MODULE patchwork.settings.dev
> +ENV DEBIAN_FRONTEND noninteractive
> +ENV PYTHONUNBUFFERED 1
> +
> +# System
> +RUN apt-get update -qq && \
> + apt-get install -y --no-install-recommends \
> + python-dev python-pip python-setuptools python-wheel \
> + python3-dev python3-pip python3-setuptools python3-wheel \
> + libmysqlclient-dev mysql-client curl unzip xvfb chromium-chromedriver \
> + chromium-browser build-essential && \
> + ln -s /usr/lib/chromium-browser/chromedriver /usr/bin/
> +
> +# User
> +RUN useradd --uid=$UID --create-home patchwork
> +
> +# Python requirements.
> +# If you update requirements, you should rebuild the container.
> +# entrypoint.sh will prompt you to do this.
> +COPY requirements-*.txt /tmp/
> +RUN pip3 install virtualenv tox && \
> + pip3 install -r /tmp/requirements-dev.txt
> +# we deliberately leave the requirements files in tmp so we can
> +# ping the user in entrypoint.sh if the change them!
> +
> +COPY docker/bashrc /tmp/bashrc
> +
> +# we put the code in ~/patchwork rather than ~ so that we
> +# can put in these bashrc snippets without dirtying the
> +# working directory
> +RUN cat /tmp/bashrc >> /home/patchwork/.bashrc
> +
> +COPY docker/entrypoint.sh /usr/local/bin/entrypoint.sh
> +
> +ENTRYPOINT ["/usr/local/bin/entrypoint.sh"]
> +USER patchwork
> +WORKDIR /home/patchwork/patchwork
> diff --git a/docker/bashrc b/docker/bashrc
> new file mode 100644
> index 000000000000..eb2ed7dddb85
> --- /dev/null
> +++ b/docker/bashrc
> @@ -0,0 +1,5 @@
> +# This snippet is appended to ~/.bashrc when the container is created
> +
> +alias runserver='python3 $PROJECT_HOME/manage.py runserver 0.0.0.0:8000'
> +alias createsu='python3 $PROJECT_HOME/manage.py createsuperuser'
> +
> diff --git a/docker/db/.dockerignore b/docker/db/.dockerignore
> new file mode 100644
> index 000000000000..1269488f7fb1
> --- /dev/null
> +++ b/docker/db/.dockerignore
> @@ -0,0 +1 @@
> +data
> diff --git a/docker/db/.gitignore b/docker/db/.gitignore
> new file mode 100644
> index 000000000000..60baa9cb833f
> --- /dev/null
> +++ b/docker/db/.gitignore
> @@ -0,0 +1 @@
> +data/*
> diff --git a/docker/db/Dockerfile b/docker/db/Dockerfile
> new file mode 100644
> index 000000000000..5df9b5acb486
> --- /dev/null
> +++ b/docker/db/Dockerfile
> @@ -0,0 +1,10 @@
> +FROM mysql:5.7
> +
> +ENV MYSQL_ROOT_PASSWORD password
> +ENV MYSQL_USER patchwork
> +ENV MYSQL_PASSWORD password
> +
> +# We don't want to use the MYSQL_DATABASE env here because
> +# we want to be able to create the database with UTF-8 explictly.
> +# We also can't load in the data because it's in XML, yay.
> +
> diff --git a/docker/entrypoint.sh b/docker/entrypoint.sh
> new file mode 100755
> index 000000000000..0966cf2ec432
> --- /dev/null
> +++ b/docker/entrypoint.sh
> @@ -0,0 +1,86 @@
> +#!/bin/bash
> +set -euo pipefail
> +
> +# functions
> +test_db_connection() {
> + mysqladmin -h $PW_TEST_DB_HOST -u patchwork --password=password ping > /dev/null 2> /dev/null
> +}
> +
> +reset_data() {
> + mysql -u$db_user -p$db_pass -h $PW_TEST_DB_HOST << EOF
> +DROP DATABASE IF EXISTS patchwork;
> +CREATE DATABASE patchwork CHARACTER SET utf8;
> +GRANT ALL ON patchwork.* TO 'patchwork' IDENTIFIED BY 'password';
> +GRANT ALL PRIVILEGES ON test_patchwork.* TO 'patchwork'@'%';
> +FLUSH PRIVILEGES;
> +EOF
> +
> + # load initial data
> + python3 $PROJECT_HOME/manage.py migrate #> /dev/null
> + python3 $PROJECT_HOME/manage.py loaddata default_tags #> /dev/null
> + python3 $PROJECT_HOME/manage.py loaddata default_states #> /dev/null
> + python3 $PROJECT_HOME/manage.py loaddata default_projects #> /dev/null
> +}
> +
> +
> +# The script begins!
> +
> +# check if patchwork is mounted. Checking if we exist is a
> +# very good start!
> +if [ ! -f ~patchwork/patchwork/docker/entrypoint.sh ]; then
> + echo "The patchwork directory doesn't seem to be mounted!"
> + echo "Are you using docker-compose?"
> + echo "If not, you need -v PATH_TO_PATCHWORK:/home/patchwork/patchwork"
> + exit 1
> +fi
> +
> +# check if we need to rebuild because requirements changed
> +for x in /tmp/requirements-*.txt; do
> + if ! cmp $x ~/patchwork/$(basename $x); then
> + echo "A requirements file has changed."
> + echo "Please rebuild the patchwork image:"
> + echo " docker-compose build web"
> + exit 1
Spaces, not tabs, please: I'm no Richard Hendriks [1] :)
[1] https://www.youtube.com/watch?v=SsoOG6ZeyUI
> + fi
> +done
> +
> +# check if mysql is connected
> +if ! test_db_connection; then
> + echo "MySQL seems not to be connected, or the patchwork user is broken"
> + echo "MySQL may still be starting. Waiting 5 seconds."
> + sleep 5
> + if ! test_db_connection; then
> + echo "Still cannot connect to MySQL."
> + echo "Maybe you are starting the db for the first time. Waiting 15 seconds."
> + sleep 15
> + if ! test_db_connection; then
> + echo "Still cannot connect to MySQL. Giving up."
> + echo "Are you using docker-compose? If not, have you set up the link correctly?"
> + exit 1
> + fi
> + fi
> +fi
> +
> +# rebuild mysql db
> +# do this on --reset or if the db doesn't exist
> +if [[ "$1" == "--reset" ]]; then
> + shift
> + reset_data
> +elif ! ( echo ';' | mysql -h db -u patchwork -ppassword patchwork 2> /dev/null ); then
> + reset_data
> +fi
> +
> +if [ $# -eq 0 ]; then
> + # we probably ran with --reset and nothing else
> + # just exit cleanly
> + exit 0
> +elif [ "$1" == "--shell" ]; then
> + exec bash
> +elif [ "$1" == "--quick-test" ]; then
> + export PW_SKIP_BROWSER_TESTS=yes
> + python3 manage.py test
> +elif [ "$1" == "--test" ]; then
> + xvfb-run --server-args='-screen 0, 1024x768x16' python3 manage.py test
> +else # run whatever CMD is set to
> + $@
> +fi
> --
> 2.7.4
>
More information about the Patchwork
mailing list