[PATCH 00/13] Start generating API examples from tests

[Stephen Finucane]

On Sun, 2018-11-11 at 12:00 +1100, Daniel Axtens wrote:
> Hi Stephen,
> 
> > As part of the OpenAPI work, I realized I wanted to be able to show
> > actual API requests and responses for the documentation. We could
> > generate these manually but doing so would be tedious and would need to
> > be redone every time the API changed. There is a better way: generate
> > these things from the tests. We know that the requests and responses
> > must be correct because we're actually testing against them. It also
> > provides an opportunity to examine our test coverage and fill in gaps,
> > of which there are a few. All of these samples are currently unused but
> > they will be used once we merge in the OpenAPI work.
> > 
> > Note that some of the changes are bigger than we'd like. That's because
> > some tests were actually testing multiple things. This reduced the LoC
> > necessary but meant we didn't have a single request/response to save.
> 
> (Sorry, I would test this myself, but what with me being in an airport
> and this potentially interacting with other series, I figure it's best
> to just ask this time...)
> 
> What's the impact on the time to run tests with this? I notice it splits
> up a lot of tests; does that make the already painfully slow test suite
> any slower?

There is an impact but it's very slight.

Before:

   $ docker-compose run --rm web tox -e py27-django111 patchwork.tests.api
   ............................................
   ----------------------------------------------------------------------
   Ran 44 tests in 2.604s

   OK

After:

   $ docker-compose run --rm web tox -e py27-django111 patchwork.tests.api
   .................................................................................................
   ----------------------------------------------------------------------
   Ran 97 tests in 3.980s

   OK

Personally, I think this is acceptable as the samples are helpful and
the tests probably should have been written in this more modular
fashion the first day. As a general rule of thumb too, the REST API
tests are very fast, once you exclude the time taken to set up the
testing database (which is constant and slooooow). The slowest tests
are the ones for views, which require a lot more of Django's machinery.
We should probably try to optimize those.

Stephen

> Regards,
> Daniel
> 
> > Stephen Finucane (13):
> >   tests: Add 'store_samples' decorator
> >   tests: Add 'store_samples' decorator to 'test_bundle'
> >   tests: Add 'store_samples' decorator to 'test_project'
> >   tests: Add 'store_samples' decorator to 'test_people'
> >   tests: Add 'store_samples' decorator to 'test_user'
> >   tests: Add 'store_samples' decorator to 'test_patch'
> >   tests: Add 'store_samples' decorator to 'test_cover'
> >   tests: Add 'store_samples' decorator to 'test_series'
> >   tests: Add 'store_samples' decorator to 'test_comment'
> >   tests: Add 'store_samples' decorator to 'test_check'
> >   tests: Add tests for '/events' resource
> >   tests: Add 'store_samples' decorator to 'test_event'
> >   signals: Fix 'series-completed' event
> > 
> >  .gitignore                          |   1 +
> >  patchwork/signals.py                |  21 +++-
> >  patchwork/tests/api/test_bundle.py  |  76 +++++++++++--
> >  patchwork/tests/api/test_check.py   |  54 ++++++---
> >  patchwork/tests/api/test_comment.py |  57 ++++++----
> >  patchwork/tests/api/test_cover.py   |  51 ++++++---
> >  patchwork/tests/api/test_event.py   | 165 ++++++++++++++++++++++++++++
> >  patchwork/tests/api/test_patch.py   | 145 ++++++++++++++++--------
> >  patchwork/tests/api/test_person.py  |  37 +++++--
> >  patchwork/tests/api/test_project.py | 110 ++++++++++++++-----
> >  patchwork/tests/api/test_series.py  |  65 +++++++----
> >  patchwork/tests/api/test_user.py    |  65 +++++++++--
> >  patchwork/tests/api/utils.py        |  93 ++++++++++++++++
> >  patchwork/tests/test_events.py      |  33 +++++-
> >  patchwork/tests/utils.py            |  10 ++
> >  15 files changed, 808 insertions(+), 175 deletions(-)
> >  create mode 100644 patchwork/tests/api/test_event.py
> >  create mode 100644 patchwork/tests/api/utils.py
> > 
> > -- 
> > 2.17.2
> > 
> > _______________________________________________
> > Patchwork mailing list
> > Patchwork at lists.ozlabs.org
> > https://lists.ozlabs.org/listinfo/patchwork