Development¶
If you want to contribute to the development of
lts_workflows_sm_scrnaseq
, you need to setup a development
enviromnent and follow some basic guidelines. In addition, there are
workflow tests that provide convenient help for test-based
development. This section details some of this information.
Development environment¶
Setting up a conda development environment¶
Start by setting up the main conda development environment for python3 packages. This is where the source code will live.
$ conda env create -n lts-workflows-sm-scrnaseq-devel python=3.6
You can also use python 3.5.
For the workflow tests, you also need a conda environment for python2 packages:
$ conda env create -n py2.7 python=2.7
Hint
By default, the workflow tests are configured to use a python 2
environment named py2.7. Should you name it otherwise, you can
tell pytest with the flag --python2-conda
:
pytest -v -s --python2-conda mypython2env lts_workflows_sm_scrnaseq/tests/test_workflows_sm_scrnaseq.py
Finally, activate the development environment:
$ source activate lts-workflows-sm-scrnaseq-devel
Hint
Instead of having to remember when to activate a specific environment, maybe also setting environment variables in the process, we recommend using direnv. With direnv, you can add a hidden file .envrc to the source root directory with the following content:
source activate lts-workflows-sm-scrnaseq-devel
export DOCKER_REPO=scilifelablts
Now, every time you cd into the source directory, the correct environment will be activated. Likewise, the environment will be deactivated once you cd out of the source directory.
Checkout the source repository and install¶
Once the conda environments are setup, checkout the source
repository. If you haven’t done so already, fork the
lts_workflows_sm_scrnaseq
bitbucket repository.
$ git clone git@bitbucket.org:your_name_here/lts-workflows-sm-scrnaseq.git
Change directory to the source code directory and install in development mode:
$ cd lts-workflows-sm-scrnaseq
$ python setup.py develop
Install development requirements¶
Install the package requirements for development with pip:
$ pip install -r requirements_dev.txt
Note
In principle, conda should work as follows:
$ conda install --file requirements_dev.txt
However, for some reason the docker-py
module is named
docker on PyPI and
docker-py on
anaconda, causing the install to fail.
If everything works ok, you should now have a operating development environment! Now you are ready to create and test your new features. Follow the guidelines in Get Started! to submit a pull request.
Test-based development¶
lts_workflows_sm_scrnaseq
includes tests to ensure code and
workflow integrity. Some tests are run on the codebase itself (located
in the tests directory), whereas others execute pipeline runs
(located in the lts_workflows_sm_scrnaseq/tests directory). The
workflow tests are shipped with the package. The development
environment requirements include the module pytest-ngsfixtures
,
a pytest which provides pytest fixtures in the shape of
next generation sequencing test data sets.
The tests are located in the lts_workflows_sm_scrnaseq/tests directory. You can execute all the tests by executing
$ pytest -v -s -rs lts_workflows_sm_scrnaseq/tests
Note
Since the tests reside in the module directory, they are actually installed as a part of the module upon installation. This means you (theoretically) can run the tests like this:
$ pytest -v -s -rs --pyargs lts_workflows_sm_scrnaseq
Workflow tests¶
When developing, you are most likely only interested in runnig the workflow tests in lts_workflows_sm_scrnaseq/tests/test_lts_workflows_sm_scrnaseq.py. Running
$ pytest -v -s -rs lts_workflows_sm_scrnaseq/tests/test_lts_workflows_sm_scrnaseq.py
will run a listing of the workflow as well as an entire workflow run on two samples.
You can control some aspects of the tests by applying module-specific
pytest options. The conftest plugin
lts_workflows_sm_scrnaseq.tests.conftest
is a pytest local
per-directory plugin
that among other things adds command line options to pytest.
Note
For some reason, these options disappear when running the test from the source root directory, or with the –pyargs option.
For instance, the option –ngs-test-command controls whether workflow list or run is executed. The following command will only run the workflow.
$ cd lts_workflows_sm_scrnaseq/tests
$ pytest -v -s -rs test_lts_workflows_sm_scrnaseq.py --ngs-test-command run
A full list of test-specific options is given here:
single cell rna sequencing options:
--no-slow don't run slow tests
-H, --hide-workflow-output
hide workflow output
-T THREADS, --threads=THREADS
number of threads to use
--enable-test-conda enable test conda setup; automatically install all
dependencies in semi-persistent test conda
environments
--conda-install-dir=CONDA_INSTALL_DIR
set conda install dir
--conda-update update local conda installation
-2 PYTHON2_CONDA, --python2-conda=PYTHON2_CONDA
name of python2 conda environment [default: py2.7]
-C, --use-conda pass --use-conda flag to snakemake workflows; will
install conda environments on a rule by rule basis
--ngs-test-unit=unit [unit ...]
test unit - test source code locally or distributed in
docker container (choices: local,docker)
--smode={se} [{se} ...], --sequencing-modes={se} [{se} ...]
sequencing modes (paired end or single end) to run
--ngs-test-command=command [command ...]
test command - run or list
Rerunning workflow tests in pytest tmp directory¶
By default, pytest outputs test results in numbered subdirectories of
directory /tmp/pytest-of-user/. In order to facilitate test-based
development, for each workflow test, pytest-ngsfixtures
outputs
a file command.sh in the resulting test directory. For instance, if
the workflow test has been run, this file should exist in a
subdirectory run-se-local in one of the numbered subdirectories of
/tmp/pytest-of-user. command.sh is an executable that records the
environment and command used to run the test. Therefore, if a workflow
run has failed, the test can be rerun as follows:
$ cd /tmp/pytest-of-user/pytest-#/run-se-local
$ ./command.sh
Consequently, this is a convenient way of setting up a current test data environment for the workflow. Fix the code and rerun command.sh until the test passes.
Hint
By default, command.sh runs the all target. Edit the file to your liking to target other outputs of interest when developing a new feature.
Testing the docker images¶
Apart from providing source code and conda packages,
lts_workflows_sm_scrnaseq
also provides Docker images at
https://hub.docker.com/r/scilifelablts/lts-workflows-sm-scrnaseq/.
Important
The docker tests use docker swarm mode to maintain and manage running containers. In order to do test development on the Dockerfile, you first need to initialize docker swarm; see docker create swarm for more information.
When developing the Dockerfile and the resulting images, you can use the make commands. For instance, to build and test changes to the main Dockerfile, run (assuming you’re standing in the source root directory):
$ make docker
$ pytest -v -s -rs lts_workflows_sm_scrnaseq/tests/test_lts_workflows_sm_scrnaseq.py --ngs-test-unit docker
The option ngs-test-unit
selects what test unit to run, either
local (i.e. local source code) or docker which tests the docker
container. The test will fire up a docker stack and provide a docker
service named lts_scrnaseq_snakemake, which you can see by running
the command
$ docker service ls
Note
Developing with docker swarm can at first be confusing since on some occasions, updates to the local repository are not present in an updated image. This has to do with how docker handles version tags and repositories. It may help to restart services, but it may also be necessary to use a local registry (usually localhost:5000) and push updated images to this registry. See the docker documentation for more information.