Development

If you want to contribute to the development of lts_workflows_sm_scrnaseq, you need to setup a development enviromnent and follow some basic guidelines. In addition, there are workflow tests that provide convenient help for test-based development. This section details some of this information.

Development environment

Setting up a conda development environment

Start by setting up the main conda development environment for python3 packages. This is where the source code will live.

$ conda env create -n lts-workflows-sm-scrnaseq-devel python=3.6

For the workflow tests, you also need a conda environment for python2 packages:

$ conda env create -n py2.7 python=2.7

Hint

By default, the workflow tests are configured to use a python 2 environment named py2.7. Should you name it otherwise, you can tell pytest with the flag --python2-conda:

pytest -v -s --python2-conda mypython2env lts_workflows_sm_scrnaseq/tests/test_workflows_sm_scrnaseq.py

Finally, activate the development environment:

$ source activate lts-workflows-sm-scrnaseq-devel

Hint

Instead of having to remember when to activate a specific environment, maybe also setting environment variables in the process, we recommend using direnv. With direnv, you can add a hidden file .envrc to the source root directory with the following content:

source activate lts-workflows-sm-scrnaseq-devel
export DOCKER_REPO=scilifelablts

Now, every time you cd into the source directory, the correct environment will be activated. Likewise, the environment will be deactivated once you cd out of the source directory.

Checkout the source repository and install

Once the conda environments are setup, checkout the source repository. If you haven’t done so already, fork the lts_workflows_sm_scrnaseq bitbucket repository.

$ git clone git@bitbucket.org:your_name_here/lts-workflows-sm-scrnaseq.git

Change directory to the source code directory and install in development mode:

$ cd lts-workflows-sm-scrnaseq
$ python setup.py develop

Install development requirements

Install the package requirements for development with pip:

$ pip install -r requirements_dev.txt

Note

In principle, conda should work as follows:

$ conda install --file requirements_dev.txt

However, for some reason the docker-py module is named docker on PyPI and docker-py on anaconda, causing the install to fail.

If everything works ok, you should now have a operating development environment! Now you are ready to create and test your new features. Follow the guidelines in Get Started! to submit a pull request.

Test-based development

lts_workflows_sm_scrnaseq includes tests to ensure code and workflow integrity. Some tests are run on the codebase itself (located in the tests directory), whereas others execute pipeline runs (located in the lts_workflows_sm_scrnaseq/tests directory). The workflow tests are shipped with the package. The development environment requirements include the module pytest-ngsfixtures, a pytest which provides pytest fixtures in the shape of next generation sequencing test data sets.

The tests are located in the lts_workflows_sm_scrnaseq/tests directory. You can execute all the tests by executing

$ pytest -v -s -rs lts_workflows_sm_scrnaseq/tests

Note

Since the tests reside in the module directory, they are actually installed as a part of the module upon installation. This means you (theoretically) can run the tests like this:

$ pytest -v -s -rs --pyargs lts_workflows_sm_scrnaseq

Workflow tests

When developing, you are most likely only interested in runnig the workflow tests in lts_workflows_sm_scrnaseq/tests/test_lts_workflows_sm_scrnaseq.py. Running

$ pytest -v -s -rs lts_workflows_sm_scrnaseq/tests/test_lts_workflows_sm_scrnaseq.py

will run a listing of the workflow as well as an entire workflow run on two samples.

You can control some aspects of the tests by applying module-specific pytest options. The conftest plugin lts_workflows_sm_scrnaseq.tests.conftest is a pytest local per-directory plugin that among other things adds command line options to pytest.

Note

For some reason, these options disappear when running the test from the source root directory, or with the –pyargs option.

For instance, the option –ngs-test-command controls whether workflow list or run is executed. The following command will only run the workflow.

$ cd lts_workflows_sm_scrnaseq/tests
$ pytest -v -s -rs test_lts_workflows_sm_scrnaseq.py --ngs-test-command run

A full list of test-specific options is given here:

 single cell rna sequencing options:
--no-slow             don't run slow tests
-H, --hide-workflow-output
                      hide workflow output
-T THREADS, --threads=THREADS
                      number of threads to use
--enable-test-conda   enable test conda setup; automatically install all
                      dependencies in semi-persistent test conda
                      environments
--conda-install-dir=CONDA_INSTALL_DIR
                      set conda install dir
--conda-update        update local conda installation
-2 PYTHON2_CONDA, --python2-conda=PYTHON2_CONDA
                      name of python2 conda environment [default: py2.7]
-C, --use-conda       pass --use-conda flag to snakemake workflows; will
                      install conda environments on a rule by rule basis
--ngs-test-unit=unit [unit ...]
                      test unit - test source code locally or distributed in
                      docker container (choices: local,docker)
--smode={se} [{se} ...], --sequencing-modes={se} [{se} ...]
                      sequencing modes (paired end or single end) to run
--ngs-test-command=command [command ...]
                      test command - run or list

Rerunning workflow tests in pytest tmp directory

By default, pytest outputs test results in numbered subdirectories of directory /tmp/pytest-of-user/. In order to facilitate test-based development, for each workflow test, pytest-ngsfixtures outputs a file command.sh in the resulting test directory. For instance, if the workflow test has been run, this file should exist in a subdirectory run-se-local in one of the numbered subdirectories of /tmp/pytest-of-user. command.sh is an executable that records the environment and command used to run the test. Therefore, if a workflow run has failed, the test can be rerun as follows:

$ cd /tmp/pytest-of-user/pytest-#/run-se-local
$ ./command.sh

Consequently, this is a convenient way of setting up a current test data environment for the workflow. Fix the code and rerun command.sh until the test passes.

Hint

By default, command.sh runs the all target. Edit the file to your liking to target other outputs of interest when developing a new feature.

Testing the docker images

Apart from providing source code and conda packages, lts_workflows_sm_scrnaseq also provides Docker images at https://hub.docker.com/r/scilifelablts/lts-workflows-sm-scrnaseq/.

Important

The docker tests use docker swarm mode to maintain and manage running containers. In order to do test development on the Dockerfile, you first need to initialize docker swarm; see docker create swarm for more information.

When developing the Dockerfile and the resulting images, you can use the make commands. For instance, to build and test changes to the main Dockerfile, run (assuming you’re standing in the source root directory):

$ make docker
$ pytest -v -s -rs lts_workflows_sm_scrnaseq/tests/test_lts_workflows_sm_scrnaseq.py --ngs-test-unit docker

The option ngs-test-unit selects what test unit to run, either local (i.e. local source code) or docker which tests the docker container. The test will fire up a docker stack and provide a docker service named lts_scrnaseq_snakemake, which you can see by running the command

$ docker service ls

Note

Developing with docker swarm can at first be confusing since on some occasions, updates to the local repository are not present in an updated image. This has to do with how docker handles version tags and repositories. It may help to restart services, but it may also be necessary to use a local registry (usually localhost:5000) and push updated images to this registry. See the docker documentation for more information.