Conditional Commands in tox? (tox, travis-ci, and coveralls)

Question

tl;dr:

I'm setting up CI for a project of mine, hosted on github, using tox and travis-ci. At the end of the build, I run converalls to push the coverage reports to coveralls.io. I would like to make this command 'conditional' - for execution only when the tests are run on travis; not when they are run on my local machine. Is there a way to make this happen?

The details:

The package I'm trying to test is a python package. I'm using / planning to use the following 'infrastructure' to set up the tests :

• The tests themselves are of the py.test variety.
• The CI scripting, so to speak, is from tox. This lets me run the tests locally, which is rather important to me. I don't want to have to push to github every time I need a test run. I also use numpy and matplotlib in my package, so running an inane number of test cycles on travis-ci seems overly wasteful to me. As such, ditching tox and simply using .travis.yml alone is not an option.
• The CI server is travis-ci

The relevant test scripts look something like this :

.travis.yml

language: python
python: 2.7
env:
  - TOX_ENV=py27
install:
  - pip install tox
script:
  - tox -e $TOX_ENV

tox.ini

[tox]
envlist = py27

[testenv]
passenv = TRAVIS TRAVIS_JOB_ID TRAVIS_BRANCH
deps =
    pytest
    coverage
    pytest-cov
    coveralls
commands =
    py.test --cov={envsitepackagesdir}/mypackage --cov-report=term --basetemp={envtmpdir}
    coveralls

This file lets me run the tests locally. However, due to the final coveralls call, the test fails in principle, with :

py27 runtests: commands[1] | coveralls
You have to provide either repo_token in .coveralls.yml, or launch via Travis
ERROR: InvocationError: ...coveralls'

This is an expected error. The passenv bit sends along the necessary information from travis to be able to write to coveralls, and without travis there to provide this information, the command should fail. I don't want this to push the results to coveralls.io, either. I'd like to have coveralls run only if the test is occuring on travis-ci. Is there any way in which I can have this command run conditionally, or set up a build configuration which achieves the same effect?

I've already tried moving the coveralls portion into .travis.yml, but when that is executed coveralls seems to be unable to locate the appropriate .coverage file to send over. I made various attempts in this direction, none of which resulted in a successful submission to coveralls.io except the combination listed above. The following was what I would have hoped would work, given that when I run tox locally I do end up with a .coverage file where I'd expect it - in the root folder of my source tree.

No submission to coveralls.io

language: python
python: 2.7
env:
  - TOX_ENV=py27
install:
  - pip install tox
  - pip install python-coveralls
script:
  - tox -e $TOX_ENV
after_success:
  - coveralls

Zigmund

Answer 1

I have a similar setup with Travis, tox and coveralls. My idea was to only execute coveralls if the TRAVIS environment variable is set. However, it seems this is not so easy to do as tox has trouble parsing commands with quotes and ampersands. Additionally, this confused Travis me a lot.

Eventually I wrote a simple python script run_coveralls.py:

#!/bin/env/python

import os

from subprocess import call


if __name__ == '__main__':
    if 'TRAVIS' in os.environ:
        rc = call('coveralls')
        raise SystemExit(rc)

In tox.ini, replace your coveralls command with python {toxinidir}/run_coveralls.py.

Attorn

Nice solution! But it's better to name the script run_coveralls.py, to avoid name clash with actual coveralls package. – Affer

I wonder how run_coveralls.py should most conveniently be included/called? There is an error message can't open file 'run_coveralls.py': [Errno 2] No such file or directory as the file is not copied to the tox' virtual environment. I suppose it should not be included as part of the package. – Makassar

I think I managed to find a solution myself. python {toxinidir}/run_coveralls.py seems to work. – Makassar

You should please remove the import sys, because it is not required for your solution... – Furunculosis

See citeproc-py for an example tox.ini and a more advanced coveralls.py. – Attorn

Answer 2

An alternative solution would be to prefix the coveralls command with a dash (-) to tell tox to ignore its exit code as explained in the documentation. This way even failures from coveralls will be ignored and tox will consider the test execution as successful when executed locally.

Using the example configuration above, it would be as follows:

[tox]
envlist = py27

[testenv]
passenv = TRAVIS TRAVIS_JOB_ID TRAVIS_BRANCH
deps =
    pytest
    coverage
    pytest-cov
    coveralls
commands =
    py.test --cov={envsitepackagesdir}/mypackage --cov-report=term --basetemp={envtmpdir}
    - coveralls

Amboise

Your documentation link is now broken, but man did this save my day. – Anfractuosity

Answer 3

I am using a environmental variable to run additional commands.

tox.ini

commands =
    coverage run runtests.py
    {env:POST_COMMAND:python --version}

.travis.yml

language: python
python:
  - "3.6"
install: pip install tox-travis
script: tox
env:
  - POST_COMMAND=codecov -e TOX_ENV

Now in my local setup, it print the python version. When run from Travis it runs codecov.

Ralphralston

This is the way. {env:POST_COMMAND:python --version} is the embodiment of brilliance, because it silently reduces to a noop when that environment variable is undefined. Yes. – Ingeminate

Answer 4

Alternative solution if you use a Makefile and dont want a new py file:

define COVERALL_PYSCRIPT
import os
from subprocess import call

if __name__ == '__main__':
    if 'TRAVIS' in os.environ:
        rc = call('coveralls')
        raise SystemExit(rc)
    print("Not in Travis CI, skipping coveralls")
endef
export COVERALL_PYSCRIPT

coveralls: ## runs coveralls if TRAVIS in env
    @python -c "$$COVERALL_PYSCRIPT"

In tox.ini add make coveralls to commands

Sanburn