Automatic tests

Run the tests

We use Nix to enter an environment with all the dependencies. You can choose between two provided environments:

  • test: you need to compile Batmen yourself (see Installation). It is useful in development phases.

  • test_rebuild: recompile Batmen before entering the environment.

Batmen manually compiled: This will run the whole test suite with the version of batmen located in the folder build/. At the root of the project:

nix-shell -A test
pytest

Recompile automatically: If you want Nix to recompile batmen before entering the environment:

nix-shell -A test_rebuild
pytest

Browse the test outputs

Instead of reading the (long) outputs from the terminal, you can open test-out/test_report.html in a browser to see the test results.

Run an individual test

pytest -k EXPRESSION only runs the tests matching the given substring expression.

For example, pytest -k user will only run the tests whose name contain “user”.

How does the testing work?

pytest interprets all the functions starting with test_ as tests. For one of these functions:

  1. nix-shell puts you into an environment where batsched, batsim, robin, redis, etc. are available (code in default.nix)

  2. pytest generates combinations of test input (code in test/conftest.py)

  3. for each combination of inputs:

    1. pytest generates a robin input file

    2. pytest generates batsim and batmen input files if needed

    3. pytest executes robin or robintest on the generated file

Finally, we have special tests (in the file test/test_zexpected_output.py) verifying that the outputs of the previous tests correspond to the expected outputs.

Debugging a failed test

You can manually rerun a test with robin:

robin test-instances/FAILING-TEST.yaml

You can also run batsim and batsched in different terminals:

# feel free to hack these files — e.g., prepend commands with gdb, valgrind...
./test-out/FAILING-TEST/cmd/batsim.bash
./test-out/FAILING-TEST/cmd/sched.bash

Extra dependency

Some tests use a tool from the library batmen-tools developped separately. To do update it to the latest version:

cd batmen/test
curl -O https://gitlab.irit.fr/sepia-pub/mael/batmen-tools/-/raw/main/distance_batsim_output.py