# Automatic tests

## Run the tests
We use [Nix] to enter an environment with all the dependencies.
You can choose between two provided environments:
- `test`: you need to compile Batmen yourself (see [](../user/installation.md)). It is useful in development phases.
- `test_rebuild`: recompile Batmen before entering the environment.

**Batmen manually compiled**:
This will run the whole test suite with the version of `batmen` located in the folder `build/`.
At the root of the project:

```shell
nix-shell -A test
pytest
```

**Recompile automatically:**
If you want Nix to recompile `batmen` before entering the environment:

```shell
nix-shell -A test_rebuild
pytest
```

## Browse the test outputs
Instead of reading the (long) outputs from the terminal, you can open `test-out/test_report.html` in a browser to see the test results.

## Run an individual test
`pytest -k EXPRESSION` only runs the tests matching the given substring expression.

For example, `pytest -k user` will only run the tests whose name contain "user".

## How does the testing work?
pytest interprets all the functions starting with `test_` as tests. For one of these functions: 

0. nix-shell puts you into an environment where batsched, batsim, robin, redis, etc. are available (code in `default.nix`)
1. pytest generates combinations of test input (code in `test/conftest.py`)
2. for each combination of inputs:
   1. pytest generates a [robin] input file
   2. pytest generates batsim and batmen input files if needed
   3. pytest executes [robin] or [robintest] on the generated file

Finally, we have special tests (in the file `test/test_zexpected_output.py`) verifying that the outputs of the previous tests correspond to the expected outputs. 

## Debugging a failed test
You can manually rerun a test with robin:
```
robin test-instances/FAILING-TEST.yaml
```

You can also run batsim and batsched in different terminals:
``` bash
# feel free to hack these files — e.g., prepend commands with gdb, valgrind...
./test-out/FAILING-TEST/cmd/batsim.bash
./test-out/FAILING-TEST/cmd/sched.bash
```

## Extra dependency
Some tests use a tool from the library `batmen-tools` developped separately.
To do update it to the latest version:

```bash
cd batmen/test
curl -O https://gitlab.irit.fr/sepia-pub/mael/batmen-tools/-/raw/main/distance_batsim_output.py
```

[robin]: https://framagit.org/batsim/batexpe
[robintest]: https://framagit.org/batsim/batexpe
[Nix]: https://nixos.org/nix/