test-runner

# clean-test This tool can run other test programs and combine the results. The tool was conceived in [clean-platform#17](https://gitlab.com/clean-and-itasks/clean-platform/-/issues/19). ## Interface with tests It is assumed that the other programs adhere to the interface described in `Testing.TestEvents`. For [Gast][] tests, this can be done easily with the `exposeProperties` function. For example: ```clean Start w = exposeProperties [OutputTestEvents] [Tests 500000] [ EP prop_one , EP prop_two , EP prop_three ] w ``` Here, `prop_one`, `prop_two` and `prop_three` are Gast properties (not necessarily of the same type). `exposeProperties` generates a command-line interface. `cleantest` can run this kind of test programs (and others) by adding their executables with the `-r` flag, e.g.: ```bash cleantest -r testModuleA -r testModuleB ``` ## Options The tool has several command-line options; check `cleantest --help` for up-to-date details. Some options are explained here. ### `--parallel-job INDEX/TOTAL` This option allows you to parallelise tests over multiple machines. It was added to be used together with [GitLab parallel CI jobs](https://docs.gitlab.com/ee/ci/yaml/README.html#parallel). Concretely, set `TOTAL` to the number of parallel tests you want to run, and run one instance of `cleantest` for `INDEX=1` to `INDEX=TOTAL`. For example, to parallelise over three machines, run: ```bash cleantest --parallel-job 1/3 ... cleantest --parallel-job 2/3 ... cleantest --parallel-job 3/3 ... ``` In GitLab CI, you can use something like: ```yaml test: parallel: 2 script: - cleantest --parallel-job $CI_NODE_INDEX/$CI_NODE_TOTAL ... ``` Of course you should try to limit the overhead in such jobs, for example by pre-building the test in an earlier job and sharing it with the `test` job using [artifact passing](https://docs.gitlab.com/ee/ci/yaml/README.html#artifacts). Nevertheless, creating a container for the job will always carry some overhead, so `parallel` should only be used for large test suites. `cleantest` will run `--list` on the tests programs to get a list of all tests in the suite. It will then run the test programs with `--run` flags to select a subset of the tests that were listed with `--list`. Each parallel job selects a different subset. Because of this implementation, `--parallel-job` only works with tests that support the `--list` and `--run` command-line options. Importantly this is not the case for `testproperties` when used with `-r` (to immediately run generated tests). The following will not work: ```bash cleantest --parallel-job 1/3 -r testproperties --options '...;-r' ``` Instead, you should first only generate and compile the tests, and then run `cleantest` on the compiled tests: ```bash testproperties ... -c cleantest --parallel-job 1/3 $(ls _Tests.* | sed 's:^:-r ./:') ``` [Gast]: https://gitlab.com/clean-and-itasks/gast

#### 2.0.6 - Chore: upgrade system dependency. - Chore: upgrade generic-print dependency. #### 2.0.5 - Fix: issues with processing command line options. #### 2.0.4 - Chore: support base 2.0. #### 2.0.3 - Enhancement: do not shorten lines when printing undecodable client output. #### 2.0.2 - Chore: accept clean-platform ^v0.3 and ^v0.4 as dependency. #### 2.0.1 - Fix: update platform/property-tester versions for windows fixes to processes/files/system environments. ## 2.0.0 Initial version.