4.17. Regression Testing

Viskores has hundreds of built-in regression tests that test the functionality of the entire Viskores infrastructure on new platforms. In this chapter, we discuss how to run regression tests in Viskores and how to create new regression tests.

Did You Know?

Viskores’s regression test infrastructure is enabled by default if compiling from a git clone. If you do not need regression tests and want a faster compile time, you can disable it using the CMake configuration variable described in Section 1.2.2 (Configuring Viskores).

4.17.1. Running Regression Testing

This section details how to run Viskores’s regression tests. First, we explore how to use ctest to run these tests. ctest is the easiest option for running regression tests because it automatically sets several arguments required by the testing infrastructure. Second, we give an overview of how to run the regression tests without using ctest and list the primary command-line arguments for doing so.

4.17.1.1. Regression Testing Using ctest

The following code examples show how to run the regression tests in Viskores using ctest. Example 4.178 shows how to run all the enabled regression tests in Viskores.

Example 4.178 Running all regression tests (Unix commands).

cd viskores-build
ctest

You can get a list of all available tests by giving ctest the -N option, which suppresses actually running the tests (see Example 4.179).

Example 4.179 Listing all available regression tests (Unix commands).

cd viskores-build
ctest -N

Tests can be selected by using the -R option with ctest. The -R option is followed by a string or regular expression that matches the names of tests to run (see Example 4.180).

Example 4.180 Running a single regression test (Unix commands).

cd viskores-build
ctest -R SystemInformation

Verbose testing output can be selected by using the -V option with ctest. The -V option causes the tests to print the underlying command used to launch each test, along with detailed test progression information (see Example 4.181).

Example 4.181 Running a single regression test with verbose output (Unix commands). The verbose output first gives the exact command used to run the regression test, along with detailed test progression information.

cd viskores-build
ctest -R -V SystemInformation

Common Errors

Some of the regression tests in Viskores use data files stored in Git LFS. These files are automatically pulled when the Viskores repository is cloned. However, if the device on which you are compiling does not have Git LFS installed, these unit tests will fail.

4.17.1.2. Regression Testing Without ctest

It is also possible to run Viskores regression tests without using ctest. This can be accomplished by running individual unit-test wrappers located in the <path/to/viskores/build>/bin directory. These tests require specific command-line options to run correctly.

Example 4.182 shows how to run a specific rendering test by passing the locations of the Viskores data and baseline directories.

Example 4.182 Running a single regression test without calling ctest (Unix commands).

UnitTests_viskores_rendering_testing \
  UnitTestMapperVolume \
  --data-dir=<path/to/viskores>/data \
  --baseline-dir=<path/to/viskores>/baseline

4.17.2. Creating Regression Tests

This section details the process and expectations for new regression tests in Viskores.

4.17.2.1. How to Add Data to Viskores

Viskores uses Git LFS for all regression test data. To download or add test data to Viskores, you must have Git LFS installed. Once installed, add unit-test data to the data directory in the Viskores repository. Data in this directory is classified according to its type: structured or unstructured.

Example 4.183 Adding test data to the Viskores repository (Unix commands).

cd viskores-src-dir
cd data/data/<data-type>
git add <file-name>