Flash-X Testing Framework (FlashXTest)

Testing and maintainence of Flash-X source code is implemented using the FlashXTest python and command line utility. The utility can be installed directly from its repository.

After cloning the repository type,

./setup develop

in the root folder and it will install FlashXTest in your python environment and copy the command line script to your ~/.local/bin. FlashXTest won’t mainpulate your environment PATH variable so you may have to manually update your PATH in ~/.bashrc by adding

export PATH="$PATH:~/.local/bin"

Once this is done you can type flashxtest --help in your terminal and should be able to see following

Usage: flashxtest [OPTIONS] COMMAND [ARGS]...

  Python CLI for Flash-X Testing Utility

Options:
  --help  Show this message and exit.

Commands:
  add     Add a test from simulation directory
  init    Initialize test configuration
  remove  Remove a test from test suite
  run     Run a list of tests from xml file
  view    Launch webviewer

To setup your local testing infrastructure either create an empty folder in your desired location or go directly to your Flash-X root directory and run

flashxtest init -z <path-to-Flash-X> -s <site>

This will create a config file and a /.fxt folder which will be used to manage your test configuration.

Test information should be provided in individual Simulation/SimulationMain directories. Each simulation directory should contain a tests folder which contains a test.toml configuration file and *.par files pertaining to individual tests. Here is an example of a sample test.toml located in Simulation/SimulationMain/incompFlow/DeformingBubble/tests

# TOML file for DeformingBubble test
# TODO: handle string replacement feature used in xml files

[default]
    testNode = "UnitTest/incompFlow/DeformingBubble/2D/AMReX"
    setupOptions = "-auto -2d -nxb=16 -nyb=16 +amrex +parallelIO"
    numProcs = 1

[user-test]
    testNode = "UnitTest/incompFlow/DeformingBubble/2D/pm4dev"
    setupOptions = "-auto -2d -nxb=16 -nyb=16 --index-reorder +pm4dev -gridinterpolation=native +serialIO"
    numProcs = 2
    parFile = "test_user.par"

The keys default and user-test provide configuration of two seprate tests for this particular simulation. To add one of the tests in your local testing suite type

flashxtest add incompFlow/DeformingBubble -t <test-key>

This will create a testInfo.xml which will translate the information under <test-key> into an xml node. A jobFile will also be created which will contain the value of testNode provided in test.toml. Add more tests by repeating the flashxtest add command. An already populated set of testInfo.xml and jobFile is available in the Tests folder of FlashXTest repository, and contains all the tests that currently run by the maintainers.

To deploy a test run, type

flashxtest run

in the directory containing your config, testInfo.xml, and jobFile. The flashxtest run command will query all the nodes provided in jobFile against testInfo.xml and will run tests based on listed configuration.

There maybe situations where breaking up the jobFile is desired to organize test deployment. This is easily handled by the flashxtest run command. Simply breakup the jobFile to mulitple files with your desired naming convention and type

flashxtest run <filename1> <filename2> <filename3> ....

This project is currently under development. Please file bugs/issues at: https://github.com/Flash-X/Flash-X-Test